The Right Losses for the Right Gains: Improving the Semantic Consistency of Deep Text-to-Image Generation with Distribution-Sensitive Losses
Authors: Mahmoud Ahmed
Omer Moussa
Ismail Shaheen
Mohamed O Abdelfattah
Amr Adel Abdalla
Marwan Eid
Hesham Mohamed Eraqi
Mohamed N. Moustafa
Venue: Submitted to ICLR 2023 (2023)
Keywords: Generative Adversarial Networks
Attention
GAN
Text to Image
Contrastive learning

Abstract: One of the major challenges in training deep neural networks for text-to-image generation is the significant linguistic discrepancy between ground-truth captions of each image in most popular datasets. The large difference in the choice of words in such captions results in synthesizing images that are semantically dissimilar to each other and to their ground-truth counterparts. Moreover, existing models either fail to generate the fine-grained details of the image or require a huge number of parameters that renders them inefficient for text-to-image synthesis. To fill this gap in the literature, we propose using the contrastive learning approach with a novel combination of two loss functions:  fake-to-fake loss to increase the semantic consistency between generated images of the same caption, and fake-to-real loss to reduce the gap between the distributions of real images and fake ones. We test this approach on two baseline models: SSAGAN and AttnGAN (with style blocks to enhance the fine-grained details of the images.) Results show that our approach improves the qualitative results on AttnGAN with style blocks on the CUB dataset. Additionally, on the challenging COCO dataset, our approach achieves competitive results against the state-of-the-art Lafite model, outperforms the FID scores of SSAGAN and DALL-E models by 44% and 66.83% respectively, yet with only around 1% of the model size and training data of the huge DALL-E model.

Under review as a conference paper at ICLR 2023
THE RIGHT LOSSES FOR THE RIGHT GAINS: IMPROV- ING THE SEMANTIC CONSISTENCY OF DEEP TEXT-TO- IMAGE GENERATION WITH DISTRIBUTION-SENSIIII LOSSES
Anonymous authors
Paper under double-blind review
ABSTRACT
One of the major challenges in training deep neural networks for text-to-image generation is the significant linguistic discrepancy between ground-truth captions of each image in most popular datasets. The large difference in the choice of words in such captions results in synthesizing images that are semantically dissimilar each other and to their ground trouhd-truth counterparts. Moreover, existing models ei- ther fail to generate the fine-grained details of the image or require huge number of parameters that renders them inefficient for text-to-image synthesis. To fill this gap in the literature, we propose using the contrastive learning approach with a novel combination of two loss functions: fake-to-fake loss to increase the seman- tic consistency between generated images the same caption, and fake-to-real loss reduce the gap between the distributions of real images and fake ones. We test this approach on two baseline models: SSAGAN and AttnGAN (with style blocks enhance the fine- ine-grained details of the images.) Results show that our approach improves the qualitative results on AttnGAN with style blocks on the CUB dataset. Additionally, on the challenging COCO dataset, approach achieves competitive results against the state-of-the the FID scores of SSAGAN and DALL-E models by 44% and 66.83% respec- tively, yet with only around 1% of the model size and training data of the huge DALL-E model.
Lafite model, outperforms
INTRODUCTION
The main aim behind the Text-to-Image generation (T2I) problem is to synthesize high-quality, a challenging com-
photo-realistic images that semantically reflect input textual descriptions. puter vision problem that has many applications, including computer-aided design, image editing, and art generation. Most recent attempts at this problem utilize Generative Adversarial Networks (GANs) as the backhone model. Text-conditioned GANs have proven to be a powerful method to generate high-quality images that are semantically consistent with input captions. In practice, such models generate images that are significantly different from the ground truth. That's because, in most datasets, each image has several human-typed captions that are highly diverse in terms of con- tent and structure. Also, models have learn understand domains: textual description and visual description.
Most models in the literature either lack the details the generated images or generate fine details in the image but with less accurate match to the textual description. The former problem happens due to employing loss function that ensures sentence and word level matching between the image and the text such as the DAMSM loss (Xu et al.) 2018 Ye et al.) 2021] Zhu et al., 2019) without designing a generator capable of ensuring details are generated the image at both fine and coarse grained levels. This leads mainly washed images that have the general structure that matches the text (e.g., a bird with red wings) without the details that makes the image looks real such as the feathers and the color of the eyes. Moreover, these models mostly jet al. (2018); Zhu et al. (2019); Qiao et al.] (2019b) which can increase the aforementioned problem if the progressive growing of the details not done to ensure that image features learned well
multiple generators like Xu

---

Under review as a conference paper at ICLR 2023
at each step. This happens if the early generated image is poor, affecting the latter stages in the generator network.
Another problem with T2I is controllability and forcing the distribution of the generated images to be similar to the real ones. In general, we want similar textual descriptions to have similar im- age features and slight changes in the text to produce corresponding changes to the image without changing irrelevant features. Moreover, we want to push the generated images to look more like the real ones for all the given captions of the same real image. There are two approaches that can be done simultaneously to ensure these constrains; adding proper loss functions and changing the architecture of the generator like Hu et al. (202I).
This work aims to tackle these two problems by studying two directions: using a Style-based gen- crator with the blocks of StyleGAN (Karras et al., 2020) either using the traditional style generator or fusing the style blocks with another architecture like AttnGAN to show how it will improve an already existing architecture. The aim of style blocks is to produce good fine and coarse grained features as well as give controllability of the generation as it did for its traditional use in the uncon- ditional image generation. The other direction is to introduce contrastive learning in two flavors: real-to-fake contrastive learning and fake-to-fake contrastive learning. The purpose of these loss components is force the fake image distribution to be close to the real ones as well and to maximize the similarity between fake images for similar or the same caption. Experimentation with both di- rections lead to very promising results. Adding the style component evidently increase the quality of the generated images but produces low variability while adding the two flavors of contrastive losses gives a significant increase in visual quality of the generated images as well as the quality metrics (e.g., FID) when tried on the SSAGAN network (Hu et al., 202I), making it better than the reported state-of-theeart-moddk on the CUB birds dataset
RELATED WORK
2.1 TEXT-TO-IMAGE GENERATION
Great progress has been recently achieved in text-to-image generation by a large number of promis-
ing studies (Reed et al., 2016 Zhang et al.
2017
2018a: Xu et al., [2018 Hong et al.
2018
Zhang
ct al. 201 8b) Qiao et al. 2019b| ihhu et al. 2019 Yin et al. 2019 Li et al. [019b] Tao et al.
2020
Qiao al. 2019a Li et al. 2019a] Cha al. 2019 Hinz al. 2019 Nouby et al. 2019
Liang
iet al. 2020 Cheng et al.) 2020 Qiao et al. 2201bb Ramesh et al. 2021 Zhang et al., 2017] 2021),
most of which employ GANs as the backbone model. In this section, we provide a summary of some of the most famous and relevant models. AttnGAN (Xu et al. 2018) utilizes attention to compute the similarity between the synthesized images and their corresponding captions using Deep Attentional Multimodal Similarity Model (DAMSM) loss. In this method, both the sentence and word-level information are used to compute the DAMSM loss. The stacked GAN architecture, proposed by Zhang et al. (2017), generates images incrementally from low-resolution to high-resolution. DM- GAN (Zhu et al. 2019) generates high-quality images using a dynamic memory GAN which refines the initial generated images. It then employs memory writing gate to give more weight to rele- vant words and a response gate to enhance image representations accordingly. SD-GAN (Yin et al.) 2019) uses Siamese structure that takes pair of captions as input and employs contrastive loss to train the model. For fine-grained image generation, SD-GAN adopts conditional batch normal- ization. Contrastive loss is also utilized in XMC-GAN (Zhang et al. 2021) but, unlike SD-GAN, they reduce training complexity by not requiring mining for information negatives. CP-GAN (Liang al. 2019) adopts an object-aware image encoder together with a fine-grain fine-grained discriminator for higher-quality image generation. While it achieves promising Inception Score (Salimans et al. 2016), has been shown to perform poorly when evaluated with a stronger FID (Heusel et al. (017) metric. These approaches utilize several generators and discriminators to ultimately synthesize im- ages at high resolutions. Other approaches have proposed inferring semantic layouts and explicitly generating different objects in hierarchical models (Hong et al. 2018 Hinz et al.) 2019 Koh et al.) 2021). In such models, generation is a multi-step process that requires more detailed labels (c.g., segmentation maps and bounding boxes), which represents significant drawback. To tackle these issues, we employ a single generator and discriminator architecture in our model that is end-to-end trainable and generates much higher quality images.
2

---

Under review as a conference paper at ICLR 2023
2.2 STYLE BLOCKS
We discuss style-based image generation, focusing on the advances of the StyleGAN architecture. The StyleGAN (Karras al., 2019) generator consists of two main components: the mapping and the synthesis networks. First, the mapping network transforms a noise vector Z into a latent space W; then, it applies affine transformations on W to generate styles A. This style code A is later used to enable scale-specific control of the synthesis process allowing for better disentanglement, enabling better control over the produced features. The synthesis network the second major part of the StyleGAN generator, and it is responsible for generating the images. It consists of style blocks, where each block controls a specific level of details in the image based on style A. One of the regularization methods for style-based generators is style mixing. It works by feeding sampled styled codes A into different layers of the synthesis network independently at inference time to generate an image. The original StyleGAN (Karras et al. 2019) architecture used AdaIN to achieve style mixing; however, this resulted in artifacts in the generated images. In StyleGAN2 (Karras et al.] (020), this problem tackled by basing the normalization maps. In other words, StyleGAN2 applies modulation and demodulation to the convolution weights within each style block, an alternative that removes artifacts while maintaining full controllability. Progressive Growing (Karras et al. 2017) was used for stabilizing high-resolution image generation. but it has its own characteristic artifacts and can impair shift equivariance. StyleGAN2 (Karras al., 2020) tackled this problem by incorporating a multi-scale training technique. They tried different variations of the generator and discriminator to achieve that high-resolution synthesis, which was originally inspired by MSG-GAN (Karnewar& & Wang. 2020) architecture. Based on an ablation the authors did, the winner network is using a skip generator and residual discriminator.
the expected statistics of the incoming feature
2.3 TEXT-IMAGE CORRESPONDENCE
Part of the text-to- rext-to-image generation problem is to maximize the semantic correspondence for image- text pairs by learning their joint representations. To that end. Deep Attentional Multimodal Similar- ity Model (DAMSM) (Xu et al. 2018) loss is used to learn such low-level text-image representations. The idea behind DAMSM loss is to train an image encoder and a text encoder simultaneously to en- code certain words from the captions together with their corresponding sub-regions in the image to a common semantic space to ultimately compute the loss for image synthesis accordingly.
2.4 TEXT AND IMAGE ENCODERS
For text embeddings, Bi-directional Long Short-Term Memory (LSTM) (Schuster & Paliwal 1997) has been recently utilized to extract semantic vectors from input captions. In Bi-directional LSTM, the semantic meaning of each word is represented by concatenating its two hidden states (one for each direction). Meanwhile, the global sentence vector is produced by concatenating the last hid- den states from the Bi-directional LSTM. As for image encoding, most models in the literature use Convolutional Neural Networks (CNNs) to extract semantic vectors from images. The local image features of each image are learned by the intermediate layers of CNNs, while global features are learned from later layers. The Inception-v3 (Szegedy al. 2016) model, pre-trained on ImageNet (Russakovsky et al. 2015), is commonly used as the image encoder, where local features are ex- tracted from intermediate layers and global ones extracted from the last average pooling layer. Ultimately, these image features are encoded together with text embeddings to common feature space, where each image region is mapped visual feature vector.
2.5 CONTRASTIVE LEARNING
Contrastive learning has proven very effective in self-supervised representation learning of visual
features through various contrastive methods (Arora et al. 2019 Chen al., 2020b He al. 2020
Chen et al. 2020cl Khosla et al., 2020 Tian et al. 2020b] Robinson et al. 2020 Chuang et al.
2020 Hassani & Khasahmadi 2020 Henaft 2020 Kalantidis et al.) 2020 Misra & Maaten 2020
Tian et al. 2020al Wang & Isola 2020 Three major findings have been presented by SimCLR
(Chen et al. 2020b) to learn better representations. First, introducing a composition of data aug- mentations. Second, adding non-linear transformation (learnable) between the contrastive loss and the representation. Third, increasing the batch size and training step. Similar to Ye et al. (2021), we adopt this simple contrastive learning framework. We do with relatively small computational
3

---

Under review as a conference paper at ICLR 2023
cost and simple implementation following the first proposal by SimCLR. We improve the quality of text representations by pushing together the captions that describe the same image and pushing away those that describe different ones.
METHODOLOGY
3.1 TEXT AND IMAGE ENCODERS
AttnGAN suggested a pair of encoders, one for the text and the other for image to be trained on the DAMSM loss in order to learn the suitable text and image encodings of the same space. The idea used here is by training a simple Bi-LSTM network along with pre-trained sub-model from the Inception V3 model followed by linear layer mapping the image encoding the same dimensions of the text encoder. Using this combination allows the Bi-LSTM to learn suitable mapping from the pure tokenized text to the image encoding space representation. We tried improving on this solution by replacing the LSTM with BERT encoding. However, the models failed to converge since the layer after the image encoder was not enough to map the image encodings to the BERT encodings. Therefore, for the rest of our paper, we decided to the pre-trae pre-trained Bi-LSTM provided by Ye et al. 20011)
3.2 ATTNGAN STYLE-BASED ARCHITECTURE
Due to the state-of-the-art results achieved using StyleGAN in style-based image generation, we decided to use a combination of the style blocks with the attention mechanism of AttnGAN. The modified architecture can is shown in Figure []] The combination done by adding a mapping network composed of eight fully connected layers to map the text encoding and the random noise to a disentangled space W. The rest of the generator architecture is based on replacing the normal generator block from AttnGAN with StyleGAN2 blocks. Similarly, the discriminator blocks of AttnGAN are replaced with the Style based discriminator blocks.
Attentional Generative Network
style Hpectin
Dyde
Syle Edok
Platt
Oye Beok
Zlett
Sale Dock3
Image Cocsder
Socadoo
256x25643
G4XD4x3
128
Figure 1: AttnGAN with Style Blocks
3.3 CONTRASTIVE LOSS ON SSAGAN ARCHITECTURE
SSA-GAN (Hu al. 2021) follows the one generator-discriminato pair architecture to tackle the issue of image quality in stacked generaaor-discriminator architectures. The core element of this framework is the SSA-CN block, which uses Semantic-Spatie Condition Batch Normalization and mask predictor to fuse the text and image features effectively and deeply.
Each SSA-CN block consists of an up-sample block, a residual block, a mask predictor, and a Semantic- Spptial Condition Batch Normalization (SSCBN) block. It predicts mask map that indi- cates which parts of the current image feature maps still need to be reinforced with text information so that the refined image feature maps are more semantically consistent with the given text. The
4

---

Under review as a conference paper at ICLR 2023
CSA-Ch
Came
Oute
Con
Figure SSAGAN Contrastive Loss
SSCBN achieves semantic and spatial conditional batch normalization, which leads to accurate and deep text and image features fusion needed.
In this paper, we introduce two contrastive losses to the current SSA-GAN (Hu et al. 2021) architec- ture (as shown in Figure 2). We define the fake-to-fake contrastive loss between a pair of generated images using semantically similar captions. This allows us to increase the semantic consistency between images generated from related descriptions, while maximizing the distance between those generated from different descriptions.
calculating the real-to-fake contrastive loss between the generated images and the real ones, aiming at reducing the semantic difference between our generated images and the ground truth. (as shown in Figure 3
Similarly,
Combining these contrastive learning methods, we are able to generate semantically consistent im- ages when given semantically similar captions; in addition, also decreased the discrepancy be- tween the distributions of the real and fake images.
Irage
Img
Enoode
Fenture
Generator
Destmirato:
MJ.1)
Image Urezder
Features
Image Frooder
Img Festures
Noins (Q2.))
take
Feuture
Desimirato:
Figure 3: Contrastive Loss Architecture

---

Under review as a conference paper at ICLR 2023
3.4 Loss FUNCTIONS
We use the normal adversarial loss for both the generator and the discriminator networks. Any of the other loss functions are added on top of the generator loss. We also use the DAMSM Loss defined in (Xu et al. 2018) as is with only changes to the weight applied to it. Next, we define three other loss functions.
3.4.1 FAKE-TO- TAKE-TO-REAL CONTRASTIVE LOss
The fake-to-real contrastive loss introduced to minimize the distance between the encoding of the ground truth image and the encodings of the synthesized images ensure images similar the real image are being generated and vice versa.
We utilize the Normalized T'empertuuresccale Cross-Entropy Loss (NT-Xent) (Sohn 2016 Wu et al. 2018 Chen et al. 2020a) as the contrastive loss. For a pair of fake and real images a and b, let stm aa a b/ |||]]]]]]] || denote the dot product between la normalized and b. The loss function for the ith sample is as follows:
exp(sim((u, (1; uz)/T) Lezseppppsing Ug uz)/T)
L(t) = loo 2NN k=
(1)
where the ith and jth samples make the positive pair, 1 is an indicator function whose value is and only o i, T temperature parameter, and the batch size. The overall contrastive loss is computed across all positive pairs in mini-batch, which is computed as follow:
1 Le 2N
1 L().
#=
3.4.2 FAKE-TO-FAKE CONTRASTIVE LOSS
The fake-to-fake contrastive loss is adopted to minimize the distance between the encodings of generated images that conditioned two captions related the same image and maximize those conditioned on captions related to different images. This approach ensures that the produced images are more similar each other for the input captions related the same image. In a very similar fashion, We use the NT-Xent loss introduced earlier in Equation in order to calculate the overall fake-to-fake contrastive loss.
3.4.3 IMAGE RE-CAPTIONING LOss
The previous two losses aim to have the model converge on having similar features between the images of the same of captions and between the real and the fake images. The third loss aims to verify that the image, when fed to a captioning model, will have the same caption features as the original model. Let Fe be the captioning model, the loss can be written as:
Ler (Faaae)), Ca)
(3)
where Fake; is the fake image, C₁ is the correct caption, and L. is the normal Cross Entropy Loss.
3.4.4 OVERALL LOSS FUNCTION
After the pretraining of the image encoder and the text encoder using the DAMSM Contrastive Loss functions, the image encoder and text encoder spaces should be able to produce similar repre- sentations (image and text projected to similar space). Let T-G be the generator loss, I-AAMSM be the DAMSM loss, T.p be the recaptioing loss, T-CR he the contrastive real-to-fake loss (F2R), LFF be the contrastive fake-to-fake loss (F2F) illustrated in figure Then, the overall loss function is:
== Lo A IDAMSM X2 LoR Ag Ler Xa Lop

---

Under review as a conference paper at ICLR 2023
Where A1... b the weights of the losses. We turn of the losses don't want by setting their weight to zero so that we can test the effect individual loss components. Our typical values for these weights are X₂ A₃ 0.2, 24 L, X is about 0.05 for the SSA-GAN architecture and for the Style based architecture.
4 EXPERIMENTAL SETUP
4.1 DATASETS
The Caltech-UCSD Birds 200 (CUB-200, or CUB for short) (Wah et al. (201T) is an image dataset of 200 bird species. The CUB dataset has almost 13k images of birds belonging to 200, mostly North American, bird species and textual description for each image. CUB has one class of images, which made it casier to train on and provided a simpler mapping space for the generator network.
The Mícrosoft Common Objects in Context (MS COCO, or COCO for short) dataset (Lin al., 2014) is a large-scale object detection, segmentation, keypoint detection, and captioning dataset. The dataset consists of 328K images of 80 object categories. The numerous COCO object classes make it more challenging than the CUB-200 dataset and more difficult for text-to-image models to converge on. Additionally, the hardware requirements and the time needed to train model the COCO dataset are substantially more than those required by the CUB dataset, to an extent that was available to us. Such limitations urged us initially train our different model architectures on the CUB dataset before training them on the COCO dataset. Accordingly, some models were not trained on the COCO dataset; yet AttnGAN and SSAGAN, both with and without the contrastive loss, and SSAGAN with the real-to-fake contrastive loss were trained.
4.2 METRICS
Following previous work, now introduce the Frechet Inception Distance (FID), the Inception Score (IS), and the R-precision metrics we adopted to quantitatively evaluate the performance of our work.
4.2.1 QUALITY METRICS
(Heusel et 2017) is a prevalent evaluation metric used evaluate the quality of generated images from GANs. Specifically, it measures the Frechet distance between the distribution of the synthetic images and the real images in the feature space of a pre-trained Inception v3 network (Heusel al. 2017). The Frechet distance, also known as the Wasserstein-2 distance when the two distributions are normal distributions (Kynkäänniemi et al. 2022), a distance function used to compare the probability distributions of two variables or measure the similarity between curves taking into account the location and ordering of points along the curves. The Frechet distance between two multivariate Gaussian distributions X and Y is:
FID
FID
||
u Tr(Ex + Ey 22,
(5)
where the means and covariance matrices of X and Y are mug, muy and Eg, Ey. respectively, and Tr is the trace of the matrix.
IS Salimans et al. 2016) is another ad-hoe evaluation metric of the quality of image generative models. The IS uses pre-trained Inception v3 network and calculates the conditional probabilities for each generated image p(x-y). To that end, the Hallback-Leibler diverrive is computed for each image between the conditional and marginal class distributions:
KL divergence = p(y|=) (log(p(u(z)) log(p(v)))
(6)
where p(y-x) is the conditional class distribution, and p(y) = Σ. ( is the marginal class distribution, where P is the distribution of generated images. That is, the IS score is computed by:
IS eeppFErrp DR DRI(p(()))))p(()))
7

---

Under review as a conference paper at ICLR 2023
Architecture
IS
FTD.
R-Precision'
AttnGAN
4.36
23.98
58.06
AttnGAN +
16.34
60.52
AtyleIAUtGGAN
1.20
36.82
61.25
StyleAAUnGANN CL (F2F)
3.09
41.32
62.30
Style2AttGGAN CL (R2F)
3.59
38.51
64.53
SSAGAN
4.89
75.67
SSAGAN + CL (F2F)
4.95
9.62
70.49
SSAGAN + CL (R2F4F2F)
5.18
9.12
72.85
SSAGAN + CL (R2F4F2F+R)
5.03
9.16
72.28
Table I:
Comparison between the performance of different models when trained the CUB dataset with different combinations loss functions.
Architecture
I8 f
FID L
R-Precision
AttnGAN
6.08
30.67
83.8
AttnGAN + CL
6.29
26.89
84.24
SSAGAN
7.18
19.37
82.45
SSAGAN + CL (R2F)
7.6
12.08
85.20
SSAGAN CL (R2F F2F)
7.62
10.89
88.21
Table Comparison between the performance of different models when trained on the COCO dataset with different combinations of loss functions.
4.2.2 ACCURACY METRICS
R-precision Xu et al., 2018 standard evaluation measure for ranking retrieval results of system. is defined the top-R retrieved documents that are relevant, where R is the number of relevant documents for the current query. In other words, R- R-precision r/R when there are r relevant documents among the top-R retrieved documents. Similarly, R-precision is used in text-to-image generation tasks to measure the correlation between a generated image and its corresponding text. R-precision is especially important since the FID and the IS cannot reflect the relevance between an image and a text description. Specifically, we use the generated images to query their corresponding text descriptions. The image encoder and text encoder learned in the pre-trained DAMSM are then utilized to extract global feature vectors of the generated images and their text descriptions, which are used compare the Cosine similarities between them. Finally, candidate text descriptions for each image are ranked in descending order of similarity, and the top relevant descriptions are found to compute the R-precision. Each model generates 30,000 images from randomly selected unseen text descriptions compute the IS and the R-precision. The candidate text descriptions for each query image consist of one ground truth (i.e., R 1) and randomly selected mismatching descriptions.
5 RESULTS AND DISCUSSION
We first tackled the T2I generation problem on the CUB dataset because contains birds only, which is easier to tackle than the multi-class COCO dataset. We compare our proposed model against several architectures; however, will focus on the comparison with SSA-GAN because it is the baseline model. The main quantitative results can be shown in Table while the qualitative results can be shown in Figure 3 The variants of our proposed model notably outperform the SSA-GAN model on both FID and IS metrics. Table also shows that adding the Real-to-Fake contrastive loss leads to lower FID and higher and R- R-precision compared to using only Fake-to-Fale contrastive loss. It is also worth noting that the qualitative results of our approach are better than the baseline results. The generated images are more diverse, semantically closer to the captions, and contain fine-grained and coarse details. Table 2 shows our quantitative results the more challenging COCO dataset, while Figure 4 shows the qualitative results. The results on the COCO dataset also show that model significantly outperforms the baseline model. Particularly, the FID score is 11% better than the SSAGAN+CL (R2F) model on the COCO dataset. Comparing this to the 5% enhancement the CUB dataset, conclude that approach performs even better when the
8

---

Under review as a conference paper at ICLR 2023
this bird has wings that are red and has yellow belly
this bird has wings that are blue and has red belly
is a small light gray bird with small head and green crown hape
this has wings that are black and has white belly
this blue bird with white throat breast belly and abdomen
Yellow abdomen a black eye without an ring and beak quite short
AttnGAN CL
AtteGAN Style Blocks
AtlnGAN + Style Blocks CL
SSAGAN
SSAGAN + F2F Loss
SSAGAN F2F Loss F2R Loss
Table 3: Comparison of example images between our approach and baselines on the CUB dataset
problem gets harder. We emphasize that, on the challenging COCO dataset, our approach achieves competitive results against the state-of-the-art Lafite model, outperforms the FID scores of SSA- GAN and DALL-E models by 44% and 66.83% respectively, yet with only around % of the model size and training data of the huge DALL-E model.
5.1 WHY STYLEGAN SCORES ARE Low
As seen consistently in the results tables I and 2 despite that adding style blocks to AttnGAN architecture has a significant impact on the quality and level of details of generated images, the quality metrics (FID and IS) are low compared to AutGAN. After several experiments, we can see that this is because the variance of generated images low compared to the other architectures. While the network focuses on learning to represent both fine and coarse grained details, it learns small number of poses, sizes and shapes of birds and limited set of backgrounds. This leads to a notable decrease in the variance and consequently larger FID values. We attribute this behavior to the fact that disentanglement make the generator focus on the visual features specified in the text like colors and sizes and ignore those that are not in the text like poses and backgrounds. It is important to emphasize that the image- text matching learned well by the network, so the only issue the variability of the outputs. Moreover, the effect of contrastive learning was very evident on the visual quality in the style based architecture.
the noise via the mapping network as well as the style injection

---

Under review as a conference paper at ICLR 2023
SSAGAN + F2F Loss
AtinGAN CL
SSAGAN
SSAGAN + F2F Loss
F2R Loss
clock that on side of tower
close up of boat near land with a cloudy sky
herd of cows that are grazing on the grass
"
Table Comparison of example images between our approach and baselines on the COCO dataset.
CONCLUSION
In this paper, we propose novel approach for text-to-image generation using contrastive learning. Our goal is to address the semantic inconsistency between generated images and their corresponding captions while keeping the fine-grained details and maintaining small model size. We achieve this goal by incorporating mix of two ceerriutioonsenssitte contrastive losses: fake-to-fake and fake-to- real losses. The fake-to- to-eeee loss increases the invariability of generated images linguistic changes and the fake-to-real loss increases image quality and relevance to the text. To test our hypothesis, we use two generative baselines: SSA-GAN and a modified version AttnGAN which style blocks incorporated. Our results show increase in the quality of generated images, surpassing the SOTA FID results using both contrastive learning and style mixing on the CUB datasct. In addition, achieve competitive results the COCO dataset, outperforming the FID score of our baseline SSAGAN model by 44%, and that of DALL-E by 66.83% a with much smaller model capacity and training data.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint 2019.
arXiv:
Miriam Cha, Youngjune L Gwon, and HT Kung. Adversarial learning of semantic relevance in text to image synthesis. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3272-3279, 2019.
Ting Chen, Simon Korbblith, Mohammad Norouzi 0002, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th Inter- national Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020a. URL htmml
http::/ httt:://proceedings www.iinnnsmm.......mmmmmm..m.. .ll..rr..................m www.mmwww.......mmmmmmmmm
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In International conference machine learning. pp. 1597-1607. PMLR, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint (tle:20003.02997 2020c.
10

---

Under review as a conference paper at ICLR 2023
Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao. Rifegan: Rich feature generation for text-to-image synthesis from prior knowledge. In Proceedings of the IEEE/CVE Conference Computer Viston and Pattern Recognition, pp. 10911-10922 2020.
Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775, 2020.
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor. Tell, draw, and repeat: Gen- erating and modifying images based on continual linguistic instruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10304-10312, 2019.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference Machine Learning, pp. 4116-4126 PMLR, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, pp. 4182-4192. PMLR, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. arXiv preprint arXiv: 00686, 2019.
Seunghoon Hong, Dingdong Yang. Jongwook Choi, and Honglak Lec. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7986-7994, 2018.
Kai Hu, Wentong Liao, Michael Ying Yang, and Bodo Rosenhahn. Text to image generation with semantic-spatial aware gan. arXiv preprint arXiv:2104.00567 2021.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances Neural Information Processing Systems, 33:21798-21809 2020.
Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial net- works. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7799-7808, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10/96, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 18:1166611867,, 2020.
Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Text-to-image generation grounded by fine- sree-grained user attention. Proceedings of the IEEE/CVF Winter Conference on Applica- fions of Computer Vision, pp. 237-246, 2021.

---

Under review as a conference paper at ICLR 2023
Tuomas Kynkäänniemi, Tero Karras, Miik Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr), 'echet inception distance. arXiv preprint 2Xiv:2203.06020, 2022.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image genera- tion. Advances in Neural Information Processing Systems, 32, 2019a.
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jian-
feng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12174- 2019b.
Jiadong Liang, Wenjie Pei, and Feng Lu. Cpgan: full-spectrum content parsing generative adver-
sarial networks for text-to-image synthesis. arXiv preprint arXiv:1912.08552, 2019.
Jiadong Liang, Wenjie Pei, and Feng Lu. Cpgan: Content-parsing generative adversarial networks for text-to-image synthesis. In European Conference on Computer Vision, pp. 491-508. Springer, 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Learn, imagine and create: Text-to- image generation from prior knowledge. Advances in Neural Information Processing Systems, 32, 2019a.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1505-1514, 2019b.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.0455 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-253) 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(1 1::677326-6,,, 1997.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao- Yuan Jing, Fei Wu, and Bingkun Bao. Df- gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008. 05865, 2020.
12

---

Under review as a conference paper at ICLR 2023
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020b.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. 2011.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In International Conference Machine Learning, pp- PMLR, 2020.
Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via parammeeiiiic in- stance discrimination. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 3733-3742, Los Alamitos, CA, USA. jun 2018. IEEE Computer Society. doi: 2.11009/CVP..2288.000 10. 1109/CV/V.
URL https://dol www.m//etttteeeeeccyee..rr.o.
Tao Xu, Pengchuan Zhang. Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1316-1324, 2018.
Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to- image synthesis using contrastive learning. arXiv preprint arXiv:2107.02423 2021.
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. Semantics disen- tangling for text-to-imape generation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2327-2336, 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang. and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pp- 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan+++ Realistic image synthesis with stacked generative adversarial net- works. IEEE transactions on pattern analysis and machine intelligence, 1977-19977 2018a.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross modall contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 833-842, 2021.
Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with hierarchically-neste adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6199-62088 2018b.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative ad- versarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5802-5810, 2019.
13


Reviews:
[{"review_id": "yuBYj30kW_P", "reviewer": "Reviewer_m8jC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use contrastive learning to improve the semantic consistency of text-to-image synthesis models. In the proposed approach, the fake-to-fake loss is adopted to increase the semantic consistency between generated images of the same caption, and fake-to-real loss is adopted to reduce the gap between the distributions of real images and fake ones. The approach is built upon SSAGAN and AttnGAN, and experiments are conducted on CUB and COCO datasets.", "summary_of_the_review": "The paper simply combines contrastive learning with text-to-image synthesis frameworks, and it is of limited novelty and contribution.", "strength_and_weaknesses": "Strengths:\n1. This paper proposes to use contrastive learning to improve the semantic consistency of text-to-image synthesis models.\n2. It achieves good results on CUB and COCO datasets.\n\nWeaknesses:\n1. The writing can be improved. The current version is not polished and difficult to follow.\n2. The contribution is limited. Contrastive learning has been explored extensively in previous literature, and only applying the contrastive learning to text-to-image synthesis pipelines is not enough novelty and contribution.\n3. The visual quality of synthesized images is limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "aw5VNdnXCX", "reviewer": "Reviewer_f3ju", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper shows that the adoption of contrastive learning can improve the text-image semantic consistency and the quality of synthetic images, and the adoption of style block can enhance the fine-grained details of the images. The experiments by implementing both methods on AttnGAN and SSAGAN shows the effectiveness of them.", "summary_of_the_review": "See above weaknesses. I am happy to change my rating based on authors' responses. ", "strength_and_weaknesses": "Strength: \n1. Authors show the effectiveness of both style-based technique and contrastive learning used in multi-modal image generation task.\n\n2. Sufficient related works are discussed in the paper.\n\nWeaknesses:\n\n1. The baselines adopted in the paper might not be state-of-the-art, for example, authors do not include XMC-GAN, GLIDE, etc. Also, authors claim that \"their method achieves competitive results against the state-of-the-art Lafite model, outperforms the FID scores of SSA- GAN and DALL-E models by 44% and 66.83% respectively, yet with only around 1% of the model size and training data of the huge DALL-E model\", but there are no corresponding experimental results shown in the paper.\n\n2. The qualitative results on COCO shown in the paper might not be good enough.\n\n3. The novelty of proposed style block is limited, which mainly depends on StyleGAN. Also the paper lacks a sufficient description of the style block. Contrastive losses have been studied in XMC-GAN. Authors might discusses the difference between their method and XMC-GAN\n\n4. Could authors give more details about why the proposed method can greatly improve the performance?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NAXEd04twGX", "reviewer": "Reviewer_FV8R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a text-to-image generation model that leverages several distribution-sensitive losses for better performance under smaller model sizes. The major technical contribution is the design of the \"right\" loss function for this task, which is a linear combination of generative loss, DAMSM, fake-to-real, fake-to-fake, and re-caption loss. The authors propose that the model significantly outperforms DALL-E in terms of FID score but only has 1% of its model size and training data.", "summary_of_the_review": "Overall it's a clear paper. The technical novelty is not the strongest, and the empirical results could be demonstrated in a more convincing way. More rationales on the designs of the loss function, and more thorough comparison with other methods will put this work in a much stronger stance.", "strength_and_weaknesses": "Strength:\n- Overall easy to follow\n- This work shows great empirical results, and the proposed method is compared against multiple metrics\n- Thorough literature review\nWeakness:\n- While the design of the loss function intuitively makes sense, I have concerns about the weights assigned to those losses: 1) the assignment seems quite heuristic, and it would be good if the authors provide more justification on why certain losses should be given a higher weight; 2) the weight of DAMSM is significantly higher than other weights when style-based models are used - this choice will \"dilute\" the other loss functions emphasized by the authors, and I wonder why the authors make this decision\n- It would be nice if style based models are also compared with in the COCO task\n- DALL-E's performance should also be presented in Table 1 and 2\n- In Table 1, SSAGAN actually has a better score in R-precision", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AgJ0tZAZER", "reviewer": "Reviewer_Bi56", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to generate fine-grained details and semantically match images with the text input. Specifically, the authors propose using a StyleGAN-like block with existing generators to improve details synthesis. Besides, the authors propose fake-to-fake and real-to-fake contrastive loss to improve semantic correspondences. They conduct both quantitative and qualitative experiments on CUB and coco. \n", "summary_of_the_review": "This paper aims to synthesize more fine-grained details and semantically match images for the input text by StyleGAN-based modules and contrastive learning. However, the motivation is not convincing and the experiment results are not strong enough. If the authors address all the weaknesses mentioned in Strength And Weaknesses, I will consider raising my rating \n", "strength_and_weaknesses": "### Strength \nThis paper is overall well-written and it is easy to follow. \n\n### Weaknesses \n\n**Novelty**\n\n1. The core idea of this paper is the use of StyleGAN block and contrastive learning. However, these two techniques are not new for image synthesis. The novelty of the proposal is limited. \n\n**Clarity**\n\n1. The motivation is not clear. The authors mentioned in the second paragraph on page 1 that, the use of DAMSM loss leads to washed images without details. However, DAMSM loss is usually used to match text and images, it is not clear why it will hurt details.\n2. The motivation for using fake-to-fake contrastive learning is not clear. For example, given the text ‘dog’, there may be many different kinds of dogs that correspond to this text. Users may also need diverse results given the same caption. Forcing similar results for the same caption may lead to mode collapse.\n3. The motivation for using real-to-fake contrastive learning is not clear. As claimed on page 2, the authors use real-to-fake contrastive learning to push fake distribution close to the real distribution. However, the generative adversarial loss has worked for this motivation.\n\n**Quality**\n\n1. The authors claimed the benefit of generation controllability on Page 2, however, they forget to verify the generation controllability in experiments.\n2. The authors ignore comparisons with recent text-to-image works, e.g., DALLE, and some other diffusion-based text-to-image works, e.g., Imagen, etc. \n3. The results are not strong enough to verify the effectiveness of the proposed approach. For example, in the first case of Figure 3, the proposed approach shows the heavy color bleed issues and generates a red belly for the ‘yellow belly’ text input.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject