Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Authors: Chitwan Saharia
William Chan
Saurabh Saxena
Lala Li
Jay Whang
Emily Denton
Seyed Kamyar Seyed Ghasemipour
Raphael Gontijo-Lopes
Burcu Karagol Ayan‎
Tim Salimans
Jonathan Ho
David J. Fleet
Mohammad Norouzi
Venue: NeurIPS 2022 Accept (2022)
Keywords: text-to-image
generative models
diffusion models

Abstract: We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g., T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment.

Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding
Chitwan Saharia', William Chan', Saurabh Saxena', Lala Liţ, Jay Whang! Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan,
S. Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho!, David J Fleet", Mohammad Norouzi"
{sahariac, williamchan www.rcannnmmooozzzg@gggoooee... {srbs, lala, j jwhang, jonathanho, wwwiidfleet
Google Research, Brain Team
Toronto, Ontario, Canada
Abstract
We present Imagen, text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of tfrtt---rtttttttt FID score of 7.27 on the COCO dataset, without ever training COCO, and human raters find Imagen samples to be par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, introduce DrawBench, comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, GLIDE and DALL-E 2, and find that human raters prefer Imagen over other models side-by-side comparisons, both in terms of sample quality and image-fext alignment.
Introduction
Multimodal learning has come into prominence recently, with text-to-image synthesis [55, 12, 59] and image-text contrastive learning [51, 32, 77] the forefront. These models have transformed the research community and captured widespread public attention with creative image generation [22, 56] and editing applications [21, 43, 36). To pursue this research direction further, we introduce Imagen, text-to-image diffusion model that combines the power of transformer language models (LMs) [15, 54] with high-fidelity diffusion models [28, 29, 16, 43] to deliver an unprecedented degree of photorealism and a deep level of language understanding in text-to-image synthesis. In contrast to prior work that uses only image-text data for model training [e.g., 55, 43], the key finding behind Imagen is that text embeddings from large LMs [54, 15], pretrained text-only corpora, remarkably effective for text-to-image synthesis. See Fig. for select samples.
Imagen comprises frozen T5-XXL [54] encoder to map input text into a sequence of embeddings and a 64x64 image diffusion model, followed by two super-resolution diffusion models for generating
Equal contribution.
Core contribution.
1 DF is also affiliated with the University of Toronto and the Vector Institute
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

imagen
Sprouts in the shape of text 'Imagen' coming of A photo of Shiba Inu dog with backpack riding fairytale book.
high contrast portrait of very happy fuzzy panda dressed chef a high end kitchen making dough.
bike. wearing sunglasses and a beach hat.
There a painting flowers on the wall behind him.
Imagen
Imagenn
Imargen
Teddy bears swimming at the Olympics 400m Butter- fly event.
A cute corgi lives in house made out of sushi.
cute sloth holding small treasure chest. A bright golden glow is coming from the chest.
Imagen
Imagen
A brain riding rocketship heading towards the moon.
A dragon fruit wearing karate belt in the snow.
A strawberry mug filled with white sesame seeds. The mug is floating dark chocolate
Figure 1: Select 1024 x 1024 Imagen samples for various text inputs. We only include photorealistic images in this figure and leave artistic content to the Appendix, since generating photorealistic images more challenging from technical point of view. Figs. A.1 to A.3 show more samples.

---

256 x 256 and 1024 x 1024 images (see Fig. A.4). All diffusion models are conditioned on the text embedding sequence and use classifier-free guidance [27]. Imagen relies on new sampling techniques to allow usage of large guidance weights without sample quality degradation observed in prior work, resulting in images with higher fidelity and better image-text alignment than previously possible.
While conceptually simple and easy to train, Imagen yields surprisingly strong results. Imagen outperforms other methods on COCO [38] with zero-shot FID-30K of 7.27, significantly outper- forming prior work such as GLIDE [43] (at 12.4) and the concurrent work of DALL-E 2 [56] (at 10.4). Our zero-shot FID score is also better than sfte-of--the-a models trained on COCO, e-g.. Make-A Bake-A-Scene [22] (at 7.6). Additionally, human raters indicate that generated samples from Imagen are on- on-par image-text alignment to the reference images on COCO captions.
We introduce DrawBench, a new structured suite of text prompts for text-to-image evaluation. Draw- Bench enables deeper insights through multi-dimennionss evaluation of text-to-image models, with text prompts designed to probe different semantic properties of models. These include composi- tionality, cardinality, spatial relations, the ability to handle complex text prompts or prompts with rare words, and they include creative prompts that push the limits of models' ability to generate highly implausible scenes well beyond the scope of the training data. With DrawBench, extensive human evaluation shows that Imagen outperforms other recent methods [59, 12, 56] by significant margin. We further demonstrate some of the clear advantages of the use of large pre-trained language models [54] over multi-modal embeddings such as CLIP [51] as text encoder for Imagen.
Key contributions of the paper include:
1. We discover that large frozen language models trained only on text data are surprisingly very effective text encoders for text-to-image generation, and that scaling the size of frozen text encoder improves sample quality significantly more than scaling the size of image diffusion model.
2. We introduce dynamic thresholding, new diffusion sampling technique leverage high guidance weights and generating more photorealistic and detailed images than previously possible.
3. We highlight several important diffusion architecture design choices and propose Efficient U-Net, a new architecture variant which is simpler, converges faster and is more memory efficient.
4. We achieve new state-of-the-an COCO FID of 7.27. Human raters find Imagen to be on-par with the reference images in terms of image-text alignment.
5. We introduce Brach new comprehensive and challenging evaluation benchmark for the text-to-image task. On Draw Bench human evaluation, we find Imagen to outperform all other work, including the concurrent work of DALL-E 2 [56].
2 Imagen
Imagen consists of a text encoder that maps text to a sequence of embeddings and a cascade of conditional diffusion models that map these embeddings images of increasing resolutions (see Fig. A.4). In the following subsections, we describe each of these components in detail.
2.1 Pretrained text encoders
Text-to-image models need powerful semantic text encoders to capture the complexity and com- positionality of arbitrary natural language text inputs. Text encoders trained on paired image-text data are standard in current text-to-imaae models; they can be trained from scratch [43, 55] or pretrained on image-text data [56] (e.g., CLIP [51]). The image-text training objectives suggest that these text encoders may encode visually semantic and meaningful representations especially relevant for the text-to-image generation task. Large language models can be another models of choice to encode text for text-to-image generation. Recent progress large language models (e.g.. BERT [15], GPT [49, 50, 7]. T5 [54]) have led to leaps in textual understanding and generative capabilities. Language models are trained on text only corpus significantly larger than paired image-text data, thus being exposed to very rich and wide distribution of text. These models are also generally much larger than text encoders in current image-tex models [51, 32, 83] (c.g. PaLM [11] has 540B parameters, while CoCa [83] has R IB parameter text encoder).
thus becomes natural to explore both families of text encoders for the text-to-image task. Imagen explores pretrained text encoders: BERT [15], T5 [53] and CLIP [48]. For simplicity, we freeze the weights of these text encoders. Freezing has several advantages such as offline computation of embeddings, resulting in negligible computation or memory footprint during training of the text- to-image model. our work, we find that there a clear conviction that scaling the text encoder size improves the quality of text-to-image generation. We also find that while T5-XXL and CLIP
3

---

text encoders perform similarly on simple benchmarks such as MS-COCO, human evaluators prefer T5-XXL encoders over CLIP text encoders in both image-text alignment and image fidelity on DrawBench, a set of challenging and compositional prompts. We refer the reader to Section 4.4 for summary of our findings, and Appendix D.1 for detailed ablations.
2.2 Diffusion models and classifier-free guidance
Here we give a brief introduction to diffusion models; a precise description in Appendix A. Diffusion models [66, 28, 68] are class of generative models that convert Gaussian noise into samples from a learned data distribution via an iterative denoising process. These models can be conditional, for example class labels, text, or low-resolution images [e.g. 16, 29, 62, 61, 78, 43, 56].
diffusion model X9 is trained on a denoising objective of the form
Ex,c,e,1 w At (axx + otE,c) -x1=]
(1)
where (x, c) are data-conditioning pairs, L (1 1)), e NO I). and ar, oz Wy are functions of t that influence sample quality. Intuitively, X, is trained to denoise 2, == a+x + arE into x using squared error loss, weighted to emphasize certain values of t. Sampling such as the ancestral sampler [28] and DDIM [67] start from pure noise 21 ~ (O, and iteratively generate points tr O, that gradually decrease in noise content. These points are functions of the x-predictions x == Xa (Zr, c).
zty...... Ztr, where t >
Classifier guidance [16] is a technique to improve sample quality while reducing diversity in condi- tional diffusion models using gradients from a pretrained model p(c during sampling. Classifier- free guidance [27] is an alternative technique that avoids this pretrained model by instead jointly training a single diffusion model on conditional and unconditional objectives via randomly dropping during training (e.g. with 10% prohability). Sampling is performed using the adjusted x-prediction (Ze - oēo)/ /at, where
éa (o c) weo(2;,)) c) + (1 - ())))
Here, co(z,, e) and (z,) are conditional and unconditional e e-predictions, given by Ep == (z, axxo)/ot, and w is the guidance weight. Setting w 1 disables classifier-free guidance, while increasing w strengthens the effect of guidance. Imagen depends critically on classifier-free guidance for effective text conditioning.
2.3 Large guidance weight samplers
We corroborate the results of recent text-guided diffusion work [16, 56) and find that increasing the classifier-fre free guidance weight improves image-text alignment, but damages image fidelity producing highly saturated and unnatural images [27]. We find that this is due train-test mismatch arising from high guidance weights. At each sampling step f, the x- x-prediction x/j must be within the same bounds training data x, i.e. within [-1, 1]. but we find empirically that high guidance weights cause x- x-predictions to exceed these bounds. This is a train-test mismatch, and since the diffusion model is iteratively applied on its own output throughout sampling, the sampling process produces unnatural images and sometimes even diverges. To counter this problem, we investigate static thresholding and dynamic thresholding. See Appendix Fig. A.31 for reference implementation of the techniques and Appendix Fig. A.9 for visualizations of their effects.
Static thresholding: We refer to elementwise clipping the x-prediction to [-1, 1] static threshold- ing. This method was in fact used but not emphasized in previous work [28]. and to our knowledge its importance has not been investigated in the context of guided sampling. We discover that static thresholding is essential to sampling with large guidance weights and prevents generation of blank images. Nonetheless, static thresholding still results in over-saturated and less detailed images the guidance weight further increases.
Dynamic thresholding: We introduce a new dynamic thresholding method: at each sampling step we set 8 to a certain percentile absolute pixel value in x/. and if s > 1, then we threshold x to the range [-s, s] and then divide by Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing pixels from saturation at each step. We find that dynamic thresholding results in significantly better photorealism well as better image-text alignment, especially when using very large guidance weights.

---

2.4 Robust cascaded diffusion models
Imagen utilizes a pipeline of a base 64 x 64 model, and two text-conditional super-resolttion diffusion models to upsample 64 x 64 generated image into 256 x 256 image, and then 1024 x 1024 image. Cascaded diffusion models with noise conditioning augmentation [29] have been extremely effective progressively generating high-fidelity images. Furthermore, making the super-resolution models aware of the amount of noise added, via noise level conditioning, significantly improves the sample quality and helps improving the robustness of the super-resolution models to handle artifacts generated by lower resolution models [29]. Imagen uses noise conditioning augmentation for both the super-resolution models. We find this to be a critical for generating high fidelity images.
Given conditioning low-resolution image and augmentation level (a.k.a aug level) (e.g., strength of Gaussian noise or blur), we corrupt the lowrresouution image with the augmentation (corresponding to aug level), and condition the diffusion model on aug level. During training, aug oug_level is chosen randomly, while during inference, we sweep over its different values to find the best sample quality. In our case, we use Gaussian noise as form of augmentation, and apply variance preserving Gaussian noise augmentation resembling the forward process used in diffusion models (Appendix A). The augmentation level is specified using aug level E [0, 1]. See Fig. A.32 for reference pseudocode.
2.5 Neural network architecture
Base model: We adapt the U-Net [60] architecture from [42] for our base 64 64 text-to-image diffusion model. The network is conditioned on text embeddings via a pooled embedding vector, added to the diffusion timestep embedding similar the class embedding conditioning method used in [16, 29]. We further condition on the entire sequence of text embeddings by adding cross attention [59] over the text embeddings at multiple resolutions. We study various methods of text conditioning in Appendix 1.3.1. Furthermore, we found Layer Normalization [2] for text embeddings in the attention and pooling layers to help considerably improve performance.
Super-resolution models: For 64 x 64 - 256 x 256 super-resolution, we use the U-Net model adapted from [42, 61]. We make several modifications to this U-Net model for improving memory efficiency, inference time and convergence speed (our variant is 2-3x faster in steps/second over the U-Net used in [42, 61]). We call this variant Efficient U-Ner (See Appendix B. for more details and comparisons). Our 256 x 256 + 1024 x 1024 super-resolution model trains on 64 x 64 + 256 x 256 crops of the 1024 x 1024 image. To facilitate this, we remove the self-attention layers, however we keep the text cross-attention layers which we found to be critical. During inference, the model receives the full 256 256 low-resolution images as inputs, and returns upsampled 1024 x 1024 images as outputs. Note that we use text cross attention for both our super-resolution models.
3 Evaluating Text-to-Image Models
The COCO [38] validation set is the standard benchmark for evaluating text-to-image models for both the supervised [85, 22] and the zero-shot setting [55, 43]. The key automated performance metrics used are FID [26] to measure image fidelity, and CLIP score [25, 51] to measure image-text alignment. Consistent with previous works, we report zero-shot FID-30K, for which 30K prompts are drawn randomly from the validation set, and the model samples generated on these prompts are compared with reference images from the full validation set. Since guidance weight is an important ingredient to control image quality and text alignment, we report most of our ablation results using trade-off (or pareto) curves between CLIP and FID scores across a range of guidance weights.
Both FID and CLIP scores have limitations, for example FID is not fully aligned with perceptual quality [44], and CLIP ineffective at counting [51]. Due to these limitations, we use human evaluation to assess image quality and caption similarity, with ground truth reference caption-image pairs as a baseline. We use two experimental paradigms:
1. To probe image quality, the rater is asked to select between the model generation and reference image using the question: "Which image is more photorealistic (looks more real)?". We report the percentage of times raters choose model generations over reference images (the preference rate).
2. To probe alignment, human raters are shown an image and a prompt and asked "Does the caption accurately describe the above image?". They must respond with "yes", "somewhat", or "no"". These responses are scored as 100, 50, and O, respectively. These ratings are obtained independently for model samples and reference images, and both are reported.
5

---

NEURIPS NEURIP
brown hird and blue bear.
One cal and two dogs silting on the grass.
A sign that says 'NeurIPS..
small blue baok sitting on large red book.
A blue coloured pizza.
wine glass on top of a dog.
G++
Sume G-g-t s
by
A small vessel propelled on water by cars, an engine. Figure 2: Non-cherry picked Imagen samples for different categories of prompts from DrawBench.
A pear out into seven picces
A phoco of confused grizzly bear calculus
arranged ring.
For both cases we use 200 randomly chosen image-captions pairs from the COCO validation set. Subjects were shown batches of 50 images. We also used interleaved "control" trials, and only include rater data from those who correctly answered at least 80% of the control questions. This netted 73 and 51 ratings per image for image quality and image-text alignment evaluations, respectively.
DrawBench: While COCO a valuable benchmark, it is increasingly clear that has a limited spectrum of prompts that do not readily provide insight into differences between models (e.g., see Sec. 4.2). Recent work by [10] proposed a new evaluation set called PaintSkills to systematically evaluate visual reasoning skills and social biases beyond COCO. With similar motivation, introduce DrawBench, comprehensive and challenging set of prompts that support the evaluation and comparison of text-to-image models. DrawBench contains 11 categories of prompts, testing different capabilities of models such as the ability to faithfully render different colors, numbers of objects, spatial relations, text in the scene, and unusual interactions between objects. Categories also include complex prompts, including long, intricate textual descriptions, rare words, and also misspelled prompts. We also include sets of prompts collected from DALL-E [55], Gary Marcus et al. [40] and Reddit. Across these 11 categories, Brachench comprises 200 prompts in total, striking a good balance between the desire for large, comprehensive dataset, and small enough that human evaluation remains feasible. (Appendix C provides more detailed description of DrawBench. Fig. 2 shows example prompts from DrawBench with Imagen samples.)
We use Draw DrahBench to directly compare different models. To this end, human raters are presented with two of images, one from Model A and one from Model each of which has samples. Human raters are asked to compare Model A and Model B on sample fidelity and image-text alignment. They respond with one of three choices: Prefer Model A; Indifferent; or Prefer Model B.
Experiments
Section 4.1 describes training details, Sections 4.2 and 4.3 analyze results on MS-COCO and DrawBench, and Section 4.4 summarizes ablation studies and key findings. For all experiments below, the images are fair random samples from Imagen with no post-processing or re-ranking.
4.1 Training details
Unless specified, we train a 2B parameter model for the x text-to-image synthesis, and 600M and 400M parameter models for 64 x 64 256 x 256 and 256 x 256 - 1024 x 1024 for super- resolution respectively. We use a batch size of 2048 and 2.5M training steps for all models. We use 256 TPU-v4 chips for our base 64 x 64 model, and 128 TPU-v4 chips for both super-resolution

---

Table 1: MS-COCO 256 256 FID-30K. We use a
Table 2: COCO 256 256 human evalua- tion comparing model outputs and orig- inal images. For the bottom part (no people), we filter out prompts containing one of man, men, woman, women, person, people, child, adult, adults, boy, boys, girl, girls, guy, lady, ladies, someone, toddler, (sport) player, workers, spectators.
guidance weight of 1.35 for our 64 x 64 model, and a
guidance weight of 8.0 for our super-resolution model.
Zero-shot FID-30K
Model
FID-30K
AttGAA [79]
35.49
DM-GAN [86]
32.64
DF-GAN [72]
21.42
DM-GAN + CL [81]
20.79
XMC-GAN [84]
9.33
Model
Photorcalism
Alignment t
LAFITE [85]
8.12
Make-A-Scene [22]
7.55
Original 50.0%
Original
91.9 = 0.42 91.4 = 0.44
DALL-E [55]
17.89
Imagen
39.5 + 0.75%
LAFITE [85]
26.94
GLIDE [43]
12.24
No people
DALL-E [56]
10.39
Original
50.0%
92.2 + 0.54
Imagen
43.9 1.01%
92.1 + 0.55
Imagen (Our Work)
7.27
models. We do not find over-fitting to be an issue, and we believe further training might improve overall performance. We use Adafactor for our base 64 64 model, because initial comparisons with Adam suggested similar performance with much smaller memory footprint for Adafactor. For super- resolution models, use Adam as we found Adafactor to hurt model quality in initial ablations. For classifier-free guidance, we joint-train unconditionally via zeroing out the text embeddings with 10% probability for all three models. We train combination of internal datasets, with 460M image-text pairs, and the publicly available LAION-400M dataset [64], with 2 400M image-text pairs. There are limitations our training data, and we refer the reader to Section for details. See Appendix F for more implementation details.
4.2 Results on COCO
We evaluate Imagen on the COCO validation using FID score, similar to [55,43]. Table displays the results. Imagen achieves state of the art zero-shot FID on COCO at 7.27, outperforming the concurrent work of DALL-E [56] and even models trained COCO. Table 2 reports the human evaluation to test image quality and alignment on the COCO validation set. We report results on the original COCO validation set, as well as a filtered version in which all reference data with people have been removed. For photorealism, Imagen achieves 39.2% preference rate indicating high image quality generation. On the with people, there is boost in preference rate of Imagen 43.6%, indicating Imagen's 's limited ability to generate photorealistic people. On caption similarity, Imagen's 's score is on-par with the original reference images, suggesting Imagen's 's ability to generate images that align well with COCO captions.
4.3 Results on DrawBench
Using DrawBench, we compare Imagen with DALL-E2 2 (the public version) [56], GLIDE [43], Latent Diffusion [59], and -Lid-g VQ-GAN [12]. Fig. 3 shows the human evaluation results for pairwise comparison of Imagen with each of the three models. We report the percentage of time raters prefer Model A, Model B,
indifferent for both image fidelity and image-text alignment. We aggregate the scores across all the categories and raters. We find the human raters to exceedingly prefer Imagen over all others models in both image-text alignment and image fidelity. We refer the reader to Appendix E for a more detailed category wise comparison and qualitative comparison.
4.4 Analysis of Imagen
For a detailed analysis Imagen see Appendix D. Key findings are discussed in Fig. and below.
Scaling text encoder size is extremely effective. We observe that scaling the size of the text encoder leads to consistent improvement in both image-text alignment and image fidelity. Imagen trained with our largest text encoder, T5-XXL (4.6B parameters), yields the best results (Fig. 4a).
7

---

Imagen
Imagen
Imagen
VQGAN+CLIP
Imagen
100%
50%
0%
Aligument
Fidelity
Aligument
Fidelity
Alignment
Fidelity
Aligument
Fidelity
Figure 3: Comparison between Imagen and DALL-E 2 [56], GLIDE [43], VQ-GANNCLIP [12]
and Latent Diffusion [59] DrawBench. User preference rates (with 95% confidence intervals) for image-text alignment and image fidelity.
TS Stall
300M
statl Dredhiling
Large
S0004
dysnatic treskeldzg
18
15-00L
FED 10k
20
10
CLIP Scoce
CLIP
CLIP
(a) Impact of encoder size.
(b) Impact of U-Net size.
(c) Impact of thresholding.
Figure 4: Summary of some of the critical findings of Imagen with pareto curves sweeping over different guidance values. See Appendix D for more details.
Scaling text encoder size is more important than U-Net size. While scaling the size the diffusion model U-Net improves sample quality, we found scaling the text encoder size be significantly more impactful than the U-Net size (Fig. 4b).
Dynamic thresholding is critical. We show that dynamic thresholding results in samples with significantly better photorealism and aligument with text, over static or no thresholding, especially under the presence of large classifier-fi free guidance weights (Fig. 4c).
Human raters prefer T5-XXL over CLIP on DrawBench. The models trained with T5-XXI. and CLIP text encoders perform similarly on the COCO validation set in terms of CLIP and FID scores. However, find that human raters prefer T5-XXL over CLIP on DrawBench across all 11 categories.
Noise conditioning augmentation critical. We show that training the super-resolution models with noise conditioning augmentation leads to better CLIP and FID scores. We also show that noise conditioning augmentation enables stronger text conditioning for the super-resolution model, resulting in improved CLIP and FID scores at higher guidance weights. Adding noise to the low-res image during inference along with the use of large guidance weights allows the super-resolution models to generate diverse upsampled outputs while removing artifacts from the low-res image.
Text conditioning method is critical. We observe that conditioning the sequence of text embeddings with cross attention significantly outperforms simple mean or attention based pooling in both sample fidelity as well as image-text alignment.
Efficient U-Net is critical. Our Efficient U-Net implementation uses less memory, converges faster, and has better sample quality with faster inference.
5 Related Work
Diffusion models have seen wide success image generation [28, 42, 62, 16, 29, 61], outperforming GANs in fidelity and diversity, without training instability and mode collapse issues [6, 16, 29]. Autoregressive models [39], GANs [79, 84], VQ-VAE Transformer-base diffusion models have seen remarkable progress in text-to-image [59, 43, 59], including the concurrent
methods [55, 22], and

---

DALL-E [56], which uses a diffusion prior on CLIP text latents and cascaded diffusion models to generate high resolution 1024 1024 images; we believe Imagen is much simpler, as Imagen does not need to learn latent prior, yet achieves better results in both MS-COCO FID and human evaluation on DrawBench, GLIDE [43] also uses cascaded diffusion models for text-to-image, but we use large pretrained frozen language models, which we found to be instrumental to both image fidelity and image-text alignment. XMC-GAN [84] also uses BERT as a text encoder, but we scale to much larger text encoders and demonstrate the effectiveness thereof. The use of cascaded models is also popular throughout the literature [14, 41] and has been used with success in diffusion models to generate high resolution images [16, 29].
6 Conclusions, Limitations and Societal Impact
Imagen showcases the effectiveness of frozen large pretrained language models as text encoders for the text-to-image generation using diffusion models. Our observation that scaling the size of these language models have significantly more impact than scaling the U-Net size on overall performance encourages future research directions on exploring even bigger language models as text encoders. Furthermore, through Imagen we re-emphasize the importance of classifier-fre free guidance, and we introduce dynamic thresholding, which allows usage of much higher guidance weights than seen in previous works. With these novel components, Imagen produces 1024 x 1024 samples with unprecedented photorealism and alignment with text.
Our primary aim with Imagen is advance research on generative methods, using text-to-image synthesis as a test bed. While end-user applications of generative methods remain largely out of scope, we recognize the potential downstream applications of this research are varied and may impact society in complex ways. On the one hand, generative models have a great potential to complement, extend, and augment human creativity [30]. Text-to-image generation models, in particular, have the potential to extend image-editing capabilities and lead to the development of new tools for creative practitioners. On the other hand, generative methods can be leveraged for malicious purposes, including harassment and misinformation spread [20], and raise many concerns regarding social and cultural exclusion and bias [70, 65, 71j. These considerations inform our decision to not to release code or public demo. In future work we will explore framework for responsible externalization that balances the value of external auditing with the risks of unrestricted open-access.
Another ethical challenge relates to the large scale data requirements of text-to- rext-to-image models, which have have led researchers rely heavily on large, mostly uncurated, web-scraped datasets. While this approach has enabled rapid algorithmic advances in recent years, datasets of this nature have been critiqued and contested along various ethical dimensions. For example, public and academic discourse regarding appropriate use of public data has raised concerns regarding data subject awareness and consent [24, 18, 63, 45J, Dataset audits have revealed these datasets tend to reflect social stereotypes, oppressive viewpoints, and derogatory, or otherwise harmful, associations to marginalized identity groups [46, 4]. Training text-to-image models on this data risks reproducing these associations and causing significant representational harm that would disproportionately impact individuals and communities already experiencing marginalization, discrimination and exclusion within society. As such, there are multitude of data challenges that must be addressed before sot-te-- models like Imagen can be safely integrated into user-facing applications. While we do not directly address these challenges in this work, an awareness of the limitations of our training data guide our decision not to release Imagen for public use. We strongly caution against the use eext-to-image generation methods for any user-facing tools without close care and attention to the contents of the training dataset.
Imagen's 's training data was drawn from several pre-existing datasets of image and English alt-text pairs. 400 million examples came from FIT400M, cleaned version of the Alt-Text dataset [33, 31). This data was filtered to removed noise and undesirable content, such as pornographic imagery and toxic language. However, recent audit of another one of our data sources, LAION-400M [64], uncovered a wide range of inappropriate content including pornographic imagery, racist slurs, and harmful social stereotypes
[4]. This finding informs our assessment that Imagen is not suitable for public use at this time and also demonstrates the value of rigorous dataset audits and comprehensive dataset documentation (e.g. [23, 47]) in informing consequent decisions about the model's 's appropriate and safe use. Imagen also relies on text encoders trained on uncurated web-scale data, and thus inherits the social biases and limitations of large language models [5, 3, 52].
While we leave an in-depth empirical analysis of social and cultuaal biases encoded by Imagen to future work, our small scale internal assessments reveal several limitations that guide our decision

---

not to release Imagen at this time. First, all generative models, including Imagen, Imagen, may run into danger of dropping modes of the data distribution, which may further compound the social consequence of dataset bias. Second, Imagen exhibits serious limitations when generating images depicting people. Our human evaluations found Imagen obtains significantly higher preference rates when evaluated on images that do not portray people, indicating a degradation in image fidelity. Finally, our preliminary assessment also suggests Imagen encodes several social biases and stereotypes, including overall bias towards generating images of people with lighter skin tones and a tendency for images portraying different professions to align with Western gender stereotypes. Even when we focus generations away from people, our preliminary analysis indicates Imagen encodes range of social and cultural biases when generating images activities, events, and objects.
While there has been extensive work auditing image-to-ter to-text and image labeling models for forms of social bias (e.g- [8, 9, 71), there has been comparatively less work on social bias evaluation methods for text-to-image models, with the recent exception of [10]. We believe this is critical avenue for future research and we intend to explore benchmark evaluations for social and cultural bias in future work-for example, exploring whether it is possible to generalize the normalized pointwise mutual information metric [1] to the measurement of biases in image generation models. There is also a great need to develop conceptual vocabulary around potential harms of text-to-image models that could guide the development of evaluation metrics and inform responsible model release. We aim to address these challenges in future work.
7 Acknowledgements
We give thanks to Ben Poole for reviewing our manuscript, early discussions, and providing many helpful comments and suggestions throughout the project. Special thanks to Kathy Austin Tarango, and Sarah Laszlo for helping us incorporate important responsible Al practices around this project. We appreciate valuable feedback and support from Elizabeth Adkison, Zoubin Ghahramani, Jeff Dean, Yonghui Wu, and Eli Collins. We are grateful to Tom Small for designing the Imagen watermark. We thank Jason Baldridge, Han Zhang, and Kevin Murphy for initial discussions and feedback. We acknowledge hard work and support from Fred Alcober, Hibaq Ali, Marian Croak, Aaron Donsbach, Tulsee Doshi, Toju Duke, Douglas Eck, Jason Freidenfelds, Brian Gabricl, Molly FitzMorris, David Ha, Philip Parham, Laura Pearce, Evan Rapoport, Lauren Skelly, Johnny Soraker, Negar Rostamzadeh, Vijay Vasudevan, Tris Warkentin, Jeremy Weinstein, and Hugh Williams for giving us advice along the project and assisting with the publication process. We thank Victor Gomes and Erica Moreira for their consistent and critical help with TPU resource allocation. We also give thanks to Shekoofeh Azizi, Harris Chan, Chris A. Lee, and Nick Ma for volunteering considerable amount of their time for testing out DrawBench. We thank Aditya Ramesh, Prafulla Dhariwal, and Alex Nichol for allowing samples. We are thankful to Matthew Johnson and Roy Frostig for starting the JAX project and to the whole JAX team for building such a fantastic system for high-performance machine learning research. Special thanks to Durk Kingma, Jascha Sohl-Dickstein, Lucas Theis and the Toronto Brain team for helpful discussions and spending time Imagening!
DALL-E 2 samples and providing us with GLIDE
References
[1] Osman Aka, Ken Burke, Alex Bauerle, Christina Greer, and Margaret Mitchell. Measur- ing Model Biases in the Absence of Ground Truth. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 107.000607 2016.
[3] Emily M. Bender, Timnit Gebru, Angelina McMillan Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? a. In Proceedings of FAccT 2021, 2021.
[4] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misog- yny, pornography, and malignant stereotypes. In arXiv:2110.01963, 2021.
10

---

[5] Shikha Bordia and Samuel R. Bowman. Identifying and Reducing Gender Bias in Wore-Level Language Models. In NAACL, 2017.
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1006, 2018.
[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NeurIPS, 2020.
[8] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency, FAT 2018, 23-24 February 2018, New York, NY, USA. Proceedings of Machine Learning Research, PMLR, 2018.
[9] Kaylee Burns, Lisa Hendricks, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In European Conference on Computer Vision (ECCV), 2018.
[10] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arxiv:2202.04053 2022.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Mísra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Morcira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Melese-el Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways. In arXiv:2001.08.0 200660 2022.
[12] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204,0m 085583, 2022.
[13] Valentin De Bortoli, James Thomton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to acore-based generative modeling. Advances in Neural Information Processing Systems, 34, 2021.
[14] Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep Generative Image Models using Laplacian Pyramid of Adversarial Networks. In NIPS, 2015.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, 2019.
[16] Prafulla Dhariwal and Alex Nichol. Diffusson models beat gans on image synthesis. In NeurlPS, 2022.
[17] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
[18) Dulhanty, Chris. Issues in Computer Vision Data Collection: Bias, Consent, and Label Taxon- omy. In UWSpace, 2020.

---

[19] Patrick Esser, Robin Rombach, and Bjor Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[20] Mary Anne Franks and Ari Ezra Waldman. Sex, lies and videotape: deep fakes and free speech delusions. Maryland Law Review, 78(4):892-898, 2019.
[21] Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang. Language-Driven Image Style Transfer. arXiv preprint
2021.
[22] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint wrXiv:2203.1313: 2022.
[23] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for Datasets. 10Xiv01803.00010 [cs], March 2020.
[24] Adam Harvey and Jules LaPlace. MegaPixels: Origins and endpoints of biometric datasets "In the Wild". https://megapixels.cc, 2019.
[25] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, arXiv preprint arXiv:2104.087/8 2021.
[26] Martin Hcusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv: 1706. 08500, 2017.
[27] Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.
[29] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 2022.
[30] Rowan T. Hughes, Liming Zhu, and Tomasz Bednarz. Generative adversarial networks-enabled human-artificial intelligence collaborative applications for creative and design industries: A systematic review of current approaches and trends. Frontiers in artificial intelligence, 4, 2021.
[31] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. MURAL: Multimodal, multitask representations across languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3449- 3463, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.1888311/2222 1t0.:///1/221...nnnngeemmmmmm findings-emmlp.299.. URL https: httttps://acaaathollggeo org/ 2021. findingssemmll e.nlp.293.
[32] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung. Zhen Li, and Tom Ducrig. Scaling up visual and vision-languse representation learning with noisy text supervision. In International Conference Machine Learning, pages 4904-4916. PMLR, 2021.
[33] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen. Zarana Parekh, Hicu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lision-language representation learning with noisy text supervision. In ICML, 2021.
[34] Zahra Kadkhodaie and Fero P Simoncelli.
Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13644 2020.
[35] Zahra Kadkhodaie and Eero P Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. Advances in Neural Information Processing Systems, 34, 2021.
12

---

[36] Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:2110002111
[37] Diederik P Kingma. Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint urXiv:2107.00630) 2021.
[38] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
[39] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating Images from Captions with Attention. In ICLR, 2016.
[40] Gary Marcus, Emest Davis, and Scott Aaronson. A very preliminary analysis of DALL-E 2. In arXiv:220.113807, 2022.
[41] Jacob Menick and Nal Kalchbrenner. Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling. In ICL.R, 2019.
[42] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.00677 2021.
[43] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Bob McGrew Pamela Mishkin, Πya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In arXiv:2112. 2021.
[44] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. In CVPR, 2022.
[45] Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2(1 10:0003,,, 2021.
[46] Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision? arXiv:2006. 16923,, 2020.
[47] Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and trans- parent dataset documentation for responsible ai. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2022.
[48] Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to Generate Reviews and Discovering Sentiment. In 170iv:1704.01444 2017.
[49] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training. In preprint, 2018
[50] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. In preprint, 2019.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.
[52] Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmannn, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George Driessche, Lisa Hendricks, Maribeth Rauh, Po-Sen Huang, and Geoffrey Irving. Scaling language models: Methods, analysis insights from training gopher. a /2. 1446, 2021.
[53] Colin Raffel, Minh- Minh-Thang Luong, Peter J. Liu, Ron Weiss, and Douglas Eck. Online and Linear-Time Attention by Enforcing Monotonic Alignments. In ICML, 2017.
13

---

[54] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer. JMLR, 21(140), 2020.
[55] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-lmage Generation. In ICMI., 2021.
[56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. In arXiv, 2022.
[57] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv: 1906.00446, 2019.
[58] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pages 1060-1069. PMLR, 2016.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022.
[60] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[61] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-Image Diffusion Models. In arXiv:2111.05822 2021.
[62] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636 2021.
[63] Morgan Klaus Scheuerman, Emily L. Denton, and A. Hanna. Do datasets have politics? disciplinary values in computer vision dataset development. Proceedings of the ACM on Human-Computer Interaction, 5:1 37, 2021.
[64] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXiv preprint arXiv:2 200::2111..20 1.02114, 2021.
[65] Lucas Sequeira, Bruno Moreschi, Amanda Jurno, and Vinicius Arruda dos Santos. Which faces can AI generate? Normativity, whiteness and lack of diversity in This Person Does Not Exist. In CVPR Workshop Beyond Fairness: Towards a Just, Equitable, and Accountable Computer Vision, 2021.
[66] Jascha Sohl-Dich Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256-223 2265.266..26 PMLR, 2015.
[67] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
[68] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. NewrIPS, 2019.
[69] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[70] Ramya Srinivasan and Kanji Uchino. Biases in generative art: A causal look from the lens of art history. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, page 41-51, 2021.
14

---

[71] Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human-liks like biases. In Proceedings of the 2021 ACM Conference on Fairness, Account- ability, and Transparency, FAccT '21, page 701-713, Association for Computing Machinery, 2021.
[72] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao Yuan Jing, Fei Wu, and Bingkun Bao. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprini arXiv:2008.05865, 2020.
[73] Belinda Tzen and Maxim Raginsky. Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit. In arXiv: 1908.09883, 2019.
[74] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[75] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674, 2011.......
[76] Ting- Chen Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 8798-8807, 2018.
[77] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up large vocabulary image annotation. In Twenty-Second International Joint Conference on Artificial Intelligence, 2011.
[78] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprint a:X:v:2112.02475 2021.
[79] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. In CVPR, 2018.
[80] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- aineggraned text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316-1324, 2018.
[81] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to-image synthesis using contrastive learning. arXiv preprint arXiv:2107.02222, 2021.
[82] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv: (iw::2100.460) 14.04027. 2021.
[83] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022.
[84] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-Modal Contrastive Learning for Text-to-Image Generation. In CVPR, 2021.
[85] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxi- ang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. arXiv preprint
/tti::2//1113790 2021.
[86] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802-5810, 10, 2019.
[87] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, 2015.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] We describe the license of the public data we use Section 6.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include DrawBench prompts in the supplemental material.
(d) Did you discuss whether and how consent using/curating? [Yes] We discuss the lack of consent from data subjects in web-scraped data in Section
obtained from people whose data you're
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 6.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [No]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]
16


Reviews:
[{"review_id": "y_Zz3qo1oYm", "reviewer": "Reviewer_uoqR", "summary": "Summary:\n\nThe paper proposes a text to image diffusion model (Imagen) that generates high fidelity images that accurately match the prompts. The work uses powerful language models (T5-XXL) that have been trained on text corpora that eventually aids in improved language understanding of the final model. Additionally, the paper makes modifications to the existing diffusion models by having dynamic thresholding along with some architectural changes to the U-Net to make it more efficient. With this work, Imagen becomes the state-of-the-art on MS-COCO based on the FID scores surpassing the models that are trained on it. The authors further introduce a benchmark, DrawBench, to evaluate the quality and accuracy of the image synthesis for a battery of challenging prompts.\n\n\nStrengths:\n- The paper makes an important contribution by using the models trained on surplus text corpora (unpaired data) rather than learning from purely paired image-text data. Their experiments also confirm that scaling the pretrained text encoders improves the image generations.\n- Imagen model is one of its kind to achieve unprecedented zero-shot results on MSCOCO on the FID metric – surpassing all the existing models that are directly trained on the dataset. \n- The paper invents a plethora of ‘tricks’ that facilitated image generations. It includes Efficient U-Net, having dynamic thresholding, text cross-attention layers in the super resolution model, and noise conditioning augmentation. It will be interesting to see if some of these tricks (or principles) can be used for improved generative modeling beyond the scope of the paper.\n- The paper introduces DrawBench that aids in evaluating the quality of the model generations across various dimensions. It provides us with a way to systematically compare complementary text to image generative models along these dimensions. \n- It was nice to see that the paper uses Human evaluations to compare Imagen with existing models using fidelity and alignment scores. In my experience, CLIPScore might not be a very reliable metric to judge the image-text alignment every time. \n\nSuggestions and Weaknesses:\n\n- [1] showed that scaling the data and the model are important factors in improving performance of zero shot models. The authors show that scaling the model size in terms of text encoder is indeed important for improved performance, however, it might be worthwhile to test how well scaling laws hold in the data dimension. Along the same lines, the Table 1 of the paper compares zero-shot models with different model size and training data. For a fairer comparison of how well the architectural choice is doing, it might be good to compare with an equivalent version of Imagen in terms of model parameters and/or training dataset size.\n- Interestingly, the paper shows that Imagen achieves lower preference rate while generating people when compared to the set with no people. It is not clear if the model is finding it difficult to draw people because it has seen a lot of images without people as compared to the images with people in the training dataset or is it ‘inherently’ difficult to create photorealistic people with this architecture? How does Imagen rate when compared to the existing text to image models on the quality of images with and without people? \n\n\nTypos/Edits:\n- Figure 4’s y-axis labels are inconsistent FID-10K vs FID@10K\n- Line 338 - wrote ‘Imagen’ two times\n\nMissing references:\n- Technically, DALLE-mini also leverages the power of a BART encoder pretrained on text-only corpora. The authors should include it in their related work and discuss it a bit.\n\n*References*\n\n[1] Combined Scaling for Open-Vocabulary Image Classification: https://arxiv.org/pdf/2111.10050.pdf\n\n[2] DALLE-mini: https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-Mini-Explained--Vmlldzo4NjIxODA#our-dall-e-model-architecture\n", "questions": "Mentioned in the main review", "limitations": "Mentioned in the main review", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Mentioned in the main review", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns"]}, "extra_text": {}}, {"review_id": "qr7gHw8LDvd", "reviewer": "Reviewer_9tx5", "summary": "This paper uses large transformer language models and diffusion models for text-conditioned image generation. The image generation results are impressive, achieving the state-of-the-art FID results on COCO. This paper uses a dynamic thresholding technique to sample images. This technique achieves better results based on the claims. This paper also builds a new benchmark, DrawBench, for text-to-image tasks.", "questions": "The results are impressive, but many text-conditioned image generation works, such as DALLE-2, Parti (just come out), also have good performance. Most of these models use massive data and large models. So it seems like the improvements come from the usage of massive data and large models, which are usually not released. \n\nThis paper is a combination of existing techniques, such as language models and diffusion models. \nCould the authors explain, except using the massive data and large models, what is the technical contribution of this paper and what is the novelty?\n\nThis paper would have much greater impact if it could distinguish itself from other recent/concurrent works. What makes it different from DALLE-2 and how do those differences influence the model's performance and use cases?\n\nWill the authors release their code or data for supporting research in this area?\n\nIs it ok to have such a large figure in the second page. It seems do not fit the NeurIPS template. ", "limitations": "This paper claimed their limitations and potential negative societal impact.\n\n(-) This paper has solid results to show the effectiveness of the proposed method. However, the good results are most likely from using large data and large models. This paper simply combines transformer language models and diffusion models, which is not novel. The dynamic thresholding technique is more likely a trick for training models.\n\n(-) This is a good paper with impressive results, but it seems the novelty is limited and not a good match for NeurIPS.\n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) This paper combines transformer language models and diffusion models for image generation. Even though the techniques are not novel, the image generation results are impressive. \n\n(+) This paper observes that using larger transformer language models improves the image generation results significantly.\n\n(+) This paper proposes a new benchmark for evaluating text-to-image tasks.\n\n(-) The novelty is limited. The method is a simple combination of transformer language models and diffusion models.\n\n(-) Using large models and massive data always seems helpful for better results and zero-shot generalization. This has become an obvious \"fact\" based on recent papers, such as DALLE [1], and CLIP[2]. However, purely having good results without solid methodology contribution is hard to be accepted.\n\n[1] Learning Transferable Visual Models From Natural Language Supervision \n[2] Hierarchical Text-Conditional Image Generation with CLIP Latents\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "dZzP9BPUpki", "reviewer": "Reviewer_EZNR", "summary": "This paper proposes Image which is a text-to-image diffusion model. The major finding of this work is that using large language models pretrained only on text data as text encoders are effective for text-to-image generation and can benefit from the scaling power of language models. Dynamic thresholding and Efficient U-Net architecture are proposed to improve the training effectiveness and efficiency of the diffusion model. SOTA experimental results are achieved on COCO FID and the proposed DrawBench benchmark.", "questions": "Please refer to the weaknesses for my questions.", "limitations": "The limitations are well-addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper is well written and presented.\n* The major finding of this work is that using large language models pretrained only on text instead of trained on paired image-text data only on text data is insightful.\n* The experiment results in generating photorealistic with text alignment are amazingly impressive. \n\nWeaknesses:\n* The technique contributions of this paper are limited. The proposed dynamic thresholding heuristic and U-Net architecture are somehow technically incremental. However, dynamic thresholding seems to be very effective for training the diffusion model. If some theoretical justification can be included, I will increase my rating.\n\n* It is not clear if we should pursue the text language encoding models or paired image-text models if we have enough image and text data. In another word, if we have enough paired image-text data in the future, can the paired image-text encoding models outperform the text language encoding models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept