Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Authors: Jiahui Yu
Yuanzhong Xu
Jing Yu Koh
Thang Luong
Gunjan Baid
Zirui Wang
Vijay Vasudevan
Alexander Ku
Yinfei Yang
Burcu Karagol Ayan
Ben Hutchinson
Wei Han
Zarana Parekh
Xin Li
Han Zhang
Jason Baldridge
Yonghui Wu
Venue: Accepted by TMLR ()

Abstract: We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with a new state-of-the-art zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis on Localized Narratives as well as PartiPrompts (P2), a new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across a wide variety of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements.

Published in Transactions on Machine Learning Research (11/2022)
Scaling Autoregressive Models for Content-Rich Text-to-lmage Generation
Jiahui Yu'
Yuanzhong Xul Jing Yu Koh
Thang Luong!
Gunjan Baid
Zirui Wang
Vijay Vasudevan' Alexander Ku'
Yinfei Yang
Burcu Karagol Ayan Ben Hutchinson
Wei Han
Zarana Parekh Xin Li Han Zhang Jason Baldridge?
Yonghui Wu*
(jiahuiy,, yuanzx, jykoh, thangluong, gunjanbaid, ziruiv, vrv, alexku, jasonbaldridge, yonghui))googlee.com
Equal contribution.
Core contribution.
Google Research
Reviewed OpenReview: http:://pppaaaeeee..eee/////r/td
Abstract
We present the Pathways (Dean 2021 Autoregressive Text-to-lmage (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a modeling problem, akin to machine translation, with sequences of image tokens the target outputs rather than text tokens in another language. This strategy naturally tap into the rich body of prior work large language models, which have continued advances in capabilitios and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer- based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with new state-of- thte-offtthhrrttt zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis Localized Narratives well as PartiPrompts (P2), new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across wide variely of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements.
1 Introduction
People are generally able to conjure rich and detailed scenes through descriptions expressed in written spoken language. Supporting the ability to generate images based on such descriptions can potentially unlock creative applications in many areas of life, including the arts, design, and multimedia content creation. Recent research on text-to-image generation, e.g., DALL-E (Ramesh et al. 2021] and CogView (Ding et al. 2021), has made significant progress generating high-fidelity images and demonstrating generalization capabilities to unseen combinations of objects and concepts. Both treat the task as form of language modeling, from textual descriptions into visual words, and use modern pruuence-to-sqqueeee architectures like Transformers (Vaswani et al. 2017 to learn the relationship between language inputs and visual outputs. A key component of these approaches is the conversion of cach image into sequence of discrete units through the use of an image tokenizer such as dVAF (Rolee 2017) or VQ-VAF (Van Den Oord et. al. 2017). Visual tokenization essentially unifies the view of text and images so that both can be treated simply as sequences of discrete tokens and thus amenable to sequence- sequence-to-sequeen to-sequence models. To that end, DALL-E and Cog View
CCorespondence (jiahuiyu, jasonbaldridge, yonghui)@google.com

---

Published in Transactions on Machine Learning Research (11/2022)
Figure 1: Example images generated by Parti. Top row: "Oil-on-cansuae painting of a blue night sky with roiling energy. fuzzy and bright yellow crescent moon shining at the top. Below the exploding yellow stars and radiating swirls of blue, a distant village sits quietly on the right. Connecting earth and sky is a flame-like cypress Iree with curling and swaying branches blue hills!" (a. 67-word description of the Starry Night by Vincent toigh-contrast photo of Sydney Opera House sitting next to Eiffel tower, under blue right sky of roiling energy, exploding yellow stars, and radiating swirls of blue". Last row: Similar to the middle row, but with "anime illustration" and different landmarks (the Grent Pyramid and the Parthenon).
the left. A church spire rises as a beacon over rolling
Gogh). Middle row: "A close-up
employed docoder-only language models, similar to GPT Radffrrd al. 2018). to learn from large collection of potentially noisy text-image pairs (Changpinyo et al. 2021 Jia et al. 2021a). Make-A-Scene (Gafni et al. 2022 further expands on this two-stage modeling approach to support both text and scene-guided image generation.
A different line of research with considerable momentum involves diffusion based text-to-image models, such as GLIDE (Nichol al. 2022) and concurrent works DALL-E (Ramesh et al.] 2022) (a.k.a., unCLIP) and
Imagen (Saharia et al.] 2022). These models eschew the use of discrete image tokens in favor of diffusion models (Ho et al. 2020 Dhariwal & Nichol 2021 to directly generate images. These models improve
zero-shot Fréchet Inception Distance (FID) scores MS-COCO (Lin et al. 2014 and produce images of markedly higher-quality and greater aosthetic appeal compared to previous work. Even so, autorcgressive models for texl-to-image generation remain appealing given extensive prior work on scaling large language models (Brown et al. 2020 Cohen et al., 2022 Du et al.] 2022 Chowdhery et al. 2022a) and advances discretizing other modalitics such as images and audio so that inputs those modalitics can be trcated language-like like tokens. This work presents the Pathways Autoregressive text-to-Image (Parti) model, which generates high-quality images from text descriptions, including photo-realistic ones, paintings, drawings,
2

---

Published in Transactions on Machine Learning Research (11/2022)
SACE
am ct
Toaday!
Parti
Parti
A. photo frg reading the named "Thnday" uritten There frog peinaed
A portran the Spyptien god Anubie anator googlns, white t-abare and leather jorket. The edty of Angeles baslground. Hirro DSLR phutograah.
C. high-controst phota of pands riding Aorse. The pende uccend hot and meding book. The
lorse Colorful flowers and word "PEACE" the DELR photograph. doyttime Righting.
the
Green
BE EXCELLENT TO EACH OTHER..
EXCELLENT EACH OTHER:
BE EXCELLENT LEGELLENTTO TO EACH OTHER
Be Exnsllent Do each Otber
D. A gion! cobra snale modo from X: X ["sstes"" "pancules", "susls", "com"")
sombat sits yollows bench chais, whiie sippiny murrlisi ckst is on his luploy leglosrd. The wombal is vourtny a whiie punuma Aat and florll Hesssiia shib Out-af-forra palm treze os the background 188.. phato- groph. Wide-angie mins.
F. The mayiny "ME EXCELLENT TO EACH OTHER* oo (a) brick wall and alien (b) driltoood. (o) old wooden boot with reflection. (d) stained glass. (Sox text for full prompts.)
G. Throe-querters front vinu of coming around rurse rocusteis looking noer green volley on clouty day, DSLR photogmph, E (Mtu,, red, gritou),, Y E (1971, 1997, 2017). Z (Porsche 921, Corvetts. Ford P-130]
mossen woartey formal clethes, wwordngg a tophitt halding case. macroon Andding a parbage bay: pointing in style of X ("Rtensrondd" "Voccen
photo of on Athesico mase with peinding of playing the styln Rygption hdenyllphics. ""pandax", "tancone" "panyolion"); V. ("tonsis" "soocee", "harket. ball")
Vis Gogh"" "Hokusar", "pteel are"; "petneliiom" ", "alatruct cndtur? "Eggetion tamb hetroglyphsee", "ernditional Chs- neze pacoting", "Madhohent are")
Figure Selected Parti images. See Section 6.1 for discussion.
3

---

Published in Transactions on Machine Learning Research (11/2022)
and more (see Fig. & 2 We show that with ViT-VQGAN (Yu et al. 2022a) image tokenizer, scaling autoregressive models is an effective way to improve text-to-image generation, enabling such models to accurately integrate and visually convey world knowledge.
Parti is a sequence-to-sequenee model based on the Transformer (Vaswani et al. 2017 an architecture critical to performance on many tasks, including machine translation (Vaswani et al. (017). speech recognition (Zhang Gulati et al.] 2020), ; conversational modeling (Adiwardana et al. 2020). image captioning (Yu 2022b), and many others. Parti takes text tokens as imputs to encoder and autoregressively predicts discrete image tokens with decoder (see Figure 3). The image tokens are produced by the Transformer-basee
et al.
2020;
et al.
ViT-VQGAN image tokenizer (Yu et al.] 2022a). which produces higher-fidelity reconstructed outputs and has better codebook utilization compared with dVAE (Rolle 2017), VQ-VAE (Van Den Oord et al. 2017], and VQGAN (Esser et al.) 2021). Parti is conceptually simple: all of its components - encoder, decoder and image tokenizer are based on standard Transformers (Vaswani et al., 2017). This simplicity makes it straightforward scale our models using standard techniques and existing infrastructure (Shocybi et al.) 2019: Du et al.] 2022 Chowdhery et al.) 2022a] Xu al. 2021). To explore the limits of this two-stage text-to-image framework, scale the parameter size of Parti models up to 20B, and observe consistent quality improvements in terms of both text-image alignment and image quality. The 20B Parti model achieves new state-of-the-ar zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO.
While most recent work has focused exclusively on the MS-COCO benchmark, we also show that strong zero-shot and finetuned results can be achieved on the Localized Narratives dataset (Pont-Tuset et al. 2020), which has descriptions that are four times longer than MS-COCO's on average. These results demonstrate the strong generalization capability of Parti to longer descriptions, allowing to pile considerable complexity in our explorations with the model (see examples in Figure 2 and the Appendix, and the discussion of growing a cherry tree in Section 6.2 Nevertheless, existing captioning/ / descriptioning datasets are limited to photographs and descriptions of their contents, but much of the appeal of text-to-imaee image models is that they can produce novel outputs for fantastical prompts. Given this, we introduce PartiPrompts (P2), a rich set of over 1600 (English) prompts curated to measure model capabilities across a variety of categories and controlled dimensions of difficulty] Each prompt in P2 is associated with both a broad category (out of 12 categories, ranging from abstract to animals, vehicles, and world knowledge) and challenge dimension (out of 11 aspects, ranging from basic, to quantity, words & symbols, linguistics, and complex). Our detailed analyses and human evaluations on MS-COCO, Localized Narratives and P2, along with extensive discussion of Parti's limitations (Section 6.3 give a comprehensive picture of the strengths and weaknesses of Parti models and establish autoregressive models as strong contenders for high-quality, broadly capable, open-domain text-to-image generation models.
Our main contributions include: (1) We demonstrate that autoregressive models can achieve state-of-the-od art performance: 7.23 zero-shot and 3.22 finetuned FID on MS-COCO, and 15.97 zero-shot and 8.39 finetuned FID on Localized Narratives; (2) Scale matters: our largest Parti model (20B) is most capable at high-fidelity photo-realistic image generation and supports content-rich synthesis, particularly those involving complex compositions and world knowledge; and (3) We also introduce a holistic benchmark, the PartiPrompls (P2), propose a novel concept of Growing a Cherry Tree, establish a new precedent regarding identifying limitations of sot-to--mmae generation models, and provide a detailed breakdown, with exemplars, for error types we observed.
2 Parti Model
Similar to DALL-E (Ramesh et al. 2021). : CogView (Ding et al. 2021), , and Make-A-SS AAkke-A-Scene (Gafni et al. 2022), Parti is a two-stage model, composed of an image tokenizer and an autoregressive model, as highlighted in Figure 8 The first stage involves training tokenizer that turns image into sequence of discrete visual tokens for training and reconstruets image at inference time. The second stage trains an autoregressive sequence-to-sequence model that generates image tokens from text tokens. We describe details for these two stages below, together with other techniques for building high-performing autoregrossive text-to-image models, such as text encoder pretraining, classifier-free guidance, and reranking.
Tmagen (Saharia al. 2022 introduces 200 prompts DrawBench for similar purposes. See Section 4.3 for discussion.
4

---

Published in Transactions on Machine Learning Research (11/2022)
<eOS>>
VIT-VQGAN
Inference
Transformer Decoder
Image Detokenizer [Transformer]
Transformer Encoder
Image Tokenizer (Transformer)
Train
f2
f
<805>>
iM
Two dogs running in a field
Figure 3: Overview of Parti sequence-lo-seqqeencc auloregressive model (left) for text-lo-image generation
with ViT- VOGA the image tokenizer Yu et al. 2022a) (right).
2.1 Image Tokenizer
Autoregressive text-to-image models must linearize 2D images into 1D) sequences of patch representations. In the limit, these are just pixels, with iGPT (Chen et al. 2020 but this requires modeling very long sequences even for small images (e.g., a 256x256 x 3 RGB image leads to 196,608 rasterized values). Worse, it is based on a very low-level representation the inputs rather than a richer one informed by the position of pixel in the context the image. Previous work (Van Den Oord al.) 2017 Ramesh et al.) 2021 Yu et al.] 2022a, Gafni et al., 2022) addressed this problem by using a discrete variational auto-encoder learn quantized representations of image patches a collection of raw images. Instead of learning representations that can take any value in the latent space, visual codebook is learned that maps patch embedding to its nearest codebook entry, which is learned and indexable location in the latent space. These entries can be thought of as visual word types, and the appearance of any of these words in patch in given image is thus an image token.
To be most useful for the second stage model, the image lokenizer needs to learn an effective visual codebook that supports balanced usage of its entries across broad range of images. It also must support reconstruction of sequence of visual tokens techniques inchuding fy-nornalization codes and factorized codes, which contribute to traiming stability, reconstruction quality and codebook usage. A ViT-VQGAN image tokenizer is trained with the same losses and hyper-parameters as (Yu et al. 2022a) on images of our training data (see Section 4.1p. We first train a ViT-VQGAN-Smin configuration (8 blocks, 8 heads, model dimension 512, and hidden dimension 2048 as shown Table of Yu et al. 2022a), with about 30M total parameters), and learn 8192 image token classes for the codebook. We note that the sccond stage autorcgressive encoder-docoder training only relies on the encoder and the codebook of learned image tokenizer. To further improve visual acuity of the reconstructed images after second-stage encoder-decoe finetune a larger-size tokenizer decoder (32 blocks, 16 heads, model dimension 1280, and hidden dimension 5120, with about 600M total parameters). We use 256x256 resolution for the image tokenizer's input and output.
high-quality output image. We use VIT-VQGAN (Yu et al. 2022a. with
training, freeze the tokenizer's encoder and codebook, and
We notice visual pixelation patterns in some of the output images of VIT-VQGAN when zooming in (sce Appendix Hj, and further find ill-conditioned weight matrices of the output projection layer before the sigmoid activation function. As fix, remove the final sigmoid activation layer and the logit-laplace loss, exposing
5

---

Published in Transactions on Machine Learning Research (11/2022)
ViT- VQGAN frozen
Super-Resolution Upsampler trainable
mares covmaanpll
Original Image 256 x 256
Reconstructed Image 256 256
Original Image 1024
Reconstructed Image
Figure 4: A Iearned super-resolution module upsample 256 256 images to higher-resolution 1024 1024 ones based frozen ViT-VQGAN image tokenizer. The super-resolution module lakes 256 x 256 images as imputs without conditioning on text imputs.
Model
Encoder Layers
Decoder Layers
Model Dims
MLP Dims
Heads
Total Params
Parti-350M
12
12
1024
4096
16
350M
Parti-750M
12
36
1024
4096
16
750M
Parti-3B
12
36
2048
8192
32
3B
Parti
16
64
4096
16384
64
20B
Table 1:
Size variants of Parti. Both encoder and decoder are based on Transformers Vaswani et al, 2017). The self-attention layer in decoder transformer causally masked. Parameters of ViT VQGAN image tokenization are not included the total parameter count and can be found in Section 2.1
the raw values RGB pixel values (in range [0, 1)). Conveniently, this fix can be hot-swappable into alrcady trained image tokenizer by finetuning the decoder.
Finally, while images of resolution 256 256 capture most of the contents, structures and textures, higher- resolution images have greater visual impact. To this end, we employ a simple super-resolution module on top of the image tokenizer, shown in Figure Stacked convolutional layers with residual connections are used the super-resolution network module following WDSR (Yu et al. 2018) (12 residual blocks with 128 channels). It is learned with the same losses of ViT- ViT-VQGAN (perceptual loss, StyleGAN loss and l2 loss with same loss weighting in (Yu et al. 2022al), mapping from reconstructed images to higher-resolution reconstructed images. The repprrresolttion module has about 15M parameters for the 512x512 version and about 30M parameters for the 1024 x 1024 version. We note that diffusson models could also be used here as iterative refinement super-resolution modules, as also demonstrated DALL-E 2 (Ramesh et al. 2022 and Imagen (Saharia et al. 2022), either with or without conditioning on text inputs.
2.2 Eicoder-Decodee for Text-to-lmage Generation
As shown in Figure 3 a standard encoder-decoder Transformer model is trained at the second stage, by treating text-to-image as a copuuecc-ttosequenee modeling problem. The model takes text as input and is trained using next-token prediction of rasterized image latent codes generated from the first stage image tokenizer. For text encoding, we build sentence-piece model (Sennrich al. 2016 Kudo & Richardson, 2018) of vocabulary size 16,000 on sampled text corpus from the training data (Section 4.1). Image tokens are produced by learned VIT- VQGAN image tokenizer (see Section 2.1) At inference time, the model samples image tokens autoregressively, which are later decoded into pixels using the ViT-VQGAN decoder.
We use a maximum length of text lokens 128, and the length of image tokens are fixed to 1024 (i.e., 32x32 latent codes from a 256 x 256 input image). As an example, the 67-word description of the Starry Night prompt given Figure [ has a total length of text tokens. All decoder transformers use conv-shaped masked sparse attention (Child al. 2019) with implementation following DALL-E RRamesh al. 2021 (detailed information be found in Appendix B.1 Figure 11 in (Ramesh et al. 2021 We train four size

---

Published in Transactions on Machine Learning Research (11/2022)
variants ranging from 350 million to 20 billion parameters, as detailed in Table I Specifically, configure the Transformers following previous practice of those in scaling language models with default expansion ratio of 4x in MLP dimensions. We double the number of heads when the model dimension is doubled. In the current scaling variants, our configuration prefers a larger decoder for modcling image tokens and as a result the decoder has more layers (e.g., 3x in the 3B model and 4x in the 20B model).
Most of the existing two-stage text-to-image generation models, including DALL-E (Ramesh et al., 2021 CogView (Ding et al.] 2021) and Make-A-Soe A-Sene (Gafni et al. 2022), are decoder-only models. We found that at the model scale of 350-million to 750-miliin parameters, the encoder-decoder variants of Parti outperformed decoder-only ones, both in terms of training loss and text-to-image generation quality in our early exploration. We thus chose to focus on scaling the encoder-decoder models.
2.3 Text Encoder Pretraining
The encoder-decoder architecture also decouples lext encoding from image-token generation, so it is straight- forward to explore warm-starting the model with a pretrained text encoder. Intuitively, a text encoder with representations based on generic language training should be more capable at handling visually-grounded prompts. We pretrain the text encoder on two datasets: the Colossal Clean Crawled Corpus (C4) (Raffel et al. 2020) with BERT (Devlin et al. 2019) pretraining objective, and our image-text data (see Section 4.1h with a contrastive learning objective (image encoder from the contrastive pretraining is not used). After pretraining, continue training both encoder and decoder for text-to-image generation with softmax cross-entropy loss a vocabulary of 8192 discrete image tokens.
The text encoder after pretraining performs comparably to BERT (Devlin et al., 2019) on GLUE (see Appendix Table D); however, the text encoder degrades after the full encoder-decoder training process on text-lo-image generation. We leave this observation as a future research topic on the difference and unification of generic language representation and visually-grounder language representation. Still, the text-encoder pretraining marginally helps text-lo-image generation loss with 3B-parameter Parti models, so pretraining is used by default in our 20B model. We provide detailed training loss, GLUE evaluation of text encoders, and some qualitative comparison in Appendix G
2.4 Classifier-Free Guidance and Reranking
Classifier-free guidance (Ho & Salimans 2021 (CF-guidance in short) is critical in the context of improving the sample quality of diffusion models NNchhll et al.] 2022 Ramesh et al. 2022 Saharia et al.) 2022) without pretrained classifiers. In this setup, a generative model G is trained to be able to perform unconditional generation G(z) (where z represents random noise) and conditional generation G(z, c) (where e represents some condition, such as language descriptions). It is implemented as randomly dropping out the conditional vector (masking out or switching to a learned embedding) with some probability. During the inference process, sampling of an output I is done by using a linear combination of the unconditional and conditional predictions:
(1) where À is a hyperparameter representing the weight of classifier free guidance. Intuitively, it decreases the unconditional likelihood of the sample while increasing the conditional likelihood, which can be viewed as encouraging aligument between the generated sample and the text condition.
I= G(z) + X(G(z.c) -(())),
Classificr-free guidance has boen similarly applied in the context of autoregressive models for text-to-image generation [Crowson] 2021 Gafni et al. 2022) to great effect.. Make-A-Scene (Gafni al., 2022) finetunes the model by randomly replacing the text prompts with padded tokens. During inference, tokens are sampled from linear combination of logits sampled from an unconditional model and conditional model on a text prompt. We also apply CF-guidance in Parti, and find it has a significant improvement on the output image-text alignment, especially on challenging lext prompts. For the unconditional inputs, we simply set. text token ids to zeros and text token paddings as ones.
With batch-sampled images per text prompt, contrastive reranking is used in DALL-E (Ramesh et al., 2021 which produces image-text alignment scores after the generation. We apply contrastive reranking in
7

---

Published in Transactions on Machine Learning Research (11/2022)
all-gather
inputs
device0
device1 device2
b
b
b
device3
matmul
matmul
rell
reduce-
scatter
f
outputs
d
Figure 5: 4-way in-layer model parallelism with fully partitioned activations used to scale the 3B model training. The figure shows a simplified Transformer feed-forward layer (with the sequence dimension omitted); each color represents data on one device. We also use 128-way data parallclism.
our work and find it is complementary to classífier-free guidance. Compared with the 512 images used DALL-E (Ramesh al.) (021)), we sample just 16 images per text prompt for the experiments reported in this paper. We rerank cach output set based the alignment score of image and text embodding of a Contrastive Captioners model (CoCa) (Yu et. al. 2022b). A CoCa base-size model (Table 1 in (Yu el al. 2022bb) is trained on the same dataset with details in Section 4.1 We note that reranking over small set of batch-sampled images is computationally cheap in the text-to-image sampling process, and produces helpful image-text alignment scores among diverse image outputs.
3 Scaling
We implement our models in Lingvo (Shen et al. CloudTPUv4 hardware for both training and inference. GSPMD is an XLA compiler- based model partitioning system that allows us to treat a cluster of TPUs as a single virtual device and use sharding annotations on a few tensors to instruct the compiler automatically distribute data and compute on thousands of devices.
2019) and scale with GSPMD (uu et al. 2021 on
Training. We train both 350M and 750M models simply with data parallelism. For the 3B model, use 4-way in-layer model parallelism (see Figure 5) and 128-way data parallelism. Partitioning single dimension in each tensor is sufficient to scale a 3B model. The model weights are partitioned on the feed-forward hidden dimension and the number of attention heads dimension; the internal activation tensors of the leed-forward and attention layers are also partitioned on the hidden and heads dimensions. One difference from Megatron-LMM (Shoeybi al. 2019) is we fully partition the output activations of feed-forward and attention layers on a different dimension, with the details illustrated as the fonalized 2d sharding in the GSPMD work (Xu et al. 2021). This strategy will result in ReduceScatter and AllGather communication patterns instead of AllReduce, which significantly reduce peak activation memory.
The 20B model has 16 encoder layers, and 64 decoder layers (see Table D The size of the weights per layer is moderate (as opposed to being very wide), which makes pipeline parallelism (Huang et al. 2018) a good option for scaling. We use a generic pipelining wrapper layer allowing us to specify single-stage program, which will later be automatically transformed into multi-stage pipcliming program; the wrapper layer uses vectorization and shifting buffers to reduce pipeliming into a tensor partitioning problem (see Section 3.3 of (Xu et al. 2021)). Thus, all lover-level infrastructure two additional benefits in adopting GSPMD pipelining: 1) it allows us to conveniently configure pipclines within model sub-components, simplifying the overall complexity for encoder-decoder models, and 2) since pipelining is implemented as tensor partitioning on vectorized programs, we can reuse the same set of devices for other types of parallelism outside the transformer layers.
be reused for pipelining. There are
We configure the model to have separate encoder and decoder pipelines, each with 16 stages. We also use 64-way data parallelism in addition to pipelining. However this makes per-core batch size small, exposing
8

---

Published in Transactions on Machine Learning Research (11/2022)
split batch
devica0
device1
quantizer
device2
embedding
softmax
device15
split batch
replicate batch
layerO
layer1
layer0
layer16
layer17
layer32
layer33
layer48
layer49
layer1
layer2
++
layer15
layer2
layer31
layer 18
layer47
layer34
layer15
layer63
layer50
encoder pipeline
decoder pipeline
Figure An illustration of 16-stage GSPMD pipelines used scale the 20B model training. The figure shows how the 16 devices are used for data parallelism in the quantizer, embedding and softmax layers, but repurposed for pipelining in the encoder and decoder layers. Each color represents data layer assigned to one device. The decoder uses 4-round circular schedule to further reduce the pipeline bubble ratio. On top of this, we use additional 64-way data parallelism for all layers.
an additional challenge of excessive pipeline stalls due inter-stage data dependency (known as bubbles in pipeline parallelism (Huang et al., (2018)) To reduce the ratio of bubbles, adapt the circular schedule described in (Xu et al., 20211) in the decoder pipeline (a similar technique was also proposed in (Narayanan) le al. 2021)), )) where the 4 layers in each stage are executed in round-robin order. Outside the encoder and decoder, we use the same set of devices to do data parallelism instead of pipeliming for the embedding, softmax, and image tokenizer layers. Figure illustrates the overall distributed training strategy.
During training, Adalactor (Shazeer Stern 2018 optimizer is used lo save memory with 31 = 0.9, 32 0.96 and decoupled weight decay value of 4.5
10-2. The first moments of optimizer slot variables are also quantized from float32 to int8. We use default dropout ratio 0.1 for all models in both encoder and decoder. A deterministic version of dropout layer well vectorized version of Adafactor optimizer are used in the 20B model to enable training pipelined models. Data types are cast to bfloat16 for attention projection and feed-forward transformers layers, while all layer norms and model output are kept as float32. We use a default learning rate 4.56-5 and exponential learning rate schedule with 5,000 warm-up steps. Exponential decaying starts at training steps 85,000 with a total of 450,000 steps and final ratio of 0.025. We use a global batch size of 8192 during training. We do not use exponential moving average of the model weights to save device memory. Conv-shaped sparse allention used in the decoder transformers, similar to DALL-E (Ramesh) et al. 2021) (Appendix B.1. Architecture, Fig. 11). We additionally clip gradient norm to value of 4.0 to stabilize the training, especially at the beginning. At the output of both the encoder and decoder, we apply an additional layer normalization.
Inference. Our primary goal for inference optimization to speed up small-batcl image generation. We choose in-layer model parallelism for both the 3B and 20B models. As opposed training, we do not fully partition the output activations for foed-forward and attention layers for inference; this because 1) each step of the autoregressive decoding produces much smaller tensors and (at the time of writing) AllReduce

---

Published in Transactions on Machine Learning Research (11/2022)
Dataset
Train Val ArgWords
Caption
Image
MS-COCO (2014)
Lin al. 2014
82K
10K
10.5
bow of broccoli and applea with utensil."
Localized Narratives
(COCO subset)
Pout-Tuset et al. 2020
134K 8K
42.1
"In this picture, we ace bowd con- taining chopped apples broe- calt In the backgrownd, white table on which seeds or grains, broczoli, piece of fruit, water glass and plates are placed. This table covered with white and blue color cloth. This picture is blurred in the bockground."
Table 2: Evaluation data statistics and examples. Images from the COCO portion Localized Narratives come from the MS-COCO (2017) set; Localized Narratives descriptions are four times the length of captions in MS-COCO on average. The example above highlights the massive difference in detail between MS-COCO and Localized Narratives for the same image.
performs better on small data, 2) activation memory is not a concern during inference, which does not have backward pass.
4
Training and Evaluation Datasets
4.1 Training Datasets
We train combination of image-text datasets for all Parti models. The data includes the publicly available LATON-400M dataset (Schuhmann et al.] 2021 FIT400M, filtered subset of the full 1.8 billion examples used to train the ALIGN model (Jia et al. 2021a); ; JFT-4B dataset (Zhai et al. 2022 which has images with lext annotation labels. For lextual descriptions of JFT, we randomly switch between the original labels as text (concatenated if am image has multiple labels) or machine geeeraeeee captions from a LimVLM-Huge model (Wang al., 2022). We discuss the limitations of the data in Section 8 For all image inputs, we follow the DALL-E dVAE input processing (Section A.2. Training in (Ramesh al. 2021)) for image tokenizer training and the DALL-E Transformer input processing (Section B.2. Training in (Ramesh et al. 2021 for encoder-decoder training.
4.2 Evaluation Datasets
We evaluate our models on MS-COCO (2014) (Lin el al., 2014) and Localized Narratives (Pont-Tused el al. 2020), summarized in Table 2 finetuned text-to-image generation performance, which makes it consistent point of comparison with prior work. However, MS-COCO captions short, high-level characterizations of their corresponding images. For more comprehensive evaluation, also use the COCO portion of Localized Narratives (LN-COCO), which provides longer, detailed descriptions of images corresponding to the MS-COCO (2017) dataset, and compare Parti's performance on LN-COCO against (Koh et al.) 2021 Zhang et al. 2021 h. These long-form descriptions are typically quite different from the descript ions used to train large text-to-image generation models. This provides a measure of generalization out-of-domain distributions, as well as the finetuning capability of these models. Regardless of the community's current focus on zero-shot performance, the ability to finetune effectively is also important for adapting open-domain text-to-image generation model to work with a specific application area or domain.
MS-COCO is the current standard dataset for measuring both zero-shot and
4.3 PartiPrompts
Existing benchmarks like MS-COCO (Lin et al., 2014) and Localized Narratives (Pont- et al. 2020 are clearly useful for measuring the progress of text-to-image synthesis systems, but the descriptions available in them generally limited to everyday scenes and objects found in natural images. This limits their
10

---

Published in Transactions on Machine Learning Research (11/2022)
Category
Examples
ABSTRACT
convmonseensc;; hoppiness; hope; insight; 300; 101; golden rabio; Pibonaces wamber a Tyrennosaurus Rea; 7 dogs sitting around poker table, two of which are turning away: white rabbit in blue jogging clothes doubled over in poin while tartle wearing a red tank top dashes confidently through the finish line.
ANIMALS
ARTS
a dutch barogue painting of horse in a field of flowers; painting of a foz the style of starry night; A raccoon wearing formal clothes, wearing top hat and holding a cane. The raccoon holding garbage bag. Oil painting in the style of Vincent Van Gogh. concurrent lines; o diagram of broin fanction; concentric squares fading from yellow on the outside to deep orange on the inside; drawing of a scries of musical notes wrapped around the Earth
ILLUSTRA-
TIONS
OUTDOOR SCENES
a grand piano next to the not of a tennis court; a white country home with a wrop-eround porch; beardiful fireworks in the sky with red, white and blue; a penceful lakeside landscape with mégrating herd of sawropods
PEOPLE
the Beatles crossing Abbey road; a woman with long hair next to a lumönescend bird; a tall man stooping down to enter a low red sports car; a politicion wearing a soccer jersey and holding a wolleyball while giving speech on a stage
VERICLES
an F1; a boat, in the canals of venice; a car with tires that have yellow rims; a blue semni-truck and its trailer jumping over a vow of motorcycles. there are metal romps on either side of the motorcycles.
WORLD KNOWL- EDGE
U.S. 101; the skyline of New York City; An aerial view of Ho Long Bay without ang boets; view of the Great Wall from its base; A close-up hiigh-cnnrrst photo of Sydney Opera House sitting next to Diffel tower, under blue night sky of roiling energy, exploding gellow store, and radiating swinls of blue.
Table 3: Sample categories in the PartiPrompts (P2) benchmark. Examples (separated by semicolons) range from abstract concepts such as "golden ratio" to concrete ones such as "the skyline of New York City". For full descriptions of all categories, see Appendix D
Challenge
Examples
BASIC COMPLEX
rabbit; U.S. 101; o margarita; My pads; brain coral; The Alamo; family
the Sydney Opera House with the Eiffel tower sitting the right, and Mount Everest rising abone; whitee rubbit in blue jogging clothes doubled over in pain while turtle wearing red tank top doshes confidertly through the finish line; Oil-on-canvas painting of blue night sky with roiling energy. A fuzey and bright yellow crescent moon shining at the top. Below the exploding yellow stars and rodiating swirls of blue, a distont village sits quictly on the right. Connecting earth and sky is a flame-like cypress tree with curling and swaying branches on the left. A church spire riscs as a beacon ovcr rolling blue hills.
IMAGINATION
a four-eyed horse; a toaster shaking hands with a microwave; a peaceful lakeside landscupe with migrating herd of sauropods; a flower with large red petals growingg on the moon's surfece; A rusly spaceship blasts off in the foreground. A cily with tall skyscrapers is in the distance, with mountain and ocean in the background. A dork moon is in the sky. realistic high-contrrn anime illustration. Inomppebensihittkee;
LINGUISTIC STRUCTURES
The horse raced past the barn fell; One morning I chased an elephant in my pojamas; The dog chased the cat, which ran up free. It waited at the top; The dog chased the cut, which ran up tree. It waited at the bottom. the back buildings seen through window with rain on it; view from below of tall white ladder with just one rang leaning up against yellow brick wall
PERSPECTIVE
violin; an extreme close-up view of capybara sitting in field; tall
QUANTITY
an owl family; 7 dogs sitting around a poker table, two of which are turning away; a basketball game between a team of four cats and a team of three dogs a grumpy porcupine handing a check for $10,000 to a smillnn peacock; group of cats in a meeting. there is a whitebourd with "stack more loyers" ' writion on a; The saying 'BE EXCELLENT TO EACH OTHER" written on a red brick wall with a graffits image of a green alien wearing a tuzedo. A yellow fire hydrant is on a sidewalk in the foreground.
WRITING SYMBOLS
&
Table 4: Sample challenge aspects in the P2 benchmark. Examples (separated by semicolons) range from basic to complex ones such as a full description of the Starry Night "Oil-on-canvas painting of a blue night sky ... rolling blue hills". For full descriptions of all challenge aspects, see Appendix D
11

---

Published in Transactions on Machine Learning Research (11/2022)
representation of a broad spectrum of prompts- in particular, they lack prompts that allow to better probe model capabilities open-domain text-to-image generation. For example, MS-COCO captions are brief characterizations of high level participants and actions in images; these typically cover common scenarios and are oriented toward objects. Localized Narratives has highly-detailed descriptions, but also emphasizes natural scenes and objects. Recently, the work by (Park et al. 2021) focuses on the text-lo-image generation task, but is limited to only two scenarios, unseen object-color (c.g., "blue petal") and object-shape (e.g., "long beak"). Motivated by these shortcomings, we present PartiPrompts (P2), a set of 1600 diverse English prompts that allow to more comprehensively evaluate and test the limits of text-to-image synthesis models.
Each prompt in the P2 benchmark is associated with two labels: (1) Category, indicating broad group that a prompt belongs to, and (2) Challenge, highlighting an aspect which makes a prompt difficult. Table 3 provides a few samples of categories (out of 12 options) used in P2, ranging from abstract concepts such as the "golden ratio" to concrete world-knowledge ones such as "the skyline of New York City". Similarly, Table [ lists sample of challenge aspects (out of 11), ranging from basic ones such as "a rabbit" to complex ones such as full description of the painting Slarry Night (*Oil-on-canvas painting of blue night sky ... A church rises a beacon against rolling blue hills."). For example, the prompt "a peaceful lakeside landscape with migrating herd of sauropods" is categorized as OUTDOOR SCENES, while its challenge aspect is IMAGINATION. Similarly, the prompt "7 dogs sitting around a poker table, two of which are turning away" has ANIMALS as category and QUANTITY as challenge aspect. These two views of a prompt allows us to analyze a model's capabilities from both aspects the overall content generated and the subtle details captured.
Category
Challenge Aspect
Artifacts
Basic
17.6%
Animals
16.7
Fine-grained Detail
dorld Knowledge
Simple Detail
19.2%
19.1%
14. 14.2%
10.8%
3
3.1%
Produce & Plants
Indoor Scenes
12.6%
3.7%
2.1%
Linguistic Structures
Properties & Positioning
People
8.0%
7.6%
1,5%
4.0%
Abstract
Style & Format
1.2%
6. 9.9%
6%
5. 5.5%
Perspective
Arts
Quantity
Outdoor Scenes
Food Beverage
Illustrations
Vehicles
Imagination
Complex
Writing & Symbols
Figure 7: A summary of the PartiPrompts (P2) set of 1600 descriptions, spanning across many category labels (left) and challenge aspects (right).
We created PartiPrompts both by thinking of novel prompts and by mamually curating and sampling prompts from recent papers (Ramesh et al. 2021 Ding et al. 2021 Gu al. 2022 Nichol al. 2022, Ramesh et al., 2022) (which accounts for about 7% of prompts in P2). While it is possible to assign multiple categories and challenge aspects to a prompt, on a single primary category and challenge aspect for each prompt. For example, when there is a proper noun, we will prefer to label the prompt as WORLD KNOWLEDGE (e.g., "a painting of street in Paris") over other categories, e.g., ARTS. We also prioritize categories that have fewer examples, e.g., ARTS, over those with plenty, such as ANIMALS, as in the case of the prompt "A raccoon wearing formal clothes, wearing a tophat and holding a cane. The raccoon is holding garbage bag. Oil painting in the style of Vincent Van Gogh." The PEOPLE category always takes priority, 6.g., "a team playing baseball at the beach" is labeled PEOPLE instead of OUTDoOR SCENES; do this to ease future work on fairness and bias interested in using PartiPrompts, as it can be casily split to include or exclude prompts involving people. Figure highlights the distribution of category labels and challenge aspects over the 1600 prompls. One can group these prompts into levels of difficulties by the challenge aspects: Standard includes BASIC and SIMPLE DETAIL (about 1/3 of the prompts); Intermediate includes FINE-GRAINED DETAIL and STYLE & FORMAT (also about 1/3 of the prompts); and Challenging includes the remaining 7 challenge aspects such IMAGINATION, QUANTITY, COMPLEX, and LInGUISTIC STRUCTURES.
chose to reduce the complexity of model analysis by manually deciding
12

---

Published in Transactions on Machine Learning Research (11/2022)
It is also worth mentioning DrawBench, a concurrently developed benchmark of 200 prompts introduced in (Saharia et al. 2022). Tt has eleven labels that mix both categories (e.g., "DALI~E"") and challenging aspects (e.g., "Counting") PartiPrompts, in contrast, teases apart these two dimensions, with 12 categories and 11 challenging aspects, allowing for richer categorizations of prompts and more fined-grained analyses, together with 8x more prompts. Both benchmarks contain prompts that present strong challenges for current. best, models-inchuding DALL-E 2, Imagen and Parti-and hopefully will inspire further benchmarks to increase the level of difficulty future models continually improve.
5 Experiments
We conduct automatic evaluations on both MS-COCO and Localized Narratives to compare with previous work. On MS-COCO and PartiPrompts, we also obtain human side-by-side evaluations for Parti 20B to compare with a strong retrieval baseline, as well as the XMC-GAN model (Zhang et al. 2021). which has the best. FID out of all publicly available models the time of writing. We also conduct human evaluation on two Parti models with parameters 3B and 20B on PartiPrompts, and provide a detailed breakdown on categories. By default, Parti samples 16 images per text prompt and uses CoCa model to rank the outputs (see Section 2.40.
5.1 Retrieval Baseline
Perhaps the most compelling use for text-to-image generation models is creating novel images for situations that have never been depicted. Strong models should thus be more effective than an approach that simply retrieves candidate images from a large dataset. We implement a retrieval baseline as follows. For every training data example, we compute image embeddings from an Efficient(Net-22 ALIGN-based model (Jia et al. 2021b). Then, given text prompt, identify the nearest training image measured by the alignment between the text prompt embedding from the ELIGN-based model and the image embeddings. This be done at the scale of our data by using efficient similarity search libraries such as ScaNN (Guo et al., 2020). These retrieved examples (from the training set) are then provided as the output of the baseline, to be evaluated by images actually generated by our models. We manually visualize the retrieved images given the text prompts and observe that this retrieval approach represents a high-quality baseline, especially for common text descriptions.
To compare with Parti generated images, report retrieval baseline results under two settings that we characterize zero-shot and finetuned to align with the model evaluation terminology. For MS-COCO, retrieval over our training data is "zcro-shot", while retrieval over MS-COCO's train split is "finctuned" corresponding to out-of-dalased and in-dataset retrieval, respectively. We compare Parti generated images with retrieved images using both automated measures and human evaluation for both image realism and image-text alignment.
5.2 Evaluation Metrics
We evaluate using two primary axes: (1) generated image quality, and (2) alignment of the generated image with the input text. We report both automated quantitative metrics and human evaluation results. In addition, we show example model outputs for qualitative assessment and comparison.
Automatic image quality. Similar to prior work in text-to-image generation, we use the Fréchet Inception Distance (FID) (Heusel et al., 2017) as the primary automated metric for measuring image quality FID is computed by running generated and real images through the Inception v3 (Szegedy et al.) 2016) model, and extracting features from the last pooling layer of the model. The Inception features of the generated and real images are used to fit two separate multi-variate Gaussians. Finally, the FID score is computed by measuring the Fréchet distance between the two multivariate Gaussian distributions. Following (Xu et al. 2018 Zhang et al.) 2021 Ramesh et al.) 2021). we use 30,000 generated and real image samples for evaluation on MS-COCO (2014) using the same DALL-E input preprocessing (Section B.2. Training in (Ramesh et al.
We httppp/s//tthhh....m///sassaaass
for computing FID scores.
13

---

Published in Transactions on Machine Learning Research (11/2022)
MS-COCO FID (4)
LN-COCO FID (J)
Approach
Model Type
Zero-shot Finctuned
Zero-shot Finetuned
Random Train Images Retrieval Baseline
Gafni et al.
2022
2.47
17.97
6.82
33.59
16.48
TReCS Koh et al. 2021
GAN
48.70
XMC-GAN [Zhang et al.
2021
GAN
9.33
14.12
DALL-E Ramesh ct al.
2021
Autoregressive
~88
Vog [Ding
2021
Autoregressive
27.1
Cog View2 [Ding et 2022
Autoregressive
24.0
17.7
GLIIE (Nichol
2022
Diffusion
12.24
Make-A-Scene (Gaini et al.
2022
Autoregressive
11.84
7.55
DALL-E 2 (Ramesh et 2022
Diffusion
10.39
Imagen (Saharia ot al. 2022
Diffusion
7.27
Parti
Autoregrossive
7.23
3.22
15.97
8.39
Table 5: Comparison with previous work on the MS-COCO (2014) (Lin al. 2014 and Localized Narratives (COCO split) (Pont-Tusct et al. 2020) validation sets. When available, we report results for both zero-shot and finetuned models. Retrieval models either perform retrieval over our training set ("zero-shot"). "), or the respective MS-COCO and LN-COCO training sets (ffinetuned"). Parti samples 16 images per text prompt and uses a CoCa model to rank the outputs (Section 2.4). Similar to DALL-E 2 (Ramesh et al. 2022). we use guidance scale 1.2 for all above results. We report zero-shot FID score of other model sizes in Figure
20211) ) with 256x256 image resolution. The validation set of the Localized Narratives COCO split contains only 5,000 unique images, so we follow (Zhang et al. 2021 in oversampling the captions to acquire 30,000 generated images.
Automatic image-text alignment. Following DALL-Eval (Cho el al. 2022 J, we also measure lext-image fit through automated captioning evaluation (or captioner evaluation): an image output by the model is captioned with a pretrained VL-15 model (Cho et al. 2021) and then the similarity of the input prompt and the generated caption is assessed via. BLEU (Papineni et al.
2002 CIDEr (Vedantam et al. 2015 2016
METEOR (Denkowski & Lavie 2014 and SPICE (Anderson et al.
Human side- biiiide. We follow previous work (Zhang et al. 2021 Ramesh et al. 2021 in doing side-by-side evaluations in which human annotators are presented with two outputs for the same prompt and are asked to choose which output is a higher quality image (generally, better with respect to image realism) and which is a better match to the input prompt (image-text alignment). The models are anonymized and the pairs are randomly ordered (left us. right) for each presentation to an annotalor, and each pair is judged by five independent annotators. We graphically show the gradual breakdown of results for each model in terms of the number of examples where it obtains 0, 1, 2, 3, 4 or 5 votes. In addition, we highlight the percentage of examples where each model has obtained the majority (three or more votes), as a summary of the comparison. F for a screenshot of our annotator interface.
See Appendix
5.3 Main Results
Table 5 presents our main results of automated image quality evaluation. Parti achieves a comparable zero-shot FID score 7.23 compared with diffusion-basse model Imagen (Saharia et al. 2022). When finetuned, Parti achieves state-of-the-art FID score of 3.22, a dramatic improvement over previous best finetuned FID 7.55 from an autoregressive model, Make-a-Scene (Gafni et al., 2022). It is also better than the in-dataset retrieval baseline, with an FID score 6.82. We note that the retrieval baseline is worse than using 30,000 random samples from MS- COSC real training set images which obtains an FID of 2.47. The root cause is that the retrieval model often selects the same images for similar types of prompts, leading to duplicates in the retrieved images for evaluation. For example, there are only 17,782 unique retrieved MS-COCO train images for 30,000 validation text prompts, leading to worse diversity and poorer FID score as compared to the 30,000 random samples from the train sct. We also show qualitative comparisons (Appendix [ Figure 24 of nan-cherry--ichee Parti sampled images along with outputs of other approaches (Ramesh et al. 2021
Nichol et al. 2022 Gafni et al. 2022 Ramesh et al. 2022 on MS-COCO prompts. Parti demonstrates
14

---

Published in Transactions on Machine Learning Research (11/2022)
Approach
BLEU (1)
METEOR (1)
CIDEr (1)
SPICE (t)
Random Train Images
4.4
9.2
4.8
2.0
Retrieval Baseline
24.7
23.9
84.1
16.6
Ground Truth (upper bound)
32.5
27.5
108.3
20.4
DALL-E
9.3
12.9
20.2
5.6
ruDALL-E-XL
13.9
16.0
38.7
8.7
minDALL-E
(Kim et al.
2021
16.6
17.6
48.0
10.5
X-LXMERT
Choo
2020
18.5
19.1
55.8
12.1
Parti
26.4
23.9
83.9
16.5
Table 6: Comparison with prior work on captioner evaluation the MS-COCO 5K test set (Karpathy &
Fei-Fei 2017) with baselines from DALL-Eval (Cho et. al. 2022). Ground Truth represents the theoretical upper bound on this evaluation with captions generated using MS-COCO images as inputs to the VL-15
model (Cho et al. 2021). Parti samples 16 images per text prompt and uses a CoCa (Yu et al. 2022b model to rank the oulputs (Section 2.4
strong generalization without finetuning on specific domains like MS-COCO, and achieves a high degree of image realism often very close to that of real images.
For LN-COCO, Parti achieves finetuned FID score of 8.29, which is a massive improvement over XMC-GAN's finetuned result of 14.12 and the retrieval baseline's 16.48. Moreover, Parti achieves zero-shot FID score of 15.97, which nearly matches XMC-GAN's finetuned score (trained on LN-COCO set). We visualize and compare side-by-side with XMC-GAN and find the zero-shot images produced by Parti are qualitatively much better in realism and image-lext fit compared to images produced by XMC-GAN, which we olfer as a cautionary tale that researchers should not rely solely on FID for comparison of text-to to-image generation models.
5.4 More Results on MS-COCO
Automatic image-text alignment evaluation. Table 0 provides results of Parti on the captioner evaluation (Cho et al. 2022) as an automatic image-text alignment measure. Parti outperforms other models on this measure, and it closes much of the gap to the scores obtained for captions generated from the ground truth images. The retricval baseline performs comparable to Parti. Unlike FID scores, random train images perform considerably worse on the captioner evaluation, as expected. The captioner evaluation complements FID score evaluation as an automatic imago-text alignment measurement for text-to-image generation models; however, also note these results are limited by the captioner model's (Cho et al. 2021 ability to discriminate between outputs from different approaches.
Human evaluations. For MS-COCO, we compare our zero-shot generation results against the finetumed XMC-GAN (Zhang et al., 2021 model, which has best the FID out of all publicly available models with available images of the same MS- ISCCO prompts, at the time of writing. For each prompt, the output from Parti and XMC-GAN are anonymized and shown to 1,000 independent human evaluators. The results are summarized and shown in Figure g Even though Parti not trained on MS-COCO captions or images, XMC-GAN outputs: 91.7% preference score for image realism and 90.5% for image-text match F When compared against the retrieval model MS-COCO, Parti is evaluated as slightly worse on image realism (45.2% compared to 54.8%) but evaluated as betier on image-text match (55.2% compared to 44.8%). This shows that in nearly half of the comparisons between Parti's output and real images, the former genernted images were judged as more realistic by
our results are overwhelmingly preferred by human annotators
httpps:/ggtthbb... http://twwwb.ooeecccccraaateee// httm//lcccdestee/AAL--pooCoooo
hhttps:/rudalle.nn
We analyzed the cases in which XMC-GAN rated realistic compared to Parti, and found that most of these examples due Parti producing illustrations cartoons, rather than photo-realistic images. While these were gencrally well aligned with the given prompts, the evaluation likely disadvantages Parti since MS-COCO is entirely focused photographs and descriptions them.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Number of Votes 0/5
91.7%
90.5%
1/5
75%
54.8%
55.2%
3/5
50%
45.2%
44.8%
4/5
5/5
Majority Vote
25%
8.3%
0%
XMC-GAN
Parti
Retrieval
Parti
XMC-GAN
Parti
Retrieval
Parti
Image Realism
Image-Text Match
Figure 8: Human evaluation results over 1,000 randomly sampled prompts from the MS-COCO (2014) validation set. Each prompt is rated by 5 independent human evaluators. The zcro-shot Parti models are used in all comparisons. Our model significantly oulperforms XMC-GAN (Zhang el al. 2021), despite the latter being finetuned on MS-COCO, When compared against the retrieval model (retrieval over about 4B training images), Parti is better on image-text match, but worse on image realism (as retrieved images are real images).
people-a a strong statement of the visual quality of images produced by the model. The fact that Parti outputs are preferred for image-text match shows that generation is an important means of producing accurate visual depictions of even the mostly quotidian scenes described in MS-COCO captions.
6.5
MS-COCO (zero-shot)
Training Loss
6.0
350M
Parameters
FID
750M
350M
14.10
5.5
3B
750M
10.71
5.0
20B
3B
8.10
20B
7.23
4.5
0
10
20
30
40
Training Steps (1e4)
Figure 9: Effects of scaling Parti models of different sizes. We show zero-shot FID scores (left) on MS-COCO (2014) and the training loss curves of the corresponding models (right).
Comparison of Model Scaling. We compare four different model sizes Parti, with parameter counts ranging from 350M, 750M 3B and 20B, as shown Table D All four models are trained on the same mixture of datasets with the same image tokenizer and CoCa reranking model described in Section 2.4 Figure g summarizes the corresponding zero-shot FID scores MS-COCO (2014). Parti models are trained with next token prediction loss for text-to-image generation, using softmax cross-entropy loss image codebook. The loss is averaged by 1024 (the total output length) image tokens per example. We observe belter training loss as well as zero-shot FID on MS-COCO when we scale up the model. Specifically, a significant quality jump is achieved by scaling model from 750M to 3B; furthermore, the 20B model outperforms 3B model in more challenging prompts (e.g., text rendering). We highlight qualitatively how these models perform visually in Figures 10 and 13 using challenging prompts from the P2 benchmark (see Section 5.5)
8192-vocab
16

---

Published in Transactions on Machine Learning Research (11/2022)
Parti-350M
Parti-750M
Parti-3B
Parti-20B
WALCOME
FRIEN
POZ
500 Vee
VECME MEEKS
WEL
WELCOME COME FRIENDS!
A portrait photo of a kangaroo wearing orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House holding sign on the chest that says Welcome Friends!
NIOPNC E1 AIEE A OANT N.W C
VERY DEEP LEARNING
E 8m rIr CAR
VERY LEEEP ELEARNINS
ENAYON
green sign that says 'Very Deep Learning" and the edge of the Grand Canyon.
Puffy white clouds in the sky.
A blue Porsche 356 parked in front of a yellow brick wall.
A photo of an astronaut riding horse in the forest. There is river in front of them with water lilies.
map the United States made out of sushi. It is on a table next to glass of red wine.
Figure 10: Qualitative comparison of top-1 images sampled from Parti models of increasing sizes (350M, 750M, 3B, 20B). All Parti models sample 16 images per text prompt and rerank using the same CoCa model described in Section 2.4 We use prompts in the P2 benchmark (Section 4.3 to test WORLD KNOWLEDGE, CRE-GRAINED DETAIL, and WRITING & SYMBOLS.
17

---

Published in Transactions on Machine Learning Research (11/2022)
5.5
Results PartiPrompts
Number of Votes
0/5
73.1%
1/5
64.0%
2/5
60.8%
57.1%
3/5
42.9%
39.2%
36.0%
5/5
26.9%
Majority Vote
Retrieval Parti (20B)
(3B)
Parti (20B)
Retrieval
Parti (20B)
Parti (3B)
Parti (20B)
Image Realism
Image-Text Match
Figure 11: Human evaluation results on PartiPrompts.
Human Evaluations. In addition MS-COCO, we also conduct human evaluations on the P2 benchmark, comparing our 20B model against the 3B variant and the Retrieval baseline. Figure I1 shows that the 20B model is elearly preferred by annotators and image-text match (75.9%). These results offer complementary view to the comparison in Figure S the much more challenging P2 benchmark (Section 4.3). the retrieval bascline was unable produce matching outputs for many prompts. The 3B model closes the gap but the 20B is still preferred in terms of image realism (56.8%) and image-text match (62.7%).
the retrieval baseline both in terms of image realism (63.2%)
Categories
Challenge Aspects
1.0
1.0
0.8
0.8
0.4
0.2
0.2
0.0
Abstract (46)
Animals (307)
Artifacts (286)
Food & Beverage (74)
Arts (57)
Illustrations (122)
Indoor Scenes (39)
Outdoor Scenes (131)
Produce Plants (50)
People (177)
World Knowledge (214)
Vehicles (104)
Fine-grained Detail (311)
Basic (267)
Complex (111)
Linguistic Structures (60)
Imagination (140)
Properties & Positioning (35)
Perspective (70)
Quantity (87)
Simple Detail (232)
style & Format (206)
Writing Symbols (88)
Figure 12: Breakdown of human preferences of the Parti 20B model over the 3B model in terms of P2 categories (left) and challenge aspects (right). Each aspect is shown along with the number of prompts associated with it (e.g., the Abstract category has 46 prompts).
18

---

Published in Transactions on Machine Learning Research (11/2022)
Parti-350M
Parti-750M
Parti-3B
Parti-20B
Infinity
The back of violin
Four surrounding dog
A squirrel gives an apple to bird
Figure 13: Qualitative comparison of scaling Parti models, similar to Figure 10 We show that simple prompts from the P2 benchmark (Section 4.3) can also be quite challenging. These examples test concepts such as ABSTRACT, PERSPECTIVE, QUANTITY, and LINGUISTIC STRUCTURE.
better understand the improvements of the 20B over the 3B models, Figure 12 further brcaks down human preferences of the 20B model in terms of image-tert match across P2 categories (left) and challenge aspects (right). In terms of categories, the 20B model is clearly preferred over most categories, especially ABSTRACT,
19

---

Published in Transactions on Machine Learning Research (11/2022)
WORLD KNOWLEDGE, VEHICLES, and ARTS. The 20B and 3B models are on par for PRODUCE & PLANTS.
In terms of challenge aspects, the 20B model are better over all dimensions, especially WRITING & SYMROLS, PERSPECTIVE, and IMACINATION. See Appendix F for full breakdown on image realism and image-text match for both the Retricval basclinc and the 3B model.
Qualitative comparison. To understand qualitatively the effect of scaling, we present Figure 10 and non-cherry-picked top-1 images sampled from Parti models of increasing sizes (350M, 750M, 3B, 201). All model variants use the same image tokenizer and CoCa reranking model, described in Section 2.4 with sampling of 16 images per text prompt. We use prompts from the P2 benchmark to test the models' capabilities across a range of categories and challenging aspects.
Figure 10 clearly shows improved quality as we scale up model size. The 3B model is, sometimes, as good as the 20B one in terms of the visual quality and image-text alignment for FINE-GRAINED DETAIL prompts such as "astronaut riding a horse" with "water lilics". The "blue Porsche 356" over "ycllow brick wall" is a strong test. of world knowledge: only the 20B model gets it right. The 3B model produces a visually clean car, but it is one which never existed - it seems to merge features of multiple two-seater sports cars from the 1960s. When it comes to more challenging prompts such as those that test WORLD KNOWLEDGE and WRITING & SYMBOLS, the 20B model is able to produce better compositions, such as the "kangaroo wearing an orange hoodie and blue sunglasses" over the "Sydney Opera House" and the sushi-made "map of United States" (interestingly with wasabi in the map from the 20B model), as well as precise text outputs, such as "Welcome Friends!" and "Very Deep Learning", compared to those of the 3B model.
Figure 13 examines models from a different. perspective by demonstrating that short prompts in the P2 can also be quite challenging. The 20B model shows its strong visual abilities when generating ABSTRACT concepts, e.g., the "infinity" sign, and alypical PERSPECTIVE, e.g., "the back of a violin." While both the 3B and 20B models generate animals rather well, the 20B model shines with more photorealistic outputs, e.g., for the prompt "a squirrel gives an apple to a bird", and correct QUANTITY in the case of "Four cats surrounding a dog". Lastly, for the LINGUISTIC STRUCTURE example of considered to be the longest. English word (related to a lung disease), the 20B model generates reasonable illustration of a lung.
6 Discussion
In this section, we discuss our selected examples, then give a walk through of working with complex prompt, and finally provide a break-down (with examples) of limitations of Parti.
6.1 Selected Examples
17 and 19 in the Appendix), we hope to concretely convey some of the strengths of Parti, including its ability to handle complex prompts, multiple visual styles, words-on-image, world knowledge, and more. The top row of Figure [shoss the model accommodating very long and complex description of van Gogh's 's painting The Starry Night-the outputs all come from the same batch and show considerably visual diversity. The other rows show that the model can locate famous landmarks in common scene and adapt styles.
In Figures D and 2 (along with the additional examples in Figures 16
The top row of Figure 2 shows single images for tricky or complex prompts: (A) is short but includes writing of an intentionally misspelled word "toaday" (toad-ay) and an image-within-in siatiinn-mmageeee specification that is successfully executed; (B) has complex prompt that requires knowledge of both Anubis and the Los Angeles skyline; and (C) has extensive visual complexity covering multiple entities and their details and (literal) writing the wall, with photo-realism. (D) shows simple but effective concept combination. (E) provides four outputs from the same batch, showing both diversity and quality of multiple outputs for complex prompt. (F) demonstrates text rendering of a rcasonably long expression along with other complex details, including following the contours of driftwood, fading of the writing and its reflection in the water, and integration of words into stained glass 0 (G) shows that the model can reproduce world knowledge related to
6Te prompts in Figure panel F are: The saying 'BE EXCELLENT TO EACH OTHER' X, where is:
20

---

Published in Transactions on Machine Learning Research (11/2022)
precise visual details for multiple variants of three vehicles that have existed and changed over many decades, while also incorporating additional scene details and color specifications and producing photo-realistic outputs. (H) shows a detailed description in the style of various artists and art movements. (1) shows the adaptation of animals and sports to the style of Egyptian hicroglyphics and additionally placod on Athonian vases (which leatured a different art style), including conforming the painting to the contours of the vases.
6.2 Growing a Cherry Tree
Like other recent work on toxt-to-image generation, this paper includes novel images and the complex prompts given the model to produce them, as discussed in the previous subsection. Naturally, the most challenging and impressive examples are selected (that is, cherry picked), as noted in the captions. As such, they do not typically represent, for example, a single shot interaction in which the model directly produces such an image as its most highly ranked output. As noted in section D we are unable to release our model directly to the public, so in this section we hope to provide a bricf window into the process of increasing descriptive and visual complexity with Parti, including how, along the way, things go right or do not just work immediately.
A key concept we would like to introduce in this endeavor is that of growing the cherry tree - a concept that we believe will be useful in this space going forward. To put fine point on it, many of the prompts and resulting images that are seen in Figure 2 and others in the Appendix (noted as "selected") are not, only cherry picked: they are the result of exploring and probing the model's capabilities the product of an interactive process in which prompter tests a prompt idca, evaluates the outputs holistically, modifies the prompt, and repeats the process. Sometimes, all the outputs are great, and the prompter wants to push the model further. Other times, none of the outputs are ideal, so strategies to change or rephrase the prompt are employed. In this way, one develops a prompt in increments while interacting with the model, in order to produce cherry tree that offers up some great outputs that can ultimately be plucked. While many-perhape most-of the prompts an average user would come up with would be quite bit simpler, this is how we find the breaking points and identify the next opportunities and challenges to focus on. We also expect that designers, artists and other creatives would similarly want to push on these limits; anecdotally, this is how some artists have described working with current text-to- text-to-image models that are available to the public.
As a concrcte example, consider the process of dcveloping a complex prompt as depicted in Figure 14 This shows branching and merging process of creating prompt variations, along with two outputs for each prompt. In each box with a prompt, the best of the top eight 20B Parti outputs (as ranked from all outputs) is given on the left and the worst of the top eight is given on the right.
We start with two core entities, the sloth and the van, in Row 1.
Row 2 adds specific details for each; overall, the model accommodates this well, but notice that box 2(b) has a sloth with two books and an odd bow tie as its worst outcome of eight.
Row 3 shows the first major problems: Box 3(a) has two left arms on the sloth and Box 3(b) has a sloth but is completely missing the van mentioned in the prompt.
Row 4 shows where things start to get particularly tricky: 4(a) is the full combination of the sloth and van with all details, though only by simple concatenation of the respective prompts. The best (left) output manages to get things technically correct, but the van is out of the picture; however, the worst (right) output a confused mess. 4(b) attempts to improve matters by relating the sloth and van via positioning (The sloth slands a few feet in front of shiny VW van), but this ends up producing cartoonish outputs, and even the best output puts the cityscape in the background rather than on the van. 4(c) shows one fix, which is to switch to flowers on the van; with this, the model can produce strong, photorcalistic output that corresponds well with the prompt.
written on red brick wall with graffiti image of a green alien wearing tuzedo. A yellow fire hydrant on sidewall the foreground.
written with carved letters on driftwood.
written in faded paint on the hull of an old wooden boat and reflented in the water. Wide-ongle lens.
written in stained glass window.
21

---

Published in Transactions on Machine Learning Research (11/2022)
2
skot lowsicand bolding ousrooodd wook
with pointeton pasked
grass
Aunitin slath jacket, coebry sala lawiic The slsó hukling quaiissnafuaa log lock.
with annd
quniasoki lan lond. VW prakee graus.
parkello
b
surting lnatee jucket aowhoy skine
bowtiz ing juciet,
boetts The sloth holdking
book. the slach ssand (root aakay The van Aara dhysape parnte on in purtidd on gro
froet skizy VW with flnserr painted midr-angle in lowr fross brice:
bock The slith susadiing
val the leaber jucket couboe fot, and bowte. The sloth boldiig
dafk
vanling leaher jakct, contary tui. a kill bevilz.. The slat teiling geonemsst andiiile book
the
stack frvor of
lnaher quareeo and big book, stence.
jacid, clnaboo bowtli The slidt bolding
spanunealf and fig took. Mtgl-tt Pigh-innteeeioo parsing
Figure 14: depiction several steps in the process of adding details or rephrasing to build a complex prompt that the model responds to well. For each prompt, we show the best output of the top eight ranked images (on the left) and the worst (on the right), using the 20B Parti model. Note that for rows 1-4, the additional text " dsir photogroph. daytime lighting." " is appended the given prompt. See Section 6.2 for discussion.
22

---

Published in Transactions on Machine Learning Research (11/2022)
A second fix is given in 5(b), where the van and photo are promoted to the front of the prompt (rephrasing); alas, the sloth is missing its quarterstaff in the best output, and the worst output is cartoonish and lacking in key details. 5(a) shows that some problems of perspective and representation are resolved by going to a non-photo format (but the city is not on the van, yet again). 5(c) shows that a format change (lo ink sketch) and swapping flowers for the cilyscape again rescue the composition for the best output; however, the worst output again a confused mess with van that has a cowboy hat and sloth arm holding a quarterstaff.
We hope this diagram and its description give a sense of how model like this responds as one adds detail and rephrases prompts. In a sense, this is form of model whispering one stretches such models to their limits. That said, it is often remarkable how much descriptive complexity and diversity the model can readily accommodate. In the next section, we point to specific areas in which the Parti model still systematically runs into difficulty and are thus key areas for improvement.
6.3 Limitations
There are a number of situations that Parti currently handles poorly or inconsistently, or which lead to interesting patterns in outputs even producing some bloopers (which can at times be delightful). The likelihood of all of these errors increases with prompt complexity. Figure 13 provides example prompts and images, along with mention of specific failure modes they exemplify. Note that the examples are selected non-cherries that are often low in the ranked outputs, and that in many cases (though not all) the model produces highly ranked and high-quality output for the prompt. 7 We list the failure modes and discuss them here. Unless otherwise specified, all references are to Figure 15 and are given in terms of panel (capital letter) and image (a-d).
Color bleeding. When color is provided for one object in a description or is very strongly associated with the object itself, but left unspecified for others, it often spreads to the under-specified objects. Examples include bascballs being made ycllow when in the presence of tennis balls (A(b.c)), or a crown being given the color of a shirt (D(a.d)).
Feature blending. Similarly, when two described objects have some similarities, they can become fused as one object or the attributes of another are incorporated. Examples included baseballs with tennis ball fuzz (A(b,c)), hybrids of the Great Pyramid and Mount Everest instead of co-placement (B(c,d)), and the melding of the VW symbol into the kilt's sporran (Fig. 14 boxes 4b, 4c, 5a).
Omission, hallucination, or duplication of details. Especially in complex scenes, the model will at times either omit some mentioned details, duplicate them or hallucinate things that are not mentioned. Examples include missing baseballs in A(d), the missing space shuttle in D(a,c), the missing horse carriage and statue in I(b), and the inclusion (hallucination) of glasses in H(d).
Displaced positioning or interactions. Objects are at times put in the wrong position (especially with increased prompt complexity). Examples include the bectle not grappling with the airplane in C(a,c,d), the space shuttle and drawing of a shuttle in D(b,)), the grass and crack in H(a), and the position of the Earth in I(b).
Counting. Parti can reliably produce up to seven objects of the same type (when not also specifying other objects as in panel A mixing other details). Beyond that, it is mostly imprecise. When there are counts of
Some prompts could not fit in the figure. They are:
(a, robot painted as graffiti on a brick wall. The words "Fly an airplane" are written on the wall. sidewalk in front of the wall, and grass is growing out of cracks in the concrete.
I(a.b) Horses pulling a corriage on the moon's surface, with the Statue of Liberty and Great Pyramid in the background. The Plonet Eorth can seen in the sky. DSLR photo.
I(c) A shiny robot wearing race suit and black visor stands proudly front of an F1 race car, The sun setting on a cityscape the background. comic book ilustration.
I(d) the saying 'BE EXCELLENT TO EACH OTHER' on rough wall with graffiti image of a green alien wearing turedo.
23

---

Published in Transactions on Machine Learning Research (11/2022)
besttiirr
A. Four images generated the batch for the prompt two busclelts is the left dreee lendss lalk. Fathuras: color hineding (b): teature merging (b.c): cousting (a-d): spatial relacions (a-d). (b-d) also include (argusbly reasonable) ballucitacion ground desalls euch as gravel and grase.
B. Geucrated imagee of (a) Mount Evercst and (b)
Four imnges generated in the the prompt á chów desdle dkts size of lank groppics nel life passenger cirpione on the termar. Fatlures: dificcuty in oversiding true life aise leads to big beetle poorly merged with tanx (a) toy airplano (b), attempts pts use perspective to make big beetle (c,d), ballucination of extra beetle (h).
butch for
The Crest Pynused us references and demous inuuuiiaatin the model's knowledge. Two imager (c.4).
generated for The Grest Fyraand Gise attuzee front of Mount Eserest. Failures: feature-mergine of pyramid with pyramiidal top of Everest (d) making snowy Egyptian Pyramid (d); the poromid depicted Khafte's wot Kaufa's (Creat) Pysamid
D. Four images seucrated the same batch. For the promppt A portool of sistue of Ansons vidh crvwn svartng a yeliou sabirt beckground, Failures: wall en A bleeding (a.d); incoreet (s.b,d); (unepeet poettsoning (b.d): mixxing (sue)-
(u.b) Two images geacrated in the same bateh for a crcaeu coloral lebrubodlic seal to a ulats out welli black-tipedd aaza (c.d) images penerated same butch teu applex. Fallures: hard to assigned mostiiple disenstnngee apecttic description (Nose
F. (a,, b) Two images in she same batch for the prouptt stook of shror rod ouler aill Mue sphere on the right two groen conox the ieft. (c. Two images the same batch the prompt gluss
specs sbottle
the
aspect disglaced
onange
Juice
porttioning objecze
cuercetly had apples.)
toat--i--iiiibt.. ansotiation Hollucimation olijecto specilically mentioned absent)
FFLV
ARLARE!
Eich
BONE Z
(a.b) images suaue batela (c.d) colnrful flswers Palhures: sallncinasion diffiouity nvvrriding priors entity overriding (another horse instend of ustroneat)) (b)); cality duplication (ectaras) (e,d).
(6.b) Two iunggs genernted front sail, concovte. (r.d) Two
fail prompia) (a.b) images
geterated complex prompt
woddie
for poodle
(d) allen graffisi with writing. Palhuree: Deplication of objects (a.h); relastis zealing ba.be)) plasicaall impossible configuration (b.oa)); (umppeelied) media blending (d).
buekelt holding dictonnry in band end veiting boses on chaldboard Failures: foross//omiiion rendering toxl (all); use-mention confusion for words (b); incorrect visunl aspect (groms(();; displace positioning (cracke) (a): ballucination (d).
Figure 15: Images from 20B Parti model showing errors and limitations. In the captions, (a), (b), (c), (d) refer top left, top right, bottom left and bottom right, respectively. Note that many of the example images from batches that included successful (and typically more highly ranked) images. See Section 6.3 for detailed discussion.
24

---

Published in Transactions on Machine Learning Research (11/2022)
multiple types of entities, there is a near complete failure, shown in A(a-d). Interestingly, the reranker appears to help with counting; e.g.for five red apples the top six images all have five apples, but thereafter the count varies from four to six. For ten red apples (see E(c,d)), the counts for the top-ranked eight images for one batch were 8, 10, 9, 9, 9, 8, 11, 6.
Spatial relations. While the model often correctly depicts objects specified as above or below each other, it is still inconsistent for that and usually random for left us. right. These failures especially compound when it involves spatial relations between groups of objects (e.g., A(a) and F(a.b)).
Negation and absence. Parti tends to draw items that are mentioned, even when the prompt says a thing is missing, For example, F(c,d) shows outputs that include bananas and orange juice even though the prompt is A plate that has no bananas on it. There is a glass withoul orange juice nexl to il. F(d) has bananas off the plate, but this is just an incidentally interesting example and not an indicator of handling mention-of af-asencce well. We do find some examples with no bananas that are ranked very low, so there also appears to be a compounding effect of both the generator and reranker in this case.
Incorrect visual aspect and media blending. Especially in scenes where mixed media types are involved, such as photo-rcalistic objects along with writing and paintings on walls, some items will jump from being depicted as an object to being depicted as drawing, or vice versa. Examples include the drawn form of Amubis in D(a), the space shuttle as an object in D(b), and the grass as painted on the wall in H(a). We also see media blending, where an object's coherence is lost and it transitions from object to drawing, such as the blending of the drawn Amubis head with body in D(c) and the alien's top part being a painting that connects to non-drawn legs in I(d). While these are interesting visual effects, they were not specified in the prompt. and thus indicate a lack of control over object coherence and appearance in such situations.
Strong visual priors. Certain configurations and visual features are so strongly correlated that it can be hard to nudge the model away from them, especially in the face of other complexities in the description. For examplc, Panel C shows a failed attempt to create a tank-sized rhinoccros bootle-for the most part, the model shrinks the scene (e.g- with a loy plane) or tries to use perspeclive to make the beetle appear visually larger. Its only successful attempt to make a massive beetle errs in including an actual tank. We also found that the model is very resistant to reverse horse-and-rider situation: the only way to have the horse on the astronaut was to avoid riding and instead rephrase this as something akin to horse sitting an astronaut's 's shoulders; this produces some examples that give the requested configuration (G(a)), but, most outputs contain errors, including the astronaut on the horse, the astronaut next to the horse, or even depicting the rider as a horse astronaut (G(b)). In some cases where a horse was put onto an astronaut, the model included yet another astronaut on the horse. As another example, when generating images of statues of Abraham Lincoln wearing clothing such as a polo shirt and a bascball cap, the model can do reliably, but many of the outputs nevertheless have him wearing outfits from well-known statues and paintings of Lincoln.
Strong linguistic priors. Certain terms are highly associated with particular entities or word senses. For example, a soccer ball flying over Lincoln shows ball flying over a statue of Abraham Lincoln. The automotive Lincoln can be brought in by adding cur; SUV, etc. Similarly, a soccer ball flying over a bat produces images of baseball bats; again, this can be fixed, as it were, by adding attributes of animal bats. It is even possible to explore the tensions between competing word senses in examples such as the astronomer married a star, where the presence of astronomer invokes the heavenly body sense of star but the act of marrying invokes (more plausibly) the sense of movie star (Erk & Herbelol, (021). When given illustration of an astronomer marrying star, Parti goes with the heavenly body depiction, even when making it movic star. Switching to politician instcad of astronomer leads to outputs showing a man and woman, but also including a star (iconic five-point representation) in the image. Even with more details such as A photo of an astronomer in a tuxedo marrying a beautiful movie star. The star is wearing a white dress., the model produces some outputs with a wedding couple, but backed by an image of the cosmos. It seems likely that the overwhelming presence of stars as heavenly bodies or icons in the visual training data creates very strong combined linguistic and visual prior that is hard for the model to shift from.
Text rendering errors. It is remarkable that text-to-image models like Parti and Imagen can render text in diverse and contextually appropriate ways on images (see the examples in Figure 2 even though there is no explicitly curated or created training data for learning this ability. Nevertheless, text
Panel F, for example),
25

---

Published in Transactions on Machine Learning Research (11/2022)
rendering is often hit-or-miss. It is common to have few characters off even in simple prompts. For more complex prompis, the errors in rendering text increase with scene and descriptive complexity, as shown in Panel H. The ability to render text can also lead to situations where the model basically tries to render the entire prompt text in the image, c.g. as the title of a book. This usually happens for prompts that are not visually descriptive (e.g. How to succeed in life.), and it necessary to explicitly call out formats like oil painting or illustration in order for these to produce a non-textual output.
Usc-mention errors. The model can render text images, but at times it produces an image (use) rather than text (mention) (e.g, the drawing of an airplane in H(b)), or vice versa (rendering Space shuttle a t-shirt instead of drawing one another output related to D, but not shown).
Disentangling multiple entities. The model is often able to pack quite a lot of detail into an image that contains a single entity, but is much more challenged when there are multiple key entities. This can be seen with the sloth and van example of Figure 14 where the model struggles at their combination (row 3, box b), and where subsequent addition of complexity leads to fewer good outputs. However, it is often even more difficult when the entities are of the same type, such as two animals, demonstrated in E(a,b), where even fairly simple details are spread between the entities.
Stylistic misses. Parti can produce many styles reliably, such as pointillism and woodcut, but others like cubism and surrealism often miss the style at a deeper level, especially when applied to a complex scene. The styles of some specific painters can be modeled well, such as van Gogh and Rembrandt, but others are either lacking or very hit-or-miss, such as Michelangelo (which mostly look the same as adding oil painting.). Interestingly, scale interacts with specific painters: for example, applying in the style of van Gogh actually produces more diverse outputs consistent of style for the 3B model, while outputs are prctty much dominated by Starry Night with the 20B one.
Impossible scenes. Some outputs (usually ones ranked quite low) show entities that are inconsistently integrated, such as I(c), where the robot straddles the car in a nonsensical manner. Other cases involving mixed media lead to similarly bizarre outputs, including a drawing of Anubis's head on a photo-like body (D(c)) and a graffiti alien that extends to a depiction of real feet on the ground (I(d)). (Note that in both of these cases, these would be great outputs if the prompt explicitly specified that these effects were desired.) Other challenges come up in trying to compose complex fantastical scenes, such as the carriage, moon and earth outputs in I(a,b), where the lighting of the statue, moon and Farth are completely inconsistent (a) and the Earth appears sitting on the moon (b).
Zoom and perspective. Parti often produces outputs that are too zoomed in, e.g. showing just part of a vehicle or a subject. While can respond to directives like zoomed out, three-quarters view, wide-angle lens, this still often results in cropping on the subject. To ensure broader view, it is often necessary instead to use other details, such as including shoes flowers to get entire giraffe (as done in G(c,d)).
subject to get the feet or adding a description of a field and
Animal protagonists. Due to concerns discussed in Section g we experimented with many animals acting as stand-ins for protagonists in descriptions. As it turns out, some animals are easier to work with than others. For example, mammals with paws that are more similar to human hands, such as bears, wombats, and raccoons more reliably result in better images than gockos, inscets, and fish. This likely due to the visual and morphological similarity they have with people as well the fact that they are more commonly used as human-like protagonists in the wider visual world (e.g., in cartoons). It is also often necessary to include "photo" or "photograph" in the prompt to push the model to make photo-like like images, and even then it will often produce cartoon-like -ssecall-s with increasing description complexity.
Detailed or tricky visual effects. It is very difficult to get the model to respond to prompts such as a bear emerging from a puzzle (and variations thereof) where one might want to have an Escher-like image of part of the bear appearing as quasi-real and the other being part of the puzzle. In general, controlling such fine-grained specifications seems beyond the current model, and is likely to be better served in an interactive editing setting.
Common misconceptions. Some visual world knowledge is incorrectly understood in the broader world, and this is partially reflected in the data and then the model. For example, it is commonly thought that the
26

---

Published in Transactions on Machine Learning Research (11/2022)
central pyramid in the Giza Complex is the Great Pyramid, but in fact that is the pyramid of Khafre. The true Great Pyramid is Khufu's, which is to the north. Khafre's pyramid is often mistakenly attributed as the great one because it sits on slightly higher ground (but is actually shorter at 136.4 meters compared to Khufu's 146.6 meters) and because it sits prominently behind the Great Sphinx. This mistaken attribution is reflected in images associated with the terms the Great Pyramid, and it shows up many of Parti's outputs for the Great Pyramid (including B(d)).
We hope these observations and breakdown of limitations and error types, and their correspondences in many of the PartiPrompts, will be useful both for contextualizing the strong capabilities we present elsewhere in the paper and for inspiring future work on improving text-to-image generation models in general. In this light, it is also worth recalling WordsEye (Coyne & Sproat 2001), an automatic text-to-scene system built in 2001. It derived dependency structures from prompts, converted them to semantic structures, and then used those to select, position and scale the objects and participants the described scene. Though it is no match for current text-to-image such as Parti on open-domain, broad capabilitics, including world knowledge, it actually could precisely manage several of the above stated limitations, including counting, negation, relative scale, and positioning such that it could produce computer graphics visual (based an explicit 3D reprosentation) corresponding to complex prompts such as:
John uses the crossbow. He rides the horse by the store. The store is under the large willow. The small allosaurus is in front of the horse The dinosaur faces John. A gigantic teacup is in front of the store. The dinosaur is in front of the horse, The gigantic mushroom is in the teacup. The castle is to the right of the store.
EordsEye could also handle specifications of measurements such as lengths and heights, and make correct. visual adjustments for the output image, e.g., for prompts such as The lawn mower is 5 feet tall. John pushes the lawn mower. The cot feet behind John. The cat is 10 feet tall. With the advent of broadly capable but often imprecise models such as Dall-E 2, Imagen and Parti, this should inspire us to revisit ideas and capabilities of earlier systems such WordsFye and aspire toward models that combine breadth, visual quality, and control.
7 Related Work
Text-to-image generation. The task of text-to-image generation tackles the problem of synthosizing realistic images from natural language descriptions. Successful models enable many creative applications. WordsEye (Coyne & Sproat 2001) was a pioneering approach that was based on rule- based methods and explicit 3D reprosontations. One of the carliest models based on deep lcarning (Reed et al. 2016ab proposed using conditional GANs for generating images of birds and flowers from language descriplions. Later works improved upon generation quality by introducing progressive refinement (Zhang et al. 2017 2018a) and using cross-modal attention mechanisms (Xu et al.) 2018 Zhang et al., 2021). Several other works propose using hierarchical models that generate images by explicitly modeling the location and semantics of objects (Reed Hong et al.
et al. 201 6b
2018 Hinz et al. 2019 Koh et al. 2021
Remarkable improvement has been achieved by treating text-to-image generation as sequence modeling problem Esser et al.) 2021 Ding et al.] 2021 Ramesh et al.] 2021 Gafni et al.] 2022 Dayma et al.) 2021
trained on large-scale mmage-eex pairs. two-stage framework is usually exploited Yu et al. 2022a Chang ct al. 2022) where in the first stage the images are tokenized into discrete latent variables. With image lokenization and de-tokenization, text-to-image generation is treated as a sequence-to-sequeecc problem amenable to language models with transformers, which provide opportunities of scaling such models by
applying techniques and observations from large language models (Radford et al. 2018 Du et al. 2022
Chowdhery et al. 2022b Hoffinann et al. 2022
The most recent impressive results have been attained with diffusion models (Nichol et al. 2022 Ramesh
ct al.] 2022, Saharia et al. 2022), where the models are learned to condition on the text encoder of the CLIP Radford et al., (2021) image-text model, or frozen text encoder (like T5 (Raffel al. 2020)) pretrained by language self-supervision. Diffusion models work by positing process of iteratively adding noise to an
27

---

Published in Transactions on Machine Learning Research (11/2022)
image and then learning to reverse that noise conditioned on text imput feature. When used with diffusion model cascading (Ho et al. 2022), these models have proven effective for generating high-fidelity images from text prompts and have achieved saate-of-the-art zero-shot MS-COCO FID scores (Saharia et al. 2022).
Image tokenizers. Previous work has explored tokenizing images into discrete latent variables with a learned deep neural network. Early work like discrete Variational Auto-Encoders (dVAFs) (Rolfe] 2017) optimizes a probabilistic model with discrete latent variables to capture datasets composed of discrete classes. However, dVAEs often generate blurry pixels when applied to natural images. Recent work like VQGAN (Esser et al. 2021 (based on VQVAE (Van Den Oord et al. 2017 )) further applies adversarial loss (Karras et al 2020 and perceptual loss (Johnson et al.) 2016) Zhang et al. 2018b) to synthesize images using convolutional neural networks with self-attention modules. ViT-VQGAN (Yu et al.) 2022a) builds upon VQGAN, with improvements on both architecture and codebook learning. Transformers (Vaswani et al., 2017] are used to encode images into latent variables and decode them back to images. We use VIT- (Yu et al. 2022a
with slight modifications (see Section 2.1 as our image tokenizer.
8 Broader Impacts
Beyond the model capabilities and evaluations presented above, there are broader issues to consider with large-scale models for text-to-image generation. Some of these issues pertain to the development process itself, including the use of large, mostly uncurated training datasets of images obtained from the web with little oversight (discussed also in (Saharia et al. 2022)) ), or conceptual vagueness around constructs in the task ion (Hutchinson et al.] 2022). Since large text-lo-image models are foundation models (Bommasani it all
2021 enabling both a range of system applications as well as finetuning for specific image generation tasks they act as a form of infrastructure which shapes our conceptions of what is both possible and desirable (Hutchinson et al. 2021 Denton et al. 2021). Predicting all possible uses and consequencees of infrastructure is difficult if not impossible, and so responsible AI practices which emphasize transparently
documenting and sharing information about datasets and models are crucial (Mitchell et al.) 2019 Gebru
ct al. 2021 Pushkarna et al. 2022). Although applications are beyond scope of this paper, we discuss here some likely opportunities and risks that can be anticipated.
Creativity and art The ability of machine learned models to produce novel, high-quality images using language descriptions opens up many new possibilities for people to create unique and aesthetically appealing images, including artistic ones. Like a paint brush, these models are a kind of tool that on their own do not produce art-instcad people use these tools to develop concepts and push their creative vision forward. For artists, such models could provide new means of innovation and exploration, including opportunities to create on-the-fly generation of art that sets a theme or style while responding to viewer interactions, or to generate novel and unique visual interactions in vidco game environments. For non-artists, these affordances present a chance to explore their visual creativity through a natural language interface which does not require technical artistic ability. Text-to-image systems could also assist creativity for people with disabilities (ef. (El-Nouby et al.) 2019 Sharma et al. 2018 ), but we caution against doing so without also adopting participatory methods to increase the likelihood of actual needs being met and to avoid misconceptions about disability
[Mankoff et al. 2010).
Assessing the design merit or artistic merit (or lack thereof) of a piece created using machine learned models requires a nuanced understanding of algorithmically based art over the years, the model itself, the people involved and the broader artistic milieu (Browne) 2022). The range of artistic outputs from a model is dependent on the training data, which may have cultural biases towards Western imagery, and which may prevent models from exhibiting radically new artistic styles the way human artists can (Srinivasan & Uchino, 2021).
Visual (mis) communictin.... The Pr history of text-to-image largely consists of assisting communica- tion with non-literate groups including language learners (including children, e.g., storybook illustrations), low-literacy social groups (e.g., up until the late modern period, religious illustrations for low-literacy congre- gations), and speakers of other languages. Parti uses an architecture and stratogy that is directly connected to the neural sequence-to-sqquene models used for machine translation (Wu et al. 2016) and other communica-
tion aids such as sentence simplification (Alva-Manchego et al. 2020 and paraphrasing (Zhou & Bhat 2021
28

---

Published in Transactions on Machine Learning Research (11/2022)
This potentially strengthens the temptation to use large text-to-image models to assist with communication. However, we caution against the use of text-to-image models as communication aids, including for education (cf. (EI-Nouby et al. 2019]), until further research has examined questions of efficacy and utility, since text and image convey meaning in distinct ways and with distinct limitations. Cross-cultural considerations are of special concern, as little research has considered questions of accessibility of computer-generaleed images to members of non- non-Western cultures. Not only do visual styles differ cross-culturally, but also the form and appearance of instances of categories may radically differ across cultures (e.g., wedding attire, food, etc 2017 De Vries et al., 2019)) in ways that might lead to miscommunication.
(Shankar et al.
Deepfakes and disinformation. Given that the quality of model outputs is good enough to be confused for real photographs [ and also because output quality and realism is rapidly improving, there are obvious concerns around using such technology to create deepfakes. One way to mitigate this problem is to apply watermarks that people cannot perceive to every generated image {Luo et al.] 2020p, such that it is possible to verify whether any given image is generated by particular model such as Parti While this approach may miligate risks of disinformation, harms may still occur when an individual's likeness is reproduced without their consent.
Bias and safety. Text-to-image generation models like GLIDE, DALL-E 2, Imagen, Mako-a-Scene, CogView and Parti are all trained on large, often noisy, image-text, datasets that are known to contain biases regarding people of different backgrounds. This is particularly highlighted in Birhane et al's (Birhane et al. 2021 analysis of the LAION-400M dataset (Schuhmann et al. 2021): their study of the dataset surfaced many problems with respect to stereotyping, pornography, violence and more. Other biases include stereotypical representations of people described as lawyers, flight attendants, homemakers, and on. Models trained on such data without mitigation strategies thus risk reflecting and scaling up the underlying problems. Our primary training data is selected and highly filtered to minimize the presence of NSFW content; however, we incorporated LAION-400M during finetuning with classifier guidance - this improved model performance but also led to generation of NSFW images in some contexts. Other biases include those introduced by the use of examples that primarily have English texts and may be biased to cerlain areas of the world. In informal testing, we have noticed, for example, that prompts mentioning wedding clothes seem to produce images biased towards stercotypically female and Western attire.
Intended uses. Due to the impacts and limilations described above, and the need for further exploration of concerns, Parti is research prototype. It is not intended for use in high-risk or sensitive domains, and is not intended to be used for generating images of people.
These considerations all contribute to our decision not to release our models, code or data at this time. Instead, we will focus in follow-on work on further, careful measurement of model biases, along with mitigation strategies such as prompt filtering, output filtering and model recalibration. We also believe that it may be possible to use text-to-image generation models tools to understand biases in large image-text datasets at scale, by explicitly probing them for a suite of known types of bias and also trying to uncover other forms of hidden bias. We will also coordinate with artists to adapt capabilities of high performing text-to-image generation models toward their work, be it for purely creative ends or arl-for-hire. This is all the more important given the intense interest among many research groups and the consequent fast pace of development of models and data for training them. Idcally, these models will augment- rather than replace- human creativity and productivity, such that we all can enjoy world filled with new, varied and responsible aesthetic visual experiences.
9 Conclusion
In this work, demonstrate that autoregressive models like Parti can produce diverse, high-quality images from textual prompts, and furthermore that they present distinct scaling advantages. In particular, Parti is able to represent a broad range of visual world knowledge, such as landmarks, specific years, makes and models of vehicles, pottery types, visual styles and integrate these into novel settings and configurations. We also provide an extensive discussion of the limitations, including a breakdown of many kinds of model
FF.g. DALLL-F. 2 outputs: httpp://www. httpa://wu.natttalll. nattball.... www.lll..aa/yyee www.uuu//aaaaaaaaaa fake-aaaaa-allla- htw.:/y-ffak--aallleee---- hht. sastsa--pasad---ee the-turing-tee test./
29

---

Published in Transactions on Machine Learning Research (11/2022)
errors and challenges, that we hope will be useful both for contextualizing what the model can do and for highlighting opportunities for future research. To this end, the PartiPrompts (P2) benchmark that we release with this work are intentionally crafted to induce many of these error types.
There are also opportunities to integrate scaled autoregressive models with diffusion models, starting with having an autoregressive model generate an initial low-resolution image and then iteratively refining and super-resolving images with diffusion modules (Gu et al. 2022 Ramesh et al. 2022 Saharia al. 2022). It is also crucial to make progress the many significant evaluations and Responsible AI needs for text-to-image generation models. To this end, will conduct more experiments and comparisons with both autoregressive and diffusion models in order to understand their relative capabilities, to address key questions of fairness and bias in both classes of models and strategies for mitigating them, and to identify optimal opportunities for combining their strengths.
References
Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards human-like open-domain chatbot. arXiv preprint arXim:2001.0990 2020.
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. Data-driven sentence simplification: Survey and benchmark. Compulational Linguistics, 46(1):135 187, 2020. doi: 10.1162/coli 00370. URL
htttps//aalaannhooooo.o/////////
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic Propositional Image Caption Evaluation. In ECCV, 2016.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stercotypes. orXim:21100 01963, 2021.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bolgg, Antoine Bosselut, Emna Brunskill, et al. On the opportunities and risks of foundation models. Xiv preprint arXic:2108.0723 07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Hurbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hessc, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in. Neural Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc., 2020. URL https pdf
Kieran Browne. Who (or What) ) Is an AT Artist? Leomando, 10.1162/leon a 02092. URL https://doi. wtt. a a_02092
04 2022. ISSN 0024-094X. doi:
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In CVPR, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-texi pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Tlya Sutskever. Generative pretraining from pixels. In International Conference Machine Learning, pp.
PMLR, 2020.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. Xrxiv preprint ar Xiv: 1904. 10509, 2019.
30

---

Katherine Crowson. Classifier free guidance for autoregressive transformers. 2021. URL https://twita:: htt.://innnnaaaaaa////7/0//21660 Published in Transactions on Machine Learning Research (11/2022)
Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannanch Hajishirzi, and Aniruddha Kembhavi. X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8785 8805, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v112020..mm/ wwtp://...mmmmmam.mmm.ammm URL
https://aclaanhholog. wrg. maa.:mmmmmmlmmm.n.22m.mm 707
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unilying vision-and-language lasks via text generation. In ICML, 2021.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emiiy Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret, Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bremnan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Hellsten, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint XNi:2224.02311 2022a..
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ariic preprint arXi::224..08 002111 2022b.
Aaron Daniel Cohen, Adam Roberts, Alejandra Molina, Alena Butryna, Alicia Jin, Apoorv Kulshreshtha, Ben Hutchinson, Ben Zevenbergen, Blaise Hilary Aguera-Arcas, Chung ching Chang, Claire Cui, Cosmo Du, Daniel De Freitas Adiwardana, Dchao Chen, Dmitry (Dima) Lepikhin, Ed H. Chi, Erin Hoffman-John, Heng-Tze Cheng, Hongrae Lee, Igor Krivokon, James Qin, Jamie Hall, Joe Fenton, Johnny Soraker, Kathy Meier-Hellstern Kristen Olson, Lora Mois Aroyo, Maarten Paul Bosma, Marc Joseph Pickett, Marcelo Amorim Menegali, Marian Croak, Mark Díaz, Matthew Lamm, Maxim Krikun, Meredith Ringel Morris, Noam Shazeer, Quoc V. Le, Rachel Bernstein, Ravi Rajakumar, Ray Kurzweil, Romal Thoppilan, Steven Zheng, Taylor Bos, Toju Duke, Tulsee Doshi, Vincent. Y. Zhao, Vinodkumar Prabhakaran, Will Rusch, YaGuang Li, Yanping Huang, Yanqi Zhou, Yuanzhong Xu, and Zhifeng Chen. Lamda: Language models for dialog applications. In aiv.v. 2022.
Bob Coyne and Richard Sproat. Wordseye: an automatic text-to-scene conversion system. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 2001.
Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Le Khac, Luke Melas, wwwpah/w....oooooooaaaaaaaaeaeee
and Ritobrata. Ghosh. Dall e mini, 2021. URL
Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten, Does object recognition work for everyonc? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 52-59, 2019.
Jeff Dean. Introducing pathways: A next-generation ai architecture, 2021. URL https://blog /
Michael Denkowski and Alon Lavie. Metcor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 776--80, Baltimore,
31

---

Published in Transactions on Machine Learning Research (11/2022)
Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/w1/W144-3342 URL https://www. awwweb.lwwb.... wttp///athoollggg//14-33343
Emily Denton, Alex Hanna, Razvan Amironesci, Andrew Smart, and Hilary Nicole. On the genealogy of machine learning datasets: A critical history of ImageNet. Big Data & Sociely, 8122:20533511/221005555 2021.
Jacob Devlin, Ming-Wo Chang, Kenton Lee, and Kristina Toutanova. BERT: Pro-training of deep bidirectional
transformers for language understanding, In Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short. Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/vl 11/119-14223 URL https://aclanttholog/ www.pe....g///1111/000 1423
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Aduances in Neural Information Processing Systems, 34, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-lo-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation wttp://.rrrgaa////24////// 14211
via hierarchical transformers, 2022. URL
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-expeers. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569. PMI.R, 17-23 mlr, www.s/reessvvv..ee http://v12/////22/ htt/ccu22ccccccc html
Jul 2022. URL https://proceedee
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor. Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10304-10312, 2019.
Katrin Erk and Aurélie Herbelot. How to marry a star: Probabilistic constraints for meaning in context. In Proceedings of the Society for Compulation in Linguistics 2021, pp. 451444444 1511433, Online, February 2021. Association for Computational Linguistics. URL https: http::/oooogyoorr//...s......... scil-1.55
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Mako-a-scene: Scene-based lext-lo-image generation with human priors. ECCV, 2022.
Timmit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datashects for datascts. Communications of the ACM, 64(12):86-922 2021.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. CVPR, 2022.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-aggeenee transformer for speech recognition. INTERSPEECH, 2020.
32

---

Published in Transactions on Machine Learning Research (11/2022)
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3887-3896. PMLR, 13-18 Jul 2020. URL htttps://proceedinn mlr
press/v119/gut #Tees//v19/guo20h.tmm htt.///toohhhh.m
Martin Heusel, Hubert, Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. In ICLR, 2019.
Jonathan Ho and Tim Salimans. Classifier-free diffusson guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learnång Research, 23 (47):1-30 33, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Flena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katic Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL https://axxiv...... wwwpexii.oog//bbbe hts///bs/220 15556
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee, Inferring semantic layout for hierarchical text-to-image synthesis. In CVPR, 2018
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2018. URL
Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. Towards accountability for machine learning datasets: Practices from soft- ware engineering and infrastructure. In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transporency, pp. 560-575, 2021.
Ben Hutchinson, Jason Baldridge, and Vinodkumar Prabhakaran. Underspecification in scene description-to depiction tasks. In Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (ods.), Proceedings of the 38th International Conference on Machine Leurning, volume 139 of Proceedings of Machine Learning Research, pp. 1904-4916. PMLR, 18 Jul 2021a. URL https: httpp:://prooee http:///pocceesg..o.. mlr. ptwps//139/ html
Sung,
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021b.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Faropeaa conference on computer vision, pp. 694-711. Springer, 2016.
33

---

Published in Transactions on Machine Learning Research (11/2022)
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. 1EEE Transactions on Pattern Analysis and Machine Intelligence, 39:664-6111 676, 2017.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 8110-8119, 2020.
Sachoon Kim, Sanghum Cho, Chiheon Kim, Doyup Lee, and Woonhyuk Back. mindall-e on conceptual captions. https://gittubb.com www.coo///aaaa
2021.
Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Text-to-image generation grounded by fine-grained user attention. WACV, 2021.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brusscls, Belgium, November 2018. Association for Computational Linguistics. doi: 10. (0.18633/v1/D18-200 URL htap:::///aaaatholoogg/ org/D18-201 2012
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Europeon conference on computer vision, pp. 740-7 755. Springer, 2014.
Xiyang Luo, Ruohan Zhan, Huiwen Chang, Feng Yang, and Peyman Milanfar. Distortion agnostic deep watermarking, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13548-13555 2020.
Jennifer Mankoff, Gillian R. Hayes, and Devva Kasnitz. Disability studies as a source of critical inquiry for the field of assistive technology. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility, pp. 3-10, 2010.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220 229, 2019.
Deepak Narayanan, Mohammad Shocybi, Jared Casper, Patrick LeGrosley, Mostofa Patwary, Vijay Kor- thikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In SC21: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Tya. Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16860 16804. PMLR, 17-23 Jul 2022. URL https://proceeddngg mlr wwwpps/sssee... masss//vi22/nccho2222 http//iiccooo2a....m html
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fifth fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL
Jordi Pont-Tuset Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari, Connecting vision and language with localized narratives. In ECCV, 2020.
34

---

Published in Transactions on Machine Learning Research (11/2022)
Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data Cards: Purposeful and transparent dataset documentation for responsible AT. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1-67, 2020. httthhtt//mmmml owtp/pppers//22//00077.......
Adilya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060 1069. PMLR, 2016a.
Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schicle, and Honglak Lee. Learning what and where to draw. NewrIPS, 29, 2016b.
Jason Tyler Rolfe. Discrete variational autoencoders. In International Conference on Learning Representalions, 2017. URL httpp:///gaaaeeeeeee.ee///rrrrrp
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitser, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint Xrv::21 Stt:2111.02111 14, 2021.
Rico Semnrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
doi: 10.18653/v1 11/////////1---/// 1162. URL httpss:/aaclanthoooog..ooo wtgp:/g-......-..- 1162
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. In NeurIPS
2017 workshop: Machine Learning for the Developing World, 2017.
Shikhar Sharma, Dendi Suhubdy, Vincent Michalski, Samira Ebrahimi Kahou, and Yoshua Bengio. Chat- painter: Improving text to image generation using dialogue. In International Conference on Learning Representations: Workshop, 2018.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596 4604. PMLR, 2018.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao, Chung- Chann Chiu, et al. Lingvo: modular and scalable framework for sequence-to-sequene modeling- Xiv preprint Xix 1902. 08295, 2019.
35

---

Published in Transactions on Machine Learning Research (11/2022)
Mohammad Shocybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2019. URL bttp:://arxiiv.rg//bb///////////
08053
Ramya Srinivasan and Kanji Uchino. Biases in generative art: A causal look from the lens of art history. In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, FAceT '21, pp. 41 41-51, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781430083997 doi: URL https://doi. https:///oo..org/1... 10. 1145/3442188. 3445869
011114533442188.344
944866
Christian Szegedy, Vincent. Vanhoucke, Sergey Tofle, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In CVPR, 2015.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia. Tsvetkov, and Yuan Cao. Sim Simple visual language model pretraining with weak supervision. In International Conference Learning Representations, 2022. URL https://opeerreviee
net www.nneeeetnnneeeeeeeeeettttteee d=GUrdf
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hidcto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1600 11609.88444 2016. URL http:://rxxi:i org/ abs /tte//669..1811
Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR, 2018.
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et. al. GSPMD: general and scalable parallelization for ml computation graphs. arXiv preprint 2rXic:2105.04663 2021.
Jiahui Yu, Yuchen Fan, Jianchao Yang, Ning Xu, Zhaowen Wang, Xinchao Wang, and Thomas Huang. Wide activation for efficient and accurate image super-resolution. arXiv preprint arxi::1808 008.08718, 2018.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantian image modeling with improved VQGAN. In International Conference on Learning Representations, 2022a. URL httpp::/openrevvie: http://www..t.ooommmmooo//tti/ii
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022b. URL
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CVPR, 2022. URL httpp::/arrxw.orr//bbb./ 04560
Han Zhang, Tao Xu, Hongshong Li, Shaoting Zhang, Xiaogang Wang, Xiaolci Huang, and Dimitris Metaxas. StackGAN: Text rhoto-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
36

---

Published in Transactions on Machine Learning Research (11/2022)
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Mctaxas. 4 Realistic image synthesis with stacked generative adversarial networks. TPAMI, 2018a.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfci Yang. Cross-modal contrastive lcarning for text-to-image generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pultern Recognition, pp. 833 842, 2021.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and rmn-t loss. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1829-7833 IEFE, 2020.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shochtman, and Oliver Wang. The unreasonable effectivencss of deep features perceptual metric. In Proceedings of the IEFE conference on compuler vision and pattern recognition, pp. 586 595, 2018b.
Jianing Zhou and Suma Bhat. Paraphrase generation: A survey of the state of the art In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5075 5086, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 118865//v122211 main.414 414. URL https://aclanthooogg..o wwwp://oooyoro///// 0021 emnle-maall -aammmain.... 414
37

---

Published in Transactions on Machine Learning Research (11/2022)
A More Selected Examples
In Figure 16 Figure 17
Figure 18 and Figure 19 we present more selected examples probing the model's
capabilities to handle complex prompts, multiple visual styles, texls-on-image, world knowledge, and more.
B Image Samples for Cross Reference and Comparison
This section presents image samples with the same text prompts from related work inchuding DALL-E
Ramesh
et al. 2021 (Figure 20)_ GLIDE Nichol et al. 2022) (Figure 21}, unCL.I (Ramesh et al. 2022 (Figure 22
and Imagen (Saharia et al., 2022) (Figure 23 for cross reference and comparison. We exclude some text
prompts or replace some sub-prompts with broader impact concerns that are discussed in Section [
We also
add some new (sub-)prompts (noted in the caption) that are not from the related work to make four images per row for typography.
C Qualitative comparison on MS-COCO
Figure 24 shows qualitative comparison of non-cherry-picked Parti sampled images along with outputs
of other approaches (Ramesh et al. 2021 Nichol et. al. 2022 Gafni et al. 2022 Ramesh et al. 2022) on MS-COCO prompts. Parti demonstrates strong generalization without fine-tuning on specific domains like MS-COCO, and it achieves a high degree of image realism that is typically very close to that of real images.
D PartiPrompts
We list below the full set of calegories (Table and challenge aspects (Table logether with their descriptions and additional examples in the P2.
E
Human Evaluation Procedure
This section describes the human evaluation procedure we conducted for evaluating model outputs. The website used for conducting these evaluations are shown in Figure 25 For each image, we request five independent annotators to select which of two (anonymized) models are preferred, for image realism and image-text match. Annotators were employed as contractors and were paid hourly wages that are competitive for their locale. They have standard rights contractors. They are fluent non-native English speakers.
F
Human Evaluations on PartiPrompts
F.1 Effect of Model Scale
Figure 26 further breaks down the human preferences of the 20B model (over the 3B model) across P2 categories (left) and challenge aspects (right.) ) for image realism. We observe that as compared to the human preference for image-text match (Figure 12). the 20B model wins out on image realism by a smaller margin. This suggests that model scaling mostly benefits language understanding and image-text match, as compared to improvements on image realism. This makes intuitive sense: image realism largely dependent, on the quality of the ViT-VQGAN quantizer, and thus scaling the transformer model would primarily improve the ability of Parti to composite and represent natural language concepts.
In terms of individual categories, the 20B model is preferred over the 3B for the majority of categories, most prominently ABSTRACT, WORLD KNOWLEDGE, VEHICLES, and ARTS. For other categories, the human preference is mostly on par for the 20B and 3B models, with the 3B model being slightly preferred for outdoor scenes. Interestingly, we observe that over the challenge aspects, the 20B model outperforms the 3B model on all challenge aspects, ezcept for BASIC prompts. This suggests that the 3B model scale is sufficient handle the basic prompts PartiPrompts(whice generally contain more simple prompts). Increasing model size to 20B does not help and may slightly harm generation quality on these concepts. In contrast, the
38

---

Published in Transactions on Machine Learning Research (11/2022)
Category
Description
Additional Examples
ABSTRACT
Descriptions that represent ab- stract concepts, including single words and simple numbbers.
inspiration; derision; infinity; 42; 0; fairness; energyi gravity; intelligence; yin-yang meaning of life; A city in 4-dimensional space-time
ANIMALS
Descriptions in which the pri- mary parlicipunts ure unimals.
Stegasaurus; brain coral; A donkey is plaging tug-of-war against an oclopus. The donkey holds the rope in ils mouth. A cal is jumping over the rope; Dogs sitting around a poker table with beer bottles and chips. Their hands are holding cards.
ARTIFACTS
Descriptions of a usually sim- ple object (such as a tool or or- nament) showing human work- manship modification linguished from a natural object Descriptions of eristing paintings or intended to produce novel im- ages in the format of a painting.
a violin; a t-shirt with Carpe Diem written on it; a doorknocker shaped like a lion's head; a lavender backpack with a triceratops stuffed animal head on top; a paranoid android freaking out and jumping into the air because it is surrounded by colorful Easter eggs
dis-
ARTS
an abstract painting with blue, red and black; a super math wizard cat, richly testured oil painting; a sport car melting into clock, surrealist painting in the style of Salvador Dali; Pointing of a pemic-stricken creature, simultaneously corpselike and reminiscent of a sperm or fetus, whose contours are echoed in the swirling lines of the blood-red sky
FOOD & Descriptions of things animals, especially human beings, eat or drink.
margarito; milk pouring from glass into bowl; A bowl of Pho served with bean sprouts on top; bottle of beer next to an ashtray with a half-smoked cigarrelle; A pholo of a hamburger fighting a hot dog in a boxing ring. The hot dog is tired and up agoinst the ropes.
BEVER-
AGE
ILLUSTRAT-
Descriptions of images that in- volve specific types of graphical representations, including geo- metrical objects, diagrams, and symbols.
a metallic blue sphere the left of yellow boz made of felt; the cover of book called 'Backpropaganda by T.C. Gradients; A set of 2x2 emoji icons with happy, angry, surprised and sobbing faces. The emoji icons look like dogs. All of the dogs are wearing blue turtlenecks.
IONS
INDOOR SCENES
Descriptions about objects and participants that occur indoors.
a very fancy French restaurant; a room with two chairs and a painting of the Statue of Liberty; a small kitchen with a white goat in it; A single beam of light enter the room from the ceiling. The beam of light illmiinating an easel. On the easel there is a Rembrandt painting of raccoon
OUTDOOR
Descriplions about objects and participants that occur outdoors.
marina; while bird in frout of dinosaur slanding by some trees; robot painted as graffiti on brick wall. a sidewalk is in front of the wall, and grass is growing out of cracks in the concrete. a scientist; a family of four walking at the beach with waves covering their feet; Renaissance portrayals of the Virgin Mary, seated a loggia. Behind her is hazy and seemingly isolated londscape imagined by the artist and painted using sfumato.
SCENES
PEOPLE
Descriptions where the primary participants are human beings (but not specific individuals, liv- ing dead).
PRODUCE & PLANTS VEHICLES
Descriptions focused plants or their products (fruits, vegetables, seeds, elc).
lily pads; banana without its peel; a pineapple surfing on wave; a flower with large red petals growing on the moon's surface
Descriptions where the focus is on man-made devices for trans- portion.
a rowboat; a friendly car; A sunken ship becomes the homeland of fish; a yellow dump truck filled with soccer balls driving in coral reef. blue whale looms in the background. A Big Ben clock lowering over the city of London; the Sydney Opera House with the Biffel tower sitting on the right, and Mount Everest rising above; A portraat of metal statue of pharaoh wearing sleampank glasses and leather jacket over a whiie I-shirt that has a drawing of a space shutttle on it.
WORLD KNOWL-
Descrilpions socused on objects and places that exist in the real world.
EDGE
Table 7: Full descriptions of all calegories the PartiPrompts (P2) benchmark. We show additional categories and examples that were not included in Table
39

---

Published in Transactions on Machine Learning Research (11/2022)
Challenge
Description
Additional Examples
BASIC
Descriptions about a single subject or concept with little to no detail or embellishment.
a tornado; a hot air balloon; 300; The Starry Night; yin-yang; artificial intelligence
COMPLEX
Descriptions that include many fine-gruined, interacting delails or relationships between multiple par- ticipants.
hot air balloon with yin-yang symbol, with the moon visible in the daylime sky; A high-comitras photo of panda riding horse. The panda is wearing a wizard hat and is reading a book. The horse is standing on a street against a gray concrate wall. Colorful flowers and the word 'PEACE' are painted on the wall. Green grass grows from cracks in the street.
FINE- GRAINED DETAIL
Descriptions that include very de- tailed specifications of attributes actions of entilies or objects in scene.
a Triceratops charging down hill; The Statue of Liberty on cloudy day; bamboo ladder propped up against beautiful fireworks in the sky with red, while and blue; a full moon peeking through clouds at night; a photo of a white bird in front of dinosaur slanding by some trees; an elder politician giving a campaign speech
ook tree;
TMAGINATION
Descriptions that include partici- pants or interactions that are not, or are generally unlikely to be, found in the modern day world. Long and/or abstract words or complex synlactic structures or se- mantic ambiguities.
a smiling banana wearing a bandana; a grand piano next the net of a tennis court; the statue of Liberty next to the Washinglon Monamentt; pholo of light bulb in ouler space traveling the galaxy with a sailing boat inside the light bulb
LINGUISTIC STRUC- TURES
laptop with no letters on its keyboard; To a squirrel, a dog gives an apple; A robol gives a wombat an orange and a lemur a bonona; The trophy doesn't fit into the brown suitcase because it's s too large; a brown mouse laughing at gray cat because a 16 ton weight about to fall ils head
PERSPECTIVE Descriptions that specify particu- lar vicupoints or positioning of the subjects in a scene.
Mars rises the horizon; The frog found itself in the newspa- per; view of a giruffe and zebra in the middle of field; three quarters view of rusty old red pickup truck with white doors and a smashed windshield
PROPERTIES
Descriptions that target precise assignment of properties to enti- ties or objects (often in the con- text of multiple entities objects), and/or the relative spalial orrange- ment of entities and objects with re- spect to one another or landmarks in the scene.
A green heart with shadow; a large yellow sphere behind a small purple pyramid; a brown trash bin to the left of a blue recyling bin; concentric squares fading from yellow on the outside deep orange on the inside; pen-and-ink crosshatched drawing of sphere with durk square on il
& POSI-
TIONING
QUANTITY
Decriptions that specify particular counts of occurences of subjects in scene.
ten red apples; a pile of cash on a wooden table; a group of not more than five meerkats standing with the sun setting behind them
SIMPLE DE- TAII.
Descriptions that include only sim- ple or high-level details.
horse in field of flowers; soccer ball flying over car; subway train coming out of a tumnel; pumpkin with candle in it; an Egyplian statue in the desert
STYLE
& Descriptions that specifically focus on the visual manner in which a subject or scene must be depicted.
a thumbnail image of a gingerbread man; a horse in a field in Minecraft style; the flag of the United Kingdom painted rusty corrugated iron; Anime illustration of the Great Pyramid silling nexl to the Parthenon under a blue night sky of roiling eneryy, exploding yellow stars, and chromatic blue swirls A green sign that says *Very Deep Learning" and is at the edge of the Grand Canyon; Portrait of a liger wearing a train conductor's hat and holding skoteboard that has a yin-yang symbol on it. charcoal sketch; Two cups of coffee, one with latte art of a lovely princess. The other has latter art of frog.
FORMAT
Descriptions that require words & SYM- or symbols to be accuralely repre- BOLS
WRITING
sented in the contert of the visual
scene.
Table 8: Full descriptions of all challenge aspects in the P2. We show additional challenge aspects and examples that were not included in Table
40

---

Published in Transactions on Machine Learning Research (11/2022)
C
Parti
Parti
Parti
A math uitord richly teztored painton
A heary metal atandding reaftop soldle and jomming rlectric gutter under spotiggt. sibuttatiin..
ribby cextured ad painteng of budger deliostely sniffing pellon mase trank. mmall ucterfall bockground.
12
rusly spuccolity dhe foragneandd city alyacrupera distones, with mountuin ccam inciground dark alg. realtitic higl-onntrust undmee Hlustrationn
emeji with lappz, ungry, surpriscd auMiing jaoza, emoji wearing Y. Y ("pian, croums "dogs, bartiensck?
puinting artificia! iutzllpennce
of
"pendes, colorjad esnglasaee" "catorful macarous, couloy Aato"))
IAA
tiger wonring
jonket shouting micerppooee stendtig on photo. ("apsivrsf, stamp" "fng, fily pai", "ydalypua, boulder")
geoko woortin
kodding skateooonn pin-psng symiti
holding fag yimipong rymdet
[piolograph, alusisation, painting, statue, charooni abetch, wanieut, chil'' crayon drawsing,
["Ol coneuss, 7, statec", "Chomie" "Child's erayon dreunng", "Phatograpii" "Charmea!" "Chouese mi?" "Witerealor" "Wandest"
coler vash drawting Chicssse ink and moohh yaonting)
Figure 16: Selected Parti images.
20B model achieves marked improvement in performance more difficult. PartiP PartiPromptsprommpts such
41

---

Published in Transactions on Machine Learning Research (11/2022)
42
Parti
Parti
Parti
surrior uoudle nolshing smord and shield in lighting stnnes The vonoaa stands to fiond of the Are de Thionghhe on day shrouded muie voth the encs high do the oky. renisatee
A wlith be so bort on racd trock. The stott to Acdlinn banana in one houd. There isr banann poet on the track by the background. DSLR photogmaph.
rulot wiihh Mlack vison and the nomuler 42 on its chest. stunde yrosslly in foont of an FJ var can, The van actting on cityerape in the background. undde-zngle sieas nomsin book Clustration.
anone ibaatatton.
DA
photograph the instte trobs. There group ondraels (conse, fourd. There of and background apetng Aungripp.
Tho oups with latte N. The other Ase ("the Biffet Statue Leherty', map Africn")
sindow One them reasing seaugaper. baokgrosad. ("red "labatere", "froge", "raceoomaa 3), ('pungie" "reeer?
resp
clavessp cuo X barate untforme fighting. jumpon moterfflll color moedest illxttation. X "viamolson"" "lectlor"
photograph ktted usarting hendplonse and apeaking inde high-end microphons recording studio.
Oid painting rolet made zudbs, hoiding chop- attris.
Figure 17: Selected Parti images.
those in the COMPLEX and IMAGINATION challenge dimensions. This aligns with our observations on the significant. improvement of the 20B model prompts from the ABSTRACT category.
42

---

Published in Transactions on Machine Learning Research (11/2022)
Parti
Parti
Parti
punk rack aquirral studded loather jacket abouting interophonss while standing and holding phuto.
An otl puinting rablits the stale of American seft beam of light aboues doun armored pranite wombat sarrior statue haiding broed aword. The stuneds
Gatkte, swaring the same clockes the original.
the cxite
10000
autitery fipure shrouded miids pears up from stome strect at impposing gothic buildings surrosnding on oid-fuskiined lanpp sbisee nearöy. oil pointing.
taidy wearing moooropele habruct caps ploto. (standina frant Zock
(a) door lasorting, portil brigittly opening usy obatract
Kilokars Gastte bekded lém,, drösōug spred neer the Colden Gutee Bridgn, our verfing on a tari cst bo New York City,
through
landrouse paintings
(b) trying to food my say in a big confusing sorid, abetrues
oll painting: (c) light and happinoss dhronghont and Jisding
vittng wotonegele in Rio de Jancir with Dois frmoios in the background)
sta way to every cornerr of the world, uhedront oni painsing:
(A) un obatrnct od petcting is deep rnd and Mack wotk
thock patrhes of whiee
0.8
helicapter files over ("the Grond Congen", finnny Golabery machine mode out of risee Aorizon. ("ivadd" " "poper", "mide!")
{'Marr" "Satern",
'Yoeemitt"" 7, "the National Parks)
"Jupiter"
Figure 18: Sclocted Parti images.
43

---

Published in Transactions on Machine Learning Research (11/2022)
Parti
Parti
Parti
photo of light tull onter traosinng the palaxy und satling boet cuaide the light buts.
A telentsion made of uater that dispiags cityrcape at night. datr phote.
digriffidd berver woaring giassea, and coderful nack tie. He stands cail stack books librar. dale
photo of a X of water. ("oupgle lowf? "puln photo of a ("sxmda", "taddy losn "cronodll"" "shagonfy"]
decentee oharova sletet, ('A asction of Great moantsin 7, "The Groat Hypustgle mak", Mesoumerions pyreamia aurreucded by jungle", "Py Mohat with it reffcction))
frxe", "founsdaa clover", Yottu flowrr")
creahing ekyeeroper. potosing ("spaceedpp" style of ("oharda", "Myyrrs" "bree"), ("Hobusst", cublam" "suatenosion")
groph" "caathe")
Nateso", "Shonghaf"
"Snigon" Jussio", "Austin" "Simpapore' "Spdeey", "Wosllle "
Figure 19: Selected Parti images.
F.2 Comparison Against Retrieval Baseline
In addition comparing our 20B and 3B models, we also run human evaluations of the Parti 20B model against the retrieval baseline. Figure 27 and 28 show human preference of the 20B model (over the retrieval
44

---

Published in Transactions on Machine Learning Research (11/2022)
small bird with a pale yellow underside light brown crown and back gray tail and wing tips tip of tail feather bright yellow black eyes and black strip over eyes
armchair in the shape of an avocado
O2.
an illustration of a baby daikon radish in tutu walking dog
ACCORDING
a tapir made of accordion. tapir with the texture of an accordion.
an illustration of a baby hedgehog in a christmas sweater walking a dog
backprop
BACKPROPE
Bakpprot
BACKPROP!!
neon sign that reads "backprop". a neon sign that reads "backprop". backprop noon sign.
Figure 20: Parti image samples with text prompts from DALL-E. (Ramesh al. 2021 for cross reference and comparison.
45

---

Published in Transactions on Machine Learning Research (11/2022)
GODD C0O0
WAN
a hedgehog using calculator
corgi wearing a red bowtie and a purple party hat
robots meditating in vipassana retreat
stained glass window of a panda eating bamboo
a futuristic city in synthwave style
A cozy living room with a painting of corgi on the wall above a couch and round coffee table in frout of couch and vase of flowers on coffee table
Figure 21: Parti image samples with text prompts from GLIDE NNichol et al. 2022 for cross reference and comparison.
46

---

Published in Transactions on Machine Learning Research (11/2022)
shiba inu wearing a beret and black turtleneck
close up of handpalm with leaves growing from it
propaganda poster depicting cat dressed french emperor napoleon holding piece of cheese
A teddybear on a skateboard in Times Square.
DEEP LE $D3
A bowl of soup that looks like a monster X, where X E {"knitted out of wool", Eapray-painted on wall", "made out plasticine", "with tofu says deep learning")
An astronaut riding a horse X, where {"in photorealistic style", "in the style Pop Art", "as pencil drawing", "made of sushi" (the last sub-prompt is not from DALL-E 2))
Figure 22: Parti image samples with text prompts from DALL-E 2 RRaeesh et al. 2022 (a.k: a, unCLIP)
for cross reference and comparison.
47

---

Published in Transactions on Machine Learning Research (11/2022)
black apple and a green backpack.
panda making latte art.
A couple of glasses
sitting table.
HELLO WORLD
FIELL WORLD
Hello World
Hello World
New York Skyline with Hello World written with fireworks the sky.
A raccoon wearing cow boy hat and black leather jacket is
behind the backyard window. Rain droplets on the window.
ty 1.. 64 4 l JIM-S Rupa Po
aa
-ck {ovi-E
2.
B
d/d
2
-2
2-9
photo confused grizzly bear in calculus class.
Figure 23: Parti image samples with text prompts from Imagen and DrawBench (Saharia al. 2022) for
cross reference and comparison.
48

---

Published in Transactions on Machine Learning Research (11/2022)
Real Image
DALL B
GLIDE
Male: Scenn
DALL
Parti
"a group of skiers preparing to ski down mountain."
"a group of elephants walking in muddy water."
green coming down the tracks"
small kitchen low celling"
living area with televiaion and table"
Figure 24: Qualitative comparison of zero-shot images sampled (not cherry picked) from Parti and other modols on MS-COCO prompts. Parti model samples 16 images per text prompt and uses a CoCa model YYu et al. 2022b for re-ranking (Section 2.4
49

---

Published in Transactions on Machine Learning Research (11/2022)
Taakk : Evalaate the given images and rate them in onder
Phan soder that loak vara tru sahul
skondr and bu sen levers yilden connding imprrserenten ding
bether?
Raiigg
Figure 25: Human evaluation setup.
Image Realism (Parti 20B vs. model)
Categories
Challenge Aspects
0.8
0.6
0.6
0.4
0.2
0.0
0.0
Abstract (46)
Animals (306)
Artifacts (286)
Food Beverage (74)
Arts (57)
Illustrations (122)
Indoor Scenes (39)
Outdoor Scenes (131)
Produce & Plants (50)
People (177)
World Knowledge (214)
Vehicles (104)
Fine Fine-grained grained Detail (310)
Basic (267)
Complex (111)
Linguistic Structures (60)
Imagination (140)
Properties & Positioning (35)
Perspective (70)
Quantity (87)
Simple Detail (232)
Style & Format (206)
Writing & Symbols (88)
Figure 26: Breakdown human preferences of the realism of images from the Parti 20B model over the 3B model in terms P2 categories (left) and challenge aspects (right).
baseline) across P2. On image-text match (Figure 27), the Parti model significantly outperforms the retrieval bascline in most categorics, except for ABSTRACT. This highlights the difficulty in generating good images for ABSTRACT prompts, which may be a potential area of improvement for future work. Along different. challenge aspects, Parti outperforms the retrieval baseline in every category on image-text match, which we attribute the improved ability of Parti to synthesize images for diverse and complex prompts. Many of the harder prompts in the P2 are 'adversarial' in nature, and were designed test the limits of text-to-image synthesis models. These prompts generally unlikely have appeared in standard training datasets, and
50

---

Published in Transactions on Machine Learning Research (11/2022)
Image-Text Text Match (Parti 20B
Retrieval)
Categories
Challenge Aspects
1.0
1.0
0.8
0.6
0.4
0.4
0.2
0.0
Abstract (46)
Animals (306)
Artifacts (286)
Food & Beverage (74)
Arts (57)
Illustrations (122)
Indoor Scenes (39)
Outdoor Scenes (131)
Produce & Plants (50)
People (177)
World Knowledge (214)
Vehicles (1 (104)
Fine- mme-grained Detail (310)
Basic (267)
Complex (111)
Linguistic Structures (60)
imagination (140)
Properties & Positioning (35)
Perspective (70)
Quantity (87)
Simple Detail (232)
Style & Format (206)
Writing & Symbols (88)
Figure 27: Breakdown of human preferences of the image-text match of images from the Parti 20B model over the retrieval baseline in terms of P2 categories (left) and challenge aspects (right).
Image Realism (Parti 20B Retrieval)
Categories
Challenge Aspects
1.0
1.0
0.8
0.8
0.4
0.4
0.2
0.2
0.0
0.0
Abstract (46)
Animals (306)
Artifacts (286)
Food & Beverage (74)
Arts (57)
Illustrations (122)
Indoor Scenes (39)
Outdoor Scenes (131)
Produce & Plants (50)
People (177)
World Knowledge (214)
Vehicles (104)
Basic (267)
Complex (111)
Linguistic Structures (60)
Imagination (140)
(310) (310)
Properties& & Positioning (35)
Perspective (70)
Quantity (87)
Simple Detail (232)
Style Format (206)
Writing & Symbols (88)
Fine- Fine-grained Detail
Figure 28: Breakdown of human preferences of the realism of images from the Parti 20B model over the retrieval baseline in terms of P2 categories (left) and challenge aspects (right).
consoquently difficult for retrieval based models which rely on retrieving the closest match from our training corpus. The positive results along this evaluation highlight the strength of generative model such as Parti, and its ability to handle diverse and creative prompts.
51

---

Published in Transactions on Machine Learning Research (11/2022)
Model
CoLa(1)
SST2 (1)
RTE (1)
MRPC (t)
QQP (1)
QNLI (1)
BERT
54.6
92.5
62.5
87.6
87.4
91.0
UNITER
37 4
89.7
55.6
80.3
85.7
86.0
SimVLM
16.7
90.9
84.1
87.2
88.6
CLIP
25.4
88.2
55.2
65.0
53.9
50.5
FLAVA
50.7
90.9
37.8
86.9
87.2
87.3
Encoder after encoder pretraining
55.2
95.9
39.0
90.9
88.5
90.1
Encoder after encoder-decoder training
(w/o encoder pretraining)
15.7
82.5
49.5
81.5
76.8
66.6
(w/ encoder pretraining)
20.4
84.8
53.4
78.3
76.7
78.2
Table Results of Parti text encoder on the GLUE benchmark after pretraining with joint BERT and CLIP objectives or (contimued) training with text-to-image generation objective.
When evaluated on image realism (Figure 28 Parti fares well against the retrieval baseline, and preferred by human annotators across the majority of the P2 categories (except ABSTRACT and INDOOR SCENES), as well as preferred along all challenge aspect dimensions. We note that this is a remarkable feat, considering that the retrieval model retrieves real images from training corpus. These results highlight the ability of Parti to synthesize realistic images, which are deemed by human evaluators close to photorealistic (or even more photorealistic, in some cases).
G Encoder Pretraining
No Encoder Pretmán
Encoder Pretrain
5
"a green clock in the shape of pentagon."
"a winding river desert."
"a cat drinking pint beer."
group harbor docked next a yellow building."
Figure 29: Ablation of text encoder pretraining. In some of the prompts, observe eext-pretrained model outperforms non-pretrained encoders as examples shown above. However average, we observe significant quality improvement by warming-up text encoder. Both models are with 3B parameters trained on the same mixture of datasets for ablation.
While it is straight.orward to warm-slart the model with a pretrained text encoder, we observe the texl-encoder pretraining very marginally helps text-to-image generation loss with 3B-parameter Parti models. Qualitative examples are shown in Figure 29 and quantitative loss comparison is shown Figure 30 We leave this observation as a future research topic on the difference and unification of generic language understanding and visually-grounded language understanding.
52

---

Published in Transactions on Machine Learning Research (11/2022)
5.20
Training Loss
No Encoder Pretraining
Text Encoder Pretraining
5.15
5.10
20.0
22.5
25.0
27.5
30.0
32.5
35.0
37.5
40.0
Training Steps (1e4)
Figure 30: Ablation of text encoder pretraining. We plot text-to-image generation softmax cross-entropy training loss. The training of pretrained text encoder is only slightly better. Both models are with 3B parameters trained on the same mixture of datasets for ablation.
Pixelation Patterns of ViT-VQGAN
Original Image
Reconstructed Image
Zoom-in of Reconstructed Image
Figure 31: Example of pixelation patterns (saturated pixel values at some locations) in the out outputs of VIT-VQGAN (Yu et al. 2022a) architecture when zooming in. can be fixed by removing the final sigmoid activation layer and the logit-laplace loss, exposing the raw values as RGB pixel values (in range [0, 1l).
We notice visual pixelation patterns in some of the output images of VIT-VQGAN when zooming in (sce Appendix H. and further find ill-conditioned weight matrices of the output projection layer before the sigmoid activation function. As fix, we remove the final sigmoid activation layer and the logit-laplace loss, exposing the raw values as RGB pixel values (in range [0, 1]). Conveniently, this fix can hot-swappable into an already trained image tokenizer by finetuning the decoder.
53


Reviews:
[{"review_id": "RgObspi0Sc", "reviewer": "Reviewer_QQkZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths] \n\nOverall, Parti model has shown great potential for the auto-regressive model for image generation. The proposed method is simple, but with many novel techniques to improve the qualities of the generated images. The authors also perform a comprehensive evaluation and human study of Parti model and the limitation of the model is well discussed. \n\nIn terms of technical strength: \n- The first model shows that the auto-regressive model can achieve similar or better image generation performance compared to the recent diffusion model (DALLE2). \n- Compared to DALLE, Parti has taken advantage of recent techniques on image generation model, such as ViT-VQGAN for image tokenizer; Pre-training Encoder with NLP datasets; Classifier free Guidance similar to Make-A-SCENE; Re-Ranking using COCA; super large scale dataset size and model size. \n- The proposed 16-stage GSPMD pipelines reduce the ratio of bubbles and enable large-scale training. \n\nIn terms of evaluation strength: \n-  The authors utilize multiple evaluation metrics for Parti, such as automatic image quality, and image text alignments and achieve the best performance compared to the existing model.  \n-  The authors also create PartiPrompts, which provide very rich concepts for image synthesis evaluation and comprehensive human study has shown the clear advantage of the proposed model. \n- Ablation study with multiple model size is very informative, it shows how parameter size can change the image generation quality. \n- The discussion section and limitation section are very informative and shows the problem Parti model. \n\n[Weaknesses]\n\nMy major concern with this paper is reproducibility. In section 8, the authors mentioned the reasons not to release the models, code, or data for Parti model and it is definitely understandable. However, the concern of bias, safety, and other limitations should not be a barrier to the research communities. It would be great if the authors can release smaller models or help the academic communities to reproduce the proposed methods. Besides, there are some details not clear in the model. \n- In section 2.2, the authors mentioned the decoder use conv-shaped masked sparse attention. Could you provide more information on how the masked is created? Compared to full attention, is the mask mainly benefit the memory or speed or has better image generation qualities? \n- In section 2.4, the authors mentioned CF-guidance in Parti, I wonder what is the prompt for unconditioned training G(z).\n- In section 4.1, the training datasets contain a mix of open-sourced datasets (Laion-400m) and private datasets (FIT400M), etc. It would be great if the authors can train the proposed model on open-sourced datasets and have a fair comparison with existing diffusion models such as stable diffusion. This is also a very informative ablation study that can show the effect of data size on image generation qualities. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents pathways autoregressive text-to-image model (Parti) for high-fidelity photorealistic image generation. Parti is an auto-regressive model similar to DALLE and achieves state-of-the-art performance on image generation (compared to DALLE2 and the latest diffusion models). The authors proposed multiple techniques to improve the image generation quality of Parti: \n\n- Vit-VQGAN instead of VQ-VAE for image tokenizer; \n- Encoder-Decoder model instead of Decoder only model (DALLE); \n- Pre-training of Encoder with Large NLP datasets; \n- Classifier free Guidance for training and inference. \n- Scaling of dataset size and model size. \n\nFor better evaluation, the authors proposed PartiPrompts, which contains 1600 diverse English prompts to test the limit of test-to-image synthesis models. The authors did a comprehensive evaluation and human study for the proposed model and its ablations (model with smaller parameters). There are also very interesting discussions and \"Growing a cherry Tree\" to show the abilities and limits of the propose models. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- More information about decoder conv-shaped masked sparse attention and comparison with full attention. \n- More details about the prompt used in CF-guidance training and inference. \n- Comparison with Stable-diffusion model with similar data and model scale. \n- Discussion and thoughts on the usage proposed Parti model other than art, whether Parti can be used as a source of knowledge for other tasks such as GPT3 in the NLP domain. ", "broader_impact_concerns": "No concerns about ethical implications. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LTNb4r5JtU", "reviewer": "Reviewer_TitR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The final scaled-up Parti model achieves impressive quantitative results on two datasets, compared to previous methods. In addition, the generated images are in very good quality and the selected visualization examples are quite interesting, demonstrating the strong ability of Parti model to understand abstract and complex concepts.\n\n2. The paper presents in-depth analysis on generated images. I especially like the discussions in Section 6.2 and 6.3. Section 6.2 suggests a better visualization method than just cherry picking best results from a few independent prompts. The limitation of Parti model shown in Section 6.3 point out potential future directions for follow-up studies. \n\n3. The P2 benchmark provided can be a valuable asset for model evaluation in future studies.\n\n## Weaknesses\n\n0. It would be beneficial for the community if the smallest model trained on public dataset (for example, on C4 and LAION400M) can be released.\n\n1. As the paper emphasizes on scaling, there are some missing discussions or results regarding the scaling up recipe. Please refer to the next section for more details.\n\n2. For model pre-training and evaluation, multiple large-scale image-text models are used, for example, SimVLM to generate textual descriptions on JFT, CoCa to rerank the generated outputs, and ALIGN-base to build the retrieval baseline. One concern is that all these model are not publicly released. And the other concern is why not use CoCa for all, as CoCa achieves better results in image captioning and ZS retrieval than SimVLM and ALIGN.\n\n3. There are some missing details in addition to the missing discussions on scaling up recipe, which I listed in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In general, the paper introduces a scaling recipe for text-to-image generation, which shows impressive image generation results both qualitatively and quantitively. Here is a summary of the contributions:\n\n1. Proposing a simple encoder-decoder Transformer architecture, together with previously introduced ViT-VQGAN (as image tokenizer and detokenizer), for text-to-image generation. The encoder-decoder Transformer can be scaled up to 20B parameters, which achieves impressive results on MS-COCO/Localized Narratives benchmark.\n\n2. Proposing a P2 benchmark, with 1600 prompts covering diverse concepts and challenging aspects. This benchmark can be a useful asset to prompt and stress test future text-to-image generation models.\n\n3. In-depth analyses conducted on the generated images, such as varying model scales, comparing against retrieval-baseline and etc. The growing a cherry tree analysis is especially interesting and may pose as a better visualization tool than simply cherry picking best results from a few independent prompts.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Questions regarding scaling-up recipe\n1. As an important contribution of the paper is the scaling-up recipe, I feel there are some missing details regarding scaling law observed during pre-training. For example, Is it always the case across different model scales, the more data the better?  We have seen some studies on the scaling law on vision transformers and generative pre-training for image-to-text generation. I am wondering whether the same scaling law applies here. \n2. Another important insight benefiting future studies maybe which model component is more important when scaling Parti model. The paper provides some scaling designs, for example the decoder has much more parameters than encoder, a much bigger image tokenizer decoder is trained, in contrast to the frozen image tokenizer encoder. Some brief reasoning has been provided, but there is no quantitative results supporting these design choices. Further, it is not clear whether scaling up the image tokenizer can further benefit the model performance.\n3. It would be better if including model parameter counts of other methods and compare against Parti in Table 5. \n4. Missing insights on pre-training data scaling, especially on the multimodal pre-training datasets. I am especially curious about the following questions: (1) For JFT-4B, how important it is to randomly replace the original label with machine generated captions. (2) For FIT400M, what is the filtering rules? what if directly use the full 1.8B data from ALIGN. Would similar filtering on LAION-400M benefit the model training, as both are web-crawled images with alt-text descriptions?\n\n- Questions about retrieval baseline\n1. What is the R1/5/10 scores of retrieval baseline on MS-COCO and LN-COCO?\n\n- Questions about P2 benchmark\n1. Prompts in P2 benchmark are interesting and useful, but as there is no ground truth image for all these prompts, the only way to evaluate a model is through human evaluation. Human evaluation is generally very expensive, especially if we are comparing more and more models. This should not be considered as a drawback of P2 benchmark. However, I am wondering whether the authors have insights on alternative evaluation methods.\n\n- Additional missing details\n1. Which model size of SimVLM (for generating captions on JFT) and CoCa (for re-ranking generated outputs) is used?\n2. Regarding the frozen VIT-VQGAN tokenizer. It is not clear how much advantages are from retraining ViT-VQGAN tokenizer on \"images on our pre-training data\", in comparison to direct adoption of a pre-trained ViT-VQGAN tokenizer from the original paper. \n3. For re-ranking, 16 images per text prompt are adopted by default, while DALL-E uses 512 examples. Would more images always benefit Parti performance? Including a curve to illustrate the tradeoff between performance and computation might be better. \n\n- Minor paper edits \n1. Duplicated \"and\" found in the last sentence of \"Automatic image-text alignment.\" paragraph on page 14. \"and then the similarity of the input prompt **and and** the generated caption is assessed via BLEU\" \n2. Section 2.2 mentions \"all models use conv-shaped masked sparse attention\", which reads like both encoder and decoder across all model scales uses conv-shaped masked sparse attention. It becomes more clear in section 3 that only decoder uses conv-shaped masked sparse attention. Also need to more clearly point out the reference of Appendix B.1 Figure 11 is to the DALL-E paper.", "broader_impact_concerns": "The authors have discussed Broader Impacts in Section 8 of the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cUDvSiV2js", "reviewer": "Reviewer_Eo75", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. It is based on a simple architecture with only encoder-decoder and image tokenizer, making it easy to scale up. More importantly, the authors indeed make it scale up and demonstrate that scaling up autoregressive models improves the generation capabilities significantly. \n\n2. It demonstrates the scaling law of text-to-image generation models which enables models to generate images with complex compositions and world knowledge.\n\n3. It uses distributed and parallel mechanism of *Pathways,* making it efficient for training. \n\n4. It achieves the SOTA results on multiple common benchmarks as well as the Localized Narratives dataset whose descriptions are four times longer than MS-COCO on average.\n\n5. It has released a new prompt set *PartiPrompts* which is more diverse and multi-dimensional, giving a more comprehensive picture of text-to-image generation models. In virtue of this set, it gives a systematic analysis of Patri model with a detailed breakdown of error types.\n\n## Weaknesses\n\nI don't really notice any weakness for such a paper focusing on scaling up existing tools/concepts and making it work. The authors have already provided in-depth discussions on the limitations. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Patri, built on image tokenization and autoregressive models, has achieved state-of-the-art performance on multiple benchmarks such as MS-COCO, LN-COCO. Its architecture is very simple, with only encoder-decoder and image tokenzier, making it possible to scale up. Moreover, distributed and parallel mechanism of Pathways further makes it efficient for training. Thus, it demonstrates the scaling law of text-to-image generation models for the first time. It has also provided a comprehensive and systematic analysis of failure modes of text-to-image models with the newly released diverse and multi-dimensional PartriPrompts set which points out numerous future directions. However, due to a lack of controllability and explainability, it may suffer from low security and biases. Although it is not absolutely necessary, it may be interesting to explore or compare text-to-image generation models with diffusion models.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "No", "broader_impact_concerns": "The authors have provided sufficient discussions on broader impact concerns of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents Parti for high-fidelity photorealistic image generation. Parti is a simple auto-regressive model similar to DALLE but with an encoder-decoder architecture, and achieves state-of-the-art performance on image generation (compared to DALLE2 and the latest diffusion models). The authors have performed a comprehensive evaluation and human study of Parti model and the limitation of the model is also well discussed. On the other hand, reviewers also showed concerns about open-source plans and experiments regarding scaling laws, which is not easy to address, but we also could not blame too much on this. Overall, the editor thinks that this paper is of great significance, which shows the power of scaling up simple models for achieving impressive results.
Decision: Accept (poster)