PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Authors: Junsong Chen
Jincheng YU
Chongjian GE
Lewei Yao
Enze Xie
Zhongdao Wang
James Kwok
Ping Luo
Huchuan Lu
Zhenguo Li
Venue: ICLR 2024 spotlight (2024)
Keywords: Text-to-Image Diffusion
Transformer
Primary area: generative models
TL;DR: PixArt-α is a cost-effective, eco-friendly text-to-image diffusion model that delivers high-quality images, reduces training expenses and CO2 emissions, and fosters innovation in the AIGC community.

Abstract: The most advanced text-to-image (T2I) models require significant training costs (e.g., millions of GPU hours), seriously hindering the fundamental innovation for the AIGC community while increasing CO2 emissions. This paper introduces PixArt-$\alpha$, a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-art image generators (e.g., Imagen, SDXL, and even Midjourney), reaching near-commercial application standards. Additionally, it supports high-resolution image synthesis up to 1024px resolution with low training cost, as shown in Figure 1 and 2. To achieve this goal, three core designs are proposed: (1) Training strategy decomposition: We devise three distinct training steps that separately optimize pixel dependency, text-image alignment, and image aesthetic quality; (2) Efficient T2I Transformer: We incorporate cross-attention modules into Diffusion Transformer (DiT) to inject text conditions and streamline the computation-intensive class-condition branch; (3) High-informative data: We emphasize the significance of concept density in text-image pairs and leverage a large Vision-Language model to auto-label dense pseudo-captions to assist text-image alignment learning. As a result, PixArt-$\alpha$'s training speed markedly surpasses existing large-scale T2I models, e.g., PixArt-$\alpha$ only takes 10.8% of Stable Diffusion v1.5's training time (~675 vs. ~6,250 A100 GPU days), saving nearly \\$300,000 (\\$26,000 vs. \\$320,000) and reducing 90% CO2 emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training cost is merely 1%. Extensive experiments demonstrate that PixArt-$\alpha$ excels in image quality, artistry, and semantic control. We hope PixArt-$\alpha$ will provide new insights to the AIGC community and startups to accelerate building their own high-quality yet low-cost generative models from scratch.

Published as a conference paper at ICLR 2024
PIXART-a: FAST TRAINING OF DIFFUSION TRANS- FORMER FOR PHOTOREALISTIC TEXT-TO-IMAGE SYNTHESIS
Junsong 12,3 Jincheng Yu-.4, Chongjian Ge
Lewei Yao Enze Xie*t, Huchuan Lu Zhenguo Li*
Yue Wu', Zhongdao Wang', James Kwok', Ping Luo
Huawei Noah's Ark Lab Dalian University of Technology
"KKU 'HKUST
jschen@ma 11 d.luut.du.ccn, rhettgee@conneett. hku.hk,
(yujinchengll 1 xie...nee i whwnnuuoehuawwi.coo .com
Project Page: https: / ptppppxxatttaaaa pha. wetha...ttttt... thub. io/
ABSTRACT
The most advanced text-to-image (T2I) models require significant training costs (e.g.. millions of GPU hours), seriously hindering the fundamental innovation for the AIGC community while increasing CO2 emissions. This paper introduces PIXART-O, a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-ar afrtthe-rt image generators (e.g., Imagen, SDXL. and even Midjourney), reaching commercill application standards. Addi- tionally, supports high-resolution image synthesis up to 1024 x 1024 resolu- tion with low training cost, as shown in Figure 1 and 2. To achieve this goal, three core designs are proposed: (1) Training strategy decomposition: We de- vise three distinct training steps that respectively optimize pixel dependency, text- image alignment, and image aesthetic quality: (2) Efficient T2I Transformer: We incorporate cross-attention modules into Diffusion Transformer (DiT) to inject text conditions and streamline the computation-ime intensive class-condition branch; (3) High-informative data: We emphasize the significance of concept density in text-image pairs and leverage a large Vision-L.nnuuge model to auto-label dense pseudo-captions to assist text-image alignment learning. As result, PIXART-a's training speed markedly surpasses existing large-scale T2I models, e.g., PIX a only takes 12% of Stable Diffusion v1.5's training time (~753 vs. ~6255 A100 GPU days), saving nearly $300,000 ($28,400 vs. $320,000) and reducing 90% CO2 emissions. Moreover, compared with a larger SOTA model, RAPHAËL, our training cost is merely 1%. Extensive experiments demonstrate that PIXART-a excels in image quality, artistry, and semantic control. We hope PIXART-a will provide new insights to the AIGC community and startups to accelerate building their own high-quality yet low-cost generative models from scratch.
INTRODUCTION
Recently, the advancement of text-to-image (T2I) generative models, such as DALL-E (OpenAI, 2023), Imagen (Saharia et al., 2022), and Stable Diffusion (Rombach et al., 2022) has started a new era of photorealistic image synthesis, profoundly impacting numerous downstream applica- tions, such as image editing (Kim et al., 2022), video generation (Wu et al., 2022), 3D assets cre- ation (Poole et al., 2022), etc.
However, the training of these advanced models demands immense computational resources. For in- stance, training SDv1.5 (Podell et al., 2023) necessitates 6K A GPU days, approximately costing $320,000, and the recent larger model, RAPHAEL (Xue et al., 2023b), even costs 60K A100 GPU days requiring around $3,080.000, as detailed Table 2. Additionally, the training contributes substantial CO, emissions, posing environmental stress; e.g. RAPHAEL's (Xue et al., 2023b) train-
Equal contribution. Work doue during the intersships of the four students at Huawei Noah's Ark Lab. Project lead and corresponding author.

---

Published as a conference paper at ICLR 2024
beautiful with Whowntstins and rivers in small village
CHtulla, alien, hage towering, chuerch, an evil shatws with sicoltonn
Firate ship trapped in small cactus with happy face Sahara desert
cosinic indlstrou nobula
his hand
product #hotography, world of warcraft ore warrier, white backeround
little girl with red halr sitting table, portrait, kodak portray
paper artwork, layored colorful Chinese dragon surrounded by clouds
traveler vavigating
paper,
boat in countless mountaaiss
Chinaso ink painting
Ein,, focused yot playful, ready for a compotitive matchuppphhotoroaiin photoroalistic quality with vibes
Oppewindimer sits, on the beach on chair, watching waclear exposition with 'ange musktroomn cloud, 12.0mmm
Figure 1:
Samples produced by PIXART-a exhibit exceptional quality, characterized by remark- able level of fidelity and precision in adhering to the provided textual descriptions.
2

---

Published as a conference paper at ICLR 2024
ing results in 35 tons of CO2 emissions, equivalent to the amount one person emits over years, as shown in Figure 2. Such a huge cost imposes significant barriers for both the research community and entrepreneurs in accessing those models, causing significant hindrance to the crucial advance- ment of the AIGC community. Given these challenges, a pivotal question arises:
Can we develop a
high-quality image generator with affordable resource consumption?
Human 1 year 55 ton
CO2 Emission (ton)
RAPHAEL
7x
30
iPirrto
0.8x
0.7%
1.2%
Data Usage (B)
2% training time
15.8%
0.5%% data usage
0,08
dardata usage
GigaGAN
RAPHACI
DALL-E2
Imagen
GigaGAN
SDV1.5
$3.08M
US Dollars (S)
DALL-E
$2.14M
Imagen
2M
$366K
$320K
$245K
0.91%
300k
PIXART-a
10203
20000
30000
40000
50000
800C0
30k
$28K
Training Time (A100 GPU Days)
RAPHAEL DALLE2
Imagen
50v1.5
GIIGAN
Pisket
(a) Comparison of data usage training time
(b) Comparison of CO2 emission and training cost
Figure 2: Comparisons of CO2 emissions' and training cost among T2I generators. PIXART-a achieves an exceptionally low training cost of $28,400. Compared to RAPHAEL, our CO2 emissions and training costs are merely 1.2% and 0.91 %, respectively.
In this paper, we introduce PIXART-a, which significantly reduces computational demands of train- ing while maintaining competitive image generation quality to the current ofate-of-the-are image generators, as illustrated in Figure 1. To achieve this, we propose three core designs:
Training strategy decomposition. We decompose the intricate text-to-image generation task into three streamlined subtasks: (1) learning the pixel distribution of natural images, (2) learning text- image alignment, and (3) enhancing the aesthetic quality images. For the first subtask, we propose initializing the T2I model with low-cost class-condition model, significantly reducing the learning cost. For the second and third subtasks, we formulate a training paradigm consisting of pretraining and fine-tuning: pretraining text-image pair data rich in information density, followed by fine- tuning on data with superior aesthetic quality, boosting the training efficiency.
Efficient T2I Transformer. Based on the Diffusion Transformer (DiT) (Peebles & Xie, 2023), we incorporate cross-attention modules to inject text conditions and streamline the computation- intensive class-condition branch to improve efficiency. Furthermore, we introduce a re- parameterization technique that allows the adjusted text-to-image model to load the original class- condition model's parameters directly. Consequently, we can leverage prior knowledge learned from ImageNet (Deng et al., 2009) about natural image distribution to give reasonable initialization for the T21 Transformer and accelerate its training.
High-informative data. Our investigation reveals notable shortcomings in existing text-image pair datasets, exemplified by LAION (Schuhmann et al., 2021), where textual captions often suffer from a lack of informative content (i.e., typically describing only a partial of objects in the images) and severe long-tail effect (i.e., with a large number of nouns appearing with extremely low fre- quencies). These deficiencies significantly hamper the training efficiency for T2I models and lead to millions of iterations to learn stable text-image alignments. To address them, propose auto- labeling pipeline utilizing the state-of-the-art vision-language model (LLaVA (Liu et al., 2023)) to generate captions on the SAM (Kirillov et al., 2023). Referencing in Section 2.4, the SAM dataset is advantageous due to its rich and diverse collection of objects, making it an ideal resource for creating high-infoo www-iiinoottooennniiiei text-image pairs, more suitable for text-image alignment learning.
'The method for estimating CO2 emissions follows Alexandra Sasha Luccioni (2022).
The training cost refers the cloud GPU pricing from Microsoft (2023) Azure in September 20, 2023.
3

---

Published as a conference paper at ICLR 2024
Our effective designs result in remarkable training efficiency for model, costing only 753 A100 GPU days and 428,400. As demonstrated in Figure 2, our method consumes less than 1.25% training data volume compared to SDv1.5 and costs less than 2% training time compared to RAPHAEL. Compared to RAPHAEL, our training costs are only 1%, saving approximately $3,000,000 (PIXART-a's $28,400 vs. RAPHAEL's $3,080,000). Regarding generation quality, our user study experiments indicate that PIXART-a offers superior image quality and semantic alignment compared existing SOTA T2I models (e.g. DALL-E 2 (OpenAI, 2023), Stable Dif- fusion (Rombach et al., 2022), etc.), and its performance on T21-CompBench (Huang et al., 2023) also evidences our advantage in semantic control. We hope our attempts to train T2I models effi ciently can offer valuable insights for the AIGC community and help more individual researchers or startups create their own high-quality T2I models at lower costs.
2
METHOD
2.1 MOTIVATION
The reasons for slow T2I training lie in two aspects: the training pipeline and the data.
The T2I generation task can be decomposed into three aspects: Capturing Pixel Dependency: Generating realistic images involves understanding intricate pixel-level dependencies within images and capturing their distribution; Alignment between Text and Image: Precise alignment learning is required for understanding how to generate images that accurately match the text description; High Aesthetic Quality: Besides faithful textual descriptions, being aesthetically pleasing is another vital attribute of generated images. Current methods entangle these three problems together and directly train from scratch using vast amount of data, resulting in inefficient training. To solve this issue, we disentangle these aspects into three stages, as will be described in Section 2.2.
Another problem, depicted in Figure 3, is with the quality of captions of the current dataset. The cur- rent text-image pairs often suffer from text- text-image misalignment, deficient descriptions, infrequent diverse vocabulary usage, and inclusion of low-quality data. These problems introduce difficulties in training, resulting in unnecessarily millions of iterations to achieve stable alignment between text and images. To address this challenge, we introduce an innovative auto-labeling pipeline to generate precise image captions, as will be described in Section 2.4.
Problems
Text-image misalignment
Deficient descriptions
Infrequent vocabulary
Samples
Raw caption
AH137001950 Saudi Anbia Gold One Guiines MS-66 NOC
2018 Kawccakii Ski Ulraa S10LX in Uniooville, Vitginia
Whtt science says about pu tea?
The inage featura closoup of'a of lis with i wooden table. The descrbbe
The image shows man working on scuba diving equipmentt Biue Water Divers, The mar is siring rable, werking
The iage features man riding jet sla body of water. The ali and white, and is being med for smiling,
LLavA refined caption
which type Chisese dimll background adds
of warkshop shes, with The tolls oquigment visble in background.
kailh
equipanent possily fixing
recreatioaal purposes. The osjoying time the water. souse in beach aren.
The presence of potted
insiiating
a tonch of nature and freehness to the sceno.
Figure 3: LAION raw captions v.s LLaVA refined captions. LLaVA provides nigh-inforrmttinn density captions that aid the model in grasping more concepts per iteration and boost text-image alignment tiigmeetefficiieen efficiency.
2.2 TRAINING STRATEGY DECOMPOSITION
The model's generative capabilities can be gradually optimized by partitioning the training into three stages with different data types.
Stagel: Pixel dependency learning. The current class-guided approach (Peebles & Xie, 2023) has shown exemplary performance in generating semantically coherent and reasonable pixels in individual images. Training a class-conditional image generation model (Peebles & Xie, 2023) for natural images is relatively easy and inexpensive, as explained in Appendix A.5. Additionally,
4

---

Published as a conference paper at ICLR 2024
find that suitable initialization significantly boost training efficiency. Therefore, boost our model from an peggeNet-pretrainge model, and the architecture of our model is designed to be compatible with the pretrained weights.
Stage2: Text-image alignment learning. The primary challenge transitioning from pretrained class-guided image generation to text-to-image generation is on how to achieve accurate alignment between significantly increased text concepts and images.
This alignment process is not only time- consuming but also inherently challenging. To efficiently facilitate this process, construct a dataset consisting of precise text-image pairs with high concept density. The data creation pipeline will be described in Section 2.4. By employing accurate and information-rich data, our training process can efficiently handle larger number of nouns in each iteration while encountering considerably less ambiguity com- pared to previous datasets. This strategic ap- proach empowers network to align textual descriptions with images effectively.
Text
t block
XN
with a hoppy face the
Scale g(Ei.rr)
Sahora desert
Pointwise
Fesatooward
Scale, ptHr
T5
encoder
Shared by
N blocks
Text
feature
Multi-Head
T'me
Image
Scale
Stage3: High-resolution and aesthetic image generation. In the third stage, we fine-tune our model using high-quality aesthetic data for high-resolution image generation. Remarkably, we observe that the adaptation process in this stage converges significantly faster, primarily owing to the strong prior knowledge established in the preceding stages.
Multi-tead Self-Artention
Scale, Shift g(Fi, 15)
encoder
+ noise,
Z
Noised latent
Tunable Parameters
Frozen Parameters
Figure 4: Model architecture of PIXART-a. A cross-attention module is integrated into each block to inject textual conditions. To optimize ef- ficiency, all blocks share the same adaLN-single parameters for time conditions.
Decoupling the training process into different stages significantly alleviates the training diffi- culties and achieves highly efficient training.
2.3 EFFICIENT T2I TRANSFORMER
PIXART-a adopts the Diffusion Transformer (DiT) (Peebles & Xie, 2023) as the base architecture and innovatively tailors the Transformer blocks to handle the unique challenges of T2I tasks, as depicted in Figure 4. Several dedicated designs are proposed as follows:
Cress-Attention layer. We incorporate a multi-head cross-attention layer to the DiT block. It is positioned between the self-attention layer and feed-forward layer so that the model can ficxibly interact with the text embedding extracted from the language model. To facilitate the pretrained weights, we initialize the output projection layer in the cross-attention layer to zero, effectively acting identity mapping and preserving the input for the subsequent layers.
AdaLN-single. We find that the linear projections in the adaptive normalization layers (Perez et al., 2018) (adaLN) module of the DiT account for a substantial proportion (27%) of the parameters. Such a large number of parameters is not useful since the class condition is not employed for our T2I model. Thus, we propose adaLN-single, which only uses time embedding as input in the first block for independent control (shown on the right side of Figure 4). Specifically, in the ith block, let ((() = [3(2), a(i) (6) ,,,,7 a (i)) be tuple all the scales and shift parameters in adaLN. In the DiT, ((i) is obtained through a block block-specific MLP Sti) = f( (c++), where and denotes the class condition and time embedding, respectively. However, in adaLN- single, one global set of shifts and scales are computed as S f(t) only at the first block which shared across all the blocks. Then, 5()) obtained as 5(0) = g(3, E(O), where g is a summation function, and E(i) a layer-specific trainable embedding with the same shape as S, which adaptively adjusts the scale and shift parameters in different blocks.
5

---

Published as a conference paper at ICLR 2024
Re parameteerization To utilize the aforementioned pretrained weights, all E are initialized to values that yield the same 8(4) as the DiT without c for a selected (empirically, we use I = 500). This design effectively replaces the layer-specific MIPs with global MI. and layer-specific trainable embeddings while preserving compatibility with the pretrained weights.
Experiments demonstrate that incorporating global MLP and layer-wise embeddings for time-step information, as well as cross- roos-attention layers for handling textual information, persists the model's generative abilities while effectively reducing its size.
2.4 DATASET CONSTRUCTION
Image-text pair auto-labeling. The captions of the LAION dataset exhibit various issues, such as text-image misalignment, deficient descriptions, and infrequent vocabulary as shown in Figure 3. To generate captions with high information density, we leverage the slate-of-the-ane
vision-language model LLaVA (Liu et al., 2023). Employing the prompt, "Describe this image and its style in very detailed manner", we have significantly improved the quality of captions, as shown in Figure 3.
However, it is worth noting that the LAION dataset predominantly comprises of simplistic prod- uct previews from shopping websites, which are not ideal for training eex-tto-image generation that seeks diversity in object combinations. Consequently, illov et al., 2023), which is originally used for segmentation tasks but features imagery rich in diverse objects. By applying LLaVA to SAM, we have successfully acquired high-quality text-image pairs characterized by a high concept density, as shown in Figure 10 and Figure in the Appendix.
have opted to utilize the SAM dataset (Kir-
In the third stage, we construct our training dataset by incorporating JourneyDB (Pan et al., 2023) and a 10M internal dataset to enhance the aesthetic quality of generated images beyond realistic photographs. Refer to Appendix A.5 for details.
As a result, we show the vocabu- lary analysis (NLTK, 2023) in Ta- ble 1, and we define the valid dis- tinct nouns as those appearing more than 10 times in the dataset. We apply LLaVA on LAION to generate LAION- LLaVA. The LAION dataset has 2.46 M distinct nouns, but only 8.5% are valid. This valid noun proportion sig- nificantly increases from 8.5% to 13.3%
Table 1: Statistics of noun concepts for different datasets. VN: valid distinct nouns (appearing more than 10 times); DN: total distinct nouns; Average: average noun count per image.
Dataset
VN/DN
Total Noun
Average
LAION LAION-LLaVA
210K/2461K 8.5%
72.0M
6.4/Img
85K/646K = 13.3%
233.9M
20.9/Img
SAM-LIaVA
23K/124K = 18.6%
327.9M
29. 29.3/Img
Internal
152K/582K
26.1%
136.6M
12.2/lmg
with ILaVA--lbeled captions. Despite
LAION's s original captions containing staggering 210K distinct nouns, its total noun number is a
mere 72M. However, LAION-LLaVA contains 234M noun numbers with 85K distinct nouns, and the average number of nouns per image increases from 6.4 to 21, indicating the incompleteness of the original LAION captions. Additionally, SAM-LLaVA outperforms LAION-LLaVA with a total noun number of 328M and 30 nouns per image, demonstrating SAM contains richer objectives and superior informative density per image. Lastly, the internal data also ensures sufficient valid nouns and average information density for fine-tuning. VA-abllle captions significantly increase the valid ratio and average noun count per image, improving concept density.
3 EXPERIMENT
This section begins by outlining the detailed training and evaluation protocols. Subsequently, we provide comprehensive comparisons across three main metrics. We then delve into the critical de- signs implemented in PIXART-a to achieve superior efficiency and effectiveness through ablation studies. Finally, we demonstrate the versatility of our PIXART-a through application extensions.
3.1 IMPLEMENTATION DETAILS
Training Details. We follow Imagen (Saharia et al., 2022) and DeepFloyd (DeepFloyd, 2023) to employ the T5 large language model (ie., 4.3B F1an-T5-XXI.) as the text encoder for conditional

---

Published as a conference paper at ICLR 2024
Table We thoroughly compare the PIXART-a with recent T2I models, considering several essen- tial factors: model size, the total volume of training images, COCO FID-30K scores (zero-shot), and the computational cost (GPU days'). Our highly effective approach significantly reduces resource consumption, including training data usage and training time. The baseline data is sourced from GigaGAN (Kang et al., 2023).
the table denotes an unknown internal dataset size.
Method
Type
# Params
Images
FID-30K
GPU days
DALLE
Diff
12.0B
250M
27.50
GLIDE
Diff
5.0B
250M
12.24
LDM
Diff
1.4B
400M
12.64
DALL-E2
Diff
6,5B
650M
10.39
41,667 A100
SDv1.5
Diff
0.9B
2000M
9.62
6,250 A100
GigaGAN
GAN
0.9B
2700M
9.09
4,783 A100
Imagen
Diff
3.0B
860M
7.27
7,132 A 100
RAPHAEL
Diff
3.0B
5000M+
6.61
60,000 A A100
PIXARI-a
Diff
0.6B
25M
7.32
753 A100
feature extraction, and use DIT-X1 (Peebles & Xie, 2023) as our base network architecture. Unlike previous works that extract a standard and fixed 77 text tokens, we adjust the length of extracted text tokens to 120, as the caption curated in PIXART-a is much denser to provide more fine-grained details. To capture the latent features of input images, we employ a pre-trained and frozen VAE from LDM (Rombach et al., 2022). Before feeding the images into the VAE, we resize and center-crop them to have the same size. We also employ multi-aspect augmentation introduced in SDXL (Podell al., 2023) to enable arbitrary aspect image generation. The AdamW optimizer (Loshchilov & Hutter, 2017) is utilized with a weight decay of 0.03 and a constant 2e-5 learning rate. Our final model is trained 64 V100 for approximately 26 days. See more details in Appendix A.5.
Evaluation Metrics. We comprehensively evaluate PIXART-a via three primary metrics, iee, Fréchet Inception Distance (FID) (Heusel et al., 2017) on MSCOCO dataset (Lin et al., 2014), com- positionality on T21-CompBench (Huang et al., 2023), and human-preferent rate on user study,
3.2 PERFORMANCE COMPARISONS AND ANALYSIS
Fidelity Assessment. The FID is a metric evaluate the quality of generated images. The compar- ison between our method and other methods in terms of FID and their training time is summarized in Table 2. When tested for zero-shot performance on the COCO dataset, PIX ART-R achieves a FII) score of 7.32. It is particularly notable as it is accomplished in merely 12% of the training time (753 vs. 6250 A 110 GPU days) and merely 1.25% of the training samples (25M vs. 2B images) relative to the second most efficient method. Compared to state-of-the-ar methods typically trained using substantial resources, PIXART-a remarkably consumes approximately 2% of the training resources while achieving comparable FID performance. Although the best best-perrorming model (RAPHEAL) exhibits a lower FID, it relies on unaffordable resources (i.e., 200x more training samples, 80x longer training time, and 5x more network parameters than PIXART-a). We argue that FID may not be an appropriate metric for image quality evaluation, and it is more appropriate to use the evaluation of human users, as stated Appendix A.8. We leave scaling of PIXART-a for future exploration for performance enhancement.
Alignment Assessment. Beyond the above evaluation, we also assess the alignment between the generated images and text condition using T21- 221-Compeench (Huang et al., 2023), comprehensive benchmark for evaluating the compositional text-to-image generation capability. As depicted in Ta- ble 3, we evaluate several crucial aspects, including attribute binding, object relationships, and com- plex compositions. PIT-RT-a exhibited outstanding performance across nearly all (5/6) evaluation metrics. This remarkable performance is primarily attributed to the text-image alignment learning in Stage 2 training described in Section 2.2, where high-quality text-image pairs were leveraged to achieve superior alignment capabilities.
'To ensure fairness, we convert the V100 GPU days (1656) of our training to A100 GPU days (753), assuming 2.2x speedup in U-Net training on A100 compared to V100, or equivalent to 332 1100 GPU days with a 5x speedup in Transformer training, as per Rombach et al. (2022); NVIDIA (2023).
7

---

Published as a conference paper at ICLR 2024
Table 3: Alignment evaluation on T21-CompBench. PIXART-a demonstrated exceptional perfor-
mance in attribute binding, object relationships, and complex compositions, indicating our method achieves superior compositional generation ability. We highlight the best value in blue and the second-best value in green The bascline data are sourced from Huang et al. (2023).
Attribute Binding
Object Relationship
Model
Complext
Color
Shapet
Texturet
Spatialt
Non-Spatial
Stable vl.4
0.3765
0.3576
0.4156
0.1246
0.3079
0.3080
Stable v2
0.5065
0.4221
0,4922
0.1342
0.3096
0.3386
Composable v2
0.4063
0.3299
0.3645
0.0800
0.2980
0.2898
Structured v2
0.4990
0.4218
0.4900
0.1386
0.3111
0.3355
Attn-Exct v2
0.6400
0,4517
0.5963
0.1455
0.3109
0.3401
GORS
0.6603
0.4785
0.6287
0.1815
0.3193
0,3328
Dalle-2
0.5267
0,4747
0,5804
0.1283
0.3078
0.2967
SDXL
0.5879
0.4687
0.5299
0.2133
0.3119
0.3237
PIXART-cr
0.6690
0.4927
0.6477
0.2064
0.3197
0.3433
User Study. While quantitative evaluation metrics measure the overall distribution of two image sets, they may not comprehensively evaluate the visual quality of the images. Consequently, conducted user study to supplement our evaluation and provide a more intuitive assessment of PIXART-a's performance. Since user study involves human evaluators and can be time-consuming, we selected the top lop-performing models, namely DALLE-2, SDv2, SDXL, and DeepFloyd, which accessible through APIs and capable of generating images.
PIXART-a
DALLE-2
SDv2
SDXL
DeepFloyd
71.2
Percentage (30
81.3
64.8
54.4
53.6
51.7 48.3
53.2
50
45.6
146.8
38.7
34.8
35.2
28.8
25
01
Quality
Alignment
Quality
Alignment
Quality
Alignment
Quality
Alignment
Figure 5: User study 300 fixed prompts from Feng et al. (2023). The ratio values indicate the percentages of participants preferring the corresponding model. PIXART-a achieves a superior performance in both quality and alignment.
For each model, we employ consistent set of 300 prompts from Feng al. (2023) to generate images. These images are then distributed among 50 individuals for evaluation. Participants are asked to rank each model based the perceptual quality of the generated images and the precision of alignments between the text prompts and the corresponding images. The results presented in Figure clearly indicate that PIXART-a excels in both higher fidelity and superior alignment. For example, compared to SDv2, a current top-tier T2I model, PIXART-a exhibits a 7.2% improvement image quality and substantial 42.4% enhancement alignment.
3.3 ABLATION STUDY
We then conduct ablation studies on the crucial modifications discussed in Section 2.3, including structure modifications and re-araameterizatoon design. In Figure 6, we provide visual results and perform FID analysis. We randomly choose 8 prompts from the SAM test set for visualization and compute the zero-shot FID-5K score on SAM dataset. Details are described below.
"w/o re-param" results are generated from the model trained from scratch without re- parameterization design. We supplemented with an additional 200K iterations to compensate for the missing iterations from the pretraining stage for fair comparison. "adaLN" results are from the model following the DiT structure to use the sum of time and text feature as input to the MLP layer for the scale and shift parameters within each block. "adal_N-single" results obtained from the model using Transformer blocks with the adaL.-singll module in Section 2.3. In both "adaLN" employ the re- parameterization design and training for 200K iterations.
and "adaL.N-single
As depicted in Figure 6, despite "adaLN" performing lower FID, its visual results are on par with our BodalNN-singee design. The GPU memory consumption of " "adal."" is 29GB, whereas "adal.N.
8

---

Published as a conference paper at ICLR 2024
umandaa 6M
FID
Memory
23G
29G
23G
(Chors)
22.30
23G
61
Figure 6: Left: Visual comparison of ablation studies are presented. Right: Zero-shot FID-2K on SAM, and GPU memory usage. Our method is on par with the "adaLN" and saves 21% GPU memory. Better zoom in 200%.
single" achieves reduction to 23GB, saving 21 % in GPU memory consumption. Furthermore, considering the model parameters, the "adaIN" method consumes 833M, whereas our approach re- duces to mere 61 IM, resulting in impressive 26% reduction. "adaLN-single-2 (Ours)" results are generated from the model with same setting as "adaLN-single", but training for Longer train- ing period of 1500K iterations. Considering memory and parameter efficiency, we incorporate the "adaLN- --iigle-L- into our final design.
The visual results clearly indicate that, although the differences in FID scores between the "adaLN" and "adaLN-single"" models are relatively small, significant discrepancy exists in their visual out- comes. The "w/o re-param" model consistently displays distorted target images and lacks crucial details across the entire test set.
RELATED WORK
We review related works in three aspects: Denoising diffusion probabilistic models (DDPM), Latent Diffusion Model, and Diffusion Transformer. More related works can be found in Appendix A.1. DDPMs (Ho et al., 2020; Sohl-Dickstein al., 2015) have emerged as highly successful approaches for image generation, which employs an iterative denoising process to transform Gaussian noise into an image. Latent Diffusion Model (Rombach et al., 2022) enhances the traditional DDPMs by em- ploying score mathhing on the image latent space and introducing cross pross-attention-oe based controlling. Witnessed the success of Transformer architecture on many computer vision tasks, Diffusion Trans- former (DiT) (Peebles & Xie, 2023) and its variant (Bao et al., 2023; Zheng et al., 2023) further replace the Convolutional-basee increased scalability (Chen et al., 2023).
U-Net (Ronneberger al., 2015) backbone with Transformers for
5 CONCLUSION
In this paper, we introduced PIXART-a, a Transformer-based text-to-image (T2I) diffusion model, which achieves superior image generation quality while significantly reducing training costs and CO2 emissions. Our three core designs, including the training strategy decomposition, efficient T2I Transformer and high-informative data, contribute the success of PIXART-a. Through extensive experiments, we have demonstrated that PIXART-a achieves near commercillaa application standards in image generation quality. With the above designs, PIXART-a provides new insights to the AIGC community and startups, enabling them to build their own nigh-quality yet low-cost T2I models. We hope that work inspires further innovation and advancements in this field.

---

Published as a conference paper at ICLR 2024
Acknowledgement. We would like to express gratitude to Shuchen Xue for identifying and correcting the FID score in the paper. This research was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (Grant 16200021).
REFERENCES
Anne-Laure Ligozat Alexandra Sasha Luccioni, Sylvain Viguier. Estimating the carbon footprint of bloom,& a 176b parameter language model. arXiv preprint arXiv:2211.0000 2022.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song. Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. In arXiv, 2022.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023.
Eyal Betzalel, Coby Penso, Aviv Navon, and Ethan Fetaya. A study the evaluation of generative models. In arXiv, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into diffusion transformers for image and video generation. arXiv preprint 2rX1v:23/2.04557,20 2023.
DeepFloyd. Deepfloyd, 2023. URL https: www. http://wwwweelll......e.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020a.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition at scale. In arXiv, 2020b.
Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced dixtureeof--deooiinnn-exxxxrs In CVPR, 2023.
Chongjian Ge, Junsong Chen, Enze Xie, Zhongdao Wang. Lanqing Hong, Huchuan Lu, Zhenguo Li, and Ping Luo. Metabev: Solving sensor failures for 3d detection and map segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8721-8731, 2023.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Farde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. NeurlPS, 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
10

---

Published as a conference paper at ICLR 2024
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPS, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12598 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbenc:: A comprehensive benchmark for open-woree world compositional -ixagee generation. In ICCV, 2023.
Minguk Kang. Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image tome synthesis. In CVPR, 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2426-2435, June 2022.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In arXiv, 2013.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In ICCV, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. In arXiv, 2023.
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Beyformer: Learning bird's s-eee-view representation from multi-camera images via spa- tiotemporal transformers. In ECCV, 2022a.
Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR, 2022b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In arXiv, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In arXiv, 2017.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Microsoft. Gpu selling, 2023. URL
com/.
Midjoumnyy. Midjourney, 2023. URL https: https::/ww..idddttt
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In arXiv, 2023.
11

---

Published as a conference paper at ICLR 2024
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
NLTK. Nitk, 2023. URL htt.tttsss//ww..nttee http///wwwwwll....../
NVIDIA. Getting immediate speedups with a100 and rf32, 2023. URL httpsttss://deeelll oper. nvidia http:///w//ooootttiiaaaaaaaaeeea
watp--wwtthtaott3333 a100-tf32.
OpenAl. Dalle-2. 2023. URL wttptt:tsp//ppeeeeeeeetet http://ppenna.commaaal1111- coa.s//lemmdaale22221
Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. arXiv, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In arXiv, 2023.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. diffusion. arXiv, 2022.
Dreamfusion: Text-to-3d using 2d
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, Improving language under- standing by generative pre-training. OpenAl blog, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In arXiv, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NewrlPS, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In arXiv, 2021.
Jascha Sohl- Sohl-Dicke Dickkteein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NewrIPS, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
12

---

Published as a conference paper at ICLR 2024
Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. In arXiv, 2020.
Hugo Touvron, Matthieu Cord, Mathijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint urXiv:2212.11565 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 34:12077-12000, 2021.
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficien fine-tuning. In ICCV, 2023.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In ICCV, 2015.
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. arXiv preprint arXiv:2309.05019, 2023a.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. In arXiv, 2023b.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. In arXiv, 2023.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequeece perspective with transformers. In CVPR, 2021.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. In arXiv, 2021.
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pp. 77378-27994. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 RELATED WORK
A.1.1 DENOISING DIFFUSION PROBABILISTIC MODELS
Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) and score-based generative mod- els (Song Ermon, 2019; Song et al., 2021) have emerged as highly successful approaches for image generation, surpassing previous generative models such GANs (Goodfellow et al., 2014), VAEs (Kingma& & Welling, 2013), and Flow (Rezende & Mohamed, 2015). Unlike traditional mod- els that directly map from Gaussian distribution to the data distribution, diffusion models employ an iterative denoising process to transform Gaussian noise into an image that follows the data dis- tribution. This process can be reversely learned from untrainable forward process, where small amount of Gaussian noise is iteratively added to the original image.
A.1.2 LATENT DIFFUSION MODEL
Latent Diffusion Model (a.aa. Stable diffusion) (Rombach et al., 2022) is a recent advancement in diffusion models. This approach enhances the traditional diffusion model by employing score- matching on the image latent space and introducing gross-atentionnbasse obtained with this approach have been impressive, particularly in tasks involving high-density image generation, such text-to-image synthesis. This has served source of inspiration for numerous subsequent works aimed at improving text-to-image synthesis, including those by Saharia et al. (2022); Balaji et al. (2022); Feng al. (2023); Xue al. (2023b); Podell et al. (2023), and others. Additionally, Stable diffusion and its variants have been effectively combined with various low-cost fine-tuning (Hu et al., 2021; Xie al., 2023) and customization (Zhang et al., 2023; Mou et al., 2023) technologies.
controlling. The results
A.1.3 DIFFUSION TRANSFORMER
Transformer architecture (Vaswani et al., 2017) have achieved great success in language mod- els (Radford et al., 2018; 2019), and many recent works (Dosovitskiy et al., 2020a; He et al., 2022) show it is also promising architecture on many computer vision tasks like image classi- fication (Touvron et al., 2021; Zhou et al., 2021; Yuan et al., 2021; Han al., 2021), object de- tection (Liu et al., 2021; Wang al., 2021; 2022; Ge al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; I; Xie et al., 2021; Strudel ct al., 2021) and so on (Sun et al., 2020;
Li al., 2022b; Zhao al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a). The Diffusion Transformer (DiT) (Peebles Xie, 2023) and its variant (Bao et al., 2023; Zheng et al., 2023) fol- low the step to further replace the Convolutional- U-Net (Ronneberger et al., 2015) backbone with Transformers. This architectural choice brings about increased scalability (Chen et al., 2023) compared to U-Net-based diffusion models, allowing for the straightforward expansion of its param- eters. In our paper, we leverage DiT as a scalable foundational model and adapt it for text-to-image generation tasks.
A.2 PIXART-Q MIDJOURNEY
In Figure 7, we present the images generated using PIXART-a and the current SOTA product-level method Midjourney (Midjourney, 2023) with randomly sampled prompts online. Here, we conceal the annotations of images belonging to which method. Readers are encouraged to make assessments based on the prompts provided. The answers will be disclosed at the end of the appendix.
A.3 PIXART-a vs. PRESTIGIOUS DIFFUSION MODELS
In Figure 8 and 9, present the comparison results using test prompt selected by RAPHAEL. The instances depicted here exhibit performance that on par with, or even surpasses, that of existing powerful generative models.
14

---

Published as a conference paper at ICLR 2024
A.4 AUTO-LABELING TECHNIQUES
To generate captions with high information density, we leverage state-of-the-art vision-language models LLaVA (Liu et al., 2023). Employing the prompt, "Describe this image and its style very detailed manner", we have significantly improved the quality of captions. We show the prompt design and process of auto- labella in Figure 10, More image-text pair samples on the SAM dataset shown in Figure 11.
A.5 ADDITIONAL IMPLEMENTATION DETAILS
We include detailed information about all of our PIXART-a models in this section. As shown in Table 4, among the 256x256 phases, our model primarily focuses on the text-to-image alignment stage, with less time on fine-tuning and only 1/8 of that time spent on ImageNet pixel dependency.
PIXART-a model details. For the embedding of input timesteps. we employ a 356-dimensional frequency embedding (Dhariwal & Nichol, 2021). This is followed by a two-layer MLP that features a dimensionality matching the transformer's hidden size, coupled with SiLU activations. We adopt the DiT-XL model, which has 28 Transformer blocks in total for better performance, and the patch size of the PatchEmbed layer in ViT (Dosovitskiy et al., 2020b) is 2x.
Multi-scale training. Inspired by Podell et al. (2023), we incorporate the multi-scale training strategy into our pipeline. Specifically, We divide the image size into 40 buckets with different aspect ratios, each with varying aspect ratios ranging from 0.25 to 4, mirroring the method used in SDXL. During optimization, training batch is composed using images from single bucket, and we alternate the bucket sizes for each training step. In practice, we only apply multi-scale training in the high-acsthetics stage after pretraining the model at a fixed aspect ratio and resolution (i.e. 256px). We adopt the positional encoding trick in DiffFit (Xie et al., 2023) since the image resolution and aspect change during different training stages.
Additional time consumption. Beside the training time discussed in Table 4, data labeling and VAE training may need additional time. We treat the pre-trained VAE as ready-made component of model zoo, the same as pre-trained CLIP/T5-XXL text encoder, and our total training process does not include the training training duration of 25 hours, utilizing 64 V100 GPUs the Openlmage dataset. As for auto- labeling, we use LLAVA-7B to generate captions. LLaVA's annotation time on the SAM dataset is approximately 24 hours with 64 V100 GPUs. To ensure fair comparison, have temporarily excluded the training time and data quantity of VAE training, T5 training time, and LLa VA auto- labeling time.
VAE. However, our attempt to train VAE resulted in an approximate
Sampling algorithm. In this study, we incorporated three sampling algorithms, namely iD- DPM (Nichol & Dhariwal, 2021), DPM-Solver (Lu et al., 2022), and SA-Solver (Xue et al., 2023a). We observe these three algorithms perform similarly in terms of semantic control, albeit with minor differences in sampling frequency and color representation. To optimize computational efficiency, we ultimately chose to employ the DPM-Solver with 20 inference steps.
Table 4: We report detailed information about every PIXART-a training stage in our paper. Note that HQ (High Quality) dataset here includes 4M JourneDB (Pan et al., 2023) and 10M internal data. The count of GPU days excludes the time for VAE feature extraction and T5 text feature extraction, we offline prepare both features in advance so that they not part of the training process and contribute no extra time to it.
Method
Stage
Image Resolution
#lmages
Training Steps (K)
Batch Size
Learning Rate
GPU (V100)
PIXART-O
Pixel dependency
256x 256
IM ImageNet
300
128x8
2x
88
PUXART-Y
Text-Image align
256x 256
10M SAM
150
178x64
2x10-6
672
PIXART-0
High aesthetics
256x:26
14M HQ
90
178x64
2x10-8
416
PIXART-
High aestbetics
512x512
14M HQ
100
40x64
2x10-9
320
PIXART-
High acstbetics
1024x 1024
14M HQ
12x32
2x10-8
15

---

Published as a conference paper at ICLR 2024
A.6 HYPER-PARAMETER PARAMETERS ANALYSIS
In Figure 20, we illustrate the variations in the model's metrics under different configurations across various datasets. we first investigate FID for the model and plot FID-vs-CLIPP curves Figure 20a for 10k text-image paed from MSCOCO. The results show marginal enhancement over SDv1.5. In Figure 20b and 20c, we demonstrate the corresponding T2ICompeench scores across a range of classifier-fiee guidance (cfg) (Ho & Salimans, 2022) scales. The outcomes reveal consistent and commendable model performance under these varying scales.
A.7 MORE IMAGES GENERATED BY PIXART-O
More visual results generated by PIXART-a are shown in Figure 12, 13, and 14. The samples gen- erated by PIXART-a demonstrate outstanding quality, marked by their exceptional fidelity and pre- cision in faithfully adhering to the given textual descriptions. As depicted in Figure 15, PIXART-a demonstrates the ability to synthesize high-resolution images up to 1024 x 1024 pixels and contains rich details, and is capable of generating images with arbitrary aspect ratios, enhancing its versatil- ity for real-world applications. Figure 16 illustrates PIX ART-O's remarkable capacity to manipulate image styles through text prompts directly, demonstrating its versatility and creativity.
A.8 DISCCUSION OF FID METRIC FOR EVALUATING IMAGE QUALITY
During our experiments, we observed that the FID (Fréchet Inception Distance) score may not ac- curately reflect the visual quality of generated images. Recent studies such SDXL (Podell al., 2023) and Pick-a-pic (Kirstain et al., 2023) have presented evidence suggesting that the COCO zero-shot FID is negatively correlated with visual aesthetics.
Furthermore, it has been stated by Betzalel et al. (Betzalel et al., 2022) that the feature extraction network used in FID is pretrained on the ImageNet dataset, which exhibits limited overlap with the current text-to-image generation data. Consequently, FID may not be an appropriate metric for evaluating the generative performance of such models, and (Betzalel al., 2022) recommended employing human evaluators for more suitable assessments.
Thus, we conducted a user study validate the effectiveness of our method.
A.9 CUSTOMIZED EXTENSION
In text-to-image generation, the ability to customize generated outputs to specific style or condition crucial application. We extend the capabilities of PIXART-a by incorporating two commonly used customization methods: DreamBooth (Ruiz et al., 2022) and ControlNet (Zhang et al., 2023).
DreamBooth. DreamBooth can be scamlessly applied to PIXART-a without further modifica- tions. The process entails fine-tuning PIXART-a using a learning rate of 5c-6 for 300 steps, without the incorporation of class-preservation loss.
As depicted in Figure 17a, given a few images and text prompts, PIXART-a demonstrates the capac- ity to generate high-fidelit images. These images present natural interactions with the environment under various lighting conditions. Additionally, PiXART-a a is also capable of precisely modifying the attribute of specific object such color, shown in 1 17b. Our appealing visual results demon- strate PIXART-- can generate images of exceptional quality and its strong capability for customized extension.
ControlNet. Following the general design of ControlNet (Zhang et al., 2023), we freeze each DiT Block and create a trainable copy. augmenting with two zero linear layers before and after it. The control signal e is obtained by applying the same VAE to the control image and shared among all blocks. For each block, we process the control signal c by first passing it through the first zero linear layer, adding it to the layer input œ. and then feeding it into the trainable copy and the second zero linear layer. The processed control signal is then added to the output y of the frozen block, which is obtained from input x. We trained the ControlNet on HED (Xie Tu, 2015) signals using learning rate of 5e-6 for 20,000 steps.
16

---

Published as a conference paper at ICLR 2024
As depicted in Figure 18, when provided with reference image and control signals, such as edge maps, we leverage various text prompts to generate a wide range of high-fidelity and diverse images. Our results demonstrate the capacity of PIXART-a to yield personalized extensions of exceptional quality.
A.10 DISCUSSION ON TRANSFORMER vs. U-NET
The Transformer-basee based network's superiority over convolutional networks has been widely estab- lished in various studies, showcasing attributes such as robustness (Zhou et al., 2022; Xie et al., 2021), effective modality fusion (Girdhar et al., 2023), and scalability (Peebles Xie, 2023). Sim- ilarly, the findings multi mudti-modality fusion are consistent with our observations in this study com- pared the CNN-based generator (U-Net), For instance, Table 3 illustrates that our model, PIXART- a, significantly outperforms prevalent U-Net generators in terms of compositionality. This advan- tage is not solely due to the high-quality alignment achieved in the second training stage but also to the multi- head attention- based fusion mechanism, which excels modeling long dependencies. This mechanism effectively integrates compositional semantic information, guiding the generation of vision latent vectors more efficiently and producing images that closely align with the input texts. These findings underscore the unique advantages of Transformer architectures in effectively fusing multi-modal information.
A.11 LIMITATIONS & FAILURE CASES
In Figure 19, highlight the model's failure cases in red text and yellow circle. Our analysis reveals the model's weaknesses in accurately controlling the number of targets and handling specific details, such features of human hands. Additionally, the model's text generation capability is somewhat weak due to our data's limited number font and letter-related images. We aim to explore these unresolved issues in the generation field, enhancing the model's abilities in text generation, detail control, and quantity control in the future.
A.12 UNVEIL THE ANSWER
In Figure 7, we present a comparison between PIXART-a and Midjourney and conceal the corre- spondence between images and their respective methods, inviting the readers to guess. Finally, in Figure 21, we unveil the answer to this question. It is difficult to distinguish between PIXART-a and Midjoumny, which demonstrates PIXART-a's exceptional performance.
17

---

Published as a conference paper at ICLR 2024
small cactus with a happy face the Sahara desert
Art collection style and fashion shoot, in the style of made of glass, dark blue and light pink, paul rand, solarpunk, camille vivier, beth didonato hair, barbiecore, hyper-realistic.
The image featuros a woman wearing rod shirt with an icon. She appears to be posing for the camera, and her outfit includes pair of jeans. The woman seems to be in good mood, as she is smiling. The background the image is blurry, focusing more on the w woman and her attire.
Pirate ship trapped in a cosmic maelstrom nebula, rendered in cosmic beach whirlpool engine, volumetric lighting, spectacular, ambient lights, light pollution, cinematic atmosphere, art nouveau style, illustration art artwork by SenseiJaye, intricate detail.
poster of a mechanical cat, technical Schematics viewed from front and side view on light white blueprint paper, illustration drafting style, illustration, typography, conceptual art, dark fantasy steampunk. cinematic, dark fantasy.
PIGAN
A dog that has been meditating all the time
Beautiful scene
Figure 7: Comparisons with Midjourney. The prompts used here randomly sampled online.
To ensure a fair comparison, we select the first result generated by both models. We encourage readers to guess which image corresponds to Midjourney and which corresponds to PIXART-O. The answer is revealed at the end of the paper.
18

---

Published as a conference paper at ICLR 2024
RAPHAEL
Stable Diffusion
DeepFloyd
DALL-E
ERNIE-ViLG 2.0
PixArt-a
1. A parrot with pearl earring, Vermeer style.
2. car playing soccer, digital
3. A Pikachu with angry expression and red eyes, with lightning around it, hyper realistic style.
4. Moonlight Maiden, cute girl in school uniform, long white hair, standing under the moon, celluloid style, Japanese manga style.
Street shot a fashionable Chinese lady Shanghai, wearing black high-waisted trousers.
6. Half human, half robot, repaired human, human flesh warrior, mech display, man in mech, cyberpunk.
Figure 8: Comparisons of PIXART-a with recent representative generators, Stable Diffusion XL. DeepFloyd, DALL-E 2, ERNIE-ViLG 2.0, and RAPHAEL. They are given the same prompts as in RAPHAEL(Xue et al., 2023b), where the words that the human artists yearn to preserve within the generated images are highlighted in red. The specific prompts for each are provided at the bottom of the figure. Better zoom in 200%.
19

---

Published as a conference paper at ICLR 2024
DALL-E
ERNIE ViLG 2.0
DeepFloyd
Stable Diffusion XL
RAPHAEL
Midjourney V5.1
PixArt-a
cute little matte low poly isometric cherry blossom forest island, waterfalls, lighting, soft shadows, trending on Artstation, 3d render, monument valley, fez video game.
2. A shanty version of Tokyo, new rustic style, bold colors with all colors palette, video game, genshin, tribe, fantasy, overwatch.
3. Cartoon characters, mini characters, figures, illustrations, flower fairy, green dress, brown hair, curly long hair, elf-like wings, many flowers and leaves, natural scenery, golden eyes, detailed light and shadow, high degree of detail.
4. Cartoon characters, mini characters, hand-made, illustrations, robot kids, color expressions, boy, short brown hair, curly hair, blue eyes, technological age, cyberpunk, big eyes, cute, mini, detailed light and shadow, high detail.
Figure 9; The prompts (Xue al.. 2023b) for each column given in the figure. We give the com- parisons between DALL-E Midjourncy v5.1, Stable Diffusion XL, ERNIE ViLG 2.0, DeepFloyd, and RAPHAEL. They given the same prompts, where the words that the human artists yearn to preserve within the generated images are highlighted red. Better zoom 200%.
20

---

Published as a conference paper at ICLR 2024
FVH
Received
Received
the caption of image 'What science says about pu's e'erh tea? describe this image and its style very detailed manner.
Describe this image and its style in very detailed manner.
The image features a large, open dining area with a high ceiling and clock hanging from the wall. The room is filled with tables and chairs, creating inviting atmosphere. The clock
The image features close-up of cup of tea with saucer wooden table. The tea is described "pu'erh which type Chinese known for health benefits. The scene set in dimly room. The presence potted plant in background adds a touch of nature and freshness to the scene.
of large windows hanging from the wall adds the overall aesthetic, creating visually appealing and functional space for dining and socializing.
Figure 10: We present auto-labeling with custom prompts for LAION (left) and SAM (right), The words highlighted in green represent the original caption in LAION, while those marked in red indicate the detailed captions labeled by LLaVA.
Samples
The image features larpe statue dragon, which mythical creature with long body, wings, and tail. The drogon statue situated in a park, sumounded by lush green field flower garden. The dragon statue is positioned in front of temple, which adds overall aesthetic cultural significance of the
image features large, open dining with high ceiling and a clock hanging from the wall. The room is filled with tables and chairs, creating inviting atmospbere, The of large windows and a clock banging from the wall adds to the overall aesthetic, creeting visually appealing functional space dining and socializing.
The image features man wouring uniform, likely police officer, standing city street. He is walking down the sidewalk. survunded by various shops and people. The socne set busy urban environment, with mix of pedestrians, shops
SAM + LLaVA
Figure 11: Examples from the SAM dataset using LLaVA -uodduuudd labels. The detailed image descriptions in LLaVA captions aid the model grasp concepts per iteration and boost text-image alignment efficiency.
21

---

Published as a conference paper at ICLR 2024
Assercan
life in disiree, lest hope, food
wine hd Bi, mai meilty taper deta il poet photo
manpulition real photo
female painter with brush in hand, white backgroound, painting. looking very powerful
baby trynn
white
Cheose
mountei
happy
minted feed.
An Roate through roading neaopaper
Figure 12: The samples generated by PIXART-a demonstrate outstanding quality, marked by an exceptional level of fidelity and precision in aligning with the given textual descriptions. Better zoom 200%.
22

---

Published as a conference paper at ICLR 2024
viowed grodd nverooking
bold abyle travel
vibrantt yellow banane twnanp-rhaped couch dits in cory. bring room is curve craling pile of cal orlu nuehtone the wocdes Roor pattereed rig adde touch cham pertoo nusignt
image lemurwearing magician and blue performing magic tricks with cards garden.
window.
homsn sumcal UHD, typer debilx rith oniors, photognpe
painter study hard learn haw draw with many connagre dn the
alsaca made coloria buildig bleck. сytеrpun:
beil sca'd
cowns a litle gin the style dwaay mebiec
mar looks the stary sky lonely Eatryy lita vite of nutleatting in forest
A dog reading thick book
A Jelyduh rding rocket
Figure 13: The samples generated by PIXART-a demonstrate outstanding quality, marked by an exceptional level of fidelity and precision in aligning with the given textual descriptions. Better zoom in 200%.
23

---

Published as a conference paper at ICLR 2024
Lufty from ONEPIECE, handsome face, fantasy.
An astronaut riding horse.
Crocodile sweater.
A
Design letter A, 3D stereoscopic material Interior light blue Concoptual product design Futuristic Blind Handcrafted Exquistte effect body display Ultra-hihh precision.
The girl the car is filled with goldfish and flowers. goldfish natural posture, youthful energy and pressure, body stretching, goldfish simulation movies in the sky.
cute fluffy sentient allen from planet Avor, in the andromeda galaxy, the allen
eyes and digitigrade, high
An ancient stone Colossus with eye, Stephan Martiniére, dark yellow emerald, color zone painting, Sarazhin, silvr, robotic expressionism, high detail
A painting depicting red wave outside, trapped emotions depicted, body, depth, Dima Dmitriev, fisheye effects. Ray Collins.
Bright scone, acrial view, andient city. fantasy, gorgeous light, reflection, high detail wide angle lens.
person standing on the desert, desert waves. half red, half blue, sand, ilustration, outdoor.
A lon with dragon's head.
Futurist painting the building.
Figure 14: The samples generated by PIXART-a demonstrate outstanding quality, marked by an exceptional level of fidelity and precision in aligning with the given textual descriptions. Better zoom 200%.
24

---

Published as a conference paper at ICLR 2024
High detail 1024px
Figure 15: PIXART-a is capable of generating images with resolutions of up to 1024 x 1024 while preserving rich, complex details. Additionally, it generate images with arbitrary aspect ratios, providing flexibility in image generation.
25

---

Published as a conference paper at ICLR 2024
"Claudo Monett pointiin of
"Van Gogh painting of
"Photography of
"ixxel art of
"penoil dr awiing of
the black hole the space
a teacup on the desk
table top with vasc of flowers
birthday cake
boautiful flower
Figure 16: Prompt mixing: PIXART-a can directly manipulate the image style with text prompts. In this figure, we generate five outputs using the styles control the objects For instance, the second picture of the first sample, located at the left corner of the figure, uses the prompt Pixel Art of the black hole in the space Better zoom in 200%.
26

---

Published as a conference paper at ICLR 2024
Text prompt: [V] dog is running
Text prompt: [V] dog a doghouse
Input Images
Text prompt: A photo of dog
Text prompt: [V] dog bucket
Text prompt: [V] dog is swimming
(a) Dreambooth + PIXART-a capable of customized image generation aligned with text prompts.
Text prompt: [green] [V] car in garage
Text prompt: [white] [V] car over water
Input Images: M5
Text prompt: A photo of [grey] [V]
Text prompt: [yellow] [V] car in street
Text prompt: [black] [V] car on highway
(b) Dreambooth PIXART-a is capable of color modification of specific object such as Wenjie M5.
Figure 17: PIXART-a can be combined with Dreambooth. Given a few images and text prompts, PIXART-a can generate high-fidelity images, that exhibit natural interactions with the environ- ment 17a, precise modification of the object colors 17b, demonstrating that PIXART-a can generate images with exceptional quality, and has strong capability in customized extension.
27

---

Published as a conference paper at ICLR 2024
Reference Image
HED Edge
Flower-field
Snow
Flower
Islands
Shells
Biscuits
Reference Image
Edge
Renaissance
Gogh
Oil Paint
Star
Sketch
Cyberpunk
Figure 18: ControlNet customization samples from PIXART-a. We use the reference images generate the corresponding HED edge images and use them as the control signal for PIXART-o ControlNet. Better zoom 200%.
28

---

Published as a conference paper at ICLR 2024
A stack of books. green book on the top, sitting red book. The red book in the middlee
An expressive oil painting of a basketball dogs sitting gress dunking, depicted as an explosion nebula
Figure 19: Instances where PIXART-a encounters challenges include situations that necessitate precise counting accurate representation of human limbs. In these cases, the model may face difficulties in providing accurate results.
22
0.76
Stape
UniDet CLIP 3-in-1
0.5
Crepkc
20
0.72
B-VQA
0.68
0.4
0.64
0.2
0.24
0.25
0.26
CLIP score
cfg scale
cfg scale
(a) FID MSCOCO
(b) T2i-CompBench
(c) T2i- Be-hComBBench
Figure 20: (a) Plotting FID vs. CLIP score for different cfg scales sampled from [1.5, 2.0, 3.0, 4.0, 5.0, 6.0]. PIXART-a shows slight better performance than SDv1.5 MSCOCO. (b) and (c) demonstrate the ability of PIXART-a to maintain robustness across various cfg scales on the T21- CompBench.
29

---

Published as a conference paper at ICLR 2024
My
PiAAtLa
Mj
PixArt-a
small cactus with happy face in the Sahara desert
MJ
PixArt-a
Art collection style and fashion shoot, in the style made of glass, dark blue and light pink, paul rand, solarpunk, camille vivier, beth didonato hair, barbiecore, hyper-realistic
MI
PiAat-i
The image featuros woman wearing rod shirt with an icon. She appears be posing for the camera, her outfit includes a pair of jeans. The woman soams to be in good mood, as she is smiling. The background of the image blurry, focusing more the woman and her attire.
MJ
PixArt-a
Pirate ship trapped cosmic maelstrom nebula, rendered in cosmic beach whirlpool engine, volumetric lighting, spectacular, ambient lights, light pollution, cinematic atmosphere, art nouveau style, illustration art artwork by SenseiJaye, intricate detail.
TOE
poster of mechanical cat, technical Schematics viewed front side view on light white blueprint paper, illustration drafting style, illustration, typography, conceptual art, dark fantasy steampunk, cinematic, dark fantasy.
PrcAn
PivArt-u
PixArte
dog that has been meditating all the time
Beautiful scene
Figure 21: This figure presents the answers to the image generation quality assessment as depicted in Appendix A.2. The method utilized for each pair of images is annotated the top-left comer.
30


Reviews:
[{"review_id": "UT55rKdFW2", "reviewer": "Reviewer_VQLf", "summary": "The paper introduces several recipes to accelerate the training of text-to-image foundation models. These include\n- Use pretrained DiT\n- Use Flan-T5 XXL\n- Use LLaVA captions.\n- AdaLN / AdaLN single architectures to reduce model size.\nOverall, these methods allow training of a reasonable quality model in 10% of the resources used than Stable Diffusion, which makes it more possible to democratize the training recipes in text-to-image foundation models.", "questions": "1. The SAM dataset blurs human faces in their training, won't this cause problem in generation cases where generating a face (not closed up) is needed?\n2. How does the model generate images with more extreme aspect ratios?\n3. The DiT architecture has a fixed patch size. As resolution becomes higher, so will the number of tokens be higher. Will this cause a bottleneck in training and inference (such as 1k resolution)?\n4. DALL-E 3 technical report mentions the pitfall of \"overfitting\" to automated captions, is this the case in PixArt model? If not, how is this mitigated?\n5. Since the dataset size is smaller, does it have trouble producing named entities, such as celebrities? \n6. How critical is training DiT on ImageNet needed? While being able to start with an existing model is good it also limits the possibilities to explore different architectures. \n7. The CLIP score of the CLIP-FID curve of Pixart seems worse than SD 1.5. Is there any reason for that?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality: the empirical evaluation of synthetic captions in text to image generation is not systematically studied until DALL-E 3, and the new AdaLN architecture might be useful.\nClarify: the paper is quite clear about most details about the training, which makes reproducibly much more likely.\nSignificance: the paper mostly explores valid heuristics for training text-to-image foundation models quickly, some conclusions can be helpful in the community: 1) DiT architecture instead of UNet, 2) the use of synthetic captions, 3) the use of SAM dataset.", "weaknesses": "The paper mostly is a combination of multiple ideas that exist in the literature, so \"novelty\" in the traditional sense is somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TWcgNokh9n", "reviewer": "Reviewer_GFA3", "summary": "PIXART-α is a novel Text-to-Image (T2I) model capable of generating photorealistic images from text descriptions while maintaining low training costs. The model's approach involves decomposing the T2I task into three distinct stages, which include pixel dependency learning, text-image alignment learning, and high-resolution and aesthetic image generation. This is achieved through the modification of the diffusion transformer architecture, incorporating cross-attention layers, simplifying adaptive normalization layers, and utilizing re-parameterization techniques for an efficient T2I Transformer. Additionally, the model leverages high-informative data from a vision-language model (LLaVA) for generating quality image captions, drawing from the SAM dataset and fine-tuning using JourneyDB and an internal dataset.", "questions": "What led to the decision of utilizing ImageNet data as the pretraining source instead of a pretrained VQGAN, and how did this choice impact the model's performance and capabilities?\n\nThe selection of the Diffusion Transformer as the basis for the model's architecture over the more conventional approach of the Latent Diffusion Model (Unet) is mentioned in Appendix A10, but can a more comprehensive analysis, including ablation studies, be provided to thoroughly compare the strengths and weaknesses of these architectural choices?\n\nThe model's performance raises the important question of determining the minimum number of images required for training while still ensuring generalizability and maintaining the compositional properties of Text-to-Image models. Can the authors shed some light on what is the minimum set of images required to train such a model from scratch. What should be the properties of such a dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The PIXART-α model is particularly cost-effective to train, making it an attractive option for researchers and organizations with limited computational resources, as it minimizes the financial and hardware requirements associated with training a state-of-the-art T2I model.\n\nIt employs a straightforward and simplified approach to achieve its text-to-image synthesis, ensuring that the model's architecture and training process are accessible and comprehensible to a broader range of researchers and practitioners.\n\nNotably, PIXART-α boasts fewer tunable parameters, which not only contributes to its cost-effectiveness but also makes it more manageable and less prone to overfitting or complex hyperparameter tuning, streamlining the implementation and optimization process.", "weaknesses": "Given its smaller dataset, there are concerns about the generalizability and compositional capabilities of the model for a broader range of concepts.\n\nA more extensive test for out-of-domain generalizability would add valuable insights into the model's adaptability and effectiveness in diverse scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UACSwsOaZp", "reviewer": "Reviewer_Bg3n", "summary": "The paper proposes a new text-to-image synthesis model, called PixArt-$\\alpha$. The proposed approach introduces several improvements to conventional models on the side of the training process and architecture. In particular, the paper proposes a training strategy decomposition, in which 1) the learning of dependencies between pixels; 2) text-image alignment; 3) high-resolution synthesis, are all modelled in different subsequent phases. On the architecture side, the paper builds on top of Diffusion Transformer pre-trained on class-conditional ImageNet, proposing several tricks to adopt it to text-to-image synthesis with minimal overhead. In addition, the authors re-visit the training datasets used to train text-to-image models. By applying LLaVA to label existing datasets, much more dense labelling of images in terms of nouns/image is achieved.\n\nIn effect, the proposed approach achieves state-of-the-art quality of synthesis but significantly reduces the training time and the needed amount of training data. The method is shown to be on par or better than models like Imagen, SDXL, while using only 10% of the training time and 10% of the training samples.", "questions": "What are the plans of the authors regarding open-sourcing? \nIs it expected to be released 1) complete training code; 2) collected dataset; 3) \"internal\" dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, this is a very actual paper with very strong experimental result, big significance and potential impact.\n\n- $\\underline{\\text{Contribution}}$. The proposed method achieves state-of-the-art results with resources that are an order of magnitude lower than of current mainstream methods. Given that biggest T2I are very expensive to train, this is a big step forward for the community to democratize T2I models for broader audiences, to decrease costs and the carbon footprint required for training. \n- $\\underline{\\text{Insights}}$. The paper introduces valuable insights for the community, particularly on the side of dataset design. The paper demonstrates that the design of previous datasets like LAION suffers from deficient descriptions and infrequent diverse vocabulary usage, which led previous models for many additional epochs needed until convergence. The provided VN/DN analysis can influence the next generation of image-text datasets.\n- $\\underline{\\text{Results}}$. The paper demonstrates strong experimental results. In particular, visual results are impressive, beautiful, and have good alignment to conditioning text. The model is shown to outperform big models like DALLE-2 or SDXL by human perception. Overall, the quality of results clearly matches the bar or ICLR.\n- $\\underline{\\text{Applications}}$. It is demonstrated that the proposed method supports applications that are generally expected from big T2I models, like personalization (DreamBooth, ControlNet etc).\n- $\\underline{\\text{Presentation}}$. The paper is very well written, the ideas are very easy to follow. Explanations are high-level and clearly deliver lessons for the community. More technical discussions are presented in supplementary.", "weaknesses": "I do not see major issues that would preclude publication, but I would still ask a couple of questions:\n\n- How expensive was the dataset collection? I expect running a big image-text model like LLaVA on millions of images required a lot of time, GPU ressources, and memory. Should this non-trivial step be in some way included to the analysis of costs and co2 emissions?\n- Although using a model like LLava clearly improves VN/DN, it probably introduces new biases to the distribution of text prompts? Does this affect the scope of text prompts for which PixArt-$\\alpha$ works well or poorly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UgS2GjIVHo", "reviewer": "Reviewer_7QAx", "summary": "The authors present a text-to-image model that achieves good quality results while using much smaller networks and datasets than current SOTA models. The key to the improvement seems to be some combination of (1) transformer-based diffusion (2) multi-stage training (5 stages for a single model) (3) careful dataset curation.", "questions": "- My intuition says that training cost and CO2 emissions should be directly proportional to GPU hours, however these proportions are inconsistent. For example, Fig 2b shows a 14x increase in cost of Imagen over PixArt, while the ratio of reported GPU hours is 10. For GigaGAN, Fig2b shows a 9.4x increase, where as Table 2 shows a 7x increase in GPU hours.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Being able to train a high-quality generative model with an ~order of magnitude reduction in resources is a very important contribution, as the cost of training large models limits access to large, well-resourced organizations.\n\n- The results are good, and the evaluations are sufficient. The FID score is a bit high compared to newer models, but I agree with the authors that this is (at best) an imprecise metric for quality.\n\n- The paper is overall well written and easy to read.", "weaknesses": "I have found some inconsistencies in reported results:\n\n- The GPU days reported in A.5 contradict the main table; Summing the last column and using their 2.2 scale factor for converting from v100 to a100 days yields 753 A100 days vs. the 675 reported in the main table. It's a relatively small difference (11%) but concerning that the reported numbers don't align.\n\n- Some of the baseline numbers are inconsistent / misleading. For example, the authors report that Imagen consumed 15B images. The Imagen authors report a dataset size of 800M. It seems that 15B, at least roughly, corresponds to the total number of non-unique images seen during training, e.g. (batch size) x (training steps). If we applied this metric to PixArt-alpha (using the table in A.5), we would get a number of 31B - 2x the Imagen results. I did not dig into the other baseline numbers, but these should be checked carefully because they are central to the paper's contributions.\n\n- There are 3 primary improvements (dataset curation, multi-stage training, and unet->transformers), however there are no ablations quantifying how much of an effect each one has on model performance. While the results and final model are good, a lack of ablations limits the research contribution of the paper. \n\n- The authors do not discuss the training of the VAE; since this is an important part of \"learning the pixel distribution of natural images\", the training time and data quantity of the VAE should be included in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)