Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
Authors: Bingqi Ma
Zhuofan Zong
Guanglu Song
Hongsheng Li
Yu Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model; text-to-image generation; diffusion model
Primary area: diffusion_based_models
TL;DR: A novel paradigm to leverage decoder-only LLM as text encoder in powerful diffusion models.

Abstract: Large language models based on decoder-only transformers have demonstrated superior text understanding capabilities compared to CLIP and T5-series models.
However, the paradigm for utilizing current advanced LLMs in text-to-image diffusion models remains to be explored.
We observed an unusual phenomenon: directly using a large language model as the prompt encoder significantly degrades the prompt-following ability in image generation.
We identified two main obstacles behind this issue.
One is the misalignment between the next token prediction training in LLM and the requirement for discriminative prompt features in diffusion models.
The other is the intrinsic positional bias introduced by the decoder-only architecture.
To deal with this issue, we propose a novel framework to fully harness the capabilities of LLMs.
Through the carefully designed usage guidance, we effectively enhance the text representation capability of the LLM for prompt encoding and eliminate its inherent positional bias.
This allows us to flexibly integrate state-of-the-art LLMs into the text-to-image generation model.
Furthermore, we also provide an effective manner to fuse multiple LLMs into our framework.
Considering the excellent performance and scaling capabilities demonstrated by the transformer architecture, we further design an LLM-Infused Diffusion Transformer (LI-DIT)based on the framework.
We conduct extensive experiments to validate LI-DIT across model size and data size.
Benefiting from the inherent ability of the LLMs and our innovative designs, the prompt understanding performance of  LI-DIT easily surpasses state-of-the-art open-source models as well as mainstream closed-source commercial models including Stable Diffusion 3, DALL-E 3, and Midjourney V6.

Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
Bingqi Ma 1.
Zhuofan Zong 2
Guanglu Song'
Hongsheng LF2334
Yu Liu
SenseTime Research
CUHK MMLab
Shanghai AI Laboratory
4 CPI under InnoHK
(mabingqi,
bongguanglu/eseesettino
(zongzhuofan, Liuyuisamail@@gail.co
hsliGee. Ssli@ee.cuhk... edu. hk
DIT
Figure High-resolution (1024px) samples from our LI-DIT-1OB, showcasing capabilities in complex prompt comprehension, precise prompt following, and high image quality across various styles and resolutions. Please refer to the appendix for the prompts.
Abstract
Equal contrihution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Large language models (LLMs) based on decoder-only transformers have demon- strated superior text understanding capabilities compared to CLIP and T5-series models. However, the paradigm for utilizing current advanced LLMs in text- to-image diffussin models remains to be explored. We observed an unusual phe- nomenon: directly using large language model as the prompt encoder significantly degrades the prompt-following ability in image generation. We identified two main obstacles behind this issue. One is the misalignment between the next token pre- diction training in LLM and the requirement for discriminative prompt features in diffusion models. The other is the intrinsic positional bias introduced by the decoder-only architecture. To deal with this issue, we propose novel framework to fully harness the capabilities of LLMs. Through the carefully designed usage guidance, we effectively enhance the text representation capability for prompt encoding and eliminate its inherent positional bias. This allows to integrate state- of-the-art art L.LMs into the text-to-image generation model flexibly. Furthermore, we also provide an effective manner to fuse multiple LLMs into framework. Considering the excellent performance and scaling capabilities demonstrated by the transformer architecture, we further design an LLM-Infused Diffusion Trans- former (LI-DiT) based on the framework. We conduct extensive experiments to validate LI-DiT across model size and data size. Benefiting from the inherent ability of the LLMs and our innovative designs, the prompt understanding per- formance of LI-DiT easily surpasses state-of-the-an open-source models as well as mainstream closed-source commercial models including Stable Diffusion 3, DALL-E 3, and Midjourney V6. The Infuued Diffuser framework also one of the core technologies powering SenseMirage, a highly advanced text-to-image model.
1 Introduction
The diffusion probabilistic models 2, 3, 5] have led to significant improvement in image synthesis. With the assistance of powerful prompt encoders such as the CLIP text encoder [6] and T5 series [7], DALL-E 3 [8] and Stable Diffusion 3 [9] greatly enhance the prompt understanding ability in eex-tto-image diffusion models. Encouraged by the success of GPT [10], a series of
decoder-only large language models (LLMs) emerged and demonstrated superior text understanding capabilities compared to CLIP and T5 series models, e.g., LLaMA [11, 12]. However, methods for
effectively leveraging these powerful LLMs in diffusion models remain to be explored [13, 14].
To better understand the inherent properties of LLMs in diffusion models, we first conduct ex- periments with the transformer-based diffusion model (DiT) [15] and perform evaluations on the T21-CompBench [16] benchmark. Following the design in DiT and PixArt-a[17] the text conditional information from the last layer of LLMs is injected into the diffusion transformer by cross-attention layers. As shown in Fig. 2, although LLaMA3-SB cxhibits much stronger language understanding ability [18], it still fails to catch up to the performance of the smaller model T5-XL on the image-to- text alignment benchmark. Meanwhile, the larger variant T5-XXL achieves significant advantage over XLL The powerful capabilities of LLMs in text comprehension and logical reasoning have not been demonstrated in such a scenario. Based on this anomaly, we aim to explore the role of LLMs in prompt encoding.
We start with analyzing the difference in optimization target and model architecture between T5- like encoder-decoded models and GPT-like decoder-only models. The masked language modeling optimization and the encoder-decoder architecture design endow the T5 encoder with an inherent ability for effective information comprehension. However, the optimization target of decoder-only LLMs focuses on predicting the next token with the highest probability based on training data distribution. As presented in Fig. 4, the pre-trained LLM provides a meaningless continuation to the given image prompt. It means that the LLM does not focus on the essential elements in the given image caption and the extracted text representation of LLM is not suitable for summarizing the semantic information of the given image, leading to a misalignment with the diffusion model's demand. Meanwhile, we find that LLMs generally cause errors or omissions in comprehending
'naps://gituub..om/mtte laaaaeelaaaaammmmaaaaa
2

---

infised
15.3XL
LLaMA2 78 22--nnaaed
T1XXL
Score
CompBence
LLMMA2-17 7B
25-
LLaMA-78
196-109
Relatvve position
Figure 2: Comparisons our model, LLaMA series, and T5 series image generation and text understanding benchmarks.
Figure 3: Performance discrepancy between former and latter adj-noun compositions in LLaMA3-8B and T5-XXI..
objects or attributes mentioned in the latter part of the prompt. This observation is further validated through quantitative evaluation. We attribute this issue to the causal attention mechanism of decoder-only LLMs. In the casual attention layer, each token only attend to itself and other former tokens, while the information of the latter tokens cannot be captured. Such structured information imbalance challenges the diffusion model's ability to comprehend complex prompts. Therefore, the misalignment and positional bias significantly impede LLMs from being effective text encoders for diffusion models.
To address these issues, we propose novel framework, LLM-infusel Diffuser, to fully leverage powerful LLMs promoting diffusion models in text comprehension and following. First, we explicitly insert an instruction before prompt to mitigate information misalignment. Based on the instruction- following ability of LLMs, we leverage human instruction to encourage language models focusing on concepts related to image generation, including objects, attributes, and spatial relations. Furthermore, propose linguistic token refiner to resolve the positional bias issue. Such designs facilitate effective global representation modeling via a bi-directional attention mechanism. Finally, the collaborative refiner merges and refines text representations from multiple LLMs to further boost text comprehension ability. These targeted designs provide an effective way to leverage the capabilities of LLMs in diffusion models.
Our LL.M-infused Diffuser can easily and flexibly incorporated into diffusion models. Considering the excellent performance and scaling capabilities of the transformer architecture [15, 9], we further design an I.LM-infused Diffusion Transformer (LI-DDTT). We conduct extensive experiments to validate LI-DiT across distinct model sizes and data sizes. Benefiting from the inherent ability of the LLMs and our innovative designs, the prompt understanding performance of LI-DiT easily surpasses state-of-the-art open-source models as well mainstream closed-source commercial models including Stable Diffusion 3, DALL-E 3, and Midjourney V6. Fig. 1, We present some randomly sampled cases generated by LI-DiT-10B.
Prompt Encoding with Language Models
As outlined in Sec. 1, we observe two discrepancies between decoder-only LLMss and encoder-decoder models: optimization objective and model architecture. Specifically, the decoder-only LLMs typically optimized using the next token prediction task while the encoder-decoder models are trained with the masked language modeling task. Besides, the former tokens in sequence cannot attend the latter tokens in decoder-only LLMs while every token in the sequence can attend each other in the encoder models. Based the observations, we conduct elaborate experiments to investigate how such discrepancies affect the prompt encoding capacity of LLMs.
2.1 Exploring the Ability to Retain Prompt Information
During the pre-training of T5 models, the input sequences are formatted with masks, and the model learns from vast amounts of language data by predicting the masked content. In this process, the encoder is responsible for extracting information from all tokens in the current token sequence.

---

Input instruction
backpack orange
Output of IS-XXL
Spod-a bluc backpack and red orange and blue backpock and rol ocange and </pp
Output of LLaMA3-8B
jacket. wearing pants,
gny
Input w/ human instruction
Ssyalem, jeompee
Describe the detailing the color,
quaneity text, and spataal relationehips the objects:
blue backgurkk
orange
Output LLaMA3-8B ---mmmmmmmdmaall
The image festures two divinut clemenos bright bue bockpack and vibrant red-orange object. Overall, the focts
Color-wist In forms shppes... Their strck... terme lestunox As unangorsd oljust
sirking
bluc
Figure 4: The output of language models when feeding a prompt. We can observe that pre-trained LLaMA3-8B provides an unrelated expansion, and T5- T5-XXL repeats the input prompt. LLaMA3-8B with multi-modal fine-tuning can provide detailed information based on human instruction.
However, decoder-only language models focus representing the current text representation, which is misaligned with the diffusion model's s usage. To better understand the characteristics of how language models encode prompts, feed an image prompt into both LLaMA3-8B and T5-XXL to analyze their outputs. As shown in Fig. 4, the output of T5-XXL is the repeat of the input prompt while LLaMA3-SB generates unrelated expansion. This phenomenon further validates our hypothesis. Therefore, even though LLMs possess stronger text understanding and reasoning capabilities, such limitation harms their capacity for encoding prompts.
predicting future information rather than
2.2 Positional Bias of Decoder-only LLMs
We construct benchmark to evaluate the image-text alignment of all adj-noun compositions at different positions in image prompt. Following conventional text-to-image generation bench- marks [16, 13], we extract all adj-noun compositions and obtain their relative positions in each image prompt. These adj-noun compositions generated image and the question to a VQA model to obtain its alignment score. Please refer to the supplemental material for more details about constructing the test set. As shown in Fig. compute the average alignment score and the relative position within a prompt for each adj-noun composition. We can observe diffusion models with T5 encoders exhibit strong robustness the position change, while models with decoder-only LLMs perform poorly in latter positions. Such inherent positional bias significantly harms the prompt encoding capacity of decoder-only LLMs.
be easily converted to questions. Then, input the
LLM-infused Diffuser
3.1
Integrating LLMs and Diffusion Models
To bridge the gap between pre-training optimization and prompt encoding, we leverage the instruction- following capacity LLM to encourage focus on image contents in the given caption. Fur- thermore, we also propose the refiner modules to mitigate the inherent positional bias of L.L. text embeddings. By combining these designs, we develop a framework called LLM-infused Diffuser, which can flexibly infuse current state-of-the-an LLMs to unleash its strong text understanding capacity. As shown in Fig. 5, the pipeline of LLM-Infused Diffuser consists four parts: (1) We insert the system prompt and instruction before the image prompt to encourage the LLM focus the image contents and highlight its attributes. (2) The image prompt with instructions can be encoded by multiple frozen LLMSs separately. (3) Different linguistic token refiner modules adopted to eliminate the positional bias of text embeddings from these LLMs. (4) With collaborative refiner, text features from LLMs are collaboratively refined, resulting in more robust representations.
Input Prompt. Inspired by powerful cnaructiio-followeer capabilities of L.LMs [19], we aim to leverage such capabilities to force the LLM to attend to the crucial image contents in the prompt

---

LLM-infused Diffuser
Rativer
gankiiy, leat and werten
Model
Tolee Betlner
roos the sa, aiter raper
Large Language Model
Retiner
SCB0 20P7
tee atr.
Prorrpr With Inttuuttinn
LI-DIT
Timesicg
rowditriing
LI-DIT
VIN Encrddrr
Scale
Soaio, Shitt
Scate
Soaio, Shit
T
Scali
Trals
Bile
Safta
NX. DIT Bizch
Uigoobik Tolen Retier
Figure 5: The pipeline of LLM-infused diffuser. First, the LLM-infused diffuser inserts instruc- tion to encourage LLMs to focus on image-related concepts. The linguistic token refiner eliminates the positional bias of LLM representations. Then the collaborative refiner further refines and mixes these embeddings and provides a more robust text representation. We only show LLMs for simplic- ity.
and facilitate the alignment between the text representation and the text-to-image synthesis task. Specifically, we propose to insert the custom instruction before the conventional image description. Such instruction prompts the LLM to focus on critical image contents, such as object attributes and spatial relationships among objects in the image. In our experiments, we adopt simple instruction: Describe the image by detailing the color, shape, size, lexiure, quantity, texl, and spatial relationships of the objects. As shown in Fig. 4, the LLM tends to generate contents that are related to the image context if we do not provide explicit instruction. When feeding the instruction and an image prompt the LLM, it will follow the instruction focus on the image-relevant concepts to detailedly describe the image and provide aligned representations based on the given prompt. The output embeddings of LLMS are further processed by subsequent refiner modules.
Linguistic Token Refiner. In the causal attention layer of LLM, only the previous tokens can be attended by the current token, thus it significantly hurts the global text representation modeling. For example, the last token in the text token sequence can only be attended by itself. To mitigate such positional bias of decoder-only LLMs, biased output representations of each LLM. As shown in Fig. 5, each refiner module contains stack of transformer blocks, which consists of self-ateention layer, a feed-forward layer (FFN), and an adaptive gating module. For the self-attention layer, we directly discard the causal mask of the LLM to perform full attention, which enables the representation of the latter token can be attended by former tokens. The output feature of each layer is controlled by adaptive gating networks, whose weights are initialized as zero for better training stability. To be specific, we first perform the average pooling to the LLM representation, then the pooled representation is merged with embeddings of the timestep via element-wiss sum. The gating network takes such timestep-awart and context-aware representations as input to perform precise information injection. The final output representation of the refiner will be jointly fed into the collaborative refiner for enhancement.
insert a linguistic token refiner module refine the
Collaborative Refiner. To further improve text comprehension, we adopt multiple LLMs and linguistic token refiners for prompt encoding and collaboratively refine these representations through the proposed collaborative refiner. The representations from multiple linguistic token refiners are separately processed by multiple parallel branches and each block branch consists of cross- attention and FFN layer. Besides, we use modulation mechanism to condition each layer of collaborative refiner on the timestep and text context. This modulation takes the same input as the aforementioned gating network in the linguistic token refiner. The branches in this module are connected by multiple parallel cross-attention layers, where the text representations can be collaboratively refined. Specifically, the gross-attention layer takes the feature of the current branch as the query, and the features of other branches as the key and value refine the current feature. Finally, We truncate the output token sequence, discard the instruction tokens, and mix both representations
5

---

by concatenation. This mixed and refined representation can be flexibly integrated into diffusion models to provide discriminative text conditional information.
3.2 LLM-infused Diffusson Transformer
Our proposed LL-M-ifuueed Diffuser can be flexibly integrated into current diffusion models. Consid- ering the remarkable scaling capacity of diffusion transformers [15], we develop diffusion model named LLM-infused Diffusion Transformer (LI-DiT).
Following the paradigm of DIT, LI-DiT takes the noisy representation from the latent space of a variational eutoencoder (VAE) as input and converts the spatial input into a sequence of tokens. Each transformer block of LI-DiT contains self-attention layer, cross-attention layer, an FFN layer, and the modulation module. The cross-attention layer can inject the text conditional information extracted by LLM-infused Diffuser into the token sequence. The modulation module receives the timestep embeddings and text representation to provide extra conditional information. Unlike the 2D positional embedding designs in previous works, we adopt a convolution-basee position embedding. After the patchify layer the diffusion transformer, we directly adopt a ResBlock [20] as the positional embedding module. The translation invariance of convolutional operators can effectively introduce positional information to the transformer operators. Therefore, LI-DiT can support arbitrary resolution image generation without requiring additional design modifications.
Large-scale transformer models usually suffer from unstable gradients and numerical precision, leading to divergent loss during training. To deal with the training instability issue, we incorporate several strategies adopted in large-scale vision or language model training. First, we introduce the QK-norm [21, 22] in both self-attention layers and cross-attention layers. The RMSNorm [23] layers will normalize the query and key tokens before the dot product computation of attention score. Such operation enables the numerical stability of attention scores and avoids unstable gradients from ouitt--distribution values. Besides, considering the broader numerical representation range of 69loat16, we finally use the 6 mixed precision training [24] strategy.
4 Comparing with Other Methods Adopting LLMs
Our LLM-infused diffuser has significant differences compared to the existing methods that utilize LLMs for prompt encoding. Apart from leveraging LLMs without specific design [25], current works can be classified into three categories. The first is that LLMs generate the image layout based on the prompt, and then the diffusion model completes the image based on this layout [26, 27, 28]. The second one is training extra adapter to align LLM with frozen diffusion models like Stable Diffusion 1.5 [4] and Stable Diffusion XL. [29] for better prompt comprehension capabilities [30, 31, 14, 13).
The contribution of the LLM-infused diffuser does not conflict with the layout approach. The layout methods are usually adopted as the controllable plugin in specific areas like visual composition and number-sensitive tasks. They need to be used in conjunction with powerful diffusion model. However, the generation quality of each object in the layout still relies on the prompt understanding capability of the diffusion model. When generating single object with a complex description, the layout approach essentially falls back to directly using the diffusion model for generation. Mcanwhile, the layout can only provide the spatial relationship of objects but can not guide the generation of complex object relationships such as a boy sitting on the shoulder of a man, while the L.LM- infused diffuser can easily deal with it. The adapter-based methods have not addressed the issues. LLM4GEN [31] also observed that the performance when adopting T5-XL can also easily outperform using larger 13B decoder-only LLMs. However, they did not provide any further analysis and directly used T5-XL as the final text encoder.
5
Experiments
5.1
Implementation Details
Model Architecture. Our experiments are conducted on the smaller model LI-DiT-IB by default. We adopt the LLAMA3-8B and @owen1.5-7B [32] with multi-modal instruction fine-tuning [33] as the dual text encoders for both LI-DiT-IB and LI-DiT-1OB, For the ablation study baseline, we only keep the LLaMA3-SB to reduce training costs. We adopt 2 blocks in the linguistic token refiner

---

Table 1: The performance of LI-DiT on 121-CompBench, DPG-Bench and GenEval benchmark. We compare LI-DiT-IB with recent open-source academic works and compare LI-DiT-1OB with mainstream closed-source source commercial models. Experiments indicate the superior capabilities of LI- DiT on complex prompt understanding across the model size.
DPG-Rench
Model
color
shupe
texture
spatial
single
IWG
counting
colors
position
altribution
overall
average
SD v1.5 [4]
37.24
41.59
12.04
0.38
0.43
63.18
SD v2
50.65
42.21
49.22
13.42
0.98
0.51
0.44
0.85
0.07
0.17
0.50
68.09
[29]
20.32
74.65
SD3-1B [9]
0.97
0.72
0.52
0.78
0.16
0.34
0.58
DALL-E
57.50
54.64
63.74
12.83
PixArt-o [17]
68.36
55.82
70.44
20.82
0.98
0.50
0.44
0.80
0.08
0.07
0.48
71.11
LIDDTTIB
74.08
59.34
69.59
27.57
0.98
0.69
0.48
0.86
0.22
0.37
0.60
81.65
DALLEE3 [S]
81.10
67.50
80.70
0.96
0.47
0.47
0.83
0.43
0.45
0.67
83,50
SD3 SB
0.99
0.94
0.72
0.89
0.33
0.60
0.74
LI-DIT-10B
83.78
68.03
78.50
39.69
0.99
0.91
0.65
0.91
0.47
0.64
0.76
84.60
3
الاماا Alignment
(Ml U.I.
50%
0%
Quality
Quality
Algnment
Quality
Alignment
Figure 6: Human evaluation performance. Our LI-DiT-10B surpasses other open-source and close-
source leading text-to-image generators both quality and alignment. We observe that LI-DiT- 10B surpasses Stable Diffusion 3 and Dall-E on both quality and alignment. Compared with the most popular Midjourney V6, LI-DiT- demonstrates leading capabilities in image-text alignment with similar image-text quality performance.
and 1 block in the collaborative refiner. In our experiments, we take the text embedding from the third-to-last last transformer block as the output of each LLM. For the detailed architecture of LI-DiT-1B and LI-DiT-10B, please refer to the supplementary materials.
Training Data. All the exploration and ablation experiments are trained on the ImageNet dataset [35] and a subset the CC12M dataset [36]. We assign the text prompt of "a photo of [class/" to each sample of ImageNet and randomly select 1.3M image-text pairs from CC12M. Following previous works [9], we mix the original captions and synthetic captions generated by CogVLM [37]. When compare LI-DiT with other leading counterparts,
employ large-scale training dataset with billion-level image-text pairs, including LAION-5B [38] and other internal datasets containing both English and Chinese, which enables LI-DiT with bilingual comprehension capabilities. Following stable diffusion [4], we remove the image-text pair from LAION when its aesthetic scorer is lower than 4.7. Low-resolutin Specifically, we only sample a subset of 30M image-text pairs from this large-scale dataset to train LI-DiT-1B and use all the billion level pairs train the LI-DiT-1OB.
images and low-quality prompts including URLs and tags also removed.
Training Details. Following the paradigm of latent diffusion models (LDM) [4], we leverage a VAE encoder [39] to project the image representation into the latent space. We train VAE with 8x downsample rate and 16 channels for better image generation [9]. We do not use any data augmentation strategies. Following the multi-scale training in RAPHEL[40) we group the images based on their aspect ratio. Only images with similar aspect ratios will construct a batch. For the ablation experiments conducted on 3M image-text pairs, train the models with batch size of 256 and a learning rate of le-4 for 300k iterations at 256 resolution. For the training of LI-DiT-IB. we increase the batch size 2048 and iterations to 500k. When training LI-DiT- 10B, the batch size is 4096, and the iteration number is over 1M. We directly employ a resolution of 512 during training, and then fine-tune it to 1024 resolution with high-quality data further improve the aesthetic quality.
Evaluation Metries. For the quantitative evaluation, we mainly consider the T21-CompBench [16], DPG-Bench [13], and GenEval benchmark [41]. We also introduce human evaluations for better
7

---

Table 2: Component- ablation.
Table 3: Effects of causal mask.
instruction
token
collaborative
T2I-avg
DPG-avg
LLM
token refiner full alta
T2l-we
DPG-wv
13.36
38.72
66.15
49.52
74.63
48.41
73,45
Qwenl.5
38,11
68.61
54.02
77.08
Owent.5
83.81
76.49
56.79
78.62
LLaMA3
45.81
71.01
80.25
LLaMA3
54,02
77.08
Table 4:
Effect of instruction.
Table 5: Token refiner design.
Table 6:
Fusion design.
multi-modal
instruction
T21-ave
DPG-avg
N
gating
T21-avg
DPG-uvg
setting
T2I-avg
DPG-avg
38.72
66.15
48.25
73.83
LLaMA
56.79
78.62
38.47
65.84
2
54.02
77.08
Qwen
56.13
78.49
44.22
71.81
concar
79.04
48.41
73.45
2
53.47
76.68
refiner
60.31
80.25
comprehension of the artistic and aesthetic qualities. Note that the "T21-avg" ablation studies refers to the average score of T21-CompBench attribute metrics.
5.2 Performance Comparisons
Quantitative Evaluations. In the quantitative evaluation, focus the alignment between generated images and the input prompts. As shown Tab. 1, we choose 121-CompBench, DPG- Bench, and GenEval benchmark to evaluate the generation capability of LI-DiT-1B and LI-DiT-10B. The 121-CompBench and the GenEval benchmark are composed of short prompts, focusing on the compositional evaluation. The DPG-Bench is built with complex dense prompts. Compared with open-source academic works like SDXL and PixArt-a, LI-DiT-IB outperforms them over all benchmarks by large margin. We also compare LI-DiT- 10B with DALL-E 3 and Stable Diffusion 3 (8B), two mainstream closed-source source commercial models. The significant improvement further validates the effectiveness of our LLM-Fused Diffuser.
Human Evaluations. The quantitative evaluation metrics can directly measure the artistic and aesthetic qualities. Following previous works, we conduct the human evaluation to convincingly compare LI-DiT-10B with Stable Diffusion 3, DALL-E 3, and Midjourney V6. Our evaluation dataset contains 200 prompts with diverse styles and scenarios. The image from LI-DiT- 10B and the image from competitor will construct an evaluation pair. The human evaluator will compare the image pair from the perspective of image quality and image-text alignment. The result in Fig. 6 indicates that LI-DiT- can surpass DALLE-3 and Stable Diffusion both image-text alignment and image quality. Compared with the most popular commercial model Midjourney V6, LI-DiT- 10B demonstrates leading capabilities in image-text alignment with similar image-text quality performance. In Fig. 7, we show some randomly sampled cases to make clear comparison.
5.3 Ablation Study
Componet-wise ablation study. As shown in Tab. 2, we conduct the component wise ablation study, We adopt DiT with pre-trained LLaMA3-8B as the baseline setting. First, we observe consistent performance gains after introducing the instruction to the input prompt or incorporating the linguistic token refiner to the baseline. When leveraging both designs, the image-text alignment performances on two benchmarks continue to improve. Besides, we introduce with multi-mode fine-tuning to verify the effectiveness of the collaborative refiner. The LLM fusion strategy further enhances the prompt comprehension ability of the diffusion model. These results clearly validate the effectiveness each proposed component.
extra powerful I.I., @wwel..5-7)
Effect of causal mask. We investigate the effect of the causal mask on prompt encoding in this experiment. As presented in Tab. 3, inserting a linguistic token refiner with full attention after the LLM significantly improves the performance. However, this refiner fails to increase the performance of the T5 encoder with bi-directional attention. we introduce the causal mask of LLM to refiner, severe performance degradation occurs in both LLaMA3-81 and Qwenl 1.5-7B. These results demonstrate the causal mask is core factor that harms the prompt encoding capacity of the LLM and our proposed refiner can climinate such positional bias.

---

LI-DiT-10B
MidjourneyV6
DALL-E 3
Stable Diffusion 3
little ger thts tes leaf flosting on water, surrounded by met. entrre scene exudes quiet and peacefu stmosphere
little girlin China rowing bost. Behind her, there a huge green dragon with rod eves. The entire stmosphere tornfying
image wearing lie, and plasses remlistic style spainst background
and looking at phone while flying
Figure 7: Comparisions with Midjourney V6, DALL-E 3 and Stable Diffusion 3. The prompts randomly sampled from our human evaluation benchmark.
Effect instruction. To verify the effectiveness of the instruction, we conduct an ablation in Tab. 4. First, we find the prompt instruction fails to bring gains for the model that employs base LLaMA3-8B without instruction fine-tuning. we institute the base model for a multi-modal instruction fine-tuned variant, the alignment scores can be significantly increased. Thanks to the strong cntruuciion-follovinn capacity brought by instruction fine-tuning, inserting the instruction can further boost performance. This result demonstrates the multi-mode instruction fine-tuning data helps the LLM better describe an image and highlight key elements within the image. Besides, the instruction is able to encourage the LLM to attend to the image contents in the given prompt.
Linguistic token refiner design. As shown in Tab. linguistic token refiner. First, we compare our model with other variants with different numbers of blocks in the refiner. We observe consistent performance gains when the number of blocks in the refiner increases. However, such gain is not significant when there are 2 blocks in the linguistic token refiner. Therefore, we employ 2 blocks in the token refiner to achieve the best trade-off between complexity and performance. Besides, we also ablate the effect of the gating network in the refiner. When we remove the gating network, the performances on both benchmarks decrease. This indicates that the conditional information of time and text context contributes to better image-text alignment.
conduct experiments the design of

---

Effect of collaborative refiner. As shown in Tab. 6, we observe the model with a simple fusion technique can outperform the other counterparts with a single LLM. Besides, the collaborative refiner can further boost the performance based on this concatenation fusion. Such result indicates that an effective representation fusion method can further enhance the capabilities of L.LMs.
6 Related Work
Diftusion models. The denoising diffusion probabilistic model (DDPM) [1] provides an effective manner to generate high-quality images. To train diffusion models on limited computational resources while retaining their quality and flexibility, the latent diffusion models (LDMs) [4] project the images into the latent space of pre-trained autoencoders [39].
come-coniiiioal UNet [42] applied to denoise from the noisy latent input. Please refer to the supplementary materials for detailed information about the optimization process. The transformer architecture has achieved remarkable success in various tasks. Dit [15] is the pioneering work in adopting transformer architecture in diffusion models. Transformer models exhibit excellent scaling properties [22], which support the training of large-scale diffusion models. Recent advanced models [17, 43, 9, 25, 44, 45, 46, 47, 48] in image generation and video generation mainly consider the transformer architecture as the backbone. Apart from the DDPM paradigm, Stable Diffusion 3 [9] and Lumina-T2X [25] leverage the flow matching [49] strategy to optimize diffusion models.
Text encoder for diffusion models. The CLIP text encoder [6] is popular among various text- to-image generation models [34, 29, 4]. Under the image-text contrastive optimization, the CLIP text encoder can map prompts into unified image-text space, providing valuable information for conditional image generation. Meanwhile, utilizing CLIP text encoders with larger parameters and more extensive training data [50, 51] has significantly enhanced the diffusion model's ability to comprehend prompts [29, 9]. Imagen [5] observes that large language models like T5 [7] pre- trained on text-only corpora are surprisingly effective at encoding text for image synthesis. Recent works [17, 43, 52, 8, 9] usually adopt the T5 series as the prompt encoding model. Considering the excellent text comprehension capabilities of decoder-only LLMs [11, 12, 53, 32, 54, 55, 56), some works [25, 14, 13, 57] try to introduce LLMS into the designed framework. However, systematic comparative analysis like Vicuna [58] exhibit powerful instruction following capabilities. The multi- instruction fine-tuning [59, 60, 61, 37, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71] further enables LLMs understand visual information. These models have the potential to serve as reliable text encoders.
T5 models and LLMs is still missing. LLMs with instruction fine-tuning
7 Conclusion
In this paper, explore the role of LLMs in prompt encoding for diffusion models based on the poor performance in the text-to-image generation task when adopting a decoder-only LLM to encode prompts. Through experiments and analysis, we identified the core factors limiting decoder-only LLMs as effective text encoders for diffusion models are the misalignment between next token prediction training and the requirement for discriminative prompt features in diffusion models, and the intrinsic positional bias introduced by the decoder-only architecture. To deal with the issues, we propose novel framework to fully harness the capabilities of LLMs. We further design an LLM--lfuueed Diffusion Transformer (LI-DiT) based on the framework. LI-DiT surpasses state-of- the-art open-source models as well mainstream closed-source commercial models including Stable Diffusion 3, DALLE-3, and Midjourney V6.
8 Limitation and Potential Negative Societal Impact
Due to the limited computation resources, we conduct experiments on LLMs with 7B parameters. In future work, we will further validate the effectiveness of LLM-iifused Diffusion in larger LLMs with 13B or 70B parameters. The potential negative social impact is that images may contain misleading or false information. We will conduct extensive efforts in data processing to deal with the issue.
The work was supported by the National Key R&D Program of China under Grant 202 00212D0200000.
Acknowledgments
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840-688 1108844--681,2220 2020.
[2] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 22655. PMLR, 2015.
[3] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10699) 2022.
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-ine text-to-image ro-image diffusion models with deep language understanding, Advances in neural information processing systems, 11136479-364999 2022.
[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 21(140):1-6) 2020.
[8] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang. Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/tall-e-3 pdf, 2(3):8, 2023.
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 13:187711901, 2020.
[11] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models (2023). arXiv preprint arXiv:2302 13971, 2023.
[12] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922: 2023.
[13] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint 2xxiv:2403.05133 2024.
[14] Shihao Zhao, Shaozhe Hao, Bojia Zi, Huaizhe Xu, and Kwan-Yee K Wong. Bridging different language models and generative vision models for text-to-image generation. arXiv preprint arXiv:2403.00 07860, 2024.
[15] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEFF/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[16) Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78745 2023.
[17] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:23 2023.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 03300, 2020.
11

---

[19] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normal- ization for transformers. arXiv preprint 2rXiv:2010.04245 2020.
[22] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[23] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[24] Dhiraj Kalamkar, Dheevatsa Mudigere. Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang. Hector Yuen, et al. A study of 6 6 for deep learning training. arXiv preprint arXiv: /905. /2322, 2019.
[25] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945 2024.
[26] Weixi Feng. Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024.
[27] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023.
[28] Quynh Phung. songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7932-7942, 2024.
[29] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 2rXiv:2300.00195. 2023.
[30] Shanshan Zhong, Zhongzhan Huang, Weushao Wen, Jinghui Qin, and Liang Lin. Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models. In Proceed- ings of the 31 ACM International Conference on Multimedia, pages 567-578. 2023.
[31] Mushui Liu, Yuhang Ma, Xinfeng Zhang. Yang Zhen, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. Llm4gen: Leveraging semantic representation of 1lms for text-to-image generation. arXiv preprint arXiv:2407.00 2477007777 2024.
[32] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu, Qwen technical report. arXiv 2023.
preprint arXiv:2309.
[33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.0374, 2023.
[34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical cext-conditional image generation with clip latents. arXiv preprint 2XXi:22244006255 1(2):3 2022.
12

---

[35] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[36] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[37] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03077 2023.
[38] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
[39] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[40] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
[41] Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pages 234 241. Springer, 2015.
[43] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- (sigat- Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.. 04692, 2024.
[44] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. None, 2024.
[45] Dongzii Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. arXiv preprint arXiv:2404.03653 2024.
[46] Yun Wang, Xiaoshi Wu, Zhaoyang Huang. Xiaoyu Shi, Dazhong Shen, Guanglu Song, Yu Liu, and Hongsheng Li. Be-your-outpainterr Mastering video outpainting through input- specific adaptation. In European Conference on Computer Vision, pages 153-168. Springer, 2025.
[47] Dazhong Shen, Guanglu Song, Zeyue Xue, Fu-Yun Wang, and Yu Liu. Rethinking the spatial inconsistency in classifier- free diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9370-9377 900009979.999 2024.
[48] Fu- Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, Hongsheng Li, and Xiaogang Wang. Phased consistency model, 2024.
[49] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[50] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. None, July 2021. If you use this software, please cite it as below.
[51] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmano, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
13

---

[52] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation, 2024.
[53] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang. Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309. 10305, 2023.
[54] 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang. Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi
Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu,
Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
[55] InternLM Team. Internlm: A multilingual language model with progressively enhanced
capabilities, 2023.
[56] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
[57] Zhuofan Zong, Dongzhi Jiang, Bingqi Ma, Guanglu Song, Hao Shao, Dazhong Shen, Yu Liu, and Hongsheng Li. Easyref: aenii-gnnerllized group image reference for diffusion models via multimodal Ilm. arXiv preprint arXiv:2412.09:: 18, 2024.
[58] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.0322 2023.
[59]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[60] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[61] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual
instruction tuning, 2023.
[62] Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng
Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404. 13046, 2024.
[63] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[64] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong. Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-though reasoning in multi-modal language models. arXiv preprint 2403 240.:999999,, 2024.
[65] Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409. /2959, 2024.
[66] Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Lumen: Unleashing versatile vision-centric capabilities of large multimodal models. arXiv preprint arXiv:2403.07300 2024.
[67] Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, and Yu-Gang Jiang. Eyes can deceive: Benchmarking counterfactual reasoning abilities of multi-modal large language models. arXiv preprint arXiv:2404. 12966, 2024.
[68] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagmosing event hallucinations in video 1lms. arXiv preprint arXiv:2409. 16597, 2024.
[69] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1279) 2023.
[70] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi the right way for evaluating large rision-language Wang, Yu Qiao, Dahua Lin, al. Are 2024.
models? arXiv preprint arXiv:2403.203..
[71] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint 24XXi:240600332 2024.
14

---

A Appendix
A.1 Denoising Diffusion Probabilistic Model
The optimization target of DDPMs can be defined by maximizing the log-likelihodd the training data. Given the data distribution (xx),, the forward diffusion process is defined as:
g ( xo) II (x-);
where
xx- = Axx;; va ai)
(2)
Here, de is the noise schedule parameter. The reverse diffusion process is the key part of training, where a parameterized model pa is learned to approximate the reverse process of the data:
T
po (x(oT))) = p(xT) II
(3)
where
Po Po (x 1o ()
(4)
The optimization objective of DDPM is to minimize the Variational Lower Bound (VLB), thus maximizing the log-likelihood of the model. The optimization objective can be expressed as:
g(x1.T (Xo)
LVLB = Eg(xorr)
log
(5)
pa (x(:T)
By decomposing and rewriting, get the following form:
CVLB Ex(xxrr)
Σ (xx log po x1)0
(6)
Simplified and restated as the loss at each timestep:
L. E xxx.
co 1)13
20?
where é is the noise. These formulas describe the optimization objective of the DDPM model. By minimizing this loss function, the model can effectively learn the data distribution and progressively denoise during the generation process, producing high-quality data samples.
A.2 Detailed information of LI-DiT-1B and LI-DiT-10B
In Tab. 7, we provide the detailed architecture and training information of LI-DiT- IB and LI-DİT-1OB.
Table 7: The detailed architecture of LI-DIT-1B and LI-DiT-10B
model
depth
hidden size
head number
patch size
input resolution
batch size
iter
training data
LI-DIT-TB
28
1152
16
256
2048 4096
500k
30M
LI-DIT-10B
48
512/1024
1M
A.3 Evaluation Benchmark Construction for Positional Bias
In this chapter, we primarily discuss the construction of the positional bias evaluation benchmark. We used the attributes and nouns provided by T2l-ChmpBench to construct 1,000 prompts, each containing up 8 nouns or attributes. For each prompt, the diffusion model will generate images. We divided each prompt into segments (adj-noun composition). Following the design in T21-CompBench, we used BLIP [63] to score the alignment segments at different positions. When testing performance, we first calculate the average score within each prompt segment, then compute the overall average score for prompts to obtain the model's accuracy within that segment. We provide few samples follows:
15

---

A green bench, a red car, a blue bowl, and a pink apple.
A black banana, yellow bird, a blue dog, and a brown horse.
A metallic car, a wooden desk, a rubber band, and metallic knife.
A plastic cutlery, fabric shirt, a fluffy pillow, and leather gloves.
A big elephant, small flea, diamond pendant, and round watch.
A round bagel, rectangular knife block, tall lighthouse, and short buoy.
A.4 Prompts in Fig.
We provide the prompts adopted to generate images in Fig. 1. The prompts are arranged from left right, top to bottom.
A dramatic coastal cliff scene with waves crashing against the rocks below. The cliffside is covered in green grass and wildflowers, and lighthouse stands tall on the edge, overlooking the vast ocean. The sky is partly cloudy, with the sun peeking through.
A Chinese dragon with Pikachu on its head, featuring fire effects.
A surreal painting features giant octopus with vivid purple tentacles emerging from large teacup. while miniature ship floats on the surface. The whimsical seascape blends ocean waves with fantastical elements, creating dreamlike atmosphere. Vibrant colors and playful light reflections enhance the scene, inspired by fantasy art, and rendered in high definition for an immersive experience.
A digital art piece using C4D modeling blends Wang Ximeng's 's landscape art with jade carving and multi-layered paper-cutting Featuring the Yellow Crane Tower, white jade- carved clouds, and sculpted buildings, it incorporates crystal and glass for digital feel. Predominantly white, the artwork highlights exquisite craftsmanship and lighting.
A golden wheat field with two ears of wheat forming a heart shape the center of the image. The background is the sky under the midday sun.
A complex of buildings floating high in the sky, with a huge alloy sign reading "LI-DIT" made of high-strength transparent nanomaterials, resembling islands in the air.
A photo portrait of handsome woman and beautifiul forest, double exposure
An ink wash painting with abundant brushstrokes and heavy sense of history. It features ancient Chinese gardens with gray walls, black tiles, pavilions, boats, flowers, and trees. A stone bridge spans the water, with intricately arranged rockeries, evoking the serene atmosphere of a Jiangnan water town.
paper-cutting art piece features a little girl beneath a glowing moon, A surrounded by flying birds and flowers. The watercolor illustration uses warm colors on light background, with exquisite details and 3D rendering. Pastel hues and soft light create a dreamy, delicate atmosphere, resulting in a high-quality, visually captivating design.
A handsome little dog carrying a camera on its shoulder.
A massive treehouse built within giant conch shell, intricate wooden bridges, and lanterns adorning the shell's spiral. The background is vibrant coral reef with colorful marine life. Soft, underwater lighting with shimmering reflections.
16

---

A.5 High-quality Images Showcases,
gant wave, origamii
grass
glass colorful
Ministure photography, three tiny formers picting slant broccoll bright studio lightnn, light green background, minimalitt
Double exposure, transparent, semi transparent sihoouete of wol's face overlald with semi- sprawting
cute boy in the middle of grass field, Miyarski- style, fentasy, vastness, flowers, falling leaves. distance castle
3. 19.8 100 style,
KALIXAAKKE 66 t H, (6 A(A surreal fantosy world where dragon made of offttoe roaring, with pedestrians faintly visible)
12: pencil Illustration of lttll boy holding star, standing on the surface of earth)
Figure 8: LI-DiT-1OB exhibits an astonishing ability to understand bilingual prompts, accurately generating images even with complex descriptions and combinations of objects.
17

---

Ankie slender
linge ne deedowe eventn a dank evening sower
nutaile wendow
detailee
spuce ship abous farmir the micdle novehere
ShangHai
Farl on Shanghal dhy doviinn bls and pargle giddarr
A.uropeon grl du tgh faaiig thouuh aaves Eurinsting derappaddiic youutt enarg!
Isno, iserdri poaa
A tie eesovi firest, Bild perk Bowerr warding creekiram through.ee with diar wane Fallen poller difting eway with the orvent Dstarr veew onerrtccefec
Figure 9: LI-DiT-10B exhibits astonishing ability understand prompts, accurately generating images even with complex descriptions and combinations of objects.
18

---

A.6 Comparison with Other Models.
Real photography, princess wearing green dress, purple clothes, her hair is very long and red, very beautiful, wearing a crown on her head, living in the sea.
Figure 10: Comparisions with Midjourney V6, DALL-E 3 and Stable Diffusion 3. The prompts are randomly sampled from human evaluation benchmark. The images presented in the order of LI-DiT-1OB, Midjourney V6, DALL-E 3, and Stable Diffusion 3.
19

---

3D, Octane render, bust of white, skinned woman with light eyes, thick lips, thin nose, fine white fabric dress, with angels and flowers, Renaissance style
Figure 11: Comparisions with Midjourney V6, DALL-E 3 and Stable Diffusion 3. The prompts randomly sampled from our human evaluation benchmark. The images are presented in the order of LI-DiT-10B Midjourney V6, DALL-E 3, and Stable Diffusion 3.
20

---

Anime, peaceful snow scenery, peaceful lake surface, snow covered branches, winter wonderland in the twilight, picturesque scenery, Thomas Kinkade style, magical and peaceful, high quality.
Figure 12: Comparisions with Midjourney V6, DALL-E 3 and Stable Diffusion 3. The prompts randomly sampled from our human evaluation benchmark. The images are presented the order of LI-DiT-10B 10B, Midjourney V6, DALL-E 3, and Stable Diffusion 3.
21

---

LIQUID
A bottle of beauty care liquid sank into the sea and is surrounded by bubbles. There are too many bubbles. Soft light is refracted through the sea water. The large water ripple network makes the picture beautiful, high resolution, fine detail, front view, 8K
Figure 13: Comparisions with Midjourney V6, DALL-E 3 and Stable Diffusion 3. The prompts randomly sampled from our human evaluation benchmark. The images are presented the order of LI-DiT-10B 10B, Midjourney V6, DALL-E 3, and Stable Diffusion 3.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
23

---

Justification:
Our paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:
[No]
Justification:
We will not opensource data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.o cc/
pessoonoolccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/tterc/uuiieeeoeeuuumuimiiiiii / //nips. wwtpccccpppccceeeecce..cccccccct haabblicggiids..... /CooeSubmbie cebbbmissiii ssiiinsisnneeeeee oniioniyl) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper specifies all the training and test details
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not report the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to the appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerr..p.eee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please Section Experiments
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please Section Experiment
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "CeXuyYDLxR", "reviewer": "Reviewer_Nr61", "summary": "This work identifies two main reasons for degraded prompt-following ability in image generation with decoder-only transformers: the misalignment between pretraining objective and diffusion's need of discriminative prompt feature, as well as the intrinsic positional bias for decoder-only transformer. The solution is to enhance text representation capability and remove positional bias. This work also proposes a diffusion architecture conditioned on multiple LLMs. The proposed LLM-Infused Diffusion Transformer (LI-DiT) surpasses previous state-of-the-art open-source models in prompt understanding.", "questions": "The reviewer's questions are primarily from the weaknesses section, specifically:\n1. What are the sizes of the circles indicate in Fig. 2?\n2. What are the model sizes and the training dataset size of the models in Tab. 1?\n3. What is the performance of a model if the collaborative refiner is replaced by the simple linguistic token refiner, with input concatenated in the sequence length dimension?", "limitations": "The reviewer did not find unaddressed potential negative societal impact. The unaddressed limitations are described in the weakness section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Analysis of the reasons that LLMs not working out-of-the-box is interesting.\n* The qualitative and quantitative results demonstrate advantages over prior baselines.\n* The ablation studies are solid. The contribution of each contribution is demonstrated.\n* The paper is easy to read and understand.", "weaknesses": "The main weakness of this work is its lacked novelty.\n* The Input Prompt part (Sec 3.1) is prompt engineering. \n* The Linguistic Token Refiner (Sec 3.1) can be considered as adding more transformer blocks that are trainable and have full-attention, with the LLM weights frozen. \n* The cross-attention blocks in Collaborative Refiner (Sec 3.1) can be considered as self-attention of concatenated features without attending to each other within the same LLM tokens. \n* The technical contributions in Sec 3.1 do not exhibit significant novelty.\n\nFurthermore, the comparisons with other works are not clear.\n* What the sizes of the circles indicates in Fig. 2 is not clear.\n* The model sizes of the baseline models under comparison are not listed in Tab. 1.\n\nFinally, this work ignores an existing line of work that combines LLMs with diffusion models [1,2,3]. These works use LLM to generate intermediate representation generation and then generate images conditioned on the intermediate representation.\n\n[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. Feng, et al. NeurIPS 2023.\n\n[2] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. Lian, et al. TMLR 2024.\n\n[3] Grounded Text-to-Image Synthesis with Attention Refocusing. Phung, et al. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Nr61,\n\nThanks for your comments. We will address your concerns below.\n\n## Q1: The contribution and novelty of our work.\n\nPlease refer to the Q1 of our global response.\n\n## Q2: Comparing with other methods adopting LLMs. \n\nPlease refer to the Q2 of our global response.\n\n## Q3: The sizes of the circles in Figure 2.\n\nThe size of each circle indicates the parameter size of the text encoder model. T5-XL with the fewest parameters is marked with the smallest circle. LLaMA3-8B is marked with a large circle.\n\n## Q4: Model sizes and training dataset sizes in Table 1.\n\nWe present the parameters and training dataset sizes of models in Table 1. Some of the models do not provide parameters or training data size in their papers. Considering current works usually use unavailable internal datasets, it is hard to make fair comparisons.  For the effectiveness of our method, please refer to the experiments in the ablation study section. \n\n| Model            | SD v1.5  | SD v2 | SD XL | SD3-1B | DALL-E 2 | PixArt-$\\alpha$ | Li-DiT-1B | DALL-E 3 | SD3-8B | Li-DiT-10B |\n| ---------------- | :------: | :---: | :---: | :----: | :------: | :-------------: | :-------: | :------: | :----: | :--------: |\n| Parameters       |   0.9B   | 0.8B  | 2.6B  |   1B   |   6.5B   |      0.6B       |    1B     |    -     |   8B   |    10B     |\n| Image-text Pairs | 2000M[1] |   -   |   -   |   -    | 650M[1]  |     25M[1]      |    30M    |    -     |   -    |   1000M    |\n\n- [1] PixArt-$\\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis\n\n## Q5: Replace the collaborative refiner with a linguistic token refiner.\n\nWe conduct additional experiments to verify using a simple linguistic token refiner for fusing concatenated text embeddings. We can observe that adopting a linguistic token refiner will only bring limited improvement. Meanwhile, when fusing multiple LLMs, the self-attention mechanism in linguistic token refiner requires more computational resources and memory compared to the cross-attention mechanism collaborative refiner. \n\n| Setting | Concat | collaborative refiner | linguistic token refiner |\n| ------- | :----: | :-------------------: | :----------------------: |\n| T2I-avg | 58.32  |       **60.31**       |          58.86           |\n| DPG-avg | 79.04  |       **80.25**       |          79.32           |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mi4m4eF68H", "reviewer": "Reviewer_yHpu", "summary": "In the context of text-to-image (T2I) generation, this work addresses the problem of leveraging representations from state-of-the-art decoder-based LLMs for conditioning image generation. The authors highlight challenges of leveraging existing LLMs, namely - misalignment in representations due to differing training objectives (autoregressive v/s non-autoregressive) and the effects of positional bias. The authors then propose ways to mitigate this misalignment via prompt changes (leveraging instruction tuning of LLMs), embedding ensembling techniques (for positional bias), and a new DiT based on this ensembled conditioning mechanism. The proposed observations and method show consistent qualitative and quantitative improvements on benchmarks across the board.", "questions": "(Besides the ones in the above section) In the experiments, what is the average prompt context length (min, max, avg)?", "limitations": "Discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper addresses an important and relevant problem of aligning a) very capable LLM models with b) capable image generation models and understanding why and if there could be a mis-alignment between these two objectives. I like the motivation of leveraging instruction tuned language models and it is understandable why one might want to switch to, or at least explore these models for conditioning text to image generation. Further, the results are impressive and I appreciate the ablation studies conducted for different components proposed in the work, and the custom benchmark to probe the alignment. The paper is presented in a clear and concise manner with appropriate visuals where necessary.", "weaknesses": "While the paper shows significant empirical improvements, I have some concerns and questions where it would be great to have a discussion with the authors:\n\n1. Image generation with diffusion models is non autoregressive, while decoder-LLM based representations are trained with auto-regressive models. There is much needed discussion surrounding the choice of models (Encoder decoder, encoder only, decoder only) which does not seem to be addressed in this work. For example, it seems that encoder-only representations are a natural choice for this task (such as InstructOR[1] embeddings, which are tuned for multi task learning in an encoder-only setup, or others from the MTEB [2] benchmark). How do they, or rather would they perform? What is the intuition?\n\n\n2. The motivation behind adding ensembling, connections with positional bias, and the linguistic token refiner from different LLMs seems lacking. What is intuition here? Why does one need “..The image prompt with instructions to be encoded by multiple frozen LLMs separately”?\n\n3. Re. the positional bias: Given that the LLMs tested in the paper are still small (~2-7B), it may be possible that their “long context” understanding may be limited as an artifact of their instruction tuning. Thus, is the positional bias inherent to these decoder based LLMs or a capability drawback of the smaller LLM on long context itself? That is, can this gap be closed if one employs an LLM with impressive and uniform long context capabilities? \n\n[1] One Embedder, Any Task: Instruction-Finetuned Text Embeddings Su, Hongjin, et al.\n\n[2] Muennighoff, Niklas, et al. \"MTEB: Massive text embedding benchmark.\"\n\nSuggestions:\n1. I would encourage the authors to further expand on related work, especially focussing on the alignment between language model types and the task of (non-autoregressive) image generation.\n2. (Presentation) The coherence between claims can be improved (it seems that many observations -> design changes) are grouped together.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer yHpu,\n\nThanks for your comments. We will address your concerns below.\n\n## Q1: Analyses on the choice of text encoders. \n\nAnalyzing the choice of text encoders from encoder-only model, decoder-only model, and encoder-decoder model is one of the core contributions of our paper. In the paragraph starting from Line 41, we analyze the difference in model architecture, optimization target, and performance in the image generation task between T5-like encoder-decoder models and GPT-like decoder-only models. In Section 2.1, we also exploring the ability of different models to retain prompt information. Based on these analyses, we identified the drawbacks of decoder-only LLMs compared to the T5 encoder and designed a series of methods to address these issues. \n\nAs expressed in the paragraph starting from Line 41, our work acknowledges the advantages of encoder-only models in terms of model architecture and optimization targets. However, the current paradigm for LLMs is the decoder-only architecture. For example, the top models in the MTEB benchmark like bge-en-icl[1] and stella_en_1.5B_v5 are based on decoder-only LLMs. Encoder-only models like the InstructOR and Sentence T5-XXL[2] are limited by the capabilities of the foundation T5 model, making it difficult to achieve leading performance. We conduct experiments following in Figure 2, simply replacing the text encoder with InstructOR-XL, a fine-tuned T5-XL encoder. The average performance on the T2I-CompBench is 44.06%, similar to the performance of the T5-XL encoder with 43.47%. \n\nThe design of the LLM-infused diffuser incorporates the advantages of the encoder-only architecture and aims to leverage advancements in decoder-only LLMs to further enhance the text understanding capabilities of diffusion models.\n\n## Q2: Ensembling multiple LLMs.\n\nWe observe that different LLMs exhibit preferences for prompt encoding, and integrating multiple LLMs can effectively enhance the model's ability to understand prompts as shown in Table 6.  Ensembling multiple encoders has been widely adopted in advanced works including SDXL[3], SD3[4], and recent popular FLUX.1. This indicates that effectively leveraging the capabilities of multiple text encoders is a crucial factor in enhancing the prompt understanding ability of diffusion models.\n\n## Q3: Using LLMs with long context capabilities.\n\nThe prompt lengths used in positional bias evaluation and user input are significantly shorter than the context length of LLMs. For example, the LLaMA3-8B supports the context size of 8192 tokens. However, the prompt lengths in existing test sets and user scenarios are significantly shorter. In the following table, we use the LLaMA3-8B tokenizer to analyze the prompts of different datasets.\n\n| Data                                      | Min  | Max  | Avg  |\n| ----------------------------------------- | :--: | :--: | :--: |\n| T2I-CompBench                             |  2   |  35  |  10  |\n| DPG-Bench                                 |  40  | 175  |  82  |\n| GenEval                                   |  5   |  13  |  8   |\n| Positional Bias Evaluation Benchmark      |  16  |  25  |  19  |\n| Human Evaluation Benchmark                |  5   | 153  |  88  |\n| Sampled 1000 user prompts from Midjourney |  1   | 721  |  85  |\n\n- [1] C-Pack: Packaged Resources To Advance General Chinese Embedding\n- [2] Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models\n- [3] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis\n- [4] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHIVPL2Wnx", "reviewer": "Reviewer_NTY6", "summary": "This paper presents an investigation into the integration of Large Language Models (LLMs) into text-to-image diffusion models. It identifies issues with using LLMs as prompt encoders, namely misalignment between next-token prediction training in LLMs and the need for discriminative prompt features in diffusion models, as well as positional bias introduced by the decoder-only architecture. The authors propose a new framework to overcome these challenges and introduce an LLM-Infused Diffusion Transformer (LI-DiT) to leverage LLMs effectively in image generation tasks. The paper also discusses broader societal impacts and adheres to ethical guidelines.", "questions": "see weakness", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method addresses a significant gap in utilizing LLMs for prompt encoding in diffusion models, offering a new solution to enhance text-to-image generation.\n- The paper includes a discussion on potential societal impacts, considering both positive and negative outcomes,", "weaknesses": "- The visual results of I2T may exhibit cherry-picking, including Figure 1 and Figure 7. The authors need to provide further explanations. Although they claim there are more prompt-generated results in the appendix, this clearly does not mitigate the risk of cherry-picking.\n- Integrating LLMs into diffusion models is not uncommon, and the authors lack discussion and analysis of such works, e.g., [1][2][3]. More in-depth exploration by the authors is needed.\n- Honestly, in the visual results of Figure 7, I did not notice any particular advantages of the proposed method.\n- Besides the visual effects, the proposed method does not seem to offer significant improvements over existing methods and lacks theoretical support. There is a need for further improvement in novelty.\n\n\n\n\n[1] LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation\n\n[2] UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion\n\n[3] SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer NTY6,\n\nThanks for your comments. We will address your concerns below.\n\n## Q1: The contribution and novelty of our work.\n\nPlease refer to the Q1 of our global response.\n\n## Q2: Comparing with other methods adopting LLMs.\n\nPlease refer to the Q2 of our global response.\n\n## Q3: The risk of cherry-pick in visualization.\n\nAs stated in the paper, the images in Figure 1 and Figure 7 are randomly sampled. Apart from visualization showcases, the human preference evaluation in Figure 6 also demonstrates the powerful image generation quality and prompt following capability of LI-DiT-10B. Considering the requirements of double-blind review, we will provide the link to our online platform in the final version. \n\n## Q4: The visualization comparision in Figure 7.\n\n- The image from LI-DiT-10B follows the *tea leaf* in the prompt but it is the lotus leaf for other models. The image from LI-DiT-10B possesses remarkable aesthetic qualities in terms of light and shadow details, whereas the colors in other images appear more muted.\n- The image from LI-DiT-10B better captures the *terrifying atmosphere* described in the prompt.\n- These models can all follow prompts to generate high-quality images.\n- The image from LI-DiT-10B maintains the original form of the crab, whereas crabs in the other images have human body parts. Meanwhile, The image from LI-DiT-10B follows the *red tie* in the prompt.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBkldD54ij", "reviewer": "Reviewer_nE1N", "summary": "The paper addresses the challenges of using large language models (LLMs) as prompt encoders in text-to-image diffusion models. It identifies two primary issues: the misalignment between LLM training objectives and the requirements of discriminative prompt features in diffusion models, and the positional bias introduced by the decoder-only architecture of LLMs. To tackle these issues, the authors propose a novel framework called LLM-infused Diffuser, which leverages human instructions and linguistic token refiners to enhance text representation capabilities. They design an LLM-Infused Diffusion Transformer (LI-DiT) and demonstrate its superior performance over state-of-the-art models in both open-source and commercial systems.", "questions": "* Could the authors provide information on both the training and inference costs associated with the proposed method, such as GPU memory consumption, training time, and inference time?\n\n* The paper mentions that the LLM-infused Diffuser can be easily and flexibly integrated into diffusion models. Does this imply that once trained, the LLM-infused Diffuser can seamlessly integrate into existing diffusion models without further fine-tuning? Besides, do diffusion models require additional fine-tuning to effectively cooperate with the LLM-infused Diffuser? It would be better to see results or insights into integrating the LLM-infused Diffuser into other diffusion models.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Introduction of LLM-infused Diffuser to facilitate the integration of LLMs into diffusion models to boost generation performance.\n* Experiments validate the effectiveness of this proposed framework when compared with both SOTA open- and closed-source baselines.", "weaknesses": "* Scalability: While the paper demonstrates the superior performance of LI-DiT, the scalability of the approach to other diffusion models is not fully discussed and validated.\n* Lack of training and inference costs: Despite the superior generation quality, the paper does not provide detailed information on the training and inference costs associated with the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer nE1N,\n\nThanks for your comments. We will address your concerns below.\n\n## Q1: The training and inference cost.\n\nWe train the LI-DiT-10B model on a GPU cluster with 1024 NVIDIA A100-80G. The training framework is implemented with Pytorch. We use the gradient checkpointing, mixed precision training and Fully Sharded Data Parallel (FSDP) to enable the large-scale model optimization, and adopt the flash-attention 2 operator for high-efficient computication. The training cost of LI-DiT-10B is about 47500 GPU days, between DALL-E 2[1] (41667 GPU days)  and RAPHAEL[2] (60000 GPU days). Recent works including Stable Diffusion 3[4] and DALL-E 3[5] do not provide the training cost.\n\nWe provide the inference cost on both Nvidia A800 GPU and Nvidia H800 GPU in BFLoat16 precision when using both Pytorch and TensorRT to generate images at $1024\\times1024$ resolution in 50 steps. The inference time is an average of generating 100 images.\n\n| GPU             |      Pytorch      |     TensorRT      |\n| :-------------- | :---------------: | :---------------: |\n| Nvidia A800 80G | 33.5s / per image | 27.0s / per image |\n| Nvidia H800 80G | 17.1s / per image | 11.2s / per image |\n\n## Q2: Integrating LLM-infused Diffuser into other diffusion models.\n\nLLM-infused Diffuser provides an effective paradigm to train diffusion basemodel with powerful prompt comprehension capabilities, which can be compatible with different architectures like transformers and U-Nets. The LLM-infused diffuser is jointly trained with the denoising network, serving as a part of the diffusion model. It can not be directly used in other diffusion models. We conduct a supplementary experiment on U-Net based diffusion model with 1B parameter similar to SDXL [6]. The training data and training setting follow that in Figure 2 and Section 4.1. We can observe the significant improvement in unet-based architecture, which further verifies its compatibility.\n\n| Text Encoder                | Denoise Network | LLaMA3-8B | LLaMA3-8B-infused diffuser |\n| --------------------------- | :-------------: | :-------: | :------------------------: |\n| T2I CompBench Average Score |      U-Net      |   37.24   |           50.86            |\n\nIf we hope to introduce LM-infused diffuser into existing models like SDXL, it requires extra adapter module and fine-tuning to align LLMs with SDXL. We will release a LLM-infused diffuser with SDXL in the final version.\n\n- [1] Hierarchical Text-Conditional Image Generation with CLIP Latents\n- [2] RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths\n- [3] PixArt-$\\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis\n- [4] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis\n- [5] Improving Image Generation with Better Captions\n- [6] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable time and effort all reviewers have dedicated to review our work. We are pleased to learn that the reviewers generally acknowledge and commend our contributions, including:

- The importance of LLM-infused diffuser in integrating decoder-only LLMs into the diffusion framework. (yHpu, NTY6, nE1N)
- The insightful analysis of the poor performance when adopting decoder-only LLMs as text encoders. (Nr61, yHpu)

- State-of-the-art performance. (Nr61, yHpu, nE1N)
- Solid ablation studies. (Nr61, yHpu).

We express our gratitude for the insightful and constructive suggestions provided by all the reviewers. We also show more high-quality generated images in the attached file. We will provide the link to our online platform in the final version. Here, we address the common concerns raised by the reviewers.

## Q1: The contribution and novelty of our work. (Nr61, NTY6)

The core contribution and novelty of our paper is addressing the issue of poor performance when using powerful decoder-only LLMs as diffusion text encoders. We put more emphasis on investigating the inherent properties of T5-like encoder-decoder models and GPT-like decoder-only models serving as text encoders (paragraph starts with line41), and the reasons behind the poor performance of decoder-only LLMs (Section 2). The simple and effective LLM-infused Diffuser is designed based on novel and contributional observations. Similarly, the key discovery of Imagen[1] is that T5 series models are surprisingly effective, which leads subsequent advanced works to adopt the T5 series models. Our work further advances the application of LLMs within the diffusion framework, enabling the continual development of LLMs to better enhance the text understanding capabilities.

## Q2: Comparing with other methods adopting LLMs. (Nr61, NTY6)

We argue that our approach has significant differences compared to the existing methods that utilize LLMs for prompt encoding. 

Current works that utilize LLMs can be categorized into: 1) LLMs first generate the image layout based on the prompt, then the diffusion model completes the image based on this layout[2,3,4] mentioned by reviewer Nr61. 2) Training an extra adapter to align LLM with frozen diffusion models like SD1.5[5] and SDXL[6] for better prompt comprehension capabilities[7,8] mentioned by reviewer NTY6. 3) Leveraging LLMs as text encoders without specific design[9]. UNIMO-G[10] mentioned by reviewer NTY6 adopts MLLMs to take extra images as conditional information, which is not the same task as text-to-image generation.

The contribution of the LLM-infused diffuser does not conflict with the layout approach. The layout methods are usually adopted as the controllable plugin in specific areas like visual composition and number-sensitive tasks. They need to be used in conjunction with a powerful diffusion model. However, the generation quality of each object in the layout still relies on the prompt understanding capability of the diffusion model. When generating a single object with a complex description, the layout approach essentially falls back to directly using the diffusion model for generation. Meanwhile, the layout can only provide the spatial relationship of objects but can not guide the generation of complex object relationships such as *a boy sitting on the shoulder of a man*, while the LLM-infused diffuser can easily deal with it.

The adapter-based methods have not addressed the issues we observed. LLM4GEN[7], which was submitted to arxiv on 30 Jun 2024 after the submission of NeurIPS, also observed that the performance when adopting T5-XL can also easily outperform using larger 13B decoder-only LLMs. However, they did not provide any further analysis and directly used T5-XL as the final text encoder. The following table is the Table 4 in LLM4GEN.

| LLMs       |   Color   |   Shape   |  Texture  |
| ---------- | :-------: | :-------: | :-------: |
| SD 1.5     |   37.65   |   35.76   |   41.56   |
| LLaMA2-7B  |   43.21   |   40.12   |   48.91   |
| LLaMA2-13B |   43.98   |   41.03   |   49.21   |
| T5-XL-1.2B | **45.34** | **43.28** | **51.52** |

Meanwhile, LI-DiT-1B also shows outstanding performance compared with LLM4GEN with fewer parameters.

| Model        | Parameter |   Color   |   Shape   |  Texture  |  Spatial  |
| ------------ | :-------: | :-------: | :-------: | :-------: | :-------: |
| LLM4GEN SDXL |   2.6B    |   73.29   |   57.34   |   67.86   |   22.59   |
| LI-DiT-1B    |    1B     | **74.08** | **59.34** | **69.59** | **27.57** |

Lumina-T2X directly introduces decoder-only LLMs into diffusion transformers, which is similar to our baseline setting.

Compared with the related works above, our method specifically identifies the issues and provides effective solutions. We will include these analyses in the final version.

- [1] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- [2] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models.
- [3] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
- [4] Grounded Text-to-Image Synthesis with Attention Refocusing.
- [5] High-Resolution Image Synthesis with Latent Diffusion Models.
- [6] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- [7] LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation
- [8] SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models
- [9] Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
- [10] UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion

Meta-review: This paper explores the role of LLMs in prompt encoding for diffusion models, especially in terms of the learned representation and effect of positional bias. The paper received 2 weak accepts, 1 borderline reject, and 1 borderline accept recommendations from reviewers. Positive points included interesting analysis, clear writing, good results, and extensive ablation studies. Negative points included limited novelty, some missing related work, some unclear comparisons, and lacking discussion on training and inference costs and scalability. Most of these concerns were addressed by the rebuttal. The reviewer who gave the borderline reject rating did not participate in the post-rebuttal discussions; however, the ACs felt that the reviewer’s major concerns were addressed by the rebuttal. Overall, after carefully considering the paper, rebuttal, and discussions, the ACs feel that the paper makes a positive contribution and recommend accept. Please incorporate the rebuttal points into the camera ready version.
Decision: Accept (poster)