Improving Long-Text Alignment for Text-to-Image Diffusion Models

## Abstract

Abstract The rapid advancement of text-to-image (T2I) diffusion models has enabled them to generate unprecedented results from given texts. However, as text inputs become longer, existing encoding methods like CLIP face limitations, and aligning the generated images with long texts becomes challenging. To tackle these issues, we propose LongAlign , which includes a segment-level encoding method for processing long texts and a decomposed preference optimization method for effective alignment training. For segment-level encoding, long texts are divided into multiple segments and processed separately. This method overcomes the maximum input length limits of pretrained encoding models. For preference optimization, we provide decomposed CLIP-based preference models to fine-tune diffusion models. Specifically, to utilize CLIP-based preference models for T2I alignment, we delve into their scoring mechanisms and find that the preference scores can be decomposed into two components: a text-relevant part that measures T2I alignment and a text-irrelevant part that assesses other visual aspects of human preference. Additionally, we find that the text-irrelevant part contributes to a common overfitting problem during fine-tuning. To address this, we propose a reweighting strategy that assigns different weights to these two components, thereby reducing overfitting and enhancing alignment. After fine-tuning 512 × 512 512 512 512\times 512 Stable Diffusion (SD) v1.5 for about 20 hours using our method, the fine-tuned SD outperforms stronger foundation models in T2I alignment, such as PixArt- α 𝛼 \alpha and Kandinsky v2.2. The code is available at https://github.com/luping-liu/LongAlign .

## 1 Introduction

Recent advancements in diffusion models (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ; ) have significantly enhanced text-to-image (T2I) generation (Schuhmann et al., ; Rombach et al., ; Saharia et al., ; Ramesh et al., ). While text-based conditioning provides flexibility and user-friendliness, current models struggle with long and complex text descriptions that often span multiple sentences and hundreds of tokens (Chen et al., ; Zheng et al., ). Effectively encoding such lengthy text conditions and ensuring precise alignment between text and generated images remains a critical challenge for generative models.

To encode text descriptions, contrastive pre-training encoders such as CLIP (Radford et al., ) are widely used in T2I diffusion models. However, as text length increases, the maximum token limit of CLIP becomes a significant constraint, making it infeasible to encode long descriptions. Recent works have explored large language model (LLM)-based encoders like T5 (Raffel et al., ), leveraging their ability to handle longer sequences (Saharia et al., ; Chen et al., ; Hu et al., ). Nevertheless, contrastive pre-training encoders retain a key advantage: their text encoders are specifically trained to align with images, potentially offering superior alignment between text representations and generated images (Saharia et al., ; Li et al., ).

Beyond encoding, current T2I diffusion models struggle to accurately follow long-text descriptions, often generating images that only partially reflect the intended details, as demonstrated in Figure [1](#S1.F1). Inspired by advances in aligning LLMs (Ouyang et al., ; Rafailov et al., ), one potential solution is preference optimization, which generates and utilizes preference feedback when only parts of a target are satisfied. Recent works have explored collecting human preferences from T2I users and leveraging them to train preference models (Kirstain et al., ; Wu et al., ; ), enabling preference optimization in T2I diffusion models. However, since these models are typically fine-tuned from CLIP, they face the same token limit constraints. Moreover, existing human preferences blend text alignment with visual factors such as photorealism or aesthetics, which only partially support the goal of accurately aligning long and detailed text descriptions.

Figure: Figure 1: Generation results of our long Stable Diffusion and baselines. We highlight three key facts for each prompt and provide the evaluation results at the end. In each evaluation line, the four group results are arranged in order of model presentation, with S representing SD-1.5, and so on. Additionally, three $\checkmark$ or $\times$ maintain the order of the key facts corresponding to each prompt.
Refer to caption: /html/2410.11817/assets/x1.png

In this paper, to support long-text inputs for the two scenarios mentioned, we explore segment-level encoding, which involves dividing lengthy texts into shorter segments (e.g., sentences), encoding each one separately, and then merging the results for subsequent tasks. The main challenge is effectively combining these segment outputs to merge their diverse information without causing confusion. To address this, for text encoding in diffusion models, we opt for concatenating segment embeddings and explore optimal adjustments to the unintended repetition of special token embeddings from different segments. For preference models, we implement a segment-level preference training loss alongside segment-level encoding, allowing preference models to handle long inputs while generating both segment-level scores and an overall average score.

To enable preference optimization for long-text alignment, we analyze the scoring mechanisms of preference models and use these models (with segment-level encoding) to fine-tune T2I diffusion models. We find that the desired T2I alignment scores can be separated from general human preference scores. Specifically, preference scores can be divided into two components: a text-relevant part that assesses T2I alignment and a text-irrelevant part that evaluates other factors (e.g., aesthetics). In addition, we discover that the remaining text-irrelevant part leads to a common overfitting issue (Wu et al., ) during fine-tuning. To mitigate this, we propose a reweighting strategy that assigns different weights to these two components, which reduces overfitting and enhances alignment.

By integrating the methods mentioned above, we propose LongAlign. Our experiments show that segment-level encoding and training enable preference models to effectively handle long-text inputs and generate segment-level scores. Additionally, our preference decomposition method allows these models to produce T2I alignment scores alongside general preference scores. After fine-tuning the $512\times 512$ Stable Diffusion v1.5 (Rombach et al., ) using LongAlign for about 20 hours on 6 A100 GPUs, the obtained long Stable Diffusion (longSD) significantly improves alignment (see Figure [1](#S1.F1)), outperforming stronger foundation models in long-text alignment, such as PixArt-$\alpha$ (Chen et al., ) and Kandinsky v2.2 (Razzhigaev et al., ). Our contributions are as follows:

- •
We propose a segment-level encoding method that enables encoding models with limited input lengths to effectively process long-text inputs.
- •
We propose preference decomposition that enables preference models to produce T2I alignment scores alongside general preference, enhancing text alignment fine-tuning in generative models.
- •
After about 20 hours of fine-tuning, our longSD surpasses stronger foundation models in long-text alignment, demonstrating significant improvement potential beyond the model architecture.

## 2 Background

We provide an overview of diffusion models and T2I models. Next, we discuss preference models for fine-tuning T2I diffusion models, followed by an introduction to the reward fine-tuning process.

### 2.1 Diffusion model

Diffusion models (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ; ) construct a transformation from a Gaussian distribution to a target data distribution through a multi-step diffusion denoising process. Given a data distribution $x_{0}\sim q(x_{0})$, the diffusion process satisfies:

$$ $x_{t}=\alpha_{t}x_{0}+\beta_{t}\epsilon,$ (1) $$

where $\epsilon\sim\mathcal{N}(0,1)$, $t\in\{0,1,\dots,T\}$, $T$ is the maximum timestep, $\alpha_{t}^{2}+\beta_{t}^{2}=1$, and $\beta_{t}$ controls the speed of adding noise. The loss function is typically defined as follows:

$$ $\begin{split}\mathcal{L}_{t}=\mathbb{E}_{x_{0},\epsilon}||\epsilon-\epsilon_{\theta}(\alpha_{t}x_{0}+\beta_{t}\epsilon,t)||^{2},\end{split}$ (2) $$

where the model $\epsilon_{\theta}$ aims to predict the noise $\epsilon$ added to the clean data $x_{0}$.
Once $\epsilon_{\theta}$ is learned, according to DDIM (Song et al., ), the denoising process from $t=T$ to $t=0$ satisfies:

$$ $x_{t}^{*}=(x_{t}-\beta_{t}\epsilon_{\theta}(x_{t},t))/\alpha_{t},\quad x_{t-1}=\alpha_{t-1}x_{t}^{*}+\sqrt{\beta^{2}_{t-1}-\sigma^{2}_{t}}\epsilon_{\theta}(x_{t},t)+\sigma_{t}\epsilon,$ (3) $$

where $x_{T}\sim\mathcal{N}(0,1)$ and $x_{t}^{*}$ is the predicted clean image at timestep $t$. When $\sigma_{t}\equiv 0$, this is the DDIM denoising process. When $\sigma_{t}=$ $(\beta_{t-1}/\beta_{t})\sqrt{(1-\alpha^{2}_{t}/\alpha^{2}_{t-1})}$, this is the DDPM denoising process. In this paper, we choose $\sigma_{t}\equiv 0$. The corresponding DDIM process can be accelerated using numerical solvers such as PNDM (Liu et al., ) and DPM-Solver (Lu et al., ).

Stable Diffusion. Among different T2I diffusion models, Stable Diffusion (Rombach et al., ) plays a crucial role, which integrates a VAE (Kingma & Welling, ), CLIP, and a diffusion model $\epsilon_{\theta}$. During training, the pretrained VAE compresses the image $x$ into a latent $z$, while the pretrained CLIP model encodes the text prompt $p$. The diffusion model $\epsilon_{\theta}$ then attempts to fit this new distribution of latent variables $z$, conditioned on the text $p$. In the sampling process, the diffusion model $\epsilon_{\theta}$ first generates the latent $z$ based on the text prompts $p$ and then uses the VAE to decode the latent $z$ to obtain the final image $x$. For simplicity, we ignore the VAE in the following.

### 2.2 Preference model

Preference optimization (Ouyang et al., ; Rafailov et al., ) has shown its effectiveness in aligning LLMs with humans using preference feedback. To facilitate this for T2I diffusion models, prior work (Kirstain et al., ; Wu et al., ) fine-tunes these models with T2I preference models that evaluate human preferences for an image $x$ given a text prompt $p$, represented as $\mathcal{R}(x,p)$. We focus on preference models fine-tuned from pretrained CLIP models. To prepare the preference dataset for fine-tuning, prompts are paired with two generated images, and the preferred image is annotated. For $x_{i}$ preferred over $x_{j}$ (denoted as $i\succ j$), the preference training loss is:

$$ $\mathcal{L}_{i\succ j}=\frac{\exp(\mathcal{R}(x_{i},p))}{\exp(\mathcal{R}(x_{i},p))+\exp(\mathcal{R}(x_{j},p))},$ (4) $$

where $\mathcal{R}(x,p)=\mathcal{C}_{X}(x)\cdot\mathcal{C}_{P}(p)$ is the dot product of the image embeddings $\mathcal{C}_{X}(x)$ and the text embeddings $\mathcal{C}_{P}(p)$. After training, the preference model can be used to evaluate preferences or fine-tune generative models. Similar to CLIP, these preference models don’t support long-text inputs.

### 2.3 Reward fine-tuning

When fine-tuning T2I diffusion models with the preference models mentioned above, previous works (Black et al., ; Fan et al., ) typically treat preference models as reward signals defined by $\mathcal{L}_{r}=1-\mathbb{E}_{x_{T},p}\mathcal{R}(x_{0}^{*},p)$. However, fine-tuning the generator $\epsilon_{\theta}$ of diffusion models using these signals poses two challenges. First, backpropagating gradients through the entire iteration process is problematic. Second, overfitting is a concern. Previous works (Prabhudesai et al., ; Clark et al., ) have employed gradient checkpointing and LoRA (Hu et al., ) to facilitate gradient backpropagation, but these methods skip consecutive time steps at the beginning or end of the diffusion iteration to accelerate computation, which inevitably introduces optimization bias.

Recently, DRTune (Wu et al., ) shows a new method that allows for training on a uniform subset of all sampling steps. Specifically, when we apply the preference model $\mathcal{R}(x_{0}^{*},p)$ to calculate the reward signal $\mathcal{L}_{r}=1-\mathbb{E}_{x_{T},p}\mathcal{R}(x_{0}^{*},p)$, the corresponding gradient is as follows:

$$ $\partial_{\theta}(1-\mathbb{E}_{x_{T},p}\mathcal{R}(x_{0}^{*},p))\approx-\mathbb{E}_{x_{T},p}(\partial_{x_{0}^{*}}\mathcal{R})(x_{0}^{*},p)\left(\sum_{i=1}^{T-1}(\beta_{i-1}/\alpha_{i-1}-\beta_{i}/\alpha_{i})(\partial_{\theta}\epsilon_{\theta})(\text{sg}(x_{i}),i,p)\right).$ (5) $$

DRTune truncates the gradient of all $\{\text{sg}(x_{i})\}$, allowing it to optimize a uniform portion of the remaining gradient $\{(\partial_{\theta}\epsilon_{\theta})(\text{sg}(x_{i}),i,p)\}$. In this paper, we also utilize this as our gradient backpropagation method. However, overfitting remains a concern that cannot be fully addressed by early stopping alone, necessitating further analysis and additional solutions.

## 3 Segment-level text encoding

In this section, we introduce our segment-level encoding method, designed for the long-text encoding of both diffusion models and preference models. This approach divides the text into segments, encodes each one individually, and then merges the results. For diffusion models, we explore an embedding concatenation strategy during merging. For preference models, we present a segment-level loss alongside the new encoding to handle long inputs and generate detailed preference scores.

### 3.1 Text encoding of diffusion model

For text encoding in diffusion models, contrastive pre-training encoders like CLIP (Rombach et al., ) are commonly used. However, as the input text length increases, CLIP’s maximum token limitation becomes a significant issue. As a result, recent works (Saharia et al., ; Chen et al., ; Hu et al., ) have shifted to using LLMs like T5 instead of CLIP, overlooking CLIP’s distinct advantage (Radford et al., ) in image-text alignment pretraining. To leverage CLIP’s capabilities for long text, we introduce segment-level encoding. Our method divides the text into segments (e.g., sentences), encodes each one into embeddings like the original T2I diffusion models, and then merges these embeddings. Figure [9](#A1.F9) illustrates this segment-level text encoding.

In the merging process, we initially use direct embedding concatenation, which results in poor generated images (see Figure [10](#A1.F10)). This occurs because each segment includes special tokens such as <sot>, <eot>, and <pad> during individual encoding, leading to the unintended repeated presence of their embeddings in the concatenated embedding. To address this, we conduct ablation experiments on whether to keep, remove, or replace special token embeddings. The final embedding excludes the <pad> embeddings and introduces a new unique <pad*> embedding to meet the target length. It retains all <sot> embeddings while removing all <eot> embeddings, resulting in the format “<sot> Text1. <sot> Text2. … <pad*>”. More details about this experiment can be found in Appendix [A](#A1).

We can now use both CLIP and T5 for long-text encoding in T2I diffusion models. The next challenge is accurately representing all text segments in the generated images. To tackle this issue, we further fine-tune diffusion models with large-scale long texts paired with their corresponding images. We start with supervised training employing $\ell_{2}$ loss. However, we observe a clear optimization limit during training, with even the best version falling short of perfection (see the longSD(S) column in Table [2](#S5.T2)). This problem prompts us to explore additional preference optimization for long-text alignment alongside general supervised training, inspired by its success in LLMs (Ouyang et al., ).

### 3.2 Segment preference model

In the context of preference optimization in T2I diffusion models, previous studies (Kirstain et al., ; Wu et al., ) have employed CLIP-based human preference models to better align T2I diffusion models with human preferences. Since our objective—text alignment—is a key component of human preference, and CLIP-based structures are effective for large-scale training, we aim to adapt such preference models for our long-text alignment task.

The first step towards achieving the above goal is to enable CLIP-based preference models to accept long-text inputs. As shown in Section [2.2](#S2.SS2), these models introduce a new human preference training objective into the CLIP framework. However, they still have the same limited maximum input length and may struggle to reflect the varying impacts of different segments (e.g., sentences) with a single final score. To solve these problems, we split the long-text condition $p$ into $K$ segments, denoted as $\{\hat{p}_{k}\}_{k=1}^{K}$. Then, the new segment-level preference training loss is:

$$ $\mathcal{L}_{i\succ j}^{\text{seg}}=\frac{\exp(\sum_{k=i}^{K}\mathcal{R}(x_{i},\hat{p}_{k})/K)}{\exp(\sum_{k=i}^{K}\mathcal{R}(x_{i},\hat{p}_{k})/K)+\exp(\sum_{k=i}^{K}\mathcal{R}(x_{j},\hat{p}_{k})/K)}.$ (6) $$

The above loss enables weakly supervised learning (Zhou, ) during training, eliminating the need for additional segment-level annotations. In comparison to its single-value counterpart, the new preference model supports long-text inputs and generates more detailed segment-level scores $\{\mathcal{R}(x,\hat{p}_{k})\}_{k}$, along with an average final score $\sum_{k=i}^{K}\mathcal{R}(x_{i},\hat{p}_{k})/K=\mathcal{C}_{X}(x)\cdot\sum_{k=i}^{K}\mathcal{C}_{P}(\hat{p}_{k})/K$. Thus, computing this average score is equivalent to first segment-level encoding the text input and using the average embedding, denoted as $C_{P}^{\text{seg}}(p)=\sum_{k=i}^{K}\mathcal{C}_{P}(\hat{p}_{k})/K$, to compute the score. We refer to this score as Denscore, which will also function as a reward signal in the following section. If there is no confusion, we may omit the segment label in $C_{P}^{\text{seg}}(p)$.

[Uncaptioned image]: /html/2410.11817/assets/x2.png

Figure: Figure 2: (a) Schematic results for text embeddings. (b) Statistics of the projection scalar $\eta$ for three models. (c) The relationship between the original score and the two scores after decomposition using our Denscore. In the three score tables, the diagonal represents the scores for paired data, while the off-diagonal positions indicate the scores for unpaired data.

## 4 Preference decomposition

In this section, we explore preference optimization using the preference models mentioned above. We find that their preference scores comprise a text-relevant component and a text-irrelevant component, with the latter often causing overfitting in fine-tuning diffusion models. To address this, we propose a reweighting strategy for both components that reduces overfitting and enhances alignment.

### 4.1 Orthogonal decomposition

As shown in Section [2.2](#S2.SS2), the CLIP-based preference model $\mathcal{R}(x,p)=\mathcal{C}_{X}(x)\cdot\mathcal{C}_{P}(p)$ evaluates an image $x$ against a text $p$ with respect to human preferences. Some preferences concern whether the text condition $p$ is accurately represented in the image $x$, while others focus on visual factors such as photorealism and aesthetics, making them irrelevant to the text. We find a direct structural correspondence between these two types of preferences within the text embedding $\mathcal{C}_{P}(p)$. Specifically, different $\mathcal{C}_{P}(p)$ display a common direction, as illustrated in Figure [2](#S3.F2) (a). This common direction corresponds to text-irrelevant preferences, while the remainder reflects text-relevant preferences.

To support this statement, we use text embeddings from a large prompt dataset $\mathcal{P}$ to compute the common text-irrelevant direction: $\mathbf{V}:=\mathbb{E}_{p\sim\mathcal{P}}\mathcal{C}_{P}(p)/||\mathbb{E}_{p\sim\mathcal{P}}\mathcal{C}_{P}(p)||$. We then decompose the text embedding $\mathcal{C}_{P}(p)$ into two orthogonal parts: $\mathcal{C}_{P}^{\bot}(p)+\eta\mathbf{V}$, where $\eta=\mathcal{C}_{P}(p)\cdot\mathbf{V}$ is the projection scalar of $\mathcal{C}_{P}$ onto $\mathbf{V}$. For the value of $\eta$, we test CLIP, Pickscore and our Denscore, presenting the results in Figure [2](#S3.F2) (b). We observe that $\mathcal{C}_{P}$ exhibits a strong positive scalar projection onto $\mathbf{V}$ ($\eta>0.4$ for CLIP and $\eta>0.6$ for the others), forming a core in representation space. The presence of $\mathbf{V}$ is referred to as the cone effect (Gao et al., ; Liang et al., ), which results from both model initialization and contrastive training.

To better understand the differing roles of these two components in the preference score $\mathcal{R}(x,t)$, we experiment with a 5k image-text dataset. Figure [2](#S3.F2) (c) presents a score table illustrating the relationship between $\mathcal{C}_{X}(x)\cdot\mathcal{C}_{P}(p)$, $\mathcal{C}_{X}(x)\cdot\mathcal{C}^{\bot}_{P}(p)$ and $\mathcal{C}_{X}(x)\cdot\eta\mathbf{V}$. A more detailed version of this figure and real data statistics for the three scores can be found in Appendix [B.2](#A2.SS2). For the second score $\mathcal{C}_{X}(x)\cdot\mathcal{C}^{\bot}_{P}(p)$, our experiments in Section [5.2](#S5.SS2) and Appendix [B.3](#A2.SS3) show that it eliminates the influence of text-irrelevant components and focuses on measuring T2I alignment, which aligns with our objective. We refer to this aspect of Denscore as Denscore-O.

For the third score $\mathcal{C}_{X}(x)\cdot\eta\mathbf{V}=\eta(\mathcal{C}_{X}(x)\cdot\mathbf{V})$, the scalar $\eta$ is analyzed in the above two paragraphs. The remaining text-irrelevant term $\mathcal{C}_{X}(x)\cdot\mathbf{V}$ still needs further clarification. We provide images with low or high text-irrelevant scores in Figure [3](#S4.F3) to visualize the score standard. Notably, the three models assign low scores to images that share similar characteristics, marked by dull visuals and large blank spaces. Conversely, while the high-scoring images selected by CLIP lack distinct features, the two preference models show strong alignment. Their high-scoring images feature rich details, well-organized layouts, and overall higher quality. This highlights the role of training on Equation [4](#S2.E4), which provides a clear training objective—general human preference—for text-irrelevant scores. In contrast, CLIP’s original contrastive training does not incorporate this aspect.

Figure: Figure 3: Retrieval results with low or high text-irrelevant scores, using three CLIP-based models.
Refer to caption: /html/2410.11817/assets/x5.png

### 4.2 Gradient-reweight reward fine-Tuning

To use the preference model $\mathcal{R}(x,p)$ as a reward signal for fine-tuning T2I diffusion models, there are two main challenges. One challenge is backpropagating the gradient through the entire multi-step iteration process. This can be addressed by the method mentioned in Section [2.3](#S2.SS3). However, previous works still struggle with overfitting, where fine-tuned images exhibit similar patterns (see Figure [6](#S5.F6)). While these patterns may enhance reward scores, they often lead to unsatisfactory images. We find that this is primarily because the text-irrelevant parts $\mathbf{V}$ constitute a significant portion of the optimization direction. Specifically, we fine-tune the generator $\epsilon_{\theta}$ using the reward signal $\mathcal{L}_{r}=1-\mathbb{E}_{x_{T},p}\mathcal{R}(x_{0}^{*},p)$, with the gradient calculated as follows:

$$ $\begin{split}\partial_{\theta}(1-\mathbb{E}_{x_{T},p}\mathcal{R}(x_{0}^{*},p))=&-\partial_{\theta}\mathbb{E}_{x_{T},p}(\mathcal{C}_{P}(p)\cdot\mathcal{C}_{X}(x_{0}^{*}))=-\mathbb{E}_{p}(\mathcal{C}_{P}(p)^{T}\mathbb{E}_{x_{T}}\partial_{\theta}(\mathcal{C}_{X}(x_{0}^{*})))\\ =&-\mathbb{E}_{p}((\eta\mathbf{V}+\mathcal{C}_{P}^{\bot}(p))^{T}\mathbb{E}_{x_{T}}\partial_{\theta}(\mathcal{C}_{X}(x_{0}^{*}))),\end{split}$ (7) $$

where the item $\eta\mathbf{V}+\mathcal{C}_{P}^{\bot}(p)$ controls the gradient direction of $\epsilon_{\theta}$. According to Section [4.1](#S4.SS1), the text-irrelevant component $\mathbf{V}$ comprises a large portion of the entire item, overwhelming the gradient and producing similar output patterns regardless of the text input $p$. To mitigate this overfitting problem, we fine-tune the generator $\epsilon_{\theta}$ using a reweighted gradient:

$$ $\partial_{\theta}(1-\mathbb{E}_{x_{T},p}\mathcal{R}(x_{0}^{*},p))\approx-\mathbb{E}_{p}((\omega(\eta\mathbf{V})+\mathcal{C}_{P}^{\bot}(p))^{T}\mathbb{E}_{x_{T}}\partial_{\theta}(\mathcal{C}_{X}(x_{0}^{*}))),$ (8) $$

where $\omega$ is the reweighting factor for the common direction $\mathbf{V}$. This addresses the overfitting problem mentioned above. Additionally, this analysis clarifies why the original CLIP is ineffective for reward fine-tuning (see Figure [6](#S5.F6)). The issue arises because its text-irrelevant component $\mathbf{V}$ is not well-trained (see Section [4.1](#S4.SS1)), resulting in an undefined optimization direction.

By combining the methods from the two sections above, we arrive at our complete LongAlign method. Detailed algorithms are available in Appendix [A](#A1) and [C](#A3).

## 5 Experiment

In this section, we first provide our experimental setup, including models, training strategies, and evaluation metrics. We then present the results of our segment preference models. Next, we detail our long-text encoding results for diffusion models using various encoding methods. The advantages of gradient reweighting fine-tuning across different reward signals are then discussed. Finally, we present the generation results using our entire LongAlign alongside those of the baselines.

### 5.1 Experimental setup

Model. Our experiments cover three types of models: text encoders, Unets, and preference models. Specifically, we utilize the pretrained CLIP and T5 models as our text encoders. To align the embedding dimensions, we append a two-layer MLP to T5’s output, following LaVi-Bridge (Zhao et al., ). The Unet is derived from the pretrained Stable Diffusion v1.5(^1^11[https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5)). Instead of full fine-tuning, we fine-tune the Unets using LoRA with a rank of 32 and update both the ResNet blocks and attention blocks within the Unets. For the preference model, we select two pretrained models: Pickscore and HPSv2. Additionally, we introduce our new segment-level preference model, Denscore. All three models are fine-tuned from pretrained CLIP models.

Training. (1) For training the Unet, we utilize a dataset of approximately 2 million images, including 500k from SAM (Kirillov et al., ), 100k from COCO2017 (Lin et al., ), 500k from LLaVA (a subset of the LAION/CC/SBU dataset), and 1 million from JourneyDB (Sun et al., ). We randomly reserve 5k images for evaluation. All images are recaptioned using LLaVA-Next (Liu et al., ) or ShareCaptioner (Chen et al., ) and resized to $512\times 512$ pixels. We optimize the model using the AdamW optimizer with a learning rate of $3\times 10^{-5}$, a 2k-step warmup, and a total batch size of 192. Training is conducted on 6 A100-40G GPUs for 30k steps over 12 hours. (2) For the reward fine-tuning (RFT) stage of the Unet, we use the same settings as before but with a batch size of 96 and 4k total training steps over 8 hours. (3) For training the segment preference model, we use the same settings as for Pickscore (Kirstain et al., ), employing CLIP-H on Pickscore’s training data, along with LLaVA-Next captions and our new segment-level loss function. More details about training the preference model can be found in Appendix [B.1](#A2.SS1).

Evaluation. We evaluate our methods using the FID, Denscore and Denscore-O metrics on the 5k-image evaluation dataset. Additionally, we employ GPT-4o (OpenAI, ) to evaluate 1k images against baselines, mitigating the risk of overfitting to Denscore. The GPT-4o evaluation template can be found in Appendix [D](#A4). All our experiments employ UniPC (Zhao et al., ) with 25-step sampling, maintaining a consistent classifier-free guidance factor as per the original papers.

**Table 1: R@1 results for 5k text-to-image retrieval using different CLIP-based models.**
|  | CLIP-H | HPSv2 | Pickscore | Denscore |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Single | Average | Single | Average | Single | Average | Single | Average |
| $\mathcal{C}_{P}(p)$ | 86.10 | 80.40 | 42.34 | 16.72 | 54.00 | 31.84 | 83.96 | 75.90 |
| $\mathcal{C}_{P}^{\bot}(p)$ | 85.80 | 85.14 | 67.94 | 64.28 | 67.60 | 64.00 | 87.24 | 91.86 |

Figure: Figure 4: FID and Denscore results for diffusion models with different text encodings.
Refer to caption: /html/2410.11817/assets/x6.png

### 5.2 Segment preference model

Here, we compare the pretrained CLIP and single-value preference models: Pickscore and HPSv2, with our new segment-level preference model, Denscore. According to Section [4.1](#S4.SS1), the text-irrelevant embedding $\mathbf{V}$ reflects the text-irrelevant preference, while the text-relevant embedding $\mathcal{C}_{P}^{\bot}(p)$ reflects the T2I alignment, which is our objective.

To assess the ability of the text-relevant part $\mathcal{C}_{P}^{\bot}(p)$, we provide the R@1 retrieval accuracy for the four models on the 5k evaluation dataset in Table [1](#S5.T1). Here, we either utilize the embedding encoding from the first 77 tokens in a single-pass approach (Single) or take the average of segment-level embeddings $\mathcal{C}_{P}^{\text{seg}}(p)$ (Average). The final score is computed using the image embedding $\mathcal{C}_{X}(x)$ with either the full text embedding $\mathcal{C}_{P}$ or solely the text-relevant part $\mathcal{C}_{P}^{\bot}(p)$. We find that: (1) Across all preference models, the text-relevant embedding outperforms the original embedding, as it eliminates the impact of text-irrelevant factors. (2) Only Denscore with segment-level training yields better segment-level retrieval accuracy, while the other models don’t benefit from, and are confused by, the extra information provided by averaging multiple embeddings. (3) Although T2I alignment is only a partial optimization objective in Denscore, the Denscore model outperforms CLIP. This highlights the importance of segment-level encoding strategies and preference decomposition.

In Appendix [B.3](#A2.SS3), to better evaluate Denscore’s performance under varying text lengths, we employ different maximum sentence prompts to obtain R@1 retrieval accuracy. Additionally, we conduct an experiment in the same appendix to identify specific misaligned segments in long inputs, showing that segment-level scoring provides more detailed information.

**Table 2: FID and Denscore results for $512\times 512$ image generation using different foundation models. PlayG-2 is from Li et al. (), and KanD-2.2 is from Razzhigaev et al. ().**
| Model | SD-1.5 | SD-2.1 | PlayG-2 | PixArt-$\alpha$ | KanD-2.2 | longSD (S) | longSD (S+R) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FID-5k | 24.96 | 25.80 | 23.92 | 22.36 | 20.04 | 20.09 | 19.63/24.28 |
| Denscore-O | 29.20 | 30.15 | 28.80 | 33.48 | 33.30 | 31.29 | 32.83/35.26 |
| Denscore | 20.29 | 20.91 | 21.22 | 22.78 | 22.70 | 21.72 | 22.74/23.79 |

Figure: Figure 5: FID and Denscore results for diffusion models using different gradient reweighting factors.
Refer to caption: /html/2410.11817/assets/x9.png

### 5.3 Long-text encoding

Here, we provide FID and Denscore results for supervised fine-tuning (SFT), comparing various text encoding methods: CLIP with concatenation (CLIP-cat), T5 with an additional two-layer MLP (T5-mlp)(^2^22We utilize LaVi-bridge’s (Zhao et al., ) pretrained MLP, which means we only include the short-to-long fine-tuning.), and a combination of CLIP and T5 (CLIP+T5). The results in Figure [4](#S5.F4) show that CLIP-cat and T5-mlp perform similarly, while the CLIP+T5 model significantly outperforms them. This suggests that the CLIP+T5 model is preferable, as it leverages the strengths of both CLIP’s image-text paired pretraining and T5’s pure long-text encoding capabilities. This finding is consistent with the results of previous works (Saharia et al., ; Li et al., ), while these earlier models still face input length limitations due to CLIP’s maximum token count.

### 5.4 Reward fine-tuning

To simplify and accelerate ablation studies of reward fine-tuning (RFT), we leverage both LCM-LoRA (Song et al., ; Luo et al., ) and DRTune (Wu et al., ) to speed up fine-tuning using only CLIP-cat encoding for optimal strategy identification in this subsection.

Weighting Factor. In Figure [5](#S5.F5), we provide the FID and Denscore results for different gradient reweight factors $\omega$ and training steps. According to the first three experiments with different reweight factors, the FID results exhibit a parabolic shape, with $\omega=0.3$ achieving the best FID results. In the last three experiments with different training steps, only $\omega=0.3$ maintains relatively stable FID results while simultaneously improving both Denscore and Denscore-O results. In contrast, the other two options improve only one of the Denscore or Denscore-O metrics, accompanied by a significant increase in FID, indicating an apparent overfitting problem. Additionally, please note that the optimal value of $\omega$ can vary depending on the model and training strategy used.

Figure: Figure 6: Generation results using different reward signals, with and without gradient reweighting. The corresponding text conditions can be found in Appendix [F](#A6).

Reward Signal. In Figure [6](#S5.F6), we provide visual results showing the benefits of gradient reweighting on reward signals from various preference models, including CLIP, HPSv2, Pickscore, and our Denscore. Notably, this method also partially addresses the limitation of using reward signals from pretrained CLIP. However, CLIP cannot yet outperform preference models because its text-irrelevant component $\mathbf{V}$ is not well-trained, which aligns with our analysis in Section [4.2](#S4.SS2).

### 5.5 Generation result

Figure: Figure 7: GPT-4o evaluation results of T2I alignment across different models.
Refer to caption: /html/2410.11817/assets/x16.png

Foundation Model. In this subsection, we apply our entire LongAlign approach to train our long Stable Diffusion (longSD) and compare it against other baselines. Here, we present the results of SFT (S) at 28k steps and SFT+RFT (S+R) with $\omega=0.3$, at 1.25k and 3.75k steps. As shown in Table [2](#S5.T2), SFT+RFT clearly outperforms both the original SD-1.5 and the basic SFT version. Compared to other advanced foundation models, our longSD model surpasses them in terms of FID and Denscore metrics. Furthermore, we use GPT-4o (OpenAI, ) to evaluate 1k longSD(S+R) results, comparing them to baselines and mitigating the risk of overfitting. The new results (see Figure [7](#S5.F7)) are consistent with our previous scores. We also provide visualizations in Figure [8](#S5.F8). All these findings highlight the effectiveness of our method for generating high-quality images from long texts, demonstrating significant potential beyond altering the model structure.

Figure: Figure 8: The generation results show a comparison of performance before and after our fine-tuning on SD 1.5. The corresponding text conditions are provided in Appendix [F](#A6).
Refer to caption: /html/2410.11817/assets/x17.png

**Table 3: Evaluation for comparison in the P2I diffusion framework.**
| Method | 768 | 1024 |  |  |
| --- | --- | --- | --- | --- |
| P2I | +ours | P2I | +ours |  |
| FID-5k | 20.36 | 21.60 | 19.78 | 20.84 |
| Denscore-O | 34.45 | 38.71 | 34.78 | 38.51 |
| Denscore | 23.43 | 25.39 | 23.47 | 25.41 |
| GPT-4o | 240 | 583 | 289 | 536 |

Alignment Strategy. In addition to varying model structures, there are other methods that improve T2I alignment. Some of these approaches incorporate the assistance of LLMs, such as Ranni (Feng et al., ) and RPG-Diffusers (Yang et al., ), while others employ improved training strategies, such as Paragraph-to-Image (P2I) diffusion (Wu et al., ). LLM-based methods that involve additional LLM assistance increase computational requirements and encounter significant out-of-distribution (OOD) issues with long-text inputs. More details on OOD issues can be found in Appendix [E](#A5). Regarding improved training strategies, our approach is orthogonal to P2I diffusion, so we compare the original P2I diffusion with its fine-tuned version using our method. The results in Table [3](#S5.T3) also show clear improvements in terms of FID, Denscore, and GPT-4o evaluations.

## 6 Related work

Diffusion-based Generation. The popularity of diffusion models has surged, driven by breakthroughs in fast sampling methods (Song et al., ; Lu et al., ; Zhang & Chen, ) and text-conditioned generation (Saharia et al., ; Chen et al., ). Moreover, cascaded and latent space models (Ho et al., ; Rombach et al., ) have enabled the creation of high-resolution images. These advancements have also unlocked various applications beyond T2I generation, including the ability to create content with a consistent style (Hertz et al., ) and image editing tools (Hertz et al., ; Brooks et al., ). Furthermore, advancements in foundation models like U-ViT (Bao et al., ) and DiT (Peebles & Xie, ) suggest even greater potential.

Text-to-Image Evaluation. Traditional metrics, such as Inception Score (IS) (Salimans et al., ) and Fréchet Inception Distance (FID) (Heusel et al., ), have limitations in evaluating the quality of T2I generation. To address these limitations, three approaches have emerged. One approach employs Perceptual Similarity Metrics like LPIPS (Zhang et al., ), which utilize pretrained models to assess image similarity from a human perspective. Another approach involves using detection models (Huang et al., ) to extract and analyze key object alignments. A third approach fine-tunes human preference models (Xu et al., ; Kirstain et al., ; Wu et al., ) on datasets of human preferences for images generated from specific prompts, effectively turning them into proxies for human evaluation.

Reward Fine-tuning. Training text-to-image models with a reward signal can be effective in targeting specific outcomes. Two primary approaches have emerged. One leverages reinforcement learning to optimize rewards that are difficult to calculate using traditional methods, such as DPOK (Fan et al., ) and DDPO (Black et al., ). Another approach, exemplified by DiffusionCLIP (Kim et al., ), DRaFT (Clark et al., ) and AlignProp (Prabhudesai et al., ), involves backpropagation through sampling. This method leverages human preferences and other differentiable reward signals to optimize diffusion models for specific targets. Recently, DRTune (Wu et al., ) further improves training speed by stopping the gradient of the diffusion model’s input.

## 7 Discussion

In this paper, we propose LongAlign to enhance long-text to image generation. We examine segment-level text encoding strategies for processing long-text inputs in both T2I diffusion models and preference models. In addition, we enhance the role of preference models by analyzing their structure and decomposing them into text-relevant and text-irrelevant components. During reward fine-tuning, we propose a gradient reweighting strategy to reduce overfitting and enhance alignment. In our experiments, we utilize the classical SD-1.5 and effectively fine-tune it to outperform stronger foundation models, demonstrating significant potential beyond designing new model structures.

The limitation of this paper is that our method still does not fully capture the generation of the exact number of entities specified by prompts, partly due to the constraints of CLIP. In the future, we will investigate more powerful training strategies beyond CLIP-based models.

## Appendix A Concatenation strategy

For the CLIP encoder in T2I diffusion models, we first show the pipeline of our new segment-level encoding in Figure [9](#A1.F9). In this pipeline, segment-level encoding and concatenation may seem straightforward, but the optimal concatenation strategy remains unclear, as shown in Figure [10](#A1.F10). This is because each segment contains special tokens, such as <sot>, <eot>, and <pad>, leading to their corresponding embeddings’ unintended repeated presence in the final concatenated embedding. This raises the question of whether to retain, remove, or replace them in the final embeddings.

For <pad> tokens in each segment, we omit them in the final embeddings since they lack alignment information. However, we sometimes need to introduce a new <pad> embedding to ensure aligned token sequence lengths. To address this, we assign a unique embedding to each required <pad> token’s position. Specifically, our unique <pad> embedding is the average value of all <pad> tokens of an empty sentence, which we denote as <pad*>. For <sot> and <eot> tokens, we then experiment with pretrained diffusion models to find the optimal strategy. As shown in Figure [10](#A1.F10), our results indicate that the optimal approach is to keep all <sot> token embeddings and remove all <eot> token embeddings. Therefore, our final embeddings used in this paper take the form “<sot> Text1. <sot> Text2. … <pad*>”.

Figure: Figure 9: The visualization of our new segment-level text encoding for diffusion models is presented.
Refer to caption: /html/2410.11817/assets/x18.png

Figure: Figure 10: Generation results under different embedding concatenation strategies.
Refer to caption: /html/2410.11817/assets/x19.png

## Appendix B Segment preference model

Here, we provide a detailed analysis of our segment preference models with respect to training, visualization, and evaluation.

### B.1 Training

We train a segment preference model by incorporating long and detailed text conditions generated by LLaVA-Next alongside our segment-level preference loss function. Based on the analysis in Section [4.1](#S4.SS1), we choose to combine the segment-level loss for refining text-relevant aspects with the original loss for improving aspects unrelated to text, such as aesthetics. The loss function $\mathcal{L}^{\text{seg-a}}_{i\succ j}$ (where “a” denotes addition) is:

$$ $\begin{split}\mathcal{L}^{\text{seg-a}}_{i\succ j}=&\mathbb{E}_{x,\{\hat{p}_{k}\}}\sigma\left(\sum_{k=1}^{K}(\mathcal{C}_{X}(x_{i})\cdot\mathcal{C}_{P}(\hat{p}_{k})/K)-\sum_{k=1}^{K}(\mathcal{C}_{X}(x_{j})\cdot\mathcal{C}_{P}(\hat{p}_{k})/K)\right)+\\ &\mathbb{E}_{x,p}\sigma(\mathcal{C}_{X}(x_{i})\cdot\mathcal{C}_{P}(p)-\mathcal{C}_{X}(x_{j})\cdot\mathcal{C}_{P}(p)),\end{split}$ (9) $$

where $\sigma(x)=\frac{1}{1+e^{-x}}$ is the sigmoid function and $\frac{e^{m}}{e^{m}+e^{n}}=\sigma(m-n)$. In addition, we substitute $\mathcal{C}_{P}(\hat{p}_{k})$ with $\mathcal{C}_{P}^{\bot}(\hat{p}_{k})$ to help the new segment-level loss focus on the T2I alignment part and avoid influencing the text-irrelevant part. The new loss function $\mathcal{L}^{\text{seg-o}}_{i\succ j}$ (where “o” means orthogonal) is:

$$ $\begin{split}\mathcal{L}^{\text{seg-o}}_{i\succ j}=&\mathbb{E}_{x,\{\hat{p}_{k}\}}\sigma\left(\sum_{k=1}^{K}(\mathcal{C}_{X}(x_{i})\cdot\mathcal{C}_{P}^{\bot}(\hat{p}_{k})/K)-\sum_{k=1}^{K}(\mathcal{C}_{X}(x_{j})\cdot\mathcal{C}_{P}^{\bot}(\hat{p}_{k})/K)\right)+\\ &\mathbb{E}_{x,p}\sigma(\mathcal{C}_{X}(x_{i})\cdot\mathcal{C}_{P}(p)-\mathcal{C}_{X}(x_{j})\cdot\mathcal{C}_{P}(p)),\end{split}$ (10) $$

where $\{\hat{p}_{k}\}$ represents the segments split from the long text generated by LLava-Next, and $p$ is the original short text. If we prioritize alignment, $\mathcal{L}^{\text{seg-a}}_{i\succ j}$ is the better choice; if we want to balance both alignment and aesthetics, $\mathcal{L}^{\text{seg-o}}_{i\succ j}$ is the preferable option. In this paper, we choose $\mathcal{L}^{\text{seg-o}}_{i\succ j}$ to train our new segment preference model.

### B.2 Visualization

After training the new segment preference models, we want to analyze the behavior of three scores $\mathcal{C}_{X}(x)\cdot\mathcal{C}_{P}(p)$, $\mathcal{C}_{X}(x)\cdot\mathcal{C}^{\bot}_{P}(p)$ and $\mathcal{C}_{X}(x)\cdot\eta\mathbf{V}$. We are interested in the relationships among them, and additionally, in the differences in these scores between different CLIP-based models. Here, we choose CLIP, Pickscore and our Denscore as our experimental targets, as they all use the same model structure, while the last two are fine-tuned on Pickscore’s human preference dataset. We present the results in Figures [11](#A2.F11) and [12](#A2.F12). The first figure uses 20 data pairs to visualize the relationships and differences, while the second figure displays the actual statistics based on 5,000 data pairs.

For the original score $\mathcal{C}_{X}(x)\cdot\mathcal{C}_{P}(p)$, only the original CLIP behaves in such a way that its score can be close to zero when the image and text inputs are unpaired, while the other two models still assign relatively high scores to such unpaired inputs. This is because the preference models assess the image not only based on text alignment but also on other purely visual factors, such as aesthetics.

For the text-relative score $\mathcal{C}_{X}(x)\cdot\mathcal{C}^{\bot}_{P}(p)$, after removing the influence of the text-irrelevant part, all three models provide nearly zero scores for unpaired input data, which supports our analysis that this score focuses on the T2I alignment and explains why this score achieves the best retrieval results for preference models, as shown in Table [1](#S5.T1).

For the third score $\mathcal{C}_{X}(x)\cdot\eta\mathbf{V}$, we have analyzed the scalar $\eta$ in the main paper, which is determined entirely by the text input and is image-irrelevant. Here, we focus on the text-irrelevant score $\mathcal{C}_{X}(x)\cdot\mathbf{V}$. We find that this score is strongly positive for the two preference models. This corresponds to pure visual factors in preference, as shown in Figure [3](#S4.F3). On the other hand, for the original CLIP, this score is nearly zero. This indicates that the common direction of text embedding is almost orthogonal to the image embeddings, which do not contribute to the final score of CLIP.

Figure: Figure 11: Logit results for different models, both before and after orthogonal decomposition.
Refer to caption: /html/2410.11817/assets/x20.png

Figure: Figure 12: The real data statistics for the diagonal paired data and the off-diagonal unpaired data.
Refer to caption: /html/2410.11817/assets/x21.png

### B.3 Evaluation

To better support the above analysis, we design two additional experiments to test the effectiveness of our score alongside the experiment in Section [5.2](#S5.SS2).

Firstly, to evaluate Denscore’s performance under varying text lengths, we employ different maximum sentence prompts to obtain R@1 retrieval accuracy, as shown in Table [4](#A2.T4). We find that our Denscore, using segment-level training, consistently outperforms the others, except in the one-sentence setting. In this one-sentence setting, our segment-level training becomes less meaningful, but our results still outperform other existing preference models.

Additionally, to identify specific misaligned segments in long inputs, we conduct an experiment, illustrated in Figure [13](#A2.F13), to show that segment-level scoring provides more detailed information. In some cases, certain segments align better with the first image, while other segments align better with the second image. This makes the overall score relatively meaningless, while segment-level scores continue to perform well.

**Table 4: R@1 results for 5k text-to-image retrieval with varying maximum numbers of sentences.**
| max number | 1 | 2 | 3 | 4 | 6 | 8 |
| --- | --- | --- | --- | --- | --- | --- |
| CLIP | 53.06 | 70.90 | 76.70 | 79.62 | 83.00 | 84.12 |
| HPSv2 | 41.86 | 53.66 | 56.48 | 59.14 | 62.58 | 63.96 |
| Pickscore | 42.34 | 53.86 | 57.56 | 60.22 | 63.60 | 63.54 |
| Denscore | 52.72 | 72.70 | 78.78 | 83.10 | 88.16 | 89.94 |

Figure: Figure 13: Identifying the best-aligned image and segment pairs using segment-level preference scores.
Refer to caption: /html/2410.11817/assets/x24.png

## Appendix C Decomposed preference optimization

Here, we provide the pseudocode in Algorithm [1](#alg1) for the entire decomposed preference optimization pipeline discussed in this paper.

Figure: Algorithm 1 Decomposed Preference Optimization for T2I Diffusion Models

## Appendix D GPT-4o evaluation

The input for evaluation consists of a prompt and two images generated for this prompt using different models. We utilize the multimodal evaluation capabilities of GPT-4o and employ the following Python code for evaluation:

## Appendix E Out-of-distribution problem

Some of these approaches incorporate the assistance of LLMs, such as Ranni and RPG-Diffusers. LLM-based methods that involve additional LLM assistance increase computational requirements and encounter significant out-of-distribution (OOD) issues with long-text inputs.

There are several reasons why existing LLM-based methods struggle with long-text inputs. Long texts contain more facts than shorter ones, and LLMs rely on injecting different subprompts into specific subareas. Some facts may be disjoint, while others overlap (e.g., detailed descriptions of a single object), complicating the separation of information in pixel space. Additionally, some LLM-based methods fine-tune the LLM model that may not include datasets with long and detailed inputs. As a result, these models often cannot manage such situations for generating effective layout plans to assign different subprompts across different areas.

While current LLM-based methods are not fully effective, we believe this OOD problem can be resolved in the future. Additionally, LLM-based methods complement our approach: while we concentrate on training more powerful foundation models, these techniques can further enhance results during the sampling stage.

Here are some examples of OOD problems:

Ranni. In our experiment with Ranni, approximately half of the prompts resulted in errors before generating the final output. We selected examples from the remaining successful prompts. The example prompt is generated from the first image in Figure [14](#A5.F14). Although Ranni generates the layout initially, it creates too many subareas that do not contribute clearly to the final result, as shown in the second image in Figure [14](#A5.F14).

Figure: Figure 14: The reference image and the corresponding generated image using Ranni.
Refer to caption: /html/2410.11817/assets/data/ood-1.jpg

The complete input prompt is “The image is a digital artwork featuring a female character standing in a rocky environment. The character is dressed in a fantasy-style armor with a predominantly dark color scheme, highlighted by accents of blue and purple. The armor includes a corset-like bodice, a skirt, and arm guards, all adorned with intricate designs and patterns. The character’s hair is styled in a high ponytail, and she has a serious expression on her face. The armor is illuminated by a purple glow, which appears to emanate from the character’s body, creating a contrast against the darker elements of the armor and the surrounding environment. The glow also casts a soft light on the character’s face and the armor, enhancing its details and textures. The background consists of a rocky landscape with a purple hue, suggesting a magical or otherworldly setting. The rocks are jagged and uneven, with some areas appearing to be on fire, adding to the dramatic and intense atmosphere of the image. There are no visible texts or logos in the image, and the style of the artwork is realistic with a focus on fantasy elements. The image is likely intended for a gaming or fantasy-themed context, given the character’s attire and the overall aesthetic.”

RPG-Diffusers. For RPG-Diffusers, the example prompt is generated from the first image in Figure [15](#A5.F15). However, they assign two subareas: the left side and the right side of the image. Using these subprompts—“The black leather sofa is stylishly centered in the frame, showcasing its plush texture and elegant design, the distinct cushions maintain the streamlined look of the piece, offering comfort against the stark contrast of the white background.” and “The three luxurious black cushions are artfully arranged, emphasizing the sofa’s inviting nature while seamlessly blending into the cohesive monochrome theme of the setting.”—results in a repetition of “soft” in the generated image, as shown in the second image in Figure [15](#A5.F15).

Figure: Figure 15: The reference image and the corresponding generated image using RPG-Diffusers.
Refer to caption: /html/2410.11817/assets/data/ood-3.jpg

The complete input prompt is “The image presents a scene dominated by a black leather sofa, which is the central object in the frame. The sofa, with its three cushions, is positioned against a stark white background, creating a striking contrast. The cushions, like the sofa, are black and appear to be made of leather, suggesting a uniform color scheme throughout the piece. The sofa is designed with a wooden frame, adding a touch of warmth to the otherwise monochrome setting. The frame is visible on both the front and back of the sofa, providing stability and support. The arms of the sofa, like the frame, are made of wood, maintaining the overall aesthetic of the piece. The sofa is set against a white background, which accentuates its black color and wooden frame, making it the focal point of the image. There are no other objects in the image, and no text is present. The relative position of the sofa is central, with ample space around it, further emphasizing its importance in the image. The image does not depict any actions, but the stillness of the sofa suggests a sense of tranquility.”

## Appendix F Text condition for visual result

In this section, we provide the text conditions for the visual results in Figure [1](#S1.F1) and Figure [6](#S5.F6). The text conditions are as follows:

The text conditions for Figure [1](#S1.F1) are as follows:

- 1.
The image presents a 3D rendering of a horse, captured in a profile view. The horse is depicted in a state of motion, with its mane and tail flowing behind it. The horse’s body is composed of a network of lines and curves, suggesting a complex mechanical structure. This intricate design is further emphasized by the presence of gears and other mechanical components, which are integrated into the horse’s body. The background of the image is a dark blue, providing a stark contrast to the horse and its mechanical components. The overall composition of the image suggests a blend of organic and mechanical elements, creating a unique and intriguing visual.
- 2.
The image presents a close-up view of a human eye, which is the central focus. The eye is surrounded by a vibrant array of flowers, predominantly in shades of blue and purple. These flowers are arranged in a semi-circle around the eye, creating a sense of depth and perspective. The background of the image is a dark blue sky, which contrasts with the bright colors of the flowers and the eye itself. The overall composition of the image suggests a theme of nature and beauty.
Folder 1 Image 836
- 3.
The image presents a detailed illustration of a submarine, which is the central focus of the artwork. The submarine is depicted in a three-quarter view, with its bow facing towards the right side of the image. The submarine is constructed from wood, giving it a rustic and aged appearance. It features a dome-shaped conning tower, which is a common feature on submarines, and a large propeller at the front. The submarine is not alone in the image. It is surrounded by a variety of sea creatures, including fish and sharks, which are swimming around it. These creatures add a sense of life and movement to the otherwise static image of the submarine. The background of the image is a light beige color, which provides a neutral backdrop that allows the submarine and the sea creatures to stand out. However, the background is not devoid of detail. It is adorned with various lines and text, which appear to be a map or a chart of some sort. This adds an element of intrigue to the image, suggesting that the submarine might be on a mission or an expedition. Overall, the image is a detailed and intricate piece of art that captures the essence of a submarine voyage, complete with the submarine, the sea creatures, and the map in the background. It’s a snapshot of a moment in time, frozen in the image, inviting the viewer to imagine the stories and adventures that might be taking place beneath the surface of the water.
- 4.
In the image, there’s a charming scene featuring a green frog figurine. The frog, with its body painted in a vibrant shade of green, is the main subject of the image. It’s wearing a straw hat, adding a touch of whimsy to its appearance. The frog is positioned in front of a white window, which is adorned with a green plant, creating a harmonious color palette with the frog’s body. The frog appears to be looking directly at the camera, giving the impression of a friendly encounter. The overall image exudes a sense of tranquility and simplicity.
- 5.
The image portrays a female character with a fantasy-inspired design. She has long, dark hair that cascades down her shoulders. Her skin is pale, and her eyes are a striking shade of blue. The character’s face is adorned with intricate gold and pink makeup, which includes elaborate patterns and designs around her eyes and on her cheeks. Atop her head, she wears a crown made of gold and pink roses, with the roses arranged in a circular pattern. The crown is detailed, with each rose appearing to have a glossy finish. The character’s attire consists of a gold and pink dress that is embellished with what appears to be feathers or leaves, adding to the fantasy aesthetic. The background of the image is dark, which contrasts with the character’s pale skin and the bright colors of her makeup and attire. The lighting in the image highlights the character’s features and the details of her makeup and attire, creating a dramatic and captivating effect. There are no visible texts or brands in the image. The style of the image is highly stylized and artistic, with a focus on the character’s beauty and the intricate details of her makeup and attire. The image is likely a digital artwork or a concept illustration, given the level of detail and the fantastical elements present.
- 6.
The image captures a scene of a large, modern building perched on a cliff. The building, painted in shades of blue and gray, stands out against the backdrop of a cloudy sky. The cliff itself is a mix of dirt and grass, adding a touch of nature to the otherwise man-made structure. In the foreground, a group of people can be seen walking along a path that leads up to the building. Their presence adds a sense of scale to the image, highlighting the grandeur of the building. The sky above is filled with clouds, casting a soft, diffused light over the scene. This light enhances the colors of the building and the surrounding landscape, creating a visually striking image. Overall, the image presents a harmonious blend of architecture and nature, with the modern building seamlessly integrated into the natural landscape.

The text conditions for Figure [6](#S5.F6) are as follows:

- 1.
The image is a digital artwork featuring a female character standing in a rocky environment. The character is dressed in a fantasy-style armor with a predominantly dark color scheme, highlighted by accents of blue and purple. The armor includes a corset-like bodice, a skirt, and arm guards, all adorned with intricate designs and patterns. The character’s hair is styled in a high ponytail, and she has a serious expression on her face. The armor is illuminated by a purple glow, which appears to emanate from the character’s body, creating a contrast against the darker elements of the armor and the surrounding environment. The glow also casts a soft light on the character’s face and the armor, enhancing its details and textures. The background consists of a rocky landscape with a purple hue, suggesting a magical or otherworldly setting. The rocks are jagged and uneven, with some areas appearing to be on fire, adding to the dramatic and intense atmosphere of the image. There are no visible texts or logos in the image, and the style of the artwork is realistic with a focus on fantasy elements. The image is likely intended for a gaming or fantasy-themed context, given the character’s attire and the overall aesthetic.
- 2.
The image presents a scene of elegance and luxury. Dominating the center of the image is a brown Louis Vuitton suitcase, standing upright. The suitcase is adorned with a gold handle and a gold lock, adding a touch of opulence to its appearance. Emerging from the top of the suitcase is a bouquet of pink and white roses, interspersed with green leaves. The roses, in full bloom, seem to be spilling out of the suitcase, creating a sense of abundance and luxury. The entire scene is set against a white background, which accentuates the colors of the suitcase and the roses. The image does not contain any text or other discernible objects. The relative position of the objects is such that the suitcase is in the center, with the bouquet of roses emerging from its top.
- 3.
The image captures the grandeur of the Toledo Town Hall, a renowned landmark in Toledo, Spain. The building, constructed from stone, stands tall with two prominent towers on either side. Each tower is adorned with a spire, adding to the overall majesty of the structure. The facade of the building is punctuated by numerous windows and arches, hinting at the intricate architectural details within. In the foreground, a pink fountain adds a splash of color to the scene. A few people can be seen walking around the fountain, their figures small in comparison to the imposing structure of the town hall. The sky above is a clear blue, providing a beautiful backdrop to the scene. The image is taken from a low angle, which emphasizes the height of the town hall and gives the viewer a sense of being in the scene. The perspective also allows for a detailed view of the building and its surroundings. The image does not contain any discernible text. The relative positions of the objects confirm that the town hall is the central focus of the image, with the fountain and the people providing context to its location.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: see Weaknesses
soundness: 3
strengths: 1. The motivation for using preference models is well-founded, and the paper is well-written.
2. It is interesting to identify two distinct focuses within preference models, and the analysis provided is both reasonable and thorough.
summary: This paper presents a novel approach to enhance the alignment between long text descriptions and generated images in text-to-image diffusion models, introducing segment-level encoding to overcome input length limitations and decomposed preference optimization to mitigate overfitting and improve text-relevant alignment during fine-tuning.
weaknesses: weakness
1. I am unsure why multiple <sot> tokens are retained; regarding the retention or removal of tokens, a more detailed explanation or analysis is needed, as it currently leaves me confused.
2.After reweighting, whether there will be a noticeable difference in the aesthetic quality of the generated results (due to text-irrelevant components) remains unclear. For Appendix B.1, it would be beneficial to provide some visualizations of the outcomes from the two loss functions.
3. Segmenting to leverage CLIP's alignment effect is an intuitive innovation, but does this become irrelevant in light of the development of Vision-Language Models (VLMs)? Can the current innovation still contribute to VLMs?
4. On line 363, it mentions mitigating the risk of overfitting to Denscore. Could you clarify where the potential source of this overfitting lies?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: [1] Does your proposed segment-level encoding strategy demonstrate significant effectiveness for texts of varying lengths? Specifically, how does the model perform with very short texts (fewer than 10 words) or very long texts (over 500 words)? Could you provide additional experiments to show comparative results under different text length conditions to verify the generalizability of the segment-level encoding strategy?
[2] You mentioned using a reweighting strategy to mitigate the model's overfitting issue, but the description of this process in the paper is rather brief. Could you provide detailed steps or pseudocode to explain the implementation of this strategy? Additionally, does this method have any quantitative results to demonstrate its effectiveness in reducing overfitting in specific scenarios? Could you include comparative data from the experiments to validate the impact of this strategy?
[3] How were the 5k images in the test set specifically selected from datasets like SAM and COCO2017?
[4] Could you briefly explain the selection of models like CLIP-H and HPSv2 in the experimental section of Chapter 5, as well as the chosen evaluation metrics?
soundness: 2
strengths: [1] It introduces a segment-level encoding strategy that effectively handles long text inputs by dividing and separately encoding segments, overcoming traditional model input limitations and enhancing text-to-image alignment. 
[2] The preference model is innovatively decomposed into text-relevant and text-irrelevant components, with a reweighting strategy to reduce overfitting and improve alignment precision.
[3] The paper conducts extensive experiments, demonstrating significant improvements in long-text alignment over existing models like PixArt-α and Kandinsky v2.2, proving the method's effectiveness for complex text generation tasks.
summary: This paper proposes a novel method to improve text-to-image (T2I) diffusion models in handling long text inputs. Due to the input length limitations of existing encoders like CLIP, it becomes challenging to accurately align generated images with long texts. To address this issue, the authors propose a segment-level encoding strategy, which divides long texts into segments and encodes them separately, combined with a decomposed preference optimization method to reduce overfitting and enhance alignment. Experimental results show that the fine-tuned model surpasses several existing foundation models in long-text alignment, demonstrating significant improvements in handling long text inputs.
weaknesses: [1] The paper proposes a segment-level encoding strategy to handle long texts but does not thoroughly validate the performance of this strategy under different text length conditions. For very short or very long texts, can the segment-level encoding still maintain the same alignment effectiveness? The lack of fine-grained comparative experiments makes it difficult to adequately demonstrate the applicability of segment-level encoding across a wide range of text lengths.
[2] The paper proposes a reweighting strategy to address overfitting, but lacks detailed experimental data to demonstrate its effectiveness, failing to adequately prove its specific impact on reducing overfitting.
[3] The segment-level encoding and preference optimization strategies proposed in this paper show excellent performance in the experiments, but lack an analysis of the method's limitations. It would be beneficial to discuss whether these segment-level encoding methods might lose part of their alignment effectiveness when dealing with texts that have complex contextual dependencies or require strong semantic understanding.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: I apologize in advance if I missed it, but I do not really see clear details about the training of the Denscore model. B.1 has details on the training objectives and the fact that captions are generated by LLaVA-Next, but beyond this I do not see other implementation details (dataset, other choices etc.), so it would be great if the authors could point me to this.
soundness: 4
strengths: The paper tackles the crucial challenge of long prompt following in a very effective manner. Using a text encoder that can take the entire long prompt is a sound idea, and the Denscore preference model looks like a useful contribution in general. 
Apart from this, the reward fine-tuning with the orthogonal decomposition and the gradient reweighting looks like a good idea to deal with the "reward-hacking" problem.
Finally, the results also appear quite strong from the evaluations presented in the paper.
summary: This paper presents a method for enhancing the prompt following of text-to-image models specifically in the case of long prompts. The key contribution for tackling this problem is twofold: a) using a combination of CLIP and T5 encoders (as is becoming increasingly common these days e.g. SD3, Flux) b) the introduction of a preference model tailored for long prompts (Denscore) and applying reward fine-tuning with this Denscore model to enhance the prompt following of SD1.5 models for long prompts.
weaknesses: An important paper that is missed here is ELLA[Hu et al. 2024] for a couple of reasons. The first is that they propose replacing the CLIP encoder of SD1.5 with a T5-XL model and get significantly improved results (far superior numbers to those reported by Lavi-Bridge whose MLP adapter is used here). Therefore, this model might be a valid comparison (although the training cost of ELLA is a bit higher: 7 days with 8 A100s for SD1.5). Alternatively, the adapter provided by ELLA would have probably been a better alternative to the one used in the paper (from Lavi-Bridge). 

Apart from the comparison/use of adapter, there's also DPG-Bench introduced in the paper which is a good benchmark for long prompt following (as compared to existing benchmarks like T2I-Compbench, DSG, TIFA etc.). Evaluating on DPG-Bench would be a useful addition since the 5k evaluation set of this paper is not fully understood and only a few models have been evaluated here. Additionally, from an evaluation standpoint, even on this 5k evaluation set, VQAScore[1] might be a good option to consider, since it uses a T5-XXL model which can take long prompts, and has shown some promising results for text-to-image evaluations. 

Another aspect which is missing here is that all the experiments in this paper are conducted on SD1.5 which is a relatively older model, and there have been newer models in the past 2 years (e.g. SDXL). Therefore, it would have been nicer to also have results with any of the newer, more performant models, but I can understand that this might be a bit more computationally expensive (especially if the training has to be done at 1024 resolution). 

Overall, I dolike the paper, but I believe that incorporating these aspects (especially strengthening the paper with additional evaluations) could improve the paper significantly. 

[1] Lin et al. "Evaluating Text-to-Visual Generation with Image-to-Text Generation", ECCV 2024

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: (1) Please better explain why the proposed split-and-merge approach can address the long-text alignment issue.
(2) Please provide the ablation study clearly.
soundness: 3
strengths: (1) Comprehensive Survey of Related Work.
The paper presents a thorough and comprehensive survey of existing work in text-to-image (T2I) diffusion models, demonstrating an impressive grasp of the field. By delving deeply into previous approaches and their limitations, the authors effectively set the stage for their contributions, clarifying the gaps their method aims to fill. This background provides readers with valuable context and insight into the evolution of T2I models, particularly in handling longer, complex textual inputs. The comprehensive nature of this survey also reinforces the authors' understanding of the field's current challenges and strengths, building confidence in the relevance and timeliness of the proposed approach.

(2) Importance of the Problem and a Reasonable, Well-Motivated Solution.
The authors tackle a critical issue in T2I diffusion models: the difficulty of aligning generated images with longer text prompts. As the demand for complex, high-fidelity image generation grows, the ability to handle longer text inputs accurately is essential. The segmentation approach, paired with decomposed preference optimization, offers a well-motivated solution to this problem. Segmenting long text into manageable parts allows for better processing within the confines of existing encoding models, while the decomposed preference optimization fine-tunes the alignment, addressing the unique challenges posed by long prompts. The design choices reflect a reasonable and methodical approach to tackling these limitations, and the paper articulates the rationale for each component clearly. This structured approach suggests the authors have carefully considered the problem’s nuances, offering a solution that is not only effective but also grounded in sound methodology.

(3) Demonstrated Superiority over State-of-the-Art Models.
One of the paper's significant strengths is the demonstrated performance improvement over state-of-the-art models. Through rigorous experimentation, the authors show that their method surpasses current leading models like PixArt-α and Kandinsky v2.2 in T2I alignment, particularly for long-text prompts. By fine-tuning Stable Diffusion v1.5 with their approach, they achieve superior alignment, reducing overfitting while preserving text-relevant information in the generated images. This achievement underscores the potential of the proposed method to set a new benchmark for handling longer, more detailed textual inputs within T2I models. The improvement over established models validates the effectiveness of the segmentation and preference optimization strategy, indicating that this approach could meaningfully advance the state of the art in T2I diffusion modeling.
summary: This paper presents a new approach for long text inputs on text-to-image (T2I) alignment since the clip text encoding only allows 77 tokens. The authors address limitations of existing encoding methods like CLIP by proposing segment-level encoding, where long texts are divided and processed in parts to bypass input length constraints. They further introduce a decomposed preference optimization that separates alignment-related and non-alignment components of CLIP-based preference scores. By reweighting these components, the method reduces overfitting, achieving superior T2I alignment after fine-tuning Stable Diffusion v1.5, outperforming models such as PixArt-α and Kandinsky v2.2.
weaknesses: Although the proposed method solved an important issue, three major issues remain as listed below.
(1) Limitations and Ambiguities in the Segmentation and Merging Methodology. The segmentation and merging technique proposed in this work introduces a unique approach to handling longer text inputs but raises questions regarding its effectiveness and generalizability. When text inputs exceed 77 tokens, this method still encounters limitations, as it is fundamentally restricted by the underlying model’s capacity to handle “long” sequences since the split-and-merge process does not solve the problem. This constraint is particularly concerning as longer text inputs are common in real-world applications and often essential to producing detailed and contextually accurate image generations. The current approach of segmenting and then merging these sections seems like a workaround rather than a robust solution to handling extended texts, which may inherently limit its scalability and versatility. Furthermore, the mechanics of how segmentation and merging affect the underlying model's cross-attention dynamics remain underexplored. Cross-attention is a critical component in the alignment process between text and image features, and segmenting inputs may disrupt this alignment, especially as certain semantic connections might be lost or diluted across segmented inputs. Investigating the cross-attention differences between the original, unsegmented approach and the segment-and-merge methodology could shed light on any distortions introduced by this technique. A more thorough analysis of cross-attention’s role here could help refine segmentation methods to better retain textual coherence and improve image alignment fidelity, ultimately benefiting downstream performance.

(2) Dependency on an Outdated Baseline Model (Stable Diffusion v1.5):
The use of Stable Diffusion v1.5 as the primary evaluation model poses a significant limitation, given that the field has moved toward more advanced versions like SD-3 and SDXL. These newer versions incorporate improved architectures and training techniques, yielding enhanced performance, especially in terms of image quality and alignment with textual inputs. The reliance on an outdated model not only limits the relevance of the study’s results but also restricts the potential impact of the proposed method. Using v1.5 as the baseline reflects well on the approach’s applicability to older architectures, but it leaves unanswered questions about its efficacy on more sophisticated models that incorporate advancements in diffusion techniques, training scale, and multimodal alignment mechanisms. 
Moreover, maintaining SD-1.5 as a standard for comparison could inadvertently hold back progress within the research community. As models continue to evolve, it’s essential to align benchmark tests with the latest technologies to ensure that methods are relevant and that advancements reflect real-world capabilities. Preliminary results from newer models, such as SD-3, have demonstrated considerable improvements in T2I alignment, indicating that the proposed method may benefit even further from these architectural updates. Testing on newer models would better position the approach in the context of current technological standards, ensuring that it remains relevant and applicable as diffusion models evolve. Future work should include evaluations on SD-3 and SDXL to substantiate claims of superiority over other methods in a more current setting. The test of SD-3 with the prompt used in the first example of Fig. 1 is shown below.
https://ibb.co/CWyKQTZ

(3) Over-reliance on Long Prompt Training and Lack of Generalizability Testing.
The proposed method seems to rely heavily on training with long prompts, which could limit its flexibility and adaptability. While training on extended text inputs may enhance alignment for similar prompts, it raises concerns about the model's performance on shorter or more varied prompts. In real-world scenarios, prompt lengths and structures vary significantly, and a robust model should perform consistently across this spectrum. By focusing predominantly on long-prompt alignment, the current approach may overfit to specific input lengths, making it less effective for shorter or less detailed prompts where segmentation might not be necessary or where text segments are not sufficiently complex to benefit from this treatment.
To address this potential limitation, it would be valuable to conduct experiments that vary prompt lengths and structures systematically, assessing whether the model’s performance holds across different scenarios. Additionally, testing with alternative segmentation designs could reveal whether simpler or more complex methods yield better alignment. These experiments would enhance our understanding of how adaptable the proposed method is, providing insights into its generalizability and robustness. The community would benefit from such insights, as they could guide further development of segmentation-based approaches for T2I tasks.

Meta-review: This paper proposes a novel text encoder to deal with long conditioning texts in text-to-image diffusion models. Long texts are divided into segments and processed separately. The authors finetune a stable diffusion model based on this encoding, using a CLIP-based preference optimization method. 
Strengths mentioned in the reviews include: well written paper, good review of related work, good analysis of preference model leading to decomposition, and significant improvements over prior work. 
Weaknesses included: missing some baseline comparisons, experiments only used the SD1.5 as generative foundation model, missing evaluations as function of prompt length, there were also suggestions to include DPG-bench and VQAScore for evaluation.

Decision: Accept (Poster)