Scalable Ranked Preference Optimization for Text-to-Image Generation

Authors: Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, Anil Kag
Primary area: generative models
Keywords: text-to-image generation, Direct Preference Optimization, Learning from AI Feedback

Abstract:
Direct Preference Optimization (DPO) has emerged as a powerful approach to align text-to-image (T2I) models with human feedback. Unfortunately, successful application of DPO to T2I models requires a huge amount of resources to collect and label large-scale datasets, e.g., millions of generated paired images annotated with human preferences. In addition, these human preference datasets can get outdated quickly as the rapid improvements of T2I models lead to higher quality images. In this work, we investigate a scalable approach for collecting large-scale and fully synthetic datasets for DPO training. Specifically, the preferences for paired images are generated using a pre-trained reward function, eliminating the need for involving humans in the annotation process, greatly improving the dataset collection efficiency. Moreover, we demonstrate that such datasets allow averaging predictions across multiple models and collecting ranked preferences as opposed to pairwise preferences. Furthermore, we introduce RankDPO to enhance DPO-based methods using the ranking feedback. Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated preference dataset ``Syn-Pic'' improves both prompt-following (on benchmarks like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user studies). This pipeline presents a practical and scalable solution to develop better preference datasets to enhance the performance and safety of text-to-image models.

Decision: reject
Reviewer scores: 6, 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730201546233

thread_type:
official_review

<thread_object>
creator: reviewer_9pjp

<content>
summary:
This paper introduces a powerful and cost-effective approach for the preference optimization of text-to-image models. Specifically, this work demonstrates how synthetically generating a preference optimization dataset can enable the collection of superior signals (e.g., rankings vs. pairwise preferences and ensembling preferences across models). Moreover, this paper also introduces a simple method to leverage the stronger signals, leading to state-of-the-art results on various benchmarks for prompt following and visual quality for both the diffusion and rectified flow models.

strengths:
Inspired by the LLMs community, aligning T2I models with human feedback has become an important and practical topic to enhance existing T2I models. However, there are two major efforts in this area: 1) collecting large amounts of user preferences images for training and 2) fine-tuning with T2I models with reward functions. The former direction shows promising results when utilizing Direct Preference Optimization (DPO), but with expensive data collection cost, e.g., Pick-a-Picv2 costs nearly $50K. The later direction fine-tunes the T2I models by maximizing the reward functions with the generated images, but this process is computationally expensive and suffers from “reward hacking”, where this optimization process increases the reward scores without improving the quality of the generated images.

This paper address the above challenges and propose a scalable and cost-effective solution for aligning T2I models, which includes the following two contributions:
- Synthetically Labeled Preference Dataset (Syn-Pic).
- Ranking-based Preference Optimization (RankDPO).

The experimental results seem to consistently show the advantages of this approach, indicating that it is an effective solution.

weaknesses:
- **Lack of comparison of training costs.**  Cost-efficiency is the main claimed contribution of this work, but the paper only briefly mentions the monetary comparison (\\$200 v.s. \\$50K for Pick-a-Picv2). I believe that the gain of this work lies in the newly proposed loss function $\mathcal{L}_{\text{RankDPO}}$, so its training cost comparison, e.g., training time, gpus, flopts, and so on, should be included in main text and Tab.8.

- **Clarification of the ``*Reference model*'' definition**. The ``*reference model*'' defined in the method is not fully explained in lines 204 and 285, resulting in a less than intuitive understanding of the method.

- **Analysis of the effectiveness and limitations of Ground Truth.** I understand that Eq.5 is the core of the proposed RankDPO, which means that the model is supervised based on the reference model or the so-called ground truth (GT) preference, so I really want to know how this GT preference is obtained (Section 3.2 does not seem to mention)? Is there a subjective bias for GT that causes the performance of the model to have an upper limit (e.g., only select few intended t2I models)? Does the scale k of the adopted t2I models have an impact on the ground truth?

- **Lack of *scalable* explanations and quantitative or qualitative comparisons.** In line 92, it is claimed that this work is scalable, but there is a lack of theoretical explanation, qualitative analysis and quantitative comparison of scalability in the experiments and methods. In other words, are all previous methods not scalable? How to measure the scalability of this work? Can this work be adapted to the preference annotation of other tasks such as video and 3D? What is the connotation of this scalability? Please give a detailed explanation and clear definition.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730329497973

thread_type:
official_review

<thread_object>
creator: reviewer_pz5f

<content>
summary:
This paper proposes a scalable approach to fine-tune text-to-image generation (T2I) models using DPO without requiring human annotation. The authors introduce a synthetic preference dataset collection method, where preferences are generated using off-the-shelf reward models, eliminating the need for human annotation. Additionally, this approach leverages model predictions across multiple models, enabling dense and ranked preferences rather than just parse pairwise ones. The authors also propose RankDPO, a framework that incorporates ranking feedback to enhance the DPO process. Their method, applied to SDXL and SD3-Medium models, demonstrates improved prompt alignment and visual quality on various benchmarks. This study contributes a cost-effective, efficient pipeline to advance T2I model alignment.

strengths:
+ Scalable and cost-effective dataset collection with synthetic preferences.

+ The proposed RankDPO framework effectively leverages ranked feedback to improve text-image alignment

+ Experiments on benchmark datasets could verify the effectiveness of the proposed method.

weaknesses:
- Lack of novelty: Existing work (Wallace et al. 2024, Liu et al. 2024c) have proposed diffusion-based DPO and List-wise ranking-based DPO. It seems that the proposed method combines them to tune diffusion models. 

- More baseline methods, especially those used to generate images in the process of the construction of Syn-Pic, should be considered and compared in Tab 1, Tab 2, and Tab 3. 

- The performance improvement gained by RankDPO over DPO (5 Rewards) in Tab 4 seems marginal. The effectiveness of the list-wise or rank-wise DPO method may be not strongly verified.

Minors: 
It seems to be “Tab. 5” instead of “Tab. 4” in line 425.

questions:
- The improvement in the non-spatial category of T2I-CompBench, as shown in Tab 2, seems not remarkable compared with others. Why is that and how to explain it? 

- How does the proposed method influence the quality of generated images (e.g., FID), although the alignment has been improved as shown in Tab 1-3?

- Is it possible that the proposed method just “distills” the useful knowledge from the models used to generated images in the process of the construction of Syn-Pic, especially those strong base models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730553603420

thread_type:
official_review

<thread_object>
creator: reviewer_grrx

<content>
summary:
Overall, this is a very good and solid paper. The paper addresses the challenges of applying Direct Preference Optimization (DPO) to text-to-image (T2I) models, particularly the resource-intensive process of collecting human preference datasets. The authors propose a scalable approach for creating large-scale synthetic datasets for DPO training, using a pre-trained reward function to generate preferences for paired images. This method allows for averaging predictions across multiple models and collecting ranked preferences instead of just pairwise preferences. The researchers introduce RankDPO, an enhancement to DPO-based methods that utilizes ranking feedback. Applying RankDPO on SDXL and SD3-Medium models with the synthetic "Syn-Pic" dataset improves both prompt-following and visual quality, offering a practical solution to enhance the performance and safety of T2I models.

strengths:
1. The writing of this paper is clear and easy to follow. It's easy to understand the main contributions of this paper as I summarized above.
2. The contributions of this paper are solid. [Method contribution] The authors propose a new performance optimization method RankDPO, and the effectiveness is well supported by extensive experiments on SDXL, SD3-Medium models and GenEval, T2I-CompBench benchmarks. [Dataset contribution] To make the ranking-based preference optimization work, the authors collect a new dataset Synthetically Labeled Preference Dataset (Syn-Pic), which can benefit future research in text-to-image community.

weaknesses:
No obvious weaknesses. Please check the questions section.

questions:
1. How to evaluate the reliability of the Reward Models when building the Syn-Pic dataset? Do you conduct a human evaluation study, for example, do you calculate the correlation between the reward model ranking and human evaluation ranking?
2. There are some other text-to-image benchmarks as I know, for example, HRS-bench [*]. Can you explain the reason why not include other benchmarks in your paper?

[*] Bakr, E. M., Sun, P., Shen, X., Khan, F. F., Li, L. E., & Elhoseiny, M. (2023). Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20041-20053).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730606849268

thread_type:
official_review

<thread_object>
creator: reviewer_rfve

<content>
summary:
This paper studies the preference optimization of T2I diffusion models, particularly SD3-Medium and SDXL. The main contribution made by this paper has two folds. 1) This paper proposed a new synthetic preference dataset containing rankings annotated by existing preference models, which cost less than similar datasets with human annotation. 2) Based on the ranking-based dataset, this paper proposes a new DPO method using the ranking information to improve the prompt following and generation quality of T2I diffusion models.

strengths:
- The overall presentation of this paper is clear.
- I like the idea that extending existing human preference dataset with (win, lose) tuple to having more items, naturally leading to a rank based preference dataset.
- The experiments are conducted on both big and medium sized models: SDXL and SD3-Medium, which is adequate. The comparison experiments with other models finetuned with DPO and different preference datasets show that the proposed new dataset and method has better performance than others.

weaknesses:
- Overall, I like the idea of using rank-based preference dataset and DPO, which should be able to provide more fine-grained preference guidance than (win, lose) style DPO and datasets. My only concern is that it is unclear whether the performance improvement of the proposed RankDPO is more because of the new synthetic dataset or the rank-based DPO. According to my understanding, in Table 3, DPO-SDXL, MaPO SDXL, SPO SDXL, and the proposed RankDPO SDXL are fine-tuned with different preference datasets, which mixes the impact of the new dataset and new DPO model. It would be better if the author could provide the insight about whether dataset or the rank-based DPO, is more important to the performance improvement over other preference optimization models. What if train DPO/SPO/MaPO with newly collected rank preference dataset? For example, randomly pick two images from a rank set and set the one with the lower rank to be the "win" and the other to be the "lose". What if finetune with RankDPO but using existing preference datasets? For example, set the "win" sample to be rank 1 and the "lose" sample to be rank 2. In my opinion, these insight is super important to this paper and would be beneficial to the community.

questions:
No
</content>
</thread_object>

</thread>
<thread>
creation date:
1732137796119

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for providing valuable feedback on our paper. We are pleased that they find our paper to have solid contributions (Reviewer GRrX), clear writing (Reviewers Rfve, GRrX),  and providing an effective solution (Reviewers 9PJP, Pz5f).  While we have addressed individual concerns in our replies, we outline the key additions to the paper below:

**a) Additional Baselines (Pz5f):** We have added PixArt-Sigma and Stable Cascade to Tab. 1,2,3 completing all the models that we use for generating Syn-Pic. We continue to observe that RankDPO applied to SDXL and SD3-Medium obtains strong results compared to other methods.


**b) FID-based evaluation (Pz5f):** We already evaluate our methods against baselines on several popular quantitative benchmarks (Tab. 1,2,3) and through user-studies (Fig. 3). In addition, in Tab. 10, we have included an evaluation on the MJHQ-30K set using FID against Midjourney images as an additional evaluation metric to measure image quality. We observe consistent improvements in FID scores as compared to existing results with SDXL. Although, it is worth noting that FID scores are sensitive to the choice of reference images and the pre-processing of features. 
| **Category**   	| **Animals** | **Art**   | **Fashion** | **Food**  | **Indoor** | **Landscape** | **Logo**   | **People** | **Plants**  | **Vehicles** | **Overall** |
|---------------------|-------------|-----------|-------------|-----------|------------|---------------|------------|------------|-------------|--------------|-------------|
| **SDXL-Refiner**	| 28.93   	| 31.05 	| 28.90   	| 30.09 	| 28.83  	| 30.78     	| 36.67  	| 35.56  	| **28.42**   | 24.45    	| 9.55    	|
| **RankDPO SDXL**	| **24.37**   | **27.22** | **22.91**   | **25.02** | **26.01**  | **28.61** 	| **29.74**  | **27.54**  | 30.27   	| **21.83**	| **7.99**	|



**c) Clarify RLHF formulation (9PJP):** We now have a more detailed introduction of RLHF (Sec. 3.1) explaining the objective as well as the role of the reference model. 


**d) Reward Model Strengths and Limitations (GRrX, 9PJP):** We now have a more detailed explanation for the reward models (Sec. 5), including discussing their high correlation to overall human judgment on various benchmarks, making them an ideal choice for labeling our preferences. 


**e) Disentangling gains from Syn-Pic and RankDPO (Rfve, Pz5f):** Our ablations in Tab. 4 and 6 aim to break-down the gains from our proposed preference dataset and learning objective.  
In Tab. 4, DPO (Pick-a-Picv2) vs DPO (5 rewards) compares the performance of DPO when trained on Pick-a-Picv2 and Syn-Pic datasets. It shows that training on the Syn-Pic dataset yields considerable improvements over training on Pick-a-Picv2. 
Further, in Tab. 4, DPO (5 rewards) vs RankDPO compares the performance of RankDPO vs DPO when trained on the same Syn-Pic dataset, where we observe improvements over DPO. Additionally, in Tab. 6, we train DPO and RankDPO on the Syn-Pic dataset and evaluate their performance on GenEval benchmark. It shows that RankDPO consistently outperforms DPO.

| **Model Name**         	| **DSG Score** | **VQA Score** | **Q-Align Score** |
|-----------------------------|---------------|---------------|-------------------|
| SDXL                   	| 74.65     	| 84.33     	| 0.72          	|
| DPO (Pick-a-Picv2)     	| 76.74     	| 85.67     	| 0.74          	|
| DPO (5 Rewards)        	| 78.84     	| 86.27     	| 0.81          	|
| **RankDPO (Only SDXL)** 	| 78.40 	| 86.76 	| 0.74      	|
| **RankDPO**            	| 79.26 	| 87.52 	| 0.81      	|


| **Model**    	| **Mean ↑** | **Single ↑** | **Two ↑** | **Counting ↑** | **Colors ↑** | **Position ↑** | **Color Attribution ↑** |
|-------------------|------------|---------------|------------|-----------------|---------------|-----------------|--------------------------|
| SDXL         	| 0.55   	| 0.98      	| 0.74   	| 0.39        	| 0.85      	| 0.15        	| 0.23                 	|
| DPO	| 0.59   	| 0.99      	| 0.84   	| 0.49        	| 0.87      	| 0.13        	| 0.24                 	|
| **RankDPO**   	| **0.61**   | 1.00  	| 0.86   | 0.46    	| 0.90  	| 0.14    	| 0.29             	|
|-------------------|------------|---------------|------------|-----------------|---------------|-----------------|--------------------------|
| SD3-Medium   	| 0.70   	| 1.00      	| 0.87   	| 0.63        	| 0.84      	| 0.28        	| 0.58                 	|
| DPO	| 0.72   	| 1.00      	| 0.90   	| 0.63        	| 0.87      	| 0.32        	| 0.58                 	|
| **RankDPO**   	| **0.74**   | 1.00  	| 0.91   | 0.72    	| 0.87  	| 0.31    	| 0.66             	|
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper has two major contributions: Syn-Pic, a synthetic preference dataset collection method, where preferences are generated by using multiple existing reward models; RankDPO, a extended DPO method with list-wise preference. 

Major paper strengths: 1) using existing reward models to create synthetic list-wise preference data and  applying RankDPO to this listwise preference data is a reasonable solution 2) Experiments on multiple diffusion models like SDXL and SD3-Medium on multiple benchmarks shows the effectiveness of the proposed approach.

Major weaknesses: 1) As pointed out by reviewer Pz5f and Rfve, the technical novelty of this paper is not high. RankDPO is essentially similar as LiPO: Listwise Preference Optimization through Learning-to-Rank (https://arxiv.org/html/2402.01878v1) . And the idea of using existing reward models to generate synthetic preference data is straightforward. 2) The motivation of using existing reward models in Syn-Pic is to eliminate human labeled data, however, human labeled data are needed when creating the reward models. So it does not really eliminate human labeled data, but instead implicitly leverage existing human labeled data in previous reward models. Moreover, existing reward models may not generalize well to future generative models (when model quality and data distribution are different). So this method will not eliminate human labeled data for future generative models either.

Given the two major weakness and the overall average score of this paper is 6, not very high, I will recommend "Reject".