Iterative DPO with An Improvement Model for Fine-tuning Diffusion Models

Authors: Ying Fan, Fei Deng, Yang Zhao, Sahil Singla, Rahul Jain, Tingbo Hou, Kangwook Lee, Feng Yang, Deepak Ramachandran, Qifei Wang
Primary area: generative models
Keywords: dpo, diffusion models

Abstract:
Direct Preference Optimization (DPO) has been proven as an effective solution in aligning generative models with human preferences. However, as shown in recent works, DPO could suffer from constraints from the offline preference dataset. This paper introduces a novel improvement approach for online iterative optimization of the diffusion models without introducing extra annotation of the online data. We propose to learn a preference improvement model to extract the implicit preference from the preference dataset. The learned improvement model is then used to generate winning images from the images generated by the current diffusion model. We can construct new pairs of preference data by using images generated by the current diffusion model as losing images, and its corresponding improved images as winning images. The diffusion model can therefore be optimized via iteratively applying online preference datasets. This method enables online improvement beyond offline DPO training without requiring additional human labeling or risking overfitting the reward model. Results demonstrate improvements in preference alignment with higher diversity compared with other fine-tuning methods. Our work bridges the gap between offline preference learning and online improvement, offering a promising direction for enhancing diffusion models in image generation tasks with limited preference data.

Decision: withdrawn
Reviewer scores: 5, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729061044247

thread_type:
official_review

<thread_object>
creator: reviewer_mw3m

<content>
summary:
The paper introduces the drawback of standard DPO methods that they are limited by their training data as the motivation of the paper.

To be specific: It first use pre-collected paired data to train a model similar to instruct-p2p (improvement model), which can be used to enhance the quality of input images.

The proposed method iteratively trains the   diffusion model and generate new "bad" samples with the diffusion model. The improvement model is used to generate "good" data given the new model.

strengths:
1. Reasonable storytelling and clear figures.

2. The method is easy to understand and implement.

weaknesses:
**Validation**

1. The performance improvement is very minor. From Table 4, we can see that the win-rate improvement is at most 0.9% compared to SPIN. I also think the presented paired visual comparisons in the paper (those figures) are not impressive.

2. The validation is not sufficient. It only validates performance on stable diffusion 1.5. Many works including Diffusion-DPO report their performance on both stable diffusion 1.5 and stable diffusion XL. The generation quality of stable diffusion 1.5 is not good and therefore improving generation quality from it is much easier than stable diffusion XL. It makes me suspicious about its effectiveness on better and larger models.

3. Validation of the effectiveness of the improvement model. I didn't find any validation on how much the improvement model can improve the given images. (I am sorry if I missed them.) The only available evaluation on the improvement seems to be the Table 3, which can not reflect the ability of the improvement model. The improvement model even performs worse than SPIN on test set. How can we expect it to improve the quality of generated images.

4. Many experimental details are not clearly stated. How is the win-rate calculated in Table 4? 

5. Lack of user study including comparison and  details. 


**Reasonability**

1. Since all good images are generated with the improvement model, I think the generation quality might be upper-bounded by the improvement model. I am suspicious about the reasonability of the proposed method.

2. Generating new samples during training takes a lot of computation and time. I noticed in the paper that the authors train the improvement model 200K steps with batch size 2048, which is a very costly training compared to standard diffusion-DPO (They train their model with only 2K iterations with a similar batch size 2048).   The very high training cost might also be the reason that they do not train their model on SDXL or even larger models. 

**Presentation**
1. Most presented images are not visually appealing and their prompts are generally very short. It can not reflect the text-image alignment.


Due to the above reason, I think the paper still has a relatively large gap for publication and might not meet the bar of ICLR.

questions:
I suggest the authors validating their method on larger models like Stable Diffusion XL and Flux. Otherwise I am deeply suspicious of its effectiveness on larger models. They should show the potential of scaling up of their method for practical application.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634931966

thread_type:
official_review

<thread_object>
creator: reviewer_bg6f

<content>
summary:
In this paper, authors propose a diffusion model alignment method with a limited number of preference datasets. Specifically, an improvement model is trained for generating high-quality samples. Then DPO is achieved on an extended synthetic dataset.

strengths:
The method is easy to understand. The visualization is good. The method is reasonable.

weaknesses:
1. The authors should provide precise definitions and annotations for each concept, such as $\epsilon^w$ and $\epsilon^l$.
2. The proposed method introduces a semi-supervised learning framework to utilize a preference dataset for training a "high-quality" model, followed by fine-tuning the diffusion model using generated high- and low-quality data pairs. However, this approach appears to complicate the alignment process for diffusion models. A more straightforward solution would be to train a reward model directly, enabling reinforcement learning without the need to construct a high-quality model and apply DPO in such a rudimentary way.
3. Additionally, the performance gains achieved by the proposed method appear limited. As shown in Table 1, the results are only comparable to those of SPIN.
4. The authors should refine the contributions section, as the current version lacks clarity and specificity.
5. The authors should consider including a comparison with alignment methods based explicitly on reward model [1].
[1] Zhang, Y., Tzeng, E., Du, Y., & Kislyuk, D. (2024). Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401.12244, 4.

questions:
Please refer to the weakness section. Moreover, authors should illustrate the detailed reasons to choose such a complicated alignment process instead of directly training a reward model and applying RL.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659641248

thread_type:
official_review

<thread_object>
creator: reviewer_fmrq

<content>
summary:
This paper introduces an approach for online iterative optimization of the diffusion models without introducing extra annotation of the online data. It uses the generated images from the improved models as winning samples to continue improving the base model in multiple iterations.

strengths:
The writing is relatively clear, and the investigation is fairly thorough.

weaknesses:
**Idea is unreasonable:** Using the generated images from the improved models as winning samples is not a reasonable approach. Due to the instability of the generative model, even with the same seed, it is difficult to ensure that the images generated by the improved models are consistently better in quality than those from the base model. This introduces a significant amount of noisy data, which can severely affect the quality of model training. Furthermore, the method involves multiple iterations of training, which exacerbates this issue.

**Lack of novelty:** The paper merely proposes an iterative DPO method without providing a well-thought-out solution to the core problem, making its novelty and contributions quite limited.

**Unreasonable experiments:** Firstly, a considerable number of the visual examples presented in the paper do not demonstrate the advantages of the proposed algorithm. The generated images do not align well with the prompts, and the improvements over the baseline model seem quite limited. Does this suggest that the method’s effectiveness is rather underwhelming? To verify the validity of this approach, I think it is necessary to conduct experimental analysis on the number of iterations.

questions:
You need to provide a detailed explanation addressing the issues raised in **Weakness 1**, specifically whether this situation was encountered during implementation.

Also, conduct experiments on the number of iterations to verify the reasonableness of the iterative training process.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711634013

thread_type:
official_review

<thread_object>
creator: reviewer_fznm

<content>
summary:
This paper proposes an online DPO method to iteratively improve the preference of the diffusion model, and expand the limitation of fixed size preference dataset in optimization. The preference pairs are generated from the original diffusion model and the preference improvement model that trained on the offline preference dataset. Experiments show the effectiveness of the proposed method.

strengths:
1. This paper proposes an online DPO method with preference improvement model to expand the size limitation of the preference dataset in optimization.
2. Muti-task learning objectives are proposed for the preference improvement model for ensured quality and diversity.
3. Experiments show the effectiveness of the proposed method.

weaknesses:
1. The main idea of this work is to provide online preference positive samples through constructing the preference improvement model that maps the outputs from the original diffusion model to high-quality ones. It is still under the static Diffusion-DPO framework without dynamic adaption, which is somewhat less convincing.
2. It is wondered whether the preference improvement model needs to be evolved in iteration, as the fixed improvement model can be hard to adapt to the shifted distribution of the evolved diffusion model, and the positive samples provided by consequent preference improvement are doubted to bias from the original user preference. Additionally, the fixed improvement model is still setting the upper bound of the optimized diffusion model, with only enlarged optimization samples.
3. It is recommended to provide the per-iteration result to monitor the diffusion model behavior.
4. The role of guidance weights in yielding positive samples is not well illustrated with corresponding qualitative or quantitative results. Basically, we have no idea about the behavior of the improvement model in terms of its multi-objective learning. The validation experiments should be provided.

questions:
Is the diffusion model sensitive to the CFG weights of positive samples in preference optimization?
</content>
</thread_object>

</thread>


Metareview: