|
---
|
|
license: apache-2.0
|
|
---
|
|
# Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step
|
|
|
|
<a href="https://arxiv.org/abs/2406.04314"><img src="https://img.shields.io/badge/Paper-arXiv-red?style=for-the-badge" height=22.5></a>
|
|
<a href="https://github.com/RockeyCoss/SPO"><img src="https://img.shields.io/badge/Gihub-Code-succees?style=for-the-badge&logo=GitHub" height=22.5></a>
|
|
<a href="https://rockeycoss.github.io/spo.github.io/"><img src="https://img.shields.io/badge/Project-Page-blue?style=for-the-badge" height=22.5></a>
|
|
|
|
## Abstract
|
|
<p>
|
|
Recently, Direct Preference Optimization (DPO) has extended its success from aligning large language models (LLMs) to aligning text-to-image diffusion models with human preferences.
|
|
Unlike most existing DPO methods that assume all diffusion steps share a consistent preference order with the final generated images, we argue that this assumption neglects step-specific denoising performance and that preference labels should be tailored to each step's contribution.
|
|
</p>
|
|
<p>
|
|
To address this limitation, we propose Step-aware Preference Optimization (SPO), a novel post-training approach that independently evaluates and adjusts the denoising performance at each step, using a <em>step-aware preference model</em> and a <em>step-wise resampler</em> to ensure accurate step-aware supervision.
|
|
Specifically, at each denoising step, we sample a pool of images, find a suitable win-lose pair, and, most importantly, randomly select a single image from the pool to initialize the next denoising step. This step-wise resampler process ensures the next win-lose image pair comes from the same image, making the win-lose comparison independent of the previous step. To assess the preferences at each step, we train a separate step-aware preference model that can be applied to both noisy and clean images.
|
|
</p>
|
|
<p>
|
|
Our experiments with Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms the latest Diffusion-DPO in aligning generated images with complex, detailed prompts and enhancing aesthetics, while also achieving more than 20× times faster in training efficiency. Code and model: <a ref="https://rockeycoss.github.io/spo.github.io/">https://rockeycoss.github.io/spo.github.io/</a>
|
|
</p>
|
|
|
|
## Model Description
|
|
The models in this repository are step-aware preference models used for fine-tuning SD v1.5 and SDXL. For more details, please visit our [GitHub repository](https://github.com/RockeyCoss/SPO).
|
|
|
|
## Citation
|
|
If you find our work or codebase useful, please consider giving us a star and citing our work.
|
|
```
|
|
@article{liang2024step,
|
|
title={Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step},
|
|
author={Liang, Zhanhao and Yuan, Yuhui and Gu, Shuyang and Chen, Bohan and Hang, Tiankai and Li, Ji and Zheng, Liang},
|
|
journal={arXiv preprint arXiv:2406.04314},
|
|
year={2024}
|
|
}
|
|
``` |
