Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion
  Models
Authors: name: Minghao Fu; name: Guo-Hua Wang; name: Tianyu Cui; name: Qing-Guo Chen; name: Zhao Xu; name: Weihua Luo; name: Kaifu Zhang
Published: 2025-11-05 09:30:49

Abstract: Text-to-image diffusion models deliver high-quality images, yet aligning them
with human preferences remains challenging. We revisit diffusion-based Direct
Preference Optimization (DPO) for these models and identify a critical
pathology: enlarging the preference margin does not necessarily improve
generation quality. In particular, the standard Diffusion-DPO objective can
increase the reconstruction error of both winner and loser branches.
Consequently, degradation of the less-preferred outputs can become sufficiently
severe that the preferred branch is also adversely affected even as the margin
grows. To address this, we introduce Diffusion-SDPO, a safeguarded update rule
that preserves the winner by adaptively scaling the loser gradient according to
its alignment with the winner gradient. A first-order analysis yields a
closed-form scaling coefficient that guarantees the error of the preferred
output is non-increasing at each optimization step. Our method is simple,
model-agnostic, broadly compatible with existing DPO-style alignment frameworks
and adds only marginal computational overhead. Across standard text-to-image
benchmarks, Diffusion-SDPO delivers consistent gains over preference-learning
baselines on automated preference, aesthetic, and prompt alignment metrics.
Code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO.