SePPO: Semi-Policy Preference Optimization for Diffusion Alignment
Authors: name: Daoan Zhang; name: Guangchen Lan; name: Dong-Jun Han; name: Wenlin Yao; name: Xiaoman Pan; name: Hongming Zhang; name: Mingxiao Li; name: Pengcheng Chen; name: Yu Dong; name: Christopher Brinton; name: Jiebo Luo
Published: 2024-10-07 17:56:53

Abstract: Reinforcement learning from human feedback (RLHF) methods are emerging as a
way to fine-tune diffusion models (DMs) for visual generation. However,
commonly used on-policy strategies are limited by the generalization capability
of the reward model, while off-policy approaches require large amounts of
difficult-to-obtain paired human-annotated data, particularly in visual
generation tasks. To address the limitations of both on- and off-policy RLHF,
we propose a preference optimization method that aligns DMs with preferences
without relying on reward models or paired human-annotated data. Specifically,
we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO
leverages previous checkpoints as reference models while using them to generate
on-policy reference samples, which replace "losing images" in preference pairs.
This approach allows us to optimize using only off-policy "winning images."
Furthermore, we design a strategy for reference model selection that expands
the exploration in the policy space. Notably, we do not simply treat reference
samples as negative examples for learning. Instead, we design an anchor-based
criterion to assess whether the reference samples are likely to be winning or
losing images, allowing the model to selectively learn from the generated
reference samples. This approach mitigates performance degradation caused by
the uncertainty in reference sample quality. We validate SePPO across both
text-to-image and text-to-video benchmarks. SePPO surpasses all previous
approaches on the text-to-image benchmarks and also demonstrates outstanding
performance on the text-to-video benchmarks. Code will be released in
https://github.com/DwanZhang-AI/SePPO.