Title: Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance
Primary area: generative models
Keywords: Preference learning; Diffusion models; Text-to-image generation

Abstract: Aligning large-scale text-to-image diffusion models with nuanced human preferences remains a significant challenge. Direct preference optimization (DPO), while efficient and effective, often suffers from generalization gap in large-scale finetuning. We take inspiration from test-time guidance techniques and view preference alignment as a variant of classifier-free guidance (CFG), where a finetuned preference model serves as an external control signal. This perspective yields a simple and effective method that improves alignment with human preferences. To further improve generalization, we decouple preference learning into two modules trained on positive and negative samples, whose combination at inference can yield a more effective alignment signal. We quantitatively and qualitatively validate our approach on Stable Diffusion 1.5 and Stable Diffusion XL using standard image preference datasets such as Pick-a-Pic v2 and HPDv3.

TL;DR: Rethinking the role of CFG-style inference in preference learning for diffusion models.