Aligning Diffusion Models by Optimizing Human Utility
Authors: name: Shufan Li; name: Konstantinos Kallidromitis; name: Akash Gokul; name: Yusuke Kato; name: Kazuki Kozuka
Published: 2024-04-06 01:23:23

Abstract: We present Diffusion-KTO, a novel approach for aligning text-to-image
diffusion models by formulating the alignment objective as the maximization of
expected human utility. Since this objective applies to each generation
independently, Diffusion-KTO does not require collecting costly pairwise
preference data nor training a complex reward model. Instead, our objective
requires simple per-image binary feedback signals, e.g. likes or dislikes,
which are abundantly available. After fine-tuning using Diffusion-KTO,
text-to-image diffusion models exhibit superior performance compared to
existing techniques, including supervised fine-tuning and Diffusion-DPO, both
in terms of human judgment and automatic evaluation metrics such as PickScore
and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging
readily available per-image binary signals and broadens the applicability of
aligning text-to-image diffusion models with human preferences.