Dual Caption Preference Optimization for Diffusion Models
Authors: name: Amir Saeidi, name: Yiran Luo, name: Agneet Chatterjee, name: Shamanthak Hegde, name: Bimsara Pathiraja, name: Yezhou Yang, name: Chitta Baral

Recent advancements in human preference optimization, originally developed
for Large Language Models (LLMs), have shown significant potential in improving
text-to-image diffusion models. These methods aim to learn the distribution of
preferred samples while distinguishing them from less preferred ones. However,
existing preference datasets often exhibit overlap between these distributions,
leading to a conflict distribution. Additionally, we identified that input
prompts contain irrelevant information for less preferred images, limiting the
denoising network's ability to accurately predict noise in preference
optimization methods, known as the irrelevant prompt issue. To address these
challenges, we propose Dual Caption Preference Optimization (DCPO), a novel
approach that utilizes two distinct captions to mitigate irrelevant prompts. To
tackle conflict distribution, we introduce the Pick-Double Caption dataset, a
modified version of Pick-a-Pic v2 with separate captions for preferred and less
preferred images. We further propose three different strategies for generating
distinct captions: captioning, perturbation, and hybrid methods. Our
experiments show that DCPO significantly improves image quality and relevance
to prompts, outperforming Stable Diffusion (SD) 2.1, SFT_Chosen, Diffusion-DPO,
and MaPO across multiple metrics, including Pickscore, HPSv2.1, GenEval,
CLIPscore, and ImageReward, fine-tuned on SD 2.1 as the backbone.