Dual Caption Preference Optimization for Diffusion Models
Authors: Amir Saeidi
Yiran Lawrence Luo
Agneet Chatterjee
Shamanthak Hegde
Bimsara Pathiraja
Yezhou Yang
Chitta Baral
Venue: Accepted by TMLR ()

Abstract: Recent advancements in human preference optimization, originally developed for Large Language Models (LLMs), have shown significant potential in improving text-to-image diffusion models. These methods aim to learn the distribution of preferred samples while distinguishing them from less preferred ones. However, within the existing preference datasets, the original caption often does not clearly favor the preferred image over the alternative, which weakens the supervision signal available during training. To address this issue, we introduce Dual Caption Preference Optimization (DCPO), a data augmentation and optimization framework that reinforces the learning signal by assigning two distinct captions to each preference pair. This encourages the model to better differentiate between preferred and less-preferred outcomes during training. We also construct Pick-Double Caption, a modified version of Pick-a-Pic v2 with separate captions for each image, and propose three different strategies for generating distinct captions: captioning, perturbation, and hybrid methods. Our experiments show that DCPO significantly improves image quality and relevance to prompts, outperforming Stable Diffusion (SD) 2.1, SFT_Chosen, Diffusion-DPO and MaPO across multiple metrics, including Pickscore, HPSv2.1, GenEval, CLIPscore, and ImageReward, fine-tuned on SD 2.1 as the backbone.

Amir Sacidi' Published in Transactions on Machine Learning Research (10/2025)
Dual Caption Preference Optimization for Diffusion Models
ssacidi Casu. edu
School of Computing and Augmented Intelligence
Arizona State University
Yiran Lawrence Luo"
yluo97@asu.co edu
School of Computing and Augmented Intelligence Arizona State University
Agneet Chatterjee
agnest@asu.ed edu
School of Computing and Augmented Intelligence Arizona State University
Shamanthak Hegde
shegde223@asu edu
School of Computing and Augmented Intellignnce
Arizona State University
Bimsara Pathiraja
bpathir @asu. edu
School of Computing and Augmented Intelligence Arizona State University
Yezhou Yang
yz. yz.yong@asu.edu
School of Computing and Augmented Intelligence Arizona State University
Chitta Baral
chitto Casu. edu
School Computing and Augmented Intelligence
Arizona State University
Reviewed on OpenReview: https: // openreview. net/ met/ forum? id= ruZks JBBd
Abstract
Recent advanccments in human preference optimization, originally developed for Large Lan- guage Models (LLMs), have shown significant potential in improving text-to-image diffusion models. These methods aim to learn the distribution of preferred samples while distinguish- ing them from less preferred ones. However, within the existing preference datasets, the original caption often does not clearly favor the preferred image over the alternative, which weakens the supervision signal available during training. To address this issue, introduce Dual Caption Preference Optimization (DCPO). data augmentation and optimiza- lion framework that reinforces the learning signal by assigning two distinct caplions to each preference pair. This encourages the model to better differentiate between preferred and less-preferred outcomes during training. We also construct Pick-Double Caption, a modified version of Pick-a-Pic v2 with separate captions for each image, and propose three different strategies for generating distinct captions: captioning, perturbation, and hybrid methods. Our experiments show that DCPO significantly improves image quality and rele- vance to prompts, outperforming Stable Diffusion (SD) 2.1, SFT Chosehes Diffusion-DPO and MaPO across multiple metrics, including Pickscore, HPSv2.1, GenEval, CLIPscore, and RewaeR fine-tuned on SD 2.1 as the backbone.
* Equal contribution. Correspondence ssaeidil Rasu.edu

---

Published in Transactions on Machine Learning Research (10/2025)
SD Bove
SFT chose
900
100
Difiuionn DPO
Maro
DCPOR (ours)
From for Par ComnaC alan
Figure 1: Sample images generated by different methods on the HPSv2, Geneval, and Pickscore benchmarks. After fine-tuning SD 2.1 with SFTChosen: Diffusion-DPO, MaPO, and DCPO on Pick-a-Piev2 and Pick- Double Caption datasets, DCPO produces images with notably higher preference and visual appeal (See more examples in Appendix H
1 Introduction
Image synthesis models (Rombach et al. 2022 Esser et al. 2024 have achieved remarkable advancements in generating photo-realistic and high-quality images. Text-conditioned diffussio (Song et al. 2020) models have led this progress due to their strong generalization abilities and proficiency modeling high-dimensional data distributions. As result, they have found wide range of applications in image editing (Brooks et al.,
2023), video generation (Wu et al. 2023a) and robotics (Carvalho et al., 2023 Consequently, efforts have locused on aligning them with human preferences, largeting specilic altribules like safety (Liu et al., (2024b). style (Everaert et al. 2023), and personalization (Ruiz et al. 2023), thereby improving their usability and adaptability.
Similar the aligument, process of Large Language Models (LLMs), aligning diffusion models involves two main steps: 1. Pre-training and Supervised Pine-Tuning (SFT). Recent fine-tuning based methods have been introduced to optimize diffusion modcls according to human preferences by leveraging Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al. 2022), the aim of which is maximize an explicit reward. However, challenges such fine-tuning separate reward model and reward hacking have led to the adoption of Direct Preference Optimization (DPO) (Rafailov et al. 2024) techniques like Diffusion-DPO (Wallace et al. 2024). Intuitively, Diffusion-DPO involves maximizing the difference between a preferred image and a less preferred image for a given prompt.
Although DPO-based methods are effective in comparison SFT-based approaches, applying direct op- timization in multi-modal settings presents certain challenges. Current preference optimization datasets consist of a preferred (z") and less preferred (x') image for given prompt (c). Ideally, x" should show higher corrclation with compared to x'. However, we find that in current datascts, the preferred and less-preferred images exhibit a substantial overlap in their semantic distributions for the given prompl. e, which we refer to as semantic overlap in the data. Additionally, irrelevant information prompt e restricts the U-Net's ability to predict noises from x' in the diffusion reverse process, which we refer to as irrelevant prompts. This entails that there is a lack of sufficient distinguishing features between the two pairs (x" c). (xr,, c), thereby increasing the complexity of the optimization process.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Preferred Image
DCPO-c Captioning Method
VLM
Caption zt
Caption 2nd
"The image shows a chessboard with white pieces set up a lunar surface, suggesting chess fournament the
"The image shows a chessboard: with white pieces set up lunar surface, suggesting chess fournamen the
Prompt c "playing chess foumament on the moon."
LLM
Weak Perfurbation
Caption z!
Caption x
"Two astronauts on moon, standing near chess set, with the moon's surface as the backdrop."
DIPPER (TS-XXL)
"The moon landing. Two: moonwalkers near
chessboard. The moon
surface backdrop.
Pick-Double Caption Dataset (xgg a/g. 2²,
Preferred Image 2
Medium
LLM
Perlurbaion
Medium
Stoong Pertubaaiion
Caption
'Caption's n "playing chess foumament on the moon."
Caption z "The moon chess toumamentt
Caption z
Caption =j Playiig chess on the
"Two lunar explorers chessboard, moonscape as backdrop:
DIPPER (TS-XXL)
"Onee upon a time on the moon, there was chess foummamet "
DCPO-p Perturbation Method
DCPO-h Hybrid Method
Figure 2: The DCPO pipeline in 3 variants: DCPO-c, DCPO-p, and DCPO-h, all of which require a duo of a captioned preferred image (x0 and a captioned prss-preeerred image (z z*). DCPO-c (Top Left): We use a captioning model to generate distinctive captions respectively for images z and zój given the shared prompt c. DCPO-p (Bottom Left): We take prompt e as the caption for image zb. then we use a Large Language Model (LLM) to generate semantically perturbed prompt z given prompt
the caption for image an- DCPO-h (Right): hybrid method where the generated caption af now perturbed into ?p for image wh- Our Double Caption Dataset discussed in Section 4.1 is constructed using DCPO-c.
To address the aforementioned bottleneck, we propose DCPO: Dual Caption Preference Optimization, novel preference optimization technique designed to align diffusion models by utilizing two distinct captions corresponding to the preferred and less preferred image. DCPO broadly consists of two steps: a text generation framework that develops better-aligned captions and novel objective function that utilizes these captions as part of the training process.
The text generation framework aims to mitigate the semanlic overlop between the distributions of preferred and less less-preferred images, given the same prompts, in existing datasets. We hypothesize that prompt e does not serve the optimal signal for optimization because they do not convey the reasons why image is preferred or dis- tis-prefe;r:;; based the above, devise the following techniques to generate better aligned captions. The first method involves using captioning model (+<=*(x',,)) which generates new prompt z' based on an image x' and the original prompt where i E (w,1). The second method introduces perturbation techniques f, such that c = 2 z = f(c); ie., generating 2", to represent the less preferred image, considering the original prompt e as the prompt aligned with the preferred image. We investigate multiple semantic variants of f, where cach variant differs in the degree of perturbation applied to the original caption c. Finally, we also explore a hybrid combination of the above methods, where we combine the strong prior of the captioning model and the efficient nature of the perturbation method. All the above methods are designed to generate captions that effectively discriminate between the preferred and less preferred images.
We introduce a novel objective function that allows DCPO to incorporate z" and af into its optimization process. Specifically, during optimization, the policy model pa increases the likelihood of the preferred image x conditioned on the prompt n", while simultancously decreasing the likelihood of the less preferred image z' conditioned on the prompt z'. The results in Tables [nd demonstrate that DCPO consistently outperforms other methods, with notable improvements of +0.21 in Pickscore, +0.31 in HPSv2.1, +1.8 in
3

---

Published in Transactions on Machine Learning Research (10/2025)
normalized ImageReward, +0.15 in CLIPscore, and +2% in GenEval. Additionally, DCPO achieved 58% in general preference and 66% in visual appeal compared to Diflusion-DPO on the PartiPrompts dataset, evaluated by GPT-4o (see Figure 7
In summary, our contributions are as follows
Double Caption Generation: We introduce the Captioning and Perturbation methods address the issue of overlapping semantic distributions in current datasets, as illustrated in Figure In the Captioning method, we employ state-of-the-art models like LLaVA (Liu et al. 2024a and Emu2 (Sun et al. 2024) to generale caption based on the image and prompt c. Additionally, we use DIPPER (Krishna et al.] 2024), paraphrase generation model built by fine-tuning the T5-XXL model to create three levels perturbation from the prompt c.
Dual Caption Preference Optimization (DCPO): We propose DCPO, a modified version of Diffusion-DDP -PO, that leverages the U-Net encoder embedding space for preference optimization. This method enhances diffusion models by aligning them more closely with human preferences, using two distinet captions for the preferred and less preferred images during optimization.
Improved Model Porformance: We demonstrate that our approach significantly outperforms SD 2.1, SFT, Diffusion-DPO, and MaPO across metrics such Pickscore, HPSv2.1, GenEval, CLIPscore, normalized ImageReward, and GPT-4o (Achiam et al. 2023) evaluations.
2 Preliminary
Aligning generative model typically involves fine-tuning it to produce outputs that are more aligned with human preferences. Estimating the reward model r based on human preference generally challenging, as do not. have direct access to the reward model. However, if assume the availability of ranked data generated under given condition c, where zi > zgle (with zo representing the preferred sample and z the less-preferred sample), we can apply the Bradlcy-Terry theory to model these preferences. The Bradlcy-Terry (BT) model expresses human preferences as follows:
PBr aglc) = o(((c, - r(c)
(1)
where a(-) the sigmoid function, and r(zo, c) derivod from a neural network parameterized by Φ. Subsequently, o is estimated by maximum likelihood training for binary classification follows:
Ler(Ø) = -Ee,ag w [
(2)
where prompt e and data pair (xy) wg) are sourced from a human-annotated dataset.
This approach to reward modeling has gained popularity in aligning large language models, particularly when combined with reinforcement learning (RL) techniques like proximal policy optimization (PPO) (Schulman)
et al. 2017 to fine-tune the model based on rewards learned from human preferences, known as Reinforce- ment Learning from Human Feedback (RLHF) (Ouyang et al. (022). The goal of RLHF is to optimize the conditional distribution p(æalc) (where ~ Dz) such that the reward model r(c, xo) is maximized, while keeping the policy model within the desired distribution using a KL-divergence term to ensure it remains reachable under the following objective:
Pew.D...orppptta 3TD po
(3)
where 3 controls how far the policy model po can deviate from the reference model Pref-
It can be demonstrated that the objective in Equation converges to the following policy model:
4

---

Published in Transactions on Machine Learning Research (10/2025)
(o(oo))) =
where is the partition function.
The training objective for pe, inspired by DPO, has been derived to be equivalent to Equation 2 without the need for an explicit reward model c),, Instead, RLHF learns directly from the preference data (c, x()) ) ~ D:
(0) = -Ba,y .nd log o log (x
po(zglle) Prot (c)
3 log
(5)
(rer(=0)())
where o(-) the sigmoid function.
Through this e-parameeerization, instead of optimizing the reward function before applying reinforcement learning, the RLHF method directly optimizes the conditional distribution po (zolc).
3 Method
In this section, we present the semantic overlop issue preference datasets, where the semantic distribution of preferred and less- peess-refereed images generated from the same prompt exhibit significant similarity. We also explain the irrelevant prompt issue found previous direct preference optimization methods. To address these challenges, we propose Dual Caption Preference Optimization (DCPO), a method that uses distinet captions for preferred and less preferred images to improve diffusion model aligmment.
3.1 The Challenges
Generally, to optimize Large Language Model (LLM) using preference algorithms, need dataset D " , }, where ydd and y' represent the preferred and less preferred responses to given prompt C. Recent studies (Sun et al. 2025 Deng et al. 2025) have shown that high semantic similarity between preferred and less-preferred responses negatively impacts preference optimization and should be carefully considered during dataset curation. Similarly, in diffusson model alignment, the semantic distributions of preferred and less preferred images should be distinct for the same prompt c. However, our analysis shows substantial overlap between these distributions, which we call semantic overlap, as illustrated in Figure For more details, refer to Appendix B.1
Another issue emerges when direct preference optimizes diffu- sion model. In the reverse denoising process, the U-Net model predicts noise for both preferred and less preferred images us- ing the same prompt c. As prompt c is more relevant to the preferred image, becomes less effective for predicting the less preferred one, leading to reduced performance. We call this the irrelevant prompts problem.
25.418, 4M 26.746, Au 1.328
Occurence
3.2 DCPO: Dual Caption Preference Optimization
prompt c. less preferred XG preferred
Motivaled by the semantic overlap and irrelevand prompls is- sues, propose DCPO, that optimizes diffusion models using two distinct captions. DCPO is a refined version of Diffusion-DPO designed to ad- dress these challenges. More details are in Appendix A
10
15
25
30
35
45
50
CLIPScore
preference optimization method
Figure 3: The semantic overlap issue in the Pick-a-Pic v2 dataset. μ³ and μM represent the average CLIPscore of preferred and less preferred images for prompt c, respectively. Also, Aμ shows the difference between the distributions.
We start with fixed dataset D = fc,=," m/); where each entry contains prompt and a pair of images generated by reference model Pref- The human labels indicate preference,
5

---

Published in Transactions on Machine Learning Research (10/2025)
with zi preferred over z We assume the existence of model Ro(z(c, which generates a caption & given prompt с and an image Using this model, we transform the dataset into D' {= 2", 500 ; ; where z" and s are captions for the preferred image Tp and the less- ress-preferred image z/g respectively. Our goal is to train a new model Po, aligned with human preferences, to generate outputs that are more desirable than those produced by the reference model.
The objective of RLHIF maximize the reward r(c, z0) for the reverse process pe p=(zor))=) while maintaining alignment with the original reference reverse process distribution. Building prior work (Wallace et al. 2024h, the DCPO objective is defined by direct optimization through the conditional distribution pe p2(20 as follows:
Po Po(x0T12")
-log po (a
(DC = (1 wD
(obb-/ log a B poo ="", pp log log Prof Prof("0.712 Pret (
(6)
where log(-] is the sigmoid function.
However, as noted in Diffusion-DPO (Wallace et al. 2024). the sampling process 21.7 ~ p(r1er (ro) is inefficient and intractable. To overcome this, we follow a similar approach by applying Jensen's inequality and utilizing the convexity of the - log(-) function to bring the expectation outside. By approximating the reverse process po (xp, 2) with the forward process g(Z1r1/z)) and through algebraic manipulation and simplification, the DCPO loss can be expressed as:
(fc = -- [Ry -Baay (nu((,))) "t),:tttttt,,, ~o(a"ll (a3
(a/ 2., -012--11e log a (3Tw(d)) (clc"" =0, ,,,,) ||c (ref(xx" =" 12)113 (llc
108--llc-rrtaate -ooaa 2 +(,11)))
(7)
where 27 = ax²t + oiE", and ex (N, I) is a sample drawn from g(x)(=0)) A = af/o? represents the signal-to-noise ratio, and w({,) weighting function.
To optimize diffusion model using DCPO, dataset D= (=", "0, =0 paired with the images. However, the current preference dataset only contains prompts and image pairs without captions. To address this, propose three methods for generating captions and introduce a new high-quality dataset, Pick- Double Caption, which provides specific captions for each image, based on Pick-a-Pic v2 (Kirstain et al. 2023).
w0) =
required, where captions are
3.2.1 DCPO-c: Captioning Method
In this method, the captioning model Qo(2lc, generates the caption based the image and the original prompt As result, obtain preferred caption =" "0(="le,<>") for the preferred image and less preferred caption &' Qo(2"|e, x²) for the less preferred image, as illustrated in a sample in Figure Thus, based on the generated captions " and 2 we can optimize a diffiusion model using the DCPO method.
2
In the experiment section, we evaluate the performance of DCPO-c and demonstrate that this method effectively mitigates the semantic overlap by creating two differentiable distributions. However, the question of how much divergence is needed between the two distributions remains. To investigate this, we propose Hypothesis 1.
Hyphothesis 1. Let d(2,z) represent the semantic distribution between a caption & and an image x, with p being the mean of the distribution d, and Aμ = =u(d(2) x²())) u(d(2), x/)) as the difference between the two distributions. Increasing Ap between the preferred and less-preferred image distributions in a preference dalaset beyond threshold (i.e., Ap > 1), can improve the performance of the model po-

---

Published in Transactions on Machine Learning Research (10/2025)
uld") 30.856, utd') 26.481, Auid", d)) 4.375
d': caption e', less preferred image x caption 25, prefeered image xf
(a)
اld 24.416. 4n1 4 6.499
perturbed caption x less preferre image X
Occurence
(b)
Lid 380 di d 6.755
% perturbedd castion (medium)
less preferred Image
(c)
M 22 566. wuid d - 82267
di perturbed caption (stre ng) z
less preferred Image x/
(d)
10 15
25
30
35
40
45
50
CLIPScore
Figure 4: Effect of the perturbation method on semantic distributions in terms of CLIPScore. (a) shows the distributions that feature the captions z" and s' generated by the LLaVA model, while (b), (c), and (d) represent different levels of perturbation on top of the caption af. The figure demonstrates that as the level of perturbation increases, the distance between the distributions of captions =" and at increases. For more details on the perturbation method, refer Appendix D
Our hypothesis suggests that increasing the distance between the two distributions up to certain threshold can improve alignment performance. To examine this, we propose the perturbation mothod to control the distance between the two distributions, represented by Ap.
3.2.2 DCPO-p: Perturbation Method
While using captioning model is an effective way to address the semantic overlap, it risks deviating from the original distribution of prompt e, and the distributions of preferred and less preferred images may still remain close. To tackle these issues, we propose a perturbation method. In this approach, we assume that prompt e highly relevant to the preferred image = and aim generate less relevant caption, denoted as cp. based on prompt c. To achieve this, we use the model Wo(eple), which generates perturbed version of prompt e, altering its semantic meaning. In this framework, prompt. e corresponds to the preferred caption z" (c ))), while the perturbed prompt Cp represents the less- preserreferee caption z (cp =')==)).
For the perturbation model Wg. we choose the highly customizeable DIPPER model (Krishna et al. 2024 which is built. by fine-tuning T5-XXT. (Chung et al. 2022), to produce a degraded version of the prompt c. DIPPER is equipped with flexible steerability for controllable textual perturbation, which supports adjustable hyperparameters that control both lexical diversity and word order variation at the output.
We define three levels of perturbation strength: 1) Weak: where prompt Gp has high semantic similarity to prompt c, with minimal differences. 2) Medium: where the semantic difference between prompt cp and is more pronounced than in the weak level. 3) Strong: where the majority of the semantics in prompt Cp differ significantly from prompt c.
Perturbation Selection Process. Given each input prompt c, we first perform a hyperparameter sweep over DIPPER's controllable dimensions to construct a larger diverse pool of perturbed prompt candidates. We then select the candidates that are well-scparated along the CLIPScore scale from the pool, ensuring clear semantic gradient while always preserving the original input as the one with the highest image- text aligument in terms of CLIPScore affinity. The three resulting perturbed prompts Cp in levels of Weak, Medium, and Strong, respectively - reflect progressively degraded semantic fidelity relative to the associated image of the input, visualized as the shifting distributions in terms of CLIPScore semantic similarity in Figure For further details on the perturbation method, see Appendix D
Compared with DCPO-c, DCPO-p is, first of all, less computationally costly. As described in Figure 2 DCPO-p requires only the textual perturbation as the less dess-preferred caption. In addition, by using the original prompt e directly the preferred caption, DCPO-p keeps the original distribution at bay, mitigating
7

---

Published in Transactions on Machine Learning Research (10/2025)
DCPO-c's risk where both the preferred and the less preferred may deviate together. However, we observe that the quality of the captions in DCPO-c outperforms that of the original prompt e, shown in Table in Appendix C Based on this observation, we propose a hybrid method to improve the alignment performance by combining the captioning and perturbation techniques from DCPO-c and DCPO-p, altogether.
3.2.3 DCPO-h: Hybrid Method
In this method, instead of perturbing the prompt e, we perturb the caption z generated by the model Qs(2/2, c) based on the image æ and prompt c. As discussed in Scction 3.2.1 the goal of the perturbation method is to increase the distance between the two distributions. However, the correlation between the image zg and prompt e significantly impacts alignment performance. Therefore, we propose Hypothesis 2.
Hypothesis 2. Let S(c, represent the corvelation score between prompl and image x, and ((po(ci,c)) denote the performance of model pe optimized on captions ej and 0g with DCPO, where We is the perturbation P(polc, WW(cple))),
model. If S(2,x)) > s(c,2), then P(po(2", , W Wo(="12<'))
In Section 4.3
provide experimental evidence supporting Hypothesis 2 and investigale the potential of W (2)) 22) as the less- preferree caption 2², instead of zy ~ WW (2(2"12") as originally proposed in Hypothesis 2.
using z/p
4 Experiments
We fine-tuned the U-Net model of Stable Diffusion (SD) 2.1 using DCPO on the Pick-Double Caption dataset and compared it with SD 2.1 models fine- tuned with Chosen: Diffusion-DPO, and MaPO on Pick-a-Piev2 in various metries. We first describe the Pick-Double Caption dataset and compare it to Pick-a-Piev2. Subsequently, we provide in-depth analysis of the results. Details on fine-tuning are in Appendix F and further comparisons are in Ap-
Table 1: Results on PickScore, HPSv2.1, ImageRe-
ward (normalized), and CLIPScore. We show that DCPO significantly improves Pickscore, HPSv2.1, and ImageReward.
Pick (1) Score
HPSv2.1 (1)
Reward
Image
(1)
Score
CLIP (t)
Results from other methods
SD
20.30
25.17
SPTcssn DiflasionDPPO
20.35
25.09
56i-4
26.98
20.36
25.10
566.4
26.98
MaPO
20.31
25:31
55.6
26.78
pendix
Results from our methods
DCPO-e (LLaVA)
20.46
25.10
56.5
27.00
DCPO-e (Kmuz)
20.46
25.06
566.6
26.997
DCPO-p
20.28
25.42
54.2
26.98
4.1 Pick-Double Caption Dataset
DCPO-b (LL&VA)
20.57
25.62
58.2
27.13
Motivated by the semandic overlap observed in previous preference datasets, we applied the captioning method described in Section 3.2.T to generate unique captions for each image in the Pick-a-Pic v2 dataset. For the Pick-Double Caption dataset, we sampled 20,000 instances from Pick-a-Pic v2 and cleaned the samples detailed in Appendix C We then employed two stato-of-the-art captioning models, LLaVa-1.6- 34B and Emu2-37B, to generate captions for both the preferred and less preferred images, shown in Figure
To generate the captions, we used two different prompting strategics: 1) Conditional prompt: where the model was explicitly instructed to generate caption for image x based on the given prompt c, and 2) Non-conditional prompt: where the model provided a general description of the image in one sentence without referring specific prompt. More details are in Appendix C
We evaluated the captions generated by LLaVA and Emu2 using CLIPscore, which revealed several key insights. LLaVA produced captions that have more correlation with the images for both preferred and less preferred samples compared to Emu2 and the original captions, although LLaVA's 's captions were significantly longer (see Table g in Appendix Cp. Models fine-tuned on captions from the conditional prompt strategy outperformed those using the non-conditional approach, though the conditional prompt captions were twice as long. Interestingly, despite Emu2 generating much shorter captions, the models fine-tuned on Emu2 were comparable to those fine-tuned on the original prompts from Pick-a-Pic v2.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Results on the GenEval Benchmark. DCPO contributes model performance in generating the correct number of objects, improving image quality in terms of colors, and constructing attributes accurately.
Method
Overall
Single object
Two objects
Counting
Colors
Position
Attribute binding
Results from other methods
SD 2.1²
0.4775
0.96
0.52
0.35
0.80
0.09
0.15
SFTThheen
0.4797
1.00
0.42
0.42
0.81
0.07
0.14
Diffusion-DPe
0.4857
0.99
0.48
0.46
0.83
0.04
0.11
MaPO
0.4938
1.00
0.45
0.45
0.80
0.09
0.16
Results from our methods
DCPO-c (LLaVA)
0.4971
1.00
0.43
0.53
0.85
0.02
0.14
DCPO-c (Emn?)
0.4925
1.00
0.41
0.04
DCPO-p
0.4906
1.00
0.41
0.50
0.83
0.03
0.17
DCPO-h (LLaVA)
0.5100
0.99
0.51
0.54
0.84
0.05
0.14
Table 3:
Performance comparison of DCPO-h and DCPO-p across different perturbation levels. turbation method has a strong impact on captions that are more closely correlated with images.
The per-
Method
Pair Caption
Perturbed Level
Pickscore (1)
HPSv2.1 (t)
ImageReward (1)
CLIPscore (1)
GenEval (1)
DCPO-p
(c,0g) (2")
weak
20.28
25.42
54.20
26.98
0.4906
DCPO-b
weak
20.35
25.61
57.70
27.07
0.5070
DCPO-h
(g%)
27.15
DCPO-p
(c.0g)
medium
20.21
25.34
26.87
DCPO-h
(a"
medium
20.59
25.73
27.12
0,5008
DCPO-b
(a" (.))
medium
20.37
25.62
58.20
27.13
0.5100
DCPO-p
(c.cy)
strong
20.31
25.06
5-1.00
27.03
0.4868
DCPO-b
strong
20.37
25.27
57.43 43
27.15 18
0.5110
DCPO-h
strong
20.58
25.43
57.90
27.21
0.4993
A key challenge is generating captions for the less preferred images using the captioning method. We observed that in both prompting strategies, the captions for the preferred images more aligned with the original prompt distribution. However, the non-conditional prompt strategy often produces captions for less preferred images that are out-of-distribation (OOD) from the original prompt in most cases. We will explore this further in Section 4.3
Finally, we observe that the key advantage the Pick-Double Caption dataset is the greater difference CLIPscore (Aμ) between preferred and less preferred images compared the original prompts. Specifically, while the original prompt has a Aμ of 1.3, LLaVA shows a much larger difference at 4.3, and Emu2 at 2.8. This increased gap reflects improved alignment performance in models fine-tuned on this dataset, indicating that the captioning method mitigates the semantic overlap.
4.2
Performance Comparisons
As shown in Table 1 evaluated all methods 2,500 unique prompts from the Pick-a-Piev2 (Kistaann
et al. 2023) dataset, measuring performance using Pickscore (Kirstain et al. 2023). CLIPscore (Hessel et al. 2022). and Normalized ImageReward (Xu et al., 2023). We also generated images from 3,200 prompts the HPSv2 (Wu et. al. 2023b) benchmark and evaluated them using the HPSv2.1 model. To provide a more fine-grained evaluation, in Table 2
we also compared the methods using GenEval [Ghosh et al., 2023). focusing on how well the fine-tuned models generated images with the correct number of objects, accurate colors, and proper object positioning.
We compared our different variants of DCPO, including the captioning (DCPO-c), perturbation (DCPO-p). and hybrid (DCPO-h) methods, with other approaches, as outlined in Section 3.2 For more information on the fine- fine-tuing process of the models, refer to Appendix F
Note that we all the models the same seeds to fair comparison.

---

Published in Transactions on Machine Learning Research (10/2025)
DCPO-c (Baseline)
DCPD-h (Perturb. Weak)
DCPO-h (Perturb. Medium)
DCPO-h (Perturb. Strang)
26.25
Picksore
20.65
26.00
20.60
58
25.50
20.50
25:25
20:45-
CUPscore
27.2
GenEval
awarza
40
27.1
50-
27.0
48-
6
Qutz"
Auiz", 2))
buiz",
Figure 5: Performance comparison of DCPO-c and DCPO-h on different perturbation levels. We plotted regression lines for the four models, showing that as Ap increases, performance improves but drops after a threshold (orange boundary).
The results in Tables and 2 show that DCPO-h significantly outperforms the best scores from other methods, with improvements of +0.21 in Pickscore, +0.31 in HPSv2.1, +1.8 in ImageReward, +0.15 in CLIPscore, and +2% GenEval. Additionally, the results demonstrate that DCPO-c outperforms all other methods on GvnEval, Pickscore, and CLIPscore. While DCPO-p performs slightly worse than DCPO-c, it still exceeds SD 2.1, SFT, Diffusion-DPO, and MaPO GenEval. However, its scores on ImageReward and Pickscore suggest that it underperforms compared to the other approaches. Importantly, DCPO-p shows significant improvement over the other methods on HPSv2.1, highlighting the effectiveness of the perturbation method.
4.3 Ablation Studies and Analyses
we defined three levels of perturbation: weak, medium, and strong. In Hypothesis 1, we proposed that increasing the distance between the distributions of preferred and less preferred images Aμ improves model alignment performance. To explore this, we fine-tuned SD 2.1 using the DCPO-h method with three levels perturbation applied to the less preferred captions z' generated by LLaVA. The results in Figure 5 show that increasing the distance Ap between the two distributions enhances performance. However, this distance must be controlled and kept below threshold t, hyperparameter that may vary depending the task. These findings support our hypothesis.
Support of Hypothesis 1. As described in Section 3.2.2
Support of Hypothesis 2. To illustrate the impact of the correlation between the prompt and image x on the perturbation method, we perturbed both the original prompt and the less preferred caption z", generated by the model Qo. where =" ~ generated by Qs for the preferred image as the preferred caption, z" ~ Q(= a c). In this case, we assume Q+ LLaVA and Wa DIPPER. The results in Table g in Appendix C show that the caption generated by LL more correlated with the image x than the original prompt e, indicating that 5(2,x) s(c,x). Based on the results in Table 3. we conclude that perturbing more correlated captions leads to better performance.
c)). At the same time, we kept the caption
In- vs. Out-of Distribution. We evaluated DCPO on in-distribution and out-of-distribution (OOD) data. As discussed in Section 4.1 the captioning model can generate OOD captions. To explore this, we fine-tuned SD 2.1 with DCPO-h using LLaVA and Emu2 captions at a medium perturbation level. Figure 0 shows that in-distribution data significantly improve alignment performance, while OOD results for LLaVA in GenEval, Pickscore, and CLIPscore are comparable Diffusion-DPP DPO. Similar behavior was observed for DCPO-c, as noted in Appendix
10

---

Published in Transactions on Machine Learning Research (10/2025)
GenFval
Pickscore
HPSv2.1
ImageRoward
CliPscore
0.52
21.0
26.0
59
25.62
0.5100
25.5
0.51
25.0-
25.0-
27.2
6.6952
20.6- 20.57
56-
27.05
27.06
0.4017
54.27
27.0-
0.49
20.4
20:36
20.32
24.0
21.64
26.89
20:25
23.5-
23.40
53-
23.0-
23.01
52.40
26,8
Emiuk
Emat out
Latt
Emiz-in
Emit ou
LLav
Emi2-
Emiz out
Llav
Uatd
Emi2-
Emaz out
Laak
Emi
Emauz out
Figure 6: Comparison of DCPO-h performance on in-distribution and out-of-distribution data.
Table 4: Performance comparison of DCPO and Diffusion-DPO fine-tuned on the dataset. While larger captions improve the performance of Diffusion-PPO, DCPO-h still significantly out-
Caption
performs Diffusion-DPO.
Method
Input Prompt
Token Length (Avg)
Pickscore (1)
HPSv2.1 (1)
ImageReward (t) CLIPscore (t)
GenEval (t)
Diffusion-DDD DPO
prompt
15.95
20.36
25.10
56.4
26,98
0,4857
DiffusionDDPO
csption 2" (LLaVA)
32,32
20.40
25.19
56.6
27.10
0.4958
Diffusion DPO
caption (Emm2)
7.75
20.36
25.08
56.3
26,98
0.4960
DCPD-h (LJ&VA)
Pair (art A)
(32.32, 31.17)
20.57
25.62
58.2
27.13
0.610U
DCPO-b (LLaVA)
Pair (5t 2g)
(32.32, 27.01)
25.27
27.18
0.5110
Effectiveness of the DCPO. Our analysis shows that LLaVA captions are twice the length of the original prompt raising the question of whether DCPO's improvement is due to data quality or the optimization melhod. To explore this, we fine-tuned SD 2.1 with Diflusion-DPO using LLaVA and Emu2 captions instead of the original prompt. The results in Table show that models fine-tuned on LLaVA captions outperform Diffusion-DPO with the original prompt. However, DCPO-h still surpasses the new Diffusion-DPP models, demonstrating the effectiveness of the proposed optimization algorithm.
Hyperparameter Tuning. We conducted systematic hyperparameter study for fair comparison across DCPO and all baselines. Under controlled protocol with batch size fixed at 128 and learning rate fixed at (1 x 10-8), we swept / for DCPO and DPO over the set [500, 1000, 2000, 2500, 5000). DCPO achieved its strongest performance at 3 value of 5000, outperforming alternatives on Pickscore, GenEval, HPSv2.1, ImageReward, and CLIPscore, while Diffusion-DPO reached its optimum at a B value of 2000. SFT performed best with batch size 64 and learning rate (1 x 10-9). MaPO attained its optimum at a much smaller coefficient, with a value of 0.01, indicating a preference for lighter regularization relative to DCPO and DPO. These outcomes align with the objectives of each method, where MaPO's margin-oriented formulation is sensitive to over- weightigg the margin and thus favors small B, DCPO's diffusiinn-cnniisennt objective with dual captions scales the preference signal while preserving stable updates and thus benefits from larger 3, and Diffusion-DPO's balance between the preference term and a reference-model KL naturally lavors a moderate 3. For additional details on the hyperparameter luning process refer to Appendix
DCPO-h vs Diffusion-DPO on GPT-4o Judgment. We evaluated DCPO-h and Diffusion-DPO using GPT-4o on the PartiPrompls benchmark, consisting of 1,632 prompts. GPT-4o assessed images based on three criteria: Q1) General Preference (Which image do you prefer given the prompt?), Q2) Visual Appeal Whhich image is more visually appealing and Q3) Prompt Alignment (Which image better fits the text descriplion?). As shown in Figure| 7 DCPO-h outperformed Dillusion-DPO in Q1 and Q2, with win rates of 58% and 66%. Beyond GPT-4o, we conducted a small human study for Q2 to measure the alignment of GPT-40 with human preference; results in Table 14 in Appendix G indicate that GPT-4o's judgments align closely with human preferences. See Appendix for more details.
Alternative MLLM-as-a-Judge for Controllable Perturbation. We use CLIPScore as the selection mechanism for incorporating the three levels of perturbations used in our PickDouble dataset. To confirm the effectiveness of CLIPSSore--s-a-Juuue perturbation selection, we further conducted a series of controlled
11

---

Published in Transactions on Machine Learning Research (10/2025)
Diffusion-
(ours)
(ours)
Wi Pate
Loss Rale
PartiPrompts:
Offfulloo DPD
DCPO
PartiPrompte: Visual Appeal
66
Prompt aligmmen
'A racessn weettng formul clalhws, meeeing tophat and hoeding cone. The racooon is holsing garüage bug OV paestng fiee atyle of pinal art.
'A mixdd merla image with photograph of woman with long orange hoi over background that in sholch of city skyliee"
GPT-4o
Figure 7: (Left) PartiPrompts benchmark results for three evaluation questions, voted by GPT-4o. (Right) Qualitative comparison between DCPO-h and Diffusion-DPO fine-tuned on SD 2.1. DCPO-h shows better prompt. adherence and realism, with outputs that align more closely with human preferences. emphasizing high contrast, vivid colors, fine detail, and well-focused composition.
20.50
0.51
30.00
25.b
9.50
57.00 00
27.00
24.75
25.00
20
0.49
641
218
53.00
$5.00
27,50
14:77
25:1
3500
25.9
22.45
23.00
2044
2344
Cower Clue
Owen 488-9
Owen 4-0-2
cLa
LANI 6226
Olear Chae
Owent 500-2
L
GLPScO
Olear chae
Owned 548-
4avtt 4-98
CLPStOo
Oheen chee
Figure & Benchmarking results by alternative perturbation selection mechanisins (-LI mod- els) alongside our original CLIPScore- based judge. All experiments are under the same SD 2.1 and DCPO-h + LLaVa captions sctting. For clarity, we only show the averages across all perturbation levels, per pertur- bation judge and benchmark.
ablation studies using perturbations ranked by alternative MLLM-basod judges inspired by Chen et al. (2024). We experimented with the following judge models Qwen Chat, Qwen2.5- Lla Va1.6-7B, and Lla Val.6- 1.66 13B. The results in Figure show that all MLLM-as-a-Judge variants consistently underperform compared to our CTIPScore-based baselines across all five downstream benchmarks, regardless of the judge model's s scale or the granularity of its perturbation strengths. This further validates that our original method identifies more effect perturbation ions (i.e., modified prompls) that lead to improved downstream performance across multiple benchmarks. More details can be found in Appendix D
DCPO vs Online and Iterative Methods. Although DCPO is an offline method, we compare it against online and iterative approaches such as ScPPO Zhang ct al. 2024 and SPO Liang et al. (2025) to assess eveectiveness. Under a small budget of optimization steps and limited training dala, DCPO consistently outperforms both, demonstrating superior sample and compute efficiency. By contrast, SPO typically re- quires substantially more training itcrations and data reach similar performance, and ScPPO depends on iterative online updates to approach comparable results. These findings indicate that DCPO achieves strong alignment with fewer resources while avoiding the complexity of online optimization. Refer to Appendix B.3 for more details.
5 Related Works
Aligning Diffusion Models. Recent advances in preference alignment for text-to-image diffusion models
show that RL-free methods (Yang et al.
2024a Li ct al.
2024 Yuan et al.) 2024
Gambashidze et al. 2024
Park el al. 2024) outperform RI-based ones [Fan & Lee 2023
Fan et al., 2023 Hao el al.) 2023 Lee el al.
2023 Prabhudesai et al. 2024 Black et al. 2024 Clark et al. 2024] by removing the need for explicit reward
models. Methods like Diffusion-DPO Waallace et al. 2024). which adapts DPO (Rafailov et al. 2024). and Diffusion-KII (Li et al. 2024), which uses binary feedback instead of pairwise data, streamline alignment.
MaPO (Hong et al.] 2024 further increases flexibility by eliminating the dependency on reference models.
12

---

Published in Transactions on Machine Learning Research (10/2025)
However, these methods often align based on single- image pairs, leading to issues with irrelevant prompts (see Section 3. While online preference optimization shows strong performance (Fan et al. 2023 Yang et al. 2024b; Lou et al.) 2024), it requires a reward model and more optimization steps, making it vulnerable to reward hacking. In this work, we focus on comparing DCPO with offline methods that do not rely on extra models for ranking of diffusion outputs.
Text-to-image Preference Datasets. Text-to-image image preference datasets commonly involve the text prompt to generate the images, and two (Wu et al. 2023c) and HPSv2 (Wu et al.) 2023b) create multiple images using series of image generation models for a single prompt, and the images are ranked according to real-world human preferences. Moreover, a classifier is trained using the gathered preference dataset, which can be used as a metric for image-aligning tasks. Also, Pick-a-Pie v2 (Kirstain et al. 2023 follows a similar structure to create a pairwise preference dataset along with their CLIP (Radford et al. 2021] based scoring function, Pickscore. While these datasets are carefully created, having only one prompt for both or all the images introduces semantic overlap, which will be further discussed in Section 3.1 For this reason, we modified the Pick-a-Pic v2 dataset. using recaptioning and perturbation methods to improve image alignment performance.
images ranked according to human preference. HPS
6 Conclusion
In this paper, present novel preference optimization method for aligning text-to-image diffusion models called Dual Caption Preference Optimization (DCPO). We tackle two major challenges in previous preference datasets and optimization algorithms: the semantic overlap and irrelevant prompt. To overcome these issues, we introduce the Pick-Double Caption dataset, modified version of the Pick-a-Pic v2 dataset. We also identify difficulties in generating captions, particularly the risk of diiitii-iitiibttiion captions for images, and propose three approaches: 1) captioning (DCPO-c), 2) perturbation (DCPO-p), and 3) a hybrid method (DCPO-h). Our results show that DCPO-h significantly enhances alignment performance, outperforming methods like MaPO and Diflusion-DPO across multiple metrics.
Limitation
While DCPO achieves strong performance on various benchmarking metrics, its captioning and perturbation processes require considerable computational resources. We encourage future research to explore more effi- cient and cost-effective alternatives to further enhance its practicality. The preliminary results in Appendix B.5 indicate that DCPO outperforms Diffusion-DPO across different backbones, such as Stable Diffusion XL (SDXL) (Podell et al. 2023). However, further investigation into its performance with emerging state-of- the-art models remains essential. Additionally, exploring DCPO's applications in safety-related tasks could a valuable ionection for future work. We believe our research will contribute significantly to the alignment community and inspire further advancements in this field.
Ethical Considerations
The authors state that this work, AT assistants, specifically Grammarly and ChatGPT, were utilized correct grammatical errors and restructure sentences.
Acknowledgments
We thank the Research Computing (RC) at Arizona State University (ASU) and cr8dllai for their generous support in providing computing resources. The views and opinions of the authors expressed herein do not necessarily state reflect those of the funding agencies and employers.
13

---

Published in Transactions on Machine Learning Research (10/2025)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. anXiv preprint arXit:2303.08771 2023.
Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya Grover. Com- paring bad apples to good oranges: Aligning large language models via joint preference optimization. arXiv preprint arXivx2404.0083 2024.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Represenlations, 2024. URL
wtwpeerrwwwee.eeeforurrurr/aaaee https://openrreview net/ net/fooruuridddoowjjnr..
Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392 18402, 2023.
Joao Carvalho, An T Le, Mark Baierl, Dorothea Koert, and Jan Peters. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (TROS), pp- 1916-1923, TEFE, 2023.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-aajjude:: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-firss frrty-firsttto International Conference on Machine Learning, 2024.
Dimitrios Christodoulou and Mads Kuhlmann--Jrreenenn Finding the subjective truth: Collecting million votes for comprehensive gen-ai model evaluation, 2024. URL
11904
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hougkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetune language models, 2022. URL
https://arxiv. oww.oogg//..o..oo aaaaabaa/2/0 /atp:/221.... 11416
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference Learning Representations, 2024. URL
https::/openrrevier http:://ppeuueeeeeeeeeeoo/ooooop
Xun Deng, Han Zhong, Rui Ai, Fuli Feng. Zheng Wang, and Xiangnan He. Less is more: Improving Ilm alignment via preference data selection. arXiv preprint orXiv:2502.14500 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Martin Nicolas Everaert, Marco Bocchio, Sami Arpa, Sabine Süsstrunk, and Radhakrishna Achanta. Dif- fusion in style. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2251-2261, October 2023.
Ying Fan and Kangwook Lee. Optimizing DDPM sampling with shortcut fine-tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 9623-9638 PMLR, 23 29 Jul 2023. URL pttps://prooeediing... mlr. press/v202/fan225 html
14

---

Published in Transactions on Machine Learning Research (10/2025)
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Picter Abbeel, Mo- hammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: Reinforcement, learning for fine-tuning text-to-image diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Lovine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 79858-79885. Cur- ran Associales, Inc., 2023. URL httpps:// newr www.ssss......pppppppr..... filess/paper/222//iilel 16655388808335556666666666111113 Seapraaeeeeeeilllaaaooo----ooeer
Alexander Gambashidze, Anton Kulikov, Yuriy Sosnin, and Nla Makarov. Aligning diffusion models with noise-conditioned perception, 2024. URL
Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt, Geneval: An object-focused framework for evaluating text-to-image alignment, 2023. URL huttps://arxiy/ http///aaw...oaa////////// 11513
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. In Thirly- seventh Conference on Neural Information Processing Systems, 2023. URL https://oppenreviex net/
forum?id BsNNWDDaaa
Jack Hossel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yojin Choi. Clipscorc: A reference-free evaluation metric for image captioning, 2022. URL
08711
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thorne, and Jongheon Jeong. Margin-aware preference optimization for aligning diffusion models without reference, 2024. URL https://arxiv.org 0bs/2406.06424
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive bench- mark for open-worke world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723 78747, 2023.
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata, If at first you don't succeed, try, try again: Faithful diffusion-based text-to-image generation by selection. arXiv preprint arXiv:2905.13308, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Thirby-sewenlh Conference on Neural Information Processing Systems, 2023. URL https:://pperrevie
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retricval is an effective defense. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mo- hammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback, 2023.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka, Aligning diffusion models by optimizing human utility, 2024.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13199- 13208, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. Latent guard: a safety framework for text-to-image generation. arXiv preprint 2rx::2404,00001, 2024b.
Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaiqi Huang. Spo: dimeni-aimensional preference sequential alignment with implicit reward modeling. arxiv preprint arXiv:2405. 1 12739, 2024.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Noug-Hyun Park, Sangdoo Yun, Jin-Hwa Kim, Junho Kim, Geonhui Jang, Yonghyun Jeong, Junghyo Jo, and Gayoung Lec. Direct unlearning optimization for robust and safe text-to-image models, 2024. URL
http:////axivv.rr/bbbbbbo 2407 21035
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach, Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL
attps://axxiv.orr/abb/////
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadakii Aligming tix-eee diffu- sion models with reward backpropagation, 2024. URL http::////ppppeeeeee....////r/et
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL htt.l::ttt:ssiiiiii.
00020
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, pp. 10684-10695, 2022.
Sheldon M Ross. Introduction to probability models. Academic press, 2014.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500 22510, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint vrXiv:1707.000 06347, 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Demoising difflusion implicit models. orXiv:2010.02502. October 2020. URL attps:://arxiy www.ab//g////eee/eee/.oooeeeee/e abs/2010. 200..000000000222222200200000200
Jie Sun, Junkang Wu, Jiancan Wu, Zhibo Zhu, Xingyu Lu, Jun Zhou, Lintao Ma, and Xiang Wang. Robust preference optimization via dynamic target margins. arXiv preprint arXiv:2506. 096900 2025.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 14398 14409, 2024.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkan, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9228-8238. 2024.
Jay Zhangjie Wu, Yixiao Gc, Xintao Wang, Stan Weixian Loi, Yuchao Gu, Yufci Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shol tuning of image diffuusio models for text-to-video generation. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 7623-7633, 2023a.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: solid benchmark for evaluating human preferences of text-to to--to---imi to-image synthesis, 2023b. URL httpps///axxii.......m
16

---

Published in Transactions on Machine Learning Research (10/2025)
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/ /CVF International Conference on Computer Vision, pp. 2096 2105, 2023c.
Jiazheng Xu, Xiao Liu, Yuchen Wi, Yuxuan Tong, Qinkai Lá, Ming Ding, Jie Tang, and Yuxiao Dong. Imagcreward: Learning and evaluating human preferences for text-to-image generation, 2023. URL https:
//arxiv.org./ wtt.orr////....000000/30 05977
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human leedback to fine-tune diffusion models without any reward model, 2024a.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024b.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation, 2024.
Daoan Zhang, Guangehen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Yu Dong, Christopher Brinton, et al. Seppo: Semi-policy preference optimization for diffusion aligmment. arXiv preprint orXiv:2410.05255, 2024.
17

---

Published in Transactions on Machine Learning Research (10/2025)
A Formalized Proofs of DCPO
A.1 Optimizing the DCPO Loss is Optimizing the DPO Loss
Inspired by Bansal et al. (024), we can intuitively assume that the DCPO objective is to learn an aligned model po by weighting the joint probability of preferred images pa(x 2²)) over less preferred images po po =²). We the optimization objective of DCPO is minimize the following:
(Dcro(() [uy 22))= log a (BE "r (1 ( z²)) pa (1birr 24
(8)
[log
log P
Pref (2 2²)
Here, we highlight. that reducing LDCPO (0) is equivalently reducing (D) when the captions the for the preferred and less preferred images.
Lemma 1. Under the case where Daefine = {-ū c,,b, c), that is, the image caplions identical for the given pair of preferred and less preferred images (2ll ab)) we have LDPO(0; D 3 Pres) = LDcPo(0; deefine 3; Pref), in which D DDPO = 20 x0)-
Proof of Lemma
LDc D', D', Pret))
Beay affis"
log (o (Blog (e(a =")
po(a1r 2²)
Proflar'' zw) B log Pref(aty 2!)
(9)
loo (o (B11gg
Po Po(m 2 = (2()
-B log po(agl> po(ogl=')pe(>* (2)(z")) Pres(ag) (2 p()))
Pretl n (re)(
ma= =c
(pcro((; Déefines Pret)
E Cay (ayy e)
log (a
po(c) Pree(c)
Blog po(28(c)
log po(orlle Prot(zblc)
Pref (ee((x))c)
pgle) =0 Prorle]
(10)
I (at (aaraatie) (abie)-DDDo
Log a (C. Bog
po pa
C B log pu(aglee Pres(
Pres(at
(OP DDPO, 3, Prot)
In Equation
10 C' is constant value that equates to pale) Prafle)
The proof above follows the Bayes rule by
substituting according to 2 = - =cc
A.2 Analyses of DCPO's Effectiveness
In this section, present the formal proofs of why our DCPO leads to more optimized L(O) of Diffusion- based model and, consequently, better performance in preference alignment tasks.
Proof 1. Increasing the difference between Apreferred and Atess-preferred improves the optimization of L(0). For better clarity, the loss function I.(0) can be written as:
T((0) -E [logo (Tw(())) M)]
18

---

Published in Transactions on Machine Learning Research (10/2025)
where (x) is the sigmoid function that squashes its imput into the output range (0, and M A preferred A les-preforrdd L.e, the margin between the respective importance of the preferred and less preferred predictions.
Characterislicallyy the gradient of (x)
maximum near x 0 and decreases as |æ|| increases. A larger margin in terms of M makes casier for the optimization to drive the sigmoid function towards its asymptotes, reducing loss.
When M is small (IM) = 0): The sigmoid o(-BTw(Xx) M) is near 0.5 (its midpoint). Also, the gradient of (ar) is the largest near this point, meaning the model struggles to differentiate between preferred and less preferred predictions effectively.
When M is large (IM) > 0): The sigmoid o(-33T(A\)- M) moves closer to 0 or 1, depending on the sign of M. For a well-aligned model, if the preferred predictions are correct, M > and a(-BTw(>,)- M) approach 1, thus minimizing the loss.
Intuitively, an ideally large M represents a clear distinction between the preferred image-caption versus the less preferred image-caption. Thus, by maximizing M, we may push the loss L(0) towards its minimum, leading to better soft-margin optimization.
Proof 2.
Replacing caption with the specifically generated caption of for the less ress-preferred image xh decreases tess-preeerrdd
To analyze how replacing с with z', where C z' and z' ~ Q(z1(a' c), for the less cess-preferred imago xh improves the optimization, we delve into how the loss function affected by this substitution. The term relevant the less-preferred image x in the loss is:
A less preffrred |c' co (x 1, c)ll ||e' 4,
Replacing with z' modifies the predicted noise term co(x), t, c) to co(x t22). Since z' better represents x/, we have:
I co(,, 2. < ||c' ca (o(xx, t,)l22
(11)
When ||e' - (, becomes smaller, the term A Ales-preferre decreases. This leads to Apreferred Aless Jles-preferred becoming larger, which improves the soft-margin optimization in the loss function L(0) that we have shown in Proof 1.
We further elaborate on why Equation 11 true. In the context of mean squared error (MSF) minimization, the optimal predictor of cl given some information is the conditional expectation:
When conditioned on (xL, c):
(x t, c) =E x/.t,c]
When conditioned on (x),,,21):
cG 2,) = [e' x [.
The total variance of el can be decomposed as by the Law Total Variance (conditional variance formula)
Ross
2014h:
Var (6²)
[ar (6' xL, ())] + Var (E[e') (,
19

---

Published in Transactions on Machine Learning Research (10/2025)
Similarly, when conditioning on z²: Var (e)) = I. [Var (e') 22, + Var (B xx,(,2])
Since с C 2', the information provided by 2! richer than that of c. In probability theory, conditioning on more information does not increase the conditional variance:
Var (é') x,,,,²)) S Var (e' xx,t,,) t)
(12)
This inequality holds because conditioning on additional information (z*) can only reduce or leave unchanged the uncertainty (variance) about e'.
The expected squared error when using the optimal predictor is equal to the conditional variance:
E ||e' (((x),,,))1 E Vaar (e' X2, c))
Similarly,
E ||e' (x 21)) -E [Var (E' (2))
From 12
we have:
Var (6') x,,,t,z) S Var (e' k) Taking expectations on both sides:
E Var (6') |x(xxx (x,], S E (Var (e' xx,4,c]]
Therefore,
||e' - [|le-=E(xxxe (xt,,, 20) s SE
[llc'' -50-00(((2 cx(x), 1, e)12
Assuming that the neural network еœ is capable of approximating the optimal predictor eq especially training progresses and the model capacity is sufficient, we can write:
le' - co(xx, = =||e' txxx),,,)))
Similarly for e
|| (2,4 - - +x)
Therefore, the expected squared error satisfies:
E Blll'-ccols
- ca (x(x), 21) S SB [|le [ 4, e012]]
Since the term of Aies-preferred in the loss function involves the difference of squared errors, using of instead of for the less preferred sample results a lower error term:
A (5') less- preferred = =l|e² 1, ||e' (ntr( 1, 2/12
20

---

Published in Transactions on Machine Learning Research (10/2025)
Comparing with the original:
A
peee-ereeereed = ||e' co (xx/, t, e) ||e' tt,
(c)
Assuming the reference model Gre remains the same also benefits similarly from the additional informa- tion in 2 the net effect is that the first term decreases more than the second term, leading to a reduced A Ales-preferred-
Proof 3 Replacing coption with the specifically generated caption 2" for the preferred image x increases
A preferred-
To prove that replacing с with z Q(2"1110 ((""), c), where C z", for x also contributes a better optimized loss L(0), we examine how this particular substitution affects the loss function.
We let
Ro(c) ||E" co(x)" .t.x)" t.c)
Ree((c) = ||e - cxet(xx"' t, c)
The rate of decrease in Rg due to z" is proportional to the model's ability to exploit the additional condi- tioning, Since Eg is learnable, it can more effectively leverage z" than Eref, yielding:
ARo Ro(c) Ro(z") > ARref Ree((C) Rret(z").
We further elaborate on why the learnable model's noise prediction residual (Ro) decreases faster than the reference model's residual (Bret) when is replaced by z". The residuals for the learnable and reference models are defined as:
Ro(c) ||c" co(x;" t, c)l.2. Ryar(c) = |cc - cxr( c, c)
When e is replaced with nd (where C )),, the residuals become:
Ro (z(z = ||E" - cx(xx t,, )1
(xed = =||E" Grot(x;" t,, x 112
The rate decrease for each residual defined as:
ARg Ro(c) Rz(z"),
A Rie((C) Rre((z").
The quality conditioning, Q(c), represents how well the conditioning e aligns with the true noise " We assume that
Q(z")) > Q(c),
where the improvement in conditioning quality AQ is defined as
AQ Q(z²)) Q(c).
21

---

Published in Transactions on Machine Learning Research (10/2025)
The residual for Rg is proportional to the misalignment between Q(c) and ex:
1
Ro(c) & Q(c)
Replacing with n (higher Q) results in larger proportional reduction:
Rx(x")) Q(z)) with Aoo & AQ.
The reference model's residual Rzef depends weakly on Q(c), as it is fixed or less adaptable:
Rea((C) Qref(c)
where Qro((c) is less sensitive to changes in c.
Thus, the proportional improvement Rg due to AQ is significantly larger than for Ryel-
The preferred difference term is:
Preefrred Ro Raef-
As Rg decreases significantly more than Roef the gap Rg - Ryef becomes larger, increasing Apreferred:
ARo Raye
preferred increases.
The learnable model Eg benefits more from the improved conditioning zt because of its adaptability and training dynamics. This results in larger reduction in Ro compared to Reot- Mathematically, the relative rate of decrease:
A ARref > 1,
Relative Rate =
which ensures that Apreferred also increases, hence improving the optimization process in L(O) and helping the model distinguish predictions on preferred and less preferred image-captions more effectively.
B Supplementary Experiments and Analyses on DCPO
0.703, μr" 0.762, 0.059
B.1 Semantic Overlap Distribution Measured in VQAScore
prompt less preferred x6 prompt preferred
We also investigated the conflict distribution challenge pref- erence optimization datasets described in Section 3.1 using more recent. prompt-image alignment measures, such as VQAS- core (Karthik al., 2023 Huang al.) 2023). Likewise, we caleulated the VQAScores between each prompt and its pre-
Occurence
ferred image, as well between each prompl and its less preferred image, from the Pick-a-Pic V2 dataset. Our results in Figure g
indicate that, similar to the counterpart in terms of CLIPScore in Figure g there exists a consistently significant conflict between the semantic distriiuutions of the preferred and less- preeerefer images with respect to the prompt
VQAScore
Figure 9: The conflict distribution that ex- ists in the Pick-a-Pic V2 dataset terms of VQAScore.
B.2 Comparison with Diffusion- KTO
A preference alignment datasct, such as Pick-a-Pic (Kirstain et al. (2023). is defined as fc, x ,²), where = and x' represent the preferred and less preferred images for the prompt c. Diffusion-KTO (Li et al. 2024 hypothesizes the optimization of a diffusion model using
22

---

Published in Transactions on Machine Learning Research (10/2025)
Table Comparison of DCPO-h and Diffusion-KTV across various benchmarks.
Method
GenEval (1) Pickscore (f)
HPSv2.1 (t)
ImageReward (t)
CLIPscore (t)
Diffusion-TTO 0.5008
20.41
24.80
55.5
26.95
DCPO-h
0.5100
20.57
25.62
58.2
27.13
only single preference label based whether an image is suitable or not for given prompt Diffusion- KTO uses a differently formatted input dataset D = {c,4), where æ is generated image corresponding to the prompt c.
Diflusion-KTO's hypothesis is fundamentally different. from our DCPO's. While Diflusion-KTO focuses on binary preferences
for individual image-prompt pairs, our approach involves paired preferences. We observe that using the same prompt for both preferred and less preferred images may not be idcal. To address this, we propose optimizing dilfussion model using dataset in terms of D {=" x 2x ,,'',, where 2 and are the captions generated by static captioning model Qa for the preferred and less preferred images, respectively, referring to the original prompt.
We nonetheless conduct comparisons between Diffusion-KT and DCPO on various preference alignment benchmarks. The results in Table 5 show that our DCPO-h consistently outperforms Diffusion-KTO on all benchmarks, demonstrating the effectiveness of DCPO method.
B.3 Comparison with Online and Interactive Methods
Boyond offline preference optimization methods such as Diffusion-DPO, MaPO, and Diffusion-KTO, we also evaluate iterative algorithms (e.g., SePPO) and online algorithms (e.g.. SPO). Under a fixed training budget of 2,000 steps and dataset of 20,000 samples, the results in Table t show that DCPO outperforms both SPO and SePPO, indicating greater cost efficiency and stronger alignment with fewer optimization steps and limited data. We leave comprehensive comparison of offline and online methods under larger training budgets and datasets for future work.
Table 6: Comparison of DCPO-h against online (SPO) and iterative (SePPO) methods under a fixed training budget.
Method
GenEval (1)
Pickscore (t)
HPSv2.1 (t)
ImageReward (t)
CLIPscore (1)
SPO
0.1449
19.30
24.40
16.45
25.91
SePPO (Ir 1 10-8)
0.50567
20.34
25.09
55.21
26.83
SePPO (Ir 5 x 10-9)
0.50232
20.32
25.08
55.24
26.82
DCPO-h
0.5100
20.57
25.62
58.20
27.13
B.4 Benchmarking Performance on Rapidata
To further demonstrate DCPO's versatility, we fine-tune Stable Diffusion 2.1 using Diflusion-DPo and DCPO on another high-quality preference dataset Rapidata (Christodoulou & Kuhlmann-Jorgense 2024 Table 7 shows that our DCPO variants consistently outperform the Diffusion-DPO bascline on multiple benchmarking metrics, including GenEval, Pickscore, HPSv2.1, ImageReward, and CLIPscore.
Table 7: DCPO performances using SD2.1 on Rapidata Christodoulon & Kuhlmann-Jorgensee
2024
Method (SD2.1)
Geneval
Pickscore
HPSv2.1
ImageReward CLIPscore
Diffusion-DDe DPO
0.4813
20.34
25.10
55.4
26.84
DCPO-e DCPO-h
0.4867
20.44
25.43
55.7
26.86
0.4978
20.42
25.10
55.6
26.91
23

---

Published in Transactions on Machine Learning Research (10/2025)
B.5 Performance of Using SDXL as the Backbone of DCPO
We also perform additional experiments to evaluate the performance of DCPO using Stable Diffusion XL (SDXL) (Podell et al., 2023), a larger alternative backbone model for T2I instead our default SD 2.1. Duc to the larger parameter size of SDXL, we conduct LoRA fine-tuning with minimal hyperparameter search. The results in Table X show that DCPO outperforms Diffusion-DPO on key metrics such as Pickscore, Geneval, HPSv2, and Ccorscore, while achieving comparable performance on ImageReward. We hope that our results would encourage other researchers to further explore DCPO's effectiveness on different datasets in future, gaining more valuable insights into its broader applicability and robustness.
Table 8: DCPO performances of using Stable Diffusion XL (SDXL)
Podell et al.
2023
the backbone
model.
Method (SDXL)
Geneval (Overall)
Pickscore
HPSv2.1
ImageReward
CLIPscore
Diffusion-DPO
0.5645
21.77
28.64
71.2
28.61
DCPO-c
0.5758
21.87
28.65
71.2
28.63
DCPO-h-weak
0.5704
21.87
28.64
71.2
28.62
0.5700
21.86
28.64
71.2
28.63
DCPO-h-strong
0.5696
21.86
28.64
71.2
28.62
C Pick-Double Caption Dataset
In this section, we provide details about the Double Caption dataset. As discussed in Section 4.1 we sampled 20,000 instances from the Pick-a-Pic dataset and excluded those with equal preference scores. We plot the distribution of the original prompts, as shown in Figure 10
Occurence
50
100
150
200
250
300
350
400
Length of Prompt in Pick-a-Pic v2
Figure 10: The distribution of token lengths in the original prompts.
We observed that some prompts contained only one or two words, while others were excessively long. To ensure fair comparison, we removed prompts that were too short or too long, leaving us with approximately 17,000 instances. We then generated captions using two state-of-the-art models, LLa AAaV6-666-34B, and Emu2- 32B. The construction the Boob-e Caption dataset illustrated in Figure which provides several examples.
We acknowledge that while captioning and perturbation introduce additional computational costs, these are one-time expenses incurred only during pre- pre-processing and do not affect training evaluation time. We introduce three variants of DCPO-c, p, and h-cach with different processing requirements. For instance, DCPO-c and DCPO-h involve captioning, whereas DCPO-p is more computationally efficient as it only perturbs less preferred caption. Furthermore, captioning 20,000 images using LLaVA requires less than 12 hours on a single A100:80G GPU, and this process can be significantly accelerated using multiple GPUs.
As explained in Section 4.1 we utilized two types of prompts to generate captions: 1) Conditional prompt and 2) Non-conditional prompt. Below, outline the specific prompts used for each captioning method.
24

---

Published in Transactions on Machine Learning Research (10/2025)
Example of Conditional Prompt
Using one sentence, describe the image based on the following prompt: playing chess tournament on the moon.
Example of Non-Conditional Prompt
Using one sentence, describe the image.
Table 9 presents a statistical analysis of the Pick- Double Caplion dataset. With the non-conditional prompt method, we found that the average token length of captions generated by LLaVA similar to that of the original prompts. However, captions generated by LLaVA using conditional prompts are twice long the original prompls. Additionally, Emu2 generated captions (hat, on average, are half the length of the original prompts for both methods.
Table 9: Statistical information on the Double Caption dataset, including the CLIPscore of in-distribution data and average token count of captions generated by LLaVA and Emu2 for both in- distribution and out-of-distribution data.
Token Len. Token Len. (Avg-in)
CLIP score (in)
CLIP score (out)
Text
(Avg-out)
prompt
15.95
15.95
(26.74, 25.41)
(26.74, 25.41)
caption 2 (LLaVA)
32.32
17.69
30.85
29.04
caption z! (A)
32.83
17.91
26.48
28.29
caption 2" (Emu2)
7.75
8.40
25.44
25.18
caption z! (Emu2)
7.84
8.44
22.64
24.88
Prompt e "Soon" written smoke"
Prompt
Prompt c
Glamorous dreesing room with jarge minor
Melme, Peetyy Worrnn in Blow Dreas Satarl Harzo,
SCNN
SODN
Prefered Image
Less Prevemed Image
Prefered Image
Less Proferre Image
Prefened Image
Less Prefered Image
Caption LLaVA
Caption LLaVA
Caption LLaVA
Caption LLaVA
Caption LLaVA
Caption LLaVA
Paut dge pedting
aod SD0W farortt
holding aasebal but atansss fiont Heigo fory "toon" sign.
modem bethroam with large minor and vanily:
And owere vaans and sungany
and
An ancemite chamczer aociingg apaicer sunder day
uncling nur of surtor over
youig woren
Caption Emu2
Caption Emu2
Caption Emu2
Caption Emu2
Caption Emu2
Caption Emu2
The word poon writtan flamos
A standing
A chessking roont with two sinks and miimooa.
wardirbbe
An aniine in blue cress atandlin the beach
aitiing loakingg the sunset.
Figure 11: Examples of Piub-D Caption dataset.
D Specifications of Creating Perturbed Captions
We provide the sctups for the LLM-based perturbation process involved in the DCPO-p and DCPO-h pipelines. Similarly to the method of constructing paraphrasing adversarial attacks as synonym-swapping perturbation by Krishna et al.] (2024), we use DIPPER, a text generation model built by fine-tuning T5- XXL Chung et al.) 2022), to create semantically perturbed captions or prompts, as shown in Table [0 Our three levels of perturbation are achieved by only altering the setting of lexicon diversity (0 100) DIPPER- we 40 for Weak, 60 for Medium, and 80 for Strong. We also "Terl pertarbalion for variable text-to-image prompt." to prompt the perturbation. We hereby provide code snippet to showcase the whole process to perturb sample input as in Figure
25

---

Published in Transactions on Machine Learning Research (10/2025)
fron transformers import TSTokenizer, TEForconditionnsGGeeetttoi class DapptPraprrer(ehjet)) # As defined in https://hggingface... /tgp
prompt "Text perturbation variable texagee prompt."
inxut_text "playing chess tournament
moon."
dp DipperParaphraser((
lex_ lex_diversity=40, prefix-prompt, do_sample=True, top-P 75,7,, top_ hop_k=None, max_length=256) 11 cap mep_ne = dp. araaphraeeiinput....
10 cap_weak dp (inppprraeeiinuut_...
ler_diversity=60, prefix-prompt, do do_sample=True, top_p=0.75, top_k=None, max_length=250) 12 cap_strong = dp. paraphraaeeupppt_ttt
(input_text, lex_ eex_diversity-80 prefix-prompt, do_ co_sample=True,
top_p=0.75, top_k=None, max max_length-22 256))
Figure 12: Sample source code to create multiple perturbations (modified prompts) using DIPPER.
Table 10: Examples of perturbed prompts and captions after applying different levels perturbation.
Weak
Medium
Strong
Prompt
Oryptocrystalline quarts, melted gem- stones, telepathic style.
Painting of oryptnerystallime quarts. Melted gema, Sacred geometry.
Cryptocrystalline quartz with melted stones, In telepathic style.
Caption
digital artwork featuring sym- metrical, kaleidoscopic pattern with vibrant colors and central star-like motif.
digital artwork featuring a sym- metrical, kaleidosoopie pattern with coutrasting colors and a central star- motif.
kaleidosrope with symmetriral and colourful patterns and central starlike motif.
(LLaVA)
Caption (LLaVA)
A vivid circular stained-glase art with symmetrical star design its center.
The image is of radially symmettri- stained-glass window.
A colorful, round stained-glass design with a symmetrical star the center.
Caption sp
Abstract image with glass.
An abstract image colorful stained glass.
An abstract picture with glass many colors.
(Emu2)
Caption (Emu2)
An abstract circular desigu with leaves.
colourful round design with leaves.
Brightly colored circular design.
Original Prompt c Painting of cryptocrystalline quarts melted gemstones sacred geometry pattern telepathicc AI style
E Specifications of Perturbation Quality Control Using MLLM-as-a-Judge
We provide the evaluation prompt that defines the rubrics used for perturbation quality assessment, ranked by MLLM judges, as follows.
Per input, we ensure that any perturbation candidate does not outrank the input (i.e., the original prompts or the captions generated by LLaVa). For example, the input is ranked Good (4), its corresponding perturbations would be selected only from those that are ranked Average (3) below.
F Specifications of Model Fine-tuning
In this section, we dotail the fine-tuning methods and hyperparametor searches conducted under a unified SD 2.1 setup using eight A100 GB GPUs. We fine-tune SD 2.1 for 2,000 training steps with each method. The dataset D sampled and cleaned subset of Pick-a-Pic v2; each record comprises a text prompt e and two images, where " denotes the aunotator-preferee image for and at the less-preferred image.
SFTThhosen We fine-tune SD 2.1 on pairs x" We sweep batch size in {64, 128} and learning rate in (10-10, 10 10-9 10-8, 10-7). The strongest configuration uses batch size with learning rate 10-9. Larger learning rates substantially degrade quality, and increasing the batch size 128 at fixed learning 10-* reduces overall performance.
26

---

Published in Transactions on Machine Learning Research (10/2025)
Evaluation Prompt for Assessing Perturbation (Modified Prompt) Quality Using an MLLM Judge
You are given an image, and a text which is an image caption generated by another Al model for the given image. Rate how well the caption describes the image on a discrete scale from 1 to 5. Here is the detailed scoring rubric for evaluating the quality of the caption:
Excellent (5): the caption perfectly adheres to the image, excelling in relevance, accuracy, comprehensiveness, creativity, and gramularity. It provides insightful, detailed, and thorough answer, indicating a deep and nuanced understanding of the image.
Good (4): the caption is well-aligned with the image, demonstrating a high degree of rele- vance, accuracy, and comprehensiveness shows creativity and a nuanced understanding of the topic, with detailed granularity that enhances the response quality.
the caption adequately addresses the image, showing fair level of relevance, accuracy, and comprehensiveness It reflects a basic level of creativity and granularity but may lack sophistication or depth in fully capturing the image.
Average (3):
Fair (2): the caption addresses the user's instruction partially, with evident shortcomings in relovance, accuracy, or comprchensiveness. lacks depth creativity and granularity, indicating a superficial understanding of the image.
Poor (1): the caption significantly deviates from the image and fails to address the query ef- fectively. It shows lack of relevance, accuracy, and comprehensiveness. Creativity and granularity are absent or poorly executed.
Diffusion-DPO. We train SD 2.1 on triples {c, ",, "c,"" with batch size fixed to 128 and learning rate fixed to 10-8, We sweep the regularization coefficient 8 over (500, 1000, 2000, 2500, 5000}. Performance peaks at a moderate 3 around 2000, with small but consistent declines away from this setting, which aligns with the expected trade-off governed by
MaPO. We also use triples fe ...} with batch size fixed to 128 to fine-tune SD 2.1. Guided by the design of the method, we focus the search on the small-3 regime with 8 E (0.01, 0.1} and learning rate in {10-8, 10007). The most favorable setting is 8 = 0.01 with learning rate 10 10-8, Increasing 3 or the learning rale degrades performance, reflecting sensitivity to heavier margin weighting and aggressive steps.
We explore two families of configurations that differ only in how captions and perturbations are constructed while keeping the SD 2.1 backbone and compute budget unchanged. DCPO-c and DCPO-p fine-tune SD 2.1 with the DCPO objective using captions produced by LLaVA and Emu2, respectively, and evaluate three perturbation levels. Unless otherwise noted, the main results for DCPO-p use a weak perturbation applied to the original prompt, and Table g reports DCPO-p performance across the other perturbation levels. DCPQ-h uses distinct captions for each image; for every training instance we form caption image pairs, (c" xx") and (d,z'), where c describes the preferred image and el describes the less-preferred image. Captions are generated with LLaVA and then perturbed according to the DCPO-h protocol. For DCPO and Diffusion-DPo
DCPO.
sweep 3 over {500, 1000, 2000, 2500, 5000} with batch size 128 and learning rate 10-8, Within this range DCPO attains its best overall performance at B = 5000, and the reported DCPO-h results correspond to a medium perturbation applied to d'. Table presents DCPO-h performance across additional perturbation levels, including experiments that perturb c" and Table [[] reports DCPO-h results when captions are generated by Emu2. The key findings indicate that perturbations applied to short captions do not improve performance and often produce worse outcomes compared DCPO-c (Fmu2).
27

---

Published in Transactions on Machine Learning Research (10/2025)
We also examine sensitivity to distribution shift by constructing in-distribution and out-of-distribution splits using V. and Emu2 within the captioning setup. As shown in Figure 13 in-distribution data generally outperforms out-of-distribution data, while the most significant improvement is observed with the hybrid configuration reported in Figure 6
Finally, Table 12 reports the sweep for DCPO-h under the medium perturbation setting along- side Diffusion-DPO. The results highlight that, across range of values, DCPO-h can outperform Diffusion-DPO, underscoring the effectiveness of the DCPO framework. Based on these observations, we recommend validatting / for each dataset and task, since the preferred setting shift with distributional differences and metric emphasis.
Table 11: Results of the perturbation method applied to Emu2 captions across different levels.
Method
Pair Caption Perturbed Level Pickscore (t)
HPSv2.1 (1)
ImageReward (1)
CLIPscore (1)
GenEval (1)
DCPO-b
(2"
weak
20.10
21.23
49.7
26.87
0.5003
DCPO-h
weak
20.32
23.4
53.8
27.06
0.5070
DCPO-b
(2"
medium 20.31
23.08
53.2
27.01
0.4893
DCPQ-h
(a
medium
20.33
23.22
53.8
27.09
0.5009
DCPO-b
(a
strong
20.31
22.95
53.1
27.
0.4878
DCPO-h
strong
20.35
23.24
53.63
27.08
0.5050
GenEval
Pickscore
HPSv2.1
ImageReward
CUPscore
25.4-
27.4-
0.50 0.4971
0,4869
23.6
25:10
25.12
25.06
25.10
56.6
56.60
56.60
27.2-
0,493210
26.95
26.97
20.4-
20.37
56.4-
26.8
26.8
24.6-
0.47
Le
Laaitnntt
Em
Emai oid
20.0
Le
Laaitaot
Emuzión
Emuzeot
Le
Laaitaot
Emazt
Emuzzoit
56,0
LLA
Lasitoot
Emazt
Emuzeon
LLAVk
Lasttoot
Emaut
Emuzi ost
Figure 13:
Comparison of DCPO-c performance on in-distribution and out-of-distribee data.
Table 12: Results DCPO-h across different 3.
Method
8
Pickscore (t)
HPSv2.1 (t)
ImageReward (t)
CLIPscore (t)
GenEval (t)
DCPO-h
500
20.43
26.42
58.10
27.02
0.5208
Dif
500
20.32
24.79
54.22
26.83
0.4926
DCPO-h
1000
20.51
26.12
58.2
27.10
0.4900
Diflusion-DPO
1000
20.34
21.99
55.06
26.86
0.4935
DCPO-h
2000
20.61
25.83
58.36
27.19
0.5056
Diflusion-DPO
2000
20.36
25.10
56.43
26.98
0.4858
DCPO-h
2500
20.53
25.81
58.00
27.02
0.5036
Diflusion-DPO
2500
20.35
25.11
55.24
26.86
0.4920
DCPO-h
5000
20.57
25.62
58.2
27.13
0.5100
DiP
5000
20.34
25.16
55.49
26.87
0.4974
G GPT-4o as Evaluator
Positional Bias. To obtain binary preferences from the API evaluator, we follow the approach outlined
the MaPO paper (Hong et al.] (024)). To address positional bias in GPT-4o's evaluations, we alternate the positions of the images across different criteria 100 samples (explained in Section 4 The results in Table
28

---

Published in Transactions on Machine Learning Research (10/2025)
show that DCPO consistently achieves better performance than Diffusion-DPO, even when positional bias is accounted for.
13
Table 13: Comparison of DCPO and Diffusion-DPO across different positions in prompt of GPT-4o based various criteria. Method
General Preference
Visual Appeal
Prompt Alignment
DCPO-h Diffusion-DPo
58%
64.5%
56.5%
42%
35.5%
43.5%
Human Study. While recent work Hong et al. (2024) shows that GPT-40 can judge two images given the same prompl, conducted small human study independently assess this aligument. We randomly sampled 100 image pairs produced by ST) 2.1 fine-tuned with DCPO-h and Diffusion-DPO, and recruited three volunteer researchers to evaluate visual appeal. Each annotator answered the question "Which image The comparison performance of DCPO-h and Diffusion-DPOO on Visual Appeal, reported in Tables 13 and 14 indicates that GPT-4o judgments closely align with human preferences.
is more visually appealing? ?
Table 14:
Comparison of DCPO and Diflusion-DPO across different positions in the prompt of GPT-4o
based on various criteria.
Method
Volunteer
Volunteer 2
Volunteer
Average
Win Rate of DCPO-h
64%
66%
69%
66.33%
Win Rate of Diffusion-DEF
36%
34%
41%
33.67%
Judgment Prompts. Similarly Diffusion-DPOO we use three distinct questions to evaluate the images generated by DCPO-h and other baseline models, all utilizing SD 2.1 as the backbone. These questions are presented to GPT-4o to identify the preferred image. In the following, we provide details of the prompts used.
GPT-4o Evaluation Prompt for Q1: General Preference
Select the output (a) or (b) that best matches the given prompt. Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) or Output (b).
## # Prompt: {prompt}
## Output (a): The first image attached.
## Output (b): The second image attached.
# Which image do you prefer given the prompt?
29

---

Published in Transactions on Machine Learning Research (10/2025)
GPT-4o Evaluation Prompt for Q2:
Visual Appeal
Select the output (a) or (b) that best matches the given prompt. Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) or Output (b).
## Prompt: {prompt}
## Output (a): The first image attached.
## Output (b): The second image attached.
## Which image is more visually appealing?
GPT-4o Evaluation Prompt for Q3: Prompt Aligmment
Select the output (a) (b) that best matches the given prompt. Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) or Output (b).
## # Prompt: (prompt}
## Output (a): The first image attached.
## Output (b):
The second image attached.
# Which image better fits the text description?
Additional Generated Samples
We also present additional samples for qualitative comparison generated by SD 2.1, Chosen: Diffusion- DPO, MaPO, and DCPO-h from prompts on Pickscore, HPSv2, and GenEval benchmarks.
30

---

Published in Transactions on Machine Learning Research (10/2025)
SD 2.1 Base
SFT chosen
Diffusion-DPO
MaPO
DCPO-h (ours)
'Photo Violet Parr from The Incredibles in two- piece dress at the beach.
'A ps2 anime witch from madoka magicka is flying a broom through New York causing people to run for lives due terrorist attack.'
'Portrait of an spe wearing an astronaut helmet.."
close-up portrait of Sailor Moon standing front of panel house with a grey winter landscape and an Orthodox church in the background.
Woman with a motorcycle staring over bridge at wetlands.
Figure 14: Additional generated outcomes using prompts from HPSv2 benchmark.
31

---

Published in Transactions on Machine Learning Research (10/2025)
SD 2.1 Base
SFT
chosen
Diffusion-DPO
MaPO
DCPO-h (ours)
Mugshot of
side profile in 1950's style black and white grainy image'
Fantasy landscape painting valey.
mountahous terrain autamon season, fantasy art meadows and and frall frees along the meanderitg banks, midday
Cinematic shot of Ferrari car desert'
'Raalstic portrait of Thanos'
la logo of gym with lion and barbeil or plate'
GYM
GYM
GYMM
GYM
Figure 15: Additional generated outcomes using prompts from Pickscore benchmark.
32

---

Published in Transactions on Machine Learning Research (10/2025)
SD 2.1 Base
SFT
chosen
Diffusion-DPO
MaPO
DCPO-h (ours)
photo toaster'
photo cat'
photo book and laptop'
photo broccoli and parking meter!
7
7
7
'A photo of
W
orange snowboard and green cat!
Figure 16: Additional generated outcomes using prompts from GenEval benchmark.
33


Reviews:
[{"review_id": "6MpE90HASM", "reviewer": "Reviewer_GyAx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The paper identifies two under-explored issues in preference datasets and proposes a solution. This analysis adds novel insights into the limitations of current diffusion alignment methods.\n2. The paper is clearly written and easy to follow. The figures are well-designed and effectively illustrate the proposed method.\n3. The empirical results are strong, releasing a new dataset and showing consistent improvements across a wide range of benchmarks.\n4. The paper introduces a simple and effective extension of DPO and explains why dual captioning improves optimization.\n\n# Weaknesses\n1. The motivation behind the conflict distribution issue could be better justified. It is not clearly shown why semantic overlap between preferred and less-preferred samples is inherently problematic.\n2. It is not well explained how Equation (8) is derived or verified. Specifically, how the use of two distinct captions ($z^w$, $z^l$) directly improves the optimization process. How exactly does this modification alter the RLHF-style objective?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to improve human preference alignment in diffusion models. They identify two issues in existing datasets: conflict distribution and irrelevant prompts, and address them using DCPO, which augments image pairs with distinct captions generated by VLMs and LLMs. Empirically, DCPO improves performance on several text-to-image generation benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The notion of conflict distribution is not clearly justified. Specifically, why should the distributions of $y^w$ and $y^l$ differ significantly under the same prompt $c$? Since both $y^w$ and $y^l$ are sampled from $p_\\theta(\\cdot | c)$, one would expect considerable overlap in their distribution. It is unclear whether this overlap is inherently problematic or simply reflects the stochasticity of the generative model. Additionally, in Figure 3, it is not specified whether the authors are measuring $p_\\theta(\\cdot | c)$ using a single shared prompt or $p_\\theta(\\cdot)$ using multiple prompts. \n\n2. Extra line space before Equation (9).\n\n3. While the empirical benefit of using distinct captions is demonstrated, it remains unclear how introducing $z^w$ and $z^l$ improves the model in terms of the underlying data distribution or optimality. For example, how does the inclusion of dual captions change the effective training distribution or the optimization objective? A formal or conceptual explanation of the role of $z^w$ and $z^l$ in enhancing preference separation would strengthen the contribution.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W1zLLfQmKH", "reviewer": "Reviewer_HcS5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n\n1. Theoretical Motivation and Hypothesis-Driven Evaluation. The paper is not only empirical but also theory-informed. \n\n---\n\nWeaknesses.\n\n1. Computational Overhead in Caption Generation. A key limitation of DCPO lies in the computational cost and scalability of its pipeline. As stated in the Limitations section, both the captioning (DCPO-c) and perturbation (DCPO-p/h) methods require significant computational resources.\n\n2. Limited baselines. Several related baselines are missed in comparison, e.g., SePPO [1].\n\n3. Limited Generalization Beyond Pick-a-Pic v2. While the authors introduce the Pick-Double Caption dataset in Section 4.1, it is derived solely from Pick-a-Pic v2, limiting diversity in training data sources. \n\n4. No Human Evaluation Beyond GPT-4o. While the paper includes GPT-4o-based judgments in Figure 7, no direct human evaluation, e.g., crowdsourced pairwise comparisons, is conducted.\n\n[1] Zhang, Daoan, et al. \"SePPO: Semi-Policy Preference Optimization for Diffusion Alignment.\" arXiv preprint arXiv:2410.05255 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Dual Caption Preference Optimization (DCPO), a framework for aligning text-to-image diffusion models with human preferences, to address two major issues in existing preference optimization approaches:\n\n1. Conflict distribution. Preferred and less-preferred images often share too much semantic overlap under the same prompt.\n\n2. Irrelevant prompts. The same prompt can be poorly aligned with the less-preferred image, reducing optimization effectiveness.\n\nIt also introduces a new dataset, Pick-Double Caption, derived from Pick-a-Pic v2, which contains dual captions to reduce semantic overlap and improve alignment performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. Comparisons with more baselines.\n\n2. Justification of computational overhead.\n\n3. (not necessary) Human Evaluation.", "broader_impact_concerns": "N/A. Ethical Consideration is discussed on page 12.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e1Mo6vHWSw", "reviewer": "Reviewer_w5K7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strengths of the paper**\n\n1. The paper adopts diverse baseline methods to present DCPO's empirical strength, including Diffusion-DPO, MaPO, and Diffusion-KTO.\n\n2. Diverse alternatives for modifying the prompts of binary preference dataset are explored. This provides a better empirical understanding that the prompts' contextual diversity could be crucial for T2I diffusion model alignment.\n\n\n### **Questions and Weaknesses of the paper**\n\n1. **Is \"Conflict Distribution\" truly a problem?** - While the method is built on the *conflict distribution* as a core issue in the existing pairwise preference datasets for the T2I diffusion models, it is not clearly justified. \n    - First, the concept is not clearly defined in the paper. The use of arbitrary metrics like CLIPScore (Figure 3) and VQAScore (Figure 9) confuses what exactly the conflict distribution is referring to. The authors mention *\"Ideally, the distributions of these responses\nshould differ significantly\"*, but CLIPScore and VQAScore are not measuring the difference between $y^w$ and $y^l$, they measure the alignment between the prompt and the corresponding image. This further makes the concept of *conflict distribution* unclear.\n    - Second, chosen and rejected images both being similarly aligned to the prompt is not an issue. The early attempts in the preference alignment leveraged pairwise datasets that comprise multiple models (which is not $p_\\mathrm{ref}$) that generate responses. However, it is shown that using the responses (images) generated from $p_\\mathrm{ref}$ led to a better alignment both empirically [1] and theoretically based on the derivation of DPO [2]. Thus, the CLIPScore/VQAScore of $(x^{(i)}, c)$ for $x^{(i)} \\sim p_\\mathrm{ref}(\\cdot|c)$ being invariant between the chosen and rejected sets is a natural phenomenon. Again, the authors are relying on *\"Ideally, the distributions of these responses should differ significantly\"* to justify that this is problematic, but the observations in the Figures 3 and 9 are entailed from curating the preference datasets with $(x^{(i)}, c)$ for $x^{(i)} \\sim p_\\mathrm{ref}(\\cdot|c)$. While the empirical performance increase with different prompts could be one evidence (which is similarly reported in LLM-based reward modeling context [5]), it does not justify that the conflict distribution is problematic. The authors need to be more specific about why this is problematic.\n\n2. **Correlation between the prompt modifier's strength and the downstream results** - As the caption/perturbed prompt generator (prompt generator from now on) is the key component that differentiates DCPO from DPO, the reader would expect a fine-grained study on the strength of the prompt generator. The paper uses LLaVa-1.6-34B and EMU2 as a caption generators and T5-XXL based model for perturbation. However, there are a lot more models that have a better performance with a better steerability, e.g., more fine-grained control on the degree of perturbation, including the proprietary models [3, 4]. Plus, the choice of LLaVa and EMU2 does not have a clear reason or standard behind it. A clearer ablation on the prompt generators will further support the strength of DCPO to the practitioners.\n\n3. **Hyperparameter tuning for compared baselines** - The paper mentions *\"We fine-tuned SD 2.1 with the best hyperparameters reported in the original papers\"* (Appendix E) for the training configurations for the baseline methods, including $\\mathrm{SFT}_\\mathrm{Chosen}$, Diffusion-DPO, and MaPO. However, both Diffusion-DPO and MaPO papers did not report the optimal hyperparameters for SD 2.1, as both papers employed SDXL or SD 1.5. As all the models have different sizes and configurations, a separate hyperparameter tuning for the baseline methods are crucial for the fair comparison. Including those baselines and the proposed method, every training configuration should be clearly reported in the paper.\n\n&nbsp;\n\n**References**\n\n[1] Meng et al., 2024, \"SimPO: Simple Preference Optimization with a Reference-Free Reward\".\n\n[2] Liu et al., 2023, \"Statistical Rejection Sampling Improves Preference Optimization\".\n\n[3] Chen et al., 2024, \"MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark\". \n\n[4] Yue et al., 2023, \"MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI\".\n\n[5] Sun et al., 2025, \"Rethinking Reward Modeling in Preference-based Large Language Model Alignment\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the variant of direct alignment algorithms in the text-to-image diffusion models. The paper proposes modifying the prompts to ensure the distributional gap between the chosen and rejected contents, resulting in the DCPO methods. Multiple alignment metrics and previous methods were employed to demonstrate the empirical validity of DCPO. The paper's insights highlight the importance of carefully considering the quality of the prompt-image alignment for the better results in the downstream alignment performance of the diffusion models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Detailed requests for changes are discussed above. All three weaknesses should be addressed. Especially, the first weakness should be addressed to strengthen the theoretical foundation of the proposed method, and the third weakness should be addressed to strengthen the empirical justification that DCPO outperforms the baseline methods. Without addressing the two weaknesses, the DCPO lacks justification of its necessity both theoretically and empirically.", "broader_impact_concerns": "No other concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)