DSPO: Direct Score Preference Optimization for Diffusion Model Alignment
Authors: Huaisheng Zhu
Teng Xiao
Vasant G Honavar
Venue: ICLR 2025 Oral (2025)
Keywords: Text-to-image generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Diffusion-based Text-to-Image (T2I) models have achieved impressive success in generating high-quality images from textual prompts. While large language models (LLMs) effectively leverage Direct Preference Optimization (DPO) for fine-tuning on human preference data without the need for reward models, diffusion models have not been extensively explored in this area. Current preference learning methods applied to T2I diffusion models immediately adapt existing techniques from LLMs. However, this direct adaptation introduces an estimated loss specific to T2I diffusion models. This estimation can potentially lead to suboptimal performance through our empirical results.  In this work, we  propose Direct Score Preference Optimization (DSPO), a novel algorithm that aligns the pretraining and fine-tuning objectives of diffusion models by leveraging score matching, the same objective used during pretraining. It introduces a new perspective on preference learning for diffusion models. Specifically, DSPO distills the score function of human-preferred image distributions into pretrained diffusion models, fine-tuning the model to generate outputs that align with human preferences. We theoretically show that DSPO shares the same optimization direction as reinforcement learning algorithms in diffusion models under certain conditions. Our experimental results demonstrate that DSPO outperforms preference learning baselines for T2I diffusion models in human preference evaluation tasks and enhances both visual appeal and prompt alignment of generated images.

Published as a conference paper at ICLR 2025
DSPO: DIRECT SCORE PREFERENCE OPTIMIZATION FOR DIFFUSION MODEL ALIGNMENT
Huaisheng Zhu, Teng Xiao & Vasant G Honavar Pennsylvania State University (hvz5312 tengxias ao, vhonavar @psu. edu
Figure Sample images generated by Stable Diffusion v1.5, fine-tuned using our proposed Di- rect Score Preference Optimization (DSPO). DSPO aligns human preferences of images through preference score matching, maintaining consistency with the pretraining objective. With DSPO fine-tuning, Stable Diffusion v1.5 produces high-quality images that not only adhere more closely to the text prompts but are also visually striking and more appealing.
ABSTRACT
Diffusion-based Text-to-lmage (T2I) models have achieved impressive success generating high-quality images from textual prompts. While large language models (LLMs) effectively leverage Direct Preference Optimization (DPO) for fine-tuning on human preference data without the need for reward models, dif- fusion models have not been extensively explored in this area. Current prefer- ence learning methods applied to T2I diffusion models immediately adapt exist- ing techniques from LLMs. However, this direct adaptation introduces esti- mated loss specific to T2I diffusion models. This estimation can potentially lead suboptimal performance through empirical results. In this work, pro- pose Direct Score Preference Optimization (DSPO), a novel algorithm that aligns the pretraining and fine-tuning objectives of diffusion models by leveraging score

---

Published as a conference paper at ICLR 2025
matching, the same objective used during pretraining. It introduces new per- spective on preference learning for diffusion models. Specifically, DSPO distills the score function of human- numan-preferred image distributions into pretrained diffu- sion models, fine-tuning the model to generate outputs that align with human preferences. We theoretically show that DSPO shares the same optimization di- rection as reinforcement learning algorithms in diffusion models under certain conditions. Our experimental results demonstrate that DSPO outperforms pref- erence learning baselines for T2I diffusion models in human preference eval- uation tasks and enhances both visual appeal and prompt alignment of gener- ated images. The source code for DSPO is publicly available at the Github: https:/ //gi::////ii thub. cow hwa shengzhu/DSPO.
INTRODUCTION
Diffusion-based Text-to-Image (T2I) models have achieved remarkable success in generating high- quality images from textual prompts (Ramesh et al., 2021; Saharia et al., 2022; Rombach et al., 2022). These models are generally trained in single stage, utilizing web-scale datasets of text- image pairs and employing the diffusion objective to guide the learning process. While large lan- guage models (LLMS) have made substantial progress in generating text that addresses wide array of human needs, they achieve this through a two-step process: pretraining from the web, followed by fine-tuning on smaller, more specific datasets to align with user prefer- ences (Achiam et al., 2023; Dubey et al., 2024). This fine-tuning phase refines the model's outputs to better meet human expectations, without significantly compromising the broader capabilities gained during pretraining. Applying this fine-tuning approach to text-to-image -m-i models could similarly en- hance image generation in line with user preferences an area that, to date, has been relatively underexplored compared advancements in the language domain.
vast, noisy datasets
Several recent studies have focused on fine-tuning diffusion- -hased T21 models to better align with human preferences after large-scale pretraining, which is often achieved through Reinforcement Learning from Human Feedback (RLHF) (Black et al., 2023; Clark et al., 2023; Fan et al., 2024; Lee al., 2023; Prabhudesai et al., 2023; Uehara al., 2024). These approaches typically involve fitting reward model to a dataset of human preferences and optimizing the diffusion model to generate images that receive high reward scores, while avoiding significant deviation from the original model. However, building reliable reward model for diverse tasks poses challenges, often requiring large collection of images and substantial training resources (Wallace al., 2024; Rafailov et al., 2024).
To address this issue, several recent works (Wal- lace al., 2024; Yang al., 2024; Li et al., 2024; Gu et al., 2024) have introduced preference learn- ing methods that eliminate the need of reward models when fine-tuming diffusion models for hu- man preferences inspired by the success of Direct Preference Optimization (DPO) (Rafailov al., 2024). These approaches directly adapt the objec- tives used in LLMs for human preference align- ment to diffusion models, adjusting them to fit the specific formulation of diffusion models. Im- mediate adaptation results in an estimated loss on diffusion models based on the original DPO ob- jectives. For example, the loss of Diffusion-DPO upper-bounded by the original DPO loss (Wal- lace et al., 2024). This estimation may result in suboptimal performance when fine-tuning diffusion models for human preference alignment through empirical results, as demonstrated in Figure 2. The figure shows results from human preference alignment experiments three widely used datasets, comparing DSPO with existing baselines for preference learning in T2I diffusion models.
80
Diff--DPO
Dif
DSPO
75
Win rate
70
PickV2
Parti-Prompt
HPSV2
Dataset Figure 2: Win-rate (vs SD15) for DSPO and preference learning baselines based on Aesthet- ics reward. "Diff." represents "Diffusion".
In this paper, we introduce ing their output distribution with human preferences through score matching, the same technique used in pretraining. It is known that diffusion models can be formulated as stochastic differential
perspective about how fine-tune diffusion models by align-
2

---

Published as a conference paper at ICLR 2025
equations (SDEs) and are trained using score matching objectives, which is why they are also ferred to as score functions or score models (Song et al.. 2020). Based on this formulation, we propose Direct Score Preference Optimization (DSPO), algorithm that distills the score function of human-preferred image distributions into pretrained score functions (diffusion models). Here, introduce the target human-preferred score function by combining the ground-truth score of the data distribution with the score from theoretical preference model, such as the Bradley-Terry model (Bradley & Terry, 1952). To simplify this process, we incorporate the implicit reward formu- lation from DPO, eliminating the need for additional training of the preference model. This target score function models the image distribution aligned with human preferences, and fine-tuning the pretrained score models to match this target guides the diffusion process toward human-pret preaerr outputs. Furthermore, we theoretically demonstrate that the optimization direction of this preference score matching loss (under some conditions) is equívalent to the direction required to optimize the RLHF objective introduced in diffusion models using reinforcement learning.
The main contributions of this paper are: (i) To the best of our knowledge, we are the first to fine- tune diffusion models based on human preferences using a score-matching approach that aligns the pretraining and fine-tuning objectives. This introduces a novel perspective for designing preference learning algorithms for diffusion models. (ii) we theoretically prove that DSPO shares the same optimization direction with RLHF objectives in diffusion models under certrain conditions. (iii) DSPO outperforms preference learning baselines on evaluations of human preference tasks.
2
RELATED WORKS
Text-fo-image Diffusion Models. Denoising diffusion probabilistic models have proven to be pow- erful tools for generating diverse data types (Ho et al., 2020). Additionally, the sampling process of diffusion models can be interpreted as stochastic differential equations (SDEs) and is trained using score matching objectives based on this formulation (Song et al., 2020). These models have been successfully applied in various fields, including image synthesis, video generation, and robotics con- trol. Notably, text-to-image diffusion models have enabled the generation of highly realistic images from textual descriptions, paving the way for new possibilities in digital art and design (Ramesh et al., 2021: Saharia et al., 2022). Recent research has focused on improving the control and pre- cision of diffusion models during the generative process. Techniques such as adapters and compo- sitional approaches have been introduced to incorporate additional constraints and blend multiple models, enhancing both image quality and generation control (Zhang et al., 2023: Du et al., 2023). Additionally, bessifferrbbsed and classifier-free guidance methods have significantly advanced the autonomy of diffusion models (Dhariwal Nichol, 2021; Ho & Salimans, 2022), allowing them to generate outputs that closely align with user intentions. In our work, we adopt Stable Diffu- sion (Rombach et al., 2022) to generate images based on specific textual prompts.
Reinforcement Learning from Human Feedback. After web-scale pretraining, large language models are further enhanced through two-step process: first, by supervised fine-tuning on demon- stration data, and then by applying reinforcement learning to incorporate human feedback. Rein- forcement learning from human feedback (RLHF) has proven to be an effective method for both im- proving the performance of large language models and aligning them with user preferences (Akrour et al., 2011; 1; Christiano et al., 2017; Dubois et al., 2024; Dubey et al., 2024; Stiennon et al., 2020; Xiao et al., 2024b;a; Xu & Zhu, 2024). However, the alignment of text-to-image diffusion mod- els with human preferences has been significantly less explored compared to LLMs. To bridge this gap, mutiple methods propose apply supervised fine-tuning to improve text-to-image diffu- sion models. These approaches curate datasets by combining several methods, including preference models (Podell et al., 2023), pre-trained image models (Betker et al., 2023; Dong et al., 2023; Wu et al., 2023), such as image captioning models, and filtering data with the help of human experts (Dai et al., 2023). In the field of aligning and improving diffusion models, several studies have explored fine-tuning these models by leveraging reward models, either by directly increasing the reward of generated images (Clark et al., 2023; Prabhudesai et al., 2023; Hao et al., 2024) or through rein- forcement learning techniques (Fan et al., 2024; Black et al., 2023). This process typically involves pretraining a reward model to capture specific human preferences. However, building a reliable reward model that accurately reflects human preferences is both challenging and computationally intensive. Furthermore, over-optimizing the reward model can result in severe issues, such as model collapse (Lee et al., 2023; Prabhudesai et al., 2023).
3

---

Published as a conference paper at ICLR 2025
Direct Preference Optimization. Recently, several studies have proposed methods for directly op- timizing preferences, such as Direct Preference Optimization (DPO) (Rafailov et al., 2024). These approaches bypass the need for a separate reward model training phase by directly fine-tuning mod- els using preference data, often achieving better performance than RLHF-based methods (Ethayarajh al., 2024; Azar al., 2024; Zhao et al., 2023; Munos al., 2023). Inspired by the success of these approaches, multiple recent methods directly adopt these preference learning methods originally de- signed for LLMs to fine-tune T2I diffusion models align with human preferencess (Wallace al., 2024; Yang et al., 2024; Li et al., 2024; Yuan et al., 2024; Gu et al., 2024). However, Moreover, directly adapting these algorithms from LLM domains results in an estimated loss. For instance, Diffusion-DPO is upper-bounded by the original DPO loss (Wallace et al., 2024). This estimation can lead to suboptimal performance when fine-tuning diffusion models to align with human pref- erences shown in Figure 1 through empirical results. To address this, we propose Direct Score Preference Optimization (DSPO), a method for fine-tuning diffusion models by aligning their output distribution with human preferences using score matching. This approach is the first to apply pref- erence learning from the perspective score matching, offering a novel framework for designing effective preference learning algorithms for diffusion models.
NOTATIONS AND PRELIMINARIES
Diffusion Model. Denoising Diffusion Probabilistic Models (DDPMs) (Ho al., 2020) represent the image generation process as a Markovian process. Starting with data Xo, the forward process gradually adds noise using a predefined variance schedule, ,r. which is defined as follows:
(XI.T xn) ==
1=1 a (xt x;-1), (x, xx-1) ==
= N NN (x:5 / 3ix;- Be1).
(I)
The training of diffusion models involves parameterizing the reverse process (x,-1 xx) using neural network in DDPM (Ho et al., 2020), which defined as:
Po (x: X1+1, c) N
Xti
az
X(+1
Bi+1
co (x41, of oftiT):
(2)
ac+1
VI āu+1
where of41 I-Ge41 Br41, 1, 0t2
Bc, ar. = II Os, Then, the evidence lower bound (ELBO)
minimized to train the diffusion model with the following equation:
LDDPM = Ex..t.s (A(() ||e - Eg (Xx, t)]]"]
(3)
where e ~ (O, 1). L ~ u(0, T), x, q (x, xo) = N (xx: Vaxxo (1 - (1) I). X(1) is a time-
dependent weighting function and 0 represents learnable parameters.
RLHF on T21 Diffusion Models. RLHF typically involves fitting a reward model to human pref- erence data and then fine-tuning the generative model to maximize expected reward through rein- forcement learning. In the reward fitting process. human preferences can be modelled using the Bradley-Tery Terry (BT) model (Bradley & Terry, 1952). To adapt the BT model diffusion models, we define the posterior of human preferences for each time step / with the following formula:
PET (x""
x ()) - a (r (c,
xc.
where (.) denotes the sigmoid function, e is the textual prompt, x;" and x² are pair of winning and losing image samples the time step of diffusion models.
After the reward function is learned, the generative model is optimized using reinforcement learning based on the reward feedback. By conceptualizing the denoising process of the diffusion model as multi-step Markov Decision Process (MDP) and following Wallace et al. (2024); Fan et al. (2024); Yang et al. (2024); Li et al. (2024), we consider reward models each step to define the objective:
L BO-DEEpaxxe (c) Σ r (Xt; c) - AIDx1.. [pe (xo. [c)
(5)
where Pref (XorT) is the leartt distribution from pretrained diffusion models, c is the textual prompt sampled from dataset DD D.... [:]l-] represents the KI. divergence between two distributions and
4

---

Published as a conference paper at ICLR 2025
is the hyperparameter to control the weight of this KL term. We put more details about RLHF and modeling diffusion models as MDP into Appendix due to space constraints.
DPO on T21 Diffusion Models. To simplify RLHF, DPO (Rafailov et al.. 2024) uses the log- likelihood of the learning policy to implicitly represent the reward function. In the context of T2I diffusion models, the step-wise reward function for them be defined
Pg (x, X4+1, c)
r(x,, c) Alog
(6)
Pref (x; X(+1. c)
Following this formulation, existing works (Wallace et al., 2024; Yang et al., 2024) adapt DPO algorithms, which aims to optimize pe based on the BT model in Equation (4), to diffusion models by framing them as MDPs. The objective defined as follows:
Po (xi
Xit1 c)
A pa (x} x)+1
)]
LDiffusion - - E log
Alog
Prof (x} x x(+1,
(7)
Prof (
x)+1,
where (xx² xy. c) ~ D, t ~ U(0, xi,+1 ~ p (xi,++1 x5x) and xi-1,1 ~ p (xl,442 xt). To
simplify notation, we use x2 to represent xi in the following section.
4 METHOD
ạ (x,+1 x).
We introduce Direct Score Preference Optimization (DSPO), a preference learn- ing algorithm grounded in score match- ing principles, tailored for fine-tuning dif- fusion models. Our approach begins by defining a target numan--referre score function, which combines the ground- truth data distribution score with pref- erence model. We then fine-tune the pre- trained score models to align with this target, guiding the diffusion process toward generating human-preferred outputs. The illustration of the model framework is displayed in Figure 3.
Pa (x,
Xx+11 C
X
X,
Xg
V xs log (x,
C
Xx log (x, c.yl
X
pxx, x C,y
Figure 3: The illustration of the model framework.
4.1 HUMAN PREFERENCE SCORE MODEL
In this section, we introduce the target human preference score model, which is aligned with human preferences. Our goal to use this target model for fine-tuning the pretrained score diffusion models to match the this target model. Before presenting this, we first introduce the score model used to leverage the connection between diffusion models and score matching (Song et al., 2020). be derived as Vx, log pe (x, c). By incorporating
And the corresponding score function ditional constraints T2I diffusion models, we can get the score model Vx, log pe (x1 c,y) for the conditional variables y. This can be derived using Bayes' rule follows:
(8) Based on this formulation, we can treat human numan-preferred properties as constrained conditions for T2I diffusion models, which is represented by the variable The probability of whether the input images p (y Xx, c) align with human preferences can obtained using Equation (4):
Vx, log Po (x, c, Ux, log po (x, c) Vx, log (y x),
( Xx, c) = ( Xr, x c) p (x, x xL_ c) =a (r xx) (c,x)))
By treating the variable y as human--rrefeet prefer-ee conditions, we can derive a human preference score model. To achieve image generation based on human-preferee preferred conditions in a training free manner, we can first naively train preference model to estimate pş (x, > x x c). This trained model can then be used to replace the second term in Equation (8), following the widely-used classifier guidance method for diffusion models (Dhariwal Nichol, 2021). I). However, this approach has two major drawbacks: (i) To determine the probability of human-preferred images, Ps. must input x at each time step. which requires providing negative samples for every target prompt-a task that is impractical in real-world applications. (ii) Calculating the gradient of the trained classifier increases inference time, and training a robust classifier for all reverse steps, especially for highly noisy inputs at the initial steps, is significant challenge (Ho & Salimans, 2022).
5

---

Published as a conference paper at ICLR 2025
4.2 DIRECT SCORE PREFERENCE OPTIMIZATION
In this paper, we focus on a novel fine-tuning method instead of training-free method for pretrained T21 diffusion models better align with human preferences. Our approach ensures that the fine- tuning objective is consistent with the objective used during the pretraining stage, unlike current methods that adapt fine-tuning techniques from LLMs (Wallace et al., 2024; Yang et al.. 2024; Li et al., 2024), which differ from the pretraining objectives and may lead to suboptimal results. Specifically, we propose fine-tuning the pretrained T2I diffusion model to match the target human preference score model for each time step i introduced in Section 4.1, which is defined as follows:
m() V log (x, c) (Vx, logp (xt c) + 7 Vx. logp (y Xt, c))
(10)
0
where w(t) is a dimendependent function for score matching as introduced in Song et al. (2020).
Y is used to control the weight of conditional constraints towards human preferred image genera- tion and p (y Xx, = (r x,) - r (c. xx)))) is represented as Equation (9) for human preference conditions. To avoid training an extra probability model, we use the implicit reward r (c, x²) de- fined in Equation (6) to replace the reward model in Equation (9). Based on the reverse process po (X/ X(+1, c) of T2I diffusion models in Equation (2), we can get the following reward r(x,, c):
A3411 ar (Ileo (32+11 +1 c++1 |Eref (I++11, /. 1) c (11) r(x,, c) 211 āx) 04+1
Details about achieving this equation are put into Appendix B.1. We use the score function of the true data distribution instead of the pretrained model Pref in the second term of Equation (10) because the pretrained model may not accurately reflect the true data distribution's score function. Based on Equation (11), we get the following objective after derivations on Equation (10):
min A(t) [() (EA,441 -E+11) - Ar (1 a (c, -r (c,xx)) (E0,141 1)) (12)
or Li. Co,141 = Cy c,, 1), À is a hyperparameter that where A(t) w(t) 4att ac+1 1-6++1"
determines the weight used to control the KL divergence in Equation (5), similarly for Cref, and r()) is defined in Equation (11). B(t) is a time-dependent parameter whose specific form is provided in Appendix B.2 due to space constrains. Based on our empirical findings and to further simplify the loss function, we omit B() in our experiment, arriving at our final objective:
CIsPo A(t) |oo.+1 c++1 by (1 a (r (c.x.) r (c,x)) (co,141 Ge) (13)
We set ? to avoid extra hyperparameter for one-tuning diffusion models. The derivations are put into Appendix B.2. Following DDPM (Ho et al., 2020) and Diffusion-DPO (Wallace et al., 2024),
we disregard A(l) and the associated parameters about a, and B, at the beginning of Equation (13). In our settings where only preference data are accessible, we have our following final objectives:
min E (xy (...))) x) sfpp(x x/.c) Cisso
(14)
4.3 THEORETICAL ANALYSIS
In this section, we provide theoretical analysis about DSPO. Our analyses show the relation with RLHF objectives on diffusion models in Equation (5). Specifically, We demonstrate that, under cer- tain conditions, minimizing DSPO by sampling data from the trained diffusion model and distilling the score from the reference model shares similar optimization directions with maximizing RL objectives in Equation (5). Because of space constraints, all proofs are put into the Appendix C.
Next, we start by deriving an equivalent form of the RLHF objective on T21 diffusion models in Equation (5) by rearranging the elements in this equation. We can view the RLHF objective as opti- mizing a reverse KL-divergence between po(-) and p" (.) from the probability matching perspective:
Lint = EE Poo c))
ADKI. |po(Xe I Xe+1, c)llp" (xx Xx41, c)] + log Z (c), (15)
where Z(c) = f exp((2n0 r(x,, c)/ //eet (XoTTe)) ( is independent of learnable parameter 0 and p" (xx Xx+1, c) Pret (xx 1e41, c) el)AA, The details of derivations for this equation

---

Published as a conference paper at ICLR 2025
SD15
Diff--DPO
Diff.-TTO
SFT
DSPO
Pick Score
HPS
Aesthetics
22.0
28.0
5.80
21.6
27.4
5.65
Score
Score
5
21.2
26.8
5.50
20.8
26.2
5.35
20.4
25.6
5.20
PickV2 Parti Prompt HPSV2
PickV2 Parti Prompt HPSV2
PickV2 Parti Prompt HPSV2
Dataset
Dataset
Dataset
Figure 4: Reward score comparisons all datasets for various baselines by different reward models.
are put into Appendix C.I. In the following theoretical demonstration, we show that optimizing our DSPO shares the same optimization direction as maximizing the RLHF objective. This is achieved by matching the hyperparameters w()) and y with those in the RLHF framework.
Theorem Following w(t) 20/-1//. x 1/2x, reward model r(-) defined in Equation (4) and P- (-) as the reference model for RLHF of diffusion models in Equation (15), the gradient of DSPO objective in Equation (13) by sampling data from po satisfies:
T
VoLunt = VoBE-DE (x (Xar
DSPO
(16)
t=
Theorem indicates the optimization direction for pe during intermediate steps of minimizing LDSPO, when sampling data from aligns with the direction required to maximize Lime asymp- totically, given a sufficiently large dataset. Our final optimized loss Equation (14) an empirical estimate of the loss in Equation (16) on preference feedback by sampling pairs of images, x" x'. Moreover, we consider the reward model r(.) = PET (x," X c) as defined Equation (4), which evaluates pair of images based human feedback. Alternatively, we can retain the original reward model format, i.c. r(x), which assesses individual images rather than comparing pairs based on human feedback. Maximizing the RLHF objective while maintaining the original reward struc- ture is equivalent to minimizing our objective by setting p (y X2, = exp (r(x,, c)/A) /c(c). where Z(c) I exp (r(x,, c)/A) dxx,, under the same condition in Theorem 1. Detailed derivation are put into Appendix C.3. We conduct an ablation study on this approach in Section 5.3.
5 EXPERIMENT
5.1 EXPERIMENTAL SETUP
Datasets and Models. We fine-tune Stable Diffusion v1.5 (SD1.5) using the DSPO objective image pairs based on human feedback, as described in Equation (14), following the Diffusion-DPO approach (Wallace et al., 2024). This is done using the Pick-a-Pic v2 (Pick V2) dataset (Kirstain et al.. 2023), which contains image preference pairs for each prompt. To evaluate the model, we use test prompts from Pick V2, the HPSV2 benchmark prompts (Wu et al., 2023), and the Parti- Prompts dataset (Yu et al., 2022). We conduct the image editing experiment with text instructions on InstructPix2Pix dataset (Brooks al., 2023). The details of dataset are put into Appendix D.2.
Baselines. We evaluate the effectiveness of aligning T2I diffusion models with DSPO by comparing the generations from our DSPO aligned model to those from other existing methods, including the original pretrained 5D1.5mm or SDXL, supervised fine-tuning (SFT) approaches, Diffusion-DPO (Wal- lace et al., 2024), MaPO (Hong et al., 2024) and Diffusion-KT KTO (Li al., 2024). Note that when training the SDXL model, the quality of training data in Pick is lower than that of images gen- crated by SDXL. Therefore, we use the reference model for p(x; c). A detailed discussion of the training method SDXL is provided in Appendix D.I.
Evaluation. To assess human preference alignment, we perform text-to-image (T2I) genera- tion and text-guided image editing. We evaluated each task with several metrics, including Pick Score (Kirstain et al., 2023), HPSV2 (Wu et al., 2023), LAION Aesthetics Score (Schuhmann, 2022), CLIP (Radford et al., 2021), and ImageReward (Xu al., 2024). For each reward model,
7

---

Published as a conference paper at ICLR 2025
Table 1: Win-rate comparison between DSPO and other baselines versus SD1.5, evaluated on dif- ferent reward models using prompts from the PickV2, HPSV2, and Parti-Prompt datasets (T2I Gen- eration). For simplicity, "Diff." represents "Diffusion" Best results are highlighted in boldface.
Dataset
Method
Pick Score
HPS
Aesthetics
CLIP
Image Reward
SFT
70.20
84.20
75.80
61.20
76.40
Diff.-DPO
71.60
70.20
66.20
58.80
63.60
PíckV2
DiTT--KTO
71.40
84.40
72.60
60.02
77.00
DSPO
73.60
84.80
76.20
61.80
78.00
SFT
64.27
85.72
75.74
54.72
71.38
Dif
61.18
66.48
60.42
55,45
62.19
Prompt
Diff.-KTO
64.80
86.16
72.92
54.34
71.51
DSPO
65.32
87.50
76.96
54.86
71.75
SFT
79.03
91.97
78.56
60.47
80.78
Diff.-DPO
76.06
72.13
66.00
58.50
64.22
HPSV2
Diff--KTO
79.18
92.15
77.87
59.28
81.96
DSPO
79.90
92.56
80.59
61.13
82.31
Table
Win-rate comparison between DSPO and other baselines versus SDXL, evaluated on dif- ferent reward models using prompts from the PickV2, HPSV2, and Parti Prrti-Prompt datasets (T2I Gen- eration). Note that the DSPO this table is fine-tuned on SDXL.
Dataset
Method
Pick Score
HPS
Aesthetics
CLIP
Image Reward
SFT
20.80
40.60
23.20
44.80
34.40
Diff.-DPO
75,20
76.20
54.10
59.40
65.20
PickV2
MaPO
54.40
69.60
68.20
51.20
61.40
DSPO
74.00
80.00
54.20
59.60
68.60
SFT
17.03
33.02
27.81
36.58
37.18
Diff.-DPO
65.44
74.08
56.86
60.54
66.85
Parti-Po Parti-Prompt
MaPO
58.34
66.54
68.23
47.43
58.64
DSPO
67.46
81.80
57.84
55.02
73.47
SFT
18.18
45.28
26.72
39.13
47.22
Diff-DDOO
70.31
80.81
50.78
59.31
68.75
HPSV2
MaPO
59.62
77.90
62.31
50.90
62.09
DSPO
72.59
83.47
51.41
57.34
70.09
report both the average scores for all models and win rates between DSPO baselines and Stable Diffusion v1.5. For fair comparison, we use the default hyperparameters for the T21 diffusion model to sample images across all baselines and DSPO used in Rafailov al. (2024), ensur- ing consistency in evaluation, i.e., guidance scale as 7.5 and the number of sampling steps as 50. Note that for evaluation experiments, Diffusion-KTT provided by the authors. Additionally, we train the SFT model following Diffusion- DPO for our evaluations. We conduct five sampling runs for each algorithm using different seeds, and the average results are reported. Implementation details of DSPO are provided in Appendix D.3.
directly the checkpoints for Diffusion-DPO and
5.2 PERFORMANCE COMPARISON HUMAN PREFERENCE ALIGNMENT
We present the results of real rewards from var- ious reward models across all datasets of T2I generation in the Figure 4, comparing DSPO with SFT, Diff.-DPO, KTO, and SD1.5. Due to space constraints, additional reward score results (CLIP and Image Reward) are pro- vided in Appendix E.2. The results consis- tently show that DSPO outperforms all base- fine-tuned T2I diffusion
Table 3: Computational costs of Diffusion-DPOO and DSPO using 1 NVIDIA A100s. Training time ("Time") for each optimization step and peak GPU memory without the model ("GPU Mem.") measured with batch size and 128 accumula- tion gradient step in fine-tuning SD15 PickV2.
Diffusion-DPO
DSPO
lines. Notably, model significantly surpasses the original base model SD1.5. For example, SDI.5 achieves
Time (L)
4.15 min 60.2
4.18 min
GPU Mem. (4)
60.5
8

---

Published as a conference paper at ICLR 2025
Table 4: Win-rate comparison of InstructPix2Pix dataset for text-guided image editing.
Dataset
Method
Pick Score
HPS
Aesthetics
CLIP
Image Reward
SFT
57.10
66.60
73.10
48.60
61.10
Dift--DPO
51,40
52.00
52.80
46.80
47.00
InstructPix2Pix
DiO
53.60
69.20
72.20
50.00
61.00
DSPO
58.40
69.30
73.80
51.30
61.10
Table 5: Win-rate comparison between DSPO and its variant DSPO-E versus SD1.5, evaluated
across different reward models using prompts from the PickV2, HPSV2, and Parti-Prompp datasets.
Dataset
Method
Pick Score
HPS
Aesthetics
CLIP
Image Reward
DSPO-E
70.20
84.00
73.20
60.60
75.80
PickV2
DSPO
73.60
84.80
76.20
61.80
78.00
DSPO-E
62.86
85.31
75.91
54.81
71.69
Parti-Prompt
DSPO
65.32
87.50
76.96
54.86
71.75
DSPO-E
75.06
91.28
77.65
59.59
80.93
HPSV2
DSPO
79.90
92.56
80.59
61.13
82.31
Image Reward score of only 0.018, while DSPO attains a much higher score of 0.568. Additionally, DSPO outperforms both DDO and Diff. which also use preference learning algorithms. This validates the effectiveness of our model in aligning with human preferences.
Table presents the win-rate comparison of SFT, Diffusion-DPO (Diff-DDPO) Diffusion-KTO (Diff.-KTO), and DSPO aligned SD1.5 against the original pretrained SD1.5 for T21 generation. In general, DSPO achieves the best performance compared to recent baselines across all datasets and nearly all reward models, demonstrating its effectiveness. Notably. DSPO significantly hances alignment for the base SD1.5 model, achieving win-rates as high as 92.56% according to the HPSV2 reward model. Furthermore, DSPO outperforms existing baselines, such as Diff.-DPO and Diff.-KTO, which adapt algorithms from L.M domains to diffusion models, across nearly all reward models. Specifically, DSPO achieves absolute win-rate improvement of 16.54% and 4.04% Diff.-DPO and Diff. respectively, on the Parti-Promp dataset for the Aesthetics reward model. This validates the motivation that matching the loss objectives during both the pretraining and fine- tuning stages of T2I diffusion models enhances overall model performance. Table 2 presents the results of DSPO fine-tuned on SDXL, alongside the corresponding baselines. The results demon- strate that our models outperform the baselines across most metrics on all three datasets, further validating the effectiveness of our approach. We also conduct memory and wall-clock experiments in Table 3. Compared to Diffusion-DPO, DSPO shows comparable runtime and memory usage.
Table presents the win-rate comparison results for the rext-guided image editing task. Similarly, DSPO outperforms all baselines across different reward models, further demonstrating its effective- ness and potential applicability to a wide range of text-based image generation tasks.
5.3 ABLATION STUDY
We display the results of DSPO and its variant DSPO-E performance in Table 5, Specifically, as outlined in Section 4.3, we can express p (y x,, c) as an energy-based distribution (y x,, c) = exp (r(x;, A)) (e), where 7(c) = exxp exp (r(x,, e)(x) dx,, Optimizing this variant of DSPO follows the same optimization direction as the RLHF objective, as demonstrated in Theorem without assuming reward functions as BT models. Further details on this variant are provided in Appendix C.3 and we denote it as DSPO-E. We observe that models outperform the DSPO-E variant on all datasets for all reward models, highlighting the effectiveness of using the BT model for human preference learning, outlined in Equation (4). Unlike DSPO-E, which relies implicit reward models for single images, DSPO leverages image pairs from human preference feedback, providing richer information and enhancing overall performance.

---

Published as a conference paper at ICLR 2025
SD15
SFT
DPO
KTO DSPO
Figure 5: We show the images generated by different models from one prompt, which is "Frontal portrait of an anime girl with pink hair and sunglasses wearing white tshirt.". Original
SD15
Figure 6: We show the text-guided image editing task with the prompt "The siblings are all robots".
5.4 QUALITATIVE ANALYSIS
Figure 5 showcase the qualitative performance of our model on T2I image generation used in this paper. Compared to the baseline methods, DSPO exhibits a clear enhancement image quality, which is even more pronounced than the improvements reflected in the reward scores. Specifically, DSPO accurately generates details such as sunglasses, pink hair, and an anime girl, while simultane- ously creating more visually appealing image compared to other baselines. Furthermore, Figure presents the qualitative results of DSPO on text-guided image editing. In comparison to other base- lines, DSPO not only faithfully adheres the textual description when transforming siblings into robots, but also generates more realistic and visually acceptable images. In summary, the advantages of generated images from DSPO particularly evident in key aspects such alignment, visual appeal, and the intricacy of details within each image. These qualitative results emphasize DSPO's ability to generate images that are not only contextually accurate but also visually superior those produced by existing models. Additional prompts and qualitative results for both of experiments are provided in Appendix E.3 due to space constrains in the main paper.
CONCLUSION
In this paper, we propose Direct Score Preference Optimization (DSPO), a novel approach for fine- tuning diffusion- text-to-image (T2I) models by aligning their pretraining and fine-tuning ob- jectives through score matching. By leveraging the inherent score function of diffusion models and incorporating human preference feedback without relying on complex reward models, DSPO ad- dresses the performance gaps observed with existing fine-tuning techniques such as Diffusion-DDP We theoretically demonstrate that optimizing DSPO shares the same optimization directions op- timizing Reinforcement Learning from Human Feedback (RLHF) objectives, ensuring the effective- of the fine-tuning process. Our empirical results show that DSPO consistently outperforms other preference learning methods, confirming its capability to enhance image generation for human preferences. This approach offers new direction for preference alignment in diffusion models, bridging the gap between pretraining and fine-tuning for more user-aligned outputs.
ACKNOWLEDGMENT
The work of Vasant G Honavar, Huaisheng Zhu and Teng Xiao was supported in part by grants from the National Science Foundation (2226025, 2225824), the National Center for Advancing Translational Sciences, and the National Institutes of Health (UL1 TR002014).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint (xXiv:2303.07777 2023.
Riad Akrour, Mare Schoenauer, and Michele Sehag. Preference-based policy learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML. PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part 11, pp. 12-27. Springer, 2011.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e3. pdf. 2(3):8, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13300 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. /7400, 2023.
Xiaoliang Dai, Ji Hou, Chih- Yao Ma. Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309. 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8894, 2021.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 2023.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. International con- ference machine learning, pp. 4489-8510. PMLR, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
11

---

Published as a conference paper at ICLR 2025
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadch, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning dif- fusion models through relative preference optimization. arXiv preprini arXiv:2406.06382. 2024.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jiwoo Hong. Sayak Paul, Noah Lee, Kashif Rasul, James Thome, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. arXiv preprint arXiv:2406.066424 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36662-3366666 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192. 2023.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.044 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang. Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:23 200088 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffuuion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine Pmir, 2021.
learning, pp. 8821-8831
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 35:36479-36494, 2022.
12

---

Published as a conference paper at ICLR 2025
Christoph Schuhmann.
Laion-assthetics. Laion-aestheticon 2022. Accessed: 2023 11- 10.
https:///adon.al/bblog.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous- time diffusion models as entropy-regularizn control. arXiv preprint 24X1::2402.1194,, 2024.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238, 2024.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2096-2105, 2023.
Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? self-imitation learning perspective. arXiv preprint arXiv:2410.1000 2024a.
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. arXiv preprint 2..::2242..4511 2024b.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Siyuan Xu and Minghui Zhu. Meta-reinforremmet learning with universal policy adaptation: Prov- able optiaolt under all-task optimum comparator. In Neural Information Processing Sys- tems, 2024.
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8941-89511 2024.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206.10789) 2(3):5, 2022.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. arXiv preprint arXiv:2402.102111 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
13

---

Published as a conference paper at ICLR 2025
A OMITTED DETAILS OF RLHF ON DIFFUSION MODELS
In this section, we follow the approach of Black et al. (2023); Fan (2024) to model the diffusion reverse process under the conditional generation setting Markov Decision Process (MDP), defined by .M- = (S, A. P. p). Specifically, π represents the policy network and the diffusion reverse chain is {ax)r=T with length T. This MDP can be defined as:
8t x,) a, X1-1 Pa (Su)
(a, st) A pe (x c) ((s++1 ae) - (01 (17) (s, a,) rxx-11,
(,,0,,)
where 5(.) is the measure delta measure and P (5t+1 |s;,ai from previous works, we represent each step's reward model as r(x;-1, c). Based on this MDP, we get the objective for RLHF diffusion models following Fan et al. (2024):
is deterministic transition. Different
E T (x AlDKLL [pe (x Peee (xo:Tec]]
E Eew rlc)
(18)
For notation simplicity in the following part, we set the range from to T and get the following equation:
BewD Booxorp Ene cx AlEEL Ips (poo
(19)
B DERIVATIONS IN SECTION 4.2
B.1 DERIVATIONS OF EQUATION (11)
In this section, we provide a detailed derivation of Equation (I0) for the implicit reward model:
ReeD Bpp (xa.rlc) Σ T-1
r(x:, c) ATRIL (xxxxxrce)llppec(xxoo
(20)
1=0
Following DDPM (Ho et al., 2020), the definition of po (xt (x+1,C) is as follows:
Ot a++1
Be+1
co (x(x22102+11) ( obrIT
po (x, C) =N
Xxi
X
VT- ait1
at
3rt1 E0 (xxm1:
exp
Xt
Xi+1
(V2moiz.)
20 2+1
01++1
VI ae+1
(21)
where 0441
1-81 1-a1 I-a
Be41 and d is the dimension of the image.
We approximate x, with its posterior mean E [x4 Xx+1, Xo]
= =oeli
ar 1++1 v
8111
El
VI-0e41
Then, we can get the estimated po (x: Xx+1, c) as:
1 B,41 at |Iea (#0+1, 1) Ec+ 1(+1112]
exp
(22)
2 21 āz) a441
[+1
Therefore, the reward function r(.) (log po (xx xx+1, llog Pref (xx xe+1, c)) can be rep- resented as by using the estimated (xi xx+1, C):
A Br- Oa
r(x,, c) 2 (1 āe) ac+1 (IEa (2 1) 1 -1lero (I,-1, 1) ec+1
(23)
B.2 DERIVATIONS OF EQUATION (12)
In this section, we provide a detailed derivation of Equation (12) for our loss:
Po c)
wi) V log
Pdata (x+(c) V logo (r(c,x,, - (c,
(24)
po px (c)
=(())
Vx. log
- 7(1 ( (c, - (c, x:) Vx2 7 (c,x,)
Pdata (xata(xe/e c))
14

---

Published as a conference paper at ICLR 2025
Then, the reward model (xe, c) in Equation (11) and get the gradient of the reward:
ps
(x, Xx+1, c)
Vx,r (Xe, c) = Vx, log
(25)
Pref (xt.
X
Then, we get the gradient r()) based on the reverse process in Equation (21):
Pe (xe X)-1; c) Pref (x, X(+1. c)
Vx, (c,x,) x1) = >llog
(26)
But1
(co (X++1, 1) Cref (x++1, 1)).
20 a++1
VT ac+1
Using the definition of the score function which connects the score model and diffusion models as described in Song al. (2020), derive the formula for the first of Equation 24:
(x. c)
1
V X log
(Ea (x,+1, -E).
(27)
Pdata (Xr C)
v1 di
By combining Equation (23), (27), we can obtain our final objectives:
(x c)
min (t) Xx, log
Vxx logo (r (x,, c) (x (x)
Pdata (x: c)
= min A(t) B(t) (co.141 4411 xy (1 (
- (c. (c.x,)) (c0,141- Cref,t+1)
(28)
where the specific value are 60,+11 = eg (x 1), A(t) = w(t) 1 a
B(t)
20 1-6011
B4IIi1- and similarly for 6ref,t=1-
(l)) 1
Cre
11-0441
C DERIVATIONS AND PROOF IN SECTION 4.3
C.I DERIVATION OF EQUATION (15)
In this section, we provide a detailed derivation of Equation (15). Starting from the RLHF objective on T2I diffusion models in Equation (5), we can have the following equation:
T-1
E Ee-DE (xarle (c)
Σ
r (xx. c) AlDK |no[xxrrlee/llrrcxxxx///
(29)
t=
The we can get the following equation:
Epo(xx/xee,.)) C) (Xr, c) - ADRL (po (xt Xc+1, c) I|pror (xt xx+1, c)l,
= Ly ext(xre))) XEpo(x log po (X: X++1, c) Pref (xt Xe+1, c)
(30)
etr(xen) Prot (xe x)+1,c) Pe (xx x(+1, c)
=j log
We have the analytical form of p* (Xo:rle) following (Wallace et al., 2024):
p" (Xo.Tle cc) Piet (Xo.T c) (E r(x,)))/> /Z(c),
(31)
where Z (c))= = exp Σ 1 r(x,, c)/ A))r (Xor (xar) c))xxorr--- Therefore, get the following
results for Equation (30):
e MT- r(x..)) Pret (Xo:r/c)
E 1e
Pe (xo.T (c)
p" (Xo.T c) Z (c)
(32)
Eps A log
Po (Xu:2 (c)
-DDRL ((po(Xo:T(c) 1|p" (xo.Tle) c)] + Alog Z (c).
15

---

Published as a conference paper at ICLR 2025
We can get the D+L(-) form as follows:
Po (Xo:r c)
DKL (xpo(xo:r (xorree)
Po (No:T c) x log p (xo:T
dxo:T
= =s po (XoiT I cc) log
pe (XT
pe (xt Xt, c)
dxo.T
p (xT
Xx+1, C) I) p (xx
= =s |oo (xo.T (c) log
po (XT
c)
T
po (xt
X(+1,
log
dxo:
pe (xT
C)
p' (x+
X(+1,
(33)
Po (x:
Xt+1, c)
(xe Pox
log p" (x,
x c)
Ep le) Bpolxxxeetiee log
Po (x: X
p² (x: X
1= (x DEL (po (xe xx+1, c) ||p" (x: Xx+1,)]]
T (po(xoerlc) Eno DKL (xo xx+1, c)lp" (xt x)+1.c].
where p (xt Xe+1, e) x Prof (sez Be+1, e) c)r(A, Finally, we can combine the above equa-
tion with Equation (29):
Light BewDBBo(xort Σ
-ADEL (xo xe+1, (x, xx+11, c)] + Alog Z (c). (34)
C.2 PROOF OF THEOREM 1
Theorem 1, Following w(t) = 1 1/2x, reward model
r(-) as defined in Equation (4)
and paura (-) as the reference model for RLHF of T21 diffasion models in Equation (15), the gradient of DSPO objective in Equation (13) by sampling data from po satisfies:
VoLelh = Ve (xorr(e)
(35)
2=
First, we recognize the reverse process diffusion models as a Gaussian process, which we denote as po (x: c) ~ N(,, Σ) for simplicity. The corresponding log po(-) can then be defined as:
1 log pa (x: xx+1) - (x, 11² 1-1 (x,
p1) + CC,
(36)
where C' is a constant value and we can denote Pret (I: xz+1, c) by replacing i with peef similarly. logp" (x, Xc+1. c) can be estimated by Taylor expansion around x,:
logp" p" (xx X++1, c) log Pref (x) 11+1, c) e (r((),e))// /Z(x)+1,
Ro log Pref (xy Z1+1, + log (r(x).e))/A
xi=F (x: Mret) Vx, log er(xx.e)))
Ci
x:=
2 (x: - Mrro) -1 (x+ - (lrot) + (xt - Mrot) Vx; r r(x,,c) c)
1
+ C2
X
- Σxx, r (xx,,)
T
r rxx,,)) + +C3 Exx
I-1
Xt -Met
2
A
(x C) E),
= =log [logp"(a) (a) + C4,, ~ N( VV
(37)
and g represents Vx,r (xe, e) /X. We can safely ignore the constant term Ca. since it corresponds the normalizing coefficient following Dhariwal & Nichol (2021). Then, we can observe that p" (x, X(+1, c) follows the Guassian distribution. Therefore, we can further derive the KL divergence loss betwenn two Guassian distributions for RLHF in Equation (15):
where C3 -g"
DKL (xe c)lp"" (xt xx+1, c))
ΣV xt (x(, c)
7 (xx, c) ΣV,
(38)
Σ-
= 11 PTef
A
11 Pref
xi
X
16

---

Published as a conference paper at ICLR 2025
We then put the expression of μs, Meef and E into the above equation:
DKL (po(X,
xx+1, cl..ecpppsss (xx xx+1,
20 IV
Cte
Be+1
o2 - - X
Vx rxx,
(39)
ac+1 vI-
-++1 at+1
We put Equation (27) into our loss function get:
Dsro w(t)
ar
Be+1 (Et+1 - Ep) - Vx log (r (x,, (xx,
c))
20
as+1
vI ac+1
w(t) 40
IV
a
Bit1
(E1+1
En) -20 7 Vx, logo (r (xx, - (x/= c))
a++1 v1 - 04+1
(40)
Therefore, we can observe that LDSPO DKL. (po Xx+1, l)llp Xx+1, c)) when w(() 20 1/2X and we use p.atal.) as reference model, we get the following equation:
Σ T- 1=
Lithf =
EEwD F (x(:r
DDSPO + Alog 2 (c).
(41)
Finally, it completes our proof:
VoLelht = Ve Σ
-DSPO
(42)
C.3 DERIVATION Loss WITH REWARD MODEL OF RLHF
Through the proof in Section C.2, easily get the conclusion that if set p (y xx, c) p (y Xx, c) = exp (r(x,, c)/A) /c(,,, the direction of minimizing our proposed loss is same as maximizing Littt when the hyperparameters of both losses properly adjusted. Specifically, can get the optimization objective with the above format of p (y Xt, c):
(x,
(t) 0
V log
V (x,, c)
(43)
Pdata Xx
Referring to the proof in Appendix B.2 and disregarding some weighting parameters, derive the following equation for the energy-based classifier:
DDSOO = w(t)
at
B++1
(ct+1 - ca) [ (x,,
20
act1
v1 x,+1
(44)
= w((t) 10
IV
Ote
Be41
(E--1 -6) -20 (xe,
ac+1
VI
V 0441
Therefore, we can obtain the equivalent form of Equation (39) with w 2a 2t) 1/2A and we use P-ata(-) as the reference model. Based on this formulation, we can get the following objective when considering r(-) as defined in Equation (11):
Pe(x (x41, 7 Vxx, x
DDSPO-E w(t)
Xx, log
Pdnta X c)
(45)
= A(t) |E Ex+1 73 (E0 - (,e1,0+1)12.
83
where A(t) = (tt) 20:2 I- 60,1+1 = Eg (X141, C, + 1) and similarly for Eref,t+1- The detailed derivation are the same the derivation of Equation (12), which shown in Section B.2.
D EXPERIMENTAL DETAILS
D.1 THE DETAILS OF TRAINING ON SDXL MODELS
Inspired by the observation from Diffusion-DDP (Wallace et al., 2024), which highlights that when the quality of training data is lower than that of data generated by the original model, using the
17

---

Published as a conference paper at ICLR 2025
reference model becomes preferable choice. Therefore, we incorporate the reference model in Equation (10) and derive the following equation for training the SDXL models from Equation (12):
min
A(t) Vxy
(11 (1-B0) xy
B(t)
(c, - (c.x)))
(E0,0+1 rr.
-ref.4+1)
(46)
where we replace the first term ct+1 with Eref.++1- In practical implementation, we observe that the term (Eg.r-1 6ref.++1) is small, yet significantly impacts the training speed. Additionally, using small values of ((7) yields good performance. Therefore, to simplify the training process of SDXL and reduce hyperparameter, we train the following loss for SDXL:
min || - a (c, (r
(47)
Then, we replace r (c, x2) and r (c, x!) as discussed in Appendix B and ignore the relevant part for Og and 3e as the training process of diffusion models.
D.2 THE DETAILS OF DATASETS
In this section, we provide detailed descriptions of datasets:
Pick-a-Pic v2 (Pick V2) (Kirstain et al., 2023): The Pick-a-Pic dataset was developed by logging user interactions with the Pick-a-Pic web application for text-to-image generation. It contains over 500,000 examples and 35,000 distinct prompts. Each example includes prompt, two generated im- ages, and a label indicating which image is preferred there is no significant preference (tie). The dataset was generated using multiple backbone models, including Stable Diffusion 2.1, Dreamlike Photoreal 2.05, and Stable Diffusion XL variants (Rombach al., 2022), with varying classifier-free guidance scale values (Ho & Salimans, 2022).
Parti-Prompts (Yu et al., 2022): Parti-Prompe is a comprehensive dataset consisting of over 1,600 prompts written in English, designed to evaluate and benchmark the capabilities of text-to-image generation models. These prompts span wide range of categories, offering a diverse set of chal- lenges assess model performance across various dimensions.
HPSV2 (Wu et al., 2023): The dataset includes a total of 98,807 images generated from 25,205 unique prompts. For each prompt, multiple images generated, with one image selected by the user as the preferred choice while the others serve as non-preferred negatives. The number of images per prompt varies, with 23,722 prompts having four images, 953 prompts having three images, and 530 prompts having two images.
InstructPix2Pix (Brooks et al., 2023): InstructPix2Pix is dataset designed to edit images based on human-provided instructions. For example, with a prompt like "make the clouds rainy," the model will modify the input image accordingly. conditions output on both the text prompt (editing instruction) and the input image, enabling intuitive, instruction-die driivn image edits. We conducted our image editing experiment on 1,000 test samples from this dataset
D.3 IMPLEMENTATION DETAILS
We present implementation and setup details of DSPO in this section. For experiments, we use the AdamW optimizer with an effective batch size of 2048 pairs, as outlined in Wallace et al. (2024). Training is conducted on NVIDIA V100 GPUs, with a local batch size of 4 pair and gradient accumulation over 128 steps. We train at fixed square resolutions and use learning rate 2.048 10-8, scheduled with a 2000-step linear warmup. followed by inverse scaling (Rafailov et al., 2024). We use the default hyperparameters
present the DSPO results with À 0.001. For fair comparison, for the T2I diffusion model with the image editing task with text instructions, ensuring consistency in evaluation. i.c.. guidance scale as 7.5 and the strength 0.75, Our code of DSPO based on the implementation Diffusion-DPOO
18

---

Published as a conference paper at ICLR 2025
Table 6: Win-rate (VS SD15) comparison of PickV2 dataset for Ablation Studies.
Dataset
Method
Pick Score
HPS
Aesthetics
CLIP
Image Reward
DSPO-ref
65.40
75.00
68,00
57.20
64.40
PickV2
DSPO-nodup
71.20
82.60
74.00
58.60
76.60
DSPO-LoRA
68.00
79.40
74.40
60.20
70.80
DSPO
73.60
84.80
76.20
61.80
78.00
SD15
Diff.-DPO
Diff.-KTO
SFT
DSPO
Image Reward
CLIP
0.36
0.60
0.35
0.45
Score
S .cor
0.34
0.30
0.33
0.15
0.32
0.00
0.31
PickV2 Parti Prompt HP$V2
PickV2
Parti Prompt HPSV2
Dataset
Dataset
Figure 7: Reward score results for with Image Reward and CLIP models on all dataset.
E ADDITIONAL EXPERIMENT RESULTS
E.I ADDITIONAL ABLATION STUDIES AND OTHER EXPERIMENTS
In this section, we conduct the ablation studies by using Pier in Equation (10), denoted DSPO- ref. Additionally, since the PickV2 dataset contains many duplicate prompts, we conduct further experiments by removing these duplicates and rerunning our model, denoted as DSPO-nodup. To reduce memory usage and computational time, we fine-tuned SD15 using LoRA combined with DSPO, referred as DSPO-LoRA. The results of these three experiments are summarized in single table, as presented in Table 6. We get the following observation: (i) our method, which leverages the score function the true data distribution, outperforms the approach (DSPO-ref) that treats the original pretrained model as the reference model. This is because the quality of images generated by the original pretrained stable diffusion models (SD 1.5) is lower than that the fine- tuning dataset. As result, using the score function of the true data distribution outcomes; (ii) DSPO performs better compared with DSPO-nodup. It means that randomly dropping duplicate samples without careful selection may have negatively impacted the results, potentially leading to the loss of important information by randomly retaining only one instance of prompt and discarding the others; (iii) fine-tuning with LoRA enhances performance, though it slightly falls short of achieving results similar to full parameter training. However, the performance gap between these two methods is minimal, making LoRA viable approach for fine-tuning large models.
produce better
E.2 ADDITIONAL REWARD SCORE RESULTS
In this section, we present additional reward score results using CLIP and Image Reward, as illus- trated in Figure 7. Consistent with previous findings, our model surpasses all baseline methods, further demonstrating the effectiveness of DSPO.
E.3 ADDITIONAL RESULTS FOR QUALITATIVE ANALYSIS
To further verify the effectiveness of our model, we provide more Qualitative results of Text to image generation for different baselines in Figure 8. We list the prompts used in Figure 8 follows:
Minotaur
2. a painting of a fox in the style of starry night
3. The image a vibrant and intricate illustration of a man, with a focus on his shoulder and head, created using inkpen and Unreal Engine technology.
19

---

Published as a conference paper at ICLR 2025
4. portrait of young girl
5. A head shot of pretty girl dressed in eyberpunk version of Marie Antoinette's ' rococo style, depicted through detailed digital art and trending on Art Station.
6. Nine human faces from Neanderthal Modern Human and beyond depict the future of human appearance.
7. A digital painting of a young pirate with sharp features and a piereing gaze.
8. Cyberpunk cat.
Moreover, we also provide more qualitative results of the image editing task with text instructions in Figure 9. Similarly, we list the prompts used in Figure 9:
1. make it marble
2. A fantasy landscape, trending artstation
3. turn it into painting
4. make a seascape
5. make the cub tiger
6. it into a computer game
7. As an oil painting
F ETHICAL STATEMENT
This study explores algorithms fine-tune text-to-image diffusion models for human prefer- ence alignment. We use public data following the prior works (Wallace et al., 2024) in the field of human preference alignment text-to-image generation for both training and evaluation, which can be directly downloaded from Hugging Face. Moreover, no sensitive user information is exposed, and all experimental results are presented as aggregate statisties maintain reproducibility without risking information leakage. These practices comply with ethical and legal standards, ensuring responsible approach AI research.
20

---

Published as a conference paper at ICLR 2025
SD15
SFT
DPO
KTO
DSPO
Figure 8: Images generated by different models for various prompts which are selected from Pick Parti-Promph and HPSV2. Detailed prompts for these images are provided in Section E.3.
21

---

Published as a conference paper at ICLR 2025
Original
SD15
SFT
DPO
KTO
DSPO
MOKC PP MPF TMMP PE
PEPPERPEDD
B PUURPP DUAP
Figure 9: Images generated by different models for various prompts which are selected from In- StructPix2Pix of text-guided editing. Detailed prompts for these images provided in Section E.3.
22


Reviews:
[{"review_id": "ht4bDf3tOL", "reviewer": "Reviewer_HsUR", "summary": "The authors propose the first method to fine-tune text-to-image models on human preference that aligns pertaining and fine-tuning objectives, the benefit of which is shown by outperforming preference learning baselines on human preference tasks. The method is also shown to be equivalent with RLHF objectives in diffusion models under certain conditions.", "questions": "1. In table 2, Diff.-DPO seems to have a higher score in the PickV2 dataset with the Pick Score Metric, but it’s not highlighted.\n2. The paper shows the equivalence between the proposed method and RLHF, but it seems like the evaluation does not contain a direct comparison with a model that has been fine-tuned with RLHF.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The first method that fine-tunes diffusion models on human preferences using a score-matching, thus aligning with pretraining objectives, and the benefits are apparent through the baselines.\n2. Clear presentation of related works provides the reader with ample context.\n3. Comprehensive experiments covering supervised finetuning, diffusion-DPO, diffusion-KTO, and MaPO as baselines, evaluated with multiple scoring metrics, and includes evaluations on a recent model (SDXL).\n4. The proposed method outperforms baseline methods across multiple datasets and scoring metrics.", "weaknesses": "1. Figure 3 looks cramped; the equations are pitted against other visual elements, making them hard to read", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSSiykVrlq", "reviewer": "Reviewer_wk6Q", "summary": "This paper proposes a new direct score preference optimization method for diffusion model alignment that utilizes a target human-preferred score function, thereby aligning the fine-tuning objective with the pretraining objective.", "questions": "- In Figure 2, what baseline models were used to calculate the win rate? Is it the pretrained SD1.5 model?\n- During evaluation, did the authors generate images from multiple fixed seeds and average the results over them, or do the results come from a single specific seed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper takes a different approach to aligning text-to-image diffusion models, motivated by score matching, which sets this method apart from the others.\n- In terms of multiple open-source reward scores, DSPO demonstrates effectiveness in increasing reward values.", "weaknesses": "- In general, I find that many claims are too vague and ambiguous. When we examine the final loss of Diffusion-DPO and DSPO, how can we definitively say that one aligns with the pretrained loss of Stable Diffusion more clearly than the other? Additionally, why is aligning the diffusion model with direct reward optimization or RL considered suboptimal due to a mismatch in pretraining and fine-tuning objectives? Is there any theoretical justification beyond the win rates?\n- In my opinion, since the method is based on human preference, a human evaluation should be conducted to confirm whether it truly increases the reward aligned with human preference. Relying solely on open-source reward model scores seems unreliable, as these models can carry inherent biases.\n- Furthermore, why does the Diffusion-KTO result differ so significantly from the original paper? I think the authors should provide detailed explanations of their evaluation settings, including the seeds used, the number of images generated per method, and other relevant factors. Without this information, the results may appear unreliable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j8eI5NPrjC", "reviewer": "Reviewer_gpAu", "summary": "The paper introduces DSPO, presenting a score-matching formulation for fine-tuning pre-trained diffusion models on human preference data. The authors argue that since existing preference alignment fine-tuning methods have a different objective than the pre-training objective, it can lead to sub-optimal results, and they demonstrate this with empirical evidence.", "questions": "* Figure 2 could mention the base model on which the respective methods were applied.\n\n* L099 - L101: The authors mention \"... with existing baselines for preference learning in T2I diffusion models.\" However, Figure 2 compares the performance of a single base model on which the respective methods were applied. So, I think it's better to be specific and mention the base model in the statement.\n\n* Equation 12 could benefit from an expansion of the notations used. For example, I don't know where $\\lambda$ is coming from. Furthermore, it'd be beneficial to highlight the score function of the data distribution replacing $p_{ref}$.\n\n* It's not clear how DSPO incorporates $\\mathbf{x}_t^w$. Under Section 4.2, $\\mathbf{x}_t^w$ only appears in Equation 14. \n\n* L091: Typo on \"constraints\". \n\n* SD1.5 is a relatively old model. Since DSPO doesn't consider other recent models like SDXL, SD3, Flux, etc., it's unclear as to how well DSPO generalizes. I can understand that providing further results on SD3 or Flux might be computationally challenging, but I request that the authors at least consider SDXL experiments. Additionally, LoRA fine-tuning (similar to how DPOK [1] does it) when doing DSPO for larger models like SD3 and Flux might help them quickly evaluate its potential better. \n\n* Are there any sample-efficient aspects of DSPO? More specifically, I am interested to see if using the score-matching perspective of alignment fine-tuning like DSPO does can improve alignment with fewer samples than other methods.\n\n* The authors could also consider using human-benchmark arenas such as imgsys [2] for evaluation. \n\n* To assess the practical aspects of DSPO, it would be useful to report the wall-clock time and memory requirements of DSP and compare them against the existing methods. \n\n**References**\n\n[1] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models; Fan et al.; 2023.\n\n[2] imgsys; fal.ai team.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The score-matching formulation for alignment fine-tuning of diffusion models hasn't been explored before, and the paper does a good job of exploring this direction. \n* The connection between the objective covered by the RLHF methods for diffusion models and DSPO.", "weaknesses": "* The paper misses out on using MaPO [1] as a reasonable baseline even though it considers contemporary works like Diffusion KTO. The reason why I think MaPO is important here to consider is because it has similar motivations and also either performs on par with Diffusion DPO or outperforms it under various settings. \n* Lack of experimental results on models like SDXL makes it unclear as to how scalable DSPO is and if it works for models other than SD v1.5. \n* The ablations lack experiments on some of the design choices the authors make to arrive at the final objective of DSPO. For example, they use the direct score function of the underlying data distribution as opposed to using that of $p_{ref}$, but they don't justify it with sufficient experimental results. \n* Pick-a-Pic v2 contains duplicate prompts. Did the authors perform any de-duplication? If not, I think it might be better to run at least a few experiments with de-duplication to check if this improves the results.\n\n**References**\n\n[1] Margin-aware Preference Optimization for Aligning Diffusion Models without Reference; Hong et al.; 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)