Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
Authors: Yi-Lun Wu
Bo-Kai Ruan
Chiang Tseng
Hong-Han Shuai
Venue: NeurIPS 2025 poster ()
Keywords: Text to Image Generation
Inverse Reinforcement Learning
Primary area: deep_learning
TL;DR: We present a learning framework that aligns text-to-image diffusion models with human preferences through inverse reinforcement learning and a balance of offline and online training.

Abstract: Direct preference optimization (DPO) methods have shown strong potential in aligning text-to-image diffusion models with human preferences by training on paired comparisons. These methods improve training stability by avoiding the REINFORCE algorithm but still struggle with challenges such as accurately estimating image probabilities due to the non-linear nature of the sigmoid function and the limited diversity of offline datasets. In this paper, we introduce Diffusion Denoising Ranking Optimization (Diffusion-DRO), a new preference learning framework grounded in inverse reinforcement learning. Diffusion-DRO removes the dependency on a reward model by casting preference learning as a ranking problem, thereby simplifying the training objective into a denoising formulation and overcoming the non-linear estimation issues found in prior methods. Moreover, Diffusion-DRO uniquely integrates offline expert demonstrations with online policy-generated negative samples, enabling it to effectively capture human preferences while addressing the limitations of offline data. Comprehensive experiments show that Diffusion-DRO delivers improved generation quality across a range of challenging and unseen prompts, outperforming state-of-the-art baselines in both both quantitative metrics and user studies. Our source code and pre-trained models are available at https://github.com/basiclab/DiffusionDRO.

Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
Bo-Kai Ruan Institute of Electrical and Computer Engineering, National Yang Ming Chiao Tung University fyilun. ee08, bkruan. ee11, aniang.eele hhshhaal)@@ycu. edu.tw
Yi-Lun Wu
Chiang Tseng
Hong-Han Shuai
Abstract
Direct preference optimization (DPO) methods have shown strong potential in aligning text-to-image diffusion models with human preferences by training on paired comparisons. These methods improve training stability by avoiding the REINFORCE algorithm but still struggle with challenges such as accurately esti- mating image probabilities due the non-linear nature of the sigmoid function and the limited diversity of offline datasets. In this paper, we introduce Diffusion SiffuusionDROO)
Denoising Ranking Optimization framework grounded in inverse reinforcement learning. Diffusion -RO removes the dependency on reward model by casting preference learning as a ranking problem, thereby simplifying the training objective into denoising formulation and overcoming the non-linear estimation issues found prior methods. More- over, Diffusion-DRe uniquely integrates offline expert demonstrations with online policy-generated negative samples, enabling it to effectively capture human prefer- ences while addressing the limitations of offline data. Comprehensive experiments show that Diffusion-DRG delivers improved generation quality across range challenging and unseen prompts, outperforming state-of-ther of-the-at art baselines in both both quantitative metrics and user studies. Our source code and pre-trained models are available at
new preference learning
Introduction
Text-to-image diffusion models have recently emerged powerful class of generative models, achieving impressive results in synthesizing high-fidelity images from textual descriptions [37] 27
33
32 [2 These models use iterative denoising progressively transform random noise into 5
coherent visuals aligned with the input text [3]. Despite their capabilities, users often expect outputs that not only match the text but also reflect implicit aesthetic stylistic preferences that hard to encode explicitly. As a result, aligning these models with nuanced human preferences has become an emerging challenge.
Existing approaches to preference alignment in generative models have predominantly relied reinforcement learning frameworks, such as Reinforcement Learning from Human Feedback (RLHF) 66, 2 9 [6 28). In these methods, models are fine-tuned using reward signals derived from human evaluations, often requiring paired datasets where output is deemed better than another. Methods such as Direct Preference Optimization (DPO) have been employed large language models (LLMs) and diffusion models optimize for human preferences effectively 30 35 24]
However, the necessity for paired comparative data introduces substantial practical limitations. Collecting this data is labor-intensive and time-consuming, and it might not encompass the full spectrum of user preferences, potentially omitting users' favored choices. Moreover, even when successfully obtained, paired data may not effectively optimize for user preferences. For example, differentiating between two high-quality options may not yield meaningful insights for preference
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

determination, as both already meet the desired criteria. Conversely, comparing two poor examples fails to provide the model with positive references needed to avoid undesirable features. In both cases, the comparative data may be too limited to enable the model to discern and learn the nuances that truly align with human preferences, potentially resulting in inconsistent or suboptimal outputs.
In this paper, we propose a novel approach for fine-tuning diffusion models to optimize user pref- erences using only demonstration examples- images that embody the desired quihhuiss-withuut comparing them to less preferred outputs. This method departs from traditional approaches reliant on paired data, where examples are ranked against each other. Instead, we use these positive examples as direct guides, teaching the model to produce outputs that reflect the desired attributes. By theo- retically deriving a novel optimization objective. we enable the diffusion model learn from these demonstration examples while mitigating the risk of overfitting.
The contributions can be summarized as follows.
We introduce a novel framework for preference optimization in diffusion models that requires expert demonstrations only, addressing the limitations of methods that depend on paired comparative data.
We derive an optimization objective that compares the model's s outputs with the demonstra- tion examples. This formulation ensures effective learning from positive examples while preventing overfitting.
Through extensive experiments, we demonstrate that our method achieves a preference rate exceeding 70% in terms of PickScore compared to state-of-the-ar models. Our approach not only better aligns with desired human preferences but also exhibits robustness and strong generalization unseen data.
2
Related Work
To guide diffusion models toward preferred outcomes, Reinforcement Learning (RL) has been widely adopted, including DDPO [I] and TDPO-R 42), which apply REINFORCE to optimize generation trajectories based on human feedback. However, these methods often suffer from complex reward design and high variance, leading to unstable training S. To address these challenges, Fan et al. [9] propose DPOK, introducing a KI. divergence term to penalize deviation from the base model and prevent reward hacking [10]. Similarly, PRDP [ uses distillation strategy where a reward model predicts preferences to guide diffusion model updates, though remains reliant on the reward model's 's accuracy.
Recent work has also explored Direct Preference Optimization (DPO) in diffusion settings. D3PO [39 and Diffusion- -DOO [B5] adapt DPO techniques from LLMs to fine-tune diffusion models directly from preference- paired data without a separate reward model. Diffusion-KTD [17] further simplifies supervision by decoupling preference pairs into binary positive/negative sets, though this can introduce semantic biases-e.. e.g., negative sets skewed toward specific concepts such cats, the model may learn undesirable associations.
While RL and DPO-based methods advance preference alignment, they heavily rely on large-scale paired data. SPIN-Diffusion [4 circumvents this by leveraging carlier model checkpoints as negative samples, enabling self-improvement from positive-only data. However, its multi-stage pipeline demands careful hyperparameter tuning to ensure consistent performance.
In contrast, we formulate preference alignment as max-margin inverse reinforcement learning (IRL) problem, deriving single-stage objective that encourages the generation high-quality samples. This formulation eliminates the need for negative samples, thereby reducing semantic bias to some extent. Moreover, provides more stable and interpretable training process without requiring iterative self-play or stage-wise tuning.

---

3 Method
3.1 Background
We begin with a brief overview of the reinforcement learning framework and the foundational equations for our subsequent derivations. Specifically, for given image-text pair (zo. c), there exists an optimal reward model r c) that assigns a score representing the level of human preference for the provided image-text pair. Building on prior work 99
35 39,, the objective of reinforcement learning from human preferences, as defined by the reward model, is formulated as follows:
max Ee-CC. PoopCa.o-oo (( c) =I (xa pe.r
where C represents the set of prompts, 33 is the regularization weight, and po., denotes the reference distribution, typically provided by a pre-trained diffusion model. As shown in prior work 11 15. 25 26]. the optimal density function for Eq. (T) can be derived as:
(xolc)
Z(c) P(c) Pea (xolc) exp
where Z(c) Sm Pe. (z)lc) exp (a c) day is the partition function. Through algebraic manipulation, the reward function can be reformulated as:
(ool)) pe (x0
r( c) B log
+ Bog Z(c).
Nevertheless, calculating the probability of a clean image in diffusion model requires marginalizing the joint distribution pe (xole) = J J Pe (e This approach necessitates back- propagation through time 38 update the model, leading substantial increase in memory requirements for training. To mitigate this, we propose a reward model for the entire denoising trajectory, which enables a stepwise gradient calculation to improve computational efficiency.
3.2 Trajectory Reward Modeling
Conventional reward models typically predict preference scores solely for the final clean image 2tg without considering the denoising trajectory. To decompose the full denoising process into individual 42 and assume the existence of trajectory reward model R(Zor, c)
steps, we follow prior work 35
for diffusion model po P.rrl (c) such that:
r(wn, c) oe (x..)) R( c)],
where r(æo, c) aligns with human preferences defined in Eq. ( By substituting the reward function in Eq. 4 into Eq. ( and applying the data processing inequality to expand the KL divergence, we have:
Bew.C,
R(xOrT c)]
3IDki. po (eleeee)plpppaaapp
R R( c)]
kL po (a (morrlc))
(5)
Following the similar derivation as in Eq. given by:
(2. the optimal joint density for the lower bound in Eq. (5)
Z(e) pe (worTlc) exp (
2o (x).
(6)
where Z(c) = Soor Pe (x):)lc) exp R(xorT c)) dxoe is the partition function for the joint density.
3.3 Max-Margin Inverse Reinforcement Learning
Human preferences are represented by the labeled data originally used to train reward models. To avoid issues of error accumulation and reward hacking [10), advantageous remove the reward model from preference fine-tuning entirely. This approach aligns with inverse reinforcement

---

learning (IRL). which aims to learn policy based on expert demonstrations. We apply max max-ma
approach T 23 train a reward model that satisfies the the inductive step condition:
(rrci))r(aaaa ( ct] 2 Be (mol (mzo (1(20. c)
Be-C.a.-D(e)
where Zy and Zo samples from the expert demonstration ((c) and the policy model pe (xole).
respectively. Once we establish a reward model F(Z0. c) based on Eq. (7). a new policy model Po is
then obtained by maximizing the KL-regularized objective (Eq. (5) using the reward model f(zo, c). Through altering these two optimization processes, we can obtain the optimal policy model that aligns with the expert demonstrations [
However, the reward model aims maximize the margin between the expert and the policy, while the policy model seeks to minimize this margin to align more closely with the expert. This minimax optimization usually suffers from instability and an exhaustive tuning process. To this end, we further simplify the optimization procedure, First, the inductive criteria Eq. 7 can be rewritten by substituting the reward function from Eq. 4 into it:
(e [( c) =or [( e)]
(8)
We propose to parameterize the trajectory reward model R by using formulation similar to Eq. (3)
(x c) = Blog P c) + Blog Z(c),
(To.rl))
where & represents the learnalle parameters of the probability model pp. Assuming that there exist
reward model Rg, parameterized as in Eq. (9) and satisfying the inductive criteria in Eq. (8), we can further obtain the optimal policy by substituting Rs into Eq. (6):
po(xurlcc = pe,. (Tu:lIc)
1 3
R
Z(c)
exp
pen. (Fo.r
Pos (2
Z(c)
exp
log
Pe (aurlc))))))
+ log Z(c)
= Ps (Torrllc))
(10)
This result implies that the optimal policy model Po is identical reward probability model P Therefore, the alternating optimization reduces to reward modeling alone, where the maximum expected reward is implicitly achieved by Eq. (0) for any given Rs.
To optimize the reward model, subtract the right hand side from the left hand side of Eq. ( and substitute the reward parameterization from Eq. (9) into it. Moreover, we use the forward diffusion ( (To) to approximate sampling expert trajectory To.T from the expert demonstration D(c):
Peb
P (Zo.T c)
Po (c)
(in)
8 log
Pa (Fo.T [c]
EasC mor- (e) B log
pa (20.T
Through algebraic manipulation, this ranking objective is equivalent to (a detailed step-by-step derivation is provided Appendix[D
T
Σ Eo.. |lE -
e
c (Z,, e,
E es (
(12)
T Σ Ec.ar ||e- Eon - c, [
- --e--
where H, ~ (0)))))) represents samples drawn from forward diffusion with perturbation noise cœ( c, is the noise predicted by the policy diffusion model.
È ~ I), and e eo (I2,
Connection to Supervised Fine-Tuning
The ranking objective in Eq. [12 can be solved by directly minimizing the negative of the margin:
Luun (Ø)
Σ
B E,,
(e-co(a,.ct))
co(o,, c,t) - cs (I c,
(13)
Same SFT
Push

---

Algorithm Diffusion Denoising Ranking Optimization
Input: Reference diffusion model pe prompt set C, expert demonstration set (D(c) number of update steps N, policy model update interval M, batch size B, and clipping threshold m. Initialize policy model < Initialize reward model
pe Po..
2: Pe < Po..
3: for 1 N do
4:
for
to B do 1~ [
5:
sid 2o idd E(
6:
7:
ãt ~ a
Forward diffusion
8:
2t ~ po (2rlc)
Sample from policy
9:
- col It, (1, c,
C +
E (Tt, c,t) ct)
EΦ (a,, c. 1) Ep cxr C,
11:
CR + ||e (0 C,
12:
end for
CTRL (Ø) + B Σ max (,, - + fRt
Eq. 15
14:
Update & using gradient V &RLTRL
15:
if i multiple of M then
16:
Pa Ps
Update policy model
17:
end
18:
end for
We eliminate terms that do depend Φ they do not contribute to the gradients of reward model. We notice that supervised fine-tuning corresponds to optimizing the first term, which minimizes the KL-divergence between the expert distribution and the distribution induced by the reward model. The second term serves complementary purpose, where the reward model generates negative samples online guide the optimization in the correct direction.
Connection to DPO-based Approaches
aim to learn preference predictions using the Bradley-Terry 4 model. Diffusion-DPO applies Jensen's inequality to transform the objective from a probability-based form to a noise-prediction form (Fq.(14) in Wallace et al. (35). This transformation closely resembles solving the ranking problem in Eq. (12) by maximizing the margin using a cross-entropy loss:
Previous DPO-based approaches 35
39
30
40
Σ Be, (Ile E EB ( G./l -1l=
T
c e, c1)l)²)))
Loe(()) = -llog
(14)
Ccsc e, (1 (TF,, (,, -l 10 ()
where (Ix) 1/(1 + cxp(-x)) denotes the sigmoid function. Unlike prior work, our approach does not require Jensen's inequality to minimize the surrogate upper bound. The max-margin approach enables direct optimization of the margin while ensuring convergence the policy model. Specifically, sample pairs from expert demonstration and policy density, whereas DPO-based methods compare preference pairs (xx", x²), where x" is preferred over x'. In other words, the proposed inverse RL approach decouples the need for preference pairs and further eliminates the reliance on negative samples. In practice, public preferences can be obtained through simple statistical methods to rank different samples, with higher-ranked ones treated as expert demonstrations that align with public preferences. The same method can be easily extended to collect expert demonstrations reflecting individual preferences.
3.4 Thresholded Ranking Loss
While our proposed reward parameterization reduces the learnable parameters to include only the reward model, two separate models are still maintained to represent the reward and policy, respectively.
5

---

To ensure stable reward model updates, the policy model is periodically synchronized with the latest reward model after a predefined number of gradient steps (see line [n Alg. However, updating the policy too frequently can limit the reward model's ability to adapt and distinguish expert behaviors from policy outputs. Conversely, infrequent updates may cause the reward model overfit to the current policy. To balance this trade-off, we introduce thresholded ranking loss (TRL), which clips the margin loss once the inductive criterion is sufficiently satisfied:
(TRL(Ø)= = Σ Ee E, max m, (IE E- can (1,, c.
|| [
(15)
+ (e-- ( (,),,))
Eo e, -ee
where m is a predefined parameter that adjusts the baseline at which the reward margin is truncated. For samples that already satisfy the inductive criterion, further optimization of the margin is unnec- essary. By clipping the margin in these cases, the model can concentrate on rectifying incorrect rankings, thereby avoiding overfitting to samples that are already ranked correctly.
refer to the learning process with the objective ( as Diffusion Denoising Ranking Optimiza- tion Diffusion-DRO) This method learns the ranking relationships between expert demonstrations and policy behaviors. The training process is detailed in Algorithm|
Experiments
We first outline the datasets, implementation details, and evaluation protocols used in our experiments. We then evaluate Diffusion-DRe -DRO against multiple baselines using quantitative metries and supplement our findings with a user study on Amazon Mechanical Turk for qualitative comparison.
Datasets. Following prior works 17 18], we the train split of Pick-a-Pic v2 [1] (MIT license) as our training dataset. For evaluation, we adopt the test split of Pick-a-Pic v2 and the HPDv2 benchmark [36] (Apache-2.0 license), representing in-domain and out-of-domain scenarios, respectively. Each sample includes a prompt, two images, and a human preference label. Due to label sparsity, we simulate expert demonstrations using automated metrics such as PickScore [14] (MIT license) and HPSv2 [36] (Apache-2.0 2.0 license). We rank all training pairs by these scores and select the top as expert demonstrations; unless otherwise stated, K-500. Ablation results for varying K are provided in Section 4.4
Evaluation. We evaluate the human preference alignment by comparing it with various baseline meth- ods. Preference scores are computed using five different score models: PickScore [14, HPSv2 [36), Aesthetic (34) (MIT license), CLIP Score [29 (MIT license) and ImageReward 38 (Apache-2.0 license). For each preference score, we report the win rates between the Diffusion-DRO and the baseline methods, defined as the proportion of generated results with scores exceeding those of the baselines. To ensure fairness, we avoid using the same preference score for selecting the expert demonstrations and calculating the win rates, as this could inadvertently leak score prior information into the train data. Specifically, we use HPSv2 to select the expert demonstrations and calculate win rates for all metrics except for HPSv2. The experiments using different metrics to select expert demonstrations can be found in Appendix B
Implementation Details. We fine-tune Stable Diffusion 1.5 (SD v1-5) 31 (CreativeML. Open RAIL-M license) using Diffusion-DROO, ensuring consistency across all baseline methods. To sample ac from the policy model, employ DPMSolvvr++ (2]] with 20 steps, without using classifier-free guidance [2]. For inference, we use the DDPM sampler with 50 steps and a classifier-free guidance scale of 7.5 generate five images per prompt for all methods. Among these five generations, we select the image with the median PickScore as the final result. For additional implementation details, please refer Appendix A.2
4.1 Quantitative Results
We compare Diffusion-DRO with strong baselines, including SPIN-Diffusion 41 (Apache-2.0 license), Diffusion-SPO [18] (Apache-2.0 license), Diffusion-DPO (35) (Apache-2.0 license), and

---

Table 1: Automated win rates between Diffusion-Diie and baseline methods. The dagger symbol (?) indicates that the evaluation was performed the officially released model weights. Note that S1> v1-5 w/ SFT refers to SD v1-5 fine-tuned on expert demonstrations. Win rates greater than 50% are
highlighted in bold.
Pic-a-Pic v2 Test
HPDv2 Benchmark
Baseline Method
Aesthetic
CLIP
Aesthetic
CLIP
PickScore
ImageReward PickScorc
ImageReward
Score
Score
Score
Score
SD v1-5
87.80
85.20
48.40
88.60
90.47
82.91
46.59
87.69
SD v1-5 w/ SFT
71.20
58.00
66.40
57.80
70.62
57.22
64.97
62.03
SPIN-Diffusion
56.20
64,80
58.20
70.60
54.87
62.78
54.78
69.78
Diffusion-SPO
62.80
63.60
71.40
78.00
60.59
67.66
75.78
77.94
Diffusion- SPO w/ SFT
86.60
81.60
42.40
87.20
88.75
80.25
42.69
85.78
Diffusion-DDDD
78.40
83,20
41.40
84.20
79.75
79.97
39.09
82,25
Diffusion- DPO w/ SFT
64.00
55.00
59.00
56.20
63.62
56.12
59.91
58.75
Diffusion-KTT KTO
74.20
69.00
42.20
66.60
71.19
71.03
39.81
62.81
Diffusion-KTT w/ SFT
70.20
58.60
64.00
58.60
71.09
56.12
65.31
62.75
Diffusion-KTO 17 These methods remove the need for reward model and are fine-tuned from SD v1-5, consistent with our setup.
Since both expert selection and evaluation rely automated metrics, there may be concerns about potential information leakage. To address this, we additionally fine-tune SD v1-5 on our selected expert demonstrations and select the best checkpoint based on PickScore performance on the Pick-a- Pic v2 test set, denoting it as SD v1-5 w/ SFT. Using this model, we further fine-tune Diffusion-DPO Diffusion-KTOC and Diffusion-SPO with their official implementations" These variants are labeled with the postlix "w/ SFT." Table ! using various automated preference scores. Full results with raw scores and standard deviations are provided in Appendix
reports the win rates of Diffusion- DRO against all baselines
For the SD v1-5 w/ SFT, performance improves in PickScore, Aesthetic, and ImageReward compared to SD v1-5 (resulting lower win rates for our method). We attribute this to expert demonstrations enhancing preference alignment. However, an interesting observation is the decline in CLIP Score. This can attributed to the model slightly deviating from the original text encoder distribution, which was trained on large-scale data. Since Stable Diffusion and CL.IP use identical text encoder weights, this deviation leads to a decrease in CLIP Score for SD v1-5 w/ SFT. The same phenomenon is also observed in Diffusion-DDi DDO w/ SFT and Diffusion- KTO w/ SFT since they are fine-tuned from SD v1-5 w/ SFT. For the Diffusion- SPO w/ SFT, their proposed step-aware preference model leverages the CLIP vision and text encoders select the best and worst samples for fine-tuning. Therefore, the CLIP Score of Diffusion-SPP w/ SFT increases due to the consistent distribution.
We observe that Diffusion-DRo significantly outperforms all state- sfate-of-the-art art approaches across multiple metrics, including PiekScore, Aesthetic, and ImageReward. Even when compared to stronger baselines, such as Diffusion-KTO w/ SFT and Diffusion-DPO w/ SFT, Diffusion-DRO remains the preferred method terms of all automated evaluation scores. For Diffusion-SPO its reliance online sampling for step-aware preference pairs makes it susceptible to the generation quality and diversity of the pre-trained Stable Diffusion model. While fine-tuning Stable Diffusion with expert demonstrations enhances preference alignment, it also reduces the variation in step-aware preference pairs. As a result, Diffusion-SPe w/ SFT fails to gain any performance improvement over SD v1-5 w/ SFT. Consequently, Diffusion-DRe significantly outperforms Diffusion-Si SPO w/ SFT, achieving win rates exceeding 80% across PickScore, Aesthetic, and ImageReward.
SFT,
Notably, Diffusion-DROO is fine-tuned directly from SD v1-5, unlike strong baseline methods such as Diffusion-DPP w/ SFT and Diffusion-Kol w/ SFT. Despite this, Diffusion-DRO outperforms methods that start fine-tuning from SD v1-5 w/ SFT, demonstrating its capacity to effectively learn human preferences from expert demonstrations.
OPIN-Diffusion
41 does not provide valid source code for fine-tuning from SD v1-5 w/ SFT.
7

---

Diffusion- DRO Rate
DiflusinKKTO uy SPT
DPFision-IPO o/ SFT
66.677
User study results comparing Diffusion-DRO with baseline methods. The win rate represents the proportion of survey questions where users preferred Diffusion-DRO over the baselines.
Figure 1:
Diffusion-DRi SPIN- DIfN-Diffssoon
Diffusion-
Diffusion- KTO
Diffusion-So
SD v1-5
Figure 2: From top to bottom, the text prompts are: "A Pixar lemon wearing sunglasses beach." "A dragon sitting on couch in a digital illustration," " "A detailed painting of Atlantis by multiple artists, featuring intricate detailing and vibrant colors," and "A passenger jet aircraft flying in the sky."
Compared to SPIN-Diffusion, both methods use generations from diffusion models as negative samples. However, Diffusion-DRO simplifies the training process into single stage by adopting max-margin inverse reinforcement learning (IRL) formulation. This advantage allows Diffusion-DRRO achieve over 60% win rate Aesthetic Score and nearly 70% win rate ImageReward, outperforming SPIN-Diffusion.
4.2
User Study
The user study compares Diffusion-DRo with baseline methods, including SD v1-5, Diffusion-DPP w/ SFT, and Diffusion-KTO w/ SFT. Text prompts randomly sampled from the HPDv2 Benchmark to generate images for evaluation. Detailed settings of the user study are provided Appendix [
Figure D presents the results of our user study, showing that Diffusion rate against SD v1-5. This demonstrates the effectiveness of our training procedure in improving the pre-trained SD model, as human evaluators consistently prefer images generated by Diffusion-DRO. Additionally, the win rates of Diffusion-DRO against Diffusion-DPO (56.6) (56.7%) and Diffusion-KTO (66.67%) further support the reliability of Table For instance, the average win rate against Diffusion- DPO w/ SFT is 59.69 59.6%, and against Diffusion- -To w/ SFT 63.82%. closely aligning with the user study findings.
achieves a 75% win

---

Pick-a-Pic Pic v2 Test
21.5
PickS
Aesthetic
6.4
CLIP Score
0.31-
ImugeReward
0.8
21.0
Dilltion DRD
0.33
0.7
6.2
100
500
1k
5k
100
500
1k
6k
100
500
1k
5k
100
500
1k
5k
Top-K
Top-K
Top-K
Top-K
IIPDv2 Benchmark
6.7
PickScore
Assthetie +
CLIP Score
0.35
ImageReward
0.8
21.5
6.6
Diflusion DRO
0.34
21.0
SPT
0.6
100
500
1k
5k
100
500
13
5k
100
500
1k
5k
100
500
1k
5k
Top-K
K
Top-K
Top-K
K
Top-K
Figure Evaluation results of Diffusion-DRO and SFT trained with varying amounts of expert
demonstrations.
4.3
Qualitative Results
In Figure 2 we present the generation results of different methods. The example prompts from top bottom sampled from the four categories of the HPDv2 Benchmark, namely Anime, Concept Art, Paintings, and Photos. In the first row, Diffusion-DD successfully generates "lemon wearing sunglasses," while SPIN-Diffusion, Diffusion-KTO, and SD v1-5 fail to produce accurate results. In the second row, both Diffusion-DRO and Diffusion-KTT correctly generate the "dragon," the couch," and the action "sit," whereas other methods produce incorrect objects or actions. For the third row, Diffusion-DROO captures the intricate details of Atlantis, while Diffusion-DPP and Diffusion- KTO generate abstract content. the final row, only Diffusion-DRO produces realistic airplane, whereas the outputs from other methods result in implausible shapes. These examples highlight that Diffusion-DRo significantly improves both text alignment and visual fidelity.
4.4 Ablation of Expert Demonstration
In previous experiments, we observe that SFT delivers competitive performances. Therefore, we are interested in exploring the performance disparity between Diffusion-DRO and SFT under different volumes of training data. To investigate this further, we utilize HPSv2 to select varying quantities of expert demonstrations. These demonstrations are then used train both Diffusion-DRO and SFT models. The results, depicted in Figure 3 reveal that the CLIP Score consistently increases with the size of the training dataset. This phenomenon can be attributed the fact that the text encoder in SD v1-5 is identical to the one used in CLIP, resulting in improved scores as the training data volume increases.
Furthermore, across various test sets, Diffusion-DRO outperforms SFT terms of PickScore, Aes- thetic, and ImageReward metrics. These results demonstrate that the thresholded ranking loss consistently enhances Diffusion-DRO's alignment with human preferences. We attribute this improve- ment to fundamental difference in learning objectives. That is, SFT focuses solely on minimizing KL divergence, which neglects the additional expert priors embedded in expert demonstrations. In contrast, Diffusion DDRs leverages these priors by treating policy actions as negative samples, thereby enabling more effective training.
5 Conclusion
propose Diffusion-DRO, preference learning framework for text-to-image diffusion models based on inverse reinforcement learning. By reformulating the objective to remove the non-linear sigmoid function, our method simplifies optimization into a denoising task, improving training effi- ciency and stability. Diffusion-DDR DRO further balances offline and online training by combining expert

---

demonstrations with policy-generater negatives, addressing the limitations of offline data. Com- prehensive experimental evaluations and user studies demonstrate that Diffusion Eiffusion-DRO consistently outperforms state-of-the-art baseline methods across diverse and unseen prompts. By integrating human preferences more effectively, our method achieves superior generation quality, making it robust and scalable solution for preference alignment in text-to-image generation tasks.
Acknowledgments
This work is partially supported by the National Science and Technology Council, Taiwan, under Grant: NSTC-112--221---44000---11 and NSTC-112-2211--44444444
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first International Conference on Machine Learning (ICML), 2004.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL RRLhttps:: : /ttes///axxiii http://axxvv.....aaaa/// 2220/ 05862
[3] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
[4] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-355, 1952.
[5] Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-StalphaS: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations (ICLR), 2024.
[6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Ben- gio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017.
[7] Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations (ICLR), 2024.
[8] Fei Deng, Qifei Wang, Wei Wei, Tingbo Hou, and Matthias Grundmann. Prdp: Proximal reward difference prediction for large-scale reward finetuning diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 042337433, June 2024.
[9] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning ext-to-image diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 79858-79885. Curran Associates, Inc., 2023.
[10] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning (ICMI.,, volume 202, pages 10835-10866, 23-29 Jul 2023.
10

---

[11] Dongyoung Go, Tomasz Korbak, Germän Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202,
pages 11546-1 1583, 23-29 Jul 2023.
[12] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6840-6851 Curran Associates, Inc., 2020.
[14] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 36652-36663, Curran Associates, Inc., 2023.
[15] Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Mare Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic for- getting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 16220.16220. Curran Associates, Inc., 2022.
[16] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Warkins, Yuging Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback, 2023. URL https://arxxi: org/ abs/ wtw.////222..... 12192
[17] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing diffusion models by optimizing human utility. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances Neural Information Process- ing Systems (NeurIPS), volume 37, pages 24897-24925. Curran Associates, Inc., 2024.
[18] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generie preferences with step-by-step preference optimization, 2024. URL httpps://aat :tatpii///rxiv.... org/ htse//b///20 abs/2406 04314
[19] Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J Liu, and Xuanhui Wang. LiPO: Listwise preference optimization through learning-to-rn to-rnk. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pages 2404-2420, Albuquerque. New Mexico, April 2025, Association for Computational I Linguistics. ISBN 979-8-891 76-88997766111 76-119-66666666 189-..
doi: 10.1865. 18.58//5/ 12225.naacl 121.
URL
wrg.///20..r//2555../5./3 http://///...201110rrrro.oo naacl- 121/
[20] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. 2019.
[21] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, 2023. URL https:t//txxiii htgg/ab//2211...
[22] Sanghyeon Na Yonggyu Kim, and Hyunjoon Lee. Boost your human image generation model via direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
[23] Andrew Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Interna- tional Conference on Learning Representations (ICLR), volume 1, page 2, 2000.
11

---

[24] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems, volume 37, pages 116617-116637, Curran Associates, Inc., 2024. http:httts://prrceeediii neurips cc/ wwwpappppe.aeeeeeee facaper_fillee fil.s/ill//peee httpeeeiiileee/ppp//2///////// http//ppppr////////eee 337c6c2222ee555555506ccdda0o--pp
pdf
[25] Xue Bin Peng. Aviral Kumar, Grace Zhang. and Sergey Levine. weightt Simple and scalable off-policy reinforcement learning, 2019. URL https:://axxi// org/
[26] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning (ICML), pages 745-750, 2007.
[27] Dustin Podell. Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations (ICLR), 2024.
[28] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation, 2024. WtRllhttps://arxiitt org/ abs/
1310.037330
[29]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 4768-.76. PMLR, 18-24 Jul 2021.
[30] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems
(N (NeurIPS), volume 36, pages 13728-53774 Curran Associates, Inc., 2023. URL /tL/ttpps:/prrreeednng www.rpe.euure...ee.. wewurippscc/ppppp..e http://aper_fileeppaappp PooeePeaeeroooeeeeeeeeeee pdf
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-106999 June 2022.
[32] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Den- ton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. editors, Advances in Neural Informa- tion Processing Systems (NeurIPS), volume 35, pages 36479-36494. Curran Associates, Inc., 2022. URL wtt.htttss:/proccediiie pttpp//eeeeeaeeeeeeeeepee.pppppp wwtpwpppppppppppppptppcteepe. wwt..e/se/pppp///2///////// 4e775aacca00222233355000000425e0
[33] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024, pages 87-103. Springer Nature Switzerland, 2025. ISBN 978-3-031-73016-0.
[34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaezmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 25278-25294. Curran Associates, Inc., 2022.
12

---

[35] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition (CVPR), pages 8228 8238, June 2024.
[36] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023. URL httpst::: / arxii. org/ abs/ /2066 //0//.0300000011111
[37] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. In International Conference on Learning Representatinss (ICLR), 2025.
[38] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for pext-to-image generation. In A. Oh, Naumann, A. Globerson, Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 15903-15935. Curran Associates, Inc., 2023.
[39] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
pages
[40] Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to- image diffusion with preference. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 15998-56033 PMLR, 21-27 Jul 2024.
[41] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. In The Thiriy-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024. htlttttpps//ppppeviieeee net/tferuuue formmrormmmmmm wete/forum2.dggXXavkPP....
[42] Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, and Dacheng Tao. Confronting reward overoptimization for diffusion models: A perspective of inductive and primacy biases. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning (ICML). volume 235 of Proceedings of Machine Learning Research, pages 60396-60415 60413. PMLR, 21-27 Jul 2024.
13

---

A Experiment Details
A.1 Datasets
We use the Pick-a-Pic v2 [14 training set as the source of expert demonstrations, consisting of 959,040 preference pairs and 1,029,802 unique text-image pairs. Note that the train of Diffusion- SPO [8] is Pick-a-Pic v1, which is a little different from our settings, but in a comparable range. Our train is consistent with other baseline methods, i.e., Diffusion-DPO and Diffusion-KTTP [1]]. For testing, we utilize two datasets to ensure diverse evaluation scenarios. The first is the Pick-a-Pic v2 test set, which includes 500 unique text prompts collected from users of the deployed web application. The second is the HPSv2 Benchmark, divided into four categories: anime, concept art, painting, and photo. Each category contains 800 text prompts. However, we observe slight discrepancies in the number of unique prompts: 781 1 for anime. 795 for concept art, 798 for painting, and 800 for photo. To maintain consistency with prior works [7 g we use all 800 prompts (including duplicates) for each category during testing. To select the expert demonstrations from Pick-a-Pic v2, we use a preference metric to give each text-image pair a score representing the quality of being expert demonstration. We then sort all pairs in descending order by the scores and select the top K pairs as the expert demonstrations. If not otherwise specified, K 500 is used.
35
(Apache-2.0 license)
A.2 Implementation Details
We fine-tune Stable Diffusion 1.5 (SD v1-5) 31 using Diffusion-DRO. The AdamW optimizer (20] used with a learning rate of 10-4 and an effective batch size of 256 (4 samples per GPU, 32 gradient accumulation steps, yielding 4 x 4 16 256). The training consists of 1, 1,600 optimization steps, resulting in a total of 16 x 1, 600 = 25, 600 iterations when accounting for gradient accumulation. During training, 20% of prompts
randomly replaced with empty strings, which helps preserve the model's ss ability to perform unconditional generation by maintaining a balance between conditional and unconditional sampling.
Following the standard Stable Diffusion training process, we apply an exponential moving average (EMA) to aggregate the UNet weights during training, with decay rate of 0.9999. The clipping threshold m for the thresholded ranking loss (TRL) is set to -0.001, and the policy model update interval M is set to for all experiments.
with 20 steps, without utilizing classifier-free guidance [12]. To ensure all time steps in SD v1-5 are adequately fine-tuned, we uniformly perturb the sampling time steps of DPMSolver++ during training. This approach allows the time steps used during inference to differ from those used for online sampling in training. enhancing the model's robustness across all time steps.
For sampling I, from the policy model, we employ DPMSolver++
All experiments, including the reproduction of baseline methods with updated SD model weights, were conducted on four NVIDIA RTX 3090 GPUs. Training Diffusion-DRR -DOO takes approximately 20 to 25 hours.
Table 2: Automated win rates between Diffusion-DBi and baseline methods. PickScore is used to select expert demonstrations, The dagger symbol (1) indicates that the evaluation was performed on the officially released model weights. Note that SDD v1-5 w/ SFT refers to SD v1-5 fine-tuned on
expert demonstrations, Win rates greater than 50% are highlighted in bold.
Pick-a-Pic v2 Test
HPDv2 Benchmark
Baseline Method
Aesthetic Score
CLIP Score
Aesthetic Score
CLIP ImageReward
HPSv2
ImageReward
PickScore
Score
SD v1-5
94.80
79.00
58,60
87.40
97.00
79.44
51.06
88.59
SD v1-5 w/ SFT
71.40
54.40
68.80
60.60
70.34
55.84
69.06
61.53
SPIN-Diffusion
78.20
57.60
65.80
72.40
80.56
59.81
61.78
72.59
Diffusion-SPO' 84.40
57.60
76.80
78.80
85.78
65.06
79.19
79.16
Diffusion Miflusion-DPO!
92.20
79.00
51.80
83.00
94.31
77.28
44.81
84.00
Diffusion- D 1
77.40
68.60
53.60
66.20
74.25
66.22
46.91
65.22
14

---

Table 3: Preference scores of Diffusion-DRO and baseline methods evaluated the Pick-a-Pic v2 test set. The metric used to select expert demonstrations for Diffusion-DRe is indicated in parentheses after "Diffusion-DRO", e.g.. "Diffusion- DRO (PickScore)." Moreover, baseline methods with the suffix "w/ SFT" are fine-tuned from "SD v1-5 w/ SFT", which itself is fine-tuned from SD v1-5 using expert demonstrations selected by HPSv2.
Method
PickScore
HPSv2
Aesthetic
CLIP Score
ImageReward
SD v1-51 SD v1-5 w/ SFT
20.68
26.8811181
5.9311.05
0.33691.0SS
0.176511.00
21.24.11.41
28.11111.73
6.341.974
0.3293.1.059
0.76231.906
SPIN-Diffusion
11.40:11.38
27.78.11.79
6.26.1.985
0.33051..059
0.56191.971
Diffusion-SPO
21.17+1.441
2..3
6.23+.981
0.3150+
0.3278 1.07
Diffusion-SPO w/ SFT
20.75:11.37
26.93.11.75
5.93111.07
0.34191.050
0.2254
Diffusion-DPO
21.00++.440
27.22++1.80
5.96+1.00
0.3427+.057
0.3369-1.00
Diffusion-DPO w/ SFT
21.31:11.40
28.08.1.722
6.351.995
0.33341.08 .59
0.7912
Diffusion-KTTO
21.17++1.36
27.88+11777
6.20+.954
0.3438+
0.6743+.96
Diffusion-Ko TO w/ SFT
21.25++141
28.11+1.774
6.34+.974
0.3296+.009
0.7636-9910
Diffusion-DRo (IIPSv2)
21.52.11.42
28.4911..75
6.40119766
0.33901.057
0.8511 -852
Diffusion-DRi (PickScore)
21.76:11.51
28.38
6.40.1.935
0.34461.005
0.8636
Table 4: Preference scores of Diffusion-DRO and baseline methods evaluated on HPDv2 Benchmark. The score name that is used to select the expert demonstrations for Diffusion-DRO are denoted in the parentheses after "Diffusion-DRO". e.g., "Diffusion-DRR (PickScore)." Moreover, the bascline methods with suffix "w/ SFT" are fine-tuned from "SD v1-5 w/ SFT", which is also a fine-tuned from SD v1.5 with expert demonstrations selected by HPSv2.
Method
PickScore
HPSv2
Aesthetic
CLIP Score
ImageReward
SD v1-5 SD v1-5 w/ SFT SPIN-Diffusion
10.92=:1.20
27.36
6.22+.923
0.3532+.052
0.2242
1..55=1.32
28.74++.61
6.65
0.3415#.053
0.7554=
21.74+1.21
28.3841.64
6.541.843
0.3451#.054
0.6071
Diffusion-SPP
11.63=1.25
18.01+1.54
6.47
0.3217#.054
0.4141 .968
Diffusion-SPo w/ SFT
20.99+1.19
2..39
6.26:t.933
0.35561.051
0.2619
Diffusion-DPO
21.30.11.19
2..75.11.67
6.291.920
0.35841.051
0.4070
Diffusion-DDB w/ SFT
21.66=11227
28.77+1.559
6.64#.826
0.3454+.052
0.8001 861
Diffusion-KTO
21.5111.18
28.5711.61
6.471.852
0.35791.052
0.7529
Diffusion-KIIP w/ SFT
21.56+1.322
28.74+1.61
6.64
0.3416#.003
0.7557=8700
Diffusion-DRR (HPSv2)
21.82+1.229
29.05=1.66
6.70
0.3521+.053
0.8853 843
Diffusion- -DO (PickScore)
22.04:11.28
28.99::162
8.6
0.3560#.053
0.9069-833
B Additional Quantitative Results
To alleviate the variation of evaluation results, sample 5 images per prompt for all models in our benchmark. Specifically, we sample 2500 images for Pick-a-Pic v2 test and 16000 images for HPDv2 Benchmark. When calculating the win rates, sort 5 images for each prompt according to the corresponding PickScore and select the image with medium score as the comparison target.
We report the win rates of Diffusion- Diffusioon-ROO trained with HPSv2 selected expert demonstrations in Table I For the Diffusion-DRO trained with PickScore selected expert demonstrations. The win rates against baseline methods are show in Table Due to the limited computation resources, we do not reproduce the bascline methods based on the new SD model (SD v1-5 w/SFT in Table We only compare the Diffusion-DRRO with the officially released model weights.
We present the average preference scores in Table and Table including PickScore 14]
HPSv2 (6). Aesthetic 34 CLIP Score 229, and ImageReward 38
C User Study Settings
As shown in Table the automated win rates of our method are decreased after Diffusion-DDD and Diffusion KTO using the new SD model base weights, e.g., the win rate compared to Diffusson-
15

---

DPO and Diffusion-DPO w/ SFT on HPDvv2 Benchmark decreases from 79.75 to 63.62 evaluated by PickScore. This shows that Diffusion-DPO w/ SFT and Diffusion-KTTT w/ SFT are more competitive than the official released models. Therefore, we choose these two baseline models plus an SD v1-5 to be baseline methods in user studies. We use the same images generated for calculating metrics in Table four categoryies of HPDv2 Benchmark (15 prompts for each category). For each category, we use PickScore to sort the samples for each method and select the image with medium score the survey target. To avoid survey participants identifying our generation results, we re-sample the prompts for each user study between Diffusion- DRRO and baseline methods. This could prevent samples from repeated occurences in different user studies.
[ Table + and Table T
We prepare the prompts by random sampling 60 prompts from
We employ human evaluators via Amazon Mechanical Turk (MTurk) for our user studies. Although the HPDv2 Benchmark includes additional filtering steps to remove inappropriate prompts, we still indicate that the user survey may contain adult content. Before beginning the survey, users must check the box labeled "WARNING: This HIT may contain adult content. Worker discretion is advised."
On the survey page, participants can access the evaluation guidelines, which include the following instructions:
For each text prompt, Al-generated images will be displayed side by side. You evaluate which image better meets human expectations based on (but not limited to) the following criteria. The importance of each criterion depends your subjective judgment:
Completeness of details
Artistic or aesthetic quality
Alignment between the image and the given prompt
In short, select the image that you believe demonstrates better generation quality.
On each selection page, the prompt is displayed along with two images labeled Image A and Image B, accompanied by the question: "Which image do you prefer given the prompt?" Below the question, two radio buttons allow users to select either Image A or Image B, with at least one selection required before submission. To ensure fairness, the images generated by Diffusion-DRi and the baseline methods are randomly assigned to Image A and Image B. Additionally, their sources cannot be identified through the webpage's source code.
For each prompt, we collect 35 responses. the majority of these responses favor Diffusion-DRO the prompt is considered to prefer Diffusion- DRO. Finally, we compute the proportion of prompts that favor our method as the win rate, which is reported in Figure [
D Derivation of Denoising Ranking Optimization
For convenience, we repeat Eq. below:
Pe Po (To.T
Po po(ooo.le
log
Ee ao (e)
log
po (No:T
pa. (mo.r
(16)
The first term on the left-hand side (LHS) and second term the right-hand side (RHS) share similar simplification processes. We first present the derivation of the LHS:
Pe (o (c)
3 log
(17)
pa (Bo.T
Pol Po D1, c)
+ 3 log Po(xr/c) Pe (MT(C)
log
+
(18)
Pen (T,-1 (c,
T
log Pes Pa( Fe, c)
B &.) vug - (0)
+
(19)
Po Br Rt, C)
16

---

T
log Pol
B c)
3
E
q (t, [t;; 10)
log g( [ Tt)
+C
(20)
pa ( (t,, c)
T
3
E Ee P.) - Dri. [g( [e., 2o) pa ( 12, c)
+ DRi, (g((0== [2,, 20) pa (2 (8r
+0
(21)
T
1
8
E Pe- -1le-eolan =)]
20 04 (1 (4)
+ ||e (ar c, c
+C
(22)
X Dlo -|2----- c,) +
cBnl - (2,,
+C.
(23)
All diffusion hyperparameter notations, i.e., dr. Oc,, de and 3r, follow the definitions from DDPM I3
Here, 33 represents the KL regularization weight as defined Eq. ① and a constant independent of Φb. We then derive the RHS:
Pe (o(oor))))
3 log
(24)
per (xo:rlc)
T
B log Po (x1-111 (0
xc: c) pa (x)
Pe (xT((c)
-E Go:r sp
+ 8 log
+
(25)
[2, c)
pa (xT
T
PΦ (2 I c)
B
E pe
log
+0
(26)
pa @1 [,,, C)
Ps [1 D2, c)
+ log p -2,,
E me
log
+C
po (0,- Dr, c)
pa pa Dt, C)
(27)
DxL. po [ol pplk
8
E C C.
DKK po 1 120 c) p o0) +0 (28)
T
8
=3
Bo Be
- co( c, c,,
20 aq (1
āi)
+ |e c,t) c (x,e, (,m,,,,,)) c.1)11 +C (29)
T
X XBBoCCC.--pote.ll)
-I c,t) calor, t)/l
||co(2), c,t) (x,, (x,, c)) +c. (30)
17

---

Substituting Eqs.
23
and
30
into Eq.
[6 we obtain:
d DDo. -||---ea Es - c,) 1|e ca ( t)19
A,Be-C
-1 e,t) cs(ar, c,( (es(o,, c,t) ca. (2e, c,)
(31)
Following the DDPM settings, we set d obtain our final result.
E
Ethies
The Pick-a-Pic v2 dataset has been identified containing NSFW prompts, it is collected from publicly available user inputs on the internet. To minimize exposure to violent, adult, or otherwise inappropriate content, chose HPDv2 the prompt source for user studies. Participants are also informed this facts before they start the study.
For a fair comparison with previous methods, we continue to use Pick-a-Pic v2 as part of the training data. Given the strong performance of Prffiusion-RROO there is a potential risk that the model could generate NSFW content. However, Diffusion-DRO does not explicitly learn to produce NSFW images; its outputs are inherently dependent on the training dataset.
To mitigate this risk in future applications, NSFW content can be filtered at the data level by curating human preference datasets that exclude inappropriate content, thereby preventing Diffusion-Dee from learning to generate such images. Before publicly releasing our model, we will ensure the implementation of additional safety filter to prevent misuse.
F
Limitations
Despite the significant improvements Diffusion-DRO brings to aligning diffusion models with human preferences, it remains constrained by the data-dependent nature of diffusion models. Specifically, the approach relies expert demonstrations extracted from data, which may introduce distributional biases-for example, simpler prompts tend to yield better outputs and are thus more likely to be selected expert data. Iiffusion-DRO does not explicitly account for such biases and disregards non-expert demonstrations, which may result a model that performs well only in limited domains.
G Future Work
While Diffusion-DRO introduces the concept of expert demonstrations from inverse reinforcement learning perspective, future work could extend this framework beyond the current max-margin formu- lation by incorporating non-expert data enhance performance underrepresented sparse regions of the data distribution. Furthermore, our current approach treats preferences as binary rankings (i.c., preferred vs. not preferred), which results the loss of list-wise ranking information [19]. We believe that integrating such richer preference structures into the inverse reinforcement learning framework could further refine the granularity and stability of the optimization process.
18

---

H
Additional Samples
In this section, we provide additional sample comparisons with baseline methods, including Diffusion- DPO w/ SFT, Diffusion KTO w/ SFT, Diffusion-SPO,, and SD v1-5.
Diffusion-DRe
SPIN-Diffusic
Diffusion- DPO Diffusion- -anssoon-KTOO
Diffusion-
SD vl-5
The prompts used for image generation are sourced from the HPDv2 Benchmark, cate- gorized as Anime, Concept Art, Painting, and Photo from top bottom, respectively. The specific prompts, in order, are: "A portrait of smiling Dragonite in sunflower field with cloudy sky backdrop," "Amphihheater filled with crowd looking at a dumpster on fire in patriotic colors," " "Beige canvas tents set up in arctic landscape with no vegetation, surrounded by rolling hills reminiscent of romanticist puinting," and "A small bird sitting in metal wheel."
Figure 4:
19

---

Diffusion-DRO
SPIN-Diffusin
Diffusion-DIi
Diffusion-KTO
Diffusion-SID
SD vl-5
Figure 5: The prompts used for image generation are sourced from the HPDv2 Benchmark, cate- gorized Anime, Concept Art, Painting, and Photo from top bottom, respectively. The specific prompts, in order, are: "A monkey wearing a jacket," "Portrait of cyberpunk gang." "Bob Ross painting Mario easel in his office," and "A little girl holding brown stuffed animal."
20

---

Diffusion-DRe
OPIN-Diffusiaa
Diffusion-DDD
Diffusion- KTO
Diffusion-SSe
SD v1-5
Figure 6: The prompts used for image generation are sourced from the HPDv2 Benchmark, cate- gorized Anime, Concept Art, Painting, and Photo from top bottom, respectively. The specitic prompts, in order, are: "A new artwork depicting Pikachu as superhero fighting villains with dramatic lightnnng." "A futuristic cyberpunk Paris street," " "A young girl with red hat night," and "A bike parked in front of doorway."
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction are supported by theoretical derivations in Section and by empirical results, including quantitative evaluations and user studies, in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of Diffusion-DRO in Appendix including its scope of applicability and potential weaknesses in specific scenarios.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification:
We provide step-by-step derivation our theoretical results in Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the main experimental settings in Section detailed configurations in Appendix [ and include the source code in the supplementary material to ensure reproducibility of the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include the source code for training and evaluation in the supplementary material, along with instructions reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ips.cccppbllic wwtppbplcggguiiieecceeeeeoooeeee http:/uddese/eee/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The main experimental settings are provided in Section with full de- tails iilss-nnluding data splits, hyperparameters, and optimizer configurations o in Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the standard deviation of scores for all evaluated models Ap- pendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the training time of methods and the computing resources used in Appendix
A.22 A.2
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips..com http://eeerrr....ccc/bbll/eeeeee ctseeieeeeeicsscceeeeeeeeee
Answer: [Yes]
Justification: We follow the NeurlPS Code of ethics in conducting user study, with relevant details provided in Appendix(C
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
A discussion of socictal impacts is included in Appendix D as part of our ethics
review.
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: As described Appendix we will include a safety checker with the released model to help mitigate potential misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include the licenses for all publicly available assets used in our work in Section 4 along with proper attribution their original creators.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papersaiphheeeoocccpppapp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The released source code is accompanied by documentation that describes its usage and functionality.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Appendix C provides the full text of participant instructions.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: We did obtain IRB approval. However, we clearly disclosed the potential risks associated with participating in the user study, detailed in Appendix C
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not use any LLMs as part of the core methodology in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
28


Reviews:
[{"review_id": "cjSAoE6ZsR", "reviewer": "Reviewer_zy7V", "summary": "This paper introduces Diffusion-DRO, a preference learning framework for text-to-image diffusion models. The method derives a new training objective that is an linear estimation of the denoising losses and it enables fine-tuning diffusion models using only expert demonstrations with online generated examples. Empirically, the method outperforms prior methods by a large margin in terms of win-rates in across various automated metrics.", "questions": "1. What is the intuition behind Eq. 7? In line 122, the paper states that \"the reward model aims to maximize the margin between the expert and the policy,\" but this seems inconsistent. If the expert trajectory (left-hand side) is fixed and the reward model is optimized using samples from the policy (right-hand side), wouldn’t this minimize the margin? Could the authors clarify this apparent contradiction?\n2. Eq. 9 introduces a new parameterization of the trajectory reward model. What is the intuition behind this formulation? In prior work, similar parameterizations are usually derived from a definition of $R$ and training objective.\n3. In line 8 of Algorithm 1, how is the sample $x_t \\sim p_\\theta$ efficiently obtained? Does this require performing the full denoising process from step $T$ to $t$ ? If so, how is this efficient during training?\n\nI lean toward rejection primarily due to theoretical opacity and limited evaluation scope, though the core idea is novel and the results are promising. I could raise my score with improved clarity and broader experiments.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The method is novel as it replaces the standard non-linear loss with a linear, denoising-based objective for preference optimization in diffusion models, offering a simpler and more interpretable alternative to DPO-style methods.\n2. The paper is well written and technically sound, with a clear setup and most part of the derivations are rigorous. \n3. The empirical results are strong and show consistent and significant improvements over prior work\n\n## Weaknesses\n1. Parts of sections 3.2 and 3.3 are redundant and dense, with limited intuition provided. Though the derivations are complete, it is difficult to follow how the inverse reinforcement learning formulation leads to the final denoising loss (Eq. 7–12). The connection between margin-based objectives and reward-based objective could be better explained.\n2. The experiments are limited to SD 1.5, while it is now standard to also evaluate on larger models like SDXL, as done in Diffusion-DPO/SPO. This is important as gains may diminish on stronger base models.", "was_revised": true, "final_justification": "The author provides more empirical results showing the effectiveness of their method, which could have been a plus. However, the derivation does not come naturally, and the intuition is not well explained; this is a minus. I believe the latter is important to NeurIPS; thus, I lean toward rejection.", "rebuttal": "We sincerely appreciate your thoughtful review. Below, we address each of your questions in detail. Lastly, we provide clarifications for several concerns mentioned in the weaknesses section that were not explicitly formulated as questions.\n\n## Reply to Questions\n\n---\n\n- ### Q1\n    What is the intuition behind Eq. 7? In line 122, the paper states that \"the reward model aims to maximize the margin between the expert and the policy,\" but this seems inconsistent. If the expert trajectory (left-hand side) is fixed and the reward model is optimized using samples from the policy (right-hand side), wouldn’t this minimize the margin? Could the authors clarify this apparent contradiction?\n    ###\n    ### A1\n    Thank you for the insightful question. We would like to clarify the underlying optimization dynamics. Eq. 7 is given as:\n    $$\n    \\mathbb{E}\\_{\\mathbf{c}\\sim\\mathcal{C},\\bar{\\mathbf{x}}\\_0\\sim\\mathcal{D}(\\mathbf{c})}\\Big[r\\_{\\phi}(\\bar{\\mathbf{x}}\\_0,\\mathbf{c})\\Big]\\ge\\mathbb{E}\\_{\\mathbf{c}\\sim\\mathcal{C},\\mathbf{x}\\_0\\sim p\\_{\\mathbf{\\theta}}(\\mathbf{x}\\_0|\\mathbf{c})}\\Big[r\\_{\\phi}(\\mathbf{x}\\_0,\\mathbf{c})\\Big].\n    $$\n\n    The reward model $r\\_\\phi$ is trained to assign higher scores to expert demonstrations than to policy samples, thereby maximizing the margin. In contrast, the policy (updated via Eq. 10) implicitly attempts to **close the gap** by generating higher-reward samples. This creates a minimax dynamic similar to IRL.\n\n    In short:\n    - $r\\_\\phi$ maximizes the margin: $r(\\bar{x}\\_0) > r(x\\_0)$,\n    - $p\\_\\theta$ reduces the margin by improving sample quality,\n    - The contradiction is resolved by understanding the alternating optimization roles.\n\n    We will revise the text to make this dynamic more explicit.\n\n    ---\n\n- ### Q2\n    Eq. 9 introduces a new parameterization of the trajectory reward model. What is the intuition behind this formulation? In prior work, similar parameterizations are usually derived from a definition of and training objective.\n    ###\n    ### A2\n    We adopt the parameterization in Eq. 9 following the approach used in DPO [A]. As shown in Theorem 1 of DPO, this formulation can represent any reward function (up to an additive constant), ensuring no loss of generality. Importantly, our Eq. 9 mirrors the structure of Eq. 3, enabling a smooth derivation of the closed-form policy in Eq. 10. This consistency simplifies the optimization pipeline by making the reward model and policy model interchangeable under KL-regularized objectives. We believe this formulation balances theoretical soundness and practical convenience, facilitating both analysis and implementation.\n\n\n    [A] \"Direct preference optimization: Your language model is secretly a reward model.\" NeurIPS 2023.\n\n    ---\n\n- ### Q3\n    In line 8 of Algorithm 1, how is the sample $x\\_t \\sim p\\_\\theta$ efficiently obtained? Does this require performing the full denoising process from step T to t? If so, how is this efficient during training?\n    ###\n    ### A3\n    In our implementation, we explored two strategies to accelerate the sampling of $x\\_t \\sim p\\_\\theta$.\n\n    We first experimented with maintaining a replay buffer that stores intermediate samples $x\\_t$ for reuse. In each iteration, only a small portion of the buffer is updated, and full training batches are sampled from the buffer. However, we observed that when the policy is updated every step (i.e., $M = 1$ in line 15 of Algorithm 1), this strategy led to suboptimal performance because the buffer tends to accumulate outdated samples and negatively affects training. As a result, we did not adopt the replay buffer approach in our final implementation.\n\n    Afterward, we considered using more efficient samplers, such as DDIM [B] and DPM-Solver [C]. As mentioned in Section 4 of the paper, we ultimately adopted DPM-Solver++ for faster sampling. Below, we present a comparison between training with the DDPM sampler and DPM-Solver++:\n\n    |Sampler|PickScore|Aesthetic|CLIP|ImageReward|Training Time|\n    |-------|:-------:|:-------:|:--:|:---------:|:-----------:|\n    |DDPM(steps=50)       |21.50|6.48|0.3259|0.7501|46.264 hours|\n    |DPMSolver++(steps=20)|21.54|6.42|0.3390|0.8452|25.708 hours|\n\n    As shown in the table, DPM-Solver++ reduces training time by 44 percent while slightly improving performance across several metrics. We note that this setup was adopted without extensive hyperparameter tuning, suggesting that there is still considerable room for further improving training efficiency.\n\n    [B] \"Denoising diffusion implicit models.\" ICLR 2021.\n\n    [C] \"DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps.\" NeurIPS 2022.\n\n## Clarification of Weakness\n\n---\n\n- ### Weakness 1\n    The experiments are limited to SD 1.5, while it is now standard to also evaluate on larger models like SDXL, as done in Diffusion-DPO/SPO. This is important as gains may diminish on stronger base models.\n    ###\n    ### Reply to Weakness 1.\n    Thank you for pointing out this important limitation. In response to a suggestion from reviewer `NjRJ`, we extended our experiments to a larger model and dataset, specifically SDXL and MJHQ-30k [D]. We also adopted the GenEval benchmark [E], as proposed by NjRJ, to more comprehensively assess model performance.\n\n    We ranked MJHQ-30k samples using PickScore and selected the top 500 image-text pairs as expert demonstrations. Both `SDXL + SFT` and `Diffusion-DRO` were fine-tuned on this dataset. For reference, we also report results from the original `SDXL` model and `Diffusion-DPO`, the latter of which was fine-tuned on Pick-a-Pic v2. Following the GenEval protocol, each prompt was used to generate four images to ensure reliable evaluation.\n\n    | Method        |   Colors  | Color Attr. | Single Object | Two Object |  Counting |  Position |  Overall  |\n    | ------------- | :-------: | :---------: | :-----------: | :--------: | :-------: | :-------: | :-------: |\n    | SDXL          |   87.23   |    23.00    |     99.69     |    78.03   |   43.75   |   11.75   |   57.24   |\n    | Diffusion-DPO |   86.17   |    25.00    |     98.75     |    85.25   |   48.25   | **13.60** |   59.50   |\n    | SDXL + SFT    |   88.94   |  **27.00**  |     99.75     |    87.68   |   46.50   |   13.20   |   60.51   |\n    | Diffusion-DRO | **90.43** |  **27.00**  |   **100.00**  |  **87.88** | **50.00** |   12.00   | **61.22** |\n\n    This demonstrates the method’s strong generalization ability to both a larger backbone (SDXL) and a higher-quality dataset (MJHQ-30k). Notably, even when fine-tuning on expert demonstrations in the form of image-text pairs alone, without explicit preference labels, Diffusion-DRO is able to outperform SFT. For example, it achieves a 1.49 percent gain in the \"Colors\" category and a 3.5 percent improvement in \"Counting.\"\n\n    We hope this additional experiment directly addresses your concern and further demonstrates the scalability and effectiveness of our method in more advanced settings.\n\n    [D] \"Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation\", arXiv.\n    \n    [E] \"GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.\" NeurIPS 2023."}, {"review_id": "i2E3DrpINd", "reviewer": "Reviewer_NjRJ", "summary": "The paper presents a DPO style preference optimization framework for diffusion models. Different from standard Diffusion-DPO which requires paired preference data, the proposed framework instead the proposed method requires only positive samples for the task. This is achieved by having a margin-based ranking loss over \"expert\"/highly-preferred samples compared to the samples generated by the policy model for the same prompt. Comparisons on automated scoring metrics on standard prompt benchmarks indicate that the proposed method outperforms related preference optimization methods (e.g. Diffusion-DPO/KTO etc.) on the SD1.5 model.", "questions": "While the paper does an excellent job of distinguishing the proposed method from Diffusion-DPO, it might also be a good idea to explicitly have a conceptual comparison with SPIN-Diffusion (at least in the appendix) since it seems closely related as a method. \n\nLines 205-208 mention that there are 5 generations per prompt and the image with the median Pickscore value is selected. This seems like an interesting choice for evaluation: while seeds have a huge role to play, wouldn't it be a better idea to simply average the results over all generations (and then report std dev)?\n\nLines 70-73 mentions that while KTO also decouples the positives and negatives, this could introduce semantic biases. However, even after reading the paper, it's not entirely clear why this cannot happen at all for the proposed method either. Could the authors clarify a bit more about this aspect? \n\nThe choice of the expert demonstrations is also quite interesting: while Fig. 3 provides an analysis of the number of samples from the Pick-a-Pic dataset, I'm also curious about would having higher-quality images (e.g. the MJHQ-30k images) or captions (e.g. through recaptioning) as a tool to improve the results of the proposed method. \n\nThe paper might also want to acknowledge other works on ranking based DPO e.g. LiPO[a]\n\n[a] Liu et al. \"LiPO: Listwise Preference Optimization through Learning-to-Rank\", NAACL 2025", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe framework presented in the paper is quite interesting (i.e comparing expert demonstrations vs policy generations) and to the best of my knowledge has not been explored before, making it a valuable contribution.\n\nFurther, the results when compared to prior preference optimization methods (e.g. Diffusion-DPO/KTO, SPIN-Diffusion) appear quite strong making the overall methodology very compelling.  \n\nWeaknesses:\n\nOne huge drawback of the paper is that it solely relies on the SD1.5 model to make the case for the proposed method. In the last 3 years, there have been several open-source models which have since then provided superior performance (e.g. SDXL, SD3, Flux, SANA etc.). Even preference optimization methods (e.g. Diffusion-DPO etc.) have been applied to some of these methods with public checkpoints (e.g. SDXL with Diffusion-DPO), therefore it might be a good idea to investigate the quality of the proposed method on newer models. \n\nIn terms of the evaluation, it mostly relies on win-rates with reward model metrics (e.g. PickScore, ImageReward etc.). As the paper itself mentions (lines 214-215), this could potentially lead to overfitting to a single type of objective. While evaluations on different reward models and the human study is able to address the issue to some extent, different types of evaluations (e.g. object detector based GenEval or VQAScore based GenAI-bench) might also be a way to have more robust comparison. \n\n\nOverall, I would be inclined to accept the paper, however, I would love to see more robust evaluations along with results on other models before confidently recommending acceptance.", "was_revised": true, "final_justification": "The authors have provided experiments further strengthening the claims of the paper in the rebuttal (generalizing to the larger SDXL model, different sources of expert demonstrations, and evaluations beyond pure reward models). Based on these, I believe that the paper meets the bar for acceptance.", "rebuttal": "We sincerely thank you for your review. Below, we address each of your questions in detail. Several of these questions also overlap with the concerns raised in the weaknesses section, which we address jointly in our responses.\n\n## Reply to Questions\n\n---\n\n- ### Q1\n    While the paper does an excellent job of distinguishing the proposed method from Diffusion-DPO, it might also be a good idea to explicitly have a conceptual comparison with SPIN-Diffusion (at least in the appendix) since it seems closely related as a method.\n    ###\n    ### A1\n    We sincerely appreciate the reviewer’s suggestion. While our main comparison focuses on Diffusion-DPO, we agree that SPIN-Diffusion shares conceptual similarities and merits explicit discussion.\n\n    The key distinction lies in the formulation and training paradigm:\n\n    - **Our method** is grounded in a *max-margin inverse reinforcement learning* (IRL) objective (Eq. 7), which directly compares expert demonstrations against policy samples. This formulation leads to a closed-form expression (Eq. 10), where the optimal policy coincides with the reward model. As a result, the min-max optimization reduces to a *single-stage* minimization over the reward model.\n\n    - **SPIN-Diffusion**, by contrast, adopts a *discriminative training* approach. It frames reward modeling as a binary classification task, i.e., distinguishing between expert and generated samples, and relies on alternating updates of the reward and policy models. This requires *multi-stage* training and careful synchronization to ensure stable optimization.\n\n    We believe this conceptual distinction, i.e., *IRL-based ranking vs. discriminative classification*, captures the core difference between the two approaches. We will include a discussion of this comparison in the appendix to help clarify the conceptual differences more explicitly.\n\n    ---\n\n- ### Q2\n    Lines 205-208 mention that there are 5 generations per prompt and the image with the median Pickscore value is selected. This seems like an interesting choice for evaluation: while seeds have a huge role to play, wouldn't it be a better idea to simply average the results over all generations (and then report std dev)?\n    ###\n    ### A2\n    We follow the evaluation protocol adopted by Diffusion-KTO [A], which selects the image with the median Pickscore for pairwise comparisons in automated win rate evaluation (Appendix A.2 of [A]). This also allows us to use the selected images consistently in both automated evaluation and user studies. As a result, the automated win rates can be directly compared to user study results, which improves their reliability.\n\n    It is worth noting that we have provided the mean and standard deviation of preference scores over all generated samples in Table 3 and Table 4 of Appendix C to address the concern about variability across generations. These results show that the propopsed Diffusion-DRO achieves the highest average preference scores across most metrics, with the only exception being a slight gap in CLIP score on the HPSv2 dataset compared to baseline methods.\n\n    [A] \"Aligning Diffusion Models by Optimizing Human Utility.\" NeurIPS 2024.\n\n    ---\n\n- ### Q3\n    Lines 70-73 mentions that while KTO also decouples the positives and negatives, this could introduce semantic biases. However, even after reading the paper, it's not entirely clear why this cannot happen at all for the proposed method either. Could the authors clarify a bit more about this aspect?\n    ###\n    ### A3\n    The concern about semantic bias is valid and worth clarifying further. In **Diffusion-KTO**, positive and negative samples are explicitly decoupled based on a fixed dataset. If the negative set disproportionately contains certain semantic categories (e.g., images of \"cats\"), the model may inadvertently learn to avoid generating those concepts, even if they are not inherently undesirable. This fixed split makes the model vulnerable to dataset-induced semantic skew.\n\n    In contrast, **our method generates negative samples online** from the evolving policy. This has two advantages: (1) The negative samples naturally reflect the current distribution of the policy, which tends to be more diverse and balanced than a manually curated negative set. (2) The dynamic nature of online sampling reduces the chance of reinforcing static biases, since what constitutes a \"negative\" evolves over training.\n\n    While no method can entirely eliminate semantic bias, our approach is specifically designed to **reduce its impact** by avoiding hard-coded negative sets. We will clarify this point in the revised version of the paper.\n\n    ---\n\n- ### Q4 + Weakness 1 + Weakness 2\n    The choice of the expert demonstrations is also quite interesting: while Fig. 3 provides an analysis of the number of samples from the Pick-a-Pic dataset, I'm also curious about would having higher-quality images (e.g. the MJHQ-30k images) or captions (e.g. through recaptioning) as a tool to improve the results of the proposed method.\n    ###\n    ### A4\n    We sincerely thank the reviewer for the thoughtful suggestions and for pointing us to valuable resources. Following your comments in both this question and the weaknesses section, we conducted additional experiments using the larger SDXL model fine-tuned on the higher-quality MJHQ-30k [B] dataset. We also adopted the GenEval benchmark [C] as you suggested, to more comprehensively evaluate the model's performance.\n\n    We ranked MJHQ-30k samples using PickScore and selected the top 500 as expert demonstrations. Both `SDXL + SFT` and `Diffusion-DRO` were fine-tuned on this dataset. For reference, we also include the results of the original `SDXL` and `Diffusion-DPO`, the latter of which was fine-tuned on Pick-a-Pic v2. Following the GenEval protocol, each prompt was used to generate four images to ensure stable evaluation.\n    \n    | Method            | Colors  | Color Attr. | Single Object | Two Object | Counting | Position |Overall  |\n    |-------------------|:-------:|:-----------:|:-------------:|:----------:|:--------:|:--------:|:-------:|\n    | SDXL              |  87.23  |  23.00      | 99.69         |   78.03    |   43.75  |   11.75  |57.24    |\n    | Diffusion-DPO     |  86.17  |  25.00      | 98.75         |   85.25    |   48.25  | **13.60**|59.50    |\n    | SDXL + SFT        |  88.94  |**27.00**    | 99.75         |   87.68    |   46.50  |   13.20  |60.51    |\n    | Diffusion-DRO     |**90.43**|**27.00**    | **100.00**    | **87.88**  | **50.00**|   12.00  |**61.22**|\n    \n    This demonstrates the method’s strong generalization ability to both a larger backbone (SDXL) and a higher-quality dataset (MJHQ-30k). Notably, even when fine-tuning on expert demonstrations in the form of image-text pairs alone, without explicit preference labels, Diffusion-DRO is able to outperform SFT. For example, it achieves a 1.49 percent gain in the \"Colors\" category and a 3.5 percent improvement in \"Counting.\"\n\n    [B] \"Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation\", arXiv.\n\n    [C] \"GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.\" NeurIPS 2023.\n    \n    ---\n\n- ### Q5\n    The paper might also want to acknowledge other works on ranking based DPO e.g. LiPO[a]\n\n    [a] Liu et al. \"LiPO: Listwise Preference Optimization through Learning-to-Rank\", NAACL 2025\n    ###\n    ### A5\n    We sincerely thank the reviewer for pointing out this relevant work. In our current setting, we treat expert demonstrations as binary preferences (i.e., preferred vs. not preferred) without modeling any internal ordering or relative ranking among them. Therefore, our formulation is more aligned with pairwise or margin-based optimization rather than listwise ranking, which is the focus of LiPO [a].\n\n    That said, we agree that incorporating richer ranking information (such as listwise preferences or soft rankings) could offer valuable additional supervision for fine-tuning generative models. We will cite LiPO in the revised version and highlight it as a promising direction for future work that complements our current approach."}, {"review_id": "VtEcXOf1Ek", "reviewer": "Reviewer_82HS", "summary": "* This paper proposes a novel method for aligning text-to-image diffusion models with human preferences. Instead of fine-tuning a policy by preference data, this paper proposes to fine-tune a policy by using expert demonstration data. To achieve, this, the paper proposes a new algorithm, Diffusion Denoising Ranking Optimization (Diffusion-DRO), that treat the RLHF (alignment) problem as apprenticeship learning, that alternate between training a reward function and training a policy (conceptually similar to GAIL (Ho and Ermon 2016)), but for RLHF with diffusion policies setting).\n\n* The key difference between the proposed work and Supervised Fine-Tuning (SFT) in previous work is that it not only fit the reward model to match the expert demonstration as done by SFT, but also train the reward and policy in an adversarial manner, i.e., optimize policy to max the learned reward & learn a reward that damage the learned policy.\n\n* The key difference between the proposed work and DPO-based approaches in previous work is that this work is doing apprenticeship learning (IRL + policy optimization) while DPO-based approaches is learning from preferences + policy optimization in 1 step.\n\n* Results evaluate the proposed method in 2 different datasets (Pick-a-Pic v2 and HPDv2), against four baselines, including SPIN-Diffusion [40], Diffusion-SPO [18], Diffusion-DPO [34], and Diffusion-KTO [17]. Regarding the metrics, including PickScore, Aesthetic Score, ImageReward, and human preferences in a user study, the proposed method significantly outperformed baselines. Regarding the metric, CLIP Score, sometimes the proposed method is outperformed.", "questions": "* For line 102, the paper says that they make an assumption formulated in Eq.4. I am curious when this assumption holds. It seems that Eq.4 says that the reward of the image x0 given prompt c, depends on the entire diffusion process x_{1:T} that generates x0. This is a bit counterintuitive for me because I think whether an image x0 is good or bad only depends on itself, not its generation process. Could you help me ground this assumption a bit more?\n\n* For line 127, the authors say that they parameterize the trajectory reward as similar to Eq.3. I wonder whether this is a choice made by the authors for math convenience, or perhaps there is some ground truth supporting this parameterization?\n\n\n* For lines 226-231, I think the text tries to explain why the proposed method, Diffusion-DRO, is outperformed by baselines regarding the CLIP scores. But I don't quite understand the explanations from lines 226-231. Is the reason that Diffusion-DRO only uses the expert demonstrations, a small part of the preference dataset, resulting in a bad text encoder?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* This paper solves a well-motivated problem, is well-written, proposes a novel method, and demonstrates with significantly good empirical results.\n\n* One key concern of the proposed method is that it seems to lose the benefit of DPO methods that are designed to avoid first fitting a reward function and then finding a policy to optimize the reward. Instead, the proposed method in Alg.1 has to first do IRL (Alg.1 line 14) and then do policy optimization (Alg.1 line 16). It is very helpful that Sec.3.4 discusses the frequency of the two alternative steps for IRL and policy optimization. But I wonder whether the alternation between reward and policy learning results in increased computation overhead? Maybe some empirical evaluation of computation could be helpful to address this.\n    * Also, it is not quite clear to me how the pϕ in Alg.1 line 16 is computed. It would be great if the authors could clarify this.\n\n* Another concern is to find demonstration data. Preference data are largely available and easy for humans to provide. But expert demonstrations are hard to obtain. This paper proposes to use the top K images as the expert demonstrations, from a given preference dataset ranked by the preference labels. This makes sense, but introduces a hyperparameter K to tune.", "was_revised": true, "final_justification": "I think this paper contributes a new method for RLHF on diffusion models. After the rebuttal, I am still concerned that the algorithm needs high-quality demonstrations. In other words, it requires hand-designed metrics to find good demonstration data. Depending on the selection metrics, the amount of demo data will differ, making it hard to tune the K parameter. Moreover, from the table provided in the author's response, it seems that the improvement of Diffusion-DRO over Diffusion-DPO is quite small. It is unclear how significant that is in reality. So I plan to keep the score now.", "rebuttal": "We sincerely appreciate your thoughtful review. Below, we address each of your questions in detail. Lastly, we provide clarifications for several concerns mentioned in the weaknesses section that were not explicitly formulated as questions.\n\n## Reply to Questions\n\n---\n\n- ### Q1\n    For line 102, the paper says that they make an assumption formulated in Eq. 4. I am curious when this assumption holds. It seems that Eq. 4 says that the reward of the image $x\\_0$ given prompt $c$, depends on the entire diffusion process $x\\_{1:T}$ that generates $x\\_0$. This is a bit counterintuitive for me because I think whether an image $x\\_0$ is good or bad only depends on itself, not its generation process. Could you help me ground this assumption a bit more?\n    ###\n    ### A1\n    It is correct that human judgment typically depends only on the final image $\\mathbf{x}\\_0$ and not on the specific trajectory that produced it. However, in Eq. 4, the trajectory-level reward $R(\\mathbf{x}\\_{0:T}, \\mathbf{c})$ is introduced as a *modeling choice* to enable more efficient and stable optimization. The key requirement is that the **expected value** of this trajectory reward under the diffusion process aligns with the scalar reward $r(\\mathbf{x}\\_0, \\mathbf{c})$:\n    $$\n    r(\\mathbf{x}\\_0, \\mathbf{c}) = \\mathbb{E}\\_{\\mathbf{x}\\_{1:T} \\sim p(\\cdot \\mid \\mathbf{x}\\_0, \\mathbf{c})} \\left[ R(\\mathbf{x}\\_{0:T}, \\mathbf{c}) \\right].\n    $$\n\n    Importantly, this does **not** assume that human preferences depend on $\\mathbf{x}\\_{1:T}$; instead, it enables us to decompose the reward across the diffusion steps. This decomposition is beneficial for computational reasons, as it enables gradients to be computed at each denoising step, making the optimization more tractable than backpropagating through the entire diffusion process.\n\n    Moreover, the formulation is flexible. If the learned $R$ chooses to ignore $\\mathbf{x}\\_{1:T}$ and focus solely on $\\mathbf{x}\\_0$, it naturally collapses to a standard reward function:\n\n    $$\n    \\begin{aligned}\n    r(\\mathbf{x}\\_0, \\mathbf{c}) &= \\mathbb{E}\\_{\\mathbf{x}\\_{1:T}} \\left[ R(\\mathbf{x}\\_{0:T}, \\mathbf{c}) \\right] \\\\\\\\\n    &= \\mathbb{E}\\_{\\mathbf{x}\\_{1:T}} \\left[ R(\\mathbf{x}\\_0, \\mathbf{c}) \\right] \\\\\\\\\n    &= R(\\mathbf{x}\\_0, \\mathbf{c}).\n    \\end{aligned}\n    $$\n\n    This formulation allows flexibility and lets the learning process determine whether information from the full trajectory is beneficial for modeling the reward.\n\n---\n\n- ### Q2\n    For line 127, the authors say that they parameterize the trajectory reward as similar to Eq. 3. I wonder whether this is a choice made by the authors for math convenience, or perhaps there is some ground truth supporting this parameterization?\n    ###\n    ### A2\n    The parameterization in Eq. 9 is inspired by the formulation used in DPO [A], where the reward is expressed as a log-ratio of policy and reference distributions. As shown in Theorem 1 of DPO, this parameterization can represent any reward function without loss of generality. Therefore, the reward model does not lose any modeling capability with this parameterization. Moreover, this parameterization closely resembles Eq. 3 in our paper, which enables a seamless transition to the closed-form expression for the optimal policy in Eq. 10.\n\n    Thus, the parameterization serves both as a **principled modeling choice** and a **practical tool** that simplifies the derivation while preserving generality.\n\n    [A] \"Direct preference optimization: Your language model is secretly a reward model.\" NeurIPS 2023.\n\n---\n\n- ### Q3\n    For lines 226-231, I think the text tries to explain why the proposed method, Diffusion-DRO, is outperformed by baselines regarding the CLIP scores. But I don't quite understand the explanations from lines 226-231. Is the reason that Diffusion-DRO only uses the expert demonstrations, a small part of the preference dataset, resulting in a bad text encoder?\n    ###\n    ### A3\n    We would like to clarify that the text encoder used in SD 1.5 shares the same pretrained weights as the one used in CLIP. Since SD 1.5 was trained from scratch on a large dataset, the images it generates naturally align well with the CLIP text encoder, resulting in strong CLIP scores.\n\n    When SD 1.5 is fine-tuned using only a small dataset and a relatively small batch size, the model may slightly deviate from the original distribution. This shift can lead to a mild drop in CLIP score, as the generated images become less aligned with the CLIP encoder.\n\n    This phenomenon can also be observed in models fine-tuned from `SD 1.5 w/ SFT`. For example, in Table 3, `Diffusion-DPO w/ SFT` shows a lower CLIP score compared to the version using the official weights (`Diffusion-DPO`), even though other metrics improve.\n\n## Clarification of Weakness\n\n---\n\n- ### Weakness 1\n    \"One key concern of the proposed method is that it seems to lose the benefit of DPO methods that are designed to avoid first fitting a reward function and then finding a policy to optimize the reward. Instead, the proposed method in Alg.1 has to first do IRL (Alg.1 line 14) and then do policy optimization (Alg.1 line 16). It is very helpful that Sec.3.4 discusses the frequency of the two alternative steps for IRL and policy optimization. But I wonder whether the alternation between reward and policy learning results in increased computation overhead? Maybe some empirical evaluation of computation could be helpful to address this.\n\n    Also, it is not quite clear to me how the $p\\_\\phi$ in Alg.1 line 16 is computed. It would be great if the authors could clarify this.\"\n    ###\n    ### Response to Weakness 1\n    We appreciate the reviewer’s concern regarding the potential overhead of alternating between reward and policy optimization. We would like to clarify that in our framework, the **policy model $p\\_\\theta$ is always identical to the reward model $p\\_\\phi$**, as formally derived in Eq. 10. This means that **every update to the reward model simultaneously updates the policy**—no additional optimization step is required for the policy. Line 16 in Algorithm 1 simply copies the parameters ($p\\_\\theta \\leftarrow p\\_\\phi$) and introduces **no computational overhead**, regardless of the value of $M$. In all of our experiments, we set $M = 1$ so that the policy and reward model remain synchronized at all times. We will add this detail explicitly in Appendix A.2 of the revised version.\n\n    Please note that this design closely mirrors the efficiency advantage of DPO: by training only a single model (the reward model), the generation policy is optimized implicitly. Our proposed Diffusion-DRO thus retains the same **single-model training benefit** as DPO, despite being derived from an IRL perspective. We will revise the paper to make this explanation more explicit and appreciate the reviewer’s feedback on this point.\n\n- ### Weakness 2\n    Another concern is to find demonstration data. Preference data are largely available and easy for humans to provide. But expert demonstrations are hard to obtain. This paper proposes to use the top K images as the expert demonstrations, from a given preference dataset ranked by the preference labels. This makes sense, but introduces a hyperparameter K to tune.\n    ###\n    ### Response to Weakness 2\n    The table below shows the original data used to produce Figure 3 in our paper. As noted in the main text, expert demonstrations are selected based on HPSv2, so we omit HPSv2 scores here.\n\n    As shown, when $K$ ranges from 100 to 1000, Diffusion-DRO consistently outperforms the baselines across most metrics and performs comparably in the remaining cases. While our method introduces a hyperparameter $K$, we find that it performs robustly across a reasonably range of values. It is not necessary to fine-tune $K$ precisely to obtain strong performance.\n    |   Method                 | PickScore | Aesthetic |  CLIP    | ImageReward |\n    |:-------------------------|:---------:|:---------:|:--------:|:-----------:|\n    | Diffusion-DPO            |  21.31    |   6.35    |  0.3334  |   0.7912    |\n    | Diffusion-KTO            |  21.25    |   6.34    |  0.3296  |   0.7636    |\n    | SPIN-Diffusion           |  21.40    |   6.26    |  0.3305  |   0.5619    |\n    | Diffusion-DRO($K=100$)   |  21.55    | **6.46**  |  0.3373  |   0.7876    |\n    | Diffusion-DRO($K=500$)   |**21.54**  |   6.42    |  0.3390  | **0.8452**  |\n    | Diffusion-DRO($K=1000$)  |  21.46    |   6.34    |  0.3409  |   0.8188    |\n    | Diffusion-DRO($K=5000$)  |  21.25    |   6.30    |**0.3439**|   0.7686    |"}, {"review_id": "Lr4rW5f9Fl", "reviewer": "Reviewer_Y9sn", "summary": "This paper is quite formula-heavy, so I’m trying to clarify its main points. The authors aim to improve the performance of diffusion models using reinforcement learning (RL). The abstract notes that previous methods improved training stability by avoiding the REINFORCE algorithm but still face challenges such as accurately estimating image probabilities due to the nonlinear nature of the sigmoid function and the limited diversity of offline datasets.\n\nThe main challenges are thus “accurately estimating image probabilities” (which I interpret as accurately estimating the reward for images) and the limited diversity of offline datasets. The abstract then proposes a method called Diffusion-DRO. By casting preference learning as a ranking problem, this approach overcomes the nonlinear estimation issues. It also integrates offline expert demonstrations with online policy-generated negative samples to address the limited diversity of offline datasets.\n\nNow, looking at how Diffusion-DRO achieves these two goals: equations (1) to (6) explain the basics of RL and reward functions, but equations (7) to (12) are quite obscure. My understanding is that the authors want there to exist an oracle image from the distribution D(c), and the model-generated images should be as close as possible in quality to those from D(c). However, if this were the whole story, wouldn’t simple supervised fine-tuning (SFT) suffice?\n\nThen, equation (13) mentions there is an extra term compared to SFT. The authors say that the second term serves a complementary purpose, where the reward model generates negative samples online to guide the optimization in the correct direction. I don’t quite understand this part.\n\nAccording to my shallow understanding, the SFT term encourages the model’s current noise prediction to match the ground truth noise, while the second term encourages the current model’s noise prediction to differ from that of its previous version, which helps ensure diversity. I think I understand how this addresses the limited diversity of offline datasets. However, I am still unsure whether this fully solves the problem of accurately estimating image probabilities. This seems to relate to the ranking formulation, but I am struggling to grasp how ranking is used here.\n\nJumping ahead to the experiments, the authors demonstrate the effectiveness of Diffusion-DRO by showing win rates through pairwise comparisons. The results indicate that Diffusion-DRO outperforms other models on most metrics.", "questions": "Sorry, I actually didn’t quite understand the paper. I hope the authors can provide a more straightforward and easy-to-understand explanation.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[+] Good performance.\n\n[-] Please refer to the summary, there are still many things I don't understand.\n\n[-] Although the article says that the reward model can be removed, Algorithm 1 still needs to initialize a reward model.\n\n[-] The ICML 2024 paper Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models treats the SFT dataset as oracle data, which is equivalent to the expert demonstrations in this context. In their approach, any content generated by the model is regarded as a negative example, while the SFT examples are treated as positive examples—similar to the second term in equation (13).", "was_revised": true, "final_justification": "Thank you for your reply; it has resolved all my doubts. I will raise my scores accordingly.", "rebuttal": "We sincerely thank the reviewer for the careful and thoughtful review. Below, we respond to the concerns raised in both the weaknesses and questions sections.\n\n---\n\n- ### Q1\n    Now, looking at how Diffusion-DRO achieves these two goals: equations (1) to (6) explain the basics of RL and reward functions, but equations (7) to (12) are quite obscure. My understanding is that the authors want there to exist an oracle image from the distribution $D(c)$, and the model-generated images should be as close as possible in quality to those from $D(c)$. However, if this were the whole story, wouldn’t simple supervised fine-tuning (SFT) suffice?\n    ###\n    ### A1\n\n    From a reinforcement learning perspective, inverse reinforcement learning (IRL) offers a principled alternative to supervised fine-tuning (SFT), which is also known as behavior cloning in RL terminology. We summarize the key differences in the following table:\n\n    | Aspect        | SFT               | Inverse RL |\n    | ------------- | ------------------------------------------ | ------------------------------------ |\n    | Objective     | Imitate expert policy                      | Recover reward function              |\n    | Required Data | Expert text-image pairs                    | Expert text-image pairs              |\n    | Advantages    | Simple to implement; no interaction required | Interpretable objectives; better generalization |\n    | Disadvantages | Susceptible to compounding errors; limited extrapolation | Harder to optimize; sample inefficient  |\n\n    In summary, we choose inverse RL over direct supervised fine-tuning in order to improve generalization. Rather than directly imitating expert demonstrations, our method learns a reward model that implicitly evaluates generation quality. This enables stronger learning signals and allows the policy to generalize beyond the specific modes captured in the demonstration data.\n\n    ---\n\n- ### Q2\n    According to my shallow understanding, the SFT term encourages the model’s current noise prediction to match the ground truth noise, while the second term encourages the current model’s noise prediction to differ from that of its previous version, which helps ensure diversity. I think I understand how this addresses the limited diversity of offline datasets. However, I am still unsure whether this fully solves the problem of accurately estimating image probabilities. This seems to relate to the ranking formulation, but I am struggling to grasp how ranking is used here.\n    ###\n    ### A2\n    The first term (SFT-style loss) indeed encourages accurate noise prediction, while the second term pushes the reward model to assign lower scores to policy-generated samples, helping to diversify and improve generation quality.\n\n    The core of our contribution lies in the **ranking-based formulation** introduced in Eq. 7, which specifies that expert demonstrations should receive higher rewards than policy samples. This ranking constraint is enforced via a **max-margin objective**, allowing the reward model to learn preference-consistent scores without requiring exact image likelihoods. Unlike Diffusion-DPO, which uses a non-linear transformation $\\log \\sigma(\\cdot)$ to convert rewards into classification logits (Eq. 14), our method **maintains a linear relationship** between expert and policy scores. This avoids the use of Jensen’s inequality, which introduces a relaxation gap in the optimization. As a result, our approach can **more precisely optimize the reward margin**, leading to stronger alignment with expert demonstrations.\n\n    While we do not attempt to estimate exact image probabilities, which is known to be difficult in diffusion models, our ranking loss offers a more **faithful training signal** for learning sample quality preferences. This directly addresses the challenge of aligning generation quality with human-like judgments without relying on brittle probabilistic modeling.\n\n    We hope this clarifies how ranking is directly utilized in our formulation to improve the fidelity of reward modeling.\n\n    ---\n\n- ### Q3\n    Although the article says that the reward model can be removed, Algorithm 1 still needs to initialize a reward model.\n    ###\n    ### A3\n    In conventional reinforcement learning from human feedback (RLHF), a reward model is typically trained on preference data first, and then used to fine-tune the generative model. This results in a two-stage training pipeline.\n\n    In contrast, our method does not require pretraining a separate reward model. As shown in Eq. 10, our reward model and policy model are essentially equivalent. Therefore, we only need to train the policy model once, and the reward signal is implicitly defined through this joint formulation. This allows us to bypass the need for a standalone reward model and simplifies the training process into a single stage.\n\n    ---\n\n- ### Q4\n    The ICML 2024 paper Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models treats the SFT dataset as oracle data, which is equivalent to the expert demonstrations in this context. In their approach, any content generated by the model is regarded as a negative example, while the SFT examples are treated as positive examples—similar to the second term in equation (13).\n    ###\n    ### A4\n    While both SPIN [A] and SPIN-Diffusion [B] make use of oracle data, the notion of expert demonstrations in our work differs in a fundamental way. Specifically, our expert demonstrations are required to satisfy the max-margin condition in Eq. 7, meaning that their quality must be optimal with respect to the reward model.\n\n    In contrast, SPIN and SPIN-Diffusion train a discriminator to distinguish whether a sample originates from the dataset or from the model. This is particularly evident in SPIN-Diffusion, where the discriminator must be updated frequently to maintain its effectiveness. As a result, their training procedure follows a *multi-stage* process.\n\n    Although our final formulations may appear similar, the role and interpretation of expert demonstrations in our method are conceptually distinct from the discriminators used in SPIN-based approaches.\n\n    [A] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024.\n    \n    [B] Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation, NeurIPS 2024."}, {"review_id": "yU3Gf9cvol", "reviewer": "Reviewer_Fj6s", "summary": "The authors propose Diffusion-DRO, a novel algorithm inspired by Inverse Reinforcement Learning for aligning text-to-image (T2I) diffusion models with human preferences using expert knowledge. Diffusion-DRO employs a minimax optimization framework where a reward model maximizes the margin between expert and policy generations, while the policy network minimizes this margin. The authors validate their approach through experiments on Stable Diffusion v1.5, demonstrating superior performance compared to baseline alignment methods such as KTO and SPO using both automatic metrics and human evaluation.", "questions": "* As mentioned in **Strengths And Weaknesses** section, could the authors provide preliminary results on the generalizability of the DRO on larger size models, say SDXL if time permits.\n* It appears to me that the threshold $m$ is an crucial factor in preventing overfitting, could the authors provide some ablation studies on the behavior of Diffusion-DRO w.r.t different choices of m?\n* It seems a bit unnatural that the aesthetics focused scores are dropping as the training set size increases with both SFT and DRO, given that in table 1 SFT improves such scores for SD1.5. could the authors elaborate a bit more on this?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "* Strengths:\n  * Novel theoretical perspective from Inverse RL. The Max Margin loss can be simplified to an intuitive form of SFT with winning data plus contrastive loss between policy and reward. The overall methodological contribution is solid.\n  * Comprehensive experiments for both automatic and human preference evaluation support the authors' claims.\n\n  * Clear writing with detailed evaluation protocols ensuring fair comparison with baselines.\n\n* Weakness:\n  * My major concern is the generalizability of Diffusion-DRO. The Diffusion-DRO loss handles winning images from expert data, which could be very effective on medium-sized T2I models like SD1.5. However, [1] suggests that SFT with Pick-a-Pic v2 causes performance degradation on SDXL. This indicates we might need higher-quality expert data to scale up to slightly larger models. It would be interesting if the authors could provide evidence that Diffusion-DRO does not hurt SDXL performance within a few optimization steps.\n  * Another potential issue with Diffusion-DRO is tuning, the push-away term $||\\epsilon_{\\theta}(x_t, c,t) - \\epsilon_{\\phi}(x_t, c,t) ||_2^2$ could \"push too far\" and cause overfitting/collapse when the SFT loss component saturates, tackling this issue with different T2I models could require significant tuning efforts on hyper-parameters\n\n\n[1]  Diffusion model alignment using direct preference optimization. Wallace et,al, 2024", "was_revised": true, "final_justification": "As mentioned in comments", "rebuttal": "We sincerely thank the reviewer for the valuable feedback. Below, we address each of the questions in detail. As the concerns raised in the weaknesses section are directly reflected in these questions, we focus our response on answering the questions provided.\n\n---\n\n- ### Q1\n    As mentioned in Strengths And Weaknesses section, could the authors provide preliminary results on the generalizability of the DRO on larger size models, say SDXL if time permits.\n    ###\n    ### A1\n    In response to a suggestion from reviewer `NjRJ`, we extended our experiments to a larger model and dataset, specifically SDXL and MJHQ-30k [A]. We also adopted the GenEval benchmark [B], as proposed by NjRJ, to more comprehensively assess model performance.\n\n    We ranked MJHQ-30k samples using PickScore and selected the top 500 image-text pairs as expert demonstrations. Both `SDXL + SFT` and `Diffusion-DRO` were fine-tuned on this dataset. For reference, we also report results from the original `SDXL` model and `Diffusion-DPO`, the latter of which was fine-tuned on Pick-a-Pic v2. Following the GenEval protocol, each prompt was used to generate four images to ensure reliable evaluation.\n\n    | Method        |   Colors  | Color Attr. | Single Object | Two Object |  Counting |  Position |  Overall  |\n    | ------------- | :-------: | :---------: | :-----------: | :--------: | :-------: | :-------: | :-------: |\n    | SDXL          |   87.23   |    23.00    |     99.69     |    78.03   |   43.75   |   11.75   |   57.24   |\n    | Diffusion-DPO |   86.17   |    25.00    |     98.75     |    85.25   |   48.25   | **13.60** |   59.50   |\n    | SDXL + SFT    |   88.94   |  **27.00**  |     99.75     |    87.68   |   46.50   |   13.20   |   60.51   |\n    | Diffusion-DRO | **90.43** |  **27.00**  |   **100.00**  |  **87.88** | **50.00** |   12.00   | **61.22** |\n\n    This demonstrates the method’s strong generalization ability to both a larger backbone (SDXL) and a higher-quality dataset (MJHQ-30k). Notably, even when fine-tuning on expert demonstrations in the form of image-text pairs alone, without explicit preference labels, Diffusion-DRO is able to outperform SFT. For example, it achieves a 1.49 percent gain in the \"Colors\" category and a 3.5 percent improvement in \"Counting.\"\n\n    [A] \"Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation\", arXiv.\n    \n    [B] \"GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.\" NeurIPS 2023.\n\n    ---\n\n- ### Q2\n    It appears to me that the threshold is an crucial factor in preventing overfitting, could the authors provide some ablation studies on the behavior of Diffusion-DRO w.r.t different choices of m?\n    ###\n    ### A3\n    We fully agree that the threshold $𝑚$ can play a critical role in preventing the model from \"pushing too far.\" Below, we provide an ablation study evaluating Diffusion-DRO with different values of $m$ on the Pick-a-Pic v2 test set, keeping all other settings consistent with those in Section 4.\n\n    | Method                      |PickScore|Aesthetic |CLIP       |IR        |\n    |:----------------------------|:-------:|:--------:|:---------:|:--------:|\n    | Diffusion-DPO               |21.31    |6.35      |0.3334     |0.7912    |\n    | Diffusion-KTO               |21.25    |6.34      |0.3296     |0.7636    |\n    | SPIN-Diffusion              |21.40    |6.26      |0.3305     |0.5619    |\n    | Diffusion-DRO ($m=10^{0}$)  |21.37    |6.38      |0.3290     |0.7279    |\n    | Diffusion-DRO ($m=10^{-1}$) |21.39    |6.40      |0.3267     |0.7076    |\n    | Diffusion-DRO ($m=10^{-2}$) |21.49    |**6.47**  |0.3268     |0.7399    |\n    | Diffusion-DRO ($m=10^{-3}$ used in paper) |**21.54**|**28.48**|6.42      |0.3390     |**0.8452**|\n    | Diffusion-DRO ($m=10^{-4}$) |21.29    |6.25      |0.3431     |0.7104    |\n    | Diffusion-DRO ($m=0$)       |21.13    |6.18      |**0.3426** |0.5955    |\n\n    From the results, we observe that when $m$ becomes extremely small, such as $10^{-4}$ or $0$, performance drops due to the \"push too far\" effect. The best performance is achieved at $m = 10^{-3}$, and while increasing $m$ further slightly reduces the scores, Diffusion-DRO still consistently outperforms both Diffusion-DPO and Diffusion-KTO across most metrics.\n\n    We hope this ablation provides a clearer understanding of how the threshold affects model performance.\n\n    ---\n\n- ### Q3\n    It seems a bit unnatural that the aesthetics focused scores are dropping as the training set size increases with both SFT and DRO, given that in table 1 SFT improves such scores for SD1.5. could the authors elaborate a bit more on this?\n    ###\n    ### A3\n    Thank you for raising this point. We believe you are referring to the trend shown in Figure 3, but please feel free to correct us if we misunderstood.\n\n    In this figure, the so-called training set is not a random subset of the full dataset. Instead, it is constructed by ranking the training data using the HPSv2 preference model and selecting the top K samples. As K increases, the average quality of the selected expert demonstrations gradually decreases. This leads to a drop in performance for both SFT and DRO, especially on metrics related to aesthetics.\n\n    To avoid confusion, we will revise the x-axis label in the subfigures of Figure 3 from \"Train Size\" to \"Top K\" to better reflect that it refers to the size of the expert demonstration subset, ranked by preference quality."}]

Author rebuttal:

Meta-review: This paper proposes a method for finetuning diffusion models called Diffusion Denoising Ranking Optimization (Diffusion-DRO). This method does not require paired preference data like DPO, and instead relies on a set of expert demonstrations (high quality images) that serve as positive examples for finetuning. The authors derive Diffusion-DRO from the perspective of max-margin inverse RL. They propose a simple update formula and provide interpretations in terms of its relationship to SFT and DPO.

Empirically, the authors use Diffusion-DRO to finetune Stable Diffusion 1.5, using the 500 top-scoring training examples from Pick-a-Pic v2 as ranked by PickScore and HPSv2 metrics, as the expert demonstrations. They report performance on Pick-a-Pic v2 and HPDv2 using PickScore, HPSv2, Aesthetic score, CLIP score, and ImageReward metrics. They show that Diffusion-DRO outperforms the baselines on most metrics in terms of automated win rates.

The paper also compares to SFT-augmented versions of baselines, by first optimizing the base model with SFT on the same set of expert demonstrations, and then applying a given finetuning algorithm such as DPO. This seems like a nice way to ensure fair comparisons and show the benefit of Diffusion-DRO.

The authors also perform a user study that shows that Diffusion-DRO is preferred to two good baselines (Diffusion-KTO w/ SFT and Diffusion-DPO w/ SFT). They also provide an ablation over the number of expert demonstrations, where some metrics increase and some decrease given more demonstrations.

Reviewer zy7V found the idea to be novel and the results promising, and raised concerns regarding the intuition of the derivation and the scope of the experiments. The reviewer still had concerns regarding the derivation after the rebuttal, but the new results the authors provided on SDXL were helpful to show the effectiveness of the method.

Reviewer NjRJ found the framework novel and interesting, with good empirical results compared to baselines. This reviewer was concerned that the experiments used only Stable Diffusion 1.5 and focused on win-rates with reward model metrics. The rebuttal addressed these concerns by providing results using SDXL, a different source of expert demonstrations, and different evaluation metrics using the GenEval benchmark. Regarding the evaluation using automated win rates, the reward model scores, with means and standard deviations, are provided in Appendix Tables 3 and 4, and they also support the claim that Diffusion-DRO outperforms the baselines.

Reviewer 82HS found the paper to be well-motivated and well-written, and the empirical results to be good. This reviewer raised questions regarding some derivations, comparisons to DPO, and the selection of expert demonstrations. These results were partially addressed by the rebuttal with new results.

Reviewer Y9sn found the paper to have good performance, and had questions regarding the theoretical framework. The rebuttal resolved all of this reviewer’s concerns.

Reviewer Fj6s found the paper to be novel, clearly written, and containing comprehensive experiments. This reviewer raised concerns regarding the generalizability of Diffusion-DRO to larger models, and the rebuttal provided results on SDXL.

Overall, Diffusion-DRO is a well-motivated method that performs well compared to baselines. The perspective from inverse RL is interesting, the paper is clearly written. I think it would be a useful contribution for the community.
Decision: Accept (poster)