SEE-DPO: Self Entropy Enhanced Direct Preference Optimization
Authors: Shivanshu Shekhar
Shreyas Singh
Tong Zhang
Venue: Accepted by TMLR ()

Abstract: Direct Preference Optimization (DPO) has been successfully used to align large language models (LLMs) according to human preferences, and more recently it has also been applied to improving the quality of text-to-image diffusion models. However, DPO-based methods such as SPO, Diffusion-DPO, and D3PO are highly susceptible to overfitting and reward hacking, especially when the generative model is optimized to fit out-of-distribution during prolonged training. To overcome these challenges and stabilize the training of diffusion models, we introduce a self-entropy regularization mechanism in reinforcement learning from human feedback. This enhancement improves DPO training by encouraging broader exploration and greater robustness. Our regularization technique effectively mitigates reward hacking, leading to improved stability and enhanced image quality across the latent space. Extensive experiments demonstrate that integrating human feedback with self-entropy regularization can significantly boost image diversity and specificity, achieving state-of-the-art results on key image generation metrics.

Published in Transactions on Machine Learning Research (06/2025)
SEE-DPO: Self Entropy Enhanced Direct Preference Opti- mization
Shivanshu Shekhar
shekhar6@rilinois.edd
Siebel School of Computing and Data Science University of Illmoois Urbona- Champaign
Shreyas Singh
sireyas. @fractal..ccoccoooo
Practal AI Research
Tong Zhang
tozhang@hllinois.. edu
Siebel School of Computing and Data Science University of Illinois Urbana- Chaan--Caa
Reviewed on OpenReview: https: // openreview. net/ forum? id= d==QbRFFjjod gCL
Abstract
Direct Preference Optimization (DPO) has been successfully used to align large language models (LLMs) according to human preferences, and more recently it has also been applied to improving the quality texl-to-image diffusion models. However, DPO-based methods such as SPO, Diffusion-DPO, and D3PO are highly susceptible to overfitting and reward hacking, especially when the generative model is optimized to fit out-of-distribution during prolonged training. To overcome these challenges and stabilize the training of diffusion models, introduce a self-entropy regularization mechanism in reinforcement learning from human feedback. This enhancement improves DPO training by encouraging broader exploration and greater robustness. Our regularization technique effectively mitigates reward hacking, leading to improved stability and enhanced image quality across the latent. space. Extensive experiments demonstrate that integrating human foodback with self-entropy regularization can significanlly boost image diversity and specificity, achieving stale-of-the-art results on key image generation metrics.
1 Introduction
Recent advancements in text-to-image generation models have achieved remarkable success Rombach al.) (0022] Podell al. (2023); Saharia et al. (2022); Schneuing et al. (2024), enabling the creation of high- quality images that are both visually striking and semantically coherent. By leveraging large-scale image datasets, these models have demonstrated an unprecedented ability to generate images that align closely with the intended descriptions. This progress has garnered significant attention, highlighting the vast potential for practical applications fields such as art, design, and content generation, while also raising important questions rogarding the broader implications of such technology. Saharia et al. 2022 Schncuing al. 2024 Dhariwal & Nichol (2021)
Text-to-image generation models typically pretrained vast datasets, which often contain low-quality contents, leading to the introduction of various deficiencies in the trained models Rombach et al. (2022). To mitigate these issues and improve the quality of generated outputs, addit ional post-training steps are usually required. Common approaches for post-training include Supervised Fine-Tuning (SFT) Brown et al.
(2020a) and Reinforcement Learning with Human Feodback (RLHF) Christiano al.) (2023) RL.FF has gained popularity in recent years due to its ability to improve generative model's ability using self generated data Bai al. 2022). RLHF has been highly successful in aligning L.LMs Rafailov et. al. (2024b) Brown

---

Published in Transactions on Machine Learning Research (06/2025)
et al. (2020b); OpenAl et al. (2024), and more recently it has demonstrated great potential for enhancing
the generation quality of diffusion models Yang et al. (2024): Wallace et al. (2023]; Black et al. (2024) |
Fan
et al.
p:23);
Liang et al. (2024).
Standard RLHF algorithms rely on a reward model to evaluate output images and a learning algorithm,
such as Proximal Policy Optimization (PPO) Schulman et al. (2017) or REINFORCE) Zhang et al. (2020)
to adjust the generative model based on these rewards. Reward models are often trained on human preference
data Kirstain et al. (2023) : Wu et al. (2023). More recently, Direct Preference Optimization (DPO) fine-
tunes a large language model directly on preference data without reward modeling Rafailov et al. (0224b
The algorithm became popular because it yields competitive performance and is more stable to train than
PPO, and has since been applied to diffusson models as well Wallace et al. (2023); Yang et al. (2024 ; Liang
et al.
2024). Methods like D3PO Yang et al.] (024)), SPO Liang et al. (2024) and Diffusion- DPO Wallace 2023) adapt the DPO objective to train diffusion models. Some more recent studies showed that for
ct al.
large (2024):
language models, online iterative DPO plus reward model learning, is superior to DPO Yuan et al.)
5 Xiong et. al. (2024), and similar techniques can be applied to diffusion models as well. Online DPO algorithms consist of two key steps: model training and sample generation. At each timestep, the current model undergoes finc-tuning on the available dataset using DPO. Following this, new samples are generated from the updated model. These samples are subsequently evaluated and ranked by an existing reward model. The newly scored data is then appended to the existing dataset, ensuring contimnous improvement and adaptation of the model. This is the approach we will employ in this paper.
However, for large language models, when we use online iterative DPO for multiple iterations, the generation model deteriorates significantly due to major shifts from the original distribution which leads to artificially high rewards and mode collapse. This phenomenon, referred to reward hacking Xu et al. (0224) is a challenging problem that we have also observed in diffusion models. [Fig:
A commonly used method to mitigate reward hacking is to include a KI. regularization term in the objective function that enforces the trained language model to be close to the pretrained model (reference distribution) Christiano et al. (2023). We observe that this regularization, while mitigating the reward hacking problem, is insufficient to resolve it for diffusion models. We identify a key reason for this to be that pretrained diffussinn model's generation is not diverse cnough, and thus after RLHF, the trained model would have further reduced diversily. To address this problem, we propose lo incorporate improve diversity. Reduced diversity means that the generation distribution is highly concentrated at a few points, causing the model to repeatedly output very similar images when prompted. Mathematically, we demonstrate that our approach flattens the reference distribution in the KL-divergence term. Therefore the regularization condition encourages the model to be more exploratory, effectively reducing overfitting and mitigating reward hacking. Moreover, our method achieves state-of-the-art performance across various metrics. Finally, we qualitatively show that our approach generates more diverse and aesthetically superior images compared to existing methods, which tend to produce high-quality results only for limited subset of the input latent space for given prompt (that is, lacks diversity). Our main contributions are as follows:
additional self-entropy regularization to
1. We introduce a novel self-entropy regularization term into the standard KT-rogularized formulation of RLHF objective function. We show that this additional term can effectively improve RLHF results for diffusion models.
2. Our method is effective in mitigating reward hacking in DPO-based algorithms for diffusion models. When combined with the prior state-of-the-art methods D3PO, Diffusion-DPG and SPO we are able to obtain new state-of-the-arr results on various image quality metrics.
3. We carry out empirical studies to demonstrate that this regularization technique encourages broader exploration of the solution space, reducing overfitting and preventing reward hacking. Moreover, we achieve enhanced stability and image quality across the latent space of the diffusion model, leading to more consistent outputs.
2

---

Published in Transactions on Machine Learning Research (06/2025)
2 Related Works
Diffusion Model:
Diffusion models have become state-of-the-art in image generation Rombach et al. (0022)
Podell et al. (2023), with methods like Denoising Diffusion Probabilistic Models (DDPM) Ho et al. (2020)
and Denoising Diffusion Implicit Models (DDIM) Sonn al.] (2022) significantly improving both image
quality and generation speed. Text-to-image diffusion models, in particular, have enabled the creation of
highly detailed and realistic digital art Rombach et al. (2022); Podell et al. (2023) Saharia et al. (2022)
There has been an increased interest in adapting these models to specialized domains Li et al. (2023) Chi
et al. (2024); Ajay et al. (2023); Ho et al.] (2022), such as chemical structure generation [Schneuing et al. (2024), where tuning the full model is computationally expensive.
Works like Hu et al. (2021) ; Lai et al. (2024): : Luo et al. (2024) have introduced methods such as adapters and compositional approaches to combine multiple models, enhancing specificity, control, and output qual- ity while lowering the computational needs for training. Additionally, classifier-free guidance (CFG) Ho & Salimans (2022) has streamlined the sampling process by removing the need for separate classifier, instead relying on the model's predictions of both conditional and unconditional noise to improve quality and diver- sity. In contrast, classifier-based guidance Dhariwal & Nichol (2021), while providing more precise control, comes with higher computational costs due to the need for an external classifier.
RLHF: RLHF Christiano et al. (2023) has become a pivotal technique for aligning the outputs of founda- tional models with human preferences by incorporating human leedback during the training process. This method enables models to be fine-tuned to generate behaviorally aligned outputs, addressing issues like bias or harmful content. RLHF has been successfully applied across diverse domains, from developing policies for Atari games to training robotic systems
Bai et al. (2022).
Notably, OpenAT's GPT series OpenAT et al. 2024 has showcased the impact of RLHF in enhancing language model performance, other parallel works have also shown a similar improvement across tasks Anthropic (2024]: Google (2024); Touvron et al. (2023), particularly in improving coherence, relevance, and user experience. In parallel, Reinforcement Learning from AT Feedback (RLAIF) has been proposed cost-cffective alternative, utilizing AI systems for feedback to scale model refinement. In this paper, adopt RLAIF to generate high-quality images and achieve stale-of-the-arl performance across multiple evaluation metrics.
DPO: A key challenge in RLHF lics in designing an effective reward function, which typically requires a large, curated dataset for training order achieve competitive results Christiano el al. (2023). Even after constructing the reward model, it is vulnerable to manipulation by policies, especially for out-of- distribution (OOD) samples, leading to artificially inflated rewards. Additionally, training the main policy while simultaneously loading the reward model into memory introduces further computational complexity.
To address these issues, Rafailov et al. 2024b) introduced Direct Preference Optimization (DPO), a method that fine-tunes language models directly from user preferences, leveraging the correlation between reward functions and optimal policies. However, this approach does not directly extend to diffusion models, as DPO operates within the Contextual Bandit framework, which treats generation as a single-step process, whereas diffusion models require handling latent states over multiple timesteps, making training infeasible even with techniques like LoRA Hu et al. (2021). Diffusion-DPO Wallace et al. (2023) adapts this by optimizing upper bound on the true loss function, D3PO Yang et al. (2024) models the diffusion process as Markov Decision Process (MDP), using the action-value function instead of a traditional reward model at each generation step. This approach assumes that the chain generating the preferred response is best at every step. In contrast, SPO Liang et al. (2024) introduces training a reward model that can score samples across varying noise levels, allowing for effective optimization of the DPO objective throughout the diffusion process. This method relaxes the strict assumptions made by D3PO, providing a more flexible approach diffusion modeling.
3

---

Published in Transactions on Machine Learning Research (06/2025)
Epoch: 130
Epoch: 150
Epoch: 160
Epoch: 130
Epoch: 150
Epoch: 160
Entropy Regularization
Entropy Regularization
Figure 1: Effect of Entropy Regularization reward hacking. This figure demonstrates the effective- ness of entropy regularization for mitigating reward hacking problems during Direct Preference Optimization (DPO) Xu et al. (2024). The A images are distorted whereas the B images are acsthetically better at the same epochs.
3 Background
Diffusion Models: Diffusion models are generative models that learn represent the data distribution p(xo) by gradually corrupting data with noise in a forward diffusion process, and subsequently learning reverse this process. Denoising diffusion models parameterize the data distribution (o(x)), where the reverse process is modeled as Markov chain. This reverse process is discrete in time and follows the structure:
po(Xo.T) p(xr) pa (xt-1 Xt)
(1)
where
2 pa xx)) = (I=IiiuO(Zz)
of
p(xr) - (, (x);
The model is trained by maximizing the evidence lower bound (ELBO) on the data likelihood. The training objective for diffusion models is defined
LDM ExpEtx (w)))]lee (x1, 1)
(2)
Here, the noise sampled from a normal distribution, i.c., ~ N(H, I), and the time step sampled uniformly from the range L((O T) where T the total number of diffusion steps. The corrupted image x, is sampled from Gaussian distribution conditioned the original image, q(x, Xn) (xxx;; 06X0, ofT).

---

Published in Transactions on Machine Learning Research (06/2025)
where X = af/o? represents the signal-to-noise ratio. The function ((A) is predefined weighting function, often chosen as a constant for simplicity.
Reward Modelling: Reward modeling focuses on modeling human preferences for a generated sample x under a specific conditioning c. To achieve this a human-annotated dataset is usually used which contain triplets (Xw: X1, c), where Xw represents the preferred (winning) sample and xi represents the less preferred (losing) sample given the conditioning C.
For preference learning which is used in online iterative DPO, usually the Bradley-Terry (BT) model Bradley & Terry 1952 is used to describe the probability of Xw being preferred over conditioned on c, which is formalized as:
PBT > xi(c) = (r(c, xw) r(c,
(3)
where r(c,x) denotes the reward function and o(-) is the sigmoid function. The reward function is parametrized as (),, where & represents the model parameters, and we finally optimize the log-likelihood of Eq. 3 as the primary learning objective:
Lør(Ø) Ex,, [log (ra(c, - r (c, x1))]
DPO: Reinforcement Learning from Human Foodback (RLHF) aims to optimize the parameters 0 of a con- ditional distribution (xol(c) over latents Xg conditioned on context c. The objective of RLHF commonly framed as maximizing the expected reward while minimizing divergence from reference policy. Mathemat- ically, the objective can be expressed as:
max De [r(c, mo)] 6DKL [po(zo)))llraazzoee] Pa
In Eq. 5 the first term encourages the policy to maximize the expected reward, while the second term, weighted by byperparameter 3, introduces a regularization to keep the current policy close to reference policy Pref- In this formulation, & controls the tradc-off between exploration (reward maximization) and exploitation (closeness to the reference policy).
However, this formulation assumes that the reward model ideal, which is often not the case practice. Training and querying reward model typically incurs high computational costs, making the approach challenging to scale. To address this issue, DPO Rafailov et al. (024b) reformulates the problem to directly solve for the optimal policy Pe in closed form, defined as:
(r(c,x)) xo)
py (x (xor(x)lc) exp
/Z(c)
(6)
Here, Z(c) is the partition function:
rr(c, xo)
Z(c)= = Σ
Preel ( colc exp
(7)
B
20
This leads to rewritten form of the reward function:
r(c, = Bog (5(x) Pros(xolc)
+ 33 log Z(c)
(8)
Substituting this into the original objective in Eq.
the reward objective for DPO becomes:
Pret(ag (c) Prer ]
Po (o
po(zblc)
LOPO (0) FRe. ai an log a
B Blog
3 log
(9)
This reparameterization allows DPO to bypass the need for reward function optimization and instead directly optimizes the conditional distribution po(zulc). All DPO-based diffusion methods (Diffusion-DPO, D3PO & SPO) introduce approximation techniques to apply this approach to diffusion models.
5

---

Published in Transactions on Machine Learning Research (06/2025)
D3PO: Instead of directly optimizing the reward model, Yang et al. (2024) formulate the generation process 8s Markov Decision Process (MDP) and focus learning a policy that maximizes the action-value function.
The corresponding objective is given in Eq. 10
max Ba (116) (* - 3 DKL [[(a| ((
(10)
with Q(s, a) inplace of r(c, mo) Consequently, we can adopt an optimization approach similar DPO derive their final loss function:
As we can see Eq- similar to Eq. 5
LDSPO( (.( log p
(B. (118
log To (s))
loo To(a((lff)
(11)
( (s)))
(f (s)))
Diffsuion-DPO: To address the memory limitations that arisc when directly applying DPO to diffusion models Wallace et. al. (2023) upper bound Eq. g using the Jensen-Shannon inequality. They approximate the reverse process pa (Z1:T 20) with the forward process q( and leverage the fact that the forward process follows Gaussian distribution to derive their final objective, as shown below:
-BT (Iee - co(ax, t)1 -Ilé' -ee -e(ett.))l)) t))l²)
LDiffusionDPO -Ei,e=, log
-notul 1)1 ||e' -nn(((b))))) 1)
(12)
SPO: D3PO assumes that a preferred image is associated with corresponding preferred diffusion chain, which may be a strong assumption. In contrast, SPO follows the same MDP formulation as D3PO but, instead of optimizing an action-value function, utilizes a per-step reward model capable of scoring noisy images and providing reliable preference signals al each step. This allows SPO to generate preferences iteratively without relying on D3PO's assumption. As result, SPO can optimize Eq. at each diffusion step, leading to final objective that aligns with D3PO but applied every step of the diffusson process.
In this paper, we show that all these approaches are mathematically similar and we further introduce our self-entropy regularization in this unified framework.
4 Methodology
In this section, we prove the equivalence of D3PO, SPO, and Diffusion-DPO. We model the diffusion process as an MDP, we borrow the notations from Yang et al. (2024):
A (c,t,IT-1)
P(st41
s1,a,) (5e, d141, 5xT-1-1)
at A 8T-1-t
(a; s:) è (o
(T))
r(s a:) - r ( t, Iт 1T-
Where T the maximum number of denoising timesteps and 8 represents Dirac delta distribution. Wo want to optimize Eq. we can re-write it as:
Pa 1 3 log II II
pe (=(z_112()
(0) -Beay on log
Bog
(13)
(ret(x?" (1:"" (xc(0)
Pref 1et

---

Published in Transactions on Machine Learning Research (06/2025)
We assume that the number of genration timestep (T) is same for both winning and lossing sample, by using We don't write the conditioning on
the property logæl x22 = log 1:1 + log x:2 Vrl,22 22 E R+ we simplify 13
explicitly in Eq. 13 for conciseness:
(0) = -Bagy in log
1 BT
Σ
log Pe(x7_1aa7 1x(")) Pret 127))
BT
1
Σ
log
(0 Prot(1/-1
)
(14)
T
T
We can define & as random variable which is sampled from uniform distribution with support from 0 to T. Using this and Jenson Shannon inequality we arrive at:
po 20(42_1111 Prot(ar/- (2e)) ]
(DPO(0)= S Beeey al ((000T) log a BTT log Pe (xx") Pret (T (2)0)
BT BT log
(15)
This objective is similar to Diffusion-DPO, instead we consider optimization at each step then the objec- tive translates SPO and D3PO's objective. Now we introduce our self-entropy regularization into this framework. We start with D3PO's formulation:
max En (-1))1)))))))))))))))))) as) - B DKL [[ + (alss) log x(als)
B x(als) log <{ass)
max Eg ( Q²"(s, a)]
+ (alss) {(als)
Tre((als)
((als)
max Eg (11)) (2(ss, - 37 ()T(()aa)) log
Troel (a) (
max Eg am)))-1) (s((s. - B(v + 1) DKi, [[(o)))
(16)
We can that the only effect of adding our self entropy (erm is on Tref, particularly we can simplify Eq. 16 using the usual DPO derivation Rafailov et al. (2024b) to show that adding the self entropy is equivalent to raising Tref by + y. Hence can use the same procedure as D3PO to arrive at our update objective:
L(0)- ((( log p ( (1(+1))+)))))11 (1og [ogg
Te(a log To 15h
(17)
Trof (x (s नें Tref (a 7FT
Since SPO follows the same formulation as D3PO Eq. 17 can also be applied to train SPO. From Eq. 17 see that for 7 > 0 the final effect of adding the self-entropy term to flatten out the reference distribution which in return forces the policy to be more explorative (appendix C.1 we can also write this in terms of the noise objective, similar to Diffusion-DPO.
L. (1+7) (Ile" - c0(2)" se(x"",))) - -Ile'
t)]1
e (o(s/,t))1 t))1²)
Ia -1c -ret(aa"""" 2 t)]1 - ||e' - (ret(at, 1))1²)
-B logo (T -an--ii) Li)).
We could also start from the formulation of Diffusion-DPo to get an alternate version of our objective which is different only by a constant, we follow the same steps as of Diffusion-DPP to arrive at our final objective the exact derivation can be found in the appendix C.2

---

Published in Transactions on Machine Learning Research (06/2025)
Twu (Iee" - -11e² -co(s)
=o(())+1)1²)) 1)l1²
L
(1 + 7) (1ee f t)[l -Ile' cre((a/;
[I Lsmaal S -Rr,e log ( - Li))
We can see that to a constant factor this is similar to one derived directly from Eq. 17 Intuitively, in this formulation, the loss associated with the reference model is reduced when 7 > 0. Therefore the resulting diffusion model relies less on the reference model. This allows the trained model to explore the low probability space of the reference model and thus increases the diversity of the trained model.
5 Experiments
C30
C210 16.11
33
23.05
06.9
33.55
300
SE50R0
0E6:00TA00ORO (851
2030
Sept
Figure 2: Accuracy vs Steps A: Step- Wise PickScore Liang et al. (2024), B: Aesthetic Scord Wang et al.
22022,, C: PickScore Kirstain et al. (023), demonstrate faster convergence, improved training stability and robustness to reward hacking.
General Preference
25.
Visual Appeal
23.2
SEE-SPO Win.
Draw
Prompt Aligument
40.5
SPO Win
Diversity
Diversity
23.4
General
DilfusionDPP Win
Draw DiflusionDPO Win
Visual Appeal
41.2
Prompt Aligunent
41.7
General Preference
18.2
Visual Appeal
20.6
SEE- DSPO Win
Draw
Prompt Aligument
D3PO Win
Diversity
20
40
60
80
100
Figure User study. We compare our best models with counterparts fine-tuned using the PickScore reward model Kirstain al. (2023). Following SPO Liang al. (2024), we randomly select 300 prompts (2022 and HPS Wu et al. (2023) in 1:2 ratio. Participants
shown reference image alongside three generated images from the same prompt but with different initial latents. They evaluate image quality and diversity relative to the reference to estimate the diversity metric. For other metrics, users assess single images.
from PartiPrompts Yu et al.
Experimental Setting: For a fair comparison, we used the official implementations of D3PO, Diffusion- DPO, and SPO with their default parameters. We trained these models using our proposed regularized loss

---

Published in Transactions on Machine Learning Research (06/2025)
8
SC Wit.Oaatoooo
030
SXC 0UP0
Figure 4: Enhancing Generation Diversity. Our method demonstrates superior image quality, producing clearer and more detailed results across different latent variables. In contrast, existing approaches show significant degradation in image quality, with blurring and distortions becoming evident as latent variations increase.
9

---

Published in Transactions on Machine Learning Research (06/2025)
Method
PickScore
HPS f
Aesthetic Score f
BLIP 1
ImageReward
CLIP
D3PO
20.306
0.227
5.050
0.470
-0.209
0.261
Diffusion-DPO
20.791
0.262
4.711
0.486
0.139
0.263
SPO
20.919
0.266
4.702
0.472
0.191
0.249
SEE-D8PO
20.590
0.247
4.851
0.491
-0.003
0.271
SEE-Diffusion
21.014
0.273
4.721
0.497
0.333
0.266
SEE-SPO
21.295
0.287
4.679
0.510
0.585
0.275
Table 1: Quality Metrics.All numbers are reported on the validation-unique split of Pick-a-Pic V1 dataset.
We evaluate on CL.IP Score Hessel et al.
(2022). BLIP Score Li et al.
(2022), Aesthetic Score Wang et al.
2022), PickScore Kirstain al. (2023). PPv2
Score Wu al. (2023)
Method
RMSE t
PSNR 4
SSIM 4
1
f
StableDiffusion-LL.
0.3745
8,6943
0.2100
7.4627
7.4620
Diffusion-DPO
0.3736
8.7047
0.2406
7.4954
7.4945
SFFF-DiffusioDDO
0.4299
8.1743
0.2056
7.6517
7.6587
D3PO
0.2455
12.5599
0.4190
7.0853
7.0911
SEE-D3PO
0.3052
10.5240
0.3183
7.3650
7.3692
SPO
0.4446
7.1218
0.1666
7.5674
7.5635
SEE-SPO
0.4383
7.0818
0.1575
7.7120
7.7101
Table 2: Diversity Metrics.All values are reported the validation-umique split of the Pick-a-Pic- V1 dataset Kirstain et al. (002)). For each prompt, we first sample a base image and then sample 10 different images to get all the diversity metrics. Evaluation metrics include RMSE, PSNR, SSIM, and entropy energy scores (E1 and E2) Sun et al. 2024).
function, described in the Methodology section. When applying our method, we treated only y and B as hyperparameters while keeping all other settings at their default values to ensure fair evaluation. During inference, we set the guidance scale to 7.5 for consistency across models.
For training, we used 4,000 prompts from the Pick-a-Pic-VI dataset Kirstain et al. 2023 for SPO and D3PO, following the dataset provided in SPO's official GitHub repository. For Diffusion-DPOO we used 800,000 prompts from the same dataset. Fach model trained using the same setup and data splits specified in their original implementations. To establish fair basis for comparison, selected Stable- Dilfusion v1.5 as the benchmark model.
Since our training approach is online, we relied on automatic preference generators. Specifically, we used the time-conditional PickScore Liang et al. (2024) as a proxy for human preference in SPO, human annotations from the Pick-a-Pic-VV dataset for Cirtusiin-DPO. and PickScore Kirstain el al. (2023) for D3PO. Notably, we did use online training for Diffusion-DDi to remain consistent with its original implementation.
Evaluation: To assess the quality of generated images, utilize automatic evaluation metrics, inchuding PickScore Kirstain et al. (2023). HPS Wu et. al. (2023), hesth Score Wang el. al. (022)) CLIP Hes- sel et al. (0022), BLIP Li et al. (002)). and ImageReward Xu et al. (2023). We report results on the vali validation_ unique split of the Pick-a-Pic V1 dataset, which contains 500 prompts, as shown in Tables
and 5
For a comprchensive asscssment of generation diversity, we employ five additional metrics. Root Mcan Square Error (RMSE), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM) are commonly used to evaluate image similarity; here, we leverage them to quantify diversity in generated images. Additionally, we compute image entropy, including both 1D and 2D entropy, to measure the information content within images. Higher entropy values indicate greater diversity and richness in image content.
To evaluate diversity, we generate 10 images per prompt from the validation validationnumique set and compute the aforementioned diversity metrics Sun et al. (2024). We do not rely on embedding- similarity, as images generated from the same prompt, despite visual differences, often cluster closely in embedding space. An effectively aligned model should achieve high reward scores while producing diverse outputs. Our results
10

---

Published in Transactions on Machine Learning Research (06/2025)
demonstrate that the regularized versions of D3PO, SPO, and Diffusion-DPO achieve higher mean rewards across multiple metries while improving diversity scores, highlighting enhanced performance in both diversity and image quality compared to baseline methods.
Additionally, we conduct a user study similar to Liang et al. (2024). We recruit 10 participants to evaluate images generated by different models based on 300 prompts sampled from PartiPrompts and HPSv2 in a 1:2 ratio. Participants assess four aspects: Prompt Aligument, Visual Appeal, General Preference, and Diversity. To evaluate diversity, provide each participant with reference image generated by model alongside three additional images generated from different latent codes. Participants rate both their general preference and the diversity of the additional images based on their differences from the reference image.
Model
B
D3PO
5 0.01
Diffusion-DPO
3
4
SPO
3
0.1
Table 3: Hyperparameter values: All the other hyperparameters values were fixed to the original imple- mentation
5.1 Results
Our models, trained using the proposed objective, outperform or comparable to baseline methods across all quality metrics, as shown in Table 2 Specifically, SEE-D3PO improves PickScore by 1.41%, HPS by 8.8,, and Imageleward by 98.5%. SEE-DiffusionDPO yields improvements of 1.08% in PickScore, 4.2% in HPS, and 139.5% in ImageReward. Lastly, SEE-SPO shows gains of 1.8% in SPO, 7.9% in HPS, and 206.3% in ImageReward.
In terms of image diversity metrics, all our models outperform their respective counterparts across all six metrics. This trend is also reflected in the user study [Fig: where our methods achieve least 6% improvement in generation diversity, Furthermore, our approach enhances all other evaluation aspects, though we observe minimal gains in the Prompt Alignment task. Example samples are provided in the appendix and Fig:
5.2 Ablations
In our experiments, we explored the effect of different values of on both training stability and overall model performance, As shown in Fig: - when 7 = 0, representing the baseline or vanilla model, we observe an initial steep drop in reward values, followed by a suddon and rapid increase during training phases of all the models SPO and D3PO training phases. Examinnig the generated images in this region, we note gradual decline in quality corresponding to the initial drop in scores, followed by further degradation but the reward values continue to rise. This pattern signals that the model begins to overfit prematurely, define as reward hacking. In
leading to the generation of repetitive and noisy images a phenomenon this scenario, the model exploits reward dynamics by producing images that maximize the reward function but lead to distorted images as shown in Fig: [ Through our experimentation, we observed two distinct types of overfitting. In the first, seen in D3PO, the reward values continue to increase despite a progressive decline in image quality, indicating that the model is optimizing for reward at the expense of meaningful outputs. In the second, observed in SPO, the reward model assigns consistently low scores, and the model fails to improve beyond this point, becoming stuck in a suboptimal state. Notably, this trend is absent in the Diffusion-DPO setup, which utilizes the extensive Pick-a-Pic- V1 dataset containing over 800,000 images, likely providing more comprehensive training data that mitigates carly overfitting. For diffusion DPO we see that the model with = -0.5 initially increases in accuracy and then decreases. Across all configurations, our model achieves higher scores faster, with more consistent. stability in training and fewer abrupt changes in reward values.
As training progresses, our model not only stabilizes but also consistently reaches higher reward range. We attribute this improvement to the implicit flattening of the reference distribution that higher y values
11

---

Published in Transactions on Machine Learning Research (06/2025)
21.2
21.0
21.1
Pick Score
21.0
20.8
20.9
20.8
20.7
10
10a
Figure 5: Ablation Study for y & 3: This study investigates the impact of y and on image quality, specifically using PickScore the metric. In the first ablation, cxamine the effect of varying y while keeping all other parameters fixed, with 8 set to 0.1. In the second ablation, explore the influence of different values of B for 3 settings.
introduce. This flattening effect broadens the policy distribution, enabling more exploratory behavior and decreasing the likelihood of overfitting to repetitive patterns.
Further, we observe empirically poorer results compared to the base model when = -0.5, as shown in Fig: 3 D3PO demonstrates early overfitting tendencies, resulting in sharp reward increases followed by rapid declines. Meanwhile, Diffusion-DPO consistently underperforms relative to other models, as the negative 7 value restricts exploratory actions, yielding repetitive outputs that fail to capture meaningful image variations. In the SPO pipcline, overfitting occurs somewhat later than in the vanilla model. This difference can be altributed to SPO's update policy, which only updates the model when it generates images with reward differences that exceed a predetermined threshold. With negative 7, the generation distribution narrows, limiting the diversity of generated images and, consequently, the froquency of model updates. Further evidence of these observations can be seen by examining the rewards vs. steps graph for SPO == 005,, where the reward values remain nearly constant. throughout most of the training duration before ultimately showing signs of overfitting. This steady reward trend reinforces the observation that the model fails to explore diverse solutions, hindering its ability to generalize and adapt, particularly under negative y settings.
The optimal values for 3 and 7 are shown in Table 3 Notably, the optimal 3 for Diffusion-DPO is two orders of magnitude larger than that of D3PO, which is expected since the for Diffusion- DPO is multiplied by T, term of order Tn the case of SPO, we update at each timestep, which is of order 1, so we also expect / increase by one order to prevent significant deviation from the reference distribution.
We also conducted deeper ablation study using SPO, exploring various values of B and y. In Fig: 5 the left all experiments achieve high PickScore, whereas for y < O, the performance significantly drops. As noted carlier, 7 < 0 lcads to reduced diversity and poor training, with results comparable to the vanilla Stable-Diffusion 1.5 model. the right image, we fix y = 3 and vary 8 while keeping other hyperparameters constant. Initially, for low values of 3, the performance the model's increasing reliance on the initial distribution as the regularization strength 3 rises, limiting exploration and slowing training, which results in lower scores.
image shows the results of keeping / fixed at 0.1 while varying 7. We observe that for >>> >
high, but it declines steeply as / increases. This decline may be due
12

---

Published in Transactions on Machine Learning Research (06/2025)
6 Conclusion
In this paper, we introduced Self-Entropy Enhanced Direct Preference Optimization (SEE-DPO) for fine- tuning diffusion models. We showed cxisting approaches that optimize Eq. often suffer from overfitting and distribution shift issues. To address these challenges, we introduced self-entropy regularization term into the objective function, transforming into the final optimization objective outlined in Eq. 17 Our empirical results demonstrate that the proposed SEE-DPO, SEIE-DfftusioDDOO, and SEE-DPO method significantly outperforms their respective counterparss, effectively mitigating the reward hacking problem commonly associated with RLHF approaches and resulting in robust training. Additionally, our method ensures diversity of image generation across the latent space, highlighting its robustness and effectiveness in maintaining high-quality outputs.
Our approach was straightforward to implement within the current framework, specifically DPO with KL- regularized reward optimization. However, extending our method to incorporate other forms of regulariza- tion, such Hellinger distance, may introduce additional mathematical complexity. We leave the exploration of alternative regularizations and their impact performance to future work. Furthermore, although our diversity metrics (RMSE and PSNR) showed minimal improvement with SPO, the user study reported a higher diversity. This discrepancy highlights the need for a diversity metric that better aligns with human judgment, which we plan to investigate in future work.
7 Impact Statement
Generative models for media, such as image generation, present both significant opportunities and challenges. On one hand, these models enable wide range of creative applications and, with advancements that reduce training and inference costs, have the potential to make this technology more accessible and encourage broader exploration. On the other hand, they also pose risks by making it easier to create and disseminate manipulated data, which can contribute to the spread of misinformation and harmful content, including the prevalent issue of deepfakes. Additionally, generalive models may inadvertently expose sensitive or personal information from their training data, raising concerns about privacy, particularly when such data is collected without consent. The potential for this issue in diffusion models, specifically image generation, remains an area requiring further investigation. Moreover, deep learning systems, including diffusson models, can perpetuate amplify biases inherent in their training data. Our methods explores increasing diversity of generation which could positively impact the bias-corretion problem. For more detailed discussion of the ethical considerations of deep generative models we refer readers to Hagendorff (2024).
References
Amurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is condi- tional generative modeling all you need for decision-making?, 2023. URL https:///axxit httpap///axvvoor//babbba org/abs/2211.
15657
Authropic. Claude: An ai assistant. https://www..athhoppicc.
2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Tanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Doddd, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from feedback, 2022. URL https:
org/abs//22
08073
Kevin Black, Michael Janner, Yihun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning, 2024. URL https://aaxxi: http:///axxxvvv../aabba abs /200e.//055.00 13301
13

---

Published in Transactions on Machine Learning Research (06/2025)
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):3243445 1952.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Von Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Danicl M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πla Sutskever, and Dario Amodei. Language models are few-shot learners, 2020a. URL httpp:///arxat http:://axxvv....//a//// wtwg/b/////g.g. httg//bbs20005...o 14165
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vose Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b. URL https://aaxii org/ abaaaaas///0/ 14165
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Todrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2024. URL https:
//arxiv.org/abby ats///bb//2/3/ 04137
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment. learning from human preferences, 2023. URL https::/aaxxv..org/////////// wwwpaaawwww.vv..aabaaaaeeaeeeaee /706 03741
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. URL https
//axxiv org/ wttg/////////ao///////////////// 225//23333000333
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tunngg text-to-iie to-image diffusion models, 2023. URL htttps://arxiv.rgg wttpo//g///b///oeeeeee 16381
Google. Google bard: An ai language model. https://bard.googgt
2024.
Thilo HagendorfT. Mapping the ethics of generative ai: A comprehensivee scoping review. Minds and Machines, 34(4), September 2024. ISSN 1572-8641. doi: 10.0007/111002-004---0666 URL http://dx. doi org/
10.1007//10028-024- 09694-1
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscorc: A reference-free evaluation metric for image captioning, 2022. URL http://aawiv.orgg///////////////
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL https: /tt::://axxvvevee www.iv.orggabe//// 2207.12599
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL https:
7/arxiv. wwwgg//s//0/..../////// 11200006.11199
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Dioderik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022. URL
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv..rgg
abb/2116.0966.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023. URL https://arxiv..r aba/
2305.0155 01569
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model, 2024. URL http:/://wwwwwwww////a/////s////
14

---

Published in Transactions on Machine Learning Research (06/2025)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping langagg--iga pre-training for unified vision-language understanding and generation, 2022. URL https: http::://axxivorr//bbbb.s. www.oogg////////oooooo.oooo.er/o 2201. 12086
Ronghui Li, Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang, Yansong Tang, and Xiu Li. Finedance: A fine-grained choreography dataset for 3d full body dance generation, 2023. URL
https://arxiv wrggabs//212......
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step- aware preference optimization: Aligning preference with denoising performance at each step, 2024, URL
attps://arxiv.on h/////bb//// 2406 .43144
Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-cffcienn fino-tuning for large language models, 2024. URL
https://arxiie sttps///arziv.orrg/aa///// wwt./oo//////aeeeeoooooo 2402. 12851
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cumminns, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Elcti, Tyna Eloundou, David Farhi, Liam Fodus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Montijo-Lopess Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinnoy, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pawlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, 1lya Sutskever, Jie Tang, Nikolas Tezak, Madcleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizaboth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wicthoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang,
15

---

Published in Transactions on Machine Learning Research (06/2025)
Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL htttp://arxivv wttpg/abs/2303..87777
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL
https://arxiv.oot wwg.aaiiwi......aae.. wwwp:///g//aaaaaaoo/ooooooss 2307. 01952
Rafael Rafailov, Joey Hejua, Ryan Park, and Chelsea Finn. From to q": Your language model is secretly 12358
a q-function, 2024a. URL
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.
Direct preference optimization: Your language model is secretly reward model, 2024b. URL https:
//arxiv.org/abbat wtw..//./.bb./..... 2201/
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL htttp://arxiiv.or//aaaa wttp:oo//bb......//.....
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022. URL
www.
www.ooog//////oe..ooooooe abs a20.. 10066.1444 11487
Arne Schneuing, Charles Harris, Yuanqi Du, Kieran Didi, Arian Jamasb, Ilia Igashov, Weitao Du, Carla Gomes, Tom Blundell, Pietro Lio, Max Welling, Michael Bronstein, and Bruno Correia. Structure-basel drug design with equivariant diffusion models, 2024. URL httpss//arxiv.. wtgggb/////////oo//////g////////
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms, 2017. URL
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL https: //arxiv. wttp:///gaa...oo////// 02502
Haoyuan Sun, Bo Xia, Yongzhe Chang, and Xueqian Wang. Generalizing alignment paradigın of text-to- image generation with preferences through f-divergence minimization. ArXíe, abs/ hos/2409.09774, 2024. URL
2726992234
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswarni, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyyan Zarov, Yuchen Zhang, Angela Fan, Mclanic Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models, 2023. URL https:///axxi wwwriii.rrgg/baa orgg/bs//// 22////337000929
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023. URL httts:://arti:: org/ aww.aaaaa... 12908
Zijic J. Wang, Evan Montoya, David Munochika, Haoyang Yang. Benjamin Hoover, and Duen Horng Chau. Large-scale prompt gallery dataset for text-to-image generative models, arXiv:2210. 14896 (cs}, 2022. URL https://arxiv. httpl/.org//bab///abeeeeee
16

---

Published in Transactions on Machine Learning Research (06/2025)
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023.
URL https://arxiv. wttp:///i...g/////222666// 09341
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024.
URL
org abtp//bbs/311.... 11456
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxnan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023. URL https:
flaxxiv.org/bb//3044... 05977
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mci, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm aligument? a comprehensive study, 2024. URL
org/ abs/ ab0//444.//2...////// 10719
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Ti, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human fcodback fine-tune diffusion models without any reward model, 2024. URL
https: hopp:://arxivv orgoooggg/oo. aba/ abwaaaaa.a...../ 2211. 13231
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexan- der Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich soxt-to-image generation, 2022. URL https://arxiv httpg/abs/22006
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Sell-rewarding language models, 2024. URL http::::///xxx..rrr//////////// 10020
Junzi Zhang, Jongho Kim, Brendan O'Donoghue, and Stephen Boyd. Sample efficient reinforcement learning with reinforce, 2020. URL httL:ttt///arri..or//////00...00
A
Appendix
This supplementary document provides additional details to complement the main paper:
Additional experiment results
Additional qualitative results
Proofs
B Additional Experiment Results
In this section, we present additional evidence of the statistical significance of our results, as shown in Tables 5 All experiments were conducted over five randomly chosen seeds to ensure robustness. Furthermore, for Fig: we generated five images per prompt using different latent variables while keeping the prompts consistent.
and
Our model consistently outperforms its counterparts, producing images with significantly finer details, thereby demonstrating its robustness and effectiveness in generating high-quality outputs.
Among the evaluated visual metrics, we observe that SSIM consistently aligns with the visual improvements, we attribute this to the fact that it effectively measures fine-grained structural similarities. In contrast, RMSE and PSNR, being coarso-grained metrics, failed to account for the nuanced details in the generated images. These results highlight the importance of using metrics like SSIM for a more comprehensive evaluation of fine-grained image goneration tasks.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Dfteeee DFO
D30
BCC DPPO
Figure 6: Images are generated using the same prompts across 5 different latents, demonstrating that our models consistently outperform their counterparts across all cases.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Method
PickScore f
HPS 1
Aesthetic Score f
BLIP 1
ImageReward
CLIP 1
D3PO
20.306 + 0.016
0.227 + 0.001
5.050 = 0.005
0.470 + 0.001
-0.209 + 0.038
0.261 + 0.002
Diffusion-DPO
20.791 + 0.021
0.262 0.000
4.711 + 0.010
0.486 + 0.002
0.139 + 0.034
0.263 + 0.001
SPO
20.919 = 0.024
0.266 0.002
4.702 + 0.019
0.472 + 0.002
0.191 + 0.028
0.249 + 0.001
SEE-D3PO
20.590 = 0.017
0.247 0.001
4.851 = 0.016
0.491 = 0.001
-0.003 = 0.018
0.271 + 0.002
SEE- DIB-DiifusionDP
21.014 = 0.032
0.273 = 0.003
4.721 = 0.015
0.497 = 0.002
0.333 = 0.052
0.266 + 0.001
SEE-SPO
21.295 + 0.041
0.283 1 0.001
4.679 + 0.010
0.510 + 0.001
0.585 + 0.028
0.275 1 0.001
Table 4: Results are averaged over random seeds, with mean and standard deviation re- ported.All numbers are reported the irrdaatio-unqqque split of Pick-a-Pic V1 dataset. We evaluate CLIP Score, BLIP Score, Aesthetic Score, PickScore, HPSv2 Score
Method
RMSE T
PSNR 4
SSIM
E1
E2 f
D3PO
0.2455 + 0.0006
12.5599 + 0.0248
0.4190
0.0014
7.0853 + 0.0075
7.0911 0.0074
Diffusion-DPO
0.3736 = 0.0017
8.7047 0.0401
0.2406 0.0013
7.4954 + 0.0071
7.4945 0.0070
SPO
0.4446 = 0.0008
7.1218 0.0154
0.1666 = 0.0015
7.5674 0.0118
7.5635 + 0.0118
SEE-DiffusionDPO
0.4299 + 0.0005
8.1743
0.0095
0.2056 0.0025
7.6517 1 0.0059
7.6587 0.0060
SEE-D3PO
0.3052 = 0.0003
10.5240 = 0.0100
0.3183 + 0.0021
7.3650 1 0.0123
7.3692 + 0.0124
SEE-SPO
0.4383 + 0.0021
7.0818 0.0351
0.1575 0.0018
7.7120 + 0.0072
7.7101 0.0074
Table 5: Results are averaged over 5 random seeds, with mean and standard deviation re- All values are reported on the validation-uique split of the Pick-a-Pic-VV dataset. For each prompt, we first sample a base image and then sample 10 different images to gel all the diversity metrics. Evaluation metrics include RMSE, PSNR., SSIM, and entropy energy scores (E1 and E2).
C Proofs
C.1
Effects of ?
The parameter y serves to flatten the reference distribution. As seen in Eq. I6 the objective includes KL divergence term that penalizes deviation from the reference policy, which in turn can restrict the training policy from exploring potentially higher-reward regions. By flattening the reference distribution via higher 7: the training policy is allowed greater flexibility to explore wider range of actions. This effect is illustrated in Fig. 7 where the high-reward region is initially assigned very low probability under the standard reference policy, making it difficult for the training policy to reach it. it-resulting in slower learning. In contrast, with a larger 7, the reference distribution becomes more uniform, assigning greater probability mass to the high-reward region. This facilitates more effective exploration and accelerates training.
C.2 Diffusion-DPe DPO equivalence
We could also start from the formulation of Diflusion-DPO to get alternate version of our objective which is different only by a constant, we follow the same steps as of Diffusion-DPPO to arrive at our final objective:
max Eew Bew.D..a www.De.ennrrrrttureolllooo va) PI.. [po c)) Pref(Zror [c] Brypo ( log c) (18)
Further let r(c, zo) = (x) [R(c, No.T)), plugging this to Eq. 18
upper-bounding KL-divergence and
using jensen's inequality we get our objective:
L = w((e) (Iee "0e"""a(""" 1)1 -Ile' - -e--eaadd ca
((()-)))())))))))) 0)1(²)
L, = w(eel + +))) (I6"" emt(x " - -lle' - ever(a(,,)) 11)²)
19

---

Published in Transactions on Machine Learning Research (06/2025)
Gaussian and Gaussiantty in High Reward Region
3
Gaussian
Gaussian
2
1
Yaxiis
High reward region
-1
-2
-3
-3
-2
-1
0
1
2
3
X-axis
Figure 7: A toy example illustrating how incorporating encourages greater exploration, po- tentially guiding the model toward regions with higher rewards.
20

---

Published in Transactions on Machine Learning Research (06/2025)
[hinal S -Ri,e log - (Tew Ta))
Further w()) 1/o?, here of refers to the variance at timestep L. In most work Wallace al. 2023
Song
al. 2022 Ho et al. 2020 w(() is taken constant. Combining both we write final objective
Lau (1ee--entxz) 011 - -Ile' co(xt, 011²))
L (1 (1+1) 7) (I1E" crerlay 11)1 -llé' exee(1) () I1²)
1
T.nnal (0) S -B log (- -B I.)).
21


Reviews:
[{"review_id": "PPujKIgtB1", "reviewer": "Reviewer_ruq3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Conceptual Insight: The work introduces a regularization term – self-entropy maximization – into the RLHF objective for diffusion models. This idea directly targets the identified problem of reduced diversity. This is a simple yet insightful contribution.\n\n- Soundness: The modified objective (Eq. 17) is derived by augmenting the standard DPO loss with the entropy term. The authors show that this leads to an effective flattening of the reference distribution, which may force the policy/model to be more exploratory. The derivation appears sound and follows the DPO framework closely. Moreover, the approach is straightforward to implement, as it just adds an extra term to the existing KL-regularized DPO objective. It can be easily combined with any DPO-based algorithm without much overhead.\n\n- Performance: The experiments are well-designed to ensure fairness with baselines, cover both objective metrics and human evaluation, and demonstrate improvements due to the proposed method.\n\n**Weaknesses**\n- Incremental Novelty: The idea of adding an entropy-based regularization to encourage exploration, while effective, is somewhat incremental. In this paper, the novelty lies in recognizing the importance of such a term for diffusion models and integrating it into the DPO framework. This is a valuable contribution, but also a relatively straightforward extension of existing methods. From a novelty standpoint, it may be seen as an incremental improvement on prior art (D3PO, Diffusion-DPO, etc.) rather than a completely new paradigm.\n\n- Clarity: While the paper provides a correct derivation of the new objective, the theoretical treatment could be clearer. The notation and equations (particularly around Eq. (17)) are quite dense. Readers who are not already familiar with the DPO formulation for diffusion models might struggle to follow this part. The paper could do more to intuitively explain each step of the derivation. For example, the connection between adding the entropy term and effectively raising the reference distribution to a power of $1/(1+\\gamma)$ (thus flattening it) is mentioned but not deeply expanded on in the main text. Additionally, some background on how DPO is applied to diffusion (e.g. the role of “winner” and “loser” samples, or how the noise prediction terms relate to preferences) would help make the paper self-contained.\n\n- Limited evaluation scope: The experiments are focused on a single benchmark (Pick-a-Pic V1) and one base model (Stable Diffusion 1.5). It remains uncertain how well SEE-DPO would generalize to other settings. For instance, would the approach yield similar benefits on a different text-to-image dataset or with a different diffusion model (e.g. Stable Diffusion 2.x or another architecture)? This narrow scope means the generality of the conclusions is not fully tested.\n\n- Effectiveness of method on diversity: The qualitative results (Fig.3,  in the paper that aim to show increased diversity do not seem convincing. While the visual quality seems to be improved compared to the baselines, it does not seem to demonstrate increased diversity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a simple yet effective contribution to the field of generative model fine-tuning. It tackles an important problem – the tendency of RLHF methods to collapse model diversity and overfit to the reward – specifically in the context of diffusion models. The proposed solution, incorporating a self-entropy term into the preference optimization objective, is well-motivated. Empirical results indicate that this solution improves both quantitative metrics and qualitative outcomes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Improve theoretical clarity: While the core idea comes across, the mathematical derivation in Section 4 could be made clearer. One suggestion is to add a bit more step-by-step commentary around Equation (17). Moreover, the authors should include a short explanatory paragraph or a simplified toy example to illustrate how increasing entropy = flattening the reference distribution. \n\n- Broaden experimental evaluation: To strengthen the paper, the authors could evaluate SEE-DPO on additional model (e.g. Stable Diffusion 2.x, SDXL, or PixArt). This will confirm that the entropy regularizer consistently improves alignment and diversity across architectures.\n\n- Enhance figure readability: Redesign the training‑curve plots and figures (e.g. Figure 2, 4, 5) with larger fonts, clearer line styles or markers, and higher-contrast color choices so that key trends and comparisons remain legible even at reduced sizes.", "broader_impact_concerns": "The paper raises several ethical considerations that should be added in an explicit Broader Impact Statement section, such as misuse for synthetic misinformation and deepfakes, amplification of bias and stereotypes, and copyright issues.\nWe recommend the authors include a dedicated Broader Impact section that acknowledges these potential harms and outlines mitigation strategies.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5k7etIqwE1", "reviewer": "Reviewer_1Kva", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The problem to address is important and will have a lot of interesting applications.\n2. The authors provide some theoretical analysis and formulate their method as a self-entropy  regularization trick.\n3. The paper is generally well written. I can follow most of the things. And the related work part is comprehensive which details the literature contexts for me.\n\n**Weakness**:\n1. In Table 1, the SEE based methods have similar metric results to other baselines, like Diffusion-DPO. Could you please provide some explanation?\n2. Although I do not want to be picky, I cannot perceive notable visual improvements over baseline in Figure 3. For instance, for the last example, the fox generated by D3PO even presents more details than the proposed method. And the results of both methods are quite close in the first row.\n3. The visual results are quite limited. I can only see comparisons in Figure 3 and Figure 6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a self-entropy regularization mechanism to overcome the problems of overfitting and reward hacking for Direct Preference Optimization (DPO) methods. The authors present some results to show that their SEE-DPO method can improve DPO training with better network stability, result quality and image diversity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am not an expert in this area. To be honest, I have no idea of what requested changes the authors should have. So I will count on other reviewers' comments.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t4RRUzGL1O", "reviewer": "Reviewer_NzNp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n   * The proposed regularization is general and straightforward.\n   * It can be applied to multiple DPO variants, including D3PO, SPO, and Diffusion-DPO, and its effectiveness is demonstrated.\n\n**Weakness**\n   * While effective, entropy regularization is not a novel concept in RL.\n   * The experiments are limited to Stable Diffusion v1.5.\n   * Figure 2 is difficult to interpret based on the description on page 11. The text mentions an initial steep drop in reward values during both SPO and D3PO training phases, but the explanation is unclear. Could the authors please elaborate on how to properly interpret these plots?\n   * Please see the requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Self-Entropy DPO (SEE-DPO), a method designed to mitigate the reward hacking phenomenon during DPO for diffusion models by incorporating a self-entropy regularization term. They prove the equivalence of three DPO variants (D3PO, SPO, and Diffusion-DPO) and demonstrate that the proposed regularization can be applied to all of them. Experimental results validate the effectiveness of the regularization across these three variants, using Stable Diffusion v1.5 as the benchmark model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I recommend that the authors carefully proofread the paper and ensure formatting and text consistency. Some issues that need to be addressed include:\n\n1. Table 5 in Appendix B: the rows are out of order, could the authors please arrange them properly, matching the format of Table 4 for easy comparison.\n2. Page 6, first line: There should be a space before \"Consequently, we …\".\n3. Page 6, Equation (11): It is inappropriate to state \"derive our final loss function\" since this function was originally proposed by D3PO. Please clarify this.\n4. Inconsistent formatting in captions: Captions for Table 1, Table 2, Figure 4, and Figure 6 are bolded, whereas other figures and tables are not. Please ensure consistency.\n5. I think Figure 4 is not referenced anywhere in the text, please consider include a reference and explanation (maybe in first paragraph of page 11).\n6. Inconsistent figure notations: Some figures are labeled as \"Figure X\" while others use \"Fig: X\" (e.g., \"Figure 5\" vs. \"Fig: 3\"). Please standardize the notation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has several merits, including,
(1) clear problem motivation with broad applicability: the paper tackles a well-recognized limitation in RLHF for diffusion models—namely, model collapse due to overfitting and reward hacking. this problem is relevant and significant across a growing range of generative ai applications.
(2) novelty through self-entropy regularization: the proposed method introduces a self-entropy maximization term into the direct preference optimization (DPO) objective. this encourages broader policy exploration and mitigates over-optimization, resulting in more diverse and stable generation.
(3) theoretical soundness and generalizability:  the authors demonstrate a mathematically sound derivation of the modified loss function and prove equivalence across several DPO variants.

The authors thoughtfully addressed reviewer concerns, notably by re-running key experiments (e.g., Diffusion-DPO), clarifying theoretical derivations (Equation 17), including a toy example, enhancing figure clarity, and so on. Despite one reviewer still has the negative final score, the reviewers largely agreed that SEE-DPO is a meaningful, well-argued contribution. While the entropy regularization idea is not novel in general, applying it systematically and successfully within DPO for diffusion models—while empirically verifying its benefits—is a clear strength.
Decision: Accept (poster)