Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Authors: Yunhong Lu
Qichao Wang
Hengyuan Cao
Xiaoyin Xu
Min Zhang
Venue: ICML 2025 poster (2025)
Keywords: diffusion models
human preference
preference optimization
Primary area: applications->computer_vision
TL;DR: An efficient method designed to align diffusion models with varied human preferences.

Abstract: Direct Preference Optimization (DPO) aligns text-to-image (T2I) generation models with human preferences using pairwise preference data. Although substantial resources are expended in collecting and labeling datasets, a critical aspect is often neglected: *preferences vary across individuals and should be represented with more granularity.* To address this, we propose SmPO-Diffusion, a novel method for modeling preference distributions to improve the DPO objective, along with a numerical upper bound estimation for the diffusion optimization objective. First, we introduce a smoothed preference distribution to replace the original binary distribution. We employ a reward model to simulate human preferences and apply preference likelihood averaging to improve the DPO loss, such that the loss function approaches zero when preferences are similar. Furthermore, we utilize an inversion technique to simulate the trajectory preference distribution of the diffusion model, enabling more accurate alignment with the optimization objective. Our approach effectively mitigates issues of excessive optimization and objective misalignment present in existing methods through straightforward modifications. Experimental results demonstrate that our method achieves state-of-the-art performance in preference evaluation tasks, surpassing baselines across various metrics, while reducing the training costs.

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Yunhong Lu
Qichao Wang
Hengyuan Cao
Xiaoyin Xu²
Min Zhang 13
Abstract
several challenges remain, such limited text rendering capabilities (Chen et al., 2023a), unrealistic spatial layouts (Lin et al., 2024) and improper illumination (Ren et al., 2024). Current approaches to addressing these challenges primarily focus on optimizing training strategies (Esser et al., 2024), improving network architectures (Peebles & Xie, 2022), augmenting datasets (Gadre al., 2023) and incorpo- rating richer semantic conditions (Chen al., 2024; Pernias et al., 2023). Nevertheless, these improvements typically necessitate retraining the models from scratch, making them unsuitable for enhancing pre-existing models. Inspired by reinforcement learning with human feedback (RLHF) meth- ods used in large language models (LLMs) (Wang et al., 2024), aligning T2I models with human preferences has emerged as a promising direction to improve model perfor- mance (Liu al., 2024), though it is currently under active exploration and holds significant potential.
Direct Preference Optimization (DPO) aligns text- to-image (T2I) generation models with human preferences using pairwise preference data. Al- though substantial resources are expended in col- lecting and labeling datasets, a critical aspect is of- ten neglected: preferences vary across individuals and should be represented with more granularity. To address this, we propose novel method for modeling preference distribu- tions to improve the DPO objective, along with a numerical upper bound estimation for the diffu- sion optimization objective. First, we introduce smoothed preference distribution replace the original binary distribution. We employ reward model to simulate human preferences and apply preference likelihood averaging to improve the DPO loss, such that the loss function approaches zero when preferences are similar. Furthermore, we utilize an inversion technique to simulate the trajectory preference distribution of the diffusion model, enabling more accurate alignment with the optimization objective. Our approach effec- tively mitigates issues of excessive optimization and objective misalignment present in existing methods through straightforward modifications. Experimental results demonstrate that our method achieves state-of-the ofetherrrt performance in prefer- ence evaluation tasks, surpassing baselines across various metrics, while reducing the training costs.
Current research primarily focuses on two directions. The first involves collecting human-labeled preference images to train models (Lee et al., 2023b; Liang et al., 2023; Dai et al., 2023). These datasets typically assume binary pref- erence distribution (Kirstain et al., 2023), which not only requires substantial human resources but also neglects the inherent variability of human preferences, often resulting in excessive optimization. The second direction focuses on designing methods to fine-tune T2I models using human feedback data. Many previous approaches propagate reward maximization through reward models (Clark et al., 2023; Prabhudesai al., 2023), leading to reward hacking. Alter- natively, some approaches model the sampling process of diffusion models as a Markov chain and employ reinforce- ment learning (RL) objectives based on reward feedback (Black et al., 2023; Fan et al., 2023; Zhang et al., 2024b; Li et al., 2024b). However, these methods require extensive online evaluations and suffer from training instability. More- over, DPO methods (Wallace et al., 2023; Yang et al., 2023; 2024) optimize the RL objective toward the optimal trajec- tory strategy for diffusion models. Nevertheless, estimation based on the forward process often results in misalignment between the optimization objective and the desired outcome.
Introduction
T2I diffusion models (Rombach et al., 2021; Podell et al., 2023) have recently gained widespread popularity. However,
College of Computer Science and Technology, Zhejiang Uni- versity. *College of Biomedical Engineering and Instrument Sci- ence, Zhejiang University. "Shanghai Institute for Advanced Study-Zhejiang University. Correspondence to: Min Zhang <min_zhang zju.edu.en>>
To address these issues, we propose an adaptive (with image- dependent loss), and efficient approach for preference align- ment of T2I diffusion models. Our insight is that, within
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
ROME PRNNT
SDXL
SmPO-SDXI
DONT PANIC
"A book with the words 'Don't Panic!' written
"Baby quokka in teacup, photography, minimalistic, 8k."
"A pineapple with one beer to its left and two beers its right."
"A cute cartoon giralle toy wearing ballet skirt, dancing..."
"Beautiful rose with dew, lens focused on rose....
We introduce AmPO-Diffusioon designed to address the variability of human preferences, as an adaptive and efficient approach to aligning diffusion models with human preferences. The images in the first row are generated by the SDXL model, while the second row presents images generated by the SDXL model fine-tuned using SmPO-Diffusion (denoted as SmPO-SDXL.). The results demonstrate that SmPO-SDXL. generates higher-quality images, with significant enhancements in spatial layouts, text rendering, proper illumination and visual appeal.
Figure I: Enhancement over Vanilla SDXL.
the diffusion model framework, existing methods exhibit significant inaccuracies in both modeling human prefer- ences and their estimation techniques. Thus, we introduce SmPO-Diffusion for modeling preference distributions to improve the DPO loss function, along with a numerical upper bound estimation for the diffusion optimization ob- jective. To this end, introduce two key contributions:
of-the-art baselines for human preference alignment. Our approach outperforms existing methods across multiple hu- man preference evaluation metrics (refer to Table 2) and demonstrates significant improvements in image generation quality (examples shown Figures 1, 3 and 4) and training efficiency (26 faster than Diffusion-KTT per Table 1).
Smoothed Preference Modeling. As the adage goes, "Beauty is in the eye of the beholder." Given the inherent variability in human preferences, we propose the use of smoothed preference distributions to replace binary prefer- ences. This approach mitigates label bias by integrating the average likelihood of preferences into the DPO loss func- tion, the loss function approaches zero when preferences similar. In practice, method eliminates the need for manual annotation by automatically generating smooth preferences through reward model, significantly reducing data collection costs and improving adaptability.
2 Related Works
Text-to-Imgae Generative Models. Early research utilizes
GANs for T2I generation (Reed et al., 6;16; Karras et al., 2018), while diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021) and flow matching (Liu et al., 2022; Lip- man al., 2022: Albergo & Sanden-Eijnden, 2022) have become the dominant approaches for image synthesis. De- spite the ability of Stable Diffusion models (Rombach et al., 2021; Podell et al., 2023; Esser et al., 2024) produce high- quality images, these models are generally trained on large, noisy datasets, which can lead to results that are contrary to human intent. Our work explores the effectiveness using synthetically preference dataset (Kirstain al., 2023) im- prove pre-trained T2I generative models through techniques that leam from human/Al feedback.
Optimization via Renoise Inversion. In addition, to accu- rately estimate the trajectory preference distribution of the diffusion model, employ the Renoise Inversion method for estimating the sampling trajectory. This approach places the forward-process-bon based estimation used in (Wal- lace et al., 2023). effectively addressing the issue of objec- tive misalignment and facilitating more efficient fine-tuning.
Preference A Alignment of Diffusion Models. Inspired by RLHF methos in LLMs, text-image reward models (Schuh- mann et al., 2022; Radford et al., 2021a; Wu et al., 2023; Zhang et al., 2024a; Xu et al., 2023; Wu et al., 2024) have been increasingly developed and applied for fine-tuning
Extensive experiments demonstrate the advantages our contributions. We compare SmPO-Diffusion with state-
2

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
T21 generation models. With these reward models, DRaFT (Clark et al., 2023) and AlignProp (Prabhudesai al., 2023) update the diffusion model's sampling trajectory using dif- ferentiable reward propagation, while DPOK (Fan et al., 2023) and DDPO (Black et al., 2023) regard the sampling process as a Markov decision process and apply policy gra- dient methods for training. However, these RL techniques require resouree-intensive training objectives and are prone to reward hacking. To address these issues, Diffusion-DPO (Wallace et al., 2023) and D3PO (Yang et al., 2023) fine- tune the diffusion model for the unique global optimal pol- icy using DPO techniques, while DDIM-InPO (Lu et al., 2025) uses DDIM Inversion to align specific latent variables, leading to a series of variants (Yang et al., 2024; Li et al., 2024a; Hong et al., 2024b; Lou et al., 2024; Gu et al., 2024). However, the flawed design of human preferences in the dataset often results in over-optimization and low training efficiency for these methods. Differently, our method inves- tigates more accurate modeling and estimation of human preferences.
zi or loser ag based on human preferences. The Bradley-
Terry (BT) (Bradley & Terry, 1952) model defines pairwise preference follows:
PBT > ædjcc) o (1(25) c) r( c))
(3)
where (-)) is the sigmoid function and r r(zz: c) is reward model which can be parameterized by neural network and trained with maximum likelihood estimation (MLE).
Building on this, (Rafailov et al., 2024) design the following DPO loss function and demonstrate its equivalence a reinforcement learning (Sutton & Barto, 2018) process (such as PPO REINFORCE) with explicit reward model:
(= [c) Piol (2% [c) -Blog
(a( (c) Pret (oet(at (x))
LDPO = Laay an log (8log
(4)
where (rer(a)le) is the reference distribution and / is a
hyperparameter that governs regularization.
DPO for diffusion models. For diffusion models, since directly computing the image's probability distribution p(æā(e) is not feasible, Wallace al. (2023) propose method for optimizing the upper bound of the original DPO objective function and derive a differentiable objective:
3 Preliminaries
Diffusion Models. Diffusion models (Nichol & Dhari- wal, 2021) consist of a forward process g(wo:T) that pro- gressively adds noise to data ~ g(2o), and a learned denoising process po (ForT) to reconstruct the data from Po(or) ~ I). The forward process is defined as a Markov process g( -1) = (a;;; -a (I), where ay is the noise schedule. The denoising process takes the form of po(x; p= (ar)) = N(x,, 1: (2+)), (x:)). The training objective is to maximize the variational lower bound (VLB) associated with the parameterized denoiser cq::
LDPO-Diffusion (0) = - -E(ay .ab,c)/D log
log p((xx):)) pf (2
log po(c))r)) Diet (at
B by o Paler ( pp
where Pz(-) represents pe(-Ic) for compactness. They esti-
mate Equation (5) based on the following formula:
[( CDM = Exo, (A(t) [|(t) -
[ (1)
[())- = (Baag ml c)-D,, log (02()
c)))
where 2y ~ q(z0), xt ((x;) (20)= = Na Vaiivvata varzo, (1- I.).). ā. Im Og and A(t) is pre-specified positive weighting function (Ho al., 2020; Song et al., 2020b).
(6)
Here the score function si is defined as:
st s/ c) = ||e" - c/ ( c)ll2 - ||e" cte c) (7)
Denoising Diffusion Implicit Models (DDIM) (Song et al., 2020a) formulate the denoising process as an ODE, de- terministically generating data. The discrete form of the process can be expressed as:
where e* (O, and x. Varag v1 -a,
4 Methodology
/ ai x1-1 x: V ar41
(V
11- Ote
a1+1
(x141). Cp
In this section, we introduce our SmPO-Diffusion for align- ing diffusion models with human preference. Our moti- vation is to enable more precise modeling and estimation of human preferences within the framework of diffusion preference optimization. First, to account for the inherent variability in human preferences, we employ a smooth pref- erence distribution for modeling (Section 4.1). Furthermore. we estimate the trajectory preference distribution of the dif- fusion model through Renoise Inversion (Section 4.2). The overview of the
0u1
(2) This proccess can also be reverse-integrated, starting from noise-free image Ty to estimate æ; at any time step which is known as DDIM Inversion (Mokady et al., 2023).
Direct Preference Optimization. Given a predefined hu- man preference dataset D = {(c, xx x))), each sample consists of a prompt e and a pair of images generated by a reference model, with each pair labeled as the winner
is illustrated in Figure
3

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Step1 :Smotthd Preference Modeling
Prompt: A 3d C6I burger icon.
0.7
0.7
Expert Rating
Human
x
Voting
0.3
222
0.3
p(x% > x6(c) = 0.7 y
Normed
Softhaa
0.7
Reward Model
xb
Collected Data
0.3
p(x0 < xo|c)
0.3
Y
OpiiiaOpt via Renoise Inversion
Maximize smoothed likelihood on Winner Minimize smoothed likelihood on Loser
D
0.7
t
Renoise
X
Text-to- Diffusion Model
0.3
x
Dataset
Prompt:
3d
CGI burger icon. + X
Inversion
1
Random sampling
Inversion
RT
Figure 2: Overview of SmPOODiffusion. We present two steps of SmPO-Diffusion: (1) Smoothed preference modeling. We calculate smoothed preference labels for all image pairs in the dataset. Unlike traditional methods such as expert ratings and human voting, we employ reward model (PickScore) to assign scores to preference pairs. These reward scores are subsequently normalized and processed using a Softmax function to derive the final smoothed labels. (2) Optimization via Renoise Inversion. To estimate Equation (10), we use Renoise Inversion (repeat a few steps of Equation (13) for Inversion, and perform one step of Renoise according to Equation (14)) to estimate the diffusion model sampling trajectory, maximizing the smoothed oggliikelhoodd of preferences to diffusion models. The final loss function is formulated as Equation (15).
4.1 Smoothed Preference Modeling
with p(-lc), yields the following training objective:
When sampling any pair of images from the dataset D, in- dividual preferences exhibit significant variability. In this case, we employ smooth distribution model hu- man preferences (Furuta et al., 2024). Assuming that the datasets the winners 2. and losers abb are sampled from a smoothed probability density p(-lc), we model the complex probability distribution using weighted average:
(o(x))) (Blog
(a(at))
CSmPO = (o ag log
B log
pr. (f.((()))
Dict (xrb)
ps (3x))
pd (500))
-F mb log ())(1) .e/uD
log
proc (xx))
Pfat (2rb)
(9)
Equation (8) represents of the options for regulariza-
tion design. If we use binary labels, i.e., 1, 1,
Equation (9) reduces to the original form Equation (4).
j(x)'e (c) p(x1'1 (az"'())'pp ())pp(()))))))))))())( T
Modeling for diffusion models. Following (Wallace et al., 2023), we redistribute the rewards r(æ0, e) over all poten- tial diffusion trajectories pe (#1.7) 2o. c), with the goal of minimizing the KL-divergence between the joint probabil- ity distributions D& (po (Zoer c) (Zarrlc)), Thus, arrive at the following objective for diffusion models:
7%
(8)
p(z//c) = (==(()))~=p((b)) c)
(c)
where factors (c) = p(a p(al)c)) and Z/1 (c) (c)==E;; p(a""ee""~p((rjpp (e)c)a are for normaliza- tion. Since these values are challenging to estimate, the normalization term is approximated
-SmPO-Diffus = -E(ay wb.c)// log o (2a - 77
constant and omit- ted. Here, the weighting factor balances the relative con- tributions of the winner and loser distributions, while the sensitivity factor governs the sensitivity of the mixture distribution to variations in parameters. Weighted averaging serves as a smoothing technique, enabling effective modula- tion of the likelihood scale. Replacing p(: ((((c) Equation (4)
3 pp lag)) x o p8 (wl.tk)))
log
log
pfe (x pf p
(10)
When preferences are more similar, meaning humans find it harder to differentiate image quality, the loss function decreases further; otherwise, it increases. This modeling

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
approach more accurately reflects human preferences. To achieve this goal, how should and y be formulated? In our work, we define the weight-to-sensstivity ratio a//y as the probability of the winner p(æ"" > eylc), ensuring better alignment with human preferences.
steps are taken, which results in time-intensive inversion process that is detrimental to improving training efficiency. Therefore, inspired from (Garibi et al., 2024), we employ few-step DDIM Inversion (fewer than 10 steps), followed by an additional single ReNoise step:
Smoothed Preference of Reward Model. Inspired from prior works (Lee et al., 2023a; Black et al., 2023; Fan al., 2023), we adopt the text-image reward model as a reliable alternative expert rating human voting. It is because AI reward models demonstrate strong consistency with human preferences, a concept commonly used in the RLHF litera- ture. To produce smooth preference labels, reward scores are computed for all image pairs, with the higher-scoring im- age selected as the winner. Specifically, we feed the prompt and image Vū into a reward model r derive reward scores r r c). Given D, we obtain set of reward scores, denoted as Rp- = (r(arg) c), r raath c)) .))e)) To achieve a balanced probability distribution, we normalize the reward score as described below:
(V--o
ct V
21-1 +
an-1
c e).
at-1
at:
at-1
(14)
Following (Wallace et al., 2023), once we obtain D, and we can estimate Equation (10) in the following manner:
(()) EE log ( B) c) (2/g c)))
(15)
where score function is defined as:
3 (z). c) = ||, - ( -11 - Soe (#t, 1) (16)
where T.¹ = ( - // //IVVV=/ 04. This loss function drives denoising zü at point "."" to improve more signifi- cantly than denoising denoising 2 at point ă². Unlike the random noise addition process in Diffusion-DPO, this ap- proach enables fine-tuning variables highly correlated with the image, thereby enhancing training efficiency.
r( ( c) - max (RD)
r' r c)
(11)
max (Rx) min (RD)
For various preference data pairs, the weight-to-sensstivity ratio a/y is modeled as:
5 Experiments
exp(r' c)) (a 10 arb)= exp(r'(xl) c)) + exp(r'(xt)) c))
5.1 Setup
(12)
Implementation Details. Our method is applied to align the Stable Diffusion 1.5 (SD1.5) and Stable Diffusion XL-base- 1.0 (SDXL). We utilize the Pick-a-Pic v2 dataset, which contains 851,293 data pairs and 58,960 unique text prompts as (Wallace et al., 2023). We employ Adam (Loshchilov, 2017) as the optimizer for SD1.5 and Adafactor (Shazeer Stem, 2018) for SDXL. All experiments are performed 8 A800 GPUs, with each GPU handling batch size of data pair. Through 128 gradient accumulation steps, an batch size of 1024 data pairs is achieved. The learning rate is set to 2000 2.048-$, 8 with a linear warm-up phase. For SD1.5, 3 3 is set to 2000, while for SDXL, , is set to 5000.
In our work, we employ PickScore (Kirstain et al., 2023) as the reward model. To control the effect of weighting factor on the fluctuations of the loss function, we set sensitivity parameter 7 to fixed value.
4.2 Optimization via Renoise Inversion
To perform optimization on Equation (10), we need to sam- ple æ1.r ~ ((a):rr/a)))) Given that the sampling pro- cess is intractable, previous methods (Wallace et al., 2023; Hong et al., 2024b) replace the reverse process with the for- ward noisy process g (z)) Since the noise is randomly drawn from a Gaussian distribution, it results in inaccurate estimation of the loss function. We consider this to be the primary cause of the observed inefficiency in training.
Evaluation. We compare SmPO-Diffusion with existing baselines across three dimensions: automatic preference metrics, user studies, and training resource consumption. In this work, we compare the following baseline methods: Supervised Fine-Tuning (SFT), Diffusion-DPO. Diffusion- KTTO (for SD1.5), and MaPO (for SDXL). These baselines share a common focus on human-aligned image genera- tion but differ fundamentally in their technical mechanisms: Diffusion-KTO adopts the Kahneman-T Tversky model to represent human utility instead of maximizing the log- likelihood of preferences. MaPO jointly maximizes the likelihood margin between the preferred and dispreferred datasets and the likelihood of the preferred sets, learning preference without reference model. To ensure a fair com- parison, all models are fine-tuned on the Pick-a-Pic v2
In order to improve the estimation of Equation (10), we apply an Inversion technique to evaluate the diffusion sam- pling process y ((x1:)(a)))) (a0o). Given image Da to compute at any time step f, we first apply DDIM Inversion (Mokady et al., 2023) iteratively to derive an approximate estimate:
(1- a ate
x1-1-
Eg
( c).
0c-1
C4-1
(13)
Here, we denote the estimate of x, obtained as b.. However, Equation (13) assumes that sufficient number of inversion
5

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
SmPO-SDXL MaPO-SDXL DPO-SDXL SDXL+Refiner
STOP
STO
STOP STPP
STOP
...., 8k portrait render, ungel wings, be.be tiful lighting....
rich daddy, wear grey suits and sunglasses...
"Three cats standing next to an orange cat."
"A large upple with two small green word 'stop" printed in red on it.
"A towel with the
"A heart made of water."
upples on its left."
Figure 3: Qualitative comparison. We provide qualitative comparison of SmPO-Diffusion and other different preference optimization methods (Refiner. Diffusion-DPO. MaPO) for SDXL. The results indicate that our model generates the highest quality images, including aspects such as llumination and spatial composition, text rendering, visual appeal, and others.
dataset. To comprehensively assess quality of image gener- ation, we use five reward evaluators: CLIP (Radford et al.,
Table 1: Computational cost comparison. We report the NVIDIA A800 GPU hours required for training our SmPO- Diffusion and the baselines SDXL and SD1.5
2021a) for text-image alignment, LAION aesthetic classi- fier and Imagereward (Xu et al., 2023) for image quality, PickScore (Kirstain al., 2023) and HPSv2.1 (Wu al., 2023) for simulating human preferences. During testing, im- ages are generated using the Parti- Prompts (1632 prompts) (Yu et al., 2022) and HPDv2 (3200 prompts) (Wu et al., 2023) text test sets with the same seed. We use the median of reward scores and the win-rate as automatic preference met- rics, where the win-rate indicates the frequency with which the reward evaluator prefers images from SmPO-Diffusion those from the baselines. Additionally, we compare the GPU hours required for training across different methods.
Model
GPU Hours +
Model
GPU Hours 4
DPO-SDXL
~ -.96.00
DPO-SDI.5
~ 204,8
MaPO-SDXL
834.4
KTO-SDI.5
1056.0
SmPO-SDXL 150.8
SmPO-SDI.5
41.3
illustrated in Figure 1, SmPO-SDX improves the quality of generated images while achieving better alignment with human preferences, and also resolves some failure cases. Additionally, as demonstrated Figure 3, compared to other advanced preference optimization methods for SDXL (Refiner, DPO-SDXL, MaPO-SDXL)), our SmPO-SDXL produces images of the highest quality, with notable hancements in illumination, spatial composition, text ren- dering, visual appeal, and other key aspects. These hidden advantages lie within human preferences, further validating the effectiveness our method.
5.2 Primary Results
Qualitative results. Figures and present the qualita- tive comparison results of our SmPO-Diffusion and other baselines used for fine-tuning base SDXL model. First,
6

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Table 2: Quantitive comparison. We employ the HPDv2 and Parti-Prompts test sets to generate images using each SDXL and SD1.5 model. By performing automatic evaluation each reward evaluator, we report the median reward score (Score) of the images generated by all models and show the win-rate (%) of our SmPO-Diffusion compared to the corresponding baseline models (WR). Higher scores and win-rates demonstrate the superiority of our method. In the Score column, the highest value is displayed in bold, the second highest is underlined. Additionally, win-rates exceeding 50 % highlighted.
HPDv2 (3200 prompts) ImRewardf
Parti-Prompts (1632 prompts)
PickScore?
HPSv2.11
Aesthetict
CLIP
PickScoref
HPSv2.1
ImReward
Aestheticf
CLIP
Model
Score
WR
Score
WR
Score
WR
Score
WR
Score
WR
Score
WR
Score
WR
Score
WR
Score
WR
Score
WR
SDXL
22.75
88.1
28.45
94.5
0.881
79.3
6.114
64.7 38.36
56.2
22.63
86.2 27.45
92.6
0.929
81.9
5.753
74.0
35.53
55.6
SFT
22.17
95.4
28.39
90.3
0.756
79.5
5.989
72.8
37.66
62.9
21.98
95.2
27.00
91.1
0.660
83.7
5.705
74.2
34.77
64.1
DPO
23.13
77.2
30.06
86.7
1.184
62.9
6.112
67.7
38,86
50.5
22.93
77.2
28.89
82.5
1.280
67.8
5,813
70.6
36.30
51.5
MaPO
22.81
86.3
29.11
90.8
1.224
60.5
6.309
47.6
38.17
58.1
22.83
80.2
28.47
84.9
1.269
63.5
6.012
47.3
35.24
60.7
SmPO
23.62
32.53
1.331
6.264
38.88
23.35
30.73
1.429
5.959
36.34
SD1.5
20.83
88.0
23.61
-0.078
85.9
5.390
81.5
34.71
69.3
21.40
74.5
24.97
89.2
0.121
77.3
5.355
76.4
33.11
63.1
SFT
21.64
75.8
28.60
65.7
0.738
62.1
5.725
63.3
36.24
61.4
21.77
65.6
28.17
62.3
0.702
59.8
5.592
55.9
34.07
59.5
DPO
21.29
79.1
25.11
89.5
0.195
80.1
5.530
72.7
35.58
64.6
21.62
66.5
25.79
84.9
0.386
71.2
5.442
70.2
33.54
59.7
KTO
21.54
73.4
28.28
66.8
0.706
61.9
5.692
65.1
35.92
64.5
21.77
65.8
28.05
64.5
0.677
59.9
5.547
62.8
34.11
59.0
SmPO
22.08
29.31
0.885
5.831
37.19
22.00
28.81
0.911
5.636
34.72
Control
SDXL
DPO-SDXL
MaPO-SDXI.
SmPO-SDXL
Prompt
majestic pyramid center of a vast desert, with golden light from a setting sun, The surrounding dunes stretch andlessly, realistic and detailed.
Canny
Close portrait painting a bearded wilderness man, ultra realistic, concept art, intricate details, scrious, highly detailed, photorealistic, render, 8k, unreal engine.
Depth
Figure 4: Conditional generation comparison. SmPO-SDXL produces images with the highest quality (ef. Section 5.3).
Computational cost. illustrated in Table SmPO- Diffusion requires around 150.8 and 41.3 A800 GPU hours to train SDXL and SD1.5, respectively. These include 5.3 A800 GPU hours of PickScore reward model train- ing. In comparison, fine-tuning SDXL and SD1.5 with Diffusion-DPPP demands 976 and 204.8 GPU hours, re- spectively. The results demonstrate that SmPO-SDXL and SmPO-SD1.5 achieve significant improvements in gener- ation quality while reducing training time only 15.5% and 20.2% of that required by DPO-SDXL and DPO-SDI respectively. Compared to the latest methods, SmPO-SDX requires only 18.1% 1% of the training time of MaPO-SDXX and SmPO-SD1.5 requires merely 3.9% of the training time of KTO-SDI.5, resulting in 26-fold efficiency improve- ment. We attribute this remarkable efficiency gain to the more precise modeling and estimation of human preferences,
consistent with our motivation.
Quantitative results. Table 2 presents the reward scores of our SmPO-aligned diffusion models and baseline mod- els, along with the win-rates of our model compared to basclines. Overall, our SmPO fine-tuned SDXL and SD1.5 models demonstrate superior performance over the base- line models on nearly all test datasets and reward evalua- tion metrics. For instance, on the HPDv2 set, the me- dian HPSv2.1 scores for SmPO-SDXL and SmPO-SD1.5 reach 32.53 and 29.31, respectively, reflecting significant im- provements of +4.08 and +4.7 compared to Base-SDXI. and Base-SDI.5. When compared to tlate-of-the-ar baselines, the HPSv2.1 metric reveals that SmPO-SDXL achieves an 86.7% win-rate against DPO-SDXL on the HPDv2 test set, while SmPO-SD1.5 achieves 66.8% win-rate against KTO-

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Table 3: Enhancements from the proposed method modules.
Table 4: Impact of maximum DDIM Inversion steps.
PickScoret
HPSt
ImRewardf
Aesthet
CLIP1
InvStep
PickScoret
HPSt
ImReward
Aesthet
CLIPt
G-H.
DPO
21.29
25.11
0.1947
5.530
35.58
2
22.06
29.22
0.8738
5.828
37.01 ~ 27.8
Inversion
21.72
28.71
0.7608
5.748
36.41
4
22.06
29.23
0.8764
5.822
36.92 ~ 32.3
Renoise
21.87
29.01
0.7778
5.786
36.73
9
22.08
29.31
0.8854
5,831
37.19
~ 41.3
+Smoothed
22.08
29.31
0.8854
5,831
37.19
19
22.11
29.50
0,8964
5.844
37.13 67.3
Table 5: Impact of sensitivity factor yy and regularization B.
Table 6: Impact of CFG during DDIM Inversion.
(v,8)
PickScoret
HPSt
ImReward
Aesthet
CLIPT
InvCFG
PickScoret
HPS
ImReward
Acsthet
CLIPT
G-H4
(2,2000)
21.71
28.50
0.7746
5.712
36.41
-7.5
21.82
29.06
0.7772
5.766
36.51
70.3
(5,2000)
21.94
28.57
0.7716
5,812
36.79
-5
21.89
29.12
0.8047
5.776
36.77
~ 70.3
(20,2000)
21.96
29.27
0.8542
5.798
36.93
-1
21.79
29.27
0.8310
5.753
36.62
70.3
(10,2000)
22.08
29.31
0.8854
5,831
37.19
0
21.78
29.22
0.8311
5.734
36.51
41.3
(10,1000)
21.98
28.69
0.8313
5.823
36,97
1
22.08
29.31
0.8854
5,831
37.19
41.3
(10,3000)
21.95
29.13
0.8504
5,806
37.06
5
21.83
29.12
0.7972
5.741
36.44
~ 70.3
(10,5000)
21.90
29.12
0.8312
5.795
36.86
7.5
21.81
29.01
0.7931
5.752
36.38
70.3
SD1.5, Similar trends are observed across other metrics, further substantiating the superiority of our approach.
Table 7: Impact of weight-to-sensitivvn ratio using different reward models. We report the performance comparison where rows represent different training signals and columns denote evaluation metries. Upward arrows (1) in column headers indicate higher values are preferable. The highest value is displayed in bold, the second highest is underlined.
5.3
Conditional generation results
Without the need for further training, the model can be di- rectly applied to conditional generation tasks. We employ ControlNet to provide conditional control, which has been jointly pre-trained trained with the Base-SDXL model. As shown in Figure 4, we use canny map and depth map as additional con- ditions to control the T2I generation process. Experimental results demonstrate that the generated images successfully retain the strengths of the
Reward PickScore
PickScoret
1PSv2.11
ImReward
Acsthe
CLIPT
22.08
29.31
0.8854
5,831
37.19
HPSv2.1
21.87
29.50
0.8891
5.790
36.81
ImReward
21.77
28.98
0.8638
5.766
36.71
Aesth
22.01
29.26
0.8321
5.958
36.16
model.
CLIP
21.78
28.77
0.8428
5.759
37.08
5.4 Ablation Studies and Analysis
Experimental results indicate that configuring the step count to 19 achieves optimal image quality. However, this con- figuration substantially escalates the demand for training resources. To balance quality and efficiency, we fix the inver- sion steps at Furthermore, classifier-free guidance (CFG) also plays significant role in determining the efficacy of the inversion process. Mokady et al. (2023) point out that DDIM inversion is sensitive to the prompt c. Table 6 reports the influence of CFG during DDIM Inversion on the results, with experimental results demonstrate that optimal perfor- mance achieved when CFG=1. Additionally, we set the sensitivity parameter 7 to 10. As shown in rows to 5 of Ta- ble 5, if y is too small, the model is insensitive to changes in the reward score, resulting in suboptimal optimization. Con- versely, setting 7 too high may cause vverooptimization of the reward model. Moreover, we set 3 to 2000. Rows 5 to 8 of Table 5 reveal that if /3 is set too low, the diffusion model degenerates into a pure reward scoring model, while if set too high, the KL divergence penalty term overly restricts the model's flexibility during the adjustment process.
Tables 3 to 7 present the results of our ablation experiments on SD1.5. We report the median score of the reward eval- uators on the HPDv2 test set across different experimental configurations, along with an analysis of the results.
Proposed Method Enhancements. The core of our method lies in estimating the diffusion sampling process using Renoise Inversion and employing smooth preference modeling approach. As demonstrated in Table 3, estimating the latent variable a, via DDIM Inversion, followed by cor- recting 2, with an additional step of Renoise, significantly improves the model's 's performance. This highlights the criti- cal importance of accurately estimating the diffusion sam- pling process for diffusion models alignment. Furthermore, the integration of varied human preferences through smooth preference modeling substantially improves the quality of T21 generation. These two components comprehensively validate the effectiveness of our proposed method.
Parameter Selections. Table 4 illustrates the influence of the maximum DDIM Inversion steps required on results.
Choice of Reward Model. As shown in Table 7, train- ing with text-aware preference estimation models, such as
8

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
PickScore and 1PSv2.1, enhances both the visual appeal and the text rendering capabilities of images. However, other reward models prioritize enhancing specific performance metrics, such as aesthetic optimization, which improves the corresponding model capabilities but sacrifice some of the text rendering ability. We note that PickScore can be viewed as pseudo-labels from the Pick-a-Pick dataset, equivalent to cleaned data, making more suitable as a reward model.
References
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin. O., Üstün, A.. and Hooker, S. Back basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
Albergo, M. S. and Vanden-Fijnden, E. Building nor- malizing flows with stochastic interpolants. ArXiv, 00s/2209.15571,200
Conclusion and Discussion
In our work, recognizing the variability of human prefer- ences, we introduce SmmPO-Diffusion, a novel method for modeling preference distributions with a numerical upper bound estimation for optimizing the diffusion DPO objec- tive. Initially, we replace the binary distribution with smooth preference distribution modeled using a reward model. Additionally, we apply Renoise Inversion to es- timate the trajectory preference distribution. Experimental results demonstrate that our method achieves SOTA perfor- mance on various human preference evaluation tasks while significantly reducing consumption of training resources. We posit that integrating preference alignment into online learning represents a promising future direction, enabling models to undergo continuous performance enhancement.
Azar, M. G., Guo, Z. D.. Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Bao, F, Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669-22679, 2023,
Bao, F. Xiang, C.,, Yue, G., He, G., Zhu, H., Zheng, K., Zhao, M., Liu, S., Wang, Y., and Zhu, Vidu: highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024.
Limitations. In this paper, we utilize the Pick-a-Pic v2 dataset for model training and evaluation. While this dataset provides a diverse range of image-text pairs, it is impor- tant to acknowledge that it may contain harmful or biased content, which could inadvertently influence our model's outputs. Specifically, we observe that certain societal biases within the sataset-such as gendered stereotypes- may lead the model to generate overly feminized representations in response to neutral or non-gendered prompts. This potential bias necessitates careful consideration during both training and inference to mitigate unintended consequences.
Black, K., Janner, M., Du, Y., Kostrikov, I,, and Levine, S. Training diffusion models with reinforcement learning. ArXiv, abs/2305. 133001 2023.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231.155127, 2023a.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575 2023b.
Acknowledgments
This work was supported by the National Major Science and Technology Projects (the grant number 2022ZD01 17000) and the National Natural Science Foundation of China (grant number 62202426).
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. While sharing the same abuse poten- tial as conventional preference alignment methods where malicious actors could train harmful T2I models by aligning with unethical preferences- our approach's 's negative effects can be substantially mitigated through rigorous dataset au- diting and preference source monitoring.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng. C., Wang, R., and Ramesh, A. Video generation models as world simulators. httpppaaa....//ee/eeet////// rewarrhhssechh///
2024. URL
-s -imulators.

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Chen, J., Huang, Y., Lv, T., Cui, L., Chen, Q., and Wei, F. Textdiffuser: Diffusion models as text painters. ArXiv, abs/2305. 10855, 2023a.
Furuta, H., Lee, K.-H., Gu, S. S., Matsuo, Y., Faust, A., Zen, H., and Gur, I. Geometric-averaged preference optimization for soft preference labels. arXiv preprint WrXiv:240900669/ 2024.
Chen, Yu, J., Ge, C., Yao, I.,, Xie, E.,, Wu, Y., Wang, Z Kwok, J., Luo, P., Lu, H., et al. Pixart-\aa: Fast training of diffusion transformer for photorealistic -omagge-mmae synthesis. arXiv preprint 2XXv:23310000226 2023b.
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smymiss G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang. J., Orgad, E., Entezari, R., Daras, G., Pratt, S., Ramanujan, V., Bitton, Y., Marathe, K., Mussmann, S., Vencu, R., Cherti, M., Krishna, R., Koh, P. W. Saukh, O., Ratner, A. J., Song, S., Hajishirzi, H., Farhadi, A., Beaumont, R., Oh, S., Dimakis, A. G., Jitsev, J., Carmon, Y., Shankar, V., and Schmidt, Datacomp: In search of the next generation of multimodal datasets. ArXiv, abs/2304.14100 2023.
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-o: Weak-to-stree so-sonng training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, 2024.
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-la: Weak-to-strong training of diffusion transformer for 4k cext-to-image generation. In European Conference on Computer Vision, pp. 74-91. Springer, 2025.
Garibi, D., Patashnik, O., Voynov, A., Averbuch- -loor,, H., and Cohen Or, D. Renoise: Real image inversion through iterative noising. ArXiv, abs/2403. 14602, 2024.
Christiano, P. F,, Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Gu, Y., Wang. Z., Yin, Y., Xie, Y., and Zhou, M. Diffusion- rpo: Aligning diffusion models through relative prefer- ence optimization. ArXiv, abs/2406.06$3 2024.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-prompl im- age editing with cross attention control. arXiv preprint a6X6v22088.01666, 2022.
Clark, K., Vicol, P., Swersky, K., and Fleet, J. Directly fine-tuning diffusion models on differentiable rewards. ArXiv, abs/2309. 17400, 2023.
Dai, X., Hou, J., Ma, C., Tsai, S. S., Wang. J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., Yu, M., Kadian, A., Radenovic, F., Mahajan, D. K., Li, K., Zhao, Y., Petrovic, V., Singh, M. K., Motwani, S., Wen, Y., Song, Y.-Z., Sumbaly. R., Ramanathan, V., He. Z., Vajda, P., and Parikh, D. Emu: Enhancing image generation models using photogenic needles in a haystack. ArXiv, abs/2309.15807, 2023.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. ArXiv, abs/2006.112333 2020.
Hong, J., Lee, N., and Thorne, J. Reference-free monolithic preference optimization with odds ratio. arXiv e-prints, pp. arXiv-2403, 2024a.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06766 2023.
Hong, J., Paul, S., Lee, N., Rasul, K., Thorne, J., and Jeong, J. Margin aware preference optimization for aligning dif- fusion models without reference. ArXiv, abs/2406. 06644, 2024b.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Muller, J. Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, Podell, D., Dockhorn, T., English, Z., Lacey, K., Good- win, A., Marek, Y., and Rombach, Scaling rectified flow transformers for high-resolution image synthesis. ArXiv, abs/2400.003066 2024.
Huang, Y., Huang, J., Liu, Y., Yan, M., Lv, J., Liu, J., Xiong, W., Zhang, H., Chen, S., and Cao, L. Diffusion model-based image editing: A survey. arXiv preprint arXiv:2402. 17525, 2024.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4396 4405, 2018.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic 2024.
optimization. arXiv preprint arXiv:2402.00
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M.. Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Dpok: Reinforcement learning for fine-tuning text-to- image diffusion models. ArXiv, abs/2305.1633 16381, 2023.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation. ArXiv, abs/2305.01 2023.
10

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023a.
Lu, Y., Wang. Q., Cao, H., Wang, X., Xu, X., and Zhang, M. Inpo: Inversion preference optimization with reparametrized ddim for efficient diffusion model align- ment. arXiv preprint arXiv: 18454, 2025.
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abheel, P., Ghavamzadeh, M., and Gu, S. S. Align- ing text-to-image models using human feedback. ArXiv, abs/2302.12192, 2023b.
Meng, C., He, Y., Song, Y., Song. J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations. arXiv preprint 2021.
Li, S., Kallidromitis, K., Gokul, A., Kato, Y., and Kozuka, K. Aligning diffusion models by optimizing human utility. ArXiv, abs/2404.04465, 2024a.
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images us- ing guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Li, Y., Liu, X., Kag, A., Hu, J., Idelbayev, Y., Sagar, D., Wang, Y. Tulyakov, S., and Ren, Textcraftor: Your text encoder can be image quality controller. 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7985-7995 2024b.
Nichol, A. and Dhariwal, P. Improved denoising diffusion prohabilistic models. ArXiv, abs/2102.0900 026722 2021.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Liang, Y., He, J., Li, G., Li, P., Klimovskiy, A., Carolan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F,, Ke, J., Dvijotham, K. D., Collins, K., Luo, Y., Li, Y., Kohlhoff, K., Ramachandran, D., and Navalpakkam, V. Rich human feedback for text-to-image generation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19401-19411, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision pp. 4195-4205, 2023.
Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, 2024.
Peebles, W. S. and Xie, S. Scalable diffusion models with transformers. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4172 4182, 2022.
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. ArXiv, abs/2210.02747, 2022.
Pernias, P., Rampas, D., Richter, M. L.,, Pal, C., and Aubre- ville, M. Wuerstchen: An efficient architecture for large- scale text-to-image diffusion models. 2023.
Liu, B., Shao, S., Li, B., Bai, L., Xu, Z.,, Xiong, H., Kwok, J., Helal, A., and Xie, Z. Alignment of diffusion models: Fundamentals, challenges, and future. ArXiv, abs/2409.07253, 2024.
Podell, D., English, 7., Lacey, K., Blattmann, A., Dockhorn, T., Muller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. ArXiv, abs/2307.01952. 2023.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Salch, M., Liu, P. J., and Liu, J. Statistical rejection sam- pling improves preference optimization. arXiv preprint 2rXiv:2309.06657 2023.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning generate and transfer data with rectified flow. ArXiv, abs/2209, 00000,, 2022.
Prabhudesai, M., Goyal, A., Pathak, D., and Fragkiadaki, K. Aligning text-to-image diffusion models with reward backpropagation. ArXiv, abs/2310.03773 2023.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint /71 2017.
Prabhudesai, M., Mendonca, R., Qin, 7.,, Fragkiadaki, K., and Pathak, D. Video diffusion alignment via reward gradients. arXiv preprint w=XXi:2407.08737. 2024.
Lou, X., Zhang, J., Xie, J., Liu, L., Yan, D., and Huang, K. Spo: multi-dimensional preference sequential alignment with implicit reward modeling. ArXiv, 2tsp/2255.1799, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
visual models from natural language supervision. In International Conference Machine Learning, 2021a.
Song, Y., Sohl-Dickstein, J. N., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. ArXiv, abs/201 1.0001...11 131465, 2020b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021b.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Wallace, B., Dang, M., Rafailov, R., Zhou, L.,, Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S. R., and Naik, N. Diffusion model alignment using direct preference optimization. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8228 8238, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er-
mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Reed, S. E., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H. Generative adversarial text to image synthesis. In International Conference on Machine Learning, 2016.
Wang. Z., Bi, B., Pentyala, K., Ramnath, K., Chaud- huri, S., Mehrotra, S., Zhu, Z., Mao, X.-B., Asur, S., and Cheng, N. A comprehensive survey of Ilm align- ment techniques: Rlhf, rlaif, ppo, dpo and more. ArXiv, abs/2407.16216 2024.
Ren, M., Xiong, W., Yoon, J. S., Shu, Z., Zhang, J., Jung, H., Gerig, G., and Zhang. H. Relightful harmonization: Lighting-awant aware portrait background replacement. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 6452-6462 2024.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. ArXiv, 000//336..9341,, 2023.
Rombach, R., Blattmann, A., Lorenz, D.. Esser, P., and
Ommer, B. High-resolution image synthesis with la- tent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 10674-10685, 2021.
Wu, X., Huang, S., and Wei, F. Multimodal large language model is human-aligned annotator for text-to-image generation. ArXiv, abs/2404. 2024.
Xu, J., Liu, X., Wu, Y., Tong. Y., Li, Q., Ding, M., Tang, J., and Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. ArXiv, abs/2304.05977, 2023.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crow- son, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next generation image-text models. ArXiv, 10.0220 2022.
Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Li, Q., Shen, W., Zhu, X., and Li, X. Using human feedback to fine- tune diffusion models without any reward model. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8941-8951, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017.
arXiv preprint
Yang, S., Chen, T., and Zhou, M. A dense reward view on aligning text-to- -atate- diffusion with preference. ArXiv, Abss2440.082266 2024.
Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In International Con- ference on Machine Learning, pp. 4596-4604 PMLR, 2018.
Ye, J., Liu, F., Li, Q., Wang, Z., Wang, Y., Wang, X., Duan, Y., and Zhu, J. Dreamreward: Text-to-3d generation with human preference. In European Conference on Computer Vision, pp. 259-276. Springer, 2025.
Song, F., Li, Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 18990-18992) 2024.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Va- sudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich rext-to-image gen- eration. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. ArXiv, abs/2010.02502, 2020a.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yuan, H., Zhang, S., Wang, X., Wei, Y., Feng, T., Pan, Y., Zhang, Y., Liu, Z., Albanic, S., and Ni, D. Instructvideo: instructing video diffusion models with human feedback.
12

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 6463-6474, 2024.
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses align language mod- els with human feedback without tears. arXiv preprint arXiv:2304.05302. 2023.
Zhang, C.,, Zhang. C.,, Zheng, S., Zhang, M., Qamar, M., Bac, S.-H., and Kweon, S. A survey audio diffusion models: Text speech synthesis and enhancement in generative ai. arXiv preprint 2rXIv:2303. 13336, 2023a.
Zhang, L.., Rao, A., and Agrawala, M. Adding conditional control text-lo-image diffussinn models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3843 2023b.
Zhang. S., Wang, B., Wu, J., Li, Y., Gao, T., Zhang, D., and human preference for text-to-image generation. 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 8018-8027, 2024a.
Wang, Z. Learning multi-dimension
Zhang, Y., Tzeng, E.., Du, Y., and Kislyuk, D. Large-scale reinforcement learning for diffusion models. In European Conference Computer Vision, 2024b.
13

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
Appendix
A Background
Conditional Generative Models. Diffusion models represent a category of generative models that create data by inverting a coise-introducing forward process. During the training phase, the neural network learns to approximate the reversal of this forward process, which systematically corrupts data with noise. By harnessing the generalization and approximation abilities of neural networks, diffusion models (Nichol & Dhariwal, 2021; Ho al., 2020) are capable of generating diverse data samples that align with the distribution of the training dataset. These models can be broadly categorized into two main paradigms: denoising diffusion (Ho et al., 2020) and scsre-based matching (Song & Ermon, 2019; Song al., 2020b), each providing unique theoretical frameworks and computational strategies for the generation process. In recent years, diffusion models and their variants, including Rectified Flow (Liu et al., 2022), have risen to prominence as the leading framework in generative modeling, showcasing exceptional performance in both output quality and training stability compared to earlier methods. Thanks to advancements in diffusion models (Peebles & Xie, 2023; Bao et al., 2023), they have also achieved remarkable progress in areas such as conditional image generation (Chen et al., 2023b; 2025), audio synthesis (Zhang et al., 2023a), and video creation (Bao et al., 2024; Blattmann al., 2023a; Brooks et al., 2024; Blattmann et al., 2023b). In this study, we concentrate primarily conditional image generation (Huang et al., 2024; Meng et al., 2021). A common strategy for generating desired images involves the use of text prompts (Hertz et al., 2022) as guidance. Textual information is typically converted into text embeddings using pre-trained text encoder (Radford et al., 2021b), enabling text-to-image models to achieve impressive results. However, relying solely on textual input often results in limited precision. To improve image control with additional conditions, such as sketches, the prevailing approach involves training a dedicated control network and integrating it with the generative model. A prominent example ControlNet (Zhang et al., 2023b), which trains an independent control network for each conditional image, fostering the development of various control techniques. In our work, we utilize edge maps and depth maps guide the text-to-image model.
Preference Alignment of Large Language Models. Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) is a pivotal technique for aligning large language models (LLMs) with human preferences (Ouyang et al., 2022). This methodology typically involves two main steps: first, training a reward model to approximate human preferences, and second, applying reinforcement learning to optimize the policy, aiming to maximize the reward feedback received by the model. Leading LLMs, such as ChatGPT, have effectively integrated this fine-tuning approach. A central algorithm in reinforcement learning, Proximal Policy Optimization (PPO) (Schulman et al., 2017), requires the concurrent loading of multiple models, including the training model, reference model, critic model, and reward model. The substantial compu- tational overhead and intricate training objectives associated with PPO often make its practical optimization challenging (Ouyang et al., 2022). To address these computational demands and improve training efficiency, Ahmadian et al. (2024) has shown that REINFORCE-style methods and their variants are highly effective within the RLHF framework. Additionally, alternative methods circumvent traditional reinforcement learning by using the reward model to rank prompt samples from LLMs and then collecting preference data for fine fine-tuning, For example, RAFT (Dong et al., 2023) focuses on supervised fine-tuning of high-reward samples, RRHF (Yuan et al., 2023) uses ranking loss for alignment, and Liu et al. (2023) applies rejection sampling optimization to gather preference data from the optimal policy.
Direct Preference Optimization (DPO) (Rafailov et al., 2024) climinates the need for an explicit reward model by directly optimizing the policy, implicitly refining the reward scores embedded in the Bradley-Temy (BT) model. Similarly, IPO (Azar et al., 2024) argues that pairwise preferences cannot be replaced by pointwise rewards and provides method for direct optimization based on preference probabilities. Furthermore, ORPO (Hong al., 2024a) removes the requirement for a reference model by simultaneously performing supervised fine-tuning and preference optimization. In alternative reward configurations, KTO (Ethayarajh et al., 2024) adopts the Kahneman-Tverr maximizing the log-likelihood of preferences, while PRO (Song et al., 2024) utilizes reward ranking information to optimize the large language models.
model to represent human utility instead of
Further Diffusion Models Alignment. Beyond preference alignment in text-to-image diffusion models, other generative domains, such as video and 3D. have also adopted alignment methods customized for their unique data characteristics. For example, InstructVideo (Yuan et al., 2024) enhances text-to-video diffusion models by incorporating reward fine-tuning supplemented with human feedback. This method utilizes an image reward model to improve video quality while reducing fine-tuning costs through partial DDIM sampling. Similarly, Prabhudesai et al. (2024) aligns the base video diffusion model
14

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
using gradients from a publicly available pre-trained visual reward model. In the 3D domain, DreamReward (Ye et al., 2025) creates text-based 3D dataset to train reward model (Reward3D) and integrates the reward model's gradients into the
score distillation sampling (Poole et al., 2022) framework. The alignment of human preferences in diffusion models is still in its early stages, with future advancements potentially including the adaptation of Large Language Model (LLM) alignment techniques to diffusion models and the extension of alignment approaches to additional modalities, such as audio and tactile feedback.
B Details of the Primary Derivation
In this section of the paper, we provide the full derivation of Equation (9):
(2 [c) Peet (2r) c)
3 log Do (c)
LSmO - [a an e)/DD log
B log
Pret (2) c)
pu pol prot (at)
-a an e) log
B log
(e c) (x)) c))
polzed le)" po
Prot (20 (c)
Pref (2/)1c)
(po (c)
25
-a ay e) log
Blog
Prot Prot [e Pent pra (cylee))
po(2 c) (a(
2 2
pg (c)
@ log Po
(2 Po(ar)
(2(h)e)))) ()) Pref x c) -- Pref (ar) le))
(17)
--E(ay a e)D log a
Pret pret (a)le)) app (2 (c) 2 po (an) wollle)) le)a
po(x)) (0llc)22222_) Pref (2/ (2/lec220 2a) Peef (3%) cc)aa-7pp (ay) [clccc2211
-aayy a5 e)wD log
f log
(2a - 1--7)88ogg Pa(2) Pro (2) (c)
-EE o) logo
Pret (x)' c)po (ar (c)
po (2) c).
log po po(aofle) Pref (2/ (c)
- Lang a .D log
(2a - 3)8(1og
Prof (x)) c))
In the main body, p.(.) represents po(-Ic) for compactness.
C Further Discussion
Human preferences are influenced by variety of factors, including culture and geographical location, leading significant variability among individuals. Consequently, our motivation is both well-founded and explicitly defined. In our experiments, we utilize the Pick-a-Pic v2 dataset, which features image quality ranging between SD1.5 and SDXL standards. The quantitative results (refer to Table 2) demonstrate that our method outperforms supervised fine-tuning and other preference optimization methods across both models, exhibiting superior adaptability, Our approach employs an offline fine-tuning methodology that relies on pre-existing datasets. However, it can be seamlessly transitioned to an online learning framework, enhancing its practical applicability.
D Experiment Details
Additional Implementation details. During the text-to-image generation evaluation, the CFG (Classifier-Free Guidance) (Ho& & Salimans, 2022) values for SD1.5 and SDXL are set to 7.5 and 5, respectively, following widely accepted standards. For tasks involving depth maps and Canny edges, the controlnet conditioning scales are set to 0.5 and 0.3, respectively, with ensure reproducibility.
CFG fixed at for both. The random seeds for all comparative experiments are fixed
Explanation of Chart. To facilitate clearer comparison, multiply the PickScore, HPSv2.1, and CLIP scores by factor of 100, ensuring readability while maintaining precision with four significant digits.
Pick-a-Pic. The Pick-a-Pic dataset comprises a curated set of text-to-image pairs, meticulously gathered from user interactions within the Pick-a-Pic web application. Each entry in the dataset features a duo of images, accompanied by
15

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
descriptive text prompt and label that captures user preferences. This dataset is enriched with images produced by spectrum of text-to-image generation models, such as Stable Diffusion 2.1, Dreamlike Photoreal 2.05, and several iterations of Stable Diffusion XL, each tested across variety of CFG scales. For the purposes of this research, we focus on the training segment of the dataset. Moreover, to substantiate the robustness of our methodology, we include a detailed quantitative analysis of the test dataset in the supplementary materials, providing additional insights and validation of our findings.
HPDv2. The HPDv2 dataset is meticulously compiled by collecting a vast array of human preference data through the "Dreambot" channel on the Stable Foundation Discord server. This comprehensive dataset includes 25,205 unique text prompts and an impressive total of 98,807 images generated from these prompts. Each prompt is associated with several images and is annotated with preference labels that reflect user selections among image pairs. To rigorously evaluate the effectiveness of our approach, we employ a dedicated test dataset consisting of 3,200 text prompts. This test set plays pivotal role in benchmarking the HPDv2 model's capability to accurately assess and predict human preferences in image evaluation tasks.
Parti-Prompts. Parti- Prrti-rromps a meticulously curated collection of 1,632 text prompts specifically crafted to assess the efficacy of text-to-image generative models. This dataset encompasses wide range of categories and integrates numerous intricate tasks aimed at comprehensively evaluating the models' generative prowess across various dimensions. The prompts are structured to provide a rigorous testing environment, facilitating an in-depth analysis of model performance. They are particularly effective in gauging the models' flexibility and precision when confronted with complex and diverse scenarios, thereby offering thorough measure of their capabilities in real-world applications.
E
User Study
Here, We perform a user study consisting of three questions: (1) Overall, which image do you consider to have the best quality? (2) Which image the most visually appealing? (3) Which image has the highest alignment between text and image content?
Base SDXL
DPO-SDXL
MaPO MAPO-SDXL
CmPO-SDXL
Text
Aligrment
6%
21%
6%
67%
Visual Attrativenees
3%
22%
12%
63%
Overall
3%
22%
3%
72%
Prefexence
20
40
60
80
100
Figure 5: User Studies. We present the comparative user evaluation results of SmPO-SDXL against Base-SDXL, DPO- SDXL, and MaPO-SDXL. The first row displays the user study results for text alignment, the second row for visual attractiveness, and the third row for overall user preference. The results demonstrate that our method outperforms all existing baselines.
We randomly selected 50 prompts from Parti-Prompt and HPDv2 for image generation and invited 6 participants to evaluate the generated images based on three criteria. As illustrated in Figure 3, SmPO-SDXL achieved 72% preference rate in overall evaluation, while DPO-SDXI,, MaPO-SDXI, and Base-SIXXI. obtained 22%, 3%, and 3%, respectively. Furthermore, SmPO-SDXL demonstrated comparable performance in both visual Attractiveness and text alignment, with preference rates of 63% and 67%, respectively. Our results indicate that the SmPO fine-tuned SDXL base model significantly outperforms the baseline Base-SDXL model. Additionally, exhibits superior performance compared to the advanced baselines, Diffusion-DPO and MaPO, across all evaluated metrics.
16

---

Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
F
Additional Quantitative Results
In this section, we provide additional quantitative evaluations.
Table 8 presents the quantitative comparison against SDXL baselines on the Pick-a-Pic v2 test set.
Table presents the quantitative comparison against SD1.5 baselines on the Pick-a-Pic v2 test set.
The experimental outcomes indicate that our method outperforms the baseline models across nearly all reward-based evaluation metrics, highlighting its superior performance.
Table 8: Additional quantitative comparison with SDXL baselines. We apply the prompts from the Pick-a-Pic v2 test set to compare our model with the existing alignment baseline on the SDXL model. We report the median and mean values of five reward evaluators on the Pick-a-Pic v2 test set, retaining five significant figures. In the table, the highest value each column is highlighted in bold, and the second highest is underlined. shown, our model achieves the best results in nearly all reward evaluations.
PickScore Median
HPSv2.1
ImageReward Median
Aesthetic
CLIP
Baselines
Mcan
Median
Mean
Mean
Median Mcan
Median
Mean
Base-SDXL
22.226
22.153 28.482 28. 18.102
0.7377 0.5622 5.9719 6.0061 36.512 36.138
SFT-SDXI.
21.658
21.678
28.070
27.747
0.4851
0.3991
5.8693
5.8594
36.244 35.749
DPO-SDXL
22.600
22.627
29.708
29.612
1.0345
0.7993
6.0252 6.0168
37.386 37.376
MaPO-SDXI.
22,200
22.272
29.251
28.978
1.1718
0.9135
6.2382
6.1941
36.628
36.072
SmPO-SDXL
23.123
23.052 32.173 31.691
1.3288
1.0742
6.1357
6.1259
37.634 37.381
Table Additional quantitative comparison with SD1.5 baselines. We apply the prompts from the Pick-a-Pic v2 test set compare our model with the existing alignment baseline on the SD1.5 model. We report the median and mean values of five reward evaluators on the Pick-a-Pic v2 test set, retaining five significant figures. In the table, the highest value in each column is highlighted in bold, and the second highest is underlined. As shown, model achieves the best results in nearly all reward evaluations.
PickScore Median
HPSv2.1
ImageReward Median
Aesthetic
CLIP
Baselines
Mean Median
Mean
Mean
Median
Mean
Median
Mean
Base-SD1.5
20.633
20.662 24.642 24.237 -0.0865 -0.1505 5,3334
5.3268 33.046
32.637
SFT-SD1.5
21.184
21.255 27.949 27.728 0.6389
0.4771
5,6314
5.6220
34.261
34.055
DPO-SD1.5
21.033
21.052 25.760 25.384
0.1653
0.0707
5.5254
5,4706
33.272 33.276
KTO-SD1.5
21.200
21.193
27.917
27.719
0.6320
0.5395
5.5996
5.5820
33.996
33.940
SmPO-SD1.5
21.628
21.576
28.944
28.507
0.8965
0.6401
5.7449
5.6997
35.020
34.864
17


Reviews:
[{"review_id": "P2MqqAZJDN", "reviewer": "Reviewer_UNNR", "summary": "This paper proposes a post-training method for diffusion models, named SmPO, which is modified from diffusion-dpo. SmPO recognize the variability of human preferences by replacing binary preferences with smoothed preference distributions, thereby mitigating label bias. In addition, Renoise Inversion method is employed to estimate the sampling trajectory. Compared to previous methods that randomly sample noise from a Gaussian distribution, this inversion method provides a more accurate estimation of the trajectory preference distribution. Extensive experiments demonstrate the strong performance of SmPO and the effectiveness of the proposed modules.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are honored by your favorable evaluation and have carefully considered your suggestions!\n\n---\n\n**Q1**: *The rationale for designing the weight-to-sensitivity ratio as given in Equ (12) is unclear.*\n\n**A1**: Thank you for your feedback!  According to Equ (8), $\\tilde{p}(x_{0}^{w}|c) = \\frac{p(x_{0}^{w}|c)^{\\alpha}p(x_{0}^{l}|c)^{\\gamma-\\alpha}}{Z_{p}^{w}(c)}=\\frac{(p(x_{0}^{w}|c)^{\\frac{\\alpha}{\\gamma}}p(x_{0}^{l}|c)^{1-\\frac{\\alpha}{\\gamma}})^{\\gamma}}{Z_{p}^{w}(c)}$\n\nwhere weight-to-sensitivity ratio could be regarded as the pairwise preference probability $p(x_{0}^{w} \\succ x_{0}^{l}|c)$. Since a pairwise preference is hard to model directly, we adopt the well-established Bradley-Terry framework through the reward model as Equ (12).\n\n---\n\n**Q2:** *The GPU hours for SmPO should include the training time of the reward model.*\n\n**A2**: Thank you for the feedback! Training PickScore requires **8×A800 GPUs for ~40 minutes** [1]. We have updated **Table 1** to include this cost.\n\nSupp Table 1: Computational cost comparison\n\n| Model     | GPU Hours              |\n| --------- | ---------------------- |\n| DPO-SDXL  | ~976.0                 |\n| MaPO-SDXL | ~834.4                 |\n| SmPO-SDXL | ~**150.8 (145.5+5.3)** |\n\n| Model      | GPU Hours            |\n| ---------- | -------------------- |\n| DPO-SD1.5  | ~204.8               |\n| KTO-SD1.5  | ~1056.0              |\n| SmPO-SD1.5 | ~**41.3 (36.0+5.3)** |\n\n[1] https://github.com/yuvalkirstain/PickScore\n\n---\n\n**Q3:** *PRDP, SPO and RankDPO should be included.*\n\n**A3:** We appreciate the suggestion! We'll update our related work with these papers and include the SmPO-SPO comparison.\n\nSupp Table 2: Median score comparison of SD1.5 on HPD v2.\n\n|            | PickScore | HPSv2.1   | ImageReward | Aesthetic | CLIP      |\n| ---------- | --------- | --------- | ----------- | --------- | --------- |\n| SPO-SD1.5  | 21.49     | 26.74     | 0.181       | 5.655     | 32.513    |\n| SmPO-SD1.5 | **23.62** | **32.53** | **1.331**   | **6.264** | **38.88** |\n\nSupp Table 3: Win-rate comparison on Parti-Prompts. \n\n|                         | PickScore | HPSv2.1 | ImReward | Aesthetic | CLIP  |\n| ----------------------- | --------- | ------- | -------- | --------- | ----- |\n| SmPO-SD1.5 vs SPO-SD1.5 | 70.81     | 75.47   | 76.62    | 65.03     | 79.96 |\n| SmPO-SDXL vs SPO-SDXL   | 58.37     | 52.44   | 63.72    | 52.36     | 74.25 |\n\n**However, Comparisons with PRDP (different SD-v1.4 base) and RankDPO (different training data) are limited by the unavailability of checkpoints.**\n\n---\n\n**Q4:** *There are no experiments validating the design of the weight-to-sensitivity ratio.* \n\n**A4:**  Thank you for the feedback. We have designed two experiments to verify this aspect:\n\n1. **Ablation Study (Table 3):**\n   The results demonstrate that incorporating our PickScore-based weight-to-sensitivity ratio (*+Smoothed*) leads to **significant performance gains**, validating its effectiveness.\n2. **Reward Model Analysis (Table 7):**\n   We have further compared the weight-to-sensitivity ratio with different reward models, providing additional empirical support for our design choice.\n\n---\n\n**Q5:**  *Could you provide quantitative results on the GenEval benchmark?* \n\n**A5:**  Yes! Thank you for highlighting this important aspect. We have conducted quantitative results on the GenEval.\n\nSupp Table 4: GenEval scores over SD1.5 baselines.\n\n| Model | single object | two object | counting | Attribute binding | position | colors   | overall  |\n| ----- | ------------- | ---------- | -------- | ----------------- | -------- | -------- | -------- |\n| SD1.5 | **0.96**      | 0.38       | 0.35     | 0.04              | 0.03     | 0.76     | 0.42     |\n| DPO   | **0.96**      | 0.40       | 0.38     | 0.05              | 0.04     | 0.77     | 0.43     |\n| KTO   | **0.96**      | 0.44       | 0.39     | **0.08**          | 0.07     | 0.78     | 0.45     |\n| SPO   | **0.96**      | 0.35       | 0.36     | 0.06              | 0.05     | 0.77     | 0.43     |\n| SmPO  | **0.96**      | **0.52**   | **0.40** | **0.08**          | **0.08** | **0.80** | **0.47** |\n\n\nSupp Table 5: GenEval scores over SDXL baselines.\n\n| Model | single object | two object | counting | Attribute binding | position | colors   | overall  |\n| ----- | ------------- | ---------- | -------- | ----------------- | -------- | -------- | -------- |\n| SDXL  | 0.97          | 0.70       | 0.41     | 0.22              | 0.10     | 0.87     | 0.55     |\n| DPO   | 0.98          | **0.80**   | 0.45     | 0.24              | 0.11     | 0.88     | 0.58     |\n| MaPO  | 0.95          | 0.70       | 0.36     | 0.26              | 0.10     | 0.88     | 0.54     |\n| SPO   | 0.97          | 0.73       | 0.38     | 0.17              | 0.10     | 0.86     | 0.53     |\n| SmPO  | **0.99**      | **0.80**   | **0.46** | **0.30**          | **0.15** | **0.89** | **0.60** |\n\nExperimental results indicate that SmPO consistently enhances image-text alignment performance.", "claims_and_evidence": "Claim: Human preference is variable. Simply considering binary preference of a pair of image causes label bias. \n\nEvidence: Exp in Table 3\n\n\n\n\nClaim: inversion based method can provide better estimation of the trajectory preference distribution.\n\nEvidence: Exp in Table 3", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "I have reviewed the derivation of the loss function in equation 16, and it appears correct to me. However, I will also consider the opinions of other reviewers.", "experimental_designs_or_analyses": "1. Computational cost comparison in Table 1: The GPU hours for SmPO-SDXL should include the training time of the reward model, as other methods rely only on preference pair data, whereas SmPO-SDXL requires a reward model.\n\n2. There are no experiments validating the design of the weight-to-sensitivity ratio.\n\n3. In the qualitative results, the image-text alignment appears to be improved. Could you provide quantitative results on the GenEval [1] benchmark to verify this?\n\n[1] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. \"Geneval: An object-focused framework for evaluating text-to-image alignment.\" Advances in Neural Information Processing Systems 36 (2023): 52132-52152.", "relation_to_broader_scientific_literature": "1. The standard practice of previous dpo-based methods [1,2,3] is to use binary label. This paper proposes a new approach that instead employs smoothed preference distributions and demonstrates the effectiveness of this replacement through experiments\n\n2. While using q(x_{1:T}|x_0) to approximate the reverse process is proposed by [1] and is widely accepted by the community, this paper proposes using inversion to achieve a more accurate estimation.\n\n[1] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Yang, Kai, et al. \"Using human feedback to fine-tune diffusion models without any reward model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. \"A dense reward view on aligning text-to-image diffusion with preference.\" arXiv preprint arXiv:2402.08265 (2024).", "essential_references_not_discussed": "I believe the following papers should be mentioned in the related works section or even compared against.\n\n[1] Deng, Fei, et al. \"Prdp: Proximal reward difference prediction for large-scale reward finetuning of diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Liang, Zhanhao, et al. \"Step-aware preference optimization: Aligning preference with denoising performance at each step.\" arXiv preprint arXiv:2406.04314 2.5 (2024): 7.\n\n[3] Karthik, Shyamgopal, et al. \"Scalable ranked preference optimization for text-to-image generation.\" arXiv preprint arXiv:2410.18013 (2024).", "other_strengths_and_weaknesses": "The rationale for designing the weight-to-sensitivity ratio as given in Equation 12 is unclear.", "other_comments_or_suggestions": "I will modify the score according to the opinions of other reviewers and the author's response.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Hn85oHayk", "reviewer": "Reviewer_4khg", "summary": "The paper introduces SmPO-Diffusion, an approach for aligning text-to-image diffusion models with AI preferences by refining the Direct Preference Optimization framework. Instead of using a binary preference, the authors propose a smoothed preference distribution based on a reward model.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and we'll do our utmost to resolve your concerns.\n\n---\n\n**Q1:**  *For evaluation, Image Reward, and Aesthetic score could also be considered.*\n\n**A1:**  We have incorporated both metrics: **Image Reward** and **Aesthetic Score**  are reported in **Table 2,8 and 9** (Quantitive comparison), **Table 3-7** (Ablation studies) and discussed in Line 262-263 of **Section 5.1**. \n\n---\n\n**Q2:** *There is no formal proof in this paper.*\n\n**A2:** The detailed derivation of SmPO loss is provided in **Appendix B**. \n\n---\n\n**Q3:** *The ablation study on hyperparameters are not comprehensive.*\n\n**A3:** Our **ablation studies** on hyperparameters were systematically designed along five key dimensions to validate our method’s robustness:\n\n+ **Component Effectiveness: Table 3** shows each module's contribution through progressive ablation.\n+ **DDIM Inversion Steps: Table 4** shows step selection's impact on output quality and training efficiency.\n+ **$(\\gamma,\\beta)$ combinations : Table 5** shows hyperparameter interaction effects.\n+ **CFG in Inversion: Table 6** shows CFG's influence during DDIM inversion.\n+ **Weight-to-Sensitivity Ratio $\\frac{\\alpha}{\\gamma}$: Table 7** shows reward-model derived calculations.\n\n---\n\n**Q4:** *Lack of comparison and discussion with many Diffusion DPO variants.*\n\n**A4:** Thank you for your feedback. We've compared our method with key baselines: **SFT, Diffusion-DPO,** its variants (**Diffusion-KTO and MaPO**). Additional **SPO** comparison are included (see response **A3** to Reviewer UNNR). To our knowledge, these are the most relevant and publicly available baselines. We welcome suggestions for additional baselines with references.\n\n---\n\n**Q5:** *Label smoothing has been proposed by the DPO authors in 2023, Conservative DPO (cDPO).*\n\n**A5:** Thank you for your feedback. In fact, cDPO **differs** fundamentally from our proposed SmPO.\n\n1. cDPO focuses on noisy labels (where labels may be **flipped** with some probability) and applies a **linear weighting** of swapped DPO losses: $L_{cDPO}(x_0^{w},x_0^{l})=(1-\\epsilon)L_{DPO}(x_0^{w},x_0^{l})+\\epsilon L_{DPO}(x_0^{l},x_0^{w})$. Our SmPO assigns **fine-grained** labels to each image pair and incorporates them into the DPO loss function through distribution averaging, yielding Equ (9).\n2. In cDPO, $\\epsilon$ is a is a **fixed** manually-tuned hyperparameter (refer to Line 50 and 84 of [1]), whereas our SmPO introduces reward-adaptive smoothing - zeroing loss for similar pairs, amplifying gradients for dissimilar ones. In other words, distinct image pairs are assigned different weights. As such, in Equ (12), $\\frac{\\alpha}{\\gamma}$ could be more precisely represented as $\\frac{\\alpha}{\\gamma}(x_{0}^{w},x_{0}^{l})$. \n\n[1] https://github.com/eric-mitchell/direct-preference-optimization/blob/main/trainers.py\n\n---\n\n**Q6:** *I cannot understand what the contribution is.* \n\n**A6:** We appreciate your feedback and are happy to clarify our key contributions:\n\n1. **Smoothed Preference:** We propose a novel *smoothed preference distribution* replacing DPO's binary modeling. Unlike fixed-weight approaches, our *(2α-γ)* scaling factor acts as a **pair-dependent dynamic regulator**, automatically driving loss to zero for similar pairs while amplifying gradients for dissimilar ones with each image pair's reward signal.\n\n2. **Precision Optimization:**\n   Unlike Diffusion-DPO's random noise injection, our *Renoise Inversion* technique enables **direct optimization of image-correlated variables.** This contribution provides more stable training and higher efficiency.\n\n---\n\n**Q7:** *It seems that $\\gamma$ can be absorb in $\\beta$ by $(2\\alpha-\\gamma)\\beta=(\\frac{2\\alpha}{\\gamma}-1)\\gamma\\beta$ is calculated by equation (12).*\n\n**A7:** We appreciate this observation.  \n\nThe term $(2\\alpha-\\gamma)$ is *image-pair-dependent*, automatically adjusted based on reward differences.  The core idea of $(2\\alpha-\\gamma)$ is to ensure that *\"the loss decreases when preferences are more similar, and increases otherwise.\"*  **This can be implemented via different parameterization strategies.** For example, $\\gamma$ can be step- or epoch-aware, adjusting dynamically to avoid being absorbed into $\\beta$. We adopt a simple formulation: $(2\\alpha-\\gamma)$ is implemented as $(\\frac{2\\alpha}{\\gamma}-1)\\gamma$, balancing adaptability and numerical stability.  Ablations (Table 5) validate the impact of different $(\\gamma,\\beta)$ pair configurations.\n\n---\n\n**Q8:** *Optimization via Renoise Inversion. There is no optimality analysis.*\n\n**A8:**  Thank you for your feedback. Optimizing Equ  (10) requires sampling $x_{1:T}\\sim p_{\\theta}^{c}(x_{1:T}|x_{0})$. While Diffusion-DPO approximates this with $q(x_{1:T}|x_{0})$, we argue precise $x_{0}$ reconstruction needs more accurate latents. This motivates our use of diffusion reconstruction methods (DDIM Inversion), supported by additional theoretical analysis (refer to response **A2** to Reviewer 5qKt).", "claims_and_evidence": "1. Smoothed Preference Modeling. Smoothed Labeling has been used in DPO in 2023. See *Essential References Not Discussed*.\n\n2. Optimization via Renoise Inversion. There is no optimality analysis.", "methods_and_evaluation_criteria": "For evaluation, Image Reward, and Aesthetic score could also be considered as benchmarks.", "theoretical_claims": "There is no formal proof in this paper.", "experimental_designs_or_analyses": "The ablation study on hyperparameters are not comprehensive.", "relation_to_broader_scientific_literature": "I cannot understand what the contribution is in this paper. It seems that this paper changes the hyperparameter $\\beta$ to $(2\\alpha -\\gamma)\\beta$, and tunes the hypermeters on specific datasets. That is all.", "essential_references_not_discussed": "1. Lack of comparison and discussion with many Diffusion DPO variants.\n\n2. Label smoothing has been proposed by the DPO authors in 2023 [1].\n\n[1] A note on DPO with noisy preferences & relationship to IPO. https://ericmitchell.ai/cdpo.pdf", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. I cannot understand what the contribution is but adding more hyperparameters in the loss function. I would more than appreciate it if the authors can clarify it.\n\n2. It seems that $\\gamma$ can be absorb in $\\beta$ by $({2\\alpha} - \\gamma)\\beta = (\\frac{2\\alpha}{\\gamma} - 1)\\gamma\\beta$ and $\\frac{\\alpha}{\\gamma}$ is calculated by equation (12).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6qLu2d0UO0", "reviewer": "Reviewer_HnYu", "summary": "This paper proposes a smoothed extension to DPO, where the preference data is smoothed to incorporate non-binary preference labels. The authors first created smoothed preference labels for image pairs using the likelihood estimation of a reward model. Then it uses noise-inversion to provide a better posterior estimation of the forward process during the optimization step. The author conducted experiments on a variety of models (SDXL, SD1.5) and showed that the proposed method outperforms existing baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're truly grateful for your enthusiastic reception of our manuscript and your insightful feedback!\n\n---\n\n**Q1：** *Table 7 caption can be improved.*\n\n**A1：** We sincerely appreciate your constructive feedback! We have implemented the following improvements:\n\n+ **Caption Revision**: We have revised the table caption to explicitly state:\n  **\"Table 7: Impact of weight-to-sensitivity ratio using different reward models. We report the performance comparison where rows represent different training signals and columns denote evaluation metrics. Upward arrows (↑) in column headers indicate higher values are preferable.\"**\n+ **Visual Reinforcement**: To further enhance clarity: We will include a brief structural description in the Results section (Section 5.4) when first referencing the table in \"Choice of Reward Model\" subsection.\n\n---\n\n**Q2**：*Discuss the statistic significance of the gap in table 7, 8, 9.*\n\n**A2:**  Thank you for this helpful suggestion! \n\nTo provide a comprehensive statistical evaluation of our approach, we present both **win-rate analysis** and **effect size measurements**. \n\n+ The win-rate results in *Supp Table 1-2* demonstrate how frequently evaluators prefer SmPO generations over baseline methods, with values exceeding **50**% (indicating statistical majority preference) highlighted in bold. \n\nSupp Table 1: Win-rate comparison between SmPO-SDXL and baselines on Pick-a-Pic v2 test set.  \n\n|               | PickScore | HPSv2.1  | ImageReward | Aesthetic | CLIP     |\n| ------------- | --------- | -------- | ----------- | --------- | -------- |\n| vs. SDXL      | **89.6**  | **93.6** | **81.5**    | **65.5**  | **60.6** |\n| vs. SFT-SDXL  | **96.6**  | **94.2** | **81.3**    | **76.3**  | **67.3** |\n| vs. DPO-SDXL  | **75.7**  | **85.9** | **71.9**    | **63.9**  | **55.4** |\n| vs. MaPO-SDXL | **83.5**  | **86.5** | **61.0**    | 47.4      | **60.2** |\n\n\n\nSupp Table 2: Win-rate comparison between SmPO-SD1.5 and baselines on Pick-a-Pic v2 test set.  \n\n|               | PickScore | HPSv2.1  | ImageReward | Aesthetic | CLIP     |\n| ------------- | --------- | -------- | ----------- | --------- | -------- |\n| vs. SD1.5     | **82.5**  | **88.0** | **80.1**    | **79.3**  | **66.5** |\n| vs. SFT-SD1.5 | **68.5**  | **66.1** | **62.2**    | **59.0**  | **60.0** |\n| vs. DPO-SD1.5 | **72.3**  | **83.7** | **76.1**    | **69.7**  | **63.3** |\n| vs. KTO-SD1.5 | **68.3**  | **67.3** | **57.0**    | **64.7**  | **59.8** |\n\n\n\n+ Additionally, we compute **Cohen's d** to quantify the **effect sizes** between SmPO and baselines in *Supp Table 3-6*, following conventional interpretations: |d|<0.2 (small), 0.2≤|d|<0.5 (medium), 0.5≤|d|<0.8 (large), and |d|≥0.8 (very large). **Cohen's d measures the standardized mean difference in standard deviation units, making it unit-free.**\n\n\n\nSupp Table 3: Cohen's d for comparison between SmPO-SDXL and baselines on Pick-a-Pic v2 test set. \n\n|               | PickScore | HPSv2.1 | ImageReward | Aesthetic | CLIP  |\n| ------------- | --------- | ------- | ----------- | --------- | ----- |\n| vs. SDXL      | 0.600     | 0.860   | 0.537       | 0.205     | 0.203 |\n| vs. SFT-SDXL  | 0.937     | 0.960   | 0.693       | 0.480     | 0.266 |\n| vs. DPO-SDXL  | 0.282     | 0.505   | 0.290       | 0.192     | 0.010 |\n| vs. MaPO-SDXL | 0.508     | 0.648   | 0.176       | -0.120    | 0.213 |\n\n\n\nSupp Table 4: Cohen's d for comparison between SmPO-SD1.5 and baselines on Pick-a-Pic v2 test set. \n\n|               | PickScore | HPSv2.1 | ImageReward | Aesthetic | CLIP  |\n| ------------- | --------- | ------- | ----------- | --------- | ----- |\n| vs. SD1.5     | 0.650     | 1.010   | 0.730       | 0.652     | 0.385 |\n| vs. SFT-SD1.5 | 0.227     | 0.188   | 0.157       | 0.136     | 0.139 |\n| vs. DPO-SD1.5 | 0.368     | 0.732   | 0.529       | 0.392     | 0.275 |\n| vs. KTO-SD1.5 | 0.272     | 0.192   | 0.099       | 0.209     | 0.161 |\n\n\n\nSupp Table 5: Cohen's d for comparison between SmPO-SDXL and baselines on HPD v2 test set. \n\n|               | PickScore | HPSv2.1 | ImageReward | Aesthetic | CLIP  |\n| ------------- | --------- | ------- | ----------- | --------- | ----- |\n| vs. SDXL      | 0.601     | 0.914   | 0.498       | 0.206     | 0.124 |\n| vs. SFT-SDXL  | 1.084     | 0.957   | 0.647       | 0.419     | 0.272 |\n| vs. DPO-SDXL  | 0.336     | 0.527   | 0.192       | 0.219     | 0.012 |\n| vs. MaPO-SDXL | 0.537     | 0.772   | 0.162       | 0.021     | 0.149 |\n\n\n\nSupp Table 6: Cohen's d for comparison between SmPO-SD1.5 and baselines on HPD v2 test set. \n\n|               | PickScore | HPSv2.1 | ImageReward | Aesthetic | CLIP  |\n| ------------- | --------- | ------- | ----------- | --------- | ----- |\n| vs. SD1.5     | 1.007     | 1.360   | 0.946       | 0.778     | 0.473 |\n| vs. SFT-SD1.5 | 0.362     | 0.181   | 0.165       | 0.201     | 0.185 |\n| vs. DPO-SD1.5 | 0.641     | 0.979   | 0.686       | 0.481     | 0.327 |\n| vs. KTO-SD1.5 | 0.413     | 0.246   | 0.173       | 0.244     | 0.246 |", "claims_and_evidence": "The author made two central claim: a) the existing framework based on binary feedbacks do not adequately address noise and inconsistencies in the human preference data, necessitates the incorporation of a smoothed relaxation. b)  the proposed noise inversion technique addresses the improves training process of preference alignment algorithm by providing better estimates of forward sampling trajectory. The author was able to support both of these claims with theoretical analysis and emprical experiments.", "methods_and_evaluation_criteria": "The author employed common evaluation metrics (e.g. PickScore, HPSv2) and datasets (e.g. Parti-Prompts) that are widely used by related literature. The author also included relevant baselines such as Diffusion-DPO, Diffusion-KTO. The authors additionally conducted experiments on multiple models (SD1.5, and SDXL) to showcase the generalizability of the proposed method.  These results are comprehensive.", "theoretical_claims": "I checked derivation in the main paper and appendix, they look good to me.", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria section. The author employs a set of common setup for their experiments.", "relation_to_broader_scientific_literature": "The proposed work proposed a novel solution to a problem that is well-recognized by the community (i.e. noises and inconsistencies of working with human preference data).  This work provide meaningful insight to the problem and can inspire future solutions.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The authors are suggested to discuss the statistic significance of the gap in table 7, 8 and table 9. These numerical metrics can be hard to interpret for people with limited exposure to prior literature, as they have drastically different scales and some gap may be perceived as \"insignificant\".", "other_comments_or_suggestions": "Table 7 caption can be improved, as it is not immediately clear if rows are reward model and columns are evaluation metric, or vice versa. It was only clearly after noticing the uparrow in column titles. I suggest the authors clearly state in the caption that rows represents different training signals while columns are evaluated metrics.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsDlRSBEyJ", "reviewer": "Reviewer_5qKt", "summary": "This paper proposes SmPO-Diffusion, a novel method for aligning text-to-image diffusion models with varied human preferences. The authors introduce two core contributions: (1) a smoothed preference modeling approach, replacing the binary preference distribution with a smooth distribution derived by reward models. (2) an optimization strategy called ReNoise Inversion, designed to estimate the dpo preference distribution by inversion and renoise.  Experimentally, the method achieves state-of-the-art results, surpassing existing baselines across multiple human preference evaluation metrics, and notably reduces training resource consumption.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highly recognizing the value of our study and helpful feedback! \n\n---\n\n**Q1:** *Why do reward models outperform large human preference datasets despite their simplicity?*\n\n**A1:** This is a great question! We believe reward models have the following advantages:\n\n1、We note that PickScore (reward model) feedback is interpretable as **pseudo-labeling** the Pick-a-Pic dataset—a form of data cleaning [1] [2]. Specifically, the PickScore reward model serves not only as a proxy for human preferences but also as a mechanism for **data refinement**. \n\n2、**Granular Feedback vs. Binary Human Labels**\n\nHuman preference datasets inherently provide *binary* labels (e.g., \"A > B\"), limiting their ability to represent nuanced preference distributions. In contrast, reward models assign **continuous scores**, enabling two key advantages:\n\n+ **Smooth Supervision**: Loss functions can adaptively weight samples based on reward differences,avoiding the brittleness of binary thresholds.\n\n+ **Relative Calibration**: Fine-grained rewards better capture perceptual similarity (e.g., distinguishing \"slightly better\" from \"significantly better\" cases), which is critical for guiding iterative model updates.\n\nThis explains why reward-guided training can outperform raw human datasets: it amplifies signal-to-noise ratios in human preferences while retaining their semantic intent. We emphasize that our approach complements (not replaces) human feedback, as reward models are trained on human-annotated data.\n\n[1] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet. CVPR 2020\n\n[2] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pretraining and self-training. NeurIPS 2020\n\n---\n\n**Q2:** *The paper uses DDIM inversion in DPO training but lacks theoretical justification of its advantage over basic noise addition.*\n\n**A2:**  Thank you for the valuable feedback! We first derive the minimal proof. Note that for Equ (5), Diffusion-DPO utilizes $q(x_{1:T}|x_{0})$ to approximate $p_{\\theta}(x_{1:T}|x_{0})$. For each step, Diffusion-DPO used $q(x_{t-1,t}|x_{0})$ to approximate $p_{\\theta}(x_{t-1,t}|x_{0})$. Suppose we replace standard noise injection at $x_{t-1}$ with **a single-step DDIM inversion**, when given $x_{0}$. Formally, we propose to use $q(x_{t-1}|x_{0})p_{\\theta}(x_{t}|x_{t-1})$ for approximation. And this approximation yields lower error because $$D_{KL}(q(x_{t-1}|x_{0})p_{\\theta}(x_{t}|x_{t-1})||p_{\\theta}(x_{t-1,t}|x_{0})) = D_{KL}(q(x_{t-1}|x_{0})||p_{\\theta}(x_{t-1}|x_{0})) <D_{KL}(q(x_{t-1,t}|x_{0})||p_{\\theta}(x_{t-1,t}|x_{0})).$$\n\nThe complete derivation employing full inversion will be detailed in the paper.\n\n---\n\n**Q3:** *Could DDIM inversion be generalized and applied as an standard alternative to noise injection in diffusion training?*\n\n**A3:** Yes, generalizable!  We greatly appreciate the opportunity to discuss this point! \n\nCurrently, the training of large-scale diffusion models/flow matching primarily consists of two stages: **pre-training** and **post-training**. The pre-training stage focuses on establishing the trajectory from a noise distribution to a data distribution, while we argue that the post-training stage should specifically fine-tune variables that are **highly correlated with the target images**.\n\nThis idea shares some similarities with the **Reflow** method in **Rectified Flow** [3], which trains specialized (noise, image) pairs. However, a key distinction is that most real-world datasets contain only images (without paired noise). Thus, our motivation is to **adaptively adjust image-dependent variables** during post-training. We believe this approach could become a dominant paradigm in future post-training strategies, as it **preserves the model’s inherited capabilities while enhancing its task-specific performance**. \n\n[3] Liu, Xingchao, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR2023\n\n---\n\n**Q4:** *Please provide a clearer introduction and context for the compared baselines.* \n\n**A4:** Thank you for the suggestion! We fully agree that providing clearer motivation for method selection is critical for readability. Below, we outline our planned revisions to address this concern:\n\nThese baselines share a common focus on **human-aligned image generation** but **differ fundamentally in their technical mechanisms**:  Diffusion-KTO adopts the Kahneman-Tversky [4] model to represent human utility instead of maximizing the log-likelihood of preferences. Margin-aware Preference Optimization (MaPO)  jointly maximizes the likelihood margin between the preferred and dispreferred datasets and the likelihood of the preferred sets, simultaneously learning general stylistic features and preference without reference model.\n\n[4] Tversky, A. and Kahneman, D. Advances in prospect theory: Cumulative representation of uncertainty.", "claims_and_evidence": "Yes. The claims in the submission are supported by convincing evidence. The paper provides extensive experiments and ablation studies to support the claims.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.\nI've checked the correctness of any proofs for theoretical claims.\nThe author provided the derivation of eq.9 in the appendix and it is correct.", "experimental_designs_or_analyses": "Yes.\n1. Primarily used the Pick-a-Pic v2 dataset and benchmarked against established baselines (e.g., Diffusion-DPO, MaPO, KTO, SFT), which is appropriate and valid.\n2. Evaluated results using widely-accepted automatic metrics (e.g., PickScore, HPSv2.1), effectively capturing text-to-image quality.\n3. Conducted detailed ablation studies clearly demonstrating the contributions of each method component; analyses were very convincing.\n4. Included practical and valid computational efficiency analyses.", "relation_to_broader_scientific_literature": "Yes.\n\n1. The paper builds upon Direct Preference Optimization (DPO) techniques, extending recent methods such as Diffusion-DPO and MaPO, and addresses known issues like excessive optimization and misalignment.\n2. The work utilizes DDIM and ReNoise inversion techniques from prior literature but uniquely applies them to preference optimization, demonstrating improved alignment and computational efficiency compared to earlier methods.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The proposed smoothed preference distribution appears novel and well-motivated. Incorporating DDIM inversion into DPO training is innovative, providing deeper insights into preference modeling and demonstrating notable originality.\n\n2. The experimental results are convincing and impressive, clearly highlighting the significant potential of the proposed method.\n\n3. The paper is clearly written and well-structured. The presented method achieves a good balance between simplicity and effectiveness.\n\nWeaknesses:\n\n1. While using automated reward models is practically advantageous, including explicit human validation results in the main text would considerably strengthen the claim that the proposed method enhances real-world alignment. Relying solely on reward models to capture fine-grained preference distributions appears overly simplistic, raising questions about why this method can outperform large-scale human preference datasets.\n\n2. A key contribution of this paper is adapting DDIM inversion for sampling and noise injection within DPO training. However, additional theoretical justification is required. Specifically, it remains unclear why DDIM inversion is more effective than simple noise addition during training. Moreover, discussing whether this approach generalizes beyond DPO to other training paradigms would enhance the paper's impact.", "other_comments_or_suggestions": "Please provide a clearer introduction and context for the compared methods, such as MaPO and Diffusion-KTO. Currently, these comparisons appear abruptly without prior explanation, making it confusing for readers. Clarifying why these particular methods were selected for comparison and briefly outlining their relevance to your work would significantly enhance readability and coherence.", "questions_for_authors": "1. The idea of using DDIM inversion rather than simply adding noise during training is intriguing. Could this technique be generalized and applied effectively to other diffusion model training paradigms beyond DPO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates preferences optimization for T2I diffusion models and introduces an alternative to the binary reward assignment in their method, SmPO-Diffusion. It is argued that the current preference labeling scheme lacks granularity and thus calls for a smooth reward modeling. In addition, Renoise Inversion is introduced to estimate the sampling trajectory.

The paper is rated by the reviewers as WA, R, A, A. While there are concerns over the lack of comparison to related work (e.g., PRDP, SPO and RankDPO) and insufficient study on hyperparameters, the overall feeling on the paper remains positive, evidenced by the recognition of the strong experimental results, the well motivated method design and the theoretical analysis for support, and the paper being written and well-structured. Reviewer 4khg raised a few concerns such as similarity to label smoothing, lack of proof, and lack comparisons with existing methods. The author rebuttal addressed these concerns. 

Overall, the AC recommends accept and encourages the authors to fully address the mention issues by the reviewers in their final version.
Decision: Accept (poster)