A Gradient Guidance Perspective on Stepwise Preference Optimization for Diffusion Models
Authors: Joshua Tian Jin Tee
Hee Suk Yoon
Abu Hanif Muhammad Syarubany
Eunseop Yoon
Chang D. Yoo
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Human Preference Alignment
Primary area: deep_learning
TL;DR: GradSPO reinterprets Stepwise Preference Optimization (SPO) through a novel gradient guidance lens, enabling a simplified objective and integrated noise reduction to achieve superior human preference alignment in text-to-image models.

Abstract: Direct Preference Optimization (DPO) is a key framework for aligning text-to-image models with human preferences, extended by Stepwise Preference Optimization (SPO) to leverage intermediate steps for preference learning, generating more aesthetically pleasing images with significantly less computational cost. While effective, SPO's underlying mechanisms remain underexplored. In light of this, we critically re-examine SPO by formalizing its mechanism as gradient guidance. This new lens shows that SPO uses biased temporal weighting, giving too little weight to later generative steps, and unlike likelihood centric views it reveals substantial noise in the gradient estimates. Leveraging these insights, our GradSPO algorithm introduces a simplified loss and a targeted, variance-informed noise reduction strategy, enhancing training stability. Evaluations on SD 1.5 and SDXL show GradSPO substantially outperforms leading baselines in human preference, yielding images with markedly improved aesthetics and semantic faithfulness, leading to more robust alignment. Code and models are available at https://github.com/JoshuaTTJ/GradSPO.

A Gradient Guidance Perspective on Stepwise Preference Optimization for Diffusion Models
Joshua Tian Jin Tee Abu Hanif Muhammad Syarubany
Hee Suk Yoon
Eunseop Yoon Chang D. Yoo"
Korea Advanced Institute of Science and Technology (KAIST)
(joshuateetj, hskyoon, hanif. www.f.asarubaa.... syarubany, esyoon97,
Abstract
Direct Preference Optimization (DPO) key framework for aligning text-to- image models with human preferences, extended by Stepwise Preference Opti- mization (SPO) to leverage intermediate steps for preference learning, generating more aesthetically pleasing images with significantly less computational cost. While effective, SPO's underlying mechanisms remain underexplored. light of this, we critically re-examine SPO by formalizing its mechanism gradient guidance. This new lens shows that SPO uses biased temporal weighting, giv- ing too little weight to later generative steps, and unlike likelihood centric views it reveals substantial noise in the gradient estimates. Leveraging these insights, GradSPO algorithm introduces simplified loss and targeted, variance- informed noise reduction strategy, enhancing training stability. Evaluations on SD 1.5 and SDXL show GradSPO substantially outperforms leading baselines in hu- man preference, yielding images with markedly improved aesthetics and semantic faithfulness, leading more robust alignment. Code and models are available https: btttps//gittube gggthab.ccmm/Dsshaaa////rrddd
Introduction
The rise of diffusion models has transformed the generation of high-quality images from textual prompts, representing major leap forward in generative artificial intelligence. Traditional text-to- image (T2I) models 3 typically rely single-stage training process, learning synthesize images directly from large-scale paired text-image datasets. While this approach has yielded im- pressive results, it lacks an explicit mechanism adapt outputs align with user preferences or specific application needs. In contrast, large language models (LLMs) have embraced a more nuanced multi-stage training strategy. They undergo extensive pretraining large, diverse, and sometimes noisy datasets, followed by a fine-tuning stage on datasets annotated with human preferences [4. This fine-tuning step crucial for enhancing the models' practical utility, safety, and responsiveness, all while preserving their broad foundational knowledge. Applying this two-stage training approach text-to-image diffusion models presents promising opportunity better align generated images with human preferences, resulting in outputs that more faithfully capture user intent.
Recent advances in language modeling have sparked increasing interest in incorporating human preference feedback into the training of text-to-image diffusion models. This emerging area leverages human judgments to refine model outputs, thereby enhancing both alignment with user intent and overall visual fidelity. Prominent works in this field are mostly based on Reinforcement Learning from Human Feedback (RLHF), which uses human comparisons rank generated images for guiding model training [6-9). Among these, Diffusion DPO 6 adapts the Direct Preference
*Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Optimization (DPO) framework inittally introduced for Large Language Models (LLMs) to the unique characteristics of diffusion models. This specific adaptation enables the model inherently learn to favor higher-quality images, and consequently, this strategy has demonstrated notable improvements in producing images that both visually appealing and semantically aligned with user prompts. Building on this foundation, recent work, Stepwise Preference Optimization (SPO) [TO] introduces preference learning at intermediate diffusion steps, providing more precise reward signals during the denoising process. This approach enhances training efficiency by improving the stepwise likelihood favorable transitions throughout the diffusion trajectory.
While effective, the SPO framework introduces a learning paradigm that diverges from traditional diffusion training; the latter based on distribution matching, whereas SPO relying on maximizing sample-wist likelihoods. This paper presents a critical re-examination of SPO, positing that its mechanism can be more accurately and beneficially characterized as form of Direct Preference Optimization (DPO) with respect to score functions augmented by the gradients of reward models. This novel theoretical lens instrumental: firstly, reveals an inherent bias in SPO's implicit weight- ing of generative timesteps, leading to underemphasis on crucial final-stage details. Secondly, by reframing the optimization from a likelihood-badd -based perspective to a gradient-centrie one, our approach uniquely facilitates the quantification and analysis of noise inherent in the preference-guide learning process, specifically through the variance of the guiding reward gradient approximations.
Our Contributions. Building upon the re-examination of SPO through a gradient guidance lens, this paper introduces GradSPO and makes the following key contributions:
Novel Theoretical Framework for SPO: score matching. This perspective uncovers inherent bias in SPO's weighting of generative timesteps and introduces a new method for quantifying learning noise via the variance of gradient approximations.
We formally reinterpret SPO through the lens of guided
Principled Design of GradSPO: Building on this framework, we propose GradSPO, an algorithm that incorporates: (1) a simplified loss objective with uniform timestep weighting and fixed guidance scale to directly address the identified SPO from this new viewpoint, and (2) an integrated noise reduction strategy grounded in the gradient perspective improve training stability and enhance preference fidelity.
State-of-the-Art Preference Alignment: Through extensive experiments, show that GradSPO significantly outperforms existing preference learning baselines, achieving superior alignment with fine-gre gre-grained human preferences in text-to-image diffusion models, leading to notable gains in both visual quality and semantic accuracy.
SDXL
SPO
GradSPO
Figure 1: Qualitative comparison between vanilla SDXL, SPO, and GradSPO. GradSPO demonstrates superior aesthetic quality and prompt alignment compared to both SDXL and SPO.

---

2
Notations and Preliminaries
This section introduces the foundational concepts and notation for diffusion models and preference optimization techniques relevant to our work.
2.1 Diffusion Models
Diffusion models define a forward noising process that gradually transforms a clean data sample Xg Rd into pure noise through sequence of latent variables ,,,--.;XT where T denotes the total number of timesteps. Each X, represents a progressively noisier version of xo. The forward process defined as Markov chain:
(x11r Xu)
II ((xx(xx g(xx(xx-1) = N(xi /1 1, (=1
(I)
where E (0, 1) controls the noise variance timestep useful property of this process is that X₂ can also be sampled directly from X via closed-form expression:
(xx xn) = (xxx; Va,xo, (1 oz I))
where a, 1 3r and a = II--1 as the cumulative product of a, up to timestep I.
The generative model seeks to reverse this process by learning a parameterized reverse distribution po(x X c), where denotes optional conditioning information (e.g., a text prompt). This reverse process is modeled using a neural network Eg (Xe, c,t), which predicts the noise component e that each reverse transition defined
was added to X obtain X,. Following Ho et
(x c) = N (x (, 1), 10,
Og
Bit1 V1 - ā+t
co(xu)1,,,++1))) of
1- āi
where
140
X++1
3141-
a++1
ōt+1
(2)
The model is trained by minimizing the variational bound, which reduces a simplified objective known as the denoising score matching loss:
CDDPM Exx.,et,e [a(t) ||c colVarx v1 - a,,,
where f sampled uniformly from {1....,T). and X(t) is a time-dependent weighting function. The term inside the expectation represents the squared error between the true noise and the model's prediction Co at timestep t.
2.2 Stepwise Preference Optimization (SPO)
Among various preference alignment training schemes, many recent methods adopt a DPO-style training framework 112 6 g 10, [3]. Given a preference dataset 'D, these methods often aim to minimize objective related to the DPO loss. For Diffusion DPO (6) this can be framed as minimizing upper bound the DPO loss [2], defined as:
Lpmmro E log
(-ST (DK( (DEL (q(x)", x²" xx" c) po po(" xx"" c))
DxI. (g(x²"1
xx= x c) prer(x xx)
(4)
DxL. (g(x{_1
x x/y c) Pa (x{_1 1 x(,c)
+ DD (g(x/_1
x x c) Pee (x| x c))

---

where (xx) x c) ~ D, t ~ 4 T). and samples x"" and x are drawn from (x]]
xx)) and
(x)] x6) respectively.
While the original Diffusion-DPO
6 formulation often utilizes clean preference pairs (x) x)) from offline datasets to approximate such objectives, the recently introduced Stepwise Preference Optimization (SPO) [T0] approach leverages intermediate samples generated by an online model as preference pairs. The SPO loss is typically expressed in more direct log- iigllikeihhod ratio form, analogous to the general DPO principle:
LsP -Bcx Too (3 (x]""_] x, Prer(x (x C, x,)
Po(x Pref x
C, C,X,)
log
(5)
C, Xt)
The timestep t uniformly sampled as t U[1, k] for a fixed constant x. Textual prompts с are drawn from a distribution p(c), and the initial latent state Xy is sampled from ( I). Intermediate latent states x are obtained by applying the discrete reverse diffusion process (using Eq. literatively..
The winning x and losing x samples for the transition from x; x;-1 determined by step-aware reward model :)
X argmax -(x-1,,-11) 1), x
argmin r(x--1, 1)), 1),
(6)
x
where {x are candidate samples generated from (x c,
SPO contrasts with standard DPO applications by operating on preferences over intermediate tran- sitions rather than only final outputs. By integrating feedback throughout the generation process, SPO provides denser reward signals. This stepwise learning mechanism aims to enhance training efficiency and offer finer-grained control for aligning the model's generation trajectory with human preferences while requiring significantly less computational cost.
3. GradSPO: A Gradient Guidance Perspective on Stepwise Preference Optimization for Diffusion Models
Stepwise Preference Optimization (SPO) [0]. as outlined in Section 2.2 aligns diffusion models by optimizing preferences at intermediate generative steps. Recall from Eq. 2 that the generation of X(-1 from a given and context involves sampling noise ~ (Q. such that x;-1 po (xx, (x(, c, + 022. Consequently, selecting x;-1 to maximize or minimize step-aware reward r(x-1,, 1) (Eq. 6 is equivalent to finding the optimal noise for that step. This can be expressed
as:
argmax ggmmxxr(ue(xx, c, + 022 t 1),
argmin r(uo X t) 0,2,t-1 1).
A key insight from Huang et al. [14] is the interpretation this argmax noise serves approximation of a scaled, noisy gradient of the reward model:
TV. +
(8)
where ~ (O0 I)- This approximation motivates us view SPO from reward guided perspec- tive. Within this view, an ideal reward-guided score function can be defined as dileal c, = so (x(x,, rx,,, where sa(x,, C, () = Vx, logro( (x, c) is the original model score and the guidance scale. From this point of view, SPO can be seen as approximating this ideal guidance term -v,xx t,t, relying the connection between the gradient of the reward models and 2 in Eq. Specifically, we define our approximate guided scores s as:
$g (x,, c, t) = Vx, log pe (x+|c) + 7/2
Here, = defined Eq. and 7/ a time-dependent scaling factor related to 7. Drawing an analogy to how Diffusion DPO handles preferred and rejected samples, we introduce GradSPO,

---

method that performs model alignment by extending DPO principles to operate on guided scores:
LemassoO Oref Xx, c, - -E log (3wo())[) c, 1) - sg(68 (Xt, c, 1))
- S c, sg( (x,, c, t))
(10)
- (X,,(X C, f) (x,, c, t
{X),,o((X,, C, t) sg( sg(5) (x,, t))
where (x(,,, c, is the current model's score prediction (e.g., related to EØ), Syef is the reference model's score, sg(-) denotes the stop-gradient operator (which prevents gradients from flowing through its argument), and w() is a time-dependent weighting function. The stop-gradient operator essential for stable training, as detaches the target scores, i.e., sg(sk) ) and sg((),,), from gradient updates. With these fixed targets, Eq. 10 trains the model by pulling the current score so(X,, c, t) closer to the winning score $i while pushing away from the losing score s'g. Without this detachment, the winning score would drift toward the current prediction (s% + 5g), weakening the supervision signal and destabilizing learning.
Theorem (GradSPO Loss as Upper Bound). Let LGradsPo(se, Sief, TT, denote the GradSPO loss functional as defined in Eq. 10 where T and T' are the target winning and losing scores. Let Vient = + V t) and Sydea S V x, r (x,, t) be the ideal target scores based on true reward gradients. Let sy and $ (as defined in Eq. 9 be the target scores constructed using
the approximated, potentially noisy, gradient signal. Then, the GradSPO loss computed with the approximated targets forms upper bound the loss computed with ideal targets:
Lraasso ( 59, Stefs Sideal sideaa) S. LGraasrol ( (S0, Staf 8 s))).
30
(11)
A detailed derivation of Theorem I provided in Appendix A This theorem is significant as it demonstrates that our practical GradSPO loss (the right-hand side, which we minimize) serves as an upper bound on idcalized loss formulated with exact reward gradients. Analogous to ELBO maximization in variational inference, minimizing this upper bound provides a principled approach to optimizing for the underlying clean-gradient objective. This is particularly advantageous in scenarios like latent diffusion models, where precise gradient computations Vx, rx,, t) might involve costly backpropagation through components like the VAE decoder. The GradSPO loss objective from Eq. 10 when expressed in terms of noise predictions ex(X,, c, I) (where 89 Xx, c, -eo (x,, C, 1)L/vV 0g can be written
CGmasPo = -E log σ -Ba(t) (IEoo c, sg(Eo(X, c,t) Act)
|lcer(Xee, c, sg(co(X), c,t) Act)
(12)
| c, sg(co(X), c, -
+ +Xkeer c, sg(co(xi,, c, AC))
where Act -1V1 ā, a(t) is time-dependent weighting function, and 33 is scalar efficient. Notably, the standard SPO loss can be interpreted within this guided noise prediction framework by specific choices of a(t) and y, (see Appendix for details).
This reinterpretation of SPO through a gradient guidance perspective offers critical insights into its behavior and limitations. For instance, it reveals that SPO's interval-based training [IO) can be viewed as an application of the recently introduced interval guidance [15), which has been shown to improve sample quality and diversity.
More critically, this re-framing reveals that the effective weighting function a(t) in the original SPO formulation (as shown in Figure 2 disproportionately underweights the later diffusion steps. similar pattern emerges when analyzing the Pick Score difference between the base model and its
counterpart: the improvement decays exponentially over time, even though later steps are crucial for capturing fine-grained details. This suggests that SPO primarily enhances early timesteps, with minimal gains in the later stages of the denoising process. To address this imbalance and ensure more uniform supervision across timesteps, we adopt a simplified objective in which both att) and y are set to constant values.
5

---

0.008
Score Difference
4-
34
0.000
600
400
200
600
400
200
Timestep
Timestep
(a) Timestep weighting function
difference
Figure 2: (a) Timestep weighting function a(t) used SPO. (b) Pick Score difference between SPO and the base model across timesteps.
3.1 Noise Reduction Techniques
The 2 values (Eq. 8). which serve as noisy reward gradient approximations also include an inherent noise component signals, potentially hindering training performance. To address this, we propose a technique to reduce noise in the estimated gradient direction. Recall z s = Vx, t). The term +/T3 Vx. (Xx, t) is the desired gradient signal. We define a noise-reduced estimate as:
_N, I). This inherent noise in = can introduce variance into the target
(13)
mmax-min
2
This estimator leverages the opposing nature of the target gradient components in 2 and 2
Theorem 2 (Variance Reduction of Estimated Gradient Signal). Let = G + and z- -G+ 22, where = /II V. rxx, is the true scaled gradient signal, and 2ı, 22 NO, 1) are independent noise terms. The noise-reduced estimate max-min has variance half that of the original estimation ==
Var(=
Var Var mmxx--min)
(14)
2
proof is provided Appendix This demonstrates that maxx-min offers more stable estimate of the desired gradient signal, leading to more stable training process that converges effectively, resulting in the generation of better aligned images.
While noise reduction techniques are beneficial, noise cannot be entirely removed. The persistence of this inherent noise in the gradient estimates necessitates the development of training algorithms that are robust to its presence. Towards this end, to further enhance stability and promote convergence, we employ an Exponential Moving Average (EMA) of the model parameters 0 during training and online sampling. The EMA parameters GEMA are updated as:
PEMA + μωEMA + (1 - 11)0,
(15)
where E [0, 1) is decay rate (e.g., 0.999). EMA parameters often provide a more stable representation of the learned model particularly in noisy environments [16 17..
By incorporating these noise reduction strategies, we arrive at the final GradSPO framework. The target noise predictions in the loss function are now constructed using the EMA model's prediction 6BEM and the noise-reduced gradient signal =max-minn The resulting GradSPO loss is:
LGaadSPO = -E log a (Ba(t) ((Ee())(leexxe,,) c, sg(sgg((sssn (x1, c) - A +tduced) ||
+
||eur(Xx, c, sg(Come (Xe, c, A dt 1
(16)
||ee(Xe, c, () -g((E (Xe, C, A (educedd >
+llexa(x,, c) sg(Ceeus c,t) Edduced l1
where = - = T 04 max min This changes brought about by this new viewpoint on SPO substantially improves the alignment capabilities of stepwise preference optimization.

---

Experimental Results
4.1 Experimental Setup
Datasets and Models. We fine-tune both Stable Diffusion 1.5 [8] (Creativeml-openrai-n License) and SDXL. [99] (Openrail++ License) models using the GradSPO objective, detailed in Section 3 Following the SPO training scheme, we train the models on 4,000 randomly sampled prompts from the Pick-a-Pic vI dataset [20), which contains 580,000 pairs of image preference for each prompt. For evaluation, unless stated otherwise, we used the test set consisting of 500 prompts sourced from the Pick-a-Pic v2 dataset, similar to previous work in the field
Implementation Details. Since GradSPO builds upon the SPO framework [0], we retain SPO's s base hyperparameterss to ensure fair and direct comparison. Additionally, because we do not modify the stepwise-aware preference model, we reuse the same reward model from SPO rather than training performance: the time-dependent weight function Og is set to 1, the guidance scale 7/e is fixed at 0.5, and the Exponential Moving Average (EMA) decay rate μ
one. However, GradSPO introduces several unique hyperparameters to optimize
0.9. These settings are applied consistently across both SDXL [19] and Stable Diffusion 1.5 [18] models. A complete listing of all hyperparameters used in our experiments can be found in Appendix D for reproducibility and further reference.
Baselines. To evaluate the effectiveness of GradSPO in aligning with human preferences, benchmark its performance against several strong baselines. These include the original pre-trained models, Stable Diffusion 1.5 18] and SDXL [19), well as several recent preference learning methods: Direct Preference Optimization (DPO) 12], Stepwise Preference Optimization (SPO) [T), InPO (8), and MaPO [13.. For consistency across comparisons, utilize publicty available pretrained checkpoints for all baseline methods and apply identical evaluation protocols.
Evaluation. We assess model performance using four widely accepted metrics that quantify alignment with human preferences: HPS v2 [2 (Apache-2.00 License), PickScore P2] (MIT License), Aesthetic score (23] (MIT License), and Image Reward 24 (Apache-2..) 2.0 License). For each prompt in the evaluation set, we compute and report the average scores across all compared models to provide a comprehensive view of performance. In line with the evaluation procedure used in SPO [10], generate images for all models using DDIM sampling [25] with 20 diffusion steps and a classifier-free guidance scale of 5.0 (26). This consistent inference setup ensures fair comparisons of image quality and alignment metrics across methods.
4.2 Quantitative Results
Table 1 presents quantitative comparisons of GradSPO against established baselines on both SD 1.5 and SDXL. backbones. For the SD 1.5 backbone, GradSPO demonstrates strong performance, surpassing most existing methods. While InPO achieves a marginally higher score the Image Reward metric, it is important to contextualize this: GradSPO builds upon the SPO framework, which itself registers the lowest Image Reward among the compared alignment techniques. Despite this foundational starting point, GradSPO substantially elevates SPO's Image Reward score to 0.4747, securing the aecondhiighess position for this metric. Furthermore, GradSPO markedly improves SPO's s aesthetic score, underscoring the significant advantages of reinterpreting SPO through our gradient guidance perspective.
On the SDXL backbone, GradSPO's performance is particularly compelling, demonstrating clear superiority across all evaluated metrics. It achieves, for instance, a leading Aesthetic Score of 6.2985 and a Pick Score of 28.93. The improvement over its direct precursor, Stepwise Preference Optimiza- the HPSv2 score from 28.27 (SPO) to 28.93 (GradSPO). These
tion (SPO), is sggificant-bbostiin outcomes strongly affirm the benefits of our gradient guidance reinterpretation of SPO, particularly its advantages over conventional likelihood baaede optimization strategies.
To further assess the effectiveness of our GradSPO training paradigm on SDXL, we conducted user study with five judges using 100 prompts randomly sampled from the HPSv2 benchmark [21]. For each prompt, participants were shown two images, one generated by GradSPO and the other by a competing method, and asked to indicate preference or select tie if neither image was clearly better. Judgmentss were based on three criteria: overall image quality, image-text alignment, and aesthetic appeal.
7

---

Model
Method
HPSv2
Pick Score
Aesthetic Score
Image Reward
Baseline
26.26
20.62
5,2687
0.0741
DPO
26.56
21.01
5.3704
0.2704
SD 1.5 18
InPO
26.86
21.21
5.4674
0.5135
SPO 10
26.47
21.11
5.5898
0.1945
GradSPO (Ours)
26.86
21.38
5.7651
0.4747
Bascline
27.06
21.85
5.8253
0.4749
DPO 6
27.81
22.41
5.8412
0.7466
MaPO R3
27.30
21.95
5.9684
0.5868
SDXL
InPO 8
28.07
22.46
5.9046
0.8546
SPO
10
28.27
22.93
6.2236
0.9982
GradSPO (Ours)
28.93
23.45
6.2985
1.0861
Table 1: Comparison of GradSPO with baseline methods SD 1.5 and SDXL backbones. GradSPO attains highest scores across most human preference metrics, demonstrating superior alignment visual quality. For each metric, the top-performing method bolded, while the second-hest is underlined.
General Preference
16.7
GradSPO
Visual Appeal
53.2
30.9
15.9
Tie
SOXL Win
Prompt Alignment
50 Percentage (%)
100
Preference
GradSPO Win
Visual Appeal
26.4
55.0
18.7
lie
Prompt Alignment
56.2
21.1
22.6
50
100
Percentage (%)
Figure
User study results comparing GradSPO against two baselines. Top: comparison with SDXI.. Bottom:
comparison with SPO.
As summarized in Figure 3 the results indicate a clear human preference for images generated by GradSPO, reflecting notable improvement in overall quality. GradSPO achieved win rate of 75.6% against SDXL and 53,0 33.0% against SPO. Notably, the strongest gains were observed in aesthetic quality, where GradSPO attained win rates of 81 81.1% % and 56.2% against SDXL and SPO, respectively.
4.3 Qualitative Results
Figure illustrates the qualitative performance of model. GradSPO (using the SDXL backbone), compared to other Text-to-lmage (T2I) methods. In the first row, for the prompt "The Best," GradSPO successfully generates image accurately containing the desired text, outperforming baseline methods. Notably, while the InPO approach nearly generates the correct text, it includes visible artifacts. For the second row, corresponding methods (excluding InPO) fail to generate images following the provided textual prompt. However, comparing GradSPO and InPO, GradSPO produces more aesthetically pleasing image. Lastly, in the final row, all baseline methods fail to effectively render the "A gorgeous queen" following the textual prompt, whereas GradSPO reliably and accurately generates the image. Collectively, these examples in Figure qualitatively demonstrate GradSPO's superior capability in prompt adherence, text rendering, and aesthetic quality compared to baseline methods.
the prompt "Buff Harry Potter," most baseline
Ablation on EMA Momentum and Reward Guidance Scale
To better understand the performance and stability of GradSPO, we conduct ablations on two key hyperparameters: the EMA momentum and the reward guidance scale 7. All other settings are fixed for SD 1.5, with results reported on the Pick-a-Pie dataset. When varying f4, we set y 0.5. and when varying y, fix 0.9. The results summarized in Table

---

Text
SDXL
Diff-DPO
MAPO
SPO
InPO
GradSPO (Ours)
Photo takon throuugh social media of bloude woman holding a sign with "the Best" word written on
THE BET BOST THE ST BESST
BA
THE BEST
THE ST
BEEST
THE ST
THE BEST
Buff Harry Pottor
A gorgeons oucen with cat oyes
Figure 4: Side-by-side comparison of images generated by related methods using SDXL. GradSPO demonstrates significant improvement terms of aesthetic appeal and fidelity the caption.
HSP+2
Pick Score
Aesthetic
ImageRwd
14
HSP+2
Pick Score
Aesthetic
ImageRwd
0,25
26.67
20.71
5,5278
0.452
0.90
26.86
21.38
5.7651
0.4747
0.50
26.86
21.38
5,7681
0.4747
0.95
26.74
21.12
5.6352
0.4153
5.4630
21.22
(a)
(b)
Table 2: Ablation study of GradSPO on SD 1.5 (Pick-a-Pic v2). (a) Varying the reward guidance scale with EMA momentum fixed at μ 0.9; (b) Varying the EMA momentum j with reward scale fixed 7 0.5.
We observe that GradSPO is relatively robust to changes in the EMA momentum μu, showing only minor performance degradation at higher values. attribute this to an oversmoothing effect, where the EMA model lags behind the current model and adapts too slowly.
In contrast, the reward guidance scale y exerts stronger influence. A large y increases the variance of the reward signal, especially in noisy regions, leading to less stable convergence. Conversely, very small ? weakens the learning signal, preventing the model from capturing meaningful preferences. Empirically, we find 0.5 provides a good balance between stability and signal strength, and adopt this setting across both SDXL. and SD) 1.5 experiments.
4.5 Effectiveness of Gradient-Guided Objectives and Noise Mitigation Strategies
To dissect the contributions of our gradient guidance perspective on Stepwise Preference Optimization (SPO) and to validate the efficacy of the proposed noise mit- igation strategies, we conducted systematic ablation experiments. The results for experiments conducted on
SPO
SPO+Simple
0.9
SPO+Simple+Marmin
GradSPO
Win rate vv.. SD 1.5)
the SDXL backbone, presented in Figure 5 illustrate the progressive performance enhancements (measured by win-rates across diverse automated reward models) as each component of GradSPO is incrementally intro- duced, starting from a standard SPO baseline.
0.7
First, adopting the simplified objective derived from our cuudient-guided reinterpretation (denoted SPO+Simple) led to improvements across most metrics, with only slight drop in aesthetic score. This highlights the advan- tage of our reformulated objective, including uniform timestep weighting.
0.6
HSPV2
PickScore
Aesthetic Score Image Reward Average
Figure 5: Win-rate comparisons across SPO- Simple, S-i-SSiimaaaaaieee + and GradSPO on popular reward metrics.
Subsequently, integrating the max-min noise reduction technique for the gradient signal, - min (termed SPO+Sinple+Maxmin) yielded further discernible boost in performance, increasing the

---

average win rate across all 4 metrics. This gain is primarily attributed to the reduced variance in the estimated reward gradients, demonstrating the effectiveness of this stabilization approach.
Finally, the incorporation of Exponential Moving Average (EMA) for model parameters during train- ing (Eq. [5), which constitutes the full GradSPÓ method, achieved the highest average performance among all configurations. This result highlights EMA's crucial role in further stabilizing the training dynamics within the inherently stochastic preference learning landscape.
In summary, the ablation study (Figure 5) demonstrates that each core component of GradSPO-the simplified gradient-guided objective, max-min noise reduction, and EMA makes distinct and measurable contribution to the overall performance
Related Works
5.1 Diffusion Models for Text-to-lmage Generation
Denoising Diffusion Probabilistic Models (DDPMs) [11] are class of powerful generative models that have recently gained popularity for their ability to effectively model high-dimensional data. This has led to their successful adoption across a range of applications. including image synthesis [ 19], video generation (27-29), and text-to-speech synthesis (30, Among these, text-to-image generation has been particularly impactful, enabling the creation of complex visuals directly from textual descriptions [2 3] and unlocking new possibilities in creative fields such as digital art and design. Despite their ability produce high-quality images after pretraining, these models often struggle to capture nuanced human preferences, frequently resulting in visual artifacts such as poorly rendered hands and faces 7 These limitations have spurred ongoing research into improving
sampling efficiency
B3 35 and enhancing alignment with textual inputs (6] 10 g
5.2 Human-Preeerence Alignment for Diffusion
Human preference alignment has long been recognized as beneficial in Large Language Models (LLMs), where techniques like Reinforcement Learning from Human Feedback (RLHF) have substan- tially improved performance, helpfulness, and safety [5 4 12 36 37.. Inspired by these successes, recent work has explored applying human preference learning to text-to-image generation (6, A prominent example is Diffusion DPO [6], which adapts Direct Preference Optimization (DPO) for diffusion models, steering the model toward preferred images and away from dispreferred ones using an offline dataset. While Diffusion DPO significantly enhances text alignment and aesthetic quality, generating high-quality images, it demands considerable computational resources. More recently, Stepwise Preference Optimization (SPO) [0]] was introduced, performing preference learning at intermediate diffusion steps on a per-sample basis. This approach provides stronger learning signals, greatly improving computational efficiency and text alignment.
Despite SPO's s success, its sample-wise formulation deviates markedly from the typical score matching training paradigm used in diffusion models (II. In this work, we reinterpret SPO through a score matching lens, showing that rather than maximizing sample-wise likelihoods, SPO effectively Icarns to move towards preferred scores and away from dispreferred ones. This perspective exposes limitations in the original SPO training framework. Building on this insight, we propose GradSPO, a score matching inspired approach that substantially enhances image generation quality.
Conclusion
In this paper, we revisit Stepwise Preference Optimization (SPO) through the lens of score matching. We establish novel theoretical connection, demonstrating that SPO is equivalent to Direct Preference Optimization (DPO) when using winning and losing score functions derived from reward models with added noise. Leveraging this perspective, we propose a simplified and more intuitive optimization objective, alongside effective noise reduction techniques that significantly mitigate approximation errors caused by such noise. Empirical evaluations demonstrate that our proposed method, GradSPO, consistently outperforms existing preference learning approaches, highlighting its superior capability in generating images that align closely with human preferences. Furthermore, this new interpretation of SPO links gradient guidance to the SPO training objective allowing for the integration of improved gradient guidance techniques for more user aligned images which we leave for future work.
10

---

7 Acknowledgements
This work was supported by Institute for Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea MoXernmen((MSIT) (No.RS-2021- II21 1381, Develop- ment of Causal AI through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSTT) (No.RS- ARR-2022-112200) 18, Development and Study of Al Technologies Inexpensively Conform Evolving Policy on Ethics)
References
[1] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 22563-22575, June 2023.
[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. URL http::ttpp://rrtt org/ /bs/
2204.06125.
[3] Black Forest Labs. Flux.
lamstttlab/fffxu 2024.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. URL https: /tatps///rrxvv org/ abs/ 22112.80733
[5]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreett For
net/
[6] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8228 8238, June 2024.
[7] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge. Jiaxin Chen. Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8941-8951, June 2024.
[8] Yunhong Lu, Qichao Wang, Hengyuan Cao, Xierui Wang, Xiaoyin Xu, and Min Zhang. Inpo: Inversion preference optimization with reparametrized ddim for efficient diffusion model alignment, 2025. httptttps///artt
18454
[9] Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. DSPO: Direct score preference optimization for diffusion model alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL httt:::://pperrrrii..eee///torre

---

[10] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-tramnig diffusion models from generic preferences with step-by-step preference optimization, 2025. BRI.https:: htttps::/saaii.orrrrrt/ wwt.orgggab.//sooeae 04314
[11] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020. URL htR./tttps::/prcceedaage neurips. cc/ cc/paper__.. _f files/ 4cbbccec664440067777110000000ed2
[12] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Shirty-seventhh Conference on Neural Information Processing Systems, 2023. URL httpss//ppeneevieeeeeeeefoooo
[13] Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thorne, and Jongheon Jeong, Margin- aware preference optimization for aligning diffusion models without reference, 2024. URL
https://arxiv.ou org/ abs 06424
[14] Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli Shama Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non-differentiable rule guided diffusion. ICML, 2024. URL http::t//ppeereviiiii net/
forum?id g
[15] Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. The Thirty-eighth Annual Conference on Neural Information Processing net met/foorumiin
Systems, 2024. URL
[16] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. URL https://arxiv> org/ abs 2203. 01469
[17] Daniel Morales-Brotons, Thijs Vogels, and Hadrien Hendrikx. Exponential moving average of weights in deep learning: Dynamics and benefits. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL /ttlhtts://poeereiiieee .ett://ooruuiiid-MMMnnnaa
[18] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[19] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[20] Yuval Kirstain, Adam Polyak. Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663 2023.
[21] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.0334/, 2023.
[22] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset user preferences for text-to-image generation. 2023.
[23] Christoph Schuhmann. Laion-assthetics.
2022. Accessed: 2025-05-15.
[24] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image gente--aatagggeeeattiionon Advances in Neural Information Processing Systems, -5955 2023.
12

---

[25] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL /ttps//////ooo net/forumg Aet-StlgiarCii
[26] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL /ttLitst/////eeeeeiii het//forun/ido-gwBBkxtttbl
[27] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
URL httph: org/ abs///b///// ab3//31.. 15127
[28] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Zhang, Weihan Wang. Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with expert transformer. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/ http:/enrwiiieeeee//eeeeeeeeeeee 2tt
[29] Sunjae Yoon, Gwanhycong Koo, Younghwan Lee, and Chang Yoo. Tpc: Test-time procrustes calibration for diffusion- human image animation. Advances in Neural Information Processing Systems, 37:1 18654-1641111 10---1187777 2024.
[30] Vadim Popov, Ivan Vovk. Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- its: A diffusion probabilistic model for text-to-speech, 2021. URLhttps: //ap:::///axxxv org/abs/ 2105. 06337
[31] Matthew Le, Apoory Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text-guided multilingual universal speech generation at scale. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL http::////eeeeee.neeet..e. gzC5252hCO
[32] Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, sheng zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, 2024. URL httpp:///ppenrevier.. www.:iiwww.eeee.ooooeeeee www.:eeeeoouinte.. 2ttp///rrrmmddddac//aaa.....
[33] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors. Advances in Neural Information Processing Systems, 2022. https: //penreviie net/forru?idd
[34] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. URL https: attts://arxxi //ar:iv///iii.//aaa org/ abs/ 2303. 01469
[35] Joshua Tian Jin Tee, Kang Zhang, Chanwoo Kim, Dhananjaya Nagaraja Gowda, Hee Suk Yoon, and Chang D. Yoo. Physics informed distillation for diffusion models, 2024. URL
https://openreeieew..ttt httppp///pppeeeeeeerreeee//r//aa http://www....tteooooooo///////p
[36] Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung- Woon On, Mark A Hasegawa-Johnson, Sungwoong Kim, and Chang D Yoo. Tler: Token-level continuous reward for ann-ggained reinforcement learning from human feedback. arXiv preprint 2rXiv:2407.16577 2024.
[37] Hee Suk Yoon, Eunscop Yoon, Mark A. Hasegawa-Johnse Sungwoong Kim, and Chang D. Yoo. ConfPO: Exploiting policy model confidence for critical token selection in preference optimization. In Forry-second International Conference on Machine Learning, 2025. URL https://openrevie:: nett/forum2dd227770pp3333
[38] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https //arxiv. //arxiv.rggaabs/ 05101
13

---

[39] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022. URL, https:: /tattps///axivv org/ abs/ 2206.10789.
[40] Yutong He, Naoki Murata, Chich-Has Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion, 2023. URL
16424
[41] Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. Cfg++: Manifold-conssrainee classifier free guidance for diffusion models, 2024. URL https://
arxiv. orxiv.org/aoo ab///406..2.2... 08070
[42] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7441-7451, October 2023.
[43] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 11472- 1 14811, June 2022.
14

---

Contents
Proof of Theorem
Proof of Theorem 2
3
Link Between GradSPO and SPO
D Additional Experimental Details
4
Limitations
Evaluations on Parti Prompts and HPDv2
Comparisons with Clean Gradient
Connection Between Maximal/Minimal Noise and Reward Gradient
GenEval Results
J
Effect of Timestep Weighting
Additional Qualitative Results

---

A Proof of Theorem 1
Theorem (GradSPO Loss as Upper Bound). Let LGraaspo(se, Sref T denote the GradSPO loss functional as defined in Eq. 10, where T and T are the targel winning and losing scores. Let ideet = Sø + XVx,T(X,, t) and Syeetl se xx.x. xxx,(xx, t) be the ideal target scores based on true reward gradients. Let Sø and 8 {as defined in Eq. 9 be the target scores constructed using the approximated, potentially noisy, gradient signal. Then, the GradSPO loss computed with the approximated targets forms an upper bound the loss computed with ideal targets:
Lemdssool su, Srefs Siteat ddea)) S Lerassol (8g, Srafs 86 8%)
(17)
Proof. From Eq.
we have:
[((7/=-)) ty Va, r(x,,)) V Var(T
(18)
It follows that:
E E( u.t Var(s I.
ideal
(19)
Next, consider the GradSPO loss in Eq. 10 (where we omit the stop gradient operator sg(.) for simplicity):
LGraaSPOO (80 Stefs Iddal dca)) = log a (BBr(()) (100 (
c, - Sdcall (x,, c, 4)11
- |/seet c,t) sideal (xe,, t)
(20)
[]50 c,
S
(x5e, (xe,, C, t
+|[ c, s/acal (Xe. 00)11)
Using E((y) - and (E(3)) S/acat we can rewrite the loss as:
LeradSPO (onuusooss9,) (S0, Stet: ideall ((ca) = E log a (Bww(t) [1s0 (1150 c, t) E (xe, c, t)]1
||stor(X,, c,t) - E[3 (x t)]1
(21)
||s0(Xe, 15ss' Xxx, 1)]
+||sne(xx,, c,t) Exs c, t]]]33
since
E (Xee c, 35" (xr, c, 1) = =|[[[rer(xx Xr, c, () - c,)122 (ar(3) c, I));
where denotes the dimension of the variable $9. Since the variance terms equal, they cancel each other out, allowing us to move the expectation out of the norm:
LGradSPo sg, Stefi Sideals s
- log or
(logoo(Buii (-Bw(t)
(Xi, c,t) 8/ (x,, c, 1
E Ster (x,, C, -8 (x,, c, t
(22)
E|||sa (X c, 8/ c, [
[ell c, 5%,,,,, c, 00111

---

Finally, applying Jensen's 's inequality to the convex function - --) yields
GradsPO (sa, Stef; Sdeat (deul) S -B log o (-Bw(t)) (xe, c,t) - 38 (x,, (xx,,,t))) c,
|xtrr(Xxe c,t) 88 (xx, C, 1)l
|s50 C, 86(,,, c, t)
(23)
||stet (Xe, c, (x,, t))l
GGaasPO (80, Stef; 88 8%).
Thus, the approximate GradSPO loss serves as upper bound on the exact variant.
B Proof of Theorem 2
Theorem 2 (Variance Reduction of Estimated Gradient Signal), Let 2 -G+ 22, where G V t) is the true scaled gradient signal, and 21, 22 NO, are independent noise terms. The noise-reducee estimate = estimation z=:
+ 2 and x
max min has variance half that of the original
Var(=*
Var 2 maxx-mia)
(24)
2
Proof.
We are given two noisy estimates
= G and = 22.
where 21, ~ .N(0 I) are independent. Consequently,
x ~(G,1 and x ~ NI-G,I),
so Var(=*)= = Var(=-) = I. We define the noise-reduced estimate
max min
2
First, to see that this estimator is unbiased, note
= - - - +G.
max-min
Next, compute its variance, observe that = and =T are independent, so
Var + +
1 1 4 (Var(=*) Var(2 (1 -
Var - min
Thus,
1 Var(=+) = Var(="),
1
Var = min)
showing that the variance of - min exactly half the variance the original noisy estimates = or Therefore, + = is unbiased estimator of =G whose variance reduced by a factor of compared to the individual noisy vectors.
C Link Between GradSPO and SPO
The Step-by-step Preference Optimization (SPO) framework utilizes the Denoising Diffusion Implicit Models (DDIM) (25] transition probability. This probability, (x (xx), from X, to X is defined as Gaussian distribution:
(x N (x c,t)
(25)

---

where
Xx (xi
vI (xea(
Ma(x;, c, vā--1
+ 0x- - of co(x,, c, t).
111-0-1--0
(26)
Voa
In this formulation, the terms of āe, and follow the definitions provided in Section 2.1. Given these definitions, the log- reg-robability of specific sample x can be expressed as:
1
log Pe x [x,) = 207
Me (x c, x +C,
(27)
where the constant C' is given by C' = - log (2Ta?), with d being the dimensionality of the variable Xt.
Rewriting the log-probability where x 1 is expressed using auxiliary variable ==:
P [x,) = 20,
[xx c, 1) - uo c, o, 12 + CC
(28)
This expression be further reformulated in terms of the noise prediction cØ cxtx
log Po [x = -(1) |x c, () - (Ea (x,, c, B(t) - Rz 11 C.
(29)
The time-dependent coefficients A(t) and (tt) are defined as:
A(t) = 20 Vār
Øl-1 -
(30)
B(t) := VI- Vāe-i(L-ar) Vol
at
(31)
V1 -ā o0
By comparing the rewritten form log Po (x (x,) with the objective function of GradSPO in Eq. 12 it becomes evident that SPO can be viewed specific instance of GradSPO. The correspondence is established by setting the GradSPO parameters a(t) and "/t as follows:
a(t) A(t) and 2/2 Btt).
(32)
This demonstrates that GradSPO can be considered a generalization of the original SPO method, where SPO emerges under a particular choice for the functions a(t) and ?r-
D Additional Experimental Details
Hyperparameters
1.5
SDXL
I Learning rate of epochs
6e-5
lc-5
10
Batch size
40
16
0.9
0.9
10
10
[0, 750]
[0, 750]
LoRA rank
4
64
efg during training
5.0
5.0
of samples per step Sampling steps during training GPU Setup
20
20
4x NVIDIA A100
4x NVIDIA A100
Table 3: Hyperparameter settings used for SD 1.5 and SDXL. backhones.
We generally adopted hyperparameters aligned with those used in SPO, training all models with AdamW W [38 using a weight decay of 1e-4. Additional hyperparameter details are provided in Table For SDXL, however, we deviated from the standard multi-step preference optimization procedure of SPO, instead employing a single-step approach analogous to the one utilized SD1.5. Although the multi-step preference optimization typically yields superior outcomes compared to the single- step method within SPO, our single-step procedure for SDXL. nonetheless achieved performance surpassing that of SPO.

---

E
Limitations
While GradSPO significantly improves upon the original SPO method, our current implementation remains restricted single-step preference optimization and has not yet been extended multi-step preference optimization. Nonetheless, GradSPO consistently outperforms SPO, even though SPO employs multi-step preference optimization with models such as SDXL,, underscoring the robustness and effectiveness of GradSPO's gradient-based viewpoint. Additionally, although GradSPO training does not necessitate an explicit image dataset, relies on a prompt dataset and trained reward model, rendering it susceptible to biases inherent in the training data. These biases can propagate throughout the optimization process, potentially affecting the quality, fairness, and generalizability of outcomes. Despite these limitations, GradSPO provides valuable thcoretical insights by generalizing and improving upon SPO, clarifying both the fundamental advantages and inherent challenges associated with stepwise mefferencebbssed optimization techniques.
Evaluations on Parti Prompts and HPDv2
This section presents additional comparative evaluations conducted using the Parti Prompts [39 consisting of over 1600 prompts and HPDv2 [2] datasets consisting of 400 prompts. The results,
Model
Method
HPSv2
Pickscore
Aesthetic Score
Image Reward
Baseline
26.29
20.63
5.2191
-0.2901
DPO 6
26.71
21.22
5,4662
0.1179
SD
InPO
27.60
21.62
5.6009
0.6178
SPO 10
27.25
21.65
5.8101
0.2975
GradSPO (Ours)
27.70
21.93
5,8273
0.5325
Baseline
27.63
22.58
5.9639
0.6726
DPO 6
28.28
23.03
6.0127
0.9613
MaPO 13
27.92
22,66
6,0699
0.7413
SDXL 19
InPO
28.78
23.18
6.0504
0.9599
SPO 10
29.00
23.75
6.1341
0.9441
GradSPO (Ours)
29.55
24.26
6.3470
1.1498
Table 4: HPDv2 results. Results for GradSPO preference alignment baselines on HPDv2. For each metric, the top-performing method bolded, while the second-bor boand-bestti is undertined.
summarized in Table clearly illustrate the superior performance of GradSPO on the HPDv2 dataset across both SDI.5 and SDXL models. GradSPO significantly outperforms the baseline SPO method, achieving notably higher average scores 2cores-29.55 for HPSv2 and 24.26 for whchccore-whic underscores its efficacy in enhancing alignment with human preferences.
Model
Method
HPSv2
Pickscore
Aesthetic Score
Image Reward
Baseline
26.50
21.27
5.1481
0.0505
DPO
6
26.78
21.56
5.2137
0.2274
SD 1.5 18
InPO g
27.62
21.85
5.4610
0.6067
SPO 10
27.16
21.72
5.5289
0.3986
GradSPO (Ours)
27.49
21.99
5.6794
0.59S1
Baseline
27.38
22.24
5,5929
0.4864
DPO 6
28.29
22.83
5,6826
0.9943
MaPO
27.72
22.41
5.7819
0.6793
SDXL 19
InPO
8
28.48
22.87
5.7093
0.9252
SPO 10
28.78
23.32
6.0509
0.9762
GradSPO (Ours)
29.35
23.80
6.1202
1.1005
Table 5: Parti prompt results. Comparison of GradSPO and preference alignment baselines evaluated on Parti prompts. The best dest-perfommig method for each metrie is highlighted in bold, and the runner-up up underlined.

---

As detailed in Table GradSPO performs comparably to the leading method, InPO, when evaluated with the SD1.5 model on the Parti Prompts dataset, showing only minor differences. Crucially, GradSPO consistently outperforms SPO, highlighting the advantages of our proposed gradient-based optimization approach. Furthermore, when using the SDXI. model with the Parti Prompts dataset, GradSPO surpasses all baseline methods, confirming its robust ability to generate visually appealing images.
Overall, GradSPO achieves state-of-the-art or highly competitive results across various reward metrics for both datasets. The consistent and significant improvements over the original SPO variant further validate the effectiveness and practical benefits of viewing SPO from this gradient- optimization perspective.
G Comparisons with Clean Gradient
In this section, we present an ablation study comparing GradSPO variant trained with exact gradi- ents, which we term "Clean Gradient." Unlike GradSPO, which uses a noisy gradient approximation, Clean Gradient utilizes exact gradients obtained by backpropagating through the reward model. We
observed instability when training with gradients with respect X, (i.e., Vx,). consistent with prior findings 40 4T. Consequently, for our Clean Gradient experiments, we used gradients with respect to the estimated clean latent, Xy (i.e., Vx,), for the Clean Gradient experiments.
Method
Comp. Cost
Reward Metries
GPU Mem. (GB) Time HPSv2
Pickscore Aesthetic Score Image Reward
Clean Gradient
52
25
27.11
21.67
5.7058
0.5211 0.4747
GradSPO
28
17
26.86
21.38
5.7651
Table 6: Comparison of computational costs and performance metrics for GradSPO and Clean Gradient Pick-a-Pic v2. Best results per metric are bold. Computational costs per optimization step were measured using a single NVIDIA A100 GPU with a batch size of 5.
Table demonstrates that Clean Gradients generally outperforms GradSPO across most metrics, with the notable exception of the aesthetic score. We primarily attribute this performance gap the inherent noise in GradSPO's gradient estimation. This highlights the critical role of noise reduction strategies of GradSPO in narrowing this performance difference. Despite Clean Gradient' ss better performance, Clean Gradient incurs higher memory cost, it requires backpropagation through both the VAE and the reward model (see Table [) In contrast, GradSPO offers a more memory-effieient alternative that remains applicable even when the reward model isn' sn't explicitly differentiable.
H
Connection Between Maximal/Minimal Noise and Reward Gradient
This section briefly explains the connection between the maximal and minimal noise, z=, and the gradient reward model, Vz,r(xe; (). From an energy-based perspective, the reward function models the conditional likelihood:
(x((Xe
ply xx)
(33)
Z
where Z is the normalization factor and À serves as a guidance scale.
From an optimal control viewpoint, guiding the model to sample from p(y x;) equates to sampling from the augmented stochastic differential equation (SDE):
1 T T3, log di VTB, (u" (x,,x, f)dt dw), dx,
(34)
where u²"xx (xx, L) denotes the optimal control function minimizing a cost function (negative of the terminal reward), and du denotes standard Brownian motion.

---

As detailed in Huang et al. [1] (Section 4.4), this optimal control function is connected to the reward gradient:
(x,, t) = VT3, logp(y x()
(35)
=VTTA x, rxx,
Additionally, the noise maximizing this reward (Eq. 7 Z + provides an upper bound to this optimal control function, u" (x,, f). Hence, [4] employs an approximation for the sum of control and Brownian increments (see Section 4.3
(14)):
(u" (xx, t) + 2) se 2 dt.
(36)
similar reasoning can be applied noise minimizing the reward, approximating the negative reward gradient Vx, rxx,, with minimal noise =
I GenEval Results
To further assess the compositional and perceptual capabilities of our models, we evaluate them on the GenEval benchmark using SDXL-based backbones. As shown in Table GradSPO achieves notable gains over SPO-not not only aesthetic quality but also compositional fidelity fidelity-reuulting in a higher overall score. However, consistent with prior findings from the SPO study, GradSPO still falls short of Diff-DPO in overall GenEval performance. This suggests that, like SPO, GradSPO tends to prioritize aesthetic reward signals over fine-grained compositional alignment, despite outperforming Diff-DPO on all individual image-level reward metrics.
Method
Single
Two
Count
Color
Pos
AttrBind
Overall
SDXL
97.81
68.43
40.62
86.70
12.00
23.00
54.76
Diff-DPO
99.69
81.06
48.44
89.63
13.25
27.75
59.84
SPO
97.81
73.74
41.25
86.44
13.00
20.25
55,41
GradSPO
99.06
77.78
47.50
88.03
13.00
22.25
57.07
Table GenEval results for SDXL. models. Each column reports accuracy (%) for specific evaluation dimension, and final column shows the overall GenEval score.
J Effect of Timestep Weighting
To investigate the impact of different timestep weighting strategies on preference-based diffusion training, compare three commonly used schemes: min-SNR weighting [42), P2 weighting [43], and uniform weighting. Table presents results for GradSPO trained on SD 1.5 using the Pick-a-Pic v2 dataset.
Unlike standard diffusion training- where non-uniform schemes such as min-SNR or P2 often outperform uniform weighting- we observe that uniform weighting performs slightly better across most preference-learning metrics. This indicates that, although these weighting schemes have been widely adopted for diffusion model training, their effectiveness does not directly transfer to the preference optimization setting.
We hypothesize that this discrepancy arises from the differing objectives of diffusion generation and diffusion preference learning. While the former seeks to approximate data distribution, preference learning focuses on maximizing the margin between preferred and non-preferred samples. Consequently, uniform weighting may provide better inductive bias for stable optimization in margin-based objectives.
K
Additional Qualitative Results
present Figure/Bland F present additional image samples generated by GradSPO and various preference alignment methods on SDXL for qualitative comparisons on Parti Prompts and HPDv2, respectively.
7

---

Weighting
HSPv2
Pick Score
Aesthetic Score
Image Reward
MNR-SNRRR 42
26.72
21.26
5.6590
0.5165
P2 43
26.79
21.25
5.6028
0.5318
Uniform
26.86
21.38
5.7651
0.4747
Table Comparison of timestep weighting schemes for GradSPO trained on SD with the Pick-a-Pie v2 dataset. The best result in each column is highlighted in bold.
SDXL
Diff.-DPO
MAPO
SPO
InPO
GradSPO (Ours)
a dolphin jumping over a rowboat
Anubis wearing sunglasses and sitting astride a hog motorayle
frustrated child
child and penguin sitting on the moon
a top down view of horse rumning in field
Figure 6: Side-by-side comparison of images generated by related methods on Parti Prompts using SDXL.

---

SDXL
Diff.-DPO
MAPO
SPO
InPO
GradSPO (Ours)
Popcorn in mouth.
A white polar bear cub wearing sunglasses sits in a meadow with flowers.
B2
A baseball player pitching a baseball on a field.
Spider-Wan MMn holding a ginger cat
MAST
Looking down at a destroyed city from plane
Figure 7: Side-by-side comparison of images generated by related methods HPDv2 using SDXL.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims reflect the paper's contributions and scope as stated in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
10

---

Assumptions are clearly stated in Section 3 and Proofs are provided in the Appendix and
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: To reproduce the experimental result, we provide the corresponding setup in Subsection
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: An open-source implementation is publicly available at htttps://githi com/ Saam//oohhaTTT//rrdSS
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc.
aublic/guudees/Codeetbbiis wtw.eeeeuuuuisssoooooooo.o for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wtaeeee///ooSSemmsssseeeeeee ssi.nnoliccyy for more details. / wttp:/cccpblccgggiiiiieeeec
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included the experimental details Section 4.1 with more details in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are included in figures where applicable to indicate variability or uncertainty, such as in user study results in Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
12

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Compute resource was provided in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:
Answer: [Yes]
Justification:
We confirm that this paper conforms with the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Given that our work primarily focuses on improving the aesthetic quality and faithfulness of generated images, the associated societal impacts, both positive (e.g., enhanced digital art) and negative (e.g., misuse for disinformation,, align with those broadly discussed for generative image models and do not introduce novel ethical considerations requiring dedicated section.
13

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have properly credited these in Subsection 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
14

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssii paperswi thcode. www.coo//aaaaa datasess package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
15

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:e///eerrrrss.cc/////rreeree for what should or should not be described.
16


Reviews:
[{"review_id": "DZVPHz9s7g", "reviewer": "Reviewer_wrXA", "summary": "The paper introduces GradSPO. It provides an analytical study on top of stepwise preference optimization (SPO), which is a promising direction in the field, incorporating step awareness during the process.\n\nAfter presenting the analysis, the paper proposes a simpler loss function compared to SPO and obtains strong results.", "questions": "* Could the authors show results on a recent flow-based model? I believe models like Lumina2 [1], SANA [2], etc., are small enough to conduct an experiment.\n* Even though there is MaPO in the results section, I think it also deserves to be in the other relevant sections of the paper (such as Introduction).\n* I found the clarity of the symbols used throughout the main text of the paper to be somewhat hard to follow. Could the authors try to be more explicit with those?\n* In equation 3 and elsewhere, the authors mention the use of a timestep-dependent weighting factor. Even though it was there in DDPM [3], it's not used much in practice. Could the authors comment on that?\n* The existing reward models (like PickScore [4]) are not typically timestep-aware. This work, however, relies on timestep-aware reward models. Could the authors comment on it a bit more?\n* Equation 13 - can't we scale with the variance of $z$?\n\n## References\n\n[1] Lumina-Image 2.0: A Unified and Efficient Image Generative Framework\n\n[2] SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers\n\n[3] Denoising Diffusion Probabilistic Models\n\n[4] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation", "limitations": "The authors didn't include a separate section on limitations in the main text nor did they make them clear in the main text.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n* Stepwise preference optimization is a promising direction in aligning diffusion models. Therefore, I believe the authors identified a good testbed to start with and trying to further reason about it and repurpose it.\n* Since the method relies on a noisy approximation, the authors also introduce a noisy reduction strategy to mitigate its repercussions.\n* The authors substantiate the simplification with impressive results on widely used metrics in the area.\n\n## Weaknesses\n\nSeveral important ablations and details seem missing from the main text:\n\n  * It uses uniform timestep weighting. How about other weighting schemes?\n  * What is the impact of using components like EMA and noise reduction? Even Figure 5 has win-rate comparisons, I believe a better ablation would have been to show how the loss landscape gets affected by the noise reduction strategy and also some qualitative samples.\n  * For step-aware models, memory and convergence speed can be an issue. Could the authors include a discussion on this?\n  * DPO and MaPO were trained on the Pick-a-Pic v2 dataset while GradSPO uses v1. Could the authors ensure that the methods being compared use the same base dataset?\n  *  The effect of the stop-gradient operator in equation 10 isn't quite clear.", "was_revised": true, "final_justification": "I had asked a bunch of questions in my reviews and also raised some concerns regarding the design of experiments. The authors responded to them firmly and also validated some of the design factors with results. For example, I had asked for the effect of uniform timestep weighting, EMA, etc. They provided enough evidence to support their use. So, I will raise my score.", "rebuttal": "Thank you for your thoughtful review. We address each of your comments below, following their order in the original review.\n\n---\n### **Weaknesses (W)**\n**[W-Q1] Weighting schemes other than the uniform timestep weighting**\n\n**[W-A1]** **Table A** compares uniform timestep weighting with alternative schemes, including the widely used min-SNR weighting [1], and P2 weighting [2] for SD 1.5 on Pick-a-Pic v2. Unlike in standard diffusion training, where schemes like min-SNR or P2 often outperform uniform weighting, we find that uniform weighting performs slightly better in most metrics.\n\nWhile non-uniform weighting schemes [1,2] have seen wide use in diffusion model training, their use in preference learning remains limited; we find that most preference learning for text-to-image diffusion models uses uniform weighting. Although the two objectives are related, they differ in nature: standard diffusion training aims to match a target distribution, whereas diffusion preference learning focuses on maximizing the margin between winning and losing samples. This leads to different optimization dynamics, where uniform weighting may offer a better inductive bias for margin-based learning.\n\n**Table A.** GradSPO with different weighting schemes for SD 1.5 on Pick-a-Pic v2.\n|Weighting|HSPv2|Pick Score|Aesthetic Score|Image Reward|\n|--|--|--|--|--|\n|min-SNR [1]|26.72|21.26|5.6590|0.5165|\n|P2 [2]|26.79|21.25|5.6028|**0.5318**|\n|uniform|**26.86**|**21.38**|**5.7651**|0.4747|\n\n> [1] Hang et al., Efficient Diffusion Training via Min-SNR Weighting Strategy\n\n> [2] Choi et al., Perception Prioritized Training of Diffusion Models\n\n\n**[W-Q2] Impact of using components like EMA and noise reduction**\n\n**[W-A2]** We will include the training curve in the main paper to illustrate the impact of EMA and noise reduction. These components lead to ~2 times faster convergence and a higher plateau in average preference score (GradSPO: ~17.8 vs. SPO: ~17.4) during training on SD 1.5. We attribute the faster convergence and higher training reward to reduced noise, which makes it easier for the diffusion model to optimize toward higher rewards.\n\n\n**[W-Q3] Issue of memory and convergence speed issue for a step-aware model**\n\n**[W-A3]** Stepwise preference methods (SPO and our GradSPO) can introduce two sources of computational overhead compared to standard Diffusion DPO [1].\n\n1) They require a separate step-aware preference model. In our case, we reuse the publicly available pretrained checkpoint from SPO for fairness, rather than training a new model.\n\n2) Both SPO and GradSPO perform online sampling and labeling during training using the step-aware preference model, whereas standard Diffusion DPO relies on offline-labeled data (e.g., from human or AI annotators).\n\nDespite these costs, stepwise methods converges significantly faster. Our full finetuning on SD1.5 completes in 24 hours on 4×A6000 GPUs, compared to 768 hours reported in the original Diffusion DPO paper on the same hardware.\n\n> [1] Wallace et al., Diffusion Model Alignment Using Direct Preference Optimization\n\n\n**[W-Q4] Dataset used for DPO&MaPO vs GradSPO**\n\n**[W-A4]** Pick-a-Pic v2 is an extended version of v1, preserving the same prompt quality while expanding the dataset size. Following SPO, GradSPO uses only the prompt dataset (no image supervision), selecting 4,000 prompts from Pick-a-Pic v1 and generates the images via online sampling. To ensure fairness, we compare against baselines using the same underlying prompt distribution. Although DPO and MaPO are trained on the larger v2 dataset, the prompt quality is equivalent to v1. Since our method, like SPO, depends only on prompt-level supervision, the comparison remains valid despite the size difference.\n\n\nSPO uses online sampling and relies only on the prompt dataset (no image required), selecting a subset of 4,000 prompts from Pick-a-Pic v1.\n\nFor a fair comparison with SPO, we use the same prompt dataset that SPO was built on.\n\nSince our method also depends solely on a small prompt dataset, the extended size of Pick-a-Pic v2 is still a valid comparison as they share the same prompt quality.\n\n\n**[W-Q5] Effect of stop-gradient (Eq. 10) is not clear**\n\n**[W-A5]** In Eq. 10, the training objective updates the current score model $s_\\theta(x_t, c, t)$ by pulling it closer to the winning score $s^w_\\theta(x_t, c, t)$, and pushing it away from the losing score $s^l_\\theta(x_t, c, t)$. To ensure a stable learning signal, the targets are detached using stop-gradient: $sg(s^w_{\\theta_0})$ and $sg(s^l_{\\theta_0})$. \n\nIf the stop-gradient is not applied, the winning score would be updated during training to move toward the current score, collapsing $s^w_\\theta \\rightarrow s_\\theta$, thereby weakening the supervision signal and destabilizing training.\n\n---\n### **Questions (Q)**\n**[Q-Q1] Results on flow-based model**\n\n**[Q-Q1]** Thank you for the suggestion. We agree that evaluating on recent flow-based models such as Lumina2 [1] and SANA [2] would be useful. However, due to the limited time of the rebuttal period, we were unable to conduct these new expriments on additional architectures. We note, however, that our current results, based on widely adopted benchmarks and models in diffusion preference learning literature consistently show performance improvements,  demonstrating the effectiveness of GradSPO within the established preference learning setting.\n\n> [1] Qin et al., Lumina-Image 2.0: A Unified and Efficient Image Generative Framework\n \n> [2] Xie et al., SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer\n\n\n**[Q-Q2] Mention of MaPO in the introduction**\n\n**[Q-Q2]** Thank you for the suggestion. We will include a mention of MaPO in the introduction in the camera-ready version.\n\n**[Q-Q3] Clarity of symbols throughout the text**\n\n**[Q-A3]** We acknowledge that some symbols, such as $x$ (image), $c$ (conditioning input, e.g., text prompt), and $T$ (total number of diffusion timesteps), are used without explicit definitions when first introduced. We agree that providing clear definitions would improve clarity for a broader audience. We will revise the paper to explicitly define all key symbols when first mentioned, especially in the Notation and Preliminaries section.\n\n**[Q-Q4] In equation 3 and elsewhere, the authors mention the use of a timestep-dependent weighting factor. Even though it was there in DDPM, it's not used much in practice. Could the authors comment on that?**\n\n**[Q-A4]** In DDPM, the timestep weighting function corresponds to the ELBO. This timestep weighting is useful when modeling likelihoods with VDM [1] and DDPM [2] achieving better likelihood estimation with an ELBO timestep weighting scheme in contrast to the uniform weighting scheme, however this weighting scheme has been shown to hurt image quality achieving lower FID on standard image benchmarks due to a timestep bias issue during training. As most image generation as well as our research is more concerned with image perceptual quality than accurate likelihood estimation, we opt for using uniform weighting instead of a timestep dependent weighting factor.\n\n> [1] Kingma et al., Variational Diffusion Models\n\n> [2] Ho et al., Denoising Diffusion Probabilistic Models\n\n\n\n**[Q-Q5] The existing reward models (like PickScore) are not typically timestep-aware. This work, however, relies on timestep-aware reward models. Could the authors comment on it a bit more?**\n\n**[Q-A5]** Our approach requires a timestep-aware reward model primarily because both SPO and GradSPO rely on *dense* supervision across intermediate denoising steps, rather than *sparse* supervision only at the final output. This is analogous to findings in LLM RL training, where dense token-level rewards have been shown to lead to better alignment than sparse response-level rewards [1,2,3].\n\nIn the diffusion setting, this means scoring not just the final (clean) image but also the intermediate, noisy ones during denoising. Existing reward models like PickScore are trained only on fully denoised images and therefore produce unreliable or uninformative outputs when applied to intermediate blurry samples. To enable effective preference learning across the full denoising trajectory, we use timestep-aware reward models specifically trained on such intermediate images.\n\n> [1] Yoon et al., TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback\n\n> [2] Zeng et al., Token-level Direct Preference Optimization\n\n> [3] Chan et al., Dense Reward for Free in Reinforcement Learning from Human Feedback\n\n\n**[Q-Q6] Equation 13 - can't we scale with the variance of z?**\n\n**[Q-A6]** While scaling with the variance of $z$ can reduce the variance of the estimator, it also unintentionally scales down its mean, $\\sqrt{T} \\, \\beta_t \\, \\nabla_{x_t} r(x_t, t)$. This weakens the strength of the guidance signal. Our goal with the variance-reduced estimator in Eq. 13 is to reduce variance *without* altering the mean, so that the average reward signal strength remains consistent while the noise is suppressed. This ensures that the model receives a stable yet strong training signal. We will include this rationale in the main paper.\n\n\n**[Limitations Section Presentation]**  \nWe acknowledge the concern and agree that limitations should be clearly presented in the main paper. We will move the limitations section to the main paper to ensure better visibility."}, {"review_id": "XLOoOyK1pr", "reviewer": "Reviewer_XP6v", "summary": "This paper proposes GradSPO, a method designed to alleviate the time-step weighting bias inherent in SPO. The authors support their approach with theoretical analysis and further validate it through experiments on both SD1.5 and SDXL backbones.", "questions": "As highlighted in the weaknesses section, the primary concern centers around the simplicity of the proposed function a(t). While the idea is intuitive, the paper does not provide any ablation studies or empirical analysis to justify this specific design choice. Without such evidence, it remains unclear whether the performance gains are inherently tied to this formulation or if alternative strategies—such as more complex or learnable functions—could yield better results.\nFurthermore, the absence of robustness checks raises questions about the general applicability and stability of the method under different settings or perturbations.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is clearly written and logically organized, making it easy to follow the proposed methodology and its motivations.\n2. The GradSPO framework is theoretically well-founded, with analytical derivations that support its design and justify its components.\n3. The method’s effectiveness is demonstrated through experiments on two different backbone architectures, yielding compelling qualitative results that underscore its generalizability.\n\nWeaknesses\n1. The proposed function a(t) in GradSPO appears to be overly simplistic. It would be beneficial to explore alternative formulations—such as non-linear functions—or investigate whether a learnable parameterization could better capture the desired temporal behavior.\n2. While Equations (10)–(12) offer a solid theoretical argument, the paper would benefit from additional empirical evidence or illustrative visualizations to more concretely demonstrate the time-step weighting bias issue inherent in SPO.\n3. The experimental validation is limited in scope. It lacks evaluations on more recent and competitive backbone models such as DiT (e.g., SD3 or Flux). Moreover, it remains unclear whether the proposed method can be extended or adapted to the video generation domain, which would further demonstrate its versatility.", "was_revised": true, "final_justification": "I appreciate the authors for the helpful feedback. My main concerns are addressed and I would raise my final rating to 4. borderline accept.", "rebuttal": "Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.\n\n---\n### **Weaknesses (W)**\n**[W-Q1] The proposed function a(t) in GradSPO appears to be overly simplistic. It would be beneficial to explore alternative formulations, such as non-linear functions.**\n\n**[W-A1]** **Table A** compares uniform timestep weighting with alternative schemes, including the widely used min-SNR weighting [1], and P2 weighting [2] for SD 1.5 on Pick-a-Pic v2. Unlike in standard diffusion training, where schemes like min-SNR or P2 often outperform uniform weighting, we find that uniform weighting performs slightly better in most metrics.\n\nWhile non-uniform weighting schemes [1,2] have seen wide use in diffusion model training, their use in preference learning remains limited; we find that most preference learning for text-to-image diffusion models uses uniform weighting. Although the two objectives are related, they differ in nature: standard diffusion training aims to match a target distribution, whereas diffusion preference learning focuses on maximizing the margin between winning and losing samples. This leads to different optimization dynamics, where uniform weighting may offer a better inductive bias for margin-based learning.\n\n**Table A.** GradSPO with different weighting schemes for SD 1.5 on Pick-a-Pic v2.\n|Weighting|HSPv2|Pick Score|Aesthetic Score|Image Reward|\n|--|--|--|--|--|\n|min-SNR [1]|26.72|21.26|5.6590|0.5165|\n|P2 [2]|26.79|21.25|5.6028|**0.5318**|\n|uniform|**26.86**|**21.38**|**5.7651**|0.4747|\n\n> [1] Hang et al., Efficient Diffusion Training via Min-SNR Weighting Strategy\n\n> [2] Choi et al., Perception Prioritized Training of Diffusion Models\n\n\n**[W-Q2] While Equations (10)–(12) offer a solid theoretical argument, the paper would benefit from additional empirical evidence or illustrative visualizations to more concretely demonstrate the time-step weighting bias issue inherent in SPO.**\n\n**[W-A2]** We will add the plot in the main text to provide clear empirical evidence for the timestep bias issue. To visualize its impact, we measure the difference in preference scores of $\\mathbf{x}_0$ predictions at each timestep before and after preference finetuning, using noisy images from the Pick-a-Pic v2 win set (SD 1.5).\n\nWe observe that in SPO, the difference in preference scores is high at early timesteps but drops off sharply in later timesteps. This exponential decrease reflects a timestep bias, where the model allocates more capacity to optimizing preference scores at earlier denoising stages, while neglecting later ones. In contrast, GradSPO exhibits a more linear decrease, indicating more balanced improvements across all timesteps, including the later ones.\n\n\n\n\n**[W-Q3] It lacks evaluations on recent backbone models.**\n\n**[W-A3]** Thank you for the suggestion. We agree that evaluating on more recent architectures would provide a stronger justification on the generality of our method. However, due to the limited time and computational constraints during the rebuttal period, we were unable to conduct additional experiments on these more resource-intensive models.\n\nThat said, our experiments on SD 1.5 and SDXL, two widely adopted diffusion backbones, demonstrate consistent performance gains across multiple datasets and metrics, providing strong evidence of the effectiveness of GradSPO over prior preference learning methods.\n\n---\n### **Questions (Q)**\n**[Q-Q1] addressed in [W-Q1]**\n\n**[Q-Q2] Robustness checks under different settings.**\n\n**[Q-A2]** To properly analyze the performance and stability of GradSPO, we conduct ablations on key hyperparameters with all other settings fixed for SD 1.5 with results reported on Pick-a-Pic v2 (we set $\\gamma=0.5$ when varying $\\mu$ and $\\mu=0.9$ when varying $\\gamma$). Results are shown in **Table B**.\n\nWe find that the method is relatively robust to variations in the EMA momentum parameter $\\mu$, with only a slight drop in performance at higher values. We attribute this to an oversmoothing effect, where the EMA model lags behind the current model and adapts too slowly.\n\nIn contrast, the reward guidance scale $\\gamma$ has a stronger impact. Setting $\\gamma$ too high increases the variance of the reward signal, especially in noisy regions, which makes convergence less stable. On the other hand, a very low $\\gamma$ weakens the signal and prevents the model from learning meaningful preferences. We find $\\gamma = 0.5$ to be a good balance between signal strength and noise, and use this value across both SDXL and SD 1.5 experiments.\n\n**Table B.** Ablation study on EMA momentum $\\mu$ and reward guidance scale $\\gamma$ for GradSPO on SD 1.5 for Pick-a-Pic v2.\n| $\\gamma$ |HSPv2|Pick Score|Aesthetic Score|Image Reward|\n|-------|--------|-------------|------------------|----------------|\n| 0.25  | 26.67 | 20.71      | 5.5278           | 0.452          |\n| 0.5   | 26.86 | 21.38      | 5.7651           | 0.4747         |\n| 0.75  | 26.70  | 20.99      | 5.463            | 0.3878         |\n\n| $\\mu$  | HSPv2 | Pick Score | Aesthetic Score | Image Reward |\n|--------|--------|-------------|------------------|----------------|\n| 0.9    | 26.86 | 21.38      | 5.7651           | 0.4747         |\n| 0.95   | 26.74 | 21.12      | 5.6352           | 0.4153         |\n| 0.99   | 26.68 | 21.22      | 5.7591           | 0.3871         |"}, {"review_id": "917bUqtnsf", "reviewer": "Reviewer_ccWi", "summary": "The paper proposes GradSPO, a follow-up to the SPO approach for preference optimization and text-to-image alignment. Specifically, instead of optimizing likelihood during denoising steps it directly approximates reward gradients using noise vectors that produce the best and worst samples and constructs guided targets by combining these gradient directions.\nThe key contributions of the paper are:\n- Theoretical contribution showing SPO can be interpreted as gradient based score matching.\n- GradSPO adds a timestep-uniform objective that removes the need for heuristic weighting.\n- Adds a max-min noise estimator to reduce variance in the gradient leading to more stable optimization.\n- EMA-based target smoothing which stabilizes training.\n- SOTA results across multiple aesthetic benchmarks, datasets and models.", "questions": "- The fact that the author did not have to implement \"multi-step\" SPO to make SDXL work is very interesting but under-explored in the paper.\n\n- For the method to stand up on its own it is important to consider redoing key ablation studies that could have changed from SPO: such as number of timesteps, guidance scale, and some new parameters (like /mu).\n\n- Adding specific failure cases would be very interesting to examine, especially if they are not shared with SPO, which can inspire future research into the approach.\n\n - The authors should also consider evaluating on GenEval results which are missing here but are in the SPO paper\n\n- A figure would be very useful to show a comparison of the GradSPO and SPO methods as a quick reference and introduction to the paper.\n\n- I found Figure 4 confusing. The side-by-side comparisons expected typically for the preference models is similar to Figures 6 - 7 in the Appendix, meaning the difference is clearly visible, but the images are relatively the same. Doing some testing on SDXL and DPO demos for the prompts specified: a) \"Buff Harry Potter\" and b) \"a gorgeous queen with cat like eyes\", I was able to get a correct figure for a) SDXL (6/10), DPO (10/10) b) SDXL (9/10), DPO (8/10). These are not rigorous tests,but they suggest handpicked results or preferential selection for GradSPO. I do not have access to GradSPO directly so I cannot perform a fair evaluation. However, the demos demonstrated for the GradSPO in Figure 4 are better even among the ones I sampled so I do not see a reason why the authors would pick poor images for the other methods, which damages the credibility of the comparison. I suggest revising the qualitative demos in Figure 4 to match the extended ones in the Appendix. For a fairer comparison multiple demos can be sampled for each model, scored and the best one for each method selected.\n\n- The user evaluation also raises concerns. Human preference studies are easy to bias via position effects or presentation order. The performance of GradSPO vs SDXL (~90% win rate) is a unusually high for aesthetic alignment so I urge the authors to describe their process and guarantee all the necessary steps have been taken for a fair comparison. It would also be better to follow SPO’s breakdown and perform a user study based on General Preference, Visual Appeal and Prompt Alignment categories.\n\n- Minor point: the ablation studies in Section 4.4 make a strong argument in favor of each component in isolation. However, winrate is a non transitive metric so SPO>SDXL and GradSPO>>SDXL does not mean GradSPO>>SPO, so a winrate vs SPO or a table with scores is more informative.", "limitations": "Addressed", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Clear theoretical motivation for interpreting SPO as score matching.\n- Simple objective that avoids timestep-specific weighting, noise variance reduction and EMA smoothing.\n- GradSPO has good results results across multiple aesthetic benchmarks and ablation studies show each component is well motivated.\n- GradSPO is robust and works without requiring multi-step implementation which is central to SPO.\n\nWeaknesses\n- The method relies heavily on SPO but does not re-examine key hyperparameters that were ablated in SPO (see Questions).\n- No visual side-by-side comparison with SPO which could have been useful to demonstrate the differences.\n- SPO evaluations are missing and key issues with qualitative results and user studies (see Questions)\n- Lack of specific qualitative failure cases to examine the method in detail and guide future improvements.", "was_revised": true, "final_justification": "As mentioned in the discussion my main concerns revolved around the comparison with other models which have been addressed and I am thus increasing my score.", "rebuttal": "We thank the reviewer for the insightful comments and careful evaluation. Our responses are organized according to the sequence of points raised in the review.\n\n### **Weaknesses (W)**\n**[W-Q1] addressed in [Q-Q2] below**\n\n**[W-Q2] No visual side-by-side comparison with SPO.**\n\n**[W-A2]** Side-by-side visual comparisons with SPO are already included in Figure 4 of the main paper, as well as Figures 6–7 in the Appendix. These figures present qualitative comparisons between GradSPO, SPO, and other widely used preference learning methods to highlight their differences more clearly.\n\n**[W-Q3] addressed in [Q-Q6] and [Q-Q7] below**\n\n**[W-Q4] addressed in [Q-Q3] below**\n\n\n---\n### **Questions (Q)**\n**[Q-Q1] The fact that the author did not have to implement \"multi-step\" SPO to make SDXL work is very interesting but under-explored in the paper.**\n\n**[Q-A1]** 'Multi-step' SPO was originally introduced to strengthen reward signals by simulating longer trajectories. However, in GradSPO, the use of noise reduction stabilizes gradient estimates, effectively reducing the need for such rollouts. This allows SDXL to achieve strong performance without the added complexity of multi-step simulation, highlighting noise reduction as a more effective alternative in this setting.\n\n**[Q-Q2] Lack of ablation on key hyperparameters.**\n\n**[Q-A2]** In **Table A**, we present ablation results of our GradSPO on SPO-specific hyperparameters, including guidance scale and timestep count for SD 1.5 on Pick-a-Pic v2 with all other hyperparameters fixed to the ones used in the main results.\n\nWe observe that increasing the guidance scale leads to a consistent drop in performance, while reducing it improves results. We hypothesize this is due to high CFG scales introducing oversaturation artifacts during sampling [1]. While high CFG can yield visually appealing images during inference, training with these settings amplifies artifacts, ultimately degrading generation quality.\n\nWe also examine the impact of increasing the number of sampling steps from 20 to 50. As shown in the table, this change yields no meaningful performance improvement, with results remaining comparable to those obtained at 20 steps. Additionally, it nearly doubles the training time due to the increased sampling overhead.\n\n**Table A.** Ablation study on classifier-free guidance scale and number of timesteps in GradSPO on SD 1.5.\n|Guidance|HSPv2|Pick Score|Aesthetic Score|Image Reward|\n|-|-|-|-|-|\n|7.5|26.60|21.19|5.5051|0.3066|\n|5.0|26.86|21.38|5.7651|0.4747|\n|4.0|27.19|21.44|5.7669|0.5852|\n\n|Steps|HSPv2|Pick Score|Aesthetic Score|Image Reward|\n|-|-|-|-|-|\n|20|26.86|21.38|5.7651|0.4747|\n|50|26.87|21.21|5.7939|0.4563|\n\nAdditionally, in **Table B**, we conduct ablations on GradSPO specific hyperparameters, $\\gamma$ and $\\mu$, with all other settings fixed for SD 1.5 with results reported on Pick-a-Pic v2 (we set $\\gamma=0.5$ when varying $\\mu$ and $\\mu=0.9$ when varying $\\gamma$).\n\n**Table B.** Ablation results on reward guidance scale $\\gamma$ and EMA momentum $\\mu$ for GradSPO on SD 1.5.\n| $\\gamma$ |HSPv2|Pick Score|Aesthetic Score|Image Reward|\n|-|-|-|-|-|\n|0.25|26.67|20.71|5.5278|0.452|\n|0.5|26.86|21.38|5.7651|0.4747|\n|0.75|26.70|20.99|5.463|0.3878|\n\n|$\\mu$|HSPv2|Pick Score|Aesthetic Score|Image Reward|\n|-|-|-|-|-|\n|0.9|26.86|21.38|5.7651|0.4747|\n|0.95|26.74|21.12|5.6352|0.4153|\n|0.99|26.68|21.22|5.7591|0.3871|\n\nWe find that the $\\mu$ parameter is relatively robust, with only a slight drop in performance at lower values. This may be attributed to oversmoothing, where the EMA model updates too slowly. In contrast, modifying the $\\gamma$ parameter in either direction results in noticeable performance degradation.\n\nAt higher guidance scales, increasing $\\gamma$ amplifies noise, leading to higher gradient variance and poorer convergence. Conversely, lowering $\\gamma$ weakens the reward signal, preventing the model from optimizing effectively. Overall, we find that $\\gamma = 0.5$ provides a strong balance between reward strength and signal stability, and we adopt this value for both SDXL and SD 1.5 experiments.\n\n> [1] Sadat et al., Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models\n\n**[Q-Q3] Adding specific failure cases would be very interesting.**\n\n**[W-A3]** One potential failure area worth exploring in future work is the reduction in diversity after alignment. Specifically, we observe that stronger alignment to the preference signal tends to reduce output diversity. To investigate this, we measured diversity scores across 10 randomly sampled prompts with 10 different seeds each, following the setup in [1].\n\nThe results show a consistent drop in diversity: SDXL scores *12.54*, SPO drops to *10.32*, and GradSPO further to *9.40*. This mirrors patterns seen in preference learning for large language models [2], where optimization toward high-reward outputs often comes at the cost of diversity. Understanding and mitigating this trade-off remains an open research challenge.\n\n> [1] Ibarrola et al., Measuring Diversity in Co-creative Image Generation\n\n> [2] Cui et al., The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models\n\n\n**[Q-Q4] The authors should also consider evaluating on GenEval results.**\n\n\n**[Q-A4]** **Table C.** below presents the GenEval results for SDXL. From these results, we observe that GradSPO achieves notable improvements over SPO, not only in aesthetic quality but also in compositional generation, resulting in a higher overall score. However, similar to what was reported in the SPO paper, GradSPO still falls short of Diff-DPO in overall GenEval performance. This suggests that, similar to SPO, GradSPO tends to prioritize optimizing for aesthetic reward signals over learning fine-grained compositional alignment, despite outperforming Diff-DPO on all individual image reward metrics.\n\n**Table C.** GenEval results for SDXL models.\n|Method|Single|Two|Count|Color|Pos|AttrBind|Overall|\n|-|-|-|-|-|-|-|-|\n|SDXL|97.81|68.43|40.62|86.70|12.00|23.00|54.76|\n|Diff.-DPO|**99.69**|**81.06**|**48.44**|**89.63**|**13.25**|**27.75**|**59.84**|\n|SPO|97.81|73.74|41.25|86.44|13.00|20.25|55.41|\n|GradSPO|99.06|77.78|47.50|88.03|13.00|22.25|57.07|\n\n\n**[Q-Q5] A figure comparing GradSPO and SPO would be helpful as a quick reference in the introduction.**\n\n**[Q-A5]** We recognize that including such a comparison earlier in the paper could improve clarity and accessibility. For the camera-ready version, we will revise the introduction to feature a clearer visual reference as a quick overview of the improvements introduced by GradSPO.\n\n\n**[Q-Q6] Curation details for Figure 4**\n\n**[Q-A6]** We appreciate the reviewer’s careful analysis and thoughtful suggestion. We acknowledge that without a clearly defined sampling or scoring procedure, the presentation may give the impression of preferential selection.\n\nTo address this, we will revise Figure 4 in the camera-ready version to follow a more rigorous and transparent protocol. Specifically, we will sample multiple generations per model for each prompt, score them using consistent preference criteria (e.g., reward model scores), and display the top result per method. This should provide a more credible and balanced qualitative comparison. We will also clarify this selection process in the caption and main text.\n\n**[Q-Q7] Concerns regarding user study**\n\n**[Q-A7]** In our original user study, participants were shown image pairs (Image A and Image B) generated using the same prompt and initial noise, each corresponding to a different method. Participants were asked to choose the image they preferred. To mitigate presentation order bias, the assignment of methods to Image A/B was randomized for each comparison. The high win rate in the user study is due to the lack of a \"tie\" option, which may have inflated the win rate of GradSPO over SDXL by forcing a decision even when the quality was comparable. \n\nTo address this, we conducted an additional fine-grained user study with 100 prompt samples per method and 5 independent raters. In this follow-up, participants were allowed to select one of three options: *Image A better*, *Image B better*, or *Tie*. Additionally, similar to SPO, the preference was done over General Preference, Visual Appeal and Prompt Alignment (**Table D** and **Table E**). \n\n**Table D.** User preference results: **GradSPO vs SDXL**.\n|Category|Win|Tie|Lose|\n|-|-|-|-|\n|Overall|75.62%|5.72%|18.66%|\n|Instruction|53.23%|30.85%|15.92%|\n|Aesthetic|81.09%|3.98%|14.93%|\n\n**Table E.** User preference results: **GradSPO vs SPO**.\n|Category|Win|Tie|Lose|\n|-|-|-|-|\n|Overall|52.99%|21.89%|25.12%|\n|Instruction|26.37%|54.98%|18.66%|\n|Aesthetic|56.22%|21.14%|22.64%|\n\nThese results confirm that GradSPO maintains a strong preference margin over SDXL and SPO, even with the tie option included, further validating the perceptual improvements introduced by our method.\n\n**[Q-Q8] Non-transitivity of win rates and need for direct comparison with SPO**\n\n**[Q-A8]** To address the concern regarding the non-transitivity of win rates, we provide a direct comparison of each ablation variant against SPO using standard preference metrics in **Table F**. The results show a consistent upward trend as each component is added, reinforcing the value of the proposed design choices in GradSPO.\n\n**Table F.** Preference metrics comparing ablation variants of GradSPO against SPO on SDXL. Gradual inclusion of key components (e.g., max-min guidance) leads to consistent improvements across all metrics.\n|Method|HSPv2|Pick Score|Aesthetic|Image Reward|Avg|\n|-|-|-|-|-|-|\n|SPO + Simple|0.518|0.570|0.646|0.624|0.5895|\n|SPO + Simple + MaxMin| 0.618 | 0.566 |0.718|0.650|0.6380|\n|GradSPO|0.604|0.648|0.800|0.684|0.6840|\n\n---"}, {"review_id": "ekkjGMczLM", "reviewer": "Reviewer_VBZ5", "summary": "This paper introduces GradSPO, a novel framework that reinterprets Stepwise Preference Optimization (SPO) for aligning text-to-image diffusion models with human preferences through a gradient guidance perspective. While SPO improves upon Direct Preference Optimization (DPO) by leveraging intermediate diffusion steps for preference learning, the authors identify two key limitations: (1) an imbalanced temporal weighting that under-prioritizes later denoising steps critical for fine details, and (2) high noise in gradient estimates from reward models. GradSPO addresses these issues by reformulating SPO’s mechanism as gradient-based score matching, leading to a simplified loss with uniform timestep weighting and a variance-aware noise reduction strategy to stabilize training. Experiments on Stable Diffusion 1.5 and SDXL demonstrate that GradSPO outperforms existing methods (e.g., DPO, SPO) in human preference alignment, generating images with enhanced aesthetic quality and prompt faithfulness.", "questions": "No", "limitations": "See Weaknesses", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Theoretical Rigor: The paper provides a solid theoretical foundation for GradSPO, rigorously addressing the underweighting issue in SPO. The derivations are clear and well-justified, making a convincing case for the proposed method from a theoretical perspective.\n2. Problem Identification: The authors correctly identify a potential limitation in SPO—underweighting later generative steps—and propose a principled solution. This is a meaningful contribution, as later diffusion steps are crucial for high-frequency detail refinement.\n3. Novelty in Formulation: GradSPO introduces a new gradient-based weighting scheme, which is a non-trivial extension of SPO. The idea of dynamically adjusting step weights to stabilize training is theoretically appealing.\n\nWeaknesses:\n\n1. Lack of Empirical Validation:\n   1. The paper claims that underweighting later steps harms output quality (e.g., blurriness, incoherence), but no fine-grained analysis (e.g., per-step contribution to image sharpness, edge quality) is provided.\n   2. The claim that GradSPO improves training stability is unsupported by experiments. There are no training curves, loss variance comparisons, or reward convergence plots between SPO and GradSPO.\n2. Weak Experimental Section:\n   1. The experiments focus primarily on downstream metrics (e.g., FID, IS) without isolating the impact of GradSPO’s weighting mechanism. It’s unclear whether improvements stem from better weighting or other factors.\n   2. No ablation studies (e.g., removing GradSPO’s gradient correction) to validate its necessity.\n3. Overstated Practical Impact:\n   1. The theoretical benefits of GradSPO are not matched by empirical gains. The performance improvements over SPO (if any) are marginal in the presented results.\n   2. The paper does not explore failure cases or scenarios where GradSPO might degrade performance (e.g., noisy gradients exacerbating instability).", "was_revised": true, "final_justification": "Thank you for your detailed response. The clarification is neat and clear, but after considering the many missing experiments and ablations in the paper, I would recommend authors to thoroughly revise thier paper for a next submission. I would maintain my score.", "rebuttal": "Thank you for taking the time to review our submission and provide detailed feedback. We’ve addressed your comments below in the order they were raised.\n\n---\n### **Weaknesses (W)**\n\n**[W-Q1.1] Lack of empirical justification for timestep bias problem raised**\n\n**[W-A1.1]** To clarify the effect of timestep bias, we conducted an additional empirical study. Specifically, we plotted the change in preference scores of the $𝑥_0$ predictions at each timestep before and after preference fine-tuning, for both SPO and GradSPO, using noisy images from the Win subset of the Pick-a-Pic v2 dataset on Stable Diffusion 1.5.\n\nThe results show that SPO significantly improves preference scores in early timesteps, but the improvements sharply decline in later steps, indicating an exponential bias toward early timesteps. In contrast, GradSPO exhibits a more gradual, linear decline, suggesting it distributes learning improvements more evenly across timesteps.\n\nThis supports the presence of timestep bias in SPO and highlights GradSPO's advantage. ***The corresponding plot and discussions will be included in the main paper.***\n\n**[W-Q1.2] Lack of empirical results supporting improved training stability claims of GradSPO**\n\n**[W-A1.2]** We thank the reviewer for highlighting this point. To support our claim regarding GradSPO’s improved training stability, we will include a training reward convergence curve in the camera-ready version. This curve demonstrates that GradSPO consistently achieves higher training rewards compared to SPO, which we attribute to reduced gradient noise and more stable optimization. We will also revise the introduction and abstract to clarify our definition of '*training stability*' and better align the narrative with the empirical evidence presented.\n\n**[W-Q2.1] Lack of ablation experiments isolating the weighting mechanism**\n\n**[W-A2.1]** In Section 4.4 (Figure 5) for our main paper, we provide an ablation study to isolate the weighting mechanism of GradSPO. Specifically, we introduce a simplified objective that incorporates only the new weighting mechanism to standard SPO (*denoted as SPO+Simple*), to better assess the individual contribution of the weighting mechanism. Our results show that the weighting mechanism contributes positively, leading to improved win rates over the baseline SPO on the SD 1.5 model.\n\n**[W-Q2.2] Lack of ablation experiments for gradient correction**\n\n**[W-A2.2]** An ablation study analyzing the effect of the gradient correction term is also provided in Section 4.4 (Figure 5) (*denoted as SPO+Simple+Maxmin*). It shows a clear performance gain over the simplified objective alone (SPO+Simple), demonstrating the contribution of each component to the overall effectiveness of GradSPO*.\n\n*(note that GradSPO would be equivalent to *SPO+Simple+Maxmin+EMA*).\n\n**[W-Q3.1] The theoretical benefits of GradSPO are not matched by empirical gains.**\n\n**[W-A3.1]** We respectfully disagree with this assessment. GradSPO shows consistent improvements over SPO across all benchmarks. While the absolute gains (~0.2 across preference metrics) may seem modest, they are in fact meaningful, especially when compared to absolute gains of DPO to SPO on both SD 1.5 and SDXL.\n\nThese results indicate that GradSPO not only offers theoretical advantages but also translates them into empirical gains, reinforcing its effectiveness as a practical preference learning approach.\n\n**[W-Q3.2] The paper does not explore failure cases**\n\n**[W-A3.2]** To better understand the failure cases in terms of training of GradSPO, we vary the $\\gamma$ parameter on SD1.5 on Pick-a-Pic v2 while keeping the rest fixed to the setting used in the main results.\n\n**Table A** summarizes the effect of varying $\\gamma$ on key metrics such as HSPv2, Pick Score, Aesthetic Score, and Image Reward. The results suggest that at higher guidance scales, increasing $\\gamma$ introduces more variance due to noise, leading to worse convergence. Conversely, lower $\\gamma$ values result in a weaker reward signal, preventing the model from optimizing effectively. We find that $\\gamma = 0.5$ offers a good trade-off between stability and reward strength, and we adopt this setting in both SDXL and SD1.5 experiments.\n\n**Table A.** Ablation study on reward guidance scale $\\gamma$ for GradSPO on SD 1.5 for Pick-a-Pic v2.\n| Gamma | HSPv2 | Pick Score | Aesthetic Score | Image Reward |\n|-------|--------|-------------|------------------|----------------|\n| 0.25  | 26.67 | 20.71      | 5.5278           | 0.452          |\n| 0.5   | 26.86 | 21.38      | 5.7651           | 0.4747         |\n| 0.75  | 26.70  | 20.99      | 5.463            | 0.3878         |\n\n\nMoreoever, one potential failure area worth exploring in future work is the reduction in diversity after alignment. Specifically, we observe that stronger alignment to the preference signal tends to reduce output diversity. To investigate this, we measured diversity scores across 10 randomly sampled prompts with 10 different seeds each, following the setup in [1].\n\nThe results show a consistent drop in diversity: SDXL scores *12.54*, SPO drops to *10.32*, and GradSPO further to *9.40*. This mirrors patterns seen in preference learning for large language models [2], where optimization toward high-reward outputs often comes at the cost of diversity. Understanding and mitigating this trade-off remains an open research challenge.\n\n> [1] Ibarrola et al., Measuring Diversity in Co-creative Image Generation\n\n> [2] Cui et al., The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models"}]

Author rebuttal:

Meta-review: The paper takes the recent SPO algorithm for Diffusion model preference fine-tuning and does a theoretical analysis in terms of a gradient guidance perspective. It then uses that framework to improve the design of the algorithm with noise reduction to prevent a temporal weighting bias. This algorithm leads to improved aesthetics and human preference alignment for image generation tasks.

Strengths: All reviewers agreed that the development of the algorithm was interesting, novel and impactful. 

The bulk of complaints centered on some minor weaknesses in experiments. some ablations were suggested by reviewers and in most cases done by the authors and results shared in rebuttal. There was a concern that the gains are marginal, but I think this arises from a misunderstanding of how to interpret HPS score and other metrics. There is a concern that results are not done on the latest T2i models (like flux).  One additional concern i have is that it is unclear how important the original SPO algorithm is, since it only has 5 citations so far...so how impactful is a paper that adds analysis and improvements?

Overall however, it seems that the results justify acceptance, since it is an important problem, the derivation is non-trivial and the results are good.
Decision: Accept (poster)