Title: A Gradient Guidance Perspective on Stepwise Preference Optimization for Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

Human Preference Alignment
Abstract: Direct Preference Optimization (DPO) is a key framework for aligning text-to-image models with human preferences, extended by Stepwise Preference Optimization (SPO) to leverage intermediate steps for preference learning, generating more aesthetically pleasing images with significantly less computational cost. While effective, SPO's underlying mechanisms remain underexplored. In light of this, we critically re-examine SPO by formalizing its mechanism as gradient guidance. This new lens shows that SPO uses biased temporal weighting, giving too little weight to later generative steps, and unlike likelihood centric views it reveals substantial noise in the gradient estimates. Leveraging these insights, our GradSPO algorithm introduces a simplified loss and a targeted, variance-informed noise reduction strategy, enhancing training stability. Evaluations on SD 1.5 and SDXL show GradSPO substantially outperforms leading baselines in human preference, yielding images with markedly improved aesthetics and semantic faithfulness, leading to more robust alignment. Code and models are available at https://github.com/JoshuaTTJ/GradSPO.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GradSPO, a novel framework that reinterprets Stepwise Preference Optimization (SPO) for aligning text-to-image diffusion models with human preferences through a gradient guidance perspective. While SPO improves upon Direct Preference Optimization (DPO) by leveraging intermediate diffusion steps for preference learning, the authors identify two key limitations: (1) an imbalanced temporal weighting that under-prioritizes later denoising steps critical for fine details, and (2) high noise in gradient estimates from reward models. GradSPO addresses these issues by reformulating SPO’s mechanism as gradient-based score matching, leading to a simplified loss with uniform timestep weighting and a variance-aware noise reduction strategy to stabilize training. Experiments on Stable Diffusion 1.5 and SDXL demonstrate that GradSPO outperforms existing methods (e.g., DPO, SPO) in human preference alignment, generating images with enhanced aesthetic quality and prompt faithfulness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Theoretical Rigor: The paper provides a solid theoretical foundation for GradSPO, rigorously addressing the underweighting issue in SPO. The derivations are clear and well-justified, making a convincing case for the proposed method from a theoretical perspective.\n2. Problem Identification: The authors correctly identify a potential limitation in SPO—underweighting later generative steps—and propose a principled solution. This is a meaningful contribution, as later diffusion steps are crucial for high-frequency detail refinement.\n3. Novelty in Formulation: GradSPO introduces a new gradient-based weighting scheme, which is a non-trivial extension of SPO. The idea of dynamically adjusting step weights to stabilize training is theoretically appealing.\n\nWeaknesses:\n\n1. Lack of Empirical Validation:\n   1. The paper claims that underweighting later steps harms output quality (e.g., blurriness, incoherence), but no fine-grained analysis (e.g., per-step contribution to image sharpness, edge quality) is provided.\n   2. The claim that GradSPO improves training stability is unsupported by experiments. There are no training curves, loss variance comparisons, or reward convergence plots between SPO and GradSPO.\n2. Weak Experimental Section:\n   1. The experiments focus primarily on downstream metrics (e.g., FID, IS) without isolating the impact of GradSPO’s weighting mechanism. It’s unclear whether improvements stem from better weighting or other factors.\n   2. No ablation studies (e.g., removing GradSPO’s gradient correction) to validate its necessity.\n3. Overstated Practical Impact:\n   1. The theoretical benefits of GradSPO are not matched by empirical gains. The performance improvements over SPO (if any) are marginal in the presented results.\n   2. The paper does not explore failure cases or scenarios where GradSPO might degrade performance (e.g., noisy gradients exacerbating instability).\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes GradSPO, a follow-up to the SPO approach for preference optimization and text-to-image alignment. Specifically, instead of optimizing likelihood during denoising steps it directly approximates reward gradients using noise vectors that produce the best and worst samples and constructs guided targets by combining these gradient directions.\nThe key contributions of the paper are:\n- Theoretical contribution showing SPO can be interpreted as gradient based score matching.\n- GradSPO adds a timestep-uniform objective that removes the need for heuristic weighting.\n- Adds a max-min noise estimator to reduce variance in the gradient leading to more stable optimization.\n- EMA-based target smoothing which stabilizes training.\n- SOTA results across multiple aesthetic benchmarks, datasets and models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Clear theoretical motivation for interpreting SPO as score matching.\n- Simple objective that avoids timestep-specific weighting, noise variance reduction and EMA smoothing.\n- GradSPO has good results results across multiple aesthetic benchmarks and ablation studies show each component is well motivated.\n- GradSPO is robust and works without requiring multi-step implementation which is central to SPO.\n\nWeaknesses\n- The method relies heavily on SPO but does not re-examine key hyperparameters that were ablated in SPO (see Questions).\n- No visual side-by-side comparison with SPO which could have been useful to demonstrate the differences.\n- SPO evaluations are missing and key issues with qualitative results and user studies (see Questions)\n- Lack of specific qualitative failure cases to examine the method in detail and guide future improvements.\n\nQUESTIONS:\n- The fact that the author did not have to implement \"multi-step\" SPO to make SDXL work is very interesting but under-explored in the paper.\n\n- For the method to stand up on its own it is important to consider redoing key ablation studies that could have changed from SPO: such as number of timesteps, guidance scale, and some new parameters (like /mu).\n\n- Adding specific failure cases would be very interesting to examine, especially if they are not shared with SPO, which can inspire future research into the approach.\n\n - The authors should also consider evaluating on GenEval results which are missing here but are in the SPO paper\n\n- A figure would be very useful to show a comparison of the GradSPO and SPO methods as a quick reference and introduction to the paper.\n\n- I found Figure 4 confusing. The side-by-side comparisons expected typically for the preference models is similar to Figures 6 - 7 in the Appendix, meaning the difference is clearly visible, but the images are relatively the same. Doing some testing on SDXL and DPO demos for the prompts specified: a) \"Buff Harry Potter\" and b) \"a gorgeous queen with cat like eyes\", I was able to get a correct figure for a) SDXL (6/10), DPO (10/10) b) SDXL (9/10), DPO (8/10). These are not rigorous tests,but they suggest handpicked results or preferential selection for GradSPO. I do not have access to GradSPO directly so I cannot perform a fair evaluation. However, the demos demonstrated for the GradSPO in Figure 4 are better even among the ones I sampled so I do not see a reason why the authors would pick poor images for the other methods, which damages the credibility of the comparison. I suggest revising the qualitative demos in Figure 4 to match the extended ones in the Appendix. For a fairer comparison multiple demos can be sampled for each model, scored and the best one for each method selected.\n\n- The user evaluation also raises concerns. Human preference studies are easy to bias via position effects or presentation order. The performance of GradSPO vs SDXL (~90% win rate) is a unusually high for aesthetic alignment so I urge the authors to describe their process and guarantee all the necessary steps have been taken for a fair comparison. It would also be better to follow SPO’s breakdown and perform a user study based on General Preference, Visual Appeal and Prompt Alignment categories.\n\n- Minor point: the ablation studies in Section 4.4 make a strong argument in favor of each component in isolation. However, winrate is a non transitive metric so SPO>SDXL and GradSPO>>SDXL does not mean GradSPO>>SPO, so a winrate vs SPO or a table with scores is more informative.\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GradSPO, a method designed to alleviate the time-step weighting bias inherent in SPO. The authors support their approach with theoretical analysis and further validate it through experiments on both SD1.5 and SDXL backbones.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is clearly written and logically organized, making it easy to follow the proposed methodology and its motivations.\n2. The GradSPO framework is theoretically well-founded, with analytical derivations that support its design and justify its components.\n3. The method’s effectiveness is demonstrated through experiments on two different backbone architectures, yielding compelling qualitative results that underscore its generalizability.\n\nWeaknesses\n1. The proposed function a(t) in GradSPO appears to be overly simplistic. It would be beneficial to explore alternative formulations—such as non-linear functions—or investigate whether a learnable parameterization could better capture the desired temporal behavior.\n2. While Equations (10)–(12) offer a solid theoretical argument, the paper would benefit from additional empirical evidence or illustrative visualizations to more concretely demonstrate the time-step weighting bias issue inherent in SPO.\n3. The experimental validation is limited in scope. It lacks evaluations on more recent and competitive backbone models such as DiT (e.g., SD3 or Flux). Moreover, it remains unclear whether the proposed method can be extended or adapted to the video generation domain, which would further demonstrate its versatility.\n\nQUESTIONS:\nAs highlighted in the weaknesses section, the primary concern centers around the simplicity of the proposed function a(t). While the idea is intuitive, the paper does not provide any ablation studies or empirical analysis to justify this specific design choice. Without such evidence, it remains unclear whether the performance gains are inherently tied to this formulation or if alternative strategies—such as more complex or learnable functions—could yield better results.\nFurthermore, the absence of robustness checks raises questions about the general applicability and stability of the method under different settings or perturbations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces GradSPO. It provides an analytical study on top of stepwise preference optimization (SPO), which is a promising direction in the field, incorporating step awareness during the process.\n\nAfter presenting the analysis, the paper proposes a simpler loss function compared to SPO and obtains strong results.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* Stepwise preference optimization is a promising direction in aligning diffusion models. Therefore, I believe the authors identified a good testbed to start with and trying to further reason about it and repurpose it.\n* Since the method relies on a noisy approximation, the authors also introduce a noisy reduction strategy to mitigate its repercussions.\n* The authors substantiate the simplification with impressive results on widely used metrics in the area.\n\n## Weaknesses\n\nSeveral important ablations and details seem missing from the main text:\n\n  * It uses uniform timestep weighting. How about other weighting schemes?\n  * What is the impact of using components like EMA and noise reduction? Even Figure 5 has win-rate comparisons, I believe a better ablation would have been to show how the loss landscape gets affected by the noise reduction strategy and also some qualitative samples.\n  * For step-aware models, memory and convergence speed can be an issue. Could the authors include a discussion on this?\n  * DPO and MaPO were trained on the Pick-a-Pic v2 dataset while GradSPO uses v1. Could the authors ensure that the methods being compared use the same base dataset?\n  *  The effect of the stop-gradient operator in equation 10 isn't quite clear.\n\nQUESTIONS:\n* Could the authors show results on a recent flow-based model? I believe models like Lumina2 [1], SANA [2], etc., are small enough to conduct an experiment.\n* Even though there is MaPO in the results section, I think it also deserves to be in the other relevant sections of the paper (such as Introduction).\n* I found the clarity of the symbols used throughout the main text of the paper to be somewhat hard to follow. Could the authors try to be more explicit with those?\n* In equation 3 and elsewhere, the authors mention the use of a timestep-dependent weighting factor. Even though it was there in DDPM [3], it's not used much in practice. Could the authors comment on that?\n* The existing reward models (like PickScore [4]) are not typically timestep-aware. This work, however, relies on timestep-aware reward models. Could the authors comment on it a bit more?\n* Equation 13 - can't we scale with the variance of $z$?\n\n## References\n\n[1] Lumina-Image 2.0: A Unified and Efficient Image Generative Framework\n\n[2] SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers\n\n[3] Denoising Diffusion Probabilistic Models\n\n[4] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation\n\nLIMITATIONS:\nThe authors didn't include a separate section on limitations in the main text nor did they make them clear in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}