Title: Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reward-Centric Efficient Image Synthesis
Abstract: This paper addresses the challenge of achieving high-quality and fast image generation that aligns with complex human preferences. While recent advancements in diffusion models and distillation have enabled rapid generation, the effective integration of reward feedback for improved abilities like controllability and preference alignment remains a key open problem. 
Existing reward-guided post-training approaches targeting accelerated few-step generation often deem diffusion distillation losses indispensable.
However, in this paper, we identify an interesting yet fundamental paradigm shift: as conditions become more specific, well-designed reward functions emerge as the primary driving force in training strong, few-step image generative models. Motivated by this insight, we introduce Reward-Instruct, a novel and surprisingly simple reward-centric approach for converting pre-trained base diffusion models into reward-enhanced few-step generators. Unlike existing methods, Reward-Instruct does not rely on expensive yet tricky diffusion distillation losses. Instead, it iteratively updates the few-step generator's parameters by directly sampling from a reward-tilted parameter distribution. Such a training approach entirely bypasses the need for expensive diffusion distillation losses, making it favorable to scale in high image resolutions. Despite its simplicity, Reward-Instruct yields surprisingly strong performance. Our extensive experiments on text-to-image generation have demonstrated that Reward-Instruct achieves state-of-the-art results in visual quality and quantitative metrics compared to distillation-reliant methods, while also exhibiting greater robustness to the choice of reward function.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the effect of rewards for fast image generation using pre-trained diffusion models. \nThe authors reformulate the problem of conditional generation with rewards, focusing on the reward and searching for samples that maximize it, while using the base generative model as a regularizer.\nAdditionally, they experiment with various reward functions and show that using a weighted normalized average of multiple rewards yields the best results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Clear background and motivation.\n* The authors offer an interesting re-formulation of image generation formula post pre-training by placing the reward in the center.\n* The evaluation is thorough and includes ablations and comparisons with relevant previous works.\n\nWeaknesses:\n* The paper needs proofreading as it contains many typos and grammatical issues, for example: \n“Our journey starts with an analysis in Section 3 shows that” (line 57), “our methods directly operates..” (line 66), “advanced prior distribution will definitely leads to..” (line 166), “converts pretrained pre-trained base..” (line 180), and more..\n* The limitations section is quite limited, I think the paper could benefit from further limitations discussion including images of failure cases.\n* There aren't many visual examples, and there aren't complex visual examples of the method as the ones presented in Figure 2 of limitations of the competing method.\n\nQUESTIONS:\n1. What are the results of Reward-Instruct over more complex prompts such as the ones presented in Figure 2?\n2. “Figure 6: The prior distillation-based reward maximization methods collapse when the reward is chosen to be HPS v2.1.” – what happens for prior approaches vs. Reward-Instruct when not using HPS v2.1? (e.g. using HPS v2.0 [60] or Image Reward [61] as in their original approach?)\n3. Figure 7 – there is a mild improvement when increasing the number of rewards used, but it seems more like enhancing the gradient step because the improvement is generally toward the same direction. My question is – what happens if you use a higher weight for one of the regularization methods instead of using the weighted average? \n4. In addition to text proofreading that is needed, in some figures it is not clear what are the prompts for the examples, e.g. Figure 8, or it looks like the wrong prompt/images, e.g. in Figure 7 the \"squirrel\" example which shows bird images. These should be fixed.\n\nLIMITATIONS:\nI think the paper could benefit from a broader discussion of the method's limitations, including visual examples of them.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a finetuning method that enables few-shot, preference-aligned text-to-image generation using an off-the-shelf pretrained diffusion model. The authors critically examine the importance and difficulties of reward maximization in the setting of diffusion models and alignment to human preferences, and they develop a novel yet simple objective to enable alignment and few-shot sampling. Reward-Instruct shows strong results in effectively maximizing human preference, per a set of image-based reward models, however the generalization capabilities of the final model, i.e. alignment w.r.t to reward models not seen in training, remains unclear.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Strong results w.r.t to few-step baselines and 50-step baselines\n\n- Principled formulation shows how to enable reward maximization and even shows how to extend guidance into intermediate denoising steps. Also demonstrates the benefit of reward ensembling.\n\n- Show extensions to image editing and super-resolution.\n\n- Paper is well written and motivations are clearly expressed. \n\n\n**Weaknesses**\n\n- Generalization of the final model is unclear. The authors seem to maximize the combination of HPS, ImageReward, and CLIP scores (I would appreciate it if the authors could confirm this and also clarify which HPS reward model in particular). However, their evaluations measure the scores from these same reward models plus the LAION Aesthetics model. Thus, it’s unclear if the preference alignment generalizes outside of these reward models– e.g. how do the results compare if measuring Pick-a-Pic reward model score, or a different version of HPS rm, or ideally if using human annotators?\n\n- Comparisons to reward-centric approaches. Many of the reported baselines, e.g. ReFL, optimize their model w.r.t to one reward signal whereas the benefit of ensembling rewards is a key finding of this paper. I understand that re-training such baselines may be infeasible, however I think it should be noted that the method benefits from multiple signals whereas many other baselines do not, when in fact their performance may improve with multiple signals too. \n\n- Comparisons to offline approaches. Given the prevalence of offline approaches like Diffusion-DPO, how does Reward-Instruct compare to SD-1.5 aligned via Diffusion-DPO or Diffusion-KTO?\n\nQUESTIONS:\n- Could the authors please provide results that measure preference alignment using a reward model that was not used during Reward-Instruct’s training? I acknowledge the Aesthetics results but the Aesthetics model is biased towards simple image aesthetics and may not capture much of the intricacies of preference alignment in the context of text-to-image generation. I think using Pick-a-Pic reward model could be a good experiment. \n\n- How does Reward-Instruct compare with Diffusion-DPO and/or KTO SD1.5 or other offline approaches?\n\n- Could the authors please elaborate on the design decisions when selecting reward models? The justification for reward ensembling highlights that maximizing the reward based on shared data modes across different reward models can help avoid reward hacking. In the paper however, the reward models used are somewhat similar in that they are optimized for some combination of image aesthetics and/or image-text alignment. Does this mean the reward models must be “sufficiently similar” in terms of what they reward? As a hypothetical example, if you included a safety classifier as one of the reward models would the training still be stable and, most importantly, effective?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Reward-Instruct, a novel framework for creating high-quality, few-step image generators w/o complex diffusion distillation losses. The core idea is to reframe the task as regularized reward maximization, where a few-step generator's parameters are directly optimized to satisfy reward functions. To prevent \"reward hacking,\" the method employs simple regularization techniques utilizing the models base parameters to approximate the base image distribution.\nThe authors investigate how to balance different reward functions and their impact on the downstream model. The experimental evaluation demonstrates that this distillation-free approach achieves state-of-the-art performance, surpassing previous methods in visual quality and quantitative metrics\n\nSTRENGTHS AND WEAKNESSES:\n# Strength \n\n- Unique and novel perspective on reward modeling and DM distillation makes for a strong methodological contribution \n- Rigorous methodological section with strong motivation and good mathematical background \n- Interesting insights on reward hacking, as well as  balancing different reward function and their influence \n- Experimental demonstration of effectiveness on 2 different DM paradigms: SD 1.5 for U-Net backbone and noise-estimate score function + SD3.5 for DIT backbone w/ rectified flow \n- Qualitative and empirical performance improvements look solid \n- Additional improvements in computational requirements \n\n## Weaknesses \nThere remain some minor weaknesses that could be improved \n\n- Many of the provided insights are only underpinned by a handful of cherry-picked examples (e.g. Fig. 4, Fig. 6, Fig. 7, Fig. 8, Fig. 11). The paper would benefit from corroborating these findings with quantitative experiments\n- Use of LAION Aesthetics classifier v1 instead of v2, despite the improved version being readily available \n- Inferring improved convergence from Fig. 10 (RewardInstruct+) is a bit of stretch. No improvement for HPS for example \n- The presentation can be improved. Especially the sudden introduction of RewardInstruct+ over more substantive experiments may confuse some readers \n- The paper would immensely benefit from a human user study on actual improvements in quality, since automated metrics are known to be faulty (and as discussed in the paper there may be reward hacking)\n\n-\n\nQUESTIONS:\n- Could you elaborate further on the intuition behind $\\hat{\\epsilon}_\\eta$ (L146)? Is this routed in prior literature on DM sampling? \n- What is the importance of randomly sampling $\\eta$ at different diffusion steps? Could you provide further analysis of the impact of that design choice?\n\nLIMITATIONS:\nDespite the authors claim in the NeurIPS checklist the paper does not discuss the limitations of the proposed method, nor do the authors provide insights on its failure cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Reward-Instruct, a reward-centric training approach designed to efficiently convert pre-trained diffusion models into reward-aligned few-step generative models without relying on diffusion distillation losses. The key claim is that reward maximization alone, guided by a stochastic Langevin dynamics-inspired optimization, can effectively yield high-quality few-step generation. Extensive experiments demonstrate strong quantitative and qualitative performance on text-to-image benchmarks, showcasing superior visual quality compared to existing distillation-based methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n\\* Reward-Centric Simplicity: The proposed approach elegantly bypasses the computational complexity and implementation difficulties inherent in diffusion distillation.\n\n\\* Strong Empirical Results: Impressive visual quality and improved performance metrics (FID, HPS, CLIP scores) across various text-to-image tasks.\n\n\\* Robustness via Multiple Rewards: Incorporates multiple reward signals effectively to mitigate common reward hacking pitfalls, achieving better image quality and stability.\n\nMajor Weakness\n\n\\*Conceptual Ambiguity Regarding Diffusion Sampling:\nThe proposed training method updates parameters by directly maximizing rewards in a few-step generation process, significantly deviating from standard diffusion score matching procedures. While the model structure still resembles diffusion-based epsilon-prediction, the learned outputs may no longer represent valid diffusion score functions. Thus, it is debatable whether the final model should be categorized as a diffusion model or rather a general generator shaped merely in the mathematical form of an epsilon predictor. The paper fails to clearly acknowledge or analyze this conceptual discrepancy.\n\n\\*Lack of Justification Regarding Diffusion Trajectory Preservation:\nTraditional diffusion distillation methods explicitly attempt to preserve the trajectories defined by a pretrained diffusion model, enabling stable and theoretically grounded sampling dynamics. Reward-Instruct completely omits trajectory-level constraints, potentially collapsing diffusion trajectories into reward-driven but arbitrary generation paths. Without any formal analysis or justification, it remains unclear whether this approach maintains meaningful diffusion trajectories or merely exploits reward-driven shortcuts.\n\n\\*Insufficient Justification of Reward Hacking Robustness:\nThe paper claims RI’s robustness against reward hacking compared to diffusion-distillation-based methods like DI++ or RG-LCM. However, no clear theoretical explanation or thorough empirical demonstration is provided. Moreover, it is unclear why similar robustness could not be achieved simply by applying reward maximization after standard diffusion distillation.\n\n\\*Critical Omission of Relevant Recent Methods:\nSeveral closely related recent works addressing reward-guided few-step diffusion methods are notably missing:\n\"[ICLR 2025] Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization\"\n\"[CVPR 2025] Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewards\"\n\"[ICLR 2025 Oral] DSPO: Direct Score Preference Optimization for Diffusion Model Alignment (DPO-based)\"\n\nMinor Weakness\n\n\\*Inconsistent Naming Conventions:\nThe method is inconsistently named (Reward-Instruct, RI, RO), creating confusion (e.g., Figure 5). Clearly choose and consistently use one abbreviation throughout the paper.\n\n\\*Unclear CFG Loss Description:\nThe algorithm presented in the appendix includes a term named \"CFG Loss\" without any explicit description or justification within the main body, making the motivation and necessity unclear.\n\n\\*Missing Algorithm Description for Reward-Instruct+:\nDespite introducing Reward-Instruct+ (intermediate-step supervision), the paper lacks a clear, formal algorithmic presentation for this variant.\n\n\\*Lack of SDXL Comparison:\nWhile experiments are conducted on Stable Diffusion v1.5 and SD3-Medium, no comparisons or experiments with SDXL—currently a prevalent and practically important baseline—are provided. Additionally, SD3-Medium results are presented without comparison to existing methods, reducing interpretability and value.\n\nQUESTIONS:\nSince Reward-Instruct directly maximizes rewards over a few-step sampling process without explicit diffusion distillation, can the resulting outputs still be interpreted as valid diffusion score functions? Or should the model be viewed simply as a general epsilon-conditioned generator lacking traditional diffusion properties?\n\nWithout explicit diffusion distillation, how can RI theoretically or empirically guarantee meaningful preservation of the original pretrained diffusion model’s trajectories?\n\nWhy exactly does Reward-Instruct provide increased robustness against reward hacking compared to methods incorporating explicit diffusion distillation losses (e.g., DI++, RG-LCM)? Could similar robustness be achieved simply by applying RI-style reward maximization after completing standard diffusion distillation training?\n\nHow sensitive is RI to variations in reward scales or reward noise? Can RI reliably maintain stability and image quality if rewards become noisy or biased?\n\nCould authors provide experiments using Stable Diffusion XL (SDXL), currently a prevalent baseline, to fully understand RI’s practical advantages and limitations compared to state-of-the-art few-step diffusion baselines?\n\nLIMITATIONS:\nRI relies strictly on differentiable reward signals, potentially restricting applicability to black-box or human-derived feedback scenarios.\n\nThe method significantly deviates from traditional diffusion model inference procedures, compromising interpretability and the theoretical underpinnings of diffusion sampling.\n\nRI’s robustness against reward hacking is claimed without rigorous theoretical or empirical validation, limiting the trustworthiness of this key assertion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}