Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation
Authors: Yihong Luo
Tianyang Hu
Weijian Luo
Kenji Kawaguchi
Jing Tang
Venue: NeurIPS 2025 poster ()
Keywords: Reward-Centric Efficient Image Synthesis
Primary area: reinforcement_learning
TL;DR: We propose a reward-centric approach to fast image generation that converts pretrained diffusion models to reward-enhanced few-step generators, without relying on complicated diffusion distillation losses or training images..

Abstract: This paper addresses the challenge of achieving high-quality and fast image generation that aligns with complex human preferences. While recent advancements in diffusion models and distillation have enabled rapid generation, the effective integration of reward feedback for improved abilities like controllability and preference alignment remains a key open problem. 
Existing reward-guided post-training approaches targeting accelerated few-step generation often deem diffusion distillation losses indispensable.
However, in this paper, we identify an interesting yet fundamental paradigm shift: as conditions become more specific, well-designed reward functions emerge as the primary driving force in training strong, few-step image generative models. Motivated by this insight, we introduce Reward-Instruct, a novel and surprisingly simple reward-centric approach for converting pre-trained base diffusion models into reward-enhanced few-step generators. Unlike existing methods, Reward-Instruct does not rely on expensive yet tricky diffusion distillation losses. Instead, it iteratively updates the few-step generator's parameters by directly sampling from a reward-tilted parameter distribution. Such a training approach entirely bypasses the need for expensive diffusion distillation losses, making it favorable to scale in high image resolutions. Despite its simplicity, Reward-Instruct yields surprisingly strong performance. Our extensive experiments on text-to-image generation have demonstrated that Reward-Instruct achieves state-of-the-art results in visual quality and quantitative metrics compared to distillation-reliant methods, while also exhibiting greater robustness to the choice of reward function.

Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation
Yihong Luo'; Tianyang Hu Weijian Luo², Kenji Kawaguchi', Jing Tang HKKUST *CUHK (sz) SXiachongshu Inc "NUS SHKUST (GZ)
Abstract
This paper addresses the challenge achieving high-quality and fast image gener- ation that aligns with complex human preferences. While recent advancements in diffusion models and distillation have enabled rapid generation, the effective inte- gration of reward feedback for improved abilities like controllability and preference alignment remains a key open problem. Existing reward suuddg post-training ap- proaches targeting accelerated few-step generation often deem diffusion distillation losses indispensable. However, in this paper, we identify an interesting yet funda- mental paradigm shift: conditions become more specific, well-designed reward functions emerge as the primary driving force in training strong, few-step image generative models. Motivated by this insight, we introduce Reward-Instruct novel and surprisingly simple reward-centric approach for converting pre-trained base diffusion models into reward-enhanced few-step generators. Unlike existing methods, Reward- Instructt does not rely on expensive yet tricky diffusion distillation losses. Instead, it iteratively updates the few-step generator's 's parameters by directly sampling from a reward- -eward-titedd parameter distribution. Such a training approach entirely bypasses the need for expensive diffusion distillation losses, making it favorable to scale in high image resolutions. Despite its simplicity, yields surprisingly strong performance. Our extensive experiments on text-to- -o-ige- generation have demonstrated that Reward- Instruct achieves state-of state-of-the-ar thet-rt results in visual quality and quantitative metrics compared iistillation-relian methods, while also exhibiting greater robustness to the choice of reward function.
seward-Instruee
Introduction
High-quality, controllable, and fast image generation stands as a paramount goal in the field of Artificial Intelligence Generated Content (AIGC), Recent advancements in diffusion models (57 [8
(33 and, particularly, diffusion distillation [56 28 have yielded impressive few-step image generators
capable of rapid synthesis of these advancements have significantly improved generation speed and visual fidelity, how to achieve improved controllability and alignment with complex human preferences remains challenging.
images and videos 47
48
68
30
32. While
Inspired by the success of reinforcement learning (RL) and more general reward-driven methodologics in large-language models 73 36 I1, the image generation community has made considerable strides in developing effective reward functions for images. These rewards, broadly include any discriminative model capable of evaluating image quality or adherence specific criteria, offer promising avenue for guiding generative models towards desired attributes, such as human preferences, instruction following, as well safety. Yet, the optimal strategies for effectively integrating and leveraging these reward signals in image generation workflows are still actively being investigated.
"Equal contribution.
1 Corresponding author: Jing Tang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

REWARD INSTURCT
strongly Figure I: 4-step samples at 1024 resolution generated by Reward-Instruct. The here is trained from SD3-medium purely by reward maximization.
Current standard practices for incorporating reward signals into image generation pipelines pre- dominantly occur as a post-training stage. These methods generally fall into two main categories: (I). integrating reward optimization directly into pre-trained diffusion model, often followed by a distillation process for efficient sampling, as exemplified by SDE control and other reward- based post-training techniques [13 8 (2). Developing methods that post-train some already distilled, fast sampling models by applying reward fine-tuning methods together with expensive diffusion distillation losses. Typical works this line Diff-Instruct+++ (DI++) (27]. However, both lines of approaches are considered computationally demanding and not inherently designed for directly converting pre-trained base diffusion models into few-step, reward-enhanced generators. Furthermore, despite the seemingly good metrics on benchmarks, close examination of sate-of-the-ar reward- enhanced few-step generators reveals evidence of reward hacking certain artifacts or repeated objects in the background, as illustrated in Fig. 2
Achieving reward-enhanced fast image generation hinges two key driving forces: the knowledge embedded within the base pre-trained generative model and the knowledge derived from reward signals. While language modeling has demonstrated the primacy of the latter, reward signals, such reward-centric approaches remain underexplored in image modeling, as diffusion distillation losses often considered indispensable in current works. With these observations, motivated by an important scientific problem:
Can we develop a reward-centric training approach, that can result in fast generation speed without the need of tricky yet expensive diffusion distillation losses?
In this paper, we give a positive answer to this question and introduce reward-centric method for training few-step generative models. Our journey starts with an analysis in Section 3 which shows that the post-training of DI++ is largely driven by reward signals, rather than the diffusion distillation objective for preserving knowledge of the image distribution from pretrained base models. This points towards phase transition to modern conditional generation tasks the increasing specificity and diversity of desired conditions or reward signals are causing fundamental shift in the generation process, moving away from primarily modeling the conditional distribution and towards regularized reward maximization.
Taking this perspective its logical conclusion, we propose novel reward-centric approach termed Reward-Instruct (RI), which is capable of efficiently training few-step generative models using reward. Fundamentally differs from existing approaches (27 (23] our methods directly operate on the base diffusion model and performs a direct reward maximization with simple yet effective regularization without the explicit requirement of separate distillation loss or training images. Specifically, we start from few-step sampler from pretrained diffusion models with enhanced stochasticity (via random 7) sampling in Section 3.2.J at each step. Our subsequent training process can be conceptually understood as directly sampling from reward-tilter distribution within the

---

Phrt Manuty thage
Meoil Baikic
sclortd
Prepoo withIc sOrr
Figure 3: Log Gradient norm curve of DI++ through train- ing process. We use the best configuration reported in DI++ 227
Figure 2: Samples are taken from the corresponding papers of DI++ and RG-LCM. It can be observed that certain artifacts exist in samples, e.g., repeated text/objects in the background. We hypothesize this comes from reward hacking.
generator's parameter space. Such a simple reward-centri design significantly improves the training efficiency, training stability, resulting in state-of the-art results in few-step text-to-image generation with high visual quality (Fig. []. Particularly, our method outperforms previous methods that combine diffusion distillation and reward learning regarding visual quality and machine metrics (Fig. and Table D). while being more robust to the reward choice (Fig. 6).
2 Preliminary
Diffusion Models. Diffusion models (DMs) 53 57 define a forward diffusion process that 2t 02 oge, where is sampled from the data distribution, and d specifies the noise schedule. By training a denoising function to predict the added noise, the model implicitly learns the score of the data distribution, enabling the generation of new samples by simulating reverse stochastic differential equation or ordinary differential equation [5754] 25] 77, 65). Conditional generation is
often achieved through techniques like classifier-free guidance (CFG), which modulates the denoising
process based on the desired conditions (57, 25 70 65 31. As illustrated in [27] CFG can be
seen an implicit reward condition alignment.
have been proposed for developing preference align few-step text-to-image models. Their approach can be sum- marized as a combination of distillation loss and reward loss: ming = Ldistin (20)) R[ c), where By denotes the model samples and R(xs, c) denotes the reward measure the alignment be- tween zo and condition c. The distillation loss can be consistency distillation [23] reverse-Ki KL divergence [27]. The previous method either requires real data for training (2], or requires training an extra online score model (27). These components increase the complexity of the post-training and serve solely as an overly complicated regularization.
Preference Alignment with Diffusion Distillation. Currently several methods 27
29 23
3 Reward-Instruct: A Reward-Centric Approach to Image Generation
3.1 Discriminative vs Generative: The Phase Transition
The concerning phenomenon of reward hacking, as visually evidenced Fig. strongly suggests potential imbalance where the generation process becomes excessively driven by reward models, potentially at the expense of image quality and generalization. To further verify this, our examination of the gradient norms of the reward loss and the distillation loss in ce-implementation of DI++ (27). shown in Figure 3 clearly indicates that updates in DI++ are overwhelmingly dominated by the reward term, relegating the diffusion distillation objective secondary role.
Extending this observation, we observe similar phenomena when considering generalized concept of rewards that encompasses any discriminative model capable of judging the goodness of generated samples. In modern text-to-image generation models, various guidance modules for aligning con- ditions [19 2 33 are disproportionately amplified compared with the seemingly more important diffusion generation component. For instance, large CFG coefficients are indispensable (7.5 by default in Stable Diffusion [46] and 100 in DreamFusion 42]). When dealing with strong conditions, such as generating an image of "a red cube on top of blue sphere behind a green pyramid"" a higher CFG value leads to more semantically compliant images.
These findings prompt a fundamental rethinking conditional generation tasks with strong conditions, such aligning to various reward functions in text-to-image generation. For these tasks, we usually model the conditional density p(rly) through the decomposition p(mly) x p(x)) p(ylar), where the terms correspond the marginal density and discriminative model respectively. In diffusion
3

---

7
1 0.3
? ?)
Samples with 100 NFE and 7.5 CFG by varying the r sampling. The samples are generated from the same initial noise.
Figure 4:
models, learning the marginal density p(z) is usually the main focus while the condition part is usually handled by CFG external guidance in diffusion models [3 However, conditions get stronger and more complicated, the conditional distribution may ill-conditioned to estimate, since the sample size for each condition is oftentimes only 1.
This leads us the key insight: for tasks with strong conditions, there is a fundamental shift in the nature of the problem moving away from directly estimating the conditional distribution towards what more accurately described as regularized rewards maximization, where the discriminative part p(y) æ) captured by various rewards becomes the primary driving force, and the base generative model acts as regularizer. the following section, we will formally reformulate the problem of conditional generation with rewards.
3.2 Regularized Reward Maximization for Image Generation
Let R(x) denote a reward function mapping Tid + R. We can characterize the reward-centric generation task as searching in the image domain for samples that maximize rewards, which is naturally an optimization problem. To characterize the image domain constraint, consider having access the image likelihood p(x) for natural images. If p(æ) is below certain threshold, can be deemed outside the image domain. Therefore, we can rethink the generation as a regularized reward maximization problem with the following objective:
max R(x) s.t. p(x) S. ØER
(3.1)
where e >> is some threshold. With this formulation, there are important design choices to consider: (1) How to parameterize images? (2) How to effectively optimize the generator while preventing reward hacking? We will demonstrate in later sections that with well-parameterized generator and proper regularization, we can achieve state-of-the-an text-to-image generation with only few reward functions, without relying on diffusion distillation.
3.2.1 Parameterizing Images with Generator
To achieve fast image generation, natural strategy is to utilize GAN-style generators that directly transform random noise into images. With this formulation, the optimization problem shifts from the image space to the generator's parameter space. Ideally, the generator should have large-enough capacity and reasonable initial grasp of the image distribution. We address this by proposing the use of an unrolled few-step sampler directly from pre- pre-trained diffusion models. Conveniently, the number unrolled steps allows us to control both the generator's capacity and its initial knowledge.
Specifically, we parameterize the generator go.g to accept K noisy levels of a diffusion model as inputs, creating network that progresses from noise to obtain clean samples in K steps. The specific parameterization is as follows:
(z) = go.or.n 90....m. 90. (2), 90. (0..).((mx.) = V1 1- 0z_
(cca(eE) v1
o (3.2)
where ē = ( ) Vi- σk 1 and z, (0, I). We can use pre-trained score net
as the initialization for enough model capacity and better initial images. Our optimization target is 0, which can be the full score network parameters or low-rank adaptation them (LoRA) 20
Random 7-Sampligg. The 7/ plays important role in our parameterized generator go.n- In
particular, when 7) = 1, the generator is parameterized into the discrete format of DDIM sampler (54) In practice, find that varying 77 in diffusion sampling results in significant differences in the style

---

and layout of the generated images, as shown in Fig. Motivated by this, we propose to parameterize our generator by inputting random 17 at each step for augmenting the generator distribution that allows the generator explore more diverse area:
g (2) = 90, 00.02.0--- o 20,0x (z);
(3.3)
where 0 1] and 1, K. After training. we can fix 77 [0, 1] in sampling. The design can effectively augment the generator distribution by randomly adding stochastic, serving as an effective regularization to enhance performance.
3.2.2 Optimization as Sampling from Reward-tilted Distribution
The generator (z) initialized from the base diffusion model provides non-trivial starting point for reward-centric training. We aim to adjust the parameter 0 such that generated images are more preferred by the rewards. Formally, we can define the target 0* to be distributed following reward- informed posterior distribution
p (0) oc (o()) exp( [(0)).
where = =(0)= Ez (2))) and po( is the initial distribution from the base diffusion models, which can be viewed as the likelihood to specify the image domain likelihood. For simplicity, we choose (0) to be Gaussian distribution centered in the pretrained 0g with variance g² More advanced prior distributions will definitely lead to better performance.
Remark 3.1. The above formulation is similar to existing works in reward-driven approaches 40 [1
44 8] Given base generative model with base distribution (x),, the reward-tilted target is usually defined as p" (z) x po exp {.())). }. The key difference that our formulation on the generator parameter space.
39
With the target distribution p" specified, a trraightfforward method obtain 0* ~ p" (0) using stochas- tic Langevin dynamics. Concretely, starting from ((0) = 0o. ((++1) can iteratively calculated via
the stochastic gradient Langevin dynamics (SGLD) (59
((++1) g((0 = AVlog (po)) exp (R(((()) + +V 2
\VR(O
Vi100 0o113 22
(3.4)
20
Rewards maximization
Regularization
where À is the learning rate and é ~ (0, 1) a random noise. As can be seen above, each update constitutes of reward maximization and regularization. The regularization is also two-fold, one being an 12-penalty or weight decay, the other one being random noise perturbation.
Elucidating the Design Space of Reward-Instruct
By incorporating all aforementioned designs, we have developed surprisingly simple yet effective reward-centric approach to fast image generation. Rooted regularized reward maximization, we call our method Reward-Instrus and it is framework that directly converts pre-trained base diffusion models into reward-enhanced few-step generators. Detailed algorithms are summarized in Algorithm in in the Appendix.
Such simple reward-centric design significantly improves the training efficiency, training stability, resulting in state-of-the-an results in few-step ext-to-image generation. Fig. iillustrss the qualita- tive comparisons among other methods. Detailed comparison for computation cost are deferred to Appendix B.I. Next, We proceed to explore the various facets of its design space.
3.3.1 Form of Regularization
The primary role of regularization is constrain the generator's distribution to the vicinity of the image manifold, thus avoiding reward hacking. In our formulation, the specific form of the O-regularization dependent on the form (0). Choosing Gaussian distribution will give rise to l2 regularization. This is very similar to the KL penalty in RL to control the update to be not too large (66) The random noises introduced in the SGLD sampling algorithm also provide regularization effect against reward hacking. Its effect is ablated in Appendix B.2
5

---

photo of on sunglass
photo of Pikachu, quality
photo of quality
small cechus with heppy desert.
birthday
monkey making
estronaut walking
Base DO NFE
Hyper 4NFE
Distillations w/HFF
RG-LCM 4NFE
DI 4NFE
4NFE
HFL only
RI+ 4NFE
LEL
Figure 5: Qualitative comparisons of Reward-Instruct against distillation based and diffusion based models on SD-v1.5 backbone. All images are generated by the same initial noise. We surprisingly observe that our proposed Reward-Instruct has better image quality and text-image alignment com- pared to prior distillation-ban based reward maximization methods 4-step text-to-image generation.
Diffusion distillation as regularization. More closely related to our generator parameterization, existing works such as RG-LCM (23) and DI++ 27 introduce reward maximization into diffusion distillation. These methods typically require training an additional score model for the distilled generator to ensure its closeness to the original model, which is memory and computation-intensin Surprisingly, we found that the reward gradient dominates throughout the training process, turning diffusion distillation into sort of costly regularization role (Fig. 3). Specifically, the RG-LCM and DI++ employ HPS v2.0 [6]] or Image Reward (62) as the reward loss their original approach, but discovered that if the reward is
HPS v2.1, it would cause the generators of RG-LCM and DI++ to collapse into undesirable distributions, as shown in Fig. 6
Built upon the above formulation and regularization techniques, we found that our method integrated with HPS v2.1 1 can generate high-quality images, suffering less from artifacts compared to RG-LCM and DI++ (Fig. 7 where each regularization effectively improves image quality. This indicates the importance of proper regularization in reward maximization. However, we found that optimization with single explicit reward still suffers from the over-saturation issue. To address this, we suggest maximizing multiple explicit rewards. This indicates the importance of proper regularization in reward maximization.
3.3.2 Power of Multiple Rewards
Thanks plethora preference data and powerful RL methods, we have access to diverse collection of learned reward models. Although learning from zoo of pre-trained models has long been studied various vision tasks 660 9 10), utilizing multiple reward models is relatively underexplored. Denote Rí (x)).... ... R., (x: as the reward functions, each with its own set of modes, some genuinely good and some corresponding to artifacts. Our assumption that the good ones are associated with the ground truth, while the bad ones random and not shared with other reward

---

of
photo dey
pleto
RI (SFG) random cta wight negulanzztion
RG- LCW
wught regulorzzttion
RE (CHG-165)
CF only
Figure 6: The prior distillation- based reward maximization methods collapse when the reward is chosen to be HPS v2.1. In contrast, our deward-Instrued still works well, benefiting from the proposed effective regularization technique.
Alackua toked
Nlocky bakind
Pilccial heleed
HFS
HP5+ Image Revard
HP5- Lmage Reward <clp Scone
Figure 7: Four-step samples generated by Reward- Instruutt We observed that the quality of the image monotonically increases with the gradual increment of the reward count.
functions. Therefore, with a diverse collection of different reward functions focusing on different aspects of the data, images preferred by all of them tend to be good.
Therefore, the goal of utilizing multiple rewards is find common data modes that have high rewards from different perspectives. This design can effectively prevent the model from hacking each individual reward, therefore significantly stabilizing the training process. We can extend (3.1) to
m
max
wi (x) s.t. p(x) c.
(3.5)
BER
where Wj's 's are positive weights for balancing the rewards. However, all have is gradient pointing towards the direction of the steepest climb. How to find the most effective direction is of critical importance. In practice, combination may fail to maximize all rewards in the training process. As shown in Fig. 9 the clip score does not converge
found that optimizing multiple rewards with naive weighted
high value.
To balance the learning of different rewards, we suggest gradient normalization, normalizing the gradients from each reward and forming the average direction. Therefore, can choose the weights w; in 3.1 as
W, [sg(( (g((((z)) 0g(z) (12).
(3.6)
This is equal to setting a dynamic weighting. Note that the normalizing operation is also performed for the implicit reward (CFG). Fig. 9 shows that after applying the gradient normalization, can maximize multiple rewards well. Fig. 7 demonstrates that the image quality and image-text alignment become significantly better when maximize multiple rewards.
The Complementary Effect of Rewards. Since the common modes of multiple rewards tend to be more well-behaved than those from signal rewards, the combination of multiple rewards in (3.6) also serves kind of implicit regularization for the generator. To verify this, we maximize HPS. image reward, and clip score separately, without using random-1j sampling. We find that individual rewards perform poorly. but when combined, they can generate images of reasonable quality as shown in Fig. g which highly emphasizes the complementarity between rewards and their effectiveness as implicit regularization.
7

---

Only
Image Ony
Cip Score Only
Muttiple Rewards
Figure 8: The complementary effect of different reward. We do not use random sampling and set small weight regularization in training here to highlight the complementary effect between rewards.
3.04
Rarmal leed
3.28
91-
8.71
28 Parabore 1550)
20 20
IF
13 23
11
30
13 25 30
Figure 9: Training progress of various metrics over iterations. It can be seen that the normal- ized gradient shows better performance. This is evaluated on 1 prompts from HPS benchmark.
Figure 10: Comparison on the convergence speed between RI and RI+. We note that each training iteration of RI+ takes only 65% of the time required by RI.
3.4 Reward-Instruct+: Additional Supervision on Intermediate Steps
The reward supervision in Reward-Instruct only the final generated samples in end-to-end fashion, which is similar to the DeepSeck-RI I 77. To further enhance performance, we can incorporate extra supervision into intermediate generation steps to form neward-Instruck.
Although Reward-Insture is already capable of generating high-quality images in a setting where only reward maximization is considered, the reward signal is only provided at the end. This leads to two issues: on one hand, the efficiency of reward maximization is low because there is no direct reward signal feedback during the process; the other hand, the gradient needs to be backpropagated through the entire generator, resulting in high memory usage and significant computational costs.
To address the above issues, we propose learning the generator with intermediate supervision well, forming Reward-Instrucet. We rewrite the generator as follows:
sg(g8, 724 14:3443 go (z)),
o #k+1 044100(1221)
okkma a (k)
16+1 T 164
(3.7)
36
-
o
where k = 1,2, K, U0, 1]. En, ThEa (x)+1) + For computing CFG, we diffuse samples from æg. The k randomly sampled during training. The update of 0 has the same form as Eq. (3.4), differing in how to obtain samples.
ox = and 2, NNO I).
By doing so, can effectively supervise the intermediate process in the generation process. Fig. 10 shows that after applying the intermediate supervision, the convergence speed of Neward-Instrucee has significantly improved compared to than RI, since each training iteration of RI+ takes only 65% of the time required by RI.
Moreover, training RI+ is more efficient
Reward- Comparison on Generation Path Between Reward-Instruct and Reward-Instruce Instruct and Instrucct are significantly different and it is interesting to explore the generation paths corresponding to these two models. In Fig. 16 we can observe that Reward-Instruce generates much clearer images during the early process compared to Neward-Instruee Although Reward- Instruct+ struggles with severe artifacts in the early stages of generation, surprisingly, these artifacts are gradually removed rather than accumulating throughout the process. In contrast, Reward- path progresses from blurry to clear, with less affected by artifacts during the process.
4 Evaluations
To verify the effectiveness of the Reward-Instruc and Reward-Instruce we compare them with previous distillation-banl reward maximization methods. We put the experiment details in the Appendix.

---

Table 1: Comparison of machine metrics on text-to-image generation across SOTA methods. We
highlight the best among fast sampling methods. The FID is measured based on COCO-5k dataset.
Model
Type
Backbone
NFE
HPSt
Aest
CSt
FID.
Image Reward
Base Model (Realistic-vision)
SD-v1.
30.19
5.87
34.28
29.11
0.81
Hyper-SD 45
SD-v1.5
4
30.24
5.78
31.49
30.32
0.90
RG-LCM
Distillation Reward
SD-v1.5
31.44
6.12
29.14
52.01
0.67
DI++ 27
SD-v1.5
31.83
6.09
29.22
55.52
0.72
ReFL. 62
SD-v1.5
50
31.82
5.97
31.78
39.38
1.16
Reward Centric
DRaFT
SD-v1.5
33.10
6.18
30.70
37.10
0.85
Inward-Instruct (Ours)
Reward Centric
SD-v1.5
33.70
6.11
32.13
33.79
1.22
Reward-Instructe Instruct+ (Ours)
SD-v1.5
34.37
6.20
32.97
37.53
1.27
Base Model
SD3-Medium SD3-Medium
31.37
5.84
34.13
28.72
1.07
Reward- Reward-Instruct Instruct (Ours)
Reward Centric
4
34.04
6.27
33.89
31.97
1.13
w/ Dreamshaper
Image Editing: squirnel -> cat
ControlNett: Conny + Image
Figure 11:
Qualitative comparison against competing methods and applications in downstream tasks.
Table 3: Ablation study on the proposed components in our Reward-Instruct.
Model
Backhone
Steps
HPS
Aest
CSt
FIDJ
Image Rewardf
Reward-Instruce (Ours)
SD-v1.5
4
33.70
6.11
32.13
33.79
1.22
Inward-Instruct w/ single reward
SD-v1.5
4.
32.08
5.80
31.01
38.21
0,89
Reward-Instrus w/o random-ry-sama -amplligg
SD-v1.5
4
33.41
6.12
32.11
36.90
1.18
Reward-Instrucd w/o weight regularization
SD-v1.5
34,26
6.12
32.45
39.25
1.27
Metric. We assess image quality using the Aesthetic Score (AeS)
150
image-text alignment and
human preference with the Human Preference Score (HPS) v2.1 [1] and Image Reward, and image- text alignment with the CLIP score (CS) [16]. Additionally, we use zero-shot FID) on the COCO-5k dataset for more comprehensive evaluation.
Qualitative Results. We present the qualitative results in Fig. 5 It can be observed that our proposed Reward- Instruct and image quality and text-image alignment compared to existing distillation based reward maximization methods and RL-baaed finetuning methods. We also include additional visualization regarding complex prompts in Fig. 15
without using a distillation loss, demonstrate better
User Study. To further verify the effectiveness of our proposed method without concern for reward- wackddh we conduct a user study on SD-v1.5 backbone. We refer to Section B.2 for details of the user study. The results in Table 2 show that our method outperforms the base diffusion model.
Table 2: User Preference Study.
Model
NFE
User Preferencet
Base Model RI (Ours) 4
50 41.7%
58.3%
Quantitative Results. We present the quantitative results in Table I Our proposed Reward- Instruct and Reward- achieve state-of-the-art (SOTA) performance across various text-image alignment and human preference metrics. Notably, our model also achieves a zero-shot COCO FID comparable to the original model, which demonstrates that our method does not suffer from artifacts.
Ablation Study. We provide quantitative results about removing one technique at time to show the effectiveness. The results in Table 3 demonstrate the critical role of each component: 1) Multiple rewards: Improves both rewards and FID, mitigating artifacts and reward hacking. 2) Random- sampling: Maintains similar reward performance but significantly improves FID, aiding to find better mode with fewer artifacts. 3) Weight regularization: Trades slight reward gains for better FID, ensuring the generator stays within the image manifold.
Additional Application. We show our Reward-Instruct's capabilities various tasks: 1) Image-to- Image Editing: As shown in Fig. 11 Reward-Instrus performs high-quality image editing (35 in

---

just four steps; 2) Compatibility with ControlNet and Base Models: Illustrated in Fig. II Reward- Instruct-LoRA is compatible with ControlNet [6 and works seamlessly with various fine-tuned base models (e.g. Dreamshaper from SD 1.5), preserving their unique styles.
Discussion
Our work targets reward-centric approach photo-realistic image generation. Our proposed Reward-Instruct demonstrates that via regularized reward maximization, we can convert pretrained base diffusion models reward-aligned few-step generators, without diffusion distillation losses or training images. Our method enables 4-step 1024px generation (Fig. T. matching or exceeding both
the inference speed and sample quality of previous approaches. In Appendix C. we further explore more application scenarios and extensions where our method can shine.
Our approach is inspiring but have its limitations. For instance, method relies trained differentiable reward functions. It's an interesting future work to extend such reward-centric approach to include black-box reward functions. It would be exciting to explore DPO-style [44 pipelines that directly fine-tune generators with raw preference data, and extending these principles to other generative domains like video and 3D content. Moreover, our Reward-Instruct framework can potentially work with any few-step generator and doesn't it have to relate to diffusion models but this is not explored in the current work. Making this investigating is an important extension.
Acknowledgments
Jing Tang's work is partially supported by National Key R&D Program of China under Grant No. 0024YFA1012700 and No. 2023YFF0725101 by the National Natural Science Foundation of China (NSFC) under Grant No. 62402410 and No. U22B2060, by Guangdong Provincial Project (No. 20230N10X025), by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2023A15151 10131, by Guangzhou Municipal Science and Technology Bureau under Grant No. 202440414454, by Guangzhou Municipal Education Bureau (No. 2024312263), and by Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628) and Guangzhou Municipal Key Laboratory of Financial Technology Cutting-Edge Research (No. 2024A0300630).
References
[1] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint urXiv:2204.05800 2022.
[2] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
[3] Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13. 30.3005513300 2023.
[4] Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi-Ming Ma. Explore and exploit the diverse knowledge in model zoo for domain generalization. In International Conference on Machine Learning, pages 4623-4640 PMLR, 2023.
[5] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprini 230.::2309.740000 2023.
[6] Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang. Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15800 2023.
[7] DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability lmms via reinforcement learning, 2025.
10

---

[8] Carles Kaningo-Enniche Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprini arXiv:2409.08861, 2024.
[9] Qishi Dong, Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung-Ho Bae, and Zhenguo Li. Zood: Exploiting model zoo for out-of-distribution generaliza- tion. Advances in Neural Information Processing Systems Volume 35, 2022.
[10] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dicken Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and memc. In International conference machine learning, pages 1889- PMLR, 2023.
[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
[12] Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimization, 2024.
[13] Ying Fan and Kangwook Lee. Optimizing ddpm sampling with shortcut fine-tuning, arXiv preprint arXiv: 13362, 2023.
[14] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuming text-to-ine eex-to--mmaee diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[15] Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and Zico Kolter. Consistency models made easy. arXiv preprini arXiv:2406. 140.44448884488, 2024.
[16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, arXiv preprint arXiv: (1t::2004008716 2021.
[17] Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 0851, 2020.
[19] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:200..12598 2022.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[21] Dongjun Kim, Chich- Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji. and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:3310.02279, 2023.
[22] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. /2192, 2023.
[23] Jiachen Li, Weixi Feng, Wenhu Chen, and William Yang Wang. Reward guided latent consis- tency distillation. arXiv preprint arXiv:2403. //07, 2024.
[24] Xuantong Liu, Tianyang Hu, Wenjia Wang, Kenji Kawaguchi, and Yuan Yao. Referee can play: An alternative approach to conditional generation via model inversion. arXiv preprint 163. 2024.

---

[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0099 00927, 2022.
[26] Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models. arXiv preprint arXivv2330..42222 2023.
[27] Weijian Luo. Diff-instruct++: Training one-step -ontteo-mmae generator model to align with human preferences. arXiv preprint arXiv:2410. 18881, 2024.
[28] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances Neural Information Processing Systems, 36, 2023.
[29] Weijian Luo, Colin Zhang, Debing Zhang, and Zhengyang Geng. Diff-instruer Towards human-preferred one-step text-to-image generative models. arXiv preprint 44X1v2241,,20898, 2024.
[30] Yihong Luo, Xiaolong Chen, Xinghua Qu, Tianyang Hu, and Jing Tang. You only sample once: Taming one-step text-to-imaae synthesis by self-cooperative diffusion gans, 2024.
[31] Yihong Luo, Tianyang Hu, Yifan Song, Jiacheng Sun, Zhenguo Li, and Jing Tang. Adding additional control to one-step diffusion with joint distribution matching. arXiv preprint arXiv:2503.06652, 2025.
[32] Yihong Luo, Tianyang Hu, Jiacheng Sun, Yujun Cai, and Jing Tang. Learning few-step diffusion models by trajectory distribution matching. arXiv preprini arXiv:2503.0663 2025.
[33] Jiajun Ma Tianyang Hu, Wenjia Wang, and Jiacheng Sun. Elucidating the design space of classifier-guided diffusion generation. arXiv preprint 1331 2023.
[34] Jiajun Ma, Shuchen Xue, Tianyang Hu, Wenjia Wang, Zhaoqiang Liu, Zhenguo Li, Zhi-Ming Ma, and Kenji Kawaguchi. The surprising effectiveness of skip-tuning in diffusion sampling. arXiv preprint arXiv:2402.151700 2024.
[35] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01011 2021.
[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[37] Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb: A benchmark for generative image understanding, 2023.
[38] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205 2023.
[39] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117 2019.
[40] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750 2007.
[41] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.0199: 2023.
[42] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2022.
2d diffusion. Xixiv preprint
12

---

[43] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.037.9, 2023.
[44] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024.
[45] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis, 2024.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[47] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representutions, 2022.
[48] Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching. arXiv preprini urxiv:2446.004133, 2024.
[49] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311. 2023.
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25222 2022.
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[52] Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4733-4743, 2024.
[53] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[54] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502,, 2020.
[55] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations.
[56] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint trXiv:2303.01469, 2023.
[57] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[58] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 82388, 2024.
[59] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-II), pages 681-688, Citeseer, 2011.
13

---

[60] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontij-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965- PMIR, 2022.
[61] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.099 2023.
[62] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings the 37th International Conference on Neural Information Processing Systems, pages 15903-15935, 2023.
[63] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8196-8206, 2024.
[64] Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. arXiv preprint arXiv:2402.17376 2024.
[65] Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi- Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36:77632-77674, 2023.
[66] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2024.
[67] Zilyu Ye, Zhiyang Chen, Tiancheng Li, Zemin Huang, Weijian Luo, and Guo-Jun Qi. Schedule on the fly: Diffusion time prediction for faster and better image generation. arXiv preprint arXiv:2412.01243, 2024.
[68] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311 18828, 2023.
[69] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[70] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprini arXiv:2204. 13902, 2022.
[71] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing $ystems,,6:49844 36:49842 49869, 2023.
[72] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024.
[73] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909: 190.00899900 2019.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The introduction part outlines the paper's structure. The abstract summarizes the main body of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

We have discussed our limitations in the Discussion section, and those limita- tions leads future work.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This work doesn't contain any theorem. However, for math derivations, necessary details are included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the main part, we discussed the algorithm and experiment setup. More details for reproducibility included in the appendix. Furthermore, we will release codes
got accepted.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We will provide code and data for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.:biiiiiiiooooooo.iooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experiment settings and details are included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Most experiments are very expensive to run and do not have multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information on the computer resources is discussed in detail in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpps:/neuurpps. http:///wwarr..ccoobb////cccccce
Answer: [Yes]
Justification:
The research conducted in this paper conform with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss it in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
19

---

Justification:
We discussed it in the appendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used this paper have been properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not release new assets in the submission phase.
Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Additional Related Works
Few-Step Diffusion Sampling. Despite significant advancements in training- free accelerated sam- pling of DMs (25 7T 64 34), diffusion distillation [26] is dispensable for satisfactory few-step
Sampling. Typically, the distilled sampler involves a single or multiple transformations from random noise to images. Among various approaches, trajectory matching [56 21 55
15 43 and distribution
matching 688 28 72
49
63 30 are the most popular methods for diffusion distillation in few-step
diffusion sampling. Very recently, trajectory distribution matching 32 has shown its promising
performance in distillation.
Preference Aligmment for Text-to-Image Models. In recent years, significant efforts have been made to align diffusion models with human preferences. These approaches can be broadly categorized into three main lines of work: 1) miee-tunigg DMs on carefully curated image-prompl datasets [6]
2) maximizing explicit reward functions, either through multi-step diffusion generation outputs 43
22 17 or policy gradient- based reinforcement learning (RI. methods [14] 3 67]. 3) implicit reward
maximization, exemplified by Diffusion-DPO [58] and Diffusion-KTT KTO [66] directly utilizes raw
preference data without the need for explicit reward functions.
Reward driven image generation. There is an active line of research investigating using various rewards in the post-training process for improved image alignment. For instance, [3] proposed to fine-tune DDPM samplers via policy gradient, achieving good few-step sampling performance. K considered reward fine-tuning diffusion models or flow models via stochastic optimal control (SOC), and proposed Adjoint Matching which outperforms existing SOC algorithms. ReNO [2] optimizes the initial noise by maximizing multiple rewards for enhanced performance given frozen one-step generator. Our Reward-Instrus significantly differs from existing works in that it is the first reward-driven few-step image generation method that directly converts pretrained diffusion models to reward-enhanced few-step generators, without relying on complicated diffusion distillation or training data.
Another closely related work "Referee can play" where the authors emphasized the importance of discriminative models in conditional generation and presented text-to-image generation pipeline by inverting Vision Language Models (VI.Ms). Specifically, they utilized the decoder from stable diffusion and optimized the latent for maximizing the alignment score given by VLMs. On high- level, the VLMS employed in 24 can also be viewed as reward models, providing matching scores for text-image pairs. Even though [24] provides an interesting proof-offconcept demonstration, it lacks a systematic formulation and its generating process has major downsides in real-world scenarios. (1) The optimization process is highly sensitive to initialization and hyperparameter choices, which is not robust. (2) To generate a new image, they have to do hundreds of function evaluations (NFEs), which is computationally intensive. Visual comparisons are illustrated in Figure 12
A brown jacket and black hat
A black squirrel and a brown nut
Referee Can Play
Reward-Instruct
Figure 12: Comparison with Referee can play (44. The baseline samples taken from their paper. can be seen that our Reward-Instruct has significantly better visual quality.
22

---

Algorithm RewarddInnstruct
Require: Generator fo, Pre-trase Pre-trained score fws Reward models {ri}, desired sampling steps K, total iterations N, learning rate
Ensure: optimized generator fo- Initialize weights by x/;
2: for i to N do
3: Sample noise e from standard normal distribution;
4: Sample with initialized noise from generator fo by K steps via random-n) sampling.
5: # Compute regularization loss
6: Grey + 0 1
7: # Compute Rewards
8:
Creward + E;
Rx(2,y)
sg(ll ( (2)
9:
# Compute CFG guidance loss
10:
Sample 2t g(x;(x)
11:
cfg grad f- Va, log (cl
Lafa - |x, - sg(a, + cfg grad)
13:
# Compute Total Loss and Update
14:
Ltatal f- Wreghreg + Lreward + Wefghefa
15:
Update f using SGLD step with temperature r: 0 t VV + 22c..
end for
Listing 1: Torch-style pseudo code of SGLD step with temperature т.
loss backvard
noise_scale (2 learning_rate rate tau) ** 0.5 / for param in model. parameters
learning.rate
with torch. (o_grad():
if param grad not None: noise_para torch. randn_ like (param grad data)
noise_scale
param grad. data, add_ (noise_para)
optimizer (t
optimizer zero ()
B Experiment Details
Baseline Models. We perform experiments SD-v1.5 [46] and SD3-medium [1], including both
UNet and MM-DiT [I] 38 architectures, indicating the broad applicability of our approach.
Experiment Setting. Training is performed on the JourneyDB dataset [37] using prompts, without requiring images, as our method is image-free. We primarily compare with previous distillation reward maximization methods. For Hyper-SD, we use the public checkpoint, while for RG-LCM and DI++, we reproduce their methods. We adopt the AdamW optimizer with 3 = 0.9, B2 0.95, and the learning rate of 2e 5. We use a batch size 256.
Details of User Study. We randomly selected 20 prompts for image generation. Around 20 user responses are collected on 20 pairs in total. We note that multiple users review the same pair of images, which potentially reduces evaluation bias.
B.1 Training Efficiency of Reward-Instruct
Our proposed Reward-Instruct is efficient to train, since does not require training images and does not require online auxiliary models during training. And its overall training cost is also not expensive compared to multi-step RI.-based methods and distillation based RI. methods, as shown in the Table
23

---

Method
Few-step
Image-Free
Single Trainable Network Training Cost
DRaFT
Hours
ReFL
X
64 Hours
RG-LCM
X
20 Hours
DI++
36 Hours
Reward-Instruct (Ours)
28 Hours
Table 4: Comparison the training efficiency. Our method is efficient for training and deploying in various aspects. The training cost is measured by GPU hours on RTX-4090. We use the same batch size and iterations for each method.
squirmel car
airplanc flying
pretty
squirnel
airplone flying
pretty
like
lastion
reguionzzation
Figure 13: The effect regularization loss. Images are from the same initial noise.
spider
photo of cat
bot
corgi sopcorn
photo of bet man
popcorn
w/o sgld upocte
sgid spdate
Figure 14: The effect SGLD update. Images are from the same initial noise.
Table 5: Comparison between Reward- using different rewards.
Model
Backbone
Steps
HPSt
Acst
CSt
FID.
Image Reward
Reward-Instruct (Ours)
SD-v1.5
33.70
6.11
32.13
33.79
1.22
Reward-Instruct w/ HPS Clip Score + AcS (Ours)
SD-v1.5
4
32.35
6.21
32.43
31.26
1.01
B.2
Additional Ablation
The Indispensability of Weight Regularization. The regularization loss is the key to learning good generator. Without weight regularization, even if we employ multiple rewards, the generator can easily converge to an undesirable distribution, which underscores the importance of explicitly regularizing the generator, as shown in Fig. B3
The Indispensability of SGLD's randomness. The SGLD randomness also serves key to regularize the generator. We observe that by introducing SGLD's randomness learning, the model more robust to avoid artifacts. Under larger reward weightings, without SGLD's randomness, the model may generate some repeated objects in the background related to the prompts, as shown in Fig. 14
Different Rewards for Training Reward-Instruct. We provided additional experiments incorporat- ing Aesthetic Score (AeS) in Table The results demonstrate that incorporating AeS reward focused solely on image aesthetics maintains strong performance and even improves the zero-shot FID (from 33.79 31.26). This improvement may stem from AcS's complementary effect, as emphasizes visual aesthetics differently from HPS and CLIP, potentially enhancing the regularization effect. These findings suggest that other rewards can indeed be effective to train our
Verify the Complementarity of Rewards. We can assess the complementarity of the rewards to some extent by examining their correlations on validation set. Specifically, we computed the Pearson correlation among the rewards used during training, utilizing the COCO5k dataset. The
24

---

HPS
Image Reward
Clip Score
HPS
0.51
0,46
Image Reward
0.51
0,41
Clip Score
0.46
0.41
Table 6: Pearson correlation between rewards computed on the COCO5k dataset
walks Tokye street filldd with
worn glawing neon and osimated city signage. She
clooe up phicto of womon Shx wore blue groy
weors a black toather jocket long red drees. boots,
blue and bland hair, and weard pair of carrings.
lipetick She woles
canti contidentry and casuaily.
The street donp and
reflactive, creating
mirror effect of lights. Many pedestmons
An extreme caste-up of groy- holred non with beard thought pondering history Poris, his of fecoism os he pits mostly motionieds. he is dhesseed in weol coct ouf coot with buttonn down shirt a brown very oppearance.
style snowooarder dir perforns sweetchirt rall,
welk
block pants, with orms outstrenched. The senone snowey landecape baskground dotted with trees. comp ements scone.
Figure 15: Samples generated by Reward-Instruct (4 NFE) with long and complex prompts. The Reward-Instruct here is post-trained from -03-medium.
JNT
2N
JM1
LUST
Fighreat
Figure 16: Path comparison between Reward-Instruct and Reward- Instructt. The prompt "Two dogs, best quality".
Figure 17: The effect of high- resolution guidance loss. Im- ages are from the same noise.
results in Table indiicate that correlations among the different rewards are positive and relatively strong.
C
More Applications for Reward-Instruce
C.1 Super-resolution
Tackling High-Resolution Generation. Existing reward functions for text-to-image generation are mostly trained on low-resolution inputs (e.g., 224x224 pixels). This creates a fundamental limitation: directly optimizing such rewards during high-resolution (e.g., 1024x1024 pixels) synthesis struggles to preserve fine-grained details. To address this, we propose training high-resolution classifier as a complementary guidance signal. This classifier explicitly prioritizes perceptual quality in high-resolution outputs.
25

---

Implicit High- Righ-Resolution Classifier.
We can form a high-resolution classifier via the Bayesian rule:
log p(Hgghies (x)) log p( Hoie[HHighResssHHgghhee
(C.1)
p(z; (LowRes)
where z; denotes the noisy samples timesteps Its gradient can be obtained as follows:
Va, log (esigh)Res((:)) Va, (oighs/ Vz, (o
(C.2)
The Va p HighRes) can be directly replaced by the original diffusion model, since it has the capability to generate high-resolution images. For obtaining V log log petp:///oooeree
finetune
the pre-trained diffusion over low-resolution data. By doing so, we can obtain a powerful Implicit classifier for high-resolution guidance. Fig. 17 shows that after applying the implicit high-resolution guidance proposed by us, the generated images are significantly clearer.
D Limitations
Our model, akin to most rext-to-image diffusion models, may not always perform perfectly regarding fairness and the accurate depiction of specific details. We intend to investigate these outstanding challenges within the generation field in our subsequent research. The goal of this future work will be to improve the model's ' proficiency in text generation, ensure fairer outcomes, and provide finer control over generated details.
E
Broader Impacts
This work introduces Reward-Instree reward-centric approach for developing efficient text-to- image models. From a positive perspective, although this an academic study, we believe that the proposed Seward-Instruce could be widely adopted in industry. Conversely, if these swift generation models are exploited by individuals with malicious intent, they could also streamline and expedite the production of detrimental content. While work centered scientific inquiry, committed to implementing measures to mitigate the spread of harmful information, such as by removing inappropriate content in the dataset.
F
Safeguards
The Reward-Instruct is trained on the prompt of JourneyDB dataset [37] which has undergone rigorous human and machine-based filtering ensure that there are harmful or violent prompts in the dataset.
26


Reviews:
[{"review_id": "fEzd7JzlUo", "reviewer": "Reviewer_Bbgh", "summary": "This paper proposes Reward-Instruct, a reward-centric training approach designed to efficiently convert pre-trained diffusion models into reward-aligned few-step generative models without relying on diffusion distillation losses. The key claim is that reward maximization alone, guided by a stochastic Langevin dynamics-inspired optimization, can effectively yield high-quality few-step generation. Extensive experiments demonstrate strong quantitative and qualitative performance on text-to-image benchmarks, showcasing superior visual quality compared to existing distillation-based methods.", "questions": "Since Reward-Instruct directly maximizes rewards over a few-step sampling process without explicit diffusion distillation, can the resulting outputs still be interpreted as valid diffusion score functions? Or should the model be viewed simply as a general epsilon-conditioned generator lacking traditional diffusion properties?\n\nWithout explicit diffusion distillation, how can RI theoretically or empirically guarantee meaningful preservation of the original pretrained diffusion model’s trajectories?\n\nWhy exactly does Reward-Instruct provide increased robustness against reward hacking compared to methods incorporating explicit diffusion distillation losses (e.g., DI++, RG-LCM)? Could similar robustness be achieved simply by applying RI-style reward maximization after completing standard diffusion distillation training?\n\nHow sensitive is RI to variations in reward scales or reward noise? Can RI reliably maintain stability and image quality if rewards become noisy or biased?\n\nCould authors provide experiments using Stable Diffusion XL (SDXL), currently a prevalent baseline, to fully understand RI’s practical advantages and limitations compared to state-of-the-art few-step diffusion baselines?", "limitations": "RI relies strictly on differentiable reward signals, potentially restricting applicability to black-box or human-derived feedback scenarios.\n\nThe method significantly deviates from traditional diffusion model inference procedures, compromising interpretability and the theoretical underpinnings of diffusion sampling.\n\nRI’s robustness against reward hacking is claimed without rigorous theoretical or empirical validation, limiting the trustworthiness of this key assertion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\n\\* Reward-Centric Simplicity: The proposed approach elegantly bypasses the computational complexity and implementation difficulties inherent in diffusion distillation.\n\n\\* Strong Empirical Results: Impressive visual quality and improved performance metrics (FID, HPS, CLIP scores) across various text-to-image tasks.\n\n\\* Robustness via Multiple Rewards: Incorporates multiple reward signals effectively to mitigate common reward hacking pitfalls, achieving better image quality and stability.\n\nMajor Weakness\n\n\\*Conceptual Ambiguity Regarding Diffusion Sampling:\nThe proposed training method updates parameters by directly maximizing rewards in a few-step generation process, significantly deviating from standard diffusion score matching procedures. While the model structure still resembles diffusion-based epsilon-prediction, the learned outputs may no longer represent valid diffusion score functions. Thus, it is debatable whether the final model should be categorized as a diffusion model or rather a general generator shaped merely in the mathematical form of an epsilon predictor. The paper fails to clearly acknowledge or analyze this conceptual discrepancy.\n\n\\*Lack of Justification Regarding Diffusion Trajectory Preservation:\nTraditional diffusion distillation methods explicitly attempt to preserve the trajectories defined by a pretrained diffusion model, enabling stable and theoretically grounded sampling dynamics. Reward-Instruct completely omits trajectory-level constraints, potentially collapsing diffusion trajectories into reward-driven but arbitrary generation paths. Without any formal analysis or justification, it remains unclear whether this approach maintains meaningful diffusion trajectories or merely exploits reward-driven shortcuts.\n\n\\*Insufficient Justification of Reward Hacking Robustness:\nThe paper claims RI’s robustness against reward hacking compared to diffusion-distillation-based methods like DI++ or RG-LCM. However, no clear theoretical explanation or thorough empirical demonstration is provided. Moreover, it is unclear why similar robustness could not be achieved simply by applying reward maximization after standard diffusion distillation.\n\n\\*Critical Omission of Relevant Recent Methods:\nSeveral closely related recent works addressing reward-guided few-step diffusion methods are notably missing:\n\"[ICLR 2025] Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization\"\n\"[CVPR 2025] Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewards\"\n\"[ICLR 2025 Oral] DSPO: Direct Score Preference Optimization for Diffusion Model Alignment (DPO-based)\"\n\nMinor Weakness\n\n\\*Inconsistent Naming Conventions:\nThe method is inconsistently named (Reward-Instruct, RI, RO), creating confusion (e.g., Figure 5). Clearly choose and consistently use one abbreviation throughout the paper.\n\n\\*Unclear CFG Loss Description:\nThe algorithm presented in the appendix includes a term named \"CFG Loss\" without any explicit description or justification within the main body, making the motivation and necessity unclear.\n\n\\*Missing Algorithm Description for Reward-Instruct+:\nDespite introducing Reward-Instruct+ (intermediate-step supervision), the paper lacks a clear, formal algorithmic presentation for this variant.\n\n\\*Lack of SDXL Comparison:\nWhile experiments are conducted on Stable Diffusion v1.5 and SD3-Medium, no comparisons or experiments with SDXL—currently a prevalent and practically important baseline—are provided. Additionally, SD3-Medium results are presented without comparison to existing methods, reducing interpretability and value.", "was_revised": true, "final_justification": "During rebuttal period, the author actively engage in discussion, and address most of my concerns. Therefore, I vote positive score.", "rebuttal": "We sincerely thank you for the valuable feedback. We address each comment below.\n\n> Since Reward-Instruct directly maximizes rewards over a few-step sampling process without explicit diffusion distillation, can the resulting outputs still be interpreted as valid diffusion score functions? Or should the model be viewed simply as a general epsilon-conditioned generator lacking traditional diffusion properties?\n\nThe generator trained by Reward-Instruct is no longer diffusion score functions. We note that almost all distilled diffusion models are also no longer diffusion score functions, since their objectives are not for learning scores.\n\n> Without explicit diffusion distillation, how can RI theoretically or empirically guarantee meaningful preservation of the original pretrained diffusion model’s trajectories?\n\nCurrently, there are usually two mainstream diffusion distillation approaches: one is distribution matching, which aims to align the student and teacher at the distribution level (such as SD-Turbo [48] and Diff-instruct [28]); the other is trajectory distillation, which aims to simulate the original diffusion model’s trajectories with fewer steps (such as Consistency Models [5] and CTM [21]). Only the latter aims to preserve the original diffusion model’s trajectories, while the former performs much better than the latter and is the most popular approach currently. Our method, different from traditional diffusion distillation approaches, is a novel reward-centric approach. Note that, similar to distribution matching, our approach achieves strong performance without preserving the original diffusion model’s trajectories.\n\nIt is also worth emphasizing that there has recently been a paradigm shift from supervised training (data-centric) to RL training (reward-centric). In the LLM field, the community is actively exploring reward-centric post-training approaches that can demonstrate powerful emergent capabilities, e.g., the GRPO adopted in DeepSeek R1. **Our work represents a systematic study of the first reward-centric approaches to post-train pre-trained diffusion models into powerful few-step generators.** By our reward-centric RI, a simple yet elegant approach that doesn't rely on traditional diffusion distillation techniques, can achieve surprisingly good few-step generation performance.  \nWe hope our investigation is helpful for the community to make further explorations along this novel path. \n\n> Why exactly does Reward-Instruct provide increased robustness against reward hacking compared to methods incorporating explicit diffusion distillation losses (e.g., DI++, RG-LCM)? Could similar robustness be achieved simply by applying RI-style reward maximization after completing standard diffusion distillation training?\n\nOur work aims to explore a reward-centric approach to achieve fast generation without the need for tricky yet expensive diffusion distillation. To achieve this, we investigate several simply yet effective regularizations to alleviate reward hacking, which could also potentially benefit the distillation-based reward maximization for better performance. \nIt is our experimental observation that our RI delivers surprisingly robust performance with simple regularization. \nHowever as you suggested, we agree that it is exciting to further explore the theoretical properties of regularizations introduced with RI in the future.\n\nInterestingly, we can explore combining reward-centric RI training with traditional distillation in our future work to produce more powerful models.\n\nStarting from the distilled model is a much stronger initialization compared to pre-trained diffusion models. We can directly initialize from the distilled model, and replace the prior with the weight of the distilled model to construct RI for fine-tuning the distilled model. We show results below:\n\n| Model                  | Backbone | NFE  | HPS↑  | Aes↑ | CS↑   | FID↓  | Image Reward↑ |\n| ---------------------- | -------- | ---- | ----- | ---- | ----- | ----- | ------------- |\n| Base Model             | SD-v1.5  | 50   | 30.19 | 5.87 | 34.28 | 29.11 | 0.81          |\n| RI, init w/ LCM (Ours) | SD-v1.5  | 4    | 33.62 | 6.15 | 33.58 | 32.05 | 1.19          |\n| RI (Ours)              | SD-v1.5  | 4    | 33.70 | 6.11 | 32.13 | 33.79 | 1.22          |\n\nIt can be seen that RI can effectively fine-tune the distilled models with better performance compared to initializing from pre-trained models.\n\n> Several closely related recent works addressing reward-guided few-step diffusion methods are missing: PSO [a], LaSPO [b], DSPO [c].\n\nThank you for bringing our attention to these interesting works. We discuss the mentioned works below:\n\n- **DSPO, PSO, and LaSPO focus on essentially different tasks compared to us**. DSPO focus on fine-tuning pre-trained diffusion models, which is not for constructing few-step models. PSO and LaSRO aim to fine-tune the well-trained few-step models, which is a different and much easier setting compared to our method and our baselines. **The notable contribution and finding of our work is that: we show that reward-centric approach can fine-tuning form pre-trained diffusion models to few-step generator.**\n\n- **PSO and DSPO belong to DPO-like methods, which are essentially different from RL methods** that aim to maximize a certain reward. In their papers [a, c], there is no comparison between DPO-like methods and RL-based methods.\n\n- However, we are also glad to provide a comparison below:\n\n| Model                  | Backbone | NFE  | HPS↑  | Aes↑ | CS↑   | FID↓  | Image Reward↑ |\n| ---------------------- | -------- | ---- | ----- | ---- | ----- | ----- | ------------- |\n| Base Model             | SD-v1.5  | 50   | 30.19 | 5.87 | 34.28 | 29.11 | 0.81    |\n| DSPO                   | SD-v1.5  | 50   | 30.76 | 5.96 | 33.81 | 30.86 | 0.92   |\n| PSO, init w/ LCM       | SD-v1.5  | 4    | 29.24 | 5.99 | 32.03 | 33.24 | 0.85   |\n| LaSPO, init w/ LCM     | SD-v1.5  | 4    | 33.16 | 6.13 | 30.83 | 34.73 | 0.94   |\n| RI, init w/ LCM (Ours) | SD-v1.5  | 4    | 33.62 | 6.15 | 33.58 | 32.05 | 1.19|\n| RI (Ours)              | SD-v1.5  | 4    | 33.70 | 6.11 | 32.13 | 33.79 | 1.22|\n\nIt can be seen that applying reward-instruct for post-training distill model (LCM) is more effective than starting from pre-trained diffusion models, while reward-instruct init w/ LCM outperforms the prior works in fine-tuning LCM regarding both reward metrics and FID.\n\n[a] Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization\n\n[b] Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewards\n\n[c] DSPO: Direct Score Preference Optimization for Diffusion Model Alignment\n\n> How sensitive is RI to variations in reward scales or reward noise? Can RI reliably maintain stability and image quality if rewards become noisy or biased?\n\nFor reward scales, we have applied gradient normalization to balance different rewards, which effectively balances contributions from different rewards regardless of their individual scales. Hence, our method is robust to reward scales. We have verified the effectiveness of gradient normalization in Figure 9. \n\nIn the case of biased rewards, we expect RI with multiple rewards to be also relatively robust. \nOn one hand, if only one reward is biased, its negative effect will not be significant since the supervision is an average of all rewards. \nOn the other hand, even if all rewards are biased, we believe that as long as the biases are not consistent (e.g., biases are random with zero mean), their average effect should be neutral. \n\nIt would be an interesting future work to conduct exact experiments about this matter. However, such an investigation is difficult to conduct in a rigorous manner since, in the current community, it is difficult to obtain a large pool of rewards with different sources of quantifiable biases. \n\n> Inconsistent Naming Conventions\n\nRI is short for Reward-Instruct, and R0 is a typo. Sorry for the confusion. \nWe will clean up these inconsistent abbreviations and use RI as our abbreviation.\n\n> Unclear CFG Loss Description\n\nIt has been pointed out in [27] that CFG can be viewed as an implicit reward function. Using it in our RI incurs a slight difference compared with other rewards, since it does not have an explicit value. \n\nHowever, we can compute its gradient. Hence, the CFG loss is a surrogate loss that has the same gradient of $-E_{x_t,t}\\log p_\\psi(\\mathrm{text}|x_t) = -E_{x_t,t}\\log \\frac{p_\\psi(x_t|\\mathrm{text})}{p_\\psi(x_t|\\emptyset)}$. We will add a detailed description in our revision.\n\n> Missing Algorithm Description for Reward-Instruct+\n\nThank you for your suggestion. \nWe have detailed our RI+ in Section 3.4, but didn't summarize it into an algorithm.\nWe will add an algorithm description for Reward-Insturct+ in revision.\n\n> SD3-Medium results are presented without comparison to existing methods, reducing interpretability and value.\n\nWe note that the multi-step SD3-Medium itself is a strong baseline.\nThe comparison between multi-step SD3-Medium and 4-step RI shows that **RI, as a reward-centric approach, is capable of post-training powerful flow-based models to few-step generators with better reward metrics and comparable zero-shot FID**. This highlights RI's effectiveness and its wide application.\n\n> Lack of SDXL Comparison.\n\nWe compare to the base model (SDXL) and distillation-based RL methods, i.e., RG-LCM and DI++. \n\n| Model | NFE | HPS↑ | Aes↑ | CS↑ | FID↓ | Image Reward↑ |\n|---------|-----|--------|--------|--------|---------|----------------|\n| Base | 50 | 33.62 | 6.24 | 36.15 | 28.61 | 1.12 |\n| RG-LCM | 4 | 32.5 | 6.08 | 32.96 | 36.13 | 1.27 |\n| DI++ | 4 | 33.96 | 6.13 | 32.75 | 34.02 | 1.35 |\n| RI (Ours) | 4 | 34.83 | 6.30 | 34.50 | 32.45 | 1.40 |\n\nThe results shown above show that the proposed Reward-centric approach - Reward-instruct also has advantages over prior distillation-based RL methods on the SDXL backbone."}, {"review_id": "gtRbv3FwAS", "reviewer": "Reviewer_qewY", "summary": "This paper introduces Reward-Instruct, a novel framework for creating high-quality, few-step image generators w/o complex diffusion distillation losses. The core idea is to reframe the task as regularized reward maximization, where a few-step generator's parameters are directly optimized to satisfy reward functions. To prevent \"reward hacking,\" the method employs simple regularization techniques utilizing the models base parameters to approximate the base image distribution.\nThe authors investigate how to balance different reward functions and their impact on the downstream model. The experimental evaluation demonstrates that this distillation-free approach achieves state-of-the-art performance, surpassing previous methods in visual quality and quantitative metrics", "questions": "- Could you elaborate further on the intuition behind $\\hat{\\epsilon}_\\eta$ (L146)? Is this routed in prior literature on DM sampling? \n- What is the importance of randomly sampling $\\eta$ at different diffusion steps? Could you provide further analysis of the impact of that design choice?", "limitations": "Despite the authors claim in the NeurIPS checklist the paper does not discuss the limitations of the proposed method, nor do the authors provide insights on its failure cases.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strength \n\n- Unique and novel perspective on reward modeling and DM distillation makes for a strong methodological contribution \n- Rigorous methodological section with strong motivation and good mathematical background \n- Interesting insights on reward hacking, as well as  balancing different reward function and their influence \n- Experimental demonstration of effectiveness on 2 different DM paradigms: SD 1.5 for U-Net backbone and noise-estimate score function + SD3.5 for DIT backbone w/ rectified flow \n- Qualitative and empirical performance improvements look solid \n- Additional improvements in computational requirements \n\n## Weaknesses \nThere remain some minor weaknesses that could be improved \n\n- Many of the provided insights are only underpinned by a handful of cherry-picked examples (e.g. Fig. 4, Fig. 6, Fig. 7, Fig. 8, Fig. 11). The paper would benefit from corroborating these findings with quantitative experiments\n- Use of LAION Aesthetics classifier v1 instead of v2, despite the improved version being readily available \n- Inferring improved convergence from Fig. 10 (RewardInstruct+) is a bit of stretch. No improvement for HPS for example \n- The presentation can be improved. Especially the sudden introduction of RewardInstruct+ over more substantive experiments may confuse some readers \n- The paper would immensely benefit from a human user study on actual improvements in quality, since automated metrics are known to be faulty (and as discussed in the paper there may be reward hacking)\n\n-", "was_revised": true, "final_justification": "Taking into consideration the rebuttal response and author reviewer feedback I retain my positive rating.", "rebuttal": "We sincerely thank you for the valuable feedback. We address each comment below.\n\n> Many of the provided insights are only underpinned by a handful of cherry-picked examples (e.g. Fig. 4, Fig. 6, Fig. 7, Fig. 8, Fig. 11). The paper would benefit from corroborating these findings with quantitative experiments\n\nWe provide quantitative results about removing one technique at a time to show the effectiveness. \n\n| Model                        | Backbone | Steps | HPS↑  | Aes↑ | CS↑   | FID↓  | Image Reward↑ |\n| ---------------------------- | -------- | ----- | ----- | ---- | ----- | ----- | ------------- |\n| RI (Ours)                    | SD-v1.5  | 4     | 33.70 | 6.11 | 32.13 | 33.79 | 1.22          |\n| RI w/ single reward          | SD-v1.5  | 4     | 32.08 | 5.80 | 31.01 | 38.21 | 0.89          |\n| RI w/o random-η-sampling     | SD-v1.5  | 4     | 33.41 | 6.12 | 32.11 | 36.90 | 1.18          |\n| RI w/o weight regularization | SD-v1.5  | 4     | 34.26 | 6.12 | 32.45 | 39.25 | 1.27          |\n\nThe results, presented in above, demonstrate the critical role of each component:\n\n1) Multiple rewards: Improves both rewards and FID, mitigating artifacts and reward hacking.\n2) Random-$\\eta$-sampling: Maintains similar reward performance but significantly improves FID, aiding to find better mode with fewer artifacts.\n3) Weight regularization: Trades slight reward gains for better FID, ensuring the generator stays within the image manifold.\n\n> Use of LAION Aesthetics classifier v1 instead of v2, despite the improved version being readily available\n\nWe clarify that AeS v2 is used in our experiments.\n\n> Inferring improved convergence from Fig. 10 (Reward Instruct+) is a bit of stretch. No improvement for HPS for example\n\nReward-instruct only provides supervision at the endpoint and requires backpropagation to flow through the entire generator (multiple denoising steps), while Reward-instruct+ provides signal supervision during the process, which intuitively enables faster convergence and higher training efficiency, as we only allow RI+'s gradients to backpropagate one step rather than through the entire generator. Reward-instruct+ can converge faster than Reward-instruct regarding most rewards. Although it doesn't show significant gains on HPS, RI+'s training is cheaper (each iteration requires only 65\\% of Reward-instruct's time), which makes it still have clear advantages over Reward-instruct overall. Exploring how to make RI+ have more consistent gains would be an interesting future direction.\n\n> The presentation can be improved. Especially the sudden introduction of Reward Instruct+ over more substantive experiments may confuse some readers\n\nThank you for your valuable comments. We will add more motivating analysis and illustrations for the introduction of Reward-Instruct+ in our revision. \n\n> The paper would immensely benefit from a human user study on actual improvements in quality, since automated metrics are known to be faulty (and as discussed in the paper, there may be reward hacking)\n\nGreat suggestion. We provide a user study to compare the base model and our reward-instruct. The user study is conducted by presenting users with two anonymous images generated by different models and asking them to select the sample with higher image quality and better prompt alignment. We randomly selected 20 prompts for image generation. Due to the limited time in the rebuttal phase, we collected approximately 20 user responses on 20 pairs in total.  \n\n| Model      | Backbone | NFE  | User Preference↑ |\n| ---------- | -------- | ---- | ---------------- |\n| Base Model | SD-v1.5  | 50   | 41.7%            |\n| RI (Ours)  | SD-v1.5  | 4    | 58.3%            |\n\nResults in above shows that the proposed RI achieves better user preference compared to the base models, indicating the effectiveness of the proposed reward-centric approach in aligning human preference. \n\n> Could you elaborate further on the intuition $\\epsilon_\\eta$ behind (L146)? Is this routed in prior literature on DM sampling?\n\nThis refers to an interpolation between the Consistency Model (CM) sampler and the DDIM sampler. The former is a purely stochastic sampler ($\\eta=0$), while the latter is a deterministic sampler ($\\eta$ = 1). There is a notable difference in both the sampling paths and the style of the resulting samples generated by these two distinct methods. Our experiments in Figure 4 further show that different $\\eta$ leads to different styles.\nThis discrepancy drives the strategy of interpolating between them by randomizing the eta parameter during sampling.\n\n> What is the importance of randomly sampling $\\eta$ at different diffusion steps? Could you provide further analysis of the impact of that design choice?\n\nRandomly sampling $\\eta$ at different diffusion steps can augment the generator's output distribution and broaden the scope of its exploration, since different $\\eta$ deliver different sampling trajectories and styles. Consequently, we believe this can force the generator to learn a more robust policy by maximizing the rewards across these varied sampling paths, which ultimately improves its robustness.\n\nIf we fixed $\\eta$ in sampling during training, the generator only needs to learn to maximize the reward under a specific sampling trajectory and style, which makes it easier for the generator to hack the reward.\n\n| Model         | Backbone | Steps | HPS↑  | Aes↑ | CS↑   | FID↓  | Image Reward↑ |\n| ------------- | -------- | ----- | ----- | ---- | ----- | ----- | ------------- |\n| RI (Ours)     | SD-v1.5  | 4     | 33.70 | 6.11 | 32.13 | 33.79 | 1.22          |\n| RI w/ $\\eta$ = 0   | SD-v1.5  | 4     | 33.29 | 6.05 | 31.50 | 37.57 | 1.09          |\n| RI w/ $\\eta$ = 0.7 | SD-v1.5  | 4     | 33.74 | 6.19 | 32.30 | 36.09 | 1.10          |\n| RI w/ $\\eta$ = 1   | SD-v1.5  | 4     | 33.41 | 6.12 | 32.11 | 36.90 | 1.18          |\n\nThe results above show that fixed $\\eta$ leads to inferior results in both FID and reward metrics."}, {"review_id": "DdTIVVdRp2", "reviewer": "Reviewer_JABx", "summary": "This paper introduces a finetuning method that enables few-shot, preference-aligned text-to-image generation using an off-the-shelf pretrained diffusion model. The authors critically examine the importance and difficulties of reward maximization in the setting of diffusion models and alignment to human preferences, and they develop a novel yet simple objective to enable alignment and few-shot sampling. Reward-Instruct shows strong results in effectively maximizing human preference, per a set of image-based reward models, however the generalization capabilities of the final model, i.e. alignment w.r.t to reward models not seen in training, remains unclear.", "questions": "- Could the authors please provide results that measure preference alignment using a reward model that was not used during Reward-Instruct’s training? I acknowledge the Aesthetics results but the Aesthetics model is biased towards simple image aesthetics and may not capture much of the intricacies of preference alignment in the context of text-to-image generation. I think using Pick-a-Pic reward model could be a good experiment. \n\n- How does Reward-Instruct compare with Diffusion-DPO and/or KTO SD1.5 or other offline approaches?\n\n- Could the authors please elaborate on the design decisions when selecting reward models? The justification for reward ensembling highlights that maximizing the reward based on shared data modes across different reward models can help avoid reward hacking. In the paper however, the reward models used are somewhat similar in that they are optimized for some combination of image aesthetics and/or image-text alignment. Does this mean the reward models must be “sufficiently similar” in terms of what they reward? As a hypothetical example, if you included a safety classifier as one of the reward models would the training still be stable and, most importantly, effective?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Strong results w.r.t to few-step baselines and 50-step baselines\n\n- Principled formulation shows how to enable reward maximization and even shows how to extend guidance into intermediate denoising steps. Also demonstrates the benefit of reward ensembling.\n\n- Show extensions to image editing and super-resolution.\n\n- Paper is well written and motivations are clearly expressed. \n\n\n**Weaknesses**\n\n- Generalization of the final model is unclear. The authors seem to maximize the combination of HPS, ImageReward, and CLIP scores (I would appreciate it if the authors could confirm this and also clarify which HPS reward model in particular). However, their evaluations measure the scores from these same reward models plus the LAION Aesthetics model. Thus, it’s unclear if the preference alignment generalizes outside of these reward models– e.g. how do the results compare if measuring Pick-a-Pic reward model score, or a different version of HPS rm, or ideally if using human annotators?\n\n- Comparisons to reward-centric approaches. Many of the reported baselines, e.g. ReFL, optimize their model w.r.t to one reward signal whereas the benefit of ensembling rewards is a key finding of this paper. I understand that re-training such baselines may be infeasible, however I think it should be noted that the method benefits from multiple signals whereas many other baselines do not, when in fact their performance may improve with multiple signals too. \n\n- Comparisons to offline approaches. Given the prevalence of offline approaches like Diffusion-DPO, how does Reward-Instruct compare to SD-1.5 aligned via Diffusion-DPO or Diffusion-KTO?", "was_revised": true, "final_justification": "The paper presents an effective method that combines preference alignment and few-shot sampling. The results in the paper and additional results presented in the rebuttal validate the significance of these results. I believe the novelty, applicability, and effectiveness of this method merits acceptance.", "rebuttal": "We sincerely thank you for the valuable feedback. We address each comment below.\n\n> Generalization of the final model is unclear. The authors seem to maximize the combination of HPS, ImageReward, and CLIP scores (I would appreciate it if the authors could confirm this and also clarify which HPS reward model in particular). However, their evaluations measure the scores from these same reward models plus the LAION Aesthetics model. Thus, it’s unclear if the preference alignment generalizes outside of these reward models– e.g., how do the results compare if measuring Pick-a-Pic reward model score, or a different version of HPS rm, or ideally if using human annotators?\n\nGreat suggestion. We maximize HPS V2.1, ImageReward, and Clip Scores. To fully confirm the effectiveness of our Reward-Instruct in generating high-quality samples, we conduct a user study between the base model (50 NFE) and Reward-Instruct (4 NFE). The user study is conducted by presenting users with two anonymous images generated by different models and asking them to select the sample with higher image quality and better prompt alignment. We randomly selected 20 prompts for image generation. In total, we collected approximately 20 user responses on 20 pairs.\n\n| Model      | Backbone | NFE  | HPS v2.0↑ | Pick Score↑ | User Preference↑ |\n| ---------- | -------- | ---- | --------- | ----------- | ---------------- |\n| Base Model | SD-v1.5  | 50   | 27.22     | 22.19       | 41.7%            |\n| RI (Ours)  | SD-v1.5  | 4    | 28.83     | 22.40       | 58.3%            |\n\nIt can be seen that Reward-Instruct is able to generalize to unseen rewards, and also achieves better performance regarding user preference compared to the pre-trained base models.\n\n>Comparisons to reward-centric approaches. Many of the reported baselines, e.g. ReFL, optimize their model w.r.t to one reward signal whereas the benefit of ensembling rewards is a key finding of this paper. I understand that re-training such baselines may be infeasible, however I think it should be noted that the method benefits from multiple signals whereas many other baselines do not, when in fact their performance may improve with multiple signals too.\n    \nGreat suggestion. This is a very interesting direction. Indeed, some other RL methods that integrate multiple rewards could also potentially further improve performance. We will discuss this point in the revision and further explore it in future work. Nevertheless, it is worth noting that our reward-centric approach directly produces few-step samplers, without the need for diffusion distillation. In this sense, ReFL (standard diffusion sampling) is not a direct baseline in our setting.\n\n\n\n> Comparisons to offline approaches. Given the prevalence of offline approaches like Diffusion-DPO, how does Reward-Instruct compare to SD-1.5 aligned via Diffusion-DPO or Diffusion-KTO?\n\nWe note that the DPO-like methods belong to another kind of method. We are also glad to provide the comparison between DPO and our Reward-Instruct below:\n\n| Model               | Backbone | NFE  | HPS↑  | Aes↑ | CS↑   | FID↓  | Image Reward↑ |\n| ------------------- | -------- | ---- | ----- | ---- | ----- | ----- | ------------- |\n| Base Model          | SD-v1.5  | 50   | 30.19 | 5.87 | 34.28 | 29.11 | 0.81          |\n| DPO                 | SD-v1.5  | 50   | 30.79 | 5.93 | 33.84 | 31.59 | 0.90          |\n| **Reward-centric:** |          |      |       |      |       |       |               |\n| ReFL                | SD-v1.5  | 50   | 31.82 | 5.97 | 31.78 | 39.38 | 1.16          |\n| RI (Ours)           | SD-v1.5  | 4    | 33.70 | 6.11 | 32.13 | 33.79 | 1.22          |\n\nWe find that DPO typically aligns less with human preference compared to reward-centric approaches, but performs better regarding the FID. We believe that a possible reason is that DPO training still relies on curated data, which benefits the FID. However, it is clear that RI is not in conflict with DPO in that one can readily use a DPO-aligned diffusion model as a base model and apply RI after it to get the advantage of both. However, in this paper, we evaluate RI without the influence of DPO to clearly study its pros and cons.\n\n> Could the authors please elaborate on the design decisions when selecting reward models? The justification for reward ensembling highlights that maximizing the reward based on shared data modes across different reward models can help avoid reward hacking. In the paper however, the reward models used are somewhat similar in that they are optimized for some combination of image aesthetics and/or image-text alignment. Does this mean the reward models must be “sufficiently similar” in terms of what they reward? As a hypothetical example, if you included a safety classifier as one of the reward models would the training still be stable and, most importantly, effective?\n\nWe believe that rewards do not need to be sufficiently similar to each other, and that the model may benefit from combinations of different types of rewards. In Table 3 of the Appendix, we maximize HPS, CLIP scores, and AeS, where each reward represents a different type. In particular,  HPS represents human preference, CLIP score represents image-text alignment, and AeS represents pure aesthetics. The results in Table 3 demonstrate that this combination can achieve similar rewards as well as notably better zero-shot FID, suggesting better avoidance of reward hacking. To further verify this, we took your valuable suggestion and incorporated a \"safety classifier\" into our training. \n\n| Model                               | Backbone | Steps | HPS↑  | Aes↑ | CS↑   | FID↓  | Image Reward↑ |\n| ----------------------------------- | -------- | ----- | ----- | ---- | ----- | ----- | ------------- |\n| RI w/ HPS + Clip Score + AeS (Ours) | SD-v1.5  | 4     | 32.35 | 6.21 | 32.43 | 31.26 | 1.01          |\n| RI, w/ safety checker               | SD-v1.5  | 4     | 33.62 | 6.08 | 32.47 | 33.51 | 1.18          |\n| RI (Ours)                           | SD-v1.5  | 4     | 33.70 | 6.11 | 32.13 | 33.79 | 1.22          |\n\nResults in above show that after integrating the safety classifier into the training, we can still achieve comparable performance and slightly better zero-shot FID."}, {"review_id": "zfReklGo6B", "reviewer": "Reviewer_oEgS", "summary": "The paper investigates the effect of rewards for fast image generation using pre-trained diffusion models. \nThe authors reformulate the problem of conditional generation with rewards, focusing on the reward and searching for samples that maximize it, while using the base generative model as a regularizer.\nAdditionally, they experiment with various reward functions and show that using a weighted normalized average of multiple rewards yields the best results.", "questions": "1. What are the results of Reward-Instruct over more complex prompts such as the ones presented in Figure 2?\n2. “Figure 6: The prior distillation-based reward maximization methods collapse when the reward is chosen to be HPS v2.1.” – what happens for prior approaches vs. Reward-Instruct when not using HPS v2.1? (e.g. using HPS v2.0 [60] or Image Reward [61] as in their original approach?)\n3. Figure 7 – there is a mild improvement when increasing the number of rewards used, but it seems more like enhancing the gradient step because the improvement is generally toward the same direction. My question is – what happens if you use a higher weight for one of the regularization methods instead of using the weighted average? \n4. In addition to text proofreading that is needed, in some figures it is not clear what are the prompts for the examples, e.g. Figure 8, or it looks like the wrong prompt/images, e.g. in Figure 7 the \"squirrel\" example which shows bird images. These should be fixed.", "limitations": "I think the paper could benefit from a broader discussion of the method's limitations, including visual examples of them.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Clear background and motivation.\n* The authors offer an interesting re-formulation of image generation formula post pre-training by placing the reward in the center.\n* The evaluation is thorough and includes ablations and comparisons with relevant previous works.\n\nWeaknesses:\n* The paper needs proofreading as it contains many typos and grammatical issues, for example: \n“Our journey starts with an analysis in Section 3 shows that” (line 57), “our methods directly operates..” (line 66), “advanced prior distribution will definitely leads to..” (line 166), “converts pretrained pre-trained base..” (line 180), and more..\n* The limitations section is quite limited, I think the paper could benefit from further limitations discussion including images of failure cases.\n* There aren't many visual examples, and there aren't complex visual examples of the method as the ones presented in Figure 2 of limitations of the competing method.", "was_revised": true, "final_justification": "The authors addressed most of my concerns by providing clarifications and additional experiments.\nAssuming the authors will proofread the paper and include more complex prompts comparisons with competing methods, I lean toward acceptance.", "rebuttal": "We sincerely thank you for the valuable feedback. We address each comment below.\n\n> The paper needs proofreading as it contains many typos and grammatical issues\n\nThank you for your valuable comments. We will revise those typos and grammatical issues in the revision.\n\n> The limitations section is quite limited, I think the paper could benefit from further limitations discussion including images of failure cases.\n\nThank you for your great suggestion. We will discuss our limitations in greater detail in our revision. Specifically, we will add some failure cases and corresponding analysis.\n\n> What are the results of Reward-Instruct over more complex prompts such as the ones presented in Figure 2?\n\nOur model also performs well on complex prompts. For example, the prompts used in Figure 1 have some complex cases: \"A alpaca made of colorful building blocks, realism\", \"A professional portrait photo of an anthropomorphic capybara wearing fancy gentleman hat and jacket walking in autumn forest\". We will add more samples on complex prompts in revision, but we cannot provide it in the rebuttal stage due to the policy.\n\n> “Figure 6: The prior distillation-based reward maximization methods collapse when the reward is chosen to be HPS v2.1.” – what happens for prior approaches vs. Reward-Instruct when not using HPS v2.1? (e.g. using HPS v2.0 [60] or Image Reward [61] as in their original approach?)\n\nGreat question. The results in Figure 5 and Table 1 for RG-LCM and DI++ are obtained by using the reward in their original approach. It can be seen that our proposed Reward-Instruct achieves better performance.\n\n> Figure 7 – there is a mild improvement when increasing the number of rewards used, but it seems more like enhancing the gradient step because the improvement is generally toward the same direction. My question is – what happens if you use a higher weight for one of the regularization methods instead of using the weighted average?\n\nWe argue that the performance improvement brought by multiple rewards is not merely due to the enhancement of gradient step. We presented the results using weak regularization in Figure 8 (small weight regularization coefficient and without using random eta sampling), *equivalent to a higher weight of rewards*. From Figure 8, it can be observed that using a single reward leads to severe artifacts, while ensembling through multiple rewards can generate reasonably high-quality images.\n\nBesides, we conduct an extra study on using a large weight regularization coefficient (5 times larger). The results are shown below:\n\n| Model                      | Backbone | Steps | HPS↑  | Aes↑ | CS↑   | FID↓  | Image Reward↑ |\n| -------------------------- | -------- | ----- | ----- | ---- | ----- | ----- | ------------- |\n| R0 w/ single reward        | SD-v1.5  | 4     | 32.08 | 5.80 | 31.01 | 38.21 | 0.89          |\n| R0 w/ large regularization | SD-v1.5  | 4     | 32.86 | 5.98 | 31.50 | 32.86 | 0.95          |\n| R0 (Ours)                  | SD-v1.5  | 4     | 33.70 | 6.11 | 32.13 | 33.79 | 1.22          |\n\nIt can be seen that with large regularization and multiple rewards, the performance on rewards is degraded while the zero-shot FID becomes better. And its performance is way better than using only single reward.\n\n> In addition to text proofreading that is needed, in some figures it is not clear what are the prompts for the examples, e.g. Figure 8, or it looks like the wrong prompt/images, e.g. in Figure 7 the \"squirrel\" example which shows bird images. These should be fixed.\n\nThank you for your effort! We will add the prompt descriptions in Figure 8 and will revise the wrong prompts (which should be \"bird\") in Figure 7. We will also carefully check typos and improve the clarity of the presented figures."}]

Author rebuttal:

Meta-review: The reviewers engaged in extensive discussion with the authors. Following the rebuttal, all reviewers reached a consensus in favor of acceptance. The AC finds no major outstanding concerns, aside from improving the writing and clarifying certain points in revision. Therefore, the AC agrees with the reviewers and recommends acceptance.
Decision: Accept (poster)