DiffusionNFT: Online Diffusion Reinforcement with Forward Process
Authors: Kaiwen Zheng
Huayu Chen
Haotian Ye
Haoxiang Wang
Qinsheng Zhang
Kai Jiang
Hang Su
Stefano Ermon
Jun Zhu
Ming-Yu Liu
Venue: ICLR 2026 Oral (2026)
Keywords: Diffusion Models
Reinforcement Learning
Flow Matching
Primary area: generative models
TL;DR: We propose a new online reinforcement learning (RL) algorithm for diffusion and flow models based on forward process.

Abstract: Online reinforcement learning (RL) has been central to post-training language models, but its extension to diffusion models remains challenging due to intractable likelihoods. Recent works discretize the reverse sampling process to enable GRPO-style training, yet they inherit fundamental drawbacks, including solver restrictions, forward–reverse inconsistency, and complicated integration with classifier-free guidance (CFG). We introduce Diffusion Negative-aware FineTuning (DiffusionNFT), a new online RL paradigm that optimizes diffusion models directly on the forward process via flow matching. DiffusionNFT contrasts positive and negative generations to define an implicit policy improvement direction, naturally incorporating reinforcement signals into the supervised learning objective. 
This formulation enables training with arbitrary black-box solvers, eliminates the need for likelihood estimation, and requires only clean images rather than sampling trajectories for policy optimization. DiffusionNFT is up to $25\times$ more efficient than FlowGRPO in head-to-head comparisons, while being CFG-free. For instance, DiffusionNFT improves the GenEval score from 0.24 to 0.98 within 1k steps, while FlowGRPO achieves 0.95 with over 5k steps and additional CFG employment. By leveraging multiple reward models, DiffusionNFT significantly boosts the performance of SD3.5-Medium in every benchmark tested.

Published as a conference paper at ICLR 2026
DIFFUSIONNFT: ONLINE DIFFUSION REINFORCE- MENT WITH FORWARD PROCESS
Kaiwen Zheng 1.2
Huayu Chen 1,,,
Haotian Ye
Haoxiang Wang?
Qinsheng Zhang?
Kai Jiang'
Hang Su'
Stefano Ermon
Jun Zhu lif
Ming- Yu Liu?
1 Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch MI Center, AI Institute, Tsinghua
2NVIDIA *Stanford University
https://research.nn//
ABSTRACT
Online reinforcement learning (RL) has been central to post-training language models, but its extension to diffusion models remains challenging due in- tractable likelihoods. Recent works discretize the reverse sampling process enable GRPO-style training, yet they inherit fundamental drawbacks, including solver restrictions, forward-reverse inconsistency, and complicated integration with classifier free guidance (CFG). We introduce Diffusion Negative-awart Fine- Tuning (DiffusionNFTT), new online RL paradigm that optimizes diffusion mod- els directly on the forward process via flow matching. DiffusionNFT contrasts positive and negative generations to define an implicit policy improvement direc- tion, naturally incorporating reinforcement signals into the supervised learning ob- jective. This formulation enables training with arbitrary black-bor solvers, elim- inates the need for likelihood estimation, and requires only clean images rather than sampling trajectories for policy optimization. DiffusionNFT is up 25 more efficient than FlowGRPO in head-to-head comparisons, while being CFG- free. For instance, DiffusionNFT improves the GenEval score from 0.24 to 0.98 within Ik steps, while FlowGRPO achieves 0.95 with over 5k steps and additional CFG employment. By leveraging multiple reward models, DiffusionNFT signifi- cantly boosts the performance of 3D3.5-Medium in every benchmark tested.
DiffusioNNFT
S-M
0.95
25c Ericency
(0.95)
Flow-GRPG
Clipsoree
5035. CHOI
FL001 Dov
OCR
GenEval Score
0.80
0.75
0.70
HPSV2.11
Genniivl
SD3.5-M (0.63)
0.60
CFG (0-24)
0.20
Acathutic
Unffiekkoward
1000
1500
2500
Training
(a)
(b)
Figure 1: Performance DiffusionNFT.. (a) Head-to-head comparison with FlowGRPO on the GenEval task. (b) By employing multiple reward models, DiffusionNFT significantly boosts the performance of SD3.5-Medium in every benchmark tested, while being fully CFG-free.
INTRODUCTION
Online Reinforcement Learning (RL) has been pivotal in the post-training of LLMs, driving recent advances in LLMs' alignment and reasoning abilities (Achiam et al.) 2023] Guo et 2025 How-
Equull Contribution
Corresponding Author

---

Published as a conference paper at ICLR 2026
Discretized Reverse SDE Process
To (2 x oold (xp1x)+1) r(x))
XT w
X2
X1
Xo
+ *GROO/PO
stored
prompt
((xlo
To
+v
vo((e)
(x)d(x)) NFT r(x))
Black-box Solver
x0
X
V
f ((x))
Forward Process
Comparison between Forward Process RL (NFT) and Reverse-Procee RL (GRPO). NFT allows using any solvers and does not require storing the whole sampling trajectory for optimization.
Figure 2:
ever, replicating similar success for diffusion models in visual generation is not straightforward. Policy Gradient algorithms assume that model likelihoods are exactly computable. This assumption holds for autoregressive models, but is inherently violated by diffusion models, where likelihoods can only be approximated via costly probabilistic ODE variational bounds of SDE (Song al.) 2021). Recent works circumvent this barrier by discretizing the reverse sampling process, refram- ing diffusion generation
multi-step decision-making problem (Black et al. (002)). This makes transitions between adjacent steps tractable Gaussians, enabling direct application of existing RL
algorithms like GRPO to the diffusion domain (Xue al. 2025 Liu al. 2025).
Despite promising efforts made, we argue that GRPO-style diffusion reinforcement still faces fun- damental limitations: (1) Forward inconsistency. Focusing solely on the reverse sampling process breaks adherence to the forward diffusion process, risking the model degenerating into cascaded Gaussians. (2) Solver restriction. The data collection process relies on first-order SDE samplers, precluding the full utilization of ODE or high-order solvers that are default to flow models and ad- vantageous for generation efficiency. (3) Complicated CFG integration. Diffusion models heavily rely on Classifier-Free Guidance (CFG) (Ho & Salimans, 2022), which requires training both con- ditional and unconditional models. Current RL practices typically incorporate CFG in post- training, leading to complicated and inefficient two-model optimization scheme.
We aim to disentangle data collection, remove solver restriction, and maintain consistency with stan- dard supervised pretraining in diffusion RL. As a diffusion policy admits single forward (noising) process but multiple reverse (denoising) processes (e.g., different samplers), a natural question is:
Can diffusion reinforcement be performed on the forward process instead of the reverse?
This paper proposes novel online RL paradigm named Diffusion Negative-aware FineTuning (Dif- fusionNFT). Instead of building upon the conventional policy gradient framework, DiffusionNFT directly performs policy optimization objective. Intuitively, it defines a contrastive improvement direction between two implicit policies learned "positive" and "negative" generated samples split by reward signals, and optimizes ward the positive policy without modifying the sampling process.
the forward diffusion process through the flow matching
The forward-process RL formulation provides several practical benefits (Figure 2}. sionNFT allows data collection with arbitrary black-box solvers, rather than relying on first-order SDE samplers. Second, it eliminates the need to store entire sampling trajectories, requiring only clean images for policy optimization. Third, it is fully compatible with standard diffusion training, requiring minimal modifications to existing codebases. Finally, it is a native off-policy algorithm, naturally allowing decoupled training and sampling policies without importance sampling.
First, Diffu-
We evaluate DiffusionNFT by post-training Medduu (Esser et al., 2024) on multiple reward models. The entire training process deliberately operates in a CFG-free setting. Although this results in a significantly lower initialization performance, we find DiffusionNFFT substantially improves performance across both in-domain and out-of-domain rewards, rapidly outperforming CFG and the GRPO baseline. We also conduct head-to-hear comparisons against FlowGRPO in single-reward settings. Across four tasks tested, DiffusionNFT consistently exhibits 3x to 25 x efficiency and achieves better final scores. For instance, it improves the GenEval score from 0.24 to 0.98 within Ik steps, while FlowGRPO achieves only 0.95 with over 5k steps and additional CFG employment.
2

---

Published as a conference paper at ICLR 2026
DiffusionNFT is direct RL alternative to conventional Policy Gradient methods, introducing the Negative-aware FineTuning (NFT) paradigm (Chen et al.] (025c) into the diffusion domain. Grounded in supervised learning foundation, believe this paradigm offers valid path toward a general, unified, and native off-policy RI. recipe across various modalities.
2 BACKGROUND
2.1 DIFFUSION AND FLOW MODELS
Diffusion models (Ho et al. 2020 Song et al.! 2020b) learn continuous data distributions by grad- ually perturbing clean data Zo
Pdata with Gaussian noise according to a forward process. Then, data can be generated by learning to reverse this process.
The forward noising process admits a closed-form transition kernel (ejo(Z; æo) N (aa of with specific noise schedule oe,, ot, enabling reparameterizzttin
= 0; + - ((0,,~
One way to learn diffusion models is to adopt the velocity parameterization o9 b) Zheng al. 2023b), which predicts the tangent of the trajectory, trained by minimizing
Bo. [vo 1) - v 11
where the target velocity is defined by the schedule's time derivatives as v 6,z0 + O,e under the notation Jt == /dl, and w(L) is some weighting function. Reverse sampling typically follows the ODE form (Song et al. 2020b) of the diffusion model, which is reduced to dæ, (0 t) using Vg. This lormulation is known as flow matching (Lipman et al. (022). where simple Euler discretization serves
effective ODE solver, equivalent to DDIM (Song et al. 2020a).
Rectified flow (Liu et al. 2022) can be considered as special case of the above-discussed diffusion models, where 03 =1- t, t, which simplifies the velocity target to c Vg.
2.2 POLICY GRADIENT ALGORITHMS FOR DIFFUSION MODELS
In order to apply Policy Gradient algorithms such as PPO Schulman et al. 2017) or GRPO (Saa al. 2024) to diffusion models, recent works (Black al. 2023 Fan et 2023 Liu al. 2025 Xue et al. 2025) formulate the diffusion sampling as multi step Markov Decision Process (MDP). This can be achieved by discretizing the reverse sampling process of diffusion models.
While flow models naturally admit simple and efficient sampling through ODE, the lack of stochas- ticity hinders the application of GRPO. FlowGRPO (Liu et al., 2025) addresses this by using the SDE form (Song et al. 2020b) under the velocity parameterization vo (see Appendix B.I.
g (ar: + (1 - t))o( 0) dt + g.dwo
dæ.- = (o (1,, 1) +
(2)
2t
where 9t a. controls the level of injected stochasticity. Discretizing it with Euler yields
ot (æ, + (1 (2, 1)) | At, of Ac)..
T0(0;-11 w.) = .N (o 1)
This makes transition kernels between adjacent steps likelihood tractable Gaussians, enabling the direct application of existing policy gradient algorithms, such as GRPO.
3 DIFFUSION REINFORCEMENT VIA NEGATIVE- FINETUNING
PROBLEM SETUP
Online RL. Consider a pretrained diffusion policy old and prompt datasets {c). At each iteration, we sample K images xn for prompt c, and then evaluate each image with scalar reward function E [0. representing its optimality probability r(wo, c) p(o 1/ c) (Levine 2018
3

---

Published as a conference paper at ICLR 2026
This optimality serves bridge from continuous- valued rewards to binary partition. Collected data can be randomly split into two imaginary subsets. An image 2, will have a probability r of falling into the positive dataset D and otherwise the negative dataset D Given infinite samples, the underlying distributions of these two subsets are respectively
x (xolc)) == nTl (2olo 1, p = 20 c) (xolc)
r(xo, c)
old (xale)
Py (o (e)
P (0 1 cc
π (xooec) == ف (z0 = 0, plo 0| 0 c)x old (2pp (c)
r (o c)
π (xolc)
p 0/c)
1 Panao (o 1(c)
RL requires performing policy improvement at each iteration. The optimized policy .' satisfies (xle c) (e00,)) (denoted n x old)
Policy Improvement on Positive Data. thus straightforward improvement of no can be x" x+ To achieve this, previous work (Lee et al. 2023) performs diffusion training solely on D++ known as Rejection Tuninu (RFT).
is easy to prove that π > nld > π constantly holds,
Despite the simplicity, RFT cannot effectively leverage negative data in D (Chen al. 2025c).
Reinforcement Guidance. We posit that negative feedback is crucial to policy improvement, es- pecially for diffusion Rather than treating - an optimization point, we leverage both negative and positive data to derive an improvement direction A E R". The training target is defined as
1 (xr, e, 1) + ( (,).,
(tt C, 1) = oldd (r,,
where is the veloeity predictor of the diffusion model, 3 is hyperparameter. This definition formally resembles diffusion guidance such as Classifier-Free Guidance (CFG) (Ho & Salimans 2022). We term A(x;, t) I reinforcement guidance, and 1/4 E guidance strength.
In Section 3.2 we address two challenges: 1. What is an appropriate form of A that enables policy improvement? 2. How to directly optimize Uy + v" leveraging collected dataset Dt and D
3.2 NEGATIVE-AWARE DIFFUSION REINFORCEMENT WITH FORWARD PROCESS
In Eq. (3), A corresponds to the distributional shift between an improved policy and the original policy. To formalize this, we first study the distributional difference between n+ > 7 > -.
Theorem 3.1 (Improvement Direction). Consider diffusion models ut, w² and old for the policy triplet n+, π. .nndd ald, The directional differences between these models are proportional:
A : a(x))
[odd (I,, -v "0(x, c,t) (Reinforcement Guidance) I0 (xt,,,,,))) (
a(x))
(4)
where 0 S a(xx)) S 1 is scalar coefficient:
Re (x)lc)
a(x:)
F n (x
(zole) r(
Eq. indicates an ideal guidance direction
D
D
Guidance
A
A for improving over oold With appropriate guidance strength, policy improvement can be guaranteed. For instance, let 3 a(x,) in
1.3
Eq. 33) we have V² (x,, c, = vold( (x, c, + 4 (a c, t), such that T = + > olld holds. Figure contains an illustration for the improvement direction A.
D
Vo
r=03
((a)) A(x,, c,
volld
Figure 3: Improvement Direction.
Having defined valid optimization target v* with Eq. (3) and we now introduce a training objective that directly optimizes vy towards e":
We find that finetuning only on the positive data leads to collapse (Section 4.4
4

---

Published as a conference paper at ICLR 2026
Implicit Positive Policy
Condition
Images
x
Noisy x k
[a cute dog)
Vaa
v (x,lc)
112
+
min()
π old
B
Volla
Va (xelc):
112
P E [0, 1] Optimality Rewards
1 ++ Implicit Negative Policy
Figure 4: Nif jointly optimizes two dual diffusion objectives, on both positive (r = 1) and negative (r 0) branches. Rather than training two independent models v and vo it adopts an implicit parameterization technique that directly optimizes a single target policy vo-
Theorem 3.2 (Policy Optimization). Consider the training objective:
(()) (,,.""( rl (at, c, 1) - 012 + - -r)llo (2t, e, - -
(5)
where Ug (2), c, == (1 (1- 3) 3) (a,, c, + (xr. c, t),
(Implicit positive policy)
and on (xi, C, 1) = (1 + 3) 30 (at;, c, 3 (t. c, e.
(Implicit negative policy)
Given unlimited data and model capacity, the optimal solution of Eq. 5 satisfies
2 Vg- (g c, odil x (,,, A(x,.
(6)
Theorem presents new off- policy RL paradigm (Figure Instead of applying Policy Gradient, it adopts supervised learning (SL) objectives, but additionally trains on online negative data D This renders the algorithm highly versatile, compatible with existing SL methods. We term our method Diffuasion Negative-awar FineTuning (DiffusionNFT)) highlighting its negative-aware SL nature and conceptual similarity to parallel algorithm NFT in language models (Chen et al. 2025c
Below, we discuss several distinctive advantages of DiffusionNFT.
1. Forward Consistency. In contrast policy gradient methods (e.g.. FlowGRPO), which for- mulated RL on the reverse diffusion process, DiffusionNFT defines a typical diffusion loss on the forward process. This preserves what term forward consistency- the adherence of the diffusion model's underlying probability density to the Fokker- Planck equation (Øksendal, 22033] Song et al.) 2020b), ensuring that the learned model corresponds coupled with Øn through a joint distribution ao (T æo) = T (2o) Tejo (2; |zo)).
valid forward process (i.e., 3, are correctly
2. Solver Flexibility. DiffusionNFT fully decouples policy training and data sampling. This enables the full utilization of any black-box solvers throughout sampling, rather than relying on first-order SDE samplers. also eliminates the need store the entire sampling trajectory during data collec- tion, requiring only clean images with their associated rewards for training.
Guidance Integration. Intuitively, DiffusionNFT defines guidance direction A and apply such guidance to the old policy nidd (Eq. (6)). However, instead of learning separate guid- ance model Ag and employing guided sampling, adopts an implicit parameterization technique that enables direct integration of reinforcement guidance into the learned policy. This technique, inspired by recent advances in guidance-free free training (Chen al., 2025a). allows us to perform RI. continuously single policy model, which is crucial online reinforcement.
Implicit
4. Likelihood-Fai Formulation. Previous diffusion RL methods are fundamentally constrained by their reliance on likelihood approximation. Whether approximating the marginal data likelihood with variational bounds and applying Jensen's 's inequality to reduce loss computation cost (Wallace et al. 2024), or discretizing the reverse process to estimate sequence likelihood (Black al. (023)), they inevitably introduce systematic estimation bias into diffusion post-training. In contrast, Diffu- sionNFT is inherently likelihood bypassing such compromises.
3.3 PRACTICAL. IMPLEMENTATION
We provide DiffusionNFT pseudo code in Algorithm Below, elaborate on key design choices.
5

---

Published as a conference paper at ICLR 2026
Algorithm Diffusion Negative-aware FineTuning (DiffusionNFT)
Require:
Pretrained diffusion policy vdd raw reward function y (-) E IR, prompt dataset {c}. Initialize: Data collection policy yold v'e training policy by oo data buffer D < 0.
1: for each iteration do
2: for each sampled prompt c. do
11 Rollout Step, Data Collection
3: Collect K clean images æb:K, and evaluate their rewards { 1
4: Normalize raw rewards in group: p = p mean [I 1:
5: Define optimality probability 0.5 + 0.5 clip
7e. 1, 1.
6:
4 {c, mo , E [0,
7:
end for
8:
for each mini batch {c, 2o. E E do
1/ Gradient Step, Policy Optimization Forward diffusion process: æ; = a. + d;c; v = d. + die.
9:
10:
Implicit positive velocity: (2s; c, == (1 3)
(x) (xr, c, + ( (z:; C, t).
Implicit negative velocity: by (3t, c, == (1 + 9)0 ( (ax, c,t) Brs (Te, C, t). 0 < 0 A [rllog (It, e, 21l2 + (1 r)ll (mx, e, =113]- (Eq. 55
12:
13:
end for
Update data collection policy Oold <- r,000l (1 - 70). clear buffer < Online Update
15:
end for
Output: Up
Optimality Reward. In most visual reinforcement settings. rewards manifest as unconstrained continuous scalars rather than binary optimality signals. Motivated by existing GRPO practices (hao et al.) 2024 Liu et al. 2025; Xue et al. 2025), we first transform the raw reward faw into r E 00 which represents the optimality probability:
y. ( c) Ex (xo, c) 1,1
r(zo, e)
clip
Ze
Ze > 0 is some normalizing factor, which could take the form of a global reward std. We sample K images for each prompt e during data collection, so the average reward (= ( (xu, c) for each prompt can be estimated.
Soft Update Sampling Policy. The off-policy nature of DiffusionNFT decouples the sampling policy old from the training policy тo. This obviates the need for a "hard" update (I x- x²) after each itcration. Instead, we leverage this property to employ "soft" EMA update:
gold <- nodold + (1 77:)
where i is the iteration number. The parameter governs a trade-off between learning speed and stability. strictly on-policy scheme (n7 0) yields rapid initial progress but is prone to severe instability, leading to catastrophic collapse. Conversely, nearly offline approach (1) - 1) is robustly stable but suffers from impractically slow convergence (Figure
Adaptive Loss Weighting. Typical diffusion loss includes a time-dependent weighting w(t) (Eq. (I). Instead of manual tuning, we adopt an adaptive weighting scheme. The velocity predictor Vy can be equivalently transformed into 2o predictor, denoted rectified flow schedule). We replace the weighting with a form of self normalzedd ay regression, motivated by the diffusion distillation method DMD (Yin et al. 2024):
(e.g., Ey tvo under
[2 C, xoll
w(t)ll e, 21| sg ag
c,t) xo)))
where sq is the stop-grt graddentt operator. We find typically leads to faster training (Figure
CFG-Free Optimization. Classifier-Frec Guidance (CFG) (Ho & Salimans 2022) is a default tech- nique to enhance generation quality at inference time, yet complicates post-training and reduces efficiency. Conceptually, we interpret CFG as an offline form of reinforcement guidance (Eq. (4). where conditional and unconditional models correspond to positive and negative signals. With this understanding, discard CFG in our algorithm design, and the policy is initialized solely by the conditional model. Despite this seemingly poor initialization, we observe that performance surges and quickly surpasses the CFG baseline (Figure T. This suggests that the functionality of CFG can be effectively learned or substituted through RL wost-training, echoing recent studies that achieve strong performance without CFG through post-training (Chen et al. 2025b
Zheng et al. 2025).

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation Results. Gray-colored: In-domain reward.
Evaluated on official checkpoints.
Evaluated under x 1024 resolution. Bold: best; Underline: second best.
Rule-Based
Model-Based
Model
#Iter
GenEval
OCR
PickScore
ClipScore
HPPSv..1
Aesthetic
ImgRwd
UniRwd
SD-XIF
0.55
0.14
22.42
0.287
0.280
5.60
0.76
2,93
SD3..5LLF
0.71
0.68
22,91
0.289
0.288
5.50
0.96
3.25
F-UX.
0.59
22.84
0.295
0.274
5.71
0.96
3.27
SD3.5-M (w/o CFG)
0.24
0.12
20.51
0.237
0.204
5.13
-0.58
2.02
+ CFG
0.63
0.59
22.34
0.285
0.279
5.36
0.85
3.03
PlowGRPO
>5k
0.95
0.66
22,51
0.293
0.274
5.32
1.06
3.18
0.66
0.92
22.41
0.290
0.280
5.32
0.95
3.15
0.68
23.50
0.280
0.316
5.90
1.29
3.37
Ours
1.7k
0.91
23.80
0.293
0.331
6.01
1.49
3.49
4 EXPERIMENTS
We demonstrate the potential of DiffusionNFFT through three perspectives: (1) multi-reward joint training for strong CFG-free performance, (2) head-to-head comparison with FlowGRPO on single rewards, and (3) ablation studies key design choices.
4.1 EXPERIMENTAL SETUP
Our experiments are based on SD3. 5 -MMiiiu um (Eser al. 2024) 512x512 resolution, with most settings aligned with FlowGRPO (Liu et al. 2025).
Reward Models. (1) Rule-based rewards, including GenEval (Ghosh al. 2023 for compo- sitional image generation and OCR for visual text rendering, where the partial reward assignment strategies follow FlowGRPO. (2) Model-based rewards, including PickScore (Kirstain et al.)
2023). C1 iScorre (Hessel et al. 2021). HPSv2. 1 (Wu et al., 2023), Aest Aesthet i (Schuh- mann 2022 ImageReward (Xu al. 2023) and Un Feedeeaadd (Wang al. 2025). which
measure image quality, image-text alignment and human preference.
Prompt Datasets. For GenEval and OCR, we use the corresponding training and test sets from FlowGRPO. For other rewards, we train on Pick-a-Pic (Kirstain et al., 2023) and evaluate on DrawBench (Saharia et al., 2022).
We finetune with LoRA (a 64, 32). Each epoch consists of 48 groups with group size G = 24. We use 10 rollout sampling steps for head-to-head comparison and ablation studies, and 40 steps for best visual quality in multi-reward training. Evaluation is performed with 40-step first-order ODE sampler. Additional details are provided in Appendix C
Training and Evaluation.
4.2 MULTI-REWARD JOINT TRAINING
We first assess DiffusionNNFTss cefioonNNFTsfffffctteeeeeeee effectiveness in comprehensively enhancing the base model. Starting from the CFG-free SD3.5-M (2.5B parameters), we jointly optimize five rewards: GenEval. OCR, PickScore, ClipScore, and HPSv2. 1. Since the rewards are based on different prompts, we first train on Pick-a-Pic with model- rewards to strengthen alignment and human prefer- ence, followed by rule-based rewards (GenEval, OCR). Out-of-domail evaluation is conducted on Aesthetic iss ImageRewa rd, and Un FiedRwardd.
As shown in T our final CFG-free model not only surpasses CFG and matches FlowGRPO (fit- ted only single rewards) on both in-domain and out-of-domain metrics, also outperforms CFG- based larger models such as SD3.5-L (8B parameters) and FLUX. (12B parameters) (Labs) 2024). Qualitative comparison in Figure demonstrates the superior visual quality of our method.
4.3 THEAD-TO-HEE HEAD COMPARISON
We conduct head-to-heds head comparisons with FlowGRPO on single training rewards. As shown in Figure [a) and Figure 6, our method is 3x to 25x more efficient in terms of wall-clock time,

---

Published as a conference paper at ICLR 2026
photo of pizaa and yellow bascball glove
wibrant bon alley groff prominently spray- pointed Street Rules", surrounded by ful tags murals, surey sky.
New York Skyline with (Googl Research writton fir Firworrke sky.
colored
An shotoyraph 1920s airshp shoped pig. flasting over wincst field,
Google Te Para
SD3.5-M
RMEES
Google
Pizzarh
Cafe
FlowGRPO
RAES
Google Research Pizza
DiffusionNFT
STEET ART RILES
Figure 5: Qualitative Comparison. The prompts are taken from GenEval, 1, OCR and DrawBench respectively, where we compare the corresponding FlowGRPO model with our model.
24.00
23.75
34X Effciine
23.50
HPSV2.1 Score
Emfcieny
Efficency
23:00
22.75
FlowGRPO
FlowGRPO
FlowGRPO
DiffusionNFT
22.25
NiffusionNFT
Niffusion
22.00
0.28
200
600
800
1000
250
500
750
1000 1250 1500
Training Time (GPU Hours)
Training Time (GPU Hours)
Training Time (GPU Hours)
(a)
(b)
(c)
Figure Head-to-head comparison between DiffusionNFT with FlowGRPO on single rewards.
achieving GenEval score of 0.98 within only 1k iterations. This demonstrates that CFG-free models can rapidly adapt to specific reward environments under our framework.
4.4
ABLATION STUDIES
1.00
1.0
0.95
0.90
23.0
0.80
0.83
22.5
Centuoll
0.7
minco 0016 0.5)
22.0
minco.01i.00 0.8)
0.75
- 0.0
1stiorder SDE
1st: order SDE
0.4
2ndiorder ODE
28.5
2nd order ODE
ODE
0.3
0.55
203
Training Iterations
600
1000
10o Training iterations
1000
100
Training Iterations
200
400
500
(a)
(b)
Figure 7: Different diffusion samplers for data collection.
Figure 8: Soft-update strategies.
We analyze the impact of our design choices:
Negative LUss The negative-aware component crucial in DiffusionNFT. Without the negative policy loss ey find rewards collapse almost instantly during online training. highlighting the
8

---

Published as a conference paper at ICLR 2026
21.0
0.9
teentvel Scre
9.5
22.5
0.8
PickScare
GenEval
0.6
wi!!
0.3
9.5
with
21.0
f=0001
wit!
MV3 =
B=10
0.4
Adaptive
Adeptivee
f- 10.0
120
200
320
400
500
100
200
200
400
500
100
263
403
Training Iterations
Taining Iterations
Training Iterations
(a)
(b)
Figure 9: Different time- time-dependent weighting strategies.
Figure 10: Choices of strength B.
essential role of negative signals in diffusion RL. This phenomenon is divergent from observations in LLMs, where RFT remains strong bascline (Xiong et al. 2025 Chen et al. (025c)).
Diffusion Sampler. Online samples DiffusionNFT are used both for reward evaluation and as training data, making quality critical. Figure 7 shows that ODE samplers outperform SDE ones, especially on PickScore, which is soise-sensitive. Second-order ODE slightly outperforms first- order on GenEval, while being comparable on P PickScore.
daptivv Weighting. We find stability improves when the flow-matching loss given higher weight at larger i, whereas inverse strategies (e.g., w(t)= = 1 - 1) lead to collapse (Figure 9 schedule consistently matches or exceeds heuristic choices.
Our adaptive
Soft Update. We compare different 7/ schedules for the soft update in Figure 8 Fully on-policy
(% 0) accelerates early progress but destabilizes training, while overly off-policy (1) = 0.9) slows convergence. We find that starting with a small 1) and gradually increasing it to a larger value in later stages strikes an effective balance between convergence speed and training stability.
Guidance Strength. As shown in Figure 10 the guidance parameter 3 also governs trade-off between stability and convergence speed. We find that 3 near I performs stably and select 3 as I or 0.1 (for faster reward increase) in practice.
5 RELATED WORK
The transition of RL algorithms from discrete autoregressive (AR) to continuous diffusion models poses central challenge: the inherent difficulty of diffusion models for computing exact model likelihoods (Song al. 2021), which are nonetheless crucial for RL (Chen al. 2023 Liu et al. 2025). To address this challenge, existing efforts include:
Likelihood-free methods: (1) Reward Backpropagation (Xu et al. 2023 Prabhudesai et al. 2023
Clark et al. 2023) Prabhudesai et al.) (024) proves highly effective, yet is limited to differentiable rewards and only tune low-noise timesteps due memory costs and gradient explosion when unrolling long denoising chains. (2) Reward- Weighted Regression (RWR) (Lee et al., (023) is an
offline finetuning method but lacks negative policy objective penalize low-reward generations.
(3) Policy Guidance. This includes energy guidance (Janner et al. [022]] Lu et al.) 2023) and CFG-
style guidance (Frans al., 2025) Jin al. 2025). These methods all require combining multiple models for guided sampling, thus complicating online optimization. (4) based RL. These methods try to perform RL directly on the score rather than the likelihood field (Zhu et 2025).
Likelihood-based methods: (1) Diffusion-DPo (Wallace et al. 2024 Yang et al. 2024 Liang et al.)
2024 Yuan et al., 22244 [L al. 2025a) adapts DPO to diffusion for paired human preference data but requires additional likelihood and loss approximations compared to AR; DDO (Zheng et al.) 2025) uses high-quality dataset as positive signals and self-generated samples as negative signals to avoid the requirement of paired data, achieving state-of-the-art CFG-free FIDs in visual generation, while still relying on likelihood approximation for the diffusion case. (2) Policy gradient methods, starting from PPO style (Black et al. 2023 Fan et al. 2023), decompose trajectory likelihoods step by step without considering forward consistency. Recent GRPO extensions (Liu et al.) [025 Xue
et al. 2025) prove effective and scalable for diffusion RI, but they couple the training loss with

---

Published as a conference paper at ICLR 2026
SDE samplers and face efficiency bottlenecks. MixGRPO (Li et al. 2025b) improves efficiency by mixing SDE and ODE, while issues of coupling and forward inconsistency remain.
CONCLUSION
We introduce Diffusion Negative-awes -warr FineTuning (DiffusionNFT)), new paradigm for online reinforcement learning of diffusion models that directly operates the forward process. By formu- lating policy improvement as a contrast between positive and negative generations, DiffusionNFF integrates reinforcement signals seamlessly into the standard diffusion objective, eliminating the re- liance on likelihood estimation and SDE-based reverse process. Empirically, DiffusionNFT demon- strates strong and efficient reward optimization, achieving up 25x higher efficiency than Flow- GRPO while producing a single model that outperforms CFG baselines across diverse in-domain and out-of-domain rewards. We believe this work represents step toward unifying supervised and reinforcement learning in diffusion, and highlights the forward process as promising foundation for scalable, efficient, and theoretically principled diffusion RL.
THE USE LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMs) solely as writing assistant for language polishing and improving clarity of presentation. The LLMs were not involved in research ideation, methodolog- ical design, experimental execution, or result analysis. All scientific contributions and substantive writing were carried out by the authors.
ACKNOWLEDGMENTS
This work supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. /YB2025XDXM101), NSF of China Projects (Nos. 62550004, U25B6003, 92370124, 92248303); Beijing Natural Science Foundation L247011; the High Perfor- mance Computing Center, Tsinghua University. J.Z was also supported by the XPlorer Prize.
We thank Cheng Lu, Hanzi Mao, Zekun Hao, Tao Yang. Zhanhao Liang, Shuhuai Ren, Tenglong Ao, Xintao Wang, Haoqi Fan, Jiajun Liang, Yuji Wang, and Hongzhou Zhu for the valuable discussion.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.1 /3301, 2023.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023.
Huayu Chen, Kai Jiang, Kaiwen Zheng, Jianfei Chen, Hang Su, and Jun Zhu. Visual generation without guidance. Forty-second international conference machine learning, 2025a.
Huayu Chen, Hang Su, Peize Sun, and Jun Zhu. Toward guidance-free visual generation via condition contrastive alignment. ICLR, 2025b.
Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Yin Cui, Haotian Ye, Tsung-Yi Lin, Ming-Yu Yu Liu, Jun Zhu, and Haoxiang Wang. Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint arXiv:2505.18116 2025c.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models differentiable rewards. arXiv preprint arXiv:2309. (ato:2339.74400 /7400, 2023.
10

---

Published as a conference paper at ICLR 2026
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image tomee diffusion models. Advances in Neural Information Processing Systems, (9.79858-79888 2023.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is controllable policy improvement operator. arXiv preprint arXiv:2505. 234,255,00..24888, 2025.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt, Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
Martin Gonzalez, Nelson Fernandez Pinto, Thuy Tran, Hatem Hajri, Nader Masmoudi, et al. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models. Advances Neural Information Processing Systems, 36:68061-681 20, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12590 2022.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 6851, 2020.
Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209-288 2010.
Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective diffusion- based generative models and score matching. Advances in Neural Information Processing Sys- tems, 34:22863-22876, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
Luozhijie Jin, Zijie Qiu, Jie Liu, Zijie Diao, Lifeng Qiao, Ning Ding, Alex Lamb, and Xipeng Qiu. Inference-ting time alignment control for diffusion models with reinforcement learning guidance. arXiv preprint arXiv:2508.2 1016, 2025.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances neural information processing systems, 34:21696-21707, 2021.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances neural information processing systems, 36:36652-36663,, 2023.
Black Forest Labs. htrp://prr./gggtooooooooooooooon
2024.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 018880050099.9 2018.

---

Published as a conference paper at ICLR 2026
Binxu Li, Minkai Xu, Meihua Dang, and Stefano Ermon. Divergence minimization preference optimization for diffusion model alignment. arXiv preprint arXiv:2507.07510, 2025a.
Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Miles Yang, and Zhao Zhong, Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint urXiv:2507.2180) 2025b.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint arXiv:2406.04331 2(5):7, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02744, 2022.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint M.Xi:250550057000 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint vxXv:2200.00303,, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in neural information processing systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++; Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:221..01095, 2022b.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu, Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning, arXiv preprint arXiv:2304. 12824, 2023.
Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations: an introduc- tion with applications, pp- 38 18.50. Springer, 2003.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. 2023.
Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint arXiv:2407.08737 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural informa- tion processing systems, 35:36479- -34444, 2022.
Christoph Schuhmann.
Laion-aesthetics.
htttps://aiionaa//bll
Laion-aestheticn 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 133456 2020b.
12

---

Published as a conference paper at ICLR 2026
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. In Advances in Neural Information Processing Systems, volume 34, pp. 1415-1428, 2021.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1228-8238 2024.
Feng Wang and Zihao Yu. Coefficients-preseeve sampling for reinforcement learning with flow matching. arXiv preprint arXivv2509.0552) 2025.
Yihin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multi- modal understanding and generation. arXiv preprint arXiv:2503.052.6. 2025.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score:
Better aligning lext-to-image models with human preference. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 2096-2105, 2023.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. 11343, 2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 15033-1553, 2023.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.072 078188 2025.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941 191118991,,1 2024.
Tianwei Yin, Michaäl Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6613-6623, 2024.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. Advances in Neural Information Processing Systems, 37: 73366-73398, 2024.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprini arXiv:2204. /3902, 2022.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-ssover-v3: Improved diffusion ode solver with empirical model statistics. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023a.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, pp. 42363- 42389. PMLR, 2023b.
Kaiwen Zheng, Guande He, Jianfei Chen, Fan Bao, and Jun Zhu. Diffusion bridge implicit models. arXiv preprint arXiv:2405.15885 2024.
Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Direct discriminative optimization: Your likelihood-basee visual generative model is secretly gan discriminator. In ICML, 2025.
Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model alignment. In The Thirteenth International Conference on Learning Representa- fions, 2025.
13

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREMS
Lemma A.1 (Distribution Split). Consider the distribution triplet n²,, nt, and n as defined in
Section 3.7
(x)l)) = n (x00 o 1, p(o 1 1, c) C) (xolee
r( c)
7 (xolc)
(7)
Pa (0 = 1/c)
Paou = 1/c)
π =xxolc) = T TTall (xool 0,c) plo = 20 C)x old (xu(e)
1 r( c)
1 Paiu 1(c) x
(xule)
P 0/c)
(8)
n (x0l)c) is as linear combination between its positive splis π (xooc) and negative split (zalc))
ald add ( c) = P 1|c)x (xolc) + - - 1|c]] (
(xole)
Proof. The result follows directly from Eq. and ().
Lemma A.2 (Posterior Split). The diffusion posteriors for distribution triplet xt, xt, and nold satisfy:
π (x0 æt, c) ( ( (2e c) + a ( (x c)
where a(xx) == ni (x:|c)
E (Io. c)
nolll (x:(c)
Proof. Leveraging Bayes' Rule:
[olcc) nt (x: c) maje (wo (x c)
x (2> (o)
Replacing all distributions in Eq. 99) (Lemmaa A. T/ (z (c) mojd (xo xe. c) T (12 (00)
we get
(x) (eglc) ( (Io xee c)
Pa (o = 1/c)
<xx mo)
Te (arel () (20 l c)
+ [1 - - -1/c)]
(x; 2o))
( (wo (00 c) =Pnna 00= 1/c) n. [ (c)
7 Taje
Toje (#o ac,,
= (x: cc)
T. (x/
P (o 1|c)
T (wo (x2, c)
(x: cc)
Diffuse both sides of Eq.
9 have
(x: a (o 1/c)m (x:|c) + +[1 -1-Ppau 1 (x:/e)
P
क. (z,[c)
T.. (z)
P
1/c)
+ Po 1[c]
π (x)(c)
gold (x c)
Note that
P (o 1/c) = (x))
We have
Tojt Tajt (2 [ c) = axx)) (Iu x c) + - a(x:))()))))))) (To (zulze,
old
Theorem A.3 (Improvement Direction). Consider diffusion models vt, and old for the dis- tribution triplet πo, π and oldd The directional differences between these models are parallel:
A := a(x:)) n0 (æx, C, 1) -- (a,, c, 1)] (Reinforcement Guidance) = a<(x))) (x= c, old (xt,, c, 1)]-
where 0 S a(xx)) S a scalar coefficient:
oxxx) == me (x lc)
Told(x (x)(c)
14

---

Published as a conference paper at ICLR 2026
Proof. According to the relationship between the optimal velocity predictor and the posterior mean of zo (i.e., the optimal zy predictor) (Zheng et al. 2023b):
o (at, C, () a,3; Dy aw
.) [00
v [1, c,t) = a, + b, (mo m [00]
( () = 0, + by 1x (ar cc) [n
b, 0,0 where 0r 01
de,
Based on Lemma A.2 have
a
oodd (a., t) = ax (æz, 2, + [1 a(xx)) (xte, c,
Rearranging the equation, we complete the proof.
Theorem A.4 (Reinforcement Guidance Optimization). Consider the training objective:
(()) = - r (at, c,t) - v12 + (1-r)llv, (xx, c, u
(10)
where og == (1- (1--))))) (at, c, + 300 [1: (x:: c, t):
(Implicis positive policy)
and vo c,,, := (1 3) (I2, c, B (2); c, c,
(Implicit negative policy)
Given unlimited data and model capacity, the optimal solution of Eq. (10) satisfies
2 Vg- (It, c, () o (x,, c,t) + A c.
Proof.
[(()- = (m. (mol).)) r( c) (a,,
(.,, c,t) v122 + r( c))l,, c,t, - -
tf (mxle) (e (20 c) c) (a,, c,t) 2112
E Eag (mro (m)) [1 r( c) 25 (2 c,t) v 21
From Lemma
A.1 have r(zp,
P (o 1 therefore:
nold( (xollc))(() (m)
r(Zu, mo (xp x c) r(zu, c)
olld (mx (c)
a (xx [c) n (z) (D))
= Pami = 1/c)
old [x, c)
n (xx(c)
: (wi cc)
P0 1/c)
Tofe (wo ax, c)
+
old (x:
a(xx))) )moje (2r0 (2
Similarly,
[1 - r(wo, c)] c) = axxx)l (200 c)
Then,
((0) = exx (a )R (ooten))) c,t) 2l12
+ [1 - (zx (wo e) bl c,t, - -1/2)
= [B.
(ot, c,t) - E a (o) [0)1
+[1 - a ( c) E ( [b] [w]13 +
=E [ (2,, c, - ut c,t)
[1 - - ( v (x,, ())
15

---

Published as a conference paper at ICLR 2026
Combining Theorem A.3 0g (ze, C, 1) xz (ax, C, () (1 - -3)00 C, () +
observe that
-² (xe, c,
= =B(00 - aold A
3 (a())))
1ig (,,) - x- (ar, = (1 + B) (12,, C. ()
(20 C, -v -0 c,
A
= -00000 olu
B 1
Substituting these results into L(0):
1 A
((0) = a, (an le) (a( 100 oold
3 (xx))
1
+ [1 |1- a old
[2] + C1
B
((x))
A
F o (m))) (axaa)))1ee)
(0
+
B Ba(x)) )11
1
A
+ [1 a<x)))))l (2rald
)1121 + C1
8
a (xx)
A
E Be.tt (a cc) 10 (z
+
- a a<x;)) + 31- a(x;)
112 + C2
B axx)))
2 A) C2
E |100 oo
from which is obvious that the optimal 8* satisfies va- (xte c, uold c,t) + A c,t).
B
THEORETICAL DISCUSSIONS
B.1 FLOW SDE
As flow models are a special case of diffusion models under the rectified schedule a; = 1- = t, the earliest results diffusion SDEs (Song et al. 2020b) be directly applied without difficulty. FlowGRPO (Liu et al. 2025) and DanceGRPO (Xue et al. 2025) derive the flow SDE with unex-
plained hyperparameters Sr 0. - additional complexity. We provide simpler and more principled perspective based solely on the diffusion model framework.
To leverage the diffusion SDE formulation in /Song et al. (2020b), we need to match its forward SDE 0rAE0 + a,E. As noted in the
dæ, = f())x,dt + g(()dw, with the forward transition kernel first two arXiv versions of the VDM paper (Kingma et al. 2021 (()),g(() are related to ar, az by
f(1) log (2) dt
do? dogag 0z Setting at
t, or
have
de
1
2t
S(I) 1- 1.
((())
(11)
1-1
for rectified flow. According to (Huang et al. 2021
the generalized reverse SDE takes the form:
dæ; [(1)a, 1+ X2
2 (2(() se log (xx:) di + Xrg(()do
(12)
where Xa E [0, 1]. Equivalently, it amounts to introducing Langevin dynamics top of the diffusion ODE, with X, = 0 corresponding to ODE, and A = corresponding to the maximum variance SDE in Song et (0020b). The score function s0 1) V Va log (xx noise predictor c0 !). data predictor 2e [e, () and velocity predictor up (Ze, !) are interconvertible under general noise schedules (Zheng et al. 2023b):
60(0;, 1) = -=0,80(21,1) (), 2n 1)
volar, 1) ao ( ()
04
(13)
16

---

Published as a conference paper at ICLR 2026
Applying these relations to the rectified flow schedule, can derive:
3t + (1 t)
(a 1)
(14)
Substituting Eq. and Eq. (14 into Eq. 12 we have the diffusion SDE under rectified flow:
X2
21 dw
da (1 2 ( 1)
Dt de
(15)
2t Therefore, the flow SDE in Eq. 2 is essentially conducting transformation 9: A I-r from the interpolation parameter À E [0, 1] to the variance parameter St- This also explains the choice = in FlowGRPO, where a = v2x, is a scaled version of Ax, with a = v2 corresponding to the maximum variance SDE. In comparison, DanceGRPO adopts a fixed variance 9t across timesteps, which is less effective on image models while more stable on video models.
VVT I-t
FRoWGRPO and DanceGRPO directly take the Euler discretization of the flow SDE. In principle, there are more accurate ways, such as utilizing the idea of diffusion implicit models (Song et al., 2020a Zheng et al. 2024), which is equivalent the first-order discretization after applying expo- nential integrators (Hochbruck & Ostermann 2010 Zhang & Chen 2022 Gonzalez et al. 2023).
Specifically, the sampling step from i to t can be derived as:
x. = -1 8)
82
p
(1 8)t - - p² -t)
uo 1) (1
c~ I
(16)
(1- where Pt 7:8 ((--))2+ and 1/. E j0, 1] interpolates between ODE and maximum variance SDE. Compared to the Euler discretization, the DDIM-style discretization avoids singularities at boundaries and is expected to reduce sampling errors. However, we did not observe notable advan- tages by replacing the SDE sampler with stochastic DDIM. Concurrent work (Wang & Yu (025)
improves the SDE sampler through the Coefficients-Pessevvngg Sampling (CPS) principle.
B.2 HIGH-ORDER FLOW ODE SAMPLER
We implement the 2nd-order ODE sampler for flow models based on the DPM-Solver series (Lu et al. 2022a by Zheng et al. 2023a), which uses the multistep method and half the log signal-to-noise ratio (SNR) A log(os, ) for time discretization. Specifically, for three consecutive timesteps < 4-11 < 4-2. where Lt, 1: F4 are already obtained, the update rule for æz, is:
1 Ft,- = 0t. :t,-, - ar, (e - 1) [[1+ 2r:
xe (Tt
2r: we x (12)
(17)
Ut.-1
tug for rectified flow. High- order solvers are also adopted in MixGRPO (Li et al. 2025b) but only for certain steps. Adopting the 2nd-order solver throughout the entire sampling process infeasible, as d will be infinity at
where h; do. A h-1 and the data predictor
boundaries & 0 or f 1. Following common practices, the first and last steps degrade to the first-order solver, which is the default Euler discretization for flow models.
B.3 INTUITION BEHIND THE FLOWGRPO OBJECTIVE
We provide some insight into reverse- prerss- diffusion RL by inspecting the FlowGRPO objective in campler-agnostic manner. For any first-order SDE sampler, the reverse sampling step from to 8 < I can be expressed as
v, = [(s, ((s,t)æ, - m(s, ())g, t) + (nsss, t)e, e~ N(0,1)
(18)
where l(s, t), m(s, s, t), n(s s, t) depend only on t and the sampler. Consider the on on-policy case and the branching strategy in MixGRPO. Starting from a shared æ;, a group of N noises e(1) ...., (N)
sampled and incorporated into the reverse step produce multiple samples (1)
1.... (N)
17

---

Published as a conference paper at ICLR 2026
They go through further sampling, yielding N clean samples and corresponding advantages
A ,..., A(N). On-policy GRPO minimizes the negative weighte log likelihoods:
N A 4 log po (o)
((0)
(19)
where
|x - (1(s, 1) - m( (2), (ze, 1)) +
log po (2x a:1)
(n²(s, t)
(20)
mms, ()) t) m t)Us (2 t) + n(s, () (n,
sg emerges because the samples x." ..... as () are gradient-free. The gradient of the reverse-step log likelihood w.r.t. & can be surprisingly reduced to simple form:
m)(,() Vo mo(x,,))) Vo lo po (a: (x:) n(s,t)
(21)
and
VoC(O) m(s, t) n(s,t)
1 (1
Vo
(22)
Therefore, FlowGRPO essentially aligns the velocity field with the advantage-weighted noise, while the choice of timesteps and sampler only influences the weighting m(s.() across sampling steps. In
m(a.t)
the following, we show a further conclusion that FlowGRPO can be viewed as gradient estimation of reward backpropagation.
Denote Te (2xe as the implicit gradient-free function that solves the PF-ODE from to 0 and fetches the reward on the cleaned sample. The rewards can be expressed as
( = rs (1(s, t)xx, mmss, ()v 1) + ne ((sst)e()) ())
(23)
According to Stein's 's identity, we have
( ( x BE-NiO,) Irs (L(s, t)x; ()ss,t)uo(Ee + ()s,
N
(24)
nss (NBeNiiB (rr (1(x, ()ar. ((s,())ooaa.. 1) + ()6)
Therefore,
1
A (1)) oo(
0
N
n(s,()
BENN(O, Vrs (1(s, f)=, m(s, (a,, () + n(ss, t)e) (ovo(a;, ()
(25)
n(s,t)
m(s,t)c Be-N(O.) (Vors (1ss ()ar m(s, (mr, () n( ()))
where is the global std used in GRPO normalization. Therefore, the GRPO loss gradient is
voL(O) R - Be-w(o.) Vor, (I(,, m(s, t) f) + ni,s, t)c)]
(26)
From the above gradient, GRPO optimizes the reverse transition + when the remaining tra-
jectory s - 0 is gradient-free. Compared to works like ReFL (Xu et al. 2023). which conduct direct gradient backpropagation and approximate
0 with single forward pass (æo-prediction), GRPO introduces higher estimation variance but avoids backpropagation through the s + process, allowing larger and longer sampling chain for O.
18

---

Published as a conference paper at ICLR 2026
C EXPERIMENT DETAILS
Training Configurations. Our setup largely follows FlowGRPO, adopting the same number of groups per epoch (48), group size (24), LoRA configuration (a 64, r 32), and learning rate (3e - 4). For each collected clean image, forward noising and loss computation are performed exactly on the corresponding sampling timesteps. We employ 2nd-order ODE sampler for data collection and enable adaptive time weighting by default.
Single-Reward. For a head-to-head comparison with FlowGRPO under single-reward settings, we fix the number of sampling steps to 10 to ensure fairness. By default, we set 3 1 and fii = min(0.001, 0.5), which work stably for most reward models. In the case of OCR, the reward rapidly approaches within 100 iterations but suffers from instability. To address this, we adopt more conservative soft-update strategy with 7mma 0.999.
Multi-Reward. To comprehensively improve the base model across multiple rewards, we adopt multi-stage training scheme. The training setup involves three categories of rewards and datasets: (1) PickScore, CLIPScore, and HPSv2. rewards the Pick-a-Pic dataset; (2) GenEval reward with the three rewards above on the GenEval dataset; and (3) OCR reward with the three rewards above on the OCR dataset. Since the initial CFG-free generation is of low quality, first train (1) for 800 iterations to enhance image quality, followed by (2) for 300 iterations, (1) for 200 iterations, (2) for 200 iterations, and finally (3) for 100 iterations. All rewards are equally weighted, with PickScore divided by 26 for normalization to [0, 1j. By default, we use 3 = 0.1 and y min(0.0014 0.5), while setting Mmax 0.95 for OCR stabilize training. The number of sampling steps fixed to 40 to ensure sigh-fidelity data collection.
D ADDITIONAL RESULTS
Table 2: Evaluation results of FlowGRPO and DiffusionNFT trained on single rewards, both ini- tialized from CFG-free base model. Gray-colored: In-domain reward. We observe that training exclusively on the OCR reward impairs generalization to other metrics; to compensate this, we en- able CFG when evaluating non- non-OCR rewards for OCR-traieed models.
Rule-Based
Model-Based
Model
#Iter
GenEval
OCR
PickScore
ClipScore
HPSv2.1
Aesthetic
ImgRwd
UniRwd
SD3.55M (wlo CFG)
0.24
0.12
20.51
0.237
0.204
5.13
-0.58
2.02
CFG
0.63
0.59
22.34
0.285
0.279
5,36
0.85
3,03
FlowGRPO
4k
0.97
0.30
21.78
0.277
0.248
5.15
0.74
2.87
Ik
0.66
0.96
21.94
0,280
0.257
5.18
0.31
2,86
4k
0.54
0.60
23.62
0.257
0.295
6.42
1.17
3.17
Ours
1k
0.98
0.36
21.92
01,271
0.251
5.33
0.68
2,91
150
0.54
0.97
21.63
0,281
0.246
5.19
0.37
2,81
2k
0.53
0.64
24.03
0.270
0.315
6.17
1.29
3.40
We provide more qualitative comparison between the base model, FlowGRPO and our multi-reward optimized model in Figure II Figure 12 and Figure
19

---

Published as a conference paper at ICLR 2026
SD3.5-M (w/o CFG)
SD3.5-M (w/ CFG)
+FlowGRPO (w/ CFG)
+DiffusionNFT (w/o CFG)
a photo of a red dog
photo of a tie above a sink
photo of a toothbrush below pizza
photo of black potted plant and a yellow toilet
photo of brown hot dog and purple pizza
Figure 11: Qualitative comparison between FlowGRPO and our model on GenEval prompts.
20

---

Published as a conference paper at ICLR 2026
SD3.5-M (w/o CFG)
SD3.5-M (w/ CFG)
+FlowGRPO CF/C
+DiffusionNFT (w/o CFG)
widt WARUNG
Consult
ate
Doctor
Consut
Doctor
Consult Doctor
close-up up of a medicine bottle with prominent warning label that reads "Consult Doctor", set against neutral background, emphasizing the clarity and visibility of the text.
ORDE I
Order Court
ORDER IN THE COURT
courtroom scene with judge's gavel resting on wooden ploque that reads "Order in the Court", set against the backdrop of quiet, solemn courtroom.
Al TRAINING
Al
realistic photo a tech campus courtyard at night, featuring glowing "AI Training Zone" hologram floating in the center, surrounded by futuristic buildings and greenery. with soft ambient lighting enhancing the futuristic atmosphere.
Chepter
Dark Nijt
Qwateer
antique typewriter with sheet of paper inserted, prominently displaying the typed words: "Chapter 1 It Was Dark Night". The scene set dimly lit, vintage study with single desk lamp casting glow over the typewriter.
Life sa Beach 2024
Life 's Beach 2024
Life's Beach 2024
Com Zork
vibrant beach scene featuring a colorful towel with the phrase "Life's 's Beach 2024" prominently displayed, surrounded by seashells, sunglasses, and flip-flop, set against a backdrop clear blue water and golden sand.
Figure 12: Qualitative comparison between FlowGRPO and our model on OCR prompts.
21

---

Published as a conference paper at ICLR 2026
SD3.5-M (w/o CFG)
SD3.5-M (w/ CFG)
+FIowGRPO (w/ CFG)
+DiffusionNFF (w/o CFG)
wine gloss top of dog.
baby fennec sneezing onto strawberry, detailed, macro, studio light, droplets, backlit ears.
NEURIPS
NEURIPS
NeuRIPS
NeurlPs
A storefront with 'NeurIPS' written on it.
large keyboard musical instrument with wooden case enclosing soundboard and metal strings, which are struck by hammers when the keys are depressed. The strings' vibration is stopped by dampers when the keys are released and can regulated for length and volume by two three pecals.
A side view sitting field.
Figure 13: Qualitative comparison between FlowGRPO and our model on Draw BrawBench prompts.
22


Reviews:
[{"review_id": "1hg95hE2Id", "reviewer": "Reviewer_fF9P", "summary": "This paper proposes to optimize diffusion models directly through forward process, opposed to prior works that will discretize the reverse process. The proposed algorithm yields a supervised learning objective for diffusion models fine-tuning as an off-policy RL paradigm, achieveing comparable performance to mainstream online RL based methods while using much less compute.", "questions": "1. The definition of reward functions in Line 156 is non standard. Why adopting such kind of reward and its optimality probability, opposed to the standard human satisfactory about the generations as an elo rating in existing works like DDPO and FlowGRPO? Specifically most existing reward models are also trained based on elo ratings, the current setup is a bit confusing. In addition to the pratical transformation in Section 3.3, ss there any theorectical equivalence or connection of the optimal solutions with the standard setup? \n\n2. Why the forward consistency is ensured? There is no direct proof on this part. Especially I do not see the straightfoward difference between Diffusion-NFT and RL based method in ensuring such, since $x_t$ sampled from adding noise to $x_0$ is not from the true trajectory that samples $x_0$ in Diffusion-NFT. It will be beneficial to see experiments to showcase the possible distriution shift of RL based methods compared to Diffusion-NFT.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The general structure of this paper is clear. The introductions and preliminaries are clean and instinct. The paper explains clearly about the motivations, existing methods' limitations and the advantages of proposed methods.\n\n2. There are several insightful theorems proposed that characterizes the policy improvement direction and a tractable supervised learning objective for policy optimization. \n\n3. The experimental results are quite impressive in terms of achieveing comparable performance to the RL based counterparts like Flow-GRPO while using much less compute.", "weaknesses": "1. There lacks important baselines for comparisons, e.g. iterative DPO based methods for diffusion models, which also proposed supervised learning based objectives sampled from on-policy data. Overall the Diffusion-NFT objectives are quite similar, more like some weighted version of iterative Diffusion KTO, thus it will be beneficial to discuss the contributions of this paper over these baseline methods including the direct comparison of performance.\n\n2. There lacks more recent models performance like FLUX and Video Generative Models, the reported results are only limited to Stable Diffusion Medium. Given the claimed efficiency of the proposed method, it seems practical to include the results of these other models.\n\n3. Some presentations and clarity of the paper still needs to be improved. There are many missing explanations of the equation elements, for example:\n* $o$ in Line 156 and $p_{\\pi_{\\text{old}}}$ in Line 161 are not explained. In addition, why introducing the two concepts of positive dataset and negative data in the online RL part, which seems irrelevant?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rBXW62Kjat", "reviewer": "Reviewer_niMD", "summary": "This paper introduces DiffusionNFT, a new RL paradigm that optimizes diffusion models directly on the forward process w.r.t. a reward function. It contrasts the positive and negative generations to define an implicit policy improvement direction. DiffusionNFT is compatible with any black-box ODE solvers and does not need to estimate the likelihood. It demonstrates significant improvement on SD3.5.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. this paper reduces RL-based fine-tuning to supervised learning, which could be more stable and efficient compared to conventional Policy Gradient methods;\n2. the training process utilizes both positive and negative signals\n3. the empirical results demonstrate significant improvements", "weaknesses": "This paper optimizes the diffusion model on the forward process. While it brings several benefits, the distribution shift issue is not discussed in this paper. The training objective function is an expectation over the whole dataset. But the inference procedure only covers the region of trajectories from the positive samples. As a result, it's not very clear if the negative samples are used efficiently during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CF6ZQPWnOv", "reviewer": "Reviewer_w5yg", "summary": "The paper introduces an online RL framework for optimizing diffusion models via flow matching on the forward process of the diffusion model.", "questions": "See weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents what is, to my knowledge, a novel approach to optimizing diffusion models via the forward process. The theoretical results seem sound, and DiffusionNFT shows drastic improvement in runtime efficiency over FlowGRPO. The approach can work with any solver for sampling, offering some generality/flexibility.", "weaknesses": "The paper compares only to FlowGRPO as the fine-tuning baseline. Other baselines such as [1] are not considered. Additionally, in Section 3.3, the paper discussed CFG-free optimization. How CFG is integrated as an offline form of RL guidance is slightly unclear and could benefit from greater exposition. Is $v^{old}$ updated during this offline RL style procedure? \n\nWhy does DiffusionNFT seem to dominate the Model-Based scores but not Rule-Based?\n\n[1] Xue et al. “DanceGRPO: Unleashing GRPO on Visual Generation”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)