Reinforcing Diffusion Models by Direct Group Preference Optimization
Authors: Yihong Luo
Tianyang Hu
Jing Tang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Models; Reinforcement Learning;
Primary area: reinforcement learning

Abstract: While reinforcement learning methods such as Group Relative Preference Optimization (GRPO) have significantly enhanced Large Language Models, adapting them to diffusion models remains challenging. In particular, GRPO demands a stochastic policy, yet the most cost‑effective diffusion samplers are based on deterministic ODEs. Recent work addresses this issue by using inefficient SDE-based samplers to induce stochasticity, but this reliance on model-agnostic Gaussian noise leads to slow convergence. To resolve this conflict, we propose Direct Group Preference Optimization (DGPO), a new online RL algorithm that dispenses with the policy-gradient framework entirely. DGPO learns directly from group-level preferences, which utilize relative information of samples within groups. This design eliminates the need for inefficient stochastic policies, unlocking the use of efficient deterministic ODE samplers and faster training. Extensive results show that DGPO trains around 20 times faster than existing state-of-the-art methods and achieves superior performance on both in-domain and out-of-domain reward metrics.

Published as a conference paper at ICLR 2026
REINFORCING DIFFUSION MODELS BY DIRECT GROUP PREFERENCE OPTIMIZATION
Yihong Luo!
Tianyang Hu² Jing Ta
1 HKUST 2 CUHK (SZ)
3 HKUST (GZ.)
GenEval Performance
Relative Performance (SD3.5-M
1.30
DGPO (0.97)
GRPO (0.9g)
1.25
Image Quality
Preference Score
DGPO is - 30x Faster than Flow-GRPO
Relative Performance
0.9
GenEval Score
1.20
1.15
SD3.5-M
0.8
Flow-GRFO DGPO (Ours)
DGPO (Ours)
1.05
0.7
1.00
Buseline (6:63)
0.95
0.6
500
1000
1500
2000
Nestbelic DeQA Imglldd PickSe. Unilwai Out-of-Domaln Metrics
Training Time (GPU Hours)
Figure/ 1: Our proposed DGPO shows a near 30 times faster training compared to Flow-GRPO on improving GenEval score (Left Figure). The notable improvement achieved while maintaining strong performance on other out-of-domain metrics (Right Figure).
ABSTRACT
While reinforcement learning methods such as Group Relative Preference Opti- mization (GRPO) have significantly enhanced Large Language Models, adapting them to diffusion models remains challenging. In particular, GRPO demands a stochastic policy, yet the most cost-effective diffusion samplers are based on deter- ministic ODEs. Recent work addresses this issue by using inefficient SDE-based samplers to induce stochasticity, noise leads to slow convergence. To resolve this conflict, propose Direct Group Preference Optimization (DGPO), new online RL algorithm that dispenses with the policy-gradient framework entirely. DGPO learns directly from group-level preferences, which utilize relative information samples within groups. This de- sign eliminates the need for inefficient stochastic policies, unlocking the use of efficient deterministic ODE samplers and faster training. Extensive results show that DGPO trains around 20 times faster than existing state-of-the-art methods and achieves superior performance on both in-domain and out-of-domain reward metrics. Code is available at hettps://gitubte com/ Luo- Yihong/DGPO
this reliance on model-agnostic Gaussian
INTRODUCTION
Reinforcement Learning (RL) has become a cornerstone for the post-training of Large Language Models (LLMs),, significantly enhancing their capabilities (Ziegler et al. 2019 Ouyang et al. 2022
Bai et al. 2022). In particular, methods like Group Relative Policy Optimization (GRPO) (Shao et al. 2024) have demonstrated remarkable success in substantially improving the complex reason- ing abilities of LLMs (DeepSeek-AA) 2025). However, progress in applying RL for post-training diffusion models has lagged considerably behind that of language models, leaving a significant gap in methods for aligning generative models with human preferences and complex quality metrics.
A central obstacle is the mismatch between GRPO's policy gradient-based framework (short for policy framework the following) and the mechanics of diffusion generation. GRPO requires ac- cess to stochastic policy to enable effective training and exploration. This requirement is naturally
"Corresponding Author: Jing Tang

---

Published as a conference paper at ICLR 2026
A photo of sandwich left of bear
photo blue burger the head of red dog
photo of a girl with red dress right of a green cat
À photo of five cats
A photo of four cakes
3.5M
SD 3.5M w/ Flow- GRPO
SD 3.5M w/ DGPO (Ours)
Figure 2: Qualitative comparisons of DGPO against competing methods. can be seen that our proposed DGPO not only accurately follows the instructions, but also keeps strong visual quality. All images are generated by the same initial noise.
met by LLMs, which inherently output a probability distribution over a vocabulary. In contrast, diffusion models predominantly rely deterministic ODE-based samplers strike better balance between sample quality and cost (Song et al. 2020 Luo et al. (202cc), and thus do not naturally provide stochastic policy. To bridge this gap, prior work has resorted to forced adaptation: using stochastic SDE-based sampling to induce conditional Gaussian policy suitable for GRPO's pol- icy framework (Liu al., 2025 Xue al.) 2025). This workaround, however, introduces severe negative consequences: (1) SDE-based rollouts are less efficient than their ODE counterparts and produce lower-quality samples under fixed computational budget (Lu al., 2022a Bao et al,, 2022 Song et al., 2020); (2) The policy's stochasticity comes from model-agnostic Gaussian noise, which provides a weak learning signal and results in slow convergence; and (3) Training is per- formed over the entire sampling trajectory, making each iteration computationally expensive and time-consuming.
We argue that the practical success of GRPO stems less from its policy- golice-gradient formulation, and more from its ability to utilize fine-grained relative preference information within group. Based on the insight, an ideal RL method for diffusion models should be capable of leveraging this powerful group-level information while dispensing with the need for a stochastic policy and its associated negative effects. To this end, we introduce Direct Group Preference Optimization (DGPO), new online RL method tailored to diffusion models. DGPO circumvents the policy-gradient framework entirely, instead optimizing the model by directly learning from the group-level preference between set of "good" samples and set of "bad" samples. Concretely, for each prompt, we generate G samples using efficient bDE-based rollouts, partition them into positive and negative groups, and directly optimize the model by maximizing the likelihood of these group-wise preferences. Conceptually, DGPO can be understood as natural extension of Direct Preference Optimization (DPO) (Wallace et al.) 2024) that incorporates group-wise information, and as diffusion-natiio re- re-imagination GRPO.
This proposed methodology allows to bypass the dependency on stochastic policy, which yields several benefits: (1) Efficient Sampling and Learning: by using high-fidelity ODE samplers, DGPO leams from higher-qualit rollouts, leading to more effective learning. (2) Efficient Con- vergence:
optimization is directly guided by grvup-level preferences rather than inefficient model- agnostic random exploration, leading to faster convergence. (3) Efficient Training: approach
2

---

Published as a conference paper at ICLR 2026
avoids training the entire sampling trajectory, notably reducing the computational cost of each training iteration. Together, these advantages establish DGPO as a highly efficient and powerful online RL algorithm for diffusion models. Our extensive experiments show that DGPO achieves around 20x faster training than prior state-of-the-art Flow-GRPO (Liu et al. 2025), while delivering superior performance both in-domain and out-of-domain metries. Most notably, the challeng- ing GenEval benchmark (Ghosh et al. 2023). DGPO trains nearly 30x faster than Flow-GRPO and boosts the base model's performance from 63% 97% (Fig. []. These compelling results demon- strate DGPO's potential as powerful technique for aligning diffusion models.
PRELIMINARIES
Diffusion Models (DMs) DMs (Sohl- (Sohl-Dickstein et al. 2015 Ho et al. 2020) define forward dif- fusion mechanism that progressively introduces Gaussian noise input data across T sequential timesteps. The forward process follows the distribution q(x, ix) A [(x;; oxx, oi1). where the hy- perparameters a and control the noise scheduling strategy. At each timestep. noisy samples are obtained by X, = axx Jee, where e A( I). The parameterized reversed diffusion process is defined by: Po (x (x,) & N (x+ 1: Ho (x, t). n/ I). The model's neural network fo is learned by denoising Ax Aelff t) x B. We note that the flow matching and DMs are equivalent in the context of diffusing by Gaussian noise (Gao et al.) 2025).
Reward Modeling Given ranked pairs generated from certain conditioning x > xblc, where xi" and x denote the "better" and "worse" samples. The Bradley-Terry (BT) model formulates the preferences as:
pBT(xp xolc) = (r((,, x²) r(c, xh))
(I)
where a--) denotes the sigmoid function. A network rá that models reward can be trained by maximum likelihood as follows:
LoT(Ø) = Exxx x [logo (c((xx())))
r (c,
(2)
RLHF RLHF typically aims to optimize conditional density pe (xo ()) to maximize a underlying reward r(c, xo) while staying close to a reference distribution Prer via KL regularization, i.c.
max (x [r(c, Xo)] - BKL. [po (Peer(())))))))
(3)
where the hyperparameter 3 controls strength of regularization.
GRPO Objective (Shao et al. 2024) The RLHF objective in Eq. 3 can be optimized by policy-
based learning (we omit the KL. term and clip term hereinafter for brevity):
po (x [x,, c)
max ((xo,xi," (x pr) (1))
A A(Xx+1),
(4)
k=0 Peas (xk-1
Xk: c)
where A(x++1) denotes the advantage of X which can be directly computed by reward r(x++1), or introduce additional value model for reducing variance.
The GRPO proposes to sample a group of outputs for each prompt e from the old policy, then compute the advantage of each sample by normalization among groups, i.e., A, = (rr mean(fri; ... ,r., tween X; and X follows stochastic distribution. To meet this requirement for a stochastic policy, recent works (Liu et al.) 2025) Xue et al., 2025) employ a stochastic SDE for sampling, rather than the more efficient deterministic ODE. However, the SDE itself is less effective in sam- pling high-quality samples with insufficient steps. Besides, the policy-based method requires per- forming training on the whole trajectory, which further leads to slow training. More importantly, unlike LLMs, which directly output distribution, the stochasticity in DM's 's policy comes from model- agnostic Gaussian Noise. This makes the stochastic exploration rely on the model -adel-agnostic Gaussian noise, which is extremely inefficient in dige-dimeas space.
.rra)) ()ddd))ss(((rrr))) (2,,,2,.... .r,.)) The policy learning requires that the transition be-
3

---

Published as a conference paper at ICLR 2026
DPO Objective (Rafailov et al. 2024) The unique global optimal density pa of the RLHF objec- tive (Eq. (3) is given by:
(x (xo((X exp (r(c, (Z(c)
where 2(c)= (xo exp (o)(3)))/3) is intractable partition function. We can compute the reward function follows:
(xalc)) + 3 log Z(c)
r(c,x)) B log
(ra)(Xolc)
After obtaining the parameterization of the reward function, the DPO optimizes the models by the reward learning objective in Eq. Q
log a (Blog Pa Px(xx c) Pref (x Prer(xx51
po (xbblc Pret (x) ()(cc)
(O) = = B c, xeb
B log
Diffusion DPO (Wallace et al. 2024 has adapted DPO to Diffusion models by defining reward over the diffusion paths XO:T, which does not require a stochastic policy. However, it strictly relies on pairwise samples for optimization due to the intrinsic restriction of the intractable partition (c)), preventing the use of the fine- fine-grained preference information of each sample.
3
METHOD
We believe that the key to GRPO's 's success lies in its ability to utilize fine-grained relative preference information within groups. However, existing GRPO-style methods (Liu et al. 2025; Xue et al., 2025) require using an inefficient stochastic policy. Although existing DPO-style methods (Wallace) al. 2024) provide framework without the need for a stochastic policy, they require performing training on pairwise samples to eliminate the intractable partition Zcc). To this end, we propose Direct Group Preference Optimization (DGPO), which eliminates the inefficient stochastic policy and allows us to directly optimize inter-group preferences without concerning ourselves with an intractable partition, leveraging fine-grained reward information to significantly improve training efficiency. The pseudo code of DGPO is summarized in Algorithm
Problem Setup Let Peet denote a pre-trained reference diffusion model with parameter 0 We have a dataset of conditions D. { and reward function ra(s-): X x C + R that evaluates the quality of generated samples x E X' given condition с E C. Our goal is to enhance a diffusion model pe. initialized from Pref, according the reward signal. At each training iteration. we use an online model Pa- to generate group of samples conditioned on c E De. where 0 can be set as the current parameters
an exponential moving average (EMA) version of previous A's. These generated samples form dataset 'D {(G, )= c.) X Pa- (.(c,),, which are then evaluated by the reward function to provide reward signals for splitting positive or negative groups.
3.1 DIRECT GROUP PREFERENCE OPTIMIZATION
In order to leverage relative information within groups, we propose directly learn the group-level preferences using the Bradley- Terry model via maximum likelihood:
max Bi -- "-,e)>
log log p(g 9 (c) = Big+ ....----.o "-,e)= log (R (c) - (R (8)
where -+ and g represent positive and negative groups respectively, with g+ U - = G. and
G = {x] x being the complete group of samples for conditioning Intuitively, the objec- tive in Eq. S can leverage fine- fine-grained preference information of each sample within groups with appropriate parameterization.
Therefore, we propose parameterizing the group-level reward as a weighted sum of rewards ro (c) for each sample within the group:
(glc) = Σ w(x)) ro(c, Xp),
X
where w controls the 'importance level" of the sample within the group. The parameterization of group-level reward can reflect the fine-grained information of each sample. And the reward of single
4

---

Published as a conference paper at ICLR 2026
Algorithm Direct Group Preference Optimization (DGPO)
Require: Diffusion model fa. Reference model fiet. Reward model ra. Group size G, Hyperparam- eter 3. Minimum training timestep min. Learning rate Iterations N, EMA decay ja (optional).
Ensure: Optimized model fo.
1: for < to N do
2: # Sample conditioning and generate group
3: Sample conditioning D.
4: Generate group G = {x,"".;; by sampling from po- (-Ic)
5: # Compute advantages
6: (ri} + (ro(c, xb)]
7: A; (r for all (r)
8:
Partition into positive and negative groups
9:
g++ + {x) : A; > 0} and g + {x] A; 0}
10:
# Compute DGPO loss
11:
Sample L U[min. e
12:
x - a,xy + ofE for all /
13:
Compute LDGPO by Eq. 17
14:
Update 0 + V L
15:
Update 0- A or f + 1 (1 1))
16:
end for
sample can be parameterized by following Eq. and Diffusion-DPO Wallace et al. 2024):
log px (c)
ro Xo) HE pe (x0)
+ Bog Z(c).
Pret (at:xxr (c)
(10)
po (x):rllc))
gg( log
3 log Z(c)
Pietl (x.r
where Z(c) is intractable partition function. We note that since sampling from the inversion chain po(x 2o) is expensive, the forward diffusion (xx1:T (x.) has been utilized as an approximation in practice (Wallace et al. 2024).
By combining Eq. and Eq. 88 we can derive the desired training objective:
(x + [(c)
I(O)= -g (gg.gg--aa -o DD g-D log (
Px(xxrlxo)) Bu (xo) [log Prer(Xo:T (c)
xgEg!
(x(:TTlc) + Z(c)l))
Σ
Eg Ex(xi. (bo) (10(())) [log
Prer(Xo:T
Po Po (x (c)
-Eggt. Egggtg-aa D- log 0 (3[Bg(xirree (x))
w(xo) logg
Pret xg.7
Xo Eg
(x c)
[x w (xo) log
w(x))2(c)
w(x)) (xxx))2( Z
Pret (x.o
X
XpEg
(11)
remaining crucial challenge is that the partition function Z(c) is intractable for training. We have to carefully select appropriate weighting w; for each sample eliminate the intractable partition function Z (c). Generally speaking, good weighting strategy should satisfy the following:
Larger weights correspond to better samples in G and worse samples in g-
The weights satisfy: Ex,eg+ ur(x)) = =xx w(xo), such that Exoeg+ w(x))Z(c) Expeg- w(x,) ((x))Z(c)) [cc)) for eliminating the intractable Z(c).
3.2 ADVANTAGE-BASEI WEIGHT DESIGN
We propose using advantage--aseed weights derived from GRPO-style normalization to address the aforementioned issues. Given group G x/_ x3 x with corresponding rewards T1, 72, m. TG.
5

---

Published as a conference paper at ICLR 2026
compute advantages:
A(xh) mean (r;),-1)
(12)
[[((r)]]]]
We then partition the group based on advantages:
g+ {xg A(xx) > 0}, g- {(x) A(xx) s 03.
(13)
And we set weights as:
w(xo) |A(x))]
(14)
This choice ensures E Σxyegi =xx))= Exxes w(xo) due to the zero-mean property of the nor- malized advantages. It also dynamically assigns larger weights to samples that deviate more from the average, which enables the model to more effectively learn relative preference relationships. More importantly, this weighting turns the objective in Eq. to:
Bxox log po (xo.T
I(0) -Ei -- (-)))-Dlogo(a
Σ
Pret (xu:rl))))))
xnEg1
(15)
pe (xO.T(/))
log
Pret (X1.7
XoEG-
By using Jensen's inequality and the convexity of - logg o, we can move the expectation outside:
I(0) S 1g+ 8- Be logo
Σ w(xx), gx- (TT
po (2(X Xe. C) xx) log
Pref x, X,, c)
XoEg+
(16)
Po (x X2, c) Σ /(x. log x-Eg-
Pref
Xt-1 (r-
X1: C)
where -= G We note that reduce the variance, the sampled noise is shared among samples within the same complete groups. By some simplification, we obtain our final training objective of the proposed DGPO:
LDGPO(0) -Risst S .c) logol->,
Σ
w w(x) (x, x1, I. (x, X1, c)]
Xeg+
Σ XEG-
w(x) Xx,
xe. c) dom (x, Xx, c)])),
(17)
where T.G (x, Xx, c) [) (x), ()) T. dsm (x, Xx, [1 (x) A is weighting function and the constant 7 can be factored into B. We defer the derivation from Eq. (55) to Eq. (17) the Appendix C The advantage of the derived DGPO objective is fourfold: 1) Leverages Relative information: It directly learns preferences between groups of samples, which leverages the fine- grained relative preference information of individual samples within groups. 2) Enhances training efficiency: It does not require training on the entire sampling trajectory, which notably reduces the computational cost per iteration. 3) Enables effective learning: sidesteps the need for an inefficient stochastic policy, thus avoiding inefficient model-agnostic exploration and allowing the model to learn more effectively and directly from the preference data. 4) Efficient Sampling and Learning: It allows the usage of deterministic ODE sampling for rollouts. This yields higher-quality training samples compared to inefficient SDE sampling, all while using the same inference budget.
Timestep Clip Strategy The considered online setting requires generating samples from the on- line model which might be expensive; thus, take few steps (c.g., 10) for generating samples to reduce the inference cost following Flow-GRPO (Liu et al. 2025). However, naively performing DGPO's training these samples generated by few steps would lead serious performance degra- dation due to the poor sample quality. To mitigate this, we propose the simple yet effective Timestep Clip Strategy: during training, we only sample timesteps from the range [kmin: 2'] with chosen min- imum timestep bmin > 0. This could effectively prevent the model from overfitting specific artifacts (e.g., blurriness) of the generated samples by few steps (see ablation in Fig.

---

Published as a conference paper at ICLR 2026
Table 1: GenEval Result. We highlight the best scores.
Obj.: Object; Attr.: Attribution.
Model
Overall
Single Obj.
Two Obj.
Counting
Colors
Position
Attr. Binding
Autoregressive Models:
Show-o Xie
2024
0.53
0.95
0.52
0.49
0.82
0.11
0.28
Emu3-Gen Wanng el 2024g
0.54
0.98
0.71
0.34
0.81
0.17
0.21
JanusFlow (Ma il. 2025
0.63
0.97
0.59
0.45
0.83
0.53
0.42
Janus-Pro-77 (Chen
0.79
0.66
GPT-4o Hursteraa
0.84
0.99
0.92
0.85
0.92
0.75
0.61
Diffusion Models:
LDM (Rombach al.
2022
0.37
0.92
0.29
0.23
0.70
0.02
0.05
SD1.
Rombach et al.
2022
0.43
0.97
0,38
0.35
0.76
0.04
0.06
SD2.1
(Rumbach al.
2027
0,98
0.51
0.44
0.07
0.17
SD-XL
(Podell
2016
0.55
0,98
0.74
0.39
0.85
0.15
0.23
DALLE
OpenAT
2074
0.52
0.94
0.66
0.49
0.77
0,10
0.19
DALLE-3
Belker et 2023
0.67
0.96
0.87
0.47
0.83
0.43
0.45
Lahg
0.22
0.45
SD3.5-L
0.71
0.98
0.89
0.73
0.83
0.34
0.47
SANA-1 1,5 4,88 [Xie ot 2025
0.81
0.99
0.93
0.86
0.84
0,59
0.65
SD3.5-M [Eserr et 2024
0.63
0.98
0.78
0.50
0.81
0.24
0.52
w/ Flow-GKPU et al.
2025
0.99
0.95
0.92
0.99
SD3.5-M w/ DGPO (Ours)
0.97
1.00
0.99
0.97
0.95
0.99
0.91
Table 2: Performance on Compositional Image Generation, Visual Text Rendering, and Hu- man Preference benchmarks. ImgRwd: ImwgeReward; WniRwd: UnifiedReward.
Metric
Image Quality
Preference Score
Model
GenEval
OCR Acc.
PickScore
Aesthetic
DeQA
ImgRwd
PickScore
UniRwd
SD3.5-M
0.63
0.59
21.72
5.39
4.07
0.87
22.34
3.33
Compositional Image Generation:
Flow-GRPO
0.95
5.25
4.01
1.03
22.37
3,51
DGPO (Ours)
0.97
5.31
4.03
1.08
22.41
3.60
Visual Text Rendering:
Flow-GRPO DGPO (Ours)
0.92
5.32
0.95
22.44
3.42
0.96
5.37
4.09
1.02
22.52
3.48
Human Preference Alignment:
Flow-GRD CRW-GRPO
23.31
5.92
4.22
1.28
23.53
3.66
DGPO (Ours)
23.89
6.08
4.40
1.32
23.91
3.74
EXPERIMENTS
In this section, we comprehensively evaluate the proposed DGPO. Specifically, we benchmark im- provements three tasks dasks-comoostiional image generation, visual text rendering, and human pref- erence alignment Tabllssaand We also present qualitative comparisons and training efficiency (Figs. and 3). We further conduct ablations key components (Figs. and
4.1 EXPERIMENTAL SETUP
We evaluate the DGPO on post-training the SD3.5-M (Esser et al. 2024) across three distinct valuable tasks: 1) compositional image generation, using GenEval ing, spatial relations, and attribute binding; 2) visual text rendering (Gong et al.) 2025), measuring accuracy of rendering text in generated images, and 3) human preference alignment, using PickScore to assess visual quality and text-image alignment. Details are provided in the Section
Evaluation Tasks
object count-
Out-of-Domain Evaluation Metrics To fairly evaluate model performance and guard against re- ward hacking-where models may overfit to training rewards signal while compromising actual image quality-wa we employ four independent image quality metrics not used during training as out- of-domain evaluations: Aesthetic Score (Schuhmann et al.) 2022) DeQA (You et al., 2025). Im- ageReward (Xu et al. (202). and UnifiedReward (Wang et al. 2005)) We compute these metrics on DrawBench (Saharia al. 2022). comprehensive benchmark featuring diverse prompts.

---

Published as a conference paper at ICLR 2026
(a) OCR Accuracy Comparison
(b) Pick Score Comparison
0.5
23.5
0.5
23.0
Paster
Plck
0.75
Flow-CIRP DGPD (Ours)
22.0
(Oun)
25
30
75
100
125
150
175
200
21.53
50
160
150
200
250
300
Training Time (Hours)
Training Time (Houss)
Figure 3: Compare the training speed of Flow-GRPO and proposed DGPO.
dragon with Subtities "Ultra- Fast Training"
A panda holds the sign says "Timestep Clip Matters"
dragon with Subtitles "Uitra- Fast Training"
panda holds the says "Timestep Clip Matters"
A coffee shop with sign "DGPO"
A coffee shop with sign "DSPO"
DGPO
DGPO
Timestep
Timestop Clip Matters
Clip
Ultra-Fasd Fast Trainhg.
Ultra-Fasd Training
DGPO Timestep Clip Strategy
DGPO (Ours)
Figure 4: Visual comparisons among variants. It can be seen that without the proposed timestep clip strategy, although it can still accurately follow the instruction, the visual quality notably degrades
4.2 MAIN RESULTS
Quantitative Results Table [ shows that DGPO achieves state-of-the-art performance on GenEval, notably surpassing prior SOTA methods such GPT-4o and Flow-GRPO. This improve- ment is achieved while maintaining performance across various out-of-domant metrics (such as AeS, DeQA, and Image Reward), indicated by Table Beyond compositional image generation, provides detailed evaluation results on visual text rendering and human preference tasks, where DGPO similarly demonstrates significant improvements the target optimization metrics while maintaining performance across various out-of-domain metrics.
Table 2
Qualitative Comparison We present the qualitative comparisons of methods trained with GenEval's 'ss signal in Fig. It is clear that the proposed DGPO can follow the instructions more accurately compared to the base diffusion model and also the Flow-GRPO. Although Flow-GRPO also shows accurate instruction following, its image quality degrades seriously, while our method shows notably better visual quality. We present additional visual samples in Appendix
Training Cost The overall training of the proposed DGPO is quick, since we do not require the inefficient stochastic policy for training. Besides, the training the DGPO is efficient per iteration, since we do not perform training on the whole trajectory. Benefit from these points, the overall training of DGPO for reinforcement post-training much faster than prior SOTA Flow-GRPO, As the overall training of DGPO is generally around 20 x faster than Flow-
shown in Figs. and 3 GRPO.
4.3 ABLATION STUDY
Effeet of Timestep Clip Strategy We found that without the proposed timestep clip strategy, the reward metrie slightly degrades from 0.96 to 0.95 regarding OCR Accuracy, while the visual quality seriously degrades as shown in Fig.
ODE Rollout vs. SDE Rollout core advantage of our work compared to prior GRPO-style works (Liu et al. (025) is the ability to the efficient ODE solvers for generating samples. This can deliver samples with better quality and rewards. Results in Fig Flshow that ODE rollout notably outperforms SDE rollout in both convergence speed and ultimate metrics. This suggests that the use
8

---

Published as a conference paper at ICLR 2026
(a) Offinee Methods
(b) Online Methods
1.0-
Eval Accuracy
0.65
OCk Eval Accuracy
0.50
0.70
0.65
Online (Durs)
Offine DGPD (Qurs)
DGPO (Ound
0.58
200
400
Rico
800
1000
200
400
600
1000
Training Steps
Training Steps
Figure 5: Comparison of visual text rendering across variants.
of SDE in prior works may have been a requirement of the policy gradient framework rather than providing more diverse samples for training.
Our work can be easily adapted to the offline setting by using the reference model Prer for generating the training dataset. Results in Fig. Fig.55 show that our offline DGPO can reasonably boost the performance over the baseline, but its performance is notably worse than the online setting.
Offline DGPO
Compared to Diffusion DPO Diffusion DPO can also avoid the need for the stochastic policy, however, it cannot leverage the fine-grained reward signals of each sample. Results in Fig. show that our DGPO notably outperforms the DPO in both online and offline settings, indicating the effectiveness of our proposed DGPO in leveraging fine-gras ine-grained group relative information.
RELATED WORKS
Recent research has focused extensively aligning DMs with human preferences through three primary approaches. The first involves fine-tuning diffusion models on carefully curated image- prompt datasets (Dai al.) 2023 Podell et al. (023). The second approach maximizes explicit
reward functions, either by evaluating multi-step diffusion generation outputs (Prabhudesai et al.)
2023 Clark al., 2023 Lee et al.
2025a b) or through policy 2024 The third category gradient-based learning (Fan et al. 2024 Black et al. 2023] Ye et al.
2023 Ho al. 2022 Luo et al.
employs implicit reward maximization, as demonstrated by Diffusion- DPO (Wallace al.) 2024)
and Diffusion-KTO (Yang et al. 2024). which directly leverage raw preference data. A concur- rent work (Chen al. (2025a) explores utilizing the group information in DPO by enumerating all pairwise comparisons within the group. In contrast, our work defines single group-level re- learning on that group-level reward. Recent works have also adapted GRPO to DMs (Liu et al., 2025 Xue et al. 2025) under policy-gradient framework, demonstrating promising scalability and impressive performance improvements. How- ever, a notable drawback of existing GRPO-style approaches is their reliance stochastic policy, which requires inefficient SDE-based rollouts during training. Our work identifies group relative information as the critical component of GRPO and introduces DGPO directly optimize group preferences, thereby exploiting fine-grainec group relative information without requiring stochastic policies. As result, DGPO achieves significantly faster training and superior performance on both in-domain and out-of-domain reward benchmarks compared to prior GRPO-style methods.
ward and reinforces the DMs with maximum-likelibon
6 CONCLUSION
In this work, we introduce Direct Group Preference Optimization (DGPO), a novel online rein- forcement learning method specifically designed for post-training diffusion models. Our approach addresses the fundamental mismatch between policy gradient methods like GRPO and the inher- ent mechanies of diffusion generation. By recognizing that GRPO's effectiveness stems primarily from its utilization of group relative preference information within the group rather than its policy- gradient nature, we developed a method that preserves this key strength while eliminating the need for stochastic policies. DGPO's direct optimization approach offers substantial practical advan- tages over existing methods. By enabling the use of efficient ODE-based samplers, eliminating reliance on model-agnostic noise for exploration, and avoiding expensive trajectory- based training, DGPO achieves around 20x speedup in overall training time compared to Flow-GRPO. More im- portantly, experiments demonstrate that this efficiency gain comes with superior performance, as

---

Published as a conference paper at ICLR 2026
DGPO consistently outperforms baseline methods across both in-domain and out-of-domain evalu- ation metrics.
ETHICS STATEMENT
This work did not involve human or animal subjects, sensitive data, or any other elements that would necessitate ethical review. We have identified potential for misuse negative societal impact.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, complete code and experimental setup, which builds upon the open-source Flow-GRPO codebase eLi al. 2025), will be made publicly available
ACKNOWLEDGMENTS
Jing Tang's work is partially supported by National Key R&D Program of China under Grant No. 2024YFA 1012700, by the National Natural Science Foundation of China (NSFC) under Grant No. 62402410, and by Guangdong Provincial Project (No. 2023QN10X025).
REFERENCES
Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wrXiv:220400882, 2022.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang, Analytic-pmm: analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint arXiv:2201.06503 2022.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang. Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall-e- pdf. 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion based maximum entropy reinforcement learning. arXiv preprint arXiv:2502.02316, 2025.
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters. Advances in Neural Information Processing Systems, 36:9353-
Renjic Chen, Wenfeng Lin, Yichen Zhang, Jiangchuan Wei, Boyuan Liu, Chao Feng, Jiao Ran, and Mingyu Guo. Towards self-improvemmnt of diffusion models via group preference optimization, 2025a. Uttttts::///axx
2005..225551111 11070
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint arXiv:2501.17811 2025b.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309, /7400, 2023.
Xiaoliang Dai. Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang. Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint 2oo.:2230..8817 2023.
10

---

Published as a conference paper at ICLR 2026
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. hRRtstts:///att
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL https::/arxiv.orr///////
2403 03206
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin Patrick Murphy, and Tim Salimans. Diffusion models and gaussian flow matching: Two sides of the same coin. In The Fourth Blogpost Track at ICLR 2025, 2025. Cd=C8Yyg9
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating ext-to-image alignment. Advances Neural Information Processing Systems, 36: 52132-52152, 2023.
Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, et al. Seedream 2.0: native chinese-english bilingual image generation foundation model. arXiv preprint arXiv:2503.07703, 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in 2020.
Neural Information Processing Systems,
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300. 2022.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image image generation. Advances in neural information processing systems, 36:36652-3666666 2023.
Black Forest Labs. Flux.
forest Labs/Etaab 2024.
Kimin Lec, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.11 12122, 2023.
Jie Liu. Gongye Liu, Jiajun Liang Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXivv2505.00.: 05470, 2025.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022a
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX0v:2211.01095, 2022b.
Yihong Luo, Tianyang Hu, Weijian Luo, Kenji Kawaguchi, and Jing Tang. Reward- instruct: reward-centrie approach to fast photo-realistic image generation. arXiv preprint arXiv:2503. 13070, 2025a.
11

---

Published as a conference paper at ICLR 2026
Yihong Luo, Tianyang Hu, Yifan Song, Jiacheng Sun, Zhenguo Li, and Jing Tang. Adding additional control to one-step diffusion with joint distribution matching. arXiv preprint arXiv:2503.06632 2025b.
Yihong Luo, Tianyang Hu, Jiacheng Sun, Yujun Cai, and Jing Tang. Learning few-step diffusion models by trajectory distribution matching. arXiv preprint arXiv:2503. 06674, 2025c.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xic, Haowei Zhang, Xingkai Yu, et al. Janusflow: Harmonizing autoregression and rec- tified flow for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7739-7751, 2025.
OpenAl. Dalle- 2023. URL httLttttt/pppeaaiii www.coooomaaalooooooa htse/dallleee
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint (rXiv:2310.03739, 2023.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. arXiv preprint 1xx.::23/2.111/: 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024.
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409. 00588, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition, pp. 10684-10695 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 35:36479-36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li. Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint 03Xv:0240..03300, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynanics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
12

---

Published as a conference paper at ICLR 2026
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 9228-8236 2024.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409, 18869, 2024a.
Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multi- modal understanding and generation. arXiv preprint arXiv:2503.05236, 2025.
Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-54206 2024b.
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of training-time and inference- time compute in linear diffusion transformer. arXiv preprint FrXXiv250011842 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408, 12528, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 2023.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.0785
2025.
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024.
Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for rein- forcement learning. arXiv preprint arXiv:2305. 13 122, 2023.
Zilyu Ye, Zhiyang Chen, Tiancheng Li, Zemin Huang, Weijian Luo, and Guo-Jun Qi. Schedule on the fly: Diffusion time prediction for faster and better image generation. arXiv preprint arXiv:2412.01243. 2024.
Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress accurate image quality scores using score distribution. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14483-14494, 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint XwX.::v119900593.33 2019.
A USE OF LARGE LANGUAGE MODELS
The use of large language models (LLMs) was strictly limited to language refinement and minor editorial tasks. The authors affirm that LLMs played no part in the substantive phases of the research, which include the ideation, experimental design. data analysis, and the interpretation of results. All scientific content, methodologies, and conclusions presented herein were conceived and developed exclusively by the authors.
13

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RELATED WORKS
Diffusion-Basee Policies in RL. Diffusion policies have gained significant traction in recent on- line RL research. Existing methods have investigated optimizing these policies via different ap- proaches. The first uses reparameterized policy gradients for optimization (Wang et al. 2024b Ren et al., 2024, Celik et al., 2025); The second explores optimizing via variants of score matching. There are also works exploring optimizing via weighted schemes (Psenka al., 2023] Yang et al,, 2023). These works focus using diffusion policy for action learning. In contrast, work plores reinforcing diffusion models more ceffusiion-native way for achieving better performance in the image generation domain.
C DERIVATION
C.I PRELIMINARY: JENSEN'S INEQUALITY.
Let Φ : R + R be a convex function. For a random variable X, Jensen's inequality states that the function of the expectation is less than equal to the expectation of the function:
<<E[X] < E[o(X)).
(18)
C.2 DERIVATION OF EQ. 16
For simplicity and without loss of generality, we omit the condition in the derivation. The Eq. can be rewritten as:
15
Po (XO:T)
((0) = ig
((gttg-)~0 --)~D logo
Σ
Eg(x (x0) (x6) log Prer (Xo:T)
NoE
w(Xo) Eytrr xo) log Po (xu:2)
Σ
Pret (Xo:T)
X.Eg-
po ( Piet(x; (2)
Eg & --) log (3E (x
Σ
w(xx)
Σ
log
XIE
1=1
T
Po (x (xt)
Σ wxxo)
log
Pret x xx,
x
Po (x{)
- (gt & g-D log (x)
Σ
T(.))TE, log
Pret (x,-1 Xx)
NaEgi
w(xo) ),E, log Po Po(x1-11x0 Pret X1_ X2
x0
w(xo) log po 1 Pies (x Xx))
= E - log ( BTE, (xx) 1oo))
Σ
E
w(xo) log (x+ (xx)
Pret Xx-1 xx,)
(19)
By Jensen's inequality (Section C.I and the convexity of - -log d, we have:
Σ w(xo) BTTE (x) log Po Po (x()
L(0)= < -E -0--~D
JNDEr, No) logot o
Pietf (xt xx))
x
(20)
w(xx) F (x. log Po Px(xx-11 1xx))
Pret ( (x+)
14

---

Published as a conference paper at ICLR 2026
C.3 DERIVATION OF EQ.
For simplicity and without loss of generality, we omit the condition с in the derivation. The DGPO objective in Eq. T6 be rewritten
1 x1x. log P Pret (
L(0)= v Eig+ (--+,--)-00 a(x logo
Σ
w(xx)
x
log po (X:- (xx)
Σ w(xx)) Byx xox)xx)
Pret (x+ (xx)
XpEG-
--Bigt gr log (-331
w(xo)- BKkko///KKK/sss/..x/////loo-//b
X
Σ w(xo)
X.EG
(21)
With the Gaussian parameterization
Song et al. 2020) of the posterior ((xx-1|x;, Xo)
N(a:-1X0 + VoL-1 1/2 - m I) and reverse sampling (x(x(-1|x;)
(xxx-1 / oo 1/-1 fo(x.)) 1), the KI. divergence de (xx-1 1(x))) can be computed by:
term
KI(g(Xt-11 (xl, x0),
KL(g(x--1\x,) X1, (x-
||0-1800 + V 0o6 . X1 -4x0 (ax= + oo- -
- nh X; at (x t)
21/211
at )
a4-- 0001 - n
||x0 tx(x,,t))l
2n/21
01
01-1 - 1 of 1/ 04 2 20/21
1
=>el1x0 - fo( t)l where de =
01
(22)
Similarly, have (L(g(x;-1 X2, xo) P (x.)
du X1l1x00 (x1, t)112.
Substituting the
computed KL divergence into Eq. 21 we can obtain Eq.
D VISUALIZATION OF REWARD HACKING
We set 8 to be smaller (e.g., = 10) than its normal value (c.g., 8 100) and extended the training iterations make the model over-optimize the rewards. We visualize some failure modes of overooptg rewards Section D
E EXPERIMENT DETAILS
Compositional Image Generation. We evaluate text-to-image models on complex compositional prompts using GenEval (Ghosh al. 2023), which tests six challenging compositional generation tasks including object counting, spatial relations, and attribute binding.
Visual Text Rendering. Following the methodology in TextDiffuser (Chen et al. 2023 and Flow- GRPO's experimental setup, we evaluate models' ability to accurately render text within generated images. Each prompt follows the template structure "A sign that says 'text"", where 'text' represents the exact string to be rendered in the image. We measure text fidelity (Gong al. 2025
follows:
r= max - N Ne/Ner: 0)
where N₂ denotes the minimum edit distance between rendered and target text, and Niur represents the character count within the prompt's quotation marks.
15

---

Published as a conference paper at ICLR 2026
w/ OCR Acc
w/ HPS
Over optimized DGPO
Over-optimize Test
Over- optimize Test
OVER OPTIMIZE
TEST
DGPO
Test
Over optimize
Test
Test
optimize
DGPO
DOPO
DGPO!
A photo of panda holding sign says Over optimize Test
photo of panda halding sign says 'DSPO in forest
Visualization of reward hacking- Over-optimizing the rule-based reward (i.e., OCR Acc) preserves text accuracy but degrades image quality. In contrast, over-optimizing the model-based reward (i.e., HPS) introduces specific artifacts, such repeated objects in the background.
Human Preference Alignment. To align text-to-image models with human preferences, we em- play PickScore (Kirstain al. (023) as the reward signal. The PickScore model, trained large- scale human preference data, evaluates both visual quality and text-image alignment, providing a comprehensive assessment of generation quality from human-centrie perspective.
Setup Details. We generate 24 samples for each group for training. We adopt the Flow-DPM- Solver (Xie et al., (025) with steps of 10 for rollout during training. We adopt LoRA fine-tuning with rank of 32. The 3 is set to be 100 by default. Defaultly, We update 0 by identity mapping, i.e. 0= + 0 within 200 steps, and update 0- by EMA with decay 0.3 in the remaining training. Experiments performed over 512 resolution. We training. The experiments are performed over A100. The reported GPU hours are A100 hours.
probability of 0.05 to drop text during
Details of the out-of-domain evaluation metries We outline the specific out-of-domain metrics used to assess quality: The aesthetic score (Schuhmann et al. [2022) employs a linear regres- sion model based on CLIP evaluate the visual appeal of generated images; For assessing image quality degradation, we utilize the DeCA score (You et al. (025). This metric leverages a multi- modal large language model architecture measure the impact of various imperfections-inee includding distortions, textural degradation, and low-level visual artifacts-on on the overall perceived quality of images; ImageReward (Xu et al. 2023) serves as comprehensive human preference model for text-to-image generation tasks. This reward function evaluates multiple dimensions including the coherence between textual descriptions and visual content, the fidelity generated visuals; Finally, Uni waiffedReewr (Wang al. 2025) represents the latest advancement in this area. This inte- grated reward framework can evaluate both multimodal understanding and generation tasks, and has demonstrated superior performance compared to existing methods the human preference assess- ment leaderboard.
F ADDITIONAL QUALITATIVE COMPARISON
We present additional visual samples in Figs. 6 and
G LIMITATIONS AND FUTURE WORKS
Our work focuses on text-to-image synthesis; however, also has the potential to be adapted to enhance text-to-video synthesis. Exploring the extension would be interesting future work.
16

---

Published as a conference paper at ICLR 2026
A vibrant urban alley graffiti wall prominently spray-painted "Street Art Rules"
A photo of a coffee shop with sign "OCR Test"
A photo of a book with title "Diffusion RL"
photo of cat holds a sign "DGPO SOTA"
photo of worn road sign "Danger"
DANGERR
Diffusion RL
SD 3.5M
TUTES
DGPO SOTA
SD 3.5M w/ Flow- GRPO
OCR Test
Difusion RL
DANGER
Street Art Rules
DGPO SOTA
OCR Test
Diffusion RL
SD 3, A. w/ DGPO (Ours)
DANGER
Street Art Rules
DGPO SOTA
Figure 6: Qualitative comparisons DGPO against competing methods. The training signal is
given by OCR Accuracy. All images generated by the same initial noise.
cute baby playing with toys in the
photo Asia girl with sungloss
small cactus with happy face in the Sahara desert
A photo of woman on top of horse
A photo of monkey making latte
3,5M
SD 3.5M w/ Flow- GRPO
SD 3.5M w/ D&PO (Ours)
Figure 7:
Qualitative comparisons DGPO against competing methods. The training signal is given by PickScore. All images are generated by the same initial noise.
17


Reviews:
[{"review_id": "LcaY1bJJE2", "reviewer": "Reviewer_3i9L", "summary": "This paper introduces Direct Group Preference Optimization (DGPO), a reinforcement learning algorithm to fine-tune diffusion models more efficiently. Current methods are slow because they require inefficient stochastic samplers. DGPO avoids this requirement by learning directly from group-level preferences, which allows the use of efficient deterministic ODE samplers. As a result, DGPO is shown to train up to 30 times faster than existing methods like Flow-GRPO while achieving superior performance on compositional generation, text rendering, and human preference alignment tasks.", "questions": "- In \"Timestep Clip Strategy,\" a minimum timestep $t_{\\min}$ is used during training to avoid overfitting to artifacts from few-step generation. How sensitive is the final model's performance to the choice of $t_{\\min}$? Is there a principled way to select this value, or is it found empirically through hyperparameter tuning?\n- How do you cope with Classifier-Free Guidance (CFG) in training and sampling? As I understand, the diffusion loss cannot be applied to the CFG model directly. For example, [1] claims that they abandon CFG entirely during both training and sampling, while still achieving strong performance.\n\n[1] DiffusionNFT: Online Diffusion Reinforcement with Forward Process", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Significant Efficiency Gains. The most compelling advantage of DGPO is its dramatic improvement in training efficiency. The claim of being ~20-30 times faster than Flow-GRPO is well-supported by the experiments shown in Figures 1 and 3. This is a substantial practical contribution that could make reinforcement learning-based fine-tuning of diffusion models much more accessible.\n- Strong Empirical Performance. DGPO not only trains faster but also achieves state-of-the-art results. It surpasses the performance of the baseline SD3.5-M model and the strong Flow-GRPO competitor on the challenging GenEval benchmark, boosting the score from 63% to 97% (Table 1). The method also shows consistent improvements across visual text rendering and human preference alignment tasks (Table 2).\n- Well-Motivated. The paper clearly identifies a key limitation in applying GRPO-style methods to diffusion models (the reliance on inefficient stochastic SDE samplers).\n- Comprehensive Experimental Evaluation. The authors provide a thorough evaluation, including qualitative and quantitative comparisons on multiple benchmarks, ablation studies that validate key design choices like the \"Timestep Clip Strategy\" and the use of ODE rollouts over SDE rollouts, and evaluation on a range of out-of-domain metrics to check for reward hacking.", "weaknesses": "- Incremental Novelty. The core idea can be seen as a combination of two existing lines of work: Direct Preference Optimization (DPO) and Group Relative Preference Optimization (GRPO). The main novelty lies in successfully extending DPO to a group-wise setting for diffusion models, but it does not introduce a fundamentally new optimization paradigm. The resulting objective still relies on approximations like Jensen's inequality, making it less theoretically principled and elegant.\n- Limited Discussion on Reward Quality. The success of any RL-based fine-tuning heavily relies on the quality of the reward function. The paper uses established reward models like GenEval, OCR accuracy, and PickScore. However, a brief discussion on the potential failure modes or biases of these reward models and how they might affect DGPO's output would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zyLrICHc0J", "reviewer": "Reviewer_Tf9g", "summary": "The paper presents an online reinforcement learning method (DGPO), by combining GRPO and DPO. It utilizes the fine-grained preference information from GRPO, but uses ODE-based samplers for efficient training. Once preference groups are constructed, DGPO directly learns from them. Experiments show that DGPO is 20x faster in training, compared with existing methods, and can achieve better performance.", "questions": "- How to choose group size, and how does it affect the performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- DGPO utilizes group-level preference information.\n- DGPO uses ODE-based samplers and therefore trains faster.\n- DGPO outperforms Flow-GRPO.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mPn4tX1lY4", "reviewer": "Reviewer_31Y4", "summary": "The paper presents DGPO, a clear, diffusion-native alternative to GRPO that removes the stochastic-policy requirement while exploiting group-relative preferences.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed DGPO achieves significant training efficiency improvements and superior performance on benchmark metrics.\n2. DGPO integrates the idea of group advantage estimation from GRPO into DPO, thereby avoiding the need for stochastic-policy and eliminating the intractable partition function through an advantage-based weight design.", "weaknesses": "1. The mathematical transition from the intermediate objective $\\mathcal{L}(\\theta)$ (Eq. 15) to the final DGPO training objective $\\mathcal{L}_{DGPO}(\\theta)$ (Eq. 17) is not explicitly detailed, as the steps involving Jensen's inequality and the convexity of $-\\log \\sigma$ are only summarized. Try to clarify it.\n2. The definition of the term $\\log \\frac{p _ {\\theta}(x _ {t-1}|x _ t,c)}{p _ {ref}(x _ {t-1}|x _ t,c)}$ in $\\mathcal{L} _ {DGPO}(\\theta)$ is related to the difference in the DSM losses, $L_{dsm}^{\\theta}-L_{dsm}^{\\theta_{ref}}$ (Eq. 17), but the exact derivation of this equivalence is omitted.\n3. The hyperparameter $\\beta$ is a critical part of the loss function (Eq. 15, Eq. 17), but its specific value for the main experiments is not provided in the Setup Details, and no ablation study is conducted on its effect, making it difficult to assess the stability and sensitivity of the method to this parameter.\n4. The choice of $G=24$ samples per group is mentioned, but the impact of the group size $G$ on the training speed, final performance, and the robustness of the advantage normalization is not investigated.\n5. DiffusionNFT [1] also addresses the same problem of Flow-GRPO and has publicly released its source code. It would be beneficial to include a comparison with this method and incorporate the evaluation metrics mentioned in their work.\n6. Add related works on diffusion models integrated with online off-policy reinforcement learning, such as QSM [2], DACER [3], and DIME [4], to provide a more comprehensive discussion of recent advancements in this area.\n\n[1] Zheng, Kaiwen, et al. \"Diffusionnft: Online diffusion reinforcement with forward process.\" arXiv preprint arXiv:2509.16117 (2025).\n\n[2] Psenka, Michael, et al. \"Learning a diffusion model policy from rewards via q-score matching.\" arXiv preprint arXiv:2312.11752 (2023).\n\n[3] Wang Y, Wang L, Jiang Y, et al. Diffusion actor-critic with entropy regulator[J]. Advances in Neural Information Processing Systems, 2024, 37: 54183-54204.\n\n[4] Celik, Onur, et al. \"Dime: Diffusion-based maximum entropy reinforcement learning.\" arXiv preprint arXiv:2502.02316 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7rQkvsnvb7", "reviewer": "Reviewer_rxkG", "summary": "- This paper proposes DGPO, a method that brings GRPO’s within-prompt, group-relative feedback into diffusion alignment, addressing the technical mismatch that has made GRPO difficult for diffusion models.\n- The approach distills GRPO’s group information into a tractable objective inspired by diffusion-DPO’s density-ratio formulation, avoiding stochastic policy gradients and full-trajectory training.\n- Empirically, DGPO achieves strong preference alignment and image quality compared to existing methods, while remaining computationally efficient.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "questions": "- In Equation (17), the DGPO loss contains two separate summations over \\(x_0 \\in \\mathcal{G}^+\\) and \\(x_0 \\in \\mathcal{G}^-\\).  \n  Can these be rewritten as a single summation, such as  \n  $\\sum_{x_0 \\in \\mathcal{G}} A(x_0) (L^{\\theta}-L^{\\theta_{\\mathrm{ref}}})$,  \n  by using Equation (14)?\n\n- How would you incorporate PPO-style clipping into DGPO? The original GRPO introduced PPO clipping [1].\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a novel online RL method for diffusion alignment that efficiently leverages rich group-level information.\n- They reformulate the core idea of GRPO—relative, within-prompt group information—and make it applicable to diffusion models using a fast ODE sampler.\n- Their DGPO achieves higher benchmark scores than GPT-4o and Flow-GRPO.\n- Experiments show DGPO is roughly 15–20× faster than Flow-GRPO at comparable scores.\n- They also examine the effectiveness of timestep clipping, ODE rollouts, and the online setting.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "weaknesses": "- My concern is that the authors’ formulation relies on the Bradley–Terry model, which might alter core characteristics of the original GRPO strategy.\n- The formulation explicitly partitions each within-prompt group into \"good\" and \"bad\" subsets—something the original GRPO did not do—which may change the objective’s behavior compared with the original approach (see my question).\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)