DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment
Authors: Haoyou Deng
Keyu Yan
Chaojie Mao
Xiang Wang
Yu Liu
Changxin Gao
Nong Sang
Venue: ICLR 2026 Poster (2026)
Keywords: text-to-image generation
reinforcement learning
flow matching
preference alignment
group relative policy optimization
Primary area: reinforcement learning

Abstract: Recent GRPO-based approaches built on flow matching models have shown remarkable improvements in human preference alignment for text-to-image generation.
Nevertheless, they still suffer from the sparse reward problem: the terminal reward of the entire denoising trajectory is applied to all intermediate steps, resulting in a mismatch between the global feedback signals and the exact fine-grained contributions at intermediate denoising steps.
To address this issue, we introduce \textbf{DenseGRPO}, a novel framework that aligns human preference with dense rewards, which evaluates the fine-grained contribution of each denoising step.
Specifically, our approach includes two key components: (1) we propose to predict the step-wise reward gain as dense reward of each denoising step, which applies a reward model on the intermediate clean images via an ODE-based approach. This manner ensures an alignment between feedback signals and the contributions of individual steps, facilitating effective training;
and (2) based on the estimated dense rewards, a mismatch drawback between the uniform exploration setting and the time-varying noise intensity in existing GRPO-based methods is revealed, leading to an inappropriate exploration space.
Thus, we propose a reward-aware scheme to calibrate the exploration space by adaptively adjusting a timestep-specific stochasticity injection in the SDE sampler, ensuring a suitable exploration space at all timesteps.
Extensive experiments on multiple standard benchmarks demonstrate the effectiveness of the proposed DenseGRPO and highlight the critical role of the valid dense rewards in flow matching model alignment.

Published as a conference paper at ICLR 2026
DENSEGRPO: FROM SPARSE TO DENSE REWARD FOR FLOW MATCHING MODEL ALIGNMENT
Haoyou Deng 1.2 Keyu Yan Changxin Gao',
Chaojie Mao?,
Xiang Wang', Yu Liu?,
Nong Sanglt
Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology TTogyy Lab, Alibaba Group
(haoyoudeng, wxiang, egao, nsang]ehust. edu. en
yankeyu66@aliyum com {chaojie maj, aya00369)aalibbbaaaaa com
ABSTRACT
Recent GRPO-based approaches built on flow matching models have shown markable improvements in human preference aligmment for text-to-image gener- ation. Nevertheless, they still suffer from the sparse reward problem: the termi- nal reward of the entire denoising trajectory is applied to all intermediate steps, resulting mismatch between the global feedback signals and the exact fine- grained contributions at intermediate denoising steps. To address this issue, we introduce DenseGRPO, novel framework that aligns human preference with dense rewards, which evaluates the fine- grained contribution of each denoising step. Specifically, our approach includes two key components: (1) we propose predict the step-wise reward gain as dense reward of each denoising step, which applies a reward model on the intermediate clean images via an ODE-based ap- proach. This manner ensures alignment between feedback signals and the contributions of individual steps, facilitating effective training; and (2) based on the estimated dense rewards, mismatch drawback between the uniform explo- ration setting and the time-varying noise intensity in existing GRPO-basee meth- ods is revealed, leading a reward-aware scheme to calibrate the exploration space by adaptively adjusting timestep-specific stochasticity injection in the SDE sampler, ensuring suitable exploration space at all timesteps. Extensive experiments on multiple standard benchmarks demonstrate the effectiveness of the proposed DenseGRPO and high- light the critical role of the valid dense rewards in flow matching model alignment.
inappropriate exploration space. Thus, we propose
INTRODUCTION
Flow matching models (Lipman et al. 2022 Liu et al. 2022 Wang et al. 2025a have achieved remarkable advancement in the text-to-image generation task, yet aligning them with human pref- erence remains a critical challenge. Recent progresses (Liu et al., 2025 Xue et al. 2025 Wang 2025b) highlight reinforcement learning (RL) as promising solution by maximizing rewards during the post-training stage. Among these, Group Relative Policy Optimization (GRPO) (Shao
al.
al.
2024) has attracted substantial attention, with numerous studies (Liu al., 2025 Xue al.]
2025
Li et a
2025
He et al. 2025) reporting significant gains in human preference alignment.
Although effective, existing GRPO-based approaches, egg., Flow-GRPO [Liu et al., 2025) and DanceGRPO (Xue et al. 2025), still suffer from the sparse reward problem: the terminal reward of the entire denoising trajectory directly adopted to optimize intermediate denoising steps. As shown in Fig. (a), for the i-th T-step generation trajectory in a GRPO sampled group, they only predict& a single, sparse reward Rt from the terminal generated image, and naively adopt It' to opti- mize all intermediate denoising steps. However, as R represents the cumulative contribution of all
"qqual Contribution CCorespponing Author

---

Published as a conference paper at ICLR 2026
T denoising steps, directly applying R to optimize single step timestep leads to a mis- match between the assigned global trajectory-lever feedback and the exact Line-grained step-wise contribution, misleading policy optimization.
To address the aforementioned issue, we introduce DenseGRPO, novel RL frame- work that aligns human preference with dense rewards, depicted in Fig. [ (b). The key idea of dense rewards is to evalu- ate the step-wise contribution of each de- noising step, thereby aligning the feed- back signals with the fine-grained contri- bution. Intuitively, training process re- ward model presents a promising approach to estimate dense rewards (Zhang et al. 2024), yet encounters two limitations: increased training costs due additional models and limited adaptability to other tasks. In DenseGRPO, we adopt simple yet effective approach that eliminates the need for additional specialized models and can seamlessly integrate with any estab- lished reward model. Specifically, since the contribution of a denoising step can be accessed by latent change, we propose predict the reward gain between the cur- rent step and the next step latent as dense reward of each denoising step. To estimate the reward of an intermediate latent, we leverage the deterministic nature of Ordinary Differential Equation (ODE) and apply a reward model on the intermediate clean images via ODE denoising. Then, we assign the reward feedback latent rewards and thus obtain the dense rewards by computing reward gains at each step. By this means, the estimated dense rewards ensure an alignment between feedback signals and the contribution of individual steps, thereby facilitating human preference alignment.
Trajectory-Wiis Sparse Reward
Reward Model
x7
x
(a) Existing Approaches (e.g. Flow-GRPO)
Step- Wise Dense Reward
ARt
AR
ARf xb
Reward Model
R
x
x--1
x
x
(b) DenseGRPO
Figure 1: (a) Existing approaches only prediet a sin- gle, sparse reward at the end of the denoising trajectory, which is naively applied to optimize all intermediate steps. (b) DenseGRPO estimates step-wise rewards of individual steps, densifying the feedback signal for the denoising process.
Moreover, leveraging the step-wise dense rewards estimated above, a mismatch drawback between the uniform exploration setting and the time-varying noise intensity in existing GRPO-based ap- proaches (Liu et al., 22055] Xue et al. (025) is revealed. In general, the amount of noise is consistent between the diffusion and denoising processes in diffusion models (Song et al. 2020). Since RL re- lies on stochastic exploration, Flow-GRPO proposes Stochastic Differential Equation (SDE) sam- pler that relaxes this consistency and injects increased noise, allowing diverse sampling. However, the current uniform setting of noise injection fails to align with the tary-ng nature of the gener- ation process, often resulting in either excessive or insufficient stochasticity. As evidenced by Fig. 3 (a), where nearly all samples receive negative rewards at late timesteps, the distribution of dense rewards is imbalanced, indicating an inappropriate exploration space. To mitigate this, we propose a reward-aware scheme to calibrate the exploration space by adaptively adjusting a timestep-spedd stochasticity injection in the SDE sampler, ensuring a suitable exploration space for effective GRPO learning. We conduct extensive experiments across multiple benchmarks, and the superior perfor- mance of DenseGRPO demonstrates its effectiveness and underscores the critical role of valid dense rewards in flow matching model alignment.
To summarize, the main contributions of our work are as follows:
We introduce DenseGRPO, which aligns human preference with dense reward, evaluating the fine-grained contribution of each denoising step. Leveraging ODE-based approach, DenseGRPO estimates a reliable step dense reward that aligns with the contribution.
Informed by the estimated dense rewards, we propose a reward-aware scheme to calibrate the exploration space, balancing the dense reward distribution all timesteps.
Comprehensive experiments on multiple text-to-image benchmarks demonstrate the state- of-the-art performance of the proposed DenseGRPO and highlight the critical role of dense rewards in flow matching model alignment.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Aligmment for Text-to-Image Generation. Aligning text-to-image models (Wei et al. 2025 with human preferences has attracted considerable attention. Early works are directly driven by the pref- erence signals with scalar rewards (Prabhudesai et al.) 2023; Xu et al. 2023) or reward weighted regression (Lee et al. 2023 Furuta et al.; 2024). To obviate the need for a reward model, some approaches (Wallace et al. 2024 Yang et al. (024a) adopt offline Direct Preference Optimization (DPO) (Rafailov et al. 2023) with win-lose pairwise data to directly learn from human feedback. In parallel. to tackle the distribution shift induced by offline win-lose pairwise data relative to the policy model during training, several methods (Black et al.) 2023 Fan et al. (023) utilize Proximal Policy Optimization (PPO) (Schulman et al. 2017) for online reinforcement learning, optimizing the score function through policy gradient methods. More recently, Group Relative Policy Optimiza- tion (GRPO) (Shao et al., 2024) has further improved the alignment task. Specifically, pioneering efforts, e.g., Flow-GRPO (Liu et al. 2025) and DanceGRPO (Xue al. 2025). introduce the GRPO framework on flow matching models and enable diversity exploration by converting the determinis- tic ODE sampler into an equivalent SDE sampler. Despite subsequent GRPO-based advances (He et al. 2025 Li et al., 2025 Wang et al., (025b), existing methods still exhibit mismatch be- tween the global terminal reward feedback and exact fine-graine -ne-grained contribution each denoising step, thereby limiting performance. To tackle this issue, propose DenseGRPO that estimates and assigns accurate reward signals for each denoising step, thereby facilitating effective optimization.
Dense Reward. In sequential generation model alignment, dense reward has proven effective in addressing the sparse reward issue, which is inherent in the trajectory-level feedback. In text gen- eration, to densify the sparse reward, several methods incorporate a per-step KL penalty into the training objective |Ramamurthy et al. [0222] Castricato et al. 2022). Additionally, Tan & Pan (2025) dynamically weights the rewards using token-level entropy for dense reward prediction, achieving true levenl credit assignment within GRPO framework. Similarly, dense reward has been plored for training text-to-image generation models. Specifically, within DPO-style methods, Yang et al. 2024bb fines the per-step reward signal and introduces temporal discounting into the training objective, and SPO (Liang) 2024) trains a step-aware performance model for both noise and clean images. In PPO-style approaches, Zhang et al. (2024) assigns each intermediate denoising timestep temporal reward by learning a temporal critic function. Besides, TempFlow-GRPO (He et al. (025) proposes a trajectory branching mechanism that provides per-timeese timestep reward in GRPO-based align- ment, yet adopts a trajectory-wiss signal for step optimization. Most closely related to our work, CoCA (Liao et al., 2025) estimates the contribution of each step by assigning the terminal reward in proportion to the latent similarity. However, it still assigns the trajectory-wise reward signals to optimize an intermediate denoising step, where optimization mismatch persists. In contrast, we present DenseGRPO that aims to train with the step-wise dense reward, which captures the exact fine-grained contribution of each denoising step.
3 PRELIMINARY
In this section, we briefly review some concepts from a typical previous work (Liu al. 2025) to provide preliminary details about the application of GRPO in flow matching models, including (1) the formulation of RL. on flow matching models, (2) the GRPO framework, and (3) the SDE sampler.
RL on Flow Matching Models. Within reinforcement learning, a sequential decision-making problem is commonly formulated as a Markov Decision Process (MDP). An MDP is characterized by a tuple A. Pu. P. R), where S denotes the state space. AA represents the action space, Pu is the distribution of initial states, P is the transition kernel, and R is reward function. At timestep with a state 8, E S, the agent takes an action a; E A according to a policy π (a s), and thereby receives reward R(s,, a.), moving new state St+1 P(s++1 St, a.). Following Flow-GRPP (Liu et al. 2025), the iterative denoising process in flow matching models can be formulated as an MDP:
St (c,t,
T(a 5t) A p( Lt. c).
P(s
st, st,
(5c, 8 55 1,
a e x1-1 R(s,, ar) A R()
ift= otherwise'
(1)
Po(so) & (p(c), 87, &(,,N(
0,
3

---

Published as a conference paper at ICLR 2026
ARD
R
R
AR
ARF
AR
X--2
x²
x
X
X
X7-1
x7
R
Reward Model
c--0
ODE Denoising
Ri
Figure 2: Overview of DenseGRPO. Given the i-th trajectory within GRPO group, we first predict the rewards { {Ri} } of latents { via ODE denoising. By capturing the reward gain {ARj] each step, we obtain the dense reward that reliably evaluates the step- seppwwse contribution.
Here, the state at timestep includes the prompt c, the timestep t, and the latent 2t. The action the #t-1 predicted by policy model p(æ;-1 æt c). And by is the Dirac delta distribution with nonzero density only at y. Notably, the reward acts as a trajectory-wis single, sparse reward only the terminal state and provides zero reward at intermediate steps. This formulation assigns the reward of the entire trajectory to the final denoising step, thereby overlooking the fine-gre une-grrined contributions of intermediate steps. As result, existing methods adopt this sparse reward to optimize all timesteps, leading to DenseGRPO explicitly estimates the step-wise dense rewards, thereby aligning the reward feedback with the contribution at each step.
feedhack signal that predicts
mismatch. To address this,
GRPO Framework. Flow-GRPO adopts the GRPO framework (Shao et al. 2024) to align flow matching models. Specifically, given prompt the flow matching model pe samples a group of G individual images { 10 with timesteps and the corresponding denoising trajectories
((r/), - )=));- Using a reward model R, the advantage of the i-th image is estimated by group normalization as follows:
A R( mean mean((R(ar) c)
(2)
std std( 2)
Subsequently, the policy is optimized by maximizing the following objective: (,C,,,,)
JHoowGGROO GRPO (
(3)
where
1
[(r A,0,c,
(min (Te (4)
G
1=1
t=
with r()) Polai pola|_plamjen (c)
Notably, the advantage A obtained via Eq. 2 exclusively determined
Pole [z) c)
by the reward signal R( e) of the entire trajectory, rendering independent of any particular timestep t. In other words, policy optimization across different timesteps utilizes identical trajectory- wise reward feedback, exhibiting a mismatch between the assigned trajectory- wise feedback and the step-wise contributions of each timestep.
SDE Sampler. Typically, flow matching models predict the velocity o, and employ a deterministic ODE for the denoising process:
dx, = v.dt.
(5)
Yet, GRPO requires stochastic sampling to generate diverse trajectories for exploration. To this end, Flow-GRPO injects additional noise to sampling by converting the deterministic ODE sampler to an equivalent SDE sampler:
Et+Ai = It [vo t) + of (a, + (1 (2), A.AAt..
(6)
2t
Here, G; = a - and e ~ N(00, I) inject stochasticity, where a a scalar hyper-paranan for noise level control.
4

---

Published as a conference paper at ICLR 2026
Seep See-Wibe Deare Revard Notse
1.50
050
600
2.00
-100
Step-Wtse
-2.0
Balanced
All Negotive
1.50
-2.00
Znsut/icient Exploration
Timestep
Timestop
(a) Existing GRPO-based Methods (a 0.7)
(b) Existing GRPO-based Methods (a = 0.5)
Step-Wiss Deuse Reward Nodss
Nolon
100
Diverse Exploration
1.00
850
4000
+3.0
32.00
Step-Wiie -3.0
Balanced
Imbolanced
Exptoration
2000
Timestep
Timestep
(c) Existing GRPO-based Methods 0.8)
(d) DenseGRPO (((())
Figure 3: Visualization of dense rewards, where each polyline denotes an SDE-sampled trajectory: (a) existing GRPO-based methods utilize uniform setting of noise level such as 0.7, 0.5, and 0.8, leading an inappropriate exploration space; (d) DenseGRPO calibrates timestep-speciifcc noise intensity (t)), enabling suitable exploration space for all timesteps.
DENSEGRPO
In this section, we present DenseGRPO that aligns flow matching models using the step-wise dense rewards. Below, we begin by showing how to explicitly estimate the dense reward, evaluating the contribution of each step. Subsequently, we introduce the reward-aware scheme that calibrates the exploration space in the SDE sampler, providing suitable exploration space for GRPO training.
4.1 STEP-WISE DENSE REWARD
As shown in Fig. existing approaches estimate a single reward R of the whole trajectory, and directly apply R to optimize intermediate steps. Since R is achieved by all steps, this manner encounters a mismatch between the trajectory-wise feedback signal and the step-wise contribution. To tackle this, we propose to estimate dense rewards that evaluate the contribution of each step, thereby providing step-wise feedback signal. From the perspective of reward RI., each action (e.g., æt) receives a reward feedhack (e.g., Rt)) that evaluates its corresponding future outcome. At timestep = t, the one-step denoising process xj - xi-1 contributes to a reward raising from Ii to Ri_1- Therefore, we define the step-wise dense reward A R timestep = as the reward gain:
AR
R Ri.
To this end, we first estimate the reward of any intermediate latent, i.e., Rú Typically, classical RL methods learn critic function to immediately estimate the influence on the future outcome, which serves as a proxy of the reward at the current action (Pignatelli et al. 2023 Zhang et al., (022)). However, the critic function incurs increased training overhead and lacks adaptability to other tasks. In contrast, we implement simple yet effective approach that eliminates the need for additional specialized models. Specifically, our approach leverages the deterministic nature of ODE sampler in flow matching models: given a latent æ; timestep corresponding clean latent, and hence the final clean image, are fully determined. Therefore, this promising
the ODE denoising trajectory, the
one-to-one mapping allows the clean image obtained by ODE denoising to future counterpart for any latent x; Building on these analyses, we propose that the reward latent of be reliably assigned that of the corresponding clean image via ODE denoising.
5

---

Published as a conference paper at ICLR 2026
As illustrated in Fig. 2 for the i-th trajectory {0]]?_r within sampled group of GRPO, first employ an n-step ODE denoising to obtain the underlying clean latent o 1, for latent a/:
$ ODE,, (a), c).
/ (.1//v] oi.0. where &i It/m/ is the latent generated by ODE sampler at timestep [t/n] and n may be any integer in [1, 1). In our experiments, we set nt for improved performance (See Sec. 5.3 for its impact). After that, we decode the clean image from æf,0 and apply reward model R predict its reward Ri as the latent reward for at:
Here, $b.0 æyj and ODE, involves n. ODE denoising steps: æ)
ODE
ODE,
ODE,
R & R R( c).
Notably, since Li belongs to the clean distribution, plenty of established reward models can be seamlessly integrated as R for reward prediction. With the estimated { {R; 1t-1+ we obtain the dense reward (AR;) of timestep step's s contribution. During GRPO training, we replace the sparse R(x C) in Eq. 2 with the dense A ARt at timestep t, and thereby the advantage is calculated by:
1T
by computing the reward gain via Eq. which represents each
ARt mean (R))
(10)
std( ( R
As a result, we align the reward signal with the contribution of denoising each denoising step, facilitating effective policy optimization.
4.2 EXPLORATION SPACE CALIBRATION
Based on the estimated per- timestep dense reward above, a mismatch drawback between the exploration space and the denoising timestep schedule in existing GRPO-based methods revealed. To promote diverse
Algorithm 1
Exploration Space Calibration
Require: policy model po, reward model R. initial noise level (B).. prompt dataset C, total sampling steps number of samples N, small constants (21
1:
for iteration & = 1, do for sample i = 1 to N do
2:
3:
Init noise x² ~ (O
exploration for RI, Flow-GRPO proposes an SDE sampler that injects additional noise during trajectory sampling. This injec- tion leads to a greater amount of noise than the denoising process, sampling out-of-distribution tra-
4:
Sample c ~~' C
5:
Sample trajectory { )-- via SDE with (b)))
6:
Predict latent rewards { RR] = R(ODE,, (m), c),
7:
Calculate dense rewards { {AR) R R Ri end for
8:
9:
for timestep = 7 to do
10:
[num (1 ((RR) Ri > 0}) mim((A)) < 0}) < then
jectories. Therefore, a suitable noise injection is critical, as an inappropriate setting often results
11:
(1) <- ((() +
12:
clse
13:
(1) (b())
in either excessive or insuffi-
14:
end if
15: end for
cient stochasticity. However,
16: end for
the current uniform setting of noise injection fails align with the time-varying nature of the generation process, which all timesteps share an identical noise level in Eq. 6 As plotted in Fig. 3 (a), we collect the step-wise dense reward of several trajectories with a = 0.7 using PickScore (Kirstain et al., 2023) as the reward model. The results show that trajectories receive negative rewards at timestep = 2, indicating that nearly all samples in the current exploration space perform worse than the default. Lacking positive guidance, this inappropriate exploration space undermines effective policy optimization. We hypothesize that this issue may arise from the excessive noise injection in the current setting (a = 0.7). Hence, we further reduce the stochastic noise injection by lowering exploration space yet improves the reward balance, enabling a more fair distribution of positive and negative feedback, particularly timestep 2. Conversely, increasing the noise level to 0.8
17: return (+)
0.5. As depicted in Fig. (b), this adjustment constrains the

---

Published as a conference paper at ICLR 2026
Table 1: Performance Compositional Image Generation, Visual Text Rendering, and Human Preference benchmarks, evaluated by task performance on test prompts, and by image quality and preference scores on DrawBench prompts. ImgRwd: ImageReward: UniRwd: UnifiedReward, UniRwd": our evaluation results of the official checkpoints and our method with UnifieRRewrrd
Task Metric
Image Quality
Preference Score
Model
GenEvalt
OCR Aec.7
PickSoone?
Aesthetict
DeQA
ImgRwdt
PickScoref
UniRwd?
UniRwd
SD3.5-M
0.63
0.59
21.72
5.39
4,07
0.87
22.34
3,33
3,06
Compostioonl Image Goneration
Flow-GRPO
0.95
5.25
4.01
1.03
22.37
3.51
3.18
GRw-GRRO&COCAA
0.96
5.25
3,92
0.93
22.34
3.05
DeneeGRPO
0.97
5.33
3,83
1.02
22.28
3.03
Visual Teit Rendering
Flow-GRPO
0.92
5.32
4,06
0.95
22.44
3,42
3.17
Flow- GOw.GRPO+GGCA
0.93
5.32
4.08
0.92
22.47
3.11
DemeeGRP)
0.95
5.31
4,02
0.91
22.44
3.12
Human Preforence Aligument
Flow-GRPO
23.31
5.92
4,22
1.28
23.53
3.66
3.38
NOw-GRPOHCOCA
23.63
6.22
4.16
1.32
23,80
3.38
DemeeGRPO
24.64
6.35
4.06
1.41
24.55
3.39
1.00
2500
GuubulSorr
8.40
DCN Accuraii
1048
2100
22:00
GRPO
8.50
21.00
1068 206
2048 3000 3590
10.0
1506
2200 3000 Training
2006
Steps
Compositional Image Generation
(b) Visual Text Rendering
Human Preference Alignment
Figure 4: Comparison of learning curves. Figures (a) to (c) correspond to the tasks of compositional image generation, visual text rendering, and human preference alignment, respectively.
expands the exploration space, as evidenced by the greater diversity of rewards at timestep = 10 in Fig. 3 (c). However, more pronounced imbalance arises at certain timesteps, e.g., timestep and 2. These findings underscore the limitation that uniform noise injection setting fails to produce suitable exploration space for all timesteps. Therefore, timestep-speciff noise injection setting is expected to align with the time-varying nature of the generation process.
To mitigate this, we propose calibrate the exploration space by adaptively adjusting the stochas- ticity injection in the SDE sampler suitable for all timesteps, yielding timestep-specifie noise level (t). We suggest that an ideal exploration space supposed to provide diverse trajectories while preserving dense reward balance. Based on the above observation, a higher noise level facilitates exploration diversity, while a lower noise level benefits reward balance. Consequently, we advocate for using higher feasible noise intensity enhance exploration diversity, up the point where reward imbalance occurs. As illustrated in Algorithm [[] we start by sampling plenty of trajecto- ries { -4-1: + and then predict their dense rewards { (AR)). Subsequently, for each timestep, we increase the noise level slightly when dense rewards are balanced (iee., the disparity between the number of positive and negative samples is minimal), or decrease otherwise. By it- cratively updating, we obtain suitable (t)) output, which ensures a balanced exploration space
for all timesteps. Accordingly, a, in Eq. is employed as o₂ ((t) / Given that (b()) is
self-adjusting function with respect to the item / / which is constant for I,, can be incorporated into the calibration process. Hence, we unify the formulation and employ a, as follows:
a, (t)).
(11)
'Our experiments reveal discrepancy with the results reported in Flow-GRPO paper when evaluating the official checkpoints with UnificdReward. This may stem from updates of the UnifiedReward checkpoint or the sglang package, as discussed in httpttttt///teeeeee com/yi fan htwp/////o/oo//o/oeeeeeeee

---

Published as a conference paper at ICLR 2026
Flow-GRPO
Flov-GRPO+CuCA
DenseGRPO
Compesttinal Insgee Genoration
A photo a black broecoli and yellow cake.
Fount Rendering
Search Callegg Here
Search Calidag Here
Soarch Cotalo Herr
libvary sleek bookrkelves computers along one sign computers Catalog Here", Seft, natural light filters shrough large windowws, cassing a warm glow over the space.
Human Proferunc Aligument
Laulying on top of tondstood.
Qualitative comparison on three benchmarks: Compositional Image Generation, Visual Text Rendering, and Human Preference Alignment. Our DenseGRPO generates high-quality out- comes across all tasks, excelling in color accuracy, text fidelity, and content alignment.
Figure 5:
5
EXPERIMENT
5.1
IMPLEMENTATION DETAIL
evaluate our method on three text-to-image tasks: Compositional Image Generation, employing GenEval (Ghosh et al. 2023) as the reward model, (2) Human Pref- erence Alignment, utilizing PickScore (Kirstain et 2023). (3) Visual Text Rendering, predicting OCR accuracy (Gong et al. 2025) as reward. The experimental setup aligns with Flow-GRPO, in- cluding a sampling timestep
Following Flow-GRPO,
10, an evaluation timestep = 40, a group size G 24, and an image resolution of 512. The KL ratio B in Eq. is set to 0.04 for compositional image generation and visual text rendering, and 0.01 for human preference alignment.
5.2 MAIN RESULT
We compare the proposed DenseGRPO with Flow-GRPO (Lu ct al. 2025) and CoCA (Liao et al. 2025). Since the official CoCAA is designed DDPMs, implement their idea flow match- ing models by tracking the latent similarity for nep-wise reward, denoted by "Flow-GRP -OOOWGGRP+CCCCA"" As summarized in Tab. [ and Fig. our DenseGRPO achieves superior performance, outperforming competitors across all three tasks. Notably, in the task of human preference alignment, our DenseG- RPO significantly surpasses the competitors by at least 1.01 of PickScore. In addition, compared to -ROo-GRRO+COCA" which leverages latent similarity to estimate step-wise feedback signal, the substantial gains of our ODE-bbsed approach validate its advancement to provide more accurate dense reward. Moreover, shown in Fig. our DenseGRPP generates favorable outcomes with higher visual and semantic quality. For instance, in the third row, only our DenseGRPO successfully generates the positional relationship of "on top of", whereas other methods produce combination of "ladybug" and "toadstool". These results demonstrate the significant advantages DenseGRPO in aligning the target preference.
8

---

Published as a conference paper at ICLR 2026
25.03
25.0
2433
2450
PicSoore
10.99
PickSlorr
23:00
19.83
2300
22.09
29.33
2133
Denze Bewoort (Bandlre)
7100
2153
Our #5G
=
GAPO
0.7
2200
2150
2100
230
12:0 1443 1630
830
1000
1230
1489 1693
400
Training Staps
Training Steps
TV Training Time (GPO Hours)
(a) Effect of Dense Reward
(b) Effect of Noise Lovel
(c) Effect of ODE Denoising Steps
Figure 6: Ablation studies our critical designs. (a) Step-wise dense reward aligns with contribu-
tion, surpassing trajectory-wiss sparse reward. (b) Our time-specific noise level enables a suitable
exploration space. (c) Increased ODE denoising steps (n) improve dense reward accuracy, yielding superior results. The vertical axis denotes the PickScore results. The horizontal axis of (a) and (b) training steps, while the horizontal axis of (c) denotes training time for training cost comparison.
5.3 ANALYSIS
Effeet of Dense Reward. To investigate whether RL benefits more from sparse rewards or dense rewards, we include another setting for comparison, namely "Dense Reward (Baseline)", which directly applies the _;_ reward () for optimizing denoising step at timestep = f. During
[ 1.m1)) As illustrated in Fig. (a).
GRPO training, its advantage is computed as A₁
sd )2
"Dense Reward (Baseline)" offers greater benefits than Flow-GRPO, highlighting the effectiveness of dense reward. This advancement is further confirmed in Tab. [ Fig. By employing step- wise rewards, "Flow-GRP -ROOGRROO+CCC' outperforms the vanilla Flow-GRPO. These findings highlight the critical role of step-wise dense rewards, which align the feedback signal more closely with the contribution for each denoising step, thereby facilitating policy optimization.
Effeet of Exploration Space Calibration. To evaluate the effectiveness of the calibrated explo- ration space in Sec. 4.2 we make comparison by applying the existing uniform setting (a = 0.7) in the proposed DenseGRPO. As presented in Fig. (b). we find that our time time-specific noise level advances the alignment task, indicating a more suitable exploration space for all timesteps and val- idating the success of our reward-aware calibration scheme. Besides, even if using the uniform 0.7 setting, our DenseGRPO also yields improved performance than Flow- further vali- dating DenseGRPO's superiority and the benefit of step-wise dense reward.
Effect of Different ODE Denoising Steps. The proposed DenseGRPO adopts n-step ODE denoising (Eq. g to obtain clean latents. To evaluate the impact of different n, we perform ablation studies with n 1, and t, respectively. As depicted in Fig. (c), we can draw two findings: (1) increasing the number of ODE denoising steps improves performance; (2) a single-step ODE yields suboptimal results, performing worse than Flow-GRPO. These findings suggest that a more accurate dense reward offers more benefits. Since existing reward models are primarily tailored for well-denoised images, utilizing more ODE steps is closer to a precise rollout, and thus receives more accurate rewards. In contrast, single-step ODE deviates far from this domain, resulting in less accurate rewards and degraded performance. Furthermore, under the same experimental setting, require 11, 13, and 19 GPU hours for training 20 steps, respectively.
= 1, 2, and Although a larger n incurs higher computational overheads, it offers improved performance with the same GPU training time, underscoring the critical role of dense reward accuracy.
Discussion of Reward Hacking. Following Flow-GRPO, we evaluate our method on Draw- Bench (Saharia et al. 2022) using four additional metrics: Aesthetic Score (Schuhmann 2022
DeQA You et al.] (005). ImageReward (Xu et al.) 2003), and UnifiedReward (Wang et al. 2025c As shown Tab. [ our DenseGRPP exhibits outstanding alignment capability with slight reward hacking in parts of tasks. Notably, in the human preference alignment, while achieving pronounced improvement on the PickScore metric, our method also performs strongly across other metrics. For example, in terms of the Aesthetic score, our DenseGRPO outperforms Flow-GRPO by 0.43, in- dicating more visually pleasant outcomes by DenseGRPO. These advancements demonstrate the strong robustness of the proposed DenseGRPO.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We present DenseGRPO to address the mismatch between trajector- reward feedback and step-wise contribution. By estimating per-timestep dense rewards via an ODE-based approach, DenseGRPO aligns the reward feedback with the contribution of each denoising step, enabling fine-grained credit assignment and facilitating effective optimization. Based on the estimated dense rewards, address the current imbalance exploration in the SDE sampler, we propose reward- aware scheme that calibrates timestep-specific noise injection, ensuring suitable exploration space for all timesteps. Extensive experiments demonstrate the substantial gains achieved the proposed DenseGRPO and validate the effectiveness of dense reward in flow matching model alignment.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. All datasets utilized in this study are publicly available and used in accordance with their respective licenses. The research does not involve human subjects, sensitive personal information, or proprietary content. Besides, the methods proposed in this paper do not present any foreseeable risks of misuse or harm.
REPRODUCIBILIIT STATEMENT
We committed ensuring the reproducibility of our research. A comprehensive description of the proposed DenseGRPO is provided in Sec. 4. Implementation details, including the experimental setup, hyperparameter configurations, training pipeline, and evaluation metrics, are introduced in Sec. 5.1 and further elaborated in Sec. A the Appendix. Additionally, all datasets utilized in this study are publicly available and described in detail in Sec. 5.1.
ACKNOWLEDGMENT
This work is supported by the National Natural Science Foundation of China under grants U22B2053 and 623B2039.
REFERENCES
Forest Labs Black et al. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.1574 2025.
Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, Ian Yang, Spencer Frazier, and Mark Riedl. Robust preference learning for storytelling via contrastive reinforcement learning. arXiv preprint arXiv:2210.07799, 2022.
Ying Fan, Olivia Watkins, Yuqing Du Hao Liu, Moonkyung Ryu, Craig Boutilier, Picter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. In Advances in Neural Information Processing Systems, 2023.
Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, and Sherry Yang. Improving dynamic object interactions in text-to-video generation with ai feedback. arXiv preprint arXiv:24/2.02611, 2024.
Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An
framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 22132-52152. 2023.
Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, et al. Seedream 2.0: native chinese-english bilingual image generation foundation model. arXiv preprint arXiv:2503.07770 2025.
10

---

Published as a conference paper at ICLR 2026
Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, and Bo Zhang. Tempflow-gug -rrp:: When timing matters for grpo in flow models. arXiv preprint arXiv:2508. 04324, 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to- text-to-image generation. Advances in neural information processing systems, 36:36652-36663, 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Warkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. /2192, 2023.
Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Miles Yang, and Zhao Zhong. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint 1rXiv:2507.21802 2025.
Zhanhao Liang. Step-aware preference optimization: Aligning preference with denoising perfor- mance at each step. arXiv preprint arXiv:2406.04311 2(5):7, 2024.
Xinyao Liao, Wei Wei, Xiaoye Qu, and Yu Cheng. Step-level reward for free in rl-based t2i diffusion model fine-tuning, arXiv preprint arXiv:2505.19196, 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Jie Liu. Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505. 000400,, 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0.3: 000033, 2022.
Eduardo Pignatelli, Johan Ferret, Matthicu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. survey of temporal credit assignment deep reinforcement learning. arXiv preprint arXiv:2312.01077 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 11553728-537411 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint urXiv:2210.0124), 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings the the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 36494, 2022.
Christoph Schuhmann.
Laion-acsthetics.
https://laion.a//bllos
Taion-aestheiicss 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347. 2017.

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Hongze Tan and Jianfei Pan. Gipo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508. 04344, 2025.
Bram Wallace, Mcihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238,, 2024.
Xiang Wang, Zhifei Zhang, He Zhang. Zhe Lin, Yuqian Zhou, Qing Liu, Shiwei Zhang, Yijun Li, Shaoteng Liu, Haitian Zheng, et al. Hbridge: H-shape bridging of heterogeneous experts for unified multimodal understanding and generation. arXiv preprint arXiv:2511.20520 2025a.
Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning. arXiv preprint 25Xiv2550..205:: 2025b.
Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multi- modal understanding and generation. arXiv preprint arXiv:2503.05236. 2025c.
Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with customized sub- ject and motion. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 6537-6549, 2024.
Yujie Wei, Shiwei Zhang, Hangjie Yuan, Yujin Han, Zhekai Chen, Jiayu Wang, Difan Zou, Xihui Liu, Yingya Zhang. Yu Liu, et al. Routing matters in moe: Scaling diffusion transformers with explicit routing guidance. arXiv preprint arXiv:2510.244
2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 01115533-19355 2023.
Dong. Imagereward:
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.0788 07818, 2025.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024a.
Shentao Yang. Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with preference. In International Conference on Machine Learning, pp. 55998 56032.. PMLR, 2024b.
Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress accurate image quality scores using score distribution. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14483 14494, 2025.
Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, and Dacheng Tao. Confronting reward overoptimization for diffusion models: A perspective of inductive and primacy biases. In International Conference on Machine Learning, pp. 60396-60413, PMLR, 2024.
12

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAIL
Our experiments are conducted based on the official implementation D of Flow-GRPO (Liu et al. 2025). The models are trained using 16 NVIDIA A100 GPUs. Before training, we first perform the exploration space calibration strategy to generate the noise level (t()), as presented in Algorithm 1 where E and 82 are set to 2 and 0.01. Note that the obtained (+() is fixed in the training process. To ensure fair comparison, we adopt the same experimental settings Flow-GRPO. Specifically, we apply LoRA with & = 64 and r = 32. During training, we use the Adam optimizer with a learning rate of 3 x 10 10-4 ,8 0.9, B2 = 0.999, and weight decay of 1 10 The global batch size is set to 144, with a gradient accumulation step of 8. The total number of training iterations is 4500, 1500, and 4500 for the tasks of compositional image generation, visual text rendering, and human preference alignment tasks, respectively. Upon completing training, inference is conducted using the standard ODE sampler of flow matching models for text-to-image generation.
B MORE RESULT
B. TRAINING CURVE OF KL LOSS
We present visualization of the KL loss evolution during training in Fig. 7 The results show that the KI. loss of Pense-GRPO is slightly larger than that of Flow-GRPO. This difference arises from the incorporation of the timestep-specife noise level, which encourages a more diverse exploration space and thereby pushes the model to deviate further from the original model.
DenoeGRPO
DasseGRPO
DesseGRPO
Flew GRPO
How GRPO
How GIPO
0U802)
KL Lanee
10. Lenes
02030
8:004
01891
0001
00005
30 v00
100 1166 1330 1483 sin 1991
Traininn Steps
Traiing Stepe
Training Steps
(a) Compositional Image Generation
(b) Visual Text Rendering
(c) Haman Preference Alignment
Figure 7: Training curves of KL loss. Figures (a) to (c) correspond to the tasks of compositional image generation, visual text rendering, and human preference alignment, respectively.
B.2 ACCURACY OF DENSEGRPO'S S REWARD
In DenseGRPO, we estimate step- wise dense rewards by calculating the reward gain at each denoising step and utilize an ODE-based method predict the reward for intermedi- ate latents. To evaluate the accu- racy of DenseGRPO's reward, we make a comparison between the pre-
2430
2330
23.00
OfF-baadd Latent Rewverd
22.00
22.00
21.00
21.00
2030
dicted latent reward and the terminal reward of the SDE sampling trajec- tory with PickScore, as visualized in Fig. & Note that the latent reward at timestep = 0 is directly predicted by the reward model without requiring ODE sampling, and therefore corre- sponds to the terminal reward of the SDE sampling trajectory. The results show that the difference between the predicted latent rewards and the ter- minal trajectory rewards is minimal. Furthermore, the relative ranking of rewards across different samples consistently aligns across all timesteps. These findings confirm the accuracy of the reward predictions in DenseGRPO.
70.00
1850
1900
10
Timestop
Figure 8: Visualization of ODE-based latent rewards, i.e. R; predicted by Eq. 9 where each polyline denotes sam- pled trajectory. timestep 0 represents the terminal re- ward of the SDE sampling trajectory.
http://gtttuubccco http:::///itbbbb.com/y/ fan
13

---

Published as a conference paper at ICLR 2026
B.3 MORE EXPERIMENT
Experiment on FLUX.I-Dev. We further evaluate the performance of our method against Flow- GRPO on FLUX. 1-ev (Black et al. 2025) model using PickScore as the reward model. As shown in Fig. aa), the proposed DenseGRPO achieves substantial improvements over Flow-GRPO, sug- gesting the superiority and robustness of the estimated dense rewards.
25.00
2403
21.00
2400
PlelSpore
20.03
21.20
2140
2100
22.00
2330
DenseGRPO
22.1
3840
19.0
2151
34:00
Training Steps
Training Steps
Training Stops
(a) Performance on FLUX..
(b) Performance on SD 3.5-M (1024)
(c) Perforancee Diffission Model
Figure 9; Performance of DenseGRPO compared with Flow-GRPO additional models: (a)
FLUX.I-dex, (b) SD 3.5-M on 1024 x 1024 resolution, and (c) diffusion model.
Experiment on High Resolution. As shown in Fig. b)), we raise the training and inference resolution to higher resolution 1024 x 1024 on the SD 3.5-M model, utilizing PickScore as the reward model. The results reveal that DenseGRPO also yields significant gain over Flow-GRPO, indicating the strong scalability of DenseGRPO.
Experiment on Diffusion Model. Though DenseGRPO focuses on flow matching models, it can also generalize to other generative models (Wei et al., 2024) by employing deterministic sampler to predict dense rewards. This deterministic nature enables a one-to-one mapping between interme- diate latents and clean latents, ensuring an accurate prediction of latent rewards and step-wise dense rewards. To validate this capability, we use SD 1.5 (Rombach et al., 2022) as the base model with an ODE sampler predict Ai from x/. The reward of #2,0 is then assigned that of æ/, and the reward gain is calculated as the step-wise dense reward. As presented in Fig. 9()), the performance improvement of dense reward within DenseGRPO demonstrates the accuracy and effectiveness of dense reward on diffusion models, These findings show that DenseGRPO is capable of generalizing other generative families via a deterministic denoising sampler.
B.4 REWARD HACKING ANALYSIS
Figure 10 illustrates examples of re- ward hacking. When GenEval is used as the reward model for compo- sitional image generation, DenseG- RPO achieves notable gains in com- positional accuracy, such as object counting, but may occasionally expe- rience a decline in image quality. A similar issue is observed in the task of visual text rendering. This prob- lem arises from the step-wise dense reward in DenseGRPO, which aligns feedback with the contributions of in- dividual steps. providing a more pre-
Flew-GRPO
DePoGRPO
Compuriioonl Emegge Generabto
A
Fimal Jest Rraderiog
SUPPLIES CHECKED
SUPPLIES CHECKED
cise signal. While this increased re-
hike paaner insemnse teal doir Auclpae pesmissslly doskecnn a "Suysiis Cheokea" sgg rigord
ward accuracy enhances the learning process, it may also make the model more susceptible to overfitting the ward model, thereby amplifying the risk of reward hacking. One potential solution is to employ a large-scale reward model to provide higher-quality reward signals.
csseetioi adbrusaaress spini jaureyy.
Figure 10: Visualization of reward hacking.
C LLM USAGE
We use LLMs to assist with writing refinement, but do not involve them in core idea development.
14


Reviews:
[{"review_id": "7LuQGOiO7O", "reviewer": "Reviewer_34g2", "summary": "This paper introduces DenseGRPO, which is a framework that enhances flow-matching based GRPO by introducing dense, step-wise rewards to better align human preferences with intermediate denoising contributions. It further calibrates the exploration space through a reward-aware, timestep-adaptive scheme, achieving more efficient and consistent alignment across benchmarks.", "questions": "What is the difference between the sparse reward setting in Figure 6(a) and the original Flow-GRPO, and why does it perform significantly better than the original version?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper highlights the importance of **dense rewards** for effective **credit assignment** in reinforcement learning tasks. This is indeed a crucial factor that can significantly improve policy optimization stability and efficiency.\n* Figure 6(c) provides informative results. It clearly demonstrates that the most straightforward approach of predicting the final $x_0$ in a single step to compute delta rewards performs even worse than the original Flow-GRPO, underscoring the necessity of proper dense reward design.", "weaknesses": "*  The proposed Dense-GRPO algorithm introduces many additional ODE denoising steps, which inevitably increase computation time. However, the paper lacks a comparison with Flow-GRPO under the same training-time horizontal axis to show efficiency differences.\n\n*  Since the modified algorithm may alter KL-consumption behavior, it would be helpful to visualize how the KL loss evolves during training for both Flow-GRPO and Dense-GRPO.\n\n*  The proposed Exploration Space Calibration module seems designed specifically for dense rewards. It would be valuable to clarify whether this technique can be directly applied to Flow-GRPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4fIWGZrA8i", "reviewer": "Reviewer_q5r8", "summary": "This paper proposes DenseGRPO, a reinforcement learning framework for aligning flow-matching models (e.g., text-to-image generation) with human preferences.\nWhile previous GRPO-based approaches such as Flow-GRPO and DanceGRPO rely on a sparse reward that assigns a single trajectory-level reward to all denoising steps, DenseGRPO introduces a step-wise dense reward estimation method. Specifically, it leverages ODE-based denoising to obtain intermediate clean images and applies a reward model to compute per-step reward gains. In addition, the paper identifies a mismatch between uniform noise injection and the time-varying denoising process in existing methods, and proposes a reward-aware exploration space calibration scheme that adaptively adjusts timestep-specific stochasticity.\nEmpirical results on multiple text-to-image benchmarks show consistent improvements over Flow-GRPO and Flow-GRPO+CoCA.", "questions": "- Since dense rewards are computed using ODE rollouts while policy trajectories come from SDE sampling, how do the authors justify that the resulting reward estimates remain unbiased with respect to the actual policy distribution?\n- Theoretical framing: Could the authors connect their dense reward to existing temporal credit-assignment frameworks (e.g., potential-based shaping or advantage decomposition)?\n- Would DenseGRPO still hold if the base model were trained with a consistency loss or direct velocity regression instead of flow matching? Is the dense reward specific to ODE semantics?\n- Is the reward gain $\\Delta R_t = R_{t-1}-R_t$ guaranteed to reflect incremental improvement rather than numerical noise from the reward model? Has the variance of this difference been analyzed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the well-known issue of sparse reward assignment in GRPO-based flow matching models and provides a conceptually simple fix.\n\n- DenseGRPO demonstrates noticeable improvements across several text-to-image benchmarks and alignment metrics (e.g., +1.0 PickScore gain).\n\n- Figures and algorithm descriptions are intuitive, especially the visualization of dense reward distributions and the adaptive exploration scheme.", "weaknesses": "- Conceptual Incrementality without a Deeper Credit Assignment Analysis\n\nWhile the shift from sparse to dense reward seems natural, the paper does not provide a principled analysis of why ODE-based reward estimation captures per-step contribution more faithfully. The method assumes that ODE rollouts preserve semantic consistency, yet no theoretical or empirical verification supports this. Without a formal treatment of credit assignment or causality, the approach reads more as a heuristic refinement than a conceptual advance.\n\n- Limited Perspective Beyond Flow Models\n\nThe method is tightly coupled with ODE-based denoising, making it unclear whether the same principle generalizes to other generative families (e.g., DDPM, consistency models). Without this, the claimed “general framework for dense reward” seems overstated.\n\n- Unclear Attribution of Performance Gains\n\nThe performance boost could stem from either (a) the dense reward, (b) better noise scheduling, or (c) hyperparameter tuning. The ablation is not disentangled enough to isolate these effects. For example, the “uniform a=0.7” baseline in Fig.6(b) still benefits from DenseGRPO, which suggests confounding factors that aren’t fully analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aL8AmkrRRZ", "reviewer": "Reviewer_Fc1V", "summary": "This paper introduces DenseGRPO, a novel reinforcement learning framework for aligning flow matching models with human preferences in text-to-image generation. The key innovation addresses the \"sparse reward problem\" in existing GRPO-based approaches, where a single terminal reward for the entire denoising trajectory is naively applied to optimize all intermediate steps, creating a mismatch between global feedback and fine-grained step-wise contributions.\n\nDenseGRPO proposes two main components: (1) **Step-wise Dense Reward Estimation**: Instead of using a single terminal reward, the method predicts reward gains at each denoising step by applying a reward model to intermediate clean images obtained through ODE-based denoising. This approach assigns $\\Delta R_t^i = R_{t-1}^i - R_t^i$ as the dense reward for each step, aligning feedback signals with actual step contributions. (2) **Reward-Aware Exploration Space Calibration**: Based on the estimated dense rewards, the authors identify that existing uniform noise injection in SDE samplers creates inappropriate exploration spaces. They propose an adaptive scheme that adjusts timestep-specific stochasticity  to balance dense reward distributions across all timesteps.\n\nExperiments on three benchmarks (compositional image generation, visual text rendering, and human preference alignment) demonstrate superior performance over Flow-GRPO and CoCA baselines. The method achieves notable improvements, particularly +1.01 PickScore on human preference alignment, while maintaining good generalization across multiple evaluation metrics.", "questions": "1. How many iterations does Algorithm 1 require for $\\psi(t)$ to converge? What are the sensitivity and recommended values for hyperparameters $\\epsilon_1$ and $\\epsilon_2$?\n2. Is exploration space calibration performed once as preprocessing or continuously during training? If continuous, how frequently?\n3. How does the accuracy of dense reward estimates degrade as you move further back in timesteps (larger t)? Have you analyzed the reward estimation error at different timesteps?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality:** The paper presents a genuinely novel perspective on the sparse reward problem in flow matching model alignment. While dense rewards have been explored in text generation and diffusion models, the specific ODE-based approach for estimating step-wise rewards without additional specialized models is creative and practical. The identification of exploration space mismatch through dense reward analysis is an insightful observation that leads naturally to the second contribution. The paper distinguishes itself from CoCA by actually training with step-wise rewards rather than just using trajectory-wise signals proportionally weighted.\n\n**Quality:** The experimental validation is comprehensive, covering three distinct tasks with multiple evaluation metrics. The ablation studies systematically validate each design choice: dense vs. sparse rewards (Figure 6a), exploration space calibration (Figure 6b), and ODE denoising steps (Figure 6c). The visualization of dense reward distributions (Figure 3) provides compelling evidence for the exploration space problem. The method demonstrates consistent improvements across all tasks, with particularly strong results on human preference alignment. The authors appropriately acknowledge discrepancies in reproducing baseline results with UnifiedReward and provide their own evaluations for fair comparison.\n\n**Clarity:** The paper is generally well-written with clear motivation. Figure 1 effectively illustrates the core problem and proposed solution. The mathematical formulation is precise, and the algorithm pseudocode (Algorithm 1) provides implementation details. The progression from problem identification to solution is logical and easy to follow.\n\n**Significance:** The work addresses a fundamental limitation in current GRPO-based alignment methods for flow matching models. The ODE-based dense reward estimation is elegant and can integrate with any existing reward model, making it broadly applicable. The reward-aware exploration calibration provides insights that could benefit other RL-based generative model training approaches. The consistent improvements across diverse tasks suggest practical value for the community.", "weaknesses": "**1. Limited Analysis of Reward Hacking:** While Table 1 shows some reward hacking (e.g., Aesthetic Score degradation in compositional generation), the discussion is superficial. The paper doesn't analyze: (a) why certain metrics degrade while others improve, (b) whether dense rewards exacerbate or mitigate reward hacking compared to sparse rewards, (c) qualitative failure cases where the model exploits the reward function, or (d) potential solutions. Given that reward hacking is a critical concern in RL-based alignment, this deserves deeper investigation.\n\n**2. Generalization Questions:** Several aspects of generalization remain unexplored: (a) Does the method work with different base models beyond SD3.5-M? (b) How does it scale to higher resolution generation or longer sampling trajectories? (c) Does it apply to other generative models (e.g., diffusion models, consistency models)? (d) How does it perform on out-of-distribution prompts or edge cases?\n\n**3. Writing and Presentation Issues:** Minor clarity issues: (a) Notation inconsistency: sometimes $x_t^i$ and sometimes $x^i_t$, (b) Figure 2 could be clearer about which components run during training vs. inference, (c) The connection between ODE denoising steps n and timestep t is confusing (\"n may be any integer in [1,t]\" but then \"we set n=t\"), (d) Some experimental details are missing (e.g., learning rates, number of training iterations to convergence, batch sizes).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E28F5lghPY", "reviewer": "Reviewer_9Dzz", "summary": "Recent GRPO-based flow matching models improve human preference alignment in text-to-image generation but still suffer from sparse rewards, where a single terminal reward is applied to all denoising steps. DenseGRPO addresses this by introducing dense, step-wise rewards that evaluate each denoising step via an ODE-based reward model. Furthermore, it adaptively calibrates the sampler’s exploration based on reward feedback, ensuring appropriate stochasticity across timesteps and yielding stronger alignment performance.", "questions": "- Why does the noise scale aim to guide the distribution of dense rewards to be balanced? Why not guide the distribution to fall in the positive reward region?\n- In the ablation study 'Effect of Different ODE Denoising Steps', what about using $n=t/C$, where $C$ is a constant? This is proportional to $t$, while faster than $n=t$.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a step-wise reward for RL finetuning.\n- This paper designs an adaptive noise scale.", "weaknesses": "- There is no 'return' in Algorithm 1.\n- There are many metrics in Table 1, and they have different directions for better performance. It will be clearer to add an arrow after them to denote which direction (up/down) is better.\n- Only results of $a=0.7$  are provided in the ablation 'Effect of Exploration Space Calibration'. What about other $a$? Is $a=0.7$ the best hyperparameter?\n- The return defined in reinforcement learning is $R_{t_0}=\\sum_{t=t_0}^\\infty \\gamma^{t-t_0} r$, so the reward is not $R_{t+1}-R_t$. \n- There is no code repository and reproducibility statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)