TEMPFLOW-GRPO: WHEN TIMING MATTERS FOR GRPO IN FLOW MODELS
Authors: Xiaoxuan He
Siming Fu
Yuke Zhao
Wanli Li
Jian Yang
Dacheng Yin
Fengyun Rao
Bo Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: GRPO; Flow Matching
Primary area: generative models

Abstract: Recent flow matching models for text-to-image generation have achieved remarkable quality, yet their integration with reinforcement learning for human preference alignment remains suboptimal, hindering fine-grained reward-based optimization. We observe that the key impediment to effective GRPO training of flow models is the temporal uniformity assumption in existing approaches: sparse terminal rewards with uniform credit assignment fail to capture the varying criticality of decisions across generation timesteps, resulting in inefficient exploration and suboptimal convergence. To remedy this shortcoming, we introduce TempFlow-GRPO (Temporal Flow-GRPO), a principled GRPO framework that captures and exploits the temporal structure inherent in flow-based generation. TempFlow-GRPO introduces three key innovations: (i) a trajectory branching mechanism that provides process rewards by concentrating stochasticity at designated branching points, enabling precise credit assignment without requiring specialized intermediate reward models; (ii) a noise-aware weighting scheme that modulates policy optimization according to the intrinsic exploration potential of each timestep, prioritizing learning during high-impact early stages while ensuring stable refinement in later phases; and (iii) a seed group strategy that controls for initialization effects to isolate exploration contributions. These innovations endow the model with temporally-aware optimization that respects the underlying generative dynamics, leading to state-of-the-art performance in human preference alignment and text-to-image benchmarks.

Published as a conference paper at ICLR 2026
TEMPFLOW-GRPO: WHEN TIMING MATTERS FOR GRPO IN FLOW MODELS
Xiaoxuan He 2 2- Siming Fu'; Yuke Zhao'; Wanli Li', Jian Yang', Dacheng Yin Fengyun Rao? Bo Zhang!
ZheJiang University.
2 WeChat Vision, Tencent Inc
TempFlow GRPO
Images generated by our proposed TempFlow-GRPe with FLUX. It substantially improves the baseline models, achieving superior photorealism and enhanced fine-grained detail.
Figure 1:
ABSTRACT
Recent flow matching models for text-to-image generation have achieved remark- able quality, yet their integration with reinforcement learning for human prefer- ence alignment remains suboptimal, hindering fine-grained reward-based opti- mization. We observe that the key impediment to effective GRPO training of flow models is the temporal uniformity assumption in cxisting approaches: sparse ter- minal rewards with uniform credit assignment fail capture the varying criticality decisions across generation timesteps, resulting in inefficient exploration and suboptimal convergence. To remedy this shortcoming, introduce TempFlow- GRPO (Temporal Flow-GRPO), a principled GRPO framework that captures and
Egual Contribution.
1 Project Leader.
Corresponding authors.

---

Published as a conference paper at ICLR 2026
exploits the temporal structure inherent in flow-based generation. TempFlow- GRPO introduces three key innovations: (i) a trajectory branching mechanism that provides process rewards by concentrating stochasticity at designated branching points, enabling precise credit assignment without requiring specialized interme- diate reward models; (ii) a noise-aware weighting scheme that modulates policy optimization according to the intrinsic exploration potential of each timestep. pri- oritizing learning during high-impact early stages while ensuring stable refine- ment in later phases; and (iii) seed group strategy that controls for initialization effects to isolate exploration contributions. These innovations endow the model with temporally-awan
optimization that respects the underlying generative dy- namics, leading to state-of-the-art performance in human preference alignment and text-to-image benchmarks.
INTRODUCTION
While text-to-image diffusion models have achieved unprecedented visual quality and semantic con- trol (Esser et al., 2024; Xie et al. 2024a Labs, 2024), aligning their outputs with human preference remains formidable challenge. Reinforcement learning (Shao et al.) 2024]] Wu et al. 2025b) has emerged as a promising solution, giving rise to the field of Diffusion RL (Wallace et al. 2024 B Black et al. 2023 Fan et al. 2023). However, the performance of these methods remains suboptimal, hindered by two fundamental limitations that have been largely overlooked: ignoring the temporal dynamics of generation and lacking intermediate feedback signals. These approaches apply uni- form optimization across all timesteps and provide rewards only at completion, missing the varying importance of decisions throughout the generation process.
The majority of existing approaches (Gu et al. 2024 Hong et al. 2024), including recent works like Flow-GRPO (Liu et al.) (025) and DanceGRPP (Xue al. 2025). treat the multi-step gener- ation process as a "black box" with temporally agnostic optimization. They apply uniform updates across all timesteps despite the fact that each timestep operates under different noise conditions and contributes differently to final image quality. Specifically, we plot the Figure 2 (left) with applying SDE only one timestep in the entire ODE trajectory. which ensures that any deviation in the final reward can be attributed to the stochastic exploration introduced at that specific step. As shown in Figure 2 (left,, the std of the reward varies dramatically between timesteps, reaching a peak during early structural decisions (steps (-2) and approaching zero during final refinements (steps 6-8). Yet Flow-GRPO maintains uniform treatment throughout, squandering high-impact exploration oppor- tunities. Alternative approaches like SPO (Liang et al. (024)) attempt address temporal dynamics through process reward models, but training such models on semantically ambiguous intermediate states is notoriously difficult. This raises fundamental question: how can effectively achieve precise credit assignment for intermediate actions while adapting optimization intensity to each timestep's exploration capacity?
We address these limitations with TempFlow-GRPO, a temporally-aware RL framework built on two key insights. First, we define the visualization method in the left of Figure 2 as trajectory branching. which enables precise credit assignment by strategically introducing stochasticity at in- dividual timesteps while maintaining deterministic evolution elsewhere (Figure 2 Right). This pro- vides provable guarantees: (1) reward variance localizes to the branching point, (2) improvements are directly attributable specific exploration outcomes, and (3) existing reward models require no modification. Second, noise-aware policy weighting modulates optimization intensity based on each timestep's is intrinsic noise level. Early high-noise stages receive larger weight updates to encourage structural exploration, while late low-noise stages receive gentler updates to preserve learned fea- tures. Third, we introduce a seed-level grouping strategy that controls for initial noise influence, ensuring reward variations attributed to branching exploration rather than random initialization. Together, these mechanisms create a framework that is conceptually simple, computationally effi- cient, and seamlessly integrates into existing flow matching architectures all while respecting the temporal dynamics that uniform approaches ignore.
Our main contributions are threefold:
2

---

diffusion mod- Published as a conference paper at ICLR 2026
1.4
PickScore Std
0.8
0.6
0.0
0
2
Timestop
SDE Explocaton
ODE Tnjeetury
Figure 2: (Left) Reward Variance Analysis: We plot the standard deviation of PickScore each denoising step for 200 prompts, per prompt group size is 24. The results, obtained via applying SDE only one step, reveal that reward variance highest the initial steps, indicating that early- stage interventions are most impactful for exploration. (Right) Method Illustration: By branching stochastic (SDE) exploration from a specific, known state on deterministic (ODE) trajectory, the resulting difference in the final reward can be unambiguously attributed to the exploration action taken at that precise branching point.
We pinpoint temporal uniformity- the the equal treatment of all timesteps- as the primary limitation of flow-based GRPO. Our proposed TempFlow-GRPO overcomes this by intro- ducing two key innovations: precise credit assignment to intermediate actions and noise- aware adaptation of optimization intensity.
We introduce trajectory branching and noise-aware reweighting mechanisms learn lemporally-structuree policies that respect the inherent dynamics of generative models. Ad- ditionally, we propose an efficient seed group strategy that effectively isolates exploration effects and considerably enhances overall performance.
standard text-to-image benchmarks, achieving superior sample quality, human preference alignment, and compositional image generation compared to existing flow-based RL methods.
We demonstrate state-of-the-art performance
2 RELATED WORK
Alignment for Diffusion Models. Alignment for diffusion models has emerged key research area. D3PO (Yang et al., (024) introduces Direct Preference for Denoising Diffusion Policy Opti- mization to directly fine-tune diffusion models. Diffusion-DPO (Wallace et al. 2024) adapts Direct Preference Optimization (DPO (Rafailov et al.] 2023)) as a simpler RLHF alternative that directly optimizes policies satisfying human preferences. DyMO (Xie & Gong. 2025) proposes training- free alignment method for inference-time preference alignment. Flow-GRPO (Liu et al. 2025) first integrates online reinforcement learning into flow matching models. However, Flow-GRPO applies uniform optimization across timesteps and suffers from sparse terminal rewards, ignoring time- varying exploration potential in stochastic processes. Our TempFlow-GRPP addresses these limi- tations through trajectory branching for precise credit assignment and noise-aware policy weighting aligned with natural exploration capacity.
Process Reward. Shaping reward processes beyond sparse terminal rewards significantly improves learning and policy performance. Zhang (Zhang al. 2025) combines response-level and step-level evaluation metrics. ThinkPRM (Khalifa et al.) (0255 builds data-efficient process reward models (PRMs) using verification chain-of-thought for step-wise verification.
els, SPO (Liang et al. 2024) trains step-aware preference models for both noisy and clean images. However, PRMs require expensive step-level supervision. Methods like PRIME (Cui et al., 2025) enable online PRM updates using only outcome labels through implicit process rewards. Math- Shepherd (Wang et al. 2023) assigns reward scores to solution steps. Given challenges in scoring noisy images, algorithms enabling process rewards for flow models needed. Our TempFlow-
3

---

Published as a conference paper at ICLR 2026
TempFlow-GRPO
Pick
24
Flaw-GRPO (Prompt)
Flow-GRPO
22
200
400
600
800
1000
1200
0.85
Step
Geneval
0.80
0.75
TempHlow-GRPP
PickScore
24
Flow-GRPO (Promept) Flow-GRPO
1xEfffiiency
0.70
23
TempFlow GR20
FLUX.1 Dev (0.66)
Flow-CiRPO (Promp)
0.65
SD3.5-Medium (0.63
22
Flew-GRD)
0
200
400
600
800
1000
1000
2000
3000
4000
Training Time (GPU Hours)
Step
Figure 3: (Left) Performance comparison the PickScore benchmark across training steps and GPU hours. Flow-GRPO (Prompt) represents an improved baseline with group-wise standard deviation stabilization. TempFlow-GRPO consistently outperforms both Flow-GRPO variants in both sample efficiency (steps) and computational efficiency (GPU hours), demonstrating superior training efficiency while achieving the best performance. (Right) On the Geneval benchmark, TempFlow-GRIP achieves the highest performance, significantly outperforming Flow-GRPP and surpassing state-of-te oottt-o-----rrrrr- models including GPT-4o, FLUX.1 Dev, and SD3.5-Medium.
GRPO circumvents specialized process reward models by directly attributing outcome-basee sig- nals to intermediate actions, enabling precise credit assignment without computational overhead of training step-level evaluators for semantically ambiguous states.
3 PRELIMINARY: FLOW-GRPO
GRPO. RL aims to learn policy that maximizes the expected cumulative reward. GRPO optimizes the policy model by maximizing the following objective:
JFlow-GRPO (0) [x- Boo.ccoo A,0
(I)
T-1
[(r, Ar,A,0,e,, G
3Dxr 3Tol(TTllTTe
1=1
1=
x c)
r)(
T is the timestep.
Po... (x))
| c)
Given a prompt c, the flow model pe samples group of G individual images { and the corresponding reverse-time trajectories {(x), 1/- ) Then, the advantage of the i-th image is calculated by normalizing the group-level rewards as follows:
A R(x) c) mean( (R(()) e))
(3)
std([R(z), e))
Convert ODE to SDE. Flow-GRPO converts the deterministic ODE into equivalent SDE that matches the original model's marginal probability density function at all timesteps. The ODE and SDE is follows:
dx: v.dt.
4

---

Published as a conference paper at ICLR 2026
%
X
R
ODE
ODE
X
R
SDE
SDE
(
-vV Aee
0&V
Ale
ooy Ate
Xx
X1
X1
Xo
==
ODE
t 0
Figure 4: Overview of TempFlow-GRPO Framework. Our method performs trajectory branching by switching from ODE to SDE sampling at selected timesteps (t=k, j. i), injecting noise o a.v Me to create exploratory branches. Each branch generates a distinct outcome with reward R,, enabling precise credit assignment. The framework applies noise-aware weighting where w; W/ was prioritizing optimization at high-noise early stages (larger circles) over low-noise refinement phases (smaller circles), aligning learning intensity with each timestep's intrinsic exploration capacity. We visualize the model's 's learning process as an astronaut exploring unknown planets: in early stages, the model explores vast possibility spaces with high uncertainty, while later stages involve focused navigation toward the final destination.
D++Ar Ft + (m0, 1) + o (x + (1 t) 1)) + av Ale
(5)
2t
where e -0, I) injects stochasticity and 0r av the reference policy Teer is closed form:
T And the KL divergence between πī and
A1,0 Te+ At.
Al oe 1-)
D (x (TTe)
2t +
P1l1o (x) (6)
207 At
2
4 METHODS
4.1
TEMPORAL FLOW-GRPO
Flow-GRPO advances online RL for flow matching but overlooks time-dependent generative dy- namics. We identify two key limitations: (1) Sparse Terminal Rewards: uniform credit assignment across timesteps fails to distinguish eritical early decisions from later fine-tuning: (2) Uniform Op- timization: ignoring non-unin non-uniform noise in SDE sampling. where high-noise early steps offer greater exploration potential than low ooin refinement phases (Xie Gong (025). To address these issucs, propose HempFlow-GRPe (Figure introducing process rewards via trajectory branching and noise-aware policy weighting.
4.1.1 TRAJECTORY BRANCHING FOR PROCESS REWARDS
Traditional process reward methods require specialized reward models evaluate noisy intermedi- ate states which is exceptionally difficult due to the semantic ambiguity of partially-denoiser representations. We propose an elegant alternative that leverages the interministio-stccasstt sam- pling methods of flow matching models.
Key Insight: Instead of training complex process reward models, we use existing high-quality outcome-based reward models and directly attribute their scores to intermediate exploratory actions through a novel trajectory branching mechanism.
We define trajectory branching operation where trajectory evolves deterministically until a designated branching timestep where 24 is obtained from ini- tial noise Fy using Equation
Proposition (Trajectory Branching):
4 At this branching point, stochasticity is introduced via noise variable e in Equation 5 yielding Øk-1 SDE (x E). The remainder of the trajectory Tk-2; Fk-3....,
generated deterministically as Zo ODE* k- (xx-1), * denotes & times ODE.
Theorem (Credit Localization): Since all stochasticity and model controllability concentrated at the branching point, the total reward variance and all pnrameter-dependeer improvements are en- tirely attributable to the outcome of noise injection at k. This enables rigorous and efficient credit
5

---

Published as a conference paper at ICLR 2026
Normalized Value
1.75
Noise Level
0.8
w/o reweight
Scale Term
1.50
Reward Std
0.6
w reweight
1.25
reweight, flow shift =1
0.4
1.00
0.75
0.2
0.50
0
5
8
2
5
8
Timestep
Timestep
Figure 5: (Left) Strong correlation between reward standard deviation and noise level across gen- erative timesteps. (Right) Scale term analysis reveals fundamental mismatch in standard GRPO: scale terms are inversely proportional noise levels, causing low-noise refinement steps to domi- nate optimization despite minimal impact on image content.
assignment localized to the branching point.
In practice, we replace the reward for the k-th step from R(x e) to "(DDE* e')), c)), where æi is sampled with SDE and "SDE<() <¹)) is sampled with ODE-SDE-ODE, Tra- jectory branching allows for the precise attribution of the terminal reward to step k. effectively creating temporally-awant reward signal.
4.1.2 NOISE-AWARE POLICY WEIGHITIGG
While trajectory branching provides precise reward signals for individual timesteps, the generative process consists of sequence of I'' potential branching points with fundamentally diffeerent char- acteristics. The SDE sampler exhibits time-varying stochasticity: the noise injection magnitude A.vAt is large during initial generation stages and diminishes to near zero during final refinement stages. This non tni-unfform noise distribution implies that exploration capacity varies dramatically across timesteps. An exploratory action at an early stage has vastly different impact and risk com- pared perturbations near-perfect images. However, standard GRPO applies uniform optimiza- tion pressure, implicitly assuming equal learning importance at all stages.
Reweighting by Noise Level. We wonder if the level of the exploration space itself serve proxy for a reweighting factor. We visualized the noise level alongside the reward standard deviation (Figure 5 left) and observed striking correspondence between the two. This strong correlation suggests that the noise level, serves as an excellent and intrinsic proxy for the exploration capacity and associated risk at each timestep.
We therefore propose to reweight the policy loss directly using the noise level. For each timestep f, we introduce weighting factor proportional to the noise level. Specifically, we modify the original policy loss function to the following weighted form:
Jpolicy (0)
G G
Σ 1=
Norm(o, v (t)( (l)p ())
The intuition behind this weighting strategy is to align the optimization pressure with the inherent properties the generative process. In the early stages generation, noise large, amplifying the learning signal during these high-noise, high-impact phases and encouraging the model to perform effective macroscopic exploration. As generation proceeds, noise diminishes, shifts the optimiza- tion focus towards fine-grained adjustments and stability, preventing aggressive exploration from corrupting high-fidelity state with noise or artifacts.
4.2 POLICY GRADIENT-BASED THEORETICAL JUSTIFICATION
Consider generative process parameterized by 8, the policy gradient can be written as:
VoJ(O) = Vo
(8)

---

Published as a conference paper at ICLR 2026
This yields:
7-1 VoJ(0) Σ
(=+
A 1 k) e
E Bum--(00
k)Ap
(9)
k
k=0
Scale Term
This reveals that the natural gradient coefficient is proportional to V
Ak(1-k] which captures the
intrinsic exploration potential at timestep k. After reweighting, we have the following derivation:
((2 +
(a+z2)
VoJ(Ot Σ
a
Ak e VovoOO A)
(10)
Scale Term
where [c(eAA]] 1lok indicating that the norm of TE, [cAA]] is invariant among the timesteps. Ad- ditional details about these equations are provided Appendix A.I In Equation Jand Equation [0] the scale terms that modulate how each timestep's 's model gradient Ve (K), k) contributes to the
overall reward gradient simplify distinct functions, which denote AR(1-A) and Ak. We visualize these scale terms under different flow shifts in (right)) The "w/o reweighting" set- ting exhibits highly imbalanced gradient contributions across timesteps, where early denoising steps performing broad structural exploration contribute significantly less than late steps focused on fine- grained refinement. Our noise-aware policy reweighting mitigates this issue by simplifying the scale term to be proportional to step size Ak. When flow shift equals I, our method achieves perfect equi- librium with equal gradient contributions from all timesteps. completely balancing the effect across the generation trajectory. Additional details about this section are provided in Appendix A.1
4.3 GROUP STRATEGY
framework employs group-level strategy, where trajectories are grouped based on a shared prompt. To mitigate issues of overfitting and instability, Reinforce++ (Hu al. 2025) later introduced batch-level normalization.
Batch Group & Prompt Group. The original GRPO (Shao et al. 2024
Seed Group. In our work, trajectory branching requires K distinct explorations at each timestep. Furthermore, propose novel seed-level grouping strategy. Under this approach, trajectories originating from the same prompt are further grouped if they share an identical initial noise. This methodology effectively controls for the influence of the initial noise, thereby ensuring that varia- tions in reward can be attributed solely to the exploration conducted during the branching process. The experimental results, presented Figure validate our approach. As shown, TempFlow- consistently achieves superior performance compared to GRo-G regardless of the grouping strategy employed.
EXPERIMENT
Following Flow-GRPO, validate approach on the Com- positional Image Generation benchmark Geneval (Ghosh al. 2023) and the Human Preference Alignment task in PickScore (Kirstain et al.) 2023). To ensure a fair compar- ison, our experimental setup is kept consistent with that of Flow-GRPO. For instance, we normalized the weights ap- plied the policy loss to have mean of 1 at all timesteps. While our method introduces trajectory branching strategy, we maintain an identical group configuration by setting ini- tial noise seeds and a branching factor (K) of 6. This re- sults in a total group size of 24 (4 seeds x 6 branches) and 48 groups, perfectly matching the setup in Flow-GRPO. Fur- thermore, to benchmark our method against concurrent work, conduct additional experiments the FLUX.1-dew model. More detailed experimental settings (include comparsion with DanceGRPO) can be found in the Appendix A.2
345
PickScure
23.0
22.5
TexpfTon GRrO Swet
Tempflow-GRro
Hew.dero
How-Grro
200
400
600
200
1860
1200
Step
Figure TempFlow-GRu GRPO different group strategy.
6: Performance of and Flow-

---

Published as a conference paper at ICLR 2026
Table I: GenEval Result. Best scores are in blue second-best in green from Flow-GRPO. Obj.: Object; Attr.: Attribution.
Results for models are
Model
Step Overall
Single Obj-
Two Obj-
Counting 1
Colors 1
Position
Attr. Binding
Difiusion
0.70
0.02
SDL
0.43
0.97
0.38
0.35
0.76
0.04
0.06
SD2
Kommen Mi bie
0.50
0.98
0151
0.44
0.85
0.07
0.17
SD-XL
Podeilerd 2013
0.55
0.98
0.74
0.39
0.85
0.15
0.23
DALLE
DALLE-3
Belker 6
0.67
0.96
0.87
0.47
0.83
0.43
0.45
Antoregeezivee Models
Show-o Xal 2024b
0.53
0.95
0.52
0.49
0.82
0.11
0.28
Emm3-Gam
0.21
Mare
0.45
0.53
Janu--ro
7 hen
2025
0.80
0.99
0.89
0.59
0.90
0,79
0,66
GPT-4o (Hurst
0.84
0.99
0.92
0.85
0.92
0.75
0.61
Flow Matching Modela
FLUXXI
Des (1 Labe N14
0.66
0.74
0.79
0..22
0.45
SD3
0.34
SANA-1 438 Ni e 2025
(1.81
1399
0.93
0.86
0184
0.59
0.65
SD3.5-M
(Eser
0.63
0.50
0.24
0.52
GRPO Methoass
SD3.5-M-- 333.5-M-Foon-GAA GRPO
2025
5600
0.955
1.00
0.99
0.95
0.92
0.99
0.86
SD3 MM-------RPPP
li et
M123
3800
0.88
0.99
0.96
0.90
0.87
0.88
0.78
5.1 MAIN RESULTS
Compositional Image Generation. We evaluate the compositional image generation capability of TempFlow-GRPO on the Geneval benchmark with its corresponding reward model. The ex- perimental results summarized in Table [ As shown, approach significantly improves the performance of the base model, increasing the overall score from 0.63 to 0.97. Furthermore, among GRPO-based methods, our method substantially outperforms Flow-GRPO: it achieves per- formance of 0.97 within only 3,800 steps, whereas Flow-GRPO reaches only 0.88 under the same conditions. Additionally, as illustrated in Figure 3 our method requires only about 2,000 steps to achieve score of 0.95, while Flow-GRPO needs approximately 5,600 steps to reach the same level. Overall, these results demonstrate that TempFlow- GRRPO not only accelerates convergence but also achieves superior final performance compared to existing approaches.
Human Preference Alignment. To further validate the generalizability of our approach, we con- ducted experiments on the PickScore benchmark, using PickScore as the reward model. As shown in achieves the highest performance, surpassing the original Flow- GRPO by approximately 1.7% and outperforming the improved baseline Flow-GRPO (Prompt) by about 1.0%. Notably, our method requires only 100-200 training steps to match the performance of Flow-GRPO, and just 300-400 steps reach the level of Flow-GRPO (Prompt). These results PickScore further demonstrate the general applicability of our method as a unified flow-based RL. algorithm across different reward models.
Figure (left), TempFlow-GRe
333
TempFlow-GRPO
Loss
0.003
TempF low-GR
Flow GRPO (Prompt)
Flow COw-GRPO (Prompt)
0.001
0.000
100
150
250
300
50
100
150
200
250
300
Step
Step
Figure 7: The performance and KI. loss (smoothed) of HPSv3 on PickScore dataset (FLUX.1-dey)
HPSv3 on FLUX.1-dev. To further validate the performance of TempFlow- -RPPOO across differ- ent model and reward models, select FLUX. 1-de our base model with 1024 resolution and employ HPSv3 (Ma et al., (2025b) as the reward model. As shown in Figure 7 which presents the results, method requires only 80 steps to match the performance that Flow-GPO achieves in 300
8

---

Published as a conference paper at ICLR 2026
PickSccre
PickScore
24
Geneval
Creup
Reneigt
Tal
0.8
Five GLTD (Poopt)
Vempliwe GRPO
23
Roweight
Tepp GRMO
Brench
0.7
Trapelow
TIO0-TRPO (Prorge)
200
400
600
800
1000
1200
Step
Step
Step
Figure 8: Ablation studies trajectory branch and noise-aware policy reweighting.
steps. Moreover, our approach maintains a lower and more stable KL. loss. Further experimental results and visualizations for the FLUX model can be found in the Appendix. A.3
5.2 ANALYSIS
Ablation of Trajectory branching. In our trajectory branching strategy, branching is performed each step. To ensure fair comparison with Flow-GRPO, maintain constant total group size of 24. Therefore, we tested several branching configurations: 2 initial noises with 12 branches each (2x 12), 4 initial noises with branches each (4x6), and 6 initial noises with branches each (6x4). The experimental results are shown in the left of Figure X The figure indicates that larger number of initial noises helps to accelerate performance improvement the early stages of training. However, training progresses, higher number of branches yields better results. To strike balance, we ultimately selected 4x6 configuration as the default setting for our paper.
FLUX.1-dev
(row)RRP((roopr))
TempFlow-GRPO
FLUX.1-dev
(Oo-GR)PO(rmmot)
TempFlow-GRPO
Beautiful petite femole hand with almond long french noils
A man and
hugging each other during sunset, colorful, depth field, best quality
An exquisite fine lace motherboard, GPU, circuitry, radial symmetry, black background
Diamond ferrani, made entirely of diamand mirror glass
selfie photo monkey bronch www.sing.wwerllldooooooo overlooking mxssivv rainforest
of dinasour can, inspired by Adam **cinematic, 1993, heartbreaking, promo image, oction shot, an ultra realistic
Figure 9: Qualitative comparison between FLUX.1-dew Flow-GRPO (Prompt) and TempFlow- GRPO with HPSv3 as reward on PickScore prompts.
Ablation of TempFloo-GRRO. We conducted ablation studies to explore the effectiveness of our proposed components: the trajectory branch, noise-aware policy weighting and seed group. These ablations were performed on both the Geneval and PickScore benchmarks. As shown in the second and third subfigures of Figure 8 the PickScore benchmark, introducing the trajectory branch further improves the performance of Flow-GRPO (Prompt), and applying noise-aware reweighting on top of this further boosted the performance. Finally, achieved the highest performance by

---

Published as a conference paper at ICLR 2026
applying seed group. On the Geneval benchmark, the benefit of the noise-aware strategy is more significant: compared to Flow-GRPO, noise-aware policy reweighting boosts performance from 0.82 to 0.92 in 1200 step, 10% improvement while the trajectory branch also brings about a substantial gain of approximately 5% and seed group achieves 2% improvement. These ablation results clearly demonstrate the effectiveness of proposed methods.
Qualitative Result. We also conducted qualitative analyses on the PLUX.1-der' Flow-GRPO (Prompt), and TempFlow-GRPO. As shown in Figure [ compared Flow-GRPO (Prompt), TempFlow-GRPO produces images with noticeably finer details and fewer visual artifacts or mis- takes. In particular, our approach demonstrates superior capability in preserving complex struc- tures and realistic textures. These qualitative improvements further highlight the advantages of our method in generating high-quality, Visually appealing images.
CONCLUSION
We presented TempFlow-GRPO, a temporally-aware reinforcement learning framework that ad- dresses fundamental limitations in existing flow-based GRPO methods. Through trajectory branch- ing, we enable precise credit assignment to intermediate actions without requiring specialized pro- cess reward models. Through noise-aware weighting, ensure that optimization intensity matches each timestep's exploration potential. We also introduce principled seed group strategy that controls for initialization effects. Our extensive experiments demonstrate that TempFlow-GRPO achieves SOTA performance on human preference.
Limitations. Although our method achieves significant improvements in both performance and image quality, the current experiments focus primarily on algorithmic innovations rather than reward model enhancements. In future work, we plan to explore how to effectively incorporate multi- modal rewards from more powerful foundation models and develop comprehensive reward frameworks, aiming enhance performance across multiple evaluation dimensions.
Use of LLMs. We utilize LLMs to assist with formula derivations and writing refinement.
ACKNOWLEDGEMENTS
This work was partially supported by the National Natural Science Foundation of China under Grant No. 62402434.
REFERENCES
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallee3. pdf. 2(3):8, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 2023.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. /78/ 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Ying Fan, Olivia Watkins, Yuging Du Hao Liu, Moonkyung Ryu, Craig Boutilier, Picter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for
10

---

Published as a conference paper at ICLR 2026
fine-tuning text-to- image diffusion models. Advances in Neural Information Processing Systems, 36:79858 7.779888-77885, 2023.
Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpo: Aligning dif- fusion models through relative preference optimization. arXiv preprint arXiv:2406.06388. 2024.
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thome, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness both prompt and reward models. arXiv preprint arXiv:2501.03262 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moon- tae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint arXiv:2504. 16828, 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36662-36666666 2023.
Black Forest Labs. Flux.
http:///ttttbb.coooocaccooo-aa hartaacc--ooretttt ffuxx 2024.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng- Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint 2444:2240000.34....2 2(5)77, 2024.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint xxXv:22505.05400. 2025.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, et Janusflow: Harmonizing autoregression and rec- tified flow for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 7739-7751, 2025a.
Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. arXiv preprint arXiv:2508.03777 2025b.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955. 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 011557228-37444 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2XXi:22044006255 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint 20Xv::240..03300, 2024.
11

---

Published as a conference paper at ICLR 2026
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 9228-8238, 2024.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Wath-shepherds Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.08953 2023.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint ar www::::0...8800000 2024.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng- ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.0233 2025a.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng. Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Miung-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint 250.00500,0 2025b.
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, al. Sana: Efficient high-resolution image synthesis with linear diffu- sion transformers. arXiv preprint arXiv:2410. 10629, 2024a.
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of training-time time and inference- time compute in linear diffusion transformer. arXiv preprint arXiv:2501.18427 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024b.
Xin Xie and Dong Gong. Dymo: Training-free free diffusion model aligmment with dynamic multi- objective scheduling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13220-13230, 2025.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo visual generation. arXiv preprint arXiv:2505.078/8 2025.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons developing process reward models in mathematical reasoning. arXiv preprint 2501:.250....301, 2025.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 POLICY TRADIENT-BASER THEORETICAL FRAMEWORK
To provide deeper understanding of our approach, we now examine it from the policy gradi- perspective. Note that in the summation Σ the index denotes the timestep, while in subsequent equations, k represents the timestep value. Simplifying Equation 5 #k-1 N (po(a,, k), of A))),, where:
we obtain
of (ora (1 () k)) Ak
po (xk, k) 2h (0o(
(11)
21:
Starting from the policy gradient formulation Equation 8 we have:
T
VoJ(O))
Σ
(12)
Substituting xt-1 in the log- probbabbbbil:t:
(- [k-1 1o ( 1i)1
VoJ(0)
E No.
Va log exp
Av
(13)
A4:
Vo (- | - (0o 1:)11
Akk
(14)
20 Ak
Taking the gradient with respect to 8:
T-1
Øk-1 ps ( (OH 1:) k)
VoJ(
E
N(.1))
(15)
of Ak
k=
Since th-1 ua( k) on Ak where E ~ N(0, I):
T-1
VoJ(O) =
Σ 0=
E N
VoHo(I A)
(16)
Expanding (2), k):
of
Vona( k) Vo
2h vo(2xx k.)
(æt + (1 k)oo( h))) Ak
(17)
2k
o2(1
k)
V
Ve (a 1))
Uo v (Ik k)) Ak
(18)
2k
o2(1 (1 k)
Ak (x k)
(19)
2h:
Substituting back:
T-1
22 k)
VoJ(A)
Eor--NO.
c
1
Ak (o
(20)
2k
0m
VAk
σk v Ak(1 - -k)
E (o..NN(0..))...NV..
(21)
Ok
2k.
K=
13

---

Published as a conference paper at ICLR 2026
With 0 = av
A I-
we get:
A
A
Ak(1 k)
(22)
k
KK V Ak(1 b)
VV I-K
V 1s)
0
Ak(1 k)
(23)
2k
2k
2
k
Therefore:
T-1
(a (1+2)) 2
Ak(1 k)
VoJ(O)= = Σ Noy-NNoo
k c (2
(24)
Scale Term
This reveals that the natural gradient coefficient is proportional to V L VAk, which captures the intrinsic exploration potential at timestep k. After reweighting, we have the following derivation (don't consider and norm coefficient):
(
VoJ(O)
B Wir.-woo
+ (
a
Ak E
(evo( K)Aa
(25)
k=
Scale
Consider E.~N((, IcA.)), suppose the final reward is function of the small noise vector Øμ / Akee applied at a certain step. When de σ&V Ake is small (and drawn from a zero-mean Gaussian), we can approximate the reward using first-order Taylor expansion:
Ry(ox (a(ovVAke)
R2 (0) ax Ake T
OLVALE Ra RelorvATEED
(26)
Since A is normalized version of Rg, the mean and std are as follows (let Vov'kke Bolor gk)):
mean E oy P ARe RI (0) + V Akke gul R0)
std
B
mean)
(27)
B BoLVAne (oo.AeellOVAACA AKE gk)
= o v Ak|loul1
Therefore:
A Rs mean
T V AKE
std
O
Ak
[g*||
(28)
Eg [EA = Esl EEE gk:
gk 1gg
|19111
Equation 28 indicates that the norm of E. [cAAA is invariant among the timesteps.
A.2 EXPERIMENTAL SETTING DETAILS
Dataset. We evaluate Compositional Image Generation on Geneval and Human Preference Align- ment PickScore. Meanwhile, also test method HPDv2 (Wu et al. 2023).
Reward Model. In TempFlow-GRB GRPO, we introduce four reward models, including clip-based PickScore, HPSv2 (Wuetal 2023), VLMM-assed HPSv3 (Ma et al. (2225)) and framework Geneval.
14

---

Published as a conference paper at ICLR 2026
Human Preference Alignment. Following Flow-GRPO, evaluate method PickScore
prompts using the PickScore reward function. PickScore is a CLIP-based scoring function that demonstrates superhuman performance in predicting human preferences for generated images. To ensure consistency with Flow-GRPO, we set the KI. divergence regularization coefficient ,3 0.001.
Compositional Image Generation. We adopt the GenEval framework following Flow-GRPO's experimental protocol. The training prompts are sourced from the Flow-GRPO dataset. GenEval provides an object-focused evaluation framework that assesses compositional image properties, in- cluding object co-occurrence, spatial positioning, object count, and color attributes. For fair com- parison, we maintain the KL divergence regularization coefficient at 3 = 0.004.
HPSv3 Evaluation on FLUX.I-dev. For experiments on the FLUX.I-dew model, we configure the noise level at 0.9 for both Flow-GRPO and our TempFlow- GRPP method. The KL divergence weight is set 0.004. During training, we employ 10 sampling steps, while evaluation uses 50 steps to ensure high-quality generation. The batch configuration consists of group size of 24 (arranged as 6) with 48 groups total.
Comparison with DanceG RPO. DanceGRPO and Flow-GRPO represent concurrent developments in this research area. The SDE sampling methodology employed in DanceGRPO corresponds to Equation 12 in the Flow-GRPO formulation. However, the theoretical derivation in DanceGRPO terminates at its Equation without developing an analogous final formulation. To comprehensively validate method's effectiveness, conduct direct comparison with DanceGRPO. For equi- table comparison, we train TempFlow-GRPe for 300 iterations on the HPDv2 dataset using the HPSv2 reward function.
A.3 EXPERIMENTS ON FLUX.1-DEY
We further evaluate the performance our method against Flow-GRPO on FLUX. Iddev, utilizing the HPDv2 dataset with HPSv3 the reward model. As illustrated in Figure IO method demon- strates substantial improvements over the baseline. After 300 training steps, our approach achieves performance gain of approximately 0.3 compared Flow-GRPO. More significantly, method exhibits superior training efficiency: reaches the performance level that Flow-GRPO achieves at 300 steps in merely 80 steps, representing 3.75x speedup in convergence. Additionally, the re- sults indicate that our method maintains lower and more stable KL divergence values throughout the optimization process, suggesting improved training dynamics and better preservation of the original model's 's distribution compared to Flow-GRPO.
TempFlow-GRPP
HPSv3
KL Loss
0.002
Floa-GRPO (Prompt)
14.5
0.001
TompFlow-CRPP
14.0
(Prompt)
0.000
Step
Step
Figure 10: The performance and KL loss (smoothed) of HPSv3 on HPDv2 dataset with FLUX. 1-dev as base model.
A.4 COMPARSION WITH DANCEGRPO
As illustrated in Figure TI (Left), we present the evaluation curves of our method on the HPDv2 dataset. The results demonstrate clear superiority our approach over the baseline. While Dance- GRPO achieves final performance score of 37.2 after 300 iterations, our method attains higher score of 38.5 at 280 iterations, representing a 1.3% improvement over the DanceGRPO. More signif- icantly, approach exhibits substantially enhanced training efficiency: it matches DanceGRPO's final performance (achieved at 300 iterations) in merely 150 iterations, yielding a 2x speedup in convergence. Notably, this superior performance is achieved without any specialized hyperparame-
15

---

Published as a conference paper at ICLR 2026
24.00
0.38
23.75
PR
0.37
23.50
HPSv2
0.36
PickScore
23.25
0.35
23.00
0.34
22.75
0.33
22.50
0.32
22.25
TempFlow-GRPO
0.31
TempFlow-GRPO
22.00
Flow-GRPO (Prompl)
50
100
150
200
250
100
200
300
400
Step
Step
Figure 11: (Left) Performance of TempFlow-GRPO on HPDv2 with HPSv2 as reward compared DanceGRPO. (Right) Comparsion on PickScore benchmark (PickScore reward, SD3.5-M 1024).
ter tuning, as we directly adopt the configuration settings from Flow-GRPO, further validating the robustness and generalizability of our method.
A.5 EXPERIMENTS SD3.5-M WITH 1024x 1024 RESOLUTION
We investigate the effectiveness of our approach at higher image resolutions, employing PickScore as the reward model for evaluation. As illustrated in Figure II (Right), TempFlow-GRE achieves 1.0% improvement in PickScore after 450 training steps compared the Flow-GRPO. More notably, our method demonstrates exceptional training efficiency, requiring only approximately 100 steps to match the performance that result further validates the efficiency and effectiveness of our proposed method across varying image resolutions.
(Prompt) achieves after 450 steps a 4.5x speedup. This
A.6 TRAINING HOURS
discussion of the computational cost of our method is warranted. Due to the K-branch ex- ploration at each timestep. our sampling process incurs higher computational overhead compared to Flow-GRPO. Specifically, for K = 10, the average number branches is approximately (9 +.. + 1)/10 4.5 times that of Flow-GRPO. However, the training time per iteration mains identical to Flow-GRPO Despite this increased per-sample computational cost, our method demonstrates substantially superior overall training efficiency. As illustrated in Figure 3 (bottom left, main paper) and Figure [2 where we plot performance against wall-clock training time, our method exhibits faster convergence across all evaluated benchmarks. This efficiency gain becomes particularly pronounced at higher resolutions, where our approach achieves Flow-GRPO's 'ss final per- formance using only 33% 50% of the total training time.
Furthermore, our approach not only converges faster but also achieves superior final performance. The PickScore experiments presented the main paper reveal that even after extended training of 1200 steps, our method continues to outperform Flow-GRPO providing compelling evidence that our approach possesses a higher performance ceiling.
A.7 ADDITIONAL DETAILS ON GROUP STRATEGY
In Section 4.3 of the main paper, we analyze the impact the group strategy on performance. Fig- ure 6 and Figure 13 demonstrate that TempFlow-GRD achieves substantial improvements when employing an identical group strategy to the bascline, indicating the inherent advantages of proposed modules. To systematically disentangle the contributions of the group strategy
16

---

Published as a conference paper at ICLR 2026
TempFlow-GRPO
TampElow-GPro
Flow-GRPO (Promps)
12.0
Flow-GRPO (Prompt)
0.90
Flow-GRPO
0.85
11.5
Geneval
HPSv3
0.80
11.0
0.75
10.5
0.70
10.0
0.65
500
1000
1500
2000
0
800 1000 1200 1400 Training Time (GPU Hours)
200 400
600
Training Time (GPU Hours)
(a) Training time comparison on GenEval benchmark (Geneval reward, SD3.5-M 512).
(c) Training comparison on PickScore prompts (HPSv3 as reward, FLUX. 1-dev 1024).
23.4
Temphlow-GRPO
15.4
TempFlow-GRPP GRPO
Flow-GRPO (Promps)
Flow-GRPO (Prompt)
23.2
15.2
23.0
15.0
PickScore
22.8
HPSv3
14.8
22.6
14.6
22.4
14.4
22.2
14.2
22.0
200
400
600
800
1000
800 1000 1200 1400 Training Time (GPU Hours)
200 400 600
Training Time (GPU Hours)
(b) Training time comparison on PickScore prompts (PickScore reward, SD3.5-M 1024).
(d) Training time comparison on HPDv2 prompts (HPSv3 reward, FLUX. 1-dev 1024).
Figure 12: Comprehensive comparison training time across multiple benchmarks and resolutions. TempFlow-GRPo consistently achieves superior performance with reduced training time compared to Flow-GRPO baseline.
from our two fundamental components- branching and noise reweighting-we conduct controlled ablation studies. In these experiments, maintain constant group strategy config- uration to ensure fair comparison. The results unequivocally demonstrate that our core modules independently yield significant performance gains. These components not only enable the model to match Flow-GRPO's multi-step performance in substantially fewer iterations but also elevate the final performance ceiling, validating the effectiveness our theoretical framework.
A.8 KL DIVERGENCE ANALYSIS
To further substantiate the training stability of method, we provide comprehensive anal- ysis of the KL divergence dynamics during optimization. The results, presented in Figure [4] clearly demonstrate that our method consistently maintains significantly lower KL divergence values throughout the training process compared to Flow-GRPO. This reduced divergence indicates better preservation of the original model distribution while still achieving effective preference alignment.
17

---

Published as a conference paper at ICLR 2026
23.5
15.0
12.5
23.2
12.0
15.00
PickScore
HPSv3
HPPSv3
14.75
22.4
11.0
14.90
22.4
14.25
Jempllow-CHe GRP0
TempFloo-CRRP
Terpltow-GRP
Teup GR20 (Proap
TenyFlow GR2O (Pwenyt
14.0
Teayflow GRPO (PecppO)
Prorppt)
9.-
1526
Plre-GKPO
140
200 Step
250
150
200
150
150
200
Step
Step
(a) Group strategy analysis on (b) Group strategy analysis on (c) Group strategy analysis on PickScore prompts (PickScore as PickScore prompts (HPSv3 as re- HPDv2 prompts (HPSv3 as re- reward, SD3.5-M 1024).
ward, FI.UX.1-des 1024).
ward, FL.UX.1-dev 1024).
Figure 13: Comprehensive analysis of group strategy impact across different models and bench- marks, demonstrating consistent performance improvements with our proposed method and seed group approach.
0.012
0.0025
0.010
t cartt mype
0.0020
KL KLLoss Loss
0.008
KL Loss
0.006
0.0010
0.004
CempploowGRRP
0.0005
TempFlow-GRPO
0.002
TempFlow-GRPP (Prompt)
TempFlow-GRPO (Prompt)
0.000
Flow-GRPO (Prompt)
0.0000
Flow-GRPO (Prompt)
250
500
1000
1250
0
100
200
300
Step
Step
(a) KL. Loss on PickScore (SD3.5-M 512).
(b) KL Loss on PickScore (SD3.5-M 1024).
Figure 14: Comprehensive analysis of KL divergence dynamics during training across different res- olutions. TempFlow-GRPO consistently maintains lower KL divergence values compared to Flow- GRPO, indicating better preservation of the original model distribution while achieving superior preference alignment.
Furthermore, we analyze the impact of different grouping strategies KL divergence. While adopt- ing the "seed group" strategy results in a marginal increase in KL divergence compared to our de- fault configuration, the values remain substantially below those of the Flow-GRPO baseline across all training iterations. This observation provides strong evidence that our core modules -trajectory enhance training stability and robustly maintain dis-
branching and noise reweighting-inhee tributional proximity, regardless of the specific grouping strategy employed. The consistent stability across different configurations underscores the fundamental advantages of our theoretical framework in balancing exploration and exploitation during the optimization process.
A.9 MULTI-REWARD OPTIMIZATION
We evaluate the performance of our method in multi-reward optimization scenarios. For this ex- periment, employ FLUX.I-dev the base model and train at high resolution of 1024x 1024. We simultaneously optimize for two reward functions- HPPv3 and PickScore-we wwth weight ratio of 1 1:0.26. After 120 training steps, the results presented in Table demonstrate method's
18

---

Published as a conference paper at ICLR 2026
Table Performance comparison for multi-reward optimization on PickScore prompts, demonstrat- ing simultaneous improvement across multiple objectives.
HPSv3
PickScore
FLUX.1-dev
9.42
22.34
Flow-GRPO (Prompt)
10.41 (+0.99)
22.44 (+0.1)
TempFlow-GRPO
12.23 (+1.81)
22.52 (+0.18)
bust capability for multi-objective optimization. Compared to Flow-GRPO, our approach achieves substantial improvement of 0.82 the HPSv3 metric while simultaneously maintaining consis- tent performance gains on PickScore. These results validate that our TempFlow-GREN effectively balances multiple objectives without sacrificing performance on individual metrics, highlighting the versatility of our approach in complex preference alignment scenarios where multiple criteria must be optimized concurrently.
0.90
TempFFav-CCRPP (Seci)
TempFlow
0.8
Flow: GRPO (Prompe)
0.85
TempFTow CRPO (Bexc0)
OCR Score
Geneval
Flow GRPO (Batch)
0.80
0.75
0.70
0.70
TempElow GRPO (Seed)
Tempé GIUO (Prompt)
Floe-GitPO (Pronpp)
200
4001
600
800
1000
100
200
300
400
Step
Step
Figure 15: (Left) Performance of TempFlow-GRPe and Flow-GRPO on different group strategies (Geneval reward, SD3.5-M 512). (Right) Comparsion on Visual Text Rendering benchmark (OCR reward, SD3.5-M
A. GROUP STRATEGY ON GENEVAL
To further validate the robustness of TempFlow- GRPO across different group strategies, conduct additional experiments on the GenEval benchmark. As illustrated in the left panel of Figure [5] the results demonstrate significant improvements with CempFlow-GRPO Specifically, under the batch grouping strategy, TempFlow-GRPO achieves a performance gain of approximately 7% within 1,000 steps. This margin expands to around 10% when employing the prompt group strategy. Most notably, our proposed seed group strategy yields superior performance, attaining a GenEval score of 91% % 1,000 steps. These findings conclusively demonstrate that TempFlow-GRPO is robust and highly effective method, regardless of the chosen group strategy.
A.11 VISUAL TEXT RENDERING
To further demonstrate the superiority TempFlow-GRPO, we conduct a comparative evaluation on the Visual Text Rendering task.
Experimental Setup. Following the configuration in Flow-GRPO, we use SD3.5-M as the base model with resolution of 512x512. The noise level is set to 0.7, and the KL loss weight is set to 0.004. Additionally, we employ PaddleOCR as the reward model. We evaluate both the prompt grouping and proposed seed grouping strategies.
TempFlow-GRB GRPPO exhibits significant performance gains this task. Specifically, TempFlow-Gre GRPO (using the prompt grouping strategy) outperforms the Flow-GRPO baseline, achieving an improvement of approximately 5.5% at around 500 steps. Furthermore, seed grouping variant attains the highest performance, reaching score of ap-
As illustrated the right panel Figure 15
19

---

Published as a conference paper at ICLR 2026
proximately 89.5%, which surpasses the Flow-GRPO baseline by 7%. These results underscore the effectiveness and versatility of the proposed TempFlow-GRPO framework.
24.25
24.00
TempFlon-Git (Prompt)
Templtow GRPO (Seed)
23.75
Tenpfiav-GiPO (hrone))
Tempplary Bead
GRPO (Pringe)
23.75
Farv-CRPO (Proepp))
23.50
Sioke
23.50
Score
23.25
23.25
23.00
22.75
22.75
22:50
22.50
22.25
22.25
0
100
200
300
400
0
250
500
750
1000
1250 1500
1750
Step
Training Time (GPU Hours)
Figure 16: (Left) Performance comparison on the PickScore benchmark across training steps
(Qwen-Image 512). (Right) Performance comparison on the PickScore benchmark across GPU hours (Qwen-Image 512).
A.12 TEMPFL.OW-GRO ON QWEN-IMAGE
We validate TempFlow-GRPO on Qwen-Image Wu et al. (202a)). The experimental setup involves LoRA fine-tuning with KL weight of O and noise level of 1.2, utilizing PickScore as the reward model (at resolution of 512).
The results, illustrated in Figure 16 highlight significant improvements. Notably, even without ad- ditional hyperparameter tuning, our method (configured with 4x6 group size and simple reweight- ing) substantially outperforms Flow-GRPO (prompt). It achieves 1.3-point performance gain over the baseline. Critically, TempFlow-GRPe matches the performance level that Flow-GRPO achieves at 400 steps in only 100 steps. Regarding computational cost, our method requires only 43% of the total training time. This confirms the high efficiency and ease of integration of our proposed approach.
23.6
24.0
23.4
PickScore
23.2
PickScore
23.5
23.0
23.0
22.8
22.6
TempFlow GRPO-Fasti (Seed)
TempFlow (Seed)
22.5
22.4
SemmpEloo-GRPO-EEE (Prompe)
TempF Pea---GRRO-Faattt (Prompt)
Flon-GRPO.FF Fas4 reweight (Prompt)
Flom:GRPO Fasti reweight (Prompt)
Hlow-GRPO-FF (Prompe)
Flow-GRPO-Fase (Prompr)
22.0
100
200
500
400
500
600
100
200
300
400
500
Step
Step
Figure 17: (Left) Performance of TempFlow-GRPO-IIE and Flow-GRPO-Fast in 4 steps (PickScore, SD3.5-M 512). (Right) Performance of GRRP-Fa and Fow--GPOOFFast in 6 steps (PickScore, 512)).
A.13 HYBRID VARIANTS OF TEMPFLOW-GRPO
In this section, introduce variant of Flow-GRPO, Flow-GRPO-Fast designed to accelerate the training process. This is achieved by strategically limiting the SDE formulation to the initial timesteps of the diffusion process. Figure 17 illustrates the performance of two configurations:
20

---

Published as a conference paper at ICLR 2026
24.5
24.0
PickScore
23.5
23.0
TempFlow-GRPO (Seed)
TempFlow-GRRPP-Fartt (Seed)
22.5
TempFlow-GRPP-Faart (Seed)
200
400
600
800
1000
1200
1400
Step
Figure 18: Performance of TempFlow-GRPO-Faa in different steps.
Flow-GRPO-Faste and Flow-GRPO-Fast6, which correspond to training with the first 4 and steps, respectively.
Reweighting on Flow-GRPO-Fast. To validate the effectiveness of our reweighting strategy, we conduct ablation study. In this experiment, omit the trajectory branching and apply the reweighting strategy directly to Flow-GRPO-Fast The results are presented Figure 17 where the left panel shows the 4-step setting and the right panel displays the 6-step setting. As observed, applying the reweighting strategy alone yields considerable performance improvements, particularly in these few-step settings. Notably, this performance gain achieved zero additional computa- tional cost, as the reweighting mechanism introduces overhead to either the training sampling time.
SempFlow-GRPO-Eas Next, we introduce a variant named ShoppFlow-GRP-EER to evaluate the effectiveness of our framework in few-step settings. This variant demonstrates the robustness of CempFlow-GRPO across varying numbers of steps. As depicted in Figure [7] the results confirm consistent improvements. In the 4-step setting, TempFlow-GRPO-Fast (Prompt) and TempFlow- GRPO-Fast (Seed) both achieve a performance gain of 0.3. These improvements become more pronounced in the 6-step setting, where the Prompt and Seed strategies yield gains of 0.3 and 0.5, respectively. These findings validate that HempFlow-GRPO maintains its performance superiority even in computationally constrained, few-step settings. Note that reducing the number of steps negatively impacts overall performance, illustrated in Figure 18| This because each step in the diffusion process contributes to the exploration of the solution space.
14
THE DIVERSITY OF TEMPFLOW-GRPO
Figure 19: The diversity of TempFlow-GRPe FLUX.1-dew (Prompt: "The ground is mixture of melting liquid gold and scattered LEGO bricks. raining colorful jelly beans from the sky. Macro photography, shallow depth of field, hyper-realistic light refraction, tactile textures, unreal engine render, volumetric lighting").
21

---

Published as a conference paper at ICLR 2026
To demonstrate the diversity of approach, present visualizations in Figure 19 indicate that TempFlow- GRPO maintains high diversity while achieving superior prompt alignment and image quality. To assess potential reward hacking, evaluate the PickScore metric models optimized for the GenEval benchmark. After approximately 3,800 steps, the scores for both our method and the baseline decrease by approximately 0.234, which indicates that approach does not introduce additional reward hacking issues. Furthermore, TempFlow-GRe GRRO exhibits lower and more stable KL divergence. We acknowledge that reward hacking remains an inherent challenge in RL-based strategies, and we plan to address this issue from both algorithmic and reward-modellng perspectives in future work.
The results
A.15 ALGORITHM TEMPFLOW-GRPO
Algorithm details the operational workflow of Tempflow-GRPO for single input prompt, specif- ically configured with a N M group size.
Algorithm TempFlow-GGPe (Take a prompt as case)
Require: Prompt group size G(N M). total timesteps T,, reward models R.
Ensure: Optimized policy parameters 0
1: Initialize policy parameters 0, reference policy The
2: repeat
3: // Do this N GPUs
4: // Sample
5: Sample initial noise By (O I) // Ensure the same initial noise
6: for = to 0 do
7:
Tt-1 2; w,dt // Equation // Traj Branch, M is the branch batch size for i to M do
8:
9:
# 2, +
[v 1r, t) + 2 ( + ()vo(ar, + o. Ate Equation
11:
for k T to do
$ -
24 dk // Equation
13:
end for end for
15:
Compute rewards (R(#)) c).)
16:
end for
17:
I/ Compute Advantages
18:
for to do
19:
Compute mean, std of (R(#), MM // The group is initialized with same noise Compute (A(#)) M with Equation
20:
21:
end for
22:
// Training
23:
for = to . do
24:
for f. T to 0 do
25:
// Policc Reweighting
26:
Ci -Jppltyy, DKa, with Equation 6 and Equation 7 end for
28:
end for
L MT M ΣT Cj
29:
30:
A - o
31:
until convergence
A.16
GENERALIZED FORMULATION OF REWEIGHTING STRATEGY
To provide more general understanding of our approach, we derive the generalized reweighting coefficient 3 for arbitrary æ schedules. Based on Equation an] Equation [5] we obtain:
V
(1)
B
Ak
(29)
Ok
2k
a
22

---

1. 16-year-old teenager wearing white bear-ear hat with a smirk on their face. photo well done salmon dinner, 8K, Global Illumination, Ray Tracing Reflections Published as a conference paper at ICLR 2026
23.50
23.25
PickScore
23.00
22.75
22.50
22.25
TempFlioe-GiP0 (Preppp
22.00
Hon-ERPP (Prompt) TTaj Beanch
(Prompt)
21:75
50
100
150
200
250
300
350
400
450
Step
Figure 20: Performance of TempFlow-GRPO under = ak (PickScore, SD3.5-M, 512).
Solving for 3 yields:
a
A
8 =
(30)
2
mal1-k)
24
a
2k A
(31)
2
2k + o - k)
In the specific case of Flow-GRPO, where = av T- this simplifies to:
k
8
A
(32)
Note that Flow-GRPO determines its optimal 0j. by specifically formulating as the Signal-to- Noise Ratio (SNR). According to Equation 31 our method applies to any σы- Consequently, our weighting coefficient remains effective across varying formulations. We demonstrate this flexibility in Figure 20, where our method consistently outperforms the baseline even when dg is set to ak.
A.17
VISUALIZATION
We present comprehensive qualitative comparisons to demonstrate the superior visual quality achieved TempFlow-GRPe method. The visualization results consistently show that approach generates images with enhanced fidelity, better adherence to complex prompts, and fewer visual artifacts compared to baseline methods.
Prompts in Figure 1. The prompts in Figure are as follows:
3. A lemon with McDonald's 's hat.
4. An image of an emo with dark brown hair in a messy pixie cut, large entirely-black eyes, wearing black clothing.
The image is a mixed media collage with broken glass and torn paper elements, featuring intricate oil details and canvas texture, in contemporary art style.
6. An epic deep space photograph. Gigantic, monolithic letters forming the word 'TempFlow-" hang silently in the void, their surfaces like ancient, cracked obsidian reflect- ing distant starfields. Far below, the letters 'GRPO' are formed by a vast, tranquil nebula glowing with soft, ethereal light, like a cosmic ocean. The sense of scale is immense and humbling. Perfect, sese-sensieive lettering. Moody, atmospheric, photorealistic. cinematic wide-angle shot, 4K UHD.
7. Kiwi fruit, mint leaves, ice cubes, background yellow, splashing water, soft box, back light, creative food photography, Art by Alberto Seveso,
8. Claymation of Futurama characters.
9. A group of four friends commemorating a ski trip in the snow.
10. an empty bench next to a busy street.
11. a 12 year old girl and her pet raccoon
23

---

Published as a conference paper at ICLR 2026
FLUX.1-dev
Flow-GRPO (Prompt)
TempFlow-GRPO
a papaya fruit dressed sailor,
A 3D render of volcanic icon rocky background, in isometric perspective and darkly lit.
"A portrait painting of Batman on Artstation."
Figure 21: Qualitative comparison between FLUX. I-dev, Flow-GRPO (Prompt) and TempFlow- GRPO with HPSv3 rewards on HPDv2 prompts.
FLUX.1-dev
Flow-GRPO (Prompt)
TempFlow-GRPO
Cliffside megastructure by the Grand Canyon
render of pc mouse made out of cheese
Figure 22: Qualitative comparison between FLUX.I-dev, Flow-GRPO (Prompt) and TempFlow- GRPO with multi rewards on PickScore prompts.
24

---

Published as a conference paper at ICLR 2026
FLUX.1-dev
Flow-GRPO (Prompt)
TempFlow-GRPO
a whimsical and playful image of a pizza helicopter with toppings like pepperoni, mushrooms, peppers, and olives. The pizza should be transformed into the shape of a helicopter, complete with rotors, landing gear, and cockpit windows.
octopus on top of horse statue
Lightning Greaves: This artifact equipment card gives the equipped creature haste and shroud, making it a popular choice for protecting and enhancing creatures in combat. Its low cost of mana also makes it versatile card in many decks...
WOAY TO HELL
HELL
A road sign showing the way to hell
miniature fantasy castle with pool of water, sharp focus, photo taken with eos 5d, ultra realism, hyperrealism, professional photography, 8k uhd, ray tracing, ssao, film grain, long shot, wide shot
Figure 23:
Qualitative comparison between FLUX.I-dev, Flow-GRPO (Prompt) and TempFlow- GRPO with multi rewards on PickScore prompts.
25

---

Published as a conference paper at ICLR 2026
FLUX.1-dev
Flow-GRPO (Prompt)
TempFlow-GRPO
Inside computer, high tech landscape, tron, intricately detailed, best quality
arcane style, one single colorful potion in a round bottle with a glowing galactic
landscape inside of it messy brown table, papers and books, sunlight from
a window, soft lighting, atmospheric, bottle is the focus. by makoto shinkai, stanley artgerm lau, wlop, rossdraws, james jean, andrei riabovitchev, marc simonetti, krenz cushart, sakimichan,
Mena
Photo of a woman sitting in a restaurant holding menu that says "Menu"
An image hyper realistic of dog in the sky with dragon wings
chihuahua with knife Burger King
Figure 24: Qualitative comparison between FLUUX.1-de) Flow-GRPO (Prompt) and TempFlow- GRPO with multi rewards PickScore prompts.
26

---

Published as a conference paper at ICLR 2026
FLUX.1-dev
Flow-GRPO (Prompt)
TempFlow-GRPO
panda a velo dans la montagne
single chocolate cereal ball serving as an astronaut's ss head.
A very tiny kitten taking bath in teacup, hd, uhd, uhdr, hdr, 8k, 35mm, ultra high quality
aerial view of giant fish tank shaped like a tower in the middle of new york city, 8k octane render, photorealistic
A panda bear as a mad scientista cup of coffee splashing morph into a rising phoenix rooster with bramble and sparkle sparks spark particles, beautiful studio lighting, appetizing lighting
Qualitative comparison between FLUX.I-dev, Flow-GRPO (Prompt) and TempFlow- GRPO with multi rewards PickScore prompts.
Figure 25:
27


Reviews:
[{"review_id": "Z7kHXzXm0N", "reviewer": "Reviewer_RKyA", "summary": "TempFlow-GRPO is a temporally-aware reinforcement learning framework for flow matching models that improves human preference alignment by introducing trajectory branching, noise-aware weighting, and seed grouping to achieve precise credit assignment and efficient optimization across timesteps.", "questions": "The comparison between batch std and global std is only evaluated on PickScore. How does this observation generalize to other tasks?\n\nCan the proposed reweighting mechanism be applied to hybrid variants (FlowGRPO-Fast/MixGRPO) where only a subset of steps follows an SDE formulation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "For reinforcement learning tasks, dense rewards are crucial for effective credit assignment. The proposed Trajectory Branching mechanism provides an elegant and effective way to obtain dense rewards along the denoising trajectory.\n\nThe introduced reweighting mechanism offers a valuable analysis of how gradients evolve across steps in baseline algorithms and presents a solution to mitigate the identified issues.", "weaknesses": "The proposed method involves numerous ODE denoising steps, which substantially increase computational overhead. However, the paper lacks a comparison against the baseline method using training time as the horizontal axis to illustrate efficiency trade-offs.\n\nThe authors should evaluate the performance of the reweighting mechanism under different $\\sigma_t$ schedulers rather than relying solely on the one used in Flow-GRPO, to examine how the choice of scheduler influences its effectiveness. It remains unclear whether simply reweighting the coefficients in the earlier part to 1 would yield good results under different schedulers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XJXdsO9lSo", "reviewer": "Reviewer_zj9Z", "summary": "This paper addresses the sparse terminal reward and uniform credit assignment problem in GRPO training of flow models. The authors propose TempFlowGRPO, which includes: (1) Trajectory Branching, where only one step of SDE is used at timestep k; (2) Noise-Aware Policy Weighting by reweighting according to noise level; and (3) a seed group strategy. The method achieves state-of-the-art performance in human preference alignment and text-to-image benchmarks.", "questions": "1.\tThe trajectory branching mechanism appears similar to MixGRPO limited with a single-timestep window. How do their efficiency and effectiveness compare?\n2.\tThe paper claims that Flow-GRPO (Prompt) is an improved baseline with group standard deviation stabilization, but does not provide much detail. Could the authors elaborate on this improved method?\n3.\tWhy are the Pickscore curve trends by steps and GPU hours on the left of Figure 3 inconsistent?\n4.\tCompare to FlowGRPO, the experiment of Visual Text Rendering is not addressed. How well does TempFlow-GRPO perform on this particular task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors astutely identify that the FLOW-GRPO algorithm treats all timesteps equally, and tackle this issue via single-timestep SDE optimization.\n2.\tThe noise reweighting method is shown to be effective through both soild theoretical analysis and experiment results.\n3.\tThe paper is generally well written with a clear logical structure.", "weaknesses": "1.\tThe contribution of seed group strategy is relatively small to other parts of the work, and the paper should provide additional details of the seed group strategy.\n2.\tSimilarly, MixGRPO [1] proposes a training window of SDE time steps that also tackles the issue of treating all timesteps equally. However, there is limited discussion comparing with MixGRPO.\n3.\tThe paper does not discuss the phenomenon of reward hacking, which is an inevitable problem for the GRPO method.\n\n[1] Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ll3uG2W6Wk", "reviewer": "Reviewer_wmxk", "summary": "This paper presents TempFlow-GRPO, a new reinforcement learning framework that addresses the limitation of uniform credit assignment across timesteps. The method introduces trajectory branching, which switches from ODE to SDE sampling at selected timesteps to generate exploratory branches and assign their rewards to intermediate states. This paper further proposes noise-aware policy weighting, prioritizing optimization at high-noise early stages over low-noise refinement phases. Experiments show that TempFlow-GRPO achieves substantially improved efficiency and final performance compared to the baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well-written and easy to follow.\n- The motivation and the proposed method are clear and straightforward: addresses the temporal inhomogeneity and credit assignment problems through intermediate resampling for intermediate value estimation and noise-aware reweighting.\n- The proposed method shows strong empirical performance in both efficiency and end-level performance, with comparisons that include GPU time.", "weaknesses": "- Theorem 1 is intuitively reasonable, but labeling it as a Theorem feels overstated since the underlying assumptions and proof sketch are insufficiently formalized. The analytical depth is also somewhat limited.\n- The explanation around line 847 (regarding why the average number of branches is 4.5× when K = 10) is unclear. It is not obvious how this factor arises or how the branching schedule operates, and the paper does not explicitly describe it.\n- Adding more algorithmic details or pseudocode would improve readability and make the proposed procedure easier to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bwnN4Dpw2N", "reviewer": "Reviewer_rGTd", "summary": "The paper proposed TempFlow-GRPO, a framework that makes the optimization process temporally aware to address the key limitation of temporal uniformity in previous RLHF works. The paper introduces a mixture of ODE and SDE sampling, along with a noise-aware policy weighting scheme, to balance exploration and reward exploitation. Experiments demonstrate that TempFlow-GRPO achieves state-of-the-art performance, yielding higher rewards than standard GRPO approaches.", "questions": "- How is the performance affected by the number of branches (K) at each step, the specific timesteps chosen for branching, or the exact function used for noise-aware weighting? The ablation study (Fig. 8) shows that the 4x6 (seed x branch) configuration was chosen, but it's unclear how much tuning is required to find the optimal setup for a new model or dataset. A discussion on how to choose these hyperparameters will be useful for general applications of the proposed framework.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper pinpoints temporal uniformity as the primary limitation of existing flow-based GRPO methods and proposes TempFlow-GRPO to solve it with precise credit assignment and noise-aware optimization. The authors demonstrate this non-uniformity well with empirical evidence from rewards, supporting the need for temporal information.\n- The paper introduces the core mechanisms of trajectory branching and noise-aware reweighting to create temporally-structured policies that respect the dynamics of the generative process. The authors also provide a theoretical justification from the policy gradient perspective, further supporting the use of noise-aware reweighting.\n- The proposed TempFlow-GRPO achieves state-of-the-art performance compared to the existing vanilla GRPO approach, demonstrating the effectiveness of the method. The authors also include comprehensive ablation studies to better understand the dynamics of this model.", "weaknesses": "- The computational cost, as thoroughly analyzed in Appendix A.6, will be higher than the vanilla GRPO models due to the branching process. Nonetheless, this is more like a trade-off between quality and time, given the superior quality metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)