TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
Authors: name: Xiaoxuan He; name: Siming Fu; name: Yuke Zhao; name: Wanli Li; name: Jian Yang; name: Dacheng Yin; name: Fengyun Rao; name: Bo Zhang
Published: 2025-08-06 11:10:39

Abstract: Recent flow matching models for text-to-image generation have achieved
remarkable quality, yet their integration with reinforcement learning for human
preference alignment remains suboptimal, hindering fine-grained reward-based
optimization. We observe that the key impediment to effective GRPO training of
flow models is the temporal uniformity assumption in existing approaches:
sparse terminal rewards with uniform credit assignment fail to capture the
varying criticality of decisions across generation timesteps, resulting in
inefficient exploration and suboptimal convergence. To remedy this shortcoming,
we introduce TempFlow-GRPO (Temporal Flow GRPO), a principled GRPO
framework that captures and exploits the temporal structure inherent in
flow-based generation. TempFlow-GRPO introduces two key innovations: (i) a
trajectory branching mechanism that provides process rewards by concentrating
stochasticity at designated branching points, enabling precise credit
assignment without requiring specialized intermediate reward models; and (ii) a
noise-aware weighting scheme that modulates policy optimization according to
the intrinsic exploration potential of each timestep, prioritizing learning
during high-impact early stages while ensuring stable refinement in later
phases. These innovations endow the model with temporally-aware optimization
that respects the underlying generative dynamics, leading to state-of-the-art
performance in human preference alignment and standard text-to-image
benchmarks.