Flow-GRPO: Training Flow Matching Models via Online RL
Authors: Jie Liu
Gongye Liu
Jiajun Liang
Yangguang Li
Jiaheng Liu
Xintao Wang
Pengfei Wan
Di ZHANG
Wanli Ouyang
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
flow matching
text to image generation
Primary area: deep_learning
TL;DR: We propose Flow-GRPO, the first method to integrate online RL into flow matching models, significantly enhancing text-to-image generation performance.

Abstract: We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into an equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks. For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from $63\%$ to $95\%$. In visual text rendering, accuracy improves from $59\%$ to $92\%$, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.

Flow-GRPO: Training Flow Matching Models via Online RL
Jie Liu
Gongye Liu
Jiajun Liang*
Yangguang Li!
Jiaheng Liu
Xintao Wang* Pengfei Wan
Di Zhang* Wanli Ouyang!
MMLla, CUHK
Tsinghua University
*Kling Team, Kuaishou Technology
"Nanjing University
Shanghai Al Laboratory
jieliuflink. cuhk. edu.hk wlouyang@ie. cuhk. edu.hk
Code: nttps://github com httpa//.omoomlfaa/////////gggg
Abstract
We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks, For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from 63% 95%. In visual text rendering, accuracy improves from 59% to 92%, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.
Introduction
Flow matching [2, 3] models have become dominant in image generation [4. 5] due to their solid theoretical foundations and strong performance in producing high quality images. However, they often struggle with composing complex scenes involving multiple objects, attributes, and relationships [6, 7], as well as text rendering [8]. At the same time, online reinforcement learning (RL) [9] has proven highly effective in enhancing the reasoning capabilities of large language models (LLMs) [10, 11]. While previous research has mainly focused on applying RL to early diffusion- models [12] and offline RL techniques like direct preference optimization [13] for flow-based generative models [14, 15), the potential of online RL in advancing flow matching generative models remains largely unexplored. In this study, explore how online RI. can be leveraged to effectively improve flow matching models.
generative
Training flow models with RL presents several critical challenges: (1) Flow models rely on deterministic generative process based on ODES [3], meaning they cannot sample stochastically during inference. In contrast, RL relies on stochastic sampling to explore the environment, learning by tryng different actions and improving based on rewards. This need for stochasticity in RL conflicts with the deterministic nature flow matching models. (2) Online RL depends on efficient sampling
Equal contribution, 'Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) GenEval Performance
(b) Image Quality
1.00
303.5-Medium FloW-GRPO
0.95
(0.95)
0.90
GenEval Score
0.85
GPT-4o (0.84)
Austhatic
Deng
0.80
(c) Preference Score
0.75
FLUX.1 Dev (0.66)
1.5
23
3:33
2:54
0.65
SD3.5-Medium (0.63)
1.03
0.60
0.55
500
1000
1500
2000
2500
Inegefleward
Pickocore
Uniivetleward
Training Time (GPU Hours)
Figure 1: (a) GenEval performance rises steadily throughout Flow-GRPO's training and outperforms GPT-4o. (b) Image quality metrics on Draw Bench [1] remain essentially unchanged. (c) Human Preference Scores on DrawBench improves after training. Results show that Flow-GRIP enhances the desired capability while preserving image quality and exhibiting minimal reward- hhckidgg
to collect training data, but flow models typically require many iterative steps to generate each sample, limiting efficiency. This issue is more pronounced with large models [5, 4]. To make RL practical for tasks like image video generation, improving sampling efficiency is essential.
To address these challenges, we propose Flow-GRPO, which integrates GRPO [16] into flow matching models for text-to-image (T2I) generation, using two key strategies. First, we adopt the ODE-to-SDE strategy to overcome the deterministic nature of the original flow model. By converting the ODE-based flow into an equivalent Stochastic Differential Equation (SDE) framework, we introduce randomness while preserving the original marginal distributions. Second, to improve sampling efficiency in online RL, we apply the Denoising Reduction strategy, which reduces denoising steps during training while keeping the full schedule during inference. Our experiments show that using fewer steps maintains performance while significantly reducing data generation costs.
We evaluate Flow-GRPO on T2I tasks with various reward types. (1) Verifiable rewards, using the GenEval [17] benchmark and visual text rendering task. GenEval includes compositional image generation tasks (e.g., generating specific object counts, colors, and spatial relationships). which can be automatically assessed with object detection methods. Flow-GRPO improves the accuracy of Stable Diffusion 3.5 Medium (SD3.5-M) [4] from 63% 95% on GenEval, outperforming the state-of-the-art GPT-4o [18] model. For visual text rendering, SD3.5 3D3.5-M's accuracy increases from 59% to 92%, greatly enhancing its text generation ability. (2) Model-based rewards, such as the human preference Pickscore [19] reward. These results show that our framework is task independent, demonstrating generalizability and robustness. Importantly, all improvements are achieved with very little reward hacking, demonstrated in Figure 1.
To summarize, the contributions of Flow-GRPO are as follows:
We are the first to introduce GRPO to flow matching models by converting deterministic ODE sampling into SDE sampling, showing the effectiveness online RL for T2I tasks. Flow-GRPO improves SD3.5-M accuracy from 63% to 95% without noticeably compromising image quality.
We find that online RL for flow matching models does require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process.
We show that Kullback-Leible (KL) constraint effectively prevents reward hacking, where reward increases at the cost of image quality or diversity. KI. regularization is not empirically equivalent to early stopping. With proper KI. term, we can match the high reward of the KI.-free version while preserving image quality, albeit with longer training.

---

2 Related Work
RL for LLM. Online RL has effectively improved the reasoning abilities of LLMs, such as DeepSeek-R1 [10] and OpenAI-ol [11], using policy gradient methods like PPO [20] or value-free GRPO [16]. GRPO is more memory efficient by removing the need for value network, so we adopt it in this work. PPO can also be applied to flow matching in similar way.
Diffusion and Flow Matching, Diffusion models [21, 22, 23] add Gaussian noise to data and train neural network reverse the process. Sampling uses discrete DDPM steps probability flow SDE solvers to generate high-fidelity outputs. Flow matching [2. 3] learns a continuous-timg normalizing flow by directly matching the velocity field, allowing efficient deterministic sampling with only few ODE steps. It achieves competitive FID with far fewer denoising steps than diffusion, making it the dominant choice recent image [4, 5] and video [24, 25, 26, 27] generation models. Recent work [28, 29] unifies diffusion and flow models under an SDE/ODE framework. Our work builds on their theoretical foundations and introduces GRPO to based models.
Alignment for T21. Recent efforts to align pretrained T2I models with human preferences follow five main directions: (1) direct fine-tuning with differentiable rewards [30, 31, 32, 33]; (2) Reward Weighted Regression (RWR) [34, 35, 36, 37]; (3) Direct Preference Optimization (DPO) and vari- ants [38, 39, 14, 40, 41, 42, 43, 44, 45, 46]: (4) PPO-style policy gradients [47, 48, 49, 50, 51, 52]; (5) training-free free alignment methods [53, 54, 55]. These methods have successfully aligned T2I models with human preferences, improving aesthetics and semantic consistency. Building on this progress, introduce GRPO for flow matching models, the backbone of today's state-of-the-art T21 systems. Concurrent work [56] applies GRPO to text-to-speech flow models, but instead of convert- ing the ODE to SDE inject stochasticity, they reformulate velocity prediction by estimating Gaussian distribution (predicting both the mean and variance of velocity), which requires retraining the pre-trained model. Another study [57] also explores SDE-based stochasticity but focuses on inference-time scaling.
3 Preliminaries
In this section, we introduce the mathematical formulation of flow matching and describe how the denoising process can be mapped multi-step MDP.
Flow Matching. Let wo ~ Xn be a data sample from the true distribution, and æ ~ X2 denote a noise sample. Recent advanced mage-ggeneration models (e.g., [4, 5)) and video-generation models (e.g., [24, 26, 25, 27]) adopt the Rectified Flow [3] framework, which defines the "noised" data æ as
C; (1 1) 2o0 + L L0
(1)
fort E [0, 1]. Then transformer model are trained to directly regress the velocity field vy (2,,, f) by minimizing the Flow Matching objective [2,
[())= = E,, m No. x [|v vo(2,, 1)1?
where the target velocity field v x1 Zo.
Denoising an MDP. As shown in [12], the iterative denoising process in flow matching models can be formulated as a Markov Decision Process (MDP) (S,A, Po. P. R). The state at step is 8, A (c, t, R, the action is the denoised sample a, 2;-1 predicted by the model, and the policy is T(a 8x) pa (#;-1 c), The transition is deterministic: P(8 |8,, a;) A (8e, 8,-1, 6,._),, and the initial state distribution is Po( (p(c), 8r, (0, I)), where 8y is the Dirac delta distribution centered at y. The reward only given the final step: R(st, a.) & r(wo, c) if = O, and 0 otherwise.
4 FIoW-GRPO
In this section, we present Flow-GRPO, which enhances flow models using online RI. We begin by revisiting the core idea of GRPO [16] and adapting to flow matching. We then show how to convert
3

---

Prompt: photo of four cups.
Policy Optimization
Group Relative Policy Optimization
SDE Sampling
Matching Model
Advantages
Markov Decision Process via SDE Sampling
Sq
Group Computation
(0,)
Rewards
dx, fo dt o,dw
g+
A+
g+
R
RP
%2
Reward
xy (acls)
Function
ST
Wow, few-step, low-quality sumples suffice for RL
Denoising Reduction (T 10)
Figure 2: Overview of Flow-GRPO. Given a prompt set, we introduce an ODE-to-SDE strategy enable stochastic sampling for online RL. With Denoising Reduction (only T 10 steps), efficiently gather low-quality but still informative trajectories. Rewards from these trajectories feed the GRPO loss, which updates the model online and yields an aligned policy.
the deterministic ODE sampler into SDE sampler with the same marginal distribution, introducing the stochasticity needed for applying GRPO. Finally, we introduce Denoise Reduction, practical sampling strategy that significantly speeds up training without sacrificing performance.
GRPO on Flow Matching, RL aims to learn policy that maximizes the expected cumulative reward. This is often formulated as optimizing a policy Ty with regularized objective:
R(sr, a.) - (DKL(TT( ( st)) ]
max Ei Gg..... ay 0
(3)
Unlike other policy based methods like PPO [20], GRPO [16] provides a lightweight alternative, which introduces a group relative formulation estimate the advantage.
Recall that the denoising process can be formulated as an MDP, as shown Section 3. Given a prompt the flow model pe samples group of G individual images 1 and the corresponding reverse- time trajectories {(r}r 1/1-1 an) ) - Then, the advantage of the i-th image is calculated by normalizing the group-level rewards as follows:
A R(* c) mean ([(ln c)
(4)
(d()) c)
GRPO optimizes the policy model by maximizing the following objective:
Jrwow-CRPO(P) = Ee-C ( Mo (o..
(5)
where
6
T-1
f(r, f(r,A,0,
Σ
min (1(())A,, clip (r((0),1-E, =, AA)) - 33Dxx 3Dex (Tol(
T
1=1
1=0
Po(0)-1
c)
ri(0)
x
From ODE to SDE GRPO relies on stochastic sampling Eq. and Eq. 5 to generate diverse trajectories for advantage estimation and exploration. Diffusion models naturally support this: the

---

forward process adds Gaussian noise step by step, and the reverse process approximates a score-based SDE solver via Markov chain with decreasing variance. In contrast, flow matching models use deterministic ODE for the forward process:
dæ = v,di,
(6)
where V2 is learned via the flow matching objective in Eq. 2. A common sampling method is to discretize this ODE, yielding one-to-one mapping between successive time steps.
This deterministic approach fails to meet the GRPO policy update requirements two key ways: (I) (0) in Eq. 5 requires computing p(æ(-1 Dt. c). which becomes computationally expensive under deterministic dynamics due to divergence estimation. (2) More importantly, RL depends on exploration. As shown in Section 5.3, reduced randomness greatly lowers training efficiency. Deterministic sampling, with no randomness beyond the initial seed, is especially problematic.
To address this limitation, we convert the deterministic Flow-ODE from Eq. 6 into an equivalent SDE that matches the original model's marginal probability density function all timesteps. We outline the key process here. A detailed proof is provided in Appendix A. Following [23, 28, 29], we construct a reverse-time SDE formulation that preserves the marginal distribution:
of v log (((s)) dt + o,dw,
dæ;- = U1 (x:))
2
where dw denotes Wiener process increments and oR control the level of stachasticity during genera- tion. For rectified flow, Eq. 7 is specified
of (xx +1 (1-))()))((zz))) dt + owdw.
dæ; (x))
(8)
2t
Applying Fuler-Maruyama discretization yields the final update rule:
of (a, 101-17 (200 Al Al+o.VALe
A++ 27, (a
2t
where e ~ (0, I) injects stochasticity. We use = I-t in this paper, where is a scalar hyper-parameter that controls the noise level (See Section 5.3 for its impact on performance).
Eq. reveals that the policy (1 æt, c) is an isotropic Gaussian distribution. We can easily compute the KL divergence between Ty and the reference policy Teet in Eq. 5 as a closed form:
Te Ar.0
ateell
At
[o (or(1-t) -)
1 |[vo(2), () Uerr(Ze, +)1
DKL (Tee
(Tee)) (ree)
20
2
21
Denoising Reduction. To produce high-quality images, flow models typically require many de- noising steps, making data collection costly for online RL. However, we find that large timesteps are unnecessary during online RL training. We can use significantly fewer denoising steps during sample generation, while retaining the original denoising steps during inference to get high-quality samples. Note that we set the timestep I as 10 in training, while the inference timestep I' is set as the original default setting fast training without sacrificing image quality at test time.
40) for SD3.5-M. Our experiments reveals that this approach enables
Experiments
This section empirically evaluates Flow-GRPO's ability to improve flow matching models three tasks. (1) Composition Image Generation: This task requires precise object arrangement and attribute control. We report the results on GenEval. (2) Visual Text Rendering: a rule-based task that evaluates the accurate rendering of the text specified in the prompt. (3) Human Preference Alignment: This task aims align T2I models with human preferences.
5.1 Experimental Setup
We introduce three tasks, detailing their respective prompts and reward definitions. For hyperparame- ter details and compute resource specifications, please refer to Appendix B.3 and Appendix B.4.
5

---

Compositional Image Generation. GenEval [17] assesses T2I models on complex compositional prompts- like object counting, spatial relations, and attribute binding- across six difficult composi- tional image generation tasks. We use its official evaluation pipeline, which detects object bounding boxes and colors, then infers their spatial relations. Training prompts are generated using official GenEval scripts, which apply templates and random combinations to construct the prompt dataset. The test set is strictly deduplicated: prompts differing only in object order (e.g., "a photo of A and B" vs. "a photo of B and A") are treated as identical, and these variants are removed from the training set. Based on the base model's 's initial accuracy across the six tasks, we set the prompt ratio as Position: : Counting Attribute Binding. : Colors Two Objects Single Object 7:5: 3 : :111 1 0. Rewards are based: (1) Counting: r = 1 |NVeen / (2) Position / Color: If the object count is correct, partial reward is assigned; the remainder is granted when the predicted position or color is also correct.
Visual Text Rendering [8]. Text is common in images such as posters, book covers, and memes, so the ability to place accurate and coherent text inside the generated images is crucial for T2I models. In our settings, we define an text rendering task, where each prompt follows the template "A sign that says "text", Specifically, the placcholder ' "text" is the exact string that should appear in the image. We use GPT4o to produce 20K training prompts and IK test prompts. Following [58], measure text fidelity with the reward distance between the rendered text and the target text and Niar is the number of characters inside the quotation marks in the prompt. This reward also serves our metric of text accuracy.
max N Ne/N 0), where N, is the minimum edit
Human Preference Alignment [19]. This task aims to align T21 models with human preferences. We use PickScore [19] as our reward model, which is based on large-scale human annotated pairwise comparisons of images generated from the same prompt. For each image and prompt pair, PickScore provides an overall score that evaluates multiple criteria, such as the alignment of the image with the prompt and its visual quality.
Image Quality Evaluation Metric. Since the T21 model is trained to maximize a predefined reward, it vulnerable to reward hacking, where the reward increases but image quality or diversity declines. This study aims to make online RL effective for T2I generation without noticeably compromising quality or diversity. To detect reward hacking beyond task-specific accuracy, we evaluate four automatic image quality metrics: Aesthetic Score [59]. DeQA [60], ImageReward [32], and Unird [61] (see Appendix B.1 for details), All metries are computed on DrawBench [1], comprehensive benchmark with diverse prompts for T2I models.
5.2
Main Results
Figure and Table show Flow-GRPO's GenEval performance steadily improving during training, ultimately outperforming GPT-4o. This occurs while maintaining both image quality metrics and preference scores on DrawBench, a benchmark with diverse and comprehensive prompts for evalu- ating general model capabilitics. Figure offers qualitative comparisons. Beyond Compositional Image Generation, Table 2 details evaluations Visual Text Rendering and Human Preference tasks. Flow-GRPO improved text rendering ability, again without decreasing image quality metrics and preference scores on Drawhench. See Figures 13, 14 & 15 in Appendix C.6 for related qualitative examples. For the Human Preference task, image quality did not decrease without KL regulariza- tion. However, we found that omitting KL caused collapse in visual diversity, form of reward hacking discussed further in Section 5.3. These results demonstrate that Flow-GRPO boosts desired capabilities while causing very little degradation to image quality visual diversity.
Flow-GRPO vs. Other Alignment Methods. We compare Flow-GRPO with several alignment methods: supervised fine-tuning (SFT), Flow-DPO [14, 39], and their online variants. Flow-GRPO consistently outperforms all baselines by significant margin. At each step, we generate group of images using the same group size as in Flow-GRPO. The only difference lies in the update rule:
SFT: Select the highest-reward image each group and fine-tune on
Flow-DPO: Use the highest-reward image in each group as the chosen sample and the lowest as the rejected, then apply the DPO loss.

---

Table 1: GenEval Result. Best scores are in blue second-best in green Results for models other than SD3.5-M are from [7] their original papers. Obj.: Object; Attr: Attribution.
Model
Overall
Single Obj.
Two Obj.
Counting
Colors
Position
Attr. Binding
Diffusion Models
LDM [62]
0.37
0.92
0.29
0.23
0.70
0.02
0.05
SD1.5 [62]
0.43
0.97
0.38
0.35
0.76
0.04
0.06
SD2.1 [62]
0,50
0,98
0,51
0.44
0.85
0.07
0.17
SD-XL [63]
0,55
0,98
0,74
0.39
0.85
0,15
0.23
DALLE-2 [64]
0.52
0.94
0.66
0.49
0.77
0,10
0.19
DALLE-3 [65]
0.67
0.96
0.87
0.47
0.83
0.43
0.45
Autoregressive Models
Show-o [66]
0.53
0.95
0.52
0.49
0.11
Emu3-Gen [67]
0.54
0.98
0.71
0.34
0.81
0.17
0.21
JanusFlow [68]
0.63
0.97
0.59
0.45
0.83
0.53
0.42
Janus-Pro- Pro-7B [69]
0.80
0.99
0.89
0,59
0.90
0.79
0.66
GPT-4o [18]
0.84
0.99
0.92
0.85
0.92
0.75
0.61
Flow Matching Models
FLUX.1 1 Dev [5]
0.66
0,98
0.81
0.74
0.79
0,22
0.45
SD3.5-L [4]
0.71
0.98
0.89
0.73
0.83
0,34
0.47
SANA-I.5 4.8B [70]
0.81
0.99
0.93
0.86
0.84
0.59
0.65
SD3.5-M [4]
0.63
0.98
0.78
0.50
0.81
0.24
0.52
SD3.5-M+Flow-GRB
0.95
1.00
0.99
0.95
0.92
0.99
0.86
photo of four giraffes
A photo of white sandwich
photo of red dog
brown giraffe white stop sign
red orange purple broccoli
photo of a bench left of
FLUX.1
STOP
GPT-4o
STOP
STOP
SD-3.5-M
SD-3.5-M Flow-GRPO
STOP
Figure 3: Qualitative Comparison on the GenEval Benchmark. Our approach demonstrates superior performance in Counting, Colors, Attribute Binding, and Position.
Offline variants use fixed pretrained model for data collection, while online variants update their data collection models every 40 steps. As shown in Figure 4, Goo- outperforms all base- lines. Online DPO also surpasses offline counterpart, consistent with [15]. For the second-best online DPO, hyperparameter search on its key parameter 3 revealed that smaller values are not always optimal; excessively small 3 values can cause training collapse. Appendix presents more comprchensive comparisons covering additional methods and tasks.

---

Table 2: Performance Compositional Image Generation, Visual Text Rendering, and Human Preference benchmarks, evaluated by task performance on test prompts, and by image quality and preference scores DrawBench prompts. ImgRwd: ImageReward; UniRwd: UnifiedReward.
Task Metrie
Image Quality
Preference Score
Model
GenEval
OCR Acc.
PickScore
Aesthetic
DeQA
ImgRwd
PickScore
UniRwd
SD3.5-M
0.63
0.59
21.72
5.39
0.87
22.34
Compositional Image Generation
Flow-GRPO (wlo KL)
0.95
4.93
2.77
0.44
21.16
2,94
Flow-GRPO (w/ KL)
0.95
5.25
4.01
1.03
22.37
3.51
Visual Text Rendering
Flow-GRPO (w/ KL)
0.93
5.13
3.66
0.58
21.79
3.15
Flow-GRPO (w/ KL)
0.92
5.32
4.06
0.95
22.44
3.42
Human Preference Alignment
Flow-GRPO (w/o KL) Flow-GRPO (w/ KL)
23.41
6.15
4.16
1.24
23,56
3.57
23.31
5.92
4.22
1.28
23.53
3.66
PrekScore Evel
0.8
GenEvval
Plon
190
Flow GRPO (G = 24]
nel SFT
offina DPO 15
21
Flow GRPO [G 12)
Officc SPT
Flow GRPO (G 6)
0.5
3000
10000
15000
20030
25000
30000
Prompt
Training Step
Figure 4:
Comparison with Other Alignment Methods on the Compositional Generation Task.
Figure 5: Ablation Studies on Different Group
Size G. Higher group size performs better.
5.3 Analysis
This section presents several analyses better understand the behavior and robustness of We examine issues such as reward hacking, the impact of denoising reduction and noise levels, the effect of group size, and the model's generalization ability. We provide additional analyses in Appendix C.
Reward Hacking. We use KL regularization to mitigate reward hacking by tuning the KL coef- ficient keep the divergence small and nearly constant during training, keeping the model close to its pretrained weights. This allows task-specific reward optimization without harming overall performance. As shown in Table 2, removing the KL constraint for Compositional Image Generation and Visual Text Rendering significantly reduces image quality and preference scores on DrawBench. In contrast, a properly tuned KL preserves quality while achieving similar gains on task-specific metrics. In the Human Preference Alignment task, removing KI does not affect image quality, likely due to overlap between PickScore and evaluation metrics, but causes collapse in visual diversity. Outputs converge to single style, with different seeds producing nearly identical results. KL regularization prevents this collapse and maintains diversity. See Figure 12 in Appendix C.5 for training curves and Figure 6 for more examples.
Effect of Denoising Reduction. Figure (a) highlights Denoising Reduction's 's significant impact on accelerating training. explore how different timesteps affect optimization, these experiments are conducted without the KI. constraint. Reducing data collection timesteps from 40 to 10 achieves over 4x speedup across all three tasks, without impacting final reward. Further reducing to 5 does not consistently improve speed and sometimes slows training, we choose 10 timesteps for later

---

Quality Degradation
Diversity Decline Ours (W/
Ours KL)
Ours (W// KL)
Ours (W/o
SD-3.5-M
photo apples
FUEL LOW
BBL
spaceship cockpit digite! displaying warming, set against backdrop star stucded çalexy, with control panels futuristic interfaces around.
Lincoln outdoor large
Figure 6: Effect of KL Regularization. The KL penalty effectively suppresses reward hacking, preventing Quality Degradation (for GenEval and OCR) and Diversity Decline (for PickScore).
experiments. For the other two tasks, learning curves reward versus training time are presented in Figure in the Appendix C.2.
Effect of Noise Level. Higher oN in the SDE boosts image diversity and exploration, vital for RL training. We control this exploration with a noise level a (Eq. 9). Figure 7 (b) shows the impact of on performance. A small (e.g., 0.1) limits exploration and slows reward improvement. Increasing (up 0.7) boosts exploration and speeds up reward gains. Beyond this point (e.g.. from 0.7 to 1.0), further increases provide no additional benefit, as exploration is already sufficient. We also observe that injecting too much noise by further increasing a degrades image quality, resulting zero reward and failed training.
Accuracy
Score Score
Genbval
0.7
OCR, a 0.1
GenEval, Step
OCR, 0.4
GenEvai, Step 10
0.6
GenEval, Step 40
OCR, 1.0
0.5
1500
1000
1260
Training Time (GPU Hours)
Training Steps
(a). Effect Denoising Reduction on GenEval
(b). Effeect of Noise Level Ablation on the OCR
Figure 7: Ablation studies critical design choices. (a) Denoising Reduction: Fewer denoising
steps accelerate convergence and yield similar performance. (b) Noise Level: Moderate noise level (a 0.7) maximises OCR accuracy, while too little noise hampers exploration.
Effect of Group Size. Figure shows the effect of group size using PickScore as the reward function. When the group size reduced G 12 and G = 6, training became unstable and eventually collapsed, whereas G = 24 remained stable throughout the process. We observe that smaller group sizes produce inaccurate advantage estimates, increasing variance and leading to training collapse, a phenomenon also reported in [71, 72].
Generalization Analysis. Flow-GRPO demonstrates strong generalization on unseen scenarios from GenEval (Table 4). Specifically, it captures object number, color, and spatial relations, generaliz- ing well to unseen object classes. It also effectively controls object count, generalizing from training on 2 4 objects to generate 5 - 6 or 12 objects. Furthermore, Table 3 shows Flow-GRPO achieves

---

significant gains on 121-CompBench++ [6, 73]. This comprehensive benchmark for open- open-world compositional T2I generation features object classes and relationships substantially different from our model's GenEval-style training data.
Table 3: 121-CompBench++ Result. This evaluation uses the same model presented in Table 1,
which was trained on the -eeneaa-geeeratee dataset. The best score is in blue
Model
Color
Shape
Texture
2D-Spatial
3D-Spatial
Numeracy
Non-Spatial
Monus-rr-7BB [69]
0.5145
0.3323
0.4069
0.1566
0.2753
0.4406
0.3137
EMU3 [67]
0.7913
0.5846
0.7422
FLUX.1 1 Dev [S]
0.7407
0.5718
0.6922
0,2863
0.3866
0.6185
0,3127
SD3.5-M [4]
0.7994
0,5669
0.7338
0,2850
0.3739
0.5927
0.3146
SD3.5-M-Plow-GRPO
0.8379
0.6130
0.7236
0.5447
0.4471
0.6752
0.3195
Table 4: Flow-GRPO demonstrates strong generalization. Unseen Objects: Trained on 60 object classes, evaluated on 20 unseen classes. Unseen Counting: Trained to render 3, or 4 objects, and evaluated in two settings: rendering 5 or 6 objects, and rendering 12 objects.
Method
Unseen Objects
Unseen Counting
Overall
Single Obj.
Two Obj.
Counting
Colors
Position
Binding
5-6 Objects
12 Objects
SD3.5-M
0.64
0.96
0.73
0.53
0.87
0.26
0.47
0.13
0.02
SD3.5MM-FlowGRPO
0.90
1.00
0.94
0.86
0.97
0.84
0.77
0.48
0.12
Conclusion
We have presented Flow-GRPO, the first method integrate online policy gradient RL into flow matching models. By converting deterministic ODEs to SDEs and reducing denoising steps during training,
enables efficient RL-based optimization without noticeably compromising im- age quality or diversity. Our method significantly improves performance on compositional generation, text rendering, and human preference alignment, with minimal reward hacking. Flow-GRPO offers simple and general framework for applying online RL to flow-based generative models.
Limitations & Future Work. Although this work focuses T2I tasks, Flow-GRPO has potential for video generation [25, 27], raising several future directions: (1) Reward Design: Simple heuristics, such using object detectors trackers rule-based rewards, can encourage physical realism and temporal consistency, but more advanced reward models are needed. (2) Balancing Multiple Rewards: Video generation requires optimizing multiple objectives, including realism, smoothness, and coherence. Balancing these competing goals remains challenging and demands careful tuning. (3) Scalability: Video generation is far more resource- intenci-nteesive than T2I. applying Flow-GRPO at scale requires more efficient data collection and training pipelines. Additionally, better methods for preventing reward hacking are worth exploring. While KL regularization helps significantly, requires longer training and occasional reward hacking occurs for certain prompts.
Acknowledgements
This work was partially supported by the JC STEM Lab of AI for Science and Engineering, funded by The Hong Kong Jockey Club Charities Trust, the Research Grants Council of Hong Kong (Project No. CUHK 14213224). We gratefully acknowledge Mingwu Zheng for his insightful discussions the proof and Zhanhui Zhou for his valuable comments that improved the clarity of this paper.
References
[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-36494, 2022.
[2] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint 27.0042210.02747, 2022.
10

---

[3] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 020000200,,3000,, 2022.
[4] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[5] Black Forest Labs. Flux, https:///gtthiie http:://gttubb..oommaaaaa-freree latssllabb/fffuuu 2024.
[6] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78747, 2023.
[7] Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang. Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation. arXiv preprint arXiv:2504.02782 2025.
[8] Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters, Advances in Neural Information Processing Systems, 36:9353- 9387, 2023.
[9] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[10] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arviv preprint arXiv:2501.12948, 2025.
[11] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card. arXiv preprint arXiv:2412.
2024.
[12] Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 201223055333000 13301, 2023.
[13] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53744 2023.
[14] Jie Liu, Gongye Liu, Jiajun Liang. Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint arXiv:2501. 13918, 2025.
[15] Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Juncheng Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengchen Ma, al. Skyreels-v2: Infinite-length film generative model. arXiv preprint arXiv:2504. 13074, 2025.
[16) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.0333 2024.
[17] Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132-52152. 2023.
[18] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card, arXiv preprint arXiv:2410.21276, 2024.
[19] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for ext-tto-mmage generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.

---

[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.06347. 2017.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[22] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[23] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13346 2020.
[24] Kuaishou. Kling ai.
www.nkkkkiihhoonnnoo. 2024.
[25] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint wrXhv:2503.20344 14, 2025.
[26] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, al. Video generation models as world simulators. OpenAl Blog, 1:8, 2024.
[27] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min. Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024.
[28] Michael S Albergo, Nicholas M Boffi, and Eric Vanden- Vanden-Fijnden. Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
[29] Carles Domingo-Enrinn Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffussion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.0886/ 2024.
[30] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739 2023.
[31] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 17400, 2023.
[32] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[33] Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint urXiv:2407.08737, 2024.
[34] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1970.00177 2019.
[35] Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu. Online weighte fine-tuning of flow matching with wasserstein regularization. In The Thirteenth International Conference on Learning Representations, 2025.
[36] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
[37] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06760 2023.
12

---

[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[39] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1228-8238, 2024.
[40] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941-8951, 2024.
[41] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance each step. arXiv preprint arXiv:2406.04331 0431,, 2024.
[42] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-inne texl-to-image generation. arXiv preprint arXiv:2402. 2024.
[43] Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. Vidcodpo: Omni- aani-preference alignment for video diffusion generation. arXiv preprint arXiv:2412. 14167, 2024.
[44] Jiacheng Zhang, Jie Wu, Weifeng Chen, Yatai Ji, Xuefeng Xiao, Weilin Huang, and Kai Han. Onlinevpo: Align video diffusion model with online video-centric preference optimization. arXiv preprint arXiv:24/2. 15159, 2024.
[45] Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, and Sherry Yang. Improving dynamic object interactions in text-to-video generation with ai feedback. arXiv preprint arXiv:2412.026117 2024.
[46] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generie preferences with step-by-step preference optimization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13199-13208, 2025.
[47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066 06347, 2017.
[48] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 2023.
[49] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee, Reinforcement learning for fine-tuning text-to-imet -xt-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[50] Shashank Gupta, Chaitanya Ahuja, Tsung- Yu Lin, Sreya Dutta Roy. Harrie Oosterhuis, Maarten de Rijke, and Satya Narayan Shukla. simple and effective reinforcement learning method for text-to-ing text-to-image image diffusion tie-ttning. arXiv preprint :.t.:::250300099)) 2025.
[51] Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10844-10853, 2024.
[52] Hanyang Zhao, Haoxian Chen, Zhang, David D Yao, and Wenpin Tang. Score as action: Fine-tuning diffusion generative models by continuous-time reinforcement learning. arXiv preprint arXiv:2502.01889 2025.
[53] Po-Hung Yeh, Kuang-Huei Lee, and Jun-Cheng Chen. Training-free diffusion model alignment with sampling demons. arXiv preprint arXiv:2410.05760, 2024.
13

---

[54] Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, and Tsung-Hie Hui Chang. Tuning-free alignment of diffusion models with direct noise optimization. arXiv preprini arXiv:2405. 18881, 2024.
[55] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference Machine Learning, pages 32483-32499 PMIR, 2023.
[56] Xiaohui Sun, Ruitong Xiao, Jianye Mo. Bowen Wu, Qun Yu, and Baoxun Wang. F5r-tts: Improving flow matching based text-to-speech with group relative policy optimization. arXiv preprint arXiv:2504. 02400, 2025.
[57] Jaihoon Kim, Taehoon Yoon, Jisung Hwang, and Minhyuk Sung. Inference-time scaling for flow models via stochastic generation and rollover budget forcing. arXiv preprint arXiv:2503. 19385, 2025.
[58] Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model. arXiv preprint arXiv:2503.07703 2025.
[59] Chrisoph Schuhmann. Laion aesthetics, Aug 2022.
[60] Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress accurate image quality scores using score distribution. arXiv preprint arXiv:2501.11561, 2025.
[61] Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.052 05236, 2025.
[62] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[63] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
[64] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint 0004022204. 1(2):3, 2022.
[65] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo. et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall--- 3. pdf. 2(3):8, 2023.
[66] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijic Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
[67] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint 2409 18869, 2024.
[68] Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation. arXiv preprint arXiv:241 1.07975, 2024.
[69] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint 17811 2025.
14

---

[70] Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, et al. Sana 1.5: Efficient scaling of training- time and inference-time compute in linear diffusion transformer. arXiv preprint (rXiv:2501.8442) 2025.
[71] Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXh: 240.24464, 2025.
[72] Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-neemotron Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505. 16400, 2025.
[73] Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i- compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[74] Bernt Øksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
[75] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326 1982.
[76] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weigghted regression: Simple and scalable off-policy reinforcement learning, arXiv preprint arXiv:/910.00177, 2019.
15

---

Appendix of Flow-GRPO: Training Flow Matching Models via Online RL
Mathematical Derivations for Stochastic Sampling using Flow Models
17
Further Details on the Experimental Setup
18
B.1
Quality Metrics
18
B.2 Model Specification
19
B.3 Hyperparameters Specification
19
B.4 Compute Resources Specification
19
Extended Experimental Results
19
C.1 Flow-GRPO vs. Other Alignment Methods
19
C.2 Effect of Denoising Reduction
21
C33 Effect of Initial Noise
................................
C,4 Additional Results on FLUX.I-Dev
22
C.5 Learning Curves with or without KL
22
C.6 Additional Qualitative Results
22
C.7 Evolution Evaluation Images During Flow-GRPO Training
22
Training Sample Visualization with Denoising Reduction
22
Our Appendix consists of sections. Readers can click on each section number to navigate to the corresponding section:
Section A provides detailed derivations of stochastic sampling in flow matching models.
Section presents details about our experimental setup.
Section offers some additional experimental results, including 1) the comparison with other alignment methods, 2) ablation of denoising reduction on OCR accuracy and pickscore, 3) ablation initial noise, 4) additional results on FLUX.I-Dev, 5) the learning curves of Flow- GRPO on three tasks, 6) additional qualitative results, and 7) evolution of evaluation images during training.
Section provides a visualization of training samples under the denoising reduction strategy.
In addition to this Appendix, we also provide more visualization results, see this website. We encourage the readers consult this HTML page for more intuitive assessment of the improvements brought by Flow-GRPO.

---

A Mathematical Derivations for Stochastic Sampling using Flow Models
We present detailed proof here. To compute (0 3t, c) Equation during forward sampling, we adapt flow models to stochastic differential equation (SDE). While flow models normally follow a deterministic ODE:
dæ. 0,d
(10)
We consider its stochastic counterpart. Inspired by the derivation from SDE to its probability flow ODE in SGMs [23], we aim construct forward SDE with specific drift and diffusion coefficients so that its marginal distribution matches that of Eq. 10. We begin with the generie form of SDE:
dæ; = /SDE (r,, t)dt + o,dw,
(11)
Its marginal probability density pe (x) evolves according to the Fokker- equation [74], i...
dept (x) (< p) (z)
(12)
Similarly, the marginal probability density associated with Eq. 10 evolves:
d.p.(x) - [w (a))]
(13)
To ensure that the stochastic process shares the same marginal distribution as the ODE, we impose:
LISDE Pi (2:) + ] [o]p(())
[ur( (x)]
(14)
Observing that
(o(opp((<)) = pr
= o (Vpc(z))
(15)
= -
Substituting Eq. 15 to Eq. 14, we arrive at the drift coefficients of the target forward SDE:
fSDE = ve( t) + of V (x))
(16)
Hence, we can rewrite the forward SDE in Eq. 11 as:
ty (xx)) + of V (og) dt o,dur
dæ;
(17)
where dw denotes Wiener process increments, and of the diffusion coefficient controlling the level of stochasticity during sampling.
The relationship between forward and reverse-time SDEs has been established in [75, 23). Specifically, the forward SDE takes the form
dæ. f(x,, g(t) dw,
(18)
then the corresponding reverse-time SDE is
dx; = [f - (ogg((())) dt + g(t) dw.
(19)
Setting g(t) = Gt. we obtain the reverse-time SDE corresponding to Eq. 17 as
oi v Voggp(( o,V lo di + o,dW.
dæ;
(x(e))
(20)
We thus arrive at the final form of the reverse-time SDE:
07
dæ;, = (xx(xx))
log p((p)))) dt + o,dw,
(21)
2
17

---

Once the score function V Voog (zz) is available, the process can be simulated directly. For flow matching. this score is implicitly linked the velocity field vt.
Specifically, let di = / All expectations are over Bo ~ Xo and æ1 ~ (N, I), where Xo is the data distribution.
For the linear interpolation a; = 0,30 + 3,01, we have:
Po ( (r)) N (æx, 0g 8 II)
(22)
yielding the conditional score:
04300
log (=)
(23)
$3
Bg
The marginal score becomes:
lo
E V log Pija (201(200) 2
1
Blær
(24)
3g
For the velocity field oy (2;), we derive:
0r (x)
di
3,00
04 = x
di [
x]]
+ B,11B2
xz = æc]
di
Dt
9,301
2t
+ 1//EE
æt xx)
(25)
x
E[z
+ 8EEE1
Dt = x
at
di
ai
B,
V lo).
at
at
Substituting a; 1 and Br I simplifies Equation 25 to:
t
vx(x)
(z))
(26)
t
Solving for the score yields:
1-t
logg,(
Us (x).
(27)
Substituting Equation 27 into 21 gives the final SDE:
dæ; (x(1 + of (x: (1 - t))(( dt + owdw.
(28)
2t
Applying Euler-Maruyama discretization yields the update rule:
o² (xc + (1 - t) t)) At At+ oooott
K++A: xz +
vo(x,, t) +
(29)
2t
where c ~ (o. I) injects stochasticity.
B
Further Details on the Experimental Setup
B.1
Quality Metries
The details of quality metrics are as follows:
18

---

Aesthetic score [59]: CLIP-based linear regressor that predicts an image's aesthetic score.
DeQA score [60]: a multimodal large language model based image-quality assessment (IQA) model that quantifies how distortions, texture damage, and other low-level artefacts affect per- ceived quality.
ImageReward [32]: a general purpose T2I human preference reward model that captures text-image alignment, visual fidelity, and harmlessness.
UnifiedReward [61]: recently proposed unified reward model for multimodal understanding and generation that currently achieves state-of-the-art performance on the human preference assessment leaderboard.
B.2
Model Specification
The following table lists the base model and the reward models and their corresponding links.
Models
Links
SD3.5-M.
5-medium
Aesthetic Score [59]
httpa://github. httpp///gggtttt..o////aaaaaa/att
PickScore [19]
https:
DeQA score [60]
ImageReward [32]
UnifiedReward [61]
B.3 Hyperparameters Specification
Except for 3, GRPO hyperparameters fixed across tasks. We sampling timestep T 10 and an evaluation timestep I' = 40. Other settings include a group size G = 24, an noise level a = 0.7 and an image resolution of 512. The KL ratio 3 is 0.04 for GenEval and Text Rendering, and 0.01 for Pickscore. We use Lora with 64 and r 32.
B.4 Compute Resources Specification
We train our model using 24 NVIDIA A800 GPUs. The learning curves in Appendix C.5 provide details on the specific GPU hours.
C Extended Experimental Results
C.1 Flow-GRPO vs. Other Alignment Methods
We compare Flow-GRPO with several alignment methods: supervised fine-tuning (SFT), reward- weighted regression (Flow-RWR [14, 76]). Flow-DPO [14], and their online variants. Flow-GRPO consistently outperforms all baselines by significant margin. At each step, generate group of images using the same group size as in Flow-GRPO. The only difference lies in the update rule:
SFT: Select the highest-reward image each group and fine-tune on
Flow-RWR [14, 76]: Apply softmax over rewards in each group and perform reward-weighted likelihood maximization.
Flow-DPO [14, 39]: Use the highest-reward image in each group as the chosen sample and the lowest the rejected, then apply the DPO loss.
Offline variants use fixed pretrained model for data collection, while online variants update their data collection model every 40 steps. As shown in Figure 8, Flow-GRPE outperforms all other methods. The figure also indicates that DPO and SFT improve over time. In contrast, RWR does not, which aligns with experimental findings on RWR [12]. Additionally, Online DPO surpasses offline DPO, aligning with [15]'s finding that online DPO performs better. For the second-best online DPO, a hyperparameter search on its key parameter revealed that smaller values are not always optimal; excessively small 3 values cause training collapse.
19

---

24
Flow GRPO
PickScore Evel
Online DPO 100)
Offlline
Online SFT
Offline SFT
22
Offine RWR
2500
5000
7500
10000
12500
15000
17500
Training Prompt Number
23
PickScore Eval
Flow GRPO
DDPO
400
600
800
1000
1200
1400
Training Steps
Figure 8: Comparison Flow-GRPO and Other Alignment Methods on the Human Preference Alignment task. Since methods like DPO use different tuned batch sizes from Flow-GRPO use the number of training prompts on the x-axis for a fair comparison across these methods.
DDPO. DDPO [12] was originally developed for Liffusion-based backbones, so we adapted it to how-matching models via our ODE-to-SDD conversion. Using SD3.5-M as the base model and PickScore the reward signal, we track the evaluation reward throughout the entire training process in Figure 8, We find that DDPO's reward increases more slowly than Flow-GRPO's 'ss and eventually collapses in the later stages, whereas Flow-GRPO trains stably and continues to improve consistently over time.
ReFL. ReFL [32] directly fine-tunes diffusion models by viewing reward model scores as human preference losses and pack-propagating gradients randomly-picked late timestep (. Following lewae [32], we pack-propagate gradients to a randomly chosen late timestep / E [30, 40) during denoising. Figure shows that GRPO surpasses ReFL. when the reward differentiable, indi- cating that GRPO maintains strong performance in settings where ReFL applies. More importantly, GRPO does not require differentiable rewards, enabling direct use of state-of-the-ar Vision-Language Models (VLMs) as reward providers. This offers two key advantages:
Sophisticated, General-Purport Rewards: VLMs can conduct human-liie like evaluations through a structured reasoning process. Given prompt, a VLM can decompose it into key criteria, reason step by step to verify each aspect in the generated image, and then provide comprehensive overall score. This enables single, unified reward model to handle diverse tasks, from text-to-image generation to complex instruction-
image editing.
The field of VLMSs is advancing at a breathtaking pace. By using VLM the reward source, our framework automatically benefits from these
Future-Proof and Cost-Free Upgrades:
20

---

improvements. As VLMs become more capable, the reward model becomes stronger without any additional training data or computational cost.
ORW. ORW [35] is an online reward-weighted regression method that guides the model to prioritize high-reward regions. Unlike KL regularization, it employs Wasserstein-2 regularization prevent policy collapse and maintain diversity. To ensure fair comparison, we adopt the same experimental setup as in our Human Preference Alignment task. For ORW, we set 3 0.5 and = (lower values led to unstable training). The steps_per_eppee _ess__eepppchhhh parameter, which controls how frequently data-collecting policy updated, was chosen from 20, 40, 100, 400 based on best performance. Table 5 reports reward scores the test set across training steps. Following ORW's Table randomly sampled 50 DrawBench prompts and generated 64 images per prompt to compute CLIP and Diversity scores. As shown in Table 6, Flow-GRPO outperforms ORW on both metrics.
Table Reward scores on the set over training steps.
Method
Step 0
Step 240
Step 480
Step 720
Step 960
SD3.5 M + ORW
28.79
29.05 29.10
29.15 29.17
27.58 29.51
23.05
SD3.5-M Flow-GRPO
28.79
29.89
Table 6: Comparison of CLIP and diversity scores across different fine-tuning methods.
Method
CLIP Score f
Diversity Score
SD3.5-M
27.99
0.96
SD3.5-M ORW
28.40
0.97
SD3.5-M Flow-GRPO
30.18
1.02
C.2 Effect of Denoising Reduction
We show the extended Denoising Reduction ablations of Visual Text Rendering and Human Preference Alignment tasks in Figure 9.
DCR Eval Accurscy
Piokscore
OCR, Step
PickScore, Step
OCR, Step 10
PickScore, Step 10 PickScore, Step
Step
200
400
600
800
1000
200
400
600
800
1000
1200
1400
1600
Training (GPU
Training
(b) Human Preference Alignment
(a) Visual Text Rendering
Figure 9: Effect of Denoising Reduction
C.3 Efffet of Initial Noise
We initialize each rollout with difference random noise to increase exploratory diversity during RL training. We perform an additioanl ablation to confirm this claim. With SD3.5-M as the base model and PickScore the reward, we compare Flow-GRPO with different initial noise against RROwGGPOO with the same initial noise. Figure 10 shows the variant with different noise consistently achieved high rewards during the training process.
21

---

23.5
FLUX-1
Flow-GRPO
(23:43)
Pickscore Evel
23
23.0
PickScoree
22.5
22
22.0
FLUX. Bev (21.94)
GRPO,
Flow GRPO, same inital noise
21.5
200
400
600
800
1000
1200
1400
Training Steps
Training Stops
Figure 10: Effect of Initial Noise
Figure 11: Additional Results on FLUX.I-Dev
C.4
Additional Results on FLUX.I-Dev
We run Flow-GRPO on FLUX. [5] using PickScore as the reward signal. The reward curve rises steadily throughout training without noticeable reward hacking. Figure 11 shows the reward values over the training process, and Table 7 compares FLUX.1-Ded with FLUX.I-Dev Flow-GRPO on DrawBench.
Table Comparison of FLUX.I-Dee and Flow-GRPO fine-tuned models.
Model
Aesthetic
DeQA
ImageReward
PickScore
UnifiedReward
FLUX.I-Dev
5.71
4.31 4.24
0.85
22.62
3.65
FLUX.1-DDO + Flow-GRPO
6.02
1.32
23.97
3.81
C.5 Learning Curves with or without KL.
Figure shows learning curves for three tasks, with and without KL. These results emphasize that KL regularization is not empirically equivalent to early stopping. Adding appropriate KL can achieve same high reward as the KL-free version and maintain image quality, though it requires longer training.
C.6 Additional Qualitative Results
Figures 13, 14 & 15 qualitatively compare SD3.5-M with its Flow-GRPO enhanced versions (with and without KL regularization) using GenEval, OCR and PickScore rewards, respectively. Flow- GRPO with KL regularization improves the target capability while maintaining image quality and minimizing
Conversely, removing the KL constraint significantly degrades image
quality and diversity.
C.7 Evolution of Evaluation Images During Flow-GRPO Training
To better understand the training dynamics of our proposed Flow-GRPO framework, we visualize the evolution of generated samples corresponding to fixed evaluation prompts at regular intervals during training in Figure 16, 17 & 18. For consistency, all visualizations are produced using a 40-step baeed sampling schedule. These qualitative results provide visual representation of how the model progressively improves its generation quality and alignment with task objectives over time.
D Training Sample Visualization with Denoising Reduction
In this section, we compare images obtained with SDE sampling at various steps against those produced by ODE sampling, and offer an intuitive view of the denoising reduction strategy. Figure 19 presents SD3.55Medium samples under four inference settings:
(a) ODE sampling with 40 steps; (b) SDE sampling with 40 steps; (c) SDE sampling with 10 steps; (d) SDE sampling with steps.
22

---

OCR, Eval Accuracy
GenEval Score Score
0.8
0.7
0.7
GenEval,
0.6
GenEval,
0.5
1,000
5000
0.5
1000
Training Stops
Traininng Steps
(a) Compositional Image Generation
(b) Visual Text Rendering
24.0
23.5
23.0
Prolscore
22.5
22.0
21.5
RickScore, PickScore, w/ KL
21.0
20000
2500
300c
Training Stops
(c) Human Preference Alignment
Figure 12: Learning Curves with and without KL. KL penalty slows early training yet effectively suppresses reward hacking.
The 40-step ODE and SDE runs yield visually indistinguishable images, confirming that our SDE sampler preserves quality. Shortening the SDE schedule to 10 and 5 steps introduces conspicuous artifacts, like color drift and fine details blur. Contrary expectation that such low-quality samples might hinder optimization. it actually do just the opposite and accelerate optimization. Because Hlow-GRPO relies on relative preferences, it still extracts a useful reward signal, while the shorter trajectories signifactly cut wall-clock time. Consequently, Flow-GRPO with denoising reduction strategy converges more quickly on both layout-oriented benchmarks such as GenEval and quality- focused metrics such as PickScore, without sacrificing final performance.
23

---

SD-3.5-M
Flow-GRPH GRPO
Flow-GRPO(w/o KL)
photo traffic lightand backpack
photo of fire hydrant and a tennis racket
STOP
STOP
STOP
STOP
STOP
photo yellow stop sign potted plant
photo of a blue pizza yellow baseball glove
photo zebra left elephant
photo a above sink
photo three hydrants
Figure 13: Additional Qualitative comparison between the SD3.5-M and SD3.5-M Flow-GRPO trained with GenEval reward.
24

---

3D-3.5-M
Flow-GRPO
Flow-GRPO(w/o KL)
BINARY
BINARY STANDUP
TINARY STANDUP CLUB
CL
STANDUP
bustingg city street at night, illuminated by neon lights, featuring retro futuristic mhot comedy club with a vihrant marquee that meas "Tonight Binary StandUpp Robots guther laughing chottingg, creating, lively alimusphere
AJ
Al Training Zone
Training zone
TRAINING
realistic phuto afa tech gord night, featuring gloving 'Al Training Zone" bulogram flusting the center, surrounded
buildings greenery, soft ambient lighting enkancing the futuristic atmossphre.
STREET ART RULES
K
RULES STREET ART
STREET -ART RULESS
STREET RILES
STA RieS
TOWARLES
vibrant urban alley with graffiii wall prominently spray-painted 'Street Rules", surrounded colorful and murals, under
sunny sky.
COLCOME VEGAS
Welcome Vegas
WelcoMe TO VEGAS
EGAS
WELCOME VEGAS
ncon museum sign in retro style, gloving hrightly with the wards "Welrame to Vegas", setagainst dark urban nighe with hint buckground.
DANGER
Edge Unstable
DANSER
EXEE INSTABLE
DANGER EDGE UNSTABLE
DANGER EDGE UNSTABLE
A reolistic photograph ofa WOTT rood sign neara rugged clift, reading "D Jngger Edgge Unstable", with the vast, misty landscupe stretching beyond.
WANTED
WANTED
BANDIT STOLE THE
WANTED
BANDIT STOLLE
BANDIT "Whe Stale the the D
D
Wanted poster with rugged, old parchment texturs, offering substantial reward for capture of "Bandit Who Stole the D" The puster is tattered, with faded sketch of the bandit, and pinned to wooden beard a busting, dusty frontier town.
WORD'S BEST DELI
WORLD'S BEST DELI
VIRLT DU
Voris Bas
bustlingg street charming sturefrunt prosninenty displaying "Wurld's elegent lettering surrounded by vibrant window displays and happy custumers.
Figure 14:
Additional Qualitative comparison between the SD3.5-M and SD3.5-M Flow-GRPO trained with OCR reward.
25

---

SD-3.5-M
Flow-RRPO
Flow- (WW- KL)
Lunch Bavaria painting
Young man with an orange beard, cartoon style
ALL ALL
ALL
ALL
ALL
ALL
the word "ALL" text grafftii style, shirt design, detailed
fantasy, pastel, abourdist, photo, pineapple
quick doodle of medium hair ith long bangs, hd detailed detailed
CAFE
CAFE
A modern and minimalist lago for cafe, tescup icon
fantasy, pastel, absurdist st, photo, tiny teapot house matchbox
Figure 15: Additional Qualitative comparison between the SD3.5-M and SD3.5-M Flow-GRPO trained with PickScore reward.
26

---

Training Process on GenEval Task
photo of four boats
STOP
STOP
STOP
STOP
STOP
STOP
STOP
STOP
photo of left stop sign.
photo of a blue pizza and a yellow baseball glove.
Figure 16: We visualize the generated samples across successive training iterations during the optimization of SD3.5-Medium on the GenEval task.
Training Process on OCR Task
COISTON
Casphne Susjectts
CANTIRN
GE
laboratory setting with cage prominently displayed. the cage label reads caution telepathic subjects in bold letters, with a warning symbol. the environment is sterile and clinical, emphasizing unusual nature experiment.
Tost City
Cles Neur
M
City Near
Last
Les Now
Lunciy
a weathered cave explorer's journal page, with the phrase lost city near prominently written faded ink, surrounded by sketches of ancient ruins and cryptic symbols, under dim, mystical light.
THY NEW
BURGER
NEW BURGER
TRY NEW BURGER
TerOre NEW BURGER
Try New BURGER
BUNSE
BURGER
a realistic photograph of a fast food drive thru menu board at dusk, featuring a bold and colorful advertisement that reads try burger with an appetizing image burger below, against the backdrop of busy suburban street.
Figure 17:
We visualize the generated samples across successive training iterations during the optimization of SD3.5-Medium on the OCR task.
27

---

Training Process on PickScore Task
the and robots, his study, with pictures his desk the wall
Teddy car, workshop spaceship, inside model of mgb, trek shuttle bay
woman on top of horse
Figure 18: We visualize the generated samples across successive training iterations during the optimization of SD3.5-Medium on the PickScore task.
28

---

ODE Sampling (Step = 40)
SDE Sampling (Step 40)
Hupey
HaWpyy
the
Burbony
8000 Fast
SDE Sampling (Step 10)
SDE sampling (Step = 5)
Bithosy
Swidit Loutse
Figure 19: Visualization of training samples under difference inference settings.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our main contribution are detailed in Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Sec. for limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Yes. We provide the detailed derivations of SDE sampling in flow models, see Appendix of our attached file.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail the experimental settings See 5.1. We believe our experimental results can be easily reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code and data in the supplementary material, along with detailed instructions for installation and usage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
caasssinoollcyy icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cccpublic//eeecccccccccccccc wwtpc//gg//eeeeeee ottp://oouuniioooooooooooooo.o
ssi ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detail the experimental settings Sec 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due the resource limitation, we do report error bars. As reported in Ap- pendix B.4, we spent numerous resources for our experiments, which makes prohibitively expensive to run each experiments for multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the GPU resources used in our experiments in Appendix B.4, and provide GPU hour statistics in Fig. 9.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:://neuupppeee wtt.pppp..ppppppeeeeepeepppppppp cc/public/Ethie csGuidelines"
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work focuses publicly available academic benchmarks and datasets (c.g., GenEval), and does not involve any private or personal data. So we do not identify any explicit negative societal impacts arising from this work.
Guidelines:
33

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode wwwccoo///////// datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htaps::/nnerris... http///rppp.ccoooooorereeee/e/// for what should or should not be described.
36


Reviews:
[{"review_id": "UYedriBA8C", "reviewer": "Reviewer_2Psq", "summary": "- This work proposes Flow-GRPO, the first method to perform online RL in flow matching models. \n- There are two problems to applying online RL methods like GRPO directly to flow matching models. 1) Flow matching models rely on a deterministic process based on ODE. As a result, the sampling during inference is not stochastic while online RL methods rely on exploration i.e. stochasticity during inference and 2) Online RL methods rely on efficient sampling to collect training data. Flow models inherently require multiple denoising steps during inference for 1 generation. This affects the efficiency of online RL methods.\n- Flow GRPO solves both these problems by 1) converting the **ODE to an SDE** thereby making the inference stochastic and 2) applying **Denoising reduction** strategy. The denoising reduction strategy uses fewer de-noising steps during training but uses the full schedule during inference.\n- Authors show that unlike other online RL methods, Flow-GRPO shows minimal reward hacking and gets almost perfect score on GenEval outperforming GPT-4o. On visual text-rendering, SD3-M's accuracy increases from 59% to 92%.", "questions": "- Effect of ODE-to-SDE conversion. How much does this affect metrics like FID/SSIM/ human evaluations? \n- Why does using fewer steps improve reward? In Fig. 5a, Step=10 outperforms Step=40 in GenEval score as well. I understand that speed is definitely improved with 4x reduction in steps but the reward also doesn't go above 0.9 with Steps=40? Is it because of the number of updates? Is the graph adjusted for the number of updates under each setting?\n- How strong is the generalization? In Table-4, authors show that the model generalizes well for objects > 4. When does it break? \n- All the Text rendering examples show text with few words. Will the model be robust to long visual text as well? When does the model break?", "limitations": "yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- This paper proposes a solution to a very hard and important problem, applying online RL to diffusion/flow matching methods and shows that, similar to text generation, online RL improves media generation significantly as well. This work would be very useful to the community.\n- The paper is well written and has enough details for proper reproduction. \n- Qualitative and quantitative results are impressive.\n\n**Weaknesses**\n- I see tremendous value in the current work for media generation community and I don't see any potential weaknesses of this work.", "was_revised": true, "final_justification": "I thank the authors for addressing my questions. Authors have satisfactorily addressed my questions. After going through other reviews and author responses, I vote to retain my initial rating.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n\n` [Q1] How does this ODE-to-SDE conversion affect metrics like FID/SSIM/ human evaluations? `\n\n\nWe evaluate the impact of converting ODE sampling to SDE sampling using SD‑3.5‑M as the base model. We compared 40-step ODE and 40-step SDE sampling on MSCOCO‑2014 for FID/IS/CLIP Score, and on DrawBench for aesthetic/DEQA/ImageReward/PickScore/UnifiedReward. The two settings yield essentially the same performance at 40 steps, which aligns with our derivation in Sec. A showing that both share the same marginal distribution. For clarity, SDE sampling is only used during training; all reported metrics are computed from 40-step ODE samples. Consequently, the ODE-to-SDE conversion has no material impact on FID/ID or human-preference metrics in our experiments.\n\n\n\n$$\n\\begin{array}{l|ccc|ccccc}\n\\hline\n & & \\textbf{MSCOCO-2014} & & & & \\textbf{DrawBench} & & \\newline\n\\textbf{Model} & \\textbf{FID ↓} & \\textbf{IS ↓} & \\textbf{CLIP Score ↑} & \\textbf{Aesthetic ↑} & \\textbf{Deqa ↑} & \\textbf{ImageReward ↑} & \\textbf{PickScore ↑} & \\textbf{UnifiedReward ↑} \\newline\n\\hline\n\\text{ODE, 40 steps} & 17.50 & 40.81 & 31.95 & 5.39 & 4.07 & 0.87 & 22.34 & 3.33 \\newline\n\\text{SDE, 40 steps} & 16.43 & 41.22 & 31.70 & 5.36 & 4.01 & 0.85 & 22.34 & 3.33 \\newline\n\\hline\n\\end{array}\n$$\n\n\n\n---\n\n` [Q2-1] Why does using fewer steps improve reward? `\n\n\nThank you for the question. We view denoising reduction as a strategy to boost efficiency, not to raise the ultimate reward ceiling. In addition to Figure 5, Section C.2 in the Appendix extended ablations on visual text rendering and human-preference alignment. Across these three tasks, a 10-step schedule converges to similar final rewards compared to the 40-step baseline while cutting wall-clock training time by roughly 4x. The higher reward observed therefore reflects faster convergence, not a higher performance.\n\n---\n\n` [Q2-2] The number of updates in the graph `\n\nIn Figure 5, we measure progress by GPU‑hours, i.e., wall‑clock cost. If we re‑index the curves by the number of optimizer updates, the 10‑step and 40‑step schedules are nearly equivalent. One difference is that each update in the 40-step schedule requires four times the accumulation of the 10‑step schedule's, owing to its 4x longer denoising steps.\n\n\n\n\n---\n\n` [Q3] How strong is the generalization? In Table 4, the authors show that the model generalizes well for objects > 4. When does it break? `\n\nThank you for the suggestion. Following the protocol of Table 4, we extend the generalization test and require the model to generate images with 8/10/12 objects, an extremely challenging setting in which the base SD-3.5-M model almost never succeeds. Flow-GRPO still achieves a substantial improvement, although the absolute success rate naturally decreases as the task becomes more difficult. The results are summarised below:\n\n\n$$\n\\begin{array}{l|ccc}\n\\hline\n\\text{Method} & \\text{8 objects} & \\text{10 objects} & \\text{12 objects} \\newline\n\\hline\n\\text{SD3.5-M}  & 0.02 & 0.01 & 0.02  \\newline\n\\textbf{SD3.5-M + Flow-GRPO}  & \\textbf{0.19} & \\textbf{0.07} & \\textbf{0.12}  \\newline\n\\hline\n\\end{array}\n$$\n\n---\n\n` [Q4] Will the model be robust to long visual text as well? When does the model break? `\n\n\nThank you for the suggestion. Our text-rendering checkpoint was trained on short phrases (2-4 words), so we further evaluate its generalization on prompts containing a full 10-word sentence. For illustration, a representative long‑text prompt is:\n\n> The final, unplaced piece of a giant, intricate jigsaw puzzle, held in a hand above the puzzle. The piece itself has the words, “The final piece that makes the entire picture make perfect sense.\n\n\n\nIn this harder setting the base SD-3.5-M model degrades sharply in accuracy, whereas Flow-GRPO maintains most of its performance. The results are summarised below:\n\n\n$$\n\\begin{array}{l|cc}\n\\hline\n\\text{Method} & \\text{2-4 words} & \\text{10 words} \\newline\n\\hline\n\\text{SD3.5-M}  & 0.59 & 0.28   \\newline\n\\textbf{SD3.5-M + Flow-GRPO} & \\textbf{0.92} & \\textbf{0.82}   \\newline\n\\hline\n\\end{array}\n$$"}, {"review_id": "GMMHKkok1N", "reviewer": "Reviewer_UUcc", "summary": "This paper proposes a method to fine-tune a pre-trained Flow-Matching (FM) sampler via online reinforcement learning (RL). By leveraging the SDE formulation of FM, the authors successfully reduce the number of FM sampling steps required at inference time.", "questions": "- **Q1**: Why choose flow matching as the base model? How would the approach differ if a diffusion model were used?  \n- **Q2**: As I am not specialized in this area, could you explain the design principles behind the reward function $f$ defined on line 119?  \n- **Q3**: Again, from a non-expert perspective, what are the main factors limiting the scalability of the proposed method?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**  \n- **Quality**: The utility of the method is supported by comprehensive, multi-faceted experiments.  \n- **Clarity**: The core ideas are presented in a clear and understandable manner.  \n- **Significance**: Fine-tuning generative samplers offline via RL is novel, and the fact that training steps can be drastically reduced without loss of performance is particularly interesting.\n\n**Weaknesses**  \n- **Originality**: For readers already familiar with FM theory, the idea may appear straightforward.", "was_revised": true, "final_justification": "## Resolved Issues\n\n- **Q1 (Choice of Flow Matching):**  \n  Authors explained that flow-matching is now the leading backbone in SOTA image/video generators, motivating direct post-training with RL.\n\n- **Q2 (Reward Function Design):**  \n  Clarified that `f` is the GRPO surrogate objective, and its clipping-based trust region, advantage weighting, and KL penalty jointly ensure stable, monotonic policy improvement.\n\n- **Q3 (Scalability Factors):**  \n  Denoising-reduction for ~4× speed-up and the KL term’s role in curbing reward hacking are now clearly articulated.\n\n## Remaining Concerns\n\n- The core idea—injecting stochasticity via an SDE formulation and applying a standard GRPO update—is relatively simple. Some readers may seek deeper theoretical novelty.\n\n## Weighting of Aspects\n\n- **Empirical Validation (High weight):**  \n  Comprehensive experiments (GenEval, text rendering, human preferences) convincingly demonstrate effectiveness.\n\n- **Clarity (Medium weight):**  \n  Rebuttal improved my understanding; clarity score raised to 3.\n\n- **Originality (Low–Medium weight):**  \n  While straightforward, the work fills an important gap by validating RL on flow-matching samplers; originality score raised to 3.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n\n---\n\n\n` [W1] About the originality; `\n\n\nThank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: *Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models.* Before our work, PPO/GRPO have shown remarkable success in understanding tasks with LLMs, while their applicability to flow-matching generation models has not yet been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:\n\n- **We establish a well-defined MDP for flow-matching generators.** GRPO presupposes stochastic, step-by-step transitions so that a policy $p_{\\theta}(a_t \\mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with controlled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.\n- **We introduce several practical strategies that make online RL both efficient and robust.** We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speedup. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.\n- **We provide empirical validation of the effectiveness of Flow-GRPO.** We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framework is a promising path for enhancing SOTA generative models.\n\n\n---\n\n` [Q1-1] Why choose flow matching as the base model?`\n\nWe choose flow matching models because it provides faster sampling speed at comparable quality. Contemporary leading image generation models (e.g., SD3 [1], FLUX [2]) and video generation models (e.g., Sora [3], Wan2.1 [4], Seedance [5]) now adopt flow matching in place of traditional diffusion, which means our method could apply directly to their post‑training stage.\n\n[1]. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.  \n[2]. FLUX.  \n[3]. Sora: Video generation models as world simulators.  \n[4]. Wan: Open and Advanced Large-Scale Video Generative Models.  \n[5]. Seedance 1.0: Exploring the Boundaries of Video Generation Models.  \n\n---\n\n` [Q1-2] How would the approach differ if a diffusion model were used`\n\nThanks for asking this clarification question! \n\n1. **GRPO requires an MDP with stochastic step-to-step transitions so that a policy $p_{\\theta}(a_t \\mid s_t)$ is well defined.** A flow-matching model, in its standard ODE form, is purely deterministic, and there is no randomness in a single denoising step. To address it, we therefore convert the ODE to an equivalent SDE to inject randomness, creating the stochastic transitions needed to instantiate the MDP.\n2. **Our proposed method could be applied directly in diffusion models without modification.** If a diffusion model were used instead, this stochasticity is already built in their DDPM-style sampling, so the same pipeline in Flow-GRPO can be directly utilized in diffusion models' optimization. \n\n\n\n---\n\n` [Q2] The design principles behind the reward function f defined on line 119? `\n\n\nThank you for the question. There seems to be a misunderstanding: in our notation, `f` refers to the GRPO objective function, not the reward model (which is denoted by `r`).  A proper reward model typically needs to assign a scalar score to each generated sample (higher means better), and does not need to be differentiable. The three reward functions we adopt in the manuscript are defined on Line 168, Line 169, and Line 176.\n\nThe GRPO objective (`f`) is designed to ensure stable and monotonic policy improvement through three key principles:\n\n1. **Trust Region.** The core of the function is a clipping mechanism that uses `min` and `clip` functions to limit the magnitude of policy updates. This creates a **trust region**, allowing only small, safe adjustments around the current policy. This effectively prevents performance collapse due to overly large update steps and ensures training **stability**.\n\n2. **Advantage-Weighted Updates.** Our goal is to maximize the objective `f`, which corresponds to maximizing $r_t Â_t$. Therefore, the direction of updates is guided by the **advantage function** $Â_t$:\n\n * When $Â_t > 0$ (i.e., the action is better than average), the objective reduces to maximizing $r_t$. Since the denominator of $r_t$ has no gradient, this effectively means increasing the numerator — that is, increasing the probability of that action.\n * When $Â_t < 0$ (i.e., the action is worse than average), the probability of that action is decreased.\n\n3. **KL Divergence Regularization.** The loss function includes a KL divergence penalty term, $-β D_{KL}(π_θ || π_{ref})$. It acts as a **soft constraint**, preventing the new policy $π_θ$ from deviating too far from a known reference policy $π_{ref}$ (often the pretrained policy).\n\n\n\n---\n\n\n` [Q3] What are the main factors limiting the scalability of the proposed method? ` \n\nThank you for raising this point. We discuss two main factors that govern scalability.\n\n1. **Efficiency.** Scaling the post-training stage requires a more efficient training strategy. Our manuscript introduces a denoising reduction strategy that trains the model with only a quarter of the normal sampling steps, achieving roughly a 4x speed-up with very subtle loss in quality. Further works could attain gains by adopting a dynamic data selection, for instance, periodically discarding samples whose advantage estimates show low variance.\n2. **Avoid Reward Hacking.** Reward hacking would become increasingly problematic as the training scale grows. We observe that a KL regulariser can curb the hacking effectively, allowing more optimisation steps before the model starts to hack the reward function (see Figure 8). Even so, an ultimate safeguard is a robust and accurate reward model, which makes it harder for the generator to discover loopholes. Further work on powerful, ensemble-based rewards will therefore be one key to scaling the algorithm."}, {"review_id": "yftTitfnIm", "reviewer": "Reviewer_cqnV", "summary": "This paper presents Flow-GRPO, a method that enables online reinforcement learning for flow matching generative models by introducing an ODE-to-SDE conversion and a denoising reduction strategy. The approach achieves strong improvements in compositional image generation, text rendering, and human preference alignment on several benchmarks, while preserving image quality and diversity.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength:**\n\n1. This paper proposes to apply online RL to flow matching models by the ODE-to-SDE conversion to introduce essential stochasticity for RL.\n\n2. Denoising reduction greatly speeds up training without sacrificing output quality and show substantial gains across standard T2I benchmarks, with thorough ablations on reward hacking and generalization.\n\n\n**Weakness:**\n\n1. The conversion from Flow ODE to SDE is not a novel contribution, as stochastic sampling for flow-based models has already been explored in prior works such as [1].\n\n2. The paper’s main contributions are: (a) transferring Flow ODE to SDE, which is well-studied in prior literature, and (b) applying GRPO to flow models. Since GRPO has already been extensively explored in the context of LLMs, this work appears to be largely an application of existing techniques to diffusion models, which limits its novelty. How does the exploration-exploitation tradeoff in the Flow-GRPO compare with diffusion models or standard RL settings?\n\n[1] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers\n\nOther questions:\n\n1. The alation on the group size $G$ on GRPO.", "was_revised": true, "final_justification": "My questions have been resolved. However, I still find the novelty limited, as there are no new theoretical contributions in applying GRPO to flow matching models.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n\n---\n\n`  [W1] The conversion from Flow ODE to SDE is not a novel contribution, as stochastic sampling for flow-based models has already been explored in prior works. ` \n\n\n\nThank you for raising this concern. Our ODE-to-SDE conversion is not intended as a standalone theoretical contribution, but as a practical tool that equips Flow-GRPO with the stochastic transitions required for a valid MDP. Applying GRPO-style optimization in FM models demands a sampling procedure whose step-by-step randomness does not disturb the model's marginal distribution. We therefore construct an SDE that is distribution-equivalent to the original ODE, which is largely inspired by the prior flow-matching theory [1] and shares a similar formulation with prior works such as [2]. **The contribution lies not in the mathematics of the conversion itself, but in adapting and repurposing it to make online RL feasible and empirically effective for contemporary flow-matching backbones.**\n\n\n[1]. Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control. ICLR 2025.  \n[2]. SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers. ECCV 2024.  \n\n---\n\n`  [W2-1] This work appears to be largely an application of existing techniques to diffusion models, which limits its novelty. `\n\n\nThank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: *Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models.* Before our work, PPO/GRPO have shown remarkable success in understanding tasks with LLMs, while their applicability to flow-matching generation models has not yet been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:\n\n- **We establish a well-defined MDP for flow-matching generators.** GRPO presupposes stochastic, step-by-step transitions so that a policy $p_{\\theta}(a_t \\mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with controlled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.\n- **We introduce several practical strategies that make online RL both efficient and robust.** We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speedup. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.\n- **We provide empirical validation of the effectiveness of Flow-GRPO.** We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framework is a promising path for enhancing SOTA generative models.\n\n\n---\n\n`  [W2-2] How does the exploration-exploitation tradeoff in the Flow-GRPO compare with diffusion models or standard RL settings? `\n\nThank you for the question. In Flow-GRPO, the exploration–exploitation tradeoff is governed by the noise scale $\\sigma_t$ in Equation (9). Conceptually, $\\sigma_t$ plays a role similar to the $\\epsilon$ in $\\epsilon$-greedy strategies in standard RL settings: a larger $\\sigma_t$ increases exploration by introducing greater stochasticity into the denoising process, but at the cost of reduced policy quality and potential instability or suboptimal behaviors.\n\nWe provide an empirical analysis of this tradeoff in the paragraph starting at line 225, where we systematically vary $\\sigma_t$ and study its effect on final performance. The results demonstrate a clear tradeoff, highlighting the importance of tuning $\\sigma_t$ appropriately to balance exploration and exploitation.\n\n\n---\n\n` [W3] The ablation on the group size G on GRPO. `\n\n\nThank you for this valuable suggestion. We have added an ablation study on the group size $G$ using PickScore as the reward function. When we reduced the group size to $G=12$ and $G=6$, training became unstable and collapsed at step 778 and step 742, respectively. In contrast, $G=24$ remained stable throughout the full process. We logged the PickScore on the test set before and after the collapse, as shown below.\n\n\n$$\n\\begin{array}{l|ccc}\n\\hline\n\\textbf{Group Size} & \\textbf{Step 0} & \\textbf{Step 720} & \\textbf{Step 956} \\newline\n\\hline\nG=24 & 21.72 & 23.31 & 23.41  \\newline\nG=12 & 21.72 & 22.95 & 20.96  \\newline\nG=6  & 21.72 & 22.64 & 20.68  \\newline\n\\hline\n\\end{array}\n$$\n\nOur conclusion is that a small group size leads to inaccurate advantage estimates, inflating the variance and eventually triggering collapse. A similar phenomenon is also reported in [3, 4]. We will add these results in the revision.\n\n[3] ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. Arxiv 2025.  \n[4] AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning. Arxiv 2025."}, {"review_id": "9nmDVNtK5O", "reviewer": "Reviewer_JifQ", "summary": "This paper proposed Flow-GPRO, which adopts the original GPRO technique for pre-trained flow-based models. Specifically, the authors transformed the flow ODE into the corresponding SDE and then applied the standard GPRO technique. Experimental results on text-to-image alignment demonstrated the superior performance compared to the original model.", "questions": "See the weaknesses listed above. In addition:\n- The CLIP score is the standard evaluation metric for text-to-image alignment. Can you provide additional results for the CLIP score?\n- Can you compare the proposed method with more recent online RL approaches like ORW mentioned in the Weakness section?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n- The idea behind Flow-GRPO was clear and well-explained in the paper. The delivery and organization of the paper were concise.\n- The experimental results are significantly better than the original pre-trained model, demonstrating the effectiveness of the proposed approach.\n\n## Weakness\n- The paper claimed to be the \"first method\" to incorporate online RL into flow-based models, which I find to be **an overclaim**. For example, the Stable Diffusion 3 paper itself already applied Diffusion DPO [1] to finetune its flow-based text-to-image model. Meanwhile, the online RL methods explicitly designed for flow matching models also exist, e.g., ORW [2] that utilizes the reward-weighted loss. Therefore, the authors should avoid overclaiming. Furthermore, there are abundant approaches in RLHF that are model-agnostic and can be directly applied to align flow matching models in an online setup, e.g., RAFT [3] and DRaFT [4].\n- **The contributions seem limited to me**. The proposed method is a simple combination of GPRO and the well-known connection between flow and diffusion models (e.g., [5,6]). The authors also admitted that they directly followed the previous formulation of the conversion between ODE and SDE.\n- **RL baselines were very limited**. The authors provided comparisons with existing text-to-image models without such alignment, which are less meaningful in demonstrating the effectiveness of the proposed online RL algorithm. In this way, only the last two rows in Table 1 provided meaningful comparison, which were **insufficient in demonstrating the superior performance over other RL approaches**. Only a **limited comparison with the RL baselines** was provided in the appendix on a single metric. As this paper was not for the benchmark track, such an organization might be misleading for the RL community.\n\n[1] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Fan, Jiajun, et al. \"Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization.\" arXiv preprint arXiv:2502.06061 (2025).\n\n[3] Dong, Hanze, et al. \"Raft: Reward ranked finetuning for generative foundation model alignment.\" arXiv preprint arXiv:2304.06767 (2023).\n\n[4] Clark, Kevin, et al. \"Directly fine-tuning diffusion models on differentiable rewards.\" arXiv preprint arXiv:2309.17400 (2023).\n\n[5] Albergo, Michael S., Nicholas M. Boffi, and Eric Vanden-Eijnden. \"Stochastic interpolants: A unifying framework for flows and diffusions.\" arXiv preprint arXiv:2303.08797 (2023).\n\n[6] Domingo-Enrich, Carles, et al. \"Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control.\" arXiv preprint arXiv:2409.08861 (2024).", "was_revised": true, "final_justification": "As I have mentioned in my review and additional response to the authors' rebuttal, the original paper **overclaimed** itself as the first online RL approach for flow models, but surprisingly **did not provide any comparison** with existing online RL baselines. The new results in the rebuttal make more sense, but require a **significant rewrite** of the current paper, as **none of the original baseline results are meaningful**. \n\nBased on my understanding of the NeurIPS guideline, the paper should not be significantly different from the initial submission. In view of the above limitation, I do not think the paper should be accepted in its current form, but it should be completely revised for future submission. However, as the new results offer more reasonable comparisons, I will leave the final decision to AC on whether such a large modification is acceptable.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n\n` [W1] The paper claimed to be the \"first method\" to incorporate online RL into flow-based models, which I find to be an overclaim. ` \n\nWe appreciate the reviewer’s comment and agree that it's important to clarify what we mean by “online reinforcement learning” in this context. In our work, we specifically refer to algorithms from the typical reinforcement learning literature, such as PPO [1], GRPO [2], DQN [3], DDPG [4], and SAC [5].\n\nThis categorization is consistent with the distinctions made in prior work. For example, the Diffusion-DPO [6] paper explicitly classifies DPOK [7] and DDPO [8] as RL-based methods, while grouping other approaches into separate categories. Furthermore, the original DPO [9] paper characterizes its own contribution as “a simple **RL-free** algorithm for training language models,” reinforcing the distinction between RL-based and non-RL-based alignment methods.\n\nWe will revise the manuscript and acknowledge prior work such as Diffusion-DPO [6], ORW [10], RAFT [11], and DRaFT [12], and discuss how they relate to our setting.\n\n[1] Proximal policy optimization algorithms  \n[2] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models  \n[3] Playing Atari with Deep Reinforcement Learning  \n[4] Continuous control with deep reinforcement learning  \n[5] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor  \n[6] Diffusion Model Alignment Using Direct Preference Optimization  \n[7] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models  \n[8] Training Diffusion Models with Reinforcement Learning  \n[9] Direct Preference Optimization: Your Language Model is Secretly a Reward Model  \n[10] Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization  \n[11] Raft: Reward ranked finetuning for generative foundation model alignment  \n[12] Directly Fine-Tuning Diffusion Models on Differentiable Rewards  \n\n\n\n---\n\n` [W2] The contributions seem limited to me. `\n\n\nThank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: *Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models.* Before our work, PPO/GRPO have shown remarkable success on understanding tasks with LLMs, while their applicability to flow-matching generation models have yet not been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:\n\n- **We establish a well-defined MDP for flow-matching generators.** GRPO presupposes stochestic, step-by-step transitions so that a policy $p_{\\theta}(a_t \\mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with conytolled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.\n- **We introduce several practical strategies that make online RL both efficient and robust.** We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speed up. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.\n- **We provide empirical validation of the effectiveness of Flow-GRPO.** We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framewrok is a promising path for enhancing SOTA generative models.\n\n\n\n---\n\n` [W3] Only a limited comparison with the RL baselines was provided in the appendix on a single metric. As this paper was not for the benchmark track, such an organization might be misleading for the RL community `\n\n\n\nThank you for pointing out this issue! We agree that a broader evaluation is necessary. We have therefore added some extensive experiments that will be included in the revision. These experiments are summarised below:\n\n\n### 1. Comparison with other RL baselines on GenEval.\n\nSimilar to the Figure 6 in the appendix, we compare Flow-GPRO against offline SFT/DPO and their online variants. For each method we track the GenEval evaluation reward and apply early stopping when visual quality collapse is observed. The reward scores, measured at increasing number of training prompts, are list as below:\n\n\n$$\n\\begin{array}{lccccccc}\n\\hline\n\\text{Method} & \\text{0} & \\text{2000} & \\text{5000} & \\text{10000} & \\text{15000} & \\text{20000} & \\text{25000} \\newline\n\\hline\n\\textit{Offline Methods} & & & & & & & \\newline\n\\text{Offline DPO} (\\beta = 100) & 0.63 & 0.71 & 0.76 & 0.78 & 0.74 & 0.76 & -  \\newline\n\\text{Offline DPO} (\\beta = 1)   & 0.63 & 0.65 & 0.46 & -    & -    & -    & -  \\newline\n\\text{Offline SFT}               & 0.63 & 0.72 & 0.74 & 0.74 & 0.76 & 0.75 & 0.75  \\newline\n\\hline\n\\textit{Online Methods} & & & & & & & \\newline\n\\text{Online DPO} (\\beta = 100)  & 0.63 & 0.70 & 0.81 & 0.72 & 0.65 & 0.47 & - \\newline\n\\text{Online SFT}                & 0.63 & 0.72 & 0.45 & 0.0  & -     & -    & -  \\newline\n\\textbf{Flow GRPO (ours)}        & \\textbf{0.63} & \\textbf{0.65} & \\textbf{0.72} & \\textbf{0.80} & \\textbf{0.84} & \\textbf{0.92} & \\textbf{0.95}  \\newline\n\\hline\n\\end{array}\n$$\n\nWe observe that offline baselines like offline DPO and offline SFT hit a ceiling early, with the reward converging at roughly 0.76. Their online variants improve faster but are fragile, which emerges severe reward-hacking and ultimately causes the models to collapse. By contrast, our Flow-GRPO trains smoothly, pushing the reward up to about 0.95 without incuring significant hacking issues. \n\nNote that the reviewer mentioned RAFT: for each input $x_t$, it generates a group of $G$ responses and selects the one with the highest reward $y_t$ within the group for supervised fine-tuning. In fact, this is essentially the same as our baseline labeled \"SFT\" in the paper.\n\n\n\n### 2. Comparison with DDPO on pickscore\n\nDDPO was originally designed for diffusion backbones, so we adapted it to flow-matching models by applying our ODE-to-SDE conversion. Using SD-3.5-M as the base model and PickScore as the reward signal, we track the evaluation reward over the entire training process, as shown below:\n\n\n$$\n\\begin{array}{l|ccccc}\n\\hline\n\\text{Method} & \\text{Step 0} & \\text{Step 240} & \\text{Step 480} & \\text{Step 720} & \\text{Step 960} \\newline\n\\hline\n\\text{DDPO (with ODE-to-SDE)}  & 21.72 & 21.91 & 22.01 & 21.93 & 17.64   \\newline\n\\textbf{Flow-GRPO} & \\textbf{21.72} & \\textbf{22.53} & \\textbf{23.01} & \\textbf{23.31} & \\textbf{23.41}   \\newline\n\\hline\n\\end{array}\n$$\n\nWe observe that DDPO's reward rises more slowly than Flow-GRPO's and collapses in the late stages, whereas Flow-GRPO trains steadily and continues to improve throughout the training process.\n\n---\n\n` [Q1 & Q2] Results for CLIP Score and ORW `\n\nThank you for the suggestion. Since the official ORW repository only provides a partial implementation, we re-implemented ORW based on their repo within our reward-weighted regression baseline. The loss function is as follows:\n\n```python\nfm_loss = ((vt - ut) ** 2).mean(dim=(1, 2, 3))\nw2_loss = ((vt - vt_ref) ** 2).mean(dim=(1, 2, 3))\ntotal_loss = torch.mean(torch.exp(beta * rewards) * fm_loss + alpha * w2_loss)\n```\n\n**Experimental setup:**\n\n* **Training prompts:** Same as those used in our Human Preference Alignment task.\n* **ORW hyperparameters:** β = 0.5, α = 1 (lower values led to unstable training); `steps_per_epoch` (i.e., how frequently the data-collecting policy is updated) was selected from {20, 40, 100, 400} based on best performance.\n\n**Reward scores** on the test set over training steps:\n\n$$\n\\begin{array}{l|ccccc}\n\\hline\n\\text{Method} & \\text{Step 0} & \\text{Step 240} & \\text{Step 480} & \\text{Step 720} & \\text{Step 960} \\newline\n\\hline\n\\text{SD3.5-M + ORW}  & 28.79 & 29.05 & 29.15 & 27.58 & 23.05    \\newline\n\\textbf{SD3.5-M + Flow-GRPO} & 28.79 & \\textbf{29.10} & \\textbf{29.17} & \\textbf{29.51} & \\textbf{29.89}   \\newline\n\\hline\n\\end{array}\n$$\n\nFollowing ORW’s Table 1, we randomly sampled 50 DrawBench prompts and generated 64 images per prompt to compute CLIP and Diversity scores. We used ORW’s best-performing checkpoint on the test set (at step 480). Flow-GRPO outperforms ORW on both metrics:\n\n$$\n\\begin{array}{l|cc}\n\\hline\n\\text{Method} & \\text{CLIP Score}\\uparrow & \\text{Diversity Score}\\uparrow \\newline\n\\hline\n\\text{SD3.5-M}  & 27.99 & 0.96   \\newline\n\\textbf{SD3.5-M + ORW} & 28.40 & 0.97   \\newline\n\\textbf{SD3.5-M + Flow-GRPO} & \\textbf{30.18} & \\textbf{1.02}   \\newline\n\\hline\n\\end{array}\n$$\n\nWe will include these results in the revision."}, {"review_id": "1i9Iac7jhl", "reviewer": "Reviewer_L1rz", "summary": "This paper attempts to integrate the GRPO algorithm from LLM Online RL into Flow Matching models to enhance T2I models' capabilities in generating complex scene compositions involving multiple objects, attributes, relationships, and text rendering. To achieve this, the authors propose an ODE-to-SDE conversion that transforms the deterministic ODE into an equivalent stochastic differential equation (SDE) that matches the original model's marginal distributions across all time steps, thereby introducing the necessary stochasticity for RL exploration. Additionally, to improve RL training efficiency, the method significantly reduces denoising steps during RL training while preserving the original full step count during inference. The authors demonstrate substantial performance improvements across multiple benchmarks and claim that their approach exhibits little to no reward hacking.", "questions": "1. **Sampling strategy**: Why does the GRPO computation use different random noise to generate different final images for the same prompt, rather than using the same initial noise with different sampling paths? The authors should provide an additional experiment comparing different initial noise + ODE sampling paths versus the current approach.\n2. **Prompt ratio justification**: In Compositional Image Generation, the authors set the prompt ratio as Position : Counting : Attribute Binding : Colors : Two Objects : Single Object = 7 : 5 : 3 : 1 : 1 : 0, but provide no justification for this specific setting nor ablation studies analyzing its impact.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The authors successfully adapt LLM Online RL algorithms to Flow Matching models, improving T2I generation capabilities for complex compositional scenes involving multiple objects, attributes, relationships, and text rendering.\n2. The proposed conversion from deterministic ODE to an equivalent SDE that maintains marginal distributions across all time steps is theoretically sound. This introduces necessary stochasticity for RL exploration while preserving the model's generative characteristics.\n3. The approach of significantly reducing denoising steps during RL training while maintaining full steps during inference substantially improves sampling efficiency in the RL process and accelerates training without sacrificing inference-time image quality.\n4. Flow-GRPO demonstrates good generalization to unseen object categories and quantities. For instance, despite being trained on scenes with 2-4 objects, the model can generate scenes with 5-6 objects.\n5. The paper is well-written with clear explanations and high-quality figures and tables.\n\n\n## Weaknesses\n1. The most significant limitation is the absence of comparisons with existing Diffusion/Flow Model RL methods such as Online DPO [1], SPO [2], and D3PO/DDPO [3,4]. Without these comparisons, it is difficult to assess the true advantages of GRPO over established approaches. Specifically, Online DPO could serve as a baseline by training SD3.5-M using good-bad pairs generated based on reward model scores; SPO results could provide aesthetic quality baselines; and D3PO/DDPO could serve as baselines for other reward models.\n2. While Table 1 shows excellent results for Counting, Position, and Color reward scores, the paper lacks evaluation on other critical aspects such as aesthetic quality and text-image alignment. It remains unclear whether improvements in specific metrics come at the cost of degradation in others. If not, the authors should explain why GRPO better mitigates this issue compared to existing Diffusion/Flow Model RL algorithms.\n3. The authors' claims about reward hacking mitigation in the abstract and introduction are misleading. Upon careful reading, the alleviation of reward hacking stems primarily from reward ensembling and KL constraints rather than the GRPO algorithm itself. This raises questions about the paper's core contributions.\n4. The evaluation is restricted to SD3.5-M, leaving the effectiveness on other Flow-based models (e.g., HunyuanVideo, Flux) unclear. Additionally, the exclusive focus on Flow-based models is questionable, as Diffusion-based models can also benefit from Online RL approaches.\n5. Despite Table 4 demonstrating some generalization in categories and quantities, this capability appears constrained. For example, Table 3 shows that Flow-GRPO performs worse than the original SD3.5-M on texture generation.\n6. The technical contributions are relatively incremental. ODE-to-SDE conversion has been explored in prior works [5,6], and the authors make no algorithmic innovations to GRPO itself. While demonstrating GRPO's applicability to Flow Models, the absence of Online RL baselines and failure to establish GRPO's superiority over other Online RL algorithms in image generation (noting that reward hacking mitigation is not GRPO's achievement) weakens the contribution.\n\n## Summary\nWhile the paper presents an interesting application of GRPO to Flow Matching models, the significant limitations in experimental evaluation, particularly the absence of critical baselines and overclaimed contributions regarding reward hacking, substantially weaken its impact. The work would benefit from comprehensive comparisons with existing methods and more honest characterization of its contributions.\n\n### **Reference**\n- [1] Diffusion Model Alignment Using Direct Preference Optimization, CVPR 2024\n- [2] Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization, CVPR 2025\n- [3] Direct Preference for Denoising Diffusion Policy Optimization, CVPR 2024\n- [4] Training Diffusion Models with Reinforcement Learning, ICLR 2024\n- [5] Semantic Image Inversion and Editing using Stochastic Rectified Differential Equations, ICLR 2025\n- [6] Stochastic Interpolants: A Unifying Framework for Flows and Diffusions, arXiv'23", "was_revised": true, "final_justification": "The authors successfully addressed all my concerns, so I increase my final score to \"5: Accept\"", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. Due to space limits, part of the responses to `[W1` is included in other reviewers' rebuttals. We apologize for the inconvenience and kindly refer the reviewer to those sections for details.\n\n\n---\n\n` [W1] Comparison with other RL methods`\n\n\nThanks for your suggestions. First, in Appendix C.2, we already provide the comparison with other RL baselines, including offline SFT/DPO/RWR and their online variants. Results show that Flow-GRPO trains more stably than other RL baselines and ultimately attains the highest evaluation reward. In addition, SPO is a very insightful work, but it requires a step-aware preference model to label win-lose pairs. In contrast, our method only needs a reward signal from the final, clean image. The final-image equivalent of SPO is DPO, which we have already compared against. We will cite SPO in our revision and clarify this distinction.\n\n\nBeyond the existing results, we have also conducted additional comparative experiments, these results will be included in the revision. Due to space limitations, we kindly refer the reviewer to `Reviewer JifQ's ［W3］and [Q1 & Q2]` for further details. We will include this information in the revised version of the main paper. We summarised below:\n\n\n### 1. Comparison with other RL baselines on GenEval.\nPlease see `Reviewer JifQ's ［W3］`\n\n### 2. Comparison with DDPO on PickScore.\nPlease see `Reviewer JifQ's ［W3］`\n\n### 3. Comparison with ORW on CLIP Score. \nPlease see `Reviewer JifQ's [Q1 & Q2]`\n\n---\n\n` [W2] Whether improvements in specific metrics come at the cost of degradation in others ` \n\nIn Table 2, we have already covered the other critical aspects you mentioned. Specifically, it reports image quality metrics such as Aesthetic and DeQA, along with human preference metrics like ImageReward, PickScore, and UnifiedReward. **Flow-GRPO improves the task-specific metrics without degrading these additioanl metrics.**\n\n\n---\n\n` [W3] The alleviation of reward hacking stems primarily from GRPO itself.`\n\nWe emphasize that we do not claim to \"invent\" the KL constraint, instead our results show that a carefully tuned KL penalty can match the high rewards of a KL-free run while preserving image quality, albeit at the cost of slightly longer training. Importantly, the KL term is not viewed as a necessary component of GRPO, previous studies [1] have even reported improved scores after removing it. **Our contribution is that, through comprehensive ablation studies, we demonstrate that the KL constraint is critical to the Flow‑GRPO algorithm, enabling the reward to keep rising without any collapse in fidelity or diversity.** This is not a general consensus in online RL for generative models, we believe our finding offers useful empirical guidance to the community. \n\n\n[1]. DAPO: An Open-Source LLM Reinforcement Learning System at Scale.\n\n---\n\n\n` [W4] The effectiveness on other Flow-based models/diffusion-based models`\n\n\nThank you for this valuable suggestion. We have run Flow-GRPO on FLUX.1-Dev using PickScore as the reward signal. The reward curve rises steadily throughout the training without noticeable reward hacking. We report: \n\n**(1) The reward value over the training process**\n\n$$\n\\begin{array}{lccccc}\n\\hline\n\\text{Method} & \\text{Step 0} & \\text{Step 240} & \\text{Step 480} & \\text{Step 720} & \\text{Step 960} \\newline\n\\hline\n\\text{FLUX.1-Dev + Flow-GRPO}  & 21.94 & 22.78 & 22.98 & 23.19 & 23.36   \\newline\n\\hline\n\\end{array}\n$$\n\n**(2) Comparison between *FLUX.1-Dev* and *FLUX.1-Dev + Flow-GRPO* on DrawBench.**\n\n$$\n\\begin{array}{lccccc}\n\\hline\n\\text{Model} & \\text{Aesthetic} & \\text{DeQA} & \\text{ImageReward} & \\text{PickScore} & \\text{UnifiedReward} \\newline\n\\hline\n\\text{FLUX.1-Dev} & 5.71 & 4.31 & 0.85 & 22.62 & 3.65   \\newline\n\\text{FLUX.1-Dev + Flow-GRPO}  & 6.02 & 4.24 & 1.32 & 23.97 & 3.81    \\newline\n\\hline\n\\end{array}\n$$\n\n\nAlthough our experiment mainly focuses on flow-matching models, Flow-GRPO could be applied directly in diffusion models without modification. If a diffusion model were used instead, this stochasticity is already built in their DDPM-style sampling, so the same pipeline in Flow-GRPO can be directly utilized in diffusion models' optimization. \n\n\n---\n\n` [W5] Flow-GRPO performs worse than the original SD3.5-M on texture generation.`\n\nIn T2I‑CompBench++, the “texture generation” evaluates the ability to render material appearance (e.g., plastic, metallic, and wooden). **This axis is essentially orthogonal to the objectives we optimise in GenEval**, which focus on counting, color binding, and 2D spatial relations, and our training prompts contain no vocabulary or descriptions of object material. Given that the model receives no reward signal in that direction, a slight decline from 0.7338 to 0.7236 is unsurprising and negligible when set against the broader gains Flow‑GRPO achieves on its target tasks.\n\n\n---\n\n\n` [W6] The technical contributions are relatively incremental `\n\n\nThank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: *Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models.* Before our work, PPO/GRPO have shown remarkable success in understanding tasks with LLMs, while their applicability to flow-matching generation models has not yet been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:\n\n- **We establish a well-defined MDP for flow-matching generators.** GRPO presupposes stochastic, step-by-step transitions so that a policy $p_{\\theta}(a_t \\mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with controlled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.\n- **We introduce several practical strategies that make online RL both efficient and robust.** We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speedup. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.\n- **We provide empirical validation of the effectiveness of Flow-GRPO.** We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framework is a promising path for enhancing SOTA generative models.\n\n\n\n---\n\n` [Q1] Why use different random noise? The authors should provide an additional experiment comparing different initial noise + ODE sampling paths versus the current approach ` \n\nWe initialize each rollout with different random noise to increase exploratory diversity during RL training. We perform an additioanl ablation to confirm this claim. With SD-3.5-M as the base model and PickScore as the reward, we compare Flow-GRPO with different initial noise against Flow-GRPO with the same initial noise. The variant with different noise consistently achieved high rewards during the training process. The detailed results are reported below:\n\n\n$$\n\\begin{array}{lccccc}\n\\hline\n\\text{Method} & \\text{Step 0} & \\text{Step 240} & \\text{Step 480} & \\text{Step 720} & \\text{Step 960} \\newline\n\\hline\n\\text{Flow-GRPO, same inital noise}  & 21.72 & 22.53 & 22.98 & 23.12 & 23.30   \\newline\n\\textbf{Flow-GRPO, different inital noise}  & \\textbf{21.72} & \\textbf{22.53} & \\textbf{23.01} & \\textbf{23.31} & \\textbf{23.41}   \\newline\n\\hline\n\\end{array}\n$$\n\n\nA comparison based on “different noise + pure ODE sampling” is inapplicable. GRPO requires a stochastic, step‑by‑step transition model so that the policy $p_{\\theta}(a_t \\mid s_t)$ is well defined. ODE sampling is deterministic, therefore it does not furnish an MDP; under that setting the GRPO objective cannot be optimised.\n\n---\n\n\n`  [Q2] Prompt ratio justification ` \n\nThe prompt ratio was selected empirically according to the relative difficulty of each task. The base SD-3.5-M model handled single-object and two-object scenes almost flawlessly, yet struggled with position, counting, and attribute binding. We therefore allocated a larger portion of training prompts to the harder categories and a smaller portion to the easier ones. This trick does not affect our contributions and conclusions, as each RL baseline is trained with the identical backbone and the same training prompts."}]

Author rebuttal:

Meta-review: The authors introduce Flow-GRPO, an application of GRPO to perform preference alignment on flows, which is claimed to be the first application of an online RL algorithm to flows. As policy gradient algorithms require a stochastic policy, an ODE-to-SDE conversion is used define a stochastic sampler for the flow to use with GRPO. Denoising Reduction strategy which involves using fewer denoising steps when running GRPO and using more at evaluation time is introduced to increase training efficiency. Empirically, Flow-GRPO is shown to significantly improve text-to-image generation across multiple tasks, including compositional generation, visual text rendering, and human preference alignment. The method also demonstrates minimal reward hacking due to KL regularization and strong generalization capabilities.

Strengths:
- A simple and general way of applying online RL to flow matching models.
- The quantitative and qualitative results across diverse text-to-image tasks, outperforming GPT-4o, highlight its effectiveness.
- Flow-GRPO is shown to have some generalization capabilities to unseen object categories and counts, and robustness to reward hacking.

Weaknesses:
- Limited novelty, as both the ODE-to-SDE conversion and GRPO are existing techniques.
- Reviewers have taken issue with the claim that Flow-GRPO was the first online RL method applied to flows and the lack of insufficient discussion of prior work on preference alignment methods for flows, though the authors' response has mostly addressed this.
- The lack comparison to any preference alignment baselines in the body of the paper weakens it considerably by leaving some key questions unanswered. There were some such results in the supplementary material and more were provided in response to the reviewers, but they really need to be integrated into the paper.
- All the results in the paper are based on single runs, which is not a good practice, especially for RL algorithms.

The paper in its current form does not meet the bar for acceptance due to missing key baselines and lack of clarity about prior work, but with these issues addressed as promised in the discussion with the reviewers, it would make for a solid contribution to the literature.
Decision: Accept (poster)