Title: Flow-GRPO: Training Flow Matching Models via Online RL
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

flow matching

text to image generation
Abstract: We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into an equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks. For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from $63\%$ to $95\%$. In visual text rendering, accuracy improves from $59\%$ to $92\%$, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to integrate the GRPO algorithm from LLM Online RL into Flow Matching models to enhance T2I models' capabilities in generating complex scene compositions involving multiple objects, attributes, relationships, and text rendering. To achieve this, the authors propose an ODE-to-SDE conversion that transforms the deterministic ODE into an equivalent stochastic differential equation (SDE) that matches the original model's marginal distributions across all time steps, thereby introducing the necessary stochasticity for RL exploration. Additionally, to improve RL training efficiency, the method significantly reduces denoising steps during RL training while preserving the original full step count during inference. The authors demonstrate substantial performance improvements across multiple benchmarks and claim that their approach exhibits little to no reward hacking.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The authors successfully adapt LLM Online RL algorithms to Flow Matching models, improving T2I generation capabilities for complex compositional scenes involving multiple objects, attributes, relationships, and text rendering.\n2. The proposed conversion from deterministic ODE to an equivalent SDE that maintains marginal distributions across all time steps is theoretically sound. This introduces necessary stochasticity for RL exploration while preserving the model's generative characteristics.\n3. The approach of significantly reducing denoising steps during RL training while maintaining full steps during inference substantially improves sampling efficiency in the RL process and accelerates training without sacrificing inference-time image quality.\n4. Flow-GRPO demonstrates good generalization to unseen object categories and quantities. For instance, despite being trained on scenes with 2-4 objects, the model can generate scenes with 5-6 objects.\n5. The paper is well-written with clear explanations and high-quality figures and tables.\n\n\n## Weaknesses\n1. The most significant limitation is the absence of comparisons with existing Diffusion/Flow Model RL methods such as Online DPO [1], SPO [2], and D3PO/DDPO [3,4]. Without these comparisons, it is difficult to assess the true advantages of GRPO over established approaches. Specifically, Online DPO could serve as a baseline by training SD3.5-M using good-bad pairs generated based on reward model scores; SPO results could provide aesthetic quality baselines; and D3PO/DDPO could serve as baselines for other reward models.\n2. While Table 1 shows excellent results for Counting, Position, and Color reward scores, the paper lacks evaluation on other critical aspects such as aesthetic quality and text-image alignment. It remains unclear whether improvements in specific metrics come at the cost of degradation in others. If not, the authors should explain why GRPO better mitigates this issue compared to existing Diffusion/Flow Model RL algorithms.\n3. The authors' claims about reward hacking mitigation in the abstract and introduction are misleading. Upon careful reading, the alleviation of reward hacking stems primarily from reward ensembling and KL constraints rather than the GRPO algorithm itself. This raises questions about the paper's core contributions.\n4. The evaluation is restricted to SD3.5-M, leaving the effectiveness on other Flow-based models (e.g., HunyuanVideo, Flux) unclear. Additionally, the exclusive focus on Flow-based models is questionable, as Diffusion-based models can also benefit from Online RL approaches.\n5. Despite Table 4 demonstrating some generalization in categories and quantities, this capability appears constrained. For example, Table 3 shows that Flow-GRPO performs worse than the original SD3.5-M on texture generation.\n6. The technical contributions are relatively incremental. ODE-to-SDE conversion has been explored in prior works [5,6], and the authors make no algorithmic innovations to GRPO itself. While demonstrating GRPO's applicability to Flow Models, the absence of Online RL baselines and failure to establish GRPO's superiority over other Online RL algorithms in image generation (noting that reward hacking mitigation is not GRPO's achievement) weakens the contribution.\n\n## Summary\nWhile the paper presents an interesting application of GRPO to Flow Matching models, the significant limitations in experimental evaluation, particularly the absence of critical baselines and overclaimed contributions regarding reward hacking, substantially weaken its impact. The work would benefit from comprehensive comparisons with existing methods and more honest characterization of its contributions.\n\n### **Reference**\n- [1] Diffusion Model Alignment Using Direct Preference Optimization, CVPR 2024\n- [2] Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization, CVPR 2025\n- [3] Direct Preference for Denoising Diffusion Policy Optimization, CVPR 2024\n- [4] Training Diffusion Models with Reinforcement Learning, ICLR 2024\n- [5] Semantic Image Inversion and Editing using Stochastic Rectified Differential Equations, ICLR 2025\n- [6] Stochastic Interpolants: A Unifying Framework for Flows and Diffusions, arXiv'23\n\nQUESTIONS:\n1. **Sampling strategy**: Why does the GRPO computation use different random noise to generate different final images for the same prompt, rather than using the same initial noise with different sampling paths? The authors should provide an additional experiment comparing different initial noise + ODE sampling paths versus the current approach.\n2. **Prompt ratio justification**: In Compositional Image Generation, the authors set the prompt ratio as Position : Counting : Attribute Binding : Colors : Two Objects : Single Object = 7 : 5 : 3 : 1 : 1 : 0, but provide no justification for this specific setting nor ablation studies analyzing its impact.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed Flow-GPRO, which adopts the original GPRO technique for pre-trained flow-based models. Specifically, the authors transformed the flow ODE into the corresponding SDE and then applied the standard GPRO technique. Experimental results on text-to-image alignment demonstrated the superior performance compared to the original model.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n- The idea behind Flow-GRPO was clear and well-explained in the paper. The delivery and organization of the paper were concise.\n- The experimental results are significantly better than the original pre-trained model, demonstrating the effectiveness of the proposed approach.\n\n## Weakness\n- The paper claimed to be the \"first method\" to incorporate online RL into flow-based models, which I find to be **an overclaim**. For example, the Stable Diffusion 3 paper itself already applied Diffusion DPO [1] to finetune its flow-based text-to-image model. Meanwhile, the online RL methods explicitly designed for flow matching models also exist, e.g., ORW [2] that utilizes the reward-weighted loss. Therefore, the authors should avoid overclaiming. Furthermore, there are abundant approaches in RLHF that are model-agnostic and can be directly applied to align flow matching models in an online setup, e.g., RAFT [3] and DRaFT [4].\n- **The contributions seem limited to me**. The proposed method is a simple combination of GPRO and the well-known connection between flow and diffusion models (e.g., [5,6]). The authors also admitted that they directly followed the previous formulation of the conversion between ODE and SDE.\n- **RL baselines were very limited**. The authors provided comparisons with existing text-to-image models without such alignment, which are less meaningful in demonstrating the effectiveness of the proposed online RL algorithm. In this way, only the last two rows in Table 1 provided meaningful comparison, which were **insufficient in demonstrating the superior performance over other RL approaches**. Only a **limited comparison with the RL baselines** was provided in the appendix on a single metric. As this paper was not for the benchmark track, such an organization might be misleading for the RL community.\n\n[1] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Fan, Jiajun, et al. \"Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization.\" arXiv preprint arXiv:2502.06061 (2025).\n\n[3] Dong, Hanze, et al. \"Raft: Reward ranked finetuning for generative foundation model alignment.\" arXiv preprint arXiv:2304.06767 (2023).\n\n[4] Clark, Kevin, et al. \"Directly fine-tuning diffusion models on differentiable rewards.\" arXiv preprint arXiv:2309.17400 (2023).\n\n[5] Albergo, Michael S., Nicholas M. Boffi, and Eric Vanden-Eijnden. \"Stochastic interpolants: A unifying framework for flows and diffusions.\" arXiv preprint arXiv:2303.08797 (2023).\n\n[6] Domingo-Enrich, Carles, et al. \"Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control.\" arXiv preprint arXiv:2409.08861 (2024).\n\nQUESTIONS:\nSee the weaknesses listed above. In addition:\n- The CLIP score is the standard evaluation metric for text-to-image alignment. Can you provide additional results for the CLIP score?\n- Can you compare the proposed method with more recent online RL approaches like ORW mentioned in the Weakness section?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents Flow-GRPO, a method that enables online reinforcement learning for flow matching generative models by introducing an ODE-to-SDE conversion and a denoising reduction strategy. The approach achieves strong improvements in compositional image generation, text rendering, and human preference alignment on several benchmarks, while preserving image quality and diversity.

STRENGTHS AND WEAKNESSES:
**Strength:**

1. This paper proposes to apply online RL to flow matching models by the ODE-to-SDE conversion to introduce essential stochasticity for RL.

2. Denoising reduction greatly speeds up training without sacrificing output quality and show substantial gains across standard T2I benchmarks, with thorough ablations on reward hacking and generalization.


**Weakness:**

1. The conversion from Flow ODE to SDE is not a novel contribution, as stochastic sampling for flow-based models has already been explored in prior works such as [1].

2. The paper’s main contributions are: (a) transferring Flow ODE to SDE, which is well-studied in prior literature, and (b) applying GRPO to flow models. Since GRPO has already been extensively explored in the context of LLMs, this work appears to be largely an application of existing techniques to diffusion models, which limits its novelty. How does the exploration-exploitation tradeoff in the Flow-GRPO compare with diffusion models or standard RL settings?

[1] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers

Other questions:

1. The alation on the group size $G$ on GRPO.

QUESTIONS:
See weakness

LIMITATIONS:
See weakness
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
This paper proposes a method to fine-tune a pre-trained Flow-Matching (FM) sampler via online reinforcement learning (RL). By leveraging the SDE formulation of FM, the authors successfully reduce the number of FM sampling steps required at inference time.

STRENGTHS AND WEAKNESSES:
**Strengths**  
- **Quality**: The utility of the method is supported by comprehensive, multi-faceted experiments.  
- **Clarity**: The core ideas are presented in a clear and understandable manner.  
- **Significance**: Fine-tuning generative samplers offline via RL is novel, and the fact that training steps can be drastically reduced without loss of performance is particularly interesting.

**Weaknesses**  
- **Originality**: For readers already familiar with FM theory, the idea may appear straightforward.

QUESTIONS:
- **Q1**: Why choose flow matching as the base model? How would the approach differ if a diffusion model were used?  
- **Q2**: As I am not specialized in this area, could you explain the design principles behind the reward function $f$ defined on line 119?  
- **Q3**: Again, from a non-expert perspective, what are the main factors limiting the scalability of the proposed method?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This work proposes Flow-GRPO, the first method to perform online RL in flow matching models. \n- There are two problems to applying online RL methods like GRPO directly to flow matching models. 1) Flow matching models rely on a deterministic process based on ODE. As a result, the sampling during inference is not stochastic while online RL methods rely on exploration i.e. stochasticity during inference and 2) Online RL methods rely on efficient sampling to collect training data. Flow models inherently require multiple denoising steps during inference for 1 generation. This affects the efficiency of online RL methods.\n- Flow GRPO solves both these problems by 1) converting the **ODE to an SDE** thereby making the inference stochastic and 2) applying **Denoising reduction** strategy. The denoising reduction strategy uses fewer de-noising steps during training but uses the full schedule during inference.\n- Authors show that unlike other online RL methods, Flow-GRPO shows minimal reward hacking and gets almost perfect score on GenEval outperforming GPT-4o. On visual text-rendering, SD3-M's accuracy increases from 59% to 92%.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This paper proposes a solution to a very hard and important problem, applying online RL to diffusion/flow matching methods and shows that, similar to text generation, online RL improves media generation significantly as well. This work would be very useful to the community.\n- The paper is well written and has enough details for proper reproduction. \n- Qualitative and quantitative results are impressive.\n\n**Weaknesses**\n- I see tremendous value in the current work for media generation community and I don't see any potential weaknesses of this work.\n\nQUESTIONS:\n- Effect of ODE-to-SDE conversion. How much does this affect metrics like FID/SSIM/ human evaluations? \n- Why does using fewer steps improve reward? In Fig. 5a, Step=10 outperforms Step=40 in GenEval score as well. I understand that speed is definitely improved with 4x reduction in steps but the reward also doesn't go above 0.9 with Steps=40? Is it because of the number of updates? Is the graph adjusted for the number of updates under each setting?\n- How strong is the generalization? In Table-4, authors show that the model generalizes well for objects > 4. When does it break? \n- All the Text rendering examples show text with few words. Will the model be robust to long visual text as well? When does the model break?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}