ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
Authors: Tonghe Zhang
Chao Yu
Sichang Su
Yu Wang
Venue: NeurIPS 2025 poster ()
Keywords: Online Reinforcement Learning
Flow Matching Policy
Fine-tuning
Robot Learning
Primary area: reinforcement_learning
TL;DR: We propose an online reinforcement learning technique to fine-tune a family of flow matching policies for robot learning.

Abstract: We propose ReinFlow, a simple yet effective online reinforcement learning (RL)
framework that fine-tunes a family of flow matching policies for continuous robotic
control. Derived from rigorous RL theory, ReinFlow injects learnable noise into a
flow policy’s deterministic path, converting the flow into a discrete-time Markov
Process for exact and straightforward likelihood computation. This conversion
facilitates exploration and ensures training stability, enabling ReinFlow to fine-tune
diverse flow model variants stably, including Rectified Flow [34] and Shortcut
Models [18], particularly at very few or even one denoising step. We benchmark
ReinFlow in representative locomotion and manipulation tasks, including long-
horizon planning with visual input and sparse reward. The episode reward of
Rectified Flow policies obtained an average net growth of 135.36% after fine-tuning
in challenging legged locomotion tasks while saving denoising steps and 82.63% of
wall time compared to state-of-the-art diffusion RL fine-tuning method DPPO [42].
The success rate of the Shortcut Model policies in state and visual manipulation
tasks achieved an average net increase of 40.34% after fine-tuning with ReinFlow
at four or even one denoising step, whose performance is comparable to fine-tuned
DDIM policies while saving computation time for an average of 23.20% . Code,
model, and checkpoints available on the project website: https://reinflow.github.io/

ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
Tonghe Zhang Robotics Institute Camegie Mellon University tonghezGandrew. cmu edu
Chao Yu*
Shenzhen International Graduate School Tsinghua University
zoeyuchao@gmail.c coo
Sichang Su Department of Aerospace Engineering The University of Texas Austin Schanggss@uuexee edu
Yu Wang
Department Electronic Engineering
Tsinghua University
yu-wangOmail. tsinggua edu. cn
Abstract
We propose ReinFlow, a simple yet effective online reinforcement learning (RL) framework that fine-tunes family of flow matching policies for continuous robotic control. Derived from rigorous RL theory, ReinFlow injects learnable noise into a flow policy's i deterministic path, converting the flow into discrete-time Markov Process for exact and straightforward likelihood computation. This conversion facilitates exploration and ensures training stability, enabling ReinFlow fine-tune diverse flow model variants stably, including Rectified Flow [34] and Shortcut Models [18. particularly very few even denoising step. We benchmark ReinFlow in representative locomotion and manipulation tasks, including long- horizon planning with visual input and sparse reward. The episode reward of Rectified Flow policies obtained an average net growth of 135.36% after fine-tuning in challenging legged locomotion tasks while saving denoising steps and 82.63% of wall time compared to state-of of-the-art diffusion RL fine-tuning method DPPO [42]. The success rate of the Shorteut Model policies in state and visual manipulation tasks achieved an average net increase of 40.34% after fine- fine-tuning with ReinFlow at four or even one denoising step, whose performance comparable to fine-tuned DDIM policies while saving computation time for an average of 23. 23.20%. Code, model, and checkpoints available on the project website: htttps:/reinftoow.ghee gttthi///ooo.ttuuiooooo
1 Introduction
Recent years have seen rapid progress in the training of robotic models via imitation learning. Flow matching models, which combine the trifecta of precise modeling, fast inference, and minimal implementation, have emerged as robust alternative to diffusion policies and a popular choice for robot action generation X 57].
However, due to the scarcity of robot data and the embodiment gap, flow policies and many other imi- tation learning policies often trained from datasets with mixed quality [13. incurring suboptimal success rates even after supervised fine-tuning [6]. Although collecting more data may alleviate this problem, recent work (31 suggests that scaling the quantity of data may be a panacea because, in a given environment, the success rate quickly plateaus when we only increase the number of demonstrations. Worse still, as imitation learning method, flow policies lack built-in exploration
"ooresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

mechanism, implying that robots trained on imperfect data could struggle to complete challenging tasks where the agent needs to surpass expert demonstrations [5]
Online reinforcement learning (RL) offers a promising solution to these challenges. By learning through trial and error, RL promises overcome limitations associated with imperfect expert data and even achieve superhuman performance. Although recent studies have shown that it is possible to fine-tune diffusion policies via RL 30 42). training flow models with online RL remains technically challenging.
First, the theory has established that the refinement of stochastic policy is vital for continuous control problems [49]. However, for conditional flows, whose sample path is governed by a neural ordinary differential equation (ODE), even the log probability-a key factor that measures this stochasticity- could be challenging to obtain and unstable to backpropagate through. This problem is exacerbated when we infer flow policies very few denoising steps, where the discretization error becomes large at a low inference cost. Second, compared to offline RL methods, online RL fine-tuning requires the policy balance exploration and exploitation, especially in sparse reward settings [20]. However, how to design a principled exploration mechanism remains elusive for conditional flows with a deterministic path.
This paper addresses these challenges head-on and proposes the first online RL approach to fine-tune pre-trained flow matching policy. The contributions of this work are summarized as follows:
Algorithm Design. We propose ReinFlow, the first online RL algorithm to stably fine-tune a family of flow matching policies, especially in very few or even in one denoising step. We train a noise injection network to convert flows to a discrete-time Markov process with Gaussian transition probabilities for exact and tractable likelihood computation. Our design allows the noise net to automatically balance exploration with exploitation; enjoys lightweight implementation, built-in exploration, and broad applicability to various vari- ants of flow policy, including those parameterized with Rectified Flow [34] and Shortcut Models [18].
Empirical Validation. We perform extensive experiments in representative robot locomotion and manipulation tasks, with the agent receiving state or pixel observations and possibly accepting sparse rewards. Without reward shaping or scaling off-line data, our method, on average, improves the success rate the pre- pre-trained manipulation policy by 40.34% and increases the reward of locomotion policies by 135.36%. We achieve this improvement with wall time reduction of 62.82% for all tasks compared to the oa-te-of-the-aie art diffusion RL method DPPO [42].
Scientific Understanding. We carry out systematic sensitivity analysis design choices and key factors affecting the performance of our method, ReinFlow, including the scale of pretrained data, the number of denoising steps, the conditioning of the noise network, the level of noise and the type and intensity of different regularizations.
2 Related Work
In this section, we provide an outline of the relevant work. We defer detailed introduction for several key basclines to the Appendix
Online RL for improving diffusion- policies. Training diffusion- based policies from demon- strations has recently achieved impressive results in a variety of robot learning tasks [3] 39 43 50
53 156. However, their performance remains highly dependent the quality of the demonstration data. In practice, demonstrations often contain mixed or suboptimal trajectories, which limits policy effectiveness and motivates the need for online fine-tuning.
Several offline diffusion- -ased RL methods can be extended online settings. For example, Diffusion Q-Learning (DQL) [54] and Implicit Diffusion Q-Learnigg (IDQL) [2]] treat diffusion models as stochastic action policies and apply Q-learning updates. DIPO (55) adopts similar idea, but employs a critic to directly refine the actions sampled using action gradients. Although these approaches leverage Q-function approximations to guide the diffusion actor, inaccurate Q estimates can introduce bias and destabilize training.

---

Alternatiell, methods such as Q-Score Matching (QSM) AT and Diffusion Policy Policy Optimiza- tion (DPPO) 42 fine-tune pre-trained diffusion policies using policy gradient techniques. Despite the growing interest in online RI for diffusion-based policies, by the time we release this work, few analogous methods exist for flow-based policies. The underlying mathematical distinctions between diffusion and flow models prevent direct adaptation of existing techniques [3 which poses additional challenges to develop effective online RL algorithms for flow policies.
Reinforcement Learning for Flow Matching Models. Flow matching models are more efficient
than diffusion models, offering faster training, sampling, and better generalization [22 58). Following
their success in imitation learning 8 57 and vision tasks [32 16 27 52 recent work has begun
integrating them into reinforcement learning (RL).
but lacks exploration during online adaptation, leading to suboptimal convergence. ReinFlow addresses this by injecting bounded, learnable noise into flow trajectories to promote exploration.
Flow Q-Learning (FQL) (38 trains flow policies offline and distills them for
While Flow-GRPO 333 and ORW-CFM-W- 17 also study online RL for flow models, they focus on vision tasks rather than continuous robotic control. ReinFlow instead provides general policy gradient framework with compact noise network and exact likelihood computation, unlike Flow- GRPO's fixed noise or ORW-CFM-W- C2''ss reward weeghd-weighted regression approach.
3 Problem Formulation
Robot Learning as a Decision Process We formulate rohot learning as an infinite-horizon Partially Observable Markov Decision Process (POMDP) in continuous state space S E RRds action space A E 1 observation space O E TRdo, with a reward discount factor 7y E (0, 1). The robot plays in an environment with an unknown transition kernel o-ls and an emission kernel O,(sss). The interactions start with an initial state Sn drawn from distribution E (S)). In step h E Zzo. agent 5h+1 Th, (-1sh)
observes Oh Oy (-Isa), takes an action ag. before the state s; transitions with reward The For simplicity, we study reactive policies, which map the latest observation to an action distribution. The agent aims to maximize the discounted accumulated reward J(T)
E"
The Q function value function and the advantage function defined as Σ En=o ("ra(on,
on)
+00
(%, an)
E=
Σ Lr
oh: on (o,) = E" ("I A7, : Q Vi ,r
We drop the subscript h for Vr Qh, and AA when the policy and POMDP are stationary.
32 transforms random variables from one distribution process is associated with an ODE: d
Flow Matching Models Flow matching
Po to another P1 with flow mappings vb [0. 1 x X + X, where X, == t/a (Xo). t E 1). This
de (Xoo = vft, (Xo)) where Xo ~ Po- Rectified flow
(ReFlow) 334 is simple flow model with straight ODE path X, iX, + (1 Xo. The velocity d
field for Rectified Flow satisfies vt,,X,) di dx, = X1 Xo. which implies that the training objective for Rectified Flow given by
i arg min Xxxx unmifjj.. [|xx, Xo no (1, x,)I
Practitioners also sample from the beta distribution (7] or the logit normal distribution 16. T8 proposes "Shortcut Models" to further improve the generation quality of Rectified Flow at very few denoising steps by enforcing the velocity generated by two steps align with that generated by single step. During inference, we numerically solve the transport equation by integrating the learned velocity field: X1 = Xo + Σk -1
k=o (t,t,, Xx, At, where K is the number of denoising steps, 0= lo < 4 .... < k = [ are the discretized time steps, At, = 4+1 - 4g is the step size.
When we instantiate a flow-matching model in the action-generann generation setting, we obtain a matching policy for robot learning. We denote by of the denoised action at time t
Flow Matching Policy
For simplicity, we only consider 1- I-ReFlow and use I-ReFlow and ReFlow in this work interchangeably.
3

---

generated during the h-th step of the episode. A' will be the action space A. will be the standard normal distribution, and X, corresponds the robot's denoised actions. The velocity field up also conditions the observations.
Flow policies offer significantly faster inference than autoregressive and diffusion models 57 A fast policy is valuable for robot learning, as a high inference frequency enhances the robot's dexterity, and faster rollouts also reduce the wall-clock time of RL fine-tuning. Reducing denoising steps is arguably the most straightforward method to further accelerate flow policies, but when numerically solving the ODE, fewer integration steps increase the discretization error and reduce the quality of action generation [45]. This work seeks to use RL to improve flow policies with minimal denoising steps, ideally single step, achieve the best of both worlds: rapid inference and high success rates.
4 Algorithm Design
In this section, we detail the design of our algorithm "ReinFlow". We begin Section 4.1 by introducing methodology that precisely computes the logarithmic probability of the action in simple closed form. This approach leverages the injection of Markov noise and climinates the discretization error, making it applicable even with minimal denoising steps. Next, in Section 4.2 derive a general policy gradient loss for arbitrary policies parameterized by a discrete-ting Markov process, and then specialize this result in noise- injected flow policy. Following this, Sections 4.3 and detail the design choices for the noise injection network and the regularization techniques employed to enhance stability and promote exploration.
Algorithm 1 ReinFlow
1: Input pre-trained flow matching policy's velocity field ug: denoising step number K, discount factor 7, batch size B, discretization scheme to < t ... tk 1} with At,
f - l regularization function R with intensity coefficient a E IR
2: Initialize noise injection network 6'.
3: while not converged do
4: Restore last iteration's parameters: Oold f- stop stop_grad([0, 0'))
5: Reset environment and receive initial observation
6: while not done do
Rollout policy m.
7:
Sample a (0, Ida) for denoising step in (0.1....,, K 1} do k+ a vo a A) or (tt, at o)c, e~ (0, Ia,)
8:
Inject noise and integrate.
9:
10:
end for
11:
Record denoised actions a,, at....,a k in a buffer.
12:
Play action = a receive reward r and done flag d, update observation Store (a,o, r, d) to buffer, where a = (aa at .ak
14:
end while
15:
Sample a batch of data {a,, os, Ti, d, - from buffer. Compute the policy's 's transition probability for each denoising step & by Eq. 7
> Optimize policy.
16:
17:
In n (a/++1 o,) In (a (dt+110 |aa + vo (ta, , o:) Ats. og (tk, af
18:
Compute the regularization function R evaluated on each tuple, denoted as R(ai, o;; Oaia).
19:
Call policy gradient sub-routine. such Alg. 2 to jointly optimize O and 6' by Eq. (
0,0' arggmnn 1
Σ
A Doid (o;, a4)
Σ k=0
In (a (a/++ laf. o;) + R(a,, 0, 0, Bosa)
0.0"
B
(3)
where 0: 10
20:
end while
21: Return fine-tuned flow matching policy's 's velocity field do.

---

Algorithm 2 ReinFlow Subroutine for Policy Optimization (PPO implementation)
1:
Input clipping range E (0, 1), policy parameters the current iteration 0 == 00. 0' and the regularization
last iteration fold. data {ai, 0z, d; r function values (R(a;, Oz; 0 foid) )=) with intensity a IR.
2: Compute the advantage estimates A, == Å(o, ( by methods such as GAE [46]
3:
Jointly optimize the velocity net and noise net 0' by taking gradient step on the clipped surrogate loss:
B
(T( As clip Ta (ailo)))) 1 1+e) A. + a. R(a; oz; 0, Bota) Baaa (a)los))
min
B
(a: oi:)
t=1
4: Return updated parameters 0, 0'
4.1 Likelihood Computation Short Denoising Trajectory
The log probability a policy, which quantifies action stochasticity, essential for policy gradient methods in continuous control 22 47
However, computing the log probability for flow-matching policies with minimal denoising steps is challenging.
Although exact log probability expression exists
[|] for continuous-time flow models,
(v)((() = In (npo(((()))
v(l, x x~~ po()
in practice, we need to simulate this integral with numerical solvers and estimate the divergence [24
K
Impi(z) = (o))
Σ tr [Z Ox At.. k=0
(5)
where Z is zero-mean random vector with an identity covariance matrix. However, the trace estimator involves Monte-Carlo error; Simulating the integral introduces discretization error, which becomes more pronounced with larger step sizes (fewer denoising steps) 445 critical limitation for fast inference in robotic action generation. Treating the flow process at inference as discrete-ting time Markov process can mitigate this issue. However, the intermediate variables follow a deterministic transition p(X++A = x x,) = 5 (x X₂ vy (t, At), which renders the computation of the
probability impossible.
Our approach differs from previous methods by injecting learnable noise directly into the flow model's 's trajectory, thereby transforming the flow into discrete-time Markov process with closed- form transitions. During generation, actions are sampled from a normal distribution whose mean is given by the velocity field and whose standard deviation is parameterized by noise injection network 8', For robotic policies, this yields:
a ~'0 Ia,), a+1 ~ ~N( |a vo(tz, At,. (t,, a o)))
(6)
We condition the noise on the current denoised action and time to preserve the Markov property the flow process. Consequently, the joint log probability of the denoising process admits the following expansion:
K
In Taa x 10: 11n:=-nWW0000 In. +
Σ In N (a + no (te, (te,a Ats. o (ta, at o))
where t; = * and At 1 under uniform discretization.
We treat the flow model at inference as discrete-time process, and we have complete knowledge of the noise statistics. Unlike the empirical estimate in Eq. (5). the joint probability expression in Eq. (7) exact, even for arbitrarily large step sizes (or equivalently, minimal denoising steps). This precision guarantees stable fine-tuning of policies, particularly for very few or even one-step inference. By avoiding trace estimation, our method eliminates errors and computational overhead associated with Monte-Carlo estimation.
5

---

4.2 Policy Gradient of Discrete-time Markov Process
Although Eq. 7 provides the joint probability for the discrete-time Markov process that generates the action through denoising, it does not directly describe the marginal probability of the final action that the robot ultimately executes in the environment. Fortunately, we can establish a policy gradient theorem for policies parameterized by discrete-time Markov process. This enables us to leverage Eq. 7 effectively to optimize the noise-injected flows.
Theorem 4.1 (Policy Gradient Theorem for Markov Process Policy), For a POMDP with reactive 0 policy x² described by discrete-tine Markov Process Oh + 0% - dj _... of ah the policy gradient is
toc
VoJ(xT)= E=
A (oh, an) Vn Voln (a op................ Σe
an (on)
(8)
Further assuming that the policy and POMDP are stationary, the policy gradient takes the form of
K-1 In (a ,0)
1
VoJ(x=))
E 1 1-7 onddy Bo K
Q (o.a)
(9a)
E uond?" (.) Boo "...a
(
In x (a
(9b)
k=
where d is the discounted observation visitation frequency, defined as
+oc
(0) (1 -1) -p(.)
Σ xh p(o: o|s1;m)
(10)
Theorem 4. builds Theorem B.I in 40) and established results in policy gradient theory provide the proofs in Appendix A
2
We
enable the application of various modern deep policy gradient algorithms to optimize a policy parameterized by a discrete- time Markov process, including the noise- crise-injected flows introduced in Section 4.1 which are of particular interest. Specifically, we can optimize the policy using either Eq. (9a) or Eq. (9b). In this work, we implement Eq. (9h) with the clipped surrogate loss 47 due to stability.
Using Eq. 7
By combining Eq. 7 with Eq. ( we derive our algorithm, "ReinFlow", outlined Alg. [ We also present one possible policy optimization subroutine in Alg. 2 In Alg. [ denote the denoised action sequence a " as a boldfaced a and represent the combined parameters of the velocity and noise networks 0 10,
We also note that, beyond the implementations in Alg. and Alg. we can optimize Eq. (96) by replacing Eq. (3) with:
0, 0' argmin 0,0
1 B
Σ
Q Bold (oz, a;)
Σ In Th (a l 0i) + a R(a,,o;;0) where 0: 10,0'
(11)
and employ off- off-policy methods, such SAC 22 to update the policy.
In practice, we adopt "action chunking" during policy execution (29). in which the policy outputs multiple actions in batch given observation. We explain how action chunking affects likelihood computation in Appendix A.2
4.3 Noise Injection Network
The noise has the same shape of the joint torques and is applied to the whole denoising process of action generation. With the noise, we characterize the policy's probability in closed form. The price we take is slight increase noise net parameters, which is only a fraction of the pre-trained policy, as indicated by Table 2

---

During fine-tuning the noise injection network 6' is trained with the velocity network 0 to add diversity to the sample path for better exploration. The loss function for 0' is the loss of the policy gradient (Fq. (9), the same as for the net velocity. After fine-tuning, we discard the noise net O' and recover the flow matching policy, which is still made up of deterministic maps. By optimizing the noise net, the agent automatically adjusts her exploration level over the course of training.
The noise network could condition on o, (o, t), even output fixed constants. We provide comparative study in Section 6 to analyze how the noise conditions affect ReinFlow's performance. When the noise injection network learnable, uses features from the pre-trained flow policy to save parameters and ensure consistency.
We determine the noise limit using set of key hyperparameters that vary according the mechanical structural constraints of each joint. We study the effect of the noise bounds Section [ Empirically, find that when the success rate converges 100%, the noise level naturally decays. We can also manually control the noise level by adjusting the limits, or even increase the standard deviation of the noise with regularization of the entropy promote exploration (see Section 4.4).
4.4 Regularization
ReinFlow also supports various ways to regularize the fine-tuned policy.
Wasserstein-2 (W2) Regularization. One approach is to adopt Wasserstein regularization 28, 38
which constrains the Wasserstein-2 distance from the fine-tuned policy to the pre-trained policy. Studies show that W2 regularization could improve training stability for generative models 4 22 In practice, we minimize tractable upper bound on this distance to enforce such constraint:
Rw, (0, Oata)
2 Ilaa Gnld 2
(-lo) ao (110
(12)
2 E, inf En |
v1l2] = E. [W3 (T (110), To(-))))]
AEA
In Eq. 122 A. indicates the set of distributions over A² whose marginals are Ta (-10) and T-(-)o).
To implement Eq. (12), we replace the expectations with the sample mean over a batch of data.
Inspired by (38), we integrate and Gold from the same starting noise dg (O, Ia,), ), to control the
stochasticity of the initial denoise action. We also note that when computing Rw,,, we do not inject noise when sampling from Te.
Entropy Regularization. Another technique is entropy regularization, which theory has shown accelerates convergence TO) and encourages exploration [22] for simple policy classes. For flow matching policy parameterized with discrete-time Markov process, we adopt the negative per-symbol entropy rate (or block entropy) and policy, we define the regularizer as
48 the entropy regularizer. For stationary POMDP
(n) == K +
[h(a" a' a'......'..........
a [o,0]]
1 K+1 ((V(0,
Σ hh (at vo(t,, oo) og (ta, at .o)))
Here, the last step is due to the definition of the noise-injected flow process in Eq. (6) h is the differential entropy operator [14), of which normal distributions possess simple closed-form expression specified Section 3 According to [48,, the per-symbol entropy measures the Shannon entropy of finite symbol sequence with long-range correlations. By minimizing Rn. we promote the agent to seek more diverse actions, enhancing exploration.
We carry out experiments in Section 5 to compare the effects of different regularizations the performance of ReinFlow. By default, we adopt entropy regularization in Algorithm | for statc-input tasks and do not adopt regularization for visual manipulation tasks.
5 Experiments
We run simulated robot learning experiments to test the effectiveness and flexibility of our method, ReinFlow. We aim to adopt ReinFlow to significantly enhance the success rate of pre-trained flow
7

---

matching policies trained on mediocre expert data. We also managed to fine-tune at very few even one denoising step for Rectified Flow (indicated by "ReinFlow-R"" and Shortcut Models (indicated by "ReinFlow-S" where fine-tuning the two types of pre-trained models shares the same training hyperparameters.
We adopt PPO-based implementation of our algorithm due to its stability, Although alternative implementations may offer higher sample efficiency, we prioritize wall-time efficiency in simulated environments over sample cost. Exploring more sample-efficient RL algorithms for ReinFlow, especially real-world scenarios where data collection expensive, is an interesting direction for future work.
We compare ReinFlow against various RL methods for fine-tuning diffusion and flow-based policies. particularly DPPO and FQL. DPPO is a strong online RI. algorithm for diffusion policies, developed under bilevel MDP formulation [42). FQL represents state-of-the-an offline RI. for flow-mathhing policies and applies to offline-to-online fine-tuning. Comparisons with other baselines are provided the Appendix E
5.1 Environment Setup and Data Curation
We compare the algorithms in locomotion and manipulation tasks. In locomotion, the agent receives state input and a dense reward, similar to sim2real RL for legged robots [44). We also consider manipulation tasks, where agents receive pixel and/or state inputs with sparse rewards.
OpenAT Gym 2 We fine-tuned flow matching policies with ReinFlow in "Hopper", "Walker2d", "Ant" and "Humanoid" where these tasks are listed in ascending difficulty, and the last two involve high-dimensional state inputs and are considered very challenging continuous control problems. Ex- pert data are medium- or medium-expert-lewe demonstrations collected from the D4RL dataset [19]
Franka Kitchen (21). In Franka Kitchen, Franka robot learns long- long-horizon multitask planning by completing four state-based manipulation tasks sequentially. We pre-train the models with human teleoperated data with complete, mixed, or partial demonstrations of the four tasks.
Robomimic [36]. We adopt Robomimic visual manipulation tasks: PickPlaceCan (Can), NutAssem- blySquare (Square), and Aroorrrraanspo (Transport). Data collected via human teleportation and processed following DPPO [42] containing fewer and lower-quality data than proficient human.
5.2 Experiments
ReinFlow demonstrates strong training stability with a significant increase the success rate or success rate in all tasks. In Gym and Franka Kitchen benchmarks, it achieves the best overall efficiency and performance improvement.
1201
$300
3000
2002
Spea
2800
2300
WallCCacc (hours)
Wall Cocc Time thours)
Wal Cock Time thourg)
Wat clcckiicmma thours)
(a) Hopper-v2
(b) Walker-v2
(c) Ant-v2
(d) Humanoid-v3
ReinFlow-S (ours)
ReinFlow-R (ours)
DPPO
FQL
Figure 1: Wall time efficiency in OpenAl Gym. Dashed lines indicate the behavior cloning level.
Across three Robomimic visual manipulation tasks, ReinFlow-S and ReinFlow-R improve the success rate of the pre-trained policy by an average of 45.77% ReinFlow achieves success rates comparable to DPPO, requiring significantly fewer fine-tuning steps and less wall-clock time. Notably, it uses fewer denoising steps: just one in can and square, and a four-step flow transport, compared the five-step DDIM used by DPPO.
EEceep for "Humanoid" task, where the data sampled from own pre-trained SAC agent.

---

1.0
1.0
Taak Completion 8200
0.8
0..8
0.8
Compan
0.4
0.4
0.2
0.2
0.2
Samples
l06
Samples
105
Samples
lc6
(a) Kitchen-complete
(b) Kitchen-mixed
(c) Kitchen--artial
ReinFlow-S (ours)
DPPO
FQL
Figure 2: Task completion rates of state-input manipulation tasks in Franka Kitchen
1.0
0.9
Rate
0.8
0.8
Sucoess
Barrilice tours
0.6
0.7
BFRO Geuao
Barchiceel tours)
D1ro
0.4
6490
Gn
0.2
Getoran
0.5
Samples
le6
Samples
107
Samples
107
(a) Can
(b) Square
(c) Transport
ReinFlow-S (ours)
ReinFlow-R R (ours)
DPPO
Gaussian
Figure Success rates in visual manipulation tasks in Robomimic.
The Design Choice and Key Factors Affecting ReinFlow
This section analyzes how the pre-trained model and denoising steps affect our algorithm. We also study the effects of the noise level, the type, and the intensity of regularization.
Scaling, We fine-tune flow matching policies trained on datasets with different numbers of episodes and test the performance of pre-trained and fine-tuned models at different denoising steps. Fig reveals that scaling inference steps and/or pre-training data quantity does not consistently improve the reward, which consistent with the findings in [31]. However, ReinFlow consistently improves the success rate reward over the pre-trained policies regardless of the pre-training scale.
Average Reward
0.8
+
Fine-tuned
0.8
Pre-trained
3002
Success Rate
0.6
2300
0.6
8.4
beta Isg bnommaa unilom
1502
bultii bull tre reemineed Spisedt
EC
0.2
Inference Stop
16
32
64
128
0.00
0.25
0.50
0.75
1.00
1.25
Protrained Episades
Samples
1e7
ReFlow Policy in Hopper-v2
(b) Shorteut Policy in Square
(c) ReFlow Policy in Square
Figure 4: RL offers an orthogonal scaling path beyond data or inference. The gain is invariant to denoising steps-at 4 steps in Hopper and in Square.
Flow Matching's Time Distribution. The effectiveness of ReinFlow is not affected by altering the time sampling distribution the pre-trained flow matching policy. However, the beta distribution slightly stronger when fine-tuning in one denoising step, as in the case in Fig. 4c Inputs. The inputs the noise injection network affect the performance of fine-tuned flow policies. As shown in Fig. 5 conditioning both on observations and time often yields higher
Noise
success rate, as this approach allows the noise network learn how to create more diverse actions by altering the noise intensity different denoising steps.

---

Average Episode Reward
eelleetionn Rate
Task Compieetion
ReinFlow-R:
ReinFlow ats.
Reinflow-R: els)
ReinFlow ats)
BC
0.4
1000
0.0
0.4
0.8
Samples
1e7
Samples
1e7
(a) Noise Input's Effect in Ant
(b) Noise Condition's 's Effect in Kitchen-partial
Figure 5: Conditioning on state and time yields higher success rate than only conditioning on states.
Noise Level and Exploration. The noise magnitude is the key factor that influences the performance of ReinFlow. Fig. shows small noise leads limited exploration, while moderate noise enables rapid improvement, up to three times higher rewards. Beyond this threshold, performance becomes less sensitive to noise. Reducing noise proves beneficial for visuomotor policics, precision-critical tasks, longer denoising chains, and weakly pret-trained models.
x10º
Average Episode Reward
3000
BC
std-0.150
std-0.120
2000
stdl00200
1000
BC
8
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Samples
1e7
Samples
187
(a) Noise Affects Exploration
(b) W2 (blue) and Entropy (red) Regularization
Figure 6: Effect noise and regularization Ant-v0 (left) and Humanoid-v2 (right)
Regularization and Exploration. ReinFlow supports various regularizations. Adding entropy regularization, especially in locomotion tasks, is generally more effective than constraining the policy with Wa regularizer. As shown in Fig. 6b] reducing the W2 coefficient 3 allows the policy to surpass the behavior cloning baseline and learn more robust actions, approaching the performance achieved with entropy regularization (a 0.03). This also helps explain why the offline RL method FQL., which enforces W2 constraints during training, underperforms compared to our online approach.
7 Conclusion, Limitations, and Future Work
This work introduces ReinFlow, the first online reinforcement learning (RL) framework that stably fine-tunes family of flow matching policies for continuous robotic control. In state-input locomotion and visual manipulation tasks, ReinFlow surpasses existing methods that fine-tune diffusion or flow models using RL, while reducing wall-clock time over 50% compared the state-of-the-and diffusion RI. algorithm. We conducted a sensitivity analysis to identify the key factors affecting ReinFlow's performance.
ReinFlow's s current implementation has several limitations. Although the on on-policy design saves wall- time with parallelism, future work should explore sample-efficient implementation and adapting ReinFlow to real-world RL. Another open direction is to reduce its sensitivity to noise magnitude, auto-tune remove these hyperparameters. Lastly, our current experiments use relatively small networks, and scaling ReinFlow to large flow- base laniuanl -ction (VLA) models remains an exciting challenge. We leave these to future work.
10

---

Acknowledgements This work was supported by National Natural Science Foundation of China (No.62406159, 62325405), Postdoctoral Fellowship Program of CPSF under Grant Number (G7C202408300, 2024M761676). China Postdoctoral Science Special Foundation 2024T170496, and Beijing Zhongguancun Academy Project C20250301. The authors are grateful to Shu' ang Yu for reviewing the earlier versions of the paper. We also thank Feng Gao, Cheng Yin, and Ningyuan Yang for many fruitful discussions and comments.
References
[1] A. Agarwal, N. Jiang, S. M. Kakade, and W. Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32, 2019.
[2] A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
[3] L. Ankile, A. Simeonov, I. Shenfeld, and P. Agrawal. Juicer: Data-efficient imitation learning for robotic assembly, 2024.
[4] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In D. Precup and Y. W. Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214-223. PMLR, 06-11 Aug 2017.
[5] S. Belkhale, Y. Cui, and D. Sadigh. Data quality in imitation learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages
Curran Associates, Inc., 2023.
[6] K. Black, N. Brown, D. Driess, Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. lchter, S. Jakubzzk,, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, I.. Shi, Tanner, Q Vuong, A. Walling, H. Wang, and U. Zhilinsky. Tnž vision-language- action flow model for general robot control, 2024.
[7] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubezak, T. Jones, I.. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q Vuong, A. Walling, H. Wang, and U, Zhilinsky, mo vision-language- action flow model for general robot control, 2024.
[8] M. Braun, N. Jaquier, I.. Rozo, and T. Asfour. Riemannian flow matching policy for robot motion learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5144-5151, 2024.
[9] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[10] S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Oper. Res., 70(4):2563-25 2578, July 2022.
[11] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianch and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[12] C. Chi, Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2024.
[13] O. X.-E. Collaboration, A. O'Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar,
A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie,
A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess- Limerick, Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang,
C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton,
11

---

D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp. G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang. G. Wang. H. Su, H.-S. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj. H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra,
J. Kim, J. Drake, J. Peters, J. Schneider, Hsu, J. Vakil, J. Bohg. J. Bingham, J. Wu, Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang. J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go. K. Gopalakrishnan, K. Goldberg, K. Byme, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh. K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto,
I.. Fei-Fei, I. Tan, I. J. Fan, L. Ott, L.. Lee, I. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart'in-Mart'nn in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine. S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, Yu, T. Ding, Davchev, Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang. X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang. Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao. Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embbdiment: Robotic learning datasets and RT-X models.
https://arxiv:: org/ abs/2310...... 08864 2023.
[14] T. M. Cover. Elements of information theory. John Wiley & Sons, 1999.
[15] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[16] Esser, S. Kulal, Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[17] Fan, S. Shen, C. Cheng, Y. Chen, C. Liang, and G. Liu. Online reward-weighted fine-tuning of flow matching with wasserstein regularization, 2025.
[18] K. Frans, D. Hafner, S. Levine, and P. Abbeel. One step diffusion via shortcut models. arXiv preprint 11Xx::2410.1255:1 2024.
[19] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4(rl): Datasets for deep data-driven reinforcement learning, 2021.
[20] Y. Guo, J. Gao, Z. Wu, C. Shi, and J. Chen. Reinforcement learning with demonstrations from mismatched task under sparse reward. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1146-1156. PMLR, 14-18 Dec 2023.
[21] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In L. P. Kaclbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 1025-1037. PMLR, 30 Oct-01 Nov 2020.
[22] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In J. Dy and A. Krause, editors, Proceedings
12

---

of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018.
[23] Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. 2023.
[24] M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulttie and Computation, 18(3): 1059- 1076, 1989.
[25] M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
[26] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.
[27] W. Kong, Q. Tian, Z. Zhang, R. Min. Z. Dai, J. Zhou, Xiong, X. Li, B. Wu, J. Zhang, K. Wu, Q. Lin, J. Yuan, Y. Long. A. Wang, A. Wang, C. Li, D. Huang, F. Yang, H. Tan, H. Wang, J. Song, J. Bai, J. Wu, J. Xue, J. Wang, K. Wang. M. Liu, P. Li, S. Li, W. Wang, W. Yu, X. Deng, Y. Li, Y. Chen, Y. Cui, Y. Peng. Z. Yu, 7. He, Z. Xu, 7. Zhou, 7. Xu, Y. Tao, Q. Lu, S. Liu, Wang, Y. Liu, J. Jiang, and Zhong. Hunyuanvideo: systematic framework for large video generative models, 2025.
D. Zhou, H. Wang, Y. Yang,
[28] A. Korotin, V. Egiazarian, A. Asadulaev, A. Safin, and E. Burnaev. Wasserstein-2 generative networks. In International Conference on Learning Representations, 2021.
[29] L. Lai, A. Z. Huang, and S. J. Gershman, Action chunking as conditional policy compression, Sep 2022.
[30] Li, R. Krohn, T. Chen, A. Ajay, Agrawal, and G. Chalvatzaki. Learning multimodal behav- iors from scratch with diffusion policy gradient. Advances in Neural Information Processing Systems, 37:38456-38479, 2024.
[31] Lin, Y. Hu, P. Sheng, C. Wen, J. You, and Y. Gao. Data scaling laws in imitation learning for robotic manipulation. In The Thirteenth International Conference on Learning Representations, 2025.
[32] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.
[33] J. Liu, G. Liu, J. Liang. Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang. Flow-grpo: Training flow matching models via online rl, 2025.
[34] X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300. 2022.
[35] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with restarts. CoRR, abs/1608.03999 2016.
[36] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany. C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1678-1690 PMLR, 08-11 Nov 2022.
[37] D. Misra. Mish: A self regularized non-monotonic activation function, 2020.
[38] S. Park, Q. Li, and S. Levine. Flow q-learning, 2025.
[39] Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, Georgeseu, S. V. Macua, Z. Tan, I. Momennejad, K. Hofmann, and S. Devlin. Imitating human behaviour with diffusion models, 2023.
[40] M. Psenka, A. Escontrela, P. Abbeel, and Ma. Learning a diffusion model policy from rewards via s-score matching, 2024.
13

---

[41] M. Psenka, A. Escontrela, P. Abbeel, and Y. Ma. Learning a diffusion model policy from rewards via q-score matching, 2025.
[42] A. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B, Burchfiel, H. Dai, and M. Simchowitz. Diffusion policy policy optimization, 2024.
[43] M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies, 2023.
[44] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[45] T. Sauer. Numerical analysis. Addison- Addison-Wesley Publishing Company, 2011.
[46] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018.
[47] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[48] T. Schürmann and P. Grassberger. Entropy estimation of symbol sequences. Chaos: An Interdisciplinary Journal of Nonlinear Science, 6(3):414-427, 1996.
[49] M. Simchowitz, D. Pfrommer, and A. Jadbabaie. The pitfalls of imitation learning when actions are continuous, 2025.
[50] A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration, 2023.
[51] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[52] T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang. J. Zeng,
J. Wang, J. Zhang, J. Zhou, J. Wang, Chen, K. Zhu, K. Zhao, K. Yan, L.. Huang, M. Feng, N. Zhang, P. Li, Wu, R. Chu, R. Feng, Zhang, S. Sun, T. Fang, T. Wang, T. Gui, Weng.
T. Shen, W. Lin, W. Wang, W. Wang, W. Zhou, W. Wang, W. Shen, W. Yu, X. Shi, X. Huang,
X. Xu, Y. Kou, Lv, Y. Li, Y. Liu, Wang, Y. Zhang, Y. Huang, Y. Li, Y. Wu, Liu, Y. Pan, Y. Zheng, Y. Hong, Y. Shi, Y. Feng. Z. Jiang, Z. Han, Z.-F. Wu, and Z. Liu. Wan: Open and advanced large-scale video generative models, 2025.
[53] L. Wang, J. Zhao, Y. Du, E. H. Adelson, and Tedrake. Poco: Policy composition from and for heterogeneous robot learning, 2024.
[54] Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies expressive policy class for offline reinforcement learning, 2023.
[55] L. Yang, Z. Huang, F. Lei, Y. Zhong, Y. Yang, C. Fang. S. Wen, B. Zhou, and Z. Lin. Policy representation via diffusion probability model for reinforcement learning, 2023.
[56] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations, 2024.
[57] F. Zhang and M. Gienger. Affordance-basse robot manipulation with flow matching, 2025.
[58] Q. Zheng, M. Le. N. Shaul, Y. Lipman, A. Grover, and R. T. Q. Chen. Guided flows for generative modeling and decision making, 2023.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: YES
Justification:
The claims are derived from experiments which is reproduceable.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:See the last section.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
See the proof in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification: in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification:
Will be released few weeks later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeubiiissioooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification:
In appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: Results all have shadings by defaul indicate 1-sigma range.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: See appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes].
Justification: We doing what NIPS asks us to do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: Provided in appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [No]
Justification: Open source data.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Will be disclosed in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: In appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No].
Justification: No such experiments.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No].
Justification: No such risk.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes].
Justification: In OpenReview webpage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.M policy (ttr::e///errrppccccooofeereee/r for what should or should not be described.
21

---

We provide schematic in Fig 7 to illustrate the procedure of fine-tuning flow matching policy with ReinFlow.
Noise Injection Net
0
U0
Sensors
Encoder
Visuomotor
Feature
Velocity Head
Action Generation
Figure 7: Fine-tuning a flow matching policy with online RL algorithm ReinFlow (Alg.
Through interactions with the environment, the robot collects visual and proprioceptive signals, from which a pre-trained policy extracts features and outputs the velocity field of the following action, Vg. A noise injection network dgr shares the extracted features with ty and outputs Gaussian noise that smoothens the flow's 's deterministic ODE path, converting flows process with Gaussian transition probabilities. The noise injection yields an exact and straightforward likelihood expression any denoising steps, which is friendly for policy gradient optimization. The noise injection net dø, surrounded by the dot lines, co-trained with Up but will discarded after fine-tuning. The size of ag only fraction of the pre-trained flow policy. We outline our findings on a webpage: https://reinflow. www.hb.tthoooooo In what follows, we provide the theoretical background of our algorithm, report the findings omitted by the main text, and elaborate on the implementation details required to reproduce our experimental results.
discrete-ting Markov
A Theoretical Support
A.1 Proof of Theorem 4.1
In this section, we prove Theorem 4.1 the policy gradient theorem for discrete-time Markov process
policies.
For notation simplicity, we only consider an infinite-horizon POMDP with a reactive policy. We obtain the result for the H. For the reader's 's convenience, we recall several fundamental definitions of RL. The objective
setting by imposing t4 0 for larger than the finite horizon
function for RL given by J(T) E" Σ h ( an) The value function, Q function, and
the advantage functions are defined as
+o0
VT (ox) ==
"r (ar,
(13)
(o, ax) ==
Σ r=hh
(
oh an
A,, ax) = (% ax) - (or)
We first show that we can express the policy gradient for POMDP in terms of the advantage function and the action log probability's gradient:
+00
VoJ(
ATA (or, (or
(14)
Remark A.1. In Eq. 14 we use 0 to indicate general policy parameters. 0 should be understood as 0 = [0, 0'), that is, the combination of the velocity and noise nets when we instantiate the policy as noise-injected flow matching policy.
22

---

Proof.
+00
I rn(on: an) aulla")
VoJ(TT) = Σn t
Oxxx
h=0
toc
Σ
Toxar ra(on; an) Vo p( (1017)
toc
louar ra(on: and plona / in p(po am
h=0
ra(on, un)
p(o p(o una/
V In mo
Oxda
h=0
+00
E
(on, an Σ Vo Voln
+oo
E
(on, on) Vo
Change summation order
toc
to0 Vo In
rh (on. an)
h=T
=E"
7 VV
(n - an) or
+00
7 10= (or, ar (on.a.)VlnnToaaaa
I/ Definition in Eq.
[3
t
V' To To(a (00)
/ da-mo(a-1o+)
Σ 7 -²= (or, a. (or
Vo
+oc
=E"
Σ r A3 (ot, (oo
I/ Definition in Eq.
13
where (i) is due the Markov property:
V Inplonn,aaalt")
=V (on (01(1))) vo(a)lo)) TA (82 (1) . On, =T(anlo))) (15)
(oolng((r)o)) + Vo (olnmo(a2)o)) .... + Vo ma
Next, we extend Theorem in [40] and study the case when the action is generated via Markov Process. The action probability expressed by
To(anlon) = -/ daj ... da K-1 To(ah, a....., af (ox)
Ax
1 Ax dafdah ... da -l To(aglon)
K-1
(16)
II
(a 8+1 (
on)
k=
23

---

where we write = a Bringing Eq. (66 to Eq.
14
we obtain
VaJ(T")
t00
=F"
Σ
A7 ar a Vo
L=
1:00
Σ
da, da (or: ar) xala=to=)
V VOTo((rloo7)
r=0
/ da A3 ar) / da .a. Vo To(a
K
a la on)) e
II
T=0
1=0
da ......de .. AF ar) [Vo expll at a [o+)]
AK+
da ... da ... AF (0,, at at o+) Vo oo
AR+1
+000
=
Σ
7" A (or, a,r,a,)ve ot.. 10,)
+000
K-1 Σ In In S
=F""
7 A (or a, V
la
r==
(17)
In what follows, we consider the case where the POMDP has stationary transition kernels, and the policy is reactivate and stationary. This setting makes A Q", V time independent, drop their subscripts h for brevity:
K-1
VeJ(n B.
TrA A ==
Σ
(o,a K V
Σ In (a (a .)) k=0
(18)
To proceed from the RHS of Eq. (18). we first show that for any function f(;)::0x A and reactive, stationary policy the following relation holds:
E
Σ
+
f(on, an)
1 - Evndz() [aant())) [)))
(19)
where we take the expectation of observation o with respect d (o), the discounted average visitation frequency to that observation given initial state distribution. Concretely speaking, (o), which we call the "observation visitation measure", defined by
d, (0) (1 7) Σn A / do po = 0(81;4)
+
(20a)
h=0
d (0) = = Espp())) (1)(
(20b)
The MDP version for this result can found in classical RI. theory textbooks, such Eq. (0.10) on page 27 of [ Below, we extend the MDP result to POMDPs.
24

---

Proof.
+00 1si p(81)
ford dogdan plon, (x))) as)
LHS
Σn
7th
OxA
4=0
1si p(81)
Σ
to0
7t
/ dondan ((oNlsi)TT(QOllo))[
OxA
A=U
/ dsi p(s1) (1-
Σ n" / don p(onlsi) / dan malanlon)f(n,,e m.))
ddon
1
h=0
t00 E Bernp()) -7)
1
Σ
Io do plon o/81 (o
//Re-labeling
h=0
E Ex Bonds (a f(o,a) l/Eq.
20a
1
Bond E Ban B.q. 20b
1 Bard, (:) () (aan //Stationary Policy
Instantiate the function ff in Eq. (19) as the product of the advantage function times and the gradient of the joint log probability, we arrive at the following result by plugging Eq. (19h into Eq. (18):
K-1
o0) o))
VoJ(==) = 1- Boddz" () Ba .... (10)
^= (o.
Σ In (a k=
(21)
where a stands for ak. We take the expectation observation concerning its visitation measure and the expectation of intermediate actions for the distribution induced by the policy's s Markov Process.
We remind the reader that Eq. 21 holds only for POMDPs with stationary and reactive policy.
A.2 Action Chunking
In practice, we implement a flow matching policy with action chunking, which slightly alters how we compute the policy's 's log probability. Our robots interact with the environment in manner similar to Diffusion Policy [12), where the agent receives one observation, outputs sequence of actions, executes each action, collects rewards per step, and accumulates them for optimization.
This interaction protocol implies that actions remain unaffected by later observations that may change during execution. Thus, actions within chunk are conditionally independent, given the initial observation. In our setup, we flatten the actions of the chunk into a single vector as the actor's 's output, ensuring that the actions in the chunk are independent given the network's inputs, which include the action chunk at the last denoising step, the observation, and the denoising time. Hence, the log probability of a chunk of size C is the sum of the log probabilities of its internal actions. Formally:
chunk fixed after the first observation and
In ny (d 14,, (l,,,l,,,,tt.t... dh 0n)
k+1
In k+1 & ah- on) //Conditional Independence
(22)
am0 0
C-1
Σ o=0
In
x+1 ahe
affee + [volame At loo
In Eq. 22 the terms vo and der are conditioned on (Is, Ch on). We use [u]; to denote the i-th element vector u, and Way to represent the sub-vector formed by concatenating the i-th the j-th coordinates of u.
25

---

B Extended Related Work
Diffusion Policy Policy Optimization 422
Diffusion Policy Policy Optimization (DPPO) trains DDIM policies with PPO. They design the algorithm on top of bi-level MDP formulation, which involves calculating the advantage function for the denoised actions.
To (5t, az) Th (8t, 0t)
Ro (5t, ai) (5,, ai)
Ca E min
A (5i, (t)
Am (5t;ūg) clip
E, e, +
(23)
The term f = E(t, k) represents the k-th denoising step in the t-th interaction. They define the advantage function these denoising steps by applying additional discount factor, "denoise the advantage function of the action taken.
DPPO does not provide theoretical guarantee for its design in Eq. 23 Computing the advantage function at denoised steps also slows down computation. Our method does not perform such calculations, and we derive our algorithm from rigorous reinforcement learning theory for POMDPs claborated in Appendix A
Empirically, Section 3 shows our method, ReinFlow, outperforms DPPO in almost all tasks in Gym and Franka Kitchen while reducing the denoising step count from 10 in DPPO to 4 in ReinFlow, significantly reducing wall time as shown in Table [ Shortcut Model policies trained in visual manipulation tasks achieved a comparable or slightly better success rate than DPPO, using as few as one step in Can and Square and four steps in Transport. DDIM policies in DPPO adopt five steps for these tasks.
Flow Q Learning (FQL) is an offline reinforcement learning algorithm designed for flow matching policies, which can also be used for offline-to-online fine-tuning.
Flow Q Learning
Unlike our approach, FQL does not directly fine-tune a flow model with multiple denoising steps. Instead, it first trains multi-step flow policy Te using the objective in Eq. (2). Then, it learns function Qo from an offline RL dataset using one-step temporal difference (TD) learning, as described in [51.
(()) = = [((o(s,,a) - ~Qgls' ms (6',,)) =))²
(24)
At the same time, FQL distills a one-step policy from the pre-trained flow via minimizing the following loss:
((w) = B [ (s, (w) 2)) + a (ew Mo(s, 2)1[
(25)
The one-step policy will be deployed after FQL fine-tuning.
In the official FQL implementation, gradients flow to the multi-step policy during the distillation of the one-step policy, which can interfere with the pre-trained loss. As result, discover that even when we align the sample consumption during the offline RL phase, the multi-step FQL policy may struggle to match the reward achieved by pure behavior cloning pre-training.
In contrast, ReinFlow's method directly and stably fine-tunes a multi-step flow matching policy, offering richer representation capabilities than one-step distilled models in FQL. Empirically, Re- inFlow outperforms FQL asymptotically and converges faster regarding wall-clock time, as shown in Figures [ and 2 Additionally, ReinFlow does not rely distillation. As a purely online RL algorithm, ReinFlow does not require labeled rewards for expert demonstrations.
Other Methods DPPO 42 and FQL 38 provided an excellent introduction to a line of algorithms that fine-tune diffusion models with online RL and flow models with offline RL, including IDQL 23 QSM 40 DAWR 422,, DIPO (5), DRWR 42.. DQL [44, IFQL 38 etc. DPPO and FQL are generally superior to these methods regarding asymptotic reward, training stability, and wall time.
Although we have shown the advantage of ReinFlow over DPPO and FQL in Section 5 for com- pleteness, we provide a brief comparison between the diffusion RL baselines with our method in few representative continuous control tasks with the same set of hyperparameters indicated in 42 Please refer to Appendix E for details.
26

---

C
Environment and Dataset Configuration
Tasks. Figures 8 and 9
demonstrate the locomotion and manipulation tasks considered in this work.
(a) Hopper
(b) Walker2d
(c) Ant
(d) Humanoid
Figure 8: Four OpenAl Gym locomotion Tasks: Hopper, Walker2d, Ant, and Humanoid.
(a) Franka Kitchen
(b) PickPlaceCan
(c) NutAssembly
(d) TwoArm TaAkAmrrrnsottt
Figure 9: Four manipulation tasks state-input Franka Kitchen and pixel-input Robomimic environ- ments.
Environment Configuration. We also list the environment configurations in different tasks in Table
Table I: Environment Configuration
State Dim
Action Chunk
Max. Eps.
Environment
Task or Dataset
Image Shape
Reward
Len.H
Hopper-v2
x 4
1000
Dense
OpenAl Gym
Walker2d-v2
17
4
1000
Dense
Ant-v0
111
x 4
1000
Dense
Humanoid-v3
376
17 x 4
1000
Dense
Kitchen Maahhen-oomlete--eee
60
4
280
Sparse
Franka Kitchen
-itheen-iixed-00
60
9 X 4
280
Sparse
60
4
280
Sparse
Can
[3,96,96] X 1
x 4
300
Sparse
Square
9
[3,96.96] X 1
X 4
400
Sparse
Robomimic
Transport
18
[3,96,9]] x 2
14 x
8
800
Sparse
In Tab. [ the "State Dim" indicates the dimension of proprioception inputs. "Action Chunk" expresses the dimension of a single action and the size of the action chunk. "Max.Eps.Lem H" indicates the maximum steps a robot can take in single rollout. When the reward is "sparse, we award the agent reward of +1 only upon task completion. Otherwise, the agent receives a 0 reward. The sparse reward realistic, straightforward, and directly associated with the success rate of manipulation tasks. A "dense" reward system assigns the reward based on the robot's dynamics and kinematic properties. It is usually floating number after each step of action execution. Dense reward systems need careful design. We adopt them primarily in sim2real training for legged locomotion tasks.
27

---

The Robomimic Dataset Provided by DPPO
42 We remark that, in Robomimic environments, 42 which makes the shapes of the state vector smaller
we adhere to the configuration of DPPO
than the official default implementation in 36 This processing simplifies robot learning and makes the pre-trained data provided by [42 smaller and simpler than Robomimic's official datasets. As acknowledged by the authors of DPPO, the datasets provided by 42) are of lower quality and/or quantity than the official release, so the pre-trained policy is trained on inferior data. Consequently, the pre-trained checkpoints of both DDPM and Shortcut policies have lower success rate than the best-reported results in the literature [8].
We can train flow matching or diffusion policies with RoboMimic's ' default configuration and datasets, which will help us align with state-of-the art results in the imitation learning literature. However, we may need to adopt larger neural networks and spend more time pre-training and fine-tuning.
Datasets for OpenAI Gym Tasks To faithfully replicate the results of DPPO 42 both DPPO and ReinFlow agents using the behavior cloning (BC) dataset provided by Ren al. (2024) 42). For consistency, we followed their dataset choices for all experiments in the Appendix that do not involve offline reinforcement learning (RI.) algorithms, such as FQI..
we trained
However, the DPPO dataset for OpenAI Gym tasks lacks the offfine rewards necessary to train an FQL agent, and the authors did not specify how they constructed their Gym dataset. We hypothesize that their BC datasets were derived and augmented from D4RL [1)] Therefore, in the experiments that involve FQL (described in Section of the main text), we adopted D4RL offline RL datasets to train DPPO, FQL. and ReinFlow agents. We also remark that while we chose the "Ant-v2" environment in these experiments, we switched to "Ant-vO" in the sensitivity analysis in Section 4.3 and the Appendix.
The minor differences in OpenAl Gym Tasks mentioned above resulted in minimal discrepancy in reward curves.
D Implementation Details
D.1 Model Architecture
For fair comparison, we try to make the model architecture of flow matching policies align with diffusion policies proposed DPPO [42 as much as possible.
State-input Tasks. state-imput tasks (OpenAI Gym and Franka Kitchen), the velocity nets os of I-ReFlow policies are parameterized with Multi-layer Perceptrons (MLP) that receive action chunk, state, and time features. The time input t E (0. 1) is encoded by sinusoidal positional embedding (25 and linear projections, with Mish (37) activation function in between. We implement Shortcut Models similarly, passing the time and inference step counts through the same sinusoidal positional embedding and concatenating. At the same time, we also encode the state input with small MLP and add the state feature with the time-step features. The sizes of Shortcut Models are often slightly smaller than I-ReFlow policies in the same task. Critic networks are also MLPs with the same half the width.
Pixel-input Tasks. In Robomimic, where the agent receives pixel and proprioception inputs, the actor and the critic adopt a single-layer Visual Transformer [15] with random shift augmentation as the visual encoder and compresses the proprioception information with a small MLP. We pass the features of the visuomotor condition, time embedding, and the raw action chunk through the actor's velocity head, which outputs a velocity estimate. The critic only receives features from time and condition.
Noise Injection Net The noise injection network 6' shares the input features with the actor and outputs the standard deviation of the noise at each coordinate of the actions in the action chunk. The output of network 0' is passed through Tanh function coupled with affine transform, to ensure bounded output and smooth gradients. The upper and lower bounds of the noise standard deviation Fmax, min are a group of essential hyperparameters of ReinFlow, and we have provided an analysis in Section to study how they influence exploration.
28

---

We discard dg after fine-tuning. Although noise-injected process exists during RL, ReinFlow still returns a policy with an ODE inference path.
During the evaluation, we also did not inject noise into the flow policy; We observed that the reward often higher than the noise-injected version, aligning with findings in classical RL literature [22
Parameter Scale We summarize the sizes of the actor, critic, and noise injection networks in Table 2 which shows that for state-input tasks, a negligible parameter increase (< 6%) brought by noise injection network returns promises a 135.36% net increase in reward and 31 11.29% net increase in success rate.
For complex visual manipulation tasks beyond robomimic, since the noise network O' shares the same representation backbone as the velocity network, the parameter increase should also remain minimal when scaling to larger models with more complex backbones, such as multi-layer Transformers, instead of the single-layer Transformer used in our experiments.
Designing an efficient noise injection network architecture that balances reward improvement and parameter count is an interesting problem, which we leave for future work.
Table 2: Model Parameter Counts
Pre-trained Actor 0/M
Noise Net /
Fine-tuned Actor 0/M
Critic /M
Total /M
Noise/
Task
Model
Velocity
Hopper-v2
I-ReFlow
0.55
0.01
0.56
0.13
0.69
1.22%
Walker2d-v2
I-ReFlow
0.57
0.01
0.58
0.14
0.71
1.39%
Ant-v3
I-ReFlow
0.62
0.01
0.64
0.16
0.80
2.31%
Humanoid-v3
1 R-ReFlow
0.80
0.03
0.83
0.23
1.06
4.23%
Kitchen
Shorteut
0.16
0.01
0.17
0.15
0.31
5.46%
Can
Shortcut
1.01
0.15
1.16
0.59
1.74
14.59%
Square
Shortcut
1.69
0.32
2.01
0.59
2.59
18.91%
Transport
Shortcut
1.87
0.35
2.22
0.66
2.87
18.84%
D.2 Enhancing Training Stability
Clipping Probability Ratio For stability reasons, we clip the log probability ratio in Algorithm) | with 6 0.01 for state-input tasks and 0.001 for pixel-input tasks. This choice follows
DPPO
42
Clipping Denoised Actions Although unnecessary for the pre-trained flow policies, we discover that during fine-tuning, it is beneficial to clip the denoised actions of a flow matching policy because this helps prevent the injected noise from interrupting the integration path too violently. After fine- tuning with ReinFlow, policies should also clip the denoised actions during inference; otherwise, performance may likely deteriorate.
Critic Warmup Training the critic network for several iterations before updating the actor is erucial for stable training and rapid convergence, particularly for larger models with visual inputs. We refer to this phase as "critic warm-up". Since we use RL as a fine-tuning approach, the critic should output reasonably large value (at least positive) before the policy gradient starts. An excessively small or even negative initial output misleads the actor into considering its current actions as excessively unsatisfying, resulting in rapidly degrading rewards and value function estimates during the fine-tuning process.
Empirically, recommend initializing the critic and/or adjusting the critic warm-up iteration number according to the initial reward of the pre-trained policy, the rollout step number, and the discount factor.
Critic Overfitting and Initialization An excessively long warm-up period may lead to overfitting of the critic network, especially in cases where the critic is significantly smaller than the pre-trained policy (Table [p. When the critic overfits, the policy gradient loss could oscillate violently during fine-tuning.
29

---

One possible solution to address critic overfitting is to incorporate regularizations into the critic's training procedure. Another method is limit the warm-up up iterations and initialize the critic's last fully connected layer with positive bias. The bias can be estimated by the pre-trained success rate. With positive initial output, the critic requires fewer warm-up iterations to output an appropriate value estimate without overfitting the pre-trained policy's ' distributions.
Additional Experimental Results
Random seeds Unless specified, we train the algorithms with three seeds all tasks except for Franka Kitchen with mixed partial data, where we adopt two extra seeds, in the main experiment section [ These tasks involve long-horizon multi-task planning, and we discover that DPPO and ReinFlow exhibit high variations across different runs.
The RL fine-tuning curves show the reward success rate averaged over these seeds, with shading representing the mean + standard deviation to indicate variability across runs.
During training and wall time testing, the MuJoCo graphics rendering backend (MUJOCO_a) ( is set to Embedded System Graphics Library (EGL. to accelerate the computation with GPU. If users do not have EGL support and they switch to software rendering (osmesa), or multiple threads are running together on the same group of CPU kernels or the same GPU, the compute time may be longer than that described in Table 3
Rendering Backend
Table Per Iteration Wall-clock Time
Task
Algorithm Single Iteration Time/second
Average
First seed
Second seed
Third seed
Mean + Std
ReinFlow-R
11.598
11.704
11.843
11.715 0.123
ReinFlow-S
12.051
12.127
12.372
12.290 + 0.141
Hopper-v2
DPPO
99.502
99.616
98.021
99.046 0.890
FQL
4.373
4.366
4.515
4.418 + 0.084
ReinFlow-R 11,861
11.446
11.382
11.563 + 0.260
ReinFlow-S
12.393
12.690
13.975
13.019 0.841
Walker2d-v2
DPPO
101.151
106.125
98.470
101.915 = 3.884
FQL
5.248
4.597
5.207
5.017 + 0.365
ReinFlow-R
17.210
17.685
17.524
17.473 0.242
ReinFlow-S
17.291
17.821
18.090
17.734 + 0.407
Ant-v0
DPPO
102.362
104.632
99.042
102.012 = 2.811
FQL
5.242
4.950
5.3086
5.167 + 0.191
ReinFlow-R
31.437
30.223
31,088
30,916 + 0.625
ReinFlow-S
30.499
30.058
31.029
30.529 + 0.486
Humanoid-v3
DPPO
109.884
105.455
113.358
109.566 3.961
FQL
5.245
4.981
5.522
5.249 + 0.271
ReinFlow-S
26.655
26.328
26.628
26.537 0.182
Franka Kitchen
DPPO
81.584
84.646
83.245
83.158 + 1.533
FQL
5.245
4.981
5.522
5.249 + 0.271
ReinFlow-S DPPO
219.943
216.529
217.711
218.061
1.734
Can (image)
310.974
307.811
308.014
308.933
1.771
ReinFlow-S DPPO
313.457
312.3
313.862
313.206
0.811
Square (image)
438.506
440.212
434.773
437.830 = 2.782
ReinFlow-S DPPO
554.196
557.712
559.006
558.359 - 0.915
Transport (image)
406.607
439.268
412.077
419.317
17.493
30

---

Wall-clock Time For fair comparison, we measure all algorithms' wall-clock time in all the tasks (except Transport) on a single NVIDIA RTX 3090 GPU with EGL rendering. We evaluate the wall time of Robomimic Transport on two NVIDIA A100 GPUs with EGL rendering, as this task occupies significantly more memory than a single RTX 3090 GPU. The measurement taken sequentially for three random seeds without interfering with other processes. We obtain the total wall-clock time for each algorithm by multiplying the average iteration time by the number of iterations. Table 3 provides detailed record of these measurements, where the results recorded in seconds and accurate to three decimal places.
Although FQL's per-iteration wall time is significantly shorter than other methods, this does not imply that it is more efficient overall: FQL's batch size is considerably smaller than that PPO-based methods, including ReinFlow and DPPO, and its total training iterations are significantly larger than those of the other two methods; neither FQL designed for parallel computing like DPPO and ReinFlow.
Performance Increase We report the performance improvement after fine-tuning flow matching policies with ReinFlow across various simulation tasks in Table
For locomotion tasks, we compute the reward increase ratio follows:
Locomotion Reward Net Increase Ratio == Fine-tuned Reward Fine-tuned Reward
(26)
Pre-trained Reward
For manipulation tasks, we compute the success rate increase:
Manipulation Success Rate Net Increase Fine-tuned Success Rate Pre-trained Success Rate
(27)
Policies fine-tuned with ReinFlow achieved an average episode reward net increase of 135.36% in OpenAl Gym locomotion tasks using D4RL, datasets, with an average success rate increase of 31.29% in Franka Kitchen, 77% in Robomimic, and 40.34% in all manipulation tasks. The results are accurate to two decimal places.
Sample Complexity in Gym Tasks Here, also compare the sample complexity in OpenAI Gym tasks omitted the main text. While FQL is more sample efficient simpler tasks such as Hopper and Walker2d, it generally struggles to tackle more complex locomotion task, where it is asymptotically inferior to DPPO and ReinFlow.
We compare the diffusion RL baselines with method in a few representative continuous control tasks. For these baselines, we adopt the same set of hyperparameters described in 42
Comparison with Other Diffusion RL Methods
Figs and 12 show that ReinFlow overall outperforms other methods concerning the stability w.n. random seeds and asymptotic performance.
Average Episode Reward
4000
3000
2000
1000
Samples
Beinfloeed (ours)
DIPO
123L BRWR
DAWIL
Reinfloer (ours)
DGL
Figure 11: Fine-tuning locomotion task Ant-vō with Diffusion RL baselines and ReinFlow.
31

---

Table 4: Performance Metrics for ReinFlow Across Tasks.
(a) Average Episode Reward in Locomotion Tasks.
Pre-trained
Fine-tuned
Reward Net
Task
Algorithm
Episode Reward
Episode Reward
Increase Ratio
ReinFlow-R
1431.80#27.55
3205.33+32.09
123.87%
Hopper-v2
ReinFlow-S
1528.34+14.91
3283.27427.48
114.83%
ReinFlow-R
2739.90174.57
4108.57451.77
49.95%
Walker2d-v2
ReinFlow-S
2739.19#134.30
4254.87+56.56
55.33%
ReinFlow-R
1230.5448.18
4009.18+44:
225.81%
Ant-v2
ReinFlow-S
2088.06479.34
4106.31+79.45
225.81 %
ReinFlow-R
1926.48:41.4B
5076. 1076.12+37.47
163.49%
Humanoid-v3
ReinFlow-S
2122.03=105.01
4748.55#70.71
123.77%
(b) Average Success Rate in Manipulation Tasks.
Pre-trained Success Rate
Fine-tuned Success Rate
Success Rate
Environment and Task
Algorithm
Net Increase
Kitchen-complete
ReinFlow-S
73.16+0.849%
96. 96.17+3.657
23.01 %
Kitchen-mixed
ReinFlow-S
48.37=0.78%
74.63+0.36%%
26.26%
Kitchen-partial
ReinFlow-S 40.00=0.228%
84.59+12.38%%
44.59%
ReinFlow-R
59.00=3,088 00%
39.67%
Can (image)
ReinFlow-S 77.83+1.25%
98.50+0.7199
40.67%
ReinFlow-R
25.00+1.4799
74.83+0.2449
49.83%
Square (image)
ReinFlow-S 34.50+ 1.22%
74.67+2.666
40.17%
Transport (image)
ReinFlow-S
30.17+2.46%
88.6744.400
58.50%
Average Egisane Roward
35.00
Average Episade Reward
5000
8000
25.00
6000
2000
3000
15:00
2000
1.000
Samples
Renflews (uui
DARO
IDO1. DOL
GSM DRWM
DOWR
Rendilow Jount
DAPO
DOL
DRWS
D0WS
herficw (oers))
IDCL
(a) Hopper-v2
(b) Walker2d-v2
Figure 12: Fine-tuning locomotion task Hopper-v2 and Walker2d-v with Diffusion RL baselines and ReinFlow.
Changing the Behavior Cloning Dataset's Scale Square Tab. 5 shows how the fine-tuned performance of ReinFlow is affected by the scale of the behavior cloning dataset in robomimic square. Fine-tuning the policy trained on 16 episodes failed due to overly low initial success rate. We adopt the same hyperparameter set when fine-tuning policies trained on 64 and 100 episodes, restricting the
32

---

K10
Episode Reward
Samples
(a) Hopper-v2
X10
Episode Reward
Samples
Walker2d-v2
x10
Episade Reward
2
1
2
Samples
(c) Ant-v2
x10a
Episode Reward
2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Samples
307
(d) Humanoid-v3
ReinFlow-S (ours)
ReinFlow-R (ours)
DPPO
FQL
Figure 10: Sample efficiency results of state-based locomotion tasks in OpenAI Gym. For better visualization, we down-sampled FQL's data by five times in the first three tasks and three times in "Humanoid-väs v3".. Although FQL is more sample-efficient than DPPO and ReinFlow in simpler tasks, it struggles to achieve high reward in more challenging locomotion tasks.
noise standard deviation std (0.08, 0.14] with entropy coefficient 0.01. We slightly tuned down the noise to std E [0.06, 0.10) and removed entropy regularization, as we find it more beneficial to limit exploration when the pre-trained policy performs poorly.
Table 5: Fine-tuned Success Rates in Square Across Different Pre-trained Episodes
Pre-trained Episodes
Pre-trained Success Rate
Fine-tuned Success Rate
Average Fine-tuned
Success Rate
Fist seed
Second seed
Third seed
Mean
Std
16
3.08 %
0.00 %
0.00 %
0.00 %
3.08 %
0.00
32
10.15%
22.20%
22.00%
18.50%
20.90%
1.70 %
64
27.67%
65.50%
62.20%
56.20%
61.30%
3.85 %
100
25.14%
79.50%
78.20%
74.50%
77.40%
2.12
Changing the Number of Fine-tuned Denoising Steps number K could affect ReinFlow's performance since the pre-trained policy has different rewards when evaluated at different steps.
Altering the fine-tuned denoising step
33

---

Fig. 13 shows the difference when fine-tuning a Shortcut Policy in Franka Kitchen at K = 1, 2, and 4 denoising steps.
1.0
Task Completion Rate
0.8
ReinFlow-S (4 steps)
0.4
ReinFlow-S S (2 steps)
ReinFlow-S (1 step)
0
4
6
8
Samples
1e6
Figure 13: Fine-tuning Shortcut Policy in Kitchen-complete-00 at Different Denoising Steps 1,2, and 4.
In Fig. 13, all the run hyperparameters except K and random seeds are the same. The pre-trained policy adopts complete demonstration data.
Increasing K improves the reward at the beginning of fine-tuning, A better starting point shrinks the space of exploration and thus accelerates convergence. However, longer denoising trajectory also consumes more simulation time. Fig. 13 and Fig. 4a also show that the success rate or episode reward quickly plateaus when scaling K.
In more challenging tasks such as visual manipulation, we also find that reducing the noise standard deviation is beneficial when increase the number of denoising steps K, especially when the pre-trained policy has a low success rate.
F
Discussion
In this section we provide additional discussion to help the readers better understand our design.
F1
Noise Injection versus Entropy Regularization
Both noise injection and entropy regularization serve to enhance exploration in reinforcement learning, but they play distinct and complementary roles in ReinFlow.
Purpose and mechanism. Noise injection is primarily used replace the log ı function, making the action distribution tractable. The added stochasticity naturally aids exploration byproduct. In contrast, entropy regularization provides an explicit mechanism coefficient a. The noise network enables ReinFlow to learn diverse actions by adapting noise intensity across denoising steps, while entropy regularization offers direct, tunable control over the opploratio-eepploiaaioon trade-off. Importantly, ReinFlow remains mathematically valid even without the entropy term
control exploration via its
Generalization beyond locomotion. Entropy regularization is effective across various types of tasks, only in locomotion domains. We successfully adopt in the Robomimic visual manipulation task "square" (Table 9b, Appendix). Furthermore, in the "Franka Kitchen Kitchen-complete" task state-input, sparse-reward manipulation environment- entropy regularization yields meaningful improvements in success rate. Ablation experiments averaged over seeds show an increase from 96.17 + 3.65% (without regularization of the entropy) 99.00 0.75% (with regularization of the entropy, a = 0.1).
F.2 Wall-time Efficiency
ReinFlow achieves faster wall-time performance for three key reasons:
(a) Fewer Denoising Steps: Flow models (e-g., ReFlows) require fewer steps than diffusion models (c.g., DDPM, DDIM) to reach high initial reward.
34

---

(b) Exact Probability Description in Few Steps: The probability of flow ODEs depends on the number of denoising steps. However, ReinFlow's expression remains exact even with very few steps, making it possible to take advantage of flow's property, enabling efficient fine-tuning without compromising reward.
(c) Simplified Optimization: We simplify DPPO's policy loss by not computing value/ advantage for denoised actions. This further reduces the computational cost.
F.3
Additional Ablation Studies
We conducted additional ablation studies across various tasks during the rebuttal period, confirming that the observed trends are consistent.
Noise conditioning across environments. We compared the effect of different noise conditioning strategies in the Humanoid-vž environment, conducting experiments with three random seeds. We evaluated conditioning only on state (ar (s)) versus conditioning on both state and time (a t)):
Task
a (s)
Tg (s,t)
Humanoid
4987.39 + 97.82
5076. + 37.47
These results demonstrate that conditioning the noise on both time and state yields higher rewards, consistent with the findings presented in Fig. 5 (page 9).
Noise scale in sparse-reward manipulation. We trained ReinFlow agents with different noise scales in the Franka Kitchen-comple task, with results averaged over three seeds:
Noise std
Success Rate
0.001
70.42% + 3.21% 90.67% 14.87%
0.08
0.16
99.08% + 1.01%
This experiment reveals that larger noise levels promote exploration. Together, these additional experiments demonstrate that the design choices and trends identified are robustly generalized across different types of environment, reward structures, and task complexities.
G Reproducing Our Findings
We list the key hyper-parametee and model architectures needed to reproduce the experiment results of ReinFlow and other baseline algorithms.
G.1 Hyperparameters of ReinFlow
We adopt the same batch size suggested in DPPO [42 and follow their implementation to normalize the reward with time-reversed running variance.
is essential to adjust the number of critic warmup iterations according to the performance of the pre-trained policy. A larger initial reward requires more warmup steps.
We clipped the denoised actions within [-1, 1] enhance training stability in implementation. Table indicates this option with "clip intermediate actions". We also find it beneficial to slightly reduce the maximum noise standard deviation the latter training course to postpone the reward decrease. cos(r1,77 r2) indicates learning rate that decays in the rate of cosine function from r1 to 72-
35

---

Table 6: ReinFlow's Shared Hyperparameters Across All Tasks
Parameter
Value
critic loss coefficient
0.50
GAE lambda À reward scale reward normalization
0.95
1.0
True
actor optimizer
Adam [26
actor learning rate weight decay
0
actor learning rate scheduler
CosineAmnealingWarmnn WarmupRestart 35
actor learning rate eycle steps
100
critic optimizer
Adam
critic learning rate scheduler critic scheduler warmup critic learning rate cycle steps
10
100
Table 7: ReinFllow's Hyperparameters in OpenAl Gym Locomotion Tasks
(a) Shared Hyperparameters Across OpenAl Gym Tasks
Parameter
Value
critic learning rate weight decay
le-5
number of parallel environments
40
reward discount factor y action chunking size condition stacking number batch size
0.99
4
1
50k
maximum episode steps number of rollout steps
1000
500
update epochs number of training iterations number of denoising steps clipping ratio e clip intermediate actions target KI divergence
5
1000
4
0.01
True
1.0
noise std upper bound hold for
35% of total iteration
noise std upper bound decay to entropy coefficient
0.3 Jmin 0.7 o mmax
0.03
BC loss (W2 regularization) coefficient
0.00
(b) Task-Specific Hyperparameters in OpenAl Gym Environment
Ant-v0
Parameter
Hopper-v2
Walker2d-v2
Humanoid-v3
minimum noise Ømin
0.10
0.10
0.08
maximum noise std max
0.24
0.24
0.16
critic warmup iters
0
actor learning rate
cos(4.5e-5, 2.0e=5) cos(6.5e-4, 3.0e-4)
cos(4.5c-4, 4.0c-4)
cos(4.5e-5, 2.0c-5)
critic learning rate
cos(4.0e-3, 4.0c-3)
cos(6.5e-4, 3.0c-4)
36

---

Table ReinFlow's Hyperparameters in Franka Kitchen State State-input Manipulation Tasks
Parameter
Value
critic learning rate weight decay
le-5
number of parallel environments reward discount factor y action chunking size condition stacking number batch size
40
0.99
4
5600
maximum episode steps number of rollout steps
280
200
update epochs number of training iterations number denoising steps clipping ratio c clip intermediate actions minimum noise std min maximum noise std mmax noise std upper bound hold for noise std upper bound decay to entropy regularization coefficient a BC loss (W2 regularization) coefficient 8
10
301
0.01
True
0.05
0.12
100% of total iteration
0 max
0.00
0.00
Table ReinFlow's Hyperparameters' Robomimic Visual Manipulation Tasks
Shared Hyperparameters Across Robomimic Tasks
Parameter
Value
critic learning rate weight decay
0
number of parallel environments
50
reward discount factor
0.999
condition stacking number pixel input shape
[3, 96, 96]
image augmentation gradient accumulation steps
RandomShift (padding=4)
15
batch size
500
update epochs
10
clipping ratio clip intermediate actions target KL divergence
0.001
True
lc-2
noise std upper bound hold for
100% of total iteration
noise std upper bound decay to
Fuxx
BC loss (W2 regularization) coefficient 3
0.00
(b) Task-Specific Hyperparameters in Robomimic Environment
Parameter
PickPlaceCan
NutAssemblySquare
TwoAmmTransport
minimum noise std Omin maximum noise std Fmax
0.08
0.08
0.05
0.14
0.14
0.10
number of denoising steps entropy coefficient
4
0.00
0.01
0.00
critic warmup iters
2
2
5
critic output layer bias actor learning rate warmup
0.0
0.0
4.0
10
25
10
actor learning rate critic learning rate
cos(2.0e-5, 1.0e-5)
cos(3.5e-6, 3.5e-6) cos(4.5e-4 3.0,e-444
cos(3.5e-2 3.5e-6)
cos(6.5e-4, 3.0e-4)
cos cos(3.2e-4, 3.0e-4)
action chunking size
8
number of cameras maximum episode steps
1
2
300
400
800
number of rollout steps
300
400
400
37
number of training iterations
150
300
200

---

G.2 Hyperparameters of DPPO
We strictly follow the hyperparameter setting of the official implementation DPPO. Please refer to Section E.10 of 42 for details.
The only changes make incorporating seeds 509 and 2025 for furniture tasks and elongating the rollout steps from 70 to 200 (while we still keep the maximum episode length as 280 in the environment configuration). DPPO and ReinFlow exhibit higher variability in Franka kitchen tasks across random seeds and even different runs, so we adopt five random seeds. We also find that a longer sampling trajectory helps the DPPO agent discover optimal strategies. For this reason, DPPO achieves a higher task completion rate than that reported by DPPO's original paper.
G.3 Hyperparameters of FQL.
We list the hyperparameters adopted by FQL in Table 10 steps such that the total sample consumption during the offline phase no less than the pre-trained consumption of DPPO ReinFlow in Gym (D4RL) and Franka Kitchen tasks.
We set the number of offline pre-traine pre-training
[38. the temperature coefficient, aFQL, is the most important hyperparametero FFQL. We followed the instructions of the original paper and scanned arQL in [0.03, 0.1, 0.3, 1, 10) to obtain a proper value for the temperature in Hopper-v2. and adopted the same value for all state-input tasks.
As described
Table 10: Hyperparameters for FQL in Gym and Franka Kitchen Tasks
Shared Hyperparameters for FQI.
Parameter
Value
number of denoising steps of the base policy лe
4
number of steps for evaluation
500
number of evaluation episodes reward discount factor y actor learning rate actor weight decay actor learning rate cycle steps actor scheduler warmup critic learning rate critic learning rate weight decay
10
0.99
1e
0
1000
10
3e-4
0
critic scheduler warmup
10
batch size
256
target EMA rate
0.005
reward scale
1.0
buffer size
1e6
Behavior cloning coefficient
3.0
actor update repeat
1
online steps
569,936
(b) Task-Specific Hyperparameters in Robomimic Environment
Parameter
Gym and Kitchen-zomplete-v00 200,000
Kitchen-mixed-v0 and Kitchen-partial-v0
offline steps
600,000
38


Reviews:
[{"review_id": "JNc5cR1oA8", "reviewer": "Reviewer_h3PM", "summary": "This paper proposes ReinFlow, which uses online reinforcement learning (RL) to fine-tune a family of flow-matching polices for continuous robotic control. The essential idea of ReinFlow is to inject noise into a flow policy’s deterministic path and convert the flow into a discrete-time Markov Process for tractable likelihood computation. This allows ReinFlow to stably fine-tune diverse flow models using online RL. Empirical evaluations on robotic locomotion and robotic manipulation tasks demonstrated that ReinFlow can increase success rate, episode reward, and save wall-time compared to the state-of-the-art diffusion RL method.", "questions": "1.\tHow does the noise injection net automatically balance exploration and exploitation?\n\n2.\tAs claimed in the paper, the noise network enables ReinFlow to learn to create more diverse actions by altering the intensity of noise at different denoising steps. The entropy regularization in Section 4.4 also promotes agents to seek more diverse actions. If the noise network itself already enables ReinFlow to have a good exploration ability, what is the reason to introduce the additional entropy regularization? Does the entropy regularization further promote the exploration ability of agents? Is this promotion scalable to different tasks, instead of only in locomotion tasks?\n\n**Minors:**\n\n- Line 157: For completeness, we provide the proofs in Appendix ??.", "limitations": "1.\tThe upper and lower bounds of the noise net is a hyper-parameter of ReinFlow that varies in different tasks and robot embodiments. This limits the scalability of ReinFlow to more diverse tasks without prior knowledge of the robot’s embodiments.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- This paper highlights the challenges of unstable back-propagation in fine-tuning conditional flow policies in very few steps for continuous control problems. In addition, designing an exploration mechanism for conditional flows with a deterministic path is also elusive. ReinFlow provides a stable fine-tuning solution, enables light implementation, and plugs into diverse flow models, e.g., ReFlow and Shortcut Models. \n\n- The idea of converting flows to a discrete-time Markov Process using a noise injection network is interesting and efficient. ReinFlow uses RL to improve flow policies at very few or even one denoising step, and enables faster inference and high success rate after fine-tuning. \n- The proposed method is evaluated in extensive experiments in robotic locomotion and robotic manipulation tasks. ReinFlow increases the success rate of pre-trained manipulation policies, and increases the episode rewards of locomotion policies significantly, with a reduction of wall-time compared the state-of-the-art diffusion RL method. \n\n**Weaknesses**:\n\n- Need more details and discussion on how the noise injection net can convert flow into a discrete-time Markov Process.  \n\n- Both noise injection and entropy regularization can enhance exploration. Missing discussion on the differences between the two methods.\n\n- Hyperparameters potentially need to be tuned in different tasks.", "was_revised": true, "final_justification": "The author's rebuttal has addressed my concerns. Thank you.", "rebuttal": "**General response to Reviewer h3PM**\n\nWe thank Reviewer h3PM for the thoughtful review and encouraging comments. We appreciate your recognition of our well-motivated problem formulation, the flexibility of our noise injection technique, and the effectiveness and efficiency demonstrated in our extensive experiments.\n\nBelow, we are happy to address your questions in detail.\n\n**1. Noise injection v.s. entropy regularization**\n> \"Both noise injection and entropy regularization can enhance exploration. Missing discussion on the differences between the two methods. \"\n\n> \"As claimed in the paper, the noise network enables ReinFlow to learn to create more diverse actions by altering the intensity of noise at different denoising steps. The entropy regularization in Section 4.4 also promotes agents to seek more diverse actions. If the noise network itself already enables ReinFlow to have a good exploration ability, what is the reason to introduce the additional entropy regularization? Does the entropy regularization further promote the exploration ability of agents? Is this promotion scalable to different tasks, instead of only in locomotion tasks?\"\n\nThank you for raising this important point! Here's a clarification:\n\n* **The purpose of using noise and entropy**\nAs described in Section 4.1 (page 4, lines 135–142), noise injection is primarily used to replace the $\\delta(\\cdot)$ function, making the action distribution tractable. The added stochasticity also naturally aids exploration. Entropy regularization (Section 4.4, page 6), on the other hand, provides an explicit mechanism to control exploration via its coefficient $\\alpha$. Importantly, the method remains mathematically valid even without the entropy term.\n\n* **Interaction between noise and entropy.**\nWe observe that during training, the noise network tends to reduce its output variance to stabilize returns, especially in tasks with dense rewards (e.g., locomotion). This behavior can prematurely dampen exploration, leading to convergence to suboptimal policies. Entropy regularization mitigates this by encouraging higher noise levels, thereby sustaining exploration throughout training.\n\n* **Entropy works in tasks beyond locomotion.**\n  - We successfully adopt entropy regularization in Robomimic visual manipulation task \"square\", as is shown in Table 9 b, page 35 in the Appendix. \n  - This technique also yields additional success rate in \"Franka Kitchen-complete\", which is a state-input, sparse-reward manipulation task. This is supported by an ablation experiment conducted during the rebuttal, where results are averaged over 3 seeds: \n| Without Entropy Reg.  | With Entropy Reg. ($\\alpha=0.01$) |\n|-------------------------------|-----------------------------------------|\n| 96.17 ± 3.65%             | 99.00 ± 0.75%                         |\n\nWe will follow your insightful advice and add a dedicated discussion to clarify the relationship between noise and entropy in the revised manuscript.\n\n**2. Conversion to a Discrete-time Markov Process**\n> \"Need more details and discussion on how the noise injection net can convert flow into a discrete-time Markov Process.\"\n\nThank you for the suggestion on emphasizing the functionality of the noise injection net. We explained in Section 4.1(lines 135 to 143, page 4) that the noise conditions on the current state and action, ensuring that the resulting process maintains the Markov property. We will follow your advice and highlight this mechanism in our revision. \n\n**3. Automatic Exploration–Exploitation Trade-off**\n> \"How does the noise injection net automatically balance exploration and exploitation?\"\n\nThat is an insightful observation! As described in Section 6 (page 8), the stochasticity introduced by the noise net promotes exploration early in training, allowing the policy to exceed pre-trained performance. Over time, we also observe the network learns to reduce noise levels, shifting toward exploitation. We will emphasize this adaptive behavior in the revised manuscript.\n\n**4. Tuning the hyperparameter of noise levels.**\n> \"The upper and lower bounds of the noise net is a hyper-parameter of ReinFlow that varies in different tasks and robot embodiments. This limits the scalability of ReinFlow to more diverse tasks without prior knowledge of the robot’s embodiments.\"\n\nThank you for your thoughtful review! We report the noise bounds in Tables 7 to 9 (page 33 to 35 in the Appendix). While task-specific tuning may yield minor gains, we find (lines 262 to 265, page 8) that ReinFlow is relatively robust to noise level changes within a reasonable range: a noise standard deviation of 0.08-0.12 generally works well across most tasks. Nonetheless, reducing ReinFlow's sensitivity to noise level is a promising direction for future work!\n\n**5. Typo**\n> \"Line 157: For completeness, we provide the proofs in Appendix ??.\"\n\nThank you—the proof presents in Appendix B (page 22). We will correct this and ensure all appendix references are properly linked.\n\n**Concluding remark**\n\nWe hope our response has addressed your concerns and the questions raised. We would be grateful if you kindly consider improving our rating. Please do not hesitate to let us know if you have any further questions."}, {"review_id": "1M3bFWBecY", "reviewer": "Reviewer_6Jdc", "summary": "This paper proposes ReinFlow, an online reinforcement learning framework for fine-tuning flow matching policies, especially those trained from imperfect imitation data. The method injects learnable, bounded Gaussian noise into the deterministic flow policy to turn it into a discrete-time Markov process. The paper demonstrates improvements across both locomotion and manipulation tasks (with state or pixel inputs, dense or sparse rewards), using as few as one denoising step, outperforming or matching baselines like DPPO and FQL while reducing wall-time by up to 80%.", "questions": "1. From Figure 6, it appears that the performance of ReinFlow is quite sensitive to small changes in noise magnitude and regularization strength, with no consistently clear trends. This raises the concern that significant per-task hyperparameter tuning may be required, which limits practical scalability. Do the authors have any thoughts on improving robustness or automatically adapting these hyperparameters during training? Also, could you clarify the meaning of $\\alpha$ in this context?\n\nMinor:\n1. Figure 1 (c) and (d) should be \"v3\" instead of \"v2\"?\n2. Missing caption for the last figure (figure 6).", "limitations": "The authors have mentioned some future work in the last section.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- First practical online RL framework for flow matching policies in robot learning. The proposed method is principled and mathematically sound.\n- Good readability for most of the part. \n- Significant wall-time savings compared to diffusion-based methods, and it works effectively with very few denoising steps, which is critical for real-time robot control.\n\nWeakness:\n- The authors claimed to \"prioritize wall-time efficiency in simulated environments over sample cost\", while they chose to present sample cost instead of wall-time efficiency for `Kitchen-mixed`, `Kitchen-partial` and `Robomimic` benchmarks. This creates a confusing inconsistency and makes it harder to verify their wall-time efficiency claims in those settings.\n- Section 6 presents only partial results for the design choices and ablation studies. It's unclear whether the observed trends and results generalize across all tested environments.", "was_revised": true, "final_justification": "Most of the concerns are addressed in the response, while the sensisitivity still remains an issue, the authors provided several possible solutions.", "rebuttal": "Thank you for the encouraging feedback! \n\nWe appreciate your acknowledgement on the novelty and theoretical soundness of our algorithm, the clarity of our presentation, and ReinFlow's advantage in wall-time saving. \n\nWe are happy to address your questions and concerns in what follows. \n\n****\n\n**1. Use a Consistent Horizontal Axis**\n> \"The authors claimed to 'prioritize wall-time efficiency in simulated environments over sample cost', while they chose to present sample cost instead of wall-time efficiency for Kitchen-mixed, Kitchen-partial and Robomimic benchmarks. This creates a confusing inconsistency and makes it harder to verify their wall-time efficiency claims in those settings.\"\n\nWe thank the reviewer's helpful suggestion to emphasize wall-time efficiency in the main text. Due to space constraints, we included detailed wall-time statistics in Table 3 (page 28) of the appendix. In future revisions, we will follow your advice and include wall-time efficiency plots for the Franka Kitchen-mixed, Kitchen-partial, and Robomimic tasks directly in the main text to improve clarity and consistency of our work.\n\n****\n\n**2. Ablation studies in More Environments**\n\n> \"Section 6 presents only partial results for the design choices and ablation studies. It's unclear whether the observed trends and results generalize across all tested environments.\"\n\nThank you for your feedback on the ablation studies. In Section 6, we presented 7 detailed sensitivity analyses across 5 representative environments, and we observed similar trends in other tasks. \n* During the rebuttal, we compare the effect of different types of noise conditions in Humanoid-v3 environment. We run the group that conditions only on the state ($\\sigma(s)$) and the group that conditions on both state and time ($\\sigma(s,t)$) with three random seeds. \n| Task    | $\\sigma(s)$  | $\\sigma(s,t)$  |  \n|-------------------|--------------|------------------|  \n| Humanoid| 4987.39\t± 97.82  |  5076.12 ± 37.47   |  \n\n  The results show that conditioning the noise on both time and state is beneficial to obtain higher reward, coinciding with the results in Fig. 5 (page 9). \n\n* We also train ReinFlow agent with different noise scales in Franka-kichen complete, and result is averaged over three seeds. \n| Noise std| Success Rate |\n|-------------|----------------------------------|\n| 0.001       | 70.42 % ± 3.21  %         |\n| 0.08        | 90.67 % ± 14.87 %         |\n| 0.16        | 99.08 % ± 1.01   %         |\n\n  This experiment reveals that a larger noise level promotes exploration and it is also supported by lines 262-265 (page 8). \n\nDue to time constraint in the rebuttal phase, we are still running the rest of the new experiments, and we will update results when we are ready and incorporate it in our subsequent revisions. \n\n****\n\n**3. Understanding the Sensitivity Analysis**\n\n> \"From Figure 6, it appears that the performance of ReinFlow is quite sensitive to small changes in noise magnitude and regularization strength, with no consistently clear trends. This raises the concern that significant per-task hyperparameter tuning may be required, which limits practical scalability. Do the authors have any thoughts on improving robustness or automatically adapting these hyperparameters during training? Also, could you clarify the meaning of $\\alpha$ in this context?\"\n\nWe appreciate the reviewer’s insightful comments on hyperparameter sensitivity and offer the following clarifications:\n\n* **Trend in Noise Magnitude**. \n  - As discussed in lines 262 to 268 (page 8), small noise levels (0.001–0.040) leads to insufficient exploration, resulting in performance close to the pre-trained policy. However, a moderate noise range (0.04–0.20) yields high reward, with robustness to variations within this range. This demonstrates ReinFlow is **not overly sensitive** once past a noise threshold.\n  - Furthermore, we provide the chosen noise valuess for each task in the Appendix (page 33-35), where a noise range between 0.06 to 0.16 is reasonable to most tasks. Although it may yield some benefits, extensive per-task tuning is not necessary. \n\n* **Trend in Regularization Strength**\n\n  In Fig 6(a) (page 9), we illustrate the effects of two distinct types of regularization:\n  - **Wasserstein-2 Regularization** (blue curves, Eq. 9, page 6) with coefficient $\\beta$ penalizes deviations from the pre-trained model. As expected, as $\\beta$ increases from 0.01 to 1.0, the policy becomes more constrained, leading to a clear decline in reward back to the pre-trained level. \n   - **Entropy Regularization** (red curves, lines 195–205, page 6), with coefficient $\\alpha$, encourages exploration and consistently achieves higher rewards. This suggests that entropy regularization is more effective than conservative constraints in our setting. The coefficient $\\alpha$ is defined in Algorithm 1 (line 19, page 4).\n\n    To improve robustness, future work could explore **adaptive tuning** of $\\alpha$ using primal-dual methods, as done in Soft Actor-Critic [1].\n\n****\n\n**4. Typos**\n> \"Figure 1 (c) and (d) should be  'v3' instead of 'v2'? \"\n> \"Missing caption for the last figure (figure 6).\"\n\nWe appreciate the reviewer's suggestions!. We will be sure to revise those typos based on your valuable feedback. \n\n****\n\n**Final remark**\n\nWe hope our response have addressed your concerns effectively, and we kindly ask whether you would consider improving our rating. We are happy to clarify any remaining questions. Thank you for your time and thoughtful feedback!\n\n****\n\n**References**\n\n[1] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. Pmlr, 2018."}, {"review_id": "Mdw4sd91BM", "reviewer": "Reviewer_S8sj", "summary": "The document introduces ReinFlow, a novel online reinforcement learning framework designed to fine-tune flow matching policies for continuous robotic control. Unlike previous methods, ReinFlow addresses challenges in likelihood computation and exploration for flow models by injecting learnable noise, converting the deterministic flow into a discrete-time Markov process. This approach significantly improves the success rate and reward of pre-trained policies in various locomotion and manipulation tasks, including those with visual input and sparse rewards. The paper demonstrates ReinFlow's better efficiency compared to state-of-the-art diffusion RL methods and examines key factors influencing its performance, such as noise levels and regularization techniques.", "questions": "For the results in Figure 1. Is the ReinFlow baseline based on similar implementations as the baselines? It would be nice to know how the wall-clock savings can be broken down as implementation efficiency + algorithmic efficiency.\nFor evaluation, it looks like you keep most hyperaparameters of the baseline (DPPO) intact. That implies that DPPO is using a lot more denoising steps which is computationally expensive. I would like to see some evaluation between ReinFlow and DPPO with comparable denoising steps in terms of sample complexity and wall-clock time. That would allow a better assessment of the algorithmic improvements made in this work.\n\nI am willing to increase my score if the authors can address my comments above.\n\nNit:\n* Line 269 Figure 6 - you are missing the super caption for this figure. 6(a) the color encoding is repeated for std = 0.160 and 0.040. Also, the color coding is difficult to see.\n* Line 1011 \"asymptotically inferior” In the evaluation of FQL on sample complexity, it looks like it is not run with the same number of total steps.\n* 5.2 Performance evaluation x-axis as number of frames instead of wall-clock time. What is the confidence interval reported?\n* Line 157 missing reference to Appendix.\n* DPPO evaluated on halfcheetah. Lift - state (https://arxiv.org/pdf/2409.00588). It would be great if you can also include it in your evaluation.\n* Line 127 “infact”", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The biggest strength of the proposed method is the improvement in wall-clock efficiency - Reinflow can achieve comparable performance as DPPO at a fraction of the wall clock time. There are also improvements in asymptotic performance in various tasks.\nThere are good evaluation and ablation on numerous environments which help understand the effect of ReinFlow’s hyperparameters.\n\nI have some questions regarding some experimental evaluation which I have detailed in the questions section.", "was_revised": true, "final_justification": "The authors have addressed my points.", "rebuttal": "**General Response to Reviewer S8sj**\n\nThank you for your positive and constructive feedback!\n\nWe appreciate your recognition of the originality and novelty of our approach, how we address the challenge of likelihood computation and ReinFlow's significant improvement in success rate and wall-time efficiency. We are also grateful for your comments on the thoroughness of our sensitivity analysis. \n\n**** \n\n**1. Mechanism Behind Wall-Time Reduction**\n\n> ...It would be nice to know how the wall-clock savings can be broken down as implementation efficiency + algorithmic efficiency...I would like to see some evaluation between ReinFlow and DPPO with comparable denoising steps in terms of sample complexity and wall-clock time. That would allow a better assessment of the algorithmic improvements made in this work.\n\n> I am willing to increase my score if the authors can address my comments above.\n\nThank you for raising this important point. We address it below:\n\n* **Implementation Consistency**. \nWe use the same implementation as DPPO wherever possible, including training loops, buffer structures, and shared hyperparameters. DPPO's hyperparameters remain unchanged from its official release, and our submitted code in the supplementary materials reflects this.\n\n* **Why is ReinFlow faster?** The observed wall-time savings stem primarily from algorithmic efficiency. \n  - a). **Fewer Denoising Steps**: Flow policies such as Rectified Flow need fewer integration steps than DDPM/DDIM to achieve similar precision, as evidenced by [1,2,3]. \n  - b). **Exact Likelihood**: ReinFlow precisely calculates the flow policy’s probability at any steps, enabling training with as few as four or even one step (Section 4.1, page 3).  \n  - c). **Stabilized Fine-Tuning**: By controlling the magnitude of noise and denoised actions (Appendix, lines 845–852, page 22), ReinFlow avoids abrupt policy update during fine-tuning, ensuring a stable increase in success rates \n  - d). **Simplified Computation in Optimization**: Unlike DPPO, ReinFlow avoids computing the value and advantage functions at each denoised step, saving optimization time (Appendix C, page 25). \n\n* **If DPPO uses the same steps as ReinFlow, what will happen?**\n\nWe thank the reviewer for the helpful feedback. To address the question directly, we ran both DPPO and ReinFlow with 4 denoising steps during evaluation and RL training. Experiments were conducted on a single NVIDIA RTX 3090 GPU with EGL backend and averaged over seeds 1, 2, and 3.\n\n  - **Inference Time per Action (ms)**  \n| Task              | DPPO         | ReinFlow         |  \n|-------------------|--------------|------------------|  \n| Hopper            | 4.25 ± 0.16  | 3.71 ± 0.21   |  \n| Walker            | 4.47 ± 0.25  | 3.90 ± 0.19   |  \n| Kitchen-complete  | 7.22 ± 0.08  | 6.67 ± 0.06 |  \n\n  - **RL Reward / Success Rate**  \n| Task              | DPPO                | ReinFlow             |  \n|-------------------|---------------------|-----------------------|  \n| Hopper            | 2689.54 ± 35.47     | 3197.33 ± 60.95   |  \n| Walker            | 3688.90 ± 60.83     | 4171.81 ± 141.88  |  \n| Kitchen-complete  | 5.83% ± 0.95%       | 98.25% ± 0.66%    |  \n\n  - **RL Wall-Time per Iteration (s)**  \n| Task              | DPPO         | ReinFlow         |  \n|-------------------|--------------|------------------|  \n| Hopper |  43.95 ± 1.41 | 14.17 ± 0.35 |\n|Walker| 45.03 ± 0.59 | 14.46 ± 0.21 | \n| Kitchen-complete  | 41.71 ± 0.52 | 26.56 ± 0.38  |  \n\n  - **Conclusion:**  \n    - During inference, DPPO and ReinFlow have similar per-action time, with ReinFlow slightly faster due to the use of an ODE solver rather than an SDE solver.  \n    - During RL, ReinFlow reduces wall-time per iteration by accelerating both rollout and optimization, resulting in faster training.  \n    - With the same number of denoising steps, DPPO starts with lower rewards and continues to lag behind, while ReinFlow achieves significantly better performance, combining fast inference with high reward.  These trends align with prior findings ([1], Fig. 5.4 and 5.5, p. 9). \n\nWe appreciate the reviewer’s suggestion and will include further discussion on ReinFlow’s wall-time efficiency in future versions.\n\n**** \n\n**2. Confidence interval**\n\n> What is the confidence interval reported?\n\nThank you for this question! As noted in Appendix (Page 28, Lines 981–982), we report **one-sigma error bars**, with shaded areas indicating one standard deviation from the mean. We will clarify this in the main text to improve its visibility.\n\n**** \n\n**3. Additional experiments**\n\n>   DPPO evaluated on halfcheetah. Lift - state... It would be great if you can also include it in your evaluation. \n\nWe thank the reviewer for the suggestions to benchmark in additional state-based tasks. In Section 5.2 (page 7), we have chosen a diverse set of tasks--both state and visual input--with varying difficulty for a representative evaluation. \n\nWhile time constraints prevent us from adding all suggested tasks during the rebuttal phase, we agree with the value of this suggestion and plan to incorporate them in future work or a camera-ready revision.\n\n**** \n\n**4. Typos and Formatting**\n\n> Line 269 Figure 6 - you are missing the super caption for this figure. 6(a) the color encoding is repeated for std = 0.160 and 0.040. Also, the color coding is difficult to see.\n> Line 127 “in fact” \n> 5.2 Performance evaluation x-axis as number of frames instead of wall-clock time.\n> Line 157 missing reference to Appendix. \n> Line 1011 \"asymptotically inferior” In the evaluation of FQL on sample complexity, it looks like it is not run with the same number of total steps.\n\nThank you for the thorough review! We will fix these issues in our subsequent revisions. \n\n**** \n\n**Final Remark**\n\nWe hope we have addressed your concerns and we appreciate it if you kindly consider improving our rating. Please also let us know if you have any remaining questions!\n\n****\n\n**References**\n\n[1] Zheng, Qinqing, et al. \"Guided flows for generative modeling and decision making.\" arXiv preprint arXiv:2311.13443 (2023).\n\n[2] Zhang, Fan, and Michael Gienger. \"Affordance-based robot manipulation with flow matching.\" arXiv preprint arXiv:2409.01083 (2024). \n\n[3] Liu, Xingchao, and Chengyue Gong. \"Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.\" The Eleventh International Conference on Learning Representations."}, {"review_id": "zmawp7kTLm", "reviewer": "Reviewer_aYCb", "summary": "The authors propose ReinFlow, a method to fine-tune flow-matching policies for online RL. The main difficulty of fine-tuning flow policies with PPO-like online RL algorithms is in computing the log likelihood of actions. To deal with this, the authors convert the flow ODE into an SDE with noise injected at each step. The strength of the noises is controlled by another learnable network trained in an end-to-end manner. They show that ReinFlow outperforms DPPO and FQL on various robotics domains.", "questions": "* What's the disadvantage of using Eq. (4)? It seems this is the most straightforward way to fine-tune a flow policy with online RL. Have the authors empirically compared ReinFlow to Eq. (4) with (say) 10 flow steps?", "limitations": "Limitations are not explicitly discussed, even though the authors claim in the checklist that they are discussed in the last section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n* The method is straightforward and reasonable.\n* The paper is easy to understand.\n* The authors evaluate ReinFlow across diverse settings (including pixel-based tasks).\n* The authors provide quite extensive ablation studies about ReinFlow's different design choices.\n\nWeaknesses\n* The performance improvement over DPPO seems marginal in general. While the authors show that ReinFlow is better than DPPO in terms of *wall-clock* time, ReinFlow and DPPO achieve similar performance in terms of environment steps (Fig. 10 in the Appendix). Also, are there any reasons why ReinFlow should be faster than DPPO *in principle*, given they are both based on the same PPO framework with a similar iterative policy training scheme? Moreover, vanilla SAC achieves significantly better sample efficiency on all these Gym MuJoCo tasks (see the original SAC paper).\n* Only two flow/diffusion baselines are considered in the main text. Since FQL is essentially an offline RL algorithm, the comparison is mainly done only against DPPO. While the authors compare with more baselines in Fig. 11 and 12 in the Appendix, again, the improvement is marginal, and vanilla SAC seems to outperform all of the baselines. \n* The authors use only 3 seeds in most plots. In the Appendix, the authors mention using seeds {0, 42, 3407} (and additionally {509, 2025} for some Kitchen experiments). This seems quite arbitrary; it'd have been better if the authors had used more systematically generated numbers so that they didn't look cherry-picked.\n\n\nMinor comments\n* L47: \"We propose ReinFlow, the first online RL algorithm to stably fine-tune a family of flow matching policies\" -- I think this is an overclaim given the presence of Flow-GRPO.\n* L127: Infact -> In fact\n* L142: \"the generated sigma algebra of $a_t$ and $t$\" -- While random variables do generate sigma algebras, $a_t$ and $t$ seem deterministic variables here, technically.\n* I'd highly recommend using consistent notations to denote times in flow matching (e.g., subscripts in Eq. (5) vs. superscripts in Eq. (6)).\n* L157: Missing reference.\n* L202: Use \\citet.", "was_revised": true, "final_justification": "My initial concerns about statistical significance have been resolved, and I see the relevance of this paper in that it proposes one of the first approaches that fine-tune a flow policy with policy gradients. That said, I'm still not convinced by the significance of their empirical results, especially in comparison to previous baselines (DPPO, SAC, etc.).\n\nGiven the above, I've adjusted my score to 4 (borderline accept), but I wouldn't be opposed to rejection either.", "rebuttal": "**General response to Reviewer aYCb**\n\nWe thank your detailed review and the positive comment on the clarity of our presentation, the soundness of our algorithm design, and the comprehensiveness of our empirical results! \n\n* **The Reviewer’s Concern.**\nThe reviewer raises concerns about ReinFlow's success rate relative to SAC and DPPO. However, _these methods serve different purposes_: SAC trains Gaussian policies from scratch, and DPPO fine-tunes Diffusion Policies. In contrast, ReinFlow is designed _as an online RL framework for fine-tuning flow matching policies obtained via imitation learning_—a growing focus in robot learning [1,3,4].\n\n* **Intent and Scope of Our Contribution.**\nWhile SAC and DPPO achieve near-optimal success rates in several tasks, focusing solely on this metric overlooks our contributions: a) ReinFlow improves performance _over imitation learning baseline_, with significant wall-time efficiency.\nb) It is explicitly designed for a broad class of _flow_ policies, an area not addressed by SAC or DPPO.\n\n****\n\n**1. ReinFlow and SAC**\n> \"Vanilla SAC achieves significantly better sample efficiency on Gym MuJoCo tasks (see the original SAC paper).\"\n\nSAC is effective in simple MuJoCo tasks, but this does not diminish our contributions. In fact, its limitations in more complex domains highlight ReinFlow’s broader applicability.\n\n* **Different design goals**. SAC trains _Gaussian_ policies _from scratch_ [5], which often lacks the expressiveness needed in complex tasks [2]. ReinFlow instead fine-tunes _flow matching_ policies _following imitation learning_—a important direction gaining significant traction [1,3,4].\n\n* **ReinFlow's flexible implementation**. While ReinFlow currently builds on PPO, it can, in principle, incorporate SAC-style updates via Theorem 4.1 (lines 158–162), potentially improving sample efficiency (lines 212–214).\n\n* **ReinFlow excels in complex tasks, where SAC fails**\n  - Although effective in dense-reward MuJoCo tasks, _SAC performs poorly_ in sparse-reward, long-horizon manipulation tasks, such as Franka Kitchen ([7], page 8, Fig. 5) and Robomimic ([6], page 17).\n  - DPPO [2] explicitly reports it is \"...the first RL algorithm to solve Transport with >50% success rates\" (page 11), implying SAC's limitations.\n  - In contrast, _ReinFlow_ achieves >70% success rates across all tested sparse-reward manipulation tasks (Appendix, page 30), _demonstrating robust performance where SAC fails_.\n\n* **ReinFlow matches or outperforms SAC’s flow variant, FQL.**\nFQL is a SAC-based method for flow policies [1], and ReinFlow matches or exceeds its reward in MuJoCo and Franka Kitchen , while achieving superior wall-time efficiency (Section 5.2, page 7).\n\nWe thank the reviewer for highlighting the importance of sample efficiency. We plan to integrate SAC-style off-policy updates for ReinFlow in future work.\n\n****\n\n**2. ReinFlow and DPPO**\n\n> 2.1 Only two flow/diffusion baselines are considered in the main text...The performance improvement over DPPO seems marginal in general. \n\n* **On the choice of baselines**: We emphasize FQL and DPPO in the main text because they represent state-of-the-art methods for flow and diffusion models at the time of submission, respectively [1,2]. Moreover, six additional baselines are included in the Appendix (page 31). \n\n* **Success Rate and the Intention of Our Work**. \n  - **Our goal** is not to significantly outperform DPPO in success rate, as it is nearly-optimal in many tasks, especially in Robomimic [2]. Rather, we aim to introduce the first online RL framework tailored for flow policies, and improve the success rate when data is suboptimal (lines 22-34, page 1)—an area that lacked an equivalent to DPPO and is gaining significant attention [1,3,4]. \n  - **Our contribution**: Unlike diffusion which rely on SDEs, flows based on ODEs and require different treatment. ReinFlow fills this gap, matching or beating DPPO while offering substantial wall-time reduction (Section 5.2, page 7). In this sense, ReinFlow serves flow policies as effectively as DPPO does for diffusion policies, if not better. \n\n> 2.2 ReinFlow and DPPO achieve similar performance in terms of environment steps. \n\nWe focus on wall-time efficiency, not sample efficiency, given the simulated nature of our experiments (page 6). While improving sample efficiency is important—especially for real-world RL—we leave this as promising future work.\n\n> 2.3 Why ReinFlow should be faster than DPPO in _principle_ ?\n\nReinFlow achieves faster wall-time performance for three key reasons:\n * (a)  **Fewer Denoising Steps:**  Flow models (e.g., ReFlows) require fewer steps than diffusion models (e.g., DDPM, DDIM) to reach high initial reward. ([8], Fig. 5.4; [9], page 8). \n*  (b)  **Exact Probability Description at Few Steps:**  The probability of flow ODEs depends on the denoising step number. However, ReinFlow's expression remains exact even with very few steps (lines 138–147), making it possible to take advantage of flow's property, enabling efficient fine-tuning without compromising reward.\n*  (c)  **Simplified Optimization:**  We simplify DPPO’s policy loss by not computing value/ advantage for denoised actions (Appendix C, page 25). This further reduces computational cost.\n\nWe appreciate your question and plan to better explain wall-time efficiency in our revision. \n\n****\n\n**3. Random Seeds**\n> ...it'd have been better if ...used more systematically generated numbers...\n\nWe took your advice seriously and ran additional experiments using seeds 1, 2, 3, 4, 5.\n\nDue to time limit, we train Rectified Flows in locomotion and Shortcut Models in manipulation. \n\n|Task|Original Seeds|New seeds|\n|----|----|---|\n|Hopper|3205.33 ± 32.09|3197.33 ± 60.95|\n|Walker2d|4108.57 ± 51.77|4171.81 ± 141.88|\n|Ant|4009.18 ± 44.60|3884.52 ± 161.51|\n|Humanoid|5076.12 ± 37.47|5007.35 ± 88.25|\n|Kitchen-complete|96.17% ± 3.65%|98.25% ± 0.66%|\n|Kitchen-mixed|74.63% ± 0.36%|74.85% ± 0.14%|\n|Kitchen-partial|84.59% ± 12.38%|70.60% ± 19.73%|\n|Can|98.50% ± 0.71%|98.70% ± 1.35%|\n|Square|74.67% ± 2.66%|71.10% ± 4.64%|\n|Transport|88.67% ± 4.40%|85.75% ± 2.90%|\n\nWhile the differences are not substantial, we will follow your advice and update the experimental section accordingly. \n\n****\n\n**4. Minor comments**\n\n> L47: \"ReinFlow, the first online RL algorithm to stably fine-tune a family of flow matching policies\" -- this is an overclaim given Flow-GRPO\"\n\nFlow-GRPO is a concurrent work and we clarified the distinctions in Section 2 (lines 88–90, page 3). To the best of our knowledge, our claim is valid at the time of submission. \n\n* Flow-GRPO does not train flow-matching *policies* for robotic control. It targets flow-matching models for image generation ([11] , Section 5, page 5). \n* Flow-GRPO's mathematical derivations are specific to Rectified Flows (Section 3, page 3; Eqs. 26–29, page 17 of the Appendix). In contrast, ReinFlow applies to broader flow model families, including Shortcut Models [10] (Section 5.2, page 7, Theorem 4.1 page 5).\n\n> \"L142: generated sigma algebra.\"\n\nWe refer to the \"generated sigma algebra\" because in ReinFlow, the denoised actions are stochastic due to the injection of Gaussian noise (Algorithm 1, page 4). \n\n> \"What's the disadvantage of using Eq. (4)? \" \n\nThank you for raising this point!  Eq. (4) is accurate only when the step size is very small (lines 132–134, page 4)—contrary to our goal of high inference speed. Moreover, Eq. (4) introduces Monte Carlo noise, whereas our method offers exact estimation of the action log probability (lines 138–147) at arbitrarily few steps. \n\nWe observed that using Eq. (4) with deterministic policy gradient methods (e.g., TD3) led to unstable training, while ReinFlow produced stable and reliable rewards (Section 5.2, page 7). We will emphasize this in future revisions.\n\n> Limitations are not explicitly discussed. \n\nWe do acknowledge the limitations of our work in Section 7, page 9. \n\n> I'd highly recommend using consistent notations to denote times in flow matching. L157: Missing reference. L202: Use \\citet. L127: Infact -> In fact\n\nThank you! We will fix the typos in the revised manuscript.\n\n**** \n\n**Final Remark**\n\nThe reviewer is concerned with the comparison to SAC in locomotion. While SAC is effective on simple benchmark, ReinFlow excels in **more complex tasks** and is designed for **a new purpose**--**to improve flow policy after imitation learning**. We respectfully ask the reviewer to reconsider the assessment of our work's quality and significance, and we're happy to clarify any questions.\n\n**** \n\n**References**\n\n[1] Park, Seohong, et al. \"Flow q-learning.\" arXiv:2502.02538 (2025). \n\n[2] Ren, Allen Z., et al. \"Diffusion policy policy optimization.\" arXiv:2409.00588 (2024).\n\n[3] Braun, Max, et al. \"Riemannian flow matching policy for robot motion learning.\" 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024.\n\n[4] Black, Kevin, et al. \"$\\pi_0 $: A Vision-Language-Action Flow Model for General Robot Control.\" arXiv:2410.24164 (2024).\n\n[5] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. Pmlr, 2018.\n\n[6] Zhu, Yuke, et al. \"robosuite: A modular simulation framework and benchmark for robot learning.\" arXiv:2009.12293 (2020).\n\n[7] Zhang, Jesse, et al. \"EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data.\" arXiv:2406.17768 (2024).\n\n[8] Zheng, Qinqing, et al. \"Guided flows for generative modeling and decision making.\" arXiv:2311.13443 (2023).\n\n[9] Zhang, Fan, and Michael Gienger. \"Affordance-based robot manipulation with flow matching.\" arXiv:2409.01083 (2024).\n\n[10] Liu, Jie, et al. \"Flow-grpo: Training flow matching models via online rl.\" arXiv:2505.05470 (2025).\n\n[11] Frans, Kevin, et al. \"One step diffusion via shortcut models.\" arXiv:2410.12557 (2024)."}]

Author rebuttal:

Meta-review: The paper proposes a new approach to perform online finetuning of flow-matching-based policies through a specific way to inject noise in the diffusion process, which converts the flow formulation into a discrete-time Markov chain with tractable likelihood computation. The empirical shows that the resulting method can significantly improve the initial policy in simulated locomotion and manipulation domains while reducing the computational complexity wrt to flow-based baselines.

In the rebuttal phase, the authors addressed most of the concerns from the reviewers, notably expanding the empirical validation to include additional ablations (e.g., to clarify the difference between noise injection and entropy regularization) and clarified aspects such as wall-time computation and comparison with baselines. One remaining among reviewers after the rebuttal was the marginal improvement obtained wrt to SAC and DPPO (at least in some domains). While the authors pointed out that SAC and ReinFlow are effective in different domains and DPPO is less efficient, I think this stands as a valid concern and I strongly encourage the authors to acknowledge this aspect in the updated version of the paper and expanding on the discussion in the rebuttal. Despite this aspect, there is an increasing evidence that diffusion/flow-based policies are the most effective representation in many robotic domains and proposing a computationally efficient and performance effective fine-tuning mechanism is of paramount importance in this context. For this reason and given the overall consensus from reviewers, I propose acceptance for the paper.
Decision: Accept (poster)