Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generation
Authors: Guojian Zhan
Letian Tao
Pengcheng Wang
Yixiao Wang
Yuxin Chen
Yiheng Li
Hongyang Li
Masayoshi Tomizuka
Shengbo Eben Li
Venue: ICLR 2026 Oral (2026)
Keywords: Reinforcement learning
Generative policy
Primary area: reinforcement learning
TL;DR: We introduce the mean velocity policy, a new RL policy that, along with a novel instantaneous velocity constraint, achieves state-of-the-art performance and the fastest training and inference speed.

Abstract: Learning expressive and efficient policy functions is a promising direction in reinforcement learning (RL). While flow-based policies have recently proven effective in modeling complex action distributions with a fast deterministic sampling process, they still face a trade-off between expressiveness and computational burden, which is typically controlled by the number of flow steps. In this work, we propose mean velocity policy (MVP), a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation. To ensure its high expressiveness, an instantaneous velocity constraint (IVC) is introduced on the mean velocity field during training. We theoretically prove that this design explicitly serves as a crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness. Empirically, our MVP achieves state-of-the-art success rates across several challenging robotic manipulation tasks from Robomimic and OGBench. It also delivers substantial improvements in training and inference speed over existing flow-based policy baselines.

Published as a conference paper at ICLR 2026
MEAN FLOW POLICY WITH INSTANTANEOUS VELOC- ITY CONSTRAINT FOR ONE-STEP ACTION GENERATION
Guojian Zhan Letian Tao Pengcheng Wang?, Yixiao Wang', Yiheng Li?, Yuxin Chen?,
Hongyang Li Masayoshi Tomizuka', Shengbo Eben Li
1 School of Vehicle and Mobility & College of AI, Tsinghua University
Berkeley AI Research (BAIR), UC Berkeley
3 The University of Hong Kong
ABSTRACT
Learning expressive and efficient policy functions is promising direction in rein- forcement learning (RL). While flow-based policies have recently proven effective in modeling complex action distributions with a fast deterministic sampling process, they still face trade-off between expressiveness and computational burden, which is typically controlled by the number of flow steps. In this work, we propose mean velocity policy (MVP). new generative policy function that models the mean velocity field to achieve the fastest one-step action generation. expressiveness, an instantaneous velocity constraint (IVC) is introduced on the mean velocity field during training. We theoretically prove that this design explic- itly serves crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness. Empirically, our MVP achieves state-of- the-art success rates across several challenging robotic manipulation tasks from Robomimic and OGBench. It also delivers substantial improvements in training and inference speed over existing flow-based policy baselines.
ensure high
INTRODUCTION
promising topic reinforcement learning (RL) community is develop expressive and ef- ficient policies. particularly in complex control environments where action distributions can be multi-modal (Zhu al., 2023; Wang et al., 2023). Generative policies, such as diffusion model and flow matching, have recently emerged as powerful alternative to Gaussian mixture policies by transforming simple base distributions into flexible action distributions via learn- able transformations (Song et al., 2020; Chi et al., 2023). However, a key limitation of exist- ing generative policies is their dependence on itcrative multi-step refinement from noise to ac- tions (Wang et al., 2024a; 2025; Ding et al., 2024). This computational dependency imposes a significant overhead that hinders training speed, particularly for online RL where action sam- pling is per-step requirement (Li, 2023; Yang et al., 2023). Moreover, this overhead trans- lates to considerable inference latency, which is a major impediment to achieving high closed- loop performance in real-time control systems (Zhan al., 2024; 2025; Jiang al., 2024).
question naturally arises: Can we unify the expressive- ness of generative policies with the efficiency of one-step action generation for online RL?
1.0
Success rate
0.8
QC
MVP (ours)
In this paper, we propose the mean velocity policy (MVP) as an affirmative answer. While existing flow policies learn instantancous velocities and require multi- step iterative sampling (Lipman et al., 2023; Park et al., 2025; Bharadhwaj et al., 2024), MVP instead learns the mean velocity field (Geng et al., 2025a). This design
0.4
BFN
50
75
100
125
150
175
Online training speed (Herrs)
Figure 1: Performance--ffiriencyy com-
"quual contribution.
parison on 9 robotic manipulation tasks.
Corresponding author. 1 shbo@ts nghua, edu. en.

---

Published as a conference paper at ICLR 2026
enables direct, single-step mapping from base Gaussian noise multi-moda action distribution, thereby preserving the expressive power of flow-based models while drastically improving training and inference efficiency (Kornilov et al., 2024).
Although the time-efficiency gains of MVP are very promising, its learning difficulty is higher than that of standard flow policy. One reason is that our MVP requires modeling the mean velocity for any time interval specified by two time points (Geng et al., 2025a). A more significant reason is that its learning process governed by first-order ordinary differential equation (ODE) derived from the definition of mean velocity. However, this ODE theoretically suffers from the problem of multiple solutions due a lack of explicit boundary conditions, that is, the value at any boundary point not enforced. This poses non-trivial challenge to learning accuracy and consequently affects policy expressiveness (Birkhoff Langer, 1923).
To address this, we introduce instantaneous velocity constraint (IVC) to compensate for the lack of boundary conditions. Intuitively, IVC pairs the average velocity loss for each interval with an instantaneous velocity loss at the interval's start point. In practice, IVC is implemented as auxiliary policy loss, adding negligible computational overhead while materially improving accuracy. We evaluate our MVP on Robomimic (Mandlekar et al., 2021) and OGBench (Park et al., 2024), two demanding robot manipulation benchmarks. As shown Figure and Table 3, MVP achieves state-of the-art success rates while delivering substantial speed-ups in training and per-step inference over strong flow-policy baselines, on average across both suites.
Our contributions are summarized threefold:
We propose new flow-based policy, namely mean velocity policy (MVP), that enables fastest one-step action generation. By modeling the mean velocity field, MVP retains the expressiveness of generative policies while eliminating multi-step sampling overhead.
We design training enhancement technique, namely instantancous velocity constraint (IVC), to improve the learning accuracy of mean velocity field. This technique explicitly serves as boundary condition, thereby stabilizing learning and enhancing policy expressiveness.
We empirically achieve state-of-the the-art success rates on two challenging robotic manipulation benchmarks: Robomimic and OGBench. Moreover, our approach provides a substantial speedup in both training and inference over existing flow-policy baselines, highlighting its practicality for real-time application.
PRELIMINARIES
Reinforcement Learning. We consider agent interacting with an environment modeled as a Markov Decision Process (MDP), defined by a tuple M (S, A, P,r,7) (Li, 2023). The components are the state space S C R the action space A C Rm the state transition function P(s'|s, (6((ss,a) a), the reward function r(s, a), and the discount factor y E [o, 1). The primary goal in reinforcement learning (RL) is to learn policy (als) that maximizes the expected cumulative discounted reward, namely return, given by
20 Σ
TraKs an)
J. Er,P
Grounded in the off-policy learning paradigm, our approach utilizes an action-value function (Q- function) to guide policy improvement, which denotes the expected cumulative return for taking an action a in state and thereafter following the policy x.
(T a) ET.P
= s, a
Σ
(2)
The optimal action-valle function, Q* a), represents the maximum expected return achievable from state s by taking action a. The optimal policy x² can then be found by selecting the action that maximizes this function: x= (s) arg max Q" (s)
2

---

Published as a conference paper at ICLR 2026
Flow Matching. Flow matching is principled methodology for constructing continuous time generative models (Lipman et al., 2023). In contrast to diffusion models, which employ stochastic differential equations (SDEs) (Song et al., 2020), flow matching is built upon deterministic dynam- ics governed by an ordinary differential equation (ODE). By directly learning a continuous- instantaneous vector field, this approach simplifies the training objective and enables more efficient sampling. Specifically, it trains a neural network Ug R² [0, 1] + R to parameterize velocity field (x(()), t) that matches a predefined conditional target velocity v(1).
For source distribution (()())) and a target distribution p(z(1)), the velocity field is trained by minimizing the flow matching loss (Lipman et al., 2024):
(0) E
(- =(1) =(U)rg
o1 (z()), (z)()))))
where any intermediate point along the generating path x(t) = tI( + (1 - ()x(()) is a linear interpolation between source and target points. The velocity for this path is assumed to be constant vector (zx(t), = x(1) z00). This formulation defines the target vector field along a straight path between the source and target samples. Once trained, the learned field ve defines probability flow via the following ODE:
dx(t) = vo(x(t), t), z00)
(4)
dt
which allows us to effectively generate samples of the target distribution p, starting from samples of the source distribution q. Although flow matching is conceptually designed to generate along a straight line, the paths still curve in practice when fitting between two distributions (Song et al., 2023). Therefore, multi-step discretization and numerical methods like the Euler method are often required to solve the ODE in Eq. (4) to obtain high-quality generated results (Park et al., 2025).
METHOD
First, we introduce the mean velocity policy (MVP), showing how its integration with a "generate- and-select" mechanism enables a direct mapping from noise to optimal actions. We then present the instantaneous velocity constraint (IVC) and theoretically justify its role in improving the learning accuracy. Finally, the complete pseudo-code for our mean flow RI. algorithm is provided.
3.1 MRAN VELOCITY POLICY
In RL, policy T-|ss) defines distribution over actions given state s. For standard flow-based policies, this mapping is framed as a generative process: velocity model, v(a(t),,, s). transforms a standard Gaussian noise (source) into the optimal action (larget), with the state serving conditioning input. The final output action a(1) is calculated by
0-00 Lass
Ya.
Live
a(1) a ( o(a((), a(0) NOO, (5)
Figure 2: Velocity field: blue arrows de- note the mean velocity over time in- terval, with red arrows representing the instantaneous velocity at time point.
Unlike standard flow policies that learn the instantaneous velocity field v(a(t),t,s) s), we center on the mean velocity
field u u(a(t), ł, r, s). modeling the mean velocity over any given time interval rt
=) r f. c(a(F)
(6)
The relationship between these two types of velocities is shown in Figure If the mean velocity model u" is ideally learned, the policy inference is formulated
((1) a() nz (a(0), 0, 1,s,, a(0) A( T)-
(7)
3

---

Published as a conference paper at ICLR 2026
(1) How to imitate given target action. To train the mean velocity model u, first multiply both sides of Eq. (6) by (r - t) and then differentiate with respect to treatiing r as independent of t), which yields the mean flow identity:
w (Ma(t)),,,r + (r ou-02 d dt ( =
(8)
We assume there is a target optimal action a', serving as a(1) for imitation. For two randomly sampled time points, & and (with t < r), the intermediate point a(t) is defined by the linear interpolation: a(t) t a(1) + (1 t) a(0), and
a(0). Let denote the learnable parameters, the training objective is to minimize the residual of the mean flow identity in Eq. (8) as
d ( rt,,(() t,,, sg V (t (t) ( (o(a(())t. |.
(9) where "sg" denotes the stop-gradient operator to stabilize training. Calculating this training loss requires computing the total time derivative term, at u. Using the chain rule, this term is expanded as shown in Eq. (10), and its computation can be performed efficiently using the Jacobian-vector product (JVP) modern automatic differentiation libraries.
d dt " (M(()())) (,, v(a((),4,)) 4), Oa.
8
mo(a(()) r, (,, uo (o(a((),r,,,)))
(10)
Dt
By minimizing Eq. (9), the mean velocity model can effectively transform a standard Gaussian noise into the desired target action distribution. In RL, however, there is no ground-truv dataset of optimal actions to imitate. Next, we will detail how to gradually find better actions and eventually approach the optimal target action.
(2) How to find the optimal target action to imitate. Finding the optimal action a' directly is not realistic. However, we can use the Q-function to progressively find better actions as imitation targets, and eventually find the optimal action a" in a bootstrap manner. This mechanism called "generate- and-seeet or "best-of-N"..
In practice, at any given state s, the agent first generate N diverse candidate actions as a' at = + uo(c, 0, s,),, a'()) = c' ~ (O0 I). i = 1, N.
(11)
Then the critic function Q+ parameterized by is employed evaluate all candidates, and the action yielding the highest Q-value is identified as the final output action for that state:
arg max Qs)saa"). a²
(12)
We treat the combined mean flow generation process in Eq. (II) and the best-of-NV mechanism in Eq (12) as unified policy ryjnified or simply denoted as The resulting action, a", then serves three purposes: (1) interacting with the environment, (2) acting as the target action for policy training, and (3) calculating the target value for value training.
Although this bootstrap mechanism is intuitive, next we will formally prove that such imitation- based update guarantees policy improvement by the following theorem.
Theorem (mean velocity policy Improvement). Let the new policy knew be derived from an old policy wold via the N-candidate generative update process described above. Under assumptions of bounded Q-function error (eq), Lpshiith continuity of the Q-function, and a bounded mean flow matching error (EA), the performance difference between these two policies bower-bounded by
2eQ + +QEAA
V Tuer (s) - -V)-VTar (s) 2 E Eiaeetee
Σ o
Falt (s:)
(13)
1-7
fating error
BFN improvement term dn
where (ss) the state-value functiton, and Th" (s) is the best-of-N advantage gain, satisfying
A Nole (s) = E
max :
QT (s, an) V (s) 2
(14)
Proof. See Appendix A.
4

---

Published as a conference paper at ICLR 2026
This theorem decomposes the performance difference into two distinct components: BFN improve- ment term A1 and a firting error term Σg. As we prove in Appendix A.4, the best-of-N advantage gain (A ( in A is strictly non-negative, implying that the policy improvement can be driven by the benefit of sampling N diverse action candidates. It is also important to note that this improvement is partially counteracted by the fitting error term A2, which stems from the critic's 's inaccuracy (eq) and the mean flow matching error (EA). This highlights the importance of reducing e to further enhance policy performance. In the next subsection, we will introduce the instantaneous velocity constraint (IVC) to help reduce this fitting emror.
3.2 THE INSTANTANEOUS VELOCITY CONSTRAINT AS A BOUNDARY CONDITION
As we mentioned above, the underlying principle for training the mean velocity model is Eq. (8), which is a first-order ordinary differential equation (ODE) with respect to u. Mathematically speaking, we need to know both the dynamics and at least a boundary condition to accurately solve the unique solution of a ODE. Back to our practical setting, Eq. (8) only provides the dynamics given < r. Although sampling pairs where is very close to could implicitly serve as the boundary condition, such events are too rare to ensure robust training.
Inspired by this, we introduce the instantaneous velocity constraint (IVC), training objective that explicitly enforces a boundary condition at t. The mean velocity from to t is exactly the known instantaneous velocity a" a()), so the IVC objective is expressed as:
Live (f) = ax) [|uo(a(t), - v
(15)
To understand the effectiveness of this IVC objective, we first derive the following theorem, which demonstrates the multiplicity of solutions for Eq. (8) the absence of boundary condition.
Theorem 2 (Multiplicity of Solutions for the Mean Flow Identity). Let the cumulative error A. be defined as the difference between the learned and the true mean velocity field, Au(a(t),t,r) t.r) & wo uo(a(t), (,r) - u² (a(l), r)). Assume un perfectly satisfies the mean flow identity (Eq, (8)) for all t < r. Then, its cumulative error Du satisfies
C(a,r) An(a(1), r r-t
(16)
where C(a, r) is an integration constant independent of time
Proof. By assumption, both uy and u² precisely satisfy the mean flow identity. Subtracting the identity for u* from that of ug yields homogeneous linear differential equation with respect the cumulative error Au as
A_ + (t - A. 0.
(17)
By the product rule for derivatives, this equivalent to ar. [(r -t)A,] Integrating with respect to yields (r - t) = C(a,r) r), which completes the proof.
Theorem reveals that any solution learned by perfectly minimizing the LMF loss belongs to a family of functions with a unknown constant C(a, r). Because the LMF loss is blind to the boundary, it cannot provide a gradient to force C(a, r) to zero. This allows an arbitrary, persistent bias to exist in the learned ug. Next, we derive the following theorem to prove that the explicit introduction of IVC eliminates this degree of freedom and restricts the solution space to the unique correct u".
Theorem (Uniqueness via the Instantaneous Velocity Constraint). Let the boundary error Ag be defined as the error at the boundary t, (a(t),, t) & t,(((t),,,t)-e (a(t), t). Minimizing the IVC loss, Live = ||||| Au []2]], forces the integration constant C(a, in Theorem 2 to zero. This eliminates the boundary error and ensures the cumulative error Du vanishes for all < r.
Proof. The boundary error A. is the limit of the cumulative error Au as + tt_ Applying Eq. (16):
lim C(a,r)
A. (a(1))t)
(18)
r-t!! r
5

---

Published as a conference paper at ICLR 2026
This limit diverges if r(a,r) * 0. To keep the IVC loss finite, the optimization must prevent this divergence. which requires C(a,r) = 0. With C(a,r) = O, the boundary error Ap is zero. Consequently, by invoking Theorem 2, the cumulative error A.. also becomes zero. This completes the proof.
In essence, Theorem 3 proves that the IVC provides the necessary boundary condition make the learning problem well-posed. By forcing the eπror constant to zero at the boundary, the IVC eliminates the cumulative error inherent the mean flow objective, thereby retaining the high expressive power of the generative model. Moreover, the resulted smaller mean flow matching emor 6 in Theorem 1, which helps enable more effective policy improvement with each update.
3.3 MEAN FLOW REINFORCEMENT LEARNING
This section systematically presents the complete picture of our mean flow RL. The policy training loss Epolicy combines the mean velocity model loss in Eq. (9) with the IVC loss in Eq. (15):
Cpolicy (0) MM ()) + (O))
(19)
where the balancing hyperparameter À > 0 is called IVC coefficient, and the default value is 1.0.
Concurrently, the critic Qa is trained to minimize a standard TD-error in Eq. (20) on transitions (sk, 0k, 8k+1) from the replay buffer, where 1: is the step index.
Lo(
[Qp(sk) ax) (rx + 7Q d 1))
(20)
Recall that we treat the combined mean flow generation process in Eq. (11) and the best-of-N mechanism in Eq. (12) as a unified policy so the calculation of a also involves a generative- then-select process for ensuring an unbiased policy evaluation.
The overall training scheme decouples the generative policy's imitative training from the Q-value gradient. Instead, the policy improvement is guaranteed by a best-of-N mechanism under the guidance of Q-value selection. This design allows us to leverage the policy's 's full expressive power while ensuring stable and effective policy improvement. The complete algorithm is shown in Algorithm 1.
Algorithm 1 Mean Flow RL
Input: mean velocity policy Te. where 0 is the parameters of ug. Critic Qs. offline dataset Dotfline Initialize replay buffer 'D with D Dottine
b Phase 1: Offline Pre-training
for offline training step do Replay mini-batch from D
Update policy Ty by minimizing Epolicy (f) with Eq. (19) Update critic Q& by minimizing (o) with Eq. (20)
end for
Phase Online Interaction and Fine-tuning
for online training step k = 1,1,2 1,2.... do
Observe $g. execute at no (s (sw), receive s Tki and store (sg, at. Tk. Sk+1) into D
Replay a mini-batch from D Update policy my by minimizing frllicy (0) with Eq. (19)
Update critic Q by minimizing (0) with Eq. (20)
end for
EXPERIMENTS
4.1 MAIN EXPERIMENT
Benchmark.
We consider a total of sparse-reward robotic manipulation tasks with varying diffi- culties. This includes tasks from the Robomimic benchmark (Mandlekar et al., 2021), Lift, Can and Square, and tasks from OGBench (Park al., 2024), cube dobb--uubleeaask 2 2/3/4 4 and cube-triple-te
2/3/4 4. For Robomimic, we use the multi-hummn datasets. For OGBench, use the default play-style datasets. See more details of these tasks in Appendix C.

---

Published as a conference paper at ICLR 2026
Baselines and our method. We compare with three latest strong offline-to-online RL baselines.
(1) FQL (flow Q learning) (Park et al., 2025) first uses behavioral cloning to train a multi-step flow policy on offline data. It then trains separate one-step policy that imitates the multi-step policy and maximizes Q-values, enabling efficient and stable learning within the data distribution. (2) BFN (best-of-N) (Ghasemipour et al., 2021) combines the best-of-N sampling with expressive multi-step flow policy. Specifically, BFN first generates N candidate actions and picks the action (out of N) that maximizes the current Q-value. (3) QC (Q-chunking) (Li al., 2025) applies action chunking (Bharadhwaj et al., 2024) on the basis of BFN to improve exploration and sample efficiency. (4) Ours MVP also adopts the chunking trick, but leverages more efficient mean velocity policy to achieve the fastest one-step action generation with maintained even enhanced expressiveness.
Buccea hew
Nucees Hatt
Loop Abco
0.
0.0
8.0
8.8
(x10f)
(1:25))
(a) Rohomimic-lift
(b) Rohomimic-can
(c) Robomimic-square
Racemes wake
hood
0.0
1.0
1.0
Octof
(d) Cube-double-task2
(c) tube-double-tasks
(f) Cube-double-taskel
1.0
16
Suooee hal
Rucorea Cork
155
1.0
1.0
Slep (x10%)
Stape (x
Stage 04109)
(g) Lube-triple-tasee
(h) Cube-triple-tan task3
(i) Cube- Sube-triple-task/
MVP (ours)
Figure 3: Training curves on benchmarks. The solid lines correspond to mean and shaded regions correspond to 95% confidence interval over five runs. The shadow background indicates the offline training phase, while the white background indicates the online training phase.
Main results. As shown in Table our MVP matches exceeds state-of-the-art multi-step flow-matching baselines on eight of nine tasks. On the remaining task, MVP ranks second, with a performance of 0.92, which is just 0.02 points below the top-performing baseline's 's score of 0.94. While all methods achieve near- serr-perfect performance on simpler tasks like Robomimic-line Cube- double-task2, and tube-double-task3, MVP demonstrates clear superiority on the more challenging tasks. Specifically, MVP consistently outperforms all baselines on Robomimic-squaa
Cube-double- task4, and all Cube-triple tasks, where consistently achieves the highest success rates. For instance, on the most difficult task, Cube-triple-task4, MVP achieves success rate of 0.52 0.11, which is significantly higher than the best baseline. QC (0.46 + 0.13), and substantially exceeds both FQL and BFN. Overall, our MVP secures the top position with an average success rate of 0.88 + 0.05. This result highlights its strong capability that is competitive with multi-step flow policies in solving long-horizon, sparse-reward tasks.
7

---

Published as a conference paper at ICLR 2026
Table 1: Success rates, Mean Std over 5 seeds. Bold best, underlined 2nd--best.
Task
FQL
BFN
QC
MVP (ours)
Robomimic-lift
0.96 + 0.03
1.00 = 0.01
1.00 + 0.00
1.00 0.00
Robomimic-can
0.74 0.11
0.82 0.03
0.94 = 0.06
0.92 = 0.07
Robomimic-square
0.12 0.05
0.34
0,92 + 0.01
0.93 = 0.01
Cube-double-task2
0.95 + 0.04
0.88 +.05
1.00 + 0.00
1.00 - 0.00
Cube-double-taske
0.97 0.04
0.00
1.00 = 0.00
1.00 0.00
Cube-double-taske
0.08 + 0.04
0.09
0.93 - 0.08
0.95 = 0.04
Cube-triple-task2
0.01 0.02
0.06
0.82 + 0.10
0.88 0.03
Cube-triple-task3
0.12+
0.26
0.69 + 0.05
0.71 + 0.06
Cube-triple-taske
0.00 0.00
0.02
0.46 + 0.13
0.52 = 0.11
Average
0.44 + 0.05
0.52 + 0.06
0.86 + 0.05
0.88 0.05
4.2 ABLATION STUDY AND SUPPLEMENTARY EXPERIMENTS
(1) Ablation on the instantaneous velocity constraint (IVC). We perform an ablation study on the ICC coefficient X. Our full version (A 1.0) was compared against variants with reduced constraint (A = 0.5) and without the constraint (λ = 0.0). The results, as shown in Figure 4, indicate positive correlation between the IVC weight and performance, while also demonstrating that the method not overly sensitive to A. For example, the success rate on the challenging Cube-triple-taske significantly increases from 0.30 = 0.21 (with no IVC) to 0.45 = 0.15 (with a partial IVC), and further to 0.52 0.11 (with full IVC). Detailed numerical results are listed in Table in Appendix B. These findings empirically validate our theoretical claims, confirming the IVC's role as a crucial component for modeling an accurate mean velocity field and consequently achieving significant performance gains.
Sucoes Rate
Succuss Rate
(A=0.0)
0.5
0.5
MVP (A-0.5)
MVP (A=1.0)
2.0
20
Steps (x10*)
Stepe (x10*)
(a) Cube-triple-task3
(b) Cube-triple-tssk4
Figure 4: Training curves of ablation the IVC.
(2) Comparison with one-step variants of the aforementioned baselines. We compared our MVP against one-step variants of the aforementioned baselines: FQL-Onestep BFN- BFN-Onestep, and QC-Onestep. As shown Figure 5, a naive one-step configuration insufficient for solving these complex, long-horizon tasks, with baselines achieving success rates near zero on both Cube-triple- task3 and Cube-triple-taske In stark contrast, our MVP achieves success rates of 0.71 = 0.06 and 0.52 + 0.11 on these tasks, respectively. Detailed numerical results can be seen in Table in Appendix B.1. This supplementary comparison highlights that simply using a one-step standard flow is not enough; the superior expressive capability and stable learning process of our mean velocity policy critical for tackling these challenging long-horizon manipulation tasks.
(3) Training and inference time analysis. Figure/ I presents comparison of the average success rate (%) versus online training speed (iter/s) across the all 9 tasks. Our MVP achieves highest success rate and fastest training speed. This superior training efficiency stems directly from our one-step action generation, which eliminates the expensive iterative sampling process required by prior multi-step flow policies.
8

---

Published as a conference paper at ICLR 2026
1.0
1.0
-nestep
Suocuss Rude
Success
0.5
0.5
BFN OFN-Onestep
Or-Onnetepp
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.5
2.8
MVP (ours)
(X10*)
(x10f)
(a) Cube-triple-task3
(b) Cube-triple-task4
Figure 5: Training curves of comparison with one-step flow.
Table 2: Comparison of online training speed
Metric
FQL
BFN
QC
MVP (ours)
Average
108.5 + 7.7 iter/s 68.0 + 5.8 iter/s 92.6 + 6.3 iter/s
153.6 11.5 iter/s
Regarding inference time-efficiency, conducted evaluation with focus its suitability for real-world robotic deployment. Since robotic platforms often have limited computational resources, our experiments were conducted on CPU-only environment, AMD Ryzen Threadripper 3960X 24-Core Processor. To simulate a more realistic deployment scenario without hardware acceleration, disabled JAX's Just-In Time (JIT) compilation during all evaluations.
The results are listed in Table 3. our MVP and FQL exhibit very similar inference times, with both approaches being significantly faster than BFN and QC.
Why BFN and QC are slow. The poor performance of BFN and QC is primarily because they rely on a 10-step flow policy, which requires iterative computation to transform noise into an action.
Why FQL is fast. FQL simultaneously learns both a 10-step flow policy for high-accuracy imitation and a separate one-step flow policy. The one-step policy is obtained through a distillation process combined with a Q-function maximization loss, which allows FQL achieve high inference speed comparable to our MVP.
Why MVP is better than FQL comprehensively. Despite FQL's relatively fast inference, its training process is very slow due the involvement of multiple policies, including multi-step flow policy. Furthermore, benchmark results indicate that its success rate is very low, averaging only half of our MVP's. When considering FQL's overall low success rate and slow training speed, MVP still maintains a significant advantage.
Table 3: Comparison of inference time
Metric
FQL
BFN
QC
MVP (ours)
Average
10.76 1.02 ms
117.3 + 13.23 ms
113.22 + 11.92 ms
10.93 + 0.95 ms
RELATED WORK
Witline-to-online RL. The offline-to-online RL first uses a static, pre-collected dataset to offline train initial policy and Q-value functions (Levine et al., 2020; Kumar et al., 2020). This process provides a warm start, giving the agent a foundational understanding of the environment that significantly accelerates and improves the efficiency of subsequent online interactive fine-tuning (Lee al., 2022). Numerous algorithmic designs have been proposed to improve offline-to-online RL, including behavioral regularization (Ashvin al., 2020; Tarasov al., 2023), conservatism (Kumar al., 2020),

---

Published as a conference paper at ICLR 2026
in in-sample maximization (Kostrikov et al., 2022; Garg et al., 2023), out-of-distributend detection (Yu et al., 2020; Kidambi et al., 2020; Nikulin et al., 2023) and dual RL (Lee et al., 2021; ;; Sikchi et al., 2024). Beyond these algorithmic solutions, the choice of policy function also plays an important role (Wang et al., 2023). A policy network with high expressiveness can better capture the intricate distribution of the behavioral policy during offiine training stage. This capability is also crucial for adapting to target environments during online fine-tuning, as optimal actions in these settings often possess a naturally multi-modal nature (Park et al., 2025; Li al., 2025). Since the policy must infer actions at every step during both the online fine-tuning and real-time deployment stages, the inference time efficiency of the policy function is also critical (Li, 2023; Park al., 2025). Our work contributes a new policy function, MVP, which achieves the fastest single-step action generation and maintains high expressive capacity to achieve high performance.
Generative models as RL policies. The expressive power of generative models, such as denoising diffusion models (Ho et al., 2020) and flow matching (Lipman et al., 2022), makes them promising for representing complex, multi-modal policies in both offline (Janner et al., 2022: Chi et al., 2023) and online (Yang et al., 2023; Wang et al., 2024a; Ding et al., 2024) RL. However, their iterative sampling process requires high number of function evaluations (NFE), creating prohibitive latency for the tightthroughput nature of online RL. A common approach in RL to improving time-efficiency of generative policies is distillation, which compresses trained iterative model into one-step policy (Wang et al., 2024b; Park et al., 2025). Beyond the field of RI., recent studies have begun to explore training a single-step generative flow directly for tasks like image generation (Lu & Song, 2025). These methods typically operate either by enforcing consistency constraints on the model's outputs at different time steps (Song et al., 2023; Song & Dhariwal, 2024; Geng al., 2025b) or by explicitly modeling the flow velocity over a specific time interval (Boffi et al., 2024; Frans et al., 2024). The latter class of methods generally requires more iterations to train, but ultimately performs better (Frans et al., 2024). A typical representative, mean flow, has achieved the best one-step fitting performance on Imagenet (Geng et al., 2025a). We propose a new policy function, MVP, which combines mean flow with RL to enable the fastest single-step action generation. In the context of RL, the dynamic shifts in data distribution during sampling place higher demands on imitation based flow matching training. To address this, our proposed IVC serves an explicit boundary condition during MVP training, which reduces fitting error and consequently ensures strong policy expressiveness.
CONCLUSION
We propose the mean velocity policy (MVP), a new generative RL policy that enjoys both high time-efficiency and expressiveness. The former stems from the fastest one-step action generation, and the latter contributes to our designed instantaneous velocity constraint (IVC), which explicitly serves necessary boundary condition and reliably improves the learning accuracy. Empirical results on the Robomimic and OGBench benchmarks confirm that our MVP achieves state-of-the-art success rates and offers substantial improvements in training and inference speed. We believe that this work represents a significant step towards developing highly expressive and efficient policy functions for complex robotic control tasks. Regarding limitation, primary one is the additional GPU memory consumption during training, which stems from the Jacobian- Vector Product (JVP) operation. In future work, we plan to validate our method on more tasks and real robotic platforms.
7
ACKNOWLEDGMENT
This study supported by the Tsinghua University-Toyodt Joint Research Center for AI Technology of Automated Vehicle, Beijing Natural Science Foundation (I.257002), and SunRisingAl Lab.
REFERENCES
Nair Ashvin, Dalal Murtaza, Gupta Abhishek, and L Sergey. Accelerating online reinforcement learning with offline datasets. CoRR, vol. 0ab//2006.09351 2020.
Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Ku- mar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations
10

---

Published as a conference paper at ICLR 2026
and action chunking. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 4788 4795. IEEE, 2024.
George D Birkhoff and Rudolph E Langer. The boundary problems and developments associated with system of ordinary linear differential equations of the first order. In Proceedings of the American Academy of Arts and Sciences, volume 58, pp. 51-128. JSTOR, 1923.
Nicholas M Boffi, Michael S Albergo, and Eric Vanden-Eijndee Flow map matching, arXiv preprins S0Mi:22406.055077 2, 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel. Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal Robotics Research, pp. 02783649241273668, 2023.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion- based reinforcement learning via q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945-53966 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arViv preprint arXiv:2410. 2024.
Divyansh Garg, Jocy Hejna, Matthicu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In The Eleventh International Conference Learning Representations, 2023.
Zhengyang Geng, Mingyang Deng, Xingjian Bai, J. Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling, 2025a. URL hitps: /awxiv.vvv...//.io/...///aavavvt org. htrrg/abs/250000 13447.
Zhengyang Geng, Ashwini Pokle, Weijian Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. In The Thirteenth International Conference Learning Representations, 2025b.
Scyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-mas q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
Yuxuan Jiang, Yujie Yang. Zhiqian Lan, Guojian Zhan, Shengbo Eben Li, Qi Sun, Jian Ma, Tianwen Yu, and Changwu Zhang. Rocket landing control with random annealing jump start reinforcement learning. In 2024 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 14026-14033. IEEE, 2024.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proc. 19th International Conference Machine Learning. Citeseer, 2002.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 11810-21823, 2020.
Nikita Kornilov, Petr Mokrov, Alexander Gasnikov, and Aleksandr Korotin. Optimal flow matching: Learning straight trajectories in just one step. Advances in Neural Information Processing Systems, 2024.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1 199119999991111 2020.
Jongmin Lee. Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference Machine Learning, pp. 6120-6130. PMLR, 2021.

---

Published as a conference paper at ICLR 2026
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-onling reinforcement learning via balanced replay and pessimistic Qeensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement leamming Tutorial, review, and perspectives open problems. arXiv preprint urXiv:2005.01643, 2020.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint FrXx:25507007966 2025.
Shengbo Eben Li. Reinforcement Learning for Sequential Decision and Optimal Control. Springer Verlag, Singapore, 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Nem-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint 1rXiv:24/2.062644 2024.
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. In The Thirteenth International Conference on Learning Representaatioss, 2025.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni. Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.0322 2021.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International conference on machine learning, pp. 2-2244.. PMLR, 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. ArXiv, 2024.
Scohong Park, Qiyang Li, and Sergey Levine. Flow Q-learning, arXiv preprint arXiv:2502.02538, 2025.
Sikchi, A Zhang, and Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. In International Conference on Learning Representations. International Conference on Learning Representations, 2024.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint D.Xiv:2011.13455 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. International Conference Machine Leurning, pp. 3221 1221123 PMLR, 2023.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 36:11592-11629 2023.
Yinuo Wang, Likun Wang, Yuxuan Jiang. Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 77:54183-542044 2024a.
12

---

Published as a conference paper at ICLR 2026
Yinuo Wang. Mining Tan, Wenjun Zou, Haotian Lin, Xujie Song, Wenxuan Wang. Tong Liu, Likun Wang, Guojian Zhan, Tianze Zhu, et al. Enhanced dacer algorithm with high diffusion efficiency. arXiv preprint arXiv:2505.23426 2025.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations. 2023.
Zhendong Wang. Zhaoshuo Li, Ajay Mandlekar, Zhenjia Xu, Jiaojiao Fan, Yashraj Narang, Linxi Fan, Yuke Zhu, Yogesh Balaji, Mingyuan Zhou, et al. One-step diffusion policy: Fast visuomotor 2024b.
policies diffusion distillation. arXiv preprint
Long Yang. Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305. 13122, 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances Neural Information Processing Systems, 33:14129 -41422 2020.
Guojian Zhan, Yuxuan Jiang. Shengbo Eben Li, Yao Lyu, Xiangteng Zhang, and Yuming Yin. A aaassfrmmation-gggrggatoo framework for state representation of autonomous driving systems. IEEE Transactions Intelligent Transportation Systems, 25(7):7311 (170))3111-33331 1--322, 2024.
Guojian Zhan, Xin An, Yuxuan liang, Jingliang Duan, Huichan Zhao, and Shengbo Eben Li. Physics informed neural pose estimation for real-time shape reconstruction of soft continuum robots. IEEE Robotics and Automation Letters, 2025.
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, arXiv preprint arXiv:2311.01223, 2023.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS ON THE MEAN VELOCITY POLICY IMPROVEMENT
In this section, we provide the detailed thcoretical analysis for the policy improvement guarantee of training paradigm. It includes the implementation procedures of policy update detailed in A.I, core assumptions and useful lemma in A.2, the formal proof of the mean velocity policy improvement theorem in A.3, and three key properties of the improvement gain in A.4.
A.1 IMPLEMENTATION PROCEDURES POLICY UPDATE
Let Told denote the base mean velocity policy prior to an update. At the same time, we have learned critic, Qp(s, a)), which is an approximation of the true action-value function of the base policy, QTa (s, a). Let nnew denote the updated policy, which is derived from Vold through the following step generative process for any given state s:
1. Sample: Generate N candidate actions
aNi by sampling from the base policy, i.e.,
a; (-ss)
2. Seleet: Use the learned critic Qo to select the best action from the candidates, which becomes the target action a" (s)): a' (s) = arg max Qo(s,
3. Match: The new policy Tnew is obtained by training the parameters of Vold to match the target action a" (s). Note that an action Onew ~ Tnew is not guaranteed to perfectly match a" (s(ss. We formally bound the expected distance between them in Assumption 3.
For analytical clarity, we introduce a conditional matching distribution, M(-la"), to model the outcome of the matching process (Step 3). Specifically, we assume the specific target a" has been selected in the previous Steps 1 and 2, then generation the final action anew is expressed as
Onew ~ M (-)a").
The difference between M (-la") and Nnww that M (-la") describes the action distribution conditioned specific target action a", whereas Tnew represents the marginal distribution of the action, which results from averaging over all possible targets a" (s) generated in Steps and
A.2 CORE ASSUMPTIONS AND USEFUL LEMMA
Our analysis relies the following assumptions and lemma.
Assumption (Bounded Q- Valuuue Fitting Error). The error between the learned critic Qs and the true Q-function of the base policy QTT is uniformly bounded by constant EQ 2 0.
((s,a) SS
[Q((s,a) - (s, EQ.
Assumption 2 (Qluelue Smoothness). The true Q-value of the base policy, QTW a), is Lo Lipschitz continuous with respect to the action a.
Vs E S, a1 0g E A, (s, a1) - Q (s. & LQ d(a1,a2)
Assumption 3 (Bounded Mean Flow Matching Error). The expected distance between the action anew generated by the new policy and the target action a" is bounded by constant ć 2 0.
Vs Es B (-Ia") (dleaaa)d((aaea [d([nev a" (s)) S AAA
Lemma (Performance Difference Lemma (Kakade & Langford, 2002)). Let M
be Markov Decision Process, and let π. and n' be two arbitrary policies. The difference in the value function a starting state s, or denoted as su, can be expressed in terms of the advantage function of the new policy n' with respect to the old policy n
J - -V= = Egnay' .amn'' 8)
Σ 1=0
AT A
14

---

Published as a conference paper at ICLR 2026
where d= (s) is the discounted state visitation distribution under policy x', and (s, as - (s) is the advantage function of policy π.
We start with the Proof. The proof relies on the key identity V=(s) [(as) definition of the value difference for state s:
V* (8) V (8) V"
Σ (s.
E (=. (*
aEA
aEd
-V (s)
Q"
Σ (= - (*))))
aEA
aEA
VT (8) n' Q"aal=)0"
Σ (s(als)AA(ssa))
aEA
aEA
Using the Bellman expectation equation for V (s):
V (8) Σ
r''''s)
R(s,a') + M P(dis,'')vv(())
a'EA
#'ES
And the definition of (s)
(s) R(s,a) Σ P(d(s,a)))(1))
8'ES
Substitute these into our first equation:
v" (s) - (s)
= Σ (d'lss
+
(()) (r((0)(()))) + Σ ('d''s))A"ssaa")
d'EA
s'ES
a'EA
"E x'('sss)
7 Σ 5 (8) + (1o x'ES
a'EA
(H)) (8') ("'))) (e)x (s.
This recursive formula shows that the value difference at state depends on the expected value difference at the next state s'. By recursively unrolling this expression over time and averaging over the state visitation distribution dt' (s), we arrive at the final result.
/" (s) - V= = E Egnayr amat' a)
Σ A''A 1=0
This completes the proof.
A.3 PROOF OF THE MEAN VELOCITY POLICY IMPROVEMENT THEOREM
This subsection provides formal proof of the mean velocity policy improvement theorem, that is, Theorem the main text.
Proof. Our proof begins by invoking the Performance Difference Lemma, i.e., Lemma 1, which yields
Faee (s) Foll (s) EE Buraa
Σ Ao
(st, ai)
(21)
This equation connects the improvement in value function to the expected advantage of the policy's 's actions, measured with respect to the old policy.
The following proof proceeds in two parts: first, derive the single-step advantage bound Eg (s,) [A (s, at) and second, we aggregate them over time to obtain the required bound of value difference, which completes the proof.
15

---

Published as a conference paper at ICLR 2026
Part I: Lower bound the single-step expected advantage We begin by analyzing the Q-value decay due to the imperfect matching for given target action a" The final action Onew is distributed according to M-la"). We can bound the expected difference as follows:
QTIe (s, a²)
[QT Gnew
BBas [QTHe( (8, aa" QTwe (s, Gnew) ]]
BBan [1Q (s,a")
Q (5, anew)]
(by Jensen's incquality)
Baan L d(a", Onew)]
(by Q-Value Smoothness, Assumption. 2)
SLQCA
(by Bounded Matching Error, Assumption 3)
Rearranging gives us the following bound for given a":
M( (a-) (QT (s, Onew
2
LQEA-
Now, we take the expectation over the distribution of target actions a" (s) to get the bound for the marginal policy Tnew:
E 1Qaaa a)] = BBaa)~-Te M( (QT anew]]]
2 Baa.)-ma [QTH0 (s, a²)] - LQEA-
Next, we relate the Q-value of the target action, TTM (s, a²), back to the Q-values of the original candidates {a:):
QTTn (s, a") 2 Q0(8, - =e
(by Bounded Q- Fitting Error, Assumption 1) (by definition of a*)
max i=
Qo(s, az) CQ
2 max EI...,,
(s, ai) - 2EQ-
(by Assumption on each candidate a:)
Substituting this result back into our bound for Enwwi TQ (s, a)}, we get:
E ENNTER Taaass (s, a)) 2 Baaibostas
max il.....
(5, 2EQ LQ
Finally, subtracting Vad (8) Eag Quiad (s, from both sides gives the desired single-step advantage bound:
Egwttn [A Aa (s, a)] 2 (r 2cq - LOCA.
where A (8) ) is the best-of-N advantage gain, defined as
E EaI....aN
max I,...,N
Q (8,
Val
Part 2: Extension to the value function difference lished, we substitute it into Eq. (21) and obtain:
With the single-step advantage bound estab-
V (8) V (s)
E Sornaee Tenn
Σ 1=0
r A (sc, ac)
0
Eng (aan()))))))) [ATin (Am)) at)]]
(by law of total expectation)
t=0
Aa (st) -EEQ LQEAl
(by substituting the bound from Part 1)
t=0
2rq + LQCA 1-7
=B
Σ
EA'A (si)
(by rearranging the geometric series)
1=0
This completes the proof.
16

---

Published as a conference paper at ICLR 2026
A.4 KEY PROPERTIES THE BEST-OF-N ADVANTAGE GAIN
The term A N (s) has several important properties:
Non-negativity: A Tuid (+) 2
2. Monotonicity with N: N+1 (8) 2 A AN
Tad
3. Special case (N = 1): A Tal (S) = O.
Proof. Our proof sketch is to first rewrite A ( as integral involving the cumulative distribution function (CDF) of the Q-values, and then demonstrate the claimed properties of the rewritten form.
Let us define random variable X representing the candidate Q-values. For a given state s. the randomness of X is induced by the candidate action a independently sampled from the base policy:
X Q Tall (s, a), where (-aa(-18))
Recall that the definition of the best-of-N advantage gain is
E a
Q ai) i=1....N
max
Vala (s).
We can equivalently describe this definition using the CDF notation of the random variable X.
Let Fx (z) be the CDF of X, and XI...., Ny be N independent and identically distributed (i.i..)) samples of X, and let Yv max(X,,
Xx). The CDF NN is FYw (v) [F)( Therefore,
we have A Tai (s) = [[[[] ] - E[X].
Moving forward, since the expectation of random variable X in terms of its CDF satisfies E[X] [o (1 Fx (dx dx - _ Fx (x))dr,, we can further rewrite A Thu (s) as:
A (s) = (1 (=)) S FYw (z)dx)
( (1-((()dd - f Fx Fx(x)d
-1, (1 -x=x(x))d
(Fx(x) - Frr (x))
Substituting FyN (x) [x we arrive at the final rewritten form:
Fall /
(Fx(x) [Fx (x))
Next we use this form to prove the three key properties of AT
1. Proof of non-negativity (A TaT (s) 2 0)
The CDF Fx (x) takes values in the range [o, 1]. For any value p E [0, 1] and any integer > 1, we have D < p. Therefore, the integrand Fx [F'x (x)] is always greater than equal to zero for all x. The integral a non-negative function is non-negane non-negative. Thus, A Tau (s) 2 0.
2. Proof of monotonicity with N (A N > A (s))
We examine the difference between consecutive terms using the integral form:
(xx - (Fx Ido L (Fx(x) - (xx))))
Rad (s)
Fou
([Fx(T)) - [Fx(
[Fx())) (1-x((ddddd
Since (x) [0, 1]. both terms in the integrand, [Fx (x)] and (1 Fx(x)), non-negative. Their product is therefore non-negative. The integral of this non-negative function is non-negative, which implies A Feu (s) T (s) 2 0.
17

---

Published as a conference paper at ICLR 2026
3. Proof of special case (AT 0)
Substituting N = into the integral identity yields:
AT = _ -Fx(x))[Fx((x) (Fx()))de / (Fx(x) ( -x(e))d
/ -00
Od =
This confirms the special case.
B SUPPLEMENTARY RESULTS
B.1
NUMERICAL RESULTS OF ABLATION STUDY
Table 4: Ablation on the impact of IVC.
Task
MVP (A = 0.0)
MVP (A = 0.5)
MVP (A= 1.0)
Cube-triple-task3 tube-triple-tas44
0.65 + 0.05
0.70 + 0.14
0.71 + 0.06
0.30 = 0.21
0.44 + 0.08
0.52 + 0.11
Table 5: Comparison with one-step variants of the aforementioned baselines.
Task
FQL-Onestep
BFN-Onestep
QC-Onestep
MVP (ours)
Cube-triple-task3
0.00 = 0.01
0.00 + 0.00
0.02 + 0.03
0.71 + 0.06
Cube-triple-taske
0.00 + 0.00
0.00 + 0.00
0.01 + 0.01
0.52 + 0.11
C ENVIRONMENTS DESCRIPTION
Robomimic benchmark. We use three challenging tasks from the robomimic domain (Mandlekar et al., 2021). We use the multi-human datasets that were collected by six human operators. Each dataset consists of 50 trajectories provided by each operator, for total of 300 successful trajectories. The operators were varied in proficiency there were 2 "worse" operators, "okay" operators, and "better" operators, resulting in diverse, mixed quality datasets. The three tasks are described below.
(1) lift: requires the robot arm to pick small cube. This is the simplest task of the benchmark.
can: requires the robot
pick up coke can and place in a smaller container bin.
(3) square: requires the robot arm to pick square nut and place it on rod. The nut is slightly bigger than the rod and requires the to move precisely complete the task successfully.
All of the three robomimic tasks use binary task completion rewards where the agent receives -1 reward when the task not completed and 0 reward when the task is completed.
OGBench cube-double /triple: These three domains contain 2/3 cubes respectively. The tasks in the two domains all involve moving the cubes to their desired locations. The reward is -fwrong where fhwoogg is the number of the cubes that at the wrong position. The episode terminates when all cubes are at the correct position (reward is 0).
Below highlight three representative tasks from each environment:
(4) Cube-double-task2 (move): Two cubes in different colors are initialized at (0.35, -0.1, 0.02) and (0.50, 0.1, 0.02), and the goal is move them to (0.35, 0.1, 0.02) and (0.50, 0.1, 0.02).
(5) Cube-double-task3 (move): The initial cube positions are (0.35, 0.0, 0.02) and (0.50, 0.0, 0.02). and they must be placed (0.425, -0. 0.2, 0.02) and (0.425, 0.2, 0.02).
(6) Cube-double-tssk (swap): Two cubes start from (0.425, -0.1, 0.02) and (0.425, 0.1, 0.02).
and the objective is swap their positions to (0.425, 0.1, 0.02) and (0.425, -0.1, 0.02).
18

---

Published as a conference paper at ICLR 2026
(a) Lift
(b) Can
cc) Square
(d) Cube-double-task2
(e) Cube-double-task3
(f) Cube-double-taske
(g) Cube- tea--trppeelaske
(h) tube-triple-taske
(i) sabe-triple-tan
Figure Snapshots the challenging long-horizon, searse-reward manipulation tasks.
(7) Cube-triple-task2 (move): Three cubes are initialized at (0.35, 002,, 0.02), (0.35, 0.0, 0.02), and (0.35, 0.2, 0.02), with goals at (0.50, 0.O, 0.02), (0.50, 0.2, 0.02), and (0.50, -0.2, 0.02).
(8) Cube-triple-task3 (stack): A stacked tower of three cubes is initialized (0.425, 0.2, 0.02). (0.425, 0.2, 0.06), and (0.425, 0.2, 0,10), and the robot must relocate them to (0.35, -0.1, 0.02). (0.50, 0.2, 0.02), and (0.50, 0.0, 0.02), respectively.
(9) Cube-triple-task4 (swap): Three cubes are initialized at (0.35, 0.0, 0.02). (0.50, 0.1, and (0.50, 0.1, 0.02), and they must be cyclically rearranged to (0.50, -0.1, (0.50, 0.1, and (0.35, 0.0, 0.02).
These tasks highlight increasingly complex spatial reasoning requirements, ranging from coordinated multi-cube relocation to disassembly of stacked structures and cyclic rearrangement.
19

---

Published as a conference paper at ICLR 2026
D VISUALIZATIONS
This section provides supplementary visual material to demonstrate our model's performance. The included visualizations of successful episodes highlight policy's ability to generate precise and robust trajectories, showcasing its effectiveness in handling the complexities of long-horizon reasoning and sparse rewards inherent in these robotic manipulation tasks.
(a) Lift
(b) slep 5
(c) slep 10
(d) slep 15
(e) slep 20
(I) Can
(g) step 15
(h) step 30
ii) step 45
i step 65
(k) Square
(I) step 20
(m) step = 40
(n) step 60
(o) step 70
(p) Double-task2
(g) step 15
(r) step 30
(s) step 45
(l) step = 60
(u) Triple-task2
(v) slep 20
(w) slep 30
(x) slep 45
(y) slep 50
Figure 7: Visualizations of typical success episodes: Robomimic-lift, Robomimic-can, Robomimic-square, Cube-double-task2 and 3ube-double-task3.
20

---

Published as a conference paper at ICLR 2026
(a) Double-task4
(b) step = 40
(c) step 75
(d) step = 120
(c) step = 155
Triple-task2
g step 30
(h) step 60
(i step = 90
i step 125
(k) Triple-task3
(I) step 40
(m) step = 75
(n) step = 120
(0) step = 165
Triple-task4
(g) step 25
(r) step = 50
(s) step = 75
(t) step = 100
(u) step = 125
(V) step = 150
(w) step = 175
(x) step 200
(y) step = 235
Figure 8: Visualizations of typical success episodes: Cube-double-task4 Cube-triple-task2, Cube-triple-
task3, and Cube-triple-task4.
E REPRODUCIBILITY STATEMENT
The hyperparameters of all algorithms are shown Table
21

---

Published as a conference paper at ICLR 2026
Parameter
Value
Shared
Batch size
256
Discount factor (r)
0.99
Optimizer
Adam
Learning rate
3 x 10-4
Target network update rate (r)
5 x 10-3
UTD Ratio
Evaluation interval
5000
Number of evaluation episodes
50
Number of offline training steps
10th (IM)
Number of online training steps
10 (1M)
Number of flow steps (T)
10
Policy network width Policy network depth Policy activation function Policy layer normalization
512
4 hidden layers GELU False
Value network width
512
Value network depth
4 hidden layers GELU
Value activation function
Value layer normalization
True
Value ensemble size (K)
Value ensemble operator
MEAN
FQL
Flow step
10
10000 for lift, can and square 300 for cube-doubl double------eo and cube-triples
BC weight (a)
QC
Chunking horizon length Flow step
10
16 for lift, can and square 32 for cube-double- and cube-triple-+
Number of best-of-N
MVP-JVC (ours)
IVC ratio (A)
1.0
for lift, can and square 32 for cube-double- citooubbeesee and cube-triple-+
Number of best-of-N
Table 6: Detailed hyperparameters.
22

---

Published as a conference paper at ICLR 2026
F LLM USAGE DISCLOSURE
We used ChatGPT to refine the grammar and improve the clarity of the text. All LLM-generated suggestions were reviewed and edited by the authors, who take full responsibility for the final content.
23


Reviews:
[{"review_id": "BuHSMx4wIJ", "reviewer": "Reviewer_Ruko", "summary": "This paper introduces the Mean Flow Policy (MFP) which is an improvement for flow-matching based policies using the idea of mean flows. The key insight is that in traditional flow policies, one has to integrate the instantaneous velocity from 0 to 1 to transform from the standard normal distribution to the output distribution which selects the action. Instead of learning instantaneous velocities (v(t)) and then performing numerical integration, this paper shows that you can learn a mean-flow (u(t,r)) which is the velocity from t to r. This allows one to get the target distribution in one step as opposed to the several needed in typical flow matching papers. To ensure accurate learning of this mean velocity, the authors propose an Instantaneous Velocity Constraint (IVC), which serves as a boundary condition to eliminate solution ambiguity in the mean flow ordinary differential equation. Theoretically, the paper proves that IVC ensures a unique and accurate solution and that MFP guarantees policy improvement under bounded fitting errors. Empirically, MFP achieves state-of-the-art success rates but provides training/inference speedups on two standard simulation benchmarks.", "questions": "1. The paper’s main claim is that Mean-Flow Policy achieves faster inference than standard flow or diffusion policies. However, the runtime comparisons are only briefly mentioned in the appendix. Could you provide more details on the experimental setup for these timing results e.g., hardware used, batch size, and number of integration or denoising steps for baselines — and consider moving a quantitative runtime table into the main text?\n\n2. The reported environments (Robomimic and OGBench) seem relatively saturated, with strong baseline performance. Have you considered evaluating on more complex or long-horizon benchmarks to better stress-test the method’s benefits? If not, could you explain the rationale behind choosing these two domains?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a strong paper that introduces a new formulation for flow matching, which is prevalent tool in robot learning. A major pain point for flow-matching or diffusion style approaches is the slow inference times. We need several denoising steps or numerical integration, that greatly slows down inference and the rate at which such policies can be applied. Unlike \"hacks\", this paper presents a clean reformulation using the ideas of mean-flow to reduce the inference time. The theoretical analysis is strong. I expect this to be the starting point for several follow-up ideas that can greatly improve the empirical performance as well.", "weaknesses": "An unfortunate (IMO) weakness of the paper is the limited empirical results. It uses only two simulation environments and the improvements are minimal over the baselines. In most cases, it matches or only slightly exceeds the baseline. In a way that's not surprising to me. The claim isn't that this is a better method (in terms of higher success rates) than baselines but that it is a faster method (during inference). However, the inference time comparisons are only in the appendix with some short description in the main body. I would hav liked to see more discussion on that aspect. \n\nAnother issue is that the two simulation seem \"saturated\" in that the baselines already perform pretty well. It would have been nice to look at more complex benchmarks.\n\nIt would have been especially nice to see how fixing the inference time (for example) results in better/worse performance for the several baselines. For diffusion style methods, it is not uncommon to perform comparisons with varying the number of denoising steps (which controls for the inference time). An equivalent study here would have really benefitted the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ezCUOBq5dP", "reviewer": "Reviewer_aBKZ", "summary": "This paper introduces the Mean Flow Policy (MFP), a novel generative policy for reinforcement learning that is both highly expressive and computationally efficient.\n\nThe core idea is to represent the policy using MeanFlow [1]. As MeanFlow is trained via supervised learning, it is incompatible with standard reinforcement learning frameworks. To bridge this gap, the proposed method samples N actions from the policy, selects the action with the highest Q-value, and uses this optimal action as a supervised target to train the MeanFlow policy.\n\nFurthermore, to ensure stable and accurate learning, an Instantaneous Velocity Constraint (IVC) is introduced. This constraint imposes a necessary boundary condition on the underlying ordinary differential equation, which stabilizes the training process and enhances model fidelity.\n\nEmpirically, MFP achieves state-of-the-art success rates on several challenging robotic manipulation benchmarks (Robomimic and OGBench), while demonstrating significant computational speedups over existing multi-step, flow-based baselines.\n\n[1] Geng, Zhengyang, et al. \"Mean flows for one-step generative modeling.\" arXiv preprint arXiv:2505.13447 (2025).", "questions": "- In the MeanFlow paper [1], the authors train the MeanFlow model using the MeanFlow objective for only 25% of the training time, while employing the Flow Matching objective (where r=t) for the remaining 75%. Similarly, FACM [2] incorporates the original Flow Matching loss into its training objective. How does the Instantaneous Velocity Constraint (IVC) proposed in this paper differ from them?\n\n\n[1] Geng, Zhengyang, et al. \"Mean flows for one-step generative modeling.\" arXiv preprint arXiv:2505.13447 (2025).\n\n[2] Peng, Yansong, et al. \"Flow-anchored consistency models.\" arXiv preprint arXiv:2507.03738 (2025).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- It has always been a question for me how diffusion / flow-matching policies can be used in RL. The “generate-and-select” mechanism used in this paper seems simple and straightforward, yet the authors demonstrate that it works remarkably well. I really appreciate this idea.\n\n- By incorporating recent advances from generative AI (specifically, MeanFlow), the method enables fast online RL training while preserving the generative model’s ability to represent complex, multimodal action distributions.\n\n- The approach achieves the highest or second-highest success rates across nine challenging robotic manipulation benchmarks, showcasing its effectiveness in solving complex, long-horizon tasks.\n\n- The authors provide rigorous mathematical proofs to justify their framework. They show that the implicit value constraint (IVC) ensures a unique solution for the mean flow identity and that their overall algorithm guarantees consistent policy improvement.", "weaknesses": "- The paper emphasizes its one-step inference speed but understates the associated training cost. The core training loss (Eq. 9) necessitates a Jacobian-vector product (JVP) to compute the $\\frac{d}{dt} \\mathbf{u}_\\theta$ term. This operation is often incompatible with optimized attention implementations like FlashAttention, potentially limiting the method's training efficiency and scalability.\n\n- The paper repeatedly claims suitability for \"real-time control systems\" and \"real-time deployment,\" yet these claims lack empirical validation on physical robotic hardware, as all experiments are conducted in simulation.\n\n- The theoretical proofs for policy improvement (Theorem 1) and solution uniqueness (Theorem 3) are elegant but rely on strong assumptions that are often violated in practice. For instance, Theorem 1 assumes bounded Q-function and mean flow matching errors (epsilon_Q, epsilon_A). In deep RL, these errors can be large and unstable. Similarly, the proof for IVC's effectiveness assumes the mean flow loss (L_MF) is perfectly minimized, which is never the case with non-convex optimization. While the theory provides intuition, its practical guarantees are much weaker than presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcVaCka4kt", "reviewer": "Reviewer_dkbj", "summary": "This paper presents Mean Flow Policy (MFP), a reinforcement learning approach that adapts the mean-flow generative modeling framework for efficient one-step action generation. Instead of learning the instantaneous velocity field as in standard flow matching, MFP models the mean velocity field, enabling direct sampling without multi-step integration. To address the underdetermined mean-flow ODE, the authors add an Instantaneous Velocity Constraint (IVC) enforcing equality between mean and instantaneous velocities at the boundary r=t. Combined with a best-of-N Q-guided action selection scheme, MFP achieves competitive or superior results to existing flow-based RL methods (FQL, BFN, QC) on Robomimic and OGBench, while significantly improving training and inference speed.", "questions": "The IVC condition appears to be mathematically implied by the mean-flow definition (Eq. 6) and the mean-flow loss (Eq. 9) as $r \\rightarrow t$. The added IVC term seems to primarily emphasize the boundary condition rather than address a missing component of the mean-flow formulation. Could the authors clarify whether this term introduces any new theoretical insight beyond reinforcing an already implied constraint?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear theoretical presentation with formal analysis of the mean-flow ODE’s non-uniqueness and the effect of the IVC boundary constraint.\n- Empirical results show consistent efficiency gains over multi-step flow policies.\n- The IVC ablation confirms its stabilizing effect during training.\n- Writing and experimental setup are clear and reproducible.", "weaknesses": "- The mean-flow formulation itself is taken directly from prior generative modeling work (*Geng et al., 2025a*); this paper mainly applies it to RL.\n- The proposed IVC is conceptually implied by the mean-flow definition when r\\!\\to\\!t, so it should not be considered a new theoretical contribution.\n- The best-of-N action-selection mechanism is standard in prior RL methods (EMaQ, BFN, FQL).\n- The experimental scope is limited: only a small number of state-based manipulation tasks are used, with no visual-policy or diverse-domain evaluation. This constrains the empirical strength of the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpifwrquHe", "reviewer": "Reviewer_ac7s", "summary": "This paper proposes builds on the use of generative modelling for policy learning by introducing a new policy learning framework with MeanFlow, a one-step alternative to popularly used flow matching based approaches. Additionally, to improve the accuracy of the policy learning, the authors propose a new constraint on the MeanFlow generation framework that ensures the consistency of the instantaneous velocity in modelling the velocity field of flow matching. This is proven both theoretically and experimentally. The policy is then trained in an offline-to-online reinforcement learning setting wherein the Q-values from the critic are used to select the action targets in a best-of-N manner to train the MeanFlow policy network.", "questions": "- How well does the approach perform with an image-based backbone?\n- Since the authors train their model from offline data as well, does similar performance hold for Imitation Learning as well?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed approach shows a novel way of modelling policies using MeanFlow rather than a naive approach that just shows the use of a new generative modelling method to policy learning. The authors show how their instantaneous velocity constraint helps improve the policy learning. The results show a good improvement over previous approaches that use different combinations of flow matching and a best-of-N selection. The presentation clarity of the paper is easy to follow, and the contributions are highlighted well both in terms of the writing and in terms of experimentation.", "weaknesses": "Given the growing use of generative modelling in Imitation Learning settings, and since the authors explore the offline-to-online RL case, a comparison in a simpler behavioral cloning paradigm can help show the advantage of the proposed approach better, which in general can be useful for learning much more complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)