One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning
Authors: Thanh Xuan Nguyen
Chang D. Yoo
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
Diffusion Model
Flow Matching
Offline Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We introduce OFQL, a robust and efficient diffusion-based offline RL method that enables one-step action generation during training and inference via flow matching with average velocity modeling.

Abstract: Diffusion Q-Learning (DQL) has established diffusion policies as a high-performing paradigm for offline reinforcement learning, but its reliance on multi-step denoising for action generation renders both training and inference slow and fragile. Existing efforts to accelerate DQL toward one-step denoising typically rely on auxiliary modules or policy distillation, sacrificing either simplicity or performance. It remains unclear whether a one-step policy can be trained directly without such trade-offs. To this end, we introduce One-Step Flow Q-Learning (OFQL), a novel framework that enables effective one-step action generation during both training and inference, without auxiliary modules or distillation. OFQL reformulates the DQL policy within the Flow Matching (FM) paradigm but departs from conventional FM by learning an average velocity field that directly supports accurate one-step action generation. This design removes the need for multi-step denoising and backpropagation-through-time updates, resulting in substantially faster and more robust learning. Extensive experiments on the D4RL benchmark show that OFQL, despite generating actions in a single step, not only significantly reduces computation during both training and inference but also outperforms multi-step DQL by a large margin. Furthermore, OFQL surpasses all other baselines, achieving state-of-the-art performance in D4RL.

Published as a conference paper at ICLR 2026
ONE-STEP FLOW Q-LEARNING: ADDRESSING THE DIF- FUSION POLICY BOTTLENECK IN OFFLINE REINFORCE- MENT LEARNING
Thanh Nguyen & Chang Yoo
Department of Electrical Engineering
Korea Advanced Institute Science and Technology (KAIST)
Daejeon, Korea
ABSTRACT
Diffusion Q-Learning (DQL) has established diffusion policies as a high- performing paradigm for offline reinforcement learning, but its reliance multi- step denoising for action generation renders both training and inference slow and fragile. Existing efforts accelerate DQL toward one-step denoising typically rely on auxiliary modules or policy distillation, sacrificing either simplicity or performance. It remains unclear whether a one-step policy can be trained directly without such trade-offs. To this end, we introduce One-Step Flow Q-Learning (OFQL), a novel framework that enables effective one-step action generation dur- ing both training and inference, without auxiliary modules or distillation. OFQL reformulates the DQL policy within the Flow Matching (FM) paradigm but departs from conventional FM by learning average velocity field that directly supports accurate one-step action generation. This design removes the need for multi-step denoising and faster and more robust learning. Extensive experiments on the D4RL benchmark show that OFQL,, despite generating actions in single step, not only significantly reduces computation during both training and inference but also outperforms multi- step DQL by a large margin. Furthermore, OFQL surpasses all other baselines, achieving state-of-the-arr performance in D4RL.
parpopaaaationthroughhh time updates, resulting in substantially
INTRODUCTION
95.0
In recent years, offline reinforcement learning (Offline RL) has achieved impressive progress through the integration of diffusion models, lead- ing to many high-performance algorithms. A prominent example is Diffusion Q-Learning (DQL) (Wang et al.) 2022). which replaces the conventional diagonal Gaussian policy in TD3-BC (Fujimoto & Gu 2021) with a de- noising diffusion probabilistic model (DDPM) (Ho et al. 2020). This approach has demon- strated substantial performance gains and has spurred widespread interest in leveraging gen- erative models for policy learning. Notably, DQL remains competitive and often outperforms many more recent methods both diffusion- based planning and policy optimization (Lu et al. 2025a Dong et al. 2024).
92.5
Ours
90.0
Performance
77.5
* OFQL FQL
DQL
V
Diffuser
75.0
Decision Frequency (Hz)
Figure 1: Performance and decision frequency. Performance (i.e., normalized score) and decision frequency are measured eraged across MuloCo tasks from D4RL. OFQL achieves both high inference speed and strong per- formance, clearly outperforming prior baselines.
A100 GPU and av-
Despite its strong empirical results, DQL faces key practical limitations, including high computational demands during training and inference (Kang et al. 2023) Wang et al.) (2022), as well as optimization fragility causing reduced performance (Park et al.
2025 Upon closer analysis, identify that the

---

Published as a conference paper at ICLR 2026
key bottleneck lies in its use of DDPM diffusion policy (Ho et al. 2020), which involves multiple denoising steps per action, leading to slow inference. Furthermore, the training speed is doubly affected: beyond the diffusion loss, DQL requires two rounds of policy sampling per iteration- for the current action and another for the next-to compute all loss components. In addition, DQL. leverages the reparameterization trick backpropagate through the entire denoising chain, amplifying computational load and impeding convergence to optimal solutions. These characteristies collectively hinder DQL's efficiency and robustness.
It is worth noting that several recent approaches have partially addressed these limitations- through improved denoising solvers (Kang et al.) 2023). IQL-based learning (Hansen Estruch al. 2023). or the use of an auxiliary policy and policy distillation strategies (Park et al. [202]] Chen et al.
-aatsnn-Estruch
2023
2024 Lu et al. 2025b). Nevertheless, such solutions typically introduce additional complexity,
multi- multi--phase training procedures, or undesirable trade-offs in scalability and policy quality.
Recognizing that the diffusion policy itself is the bottleneck, we adopt a more direct approach by introducing One-Step Flow Q-Learning (OFQL), a novel framework specifically designed to enable effective one-step action generation during both training and inference, without the need for auxiliary models, policy distillation, or multi-stage training. At the heart of OFQL is the elimination of DDPM's computationally intensive multi-step denoising and associated reparameterization trick. By recasting DQL policy under the Flow Matching (Lipman et al. 2022) paradigm, we facilitate its efficient action sampling. However, conventional Flow Matching trequently yields curved trajectories, limiting one-step inference accuracy- issue rooted in the intrinsic properties of the marginal velocity field it models. We address this by learning an average velocity field instead, enabling accurate direct action prediction from single step. As result, OFQL eliminates the necessity of iterative denoising and recursive gradient propagation, providing a faster, more stable, one-step mraiiing-ifferenee inference pipeline. Extensive empirical evaluations on the D4RL benchmark demonstrate that OFQL not only surpasses DQL in performance but also significantly improves both training and inference efficiency, all while maintaining a simple learning pipeline. Compared to other approaches, OFQI. delivers consistently stronger results, establishing it as a fast and sfate-of-the-arr algorithm on D4RL.
training-
2 RELATED WORK
Diffusion Models in Offline Reinforcement Learning. Offline RL aims to learn effective policies from fixed datasets without environment interaction (Levine et al., 2020), with early methods addressing distributional shift via conservative objectives e.g., CQL (Kumar et al. 2020 TD3+BC (Fujimoto& & Gu 2021), and IQL. (Kostrikov et al. 2021). However, these approaches often rely on unimodal Gaussian policies, which struggle to model complex, multi-modal action distributions. To address this, recent approaches have adopted diffusion models (Ho et al. 2020 Song et al. 2020b) for offline RL. These models excel in representing complex distributions and have been applied in various forms: as planners for trajectory generation (Janner et al.) 2022 Ajay et al. (2022), as expressive policy networks (Wang et al. 2022 Hansen- et al. 2023 and as data synthesizers
to augment training (Zhu et al. 2023).
Among diffusion-basee methods, Diffusion Q-Learning (DQL) WWang et al. 2022) stands out as a strong baseline, replacing Gaussian policies in TD3+BC with diffusion model better capture multi- modal actions. Follow-up evaluations, including those by Clean Diffuser (Dong et al. (024)) and recent empirical studies (Lu al. 2025a b). confirm DQL's consistent advantage over policy-based and planner-based methods. Despite its effectiveness, DQL remains significant computational overhead during both training and inference (Kang et al. 2023). and is prone to suboptimal convergence (Park
et al. 2025)
Subsequent works have attempted to mitigate these issues. For instance, early approaches use an efficient solver that reduces the number of denoising steps (Kang et al., 2023). Other approaches bypass backpropagation through time (BPTT) of the DQL policy update by training a diffusion policy to clone the behavior policy, with actions reweighted by separately learned IQL-based value function Staansnn-Estrucl et al., 2023). Others further apply distillation to obtain a one-step policy (Chen et al.) 2024)2024). However, based methods are generally less effective than actor-crit coioic learning (Park et al. 2025). To address this, (Park et al.) 2025) adopts a flow model to clone the behavior policy and distill it into one-step policy for actor-critic updates. While yielding a one-step inference
2

---

Published as a conference paper at ICLR 2026
policy, the distillation process still requires repeated queries the underlying multi-step diffusion or flow model. Overall, although these methods partially alleviate DQL's limitations, they introduce additional components and multi-phase training procedures, thereby increasing system complexity and limiting practical generality. Departs from prior work, our method pinpoints the diffusion policy itself as the primary source of inefficiency and instability in DQL. We directly solve it with highly performant one-step policy alternative that provides a simpler and more robust solution-wiihhww relying on auxiliary policies, distillation, or sacrificing policy expressivity.
Efficient One-Step Diffusion. Our work is inspired by advances in efficient generative modeling with diffusion models and flow-based models -Sohl-Dickstein al.) 2015 Ho et al. 2020) Song & Ermon 2019). To accelerate generation, one line of work focuses on distillation techniques that
compress multi-step models into fewer steps (Salimans & Ho 2022, Sauer et al. 2024 Yin et al.
2024), while another pursues flow matching approaches that learn time-dependent velocity fields for straight-through sampling (Lipman et al., 2022] Liu et al. 2022). Consistency Models (Song
ct al. 2023) offer another path to one-step generation, but suffer from training instabilities (Song &
Dhariwal 22033 Lu & Song 2024). More recent methods (Frans et al. 2024| Geng et al. 2025 Zhou
et al. 2002) address these limitations by exploiting different physical parametrization using dual time variables, showing improved stability and performance. Leveraging the Mean Flow modeling (Geng al., (2025), OFQL realizes one-step flow-based policy while uniquely incorporating the Q-gradient to guide velocity learning, instead of relying solely on supervised learning.
3 PRELIMINARIES
3.1 OFFLINE RI..
Reinforcement learning (RL) is typically formalized as a Markov Decision Process (MDP), defined by M =
r). Here, S and A denote the state and action spaces, P(s' a) denotes the transition probability of moving from s to s' given action a, and R(s, a) denotes the reward function. The discount factor E [0. 1) governs the trade-off between immediate and future rewards. In RL, the objective is to learn policy xa (a s). parameterized by that maximizes the expected discounted return E,, _ n R(s,, an)).
To support policy learning, the action (Q) function is defined:
(" = Ex
7 h R(sh, an) 80 = 8, ag = a
(I)
which measures the expected cumulative return starting from state s and action a under policy x.
Offline RL is setting of RL where the agent does not interact with the environment but instead learns from fixed dataset of transitions = {(s,, ag. Sh+1, rx)]. The challenge lies in learning an optimal policy solely from this static dataset that often contains suboptimal behavior, without any further exploration.
3.2 DIFFUSION Q-LEARNING (DQL)
Modeling Policy Diffusson Model. The Denoising Diffussonn Probabilistic Model (DDPM) (Ho et al.) 2020) is a powerful generative framework that formulates a forward diffusion process as a fixed Markov chain, progressively corrupting data into noise, and learns parametric reverse process to reconstruct the data. Once trained, DDPM is capable of generating complex data distributions by reversing the diffusion process, starting from pure noise.
Viewing actions as data, (Wang et al. 2022) formulate the reverse process of a DDPM, conditioned on state s, as parametric policy Ne:
K pa (aa a² a²
T& pe Oak |8) p(a
II
where K ~ (O, and a denotes the clean action corresponding to the actual policy output. For training, following DDPM (Ho al. 2020), DQL. parameterizes the Gaussian distribution
3

---

Published as a conference paper at ICLR 2026
Po(at-Ija", (a", with the variance fixed as Σ(a", k; s) 8 where the 13 predefined variance schedule values and the mean is defined via noise prediction model:
uo(a ki = Va aa (a-
gh a co (a" (kis))
- a', and Ey is neural network predicting Gaussian noise. To enforce behavior cloning, the score matching loss can be used as a training objective. Specifically, the model minimizes:
where a² = -
, a
(0) .a) ||ee to eo (vaaka VI- 8) ||]
where l U(1,.......... K}. (a", s) are sampled from the offline dataset D, and NO, denotes Gaussian noise.
Once trained, to generate an action (i.e., a²), the model sequentially samples from K conditional Gaussians, starting from a ~ "N(O, I):
(at-_
g
a k-1
cola", k;s) + v BBc.
Va
vI
Behavior-regullarized actor-critic. To form a complete offline RL algorithm, DQL adopts the behavior-regulene actor-critic framework (Wu et al. 2019 Fujimoto & Gu 2021 alternating between minimizing the actor and critic losses.
Specifically, the critic loss, which focuses training the Q network, is defined as:
((Φ) ((saan)))) a' ( (s: (sssss''') d') Qo, (s. a))
(6)
where E 1, indexes the two Q networks for double Q-learning, and (Ø', (') denote target network parameters updated via exponential moving average (EMA) (Fujimoto & Gu 2021). The actor loss, which focuses on learning the policy, is defined as:
((0)= LDBC(
EE-DDaww [Qo(s,)))
where a: is the weighting coefficient. To normalize for -ataset-specific Q-value scales, is adapted as a _(..a)=D[ll4> (ll.llllaaa)))))) with r being a tunable hyperparameter. The denominator is treated as constant for optimization.
7
Despite being implemented with relatively simple MLP architecture for both the diffusion policy and Q-functions, DQL has demonstrated strong performance across standard offline RL benchmarks such as D4RL (uu al. 2020), outperforming many recent diffusion- policies and planners (Dong al. 2024 Lu et al. [2025a b).
Nevertheless, DQL exhibits two notable limitations: (1) slow training and inference (Kang et al.. 2023). and (2) susceptibility to unstable or suboptimal training (Park et al. 2025).
4 RATIONALE AND METHODOLOGG
The slow inference time stems from its reliance on denoising diffusion process, where actions are sampled through reverse chain of K Gaussian transitions (Eq.5). 5). Due the Markovian nature of the DDPM framework, sampling an action during inference requires the same number of denoising steps K as those used during training. Additionally, large K is typically necessary to ensure that a approximates isotropic Gaussian. Reducing K breaks this assumption, often resulting in significant performance degradation.
In training, DQL also exhibits compounding inefficiencies. First, the critic loss (Eq. 6 requires sampling target actions a' ~ TØ,, each which must be generated via the full K-step denoising, introducing considerable computational overhead. Enforcing one-step action generation, however, can destabilize training due to diffusion sampling errors. Second, the actor loss (Eq. 7 requires sampling actions from my and performing backpropagation through all K denoising steps (i.e., BPTT)
4

---

Published as a conference paper at ICLR 2026
Flow Path
D Diffusion Path
(a)
(b)
(c)
(d)
Conditinaal Velocity
Marginal Velocity
Average Velacity
Figure 2: Comparison between diffusion and flow matching. (a) Conditional flows arise from different (e, 2) pairs, resulting in varying conditional velocities. (b) Marginal velocity is obtained by averaging over these conditional velocities. (c) Flow paths are inherently curved, but average velocity fields enable direct one-step transport from noise to data. (d) Diffusion paths are also curved but noisy, making one-step denoising challenging. Note that all the velocities exhibit symmetry under time reversal. As the model is trained to parameterize the forward flow (from data to noise), inference inverts this direction to generate samples. Accordingly, for clarity, we plot the negative velocity vector to represent the reverse generation trajectory.
using the reparameterization trick (Eq. 5). Although this enables end-to-end training, the recursive gradient flow through a long stochastic computation graph is well-known to be prone to numerical
instability and can potentially lead to suboptimal outcomes (Chen et al., 2023 Park et al. 2025).
At first glance, diffusion policy sampling appears to be the bottleneck, but resolving DQL's limitations far from straightforward. For example, using an efficient solver, e.g., the DDIM solver (Song et al. 2020a could reduce denoising steps, yet in experiments, applying DDIM for one-step action generation severely degraded policy performance. Similarly, replacing diffusion with consistency models, such as Consistency-AN (Ding & Jin 2023), still requires multiple denoising steps and yields lower performance. The most effective one-step approaches are saaiillation-basee methods (Park) et al. 2025 Chen al. 2024 2023 Lu al. 20255), which accelerate inference through student policies but incur an additional training phase or shift the inefficiencies to the distillation stage. This raises natural question: Can we design one-step policy that directly eliminates inefficiencies in both training and inference?
Designing One-Step Policy. Diffusion models generate samples through stochastic and often curved trajectories, which makes one-step sampling challenging. Flow Matching (FM) (Lipman et al. 2022) offers a principled alternative by mapping noise directly to data along smoother, more direct paths (Liu et al. 2022), as illustrated in Figure (c,d). (c,d). Modeling the policy via flow matching can potentially improve both efficiency and stability. Let us model the policy as a variant of flow matching based on linear paths and uniform time sampling. Given data s ~ D and noise ~ N(O I), FM defines a linear flow path a, and conditional velocity vi (arla;; s) for particular s as:
dug
a (1 t)a + te,
U/
where t [0.
(8)
dt
Note that by formulation, a = ag. and we use a without a subscript to denote the clean action for simplicity.
FM essentially learns the marginal velocity, parametrized by the neural network up(a,, t; s), using the Conditional Flow Matching loss:
CCFM (0) Eo.. (1, t; va | a; s)]
Once: trained, sampling an action for a state proceeds by solving the ODE da
di v(ar, t; starting from a1 ~ N(O, I), approximated using solver such as Euler's method: a;-A: = a; At v(a,, s).
Intuitively, since flows are designed so that the overall transport trajectory becomes approximately straight (Liu et al. 2022). one might expect that modeling the policy via Flow Matching could support one-step generation by setting At 1. However, in practice, the sampling trajectory straight only when the target distribution collapses to a delta distribution or when rectification or similar techniques explicitly applied. Without such conditions, the marginal velocity field typically induces curved
5

---

Published as a conference paper at ICLR 2026
overall trajectory, preventing reliable one-step action generation. Importantly, the curvature is not simply a consequence of imperfect neural approximation, but rather an inherent property of the ground-truth marginal velocity field. This phenomenon is illustrated in Figure 2 (a,b.c)
To enable high-quality one-step generation, we reinterpret the velocity field v(a,,t;)) in Flow Matching the instantaneous velocity, and instead propose model the average velocity, which directly connects any two arbitrary time steps. Specifically, we define the average velocity over an interval [r, as:
ula r, t; - --r
S. u(ar, T:))
(10)
representing the total displacement across the interval divided by its duration. Here, and denote the target and current times, respectively, with the constraint r st.l.
In general, the average velocity is functional of the instantaneous velocity, i.e., = F(u]. This field u is fully determined by the instantaneous velocity field v and is independent of any neural network. We therefore treat u as the ground-truth average velocity field and train a neural network to approximate using a loss, referred to as the Peerage-Velocio Matching loss:
Lrne. (0) Eagrsssiiss
[| r, 118) u(a,, r, (1 s)]
(11)
Once up is learned, actions can be generated in single step through the approximate endpoint map
To(c, 8)
uo r=0, 1=1;8), <~0(0, T):
(12)
which eliminates the iterative ODE integration required by standard Flow Matching. This avoids both the computational overhead and the discretization error associated with numerical ODE solvers. A formal justification for why this one-step procedure preserves the FM action accuracy is provided in Appendix G
As a result, the learned policy Ta(a 8) is the push-forward of the Gaussian prior through the approximate endpoint map:
(O,
Moreover, when used as regularizer, optimizing the average-velocity matching loss Lrbc. (0) encourages the learned one-step policy TT(- s) to remain close to the behavior policy jl( s). effectively performing behavior cloning. Importantly, this behavior cloning still preserves the ability to model complex, multimodal action distributions through the nonlinear transport map inherited from Flow Matching (see Appendix for a formal justification).
Practical Loss. In practice, computing from its definition requires integration, which is computa- tionally intractable for optimization. To address this, we adopt an equivalent reformulation based on the MeanFlow Identity (Geng al. 2025);
d ulan, (,, (1-))))) 155),
u(ae, r, 4 = v(ar, 1 1;8) -1 (1-
(13)
where the total derivative expands as d u(a,, t: v( s) pa, d.u. The computation of the derivative also remains efficient by leveraging Jacobian-vector products. Notably, when the target u reduces to the instantaneous velocity.
zi
Equation 13 mathematically equivalent to Eq. (the detailed derivation is provided in Appendix We therefore use the Eq. 13 to compute the average velocity uegt, avoiding explicit integration in Eq. 10
The resulting field can be fitted by neural network using the following loss:
LFBC Ee.rstt(a.,)D,,, ||u (az, r, t; sg(ua))l1
(14)
In this loss, a, (1 l)a le.. The v(a,, 1;8) in Eq. 13 is additionally replaced with the conditional velocity Vt, following FM, to approximate the instantaneous velocity on the fly during training. Consequently, the target velocity defined as:
Uigi (t - (v 0c, ug + Ügu0),
(15)
where D. = - a. The operator sg(+) denotes nop-gradient, preventing higher gradients on the target during optimization.

---

Published as a conference paper at ICLR 2026
One-step Flow Q-Learning. To form OFQL, model the policy with the average velocity
parameterization ug. Its behavior regularization loss is (F defined in Eq. [4] In addition, since the target average velocity is computed based on the estimate of the instantaneous velocity (Eq. [3] accurate estimatiin of the instantaneous velocity is crucial for effective average-velocity learning. To encourage this, when sampling (t,, r) we enforce certain flow ratio X, i.e., the probability that t This design biases training toward learning the instantancous velocity, while still allowing regression to the average velocity, improving the bootstrapping.
Given the policy sampling an action becomes a differentiable one-step operation:
a na ()
a= -==e-wofs,* 0, 0,6 = 1; where e~ N~,,
(16)
The critic and actor losses are updated as:
{ = (D a'rway (r+ + min Qo:(d',a') -Q((,)))))))) (E(1.2)
(17)
( = (F a E,-D, amty [Q (s,
With the new one-step policy, the main modification from DQL losses is that the behavior regulariza- tion term and action sampling require only a single step.
5 EXPERIMENT SETTING
Benchmarks. We evaluate OFQL on a diverse set of tasks from the D4RL. benchmark suite (Fu et al. 2020), widely adopted standard for offline reinforcement learning. Our evaluation spans various domains, including locomotion, navigation and manipulation tasks to demonstrate the method's generalizability. Detailed task descriptions and experimental protocols are provided in Appendix A.
Baselines. To rigorously assess OFQL's performance, we compare it against a broad spectrum of representative baselines, categorized as follows: (1) Non-Diffusion policies: Behavior Cloning (BC),
TD3-BC (Fujimoto & Gu G 2221), and IQL (Kostrikov et al., (021)) )); (2) Diffusion-based planners: Diffuser (Janner al. (022). Decision Diffuser (DD) (Ajay al.) 2022 (3) Multi-step Diffusion-
based policies: IDQL 2023 and (4) One-step Flow policies: FQL Park et al. 2025)
et al.) 2003).. DQL (Wang et al. 2022). and EDP (Kang et al.)
Implementation Details. Our approach builds directly on DQL (Wang et al. 2022). inheriting its training and inference procedures to ensure a fair comparison. We adopt the original DQL architecture for both the Q-function and policy networks. The only minor modification lies in the policy input, which is augmented by concatenating an additional positional embedding corresponding to the target step r, in addition to the standard timestep embedding L. For timestep sampling, the and are sampled from a logit-normal distribution (Esser et al. 2024) with parameters (-0.4, 1.0). The main the flow ratio and 17. Unless otherwise specified. we set the flow ratio to 0.5 and tune 3) via grid search over (0.001,0.0 ,0.00.11000 O. 10110..,0.33005. 1,0.3.055). We adopt the Adam optimizer with a learning rate of 3 x 10-4, We ensure reliability by reporting OFQL results the average D4RL normalized score (Fu et al. 2020), computed over three training seeds, with each model evaluated on 150 episodes per task. Other hyperparameters remain consistent with DQL. Additional implementation details provided in Appendix B.
hyperparameters
6 EXPERIMENTAL RESULT
Benchmark results are summarized in Table with details discussed below.
Locomotion Domain (MuJoCo). OFQL achieves strong performance locomotion tasks, surpassing competitive diffusion-basee basclines such as DQL, DD In particular, OFQL improves the average performance of DQL from 87.9 to 92.5, with notable gains on medium and medium-replay tasks, which are known to contain suboptimal and noisy trajectories. These tasks often induce complex, multi-modal action distributions that challenge standard policy learning, making expressive action modeling and stable value learning essential. The observed improvements may be attributed to two key aspects of OFQL: (1) its policy modeling remains expressive for capturing complex action distributions, and (2) the avoidance of BPTT in Q-learning, which may yield more stable value estimation, leading to better convergence. Together, these factors provide a plausible explanation for OFQL's consistent performance gains.
7

---

Published as a conference paper at ICLR 2026
Mon-Diffusion Policies
Diffusion Planners
Multi-step Diftusion Policies
One-tte Flow Policies
Datisct
BC
TD3-BC
IQr.
Difluser
DD
EDP
IDQ
DQL
FQU.
OFQI (Ous))
55.2
90.7
86.7
=0.3
95.8 0.11
01
9
Hopper Medium Expert
52.5
98.0
91.5
102.2 a 0.9
0.4
110.1
111.1
862 1.3
110.2
Walker2d Median- Expent
107.5
110.1
109.6
107.4 2 0.1
106.420.11
100.000.00
110.0.0.01
110.1
1005
113.0 0.1
HalfCheethh Medium
42.6
48.3
47.4
43.8 0.1
0.3
50.8
51.5
51.1 0:0.5
6011 0.11
63.8
Hopper Medium
529
59.3
66.3
2..
70.1
90.5
24.
0.3
Waller24-MMddian
753
83.7
78.3
79.4
88.1
07.0
72.7
67.4
HallChocahhe Macruaa-MMddda-Reeie
36.6
44.6
44.2
0.7
22.1
0..4
0..
4.3
51.1
=
Hopper Medium Reploy
18.1
60.9
94.7
0.5
0.2
9..4
6:.4:+0..55
101.9 0.7
Walker2d Medium Replay
26.0
81.8
73.9
58.3
7 0.6
$7.0
89.1
95.5
82.1
106.2 0.6
Average (MulaCo)
51.
75.3
77.0
76.2
83.2
82.4
$4.1
92.5
MreMoe-Meediun-Ue Play
0.0
106
71.2
6.7
8.0 4.3
7..3+662
67.3+
766 108
78.0 2.0
58.1 + 5.0
AntMize Large Play
0.0
0.2
39.6
1
0.0 0.0
48.744.7
d..4
84.0
$1.0 6.1
AstMuue-Ma -eediMMt-D-eeeeee
0.5
3.0
70.0
2.0 1.6
2.8
= 1.9
7 10.3
71.0 13.0
90,2 4.2
AntMare-- paaee-aar---viveeee Divens
0.0
0.0
47.5
27.3 2.4
0.000000
4.3
40.0 11.4
56664.66
83.0 4.0
76.1 666
Average (AntMaze)
0.2
3.5
57.1
13.3
3.0
50.2
Kitchen Mixed
51.
0.0
51.00
2..
7 0.0
50.2 a 1.8
60.5 #:411
62.6
50.5:16
09.0 1.5
Kitchen-Patiill
38.0
0.0
46.3
1,3
56.5 5.8
40.8 e 1.5
66.7 2:5
60.5 6.9
55,742.5
65.0 2.3
Average (Kitches)
44.8
0.0
48.7
36.1
Table I: Comparison of normalized scores on D4RL benchmark across MuJoCo, Kitchen, and AntMaze domains. Bold values indicate the best performance per row.
Compared to other acceleration approaches, although EDP and IDQL enhance efficiency and stability, they do at the expense of reducing final performance relative to DQL, whereas OFQL achieves improvements. When compared with one-step FQL (based on distillation), OFQL surpasses it by significant margin (+13.3). Overall, OFQL offers a superior combination of efficiency and effectiveness, with consistent gains across varying data regimes.
AntMaze Domain. AntMaze tasks are particularly challenging due to sparse rewards and suboptimal demonstrations, requiring accurate and stable Q-value guidance to perform well. Prior approaches (e.g., BC and Diffuser) struggle without Q-learning, whereas methods incorporating Q-learning signals (e.g.. IDQL and DQL) achieve consistently better results.
Building on the Q-learning framework, while EDP and IDQL underperform relative to DQL, OFQL achieves a substantial improvement over DQL, raising performance from 64.6 to 84.6 and outper- forming all diffusion- based baselines and the one-step FQL. Notably, FQL., which employs a one-step policy for cctor-critic training. improves upon DQL from 64.6 to 79.0. We assume Q-learning is crucial in this domain, and that OFQL and FQL may benefit from avoiding BPTT.
Kitchen Domain. The Kitchen datasets contain low-entropy, narrowly distributed behaviors (Dong et al., 2024). where action modeling plays a larger role than Q-Jearning. OFQL surpasses DQL (61.6 ++ 67.0), achieving the strongest performance across methods. Although IDQL proves competitive, FQL drops to 53. likely because its one-step policy lacks sufficient expressivity. Remarkably, OFQL's one-step formulation does not suffer this drawback, instead preserving expressivity while achieving tate-of-the-art results.
7 ABLATION STUDY
Method (Steps)
DQL (5)
DQL+DDIM (1)
FBRAC (1)
FQL (1)
OFQL (1)
Score
87.9
11.6 (-76.3)
67.1 (-20.8)
79.2 (-8.7)
92.6 (+4.7)
Table 2:
Comparison of methods (steps) using different improvement strategies toward one-step action generation across 9 MuJoCo tasks. The average normalized score is reported.
Comparison of Strategies Toward One-Step Prediction. To investigate how to effectively adapt DQL for one-step prediction, we evaluate the following strategies: (1) DQL: The base model, trained and evaluated with 5 denoising steps. (2) DQL+DDIM: A pretrained DQL model with one-step DDIM sampler applied only at inference time. (3) FBRAC: The flow policy-bassed counterpart of DQL. trained with 5 denoising steps for criorecritic updates but using single step at inference. (4) FQL: Learns a behavioral policy with a flow model, then distills it into a one-step policy for actor-critic training and inference. (5) OFQL: Trained and evaluated entirely in the one-step regime.
8

---

Published as a conference paper at ICLR 2026
Table 2 reports the average performance across 9 MuJoCo tasks. DQL+DDIM shows severe degra- dation 76.3), suggesting that direct application of improved samplers for one-step inference is ineffective. FBRAC performs better (-20.8), but still lags behind DQL, likely due to the discretiza- tion error introduced when performing one-step prediction with curved trajectories. FQL further narrows the gap (-8.7) by distilling one-step policy from multi-step flow model. In contrast, OFQL achieves the best results, exceeding DQL by (+4.7) while consistently supporting one-step sampling in both training and inference.
Training Time (hours)
50
OFOL. DOL
Decision Frequemey (H2)
1.0X
OFOL.
40
0.8%
PQL
DQL
30-
20-
10
0.2X
0.0k
KOM
hry
365
10
20
50
10
20
50
Dillusion/llow NFE Steps
Diffusion/Flow NFE Steps
Figure 3: Training Time (4) and Decision Frequency (f) over one million steps, averaged on MuloCo tasks. NFE (Number of Function Evaluations) denotes the denoising steps required by flow/diffusion model to generate one action from pure noise. During training and inference, OFQL uses only one NFE, while DQL requires multiple ones. It is worth noting that for inference, FQL runs with one-step policy, but training still relies on multi-step flow policy to construct distillation targets.
1 step
2 steps
5 steps
10 steps
1 step
Figure 4: Comparison of distribution modeling capabilities between FM with marginal velocity parameterization (left; evaluated at 1,2,5,10 steps generation) and average velocity parameterization (right; evaluated with one-step generation) toy dataset with complex multi-modal structure.
Training and Inference Efficiency Comparison. Figure 3 reports the wall-clock training time (IM steps) and decision frequency (Lu et al. 2025b) on an A100 GPU (see Appendix for experimental protocol). DQL's training time scales nearly linearly with the number of denoising steps steps-from 11.7 hours at 5 steps to 49.5 hours at 50 steps-while OFQL completes training in only 6.3 hours. At inference, OFQL reaches 846.5 Hz, compared 238.7 Hz for 5-step DQL and just 35.5 Hz for 50-step DQL.
Compared one-step FQL baseline, OFQL achieves nearly the decision frequency but enjoys shorter training time. This advantage arises because FQL requires multiple NFEs to compute distillation targets, leading to a slower training loop. Note that, despite comparable speed, FQL consistently underperforms OFQL in terms of policy performance.
Overall, OFQL achieves substantially faster training and higher decision frequency without sacrificing model expressivity, making it more practical than multi-step DQL distillation-basse FQL.
Ablation on flow ratio. We study the ef- feet of varying the flow ratio across differ- datasets in HalfCheetah (Table The best performance is obtained at flow ratio of 0.5, achieving 95.2 on Medium Expert, 63.8 on Medium, and 52.2 on Medium Re- play. In contrast, using either the flow ratio equal to (equivalent to pure flow match- ing) or setting it to 0 results in noticeable performance degradation. A moderate flow ratio serves as an effective regularizer, yielding the most stable and robust learning behavior.
Flow Ratio
0.75
0.25
Medium Expert
38.3
90.86
95.2
92.03
90.47
Medium
46.3
62.03
63.8
63.76
63.2
Medium Replay
45.2
50.2
51.2
50.3
10.5
Table 3: D4RL scores across HalfCheetah datasets der varying flow ratios.

---

Published as a conference paper at ICLR 2026
Compare Marginal Velocity and Average Velocity Parameterization. DQL has convincingly shown that employing a more expressive policy leads to superior final performance in the actor-critic training framework. To examine the expressiveness of one-step generation, we conduct toy dataset experiment comparing Flow Matching with marginal velocity (u-param) versus average velocity (u-param) parameterization across different generation steps.. As illustrated in the rightmost panel of Figure 4 samples generated by u-param in a single step already demonstrate strong mode coverage and close alignment with the target distribution. In contrast, v-param requires multiple steps to achieve comparable quality and often produces collapsed samples with fewer steps. These results underscore the advantage of modeling the average velocity field for one-step generation and give strong confidence to modeling policy. Additional experimental results and experiment setting are provided in the Appendix
8 CONCLUSION
We presented One-Step Flow Q-Learning (OFQL), a novel policy learning framework that overcomes key limitations of Diffusion Q-Learning by enabling efficient, single-step action generation during both training and inference. By reformulating DQL within the Flow Matching framework and learning an average velocity field rather than a marginal one, OFQL eliminates the need for multi-step denoising, recursive gradient propagation. This leads to faster training and inference, while surpassing the performance of state-of-the of-thhe-rrt diffusion- D4RL benchmark confirm the effectiveness and efficiency of OFQL, underscoring the promise of one- step flow policies for advancing offline RL. More broadly, OFQL facilitates accurate high-frequency decision-making, suggesting potential for real-time control and scalable deployment in complex, latency-sensitive domains.
offline RL methods. Empirical results on the
ACKNOWLEDGMENTS
This work was supported by the Institute for Information communications Technology Planning Evaluation (ITP) grant funded by the Korea government (MSIT) (No. RS-2022-110951, Development Agents Learning by Asking Questions) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) Robotic System using Continual Learning and Multimodal Language Model based Multi Attribute Feedback).
of Uncertainty-Awer
Intelligent
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision-makinn arXiv preprint arXiv:2211.15667, 2022.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859 877, 2017.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. arXiv preprint arXiv:2310.07297 2023.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou Diffussonn policies creating a trust region for offline reinforcement learning. Advances in Neural Information Processing Systems, 37: 50098-500 10009855125,,, 2024.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PmLR, 2020.
Cheng Chi, Zhenjia Xu, Siyuan Feng. Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273666, 2023.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984 2023.
10

---

Published as a conference paper at ICLR 2026
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yi Ma, Pengyi Li, and Yan Zheng. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. Advances in Neural Information Processing Systems, $7:86899-86922 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Picter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410. 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 11-20132-20144 2021.
Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint vxXiv:250../3447 2025.
Philippe Hansen- lla Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-653 2020.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 67195-67212, 2023.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 142.691412.6980, 2014.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, 2021. URL
abs 22002.00424
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 1rXiv:2110.061699 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. reinforcement learning. Advances in neural information processing systems, 33:1 179-1191, 2020.
Conservative q-learning for offline
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 17X00:2210.02777, 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.088 030003, 2022.
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2400. 1081, 2024.
Haofei Lu, Dongqi Han, Yifei Shen, and Dongsheng Li. What makes a good diffusion planner for decision making? arXiv preprint 2.Xx:25033.05555 2025a.

---

Published as a conference paper at ICLR 2026
Haofei Lu, Yifei Shen, Dongsheng Li, Junliang Xing, and Dongqi Han. Habitizing diffusion planning for efficient and effective decision making. arXiv preprint arXiv:2502.06401, 2025b.
Tung M Luu, Thang Vu, Thanh Nguyen, and Chang D Yoo. Visual pretraining via contrastive predictive model for pixel-based reinforcement learning. Sensors, 22(17);6504, 2022.
Tung M. Luu, Thanh Nguyen, Tee Joshua Tian Jin, Sungwoon Kim, and Chang D Yoo. Mitigating adversarial perturbations for deep reinforcement learning via vector quantization. In 2024 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 595-602. IEEE, 2024.
Tung M Luu, Donghoon Lee, Younghwan Lee, and Chang D Yoo. Policy learning from large vision- language model feedback without reward modeling. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11685-11692. IEEE, 2025a.
Tung Minh Luu, Younghwan Lee, Donghoon Lee, Sunho Kim, Min Jun Kim, and Chang D Yoo. Enhancing aating-based reinforcement learning to effectively leverage feedback from large vision- language models. arXiv preprint arXiv:2506. 12822, 2025b.
Thanh Nguyen, Tung Luu, Trung Pham, Sanzhar Rakhimkul, and Chang D Yoo. Robust maml: Prioritization task buffer with adaptive learning process for model-agnoston meta-learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pp. 3460-34600 3464. IEEE, 2021a.
Thanh Nguyen, Tung M Luu, Thang Vu, and Chang Yoo. Sample-efficient reinforcement learning representation learning with curiosity contrastive forward dynamics model. In 2021 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 3471-3477 IEEE, 2021b.
Thanh Nguyen, Trung Xuan Pham, Chaoning Zhang, Tung M Luu, Thang Vu, and Chang D Yoo. Dimel: dimensional contrastive learning for improving self-supervised learning. IEEE Access, 11: 21534 21545, 2023.
Thanh Nguyen, Tung M Luu, Tri Ton, Sungwoong Kim, and Chang D Yoo. Uncertainty-aware rank-one mimo q network framework for accelerated offline reinforcement learning. IEEE Access, 12:100972-100982, 2024a.
Thanh Nguyen, Tung M Luu, Tri Ton, and Chang D Yoo. Towards robust policy: enhancing offline reinforcement learning with adversarial attacks and defenses. In International Conference on Pattern Recognition and Artificial Intelligence, pp. 310-324. Springer, 2024b.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL https:://rxxit wttpor/g//p///aaaaa///// 02155
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprini arXiv:2410.20099, 2024.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arXiv:2502.02538. 2025.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluea Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint 2rXiv:2301. 10677, 2023.
Trung Xuan Pham, Jin Woong Choi, Rusty John Lloyd Mina, Thanh Xuan Nguyen, Sultan Rizky Madjid, and Chang D. Yoo. Lad: A hybrid deep learning system for benign paroxysmal positional vertigo disorders diagnostic. IEEE Access, 10:1 13995-19111111 13995-114007, 2022, doi: 10. 109/ACCESS 2022. 3215625.
Simon Ramstedt and Chris Pal. Real-time reinforcement learning. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2026
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion
distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2265.,265, pmir, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. 10. 14189, 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basee-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13446 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01466, 2023.
Sreyas Venkataraman, Yufei Wang, Ziyu Wang, Navin Sriram Ravie, Zackory Erickson, and David Held. Real-world offline reinforcement learning from vision language model feedback. In 2025 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 13452-13459.
IEEE, 2025.
Thang Vu. Kookhoi Kim, Haeyong Kang, Xuan Thanh Nguyen, Tung M Luu, and Chang D Yoo. Sphererpn: Learning spheres for high-quality region proposals on 3d point clouds object detection. In 202/ IEEE International Conference Image Processing (ICIP), pp. 3173-3177. IEEE, 2021.
Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation on point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2708-2722200 2717,22 2022.
Thang Vu, Kookhoi Kim, Thanh Nguyen, Tung M Luu, Junyeong Kim, and Chang D Yoo. Scalable softgroup for 3d instance segmentation on point clouds. IEEE transactions on pattern analysis and machine intelligence, 46(4):1981-1995, 2023.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint Stti::22008061999 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911. 11361, 2019.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6613-6623, 2024.
Hee Suk Yoon, Eunscop Yoon, Mark A. Hasegawa-Johnson Sungwoong Kim, and Chang D. Yoo. ConfPO: Exploiting policy model confidence for critical token selection in preference optimization. In Forty-second International Conference on Machine Learning, 2025, URL
htttps://openreeiew..ee net. www.fforrmiiorrmmmm htt.pparrere.aoaeeeeeooooooreeea Fo
I Linqi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching. arXiv preprint arXiv:2503.07565. 2025.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490-4499, 2018
13

---

Published as a conference paper at ICLR 2026
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, arXiv preprint arXiv:2311.01223, 2023.
A BENCHMARKING TASKS AND EVALUATION PROTOCOL.
HalfCheetah
Hopper Locomotion (Muj oCo)
Walker2D
AntMaze Medium
AntMaze Large
Manipulation (Kitchen)
Navigation (AntMaze)
Figure 5: Illustration of the benchmarking tasks examined this study. The tasks include locomotion challenges for short-term decision dccision-making, making, robotic arm manipulation tasks requiring long-term strategic decision-man makiin, and navigation tasks focused on path optimization.
Benchmarking tasks. As shown in Figure 5 we evaluate the performance of OFQL using diverse set of benchmarking tasks that span various domains of reinforcement learning. These tasks are chosen to assess OFQL's capability across a broad spectrum of environment setups, which essential for understanding the model's robustness and generalization. The selected tasks include locomotion challenges that emphasize short-term decision-make making. robotic arm manipulation tasks requiring long-term strategic decision-making, and navigation tasks focused on pathfinding. By covering this wide array tasks, we ensure a comprehensive evaluation of OFQL's performance in both simple and complex settings, facilitating deeper understanding of its strengths and limitations.
Locomotion (MuJoCo): The MuJoCo Locomotion task is a well-established benchmark in rein- forcement learning, where the agent is tasked with controlling simulated robot to navigate through a dynamic and complex environment. This task is designed to test the agent's 's ability to perform locomotion tasks, emphasizing short-term decision-making and agility in navigating unpredictable terrains.
Manipulation (Kitchen): The Kitchen (Franka Kitchen) task is robotic manipulation challenge in which the agent is required to interact with objects in a kitchen environment. This task is specifically designed evaluate the agent's proficiency in long-term strategic decision fecision-making, involves making sequences of actions for tasks such as object manipulation and coordination, which require higher levels of temporal reasoning.
Navigation (AntMaze): The AntMaze task combines locomotion and planning challenges in maze environment, where the agent must navigate through increasingly complex and variable maze configurations. This task is designed to test the agent's 's ability to perform locomotion tasks while incorporating advanced planning strategies, balancing exploration and exploitation maze with dynamic elements.
Evaluation Metric. We adopt the D4RL. (Fu et al. 2020) benchmark to report the normalized score, which allows for fair comparison across approaches. The normalized score is computed for each environment, using the following formula:
Normalized Score 100 x score random score
(18)
expert score random score
14

---

Published as a conference paper at ICLR 2026
A normalized score of 0 represents the average returns (over 100 episodes) of an agent that selects actions uniformly at random across the action space. normalized score of 100 corresponds to the average returns of a domain-specific expert (chosen by D4RL).
B ARCHITECTURAL AND IMPLEMENTATION DETAILS
Algorithm 1 OFQL Algorithm
1: Initialize policy network Tor, critic networks Qo, and Q Qot Q05
2: for each iteration do
3: Sample transition mini-batch B = {(Sh, On, Th, 8h+1] D
4: # Q-value function learning
5: Sample 04+1 -errr((aa (an+11 5h-1) ] by Eq. 16
6: Update Qs, and Qb using Eq. [Max backup Kumar et al. 2020 optional
7: # Policy learning
8:
Sample on ~ Ta(an ( (hh) by Eq. 16
9:
Update policy Te by minimizing Eq. 7
10:
# Update target networks every K iteration
O' + ~p0' + (1 p)0
12: 0f + pd + p)g, for = (1,2)
end for
Our approach generally builds directly on DQL (Wang et al. 2022). inheriting its training and inference. Below, we outline the key architectural and implementation details.
Architectural Details. We adopt the original DQL architecture for both the Q-function and policy networks, with a minor modification to the policy input. Specifically, we augment the input by concatenating an additional positional embedding corresponding to the target step r, alongside the standard timestep embedding (. More specifically, the architectures are as below:
The policy is modeled as the average velocity function uo(a,, r, L; s), where oe denotes the action latent, and r are timestep variables, and s is the state conditioning input. We adopt the same MLP-based architecture as used in DQL, with the modification of incorporating the additional timestep r. Specifically, Up parameterized as 3-layer multilayer perceptron (MLP) with Mish activations and 256 hidden units per layer, followed by linear output layer that maps to the action dimension. The input to ug is the concatenation of the action latent vector, the current state vector, and the sinusoidal positional embeddings of timesteps t and (time embedding size 64). The output is the predicted average velocity that flows from timestep to
Policy Network:
We utilize the same Q network architecture in DQL.. Specifically, we employ two Q networks, each implemented as a 3-layer MLP with Mish activations and 256 hidden units per layer, followed by linear output layer that maps to single action-value dimension. The input to each Q network is the concatenation of the action and the observation, and the output is the estimated state-action value.
Q Networks:
Training Details. The pseudo algorithm of OFQL is provided in Algorithm I where Max-Q backup applied to AntMaze tasks only, in DQL. In the training, the time variables and are sampled from logit-normal distribution (Esser et al. 2024) with parameters (-0.4, 1.0), which improves stability compared to uniform sampling. During sampling, time pairs are selected such that r # holds for 50% of the samples fiee,,fow ratio equal to 0.5). In the actor loss, the hyperparameter balances behavior regularization and Q value maximization. To normalize for dataset-specific Q-value where 17 is tunable hyperparameter. We search
scales, is adapted as
_(o.a)-D (IQ.))))[[Q((aa))) [s.aa)
over (0.001, 0.01, 0. 0.3, 0.5} since the relative importance of Q-guidance varies by domain (e.g..
the Kitchen tasks require more policy regularization and the AntMaze tasks require more Q-learning).
Training is conducted for 1000 epochs (2000 for MuJoCo tasks), with each epoch consisting of 1000 gradient steps and batch size of 256. Both the policy and Q networks are optimized with Adam (Kingma & Bal 2014), using learning rate of 3 10 For rewards, task rewards in MuJoCo Gym and Kitchen, while following CQL's modification (Kumar et al. 2020)
adopt the original
for AntMaze, consistent with DQL. For evaluation, we follow the protocol of Dong et al. (024))
15

---

Published as a conference paper at ICLR 2026
including the same sampling procedure. We report the mean normalized return averaged three training seeds, with each model evaluated on 150 episodes per task.
C DDPM, FLOW MATCHING WITH MARGINAL VELOCITY COMPARED TO AVERAGE VELOCITY PARAMETERIZATION ON TOY DATASETS
step
steps
steps
10 steps
step
40
Figure 6: Comparison of sample quality between v-parameteriization (left, steps 1, 2, 5, 10) and I-parameterization (right, one step) on toy datasets.
1 step
2 steps
steps
10 steps
1 step
Figure 7: Comparison of sample quality between DDPM in DQL (left, steps 1, 2, 5, 10) and u-parameterization in OFQL (right, one step) on toy datasets.
Experiment Setup. As DQL convincingly demonstrates, greater model expressiveness leads to stronger final performance in the actor-critic framework. To illustrate the capability of modeling complex distributions in a one-step setting, we compare (1) Flow Matching with two velocity parameterizations; marginal velocity (t-param) and average velocity (u-param) (2) DDPM in DQL with average velocity in OFQL; across three synthetic datasets: Crescent, Spiral, and Checkerboard. Each dataset challenges the models to capture distinct geometric structures.
16

---

Published as a conference paper at ICLR 2026
1. Crescent: Points form crescent shape, with outer and inner circle, testing the models' ability to avoid points in the inner circle.
2. Spiral:
Points follow noisy spiral, evaluating the models' robustness in recovering the structure amidst noise.
3. Checkerboard: Points are arranged in a checkerboard pattern, testing the models' capacity capture multimodal distributions.
Architecture. (1) e-param: estimates marginal velocity using a multi-layer perceptron (MLP) with 3 hidden layers (64 units each). with inputs: noise latent, timestep. (2) u-param: estimates average velocity using a similar MLP architecture but with an additional target time input. (3) DDPM: estimate the time step noise using multi-layer perceptron (MLP) with 3 hidden layers (64 units each), with inputs: noise latent, timestep. All models use Mish activations.
Training and Evaluation. We train both models for 100 epochs with batch size of 2048 and 40 batches for each epoch. The DDPM, e-param is evaluated with varying prediction steps (1, 2, 5, 10), while the u-param is evaluated with one denoising step. We visualize the ground truth distribution (blue points) and the generated samples (orange points) on a 2D plot to assess how well each model captures the dataset's geometric structure.
Results on u-param vs u-param While the main paper reports results on the Checkerboard dataset, Figure@additte adddiioonalyy presents results on two other toy datasets: Spiral and Moon. The results show that the u-param consistently generates rell-structured samples that match the target distribution even in single step. In contrast, the u-param exhibits significant mode collapse and noise early steps (1-2), requiring up to 10 steps to approximate the target shape. These findings reinforce our main conclusion: modeling the average velocity field enables the u-param to achieve accurate and efficient one-step generation, outperforming standard flow models across various geometrically complex distributions.
Results on DDPM u-param. Figure 7 iillsstrtts that the one-step u-parameterizzation in OFQL achieves expressivity comparable to a full DDPM, despite requiring only a single forward pass. In contrast, DDPM produces highly noisy outputs when restricted and typically requires around 10 denoising steps to approximate the target distribution reliably. Importantly, one-step generation is particularly advantageous in actor-critic RL, as it eliminates the need for backpropagation through time. This makes the one-step u-parameterization a more practical choice than DDPM for policy learning, even though both approaches exhibit similar expressive capacity.
small number of steps (1-2)
D TRAINING AND INFERENCE EFFICIENCY COMPARISON
We evaluate the decision frequency and training time of our method and baseline across Mujoco tasks. The decision frequency (Lu et al.) (025b) reflects the number of actions (or action batches) generated per second by the evaluated model.
Experiments are conducted an Ubuntu server with Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz (48 cores, 96 threads) and an A100 PCIe 80 GB GPU. The wall-clock training time (in hours) is measured over 1 million training steps and averaged across the 9 Mujoco tasks. Decision frequency is measured over 3000 action batches (batch size 2500) for each task and averaged across all tasks.
E LIMITATIONS AND FUTURE WORK
The efficiency and expressivity of OFQL. position it as compelling approach for advancing real- world reinforcement learning, in conjunction with existing approaches (Venkataraman et al. 2025 Nguyen et al. 202 Luu et al. 2024 2025b Nguyen et al. 2024b a). In particular, OFQL
enables one-step step action generation while achieving of performance, thereby supporting decision-makinn
frequencies suitable for real-time robotics, autonomous driving, medical and
related domains (Ramstedt & Pal 2019 Kiran et al.) [02T]: Ouyang et al. [002)] Yoon et all, 2025
Pham et al. 2022). Its reduced training and inference cost also lowers the computational barrier for
17

---

Published as a conference paper at ICLR 2026
scaling reinforcement learning to larger datasets and more complex tasks, providing practical path toward widespread industrial adoption.
While OFQL demonstrates compelling efficiency and performance gains in offline reinforcement learning, our current evaluation focuses primarily on single-goal state-based decision-making tasks specificclll those relying low-dimensional proprioceptive observations from the D4RL benchmark. This leaves several promising directions for future exploration.
First, extending OFQL to online reinforcement learning presents natural next step. Our one-step formulation removes the computational bottlenecks that typically hinder real-time interaction, making OFQL a promising candidate for scalable online learning. Investigating stability and sample-efficiency in this setting remains an important open question.
Second, we aim generalize OFQL to iisoonbbased or point-cloud-basse control settings, where observations are high-dimensional. An open question is whether we can leverage encoder components from domain domain-specifie specific architectures for feature extraction (e.g.. (He et al.) 2016 Zhou & Tuzel 2018 Vu et al. 2022, 2023; 2021)) or whether specialized encoder tailored to OFQL should be designed. Figuring out effective architectures and integrating them with one-step flow-based policies could pave the way for end-to-end learning in more complex and unstructured environments.
Third, future work could explore extending OFQL to goal-conditioned and multi-task RL settings. Learning conditional average velocity fields to support diverse goal-directed without resorting to separate diffusion or reward models-woull offer greater flexibility and generalization.
Overall, OFQL provides a general foundation for fast and expressive policy learning, and we hope future work expands its applicability across broader domains and learning paradigms.
F DETAILED ON DIFFUSION MODELS, FLOW MATCHING
For completeness, we provide background on diffusion models and flow matching, which serve as the foundations for our method.
Diffusion Models. Diffusion models are sample from an unknown data distribution q(z") using a dataset drawn from it (Ho et al. 2020 Song & Ermon 2019 Song et al.
class of generative models that learn to
2020b SSoh--Dicsstein et al. 2015). Denoising Diffusion Probabilistic
Models (DDPMs) (Ho et al. 2020), the canonical diffusion model used in DQL, define forward x=*11) as fixed Markov chain that gradually corrupts data with Gaussian noise over K steps, where g(a* ==11) ) = N (a+; /1 x-1 ,,9* and
diffusion process (x. I:
==== -
the variance schedule [8*] ]==1 Gaussian.
[_ is predefined such that as K + x approaches an isotropic
The corresponding reverse process, enables generating data from pure noise, is parameterized by e and defined as:
K P
(p( = (x :0 I)
II
(a )*),
(19)
k=1
which is learned by maximizing the variational lower bound Egg [log (2 q( [20]
(lei al. 2017
Ho et al. 2020).
After training, sampling from q(x:)) is approximated by drawing k NO, I) and applying the reverse Markov chain from & = K to k via the learn model pa. Conditional generation is straightforwardly supported via P.p (2
c)).
Flow Matching, Flow Matching (FM) (Lipman et al. 2022 a generative modeling framework that learns deterministic velocity fields to directly transport noise to data along smooth, stable trajectories.
Given data x ((x) and noise Pprior FM defines linear flow path:
dz:
24 a, + Bie, V2 di â + R,c,
(20)
where a2 Br are predefined schedules (e.g., a₂ = t, B, = t), and the dot notation (e.g., dz)) denotes the time derivative with respect to the continuous flow step & [0, 1]. The conditional
18

---

Published as a conference paper at ICLR 2026
velocity (2 æ) captures the direction of flow for specific sample, and the marginal velocity field defined as the expectation:
u t) E (oe (a.))
(21)
FM essentially models the marginal velocity, as it is feasible to approximate this field; parametrized by the neural network up (Zx, ()uuiing the Conditional Flow Matching loss:
CCFM (0) = =., (1,, t) (z =)]l"
(22)
where, under the commonly used schedule oz 1 t, _3 t, the conditional velocity simplifies to (e(2; .) = c x.
In inference, sampling is performed by solving the ODE in reverse time:
dæt
di v22,,), starting from 2 Periur(e),
(23)
where the solution is approximated using numerical solver, such Euler's method: Zt-At. At v(2,,)) t).
On the classifier-free guidance While classifier-free guidance (CFG) might be considered to better align generated samples with the conditioning variable с in the image generation domain, prior work has shown that CFG can lead to undesirable behaviors in sequential Specifically, CFG tends to bias the generation process toward high-density regions associated with c, which may cause agents to overlook high-return trajectories critical for long-horizon planning (Pearce et al., (023). Additionally, DQL adopts no-guidance approach. For a fair comparison, we tollow the design choices made in DQL and prior works (Chi et al.) [2023] Wang et al. 2022 and adopt nooguidance paradigm, ensuring stable and unbiased policy generation.
macksinn-making tasks.
G FORMAL JUSTIFICATION OF ACTION ACCURACY PRESERVATION IN ONE-STEP GENERATION THROUGH AVERAGE VELOCITY FIELD
We show that under perfect learning (no estimation error), learning the conditional average veloc- ity ua(at, r, 8) and applying the one-step update in Eq. 12 recovers the same endpoint map as integrating the conditional Flow Matching dynamics, thereby enabling accurate one-step action generation.
Let o² (a, 8) ) be the ground-truth conditional marginal velocity field assumed to govern the Flow Matching (FM) dynamics that generate the target action distribution Hu(- s). In FM, this velocity field transports Gaussian noise to data through the ODE
da: 0² (at, 1;
= 1).
(24)
dl
Solving this ODE backward from - sample e and the conditioning state 8. This defines the FM endpoint map
0 yields an endpoint ao that depends both the noise
(* a0= = - J. o² (ar, T;8) dr.
dr.
(25)
The forh-forward of this map over the Gaussian prior, (T*) recovers the target action distribution uu(- s).
For any interval [r. f| C [o, 1), define the average velocity as
a² (ar, r. r,t:s) - J (ar,
t-r
(26)
which represents the net displacement of the FM trajectory over this interval. Applying this definition [0, 1] gives
u (a1,, = n' =,,,0,,,,,,;; (c,0, = L'' v² (or,
(27)
19

---

Published as a conference paper at ICLR 2026
and therefore the endpoint map satisfies
(c, S (t) -
Suppose learned model ug (e.g., model by neural network) approximates this average velocity u" on the support of the Gaussian prior without estimation error. In that case, the one-step generator Eq. becomes:
Te s) = uo(c, cc -c- -
(28)
Because both maps push the Gaussian prior forward in the same way. their induced action distributions coincide:
(To) (To) (T ( ml- s).
Hence, the learned policy
TT(a {8) - (Te) (0,
matches the target distribution exactly, demonstrating that the learned average velocity preserves the FM action accuracy in a single forward pass without ODE integration.
H FORMAL JUSTIFICATION OF AVERAGE-VELLOCTE LEARNING ENCOURAGES THE LEARNED ONE-STEP POLICY TO STAY CLOSE TO THE BEHAVIOR POLICY
We show that, under general (imperfect) learning conditions, minimizing the average-velocity match- ing loss LFBC (0) ensures that the learned one-step policy Ta(- | converges toward the behavior distribution ju(- s). In particular, Lrde. (f) upper-bounds the squared 2- 2-Wasserstein distance be- tween n-(- s) and ju(- s) up to positive constant- constant-implying enforces closeness between the two distributions.
implying that small average-velocing error
Let (O, Ta) be d-dimensional standard Gaussian. For each state S, define
(* s) () -e-
v0 ff- (ar)
u (e, 1;8); To(e, uole,, 0, 0, 1;8),
so that the induced action distributions are the push-forwan
( (T* (((,)))((00,
TO(- ()) (T)
Recall the average-velocity matching loss:
Lree. (0) = Ro [l (as, r, 1; - u (at, t, +)
(29)
where de is the (deterministic) solution of the flow-matching ODE at time t given the initial noise and state s.
Assume that the sampling distribution over time pairs (r, assigns non-zero probability poi to the endpoint pair (0, 1). i.e. P((r, = (0, 1)] = Poi > Then
CPBC. (0) ((rt))s (a). r, s) at (a, s)
(30)
2 Poi (r.10) / [ 4:8) : "t (,,,,,
(31)
= Poi Eg, (
(32)
Using the endpoint parameterization
To s) uo(e, 0, 1)s)
(" = u² 0c,0,1;8)
we obtain the identity
| (e, 0,158) - -u" (e, 1; 1) [[To(e,)) T s))
20

---

Published as a conference paper at ICLR 2026
Thus
Cyec. (0) 2 Po Po [T)(( T (c, s)]1
(33)
For each state let X2 denote the joint distribution of (To(e, s), T (c, s)) induced by (N0, 7). Then X is valid coupling between TT(- s) and u(. s). i.. X A (TT( [)) s)). Therefore,
TR [IT((E,)) T (c 1 )~) [||a a
(34)
x=X(ml(-bb.u((a))) ((aaaa)~> [l|a 1
inf
(35)
W3 (- (,), pl- (()]
(36)
where W2 denotes the 2- 2-Wasserstein distance on the action space with Euclidean ground metric. Combining the inequalities yields
Lrec. (0) 2 poi E. [ (To( (( [+)]]
(37)
Thus, up the positive constant factor, LFBC (0) upper-bounds the expected squared 2- Wasseesstein distance between the learned policy Te(- s) and the target policy pa( (s) induced by flow matching.
In particular, if Croc- (0) + 0, then Egg [W3 (To( {8), uu(- s))] + 0. Consequently, average-
velocity learning regularizes Th () toward the behavior distribution μ(- [s), while still allowing complex, multimodal action distributions via the nonlinear endpoint map induced by flow- fow-matching dynamics.
GRADIENT ANALYSIS OF OFQL ACTOR LOSS
The OFQL actor minimizes
arg min [(0) = arg min (CrBc(0) minnnn a (0))
(38)
8
(Q A Eg-D,avte
aras ( [s,(s
(39)
The OFQL actor loss jointly (i) maximizes the critic value (i.e., return) and (ii) keeps the policy close to the behavior distribution via FBC (see formal justification in
We now expand the gradient of each term.
Gradient of the Q-term.
Recall that actions are sampled in one step:
(,(e,0,1;5)) N0,
(40)
The Q-term becomes
LQ(0) = Ex,
(41)
Applying the chain rule:
VoCo(O) E,,, [aRo(s, Voa
(42)
E.,c V.Qo(s. (oVVool...---ooo(. 0, 1;
(43)
Unlike diffusion-based policy parameterizations that require backpropagating through many iterative denoising steps (BPTT), the one-step mapping transformation. Thus, Vea is computed in one step without temporal unrolling, making the actor update significantly faster, more training-friendde
up(c, 0, 1: is single differentiable
Gradient of the FBC term.
The FBC objective
LPBc (0) = Exa,ter [Ilu((e, (a,,T, sg( (g(
(44)
where Uegt is stop-gradient sg(-):
at = (1 - t)a + te, Because of sg(.). the target is treated constant when differentiating. Thus
Uegt = 12 - (t - r) (vi da, us + acus),
11 = c a
(45)
V (OC 2 E. a,trr (uuo 12 uegt) Voug(as, 1;8)]
(46)
21

---

Published as a conference paper at ICLR 2026
Full OFQL actor gradient.
Combining Eqs. 42 46
VeL(O) = VeCFBC(O) - -V
(47)
= 2 2 au t; unge) Vous r, t)
(48)
+ a B,, (...,..V..(...... (oQo(s,a) Vouole, 1;))
(49)
Interpretation. The first term regularizes the policy toward the behavior distribution by match- ing average velocities, while the second term regularizes the policy to maximize the critic value through the differentiable one-step action mapping. Thus, OFQL simultaneously achieves behavior regularization and return maximization.
J EVALUATING OFQL IN HIGH-DIMENSIONAL ACTION ROBOTIC MANIPULATION
To further evaluate OFQL in high-dimensional action spaces, we conducted additional experiments on the D4RL Adroit benchmark, which features 24-dimensional control using dexterous robotic hand. We evaluated two standard tasks tasks-adroit-pe setroit-pen-humene and pewriitpee-cloned--ae where the objective to manipulate pen to match target orientation using 24-DoF hand. This domain is particularly challenging due to noisy human demonstrations, sparse rewards, and the high-dimensional action manifold. Normalized returns, following the evaluation protocol of Fu et al. (2020), reported below:
Task
BC
DQL
OFQL (ours)
adroit-pen-human pen-human
71.0
75.7
79.5
adroit-pen-cloned
52.0
60.8
62.3
The results show that OFQI. consistently outperforms both BC and DQL in high-dimensional manipulation tasks, demonstrating strong robustness and effectiveness in complex dexterous control settings.
K FEASIBILITY ON VISUAL OBSERVATION SETTING
To demonstrate the feasibility of OFQL in the visual-observation setting, we evaluate it on two OGBench (Park et al. (0024) visual manipulation tasks that require reasoning over high-dimensional image observations (64x 64 3): vi wwwceeeesssnnnn wew.eeeesielllllttttttaaas wwwssskktssa-k---a - (moderate diffi- culty) and visual-cula www--cuuuuuuubleeeee sauuuble-ply--aaee www.sssiinngnaataaaneeee tassskttsskkl-aaa -v0 (hard). We adopt the small IMPALA encoder (following FQL (Park al. 2025)) for embedding the image observation to the latent state and use simple concatenation for state conditioning. Task success rates are reported
below:
Task
FBRAC
FQL
OFQL (ours)
visual-scene- rissual-scene-siigllttss-ttttt singletask-taskl--aea visual-cube-doue double--dooee--eoe cooubbe-plaa-singltttttt siayy--iinnlttss-aaaakkk
46.0=4.0
98.0133.0
54.019.0 8.0.13.0
6.0+2.0
21.0=11..0
These results show that OFQL remains functional in visual settings, but its performance lags behind stronger visual baselines, indicating that additional architectural and algorithmic considerations are necessary for competitive results in high sish-diennsioaal pixel-basee domains.
There are several key challenges when extending OFQL to high-dimensional input scenarios such as image-based observations.
First, the learning objectives become tightly coupled. Unlike now-dimensional state spaces, visual tasks require the policy to jointly learn (i) accurate Q-values, (ii) flow-based behavior regularization, and (iii) a stable and expressive visual representation. These components are deeply interdependent: noise instability in the visual encoder propagates into Q-value estimation and flow predictions,
22

---

Published as a conference paper at ICLR 2026
while inaccuracies in the critic policy can, in turn, misguide the encoder. This tight coupling makes the overall optimization process considerably more fragile compared to low-dimensional settings.
Second, conditioning high-dimon dimenssonn latent features into the flow network is non-trivial. Simple concatenation of visual latents with the noise vector may be insufficient. figh-ddmenniioal representa- tions often require more structured fusion strategies e.g. FiLM layers, mosss-attention...- to ensure the visual features meaningfully influence the learned flow direction. Without proper conditioning, the policy may ignore or underutilize visual information.
Third, representation quality may becomes a bottleneck. Lightweight or general-purposs encoders may fail capture task-relevant spatial and semantic cues required for precise action prediction. Stronger or task-specific visual backbones, domain augmentations, or auxiliary representation- learning losses (c.g.. predictive coding (Nguyen al. 2021b), contrastive learning (Luu al. 2022
Chen et al. 2020: Nguyen et al. 2023).... may be necessary to maintain stable training.
Overall, extending OFQL to visual domains will likely require more robust encoders, improved conditioning strategies, and additional guidance signals to ensure that visual features effectively support flow-based policy learning which an interesting direction for future work.
L HANDLING OUT-OF- STATES: OFQL vs. DQL
Across all benchmarks, OFQL consistently attains higher average returns than DQL. This suggests improved robustness to out-of- out-of-distribution (OOD) states, since the average return is determined by the policy's 's interaction with the real environment, where trajectories often drift outside the trained distribution. A policy that performs better in these interactions is implicitly better at handling such OOD states.
To further support this, we compute the mean squared error between the actions produced by the trained policy (trained on medium medium-replay datasets) and the expert actions on the expert dataset. Let's denotes this MSE as OOD-MSE. This metric measures how well the learned policy aligns with the expert policy under the expert state distribution, which is largely out-of-distribution relative to the training data. A lower OOD-MSE therefore indicates stronger generalization unseen or OOD states. we provide OOD-MSE on the HalfCheetah dataset as below.
Metric (Dataset) OOD-MSE (Medium) OOD-MSE (Medium-Replay)
OFQI. DQL
0.458
0.560
0.462
0.582
The results presented in the table above show that OFQL consistently achieves lower OOD-MSE than its DQL counterpart, demonstrating that OFQL generalizes more effectively to unseen or out-of-distribution states.
M ABLATION ON TIME-SAMPLING DISTRIBUTION
We evaluate the effect of the time-sampling distribution in OFQL by comparing uniform sampling against a logit-normal distribution. An ablation on HalfCheetah is summarized below:
Time-Sampling Uniform
Logit-Normal
Medium-Fxpee 94.5++0.5
95.2 0.4
Medium 61.1++0.1
63.8 - 0.1
ReepuuyRepplay
51.7+t0.2
51.2=0011
Overall performance similar across the two strategies, though the logit-normal distribution yields a slight improvement some datasets. These results show that OFQL remains robust under different time-sampling strategies, and performance is not highly sensitive to the precise tuning of this hyperparameter. In practice, use logit-normal parameters (μr -0.4, 1.0) the default.
23

---

Published as a conference paper at ICLR 2026
N BASELINES REPRODUCIBILITY
Baseline Result. For DQL and FQL. on AntMaze, we directly report the results from the original papers. For other baselines--including BC, TD3-BC, IQL., Diffuser, DD, EDP, and IDQL-we results from the broadly accepted and standardized reimplementation CleanDiffuser (Dong et al.) 2024). For details training and evaluation procedures, we refer readers the corresponding papers.
For the FQL Locomotion and Kitchen, the official FQL implementation does support the D4RL. Locomotion or Kitchen domains. To ensure fair comparison, we extend the official JAX codebase to support these environments and additionally implement a PyTorch version of FQL within the OFQL framework (our implementation is based on PyTorch). We follow the recommendations from the official FQL repository and paper: we use the normalized- setting and perform hyperparameter search over E {0.03, 0.1, 0.3, 1, 3, 10), described in Appendix C of Park et al. (025)). For network architecture, we search over MLP sizes [512, 512, 512, 512] and [256, 256 256, 256]. We run both our extended JAX version and our PyTorch implementation and report the best-performing results. For speed measurements, we use the Py Torch version to avoid framework-led differences (JAX vs. PyTorch).
Diffusion Steps. We follow the standard diffusion-step settings recommended by each baseline: DQL (5 steps), IDQL (5 steps), EDP (15 steps), and the Flow Model used in FQL (10 steps). These configurations align with the settings reported in the respective papers or official repositories.
O ON THE MEANFLOW IDENTITY
For completeness, the derivation from (Geng et al. 2025) is revisited to provide a clear understanding of how McanFlow Identity can be used to calculate the target average velocity.
Let's consider the no-condition generation case (no state condition) for simplicity. The average velocity is defined as the displacement between two timesteps and divided by the time interval:
1 u(a,, --r o(ar,T)dr.
J.
(50)
Here, denotes the average velocity, e the instantaneous velocity (i.e., marginal velocity), and is the noise action. As + f, u converges to v.
Our purpose is to approximate using neural network, enabling single-step generation (i.e., r 0, = 1), unlike methods based on marginal velocity (a.k.a instantaneous velocity), which require time integration inference. Direct training with impractical due to the integral; instead, the definition of u is manipulated to derive a tractable optimization target.
The MeanFlow Identity. To facilitate training, the equation for is rewritten as:
(t - (1-r)ula,ra L. v(arr,)dd.. r)dT.
(51)
Differentiating both sides with respect to gives:
d - (t r)r(aa r,t) dt J. u(a,, = a((a,,,t) (t-r) d u(a,,,,t) v(a,,t) (52)
d
dt
Rearranging, the MeanFlow Identity is achieved:
d u(a,, r, () = v(ar, () - (1 u -(
(53)
This identity links and providing target for training neural network. The next step is to compute the time derivative u.
Computing the Time Derivative. To compute
d di u, we expand the total derivative:
d
da: dt
dr
dt 8,u dt
u(a
r,
Do,u u +
0,u +
(54)
dt
dt
24

---

Published as a conference paper at ICLR 2026
Using dar t), dr di O, and dt 1, we obtain:
d u (e, r, v(a,, l)0_u 0,m dt
This shows that the total derivative of is computed as the Jacobian-vector product (JVP) of the network's 's Jacobian and the tangent vector [1, 0, 1]-
Notably, the MeanFlow Identity (Eq. 53 is mathematical equivalent to Eq. 51
Geng al. 2025
We train the policy network by conditioning on the state 8, parameterizing uo(a, r, t; 8), and applying the MeanFlow Identity to define the optimization target.
P LLM USAGE
In preparing this paper, used Large Language Models (LLMs) solely an assistive tool for grammar checking and polishing text. The LLMs were not involved in research ideation, experimental design, data analysis, or substantive content generation. All research ideas, methods, analyses, and conclusions are the authors' own.
25


Reviews:
[{"review_id": "IM1RfFoaSK", "reviewer": "Reviewer_opYF", "summary": "To overcome the limitations of DQL, the paper proposes replacing the multi-step denoising policy used in training and inference with a one-step denoising policy. Unlike other one-step approaches that require an auxiliary teacher network for distillation, the paper adopts a mean-flow policy that directly approximates the denoising process. The proposed method demonstrates strong empirical performance and improved efficiency.", "questions": "The model performs worse on the Kitchen and AntMaze-Large-Diverse tasks, which are relatively more challenging within the D4RL benchmark. Do the authors have any insights into these results?\nCould it be that the mean-flow policy limits exploration during training, leading to reduced performance on tasks requiring greater stochasticity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is simple, clear, and effective. By replacing only the diffusion policy component with the mean-flow policy, the approach achieves both higher sampling efficiency and competitive performance. The toy example nicely illustrates the advantage of reparameterizing from $v$ to $u$, providing a clearer intuition for the underlying mechanism.", "weaknesses": "Given that mean-flow generative modeling has already shown strong one-step FID results on image generation tasks, it would be valuable to see this approach applied to more complex environments beyond D4RL, such as robotic control or high-dimensional decision-making settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYqzH3nxrW", "reviewer": "Reviewer_8FUi", "summary": "This paper proposes **One-Step Flow Q-Learning (OFQL)**, a novel framework for offline reinforcement learning that reformulates Diffusion Q-Learning (DQL) within the **Flow Matching (FM)** paradigm. By learning an **average velocity field** instead of a marginal one, OFQL enables accurate **one-step action generation** during both training and inference—eliminating the need for multi-step denoising and recursive backpropagation. This design substantially improves training and inference efficiency while maintaining, and even improving, performance. The authors demonstrate strong results across D4RL benchmarks.", "questions": "1. Could the authors provide a more formal justification for why **average velocity learning** preserves accuracy in one-step action generation?\n2. Are there scenarios (e.g., highly multimodal action distributions) where the **average velocity** assumption might underperform?\n\nTypo: citation in line 151", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Clear conceptual advancement:** Reformulating DQL under the flow-matching framework and introducing an average velocity field is a novel and elegant idea that directly addresses the core inefficiency of multi-step denoising.\n* **Simplicity and effectiveness:** Unlike prior one-step approaches that depend on auxiliary modules or policy distillation, OFQL remains conceptually clean while achieving superior results.\n* **Strong empirical results:** The method outperforms DQL and other diffusion-based baselines by a significant margin on D4RL, demonstrating both **efficiency** and **robustness**.\n* **Illustrative toy example:** The toy experiment effectively clarifies the intuition behind the average velocity field and supports the main claim.\n* **Readable and well-organized:** The paper is well-written, clearly structured, and easy to follow even for readers not deeply familiar with flow-matching methods.", "weaknesses": "* The theoretical justification for why learning an **average velocity field** leads to better one-step performance could be elaborated further. Currently, the paper provides an intuitive explanation but lacks a deeper analytical connection to diffusion dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfxPpPDIci", "reviewer": "Reviewer_QtuL", "summary": "This paper proposes One-Step Flow-Q-Learning, an offline RL algorithm that enables one-step action generation during both training and inference. The method is closely related to DQL but leverages the average velocity parametrization following MeanFlow. The action sampling in this parametrization can be done in a single step, which reduces inference cost and avoids the huge computation and memory costs of backpropagation through the multi-step sampling chain. Experiments on the D4RL benchmark show that OFQL surpasses all other included baselines in overall performance.", "questions": "1. How many diffusion steps are used for the multi-step diffusion policy baselines? Is the number aligned with the original papers?\n2. Can the proposed method be extended to visual observation tasks? Are there any challenges for the method in high-dimensional input scenarios?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method shows empirical advantages in policy performance, training speed and inference time. \n2. The paper is easy to follow.", "weaknesses": "1. The proposed method lacks novelty. The only main difference between the proposed method and DQL is replacing the diffusion loss in actor training with a MeanFlow loss.\n2. The experiments are not adequate. Only results on state-based D4RL tasks are included, and no visual observation task results are reported.\n3. The argument in Lines 262-264 is not clear. Flow matching cannot \"in principle, enable one-step generation\", as the sampling trajectory is straight only when the target distribution is a delta distribution or when rectification or similar techniques have been used. The following sentences in this paragraph are accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldoQrzIobr", "reviewer": "Reviewer_43m1", "summary": "To address the slow multi-step denoising and unstable optimization inherent in diffusion-based policies, this paper introduces One-Step Flow Q-Learning (OFQL). OFQL reformulates the diffusion denoising process within the Flow Matching (FM) framework and learns an average velocity field that enables direct one-step action generation. Experiments across diverse D4RL tasks demonstrate that OFQL achieves the highest average normalized scores among all compared methods. Moreover, its one-step sampling design substantially improves both training and inference efficiency.", "questions": "1.\tDoes learning an average velocity field constrain the representational power compared to DDPM’s full reverse process?\n\n2.\tHow sensitive is OFQL to flow ratio and time-sampling distribution?\n\n3.\tDoes one-step flow matching better handle out-of-distribution states than diffusion-based DQL?\n\n4.\tHow exactly does the Q-gradient interact with flow learning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "OFQL is an efficient reinforcement learning algorithm that introduces average velocity fields within the Flow Matching framework, enabling it to model complex policy distributions without relying on distillation procedures or auxiliary networks. The method offers a unified training–inference pipeline, using the same one-step model consistently in both phases. Empirically, OFQL outperforms DQL and other strong baselines in terms of both policy performance and computational efficiency.", "weaknesses": "1.\tOFQL relies on the MeanFlow Identity to enable one-step sampling for the learned policy. However, the Jacobian–vector product computation in Eq. (11) may become computationally demanding for large-scale models.\n\n2.\tAs acknowledged in the paper’s limitations, it remains unclear whether OFQL can scale to high-dimensional action spaces (e.g., humanoid control or vision-based RL). Moreover, the stability of the proposed one-step policy under non-stationary or online settings has not been investigated.\n\n3.\tThe paper provides no formal analysis establishing the expressive equivalence between the average-velocity one-step formulation and traditional multi-step diffusion policies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)