Flow Q-Learning
Authors: Seohong Park
Qiyang Li
Sergey Levine
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
Primary area: reinforcement_learning->batchoffline

Abstract: We present flow Q-learning (FQL), a simple and performant offline reinforcement learning (RL) method that leverages an expressive flow-matching policy to model arbitrarily complex action distributions in data. Training a flow policy with RL is a tricky problem, due to the iterative nature of the action generation process. We address this challenge by training an expressive one-step policy with RL, rather than directly guiding an iterative flow policy to maximize values. This way, we can completely avoid unstable recursive backpropagation, eliminate costly iterative action generation at test time, yet still mostly maintain expressivity. We experimentally show that FQL leads to strong performance across 73 challenging state- and pixel-based OGBench and D4RL tasks in offline RL and offline-to-online RL.

Flow Q-Learning
Seohong Park
Qiyang Li
Sergey Levine"
Abstract
BC flow policy po(s,
We present flow Q-learning (FQL), simple and performant offlin reinforcement learning (RL) method that leverages an expressive flow- matching policy to model arbitrarily complex ac- tion distributions in data. Training flow policy with RL tricky problem, due to the iterative nature of the action generation process. We ad- dress this challenge by training expressive one- step policy with RL, rather than directly guiding an iterative flow policy to maximize values. This way, we can completely avoid unstable recursive backpropagation, eliminate costly iterative action generation at test time, yet still mostly maintain expressivity. We experimentally show that FQL leads to strong performance across 73 challeng- ing state- and pixel-based OGBench and D4RL. tasks in offline RL and offline-to-onling RL. https: //t::///sooooogo me/ mepprojects//ql]
Distill
One-step policy (s,
t=
t 1
Figure I. Flow Q-learning. Flow-matching policies can model complex action distributions, but training an iterative flow policy with RL challenging. To address this, train expressive one-step policy [s, 2) S x TR + A to maximize Q values, while regularizing with distillation from BC flow policy.
to denoising diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020). By employing an expressive flow policy, we can ef- fectively model the arbitrarily complex action distribution of the dataset and thus enforce an accurate behavioral con- straint, which is central to many offline RL algorithms (Nair et al., 2020; Fujimoto & Gu, 2021; Tarasov al., 2023a).
1. Introduction
Offline reinforcement learning (RL) enables training ef- fective decision-making policy from previously collected dataset without costly environment interactions (Lange et al., 2012; Levine et al., 2020). The essence of offline RL constrained optimization: the agent must maximize re- turns while staying within the dataset's 's state-action distribu- tion (Levine et al., 2020). As datasets have grown larger and more diverse (Collaboration al., 2024), their behavioral distributions have become more complex and multimodal, and this often necessitates an expressive policy class (Man- dlekar et al., 2021) capable of capturing these complex dis- tributions and implementing a more precise behavioral con- straint. In this work, we aim to develop a scalable offline RL method by leveraging flow matching (Lipman et al., 2023; Liu et al., 2023; Albergo & Vanden- 2023), sim- ple yet powerful generative modeling technique alternative
However, leveraging flow or diffusion models to parameter- ize policies for offline RL is not trivial problem. Unlike with simpler policy classes, such as Gaussian policies, there is no straightforward way to train the flow or diffusion poli- cies to maximize learned value function, due to the itera- tive nature of these generative models. This is an example of a policy extraction problem, which is known to be key challenge in offline RL in general (Park et al., 2024a). Pre- vious works have devised diverse ways to extract an itera- tive generative policy from a learned value function, based on weighted regression, reparameterized policy gradient, re- jection sampling, and other techniques. While they have shown promising initial results, these extraction schemes are often limited or not necessarily scalable more com- plex problems, due to their inherent drawbacks (e.g. unsta- ble backpropagation through time, limited use of samples, and high computational cost; Section 4.1).
University of California, Berkeley. Correspondence to: Sco- hong Park <scohong@ @berkkeey.rck........
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, we propose a simple and effective way to leverage an expressive flow policy for offline RI. Our main

---

Flow Q-Learning
idea is to train an iterative flow policy only with behav- ioral cloning (BC). Instead, we train a separate, expressive one-step policy that maximizes values while distilling from the flow model (Figure 1). By lifting the burden of value maximization from the flow model, we completely avoid the problems associated with steering the iterative process, while fully leveraging the expressivity of the flow model. Moreover, this procedure yields an expressive one-step pol- icy as the output, which eliminates costly iterative flow steps at evaluation time. We call this method flow Q-learning (FQL), which constitutes our main contribution.
et al., 2023a) is one of the simplest (yet effective) offline RL frameworks. In its most basic form, it minimizes the following actor-critic losses:
(Ø( E B,,,,r.'' (ea.rs''D,,1Q0(ss.... [(Qo(s,) -r (r- a a'mzo
(+) = Bs ~ay
| QQ((s,a)) - a log i(a $) (2)
lose
BC loss
where Qo(s, a) SSAA - R is a state-action value function with parameter Φ. Qs(s, a) is a target network (Mnih et al., 2013), a is a hyperparameter that controls the strength of the behavioral cloning (BC) regularizer, and s, a, r, s' ~ D denotes uniform sampling over the dataset's 's transition tuples. Intuitively, the critic loss (o( minimizes the standard Bellman error, while the actor loss L (0) maximizes values with reparameterized gradients through a", For the actor, the BC loss is additionally applied to prevent the policy from deviating too much from the behavioral policy's distribution.
FQL is simple: thanks to the simplicity of flow matching (especially compared to denoising diffusion), it can be im- plemented within a few lines on top of the standard actor- critic framework (Algorithm 1). Yet, FQL is highly effective and efficient. Especially on complex tasks involving highly multimodal action distributions, FQL often leads to signifi- cantly better performance than both Gaussian and diffusion policy-based offline RL methods, without requiring itera- tive flow steps at test time. Moreover, FQL can be directly fine-tuned with online rollouts, often outperformng exist- ing offline-tooonliee RL methods. We empirically show the effectiveness of FQL on 73 diverse state- and pixel-based tasks across the recently proposed OGBench (Park et al., 2025) and standard D4RL (Fu et al., 2020) benchmarks.
The policy is typically modeled by Gaussian distribution to enable effective reparameteriizttio. Perhaps surprisingly, despite its simplicity, bebavior-regularized actor-critic is one of the most performant frameworks on standard D4RL tasks (Tarasov et al., 2023a). In this work, we build our flow-based offline RL method on a variant of the behavior- regularized actor-critic framework.
Flow matching. Flow matching (Lipman et al., 2023; Liu et al., 2023; Albergo & Vanden-Fijjden 2023) is sim- pler alternative to denoising diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) for training itera- tive generative models. Unlike denoising diffusion models, which are based on stochastic differential equations (SDEs), flow models are rooted in deterministic ordinary differen- tial equations (ODEs), which enable significantly simpler training and faster inference, while often achieving better quality (Esser et al., 2024; Lipman et al., 2024).
2. Preliminaries
Offline RL. In this work, we assume a Markov deci- sion process M (Sutton & Barto, 2005) defined by a tu- ple (S,A, P.P), where S is the state space, A Rdd is the d-dimensional action space, r(s,a): S x A + IR is the reward function, p(s) E A(S) is the initial state distribution, and p(s' s,a) S x A - ASS) is the transition dynamics distribution, where we denote the set of probability distributions over a space X' as A(X) and use gray to denote placeholder variables. The goal of of- fline RL is to find the parameter 0 of a policy (a s) S - (AA) that maximizes the average discounted return R( ((TT) = (E EE 7"r(s,, an)] from a dataset D = (ee")))e
Given a data distribution p(z) E A A(R*) on a d-dimensional Euclidean space, flow matching aims to fit the parameter O of a dime-dependent velocity field uo (t,,, 2) : [0, 1] I + Rd such that its corresponding flow (Lee, 2012) ()((, [0, 1] IRd Hdd, defined by the unique solution to the ODE
without environment interac- tions, where - denotes a trajectory (5o, ag ..., SH, az). y denotes a discount factor, and (r) is defined as p(so)To(ug) so)p(s1 so, 00) (...)))).lllf..... (a 8n). In this work, we also consider offline-to-online RL, whose goal is est amount of online environment interactions.
d vo(t, z) = uo (t, x)),
(3)
dt
transforms a simple distribution (e.g., unit Gaussian) at = 0 into the target data distribution p(x) at = 1.
to further fine-tune the offline pre-trained policy with mod-
In this work, we consider the simplest variant of flow match- ing based on linear paths and uniform time sampling (Lip- man al., 2024). The objective is as follows:
Behavior-regularized actor-critic.' Behavior-regularize actor-critic (Wu et al., 2019; Fujimoto & Gu, 2021; Tarasov
Herr, we use the term "behavior-regullrized actor actor-critic to refer to a general framework encompassing a family of approaches, not solety the specific BRAC method (Wu et al., 2019).
min 8
E [.o [I - (a* - -x0)1
(4)
2

---

Flow Q-Learning
where (N Ia) is the d-dimensional standard normal dis- tribution, Un, 1)) denotes the uniform distribution over the unit interval, and xf (1 t)x" + tx! is the linear in- terpolation between z and x' Intuitively, the velocity field Ny is trained to match the average direction from randomly sampled z" and z². At optimum, this objective produces a vector field that generates the data distribution p(r).. At in- ference time, we generate samples by numerically solving the ODE defined by vg. In this work, we use the simplest Euler method, which we find to be sufficient. See Lipman et al. (2024) for further details about flow matching.
BC 1oss
BPTT
π Flow
loaa
(a) Naive approach.
BC loss
T BC flow
loss
One-step
(b) Our solution.
Figure 2. The idca. Offline RL is essentially a tug-of-war between behavioral regularization and value maximization. (a) Naively do- ing this with flow policy involves costly and unstable backpropa- gation through time (BPTT). (b) We resolve this by training a sepa- rate one-step policy, which maximizes values without BPTT while being regularized by a distillation loss from a BC flow policy.
Flow policies. In this work, we use flow matching to train policies. The most basic flow mwwthhiggngg objective for behav- ioral cloning is as follows:
Crlow (0) E s,a=at>,, [Ilvo(t, 8,,") - (x² x²) )1) x ((),aa
(5)
w~ (n.1)(
where es (t, s, : 00, 1] x S Rd + Rd is state- and time-dependent vector field with parameter 0. Recall that A is defined as Rd, and flow matching happens in the action space. The state-dependent vector field generates a state- dependent flow se s) m:: : [0, 1] x S x Rd + Rd, which serves as a policy. For 8 E S and & E Rd, we simply denote the ODE's output the [1, s, 2) by (s, 2). Intuitively, μsg maps the noise R (sampled from the standard normal distribution) to the action a = ( 2) by the ODE.
for diffusion policies. However, unlike the Gaussian case, the flow or diffusion objective requires backpropagation through time in the Q loss (Equation (6) due to the recur- sion in numerical ODE solvers (e.g., the Euler method) (Fig- ure 2a). Unfortunately, this is often unstable and costly in practice, potentially leading to suboptimal performance, as we will show in our experiments.
Solution. Our main idea is to not steer the original flow policy at all. Instead, we will train the flow policy only with the BC loss, and train a separate expressive one-step policy to maximize the value function while regularizing it by a distillation loss from the full BC flow policy. Since the one- step policy does not involve any iterative procedures, we can completely avoid backpropagation through time in the Q loss (Equation (6)). We call this idea one-step guidance.
Notational warning: Note that (s 2) is deterministic function from S x IRdd to A, but serves as stochastic policy from S to A due to the stochasticity of Z ~ NO0, Ia). We denote the corresponding induced stochastic policy as (a(a s), and loosely refer to both μi and me as "policies." "
3. Flow Q-Learning
( 2)
We now introduce our method for effective data-driven decision-making flow Q-learning (FQL). Our desider- ata are twofold: we want to leverage an expressive flow- matching policy to deal with complex behavioral action dis- tributions; we also want to keep the method as simple as pos- sible so that practitioners can easily implement and use it.
Flow policy (OEE)
2
(1 2)
One-step policy
Naïve approach. Perhaps the simplest way train flow policy for offline RL is to replace the BC loss with flow- matching loss (Equation (5)) in the behavior-regularzzed actor-critic framework (Equation (2)). Formally, this naïve approach minimizes the actor loss (0) defined by
Figure 3. One-step policy. The one-step policy pa, leams the di- rect mapping from to of the flow policy μa, while simultane- ously maximizing values (this part is omitted in the figure).
(0) = BewD.a aooo l (a((s,a") a) + Flwwlow() (0)
(6)
More formally, we train flow policy μn (s(s,2) 2) only with the BC flow-matching loss (Equation (5)). Alongside, we train one-step prediction model (s, 2) S x IR + A with parameter w, whose main role is to learn the direct mapping from noise & to the output action of the full ODE flow policy a = or (8, 2), while simultaneously maximizing
Q loss
BC loss
Intuitively, the corresponding flow policy Ta is "steered" to maximize the value function while minimizing the BC loss. This is analogous to Diffusion-QL.. (Wang et al., 2023)
3

---

Flow Q-Learning
Algorithm 1 Flow Q-Learnin (FQL)
Remark:
Connection to Wasserstein Regularization
function (s) for 1 = 1......... ................................ M 1 do
BC flow policy
Our distillation loss in Equation (7) has an intriguing con- nection to Wasserstein behavioral regularization. Let & be random variable following the d-dimensional stan- dard normal distribution, (0, Ia). For s 8, let To(s), TTw (s) E ((A) be the push-forward distributions of by (s,-) -) and (s, respectively. Then, the distillation loss in Equation (7) is an upper bound on the squared 2- 2-Wasserstein distance between (w(s) and (s):):
2 -- 2 + uo (1/M, M
> Euler method
return
while not converged do
Sample batch {(5, a, a')} ~ D
> Train critic Q+
2~ (0, Ta)
a' (8),
Update @ to minimize -
b. Train vector field uo in BC flow policy To
x² ~ La)
L (w) = =~N((0,a))
anD,
[ll - mo(s, =)
x' +
Unif([0. 11)
x¹ + (1 1)e² + +1o
2 B
inf (EA(Fu
Ex v111
Update 8 to minimize (f 8,," 10-x=====> - xx0)
= BB
W2 (w,y To)
b Train one-step policy Tw (V.
where A(Tw To) denotes the of coupling distributions of nh and Tyy and W2 denotes the 2- Wassersteei distance with the Euclidean metric in the action space.
a" Ma. (x,
Update w minimize E[- (I-[Qoss, al), +alaa" - (s, =)]
return One-step policy Tw
values (Figure 3). The distillation loss is defined as follows:
Table Behavioral regularizers in offfine RL.
Offline RI Method Behavioral Regularizer
Metric-Aware?
(w) = E B> Ja,
(s, (s(s, 2) (7)
TD3+BC AWAC CQL
DEL
XXXO
DEL
x²
Recall that (s 2) denotes the output of the ODE defined by the vector field Uo (Section 2). Importantly, we note that it is possible to train an expressive one-step model that generates high-quality samples with distillation losses (Liu al., 2023; 2024; Li et al., 2024a; Ding et al., 2024b; Frans et al., 2025).
FQI. (ours)
W3
Hence, the BC term in the FQL actor loss (Equation (9) can be interpreted 2- Wasseerstei distance between the current policy Tu and the data-collecting policy approximated by ma. This Wasserstein regularizer is analogous to the KI. behav- ioral regularizer in TD3+BC (Fujimoto & Gu, 2021) and AWAC (Nair et al., 2020), and the x² behavioral regu- larizer in CQL (Kumar al., 2020; Garg al., 2023). However, unlike the KL and x² divergences, which are (in principle) invariant and agnostic to any metric struc- tures," " our 2-Wasserstein distance is aware of the metric structure over actions (which we impose as the Euclidean distance) (Table I). This metric-aware property poten- tially incorporates better inductive bias about the simi- larity between actions, akin how Wasserstein distances improve upon metric-agnostie divergences in other con- texts in machine learning (Arjovsky et al., 2017; Park et al., 2024b).
upper bound the squared
We are now ready to describe the complete objective of our method, flow Q-learning (FQL). FQL has three compo- nents: critic (o(s, a). BC flow policy ne(s, =), and one- step policy (s, 2). First, as discussed above, the BC flow policy is trained only with the BC flow-matching loss (Equa- tion (5)). The critic is trained with the original critic loss of cahavior-regularizeed actor-critic (Equation (I)), except that we use the one-step policy Tw in place of wo- Finally, the one-step policy is trained with the following actor loss:
(w) EANDDawN ( (=Q0(s, ) + DiLtist (9)
loss
"B0"
loas
Similar the naive flow actor loss above (Equation (6)), this objective maximizes both the Q and BC losses with a hyper- parameter However, it does not involve backpropagation over time as The is one-step policy. Note also that the distil- lation loss now serves as a behavioral regularizer based on the BC flow policy (Figure 2b). The output of this algorithm is the ste-step policy k.. which is what is deployed at test time. We provide a pseudocode for FQL in Algorithm 1, in
"While the original KL and x² divergences are entirely metric- agnostic, this property may be lost in practice with vari- ational approximation (e.g., with Gaussian parameterization)

---

Flow Q-Learning
which M denotes the number of steps for the Euler method, and describe the full implementation details in Appendix B.
planning and hierarchical learning (Janner et al., 2022; Ajay et al., 2023; Zheng et al., 2023; Liang al., 2023; Li et al., 2023; Suh et al., 2023; Venkatraman et al., 2024; Chen et al., 2024a), world modeling and data augmentation (Lu et al.,
Why is it a good idea? FQL has three benefits. First, it leverages reparameterized policy gradient (i.e., directly max- imizing the Q function with gradients through a²) a²), which is known to be one of the most effective policy extraction methods (Park et al., 2024a), while entirely avoiding unsta- ble and costly backpropagation through time. We will re- visit this point in more detail in Section 4.1, and empirically show its effectiveness through our experiments (Section 5). Second, FQL yields an efficient one-step policy as the out- put, which eliminates iterative flow generation processes at inference time, while maintaining most of the expressivity of the full flow model (Liu et al., 2023; Frans et al., 2025). Third, FQL is temmpleemlle and easy-to-tune: thanks to the simplicity of flow-made Now-matching, it can be implemented in few lines on top of the standard behavior-regularizee actor-critic framework, and has only one major hyperparam- eter o, without requiring tuning a noise schedule.
2023a; Ding et al., 2024c; Jackson et al., 2024; Alonso et al,,
2024), exploration (Mazoure et al., 2019; Ren et al., 2025), and policy modeling (Section 4. 1). Our method belongs to the third category, where we model a policy with an expres- sive flow network to capture the arbitrarily complex distri- bution of the behavioral policy.
4.1. How Have Previous Works Trained Diffusion and Flow Policies with RL?
Various approaches have been proposed for training diffu- sion or flow policies with RL. In this section, we provide an in-depth review of these methods, discuss their advantages and limitations, and explain how FQL relates to prior work. Prior methods can be categorized into several groups based on their policy extraction strategies (Park et al., 2024a).
(1) Weighted behavioral cloning. One straightforward ap- proach to modulating diffusion or flow policy is to as- sign weights to transition samples based on the correspond- ing learned values. The most basic form uses advantage- weighted regression (AWR) (Peters & Schaal, 2007; Peng et al., 2019; Nair et al., 2020) with the following objective:
4. Prior Work
Offiine RL and offline-to-onliee RL. The goal of offline RL is to train a policy using only previously collected data. Hundreds of offline RL methods and techniques have been proposed so far, and many of them are based on single central idea: maximizing the return while minimizing a discrepancy measure between the state-action distribution of the dataset and that of the learned policy (Levine et al., 2020; Sikchi al., 2024). Previous works have implemented this high-level objective in diverse ways through behavioral regularization (Nair et al., 2020; Fujimoto & Gu, 2021; Tarasov et al., 2023a), conservatism (Kumar et al., 2020), in- sample maximization (Kostrikov et al., 2022; Xu et al., 2023; Garg et al., 2023), out-of-distributtin detection (Yu et al., 2020; Kidambi et al., 2020; An et al., 2021; Nikulin et al., 2023), dual RL (Lee et al., 2021a; Sikchi al., 2024), and generative modeling (Chen et al., 2021; Janner et al., 2021; 2022). After finishing offline RL training, we can further fine-tune the policy with additional online rollouts. This setting is often referred to as offline-to-onliee RL, for which several techniques have been proposed (Lee et al., 202 lb; Song et al., 2023; Nakamoto et al., 2023; Ball et al., 2023; Yu & Zhang, 2023). Our method, FQL, is mainly designed for offline RL, but we show that it can also be directly fine- tuned with online rollouts without any algorithmic changes.
max E. e [ LFlow (0) 0
(10)
where a is an inverse temperature hyperparameter, and Q(s,a) : S x A- R and V(s): + IR are state-action and state value functions, respectively (Sutton & Barto, 2005). For diffusson policies, (0) is replaced with a diffusion loss. Intuitively, this objective makes the policy selectively clone transitions with high advantages. Among previous works, QGPO (Lu et al., 2023b), EDP (Kang et al., 2023), QVPO (Ding et al., 2024a), and QIPO (Zhang et al., 2025) are mainly based on weighted behavioral cloning.
Weighted behavioral cloning is simple and easy to imple- ment. However, it is known to be one of the least effec- tive policy extraction methods (Fu et al., 2022; Park et al., 2024a), due to the small number of effective samples and limited expressivity. In our experiments, we empirically show that weighted behavioral cloning generally leads to subpar performance, especially on complex tasks.
(2) Reparameterized policy gradient. Another popular approach to guide an iterative generative model is to di- rectly maximize the value function Q(s, a) with reparame- terized gradients, while regularizing it with a flow or diffu- sion loss, as in Equation (6). Among previous approaches, Diffusion-QU QL (Wang et al., 2023), DiffCPS (He el., 2023), Consistency-AC (Ding & Jin, 2024), SRDP (Ada al.,
RL with diffusion and flow models. Motivated by the re- cent successes of iterative generative modeling techniques, such as denoising diffusion Soohl-Dickstein et al., 2015; Ho et al., 2020; Dhariwal & Nichol, 2021) and flow match- ing (Lipman et al., 2023; Esser al., 2024), researchers have developed diverse ways to integrate them into RL. Pre- vious works have applied iterative generative models to
"See Park et al. (2024a) for further discussions.
5

---

Flow Q-Learning
2024), and EQL (Zhang al., 2024) implement this scheme with backpropagation through time.
Reparameterized policy gradient is known to be one of the most effective policy extraction methods for Gaussian policies (Park et al., 2024a). However, when naively applied to iterative generative models, it requires backpropagation through time (Equation (9)), which often incurs stability issues and leads to suboptimal performance (Section 5).
Figure 4. OGBench tasks.
(3) Rejection sampling. The third category is rejection sam- pling. Instead of adjusting the parameter of the generative model, we can sample N actions from fixed BC policy, and select the action that has the highest value. In other words, we treat the following formula as a policy:
effective actor-critic framework, with no backpropagation through time. In our experiments, we empirically show that
our approach leads to significantly better performance than previous degiillation-based methods (Consistency-Ae and SRPO) as well as other policy extraction schemes.
arg max ayou
Q(s,ar)
(11)
where n is a BC policy trained by a flow or diffusion objective. Among previous works, SfBC (Chen et al., 2023), IDQL (Hansen-Estruch et al., 2023), and AlignIQL (He et al., 2024) are based on (variants of) rejection sampling.
5. Experiments
In this section, we empirically evaluate the performance of FQL., comparing it to previous offline RL and offline- to-online RL approaches on a variety of challenging tasks, We also provide extensive analyses and ablations on policy extraction strategies and FQL's design choices.
Rejection sampling is simple and stable. However, it re- quires querying the policy and value function N times at ev- ery environment step during inference (and possibly during training as well, depending on the method). This can be pro- hibitive with larger models or a larger number of samples.
5.1. Experimental Setup
Benchmarks. We use the recently proposed OGBench task suite (Park et al., 2025) as the main benchmark (Fig- ure 4). OGBench provides number of diverse, challeng- ing tasks across robotic locomotion and manipulation, with both state and pixel observations, where these tasks are gen- erally more challenging than standard D4RL tasks (Fu et al., 2020), which have been saturated as of 2025 (Tarasov et al., 2023a; Rafailov et al., 2024; Park et al., 2024a). While OG- Bench was originally designed for benchmarking offline goal coalioondt RL, we use its reward- based single-task variants ("=singletask)) to make it compatible with stan- dard maward-maximizing offline RL algorithms. We employ 5 locomotion and 5 manipulation environments where each environment provides separate tasks, bringing the total to 50 state-based OGBench tasks. In addition, consider 5 diverse OGBench visual manipulation tasks to challenge the agent's 's ability to handle 64 x 64 x 3-sized image ob- servations. Finally, we also employ relatively challenging 6 antmaze and 12 adroit tasks from the D4RL benchmark.
(4) Others. Besides these three major categories, other techniques have also been proposed to guide diffusion policy to maximize the learned value function, based on some combination of the above strategies (Mao et al., 2024), action gradients (Yang et al., 2023; Psenka et al., 2024; Li al., 2024b; Mark et al., 2024; Fang et al., 2025), bi- level MDPs (Ren et al., 2025), value alignment (Chen et al.,
2024c), and implicit Q-learing (Chen et al., 2024b;d).
Contextualizing FQL in prior work. Our approach, FQL, falls into the second category, reparameterized policy gra- dient, which is known to be one of the most effective pol- icy extraction schemes (Park et al., 2024a). However, un- like the previous methods discussed above in the same cate- gory, which use backpropagation through time, we entirely bypass recursive backpropagation by only steering the one- step policy to maximize values (Equation (9)), while train- ing the flow policy solely with the BC loss. Among previous works, Consistency-AN (Ding & Jin, 2024), SRPO (Chen et al., 2024b), and DTQL (Chen et al., 2024d) also employ distillation, and in particular, Consistency-AA (Ding & Jin, 2024) shares conceptually similar ligh-level objective to our method (but with consistency models instead of direct one-step distillation). However, they either still use back- propagation through time (Ding & Jin, 2024) or are based on implicit Q-learning (Kostrikov et al., 2022), which is known to be less effective than actor-critic learning (Tarasov et al., 2023a). In contrast, we train one-step policy within a more
Methods. For our offline RL experiments, we use the fol- lowing 9 recent methods as representative examples of va- riety of algorithm types and policy extraction strategies.
(1) Gaussian policies. For standard offline RL methods that use Gaussian policies, we consider BC, IQL (Kostrikov et al., 2022), and ReBRAC (Tarasov et al., 2023a). In par- ticular, ReBRAC is known to achieve state-of-the-an perfor- mance on many D4RL, tasks (Tarasov et al., 2023b), and
6

---

Flow Q-Learning
Table 2. Offiine RL results, FQL achieves the best near-best performance on most of the 73 diverse, challenging benchmark tasks. The performances are averaged over 8 seeds (4 seeds for pixel based tasks), but the cells without the "=" sign indicate that the numbers are taken from prior works (Tarasov et al., 2023b; Mansen-Essruuce et al., 2023; Chen et al., 2024b). See Table 3 for the full results.
Caussian Policies
Diffusion Policies
Flow Policies
Task Category
BC
IQL.
ReBRAC
TDQL.
SRPO
CAC
FANAC
FBRAC
IFQL.
FQL.
OGBench wataaze-large-ingg (5 tasks)
11 11
53
81 15
21
11 1:4
33
6
00 16
28
79
OGBench artaaze-glaatt-snnllttaaa (5 tasks)
0 +0
4 +1
26 +s
0 =0
0 =0
0 =0
0 +0
4
3
+5
OGBench
(6 taska)
2 =1
33
22 2
1
1 =1
53
19
38 +s
60 +:4
58
OGBench humaaooddmazz-llarrr---iggggnni
(5 tasks)
1 10
2 21
1
0 =0
0 20
0 10
2
11
4 27
DGBench antsoccer- aiaeocceersrrem-ssiigleaaaa (5 tasks)
10
8 12
0 10
12 1:4
1 :0
2 1:4
12 pM
16 11
33 16
60 12
OGBench
OGBench cube-atnggle-atngletaae (5 taske)
=
83 =3
91 +2
95 #2
80 #5
85 =9
81 =4
79 +7
79 +2
96 +:1
(5 tasks)
2
7 +1
12 +1
15 +5
2 =1
6 +2
5 +-2
15 +3
14 +3
29 +2
OGBanch acane-singletank (5 tasks)
5 5
28 =1
41 3:3
46 1B
20 11
40 m7
30 23
45 35
30 #3
56 12
OGBench -a-e----etae (5 tasks)
2 10
9 :3
21 11
10 12
18 11
19 00
6 12
14 1:4
19 21
30 11
OCBench mazzle-4xdd-ingleeaad (5 tasks)
0 +0
7 +11
14 +1
20 #3
10 #5
=3
1 +0
13 +1
#5
17 #2
DARL antmaze (6 tasks)
17
57
78
79
74
30 =3
44 =3
61 +7
65 #7
84 +5
DARL. adreit (12 tasks)
59
11
51 11
43
29
48 21
50 12
52 21
52 21
Visual nanipulation (5 tazks)
42
60
12
22 12
23
65 12
Due to the high computational cost of pixel-bused tasks, we selectively benchmark methods that achieve strong performance on state-bused OGBench tasks.
the closest Gaussian baseline to FQL in that both are based on behavior- regulaaieee actor-critic (Section 2).
tionally, we consider two performant methods specifically designed for data-driven online RL, Cal-QL (Nakamoto et al., 2023) and RLPD (Ball et al., 2023).
(2) Diffusion policies. For diffusion policy-based offline RL methods, we consider IDQL (Hansen-Estruch et al., 2023), SRPO (Chen et al., 2024b), and Consistency-AC (CAC) (Ding & Jin, 2024). IDQL is based on rejection sampling, and SRPO and CAC are based on policy distilla- tion, as in FQL. In particular, CAC is the closest diffusion baseline to FQL, in that they both train distillation policies within the Schavior-regulariie though CAC still employs backpropagation through time (but with fewer steps) and is based on consistency models rather than direct one-step distillation.
Evaluation. For offline RL, we evaluate the performance of methods after fixed number of gradient steps; in par- ticular, we do not report the best performance across differ- ent evaluation epochs as it may bias results (Tarasov et al., 2023b). To ensure fair comparisons, we individually tune hyperparameters of the baselines with similar amounts of training budget (Appendix E.2), and use the same network size and discount factor, unless otherwise stated. We use 8 seeds for state-based tasks and 4 seeds for pixel-based tasks, and present standard deviations after "=" in tables and 95% bootstrap confidence intervals as shaded areas in plots, un- less otherwise mentioned. In tables, we denote values at or above 95% of the best performance in bold, following OG- Bench (Park et al., 2025). We refer to Appendix E for the full training and evaluation details.
actor-critic framework, al-
(3) Flow policies. Since there are currently only few prior methods that explicitly employ flow policies (Zhang et al., 2025), we consider flow variants of previous meth- ods to cover the three main policy extraction schemes dis- actor-critic (FAWAC) is flow variant of AWAC (Nair et al., 2020), which uses AWR (Equation (10)) as the policy learning ob- jective, conceptually similar to QIPO (Zhang et al., 2025).
cussed in Section 4.1. Flow advantage-weigee
5.2. Results and Q&As
We present our results via the following Q&As.
Flow behavior-regularized actor-critic (FBRAC) is the flow counterpart of Diffusion-QU -L (DQL) (Wang et al., 2023) based on the naive Q loss with backpropagation through time (Equation (6)). Implicit flow Q-learning (IFQL) is the flow counterpart of IDQL based rejection sampling (Equation (1 ())).. Notably, FAWAC and FBRAC are differ- ent from our method (FQL) only by their policy extraction strategies while sharing the exact same architectures and implementations, and thus can provide controlled ablation results on our distillation-based policy extraction scheme.
Q: How good is FQL for offline RL?
A: FQL achieves the best or near-best performance on most tasks, especially in complex manipulation environments.
Table summarizes the aggregated henchmarking result on a total of 73 state- or pixel-based offline RL tasks across robotic locomotion and manipulation. We find that FQL gen- erally achieves better performance than previous methods, including ones based Gaussian and diffusion policies. In particular, FQL leads to consistently better performance than its closest diffusion baseline (CAC), and often signifi- cantly outperforms its closest Gaussian baseline (ReBRAC) especially on manipulation tasks, which feature highly mul-
For offline-toonline RL experiments, we consider three prior offline RL methods (IQL, ReBRAC, and IFQL) that support fine-tuning and achieve strong performance. Addi-
7

---

Flow Q-Learning
timodal distributions. We also highlight that FQL achieves the best performance of 84% one of the hardest tasks in the D4RL benchmark, antmaze-large-pae
(Table 4) as well as the 10 D4RL antmaze and adroit tasks used by Tarasov et al. (2023b). Figure 6 shows the training curves of FQL and previous approaches on these 15 tasks, where online fine-tuning starts at 1M gradient steps (see Figure 12 and Table for the full results). The results show that FQL achieves the best fine-tuning performance compared to both previous offline RL approaches (includ- ing IFQL, the strongest flow-based baseline) and methods specifically designed for online fine-tuning (Cal-QL. and RLPD).
(Table 3).
Q: Can't I, just use existing policy extraction schemes?
16 TAVAC seighted regreoxion
FERAC repaтwns wish BPTT
IPQL rejjactioo xxmpl king
FQL aeoep gu gurddance
Q: What are the important hyperparameters of FQL?
Figure 5. Policy extraction is important. The bars ahove compare the performances of different policy extraction methods averaged over the 50 state- sate-based OGBench tasks in Table 2.
uateaze-lazge-iivee
cube-double
acone
Perforsance
100-
100-
302 1390
100
200
330 1300
A: You can, but previous policy extraction schemes gener- ally lead to (often much) worse performance.
D-
500K
Bsegs
Stepo
Steps
This can be seen by comparing the performances of FQL and (FAWAC, FBRAC, IFQL), which are the closest flow- based baselines to FQL., but with different policy extraction mechanisms. In particular, FBRAC is exactly the same as FQL except that it uses backpropagation through time. We emphasize again that these baselines are implemented on the same codebase, use the same architecture, and are in- dividually tuned for each environment (Table 6). Figure 5 compares their offline RL performances aggregated over the 50 state-based OGBench tasks in Table 2. The results show that policy extraction alone can significantly affect perfor- mance, consistent with findings in Gaussian policies (Park et al., 2024a). The results also indicate that our one-step guidance is the most effective, significantly outperforming the other previous extraction strategies (Section 4.
Figure 7. The BC coefficient a needs to be tuned. The plots show how different values of affect offline RL performance.
A: The most important hyperparameter is the BC coefficient.
Figure 7 shows the ablation results of the BC coefficient a on three tasks. This hyperparameter needs to be tuned for each environment based on the suboptimality of the dataset, as is typical for most offline RL methods (Tarasov et al., 2023b; Park et al., 2024a). Other than a, the default hyperparameters of FQL work well, although tuning some additional hyperparameters (e.g., target value aggregation described in Appendix B) can slightly boost performance on some tasks. We provide an extensive ablation study on a total of 4 factors of FQL in Appendix C.
Q: Can FQL be fine-tuned with online rollouts?
Q: Do need to tune flow-related hyperparameters?
1P21.
-c-
RL
-
Cal-gi
-
RATRAC
+ Tgt.
Time distribation:
unifore. (defoult) --- bosa -Q-
Jegtt normal
DGRench
(5 tasks)
D4RI, antraze (6 tasks)
DARL, adroit (4 taske)
auteaze-1arge--iveze
cube-double
100
103
100
Perforzanc
Porforaace
S0-
50
30-
50-
30
500K
13
IM
Stapa
Staga
Stege
Steps
Steps
Figure 6. Offline-to-online RL results (8 seeds). tie-ttuning starts at 1M. The D4RL results of Cal-QL, ReBRAC, and IQL are taken from Tarasov et al. (2023b). See Figure 12 for the full plots.
Figure 8. You can just use the uniform time distribution. FQL's performance is generally robust to flow-related hyperparameters.
A: No, in general.
A: Yes, FQL can be directly fine-tuned without any modifica- tions, and often significantly outperforms previous methods.
For example, Figure 8 shows how the time sampling dis- tribution for flow matching affects performance, where we consider the uniform distribution, Unif ([0, 1]) (default), the beta distribution used by Black et al. (2024), and the logit normal distribution used by Esser et al. (2024). The re- sults suggest that time distributions matter only marginally, and the simplest uniform distribution often sufficient to
Specifically, we can fine-tune FQL simply by adding new on- line transitions to the dataset D, while continuing to train all networks using the same objective as in offline training. To show how effective FQL is for fine-tuning, we evaluate it on representative OGBench tasks across different categories
8

---

Flow Q-Learning
achieve the best performance. Similarly, we find that the per- formance is generally robust to the number of flow steps (the default is 10), as long it is not too small (see Appendix C).
ing. Given that offline RL is notoriously sensitive imple- mentation details in general (Tarasov et al., 2023b), we be- lieve proposing simple yet performant method is a particu- larly important contribution to the community. We hope that FQI., with our clean, open-source implementation, spurs fu- ture research in scalable offline RL algorithms.
Q: How fast is FQL?
TZzz Douooian policy
policy
Training (Stataa)
Inferance (Statas)
5100 (ne)
Acknowledgments
We thank Chongyi Zheng for noticing an issue in our ini- tial implementation. This work was partly supported by the Korea Foundation for Advanced Studies (KFAS), AFOSR 10550-22-1-02733 and ONR 100014-20-1-2388 This re- search used the Savio computational cluster resource pro- vided by the Berkeley Research Computing program at UC Berkeley, Some figures in this work use Twemoji, an open- source emoji set created by Twitter and licensed under CC BY 4.0.
Patizd Faiae" FRRAC IRE
FR.
Rabsec
Flueer
resAc
Figure 9. Run time comparison on cube-double.
A: FQL is one of the fastest flow-based offline RL methods. Figure shows that, in terms of both training and inference costs, FQL only slightly slower than Gaussian policy- based offline RL methods, while being faster than most flow- based baselines. See Figure II for the detailed comparison results.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Q: Are flow policies better than diffusion policies?
A: Maybe, but we do not make such a claim in this paper.
The main contribution of this paper is our policy extraction scheme (one-step guidance), not just the use of flow match- ing itself. Although we show that one-step guidance com- bined with flow matching (i.e., FQL) achieves better perfor- mance than previous policy extraction schemes for diffusion and flow policies (Table 2), we believe it is possible to ap- ply our one-step guidance to diffusion policies with appro- priate modifications to convert SDEs to ODEs (Song et al., 2021) to achieve similar performance, given the equivalence between the two frameworks (Gao et al., 2024). Neverthe- less, flow matching has one arguably clear advantage over denoising diffusion: it is much simpler to implement!
References
Ada, S. E., Oztop, E., and Ugur, E. Diffusion policies for outtof-distributiin generalization in offline reinforcement learning. IEEE Robotics and Automation Letters (RA-L), 9:3116-3123, 2024.
Ajay. A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. Is conditional generative modeling all you need for decision-making? In International Conference on Learning Representations (ICLR), 2023.
Albergo, M. S. and Vanden-Eijnden E. Building normaliz- ing flows with stochastic interpolants. In International Conference on Learning Representations (ICLR), 2023.
6. Closing Remarks
We presented flow Q-learning (FQL), a simple and perfor- mant offline RL method that leverages expressive flow policy and reparameterized policy gradient, without suffer- ing from backpropagation through time. We showed that FQL generally leads to the best performance on challeng- ing tasks across robotic locomotion and manipulation, of- fline RL and offline-toonllne RL, as well as state- and pixel-based settings. FQL. however, is not perfect; see Ap- pendix A for the limitations of FQL..
Alonso, E., Jelley, A., Micheli, V., Kanervisto. A., Storkey, A., Pearce, T., and Flcuret, F. Diffusion for world model- ing: Visual details matter in atari. In Neural Information Prucessing Systems (NeurIPS), 2024.
An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- based offline reinforcement learning with diversified q- ensemble. In Neural Information Processing Systems (NeurIPS), 2021.
As closing remark, we would like to reiterate one particu- larly appealing property of FQL- - simplicity: one small algorithm box (Algorithm 1) essentially captures the entire training objectives of FQL (modulo minor details), includ- ing all of flow matching, iterative sampling, and value learn-
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gen- erative adversarial networks. In International Conference on Machine Learning (ICML.), 2017

---

Flow Q-Learning
Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv, abs/1 607. 06450, 2016.
Ding, S., Hu, K., Zhang, Z., Ren, K., Zhang, W., Yu, J., Wang, J., and Shi, Y. Diffusion-based reinforcement learning via q-weighted variational policy optimization. In Neural Information Processing Systems (NeurlPS), 2024a
Ball, P. Smith, L.., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. In Inter- national Conference on Machine Learning (ICML), 2023.
Ding, Z. and Jin, C. Consisteney models as rich and effi- cient policy class for reinforcement learning. In Interna- tional Conference on Learning Representations (ICLR), 2024.
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. To: A vision-language-actiion flow model for general robot control. ArXiv, abs/2410.24164, 2024.
Ding, Z., Jin, C., Liu, D., Zheng, H., Singh, K. K., Zhang, Q.. Kang, Y., Lin, Z., and Liu, Y. Dollar: Few-step video generation via distillation and latent reward optimization. ArXiv, abs/2412.15689, 2024b.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Miiler S., and Zhang, Q. JAX: composable transformations of Python+NumPP programs, 2018. URL http://gi thub. com/ joom/jasz maaaaammmjaas
Ding, Z., Zhang, A., Tian, Y., and Zheng, Q. Diffusion world model. ArXiv, abs/2402.035777 2024c.
Chen, C., Deng, F., Kawaguchi, K., Gulcehre, C., and Ahn, S. Simple hierarchical planning with diffusion. In Interna- tional Conference Learning Representations (ICLR), 2024a.
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, Impala: Scalable dis-
I.. Legg. S., and Kavukcuoglu, tributed deep-rl with importance weighted actor-learner architectures. In International Conference Machine Learning (ICML), 2018.
Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline rein- forcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Rep- resentations (ICLR), 2023.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024.
Chen, H., Lu, C., Wang, Z., Su, H., and Zhu, J. Score reg- ularized policy optimization through diffusion behavior. In International Conference on Learning Representations (ICLR), 2024b.
Chen, H., Zheng, K., Su, H., and Zhu, Aligning dif- fusion behaviors with q-functions for efficient continu- ous control. In Neural Information Processing Systems (NeurIPS), 2024c.
Fang. L.,, Liu, R., Zhang, J., Wang. W., and Jing, B. Dif- fusion actor-critic: Formulating constrained policy iter- ation as diffusion noise regression for offline reinforce- ment learning. In International Conference on Learning Representations (ICLR), 2025.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems (NeurIPS), 2021.
Frans, K., Hafner, D., Levine, S., and Abbeel, P. One step diffusion via shortcut models. In International Confer- ence on Learning Representations (ICLR), 2025.
Fu, J.. Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, ArXiv, 0ets/2004.072100 2020.
Chen, T., Wang, Z., and Zhou, M. Diffusion policies cre- ating a trust region for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2024d.
Fu, Y., Wu, D., and Boulet, B. A closer look at offline rl agents. In Neural Information Processing Systems (NeurIPS), 2022.
Collaboration, O. X.-E,, O'Neill, A., Rehman, A., Mad- dukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. Open x- embodiment: Robotic learning datasets and rt-x models. In IEEE International Conference on Robotics and Au- tomation (ICRA), 2024.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Neural Information Processing Systems (NeurIPS), 2021.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018
Dhariwal, P. and Nichol, A. Diffusson models beat gans on image synthesis. In Neural Information Processing Systems (NeurIPS), 2021.
10

---

Flow Q-Learning
Gao, R., Hoogeboom, E., Heek, J., Bortoli, V. D., Mur- phy, K. P., and Salimans, T. Diffusion meets flow match- ing: Two sides of the same coin, 2024. URL https: //diffusionflor www.ww..ggth.. github.io. io/.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
Lange, S., Gabel, T., and Riedmiller, M. Batch reinforce- ment learning. Reinforcement learning: State-of-the art, pp. 45-73. Springer, 2012.
Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q- learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.
Lee, H.,, Hwang. D., Kim, D., Kim, H., Tai, J. Subrama- nian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up parameters deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2025.
Hansen-Estruch P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor-critic method with diffusion policies. ArXiv, abs/2304.10573, 2023.
He, L., Shen, L.., Zhang, L., Tan, J., and Wang, X. Dif- feps: Diffusion model based constrained policy search for offline reinforcement learning. ArXiv, abs/2310.05333 2023.
Lee, J.. Jeon, W., Lee, B.-J., Pineau, J., and Kim, K.-E. Op- tidice: Offline policy optimization via stationary distribu- tion correction estimation. In International Conference on Machine Learning (ICML.), 2021a.
He, L.. Shen, L., Tan, J., and Wang, X. Aligniql: Policy alignment in implicit q-learning through constrained opti- mization. ArXiv, 0oss2405.18188/ 2024.
Lee, J. M. Introduction to Smooth Manifolds. Springer, 2012.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference on Robot Learn- ing (CoRI.), 2021
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). ArXiv, abs/1606.08415, 2016.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Neural Information Processing Sys- tems (NeurIPS), 2020.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643 2020.
Jackson, M. T., Matthews, M. T., Lu, C., Ellis, B., White- son, S., and Foerster, J. Policy-guided diffusion. Rein- forcement Learning Conference (RLC), 2024.
Li, J., Feng, W., Chen, W., and Wang. W. Y. Reward guided latent consistency distillation. Transactions on Machine Learning Research (TMLR), 2024a.
Janner, M., Li, Q., and Levine, S. Reinforcement learning as one big sequence modeling problem. In Neural Infor- mation Processing Systems (NeurIPS), 2021.
Li W., Wang. X., Jin, B., and Zha, H. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning (ICML), 2023.
Janner, M., Du, Y., Tenenbaum, B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. In International Conference Machine Learning (ICML), 2022.
Li, Z., Krohn, R., Chen, T., Ajay, A., Agrawal, P., and Chal- vatzaki, G. Learning multimodal behaviors from scratch with diffusion policy gradient. In Neural Information Pro- cessing Systems (NeurIPS), 2024b.
Kang, B., Ma, X., Du, C., Pang, T., and Yan, S. Efficient diffusion policies for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
Liang. Z., Mu, Y., Ding, M., Ni, F., Tomizuka, M., and Luo, Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning (ICML), 2023.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: basel offline reinforcement learning. In
Neural Information Processing Systems (NeurIPS), 2020.
Lipman, Y., Chen, R. T., Ben- Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In Interna- tional Conference on Learning Representations (ICLR), 2023.
Kingma, D. P. and Ba. J. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
Lipman, Y., Havasi, M., Holderricth, P., Shaul, N., Le, M., Karrer, B., Chen, R. T. Q., Lopez-Paz, D., Ben-Hamu, H., and Gat, I. Flow matching guide and code. ArXiv, abs/2412.06264, 2024.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit g-learning In International Con- ference on Learning Representations (ICLR), 2022.

---

Flow Q-Learning
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023.
Nikulin, A., Kurenkov, V., Tarasov, D., and Kolesnikov, S. Anti-exploration by random network distillation. In International Conference on Machine Learning (ICML), 2023.
Liu, X., Zhang, X., Ma. J., Peng, J., et al. Instaflow: One step is enough for high quality diffusion- based text-to- image generation. In International Conference on Learn- ing Representations (ICLR), 2024.
Park, S., Frans, K., Levine, S., and Kumar, A. Is value learning really the main bottleneck in offline ri? In Neural Information Processing Systems (NeurIPS), 2024a.
Park, S., Rybkin, O., and Levine, S. Metra: Scalable un- supervised rl with metric-aware abstraction. In Interna- tional Conference on Learning Representations (ICLR), 2024b.
Lu, C., Ball, P., Teh, Y. W., and Parker-Holder, J. Synthetic experience replay. In Neural Information Processing Systems (NeurIPS), 2023a.
Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, Contrastive energy prediction for exact energy-guide -eeigy- diffusion sampling in offline reinforcement learning. In International Conference Machine Learning (ICML), 2023b.
Park, S., Frans, K., Bysenbach, B., and Levine, S. Ogbench: Benchmarking offline goal-conditioned rl. In Interna- tional Conference on Learning Representations (ICLR), 2025.
Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage weiihted regression: Simple and scalable off- policy reinforcement learning. ArXiv, abs/1910.00177,, 2019.
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., and Mart' Bart'in-Matt'in in, R. What matters in learning from of- fline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
Peters, J. and Schaal, S. Reinforcement learning by reward- weighted regression for operational space control. In International Conference on Machine Learning (ICML), 2007.
Mao, L., Xu, H., Zhan, X., Zhang, W., and Zhang. A. Diffusion-dices In-sample diffusion guidance for offline reinforcement learning. In Neural Information Process- ing Systems (NeurIPS), 2024.
Psenka, M., Escontrela, A., Abbeel, P., and Ma, Y. Learning a diffusion model policy from rewards via q-score match- ing. In International Conference on Machine Learning (ICML), 2024.
Mark, M. S., Gao, T., Sampaio, G. G., Srirama, M. K., Sharma, A., Finn, C., and Kumar, A. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. ArXiv, abs/2412.06685, 2024.
Rafailov, R., Hatch, K. B., Singh, A., Kumar, A., Smith, L., Kostrikov, I., Hansen-Estrucd P., Kolev, V., Ball, P. J., Wu, J., et al. D5rl: Diverse datasets for data-driven deep reinforcement learning. In Reinforcement Learning Conference (RLC), 2024.
Mazoure, B., Doan, T., Durand, A., Pineau, J., and Hjelm, R. D. Leveraging exploration in off-policy algorithms via normalizing flows. In Conference on Robot Learning (CoRL),201) 2019.
Ren, A. Z., Lidard, J., Ankile, L. L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., and Simchowitz, M. Diffusion policy policy optimization. In International Conference on Learning Representations (ICLR), 2025.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602. 2013.
Sikehi, S., Zheng, Q., Zhang, A., and Niekum, Dual rl: Unification and new methods for reinforcement and imitation learning. In International Conference on Learn- ing Representations (ICLR), 2024.
Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelerating online reinforcement learning with offline datasets. ArXiv, abs/2006.09359, 2020.
Nakamoto, M., Zhai, Y., Singh, A., Mark, M. S., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. In Neural Information Processing Systems (NeurIPS), 2023.
J., Weiss, E., Maheswaranathan. N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning (ICML), 2015.
Nauman, M., Ostaszewski. M., Jankowski, K., Miloš, P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Neural Information Processing Systems (NeurIPS). 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
12

---

Flow Q-Learning
Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishna- murthy, A., and Sun, W. Hybrid rl: Using both offline and online data can make rl efficient. In International Conference on Learning Representations (ICLR), 2023.
Zhang, R., Luo, Z., Sjölund, J., Schön, T. B., and Mattsson, P. Entropy--regularized diffusion policy with q-ensembles for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2024.
Suh, H. J. T., Chou, G., Dai, H., Yang. L., Gupta, A., and Tedrake, R. Fighting uncertainty with gradients: Offline reinforcement learning via diffusion score matching. In Conference on Robot Learning (CoRL), 2023.
Zhang, S., Zhang, W., and Gu, Q. Energy-weighhe flow matching for offline reinforcement learning. In Interna- tional Conference Learning Representations (ICLR), 2025.
Zheng, Q. Le, M., Shaul, N., Lipman, Y., Grover, A., and Chen, R. T. Guided flows for generative modeling and decision making. ArXiv, abs/2311.13444 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16: 285-286, 2005.
Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the minimalist approach to offline reinforce- ment learning. In Neural Information Processing Systems (NeurIPS), 2023a.
Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., and Kolesnikov, S. Corl: Research-oriented deep offline re- inforcement learning library. In Neural Information Pro- cessing Systems (NeurIPS), 2023b.
Venkatraman, S., Khaitan, S., Akella, R. T., Dolan, J., Schneider, J., and Berseth, G. Reasoning with latent dif- fusion in offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
Wang, Z., Hunt, J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.
Wu, Y. Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. ArXiv, abs/1911. 11361, 2019.
Xu, H., Jiang, L.,, Li, J., Yang, Z., Wang, Z.,, Chan, V., and Zhan, X. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference Learning Representations (ICLR), 2023.
Yang, L., Huang, 7.,, Lei, F., Zhong, Y., Yang. Y., Fang, C., Wen. S., Zhou, B., and Lin, Z. Policy representation via diffusion probability model for reinforcement learning. ArXiv, aabb/2305.13122. 2023.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems (NeurIPS), 2020.
Yu, Z. and Zhang, X. Actor-critic aligmment for offline-to- online reinforcement learning. In International Confer- ence on Machine Learning (ICMI., 2023.
13

---

Flow Q-Learning
A. Limitations
One potential limitation of FQL is that it requires numerically solving ODEs during training to minimize the distillation loss (Equation (7)). While this is not necessarily a significant speed bottleneck on both state- and pixel-based tasks in our experiments (as shown in Figure 11) since flow matching happens in the relatively oow-dimensional action space (as opposed to image generation), we believe this may further be improved by incorporating more advanced one-step distillation method, such as shortcut models (Frans et al., 2025). Another limitation is that it does not have "built-in" exploration mechanism for online fine-tuning, For examplc, FQL does not achieve the best online fine-tuning on the puzzle-4x task (Table 4), in which exploration can help avoid local optima. While we find that FQL without any additional exploration bonuses is enough to achieve strong performance on many challenging tasks (Figure 6), we believe it can be further improved by combining FQL. with a more principled exploration strategy or additional specialized fine-tuning techniques, leaving them for future work. Finally, while we have demonstrated the performance of FQL on various simulated robotics tasks, we have not evaluated FQL real-world tasks. We believe applying FQL's distillation basodd policy extraction scheme real-world robotic tasks, potentially with a pre-trained flow BC policy (Black al., 2024), is another exciting future research direction.
B. Implementation Details
In this section, we describe the full implementation details of FQL.
Flow matching. As mentioned in Section 2, we use the simplest flow- now-matching objective (Equation (5)) based on linear paths and uniform time sampling. We use a step count of 10 for the Euler method across all tasks, and for simplicity, we do not use sinusoidal embeddings for the time variable. See Figures 10c and 10d for ablation studies on these flow-related hyperparameters.
Value learning. Following standard practice in RL, we train two Q functions to improve stability. We take the mean of the two Q values for the Q loss term in the actor objective (Equation (9)). We also use the mean for the target value in the critic objective (Equation (1)) by default, but we use the minimum of the two Q values (which is often referred to as clipped double Q-learning (Fujimoto et al., 2018)) for the adroit and OGBench antmaze- (large, giant] tasks, as we find to be slightly better. See Figure 10b for an ablation study on this choice.
Online fine-tuning, For offline-to-online RL, we simply add online transitions to the dataset, without distinguishing them from the offline transitions (i.e., we do not use balanced sampling, unlike Lee et al. (2021b); Nakamoto et al. (2023); Ball et al. (2023)). We continue to train the components of FQL with the same objective as in offline training (Algorithm 1).
Network architectures. For FQL., we use [512, 112,512,55 512, sizeed multi-layer perceptions (MIPs) for all neural networks. We apply layer normalization (Ba et al., 2016) to value networks to further stabilize training. We find that using large enough network is especially important in navigation environments (e.g., antmaze).
Image processing. For pixel-based environments, we use a smaller variant of the IMPALA encoder (Espeholt et al., 2018) and apply random-shift augmentation with a probability of 0.5, following the official implementation of Park et al. (2025). In addition, we use frame stacking with three images, which we find to be important on some pixel-based tasks, such as cube and puzzle.
Training and evaluation. We train FQL with 1M gradient steps for state-based OGBench tasks and 500K steps for D4RL and pixel-based OGBench tasks, and evaluate the agent every 100K steps using 50 episodes. For OGBench, following the official evaluation scheme (Park et al., 2025), report the average success rates across the last three evaluation epochs (800K, 900K, and 1M for state-based tasks and 300K, 400K, and 500K for pixel-based tasks). For D4RL, following Tarasov et al. (2023b), we report the performance at the last epoch. For offline-to-online RL results (Table 4), we report the performances at 1 and 2M steps.
BC coefficient a. The most important hyperparameter of FQL the BC coefficient a in Equation (9). We perform a hyperparameter search over {1000, 3000, 10000, 30000) for adroit tasks and {3. 10, 100, 300, 1000) for the other tasks, and use the best one for each environment. We use larger values for adroit tasks simply because their return scale is significantly larger than that of the other tasks. We believe normalizing the Q loss as in Fujimoto & Gu (2021) would lead to more similar values across different tasks. While we do not apply this normalization technique in our experiments, we recommend enabling Q normalization for new tasks (which is available in our official implementation) and tuning a starting from (0.03, 0.1, 0.3, 1,3, 10). See Figure 10a for an ablation study on the BC coefficient.
14

---

Flow Q-Learning
Target
mcanígl.
(defeuls)
min(q),
antaaze Large diverze
double
5ce00
antaaze large diverss
hatner- clooed
cube double
Performance
100-
:00 260 1000
100
300 1066
Perforaace
120
100
50-
0
0
0
500
500%
0
Stape
Steps
Stepc
Scepe
Stope
Steps
(a) Ablation study on the BC coefficient ax.
(b) Ablation study on the target value aggregation method.
stops:
(defacit)
Time dictribution:
waiforn (defocit)
normal
untaaneel1 argend lewrse
cube-doubl
acene
Patraase-Targee varsee
cube-dauble
acans
100-
100
100
100
100
1000
Performanc
Performance
30
0
X00X
1M
500K
0
IM
Steps
Steps
Stepe
Stepe
Steps
Sceps
(c) Ablation study on the number of flow steps.
(d) Ablation study on the flow time distribution.
Figure 10. Ablation studies. We ablate several components of FQI. and study how they affect performance. The results are averaged over 8 seeds.
Hyperparameters. We refer to Tables 5 to for the complete list of hyperparameters.
C. Ablation Study
In this section, we ablate several components of FQL. and study how they affect performance. Figure 10 shows our ablation results, where we present training curves of FQL with different hyperparameters on representative selection of tasks.
BC coefficient As discussed in the main paper, the BC coefficient is the most important hyperparameter of FQL. Figure 10a demonstrates that a needs to be tuned for each task based on the suboptimality of the dataset, as is typical for most offline RL methods (Park et al., 2024a).
Target value aggregation methods. As discussed in Appendix B, we train two Q functions (Q1 and Q2) and use their mean, (Q1 + Q2)/2, for target values in the critic loss by default, but we use their minimum, min(Q1. Q2), for some tasks, such adroit. We present the ablation results in Figure 10b with the BC coefficient individually tuned for each ablation setting. The results show that not using clipped double Q-learning often leads to better performance, which is aligned with recent findings in online RL (Ball et al., 2023; Nauman al., 2024; Lee et al., 2025).
Flow steps. To numerically solve ODEs, we use the Euler method, which requires a pre-specified number of steps. In this work, we use 10 steps for all experiments. Figure 10c shows the ablation results, which suggest that the performance generally robust to the number of flow steps, as long as it is not too small.
Time distributions for flow matching, In this work, we use the uniform distribution, Unif 1]). sample time steps for flow matching. Prior works have considered other time distributions well. For example, Esser al. (2024) use the logit normal distribution to emphasize intermediate steps (i.e., first sample from the standard normal distribution, i ~ NN(0, I), and then map it via the sigmoid function, 1/(1 + e e))) and Black al. (2024) employ beta distribution, Beta(1, 1.5), to make the flow model focus more the initial steps. We evaluate these three strategies and report the results in Figure 10d. The results suggest that the performance is generally robust to the choice of the time distribution, and the simplest uniform distribution is often enough to achieve the best performance.
15

---

Flow Q-Learning
D. Additional Results
policy
policy
Training (States)
Inference (States)
Training (Pixels)
Inference (Pixels)
2-
Step time
Bc IuL
RaBit FAWAC FBRAC
IPQL
Put.
IQL.
RaBRAA FAWAC FBRAC
IPQL
Fui.
Bc IQL
BaERAo FAWAC FBRAC
IFQL
PaL.
10L.
ReBRAC FAWAC FERAC IPQL
Fill
Figure 11. Run time comparison. FQL is only slightly slower than Gaussian policy- most other flow-based methods terms both training and inference speeds. The run times are measured on same machine using single A5000 GPU, and averaged over 8 seeds.
policy-based based offline RL methods, while being faster than
Run time comparison. Figure compares the training and inference speeds of different methods on cube-double and visual-cubo double, where consider methods implemented in the same codebase FQL for fair comparison. The results show that FQL achieves the best near-best speed in terms of both training and inference among flow-based approaches. Notably, FQL is faster than FBRAC during training as it does not use potentially costly backpropagation through time, and is faster than IFQL during inference as it does not use rejection sampling.
Full results. We present the full per-task offline RL results in Table and the full offline-to-onling RL results in Table 4 and Figure 12. The results are averaged over seeds (4 seeds for pixel-bat based tasks), and we report standard deviations after "+" in tables and 95% bootstrap confidence intervals shaded areas plots. In tables, the best performance in bold, following OGBench (Park et al., 2025). Results without standard deviations confidence intervals indicate that they are taken from prior work; the D4RL results of BC, IQL, ReBRAC, and Cal-QL are taken from Tarasov et al. (2023b), and the an mazaee results of IDQL and SRPO are from Hansen -strrcch et al. (2023) and Chen et al. (2024b), respectively.
denote values at above 95% of
16

---

Table 3. Full offiine RL results. We present the full results on the 73 OGBench and D4RL tasks, (+) indicates the default task in each environment. The results are averaged over 8 seeds (1 seeds for pixel-based tasks) unless otherwise mentioned.
Canssian Folicios
Diffasion Polictas
Flow Policies
Task
IQL
ReBRAC
FAMAC
FBRAC
IeqL
(x)
0 2n
48
91 210
4.
0 0.
42
70 sm
24 217
80
6 sa
42 10
88 44
14 3a
44
1 #1
0 a4
35 412
8 s3
57 4.0
29
72 17
sa
=10
::4
83 115
52 117
8
51 1o
84 57
62 435
45 a10
17 14
10 g3
37 S14
18 m
80 1:
10 ss
=22
90 52
2 22
1 21
55 2s
9::5
76 25
38 a1s
83 14
antaaze
(+)
0 an
0
2:23
antzaze-giant-
(1 so
1
16 617
0 so
D so
0
D
4 o7
0 a0
9
0 au
0 au
34 2:2
0 0o
20
ax
=3
0 an
0
14
19 ar
49 a:2
0 a1
0 am
0
0 so
6 a30
13 =m
16 4:4
(+)
1 =0
32 Arr
16 2m
1 1:
38 2:
6:22
25 =s
69 =1
19 19.
41
1
47 ass
40
76 a10
85 all
94
6 =a
25 as
36 a1
0 a1
2 a1
83 2:14
19 =?
27 21
40 4:99
74 114
so
2:
68
24 630
3 aa
91 as
31 m7
63 a*
98 20
97
(x)
0.
3 a:
2
0 0:a0
D ao
7 7.a
so
1:
7:s8
8
3 a1
1
2
10 a3
48 230
11
1:10
1:20
1 11
0:00
D 20
0 01
#3
2 1:
0.1
1:11
2 az
0.50
2:
14 as
0 an
44 2m
2 21
22
17 as
61 and
77
2:22
17 17
0 21
15 =22
3 21
75 23
88 1:1
0.
6
0
0
11
16
14 pm
61
(+)
1:
3
0 am
0 01
D so
0 ao
12 61
24 84
16 ss
39 14
0 an
0 am
24
0.
36: 13
10 as
88
89
95
89 67
77 u.8
81 16
73 asa
79
97 63
(+)
3 n1
85 ss
92 64
96 12
82 #16
80 130
81 1:
88 113
73
97 1:2
2 aa
91 as
93 3a
99 81
96 ss
98 a1
87 1:
82 a12
91 13
73
92
93
70 336
91
79
79 120
79
3 28
78 19
87 SM
90 am
61 212
80 2:0
78 2:10
76 a:3
77
93 23
27 3:
45 an
30 =1
21 ==
21 =7
47 211
35 =
61 #)
uubb=roobbberrpp/y--jiihh::-2212 (+)
0 am
1 =1
7 a:
16 =30
D
2
2 a1
22 212
9
36
0 10
0 =m
4 11
17 +/
0 a1
3 11
8 s5
22 65
11
0 20
4
4 A9
1 +:
D an
3
2 2
2
17 =6
19 2:10
19 a6
94 AM
95 #e
100 s>
94 1:
100 #1
87 14
96 s4
98 La
100 20
scone
sccnerppayyssiinnlltaasttttaaaa (+)
11
12 13
#13
33 #14
18 8s
46 #10
76
1 21
32
55 #30
94 44
4 44
49 11
38 6s
78 414
54 419
98 61
scene
sconerpaayytiiggllssssttttaass
2 12
0 :1
3 13
13
20
=3
1 14
m3
0
0
0
5 12
33 16
97 :4
52 132
89 8:
97 12
25 is
63 11
94 t3
90 84
1 11
3 11
0.21
21
1:
2 22
=2
16 k33
1 11
3
0 0:00
D
1
10
(+)
1 21
2 21
2
0 am
0 am
0 am
1 11
2 22
0 00
16 23
10
5 Aa
1 #1
12
26
48
24 AR
44 1
1 21
32
49
34
0 am
7
12 A:
14 1:5
0 21
0 an
5 Am
16 23
0 40
aa
15 sa
34 as
330
112
20 310
50 :14
(+)
0 20
5 :9
10
26 20
7
5 g:
21 2:
11 25
0 ao
4 m1
7 aa
24 6:33
1 81
2 62
7 m.
untzaze-unszo-et
55
66 as
ss
mB
antzaee-unsee-d2eeraee
47
54
84
80
82
06
55
82
62 a12
89
sutzaze-zeiuu
0
66
90
81
19 :24
52 112
77 =7
36 115
78 A7
intraza-rediurrd1erree
74
44 215
17 as
00 295
71 213
satzaze-arrgeppay:
0
42
52
61
54
0 sd
10 =s
32 #21
55 =9
84 o7
nntraza-large-dvveraaez
30
64
16 2:10
20 217
64 c*
83 11
pan-buman-vi
71
78
103
76 100
69 #7
64 ss
67 =1
77 #7
71 212
53
pab-cloaed-vi
52
83
103
64 47
61 67
56 a
62 410
67 a9
80 alL
74 A11
pan-expert-vi
110
128
152
134
103
11.8
119
139 as
dpor-bunan-vi
2
3
-0
6 82
3
5
2 a4
7 7
dbor-clonad-ri
0
3
0.
D a0
1 10
2 22
2
épor-expert-r1
105
107
105
105
8/3
m1
12
hamer-bunan-vl
3
2
D
2 21
1 a1
2 40
2 8.
2 gL
3
1
cloed--
1
2
2a1
2 =1
1 #1
1 10
2 e1
2 e3
11 1:
127
129
125
127
211
125
relocate-buran rel Miooaaeeclloed-v relocateresseet-vi
0
0
0
0
0 s0
0
0 =0
0 ab
0 0.:0
0 0.20
0 an
-0 29
-0 cm
-0 a0
-0 20
108
106
108
107
106
98 2s
105
105
104
107
v risual-cube-sie ieoua--obb--iinne--pllyaae
70 213
83 26
66 25
49 =7
81 232
401
1:
24
21
visual-scene-pa winggeigaa------inaaan---eeee--e
97
98
16
86 410
98
7 2:5
88 6:1
7 29
100 20
04 #1
visual- visual-puzele- wal--dddddllll--lllateeeeee
ao
26
s3
Due the high computational cost pixel-based tasks, selectively benchmark methods achieve strong performance state-based OGBench tasks.

---

Flow Q-Learning
FQL
IPQL.
-
RLPD
Cal-Qr.
ReBRAC
IQL.
humanoi dww.ee-meeeeeeeeeeee medium
antsoccer-an -rena
cube-dauble
scene
puzzle-se 1x-4x4
100
100
100
1001
1001
1M
2M
1M
2M
1M
2M
1M
2M
1M
2
antmaze aatmazz-umaze umaze
uwwazeddiveree
antmaze--eede www--mdddddddaaaeeeeee
an tatae antmaze-mediune mediun mediun-diverse
antmaze-larga larrg--pay
100
100
100
1001
1000
Performance
0-:
0
IM
2M
IM
2M
IM
2M
IM
2M
IM
2M
matmaze-large--ee divere
pen-cloned
door-cloned
hamner-cl
relocate-cloned
100
Performance
100
100
50-
50
0-
0
1M Stepe
2M
1M Stepe
2M
1M
2M
IM
2M
1M
2M
Stepe
Stepe
Stepe
Figure 12. Offline-toonline RI results. Online fine-tuning starts 1M steps. The results are averaged over seeds unless otherwise mentioned.
Table 4. Offline-to-online RL results. The results are averaged over seeds unless otherwise mentioned.
Task
ReBRAC
Cal-QL
RLPD
Fq].
hunano
211 16
16 130
Dan-4 0.
0 20
56 ass
82 1m
12
22
entsoccer- watooccce-.rraerrrrats-ioioiigee
2 23 a0
0. 1400>000 0
0 0ue - 04
28 am
sso
28 a8
86
cube cube-double-ae -bo.blooppayyriiaeleaaeeeee
01 a.
14 - 10 go
55 63o 100
1ss 50 asa
D 100
0 60 a3o
82 -: 100
puzzle-4ae -aclee--r--pay--y singleingletasse
32 -
8 24
14 :30
00 + +00.
D Do - 100 21
cs
as 2:32
antaiaze-une
96
100
antzaze-unaze
80 - 61
74 98
32 + 98
0 - 94
69 =20 93
79 810 100
72 90
98
72 99
0. 98 #:
218 93
17
antraza-raddur-dtvaree
99
62
97
55
antzaze-large-play-
38 4 64
68 4 32
32 97
0 98
214 - 80 25
66 240 2m
0.e
96
234
pen-claned-y1
84 102
74-+ 138
-3 -3
3 120
77
11 149
dnor-clonad-vl
1
0-2
-0 -0-+00
D 102 27
3 50 2:1
0 102 45
bamer-clonad-vl.
128
velocete-oloaed-v1
0
1-7
-0--0
2
-0
0.11 62
E. Experimental Details
We implement FQL and many of the baselines in JAX (Bradbury et al., 2018) top of OGBench's reference implementa- tions (Park al., 2025). We provide our full implementation and exact commands to reproduce the main results of FQL https://githur com/ conmsseoonngarrk/fg
E.1. Environments, Tasks, and Datasets
OGBench (Park et al., 2025). OGBench is main benchmark, and 10 environments, 50 state- based tasks, and 5 pixel-based tasks from OGBench. Since OGBench was originally designed for offline goal-conditioned RI., we use the single-task variants ""=singletask"" of OGBench tasks to benchmark standard reward-maximizing offline RL methods. Each OGBench environment provides five evaluation goals, each which defines a different task (=singletask-taate to -singletask -aasskk,,ak,,,...... and one of them set to be a default task (-singletask without a suffix). Given an evaluation goal, the corresponding singletask variant labels the transitions in the dataset with semi-sparse reward function. The
18

---

Flow Q-Learning
semi-sparse reward function (for the fixed task) is defined the negative of the number of remaining subtasks at given state. Locomotion tasks have only one subtask ("reach the goal"), and rewards always -1 0. Manipulation tasks usually involve more than one subtasks (e.g., "open the drawer", "turn the first button's color blue", etc.), and rewards are bounded by Rask and O, where Rask is the number of subtasks, to 16 in the set of environments we use. The episode ends when the agent achieves the goal.
In our experiments, we use the following 10 state-based and pixel-based datasets (each dataset provides different tasks).
based datasets
aattmazeellarre--aaaaaaa---
artuaaze-giant--avvigttttvv
attsocaar-arren--aaddgaaaa--
cube-singgeepllay-v0d
cube-double-plaay-rd
scene-play-v0
puzzle-3x3-play-v0
puzzle-4x4-play-vo
Pixel-based datasets
visuall-obbb--singge--ppla---
vieualloubbe-dbbbbee-pla----
visuall=scene-plaay-v0
vianaal-puzzzle--33--ppaa----
We choose these environments to cover diverse types of challenges. antmaze and humano iimanoiddmzee dmaze require controlling either a quadrupedal agent (with 8 degrees of freedom) or a humanoid agent (with 21 degrees of freedom) to reach goal position in given maze. antsoccer requires controlling a quadrupedal agent to dribble a ball to desired location. cube, scene, and puzzle require manipulating diverse objects with a robot arm, where scene involves long-horizon control of multiple objects (up to 8 subtasks) and puzzle requires combinatorial generalization. The tasks with the visual- prefix require pixel-based control solely from 64 x 64 3-sized images. For dataset types, employ the standard ones (navigate for locomotion and play for manipulation). These datasets feature high suboptimality since they consist of trajectories performing random tasks (e.g., reaching random goals manipulating random objects in the scene), and thus require high degree of "stitching" capabilities. We use all of the five tasks for each state-based environment, but we use only the first task (the one labeled as singletane singletask-task) for each pixel-based environment due to high computational cost. For evaluation, we consider binary task success rates (in percentage), following the original evaluation criterion.
D4RL (Fu al., 2020), To enable direct comparisons with previously reported results, we additionally employ 18 relatively hard D4RL tasks in experiments. We use the following 6 an antmaze and 12 adroit tasks.
antmaze-umaze-v2
minnzze-umazeeiivrree--e
mttmze-madiu-plaay--2
attnaze-rediumdddvveee2
antmaze-largg-playy22
attmaze-large-ddiere--=22
pen-human-v1
pen-clomed-vi
pen-expert-vi
door-human-vi
door-cloned-v1
door-expert-v1
hammer-human-vi
hammer-cloned-v1
hammer-eae -exper-e1
19

---

Flow Q-Learning
helocate-human-vl
relocate-cloone-vl
rellcate-expert-vv
D4RL antmaze has the same high-level objective as OGBench antmaze, but with different (relatively less challenging) maze layouts, datasets, and evaluation goals. adroit t tasks (pen, door, hammer, and relocate) require dexterous manipulation with a high-dimensional (24-D) action space. We measure binary task success rates (in percentage) for antmaze and normalized returns for adroit, following the original evaluation scheme (Fu al., 2020).
E.2. Methods and Hyperparameters
In this work, we consider a total of 11 previous offline RL and offline-to-online RL approaches. We use the same default hyperparameters, architecture, and codebase for previous methods, unless otherwise mentioned. Also, we individually tune the method-speciffe hyperparameters of prior approaches for each environment, as described in detail below. For OGBench tasks, we tune each method on the default task of each environment (i.e, the task corresponding the "-singletask" without task ID), and use the best hyperparameters for the other four tasks from the same environment.
BC. For behavioral cloning, we train a Gaussian policy with a unit standard deviation. We consider [256, 256, 256, 256]- and [512, 512, 511, 512]-sized MLPs and use the latter (which is also our default network size) for all environments.
IL (Kostrikov et al., 2022). We re-implement IQL on top of the same codebase as FQL. We perform a hyperparameter search over expectile values in {0.7, 0.9} and AWR inverse temperatures in {0.3, 3, 10}. We use a fixed expectile value of 0.9 for all environments, while the AWR inverse temperature a is individually tuned for each environment (Tables 6 and 7). We find that IQL tends to overfit on state-based OGBench manipulation tasks, and thus use smaller [256, 256, 256, 256]- sized MLPs for these state-based tasks (but not for pixel-based based tasks), which we find perform better.
ReBRAC (Tarasov et al., 2023a). We re-implement ReBRAC on the same codebase as FQL. ReBRAC has two major hyperparameters: the actor and critic BC coefficients. We consider {0.003, 0.01, 0.03, 0.1, 0.3, 1} for the actor BC coefficennt on and (0, 0.001, 0.01, 0.13 for the critic BC coefficient or2- Since actor regularization is generally (far) more important than critic regularization (Tarasov et al., 2023a), we first perform a sweep over actor BC coefficients without critic regularization, and perform second sweep over critic BC coefficients with the best actor BC coefficient. We report the individually tuned hyperparameters in Tables 6 and 7. We use the default values for the other hyperparameters (e.g., noise standard deviation, noise clipping threshold, etc.), and normalize Q values only in the actor loss, following the official implementation (Tarasov et al., 2023b).
IQL (Hansen-Estruch Estruch et al., 2023). We use the official open-source implementation of IDQL. For network architectures, we use the default residual multilayer perception (MLP) (three blocks of [256, 1024, - residual layers) for the behavioral diffusion policy and consider [[256, 256), [256, 256, 256, 256). (512,512, [512, 512, 512, 512] for the size of the value network. We find that using 4-layer value networks in this codebase Icads to unstable training, and thus choose [512, 512] for OGBench locomotion tasks and [256, 256) for OGBench manipulation tasks. We consider {0.7, 0.9) for the IQL expectile value, and (32, 64, 128} for the number of test-time action samples. We individually tune the number of action samples (N) for each task (Table 6), and use an IQL, expectile of 0.7 for OGBench locomotion and adroit t tasks and 0.9 for OGBench manipulation tasks. We use the default values for the other hyperparameters. Following the original training scheme, we train the agent for 3M steps (1.5M for value functions), three times longer than FQL's training epochs. For compatibility with our evaluation scheme, report the average performance over 2.5M, 2. 2.75M, and 3M steps for OGBench tasks, and the final performance for D4RL tasks.
SRPO (Chen et al., 2024b). For SRPO, we first used its official implementation to obtain OGBench results but were unable to achieve reasonable performance, despite initial hyperparameter sweeps. Hence, we re-implement SRPO on top of the codebase of IDQL (the closest method to SRPO), which we find to lead to better performance. We use the same tuned hyperparameters as IDQL for value learning and behavioral policy learning. For the Q coefficient (3 Chen et al. (2024b)), we perform hyperparameter search over (0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.3) and use the best one for each environment (Table 6).
Consistency-AC (CAC) (Ding & Jin, 2024). We use the official open-source implementation of Consistency-AC. We consider (0.003, 0.01, 0.03, 0.1, 0.3, 1} for the Q loss coefficient (1) in Ding & Jin (2024)) and use the best one for each environment (Table 6). For other hyperparameters for OGBench tasks, we mostly follow the default ones for D4RL antmaze tasks, as these are closest to OGBench tasks in that they both use sparse rewards and involve goal-reaching. Namely, we do
20

---

Flow Q-Learning
not normalize Q values, scale the consistency loss, and apply maximum Q backup. For D4RL antmaze, re-evaluate its performances on the -v2 tasks (the original paper uses -v0 tasks) with the hyperparameters provided in the official implementation. For D4RL adroit tasks, we mainly use the default hyperparameters tuned for adroit but perform an additional hyperparameter sweep over Q loss coefficients in (0.003, 0.01, 0.03} for the other tasks not used in the original paper (Table 6). For all tasks, we apply gradient clipping with a threshold of 5 and do not use online model selection to ensure a fair comparison.
FAWR, FBRAC, and IFQL. FAWR, FBRAC, and IFQL are implemented on top of the same codebase as FQL, shar- ing the same flw--macching implementation. To enable apples-to-apples comparisons, parameters as IQL for FIQL., and the same default ones FQL for FAWR and FBRAC. However, individually tune the policy extraction-related hyperparameters for each environment. For the inverse temperature a in FAWR (Equa- tion (10),, we consider {0.3, 1. 10}. For the number of test-time action samples N in IFQL (Equation (11)), we consider {32, 64, 128). For the BC coefficient a in FBRAC (Equation (6)), we consider {1000, 3000, 10000, 30000} for adroit tasks and {1, 3, 10, 30, 100, 300} for the other tasks. We present the sask-specific hyperparameters in Tables 6 and 7.
use the same default hyper-
Cal-QL (Nakamoto et al., 2023). We use the official implementation of Cal-QL. For the CQL regularizer coefficient a, we consider {0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10} well as its Lagrange dual variant with target action gaps 3 of {0.2, 0.5, 0.8]. We use individually tuned values of these hyperparameters for different tasks (Table 7). For the network size, consider both (256, 256, 256, 256] and [512, 512, 512, 512] -sized MIPs, and use [512, 512, 512, 512] for OGBench locomotion tasks and [256, 256, 256, 256] for OGBench manipulation tasks. We also consider scaling rewards by {1, 3, 10), and use a value of 10 to scale rewards for all tasks. We use the default values for the other hyperparameters (e.g., using a mixing ratio of 0.5, taking the maximum over 10 actions when computing target values, using importance sampling for the CQL regularizer, etc.).
RLPD (Ball al., 2023). We re-implement RLPD on top of the same codebase FQL. To ensure a fair comparison with other methods, we use update-to-data ratio of 1 and employ two Q functions. Clipped double Q-leaming is only applied to D4RL adroit tasks, as in FQL. We do not use entropy backups, we find it to be better.
FQL. See Appendix B.
We provide the complete list of hyperparameters in Table and task-specific hyperparameters in Tables 6 and
21

---

Flow Q-Learning
Table 5. Hyperparameters for FQI..
Hyperparameter
Value
Leaming rate
0.0003
Optimizer
Adam (Kingma Ba, 2015)
Gradient stops
1000000 (default), 500000 (D4RL, pixel-based OGBench)
Minibatch size
256
MLP dimensions
[512, 512, 512]
Nonlincarity
GELU (Hendrycks & Gimpel, 2016)
Target network smoothing coe coefficient
0.005
Discount factor
0.99 (default), 0.995 (antmaze -giant, humamoidnaze. antaoccer)
Image augmentation probability
0.5
Flow steps
10
Flow time sampling distribution
Unif((0, 1])
Clipped double Q-Icarning
False (default), True (adroit, antmaze- antnaze-(large, giant) glant)-navigatt navigate)
coefficient a
Tables
22

---

Table Task-specific hyperparameters for offline RL. We refer to Appendix E.2 for the description for each hyperparameter variable. We individually these hyperparameters for each task, but in OGBench, tune them the default task (denoted by (*)) and the best hyperparameters for the other four tasks. indicates that the corresponding result is taken from the prior work (or does not exist).
IQL
ReBRAC
IDQL
SRPD
CAC
FANAC
FERAC
IFQL
PQL
Task
((n),a))
NV
antaisze hagpaa/eeg.a.gegggggggeeeeggggao
(+)
10
(0.003, 0.01]
32
3
32
10
(0.001, 0.01]
32
0.3
3
32
10
10
(0.003, 0.01)
32
0.3
3
3
32
10
0.011
10
(0.003, 0.01)
32
0.3
3
32
10
(4)
10
(0.003, 0.01)
32
0.3
3
10
32
10
(0.003, 0.01)
10
(0.003, 0.01)
32
0.3
3
10
32
10
10
(0.003, 0.01]
32
0.3
3
10
312
10
10
(0.003, 0.011
10
32
bunanol bat.
(+)
10
(0.01,0.01)
32
0.3
0.03
3
30
32
30
bunanoj d 3
10
(0.01, 0.01)
32
0.3
0.03
3
30
322
30
bunanoi wwwamooiddaeeeer x
(0.01,0.01)
32
10
(0.01,0.01]
32
0.3
0.03
3
30
32
30
bunanoi b bug.
10
0.01)
3
32
Sag.a/a........o..////////gan/aa (4)
hunanoi pumanoodfeaze-laarrrrrrrrapaaaap
10
(0.01.0.01)
3
30
32
30
bunanoi
10
(0.01.0.01)
32
0.3
3
30
32
30
10
(0.01.0.01)
3
32
10
(0.01,0.01)
32
0.3
3
30
32
30
10
(0.01.0.01)
32
0.3
3
30
32
30
(0.01,0.01)
10
(0.01.0.01)
32
10
30
61
(0.01,0.01)
32
0.03
10
30
64
10
(0:01,0.00 0.01)
10
(0.01,
322
0.03
10
30
64
10
(1.0)
32
0.03
0.003
1010
322
3010
100
(1.0)
32
0.03
0.003
100
32
300
(1.0)
32
0.003
100
32
300
(1,0)
0.003
100
32
300
0.3
(0.1.0)
12
0.1
0.3
0.3
100
32
300
0.3
(0.1.0)
0.3
100
32
300
0.3
(0.1,0)
0.3
0.3
100
32
3010
0.3
(0.1,0)
32
011
0.3
0.3
100
32
300
(0.1,
scene
10 (0.1,
32
011
0.3
0.3
1010
32
3010
reeterppay-eiiniitass-eeeeeee (+)
10
(0.1,0.01)
32
0.1
0.3
0.3
100
32
300
scene
0.01)
10 (0.1,
32
0.1
0.3
0.3
100
32
300
scene play-eelay
10
(0.1, 0.01)
32
0.1
0.3
100
32
300
(0.3. 0.01)
0.3
10
(0.3,0.01)
32
0.1
0.01
0.3
100
32
1000
10
(0.3. 0.01)
32
0.1
0.01
0.3
100
32
1000
10 (0.3,
0.3
100
32
1000
(0.3, 10
322
0.1
0.01
0.3
1010
32
1000
(0.3. 0.01)
32
0.1
0.01
0.3
3010
32
1010
(0.3,
0.01
0.3
300
32
1000
(0.3,0.01)
322
0.1
0.01
0.3
300
32
1000
sue2lee-x4-play-sinng wta.kgglttttttateeeeke taak4400
(+)
(0.3, (.03)0001)
0.3
300
32
puzale-Exi-pp pxatee--x---ppyy-ii eingleeeee baacgggleeal-eoo taskkkkkk----- tv
(0.3,0.01)
32
0.1
0.01
0.3
300
32
1000
antaase uulanazeunaaeev2 uvaze-v2
0.01
3
10
32
10
3
10
antaase purp
0.01
3
10
32
10
0.01
3
10
32
10
Lamnze-large-play--22
aatnaza-largo-dvverse
3.5
3
32
pea-hur.an-rl
32
0.03
0.003
0.03
30000
32
10000
pea-cloaed-vi
0.3
pea-expert-vi
32
0.1
0.03
0.1
30000
32
3000
boor-bunen-vi
32
0.01
30000
32
30000
door-clonad-ol
10000
128
30000
door-expert-vi
32
0.01
0.03
3
30000
32
30000
hammar-hunan-vl
128
3
30000
banner-clonddvi1
32
0.1
0.003
0.03
10000
32
10000
hamar-oxpert-p1
32
0.03
0.03
3
30000
32
30000
relocate-huran-v1
0.3
30000
128
10000
rel
relocate-ccommddvr
64
0.03
0.01
0.1
3000
32
30000
relocate-oxpert-v
32
0.01
0.003
1
300100
32
30000
1010
0.3
(0.1.0)
1010
32
100
scenel-sene-plly elagllllleeetssssss taskiiv0
10
(..1,
100
32
100
sisual-puzzle-3te saszzle--3d-plyylle paappuull-----ll-----ee--- plygl--iglllaaa--ttaa
10
(0.3.
1010
32
3010
visual visual-puzzle- puzzle-4zeh 4aa--e--dpppaay
play--iingll singletack---a wlack-taaeee taski-v0
3
(0.3,0.01)
300
32
300

---

Flow Q-Learning
Table Tpee-ip hyperparameters for offinne-to-nline RL. We refer to Appendix E.2 for the description for each hyperparameter variable. We individually tune these hyperparameters for each task, and "-" indicates that the corresponding result is taken from the prior work.
IQL
ReBRAC (en. Oz)
Cal-QL (u,
IFQL
FQL
Task
busenoidnssee medi ted.
(0.01,0.01)
(-,0 0.8)
100
antwoccer aateoccer-areea narigate siggletak-o
(1001,0.0))
(- 0.2)
64
30
wube-double-ple play-singletaskk0
(0.1,0)
[0.01.-]
singleaaat
10
(0.1,
(0.01.
300
(0.3. 0.01)
(0.003.
1000
antmase-umaze-12
10
attmaze-nadimm-laay-22
aatmaze-neddun-ddvvree----
antrazeellarge-plaa--222
32
3
aatmzze-arrg-biivrree-.
pen-cleaed-v1
door-cloned-vi
128
1000
clmner-closed-v1
1000
relocate-clone-v1
10000
24


Reviews:
[{"review_id": "Iyc15gj8hQ", "reviewer": "Reviewer_YTug", "summary": "This paper proposes the offline-RL method Flow-Q-Learning (FQL) which leverages  an expressive flow-based generative model for modeling the action distribution while avoiding common issues such as unstable backprop through each time-step or less efficient re-weighting schemes. This is achieved by introducing a so-called “1-step” policy which is used to minimize the Q function while being regularized to match the behavior behavior cloning flow-model that learns the offline transitions from data. This 1-step model can then be evaluated efficiently and shows increased performance across many OGBench and D4RL tasks compared to previous offline methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the highly detailed review and constructive feedback about this work. We especially appreciate your question about the expressivity of the one-step policy, for which we conducted an additional experiment. Please find our response below.\n\n---\n\n* **Is the one-step policy expressive enough?**\n\nThanks for asking this valid question. To empirically address it, we performed a controlled ablation study to compare the expressivity of (1) Gaussian BC, (2) full flow BC, and (3) one-step distilled BC policies. Here, we evaluate their *BC* performance (not RL performance) to solely focus on their expressivity without being confounded by different policy extraction strategies (in particular, note that there's no clear, straightforward way to train a full flow policy with RL).\n\nSpecifically, we trained these three types of policies on $4$ goal-conditioned OGBench tasks, and measured their goal-conditioned BC performance. To ensure a fair comparison, we used the same architecture and common hyperparameters, and ablated only the policy class. The table below shows the mean and standard deviation across 8 seeds at 1M gradient steps (200K steps for $\\texttt{cube-single}$ due to overfitting).\n\n| Task | Gaussian | Full Flow | One-Step Distillation |\n|-|-|-|-|\n| $\\texttt{antmaze-medium}$ | $77 \\pm 3$ | $82 \\pm 6$ | $85 \\pm 6$ |\n| $\\texttt{cube-single}$ | $54 \\pm 17$ | $81 \\pm 15$ | $90 \\pm 3$ |\n| $\\texttt{cube-double}$ | $4 \\pm 3$ | $15 \\pm 9$ | $19 \\pm 9$ |\n| $\\texttt{scene}$ | $13 \\pm 3$ | $34 \\pm 5$ | $34 \\pm 8$ |\n\nThe results show that one-step distilled policies achieve nearly identical performance to full flow policies on these tasks, and both flow variants generally outperform Gaussian policies.\n\nOverall, we expect that one-step policies should be expressive enough to model complex action distributions in many practical scenarios, especially considering that one-step flow models can generate highly realistic samples [1, 2] even for high-dimensional image generation.\n\n[1] Liu et al., Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (2023). \\\n[2] Frans et al., One Step Diffusion via Shortcut Models (2025).\n\n* **Potential trade-off by using a one-step policy.**\n\nAs shown in the table above, one-step policies often have a very similar expressivity to full flow policies (at least on our benchmark tasks). Hence, we expect the \"trade-off cost\" of using a one-step policy to be generally marginal. That said, this trade-off might come into play if the task requires *highly* precise control and thus the full expressivity of iterative generative models. In this case, one solution would be to relax FQL to train a *few*-step distillation policy to strike a balance between precision and the number of recursive backpropagations. We leave this extension of FQL for future work.\n\n* **Scalability to higher state-action dimensions and real-world data sources.**\n\nThanks for the question. We would like to first note that we have evaluated FQL on a diverse set of high-dimensional benchmark tasks (e.g., $\\texttt{visual-*}$ tasks require *pixel*-based control, $\\texttt{humanoidmaze}$ requires $21$-DoF whole-body control, and $\\texttt{adroit}$ tasks require $24$-DoF dexterous manipulation). Moreover, the OGBench tasks used in this work are generally more complex than the D4RL tasks used in prior work, and they feature noisy, non-Markovian, multi-modal action distributions that (partly) resemble real-world datasets. That said, we did not evaluate FQL on real robotics data, as this work focused more on the algorithmic side of the method. We believe applying FQL to real robots with pre-trained VLA BC flow models [3] is a particularly exciting direction for future research. We will mention the lack of real-world experiments as a limitation (and discuss relevant work) in the final version of the paper.\n\n[3] Black et al., π0: A Vision-Language-Action Flow Model for General Robot Control (2024).\n\n* **Notational suggestions (e.g., $a^\\theta$ instead of $a^\\pi$) and minor clarifications.**\n\nThanks for the helpful suggestions! We will incorporate them into the camera-ready version. In the remark box, as the reviewer correctly pointed out, $\\pi^\\theta$ and $\\nu^\\theta$ correspond to the same distribution. Our original intention was to distinguish $\\nu^\\theta$ as a probability measure, but in hindsight, we feel that this distinction is unnecessary, and we will revise the paper to use the same $\\pi^\\theta$ notation in the remark box.\n\n---\n\nWe would like to thank you again for raising important questions about FQL, and we believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions.", "claims_and_evidence": "-The paper claims that FQL outperforms previous offline RL methods on a wide range of tasks and in particular this type of behavior constrained policy extraction method is better than previous flow/diffusion based extraction schemes like advantage weighted flow-matching.\n\n-They claim that this 1-step policy preserves the expressivity of the underlying action distribution learned by the BC flow-matching model.\n\n-Training FQL is more computationally efficient and effective than related flow-based policy extraction approaches.\n\nI think that all claims are pretty well substantiated by the results experimentally except for the 1-step policy expressiveness other than the remark about how the distillation loss is an upper bound for the W2 distance between the policies. For this reason it makes sense that the parameter \\alpha needs to be tuned carefully.", "methods_and_evaluation_criteria": "The proposed method is very practical with the application of offline RL in mind. All design choices are well supported by the OGBench and D4RL benchmarks.", "theoretical_claims": "This paper does not really make any theoretical claims except for the connection between the W2 distance and the distillation regularizer. I didn't check this claim carefully, but the authors do ablate the parameter alpha.", "experimental_designs_or_analyses": "The experimental section to be very thorough when comparing against previous offline RL approaches. I found the QA format of the discussion engaging and informative to read from a practitioner's standpoint.\n\nIt would be nice to better understand the limitations of FQL either in scalability due to the simulation-based loss (atleast it requires a forward pass of the ODE) or task expressiveness and when it does not compare well to simple Gaussian policies.", "relation_to_broader_scientific_literature": "The paper contextualizes its contribution quite well with existing research integrating generative models (diffusion, flow) with RL. It spends quite a lot of time distinguishing the mechanisms used from works like Diffusion-QL, IDQL, and CAC and why FQL might be more performant.", "essential_references_not_discussed": "I don't feel too strongly about this, but It might be useful to discuss how offline RL and FQL deal with more realistic data sources /real world data with noise. Perhaps this one is suitable:\n\nZhou 2022, Real World Offline Reinforcement Learning with Realistic Data Source", "other_strengths_and_weaknesses": "Strengths:\n-The methodology is clear and simple, justified from the successes and failures of previous works.\n\n-Strong empirical results and ablations\n\n-Very clear and informative discussion of previous approaches which I believe is useful to practitioners.\n\nWeaknesses:\n-Some aspects of FQL aren’t fully addressed such as scalability to state-action dimension and real world data sources. \n\n-Its not very clear how expressive the 1-step policy and how it trades off BC policy distribution and the bias/titling of the distribution towards the Q function. There is probably a trade-off between staying close to the expressive flow-model state-action representation and decreasing the Q-function more quickly.", "other_comments_or_suggestions": "This is minor, but I would find the loss function and the algorithm more readable if instead of the notation “a^\\pi” you used “a^\\omega” or “a^\\theta” to help distinguish which policy the action is sampled from. \n\nIn the Remark, It seems like \\pi^\\theta and \\nu^\\theta are the same thing?", "questions_for_authors": "1. Is there a way to clearly demonstrate the expressiveness of the 1-step policy and how it trades off BC policy distribution and the bias/titling of the distribution towards the Q function? There is probably a trade-off between staying close to the expressive flow-model state-action representation and decreasing the Q-function more quickly.\n\n2. How does FQL scale with state-action dimension and real world data sources which are potentially noisy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k2cIoklOqE", "reviewer": "Reviewer_4ovk", "summary": "This paper propose Flow Q-learning, and offline reinforcement learning that integrates expressive flow-matching policies for modeling complex action distributions.\n\n## Update after rebuttal: \nI have read the rebuttal and the discussions from other reviewers. I am maintaining my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback about this work! We would be happy to address any additional questions or concerns you may have, so please feel free to let us know. If there are no further concerns or questions, would you consider raising your rating?", "claims_and_evidence": "The paper identifies a clear challenge in offline RL related to using flow or diffusion policies. It presents an elegant solution: training an expressive one-step policy separately from the iterative flow policy, which is both theoretically sound and empirically effective.", "methods_and_evaluation_criteria": "The approach is grounded in existing behavior-regularized actor-critic frameworks but innovatively avoids costly BPTT during RL training.", "theoretical_claims": "The derivation relating the distillation loss to a Wasserstein behavioral regularizer provides additional insight and theoretical justification for why the proposed method might perform better.", "experimental_designs_or_analyses": "Extensive experiments are performed, demonstrating consistently strong performance across benchmark tasks.", "relation_to_broader_scientific_literature": "The key contributions of this paper are related to offline reinforcement learning, generative modeling (specifically flow matching), and policy extraction techniques.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EtJ5yXN5lN", "reviewer": "Reviewer_HKD1", "summary": "The paper introduces Flow Q-learning (FQL), an offline RL method that combines flow-matching policies with Q-learning to address challenges in modeling complex action distributions. FQL uses two components: (1) an expressive flow-matching policy trained via behavioral cloning (BC) to capture multimodal dataset actions, and (2) a separate one-step policy trained with Q-learning to maximize values while distilling knowledge from the flow model. This decoupling avoids unstable recursive backpropagation through iterative flow steps and eliminates costly iterative action generation during evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback on this work. We especially appreciate your pointing out related work in different domains. Please find our response below.\n\n---\n\n* **Inference details of policies.**\n\nThanks for asking this clarification question! In FQL, *neither* the BC flow policy nor the one-step policy uses CFG (or CG), as there are no separate class labels in our problem setting. More specifically, the one-step policy $\\mu_\\omega(s, z)$ is simply a standard feedforward neural network (so no iterative sampling procedure is needed), and the BC flow policy $\\mu_\\theta(s, z)$ is based on the standard Euler method for ODE solving. At evaluation (inference) time, only the one-step policy is used (L160), and no iterative process is involved. We refer to the algorithm box in the main text (Algorithm 1) as well as L679-L688 in Appendix C for the full description of the sampling procedure.\n\n* **Related works on image diffusion models.**\n\nThanks for pointing out these relevant works in image diffusion models! We will cite and discuss them in the camera-ready version.\n\n---\n\nWe would like to thank you again for raising clarification questions and suggesting several related works. Please feel free to let us know if you have any additional questions or concerns. If we have addressed your concerns, would you consider raising your rating?", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "D4RL", "relation_to_broader_scientific_literature": "The article provides a detailed discussion on the relevant literature concerning RL with diffusion and flow models.", "essential_references_not_discussed": "The method proposed by the authors bears a resemblance to reward distillation in alignment for image diffusion, where a few-step model is distilled while simultaneously maximizing the reward.\n\n[1] Reward Guided Latent Consistency Distillation​\n\n[2] DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "Could you provide more details about the inference step for the BC Flow Policy? Specifically, I would like to know whether the BC Flow Policy incorporates classifier-free guidance (CFG) during its inference process. Additionally, I’m curious if the one-step policy also utilizes CFG.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZvTWpR3CgV", "reviewer": "Reviewer_NB9x", "summary": "This paper proposes using flow models to tackle offline RL tasks. To leverage the Q-function for guiding the learning of the flow-based policy while avoiding the computational cost of multiple backpropagations, this paper proposes learning a one-step action generation policy through a flow policy constrained distillation loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback on this work. We especially appreciate the clarification questions about deterministic policies and policy constraints, as well as several helpful suggestions. We also conducted an additional ablation study on ODE solvers. Please find our response below.\n\n---\n\n* **Would a better ODE solver/different marginal probability path improve performance?**\n\nThank you for the question. Following the suggestion, we conducted an additional ablation study to compare different ODE solvers for FQL. We consider three ODE solvers in this experiment: (1) the Euler method (**default**), (2) the midpoint method, and (3) the Runge-Kutta (RK4) method. The table below compares the final performance of these ODE solvers on $3$ OGBench tasks ($8$ seeds, $\\pm$ denotes standard deviations):\n\n| Task | Euler (Default) | Midpoint | RK4 |\n|-|-|-|-|\n| $\\texttt{antmaze-large}$ | $76 \\pm 23$ | $83 \\pm 6$ | $82 \\pm 4$ |\n| $\\texttt{cube-double}$ | $56 \\pm 16$ | $54 \\pm 17$ | $7 \\pm 20$ |\n| $\\texttt{scene}$ | $87 \\pm 8$ | $83 \\pm 12$ | $83 \\pm 5$ |\n\nThe results suggest that the performance of FQL is generally robust to the choice of ODE solver, except that RK4 leads to somewhat unstable performance on $\\texttt{cube-double}$. While we have not tested different marginal probability paths, we expect similar results, given that the performance of FQL is generally robust to other flow-related hyperparameters (Figure 9). That said, we believe a better ODE solver or marginal path sampler might further improve performance in more complex tasks and datasets.\n\n* **About different kinds of policy constraints.**\n\nAs the reviewer correctly pointed out, Gaussian behavior-constrained actor-critic methods (e.g., ReBRAC) minimize the distance between the RL policy's actions and dataset actions, while FQL minimizes the distance between the RL policy's actions and the BC policy's actions. They both serve the same role of a behavioral regularizer. The main difference is that ReBRAC *directly* imposes a *Gaussian*-based constraint, while FQL *indirectly* imposes a *flow*-based constraint. We note that FBRAC in our experiment lies precisely between the two: it *directly* imposes a *flow*-based behavioral constraint.\n\nWe have empirically compared these $3$ policy extraction schemes (ReBRAC, FBRAC, and FQL) in Table 2 of the paper. The result shows that our indirect flow-based behavioral constraint (FQL) indeed leads to better performance than both the direct Gaussian-based behavioral constraint (ReBRAC) and direct flow-based behavioral constraint (FBRAC). Following the suggestion, we will further clarify the differences between these policy constraints in the final version of the paper.\n\n* **Is $\\mu_\\omega$ in L139 a deterministic policy?**\n\nAs mentioned in the \"Notational Warning\" paragraph (L126), $\\mu_\\omega(s, z)$ is a deterministic function. However, this does **not** mean that we perform distillation into a deterministic policy: even though $\\mu_\\omega(s, z)$ is a deterministic function, since $z$ is a random variable (sampled from $\\mathcal{N}(0, I)$), it serves as a *stochastic* policy $\\pi_\\omega(a \\mid s)$ when marginalizing out $z$. As a result, the one-step policy clones the flow BC policy **for each $\\mathbf{z}$** through *deterministic distillation*, while maximizing the Q function with *stochastic* actions (Eq. (8)). While we (partly) explained this subtlety in the \"Notational Warning\" paragraph, we will further clarify this point in the final draft to prevent any potential confusion.\n\n* **About the remark box.**\n\nThe purpose of the remark box is to provide further theoretical insight into the policy constraint of FQL and to discuss its relation to previous offline RL methods (TD3+BC, AWAC, CQL, etc.). We believe this section can be safely skipped if the reader is mainly interested in empirical results and methodology (which is the main reason we formatted this discussion with a separate box).\n\n* **Parameter comparisons between policies.**\n\nTo clarify, we used the same [512, 512, 512, 512]-sized MLPs *all* networks (including flow policies, one-step policies, and Gaussian policies; Table 5), unless otherwise noted, and they have almost identical numbers of parameters. There are some exceptions (e.g., IDQL), but we used smaller networks *only* when the smaller ones performed better than the default-sized ones. We fully described the way we chose these hyperparameters in Appendix F.2. We will further clarify this point in the final draft.\n\n* **Notational suggestions.**\n\nThanks for the helpful suggestions! We will incorporate them in the camera-ready version.\n\n---\n\nWe would like to thank the reviewer again for raising important questions about FQL. We believe the additional results and clarifications have significantly improved the quality of the paper. Please let us know if you have any additional questions or concerns. If we have addressed your concerns, would you consider raising your rating?", "claims_and_evidence": "Most of the claims in this paper are well supported, but some parts remain difficult to understand. For details, please refer to the questions and comments.", "methods_and_evaluation_criteria": "This paper compares several classic offline RL methods.", "theoretical_claims": "I have read the theoretical parts related to the paper.", "experimental_designs_or_analyses": "Please refer to the questions and comments for experimental concerns.", "relation_to_broader_scientific_literature": "In recent years, diffusion-based RL methods have shown great potential in offline RL tasks. As a more general generative model compared to diffusion models, applying flow models to RL is a promising research direction.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Please refer to the questions and comments for review concerns.", "other_comments_or_suggestions": "1. In line 110: To make the description clearer and more intuitive, I suggest the authors directly use notations like $a^1$ and a^0 to represent actions at different time points.\n2. In line 139 right column: Whether \\mu_w serves as deterministic policy? In other words, you want to distill a deterministic policy that maximizes the Q function and, at the same time, minimizes the output discrepancy with flow policy $\\mu_\\theta$.\n3. I suggest to add the parameters comparison between $\\mu_w$, $\\mu_\\theta$, and Gaussian policies.\n4. Compared to previous baselines, the key difference in the method used in this paper lies in the policy constraint. Specifically, it measures the distance between a deterministic policy and an expressive policy, whereas previous Gaussian policy methods impose constraints on the state-to-action mapping learned from the dataset, and the diffusion policies measure the distance between two expressive policies. It is better to understand this paper by discussing the differences among these types of applying policy constraints.", "questions_for_authors": "1. Is the analysis in the Remark redundant? It does not seem to contribute much to the understanding of the proposed method.\n2. Would using a better ODE solver lead to improved performance? Would using a different marginal probability path result in better performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents flow Q-learning (FQL), a simple and effective offline RL method using an expressive flow-matching policy to model complex action distributions in data, addressing training challenges by training a one-step policy instead of an iterative flow policy. After the rebuttal discussion, the reviewers consistently think this paper can be accepted. I personally like this paper as it tackles the fundamental challenge for modeling complex action distribution through a simple flow-based algorithm.
Decision: Accept (poster)