Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
Authors: Franki NGUIMATSIA TIOFACK
Théotime Le Hellard
Fabian Schramm
Nicolas Perrin-Gilbert
Justin Carpentier
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Reinforcement Learning
Behavior Cloning
Flow Matching
Primary area: reinforcement learning

Abstract: Offline reinforcement learning often relies on behavior regularization that enforces policies to remain close to the dataset distribution.  However, such approaches fail to distinguish between high-value and low-value actions in their regularization components. We introduce Guided Flow Policy (GFP), which couples a multi-step flow-matching policy with a distilled one-step actor. The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset rather than indiscriminately imitating all state-action pairs. In turn, the flow policy constrains the actor to remain aligned with the dataset's best transitions while maximizing the critic. This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks from the OGBench, Minari, and D4RL benchmarks, with substantial gains on suboptimal datasets and challenging tasks.

Published as a conference paper at ICLR 2026
GUIDED FLOW POLICY:
LEARNING FROM HIGH- VALUE ACTIONS IN OFFLINE REINFORCEMENT LEARNING
Franki Nguimatsia Tiofack! 1:" Nicolas P
Théotime Le Hellard" '
Fabian Schramm!
Justin Carpentier!
Inria and DI-ENS, PSL Research University Soobonne Université, CNRS, ISIR, France Equal contribution. Correspondence to: franki www.iatssiaaaaaamammmm..a teggummttsia-tiiackeeaeeaa wwwfackiinriiaa..... fr
ABSTRACT
Offline reinforcement learning often relies on behavior regularization that enforces policies to remain close to the dataset distribution. However, such approaches fail distinguish between high-value and low-value actions in their regularization components. We introduce Guided Flow Policy (GFP), which couples a multi- step flow-mal matching policy with a distilled one-step actor. The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value ac- tions from the dataset rather than indiscriminately imitating all state-action pairs. In turn, the flow policy constrains the actor to remain aligned with the dataset's s best transitions while maximizing the critic. This mutual guidance enables GFP to achieve state-of-the-an thrt performance across 144 state and pixel-based tasks from the OGBench, Minari, and D4RL benchmarks, with substantial gains on subopti- mal datasets and challenging tasks.
Webpage: hwtp://wwwoi.ohiiioooooooiiiiiaa
INTRODUCTION
Offline Reinforcement Learning (RL) aims to learn effective policies from static datasets without
further interaction with the environment (S. Lange 2012 Ernst et al. (005). This paradigm is essential in domains such as robotics and logistics, where online exploration can be unsafe or costly.
However, standard off-policy algorithms such as DDPG (Lillicrap et al. 2015) and SAC (Haarnoja
al. 2018), which are successful in online RL, tend to underperform in offline settings since the RL agent cannot interact with the environment. The main challenge is extrapolation error, corresponding to the inability to properly evaluate out-of-distribution actions Wu al. 2019 Fujimoto et al. 2019
Kumar et al. 2019 2020).
Two main lines of work have been proposed to address this challenge. The first one focuses learning a critic without querying the values of actions outside the dataset (Kostrikov et al.) 2021 Nair et al. 2020). The second one, known the Behavior Regularr-ee Actor-Critic (BRAC) fam-
i mitigates these errors by forcing the learned policy to stay "close" to the unknown behavior policy that generated the dataset (Fujimoto & Gu 2021 Tarasov al. 20233 Jaques et al. 2019
Laroche et al. 2019;; Wu et al.) 2019). The key idea is that out-of-distributie state-action pairs Q-value overestimation, while staying near the empirical distribution reduces extrapolation errors. Minimalist variants achieve this by simply adding a behavior cloning (BC) loss to the policy and/or value updates with respect to dataset actions (Fujimoto Gu [021]
are especially vulnerable
Tarasov et al. 2023). Although this approach improves stability, it also raises trade-off: regular- izing too strictly to a potentially suboptimal dataset action may restrict the policy from exploiting higher-reward actions contained in the dataset.
Until recently, most offline RL algorithms were based on Gaussian policies, with limited modeling capacities. Recent development of flow and diffusion baased expressive models (Lipman al. 2022
Ho et al. 2020 Song et al.) 2020), led to new RL algorithms to capture complex and multimodal
'The abbreviation BRAC refers to the family of methods, solely the BRAC paper (u et al. 2019

---

Published as a conference paper at ICLR 2026
Flow policy (-(s)
Offline RI inputs at a given state a Unknown behavior distribution
value-aware behavior cloning
0
Optional, see Eig
Distillation
Guidance In (s,
Actions associated
Critic Qø(s,))
Actor (-(-ss) one-step policy
Actor-critic relation
Figure 1: Overview of the Guided Flow Policy framework. GFP consists of three main components: (i) in yellow, VaBC, multi-step flow policy The trained weighted BC using the guidance term Sty (ii) green, a one-step actor ne distilled from the flow policy, and (ii) in gray. a critic Qo guiding action evaluation. Tu regularizes the actor toward high-value actions from the dataset D; in turn, the actor shapes the flow and optimizes the critic following the actor-critic approach. The different components of the figure are introduced throughout the paper. Each drawing represents the probability distribution of actions current state s, except for the gray ones, where is value of actions E AA in state s, according the critic.
A of policy, in
action distributions (Chi et al. 2023 Janner et al. 2022 Wang et al.] [022 Zhang et al. 2025). ). How-
ever, they come the risk of high computational overhead: iterative sampling slows inference, and directly optimizing the values of output actions would result in unstable backpropagation through time (BPTT). Among recent approaches to address these challenges, Park al. (2025) proposed BRAC method with a flow-matching BC model distilled into one-step policy that also optimizes the critic, enabling expressive policy learning while avoiding BPTT and iterative sampling at infer- ence. Despite these advances, a central limitation remains: the flow-based BC component, similar to standard BC, does not incorporate reward information.
We propose Guided Flow Policy (GFP), a dual-policy BRAC framework with a bidirectional guid- ance mechanism between a multi-step flow-matching policy, termed Value-aware Behavior Cloning (VaBC), and distilled one-step actor. VaBC acts distributional regularizer for the actor, en- couraging it to remain within the support of the behavior policy. VaBC is trained via a weighted-BB mechanism, close to Pene et al. (2019); Zhang et al.) (2025), but leveraging the actor and the critic to prioritize cloning high-value actions from the dataset. Unlike previous BRAC approaches, in which the BC regularization indiscriminately treats all state-action pairs (Fujimoto & Gul 2021] Park al., 2025), VaBC is a value-aware regularizer integrated into a BRAC approach. In turn, the actor opti- mizes the critic while being distilled toward VaBC, allowing align with the dataset's 's high-value actions in a given state while maximizing expected returns. Fig. iislustrttteee the GFP framework and Tab. shows how it differs in the regularization mechanisms compared to other BRAC methods.
Our contributions are threefold: (i) we introduce Guided Flow Policy, a simple yet effective BRAC method that integrates awarenesssssss in the regularization term via a jointly trained weighted flow BC policy, thereby regularizing the actor with the most promising transitions of the dataset; (ii) we extensively evaluate GFP on 144 tasks from standard offline RI benchmarks, showing strong performances with substantial gains suboptimal datasets and challenging tasks compared to prior works; (ii) we re-assess two previous state-of-the-an offline RL algorithms on these benchmarks, highlighting the critical role of hyperparameter choices and subtle implementation details, aligned in the spirit with the retrospective analysis provided in Tarasov et al. (2023).
BACKGROUND
Actor-critic framework in RL. RL problems are typically formalized as a Markov Decision Pro- cess (MDP) (Sutton al. 1998 Konda & Tsitsiklis, 1999), defined by the tuple (S, A, p, Here, denotes the state space, A the action space, the transition dynamics, r the reward func- tion, p the initial state distribution, and 7 E [0. 1) the discount factor. The behavior of the agent is
2

---

Published as a conference paper at ICLR 2026
Table 1: Overview of regularization mechanisms within the BRAC framework.
Regularization langet
Value regulatization
Expressive variant
TD3+BC (Lepmete Gn 2011
ReRRAC [haisow eral.
doteset actions
Diffusion-Q2
Wang at
PQL Park et al. 2025
leamned behavior cloning policy
GFP (ours)
leamed value- behavior claning policy
governed by a policy x. mapping states to probability distributions over actions. The objective is to maximize the expected discounted return (e(l)(())))) \_==o"r(se, at)], i.e., the expected cumula- tive reward when following x in the MDP. In actor-critic approaches, the policy æ, also referred to as the actor, is trained jointly with a critic Q, which approximates the state-action value function. The Q-function is defined as Q" (s, a) = Bana(()s))
>x 1 7'r (st, Qt) = Go a). estimating
the expected return after taking action a in state and subsequently following n.
Both actor and critic are parametrized as neural networks, with parameters @ and & respectively, and optimized by alternating gradient descent steps on the two objectives:
(A EgnD, D. (11)) C (0) = (' ((
Qo(s, ag)].
-r (Q)(s,a)-r-
(2)
where LA and L refer to the actor and critic losses, respectively, and D is the set of transi- tions (s, a, r, s') collected during training. Q3 denotes second target Q-function parameterized by a slowly updated set of weights o, maintained via Polyak averaging, a common stabilization technique in actor-critic methods.
Minimalist approaches in offline RL. In offline RL, the agent learns exclusively from a static dataset D, consisting of transitions (s, a, r, r') generated by an unknown behavior policy. In given state the distribution of actions of such a behavior policy is illustrated on the left of Fig. [ This introduces key challenge compared the online setting: the distributional shift (S. Lange 2012
Kumar et al.) 2019 Konda & Tsitsiklis 1999). Indeed, since the Iearned policy may select actions outside the dataset's ' support, value estimates for such out-of-distribution actions can be inaccurate (Kumar et al.) 2020 Fujimoto & Gu 2021). The BRAC approach addresses this issue by constraining the policy mo to remain close to the behavior policy (Jaques al., 2019 Kumar al,) 2019). As emphasized in Fujimoooo & Guj (2021), simple and effective choice is to add a BC term directly into the actor objective. Incorporating this into the actor-critic framework, the actor loss in Eq. becomes:
BC term
(0) (. Q+ ao) alla0
where is hyperparameter that balances between exploiting high Q-values and staying close to the behavior policy. This objective encourages actions that both achieve high-expected returns and remain within the support of the dataset. The critic loss in Eq. remains unchanged.
Behavior cloning with flow matching. Flow Matching (FM) (ippman et al., 2022) is generative modeling framework that learns continuous-time transformation, flow, which maps a simple base distribution (in this work, a standard Gaussian) to a target data distribution. This transformation is defined through family of intermediate, time-dependent distributions governed by an ordinary differential equation (ODE).
In the context of BC, FM is extended to conditional setting, where the goal is approximate behavior policy тu. underlying the dataset D. This is achieved by learning state and time dependent velocity field v. [0, 1] x S x Rdd Rd that governs the dynamics of a flow, where d is the action dimension. This flow tir (t, s, is the solution of the family of ODEs characterized by:
d
V di
(1,8,, = (1 the (1, (1),,
(,), =
(4)
This flow, conditioned on the state s, maps noise samples 2 ~ NO Ia) into actions distributed according to ne (. s). While sophisticated conditioning strategies can help enhance expressiveness
3

---

Published as a conference paper at ICLR 2026
Algoritiu Guided Policy (GFP)
Integrate (s,
1/ Exp Explicit discpate Eulaz integration with M sceps
0. 1....,
return
while comserged do Sample [;);)))))))
Train critic z Ni0. d= pg Update to minimize -r-0
11 step
// Step Train the distilled onerstep accor xy 2 L.). #o(a.)).
at pa(s.2)
Using the Integrate-Hu tunetion, Line.
Compute (l Update to minimize (Qu(s,a') aalot a
// Stop gradient
ff Step Train value-sware pollcy aul
Compute Sy(s. ant Daatt (at 14
// Stop gradient a
Ii la, (10, and tolttw = Update a to minimize [0, (1((,,)))) (e-d
Output:
Q
(e.g., classifier-free guidance (Ho & Salimans 2022 ). we adopt in this work the simplest variant of conditional flow matching (Holderricth & Erives (025). We further employ the optimal transport variant of FM, which uses linear interpolation with uniformly sampled time points (Lipman et al. 2022 For (s, a) ~ D, c ~ (0, Ia), and ~ 4( 1)). we define the interpolated point O₂ t)c + to, whose target velocity
The velocity field vo is then trained by least-squares regression toward this reference, yielding the conditional flw-matching BC loss (Holderrieth &
Erives 2025):
CHM-BC(W)
[lu s. a.) (a- -))3
(5)
Once the velocity field learned, the corresponding flow the [0, x x Rd A defines an approximation of the behavior policy. At inference, action is obtained by sampling random noise 2 ~ (O00, Ia) and integrating the flow from 0 to 1 using an ODE solver (e.g.. an explicit Euler method). We denote by (s, way, behavior cloning can be naturally expressed as conditional flow matching in the action space.
al 2) the value of the integrated flow time 1. In this
Flow policy for offline RI. Following the idea of Diffusion Q-Learning by Wang al. (2022), straightforward way to train flow policy for offline RL is to replace the BC term in the actor loss (Eq. 3 with the flow-matching BC loss (Eq. (5. However, the iterative sampling procedure makes training expensive, due to recursive backpropagation through time (BPTT) in the actor loss, and also results in slow inference at test time. To mitigate these limitations, Park al. (2025) suggests distilling the iterative fow-matching BC policy into step policy that also maximizes the critic.
3 GUIDED FLOW POLICY
We now detail the GFP algorithm that builds on top of Fujimoto & Gu (2021): GFP integrates a Value-aware Behavior Cloning (VaBC) flow policy with a distilled one-step actor through bidirectional guidance. VaBC leverages the actor and the critic to selectively clone high- value dataset actions, providing more targeted regularization than in standard BRAC approaches. The distilled actor, in turn, maximizes the critic while avoiding BPTT and iterative sampling. GFP composed of three main components: the critic Qs. the actor na. and the VaBC policy Tw. The complete algorithm presented in Algo. and the approach is illustrated in Fig.
: Park et al. 2025
Step 1 Learning the critic Qa. The critic is trained using the Bellman mean-squared loss:
(c =
[(Q - (r (r+20g(s' )
(6)
Bellman target y
where Q2 denotes the target network. v(s, Bellman target in actor-critic methods, which we use by default this work. Yet, since VaBC is designed prioritize cloning the most promising dataset actions for given state, we have also
(s', (''''''' a') corresponds to the standard
4

---

Published as a conference paper at ICLR 2026
Behavior cloning
Value- behavior cloning with varying temperature y
Soft filtering
Moderate filtering
Strong filtering
FQL (Park et al. 2025
1) 2 10-1
1) 10-3
17 S 10
Critic
Actor
Figure 2: Comparison of behavior cloning under different levels of guidance. Left: Prior work (e.g., FQL by Park al. )2025) no filtering. indiscriminately imitating all state-action pairs. Right: In contrast, method introduces cemperature-controlle guidance mechanism, as shown Eq. [T]] resulting in VaBC. At high temperatures, the guidance is weak, so the actor is influenced by many candidate actions. At moderate temperatures, the filtering becomes sharper, giving more woight to higher-value actions while still keeping enough regularization and exploration. At low lemperatures, the filtering very selective, concentrating almost exclusively the highest-value actions according to the critic. However, excessive concentration very low temperatures may allow actor to escape the dataset's action distribution, as shown on the right in green, leading to critic overestimation and out distittribttion issues. Importantly, VaBC cannot escape the dataset's ss action distribution even at very low temperatures, since it trains exclusively on in n--iistrbutton atatenaction pairs. The dashed blue contours in the final yellow drawings (first row) illustrate this constraint.
considered a more conservative variant of the Bellman target:
y VaBC 6') - Qg((', (o(ss,))) + (e(ss,,u((,=))) (u(d))))) ~ No0,
(7)
where (o(s',2) 2) denotes the action from the actor and Pe (s',2) the action from the VaBC pol- icy. Here, as mentioned in Sec. 2 and outlined Line [ of Alg. =)( and (s, are ac- tions sampled from (+(s)) and Tw (1ls,, respectively, with initial input noise z. The Bellman target y (s, r, s') corresponds to an averaging between two estimates of the Q-value: Qz(s', 1o ( 2)) (s,(s',
which can overestimate the real Q-value; and ( which can underestimate the real Q-value. This choice can lead substantial performance improvements in certain situations, as studied in the appendix, Sec, B.1
Step Learning the actor :T.. The actor me is trained by behavior regularized policy gradients, to maximize the Q-function while distilling the distribution of valuable actions learned by т.. This achieved by minimizing the following objective:
(A = E.
[
AQ (s, (o(s,))) + a (1o(s, 2) 2) pw (8, =) >)1[
(8)
The normalization term Σ (=.a) is based on the average absolute Q-value, estimated over mini-batches rather than over the entire dataset (Fujimoto & Gu 2021).
The distillation term encourages the actor to stay close to VaBC. In this way, the actor learns to select actions that maximize return while avoiding out-of-distribution actions, as it is constrained to remain near the support of high-value dataset behaviors.
Step 3 - Learning the flow policy Tw- The VaBC policy n. is optimized via a weighted flow- matching behavior cloning:
(a VaBC (wBC = E( B(e.) (t. (g a) (lw ar) - (a (a-
(9)
where
exp (2(o(o,a)
(s, a)
2 ~~((0,
(10)
exp (2(o(s))) + exp (#Qa(s.n0(3,2))
5

---

Published as a conference paper at ICLR 2026
Intuitively, for given state-action pair (s, a) sampled from the dataset D, 91 (s, a) compares the quality between the dataset action a and a proposal of the actor pa(s, 2) in a soft-max approach. If the dataset action has a higher Q-value, this implies that (, a) > 0.5, placing greater emphasis on cloning it. Conversely, if the dataset action is worse, (8, a) < 0.5, then it reduces its influence. This ensures that VaBC selectively clones high-value dataset behaviors. This makes sense because the actor itself's is constrained to remain close to the dataset's s action distribution. Importantly, VaBC learned jointly with the actor and the critic, not beforehand.
Here, À is the same Q-normalization factor used in the actor loss, ensuring consistent scaling across components. The parameter vj > i a temperature hyperparameter that controls the sharpness of the weighting: small makes 9.y (s, a) more selective, while large smooths the weighting. Importantly, since g (8, a) E (0, 1), VaBC avoids degeneracy during early training when the critic is unreliable, ensuring stable learning.
The key contribution of GFP is add value-awareness in the behavior regularization component of a BRAC framework, effectively combining the two predominant policy extraction methods studied by Park et al. (2024b): weighted-BC and behavior regularized policy gradients, further presented in One could employ an advantage weighted term, similar to AWR (Peng et al., 2019), using
Sec. 5
the actor to compute a baseline:
In
AWR (s, a) = =xx (Qo(ss) Q0(s
(11)
As recommended by Peng et al. (2019), in that case, clipping term should be added for stability. Instead, we propose the guiding function In, which corresponds to soft-max between (o a) and
AAR instead of gn. over Qo(s, po(s, 2)). In appendix, Sec. B.3 Tab. reports results when using gh 65 tasks.
Analysis of the temperature in the guidance function 9r- In Fig. we illustrate how the tem- perature parameter controls value-guided filtering. balancing dataset fidelity with value exploitation. Lower temperatures sharpen the filter, shifting the policy from broadly imitating the dataset to em- phasizing higher-value value actions. Moderate values achieve the best trade-off, prioritizing promising actions while preserving diversity. In contrast, excessively low temperatures over-concentrate the VaBC policy, destabilizing training and degrading the critic by pushing the actor out of distribution.
4 EXPERIMENTS
4.1 MAIN RESULTS: EXTENSIVE OFFLINE RL BENCIIARRKS
We conducted extensive experiments over a suite of robot locomotion and manipula- tion tasks, spanning three major benchmarks: D4RL (Fu et al. 2020), its successor Minari (Younis) et al. 2024). and the recently proposed OGBench (Park et al. (024a). For comparability with exist- ing works, first evaluate on D4RL's AntMaze (6 tasks) and Adroit (12 tasks). We also present sults on Minari, evaluating both GFP and FQL, to facilitate the community's 's migration from D4RL. Minari includes all available Gym-Mujoco datasets (Hopper, HalfCheetah, and Walker, each with 3 tasks), as well as Adroit (12 tasks). Our most extensive evaluation focuses on OGBench, which offers substantially more complex and challenging tasks than D4RL. Following Park et al. (2025), we use the reward-based single-task variants of OGBench. We evaluate across 9 locomotion and 1l manipulation environments, each with 5 tasks, for total of 100 state-based tasks, and we also consider 5 pixel-based tasks. Combined with D4RL and Minari, we benchmark GFP on 144 tasks overall, including evaluation of prior methods, we did about 15 000 runs in total. Our JAx-based implementation of GFP will be released after the rebuttal phase. It can complete one training run in under 30 minutes on modern GPUs
Benchmarks.
Comparison against previous works. We first compare GFP against 10 prior methods, including standard offline RL, flow, and diffusion-based methods on the 50 OGBench tasks reported by Park et al. (2025) (see Sec. 5 and Fig 5 for presentation of prior works, Tab. 10 for the full results). Performance profiles synthesizing these comparisons are presented Fig. Ba following Agarwal et al.
(2021) GFP clearly stands out compared to all these prior works. We also compare against 6
methods on D4RL (Tab. [T] and evaluate both GFP and FQL on Minari (see Tab. [2] to facilitate the community's migration from D4RL..

---

Published as a conference paper at ICLR 2026
200
3.00
RMC
mi with
0.50-
6.50
0.55
0.50
6.55
0000
122
122
OGBench laod Score
Odbench loed Score TV)
OGbench Normaizos Score
(a)
(b)
(c)
Figure 3: OGBench analysis. (a) Performance profiles for 50 tasks comparing GFP against a wide range of prior works, showing the fraction of tasks where each algorithm achieves a score ahove threshold т, using the evaluation reported by Park et al. (2025). (b) Performance profiles on 105 tasks, including more challenging ones, and carefully reevaluated prior methods. (c) Performance profiles restricted to 30 noisy and explore tasks.
Extensive study on 144 tasks. GFP is further evaluated against baselines: (i) FQL (Park et al, 2025), as it comes second only to GFP on the first 50 tasks; (ii) ReBRAC (Trasov et al. (023). as we were able to improve its performance compared to previously reported results substantially. see Sec. 433 and (iii) IQL. (Kostrikov al. 2021) to represent in-sampling approaches. Tab. 2 summa- rizes our results grouped by environment, with detailed per-task results in the appendix (Tabs.
11 and 12). Together with performance profile plots Figs. (3] 3c it confirms that GFP achieves state-of-the-art performance, with particularly substantial gains on noisy and challenging environ- ments. For instance, on the cube-double-nossy and cube-triple-noisy datasets, GFP achieves an average score of 63 and 24, respectively, compared to 38 and 4 for FQL, and 20 and 5 for ReBRAC. Similarly, GFP stands out in some very challenging locomotion tasks, such as humanoidmaze-large large- navigate (18 vs. 7 for FQL and 13 for ReBRAC), and manipulation tasks, such as cube-triple-play (16 vs. 4 for FQL and for ReBRAC).
Table Offline RL results. GFP achieves best or near-best performance on all 144 benchmark tasks. Results
are averaged over 8 seeds for state-based tasks and 4 seeds for pixel-based ones, with values reported from
prior works (Park et al. 2025 Tarasov et al. [2033] Fu et al, (2020) in italic, and values within 95% of the best performance shown bold. GFP actor me represents our primary policy, while GFP VaBC is reported
as a byproduct the training procedure. Full per-task results are provided in the appendix Tabs.
and
T2 and the comparison with 10 methods on the 50 previously evaluated OGBench tasks Tab
Task Calegory
Offinne algorithms
IQL.
ReBRAC
GFP my
GFP VaBC
OGBench cammsre-arg--maviitt----i
(5 troks)
1:
85.0 1.0.4
88.1
93.8 1:5
80.0
OGBench unimare large stitch singletask (5 tasks)
30.4 1:
89.2 1.44
58.1
d: 5.7
68.9 0.4
57.6
3.2
OGBench animare lange explore singletask (5 tasks)
cipllue-singleetsskSS
12.0 1.7
82.7 1.
87.9 5.
91.9 1.9
59.3 111
0GBench mitmare-jinn-nvvigkee mwwi-iie-----niieeeee (5 tusks)
3572
16.3 1.2
27.8 1a
0.8 1.
OGBench humancidmaze medium navigate singletask tasks)
69.2
58
35.9
OGBench humanoidmzze medium -titch singletask taks))
27.3
1.
OGBench humanniddmzz--in
mavigate--nniiigiieee singless (5 taks)
2.1
12.9 1:42
6.6 1.
17.8 1.
2.4
OGiRench resencer--renni-ee monnnnnnnngg gate--ingletak (5 tasks)
8-
55.9 1:
60 +
57.9 +
10.3 o.
OGBench antsoceer arena strich singletack tasks)
2.
22.0 1:
28.6 1 2.1
30.5 1.2
1.4 0.3
OGBench cube--ingle play-singletask tasks)
91
96 1
95.8 0.4
59.7
OGBench cube--inglo-mnos---iin -innilll--
taoks
a3.
OGBench cube double play singletack tasks)
7
12.6
OGBench cube-double- do-ddl----nniiy noicy singletask (5 talks)
4.5 0.
19.6 1
38.2 1 5.1
65.1 1:1
9.4 1.
OGBench peeee--iple--py--nneeee play-sigkktsk (5 tasks)
0.1 1.001
1.2
3.9
1
15.9
7.6 1.5
OGBench cube irppl minss--inglltask (5 tasks)
singletask
4.8
2.2
3.5
1 :.4
24.5
2:
8.6 1
OGBench puzzle3 3x
3x21--a3x-pa-----p--appaa play singltask (5 tasks)
9
21
30 11
25.1
19.2
OGBench puzzle--sc
17.1
17
26.1
OGBench puzzle-k 4x4 noi--niis--uua singletsss tasks)
0.1 0.0
1.1
OGBench scene play singletsk i5 tusks)
1
41.6 134
56 1
53.5 29
57.6 1.7
OGBench scene-mee moan--nois--aaa singletssk (5 lusks)
16.0 L.
59.9
12d
59.3
1.
57.5 19
58.5 1
OGäaench visual manipulation (5 tasks)
6O 2
60 2:2
62.8 2
D-RL
usks)
76.8
84 1.3
70.2
Adroit (asks)
46
Minuri Adroit (12 tusks)
10.6 9.4
48.5 1:13
46.1
Minari hopper (3 tasks)
79.6 = 10.3
91.7
4:43
91.5 1
Minari halfchoetah (3 tasks)
07.8 1. 2.5
109.1
1 2.
103.1
1.
Minuri walker2d bocks)
121.7
124.5 1
122.2
Average OGBench (105 taskes)
20.4
53.2
3
Average D4RL. tasks)
54.0
64.8
63.0
56.5
Average Minari (21 tasks)
63.9
74.1
71.6
sverig without pixel-besed tasks.

---

Published as a conference paper at ICLR 2026
GFP-Acto"S
GFP-VaBC
ReBRAC
0.0.01
0w0.10
0.-25
6w0.00
5.5.50
6.75.7
Tomee 900000000000
(a) Puzzle
(b) Puzzle F(g. >
(c) Cube
(d) Cube P(gn
Figure 4: Temperature analysis on challenging OGBench Puzzle (left) and Cube (right) tasks with sub- optimal data. Plots (a) and (c): performance scores across temperature values 7) for our GFP method (Actor wo and VaBC Tw) compared to baselines (FQL, ReBRAC) on noisll----ooo-yy---e task2. Plots (b) and (d): probability that the guidance term On above different thresholds 5 a function of temperature, illustrating how temperature controls the sharpness of value-guided filtering.
and cube-double-ug noisy-
Value-aware behavior cloning. VaBC serves the regularization component of GFP, encouraging the actor maximizing the value to stay close to some actions from the dataset. While the actor GFP primary policy, through our bidirectional training procedure, we obtain the VaBC policy Tw byproduct that can also be exploited and evaluated. As shown in Tab. 2 VaBC achieves good performance across benchmarks while being fundamentally in-sample, justifying its role as value-aware regularizer for the main actor.
4.2 ANALYSIS OF THE TEMPERATURE PARAMETER y)
guidance mechanism. In Fig. GFP evaluated with varying temperature on two challenging noisy tasks from OGBench, characterized as highly suboptimal data according to Park et al. (2024a). The presence of low-quality demonstra- tions makes selective action emphasis decisive for effective learning. Figs. 44a and 4c demonstrate the advantages of moderate temperatures. Very low temperatures cause training instability due to over-concentration narrow action sets, while very high temperatures fail to provide sufficient filtering of suboptimal actions. As the temperature decreases, VaBC Tw performance improves, confirming that the value-guided filtering mechanism successfully emphasizes higher-value actions, until the temperature is too low.
We investigate the impact of the cemperature-controlll
Figs. 4h and 4d illustrate the filtering behavior by showing the probability that the guidance term In exceeds various thresholds 8 (ranging from 0.01 to 0.75). At extremely low temperatures (1) S 10-5), the guidance term exhibits near-binary behavior: any slight differences between Q(s, a), with from the dataset, and Q(s, a*)). with Te (- (s), in state result in the guidance term approaching either or according to Eq. 10 in this case Pg > 0.75) s P(g,, 0.01). As the gradual transitions in the guidance values. This leads to a broader distribution of filtering probabilities across different thresholds, demonstrating how higher temperatures preserve more of the original dataset diversity. In contrast, lower temperatures create sharper distinctions between high-value and low-value actions. In the appendix, in Sec. B.4 we extend this study to two additional tasks.
temperature increases, the filtering becomes softer, creating
Sensitivity analysis to a and In Sec. B.5 we evaluate how sensitive GFP is to variations of a, the BC coefficient, and y), on 12 tasks, with comparisons FQL. It reveals how they relate and demonstrates that GFP is only mostly sensitive to the proper choice of a, as for any BRAC method.
4.3 RE-EVALUATION OF PRIOR WORKS ON OGBENCH
To obtain a fair comparison of GFP against prior methods, we reevaluate existing baselines on OGBench. During the development of our method, we observed that several task-specific hyper- parameters (e.g., discount factor 7, minibatch size B, and critic aggregation scheme for doubled Q-learning (Fujimoto et al. 2018)) have significant impact performance. Tab. 3 reports sults under these revised settings. showing that careful tuning can substantially improve the reported scores of both ReBRAC and FQL. Since the optimal values for these hyperparameters were gener- ally consistent across methods, we treated them as task-specific and, by default, applied the same settings GFP, FQL., and ReBRAC (see Tabs. 4 and in the appendix for detailed values).
8

---

Published as a conference paper at ICLR 2026
Table 3: Impact of task-specific hyperparameters on OGBench performance.
Bigger discount factor y
Task Environment
Proviously reported in Park of al. 2025
Our evaluations
ReBRAC 81 +
FQL 79 +
RcBRAC
antmaze-large- saazzz-large-naveee navigga (5 tasks)
95.3 +
0.4
88.1 + 6.5 2.
humansidmaze large navigate tasks)
2
12.9 1
Bigger minibatch size
Previously reported in Park al. 2025
Our evaluations
ReBRAC
FQL
ReBRAC
FQL 16.3 1.3++0
antmuze-giant-navigaee (5 tasks)
26 +
9 9
33.2 +
Same critic aggregation for ReBRAC as used in FQL
Previously reported in Park et al. 2025
Our evaluations
ReBRACC
ReBRAC 59.2 1 12.3
humancidmaze medium- navigute (5 tasks)
22 1
intsoccer-sscn--nna tasks)
55.9 +
cube-double-plan (5 tasks)
18 +
12.6 2..6+10 L8
scene- playpppayyy (5 tasks)
4
41.6 1--++-0
puzzle-4xtppee (5 tasks)
14 +
17.1 + L.8
5 RELATED WORK
Offiine RL. Early methods addressed distributional shift by working on both how the critic is learned and how the policy is extracted. On the critic side, Conservative Q-Learning (CQL) (Kumar et al,, 2020) penalizes out-of-distribution value estimates and Implicit Q-Learning (IQL) (Kostrikov al. 2021 avoids querying such values via expectile regression, see Fig 5a For policy extraction, weighted behavior cloning treats offline RL supervised learning, typically using the advantage value as the weight, as in AWR or AWAC (Peng et al., 2019; Nair et al., 2020). see Fig [dd] The policy extraction method found lead the best performance and scalability in standard offline- RL is behavior-regularizer policy gradient (Fujimoto & Gu 2021 Park et al., (024b)) abbreviated DDPG+BC and also called reparametized policy gradients, which consists in directly maximizing the value function, while regularizing with a behavior cloning term. The BRAC family studied in this paper uses behavior-regulariv
policy gradient and trains the critic directly using the actor, see 5ig ReBRAC (Tarasov et al. 2023) demonstrated state-of ofae-oo-the--e off-the-rt art performance from this simple idea, through careful hyperparameter tuning and architectural choices.
Control with diffusion and flow models. Expressive generative models enable multi-modal action and trajectory distributions modeling and have been integrated into control along several axes. They have been used for trajectory and motion planning. trajectory optimization, hierarchical control
(Janner et al. 2022 Ajay et al. 2022 Zheng et al., [2023] Liang et al., 2023 Li ct al. 2023 Suh Chen ct al. 2023] Venkatraman et al. [202]] Lee et al. 2023; Ma et al. [202]] hen et al.] |2024 Pan al.
2024 Le Hellard et al. 2025), as well as for world modeling (Ding et al.) 2024b] Alonso et al., 2024
Lee al. 2024). In offline RL, numerous ideas emerged to integrate diffusion and flow models in
the various policy extraction methods.
Weighted behavior cloning. A straightforward approach is add weight in front of the cloning
loss of diffusion and flow models, examples include FAWAC (Park et al. 2025), EDP (Kang et al.,
2023 and QVPO (Ding al., 2024a). Related energy-guided methods explicitly target to sample
from Ta (a]
8) Oc p(a 5) ( a)), where is the behaviour policy. This includes CEP (Lu
al. 2023 and QIPO (Zhang al. 2025 Alles et al. 2025), ) see Fig 5d
Behavior-regularizze policy gradient. Combining the most effective policy extraction method
with expressive models is motivating research direction: Diffusion-QU (Wang et al. 2022) (Fig5e)) DiffCPS (He et al.] 2023), trust-region formulations for diffusion policies (DTQL) (Chen et al.) 2024b), entropy Qeensemble regularization (Zhang et al., 22024), Consistency-AC with consistency models (Ding & Jin 2023). and diffusion policy gradients (Li et al. (004)) Yang et al. 2023). In that case, the main challenge is the drawbacks of backpropagation through time (BPTT) when trying to directly optimize the value of the outputs of iterative process. To avoid BPTT, FQL (Park) jet al. 2025) (Fig 5e) trains a flow matching with a standard BC loss, and then leverages it in the BC regularization term of a one-step BRAC actor. Beyond these families, other policy extractors include rejection sampling from a fixed BC generative policy: IDQL Mllansen-Estrucee et al. 2023)
(Fig 5b). DICE (Mao et al. 2024), and SfBC (Chen et al. 2022

---

Published as a conference paper at ICLR 2026
Critie Expocill pepression
Critic Expectil regressium
Critic learning
Critic TD learning
Critic TD learning
Critic
learning
() Actor DDPG+BC
Regularizer BC
Regularizer
Weighted BC
Actor Any policy extraction
Actor samipling
Actor Weighted BC
Actor DDPG-BC
Actor DDPG+BC
TD3-BC ReBRAC
AWAC PAWAC QIPO
(a) IQL
(b)+IDQL
(c) .DQL
(e)+FQL
(i) Our GFP
Figure 5: Overview of some offline-RL frameworks. The symbol indicates the of diffusion or flow model, and specifically in which component. Each box, provides the name of the component (c.g. critic) and the working principle that used train (c.g. TD learning). The arrows indicate how the components depend on cach other, while the dashed arrow is optional (see Eq.
Positioning GFP within prior work. Our method, GFP (illustrated Fig I and Fig 5f). extends the BRAC family of offline RL algorithms and is closely related FQL (Park et al. (2025), sharing the same goal of avoiding backpropagation through time. However, the distinctive contribution of GFP lies in integrating the weighted BC approach for the regularization component of the BRAC actor. Rather than using weighted BC to train the final actor directly (as in Fig 5d). GFP employs it to inject value-awarenes: into the regularizer VaBC. Compared to other methods using a weighted- BC policy, VaBC is not GFP's actor, its weights are computed using the one-step actor trained by DDPG+BC, with the latter using VaBC as a regularizer. This bidirectional guidance between the two policies fundamentally distinguishes GFP from other methods that weighted BC as their main actor, computing the weights based on the weighted BC policy itself, see Fig 5d
Removing the value-aware guidance from the GFP flow policy yields FQL (Fig [e), and removing the one-step actor implies using the flow policy to train the critic, yielding FAWAC (Fig 5d). Alter- natively, one could use rejection sampling instead of weighted-BC regularize the actor with the best actions found by BC policy, yet at the cost of sampling multiple actions from the flow policy for each policy update step.
6 DISCUSSION AND CONCLUSION
In this work, we revisited behavior regularization for offline RL. BRAC approaches have been shown to be highly effective for policy extraction (Tarasov et al., 2023; Park et al. 2024b). However, con- ventional BRAC approaches constrain the Tearned policy to remain close to the raw dataset dis- tribution. While this reduces instability, the regularization term itself typically does not distinguish between the high and low-value actions. This limitation is problematic in suboptimal datasets, where regularizing with all transitions indiseriminately
hinder performance.
To address this, we introduced Guided Flow Policy (GFP). GFP couples a multi-step flow-matching policy trained with value-aware behavior cloning and distilled one-step actor through bidirec- tional guidance mechanism. GFP leverages the expressiveness of flow policies while adding value awareness directly in the flow part, without the drawbacks of backpropagation through time.
Our analysis provides several insights. First, although standard behavior-regularized actor-critic methods, such as ReBRAC, are competitive with good hyperparameter tuning, their dependence on value-agnostic behavior regularization limits performance on suboptimal datasets. In particular, in prior BRAC approaches, adding low-quality transitions in the dataset will degrade performance, while the aalue-awareness of VaBC filters them out. Second, while generative models such as flow or diffusion policies can represent dataset distributions more flexibly, they inherit these same limita- tions if trained to match the raw dataset without explicit guidance. GFP leverages the benefits of two distinct policy extraction methods: the effectiveness and scalability of behavior-regullarize policy gradient, and the stability of weighted BC. This combination exploits well the expressivity of flow models, without backpropagating through time. This synergy enables GFP to consistently achieve state-of-the-an results, demonstrated by our extensive and rigorous study 144 offline RL tasks.
Nonetheless, GFP depends on the availability of a sufficiently accurate critic. In datasets lacking high-value actions when the critic cannot reliably evaluate them, improvements are limited. Fu- ture research directions could explore ways to reduce reliance on the critic or extend GFP to settings with weaker sparse reward signals.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work has received support from the French government, managed by the National Research Agency, under the France 2030 program with the references Organic Roboties Program (PEPR O2R) and "PR[AIJRIE-PSA"" AANR-3-IACCL-000S This research was funded, in part, by I Agence Na- tionale de la Recherche (ANR), projects NIMBLE project (ANR-22 ARR-22-CES -2-CE33-0008), RODEO (ANR- 24-CE23-58861 PEPR O2R AS2 (ANR-22-EXO0 11--2-EXXO--000 (000),, and PEPR O2R PI3 ASSISTMOV The European Union also supported this work through the ARTIFACT project (GA no. 101165695) and the AGIMUS project (GA no. 101070165). The Paris île-de- France Région also supported this work in the frame of the DIM A14IDF. The authors gratefully acknowledge the support and resources provided by the CLEPS infrastructure Inria Paris. This work was performed using HPC resources from the GENCI-IDRIS Jean-Za Zay cluster (Grant 2024- AD010616763) Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the funding agencies.
(ANR-22-EXOD-0004)

---

Published as a conference paper at ICLR 2026
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning the edge the statistical precipice. Advances in Neural Informa- fion Processing Systems, 2021.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211. 156577, 2022.
Marvin Alles, Nutan Chen, Patrick van der Smagt, and Botond Cseke. Flowq: Energy-guided flow policies for offline reinforcement learning. arXiv preprint arXiv:2505.14139, 2025.
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58791, 2024.
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.000 1.2644, 2024a.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:50098- 50125, 2024b.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273668. 2023.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945-53968. 2024a.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309. 16984, 2023.
Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model: Fu- ture modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint 03550 2024b.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. Pmlr, 2018.
Philippe
Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit qelearning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
12

---

Published as a conference paper at ICLR 2026
Longxiang He, Li Shen, Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffeps: Diffusion model based constrained policy search for offlie reinforcement learning. arXiv preprint S3Xt::23100.5333,, 2023.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Peter Holderrieth and Ezra Erives. An introduction flow matching and diffusion models. arXiv preprint wrXiv22506.02070 2025.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999/ 2022.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:
2019.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:67 195- 67212, 2023.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint 10.06221 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179-1191, 2020.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International conference machine learning, pp. 3652-3661, PMLR, 2019.
Théotime Le Hellard, Franki Nguimatsia Tiofack, Quentin Le Lidec, and Justin Carpentier. Sobolev diffusion policy. 2025.
Jaewoo Lee, Sujin Yun, Tacyoung Yun, and Jinkyoo Park. Gta: Generative trajectory augmentation with guidance for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:56766-56801 2024.
Kyowoon Lee, Seongun Kim, and Jaesik Choi. Refining diffusion planner for reliable behavior synthesis by automatic detection of infeasible plans. Advances in Neural Information Processing Systems, 36:24223-24222 2023.
Steven Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neu- ral Information P Processing Systems, 37:38456-38479, 2024.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pp. 20035 PMLR, 2023.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models adaptive self-evolving planners. arXiv preprint 202.0.23....1177 20233
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1509.0000000000 2015.
13

---

Published as a conference paper at ICLR 2026
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2110.02777, 2022.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu, Contrastive energy prediction for exact energy energy-guides guided diffusion sampling in offiin reinforcement learning. In Inter- national Conference on Machine Learning, pp. 22825 -2255. PMLR, 2023.
Xiao Ma, Sumit Patidar, lain Haughton, and Stephen James. Hicrarchical diffusion policy for kinematics-aware multi-task robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18081-1 18090, 2024.
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dics In- sample diffusion guidance for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:98806-98834, 2024.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09359) 2020.
Chaoyi Pan, Zeji Yi, Guanya Shi, and Guannan Qu Model-based diffusion for trajectory optimiza- tion. Advances in Neural Information Processing Systems, 77757914-57943 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking
offline goal-conditioned rl. arXiv preprint arXiv:2410.20092, 2024a.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? Advances Neural Information Processing Systems, 37:79029-79056, 2024b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arx.v:2502.00538, 2025.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weiighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv:1910.001/7,, 2019. 2019.
M. Riedmiller S. Lange, T. Gabel. Batch reinforcement learning. in: M. wiering, (eds) reinforcement learning. Adaptation, Learning, and Optimization. 12(3):729, 2012.
otterlo
Yang Song, Jascha Sohl-Dickstein, Diederík Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13356 2020.
HJ Terry Suh, Glen Chou, Hongkai Dai, Lujie Yang, Abhishek Gupta, and Russ Tedrake. Fight- ing uncertainty with gradients: Offline reinforcement learning via diffusion score matching. In Conference on Robot Learning, pp. 2878-2901 2904. PMLR, 2023.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36:1 1592-1122111 11-211 2023.
Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, John Dolan, Jeff Schneider, and Glen Berseth. Reasoning with latent diffusion in offline reinforcement learning. arXiv preprint arXiv:2309.065999 2023.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06199 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: /. 1361, 2019.
Long Yang, Zhixiong Huang, Fenghao Lei, Yueun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for rein- forcement learning. arXiv preprint arXiv:2305. 122, 2023.
14

---

Published as a conference paper at ICLR 2026
Omar G. Younis, Rodrigo Perez- Vicente, John U. Balis, Will Dudley, Alex Davey, and Jordan K Terry. Minari, September 2024. ARR/Attps://cc.org////
Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas Schön, and Per Mattsson. Entropy-regularieed diffusion policy with q-ensembles for offline reinforcement learning. Advances in Neural Infor-
mation Processing Systems, 37:98871-9889 2024.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. arXiv preprint WXXiv:5533.49975 2025.
Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, and Ricky TQ Chen. Guided flows for generative modeling and decision making. arXiv preprint arXiv:2311. 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Implementation details
16
Additional experiments
17
I
Modified Bellman target
17
B.2
Advantage weight similar to AWR
18
B.3
Modified guiding function to integrate a
18
B.4
Temperature analysis
19
B.5
Sensitivity analysis to and
19
Complete results over 144 tasks
19
Note on LLM usage: In this work, we only used LLMs for grammar and spelling corrections.
A IMPLEMENTATION DETAILS
Network architectures. All critic, actor, and flow neural networks use [512, 512, 512] multi- layer perceptrons with GeLU activations. Layer normalization applied to the critic network.
Flow matching. The number of Euler steps, M, used in Algo. [ Line is fixed to 10 for both GFP and FQL, except on the humanoidmaze- warddmaaz---a---naaa navigaa environment, where we set M 30. For GFP, we employ sinusoidal position embedding of the flow step t, with an embedding size of 64.
Doubled Q-learning, Following standard practice, two separate critic networks are trained and then aggregated to compute action values, either by taking the mean or minimum (Fujimoto et al. 2018)_ As detailed in Sec. 4.3), we find that the aggregation function has significant impact on performance for specific tasks. Specifically, by reevaluating ReBRAC on OGBench using the same aggregation function as GFP and FQL, we achieved substantial performance improvements.
Minibatch size. We use minibatch size of B 256 across most experiments, except the most challenging tasks, where we evaluate each method with both Lumanoidmaze-laree large-avvigtee environment is the only task where methods benefit from different batch sizes: GFP performs best with Note that this task, using y 0.999 substantially improves the performance of ReBRAC and FQL compared to previously reported results. For Minari Gym-Mujoco we use = 1024 following the recommendation in Tarasov et (023) for D4RL Gym-Mujoco.
256 and = 1024. The
1024, while other methods work better with = 256.
Training and evaluation. To ensure a fair comparison with FQL, we use identical training dura- tions: 1 M gradient steps OGBench state-based environments, and 500 K steps D4RI. and OGBench pixel-based tasks. For Minari, we adopt M steps following standard practices. Eval- uation differs according to the benchmark: D4RL and Minari scores are computed the end of training, while OGBench scores are averaged over the final three checkpoints (800K, 900K, and IM steps for state-based, 300K, 400K and 500K steps for pixel-based following their official evaluation protocol. All results are reported across 8 random seeds that were not used during the hyperparame- ter tuning process, and each evaluation is done over 100 episodes. Except for OGBench pixel-based tasks, where scores are computed using 50 episodes and 4 seeds, due to high computational cost, following Park et al. 20255)
Minari MuJoCo reference scores. Since reference scores for the MuJoCo locomotion tasks are not yet available in Minari (Younis et al. 2024), we use the reference scores reported in its predecessor D4RL (Fu et al. 2020).
Parameter search methodology. Tab, summarizes the set hyperparameter values shared across environments and methods. For task and method specific hyperparameters, systematic approach. First, we conduct a logarithmic sweep over the BC coefficient a, which the main parameter for all methods. For ReBRAC, we sweep over the actor coefficient on while keeping
search follows
16

---

Published as a conference paper at ICLR 2026
the critic coefficient 02 fixed 0.01. Then, sweep a2 after selecting the optimal 01. For GFP, we fix 10 first and then sweep over 7) once a is chosen. Sec. stSidies how sensitive GFP is to ny and a, and justifies first searching before adjusting 7). Hyperparameter search uses four seeds, separate from the eight seeds used for final evaluation. On OGBench, hyperparameters shared across the five tasks within each environment.
Table Summary of shared hyperparameters used across methods and benchmark evalu- ations. Environment-specifie variations are indicated where applicable.
Hyperparameter
Value
Learning
0.0003
Gradient steps
1,000,000 (OGBench, Minari), 500,000 (D4RL) 256 (default), 1024 (Minuri Gym-Mujaco), OGBench Tab (IARI., Minari), OGBench 'Tah. 10 (default). 30 (humancidmazz-lagg) mean (default), min (D4Rl-antmaze, OGBench-anntnnn
Mimibatch size
Discount
Euler integration steps
Critic aggregation function
Critic target network smoothing coefficient
Bellman larget
- r 7Qis a²) (default), y NBC (cube, (immunoddmaaz-mmmiiim
Table 5: Discount factor and minibatch size for OG Bench environments. The asterisk in some discount factors indicates cases where we modified the discount factor used in prior work, which led to significant performance improvements for the corresponding methods.
Task Category
Discount factor
Minibatch size
antmaze- Bamnuze-Jarge-nnn burge-naviuttesinnleee (5 tusks)
0.995*
256
antmaze-lun aatmaze-large-stiee hurge-tiich-ssnnletaae (5 tasks)
0.995
(except for FQL, 0.99)
256
antmaze-large antmaze-large-expy www---leeplorr-iinnnneeee
singletask tasks)
0.995
1024
antmaze-giant
(5 tasks)
0.995
1024
humanoidmazz medium navigate sittssingletssk (5 tasks)
0.995
256
humanoidmaze- caeuuze-eediu----ee medium-sticch--inn sticch--singletask (5 tusks)
0,999
256
humanoidmaze largee navigate viggtt--inngleask (5 tasks)
0.999*
256 (except for GHP, 1024)
antsoccer-arent arenn-naviaate
navigate-singletas singletask tasks)
navigate
0.99
256
antsoccer-arena- wew.a--ticc---snnleeeeee.
tasks)
0.99
256
cube-single singge-pla-ssnnleeeee (5 tusks)
0.99
256
cube single-noiss singlesi tasks)
0.99
256
cube-double-play slabbe--lay--innltassk
tasks)
0.99
256
cube-double-noff wwsul---oos-ssinletuss (5 tasks)
0.99
256
cube cube-triple triple singleinglttssk (5 tasks)
0.99
1024
sube-tiple-noisy-sii -iilloooss---nlllassk (5 tasks)
0.99
1024
puzzle-3 x ---play-singletask (5 tusks)
0.99
256
puzzle-4xx 4x4 444-- -lay--innlnt (5 tasks)
0.99
256
puzzle-4xx 100e-444-ooiiss x44-noisy-singg ---noisy--ingllttak (5 tasks)
0.99
256
sene-play-singlettdd
0.99
256
scene-noisy -ai-no-ooissllttae
0.99
256
visual tusks
0.99
256
B ADDITIONAL EXPERIMENTS
B.1 MODIFIED BELLMAN TARGET
As described in Sec. propose variant of the Bellman target, y VaBC (Eq. 7 that leverages the VaBC policy. Tab. 6 presents experimental results showing average scores over 8 seeds for the selected hyperparameters, with "~^" indicating configurations that were tested but not ultimately chosen. These experiments show that this modified target provides improvements the cube and humanoidmaze-mn medium environments, and constant slightly degraded results other environ- ments, hence y YaBC is used only for the former ones, as reported in Tab.
Implementation Note on Eq. 7 After finishing the experiments, we realized that in imple- Bellman target, we were computing
(x,())))) instead of La z)). our public release of the code, we kept this behavior by default to match the code used in our experiments, but we added an option to use Q5(s', (s. 2)). for future experi- ments.
mentation of the modified YBBC
Qá(s', (s'
17

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of the modified Bellman target for GFP.
Task Category
Standard target y
Modified y
VBC Eq
cube-double-play tasks)
47.2 + 1.6
cube-double-nois tasks)
46
63.1 4:
humanoidmaze medium navigate (5 tasks)
~ 64
72.01 + 2.8
humanoidmaze medium-stitch (5 tasks)
50
06.2 + 6.7
puzele-4x4-play tasks)
26.1
scene-play (5 tusks)
53.5 9.9
50
B.2 ADVANTAGE WEIGHT SIMILAR TO AWR
As explained at the end of Sec. 3 our In a soft-max weight comparing to a an action sampled from the actor, but one could alternatively use an advantage weighted term similar to AWR (Peng al. 2019). but using the actor compute a baseline:
AWR (s, a) In
exp (2 (Qo(s,))
(12)
Tab. reports results when using In instead of 9t, over 65 tasks, with and 1) tuned accordingly, Tab. The modified Bellman target y VaBC has also been tested for this variant, as in Sec. B.1 was found to be beneficial for the cube, humanoidmaze, and puzzle environments.
and
9n AWR achieves slightly better performance on standard tasks, but our main 9, scales better on more challenging ones, e.g., humanoid environments and cube-triple ones. Important note: GFP-AWR only changes the weighting function, keeping the general structure presented Fig and Fig 5i In particular, it does not mean first using AWR to Iearn weighted-BC policy (like FAWAC, Fig 5d) and then using it to regularize the actor. GFP-AWR preserves the bidirectional guidance, core contribution of GFP (see the arrows in Fig 5t.. As alternative, one could also learn value-function network
V(s) = (g (Q(s, a)) and use instead of the stochastic baseline Q(s, (0 z)) in the weighting functions.
Table 7: Comparison with an AWR guiding function
Task Category
GFP-9, (default)
GFP-AWR
antmaze saauzze--arge-ttee large-stitch (5 tasks)
68.9 0.8
69.8 +
antmaze- ceaiizeegginttnnee givinae (5 tasks)
27.9 1 185
29.1 8.4
humanoidmazz--melee naviggte (5 tasks)
72.0 2
60.9 3.4
humanoidmaze-meddiee wwt--m--edddddeeeeeeee (5 tasks)
66.2 5
64.8 8.5
hunanvidmaze- hargeree--agee-aviigatl tasks)
17.0
antsoccer-and aren--ticch tasks)
30.5
29.5 2.6
cube-double-ppi (5 tasks)
47.2 +.6
50.4 3.5
cube-double-noit (5 tasks)
63.1 1.3
55.7 3.4
cube-triple-ppay(5 tasks)
15.9 2
9.9 2.9
cuhe-triple-noisy (5 tasks)
24.5 +
11.2
puzzle-3x3-plas tasks)
23. 22
28.3 1
puzzle-4x4-plaz tasks)
18.8
16.5
puzzle-4x4-noif (5 tasks)
26.1 +
27.1 +
Average (65 tasks)
38.6
35.9
B.3 MODIFIED GUIDING FUNCTION TO INTEGRATE at
Our guiding function Iy, defined in Eq. TO compares a dataset action a= action sampled from the actor. Since we also sample actions using the VaBC flow policy (Algo. [ Line (0).. we can alternatively use Tw for a more conservative guiding function, leading modified guided function:
exp (AO)(s.))
In (s,
min
(13)
exp (A)o(s,a) + exp min (Qo(s,ate), (o((s,a=w))
exp
This modified guiding function is more conservative because only filters out action a when both policies produce more valuable alternatives. Initially, we employed ghin (s, a) for evaluation
18

---

Published as a conference paper at ICLR 2026
OGBench antmaze, humanoidmaze, and antsoccer environments. However, found significant performance difference, so all remaining tasks use the standard Ot defined in Eq. I0
B.4 TEMPERATURE ANALYSIS
To complete the analysis presented in Sec. 4.2 conducted experiments on two additional tasks
beyond those shown in Fig. 4 We tested humanoidmaze humanoidmazz-mmddium meaizee-edium--ttie stitch as a locomotion task and seee-ttrpl-ppaay as a non-noisy manipulation task. The results of this extended temperature analysis are reported in Fig. illustrating how temperature controls the sharpness of value-guided filtering.
GFP Actor
GFP VaBC
FQL
RaBRAC
0.0.01
5
5-0225
5-0.0
5 0.50
0.75.
Temsersture
Tonserture
Tomoz
Tomperstud
(a) Humanoidmaze score
(b) Humanoidnaae Fg. > 8)
(c) Cube triple score
(d) Cube triple F(g.,
additional very challenging tasks. Plots and (c), performance scores across temperature values 7 for our method (Actor my and VaBC m.) compared to baselines (FQI,, ReBRAC) on humanoidmaze- soddium-stitch-are and cube tiabe--riie-ppa---eeee pasyl Plots (b) and (d), probability that the guidance term is above different thresholds d as function of temperature, illustrating how temper- ature controls the sharpness of value-guided filtering.
Figure 6: Temperature analysis
B.5 SENSITIVITY ANALYSIS TO 77 AND
To study how sensitive GFP is to the temperature and the BC coefficient we evaluated 8 varia- tions around task tpsc-fpe hyperparameters (a", .7") on 12 tasks, with comparison FQL. sensitivity to a. These experiments reveal that is the most important hyperparameter of GFP, as it is for most offline RL methods (Taasovv et al. 2023 Park et al. 2024b), and that GFP is less sensitive to precise n tuning. It justifies
methodology for hyperparameter search, by first fixing 17 10 and then sweeping over 7 only once a chosen. From the results averaged on 12 tasks, it can be observed that and are partially correlated: when increasing (i.e. stronger regularization), it is better decrease (stronger filtering).
C COMPLETE RESULTS OVER 144 TASKS
This section presents the comprehensive experimental results across all 144 tasks from OGBench (Tabs. 8 and 9). D4RL (Tab. IT]. and Minari (Tab. [2] benchmarks. Tab. [O covers comparison against 10 prior works over 50 tasks. All results are averaged over 8 seeds, except for pixel-based tasks. The evaluation encompasses approximately 15,000 individual training runs, providing de- tailed performance comparisons across diverse offline RL scenarios, including navigation, manipu- lation, and locomotion tasks. The hyperparameters used are stated Tabs.
13
19

---

Published as a conference paper at ICLR 2026
6'/10
96
n1/10-
14
26
210
6'100
78
90
230
83
6//10
40
63
49
/f
97
95
77
53
26
88
89
86
67
47
x0
97
96
81
10xmn
52
12
10xm1
77
81
65
10xn'
39
8'13
d'
3xa
(a) GFP Antmaze large
(c) GFP Cube double
(c) Soone
(g) GFP Over 12 tasks
83
93
20
31
82
83
49
50
22
e'13 (b) FQL Antmaae large
3x0"
3xd
3xa"
(d) FQL Cube double
(I) FQL Scene
(h) FQL Over 12 tusks
Figure 7: Sensitivity analysis
BC coefficient a, to the temperature 77, log-scale. Evaluates the sensitivity by testing variations around task-specific hyperparameters (a",,)) reported Tab. 13 using seeds. Plots (a), (c) and (e), report GFP's sensitivity to a and 1/",
respectively. These three tasks laraaaze-arge-naviatee
cube-
chosen following FQL (Park al. 2025) ablation study on (FQL page 15). Plots (b), (d) and (f), report the sensitivity FQL with respect to a freevaluated as FQL's authors did not share exact numbers). Plots (g) and (h) aggregate the analysis over 12 tasks (cube single and double, play and noisy; scene play and noisy; antmaze navigate large and giant; humanoidmaze medium navigate and stitch, where for each environment experiments reveal that is the most important hyperparameter of GFP, it is for most offline RL methods Park al. 2024b), and GFP less sensitive to precise r tuning.
double-play-task. and ssene-play-taske
the default task). These
TTrasov et 2023
20

---

Published as a conference paper at ICLR 2026
Table 8: Offline RL full results on OGBench, part 1/2. Models were trained with random seeds and evaluated over 100 episodes, following the setup of prior work (Park et al. 2024a 2025). Scores are averaged across seeds; values within 95% of the best performance are shown bold, while italics indicate scores reported from prior work (Park et al.) 2025). GFP actor m is our primary policy, while GFP VaBC Th is reported as byproduct of training. The + symbol denotes standard deviation over seeds. See Tab 9 for the other tasks.
Offline RI. algorithms
Task
IQr.
ReBRAC
GFP actor %
VaBC R.
large navigate singletask task1-v0
48
97.7
92.6
95.4
92.0 1.33
untmaze-large-nn wawaaare----a--iiiaatte navigate-singletus--tee
48
92.2 + 1.6
80.0
10.4
92.2
3.0
88.5
98.3
93.5
untmaze-large Malnaaz-llrge-navae harge-navigate-siin sigle--innttttttuskkkee.
51
94.4 = 0.9
82.3 15.5
90.6 2.6
85.6 8.7
antmaae large navigate singletast task5-v0
96.5
animaze-lan Malnnae--lrgg-stiie large-stitch shithh-sigglttan singletask lusk-task1--D
28.2
88.4
71.8 26.4
90.3 2.1
85.0
antmaae large stitch singletask task2.
5.5 5.54-1
85.2 5.8
25.4 25.5
82.9
69.5 7.2
untmaze-large large-stitch singlsttssl--titeeeeeeee -etassk-tssk3-v0
$8.4
antmaae large stitch singletask task4
8.8 1:29
79.4 20.9
23.7
0.2 0.0.
1.2
untmaze-lane leeee-lrr---sttteeee singletask-tuske
26.2
95.1
large----eeeepeeeee singletask latkakk-takkkl-v
0.4
91.8
91.0
92.8 4.1
92.3
untmaze-largo-eee mammaze-lrgo-oxploor www.al-pploo--siiieeeeeee wogglooo-ingglenss-nskk222222
0.0
$6.4
90.8 3.4
88.5 3.6
84. 3.1
518
unimaze-large larnaee-argg-exppee sigglersssnnllttsk--aaa
9.2
54.4 80.5
89.0 2.9
87.1 2:3
87.0 2.8
antmaze large explore singletask
23.7
32.0
giant navigate singletask taskl
0.1
17.5 d: 5.8
0.8 1. 2.1
12.6
0.1 1:
antmaze-ginn giant-navigaat wwrigate-singglattee task-2-v
1+1
44.9 + 6.7
23.2 + 15.3
52.2 + 96.5
1.2
animaze giant navigale-sing single--nglesssk 3-
011
0.9
13.7
10.2
antmaze giant-naa navigate singletask task4-v0
0
20.0 90.0
9.9 +0.8
17.8
19.8
0.4 +
untmaze-ginn -in---aa---ttttt -i--ga--aanianttteeeee siaeee-iinnlttss-aaseeeeee
19
$1.4
46.9
43.2
36.7
humanoidmaze medium- navigate-singlenn siviigute-iinnlleensstttk-----
33
34.1 +
83.5
28.8
humanoidmaze-moot wewaaazz-eed--meeeee won-nvvigen ntr--avigaoo-niggluus--tttts-eee
41 =
75.8 29.
04 #
91.2 6.
60.2 9.4
bumanoidmaze meee---e------- wat.-aaal--iinnlltusseee
25.7
10.7
28.9
humanoidmaze medium navigute singletask -
0 0.1
19.5 1Y.5
3.0 3.0.1.6..
1.3
humanoidmaze medium- navigate-sn envigat--singetan wetessinnnenssnnss--w....
97.0
humanoidmaze medium stitch singletask task1-ve
26.4 1:
29.1 4:
48.0 25-4
77.9
28.7
humanoidmaze-mette medium-stitch--ee singletask-tnsk2-
27.9 8.a
94.4 1.9
87.5 3.
95.2 2 1.8
49.2 6.3
humanoidmaze medium stitch sing letask:
21.5
55.2
humanoidmaze medium- stittchshhhhhhtt singletask-tnsseeeee letask-tusk4-vo
3.7
33.1 96.4
0.8 +
3.6 9.6
14.9
humanoidmaze-
48.5
92.5 3.1
91.1 2.5
0.6
60.2 7.4
humanoidmaze www.gate--inggeeeeeeee
1
27.8
19.8 13:1
57.2 23.7
4.5 1:
humanoidmaze- seee--rrr--naa sawiiaar-singgletss-ttaaaae
0 0
0.5 6.7
0.0 0.1
0.1 0.2
0.0 1.0::: 0.0
humanoidmaze large- navigate singlstass-k-ssskk.
8.8
humanoidmaze large navigate singletask task4
8.3 14.4
1.6
3.7
0.7
humanoidmaze- wareeee---------ee navigate mavigate--singeenee www.gaawwiiigglnnnnnnss........
1.9
2.2
13.1 18.5
navi-navigtteee anvigate--ingg wiggtte-iingggetssstuskk.......
62.1 1
77
77.0 +-7
17.0
untsoocer-ane
cathoocer-arca---nee arena-navigate hame--uiiggt---ingleesssssss---
17
78.5
88 4
91.2 2.2
16.8 2.6
antsoccer- aucou--mwiaaaa--nnilcaassssssss
614
untsoccer-nn aerenceer---e----eee mage-migggg----iiggntssss------.
348
89 -
40.2 112240
5.2 9.1
untsoccer-arces navaaa-avvian www.aaa--aaiii----iinnnaaa-aa-aa
2
48.5 = 6.1
29.1
untsoccer-arent www.rrrreen-----ssineeeeeeeeeeee www.mmmttthhhiiinllllnssskk-----
3.3
41.6
53.4
untsocce--rree wtw.:attticc--iigggttt----aaaaa-
5.6
30.0
19.1
53.0 -:
3.0
antsoccer arena singletask
19.3
untooczer.on arena stitch stwicch-iinglltask...o.. singletaskk-nsk44400
0.4
14.8 4
00.00-6.4
26. +
0.1
untsoccer-arema www.rrrrrnnaa------e-ee------e-- www.maatticciiingglllsss.as..s..
1.+16
4.8
1.2
2.9
waweeeinggl-plly-eeeeeeee blny-singletusk-takki
88
97
99.1 0..
42.1
heo--innle-plly--inneee play-ssingletsskttakkkk2
85
92 4
07
99.1 0.
38.8 6.4
cube-single-pe plangeee-pay-snngee-e wig-s-innnttsttakk33...
78
92
99.1 0.
32,8
cubo-single-plee plaggll--ly--aa Mingletask-taskee
78
$7
93
97.0 L.6
36.3 6.6
100.0 0.0
cube-single norglee--os---iiiee sangletas--takk2--1
55.3 1: 6.0
96.0 :
100.0 d..
100.0 1
99.9 1 0.2
cube single noisy singletask 1a
97.4 1.8
100.0
63.2
99.7 0.5
100.0 o..
100.0 + 0.0
99.9 +
cubo-single-ue ceo---iing---oss---ieee- sooisy-sinllotss--eedee.
60.9
99.8 =2
10..0
99.9 02
99.8 +
21

---

Published as a conference paper at ICLR 2026
Table 9: Offline RL full results on OGBench, part 2/2. Models were trained with random seeds and evaluated over 100 episodes, following the setup of prior work (Park et al. 2024a 2025). Scores are averaged across seeds; values within 95% of the best performance are shown bold, while italics indicate scores reported from prior work (Park et al.) 2025). GFP actor my our primary policy, while GFP VaBC Tw is reported as byproduct of training. The + symbol denotes the standard deviation over seeds. See Tab 8 for the other tasks.
Offinee RL algorithmss
Task
IQI.
ReBRAC
FQI.
GFP actor Tw
GFP VaBC
27
43.0
61 1
76.1 16.11:40
28.5
cube-double-pla
play-singletusk-taskk22
1=1
16.2 102200 5.0
36 6
53.3 s.9
1,8
cube-double- slaneee-ll--sillltss---a-ae
0 0=0
1.3 0.4
22 =
43.3 + 8.9
0.4 0.4
cube-double-ple wlanall---a--nngeetss-----e
0.4
5
7.1 a.1.
0.8
cube-double www.aooobbab-ssnnneeeeeee play=singletuskktas5555.
2.0
19
56.3 =
0.7
cube-double-noise wow.eeeooooo--nnntsssss--------
20.8
51.3
77.1
89.5
cube-douhle-noki -okk---inglleass-eeeee t0s22-00
0.0 +00+00
21.1 +
43.1
75.7
5.8
cube-double-noisy waw.uuuleoois--snnllleoo moosy-siiggetask-tssk...
0.8 1 1.0
8.0 10
26.3 4.
75.0
3,2
cube-double-
task----
0,2 =
15.5 +
41.8
cube-douhle-noki
0.5 0.5
triple-laa-sinnletasss
0.4
14.0 S.8
17.2 7.
32.4 8.4
cube-triplepplay
playy-iinglttussktask2222
0.0
0.1
0.8
caabe-tripe-play--eee blay--inggleuukktaakkk...
1.3 1.3
0.3 033
1.3 1.3
14.9 +
3,2
0.0 0.00000
0.0
0.4
2.5 1:5::00 1.7
0.8 0.2
cube-triple-play
chaippee-ppa--aaeee plapp-llalllyssseeeeee singletusk-taskk-.v0
0.1 0.2
0.3
0.1
1.0
wriple-noisy-singlttaeee
24.0 6.0
25.3 + 18.9
17.5 + 8.0
90.7 5.2
41.0 6.6
cube-triple-noite moisy--ingleeask
0.0
0.4
0.0
0.1
0,0
11.8 7.6
0,8
tube-triple-noisy
0.0
0.0
0.0
10.8 + 6.2
0.8 0.5
sube-triple-noisy
0.0 + 0.0
0.0 0.1
0.0
0.1 + 0.2
0.0 0.0
33 6
97 44
90 4
94.8 1:
54.6 7.4
puzzle-3x3
3
3
0.9
9.7 +:
211
211
16 15
5.4
9.4
3
513
16 1
14.1 8.4
10.9 4.8
12
43.4
34 8
50.0 1.
16.2 20.2::10 3.3
7
27
1
9.9
7.0
9
2.0
46.2
10.2 ::9
puzzle-4x4pplay
5 2
9.1
11 11
17.2 11:2::2
7.4
4
0.8 0.7
7 7
7.3 5.
6.6 +:
puzzle-4x4-nne
0.1 0.1
3.9 1.2
41.0
38.5 3.6
39.6 1: 4.8
puzzle-4x4-nng
0.0 00=00
0.+00
5.9
4.7
puzzle-4x4 s---onny-snngleasskttann
0.1
hen.le-4x4-noixy hin.iiy-singlltss-tukk4444
0.0
0.404
3.0 1.0110
6.3 1:
puzzle-4 x 4-noosy- sioisy--ingleaasktuukkkkv
0.0
0.0
3.7 1.7
0.7 1.7:000
3.1 +
scene-play-singe
94 1:3
95
100 20
99.8 U.4
99.8 d.
1243
50 13
76 9
89.0 4.1
93.0 5.1
senee-ply-siiglettss-ee
32
98 11
78.0 13.2
93.5 5.1
0
0.6
scene-play-singee pla-llyy-singlluus-ttsk-----
020
010
010
0.0
0.0 -0:200
secne-noisy woiis--iigleetssktaallee
74.2
94.8 #
100.0
99.9
99.9
secne-noisy- won-noosiisilesssseeeeeeseee singletask-taskk2-00
0.1 0.2
18.1 5.
87.4 5.7
94.2 +
97.4 1.9
scene-noisy wiss-eieneettaask3335
5.7 1.1
81.1 5.5
94.4 +.7
93.3 + 4.3
95.2 + 3.2
scene-nuisy-sinee muiss--iingleuus--tskk4-4
0.0
0.0
seene-noisy noisy moissysiinglllasktaasse singlettask-takk5-v00
0.0
0.0
0.0
0.0
0.0
visual-cube-sing weaiinle-pla--sinlleeeee --0
12
2:
4
21
19.7
sosuu--seeee-pla--snnggee
97
99.3
visual wwg.g/ieeeedd....l.iinneeeaeeee
0
33
19.7
Fallowing Park et al. (205 reduce the computational cost pixel-based tasks, for these tasks we use seeds and 50 episodes per seed; moreover, chose not to evaluate the VaBC policy, it is only byproduct.
22

---

Published as a conference paper at ICLR 2026
Table 10: Offiine RL Results OGBench Tasks (only task evaluated in FQL): Performance comparison across different offline RL algorithms. All prior work results are taken from Park et al. 2025 to compare wide range of previous methods against GFP.
Task
IQL
ReBRAC
IDQL
SRPO
FAWAC
FBRAC
IHQL
FQL
TM
antonace- antmaze-torge-aae -orge-narggeeeee wawgaaaagaaeeee singletack
lask1-v
0
48 2
91 =
0
0
=
1
70 20
21
1
95.4 0.5
task2-v0
a
88 4
11
4=4
1+1
0
36 12
8 8
57 +
92.2
lask3-vU
2
72
2
3
19
12:
83
52
95.6 2.7
task4-v0
8
51#9
81
62 10=00
45 15
17=6
10
37
18
80 +
90.6
2
55
9
76
untmuaze-gian-enee niatiiattnnn----ttteeee- hataamm-nnnggtt--nniglteta----e singletarkk-tnnk
taskI-v0
0::0
0
27 +
0 0
0 0:0
0
0
0
0
1
12.6 15.1
task2-v0
16 17
0+0
0
4
0
9
52.2
msk33vv
0::-0
0.:-
34 12
0 0
0::00
0 0
0.0
0
0.0
0
13.7
10.3
task4-v0
0=0
0
1
0
0
9+
0+0
14
17.8
task5.v0
1
19 7
49 22
0.111
0.:.0
0
0.4
10
13
16
43.2
35.7
luskI-v
1.1.
32 1
16 L
1.1
38 L
6 1.1
25 1.
69
19 12
83.5
3.7
task2-v0
91.2
tnsk3 v0
211
25 L
36 1.
0.1
211
83 1
19 L
27
49
74 1.
86.3
10.7
task4-vU
3.0
tuskS vo
211
66 4
24 1.
1.1
31s
91
31 L
63
98
1
97
95.8
humanoldmaze aaggoooddmooz-arg--n--eeeee www.aagg---nii----iineeeeeeee singletask
lask1vV
3
2
0::
0::
1
0:0
011
6:
7
57.2 2:2
lask2.v
0.1
0.9
0.1
0.1
0.1.
0 0.0
0.6
0.1
0.1
0.1
0.1 1o2
lss3-vv
141
743
8:
341
121
213
111
10
48 1
14.6
task4-v0
1
1
1
0:0
0=:
0
0:0
0
1
2
3.7
lask5-vU
021
1
212
0.20
0:0
0:2
1
0.0
1
13.1
task1-v0
0:
77.0
task2-v0
2
17 7
0
15
3
0
8
8
70
88
91.2 2.2
task3-v0
0:
0.0
11:55
task4-v0
1
3
0
0+1
0=0
0
12
21
16
39
40.2 4.2
task5-v0
2+2
0++
0
0
9
0++1
36
29.1
cube-single play singletss
task1-v0
1
88 =
95
=
$1
73
90.1 0.4
task2-v0l
3+11
85 s
02 +
96 +
82 +
80 +
81 +
83 +
73
97
99.4 + 0.7
usk3-v0
task--vfl
2+11
73 6
02 +
+
70 18
91
70 +
79 +
70
94 +
99.1 +
cube-double-ploy-tieee siinllll
task1-v0
27 5
46 4
30 + 19
7 1
21 =
21 7
47 +
35
61 #
76.1 4.6
task2.v0
0+
14:1
7
16
0
2
22
36 4
53.3
av
0:iu
Dio
1
17.1 1
0
3 11111
111
22
43.33
K.
vf
0+
040
0.+1
0.0
0.4-1
7.1 1.3.1
lsk.5vv
0 0il
4
4112
1.1
0 1.:1 9
3 3.1
9.11
3.1.
17
19
56.3
11.3
scene-ploo-singleeook
lask1-vU
1
94 ::
95 2
100 2
100
87 1
96 =
98 3
100 1
99.8 =:
v0
50
tas3-vv
1#1
32 1
56
94
4 4:4
19 =
38 1
78 =
54 19
98 =1
78.0 13.2
task4-v0
2=2
a
1
0.0
6
1
0
5
0.6 1.6:00
lask5-vU
0::
010
0:
() 010
01 02:00
0 010
0 0
0
0.0
0 0
0.0 0.0
task1-v0
5
33 =3
97
52
97 2
25
63 19
94
90
91.8
task2-v0
4
1
0
4
2
1
0.3
task3-v0
1
3
3+1
0:0
0
140
1+1
0
10
0.9 19000
task4-v0
2+1
2+1
0
0
1+1
2
0
6.4
task5-v0
1
3
0::
0
0
1
2
0
16
11.1
puzzle 4nd play singletsk
task1-v0
+
48 +
21
1
31 +
50.0
mssk2-vf
0+
7 7++
4
14
0++1
0 0
0++1
5+
4 4
9.9 2.5
task3-v0
9
34
21
1+1
20
50
46.2
mss4-vfl
0++0
5-+2
10 --
26 4
7+4
1++
0+0
5++1
17.2
v0
0.1m
4.1
7.1
24
111
0.o
0.1.
4.1.3
2.1
7.
7.3
Average
2.8
23.4
30.9
22.8
14.2
24.7
16.1
28.0
30.0
43.5
51.8
tasks)
23

---

Published as a conference paper at ICLR 2026
Table 11: Offiine RL full results on D4RL. For each task, models were trained with 8 random seeds and evaluated at the end of training. Reported values are the average normalized scores over the final 100 evaluation episodes, with + denoting the standard deviation across seeds. Italics indicate scores from prior work (Fu et al., 2020; Fujimoto & Gu 2021; Tarasov et al. 2023] Park et al. (025), and bold denotes values within 95% of the best performance. GFP actor Th our primary policy, while GFP VaBC Tw is reported as a byproduct of training.
Offline RL algorithmss
Task
IQI.
ReBRAC
FQL
GFP ma
D4RL antmazc-umaze
55
74.0
87.5
78.6
97.8 13
96 +
96.8 =
94.9
D4RL antmaze-umaze-diveee
47
84.0
62.2
71.4
88.3
15.0
1
91.9 =
90.1
D4RL animaze medium play
61.2
71.2
3.0
84.0
L.4
L
81.9 L.
57.4
D4RL cottaaz--medum-diiee
52.7
70.0
19.6
76.3
13.8
71 1:
61.6
20.9
15.6
D4RL animaze large play
15.8
39.6
0.0
60.4
1.
82.6
62.6
D4RL cermmazz-lrre-diveeee
14.9
47.5
0.2
54
25.1
4
84.1
70.6
pen-buman-vl
71.5
103.5
6164
67.1
D4RL pen-cloned-v1
52
39.2
37.3
61.4
91.8
74
77.1 10.4
70.5 4.2
pen-expert-vl
153.6
146
154.1
D4RL dour-human-v!
g
9.9
4.3
-.1
0.0
0.0
01.3 0.3
4.1
D4RL door-cloned-v1
8
1.6
0.6
D4RL dour-expert-vl
105
101.5
105.3
84.6
104.6
104
104.1 1.0.
103.1
hammer human-y
4-4
1.4
0.6
0.2
D4RL clauddr-clond--v
2.1
2.1
0.8
6.7
11
12.4 5.4
2,5 0.9
hammer expert-vl
129.6
133.8
123.6
D4RL relocare-humane
0
0.20
0.1
-0.2
0.0
0
0.5 0.3
0.0 +00=00 0.0
relocate
D4RL relocate-cxpert-Vv
108
95.0
106.5
107.3
100.6
107
103.2
104.0 3.1
Table 12: Offline RL full results on Minari. For each task, models were trained using different random seeds, and evaluation performed the end of training. The reported values represent the average normalized score, computed over the final 100 evaluation episodes and averaged across the seeds. GFP actor ma represents our primary policy, while GFP VaBC Tu is reported byproduct of our training procedure.
Offline RL algorithms
Task
FQL
GFP actor Ne
GFP VaBC
Minari pen-human-v2
11.5 + 4.9
50.1 + 6.3
54.4 + 6.7
Minari pen-cloned-v2
41.8 3.7
60.4
54.0 + 5.5
Minari pen-expert-v2
92.7
115.9 - 4.5
108.7
Minari door- woor-human-v2
1.1
0.4 1.4:0. 0.1
2.7 1.9
Minari door-cloned-v2
0.4
0.3 +
0.1 0.1
Minari door-expert-v2
102.0 0.9
94.1 + 20.9
99.0 10.3
Minari hammer-human-v22
1.0
2.7
3.1
Minari hammer-cloned-v2
1.0
22.9 $4.9110 19.5
4.9
Minari h2
121.2
130.2
119.0
Minari relocate-humanc
-0.0
0.1 + 0.2
-0.0 - 0.1
Minari relocate-clomed-v2
0.0 0.0
0.3 0.2
0.0
Minari relocate-expert-v2
103.7 1.2
102.1 + 5.3
105.9 + 1.2
Minari halfcheetab--simlee-y
59.2 - 0.3
72.5 0.5
64.4 + 0.4
Minari haffchheetah-meedin--0l
100.2 + 6.7
121.3 5.8
108.6 + 5.3
Minari halfecheetah-expeert-dd
134.1
133.4
1.3
136.4 0.8
Minari hopper-simple-v0
57.2 5.9
91.6 + 4.3
87.4 6.6
Minari modper-medium-vo
81.9 + 23.9
79.6 + 14.5
78.2 + 24.2
Minari hopper-expert-vo
99.6 10.9
103.9 10.6
108.8 13.1
Minari walker2d-simples.
89.4 0.7
90.0 0.9
89.7 - 0.9
Minari walker2d-medium-oe
127.6 6
133.7 =
126.1 1 +
Minari walker2d-experr-vv0
148.0 + 1.8
149.9 + 2.0
150.8 + 0.4
24

---

Published as a conference paper at ICLR 2026
Table 13: Task-specific hyperparameters for offline RL on OGBench.
Offiine RL. algorithms
Tusk Cutegury
IQL.
ReBRAC:
PQI.
CIPP (ours)
GIP-AWR Sec R.
(ax. o2))
(a,, 7)
[a, n)
navigute singleask-sukk1122....
1e+1
(1e=2, 1e-21
1c+1
(3e=1, 1e-4)
antmuae large sinich--inggltsss (.33)
1e+1
(1e=2, 1c-2)
3e+0
(3e=2, 1e-6)
(1e-1, 3e-1)
antmuue eo (1pppeee-nnggoeeannnn22244444473
To+0
(Te=3, 1e-1)
1e40
(le=2, le-6)
antmaze-Sarg waaazzz--r--gaantttt htt.://www....nocoooaooa.d.23223
(1e=2; 2)
3641
(1e-1, 16-1)
(16-1; 3e-1)
humanoidmare- wtw.a//www.iiiiiiiiioooaoaooaaaa
(Ie=2, 16-2)
3e+1
(3e-1, 1e-3)
(3e-1; 3e-11 1)
humanoidmaze medium stithh (13_:2_6__
lo+1
(1e-2, 16-2)
le+2
(3e-1, 1e-3)
(3e-1, 1c=1)
humanoidmace hage awvippeessnnkeuuuuuuus22222431-3
(le-2; 0e++0)
Le+2
(3e-1, 1o-4)
(3e-1, le-1)
antsoccer navigate singletusk 0
(1e-2, 0c++0)
(1e-1, 1e-2)
anisoccer srent- stich singleinkk bask (223((
lo+0
(le-2, 16-3)
le+1
(le-1, 1e-2)
(le-1, le-1)
cube single play (12.2cc)
1
(1e+1. 1e-1)
3040
(le-1, le-1)
3e+1
(le41,
cube double playysing/easskuuukk/223333333.3
(1e=1, Je-1)
(1e40, 1e=2)
(1e+0, 1e=1)
cube double 1ingleetssi
3e-1
(1e-2, 1e-2)
le+1
(1e=1, 1e-4]
(le-1, le-1)
cabe-tripl--paa--iie (an.reeeeeeennnooeonn..22223...-
le+0
(1e=1, 1e-3)
3e42
(le+0, 1e-5)
(le+0, 3e+0)
3e4-D
(le=2, 0e+-0)
Te41
(le=1, 1e-5)
(3e-2; Te-0)
some-play playe-ppyaaee singkun--uukkl12233433333333
(Te-1, 16-3)
(le+1, le-S)
scene noiry singletask (22((1.)
lo+1
(3e-3, 0c++0)
3e+1
(1e+0, 1e-4)
puzzie-3 x h.glyyy.nngmmmmiaaaa222333333333
(3e40, 16-3)
(3e400 1e+-0)
puzzle-4 x 44 play wigglttassttskkkkkkkkka (1233333)
(1e-1, de+0)
(3e+0. 1e-5)
(1e+0, 3e-1)
puzzle-4 x moisy (12.
le+0
(3e-2, 1e-2)
3e+2
(3e+0, 1e-3)
(3e-1, 1e-1)
visual- cube single play slyy-sanggeaaak-mmaii-0
(1e+1, 1e-1)
Hisml-cube-druble dewuhll-pee--innnsssssaa
(3e-1, 1e-2)
visual plat-pllgllllsskkeekkkk glay-singlsaask-askkkk1
(1e41, 1e-3)
visual puzzle 3xch play singletask -00
(3e+0, 1c-2)
visual tisull-puzll--au puzzle-ix--pla plag-singleassktaae
(le+0, 1e-4]
Table 14: Task-specific hyperparameters for offline RL on D4RL and Minari.
Task
GFP (ours) (a, n)
Task
FQL
GFP (ours) (a, 1)
D4RL antmaze-umaze
1e-11,1-33
Minari pen-human-v2
le+44
(3e+0. le-
Minari p
3e 13
[lc 0, lc 4)
D4RL antmaze-umazz-divers
(1e-1, le-3)
Minari pen-expert-v2
1e433
(3c 1, lelle-
D4RL antmazz-medium -aatiluuupppyyyupo
(3e-2, le-3)
Minari door-human-v2
3e4-4
(1e-2, 1e-2)
D4RL medimaz- diverse
(3e-2, le-3)
Minari door-cloned-v2
3e+-1
(3e-2, 1e-6)
D4RL larmaaze-largeplaa
(e-2,1e-5)
Minari door-expert-v2
3e:4
(3e 3)
D4RL antmaze-large largeazi---eeg--ieeeeeeee
Be-2,1e-5)
Minari hammer-human--v
3e4-4
(1e40,
Minari hammer-cloned-v2
3e+4
(3e+0, le-5)
pen-uaannl
(3e+0,
Minari hammer-expprt--V?
let4
(1e+0, le-5)
D4RL pen-cloedddv
1e-5
Minari relocate-human-12
3e+3
(3e+0. le-5)
D4RL pen-expert-vl
(le+0,1e-3)
Minari relocate-clomed-v2
3e14
(3c+0,
D4RL door-human-vl
(1e+1,le-2)
Minari relocate-expert-v2
3e+4
(3e+0, le-3)
D4RL door-cloned-vl
(ee+1,le-2)
Minari halfcheetah-sinnleedd0
le+2
(3e-2) 2, le-
D4RL door-expert-vl
(le+1,1e-2)
Minari balfcheetah meeeheettaammim----dd
le
(lc 1, lc 3)
D4RL hammer-human-vi
(le+1,1e-5)
Minari halfcheetah vo
1e43
(3e+0. le
D4RL hammer-cloned-vl
(le+1,le-5)
Minari hupper-simple--V
3e
(3e 0, le 3)
D4RL hammer-expent-vl
(le+1,
Minari hopper-medium--v0
3e 3
(3e 0, le 3)
D4RL relocate-human-vl
(le+1,
Minari hopper-cxpert-vd v
1e+3
(3e+0. 1e-2)
D4RL relocate-cloned-vi
(le+1,1e-4)
Minari walker24-simple-00
le43
(3e+0, le-1)
D4RL relocatteexpeer--vl
(le+1,1e-4)
Minari walker24d-meiium--00
3e+2
(3e-1, le-2)
Minari x0
3e+
(3e+0. le-3)
(a) D4RI.
(b) Minari
25


Reviews:
[{"review_id": "VhyNWrVS0j", "reviewer": "Reviewer_Aadb", "summary": "The paper proposes Guided Flow Policy (GFP), an offline RL method that couples (i) a multi-step flow-matching policy trained by value-aware behavior cloning (VaBC) and (ii) a distilled one-step actor guided by a critic. The key idea is a bidirectional guidance: the actor and critic guide the flow model to preferentially clone high-value dataset actions (rather than all actions), while the flow model regularizes the actor to remain within the support of high-value dataset transitions. A temperature-controlled guidance term g_η modulates how selective the cloning is. Extensive experiments across 129 tasks spanning OGBench, Minari, and D4RL show that GFP consistently outperforms pervious methods.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The method is elegant and easy to follow. It cleanly integrates flow matching into the BRAC framework: a value-aware flow policy shapes the action distribution, and a distilled one-step actor provides fast inference. The bidirectional guidance narrative is intuitive, and the overall training loop is straightforward to implement.\nThe empirical evaluation is thorough. The paper reports results on 129 tasks spanning OGBench, Minari, and D4RL, providing broad coverage and reducing the risk of cherry-picking. Across these diverse settings, GFP demonstrates consistently strong performance.\nFinally, the paper offers a useful sensitivity analysis of the temperature η. The study clarifies how guidance sharpness trades off selectivity versus diversity, giving practitioners a clear knob to choose stable, high-performing settings.", "weaknesses": "First, the novelty is moderate. The method still fits squarely within the BRAC family: it combines an expressive flow-matching policy with value-aware cloning and then distills into a one-step actor. While the synthesis is thoughtful and practically useful, the conceptual step beyond prior behavior-regularization plus expressive policy models feels incremental.\n\nSecond, the paper’s structure could be clearer, especially in the Experiments section. Main results and analysis are interleaved, which makes it harder to extract the headline takeaways before diving into diagnostics. I recommend separating “Main Results” (per benchmark, with concise aggregates) from “Analyses & Ablations.” In particular, if my understanding is correct, Section 4.1 “Suboptimal datasets” functions primarily as a temperature sensitivity study on η; the current title is confusing. Retitling to something like “Sensitivity to Temperature on Noisy/Suboptimal Data” would improve readability.\n\nThird, the role of distillation isn’t entirely clear in the current narrative. Table 3 reports VaBC-only results, but the training pipeline still relies on a distillation phase, which makes it hard to tease apart how much of the gain comes from the value-aware flow versus the student actor. A clean ablation that removes distillation end-to-end (i.e., deploy the flow directly), paired with a short intuition for when distillation helps would clarify its necessity and practical benefit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "heb1PSVyyh", "reviewer": "Reviewer_Zzn2", "summary": "This paper introduces a framework for performing offline RL using flow policies. Compared to prior such as FQL, which first trains a BC policy as a flow network, then uses this policy to distill a one-step policy which is then trained via Q-learning, this work also connects the Q network back to the flow network. Specifically, an AWR-style weighting is used on the loss, weighting higher-advantage actions. In essence this creates *two* feedback mechanisms for the learned Q function -- it affects the flow network (through advantage-weighting) and also the resulting one-step policy (through the DDPG loss). Experiments show that this method reliably outperforms prior work on a wide range of experiments.", "questions": "In the experiments, which version of the TD learning objective is used? Equation 7 shows that actions can be taken from either the distilled policy or the flow policy. Presumably taking actions from the flow policy is computationally expensive as an ODE integration is required.\n\nIn equation 6, the notation to use pi_c is somewhat confusing.\n\nDoes the optimal temperature change if the BC weighting (lambda) is adjusted, and vice-versa? What does the landscape of these hyperparameters look like, and how sensitive is the method to both of them?\n\nIt is interesting that both the one-step policy and the VaBC policy can be used for evaluation, as mentioned in table 2. What do the average scores for the VaBC policy look like?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper presents a clean framework with empirically sound results. The idea of using the AWR weighting to train the flow network is a neat way to achieve policy improvement wrt a learned Q function. It is useful computationally that the TD errors for Q-learning can be calculated using the distilled one-step policy, avoiding the need for an expensive ODE integration during training. \n\nIn terms of clarity and quality, the paper reads well and is easy to follow. See weaknesses section for additional points here.\n\nThe significance of this paper is boosted by its strong empirical performance and wide evaluation. The paragraph describing how the results in this work were re-run for previous baselines, and in fact improved upon them, lends credibility to the evaluation. In addition, the scaling curves with relation to temperature in Figure 3 clearly show that the intuition for the temperature term is correct in practice.", "weaknesses": "As this framework in essence uses two methods of feedback from the Q function, the paper would greatly benefit from a clear analysis on the effects of both of these terms. For example, two clear baselines are the same method where either the flow policy is not weighted (or if this is exactly FQL, some text making this clear) and where the one-step policy does not utilize the DDPG loss. \n\nBuilding on the above, an analysis as to why the two feedback methods are complimentary would take this paper one step forwards. As of the current submission, it is shown that empirically it is useful to use both, but it is unclear on a deeper level why this might be true. The paper hints that \"there is no fundamental justification for preferring one divergence or distance metric over another\", and more investigation into this would be potentially fruitful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X0NBpodcS2", "reviewer": "Reviewer_AGZW", "summary": "The paper proposes guided flow policy (GFP), an offline RL method that leverages flow-matching to train expressive policies that can tackle tasks with suboptimal datasets. The proposed method trains the base flow-matching policy  $\\pi_\\omega$ via weighted behavior cloning to match $\\pi(a\\mid s) \\propto_a \\pi_\\beta \\exp(\\tau Q(s, a))$ (where $\\pi_\\beta$ is the behavior distribution described by the dataset). Then, it applies a recently proposed technique, FQL, on top of $\\pi_\\omega$ where it trains a 1-step distilled policy to stay close to the base flow policy $\\pi_\\omega$ while maximizing its value under the critic. In practice, the authors adopt a slightly different/more conservative Q-target backup by using a mixture of actions from both the more aggressive 1-step distilled policy and the more conservative multi-step base flow policy. Across multiple benchmarks (OGBench, D4RL, Minari), the proposed method outperforms FQL and Gaussian policy offline RL baselines (ReBRAC, IQL) especially on noisier offline datasets.", "questions": "Related to my concern above -- how is the proposed `value-aware behavior cloning' different from prior methods? (e.g., QIPO and FAWAC)", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper considers a comprehensive set of benchmark tasks which make the comparison between the proposed method and the baselines very convincing. I also appreciate that the authors seem to properly tune the hyperparameters for the baselines, which make the comparisons fair.\n\n- The proposed method is presented in a concise and clear manner and the algorithmic design has no technical issues.", "weaknesses": "The paper make some factually questionable claims\n- Table 1: all prior methods (IQL, TD3+BC, ReBRAC, FQL) are listed under \"handles suboptimal data (x)\" which implies that these methods cannot handle suboptimal data. This is misleading because most offline RL methods listed and in fact most offline RL methods in general can handle suboptimal data. It would be good to further clarify and be precise about what \"handles suboptimal data (x)\" actually means here.\n- The authors introduce Value-aware behavior cloning (VaBC) as a novelty in the algorithm and claim that \"this is the first integration of value-aware behavior cloning\", but as how it is currently presented it seems to be me that it is exactly equivalent to advantage weighted regression applied to flow-matching, which has been explored in many recent flow-matching/diffusion papers (e.g., FAWAC baseline in FQL[1], energy-weighted flow matching loss in QIPO [2]).\n\nThe comparison to prior work is lacking:\n- The paper only compares with a very limited number of baselines (FQL is the only baseline for expressive policies). For the similarity to prior work as mentioned above, I would encourage the authors to consider comparing to some of them such that the reviewers could be more informed about where GFP's performance stands in the literature.\n- The related work section discusses very little prior work despite a rich literature in diffusion/flow-matching in reinforcement learning and it is not clear from the section how the proposed method is different from these prior work.\n\nSmall typos:\n   - Table 4: in the value column, second row: \"50,0000\" should be \"500,000\"\n   - L248: $Q_{\\bar \\phi}(s', \\mu_\\omega(s', z)$ is missing a right bracket. \n\n[1] Park, Seohong, Qiyang Li, and Sergey Levine. \"Flow q-learning.\" arXiv preprint arXiv:2502.02538 (2025).\n\n[2] Zhang, Shiyuan, Weitong Zhang, and Quanquan Gu. \"Energy-weighted flow matching for offline reinforcement learning.\" arXiv preprint arXiv:2503.04975 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IofnxmINcP", "reviewer": "Reviewer_Zk33", "summary": "This paper proposes Guided Flow Policy (GFP), an offline RL algorithm that trains a flow policy. The method is built on top of the behavioral regularized actor-critic framework, and more specifically, flow Q-learning (FQL). The main differences between GFP and FQL are that (1) GFP uses a weighted BC flow policy instead of a vanilla BC policy, and (2) it uses a more conservative Q target by mixing the behavioral action and the policy action. The authors evaluate their method on a large number (129) of tasks across OGBench, Minari, and D4RL, showing that GFP outperforms FQL and other strong baselines.", "questions": "Have you tried vanilla AWR for learning a weighted BC policy? Why do we want to have the additional normalization term in the denominator (compared to AWR) in Eq. (10)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and easy to understand.\n* The empirical evaluation is notably thorough. Across more than 100 tasks, GFP consistently achieves better performance than other baselines (IQL, FQL, and ReBRAC). The authors also mainly use 8 seeds for each task.\n* The paper provides several ablation studies, including analyses of the effects of conservative Q targets, weighted behavioral cloning, etc. They demonstrate the importance of these techniques.", "weaknesses": "I don't see major weaknesses in this work. Two minor ones are as follows:\n* One weakness is its relatively limited novelty and contribution. Essentially, the paper can be summarized as AWR + FQL (or broadly AWR + BRAC), which is a rather straightforward extension of existing methods. While I do appreciate the simplicity of this combination, given how straightforward the method is, the paper could further be strengthened by either having more large-scale experiments (e.g., more challenging/long-horizon tasks, more realistic benchmarks (or even real robot experiments), pixel-based tasks, etc.) and/or providing additional insights beyond simply applying existing techniques to the flow RL setting.\n* Another weakness is that this method requires an additional parameter ($\\eta$) that needs to be tuned per task, compared to FQL. Generally, introducing more hyperparameters would naturally only make the method better, so the real question is how large the gap is compared to the closest baseline (FQL) with this extra degree of freedom. From Figure 4 and Table 2, the difference doesn't seem unreasonably small, but it would have been even more convincing if the proposed method had achieved an even larger performance boost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)