Flow Actor-Critic for Offline Reinforcement Learning
Authors: Jongseong Chae
Jongeui Park
Yongjae Shin
Gyeongmin Kim
Seungyul Han
Youngchul Sung
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
offline reinforcement learning
flow actor-critic
flow policies
flow matching
Primary area: reinforcement learning
TL;DR: We propose Flow Actor Critic (FAC), which leverages a flow behavior proxy policy not only to regularize the actor but also to penalize the critic via tractable behavior densities, effectively mitigating value overestimation for OOD actions.

Abstract: The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in out-of-data regions. To this end, we propose a new form of critic regularizer based on the flow behavior proxy model obtained as a byproduct of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-art performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.

Published as a conference paper at ICLR 2026
FLOW ACTOR-CRITIC FOR OFFLINE REINFORCEMENT LEARNING
Jongseong Chae!
Jongeui Park' Youngchul Sung Yongjae Shin!
Gyeongmin Kim!
Seungyul Han?
KAIST PNIST
[jongseong.. chae, yws.nggkkaist.......
ABSTRACT
The dataset distributions in offline reinforcement learning (RL) often exhibit com- plex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in oft-of-data regions. To this end, we propose a new form critic regularizer based the flow behavior proxy model obtained
byproduct
of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-ar ahe performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.
INTRODUCTION
Offline Reinforcement Learning (RL) seeks an optimal policy from a pre-collected dataset without ad- ditional enviromment interactions, which may be costly unsafe in real-world situations (Lange et al., 2012. Levine et al. 2020). Despite such advantages, offline RL suffers from value overestimation for out-of-distribution (OOD) actions due to its limited dataset coverage, which degrades performance. Many methods have been proposed to deal with this OOD value overestimation based on techniques such as constraining the target policy near the behavior policy of the dataset (Kumar et al. 2019
Fujimoto et al. 2019 Wu et al. 2022 Fujimoto & Gu [201T] Nair el al. 2020) or suppressing the value function in the OOD region with penalization (Kumar et al.) 2020
Mao et 2023). These
methods have been shown to be effective for many offline RL tasks.
As offline RL datasets accumulate and grow diverse, however, their behavioral distributions become complicated and sometimes multi-modal. Thus, simple behavior policy modeling becomes insuffi- cient (Park et al.] (202bb) and more expressive policies are required to model the behavior policy and its support. For example, diffusion policies adopted from the vision domain have garnered strong attention from the RL community as a candidate for expressive policies (Wang et al. 2022,] Chen et al. 2022, Hansen-Estruch Estruch al.) 2023 Chen et al. 2024 Zhang al.) 2025). Diffusion policies learn the underlying data distribution using forward and reverse processes and can be trained efficiently by score matching with noise models (Ho et al. 2020 Song al. 2020 They have shown impressive offline performance. Even with this advantage, diffusion models have a limitation that they can be used construct actor capable of sampling actions close to the behavior policy, but the iterative nature of their multi-step processing makes policy optimization unstable (Park et al. 2025)
Therefore, Park al. (2025) recently adopted the flow model (Dinh et al., 2014 4
Rezende al. 2014; Dinh et al.) (016) as an alternative for expressive policies. Although their work proposed one-step flow actor and demonstrated promising results, they did exploit the full advantage of the flow policy. They only considered the actor design to maximize the conventional Q function under a constraint on the distance of the target policy from a flow-hased behavior policy estimate, but overlooked the key fact that an accurate behavior density estimate itself is available with the flow model contrast to diffusion models.
"oorrespnndigg author.
Code available tttpss//ggthaaa.c.////gggggnggee

---

Published as a conference paper at ICLR 2026
In this paper, adopt the flow model and fully exploit it in a joint way to optimize offline RL policy. We not only use the flow behavior policy model as the distance anchor of the target policy but also use it for Q function penalization in the OOD region by identifying the OOD region based on the flow behavior density, which provides strong density contrast to distinguish ID region from OOD) one. Note that it has long been considered in offline RL that identifying the OOD region is difficult, and many previous works circumvent this difficulty with indirect methods (Kumar et al. 2020 Mao al. [023 Wu al., 2022). However, the highly expressive flow behavior model and accompanying density provide us with rare opportunity to directly tackle this problem. Our flow actor-critic (FAC) designed in such way exhibits outstanding performance in many current offline benchmark tasks.
PRELIMINARIES
2. OFFLINE REINFORCEMENT LEARNING
An RL problem is formulated as Markov Decision Process (MDP) M PS. 7). where S is the state space, A is the da--immenionnl action space, P (s'(s, s. a) is the transition dynamics, Po is the initial state distribution, r(s, a) is the reward function, and y is the discount factor (Sutton & Barto, 2018). The goal of offline RI. is to find a policy π S + A (A) that maximizes the expected discounted return (mO) P. [Et= 7'r(St, at)] from static dataset DD {(s, s')} collected by an unknown underlying behavior policy B. For a policy म. the action value function is defined QT a) = Es 00 7'r(st, az) [s0 8,00 al and can be estimated by iterating the Bellman operator Q a) = r(s, a) + ((s.a(((..)))
[Q(s', d-)]-
Cunstrt-Constraiied Policy Optimization. The main challenge of offline RL is Q-value overes- timation in the OOD action region outside the support of 3 (Kumar et al. 2020 Mao et al. (023).
typical approach this problem is policy optimization under distance constraint between the target policy and the behavior policy (Fujimoto et al. 2019 Peng et al. 2019 Wu et al. 2019 2022):
max Ea~D,awx [Q(s,a)] s.t. LE~D(D(T(()) 3(1ss)) < 6,
where is some distance or divergence measure. The rationale behind eq. [ is that by restricting
the distance of from 3, the action from does not fall into the OOD region of the estimated Q.
avoiding the overestimated region of Q. Practically, the hard constraint in eq. (1) is replaced with a
regularization term, yielding the following optimization (Kumar et al. 2019 Fujimoto & Gu 2021):
max Ea-D [Q(s a)] - A EEED [D (T(+))), B(1s)))
where A >>0 0 is regularization coefficient. Note that eq. 2 can be viewed as the Lagrangian of the problem
2.2 FLOW MODELS AND FLOW-BASED POLICIES
Normalizing flows are popular framework for modeling complex data distributions (Dinh al.)
2014 Rezende et al. 2014). A continuous normalizing flow (CNF) is defined as bijective function pat (Chen al. 2018), specified by time-dependent vector field U, via the Ist-order
0, ordinary differential equation:
du (z) (x(((x))),
Φ0(x)= x,
(3)
yiclding
Fu (u(o)) ze + I ue (x()))dt.
(4)
Given data samples ii ~ Pi (z) and a simple base distribution Po (e.g., standard normal), we want to learn the mapping &u such that (x)) xo time = 0 is transported to di (zx) i at time = 1. Since (1(-) is deterministic function, the density of p1 can easily be computed by the change of variables: log (21) = log (o6)) log det (00(z))
It is shown that the determinant of the Jacobian can be replaced with the divergence of the vector field v. (Chen et al. 2018 Ben-Hamu et al., 2022). Thus, the data distribution density can be written as log (z1) log (20)
azo
So Vz., Va (a(a.)du.
2

---

Published as a conference paper at ICLR 2026
A recent efficient way to learn flow for set of pairs (Zg. 21]] } is flow matching (FM), not requiring complicated likelihood maximization (Lipman et al. 2022). FM learns the velocity field such that for pair (zg. 21), the instantancous movement velocity vector Vn (z.) matches the overall displacement vector 21 - zo along the points on the line connecting zo and 01, i.e., Tu (1 u. + u.e Thus,
when the velocity vector field is parameterized with 0, the loss for learning 0 is given by
min Eu-UUni((( ().. (1) #o~Po(2)
(Tas u) - (α) xo)1
(5)
Once by is trained, samples from p1 can be generated by solving eq. 4 with an integral solver by setting = 1, and the density of the samples is available aforementioned.
Flow-based Policy. Constructing a policy from the flow model is straightforward. One can define the flow function from the action space to the action space, depending on both time and state Then, by pairing ~ (0, I) and action a (from the behavior policy), the corresponding velocity vector field parameterized with & is learned with the following loss (Lipman al. 2022 Park et al., (025):
min E (...))-...-
[| 8, u) - (a z) =)]
where ā. (1 u)= + ua. We refer to the so-Iearned flow policy with (s, a) as the behavior proxy policy B, (-1s). Then, the behavior proxy density is given by
log B (als) [)] = log Po( / Va.. oola 8, uud..
(7)
One thing to note is that samples from flow policy are stochastic samples for given due to the initial condition 2 (O, I) even though the flow function itself is deterministic. Thus, we will use the notation a(s, 2) show this dependency in the case of flow action, if necessary.
MOTIVATION: STRENGTH OF FLOW BEHAVIOR PROXY DENSITY
Now let us investigate how well the flow behavior proxy policy (-1s) tracks the actual behavior policy in terms of both sampling and density estimation. For this, we conducted an experiment with 3 being 2-D Gaussian mixture with four modes, as shown in the leftmost plot in Fig. For comparison, we considered four other behavior cloning (BC) models: simple Gaussian, conditional VAE (Kingma & Welling 2013). and diffusion models with 10 and 50 denoising steps (Ho al. 2020 Sohl-Dickstein et al. 2015: Wang et al., 2022). For the flow model, we used the Buler method with 10 steps as an integral solver for both sampling and density evaluation (Lipman et al. 2022 Chen et al. 2018). The results are shown in Fig. In the case of VAE and diffusion models, we show the ELBO since the density is not directly available. Details of the experiment are in Appendix C
As expected, simple Gaussian with single mode cannot distinguish the in in-distribution (ID) region with peak at the center of the four true modes on which the actual density is almost zero. VAE finds the four modes but generates noticeable samples outside the dataset support. Furthermore, amortized inference places probability density low-density areas when the latent variable does not
Ground
Gaussian
Diffusiinn (T=10)
Diffusion (T=50)
Flow Matching (T-10)
GT: log density
Gous.: log density
VAE: ELBO
DVf.. (T-10): ELBO
Diff. (T-: ELBO
FM (T=10) log density
Figure 1: BC models on synthetic fou--component Gaussian mixture dataset: Top row samples from each BC model. Bottom row log-density or ELBO plot.
3

---

Published as a conference paper at ICLR 2026
match the true posterior distribution, and hence the obtained ELBO is quite different from the true density as seen the bottom row of the figure. The diffusion model with 50 denoising steps tends to generate samples within the in-distribution region, while the performance degrades with 10 denoising steps. Note that the obtained ELBO of density is bad, almost not separating the four modes to make it not suited for density estimator, whereas the sampling performance is satisfactory. In contrast, the flow behavior proxy model shows good sampling performance and, importantly, yields a very strong estimate of the true density. Thus, the flow model can be used not only for actor but also as density estimator that can be used to distinguish the ID and OOD regions. In the following section, we present our algorithm design both actor and critic, exploiting these two facts.
4 METHOD: FLOW ACTOR-CRITIC
4.1 CRITIC PENALIZATION WITH FLOW BEHAVIOR PROXY DENSITY
A typical way to prevent overestimation of Q values in the OOD region for offline RL is to learn a penalized Q function. CQL (Kumar et al. 2020 and SVR (Ma et al. 2023), two representative methods, use the following losses:
min (EenD,an [0(s, a)l EaDD [Q(s, + TD Loss Q
and
((als) (Q(s,a) - Qmin)²) + TD Loss, B(als)
min a
(LnD) - Qmin) EEw
respectively, where the TD loss is given by E '' (Q(ssa))TTQQC Q(s, a))² and ((als) in SVR is distribution covering the entire action range. It is known that the CQL. penalization does not properly implement the correct Bellman operator in the ID region (Mao et al. 2023). SVR fixes this problem by using the difference of importance sampling (IS) integration when 3 does not dominate Ç. Although SVR circumvents direct OOD region identification intelligently using the IS technique, the main drawback is simple modeling of B, such Gaussian density, required to compute the IS ratio. As seen in the previous section, when the true behavior distribution is complicated and multi-modal, Gaussian modeling places high non-zero density on regions with nearly zero actual density. In such cases, the IS ratio blows up incorrectly, deteriorating the performance, as shown in Appendix D.2
With the flow behavior proxy density Be. a strong estimator for the true behavior density / at our hands, we can directly identify the ID and OOD regions and control the Q penalization term. To this end, we define the weight @
(s, a) as
50
2 a) m0 1 - (3p for some e > O.
Ave
(8)
Figure 2: weight w
The weight de (s, a) vanishes in the well-supported region with (a(s)) 2 e, and increases linearly as Gg decreases below the threshold towards zero. The threshold e is introduced because 3 is not perfect even though it is strong. We want to exclude the weak spurious density from the ID region but gradually. With this weight, we propose the following loss for critic learning:
min a EAWD, amn Tu (s, a ((s, a)] + TD Loss, Q
where a >> 0 controls the strength of penalization. In eq. our penalization term is not in the form of some difference as in CQL or SVR, but directly pinpoints the OOD region based on the strong behavior density estimator B.g When (als)) > e, i.e., support(3) with confidence, the penalization term disappears. However, the penalization term gradually kicks as the confidence decreases below e.
Proposition 1. Let 33 be the underlying behavior policy, 3 be our proxy for B, a be the learned actor, and Q be the value function of π. Consider the original Bellman operator T (FF = r(s, a) ([E.'rP(()s.a)) o' (-(a') (Q(s', a²)). Then, in the tabular setting without function approximation, the
4

---

Published as a conference paper at ICLR 2026
objective 9 yields the following operator:
(*,(s,
if B(als) 2 e, B(als) > 0
(s,
Q))
" (s.a)=(als) Bla
if B(als) <6 B(als) > 0
(10)
-c
if B(als) = O.
unless B(a|s) = 0 and (a (s, = simultaneously.
By the definition of the weight ~ a), it highly unlikely that we have B(als) and (s, a) 0 simultancously. The operator associated with our new penalization maintains the original Bellman operator inside most of the ID support region, while strongly suppressing the Q value in the OOD region. Furthermore, for weak confidence of the estimator, it gradually suppresses the Q value according to its confidence. Thus, when the proxy Aas. is a strong estimator of 3, the proposed penalization works properly. The proof of Proposition [ and rigorous analysis of the operator Tpa is provided in Appendix EE showing that the operator T has unique fixed point, yielding unbiased Q values for state-action pairs with (a]ss 2 e and underestimated conservative Q values for those with (a)s) <.
Note that in our critic penalization, determining is important. We consider two datasct-driven methods for e design: (1) dade constant threshold min (als) and (2) batch-adaptive threshold (als) using mini-batch samples B = {(ss a}} from D. The dataset-wide threshold does not exclude some of the actual ID region of the dataset, being suited datasets with wide state coverage and low action diversity, and the batch-adaptie threshold adapts to local coverage and multi-modality, being suited to datasets with limited state coverage and high action diversity.
4.2 ENHANCED FLOW ACTOR OPTIMIZATION
The typical offline RL actor optimization is done with the objective 2 That is, the objective is to maximize Q value while keeping near the behavior policy 3. One can apply this offline actor design principle to flow-based actor (Park et al. 2025). this case, the flow behavior proxy policy Big is learned as described in Section 2.2 Then, the objective (2) for the actual target policy mo can be implemented as follows (Park et al. 2025):
max Ee-D,aw ( A AB, [8)00(8) (s, =] m5
(1I)
where (s,s 2) and as 2) denote actions generated by Ty and By given (s, 2). respectively. Here, a (a, 2) is realized with velocity modeling (6) and integration (4) for full expressibility. However, the target policy my implementation is simplitied because the numerical multi-step integration based on velocity modeling makes gradient backpropagation complicated. Thus, Park et al. (2025) used one-step flow policy, which transports a directly to a via a flow as( ;;s, u = 1): =) Although this one-step flow policy Th has less expressibility than the multistep behavior proxy policy B.. it still has sufficient capability to express multi modaiimod,llty, shown in Appendix D.2 Thus, adopt this one-step flow policy as our actor too.
A A: + a.
In most cases of policy optimization via Q maximization with distance constraint like 11 including Park et al. (2025), simple Qs(8, a). just trained with the Bellman error Es,a.''D [(Qa(s.a) 7 (s, )]. is used, where & is the target Q network parameter. The rationale behind this is that the distance regularization term in (1]) attracts the policy the behavior support although the Q function is overestimated in the OOD region. However, when near-optimal ac- tions are rare compared to sub-optimal ones in the dataset, there a problem. Weak regularization in (IT) drives value maximization toward the OOD region on which unpenalized Q has overestimated values, yielding wrong actions. On the other hand, strong regularization induces the actor imitate sub-optimal actions mostly. In such cases, if we use weak distance regularization and a Q function well-penalized at the OOD region in [T]. then maximizing such a Q function yields good actions within the ID support. For such an idea to work, we need a Q function estimate precisely penalized at the OOD region. Therefore, we use the critic penalized with highly expressive flow behavior proxy proposed in Section We name this new actor-critic structure for offline RL based the flow model Flow Actor-Critt (FAC). will be shown that FAC sets new state-of of-the-art performance for difficult tasks of the OGBench in the experiment section soon.
5

---

Published as a conference paper at ICLR 2026
4.3 PRACTICAL IMPLEMENTATION
The practical implementation of FAC is provided in Algorithm three key components: a flow behavior proxy policy, a one-step flow actor, and twin crities to ensure stable Q value estimation (Fujimoto et al. 2018). The training procedure consists of two stages.
[ The proposed method comprises
In the first stage, we train a flow behavior proxy policy 3g. by eq. (6). After training the proxy policy, we evaluate behavior proxy density by eq. 7 for all state-action pairs in the dataset and store these evaluations along with the transitions. These stored evaluations as the dataset-driven threshold allow efficient identification of ID and OOD regions in the subsequent training stage.
In the second stage, we train penalized critics and a one-step flow actor concurrently. The critics Qo are trained by eq. (9) with target crities for stable learning. Specifically, the proposed critic penalization applies behavior- weights based on the behavior proxy density of actor actions and the stored evaluations the dataset. The actor my is trained by eq.
Algorithm 1:
Flow Actor-Critic
Initialize one-step flow policy network no, critic networks Q Qoz, target networks Q5, Q Qézr and flow
behavior proxy network 3. for each iteration do
Sample (s, a) ~ D, 2 N(O, I), Unif((O,
Update Be with eq. (6)
Compute {als) for the samples in D by eq. and store the evaluations D.
for each itevation do
Sample (s, (s, a, s')
D. ~~ ~NO 7) and compute e.
Update Qo. with eq.
99
E (1,2}.
Update Ta with eq.
Φe pds + (1 p)e:, E (1,2) for some
5 RELATED WORKS
Offline RL with Gaussian policies. Most methods with Gaussian policies can be grouped into two complementary approaches: actor regularization (AR) and critic penalization (CP). AR methods constrain the learned policy to the behavior policy in the dataset, such as TD3+BC (Fujimoto & Gu 202T) imposing a simple behavior cloning regularizer and SPOT (Wu et al. 2022) using VAE-based ELBO to induce high-suppon actions. Other related AR methods extract the policy through value- weighted regression (Peng et al. 2019) Nair et al., 2020 Lee et al. (20T), such as IQL (Kostrikov) et a 2021). On the other hand, CP methods, such as CQL (Kumar et al. 2020). penalize value estimates for policy actions, but the induced policy becomes overly conservative. To mitigate this, MCQ (Lyu al. 2022) introduces pseudo target value, EPQ (Ycom et al. 2024) uses state-dependent
Lee
penalty weights defined by VAE-based ELBO and a current policy, SAC-RND (Nikulin et al.. 2023) leverages RND--based anti-exploration bonus (Burda et al. 2018), and SVR (Mao et al. | (023) uses
the property of importance sampling. Based on these approaches, ReBRACT (Tarasov et al.) 2023)
shows that complementary use of AR and CP (Wu et al. 2019 Kim et al., 2025) yields strong performance when this joint approach is integrated with modern offline RL design choices.
Diffusion and flow-based policies. Expressive policies have recently emerged alternatives for modeling complex action distributions. The methods using such expressive policies can be categorized by whether they maximize their Q function explicitly or implicitly. For diffusion-baar
methods,
explicit approaches, such as CAC (Ding & Jin 2024) and SRPÓ (Chen et al. 2024), optimize Q-maximizing policies (Wang et al.] 2022 [ He et al.] 2003 Zhang et 2004).. Implicit approaches, such as IDQL (Hansen-Esttruch et al. 2023), either use value-war weighted regression (Lu et al. 2023
Kang et al. 2023) Ding et al. (0244) or apply aalue-weighted sampling (Chen et al. 2022 He et al. 2024). For flow-based methods, FQL (Park et al. 2025) trains Q-maximizing policy using the distance regularizer with a flow behavior proxy policy. Furthermore, Park et al. (2025) has shown that stable policy extraction for multistep flow policies is challenging: value- weighted regression (FAWAC), explicit Q Q-maximization (FBRAC), and walue-weighted sampling (IFQL). Another method, QIPO-OT (Zhang al. 2025), trains flow matching policy through value-weighted regression.

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation over 55 singletasks of the OGBench. For each task category, we report the final
performance averaged across its 5 singletasks, over 8 seeds (4 seeds for pixel-based tasks), with + indicating the standard deviation. Full evaluation results on the 55 singletasks are Table 3
Gaussian Policies
Diffusion Policies
Flow Policies
Task Category
BC
IQL
ReBRAC
IQLL
SRPO
CAC
FAWAC
FBRAC
IFQL
FQL
FAC (Ours)
antinaze-large cellmaze-large-aavelell narge-avvigatee
10.6
53.4
80.8
20.3
10.6
32.8
6.4
60.2
28.0
78.6
92.6125
antmaze-giaae carmaaz--iaat--ne giant-navigate
0.2
4.0
26.2
0.0
0.0
0.0
0.0
3.8
2.6
8.6
23.0a51
humanoidmaze- hatpanaanammiiiiiiiiiiieeeeeee
2.0
32.8
21.8
1,4
52.8
19.4
38.4
60.4
57.4
75.6136
humannidmaze large navigate
2.6
0.6
antsoccer-arenat wenneeeerrrren-nnviiee naviia-
1.0
8.4
0.0
11.8
1,0
1.8
12.4
16.0
33.2
60.2
67.7429
cube-single play
5.4
83.0
90.6
94.6
79.6
85.2
81.2
78.6
79.2
95.8
98.8114
cube-double-play
1.6
6.4
12.2
14.6
1,4
5.8
5.2
15.0
14.0
28.6
33.1
scene-play
4.6
27.6
40.6
46.2
20.0
39.8
29.8
44.8
30.4
55.8
71.3
puzzle-3x3-play
1.8
9.0
21.6
10.4
17.8
19.4
6.4
14,0
19.0
29.6
100.0:00
pazzle 4x4 play
0.2
7.4
14.0
29.2
10.6
14.8
0.4
13.2
25.2
17.2
32.3165
visual manipulation (pixel-based)
41.6
59,8
22.8
50.2
65.4
76.0s39
Average (state-based)
2.8
23.4
31.0
22.9
14.3
25,3
16.1
28.6
30.3
43.6
60.3
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Benchmarks. We evaluate FAC on two offline RL benchmarks: recently introduced OGBench (Park) al. 2024c) and standard D4RL (Fu et al., 2020). OGBench offers diverse goal-conditioned locomotion and manipulation environments, in which each task category consists of 5 singletask variants, which are relatively more challenging than the D4RL tasks. To align with standard offline RL, we use 55 reward based singletask variants and follow the 11 OGBench task categories. For D4RL, we use 9 MuJoCo tasks, 6 Antmaze tasks, and Adroit tasks. The details are described in Appendix G.I
Baselines. We evaluate FAC against a total of 17 baselines and categorize them by policy class: Gaussian, diffusion, and flow policies. For OGBench, we compare against Gaussian policy-based baselines BC, IQL (Kostrikov et al., 2021), ReBRAC Tarasov et al.] 2023), diffusion policy- policy-based
baselines IDQL (Hansen-Esttruch et a (203)), SRPO Chen et al. 2024), CAC (Ding & Jin 2024).
and flow policy-based baselines FAWAC, FBRAC, IFQL, FQL (Prk et al. (025). For D4RL, we
additionally compare against Gaussian policy-based baselines TD3+BC (Fujimoto & Gu 2021)
CQL (Kumar et al., 2020). MCQ (Lyu et al., (002), EPQ (Yeom et al.) 2024). SAC-RND (Nikulin
al. 2023). SPOT (Wu et al. 2022). and flow policy-based baseline QIPO-OT (Zhang et al. 2025).
The details of the basclines are described in Appendix G.2
Evaluation and hyperparameters. We report the final performance evaluated after 1M gradient steps for state-based tasks and 500K gradient steps for pixel-based tasks. For evaluation, the one-step flow policy trained by FAC samples one action at each time step. For training FAC, we mainly tune two hyperparameters: the critic penalization coefficient and the actor regularization coefficient X. All remaining settings (e.g., dataset-driven threshold schemes) are fixed per task domain. The complete configuration is in Appendix G.3
6.2 PERFORMANCE ON THE OGBENCH BENCHMARK
The evaluation results aggregated from 50 state-based and 5 pixel-based singletasks of the OGBench are summarized Table I and the full evaluation results are in Table Appendix FAC achieves the highest overall performance across all methods. FAC yields clear margin, compared to FQL among flow policy-basee baselines, CAC among diffusion policy-based baselines, ReBRAC among Gaussian -asicy-bsedd baselines. In particular, the comparison with Gaussian policy-based baselines highlights the benefit of expressive policies on datasets with multi-modal action distribution, while the gap to diffusion policy-based baselines and multi-step flow policy-based baselines (FAWAC, FBRAC, IFQL) reveals the practical difficulty of optimizing iterative generators. Leveraging a one-step flow actor enables both FAC and FQL to outperform other baselines. Furthermore, fusing penalization and actor regularization, FAC significantly improves performance over FQL, which is the actor regularization only.
critic
7

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation on 23 tasks of the D4RL. We report the final performance averaged over seeds, with + indicating the standard deviation. For MuJoCo datasets, we use the following abbreviations: m for medium, mr for medium-replay, me for mediun-expert
Gaussian Policies
Diffission Policies
Flo Policies
MuJoCo Tasks
TD3+BC
IQL
CQL
MCQ
EPQ
SPOT
ReBRAC
IQLL
SRPO
CAC
QIPO-OT
FQL
FAC (Ours)
halfcheetah-m
48.3
47.4
44,0
64.3
67.3
58.4
65.6
51.0
60.4
69.1
54.2
60.3+11
65.0a15
hopper-m
59.3
66.3
58.5
78.4
101.3
86.0
102.0
65.4
95.5
80.7
94.1
18.
91.9:
walker2d-m
83.7
78.3
72.5
91.0
87.8
86.4
82.5
82.5
84.4
83.1
87.6
77.2425
85,2:09
halfcheetah mr
44.6
45.5
56.8
62.0
45.9
51.4
58.7
48.0
49.3
55.4:27
hopper-me
60.9
94.7
95.0
101.6
97.8
100.2
98.1
92.1
101.2
99.7
101.3
49,8252
99.1
2alker2d-mr
81.8
73.9
77.2
91.3
85.3
91.6
77.3
85.1
84.6
79.5
78.6
53.1479
83.0a5s
halfchoctah-me
90.7
86.7
91.6
87.5
95.7
86.9
101.1
95.9
92.2
84.3
94.5
99.6a63
101.9:56
hopper-me
98.0
91.5
105.4
111.2
108.8
99.3
107.0
108.6
100.1
100.4
108.0
83.1-170
104.2-46
walker2d me
110.1
109.6
108.8
114.2
112.0
112.0
111.6
112.7
114.0
110.4
110.9
106.1
108.4.sa
Averuge
75.3
77.0
77.6
88.5
90.9
85.9
88.5
82.1
87.1
85.1
86.4
71.8
88.2
Gaussian Policies
Diffusion Policies
Flow Policies
Antmaze Tasks
TD3-BC
IQT.
CQL.
MCQ
EPQ
SAC-RND
ReBRAC
IQQI
SRPO
CAC
QIPO-OT
FQL.
(Ours)
umaze
78.6
87.5
74.0
98.3
99.4
97.0
97.8
94.0
97.1
75.8
93.6
96.0
98.5
umaze diverse
71.4
62.2
84.0
80.0
78.3
66.0
88.3
80.2
82.1
77.6
76.
89.0
93,5tao
medium- play
10.6
61.2
84.0
80.7
56.8
78.0
88.0.95
medium- medium--iverse
3.0
70.0
53.7
37,5
86.7
74.7
76.3
84.8
75.0
0.0
86.4
71.0
85,0a23
large playeppaya
0.2
39.6
15.8
2.5
40.0
43.9
60.4
63.5
53.6
0.0
55.5
84.0
90.0243
largo-diverse
0.0
47.5
14.9
7.5
36.7
45.7
54.4
67.9
53.6
0.0
32.1
83.0
88.0240
Average
27.3
63.0
50.6
46.4
71.0
61.0
76.9
79.2
73.7
35.0
70.6
83.5
90.5
Gaussian Policies
Diffusion Policies
Flow Policios
Adroit Tasks
TD3-BC
IQL
CQL
MCQ
EPQ
SAC-RND
ReBRAC
IDQL
SRPO
CAC
FQL
FAC (Ours)
pea-human
81.8
81.5
37.5
68.5
83,9
5.6
103.5
76.0
69.0
64.0
53.0
73.9
pen-eloned
61.4
77.2
39.2
49.4
91.8
2.5
91.8
64.0
61.0
56.0
74.0
103.2
door-baman
-0.1
3.1
9.9
2,3
13.2
0.0
0.0
6.0
3.0
5.0
0.0
5,3433
door-cloned
0.1
0.8
0.4
1.3
5.8
0.2
1.1
0.0
0.0
1.0
2.0
4. 4.1
hammer-human
0.4
2.5
0.3
3.9
-0.1
0.2
2.0
1.0
2.0
1.0
8.6254
hammer-cloned
22,8
0.1
6.7
2.0
11.0
11.1-112
relocate-buman
-0.2
0.1
0,2
0.1
0.3
0.0
0.0
0.0
0.0
0.0
0,6505
telocate-cloned
-0.1
0.2
-0.1
0.0
0.1
0.0
0.9
0.0
0.0
0.0
0.0
0,5004
Average
18.0
20.8
11.7
15.4
27.7
1.0
25,5
18.8
17.0
16.1
17.6
25.9
On navigation tasks, FAC achieves the best performance on 3 out of 5 navigation tasks. Although remains led by ReBRAC, indicating that a joint critic penalization and actor regularization approach can be competitive even with Gaussian policies. For higher- dimensional humanoid d tasks, FAC outperforms ReBRAC, suggesting that the expressive actor contributes in complex action spaces. In humano dma ze- 1 seseeennvigtttee FAC is slightly below IFQL on average but matches or exceeds this baseline on 4 of the singletasks (see Table 3 in the appendix for details.).
On state-based and pixel-based manipulation tasks, FAC outperforms all baselines across the board and achieves perfect success on puzzle -0teee-xx-ppaap In particular, the puzzle -ellel-pply tasks for solving lights-out puzzle are designed to test combinatorial reasoning and trajectory stitching, on these tasks, FAC attains 238% and 11 % performance improvements over the strongest baselines (FQL. and IDQL) on puzzle-3x3-pl ay and puzzle-4xe 4x4-x--4ppaa respectively. Moreover, FAC achieves the highest performance the pixel-based tasks, demonstrating that performance gains of the proposed method also hold in these higher-dimensional settings.
6.3 PERFORMANCE ON THE D4RL. BENCHMARK
The evaluation results on 23 tasks of the D4RL are summarized in Table Overall, FAC is competitive or superior across all three domains (MuJoCo, Antmaze, Adroit), indicating that the benefit of jointly penalizing the critic and regularizing the actor with the expressive behavior proxy policy extends beyond the OGBench and holds under diverse reward functions and state-action distributions. Among methods with expressive policies, FAC attains the best overall performance across all three domains.
On MuJoCo tasks with dense reward functions, it is susceptible to value overestimation for OOD actions. It is seen that FQL exhibits sensitivity to the value overestimation bias, while FAC matches
8

---

Published as a conference paper at ICLR 2026
cube-singles play-task2
hoppen-mediu-eggee
Fierle poyply
purde- bizti-Ixdplyyyarddd
Pattomanee
200
C.tm
Soconaliion
(a) arg max, ((s, a) evaluation
(h) Across a and X
(c) Performance under flow steps
antmsse-large
10
DWVVO BIKON
WAY KoL ORICK 0WKIEE
(d) Critic penalization weights and performance
(e) Performance of c schemes
Figure 3: Empirical analysis of components. (a) Effect of flow-based critic penalization (CP) on performance under N candidate actions. (b) Performances across actor regularization coefficient and critic penalization coefficient a. (c) Influence by the fidelity of flow behavior proxy under flow steps T. (d) Performance of weight function designs. (c) Performance under different schemes.
strong Gaussian policy-based baselines. This demonstrates that the flow-based critic penalization in FAC successfully mitigates the value overestimation.
On Antmaze tasks, as task difficulty increases from uma ze to medium to large, many baselines including the strong Gaussian policy-based baselines on the MuJoCo domain, diffusion policy-based baselines, and the multistep flow policy-based baseline exhibit pronounced performance degradation. In contrast, FAC maintains robust performance against difficulty and consistently improves over FQL.
On Adroit tasks that are notoriously challenging due to the high-dimensional action space and the sparse dataset support, FAC is comparable to the strong Gaussian policy-based baselines in overall performance. In particular, FAC attains substantial improvements on pen pen-clop oned and hammer -uuman tasks.
6.4 EMPIRICAL ANALYSIS
Effect the critic penalization. We investigate the impact of the proposed flow-based critic penalization of FAC in terms of OOD action sampling by replacing our flow-based penalized critic with an unpenalized conventional Q estimator (i.e., reducing to FQL). For this, we sample N E {1, 10, 30, 50) candidate actions a(s, 2) from the one-step flow actor with N i.id. for given and execute arg max, (s a) at each step for each method, with its best-performing configurations for X, a, and flow behavior proxy model. Note that as N increases, the chance of selecting an OOD action increases and arg max, Q0(s, a) with the unpenalized Q can check this. As shown in Fig. Ba] as N inereases, FQL exhibits significant performance degradation, showing that OOD actions are drawn even with the distance regularization. In contrast, our method maintains consistent performance with increasing N, suggesting that the flow-based critic penalization together with actor regularization is effective for handling potential OOD actions.
Sensitivity a and FAC substantially improves performance on pazzle-3x3-ppay--o0
Fig. 3b shows performance heatmap across critic penalization coefficient and actor regularization coefficient on this default singletask, with each cell presenting the mean + standard deviation over 8 seeds. With adequate À > 1, performance is robust performance becomes sensitive to a. These results demonstrate that jointly applying flow- critic penalization and flow actor regularization produces synergistic effect, enabling reliable support-aware flow policy optimization in the offline RL setting.
whereas with too small À 0.3,
Influence by the fidelity of flow behavior proxy. The step count of the Euler method directly affects the fidelity the approximated flow obtained by the flow proxy. When the learned flow represents almost linear trajectories from base noises 2o to data samples .1, single step count would be sufficient.

---

Published as a conference paper at ICLR 2026
However, for complex data distributions where the learned flow exhibit curved trajectories. higher curvature needs more integration steps (refer to Fig. in the appendix). As shown in Fig. 3c FAC attains high performance for T 2 10, demonstrating that sufficient fidelity of the flow behavior proxy is essential for the strong performance of FAC.
Impact of critic penalization weight functions. We further investigate alternative designs for the critic penalization weight w (s.a). For clarity, we refer to the original weight (eq. 8 as linear weight and introduce two alternatives defined as max(0, - (alss//) + 1)/ log 2), here 1/2 yield concave and convex weight functions, respectively, on [0, 1] of Bg/c. Fig. 3d presents the weight functions and the performance of FAC under the weights. For the tasks, the linear and concave weights achieve comparable performance, however, the convex weight leads to performance degradation. These results indicate that the linear weight is a simple but effective choice.
2 and
Impact of schemes. The choice of e is an important component of the critic penalization weight (eq- 8). We evaluate three dataset-driven threshold designs: (i) batch adaptive, (ii) dataset wide constant, and (iii) newly introduced batch wide constant, min(,..)~ Sug (als), where B denotes a mini-batch. Fig. Be shows that the three schemes yield distinct performance on hal halfcheetah-med.. un-expert-v2 and armmaee-aarge-diveeses
whereas they
exhibits nearly identical performance on antmaze -eeeeeeeeiiii um- cammpplly-a 12 Based on these observa- tions, we set e as batch adaptive for all MuJoCo tasks and as dataset wide constant for all Antmaze tasks, in order to avoid excessive selection.
Online fine-tuning, We evaluate FAC in the offline-to-onling setting. Fig. 4 presents the results in this setting, where the first 1M gradient steps de-
OGBench (5 taskst
D4RL antmage 16 tacks)
D4RL adroit tasks)
note the offline RL training phase, and the subsequent 1M steps denote the online finetuning phase. We note that FAC reuses the hyperparameters used in the offline RL settings. The details for online finetuning are provided in Appendix G.3.3
1:M0
3.24
2.50
Figure offline-oo-nling results, averaged over seeds with standard deviation. The gray shaded region indicates the offline phase, and the online fine-tuning phase is left unshaded. Full plots on 15 tasks are in Fig.
We compare FAC with FQL, which is also competitive baseline. On the OGBench and D4RL Antmaze tasks, FAC achieves strong performance even in the offline-to-online setting. These results may come from the proposed flow-based critic penalization design. The critic penalization design allows to reduce or climinate penalization for confident actions, suggesting that FAC avoids excessively suppressing high Q-value neighborhoods near the dataset support boundary. This, in turn, may enable exploration in such high Q-value regions. However, on the D4RL Adroit tasks, FAC shows smaller performance gains during online finetuning, despite outperforming FQL in offline training. Due to the notoriously sparsity of the Adroit datasets, FAC during the offline training phase provides limited guidance about high Q-value regions, and the critic penalization may unintentionally hinder exploration toward these promising neighborhoods.
7 CONCLUSION
We have proposed flow actor-critic (FAC), which leverages flow behavior proxy policy jointly for critic penalization and actor regularization. The behavior proxy provides tractable behavior densities that yield confidence weights for critic penalization so that the resulting operator preserves the original Bellman operator in confident in-distribution region and penalizes value overestimation for out-of-distribution actions in complex and multimodal datasets. This behavior proxy also regularizes the flow actor, constraining it to the dataset support. Consequently, this joint approach enables flow- based actor-criti optimization that is reliably constrained within well-supported regions. Empirically, FAC achieves consistently strong performance across tasks of the OGBench and D4RL benchmarks, effectively handling out-of- oi-diittibttion actions and maintaining high true returns under diverse reward structures and state-action distributions.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
com Our code is available at htt.://hhhu..coooooooonoonhheaao wwwgseongchaee/eeee /FAC The details of the
practical implementation and experimental setup described in Section 4.3 and Appendix G.3
ACKNOWLEDGMENTS
This work supported in part by the Institute of Information & Communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS RS--2022-11220469, Development of Core Technologies for Toik-tre Reinforcement Learning for Commercialization of Autonomous Drones) and in part by Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea govemmeett (MSIT) (No. RS-2024- 00457882, AT Research Hub Project)
REFERENCES
Ryan P Adams, Jeffrey Pennington, Matthew Johnson, Jamie Smith, Yaniv Ovadia, Brian Patton, and James Saunderson. Estimating the spectral density of large implicit matrices. arXiv preprint arXiv: 1802.0335) 2018.
Alekh Agarwal, Nan Jiang. Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle. WA, USA, Tech. Rep, 32:96, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 160700660 2016.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023,
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky TQ Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. arXiv preprint arXiv:2207.044 04.00100,11,, 2022.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milhe and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:// httt:///gthhb. /ttpt/bttttubbtt wowp/jaaaa/aapap
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
Woohyeon Bycon, Giseung Park, Jongscong Chae, Amir Leshem, and Youngchul Sung. Multi- objective reinforcement learning with max-min criterion: A game-theoretic approach. In The TThirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Jongseong Chae, Seungyul Han, Whiyoung Jung, Myungsik Cho, Sungho Choi, and Youngchul Sung. Robust imitation learning against variations environment dynamics. International Conference Machine Learning, pp. 2828-28282 2852.. PMLR, 2022.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209, 14548, 2022.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In The Twelfth International Conference Learning Representations, 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in newral information processing systems, 34:1 10044-10777. 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.

---

Published as a conference paper at ICLR 2026
Sungho Choi, Seungyul Han, Woojun Kim, Jongseong Chae, Whiyoung Jung, and Youngchul Sung. Domain adaptive imitation learning with visual observation. Advances in Neural Information Processing Systems, 36:44067-44104, 2023.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion- based reinforcement learning via q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945-53396 2024.
Zihan Ding and Chi Jin. Consistency models rich and efficient policy class for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Laurent Dinh, David Krueger, and Yoshua Bengio. estimation. arXiv preprint arXiv: 14 10.85 16, 2014.
Nice: Non-linear independent components
Laurent Dinh, Jascha Dach-Diiksttin, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint 18100.1/ 2018.
Albert Gu. Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces transfer skills with reinforcement learning. arXiv preprint arXiv:1703.02949,
2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. Pmlr, 2018
Philippe
Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. ldql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Longxiang He, Li Shen, Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffeps: Diffusion model based constrained policy search for offline reinforcement learning. arXiv preprint 10.00323 2023.
Longxiang He, Li Shen, Junbo Tan, and Xueqian Wang. Alignigl: Policy alignment in implicit q-learning through constrained optimization. arXiv preprini arXiv:2405. 181887 2024.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). 1rxiv:1606.08411 2016.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adyances in neural information processing systems, 33:6840-6851, 2020.
12

---

Published as a conference paper at ICLR 2026
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications 1989.
and Computation, 18(3):1059-1076
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 67195-67212, 2023.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. In The Twelfth International Conference on Learning Representations, 2024.
Jeonghye Kim, Yongjae Shin, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Youngehul Sung, Kanghoon Lee, and Woohyung Lim. Penalizing infeasible actions and reward scaling in reinforce- ment learning with offine data. In Forty-second International Conference Machine Learning, 2025.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 69800, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1 13:1179-1191 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-arr, pp. 43-73. Springer, 2012.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120-6130. PMLR, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
Yaron Lipman, Ricky TQ Chen, Heli Gen-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 10.027,2 2022.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben- Hamu, and Itai Gat. Flow matching guide and code, 2024. URL
https: http//aaxvvvc. org/ abs /tt.///2../2222.5522
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209 200002209.0000,,, 2022.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling offline reinforcement learning. In International Conference Machine Learning, pp. 20822222282202 22855.. PMLR, 2023.
Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, and Liqiang Nie. Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl. Advances in Neural Information Processing Systems, 37:22827-228499 2024.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
13

---

Published as a conference paper at ICLR 2026
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 40587-40609, 2023.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-359 2005.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online finc-tuning, Advances in Neural Information Processing Systems, 36:62244 622669, 2023.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International conference on machine learning, pp. 26228-26244
PMLR, 2023.
Giseung Park, Woohyeon Byeon, Seongmin Kim, Elad Havakuk, Amir Leshem, and Youngchul Sung. The max-min formulation of multi-objective reinforcement learning: From theory to a model-free algorithm. In International Conference on Machine Learning, pp. 19616-39642. PMLR, 2024a.
Jongeui Park, Myungsik Cho, and Youngchul Sung. Empo: A clustering- based on-policy algorithm for offline reinforcement learing. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024b.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking
offline goal-conditioned rl. arXiv preprint arXiv:2410.20092. 2024c.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In Forty-second International Conference on Machine Learning, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression. Simple and scalable off-policy reinforcement learning. arXiv preprint urXiv: 2019.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Scokin Sco, Byung-Jun Lee, Jongmin Lee, HyeongJoo Hwang. Hongseok Yang. and Kee-Eung Kim. Mitigating covariate shift in behavioral cloming via robust stationary distribution correction. Advances in Neural Information Processing Systems, 37: 17:100917-10920,, 2024.
Yongjae Shin, Jeonghye Kim, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Youngsoo Jang, Geon-Hyeong Kim, Jongseong Chae, Youngchul Sung, Kanghoon Lee, and Woohyung Lim. Online pre-training for offline-to-online reinforcement learning. In Forty-second International Conference Machine Learning, 2025.
Yongjae Shin, Jongseong Chae, Jongeui Park, and Youngchul Sung. Flow matching with injected noise for offline-to-onling reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. pmlr, 2015.
14

---

Published as a conference paper at ICLR 2026
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1.Xiv:2011.134560 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: an introduction, 2nd edn. adaptive computation and machine learning. 2018.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach offiin reinforcement learning. Advances in Neural Information Processing Systems, 36:1 1592-11620,, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In Interna- tional conference on machine learning, pp. 23484-23524 PMLR, 2022.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint 1xXi1:2208.061933 2022.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimiza- tion for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 12778-1121,, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1911::9111.13360 2019.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint 1rXiv:21 2021.
Taku Yamagata, Ahmed Khalil, and Raul Santos- Rodriquz. Q-learning decision transformer:
Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, pp. 38989-39007. PMLR, 2023.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
Junghyuk Yeom, Yonghyeon Jo, Jeongmo Kim, Sanghyeon Lee, and Seungyul Han. Exclusively penalized q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:1 1 3405-1134335 2024.
Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas Schön, and Per Mattsson. Entropy-regularizee diffu- sion policy with q-ensembles for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:98871-98887 2024.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy Energy-weightted flow matching for offline reinforcement learning. In The Thirteenth International Conference Learning Representations, 2025.
Zifeng Zhuang, Dengyun Peng, Jinxin Liu, Ziqi Zhang, and Donglin Wang. Reinformer: Max-return sequence modeling for offline RL. In Forty-first International Conference on Machine Learning, 2024.
15

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
The proposed method relies on the fidelity and accuracy of density evaluation of the flow behavior proxy policy for the underlying behavior policies. When the behavior proxy fails capture the multi-modal dataset distribution, the critic penalization with the flow behavior density can suppress values inappropriate regions, and the distance regularization for flow actor optimization can drive the flow actor toward the regions. In addition to this sensitivity, the method provides no mechanism for beneficial exploration, leaving its applicability to online RL. uncertain. These limitations suggest future extensions such as robust learning of the behavior proxy (Liu et al. 20222) to enable reliable support-aware policy improvement, and efficient exploitation strategies (Haarnoja al. 2018 Shin
al. 2026) for flow policies.
Moreover, our evaluation limited to simulated benchmarks, and we have not evaluated the method on real-world tasks. Real-world deployment introduces additional limitations, such as inherent stochasticity in environment dynamics (Morimoto & Doya 2005), the possibility of leveraging data from different domains due data scarcity biased data collection (Seo et al., [2024). and
the need to balance multiple conflicting objectives in reward design (Yang et al. 2019 Thus,
extending the method robust RL (Pinto al. 2017: Wang & Zou [022 Chae al. 2022), domain
adaptation (Gupta et al., 2017; Choi et al., (223), and multi-objective RL (Park ct al. 2024a Byeon
al. 2025) to handle these real-world challenges remains an important direction for future work.
B THE USE OF LARGE LANGUAGE MODELS
We used large language models to refine and polish writing.
16

---

Published as a conference paper at ICLR 2026
C BEHAVIOR CLONING MODELS
In Fig. T compare the flow behavior proxy model against other behavior cloning (BC) models. Specifically, we investigate data sampling as well as log-density or Evidence Lower Bound (ELBO) evaluations when employing flow matching model (Lipman al., 2022), simple Gaussian
probability model, a conditional VAE (Kingma & Welling 2013 Sohn et al. 2015). and diffusion
models with T {10,50) denoising steps (Ho et al. 2020
C.I
FLOW BEHAVIOR PROXY MODEL UNDER DIFFERENT STEPS NUMERICAL INTEGRATION SOLVER
Ground Truth
Flow Matching (T=3)
Flow Matching (T=5)
Flow Matching (T=10)
Flow Metching (T=30)
Flow Matching (T=50)
GT. log-density
FM (T=3): log-density
FM (T=5): log-density
FM (T=10): log-density
FM (T=30): log-densily
FM (T= densily
-10
-05
66
85
00
-5
60
10
05
35
Figure 5: Flow behavior proxy model with different step counts of the Euler method.
We investigate how the fidelity of the flow behavior proxy affects both sampling quality and density estimation. The flow behavior proxy is trained by flow matching objective 6 meaning that we can use different step counts of numerical integration for both data sampling and density estimation. We use the Euler method as the numerical integration solver, and the step counts directly influence the fidelity of the learned flow. When the flow matching model learns a nearly linear trajectory from base noise To to data sample 21, the single step count would be sufficient. However, when the underlying data distribution is complex and the learned trajectory exhibits curve, higher curvature requires larger step counts of the Euler method for sufficient sampling and density estimation.
We evaluate sampling and density estimation quality under different Euler step count T (3. 5, 10, 30, 50). Figure 3 shows the generated samples (top row) and log-density estimates (hottom row) across different T. When T is below 10, sampling quality degrades significantly, and density estimation with smallest 7 = 3 fails to separate ID and OOD regions, making it unsuitable for use in the proposed critic penalization. a level of fidelity with sufficient recovery of the multimodal distribution and accurate density contrast between ID and OOD regions. The results demonstrate that moderate number of Euler step (e.g.. T= 10) achieves strong fidelity required for density-based critic penalization.
contrast, for T > 10, both sampling and density estimation reach
C.2 IMPLEMENTATION
Dataset. We construct a dataset by generating a single state, = [0.5, -0.5,0050 0..,0.5,-00.33 -0.5]. and 2- dimensional actions from Gaussian mixture with four modes, 1 1 / Σ 0.0087).
where μ4 E ([-0.5, -0.5), (-0.5, 0.5), [0.5. -055, (0.50.55],
Network architecture. We use [512, 512, 512, 512]-sized MLP for all neural networks. For the flow matching model, we implement one neural network: velocity prediction network that takes the state Gaussian noise E Rd and flow time E [0, 1] and then outputs velocity estimate
E Rdd, For the Gaussian model, we use one neural network: Gaussian probability model, A (M(4l1g, oe). For the conditional VAE, we implement two neural networks: (1) Encoder network that takes the state and one action a then outputs latent variable 2, and (2) Decoder network that takes the state s and latent variable then outputs action estimate. For the diffusion models, use one neural network:
17

---

Published as a conference paper at ICLR 2026
conditional prediction network that takes the state 8, diffusion time and latent variable R and then outputs e-estimate.
C.3
TRAINING OBJECTIVE AND ACTION SAMPLING
Flow Matching model. We train flow matching model with the flow matching objective (6) and restate it:
min (.
[1les 8, u) - (a =)1[
For action sampling, solve the following integral equation using the Euler method with 10 steps:
a a= =+ [' cuddu = 2 + S/' us u,)uu,
u)du,
where au = Jo vedt, and ~ NOO,,)).
Gaussian model. A Gaussian model is trained by maximizing log-likelihood: max Es (honDDlloppeet [logp(a[s) Ex ((luoaa(a)ssss (o))))
We can easily sample actions from the Gaussian model.
Conditional VAE model. We train Encoder and Decoder models, 9s (z]s, a) and (e 2). respec- tively, by minimizing the ELBO of log-likeling log-likelhhood (Kingma & Welling 2013). The objective is given by
max E, [logp(a]s)] 2 Ex Ezngo [log (a 8,0
(2)s
where the prior distribution p(z) is set to (),
For action sampling, we sample z NN, I). and feed the state s and 2 to the Decoder model.
Diffusion model. We train an e-prediction model implemented as in DDPM (Ho et al. 2020).
min (ee
[||e ca solvā,a+ + vI a,e,s, s,()
(12)
where = 3, and di II (tg. Following Xiao et al. 2021 as in (Wang et al. 2022 we
use the beta schedule 3r = exp
(Bmm
2N
For action sampling, we first sample noise or NN 1), and then iteratively generate actions through the reverse process of the diffusion model:
8
04-1 ag
co(s, at, VBee
Vas
a (1(1-_))) a,)
where (N, 7). Note that in the diffusion model, the denoising time t corresponds the data index ag, while f corresponds to the noise index ar, which is the opposite of the indexing convention in flow-based models.
C.4 LOG-DENSITY AND ELBO FOR ARBITRARY ACTIONS
Flow Matching model. We can directly evaluate the log-density using the Instantancous Change of Variables (Chen et al., 2018), which a property of Continuous Normalizing Flows.
logp(als) = log (2) + (.' d logg.(als))n du
Po (2)
(V.. (ayaai u u)du,
V...
where a, a- u dt, and the base distribution Po set to normal distribution. We also use the 10 step count of the Euler method. Note that is obtained from = a₁ due to the bijective property of Continuous Normalizing Flows.
Gaussian model. We can easily compute log-density.
logp(a log N (a)a(((oossss
18

---

Published as a conference paper at ICLR 2026
Conditional VAE model. Since the VAE model cannot compute log-density exactly, instead evaluate ELBO, which is lower bound of its log-density. Using a normal distribution as the prior p(z) and implementing both the encoder O, and decoder Po Gaussian probability models, ELBO can be computed in a straightforward manner:
log p(aas)) 2 Ex [logpo(als,, Dxi
Diffusion model. Diffusion model (Sohl-Dickstein et al. 2015) defines forward process that gradually perturbs data to noise through a Markov chain, and a reverse process that learns to reconstruct data by denoising procedure that inverts the forward process. In the diffusion model, its log-likelihood is optimized via variational lower bound (ELBO).
DDPM (Ho al. 2020 simplifies the ELBO by reparameterizing the reverse process the prediction objective (12) which allows the ELBO to be expressed in a tractable form while retaining
its variational interpretation.
In the RL setting, we define the forward process ((-1)) -aaa vI 3,0+_1. 3,1) and the reverse process mo(at-ila,, s) = (a po (s, a,, t), Eo(s, a,, t)), following -Sohl-Dickstein et 2013). We therefore compute the ELBO as follows:
Po (0n:
logp(als) 2 Eg log
q(a1:r ag. s)
= E, log po (ao a1, s)
Σ DkL(a(a(-1la, ao, s)loapp((aaa-ll s))
-
(13)
where p(aT) is normal distribution.
We already have the predefined beta schedule { the forward process as in DDPM (Ho et al., 2020):
trained e-prediction model, we can choose
pa Ia, 8) ((a-11 oes 1), Bel,,
where 3.
1-āa-1 Br. I-
Using notable properties of the diffusion model (Sohl-Dickstein et al. 2015)
q(arlas. s) = - (ax: Vārav, (1 - a.))) q(a, 10,, an, 8) = N (ar-1: (e, ao). 3,7)
where [tr(a,, 0n) = v 1-a; ao + (1- and the KL-divergence between two multivariate Gaussian distributions, DRI. (N E E Σ2)).
1 tr (r + (w2 -m1) u 2iz (#2 m1) d + log det (22)
(N1(AN))
det (E))
where d denotes the dimension of the action space, each term of the ELBO (3) can be computed as
dA Dogpo(anl) 8) 1og(2*B1)=
I1a0 - no(s, (1, 1)
2,31
1
DKL D ao pellpe(a-_1100 s)) [ ao) uo ac, 12,
2,,
DEL DrLlg(arlan) ))((((()) = (árrag ag da (ār log(1 arr))
where (8, o4, t)= = fou (at vT-a, co(s,
1
9: - ()).
19

---

Published as a conference paper at ICLR 2026
D LEARNED Q COMPARISON ON A CONTINUOUS-ACTION BANDIT PROBLEM
Ground
Thue Q value & estimette
LUP DOOSHH U007Y
Tro
11
-22
Action
Action
Action
Action
value
(datass))
region
Itue valua
Q-anEimate
CCL policy
polcy
(a) Dataset and True Q value
(b) Comparison of baselines and FAC
SVR ratio (7))
log
log
Thee D valure & estimate
FQL (A=1)
FOL (A=10)
1700
1000
300
000 100
000 Staps
Action
Acton
Sops
32.ps
(c) The ratio
(d) FQL under different
Figure 6: A comparison of FAC and baselines in continuous action bandit problem. (a) Dataset and true Q value of the problem. (b) Comparison of baselines and FAC. (c) The IS ratio in the critic penalization of SVR. (d) Results of FQL under two actor regularization coefficients À {1, 10).
In order to compare the accuracy of the learned Q function, consider concrete example in this section. We mainly consider CQL (an early work), SVR (a good method with Gaussian policy), FQL recent flow-based method) and FAC (our method) for learned function comparison.
D.1 THE CONSIDERED CONTINUOUS-ACTIO BANDIT PROBLEM
Since need to know the true action value function, i.e., function, for accuracy evaluation and it is difficult to know the true Q function for complicated RL tasks, we consider a continuous-actinn bandit problem, which is modified from the bandit example in Ch.2 Sutton Barto (2018). In the considered bandit problem, the action range is from -1 to 1, and each time we perform an action hit the desired points located -0.5 and 0.5, where the location -0.5 preferred. Thus, the true action value function is designed as Gaussian mixture and given by
((a) F(a) 25e 25(a+0.5)3/2 + 10e 2(a-).5)"
The true action value function is shown in Figure 6a As seen, the true Q value exhibits two modes: the left mode with high peak and the right mode with moderately lower value. The reward each time is noisy version of the true action value, i.e.,
r(a) = F(a) +
where & is zero-mean Gaussian noise with standard deviation 0.1. Note that due to the noise in reward, we cannot know the value from single trial, and exploration to find better action and exploitation to use the collected information are necessary for this simple bandit problem to obtain large cumulated reward over time.
We construct an offline dataset composed of data {(s. a, r)) sampled from two Gaussian distributions: one centered at 0.5 and the other centered at -0.3, and 80 of data from the sampling distribution centered at 0.5 and the remaining 20% of data is from the sampling distribution centered at -0.3, as show in Figure 6a Here, $y denotes the dummy single state of the bandit problem. The reason for this dataset construction is that want model the situation in which large portion dataset corresponds to suboptimal behavior and a small portion corresponds the near-optimal behavior.
Dataset generation:
20

---

Published as a conference paper at ICLR 2026
D.2 RESULTS AND COMPARISON
The comparison result is shown in Figure (b) and (d), where the blue solid line is the true Q function and the green dotted lines the learned Q functions of the considered methods.
We first compare the performance with representative conservative critic methods: CQL and SVR, which are based Gaussian probability models. Fig. 6bb shows the learned critic and actor for each method in the continuous action bandit setting.
FAC versus Gaussian Model-Baset Methods:
The learned critic of CQL significantly underestimates the Q value between the two modes of the dataset, resulting in the actor being confined to the suboptimal ID region. In contrast, SVR tends to overestimate the value in the OOD region. The learned critic overestimates the OOD region, which induces the actor to sample OOD actions. This result stems from the instability of the importance sampling (IS) ratio used the critic penalization term SVR, as shown Fig. 6c As a consequence of this instability, the penalization term in the SVR loss ceases to play a corrective role, and the learned critic becomes unstable during training]] contrast, FAC provides more accurate Q estimates in ID region and gradually reduces estimates in OOD regions, which leads the actor to concentrate on the well-suppo high-value region.
FAC versus FQL: Now, compare FQL, which uses only the actor regularization, and our FAC. As shown in Fig. 6d the one-step flow actor, which is used in both methods, captures the multi-modal action distribution, confirming sufficient expressibility critic learning. FQL tends to overestimate the OOD region, which leads to OOD action sampling under weak regularization and to over- ver-imitation of sub-optimal actions under strong regularization, as shown in Fig. 6d In contrast, FAC preserves more accurate values on the in-distribution (ID) region and gradually suppresses values in the OOD region. Hence, the actor concentrates the well-supported high-value region even with weaker actor regularization À = 0.1, as shown in Fig. 6bb These observations demonstrate the synergy of the combination used in FAC: how flow-based critic penalization complements flow actor regularization by aligning critic learning with dataset support while retaining multi-modaa expressibility.
this problem. The key difference exists in
D.3 ALGORIHTM
We train all methods by adapting them to the considered bandit setting, where Qg is regressed from noisy reward r, similarly to Ball et al. (2023). For the flow behavior proxies of FAC and FQL. we behavior flow proxy 33 trained via the flow matching objective (6). We use the Euler method with 10 steps as an integral (ODE) solver for action sampling and evaluating behavior proxy density from B... For the behavior proxy of SVR, we use a Gaussian behavior cloning model 3gg trained via maximizing log-likelihood the dataset. The actor and critic objectives of the methods are presented below.
FAC. The objectives of FAC are given by
max ENND EaNTT [Q)(s,a]] AEZwN((n)) [llao(s, 2) as 2) [)]] R0
min a EDND art 1 (s(a))((ssa) + EeADD [(((s,)) a) - -
Qo
FQL. The objectives of FQL are given by
F EAND EANTO (Qo(s, À (ew)(( [llao(s,)) - (s, =)]]]]
max To
min E BRAND (((s,a))r(aa)) Qo
'To stabilize training. we follow the official open-soure implementation of SVR and clip the critic penaliza- tion term to the range -0' 10 Despite this practical adjustment, the IS ratio still exhibits values near the clipping threshold, limiting the effectiveness of the penalization term.
21

---

Published as a conference paper at ICLR 2026
CQL.
The objectives of CQL given by, max EDDD amme (Q, a) 17log [a(a
min EDDD [Baws [0 a)] EanD [)(( + Es [Q((),)) r(a))²
Qo
SVR. The objectives of SVR are given by,
max Rew, omns (Q)(s,a)
{(a|s) (Q(s, a) Qmin)
min a EOND Bas [(Q(s,a) - Qmin) Be-D Q+
Bu(als)
+ E D ((((s,a)-r(a))
where Quin = minp r(a)/((aa))))) (1)1))))---)) and we set to the Gaussian model with the same mean as my and larger variance, recommended in SVR (Mao al. 2023).
Network architecture. We use [256, 256, 256]-sized MLP for all neural networks. For all methods, the critic network Qa that takes state-action pair (8o, a) input and outputs scalar estimate of the true action value. For FAC and FQL, we use the behavior proxy policy 9g. that takes the dummy state so, Gaussian noise and flow time E [0, 1] input and then outputs the velocity field un- The one-step flow policy Ty that takes the state s and Gaussian noise & and then outputs a single action For CQL, we use the Gaussian probability policy that takes the dummy state so. For SVR, we use Gaussian probability models, which takes the dummy state so and outputs a action and its log-density, as the policy and behavior proxy, respectively.
22

---

Published as a conference paper at ICLR 2026
E THEORETICAL ANALYSIS
Proposition 1. Let 8 be the underlying behavior policy, be our proxy for B, ι be the learned actor, and Q be the value function of n. Consider the original Bellman operator T (=((s,)) a) r(s, a) 7lB.ss (P-laa) a' [Q(s,, (''',,0 d'} Then, in the tabular setting without function approximation, the objective ( yields the following operator:
(ss
if B(als) 2 c, (((ls)
- (a.ai(aaa)))) P(a
(s,a)
(=)(s,0)
if B(als) <, 3(als) (10)
if B(a|s) =0.
unless ((als) and w
(s. w
simultaneously.
Proof. In the tabular setting, replace D with B. Then, the objective rewritten as
can be
a E EgnD, and
w (s.a) Q(s,a) + BOND, (((s, T
T(als) a(s,a) (Q(s,,)) -TQs..))
=B ang
a a (s,
(14)
B(a)s)
We set the derivative of 14h to zero:
3 (s,a)t(als)
+
0,
B(als)
leading to the solution Q2 a), given by
" (s)
Q"(s, T -
(()((()))))
a
2 B(a(s)
Now consider three cases:
i) When 3(ass) 2 and B(als) > 0, "D (x, = 0. So, T = =T"((,
ti) When B(als) and B(als) O, (s, > 0. So, T(A T=Q(s,a)
w (e.o)x(aa)))
B(ala)
fii) When (a) s) = 0, the ratio ([als)/B(ass) = oo for any a with (s,a))((aa)) > 0. Then, its negative becomes oc.
The only case requiring special care that π (als) and B(als) = and 8h (s, a) = simultane- ously. The ratio w a)/B(als) may cancel out for ((als) 0 and w a) = O. But, it is highly
unlikely that we have B(als) = 0 and w wo,ss a) simultaneously by the definition of the weight S (s, a).
With the operator Tpx derived above, we now show that the policy (actor) æ induced by the policy improvement and policy evaluation of FAC is cunport-constrained policy, and that the Q-function (critic) under the policy yields unbiased Q-value estimates in high confidence regions of 9 > e. while providing under-estimated Q-values in low confidence regions of 3 < e.
We first define the Kumar al. (2019;; Wu et al. (2022); Mao et al. (2023).
policy, which plays a central role in our analysis, following
Definition 1. The support policy class 11 is defined as
1I (als) = 0 whenever B(als) = 0}, where 3 is the underlying behavior policy of the dataset.
23

---

Published as a conference paper at ICLR 2026
Proposition 2. For any support-constrattned policy n, the support of 3. Thee is cycontractionn operator in the Cx norm.
Proof. By Proposition
1 already have
[)) a a (1 B(als)
Let Qi and Q2 be two arbitrary functions.
For any (s, a) s.t. B(als) > and B(als) 2 ie., w (8, a) = 0, we have
Q1, Texc Q2(s,
= (T=QI(s, T2 a)l
= I/lBe'' P(1)))=)()))))))aaa((11 [Q1 a') - (2(s''
I
QQ a') - Q2(5',a)))
S >111 - -221oo
For any (s, s.t. B(als) > and B(als) < ie., w (o, > 0, we have
= T (s) 1 (s. (s.a))(als)
- Q2 e))
B(als)
-T"Q((s,a) + a (s) 2
B(als)
= =17 T Q2((saa)
S - -221
For any (s, a) s.t. (()ss) 0, by Definition [ x(als) 0, meaning that we can consider only the support of the underlying behavior policy B.
Therefore, on the support 3, || Q: (s, a) The (2 a) I sQQi
Q21loo where y [0, 1).
Theorem 1. For any support-constrained policy n,, the fixed point Q of Th is
(s, (s, B(als)
((s,a)=Q((s,a) 2 2 7)
on the support of B
where Q" (s, a) is the action value function under the policy defined Section 2.1 and d= d' a) the normalized discounted state-action distribution (occupancy measure) induced by n. We further have the following:
(i) The fixed point Q" provides unbiased Q-values for high confidence actions with Malss 2
i.e.. w (s. O.
(ii) The fixed point Q" provides under-estimated (conservative) Q-values for low confidence actions with B(ass) < Le., w (s, a) O.
Proof. By Proposition 2 Tee converges to a unique fixed point. That is, for any support-constrained policy n, the fixed point satisfies Q TT QT. Furthermore, in Proposition 2 consider only the support B.
To represent T in vector form, in prior works (umaa al. 2020 R as the vector of reward function, and pr (s, a, s', (, := P(s S, a s as the transition matrix on the state-action pair induced by n. Then, the fixed point equation is written as follows:
Mao et al. 2023 define
24

---

Published as a conference paper at ICLR 2026
Case (i): For any (s, a) s.t. B(als) ie, (s,a)
(s, - ( 1 Q-1 (s,0), Q (s,a) - - -P-)- (8, a) (s,
(0)
where (a) holds due to Proposition
| in the main paper. Thus, the fixed point provides unbiased
Q-values for (s, a) s.t. B(a|s) 2
Case (ii): For any (s, a) s.t. B(ass) < ie. (s) #0.
(² The Q" (5,a) (0) T" a w (s.a)
B(als)
" (s. a) (alss
= = [-]
(8,
a
2
B(a|s)
= (s, = (I (p"--
R a
w π
(s.
2
3
= (s, = Q" (s, - a
(1 - pp-)
(s,
2
where (a) holds due to Proposition in the main paper. The matrix (1 - 7px)- is the unnormalized discounted state-action distribution induced by the policy n. i... (s, a) -?) (refer Lemma
1.6 of Agarwal et al. (2019).. So, we have the following:
(= = Q* (s. a dd (s,
" (s)
$ Q (s.
(1-
B(als)
Thus, the fixed point provides under-estimated conservative Q-values for (s, s.t. B(als) n
Now we want to show that a policy т induced by FAC is support-constrained policy to apply Theorem [ For this we make the following assumption. Assumption 1. The well-trained flow behavior proxy in FAC exhibits a support narrower than the support of the underlying true behavior policy, i.e., supp(3) (8) C supp(3)
In practice, because the dataset has finite cardinality, i.e., [D] co, the support of the dataset is narrower than that of the underlying behavior policy 3, i.e., supp(D) S supp(33. Since a flow behavior proxy trained on such dataset inevitably fits to this narrow support, Assumption is valid. Theorem 2. Under Assumption 7 the policy π induced by Eqs. (9) and (77) in FAC is a support- constrained policy. Therefore, the learned Q-fanction under the policy x. has unbiased Q Q-values for all a) within high confidence region, i.e., M(ass) 2 and under-estimated Q- Q-values for all (s, a) within low confidence region, i.e. Baass) <c
Proof. The objective 11 of policy improvement in FAC has the following relation (Park et al. 2025
aawt (Q* a)) > (...D. [llaa(s,2) (x, < EAnD.an Q*1 (s, a)] EgwD (2(
(15)
where Q is the learned Q Q-function under the policy x, and W2( 3)2 is the squared 2- 2-Waseerseiin distance between and B.
For ease of exposition, we convert the unconstrained optimization problem 15 to the following constrained one, for some 5 2
max E,-D.ar as)
(16)
s.t. W2( 3)2 < 8.
This constrained optimization problem for policy improvement leads to the following observations:
25

---

Published as a conference paper at ICLR 2026
1. The constraint of the problem (16) offers incentive for the policy to allocate proba- bility density on actions outside the support of the behavior policy B, as ensured by the Assumption I
2. The objective of the problem (16). which yields a greedy policy whose Q value becomes -oc outside the support of the underlying behavior policy, according to Proposition [ derives the policy behavior policy B. If и assigns a positive density to those actions, i.e., ([ass) >> 0, then the objective becomes
assign zero probability density actions outside the support of the
EgwD, ave (s, -
J.ED,DEA (sa as)aa -00.
Thus, the greedy policy have to assign zero probability density actions outside the support of the underlying behavior policy B.
Based these observations, conclude that the policy learned by FAC is support-constrained policy, i.e., π E nT..
Since the policy æ FAC is a support-constrained policy, by Theorem the learned Q has unique fixed point and provides unbiased Q-values for all (s,a) within high confidence region, i... B(als) 2 6, and under-estimated Q-values for all (s, a) within low confidence region, i.e., ((als)
26

---

Published as a conference paper at ICLR 2026
F ADDITIONAL RESULTS
F. OFFLINE RL RESULTS
Table presents evaluation results on the 55 singletask variants in the OGBench. For each task category, select the best hyperparameters on its default task and evaluate the remaining tasks with the same ones. For pixel-based tasks, we reuse the hyperparameters selected for their state-based tasks; for example, vi sua leube-suueesseee lagge -vv is evaluated us- ing the hyperparameters selected for cube-ssinge single saleeppllllssseeeeeeeee seasss--nnglsss-eeeeeee The detailed experimental settings are provided in Appendix G.3
Table 3: Full offline RL evaluation results the OGBench. (*) indicates the default task in each task category. We report the final performance averaged over seeds (4 seeds for pixel-based tasks)
with + indicating the standard deviation.
Gaussian Policies
Diffusion Policies
Flow Policies
Tisk
BC
IQL. ReBRAC
IQQL
SRPO
CAC
FAWAC
FBRAC
IFQL
PQL
FAC (Ours)
anrmaze- large- navigate wineee-iinnllll-iinnneeeneeee laggeessk-kkkk--ddll
0
48
91
0
0
42
70
24
80
94.0-30
antmaze large navigate singletask task2-v0
35
antmare matmare-large large mavigate--ingl singlliask-taskeee
29
72
51
26
3
49
12
83
52
93
97,5
antmare-larg" pelllag---aviiieeee wiw.iigt---iinllessaaa
84
45
37
anrmaze-large navigate-We gingt---inlessss--natttee
10
90
2
76
83
96.0-18
untmare-gant-
27
0
anrmaze- glante----tnn novi movigate--ing wwwgggi-iinglliis-asss2-2
16
0
37.5
giant singleeeislllllkauuaae.ea..
34
0.5
antmare giant navigate navigate-ingleeat singletak isskt-t--
5
14
20.0a179
giant mavigate-sing gitll--iglleeek--oe (ask---i
19
49
0
13
50.5 16
humanoidmaze medium navigate ting mxk11~00
16
25
19
71.5
humancidmae medium--uviggit singkttssk-tok2-11
18
40
76
85
94
88 O1
humanoidmaze mediimm-uvvigtt--eeee-e- king
95.5
humanaidmaze mediaum navignte sing ttggllaaskktttkkk-
15
25.0a00
humanoidmaze medium navigatle
humamoidmaa: large mavigate tanglllttssktskkki 1-V0l
2
0
15,0
humanoidmaze- parpuree--r--ww-e wetgggg-iiglllell-eaeeeee
humanaidmaze large- navigate singletnsk-aaskk..vo
10
2014
humancidmaze lagg navigate singletask task4-v0
4.5a112
homaniddmaa-- larg mavigate singletask-tnkk---
2
1.5a21
navigute singletask task1 -v0
44
17
82.0.4m 77
antsuver aoeeeerr-ee---nnn mrem--ivvgalee
www.rrrie--iinnniitaaeeee singltask 1-002-00
15
88
93.5
soceer-arena-nn wwwaaa-revviiiaee www.rrrrvvviiieeeiiiii saaaaa-iii----inneeeeeeee singletskkkttkkkk...
0
16
62.5
ancsorcer arena navigate singlerask vxl
0
D
0
15
47.5 36
cabe-single- plaay-iing ind---innntatk-tiii1l1
cube single tlay-inngktunkktt222000
85
92
96
82
80
$1
83
73
97
100.0.00
singling play singktask-tas..
cobe-single hany--inngksskk-ttkkkkkk
92
V3
70
91
7V
79
79
98.5 94
cahe-single- paapple-pla--inieeeeee play--innggeeadk-ttaa singlleaack-ttsss...00
76
cube double playll-ppa-aaaae singlennnskkssssiee
45
39
47
cube dou-le play singletask task2vv0
36
37.5
17
22
31.5
cahe cahe-dawhle-plan sing--ingleettt--aaaaa
4.0-37
cube-double-poyy singl--sinndsd--ssse
17
19
32,5
19
M5
1000
914
100
96
98
1000
10000-00
seene seene-play- play--ingllyyedde (e
seene play singletask 42sk3-v0
32
55
94
49
38
78
54
98
97.0a2a
svene play -in-ssittottttsskfffffo
sene-play--ingleeenek.. 41--0
D
0
0
1.5+42
puzde 3t hall-------pll--snleesss-ee
52
puzole 3x ponle--x--plly--nnneettee 2-pply-sinngeetakk-ak222--
0
2
16
100.0-00
pozzle-3x bagfeexk...lly--iiggensssssk.s..
100.0.c0
puzzle 3xc bac-ppay---iii slayssiinllessskkssk4d00
100.0.co 16
puzaee-333 3xl-play singlsesiinlltsstssss----
0
100.0200 16
puzle-1s4 muzle-ts--play-snnggeee hes-ehlyyy-inngooll----11--1111
26
48
52.00201
puzae-4xx sadee-xx--paay baa----aaa-sinleeeeeee sus-pppy--nngllesss---22---
14
0
7.5 16
muzle-Ac4-play wew.-ee-ly--nnnoonnl-uassa.
15
34
21
20
62.0 18
puzzle-4e4- weaeee----plly--- plaay-singgeesskk-aad-----
10
26
21
35.04154 11
puzzle-Kxe 4x4 saag-singettekkcaakee
visual cube single play singletask tu
70.0
83.0
$5.0
49.0
$1.0
82.0a12
visual-cube double singllyl singletas--tkkkkk tusk-ll
shunt-scene-play weaa---ee---p--snnniieee pln-splllt--ssttddoooo iigletick-task 1-00
97.0
98D
46.0
86.0
98.0
99,0-30
visual- 312 play--ingleteek-ttu taskivl
88.0
7.0
100,0
95.0.10
visual puzzle visual-puzzle
4x4 play singletask uski-vd
260
0.0
8.0
33.0
56.01113
27

---

Published as a conference paper at ICLR 2026
F.1.1 COMPARISON WITH SEQUENCE MODEL BASED OFFLINE RL METHODS
We compare FAC against offline RL methods using sequence model architectures: Trans- former VVaswani et al., 2017) and state space model (Gu et al. 2022). Unlike Multi Layer Per- ceptron (MLP) based policies, sequence model based policies can condition on extended state-action- reward histories. For this comparison, we report results in the original papers and compare against Transformer-basee based methods Decision Transformer (DT) (Chen al. 2021), DC (Kim al. 2024), QDT YYamagata et al. 2023), Reinformer (Zhuang et al. 2024), and state space model based method Decision Mamba (DM) Lx et al. (2024).
On the D4RL MuJoCo tasks, FAC outperforms all sequence model based methods in terms of overall score, despite without explicit history conditioning. On the D4RL Antmaze tasks, FAC also outperforms Transformer-based methods. Notably, Reinformer exhibits strong performance on the simpler umaze tasks but degrades sharply the more challenging medium tasks, whereas FAC maintains high performance across medium and large tasks.
Table 4: Comparison with sequence model based baselines. For MuJoCo datasets, use the following abbreviations: m for medium, mr for medium-replay, me for medium-expert.
Transformer based
SSM based
MLP based
MuJoCo Tasks
DT
DC
QDT
Reinformer
DM
FAC (Ours)
halfcheetah-m
42.6
43.0
42.3
42.9
43.8
65.0=1.5
hopper-m
67.6
92.5
66.5
81.6
98.5
91.943.9
walker2d-m
74.0
79.2
67.1
80.5
80.3
85.2.109
halfcheetah-ni mr
36.6
41.3
35.6
39.0
40.8
55,4
hopper-mr
82.7
94.2
52.1
83.3
89.1
99. 1 =009
walker2d-mr
66.6
76.6
.8.2
72.9
79.3
83.0.15.8
balicheetah- me
86.8
93.0
92.0
93.5
101.9=5.6
hopper- me
107.6
110.4
107.8
111.9
104.2+44.6
walker2d-me
108.1
109.6
109.4
111.6
108.4
Average
74.7
82.2
78.8
83.2
88.2
Transformer based
SSM based
MLP based
Antmaze Tasks
DC
QDT Reinformer
DM
FAC (Ours)
umaze
85.0
84.4
100.0
98.5=3.0
umaze-diverse
78.5
90.0
93.51.60
medium-pla
13.2
88.0=9.6
medium-dived
10.6
85.0=7.3
large-play
90.014.3
large-diverse
88.0=600
Average
90.5
28

---

Published as a conference paper at ICLR 2026
F.2 OFFLINE-TO- ONLINE RL RESULTS
7 shows the offline-to-onlne results on the 5 default singletask variants in the OGBench and tasks in the D4RL Antmaze and 4 tasks in the D4RL Adroit. In all result plots, the first 1M gradient steps denote the offline training phase (the gray shaded region), and the online finetuning phase starts after 1M steps. For each tasks, FAC reuses the same hyperparameters used in the offline RL evaluation, whereas FQL uses hyperparameters specified for offline- to-oiine evaluation in its original paper (Park al,, 2025). The detailed experimental settings for the offline-to-online evaluation are provided in Appendix G.3.3
FAC
FOL
humanoldmaze medium
antsocce-aremaa
cube double
scene
puzzle 4x4
100
100
100
Performance
100
50-
2.3M
2.2M
2.2M
6M
2.2M
L.
antmare-umaze diverse
antmaze-meddum-pee play
antmaze hammmme--eddu--tiieee
antmaze-large-phy
antmare-large-diwee
Performance
stents
100
antmaaceuuaaze
pen-cloned
dogr-cloned
hammer--loned
relocate-cloned
100
100
Performance
50
7.0M
770M
7700
60
7700
7.00
Steps
Steps
Steps
Steps
Steps
Figure 7: Offline-to-online results 15 tasks, averaged over seeds with standard deviation. The gray shaded region indicates the offline phase, and the online fine-tuning phase is left unshaded.
29

---

Published as a conference paper at ICLR 2026
G EXPERIMENTAL DETAILS
We implement the proposed algorithm in JAX (Bradbury et al. 2018 on top an official implemen- tation of Park et al. (2025), with reference Lipman al. (2024).
G.1 BENCHMARKS
We evaluate our proposed method on 50 state-hased and 5 pixel-based singletasks of OGBench (Park ct al. 2024c) and 23 tasks of D4RL (Fu et al. 2020).
OGBench. We evaluate FAC across the following tasks from 10 navigation and manipulation environments:
baterhahhs..aaaiaaaaaaa...eaaaaa 5,5
-,
OGBench provides suite of environments and datasets for offline goal-conditioned RL. We use datasets from 5 navigation environments (entmaze-large, antmaze-gian sumanoidmaze-meen
antsoccer-arenal and manipula- tion environments slube-single,, cube-double, scene, puzzle-3xi 3x3, (uzzle-4x4) Each dataset offers singletask variants. For offline RL evaluation, we use the reward-based singletask variants whose rewards are relabeled align with the singletask specification. In naviga- tion tasks, a sparse reward function is used, the agent receives a reward of on successfully reaching the goal position and -1 otherwise. In manipulation tasks, each singletask is composed of multiple subtasks, and the agent receives the negative of the number of unsuccessful subtasks as its reward.
gammanoidmaz-laar
The navigation environments include antmaze and humanoi dmaze, where quadrupedal agent and humanoid one are required to reach goal positions in given maze, and antsoccer, where a quadrupedal agent dribbles ball a goal position. The manipulation environments include cube, where a robot arm is required to pick and place colored cubes, and scene, where robot arm executes a sequence subtasks, and puzzle, where a robot arm solves lights-out puzzle.
D4RL. We evaluate FAC across the following tasks from the MuJoCo, Antmaze, Adroit domains:
hopper-mediim-v22
valker2d-medium--42
happa-mmedim-reeplayy222
waller22--medium-rrpplaaaa
hal aad
hopper-medi um-expert-v2
30

---

Published as a conference paper at ICLR 2026
wellker2--meddum-eepptt
ant:aze-umazoovV
actiaae--umaee-cvveeeeev
autaaze-meciunnplayyyy
aothase--madiimmddieeeee-e
a0traze-1Larg--944-222
attaae-larreeddiveeee--
pen-human-vi
pen-cloned-vl
door-tumar-vi
door-clored-vl
hammer-human-ny
hameer-oloned-v!
relocate-human-vl
relocate-clloeed-v//
D4RL provides standard offline RL datasets for locomotion (MuJoCo domain), navigation (Antmaze domain), and dexterous manipulation (Adroit domain). In the MuJoCo domain, we evalu- on halfcheetah, hopper, walker2d with three datasets medium, um, medium-replay wwdiiiim-epper... medium dataset consists of transition rollouts from partially trained be- havior policy, and med um-rppay dataset the full replay buffer of the behavior policy, and medi um-experteet dataset mixes trajectories from the medium and expert-level behavior policies. In the Antmaze domain, we use (umaze, medi um, large large)-sized -iized mazes with two datasets play and diverse. The two datasets contain transitions to reach from start positions to goal positions. In the Adroit domain, we use pen, door, hammer, relocate tasks with two datasets human and cloned The human dataset consists of tele-operated demonstrations, and the cloned dataset contains transition rollouts from a behavior-cloned policy trained on the human dataset.
G.2 BASELINES
We primarily present the official reported performance from each baseline paper for benchmark datasets. For datasets that are not reported in the baseline papers, we refer to performances from other papers reporting performance for those datasets. For datasets without reported performances from other papers, we reproduced the results using an official implementation of baselines, with tuning hyperparameters.
Specifically, all results for method and the reproduced baselines are reported the mean + standard deviation over 8 random seeds.
For OGBench, we obtain the reported performances of BC, IQL, ReBRAC, IDQL, SRPO, CAC, FAWAC, FBRAC, IFQL,, FQL from Park et (2025).
For D4RL-MuJoCCo domain, obtain the reported performance of TD3+BC, CQL from Kostrikov ct al. (202I). Additionally, we reproduce FQL and tune broader range of hyperparameters than the ones recommended in the original paper. The hyperparameters and training settings for FQL are summarized Table 5 & 6. In particular, we consider normalizing Q-values in the actor objective, following TD3+BC (Fujimoto & Gu 2021). Empirically, enabling disabling this Q-normalization exhibits almost similar performance. To maintain consistency with default hyperparameter settings of FQL,, therefore report results obtained without the Q-normalization in the actor objective.
For D4RL-AAntaze domain, we obtain the reported performance of TD3+BC from Kostrikov et al.
202 the reported performance of MCQ from Yeom al. (2024). ones of SAC-RND from (Tarasoy]
et al.
2023). and the reported performance of CAC from the original paper and Park et al. (2025).
For D4Ri-Adroit domain, we obtain the reported performance of TD3+BC, IQL, SAC-RND from
Tarasov et al. 2023) and the reported performance of IDQL, SRPO, CAC from Park et al. 2025
31

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for FQL on the D4RL-MuJoCo domain.
Hyperparameters
Value
Learning rate
0.0003
Optimizer
Adam (Kingma & Ba 2014
Gradient Steps
1000000
Minibatch size
256
MLP dimension
[512, 512, 512, 512]
Nonlinearity
GELU (Hendrycks & Gimpel 2016
Target network smoothing coefficient
0.005
Discount factor y
0.99
Flow steps
10
Flow time sampling distribution
Unif( 1])
Clipped double Q-learning
True
BC coefficient À
Table
Q Q-normaizzation in actor loss
False
Table 6: BC Coefficient À for FQL the D4RL-MuloCo domain.
Tasks (Datasets)
BC coefficient À
nalfchoetah-meddim--22
3.0
hopper-medium-v2
100,0
walker2d-medium-V
300.0
mallichetaa-mediin--enny
30.0
hopper-medium medium -edium-replay-v2
100.0
walker2d-medium
300.0
halfcheetah-ne
30.0
hopper- medium-cxpertte
100.0
walker2d med-immeexppe-e--222222
300.0
G.3 OUR ALGORITHM
G.3.1 IMPLEMENTATION DETAILS
We describe the implementation details for flow actor-critic (FAC).
Network architecture. We use a [512, 512, 512, 512]-sized MLP for all neural networks. We apply layer normalization (Ba al. 2016) to critic networks. The critic network Qa takes state-action pair (s, a) as input and outputs a scalar estimate of the expected return under the one-step flow policy The behavior proxy policy 3. takes state Gaussian noise E Rd4, and flow time [0, as inputs and outputs the velocity field Un. The one-step flow policy Ty takes state and Gaussian noise and then outputs single action ag as(s,2) 2).
Image processing. For pixel-based tasks, following Park et al. (2025), we use a small variant of IMPALA encoder Espeholt et al. 2018) and apply random shift augmentation with probability of 0.5 and use frame stacking with three image observations.
Flow matching proxy. We use the flow matching objective (6) LLipman et al. 2022 Park et al.
2025
We use the Euler method with 10 steps to sample actions ag 2) and evaluate hehavior proxy
density (a)s).
Specifically, the proxy density can be evaluated exactly via the Instantaneous Change of Variables (Chen et al. 2018), which requires the divergence of the velocity field V... (a, 3.) This is the trace of Jacobian with respect to a,, and scales as ((dx)²² (Grathwohl et al. 2018). which may be prohibitive for high-dimensional action spaces. To reduce computational cost without introducing bias, we can approximate the divergence using Hutchinson's 's unbiased estimator (Hutchinson 1989] Adams et al. 2018), lowering the complexity to Odd)). In practice, adopt this estimator for and D4RL-Adroit domains, which have action space dimension d > 8, and using the estimator exhibits performance comparable exact evaluation.
CGGBench-Humanodee
32

---

Published as a conference paper at ICLR 2026
Critic learning. We twin critic networks to improve stability (Fujimoto al. 2018). For the empirical Bellman operator in the critic loss, we can aggregate them either by the arithmetic mean of the two Q values by their minimization. We present the aggregation choices in Table
One-step flow policy learning. Following (Fujimoto & Gul 2021). we apply the Q-value normaliza- tion in the one-step flow actor loss to balance the Q-maximizing term and the distance regularization term, i.e., maxxy Ba-=
(3.a))) Q+1
+ A DiTo, B.s). where Q2l d Σ 1Q((mm: a)] with mini-
batch {&m 1M m= and actions TT(-(sm) In practice, this normalization reduces the sensitivity to the actor regularization coefficient and enables a common tuning range across offline RL tasks.
Training and Evaluation. We train FAC for 1M gradient steps on all the state-based tasks of OGBench and D4RL, and for 500K gradient steps on the pixel-based tasks of OGBench. We evaluate it every 100K updates using 25 episodes. At evaluation, the actor samples exactly one action each time step. One thing to note is that we report the last performance measured after 1M gradient steps.
G.3.2 HYPERPARAMETER
All hyperparameters used for the main results in Table T & are summarized in Table & 7
The critic penalization coefficient a and actor regularization coefficient À are important in our method. The en- tire set of the two hyperparameters (a, X) used across all tasks a E (0.05, 0.1, 0.5, 1.0, 5.0, 10.0) 0.1,
and À {0.0003, 0.001, 0.003, 0.1, 0.3, 1.0, 3.0, 10.0}. For task-specific hyperparameters and narrow range for tuning hyperparameters, please refer to Table
Another important hyperparameter is the type of dataset-driven threshold in the weight w (s, a). As described, for datasets where the state coverage broad but the action diversity is limited, we use the dataset-wide constant threshold (OGBench: Antmaze, Antsoccer tasks / D4RL: Antmaze tasks). For all remaining tasks, we use the adaativee threshold.
Other hyperparameter is the use of clipped double Q-learning. With twin critics, the empirical Bellman target can be computed using either the minimization the mean of the two Q values for state-action pairs. We use the minimization on the D4RL tasks, and the mean on OGBench tasks other than Antmaze tasks. This selection aligns with the standard configurations of the baseline methods.
Table 7: Hyperparameters for method
Hyperparameters
Value
Learning rate Optimizer
0,0003
Adam (Kingma & Ba 2014
Gradient Steps
1M (detault), 500K (OGBench:pixel-basse tasks)
Minibatch size Epochs flow proxy model
256
(default),
125 (D4RL: OGBench:pixel -beeebesedeeeeeeee -based tasks)
MI.P dimension
[512, 512, 512]
Nonlinearity
GELU (Hendrycks & Gimpel 2016)
Target network smoothing coefficient p
0.005
Discount factor 7
0.995
Step count of the Fuler method
10
Flow time sampling distribution
Unif([O, 1))
O-normalization in actor loss Clipped double Q-learning
Truc
Min (D4RL: default CGBench AAtmmaze)
Mean (D4RL: Null OGBench: default)
Estimator for evaluating (als)
Exact (D4RL: default/ OCBench: default)
Hutchinson's is estimator with probes
[D4R:AAdroit OGBench: fumanoidmaces
Dataset-driven threshold
Batch-adaptive (D4RL: default / OGBench: default)
Datasct-wide constant
(D (D4RL: Antmaze (GBench: Antmaze, Ant soccer)
Critic penalization coefficient a Actor regularization coefficient À
Table 8
Table S
33

---

Published as a conference paper at ICLR 2026
Table 8: Critic penalization coefficient and actor regularization coefficient À of FAC
(a) OGBench
Tasks (Datasets)
a
antmaze-larg large-navigate www.giieeiinngooccaaaaa222225553
0.5
0.1
antmaze-ginn cermazz-giant-nane giant-navigate nuw.ignessngglccsscaaa123335333
1.0
0.1
0.5
0.3
0.5
0.1
antsoccer-areee htwp://wwww.aa...nnnonnoaaaaaaaa
1.0
0.1
(cube, ocanaiaeiiaaaanaaaaaaaaaaaaaaaaa {cube, setaallchhg...oooooooooiiiiaiaaa
0.5
10.0
1.0
1.0
(scene, hig.y-yayayyiiiggggaaaaaaasaaaaa
1.0
1.0
(puzzle, asuul-puzzle)
1.0
1.0
{puzzle, iisual-puzzile 4x4 1144-paay-singe (ataiyssiinggtttkkkaass....2.//. (12.33.4451
5.0
0.3
(b) D4RL: MuJoCo domain
Tasks (Datasets)
halfchhetal-mmediun-222
0,05
0.0003
hopper-medium-v2
walker2dlmeedium-v2
5.0
0.03
halfchhetth-mmddim--eeeaa-22
0.05
0.0003
hopper-meddiuum-eeplaa--2
5.0
0.1
walker2d sallerrddmmdiium-rrpaaay--
5.0
0.1
haalicheetal-mmddiuu---xpprrrr
0.003
hopper-medium--xppote
5.0
0.3
walker2d medium---emdumm
(c) D4RL: Antmaze domain
Tasks (Datascts)
antmaze-umaze-v2
1.0
0.1
antmazz-umazz--ivvrse-w
authaaze-medduu-play-222
0.5
0.03
antmaze- sutnaazemmdiium-ddierss---
5.0
0.1
haumazz-large-play-.ae
antmaze-laree-diversee.
1.0
0.03
(d) D4RL: Adroit domain
Tasks (Datasets)
pen-human-vl
10.0
0.3
pen-cloned-vl
1.0
0.1
door-human-v vl
1.0
3.0
door-cloned-vi
5.0
10.0
hammer-human v1
0.5
0.03
hammer-cloned-vi
0.5
1.0
relocate-human-vl
5.0
10.0
relocate-cloned-vl
5.0
10.0
G.3.3 DETAILS FOR ONLINE FINB-TUNING
FAC extend the online finetuning setting Ball al. 2023 Nakamoto al. 2023 Shin
ct al. 2025 2026: Kostrikov et al. 2021 Nair et al. 2020). For the offlimeto--onlnee evaluation, we initialize the one-step flow actor, critics, and flow behavior proxy using the networks trained for
34

---

Published as a conference paper at ICLR 2026
1M gradient steps in the offline training, and then continue training for additional 1M steps with online interactions. We evaluate every 100K updates. The practical implementation FAC for online finetuning is provided in Algorithm
No additional hyperparameters are introduced for the online finetuning phase, and we reuse the value of the hyperparameters used for offline training. The only modifications for the online finetuning (i) jointly training the flow behavior proxy together with the flow actor and critics, and (ii) replacing the dataset wide constant (s.a).)) (als) with batch wide constant min (a(ss) where is mini-batch for each gradient step. During online fine-tuning, the flow behavior proxy is trained so that it can represent online samples (s, a, r; s') and provide density estimates. The dataset wide constant requires estimating densities for the entire samples the replay buffer each gradient steps, even though only smaller size of mini-batch is used for each steps. This leads to unnecessary computation during online finetuning. To avoid this, while preserving the minimization- based thresholding design, we adopt the batch wide constant, which estimates the minimization of densities only for the mini-batch samples. We note that the batch adaptive threshold is used without modification during the online finetuning.
Algorithm 2: Flow Actor-Critic for online finetuning
offline-traieed one-step flow policy network We, cuftine-traince critic networks Qai Qoa, offline-traiee trained larget networks Q Qaz and offline-trained flow behavior proxy network 3.p
for each iteration do Sample (8, o, s') ~ D, ~ (((,, u), u Unif([O, 1]) and compute by eq.
Update By with eq.
Update Qo with eq. 9
E (1,2}.
Update To with eq.
&i pd: + (1 p)et,
E (1,2) for some
35

---

Published as a conference paper at ICLR 2026
H COMPUTATIONAL COSTS
Training Time (ms)
GPU Memory Usage (MiB)
Inference Time (ms)
2003
13
1933
mag
natue
rane chesp
ans meep
acc ac
est
mag
mutac
ranic choig
ars (eesg
nR rrus
ICc 14-1-11
cot
maa
cot
natsc
rase thi
rix rrut
IAc 14-0--1
Figure 8: Computational costs of FAC and baselines
We compare the computational costs of FAC against various baselines in terms of training time, GPU memory usage, and inference time. For all baselines, we primarily use their official open-source implementations whenever available. For ReBRAC, we use an open-source implementation provided with FQL Park et al. (2025). and FBRAC newly implemented. For the diffusion-basse baseline CAC, its official implementation is based on Pytorch Paszke et al. (2019). To enable a fair comparison, we additionally report the Pytorch-basee CQL implementation and the JAX-based one from the official open-source implementation of Cal-ql| Nakamoto et al. (2023).
The comparisons are conducted on single RTX 3090 GPU. Training time measured as the wall-clock time for one gradient step with batch size of 256, and inference time is measured as the time required to produce single action sample. The computational costs are shown Figure g For diffusion-basse and flow-based methods, we denote the number of denoising or flow steps as suffix to the method name (c.g., FAC (T=10)).
For training time, the PyTorch-based baselines exhibit larger training time than JAX-based methods. Within the JAX-based methods, IDQL shows the lowest training time. The training time of FAC moderately higher than IDQL and the multi-step flow policy baselines, reflecting the additional cost of density estimation. Nevertheless, FAC requires faster training time than JAX- based CQL, though both methods employ conservative critic objectives. The difference comes from simple practical implementation of FAC. The practical implementation of CQL requires auxiliary action sampling and additional gradient computing for soft maximization over Q-values, whereas FAC requires sampling only single action without additional computation.
For GPU memory usage, JAX-based CQL and the diffusion-basse baselines consume substantially higher GPU memory than other methods.
For inference time, diffusion-based baselines, CAC (T=2) and IDQL (T=5), require higher time than other baselines due to iterative denoising or diffusion steps. In contrast, the one-step flow actor used by FAC and FQL yields inference time comparable Gaussian policy based baselines.
36


Reviews:
[{"review_id": "YmhBdY9e72", "reviewer": "Reviewer_K2us", "summary": "This paper proposes FAC (Flow Actor-Critic), a new offline RL algorithm that integrates flow-based policies into the actor-critic framework. The motivation is that flow-based networks have much more expressibility than the Gaussian-based counterparts. The proposed method achieves state-of-the-art performance on the D4RL and OGBench benchmarks.", "questions": "- In the experiments of Figure 1, how does flow matching work when T=50? \n- In Equation (8), there can be other designs for giving more weight to outlier actions, for example, exponentially increasing the weight for those actions. Could the authors explain why they chose this specific formulation?\n- In L1261, the paper notes they chose the epsilon scheme according to the state coverage and the action diversity. Can the authors provide a clear definition of what the two terms mean?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The preliminary experiments strongly show that flow-matching can better estimate the density of the behavior policy.\n- The proposed method is intuitive and seems to be a nice approach to incorporate flow-matching to offline RL.\n- The proposed method shows strong performance not only on the D4RL benchmarks but also on the more recent OGBench benchmarks.", "weaknesses": "- Since the proposed method relies heavily on the accuracy of the density estimation from flow matching, there is some possibility that the proposed method may not scale well to higher-dimensional environments like pixel-based ones.\n- Importantly, the proposed method introduces many new hyperparameters (alpha, lambda, clipped double Q-learning, and epsilon), and those hyperparameters are tuned for each task. This is not a fair comparison with the baselines. For example, CQL and IQL use the same hyperparameter per domain (Mujoco, Adroit, ...).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YRRWV042js", "reviewer": "Reviewer_XLd9", "summary": "This paper proposes Flow Actor-Critic (FAC), a new offline reinforcement learning algorithm that leverages flow-based generative models to jointly regularize the actor and penalize the critic. Unlike previous flow policy methods that only used flow models for action sampling, FAC fully exploits the learned flow behavior proxy policy to estimate tractable behavior densities. These densities are then used to (1) identify out-of-distribution (OOD) regions for critic penalization and (2) constrain the actor through a one-step flow mapping toward the behavior support. The authors derive the modified Bellman operator under this framework and provide extensive empirical evaluation on OGBench and D4RL, demonstrating improved or state-of-the-art performance compared to Gaussian- and diffusion-based baselines.", "questions": "How sensitive is FAC to the choice of ε and to inaccuracies in the flow-based density estimate? Could an adaptive or uncertainty-aware threshold improve robustness?\n\nHow does FAC compare computationally to diffusion-based methods (e.g., CAC, IDQL) and multi-step flow policies (e.g., IFQL)?\n\nCan the authors provide quantitative results showing the quality of their flow density estimates versus ground-truth or surrogate densities?\n\nHow does FAC behave on datasets with limited coverage or severe distributional shift, where even flow models may fail to represent $\\beta(a|s)$?\n\nCould FAC be extended to online or offline-to-online fine-tuning, where exploration or policy improvement beyond the dataset is needed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper introduces a novel and coherent idea — using the same flow model both for actor regularization and critic penalization. This dual use of flow behavior density provides a principled way to directly detect OOD regions, addressing a key challenge in offline RL.\n\nTechnical quality: The method is conceptually sound and the derivation of the flow-based critic operator is clear. The integration of density-weighted Q penalization and flow-regularized actor optimization is well motivated.\n\nEmpirical validation: Experiments are extensive, covering 50 OGBench tasks and 23 D4RL tasks, with comparisons to strong baselines such as ReBRAC, IQL, CQL, IDQL, CAC, and FQL. FAC consistently achieves top or near-top results.\n\nSignificance: The approach bridges expressive policy modeling (via flows) and conservative value estimation in a unified framework. It represents a meaningful step forward in making expressive policies practically viable for offline RL.", "weaknesses": "Incremental over FQL: Conceptually, the work extends FQL by reusing the flow density for critic penalization rather than introducing an entirely new framework. Although the empirical gains are notable, the conceptual advance may be viewed as incremental.\n\nComputational cost: Flow-based models are typically heavier than Gaussian or VAE-based policies, but the paper does not discuss computational trade-offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PdoYbyOOpP", "reviewer": "Reviewer_Ypn5", "summary": "The paper introduces Flow Actor-Critic (FAC) for offline reinforcement learning. The key idea is to pair a flow-based behavior proxy with a one-step flow actor and to use the proxy’s density to tell when an action looks in-distribution versus out-of-distribution. The critic is then penalized only in regions where the behavior density is low, reducing the usual overestimation on OOD actions without dampening values where the data actually supports them. The actor, meanwhile, is trained to maximize Q while staying close to the behavior proxy, which helps with stability and makes good use of the expressiveness of flows without resorting to costly multi-step sampling. Empirically, FAC delivers strong results across large offline RL suites (OGBench and D4RL), beating or matching strong Gaussian, diffusion, and prior flow baselines. The ablations support the main claims: when candidate action sets get larger (which typically increases OOD risk), FAC remains robust while variants without the density-aware critic degrade.", "questions": "This method relies on a learned behavior density to decide where to be conservative. What happens if that density is hard to estimate well, for example, the dataset is multi-modal with rare but good actions, or high-dimensional with sparse coverage? In those cases, could FAC wrongly mark good actions as low-density and hurt performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and well structured. The motivation is crisp, the method is introduced in a logical sequence, and the experiments are laid out so the key results are easy to grasp before diving into details. \n\nThe core idea is clean and elegant. By using the behavior model’s density to decide when to be conservative, the critic only gets pushed down where actions look out-of-distribution and is left alone where the data provide strong support. Coupling this with a one-step flow actor keeps the approach expressive yet practical to optimize.\n\nThe empirical results are very strong. FAC matches or beats strong Gaussian, diffusion, and prior flow baselines across broad offline RL suites (OGBench and D4RL). The improvements are consistent rather than cherry-picked, and the gains on OGBench are clearly significant.\n\nThe ablation study is very insightful as well. \"Effect of the critic penalization\" section clearly shows why the density-aware critic matters. Variants without it degrade as OOD risk increases, while sweeps over key hyperparameters demonstrate robustness.", "weaknesses": "I don’t see any major weaknesses. The only area that feels under-explored is the threshold design used to decide when the behavior density is “low.” The paper offers two options—a dataset-wide constant and a batch-adaptive threshold—but stops short of examining how this choice affects robustness across tasks. It would strengthen the work to add a small, focused study on the threshold itself: for example, trying simple variants like scaling the base threshold by a weight; using per-batch quantile thresholds instead of a fixed cutoff; or annealing the threshold during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gw8vaqy2SB", "reviewer": "Reviewer_FJQh", "summary": "This paper proposes a Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies.", "questions": "1. In the Introduction, the paper mentions that diffusion policies make policy optimization computationally heavy. How does the proposed method's computational complexity compare to that of diffusion-based policies? It is suggested that the author compares the hardware efficiency of the proposed method with existing methods using metrics such as wall-clock throughput, FLOPs, and VRAM usage.\n2. In the Introduction, it is unclear why the highly expressive flow behavior model and accompanying density can solve the OOD problem.\n3. The paper lacks theoretical proof for why the proposed method provides more accurate Q-value estimation under OOD conditions compared to existing methods.\n4. The authors compare many baselines, but fewer methods from 2024 and 2025 are included. It is suggested to compare with more recent methods published in the last two years.\n5. From Table 2 in the main text, the proposed method's advantages are not evident, as in many cases, its performance is worse than the baseline. More experimental results showing the advantages of the proposed method should be added or moved from the appendix to the main text.\n6. It is suggested to include Q estimate results in the main text, comparing the Q-values of the proposed method with the baseline methods and the true Q-value to demonstrate the superiority of the proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a new critic regularizer to achieve accurate Q-value estimation under OOD conditions, and provides extensive experimental validation. The paper is overall well-organized.", "weaknesses": "1. In the Introduction, the paper mentions that diffusion policies make policy optimization computationally heavy. How does the proposed method's computational complexity compare to that of diffusion-based policies? It is suggested that the author compares the hardware efficiency of the proposed method with existing methods using metrics such as wall-clock throughput, FLOPs, and VRAM usage.\n2. In the Introduction, it is unclear why the highly expressive flow behavior model and accompanying density can solve the OOD problem.\n3. The paper lacks theoretical proof for why the proposed method provides more accurate Q-value estimation under OOD conditions compared to existing methods.\n4. The authors compare many baselines, but fewer methods from 2024 and 2025 are included. It is suggested to compare with more recent methods published in the last two years.\n5. From Table 2 in the main text, the proposed method's advantages are not evident, as in many cases, its performance is worse than the baseline. More experimental results showing the advantages of the proposed method should be added or moved from the appendix to the main text.\n6. It is suggested to include Q estimate results in the main text, comparing the Q-values of the proposed method with the baseline methods and the true Q-value to demonstrate the superiority of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)