floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
Authors: Bhavya Kumar Agrawalla
Michal Nauman
Khush Agrawal
Aviral Kumar
Venue: ICLR 2026 Poster (2026)
Keywords: offline RL
online fine-tuning
flow-matching
TD-learning
Primary area: reinforcement learning
TL;DR: We use iterative computation via flow matching to scale compute for value-based RL and obtain significant gains over monolithic critics under both parameter and compute matched settings.

Abstract: A hallmark of modern large-scale machine learning techniques is the use of training objectives that provide dense supervision to intermediate computations, such as teacher forcing the next token in language models or denoising step-by-step in diffusion models. This enables models to learn complex functions in a generalizable manner. Motivated by this observation, we investigate the benefits of iterative computation for temporal difference (TD) methods in reinforcement learning (RL). Typically, they represent value functions in a monolithic fashion, without iterative compute. We introduce floq (flow-matching Q-functions), an approach that parameterizes the Q-function using a velocity field and trains it with techniques from flow-matching, typically used in generative modeling. This velocity field underneath the flow is trained using a TD-learning objective, which bootstraps from values produced by a target velocity field, computed by running multiple steps of numerical integration. Crucially, floq allows for more fine-grained control and scaling of the Q-function capacity than monolithic architectures, by appropriately setting the number of integration steps. Across a suite of challenging offline RL benchmarks and online fine-tuning tasks, floq improves performance by nearly 1.8x. floq scales capacity far better than standard TD-learning architectures, highlighting the potential of iterative computation for value learning.

Published as a conference paper at ICLR 2026
floq: TRAINING CRITICS VIA FLOW-MATCHING FOR SCALING COMPUTE IN VALUE-BASED RL
Bhavya Agrawalla
Khush Agrawal khusha@andress cmu.
bbagrawa@andrest cmp edu
Michal Nauman
Aviral Kumar aviralku@andrew cmu, edu
nauman. mi c@gmai com
ABSTRACT
A hallmark of modern generative models is their reliance on training objectives that construct the target output iteratively, with dense supervision provided at intermediate steps, e.g., teacher forcing the next token in language models or step-by-step denoising in diffusion models. Such objectives allow models capture complex functions in a broadly generalizable way. Motivated by this observation, we study the benefits of iterative computation for temporal difference (TD) methods in reinforcement learning (RL). Typically they represent value functions in a monolithic fashion, without iterative compute. We introduce floo (fow-matehing O-functions), an approach that parameterizes the Q-function using a velocity field and trains it using techniques from flow- fow-matching. This velocity field underneath the flow is trained using TDDllarnnig objective, which bootstraps from Q-values produced by a target velocity field, computed by running multiple steps of numerical integration. Crucially, floq allows for more fine-grained control and scaling of the Q-function capacity than monolithic architectures, by appropriately setting the number of integration steps. Across suite of challenging offline RL benchmarks and online fine-tuning tasks, floq improves performance by nearly 1.8x. floq scales capacity far better than standard TD-learnng architectures, highlighting the potential of iterative computation for value learning.
Code and runs for floq can be found at: http://gittaaccom///UUAAAAAArti
INTRODUCTION
key principle in building effective models in various areas of machine learning the use of iterative computation: producing complex output functions by composing sequence of simpler operations. E.g., language models based on transformers (Vaswani et al., 2017) can generate coherent text by predicting the next token or by composing atomic reasoning strategies (Gandhi et al., 2025). Similarly, et al., 2015; Lipman et al., 2023;
diffusion and flow-matching models (Ho et al., 2020; 2023) synthesize images by progressively denoising small perturbations. Effective results from these models suggests that iterative computation a powerful tool for modeling complex functions with deep networks, by scaling compute appropriately.
Albergo & Vanden-Eijndee
Motivated by these results, this paper, we ask: can iterative computation also improve value estimation in reinforcement learning (RL)? Specifically, we are interested in improving the esti- mation of the Q-value function. While Q-functions map state-action inputs to a scalar value, they known to be highly complex and difficult fit accurately (e.g., (Dong al., 2020)). Standard cinporal-difference (TD) learning used to train Q-funcioons struggles to leverage capacity of deep networks (Kumar al., 2021; 2022: Bjorck et al., 2021: Lyle et al., 2022; Gulcehre al., 2022), often resulting in poor generalization. These problems are further exacerbated in the offline RL problem setting (Levine et al., 2020; Kumar et al., 2019), where we must learn entirely from static datasets. This motivates exploring architectures that spend compute iteratively to estimate value functions, potentially yielding better Q-values and policies.

---

Published as a conference paper at ICLR 2026
Numerical integration flow
A natural starting point for using iterative com- pute in value-based is to utilize a ResNet (He et al., 2016) Q-function, where stacking more residual blocks provides a way to run iterative computation. Recent work has obtained modest gains with ResNets (Kumar et al., 2023a;b; Fare- brother et al., 2024; Nauman al., 2024), but these methods need normalization and regulariz- ers to enable stable training (Bjorck et al., 2021; Nauman et al., 2024; Lee et al., 2024; Kumar et al., 2023a). Despite improvements, these ap- proaches lack one ingredient that makes iterative computation effective in transformers or diffu- sion models: supervision at every step of the iterative process. Just as next-token prediction supervises each generated token and diffusion supervises each denoising step, we hypothesize that stepwise loss supervision applied to TD learning might lead to improvements.
~ k(Q().))
Q-fnction
Integnaiinn
Categorical rupresentation of%
Q-ealue sample 00.9,46
Xees ER
Foutler Encedd
Figure I: floq architecture. We model the Q- function via a velocity field of flow-matching model. Over multiple calls, this velocity field converts domly sampled input z0) into sample from the Dirac- delta distribution centered at the mean Q-value. We build a fow-matching loss for training. Doing this enables us to scale computation by running numerical integration. with multiple calls to the velocity field. To train floq. we utilize a categorical representation of input #1 (Fare- brother et al., 2024) and a Fourier representation t.
With this observation, effectively leverage iterative computation with dense supervision, we design a novel architecture for parameterizing Q-functions. Instead of using single monolithic network, we represent the Q-function as a velocity field over scalar value (Figure 1). Our approach, floq (flow- moo-mhag Q-functions) samples scalar uniformly distributed noise and maps the Q-value by numerically integrating the predictions of the velocity field. We train the velocity with linear flow-matching objective (Albergo & Vanden- -iijddn,,,,,,, 2023; Lipman et al., 2023), supervised to match the evolving TD-targets. At each step, we minimize the deviation between the current Q-value estimate and the corresponding TD-target. We introduce several design choices that stabilize training and help the architecture scale capacity effectively. These include appropriately setting the support of initial noise, using a categorical representation to handle non-stationary inputs and a Fourier time embedding allow the velocity predictions to vary meaningfully across integration steps (Figure 2).
We use floq to represent the O-function for a number of complex RL (Levine et al., 2020; Kumar al., 2019) tasks from the OGBench (Park et al., 2025a) benchmark, previously studied by Park et al. (2025d). In aggregate, we find that fl outperforms offline RL algorithms that represent Q- functions using monolithic network by nearly 1.8x. floq superior even when these approaches are provided with more parameters, and more complex and higher capacity architectures. flog also outperforms existing methods when running online fine-tuning after offline RL pre-training. We also show that increasing the number of flow-matching steps results in better downstream policy performance, Allocating the same capacity via Q-network ensembles or ResNets performs worse.
RELATED WORK
Expressive generative models in RI. The most typical use of conventional generative models in RI. has been to represent the policy, with several adoptions of diffusion policies (Wang et al.;
Hansen-Estruch et al., 2023; Yang al., 2023; Bansal et al., 2023; Li et al., 2024; Ren et al., 2024), flow-based policies (Lipman et al., 2023; Albergo & Vanden-Eijnden, 2023; Park et al., 2025d), and sequence policies (Janner et al., 2021: Lee et al., 2022; Yamagata al., 2023). This shift is motivated by evidence that policy learning is often a significant bottleneck in offline RL (Kostrikov et al.; Park al., 2024). In parallel. policy-agnostie frameworks such as PA-RL (Mark et al., 2024) decouple algorithmic progress from specific architectural choices, enabling the use of diffusion, flows, or transformers interchangeably. Complementarily. we do not focus on policy expressivity and instead aim to utilize more expressive Q-functions, and opt study f floq top of FQL for simplicity.
Scaling Q-functions. Efforts to scale Q-functions in RL have taken multiple directions with new training objectives such as classification losses (Kumar et al., 2023a; Farebrother et al., 2024; Nauman et al., 2025; Seo et al., 2025), architectures (He et al., 2016; Kumar et al., 2023b; Chebotar et al., 2023; Obando-Cert et al., 2024), and regularization strategies (Kumar et al., 2021; Lyle al., 2021; Kumar et al., 2022; Nauman et al., 2024; Bhatt et al., 2024).. .. Previous work has also attempted to develop scaling laws for TD learning (Rybkin et al., 2025; Fu et al., 2025) and showing that alternatives to TD
2

---

Published as a conference paper at ICLR 2026
can scale to deeper architectures (Wang et al., 2025). Despite these advances, clear recipe for scaling value-based RL with TD-learnig has yet to emerge. Our work demonstrates that compute- efficient scaling can be realized not simply by increasing depth width, but by introducing dense intermediate supervision through multiple integration steps of the Q-function. olq introduces a novel axis of scaling, allowing for compute scaling through additional integration steps rather than depth width.
Scaling inference compute. A complementary line of work studies how more inference- compute can be traded for performance. Classical MPC-style planners coupled with learned dynamics models such as PETS (Chua et al., 2018), MPPI (Williams et al., 2017), and PDDM (Nagabandi et al., 2018) naturally allow scaling. In offline RL. MBOP (Argenson & Dulac-Arnold 2020) explicitly adopts planning with a learned model, behavior prior, and a terminal value to extend the effective horizon. Generative world models enable similar test-time scaling by planning inside the learned model (Janner et al., 2021; 2022). Similarly, performance of MCTS--tyle methods improves with more simulation (Schrittwieser et al., 2020; Hubert et al., 2021; Danihelka al., 2022: Ye et al., 2021). Across all methods listed in this paragraph, the general pattern is that increasing test-time budget (i.e. simulations, horizon, candidate trajectories) improves returns up to the limit set by model bias and value estimation error. However, none of these works use more test-time compute to better estimate a value function. Our results show that floq can not only use more integration steps at inference time amplify the "capacity" of the Q-function, but also that doing during training helps us learn better Q-functions in the first place. We are the first to show that using more integration steps is viable and effective path scaling compute for critic networks.
PRELIMINARIES AND NOTATION
that max- The goal in RL is to learn the optimal policy π. : S AA for an MDP MDP.M=
imizes cumulative discounted value function, denoted by (*(s)= = E, Bu)-((s)) (str, at) [s) = s]. S, A denote the state and action spaces. P(s'|s, ss, a) and r(s, a) are the dynamics and reward functions. p(s) denotes the initial state distribution and 7 E (0, 1) denotes the discount factor. The Q-function of a policy π is defined as (T, a) = E, (=(.)) 17 (7'r [ = s, a0 a]. and we use Q7 to denote the estimate of the Q-function of policy т as obtained via a neural net with parameters B. Value-basee RL methods train a Q-network by minimizing the temporal difference (TD) error:
L(0) = ((,,a,"") ("taara"" (r(s, + (Q('', Qu(s,
(3.1)
where D the offline dataset, Q the target Q-network, 8 denotes state, and a' an action from policy (--ss) that aims maximize (s,s, a). Offline RL methods are discussed in Appendix A.I.
Flow-matching. Given a target data distribution p(z) over x E Rd, flow-matching (Lipman et al., 2023; Liu et al., 2023; Albergo & Vanden-Eijnden, 2023) attempts to fit time-dependent field, 21 x) : [0. 1] x Id + Rd such that the solution the (t, x) to the ODE: d
di vo (t. x) = vo(t, (t. v/ (0, (0)) (0) transforms samples (0) from simple base distribution (e.g.. standard Gaussian or uniform, as we consider in this work) into samples from p(ar) at time / = 1. The most widely used approach is linear flow matching (Lipman et al., 2023), which trains the velocity flow to predict the gradient obtained along the linear interpolating path between (0) and <(1 at all intermediate points. Concretely, define x0) ~ po( be a sample from simple initial distribution, x11) p(x) be a sample from the target distribution, and t &~ Unif([0, 1J), we define interpolated points as (t) (1 t)- =00) + t x1), and train the velocity field to minimize the squared error from the slope of the straight line connecting (0) and x11). After training the velocity
field (t, æ (1)), fow-matching runs numerical integration to compute (t,,,x())), This numerical integration procedure makes several calls to compute the velocity field.
4 FLOQ: TRAINING Q-FUNCTIONS WITH FI.Ow-MATCHING
In this section, we introduce the our proposed approach, floq (flow-matching Q-functions), which leverages iterative computation with dense supervision to train Q-functions. To do so, we address the two central questions needed to make floq work: (a) how to handle moving target values in the training loss for a flow-based Q-function and (b) how to do effective not-hig over sealar Q-values without collapse for learning. Flow-matching preliminaries are discussed in Appendix A.1.
3

---

Published as a conference paper at ICLR 2026
4.1 FLOQ PARAMETERIZATION
In contrast to standard deep Q-networks that map state-action pairs to scalar values, floq parameter- izes a time-dependent,
velocity field uo(t, = s, a) over a one-dimensional latent input E R. At / = O, this input 2 is sampled from the uniform distribution Unif [1, u). where I and u are scalars that define the range of initial sample noise used for training. The velocity field transforms the initial sample z into a distribution over the Q-value. We will train flog such that the learned distribution of Q-values match a Dirac-Dellt Delta around the groundtruth Q-function, i.e. (1, 2|s, a) ~ (s..a at = We can obtain the Q-value sample by numerically integrating the ODE using the Euler method. One instantiation is shown below. Vj S K:
(a.a)
(o(i/k 8, ======+++ : Ei=1 v0 (o (i-1/K, 8, 8, a,.)) Q(s,a,z) == 1001,, ( (4.1)
An example illustration of this process is shown in Figure 1. This iterative process enables us to dynamically adjust the Q-function by varying the number of integration steps K, by controlling the number of evaluations of the velocity field uo, and thereby the "depth" of the model. Finally, we remark that although Equation 4. may appear similar to performing averaging like an ensemble, it is fundamentally different: the inputs passed to the velocity field Ug at each step depend on its own outputs from the previous step - 1. This recursive dependence introduces a form of iterative computation that is absent in conventional ensembles, that perform computation in parallel. As we demonstrate in our experiments (Section 5), this formulation enjoys greater benefits of scale than simply ensembling independent neural networks without iterative computation. In practice, the velocity field va va(*/K, -18, a) can be conditioned on the various representations of the intermediate Q-values v/ (oot- -Is.a) a) to improve the effectiveness of learning. We opt to use categorical representation of the when passing it as input to the velocity network. We discuss this in Section 4.3.
4.2 TRAINING LOSS FOR THE FLOQ ARCHITECTURE
With this parameterization in place, the next step is to design a training loss for the velocity field. Building upon TD-learning and flow- fow-matching methods, natural starting point is to iteratively train the velocity field using a loss that resembles linear flow-matching (Equation 4.2), but with targets obtained via Bellman bootstrapping. This is akin to TD-flows (Farebrother al., 2025) and y-models (Janner et al., 2020) that train a generative dynamics model with TD-bootstrapped targets. To do so, we introduce a target velocity field (g z | s, a). parameterized as a stale moving average of the main velocity field eg. similar to target networks in standard value-based RL. Given a transition (s, a, r, s'), we first sample an action a' ~ TT- | s') from the current policy at the next state s', and compute target Q-value samples (1, z'| ss,, a') by integrating the target flow, starting from some (1,
z' (via Euler integration) to obtain the predicted Q-value sample, (hg(1, 2'| 8''a').
We then average these predicted Q-value samples via (1, z' |s',a') for several values of the initial noise z' to compute an estimate of the target expected Q-value Qg(s', aa). The bootstrapped TD- target is given by: y(s, a) = r(s, a) + + +he Σn1 (1, '',a), where r(s, a) denotes the reward estimate for transition (note that this is distinct from distributional RL). We use this mean Q-target to train the Q-value at state-action pair (s, a) by regressing to the target y(s, a) via a linear flow- matching loss. Concretely, given a t ~ Unif(0, 1]. we construct an interpolant between noise z sampled at the initial step and the target Q-value y. (t) (1 - t)- z + y(, a), and train the velocity at this interpolant to match the displacement from (0) to y via flow-matching (Equation 4.2):
|oo((12(() x() s,a)
(u(s. a) - 2)
Lelog = Ez,,
(4.2)
1-0 0
4.3 PREVENTING FLOW COLLAPSE: HOW TO MAKE FLOQ WORK WELL?
So far, we have introduced a conceptual recipe for parameterizing and training a Q-function critic via flow matching. However, a na've instantiation of this idea performed no better than a standard monolithic Q-function in our initial experiments. This performance is the result of the inability of the network to meaningfully condition on the interpolant (t), leading the flow model to often collapse to monolithic Q-network (Figure 2). Interestingly, we find that this problem can is result of two peculiarities associated with applying now-matching to TD: training with constantly evolving targets and running the flow scalar Q-values. We describe our approach for handling these pathologies, and to do so, we first answer: what constitutes a "healthy" floq velocity field? Then we introduce two crucial modifications to the f f floq architecture that enable learning healthy f fl networks.
4

---

Published as a conference paper at ICLR 2026
Flow traversals with small (left) & large (right) widths [1.
x,(5.))
HL-Gauss representation
Qmia
Qzin
(R)
Pa (e)
(i)
Oue Add Gaussian noise with variance d
Ozen
Quts
(s.aa
Q o)
Founr Minbeeain
Nammo
rogion
region interpolanta
Interpolants
Oma. 0
Omax O
MMA
Target flow fiald
learned flow traversal
Figure 2: Illustrating the role of our design choices. Left: When the width of the interval [I al is small, and the overlap between this interval and the range of target Q-values we hope to see is minimal. we would expect to see more straight flow traversals, that might be independent of interpolant z. However, with wider intervals It u). the flow traversal would depend z, and hence span curved path when running numerical integration during inference. Right: Illustrating how we transform input interpolant into a categorical representation (top) and converting time t into a Fourier-basis embedding (bottom).
When is floq effective? Unlike traditional applications of flows, f floq applies them to scalar Q-values. How does flow matching on scalar work? Consider the trajectory traced by the flow during inference, as it evolves from initial noise (t 0) to the Q-value estimate produced by the network (t = 1) (Figure 2: left). If this trajectory a straight line, the velocity field vo (z(t), t) does not need to depend on í and predicting constant velocity proportional to the target Q-value is sufficient. In this case, flow matching provides no additional capacity beyond a monolithic Q-network. In contrast, if the trajectory is curved, the velocity field must utilize the interpolant =(t) and time to predict customized velocities and be able to integrate to an accurate Q-value estimate at = 1. Thus, even though training uses simple linear flow- mow-mng loss, extra capacity emerges only when the learned flows produce (slightly) curved trajectories. Here, iterative computation amplifies model capacity, allowing f oq to outperform monolithic Q-networks. Note that overly curved flows also problematic as they amplify errors in the integration process itself. Therefore, we want to attain an intermediate sweet spot in regards to the straightness of the traversals (see Figure 13).
Design choice 1: Distribution of the initial noise sample. As shown in prior works (Lipman et al., 2023; Liu al., 2023), rescaling the source noise leaves the target distribution unchanged, but alters the curvature of the transport trajectories. Interestingly, this effect seems to be particularly pronounced when applying flow-matching to scalar TD-leaming (see Figure 13 in experiments). As such, we find that that setting the bounds 1 and u for the distribution of the initial noise, Unif (7, u] greatly affects the performance of oq. We hypothesize that two aspects are important: (a) how close the target Q Q-values during training are to the chosen interval [1, ul.. and (b) the width of the interval u - I. If the width u - is too small, then the interpolants (t) span only a very limited range of values. When we then run (imperfect) TD-loss training on these interpolants, the network parameterizing the velocity field receives little meaningful variation in xt)) to associate changes in target values with. As a result, the model fails to exploit zt) effectively and degenerates into behaving like standard monolithic Q-function. Likewise, if the interval [1, u] is very disjoint from the range of target Q-values during training, then all interpolants z() are forced predict large velocities pointing in the general direction of the target Q-value. This reduces the need to learn calibrated velocity predictions conditioned on z(t) and time We show this in Figure 2, left.
Thus, we propose to choose and u using a simple heuristic. We set u = Qmx (=0 for most of our tasks). We then choose l 2 Quai to maximize the interval width u - while yielding stable learning curves and TD-error values comparable to a monolithic network. Here Quin and Qmax denote the minimal and maximal possible Q-value achievable on the task. Thus, the interval [t, 2) is likely to overlap well with the target Q-values the velocity field must predict during training. We remark that this heuristic relies on the assumption that over the course of learning, Q-value predictions will evolve from near-zero values to the target value. Since standard network initializations already produce values near zero, this assumption is not limiting and helps cover the target Q-value range we see.
Design choice 2: Representing interpolant inputs to the velocity network. The second challenge arises because the magnitudes of the scalar interpolant z(t) evolve during training. While standard TD-Jeamnng naturally handles non-stationary outputs (Q-values growing from near-zero random initialization). this is usually manageable with best practices such as activation normalization (Nauman al., 2024). In contrast, f f oq must cope with non-stationarity the imput, since the interpolant 2()
5

---

Published as a conference paper at ICLR 2026
is fed into the velocity flow. As training progresses, its magnitude grows, leading to large gradients and activations in the network. To address this, we adopt a categorical representation of input xtt) (output is still scalar like baselines), inspired by the HL-Gauss encoding of Farebrother et al. (2024), which prevents (())ss magnitude from skewing activations. Concretely, we add Gaussian noise with standard deviation a, then convert the resulting PDF (=((t), o²) into categorical histogram over N bins spanning the expected Q-value range. In contrast to Farebrother et al. (2024), we use a larger such that roughly 80% of bins receive non-zero mass at initialization, encouraging broader coverage. Finally, we also utilized Fourier basis representation of the time variable t, provided as input to the velocity network vg (t, (t)). This is illustrated in Figure (right). We show our experiments than doing so helps substantially by encouraging the network to meaningfully utilize this time.
Summary: floq architecture and training
flog parameterizes the Q-function via a learned velocity field, trained with a linear flow- matching loss against the target Q-value (Eq.4.2). To scale capacity, we sample initial noise Unif[l, u| broadly to overlap with target Q-values. The interpolant is encoded categorically, and the input to by is Fourier-encoded. See Algorithml for details.
Additional implementation details. We build on FQL., and use similar approach for training the policy. This way, we isolate the performance differences to our proposed flow-matching critic. In addition, because floq parameterizes fow-matching critic, extracting reparameterized policy gradients requires computing gradients through the full integration process with respect to the input action, which is costly. To alleviate this, we adapt a technique from Park et al. (2025c) for flow- matching policies and apply it to critics. Specifically, we train distilled critic, Qdistill (s.a), a). to approximate the predictions obtained by integrating the flow critic, Q8ow (s, a, a). Policy extraction is then performed directly on the distilled critic. We illustrate this idea in Algorithm 1.
5 EXPERIMENTAL EVALUATION
The goal of our experiments is to evaluate the efficacy of floq in improving offline RL and online fine-tuning. To this end, we compare floq to state-of-the-ar methods, and answer the following questions: (1) Does fl improve performance when compared to using similar-sized networks on benchmark tasks? and (2) How does the use of iterative computation via floq compare with the use of "parallel" computation of neural network ensemble and "sequential" iterative computation driven by ResNets of comparable size? We then run several experiments to understand the behavior of f flog critics. Furthermore,
variety of ablation studies to understand the design choices that drive the efficient use of iterative computation, including the roles of a) tuning the width of initial noise sample, b) categorical representations of the interpolant input, and e) Fourier-bass basis time embeddings.
5.1 MAIN OFFLINE RL RESULTS
Offline RL tasks and datasets. Following evaluation protocols from recent work in offline RL (Park et al., 2025d; Wagenmaker et al., 2025; Espinosa- -ice et al., 2025), we use the OGBench task suite (Park et al., 2025a) as our main evaluation benchmark (see Figure 18). OGBench provides number of diverse, challenging tasks across robotic locomotion and manipulation, where these tasks are generally more challenging than standard D4RL tasks (Fu et al., 2020), which have been saturated as of 2024 (Tarasov et al., 2023; Rafailov et al., 2024; Park et al., 2024). While OGBench was originally designed for benchmarking offline goal-conditioned RL, we use its reward single-task variants ("-s singnnngleel Meigletaa ask" from Park et al. (2025d)). We employ 5 locomotion and 5 manipulation environments where each environment provides 5 tasks, totaling to 50 state- based OGBench tasks. Some tasks are more challenging and longer- horizon (e.g.. marked in the table).
Comparisons and evaluation protocol. In addition to f floq critic, our experiments use flow- matching policies. Thus, flow-Q learning (FQL) (Park et al., 2025d), which utilizes flow-matching policy with monolithic Q-network, is our main comparison. FQL reports results with 1M training steps; we additionally re-run FQL for 2M steps and report both results. We run f floq with a default set of hyperparameters across tasks, but on the more challenging humanoi dewraaniiddzzzieeeeee dma -arge and ant maazeegiaan ant tasks we found a larger batch size of 512 to be more effective, which we use for both FQL and floq. Beyond FQL, compare against three recent SOTA offline RL algorithms, all of which rely on monolithic Q-functions: (i) ReBRAC (Tarasov et al., 2023), the strongest- strongest- method with monolithic Q-network and Gaussian policy; (ii) DSRL (Wagenmaker et al., 2025)

---

Published as a conference paper at ICLR 2026
Table 1: Offline RL results (all tasks), I Ilog achieves competitive superior performance compared to prior approaches. "Hard" environments refers the of environments where the FQI. approach attains helow 50% performance, averaged over the tasks, floq is especially more performant on these hard environments over prior comparisons, where its performance (with best configuration is around 1.8x of FQL.. We don't 't report DSRL (Wagenmaker et al., 2025) here as this prior work does not run on the exhaustive set of tasks (see Table 2 for these). A comparison on just the default tasks reveals =loq outperforms DSRL by 2x.
Gaussian Policy
Flow Policy
Flow Q Q-function (Oura)
Env. tasks each]
BC
ReBRAC
SORL
FQL (1M)
FQL (2M)
floq (Def.) floq (Beat)
antmaze-large
11
81 +s
89
79
83 t5
t5
anlmeze-giant (Hard)
0
26
9
22
27 1:31
36
51
hmmaze-
2 +1
22 +o
61 +4
57 +5
69 =20
82 =10
82 +:1
hmnaze-large (Madd)
=0
2 2=1
5
G0 #:
16 =0
28 29
antsoccer-arres
1 1o
0.
69
61 110
65
65
cube-single
5.1
91 12
97 11
96
94 15
98 13
98
cube-double (Harl)
21
12 1:
25 13
29 122
25 1.6
47 15
47 1:15
scene
41 +3
57 +2
56
57
58
puzzle-3x3 (Hard)
+0
21 +:
30
29
37 +T
37
puzzle-4x4 (Madd
0 +0
14
17 +2
9 +5
21 +5
28 +0
Avq Score (A11 knvs.)
3
31
46
47
56
59
Xvg Score (Hard Bava.)
34
38
that adapts a diffusion-based behavior cloning policy by performing RL over its latent noise space, improving over FQL; and (iii) SORL (Espinosa- to improve upon FQL. Note that none of them utilize a matthathnng Q-function, but do innovate across various properties of policy training. Comparing to the strongest methods that innovate on the policy allows us to evaluate the importance of a flow-matching Q- Q-function.
al., 2025) that leverages shortcut flow models
floq configuration. We run floq in two configurations. The "default" configuration utilizes the same hyperparameters across tasks, whereas the "best" configuration uses environment-specife hyperparameters that still are fixed across all tasks in that environment, to get a sense of the upper bound with floq. Even the default configuration of floq substantially outperforms all prior methods. The default configuration utilizes K = flow steps and sets the width of u - to be x (Q max Q min), where 0.1. The best configuration tunes the number of flow steps K {4,8) and i E 10.1, 0.25) per environment (not per task). For a fair comparison with FQL., we use a 4-layer flow critic in all environments, except in the cube (single and double) environments where employ a smaller 2-layer flow critic because we saw training instabilities with 4-layer critics.
Empirical results. Observe in Table I that floq outperforms prior methods, including FQL, on average across all 50 tasks, evaluated over seeds for each task. Also note that f floq improves over FQL most on the harder enviromments, where FQL attains performance below 50% success rate (antmaze-giane giant, hmmaze-large, cube-double, puzzle-3x3, and puzzle-4x4). For a statistically rigorous evaluation, we adopt techniques from Agarwal et al. (2021) and plot various statisties of the comparisons between floq and FQL:: 1) median and IQM scores in Figure 8, where we do not observe any overlap between the confidence intervals; 2) performance profile and P(X > Y) statistic in Figure 9, which are both strictly in favor of floq. Since DSRL (Wagenmaker et al., 2025) only evaluates on the 10 default OGBench tasks, we also provide an additional results table on only the default tasks in each environment in the appendix (Table 2). On the default tasks, find that flog outperforms DSRL (20% for DSRL vs. 45% for floq). improving by over 2x in success rate. These results establish the efficacy of floq. While floq uses an expected Q-value backup it still learns stochastic Q-function. Thus, we also compare f floq to a representative distributional RL approach (IQN (Dabney et al., 2017)) in Table 2, and we observe that floq outperforms IQN.
5.2 MAIN ONLINE TENNTTNING RESULTS
Next, we evaluate floq in the online RL fine-tuning setting. Here, we first train agents completely offline for 1M steps and subsequently fine-tune them online for an additional 2M steps (Figure 3). Across four challenging tasks cummanoddaaze-medimm humanoidmaze-large antsoccer-arena), f floq provides a substantially stronger initialization, faster learning during on- line interaction, and converges to higher final performance than FQL. Our complete set of results (Figure 10), show a similar trend establishing the efficacy of flog in online fine-tuning.
antmaze-giant and
7

---

Published as a conference paper at ICLR 2026
flo
FOL
hurransidmaze modium navigate
humano www.mmaaaeeee....o www.a//araaeenrriilaae.o
antmaze glant-navigae
antzozcer ar navigate
Sutress Rate
76
Steps
Figure 3: Learning curves for online fine-tuning of floq and FQL. floq only provides stronger
initialization from offline RI. training but also maintains its advantage throughout online fine-tuning the
hardest tasks, leading to faster adaptation, and higher final nerformance. The shaded area denotes offline RI..
(a) HM-Large
(b) Antmaze- Giint
(c) Cube-Double
(d) #MMMeedium
Rate (96)
Success Rate (%)
Success Rate (%)
GN
Success Rate (%)
70
55
58
36
24
11
18
flog
0-
12
16
12
16
12
16
12
16
Integration Steps
Integration Steps
Integration Steps
Integration Steps
Figure 4: Efffect of integration steps on floq. Performance of floq with varying flow steps, compared against monolithic Q-function (FQL.). More flow steps generally improve performance, hut too many steps can lead to diminishing or negative returns (e.g., uutmaze-giante That said, in all configurations, f floq outperforms FQL, and utilizing moderately large number of flow steps is important.
5.3 UNDERSTANDING THE SCALING PROPERTIES AND BEHAVIOR OF FLOQ
To better understand the benefits of iterative compute in floq. analyze its scaling behavior and compare it to different approaches. Specifically, we study: (i) how the number of flow-integration steps controls the expressivity of floq; (ii) how floq's iterative computation compares to monolithic critic scaling; and (iii) the importance of applying supervision to the velocity field at every flow step.
How does the performance of floq depend the number of integration steps for the flow? We now study the effect of varying the number of integration steps for the flow in floq. In Figure 4, we report the success rate of flog with K {1, 4, 16) flow steps, alongside monolithic Q-function (FQL) using the same architecture. Increasing the number of flow steps generally improves the performance of floq, with notable gains on harder tasks, hmmaze- age and antma ze-g ieee--gnnt.... Importantly, even with just 4 flow steps, floq already outperforms FQI., and the gap widens further with additional steps. However, we also observe diminishing returns and, in some cases, slight degradation beyond moderate number of steps (e.g., antmaze giant, where and 16 steps perform worse than 4). We suspect that this degradation stems from overfitting when the number of integration steps for computing the target is excessive. A similar degradation is observed for ResNets in Figure 6. We discuss this further in Appendix A.11.
2) How does fl ol compare against increasing "sequential" compute of monolithic critics? One hypothesis that fl oq could be implementing a similar iterative computation strategy such as ResNet (He al., 2016). Unlike floq, ResNet does not utilize dense supervision after each computation block. To test whether this matters, we compare f floq to ResNet. We use the same 4-layer flow critic from before and benchmark it against the best-performing ResNets with FQL. We ran cross- product over possible ResNet configurations with block sizes of 2, 4, 8, 16 layers and blocks of 8, 4, 2. These configurations cover all ways to build ResNet where 32 layers are involved in forward pass.
We compare f flog to the best ResNet configuration under a fixed total inference compute budget (that is, upper bound on the number of feed-forward layers) on humanoidmaze- large and antmaze-giant, since these harder environments should benefit from bigger ResNets. As shown in Figure 6, while ResNet critics do improve over FQL, they remain worse than floq even under matched inference compute. Note that floq does not itself utilize a ResNet, though its velocity network could use residual layers. Also note that while ResNet architectures instantiate every layer added to the network. floq uses parameters from a single 4-yerr MLP for all values of inference capacity. This shows that the gains of flow critics are not due to adding more residual
set of parameters for
8

---

Published as a conference paper at ICLR 2026
(a) HM-Large
(b)
intmaze-GGant
(c) Cube-Double
(d) HM-Medium
Success Rate (%)
86
Success Rate (%)
72
Success Rate (%)
98
37-
58
30
29
14
18
11as Eoprined Stepit
0-
Ensemble Size
Ensemble Size
Ensemble Size
Ensemble Size
Figure 5: Comparison of floq with monolithic ensembles. We evaluate ensembles of size 1, 2, 4, and 8 of FQL critics. While larger ensembles do better, even an S-critic ensemble falls short of floq with the number of flow steps, showing that flow critics provide gains beyond parallel compute.
layers parameters, but rather the dense supervision provided by supervising the velocity field at each step iterative computation.
3) How does floq compare to scal- ing monolithic critics with ensem- bles? A common way expand Q- function capacity is through ensem- bling, averaging predictions from mul- tiple critics for backups and policy up- dates. This increases parallel rather than sequential compute, and is attrac- tive if effective. We trained ensembles of 1, 2, 4, and 8 crities (each the same size as the base FQL critic) and com- pared them to floq. As shown in Figure 5, ensembles yield only modest gains FQL, and even 8 critics fail to match fl floq. Notably, floq also uses forward passes (via integration steps), so the compute is comparable. Thus, the benefits of flow critics stem
(a) Humanoidmazz--argg
(b) Antmaze- Giant
52
46
70
Sucoress
Spacess
31
0-
32
G4
FlOPs
FLOPS
Figure 6: Comparison of floq with ResNet critics on the hardest tasks: hnmaze- large, antmazz-giant outperforms the best ResNets under total hudget forward pass capacity, even after tuning over multiple residual configurations. For any given value on the x-axis, we plot the performance of the best performing ResNet configuration at that inference cost. For f flcq. integration steps at inference. Thus, approach of training floq does not simply add more depth.
4-layer flow critic
from parallel averaging.
4) Does beneftt from densely supervising the velocity at all time steps? To answer this question, We trained variant of floq where the velocity field was supervised only & with a single flow step. While one might think that training floq at t = is equivalent to baseline FQL,, this is not the case. Even with one integration step, the input to the ve- locity network is still scalar noise. The velocity field is trained to predict the difference between the target Q- value and this noise for all noise val- ues, creating several auxiliary tasks rather than the single task in baseline FQL (noise set 0). We hypothesize that fitting these auxiliary tasks yields representational benefits, consistent with prior observations linking auxiliary losses with improved TD representations (Lyle al., 2021). As shown in Figure 7, this restricted variant outperforms FQL but still underperforms floq. which supervises velocity at all E [o, 1] and uses multiple steps. On humanoidmaze-large. performance improves from 14% (FQL) 49% with only 0 training, while full floq reaches 56%. On antmaze-giant, the pattern is stronger, with scores of 11 %. 39%, and 86%. Similar trends hold on hm-medium (58%, 94%, 98%) and cube-double (36%, 38%, 72%). The smaller gap hm-medium likely reflects its lower difficulty. Thus, while t = already provides meaningful gains, supervising across all and using multiple steps is important for realizing the full potential of flog.
Train for all optimal integration vs. train a 0 only
100
FQL
Train E only
Succoss Rataa Ph3
Dus: Uaanall opairal integra
Hel.Lagge
Artiaaee Giant
H-t MWs.Millum
Gubr: Srubre
Overa
Figure 7: Comparing FQL, training floqont floq with supervision across all (and optimal integration steps chosen from (4, 8, 16)). While i = training improves over FQL, full floq consistently achieves the best performance, showing the benefits from training all integration steps.
and full
9

---

Published as a conference paper at ICLR 2026
Takeaways: properties and behavior of floq
More integration steps help, but performance saturates and can degrade at very high values. floq outperforms approaches that scale either parallel or sequential compute for monolithic Q-functions, and multiple steps are needed for best performance.
5.4 ABLATION STUDIES FOR FLQ
Finally, in Appendix A.5, we present experiments ablating various design choices and hyperparame- ters. We evaluate the sensitivity of f flog to these choices and give thumb rules for tuning them. The choices are: a) the range [1, w). for the support for the initial noise sample z x(0), b) the approach for embedding the flow step "time" t, and e) the approach for embedding the interpolant =(t).
Takeaways: ablation studies for flog
1) Utilizing HL-Gauss embedding for =tt) is crucial (Figure 14). 2) Utilizing Fourier- basis embedding of time is critical (Figure 12). 3) A moderate width of the initial noise distribution improves flow curvature, and performs best (Figure 13).
6 DISCUSSION AND PERSPECTIVES ON FUTURE WORK
In this paper, we presented fl lq. an approach for training critics in RL using flow-matching. flog formulates value learning as transforming noise into the value function via integration of a learned velocity field. This formulation enables scaling Q-function capacity by utilizing more compute during the process of integration to compute the Q-function. As result of utilizing flow-matching objective for training, floq utilizes dense supervision at every step of the integration process. We describe some important design choices to train flow-matching critics to make meaningful use of integration steps. Through our experiments, we show that floq attains state-of-the-ar results on a suite of commonly-un Q-function (e.g., via ResNet or monolithic Q-function ensemble). We also show the necessity of learning curved flow traversals to make effective use of capacity and utilizing the design choices we prescribe in this work.
offline RL tasks, and outperforms other ways of expanding capacity of
Future work. We believe floq presents an exciting approach scale Q-function capacity. Thus, there are a number of both theoretical and empirical open questions. From an empirical standpoint, it is important to understand how to appropriately set the number of integration steps as excessive steps may degrade Q-function quality. This degradation, however, is not localized to just flows but also to ResNets (Figure 6), indicating that this is perhaps bigger issue with TD-learning Another interesting direction is to build new methods and workflows for using Q-functions that rely on the property that floq inherently represents a "cascaded" family of critics with different capacities- all within one network. Can this property be used for tuning network size upon deployment, cross- validation of model size, or improving efficiency of policy extraction? Answering this question would be interesting for future work. Finally, fl eq also provides possibility for sequential or "depth" "bapth" based test-time scaling for value functions. Studying how this sort of sequential scaling can be combined with parallel scaling (i.e., ensembles) and horizon reduction techniques (Park al., 2025b) would be interesting as well.
From theoretical standpoint, quantifying iterative computation properties of floq would be impactful: in principle, curved flow traversals should enable the critic network to spend more test- time compute (i.e., integration steps) to perform equivalents of "error correction" and "backtracking" from large language models (LLMs) (Guo et al., 2025), but now in the space of scalar, continuous values to better approximate the target Q-function. We believe formalizing this aspect would not only be impactful for value-based RI, but could also shed light on methods to use test-time compute in flow/diffusion models in other domains. Second, our results show that there are substantial representation learning benefits of floq. We believe that studying the mechanisms and differences between feature learning induced by floq compared to standard TD-leamnig with regression (Kumar et al., 2022) classification (Farebrother et al., 2024) would be interesting for future value- based methods. flog also provides a rich family of auxiliary tasks to train a critic, which provides another angle explain and study its properties. All of these are impactful directions study in future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTT
We thank Max Sobol Mark, Zheyuan Hu, Aravind Venugopal, Seohong Park, Mitsuhiko Nakamoto, and Amrith Setlur for feedback on an earlier version of the paper and informative discussions. We thank members of the CMU AIRe lab for feedback and support. AK thanks Max Simchowitz for discussions on fow-matching and diffusion models. This work was supported by a Schmidt Sciences A12050 fellowship and the Office of Naval Research under N00014-24-12200
We thank the TRC program of Google Cloud and NCSA Delta for providing computational resources that supported this work. This work used the Delta advanced computing and data resource at the National Center for Supercompuing Applications (NCSA) through allocation CIS250548 from the Advanced Cyberinfrastrucctree Coordination Ecosystem: Services And Support (ACCESS) program. Delta is supported by the National Science Foundation (award OAC-2005572) and the State of Illinois, and ACCESS is supported by U.S. National Science Foundation grants 2138259, 2138286, 21 2138307, 2137603, and 2138296.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Neural Information Processing Systems (NeurIPS), 2021.
Michael S Albergo and Eric Vanden-Fijnden Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations (ICLR), 2023.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. arXiv preprint arXiv:2008.0m 05556, 2020.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36:41259-41288 2023.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. International conference on learning representations (ICLR), 2024.
Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning. arXiv preprint ar arXiv:2106.01 151, 2021.
Yevgen Chebotar, Quan Ho Vuong, Alex Irpan, Karol Hausman, F. Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Anand Sontakke, Grecia Salazar, Huong Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, and Sergey Levine. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning, 2023.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. arXiv preprint urXiv: 10044, 2017.
Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference on Learning Representations, 2022.
Sudeep Dasari, Oier Mees, Sebastian Zhao, Mohan Kumar Srirama, and Sergey Levine. The ingredients for robotic diffusion transformers. arXiv preprint vrXiv:2410.10083, 2024.
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. In International Conference on Machine Learning, pp. 2627-2637. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2026
Nicolas Espinosa-Dice Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kiante Brantley, and Wen Sun. Scaling offline rl via efficient and expressive shortcut models. arXiv preprint arXiv:2505.22866 2025.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXhv:2403.039500 2024.
Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Rémi Munos, Alessandro Lazaric, and Ahmed Touati. Temporal difference flows. arXiv preprint arXiv:2503 098117 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv, abs/2004.07219, 2020.
Preston Fu, Oleh Rybkin, Zhiyuan Zhou, Michal Nauman, Pieter Abbeel, Sergey Levine, and Aviral
Kumar. campute-optimal scaling for value-based deep rl. arXiv preprint arXiv:2508. 148881, 2025.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems,
2021.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self improving reasoners, or, four habits of highly effective stars, 2025. URL
https: //wvv//vavvvv///////e//eeeeewwvv org/ abs 2503. 01307.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.
Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matt Hoffman, Razvan Pascanu, and Arnau Doucet. An empirical study of implicit regularization in deep offline rl. arXiv preprint ar wwx::2207.000992 2022.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. ArXiv, 1at.22501.1999. 2025.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. Technical report, 2018.
Philippe Hansen-Estruuce Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprini arXiv:2304. 10573, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). ArXiv, abs/1606.08415, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems (NeurIPS), 2020.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning, pp. 4476-4486. PMLR, 2021.
Michael Janner, Igor Mordatch, and Sergey Levine. y-models: Generative temporal difference learning for infinite-horizon prediction. Advances in Neural Information Processing Systems, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
12

---

Published as a conference paper at ICLR 2026
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pp. 1 11761-11771 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 13:-119-11,,, 2020.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- parteeteriattinnooonnn inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Vasee deep reinforcement learning requires explicit regularization. International Conference Learning Representations, 2022.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline Q- learning on diverse multi-task data both scales and generalizes. In International Conference on Learning Representations, 2023a.
Aviral Kumar, Anikait Singh, Frederik Ebert, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre- training for robots: Offline rl enables learning new tasks from handful of trials. RSS 2023;
arXiv:2210.05178, 2023b.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. SimBa: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint, 2024.
Kuang- Huei Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205.1 /5241, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint, 2020.
Zechu Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: net n forum? i waw.mmiddddiiiibb3... wwtp://eeeett/nieeeeee/e
/popereeview..r
Yaron Lipman, Ricky TQ Chen, Heli Ren-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference Learning Representations (ICLR), 2023.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In International Conference Artificial Intelligence and Statistics, pp- 1-9. PMLR, 2021.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In International Conference on Machine Learning. pp. 14560-14581, PMLR, 2022.
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline and online rl fine-tuning of any class and backbone. ArXiv, abs/2412.06685 2024.
13

---

Published as a conference paper at ICLR 2026
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- mett-reiinfrremment learning. arXiv preprint 1XXv:1803.11347, 2018.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miloć, and Marek Cygan. Bigger, regularized, optimistie: Scaling for compute and sample-efficient continuous control. Advances in Neural Information Processing Systems, 2024.
Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, and Pieter Abbeel. Bigger, regularized, categorical: High-capacity value functions are efficient multi-task learners, arXiv preprint arXiv:2505. 2.t.:22555...3100 23,50, 2025.
Johan Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprint arXiv:2402.08609 2024.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline ri? arXiv preprint arXiv:2406.. 09329, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025a.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv preprint arXiv:2506.04168, 2025b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. ArXiv, abs/2502.02538, 2025.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint 2rXiv:2502.025338 2025d.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightted regression:
Simple and scalable off- off-policy reinforcement learning. arXiv preprint 19X1v:1910000177,
2019. 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning (ICML), 2007.
Rafael Rafailov, Kyle Beltran Hatch, Anikait Singh, Aviral Kumar. Laura Smith, Ilya Kostrikov, Philippe Fstruch- Victor Kolev, Philip Ball, Jiajun Wu, et al. D5rl: Diverse datasets for data-driven deep reinforcement learning. Reinforcement Learning Conference (RIC), 2024.
Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel. Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.0055 00588, 2024.
Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep rl scales predictably. arXiv preprint arXiv:2502.04322. 043377 2025.
Julian Schrittweseer, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):004-099 2020.
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3; Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint ar5iv:2505 22642, 2025.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), 2015.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
14

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint Stti::2506.15999, 2025.
Kevin Wang, Ishaan Javali, MichaL. Bortkiewicz. Benjamin Eysenbach, et al. 1000 layer networks for self-supervised rl: Scaling depth can enable new goal-reaching capabilities. arXiv preprint arXiv:2503. 14858, 2025.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning.
Ziyun Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Manfred Otto Heess, and Nando de Freitas. Critic regularized regression. In Neural Information Processing Systems (NeurIPS), 2020.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2): 344-357, 2017.
Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911. 1.11661, 2019.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.
Taku Yamagata, Ahmed Khalil, and Raul Rodriguz. Q-learning decision transformer:
Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, pp. 38989-39007. PMLR, 2023.
Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305. 13122, 2023.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25aa 2021.
15

---

Published as a conference paper at ICLR 2026
A APPENDICES
A.1 OFFLINE RI. PRELIMINARIES
We operate in the offline RL (Levine et al., 2020) problem setting, where the replay buffer P corresponds to static dataset of transitions D {(s, a, 8')) collected using behavior policy g-.- Our goal in this setting is to train good policy using the offline dataset D alone. The Q-network, Qa typically parameterized by deep network (e.g., an MLP).
Offline RL algorithms. Offlin RL methods aim to learn a policy that maximizes reward while penalizing deviation from the behavior policy πg, in order to mitigate the challenge of distributional shift. This objective has been instantiated in various ways, including behavioral regularization (Wu al., 2019; Fujimoto Gu, 2021; Tarasov et al., 2023; Park al., 2025d; Kumar et al., 2019), pessimistic value function regularization (Kumar et al., 2020), implicit policy constraints (Peters Schaal, 2007; Peng et al., 2019; Wang al., 2020; Mark al., 2024), and in-sample maximiza- tion (Kostrikov et al.; Xu et al., 2023; Garg et al., 2023). While our proposed f floq architecture for Q-function parameterization is agnostic to the choice of offline RL algorithm, we instantiate on top of FQL (Park et al., 2025d) that utilizes flow-matching policy to better model multimodal action distributions. FQL trains the Q-function using the standard temporal-differt difference (TD) error from soft actor-critic (SAC) (Haarnoja et al., 2018), shown in Equation 3.1. and optimizes the policy to stay close to a behavior policy estimated via flow-matching.
A.2 ADDITIONAL RESULTS FOR F FLOQ
In this section, we provide some additional and complete results supplementing the ones main paper.
1. Figure 8 presents median and IQM scores and Figure 9 presents performance profiles and P(X Y) statistic comparing floq with FQL all the 50 tasks studied in the paper.
2. Figure 10 presents results for online fine-tuning all 10 default tasks.
Median
IQM
Mean
Optimality Gap
(Bes)))tt) (De max (FQL(IM), FQL(2M))
FOL (2M)
(1M)
0.30
0.45
0.60
0.40
0.48
0.56
0.64
0.45
0.50
0.55
0.60
0.45
0.50
0.55
Success Rate
Figure 8: Comparison of floq against the baseline FQL across median, interquartile mean (IQM), mean and optimality gap, following Agarwal et al. (2021). Results show that floq consistently outperform FQL across all evaluation criteria with confidence interval overlap in all cases, meaning that the gains from floq significant.
Probability of Improvement
Performance Profiles
Algorithm
Algorithm Y
1.00
(oeqBestt)
FOL (2M)
0.75
flogDeff )-
FQL (2M)
0.50
floq(Best)-
max/FQL (1M) (2M))
Fraction of
0.25
RogiDer. )-
max/FQL (1M) FOL(2M)
flog10sf.)
0.00
#log18est)
0.63
0.73
0.0
0.2
0.4
0.6
0.8
1.0
Success Rate (1)
Figure 9: Comparison of floq against baseline FQL, following Agarwal et al. (2021). Left: Probability Improvement P(X > Y) showing that floq consistently outperform FQL across OGBench tasks. Right: Performance profiles illustrating that f too achieves higher scores across larger fraction of runs compared to FQL.
16

---

Published as a conference paper at ICLR 2026
1log FOL
numaroldmaze dcuble
Succei Rote
3M
2M
3M
1M
9000
wrf.mace-giantenwwwl..
antsocceer-aree www.-ann--aaavaaa.......
cube-sir glu-play
scene-play
animmaze las go-navigate
1.
Figure 10: Learning curves for online fine-tuning of floq and FQI. across all default tasks. floq not only provides stronger initialization from offline RI. training also maintains its advantage through online fine-tuning, leading to faster adaptation higher final success rates. The shaded gray area denotes offline RL training.
17

---

Published as a conference paper at ICLR 2026
Table 2: Offiiee RL results (Default Tasks), floq achieves competitive superior performance compared the baselines. "Hard" tasks refers to the set of default tasks where the FQI. haseline score is below 50% performance. floq is especially more performant on these hard tasks, more than doubling FQL''s baseline performance.
Gausaian Policy DiCE. Policy
Flow Pollcy
Flow Q- D-Eucction (Ouza)
IQN
ZQL
11:0
1log
(Det.)
Elog [Beot]
animaae 1arge
56
sxx
artmaze-giart
0.20
230
12
Am
11 :11
hmmare meae--nedddd um
12t
16 =n
34 270
67
27 #:7
23 sm2
19 gm
56 =:3
9R m1
98
hmmazc-Large
0.10
21
10 122
20
22 15
0 1.22
$.1
14 210
52 16
52 LG
antsoccer-atent
0.
28
54
44
36
19
40
40
cube-single cube-double scade
93
98
58
98
94
72
am
AM
50
$3 238
83 cm
pLzzle-3x3
142
2:1
20 45
13 64
16 45
14:44
17 10
44
puxxle-4x4
0ze
10
37 218
16
16
11 m3
5:20
12
19
Average Score IAl1)
30
38
46
39
40
64
84
86
Average (Hard)
22
21
Table 3: FQL ResNet performace on humanoi-mazz--large and antmaze-giant. (m, n)) indicates blocks each of depth m. For each fixed number of FLOPs n. the best performing architecture per environment in bold.
FLOPs (m n)]
HM-Large
Antmaze-Giant
4
35 = (2.2)
13 # (2,2)
10 (4.1)
11 (4.1)
8
11
31 13 (2,4) 22 (4.2)
21 11 (4.2)
22 11 (8.1)
+ (8.1)
16
41
32 (2,8)
(4,4)
17 1 (4.4)
26 (8.2)
46 1 (8,2)
21 (16.1)
0 1 (16,1)
32
25 1: 10 (2.16)
23 4 11 (2.16)
28 13 (4.8)
18 L: (4,8)
38 = (8.4)
9
0 (16,))
(16())
A.3 COMPATIBILITY OF FLOO WITH DIVERSE POLICY CLASSES
One natural question is whether the benefits of floq arise specifically from pairing it with flow- based policy, or whether the FloQ critic alone provides value when combined with standard policy architectures. To study this, we evaluate all combinations in the grid:
(Monolithic critic vs. floq critic) x (Gaussian policy vs. Flow policy).
In particular, we add the missing configuration: floq critic + Gaussian policy.
To isolate the effect of the critic, we integrate the floq critic into the ReBRAC framework, which employs a Gaussian policy. The policy architecture and actor update rules are unchanged; only Q-function parameterization is replaced with the flog critic.
Table summarizes the results across the 10 default OGBench tasks. The floq critic yields substantial improvements over ReBRAC, increasing the average score from 30 + 41. Gains particularly large on sparse-reward, long- long-horizon environment such as ANTMAZE-GIANT and
(27 + 91) (16 57). These improvements demonstrate that the flog critic offers benefits even when the policy is Gaussian.
Combined with the results in the main text (floq with FQL), these findings show that the floo critic is modular and compatible with diverse policy classes. Its advantages are not tied to flow-based policies, but instead arise from its parameterization of the Q value.
18

---

Published as a conference paper at ICLR 2026
Table FloQ critic is compatible with Gaussian policies. Performance on the 10 default OGBench tasks when integrating the FloQ critic into the ReBRAC framework (Gaussian policy). FloQ critic improves ReBRAC significantly, showing that FloQ's benefits are not tied to the flow policy used in FQL..
Environment (Default Task)
ReBRAC
F FloQ (ReBRAC)
FQL
FloQ (FQL)
antmaze-large
95 #4
97 =3
::4
94 14
antmaze-giane
27 122
91
14 1:0
70 18
hamaze-medium
16
57
58 2:5
98
hamaze-large
2
0 01
14 +10
52
antsoccer-areena
0 1:0
12
49 111
49 110
cube-single
92 14
89 24
10
1/2
cube-double
7 +3
3 +3
36 +6
72 #15
scenc
50 113
58
78 17
83 110
puzzlc-3x3
2
9
16
17
puzzle-4x4
10
6
11
12 14
Average Score
30
41
45
64
Table Offiine Results on D4RL Tasks. Elog performs similarly to the FQL baseline D4RL
benchmarks, showing its robustness. Values show normalized returns (mean + std).
Envirenment
FQL
FloQ
antmaze--arre--diveess-e
+6
82
antmaze-largg--play-v2
1s
75 28
urtmaze-meecium-iiveeess..
73 #6
71 +:10
untmmzze-nociimm-play--22
76 17
80 17
artmazze-umazze-divveeee v2
85 #7
78 +17
antmaze-umaze-v2
96
97
pen-human-vl
51 #10
58 #:
pen-cloned-vl
78 1s
72 25
pen-expert-vl
140 #s
140 +s
door-human-vl
#o
=00
dbor-cloned-vl
3 ::2
expr-
104 =0
104 =0
hammer-human-vl
1
hanmer-cloned-v2
15 +18
hammer-expert-v.
125 23
125 1/2
relocate-huran-vi.
0
relloatee-cloned-v1
0
relocate-expert--/
106 #2
108
+2
Median
IQM
Mean
Optimality Gap
flog
FQL
69
72
75
78
61.5
63.0
64.5
66.0
60
61
62
63
0.12
0.15
0.18
0.21
Aggregate Performance D4RL Tasks. Comparison of floq and FQL D4RL using Median, IQM, Mean, Optimality Gap metrics shows that they perform similarly.
Figure 11:
A.4 RESULTS ON THE D4RL BENCHMARK
To further evaluate the robustness of floq, we also compare its performance to FQL on the standard D4RL benchmark. As shown Table 5, floq performs similarly to the FQL baseline across all D4RL tasks, with nearly identical normalized returns average. These tasks are considerably simpler than the hong-horizon, sparse-reward OGBench environments used in the main evaluation, and the comparable results D4RL demonstrate that f floq does not sacrifice performance benchmarks. addition to per-task results, Figure 11 presents aggregate metrics using the rliable framework, including Median, IQM, Mean, and Optimality Gap. Across all four metries, floq and FQL exhibit closely aligned confidence intervals, further indicating that their performance on D4RL statistically indistinguishable.
easier
19

---

Published as a conference paper at ICLR 2026
Noise Range vs. Success Rate
Noise Range vs, Flow Curvature
100
15
Success Rate
0.01
0.05
0:10
0.25
0.50
1.00
0.01
0.05
0.10
0.25
0.50
1.00
Omnl
1M/0 Omnl
Medium
Puzzio-4x4
Medium
Glant
Double
Figure 13: Effect of variance of the initial noise sampling distribution on floq. Left: Success rates across environments function of the initial noise scaling factor (black circles denote the best setting per environment). Right: Flow curvature in HM-Medium increases with noise variance, highlighting the tradeoff' between too little curvature (flow collapses monolithic critic) and too much curvature (difficult numerical integration).
Together, the table and aggregate statistics confirm that E floq is robust across both challenging and standard offline RL benchmarks, matching FQL on D4RL while significantly outperforming it on the more challenging OGBench tasks.
A.5 ADDITIONAL. ABLATION STUDIES FOR F FLOQ
1) Ablations for the width of the [, uj interval. We study the effect of varying the variance of the initial noise sample used in critic flow match- ing by expanding the width u-l of the interval that the initial noise is sam- pled from. We present the results in Figure 13. On the left, we observe that the performance across several tasks typically peaks at intermediate variance values (note that the black circles marking the setting that yields the best success rate for each environ- ment). This means that choosing an interval [1, u] with a non-trivial width is important. As discussed in Section 4.3, Figure 13 (right) shows that the curvature of the learned flow increases as the width of the interval grows. We measure curvature by computing the magnitude of the derivative of the velocity field function of time using finite differences. Concretely, we measured the expected value (t.)(((a)) across state-action pairs the offline dataset, averaged through training.
Time Embedding Ablation
Scate Time tribed
Fouter Time Imbedd rDuna
Succisss Plat (96)
HPM
Antrade Clart
F-M Med.Ma.rmm
Cube Doub
Overal
Figure 12: Time embedding. Replacing the Fourier-basis embed- ding of time with a scalar embedding results in significantly worse performance, highlighting the importance of Fourier features for conditioning time.
Putting results in Figure 13 together, note that some degree of curvature is necessary for best performance, which is expected because otherwise, the flow collapses to behave like a monolithic critic. That said, excessive curvature makes the flow numerically harder integrate, ultimately degrading performance. Based on these observations, we recommend practitioners use k == (u 1)/(Q (Q - Qmin) in the range of {0.1, 0.25] as reliable starting points when tuning floq critic.
2) Ablations for the time embedding. In the default configuration of floq, we used a 64-dimensional Fourier embedding for the time I, provided as input to the velocity field (also see Dasari et. al Dasari et al. (2024) for a recent work training a diffusion policy also using Fourier embedding of f). As shown in Figure 12, replacing this Fourier embedding with simple scalar embedding of leads to significant drop in performance on several tasks. This highlights the importance of the Fourier embedding, which allow the velocity function to be meaningfully conditioned on f, enabling it to produce distinct behaviors at different integration times. Without such rich embeddings, the critic struggles to leverage temporal information effectively, and again collapse to the monolithic architec- ture. We therefore recommend that practitioners carefully utilize high-dimensional embeddings of time when using floq.
3) How does the approach of embedding the interpolant z(t) affect floq performance? We observe that the approach of embedding z(t) (Design Choice in Section 4.3) plays a significant role in the performance of fl ol. As shown in Figure 14, HI.-Gauss embeddings of z(1) provide
20

---

Published as a conference paper at ICLR 2026
z(t) Embedding Comparison
Success Rate (%) vs. HL-Gauss
100
Return
80
Normalized Scalar Return Embed
80
HL Gauss Return Embed (Ours)
60
Success
40
Success
HMM-Large
Glant
HM Medium
0-
Cube Lubo-Double
HM-Large (Task-5)
Antmaaee Haltaaze-Gian//. (Task-5)
16 (default)
HL-Gauss z(t) Embedding
Comparison of different approaches for representing the input interpolant in floq Left: performance on two representative tasks where that HL-Gauss embeddings outperform sealar and normalized scalar embeddings by reducing sensitivity to non-stationary inputs. Right: Ablation over HL-Gauss embedding scale for the scalar flow interpolant input, showing that larger values provide broader bin coverage and stronger performance. Default
Figure 14:
16.0.
(a) HM-Large
(b) Antmaze- Giant
(c) Cube-Double
Best
75
Success Rate (50)
Success Rate (%)
Success Rate
54
floq (4-aayer MLP)
floq (4-loyer MLP)
yer
14-ayyer
FOL. (4layar MIP!
Distil Integration Stops
Distill Integration Stops
Distill Integration on Steps
Figure 16: Effect of the number of integration steps used for policy extraction on floq performanee Even though computing the target values for TD-learnin utilizes a fixed number of 8 integration steps, in this ablation we utilize smaller number of steps for extracting the policy. Performance is more robust to the number of integration steps used for policy extraction, suggesting that long target integration is sufficiently accurate, few steps suffice for policy distillation.
a significant advantage over scalar or normalized scalar embeddings. In particular, across several tasks we found HL-Gauss embeddings (with sufficiently large value of σ) be essential for achieving strong performance, and Figure 14 (left) highlights two representative tasks in this category.
HL-Gauss embeddings with broader bin cover- age helps reduce the sensitivity of the network to non-stationary inputs, thereby stabilizing train- ing and improving performance. In domains such as HM-Large (Task 5), using raw scalar em- beddings increases the training gradient norm by roughly 4x compared to HL-Gauss (2.5 x 10* to 10"), reflecting greater instability in optimiza- tion due to non-stationary conditioning (see Fig- 15). While normalizing z(l) helps on some tasks over using the raw value, we found that HL-Gauss embeddings generally gave the best performance. In our implementation of the ve- locity network, we use HL-GGass embeddings with default scale of = 16.0.
Training Grad Norm On HM-Large (Task 5)
250000
Grad Norm
150000
100000
50000
Training Step (1e6)
HL-Gauss Encoding
Raw Scalar Encoding
Figure 15: Training Grad Norm With and Without HL- Gauss Encoding. In domains such HM-Large (Task 5), using raw scalar embeddings increases the training gradient norm by roughly 4 compared to HL-Gaass (2.5 x 10* to 10"), reflecting greater instability in opti- mization due to non-stationary conditioning
Figure 14 (right) shows an ablation over smaller values of a E [1.0, 2.0, 4.0, 8.0). Observe that larger values of consistently yield stronger performance. Intuitively, increasing leads to broader bin coverage for the HL-Gauss distribu- tion (see Figure 2, right), which helps mitigate the non-stationarity of the range of <t) over the course of training with TD-learning. These results highlight that selecting sufficiently large embedding scales is important for stabilizing learning and achieving strong downstream performance.
4) How does the number of critic flow steps used for the policy update affect the performance of flog? We next investigate the effect of varying the number of integration steps used for calculating
21

---

Published as a conference paper at ICLR 2026
the Q-value for the policy update. Since build our algorithm on top of FQL, we implement the policy update by first distilling the values produced by the flow critic into a one-step, monolithic Q-function. Then the policy extraction procedure (akin to SAC+BC) maximize the values of this distilled critic subject to a behavioral cloning loss. Note that this approach essentially decouples the number of integration steps used to compute the -artta and the number of integration steps for policy extraction. As shown in Figure 16, as long as the number of integration steps for computing the target value are fixed (to in this case), the performance of fl floq is relatively robust to the number of integration steps used for the policy update. Contrast this with the sensitivity to the number of integration steps used for computing the TD-target observed in Figure The results indicate that once the target integration steps are sufficiently large (here, 8), the policy can be effectively distilled even with small number of integration steps.
A.6 HYPERPARAMETERS AND ADDITIONAL DETAILS
In this section, present some details for floq that could not cover in the main paper, along with a pseudocode and a complete list of hyperparameters used by our approach.
Algorithm 1 Critic Flow Matching (floq) in conjunction with FQL (Park et al., 2025d)
Given: offline dataset of transitions D,
Models: flow critic, QBLOW (s, a, , distilled critic, Quiatilodd (s, a, z), flow policy TT TT(-ss), one-step policy Maw (.,.). (s, (s,0, 2)
function Q7T
P Flow Q-function, introduced by flog
for l 0, 1...,, K do
<1 1) < x11 1/K vo (/x, x(1) s,a)
Euler method, time is normalized
return x(K)
function (als)
> Flow policy from FQL., though policy training orthogonal to flog
for / = 0, 1..... .M- do
Sample =0)) N0, Ia)
att 1) a+1 1/ar ws (4/M, x(() ss)
Euler method, time is normalized
return x(M)
while not converged do Sample batch ((s, (s,,a,,,s')) ~D D
> Train vector field Ug in flow critic LLoW
a' <- Sample (-
Sample actions from policy, typically the one-step policy for FQL
z(0) Unif m). (0) 1:m Unif [1, u]
Sample initial noise for computing the Q-value
=(1) - r
1/mm Dm Q ((()))=((00) 2()))
& Use noise ={(0) for computing T'D-target
=(t) (1 (1-t) t)- x(0) +t x1)
Compute interpolant =tt) for random t
Update 0 minimize E [(00 (f) a, (z(1) =
Linear flow-matching loss
Train distill critic Quiliil for policy extraction
Update & to minimize Ez(o) [(Qasil] (s, a) Q a, als,a,((00) (0)))
Train a BC flow policy ne. analogous to FQL x00 ~ 40) æ<1) + a
t~ t~~ Unif[[0, 1])
arf) - (1-t) x0)) + xx1)
For FQL policy,,,, compute policy interpolant
Update o to minimize [[wl (t, (t))ss) - (a(1) x00) [[]]]
> Flow-matching loss for policy
D Train one-step policy μuu to maximize the learned distill critic while staying close to BC flow policy # NV La) a= < nu (8, (s)
Update w to minimize E. [-Quaill (s,a") cella" To(s, x) []]
return One-step policy mw
Hyperparameters for offline RL results. Following Park et al. Park et al. (2025d), tune the BC coefficient a on the de defultttass of each environment and then fix this value for the remaining tasks. For both FQL and fl floq, is tuned over (arQu A, OFQL, arQu + A}, where A 100
22

---

Published as a conference paper at ICLR 2026
Table 6: Hyperparameters for floq. Differences from FQL shown in light blue within brackets. Other hyperparameters are kept be same as FQI..
Hyperparameter
Value (floq)
Learning rate
0.0003
Optimizer
Adam (Kingma & Ba, 2015 Kingma & Ba (2015))
Gradient steps
2M (Oftiine), 1M 2M (Online FT)
Minibatch size
256 (default), 512 for hmm-large, antmaze-giant
Flow Q Network MLP dims
[512.512,512.512]] (default), [512,512] for cube envs [512,512.512.512] used FQI.)
Distill MI.P dims
Nonlinearity
GELU (Hendrycks & Gimpel, 2016 Hendrycks Gimpel (2016))
Target network smoothing coeff.
0.005
Discount factor Ty
0.99 (default), 0.995 antmaze-giant, humano dmaze, soccer On,
Flow time sampling distribution
Clipped double Q-learning
False (default), True (antmaze-giant) (+ antmaze untmaze-Lazge FQL)
BC coefficient
Tables 7.8
Actor Flow steps
10
Critic Flow steps
(default), Table 9 for env-wise (not used FQL)
Initial Sample Range
0.1 (default), Table 9 for env-wise (not used in FQL)
Number Of Initial Noise Samples
(not used in FQL)
Fourier Time Embed Dimension
64 (not used in FQL)
Table Enviromment-wise BC-Coefficieet (ax) for FQL and floq (Offline RL).
Environment (5 tasks each) (FQL), (floq)
Lntmaze-large
10, 10
antmaze-giant
10, 10
hmmaze-medium
30, 30
hmmaze-large
30, 20
antsoccer-arena
10, 10
cube-single
300, 300
cube-double
300, 300
scene-play
300, 300
puzzle-3x3
1000, 1000
puzzle-4x4
1000, 1000
for the puzzle, cube, and scene environments, and A 10 for the ant and humanoid environments. The baseline values aFQL are taken from Table 6 in Park et al. Park al. (2025d), and the final values for both methods are reported in Table For flog, after tuning a with the default configuration (K 8 flow steps and width (u tune K E (4, 8} and и E f0.1, 0.25] on the de defallttt- -allt-task of each environment. These values, referred f fl (Bostt,s,), are reported in Table 9. In all cases, for floq. utilize m samples of initial noise to compute the target Q-value as discussed in Section 4.2.
n x(Quua Qmax Quia) with 0.1),
Hyperparameters for online fine-tuning. Most hyperparameters (unless otherwise stated) remain similar in online fine-tuning and offline RL pre-training. For both FQL and floq, a is tuned the range [10, 100] (step size 10) for the ant and humanoid environments, and in [100, 1000] (step size 100) for the cube, scene, and puzzle environments. The selected a values are given in Table
For floq. after tuning a with the default configuration (K = 8, к = 0.1), we tune K E {4, 8, 16) and ς {0.1, 0.25) per environment. The chosen values are reported in Table 10.
Number of seeds. We ran 3 seeds for each configuration of both floq and FQL on each task, for both offline RL and online fine-tuning.
In summary, we tuned the common hyperparameters for forfflo fLoq(Def.) and FQL the same amount on the default task for each environment (following Park et al. (2025c)). For floq(Best), we additionally tuned the f oloq specific hyper-parameter K and on the default task.
23

---

K) Published as a conference paper at ICLR 2026
Table 8: Fnviromment-wise BC-Cocfficient (a) for FQI. and *1oq (Online Fine-Tuning).
Environment (5 tasks each) (FQL), a (floq)
antmaze-large
10, 10
antmaze-giant
10, 10
tmmaze-medium
80, 30
hmmaze-large
40, 20
antsoccer-arena
30, 30
cube-single
300, 300
cube-double
300, 300
scene-play
300, 300
puzzle-3x3
1000, 1000
puzzle-4x4
1000, 1000
Table 9: Environment-wise Initial Sample Range Qmay Qmin
--1
and Flow Steps (K) for oq (Best) (Offline
RL).
Environment (5 tasks each)
K) Qmas Qmin"
antmaze-large
(0.188)
antmaze-giant
(0.1,4)
hmmaze-medium
(0.1,8)
hmmaze-large
(0.1,8)
antsoccer-arena
(0.1,8)
cube-single
(0.1,8)
cube-double
(0.1,8)
scene-play
(0.1.8)
puzzle-3x3
(0.1,8)
puzzle-4x4
00.25, 8)
Table 10: Environment-wise Initial Sample Range Qmax Qmin
and Flow Steps (K) for floq (Online FT).
Environment (5 tasks each)
u-1
Quax Qmin
antmaze-large
(0.1,
antmaze-giant
(0.1.4)
hmmaze-medium
(0.1,8)
hmmaze-large
(0.1,16)
antsoccer-arena
(0.1,8)
cube-single
(0.1,8)
cube-double
(0.1,8)
scene-play
(0.1.8)
puzzle-3x3
(0.1,8)
puzzle-4x4
0.255 8)
A.7 FLOW VISUALIZATIONS
We visualize the evolution of the learned flow critic during training on cube-double Figure 17. with K = 0.1. Because raw Q-values can have large magnitudes, directly plotting them makes it difficult to assess the curvature of the learned flow. Instead, we plot advantage values, defined as the gap between the predicted Q-value obtained by integrating for k flow steps at various noise samples 2, (0), namely ds(k, 2, (k) 8, a) for E [5],, A E [I....,
.., K]. and the expected value that state-action pair after K steps, scaled linearly to à steps. Put simply, this advantage quantifies how far the intermediate estimate (k: 2, (k) s. deviates from the "straight line" path between the initial consistently non-zero
noise sample (0) and the final -avlue. We find that these deviations and vary substantially across the integration process. In many cases, they exhibit a characteristic pattern of overshooting followed by correction: larger deviations early that diminish integration
24

---

Published as a conference paper at ICLR 2026
2.0
6.3
2.5
5.
1.
60
2.5
5.
15
6.0
2.5
50
1.5
64
255
5,3
15
5.0
2.5
50
1.5
60
25
53
13
-206.5
-100
191.
-111
-145
-1313
-130
-125
Iveraledd
.022 fedd
Figure 17: Visualizing evolution of the trajectories of the flow critic during training.
Figure 18: OGBench (Park et al., 2025a) domains. These tasks include high-dimensional state and action spaces, sparse rewards, stochasticity, as well as hierarchical structure.
proceeds. These dynamics provide direct evidence that the learned flows follow curved rather than linear trajectories. We also visualize the final Q-value output x11 function of the input =0) in Figure (bottom) and find that the final x x11 depends non-lincarly on the initial noise value.
A.8 ENVIRONMENT VISUALIZATIONS
visualize OGBench tasks in Figure 18.
A.9 WALL CLOCK RUN-TIME
We report the wall clock run-times for FQL and floq in Table 11.
Table 11: Total wall-clock runtime (in 103 seconds) for FQL and floq with varying numbers of flow integration steps across four representative environments. Reported numbers correspond to 2M training steps.
floq (Flow Steps)
Environment
FQL
2
4
8
16
HM-Maze Large
14
24
28
35
50
79
HM--azze Medium
12
19
21
23
30
47
Cube-Double
16
17
Antmaze Giant
10
20
24
30
45
74
25

---

Published as a conference paper at ICLR 2026
A.10 VARIANCE OF FLOW Q VALUE SAMPLES
To assess how much the predicted Q-values vary for a fixed (s,4) a) when sampling different initial noise vectors 2, explicitly tracked the variance of the flow-matched Q-value samples during training. As shown in Figure 19, the variance decreases steadily over the course of training and eventually converges to small value, consistent with the theoretical intent of representing Dirac delta distribution.
3
Variance
2
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Training Step (1e6)
1e6
Figure 19:
Variance of Flow Q Value Samples. The variance decreases steadily over the course of training. consistent with the theoretical intent of representing a Dirac distribution.
A.11 UNDERSTANDING THE PERFORMANCE DEGRADATION AT LARGE K ON ANTMAZE-GIA GIANT (FIGURE 6)
Observe from Figure 6 that we observe degradation at large K not just for f floq but also for the monolithic ResNet approach. This makes us believe that the degradation in critic performance is largely not specific to f floq but also applicable to ResNet. Diagnostic experiments suggest that degradation with larger K stems from overfitting to very high capacity targets for both the ResNet and floc lo architecture.
For example, the best performing configuration on Antmaze-Gian (K = 4, see Figure 6) exhibits a higher TD error than the K 8 K 16 settings (see Figure 20), whose lower TD errors coincides with worse control performance. This aligns with the view that excessively small TD error often indicates overfitting than better value estimation. In support of this, when we add small amount of noise (uniformly sampled in [-0.5, 0.5)) to the TD target for the K = 16 critic, success rate healthier regime (see Figure 21).
increases by 10 percent and the TD-error also increases Taken together, these results point to shared underlying cause: very high K increases the critic capacity in a way that encourages overfitting to sharp targets, and moderate slack or noise improves the robustness needed for strong downstream performance.
A.12 EFFECT OF SIZE OF DISTILLED CRITIC
natural question is whether the capacity of the distilled critic must scale with the number of flow integration steps K. Since larger K produces more expressive flow critic, under- ander-parameterizeel distilled critic may be unable to represent faithfully, potentially degrading policy performance.
To study this, we performed controlled experiment the HMMAZB-LLAGGF task, evaluating full cross-product of: (i) distilled critic depth: a 2-layer vs. 4-layer MLP (default), and (ii) number of flow steps: K E {4, 8). For all runs, the flow critic is trained via flow matching usual, and the distilled critic is trained to regress onto the flow critic's Q-value (computed with K integration steps).
Table 12 summarizes the results. We observe that increasing the depth of the distilled critic from two to four layers consistently improves performance. Moreover,the gap between the 2-layer and 4-layer variants widens substantially for K 8. This indicates that using small distilled critic effectively "bottlenecks" the expressivity of the flow critic when K is large, confirming that distillation capacity must be sufficient preserve the benefits of larger-flow-step training.
26

---

Published as a conference paper at ICLR 2026
TD Error (Running Average) v/s K (Antmaze-Giant)
1.8
1.6
1.4
1.2
TD-Emor
1.0
0.8
0.6
0.4
0.75
1,00
1.25
1.50
1.75
Training Step (1e6)
1e6
K=4
K=8
K=16
Figure 20: TD-Error v/s Flow-Steps. The best performing configuration on Antmaze- Ginnt (K 4, see Figure 6) exhibits a higher TD error than the K or K = 16 settings, whose lower TD errors coincides with worse control performance (see Figure 6). This aligns with the view that excessively small TD error often indicates overfitting than better value estimation.
(a) % SR Change vs Target Noise
(b) TD Error vs Target Noise
Rate (SR) Change
Emar (Running Avg).
2.2
2.0
0.4
0.00
0.25
0.50
1.50 Target Noise Magnitude
0.75
1.00
1.25
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Training Step
166
Antmaze- Giant IK 16)
Target Noise 0.0
Torget Noise 0.5
Figure 21: Effect of Target Noise Success Rate And TD-Error (Antmaze-Giant, K = 16). Effect of the change in success and (b) the TD error during training. We that small
larget noise magnitude amounts of target noise (magnitude 0.5) improves performance by 10 percent and also increases TD-error to a healthier regime. However, too much target noise degrades performance, as expected.
Table 12: Effect of distilled-critic capacity for different flow-step on HMMAZE-LARGE.
shallow (2-layer) distilled critic performs worse than 4-layer critic, especially for larger K, suggesting that insufficient capacity limits the distilled critic's s ability to represent the more expressive Q-function produced by higher flow-step integration.
Distilled Critic 4 Flow Steps 8 Flow Steps
2-layer MLP
20 + 4
26 = 3
layer MI.P
24 +
52 =
A.13 DISCUSSION, CONCLUSION, AND FUTURE WORK
This work introduced floq, flow-matching approach training critics that scales Q-function capacity through iterative integration and dense supervision, achieving state-of-the-ar offline RL results and online fine-tuning results. Future directions include understanding how to set integration steps, exploiting floq's cascaded family of critics for efficiency, and combining sequential test- time scaling with ensembles. Theoretically, it is important to study how curved flows enable error correction when learning Q-values for TD-learning.
27


Reviews:
[{"review_id": "PpyDMD8yWu", "reviewer": "Reviewer_GXf4", "summary": "This work aims to improve scalability of Offline RL by introducing a flow model based Q-critic. The most closely related past work is FQL (ICML2025), which included a flow-based policy, but a regular Q-critic; the current work uses the same policy type, but adds a flow-model critic. The model works by uniformly sampling an initial noise value in the interval [Q_min, Q_max], then applying Euler integration steps to the noise with a learned velocity field for a few steps to end up at the final Q estimate. (The velocity field is also conditioned on the state and action.) Their architecture is called *floq*. Another major point in their discussion is the need for intermediate targets when training the flow which is done by a linear interpolant. The representation of the interpolant inputs is another major design choice: they use a categorical representation based on the HL-Gauss method from Farebrother (2024), and they also add a Fourier basis representation for the time t. Experimentally, they evaluate their method on OGBench (the same major testbed as used by FQL (ICML2025)), and consistently outperform FQL in terms of performance. Several ablation and sensitivity studies show the necessity of the different design choices in the current configuration, and how they alter the performance.", "questions": "I have no major question, but feel free to respond to my perceived weaknesses.\n\nA few typos:\n\nLine 138: “the our” change to “our”\n\nThe definitions of the v and q functions around lines 126 and 127 appear wrong. You add the factor $1/(1-\\gamma)$ in front, but at the same time, you added the $\\gamma^t$ inside the sum of the expectation. The original definition of these is: $\\mathbb{E}[\\sum_{t=0}^\\infty \\gamma^t r_t]$, so there will not be a $1/(1-\\gamma)$ factor in front. You get this factor, if you instead define the discounted state visitation distribution, and define the objective as the expected reward under the normalized discounted state visitation distribution (where the discounting appears in the distribution, not as a multiplier to the reward).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well written, and well presented.\n\n- Thorough experimentation. The work provides several sensitivity studies and much material for readers to examine regarding the performance. This can be a good reference for reproducing the results or building on the results.\n\n- Performance is compared on the same settings as FQL, which is an established method, the FQL results in the current paper match the published results, and the method of the current paper outperforms it clearly.\n\n- Some of the discussion was interesting, e.g., about the  usefulness of iterative computation for scaling in deeplearning, and how intermediate targets in such cases tend to be helpful.", "weaknesses": "- The computation time increases. The runtime in Appendix A.8. shows that it is quite a bit slower than FQL (up to around 7 times slower for 16 integration steps). This may reduce the practicality of the method.\n\n- Many design choices appear ad hoc to me (e.g., using HL-Gauss, and other design choices). These appear mostly engineered by testing things, without a strong theoretical backing for the choices. Such advances are also useful, but it is not clear to me whether these are likely to stand the test of time, so I list it as a weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKPycB9Wv7", "reviewer": "Reviewer_H9C5", "summary": "The paper proposes Flow-matching Q-functions (floq). floq trains a time-dependent,state-action-conditioned velocity field $v_\\theta(t,z|s,a)$ such that at the forward operator $\\psi_\\theta$ defined by the vector field equals $\\psi_\\theta(j/K, z | s, a) \\ z + \\frac{1}{K}\\sum_{i=1}^j v_\\theta(\\frac{i}{K}, \\psi_\\theta(\\frac{i-1}{K}, z| s, a) | s, a)$ with $Q(s, a, z) \\triangleq \\psi_\\theta(1, z|s,a)$ and $z_0 \\sim\\text{Unif}[l,u]$. Furthermore, $z_t = (1-t) z_0 + t y(s,a)$ where $y(s,a)$ is the boopstrapped TD-target. The authors validate floq on 50 tasks from OGBench, demonstrating approximately 1.8× improvement over baseline FQL on challenging tasks, and show it scales better than ResNets or monolithic Q-function ensembles.", "questions": "1. For a fair comparison, I think one should do scalar critic + gaussian policy, scalar critic + flow policy, floq critic + gaussian policy and floq critic + flow policy. Is there any result on floq critic + gaussian policy?\n2. floq's main contribution is on its critics. Is this applicable for other models such as REBRAC or only special to FQL?\n3. In Figure 5, why do ResNets show performance degradation at high FLOPs (e.g., antmaze-giant drops from 45% at 16 FLOPs to 31% at 32 FLOPs)? More importantly, floq also shows this degradation pattern in Figure 4 (antmaze-giant: 86% at K=4 → 70% at K=8 → 55% at K=16). What is the fundamental cause of this degradation?\n4. in Figure 6, floq achieves better performance than monolithic ensembles. Is ensemble also applicable for flow? If so, is there any benefit gain?\n5. I am not 100% convinced why HL-Gauss encoding helps here. Could you explain more in details why this helps in this scenario?\n6. The performance seems to depend heavily on hyperparameters. How easy/difficult is it to adopt the algorithm for a new test task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong ablation studies on floq \n2. Comprehensive evaluations with superior performance in difficult tasks such as puzzle-4x4 and cude-double\n3. Novel connection between flow matching and Q-learning\n4. Nice analysis of design choices in section 4.3. It gives very inspirational analysis on how to design the floq architecture.", "weaknesses": "1. Lack of theoretical understanding\n2. All implementations of floq build on FQL\n3. The idea is similar to distributional RL, but it not widely compared\n4. While design choices are grounded, they are not systematic but more results from trial-and-error", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yE6nQcb4yo", "reviewer": "Reviewer_eV3j", "summary": "This paper proposes *floq*, a flow-matching based Q-value estimation approach for offline and offline-to-online reinforcement learning. More specifically, it computes Q-values by iteratively running numerical integration of flow-matching's velocity field, allowing more fine-grained control and scaling than traditional monolithic architectures. The proposed method is evaluated on multiply OGBench tasks.", "questions": "- What is the $m$ in the TD-target equation $y(s, a)$ in line 179.\n- The authors state that Q_min and Q_max denote the minimal and maximal possible Q-value achievable on the task. But how do you calculate them in practice?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The overall paper is well written and organized, and some illustrations are clear.\n- The proposed flow-matching-based Q-learning is interesting.\n- The results are competitive, and the authors provide comprehensive analysis on their design choices.", "weaknesses": "- I can understand that applying flow-match or diffusion models can improve policy learning due to the complex, multi-modal/-peak distribution of policies (e.g., you can take different actions at a state but still get similar rewards). However, when it comes to the Q function, I am confused that if Q-value is also multimodal given a certain action.\n- The authors state that previous diffusion-based methods require supervision at every iterative step. But this can be alleviated by directly approximating the final Q-value from noise prediction, similar to DDIM.\n- It would be better to also add the flow-matching to preliminary to help readers understand the following method smoothly.\n- Although OGBench tasks are generally more challenging than standard D4RL, the comparison is still needed to illustrate the robustness of *floq*. In addition, more diffusion-based approaches should also be added to the comparison.\n- There are repeated citations for the same \"flow q-learning\" paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PRCNDtXApO", "reviewer": "Reviewer_Jc59", "summary": "The paper proposes a novel way of applying flow mathching in offline RL by predicting the Q-value function with TD learning: for that they introduce a latent variable $z$ that at $t=0$ is sampled from a uniform distribution and by following the learned velocity $v_{\\theta}(t, z)$ we should obtain $\\delta(Q(s,a))$  at $t = 1$. To learn the velocity field $v_{\\theta}(t,z)$, this approach suggests using the interpolant between the initial noise $z$ and TD-bootstrapped Q function at every step of numerical integration of the flow.  The paper builds on top of the previous work that considered flow matching for policy learning and showed that using flow matching for Q learning can bring up to x1.8 better results on some hard tasks from OGBench, where hardness is measured with respect to FQL baseline. This approach is contrasted to monolithic way of learning Q and over the series of ablation studies, authors demonstrate that this improvement cannot be attributed to just an increase of compute. Empirically, the authors show that the main ingredients to make flow matching work for TD learning are 1) the careful choice of support interval for $z$ , 2) using categorical representation for the latent input $z(t)$ with HL-Gauss encoding of high variance.\n\nThe overall idea is original and results in a clear improvement over the monolithic way of learning Q-function, but more explanation/investigation is needed to understand the degraded performance when large $K$ are considered. Moreover, the paper can improve its writing, while providing more details on the experimental setup in the main paper. For all these reasons, I recommend a weak accept.", "questions": "See weaknesses. More questions:\n\n  1. Can floq be adjusted to distributional RL setting, as currently flow matching is used to approximate a dirak distribution over $Q(s,a)$, while it can be seen as a natural framework to learn distributions over $Q$?\n2.  How do you measure Q_min and Q_max?\n3. What is a difference between default tasks and all tasks in OGbench?\n4. (L430-431) Can floq be parallelised though? Because, even if the cumulative compute time is the same, when it is parallizable you can expect that you can gain some speedup, so the actual time will reduce roughly by number of processes P. \n5. How online fine-tuning is performed, how the data is collected for online finetuning?\n6. If flow matching is done with an ODE integrator why not to try different integration schemes? According to you, can it remedy the degradation of the performance observed for the cases with the high curvature.\n7. How different are $Q(s,a)$ for fixed $s$ and $a$, but with different initial $z$? Is there a high discrepancy, even if they are trained to represent dirak distribution?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The idea of using flow matching for TD learning is very interesting and further extends the works on adapting flow matching techniques to RL. The paper shows that this approach first of all allows us to reach new state-of-the-art results for many tasks from OBbench constrained to a single task. Secondly, they showed that this algorithm proposes a good way to further scale offline RL and it is compared to other ways of scaling such as increasing the compute power in sequential way (comparison to ResNet) and in parallel way (comparison to ensemble methods). The authors also perform a vast ablation study to justify their design choices and discovering a set of default hyper parameters that work uniformly well for all tasks considered in the experiments.", "weaknesses": "I found that a lot of experimental details were not clear enough or transparent while reading the paper. What is success-rate in OGbench? Is success-rate the only metric, because in Figure 7 you mention a generic performance metric. How do you extract policy from the learned Q-furnction? Those details become more clear after consulting the appendix, but a short summary in the main paper is necessary. Moreover, it is only upon reading the appendix, it becomes clear that floq essentially builds on top of FQL and shares most of its implementation details except for training the Q function, which in FQL is done in monolithic way. Thus, in practice floq needs to distill its learned Q function from flow matching into a single MLP that can be used to obtain approximated Q in a single step. In this regard, I would like the authors to clarify whether they use a distilled Q function for all their experimental results or they use sometimes a Q function obtained directly from flow matching. \n\nMoreover, I found the study performed in Section 5.3 in the point 1) a bit unsatisfactory. First of all, I would like some more details on this experiment’s setup: does $K$ correspond to a number of integration steps used during the training and then the results are reported for the distilled Q function? From Figure 4, we see that using larger number of $K$ leads to a significant degradation on one of the tasks, and the only explanation that authors provide is that it might be overfitting, but no further analysis to confirm this hypothesis is performed. In this case, why cannot be underfitting? To my mind, you might need different number of training steps to train floq with small $K$ vs large $K$. Or you might need to consider larger networks for distillation to represent floq with $K>8$. I believe this question should be further explored, as, on the first glance, the choice of $K$ should only affect the granularity of the flow and thus having worse performance with larger $K$ is contra-intuitive.\n\nAnother possible limitation is that the authors only considered OGbench for their experiments, which is a dataset designed originally for goal-conditioned offline RL. The authors argue that the most common dataset such as D4rl has been saturated, but in this regard, why not considering d5rl for example?\n\nFinally, the authors only test their algorithm on 3 seeds, which is not enough to get reliable statistics, especially for multi-agent comparison https://arxiv.org/abs/2306.10882 (see App.G). Even if it is just for a few tasks, but performing comparisons with more seeds is important for more conclusive results.\n\nMinor:\n- L138 remove the in front of our\n- L158 refer to the papers that use averaging/doing ensembles for performing TD\n- Section 5.3: the order of the questions is non-linear: point 2) is after 3)\n- L1025: helps TO mitigate\n\n\n\nWeird sentences:\n- L194-195: Interestingly, we find that this problem can is a result of two peculiarities associated with applying flow-matching to TD\n- L235: When we then run (imperfect) TD-loss training on these interpolants, \n- L227-229: When the width of the interval $[l,u]$ is small, and the overlap between this interval and the range of target $Q$-values we hope to see is minimal, we would expect to see more straight flow traversals, that might be independent of interpolant $z$.\n- L262-263: We show in our experiments THAN doing so helps substantially by encouraging the network to meaningfully utilize this time.\n- L437: To answer this question, we trained a variant of floq where we supervised the velocity field was supervised only at t= 0 and used only a single flow step.\n- L444: oneintegration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)