Q-Learning with Adjoint Matching
Authors: Qiyang Li
Sergey Levine
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
flow-matching
Primary area: reinforcement learning

Abstract: We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm that tackles a long-standing challenge in continuous-action RL: efficient optimization of an expressive diffusion or flow-matching policy with respect to a parameterized Q-function. Effective optimization requires exploiting the first-order information of the critic, but it is challenging to do so for flow or diffusion policies because direct gradient-based optimization via backpropagation through their multi-step denoising process is numerically unstable. Existing methods work around this either by only using the value and discarding the gradient information, or by relying on approximations that sacrifice policy expressivity or bias the learned policy. QAM sidesteps both of these challenges by leveraging adjoint matching, a recently proposed technique in generative modeling, which transforms the critic's action gradient to form a step-wise objective function that is free from unstable backpropagation, while providing an unbiased, expressive policy at the optimum. Combined with temporal-difference backup for critic learning, QAM consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.

Published as a conference paper at ICLR 2026
Q- LEARNING WITH ADJOINT MATCHING
Qiyang Li UC Berkeley gcli@berkeley.c edu
Sergey Levine
UC Berkeley
sylevine@@erkeelyy.o edu
ABSTRACT
We propose Q-learning with Adjoint Matching (QAM), novel TD-based reinforce- ment learning (RL) algorithm that tackles aug-standing challenge in continuous- action RL: efficient optimization of expressive diffusion or flow-matching policy with respect to a parameterized Q-function. Effective optimization requires exploit- ing the first-order information the critic, but challenging to so for flow diffusion policies because direct gradient-basee based optimization via backpropaga- tion through their multi-step denoising process is numerically unstable. Existing methods work around this either by only using the value and discarding the gradi- ent information, or by relying on approximations that sacrifice policy expressivity bias the learned policy. QAM sidesteps both of these challenges by leveraging adjoint matching, a recently proposed technique in generative modeling, which transforms the critic's action gradient form step-wise objective function that is free from unstable backpropagation, while providing unbiased, expressive pol- icy at the optimum. Combined with cemporal-differeet backup for critic learning, QAM consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL. Code: github. http//oiiniaangiii.//c
INTRODUCTION
Actions
Offline (50 tasks)
denoling
DAM-E
Behavior Policy
a
DAM-I
2
DSRL
CBOL-L
@6. m)
FOL
8AM
35
Fine-tuned Flow Policy
Artion Gradien
Crocs
CROL-M -M
34
35
V. (.....)
FEdt
ngg ())
CCOL
30
LOAM(()
FBRAC
N
Fine Fine-tune via Adjoint Matching
10
30
50
Aggregated Score
Figure 1: QAM: Q-learning with Adjoint Matching. Left: QAM uses adjoint matching (Domingo- Enrich et al., 2025) that leverages the critic's 's action gradient directly to fine-tune flow policy towards the optimal behavvor-constrained policy: Tt(: 8) xx Tt(- Q(o.), Right: Aggregated score for offline RL on 50 OGBench (Park et al., 2025a) tasks.
A long-standing tension in continuous-action reinforcement learning (RL) especially in the offline/oflline-ttooliit setting, is between policy expressivity and optimization tractability with re- spect to critic (i.e., Q(s, a)). Simple policies, such as single-step Gaussian policies, are easy to train, since they can directly leverage the critic's action gradient (i.e., VaQ(s, a)) via the reparameter- ization trick (Haarnoja al., 2018). This optimization tractability, however, often comes the cost of expressivity. Some of the most expressive policy classes today, such as flow or diffusion policies, generate actions through multi-step denoising process. While this allows them to represent complex,

---

Published as a conference paper at ICLR 2026
multi-modal action distributions, leveraging the action gradient requires backpropagation through the entire denoising process, which often leads to instability (Park et al., 2025c). Prior work has there- fore resorted to either (1) discarding the critic's 'isi's action gradient entirely and only using its value (Ren et al., 2025; Zhang et al., 2025; McAllister et al., 2025), or (2) distilling expressive, multi-step flow policies into one-step learning efficiency and often under-perfore methods that use the critic's action gradient (Park et al., 2024; 2025c), while the latter compromises expressivity. This raises question: can we somehow keep the full expressivity of flow policies while incorporating the critic's action gradient directly into the denoising process without backpropagation instability?
approximations (Park et al., 2025c). The former sacrifices
One might be tempted to directly apply the critic's action gradient to intermediate noisy actions within the denoising process, as in diffusion classifier guidance (with the critic function being the classifier) (Dhariwal & Nichol, 2021). Intuitively, this blends two generative process together: one that generates a behavior action distribution, and another that hill-climbs the critic to maximize action value. While this approach bypasses the backpropagation instability and retains full policy expressivity, it relies on the assumption that the critic's gradient at a noisy action is a good proxy for its gradient at the corresponding denoised action. In practice, this assumption often breaks down: when the offline dataset has limited action coverage, the critic is well-trained only on a narrow distribution of noiseless actions, rendering its gradients unreliable for intermediate noisy actions that are out of distribution.
We propose Q-learning with Adjoint Matching (QAM), new RL algorithm that applies adjoint matching (Domingo-Enrich et al., 2025), a recently developed technique in generative modeling, to policy optimization. QAM leverages the critic's action gradient for training flow policies to maximize returns subject to a prior constraint (e.g., behavior constraint) (Figure 1). In general, such a constrained optimization problem on a flow model can be formulated as a stochastic optimal control (SOC) objective, which can be solved by using the continuous adjoint method (Pontryagin et al., 1962). However, this standard formulation has the same loss landscape as directly backpropagating through the SOC objective, causing instability. Instead, we leverage modified objective from Domingo-Enrion Enrich et al. (2025) that admits the same optimal solution, but does not suffer from the instability challenge. At high level, the critic's gradient at noiseless actions is directly transformed by a flow model constructed from the prior, independent from the possibly ill-conditioned flow model that is being optimized, to construct unbiased gradient estimates for optimizing the conditonnd velocity field at intermediate denoising steps. This allows the flow policy's velocity field to align directly with the optimal state-conditioned velocity field implied by the critic and the prior, without direct and potentially unstable backpropagation, while preserving the full expressivity of multi-step flow models. By combining this policy extraction procedure with a standard lemporal-difference (TD) backup for critic learning, QAM enables the flow policy to efficiently converge to the optimal policy subject to the prior constraint. In contrast, approximation methods that rely on the critic's gradients at noisy intermediate actions lack such convergence guarantees.
Our main contribution is a new TD-based RL algorithm that leverages adjoint matching to perform policy extraction effectively on a critic. Unlikely prior Q-learning methods with flow-matching that rely on approximations or throwing away the action gradient of the critic altogether, our algorithm directly uses the gradient to form an objective that at convergence recovers the optimal behavior- regularized policy. We conduct a comprehensive empirical study comparing policy extraction methods for flow/diffusion policies, including recent approaches and novel basclines, and show that QAM consistently achieves strong performance across both offline RL and offline-to-onlim RL benchmarks.
2 RELATED WORK
RL with diffusion and flow policies. Diffusion and flow policies have been explored in both policy
gradient methods (Ren et al., 2025) and actor-critic methods (Fang et al., 2025b; Kang et al., 2023; Chen et al., 2024c;a; Lu et al., 2023b; Ding et al., 2024b; Wang et al., 2023; He et al., 2023; Ding & Jin, 2024; Ada et al., 2024; Zhang et al., 2024; Hansen-Estrre
et al., 2023). The key challenge of leveraging diffusion/flow policies in TD-based RL methods is to optimize these policies against the critic function (i.e., ((s, a)). Prior work can be largely put into three categories based on how the value function is used:
(1) Post- post-processing approaches refine the action distribution from a base diffusion/flow policy with rejection sampling based the critic value (Hansen- Estruch et al., 2023; Mark al., 2025; Li et al.,
2

---

Published as a conference paper at ICLR 2026
2025; Dong al., 2025), using additional gradient steps hill climb the critic (Mark al., 2025) (l.e. ag + a + VaQ(s, a)). These approaches often reliably improve the quality of extracted policy but at the expense of additional computation during evaluation or even training (i.e., rejection sampling for value backup target (Li et al., 2025; Dong et al., 2025)). Alternatively, one may train a residual policy that modifies a base behavior poliey in either the noise space (Singh et al., 2021; Wagenmaker et al., 2025) or in the action space directly (Yuan et al., 2025; Ankile et al., 2025a;b; Dong et al., 2025).
(2) Backprop-based approaches perform direct backpropagation through both the critic and the policy ( (Wang et al.. 2023; He et al., 2023; Ding & Jin, 2023; Zhang et al., 2024; Park et al., 2025c; Espinosa- Diice et al., 2025; Chen et al., 2025). While this is the most-straightforrard implementation- wise, it requires backpropagation through the diffusion/flow policy's ' denoising process which has been observed to be unstable (Park et al., 2025c), or instead learns a distilled policy (Ding & Jin, 2023; Chen et al., 2024b; Park et al., 2025c; Espinosa-Dice et al., 2025; Chen et al., 2025), at the expense of policy expressivity.
(3) Intermediate fine-tuning approaches, which our method also belongs to, mitigate the need of the stabelit/leprressiitty trade-off in backprop-based approaches by leveraging the critic to construct an objective that provides direct step-wise supervision to the intermediate denoising process (Psenka et al., 2024a; Fang et al., 2025b; Ding et al., 2024a; Li et al., 2024b; Frans et al., 2025; Zhang et al., 2025; Ma et al., 2025; Koirala & Fleming, 2025). While these approaches remove the need for hackpropagation through the denoising process completely, the challenge lies in carefully crafting the step-wise objective that does not introduce additional biases and learning instability. Compared to prior methods that either rely on approximations (Lu et al., 2023a; Fang et al., 2025b) that do not provide theoretical guarantees (see more discussions in Section C) or directly throwing away the critic's action gradient (and use its value instead) (Ding et al., 2024a; Zhang et al., 2025; Ma et al., 2025; Koirala& & Fleming, 2025), we leverage adjoint matching allows us to use the critic's s action gradient directly to construct an direct step-wise objective for our flow policy that recovers the optimal prior regularized policy at the optimum of the objective.
Homingo--Enicce et al., 2025) which
Offlinettoonlinn reinforcement learning methods focus on leveraging offline RL. to first pre-tra train on an offline dataset, and then use the pretrained policy and value function(s) as initialization to accelerate online RL (Xie et al., 2021; Song et al., 2023; Lee et al., 2022; Agarwal et al., 2022; Zhang et al., 2023; Zheng et al., 2023; Ball et al., 2023; Nakamoto et al., 2024; Li et al., 2024a; Wilcoxson et al., 2025; Zhou et al., 2025). While it is possible to skip the offline pre- pre-training phase altogether and use online RL methods directly by treating the offline dataset as additional off-policy data that is pre-loaded into the replay buffer (Lee et al., 2022; Song et al., 2023; Ball al., 2023), these methods often under-perform the methods that leverage explicit offline pre-training, especially on more challenging tasks (Nakamoto al., 2024; Park al., 2025c). Our method also operates in this regime where we first perform offline RL pre-training and then perform online fine-tuning from the offline pre-trained initialization. In addition, we follow a common design in prior work where the same offline RL objective is used for both offline pre- trainingn and online fine-tuning (Kostrikov et al., 2022: Fujimoto & Gu, 2021; Tarasov et al., 2023; Park al., 2025c). While we focus on evaluating our method in the offine--to-nnlin
RL setting, the idea of using adjoint matching to train an expressive flow policy can be applied to other settings such as online RL.
For additional related work on diffusion and flow-matching with guidance, see Section C.
3 PRELIMINARIES
Reinforcement learning and problem setup. We consider a Markov Decision Process (MDP). M (S, P, R, p). where is the state space, A = A (A Z+) is the action space. S x A + As is the transition function, 7 E 1) is the discount factor, R: S x R - IR is the reward function, and μ E AA is the initial state distribution. We have access a dataset D consisting of a set of transitions D = {(s), at, s/ r.))!21 where s' ~ P(- s, a) and r = R(s, a). Our first goal (offline RL) is to learn a policy Ta : S + AA from D that maximizes its expected discounted return.
U= =
nh R(sks aR)
The second goal (offline-to--ne to-online RL) is to fine-tune the offline pre-trained policy T. by continuously interacting with the MDP through trajectory episodes with a task/environment dependent maximum
3

---

Published as a conference paper at ICLR 2026
episode length of H (i.e., the maximum number of time steps before the agent is reset to p). The central challenge of cilline-to-onling RL is to maximally leverage the behavior prior D to lear as sample-efficiently (high return with few environment interactions) as possible online.
Flow-matching generative model. A flow model uses a time-variant velocity field v IR x [0, Rd to estimate the marginal distribution of denoising process from noise, Xu -0, Ta). data, X = D, at each intermediate time t E [0, 1]:
X, = (1 - X) + tX
(2)
In particular, the flow model approximates the intermediate X₁ via an ordinary differential equation (ODE) starting from the noise: Xo N:
ax, = f(x,, t)dt.
(3)
Flow models are typically trained with a flow matching objective (Liu et al., 2023):
(O) E2--20001 [[]fa((1 - 1)20 + (xi, 1) 21 + xal12)
(4)
where any optimal velocity field, Vg., results in X, where its marginal distribution (r())) exactly recovers the marginal distribution of the original denoising process X,, PD(())), for each / [0, (Lipman et al., 2024). Furthermore, one may use the Fokker-Planck equations to construct family of stochastic differential equations (SDE) that admits the same marginals as well:
dx, (1(X
oft 2(1- t) ((x0))xx////) + dt a.d.,
with B, being a Brownian motion and o. O being any noise schedule.
Adjoint matching is a technique developed by Domingo-Enrinn et al. (2025) with the goal of modifying a base flow-matching generative model fo such that the resulting flow model fe generates the following tilt distribution:
Pe (X1) (e(())e
where pa the tilt distribution induced by Sa and Pæ is the base distribution induced by Sas and Q Rdd IR is any value function that up-weights or down-weights the probability of each example in the domain Rd Domingo-Enrich et al. (2025) uses a marginal-preserving SDE with a 'memory- less' noise schedule (i.e,, Xo and Xi are independent), aN = / 1)/l:
dx, = (2, - X,/t) dt + /2 - - t)/tdB.,
where the minimizer the following stochastic optimal control equation (with X, sampling from the joint distribution defined by the SDE in Equation (7)),
[."
2 o
(a0--SsXX.. (xx - di - Q(x)))
L(O)= = Ex=(xi)
(8)
gives the correct marginal rilt distribution for X1 (Domingo-Enrich et al., 2024):
p(X1) (e(X))e
(9)
Let the adjoint state be the gradient of the tilt function applied at the denoised Xy:
g(X,t Vx. [. ( llfo(Xe, (1lffxe
fo(Xx, ( dt' Q((x))
(10)
where X = {x,}, ). the random variable that represents the SDE trajectory, and satisfies
(x(X)t) Xx (2 X,
2 - di V [fo(X, - (I(. (11)
dt
'The adjoint-matching paper -Domingo-Enrich Enrich et al., 2025) uses more general framework with X; = 3.Xo + XxX1. For simplicity, assume the special where as t, & and de = /2(1 f)/t.
4

---

Published as a conference paper at ICLR 2026
with the boundary condition g t - We can compute the adjoint states by stepping through the reverse ODE (which can be efficiently computed with the vector-Jacobiat product (VJP) in most modern deep learning frameworks). Then, it can be shown that the stochastic optimal control loss in Equation (8) is equivalent to the 'basic' adjoint matching objective below:
LBAM Ex (2UEOXX00 - fx(Xa,1)) + 0.9 (X, (1, 11de
(12)
The optimal fe coincides with the optimal solution in the original SOC equation (Equation (8)), which gives the correct marginal distribution of X1 as result. However, the objective is equivalent to the objective used in the continuous adjoint method (Pontryagin al., 1962) with its gradient equal to that of backpropagation through the denoising process. Instead, Comingo-Ennice et al. (2025) derive the 'lean' adjoint state where all the terms the adjoint state that are zero the optimum are removed from the original adjoint state. The 'lean' adjoint state satisfies the following ODE:
dg(X,t) t) Vx. [2fa t) X./t) (X,, t)dt,
(13)
with the same boundary condition 3(X,1) -V Q(x))
Note that computing the 'lean' adjoint state only requires the base flow model [x and no longer needs use fa (Xee f) needed in either the basic adjoint matching objective (Equation (12)) or naive backpropagation through the denoising process. The resulting adjoint matching objective is
(0) Ex [,," [2(fo(X,,))
fo(X + ox0(X,
(14)
where again X is sampled from the marginal preserving SDE in Equation (7). Because the terms omitted in the 'lean" adjoint state are zero at the optimum, and thus do not change the optimal solution for fa. Thus, the optimal solution for the adjoint matching gives the correct tilt distribution.
4 Q-LEARNING WITH ADJOINT MATCHING (QAM)
In this section, we describe in detail how our method leverages adjoint matching to directly align the flow policy to prior regularized optimal policy without suffering from backpropagation instability.
To start with, we first define the optimal policy that we want to learn as the solution of the best policy under the standard KL behavior constraint:
arg max, m R (Q(, s.t. DT |l (T) S c(s).
(15)
or equivalently, for an appropriate T(s). Nair et al. (2020) show that
"" (a 8) n (L)
(16)
where 7 : S + R is the inverse temperature coefficient that controls the strength of the behavior constraint at each state.
We approximate the behavior policy using a flow-matching behavior policy, fs: RA that is optimized with the standard flow-matching objective:
: S x R x [0
Les (3) = B(e.a)-D..v0.),)-- [[]sa(s, (1 t) + ta, t) - a + x1
(17)
We then parameterize our approximation of the optimal policy another flow model fo S S TRA
[0, 1] + A and solve the following SOC equation:
[L'
2 of llfo(s, ar fa(s, a, t) ((s)
L(#)
E2 D.a.
(18)
where a, is defined by the following 'memory-less' SDE (e.g., ao independent from a1):
da. (2fo(s, ai/t)dt /2(1 t)/td
(19)
5

---

Published as a conference paper at ICLR 2026
Similar to the derivation by Domingo- Enrgchh---rchhhee et al. (2025), the memory-less property allows to directly conclude that the SOC equation has the optimum at
(- s) x a- s)e s)e" (*)
(20)
where T-(- ()) and mT(: 8) are the corresponding action distributions defined by fo and fa.
However, directly solving the SOC equation involves backpropagation through time that introduces additional instability. To circumvent this issue, we use the adjoint matching objective proposed by Domingo- Enrich et al. (2025) (Equation (14)) to construct similar objective for policy optimization
in our case:
V., | ac,t) - fa(ss a,, t))/ o. (lact
(AM = E.nD.(a))
(21)
where &e is the 'lean' adjoint state defined by a reverse ODE constructed from or 's:
dğ = - Va [2fa ae, g,dt.
(22)
Unlike the original SOC objective (Equation (18)) from which calculating the gradient requires backpropagating through an SDE, which suffers from stability challenges, the adjoint matching objective is constructed without backpropagation. Instead, uses the behavior velocity field fe to calculate the 'lean' adjoint states h through a series of VJPs for every SDE trajectory {az Jt- which are then used to form a squared loss the adjoint matching objective. Mathematically, backpropagation can also be interpreted as calculating the adjoint states through series of VJPs, with the key distinction that the VJPs are computed under the flow model that is being optimized (i.e., fo). This is important distinction because for direct backpropagation, any ill-conditioned action gradient in fe (i.e., Vafe(s, a, f)) would compound over the entire denoising process, contributing to the ill-condition-nes: of the overall gradient to the parameter 6, which the whole optimization process. In contrast, in adjoint matching, the action gradient of fo has no contribution to the overall gradient to which allows the optimization be much more stable.
in turn destabilize
Finally, we combine the policy optimization with the standard TD>-lermiig objective:
L(Ø)= = Es,a,'' [(((s) a) - r - r- d')] a' (ODE(f((s',)))))..) ). N)
(23)
where ODE(fa(s', -) db) = So fo(s', di, t)dt (sampling an action sample from the velocity field fo(s', :(('':;)))) :)) and Φ the exponential moving average of with a time-constant of X 0.005 (i.e., 0s+1 < (1 λ)Ø; + Ao; for each training step i).
Practical considerations. In practice, following Domingo-Enrich et al. (2025), we solve both the SDE and the reverse ODE with discrete approximation and fixed step size of h 1/T, where T is the number of discretization steps. In particular, with ag ~ N and ž, ~ N, E {0, h (T- 1)h}. the forward SDE process is approximated by
a++h + at (2fo ae, at/t) + 22 22(11t)(ta
(24)
We set the boundary condition Ø1 - (s,s, a1), where temperature coefficient т to modulate the influence of the prior Ty and we additionally clip the magnitude of the parameter gradient dement-wiss by for numerical stability. The backward adjoint state calculation process is then approximated by
state- state-nddependent inverse
- + J. + VJP(V.. (2fg(s, a..t),..))
(25)
with V.J((V,6()))2 x) [V Vyb(y) being the vector- cector-acoobian product and it can be practically implemented by carrying the 'gradient' x with backpropagation through f. For the critic, we use an ensemble of K = 10 critic functions .!.... . and use the pessimistic target value backup following Fang et al. (2025a) (originally inspired by Ghasemipour et al. (2022)). The loss function for each ..j E 21, K) is
l( (r-m
(s, -r -r [mmean (s',a') - (Qstas''
(26)
where Quean (d) = 1 E. (s' al), Quta(ss a') *. Ex Ex(e (8', a!) Quean as)),
and a' is the action sampled from the fine-tuned flow model: 10(8,';), For all our experiments,

---

Published as a conference paper at ICLR 2026
do not separate training process for fg and instead training it at the same time fo and Qp. following Park et al. (2025c); Li et al. (2025). For all our loss functions, the transition tuple (s, a, s',r) r) is drawn from D uniformly. During offline training, D is the offline data. During online fine-tuning, D is combination of the offline and online replay buffer data without any re- re-weighting. A pseudocode of our algorithm is available in Algorithm 1 in the Appendix.
Theoretical guarantees. As our algorithm builds off from Domingo-Enrich et al. (2025), we can directly extend their theoretical results our setting (see in Section G)-as long as the loss function LAM is optimized to convergence (i.e., with / = 0), the learned policy coincides with the optimal cobstiio-c policy (i.e., x²* (a 8) x T s) exp(TQ((s,))))) a)) in Equation (16)).
Expanding the constraint set beyond the KL behavior constraint. While our method, QAM, is guaranteed to converge the optimal Sehavior-constrainge policy (i.e., Oc Tа exp(Q)), can struggle to represent any policy that has a support mismatch with the behavior policy. For tasks where the optimal actions have extremely low probability under the behavior distribution, our algorithm may have trouble to represent optimal policy for these tasks. In practice, we often find it to be beneficial to relax the behavior constraint in QAM to allow actions that are 'close' (in the action space) to the behavior actions. This amounts to optimizing the policy under the following constraint:
arg mIaX Ba (-)) [Q(s, (s)) a)]
8.1.
( T&) S <($), (T, 4) S Ca.
(27)
where W is the q- Wasserstein distance between म. I under some metric d : A x A+ RJ. some constant that controls the strength of the constraint:
and 0a is
(T ( s)) ==
inf [a [d(a, (((a,)))) oEC(T,ā)
(28)
where C(TT, #) is the set of all couplings between æ and #. With such design, x not only respects KL behavior constraint, but can also represent actions that are 'close" (in the geometry of the action space) to the behavior policy. While directly optimizing this objective is intractable, we approximate it by using the solution from QAM (i.e., Ta) to approximate the optimal न. allowing us to transpose the problem into a eehavior-constrainee policy optimization problem with xe being the base policy:
π Tw arg max, maxx,eeex(ieee (1) a)]
s.t. (T(T(a s), ao(a s)) S oa.
(29)
In practice, we explore two practical variants for optimizing ı with different choices of q and d and approximations.
The first variant, QAM-FQL: For our first variant, we use q= 2 and d(a, a) = ||a - a||2, which allows us to directly use a prior flow RL method, FQL. (Park et al., 2025c). In particular, we learn 1- step noise-conditioned policy, a. (as represented by (s, =) : S x R m> TR and optimize the following objective (with a being tunable BC coefficient).
Fqu (w) = Ez-N [=Q0(8, (s, =)) + allmw 2) - ODE(So(s, 2]]]
(30)
As direct application of the theoretical result in Park et al. (2025c), the second term in the equation above is an upper-bound on squared Wasserstein distance W2 (Tw, No) between the I-step noise- conditioned policy nw and the optimal sehavior-constriiee exactly the desired constraint (i.e., W2 (Tw, TT) I (w). We then use this 1-step policy T. both to interact with the environment and to compute value targets (i.e., Q(s', a' u(s, 2 N))).. Intuitively, the -sseep policy is optimized to remain close to the QAM-fine-tuma flow policy while also maximizing the action value under the current critic. We call this first variant of our method QAM-
policy obtained via QAM, mo. This
The second variant, QAM- EDIT: Our second variant uses q = and d(a, ā) = |la - م|| To implement this, we learn an edit policy n. (A. s, a) that predicts an action edit A. to modify the action output from Tg by at most 0a in the L distance (i.e., || Ala boo I a.). This ensures that the Wasserstein distance constraint (i.e., W1 (Tw; mo) I oa) is enforced by construction. In practice, train an edit policy (Dong et al., 2025), (- ā). where it first predicts a Gaussian distribution, then squashed by tanh function to between - and 1, and finally rescaled be between -a, and oa as the action edit. To avoid the degenerate case where the edit policy always outputs the boundary action edits (i.e.. -o or oa), we also include the standard entropy term from Haarnoja et al. (2018) with the automatic entropy tuning trick such that the policy's 's entropy is close to a constant value of H(T=)) Hearges A/2. T (w) E.S, (a Qo(s, Ao + a) + 17 log TH (A. 8, a,))),
(31)
I(n) = (1 (log Tw |8, a)] - Hearget]
7

---

Published as a conference paper at ICLR 2026
where (- s) for both losses and again A. is restricted to be || da- Similarly to our first variant, we use Tu both to interact with the environment and to compute value targets. A potential side benefit of the entropy term is that it can also encourage action diversity that can be helpful for online exploration and fine-tuning. As we will show in our experiments, this QAM variant not only excels in offline RL but also fine-tunes effectively online.
5 EXPERIMENTS
We conduct experiments to evaluate the effectiveness of our method on a range of long-horizon, sparse-reward domains and compare it against set of representative baselines.
Domains and datasets. We consider 10 domains (5 tasks each) from OGBench (Park et al., 2025a): scene, puzzle-3xt -x33 (p33), puzzle-4x4 (p44), cube- (c2), cube-triple (c3). cube- quadruple (c4), humanoidmaze-mooen Semaaze-larg (al), and antmaze-gia -giant (ag). For antmaze- * and humanoi dmaze-* we use the default navigate datasets. For scene, puzzle-*, and cube-*, we use the default play datasets except for c4 and p44 where we use the larger 100M-size dataset from Park et al. (2025b), and use the sparse reward definition for {p33, p44, scene), following Li et al. (2025). In addition, for all {cube-*, scene-*, p33-*, p44-*} domains, we follow Li et al. (2025) to learn action chunking policies with an action chunking size of h = 5. Action chunking policies output high-dimensionnl ac- tions that exhibits much more complex behavior distribution, where the policy extraction becomes critical. Since our approach primarily focuses on the policy extraction aspect, these domains make an ideal testbed us compare our method to prior work. See Section D for more details on these domains.
(hm), numanoidmaze-larg (hl),
Comparisons. We carefully select 13 representative, strong baselines that can be roughly catego- rized into the following 5 categories (1) Gaussian: ReBRAC (Tarasov al., 2023), (2) Backprop: FBRAC (Park et al., 2025c) (backprop through the flow policy's ' denoising step directly), FQL (back- prop through 1-step distilled policy) (Park al., 2025c); (3) Advantage-weighted: FAWAC (Park et al., 2025c) (advantage weighted actor critic, AWAC (Nair et al., 2020), with flow policy): ): (4) Guid- ance with the critic's action gradient: DAC (Fang et al., 2025b), QSM (Psenka al., 2024a), and (three variants of classifier guidance- based methods inspired by Dhariwal & Nichol (2021)); (5) post-processing-baae baae::: DSRL (Wagemmaker et al., 2025), FEdit (flow Gaussian edit policy from Dong et al. (2025)), and IFQL (flow counterpart of IDQL (Hansen- Estruch et al., 2023)). For offline-t-online evaluations we additionally compare with RLPD (Ball et al., 2023). Finally, we compare with BAM, direct ablation from method QAM where we use the 'basic' adjoint matching (and hence the abbreviation BAM) objective in Equation (12) in- stead of the adjoint matching objective in Equation (14), and keep the rest of the implementation exactly the same. We categorize it as "backprop" method because its gradient is equivalent to that of backpropagating through the memory-less SDE
discuss above in Section 3.
Among them, RLPD does not employ any behavior constraint, so we directly train from scratch online. To make the comparison fair, we use K 10 critic networks, pessimistic value backup with - = 0.5 (except on humanoi dmaze-large where we find 0 to work better), no best-of-N sampling (i.e., N = 1) for both our methods and our baselines except for IFQL where best-of-N is used for policy extraction. See Section E for detailed description and implementation detail for each of the baselines. We also include the sppeiii hyperparameters for each baseline in Section
6 RESULTS
In this section, we present our experimental results to answer the following three questions:
(Q1) How effective is our method for offline RL.
Table reports the offline RL performance across 10 different domains (50 tasks in total). QAM outperforms all prior methods with an aggregated score of 44. In particular, among all the baselines, FAWAC also converges to the optimal behavior-constraaned distribution (Equation (16) but does not leverage the action gradient of the critic, resulting in much worse aggregated score of 8; BAM also leverages the SOC objective but optimizes it with backpropagation through the SDE chain (without using the 'lean' adjoint), leading to worse aggregated score of 35. It is worth noting that FBRAC also leverages backpropagation through time (BPTT) similar to BAM, but performs much worse than BAM. The main distinction between BAM and FBRAC is that BAM leverages the SOC formulation
8

---

Published as a conference paper at ICLR 2026
a1
ag
hm
h1
8C8D8
p33
p44
c2
c3
c4
a.l
tacko
tacke
taaks
5 sa**a
sooks
raalis
racks
takks
GAUSSIAN
ReBRAC
9-1
57
69
17
65
79
9
10
FERAC
2
0
39
50
202
11
BACKPROP
1
8
0
215
0
35
FQL
76
0
68
70
3
46
3
22
36
ADV. WEIGHTED
FARAC
17
0
23
38
3
0
0
0
76
38
0
30
COQL-M
74
211
1
355
GUIDANCE
CLQL-L
3
62
6
88
45
8
0
37
DAC
1.6
83
68
35.5
1
36
QSM
91
13
83
10
86
53
0
56
19
42
DSRL
61
3
53
3
99
0
74
'1
22
38
POST-PROCESSING
FEdit
2
2
1
33
IPQL.
7
86
24
100
0
11
0
34
QAM
81
18
67
11
97
100
0
64
33
44
ADJOINT MATCHING
QAM-F
83
12
65
12
95
99
6
65
14
45
QAM-R
7
59
2
97
100
39
65
8 46
Table I: Offine RL performance at 1M training steps (50 tasks, 12 seeds). Our method (QAM) and two of its variants, QAM-FQL (QAM-F) and QAM-EDIT (QAM-E) outperform all prior baselines. We use CGQL-M, CGQL-L, QAM-F, QAM-E as the abbreviations for CGQL-MSE: CaQL-Linex, QAM-FQL, and QAM-EDIT respectively. For results on individual tasks, see Table 2 Section
which enjoys the guarantee that the policy at the optimum recovers the optimal eehavior-constraigg policy, whereas FBRAC has not been found enjoy such guarantee. This suggests that carefully designing the loss function where its optimum coincides with the optimal Schavior-constraiie can be influential to the algorithm's performance. Furthermore, BAM ends up performing similarly to FQL, which addresses the BPTT instability issue by leveraging a 1-step distilled policy. This suggests that BPTT might not be as disadvantageous as previously reported Park et al. (2025c) in the best case, but can be sensitive methods proposed in the literature of diffusion/flow-matteeo matching and RL can perform very well when augmented with a behavior cloning term. We also find that Gaussian bascline, ReBRAC, while worse on manipulation tasks, often outperforms other flow/diffusion based methods on locomotion tasks. This makes ReBRAC the fifth-best method in terms of aggregated score, falling short compared to QSM and all three QAM variants. At last but not least, combining QAM with FQL and FEdit (QAM-F and QAM-E respectively) can push the performance even further, achieving an aggregated score of 45 and 46 respectively.
policy
implementation details. In addition, we find QSM, one of the earliest
(Q2) How effective is our method for offline-to-onliee fine-tuning?
Next, we take the best performing variant of QAM, QAM-EDIT, and evaluate its ability to online fine- tune from its offline RL initialization. Figure shows the sample efficiency curve (with x-axis being the number of environment steps). Our method outperforms all prior methods on cube-triple and the most robust method across the board. For example, compared to ours, QSM fine-tunes better on ag but struggles on all other tasks except on c2 where it performs similar to our method. FQL fine-tunes slightly better on ag but much slower on both p44 and c3.
(Q3) How sensitive is our method to hyperparameters? We also conduct sensitivity analyses for various components of our best performing variant, QAM-EDIT, including gradient clipping, number of flow steps (1"), critic ensemble size (K) and the temperature coefficient (r). As we show
'The original QSM implementation does not have a behavior cloning term. It only aligns the action gradient of the critic to the diffusion denoising process. This makes the vanilla QSM very hard to learn anything during the offline pre-training phase. To adapt it to our affline-to-online setting and make the comparison fair, we augment the original QSM loss with the standard DDPM loss as the hchavior cloning loss. See details in Section E.

---

Published as a conference paper at ICLR 2026
GAM-E
FERAC
DSRL OGOLL
QSM
RuBRAC RLPD
humänoidmaze-lage puzzia-4x4 wx4-sparse-1aaeee
cube-double
cube-quadruple-
100
Aggiegated Score
75
0-0
40
Online Environment Steps (x10*)
Figure 2: QAM online fine-tunes more effectively than prior methods (50 tasks, 12 seeds). We use the QAM-EDIT (QAM-E) variant for QAM and the CGQL-Linex variant for CGQL (CGQL-L) because they work well for online fine-tuning. For full results, see Figure 5 in the Appendix.
Gradi ent Clipping
Flow Steps (7)
Critic Ensembble Size (K)
Temperature (1)
Aggregated Score
100
100
100
100
60
40
40
-0-
T=
40
-0-
12
3
20
(ount
2 1x
team
-O- wthhot dappy
-0-
0-
0-
(sum)
0-
10
30
50
10
20
30
40
50
10
20
40
50
0
10
20
30
50
Environment Steps (x10')
Figure 3: Sensitivity analysis of QAM-EDIT on the representative task from each domain (12 seeds). For all locomotions tasks, we use task1. For all manipulation tasks, we use task2. We report the aggregated performance across these 10 representative tasks. Gradient Clipping: whether to use gradient clipping in optimizer; Flow Steps (T"): this parameter indicates the number of numerical integration steps that we use for the flow model. Critic Ensemble Size (K): number of critic network in the ensemble; Temperature (T): the parameter that modulates the influence of the prior. We rerun our method with 0.1x, 0.3x, 3x, and 10x the best 7 from tuning. Full results in Figure 6.
in Figure 3, all of these components contribute to our method's effectiveness. Among them, the temperature parameter (r) has the biggest impact on performance and need be tuned. For the other components, we find enabling gradient clipping and having a large critic ensemble size (K = 10) also help. Lastly, we find that setting the number of flow steps T 10 works well enough and increasing it further does not improve performance.
7 DISCUSSION
We present Q-learning with Adjoint Matching (QAM), new TD-based RL method that effectively leverages the critic's action gradient to extract an optimal prior-constrained policy while circumventing common limitations of prior approaches (e.g., approximations that do guarantee converge to the desired optimal solution, learning instability, or reduced expressivity from distillation). Our empirical results suggest that QAM is an effective policy extraction method both the offline RL setting and the offline-to-onlin RL setting, performing on par or better than prior methods. There are still practical challenges associated with QAM While QAM's 's effectiveness can largely attributed to how well it is able to leverage the critic's 's action gradient, this can be a double-edge sword-for for cases where the critic function is ill-conditioned, it could lead to optimization stability issue. Gradient clipping (as done in our method) can alleviate this issue, but a more principled method that combines both value and gradient information could further improve robustness and performance. Another possible extension is to apply QAM in weal-world robotic settings with action chunking policies. Our initial success (especially in the manipulation domains where we also leverage action chunking policies) may suggest that method might work more effectively in complex real-world scenarios.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was partly supported by DARPA ANSR and ONR N00014-25-1-2060 This research used the Savio computational cluster resource provided by the Berkeley Research Computing program UC Berkeley. We would like to thank William Chen, Kevin Frans, David Ma, Vivek Myers, Seohong Park, Michael Psenka, Ameesh Shah, Max Wilcoxson, Charles Xu, Bill Zheng. Zhiyuan (Paul) Zhou for helpful discussion and feedback on early drafts the paper. We would also like to thank anonymous ICLR reviewers for the insightful comments and discussions.
REFERENCES
Suzan Fee Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement learning. IEEE Robotics and Automation Letters (RA-L), 9:31 16-3123, 2024.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation accelerate progress. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 28955-28971. Curran Associates, Inc., 2022.
Lars Ankile, Zhenyu Jiang, Rocky Duan, Guanya Shi, Pieter Abbeel, and Anusha Nagabandi. Residual off-policy rl for finetuning behavior cloning policies. arXiv preprint arXiv:2509. 0eee22299999339 19301, 2025a.
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, and Pulkit Agrawal. From imitation to refinement-residuall RL for precise assembly. In 2025 IEEE International Conference on Roboties and Automation (ICRA), pp. 01-08. IEEE, 2025b.
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary. Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL http://githhtt com/jax- com/jax-ml/jan ml/jax.
Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is a predictor-corrector. arXiv preprint arXiv:2408.08m 09000, 2024.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In International Conference on Learning Representations (ICLR), 2024a.
Tianyi Chen, Haitong Ma. Na Li, Kai Wang, and Bo Dai. One-step flow policy mirror descent. arXiv preprint drXiv:2507.23675, 2025.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:50098- 50125, 2024b.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating trust region for offline reinforcement learning. Neural Information Processing Systems (NeurIPS), 2024c.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, -4:8780-8794, 2021.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via Q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945-53966 2024a.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion- based reinforcement learning via Q-weighted variational policy optimization. Neural Information Processing Systems (NeurIPS), 2024b.

---

Published as a conference paper at ICLR 2026
Zihan Ding and Chi Jin. Consistency models rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984, 2023.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. In International Conference Learning Representations (ICLR), 2024.
Carles Somming--Enrich Jiequn Han, Brandon Amos, Joan Bruna, and Ricky T. Q. Chen. Stochastic optimal control matching. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Pro- cessing Systems, volume 37, pp. 112459-112504. Curran Associates, Inc., 2024. doi: 10.52202/009011-33373. URL https://proceedingss neurips cc/paper_fille ffiees/paper/////////e/p/ wawp:eeeecsaaoaapaa----e-e--- Pawererpoeeeeeeeeeeeeeee pdf.
Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky T. Q. Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview httpe//orrummddxd(Btaaa
Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. EXPO: Stable reinforcement learning with expressive policies. arXiv preprint arXiv:2507.07900 2025.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Composi- tional generation with energy-based diffusion models and meme. In International conference on machine learning, pp. 8489 8510. PMLR, 2023.
Nicolas Espinosa-Dice Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kiante Brantley, and Wen Sun. Scaling offline RL via efficient and expressive shortcut models. arXiv preprint arXiv:2505. 22866, 2025.
Linjiajie Fang. Ruoxue Liu, Jing Zhang. Wenjia Wang, and Bingyi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. The Thirleenth International Conference on Learning Representations, 2025a. URL htt////oummmdddddeaeee
https::/ /ttp://ppp/e/ieeeeee/emee
Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bingyi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. In The Thirleenth International Conference Learning Representations, 2025b. URL
net/ fodd
Kevin Frans, Scohong Park, Picter Abbeel, and Sergey Levine. Diffusion guidance is controllable policy improvement operator. arXiv preprint arXiv:2505.234555 2025.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:18267-18281, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018.
Philippe Hansen-Esttn Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-lcarning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Aaron Havens, Benjamin Kurt Miller, Bing Yan, Carles Domingo-Eninn Enrich, Anuroop Sriram, Brandon Wood, Daniel Levine, Bin Hu, Brandon Amos, Brian Karrer, et al. Adjoint sampling: Highly scalable diffusion samplers via adjoint matching. arXiv preprint arXiv:2504.1173: 2025.
12

---

Published as a conference paper at ICLR 2026
Longxiang He, Li Shen, Linrui Zhang. Junbo Tan, and Xueqian Wang. DiffCPS: Diffusion model based constrained policy search for offline reinforcement learning. ArXiv, abs/2310.05333, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint TrXXiv2207.12599 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840- 6651,, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.
Video diffusion models. Advances neural information processing systems, 35:8633-8646 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
Prajwal Koirala and Cody Fleming. Flow-based single-step completion for efficient and expressive policy learning. arXiv preprint urXiv:2506.21427, 2025.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
learning. In International Conference on Learning Representations, 2022. URL https:// openreview nwt/neeruurt iammmmid=-mmmmm 1d=68n2s92JWFF.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024a.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. In The Tiith Annual Conference on Neural Information Processing Systems, 2025. URL https: bttaps://opeee //op//ppeeeveeeeeeeeeeeeeeeeeeee wwt.eeeettoooee formmmid----o XussiiixxxKKKK6.....
Steven Li, Rickmer Krohn, Tao Chen, Anurag Ajay. Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neural Information Processing Systems, 37:38456-38479 2024b.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben- Hen-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.06264, 2024.
Xingehao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023. URL httph::////// /ttp::///peeeeeeeeeeeee
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference Machine Learning, 2024. URL htttp:://ppereet http//pppeeeeee.eeoomeeeee
hwtp:/iwwwweeeeeeeeeee.e.eeeeeee www.feeefffee.o oowmmrddcCCiiiioo CRIVIP.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pp. 28825-22855. PMLR, 2023a.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (ICML), 2023b.
Haitong Ma, Tianyi Chen, Kai Wang. Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. arXiv preprint arXiv:2502.003661 2025.
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy-agnostic RL: Offline RL and online RL fine-tuning of any class and backbone. In 7th Robot Learning Workshop: Towards Robots with Human-Leve Abilities, 2025. URL https://
13

---

Published as a conference paper at ICLR 2026
David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients. arXiv preprint arXiv:2507.21053, 2025.
Vivek Myers, Bill Zheng. Benjamin Eysenbach, and Sergey Levine. Offline goal-conditioned reinforcement learning with quasimetric representations. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL P23UMiw71J.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0993 09359, 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL:: Calibrated offline RL pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36, 2024.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline RL? Advances Neural Information Processing Systems, 37:79029-79056, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking
offline goal-conditioned RL. In The Thirteenth International Conference on Learning Representa- tions, 2025a. URL https://opeerevieee net/ fort///rrmmm wtwpmmmiidii-iie M992mjgKZI.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes RL scalable. In The Thitty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL https: https:///penrreieee net/ htt///ormmme id= hguaupzLCU.
Scohong Park, Qiyang Li, and Sergey Levine. Flow Q-leamingg In Forty- International Confer- ence on Machine Learning, 2025c. URL https::/ www.ee//oorunidd-V/SSTT-S
Ahmad Parsian and SNUA Kirmani. Estimation under linex loss function. In Handbook of applied econometrics and statistical inference, pp. 75-98. CRC Press, 2002.
Angus Phillips, Hai-Dang Dau, Michael John Hutchinson, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Particle denoising diffusion sampler. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 40688-40724 PMLR, 21-27 Jul 2024. URL https:// proceedings mlr, .tr.sspss////... ppress/235/pps /235/phillips222 hatm.
Lev Semenovich Pontryagin, V G Boltyanskii, V Gamkrelidze, and F Mishchenko. The Mathematical Theory of Optimal Processes. Wiley, 1962.
Michael Psenka, Alejandro Escontrela, Picter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via Q-score matching. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 4186 PMLR, 21-27 Jul 2024a. URL httph: https://prroceedings mlr. ptress/v235/peetee
html.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via Q-score matching. In International Conference Machine Learning (ICML), 2024b.
Allen Z. Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. In The Thirteenth International Conference on Learning Representations, 2025.
URL https: htRps://///
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
14

---

Published as a conference paper at ICLR 2026
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In Internationall Conference on Learning Representations, 2021. URL MtLLtttt///ppeeeeeeees www..eeoooteooooooodooouuuu//a/ W......
Raghav Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. In Forty-second International Conference on Machine Learning, 2025. URL https: //openreview wwt.//et/oounnee hitp:mmmddddddpppppppppp
Marta Skreta, Tara Akhound-Sadeghe Viktor Ohanesian, Roberto Bondesan, Alán Aspuru-Guzli Arnaud Doucet, Rob Brekelmans, Alexander Tong, and Kirill Neklyudov. Feynman-kac correctors in diffusion: Annealing, guidance, and product of experts. arXiv preprint urXiv:2503.02819 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv
preprint arXiv:2010.02502. 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. Advances in neural information processing systems, 33:1 1244--12444 2020.
Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. In The Eleventh International Conference on Learning Representations, 2023. URL yyBis80iUnU.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the mini- malist approach offline reinforcement learning. In Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023. URL https: / /ttp:////eeeeeeee////mmome het.ee/eeee/ooorrr////evtttoa
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
James Thortoo,, Louis Béthune, Ruixiang ZHANG, Arwen Bradley, Preetum Nakkiran, and Shuangfei Zhai Controlled generation with distilled diffusion energy models and sequential monte carlo. In The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL https: / nnt./ faet/foouuee www.midd-6yyX0Yyaa..oo
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. Conference on Robot Learning. 2025.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.
Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine. Leveraging skills from unlabeled prior data for efficient online exploration. In International Conference Machine Learning (ICML), 2025.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information pro- cessing systems, 34:27395-27401 2021.
Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, and Hao Su. Policy decora- tor: Model-agnostic online refinement for large policy model. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttp:// http::///ppereeeeee http///ppp/eeeeeee/eeeeeeeeeeeet net met/forum?id e5jCTEiJMT.
Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforce- ment learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviex: www.ee...eeeeeeeeeeeeeeeeeeeee -334444566...
15

---

Published as a conference paper at ICLR 2026
Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas B Schön, and Per Mattsson. Ennrooy-regullate diffusion policy with Q-ensembles for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2024.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL httph::////t http:://pppeeeeieeeeeeee evneeiiew.... feet/forrmiidd-HHoollo
Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, and Jing Jiang. Adaptive policy learning for offline-to-online reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 11372-11380 13300 2023.
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforce- ment learning fine-tuning need not retain offline data. In The Thirteenth International Conference on Learning Representations, 2025, URL
16

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RESULTS
How robust is our method to data corruption? We study how robust QAM-EDIT is to data corruptions. particular, we take the noisy (for manipulation tasks) and stitch (for navigation tasks) datasets from OGBench. The noisy-style datasets were collected by expert policies with larger, uncorrelated Gaussian noise, resulting in broader state coverage but larger learning challenge. The stitch-style datasets were collected by the same expert policies the original navigati on-style datasets (what we use in all our other experiments), but the trajectories are much shorter segments (traveling up to 4 cells in the maze). These variants stress test the algorithm's ability to learn from noisy trajectories and 'stitch' short trajectory segments. We evaluate our method and compare against representative set of best-performing baselines. We use the same hyperparameters from our main experiments for each domain and report the results in Figure 4. For locomotion tasks, we observe almost no performance difference when switching from navigate-style style dataset. For manipulation tasks, our method exhibits strong robustness against action noises. In particular, while all our baselines (except BAM) fail completely on the sube-triple-nee environments, our method only suffers minor performance drop.
stitch-
um-naviggae
al-navigate
scene-play
p33-play
C2-play
C3-play
100
75
Aggregated Score
hm-stitch
slitch
scene noley
p33 naisy
c2: c2-naisy
c3 c3-ncisy
1000
75
50
30
Environment Steps (x10*)
Figure 4: Data quality analysis (12 seeds). Top: performance on the original (navigate/playy> style datasets we use in our main experiments; Bottom: performance on the (stitch/noisy)-stt datasets. Each subplot reports the aggregated score over 5 tasks in each domain. For results on individual tasks and full training curves offline, see Figure 7 below.
B FULL RESULTS
We provide full training curves and result table for all methods and tasks individually. Figure shows both the offline phase (until 1M steps) and the online phase (after 1M steps). Table reports the performance at the end of the IM-step offline training. The hyperparameters can be found in Section F. Figure shows the full training curves for our sensitivity analysis.
C ADDITIONAL DISCUSSIONS FOR RELATED WORK
Diffusion and flow-matching with guidance. Diffusion and flow-matching models have been used for generating data with different modalities ranging from images (Rombach et al., 2022), videos (Ho et al., 2022), and text (Lou et al., 2024). In most applications, the generative models trained on large- scale unlabeled data do not provide high-quality samples when conditioned on some context (c.g., language description), and common practice is to augment the sampling process with classifier guidance/classiffe-- free guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), with the goal of aligning the sampling distribution better with the posterior distribution conditioned on the context. However, most of the guidance methods suffer from a bias problem that is tricky to tackle, stemming sampling processes
from the fact that simply adding or interpolating two
17

---

Published as a conference paper at ICLR 2026
does not lead to the correct composite distribution in general (Du et al., 2023; Bradley & Nakkiran, 2024). One solution is to use Langevin dynamics sampling approaches (Song & Ermon, 2019) where only the score function for the noise-free distribution is required, but they have been known to under- perform diffusion/flow models due to the challenge of accurately estimating the score functions in low-density regions (Song & Ermon, 2020). Since then, line of work has proposed solutions to generate the correct composite distribution. Du et al. (2023), Phillips et al. (2024), Thornton al. (2025), Singhal et al. (2025) and Skreta al. (2025) propose to use Sequential Monte Carlo (SMC) that uses resampling procedures to leverage additional test-time compute to correct such bias. Rather than correcting the distribution at test-time, Domingo-Ennn Enrich et al. (2025) and Havens et al.
(2025) take different perspective by formulating it as a stochastic optimal control (SOC) objective that can be efficiently optimized as fine-tuning process while providing guarantee that the model converges to the correct distribution at the optimum. The flow/diffusion policy optimization problem guidance
in actor-critic RL methods shares a similarity to the aforementioned classifier/Classiferr- problem in generative modeling literature where the critic function serves the guidance to the generative policy model. This allows our method builds directly on top of the algorithm developed by Eanrih al. (2025) while enjoying the guarantee that the policy converges an optimal prior-regularized solution (i.e., π X T exp(p) (Q a))).
CEP (Lu et al., 2023a) and CFGRL (Frans et al., 2025). Both of them build off from the idea of chassifirr/classiffr-eees free guidance, which combines the denoising step of a base diffusion/flow policy to the denoising step of for tilt distribution.
CEP (diffusion): log az, + a loloooosss (1 a)Q(s, ar,t)
(32)
CFGRL (flow): f(s. + afs(s, an + (1 - (s)
(33)
where Q(s, tt.t) t) <- log Ea, [cQ(x.a)] is the score of the Boltzmann distribution (i.c., x ((sa)) at denoising time t and fo is the velocity field of the policy that is conditioned on optimality variable, a binary indicator of whether the policy is 'optimal' (o = 1 means it is). CEP aims at approximating t x (TT) (eQ().)))11a)) whereas CFGRL aims at approximating т x (TT))()T=))) (1-aa)
However, as discussed in many prior work (Du et al., 2023; Bradley & Nakkiran, 2024), even when both the denoising steps are exact (log π (s, de, t) for diffusion and f(..,,) t) for flow), the denoising process that uses summation of them do not lead to the correct distribution:
V as log (aa 8) * Va, log ( | 8) + T V ((.,a)))
(34)
DAC (Fang et al., 2025b). Diffusion actor critic uses the diffusion formulation where the goal is to find policy that satisfies (- s) x Te(- (())), However, their training objective is derived based on the assumption that
V at log s) Va, logps(a +V,
(35)
and additionally
V (t) V., a(s,
(36)
While these assumptions provide a convenient approximation of the objective function, it does not provide guarantees on where policy converges to at the optimum.
D DOMAIN AND EXPERIMENT DETAILS
We consider 10 domains in our experiments. The dataset size, episode length, and the action dimension for each domain is available in Table 3. For each method and each task, run 12 seeds. All plots and tables report the means with 95% confidence intervals computed via bootstrapping using 5000 samples. All our experiments are run on NVIDIA RTX-A5000 GPU and our code is written in JAX (Bradbury et al., 2018). We use NVIDIA A5000 GPU to run all our experiments. Each complete offline-to cifine-to-ooline experiment run takes around hours. To reproduce all our results in Table and Figure we estimate that it would take around hours per single run
x
17
X
50
X
12 = 51 000 GPU hours.
methods
# of tasks
of soods
18

---

Published as a conference paper at ICLR 2026
E BASELINES
In this section, we describe in detail how each of our baselines are implemented. all the loss functions below, unless specified otherwise, 8, a, s' are assumed be sampled from D and as part of the expectation even when it is not explicitly written under the expectation (i.e., EE>wV[] := B(s.orraf)=D,s=-iill)
1. Backprop-based.
FBRAC baseline considered in FQL (Park et al., 2025c) as flow counterpart of diffusion Q- learning (DQL) (Wang et al., 2023), where the multi-step flow policy is directly optimized against the Q-function with backpropagation through time (BPTT). addition maximizing the Q-value, FBRAC also has a behavior cloning term where the flow policy is trained with the standard flow- matching objective on the dataset action with a BC coefficient a.
We implement this baseline by training flow-matching policy with the following loss:
LyBRaC = aLFM(() + LBrTr((),
(37)
where LFM is the standard flow-matching loss that clones the data behavior (i.e., Equation (17)) and
T-1 + h Σ sols,a k-1 kh))
(0) -Eoo=s- Qo 8, Clip
(38)
where Clipp-Jè is an clement-wise clipping function that makes sure the actions generated from the flow model fe within the valid range 1] and {a is the discrete approximation of the ODE trajectory using Fuler's method with a step size of h = 1/T:
a i+1 = a' + -hfols, iih), ViE f0,1, .. ,T..
(39)
We use
ODE(Jo(s, Clip = >
aa(s,a'11 ,ih)
(40)
1=0
as the abbreviation for the rest of this section, and additionally use To to denote the distribution of actions generated by fe.
The critic loss is the standard TD backup:
L E.-N [(Qg(s,a) - Qa(s', (DE(fe(s',))).)))
(41)
In practice, we also use K 10 critic functions and pessimistic target value backup as described in Equation (26) and the policy we use to interact with the environment is f..
FQL (Park et al., 2025c) distill a multi-step flow policy into a one-step distillation to avoid BPTT.
This baseline is implemented by training a behavior cloning flow-matching policy (i.e., fo S x Rd x 0, 1] + RRA), and 1-step distilled noise-conditioned policy (i.e., P2w S x R + IR:);
Lequ w) LFM(0) (0) + Louestep (w)
(42)
where (f) is the standard flow-matching loss (i.c., Equation (17)) and
Tonestop (w) Erzz [ol|[R
ODE(So(s' ,, ), =))11 Q(s, (s,,
(43)
where a is the BC coefficient that controls how close the 1-step distilled policy should be relative to the BC policy fo. Finally, the critic loss is the standard TD backup:
L(Φ) = EE- [[Qa(s, a) - - Qs(s', (s' 2
(44)
In practice, we also use K 10 critic functions and pessimistic target value backup as described in Equation (26) and the policy sampled by first drawing a Gaussian noise 2 ~ N and then obtain the action by running through the one-step distilled model: 2),
to interact with the environment is πQ. where the action is
19

---

Published as a conference paper at ICLR 2026
2. Directly using the action gradient of the critic (i.e., (o)(s,a)) with approximations.
QSM (Psenka et al., 2024b) uses the action gradient the critic to train a diffusion model such that it can reconstruct the policy that follows the Boltzmann distribution of the Q-function:
(|- s) & [Q((s.)).
(45)
To approximate the score the intermediate actions, Vas log (a 8), QSM directly uses the critic evaluated at a²:
V D (a' ()) R T TV VV
(46)
where a' Va'a+ VI- with ~ N and {a", ...,...--- being any diffusion schedule with the number of diffusion steps of T' For all diffusion methods in this paper, we follow Mansen-Estruche et al. (2023) to use the variance preserving diffusion schedule:
Umin
(bmax ) +
8 exp
(47)
272
a²
Br).
(48)
j=0
for all E {0,1, T 1} with bmin 0.1, bmax 10 and T being the number of diffusion steps.
To train the diffusion model, QSM uses the following loss function:
(o) = BE-,, T- [I - - a') fo 1)
(49)
where again a' Va'a v1 - a. In practice, find that using V. (2 at) (the action gradient of the critic network) helps learning stability, so we use that instead.
To adopt QSM into the offline-to-oning RL setting, we additionally augment the loss function with the standard diffusion loss (Ho et al., 2020) as the behavior regularization:
LDDPM (0) EE so T-11 [lle So(s, a², 11
(50)
The overall actor loss is thus
L( = L0)) + (LDDra(
(51)
QSM then uses the standard diffusion denoising procedure (with the clipping to make sure generated actions are within [-1,1] 1))):
1-1 + Clip
1
(
(8'
a
V1 fols,d,)) + ei
gi
N'~
(52)
v1 - 3i
where aT N. We denote Diff(fo(s,..) as the resulting a after going through the diffusion procedure above and J. , (fo((s((ss')) as the policy that generates actions using fo- The critic loss
be defined follows:
l(Ø) = E.>N ((Q(ss,) -
Q2(s', - Tt (s)))
(53)
In practice, we also use K = 10 critic functions and pessimistic target value backup described in Equation (26) and the policy we use to interact with the environment is Ty..
DAC (Fang al., 2025b) is another method that uses similar approximation with behavior prior (i.e., var log n' (a' s) s Va log (a" | + -V Qo,, a))). To train the diffusion model, DAC matches fo to linear combination of (behavior cloning) and V. ((. (Q-maxinization)
Loxc(0) = Se TT-1 [|| TV Vo. fels,a', 11)
Bew... ||| fo(s, a², a)ll (fols, Voloss) +C, with 11 2/T. In practice, it is implemented combination of (()) EE (,0,,,, [1 [ll(fe(s,a',)) Vo. (LLDDPM(())
(54)
(55)
20

---

Published as a conference paper at ICLR 2026
DAC uses slightly different way to clip the actions:
a g
v1 a folss.d:1)) ))
à'- a'-1+ t Clip
(56)
vI - 3r
a 1- 8' 3'Va'-1
l-aa a"
+ v1 3i(1 - al---a=+aa" + VBee, ~ a'-1
(57)
where a² ~ ~ and a can be interpreted as an approximation of the denoised action (e.g., in the DDIM sampler (Song et al., 2020)). This approximated denoised action is first clipped to be within [-1, and then used reconstruct a-- using the closed-fome Gaussian conditional probability
distribution of p(a'-1 | a" = â a = a) (e.g., Eq. (7) in Ho et al. (2020)), In practice, DAC
also uses /B B as standard deviation for this conditional distribution instead of the correct one 9
vI-a--1 3* as an approximation. This is why in the expression above the multiplier before e¹ is VB". Vi-a*
Similar to QSM, the critic loss can now be defined as follows:
L( = EE-N [[Qa(s, r- Qz(ss' d' ~''- ~ (- (-
(58)
where ff is the policy that generates actions using the procedure above in Equation (56) and Equation (57). In practice, we also use K 10 critic functions and pessimistic target value backup as described in Equation (26) and the policy we use to interact with the environment is f.
CGQL is novel baseline built on top of the idea of classifier guidance (Dhariwal & Nichol, 2021). In particular, we combine the velocity field of a behavior cloning flow policy and the gradient field of the Q-function to form a new velocity field that approximates the velocity field that generates the optimal behavior-constrained action distribution.
More specifically, we implement this baseline by interpreting (s, as the score of the optimal entropy-regularized distribution log (- s) (where x*(- s) x (o(o)))) The corresponding velocity field that generates this distribution of actions can be obtained through simple conversion
(e.g., following Equation 4.79 from Lipman et al. (2024)):
cs(,,,,t) = (1 11-th -)t ((
(59)
where
Q(s.a,t) == log Be-W((..1a) [
1
(60)
is the score of the distribution over the noisy intermediate actions. In the classifier guidance literature, the score of the noisy examples is approximated by the score of the noise-free examples at the noisy examples (Dhariwal & Nichol, 2021). In our setting this translates to
fo(s, ar, (1 ()T Va a.) + ae
(61)
Empirically, both versions (fo and fa) perform similarly and we opt for a simpler design f as it does not require learning approximating Q(s, a, t) for all E [o, 1]. Finally, we add the velocity field defined by Qo directly to the behavior cloning velocity field to form our policy:
f f= f3 Ufe
(62)
where fss is trained with the standard now-matching loss (i.e., L(3) = (f)) and is coefficient that modulates influence of the guidance that we find to be helpful (e.g., 2 < 1 often works better than = 1). The critic loss uses n. to backup the target Q-value:
L(Φ) = Er- [(Qs(s, - Qs ODE(S ,)))))))))))))
(63)
In practice, we also use K critic functions and pessimistic target value backup as described in Equation (26) and the policy we use interact with the environment is n. (generated from the summation of and fo-
21

---

3. Directly using the critic value (i.e., a)). Published as a conference paper at ICLR 2026
CGOL-MSE/Linex. Alternatiely,, we can approximate Qi in Equation (60) more closely with training objective. In particular, we explore the following two regression objectives:
())(sa)) 2
MSE:
(MSE(() = [Bx.0[001).s0m
(64)
Linex: Linee (() 1B0,4[0011).00 (e a,, +
(65)
where o :- (1 - f)z + ta. The optimal solution of Q₂ in Equation (64) is not exactly the same as the desired Qo in Equation (60) but constitutes lower-bound due to Jensen's inequality:
(s on, () = Ee (Qo (s, ar) S log (eeaa((..aa)
(
(66)
The second objective resembles the classic Linex objective (Parsian & Kirmani, 2002) where the optimal solution of Q₂ in Equation (65) is the same the desired Q in Equation (60):
Qineex (s, 0, t) log E. ( (o
1
(67)
A discussion on this can be found in Myers al. (2025). For completeness, also show this below. Without loss generality, we just need to show that exp(y) = [[exp(x)]] = p(x) exp(x)dr the unique minimum for the following loss function:
/ p(x) [exp(x y) + dr
L
(68)
Taking the first and second derivatives with respect to y gives
d/ = exp(-9) +
(69)
dy
d L dy² exp(-y)
(70)
Setting Equation (69) gives y log E[exp(x:) which the second derivative is 1 (i.e., d'
1).
dy²
Furthermore, prevent the exponential blow up of exp(Qs a) Qg(s, ag, f)), we follow Myers et al. (2025) to use a Huber-style loss that locally behaves like a Linex loss but with a linear penalty when the exponential term is too large. In particular,
the following loss:
exp((A) + (8,
A
Linext (5) =
(71)
[
A
< 5
where A ((Qo(s, Qg(s,ar,t)) t)).
In practice, however, both the MSE and Linex objectives (LLinext MSS++ can still be unstable and exhibit high variances. Instead of learning Q₂ and Qa separately, we find it is often better to directly learn both of them in a single network Qs S J0. 1] IR follows:
( =E [(Qg(s, a,
2z(s' ODE 2))
(72)
(E<Eu(0.)),~ [1Qa(s t) ta, t) Q2(8, a, 1))²}
where @ is the coefficient that balances the TD backup for Qo(s, a, 1) and the noisy target regression
for Qo(s,,, a, < 1). With (s, we can define our velocity field (which is also used as the TD backup above) as
f f= where fo(s, = TV VV +
(73)
b
and i again modulates the guidance strength.
FAWAC a baseline considered in FQL (Park et al., 2025c) where it uses AWR to train the flow policy similar to QIPO (Zhang et al., 2025).
22

---

Published as a conference paper at ICLR 2026
We implement it by training a flow-matching policy with the weighted flow-matching loss:
Lemac () (s,, s, a) LFM
(74)
(8 (8, Bx-10,,1)) [Il5o (s, (1 f)= + la, - + a
(75)
where vi(s.) a) = min (v.)) 100.0). The inverse temperature parameter 7 controls how sharp the prior regularized optimal policy distribution is.
The critic function Qo(s, a) is trained with the standard TD backup and the value function Vs(() regresses to the same target:
L(Ø) = EE-- [(Qa(s, a) - Qa(s', (DE(f(d'...))))))
(76)
L(E) = Ez-- (Ve(s) - - Qz(ss,
(77)
The second line can also be alternatively implemented by regressing to the critic function (ssaa) directly. We implement in this particular way because can re-use the Q-target computed.
In practice, we also use K 10 critic functions and pessimistic target value backup as described in Equation (26) and the policy we use to interact with the environment is fo
Post-processing-bassee
FEdit is a baseline that uses the policy edit from recent offline-to-online RL method conceptually similar to EXPO (Dong et al., 2025). We implement& a Gaussian edit policy on top of a BC flow policy rather than a diffusion policy used in EXPO. EXPO also uses the standard sample-and-rank trick where it samples multiple actions and rank them based on the value. To keep computational cost down and comparisons fair other methods, only use single edited action for both value backup and evaluation.
We implement this baseline by training flow-matching policy (i.e., So : S IR x [0, RA), and a 1-ttep Gaussian edit policy (i.e., Tu S x A + A) implemented with an entropy regularized SAC policy (Haarnoja et al., 2018). The loss function can be described as follows:
LFEa1t (0, w) = (0) + LGaussian (w), s.t. EenD [ (: {)) 2 Hearget
(78)
where LFM is the standard fow-matching loss that clones the data behavior (i.e., Equation (17)). H Harrpot is the target entropy that the Gaussian policy is constrained to be above of, and
LGaussian (w) (
Qa(s, Clipla. Aa + a|L,)]
(79)
where à (s, and Clip[-là ( is an element-wiss clipping function that makes sure the actions are within the valid range [].
Intuitively, the Gaussian SAC policy edits the behavior flow policy by modifying its output action where aa is the hyperparameter that controls how much the origianl behavior actions can be edited.
The critic loss is the standard TD backup:
[ = Wa (s) - r Q2(ss, Clip lá' + to. Aa'
(80)
where again a' ODE(fa(s', 2). In practice, also use K 10 critic functions and pessimistic target value backup as described in Equation (26) and the policy we use to interact with the environment is combination of the BC policy Ty, and the Gaussian edit policy. We first sample 2 ~ A and then run it through the BC flow policy to obtain an initial action + ODE(ff(ss.:)) 2) and then both the initial action and the state is fed into the edit policy generate the final action à + Aa where Aa ~ (: a).
DSRL (Wagenmaker et al., 2025) is a recently proposed method that performs RL. directly in the noise- space of pre-trained expressive BC policy (flow or diffusion). We use the fow-matching version of DSRL as our method is also based on flow-matching policies. The original DSRL implementation does not fine-tune the BC policy during online learning while all our baselines do fine-tune the BC policy online. To make the comparison fair, we modify the DSRL implementation such that it also fine- tunes the BC policy. One additional implementation trick that allows this modification to work well is the use of target policy network for the noise-space policy. In general, we find that fine-tuning the BC policy yields better online performance, so adopt this new design of DSRL in experiments.
23

---

Published as a conference paper at ICLR 2026
More specifically, train a flow-matching policy (i.e., fo : S x RA x [0.1] RA), and 1- step Gaussian edit policy (i.., Tu S x A + A implemented with an entropy regularized SAC policy (Haarnoja et al., 2018). The loss function can be described as follows:
LosaL (0, w) = (0) + L [LatentGruusiaau(/ (w); s.t.
E [H( 8))] 2 H Htarget
(81)
where LFM is the standard flow- wow-matching loss that clones the data behavior (i.e. Equation (17)), H Haarge is the target entropy that the Gaussian policy is constrained to be above of, and
Lastentaasseian (w) = Eg- (-1s) - (s,
(82)
where 0 (8, 2) a distilled critic function in the noise space that is regressed to the original critic function, (s,
((V) E.-N ({Q](,,=) Qo(s, (D :.))()(
(83)
Intuitively, DSRL directly learns policy in the noise space by hill-climbing distilled critic that also operates in the noise space. Finally, the critic loss for the original critic function in the action space is
((d) = (g [(Qo(s,)) - Qz(s', ODE(Se(s (.) +)))
(84)
We use K = 10 critic functions and pessimistic target value backup as described in Equation (26). The policy we use to interact with the environment is a combination of the BC policy TT and the Gaussian noise-space policy. We first sample ~ (- s) and then run it through the BC flow policy to obtain the final action ODE(fo 2). One important implementation detail for stability in the offline-to-online setting is to use the target network for the BC flow policy So (instead of So- Without DSRL can become unstable sometimes when the BC flow policy changes too fast online.
IFQL is a baseline considered in FQL. (Park et al., 2025c) as a flow counterpart of implicit diffusion Q-learning (IDQL) Clansen-Estruch et al., 2023), where IQL (Kostrikov et al., 2022) is used for value learning and the policy extraction is done by sampling multiple actions from a behavior cloning diffusion policy and select the one that maximizes the Q-function value.
More specifically, we train a critic function Qs a) and a value function Vg(s) with implicit value backup:
L()) = (Q 0) - -- - Vg(s')² L(E) Pexp (Q,(s, Vs(s))
(85) (86)
where F Fu)p(u) Lucolu is the expectile regression loss function.
On top of that, we also use K = 10 critic functions and pessimistic target value backup as described in Equation (26) for training the value function (s). To extract a policy from Qo(s, a). IFQL uses rejection sampling with a base behavior cloning flow policy that is trained with the standard flow-
matching objective. In particular, the output action a" for s is selected as the following:
a" f- arg max aN (1, a1.-. ON T5o 1s).
(87)
5. Gaussian.
RLPD (Ball et al., 2023) is a strong offline-to-online RL method that trains a SAC agent from scratch online with a 50/50 sampling scheme (i.e., 50% of training examples in a batch comes from the offline dataset whereas the other 50% of training examples comes from the online replay buffer). In addition to the standard RLPD update, we also add a behavior cloning (BC) loss which we find to improve the performance further. We tune over the BC coefficient (a) and report the value we use for each domain in Table 5.
ReBRAC (Tarasov et al., 2024) is strong offline RL method that trains TD3 (Fujimoto et al., 2018) agent with behavior cloning loss. In practice, we find two hyperparameters to impact performance the most. The first hyperparameter is which controls the strength of the behavior cloning loss. The second hyperparameter is o which controls the magnitude of the Gaussian noise added in the TD3 policy. We keep the action noise clip be 0.5 and an actor delay of 2, following the original paper.
24

---

Published as a conference paper at ICLR 2026
Algorithm 1 Learning procedure in QAM.
Input: (s,a,
a, a', s)):: off-policy transition tuple, fe: behavior velocity field, fo fine-tuned velocity field, Qa: critic function.
Optimize w.r.t ((Ø) (Q a) - r-sQ0sss d' mo(- s'))]²
TD backup
a (0. ah..... a1} SDE(ff(8.;,.)
Memory-less SDE; Equation (24)
91 - - V., d1 (s(s,
> Compute the critic's 's action gradient
9h. - t LeanAdi(fsss .))91.))
Lean adjoint states; Equation (25)
Optimize 0 w.r.t L(0) - >,
2(fo(s.a) (fa.t))))Ja(allll fa() ozgt
Adjoint matching; Equation (21)
Output: fe, Q
F PSEUDOCODE AND HYPERPARAMETERS
While most methods share a common set hyperparameters (Table 4 for a fair comparison, most methods need to be tuned for each domain. We include the domain-specifee in Table 5 and the tuning range of them in Table 6. To pick the hyperparmacter for each domain for each method, we first run a sweep over all hyperparmeters in the range (specified by Table 6), or all combinations of them if there are multiple hyperparameters involved. The hyperparameter tuning runs use seeds for each method for each hyperparameter configuration for each of the two tasks per domain. For locomotion domains, we use task (the default task) and task 4. For manipulation domains, we use task 2 (the default task) and task 4. We use task in addition the default task recommended by OGBench because often find the combination of these two cover the characteristics of each domain better. We then use the combined performance of the two tuning tasks per domain per method pick the hyperparameter configuration. We include them in Table 5. Finally, for all our main results, we run all methods on all five tasks for each domain on 12 new seeds (different from the tuning seeds). We pick the hyperparmeter range such that the total number of tuning runs are similar across methods. To achieve this, we use the following strategy: For methods where more than one hyperparameter needs to be tuned, we use coarser hyperparmeter range. For methods where there is only one hyperparmeter, we use a more fine-grained sweep.
G THEORETICAL GUARANTEES
Proposition 1 (Extension of Proposition 7 in Domingo-Eniin Enrich al. (2025) to Policy Optimiza- tion.) Take (f) in Equation (14), there a unique fo such that
0 afo
LAM = 0,
(88)
and for all supp(D).
ma( ()) x TT(- Qa(caa)
(89)
Proof. Our proof mainly rewrites the assumptions and the statement of Proposition 7 of Domingo- Enrich et al. (2025) in our notations with a simple extra step that extends it to the state-conditioned version (e.g., for policy optimization).
We first define (from Equation 27 in Domingo-Enricn et al. (2025)) the residual velocity field as
frea az, () = V Be(Bárr/a - Be) (Jo(s, ar 1) - fa( az, I))-
(90)
While the original result showed for a more general case with family of ax, 3 (and later on a)) in our work we assume
a(t) 8(t) = -t,
(91)
(92)
(tf) 22 t)/t.
(93)
25

---

Published as a conference paper at ICLR 2026
This allows to simplify the expression for fres
[rea(s, dr. v T (Jo(s, (s,,
2t
(94)
Then, we restate the definition of (from Equation 27 in Comingo-Enniche et al. (2025)):
de a. ote)s ac,t)
b(s, (r.tt)) t) 2(fa(s, az, t) fo(s, ae, -2(Je(8, 1) [a(s, ae, ()) ux/l 25 ar 2fs(s, az, t) az/
(95)
(96)
(97)
We rewrite our 'lean' adjoint state definition
dg. 9, V [b(s, t)), Φ1 -=-7Vol((s.aa)
(98)
which coincides with the definition in Equation 38 in Domingo-Enrich Entingo-EErichheo al. (2025), with f 0 and g(-) = a ()) Now, we can rewrite our adjoint matching objective as
(A = D.(ae [AAa(fo:(a))
(99)
where
LAM(s, So, {arte) = S(/ (1 + (1)
2 /. S0(8,
& ar (t))
(100)
1 f. 12folss t)/o(t) + (t) dt
2
Comparing (s, for {a+)e) to the definition in Equation 37 in Domingo-Enrich et al. (2025), they different by only factor of 2 conditioned
fixed Thus, their critical points the same.
By triggering Proposition 7 in Domingo-Entnn Enrichh et al. (2025), we can conclude that for a fixed s, the only critical point of the following loss function,
Eaac [LAM(s, (a))))
(101)
f" ar, t), the velocity field that generates the following distribution,
(- 5) x TT(: exp(TQQ((s,)).
(102)
Finally, since the (O) is linear combination of (Eaai [LAM(S; So, {a+]e) over different 8, the critic point of LAM(Jo) is simply the cartesian product of over the critic points for each supp(D). Since there is only critical point for each B(o.)= [LAM(s. fe, {ai):) LAM(ffo also has only one critical point and coincides with f' a,, t). This concludes that the only critical point of (AM(fo)
results in velocity fields that satisfy
*t 8) x Tt(. ( (x)(()(((())))) vs E supp(D).
(103)
26

---

Published as a conference paper at ICLR 2026
PURAC
BAM
PAVAC
DMC
25N
173
UAN
GAR-Y
QAN-E
taxk2
98
010
0
63
58
555
80
-41
88
took2
56
60
78
40
F1
84
14
151
untaaze Jarge
TH
tooks)
taski
35
11
10
10
53
Fa
70
10
11
93
11
11
11
10
13
0
18
10
00
11
teskS
13
1
1
0
10
0
0
10
0
10
0
antwaze-gt
tuak4
74
0
0
0
0
12
10
83
15
tasks
59
01
0
10
13
23
12
0
30
423
tarke)
57
01
01
3
16
15
3
12
task1
30
50
89
F8
88
33
78
55
66
98
68
FR
83
92
77
013
98
SP
15
03
7
3
47
10
15
11
56
30
98
99
tooks]
30
23
G0
33
63
32
86
68
39
taski
36
18
0
1
0
10
11
36
1
18
tax22
71
11
11
10
70
71
tooks
111
68
teaks
10
0
13
10
12
Tosks
11
10
16
10
20
10
taske)
24
95
08
100
012
96
100
100
100
100
55
03
100
100
100
53
80
190
F1
15
88
010
68
88
100
19
033
01
00
65
37
37
89
89
11
231
172
57
510
817
87
85
97
100
88
100
acese apazae
G0
09
50
71
10
12
110
100
33
516
100
80
100
27
17
00
53
27
10
80
53
50
100
34
95
87
88
55
16 tooks]
63
50
98
78
08
74
88
68
80
00
62
514
07
86
87
tosk1
90
100
155
110
100
1000
100
txxk2
59
85
100
28
100
pusste-ata--parcc oparrtn
100
100
100
tuab4
100
100
70
0
7
100
100
403-
75
58
53
taxk2
0
50
19
0
66
0
0
0
91
0
20
85
took2
0
13
7
01
13
1
0
1
15
0
6
7
puzale- 4.4- LOCW sparss
toaks
0
16
13
0
41
0
0
0
39
0
56
Taxke
0
12
11
17
11
01
01
14
56
taxks
0
13
10
10
32
10
12
19
tosks]
16
10
0
34
10
39
taski
84
51
10
86
60
36
67
36
83
69
70
78.
11
50
178
63
33.
83.
89
T1
99
1
53
11
501
151
57
10
33.
cibe-dochle
13
11.
13
51
22
22
4
50
0
13
tarits)
9
0
47
48
12
56
31
45
33
50
74
40
11
taxk]
24
0
50
40
31
16
11
16
took2
10
11
tooks
1
13
10
10
cube-triple
Taske
tasks
tooks)
taski
33
11
11
10
16
13
04
10
11
17
09
53
taxk2
88
11
11
11
11
1
18
1
1
3
11
01
15
11
71
1
1
cobe-quadruple--i
tuak4
0
10
10
0
10
10
01
task5
0
1
11
10
10
71
<5 tazks)
17
10
15
15
17
Agg-
(b0 taaiz)
56
33
Table 2: Full offline results at 1M training steps (12 seeds).
27

---

Published as a conference paper at ICLR 2026
QAH
OANFOLL
CUREETT
BAM
13RAC
FOL
COLL
COO-MSE
COCL-Unne
DIAL
FESR
DAC
1L7p
RAERAC
Tunarac peeraceaeee proserant
Score
15
Training Steps (x10*)
Figure 5: Full training curves for our main results (12 seeds).
28

---

Published as a conference paper at ICLR 2026
Te20
01
clging birs
177 uwr.ejaarurggo pro.aiaaae daroand Sipy
Seps
poo.ttiigggggg www.s/ppww.oopoooc.......o Gradwr Ciipngg
T-36
Bix
ciging
wit therr
fime 25gnt)
wwginssteet Criec Coerba 10
7-20
Score
Ugg
white
wrth feein Lvers
Coe.apeeeenneeeeeeeee... Fan 2art
sowessileeoiieeeeeeeeo cre Exsenia 204
coodoclee byd www.aaa wwr.aaaa----ooonne... Termontae
p3doo8o08d
330000000
Txl
CA
T-36
Bix
desin
Training (x10""
Figure 6: Full sensitivity results for QAM-E E (12 seeds).
29

---

Published as a conference paper at ICLR 2026
QIME
BNN
DEAL
Fldt
QPH
PADRAC
71-
20
Score
15
119
-1-
Steps (x11')
Figure 7: Full training curves for data quality analysis (12 seeds).
Tasks
Dataset Size
Episode Length
Action Dimension (A)
cube-double-
1M
500
cube-triple--
3M
1000
cube-quadruple-100M
100M
1000
antmaze-large-
1M
1000
8
antmaze-giant-*
1M
1000
8
humanooidmazz-mmddiu
2M
2000
21
numanoidmaze--argee
2000
21
scene-sparse-x
1M
750
5
1M
500
5
muzzlee-4xd1100--ppaeee
100M
500
5
Table 3: Domain metadata.
30

---

Published as a conference paper at ICLR 2026
Figure OGBench domains. In this paper, we primarily focus on evaluating our method on some of the hardest domains on OGBench (Park et al., 2025a). cube-(triple, quadruple) (left) requires a robot arm to manipulate up to 3/4 cubes from an initial arrangement to goal arrangement. antmaze-2 giant (right) requires ant robot agent to navigate from one location to another location. All these domains are long-hone hori-ho by design and are difficult to solve from offline data alone. The offline dataset in these domains contain diverse multi-moda. behaviors that can only be accurately captured by expressive generative models like flow/diffusion policies.
Parameter
Value
Batch size
256
0.99 for (ppezz/l//ccnm//cccb/aauutzzaaa
Discount factor (7)
0.995 for
Optimizer
Adam
Learning rate
10
Target network update rate (A)
10
Critic ensemble size (K)
10
0.5 for (puzzle/scente caaccene/cubeeeee aaabbe//ntmazee-
Critic target pessimistic coefficient (p)
0 for humanoi dianooddmaee+
UTD ratio
Number of flow steps (7)
10
Number of offline training steps
10,, except RLPD (0)
Number of online environment steps
0.5 106
Network width
512
Network depth
4 hidden layers
Optimizer gradient max norm clipping
1 for QSM, DAC, BAM and QAM*, No clipping for others.
Table 4: Common hyperparameters.
DasaRO
FeBlacc
FANAC
228
2EARD
C5BL
PSALS
CORL-N
CSOL-L
BAN
GAT
GRF-2
QMB-F
(1-41
[7.40
(4.1)
(4.0.1)
(9:6.1)
[r.Fs)
(C.w)
45
(LD0-1)
164
(3.38)
1:4
113
[38-0.11
[5.01.01)
(5,00
(2,300)
0L86
18.L.0
0.8
[10.0
(30)
0.2
8.00
(10.0.1)
(1300.000
10
(1.0A)
(8.m0)
cabe-drubLes
(0.6.0)
0.5
165
100)
0.2
0.5
(10. 118.088
(1) 0001 GO01
BABL 6011
$5.00
(1.
142:
[1L
1.4
11.
(36,9000.0.1
63.811
WEltas Targe-+
0100.01
6.4
[8L 981
033
81
D.8
113
11.
118-0.1
(13.03.01)
33
10
(1.0311
(3.200
entraae-ginn
138.0.3)
(5
Eaaaee cadira-
(000-0)
115
700 (1B5
(13.1100))
Aimaooi Jazge.*
0
(K0L.)))
065
110 901
6.5
30
0A
0.5
(6..08)
125.0.100.1
10
(3.011
063.90
Table 5: Domain-specifie hyperparameters. The best hyperparameter configuration obtained from
our tuning runs. We use the same hyperparameter configuration for all tasks in each domain.
31

---

Published as a conference paper at ICLR 2026
Method Hyperparameter(S)
Sweep Range
RLPD
{0. 0.00.003 0.01, 0.03, 0.1, 0.3, 1, 10, 30)
ReBRAC
(a, o)
00,0000000..000003,,,,,,,,0,,,,, [0003,0011003311,333::000,,,,000 100.0]
FBRAC
CGQL
(0,T)
(0000.00,,1),001111100))) (001,,00)),001111,0000000... 0.13)
OGQL-MMLLJ L)
(,,
FQL
a
1020,133,00033,,00,,,,,,,,,,,,,,
DSRL
0=
(01,002.04:00.0011112222.)
FEdit
Oa
1011002203,0444050000.00000003
FAWAC
001,022200440111132244441133333
IFQL
(0.5,006.007788,00,000000
DAC
"
10,
QSM
(T,
(1. (0.1,03,,113,113300
BAM
T
QAM
(0.1,033.133,10.330
QAM-E
(T.
0.9,) oo))
QAM-F
(T,
Table Domain-specific hyperparameter tuning range. For QAM-F, oo is equivalent to the original QAM. Similarly, for QAM-E, 0 is equivalent to the original QAM. For methods with more than one hyperparameter, we tune all possible combinations within the specified sweep ranges. For example, for CGQL, sweep over all 3 3 configurations with 3 possible values of i and possible values of т.
Method
Training Speed (milliseconds/step))
Parameter Count
ReBRAC
2.97
9.226 271
FBRAC
4.54
237 535
CGQL
9.62
242 655
CGQL-(M, L}
10.40
242 655
FQL
3.58
10 082 868
DSRL
5.63
18 474 580
FEdit
3.77
10 093 642
FAWAC
3.44
10 065 952
IFQL
2.66
10 065 952
DAC, QSM
4.44
10 320 447
BAM
5.52
10 083 380
QAM
5.83
083 380
QAM-E
6.61
10 939 487
QAM-F
6.36
928 713
Table 7: Training speed and parameter count for each method on cube-triple.
32


Reviews:
[{"review_id": "NfbxDS45v0", "reviewer": "Reviewer_8dDG", "summary": "The paper addresses the instability that arises when backpropagating policy gradients through action flows, which often leads to practical difficulties when optimizing a policy with a learned Q-network. To tackle this issue, the authors employ a recent technique called adjoint matching. Rather than training a single flow-matching policy solely on an offline dataset, they utilize the behavior policy to generate adjoint states that provide direct step-to-step supervision signals for the residual policy. This results in a stable adjoint-matching objective that enables local supervision over the residual flow while still recovering an optimal prior-regularized policy. The approach is compatible with settings where a prior policy is provided, and the authors demonstrate its effectiveness through experiments in both offline and offline-to-online scenarios.", "questions": "I am also interested in the computational cost of the method—approximately how many GPU hours are required to train a policy? Also, is there any observed numerical instabilities for high dimensionality?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses a critical challenge in flow-matching-based policy learning. While the current setting is limited to cases where a behavior policy is available, the authors introduce a well-grounded formulation that establishes a clear connection to adjoint-matching techniques in generative modeling. The resulting policy maintains strong expressivity, enabling more diverse action distributions to be represented. I believe this capability can offer benefits to reinforcement learning that extend beyond the specific experimental settings evaluated in the paper.", "weaknesses": "The method relies heavily on accurate action gradients, though this dependence is common across many RL algorithms. The authors address this challenge in practice, for example by training a critic ensemble for more reliable gradient estimates. That said, there may be further opportunities to explore more advanced techniques to improve gradient accuracy.\n\nThe evaluation is limited to OGbench domains. Including results on benchmarks with more diverse characteristics would further strengthen the paper’s contributions. For example, demonstrating performance across policies of varying quality could offer additional insights into the method’s robustness and general applicability, dense rewards, offline datasets of different qualities, etc. If an offline dataset is collected using a policy with low behavioral diversity, it is unclear how this would impact the effectiveness of the proposed approach. A discussion or empirical analysis on this aspect would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pt9lKaOLrb", "reviewer": "Reviewer_VcZG", "summary": "The paper introduces Q-Learning with Adjoint Matching (QAM), an actor-critic algorithm with flow-matching policies that uses adjoint matching (a recent technique from generative flow modeling). The paper tackles the problem of optimization instability in backpropagating the critic through multi-step denoising processes in flow or diffusion models. Previous methods either discarded the action gradient and solely relied on action values or only used one-step approximations. QAM uses adjoint matching (AM) to incorporate the critic’s action gradient in training a flow policy subject to a prior constraint. QAM assumes access to a fixed behavior (prior) flow policy and then learns a residual flow policy, which is optimized towards the AM objective. Leveraging the convergence guarantees of AM, the paper claims this converges to the optimal prior regularized policy. The experiments demonstrate QAM has lower hyperparameter sensitivity and better performance compared to many baselines in offline and offline-to-online settings on various OGBench domains.", "questions": "Questions\n1. How do you plan to address the issue with CIs based on 4 seeds?\n2. Can you explain how hyperparameters were tuned? (see above)\n3. Is it possible to run an experiment compared to a stronger offline method, like IQL or Insample AC, with a Gaussian policy? Or can you explain why you do not think this is needed?\n\nThe current decision is based on these omissions, but I am willing to adjust my score based on the responses from the authors. The current scores reflect uncertainty on the outcomes, and Soundness and Contribution would particularly be increased when addressing these omissions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-motivated: the problem being addressed is clear, and there are clear arguments for using AM.\n\nThe experiments compare to many different algorithms.", "weaknesses": "The theoretical guarantees are not detailed, and the word convergence only appears in the intro and conclusion. The derivation relies strongly on SOC objectives from Adjoint Matching, and a clear motivation of the formulation in RL setting, with clearly stated assumptions can make the paper self-contained.\n\nThe experiments are quite comprehensive in scope, with many algorithms, which is laudable. However, there are currently three key issues to address. \n\nThe first is around how confidence intervals are computed. To quote:\n“We report the means and the 95% bootstrapped confidence intervals are computed over 4 seeds”\nThis is an issue. You cannot get bootstrap confidence intervals from 4 seeds. The number of seeds should be increased, or CIs should only be reported using aggregate performance across environments. I understand experiments can be expensive, however, it is key to only make claims supported by the evidence. It is possible to only show qualitative behavior, such as individual runs in an environment (with 4 lines), and then make stronger claims aggregating across environments. Or, of course, the number of runs (seeds) can be increased.\n\nThe second key concern is how hyperparameters were chosen. Appendix D lists the hyperparameters and states that they needed to be tuned for each domain. How was this done? When tuning per environment, it is key to control for the number each method gets to tune, and in general can be difficult to keep fair. Tuning hypers can be very misleading in terms of the actual utility of an algorithm, because different settings of hypers can essentially give you a different algorithm. Ideally, it is better to tune across environments or at least explain why the current tuning is valid and not misleading.\n\nThe third key concern is a missing baseline (in Table 1): a strong offline RL method that is not focused on diffusion policies. RLPD is used later, which just uses a Gaussian, but it is not an offline method. In general, for both the offline setting and the offline-online setting, it could be more informative to consider IQL. Note also that one baseline IFQL uses IQL, so it is sensible to include for this reason too. It is a strong good baseline for the offline experiments. And, for offline-online, it is also likely a stronger baseline that RLPD, since RLPD does not make as much use of the offline data as IQL. I think it is key to include IQL, or another method that has been shown to perform well in offline and offline-online, such as In-sample Actor Critic (see https://arxiv.org/abs/2302.14372). This method has an update that is a lot like SAC with a small modification to handle out-of-distribution actions.\n\n\n(Putting Minor Comments here, because there is not separate box for them. These Minor Comments are not big weaknesses.)\n1. “more convenient to directly use the same offline RL objective for both offline pre-training and online fine-tuning” Why is it more convenient? There are stronger reasons for doing so and should be listed here.\n\n2. The paragraph at the beginning of Section 2 is too long and should be split up into multiple ideas. Long paragraphs throughout should be broken into multiple, with clear topic sentences, for readability and clarity.\n\n3. It is a bit confusing to use v and r in the formulas for adjoint matching, even if that is their original terminology, due to the clash with value functions in RL and the fact that you will use q rather than r. You could simply use q right away, and consider a different variable for v\n\n4. The sensitivity analysis is highly appreciated, but there could be just a little bit more there. In the sensitivity analysis in Figure 4, there is not a lot of difference between the values of tau nor without gradient clipping. This suggests more values of tau should be tested to truly see when there is failure. This is not critical, but would make this result more useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HnGaiHN0oa", "reviewer": "Reviewer_V3Vy", "summary": "This paper proposes a TD-based reinforcement learning algorithm to efficiently optimize an expressive diffusion or flow-matching policy with respect to a parameterized value function. While the effectiveness is validated, there is a lack of theoretical proof for the proposed approach, and the experiments are not comprehensive.", "questions": "1. The paper lacks a theoretical proof to demonstrate the stability of the proposed method and to explain why it outperforms existing methods.  \n\n2. It is recommended to further demonstrate the applicability of the proposed method on tasks beyond long-horizon tasks, such as the MuJoCo tasks in D4RL.  \n\n3. A comparison with more state-of-the-art offline-to-online DRL methods should be included in the experiments.  \n\n4. How does the performance of the proposed method vary with changes in the number of critics, and does the number of critics affect Q-value evaluation?  \n\n5. The proposed method includes many components. It is recommended to analyze the time and space complexity of the proposed method, and to compare its throughput and memory usage with baseline methods in the experiments to assess practical hardware efficiency.  \n\n6. How does the performance of the proposed method change as the offline dataset size decreases and its quality declines?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novel algorithm\n2. Effectiveness validated", "weaknesses": "1. Lack of theoretical proof\n2. Incomprehensive experimental setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GH4YvMiDBb", "reviewer": "Reviewer_YuTk", "summary": "This paper proposes Q-learning with Adjoint Matching (QAM), a novel temporal-difference (TD) based reinforcement learning algorithm that addresses the challenge of optimizing expressive flow/diffusion policies with respect to a parameterized critic function. The key innovation is leveraging adjoint matching, a technique from generative modeling, to utilize the critic's action gradient directly, thereby avoiding the numerical instability that arises from backpropagation through multi-step denoising processes.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novel and well-motivated approach: The paper addresses a fundamental tension in continuous-action RL between policy expressivity and optimization tractability.\n2. Strong theoretical foundation: The method builds on solid mathematical principles from stochastic optimal control and adjoint methods.\n3. Comprehensive experimental evaluation: The paper includes extensive comparisons against eight representative baselines across multiple challenging domains. The categorization of baselines (DDPG-based, value-only, gradient-with-approximations, post-processing, Gaussian) is thoughtful and covers the landscape of existing approaches well.\n4. Practical considerations: The authors provide detailed implementation guidance, including discrete approximations, gradient clipping, and ensemble critic training.\n5. Clear presentation: The paper is generally well-written with good motivation, clear problem formulation, and helpful visual aids.", "weaknesses": "1. Insufficient stability analysis in experiments: Although the success rate is reported, the paper lacks metrics that directly assess stability, such as standard deviation across seeds or training loss curves. These would provide concrete evidence of the claimed stability advantages.\n2. Limited domain diversity: All evaluated tasks are from OGBench with similar characteristics (long-horizon, sparse rewards). The generalizability to dense reward tasks (e.g., MuJoCo locomotion in D4RL), tasks with different action space dimensions, shorter horizon tasks, and continuous reward settings remains unclear. This significantly limits the assessment of the method's broad applicability.\n3. Impact on Q-value estimation: The paper does not analyze how QAM affects the critic learning process. Since the policy used for target value backup changes during training, this could introduce additional variance or bias in TD targets. An empirical analysis that shows Q-value estimation error over training, compares learned Q-values with Monte Carlo returns, and examines the impact on critic ensemble disagreement would provide important insights.\n4. Incomplete sensitivity analysis: The impact of ensemble size and pessimistic coefficient is not studied, despite being set to specific values. No ablation on the choice of \"memoryless\" SDE vs. alternative noise schedules. The interaction between temperature and clipping is not explored.\n5. CGQL is presented as a \"novel baseline\" without prior work validation. Several baselines (DSRL, FEdit) have been modified from their original implementations, which may not represent their optimal performance. The choice to disable best-of-N sampling for IFQL may unfairly handicap it. \n6. Missing ablations: What happens if using the \"basic\" adjoint matching (Eq. 12) instead of the \"lean\" version (Eq. 14)? How does QAM perform with different critic architectures or learning rates?\n7. Limited theoretical analysis of stability: While the paper claims that adjoint matching avoids instability by not backpropagating through the residual flow model, the theoretical justification is largely intuitive. The discussion on pages 5-6 about ill-conditioned gradients is informal. A rigorous stability analysis (e.g., condition number bounds, gradient variance analysis, or convergence rate guarantees) would significantly strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)