EXPO: Stable Reinforcement Learning with Expressive Policies
Authors: Perry Dong
Qiyang Li
Dorsa Sadigh
Chelsea Finn
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Imitation Learning
Primary area: reinforcement learning

Abstract: We study the problem of training and fine-tuning expressive policies with online reinforcement learning (RL) given an offline dataset. Training expressive policy classes with online RL present a unique challenge of stable value maximization. Unlike simpler Gaussian policies commonly used in online RL, expressive policies like diffusion and flow-matching policies are parameterized by a long denoising chain, which hinders stable gradient propagation from actions to policy parameters when optimizing against some value function. Our key insight is that we can address stable value maximization by avoiding direct optimization over value with the expressive policy and instead construct an on-the-fly RL policy to maximize Q-value. We propose Expressive Policy Optimization (EXPO), a sample-efficient online RL algorithm that utilizes an on-the-fly policy to maximize value with two parameterized policies -- a larger expressive base policy trained with a stable imitation learning objective and a light-weight Gaussian edit policy that edits the actions sampled from the base policy toward a higher value distribution. The on-the-fly policy optimizes the actions from the base policy with the learned edit policy and chooses the value maximizing action from the base and edited actions for both sampling and temporal-difference (TD) backup. Our approach yields up to 2-3x improvement in sample efficiency on average over prior methods both in the setting of fine-tuning a pretrained policy given offline data and in leveraging offline data to train online.

Published as a conference paper at ICLR 2026
EXPO:
STABLE REINFORCEMENT LEARNING WITH Ex- PRESSIVE POLICIES
Perry Dong Stanford University
Qiyang Li UC Berkeley
Dorsa Sadigh Stanford University
Chelsea Finn Stanford University
ABSTRACT
We study the problem of training and fine-tuning expressive policies with online reinforcement learning (RL) given an offline dataset. Training expressive policy classes with online RL present a unique challenge of stable value maximization. Unlike simpler Gaussian policies commonly used in online RL, expressive policies like diffusion and flow-matching policies are parameterized by a long denoising chain, which hinders stable gradient propagation from actions to policy parameters when optimizing against some value function. Our key insight is that we can address stable value maximization by avoiding direct optimization over value with the expressive policy and instead construct an on-the-fly RL policy to maximize Q-value. We propose EXpressive Policy Optimization (EXPO), sample-efficient online RL algorithm that utilizes an on-the-fly policy to maximize value with two parameterized policies a larger expressive base policy trained with stable imitation learning objective and a light-weight Gaussian edit policy that edits the actions sampled from the base policy toward higher value distribution. The on-the-fly fly policy optimizes the actions from the base policy with the learned edit policy and chooses the value maximizing action from the base and edited actions for both sampling and temporal-difference (TD) backup. Our approach yields up to 2-3x improvement sample efficiency on average over prior methods both the setting of fine-tuning a pretrained policy given offline data and in leveraging offline data to train online.
Code: ht / / www.th.wo.oooc.tttoooooooo.o.ttt ccomppde pddppprrr/EPP
INTRODUCTION
Robotics has seen significant progress on challenging real-world tasks by training expressive policies on large datasets via imitation learning (Black et al., 2024). Despite promising results, imitation learning methods often struggle to achieve the high reliability and performance needed for real world use-cases, even when scaled to large datasets. Fine- tuning these policies with reinforcement learning (RL) can in principle address this problem by enabling high performance through online self-improvement. Yet, existing online reinforcement learning methods are typically designed for simple Gaussian policies (Schulman et al., 2017; Fujimoto al., 2018) and do not effectively leverage expressive pre- pre-trained policies, such as diffusion or fow-matching policies (Chi et al., 2023) typically used in imitation learning. Can we design an efficient and effective RL fine-tuning method for expressive policy classes?
Fine-tuning expressive policies with online RL comes with unique challenge not present in fine-tuning simpler Gaussian policies expressive policies like diffusion or fow-matching policies are parameterized by a long chain of denoising steps, which hinders stable gradient propagation from the action output to the policy parameters whenever we want to optimize their actions against some value functions (Ding purely online settings, many approaches have sought to avoid the gradient propagation instability by incorporating losses at intermediate denoising steps to guide the denoising process towards high-value actions (Psenka al., 2023; Fang et al., 2024), but it is still not obvious how to best perform stable value maximization for efficient online fine-tuning.
Jin, 2024; Park et al., 2025). In the adjacent purely offline or
In this work, make the key observation that value maximization of expressive policy classes can be made much more effective and stable by avoiding direct optimization over value of the expressive policy itself. Instead, train the base expressive policy using stable supervised

---

Published as a conference paper at ICLR 2026
Pretraining
Fine-tuning
Env
Average Environments
9,1 +
Offline Dateot
Keplay Buller
Average Norneliied Retums
0.5
fon-the-lly
0.-
Pretrain
Fine-tune
srgmer ?
Policy
Lyc
Liz
00
Palicy
Base
Policy
Edit
Sample Actions Action Editss
On-The-Fly Action Extraction
Envirecment Steps (x1000)
Value
EXTT
BLPO
IDQ
DAC
CAL QC
Figure I: Left: Expressive Policy Optimization (EXPO) is a stable, sample efficient method for training expressive policies with reinforcement learning by avoiding direct optimization over the value function with the expressive policy. Right: Average performance over tasks of EXPO and prior methods.
learning objective and construct an on-the-fly policy to maximize value through two steps - (1) light-weight, one-step edit policy that refines the action samples from the base expressive policy, and (2) non-parametric post-processing step that takes multiple action candidates from the base and edit policy and selects the highest-value action among base and edited actions. We impose an edit distance constraint on the edit policy such that the edited actions remain close to the original actions from the base policy. This restricts the edit policy to solve a simpler, local optimization problem. allowing it to be much smaller than the base expressive policy and enabling efficient and stable optimization. The local edits can be viewed as refining actions within modes of the base policy's s action distribution, which is complemented by the second on-the-fly, non-parametrie post-processing step, which considers multiple pairs of base and edited actions potentially from different modes and selects the best actions.
We instantiate these insights as EXPO, a sample-efficient online RL algorithm that enables stable online fine-tuning of expressive policies. EXPO consists of two parameterized policies: a base expressive policy that is initialized from offline pre-training and then online fine-tuned with an im- itation learning objective, and a small Gaussian edit policy that is trained with standard policy loss in reinforcement learning to maximize the Q-value of the edited action. The base policy is never trained to explicitly maximize value. Instead, we construct an on-the-fly policy to maximize Q-value by optimizing the actions from the base policy with the learned edit policy and selecting the best action from the base and edited actions according to their Q-values. The on-the-fly extraction has the advan- tage that any changes in the Q-function are more immediately reflected in both the agent's 's behavior and the TD valuee target, unlike standard policy extraction methods that require slow parameter updates align the policy to the Q-function. In addition, the edit policy can be trained with entropy regularization, offering convenient way to add state-dependent action noises for online exploration beyond the behavior distribution, which is often challenging to do with expressive policies alone.
Our main contribution is a simple yet effective method for online RL fine-tuning of expressive policy classes, EXPO. Our method is stable to train and unlike many prior works that focus on particular class of policies (e.g., diffusion, flow-matching), our method is agnostic to policy parameterization and can fine-tune from any pre-trained policies. We evaluate our method on 12 tasks across 4 domains and find that our approach achieves strong performance in both online RL and offline-to-onllee RL setting with up to 2-3x improvement sample efficiency on average.
2 RELATED WORKS
Reinforcement Learning with Prior Data. To improve the sample efficiency of online RL, prior works have studied the problem of using an offline dataset to accelerate online learning (Li et al., 2023; Ball et al., 2023; Hu et al., 2023; Dong et al., 2025). A common strategy in this setting is to simply initialize the replay buffer with offline data (Vecerik et al., 2018; Nair et al., 2018; Hansen et al., 2022; Ball et al., 2023). Another line of work focuses on pretraining a good value function or policy using pessimism fine-tuning (Hester et al., 2017: Lee et al., 2021; Nair et al., 2021; Song et al., 2023; Nakamoto et al., 2024). Yet, other methods maintain separate polices for offline pretraining and online fine-tuning (Yang et al., 2023a; Zhang et al., 2023; Mark et al., 2023). However, these methods still mostly rely on simple Gaussian policies. In contrast, EXPO aims to utilize the capacity of expressive policy
policy constraints typically employed in offline RL, followed by online
2

---

Published as a conference paper at ICLR 2026
classes to capture more complex behavior distributions to accelerate learning and enable fine-tuning of pre-trained models using these expressive policy classes.
Reinforcement learning with expressive policies started to gain popularity in RL to help handle more complex action distributions. A central focus of these methods is to extract an expressive policy that simultaneously maximizes the Q-function and stays close the offline dataset. Lu et al. (2023); Kang et al. (2023); Ding et al. (2024); Zhang et al. (2025) use weighted behavior cloning (BC) to imitate dataset behavior while maximizing action Q-values. While weighted BC is the most simple policy extraction method that can take into account Q-function signals, prior works (Fu et al., 2022; Park et al., 2024; 2025) have found other policy extraction methods often performs better. Yuan et al. (2024); Ankile et al. (2024) pre-trau ire-train an expressive policy on the offline data and then learn a residual policy online to refine the actions from the base policy. In contrast to these works, we focus on performing fine-tuning on the expressive policy itself, which can be crucial to fully leveraging the capabilities of the expressive policy to not only enable better sample efficiency, but also to more adaptive online. Lastly, Ren et al. (2024) reformulate the diffusion process as augmented MDP on top of the original MDP and use policy-gradient methods (e.g., PPO (Schulman et al., 2017)) to train the policy. Ankile et al. (2024) also uses on-policy method to train the residual policy. Compared to these works, we focus on developing off off-policy TD-based methods for better sample efficiency.
Fine-tuning diffusion policies with value gradients. The simplest way of leveraging the gradient of Q-functions for policy extraction is to backpropagate the Q-value into the policy parameters (Fujimoto et al., 2018; Haarnoja al., 2019). While it is possible to directly apply this technique in diffusion policies ( (Wang et al., 2022), the backpropagation can get prohibitively expensive and unstable as the number of denoising steps grows large. Ding & Jin (2024); Park al. (2025) tackles this by distilling the multi-step diffusion policy into less expressive tw0-step/single-ster policy. Psenka et al. (2023); Fang et al. (2024) use action gradients to provide direct supervision on the training of the intermediate denoising steps to bias towards high-value actions. Zhang et al. (2025); Mark et al. (2024) use action gradients as well but in a refinement manner where they first sample actions from the base policies and then improve these actions by hill climbing the Q-function using the action gradients. Our approach draws inspirations from multiple prior works, but importantly instead of backpropagating the gradient through the expressive policy, we leverage Q-function gradients through a separate policy to edit the base actions to maximize Q-value for better stability.
Sampling-baas based maximization. Some prior methods have explored sampling- based techniques to optimize Q-values. Ghasemipour et al. (2021) samples actions from the behavior policy and chooses the action that gets the highest Q-value and uses MADE (Germain et al., 2015) to model the behavior distribution. In contrast, we study more expressive policies for better performance. Hansen-Estrus Estruch al. (2023) and He et al. (2024) use expressive diffusion-based policies and sampling based Q-function maximization only for online exploration and not TD backup. In our experiments, we find using maximum action selection for both TD backup and online exploration be crucial for online sample efficiency. Chen et al. (2022) generalizes to softmax selection instead of a hard max for choosing actions based on the highest Q-values. Our method draws on ideas from these prior works, but focuses on maximizing value in stable way to address the problem of fine-tuning expressive policies. We show through experiments not only the importance of our on-the-fly action extraction, but also editing the base actions toward higher value distributions. The design choices in our algorithm enable online RL to be more than 2x more data efficient than prior works.
3 PROBLEM SETTING
We consider a Markov Decision Process (MDP), defined as (S,A, T, where S is the state space, A is the action space, r S x A + R is function defining the rewards, T(s'Ja, s) is the transition dynamics, 7 E [0, 1] is the discount factor, and p(s) is the initial state distribution. At timestep i, the RL agent observes state 8, and chooses action or by sampling from its policy <a. (s)). The goal of RL is to maximize the expected sum of discounted returns E 1 7 ot)]. In this paper, we study the setting where we additionally have access to a pre-trained expressive policy Tpre (e.g., a diffusion policy, a flow policy) as well as a prior dataset D As the agent interacts with the environment, it observes (s, a, r. s') tuples that are appended to replay buffer D for training. Our main goal is to online fine-tune the pre-trained expressive ग pre in a sample-efficient way by effectively leveraging both the prior dataset Do and the online replay buffer data D.
3

---

Published as a conference paper at ICLR 2026
4 EXPRESSIVE POLICY OPTIMIZATION (EXPO)
In this section, explain the two key components that allow EXPO to leverage a base expressive policies for sample-efficient online fine-tuning without explicitly optimizing the expressive policy for maximal rewards. The first component is an edit policy that refines the actions generated from the base policy to simultaneously maximize Q-value while encouraging exploratory actions. The second component is an on-the-fly policy parameterization for online training by selecting the maximmmax action among the original and edited actions. We also present a version of EXPO with entropy backup for data-limited regimes. Lastly, we describe the implementation details required to make our method effective in practice. The full EXPO algorithm is summarized in Algorithm 1.
Dataset Actions Basa Actions Edited Actions
4.1 Q-VALUR MAXIMIZATION AND EXPLORATION THROUGH ACTION EDITS
The edit policy transforms ac- tions of the base policy into actions that further maximize Q-value while encourag- ing exploration. The blue contour repre- sents the Q-values of actions of a single state and the orange contours represent the Gaussian distributions of actions the edit policy changes the base actions into.
Figure 2:
To avoid the unstable explicit value maximization of expres- sive policies, we use an imitation learning objective train the base policy, which has been shown to work stably and re- liable across a variety of expressive policy classes. However, training with imitation learning alone does not effectively move the distribution to high-value actions. To this end, the first component of EXPO is a Gaussian edit policy, Tedi (āls, a), that refines actions generated by the base expressive policy (a ~ Thase (+))):
(1) Intuitively, we want to train the edit policy to locally optimize the Q-function and maximize the action entropy to maintain action diversity. Such action diversity is especially important when the base expressive policy is trained on narrow behavior distribution. We do so by training the edit policy Tedi with a standard entropy- hetropy-regularized policy loss:
a à
L (Tedit) = (ss.))_D.aaaae a- (s [Qo(s, + a) log Tedit Fadi(([ a)]
(2)
with Qs(s, a) being the critic value we want implicit policy maximize.
The edit policy can be viewed as transforming each action sample from the base policy toward a higher Q-values Gaussian action distribution. We illustrate this in Figure 2. However, naively learning this edit can shift the actions too far from the behavior distribution that causes the policy to deviate from desirable behavior. We address this by simply enforcing the action edits to be close the actions sam- pled by the policy by scaling á to be between [-3, 3). where 33 is hyperparameter. In practice, / can be small (e.g., 0.05) or large (e.g., 0.7) depending on how much exploration is needed refine the ac- tions from the initial distribution of the offline dataset. This enables the policy to continuously improve upon the actions generated by the base policy while not deviating too far from reasonable behavior.
4.2 ON-TIE-FLY PARAMETERIZATION OF THE RL POLICY
Given the base and edit policies, we need a way to effectively extract dalleemaximizzng actions that account for both the expressivity of the base policy and the malue-maximization of the edits. We construct an on-the-fly (OTF) policy to perform implicit value-maxinization in two steps: (1) generating action samples using the base and the edit policy and (2) selecting the highest Q-value action. We use this on-the-fly policy for both sampling and in the TD backup.
Let hoTF be the on-the-fly policy that implicitly performs value maximization. Toor(aas,, Thase, Redit; )) is defined as arg max,- (a..a.) (o a), where a; is an action sampled from Thase and à; = a; à is the action after edit for each of N action samples. Because the edit policy is trained to maximize the Q-function, the edited actions should better represent what the Q-function views as optimal.
Taken together, the Q-function objective becomes
min E ((r, (r+ + QR (S141, a(+1)= - Qus (e))), where دح41 ~ TOTF(-1s(41) (3)
4

---

Published as a conference paper at ICLR 2026
Algorithm Expressive Policy Optimization (EXPO)
Require:
Prior dataset Daata ($1 ): optionally, expressive policy initialization Wbase-
Randomly initialize action edit policy Tedit, critic Qs. target critic Qø'. UTD ratio G. while training do
for each environment step do
Collect rollouts:
Sample az from TOTE (-1s, Those Tedits 8²)
Take action áz and observe 7y and 8141 from the environment
Store (st, at, Tt, St+1) in RL replay buffer
Update policy and critic:
for g I,..., G do
Sample mini-batch (s,a,r, s') from the replay buffer
Sample à=' from #OTF (-sp Thase Todit: o')
Compute target as = r + DQ (s', )²) Update á minimizing loss:
(y (s)
Update target networks: 0' - p0 + +1 p)-
Update Thase using the last mini-batch with supervised learning objective CII (Mass)
Update Tedit using the last mini-batch maximizing objective (s,s, + a)
a log Tedit (āls), â ~ Weau())s)
We note that because the on-the-fly policy is parameterized to maximize the Q-function and the action 4;41 is the action sample with the highest Q-value, this procedure can be viewed as equivalent to standard Q-learning update with the implicit policy.
4.3 ENTROPY BACKUP FOR DATA-LIMITED REGIMES
In scenarios where the offline dataset is not sufficiently large or broad, the agent must explore more broadly during online sampling. For these scenarios, we propose framework for incorporating an entropy bonus into both the base and edit policy training. Viewing the base and edit policies as one OTF policy, the standard entropy- regulaazeed RL loss, which has been shown to benefit exploration (Ziebart, 2010; Haarnoja et al., 2017; 2019), can be formulated as
+ 71Q (51+1, a log (or1(044115411)
L(Ø) B (e > Q~ a. a.) / ",,411 TOTF (-)5(1)4+11)
(4)
L(RorF) = - (a..)) (s,a) (Q a) - a log Toir(a)s,o))
(5)
However, one cannot directly apply this loss to the base policy. as many expressive policies such as diffusion do have closed form expression for entropy. We instead construct soft sampling distribution Tsampling that first samples N actions from Thsse and edits them into ā = a; + àg for each action, and then chooses actions following the probability distribution
Tsampling (a.ls) = Σ. (Q(r,e) We can obtain a closed-form equation for this sampling distribution and use this entropy to perform the backup. As will see in Section 5.5, this modification can improve performance when the offline dataset is small.
exp.aQ(s.a,)
4.4 PRACTICAL IMPLEMENTATIONS
In this paper, we instantiate EXPO with the base policy being diffusion policy trained using DDPM. The training objective the following:
min (ooo -cce vI ãee, s, t)
While we use diffusion policy for the main experiments canonical example of expressive policy, this framework general to any expressive policy class. We train the edit policy with simple Gaussian with entropy regularization implicitly parameterized policy even though the base expressive policy is trained with an imitation learning objective. We turn off entropy in the target for the entropy backup version of the method.
done in SAC, where the entropy promotes exploration in the
5

---

Published as a conference paper at ICLR 2026
antersaze- antrraae-legg--iivve 2
umimare-lang--pleee playy-
wntmaze-meismddivveee
131
13
Nornalized Retuens
Raturns
Normallaod Retuuns
DX
Environment Steps (K30000)
Environament steps (XTUC0)
Environment Steps (X3000)
Environment Steps (ST0C0)
BB
can
square
pmn-binary-v0
Normalase Retunis
Normalized Retunus
Ca-
28
10
18
18
30
130
Envinonment
Steps (s0000)
Environment Steps (s1000)
Envinorment Steps (s10000)
Environment Steps (s1000)
dooe binary vt
relocate binasy vtl
thasading
stack
Neeualtss Roturs
Neemalizo Roturss
Nomaltzd
Normaltad
100
200
34
200
130
130
28
130
13t
2X
Envinorumnnt Steps (=10000)
Envinonamnnt Sheps (=10000)
Envinammant Steps (=1000)
Envinonment Steps (=1000)
EXPO (ours)
KLPO
IDQL
DEP
DAC
QSM
Figure 3: Online RL results on 12 challenging sparse-reward tasks. Across almost every task, EXPO consistently exceeds or matches the performance of the best baseline- even without any pretraining.
5 EXPERIMENTS
In this section, we aim answer the following core questions through our experiments:
(Q1) Can EXPO effectively leverage offline data for online sample-efficient RL?
(Q2) How sample efficient is EXPO in fine-tuning pretrained policies compared to prior methods?
(Q3) What components EXPO are most important for performance?
5.1 BENCIMARKS
We evaluate EXPO on 12 challenging continuous control tasks spanning various embodiments. All of the tasks feature sparse rewards. We present these tasks in Figure 11 in the Appendix. The Antmaze evaluation suite from D4RL (Fu et al., 2021) features controlling quadruped ant to navigate a maze and reach the desired goal position. The suite consists of mazes medium and large sizes. The Adroit environments from D4RL involves controlling a 28-Dof to spin a pen (pen-binary-v0) open door (door-bingg www-bbiiaryyvvo vv), and relocate ball (rel Locaee-iiiiic bina bareebbiinaaaaaeeeeeee vC) The RL policy needs not only to learn dexterous behavior to operate in the high-dimensional action space but also explore beyond the narrow dataset to successfully complete the tasks. The Robomimic (Mandlekar et al., 2021) and MimicGen (Mandlekar al., 2023) tasks involve controlling DoF Franka robot arm to complete manipulation tasks. For Robomimic, we evaluate on Lift, Can, Square, which require lifting block, picking respectively. For MimicGen, we evaluate on Threading and Stack, which require threading a needle into pin and stacking small cube top of large cube, respectively. We initialize the dataset with successful demonstrations in all settings and tasks. We refer to the detailed setup in Section B.
and moving it to the correct bin, and inserting tool onto square peg,
5.2 BASELINES
evaluate our method in both the online setting (no pre-training) as well as the offline-to-online setting (offline pre-training followed by online fine-tuning). We compare method against prior state-of-the-as art methods in each setting with a focus on methods that leverage expressive policies. As there many existing fffline-to-onlig RL methods with expressive policies, also compare to existing offline RL methods with expressive policies by directly fine-tuning them online. We describe the baselines in detail in Section C.

---

Published as a conference paper at ICLR 2026
For the offline-to-online RL setting, we use imitation only to pre-train the base expressive policy of EXPO. This is different from other offline-to to-niee RL baselines such as IDQL, Cal-QL, DAC, which all use offline RL to pre-train both the policy and the value network. We only pre-train the base policy as many pre-trained robotic models do not come with a pre-trained value function. We want our method be general and be able fine-tune from any pre-trained policy. For Adroit, do not pretrain for EXPO due to the narrowness of the dataset.
5.3 CAN EXPO EFFECTIVELY LEVERAGE OFFLINE DATA FOR ONLINE RL?
We first test whether EXPO can leverage signals from offline data of demonstrations to effectively explore and learn in an online setting. We present the results in Figure 3. We find that EXPO far exceed in performance in terms of sample efficiency compared to baselines on almost every task. Comparing against RLPD, which is a method known for its fast learning in the setting of leveraging prior data, we find that EXPO consistently achieves significantly better sample efficiency with the exception of relocate -iieeribi-rrrreeee v0 which features a very narrow dataset such that it challenging for imitation learning to extract useful behavior from. All of this performance gain comes without pretraining on the offline data. While RLPD can learn efficiently by oversampling from the dataset, takes a long time for the policy to discover optimal strategies, even when the information is in the offline dataset. Because EXPO is training the base policy with imitation learning, it is able to leverage signals to learn behaviors very quickly through sampling behavior close to the behavior data, and then refine those actions through the edit policy to further explore and improve in performance. Comparing against IDQL, DIPO, and QSM which uses more expressive policy classes such diffusion, we find that these methods are often not able to learn effectively. IDQL, while also training the base policy with imitation learning and extracts actions implicitly, only does so for sampling and constrains the value function the offline data. QSM, while in principle can learn the policy by matching the diffusion loss to action gradients, in practice often struggles to learn effectively on the challenging continuous control tasks, possibly due to instabilities in the training objective. In contrast, through a stable way of value maximization, EXPO leverages the power of expressive policy classes to achieve even better performance than simpler policy classes.
5.4 HOW SAMPLE EFFICIENT EXPO IN FINE-TUNING PRETRAINED POLICIES COMPARED TO PRIOR METHODS?
Having established the effectiveness of EXPO leverage signals from offline datasets to effectively explore and learn, we turn our attention to the offline-to-online setting, where the policy is pretrained on the offline dataset and then finetuned. We present the results in Figure 4. EXPO achieves significantly better sample efficiency and asymptotic performance overall compared to baselines, despite only pretraining the policy using imitation learning. Crucially, compared traditional oflline-o-online RL methods, EXPO does not experience a large drop in performance from offline pretraining to online fine-tuning, despite randomly initializing both the Q-function and the edit policy. This is because EXPO can limit the amount of distribution shift going from offline to online as the base expressive policy generates actions that are close to the behavior distribution. While the edit policy maximizes the Q-value and expands the distribution to encourage exploration, it does so close to actions sampled by the base policy. Compared to IDQL with pretraining, we find that IDQL was generally not able to improve performance of the policy online after pretraining in the Antmaze and Adroit tasks, likely because of the policy constrained objective that constrains it too much the behavior distribution combined with a lack of exploration capabilities. Cal- and
QL obtains strong performance on easier tasks such as antmaz ze re-eedi um um-di antmaze- -ee--miiii di um- 1are-e.. but on the harder tasks has much lower overall sample efficiency despite having calibrated Q-function from offline pretraining to start, as it is not able to effectively leverage signals from the offline dataset for policy improvement. DAC obtains strong pretraining performance as it takes advantage of the expressivity of diffusion models, but collapses quickly for online training, making it infeasible for fine-tuning pretrained models. With the exception of RLPD, all baselines experience an overall drop in performance going from offline to online on the Robomimic and MimicGen tasks, likely because of the precision required to complete these fine-grained manipulation tasks. In contrast, EXPO consistently improves significantly Robomimic and MimicGen tasks with high sample efficiency as the policy stays close to the behavior distribution while continuously refining the actions in a stable manner for better performance.
all of the
7

---

Published as a conference paper at ICLR 2026
antersazee antroaae-langg--iieee 2
umimare-larg-playaee
antmaze-edss-diiveee
Nornaltzdd Retuens
Normaltzd Raturn
Nornallzed Retuens
2
25
Environment Steps (630000)
Environment steps (XT000)
Environment Steps (x3000)
Environmentt Steps (STUC0)
BR
can
square
pmn-binary-v0
331
Normalast Retumis
Normallzed Retunus
Retuma
C++
28
00
130
18
100
50
28
130
Envinonment Steps (s10000)
Environment Steps (s1000)
Envinonment Steps (s10000
Environment Steps (s1000)
dooe binory vt)
relocate binasy vt
thasading
stack
Nemallod Roturs
Noema comaallzdd Returns
Normaltad Returas
Nomaltad
2
8
3X 150
200
3-0
500
130
130
28
130
136
25
Envinonumnnt Steps (=10000)
Envinonamntt Steps (x10000)
Envinaramant Steps (=1000)
Envinonment Steps (=1000)
EXPO
RLPD
DAC
CAL-QU
Figure 4: Offlinettoonllne RL results on 12 challenging sparse-reward tasks. EXPO consistently exceeds matches the performance of the best baseline. The relative benefit of EXPO over baselines is especially large the manipulation tasks, where prior methods often struggle improve in performance. Importantly, EXPO does drop in performance going from pre-training fine-tuning.
5.5 WHAT COMPONENTS OF EXPO ARE MOST IMPORTANT FOR PERFORMANCE?
To better understand the significance of different pieces of EXPO, we ablate over three key com- ponents: (1) the importance of on-the on-the-fly policy extraction in the TD backup, (2) the effectiveness of action edits, and (3) the importance of the behavior distribution in the offline data. We present additional experiments on fine-tuning a pre-trained policy without the offine dataset in Section A.
squaee
pes-birarystl
squan
Nomalteed setume
Nomalize Rerurrs 2
Nommlted
Envimominnn Siepo (30000)
Envinomennt Steps (x1000)
Environiennt Stepa (11000)
Eavimement Stepe (x1000)
EXFO
No Implisi Panmerticatonn Backup
EXPO
Action
Figure 5: Ablation on-the-fly policy extrac- tion the backup. We that using value- maximizing actions in TD backup is vital for perfor- mance.
Figure 6: Ablation action edits, often hard to improve pretrained policies online since the base policy by itself does not effectively explore maximize Q-value.
action edits. Without
How important is the on-the-fly policy in TD backup? Prior methods such as IDQL have explored sampling from an expressive imitation learning policy and choosing the highest Q-value action for sampling. While this parameterization is different from EXPO, the Q-value is also not used as gradient signal to explicitly extract the policy. However, as the experiment results show, EXPO performs substantially better than IDQL in both online and offline-to-onllne understand the role of on-the-fly value-maximization action extraction for sampling, which corresponds to only sampling one action and using that action to compute the target Q-value, versus EXPO which extracts value maximizing actions for both sampling and backup. We present the results for Robomimic Can and Square in Figure 5. We see that on the-hh--f fly policy extraction in the TD backup is crucial for high performance and sample efficiency. This is because while the policy is trained implicitly maximized actions sampled in
settings. To better we ablate over only performing on-the-fly
8

---

Published as a conference paper at ICLR 2026
rollout, the policy is still trained with an imitation learning objective, and as such the action sampled from the policy during TD backup does not naturally maximize the Q-function and thus performs a SARSA-like objective, which is known to have slower learning than Q-learning.
How effective are the action edits? To better understand the role of action edits, we compare to not using action edits and only sampling actions from the base expressive policy and choosing the action with the highest Q-value. We conduct the ablation on requires more exploration to learn the optimal behavior, and Square, a task that benefits from more fine-grained refinements as the initial dataset contains useful signals to extract behavior policy that can get a reasonable success rate. We show the results in Figure The policy for pen-binary-ve is pretrained for 20k steps and the policy for Square is pretrained for 200k steps. We see that for both environments, action edits are crucial for better performance. On pen-bi waa--oinnnaayyaaa where the policy requires more exploration, removing action edits resulted in convergence to very suboptimal performance as the expressive policy trained with imitation learning has no mechanism to effectively explore beyond the behavior distribution. Even on Square, where the offline dataset contains good enough data to learn an imitation learning policy to a reasonable success rate, action edits are still very important to enable the policy to continuously refine its actions to improve.
an environment that
How does the offline dataset size affect perfor- mance with and without entropy backup? Be- cause EXPO trains the base expressive policy with im- itation learning, a natural question to ask is how does the offline dataset impact fine-tuning performance. To analyze the role of the offline dataset for EXPO, we subsample different number of demonstrations from the offline dataset for the Square task and plot the success rate of the online fine-tuned policy at 1M environment steps against the success rate of an imitation learning policy trained on the same subsam- pled offline dataset for both EXPO and EXPO with entropy backup. We show the results in Figure 7. We see that there is a clear pattern between fine-tuning performance and the quality of the offline dataset for EXPO without entropy, where better offline data as measured by how well an imitation learning policy trained on the data performs results in better fine- tuning performance. We note that this is perhaps not surprising as both the action edits and on-the-fly value maximization rely on the assumption that the prior contains enough signals to learn useful behaviors. This also explains the lower relative sample efficiency on rel ocate-bl eeatte-binaaee we-bbiiaryy-l as the offline dataset is very narrow and not sufficient for an imitation learning policy to extract useful behavior. However, EXPO with entropy backup using the soft sampling distribution described in Section 4.3 was able to address this problem by incentivizing exploration through the soft action sampling and the entropy bonus, where even an offline dataset with imitation learning performance less than 10% enabled EXPO to learn near perfect policy. Given an offline dataset where the initial policy can learn useful behavior, we find that EXPO with or without en- tropy bonus consistently improves significantly over the pre-trained policy with high sample efficiency.
square
1.0
Normalized Returns
04-
IL Normalized Returns
EXPO Entropy
EXPO
Varying the offline dataset. We find that better offline data, as measured by the perfor- mance of an imitation learning policy trained on the data, correlates strongly with performance of EXPO. The plot is averaged over seeds.
Figure 7:
6 DISCUSSION
In this work, we propose EXPO, a method for training expressive policies with reinforcement learning given an offline dataset. Through constructing an on-the-fly RL policy using two policies, one larger expressive base policy trained with a stable imitation learning loss and one smaller edit policy trained with a Gaussian to maximize Q-value, and choosing the action generated by the policies with the highest Q-value, we address the key challenge associated with expressive policy fine-tuning, namely stable value maximization. Despite the promising results, EXPO has limitations. First, sampling many actions for the TD backup is computationally expensive, these actions need to be sampled for every example in the batch. We leave the problem of how to improve computational efficiency for future work. Furthermore, we assume reasonable prior either through the offline dataset or policy to start training. While in practice we believe this assumption holds in many practical settings, applying our framework to setting with completely uninformed prior is an interesting direction for future work.

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
For reproducibility, we describe all components of our method in detail the main text. We include additional implementation details for hyperparameters and datasets and evaluation protocols in Section B. We also provide a link to code to run EXPO.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Tome, and Pulkit Agrawal. From imitation to refinement- residual rl for precise assembly. arXiv preprint arXiv.2407.16677, 2024.
ressiuaal
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. Wg: iision-laguuage-actton flow model for general robot control. arXiv preprint urXiv:2410.24164. 2024.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273666, 2023.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion- reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv:2405.16177 2024.
Zihan Ding and Chi Jin. Consistency models rich and efficient policy class for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: https://opentts httpss///ppeeeeeeieeee net/ forum?id=yy jwkUNNb.
Perry Dong, Alec M Lessing, Annie Chen, and Chelsea Finn. Reinforcement learning via implicit imitation guidance. arXiv preprint arXiv:2506.07505, 2025.
Linjiajie Fang, Ruoxue Liu, Jing Zhang. Wenjia Wang, and Bing- Yi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. arXiv preprint arXivv2405.2055: 20555, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021. URL https: / aarxivi... hts//bb//2000 07219.
Yuwei Fu, Di Wu, and Benoit Boulet. A closer look at offline rl agents. Advances in Neural Information Processing Systems, 33.8591-8604 2022.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-
critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation, 2015, URL
03509.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-man q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp.
PMLR, 2021.
Tuomas Haarnoja, Haoran Tang. Picter Abbeel, and Sergey Levine. Reinforcement learning with 08165.
deep energy-based policies, 2017. URL https::tpt...mm arwiv.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critie algorithms and applications, 2019, URL
05905.
Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations, 2022. URL.
https: / /aaxivv.. org/ wtgg/bss////ooooooooooooo/ooooo/ 2212. 2, 05698.
Philippe
Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q- g-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
11

---

Published as a conference paper at ICLR 2026
Longxiang He, Li Shen, Junbo Tan, and Xueqian Wang. Aligniql: Policy alignment in implicit q-learning through constrained optimization. arXiv preprint wxXiv22405.88180 2024.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Oulac-Amold, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations, 2017. URL https: /arxiv.
org/abs/ / 1704. 03732.
Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learning. arXiv preprint arX.09231,.02198, 2023.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning Advances in Neural Information Processing Systems, 36: 67195-67212, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, 2021. URL httpps////xxxw..r//bbbbb..ee
06169.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble, 2021. URL https:
//arxiv. abs 2107.000.000 2217700591..
Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36:67434-67458, 2023.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guidee grggy- diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pp. 22825-22855. PMLR, 2023.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni. Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What matters in learning from offline human demonstrations for robot manipulation, 2021. URL https: /ttpxvaxvvv///////////e..///.avl wow.iiiivrrrr.ooooooooooooo abs /2108. 03298.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations, 2023. URL https: : www.i.vvvvv.......aaaaaa
wwweiiivrrrreee wwgg/bb...............e.....e..e 2210 17596.
Max Sobol Mark, Archit Sharma, Fahim Tajwar, Rafael Rafailov, Sergey Levine, and Chelsea Finn. Offline retraining for online rl: Decoupled policy learning to mitigate exploration bias, 2023, URL
https: //axivv...iivva//aa//// org/ alss /2300 08558.
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agmostic rl: Offline and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685, 20244
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcom- ing exploration in reinforcement learning with demonstrations, 2018. URL https: //axiv.. org/abs 1709. 10089.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2021. URL 09359.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244-62261 2023.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma. Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline pre- pre-training for efficient online fine- tuning, 2024. URL httpps://aaet wttpaaaww......gg.aaae/// abs /2300 /203. 05479.
12

---

Published as a conference paper at ICLR 2026
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline r!? arXiv preprint arXiv:2406.09332) 2024.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, arXiv preprint arXiv:2502.0225 2025.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:23/2.117555 2023.
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov. Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint WxXi:2449.00588, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Yuda Song, Yifei Zhou, Ayush Sekhari, J. Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make efficient, 2023. URL https: // //aaxiv..or org/ abs/ /2200.22112222 0 06718.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang. Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning roboties problems with sparse rewards, 2018. http:: //arxiv.org/alla abs 1 1707.0000000 08817.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.065 061933 2022.
Hanlin Yang, Chao Yu, Peng Sun, and Siji Chen. Hybrid policy optimization from imperfect demonstrations. In Advances in Neural Information Processing Systems (NeurlPS), volume 36, pp. 4653-4663, 2023a.
Long Yang. Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning, 2023b. URL
http:////////////5//////// 2305. 13122.
Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, and Hao Su. Policy decorator: Model-agnostic online refinement for large policy model. arXiv preprint arXiv:2412. /3630, 2024.
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. 2023, URL https: arxiv. org/ abs abs/2302. 00935.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy- neighty flow matching for offline reinforcement learning. arXiv preprint arXiv:2503.0497 2025.
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. Phd thesis, Carnegie Mellon University, Pittsburgh, PA, December 2010. CMU-ML-10- 110.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTS
A.1 EXPO WITHOUT OFFLINE DATASETS
To better understand the role of the offline dataset as a prior in EXPO, we study EXPO in the setting of fine-tuning a pre-trained policy without the offline dataset used for pre-training. Instead of retaining the offline dataset, we use the pre-trained policy to collect data to warm-start the training. We present the results on Li ft and Can in Figure 8 and make a comparison to Cal-QL pre-training followed by SAC fine-tuning baseline. For this ablation, we collect the same number of warm-start rollouts contained in the offline dataset used for pre-training. We find that even without retaining the offline data, EXPO was able to learn to solve the tasks with high sample efficiency similar retaining the dataset. This is compared Cal-QL pre-training followed by SAC finetuning, which was not able to solve the task with this setup. This suggests the pre-train policy alone can act as a strong prior for EXPO to fine-tune and improve from, and in the context of pre-trained policies, EXPO can be used for effective, sample efficient fine-tuning even without the offline dataset used to pre-train the base policy.
Nomaliia Rattrrts
Nomallozd Returns
12 x
Envizonment Steps
Environment Stops DC1000)
FX0D FX00))
Offiine Dats
CAT--Q.Cfftee Doline
Figure Ablation on not keeping the offline dataset for fine-tuning, We find that EXPO can learn effectively even without retaining the offline dataset after pre-training.
A.2 ANALYSIS ON HYPERPARAMETERS
D200
EXOON
EXTO
D20
D20 BH02
Figure 9: Ablation over the number of samples N in the OTF policy. We find that the number of samples can affect performance to an extent.
Ablation over the scale of action edits. We find that the scale of actions edits affect perfor- mance, and is important to tune this hyperparameter for each task.
Figure 10:
We ablate over the important hyperparameters of EXPO, namely the number of samples N in the OTF policy for both sampling and TD backup and the scale of the edit policy B, to analyze the effect of these hyperparameters on performance. We present the results Lift and Square in Figure and Figure 10. We find that the performance of EXPO can be affected by both the value of N and the scale of the edit policy, where choosing the most optimal values can result in better performance. Importantly, the scale of the edit policy is a hyperparameter that is important to tune for each task. This because smaller edit scale will make the edit policy edit actions closer the base actions, which is desirable if the optimal actions are close to the base actions, while a larger edit scale allows the edits to differ more, which may be better for exploration. We provide tuning strategies in Appendix Section
B EXPERIMENT DETAILS
Hyperparameters. Hyperparameters we used for EXPO can be found in Table 1. Each training run presented is with three seeds and error bars indicating max and min. For offline-to-onlnee training, we present the number of pretraining steps for each suite. We not pretrain the online setting. We use the same residual block structure for the basc policy IDQL (Hansen-Estruch et al., 2023).
For our experiments, we find that EXPO generally works well across a fix set of hyperparameters and we only tune the edit policy 3 from [0.05, 0.1, 0.3, 0.7]. In terms of practical hyperparameter recommendations, we recommend a smaller value offline dataset, and larger value of (e.g., 0.5, 0.7) start for tasks where it is more important to explore to find the optimal strategy. While we do not extensively tune the number of action samples N, note that higher number of N might work better for higher dimensional action spaces.
(e.g., 0.05 or 0.1) to start for tasks with a good
14

---

Published as a conference paper at ICLR 2026
Hyperparameter
Robomimic
Adroit
Antmaze
Mimicgen
Optimizer
Adam
Batch Size
256
Learning Rate
3e-4
Discount Factor
0.99
Target Network Update
0.005
Q-Ensemble Size
N Action Samples
8
UTD Ratio
20
Num Min Q
2
T
Beta Schedule
Variance Preserving
Base Policy MLP Hidden Dim
256
Base Policy Num Residual Blocks
3
Edit Policy MLP Hidden Dim
256
Edit Policy MLP Hidden Layers
3
Pretraining Steps
200k
20k
500k
200k
Edit Policy Dropout Edit Policy 3 Online
None
0.1
None
None
0.05
0.7
0.05
0.05
Edit Policy 3 Offline-to-Online
0.1
0.7
0.05
0.05
Table I: Hyperparameters for EXPO.
Dataset. We list the details of the dataset used to pretrain (offline-to-online) and initialize (online) for the Robomimic and Mimicgen environments in Table 2. We subsample 10 trajectories for Lift and use the MH dataset for Can to make the tasks harder. The Adroit and Antmaze environments use the default D4RL provided datasets.
Hyperparameter
Num Data
Composition
MimicGen Stack
200
10 human and 190 generated by MimicGen
MimicGen Threading
50
10 human and 40 generated by MimicGen
Robomimic Lift
10
PH PH MH
Robomimic Square
200
Robomimic Can
300
Table Dataset details for Robomimic and MiemicGen environments.
Evaluation. Evaluation is performed every 5k steps with 100 episodes for the Adroit and Antmaze environments and every 10k steps with 50 episodes for Robomimic and MimicGen environments. The curves smoothed with running average with window size 3 or 5 depending the task for all algorithms. The shaded region is not smoothed, so the curves may appear outside the shaded regions. For the Adroit environments, normalized return is calculated the percentage of the total timesteps the task is considered solved. This is the same metric as RLPD (Ball et al., 2023), All tasks use sparse binary reward indicating whether the task has been completed successfully or not.
C BASELINES
IDQL (Hansen-Estruch) et al., 2023). IDQL similarly features training expressive diffusion policy via imitation learning and sampling multiple actions and selecting the one that maximizes the Q-value. However, the crucial differences are: (1) IDQL only uses the implicit policy for online exploration and use implicit Q-learning loss function for the TD backup (Kostrikov et al., 2021), (2) IDQL selects actions from action candidates directly sampled from the imitation learning policy.
RLPD (Ball et al., 2023). RLPD is highly sample efficient algorithm that leverages prior data and oversamples from it for learning. RLPD simpler Gaussian policy and has been shown to be
15

---

Published as a conference paper at ICLR 2026
Lift
Can
Square
Stack
Threading
Antmaze (medium, large)
Pen
Door
Relocate
Figure 11: Visualizations of 12 sparse-reward environments we evaluate on. Note that Antmaze medium and Antmaze large both have two dataset variants.
better in performance compared to many offline-to-online methods even without pretraining. For both evaluation settings, we run RLPD without offline pre-training.
DAC (Fang et al., 2024). DAC is an offline RL method that uses expressive diffusion policy. DAC includes action gradient of the Q-function as part of the diffusion loss to guide its denoising process towards generating more optimal actions. We adapt this method to the oflline-to-ollne RL setting by first pre-training it with the offline RL and the continue to fine-tune online with the same objective.
Cal-QL (Nakamoto et al., 2023) (Ofttine-to-Online only). Cal-QL is standard offline-to-onie to-oinne RL baseline that does not use an expressive policy. Instead, Cal-QL calibrates the Q-function with Monte-Carlo returns way to balance pessimism of offline RL and optimism of online fine-tuning and prevent policy unlearning from offline to online training.
DIPO (Yang et al., 2023b) (Online only). DIPO is online RL method based on diffusion policies. Instead directly maximizing value through the denoising chain, DIPO performs policy improvement by using action gradient of the Q-function to adjust towards
optimal actions.
QSM (Psenka et al., 2023) (Online only). QSM is an online RL method that trains diffusion policies by matching the diffusion loss action gradients. QSM aims avoid instability of value propagation to the expressive policy by incorporating losses to guide the denoising process.
16


Reviews:
[{"review_id": "YtUTzgyyak", "reviewer": "Reviewer_6oJM", "summary": "The paper considers a scenario where one pre-trains an expressive policy on offline data via imitation learning and would like to fine tune it online. Instead of directly modifying the expressive policy—which can be rather difficulty due to architectural complexity and poor gradient propagation—they learn a Gaussian \"edit\" policy which is optimized to produce action residuals that better maximize value over the base, expressive policy. The final policy, denoted the on-the-fly policy, then samples the edit policy multiple times, and chooses the edit which produces the largest action-value. They then update the action-values with an approximate Q-learning update, where the max operation of the bootstrap target is approximated by the aforementioned on-the-fly procedure. Across a wide variety of domains, they evaluate their approach (EXPO) empirically in both online and offline-to-online setups.", "questions": "* In the ablation over action edits, the performance in \"square\" seemed better than many of the baselines. Is this suggesting that the offline-to-online procedure of the baselines is actively making the base policy worse here?\n\n* Can the authors comment on the computational complexity of EXPO relative to the baselines? Running the OTF policy to compute the TD target for every sample in a mini-batch sounds considerably expensive if the action-values have to compute a forward pass for every sampled edit.\n\n* Have the authors considered alternate forms of the OTF policy? e.g., directly performing gradient-ascent in Q(s,.) starting from the base policy's action or base policy + mean edit, or conditional cross-entropy (Lim et al., 2018), etc. There have been various proposed methods in the literature for performing this approximate max operation for continuous-action Q-learning, that—given the interpretations/explanations for the benefit of the OTF policy—I'd like to hear whether there are any thoughts or intuitions around how this might perform within EXPO?\n\n* In some of the plots (e.g., all throughout Figure 3), the curve can be completely outside of the shaded regions. This seems like a bug, if the shaded regions supposedly represent max and min—what's going on here?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The use of an edit policy is elegant in that the approach is agnostic to the form of the base policy (e.g., compatible with non-parametric policies).\n\n* The empirical performance seemed relatively consistent, both in terms of variability and how it faired relative to baseline methods.\n\n* The paper performed a variety of ablations to justify the role of various choices in their algorithm (e.g., whether or not to re-run the on-the-fly policy for the TD target, the role of the edit policy, etc.)", "weaknesses": "* The evaluation only considered 3 seeds. This is reconciled by the breadth of environments and the results from taking it all together, but can limit the significance of individual plots. Further, there doesn't seem to be too much variability in EXPO's performance, perhaps due to an already really good base expressive policy?\n\n* The appendix states that it is presenting the max and min, which measures variability and not statistical confidence. Providing a measure of confidence is crucial for making claims about performance differences between methods here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WQ8MeuBP7z", "reviewer": "Reviewer_53K3", "summary": "This paper proposes Expressive Policy Optimization (EXPO), an online RL algorithm that enables sample-efficient training and finetuning of expressive policies such as diffusion policies. Instead of directly updating the expressive policy to maximize the Q-value, the authors propose to train a separate Gaussian edit policy with RL, while the expressive policy itself is trained via stable imitation learning. In addition, best-of-n sampling is used in both action sampling and TD backup for faster learning. Extensive experiments demonstrate strong performance in online training and finetuning with a given offline dataset.", "questions": "1. Instead of scaling the edit action to lie within the range $[-\\beta, \\beta]$, would a more natural approach such as enforcing a KL-divergence between the edited policy and the base policy also be applicable for keeping the edited action close to the original action samples?\n2. How does the proposed method perform in purely online scenario without a given offline dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is novel to the field, and the analysis is well-supported. By using a lightweight Gaussian policy for policy improvement and using imitation learning to train the expressive policy, the method obtains a good balance between policy expressiveness and training efficiency. \n2. The experiment results are solid and well analyzed. The ablation studies focus on a few of the most important components such as on-the-fly policy extraction and action edits, which demonstrate the effect of the proposed contributions well.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The experiments only cover online reinforcement learning with a given offline dataset, while no experiments are conducted in a purely online RL setting without a pre-collected dataset.\n2. The experiment does not include several strong online diffusion-policy baselines, such as DACER [1], DIPO [2], and DPMD [3], which have been shown to perform better than or comparably to QSM on various tasks.\n\n[1] Wang Y, Wang L, Jiang Y, et al. Diffusion actor-critic with entropy regulator[J]. Advances in Neural Information Processing Systems, 2024, 37: 54183-54204.\n\n[2] Yang L, Huang Z, Lei F, et al. Policy representation via diffusion probability model for reinforcement learning[J]. arXiv preprint arXiv:2305.13122, 2023.\n\n[3] Ma H, Chen T, Wang K, et al. Efficient Online Reinforcement Learning for Diffusion Policy[C]//Forty-second International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xry0m1B1FL", "reviewer": "Reviewer_5nRg", "summary": "The paper introduces Expressive Policy Optimization (EXPO), an approach for fine-tuning large pre-trained policies (like diffusion models) using a lightweight \"edit policy\" layered on top of the base policy. The key idea is to freeze or avoid altering the complex base policy and instead train a small edit policy that makes local adjustments to the base policy’s output actions to maximize rewards. The edit policy is updated with a standard policy-gradient algorithm augmented with entropy regularization (similar to SAC) to encourage exploration, while a critic (Q-function) is learned via a typical off-policy TD loss. Because the base policy is an expressive model (e.g. a diffusion model) for which computing entropy or direct gradients is difficult, the authors approximate the entropy term by sampling multiple actions from the base policy and fitting a simpler surrogate distribution for the edit policy’s updates (adding some engineering overhead but keeping the base policy’s parameters fixed). Overall, this technique defers the reward optimization to the constrained edit policy rather than directly modifying the pre-trained expressive policy, thereby avoiding the unstable gradients that would result from naive end-to-end fine-tuning of the large model. Empirically, EXPO is evaluated in an offline-to-online reinforcement learning setting (starting from offline pre-training and then fine-tuning online), and it shows significantly improved policy performance and stability.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Decoupling the reward optimization from the large expressive policy by introducing a small edit Gaussian policy is a practical idea that mitigates unstable gradients when fine-tuning complex policies. By avoiding direct backpropagation through the diffusion-based base policy and instead using a separate editable layer, the approach maintains stability in training. This design helps prevent the large pre-trained model from diverging due to high-variance gradient updates.", "weaknesses": "The core contribution of EXPO feels incremental, essentially adding a small trainable \"edit head\" on top of a fixed base policy and applying well-known fine-tuning strategies. Similar architectures have appeared in prior work – for example, learning a residual policy to refine the actions of a pre-trained policy is not a new concept. The authors combine standard elements (behavior cloning for the base policy, an actor-critic with entropy regularization for the edit policy, Q-learning with TD loss, etc.), so the method comes across as a straightforward amalgamation of known techniques rather than a fundamentally new algorithm or theoretical insight.\n\nGiven that the idea is not new, a deeper understanding of the idea in diffusion RL would be useful. The authors show that it is difficult to compute entropy in the diffusion setting and propose a solution. Yet this is also new a new trick to me. Besides this, the proposed idea mainly relies on intuitive arguments without rigorous verification. As a result, the contribution may be viewed as more of an engineering solution than a novel research insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eALnaLCluL", "reviewer": "Reviewer_RsxQ", "summary": "The manuscript introduces EXpressive Policy Optimization (EXPO), a reinforcement learning (RL) algorithm designed to enable stable and sample-efficient online fine-tuning of expressive policy classes (e.g., diffusion or flow-matching policies) using both offline data and online\ninteraction. The key innovation is to avoid direct value maximization through the expressive policy, instead employing two parameterized components: (1) a large base expressive policy trained with an imitation learning (IL) objective, and (2) a small Gaussian edit policy that locally adjusts actions toward higher Q-values. These are combined via an on-the-fly (OTF) policy that selects the highest-value action among base and edited samples for both execution and temporal-difference (TD) backup (Sec. 4.2; Eq. (3); Fig. 2). EXPO demonstrates up to 2–3× higher sample efficiency than prior methods on 12 sparse-reward tasks across Antmaze, Adroit, Robomimic, and MimicGen domains (Fig. 3–4; Sec. 5.1–5.4).\n\nThe paper is well-motivated and tackles a clear problem—stability in RL fine-tuning of expressive policy classes—with a simple yet effective hybrid approach combining imitation learning and Q-value editing (Sec. 4.1–4.2; Fig. 1–2). Empirical coverage is broad and\nconvincing, with comprehensive ablations and strong baselines (Sec. 5; Fig. 3–7). The presentation is clear, though several mathematical formulations and assumptions could be better formalized (e.g., edit distance constraint, entropy-regularized variants). Moreover,\ncomputational efficiency and theoretical analysis of stability remain underdeveloped—currently argued qualitatively (Sec. 6). Nonetheless, EXPO offers a promising direction for expressive policy fine-tuning that balances innovation with empirical rigor.", "questions": "Provide theoretical and empirical stability analyses:\n1. Include diagnostics of gradient norms, TD-error variance, or critic loss oscillations comparing EXPO to direct value-maximization baselines (Sec. 4.1–4.2).\n2. Formalize stability claims via convergence theorems or bounded-update assumptions for coupled base/edit policies (No direct evidence found in the manuscript).\n3. Report quantitative ablation of β radius vs. policy divergence to support the “local edit” assumption.\n\nReport computational cost and scaling results:\n1. Add per-step runtime, GPU hours, or FLOPs for EXPO vs. IDQL/RLPD across domains (Sec. 5.1–5.2).\n2. Analyze the impact of N (number of action samples) on both performance and compute (App. B Table 1).\n3. Explore efficient approximations (e.g., importance-weighted sampling or learned Qsampler) to mitigate the OTF sampling cost (Sec. 6).\n\nClarify mathematical notation and definitions:\n1. Introduce a global notation table clarifying β (edit radius vs. temperature), πOTFentropy definitions, and vector norm conventions (Sec. 4).\n2. Explicitly specify whether constraints are applied element-wise or via ℓ₂-norm bounds (Eq. (1); Sec. 4.1).\n3. Expand Sec. 4.3 to show derivation of Eq. (5) from SAC objective, ensuring reproducibility.\n\nBroaden evaluation and generalization claims:\n1. Include at least one non-robotic or discrete-action task (e.g., Atari or tabular MDP) to substantiate generality beyond continuous control (Sec. 5).\n2. Demonstrate compatibility with another expressive policy (e.g., flow-matching or autoregressive transformer) to validate policy-agnostic claims.\n3. Provide limited visual or textual policy diagnostics (e.g., action entropy or diversity plots) for interpretability.\n\nQuantify assumption and hyperparameter robustness:\n1. Define empirical metrics for “reasonable prior” dataset quality (e.g., imitation policy success rate threshold) and report how EXPO behaves below it (Sec. 5.5; Fig. 7).\n2. Conduct systematic sensitivity analysis on β and N, plotting performance variance vs. hyperparameter scaling (App. B Table 1).\n3. Offer practical default ranges and scaling rules-of-thumb for new user", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Clear motivation and problem framing:\n1. The paper explicitly identifies the gradient instability of expressive policy classes (diffusion/flow policies) when directly optimizing Q-values (Sec. 1; p. 2–3). Thisconnects to recent findings on denoising-step chains (Ding & Jin 2024; Park et al.2025), establishing a solid motivation.\n2. The problem is formalized as stable value maximization given a pre-trained expressive policy and dataset (Sec. 3; Eq. (1)), aligning well with practical robotic fine-tuning setups.\n3. This grounding strengthens technical soundness and real-world relevance.\n\nElegant decomposition of learning objectives:\n1. The split between base (IL-trained) and edit (Q-optimized) policies isolates instability sources and allows targeted optimization (Sec. 4.1; Eq. (2); Fig. 2).\n2. The OTF mechanism (Sec. 4.2; Eq. (3)) performs implicit value maximization without direct gradient propagation through the expressive base, offering conceptual simplicity and implementation clarity.\n3. This design contributes to both training stability and policy generality, as EXPO is agnostic to the expressive policy parameterization (Sec. 4.4).\n\nComprehensive and reproducible experimental validation:\n1. Twelve challenging continuous-control tasks across four domains (Sec. 5.1; Fig. 3–4) provide strong empirical grounding.\n2. Baselines include IDQL, Cal-QL, RLPD, and DAC—covering both expressive and nonexpressive methods (Sec. 5.2; App. C).\n3. EXPO consistently achieves higher sample efficiency and avoids offline-to-online degradation (Fig. 4; Sec. 5.4), directly supporting the main claim.\n4. Reproducibility is strengthened by hyperparameter tables (App. B; Table 1), dataset details (App. B; Table 2), and explicit evaluation protocols (App. B, p. 15).\n\nInsightful ablations and diagnostics: \n1. Ablations quantify contributions of on-the-fly TD backup (Fig. 5), edit policy (Fig. 6), and entropy backup (Sec. 5.5; Fig. 7), illustrating component necessity and interpretability.\n2. Correlation between offline dataset quality and fine-tuning performance (Fig. 7) provides useful empirical evidence for practitioners.\n3. The study in Appendix A demonstrates robustness even without retaining offline data, reinforcing adaptability.\n\nClarity and connection to prior work:\n1. Related works are extensive and well-situated (Sec. 2; p. 2–4).\n2. The manuscript differentiates from diffusion-based fine-tuning (DAC, QSM) and residual-policy methods (Ankile et al. 2024), improving conceptual clarity.\n3. Figures 1–2 concisely visualize the EXPO workflow, enhancing accessibility.", "weaknesses": "Limited theoretical justification for stability and convergence:\n1. The claim that separating imitation and Q-optimization yields “stable value maximization” (Sec. 4; 6) is intuitively argued but lacks formal analysis or empirical stability metrics (e.g., gradient variance, TD error oscillations).\n2. No theoretical bound or convergence guarantee is provided for the coupled base-edit policy updates (Eq. (2)–(5)).\n3. The assumption that local Gaussian edits suffice to capture high-value modes is unproven; sensitivity to β (edit radius) is only empirically illustrated (App. B; Table 1).\n4. This limits the technical depth relative to recent theoretical RL papers.\n\nComputational efficiency and scaling not analyzed:\n1. The OTF policy requires sampling and Q-evaluation for multiple action candidates per step (Eq. (3); Sec. 4.2), which increases computational cost; yet wall-clock or FLOP comparisons are not reported (No direct evidence found in the manuscript).\n2. The discussion acknowledges sampling overhead (Sec. 6 p. 9) but does not quantify trade-offs or propose scaling solutions.\n3. Without resource profiling, claims of “sample efficiency” may not directly translate to runtime efficiency.\n\nMathematical clarity and notation consistency:\n1. The use of “β” for both edit magnitude and softmax scaling (Sec. 4.1 vs. 4.3) can cause confusion; explicit notation conventions are missing.\n2. The edit constraint “â ∈ [−β, β]” (Eq. (1); Sec. 4.1) lacks formal definition for vectorvalued actions—element-wise, norm-bounded, or learned scaling?\n3. Entropy backup formulations (Eq. (4)–(5)) omit proofs of equivalence to standard SAC objectives under mixed sampling distributions.\nThese minor ambiguities hinder full reproducibility.\n\nEmpirical scope and generalization:\n1. Experiments focus exclusively on robotics control benchmarks; no results on nonrobotic or discrete-action tasks (Sec. 5).\n2. Although the method is claimed to be “agnostic to policy parameterization” (Sec. 6), no experiments validate this with non-diffusion expressive models.\n3. Generalization to large observation spaces (e.g., visual RL) or transfer scenarios remains unexplored.\n\nAssumption dependence and hyperparameter sensitivity\n1. Performance depends heavily on the presence of “reasonable priors” via offline datasets (Sec. 6; p. 9), but quantitative thresholds for dataset adequacy are not defined.\n2. The β and N hyperparameters control exploration and action sampling; only limited tuning guidelines are provided (App. B).\n3. Lack of systematic sensitivity analysis may limit practical deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)