What Matters for Batch Online Reinforcement Learning in Robotics?
Authors: Perry Dong
Suvir Mirchandani
Dorsa Sadigh
Chelsea Finn
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
autonomous improvement
imitation leaning
Primary area: reinforcement learning

Abstract: The ability to learn from large batches of autonomously collected data for policy improvement---a paradigm we refer to as batch online reinforcement learning---holds the promise of enabling truly scalable robot learning by significantly reducing the need for human effort of data collection while getting benefits from self-improvement. Yet, despite the promise of this paradigm, it remains challenging to achieve due to algorithms not being able to learn effectively from the autonomous data. For example, prior works have applied imitation learning and filtered imitation learning methods to the batch online RL problem, but these algorithms often fail to efficiently improve from the autonomously collected data or converge quickly to a suboptimal point. This raises the question of what matters for effective batch online reinforcement learning in robotics. Motivated by this question, we perform a systematic empirical study of three axes---(i) algorithm class, (ii) policy extraction methods, and (iii) policy expressivity---and analyze how these axes affect performance and scaling with the amount of autonomously collected data. Through our analysis, we make several observations. First, we observe that the use of Q-functions to guide batch online RL significantly improves performance over imitation-based methods. Building on this, we show that an implicit method of policy extraction---via choosing the best action in the distribution of the policy---is necessary over traditional explicit policy extraction methods from offline RL. Next, we show that an expressive policy class is preferred over less expressive policy classes. Based on this analysis, we propose a general recipe for effective batch online RL. We then show a simple addition to the recipe, namely using temporally-correlated noise to obtain more diversity, results in further performance gains. Our recipe obtains significantly better performance and scaling compared to prior methods.

Published as a conference paper at ICLR 2026
WHAT MATTERS FOR BATCH ONLINE REINFORCEMENT LEARNING IN ROBOTICS?
Perry Dong, Suvir Mirchandani, Dorsa Sadigh, Chelsea Finn Stanford University
ABSTRACT
The ability to learn from large batches of autonomously collected data for pol- icy improvement-a paradigm we refer to as batch online reinforcement learn- ing ing-holds the promise of enabling truly scalable robot learning by significantly reducing the need for human effort of data collection while getting benefits from seff-improvement. Yet, despite the promise of this paradigm, it remains challenging to achieve due to algorithms not being able to learn effectively from the autonomous data. For example, prior works have applied imitation learning and filtered im- itation learning methods to the batch online RL problem, but these algorithms often fail to efficiently improve from the autonomously collected data or converge quickly to suboptimal point. This raises the question of what matters for effective batch online reinforcement learning robotics. Motivated by this question, we perform a systematic empirical study of three axes-(i) algorithm class, (ii) policy extraction methods, and (iii) policy expressivity- and analyze how these axes affect performance and scaling with the amount of autonomously collected data. Through our analysis, we make several observations. First, we observe that the use of Q-functions to guide batch online RL significantly improves performance over imitation based methods. Building on this, we show that an implicit method of pol- icy extraction via choosing the best action in the distribution of the policy is nec- essary over traditional explicit policy extraction methods from offline RI. Next, we show that expressive policy class is preferred over less expressive policy classes. Based on this analysis, we propose general recipe for effective batch online RL. We then show simple addition to the recipe, namely using temporally-correlnn noise to obtain more diversity, results in further performance gains. Our recipe obtains significantly better performance and scaling compared to prior methods.
INTRODUCTION
The success of modern deep learning has hinged on the ability of learning methods to leverage vast amounts of data. In robotics, although recent works have focused on mitigating this gap by proposing large robotic datasets (Open X-Embodiment Collaboration, 2024; Khazatsky et al., 2024), robot learning continues to operate under a substantially smaller data regime than other fields due the amount of human supervision required for data acquisition. Even in relatively large data regimes, policies based on imitation learning often struggle to achieve reliable performance. Instead or in addition to manually collecting robotic data, a desirable alternative is to learn policies that can self-improve. Evidence in both robotics and language modeling have suggested that self self-improvement via reinforcement learning (RL) yield substantial performance gains. However, online RL settings as they have been traditionally studied in robotics can be impractical to scale up to deployment scenarios, they typically involve updating the policy frequently during policy execution make use of deployment data. Scaling up this paradigm would involve updating large models in the loop of execution and handling potentially unstable or unsafe behaviors as the policy is learning, both of which are challenging practical constraints. Instead, another setting we can consider is to iteratively perform offline improvement with batches of online deployment data. We refer to this batch online RL, training paradigm in which policies generate large batches of rollouts that are then used to iteratively refine those same policies. This paradigm has the potential to enable truly scalable robot
Videos available httts::/pp-perrgee pttpd/ddpprrttuuuuubub wtw.ggitth.....cccceeeeee ww/batchhooonll.. online-rl/

---

Published as a conference paper at ICLR 2026
learning by significantly reducing the need for human effort of data collection due to benefits from improvenpoee
Learning from autonomously collected data for policy improvement, however, remains significant challenge in robot learning as current algorithmss struggle to fully leverage this autonomous data (Mirchandani et al., 2024). Prior methods have focused on tackling this through either imitation learning (IL) or filtered-IL. methods (Liu et al., 2023; Ahn et al., 2024; Bousmalis et al., 2023). Despite their intuitive appeal, these approaches have yielded suboptimal results. IL methods have inherent limitations in their ability to leverage suboptimal demonstrations within autonomously collected datasets, while methods based on weighted or filtered IL often have diminishing returns and do not scale well with increasing amounts of autonomous data (Mirchandani et al., 2024). This raises the question: what are the necessary components enable batch online RI.? We posit that effective use of autonomously collected data requires policies that not only can collect diverse trajectories but also can learn from this diversity.
In this work, we perform a systematic empirical study to investigate what enables effective batch online RL in robotics with the goal of providing a general recipe to tackle this problem. We break down the key components of batch online RL approaches into several axes-(i) algorithm class, (ii) policy extraction methods, and (iii) policy expressivity- and analyze how each of them affects performance. The first axis we analyze is the algorithm class. Prior approaches to the batch online RL problem in roboties often focus on II. or filtered-II. methods as approaches that are easy to carry out. We compare these an algorithm class that can more effectively benefit from diverse suboptimal date-value RL. We observe that utilizing a Q-function- trained on the cumulatively collected data-1o guide the policy enables it leverage the diversity in the autonomous data learn new behaviors, thereby overcoming barriers to using autonomous data for self-improvement and scaling sigmificantly better. Among the different methods of using a Q-function, we find that the expressivity of the policy class and policy extraction method are vital choices. Specifically, an expressive policy is necessary to both generate and consume diverse data, and an implicit method of policy extraction, where we choose the best action in the distribution of the policy, significantly outperforms traditional explicit policy extraction methods.
Based these observations, we propose general recipe for effective batch online RL: train an expressive IL policy as the actor, train a Q-function on the autonomous data, and perform implicit policy extraction using the Q-function to obtain policy for autonomous rollouts. On top of the recipe, we propose a simple practical addition to induce even more diversity and achieve better sample efficiency: applying a small amount of temporally correlated noise modeled by the Ornstein-Uhlenbeck process during autonomous rollouts. Overall, our recipe results in up to 2x performance improvement over previous methods
set of six complex robotic manipulation tasks from Robomimic (Mandlekar et al., 2021). Adroit (Rajeswaran et al., 2018), and MimicGen (Mandlekar et al., 2023). Finally, we validate the practicality of the recipe on challenging real-world robotics task, improving over the initial policy by 30% in success rate in three iterations of batch online RL.
RELATED WORK
Autonomous Improvement. There has been growing interest in the robotics community in au- tonomous improvement-where an initial set of demonstrations is used to collect autonomous data which can be leveraged for learning better policies. Several works (Liu et al., 2023; Ahn et al., 2024; Bousmalis al., 2023; Mirchandani et al., 2024; Zhou et al., 2024) have attempted to use IL or filtered-IL methods as the algorithm for improvement. However, policy improvement can saturate quickly (Mirchandani et al., 2024). This likely due to the fact that the methods do not effectively leverage suboptimal or failure data, and can lack diversity in the autonomous data that they collect. Nevertheless, these methods can be appealing because of their simplicity and because they avoid some of the practical challenges of running real-world online RL. Zhou et al. (2024) provides system for autonomous improving policies. However, they operate in a goal-conditioned multi-task setting. Nakamoto et al. (2024) explores improving robot foundation models without training the models themselves, but training a value function for guidance; our recipe outperforms this approach in our real robot experiments.
2

---

Published as a conference paper at ICLR 2026
Dataset
Batch Online Data Collection
N episodes
Key Method Choices:
@6s. a)
argmax Q
Expressive policy class
Q-function
Offline
Implicit policy extraction
Policy
Training
Figure 1: Overview. We consider the batch online RL problem setting, in which a policy is trained on an initial dataset, used to collect batches of autonomous data during deployment, and then re- e-tarined on the accumulated dataset. We analyze three critical axes spectrum of approaches the batch online RL problem: policy expressivity, algorithm class, and policy extraction method. We propose general effective recipe of training an expressive IL policy as the actor, value-based RL to learn Q-function, and performing implicit policy extraction with the Q-function to get policy for rollouts.
Finetuning Offline RL. While running online RL methods from scratch can be prohibitively expensive, it possible accelerate the process by using prior demonstrations. There are several techniques for doing so, such as including the offline data in the replay buffer (Vecerík et al., 2017; Hester et al., 2018; Song et al., 2023; Ball et al., 2023) and regularizing the policy with behavior cloning loss (Rajeswaran et al., 2018) 8). Additionally, it is possible to use offline RL (Peng et al., 2019; Kumar et al., 2020) to initialize a policy and value function, and then improve these policies with an online fine-tuning procedure (Nakamoto et al., 2023; Nair et al., 2021; Kostrikov et al., 2022; Lyu et al., 2022). However, online fine-tuning can be challenging (Nakamoto et al., 2023; Lee et al., 2021) due distribution shifts and catastrophic forgetting of the initialization. We operate in different setting, where large batches of deployment data are collected autonomously using a fixed policy, and then that policy is iteratively refined offline based the deployment data. This setting has the advantage of decoupling training from data collection during policy deployment.
3 PRELIMINARIES
In this section, we describe the batch online RL problem setting, as well as ingredients from prior approaches that are relevant our empirical study Section 4.
Batch Online RL. Robotics operates under substantially smaller data regime than other fields due to the difficulty in obtaining data. Instead of or in addition to manually collecting robotic data, desirable alternative is to enable robots to self-improve While online RL methods this problem in theory, running online RL in the real-world is practically challenging because of the need train during deployment. The challenges arise from the need to update the policy in the loop of execution, which can be challenging as models scale, and also from the fact that the policy may undergo unstable unsafe behaviors as the policy is learning. A middle ground is instead to iteratively perform offline self-improvement with batches of online collected data. Various forms of this have appeared in prior works (Matsushima al., 2020; Riedmiller et al., 2022). We refer to this setting as batch online RL.
address
We formulate batch online RL in the context of a Markov Decision Process (MDP) M (S, A,,A,,,, 7.P. Mo) (Bellman, 1957). S denotes the state space and A denotes the action space. r(s, a) the reward function mapping from state and action pairs to rewards, μ4 is initial state distribution, and p(s'|s, a) denotes the transition dynamics. As in traditional RL, the objective is to find policy that maximizes the expected sum of discounted rewards Erep (+) |E of (r(,,, at)] where p" gives the likelihood of trajectory under n.
Unlike traditional RL, the policy π is frozen during a given deployment. It may, however, be updated offline after each iteration of collecting batch of data. We provide general framework for the batch online RL setting in Algorithm I. An initial policy To is trained from an offline dataset DØ- Then, for each iteration i, the policy -;-1 is used to collect rollouts D,, which is appended to the original dataset
3

---

Published as a conference paper at ICLR 2026
and trained to obtain g.. Depending the specific approach, a Q- Q-function may also be trained at each iteration and used to guide rollouts. This is repeated for N iterations to obtain the final policy.
Having introduced the problem setting, ingredients of existing works and problem settings that are relevant to our empirical analysis of batch online RL.
discuss key
Do + Collect offline dataset
Qo f (pdateValue(Dg)
To < UpdatePolicy((D))
for i in ....N do
D: < Collect M rollouts with (001out((T--1.---11)
Imitation Learning. Several works (Liu et al., 2023; Ahn al., 2024; Bousmalis al., 2023; Mirchandani et al., 2024) have attempted to use imitation learning- ceanning-based methods as an approach the batch online RI. problem. Imitation learning is often formulated as behavior cloning, which uses supervised learning to learn a policy me parameterized by 0 to maximize the log-lik ongliikeihood of actions in dataset D, (() (log (a s)].
Q. UpdateValue(U.DD))
W4 + UpdatePolicyU..D)
end
Algorithm 1: Framework of Batch Online RL
In the batch online RL setting, TT is updated by training on a combination of the base dataset Du and its own online rollouts. In the case of filtered imitation learning, transition samples are reweighted with a weighting of assigned to transitions sampled from successful trajectories and a weighting of O assigned to failures.
Value-based RL. A large body of work in offline RL (Kumar et al., 2020; Kostrikov et al., 2022; Nair al., 2021; Wu et al., 2019) has studied the problem of learning policy from a static offline dataset D. As we discuss in Section 4, we consider applying techniques from offline RL within the UpdateValue and UpdatePolicy step at each iteration of batch online RL. We emphasize the distinction from pure offline RI, where no online transitions are collected, and pure online RI., where training occurs during the rollout procedure.
In this work, we primarily consider the Implicit Q-Learning (IQL) (Kostrikov et al., 2022) value objectives given its effectiveness on a range of tasks. IQL aims to fit a value function by estimating expectiles 7 with respect to actions within the support of the data, and then uses the value function to update the Q-function. To do so, it aims to minimize the following objectives for learning a parameterized Q-function Qo (with target Q-function Qor) and value function Vp:
Lo(ø) (s (r Vo('') Qo(s,a)))
(v(w) B(,,a)D [L5 (8, (z)),
where = 1(x < O)(=². Intuitively, is hyperparameter that controls how much the value function approaches the maximum of the Q-function, with greater making the value function closer to the maximum. The policy can then be recovered from the Q-function and value function via a policy extraction step. As in IQL,, we consider Advantage-Weing 2019) as a canonical example of policy extraction method. AWR aims to maximize
Regression (AWR) (Peng et al.,
J= (0) E (ee.a)=le"(((ee.))
log (a(a)))]
where is hyperparameter to interpolate between behavior cloning and recovering the maximum of the Q-function. In the following section, we consider a spectrum of batch online RL approaches, examining algorithms leveraging IL value-based RL, different policy extraction methods, and policy expressivity.
4 EMPIRICAL ANALYSIS OF BATCH ONLINE RL
We now introduce our analysis setup. Given the general framework of batch online RL as presented in Algorithm 1, we perform systematic empirical study spectrum of approaches to understand key axes that affect performance. Specifically, we analyze the following:
Algorithm class. We perform experiments with imitation learning and filtered-IL objectives as well as balue-based RL objectives for updating the policy (Section 4.1).
2. Policy extraction method. We consider two methods of extracting policies from a value- based RL. method, which we refer to explicit and implicit policy extraction (Section 4.2).
4

---

Published as a conference paper at ICLR 2026
Figure 2: Simulation environments.
Robomimic tasks: Lift, Can, Square; MimicGen tasks:
Threading, Stack; Adroit tasks: Pen.
Nomullie Retur
100
50
Squaree
100
Threading
20
60
65
55
65
85
90
10
30
20
Valne-based RL
Pilbered II.
Figure Normalized returns of different algorithm classes over multiple iterations of improvement. Value-based RL significantly outperforms IL and filtered-IL, Runs are 3 seeds, 100 evaluations with error bars showing standard error.
Policy expressivity. We analyze the effects of policy expressivity, focusing on two policy classes: a Gaussian policy and an expressive liffusion-based policy (Section 4.3).
Experimental Setup. We perform our study in simulation and validate our findings real- robot manipulation task. We choose six challenging continuous control environments in simulation: Lift, Can, and Square from Robomimic (Mandlekar et al., 2021); Stack and Threading from Mimicgen (Mandlekar et al., 2023); and Pen from Adroit (Rajeswaran et al., 2018). We illustrate these environments Figure 2. The size of Do varies from to 100 demonstrations depending on the task difficulty; we choose this size such that the base policy performs at success rate between 30-65%, putting it a realistic scenario that leaves room for improvement. We run N=10 to 20 iterations of batch online RL with M=200 rollouts per iteration.
Based on our results, in Section we present recipe for batch online RI, and demonstrate the practicality of the recipe challenging woal-world robotic task of hanging tape hook.
4.1 WHICH ALGORITHM CLASS WORKS BEST?
We first perform controlled set of experiments identify the extent to which algorithm class affects performance in batch online RL. Three classes of algorithms commonly used in prior works; (a) IL, which is the most straightforward approach for learning from autonomous rollouts, (b) filtered-IL, which filters low-quality rollout trajectories prior to re-training to ensure that the policy only taps into successful demonstrations, and (c) value-based RL which attempts to additionally learn from negative data via learning value function and performing Bellman update.
For all of the algorithm classes, we use a diffusion-barr based policy as the default. For value-based RL, the default extraction method we consider is only using the Q-function for guidance; specifically, during rollouts, multiple actions {a, i........ are sampled from the policy and the best action arg max.; ((s, a;) is selected. Since we use diffusion policy as the base policy, this be instan- tiated as Implicit Diffusion Q-Learning (IDQL) Graasen--sttuude et al., 2023). We examine policy extraction choices individually in more detail in Section 4.2 and policy expressivity in Section 4.3.
Value-basee based RL is necessary for overcoming suboptimal convergence of Filtered-IL. In Figure 3, we present the average normalized returns over iterations of batch online RL for each algorithm class on our six tasks. We observe that value-based RI. methods tend to significantly outperform IL-bassed methods. Vanilla IL performs the worst on all tasks, which is perhaps not surprising as vanilla IL will fit the failure trajectories of the autonomous rollouts. Filtered-II, while exhibiting an initial improvement, often converges quickly to suboptimal performance compared to value-based RL.
5

---

Published as a conference paper at ICLR 2026
Value-based RL generates and consumes more diverse data than imitation-based methods. We hypothesize that the better performance of value-based RL methods is a sult of a better ability to leverage diversity in autonomously collected data. We examine heatmaps for state visitation of trajectories during batch online RL in Figure 4. We see that value-based RL methods result much more diverse tra- jectories after batch online RL. Intuitively, this makes sense because value-based RL methods can use the Q-function determine which states and actions are desirable even in fail- ure trajectories, thus allowing the policy to learn from a more diverse set of trajectories including failures.
IL
Li
Square
Figure 4: Heatmap of state visita- tions successful trajectories af- ter batch online RL for allue-based RI and filtered II. on Lift and Square. 3D plots of end-effector positions are shown for each task. Darker colors correspond to higher density of visita- tion. Note that value-based RL meth- ods achieve more state diversity in successful rollouts.
Value-based RL scales better with larger batches of
autonomous data. An important desiderata for choosing algorithm class for the batch online RL problem setting is the ability scale performance with the amount of collected at each iteration. For each environment, we set M to a small, medium, and large value: 50, 100, and 200 trajectories for Robomimic and Mimicgen, and 100, 200, and 300 for Adroit Pen. We present the results of data scaling in Figure 5. We see that value-based RL achieves the best scaling across the board. In all but one task, value-based RL performs significantly better as the amount data increases, suggesting stronger ability to leverage large batches of data for improvement. This in contrast IL or filtered-IL which tend saturate with more data collected at each iteration.
Value-based RL necessary but not sufficient for batch online RI. One takeaway from this section is that for batch online RL, we cannot get away with just doing IL or filtere--IL many prior works suggest. But if this is the case, why have prior works seen limited benefits of value-based RL in practice (Mirchandani al., 2024; Mandlekar et al., 2021)? We posit that value-basee RL as the algorithm class for updating the policy is necessary but not sufficient. There are other choices that are critical for value-based RI. methods work for batch online RL EL-specifically, policy extraction method (Section 4.2) and policy class (Section 4.3).
Normaliaed Retar
Small
Medium
Filwwed IT.
Valuesbuodd Rt.
Figure 5: Normalized returns of dif- ferent algorithm classes at various data scales averaged across all tasks.
4.2 HOw TO EXTRACT THE POLICY?
Given the advantages of value-based RL compared to IL and filtered-II. methods in the batch online RL setting from Section 4.1, the second axis we consider is how to extract the policy from the value function. We separate policy extraction into two distinct categories, explicit policy extraction and implicit policy extraction, to analyze the effect of extraction method on performance.
(a) Explicit policy extraction. Prior works in offline RL typically carry out policy extraction following the idea of maximizing the Q-value while staying close to the behavior policy. This is done through maximizing a RL objective and an IL objective offline to learn the policy. This approach has the advantage of explicitly learning signals from the Q-function, while still making the policy stay close to the behavior dataset. For our experiments, we select Advantage- Advantage-Weightee Regression (AWR) (Peng et al., 2019) as a canonical example of explicit policy extraction that follows this principle, as described in Section 3.
(b) Implicit policy extraction. In contrast separate policy extraction step that explicitly extracts a policy to maximize Q-values, an alternative for policy extraction is to optimize for the best actions in the distribution of the policy, which refer to implicit policy extraction. One simple approach for carrying this out to select actions online by querying the fuuuction, where multiple actions sampled from the policy and the highest Q-value action is selected. While implicit policy

---

Published as a conference paper at ICLR 2026
Nomwelled Rature
Laft
Can
Square
Threading
Stack
Pen
Average
100
100
Ir
55
50
50
90
90
50
50
50
18
Pre
Pook
Pre
Pook
Pre
Pool
Pie
Post
Pie
Post
Fie
Posk
Fre
Plok
Pxplicit Philay Estmetion
Implici Policy Fstrection
Figure 6: Normalized returns of explicit versus implicit policy extraction. Pre refers the initial base policy To trained on Do and Post refers to the policy after completing iterations of batch online RL. Returns are averaged over 3 seeds and 100 evaluation trials each iteration with error bars showing standard error.
extraction loses potentially useful signals from the Q-function for the policy, it has the advantage of disentangling the value function and policy training, which provides more stable learning.
Implicit policy extraction significantly outperforms explicit policy extraction in batch online RL. Figure shows the average normalized returns before (Pre) and after (Post) running batch online RL with explicit versus implicit policy extraction. The number of iterations varies for each task; refer to Figure 3 for details on iterations. Interestingly, we find that although explicit policy extraction achieves stronger initial performance in nearly every benchmark task, implicit policy extraction performs significantly better after running batch online RL. fact, explicit policy extraction does not improve the policy performance on any of the tasks. This result indicates that the policy extraction approach is critical in batch online RL, likely due to implicit policy extraction leveraging diverse autonomous data more effectively: while explicit policy extraction approaches provide more signals from the value func- tion for policy learning, this signal becomes detrimental when new, diverse data is added during batch online RL and causes shift in action distribution. The policy extracted from explicit policy extraction cannot adjust to this shift as well as implicit policy extraction, resulting in subpar performance.
4.3 DOES THE EXPRESSIVITY OF THE POLICY MATTER?
From the preceding analysis, we observe that value-based RL outperforms IL and filtered-II II. in batch online RL settings. One natural question is whether value-based RL with implicit policy extraction is sufficient, or whether the choice of policy class also matters for performance. The third axis we ana- lyze is expressivity of the policy class. We focus on two classes: Gaussian policy and an expressive diffusion- based policy. Both use supervised IL objectives and the value function objectives from IQL.
(a) Gaussian policy. Gaussian policies model the mean and variance of (als) and sample actions from the learned mean during rollouts. Though they are a less expressive class of policies, Gaussian policy are still worth examining because they fast for inference, which is especially desirable in real-world tasks. On top of that, offline RL methods such as ReBRAC (Tarasov et al., 2023) based on Gaussian policies have shown performance on par to diffusion based methods, and current online RL methods based on Gaussian policies have been successful even in challenging real-world tasks (He al., 2025; Lin al., 2024; Yin al., 2024; Lin et al., 2025), suggesting perhaps less expressive policies can be just as effective for malue-basee RL.
(b) Expressive diffusion policy. Diffusion-based policies are highly expressive policy class which use a Markovian noising and denoising process to model the behavior distribution of the data. This allows them to better model multimodal action distributions. A further advantage of diffusion- policies is that they lend themselves well to implicit policy extraction approaches as expressive policy classes can capture more diverse distribution more effectively.
Expressive diffusion policies with implicit policy extraction outperform Gaussian policies with explicit policy extraction. In Figure 7, we compare the performance of expressive policies with Gaussian policies before and after running multiple iterations of batch online RL. For the expressive policy class, we use implicit policy extraction as analyzed in Section 4.2. For Gaussian policies, since explicit policy extraction. We find that across all
the action distribution is less expressive, tasks and environments, the former significantly outperforms the latter.
Expressive policies enable better implicit policy extraction. To control for the advantages of the implicit policy extraction method in batch online RL that we observed in Section 4.2, we additionally run version of the Gaussian policy with implicit policy extraction by sampling actions from the
7

---

Published as a conference paper at ICLR 2026
Nomwelled Roture
Laft
Can
Square
Threading
Stack
Pen
Average
1.00
100
100
at
1001
50
50
90
90
50
50
80
2 Pre
Fre
Posl
Pisk
Pre
Fost
Ptc
Pist
Fie
Fost
Fre
Pos.
Pre
Pook
Ganeiin Censeien((pppic))) (Feplict)
Corowia (implici)
Diffniio (Implicit)
Figure 7: Normalized returns of value-based RL with diffusion versus Gaussian policy before and after improvement. To address confounding of policy extraction methods, we show both explicit and implicit policy extraction approaches for Gaussian policies. Returns are averaged over seeds and 100 evaluation trials at each iteration with error bars showing standard error.
learned mean and variance and using the Q-function for guidance during rollout. Although there is significant improvement over Gaussian policies with explicit policy extraction, the overall performance considerably worse than that of an expressive policy. With the expressive policy, the initial policy can better capture the action distribution which ultimately leads to stronger improvement during batch online RL. Why might less expressive Gaussian policies be sufficient for online RL but not batch online RL? In online RL, the action distribution (Tass) modeled by the policy is always changing as the policy is updated each step. This means there will always be new actions taken, so the policy does not need have good model of the action distribution from the initial dataset to enable improvement. This is in contrast to batch online RI., where to leverage diversity of the online data, the initial model needs to have captured enough of an expert action distribution collect useful trajectories that can then be used to improve the policy.
RECIPE FOR BATCH ONLINE RL
Based on our analysis from Section 4, we propose the following recipe for batch online RL: train an expressive IL policy as the actor, train a Q-function on the autonomous data, and perform implicit policy extraction using the Q-function to obtain a policy to do rollouts. We illustrate the full recipe in Figure 1. In our experiments, we instantiate the recipe with diffusion-basee policy network trained with IL. a O-function trained via the IQL objective, and implicit policy extraction through sampling actions and choosing the one with the highest Q-value during rollouts. This instantiation recovers the IDQL algorithm (Hansen-Estreee recipe defines a category of methods and can be instantiated with other expressive policy classes, implicit policy extraction methods, and Q-function objectives.
et al., 2023) for one iteration of batch online RL, though the
noise. The components in each axis share a common theme of better leveraging diverse data in autonomous collection. We apply simple addition to our recipe to induce more diversity. The idea is to add a small amount of temporally correlated -hitennnUhlenbeck process, the policy actions during the autonomous
Improving diversity with temporally-correlan
noise, modeled as an rollouts. This idea has been successful in online RL previously (Lillicrap et al., 2016) and extends to the batch online RL setting as well. We present the results of returns averaged over all tasks in Figure 8 and refer to Section 4.1 for detailed setup of scaling experiments. We find that the addition of cemporally-correlate improve data scaling because the correlated noise has the effecct of increasing the distribution the policy learns, but this increase in distribution can be naturally achieved with more data. This suggests that sypporally-corrlatte noise can be valuable addition, though our recipe does not hinge on it.
noise enables higher performance at each data level. However, it does not
Real-world robotic manipulation task. To validate the practicality of our proposed recipe, we conduct an experiment with running batch online RL on challenging real-world vision-based robotic manipulation task. The task involves controlling a 7-DoF Franka Research robot to grasp roll of tape and hang it onto a hook. We visualize the initial and final states of the task in Figure 9, We use RGB images and robot proprioceptive state (joint and end-effector positions) as input, and ResNet- 18 (He et al., 2015) the vision backbone. We use wrist camera and one camera mounted on as an external view to stream the RGB images.
We collect 5 initial demonstrations in Du and run N = 3 iterations of batch online RL, each with M = 30 rollouts. We compare our recipe with filtered-II and a steering baseline adapted from (Nakamoto al., 2024), where we train the Q-function on M 90 rollouts (as well as D₂)
8

---

Published as a conference paper at ICLR 2026
Tape
Nomaa
Smull
Medau
Treation
Valucboaed RL
Temporally Geundaaed Notoe
Filerdd E
Stowring
Becipe
Figure 9: Real-world task of hanging tape on hook. The shaded blue area (left) depicts the tape's initial distribution. The right side shows success- ful task completion.
Figure 8: Normalized returns of value-based RL with and without temporally correlated noise at different data scales, averaged over all tasks.
Figure 10: Results for real- world Tape task over itera- tions of batch online RL. Our recipe outperforms filtered-IL and steering with Q-function.
corresponding to the same amount of data our recipe trains on for the last iteration, and use a policy trained on Do to sample actions. Our recipe obtains 30% improvement over the initial policy in just 3 iterations (Figure 10). Filtered-IL does not improve upon the initial performance, likely because the policy was able to capture the action distribution of the Do fully in pre-training. The steering baseline performs the worst, indicating that it is necessary to train the policy on the new rollouts.
6 DISCUSSION
As robotic models become improve models beyond simply collecting more data. Batch online RL provides a paradigm for just that-enabling policies to leverage their own rollouts for self-improvement without the complications of online RL. For practitioners, our analysis offers a clear, practical recipe for executing batch online RL. For researchers, we bring to attention open questions for future work to optimize each component of the recipe further. For example, are techniques from offline RL the most optimal for learning Q-function, or do we need something beyond pessimism specific to batch online RL? the same vein, can we improve implicit policy extraction beyond choosing the actions with the best Q-value? We believe solving these questions will result in significantly better and more capable self-improving robotic models. We will open source the code for the final recipe.
capable, it becomes increasingly important to find ways to
7 LIMITATIONS
In this work, we empirically analyze the key axes that affect performance in batch online RL, demonstrating that the general recipe value-based RI, implicit policy extraction, and an expressive policy class enables effective self-imprus celf-improvement of policies in this setting. Our work presents general recipe on batch online RL, though it does have a number of limitations. First, we focus on robotic tasks with continuous action space for the study. In problems that have discrete action space or if we apply a discretization scheme to the actions, the results might not directly transfer as the Q-function can exhibit different properties, policy extraction is different the Q-function can be used as policy, and the notion of an expressive and non-expressive policy class is not well defined. noise in the rollouts for better sample efficiency. However,
We propose adding comporally-correlate directly adding noise may not be applicable in some deployment settings, though we find empirically that adding a small amount of noise only changes the success rate of the policy marginally. Lastly, in our experiments the initial policy achieves some level of success. How to better start from completely non-successful policy for batch online RL. an interesting direction for future work.
ACKNOWLEDGMENTS
This work is in part supported by AFOSR YIP, ONR grant N00014-22-1-2293, NSF #1941722, and the NSF CAREER award. Chelsea Finn is CIFAR fellow.

---

Published as a conference paper at ICLR 2026
REFERENCES
M. Ahn, D. Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan, K. Hausman, B. Ichter, A. Irpan, N. J. Joshi, R. Julian, S. Kirmani, I. Leal, T. E. Lee, S. Levine, Y. Lu, S. Maddineni, K. Rao, D. Sadigh, P. Sanketi, Sermanet, Q. Vuong, S. Welker, F. Xia, T. Xiao, P. Xu, S. Xu, and Z. Xu. AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents, arXiv, 2024.
Philip J. Ball, Laura M. Smith, lya Kostrikov, and Sergey Levine. Efficient Online Reinforcement Learning with Offlie Data. In International Conference on Machine Learning (ICML), 2023.
Richard Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, edition, 1957.
K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davchev, Zhou, A. Gupta, A. Raju, A. Laurens, C. Fantacci, V. Dalibard, M. Zambelli, M. Martins, R. Pevceviciute, M. Blokz- ijl. M. Denil, N. Batchelor, T. Lampe, E. Parisotto, K. Zoina, Reed, S. Colmenarejo, Scholz, A. Abdolmaleki, O. Groth, J.-B. Regli, O. Sushkov, T. Rothorl, J. E. Chen, Y. Aytar, D. Barker, Ortiz, M. Riedmiller, T. Springenberg, Hadsell, Nori, and N. Heess. RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation. Transactions on Machine Learning Research, 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine, D4RL: Datasets for Deep Reinforcement Learning. arXiv, 2020.
Philippe Hansen-Estruch, llya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies. arXiv, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang. Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi "Jim" Fan, Yuke Zhu, Changliu Liu, and Guanya Shi. ASAP: Aligning Simulation and Real-Worlde Physics for Learning Agile Humanoid Whole-Body Skills. In Robotics: Science and Systems (RSS), 2025.
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Amold, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep Q-learning From Demonstrations. In AAAI Conference on Artificial Intelligence, 2018.
Alexander Khazatsky, Karl Pertsch, et al. DROID: A Large-Scale In-The-Wil Robot Manipulation Dataset. Robotics: Science and Systems (RSS), 2024.
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learnin, In International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems (NeurlPS), 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble. In Conference on Robot Learning, 2021.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Leurning Representations (2016), 2016.
Toru Lin, Zhao-Heng Yin, Haozhi Qi, Pieter Abbecl, and Jitendra Malik. Twisting Lids Off with Two Hands. In Conference on Robot Learning (CoRL), 2024.
Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids. In Conference on Robot Learning (CoRL.), 2025.
10

---

Published as a conference paper at ICLR 2026
H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y. Zhu. Robot Learning the Job: Human-in-ther in-the-Loop Autonomy and Learning During Deployment. In Robotics: Science and Systems (RSS), 2023.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. In Conference on Robot Learning (CoRL), 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations. In Conference on Robot Learning (CoRL), 2023.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv, 2020.
Suvir Mirchandani, Suneel Belkhale, Joey Hejna, Evelyn Choi, Md Sazzad Islam, and Dorsa Sadigh. So You Think You Can Scale Up Autonomous Robot Data Collection? In Conference on Robot Learning (CoRI.), 2024.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. arXiv, 2021.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning, Advances in Neural Information Processing Systems (NeurIPS), 2023.
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance. In Conference on Robot Learning (CoRL), 2024.
Open X-Embodiment Collaboration. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. In International Conference on Robotics and Automation (ICRA), 2024.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-Weightee Regression: Simple and Scalable Off-Policy Reinforcement Learning, arXiv,, 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Robotics: Science and Systems (RSS), 2018.
Martin Riedmiller, Jost Tobias Springenberg, Roland Hafner, and Nicolas Heess. Collect & infer fresh look at data-efficient reinforcement learning. In Conference on Robot Learning (CoRL), 2022.
Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using Both Offline and Online Data Can Make RL efficient. In International Conference Learning Representations (ICLR), 2023.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the Minimalist Approach Offline Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pictquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards. arXiv, 2017.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior Regularized Offline Reinforcement Learning. arXiv, 2019.
Jessica Yin, Haozhi Qi, Jitendra Malik, James Pikul, Mark Yim, and Tess Hellebrekers. Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing. arXiv, 2024.
11

---

Published as a conference paper at ICLR 2026
Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Walke, Oier Mees, and Sergey Levine. Au- tonomous Improvement of Instruction Following Skills via Foundation Models. In Conference on Robot Learning (2024), 2024.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL EXPERIMENTS
Data scaling. In this section, present additional scaling experiments for the scaling portion of Section 4.1 and the temporally correlated noise portion of Section 5. We present the performance of the methods for each task instead of an average over the tasks.
Square
Threading
Avrrage
Narwated
Madiun Larg
Nattn Large
Mocham Large
Seettam Lange
Medken Large
Lange
Valae-boedd EIL.
PerpecstyyCennleen Nelio
Figure 11: Normalized returns of value-based RL compared with IL, filtered-IL, and temporally- correlated noise at different data scales, shown for each task.
From Figure 11, we see that value-based RL scales better with data in nearly every task, while IL-based methods either do not scale or scale in limited capacity. In addition, cemporally-correlate noise outperforms not adding temporally correlated noise for each task and data scale. Temporally correlated noise is especially useful for Adroit Pen, which has been known in the literature to benefit from more exploration.
Value-based RL with more iterations for Square and Stack. Because of compute restrictions, the results reported in the main paper for Robomimic Square and MimicGen Stack were converged for IL-based methods but not for balue-based RL. We report the results for value-based RL run for a longer number of iterations in Figure 12. We see that the difference between IL methods and value-based RL becomes larger with more iterations.
Square
Stack
Normalized Return
63
50
60
30
3
9
12
15
0
12
18
24
30
Beration
Figure 12: Normalized returns of value-based RL for Robomimic Square and MimicGen Stack. Error bars show standard error over 3 seeds.
Value-based RL with more rollouts per iteration. We also report runs for value-based RL with more rollouts per iterations for environments that saturated prematurely. From Figure 13, we see that value-based RL often exceeds premature saturation just with more data in each iteration.
Square
70
Nomaltrrd
57
67
50
30
40
30
Herratoon
Deration
Hecadion
Valua-baesd RL
Valur-tusee Trajectorios
Value-Sarwd Trspertorios
200 Trajectorios
Figure 13: Normalized returns of value-based RL for more rollouts per iteration for Robomimic Can, Square and MimicGen Stack. Error bars show standard error over 3 seeds.
13

---

Published as a conference paper at ICLR 2026
B EXPERIMENT DETAILS
Training Parameters. We set the number of iterations from N=10 20 depending on the en- vironment and M=200 rollouts per iteration. We choose the number of trajectories in the initial dataset such that the base II. policy can get 30 65% normalized returns prior to batch online RI. For temporally correlated Ornsten-UUhlenbeck (OU) noise, we select one 0 and value for each environment suite. For implementation, we use the same residual block structure for the policy as IDQL Maunsen-Estruch al., 2023) for both expressive policy RL and imitation learning. We use simple MLP for Gaussian policies.
Tasks
Parameters Dataset Size
Values
Rohomimic Lift
OU
5
OU i
0.05
Robomimic Can
Dataset Size
OU
OU o
0.05
Robomimic Square
Dataset Size
100
OU の
0.05
MimicGen Stack
Dataset Size
OU o
0.03
MimicGen Threading
Dataset Size
OU o
0.03
Adroit
Dataset
0.1
OU
0.03
Tasks
Batch
Learning Rate
3e-4
IQL Expectile
0.8
Discount
0.99
Number of Sampled Actions
64
Optimizer
Adam
Beta Schedule
Variance
Preserving
Diffusion Steps
100
Diffusion Policy: MLP Hidden Dim
256
Diffusion Policy: Num Residual
3
Blocks
Gaussian Policy: MLP Hidden Dim
256
Gaussian Policy: MLP Hidden
Layers
Table 1: Hyperparameters for each simulation task. The values specified under All Tasks are shared for different tasks.
Data Sources. For each task, the dataset consists of expert trajectories. In Robomimic tasks, we use the Proficient Human dataset provided by Mandlekar et al. (2021). In MimicGen environments, use the dataset provided by the benchmark (Mandlekar et al., 2023). For Adroit, we use the dataset from D4RL (Fu et al., 2020).
14

---

Published as a conference paper at ICLR 2026
Evaluation Protocol. Evaluations performed by rolling out the policy from start states randomly sampled from the default initial state distribution of the task. The rollout length for Lift, Can, and Square is 400; for Stack is 200; for Threading is 400; and for Pen is 100. Results in the main text report normalized return averaged over 3 seeds and 100 evaluation trials each.
C HEAL-WORLD TASK DETAILS
In this section, we provide more information on the real world Tape task our analysis.
Setup Description. The Tape task involves hanging roll of tape Franka Research robot. To successfully complete the task, the robot needs to precisely aim for and grasp the roll of tape and hang area. We illustrate an example initial state, success state, and the initial state distribution in Figure 14. The RL agent sends actions to the robot at 5Hz with maximum episode length of 200 timesteps. The robot obtains visual RGB input from two Intel RealSense D435 cameras, one on the mounted on the end effector and one mounted on the side.
rack by controlling 7-DoF
the hook. The initial distribution
roughly 15 em 15
Sample Initial State
Success State
Initial State Distribution
Figure 14: Scenes showing sample initial and success state and the initial state distribution of the real-world Tape task.
Success Detection. The Tape task contains a success state that must be reached for the rollout to be considered successful, namely having the tape the rack. We use scripted rule detect if this state has been reached and if there is a success. For each environment step, we utilize a color threshold check the color of the pixel above the hook. We manually select the pixel location and verify the error of the success detection is near zero.
Resets. We perform automatic resets of the Tape environment in experiments. For successful rollout, we replay a pre-recorded trajectory to grasp the tape and lift it off the hook. For a failed rollout, detect the location of the tape and execute primitive to lift the tape. In both cases, after lifting the tape, we sample an initial state from the initial state distribution and place the tape at the initial state location for the next episode.
15


Reviews:
[{"review_id": "s3UfHYtpId", "reviewer": "Reviewer_JZ8h", "summary": "This paper investigates algorithmic choices in the batch online RL setting, where a limited number of training and data collection iterations are interleaved. Three aspects are evaluated including value-based methods vs. imitation learning, policy extraction methods and the expressivity of the policy. These are done on simulated robotics environments and a real-world task.", "questions": "- Fig.4 is unclear. It is not entirely obvious that there is greater diversity with the value-based method, particularly on the \"Square\" task. Perhaps it would be better to add a numerical measure of the diversity of states. \nKeeping either the 2d or 3d plots alone might be more clear also.\n\n\n- Is the temporally-extended noise that important? Some papers have shown that independent Gaussian noise can be just as effective [1,2]. \n\n- How does the value-based approach compare to some model-based methods? Line 285 posits that value-based methods outperform the policy-based ones due to utilizing bad trajectories more effectively. This property would seemingly be shared by model-based approaches.\n\n[1] CleanRL implementation of DDPG. \n\n[2] \"Addressing function approximation error in actor-critic methods\" Fujimoto et al.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic---batch online RL---is relevant to current applications of RL. \nThe paper is well-written. The use of color for denoting the sections is a nice touch.\nThis paper does a good job at exploring some important algorithmic considerations for the batch online RL setting and the findings are presented clearly.\n\nSome findings are quite interesting, such as value-based methods outperforming the filtered IL after several iterations of batch training but not initially.", "weaknesses": "My main concern is the lack of certain baselines in the experimental sections.\n\n- What are the previous baseline methods for the benchmarks considered? While I see mulitple comparisons in each section to examine the impact of various choices, I do not see any explicit comparison to prior methods. While achieving the best performance is not strictly necessary, it would be helpful to have some baseline numbers to get a sense of how well the evaluated algorithm is doing.\n\n- In Fig.7, it seems like a natural baseline woudl be to also include the Diffusion policy with the explicit policy updates. Is there a particular reason this was not included?\nThis seems like a major reason why there is more diversity with the value-based method, which is presented as an advantage (line 282). It's not clear that Fig.4 is a fair comparison if the filtered-IL method used a Gaussian policy instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJZcWY2q23", "reviewer": "Reviewer_7bd7", "summary": "This paper investigates the problem of \"batch online reinforcement learning,\" a paradigm where a robotic policy is improved by collecting large batches of autonomous data and then performing offline updates. The authors conduct a systematic empirical study to determine the key components for effective learning in this setting. They analyze performance across three axes: (i) algorithm class (imitation learning vs. value-based RL), (ii) policy extraction methods (explicit vs. implicit), and (iii) policy expressivity (Gaussian vs. diffusion policies). The main findings suggest that value-based methods significantly outperform imitation-based ones, implicit policy extraction is more effective than explicit extraction, and expressive diffusion policies are superior to less expressive Gaussian ones. Based on these results, the paper proposes a general \"recipe\" for effective batch online RL in robotics.", "questions": "*   Could the authors provide a stronger justification for why these three specific axes—algorithm class, policy extraction, and policy expressivity—were chosen as the focus of the investigation?\n*   The introduction states that current algorithms \"struggle to fully leverage\" autonomous data. Could you elaborate on the specific failure modes of prior works that motivate this study?\n*   Why was the comparison of algorithm classes limited to IL, filtered-IL, and value-based RL? There are other approaches, such as those that interpolate between an IL and RL loss, that could have served as relevant baselines.\n*   What was the rationale for using a diffusion-based policy as the default across all experiments?\n*   For the filtered-IL baseline, what was the specific threshold or criterion used to filter \"low-quality\" trajectories?\n*   In the experimental setup, the initial datasets are chosen to yield a base policy with a 30-65% success rate. How can we be sure that the conclusions drawn from this \"realistic scenario\" generalize to other starting conditions (e.g., starting from a much worse or better policy)?\n*   Regarding Figure 3, the plot lines are difficult to distinguish. Could the visualization be improved for clarity? It would also be helpful for the figure to be self-contained by explicitly mentioning that the value-based method shown is IDQL.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "*   The paper addresses a practical and important problem. The \"batch online RL\" setting, which involves collecting large batches of data for offline updates, is a sensible and scalable approach for real-world robotics, reducing the need for constant human supervision and avoiding the instability of purely online updates.\n*   The work is structured as a controlled study that ablates different components of the learning pipeline (algorithm, extraction, expressivity). This systematic approach helps to isolate the factors that contribute most to performance.", "weaknesses": "*   A major weakness is the lack of motivation for choosing the three specific axes of analysis. The paper repeatedly refers to these axes but never explains why these are the most critical or representative components to study. The selection of only three algorithm classes (IL, filtered-IL, and value-based RL) also feels restrictive and lacks justification, especially when hybrid methods exist.\n*   Several of the key findings seem intuitive or are well-established principles in reinforcement learning. For instance, the conclusion that value-based methods lead to more diverse data and better performance than simple imitation learning is not a surprising result. The paper would be stronger if it more clearly situated these findings in the context of what is already known and highlighted what is genuinely new.\n*   The paper frames its final contribution as a general \"recipe\" for batch online RL, but the supporting experiments are conducted on a handful of simulation tasks and a single real-world task. This may not be sufficient evidence to support such a broad claim. Presenting the findings as a set of recommendations or case studies might be a more accurate and defensible framing. The claim that value-based RL is \"necessary but not sufficient\" is also very strong and may not be fully supported by the evidence.\n*   There appears to be a contradiction between the claims made in the text and the data presented in the figures. For example, the paper states that \"we cannot get away with just doing IL or filtered-IL,\" but the results in Figure 5 suggest that these methods do show performance gains and scale with more data, even if they saturate earlier than value-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sDjXXKHrrF", "reviewer": "Reviewer_MA26", "summary": "This paper empirically studies what matters for batch online reinforcement learning for robotic tasks. In batch online RL, where agent training and data collection are decoupled, the agent alternates between online data collection by rolling out the current policy and offline policy improvement trained on all data collected so far. The authors hypothesize that an effective batch online RL method should be able to collect diverse data throughout the process. They should also be effective in learning from the diverse dataset collected. The authors examine three key components of the method design: (1) IL-based or Value-based RL (IQL), (2) explicit (AWR) or implicit policy extraction, and (3) expressive (Diffusion-based) or non-expressive (Gaussian) policy. The authors confirm that expressive value-based RL with implicit policy extraction is a feasible recipe for batch online RL in robotics, which achieves both high performance and scaling ability on six simulation tasks and one real-world task.", "questions": "Please refer to the questions above __\"Minor Issues\"__ part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-structured and easy to read.\n- The problem setting, batch online RL, is practical and promising. And it makes sense to decompose the problem into the proposed 3 components.", "weaknesses": "__There are missing experimental details I would like to verify further:__\n\n- For experiments in Section 4.1, my understanding is: all three methods train a policy with the DDPM objective. The value-based RL one trains an additional Q function guiding policy rollout only. If this is the case, why does the initial performance of value-based RL differ from the IL baseline when both of them learn from $\\mathcal{D}_0$?\n- In Section 4.2, what is the training objective when the authors apply AWR to a diffusion-based policy with the DDPM objective? Did the authors reweigh the MSE by the advantages, like how IDQL does in their paper (Appendix F.5)?\n- In terms of the real-world task, when applying the Steering baseline, why did the authors not train the Q function in a batch online RL manner? I am aware of the Steering paper’s setting and how it is deployed in this paper, but a reasonable variant should consider iteratively improving the Q function on new online rollouts (e.g., N=3, M=30). This also benefits the understanding of whether the performance bottleneck results from the value learning for online batch RL.\n\n__Some arguments made in the paper were not well-supported:__\n\n- In my opinion, the central claim of this paper is: “policies that not only can collect diverse trajectories but also can learn from this diversity matter for batch online RL” (Lines 064-066). Intuitively, I agree that expressive RL policies with value guidance should satisfy this requirement and thus work well for batch online RL. However, the normalized score is an indirect metric supporting this claim. Like Figure 4, I expected to see more direct quantitative analysis or visualization of the diversity of the collected data in each iteration, along the three proposed axes. For example, although an implicit Gaussian policy performs worse than the recipe, showing it might be able to collect relatively more diverse data for Threading and Square tasks can actually enhance the central claim.\n- The argument made in Lines 346-351 is not supported with empirical evidence. Could the authors confirm whether implicit policy extraction can generate and leverage more diverse data? Could the authors confirm that explicit policy extraction cannot adapt policy to value functions trained on new data? Is the value learning the performance bottleneck, or the policy learning? Is the data collection the bottleneck?\n- In Lines 395-402, could the authors provide direct evidence showing that an expressive policy can capture multimodality in action distribution in the initial iteration? How does the ability of modelling action distribution change across the iterations?\n\n__Despite applying existing methods to a new problem setup, I felt the novelty of this paper is limited.__\n\n- First of all, although I agree that diverse data collection and learning from diverse data are essential for batch online RL, I think the community has been aware of their necessity for offline RL, imitation learning, and online RL. Early prior study have shown their importance in each setup, like [1, 2, 3, 4]. Given that the batch online RL setup could be treated as consecutive offline RL or “low update-to-data ratio” online RL, what are the other unique insights the reader may grasp?\n- For the second aspect of policy extraction, I recall that IDQL found that explicit policy extraction via AWR does not bring benefit to the DDPM-trained diffusion policy.\n\n__Minor Issues:__\n\n- In equation (1), the V function should be parameterized by $\\psi$.\n- In Line 292, it should be “the amount of data collected”.\n- In Line 261, the notation of N is abused.\n- In Lines 147-149, the authors argue that online RL risks of unsafe and unstable behavior. I wonder if in batch online RL, the agent also risks this since it is possible to conduct unsafe and unstable behavior during the data collection stage if the offline training was not sufficient. But I agree that this should happen less than the online RL paradigm.\n- Compared with the network used for diffusion-based policy, the network used for the Gaussian policy in Section 4.3 is smaller (3-layer MLP). Although I do not expect that increasing the model size for the Gaussian policy can significantly improve performance, the authors should mention in the main body this experimental detail.\n\n\n__References:__\n\n[1] Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning\n\n[2] Behavior Transformers: Cloning k modes with one stone\n\n[3] Synthetic Experience Replay\n\n[4] The Primacy Bias in Deep Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxQdXLNfH0", "reviewer": "Reviewer_AUz8", "summary": "The paper studies batch online RL setting for robot manipulation, where a fixed policy collects large batches of autonomous rollouts, then the policy (and optionally a Q-function) is retrained offline before the next deployment. The authors systematically vary three axes: (i) algorithm class (imitation learning or filtered imitation learning versus value-based RL), (ii) policy extraction (explicit versus implicit), and (iii) policy expressivity (Gaussian versus diffusion). They conclude that value-based RL with an expressive policy and implicit action selection, implemented by sampling multiple actions from the policy and choosing the argmax under the learned Q, works best, and that adding small Ornstein-Uhlenbeck noise during rollout yields further gains. Experiments on six simulated manipulation tasks and a real robot tape-on-hook task show strong improvements and better scaling trends than imitation-learning baselines.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well structured, easy to follow, and clearly motivated. Problem setting, assumptions, and goals are stated upfront, which makes the experimental choices and takeaways straightforward to understand.\n\nThe work decomposes batch online RL into three axes (algorithm class, policy extraction, and policy expressivity) and systematically compares design choices within each axis. This comparative study is valuable to the community because it clarifies which ingredients matter in practice, turns ad hoc intuitions into evidence-backed guidance, and provides an actionable recipe that reliably beats imitation learning baselines.", "weaknesses": "1. Limited novelty: The paper’s primary contribution is a careful, systematic comparison of existing ingredients rather than a new objective or algorithm. The proposed recipe largely recombines known components and matches IDQL when run for a single iteration, so the conceptual contribution feels incremental even though the comparative insights are useful.\n\n2. Unclear hyperparameter tuning and fairness: The paper does not clearly explain how hyperparameters for different ingredients were selected, what ranges were explored, or whether tuning effort and budgets were balanced across methods and tasks. This makes it hard to attribute gains to the method rather than choices like K (implicit samples), β or expectile τ, diffusion steps, model capacity, or critic settings. I would suggest documenting the tuning protocol (search space, selection criteria, per-task vs global tuning, seeds)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)