Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
Authors: Zhiyuan Zhou
Andy Peng
Qiyang Li
Sergey Levine
Aviral Kumar
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement learning
fast fine-tuning
Primary area: reinforcement learning
TL;DR: We find that previous RL fine-tuning methods fail because of Q-divergence when no offline data is kept, and propose a new method WSRL that can finetune more efficiently with no data retention.

Abstract: The modern paradigm in machine learning involves pre-training on diverse data, followed by task-specific fine-tuning. In reinforcement learning (RL), this translates to learning via offline RL on a diverse historical dataset, followed by rapid online RL fine-tuning using interaction data. Most RL fine-tuning methods require continued training on offline data for stability and performance. However, this is undesirable because training on diverse offline data is slow and expensive for large datasets, and should, in principle, also limit the performance improvement possible because of constraints or pessimism on offline data. In this paper, we show that retaining offline data is unnecessary as long as we use a properly-designed online RL approach for fine-tuning offline RL initializations. To build this approach, we start by analyzing the role of retaining offline data in online fine-tuning. We find that continued training on offline data is mostly useful for preventing a sudden divergence in the value function at the onset of fine-tuning, caused by a distribution mismatch between the offline data and online rollouts. This divergence typically results in unlearning and forgetting the benefits of offline pre-training. Our approach, Warm-start RL (WSRL), mitigates the catastrophic forgetting of pre-trained initializations using a very simple idea. WSRL employs a warmup phase that seeds the online RL run with a very small number of rollouts from the pre-trained policy to do fast online RL. The data collected during warmup bridges the distribution mismatch, and helps ``recalibrate'' the offline Q-function to the online distribution, allowing us to completely discard offline data without destabilizing the online RL fine-tuning. We show that WSRL is able to fine-tune without retaining any offline data, and is able to learn faster and attains higher performance than existing algorithms irrespective of whether they do or do not retain offline data.

Published as a conference paper at ICLR 2025
EFFICIENT ONLINE REINFORCEMENT LEARNING NEED NOT RETAIN OFFLINE DATA
Zhiyuan Z Andy Peng"), Qiyang Li', Sergey Levine', Aviral Kumar?
UC Berkeley, PCarnegie Mellon University
("Equal Contribution)
ABSTRACT
The modern paradigm in machine learning involves pre- pre-training on diverse data, followed by task-specific fine-tuning. In reinforcement learning (RL), this trans- lates to learning via offiin RL on a diverse historical dataset, followed by rapid online RI. fine-tuning using interaction data. Most RI. fine-tuning methods require continued training on offline data for stability and performance. However, this is undesirable because training on diverse offline data is slow and expensive for large datasets, and should, in principle, also limit the performance improvement possi- ble because of constraints or pessimism on offline data. In this paper, we show that retaining offline data is unnecessary long as use a properly-designee online RL approach for fine-tuning offline RL initializations. To build this approach, we start by analyzing the role of retaining offline data in online fine-tuning. We find that continued training on offline data is mostly useful for preventing a sudden divergence in the value function at the onset of fine-tuning, caused by distri- bution mismatch between the offline data and online rollouts. This divergence typically results in unlearning and forgetting the benefits of offline pre-training. Our approach, Warm-start RL (WSRL), mitigates the catastrophic forgetting of pre-trained initializations using a very simple idea. WSRL employs warmup phase that seeds the online RL run with very small number of rollouts from the pre-trained policy to do fast online RL. The data collected during warmup bridges the distribution mismatch, and helps "recalibrate" the offline Q-function to the online distribution, allowing us to completely discard offline data without destabilizing the online RL fine-tuning. We show that WSRL is able fine-tune without retaining any offline data, and is able to learn faster and attains higher per- formance than existing algorithms irrespective of whether they do do not retain offline data.
INTRODUCTION
The predominant paradigm for learning at scale today involves pre-training models on diverse prior data, and then fine-tuning them narrower domain-speciff data to specialize them to particular downstream tasks (Devlin et al., 2018; Brown et al., 2020; Driess et al., 2023; Radford et al., 2021; Zhai et al., 2023; Touvron et al., 2023; Zhou al., 2024). In the context of learning decision- making policies, this paradigm translates to pre-training on a large amount of previously collected static experience via offline reinforcement learning (RL), followed by fine-tuning these initializa- tions via online RL efficiently. Generally, this fine-tuning is done by continuing training with the very same offline RL algorithm, e.g., pessimistic (Kumar et al., 2020; Cheng et al., 2022) algorithms or algorithms that apply behavioral constraints (Fujimoto and Gu, 2021; Kostrikov et al., 2021), on a mixture of offline data and autonomous online data, with minor modifications the offline RL algorithm itself (Nakamoto et al., 2024).
While this paradigm has led to promising results (Kostrikov et al., 2021; Nakamoto et al., 2024), RL fine-tuning requires continued training on offline data for stability and performance ((Zhang et al., 2023; 2024); Section 3), as opposed to the standard practice in machine learning. Retaining offline data is problematic for several reasons. First, as offline datasets grow in size and diversity, continued online training on offline data becomes inefficient and expensive, and such computa- tion requirements may even deter practitioners from using online RL for fine-tuning, Second, the need for retaining offline data perhaps defeats the point of offline RL pre- training altogether: recent results (Song al., 2023), corroborated by our experiments in Section 3, indicate that current fine-

---

Published as a conference paper at ICLR 2025
No Data Retention
Fine- Tuning Paradigm
No Data Retention
With Data Retention
Offline Pre-training
Quick Adaptation
WSRL (Ours) Hybrid RL
Return
D Donlime
Offline RL Finetuning
Initialize w/ Praine
Qpre, Pre)
Forgetting
Step
Interact
Offline
Online Fine-Tuning
No data retention fine-tuning focuses on RL fine-tuning without using the offline dataset during online updates, mirroring the common paradigm in machine learning at scale today. The offline dataset is only used pre-train a policy and Q-function via offline RL initialize fine-tuning, after which the dataset is discarded and the agent only fine-tunes with online experience. Current methods struggle in this "no-retention" setting and forget knowledge learned from pre pre-training. Our goal is to develop a fine-tuning method that quickly adapts online even if we do not retain offline data.
Figure I:
tuning approaches are not able to make good use of several strong offline RL value and/or policy initializations, as shown by the superior performance of running online RL from scratch with of- fline data put in the replay buffer (Ball et al., 2023). These problems put the efficacy of current RL. fine-tuning approaches into question.
In this paper, we aim to understand and address the aforementioned shortcomings of current online fine-tuning methods and build an online RL approach that does not retain offline data. To develop our approach, we first empirically analyze the importance of retaining offline data in current online RL fine-tuning algorithms. We find that for both pessimistic (e.g., CQL (Kumar al., 2020)) and be- havioral constraint (e.g., IQL (Kostrikov et al., 2021)) algorithms, the offline Q-function undergoes "recalibration" phase at the onset of online fine-tuning where its values change substantially. This recalibration phase can lead to unlearning and even complete forgetting of the offline initialization. This manifests in the form of divergent value functions when no offline data is present for training. Even methods specifically designed for fine-tuning (e.g. CalQL. (Nakamoto et al., 2024)) still suffer from this problem with limited or no offline data. This extends the observation of Nakamoto et al. (2024) about unlearning in the standard offline-to-onligg fine-tuning setting in that it demonstrates that forgetting and unlearning pose a more severe challenge in no data retention fine-tuning.
Is it possible to fine-tune from offline RL value and policy initializations, but without retaining offline data and not forget the pre-training? Our key insight is that seeding online fine-tuning with even small amount of appropriately collected data that "simulates" offline data retention, but more in distribution to the online finettunngg task, can greatly facilitate recalibration, mitigating the distribution mismatch between pre-training and fine-tuning and preventing forgetting. Once this recalibration is over, we can run the most effective online RL approach (without pessimism or be- havioral constraints) for most sample-efficient online learning. Our approach, WSRL (Warm Start Reinforcement Learning), instantiates this idea by incorporating a warmup phase to initialize the online replay buffer with a small number of online rollouts from the pre-trained policy, and then running the best online RL method with various offline RL initializations to fine-tune. WSRL is able to learn faster and attains higher asymptotic performance than existing algorithms irrespective of whether they retain offline data or not. This approach is not a particularly novel or clever algo- rithm, but it perhaps one the more natural approaches to enable effective fine-tuning of offline initializations.
Our main contribution in this paper is the study of RL online fine-tuning with no offline data re- tention, a paradigm we call no-retention fine-tuning. We provide a detailed analysis of existing offline-to-online RL methods and find that offline data is often needed during fine- fine-tuning to mitigate the valuee divergence and the resulting forgetting due to distribution shift, but also slows down fine-tuning asymptotically. We demonstrate that a simple method of incorporating a warmup phase to initialize the replay buffer with small number of transitions from the pre-trained offline RL. pol-
2

---

Published as a conference paper at ICLR 2025
icy followed by running simple online RL algorithm is effective sample-efficient fine-tuning, without forgetting the pre-trained initialization.
2 PROBLEM FORMULATION: FINE-TUNING TUNING WITHOUT OFFLINE DATA
We operate in an infinite-horizon Markov Decision Process (MDP), . =
con-
sisting of a state space S, an action space A, a transition dynamics function P(s' ss a) : S A P(A), a reward function r S x A + R, a discount factor 7 & [0. 1). and an initial state dis- tribution p= P(S). We have access to offline RL pre-trained policy T (als) 8 P(A)
and pre-trained Q-function OP (s, a) S X A + R. Our goal to build an online fine-tuning algorithm that only uses T (als) and Q (8, a) and not Doff to maximize the discounted return: Er=o ((rr(s,,e))
(TT)) -p. (11)))
Problem setup. Note that the RL fine-tuning problem we study in this paper does not allow retaining Doft We will refer to this problem setting as no retention online fine-tuning. Conceptually, our problem setting is close to the offline- to- to-online fine-tuning problem (Nair et al., 2020; Kostrikov al., 2021; Nakamoto et al., 2024), but we do not allow data retention.
3 UNDERSTANDING THE ROLE OF OFFLINE DATA IN ONLINE FINE- -UNNNG
IOL
COL
caiq
1.0
Wile retertton
0.8
W/ retencion
Homoltze
0.6
0.4
0.0
1.0
0.6
Staps
Steps
Shaps
Figure 2: In no-retention fine-tuning IQL, CQL, and CalQL all fail fine-tune kit witchen-partia In contrast, when continually training offline data during fine-tuning, these algorithms work intended. Vertical dotted line indicates the separation between pre-training and fine-tuning.
We first attempt understand why current offline-to-onlin RL methods require retaining the offline dataset. In particular, we hope to understand the pros and cons of retaining offline data and gain insights into developing new methods that serve a similar role, but do not require retaining offline data. We center our study along two axes: (1) we analyze the role of retaining offline data at the beginning of fine-tuning, and (2) we analyze the effect of retaining offline data on asymptotic fine- tuning performance.
3.1 THE ROLE OF OFFLINE DATA AT THE BEGINNING OF -UNN-TNNIGG
Extending observations from prior work (Nakamoto et al., 2024), we find that fine-tuning offline RL initializations fails severely if offline data is retained. Specifically, observe in Figure 2 that offline RL algorithms IQL (Kostrikov et al., 2021) and CQL (Kumar et al., 2020) unlearn right at the beginning of fine-tuning, with performance dropping down to nearly 0% success rate the k citchen-partt. ittrtiiaa al task from D4RL. (Fu et al., 2020b). More importantly, they are not able to al., 2024), an offline RL approach
recover over the course of fine-tuning. CalQL (Nakamoto specifically designed for subsequent fine-tuning by learning calibrated Q-functions, initially drops in performance, but improves with further online training. That said, it still struggles improve beyond its pre- pre-trained performance.
To better understand the above results, we introduce some terminology. We define "unlearning" as the performance drop at the start of fine-tuning, with possible recovery later, and "Forgetting" as the destruction of pre-trained initialization at the beginning of fine-tuning such that recovery becomes nearly impossible with online RL training. In general, unlearning may be unavoidable due to the distribution shift over state-action pairs between offline and online (e.g. consider sparse reward problem where minor change in the policy action results in huge change in return) (Xie et al., 2021). On the other hand, forgetting the pre-trained initialization altogether is problematic since it defeats the benefits of offline RL pre- training. Our goals is for the agent to quicky recover after unlearning, which relies on the effective use of pre-trained knowledge. Observe in Figure that while all algorithms unleamn CQL. and IQI. forget. While CalQL. does not forget, does not
3

---

Published as a conference paper at ICLR 2025
Modrmalized Score
lon offiine distribution)
Loss ofline ditribution)
(anline)
32
13
31
14
33"44
13:58
Steps
0.5
Steps
Steps
When offline data is removed (to different extents) during fine-tuning, performance drops (subfigure a) because Q-function fit on offline dataset distribution diverges (subfigure b. c). even though the Q-function can the online distribution (subfigure d). This plot shows fine-tum aie-tuning CaIQL on kit chen-partial with 0/5/10/25% offiine data in each update batch. We have similar findings with IQL and CQL.
Figure 3:
improve further. This indicates a bottleneck in fine-tuning with online RL without offline data, and different offline RL initializations suffer from this challenge to different extents.
Why does not retaining offline data hurt? To build intuition of what can go wrong, we observe in Figure 3 what happens when retaining different amounts of offline data. Figure 3(b) shows that the average Q-value under the offline distribution begins to diverge as the amount of retained offline data decreases. This Q-value divergence, in turn, corresponds to divergence in the TD-error (Fig- ure 3(c)), perhaps highlighting forgetting. We will show in Section 5 how such divergence can be prevented by our method.
Diving deeper, we find that this divergence only appears under the distribution of the offline data (on which we evaluate metrics but do not train): TD-error on online distribution remain small regard- less of the amount of offline data retained (Figure 3(d);; on the other hand, the TD eπror under the offline data distribution grows substantially with a decrease in offline data (Figure3(c)). This trend is consistent across CQL., IQL, and CalQL,, though the divergence for CalQL. is the least severe per- haps thanks to its calibrated Q-function, which correlates with the stability and best performance of CaIQL in this problem setting in Figure 2. This suggests that the problem with no data retention in current offline-to tilline-to-onliee fine-tuning algorithms likely stems from a form of distribution shift be- tween the online and offline data distribution. Fine-ture Fine-tuning more on-policy data destroys how well we offline data. As we see in Figure 2, this can lead to forgetting of the pre-trained initialization.
Takeaway 1: Distribution shift between offline and online data destroys Q- function fit
Training only online experience without offline data retention can destroy how well the model fits offline data: despite attaining comparable TD-errors on the online data to the setting when offline data is retained, TD-errors under the offline distribution grow larger.
COL
CalQL
IQL
10-24
-100
10-28
Quelue
10
100
10.12
220
10-14
-250
60
10-36
10-36
Complete
Portiel
-300
20
40
450k
500c
550k
Steps
Figure 4: A downward spiral effect in CQL (left), CalQL (middle), and IQL (right) Q-functions in no- retention fine-tuning kit chtchhnmmixen tuning starts 500k steps. Q function goes on downward spiral. When it eventually recovers, the policy has already unlearned (Figure 2).
sitcche-comp Lete. and kitchen-pai sarcchnn-prriiiee iall: When fine-
Why are Q-values underestimated? Not only do the Q-values under the offline data distribution diverge, Q-values on the online distribution also go through underestimation at the onset of fine- tuning (see Figure 4). This Q-value divergence is a manifestation of the "recalibration" process at the boundary between offline RL and online fine-tuning, previously identified in Figure of Nakamoto et al. (2024). However, unlike Nakamoto et al. (2024), the recalibration process in no retention fine-tuning must operate entirely on limited on-policy data. Thus, we see that despite
4

---

Published as a conference paper at ICLR 2025
<ichen-mize--00
CMKX
Normatoss Score
6.6.
0.0
200k
2023
3006
100%
100%
2006
3004
4022
100k
200%
3004
420k
Figure 6: Retaining offline data not efficient, and is outperformed by online RL methods like RLPD three different environments, RLPD starts from scratch, and CalQL starts from pre-trained initialization at step 0.
CaIQL explicitly modifies the scale of the offline Q-function initialization, it still cannot prevent forgetting when offline data is not retained.
Next, we wish to understand why recalibration leads to divergent Q-values. Fig- intu-
Pre-trained Q-function
Online Fine-tuned Q-function
5 provides ition for such Q-value un- derestimation. Consider the very first batch of online rollouts collected from the environment. The target value computation for esti- mating TD-error on online state-action pairs will query the offline Q-function, Qpri on out-of-distribution state-
No no-retantion Update
(5.A)
(5.
Dataset
Pessimism OOD
offine makes TD target pessimistic online
Dataset
Reglay Online
Buffer Dist.
Ground Truth Q function
Pessimistic TD target makes
Learned Offline Q-function
Q-values more underestimated
Learned Online Q-function
Conservative regularizer acts on [S, A) outside replay buffer
Figure 5: Illustration to demonstrate why Q-values are under-estimated in
no-retention fine-tuning and may lead to "downward spiral".
action pairs. Due to conservative offline RL pre-training (e.g. CQL CalQL), learned Q-values at these til-of--dittibuttio state-action pairs are expected to be small. Using such pessimistic values for TD targets in the Bellman backup will, in turn, propagate these underestimation errors onto the new online state-action Figure 3(d) corroborates effective propagation of these TD targets. In ad- dition, if one continues run pessimistic RI. algorithms during fine-tuning (e.g., CQL CalQL), the conservative regularizer continues to minimize out-of-distribun Q-values: with few on-policy rollouts, Q-values for unseen actions keep getting smaller.
This mechanistic understanding hints at "downward spiral" in the learned Q-function at the begin- ning fine-tuning: By this point the Q-function recovers, the policy has forgotten its pre-training and is no longer able to recover to its offline performance (Figure 2). We find that this phenomenon becomes more detrimental as the amount of offline data in fine-tuning is reduced, as shown in Fig- ure 3. The most adverse effects arise when no offline data can be retained. Even calibrated algo- rithms, such as CalQL, though more robust, still suffer from this issue (Figure middle).
Takeaway Re-calibration Q-values leads excessive underestimation
We find that Q-value recalibration at the beginning of fine-tuning leads to excessive under- estimation due to backups with over-pessimistic TD-targets. We call this the "downward spiral".
3.2 THE ADVERSE IMPACT OF OFFLINE DATA ON ASYMPTOTIC PERFORMANCE
As shown above, offline data plays an important role in current offline-to-online algorithms by help- ing with recalibration and preventing Q-value divergence. But how does it affect performance in the long term, once recalibration is over? We find that continued training on offline data hurts final performance and efficiency. Specifically, we find in Figure that offline RL fine-tuning tends to be substantially slower than online RL algorithms from scratch that initialize the replay buffer with offline data (Ball al., 2023; Song et al., 2023). This is quite concerning because indicates that either offline RL pre-training provides no benefits for fine-tuning (unlike other fields of machine learning where pre- praitrann helps substantially) or that existing RL fine-tuning approaches from var- ious offline RL initializations show that a simple modification to online RL methods in the high updates-to -o-- (UTD) regime is able to make good use of initializations from several offline RL methods, without offline data.
not effective enough to make
of offline initializations. We will
5

---

Published as a conference paper at ICLR 2025
Takeaway 3: Retaining offline data hurts asymptotic performance
While retaining offline data appears to be crucial for preventing forgetting at the beginning of fine-tuning for current fine-tuning methods, continuing to make updates on this offline data with an (pessimistic) offline RL algorithm negatively impacts asymptotic performance and efficiency.
WSRL: FAST FINE-TUNING WITHOUT OFFLINE DATA RETENTION
So far we saw that retaining offline data in offline RL algorithms can slow down online fine-tuning but we also cannot remove offline data due to forgetting. How can we tackle both the forgetting of offline initialization and attain asymptotic sample efficiency online?
Key idea. Perhaps one straightforward approach to address asymptotic efficiency issues is to uti- lize a standard online RL approach, with no pessimism or constraints for fine-tuning, unlike cur- rent offline-to-online fine-tuning approaches that still retain offline RL specific techniques in fine- tuning. We can further accelerate online learning by operating in the high updates-to-data (UTD) regime (Ball et al., 2023; Chen al., 2021). The remaining question is: how do we tackle catas- trophic forgetting at the onset of fine-tuning that prevents further improvements online, without offline data? Our insight is that we can "simulate" continued training on offline data by collect- ing small number of warmup transitions with a frozen offline RL policy at the onset of online fine-tuning. Training on these transitions via an aggressive, high updates to-aata (UTD) online RL approach, without retaining offline data can mitigate the challenges of catastrophic forgetting. Our approach, WSRL (Warm Start Reinforcement Learning) instantiates these insights into an extremely simple and practical method that enables us to obtain strong fine-tum fine-tuning results without offline data.
WSRL algorithm. WSRL is an off-policy actor-critic algorithm (Algorithm 1), It initializes the value function and policy with the pre-trained Q-function Q and policy Th could come from any offline RL algorithm. Then, WSRL uses the first K online steps to collect few rollouts using the frozen offline RL policy to simulate the retention of offline data. We refer to this phase as the "wwarmup" phase. After warmup data collection, WSRL uses standard temporal- -ifferreee (TD) updates and policy gradient. For fine-tuning, we fine-tune with a high updates-to-datee (UTD) ratio (Fu et al., 2019; Chen et al., 2021) and follow other best practices. To combat issues such as overestimation (Hasselt, 2010)in the high UTD regime, we use an ensemble of Q functions (Chen al., 2021) and layer normalization (Hiraoka al., 2022) in both the actor and the critic.
Implementation details. Most of the results in this paper use CalQL to initialize WSRL, even though in principle other initializations could be used. See Appendix H for running WSRL with different initializations. We choose Soft Actor-Critic (Haarnoja et al., 20)8a), with an ensemble of 10 Q-networks and layer normalization, as our online fine-tuning algorithm. We use a UTD of 4. This design is inspired by the work of Ball et al. (2023). We use K = 5000 warmup steps at the start of fine-tuning. Further implementation details are in Appendix I.
5 EXPERIMENTAL EVALUATION
The goal of our experiments is to study how well WSRL is able to fine-tune online without offline data retention. We also ablate the design decisions in WSRL to understand the efficacy of WSRL. Concretely, we study the following research questions: (1) Can WSRL enable efficient fine-tuning in the no-retention setting?: (2) How does WSRL compare with methods that do retain offline data?; (3) How critical is the warmup phase in WSRL?; (4) How important is it to use online RL algorithm for online fine-tuning?, and (5) How important is it to pre-train the policy, value function, both?
5.1 BASELINES AND EXPERIMENTAL SETUP
While most prior RL fine-tuning methods are not designed explicitly for the no-retention fine-tuning setting, they can definitely be applied or repurposed to our setting. JSRL (Uchendu et al., 2023) uses pre-trained policy to roll in for some number of steps during each episode and then rolls out with the current policy. The online policy is trained from scratch with both the roll-in and rollout experience. To improve JSRL's competitiveness, we also initialize the online policy with the pre- trained policy and run it with high UTD. Offline RL methods have also been shown to tne-tune online. We consider three offline methods, CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2021), and CaIQL (Nakamoto et al., 2024), an extension to CQL that calibrates the Q-values for efficient fine-tuning. Another method, S02 (Zhang et al., 2024) attempts to balance reward maximization

---

Published as a conference paper at ICLR 2025
WSRL (Curs)
OOL
1XL
Caftx.
SAC (fest)
PSL
antmazo-ultra-de -lv.rtrrrdddve--222
antimaze largo-l
antmaaz large-play-22
kitchen partialact
1.0
Normeized Score
0.33
0.6
0A
0.2
0.3
5.
8.00
0.0
0.0
3.0
100k
2014
300k
100k
2006
30Ck
100k
201k
300k
100c
200k
3006
Steps
Steps
Steps
Stops
kitchen-miodd-vv
door-lanery-vo
pen-binary-0d
Aggregated Over All Envs
1.0
1.0
0.8
Normatted
0.4
0.4
0.4
0.2
0.2
0.2
0.0 100%
0.0
0.0
3.0
2006
300k
1006
2006
20Ck
100%
2008
200k
100g
2006
300k
Steps
Steps
Steps
Stops
Figure 7: In no-retention fine-tuning, WSRL fine-tunes efficiently and greatly outperforms all previous algorithms, which often fail to recover from initial dip in performance, JSRL, the closest baseline, uses data-collection technique similar to warmup.
WSRL (Ours)
CaloL
COL
(online dist.)
loffline dist.)
KL (online dist.)
KL (offline dist.)
0.12
000
0.10
0.08
0.06
0.10
130
0.64
0.88
0.00
Figure 8: KL. divergence of the pre-trained policy and Q-function with fine-tuned ones show that WSRL did not forget its pre-training. The four plots show DKL(mTT [ ) and DEL (softmaxx (eannaxx(@prrr online and offline state distributions. This plot aggregates the KL divergence of six different environments shown separately Figure 19 and 20, along with a more detailed discussion Appendix E.
and pessimistic pre-training via high UTD and perturbed value updates during fine-tuning. Finally, RLPD (Ball et al., 2023) an efficient online RL algorithm that learns from scratch by performing 50/50 sampling of the offline dataset and online buffer for each update batch. While the typical fine- tuning recipe involves sampling each update batch from both the offline dataset and online replay buffer (CQL, CaIQL, RLPD), initializing the replay buffer with the offline dataset (IQL. SO2), we evaluate them in the no-retention setting by only sampling from the online buffer. Since RLPD without 50/ 550 sampling is equivalent to a rapidly- updaatig Soft Actor Critic (Haarnoja et al., 201 20) agent, we refer to it as SAC (fast).
Experimental setup. We study a number of challenging benchmarks tasks and pre-training dataset compositions following protocol used by prior works (Nakamoto et al., 2024; Kostrikov et al., 2021; Kumar et al., 2020). We experiment on Antmaze, Kitchen, and Adroit tasks from D4RL (Fu et al., 2020a) and the Gym MuJoCo locomotion tasks", More discussion is in Appendix C.
5.2 CAN WSRL ENABLE EFFICIENT FINE-TUNING IN NO-RETENTION FINE-TUNING?
Figure 7 compares WSRL with the aforementioned methods applied to no-retention fine-tuning. In seven different tasks, WSRL significantly outperforms baselines, fine-tuning faster to a higher asymptotic performance. CQL, IQL, and CalQL completely fail in this setting, as noted before in Section 3. SAC (fast) completely fails in exploration-heavy environments, but can improve slowly in some environments. The most competitive baseline JSRL. While JSRL achieves the same performance on Adroit WSRL, it is significantly worse on Antmaze and Kit citchen. We hypothesize that this performance gap stems from the ability of WSRL benefit from the pre-trained value initialization, especially on datasets where the pre-trained Q-function is good (e.g. Antmazes ze). Note that while WSRL does suffer from an initial unlearning in performance (see Figure 18) recovers quickly and outperforms other methods which often do not recover all. As we noted
Results in Appendix B.

---

Published as a conference paper at ICLR 2025
WSRL (Ours)
S02
CalQL Jensemble, high UTD)
CalQU
RLPD
antriaze- ultro diverse-V2
antmaaz-largge -ioo---------
kitchen
Normaltzed Normelized Store
0.15
0.30
0.0
0.0
kichen-mixnn--o
doo--binary-v0
pen-Binary-vv
Aggrogated Over Ervs
0.6
100k
Stops
Steps
Stops
Stops
Figure 9: Compared methods that retain offline data online, WSRL, perhaps surprisingly, able to fine-tune faster or competitively despite not retaining any offline data. This highlights benefits warmup.
before, some unlearning might be unavoidable in some environments, but it is important to prevent forgetting of the pre-trained initialization. To evaluate how much the policy and Q-function forget, we measure in Figure 8 the KI. divergence between the pre-trained and fine-tuned Q-functions and separately between pre-trained and fine-tuned policies during the unlearning period. While CQL and CaIQL suffer from divergence in the Q-function, WSRL's Q-function remains stable and relatively close to its pre-trained initialization. The KI. divergence of the policy shows that T&. does not forget its pre- prainiann on states relevant in fine-tuning while unlearning potentially irrelevant information. This shows that while WSRL unlearns initially, it does not forget. See Appendix E for details.
5.3 How DOES WSRL COMPARE TO METHODS THAT RETAIN OFFLINE DATA?
In Figure 9, we compare WSRL to prior methods that still retain and utilize offline data during online fine-tuning. For example, the method labeled CalQL this figure would sample transitions from both the offline data and online data to construct an update batch. To make comparisons fair, we also compare to version of all methods that trains with high UTD of and an ensemble of 10 Q-functions. Observe that WSRL also outperforms these methods despite the fact that these prior methods retain the entire offline dataset during fine-tuning. Specifically, WSRL usually achieves higher asymptotic performance than CalQL and fine-tunes faster, indicating retaining offline data may not the best compromise for asymptotic performance, as we have also shown in Section 3. WSRL also outperforms RLPD, indicating that WSRL can effectively utilize the pre-traine pre-trained value function and policy initializations for rapid online learning.
5.4 How CRITICAL IS THE WARMUP PHASE?
We find that the warmup phase is crucial for fine-tur fine-tuning with online RL. As shown in Figure 10, WSRL without warmup performs significantly worse three different environments. Moreover, we find that using such simple warmup scheme is better than WSRL initializing with the of transitions from the offline dataset or retaining the offline dataset all together (See Appendix and K). We hypothesize that warmup helps because it helps mitigate the distribution shift and prevents divergence and the "downward spiral" at the beginning of fine-tuning. As shown in Figure 11 shows, learned Q-values do not diverge overly pessimistic values when warmup employed and the TD losses remain small on the offline data as well, avoiding the "downward spiral" in Section 3. See more detailed discussion Appendix G.
number
kichenpp
legge-2
doon-ineryyad
Nomaltoed Spore
Al Aammu
0.0
2000
Figure 10: Warmup is critical to fast fine-tuning: When WSRL does not use the initial 5000 steps of warmup, it performs worse or has much higher variance.
8

---

Published as a conference paper at ICLR 2025
Q-values
O volues loffline distribution)
TD loss loffine distribution
-10
Figure 11: Warmup phase helps prevent downward spiral: (left) Q-values during fine-tuninn tine-tuning. and warmup mitigates over-pessimistic values; (middle, right) Q-value and TD emorr evaluated on the offline distribution, where warmup prevents divergence. Data from WSRL on centmaze--azge-aa
Antmaze
Adroit
kitchen-partial-vv
Normalized Returm
Nornatbed Score
W lastion
26
0.4
0.4
Online (Curs)
Algortthim
0.0
0
50c
3011
1300
2002
30k
100k
1.50k
200k
30k
2008
120k
200k
200k
2022
200k
4006
5000
Steps
Steps
Steps
Steps
Ablation studies: the importance of using Q-function initialization (left) and online RL algorithm (right). Left: Q-function initialization is especially helpful when the pre-training dataset has high coverage (e.g. Antmazes). Each plot averages across different dataset types on domain; Right: Importance of fine-tuning with standard online RL algorithm: SAC learns faster than CalQL.
Figure 12:
pessimistic
5.5
How IMPORTANT IS USING STANDARD (NON-PESSIMISTIC) ONLINE RL ALGORITHM FOR FINE-TUNING?
In addition to a warmup phase, using a standard online RL algorithm for tinettuning is also a critical design choice in WSRL. We ablate this decision by attempting to fine-tuning. Here we choose to use CalQL, because CQL and IQL, but is still a pessimistic algorithm. We also initialize CaIQL with the pre-trained policy and Q-function, and use an identical number of warmup steps online. As shown in Figure 12 (right), using an offline algorithm is significantly worse than using a standard online RL algorithm, SAC.
offline RL algorithm during less likely to experience Q-divergence
(Section 3) by design, compared
5.6 HOW IMPORTANT IS IT TO INITIALIZE THE POLICY, VALUE FUNCTION, AND BOTH
Importance of policy initialization. At the start of fine-tuning, WSRL initializes the policy to the pre-trained policy. Since the pre-trained policy is already capable of meaningful actions, it speeds up online learning. In Figure 13, we compare WSRL's performance with and without "policy initialization". and find that initializing with the pre-trained policy crucial for fast fine-tuning.
Benefits of Q-value initialization.
kachen-mleed v
Michen partaa--0
In Figure 12 (left), we observe that while initializing the value function did not bring additional benefits in some domains, it made fine-tuning faster in others. For e.g., initializ- ing with the Q-function is especially helpful the Antmaze ze domains. We hypothesize that this is because the pre-training datasets in Antmazes exhibit much broader coverage compared to those in Adroit and Kitchen, resulting in bet- ter offline Q-function. Consequently, initializing with a more informative Q-function in Antmazes accelerates online fine-tuning.
0.0
lake Ther
100K
200k
300k
100%
200k
50ok
Steps
Figure 13:
Importance of policy initialization in WSRL: with policy initialization, WSRL performs much better in Kitchen.
6 RELATED WORK
Offline-to-onlnee RL. Offline-to-online RL focuses on leveraging an offline dataset to run online RL as sample-efficient possible (Lee et al., 2022; Nair et al., 2020). Many methods developed for this setting utilize offline pre-training followed by a dedicated fine-tuning phase (Nair et al., 2020: Kostrikov al., 2021; Agarwal et al., 2022; Hu et al., 2023; Rafailov et al., 2023; Nakamoto et al., 2024) on a mix of offline and online data. Offline RL methods can also be directly used for fine-tuning by continuing training when adding new online data the offline data buffer (Kumar

---

Published as a conference paper at ICLR 2025
et al., 2020; Kostrikov et al., 2021; Tarasov et al., 2024). Most similar to the goal in our paper is Agarwal et al. (2022), which attempts to use previous RL computations as a better initialization for downstream tasks. However, this work, along with all the methods above, still require retaining all of the pre-training data in the data buffer. As we also show, these methods completely fail without the offline data in the buffer. Our work does not retain offline data. Uchendu al. (2023) utilizes pre-trained policy to guide online fine-tuning without the need of offline data retention, but discard the value function, which typically drives learning in most actor-critic RL algorithms. Ji et al. (2023) and Luo et al. (2024) run offline RL and online RL concurrently on shared replay buffer, following the idea of tandem learning (Ostrovski et al., 2021). Although the high-level motivating principle behind this line of work is also to use offline RL to boost online RL efficienn, there's no pre-training.
Bottlenecks in online RL fine-tuning of offline RL policies. In this work, we show that offline data retention greatly stabilizes the recalibration of the Q-function at the onset of fine-tuning, which otherwise can lead to catastrophic forgetting due to state-action distribution shift. Luo et al. (2023) observe that putting offline data into the offline RL replay buffer stabilizes fine-tuning, but also slows down learning. However, it still remains unclear as to why offline data hurts fine-tuning, which our analysis aims to answer. Lee et al. (2022) identify the existence of state-action distribution shift between offline data and online rollout data, but do not explicitly analyze the negative effects of this shift in online fine-tuning. Nakamoto et al. (2024) show the poor scale calibration of offline pre-trained Q-function to be a key cause for instability of pessimistic algorithms during online fine- tuning with offline data retention. Our analysis extends this analysis to the setting which does not retain offline data and uncovers a distinct reason (state-action distribution shift) that also plagues the method of Nakamoto et al. (2024) in this regime.
Online RL with prior data but no pre-training. Another line of work bypasses offline RL pre- training altogether, directly using purely online RL agent to learn on data samples from both offline data and online interaction data from scratch (Song et al., 2022; Zhou et al., 2023; Ball et al., 2023). Despite not employing pre- vre-training, evidence shows that this recipe often outperforming online RL fine-tum line-tuning methods that utilize a separate offline pre- pre-training phase. If the most effective way utilize prior data is to include it in the replay buffer without any pre- training at all-no matter which pre-training algorithm is used-then it perhaps indicates that we are missing some important ingredients for truly scalable RL formula for pre-training and fine-tuning. In this paper, we show that at least a big part of the problem lies in online fine-tuning of offline RL initializations, and build an extremely simple approach to fix the problem.
work pretty well,
Fine-tuning RL policies with no data retention. Finally, many continual and lifelong RL methods also fine-tune policies without retaining prior experiences due to the non-stationarity assumption in the environment dynamics and task specification (Ring, 1994; Kirkpatrick et al., 2017: Huang et al., 2021: Wotezyk et al., 2021; Powers et al., 2022). Meta-RL methods (Duan et al., 2016; Rothfuss et al,,,,011 2018; Stadie et al., 2018; Rakelly et al., 2019; Arndt et al., 2020; Dorfman et al., 2021; Grigsby al., 2023) assume access to task/environment distribution to optimize for fast fine-tuning online. In contrast, we only consider the single-environment single-task setting where the pretraining and fine-tuning are in the same environment for the same task. In the same single-environment, single task setting, many prior works study on-policy RL methods (e.g., PPO (Schulman et al., 2017)) to fine-tune pre-trained policies (Schaal, 1996; Kober and Peters, 2008; Rajeswaran et al., 2017; Gupta et al., 2019; Wotezyk et al., 2024; Ren et al., 2024). Among these, Wolczyk et al. (2024) also observe unlearning at the beginning of fine-tuning and find that explicitly mitigating unlearning with techniques from continual learning improves the efficiency of fine-tuning, In contrast to these works, we focus on off-policy actor-critic RL methods, that provide an clevated sample efficiency, and require differrntt solution strategies to address this unlearning problem.
CONCLUSION
In this paper, we explore the possibility of fine-tuning RL. agents online without retaining and co- training on any offline datasets. Such setting is important for truly scalable RL, where offline RL. is used to pre-train on a diverse dataset, followed by online RL fine-tuning where keeping the offline data is expensive or impossible. We find that previous offineeto-online RL algorithms fail com- pletely in this setting because of Q-value divergence due to distribution shift. However, if we simply use online RL algorithm for fine-tuning and allow the Q-vuuus to stabilize through warmup phase, we can prevent the Q-divergence. We hope that WSRL sheds light on the challenges in no-retention fine-tuning, and inspire future research on the important paradigm of no-retention RI. fine-tuning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Seohong Park, Mitsuhiko Nakamoto, Kyle Stachowicz, and anonymous reviewers for informative discussions and feedback on an earlier version of this work. This research was supported by the AI Institute and Office of Naval Research under grants N00014-24- 1--22--- and N00014-20 1-2383. We thank TPU Research Cloud (TRC) and Google Cloud for generous compute donations that made this work possible.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Reincarnating reinforcement learning: Reusing prior computation accelerate progress. Ad- vances in neural information processing systems, 35:28955-28977 2022.
Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim-to-real domain adaptation. In 2020 IEEE international conference on robotics and automa- tion (ICRA), pages 2725-2731 IEEE, 2020.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. arXiv preprint arXiv:2302.02945 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without a model. In International Conference on Learning Representa- lions, 2021. URL https: htt.tttss://opeeeeeeeeee rewiee net met/forumm wwt.:iddd=2d=mmmee
C. Cheng, T. Xie, N. Jiang, and A. Agarwal. Adversarially Trained Actor Critic for Offline RL. ICMI., 2022.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement leaming-identtffaillty challenges and effective data collection strategies. Advances Neural Information Processing Systems, 34:4607--41888 2021.
Danny Driess, Fei Xia, Mehdi SM Saijadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03322 2023.
Yan Duan, John Schulman, Xi Chen, Peter Bartlet, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 2016.
Fu, A. Kumar, O. Nachum, G. Tucker, and Levine, D4rl: Datasets for deep data-driven rein- forcement learning. In arXiv, 2020a. URL https://arxxt wwrrxiiioiggooooo worg/orgpo..... httggppf//20...0 22004/2299...
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-
learning algorithms. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019.
Justin Fu. Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020b.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.06866 2021.
Jake Grigsby, Linxi Fan, and Yuke Zhu. Amago: Scalable in-context reinforcement learning for adaptive agents. arXiv preprint 1.Xiv:2310.09977,, 2023.

---

Published as a conference paper at ICLR 2025
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving ong-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019.
T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off--policy maximum entropy deep reinforcement learning with a stochastic actor. In arXiv, 2018a. URL https::///. /ttppss//////..//////o/eesmoee org/ pdf/1801. 01290. .df.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05540 2018b.
Hado van Hasselt. Double q-learning. Proceedings of the 23rd International Conference on Neural Information Processing Systems Volume 2, 2010.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout q-functions for doubly efficient reinforcement learning. International Confer- ence Learning Representations, 2022. URL https: ptttps::/openreviiee nee.eewwweeeeeeeeeeeeee.ee wtt.://wwww.eee///m/..... XCVJMsPV3RT.
Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learn- ing. arXiv preprint arXiv:2311.021999 2023.
Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti. Continual model-based reinforcement learning with hypernetworks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 799-805. IEEE, 2021.
Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Zhang, and Huazhe Xu. Seizing serendipity: Exploiting the value of past success in off-policy actor-critic. arXiv preprint arXiv:2306.02880 02865, 2023.
Zhengyao Jiang. Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in compact latent action space. arXiv preprint arXiv:2208. 10291, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14(13):321133566 2017.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21, 2008.
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-119-1 2020.
Seunghyun Lee, Younggyo Sco, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offiineto-online reinforcement learning via balanced replay and pessimistic Qeensemble. In Conference on Robot Learning. pages 1712-1712. PMLR, 2022.
Yicheng Luo, Jackie Kay, Edward Grefenstette, and Mare Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-off's and practical solutions. arXiv preprint arXiv:2303. 17396, 2023.
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Offline-boostee actor-critie: Adaptively blending optimal historical behaviors in deep off aff-policy rl. arXiv preprint arXiv:2405. 18520, 2024.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint MXiv:2006.093599 2020.
12

---

Published as a conference paper at ICLR 2025
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36, 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pages 11822-16847.. PMLR, 2022.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283-23295, 2021.
Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. CORA: Benchmarks, baselines, and metrics as platform for continual reinforcement learning agents. In Conference on Lifelong Learning Agents, pages 705-743. PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 9748-8763. PMLR, 2021.
Rafael Rafailov, Kyle Beltran Hatch, Victor Kolev, John D Martin, Mariano Phielipp, and Chelsea Finn. Moto: Offline pre-training to online fine-tuning for model-basee robot learning. In Confer- ence on Robot Learning, pages 3654-3671. PMLR, 2023.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 700.009.000, 2017.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinffrrcement learning via probabilistic context variables. In International conference on machine learning, pages 5331 1-340.. PMLR, 2019.
Allen Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409.00530 000000008088, 2024.
Mark Bishop Ring. Continual learning in reinforcement environments. The University of Texas Austin, 1994.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.0678. 2018.
Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv:1707.0632 1707.06347, 2017.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. arXiv preprint arXiv:2210.06718. 2022.
Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hy- brid RL: Using both offline and online data can make RL efficient. The Eleventh International Conference on Learning Representations, 2023, URL opppreveeeeeeeeeeeee/ee net forum? id=yyBis80iUntt
Bradly C Stadie, Ge Yang. Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu. Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via reinfrrennor learning. arXiv preprint 1803 2018
13

---

Published as a conference paper at ICLR 2025
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09928. 2023.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In Inter- national Conference on Machine Learning, pages 34556-34583. PMLR, 2023.
Maciej Wolezyk, Michat Zajac, Razvan Pascanu, Lukasz Kuciński, and Piotr Mitoš. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Informa- tion Processing Systems, 34:28496-28510, 2021.
Maciej Woiczyk, Bartiomiej Cupial, Mateusz Ostaszewski, Michai Bortkiewicz, Michai Zajac. Raz- van Pascanu, Łukasz Kuciński, and Piotr Miloš. Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. arXiv preprint 2iXi::2402.028688. 2024.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances neural information processing systems, 34:27395-2777 27407, 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-olliee reinforce- ment learning. arXiv preprint arXiv:3302.00935, 2023.
Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang. Yu Liu, and Wanli Ouyang. A perspective of q-value estimation on offline-to-onling reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 16908-16916, 2024.
Yifei Zhou, Ayush Sekhari, Yuda Song, and Wen Sun. Offline data enhanced on-policy policy gradient with provable guarantees. arXiv preprint 2:X11:231..0838, 2023.
Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Walke, Oier Mees, and Sergey Levine. Au- tonomous improvement of instruction following skills via foundation models. arXiv preprint arXiv:2407.20633, 2024.
14

---

Published as a conference paper at ICLR 2025
Appendices
A ADDITIONAL RESULTS ON ANTMAZE ENVIRONMENTS
In the main paper, we presented results three of the most challenging antmaaze environments. Here, addition to the set three antmaze environments shown in Figures and 9, we provide the results of WSRL all eight D4RL antmaze environments, together with strong baseline methods. The results show that WSRL significantly better than baselines.
WSRL (ours)
Caloe -----
Calqi (with retention)
RLPD
aittmazzelarge-divvreezz
antmmeze-large-pleyyv2
ertmmaze-ltra-diiveesee
antmazz--llte-pleey-2
0.25
0.0
0.8
0.20
Nemelizss solllldd
0.9
0.1
0.2
0.0
0.0
0.00
100K
2008
300k
100k
200k
300k
100K
200k
300k
100K
200k
3008
Steps
Steps
Steps
Steps
antmaaz-mmedi marmmazeeeedlu---deeeeees
antmaze-medium-ppee playy-2
artmaaze-umaze-22
antmazz-uuaze-diee diverse-v2
1.0
10
1.0
10
0.8
0.6-
0.6
0.6
0.5
Noraalzed
0.2
0.2
0.2
0.0
0.0
Steps
Steps
Steps
Steps
Figure 14: WSRL on all eight D4RL antmaze environments, along with RLPD and CalQL baselines. Step 0 shows the start of fine-tuning for WSRL and CalQL, and start of RLPD. Solid lines do not retain offline data, while dotted lines do.
B RESULTS ON MUJOCO LOCOMOTION ENVIRONMENTS
Additionally, we also apply WSRL nine different Mujoco locomotion domains in the retention fine-tuning setting. Specifically, we experiment with three different robot embodiments (Halfchees Hopper, and Walker), each with three different types of datasets. The random datasets are collected with random policy; the expert datasets are collected with policy trained completion with SAC; and the medi um- deurlllllaaayy datasets are collected with the replay buffer of policy trained to the performance approximately 1/3 of the expert. As Figure 15 shows, WSRL outperforms or similar the best baseline methods.
For WSRL, the hyperparameters were exactly as those in Section 5 and listed in Appendix I with one exception: its pre-trained policy and value function are done with CQL offline training instead of CalQL. This is because these offline datasets have dense rewards and do not end in a terminal state, and therefore do not have ground-trum return-to- to-go to support the CalQL regularizer. For the same reason did not include a CalQL baseline in Figure 15. Both the IQL and CQL bascline Figure 15 do not retain offline data, and use an ensemble of 10 Q functions, along with layer normalization in the functions. RLPD does retain offline data.
C EXPERIMENTAL SETUP
(1) The Antmaze tasks from D4RL (Fu et al., 2020a) are a class of long-horizon navigation tasks that require controlling 8-DOF Ant robot reach goal with sparse reward. The agent has to
15

---

Published as a conference paper at ICLR 2025
Ours
IQL (ensemble)
CQL (ensemble)
RLPD
haffchhettah-anddmmwz
ha halfchheetah-exeertvv2
0.75
0.75
0.75
Normalized
0.50
0.50
0.50
0.25
0.00
0.00
0.00
100k
200k
300k
100k
200k
300k
100k
200k
300k
Nommlizeed Score
hopper-random-v2
hopper-expert-v2
hopper-medium-replayyy
1.0
1.0
0.5
0.5
0.0
0.0
100k
200k
300k
100k
200k
300k
100k
200k
300k
Steps
Steps
Steps
Normalized Score
walkerrzdrandommvz
expert-v2
walker2d -inirizd-mdduum-repaaayyy
1.
10
0.5
0.5
100k
200k
300k
1004
200k
300k
100k
200k
300k
Figure 15: WSRL nine Mujaco locomotion environments with dense rewards, along baselines. Step 0 shows the start of fine-tuning for WSRL and CaIQL, and start of RLPD. Solid lines do not retain offline data, while dotted lines
learn to "stitch" experiences together from a suboptimal dataset. In addition to the original mazes from D4RL, we include Antmaze- Ulttra (Jiang et al., 2022), a larger and more challenging maze. We only include three of the hardest Antmaze environments in Section 5, and provide results on all eight Antmazes in Appendix A. (2) The Kitchen environment is long-ho nonghhoizzon manipulation task to control a 9-DoF Franka robot arm to perfrom sequential subtasks
simulated kitchen. (3) suite of dexterous manipulation tasks control 28-DoF five-
The Adroit t environments fingered hand to manipulate pen to desired position, open a door, and relocating a ball to desired position. The agent observes a binary reward when succeeds. Each data has an offline dataset that provides a narrow offline dataset of 25 human demonstrations and additional trajectories collected by behavior cloning policy. (4) The Mu Mujoco Locomot on environments in D4RL are dense reward settings where agents learn to control robotic joints to perform various locomotion tasks.
D ABLATION STUDIES ON WARMUP PHASE
Impact of different warmup types. One natural question arises: why does the simple approach of warming up the replay buffer significantly boost performance during fine-tuning? One hypothesis is that seeding the replay buffer with some data helps prevent early overfitting, as much work has found in online RL (Nikishin et al., 2022). To test this hypothesis, we plot in Figure 16 in the fine-tuning performance of initializing with random actions, and compare it to initializing with pre- trained policy actions as well as initializing the buffer at all. It is clear that seeding the buffer with random actions significantly underperform the warmup approach, and in fact does not even provide much benefit as compared to not seeding the buffer at all. This suggests that the reason warmup phase helps is not because of preventing overfitting.
Impact different length warmups. Since warmup phase seems to be critical to efficient online fine-tuning, study whether the length of this warmup phase impacts fine-tuning performance. Figure 17 shows warmup phase of lengths 1k, 5k, and 20k on three different environments. It is clear that short warmup phase (1k) sometimes lead to worse asymptotic performance during fine-tuning. On the other hand, longer warmup phases could also hurt (e.g. on Kitchen- mixed) because adds much offline-like data into the replay buffer and slows down online
instability
16

---

Published as a conference paper at ICLR 2025
kitchen-mixed-v0
kitchen-pade Oatten-pattial-v
1.0
Return
0.6
0.6-
0.4
Wermup
Random Actions Init
100K
200k
303k
4008
500R
100%
200k
300%
400x
500k
Steps
Steps
Figure 16: Comparing seeding the buffer with random actions to actions from the pre-trained policy: initializing with the pre-trained policy action works significantly better on k tichen-mi xed (left) and kitchen-partt (right).
kitchen-mixed-vO
door-binary-v0
1.0
Normalizzed Score
0.8
0..4
50k
100k Steps
1b0c
200k
50k
100k
150k
2004
30k
100k Steps
150k
200k
Steps
Figure 17: Impact of warmup phase of length 1k, 5k, 20k on Kitchen-mixxe (lcft), Antmaze- large weargeddivee diverse (middle), and Door-binary (right).
improvement. In WSRL, we did not tune the lengths of the warmup phase beyond what is shown in Figure 17, and we 5000 warmup steps for all environments.
E UNLEARNING AND RECOVERY AT THE START OF FINE-TUNING: AN ANALYSIS
To further illustrate the behavior of offline-to-online RL. agents at the start of online fine-tuning, we show in Figure 18 the performance WSRL, along with two other algorithms, evaluated at much smaller intervals than Figure 7. We fine-tune all agents for 50, 000 steps online across six different environments, and evaluate every 2, 0000 environment steps.
Figure 18 shows that WSRL experiences initial dip in policy performance after 5, 000 steps of warmup, but recovers much faster than CQL and CalQL. We hypothesize that such a dip might be inevitable at the start of online fine-tuning in the offline data retention setting because the policy experiencing different states than what it was trained on and potentially states has never seen (We analyze this with much more detail below). Moreover in some environments (e.g., binary re- ward environments), one might expect small fluctuations in the policy to manifest as large changes in the actual policy performance. However, such brief performance dip does not mean the policy/Q function has been catastrophically destroyed, which is evidenced by the fact that WSRL recovers faster than its peer algorithms and learns faster than online RL algorithms such as RLPD (See Fig- ure 9). If this initial dip would have destroyed all pre-training knowledge from the policy, then we would not expect quick recovery.
In fact, in general, it is impossible to build no data retention fine-tuning algorithm whose perfor- mance does not initially degrade as we move from offline data to online training on all environments and offline data compositions (Xie et al., 2021). Intuitively, this is because it violates sort of "no free lunch" result: for example, consider a sparse reward problem where the reward function is an arbitrary non-smooth function over actions, here catastrophic change in return. Therefore just deducing whether an algorithm has lost is prior or not based performance may not be the most informative. Instead, more meaningful metric to mea-
minor change in policy action results in
17

---

Published as a conference paper at ICLR 2025
WSRL (Ours)
calqu
COL
antmaze la divarse-v2
antmaaee large play-v2
kitheen partial-vO
0.5
Nom alieed Scate
0.4
0.3
0.2
Mtchenn-mbodd-vl
dook-binary-uo
pan-binaay-a0
Normellced Store
0.6
0.2
0.00
0.00
0100
30k
202
30c
E00
5.k
10c
20c
30%
SCR
10C
200
30%
106.
52E
Steps
Stops
Steps
Figure 18: First 50, 000 steps of fine-tuning with denser evaluation intervals. Step in the plot show the start of online fine-tuning, WSRL starts being evaluated after K = 5000 steps of warmup. All agents are evaluated in the no-retention fine-tuning setting.
sure catastrophic forgetting is to evaluate how much fine-tuning algorithm with no data retention deviates from its pre-training and how fast can adjust to the online state-action distribution.
Therefore, to investigate how much the policy and Q-function have deviated from its offline- pretraining during the initial "performance dip", we plot the KL divergence between the pre-trained policy/Q-function and the fine-tuned ones. Figure 19 shows the KL divergence between the policies (TL pre (TT)) evaluated on both the online distribution and the offline distribution. In Figure 19 (Top), we can see that DKL (" (" ( on the offline distribution generally increases during fine- tuning for all three agents. This increase indicates that the fine-tuned policy has deviated from the pre-trained policy on least some parts of the dataset distribution. This is actually expected in the no-retention fine-tuning setting because of the distribution shift from offline to online. To be more specific, for example, Antmaze ze environments, the offline dataset exhibits a very diverse state-action distribution, covering almost all the locations in the entire maze, while fine-tuming is a single-goal navigation task. no-retention fine-tuning, the agent incentivized to forget about parts of the offline dataset that is irrelevant to the fine-tuning task and specialize the online task. Compared to CQL and CalQL, WSRL generally has the same asymptotic value for DKL but reaches convergence much faster. This suggests that WSRL actively adapts the online distribution much quicker compared to its no-retention counterparts, perhaps thanks to its non-conservative objective optimized solely on online data and its high update-to-data ratio during online RL.
On the other hand, Figure 19 (Bottom) shows DKL on the online distribution for WSRL increases slightly, but is much smaller compared to CQL and CalQL (without data retention). This indicates that WSRL's policy remains almost the same on the online distribution. This is desirable because the pre-trained policy already has decent performance, and a capable fine-tuning algorithm should not forget that capability while adjusting slightly to unseen (but not gut-of-distributin states. In summary, due to the distribution shift from offline pre-training to no-retention fine-tuning, WSRL is forgetting experience in the offline dataset that was learned during offline pre-training but is in reality irrelevant for specializing to the online task. contrast, it is instead specializing to the online task.
In addition to the above analysis on the policy, in Figure 20, we plot the KL divergence of pre-trained Q-function Que and fine-tuned Q-function Qo. We normalize the Q-values into distribution with softmax and plot Dx 1so (so ) [softmar(())) (Qo)). We evaluate the D on states sampled from both the offline dataset distribution (Figure 20 Top) and the online replay buffer distribution (Figure 20 Bottom), and on actions sampled from equal-weighted mix of T (s) and (s). The results show that CQL and CalQL f-unnttions usually diverge significantly from pre-training, probably due the downward spiral phenomenon described in Section 3. In comparison, WSRL's
pre
18

---

Published as a conference paper at ICLR 2025
WSRL (Ours)
calqu
COL
antmaaee large diverse-22
antmaze- arge play
kitchen partial v0
0.14
0.10
0.12
2.03
0.00
0.00
0.09
5.4
0.50
door-binary-vo
pen-binaryuv
2.44
2.32
0.20
0.20
3.05
0.30
2.3
5.4
0.00
124
20k
30k
6Bcc
502
10x
201
324
16k
50k
10k
20c
30X
43x
S0K
Stops
Steps
WSRL [Ours)
CalQL
CCL
go play-
Kichen-partiall-0
2.10
0.20
0.00
0.26
0.20
0.02
0.04
0.04
0.00
3.02
0.39
-0.00
20%
25A
506
10c
304
k Atcheeo-miicedda
door-hisary-ao
pen-binary-ut
3.03
3.04
0.15
"ontine
3.44
0.10
0.35
0.02
0.25
0.00
201
56k
20%
Scops
Stops
Staps
Figure 19: Policy KL divergence Dxz (TT) between the pre-trained offline policy and the fine-tuned divergence evaluated the
online policy at the first 50k steps of fine-tuning. The top plot shows offline dataset distribution; the bottom plot shows the KL divergence on the online state state-and-acee and-action distribution, sampled from the replay buffer. WSRL is not plotted during the first 5, 000 steps of warmup. CQL and CalQL do not retain offline data.
Q-function remains stable on both the online and offline distributions, suggesting WSRL did not forget its pre-trained Q-function.
In summary, the above analysis on the KL divergence suggests that WSRL remains stable and quickly adapts during fine-tuning and does not forget priors learned from offline pre-training.
F DOES FREEZING THE POLICY AT THE START OF FINE-TUNING HELP?
Since Appendix E shows brief period at the start of fine-tuning where policy performance take dip, one natural question is whether such dip is avoidable. The most straight forward way avoid such a drop in policy performance is to freeze the policy during initial fine-tuning. In other words, for N steps at the on set of fine-tuning, we only pass the gradients through the function and train the Q function, but freeze the policy. After N online steps, we start training both the policy and the Q function.
Figure 21 shows the performance of WSRL after freezing the policy for N (10k, steps. It's obvious that even when we freeze the policy for some number of steps to let the Q-function adjust online. the policy still suffers dip after it is unfrozen. In fact, this is somewhat because the policy needs to adjust to the OOD online state-action distribution, as well as the new online Q-function, and such adjustment process is expected to make the policy performance worse.
expected result
19

---

Published as a conference paper at ICLR 2025
WSRL (Ours)
calqu
COL
arcmmaz largeaaaoooaoooo www.aaarrgggeeaaaaee diversa
antmaze large play
kitchen- paa--paaaaaaii
1.00
Q (oftiiee e1st)
100
1500
000
1000
50-
Ktchen-mmBedd-J0
door-binary-u0
pen-binary-v
1200
1500
10030
1000
Q. (ofTlre
000
102
20c
30c
60ic
5OC
10c
20c
30c
COc
50c
10c
20c
30c
45k
50k
Steps
WSRL [Ours]
CalqL
CI2L
adttaazee-arraa watps-larrg--ll------. play-42
éttchen-partial-o
122
15:0
250
0 KL Carime dist
75
150
52
10%
206
10k
304
5/6
10c
201
docr-binary-40
per-bbiraaruu
302
1200
Q. (orinme des.)
252
1000
200
10c
103
104 204
54k
201 321
Scap5
Scaps
Stops
[[softmaz(Q)) between the pre-trained Q- function and the fine-tuned Q- Q-function at the first 50k steps of tin-ttnnig. We evaluate the Q-functions by sampling states from the offline dataset distribution (Top) and the online buffer distribution (Bottom), and sample actions by running pre ne on the sampled states. WSRL not plotted during the first 5, 000 steps of warmup. CQL and CalQI. do not retain offline data.
Figure 20: Q-function KL divergence Dxz
G WHY WARM-UP PREVENTS Q-VALUES DIVERGENCE
In WSRL, the policy and value function is pre-trained offline with CalQL (Nakamoto et al., 2024), and the online fine-tuning process is done with SAC (Haarnoja al., 2018a). This change of RL algorithm could lead to miscalibration issues, where the pre-trained values are more pessimistic than ground truth values. As we have shown in Section 3, this hurts fine-tuning when backs up pes- simistic target Q-value through the Bellman update. This particularly hurts when the Bellman target computed on an OOD actenn pair, because OOD state-action pair have more pessimistic val- ues than state-action paris seen in the offline dataset, by the nature of pessimistic pre-training. If there were no warm-up phase, the agent will collect OOD data into the buffer, leading to Bellman backups with pessimistic target Q values, which in turn leads to Q-divergence. This is the "down- ward spiral" phenomenon in Section 3. However, warmup solves this problem by putting more offline-like like data into the replay buffer where Q-values not as pessimistic, thereby preventing the downward spiral the online Bellman backups and uses high UTD in online RL to quickly re-calibrate the Q-values.
20

---

Published as a conference paper at ICLR 2025
WSRL [Ours]
Freeze 10k
Freeze 30k
antmaze: la divarse-v2
antmaaze large play-v2
kitchen partial-v0
0.5
Nom alizedee Soute
0.4
ktchhen-mbodd-vl
daok-binary-u0
pan-biinary-a
Normalice Store
9.
0.4
0.0
0.00
504
202 30c
50k
10c
20c
30%
420
SCB
10C
2004
30%
606
52E
50ops
Steps
Figure 21: Freezing the policy for N E {10k, 30k} steps at the onset fine-tuning doesn't prevent the performance dip. In the plot, we show policy performance of WSRL vs. WSRL with initial policy freeze across six environments. Step 0 is the start of online fine-tuning.
H ABLATION STUDIES ON DIFFERENT TYPES OF VALUE INITIALIZATION
WSRL is agnostic the offfine RL pre-training algorithm. Furthermore, we find that it is erucial which specific offline RL algorithm we use to obtain the pre-trained Q values. In Figure 22, we show that the Q-values from IQL, CQL, and CalQL work just as well on three different envi- ronments, even though CQL optimizes for conservative Q-values, CaiQL is less conservative, and IQL is not conservative all. In particular, we observe that Calibrated Q- Qavalues an initializa- tion provides some small performance benefits Ki titcheeemii xed. Therefore, we use calibrated Q-values from CalQL offline pre-training for our main experiments.
Bichhen-mixe-vv
kitchen-partial-y
kitchen-comllete-00
Value Int: Type
CalOL
Normalized
0.4
0.2
0
100k
200k
300k
100k
200k
300k
100k
200k
200k
Steps
Steps
Figure 22: The offline RL algorithm used to pre-train the Q-values does not affect performance: for Kit citchhn-mixe xcd (left), xitchen-part tail (middle), and Sitchen-compleen WSRL is able to achieve similar performances by initializing with pre-trained values from CQL, IQL, and CalQL, though CalQL initializations have small benefits Kitchen waschnn-pprtiia....
I IMPLEMENTATION DETAILS
Code for WSRL is released at http:/ghhhbb..ooo////oo/oy////aa
Pseudocode. See Algorithm 1. During online RL updates, the critic is updated with standard tem- poral difference loss and the actor is updated with policy gradient (in this case, a reparameterization based policy gradient estimator) with entropy regularization in Soft Actor Critic (Haarnoja al., 2018a).
WSRL Hyperparameters. We use 5K warmup steps (K 000). For the online RL algorithm in WSRL, we use the online SAC (Haarnoja et al., 2018b) implementation in RLPD (Ball et al., 2023) with UTD of and actor delay of 4 (update the actor once for every four critic steps), batch size
21

---

Published as a conference paper at ICLR 2025
Algorithm WSRL: Warm Start Reinforcement Learning
Require: Offline RL algorithm Aotr. Pre-training dataset Doft. Q pre < TrainOflineeAAn Dott)
Offline RL pre-training
" pre Online RL algorithm Asn with UTD M, Replay buffer R + 0, warmup step 0 "
Require:
K.
Qo Q Fi 6 "
R
Initialization
while step I max steps do if step - K then
(s, t interact( pre environment)
Warmup Phase
else
(8. a,.a8', r) interact(T environment)
end if
R < R
if' step > K then
Batch1, Batthh,Batchh... Batchw ~ R
Qo < TemporaDDiffernneepdat(o.. Batch;) for M times
High UTD Critic
Update
T +- PolicyGradient Batch, U...U Batchar & Actor Update with Actor Delay
of M
end if end while
of 256, actor learning rate of 1e 4, critie learning rate of 3e 4, and temperature learning rate of
le - 4. We use and ensemble of 10 Q functions, and predict the Q value by randomly sub-sampling 2 and taking the min over the Q-functions (Chen et al., 2021). When we initialize the policy network and the Q-function network from offline RL pre-training, we keep the optimizer state of these networks.
In antmaze environments, we find it important to calculate the TD-target by taking the maximum of the TD-target over the ensemble of 10 Q-functions as done by Kumar et al. (2020). This design does not affect performance on the other environments. We hypothesize that this is because antmaze environments require more optimism online, cooperating the design decisions by Ball et al. (2023) as described below.
Environment Details. All environments use a discount factor of 0.99. We set the reward scale in the benchmark tasks following previous work (Nakamoto et al., 2024; Kostrikov et al., 2021; Kumar al., 2020). In Antmaze and Adroit environments, we use sparse reward of 5 at the goal and -5 at each step. In Kitchen, we use 0 at the goal, and - for each subtask that is not completed at the current timestep. giving possible reward values -4, 3, 2, - -,, Mujoco locomotion environment,
the original environment dense rewards. We pre-train 1M steps Antmaze, 20k steps on Adroit, 250k steps on Kitchen and Mujoco locomotion.
Baseline Hyperparameters. All SAC-based methods the same learning as WSRL ahove, and IQL-based methods use actor and critic learning rate of 3e - 4. All methods that have high update-to-data ratio or an ensemble of Q-functions use layer normalization as a regularization. The method-specific details are listed below, with hyperparameters gotten from the original papers with no further tuning.
IQL. Antmaze environments use expectile 0.9, and all other environments use expectile 0.7. For the inverse temperature in the actor, we use 10 in Antmaze, 3 in Mujoco locomotion, and 0.5 for Kitchen and Adroit. Unless otherwise specified, IQL uses two Q functions and takes the minimum over them to estimate the Q value.
CQL / CalQL. In Antmazes, we use the dual version of the CQL objective and set gap to 0.8. In Mujoco locomotion and Kitchen environments, we set CQL regularizer weight a 5.
"n the main performance comparison results in Figures and 9, kitchen environments use a maximum episode length of 280 as the default from D4RL, while other ablation and analysis experiments may use version of kitchen by Nakamoto et al. (2024) that has maximum episode length of 1000 but other changes. All comparisons plotted are the same kitchen version.
22

---

Published as a conference paper at ICLR 2025
In Adroit environments, we set 1. CQL and CalQL the same CQL regularizer for both pre-trainign and fine-tuning. In experiments that do retain offline data, each update batch samples 50% from the offline data on Antmaze, Adroit, and Mujoco locomotion environments, and 25% on Kitchen environments (Nakamoto et al., 2024). Unless otherwise specified, CQL/CalQI. uses two Q functions and takes the minimum over them to estimate the Q-value.
RLPD/ / (fast)) We use an ensemble of 10 Q-functions and a UTD of 4 with batch size 256. Following (Ball et al., 2023), in antmaze environments, we predict the Q-value from the ensemble by randomly subsampling 1 Q-function. This is needed for more optimism online in antmaze envi- ronments. In all other environments, we subsample 2 O-functions and take the minimum over them to estimate the Q-value.
JSRL. Same hyperparameters as RLPD, where we improve JSRL's competitiveness with a Q- ensemble and UTD. For each online interaction episode, JSRL decides whether to roll in the pre- trained frozen policy or roll out the fine-tuned policy with probability 0.5. This probability decreases linearly to 0 over the first 100k: fine-tuning steps. In episode where the JSRL decides the roll in the pre-trained frozen policy, the number of step it rolls in follows a geometric distribution with 7 0.99, after which the fine-tuned policy is rolled out until the end of the episode.
S02. Same hyperparameters as RLPD. Different from the original paper, we use 10 Q- ensembles to make fair comparison with WSRL and other baselines which have the same number of Q-ensembles. For the action noise, we use a standard normal with variance 0.3, and we clip the action noise to be between (-0.6, 0.6).
WARMUP WITH TRANSITIONS FROM THE OFFLINE DATASET
We have shown in Section 5 that the warmup period is essential for efficient fine-tuning with WSRL. One interesting question is whether such warmup data can be collected by sampling the offline dataset, instead of online interactions with the frozen pre-trained policy as in WSRL. Therefore, we run an ablation experiment in Figure 23 where we replace the 5, 000 steps of warmup period by initializing the online replay buffer with 5, 000 random transitions sampled from the offline dataset, which we will refer to "Dataset Warmup". As Figure 23 shows, while the two methods are similar on Adroit environments, WSRL is slightly better in Kitchen and much better on Antmaze. This is perhaps because the 5000 randomly sampled transitions might not be relevant the online fine-tuning task, especially in Antmaze where the dataset has diverse state-action coverage (See Appendix E for a more detailed discussion). When the replay buffer is initialized with less relevant data, it is less effective at preventing Q-value divergence (Section 3) and recalibrating the online Q-function and policy. This perhaps highlights the utility our approach: despite not having access to any offline data, WSRL is able to achieve similar or better performance than using transitions from the offline dataset in the no-retention fine-tuning setting.
K WSRL WITH OFFLINE DATA RETENTION
In the main paper, we have shown that WSRL can efficiently fine-tune without retaining the offline pre-training dataset. One natural question arises: can WSRL do even better if we allow offline data retention? To answer this question, we run WSRL with the online replay buffer initialized with the whole offline dataset. Figure 24 shows that average, retaining the offline data does not give WSRL any advantages, probably because it already has the necessary knowledge in the offline policy and Q-function; WSRL is actually bit faster later on in fine-tuning, perhaps due the fact that it is updating the policy on more online data.
L ABLATING THE EFFECTS OF Q-ENSEMBLE AND LAYER NORMALIZATION
In WSRL, we choose to use the most effective online RL algorithm fine-tune with high update-to- data (UTD) ratio. Following the design choices by Ball et al. (2023), we also use an ensemble of 10 Q-functions and layer normalization as regularization to stabilize online training in the high UTD regime. In Figure 7, SAC (fast) and JSRL also has high UTD and therefore we also apply both regularizations. However, implement IQL., CQL,, CalQL without these regularizations, as in the
23

---

Published as a conference paper at ICLR 2025
(0uns)
warmup
uttradiverse-22
antmaze- diverse-v2
large-p
ktthehrparttsiio
0.25
1.0
1.0
0.25
0.6
0.30
0.2
0.20
0.0
0.0
1905
2008
500k
1001
2018
3018
1008
2002
2006
1008
2012
ktchanemitaeio
dom--nnary-at
pen-tinary-al
Agg regated All
Nemalled Scote
0.5
014
0.4
0.4
00.
02
0.2
100c
300k
100g
2026
330k
100%
2004
3024
1006:
2008
3026
Steps
Steps
Steps
Stops
Figure 23: Warming up with transitions from the offline dataset is less effective than warming up with online interactions (WSRL).
WSAL (Ours)
WSRL Butfer Init
antr-ace-ultr--dieeeee
2.0
entm aze -igee-eii-ie
ent maze large playv2
kitchenn-artial--O0
5-
66
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
ittchen
ocor birary
pen binary-v
Aggregated
1.0
Nurmelue Loed Store
0.6
2.0
0.5
0.7
0.4
0.0
2006
2004
3004
1000
200k
3004
1008
2024
300k
100%
300k
305k
Stops
Steps
Steps
Sceps
Figure 24: Initializing the replay buffer with the offline dataset does not give WSRL any advantage during fine-tuning, and may make it a bit slower.
original papers. To ablate the effects of layer normalization and Q-ensemble in no-retention fine- tuning, we apply both to each of IQL, CQL, and CalQL. In Figure 25, we apply layer normalization after each dense layer in the actor and the critic MLP, and find that minimally affect performance. In Figure 26, we apply layer normalization as well as a Q-ensemble, and refer to the combination as REDQ (Chen et al., 2021). We find that REDQ helps significantly on Antma ze tasks, not huge gains in other environments. Overall, WSRL still significantly outperforms IQL., CQL., and CalQI. with extra regularizations.
M WSRL WITH DIFFERENT UPDATE-TO-DATE RATIOS
In Section 5, we mainly experiment with UTD-4 for all methods. One interesting question is whether WSRL benefit from line RL methods with higher UTDs. In Figure 27, we compare WSRL against RLPD under UTD 20, and find that while UTD 20 improves performance slightly, the difference is not huge.
higher UTD, and how it compares with other fast on-
24

---

Published as a conference paper at ICLR 2025
(0urs)
IQL
COL
ayer nom
layer nom
CaloL layer norm
artmaae wltrrlddiveeeeee diversev2
ontmmaze larnazz-irgg---herree-- x2
antimaze for play-v2
Kitchen partinn-pattll-dd
hamalize
1008
100K
200K
200K
1001
5E0K
1000
2008
5006
Steps
Staps
Stpps
Steps
Richan-maaed-00
dbe--Staryy40
pan Sanayyo
Aggregazed Oieer All Erivs
0.3
0.4
0.4
0.0
0.03
1000
200%
2006
100%
2006
200k
100%
200%
2006
100x
2008
2006
Figure 25: Layer normalization does not impact the performance of IQL, CQL. CalQL no-retention fine- tuning.
WSRL (Ours)
CDL
Ca
RFDQ
RFDQ
Calq
RFDD
on trtmmaee u traddvveree222
antmaze-large diverse-2
artmaze-lrrge plyy
kitchen partal-y0
1.0
1.01
0.5
Nor meliian
66-
0.
0.6
0.2
0.2
0.0
Stupe
Staps
Steps
Steps
don-drnarroon
pen-inaryyu
Rogragated Over All Erve
1.c
3.0
1.0
Hennaloed Saare
0.5
0.5
0.6
0.0
0.4
0.2
Figure 26: Impact layer normalization and Q-ensemble IQL, CQL, CalQL no-retention fine-tuning:
benefits ze environments greatly, but not so much on other environments.
N WSRL WITH VARYING LEVELS OF OFFLINE POLICY
We investigate how WSRL performs with varying levels of expertise of the offline pre-trained pol- icy. Specifically, we consider Ki wwwhh-ccmmleeeeecc pecially hard tasks for offline RL where pre-training with CalQL leads to poor performance. In Recolate-binary-oee
and Relocate -iiarryvv,, two
CalQL completely fails and has pre-trained performance near CQL and IQL also has pre-training performance 0, indicating that this task is inherently hard for of- fline RL. agents. (70.83%) despite our tunning efforts, which suggests there is some inherent limitation in CalQL learning good Q-funciton this domain. Not surprisingly, Figure 28 shows that WSRL also per- forms poorly: while WSRL learn somewhat in Kitchen-comple wetemmeeeeeeee with non-zero initialization, it completely fails to learn in Adroit -boeiit-bbirry-va This is expected because when pre- pre-training fails, initializing with the pre-trained network may not bring any useful information gain, and may actually hurt fine-tuning by reducing the network's plasticity (Nikishin et al., 2022), known issue in online RL. In such situations, one may wish resort different methods (e.g. online RL) rather than fine-tuning from bad initialization.
cachen-comlltte-eee CaiQL (15.47%) significantly underperforms IQL.
25

---

Published as a conference paper at ICLR 2025
WSRL IUTD 20)
WEAL (UTD
RLPO (UTO 201
RUPO (UTO 4)
4TErraece--argee diverwe--2
antimauue laggepl
par iar
0.0
0.3
0.5
U.S
0.1
1022
3002
2011
3004
500c
2002
1022
200k
Steds
Stops
Stops
Stops
pan-5
doon-binary-o
Agoragated Over All Envs
1.0
1.0
0.2
0.5
0.6
0.4
0.2
0.2
Figure 27: UTD 20 vs. UTD 4: For both WSRL and RLPD, UTD 20 performs only slightly better than UTD
Ours
RLPD
kitchen-complete.v
relocale-binary-e
1.0
Normalized Score
0.8
0.6
0.6
0.4
0.7
100k
200k
300k
100k
200k:
300k
Figure 28: On environments where the pre-training completely fails, WSRL does not work well.
26


Reviews:
[{"review_id": "DArHa8DdD8", "reviewer": "Reviewer_Wxna", "summary": "This paper explores the potential for efficient offline-to-online RL without retaining offline data. The authors investigate how offline data contributes to online fine-tuning in existing methods and propose WSRL, which leverages the offline policy and Q-function as initialization. Extensive experiments highlight the effectiveness of WSRL and its components.", "questions": "While off-policy algorithms are naturally suited for the online phase, is it possible to apply online policy methods with monotonic improvement properties (e.g., PPO, TRPO) and still achieve comparable performance during online training?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is well-written and easy to follow, tackling the interesting topic of offline-to-online RL. The study of related works is thorough and comprehensive. Building on this foundation, the paper illustrates how retaining offline data contributes to previous approaches and provides three key takeaway messages. The authors then propose WSRL, a simple and intuitive method designed to enhance the stability and efficiency of online fine-tuning.\n\nThe experimental results show that WSRL outperforms or is at least competitive with previous methods. The ablation studies are detailed and convincingly demonstrate the necessity of all components.", "weaknesses": "I did not identify any major flaws in this work. One potential limitation, however, is that WSRL relies on off-policy online algorithms, as it requires interactions from the warmup phase to stabilize the fine-tuning process.\n\n\n\nTypos:\n\nOfline --> Offline (In the box of Takeway 3, Line 3)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vh810gTf4y", "reviewer": "Reviewer_tchB", "summary": "The paper proposes a new method, Warm Start Reinforcement Learning (WSRL), for efficient online reinforcement learning (RL) fine-tuning without the need to retain offline data. Traditional RL fine-tuning often relies on retaining offline data to prevent performance degradation, but this approach is inefficient and can slow down learning. The authors argue that, by using a warmup phase that gathers initial online rollouts with a pre-trained policy, WSRL can \"recalibrate\" the offline Q-function to online data, enabling the algorithm to discard offline data and still achieve stable and effective learning. WSRL, which emphasizes a high updates-to-data (UTD) regime, proves faster and more effective across diverse tasks compared to traditional algorithms, demonstrating higher performance without offline data retention.", "questions": "- Could you provide experimental results from environments like the MuJoCo dataset, where offline performance is typically strong?\n- Could you provide experimental results using other offline algorithms (e.g., IQL, CQL) with the proposed methods applied?\n- Could you provide experimental results for all 8 Antmaze datasets?\n- Could you provide a comparison with “A Perspective of Q-Value Estimation on Offline-to-Online Reinforcement Learning”?\n- This paper excludes offline data during online fine-tuning, implements a warmup phase, and increases the UTD ratio. What would happen if we retained the offline data while implementing the warmup phase and increasing the UTD ratio?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- WSRL removes the dependency on large offline datasets for online fine-tuning, based on various experimental takeaways.\n- WSRL shows improved learning speed and higher asymptotic performance across various benchmark tasks, outperforming state-of-the-art fine-tuning methods.\n- Introducing a warmup phase and applying a high UTD regime for online RL is an efficient approach to address the problem of Q-function divergence without retaining offline data.\n- The method is adaptable across different RL environments, showing versatility and potential applicability to a broad range of tasks.", "weaknesses": "- The offline performance in the experimental setup is poor from the start (Section 6, Figures 6 and 7). The performance graph shows that, except for pen-binary, the offline score ranges from 0 to 20 points, which is quite low.  This weak offline performance may suggest that the offline data did not fit as effectively as it does with other algorithms or that, in this particular setup, the offline data may not have provided much support for online fine-tuning. The claim that the proposed method prevents unlearning is also debatable, as there was minimal offline performance to “unlearn” initially. Analyzing cases where the offline score reaches at least 60, ideally between 80 and 100, would be more informative. Conducting further experiments on Mujoco Hopper/Walker2D with medium, medium-replay, expert, or Adroit-dense expert settings—where offline performance tends to be stronger—would be beneficial.\n  - Additionally, this paper mentions using SAC with a 10-ensemble for training, but it would also be necessary to examine how other offline algorithms like IQL and CQL perform when applying the proposed methods—such as excluding offline data, using a warmup phase, and employing a high UTD ratio.\n\n- Unlearning likely occurs mostly at the very early stages of training, but the plots are spaced at intervals of about 50,000 steps. At this interval, even if unlearning did occur, it may have already been resolved by the next plotted point. To demonstrate the absence of unlearning, it would help to show early performance at smaller timestep intervals.\n\n- The paper employs strong language, such as “Should Not” and “Completely unnecessary,” though the evidence provided may not be fully convincing. Without theoretical support, it needs stronger empirical backing; however, the experiments may not comprehensively represent all RL tasks. Removing such strong expressions would not alter the flow of the paper and might actually improve its persuasiveness.\n\n- This paper conducted experiments on eight datasets across three domains (Antmaze ultra-diverse/large-diverse/large-play, Kitchen partial/mixed/complete, and Adroit door/pen), which appears insufficient compared to other conference papers. While experiments were conducted on Antmaze ultra-diverse, ultra-play was not included, which may suggest some inconsistency in selection. Given that this paper is experiment-driven and lacks a strong theoretical foundation, it would benefit from more extensive experiments, such as including Antmaze ultra/large/medium/umaze play/diverse datasets, to provide a more comprehensive set of results.\n\n- In Figure 4 of Section 4.1 and Figure 9 of Section 6.5, it’s unclear what the Q-value magnitude itself indicates. Without a comparison to the true Q-value, it could be that one is underestimated or, conversely, that the other is overestimated. Additionally, even if the overall scale is low or high, as long as the general trend follows the true Q, there is no issue (the relative comparison of Q values is more important than their absolute magnitude). For a meaningful comparison, it might be better to separate in-distribution Q from out-of-distribution Q. t. Additionally, considering the focus on Q-value analysis and the approach from a Q-value perspective, I am curious to see a comparison with the paper “A Perspective of Q-Value Estimation on Offline-to-Online Reinforcement Learning” [1].\n\nOverall, while achieving better performance than the baselines in the experimental settings (especially for Antmaze Large and Ultra) is highly positive, the points discussed above prevent the claim that \"Efficient Online Reinforcement Learning Fine-Tuning Should Not Retain Offline Data\" from being fully convincing.\n\n[1] Zhang, Yinmin, et al. \"A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zqj6VtIUZJ", "reviewer": "Reviewer_PY4G", "summary": "This paper studies the Offline to Online fine-tuning problem in RL. The author trying to remove the need of keeping the offline dataset when finetuning online, which is becoming impractical giving the growing size of pretraining dataset. Through a detailed analysis of Q-values during the fine-tuning phase, the authors reveal two critical insights:\n1. Without the offline dataset, pretrained Q-values are quickly unlearned\n2. Continuing conservative offline RL on online data leads to performance degradation\n\nIn response, the authors propose WSRL, a method that:\n1. Implements a warmup phase to seed the replay buffer with the pretrained policy to mitigate the unlearning issue\n2. Utilizes state-of-the-art online RL algorithms for fine-tuning\n\nThe approach is demonstrated through promising results on D4RL tasks.", "questions": "- Could you clarify Figure 3? The caption says \"Fine-tuning starts at step 0.\", but the plots are start from around 250k.\n- Some of the plots are truncated, i.e. not starting from 0. Could you please clarify on that? Based on the analysis, the warm-up phase should mostly solve the problem in the beginning of the finetuning phase. I would even suggest the authors to have a zoom-in version of that parts of the plot, but the truncated version seems to be missing exactly the transition part.\n- Since warm-up phase is to just seed the replay buffer with some data collected from the pretrained policy, will it also be fine to subsample a small portion of the offline dataset to initialize the replay buffer?\n- From the results in Figure 7, seem like the warmup phase does not really solve the unlearning problem, i.e. the initial value for WSRL is much lower than the CalQL initialization. Is this a desired behaviour?\n- I am not sure how the warm-up phase solves the miscalibration problem present in CalQL. For example when you initialize the policy and Q with the CQL, although the warmup data will help to combat the distribution mismatch, the change of the RL algorithm will still make the Q function miscalibrated which in turn make the policy unlearn. How do you think of this issue?\n- Won't it be beneficial to just run a bunch of gradient steps only on the Q function after you collect the warmup data until it recalibrates?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper studies an important and timely topic in RL.\n- This paper provides some analysis for transition phase to motivate the problems.\n- The proposed method is simple to implement.", "weaknesses": "A lot of the results are confusing and not convincing, as referring in the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "57v0khGKXV", "reviewer": "Reviewer_7N5M", "summary": "In this paper, the authors highlight the requirement for online fine-tuning algorithms to retain offline data in their replay buffers to remain effective. The authors thus aim to remove this requirement, by instead using a small amount of online data collected from rollouts of the frozen pre-trained policy. They show that the TD-error for the online policy under the offline data distribution diverges as less offline data is retained. The postulate that this is due to offline/online data distribution mismatch.", "questions": "1. In Figure 7, JSRL is shown as not retaining offline data. However, JSRL is implemented over IQL in Uchendu et al., and the IQL implementation linked in the paper does retain offline data. Did the authors specify to you that they chose not to retain offline data?\n2. It is hypothesised in section 4.1 that backups of small, over-pessimistic Q-values from pessimistic offline algorithms cause the Q-values divergence. However, in Figure 9 you mention the warm-up helps avoid over-pessimistic values for WSRL. I had understood that WSRL used an online, non-pessimistic algorithm (RLPD) as its base, so what would cause the underestimation in this case? Would RLPD work if initialised to the policy learned offline, without adding any data collected online?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* There is a significant decrease in the efficacy of many finetuning algorithms when offline data is not retained, and this appears to be the first work to focus on this issue\n* The experiments supporting the hypothesis of Q-value divergence over the offline data distribution are convincing\n* The results are proven over many environments\n* Clear layout and explanations", "weaknesses": "* No investigation of algorithm performance over varying levels of expertise of the offline pre-trained policy - it would have been interesting to see WSRL results when initialized from pre-trained policies of different quality levels or trained for varying numbers of steps.\n* A comparison of RLPD initialised with prior policy, but without the additional online data should have been included as a baseline.\n\nMinor\nLine 84: an warmup\n\nLine 207: on which we measure metrics on\n\nLine 253: onto the previous action pair that the TD error (?)\n\nFigure 4: Mixed, complete, partial (what does this refer to?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)