Online Pre-Training for Offline-to-Online Reinforcement Learning
Authors: Yongjae Shin
Jeonghye Kim
Whiyoung Jung
Sunghoon Hong
Deunsol Yoon
Youngsoo Jang
Geon-Hyeong Kim
Jongseong Chae
Youngchul Sung
Kanghoon Lee
Woohyung Lim
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Offline-to-Online Reinforcement Learning
Online Pre-Training
Online Fine-Tuning
Primary area: reinforcement_learning
TL;DR: Offline-to-online RL often struggles with fine-tuning the offline pre-trained agents, we propose OPT, a novel pre-training phase that mitigates this issue and demonstrates strong performance.

Abstract: Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, recent studies reveal that offline pre-trained agents often underperform during online fine-tuning due to inaccurate value estimation caused by distribution shift, with random initialization proving more effective in certain cases. In this work, we propose a novel method, Online Pre-Training for Offline-to-Online RL (OPT), explicitly designed to address the issue of inaccurate value estimation in offline pre-trained agents. OPT introduces a new learning phase, Online Pre-Training, which allows the training of a new value function tailored specifically for effective online fine-tuning. Implementation of OPT on TD3 and SPOT demonstrates an average 30\% improvement in performance across a wide range of D4RL environments, including MuJoCo, Antmaze, and Adroit.

Online Pre- Tra-Train for Offline-to-Online Reinforcement Learning
Yongjae Shin
Jeonghye Kim
Whiyoung Jung'
Sunghoon Hong' Deunsol Yoon
Youngsoo Jang'
Geonhyeong Kim
Jongseong Chae?
Youngehul Sung?
Kanghoon Lee
Woohyung Lim
Abstract
waalker2d-randoma
antmaze-umare diaeer---maa---deee--2 diverse-v2
Normalized Score
100
80
(fam
80
Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, recent studies re- veal that offline pre-trained agents often under- perform during online fine-tuning due inaccu- rate value estimation caused by distribution shift, with random initialization proving more effec- tive in certain cases. In this work, we propose a novel method, Online Pre-Training for Offline-to- Online RL (OPT), explicitly designed to address the issue of inaccurate value estimation in offline pre-trained agents. OPT introduces a new learn- ing phase, Online Pre-Training, which allows the training of cally for effective online fine-tuning. Implemen- tation of OPT on TD3 and SPOT demonstrates an average 30% improvement in performance across a wide range of D4RL environments, including MuloCo, Antmaze, and Adroit.
Normllzed
6D
TD3 lothie-on))
4D
40
SPOT+OPT tours)
20
SPOT (fro scratch)
SPOT (othlo
Environment Steps (10s)
Environment Steps 110*)
(a)
(b)
Figure 1. Comparison between offline-to-online RI. (yellow), from
scratch (green), and our method (purple). (a): A TD3+BC (Fuji- moto & Gu 202 1) pre-trained agent is fine-tuned with TD3 (Fuji- moto al. 2018). (b): A SPOT (Wu et al. 2022) pre-trained agent is fine-tuned with the same algorithm.
overcome the cost challenge of online RL and the per- formance limitation of offline RL, the offline-to-online RL approach has been introduced (Lee et al. 2022; Zhang et al. 2023; Yu & Zhang 2023). This approach entails training agent sufficiently on an offline dataset, followed by fine- tuning through additional interactions with the environment. The offline phase allows the agent to acquire a reasonable ini- tial policy without any online cost, providing a stable foun- dation for subsequent fine-tuning. Combining the strengths of both approaches, for extensive environment interactions, while enhancing the agent's 's performance through online fine-tuning.
value function tailored specifi-
RL reduces the need
1. Introduction
Although offline-to-online RL offers clear advantages, prior studies (Zhang et al. 2023; Guo et al. 2023; Nakamoto et al. 2024; Zhang et al. 2024; Kong et al. 2024; Hu et al. 2024; Luo et 2024) have shown that fine tne-ttning an offline pre- trained agent often results in worse performance compared to training from scratch. This phenomenon, described by Nakamoto et al. (2024) as counter-intuitte trends, is de- picted in Figure 1, which compares the learning curves during the online phase for both fine-tuning (off-to-on) and training from scratch with the replay buffer initialized using the offline dataset (from scratch) As shown in Figure I (a), training from scratch outperforms fine-tuning from the very beginning of the learning process, also observed by Ball et al. (2023). In Figure (b), although fine-tuning demonstrates moderate performance in the early stages, is eventually surpassed by training from seratch. This highlights the inherent challenges in fine-tuning pre- trained agents in offline-to-onling settings.
Reinforcement Learning (RL) has shown great potential in addressing complex decision-making tasks across various fields (Mnih et al. 2015; Silver et al. 2017). In particular, offline RL (Levine et al. 2020) offers the advantage of train- ing an agent on the fixed dataset, thereby mitigating the potential costs or risks associated with direct interactions in real-world environments significant limitation of online RL. However, the effectivvness of offline RL is inherently constrained by the quality of the dataset, which can impede the overall performance.
!oork done during internship at LG Al Research. 'LG AI Research, Scoul, Republic of Korea. "School of Electrical Engi- necring, KAIST, Daejcon, Republic of Korea. Correspondence to: Woohyung Lim <w.lim @ Igrescarch.ai>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Online Pre-Training for Offline-to-Olline Reinforcement Learning
Previous studies (Nakamoto al. 2024; Zhang et al. 2024) attribute this counter-intuttive trends of online fine-tuning to issues stemming from inaccurate value estimation. In response, Nakamoto et al. (2024) focuses on providing a lower bound for value updates, applied from the offline phase onward to correct value estimates throughout learning. Meanwhile, Zhang et al. (2024) introduces perturbations to value updates, aiming stabilize learning by promot- ing smoother value estimation. One notable characteristic shared by these approaches is their reliance on the single value function across both the offline and online learning phases. In contrast, we adopt a different approach by in- troducing and utilizing an entirely new value function. To investigate this, we formulate two key research questions that guide our exploration:
r, following the transition probability p(d'|s, a) and reward function r(s, a), respectively. Over successive interactions, the agent's policy x is optimized to maximize the expected cumulative return Ex 1E, 7rr(s,, 0z)], where E [0, is the discount factor.
Offline RL focuses on training agents using a static dataset D = {(s, a, r, s')}, usually generated by various policies. To address the constraints of offline RL, which are often limited by the quality of the dataset, offline-to-onie introduces an additional phase of online fine-tuning. In this framework, an offline pre-trained agent continues learning through online fine-tuning enabling further improvement beyond what the offline dataset alone can offer. This hybrid paradigm allows the agent to adapt and improve by effec- tively leveraging new experiences collected during online interaction.
RL
Q1. "Can adding a new value function resolve the issue of slow performance improvement?
Addressing Inaccurate Value Estimation. Offline RL., re- liant on a fixed dataset, is prone to extrapolation error when the value function evaluates out-of diitt---ditiiittinnnn (OOD) ac- tions (Kumar et al. 2020; Fujimoto et al. 2019; Kim et al. 2024). Several methods have been proposed to address this challenge: some focus on training the value function to as- sign lower values to OOD actions (Wu et al. 2019; Kostrikov et al. 2021a), while others aim to avoid OOD action evalua- tion altogether (Kostrikov et al. 2021b). These inaccurate value estimations in offline training not only degrade of- fline performance but can also adversely impact subsequent online fine-tuning ( YYu & Zhang 2023; Zhang et al. 2024; Nakamoto et al. 2024; Kong et al. 2024; Feng et al. 2024; Luo al. 2024; Zhou al. 2025). Although constraints applied in offline RL can be extended to online fine-tuning to mitigate this issue (Kostrikov et al. 2021b; Lyu et al. 2022; Wu et al. 2022), such strategies often impose exces- sive conservatism, limiting the potential for performance enhancement.
Q2. " "How can we best leverage the new value function?"
Through a comprehensive analysis of these research ques- tions, we propose Online Pre-Traie treaniain for Offline-to-Onling RL (OPT), a novel approach that introduces a new value function to leverage it during online fine-tuning. In response to the first research question, OPT introduces new value function, enhancing overall performance, as illustrated in Figure 1. Rather than relying solely on a value function trained in the offine phase, this additional value function allows the agent to better adapt to the changing dynam- ics of the online setting. For the second research question, OPT incorporates an additional learning phase, termed On- line Pre-Training, which focuses on learning this new value function. This value function is then utilized during online fine-tuming to guide policy learning, ensuring more effective performance improvements.
OPT consistently demonstrates strong performance across various D4RL environments within a limited setting of 300k online interactions, surpassing previous tate-offthe-aan re- sults. Additionally, since OPT introduces a new value func- tion as one of its key components, it can be broadly inte- grated into existing value function-based RL algorithms. This design enables OPT to serve as a general enhancement module, compatible with a wide range of existing meth- ods. We empirically verify this versatility through extensive experiments.
Recent advances in offline-to-online RL have aimed over- come this limitation arising from inaccurate value estima- tions (Nakamoto et al. 2024; Zhang et al. 2024; Luo et al. 2024). One approach (Nakamoto et al. 2024) addresses over- conservatism during the offline phase by providing a lower bound for value updates. However, the conservative nature of this method continues to hinder policy improvement. An- other approach (Zhang et al. 2024) introduces perturbations to value updates and increases their update frequency. While effective, this method incurs significantly higher computa- tional costs compared to standard techniques, making it less practical for general use.
2. Background and Related Work
Backbone Algorithms. Our proposed Online Pre- Tre-Training process using newly introduced value function can be ap- plied to various backbone algorithms. Among the many potential candidates, this study utilizes TD3+BC (Fujimoto & Gu 2021) and SPOT (Wu et al. 2022) as the backbone
Reinforcement Learning (RL) is modeled as Markov Deci- sion Process (MDP) (Puterman 1990). In this framework, at each time step, an agent selects an action a based on its cur- rent state s according to its policy (a)s)) The environment transitions to subsequent state s' and provides reward
2

---

Online Pre-Training for Offline-to-Olinee Reinforcement Learning
Offline Pre-Training
Online Fine-Tuning
Agent
Online Buffer
Agent
Y
In learning
Frozen
Train
Train
Offline Dataset
Offline Dataset
(a) Conventional Offline-to-Online RL
Offline Pre-Training
Online Pre-Training
Online Fine-Tuning
Agent
Agent
Online Buffer
Agent
L Offfpe
W Quii pt
Train
Train
Train
Offline Dataset
Offline Dataset
Qutrpt
Offline Dataset
π
(b) Ours: Online Pre- Training for Offline-to-Online RI.
Figure Illustrations of two different learning methods: (a) Conventional Ooffiieett-Onlinn RL (b) Ours. OPT introduces new learning phase, termed Online Pre-Traimne Pre-Training, between offline pre-training and online fine-tuning. The illustration indicates whether the value function and policy are in learning or frozen (not being trained) during each training phase.
algorithms due to their simplicity and sample efficiency. TD3+BC extends the original TD3 (Fujimoto et al. 2018) algorithm by incorporating a behavior cloning (BC) reg- ularization term into the policy improvement. The value function is trained using femporal-difference (TD) learning, with the loss functions for both the policy (s)s) and value function Q0(8, a) defined as follows:
ii Offline Pre- Fre-Training: Following the conventional offline-to-online RL, the offline pre-trained value function Q and policy = are jointly trained on the offline dataset.
(ii) Online Pre-Training: The newly initialized value function, QeeRr, is trained on both the offline dataset and newly collected online samples.
(Ø) Eg Coc-Bl-Qosssaa (T()) ((T)) a)3].
(iii) Online Fine-Tuning: The policy is updated by utiliz- ing both Q and Qee with each value function continuously updated.
( = B(aom.))-
-(r - (
where Q7 denotes delayed target value function and B is the replay buffer. SPOT extends of TD3+BC by replacing the BC regularization term with pre-trained VAE, which is then used to penalize OOD actions based on the uncertainty.
Figure illustrates the difference between conventional offline-to-online learning (Figure 2a) and our proposed method, OPT (Figure 2b). OPT introduces a new learning phase called Online Pre-Training, making it distinct from the conventional two-stage offlnneto-online RI. methods by comprising three stages. The following sections focus on the Online Pre-Training and Online Fine- Fine-Tuning phases, while the offline pre- pre-traiie vee-rraining phase adheres to the standard offline RL process.
3. Method
In this section, we introduce Online Pre- Training for Offline- to-Online RL (OPT), novel method aimed at addressing the issue of inaccurate value estimation in offline-to-onlinn RL. The proposed method employs two value functions, Qoft-pe and Q each serving distinct roles the time domain. The method consists of three distinct stages of learning:
3.1. Online Pre- Tre-Training
In the proposed method, Q'n-Pt is introduced as an additional value function specifically designed for online fine-tuning.
3

---

Online Pre-Training for Offline-to-Online Reinforcement Learning
Considering Qon-pi, one straightforward approach is add randomly initialized value function. In this case, Qanpp begins learning from the online fine-tuning, it is expected to adapt well to the new data encountered during online fine-tuning. However, since Qan-pt is required to train from scratch, it often disrupts policy learning in the early stages. To address the potential negative effects that can arise from adding Q with random initialization, we introduce a pre-training phase, termed Online Pre-Training, specifically designed to train QoDP! in advance of online fine-tuning. The following sections explore the design of the Online Pre-Tree rre-Training method in detail.
Algorithm 1 OPT: Online Pre-Traing Pre-Training for Offline-to-
Online Reinforcement Learning
1: Inputs: Offline dataset Bott. offline trained agent {Q To), online pre-training samples N,, online pre-training steps Npremain, online fine-tuning steps
Nuacume, weighting coefficient K
Initialize online replay buffer Bons value function Q
// Online Pre- Pre-Training
3: Store N₂ transitions ment interaction with TT
(8, (,,))) in Bun via environ-
4: for i = 1 to Npretrin do
5: Sample minibatch of transitions {T))=1 ~ Botf.
Designing Datasets. As the initial stage of Online Pre- Training, the only available dataset to train Q is the offline dataset Bar. Yet, given that Boft has already been extensively utilized train Qufffq further relying solely on this dataset inevitably causes Qom-pi to closely replicate Qutt-pe To address this, we incorporate online samples by initiating Online Pre-Training with the collection of N₂ samples in the online buffer Ban. generated by -प To leverage Bum: one approach to train Qon-pe with Bun- Since Qun-pi is trained based on ~oll" this prevents Quare from disrupting policy learning in the initial stage of online fine- tuning. However, as Bon is generated by the fixed policy "ff relying solely on Bon risks limiting the generalizability of Qeappi to diverse online samples. Therefore, to address both issue of similarity to Q pa and the risk of overfitting to ooll a balanced approach utilizing both Bor and Bon is necessary during training.
{ Boo
6: Update eb minimizing Q (v) by Equation (3)
7: end for
// Online Fine-Tuning
Initialize balanced replay buffer BER <- Boll U Ban
for 1 to Nineuuee do
10: Sample minibatch of transitions BBR
11: Update @ and &o minimizing L (0) and Lq (16)
respectively by Equation (2)
12: Update & minimizing Clinetune (Φ) by Equation (4)
13: end for
the Online Pre-Training, only Q is updated, with no alterations made to other components, π and Qoil-p
3.2. Online Fine-Tuning
Designing Objective Function. By leveraging both datasets, the objective for Qen-pt is to ensure its adaptability to the evolving policy samples, promoting continuous policy improvement and enhancing sample efficiency. To achieve this, we propose a meta-adaptation objective inspired by the OEMA (Guo et al. 2023), tailored to our setting. The objective function for Qon.p is outlined as follows:
As Qon-pt is trained during Online Pre-Training effectively adapt to online samples, we leverage it in the online fine- tuning to guide policy learning. Throughout this phase, the buffer Bon is continuously filled and the learning of all three components Qofff-tt Q00- and T progresses. As in the conventional offline-tooonline RL, Qolf-pt is updated using TD learning, and similarly, Qummrs, which was trained by Equation (3) during Online Pre-Training is also updated via TD learning.
Quaqs (w))
Coff Quep (w² + Con CQua( (v- - a Coff
Quaa (V))
Effectively Balancing Qe and Q One of the key aspects of OPT lies in its policy improvement strategy, which, unlike most previous approaches that rely on a single value function, effectively balances both Qoff-pt and Qo0-pt for policy improvement. Since Qual-rr provides reliable in- formation about the offline dataset, and QTT-Mr trained via anaa-adapto strategy during Online Pre-Training, is well-suited for adapting to newly collected data, we propose a policy learning strategy that effectively exploits the com- plementary strengths of both value functions. The proposed loss function for policy improvement given by:
where
Coff (w) E. R. Bar (Q (s.
- -rr + 10 00 (6', 10
(T(s'))))
Lq (1b) = E. 1Q (8,
(r 10+ 100- , TT()))))).
Here, 2% represents the target network. Equation (3) consists of two terms: the first term facilitates learning from Bor, while the second term serves as an objective to ensure that Qmnm"i adapts to Bon- Optimizing these terms allows Qon-pt to leverage Boff and Bmn to align closely with the dynamics of the current policy, thereby enabling efficient adaptation to online samples during fine-tuning. During
finctune (x) = EE- | {(1 xx) x)) (s, TT())) TT
+ mQ T (T)))))))))
(4)

---

Online Pre-Training for Offline-to-Oniing Reinforcement Learning
Table 1. Comparison of the normalized scores after online tne-tuning for each environment in MuloCo domain. r = random, medium, m-r = medium-replay. All results are reported as the mean and 95% confidence interval across ten random seeds.
Environment
TD3
Off2On
OEMA
PEX
ACA
FamO20
Cal-QL
TD3 OPT (Ours)
halfchectah-r
94,6227
92.8=35
83.8+47
62.3=35
91,0+16
38.7+24
32.2=45
90.2+16
hopper-r
86.0=13a
94.5=52
59.8+134
46.5=m24
84.6
12.0+13
10.3=11.7
108.7:117
walker2d-r
0.1101
29.4
16.7+77
10.7
38.8
9.9
10,9
88.0
halfcheetah-m
93.4
103.3
67.6=116
69,4
81.0#06
49.8+02
69.9=5.2
97.0=00
hopper-m
14.
10
107.1
84,4
101.6
74.4155
102.3
112.2::0.6
walker2d-m
103,5
112.3
98.9+4.1
84,5=1099
19.9
84,4+14
96,1=35
116.1
halfcheetah-m--r
87.2.109
97.7=1.7
18.5
55.9
67.6112
48.1
64.821.
94.3:11
hopper-m-r
94.5=11.3
103.7
108.8+08
85,8
104.7=222
98. 38.5
101.8
112.7
walker2d-m-r
103.9266
118.8
103.4129
89.3
19.
88 88.61666
98,5214
119.9
Total
752.4
860.9
694.6
588.8
738.5
504,4
586.8
939.1
Table Comparison of the normalized scores after online fine-tuning for each environment in Antmaze domain. All results are reported the mean and 95% confidence interval across ten random seeds.
Environment
SPOT
PEX
ACA
FamO20
Cal-QL
SPOT + OPT (Ours)
umaze
98.7+a8
95.6=0.8
95.0+17
93,0+16
90.8
99.7+t02
umaze-diverse
55.94159
30.7=177
94.5
43.5+14.
75.2=159
97.7100
medium-play
91.1#223
85.0=2.1
0.0+ao
89.3+21
94.6=29
97.6+kox
medium-diverse
92.02117
85.4=2.1
0.0
77.9
96.2=20
98.7106
large-play
58.0+11.4
57.5
0.0+a.
55.8+5.0
73.7
81.5
large-diverse
70.0+11.
58.3=47
0.0
56.8244
72.9
90.1
Total
465.7
412.5
189.5
416.3
503.4
565.3
where Ty is initialized as ofl and < K S 1. is weight- ing coefficient that regulates the balance between Qual-r and Qen-pr. When the discrepancy between the offline dataset and newly collected online samples is small, Qual- remains reliable, making it suitable for guiding policy improvement. Accordingly, smaller x values applied to more effectively leverage Qoff-pr As online fine-tuning progresses, Qon-pi, optimized through our meta-adaptation objective during On- line Pre-Training, quickly adapts to the online data. To leverage this adaptability, x is incrementally increased to shift the reliance towards the more rapidly adapting Qrm--r.. In summary, as online fine-tuning progresses, we schedule k to progressively shift the influence on policy learning from Quil-pi to Qon-pt, This dynamic adjustment ensures that the policy is always guided by the most informative value estimate at each stage of training.
4. Experiments
In this section, we demonstrate the effectiveness of our proposed method through experimental results. Section 1,1, we deseribe our experimental setup and compare our method against existing offline-to-onlin RL approaches across various environments. Section 4.2 examines the ap- plicability and performance of our method when applied to an alternative value- based algorithm.
4.1. Main Results
Experimental Setup. We evaluate the performance of OPT across three domains from the D4RI. benchmark (Fu et al. 2020). MuJoCo is a suite of locomotion tasks including datasets of diverse quality for each environment. Antmaze is set of navigation tasks where ant robot is controlled to navigate from starting point goal location within maze. Adroit is a set of robot manipulation tasks that require controlling a five-finger robotic hand to achieve a specific goal in each task. A detailed deseription of the environment and dataset is provided in Appendix B. For all baselines, the offline phase comprises 1M gradient steps, and the online phase consists of 300k environment steps. OPT carries out Online Pre- Pre-Training for the first 25k steps of the online phase, followed by online fine-tuning for the
Additionally, to promote the use of online samples, we employ balanced replay (Lee et al. 2022), which prioritizes samples encountered during online interactions to further accelerate the adaptation of Qon-p The overall learning phase of OPT is illustrated Figure 2b, with the algorithm presented in Algorithm 1. The implementation details are presented in Appendix 1, and the hyperparameter settings are summarized in Appendix H.
5

---

Online Pre-Training for Offline-to-Online Reinforcement Learning
Table 3. Comparison of the normalized scores after online fine- tuning for each environment in Adroit domain. All results are reported as the mean and 95% confidence interval across ten ran- dom seeds.
IQM
IQM
TD3
SPOT
Off2On
PEX
OEMA
PEX
ACA
Environment
SPOT
Cal-QI.
SPOT OPT ((mm)
ACA
FamO20
FamO20
pen-cloned
114,8+63
0.21+32
136.2+48
Cal-QL
Cal-QL
hammer-cloned
84.0+128
0.23+a08
121.9+16
TD3+OPT
SPOT+OPT
door-cloned
1.6227
-0.332000
51.1=129
relocate-cloned
0000
-0.342000
-0.06=0m
75 90 105
40 60 800
Total
200.2
-0.23
309.1
(a) MuloCo
(b) Antmaze
Figure 3. Interquartile Mean (IQM) comparison of baseline meth- ods on the MuloCo and Antmaze domains. The x-axis represents normalized scores. All results are based on 10 random seeds.
remaining 275k steps thus, like the other baselines, it also has a total 300k environment steps in the online phase. The implementation details are provided in Appendix A.
performance this domain. Table 2 shows that OPT con- sistently delivers superior performance across environ- ments. When comparing to Cal-QL (Nakamoto et al. 2024), recently proposed method with the same objective of addressing inaccurate value estimation, the results in the mmazzediveese and large-de verse environments demonstrate the efficacy of introducing new value function through Online Tre-Training in mitigating this issue.
Baselines. We compare OPT with the following baselines: method
(I) Off2On (Lee al. 2022), an ensemble-base that incorporates balanced replay to promote the use of near- on-policy samples; (2) OEMA (Guo et al. 2023), which applies an optimistic exploration strategy alongside a meta- adaptation method for policy learning: (3) PEX (Zhang et al. 2023), which utilizes a set of policies, including a frozen offline pre-trained policy and an additional learnable policy, designed to balance exploration and exploitation during online fine-tuning; (4) ACA (Yu & Zhang 2023), which post-processes the offline pre-trained value function to align it with the policy, where the post-processing is specifically applied at the transition point between offline pre-training and online fine-tuning; (5) FamO20 (Wang et al. 2024), which employs adaativ policy improvement method by leveraging both a balance model and a universal model to generalize across diverse state distributions; and (6) Cal-QL. (Nakamoto al. 2024), which trains value function with an explicit lower bound constraint to mitigate over-conservatism introduced during the offline pre-training phase.
Results on Adroit. In the Adroit, as with Antmaze, we apply OPT to the SPOT. Table 3 demonstrates that, unlike Cal-QL (Nakamoto et al. 2024), which struggles to learn from low-quality datasets such as c oned due to its con- servative nature, OPT manages to perform well even with these challenging datasets. In particular, the results in the door cl cloned environment, where SPOT fails to perform adequately, demonstrate the effectiveness of OPT.
Overall Findings The results across all domains indi- cate that OPT consistently achieves strong performance, demonstrating its versatility and effectiveness. Notably, while Off2On (Lee et al. 2022) performs competitively in the MuloCo domain, prior studies (Li et al. 2023; Lei et al. 2023) have mentioned its struggles in Antmaze-large and Adroit. In contrast, OPT demonstrates superior performance in Antmaze and Adroit, underscoring its robustness and ability to generalize across diverse domains and dataset con- ditions. The individual learning curves can be found in Appendix I.
Results on MuJoCo. In the MuJoCo, we implement OPT on the baseline which utilizes TD3+BC for the offline phase, followed by TD3 in the online phase. To further improve sample efficiency during online fine-tuning, we adopt update-to-data (UTD) ratio of 5 for TD3+OPT and apply the same setting to TD3 to ensure fair comparison. The results in Table 1 indicate that OPT demonstrates strong overall performance, notably surpassing the existing state- of-the-art (SOTA) in several environments. Its consistently high scores across different datasets highlight the robustness of OPT in handling diverse dynamics and data distributions. In particular, the results for the wa walker2d Seterddrrndom--- dataset demonstrate the remarkable efficacy of OPT, as it significantly outperforms existing approaches.
To further support the statistical significance of our results, we report Interquartile Mean (IQM) (Agarwal et al. 2021) scores, which offer a more reliable metric by averaging performance within the 25th to 75th percentile. IQM re- duces the influence of extreme outliers and better captures consistent trends across tasks. As shown in Figure 3, OPT achieves superior performance compared to other baselines, with non-overlapping 95% confidence intervals, thereby demonstrating the statistical robustness of our results. These findings provide additional evidence that OPT not only per-
Results on Antmaze. In the Antmaze, we implement OPT within the SPOT, as TD3+BC showed suboptimal
6

---

Online Pre-Training for Culline-to-OOline Reinforcement Learning
Antmaze
110
Normalized Score
85
IQL
Normaliiee Score
105
80
IQL+OPT
100
75
70
95
65
90
OPT (ours)
60
85
Pre-trained with 8,
Random Initialization
55
80
fandom
medium
medium -replay
Total
0
2
Environment Steps (10")
Figure 5. Comparing normalized score of OPT with different ini- tialization methods, averaging across all 3 environments the MuloCo domain.
Figure 4. Aggregated return curves for IQL and IQL+OPT, aver- aged across all six environments in the Antmaze domain.
Section 5.6 presents a comparison between OPT and RLPD. Additional experiments and analyses are provided Ap- pendix and Appendix D.
forms well average but also does consistently across runs and environments.
4.2. Extending OPT to IQL
5.1. Effect of Initialization Methods for Qon-pt
So far, have shown that OPT is effective algorithm for online fine-tuning when applied to TD3-based algorithms, such as TD3 and SPOT. To further validate the versatility of OPT, we changed the backbone algorithm to IQL (Kostrikov et al. 2021b). This transition allows us to examine whether OPT remains effective when applied algorithms with fundamentally different policy and value function structures. Unlike TD3-based algorithms, which employ deterministic policy and rely solely state-action value function, IQL employs stochastic policy and uses both a state-action value function and state value function. Accordingly, to implement OPT, we modified the training loss, with detailed explanations provided in Appendix A.2.
In Section 3.1, to best leverage Q during online fine- tuning, explore various approaches for initializing Q To further substantiate the effectiveness of initialization method through experimental results, evaluate three ap- proaches for initializing Qon-pt; (1) Random Initialization, (2) Pre-trained with Bon. and (3) our proposed method, OPT. (1) Random Initialization initializes QInppt from scratch and uses it directly at the beginning online fine-tuning, with- out any Online Pre- Training. Qon-pt during the Online Pre-Training using only samples from Bon and the loss objective - (1)) from Equation (3), and then uses it for online fine-tuning.
(2) Pre-trained with Bon trains
Figure 5 presents the average performance of the three ini- tialization approaches across different datasets. illustrates that random initialization consistently underperforms com- pared to OPT. This discrepancy, discussed in Section 3.1, be attributed to the fact that random initialization hin- ders policy learning from the outset, leading to suboptimal value estimates and ultimately degraded performance. Simi- larly, the results for pre-training with Bo demonstrate that learning solely from Bon. is insufficient to follow the perfor- mance OPT. In particular, the random dataset, where the policy evolves more drastically (Figure 6), highlights that an overfitted Qun-pi, trained only on Bon. not only fails adapt to this policy improvement but in some cases performs worse than random initialization. This underscores the im- portance of both Boff and Bmn for training. Detailed experi- mental results, including an additional approach and results on the Antmaze domain, are provided Appendix C.3.
The results in Figure show that applying OPT yields clear and noticeable performance improvement compared to the backbone algorithm. Notably, while IQL exhibits a gradual and steady performance increase, IQL+OPT demonstrates more rapid and substantial improvement. These results indi- cate that OPT is not limited to specific backbone algorithm but can be effectively integrated into various value-based methods, serving as a general strategy for enhancing perfor- mance. Additional experiment results in different domains, including MuJoCo and Adroit, are presented in Appendix E.
5. Discussion
In this section, we provide a detailed analysis of key compo- nents of OPT. Section 5.1 investigates the effect of different initialization strategies for Qen-Pt, Section 5.2 explores how the characteristics of offline datasets influence the schedul- ing of к. Section 5.3 examines the role of the additional value function. Section 5.4 examines the impact of the k scheduling strategy. Section 5.5 focuses on the effect of the number of Online Pre-Training samples (Nz). Finally,
5.2. Influence of Dataset Characteristics on Scheduling
In our proposed method, we utilize k to balance the con- tribution of Qoff-pt and Qon-pt during online fine-tuning. To
7

---

Online Pre- Pre-Training for Ofllinc-to-nline Reinforcement Learning
Bull
Biit
Bina
110
Normalized Score
90
80
(ours) w/o Q Q
70
60
random medium medium
Total
-replay
(a) Random
(b) Medium
(c) Medium-Replaa
Figure 7. Comparison of the normalized score for OPT and its ablation (without the addition a new value function), averaged across all 3 environments in the MuJoCo domain.
Figure t-SNE visualization of the offline dataset (Bar) and the rollout samples of the policy at the beginning (Bmi) and end (Broa) of the online fine-tuning.
Table 4. Ablation study results on x in the Antmaze domain. All values are reported as the mcan and 95% confidence interval over 5 random seeds.
effectively leverage the strengths of both value funetions, we adjust the и schedule based on the quality of the dataset, as different datasets offer varying levels of reliability for each value function. To provide insight into the rationale behind the scheduling of K, we visualize the distributional differences between the offline dataset and policy-gemm geiee-egeee samples. Specifically, we compare the state-action distri- butions of the offiin dataset and those of the policy roll- outs using t-SNE (Van der Maaten & Hinton 2008) in the walker2d environment.
Environment
0.1 0.9
0.1 0.8
0.1 0.7
0.5
umaze umaze-diverse
99.8103
100.0100
99.62a7
99.0206
97.4103
96.821a
95.2211
97.0a21
medium-plan play
98,2+11
97.6+14
98. 98.2+11
96.0s1a
medium--iverse
98,4
97.6213
96.6221
97.0
large-play
78.2
78.3239
81.0264
59.2
large-diverse
90.6132
90.0a13
88.4
75.4
Figure shows the comparison of the distribution between the offline dataset (Bor) and the rollout samples of policy both the beginning (Bini) and the end (Biia)) of the on- line fine-tuning phase. For the medium and medium-replay datasets, we observe that the distributions start of the online fine-tuning but diverse towards the end. In contrast, for the random dataset, a difference between the two distributions evident from the beginning. For the medium and medium-replay datasets, since Qoff-pt is infor- mative for the offline dataset due to offline pre-training, we initially assign it higher weight during the early stages of online fine-tuning. As the online fine-tuning progresses, the weight is gradually shifted toward Qea-pt However, for the random dataset, due to the substantial distribution dif- ference, we primarily rely on QQ""!i from the start of online fine-tuning tine-tunigg We present additional experiments related to the sensitivity of & in Appendix C.2 and provide the specific values of x in Appendix H.
in online fine-tuning is driven solely by Qoff-pt.
The results presented Figure indicate that the addition of new value function leads improved performance regardless of the dataset. Notably, this improvement is pro- nounced in the random dataset. Due to the characteristics of the random dataset, where contains few successful demon- strations, Qotl-p' becomes heavily biased. As result, it fails to benefit from Online Pre- Pre-Training and performs poorly during online fine-tuning. Detailed experiment results are provided in Appendix C.5.
similar at the
5.4. Impact of the x. Scheduling Strategy
To effectively leverage both Qullipi and Q!" during online fine-tum fine-tuning, Section 3.2 introduces weighted combination of the value functions, controlled by scheduling coef- ficient K. The value of H. is scheduled change throughout online fine-tuning, gradually shifting the weighting from Qu to Qen-pr To evaluate the impact of this scheduling strategy, we compare our proposed dynamic schedule (from K 0.1 to 0.9) with a baseline using a fixed weighting of K = 0.5. We also investigate the sensitivity of performance to the final value of x by testing alternative schedules with different endpoints.
5.3. Contribution of the Additional Value Function
key component our proposed algorithm is the intro- duction of a new value function, which is trained during the Online Pre-Training phase and subsequently utilized in online fine-tuning. To evaluate the impact of this addition, we examine the performance when the new value function is excluded. Specifically, we assess the outcome where Qall is trained during Online Pre- Pre-Training using the objective in Equation (3), without introducing Q (denoted as w/o Qon-pt in Figure 7). Under this setup, policy improvement
As shown in Table 4, the use of scheduling leads to improved performance in relatively challenging environments such as Antmaze-large This improvement suggests that gradually shifting supervision allows the agent to better adapt to on- line data distributions while maintaining stability early in
8

---

Online Pre-Training for Oflline-to-nline Reinforcement Learning
110
Table 5. Normalized score for each environment the MuloCo domain. The full results for other domains, including Antmaze and Adroit, provided in Appendix F.
Normalized Score
105
100
Environment
RLPD
(ours) baseline
Vanilla
Off-to-On
OPT (Ouns)
85
halfcheetah-r
91.5 #2.5
96.1
90.7
80
6250
12500
25000 50000 100000
hopper-r
90.2 #19.1
95.7 118.4
103.5 #3.6
walker2d-r
87.7 #14.1
74.3 13.9
79.2 4 10.0
halfchcetah-m
95.5 #1.6
96.6 #0.0
96.7
Figure 8. Comparing normalized score with varying N, of OPT, averaged across all 9 environments (3 tasks with datasets each) in the MuloCo domain.
hopper-m
91.4 #27.8
93.6 133.9
106.9 #1.5
walker2d-m
121.6 #2.3
124.1 42.4
122.8 4:300
halfcheetah-me
90.1 #1.3
90.0 #1.4
91.6 #2.1
hopper-m-r
78.9 2:4.5
94.7 26.8
107.4 1:1.
walker2d-m-r
119.0 #2.2
122.5 #2.7
120.9 #2.3
training. Notably, the results suggest that performance is not particularly sensitive to the exact value of и; rather, the key factor is the presence of gradual transition from Qo to Qon-pt Additional results using fixed x values of x. 0 and = 1 are provided in Appendix C.
MuJoCo total
866.0
887.6
918.7
demonstrated wate-of-the-e performances through the use of ensemble techniques and high UTD ratio. To assess the performance of integrating OPT with RLPD, extend RLPD by incorporating an offline phase, followed by online fine- fine-tuning, and subsequently apply OPT to evaluate its ef- fectiveness. Further implementation details are provided in Appendix A.3.
5.5. Effect of Online Pre-Training Sample Size (N+)
The Online Pre-Training phase involves two hyperparame- ters: Nr and NNpretrinn In particular, N, represents interac- tions with the environment used to collect online samples for Online Pre-Training, Since N, is also part of the envi- ronment steps within the online phase, determining the most efficient value for N, is critical for sample efficiency in dffline-to-onling RL. To investigate this, we conduct experi- ments in the MuJoCo domain comparing the results when N, is set to 1/4, 1/2, 2, and 4 times its current value (25k). In these experiments, N Nprettai is set to twice Nr, consistent with the setting proposed in our method, while the number of online fine-tuning steps is kept constant.
Table 5 reports the results of original RLPD ('Vanilla'), RLPD with an additional offline phase ('Off-to-On'), ) and RLPD combined with OPT ('OPT"). ). The experimental re- sults demonstrate that integrating OPT with RLPD leads to performance improvements, surpassing both the baseline methods. These results indicate that OPT is an effective algorithm capable of enhancing performance when applied to existing state-of-the-ee algorithms.
Figure 12 demonstrates that increasing N, enhances the effectiveness of Online Pre-Training However, beyond certain threshold, further increases do not lead additional performance gains. During environment interactions for N,, the policy remained fixed, and once online data surpasses optimal level, it no longer contributes to Online Pre- TreTrrinnig, Since offline-to-online RL aims for high performance with minimal interaction, these results suggest OPT is most ef- ficient when N, is set 25k. We adopt N, 25k and N Npretrain 50k across all domains. Appendix C.4 presents detailed per-environment results.
6. Conclusion
In this work, we propose Online Pre- Training for Offline- to-Online RL (OPT), a novel approach designed to improve the fine-tuning of offline pre-trained agents. OPT introduces an additional learning phase, Online Pre-Training, which enables the training of a new value function prior to on- line fine-tuning. By leveraging this newly learned value function, OPT facilitates more efficient fine-tuning pro- cess, effectively mitigating the performance degradation commonly observed in conventional offline-to-onling RL approaches. Through extensive experiments across various D4RL environments, we demonstrate that OPT consistently outperforms existing methods and exhibits versatility across different backbone algorithms. These results highlight OPT a robust and effective solution for enhancing performance in off to-ine RI. Our findings further emphasize in- corporating a new value function and leveraging it through Online Pre- Training to improve fine-tuning efficiency. We believe that OPT paves the way for future research on refin- ing fine-tuning strategies in reinforcement learning.
5.6. Comparison with RLPD under Offline-to-Onliee Setting
Thus far, proposed method to integrate offline pre- trained agents into online fine-tuning effectively. Recently, several studies have emerged demonstrating strong perfor- mance using online RL alone with offline datasets, without requiring an explicit offline phase (Song et al. 2022; Ball al. 2023). Among these, RLPD (Ball al. 2023) has

---

Online Pre-Training for Oflline-to-nline Reinforcement Learning
Acknowledgements
forcement learning. IEEE Transactions on Knowledge and Data Engineering, 2023.
This work was supported by LG Al Research and partly by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2025- 00557589, Generative Model Based Efficient Reinforce- ment Learning Algorithms for Multi-modal Expansion in Generalized Environments).
Hu, H., Yang, Y., Ye, J., Wu, C., Mai, Z., Hu, Y., Lv, T., Fan, C., Zhao, Q., and Zhang, C. Bayesian design princi- ples for oflline-to-online reinforcement learning. arXiv preprint arXiv:2405.209%5 2024.
Kim, J., Lee, S., Kim, W., and Sung, Y. Adaptive g-aid for conditional supervised learning in offline reinforcement learning. Advances in Neural Information Processing Systems, 37:87104 117104487733 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Kong, R., Wu, C.,, Gao, C.-X., Zhang, Z., and Li, M. Effi- cient and stable offline-to-onling reinforcement learning via continual policy revitalization. Proceedings of the Chirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp- 4317-4325, 2024.
References
Agarwal, R., Schwarzer, M., Castro, S., Courville, A. C.,, and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural in- formation processing systems, 34:29304-29322 2021.
Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Of- fline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5783, PMLR, 2021a.
Ba, J. L. Layer normalization. arXiv: 11.116000006655 2016.
arXiv preprint
Kostrikov, L,, Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint 0tfi::2100061666 2021b.
Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Ef- ficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp- 1577-1594. PMLR, 2023.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q g-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
Feng, J., Feng, M., Song, H., Zhou, W., and Li, H. Suf: Stabilized unconstrained finettuning for offline-to-online reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 11961- 11969, 2024.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble. Conference on Robot Learn- ing. pp. 1702 1712. PMLR, 2022.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Lei, K., He, Z.,, Lu, C., Hu, K., Gao, Y., and Xu, Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-pol optimization. arXiv preprint arXiv:2311.03355 2023.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 14:20132-20144 2021.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint urXiv:2005.0144
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional conference on machine learning, pp. 1587-1596. PMLR. 2018.
2020.
Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., and Zhang, Y.- Q. Proto: Iterative policy regularized offline-to-online reinforcement learning, arXiv preprint arXiv:2305.15669 2023.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Interna- tional conference machine learning, pp. 2052-2062. PMLR. 2019.
Luo, Q.- Xie, M.-K., Wang. Y., and Huang, S.-J. Opti- mistic critic reconstruction and constrained fine-tuning for general offline-to-onliig Advances in Neural Infor- mation Processing Systems, 101167-1082077 2024.
Guo, S., Zou, L.. Chen, H., Qu, B., Chi, H., Philip, Y., and Chang, Sample efficient offline-to-online rein-
10

---

Online Pre-Training for Offline-to-Ollin Reinforcement Learning
Lyu, J., Ma, X., Li, X., and Lu, Z. Mildly conservative q- learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935, 2023.
Zhang. Y., Liu, J., Li, C., Niu, Y., Yang, Y., Liu, Y., and Ouyang, W. perspective of q-value estimation on offline-to-online reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 38, pp. 16908-1699 2024.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Zhou, Z., Peng, A., Li, Q., Levine, S., and Kumar, A. Ei- ficient online reinforcement learning fine-tuning should not retain offline data. In The Thirteenth International Conference on Learning Representations, 2025.
Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y.,
Finn, C.,, Kumar, A., and Levine, S. Cal-qi: Calibrated offline rl pre-training for efficient online fine-tuning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Puterman, M. L. Markov decision processes. Handbooks in operations research and management science, 2.33-4334 1990.
Silver, D., Schrittwiser, J., Simonyan, K., Antonoglou, I, Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7666):354-333
2017.
Song. Y., Zhou, Y., Sckhari, A., Bagnell, J. A., Krishna- murthy, A., and Sun, W. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprinl arXiv:22 20.0022700.671111 18, 2022.
Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, and Kolesnikov, S. Corl: Research-oriented deep offline rein- forcement learning library. Advances in Neural Informa- lion Processing Systems, 36, 2024.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Wang, S., Yang, Q., Gao, J., Lin, M., Chen, H., Wu, L., Jia, N., Song. S., and Huang, G. Train once, get a family: State-adaptive balances for offlnn-to-online reinforce- ment learning. Advances in Neural Information Process- ing Systems, 36, 2024.
Wu, J., Wu, H., Qiu, Z., Wang, J., and Long, M. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 31278-31291, 2022.
Wu, Y., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning. arXiv preprint arXiv: 1911. 1361, 2019.
Yu, Z. and Zhang, X. Actor-critic alignment for offline-to- online reinforcement learning. In International Confer- ence on Machine Learning, pp. 2023.
PMLR,

---

Online Pre-Training for Oflline-to-Olline Reinforcement Learning
A. Implementation
A.I. Implementation Details for OPT
We implement OPT based on the codebase of cach backbone algorithm. TD3 and RLPD are based on their official implementation" 1 SPOT, IQL are built upon the CORL (Tarasov al. 2024) library 9. The primary modification introduced in our method is the addition of the Online Pre-Training phase. The Online Pre-Training phase is implemented by modifying the meta-adaptation method provided in the OEMA code', tailored specifically for value function learning. Additionally, the balanced replay (Lee et al. 2022) is implemented using the authors' official implementation", Aside from these changes, no other alterations are made to the original code. The complete training code is available at ht https: htg/tttttbbbbeg wttptttt.......ttetteeeeeeeeeee -eeetA---eeeechorreeee
A.2. Additional Implementation Details for IQL
The proposed method is applicable to various baseline algorithms. Here, we present the implementation when applied to IQL. IQL trains the state value function and the state-action value function as follows:
(v(u)= =
-v,(s))
(5)
(Q( B( [ (V Qo(s,a))))
(6)
where Cz(u) |r 1(u < 0) and E (0, 1) is the expectile value, and Qa denotes target state-action value function. Then, using the state-action value function and state value function, the policy is trained through Advantage Weighted Regression:
(o) = (s,))>> (exp( a) - (s))) log (a]s]
where 3 E [0, oo) is an inverse temperature. To apply the proposed method to IQL, we train both a new state-action value function (Qon-pt) and state value function (Voo--) in the Online Pre-Training. The state-action value function is trained identically to Eq. (3), while the state value function trained follows:
Coretrain L + LV - a VV
(8)
In the online fine-tuning, policy improvement utilizes of Quil-pt, and Voff-pt from offline pre-training as well as Qom-pi and yon-pt from Online Pre- Training. The policy is trained using an advantage weight obtained from Q'll-p' and poll-tt, and separate weight obtained from Qon-pt and prn-ppt These sets of weights are then combined to effectively train the policy:
Ln
E ( (ex((((x(()))) oll (s,, a) #
(sf)
+(1 - (Q on-pt
v (x,
(8,a) a) - V.n v()r(())))) log (a(s)))
A.3. Additional Implementation Details for RLPD
To verify the effectiveness of the proposed method when applied to ensemble techniques and high UTD ratio, conduct experiments by integrating OPT into RLPD. Since RLPD does not originally include offline phase, we incorporate an additional offline phase into its implementation. During the offline phase, we follow the RLPD learning method, performing 1M gradient steps. In the online phase, we adhere to OPT by conducting Online Pre- vee-Trainng for 25k environment steps, followed by 275k environment steps. Given that the original RLPD employs symmetric sampling in the online phase, where half of the samples are sampled from offline data and the other half from online data, we also utilize symmetric sampling instead of balanced replay.
"nttps//eituuu.commiii http:hawww..ooo..oosooooippppppp
hhtns://gituu.coo//uuussuuu
12

---

Online Pre-Training for Offline-to-Olliee Reinforcement Learning
A.4. Baseline Implementation
For comparison with OPT, we re-run all baselines. The results for all baselines are obtained using their official implementa- tions.
OFF2ON (Lee et al. 2022): httpss///gghhaa..cmmm///eeee//ss
OEMA (Guo et al. 2023): httpps///gtthabb..oomm//yyy/////
PEX (Zhang et 2023): httpes//gsttabbccc///ahhhhhhaaas
ACA (Yu & Zhang 2023): httppp//ttttuuoooessssaat/ttttaa
FamO20 (Wang et 2024) httppss/ggttto.ccommm//aaa/////2
Cal-QL (Nakamoto et al. 2024):
B. Detailed Description of the Environment and Dataset.
B.1. MuJoCo
MuJoCo consists of locomotion tasks and provides datasets of varying quality for each environment. We conduct experiments on the halfcheetah. hopper, and walker2d environments. MuJoCo environment dense reward setting, and we use the "-v2" versions of the random, medium, and medi um peedium-replee datasets for each environment.
B.2. Antmaze
Antmaze involves controlling ant robot navigate from the start of the maze to the goal. Antmaze is a sparse reward environment where the agent receives reward of +1 upon reaching the goal, and O otherwise. The maze is composed of three environments: umaze, medium, and large. For the dataset, we use the "-v2" versions of umaze, maze--iiersse, medi um-uumm-ppll.. py, medi um-di deress, large-pl -play, and
B.3. Adroit
Adroit is set of tasks controlling a hand robot with five fingers. Each environment has a different objective: in the pen environment, the task is twirling a pen; in the hammer environment, hammering nail; in the door environment, grabbing a door handle and opening it; and in the relocate, locating a ball to goal region. We utilize the "-v1" version of the cloned dataset for each environment.
13

---

Online Pre-Training for Offline-to-Oline Reinforcement Learning
C. Further Experimental Results
C.I. Ablation on Replay Buffer
The proposed method employs balanced replay (Lee et al. 2022) to enhance the use of online samples during online fine-tuning Since the balanced replay has proven effective in to-iine-o-online RL on its own, we conduct experiments assess its impact and dependency within OPT. To align with the original framework of OPT, which emphasizes rapid adaptation through more frequent learning for online samples during online fine-tuning, we test a setup that uniformly samples from online data, method we refer to Online Replay (OR).
Table Comparing normalized score for OPT with balanced replay and online replay buffer. All results reported as the mean and standard deviation across five random seeds. Improvement (%) refers to the performance gain when compared to the baseline.
Environment
OPT with BR
OPT with OR
halfchectah-r
89.0 #21
81.4 #6.2
hopper-r
109.5 13.1
109.5 1:5.1
walker2d-r
88.1 #52
89.7 #2144
halfchcetah-m
96.6 117
89.8 #3.0
hopper-m
112.0
111.2 #1.7
walker2d-m
116.1
113.9
halfcheetah-m-r
92.2 +12
85.3 +227
hopper-m-r
112.7 #1.1
111.4 118
walker2d m-r
117.7 :35
109.1 19.1
Total
933.9
901.3
Improvement (%)
24.5 %
20.2 %
The results in Table 6 demonstrate that the performance gains of OPT are not solely attributable to the effects of balanced replay. Furthermore, replacing balanced replay with this simpler Online Replay setup still results in significant performance improvements compared with the baseline. These findings indicate that the performance of OPT stems not just from balanced replay, but from other strategy that emphasizes learning from online samples, such as Online Replay.
C.2. Ablation on K
OPT adjusts the weights of Qo and T during online fine-tuning through the coefficient K. To assess the necessity of scheduling, we first conduct experiments with fixed k values of 0, 0.5, and 1. In addition, to analyze sensitivity to the scheduling parameters, we evaluate two alternative schedules where i increases linearly from 0.1 to 0.8 and from 0.1 to 0.7.
Table Ablation study results on & in the Antmaze domain. All values are reported as the mean and 95% confidence interval over 5 random seeds.
Environment
0.1 - 0.9
0.1 0.8
0.1 + 0.7
0
1
0.5
umaze
99.8+
100.0+a.
99.6#a.7
95.2470
98.8=1.1
99.0+0.
umaze-diverse
97.4+
96.8+1.2
95.2+3.2
77.64141
94.0=1.2
97.0+2.1
medium-play
98.2
97.6214
98.2+1.1
94,4218
94.6-18
96.0218
medium-diverse
98.4++111
97.6#1.3
96.6*21
96.0420
95.2=26
97,0
large-play
78.2438
78.3219
81.0264
46.0430
41.82167
59.2
large-diverse
90.64:32
90.0+1.3
88.4+3.2
52.0
49.0+11.2
75.4
The experimental results lead to two key observations. First, the comparison with fixed i values highlights the necessity of using a schedule. Second, the performance differences among various scheduled values are marginal, indicating that the precise choice of the value is not critical. Rather, the crucial factor the presence of a gradual transition between the two value functions.
These findings validate both the importance of x scheduling and the robustness of our method to variations in the scheduling configuration.
14

---

Online Pre-Training for Offline-to-nlinee Reinforcement Learning
C.3. Full Results for Comparison of Different Initialization Methods
In Section 5.1, we investigate the effectiveness of the Online Pre-Training strategy. We present the complete results of this experiment, which demonstrate that Online Pre- rre-rrannng facilitates more efficient learning of Q compared to other baselines. This improvement is particularly pronounced in the random dataset, where the distribution shift is more significant.
Table Results of the ablation study Online Pre- Pre-Training All experimental results are measured after 300k steps of online fine-tuning with 5 random seeds used for each experiment.
Environment
OPT
Random Initialization
Pre-trained with Bon
halfcheetah-r
89.0+21
76.7+1119
68.3+
hopper-r
109.5
105.8
105.5
walker2d-r
88.1
14.5
73.41289
halfcheetah-m
96.6#1.7
89.7+
95,7+ax
hopper-m
112.0=1.
11 111.3219
109.6#22
walker2d-m
116.1#47
101.0
114.7#22
halfchoctahhm--r
92.2#112
84.3=82
91.0219
hopper-m-r
112.7
111.7
112.2+0.7
walker2d-m-r
117.7=35
104.8
111.6484
Total
933.9
843.1
882.0
We further extend this evaluation to the Antmaze domain. In addition to the baselines considered in Section 5.1, we introduce a simplified alternative to our method by removing the meta-adaptation component from Equation (3). Specifically, the objective is replaced by:
pretrrin
(1b) = Qr (1b) + L (1b)
(10)
Table 9. Results of the ablation study on N,. All experimental results are measured after 300k steps of online fine-tuning, with 5 random seeds used for each experiment.
OPT
Random Initialization
Pre-trained with B.n
Pre-trained with Bar and B.o
umaze
99,8103
99.61077
98.8
99.4=007
umaze-diverse
97.4403
95.4+13
95.4+
96.6-2.2
medium-plas
98.2
92.4#22
95.6#111
96.6=20
medium-diverse
98.4+1.1
92.8#38
95.4
97.2=1.7
large-play
78.2138
1.0
0.0=0.0
0.0:200
large-diverse
90.643.
33.8
54.0#27.1
15.2+14.9
Total
562.6
415.0
439.2
405.0
As shown in Table 9, the results are consistent with those observed in Figure 5. Notably, the additional variant (pre-trained with both Bar and Bun) exhibits a sharp decline performance, especially in more challenging environments such as Antmaze-large. This degradation attributed conflicting learning dynamics between the two loss terms, which the simplified method fails reconcile.
In contrast, adaptation objective resolves this conflict and enables stable and effective online fine-tuning. These results underscore the importance of the meta-adaptation mechanism under distribution shift.
15

---

Online Pre-Training for Offline-to-Onling Reinforcement Learning
C.4. Full Results for Comparison of Different N,
In Section 5.5, we examine how the performance varies with different values of Nr. Table 10 presents the full results corresponding to Figure 12. The results show that increasing N, generally improves performance across all environments, but the performance tends to plateau beyond 25k steps.
Table 10. Results of the ablation study on N,. All experimental results measured after 300k steps online fine-tuning, with random seeds used for each experiment.
6250
12500
25000
50000
100000
halfcheetah-r
66.4492
79.9
89.0+21
93.6#21
93.7+28
hopper-r
105.8 1.3
103.7+44.
109.5+a1
108.4=522
109.24+20
walker2d-r
75.9
72.949.
88.1
92,4
93.9
balfchectah-m
98.6
97.0=2.5
96.6117
98.4
97.3
hopper-m
105.8#25
108.7+22
112.0+1.
110.2=16
110.94220
walker2d-m
112.8=70
115.3
116.1:447
119.5=4.8
116.2+11
halfcheetah-m-r
89.9
90.4+14
92.2+112
92.0=1.8
92.6+228
hopper-m-r
94,9155
111,1
112.7
112,8=22
111.6115
walker2d-m-r
111.2#23
113.0
117.7435
113.2=27
115.44228
Total
861.3
892.0
933.9
940.5
940.8
C.5. Full Results for Ablation of Addition of New Value Function
In Section 5.3, analyze the impact of introducing value function performance. Table 11 provides the full results corresponding to Figure 7. The results indicate that the addition of the new value function contributes to performance gains in almost all environments.
Table 1. Results of the ablation study on addition a new value function. experimental results are measured after 300k steps online fine-tuning, with 5 random seeds used for each experiment.
OPT
w/o Q
halfeheetah-r
89.0 21.
94.0:13.
hopper-r
109.5 127
96.91449
walker2d-r
88.1
9.349.5
halfchectah-m
96.6 114
93.1
hopper-m
112.0 +1.1
109.0=2.0
walker2d-m
116.1:4.
110.7=28
halfcheetah-m--r
92.2++10
90,6+1.5
hopper-m-r
112.7=09
109.5
walker2d-m-r
117.7:3.
107.5
Total
933.9
820.6
16

---

Online Pre-Training for Offline-to-Olline Reinforcement Learning
D. Additional Analysis
Since the motivation of our method is to improve value estimation during online fine-tuning, we conduct additional experiments to verify whether the combined value function benefits from OPT. To evaluate this, we compare the value estimation of TD3 and TD3+OPT in three environments where OPT shows significant performance gains: halfchectah- As approximation of the optimal value function,
medium-reppay--i hopper-random-v2. and train a TD3 agent with sufficient steps until it reaches near-optimal performance and use its value function as reference. For fair comparison, we sample 10 fixed state-action pairs, where states are drawn from the initial state distribution and actions are selected using the optimal policy. We then measure the value estimation bias of TD3 and TD3+OPT throughout online fine-tuning.
Hallcheetah-Meddln--eeee
Hopper-Randon
Walker2D-Medlum
Estimation Bias
300
TD3+OPT
Estimation Bias
140
120
Estimation Bias
160
140
250
TD3
100
120
200
80
100
150
80
40
60
50
20
40
0
1
2
0
1
2
3
0
2
Envionment Steps (x10²)
Envionment Steps (x10²)
Envionment Steps (x10²)
Figure 9. Estimation bias of value function, comparing TD3 and TD3+OPT against optimal value function. The hias for the OPT remains initially flat due to the Online Pre-Training phase.
Figure 9 presents how the estimation bias evolves during online fine-tuning. These results indicate that TD3 exhibits noticeable esimation bias due to distribution shift, whereas our method reduces bias early training by leveraging Q trained specifically to handle online samples. To further support this, we conduct an additional experiment using the same estimation bias metric, but this time comparing Qotl-pt and Qon-pt individually.
Halfcheetah-
Hopper-Mediee
Walker2D-Medium
Estimation Blas Estimation Bias
500
Offf-
Estimation Blas
250
Estimation Bias
250
400
Q pom-pu
200
200
300
Combined Q
150
150
200
100
100
100
50
50
0
2
1
2
0
2
3
Envionment Steps (x10²)
Envionment Steps (x10²)
Envionment Steps (x10²)
Halfcheetah-MMeduu-RRlllll
Hopper-Medium-RRplay
Walker2DDMMddum-RRplaay
300
500
Estimation Bias
250
Estimation Blas
250
200
400
200
150
300
150
200
100
100
100
50
3
2
3
Envionment Steps (x10²)
Envionment Steps (x10²)
Envionment Steps (x10))
Figure 10, Estimation bias of the value function, comparing Q """ and the combined Q against the optimal value function. The combined Q corresponds to the formulation used in OPT, as defined in Equation (4) the main paper.
As shown in Figure 10, Q reduces estimation bias more rapidly than Qoff-pt, indicating that the improvement observed above stems from the effective adaptation of Qeappt to online samples. These findings confirm that our method enhances value estimation, leading more stable and effective online fine-tuning.
17

---

Online Pre-Training for Offline-to-Onling Reinforcement Learning
E. Comparison with Backbone Algorithm
Our proposed OST is an algorithm applicable to value-based backbone algorithms. To evaluate the performance improvement achieved by applying OPT, we compare it against the performance of the backbone algorithms. Table 12 presents the performance of OPT when TD3, SPOT, and IQL are used as the backbone algorithms across all environments. When based on TD3 and SPOT, we observe average performance improvement of 30%. Additionally, when based on IQL, we observe an average performance improvement of 25%.
Table Comparison normalized scores after online fine-tuning each environment the D4RL benchmark. We denote the backbone algorithm as Vanilla and the result of the algorithm integrated with OPT as Ours. All results are reported as the mcan and 95% confidence interval over 10 random seeds.
Environment
TD3
IQL
Vanilla
Ours
Vanilla
Ours
halfcheetah-rr
94.6
90.2
32.8
45.9
hopper-r
86.0 +:3.3
108.7 11.
11.1 #1.1
15,1 +:3.8
walker2d-r
0.1 =0.1
88.0 #3.2
6.6 =1.0
#2.3
balfcheetah-m
93.4
97.0
50.1
55.1
hopper-m
89.3 +14.6
112.2 =0.6
63.2 #3.6
96.2 #0.2
walker2d-m
103.5 #3.8
116.1 #1.9
83.9 #3.0
90.4 #1.6
halfcheetah-m-rr
87.2 20.9
94.3 21.5
46.2 #0.2
50.2 1:11
hopper-m-r
94.5 +11.3
112.7 2:0.4
92.9 #8.2
90.5 #11.2
walker2d-m-r
103.9 #6.6
119.9 =2.2
91.9 #2.9
105.3
MuloCo total
752.4
939.1 (+24.8%)
478.7
558.5 (+16.6%)
SPOT
IQL
Vanilla
Ours
Vanilla
Ours
umaze
98.7 10.8
99.7 10.2
90.0 12.7
94.8 11.7
umaze-diverse
+15.9
#0.4
34.5 111.5
#7.2
medium-play
91.1
97.6
84.2
89.5
medium-diverse
92.0 11.7
98.7 =0.6
81 |4 #2.4
88.0 11.7
large-play
111.4
14..5
15.3
64.4 11.9
large-diverse
70.0 +11.5
90.1 =2.9
49.5 #3.6
62.9 #3.7
Antmaze total
465.7
565.3 (+21.3%)
389.7
489.1
(+25.5%%
pen-cloned
114.8 16.3
1:4.6
93.8 17.5
104.6 15.9
hammer-cloned
84,0 +13.8
121.9 #1.6
13.6 #3.8
22.0 =11.9
door-cloned
1.6 #2.7
51.1 #12.9
6.6 #2.2
27.9 +44.8
relocate-cloned
-0.19 20.02
-0.06 =
0.06 1:0.02
0.77 =0:42
Adroit total
200.2
309.1 (+54.4 %))
117.1
155.3 (+32.6%)
18

---

Online Pre-Training for Offlinc-to-Onling Reinforcement Learning
F. Full Results of RLPD
In Section 5.6, we compare our proposed method with RLPD, which, despite being based on different framework, has demonstrated impressive performance. Table 13 presents the full results of this comparison. The results further highlight that OPT can serve as a complementary component enhance the effectiveness of established state-of-the-ar methods.
Table 13, Average normalized final evaluation score for each environment on the D4RL benchmark, We denote the vanilla algorithm as Vani Ila, the baseline algorithm within the offine-to-onling RL framework off-to-on. and the result of the algorithm integrated with OPT as Ours, All results are reported as the mean and standard deviation across five random seeds.
Environment
RLPD
Vanilla
Off-to-on
Ours
halfcheetah-r
91.5 #2.5
96.1 #5.2
90.7 #2.2
hopper-r
90.2 +19.1
95.7 +18.4
103.5 #3366
walker2d-r
87.7 #14.1
74.3 1:3.9
79.2 10000
halfcheetah-m
95.5 21.5
96.6 20.9
96.7 11.4
hopper-m
91.4 #27.8
93.6 +13.9
106.9 #1.5
walker2d-m
121.6 =2.3
124.1
122.8 #3.0
halfcheetah-m-r
90.1
90.0
91.6 #2.1
hopper-m-r
78.9 424.5
94.7 226.8
107.4
walker2d-m-r
119.0 =2.2
122.5 #2.7
120.9 +:2.3
MuJoCo total
866.0
887.6
918.7
umaze
99.4 20.8
99.8 10.4
99.6 10.5
umaze-diverse
98.0 #1.1
99.2 #1.0
99.0 #0.6
medium-plas play
97.6 1.44
97.4 1144
99.6 +:0.6
medium-diverse
97.6
98.6
99.2 10.4
large-play
93.6 #2.4
93.0
92.2 :3.9
large-diverse
92.8 #3.2
90.4 #3.9
94.8 #22.
Antmaze total
579.0
578.4
584.4
pen-cloned
154.8 111.6
148.5 #15.2
155.5 1:1.0
hammer-cloned
139.7 =5.6
141.4 #1.0
142.1 #1.2
door-cloned
110.8 =6.1
114.6
115.7 +1.4
relocate-cloned
4.8
0.11
10.0 16.4
Adroit total
410.1
404.6
423.3
19

---

Online Pre-Training for Offline-to-Onling Reinforcement Learning
G. Extra Comparison
This section presents a comparative analysis OPT against recent baselines, including BOORL (Hu et 2024) and SO2 (Zhang et al. 2024). BOORL employs Bayesian approach to address the trade-off between optimism and pessimism in ofllnneto-online reinforcement learning, providing comprehensive framework for tackling this fundamental challenge. Similarly, SO2 incorporates advanced techniques into Q-function training to mitigate Q-value estimation errors, offering an effective strategy for improving value learning.
To ensure a fair comparison, experiments are conducted using the official implementations of BOORL and SO2, with hyperparameters set according to their original authors' recommendations. Additionally, the evaluation of these baselines are extended to 300k steps align with the experimental setup of the proposed method. The results are summarized in Table 14.
Table 14. Comparison of the normalized scores after online fine-tuning for each environment in MuJoCo domain. All results reported as the mean across five random sceds.
TD3+OPT 300k
BOORL 200k
BOORL 200k (reproduced)
BOORL 300k (reproduced)
SO2 100k
SO2 100k (reproduced)
SO2 300k (reproduced)
(ours)
(paper)
(paper)
halfcheetah-r
89.0
97.7
97.9
101.3
95.6
99.1
130.2
hopper-r
109.5
75.7
102.2
108.5
79.9
96.2
88.5
walker2d-r
88.1
93.6
4.3
0.1
62.9
36.9
66.8
halfcheetah-m
96.6
98.7
100.4
102.0
98.9
98.1
130.6
hopper-m
112.0
109.8
110.4
110.7
101.2
102.5
82.1
walker2d-m
116.1
107.7
105.3
114.5
107.6
109.1
105.7
halfcheetah-m--r
92.2
91.5
88.5
91.7
113.1
hopper-m-r
112.7
111.1
110.9
111.0
101.0
101.8
74.4
walker2d-m-r
117.7
114.4
109.6
112.3
98.2
96.8
27.1
Total
933.9
900.2
829.5
852.1
834.7
838.5
818.5
The results indicate that both BOORL and SO2 demonstrate competitive performance in specific environments, such as hal 1 fcheetah. Still, they generally underperform compared OPT regarding average performance across all tasks. Notably, SO2 performs well at 100k steps; however, its effectiveness declines significantly when extended to 300k steps. BOORL achieves comparable performance to the proposed method in several environments but shows inconsistencies in reproducibility particularly in the resulting in significantly higher computational costs during training. In contrast, OPT achieves comparable or superior performance without relying on such ensemble techniques, making a more efficient and effective solution.
task. Both BOORL and SO2 adopt ensemble-
approaches,
20

---

Online Pre-Training for Offline-to-Onling Reinforcement Learning
H. Hyper-parameterr
this paper, we present the results of applying OPT to various backbone algorithms. Aside from the hyperparameters listed below, all other hyperparameters are adopted directly from the backbone algorithms. In our proposed Online Pre- Tre-Trainnn we N 25k and Vrretsiin 50k for all environments. Additionally, for the MuJoCo domain, we use TD3 with UTD ratio of the baseline. In the Adroit domain, applied to both the actor and critic networks. Except for these modifications, all other components and training details are kept identical to the original backbone algorithms.
SPOT the baseline, trained with layer normalization (Ba 2016)
As mentioned in Section 3.2, we use the parameter A to assign higher weight to Qo during the early stages of online fine-tuning, gradually shifting to give higher weight to Qea-p as training progresses. We control x through linear scheduling. Table 15 outlines the x: scheduling for each environment, where Kinit represents the initial value of k at the start of online phase, T deeny specifies the number timesteps over which k increases, and Kend indicates the final value of after increase. Notably, in the MuJoCo random environment, demonstrated in Section 5.2 the value function pre-trate pre-trained offline exhibits significant bias, so it is not utilized during online fine-tuning.
Table 15. scheduling method for each environment.
Environment
Kinit
Tdecay
Kend
halfcheetah-r
hopper-r
1
walker2d-r
halfcheetah-m
0.3
150000
0.9
hopper-m
0.3
150000
0.9
walker2d-m
0.3
150000
0.9
halfcheetah-m-rr
0.1
150000
0.9
hopper-m-r
0.1
150000
0.9
walker2d-m-r
0.1
150000
0.9
umaze
0.1
100000
0.9
umaze-diverse
0.1
100000
0.9
medium-play
0.1
100000
0.9
medium-diverse
0.1
100000
0.9
large-play
0.1
100000
0.9
large-diverse
0.1
200000
0.9
pen-cloned
0.1
250000
0.9
hammer-cloned
0.1
250000
0.9
door-cloned
0.1
250000
0.9
relocate-cloned
0.1
250000
0.9
21

---

Online Pre-Training for Offline-to-Onling Reinforcement Learning
I. Learning Curves
OPT
Backbone
OFF2ON
OEMA
PEX
ACA
FamO20
Cal-QL
hallcheetahhcandsmme2
ballchheetah:neddimmm22
hoppet-aandmm-22
100
100
Normalized Score
Nomalized Score
80
80
Normaloud
40
40
20
20
Environientt Steps (10")
Environment Steps (10")
Environment Steps (10")
Environment Steps (10')
hopoer-medium.g
crsper-medium v2
walker2d random-v2
walker2d medium-v2
120
100
Normalized Score
100
Normalized Score
80
1000
Normatbre
60
40
60
120
40
0
Environment Steps (10P))
Environment Steps (1.P)
Environment Steps (10F)
Environment Stegs (10F)
Walker2d repleyy.22
play
100
100
100
Normalzzed Stroes
Normalzzee Score
Nommlized Score
80
80
80
60
40
40
20
20
Environment Steps (10P)
Enviranment Steps (10P)
Environment Stops (10?)
Environment Stops (10')
antmaze-reddumddeeesee
artmssrellrge--leaya
antinare-begg-divene--
pen-cloned-v1
120
Nomalzzd Scote
Nomalzzed Score
100
50
160
Nommaized
80
40
8D
20
20
0
0
0
Environment Steps (10")
Environment Steps (10")
Environment Steps (10')
Environment Steps (10')
hamme-ddnedivv
door-clerediv1
ralocate clonad-v1
Normatbed Score
Normalted Score
50
Normalized Spore
-0.05
80
-0.10
60
-0.15
-0.20
40
20
-0.30
Environment Steps (10%)
Environment Steps [10%)
Steps
Figure Learning curve of online phase over 300k steps. solid line represents the mean performance, while shaded region depicts the standard deviation across five random seeds.
22

---

Online Pre-Training for Offline-to-nline Reinforcement Learning
J. Computational Cost
To evaluate computational efficiency, we compare wall-clock running time TD3+OPT and TD3 the walker2d- random-v2 environment, where TD3+OPT achieves the most significant performance gains.
5000
Training Time (s)
4000
3000
2000
1000
TD3
Environment Steps (10*)
Figure 12. Comparison of wall-clock training time for TD3 and TD3 integrated with OPT on the walker2U-random--22 environment using single NVIDIA 1.40 GPU.
For TD3, the wall-clock time is approximately 4000 seconds. In contrast, TD3+OPT requires 6000 seconds, primarily due to the additional steps for Online Pre- Trainirinng (N, and Vpremain are set to 25k and 50k steps, respectively) and the updates associated with the addition the new value function during online fine-tuning.
Although OPT requires additional time compared TD3, this increase reflects the added effort to train the new value function, which integral enhancing adaptability and performance during online fine-tuning. The trade-off is balanced, the additional computational cost results in substantial improvements in overall performance.
23


Reviews:
[{"review_id": "Qy1yo5sSbs", "reviewer": "Reviewer_VzZL", "summary": "The authors proposed a new offline-to-online RL method called Online Pre-Training (OPT), where a new phase, online pertaining,  is added between offline pre-training and online fine-tuning to solve the inaccurate value estimation problem. OPT introduce a separate value function instead of directly continue the learning on the value function trained on offline dataset. Experiments are conducted on D4RL to confirm its superiority over other offline-to-online methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback.\nThe comments help improve the clarity and completeness of our work.\nBelow, we address each point in detail.\n\n&nbsp;\n### **R1-1. Regarding Experimental Results**\nFollowing the reviewer’s suggestion, we increase the number of random seeds from 5 to 10 to strengthen the claim regarding the effectiveness of OPT.\n\nIn Table A-C of the anonymous link (https://sites.google.com/view/icml2025opt/), we report the mean and 95% confidence interval (CI) rather than standard deviation to convey the reliability of the results.\nOPT consistently outperforms the baseline, and the minimal CI overlap supports the statistical significance of the results. \n\nWe also provide Interquartile Mean (IQM) [1-1] comparisons in Figure D for each domain, which further confirm that our method achieves strong performance.\nAdditionally, we will increase the number of random seeds for Tables 4 and 7 and include the corresponding updated results in the revised manuscript.\n\n&nbsp;\n### **R1-2. Intuitive guide of balancing strategies**\nWe appreciate the reviewer for pointing out this insightful consideration.\nAlthough we do not provide a theoretical analysis for selecting the weighting coefficient $\\kappa$, we offer the following intuitive guideline based on our empirical observations:\n\n- If the dataset quality is moderate, we recommend setting $\\kappa$ such that the emphasis on $Q^{\\text{on-pt}}$ gradually increases during online fine-tuning.\n- Conversely, if the dataset quality is relatively low, it is beneficial to set $\\kappa$ to utilize $Q^{\\text{on-pt}}$ from the early stage of online fine-tuning.\n\nWe will include this practical guideline in the revised manuscript to provide clearer guidance on applying our method across various tasks.\n\n&nbsp;\n### **R1-3. Regarding Additional Experiments on Value Estimation**\nSince the motivation of our method is to improve value estimation during online fine-tuning, we conduct additional experiments to verify whether the combined value function benefits from OPT.\n\nTo evaluate this, we compare the value estimation of TD3 and TD3+OPT in three environments where OPT shows significant performance gains: halfcheetah-medium-replay-v2, hopper-random-v2, and walker2d-medium-v2.\nAs an approximation of the optimal value function, we train a TD3 agent with sufficient steps until it reaches near-optimal performance and use its value function as a reference.\n\nFor a fair comparison, we sample 10 fixed state-action pairs, where states are drawn from the initial state distribution and actions are selected using the optimal policy.\nWe then measure the value estimation bias of TD3 and TD3+OPT throughout online fine-tuning.\n\nFigure A of the anonymous link (https://sites.google.com/view/icml2025opt/) presents how the estimation bias evolves during online fine-tuning.\nThese results indicate that TD3 exhibits noticeable esimation bias due to distribution shift, whereas our method reduces bias early in training by leveraging $Q^{\\text{on-pt}}$ trained specifically to handle online samples.\n\nTo further support this, we conduct an additional experiment using the same estimation bias metric, but this time comparing $Q^{\\text{off-pt}}$ and $Q^{\\text{on-pt}}$ individually.\nAs shown in Figure C, $Q^{\\text{on-pt}}$ reduces estimation bias more rapidly than $Q^{\\text{off-pt}}$, indicating that the improvement observed above stems from the effective adaptation of $Q^{\\text{on-pt}}$ to online samples.\n\nThese findings confirm that our method enhances value estimation, leading to more stable and effective online fine-tuning.\nWe will include this analysis in the revised manuscript to support our claims.\n\n&nbsp;\n\nWe appreciate the reviewer's detailed and insightful feedback once again. The suggestions have meaningfully contributed to improving the clarity and rigor of our work. We hope that our responses have adequately addressed the concerns raised.\n\n&nbsp;\n\n[1-1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" NeurIPS 2021.", "claims_and_evidence": "See questions.", "methods_and_evaluation_criteria": "See questions.", "theoretical_claims": "See questions.", "experimental_designs_or_analyses": "See questions.", "relation_to_broader_scientific_literature": "The idea of two separate Q function is innovative.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See questions.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The experiments results for TD3 in table 1 , Vanilla RLPD in table 4 and other initializations in table 7 seem to come with significant larger standard deviation than OPT, taking the standard deviation into consideration, OPT seems not convincingly outperform the baselines on a lot of the tasks. Would the authors consider increase the number of random seeds?\n2. The authors conducted ablation study for the weighting coefficient K on different dataset. However, for a new task, B_init and B_final are not available until you run the fine-tuning, making it hard to choose k before analysis. It would be great if authors could give a more intuitive guide of balancing strategies according to the task complexity of other properties.\n3. The main motivation that authors propose the extra online pre-training stage with separate online Q function is that there are previous empirical experiments in papers like ((Nakamoto et al. 2024) showing the smoothness of Q function learning when switching from offline to online thus the algorithms suffer from an initial unlearning effect. I think it’s would be a necessary extra experiments to show the combined Q as in (3) used in online fine-tuning indeed improved after using OPT.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NgykBE7FJA", "reviewer": "Reviewer_pJZg", "summary": "The paper studies offline-to-online RL problem and propose a new method called online pretraining. The mains to solve the sub-optimality problem brought by the offline Q value function during online fine-tuning. Specifically, the method propose to freeze the offline policy at the beginning of the online fine-tuning stage, collect online data and initialize the online Q value function with the combination of online and offline data with the OMEGA algorithm. During the rest of the online stage, the proposed method trains the policy with the combination of both the online Q value function and offline Q value function. The paper performs extensive empirical study on the effectiveness of  the proposed algorithm.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the detailed and insightful comments.\nThe suggestions are highly valuable in helping us refine both the presentation and the depth of our analysis.\nWe carefully address each point below.\n\n&nbsp;\n### **R2-1. Regarding Asymptotic Performance of Figure 4**\nWe appreciate the reviewer for raising this interesting point.\nTo investigate the asymptotic performance for different initialization strategies, we provide learning curves in Figure B at the anonymous link (https://sites.google.com/view/icml2025opt/).\nAs discussed in Section 5.1 of our paper, **Random Initialization** initially causes an early performance drop due to unstable value estimates, which in turn results in lower asymptotic performance.\n**Pre-trained with $B_{\\text{on}}$** achieves performance comparable to OPT in medium and medium-replay datasets but shows weak asymptotic performance on the random dataset.\nThese results further support the effectiveness of the initialization strategy employed in our method.\n\n&nbsp;\n### **R2-2. Regarding of $\\kappa$ Scheduling**\nThe motivation behind the design is that during online fine-tuning, $Q^{\\text{off-pt}}$ becomes less reliable due to distribution shift, resulting in inaccurate value estimation.\nTo address this, we introduce $\\kappa$ scheduling strategy that gradually shifts emphasis toward $Q^{\\text{on-pt}}$, enabling more effective online fine-tuning.\n\nWe empirically validate this approach through an experiment comparing $Q^{\\text{off-pt}}$ and $Q^{\\text{on-pt}}$ against an optimal value function during the online fine-tuning.\nThe optimal value function is estimated by training a TD3 agent until it achieves near-optimal performance.\nFor a fair evaluation, we use 10 fixed state-action pairs, where states are sampled from the initial state distribution, and actions are selected using the optimal policy.\n\nAs shown in Figure C at the anonymous link (https://sites.google.com/view/icml2025opt/), $Q^{\\text{off-pt}}$ exhibits higher estimation bias due to its inability to adapt under distribution shift.\nIn contrast, $Q^{\\text{on-pt}}$, aided by Online Pre-Training, effectively reduces estimation bias.\nThese results highlight that a gradual shift from $Q^{\\text{off-pt}}$ to $Q^{\\text{on-pt}}$ serves as an effective mechanism for mitigating estimation bias during online fine-tuning.\n\n&nbsp;\n### **R2-3. Analysis Experiments on Other Domains**\nWe agree that extending the analysis experiments in Section 5 to all environments can improve the comprehensiveness of our study.\nWe are currently conducting analysis experiments, and one preliminary result is shown in Table D at the anonymous link (https://sites.google.com/view/icml2025opt/), which exhibits patterns consistent with those in Figure 4 of the main paper.\nWe plan to include results covering all environments and analyses in the revised manuscript.\n\n&nbsp;\n### **R2-4. Regarding Different Backbone Algorithms**\nThe reason for using different backbone algorithms is that no single algorithm consistently performs well across all domains.\nFor example, Off2On[2-1] performs well in MuJoCo but is not evaluated on Antmaze, and our reproduction did not perform well there.\nConversely, SPOT[2-2] performs well in Antmaze but underperforms in MuJoCo.\nAs each algorithm tends to be specialized for certain benchmarks, we choose the most suitable backbone for each to better assess the effect of our method.\n\nThis is possible because our method is designed to be flexible and readily applicable to a wide range of algorithms.\nWe will clarify this in the revised manuscript to avoid potential confusion.\n\n&nbsp;\n### **R2-5. Regarding RLPD**\nOur work focuses on methods in the offline-to-online RL framework, which includes a distinct offline phase.\nFor this reason, the current manuscript does not include RLPD[2-3] in the main results, as it is primarily designed for online RL settings and does not involve a separate offline phase.\nHowever, to prevent confusion in baseline comparisons, we will include RLPD in Table 1 in the revised manuscript, along with a brief explanation clarifying its distinction.\n\n&nbsp;\n\nWe sincerely thank the reviewer again for the constructive comments. The feedback helped us strengthen both the empirical evidence and the clarity of our manuscript. We hope our responses have sufficiently addressed the concerns and contributed to a clearer understanding of our method.\n\n&nbsp;\n\n[2-1] Lee, Seunghyun, et al. \"Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.\" CORL 2022.\n\n[2-2] Wu, Jialong, et al. \"Supported policy optimization for offline reinforcement learning.\" NeurIPS 2024.\n\n[2-3] Ball, Philip J., et al. \"Efficient online reinforcement learning with offline data.\" ICML 2023.", "claims_and_evidence": "The paper makes claims on the strong performance of the proposed algorithm and it is validated by experiments on the relevant benchmarks over dense and sparse reward environments, and compared with a comprehensive set of baselines. The only point that might seem unfair is that the proposed method used different backbone algorithms for different tasks. \n\nAs for the effectiveness of each component of the proposed method, the paper also performs good ablation experiments on different initialization of the online Q functions. However, it would also be interesting to see if different initialization affects the asymptotic performance of the algorithm. \n\nThe decaying coefficient $\\kappa$ seems like a very intuitive idea and the paper attempts to provide an analysis on this design choice as well. However, I do not believe that the current evidence makes sense because the offline Q function is also trained on the online data as well. Maybe one way is to show the difference between online/offline Q function to the optimal Q function along the training trajectory?", "methods_and_evaluation_criteria": "As mentioned above, the paper already contains a very comprehensive set of evaluations. However, it will benefit the paper even further if the analysis experiments in Section 5 are also repeated on the whole set of environments.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "It provides a flexible method for offline2online RL with strong performance, which benefits the community.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "According to Table 4, it seems like that RLPD is actually the strongest baseline in general (by a large margin)? Thus it would be nice if it is included in the comparison  in Table 1.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3p6tg5o0Hu", "reviewer": "Reviewer_Xjkr", "summary": "This paper presents OPT, a method to improve value estimation in RL. OPT follows three phases: offline pre-training, an \"online pretraining\" phase to train a separate value function, and online fine-tuning that combines both value functions. Unlike traditional methods that use a single value function, OPT introduces a second one trained with offline data and early online samples for better performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback, which helps improve both the clarity and completeness of our analysis.\nWe address each point below.\n\n&nbsp;\n### **R3-1. Regarding Statistical Significance**\nTo reduce statistical uncertainty, we increase the number of random seeds from 5 to 10 for all baselines for Tables 1-3 and 10, and report 95% confidence intervals (CI) instead of standard deviation to better reflect the reliability of the results.\nTables A-C at the anonymous link (https://sites.google.com/view/icml2025opt/) show that OPT consistently outperforms baselines, with minimal CI overlap, indicating statistical significance.\n\nTable E further supports this trend when combined with IQL, demonstrating robustness across backbone algorithms.\nFor clarity, we bold entries whose CIs include the highest mean value among compared methods.\nWe will include these results in the revised manuscript.\n\n&nbsp;\n### **R3-2. Regarding Aggregated Statistics**\nWe appreciate the suggestion to use aggregated metrics such as the Interquartile Mean (IQM) and include corresponding comparisons.\nFigure D at the anonymous link (https://sites.google.com/view/icml2025opt/) shows our method consistently achieves the strongest performance.\n\nWe also provide aggregated learning curves.\nAs shown in Figure E, our method demonstrates consistent improvement, especially strong result in the Adroit domain.\n\nThese results support the statistical significance and overall effectiveness of our approach.\nWe will include them in the revised manuscript.\n\n&nbsp;\n### **R3-3. OPT with Simple Alternatives**\nAs described in Section 3.1, Online Pre-Training initializes the new value function to enable effective online fine-tuning.\nTo achieve this, we proposed a meta-learning strategy, formalized in Equation 3.\n\nThe second term in Equation 3 updates the value function using online samples while incorporating gradients from $\\mathcal{L}^{\\text{off}}$.\nThis aligns two terms, allowing $Q^{\\text{on-pt}}$ to generalize across offline and online data.\nTo assess its usefulness, we compare it with a simpler alternative that jointly trains on $B_{\\text{off}}$ and $B_{\\text{on}}$, as follows:\n\n$\\mathcal{L}_{Q^{\\text{on-pt}}}^{\\text{pretrain}}= \\mathcal{L}^{\\text{off}}(\\psi) + \\mathcal{L}^{\\text{on}}(\\psi)$\n\nThis experiment is shown in Table D of our anonymous link (https://sites.google.com/view/icml2025opt/).\nThe variant **Pre-trained with $B_{\\text{off}}$ and $B_{\\text{on}}$** shows a performance drop in harder tasks e.g. large mazes,  due to conflicting learning dynamics between two terms, which the simpler method cannot resolve.\nIn contrast, our method reconciles the two objectives, enabling effective online fine-tuning.\n\nThese results highlight the value of the meta-adapation objective under distribution shift.\nWe consider this a key contribution and will include this discussion in the revised manuscript.\n\n&nbsp;\n### **R3-4. Ablation Study for Equation 4**\nWe conducted experiments in Antmaze to assess the necessity of Eq. 4.\n\nTable F at the anonymous link (https://sites.google.com/view/icml2025opt/) compares $\\kappa$ scheduling with fixed alternatives.\nWhile fixed $\\kappa$ works in simple tasks, performance degrades in harder environments.\nIn particular, $\\kappa=0.5$ causes instability due to prolonged reliance on $Q^{\\text{off-pt}}$.\n\nWe also observe that performance is not sensitive to a specific $\\kappa$ value, suggesting the key factor is a gradual shift from $Q^{\\text{off-pt}}$ to $Q^{\\text{on-pt}}$.\nThese results underscore the importance of Equation 4 for stable and effective training.\n\n&nbsp;\n### **R3-5. Computational Cost**\nWe compare the wall-clock time of TD3 and our method.\nAs shown in Figure F at the anonymous link (https://sites.google.com/view/icml2025opt/), TD3 takes about 4000 seconds, while ours takes around 6000 seconds due to the added value function and Online Pre-Training ($N_\\tau = 25k$, $N_{\\text{pretrain}}=50k$).\n\nAlthough this introduces extra overhead, we believe the performance gains reasonably justify the additional cost.\nWe will include this analysis in the appendix.\n\n&nbsp;\n### **R3-6. Regarding Theoretical Justification**\nOur work aims to empirically address challenges in offline-to-online RL.\nAlthough it does not include theoretical justification, the results support the effectiveness of our design.\nWe recognize the value of theoretical analysis and consider it as a promising future direction.\n\n&nbsp;\n### **R3-7. Regarding Missing Standard Deviation**\nWe apologize for omitting standard deviations in Table 11 and Figures 4 and 6.\nWe will include them in the revised manuscript for proper assessment.\n\n&nbsp;\n\nWe appreciate the reviewer again for the valuable feedback and hope our response sufficiently addresses the concerns. We believe our method achieves statistically significant performance and is readily applicable to a wide range of algorithms. We hope this to be considered in the reviewer’s reevaluation of our work.", "claims_and_evidence": "Several claims lack sufficient statistical significance:\n\n1. The claim of \"average 30% improvement\" seems dubious when considering standard deviations, which often overlap significantly with baselines.\n2. Claims of superiority over Cal-QL (lines 308-314) aren't statistically justified given overlapping standard deviations in Table 2.\n3. Performance comparisons in Tables 1-3 highlight mean values without proper statistical significance testing, making it difficult to assess the consistency of improvements (this also includes Table 11 where authors claim superiority over BOORL without even reporting stdev).\n4. The conclusion that OPT works well across backbone algorithms is supported by empirical results, but the improvements for IQL (Table 10) show substantial overlaps in standard deviations, weakening this claim.", "methods_and_evaluation_criteria": "- The evaluation uses appropriate benchmarks (Mujoco, D4RL benchmarks) that are standard in the field.\n- However, the meta-learning objective from OEMA appears unnecessarily complex without clear justification for why simpler approaches wouldn't work. The paper lacks ablations comparing this meta-learning approach to simpler alternatives.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "As I wrote in the \"claims\" section above, several issues with experimental design and analysis undermine the paper's conclusions:\n\n1. Statistical reporting is inconsistent – standard deviations are missing entirely from Table 11 and Figure 4,6, making it impossible to assess the significance of comparisons.\n2. The paper lacks an ablation study on whether Equation 4 (weighted combination of Q-functions) is necessary, given that κ scheduling was shown to have minimal impact (like on halfcheetah env).\n\nOverall, I feel that there is a lot of overlap between the proposed method and other baselines if we factor in standard deviations, suggesting the improvements may not be statistically significant. And the selective reporting, i.e. highlighting only the mean values without statistical tests can create a misleading impression of consistent superiority. Therefore, empirically this paper does not seem sound to me.", "relation_to_broader_scientific_literature": "The problem statement is relevant to broader literature.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "## Strengths:\n\n1. Novel approach to offline-to-online RL that addresses a fundamental limitation (inaccurate value estimation)\n2. Extensive evaluation across multiple domains (MuJoCo, Antmaze, Adroit)\n3. Good ablation studies on components like initialization methods and sample sizes\n4. Paper is easy to follow \n\n## Weaknesses:\n\n1. Lack of theoretical justification for using two separate value functions (minor)\n2. Insufficient comparison with simpler alternatives to the proposed complex approach (i.e. why do we need to use OEMA in equation 3?)\n3. Selective reporting of results that emphasizes means over statistical significance and significant overlap with baselines", "other_comments_or_suggestions": "Typos: line 165: \"as\" instead of \"at\"; line 167: \"B_off.\" instead of \"B_off,\")", "questions_for_authors": "1. How much additional computational cost is incurred when using OPT vs a baseline like TD3 or IQL?\n2. Could you provide results using the rliable library [1]? The analysis can be made stronger with aggregated statistics like IQM rather than simply using means+stdev.\n3. Why is the meta-adaptation objective from OEMA necessary? Have you compared with simpler approaches for training the new value function?\n4. Can you provide \"aggregated\" learning curves with standard deviations for all baseline methods in Tables 1 and 2, similar to Figure 3, to allow for fair assessment of performance differences? (I was hoping for an aggregate curve in Appendix but could only find per-environment curves in Figure 8)\n\n[1]: Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "edbgqOjN2b", "reviewer": "Reviewer_2yWP", "summary": "The paper proposes a novel offline-to online RL approach where, at the end of the offline phase, a second critic is trained in an \"Online pre-training\" phase and used in addition to the offline dataset during the final online learning phase. During online pre-training, the offline policy and critic are frozen, and the new critic is trained from the frozen offline policy, filling a new online buffer. By leveraging both the offline critic and the pretrained online critic, offline-to online works better than with former approaches. The algorithm outperforms many state-of-the-art baselines in many environments.\n\n## update after rebuttal:\nI was in favor of accepting the paper, after reading all the reviews and exchanges with the authors, I'm still so.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the valuable and constructive feedback.\nWe respond to each point in detail below and will reflect the suggested improvements in the revised manuscript.\n\n&nbsp;\n### **R4-1. Questions**\n- Question about umaze dataset\n\nAccording to the official D4RL[4-1] paper, the Antmaze environment consists of three types of datasets: umaze, diverse, and play.\nAmong them, the ”play” dataset is collected by commanding the ant to navigate from various hand-picked start locations to hand-picked goals.\n\nIn contrast, the *umaze* dataset, which appears in the first row of Table 2, uses a single fixed start and goal location without diversity in either.\nSince this setup does not match the definition of a *play* dataset, it is not categorized as one.\n\nWe will clarify this point in the revised manuscript in Appendix B.2.\n\n- Question about training protocol\n\nAll experiments in our paper follow a training protocol consisting of a 1M offline phase and a 300k online phase.\nFor OPT, the first 25k steps of the online phase are used for online pre-training, while the remaining steps are allocated for online fine-tuning.\nThis setup is also applied in Figure 3.\n\nSince the key changes in OPT occur during the online phase, Figure 3 and Appendix H present performance curves for the online phase.\nThis is reflected in the initially flat performance observed in the graphs, as OPT does not update the policy during online pre-training.\n\nTo avoid any confusion, we will explicitly clarify this in the caption of Figure 3 and in Appendix H.\n\n&nbsp;\n### **R4-2. Sensitivity to $\\kappa$**\nTo provide clearer evidence for the sensitivity of our method, we conduct additional experiments in the Antmaze domain.\nTable F at the anonymous link (https://sites.google.com/view/icml2025opt/) highlights two key findings:\n- First, the comparison with fixed $\\kappa$ values demonstrates the necessity of scheduling $\\kappa$ during training.\n- Second, the results indicate that performance is not sensitive on a specific $\\kappa$ value; rather, the crucial factor is the gradual transition from $Q^{\\text{off-pt}}$ to $Q^{\\text{on-pt}}$\n\nThese findings validate both the importance of $\\kappa$ scheduling and the robustness of the approach to sensitivity in its exact values.\n\n&nbsp;\n### **R4-3. Regarding Writing and Reference**\nWe sincerely thank the reviewer for the detailed and thoughtful feedback on the writing.\nWe appreciate the time and care taken to point out areas for improvement, as well as for providing helpful references.\nAll suggestions will be carefully considered and reflected in the revised manuscripts, and we believe these revisions will improve the clarity and overall quality of the paper.\n\n&nbsp;\n### **R4-4. Regarding Figure 6**\nWe thank the reviewer for pointing out the insufficient information provided in Figure 6 regarding variability and statistical significance.\n\nTo provide further clarity, we include environment-specific results corresponding to Figure 6 in Table G in the anonymous link (https://sites.google.com/view/icml2025opt/).\nThese results show that performance degradation is especially noticeable in the random dataset.\nWe will include these additional results in the revised manuscript.\n\n&nbsp;\n\nOnce again, we greatly appreciate the reviewer's thoughtful comments and suggestions. The feedback has been instrumental in improving the quality and clarity of our work. We hope our responses sufficiently address the concerns raised.\n\n&nbsp;\n\n[4-1] Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" arXiv (2020).", "claims_and_evidence": "The paper benefits from a strong methodology and the authors provide satisfactory evidence for most of their claims.\n\nOne exception that I found is in Figure 6 (ablation of the new value function) where the authors should provide information on the variability of the results and use an appropriate statistical test to check that there is a statistically significant difference between the full method and its ablation.\n\nAnother one is the study of the sensitivity of the algorithm to \\kappa, which is far from clear (see comments below).", "methods_and_evaluation_criteria": "Most experimental design details are satisfactory.", "theoretical_claims": "Does not apply (no theorem in this paper)", "experimental_designs_or_analyses": "As stated above, I'm OK with all experimental design details", "relation_to_broader_scientific_literature": "The paper is clearly positioned with respect to the relevant literature.", "essential_references_not_discussed": "I could not find an essential missing reference\n\nThe authors should definitely read the WSRL paper:\n\nZhou, Z., Peng, A., Li, Q., Levine, S., & Kumar, A. (2024). Efficient online reinforcement learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762.\n\nwhich has an alternative approach to the same problem, but they cannot be blamed for not doing so as this can be considered \"concurrent work\" (out less than 4 months ago).", "other_strengths_and_weaknesses": "Strengths:\n- this is a solid paper, with a strong methodology (5 seeds, hyper-param studies, ablations, etc.) and a good contribution to the domain\n- the method is quite simple though clever, and performs well\n- beyond SOTA results, the additional studies in Section 5 are of interest\n\n\nWeaknesses:\n- writing could be improved here and there", "other_comments_or_suggestions": "Clarity and writing concerns:\n- 60% of the abstract says known things in the beginning, the authors could come quicker to the point of their paper.\n- italic \"counter-intuitive trends\" is a poor short name for the phenomenon the authors focus on, as it is too general. There are many counter-intuitive trends in many domains, find something more specific to the context of this paper\n- the \"Background and related work\" section is quite messy. This could be reorganized into two more clearly organized sections.\n- Fig. 2 could probably be smaller, if the authors need more space in the main paper\n- Section 4.1 should refer to Appendix H for the learning curves, close to the tables.\n- If possible, Fig. 4 should move to p. 7.\n- In Fig. 5, medium-replay and medium could be swapped, to be in the same order as the text mentions them. The figure is not much readable.\n- There is an issue with explanations about \\kappa defered to appendices C.2 and D: in Appendix C.2, the authors seem to compare a linear scheduling to another linear scheduling, the point they want to make is very unclear. And I believe all explanations about \\kappa in Appendix D should move together with Appendix C.2.\n- in 5.4 and appendices, all 25000 and 50000 should be rewritten 25K and 50K so that we see that it corresponds to the 25K online pre-training duration mentioned earlier\n\n\nTypos and similar concerns:\n- the authors use a lot the \"'s\" form, even for things that are not persons: \"dataset's quality\", \"algorithm's code\", etc.They shouldn't.\n- the year is often missing in references, this seems to be done on purpose but I consider this as a bad practice.\n- p. 5: ACA ... which pre-process ->processes\n- 5.1 Comparison OPT... -> Comparing\n- p. 7 \"In particular, the results on the random dataset, where policy evolves more drastically as shown in Figure 5, the results demonstrate that an overfitted Qon-pt fails to learn with this policy improvement.\" -> fix this sentence!\n- OPT is missing in the legend of Fig. 7\n- caption Fig. 7 doamin -> domain\n- A.1: TD3, RLPD -> TD3 and RLPD ... are based on its -> their\n- not dots after an equation when the sentence is not finished (\"... where...\")\n- use \"\\ eqref\" rather than \"\\ ref\" to refer to an equation.\n- B.3 has a different objective(s)\n- F: bayesian -> Bayesian", "questions_for_authors": "- Table 2, first row, umaze is umaze-play, right?\n- In Figure 3, did the authors use a 300K + 25K + 275K protocol too? Are the first 300K steps shown? The transitions between offline, online pretraining and online should appear clearly, this is also the case of learning curves in Appendix H.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work proposes a step of value function learning in between offline pretraining and online finetuning.

I was just surprised to see concerns about the statistical significance of the results after that this was the major reason for reject at a previous conference. However, the authors have addressed these concerns in the rebuttal and I expect their answers to be integrated in the final version. Overall, I agree with the reviewers and I think the paper is an interesting and relevant contribution.
Decision: Accept (poster)