Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Authors: Max Wilcoxson
Qiyang Li
Kevin Frans
Sergey Levine
Venue: ICML 2025 poster (2025)
Keywords: Offline-to-online RL
Unsupervised Pre-training
Exploration
Primary area: reinforcement_learning->online

Abstract: Unsupervised pretraining has been transformative in many supervised domains. However, applying such ideas to reinforcement learning (RL) presents a unique challenge in that fine-tuning does not involve mimicking task-specific data, but rather exploring and locating the solution through iterative self-improvement. In this work, we study how unlabeled offline trajectory data can be leveraged to learn efficient exploration strategies. While prior data can be used to pretrain a set of low-level skills, or as additional off-policy data for online RL, it has been unclear how to combine these ideas effectively for online exploration. Our method SUPE (Skills from Unlabeled Prior data for Exploration) demonstrates that a careful combination of these ideas compounds their benefits. Our method first extracts low-level skills using a variational autoencoder (VAE), and then pseudo-labels unlabeled trajectories with optimistic rewards and high-level action labels, transforming prior data into high-level, task-relevant examples that encourage novelty-seeking behavior. Finally, SUPE uses these transformed examples as additional off-policy data for online RL to learn a high-level policy that composes pretrained low-level skills to explore efficiently. In our experiments, SUPE consistently outperforms prior strategies across a suite of 42 long-horizon, sparse-reward tasks.

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Max Wilcoxson Qiyang Li
Kevin Frans'
Sergey Levine
Abstract
2018) and vision (He et al., 2022). Pretrained mod- els can be adapted efficiently new tasks with few amples, and often generalize better than models trained from scratch (Radford et al., 2019; Brown et al., 2020). However, in contrast to supervised learning, reinforcement learning (RL) presents a unique challenge in that fine- tuning does not involve further mimicking task-specific data, but rather exploring and finding the solution through iterative self-improvement. Thus, the key challenge to ad- dress in pretraining for RL is not simply to learn good rep- resentations, but to learn an effective exploration strategy for solving downstream tasks.
Unsupervised pretraining has been transforma- tive in many supervised domains. However, applying such ideas to reinforcement learning (RI.) presents a unique challenge in that fine- tuning does not involve mimicking task-specific data, but rather exploring and locating the so- lution through iterative self-imprrovmment. In this work, we study how unlabeled offline tra- jectory data can be leveraged to learn efficient exploration strategies. While prior data can be used to pretrain a set of low-level skills, or as additional off off-policy data for online RL, it has been unclear how to combine these ideas ef- fectively for online exploration. Our method SUPE (Skills from Unlabeled Prior data for Exploration) demonstrates that a careful combi- nation of these ideas compounds their benefits. Our method first extracts low-level skills using a variational autoencoder (VAE), and then pseudo- labels unlabeled trajectories with optimistic re- wards and high-level action labels, transform- ing prior data into high-level, task-relevant ex- amples that encourage novelty-secking behav- ior. Finally, SUPE uses these transformed ex- amples as additional off-policy data for online RL to learn a high-level policy that composes pretrained low-level skills to explore efficiently. In our experiments, SUPE consistently outper- forms prior strategies across suite of 42 long- horizon, sparse-reward tasks. Code: https: // //github. com/ heto/raallaa heaall-breeleyysuggee
Pretraining benefits greatly from the breadth of the data. Unlabeled trajectories (i.e., those collected from previ- ous policies whose objectives unknown, from task- agnostic data collection policies) are the most abundantly available, but using them to solve specific tasks can be dif- ficult. It is not enough to simply copy behaviors, which can differ greatly from the current task. There is an entangle- ment problem - general knowledge of the environment is mixed in with task-specific behaviors. A concrete example is learning from unlabeled locomotion behavior: we wish to learn how to move around the world, but not necessar- ily to the locations present in the pretraining data. We will revisit this setting in the experimental section.
The entanglement problem can be alleviated through hi- crarchical decomposition. Specifically, trajectories can be broken into segments of task-agnostic skills, which are composed in various ways to solve various objectives. We posit that unlabeled trajectories thus present twofold ben- efit, (1) as way to learn a diverse set of skills, and (2) as off-policy examples of composing such skills. Notably, prior online RI. methods that leverage pretrained skills ig- nore the second benefit, and discard the offline trajectories after the skills are learned (Ajay et al., 2021: Pertsch al., 2021; Hu al., 2023; Chen et al., 2024). We instead ar- gue that such trajectories up learning. We make use of simple strategy of learn- ing an optimistic reward model from online samples, and useudo-labeling past trajectories with an optimistic reward estimate. The past trajectories can thus be readily utilized as off- off-policy data, allowing for quick learning even with a very small number of online interactions.
1. Introduction
Unsupervised pretraining has been transformative in many supervised domains, such as language (Devlin et al.,
critical, and can greatly speed
Eqqual contribution Universsity of California, Berkeley. Cor- respondence Max Wilcoxson <mwilcoxson@@ kuwilloxsoo@bbeeklll....ss Qiyang Li <qcli@ eecs. becs.berkel....du...
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
unsupervised skill pretraining
pick low-level action (o)
SUPE
Trajectory Skills
DBC+JSRL ExPLORe
offline
Low-level Skills
TT [a|s,, 2)
SUPE (HILP)
HILP Skills
Aggregated Performance (42 tasks)
pick high-level action (s)
IQM Normalizze Return
1.00
Unlabeled Offline Data
High-level Policy Tea (z|s)
0.75
Environment
0.50
RL training
0.25
Labeled Off- Off-Policy Data
pseudo-labed label with UCB reward estimote (f) and online high-llvell action estimate (5)
online transitions
0.00
0.0
0.1
0.2
0.3
Environment Steps (x10f)
Figure 1: Left: SUPE utilizee unlabeled offiine data twice, both for offtine unsupervised skill pretraining and online RL for learning the high-level policy. We pseudo-label the unlabeled data with upper-confidence bound (UCB) estimates of the rewards (reward label: f) and high-level action estimates (high-level action label: 2). This transforms the unlabeled data into high-level labeled off-policy data that encourages exploration, allowing significantly more sample efficient learning online. Right: Empirically, SUPE attains strong performance across 42 challenging sparse reward tasks (see details in Section 5).
We formalize these insights as SUPE (Skills from Unlabeled Prior data for Exploration), recipe for max- imally leveraging unlabeled offline data for online explo- ration (Figure 1). The offline data is utilized during both the offline and online phases. 1 In the offline phase, we extract short segments of trajectories from the unlabeled offiine data and use them to pretrain space of low-level skills represented by latent conditioned policy (a(a,s, z). 2 In the online phase, we utilize the unlabeled offline data as additional off-policy data for learning high-level exploration policy Th (=(s) by pseudo-labeling each trajec- tory segment with an upper-confidence bound (UCB) re- ward estimate of the true rewards and high-level action estimate (2). By using offline data in these two ways, we utilize both its low and high-level structure to accelerate online exploration.
2. Related Work
Unsupervised skill discovery methods were first investi- gated in the online setting, where RL agents were tasked with learning structured behaviors in the absence of a re- ward signal (Gregor al., 2016; Bacon al., 2017; Flo- rensa et al., 2017; Achiam et al., 2018; Eysenbach et al., 2018; Sharma et al., 2020; Hansen et al., 2020; Campos et al., 2020; Laskin et al., 2021; 2022; Park et al., 2023b; Yang et al., 2023; Bai et al., 2024). These insights natu- rally transferred to the offline setting as a method of pre- training on unlabeled trajectory data. Offline skill discov- ery methods largely comprise of two categories, those who extract skills based on optimizing unsupervised reward sig- nals (in either the form of policies (Touati et al., 2022; Hu et al., 2023; Frans et al., 2024; Park et al., 2024b) or Q-functions (Chen et al., 2024)), and those who uti- lize conditional behavior-cloning over subsets of trajecto- ries (Paraschos et al., 2013; Merel et al., 2018; Shankar & Gupta, 2020; Ajay et al., 2021: Singh et al., 2021; Pertsch et al., 2021; Nasiriany al., 2022). Closest our method are Ajay et al. (2021) and Pertsch et al. (2021), who utilize trajectory-segment VAE to pretrain low-level skills of- fline and learn a high-level policy online to compose them. In contrast to these methods that utilize offline data purely for offline skill-learning, our method also utilizes the data during online learning via pseudo-lubeling. As we show in our experiments, utilizing both the low-level and high- level structure of the offline trajectories is critical for fast exploration and learning online.
Our main contribution a simple method that leverages un- labeled offline trajectory data both pretrain skills offline and compose these skills for efficient online exploration. We instantiate SUPE with a variational autoencoder (VAE) to extract low-level skills, and an off-the-shelf off-policy RL algorithm (Ball et al., 2023) to learn a high-level pol- icy from both online and offline data. Our empirical eval- uations on a set of challenging sparse reward tasks show that leveraging the unlabeled offline data during both of- fline and online learning is crucial for efficient exploration, enabling SUPE to find sparse reward signals more quickly and achieve more efficient learning compared to all prior methods (none of which utilize the offline data during both online and offline learning).
Offline-to-onlinn reinforcement learning methods (Xie et al., 2021b; Song et al., 2023; Lee et al., 2022; Agarwal et al., 2022; Zhang et al., 2023; Zheng et al., 2023; Ball et al., 2023; Nakamoto et al., 2024) focus on using task- specific offline data accelerate online learning. Many
2

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
offiine RL approaches can be applied to this setting- - sim- ply run offline RI. first the offiine data to convergence as an initialization and then continue training for online learn- ing (using the combined dataset that consists of both offline and online data) (Kumar et al., 2020; Kostrikov et al., 2021; Tarasov et al., 2024). Such approaches often result in slow online improvements as offline RL objectives tend to overly constrain the policy behaviors be close the offline data, impairing exploration. On the other hand, off- pollillcy on- line RL methods can also be directly applied in this setting by treating the offline data as additional oof-policy data in the replay buffer and learning the policy from scratch (Lee et al., 2022; Song et al., 2023; Ball et al., 2023). While re- lated in spirit, these methods cannot be directly used in our setting as they require offline data to have reward labels.
HRL methods typically learn a high-level policy to lever- age a space of low-level primitive policies online. These primitives can be either manually specified (Dalal et al., 2021), learned online (Dietterich, 2000; Kulkarni et al., 2016; Vezhnevets et al., 2016; 2017: Nachum et al., 2018), or pretrained using unsupervised skill discovery methods as discussed above. While many existing works learn or fine-tune the primitives online along with the high-level policy (Dietterich, 2000; Kulkarni et al., 2016; Vezhnevets et al., 2016; 2017; Nachum et al., 2018; Shankar & Gupta, 2020), others opt for simpler formulation where the prim- itives are fixed after an initial pre-training phase and only the high-level policy is learned online (Peng et al., 2017; Riedmiller et al., 2018; Ajay et al., 2021; Pertsch et al., 2021; Gehring et al., 2021). Our work adopts the later strat- egy where we pre-train skills offline using a static, unla- beled dataset. No prior HRL method simultancously lever- ages offine data for skill pre-training and as additional off- policy data for learning a high-level policy online. As we show in our experiments, using these approaches in combi- nation is crucial to achieving sample efficient learning on challenging sparse-reward tasks.
Data-driven exploration. A common approach for on- line exploration is to augment rewards with bonus that incentives exploration and optimize the agent with respect to these modified rewards (Stadie et al., 2015; Bellemare et al., 2016; Houthooft et al., 2016; Pathak et al., 2017: Tang et al., 2017: Ostrovski et al., 2017; Achiam & Sas- try. 2017; Merel et al., 2018; Burda al., 2018; Ermolov & Sebe, 2020; Guo al., 2022; Lobel et al., 2023). While most exploration methods operate in the purely online set- ting and focus on adding bonuses to the online replay buffer, recent works have begun to explore a more data- driven approach that makes use of unlabeled offline data to guide online exploration. Li et al. (2024) added bonuses to the offline data, incentivizing exploration around the offline data distribution. While prior methods apply the bonus directly to dataset transitions, we instead add bonuses to high-level transitions, allowing to learn a policy that composes pretrained skills effectively for exploration. Hu et al. (2023) proposes a different strategy, first learning a number of policies using offline RL that each optimizes for a random reward function, then sampling from these policies online to achieve structured exploratory behavior. This approach does not utilize offline data during the online phase and requires each policy corresponding to different reward function to be represented separately. In contrast, our method makes use of the offline data as off-policy data for updating the high-level policy and our skills are rep- resented using a single network parameterized by skill latent. As we will show, using offline data online is crucial for efficient exploration.
See more related work discussions on options and unsuper- vised pre-training in Appendix D.
3. Problem Formulation
In this paper, we consider a Markov decision process (MDP) M = AS y.r.r. p} where S is the set of all possible states, A is the set of all possible ac- tions that policy ([als) : S + P(A) may take, P(s'|s, a) S x A + P(S) is the transition function that describes the probability distribution over the next state s' given the current state and the action taken at the state, y is the discount factor, r(s, a) : S x A + R is the reward function, and p P(S) is the initial state distribution. We have access a dataset of trajectories that are collected from the same MDP with reward labels: D == (7,,+2 } (where 7' == {sh- ag s1.0j sg....) J.. During online learning, the agent may interact with the environment by taking actions and observes the next state and the reward specified by transition function P and the reward function r. We aim to develop a method that can leverage the dataset D efficiently explore in the MDP to collect reward information, and outputs a well-performing policy T(als) that achieves high cumulative return in the environment 17(T)) := Eieespp.c P((s,)))) Eizo (1'r(s,, az)]. Note that this is different from the zero-shot RL set- ting (Touati et al., 2022) where the reward function is specified for the online evaluation (only unknown during the unsupervised pretraining phase). In our setting, the agent has no knowledge of the reward function priori
Hierarchical reinforcement learning (HRL). The abil- ity of RL agents to solve long-horizon exploration tasks is an important research goal in the field of HRL (Dayan & Hinton, 1992: Dietterich, 2000; Vezhnevets et al., 2016; Daniel et al., 2016; Kulkarni et al., 2016; Vezhnevets et al., 2017; Peng al., 2017; Riedmiller et al., 2018: Nachum et al., 2018; Ajay et al., 2021: Shankar & Gupta, 2020; Pertsch et al., 2021: Gehring et al., 2021; Xie et al., 2021a).
3

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Algorithm SUPE
ticular unknown task online. We achieve this by adopting trajectory VAE design from prior methods (Ajay et al., 2021; Pertsch et al., 2021) where short trajectory segment πh {su, ag: $1, -1 -H is first fed into tra- jectory encoder [o(=[T) that outputs a distribution over the skill latent 2. Then, a skill policy To (a] s, z) is trained to re- construct the actions the segment using a latent sampled from the distribution outputted by the encoder. This design allows to use the trajectory encoder fo(=/7) obtain an estimate of the corresponding skill latent è for any trajec- tory segment. Following prior work (Pertsch et al., 2021), we learn a state-dependent prior pa to help accommo- date the difference in behavior diversity different states. Putting them all together, the loss function
Input:
Unlabeled dataset of trajectory segments D, seg- ment length H, batch size B, replay buffer Dreplay +- 0.
1. Unsupervised Skill Pretraining for each pretraining step do (TAn::" ((H) ~ D
Update 0 with V8 Ei Co(T.) (Equation 1)
2. Online RL with Pseudo-labeling for every H online environment steps do
2a. Environment Interaction
(21s)
Run (=) for H steps: {so, 0o, ro.... (5}
H- D Dreplay t Dreplay {so, 2, sH. Di=0 [y'r:))).
Lo(T) = BDKL (F(e)))))))))))))oc(lll))
2b. Pseudo-labeling Offline Data
(1)
[T ) ~ D
traj. segments of length H
(2
Σ log na (Tnlonnsss)) (onlsh)))
2 ~ fo(2lTinn) Vi
b -iihh-lvee action estimate
p (UC 2)),
UCB reward estimate assemble batch
B Bottine + [(5)) 2¹, f" sH)
Online exploration with trajectory skills. To effectively leverage the pretrained trajectory skills, we learn high- level off-policy agent that picks which skills to use to ex- plore and solve the task efficiently. Following prior work on fixed-horizon skills (Pertsch et al., 2021), our high-level policy selects latent skill every H environment steps. To transform the unlabeled offline data into high-level, task- relevant transitions, we need both the high-level action and the reward label corresponding to each length H trajectory segment 7[ For the high-level action, we use the trajec- tory encoder fa (=] to obtain an estimate of the skill latent 2. For the reward, we maintain an upper-confidese (UCB) estimate of the reward value for each state and skill latent pair (s, inspired by prior work (Li al., 2024) (where it does so directly in the state-action space a)), and pseudo-label the transition with such optimistic re- ward estimate. The reward estimate is recomputed before updating the high-level agent, since the estimate changes over time, while the trajectory encoding is computed before starting online learning, since this label does not change. The -abeelng is summarized below:
2c. Updating High-level Policy TV
B Bouline Dreplay Update va with off-policy RL on Bonline U Boffline
Output: A hierarchical policy consisting of high-level (z(s) and low-level Te (a)a)s,2)
and must actively explore the environment online to receive reward information. In addition, our setting is also different from an adjacent setting where offline data only contain observations (Ma et al., 2022; Ghosh et al., 2023; Song et al., 2024) (see Appendix D for more discussions).
bound
4. Skills from Unlabeled Prior Data for Exploration (SUPE)
In this section, we describe how we utilize the unlabeled trajectory dataset to accelerate online exploration. Our method, SUPE, can be divided into two parts. The first part is the offline pretraining phase where we extract skills from the offline data with a sngjectory-segment VAE. The second part is the online learning phase where we train a high-level -ollcy agent to compose the pretrained skills by leveraging examples from both the offline data and on- line replay buffer. Algorithm describes our method.
(800 state
2 ~ [=(=))), FUCB (So, 2),
SH )
(2)
labeled action
labeled reward
next state
Practical implementation details. Following prior work on trajectory-segnae VAEs (Ajay et al., 2021; Pertsch et al., 2021), we use Gaussian distribution (where both the mean and diagonal covariance tory encoder, the skill policy, as well as the state-dependent prior. While Pertsch et al. (2021) use KL constraint be- tween the high-level policy and the state-dependent prior, we use a simpler design without the KL constraint that performs better in practice (as we show in Appendix G).
Pretraining with trajectory VAE. Without knowledge of the downstream task, we need to capture the diverse range of multitask behavior from the unlabeled offline dataset as accurately as possible. At the same time, we aim to leverage this dataset to more efficiently learn a high- level policy which composes these skills to solve a par-
learnable) for the trajec-
4

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
To achieve this, we adapt the policy parameterization from Haarnoja al. (2018), where the action value is enforced to be between -1 and 1 using tanh transformation, and entropy regularization is applied on the squashed space. We use this policy parameterization for the high-level pol- icy {2]ss) to predict the skill action in the squashed space Asqaushert We then recover the actual skill action vector by unsquashing it according to arctanh arctanh( (Zaqaushed). so that it can be used by our skill policy To (a (ddss,)........)). 2). For upper- confidence bound (optimistic) estimation of the reward, (#o 2)), we directly borrow the UCB estimation im- plementation in Li et al. (2024) (Section 3, practical imple- mentation section in their paper), where they use a com- bination of the random network distillation (RND) (Burda et al., 2018) reward bonus and the predicted reward from a reward model (see Appendix C for more details). For the off-policy high-level agent, we follow ei al. (2024) to use RLPD (Ball et al., 2023) that takes balanced number of samples from the offline data and the online replay buffer for agent optimization. In addition to using the optimistic offline reward label, we also add the RND bonus to the on- line batch as we have found it to further accelerate online learning.
humanoi dmaze domains has three different maze lay- outs. For antmaze, we test on four different goal lo- cations for each maze layout. For humano dmaze and ant soccer we test on one goal. For humano dmaze, we use both the navi gate and stitch datasets. For ant soccer,,, we use the navigate dataset for the arena and medium maze layouts.
State-based manipulation: cube-double, scene. Next, we consider set of ma- nipulation domains that require a wide range of manip- ulation skills. kitchen is a standard benchmark from D4RL where a robotic arm needs to complete a set of ma- nipulation tasks (e.g., turn on the microwave, move the kettle) in sequence in kitchen scene. cube-si ngl e, cube-double e, and scene are three domains from OG- Bench (Park et al., 2024a). For both cube-* domains, the robotic arm must arrange one or more cube objects to desired goal locations (e.g., stacking on top of each other) in tasks that mainly involve composing pick and place mo- tions. For scene, the robotic arm can interact with a more diverse set of objects: a window, a drawer, cube and two locks that control the window and the drawer. The tasks in scene are also longer, requiring the composition of multi- ple atomic behaviors (c.g., locking and unlocking, opening the drawer/window, moving the cube).
kitchen, cube- single,
5. Experimental Results
We present a series of experiments to evaluate the effective- ness of our method in discovering fast exploration strate- gies. We specifically focus on long-horizon, sparse-reward settings, where online exploration is especially important. In particular, we aim to answer the following questions:
Pixel-based: vi sual 1 -antmaze is a challenging visual domain introduced by Park et al. (2023a). The agent has all the same proprioceptive information from the state-based antmaze ze environment except the agent's (z. y) position, which the agent must obtain from 64 64 image obser- vation of its surroundings.
1. Can SUPE leverage pretrained skills to accelerate on- line learning?
To evaluate our method on these domains, we take the datasets in these benchmarks and remove the reward label as well as any information in the transition that may reveal the information about the termination of an episode. For all of the domains above, we use the normalized return, a standard metric for D4RL (Fu et al., 2020) environments, as the main evaluation metric. For the kitchen domain, the normalized return represents the average percentage of the tasks that are solved. For tasks in other domains, the nor- malized return represents the average task success rate. For all our plots, the shaded area indicates the 95% stratified bootstrap confidence interval and the solid line indicates the interquartile mean following Agarwal et al. (2021). See more details about the tasks and rewards in Appendix E.
2. Can SUPE find goals faster than prior methods?
3. How sensitive is SUPE to hyperparameters?
5.1. Experimental setup
We conduct our experiments on 8 challenging sparse- reward domains and provide a brief description of each of the domains below (more details available in Appendix E).
State-based locomotion: antmaze, humanoi dmaze, antsoccer. The first set of domains involve control- ling and navigating robotic agent in complex environ- ment. antmaze is standard benchmark for oof online RL. from D4RL (Fu et al., 2020). humanoi dma ze and ant are locomotion domains from OGBench, a offline goal-conditioned RL benchmark (Park et al., 2024a). In antmaze and humanoid, a humanoid or ant agent must learn to navigate to various goal loca- tions. antsoccer adds the difficulty of moving soc- cer ball to desired position. Each of the entmaze and
5.2. Comparisons
While there are existing methods that pretrain unla- beled offline trajectories and use these trajectories for on- line learning, there are methods that use offline data for either purpose. We first consider a baseline that directly performs online learning with no pretraining.
5

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
SP
antmaze: three maze layouts (medium, large and ult ultra), and four goals for each layout.
PAAC
b)
a antsoccer--rrenee htt///ediuu um
d) kitchen
e) cube
scene
visual -assaal-anttaeeeeeeeeeee
Figure 2: We experiment on challenging, sparse-reward domains. a): antmaze with three different layouts and the corresponding four goal locations (denoted as the red dots); b): humano dmaze with three layouts; c): antsoccer with two layouts d): k tchee;; with colored floor and local 64 64 image observations.
e): sube-single and cube-doubles scene: g): visual-anten
EXPLORE (Li et al., 2024). Unlike our method, this base- line does not perform unsupervised skill pretraining and directly learns 1-step policy online. As we will show, pretraining is crucial for success on more challenging do- mains. Like our method, EXPLORE uses an exploration bonus on offline data to encourage exploration. noting that the original EXPLORE method does not make of online reward bonus and only uses the RND net- work to optimistically label offline data. We additionally add the reward bonus to the online batch from the replay buffer to help the agent explore beyond the offline data dis- tribution. For completeness, we include the performance of the original EXPLORE in Figure 12. For all the baselines below (including our method), add the RND bonus to the online training batch to ensure fair comparison.
DBC+JSRL. We pretrain expressive diffusion policy to imitate the offline data via behavior cloning (BC) loss. At the beginning of each online episode, we rollout this pretrained policy for a random number of steps from the initial state before using switching to the online RL agent (Uchendu et al., 2023; Li et al., 2023). One might ex- pect that a sufficiently expressive policy class can capture the action distribution of the offline data accurately enough to provide an informative initial state distribution for online exploration. This baseline is an upgraded version of the BC+JSRL baseline used in EXPLORE (Li et al., 2024), with the Gaussian BC policy replaced by a diffusion policy.
worth
TRAJECTORY SKILLS and HILP SKILLS. We also con- sider two basclines that pretrain skills offline, but discard the offline data during online learning, training the high- level policy from scratch. Each baseline uses differ- ent skill pretraining method, In addition to the trajec-
We then consider baselines that pretrain on unlabeled of- fline data but do not use the data online.
6

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
SUPE
SUPE (HILP)
Trajectory Skills
HILP Skills
DBC+JSRL
ExPLORe
antmaze
visual-antmaze Tasks
kitchen Tasks
cube-single
12 Tasks
5 Tasks
1.00
0.75
IQM Normalized Return
0.50
0.25
0.00
cube-double
scene Tasks
humanoidmaze Tasks
antsoccer Tasks
Tasks
1.00
0.75
0.50
0.25
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x106)
Figure 3: Aggregated normalized return across eight different domains. We evaluate baselines that use offline data only for pretraining (m) only during online learning (+). and methods that offline data for both pretraining and online learning SUPE matches or outperforms all other methods domains except ant soccer scene, where SUPE (HILP) performs better. Section 5.2 contains more details about the baselines. For ki chon, we use 16 seeds. For all other domains, we use seeds.
tory skill approach used by method, we also consider recently proposed unsupervised offline skill discovery method where skills are pretrained to traverse learned Hilbert representation space (Park et al., 2024b) (HILP). For both baselines, we use the same high-level RL agent as our method, but learn this high-level policy purely from online interaction with the environment. While learning high-level policy from seratch to compose trajectory VAE skills is similar SPiRL (Pertsch et al., 2021), our baseline features two additional improvements. The first improve- ment is replacing the KL constraint with entropy regular- ization (same as SUPE as described Section 4, practical implementation details). The second improvement is the online RND bonus that is also added all other methods.
Aside from skill labeling, use the exact same high-level RL agent as used in our method.
For visual antmaze, -issuallattzze,, used in RLPD (Ball et al., 2023). We also follow one of our baselines, EXPLORE (Li et al., 2024), to use ICVF (Ghosh et al., 2023), a method that uses agnossti value functions learn image/state representations from passive data. ICVF takes in an offline unlabeled trajec- tory dataset with image observations and pretrain an im- age encoder in an unsupervised manner. Following Ex- PLORE, we take the weights of the image encoder from ICVF pretraining to initialize the image encoder's weights in the RND network. To make the comparison fair, we also apply ICVF all baselines (details Appendix F).
we use the same image encoder
Finally, introduce novel baseline that also uses of- fline data during pretraining and online exploration, but uses HILP skills rather than trajectory-based skills.
5.3. Can SUPE leverage pretrained skills accelerate online learning?
SUPE (HILP). Like trajectory skills, we can also esti- mate the corresponding HILP skill latent for any trajec- tory segment. HIIPP learns a latent space of the observa- tions (via an encoder CHILP) and learns skills that move the agent in certain direction z in the latent space. For any high-level transition (so, SH),, we can simply take é x- normalize (CHIL)(HHHHHHe - ΦHILP )), the normalized dif- ference vector that points from so to 8 in the latent space.
Figure shows the aggregated performance of our ap- proach all eight domains. Our method outperforms all prior methods all domains except antsocces where HILP SKILLS performs better, and cube-single, where EXPLORE performs competitively. Additionally, we note that while SUPE (HILP) performs best on scene and antsoccer. SUPE performs best on all other tasks. This suggests that different skill pretraining

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
all
antmaze-medium
antmaze-large
antmaze-ultra
ExPLORe
DBC+JSRL
HILP Skills
Trajectory Skills
SUPE (HILP)
SUPE
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
Environment Steps (x10%)
Figure 4: Interquartile mean (QQM) of the number of environment steps taken to reach the goal (lower the better). The first goal time is considered be the maximum (0.3 x 10 steps) the agent finds the goal. SUPE is the most consistent, achieving performance better than other baselines on layouts (4 tasks/goals for each layout seeds for each task).
33
TA
Figure 5: SUPE skill policy rollouts in huwanoid-mnn saaennodd-mzze----- (meoduu///aagggg/tiaaat)) each trajectory, we first sample skill latent, then repeatedly sample lower-level action from the low-level skill policy conditioned on that fixed latent. We plot the z-y position of the agent throughout each of 16 such trajectories for each maze.
methods may excel different tasks. Unlike all prior meth- ods, both SUPE and SUPE (HILP) leverage offline data twice (both during offline pretraining and online learning), which demonstrates how this new approach is critical to surpassing prior methods. Both skill-based online learning baselines (TRAJECTORY SKILLS and HILP SKILLS) each consistently perform worse than the corresponding novel method that uses offline data during online learning (SUPE and SUPE (HILP)), which further demonstrates the importance of using offline data twice. EXPLORE uses offline data during online learning, but does not pre- train skills, leading to slower learning on all 8 domains and difficulty achieving any significant return on any domains other than the easier antmaze and cube ingle tasks. We also report the performance on individual antmaze mazes and kitchen tasks in Figure 11, and observe that method outperforms the baselines more harder vironments. This trend continues with humanoi dmaze where SUPE is the only method to achieve nonzero final return on the more difficult large and giant mazes (see Ap- pendix J.1. Figure 16). These experiments suggest that pre- training skills from prior data and leveraging offline data in
the online phase both crucial for sample-efficient learn- ing, especially in more challenging environments.
To provide more insight on the sample-efficiengy of our method, we also provide visualization of rollouts from the skill policy in Figure 5. We notice that some of the frozen skill policies are able to navigate quite far from the start- ing point, which suggests that such structured navigation behaviors allow the high-level policy effectively explore with reduced time horizon.
For the visual domain, we perform an ablation study to as- sess the importance of the ICVF pretrained representation, which we include Appendix F. While ICVF combines synergistically with our method to further accelerate learn- ing, initializing RND image encoder weights using ICVF is not critical to its success.
5.4. Can SUPE find goals faster than prior methods?
While have demonstrated that our method outperforms prior methods, it is still unclear if our method actually leads to better exploration (instead of simply learning the high-
8

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
8 (Qurs)
16
H=
(Ours)
H =
RND Coefficient (a)
Horizon Length (H)
1.00
1.00
IQM Normalized Retur
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x106)
Figure 6: Sensitivity analysis the RND coefficient (a) and the skill horizon length (HT) on subset of tasks. For different hyper- parameter values, report interquartile mean (IQM)
normalized return across seven representative tasks. The performance of SUPE is not very sensitive to the magnitude of a as long as it within reasonable range (2, 16). Without the bonus (a = 0), SUPE performs significantly worse. A skill horizon length of 4 performs significantly better than a horizon length of 2 or 8. We use skill length of 4 and 8 for all experiments with skill-based methods. The normalized return for each individual task we use for this sceds.
analysis be found in Appendix H. We average
level policy better). In this section, we study the explo- ration aspect in isolation in the antmaze domain. Figure 4 reports the number of online enviromment interaction steps for the agent reach the goal for the first time. Such a met- ric allows us assess how efficiently the agent explores in the maze. Figure 4 shows that method reaches goals faster than every baseline all three maze layouts, which confirms that our method not only learns faster, but does so by exploring more efficiently. We include the results for each individual goal location and maze layout the Ap- pendix (Table 3) for completeness.
or longer horizons perform much worse.
6. Discussion and Limitations
In this work, we propose SUPE, a method that leverages unlabeled offline trajectory data accelerate online explo- ration and learning. The key insight is to unlabeled trajectories twice, to 1) extract set of low-level skills of- fline, and 2) serve as additional data for a high-level off- policy RI. agent to compose these skills to explore in the environment. This allows us to effectively combine the strengths from unsupervised skill pretraining and sample- efficient online RL methods solve series of challenging long-horizon sparse reward tasks significantly more effi- ciently than existing methods. Our work opens up avenues in making full use of prior data for scalable, online RL al- gorithms. First, our pretrained skills remain frozen during online learning, which may hinder online learning when the skills are not learned well or need to updated as the learning progresses (see analysis in Appendix K and fail- ure in Figure 19). Such problems could be alleviated by utilizing better skill pretraining method, or allowing the low-level skills to be fine-tuned online. In addition, our approach relies on RND to maintain an upper confidence bound on the optimistic reward estimate. Although we find that RND works without ICVF on high-dimensional im- age observations in visuat antmaze, the use of RND in other high dimensional environments may require more careful consideration. Possible future directions include examining alternative methods of maintaining this bound.
5.5. How sensitive is SUPE to hyperparameters?
Finally, we analyze the sensitivity our method to hyper- parameter selection. We focus on two hyperparameters, 1) a: the amount of optimism (RND coefficient) used when labeling offline data with UCB rewards (Equation 2), and 2) H: the length of the skill. We select representative task from each state-based domain, and study how differ- ent and H values affect the performance on these tasks (Figure 6). When the RND bonus is removed (a = 0), our method performs significantly worse, highlighting the importance of optimistic labeling on efficient online explo- ration. While we observe some variability across individ- ual tasks (Appendix H, Figure 9), our method is largely not very sensitive to the RND coefficient value. The aggregated performance is similar for E {2. 8, 16). Another key hy- perparameter our method is the skill horizon length (see Figure 6, right). We find that while there some variabil- ity across individual tasks (Appendix H. Figure 10), a skill horizon length of 4 generally performs the best, and shorter

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Acknowledgments
Akhil Bagaria and George Konidaris. Option discovery ing deep skill chaining. In International Conference on Learning Representations, 2019.
This research used the Savio computational cluster re- source provided by the Berkeley Research Computing pro- gram at UC Berkeley, and was supported by ONR through N00014-22-1-2773. AFOSR FA9550-22-1-0275 and the AI Institute. We would like to thank Seohong Park for providing his implementation of OPAL which was adapted and used for skill pretraining in our method. We would also like to thank Seohong Park, Fangchen Liu, Junsu Kim, Dibya Ghosh, Katie Kang, Oleg Rybkin, Kyle Stachowicz, Zhiyuan Zhou for discussions on the method and feedback on the early draft of the paper.
Akhil Bagaria, Ben Abbatematteo, Omer Gottesman, Matt Corsaro, Sreehari Rammohan, and George Konidaris. Effectively learning initiation sets in hierarchical rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Chenjia Bai, Rushuai Yang. Qiaosheng Zhang, Kang Xu,
Yi Chen, Ting Xiao, and Xuelong Li. Constrained ensemble exploration for unsupervised skill discovery. arXiv preprint arXiv:2405. 2024.
Impact Statement
Philip J Ball, Laura Smith, Ily Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
This paper presents work whose goal is to advance the field of reinforcement learning. There are many potential soci- etal consequences of our work, none of which we feel must be specifically highlighted here.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Ad- vances in Neural Information Processing Systems, pp. 1471-1479, 2016.
References
Joshua Achiam and Shankar Sastry. Surprise-based intrin- sic motivation for deep reinforcement learning. arXiv preprint 173.0:1703.01732, 2017.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint 180.v 10299 2018.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam. Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Neural Information Processing Systems (NeurIPS), 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep rein- forcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 29304 29320, 2021
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 28955-28971. Curran Associates, Inc., 2022.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12844, 2018.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state- covering skills. In International conference on machine learning, pp. 1317-1327. PMLR, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline primitive dis- covery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021. URL https:t////... httpt:htttt//eeeeeeeeee http://ppnneeeeeeeeeeeee f 1d-V69L6WJ01IN.
Boyuan Chen, Chuning Zhu, Pulkit Agrawal, Kaiqing Zhang, and Abhishek Gupta. Self-supervised reinforce- ment learning that transfers using random features. Ad- vances in Neural Information Processing Systems, 36, 2024.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
10

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning- Ad- vances in neural information processing systems, 17, 2004.
Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.),
Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Ma- chine Learning Research, pp. 13927-13942. PMLR, 21- 27 Jul 2024. URL ht http:tttss////peeeeeee http//pooceednngg mmmg..mmm... Tatsss/v33//Eranne
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations us- ing RNN encoder-decoder for statistical machine trans- lation. arXiv preprint arXiv:/406. 1078, 2014.
html.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven re- inforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Murtaza Dalal, Deepak Pathak, and Russ R Salakhutdinov. Accelcrating robotic reinforcement learning via parame- terized action primitives. Advances in Neural Informa- tion Processing Systems, 34:21847-21859, 2021.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hicrarchical skills for efficient explo- ration. Advances in Neural Information Processing Sys- tems, 34:11553-11566 2021.
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 17(93):1-50, 2016.
Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent in- tentions. In International Conference on Machine Learn- ing, pp. 11321-11339 PMLR, 2023.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing sys- tems, 5, 1992.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wier- stra. Variational intrinsic control. arXiv preprint arXi: 2016.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018
Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Bean-Bastien Grill, Yunhao Tang, et al. Byol-explore: Exploration by bootstrapped pre- diction. Advances in neural information processing sys- tems, 35:31855-31870, 2022.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research, 13:227-303, 2000.
Aleksandr Ermolov and Nicu Sebe. Latent world mod- els for intrinsically motivated exploration. Advances in Neural Information Processing Systems, 33:5565-5575, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maximum en- tropy deep reinforcement learning with a stochastic ac- tor. In International conference machine learning, pp. 1861-1870. PMLR, 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learn- ing skills without reward function. arXiv preprint arXiv: 06070. 2018
Steven Hansen, Will Dabney, Andre Barreto, David Warde- Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor fea- tures. International Conference on Learning Rep- resentations, 2020. URL btttps://ppereviieee ttte/forumiid=BJeAlee id=BBJAHHrYDD.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochas- tic neural networks for hierarchical reinforcement learn ing. In International Conference on Learning Represen- tations, 2017. URL https:/ forum? ddd aa8a.
httttttss//pppeeeeiii http:////pp/eeeeeeeeee net/
Philippe Hlansen-Estruce Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL:: Im- plicit Q-learning an actor-critic method with diffusion policies. urXiv preprint arXiv:2304. 10573, 2023.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv: 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot reinforcement learning via functional reward encodings. In Ruslan Salakhutdi- nov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational infor- mation maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024.
Hao Hu, Yiqin Yang, Jianing Ye, Ziqing Mai, and Chongjie Zhang. Unsupervised behavior extraction via random intent priors. In shirth-eeventh Conference Information Processing Systems, 2023. URL https: //openzeview. net forum? ? SeaeeiiiiivvVVvse.
Sam Lobel, Akhil Bagaria, and George Konidaris. Flip- ping coins to estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning, pp. 22594-226133 PMLR, 2023.
Neural
Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayara- man, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and represen- tation via value-implicit pre-training. arXiv preprint arXiv:2210.00030 2022.
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefenstette, and Yuan- dong Tian. Efficient planning in a compact latent action space. arXiv preprint arXiv:2208. 10291, 2022.
Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. Dynamic abstraction in reinforcement learning via clus- tering- In Proceedings of the twenty-first international conference on Machine learning, pp. 71, 2004.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems, 32, 2019.
George Dimitri Konidaris. Autonomous robot skill acquisi- tion. University of Massachusetts Amherst, 2011.
Ishai Menache, Shie Mannor, and Nahum Shimkin. Q- cut dyt-dynamic discovery of sub-goals in reinforcement learning. In Machine Learning: ECML 2002: 13th Eu- ropean Conference on Machine Learning Helsinki, Fin- land, August 19-23, 2002 Proceedings 13, pp. 295-306. Springer, 2002.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. arXiv preprint (X.v:2110000666: 2021.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29,209, 2016.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint arXiv:1811.11111, 2018.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforce- ment ment learning. Advances in Neural Information Process- ing Systems, 33:1 17911191, 2020.
Ofir Nachum, Shixiang Shane Gu. Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing sys- tems, 31, 2018
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Al- bert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement Iearn- ing benchmark. arXiv preprint arXiv:21
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Ar- avind Rajeswaran, and Pieter Abbeel. CIC: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint Xix::2022.000666 2022.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre- training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36. 2024.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learn- ing via balanced replay and pessimistic Q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMIR, 2022.
Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill- based imitation learning, In Conference on Robot Learn- ing. 2022.
Qiyang Li, Yuexiang Zhai, Yi Ma, and Sergey Levine. Un- derstanding the complexity gains of single-task RL with a curriculum. In International Conference on Machine Learning, pp. 20412-20433 PMLR, 2023.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural den- sity models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
12

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilisticc movement primitives. Advances in neural information processing systems, 26, 2013.
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In Interna- tional Conference on Machine Learning, pp. 8624-8633, PMLR, 2020.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline goal-conditioned RL with latent states actions. In Thirty- Conference on Neural Information Processing Sys- tems, 2023a. URL https: wttps////peeeeee..ee.o net/ Forum?id=eLLCC:VVD
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Ku- mar, and Karol Hausman. Dynamics-aware unsuper- vised discovery of skills. In International Conference Learning Representations, 2020. URL https:// openreview net / forum Himmmmddd-Hllll HUgHHHHHG.K....
Özgür Şimşek and Andrew G Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 95, 2004.
Seohong Park, Olch Rybkin, and Sergey Levine. METRA: Scalable unsupervised RL with metric-aware abstrac- tion. In NeurlPS 2023 Workshop on Goal-Conditione Reinforcement Learning, 2023b. URL https://
Özgür Şimşek and Andrew G. Barto. Betweenness central- ity as basis for forming skills. Workingpaper, Univer- sity of Massachusetts Amherst, April 2007.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal- conditioned rl. ArXiv, 2024a.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinchart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In In- ternational Conference on Learning Representations, 2021. URL http:: /ttp:///opeeeeeeeeeeeeeeee httpeviwwww.eeeeeeeeeee net/ forum?
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In Forty-first International Conference on Machine Learn- ing, 2024b. URL https:/ /wtp///weeeeeeeeeeeeeee forum? (oaumid-=ar ddt.unsSAktNN
LddYsuv-WOFRE WOFeKR.
Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL ef- ficient. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview net forum? id-
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Yuda Song. J Andrew Bagnell, and Aarti Singh. Hybrid reinforcement learning from offline observation alone. arXiv preprint 22X2:22406.0253,, 2024.
Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. Acm transactions on graphics (tog), 36(4):1-13, 2017.
Aravind Srinivas, Ramnandan Krishnamurthy, Peeyush Kumar, and Balaraman Ravindran. Option discov- ery in hierarchical reinforcement learning using spatio- temporal clustering. arXiv preprint arXiv:1605.05530 2016.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accel- erating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-204. PMLR, 2021.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incen- tivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.000 2015.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188): 9, 2019.
Richard S Sutton, Doina Precup, and Satinder Singh. Be- tween MDPs and semi-MDPs: A framework for tempo- ral abstraction in reinforcement learning. Artificial intel- ligence, 1 11((122):181-111 1999.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learn- ing, pp. 4344 4353, PMLR, 2018
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip De- Turck, and Pieter Abbeel. exploration: A study of
13

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
count-based exploration for deep reinforcement learn- ing. Advances in neural information processing systems, 30, 2017.
the AAAI Conference Artificial Intelligence, vol- ume 37, pp. 11372-11380, 2023.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist ap- proach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2022.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jtap-start reinforcement learning. In International Conference on Machine Learning. pp.
34583. PMLR, 2023.
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strate- gic attentive writer for learning macro-actions. Advances in neural information processing systems, 29, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pp. 3540-3549, PMLR, 2017.
Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Ani- mesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. International Conference Learning Representations, 2021a. URL https: //openreview met/fooumm? (ttprrrmmdidxieeekk3331113
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395- 27407, 2021b.
Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behav- ior contrastive learning for unsupervised skill discov- ery. In International conference on machine learning, pp. 39183-39204. PMLR, 2023.
Haichao Zhang, Wei Xu, and Haonan Yu. Policy expan- sion for bridging offine-to-online reinforcement learn- ing. The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview net/forumm - Y34145JR62.
Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dong- sheng Li, and Jing Jiang. Adaptive policy learning for offline-to-online reinforcement learning. In Proceedings
14

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
A. Compute Resources
We ran all our experiments on NVIDIA A5000 GPU and V100 GPUs. In total, the results in this paper required approxi- mately 16600 GPU hours.
B. VAE Architecture and hyperparameters
We use VAE implementation from Park et al. (2024b). The authors kindly shared with us their OPAL implementation (which produces the results of the OPAL baseline in the paper). In this implementation, the VAE encoder is a recurrent neural network that uses gated-recurrent units (Cho et al., 2014) (GRU). takes in a short sequence of states and actions, and produces a probabilistic output of latent z. The reconstruction policy decoder is a fully-connected network with ReLU activation (Nair& & Hinton, 2010) that takes in both the state the sequence as well the latent z to output an action distribution.
Parameter Name
Value
Batch size
256
Optimizer
Adam
Learning rate
x 10-4
GRU Hidden Size
256 (antmaze, kitchen, 512 (antsoccer, humanoidmas scene, cube-+) 2 hidden layers (antmaze, kitchen, visual-antmazae
GRU Layers
3 hidden layers (antsoccer, humanoidmaze, scene, cube-*) 0.1 (antmaze, humanoidmaa dmaze, kitchen, visual aessull-antinn -ntmaaze antsoccer)
KL Coefficient (3)
0.2 (cube-+, scene)
VAE Prior
state-conditioned isotropic Gaussian distribution over the latent
VAE Posterior
isotropic Gaussian distribution over the latent isotropic Gaussian distribution over the action space
Reconstruction Policy Decoder
Latent Dimension of
8
Trajectory Segment Length (H)
4
Image Encoder Latent Dim
50
Table 1: VAE training details.
In the online phase, our bigh-level policy a Soft-Actor Critr-Critic (SAC) agent (Haarnoja et al., 2018) with critic networks, entropy backup disabled and LayerNorm added the critics following the architecture design used RLPD (Ball al., 2023). Similar to EXPLORE (Li et al., 2024), we sample 128 offline samples and 128 online samples and add an RND reward bonus to all of the samples. While the original ExPLORE paper only added reward bonus to the offline data, we add the reward bonus to both the offline data and online data, as it leads to better performance in goals where there is limited offline data coverage (see Appendix 1.2).
C. Implementation details for baselines
SUPE. We follow Li al. (2024) (ExPLORE) relabel offline data with optimistic reward estimates using RND and reward model. For completeness, descrihe the details below. We initialize two networks go(s, =), 7(8, 2) that each outputs 256-dimensional feature vector predicted from the state and (tanh-squashed) high-level action. During online leamning ((s, 2) is fixed and we only update the parameters of the other network (s 2) to minimize the L2 distance between the feature vectors predicted by the two networks on the new high-level transition (snew new new new ):
(()) = [1go(s new new) - g( new I1
In addition to the two networks, we also learn a reward model re 2) that minimizes the reward loss below on the transitions (sg. 2, r, sh) from online replay buffer:
((C) ||re(so, z) r112.
15

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
We then form an optimistic estimate of the reward value for the offline data as follows:
(s, 2) + (* =) + (s0 =) - ((50, 2)
where a controls the strength of the exploration tendency (RND coefficient). For every environment with 1/0 rewards, we find that it is sufficient to use the minimum reward, -1, label the offiine data without a performance drop, we opt for such a simpler design for our experiments. On kitchen, we use reward prediction model since the reward is the number of tasks currently completed.
For online training hatch from the replay buffer, we also add online reward bonus provided by RND with the same a coefficient. For all the baselines below, we also add the online RND bonus to the online batch to encourage exploration.
ONLINE ONLY. In the appendix, we additionally include a baseline that does not use offline data at all. This baseline directly runs the SAC agent from scratch online with no pretraining and no usage of the offline data.
DBC+JSRL, We use the diffusion model implementation from (Hansen-Estruch et al., 2023). Following the paper's implementation, we train the model for 3 million gradient steps with a dropout rate of 0.1 and a cosine decaying learning rate schedule from the learning rate of 0.0003. In the online phase, in the beginning of every episode, with probability p. we rollout the diffusion policy for a random number of steps that follows a geometric distribution Geom(1->) before sampling actions from the online agent (inspired by (Li et al., 2023)). A RND bonus is also added to the online batch on the fly with a coefficient of 2.0 to further encourage online exploration of the SAC agent. The same coefficient is used in all other non-skill based baselines. For skill based baselines, we scale up the RND coefficient by the horizon length (4) to account for different reward scale. Following the BC+JSRL baseline used in EXPLORE (Li et al., 2024), we use a SAC agent with an ensemble of 10 critic networks. one actor network, with no entropy backup and LayerNorm in the critic networks. This contiguration is used for all baselines on all environments. On antmaze, we perform a hyperparameter sweep on both p = {0.5, 0.75, 0.9} and the geometric distribution parameter ? = {0.99, 0.995, 0.997} on the large maze with the top right goal and find that p = 0.9 and 7 = 0.99 works the best. We also use these parameters for the Visual antma ze experiments. On sncco, we use the same p but raise the discount y to match the environment discount rate of 0.995. On humanoi i dmaze, we perform a sweep over p- {00.5,0. 0.75, 0.9} on humanoi i dmaze medi um-nuviiivv navi natigate and find that p 0.75 works best. We still use 7 0.995. On Scene and Cube, we do similar sweep for p on the first task of Scene, Single Cube, and Double Cube, and find that p = 0.5 works best. We still use y = 0.995. For kitchen, we perform a sweep on the parameter p = {0.2, 0.5, 0.75, 0.9} and find that 0.75 works best. We use y = 0.99. We take the minimum of one random critic for antmaze, visual-antma ze, antsoccer, and humano i dmaze, and the minimum of two random critics for kitchen, scene, and cube. For antsoccer, we performed a sweep on for each method over taking a minimum of or 2 random crities, and found that taking minimum of 1 critic was always better. For all methods, we use the same image encoder used in RLPD (Ball et al., 2023) for the vis risllll-aa -attaaz task, with latent dimension of 50 (encoded image is 50 dimensional vector), which is then concatenated with proprioceptive state observations.
antsoccer-aa aarennn-nvvigal
EXPLORE. We directly use the official open-source implementation. The only difference we make is to adjust the RND coefficient from 1.0 to 2.0 and additionally add such bonus to the online replay buffer (the original method only adds to the offline data). Empirically, we find a slightly higher RND coefficient improves performance slightly. The SAC configuration is the same as that of the DBC+JSRL baseline. We found that taking the minimum of one random critic on vi sua aall-anntzzzze worked better for EXPLORE than taking the minimum of two, all non-skill based baselines use this hyperparameter value.
This baseline is essentially our method but without using the trajectory encoder in the VAE to label trajectory segments (with high-level skill action labels), so all stated implementation decisions also apply to SUPE. Instead, we treat it directly as a high-level RL problem with the low-level skill policy completely frozen. The SAC agent is the same as the previous agents, except for on Visual AntMaze, where taking the minimum of 2 critics from the ensemble leads to better performance for SUPE, so we use this parameter setting for all skill-based benchmarks. We compute the high-level reward as the discounted sum of the rewards received every H environment steps. During the 5 x 10 steps before the start of training, we sample random actions from the state-based prior. For Visual AntMaze, we use the learned image encoder from the VAE to initialize both the critic image encoder and the RND network. If using ICVF, we initialize the RND network and critic encoder with the ICVF encoder instead.
TRAJECTORY SKILLS.
HILP SKILLS. This bascline is the same as the one above but with the skills from a recent unsupervised offline skill discovery method, HIIPP (Park et al., 2024b). We use the official open-source implementation from the authors and run the
16

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
pretraining to obtain the skill policies. Then, we freeze the skill policies and learn a high-level RL agent to select skills every H steps.
SUPE (HILP). This novel baseline is the same as HILP SKILLS, except that we also relabel the offline trajectories and use them as additional data for learning the high-level policy online (similar to our proposed method). To relabel trajectories with the estimated HILP skill, we compute the difference in the latent representation of the final state 8n and (ouuur((s)-uunrple
initial state 8o in the trajectory, so å 4 ( (1tt)))t policies use a normalized vector as input. The high-level RL agent is the same as our method, except the skill relabeling is done using the latent difference rather than the trajectory encoder.
We normalize the skill vector since the pretrained HILP
D. Additional Related Work
Options framework. Many existing works building hierarchical agents also adopt the options framework (Sutton et al., 1999; Menache et al., 2002; Chentanez et al., 2004; Mannor et al., 2004; Şimşek & Barto, 2004; Şimşek & Barto, 2007; Konidaris, 2011; Daniel et al., 2016; Srinivas et al., 2016; Fox et al., 2017; Bacon et al., 2017; Kim et al., 2019; Bagaria & Konidaris, 2019; Bagaria et al., 2024). The options framework provides a way to learn skills with varying time horizons, often defined by learnable initiation and/or termination conditions (Sutton et al., 1999). We use skills with a fixed horizon length because allows us to avoid the additional complexity of learning initiation or termination conditions, and frame the skill pretraining phase as a simple supervised learning task.
Reinforcement learning with observation-only offline data. In this problem setting, the offline data only contains observations (e.g., videos) (Ma et al., 2022; Ghosh et al., 2023; Song et al., 2024). The closest to our work are Ma et al. (2022) and Ghosh et al. (2023), which focus on pre-training on observation-only offline data to extract good image representations that are suitable for down-stream RL tasks. Different from their work, we pre-train low-level skills rather than representations for online learning.
E. Domain Details
D4RL antmaze with Additional Goal Locations. D4RL antmaze is a standard benchmark for offline-to-online RL (Fu et al., 2020; Ball et al., 2023) where an ant robot needs to navigate around a maze to a specified goal location. We benchmark on three mazes of increasing size, antmaze antmaze-medi um, antmaze- cattmaaze-aaggee large, and antmaze-ultza ra. We take the D4RL dataset for medium and large mazes as well as the dataset from Jiang et al. (2022) for the ultra maze (we use the diverse version of the datasets for all these layouts). We then remove the termination and reward information from the dataset such that the agent does not know about the goal location a priori. For each of the medium, large, ultra mazes, we test with four different goal locations that are hidden from the agent. See Figure 2a for visualization of the mazes and the four goal locations that we use for each of them. We use - 1/0 sparse reward function where the agent receives - 1 when it has not found the goal and it recieves 0 when it reaches the goal location and the episode terminates. The ant always starts from the left bottom corner of the maze, and the goal for the RL agent is to reach the goal consistently from the start location. It is worth noting that the goal is not known a priori and the offline data is unlabeled. In order to learn to navigate to the goal consistently, the agent first needs to traverse in the maze to gather information about where the goal is located.
D4RL kitchen is another standard benchmark for Franka robot arm is controlled to interact with various objects in a simulated kitchen scenario. The desired goal is to complete four tasks (open the microwave, move the kettle, flip the light switch, and slide open the cabinet door) in sequence. In the process, the agent attains a reward equal to the number of currently solved tasks. The benchmark contains three datasets, kitchen-mii xed, kitchen partial, and kitchen- comppeete.. casiest dataset, which only has demonstrations of the four tasks completed in order. kitchen-partial adds additional tasks, but the four tasks are sometimes completed in sequence. kitchen -ii xed is the hardest, where the four tasks are never completed in sequence. To adapt this benchmark in our work, we remove all the reward labels in the offline dataset.
RL (Fu et al., 2020; Nakamoto al., 2024), where
sitchen--ompla ete is the
We also include four additional domains repurposed from goal-conditioned offline RL. benchmark (Park al., 2024a).
OGBench Aumanoidmaze is a navigation task similar to antmaze, but with the Ant agent replaced by a Humanoid agent. humanoi i dmaze is much more challenging than antmaze because Humanoid control is much harder with much higher action dimensionality (21 vs. 8). We benchmark on all six datasets in the henchmark. They involve three mazes of increasing size, humanoid iedaanodd-mediunn bumanoid-lare and humanoid- -iian,, and two types of datasets,
17

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
navigate (collected by noisy expert policy that randomly navigates around the maze) and stitch (containing only short segments that test the algorithm's 's ability to stitch them together).
OGBench antsoccer is navigation task similar to antmaze, but with the added complexity where the Ant agent must first travel to the location of a soccer ball, then dribble the soccer ball to the goal location. We benchmark on the two navigate datasets, antsoccer--eeee rena-nava gate and antsooccr--mett unaevdia-nvviaaae We only benchmark on task-1.
OGBench cube-* and scene are manipulation domains with a range of manipulation tasks for each domain. Cube domains involve using robot arm arrange blocks from initial state to a goal state, which requires pick and place actions to move and/or stack blocks. We use both the cube s single and -ooble domains which include one and two blocks, respectively. We also benchmark on the Scene environment with two buttons, a drawer, and a window. The most difficult task requires the agent to perform 8 atomic actions: unlock the drawer and window, open drawer, pick up block, place block in drawer, close drawer, open window, lock drawer and window. We benchmark on all tasks for culbe-singles
mube--dubble and scene, and use the p play datasets collected by non-Markovian expert policies with temporally correlated noise. Unlike the reward function used in the singletast versions of these tasks introduced after this paper was finished, we use a binary -100 reward on the completion of the entire episode create a maximally sparse task. This is different from the reward in the recently introduced singletask versions of these tasks, where the reward is proportional to the number of subtasks completed (1 subtask for cube-s i inlle, e, 2 for cube- doieee-doulll e, and 5 for scene).
Aside from the state-based domains above, we also consider a visual domain below to test the ability of our method in scaling up to high-dimensional image observations.
visual-antmaze is a benchmark introduced by Park et al. (2023a), where the agent must rely 64 x 64 image observations of its surroundings, as well proprioceptive information including body joint positions and velocities to navigate the maze. In particular, the image is the only way for the agent to locate itself within the maze, so successfully learning to extract location information from the image is necessary for successful navigation. The floor is colored such that any image can uniquely identify a position in the maze. The maze layout is the same as the large layout in the state-based D4RL maz benchmark above and we also use the same additional goals. The reward function and the termination condition are also the same as the state- based benchmark.
F. ICVF Implementation Details and Ablation Experiments for visual-antmarz
We use the public implementation from the authors of (Ghosh et al., 2023) and run the ICVF training for 75000 gradient steps to obtain the pre-trained encoder weights, following (Li et al., 2024). Then, we initialize the encoder of the RND network with these weights before online learning. It is worth noting that this is slightly different from the prior work (Li et al., 2024) that initializes both the RND network and the critic network. Figure 7, we examine the performance of SUPE, TRAJECTORY SKILLS, and EXPLORE with and without ICVF. Both of the baselines perform much better with the ICVF initialization, suggesting that ICVF might play an important role in providing more informative exploration signal. SUPE, without using ICVF, can already outperform the baselines with ICVF. By both extracting skills from offline data and training with offline data, we are able to learn better from less informative exploration signals. We also observe that initializing the critic with ICVF (as done in the original paper (Li et al., 2024)) helps improve the performance of EXPLORE some, but does not substantially change performance.
18

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
SUPE
ExPLORe
EXPLORe (ICVF, Critic + RND) ExPLORe (ICVF, RND Only)
Trajectory Skills
ICVF Ablation
ExPLORe Critic ICVF Ablation
1.00
1.00
IQM Normalized Retum
ICVF
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.2
0.3
Environment Steps (x10%)
Figure 7: Success rate on the visual-antmaze environment with and without ICVE. SUPE works well without ICVE, almost matching the original performance. However, TRAJECTORY SKILLS achieves far worse performance without ICVF, which shows that using offline data both for extracting skills and online learning leads to better utilization of noisy exploration bonuses. Initializing the critic with ICVF helps for EXPLORE, but does substantially change performance.
SUPE
SUPE (KL)
ExPLORe
antmaze-medium
antmaze-large
antmaze-ultra
1.00
1.00
1.00
IQM Normalized Return
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x106)
Figure 8: IQM normalized return three antmaze mazes, comparing SUPE with KT. regularized alternative (OuRS (KL)). that SUPE consistently outperforms Ours (KL.) on all three mazes, with initial learning that is at least as fast and significantly improved asymptotic performance. Only SUPE is able to meet or surpass the asymptotic performance of EXPLORE on all mazes.
G. KL Penalty Ablation
In Figure compare the performance SUPE with a version of our method that uses KL-divergence penalty with the state-based prior (as used in previous skill-based method (Pertsch et al., 2021)), OURS (KL). In SUPE, as discussed in Section 4, we borrow the policy parameterization from (Haarnoja et al., 2018) and adopt tanh policy parameterization with entropy regularization on the squashed space. Pertsch et al. (2021) parameterize the higher level policy as normal distribution and explicitly constrained to a learned -tateddepeneent prior using a KL-divergence penalty, with tempera- ture parameter that is auto-tuned match some target value by using dual gradient descent on the temperature parameter. They do not use entropy regularization. Keeping everything else about our method the same, we instantiate this alternative
19

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
policy parameterization in Ours (KL). We sweep over possible target KL-divergence values (5, 10, 20, 50) and initial val- ues for the temperature parameter (100, 1, 0.1) using the performance antma ze-large, but find that these parameters do not substantially alter performance. As shown in Figure 8, SUPE performs at least as well Ours (KL) in the initial learning phase, and has better asymptotic performance on all three mazes, matching or beating ExPLORE on all three mazes. It seems likely that not having entropy regularization makes it difficult to appropriately explore online, and that explicitly constraining to the prior may prevent further optimization of the policy. Attempts at combining an entropy bonus and KL-penalty lead to instability and difficulty tuning two separate temperature parameters. Additionally, domain, the KL objective is unstable, since at some states the prior standard deviation is quite small, leading numerical instability. In contrast, adopting the tanh policy parameterization from (Haarnoja et al., 2018) is simple, performs better, and encounters none of these issues in our experiments.
Kitchen
H. Sensitivity Analysis
We picked one representative task from each domain: antmaze-large--to-rrigtt kitchen-mi xed, humanoi dmaa ze zeniiatttttttt
cube-double-taskt. scene-taskl, and antsoccer arena navigate. Figure shows the performance of method with different RND coefficients. We that an RND coefficent of zero leads poor performance antmaze and jimanoiddazzi and that the best nonzero RND) coeffcient varies between environments. For all experiments in the paper, we selected the middle value of eight, and kept it the same across all domains. Figure 10 shows the performance of our method with different skill horizon lengths. We see that while horizon length of attains better final performance on antmaze- large top right goal at the cost of slower initial exploration, performs worse than a horizon length 4 on other environments. A longer horizon length of seems to perform significantly better soccer and kitchen, but performs the same or substantially worse in all other tasks. We found that using a horizon length of generally works well all tasks, used this length for all trajectory-skill based experiments in this paper.
cube-single-taste
=
0
a = 2
α = 8 (Ours)
= 16
antmaze-large-too-rihh
kitchen-mixed
humanoid-giant-stitee
cube-single-1
1.00
1.00
1.00
1.00
0.75
0.75
0.75
W
0.75
IQM Normalized Return
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
cube-double-1
scene-play-1
antsoccer-arena
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x 106)
Figure 9: Sensitivity analysis the RND coefficient. RND is essential strong performance antmaze and bumanoidmaze. The best coefficient varies between tasks.
20

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
H
2
4 (Ours)
8
antmazz-large-toprrghh
kitchen-mixed
numanoid-giant-stoo
cube-single-1
1.00
1.00
1.00
1.00
0.75
WW
IQM Normalized Return
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.0
0.2
0.0
0.2
0.0
0.2
0.3
cube-double-1
scene-play-1
antsoccer-arena
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.1
0.2
0.0
0.2
0.0
0.2
Environment Steps (x106)
Figure 10: Sensitivity analysis of the skill horizon length. A horizon length of generally performs the best across all environments. A horizon length of 2 performs relatively well in antmaze zermmazze-rreeee large, but performs poorly in all other environments. A longer horizon length of performs better than horizon length of 4 in soccer and kitchen, but performs much in several other tasks.
State-based D4RL Results
In this section, summarize experimental results two state-based D4RL domains: antmaze and kitchen. Figure 11 shows the comparison of our method with all baselines.
(HILP)
Trajectory
EXPLORe
DBC+JSRL
antmazza-eediim
antmane-large
kitchen-mixed
kichannparttal
kitchen-camplete
I0M No-malzzed Return
0.5
22 0.5
61
42
0.5
63
0.1
0.2
02
63
Environment Steps (x110))
Figure 11: IQM normalized return on individual antmaze and kitchen tasks. SUPE achieves the strongest performance on all tasks. TRAJECTORY SKILLS learns much slower on all antmaze tasks. EXPLORE struggles to learn on kit chen, and performs worse maze size increases. None of the other baselines competitive any tasks, Each curve is average four goals with 8 seeds for antmaze, and seeds kitchen.
I.1. Exploration Efficiency
Figure 14 shows the percentage of the maze that the agent has covered throughout the training. The coverage skill- based methods that do not use prior data during online learning. TRAJECTORY SKILLS and HILP SKILLS, significantly lags behind baselines that use offline data after 50000 environment steps. Many methods achieve similar coverage um, likely because the maze is too small to differentiate the different methods. SUPE is able to achieve the highest coverage on the mntmaazeuulte and is only surpassed on antmaze lareeeeaareeeee by SUPE (HILP), which has high first goal times and slow learning. Thus, the coverage difference can likely be at least partially attributed to SUPE (HILP) struggling to find the goal and continuing to explore after finding the goal. All non-skill based methods struggle to get competitive coverage levels on antma ze-laeelleeee -lragee and antmaze mattmaze-ultae. ta.. This suggests both pretraining skills and
21

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
the ability leverage prior data online crucial for efficient exploration, and method effectively compounds their benefits.
1.2. Full D4RL AntMaze Results
We evaluate the success rate of the our algorithm compared to the same baseline suite as the main results section for each individual goal and maze layout and report the results in Figure 12. We also include EXPLORE both with and without an online RND bonus. Online RND helps EXPLORE the most for the antmaze-mede um bottom-right goal, where there is sparse offline data coverage for considerable radius around the goal. We hypothesize that with the absence of online RND, the agent encouraged to only stay close to the offine dataset, making it more difficult to find goals less well-covered regions, On the flip side, for some other goals with better offline data coverage, like the antmaze-laat large top-right goal, online RND can make the performance worse. For every goal location, SUPE consistently matches or outperforms all other methods throughout the training process.
Trajectory HILP Skills
ExPLORe ExPLORe (No Online RND)
DBC+/SRL
SUPE (HILP)
Online
meedium-too-riall
medium-topilet
medium-bottom-right
medium-tap-cente
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0,3
IQM Normalized Retur
large igh
lap-e-toplleft
large: bottorn right
large center
1.00
1.00
1.00
1.00
0.75
0.75
0.75-
0.75
0.50
0.00
0.1
0.0
0.1
0.2
0.3
0.1
top right
top
right
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.0
6.1
0'2
0/3
0.0
0.1
0.3
Environment Steps (x106)
Figure 12: IQM normalized return by goal location. The addition of online RND in ExPLORE leads to better performance goals with less offline data coverage, and slightly worse performance goals well-represented in the dataset. SUPE consistently matches outperforms other methods all goals throughout training.
We also evaluate the coverage at every goal location for every method for each maze layout and show the result in Figure 13. The coverage varies from goal location to goal location as some goal locations are harder to reach. Generally, the agent stops exploring once it has learned to reach the goal consistently. SUPE consistently has the best initial coverage for out of 12 goals, though sometimes has lower coverage compared to other methods later in training. However, this is likely due in large part successfully learning how to reach that goal quickly, and thus not exploring further.
22

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Trajectory
EXPLORe ExPLORe Online RND)
DBC+JSRL
SUPE (HILP)
HILP Skills
Online
medium-top-rigg
meedium-to--lett
medium boltom-right
medium-lan top
1.00
1,00
1.00
1.00
0.75
0.75
0.75
0.75
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
lorge-top-right
large-top-les left
bottommoill
lorge-top-center
1.00
1.00
1.00
1.00
IQM Coverage
0.75
0.75
0.75
0.75
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
ultra-top-right
ultra-top-left
ultra- bathom--tttht
uttrattop-cenor
1.00
1.00
1.00
1.00
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x106)
Figure 13: IQM coverage for every goal location three antmaze environments. There is significant variation between goals, and SUPE consistently has best initial coverage performance on 12 goals. Flattening coverage compared to other methods can he at least partially attrihuted to having already found the goal, and sucessfully optimizing reaching that goal, rather than continuing to explore after already finding the goal.
SUPE
Trajectory Skills
ExPLORe DBC+JSRL
Online
SUPE (HILP)
HILP Skills
antmaze-medium
antmaze-large
antmaze-ultra
1.00
1.00
1.00
0.75
0.75
0.75
IQM Coverage
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x 10%)
Figure 14: IQM coverage on three different AntMaze mazes, averaged over runs on four goals. SUPE has the best coverage performance on the challenging antmaze-ut ra, and is only passed by SUPE (HILP) on antmaze- weazeellrre... TRaJECtoRy SKILLS and HILP SKILLS struggle to explore after initial leamning and ONLINE ONLY and DBC+JSRL generally perform poorly all time steps.
23

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
Methods without Pretraining
Methods with Pretraining
Maze Layout
Goal Location
ONLINE ONLY
EXPLORE
DBC+JSRL
TRAJECTORY SKILLS
HILP SKILLS
SUPE (HILP)
SUPE
sop-left
71 5.0
27 3.2
60 8.1
21 4.1
120 =
27 6.2
14 3.1
top-right
100 16
29 2.8
85
160
72 36
22 3.2
mediur
bottom-righe
35
cencer
+
71
260
1.7
Aggregated
150 14
40 00 2.0
130 10
1
210 17
170 1:
1.8
top-left
72 10
33 2.9
52 3.3
22 4.2
300 1:0
300 0.0
21 1: 2.8
Lop- right
220 + 20
49
220 = 28
190 0
280 = 20
110 36
27 2.6
large
bottom-right
280 15
34 1.8
160 22
22
280
260
1.8
top-center
Aggregated
200 = 8.9
41 2.7
140 = 13
13
270 12
180 13
1.7
sop-left
76 166277 7.0
34 14:44
91 =: 11
36 # 11
39 =: 21
15 =: 5.3
17 = 3.6
top-right
300 + 0.0
92 20
290 7.8
120 14
260 + 19
150 32
37 5.5
ultra
bottem-right
300 1: 0.0
70 8.0
300 0.0
16
240
34 6.0
top-center
230 230
29 5.5
230
4.4
Aggregated
Aggregated
190 L. 6.9
46 2.3
160 6.3
80 5.9
210 1:
130 7.7
1.4
Table 3: The number of environment steps (x10*) taken before the agent find the goal. Lower is better. The first goal time is considered to be 300 x 10 steps if the agent never finds the goal. We see that our method is the most consistent, achieving performance good as or better than all other methods in each of the 4 goals across 3 different maze layouts. The error quantity indicated is standard error over seeds. The method that has the lowest mean is in bold and all the other methods with values that are not statistically significantly higher also in bold. We the one-sided f-test (p- 0.05) and bold all the methods that statistical significantly worse than best method (the method achieves the best mean performance).
I.3. D4RL Play Dataset
Since there is limited performance difference between the diverse and the play datasets, we only report the performance on the diverse datasets. For completeness, also include the results of the play datasets in Figure 15, The results on the play datasets are consistent with our results in the main body of the paper where our method outperforms all baseline approaches consistently with better sample efficiency.
SUPE
Trajectory Skills
EXPLORe
DBC+JSRL
intmaae-meddum-pa
antmaze-laa -aazze--argeeplaa-
1.00
1.00
IQM Normalized Return
MMM
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x10%)
Figure 15: IQM normalized return of our method on the play datasets. SUPE outperforms all baselines, similar to the results on the diverse datasets (Figure 11). We average over 4 seeds.
J. OGBench Results
In this section, we include the full results on individual tasks for each of the five OGBench domains (humano dmaze, cube-single, cube-double, scene, and antsoccer) (Park et al., 2024a).
24

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
J.1. HumanoidMaze
As shown in Figure 16, our method substantially outperforms all prior methods on the difficult humanoi dma ze envi- romment. It is the only method to achieve nonzero return the more difficult large and qiant mazes, and performs approximately four times better than the next best baseline, TRAJECTORY SKILLS, the medium environments, These results show that on difficult, long horizon tasks, using offline data during online learning is essential for strong exploration performance.
SUPE SUPE (HILP)
Trajectory Skills
ExPLORe DBC+JSRL
HILP Skills
medium-navigate
large-navigate
giant-navigate
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
IQM Normalized Return
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
medium-stitch
large-stitch
giant-stitch
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x10%)
Figure 16: IQM normalized return six humanoidmaze tasks. SUPE is the only method that obtain more than 0.5 IQM normalized return. All baselines either completely fail or only achieve 0.2 IQM normalized return on easier mazes (e.g.. TRAJECTORY SKILLS on medium-rave navigate and pediium-stee sticch).. We average over seeds.
J.2. Cube and Scene
As shown in Figure 17, SUPE matches or outperforms the next best baseline on of the 15 tasks. The novel baseline that introduce SUPE (HILP) which also uses offline data for skill pretraining and online learning outperforms SUPE on three of the scene tasks, which further shows how using offline data twice is critical. Additionally, on one of the difficult cube-double manipulation tasks, SUPE is the only method to achieve nonzero reward. Non-skill based methods EXPLORE and DBC+JSRL performs reasonably well on the easier cube-single and scene-taske but struggle to achieve significant return on the more difficult tasks, which shows how extracting structure from skills critical for solving challenging tasks.
25

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
SUPE SUPE (HILP)
Trajectory Skills
EXPLORe DBC+JSRL
HILP Skills
cube-single-1
cube-single-2
cube-single-3
cube-single-4
cube-single-5
1.00
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
IQM Normalized Return
0.0
0.1
0.3
0.1
0.2
0.2
0.3
0.0
0.1
0.3
0.1
0.2
0.3
cube-double-1
cube-double-2
cube-double-3
cube-double-4
cube-double-5
1.00
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
scene-1 1
scene-2
scene-3
scene-4
scene-5
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2 0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x106)
Figure 17: IQM normalized return on individual tasks in cube and scene domains. cube has 10 tasks in total (5 on cube-single and 5 on cube-double). scene has tasks total. We average over 8 seeds.
J.3. AntSoccer
shown in Figure 18, SUPE (HILP) outperforms baselines on both antsoccer-arent and antsoccer-medi We also see that on both tasks, SUPE and SUPE (HILP) outperform TRAJECTORY SKILLS and HILP SKILLS, which demonstrates the importance of using offline data twice accelerate online exploration.
SUPE
Trajectory Skills
EXPLORe DBC+JSRL
SUPE (HILP)
HILP Skills
antsoccer-rena
antsoccer-medium
1.00
1.00
IQM Normalized Return
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x10%)
Figure 18: IQM normalized return on individual tasks antsoccer domain. We average over seeds.
26

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
K. Sensitivity to Offline Data Quality
To provide more insights on how the quality of the offline data affects the performance of our method, we perform additional analysis on the an taaze- tmaze-largge environment.
SUPE
Trajectory Skills
ExPLORe
DBC+JSRL
antmazz-llarg--exne
antmaae--large-ttice
antmaze-large-exppe
1.00
1.00
1.00
1.00
1QM Normalized Return
0.75
0.75
MMMM
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x10%)
with different offline datasets. Expert: collected by non-noisy expert policy; Navigate: collected by noisy expert policy that randomly navigates the from OGBench (Park et al., 2024a); Stitch: collected by the same noisy expert policy but with much shorter trajectory length (also from OGBench (Park et al., 2024a)); Explore: random exploratory trajectories collected by moving the ant in random directions ra-saedled every 10 environment steps, with large action noise added (also from OGBench (Park et al., 2024a)). We average over 4 seeds.
Figure 19: Performance comparison on antmaze--arge--toprisi
K.1. Expert Data to Random Exploratory Data
In Figure 19, we consider four additional offline datasets for the mettmaze-large task with decreasing dataset quality:
1. Expert: collected by non-noisy expert policy that we train ourselves.
2. Navigate: collected by a noisy expert policy that randomly navigates the maze (from OGBench (Park al., 2024a)).
3. Stitch: collected by the same noisy expert policy but with much shorter trajectory length (from OGBench (Park et al., 2024a))
4. Explore: collected by moving the ant in random directions, where the direction is re-sampled every 10 environment steps. A large amount of action noise is also added (from OGBench (Park et al., 2024a)).
As expected, the baseline EXPLORE shows gradual performance degradation from Expert to Navigate, to stitch, and to Explore. All skill-based methods (including our method) fail completely on Explore. This is to be expected because the Explore dataset contains too much noise and the skills extracted from the dataset are likely very poor and meaningless. The high-level policy then would have trouble composing these bad skills to perform well in the environment. On Navigate and St Stitch, our method matches or outperforms other baselines, especially on the more challenging Stitc dataset where it is essential to stitch shorter trajectory segments together. On Expert, EXPLORE performs slightly better than all other methods. We hypothesize that this is because with the expert data, online learning does not require as much exploration, and skill-based methods are mostly beneficial when there is a need for structured exploratory behaviors. Since the expert dataset has very different distribution (much narrower) than the others, per- formed sweep the KI. coefficient in VAE training over (0.01, 0.05, 0.1, 0.2, 0.4, 0.8), and found that 0.2 performed best, so we used these skills for both SUPE and TRAJECTORY SKILLS on this dataset.
K.2. Data Corruptions
In this section, we study how robust method is against offline dataset corruptions. We perform ablation study the antmaze domain on the large maze layout with two types of data corruption applied to the offline data:
1. Insufficient Coverage: All the transitions close to the goal (within a circle with radius of 5) are removed.
27

---

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
2. 5% Data: We subsample the dataset where only 5% of the trajectories used for skill pretraining and online learning.
We report the performance on both settings in Figure 20. For the Insufficient Coverage setting, our method learns some- what slower than the full data setting, but can still reach the same asymptotic performance, and outperforms or matches all baselines the same data regime throughout the training process. For the 5% Data setting, our method also reaches the same asymptotic performance as in the full data regime, and outperforms matches all baselines throughout training. Overall, among the top performing methods in the antmaze domain, method is the most robust, consistently out- performing the other baselines that either do not use pre-trained skills (ExPLORE) or do not use the offline data during online learning (TRAJECTORY SKILLS) in these data corruption settings.
SUPE
HILP Skills
Online SUPE (Full Dataset)
Trajectory Skills
ExPLORe
5% Data
Insufficient Coverage
1.00
1.00
IQM Normalized Return
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x106)
Figure 20: Data corruption ablation on state-based antmaze-large. Top: success rate different methods on these data
corruption settings. Bottom: Visualization the data distribution for each corruption setting. We experiment with two data corruption settings. Our method performs worse than the full data setting but still consistently outperforms all baselines. We average over seeds.
28


Reviews:
[{"review_id": "tANnHcmbfj", "reviewer": "Reviewer_2HE7", "summary": "The paper studies how to leverage skills pre-trained by unsupervised RL on unlabeled data to improve exploration while solving downstream tasks online. The proposed method (SUPE) first pre-trains skills via a trajectory VAE, then labels offline data with optimistic rewards, and finally trains a policy mapping states into skill vectors while interacting with the environment. Experimentally, SUPE is shown to outperform existing baselines and several challenging continuous control tasks.\n\n## Update after rebuttal\n\nThe rebuttal clarified all my doubts, and I think the new experiments will provide good additional value to the paper. I am thus voting for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and insightful comments. We especially appreciate your clarifying question on the setting of our work. Regarding your concern on not comparing to “diversity-encouraging” objectives, we conducted additional experiments with METRA and showed that METRA finds a reward signal online much slower than our method. In addition, our experiments already involved a comparison to a variant of our method that leverages “diversity-encouraging” objectives in the offline pre-training phase (SUPE (HILP)). For your concern about the dependence on offline data quality, our ablation studies in the appendix of our original submission demonstrated the robustness of our approach to offline datasets with different levels of quality, quantity, and coverage.\n\n**Experiments with “diversity-encouraging” methods**\n\nWe conducted additional experiments to evaluate how fast a diversity-encourage objective (METRA) is able to find the reward signal (reach the desired goal) online. In the table below, we report the # of env steps that METRA took before the learned skills could reach the desired goal from the initial state (6 seeds). Our method took significantly fewer environment steps (on the order of 1/100x) compared to METRA. These results suggest that the $\\texttt{antmaze}$ domain poses a significant exploration challenge, and naively using a diversity-encouraging objective (METRA) for online exploration can be extremely sample inefficient (100x worse than our method). The table can be viewed [here](https://anonymous.4open.science/r/supe-rebuttal-8279/README.md).\n\nThese results may not be too surprising, since METRA does not leverage offline data. One may wonder what might happen if we adapt METRA to use offline data. This is actually quite similar to SUPE (HILP), a variant of our method which uses a “diversity-encouraging” objective for offline pretraining instead of a behavioral cloning objective. Similar to METRA, HILP learns a metric space of states that preserves temporal distance. SUPE (HILP) is otherwise identical to SUPE (e.g., uses the offline data for both skill pretraining and high-level agent learning online) with the only difference being that the offline skill pretraining objective is different. SUPE works better across six of the eight domains we experimented on, and is competitive with SUPE (HILP) on the other two. \n\nOur method is not tied to a specific choice of skill pre-training method or online exploration bonus. We used VAE behavioral cloning skill pretraining and an RND exploration bonus because they are simple and effective in the benchmark tasks considered. \n\n**Does SUPE only work on very high quality data?**\n\nIn Appendix J of our submission, we had some ablation studies on datasets with different levels of quality. In Figure 18&19, we show that our approach is reasonably robust against various dataset corruptions (e.g., insufficient coverage around the goal location, noisy experts with short trajectory lengths). In addition, our approach can be easily adapted to work in conjunction with different skill pre-training methods. On datasets that are diverse and “high-coverage”, we can leverage pre-trained skills that are more suitable for these properties (e.g., HILP) [1]. As mentioned in the previous section, one variant of our approach (SUPE (HILP)) leverages such skills.\n\n[1] \"Foundation policies with hilbert representations.\"\n\n**“What's the difference between this setting and (Kim et al., 2024)? Could you motivate in what real problems this is useful?”**\n\nYour understanding is correct. In the O2O paper, the reward function is available at the start of the online phase. Our setting is harder since we don’t have access to the reward function and the reward signals must be obtained through environment interactions. \n\nIn robotics, manually specifying a good reward function is challenging and it often suffers from misspecification issues (when the optimal policy of the specified reward misaligns with the expected optimal policy). For example, if you would like to specify a reward function for a home robot to clean a house, there are a lot of things to consider: should the robot clean the kitchen? What if the robot bumps into a wall? How do we define if cleanliness is acceptable? A much more appealing interaction with the robot would be that the robot first does some exploratory cleaning behavior (informed by its prior experience, possibly from its prior cleaning experience in another house) and then the user gives feedback (sparse reward signal) to the robot on whether the cleaning is done properly in a safe manner, and the robot improves upon the signal provided. We believe our work is an important first step towards making this part of our reality in the future.\n\nThank you again for your constructive feedback and detailed reviews. Please let us know if you have any other concerns or questions. **If we have successfully addressed all your concerns, could you kindly raise your rating?**", "claims_and_evidence": "The main claim that SUPE enables more efficient exploration and faster learning at downstream is supported by a good empirical evidence on several challenging exploration problems. Maybe one issue is that SUPE is compared with very few baselines: only two are existing algorithms, while the others are variants of the proposed method. While I guess this can be explained by the fact that there exist very few methods for efficient exploration in this context, I still wonder if other existing techniques could be adapted to provide further evidence about the complexity of the considered problems and the efficiency of SUPE. For instance, could \"diversity-encouraging\" objectives of recent unsupervised skill discovery methods (like DIAYN or METRA) be used as a regularizer to incentivize exploration during the online learning phase?", "methods_and_evaluation_criteria": "The proposed method makes sense. It is simple and intuitive. However, I think it has some limitations:\n1. It seems quite incremental since it combines many existing techniques (although in a smart and non-trivial way)\n2. The idea of re-using pre-training data during the online exploration phase, while effective as shown experimentally, does not seem very practical. Now skills are pre-trained on relatively small datasets, but imagine you want to scale this up to the pre-training settings of, e.g., recent large foundation models (say, with hundreds of terabytes of data). We cannot expect that these data can be deployed and moved around together with the pre-trained skills to be reused for each new task we encounter. I think the approach would be much more convincing and practical if it found a way to \"compress\" relevant knowledge to drive exploration at downstream in a learned model rather than keeping the full pre-training dataset\n3. The way skills are pre-trained, essentially through behavioral cloning, together with the experimental results make me wonder whether this approach works only on very high quality data (e.g., expert demonstrations), and possibly that covers well the space that will have to be explored at downstream. If this is the case, I think it is another significant limitation, especially because the exist skill pretraining methods (like HILP) that can make use of \"high-coverage\" data (like EXORL) which does not contain any useful behavior, but from which useful behaviors can still be extracted\n\nThe evaluation protocol makes sense. The considered tasks are sufficiently diverse and challenging to showcase exploration capabilities. Again, the only limitation may be about the pre-training data: if it is given by high-qualitity demonstrations that cover well the space to be explored at test time, than SUPE's performance would not be very surprising", "theoretical_claims": "None", "experimental_designs_or_analyses": "As far as I understand from the details given in the paper, the experimental protocol is sound and the results are convincing.", "relation_to_broader_scientific_literature": "They relate to the broader field of unsupervised pre-training followed by task-specific adaptation", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n- the paper studies a significant problem\n- the paper is well written\n- the proposed approach is simple and effective\n\nWeaknesses (see above for details)\n- the proposed approach is a bit incremental\n- need of high-quality pretrainig data\n- need to retain the pretraining data for the adaptation phase", "other_comments_or_suggestions": "None", "questions_for_authors": "1. What's the difference between this setting and the recently proposed \"Unsupervised-to-Online Reinforcement Learning\" (Kim et al., 2024)? Only that the reward function here is not given but rather only observed through samples from the environment?\n2. If this is the case, could you motivate in what real problems this is useful? For instance, in robotics most of the time the reward is chosen by the designer, so it is not unreasonable to assume that the function itself is given at downstream instead of being \"discovered\" through interaction", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOWpzQeO3x", "reviewer": "Reviewer_D8KW", "summary": "The paper presents a new algorithm on combining offline data and online RL where one can use the offline data to learn skills, and online perform exploration in the state space and skill space jointly, and relabel offline data with the exploration bonus. The paper performs comparisons with a few baselines on several tasks, and demonstrated that the sample efficiency of the proposed algorithm is superior to the previous baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and insightful comments. We especially appreciate the additional references you point out and the clarifying question on UCB. For your concern on the dependence on quality of the offline data, we had ablation studies in our appendix that demonstrate the robustness of our approach on offline datasets with different levels of quality. For your question on how skill discovery helps, we provide an additional detailed analysis that visualizes the skill latent and show how leveraging skills effectively reduces the exploration horizon, resulting in a learning speedup online. \n\n**Analysis on offline datasets with different coverage and failure modes**\n\nWe had ablation studies on datasets with different levels of quality in the Appendix J and showed our approach is robust against different dataset corruptions. A notable failure mode is the $\\texttt{explore}$ dataset where the dataset contains completely random actions, and where the skill-pretraining methods we use in SUPE cannot learn meaningful behaviors. Aside from this failure case, SUPE is able to learn efficiently from offline datasets with limited coverage (e.g., missing transitions around the goal location, Figure 19, right), limited scale (e.g., 95% of trajectories removed, Figure 19, left), and suboptimal data (e.g., the $\\texttt{stitch}$ dataset, Figure 18). We will refer to these ablation studies and discuss the failure modes of our approach in the main paper in the camera ready version.\n\n**Detailed analysis on skill discovery**\n\nWe conducted additional analysis of the skill latent to demonstrate how skill discovery helps online exploration in one of our hardest domains, $\\texttt{humanoidmaze}$. In particular, we randomly sample 16 latent vectors from our skill latent and roll-out the corresponding low-level skill policy for an entire episode and visualize the x-y position throughout each of the 16 trajectories. For comparison, we also plotted the agent’s trajectories if actions were completely random. The graphs can be viewed [here](https://anonymous.4open.science/r/supe-rebuttal-8279/latent-viz/README.md).  We can see that fixed skill latents are able to navigate quite far from the starting point. Such structured navigating behaviors allow the high-level policy to effectively operate at a reduced exploration horizon. Instead of training a low-level agent to predict $H$ correct actions in a row, the high-level policy only needs to predict 1 correct action to achieve a similar effect. As the exploration horizon is effectively reduced by a factor of $H$, we are able to train the high-level policy in SUPE to explore more efficiently and solve the task much faster.\n\n**Additional references**\n\nThanks for the additional references. In our paper, we actually do have experiments using ICVF (Ghosh et al., 2023) on the high-dimensional image domain in Appendix E. ICVF is used complementary to our proposed method and our baselines. In our experiments (Figure 6), we found that our method works well without ICVF, achieving a much better performance than our baseline “Trajectory Skills” with or without the ICVF pre-trained representations. We also found that using ICVF representations further improves our method but only marginally. Even with ICVF, many of our baselines still failed to achieve significant return (e.g., ExPLORe as shown in Figure 6, right).\n\nConceptually, both VIP (Ma et al., 2022) and ICVF are very related but different from our work in two ways: 1) they focus on pre-training on observation-only offline data whereas we assume access to actions, 2) they focus on extracting good image representations from the offline pre-training, whereas we pretrain low-level skills and retain the offline data to be used as additional off-policy transitions for online learning.\n\nBoth (Sharma et al., 2022) and (Song et al., 2024) are also relevant and we will cite and discuss them (along with the two above) in the camera ready version of our paper. \n\n**Question about UCB**\n\nIn the RL literature, UCB is typically used on the optimal value function with a confidence interval (e.g., [1]). We take the same definition of the term UCB in our paper and to describe the upper-confidence bound $r_{\\mathrm{UCB}}$ of a reward function – with high probability the reward value is at most $r_{\\mathrm{UCB}}$. As what we stated in our method section, we followed prior work (Li et al., 2024) to instantiate this estimate of the bound using an RND network and a reward model. The reward model predicts the mean of the reward estimate and the RND provides an estimate on how wide the confidence interval is. \n\n[1] \"Minimax regret bounds for reinforcement learning.\"\n\nWe would like to thank you again for your constructive feedback and detailed reviews especially on the related work references. Please let us know if you have any other concerns or questions. **If we have successfully addressed all your concerns, could you kindly raise your rating?**", "claims_and_evidence": "The claim that the proposed algorithm performs better than previous baselines is backed by evidence on a wide range of tasks.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "While the experiment demonstrates evidence of the strong empirical performance of the algorithm, one might question the source of improvement comparing to previous baselines. As the paper already compared with the baselines which also performs active exploration with offline data, it seems like the key ingredient is the skill discovery. However, a more careful analysis of how learning the skill helps the exploration or learning is lacking. The paper will benefit a lot from such zoomed-in analysis on the benefit and understanding the role of skill discovery. \n\nAlso intuitively, the method seems to heavily depend on the coverage/diversity of the offline data. I believe the paper will also benefit from conducting an analysis on offline datasets with different coverage and discover the failure mode of the algorithm.\n\nIt would also be interesting to see the performance gap between using the proposed algorithm, but offline data actually has rewards labeled.", "relation_to_broader_scientific_literature": "I believe the paper provides a strong baseline on the offline to online with unlabeled offline data problem, which is beneficial to the community.", "essential_references_not_discussed": "I think the paper overall does a great job on the literature review on skill discovery and general offline to online rl, but I think it is missing the most relevant literatures on the o2o RL with unlabeled offline RL works (I am not sure each paper will provide a meanful baseline for this paper, but would be great to compare if any of them apply here), just to list a few:\n\nSharma, Archit, Rehaan Ahmad, and Chelsea Finn. \"A state-distribution matching approach to non-episodic reinforcement learning.\" arXiv preprint arXiv:2205.05212 (2022).\n\nMa, Yecheng Jason, et al. \"Vip: Towards universal visual reward and representation via value-implicit pre-training.\" arXiv preprint arXiv:2210.00030 (2022).\n\nGhosh, Dibya, Chethan Anand Bhateja, and Sergey Levine. \"Reinforcement learning from passive data via latent intentions.\" International Conference on Machine Learning. PMLR, 2023.\n\nSong, Yuda, Drew Bagnell, and Aarti Singh. \"Hybrid Reinforcement Learning from Offline Observation Alone.\" International Conference on Machine Learning. PMLR, 2024.", "other_strengths_and_weaknesses": "Overall I think the paper is not really providing any new technique, but the performance is strong enough to make some contribution. However, the experiment should not only include performance comparison, but also conduct more detailed analysis.", "other_comments_or_suggestions": "see above", "questions_for_authors": "one minor point is on the usage of the term ucb: I do not see reference to ucb in the paper, is the bonus used in the paper just a general exploration bonus (but not really ucb)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3L6AKbxGIq", "reviewer": "Reviewer_S75k", "summary": "The paper introduces hierarchical RL to train high-level policy to utilize the unlabel offline trajectories, and train reward estiamation, high-level action (skill) and extra offline data in online learning to guide policy learning. It claims to achieve efficient exploration performance in sparse reward tasks by pseudo-labeling each trajectory segment with an upper-confidence bound (UCB) reward.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and insightful comments. We especially appreciate the additional references you point out and the clarifying question on pseudo-labeling. For your question on how policy operates conditioned on different latent skills, we provide an additional detailed analysis that visualizes the skill latent and show how leveraging skills effectively reduces the exploration horizon, resulting in learning speedup online. \n\n**\"I suggest adding visualizations, such as those in Figure 4 of BeCL [Yang et al., 2023], to illustrate how the policy operates conditioned on different latent skills\"**\n\nWe conducted additional analysis of the skill latent to demonstrate how skill discovery helps online exploration in one of our hardest domains, $\\texttt{humanoidmaze}$. In particular, we randomly sample 16 latent vectors from our skill latent and roll-out our pre-trained low-level skill policy (corresponding to each of the sampled latent vectors) for an entire episode and visualize the x-y position throughout each of the 16 trajectories. For comparison, we also plotted the agent’s trajectories if actions were completely random. As shown in the figures [here](https://anonymous.4open.science/r/supe-rebuttal-8279/latent-viz/README.md), our unsupervised offline pretrained skills were able to navigate quite far from the initial positions even before any online learning. Such structured navigating behaviors allow the high-level policy to effectively operate at a reduced exploration horizon. Instead of training a low-level agent to predict $H$ correct actions in a row, the high-level policy only needs to predict 1 correct action to achieve a similar effect. As the exploration horizon is effectively reduced by a factor of $H$, we are able to train the high-level policy in SUPE to explore more efficiently and consequently solve the task much more quickly.\n\n**\"Can you explain why skill extraction via pseudo-labeling provides a significant advantage over directly using offline data for training?\"**\n\nThe pseudo-labeling enables us to use offline data as additional off-policy data for the online training of the high-level agent. The high-level agent operates at a longer time-scale – taking only one high-level action ($z$) per 4 environment steps. Pseudo-labeling skills allows us to convert the low-level trajectories in the offline data $(s_t, a_t, s_{t+1}, a_{t+1}, \\cdots)$ into high-level trajectories $(s_t, z_t, s_{t+H}, z_{t+H}, \\cdots)$. Without pseudo-labeling, we would not be able to directly use the offline data to update the high-level RL agent online, which means that we could no longer use the offline data twice.\n\n**\"Table 1 shows that the KL coefficient and GRU layers vary across environments rather than being consistent. Can you explain the rationale for these differences?\"** \n\nThe OGBench ($\\texttt{humanoidmaze}$, $\\texttt{antsoccer}$, $\\texttt{scene}$, $\\texttt{cube-single}$, $\\texttt{cube-double}$) tasks are generally more difficult, so we use larger networks than for the simpler D4RL tasks. We select our network sizes for these OGBench environments according to the reference policy size used in the original OGBench paper [1]. For the manipulation tasks $\\texttt{cube-*}$ and $\\texttt{scene}$, the data distribution is narrower, and we found that a higher KL coefficient helps account for the difference in dataset diversity. \n\n[1] Park, Seohong, et al. \"OGBench: Benchmarking offline goal-conditioned RL.\" arXiv preprint arXiv:2410.20092 (2024).\n\n**Related work references** \n\nThanks for the additional references on unsupervised RL. We will incorporate them in our discussion in the paper and update in the camera ready version. \n\nWe would like to thank you again for your constructive feedback and detailed reviews especially on the additional related work references. Please let us know if you have any other concerns or questions. **If we have successfully addressed all your concerns, could you kindly raise your rating?**", "claims_and_evidence": "The claim that the algorithm achieves efficient exploration and faster online learning is supported by experiments in state-based and pixel-based reinforcement learning (RL) environments, demonstrating higher normalized returns compared to baselines. This claim is backed by clear and convincing empirical evidence, including detailed performance tables and learning curves.", "methods_and_evaluation_criteria": "Based on my understanding, the proposed method consists of two steps. First, a high-level policy is pretrained on unlabeled trajectory data using a VAE-based approach to learn diverse skills. Second, during online RL, a skill-conditioned policy collects data by interacting with the environment, combining offline data from the pretraining step with online data for training. The evaluation uses goal reaching task with normalized returns like D4RL to assess performance. \n\nThe inclusion of offline trajectories is reasonable for improving exploration efficiency. However, it is unclear why skill extraction is critical for online RL, as the algorithm uses offline data twice—first for skill extraction and second to augment training data—without justifying the marginal benefit of pseudo-labeling skills. Can you explain why skill extraction via pseudo-labeling provides a significant advantage over directly using offline data for training? A strong justification or a fair ablation study would strengthen the rationale as the title claimed.", "theoretical_claims": "The paper does not present formal theoretical claims or proofs. It includes serveral equations defining the loss objective to optimize the proposed model’s performance. As such, there are no proofs to verify.", "experimental_designs_or_analyses": "Although the paper lacks existing methods as it claims, it compares several online reinforcement learning (RL) methods, such as EXPLORE, DBC+JSRL, and a skill discovery approach like HILP. This section of the experiment appears robust, supported by statistical significance tests.", "relation_to_broader_scientific_literature": "The topic in this paper quite aligns with some unsupervised reinforcement learning problems, which improves efficient exploration ablility and downstream task performance by addtional pretraining phase, some papers for reference. I believe that such method can be also adopt into online RL setting.\n\n[1] URLB: Unsupervised Reinforcement Learning Benchmark\nMichael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, Pieter Abbeel\n\n[2] CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery\nMichael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, Pieter Abbeel\n\n[3] Constrained Ensemble Exploration for Unsupervised Skill Discovery\nChenjia Bai, Rushuai Yang, Qiaosheng Zhang, Kang Xu, Yi Chen, Ting Xiao, Xuelong Li\n\n[4] METRA: Scalable Unsupervised RL with Metric-Aware Abstraction\nSeohong Park, Oleh Rybkin, Sergey Levine\n\n[5] Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills\nVíctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i-Nieto, Jordi Torres\n\n[6] Behavior Contrastive Learning for Unsupervised Skill Discovery\nRushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, Xuelong Li", "essential_references_not_discussed": "See the Relation To Broader Scientific Literature section.", "other_strengths_and_weaknesses": "Strengths: \n1. clear writing and easy to follow.\n\nWeakness:\n1. see discussion in the Methods And Evaluation Criteria section.", "other_comments_or_suggestions": "The qualitative results and reported learning curves are clear and well-designed. However, it is unclear how the skill-conditioned policy guides goal-reaching tasks during evaluation. I suggest adding visualizations, such as those in Figure 4 of BeCL [Yang et al., 2023], to illustrate how the policy operates conditioned on different latent skills. A response with such visuals could enhance my confidence in the skill policy’s effectiveness.\n\n\n[1] Behavior Contrastive Learning for Unsupervised Skill Discovery\nRushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, Xuelong Li", "questions_for_authors": "1. Table 1 shows that the KL coefficient and GRU layers vary across environments rather than being consistent. Can you explain the rationale for these differences? A clear justification would strengthen the method’s design choices.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lXJ7H1ApvR", "reviewer": "Reviewer_rsMw", "summary": "In this paper, the authors propose SUPE, which combines the idea of pretraining offline skills and online exploration via hierarchical policy. SUPE first extracts low-level skills using a variational autoencoder, then pseudo-labels unlabeled trajectories with optimistic rewards and high-level action labels. Using this reward signal, SUPE train a high-level policy that composes pretrained skills for efficient exploration. In experiments, SUPE consistently outperforms previous methods across 42 long-horizon and sparse-reward tasks.\n\n## Update after rebuttal\nThe reviewers addressed my questions and updated score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and insightful comments. We especially appreciate the comments on  the motivation. For your concern on the novelty of our method, we believe there might be a misunderstanding of our method – we believe that what we were describing is actually one of our baselines, “Trajectory Skills”. In our response, we explain in detail that we are not just a naive extension of OPAL combined with UCB (that is the baseline “Trajectory Skills”) and the differentiating step in our method makes a huge difference in performance from this naive extension as demonstrated in our experiments. \n\n**\"The proposed method can be considered as an extension of OPAL (but online version), combined with a UCB approach. In other words, from a technical perspective, it seems equivalent to applying a UCB-style reward to OPAL and training it in an online setting.\"**\n\nWe believe there is a misunderstanding. Our method is not equivalent to an extension of OPAL combined with a UCB approach. Rather, our “Trajectory Skills” baseline is equivalent to an online version of OPAL combined with a UCB approach from a technical perspective. Our “Trajectory Skills” baseline pretrains skills the same way as OPAL and learns a high-level online RL agent with a UCB approach to encourage exploration. The only, yet very important difference that distinguishes SUPE from this baseline is the pseudo-relabeling of the offline data that allows it to be used as additional off-policy data during online learning. Without the relabeling, we would not be able to leverage the offline data as additional off-policy data for online learning. We showed in our paper that using the offline data this way online results in a consistent/large performance boost over the “Trajectory Skills” baseline (Figure 2 and 3). \n\n**\"It remains unclear how common in the real world the special case of offline data being un-labeled but online interactions being rewarded is.\"**\n\nIn robotics, having access to task specific data can be expensive as it often requires human demonstration (e.g., through teleoperation or carefully scripted policy). It is often much easier to have access to a diverse dataset that is not directly related to the downstream task and use unsupervised RL algorithms to pretrain on this diverse dataset, and then fine-tune on a downstream task. This paradigm also started to gain popularity in RL in recent years [1, 2, 3, 4, 5]. In particular, O2O [1], FB [5] and ExPLORe [4] all study the setting where the offline data is unlabeled but the reward information is available in the online phase. O2O and FB assume the reward function is available right from the beginning of the online phase, whereas ExPLORe and our paper assumes the reward signal can only be obtained from online interactions. While the relevance of this problem setting merits discussion (which we will add to our paper), we believe that the number of prior works on this in just the last few years suggest that the topic is of interest to the ML community.\n\n[1] \"Unsupervised-to-Online Reinforcement Learning\"\n\n[2] \"Fast Imitation via Behavior Foundation Models\"\n\n[3] \"Reinforcement learning from passive data via latent intentions\"\n\n[4] \"Accelerating exploration with unlabeled prior data\" \n\n[5] \"Learning one representation to optimize all rewards\"\n\n**“Why do the HILP-based methods achieve a non-zero return right from the start?”**\n\nThe data distribution for D4RL Kitchen is high quality, and BC methods can achieve some non-zero return [1]. Thus, since the HILP skills are effective at mimicking dataset behavior, sampling HILP skills can potentially achieve nonzero return prior to online exploration. \n\n[1] D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.\n\n**\"What are the pros and cons of SUPE into HILP and the vanilla version, and what insights can you offer on which one should be chosen depending on the environment?\"**\n\nFrom current empirical observation, SUPE appears to be more stable during offline learning, and excels at long-horizon locomotion tasks (ie. $\\texttt{visual-antmaze}$, $\\texttt{antmaze}$, $\\texttt{humanoidmaze}$). SUPE (HILP) is competitive for some manipulation tasks ($\\texttt{antsoccer}$, $\\texttt{scene}$), and achieves higher initial performance in some environments. \n\nWe would like to thank you again for your constructive feedback and detailed reviews. Please let us know if you have any other concerns or questions. **If we have successfully addressed all your concerns, could you kindly raise your rating?**", "claims_and_evidence": "The overall pipeline of using offline action primitives to train online HRL and applying UCB-style rewards is clear and convincing.\n\nHowever, as noted below, there are similar ideas in the offline RL version, and the assumption that you have access to both (a) offline data and (b) an online environment can be a strong one.\n\nThe authors argue that \"~ these methods cannot be directly used in our setting as they require offline data to have reward labels\", but it remains unclear how common in the real world the special case of offline data being un-labeled but online interactions being rewarded is.", "methods_and_evaluation_criteria": "The evaluation criteria for the proposed method seem appropriate, and experiments were conducted across a sufficient number of environments. Although the number of ablation studies is not very large, the two most important factors (the RND coefficient and skill horizon length) were well examined.", "theoretical_claims": ".", "experimental_designs_or_analyses": "The experimental design and analysis seem valid.", "relation_to_broader_scientific_literature": "As the authors have claimed, there are not many clear methods that first learn offline action primitives and then utilize them in online settings. However, similar ideas do exist in the offline domain, and it might be better to provide a direct comparison with those approaches. (For more details, please refer to the section \"Other Strengths and Weaknesses.\")", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "The proposed method can be considered as an extension of OPAL (but online version), combined with a UCB approach. In other words, from a technical perspective, it seems equivalent to applying a UCB-style reward to OPAL and training it in an online setting.", "other_comments_or_suggestions": ".", "questions_for_authors": "Why do the HILP-based methods in the kitchen environment of Figure 2 achieve a non-zero return right from the start?\n\nWhat are the pros and cons of SUPE into HILP and the vanilla version, and what insights can you offer on which one should be chosen depending on the environment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces SUPE, an approach that combines offline skill pretraining with online reinforcement learning (RL) to improve exploration. SUPE achieves faster learning and more efficient exploration compared to existing methods on multiple long-horizon and sparse-reward tasks. 

The main strengths of the paper lie in its empirical results and novel use of offline data for online exploration. There are concerns initially, but the authors' response addresses most concerns. All reviewers agree to accept this manuscript.
Decision: Accept (poster)