Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration

## Abstract

Abstract Unsupervised pretraining has been transformative in many supervised domains. However, applying such ideas to reinforcement learning (RL) presents a unique challenge in that fine-tuning does not involve mimicking task-specific data, but rather exploring and locating the solution through iterative self-improvement. In this work, we study how unlabeled prior trajectory data can be leveraged to learn efficient exploration strategies. While prior data can be used to pretrain a set of low-level skills, or as additional off-policy data for online RL, it has been unclear how to combine these ideas effectively for online exploration. Our method SUPE ( S kills from U nlabeled P rior data for E xploration) demonstrates that a careful combination of these ideas compounds their benefits. Our method first extracts low-level skills using a variational autoencoder (VAE), and then pseudo-relabels unlabeled trajectories using an optimistic reward model, transforming prior data into high-level, task-relevant examples. Finally, SUPE uses these transformed examples as additional off-policy data for online RL to learn a high-level policy that composes pretrained low-level skills to explore efficiently.
We empirically show that SUPE reliably outperforms prior strategies, successfully solving a suite of long-horizon, sparse-reward tasks. Code: https://github.com/rail-berkeley/supe .

## 1 Introduction

Unsupervised pretraining has been transformative in many supervised domains, such as language (Devlin et al., ) and vision (He et al., ). Pretrained models can adapt with small numbers of examples, and with better generality (Radford et al., ; Brown et al., ). However, in contrast to supervised learning, reinforcement learning (RL) presents a unique challenge in that fine-tuning does not involve further mimicking task-specific data, but rather exploring and locating the solution through iterative self-improvement. Thus, the key challenge to address in pretraining for RL is not simply to learn good representations, but to learn an effective exploration strategy for solving downstream tasks.

Pretraining benefits greatly from the breadth of the data. Unlabeled trajectories (i.e., those collected from previous policies whose objectives are unknown) are the most abundantly available, but using them to solve specific tasks can be difficult. It is not enough to simply copy behaviors, which can differ greatly from the current task. There is an entanglement problem – general knowledge of the environment is mixed in with task-specific behaviors. A concrete example is learning from unlabeled locomotion behavior: we wish to learn how to move around the world, but not necessarily to the locations present in the pretraining data. We will revisit this setting in the experimental section.

The entanglement problem can be alleviated through hierarchical decomposition. Specifically, trajectories can broken into segments of task-agnostic skills, which are composed in various ways to solve various objectives. We posit that unlabeled trajectories thus present a twofold benefit, (1) as a way to learn a diverse set of skills, and (2) as off-policy examples of composing such skills. Notably, prior online RL methods that leverage pretrained skills largely ignore the second benefit, and discard the prior trajectories after the skills are learned (Ajay et al., ; Pertsch et al., ; Hu et al., ; Chen et al., ). We instead argue that such trajectories are critical, and can greatly speed up learning. We make use of a simple strategy of learning an optimistic reward model from online samples, and pseudo-relabeling past trajectories with an optimistic reward estimate. The past trajectories can thus be readily utilized as off-policy data, allowing for quick learning even with a very small number of online interactions.

We formalize these insights as SUPE (Skills from Unlabeled Prior data for Exploration), a recipe for maximally leveraging unlabeled prior data in the context of exploration. The prior data is utilized in two capacities, the offline and online phases. In the offline pretraining phase, we extract short segments of trajectories and use them to learn a set of low-level skills. In the online phase, we learn a high-level exploration policy, and again utilize the prior data by labelling each trajectory segment with an optimistic reward estimate. By “double-dipping” in this way, we can utilize both the low-level and high-level structure of prior trajectories to enable efficient exploration online.

Our main contribution is a simple method that leverages unlabeled prior trajectory data to both pretrain skills offline and compose these skills efficiently online for exploration. We instantiate SUPE with a variational autoencoder (VAE) to extract low-level skills, and an off-the-shelf off-policy RL algorithm (Ball et al., ) to learn a high-level policy from both online and offline data (Figure 1). Our empirical evaluations on a set of challenging sparse reward tasks show that leveraging the unlabeled prior data during both offline and online learning is crucial for efficient exploration, enabling SUPE to find the sparse reward signal more quickly and achieve more efficient learning over all prior methods (none of which are able to utilize the data both online and offline).

## 2 Related Work

Unsupervised skill discovery.
Unsupervised skill discovery methods first began in the online setting, where RL agents were tasked with learning structured behaviors in the absence of reward signal  (Gregor et al., ; Bacon et al., ; Florensa et al., ; Achiam et al., ; Eysenbach et al., ; Sharma et al., ; Hansen et al., ; Liu & Abbeel, ; Park et al., ). These insights naturally transferred to the offline setting as a method of dealing with unlabeled trajectory data. Offline skill discovery methods largely comprise of two categories, those who extract skills based on optimizing unsupervised reward signals (in either the form of policies (Touati et al., ; Hu et al., ; Frans et al., ; Park et al., ) or Q-functions (Chen et al., )), and those who utilize conditional behavior-cloning over subsets of trajectories  (Shankar & Gupta, ; Ajay et al., ; Singh et al., ; Pertsch et al., ; Nasiriany et al., ). Closest to our method in implementation are Ajay et al. () and Pertsch et al. (), who utilize a trajectory-segment VAE to learn low-level skills, and learn a high-level policy online. However, in contrast to prior methods which all utilize offline data purely for skill-learning and do not keep it around during online training, we show that utilizing the data via relabeling is critical for fast exploration.

Offline to online reinforcement learning. The offline-to-online reinforcement learning methods (Xie et al., ; Song et al., ; Lee et al., ; Agarwal et al., ; Zhang et al., ; Zheng et al., ; Ball et al., ; Nakamoto et al., ) focus on efficient online learning with the presence of offline data (often labeled with the reward value). Many offline RL approaches can be applied to this setting – simply run offline RL first on the offline data to convergence as an initialization and then continue training for online learning (using the combined dataset that consists of both offline and online data) (Kumar et al., ; Kostrikov et al., ; Tarasov et al., ). However, such approaches often result in slow online improvements as offline RL objectives tend to overly constrain the policy behaviors to be close to the prior data, limiting the exploration capability. On the other hand, off-policy online RL methods can also be directly applied in this setting by directly treating the offline data as additional off-policy data in the replay buffer and learning the policy from scratch (Lee et al., ; Song et al., ; Ball et al., ). While related in spirit, these methods cannot be directly used in our setting as they require offline data to have reward labels.

Data-driven exploration. A common approach for online exploration is to augment reward bonuses to the perceived rewards and optimize the RL agent with respect to the augmented rewards (Stadie et al., ; Bellemare et al., ; Houthooft et al., ; Pathak et al., ; Tang et al., ; Ostrovski et al., ; Achiam & Sastry, ; Burda et al., ; Ermolov & Sebe, ; Guo et al., ; Lobel et al., ). While most exploration methods operate in the purely online setting and focus on adding bonuses to the online replay buffer, recent works also start to explore a more data-driven approach that makes use of an unlabeled prior data to guide online exploration. Li et al. () explore adding bonuses to the offline data, allowing them to optimize the RL agent to be optimistic about states in the data, encouraging exploration around the offline data distribution. Our method explores a similar idea of adding bonuses to the offline data but for training a high-level policy, allowing us to compose pretrained skills effectively for exploration. Hu et al. () explore a slightly different strategy of learning a number of policies using offline RL that each optimizes for a random reward function. Then, it samples actions from these policies online to form an action pool from which the online agent can choose to select for exploration. This approach does not utilize offline data during the online phase and require all the policies (for every random reward function) to be represented separately. In contrast, our method makes use of the offline data as off-policy data for updating the high-level policy and our skills are represented using a single network (with the skill latent being the input to our network). As we will show in our experiments, being able to use offline data is crucial for learning to explore in the environment efficiently.

Options framework. Our method is also related to the option framework (Sutton et al., ; Menache et al., ; Mannor et al., ; Şimşek & Barto, ; ; Konidaris, ; Daniel et al., ; Srinivas et al., ; Daniel et al., ; Fox et al., ; Bacon et al., ; Kim et al., ; Bagaria & Konidaris, ; Bagaria et al., ). Different from the approach we take that learns latent skills with a fixed time horizon ($H=4$ in all our experiments), the options framework provides a more flexible way to learn skills with varying time horizon, often defined by learnable initiation and/or termination conditions (Sutton et al., ). We opt for the simplified skill definition because it allows us to bypass the need to learn initiation or termination conditions, and frame the skill pretraining phase as a simple supervised learning task.

## 3 Problem Formulation

We consider a Markov decision process (MDP) $\mathcal{M}=\{\mathcal{S},\mathcal{A},\mathbf{P},\gamma,r,\rho\}$ where $\mathcal{S}$ is the set of all possible states, $\mathcal{A}$ is the set of all possible actions that a policy $\pi(a|s):\mathcal{S}\mapsto\mathcal{P}(\mathcal{A})$ may take, $\mathbf{P}(s^{\prime}|s,a):\mathcal{S}\times\mathcal{A}\mapsto\mathcal{P}(
\mathcal{S})$ is the transition function that describes the probability distribution over the next state $s^{\prime}$ given the current state and the action taken at the state, $\gamma$ is the discount factor, $r(s,a):\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is the reward function, and $\rho:\mathcal{P}(\mathcal{S})$ is the initial state distribution. We have access to a dataset of transitions that are collected from the same MDP with no reward labels: $\mathcal{D}=\{(s_{i},a_{i},s^{\prime}_{i})\}$. During online learning, the agent may interact with the environment by taking actions and observes the next state and the reward specified by transition function $\mathcal{P}$ and the reward function $r$. We aim to develop a method that can leverage the dataset $\mathcal{D}$ to efficiently explore in the MDP to collect reward information, and outputs a well-performing policy $\pi(a|s)$ that achieves good cumulative return in the environment $\eta(\pi)=\mathbb{E}_{\{s_{0}\sim\rho,a_{t}\sim\pi(a_{t}|s_{t}),s_{t+1}\sim
\mathbf{P}(\cdot|s_{t},a_{t})\}}\sum_{t=0}^{\infty}\left[\gamma^{t}r(s_{t},a_{
t})\right]$. Note that this is different from the zero-shot RL setting (Touati et al., ) where the reward function is specified for the online evaluation (only unknown during the unspervised pretraining phase). In our setting,
the agent has zero knowledge of the reward function and must actively explore in the environment to identify the task it needs to solve by receiving the reward through environment interactions.

## 4 S kills from U nlabeled P rior data for E xploration ( SUPE )

In this section, we describe in detail how we utilize the unlabeled trajectory dataset to accelerate online exploration. Our method, SUPE, can be roughly divided into two parts. The first part is an offline pretraining phase where we extract skills from the unlabeled prior data with an trajectory-segment VAE. The second part is the online learning phase where we train a high-level off-policy agent to compose the pretrained skills leveraging examples from both prior data and online replay buffer. Algorithm 1 describes our method.

Figure: Algorithm 1 SUPE

Pretraining with trajectory VAE.
Since we only have access to an unlabeled dataset of trajectories, we must capture all the behaviors in the data as accurately as possible. At the same time, we aim to make the dataset directly usable for training a high-level skill-setting policy in hope that this high-level policy can be trained in a more sample-efficient way (compared to only having access to the online samples). We achieve this by adopting a trajectory VAE design from prior methods (Ajay et al., ; Pertsch et al., ) where a short segment of trajectory $\tau=\{s_{0},a_{0},s_{1},\cdots,s_{H-1},a_{H-1}\}$ is first fed into a trajectory encoder $f_{\theta}(z|\tau)$ that outputs a distribution over the latent code $z$, then a skill policy $\pi_{\theta}(a|s,z)$ is used to reconstruct the actions in the trajectory segment. Such a design helps us directly map trajectory segments to their corresponding skill policies, effectively allowing us to transform the segment into a high-level transition in the form of $(\textit{current state: }s_{0},\textit{action: }z,\textit{next state: }s_{H})$. As result, such transition can be directly consumed by any off-policy RL algorithm in the online phase to update the high-level policy $\pi_{\psi}(z|s)$ (as we will explain in the next section in more details). We also learn a state-dependent prior $p_{\theta}(z|s)$, following the prior works (Pertsch et al., ), to help accommodate the difference in behavior diversity of different states.
Putting them all together, the loss is shown in Equation 1.

$$ $\displaystyle\mathcal{L}_{\theta}(\tau)$ $\displaystyle=\beta D_{\mathrm{KL}}(f_{\theta}(z|\tau)||p_{\theta}(z|s_{0}))- \mathbb{E}_{z\sim f_{\theta}(z|\tau)}\left[\sum_{h=0}^{H-1}\log\pi_{\theta}(a_ {h}|s_{h},z)\right].$ (1) $$

Online exploration with trajectory skills. Our main goal in the online phase is to learn a high-level off-policy agent that decides which skill to use at a regular interval of $H$ time steps to learn the task quickly. The agent consumes high-level transitions where the state and the next state are separated by $H$ time steps and the action corresponds to the trajectory latent $z$ that is used to retrieve the low-level actions from the skill policy $\pi(a|s,z)$. To make use of the prior data and generate high-level transitions from it, we need both the action and the reward label for each pair of states (that are separated by $H$ steps) in the trajectory. For the action, we can simply sample from the trajectory encoding of the trajectory segment enclosed by the state pair. For the reward, we maintain an upper-confidence bound (UCB) estimate of the reward value for each state and skill pair $(s,z)$ inspired by the prior work (Li et al., ) (where it does so directly in the state-action space $(s,a)$), and pseudo-label the transition with such an optimistic reward estimate. The optimistic reward estimate is recomputed before updating the high level agent, since the estimate changes over time, while the trajectory encoding is computed before starting online learning, since this label does not change. The relabeling is summarized below:

$$ $\displaystyle(\underbracket{s_{0}}_{\text{state}},\underbracket{\hat{z}\sim f_ {\theta}(z|\tau)}_{\text{labeled action}},\underbracket{\hat{r}=r_{\mathrm{UCB }}(s_{0},\hat{z})}_{\text{labeled reward }},\underbracket{s_{H}}_{\text{next state}}).$ (2) $$

Practical implementation details.
Following prior work on trajectory-segment VAEs (Ajay et al., ; Pertsch et al., ), we use a Gaussian distribution (with both mean and diagonal covariance learnable) for the trajectory encoder, the skill policy, as well as the state-dependent prior.
While Pertsch et al. () use a KL constraint between the high-level policy and the state-dependent prior, we use a simpler design without the KL constraint that works much better (as we show in Appendix E).
To achieve this, we adapt the policy parameterization from (Haarnoja et al., ), where the action value is enforced to be between $-1$ and $1$ using a $\mathrm{tanh}$ transformation, and entropy regularization is applied on the squashed space. We use this policy parameterization for the high-level policy $\pi(z|s)$ to predict the skill action in the squashed space $z_{\mathrm{sqaushed}}$. We then recover the actual skill action vector by unsquashing it according to $z=\mathrm{arctanh}(z_{\mathrm{sqaushed}})$, so that it can be used by our skill policy $\pi_{\theta}(a|s,z)$.
For upper-confidence bound (optimistic) estimation of the reward, ($r_{\mathrm{UCB}}(s_{0},\hat{z})$), we directly borrow the UCB estimation implementation in Li et al. () (Section 3, practical implementation section in their paper), where they use a combination of the random network distillation (RND) (Burda et al., ) reward bonus and the predicted reward from a reward model (see Appendix C, Ours for more details). For the off-policy high-level agent, we follow Li et al. () to use RLPD (Ball et al., ) that takes a balanced number of samples from the prior data and the online replay buffer for agent optimization. In addition to using the optimistic offline reward label, we also find that adding the RND reward bonus to the online batch is also helpful to encourage online exploration, so we use it in all our experiments.

## 5 Experimental Results

We present a series of experiments to evaluate the effectiveness of our method to discover fast exploration strategies. We specifically focus on long-horizon, sparse-reward settings, where online exploration is especially important.
In particular, we aim to answer the following questions:

- 1.
Can we leverage unsupervised trajectory skills to accelerate online learning?
- 2.
Is our method able to find goals faster than prior methods?
- 3.
Does offline data help our method to compose skills better for faster exploration?

Figure: (a) a) AntMaze: three maze layouts (medium, large and ultra), and four goals for each layout.
Refer to caption: x2.png

### 5.1 Experimental setup

We conduct our experiments on three challenging sparse reward domains (Figure 2).

D4RL AntMaze with Additional Goal Locations. D4RL AntMaze is a standard benchmark for offline-to-online RL (Fu et al., ; Ball et al., ) where an ant robot needs to navigate around a maze to a specified goal location. We benchmark on three mazes of increasing size, antmaze-medium, antmaze-large, and antmaze-ultra. We take the D4RL dataset for medium and large mazes as well as the dataset from Jiang et al. () for the ultra maze (we use the diverse version of the datasets for all these layouts). We then remove the termination and reward information from the dataset such that the agent does not know about the goal location a priori. For each of the medium, large, ultra mazes, we test with four different goal locations that are hidden from the agent. See Figure 2(a) for a visualization of the mazes and the four goal locations that we use for each of them. We use a $-1$/0 0 sparse reward function where the agent receives $-1$ when it has not found the goal and it recieves 0 0 when it reaches the goal location and the episode terminates. The ant always starts from the left bottom corner of the maze, and the goal for the RL agent is to reach the goal consistently from the start location. It is worth noting that the goal is not known a priori and the offline data is unlabeled. In order to learn to navigate to the goal consistently, the agent first needs to traverse in the maze to gather information about where the goal is located.

D4RL Kitchen is another standard benchmark for offline-to-online RL (Fu et al., ; Nakamoto et al., ), where a Franka robot arm is controlled to interact with various objects in a simulated kitchen scenario. The desired goal is to complete four tasks (open the microwave, move the kettle, flip the light switch, and slide open the cabinet door) in sequence. In the process, the agent attains a reward equal to the number of currently solved tasks. The benchmark contains three datasets, kitchen-mixed, kitchen-partial, and kitchen-complete. kitchen-complete is the easiest dataset, which only has demonstrations of the four tasks completed in order. kitchen-partial adds additional tasks, but the four tasks are sometimes completed in sequence. kitchen-mixed is the hardest, where the four tasks are never completed in sequence. To adapt this benchmark in our work, we remove all the reward labels in the offline dataset.

Aside from the two state-based domains above, we also consider a visual domain below to test the ability of our method in scaling up to high-dimensional image observations.

Visual AntMaze is a benchmark introduced by Park et al. (), where the agent must rely on $64\times 64$ image observations of its surroundings, as well as proprioceptive information including body joint positions and velocities to navigate the maze. In particular, the image is the only way for the agent to locate itself within the maze, so successfully learning to extract location information from the image is necessary for successful navigation. The floor is colored such that any image can uniquely identify a position in the maze. The maze layout is the same as the large layout in the state-based D4RL AntMaze benchmark above and we also use the same additional goals. The reward function and the termination condition are also the same as the state-based benchmark.

We use the normalized return, a standard metric for D4RL (Fu et al., ) environments, as the main evaluation metric. For the Kitchen domain, the normalized return represents the average percentage of the tasks that are solved. The normalized return for the AntMaze domains (state-based and visual) represents the average success rate of reaching the goal.

### 5.2 Comparisons

While there is no existing method in our problem setting that utilizes unlabeled prior data in both the pretraining phase and the online learning phase, there are methods that make use of the prior data in either phase. We first consider two baselines that do not use pretraining and instead directly perform online learning.

Online. This baseline completely discards the offline data and the exploration is done with online reward bonus implemented by random network distillation (RND) (Burda et al., ). For all the baselines below as well as our method, we add online RND bonus to the replay buffer to encourage exploration.

ExPLORe (Li et al., ). This baseline is similar to our method in the sense that it also uses exploration bonus and offline data to encourage exploration. The one crucial difference is that it does not perform unsupervised skill pretraining and learns a 1-step policy directly online. As we will show, pretraining is crucial for our method to find goals faster and lead to more efficient online learning. It is worth noting that the original ExPLORe method does not make use of online RND. To make the comparison fair, we additionally add online RND bonus to this baseline to help it explore better online. For completeness, we also include the performance of the original ExPLORe method in the Appendix (Figure 8).

We then consider additional baselines that use the prior data during a pretraining phase but do not use the data during online learning.

Diffusion BC + JSRL. This baseline is an upgraded version of the BC + JSRL baseline used in ExPLORe (Li et al., ). Instead of using a Gaussian policy (as used by Li et al. ()), we use an expressive diffusion model to behavior clone the unlabeled prior data. At the beginning of each online episode, we roll out the policy for a random number of steps from the initial state before using switching to the online RL agent (Uchendu et al., ; Li et al., ). One might expect that an expressive enough policy class can model the behavior of the prior good enough such that it can form a good prior for exploration online.

Online with Skills. We also consider two skill-based baselines where the prior data is discarded in the online phase and the high-level policy is trained from scratch online with exploration bonus. We experiment with two types of pretraining skills. The first one is the trajectory VAE skill used in our method. The second one is from a recently proposed unsupervised offline skill discovery method where skills are pretrained to be able to traverse a learned Hilbert representation space (Park et al., ) (HILP). We use the exact same high-level RL agent as our method except that the agent no longer makes use of the prior data online.

Finally, we also consider a novel baseline that also uses prior data during pretraining and online exploration, but uses HILP skills rather than trajectory-based skills.

HILP w/ Offline Data. We observe that HILP skills can also utilize the offline data via relabeling. Recall that HILP learns a latent space of the observations (via an encoder $\phi_{\mathrm{HILP}}$) and learns skills that move agent in a certain direction $z$ (skill) in the latent space. For any high-level transition $(s_{0},s_{H})$, we simply take $\hat{z}\leftarrow\frac{\phi_{\mathrm{HILP}}(s_{H})-\phi_{\mathrm{HILP}}(s_{0})
}{\|\phi_{\mathrm{HILP}}(s_{H})-\phi_{\mathrm{HILP}}(s_{0})\|_{2}}$, the normalized difference vector that points from $s_{0}$ to $s_{H}$ in the latent space. We use the normalized difference vector because the pretrained HILP skill policy takes in normalized skill vectors. We use the exact same high-level RL agent as our method except that the skill relabeling is done by computing the latent difference rather than using the trajectory encoder ($f_{\theta}(z|\tau)$).

For the visual antmaze environment, we use the same image encoder used in RLPD (Ball et al., ). We also follow one of our baselines, ExPLORe (Li et al., ), to use ICVF (Ghosh et al., ), a method that uses task-agnostic value functions to learn image/state representations from passive data. ICVF takes in an offline unlabeled trajectory dataset with image observations and pretrain an image encoder in an unsupervised manner. Following ExPLORe, we take the weights of the image encoder from ICVF pretraining to initialize the image encoder’s weights in the RND network. To make the comparison fair, we also apply ICVF to all our baselines (see details in Appendix D).

Figure: Figure 3: Aggregated normalized return across three different domains. Ours achieves the best performance through training on all three domains. ExPLORe achieves strong later stage performance on AntMaze, but struggles in high-dimensional Visual AntMaze and Kitchen tasks. Online w/ HILP Skills and HILP w/ Offline Data achieve decent initial return on Kitchen, but struggle to learn in all three domains. Online w/ Trajectory Skills consistently underperforms Ours across all three environments. Diffusion BC + JSRL learns reasonably well in Kitchen, but performs much worse in AntMaze and Visual AntMaze. Online does not perform competitively at any stage of exploration. Section 5.2 contains details on the baselines we compare with. Each curve is an average over 8 seeds. For AntMaze, we aggregate over 3 maze layouts and 4 goals. For Kitchen, we aggregate over 3 tasks. For Visual AntMaze, we aggregate over 4 goals on one maze layout.
Refer to caption: x4.png

Figure: Figure 4: Normalized return on individual AntMaze and Kitchen tasks. Ours achieves the strongest performance on all tasks. Online w/ Trajectory Skills learns much slower on all AntMaze tasks, and is asymptotically worse on two of the three more challenging Kitchen tasks. ExPLORe struggles to learn on Kitchen, and performs worse as maze size increases. None of the other baselines are competitive on any tasks. Each curve is an average over four goals with 8 seeds for AntMaze, and 16 seeds for Kitchen.
Refer to caption: x5.png

### 5.3 Can we leverage unsupervised trajectory skills to accelerate online learning?

Figure 3 shows the aggregated performance of our approach on all three domains. Overall, our method outperforms all prior methods. Both HILP-based methods (Online with HILP Skills and HILP with Offline Data) achieve a good initial return on the Kitchen domain, but improve very slowly during online training. The Online with Trajectory Skills baseline also consistently performs worse than our method across all three domains, which demonstrates the importance of using prior data for online learning of the high-level policy, since that is only difference between this baseline and Ours. ExPLORe uses offline data during online learning, but does not pretrain skills, leading to slower learning on all three environments and difficulty achieving any significant return on Visual AntMaze or Kitchen. We also report the performance on individual AntMaze mazes and Kitchen tasks in Figure 4, and observe that our method outperforms the baselines more on harder environments. For the AntMaze domain, as the maze size increases, the two best baselines, Online w/ Trajectory Skills and ExPLORe, perform worse, while Ours continues to achieve a similar level of performance. For the Kitchen domain, on the more challenging kitchen-mixed and kitchen-partial tasks, the improvement of Ours over Online w/ Trajectory Skills is also larger.
These experiments suggest that pretraining skills and the ability to leverage the prior data are both crucial for achieving efficient online learning, especially in more challenging environments.
For the visual domain, we additionally perform an ablation study to assess the importance of the ICVF pre-trained representation, which we include in Appendix D. While ICVF combines synergistically with our method to further accelerate learning and exploration, initializing RND image encoder weights using ICVF is not critical to its success.
Even though we have demonstrated that our method is able to achieve higher success rate faster than prior works, it is still not clear if our method can actually lead to better exploration (instead of simply learning the high-level policy better). In the following two sections, we study the exploration aspect in isolation in the AntMaze domain.

### 5.4 Is our method able to find goals faster than prior methods?

Table 1 shows the average number of online environment interaction steps for the agent to reach the goal for each of the four goals in each of the three maze layouts. Such a metric allows us to assess how efficiently the agent explores in the maze whereas the success rate metric only measures how good the agent is at reaching the desired goal. It is possible for an agent to be good at exploration, but bad at reaching goals consistently and vice-versa. Table 1 shows that our method’s average first goal time outperforms or matches every baseline on each of the 12 goals, which confirms that our method not only learns faster, but does so by exploring more efficiently.

Figure: Figure 5: Coverage on three different AntMaze mazes, averaged over runs on four goals. Ours has the best coverage performance on the challenging antmaze-ultra, and is only passed by HILP w/ Offline Data on antmaze-large. Online w/ Traj. Skills and Online with HILP Skills struggle to explore after initial learning, and Online and Diffusion BC + JSRL generally perform poorly at all time steps.
Refer to caption: x6.png

### 5.5 Does offline data help our method to compose skills better for faster exploration?

Figure 5 shows the percentage of the maze that the agent has covered throughout the training. The coverage of skill-based methods that do not use prior data during online learning, Online w/ Trajectory Skills and Online w/ HILP Skills, significantly lags behind baselines that use offline data after 50,000 environment steps. Many methods achieve similar coverage on antmaze-medium, likely because the maze is too small to differentiate the different methods. Ours is able to achieve the highest coverage on the antmaze-ultra, and is only surpassed on antmaze-large by HILP w/ Offline Data, which has high first goal times and slow learning. Thus, the coverage difference can likely be at least partially attributed to HILP w/ Offline Data struggling to find the goal and continuing to explore after finding the goal. All non-skill based methods struggle to get competitive coverage levels on antmaze-large and antmaze-ultra. This suggests both pretraining skills and the ability to leverage prior data online are crucial for efficient exploration, and our method effectively compounds their benefits. For completeness, we include the success rate and coverage results for each maze layout and goal location in Appendix F.

## 6 Discussion and Limitations

In this work, we propose a novel method, SUPE, that leverages unlabeled prior trajectory data to accelerate online exploration and learning. The key insight is to use unlabeled trajectories *twice*, to 1) extract a set of low-level skills offline, and 2) serve as additional data for a high-level off-policy RL agent to compose these skills to explore in the environment. This allows us to effectively combine the strengths from unsupservised skill pretraining and sample-efficient online RL methods to solve a series of challenging long-horizon sparse reward tasks significantly more efficiently than existing methods. Our work opens up avenues in making full use of prior data for scalable, online RL algorithms. First, our pre-trained skills remain frozen during online learning, which may hinder online learning when the skills are not learned well or need to be updated as the learning progresses. Such problems could be alleviated by utilizing a better skill pretraning method, or allowing the low-level skills to be fine-tuned online.
A second limitation of our approach is the reliance on RND to maintain an upper confidence bound on the optimistic reward estimate. Although we find that RND works without ICVF on high-dimensional image observations in Visual AntMaze, the use of RND in other high dimensional environments may require more careful consideration. Possible future directions include examining alternative methods of maintaining this bound.

## Acknowledgments

This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley, and was supported by ONR through N00014-22-1-2773, AFOSR FA9550-22-1-0273, and the AI Institute. We would like to thank Seohong Park for providing his implementation of OPAL which was adapted and used for skill pretraining in our method. We would also like to thank Seohong Park, Fangchen Liu, Junsu Kim, Dibya Ghosh, Katie Kang, Oleg Rybkin, Kyle Stachowicz, Zhiyuan Zhou for discussions on the method and feedback on the early draft of the paper.

## Appendix A Compute Resources

We run all our experiments on NVIDIA A5000 GPUs.

We first calculate required compute for AntMaze experiments. For each maze-goal configuration, each of our pretraining run takes about two hours and online training also takes about two hours each. To reproduce the results of our methods, it requires $8\text{
(seeds) }\times 3\text{ (maze layouts)}\times(1\text{ (pretraining)}+4\text{ (
online learning) })\times 2\text{ (hours per run) }=240$ GPU hours. We have eight baselines that take similar GPU hours, which bring the total estimated GPU hours required to be around $2160$.

The runtime per Kitchen experiment is similar. There are only 3 environments, but we do 16 seeds. This means we need about $16\text{ (seeds)}\times 3\text{ (Kitchen tasks) }\times(1\text{ (pretraining)
}+1\text{ (online learning) })\times 2\text{ (hours per run) }=192$ GPU hours for our method. We only train six baselines (we do not run ExPLORe (No Online RND) ablation, or KL ablation), so this gives 1344 hours.
We also calculate the compute used for the Visual AntMaze experiments. On average it takes approximately 8 hours to train a pretraining checkpoint and approximately 24 hours to do online learning. This means it requires $8\text{ (seeds) }\times(1\text{ (pretraining) }\times 8\text{ (hours per run)}
+4\text{ (online learning) }\times 24\text{ (hours per run)})=832$ hours for our method. We have six baselines in the main figure. Additionally, we add 4 additional baselines in the ICVF ablation. This gives a total of 9152 hours for the Visual AntMaze results.

Finally, we look at the compute required for the data ablation experiments. These are additional AntMaze experiments on just the goals in antmaze-large maze. Thus, we have approximately $8\text{
(seeds) }\times 1\text{ (maze layouts)}\times 2\text{ (data ablations)}\times(
1\text{ (pretraining)}+4\text{ (online learning) })\times 2\text{ (hours per
run) }=160$ GPU hours to reproduce our method. We include 5 additional baselines, bringing the total compute for data ablations to 960 GPU hours.
Thus, in total the results in this paper required approximately $13616$ GPU hours, or about 1.5 GPU years. Note this is an approximate upper bound, since not all methods required training checkpoints, and checkpoints were shared between different baselines that both uses trajectory skills or both used HILP skills.

## Appendix B VAE Architecture and hyperparameters

We use a VAE implementation from Park et al. (). The authors kindly shared with us their OPAL implementation (which produces the results of the OPAL baseline in the paper). In this implementation, the VAE encoder is a recurrent neural network that uses gated-recurrent units (Cho et al., ) (GRU). It takes in a short sequence of states and actions, and produces a probabilistic output of a latent $z$. The reconstruction policy decoder is a fully-connected network with ReLU activation (Nair & Hinton, ) that takes in both the state in the sequence as well as the latent $z$ to output an action distribution.

**Table 2: VAE training details.**
| Parameter Name | Value |
| --- | --- |
| Batch size | 256 |
| Optimizer | Adam |
| Learning rate | $3\times 10^{-4}$ |
| GRU Hidden Size | 256 |
| GRU Layers | 2 hidden layers |
| KL Coefficient ($\beta)$ | 0.1 |
| VAE Prior | state-conditioned isotropic Gaussian distribution over the latent |
| VAE Posterior | isotropic Gaussian distribution over the latent |
| Reconstruction Policy Decoder | isotropic Gaussian distribution over the action space |
| Latent Dimension | 8 |
| Trajectory Segment Length ($H$) | 4 |
| Image Encoder Latent Dim | 50 |

In the online phase, our high-level policy is a Soft-Actor-Critic (SAC) agent (Haarnoja et al., ) with 10 critic networks, entropy backup disabled and LayerNorm added to the critics following the architecture design used in RLPD (Ball et al., ). We follow a similar strategy in ExPLORe (Li et al., ) where we sample 128 offline samples and 128 online samples and add RND reward bonus to all of the samples. The main difference is in the original ExPLORe paper is that they only add reward bonus to the offline data as additionally adding the bonus to the online replay buffer does not help for the maze goals they tested. In our experiments, we add the reward bonus to both offline data and online data, as it leads to better performance in goals where there is limited offline data coverage (see Appendix F).

**Table 3: Hyperparameters for the online RL agent following RLPD (Ball et al., )/ExPLORe (Li et al., ). For Diffusion BC + JSRL on AntMaze, we use an initial entropy temperature of 1.0 because it works much better than $0.05$. We use a $4\times$ larger RND coefficient in skill-based methods such that the reward bonus we get for each step in the skill horizon stays roughly proportional to the non-skill-based methods.**
| Parameter | Value |
| --- | --- |
| Batch size | 256 |
| Discount factor ($\gamma$) | 0.99 |
| Optimizer | Adam |
| Learning rate | $3\times 10^{-4}$ |
| Critic ensemble size | 10 |
| Critic minimum ensemble size | 1 for all methods on AntMaze, 2 for all on Kitchen, |
|  | 1 for non-skill based methods on Visual AntMaze, |
|  | 2 for skill-based methods on Visual AntMaze. |
| UTD Ratio | 20 for AntMaze and Kitchen, 40 for Visual AntMaze |
| Actor Delay | 20 |
| Network Width | 256 |
| Network Depth | 3 hidden layers. |
| Initial Entropy Temperature | 1.0 on Kitchen, 0.05 on Visual AntMaze/AntMaze |
| Target Entropy | $-\dim(\mathcal{A})/2$ |
| Entropy Backups | False |
| Start Training | after 5K env steps (RND update starts after 10K steps) |
| RND coefficient ($\alpha$) | $2.0$ for non-skill based, $8.0$ for skill based methods |

## Appendix C Implementation details for baselines

Diffusion BC + JSRL. We use the diffusion model implementation from (Hansen-Estruch et al., ). Following the paper’s implementation, we train the model for 3 million gradient steps with a dropout rate of 0.1 and a cosine decaying learning rate schedule from the learning rate of $0.0003$. In the online phase, in the beginning of every episode, with probability $p$, we rollout the diffusion policy for a random number of steps that follows a geometric distribution $\mathrm{Geom}(1-\gamma)$ before sampling actions from the online agent (inspired by (Li et al., )). A RND bonus is also added to the online batch on the fly with a coefficient of $2.0$ to further encourage online exploration of the SAC agent. The same coefficient is used in all other non-skill based baselines. For skill based baselines, we scale up the RND coefficient by the horizon length (4) to account for different reward scale. Following the BC + JSRL baseline used in ExPLORe (Li et al., ), we use a SAC agent with an ensemble of 10 critic networks, one actor network, with no entropy backup and LayerNorm in the critic networks. This configuration is used for all baselines on all environments. On AntMaze, We perform a hyperparameter sweep on both $p=\{0.5,0.75,0.9\}$ and the geometric distribution parameter $\gamma=\{0.99,0.995,0.997\}$ on the large maze with the top right goal and find that $p=0.9$ and $\gamma=0.99$ works the best. We also use these parameters for the Visual AntMaze experiments. For Kitchen, we perform a sweep on the parameter $p=\{0.2,0.5,0.75,0.9\}$ and find that 0.75 works best. We still use $\gamma=0.99$. We take the minimum of one random critic for AntMaze and Visual AntMaze, and the minimum of two random critics for Kitchen. For all methods, we use the same image encoder used in RLPD (Ball et al., ) for the Visual AntMaze task, with a latent dimension of 50 (encoded image is a 50 dimensional vector), which is then concatenated with proprioceptive state observations.

ExPLORe. We directly use the open-source implementation from [https://github.com/facebookresearch/ExPLORe/](https://github.com/facebookresearch/ExPLORe/). The only difference we make is to adjust the RND coefficient from $1.0$ to $2.0$ and additionally add such bonus to the online replay buffer (the original method only adds to the offline data). Empirically, we find a slightly higher RND coefficient improves performance slightly. Setting the value too high (e.g., 10) can in turn hurt performance. The SAC configuration is the same as that of the Diffusion BC + JSRL agent. We found that taking the minimum of one random critic on Visual AntMaze worked better for ExPLORe than taking the minimum of two, so all non-skill based baselines use this hyperparameter value.

Online RL with trajectory skills. This baseline is essentially our method but without using the trajectory encoder in the VAE to label trajectory segments (with high-level skill action labels), so all stated implementation decisions also apply to Ours. Instead, we treat it directly as a high-level RL problem with the low-level skill policy completely frozen. The SAC agent is the same as the previous agents, except for on Visual AntMaze, where taking the minimum of 2 critics from the ensemble leads to better performance for Ours, so we use this parameter setting for all skill-based benchmarks. We compute the high-level reward as the discounted sum of the rewards received every $H$ environment steps. During the $5\times 10^{3}$ steps before the start of training, we sample random actions from the state-based prior. For Visual AntMaze, we use the learned image encoder from the VAE to initialize both the critic image encoder and the RND network. If using ICVF, we initialize the RND network with the ICVF encoder instead.

Online RL with HILP skills. This baseline is the same as the one above but with the skills from a recent unsupervised offline skill discovery method, HILP (Park et al., ). We use the official open-source implementation [https://github.com/seohongpark/HILP](https://github.com/seohongpark/HILP) and run the pretraining to obtain the skill policies. Then, we freeze the skill policies and learn a high-level RL agent to select skills every $H$ steps.

HILP w/ offline data. This novel baseline is the same as Online w/ HILP Skills, except that we also relabel the offline trajectories and use them as additional data for learning the high-level policy online (similar to our proposed method). To relabel trajectories with the estimated HILP skill, we compute the difference in the latent representation of the final state $s_{H}$ and initial state $s_{0}$ in the trajectory, so
$\hat{z}\leftarrow\frac{\phi_{\mathrm{HILP}}(s_{H})-\phi_{\mathrm{HILP}}(s_{0})
}{\|\phi_{\mathrm{HILP}}(s_{H})-\phi_{\mathrm{HILP}}(s_{0})\|_{2}}$. We normalize the skill vector since the pretrained HILP policies use a normalized vector as input. The high-level RL agent is the same as our method, except the skill relabeling is done using the latent difference rather than the trajectory encoder.

Ours. We follow Li et al. () (ExPLORe) to relabel offline data with optimistic reward estimates using RND and a reward model. For completeness, we describe the details below. We initialize two networks $g_{\phi}(s,z)$, $\bar{g}(s,z)$ that each outputs an $L$-dimensional feature vector predicted from the state and (tanh-squashed) high-level action. During online learning, $\bar{g}(s,z)$ is fixed and we only update the parameters of the other network $g_{\phi}(s,z)$ to minimize the $L_{2}$ distance between the feature vectors predicted by the two networks on the new high-level transition $(s^{\mathrm{new}}_{0},z^{\mathrm{new}},r^{\mathrm{new}},s^{\mathrm{new}}_{H})$:

$$ $\displaystyle\mathcal{L}(\phi)=\|g_{\phi}(s^{\mathrm{new}}_{0},z^{\mathrm{new} })-\bar{g}(s^{\mathrm{new}}_{0},z^{\mathrm{new}})\|_{2}^{2}.$ $$

In addition to the two networks, we also learn a reward model $r_{\psi}(s,z)$ that minimizes the reward loss below on the transitions $(s_{0},z,r,s_{H})$ from online replay buffer:

$$ $\displaystyle\mathcal{L}(\psi)=\|r_{\psi}(s_{0},z)-r\|_{2}^{2}.$ $$

We then form an optimistic estimate of the reward value for the offline data as follows:

$$ $\displaystyle r_{\mathrm{UCB}}(s,z)\leftarrow r_{\psi}(s_{0},z)+\alpha\|g_{ \phi}(s_{0},z)-\bar{g}(s_{0},z)\|_{2}^{2},$ $$

where $\alpha$ controls the strength of the exploration tendency (RND coefficient). For AntMaze environments (both state-based and visual), we find that it is sufficient to use the minimum reward, $-1$, to label the offline data without a performance drop, so we opt for such a simpler design for our experiments.

Figure: Figure 6: Success rate on Visual AntMaze environment with and without ICVF. Ours works well without ICVF, almost matching the original performance. However, the other baselines Online w/ Trajectory Skills and ExPLORe achieve far worse performance without ICVF, which shows that using offline data both for extracting skills and online learning leads to better utilization of noisy exploration bonuses. Initializing ExPLORe critic with ICVF helps, but does not substantially change performance.
Refer to caption: x7.png

## Appendix D ICVF Implementation Details and Ablation Experiments for Visual Antmaze

We use the public implementation from the authors of (Ghosh et al., ) at [https://github.com/dibyaghosh/icvf_release](https://github.com/dibyaghosh/icvf_release) and run the ICVF training for $75,000$ gradient steps to obtain the pre-trained encoder weights, following (Li et al., ). Then, we initialize the encoder of the RND network with these weights before online learning. It is worth noting that this is slightly different from the prior work (Li et al., ) that initializes both the RND network and the critic network. In Figure 6, we examine the performance of Ours, Online w/ Trajectory Skills, and ExPLORe with and without ICVF. Both of the baselines perform much better with the ICVF initialization, suggesting that ICVF might play an important role in providing more informative exploration signal. Ours, without using ICVF, can already outperform the baselines with ICVF.
By both extracting skills from offline data and training with offline data, we are able to learn better from less informative exploration signals.
We also observe that initializing the critic with ICVF (as done in the original paper (Li et al., )) helps improve the performance of ExPLORe some, but does not substantially change performance.

## Appendix E KL Penalty Ablation

Figure: Figure 7: Normalized return on three AntMaze mazes, comparing Ours with a KL regularized alternative (Ours (KL)). We that Ours consistently outperforms Ours (KL) on all three mazes, with initial learning that is at least as fast and significantly improved asymptotic performance. Only Ours is able to meet or surpass the asymptotic performance of ExPLORe on all mazes.
Refer to caption: x8.png

In Figure 7, we compare the performance of Ours with a version of our method that uses a KL-divergence penalty with the state-based prior (as used in a previous skill-based method (Pertsch et al., )), Ours (KL). In Ours, as discussed in Section 4, we borrow the policy parameterization from Haarnoja et al. () and adopt a tanh policy parameterization with entropy regularization on the squashed space. Pertsch et al. () parameterize the higher level policy as a normal distribution and is explicitly constrained to a learned state-dependent prior using a KL-divergence penalty, with a temperature parameter that is auto-tuned to match some target value by using dual gradient descent on the temperature parameter. They do not use entropy regularization. Keeping everything else about our method the same, we instantiate this alternative policy parameterization in Ours (KL). We sweep over possible target KL-divergence values $(5,10,20,50)$ and initial values for the temperature parameter $(100,1,0.1)$ using the performance on antmaze-large, but find that these parameters do not substantially alter performance. As shown in Figure 7, Ours performs at least as well as Ours (KL) in the initial learning phase, and has better asymptotic performance on all three mazes, matching or beating ExPLORe, on all three mazes. It seems likely that not having entropy regularization makes it difficult to appropriately explore online, and that explicitly constraining to the prior may prevent further optimization of the policy. Attempts at combining an entropy bonus and KL-penalty lead to instability and difficulty tuning two separate temperature parameters. Additionally, in the Kitchen domain, the KL objective is unstable, since at some states the prior standard deviation is quite small, leading to numerical instability. In contrast, adopting the tanh policy parameterization from Haarnoja et al. () is simple, performs better, and encounters none of these issues in our experiments.

## Appendix F AntMaze Results by Different Maze Layouts and Goal Locations

We evaluate the success rate of the our algorithm compared to the same baseline suite as in the main results section for each individual goal and maze layout and report the results in Figure 8. We also include ExPLORe both with and without an online RND bonus. Online RND helps ExPLORe the most for the antmaze-medium bottom-right goal, where there is sparse offline data coverage for a considerable radius around the goal. We hypothesize that with the absence of online RND, the agent is encouraged to only stay close to the offline dataset, making it more difficult to find goals in less well-covered regions. On the flip side, for some other goals with better offline data coverage, like the antmaze-large top-right goal, online RND can make the performance worse. For every goal location, Ours consistently matches or outperforms all other methods throughout the training process.

Figure: Figure 8: Success rate by goal location. The addition of online RND in ExPLORe leads to better performance on goals with less offline data coverage, and slightly worse performance on goals well-represented in the dataset. Ours consistently matches are outperforms all other methods on all goals throughout training.
Refer to caption: x9.png

We also evaluate the coverage at every goal location for every method for each maze layout and show the result in Figure 9.
The coverage varies from goal location to goal location as some goal locations are harder to reach. Generally, the agent stops exploring once it has learned to reach the goal consistently.
Ours consistently has the best initial coverage for 11 out of 12 goals, though sometimes has lower coverage compared to other methods later in training. However, this is likely due in large part to successfully learning how to reach that goal quickly, and thus not exploring further.

Figure: Figure 9: Coverage for every goal location on three antmaze environments. There is significant variation between goals, and Ours consistently has the best initial coverage performance on 11 of 12 goals. Flattening coverage compared to other methods can be at least partially attributed to having already found the goal, and sucessfully optimizing reaching that goal, rather than continuing to explore after already finding the goal.
Refer to caption: x10.png

## Appendix G Robustness Against Offline Data Corruptions

To study how robust our method is, we perform an ablation study on the AntMaze domain on the large maze layout with two types of data corruption applied to the offline data:

- 1.
*Insufficient Coverage*: All the transitions close to the goal (within a circle with a radius of 5) are removed.
- 2.
*5% Data*: We subsample the dataset where only 5% of the trajectories are used for skill pretraining and online learning.

We report the performance on both settings in Figure 10. For the *Insufficient Coverage* setting, our method learns somewhat slower than the full data setting, but can still reach the same asymptotic performance, and outperforms or matches all baselines in the same data regime throughout the training process. For the *5% Data* setting, our method also reaches the same asymptotic performance as in the full data regime, and outperforms or matches all baselines throughout training. The gap between Ours and baseline performance (in particular, ExPLORe) is smaller than in the full data regime, which is to be expected as we have less data to learn the prior skills, so the skills are likely not as good. Overall, among the top performing methods in the AntMaze domain, our method is the most robust, consistently outperforming the other baselines that either do not use pre-trained skills (ExPLORe) or do not use the offline data during online learning (Online w/ Trajectory Skills) in these data corruption settings.

Figure: Figure 10: Data corruption ablation on state-based antmaze-large. *Top*: The success rate of different methods on these data corruption settings. *Bottom*: Visualization of the data distribution for each corruption setting. We experiment with two data corruption settings. Our method performs worse than the full data setting but still consistently outperforms all baselines.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses above.
soundness: 3
strengths: - The paper is well-written and easy to understand.
- The paper proposes a simple method for leveraging offline data and showing good performance on AntMaze, visual AntMaze, and Kitchen tasks. 
- The paper conducts thorough experiments and compares a set of different methods.
summary: The paper proposes a hierarchical policy for leveraging unlabeled offline data for exploration. In the offline stage, low-level skills are extracted, and in the online stage, these skills are reused and a high-level policy is learned with optimistic rewards. The proposed method is tested on maze and manipulation tasks and shows good performance.
weaknesses: - Dependence on offline data quality: The performance of the proposed method is influenced by the quality of the offline data and the specific features of the evaluation tasks. In particular, the approach relies on a high-level policy that is updated every 
𝐻 timesteps and keeps the pre-trained skill and trajectory encoder fixed during the online phase. This limitation constrains adaptability, especially in scenarios where task distribution varies from the offline data.
- Limited discussion on Hierarchical Reinforcement Learning (HRL): Although hierarchical policy structures have been extensively explored in the HRL literature [1-8] and are closely related to the paper, the paper does not sufficiently address relevant findings from HRL research. A more comprehensive discussion of how this work could provide valuable context.
- Novelty: The paper combines elements from ExPLORe and trajectory-segment VAE to leverage offline data for exploration, but adds limited new insights beyond prior work. HRL emphasizes hierarchical structures, and the benefits of skill extraction in offline settings have already been documented. This paper simply applies existing solutions to ExPLORe.

The paper could be improved in several aspects:
- Refinement of skill extraction method: Currently, skills are extracted based on fixed-length trajectory segments, a method that may overlook important nuances in skills. A more flexible or adaptive approach could address these limitations, potentially enhancing the robustness of the extracted skills.
- Skill adaptation during the online stage: The method does not allow for online adaptation of the skill policy or trajectory encoder. Due to potential distributional shifts between the offline and online data, enabling adaptive updates to the skill set and encoder could further improve the performance.
- Training stability in HRL is often affected by interactions between high-level and low-level policies. This work could benefit from discussing how offline data might address or mitigate these stability challenges.

[1] Kulkarni, Tejas D., et al. "Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation." Advances in neural information processing systems 29 (2016).

[2] Xie, Kevin, et al. "Latent skill planning for exploration and transfer." arXiv preprint arXiv:2011.13897 (2020).

[3] Nachum, Ofir, et al. "Data-efficient hierarchical reinforcement learning." Advances in neural information processing systems 31 (2018).

[4] Bacon, Pierre-Luc, Jean Harb, and Doina Precup. "The option-critic architecture." Proceedings of the AAAI conference on artificial intelligence. Vol. 31. No. 1. 2017.

[5] Ajay, Anurag, et al. "Opal: Offline primitive discovery for accelerating offline reinforcement learning." arXiv preprint arXiv:2010.13611 (2020).

[6] Gehring, Jonas, et al. "Hierarchical skills for efficient exploration." Advances in Neural Information Processing Systems 34 (2021): 11553-11564.

[7] Dalal, Murtaza, Deepak Pathak, and Russ R. Salakhutdinov. "Accelerating robotic reinforcement learning via parameterized action primitives." Advances in Neural Information Processing Systems 34 (2021): 21847-21859.

[8] Paraschos, Alexandros, et al. "Probabilistic movement primitives." Advances in neural information processing systems 26 (2013).

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 4
questions: - Where do the authors think their empirical benefit is coming from? Why can we use trajectories twice?
- Is the algorithm robust to different design choices?
- How important is the optimistic labelling (from Li et al.)?
soundness: 3
strengths: - This paper makes an insightful empirical benefit for using trajectories twice for both low level skill pretraining in addition to optimistic labelling.
- The paper thoroughly evaluates the proposed method.
- The paper does a good job explaining the proposed method and it's significance.
summary: This paper presents SUPE, a method for using offline data (without rewards) in the online reinforcement learning setting. SUPE first extracts a set of low level skills using the offline data, and then optimistically labels the offline trajectories. It then uses an off policy high level update to update on a mix of offline (pseudo labeled trajectories) and online real trajectories. The paper empirically validates the new algorithm on three environments and does ablations on amounts of offline data.
weaknesses: - This paper could benefit from a bit deeper analysis of the contribution of the two uses of offline data. It's clear that both are necessary, but not necessarily why.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness part. I may consider increasing the score if my questions are addressed.
soundness: 2
strengths: - The approach of extracting latent “skills” from unlabeled data and employing hierarchical methods significantly enhances exploration.
- The approach of utilizing prior data twice ensures better use of the available data.
- The paper is well-structured and easy to follow.
- Extensive results demonstrate that this method outperforms previous approaches.
summary: This paper introduces SUPE, a method that leverages unsupervised learning to extract skills from unlabeled prior data, subsequently using hierarchical methods to explore more efficiently. These unlabeled data can also contribute to high-level policy training. Experimental results show that SUPE outperforms previous methods on the D4RL benchmark.
weaknesses: - The concept of using a VAE to extract latent codes and employing a high-level policy for online exploration is not novel, and it shows limited progress compared to previous work [1].
- The ablation study lacks depth. I am interested in understanding the contribution of “reusing prior data twice” to the final performance. Additionally, I’d like clarification on the design choice for the latent variable $z$ in skill discovery: how do you ensure this latent $z$  is sufficient for effective skill discovery in the dataset? Is employing trajectory-segment VAEs truly necessary for efficient exploration?


[1] Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - How does your method compare to using offline-to-online RL methods which have access to reward labels?
- How was the $\alpha$ hyperparameter set?
- Why did you not compare to other types of offline data sets?
- What should I take from the coverage results?
- In Table 1, what do the bold numbers represent?
soundness: 2
strengths: Overall, I found the paper easy to follow and I think it is addressing an important problem -- pre-training in RL -- which is of interest to the community.

The results demonstrate that learning skills from offline data is a promising approach to leverage reward-free offline data.
I think this is an interesting result.
I also like the idea of labelling the offline data using a learned reward function.
summary: This paper presents a pre-training method for reinforcement learning (RL) that can train on data sets that do not contain reward labels, i.e., the data sets are unlabeled. 
The problem setting resembles offline-to-online RL, except that there are no rewards in the data set.
In the pre-training stage, the authors propose to learn a set of skills from this unlabeled offline data.
Then, in the online fine-tuning state, the authors learn a high-level policy that selects which skill to use in a given state.
They utilize the unlabeled offline data during fine-tuning by learning an optimistic reward model and using it to add optimistic reward labels to the offline data.
They evaluate their method in the D4RL AntMaze and Kitchen benchmarks as well as the D4RL Visual AntMaze.
weaknesses: The authors consider the setting of having access to offline data but no reward labels.
Whilst I see the value in this problem setting, it is not clear if practitioners should opt for this method over standard offline-to-online RL methods when
their data sets contain reward labels.
Whilst I appreciate this is out-of-scope, ideally methods would leverage data sets both with and without reward labels.
It would be insightful if the authors could compare to offline-to-online RL methods which do leverage reward labels.
Whilst I do not expect their method to outperform these methods, I think it is an important baseline that we can gain insights from.

In my experience, optimistic-based exploration methods are very susceptible to the $\alpha$ parameter.
How was this set in practice?
Did it require a grid search to find the best value in each environment?
Please can you provide details on any hyperparameter tuning process, including the range of values tested and how sensitivity varied across environments?
This information would be valuable for reproducibility and understanding the robustness of the method.

Is there a reason the authors only considered the diverse data set for the AntMaze experiments?
Does this method require a diverse offline data set collected by an unsupervised RL method,
or can it leverage narrow offline data distributions? For example, data from solving a different task?
How does the method perform when using the AntMaze "play" data set instead of the "diverse" data set?
Even if the method performs poorly, I think it would be valuable to include these results.

I am not sure what to take from the coverage results.
I can understand why we care about coverage in unsupervised RL where our sole purpose is to explore.
However, during online training our goal is to balance exploration vs exploitation.
Please can the authors provide a clearer justification for why coverage is an important metric in this context, or include additional plots that more directly show the relationship between exploration and task performance, such as the normalized return vs coverage?

In Table 1, what do the bold numbers represent? The authors should state what statistical test was used for the bolding or at least expla8in what the bolding represents.

## Minor comments and corrections
- Line 42 - "can broken" should be "can be broken"
- Line 117 - "of an offline data" should be "of offline data"
- Line 200 - the term "latent code" is misleading. This suggests the trajectory encoder learns to map trajectories to discrete codes from a codebook and I don't think this is the case. The authors should change it to something like "latent skill".
- Line 279 - Should "Three AntMaze layouts with four different goal location configuration each." be "Three AntMaze layouts with four configurable goal locations each."
- Line 411-414 - It would make more sense for this paragraph to be at the start of Section 5.4.
- Line 407 - "Kitchen the domain" should be "Kitchen domain"
- Line 408 - "more challenging the kitchen-mixed" should be "more challenging kitchen-mixed"
- Figures - The authors have stated that the shaded area indicates the standard error. They also need to state what that solid line indicates. Is it the mean, median, etc?
- Figures - I found the figures very hard to read. I would suggest the authors colour the text "HILP w/ Offline Data", "Ours", "Online w/ HILP Skills", etc, to match the colours of the lines in the plots. This would make the text/figures much easier to read.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * See weakness above. 
* Given the similarities between SPiRL [1] and this work, apart from the online reinforcement learning stage, why isn’t SPiRL used as a baseline for comparison (despite the numerous experiments conducted) ?
* In the pre-training stage, it would also be valuable to discuss whether trajectory segment length $H$ significantly impacts the method's performance.
* I am curious whether using expert data would result in better low-level skills during the pre-training stage.

[1] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.
soundness: 2
strengths: * The paper is highly detailed, well-written and provides detailed motivation. The complete code is also provided.
* The authors conduct numerous experiments to thoroughly validate their method and address in detail several key issues that I am particularly concerned about, including its scalability, robustness.
summary: This paper proposes a two-phase framework, SUPE, which leverages data in two stages: first, extracting low-level skills during the offline pre-training phase, and then using these skills and unlabeled data in the online phase to train a high-level strategy for more efficient exploration. Building on prior works like SPiRL [1] and ExPLORe [2], the key contribution of this paper is to integrate unlabeled data with online data to accelerate exploration and training in off-policy reinforcement learning (RL) methods. In the offline pre-training stage, the authors train a set of low-level skills, while in the online phase, they develop a high-level policy by utilizing both online data and relabeled offline data. To assess the method’s effectiveness, the authors compare SUPE with several baselines using benchmarks such as D4RL, and also discuss its limitations and potential directions for future research.

[1] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.

[2] Li, Qiyang, et al. "Accelerating exploration with unlabeled prior data." Advances in Neural Information Processing Systems 36 (2024).
weaknesses: * The overall novelty of this work is somewhat limited, as it builds heavily on existing methods and concepts (mentioned in summary). 
* Although numerous experiments are conducted, the selected tasks are relatively monotonous and simplistic. The experiments test only two types of tasks: AntMaze and Kitchen.

Meta-review: This paper proposes using offline data to first extract a low-level skill policy $\pi(a|s, z)$, and then learn a high-level policy $\psi(a | s, z)$ that combines them during the online phase.

Strengths
1. Impressive gains over several baselines in Figure 3.
2. Authors added new results that show gains over offline-to-online RL approaches in Figure 8 which is a common approach. 

Weakness:
1. The approach doesn't introduce a major idea although I would say it skillfully uses existing ideas.

2. Most of the experiments are on state-based observations and many of them are grid-based. For a paper for ICLR 2025, I think the domains to be addressed should be closer to real-world challenges especially given the progress in the ML community at large.

3. The approach depends on the quality of the data. While offline data is unlabeled, meaning there are no rewards, which is nice compared to offline RL approaches, it still requires access to semantically meaningful trajectories from which skills can be extracted. E.g., if the trajectories are random walks, then such an approach wouldn't work. Where can we expect to find such data? If the approach used something such as video data which are more abundantly available, then it would make the approach more practical. 

Overall, I think this paper uses existing ideas in RL to make impressive improvements over a variety of common benchmarks. The main concern is a mismatch with real-world problems where high-quality datasets may be hard to get. Further, the experiments are not on real-world problems which matters more for a paper where the core contribution is a skillful combination of existing ideas. One way to address this is to use more visual environments. Authors have also added a lot of new experiments during the rebuttal period that need more scrutiny. E.g., Figure 8 compares offline-to-online RL approaches to the proposed approach, however, the proposed approach requires high-quality data while offline RL can work with low-quality and high-coverage data. Further, it is not clear to me how authors claim gains over Cal-QL in 6/7 domains in Figure 8. For both medium play and medium diverse both cal-ql (green) and the proposed approach (black) achieve a return of 1. Also, the green plots are truncated in some places as they are read from the previous papers. For a fair comparison, all results should have the same number of steps. 

For now, I am recommending a rejection with strong encouragement to submit again with more experiments and add more clarity for the results.

Decision: Reject