Accelerating Exploration with Unlabeled Prior Data
Authors: Qiyang Li
Jason Zhang
Dibya Ghosh
Amy Zhang
Sergey Levine
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Exploration

Abstract: Learning to solve tasks from a sparse reward signal is a major challenge for standard reinforcement learning (RL) algorithms. However, in the real world, agents rarely need to solve sparse reward tasks entirely from scratch. More often, we might possess prior experience to draw on that provides considerable guidance about which actions and outcomes are possible in the world, which we can use to explore more effectively for new tasks. In this work, we study how prior data without reward labels may be used to guide and accelerate exploration for an agent solving a new sparse reward task. We propose a simple approach that learns a reward model from online experience, labels the unlabeled prior data with optimistic rewards, and then uses it concurrently alongside the online data for downstream policy and critic optimization. This general formula leads to rapid exploration in several challenging sparse-reward domains where tabula rasa exploration is insufficient, including the AntMaze domain, Adroit hand manipulation domain, and a visual simulated robotic manipulation domain. Our results highlight the ease of incorporating unlabeled prior data into existing online RL algorithms, and the (perhaps surprising) effectiveness of doing so.

Accelerating Exploration with Unlabeled Prior Data
Qiyang Li Jason Zhang",, Dibya Ghosh", Amy Zhang" "hang"",Seraee Sergey Levine
UC Berkeley UT Austin Meta"
(qeli, jason. di bibya.ao. .arra.ghoshh (terkebereeeeye edu www.ggausttttttaa www.nn/lasunituutt.....ee utexas edu, svlevinedeecs, wtt.eeessseeeeeeyc.cooom
amy.
Abstract
Learning to solve tasks from sparse reward signal is a major challenge for stan- dard reinforcement learning (RL) algorithms. However, the real world, agents rarely need to solve sparse reward tasks entirely from scratch. More often, we might possess prior experience to draw on that provides considerable guidance about which actions and outcomes are possible in the world, which we can use to explore more effectively for new tasks. In this work, we study how prior data without reward labels may be used to guide and accelerate exploration for agent solving a new sparse reward task. We propose a simple approach that learns a re- ward model from online experience, labels the unlabeled prior data with optimistic rewards, and then uses concurrently alongside the online data for downstream policy and critic optimization. This general formula leads to rapid exploration in several challenging sparse-reward domains where tabula rasa exploration is insuf- ficient, including the AntMaze domain, Adroit hand manipulation domain, and a visual simulated robotic manipulation domain. Our results highlight the ease of incorporating unlabeled prior data into existing online RI. algorithms, and the (perhaps surprising) effectiveness doing so.
Introduction
Exploration, particularly in sparse reward environ- ments, presents major challenge for reinforcement learning, and standard exploration methods typically need to seek out all potentially novel states to cover all the places where high rewards may be located. Luckily, in many roal-world RL settings, it is straight- forward to obtain prior data that help the agent understand how the world works. For example, if trying find where we left our keys, we would not relearn how to navigate our environment, but rather might revisit locations that recall from memory. If the data has reward annotations, pre- training with offline RL provides celerate online finetuning. However, in many do- mains of interest, it is more likely for prior datasets be task-agnostic, and not be labeled with the reward for the new task we hope to solve. Despite not hav- ing reward labels, such prior data can be useful for an agent exploring for new task since it describes the environment dynamics and indicates regions of the state space that may be interesting to explore.
Interaction Data (s.a.5".r)
Replay Buffer
(s.0,5))
(s.a.s.rr
Optimitic Reward Bonus
(s.R.c.7)
1s.
minl
Batch
Agent
solution ac-
Figure I: How to leverage unlabeled prior data for efficient online exploration? We label the prior data with optimistic reward estimate and run RL on both online offline data. This allows more efficient exploration around the trajectories in the prior data, improving sample efficiency especially for hard exploration tasks.
"Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

How can this unlabeled data be utilized by deep RL algorithm? Prior data informs the agent about potentially high-reward or -ngh-information states, and behaviors to reach them. Initially, the agent may act to reach these states, leading to directed exploration in more informative subspace rather than the complete state space. This should be an iterative process, since the agent can use the new online experience to refine what directions from the prior data it chooses to explore in. After the agent begins to receive reward signals, the prior data can help the agent to more reliably reach and exploit the reward signals, steering the exploration towards more promising region.
Perhaps surprisingly, such informed exploration can be elicited in a very simple way: by labeling the prior data with optimistic rewards and adding it to the online experience of an off- off-po RL algorithm (Figure 1). Specifically, we choose to impute the missing reward on the unlabeled prior data with an optimistic upper- copper-conieeece bound (UCB) of the reward estimates obtained through the agent's collected online experience. Early in training, these optimistic reward labels will be high for states from the prior data, and optimizing the RL objective with this data will elicit policies that attempt to reach regions of state present in the prior data. As states in the prior data are visited, the labeled reward for these states will regress towards their ground-trute values, leading to policy that either explores other states from the prior data (when true reward is low) or returns to the region more consistently (when high). By training an RL policy with UCB estimate of the reward, exploration is guided by both task rewards and prior data, focusing only on visiting states from the prior data that may have high reward. In practice, we find that this general recipe works with different off off-pollcy algorithms [Ball et al., 2023, Kostrikov et al., 2021].
Our main contribution is a simple approach to leverage unlabeled prior data to improve online explo- ration and sample efficiency, particularly for sparse long-horizon tasks. Our empirical evaluations, conducted over domains with different observation modalities (e.g., states and images), such as sim- ulated robot navigation and arm and hand manipulation, show that our simple optimistic reward labeling strategy can utilize the unlabeled prior data effectively, often well as the prior best ap- proach that has access to the same prior data with labeled rewards. In addition, we can leverage offline pre-training of task-agnostic value functions (e.g.. ICVF [Ghosh et al., 2023]) from offline data to initialize our optimistic reward model to further boost the online sample efficiency in more complex settings, such image-basee based environments. In contrast to other approaches in the same setting, our method is conceptually simpler and casier to be integrated into current off-policy online RL algorithms, and performs better as we will show in our experiments.
2 Problem Formulation
We formalize our problem in an infinite-horizon Markov decision process (MDP), defined as M (S, A, P.r. 7,
7, p). consists of a state space S, an action space A, transition dynamics P A +> P(S), reward function r S A + R, a discount factor E (0, 1). an initial state distribution of in- terest p. Algorithms interact in the MDP online to learn policies x(als) : S + ((A) that maximize P(lss)()())
Ei r'r(se, ac)]- We presume access to a dataset of transitions with no reward labels, D = {(s, ag 1 hi that was previously collected from M, for example by a random policy, human demonstrator, or an agent solving different task in the same environment. The core challenge in this problem setting is to explore in a sparse reward environment where you do not know where the reward is. While struc- turally similar offfine RL or online RL with prior data, where the agent receives dataset, the reward-less setting poses a significant exploration challenge not present when reward labels are available. Typically offline RL algorithms must find the best way to use the prior data to exploit the reward signal. In our setting, algorithms must use the prior data to improve exploration to acquire reward signal. Using this prior data requires bridging the misalignment between the offline prior dataset (no reward) and the online collected data (contains reward), since standard model-free RL algorithms cannot directly use experiential data that has no reward labels.
the discounted return under the policy n(T))
neward-labellee
3 Exploration from Prior Data by Labeling Optimistic Reward (EXPLORE)
In this section, we detail a simple way to use the unlabeled prior data directly in a standard online RL algorithm. Our general approach will be to label the reward-free data with learned optimistic
2

---

rewards and include this generated experience into the buffer of an off off-py algorithm. When these rewards are correctly chosen, this will encourage the RL agent explore along directions that were present in the prior data, focusing states perceived to have high rewards or not yet seen in the agent's online experience.
Algorithm EXPLORE
1: Input: prior unlabeled data Doffline
2: Initialize the UCB estimate of the reward function: UCBR(s,a)
3: Online replay buffer D +- 0
4: Initialize off-policy RL algorithm with a policy π.
5: for each environment step do
6: Collect (s, s'rr using the policy и and add the replay buffer D.
7: Update the UCB estimate UCBR(s, a) using the new transition as, a,,,,s',7 s',
8: Relabel each transition (s, a, 3') in Doffiine with UCBR(s, a)
9: Run off- off-policy RL update on both DD and the relabeled Doffiie to improve the policy æ.
end for
General algorithm. Since the prior data contains no reward labels, the agent can only acquire information about the reward function from the online experience it collects. Specifically, at any stage of training, the collected online experience D informs some posterior or confidence set over the true reward function that it must optimize. To make use of the prior data, we label it with an upper confidence bound (UCB) estimate of the true reward function. Optimizing with this relabeled prior experience results in a policy that is "optimistic in the face uncertainty", choosing guide its exploration towards prior data states either where it knows the reward to be high, and prior data states where is highly uncertain about the reward. Unlike standard exploration with optimistic rewards for states than those seen online, or may exclude states that are less important. Consequently. the RL. agent seeks visit regions in the prior data that are promising has visited before, accelerating online learning.
RL algorithm
online, the prior data may include states that further away
To explore with prior data, algorithm maintains uncertainty model of the reward, to be used for UCB estimates, and an off-policy agent that trains jointly on relabelled prior data and online experience. As experience is added through online data collection, the agent's uncertainty model over rewards. When training the off-policy agent, we sample data from both the online replay buffer (already reward labeled) and from the prior data (no reward labels). The prior data is labelled with rewards from the UCB estimate of our uncertainty model, and then the online and prior data is jointly trained upon. Training with this data leads to policy that acts optimistically with respect to the agent's 's uncertainty about the reward function, which we use guide exploration and collect new transitions. Algorithm presents a sketch of our method.
this data to update
Notice that while this strategy is similar to exploration algorithms that use novelty-basee reward bonuses [Bellemare et al., 2016, Tang et al., 2017, Pathak et al., 2017, Burda et al., 2018, Pathak et al., 2019, Gupta et al., 2022], since the optimism is being added to the prior data, the resulting exploration behaviors differ qualitatively. Adding novelty bonuses to online experience guides the agent to states it has already reached in the online phase, whereas adding optimism to the prior experience encourages the agent learn reach new states beyond the regions it has visited through online exploration. Intuitively, while standard optimism leads to exploration along the frontier of online data, optimism on prior data leads to exploration beyond the frontier of online data.
practice, we obtain UCB estimates of the rewards using a combina- tion of reward model and newdom-networr-ddie reward bonus technique. The former provides a reward estimate of the true reward value and the lat- ter (RND) quantifies the uncertainty of the estimate, allowing to get UCB reward estimate.
Practical implementation.
(RND) [Burda et al., 2018), a novelty bbsed
For the reward model, we update randomly initialized network ro a) simultaneously with the RL agent on the online replay buffer by minimizing the squared reward loss:
[()) = (....))) (re(s,, a) - -
RND randomly initializes two networks fo a), f(s, a) that each output L-dimensional feature vector for each state and action. We keep network, f. frozen and update the parameters of the
3

---

Online
Online + RND
Naive
Ours
Env Steps (x10²)
45
30
15
Visualizations of the agent's exploration behaviors using optimistic reward labeling strat- egy antmaze antmaze-medium. The dots shown (500 points each) sampled uniformly from the online replay buffer (up 60K environment steps), colored by the training environment step which they were added to the buffer. Both Online and Online + RND do not use prior data, and the latter uses RND reward bonus [Burda et al., 2018] on top of the received online reward to encourage online exploration. Naïve learns reward model and relabels the unlabeled prior data with its reward estimate with no optimism. Ours labels the prior data with optimistic reward estimate.
Figure 2:
Oracle
Online
MinR
Naive + BC BC + JSRL
Ours
Online + RND
Naive
AntMaze (6 Tasks)
Sparse Adroit (3 Tasks)
COG (3 Tasks)
Normalized Return
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.0
0.2
0.4
0.6
0.8
0.0
0.1
0.2
Environment Steps (x106)
Figure 3: Aggregated results for 6 AntMaze tasks, 3 Adroit tasks and COG tasks. Ours is the only method that largely recovers performance vanilla RLPD (Oracle), which has access to true rewards (Ours does nor). The baselines that do not use the prior unlabeled data (Online and Online + RND) perform poorly on both AntMaze and Adroit. Naively labeling the prior data without optimism (Naïve) performs competitively on Adroit but poorly AntMaze. Scction 5.2 contains the deseription for the baselines compare with.
other network to minimize the mean reaa-sqquar-erroo between the predicted features and the frozen ones once on every new transition (Snews anew, Snow Tnew) encountered:
[(Ø) = L I1 (Suew: Ouew) -f f(Snew: Onew )l
During online exploration, [lfa(s, a) - f(s, a)l forms a measure of the agent's uncertainty about (s, a), leading the following approximate UCB estimate of the reward function
UCBR(s, rols, a) I1 f(s,
For the RL agent, we use RLPD [Ball al., 2023], which specifically designed for efficient online RL and performs learning updates on both online and labeled offline data. It is worth noting that our method is compatible with potentially any other off-policy RL algorithms that can leverage labeled online and offline data, We experiment with alternative based on implicit-Q learning in Appendix C.4. Although we only have access unlabeled data in our setting, we constantly label the prior data with UCB reward estimates, enabling direct application of RLPD updates.

---

4 Related Work
Exploration with reward bonuses. One standard approach for targeted exploration is add ex- ploration bonuses to the reward signal and train value functions and policies with respect to these optimistic rewards. Exploration bonuses seek to reward novelty, for example using an approximate density model [Bellemare et al., 2016, Tang et al., 2017], curiosity [Pathak et al., 2017, 2019], model error [Stadie et al., 2015, Houthooft et al., 2016, Achiam and Sastry, 2017, Lin and Jabri, 2023], or even prediction error to randomly initialized function [Burda et al., 2018]. When optimistic wards are calibrated correctly, the formal counterparts to these algorithms [Strehl and Littman, 2008, Dann al., 2017] enjoy strong theoretical guarantees for minimizing regret [Azar al., 2017, Jin et al., 2018]. Prior works typically augment the reward bonuses to the online replay buffer to en- courage exploration of novel states, whereas in work, add reward bonuses the offline data, encouraging the RL agent to traverse through trajectories in the prior data to explore more efficiently.
Using prior data without reward labels. Prior experience without reward annotations can be used for downstream RL in a variety of ways. One strategy is to use this data to pre-train state represen- tations and extract feature backbones for downstream networks, whether using standard image-level objectives [Srinivas et al., 2020, Xiao et al., 2022, Radosavovic et al., 2022], or those that emphasize the temporal structure of the environment [Yang and Nachum, 2021, Ma et al., 2022, Farebrother et al., 2023, Ghosh et al., 2023]. These approaches are complementary to our method, and in our experiments, we find that optimistic reward labeling using pre-trained representations lead more coherent exploration and faster learning. Prior data may also inform behavioral priors or skills [Ajay al., 2020, Tirumala et al., 2020, Pertsch et al., 2021, Nasiriany et al., 2022]; once learned, these learned behaviors can provide a bootstrap for exploration [Uchendu et al., 2022, Li et al., 2023], or transform the agent's action space to a higher level of abstraction [Pertsch et al., 2020, Singh et al.. 2020a]. In addition, if the prior data consists of optimal demonstrations for the downstream task, there exists rich literature for steering online RL with demonstrations [Schaal, 1996, Vecerik et al.. 2017, Nair et al.. 2017, Zhu et al., 2020]. Finally, for model-based RL approaches, one in theory may additionally train a dynamics model on prior data and use it to plan and reach novel states [Sekar et al., 2020, Mendonca et al., 2021] for improved exploration, but we are not aware of works that explicitly use models to explore from reward-free offline prior data.
Using prior data with reward labels. To train on unlabeled prior data with optimistic reward labels, we use RLPD [Ball et al., 2023], sample-efficient online RL algorithm designed to train simultaneously on reward-labeled offline data and incoming online experience. RLPD is an offline- to-online algorithm [Xie et al., 2021, Lee et al., 2021, Agarwal et al., 2022, Zheng et al., 2023, Hao et al., 2023, Song et al., 2023] seeking to accelerate online RL using reward-labeled prior experience. While this setting shares many challenges with our unlabeled prior data, algorithms in this setting tend to focus on correcting errors in value function stemming from training on offline data [Nakamoto al., 2023], and correctly balancing between using highly off-policy transitions from the prior data and new online experience [Lee et al., 2021, Luo et al., 2023]. Our optimistic reward labeling mechanism is largely orthogonal to these axes, since our approach is compatible with any online RL algorithm that can train on prior data.
Online unsupervised RL. Online unsupervised RL is an adjacent setting in which agents collect action-free data to accelerate finetuning some downstream task [Laskin et al., 2021]. While both deal with reward-free free learning, they crucially differ in motivation: our setting asks how existing (pre-collected) data be used to inform downstream exploration for known task, while unsuper- vised RL studies how to actively collect new datasets for a (yet to be specified) task [Yarats et al., 2022, Brandfonbrener et al., 2022]. Methods for unsupervised RL seek broad and undirected explo- ration strategies, for example through information gain [Sekar et al., 2020, Liu and Abbeel, 2021] or state entropy maximization [Lee et al., 2019, Jin et al., 2020, Wang et al., 2020] instead of the task-relevant exploration that is advantageous in our setting. Successful mechanisms developed for unsupervised RL can inform useful strategies for our setting, since both seek to explore in directed ways, whether undirected as in unsupervised RL or guided by prior data, as in ours.
Offline meta reinforcement learning. Offline meta-reinforcement learning [Dorfman et al., 2021, Pong et al., 2021, Lin et al., 2022] also seeks to use prior datasets from a task distribution to accel- erate learning new tasks sampled from the same distribution. These approaches generally tend to focus on the single-shot or few-shot setting, with methods that attempt to (approximately) solve
5

---

the Bayesian exploration POMDP [Duan et al., 2016, Zintgraf et al., 2019, Rakelly et al., 2019]. While these methods are efficient when few-shot adaptation is possible, they struggle in scenarios requiring more complex exploration or when the underlying task distribution lacks structure [Ajay al., 2022, Mandi et al., 2022]. Meta RL methods also rely on prior data containing meaningful, interesting tasks, while our optimistic relabeling strategy is agnostic to the prior task distribution.
5
Experiments
Our experiments are aimed at evaluating how effectively our method can improve exploration by leveraging prior data. Through extensive experiments, we will establish that our method able to leverage the unlabeled prior data to boost online sample efficiency consistently, often matching the sample efficiency of the previous best approach that uses labeled prior data (whereas we have no access to labeled rewards). More concretely, we will answer the following questions:
1. Can we leverage unlabeled prior data to accelerate online learning?
2. Can representation learning help obtain better reward labels?
3. Does optimistic reward labeling help online learning by improving online exploration?
4. How robust is our method in handling different offline data corruptions?
5.1 Experimental Setup
The environments that evaluate methods are all challenging sparse reward tasks, including six D4RL AntMaze tasks [Fu et al., 2020], three sparse-reward Adroit hand manipulation tasks [Nair al., 2021] following the setup in RLPD [Ball al., 2023], and three mmage-ased robotic manipu- lation tasks used by COG [Singh et al., 2020b]. See Appendix for details about each task. For all three domains, we explicitly remove the reward information from the offline dataset so that the agent must explore to solve the task. We choose these environments as they cover a range of different char- acteristics of the prior data, where each can present a unique exploration and learning challenge. In Adroit, the prior data consists of a few expert trajectories on the target task and a much larger set of data from a behavior cloning policy. In AntMaze, the prior data consists goal-directed trajectories with varying start and end locations. In COG, the prior data is mostly demonstration trajectories for sub-tasks of larger multi-stage task. These tasks are interesting for the following reasons:
Sparse reward (AII). Due to the sparse reward nature of the these tasks, the agent has no knowledge of the task, it must explore to solve the task successfully. For example, AntMaze tasks, the goal is for an ant robot to navigate in a maze to reach a fixed goal from a fixed starting point. The prior data for this task consists of sub-optimal trajectories, using starting points and goals potentially different from the current task. Without knowledge of the goal location, the agent must explore each corner of the maze identify the actual goal location before solving the task is possible.
High-dimensional image observation (COG). The three COG tasks use image observations, which poses much harder exploration challenge compared to state observations because of high dimen- sionality. As we will show, even with such high- high-dimensiooal observation spaces, our algorithm is still able to efficiently explore and accelerate online learning more than the baselines. More impor- tantly, our algorithm can incorporate pre- pre-training representations that further accelerates learning.
Multi-stage exploration (COG). The robotic manipulation tasks in COG have two stages, meaning that the robot needs to perform some sub-task (e.g., opening a drawer) before is able complete the full task (e.g.. picking out ball from the drawer). See Appendix B, Figure 9 for visualizations of images from the COG dataset showing the two stage structure.
The prior data consists only of trajectories performing individual stages by themselves, and there no prior data of complete task executions. The agent must learn explore past the first stage of the task without any reward information, before it can receive the reward for completing the full task. Therefore, this multi-stage experiment requires the agent to be able effectively make use of incomplete trajectories to aid exploration. This setting is closer to the availability of the data one may encounter the real world. For example, there may be a lot of data of robots performing tasks that are only partially related to the current task of interest. This data could also be unlabeled, as the rewards for other tasks may not match the current task. Despite being incomplete, we would still like to be able to utilize this data.

---

5.2 Comparisons
While most prior works do not study the setting where the prior data is unlabeled, focusing on either the standard fully labeled data setting utilizing online unsupervised learning phase, we can adapt some of these prior works as comparisons in our setting (see Appendix A for details):
Online: A data-efficient off-policy RI. agent that does not make use of the prior data at all.
Online RND:
An augmentation of online RL with RND novelty bonus. This baseline uses only online data, augmenting the online batch with RND bonus [Burda et al., 2018). To be clear, of the comparisons listed here bold, this is the only one that uses an online RND bonus.
Naïve Reward Labeling: This comparison labels the prior data reward with an unbiased reward estimate; this is implemented using our method but omitting the RND novelty score.
Naïve BC: This baseline additionally uses a behavioral cloning loss follow the behaviors as seen in prior data, inspired by similar regularization in offline RL [Vecerik et al., 2017].
This is an adaptation of UDS [Yu et al., 2022] the online fine-tuning setting. The original UDS method uses unlabeled prior data to improve offline RL on a smaller reward-labeled dataset, using the minimum reward of the task relabel the unlabeled data. MinR uses the same labeling strategy, but with RLPD for online RL.
MinR:
This baseline is an adaptation of JSRL [Uchendu et al., 2022]. The original JSRL method uses offline RL algorithms to pre-train a guide policy using a fully labeled prior dataset. Then, at each online episode, it uses the guide policy to roll out the trajectory up random number of steps, then switches to an exploration policy such that the initial state distribution of the exploration policy is shaped by the state visitation of the guide-policy, inducing faster learning of the exploration policy. Since in our setting the prior data has no labels, we use behavior cloning (BC) pre-training to initialize the guide-polic instead of using offline RL algorithms.
BC + JSRL:
Oracle: This is an oracle baseline that assumes access to ground truth reward labels, using the same base off-policy RL algorithm [Ball et al., 2023] with true reward labels on prior data.
We also include a behavior prior baseline (inspired by the behavior prior learning line of work [Ajay et al., 2020, Tirumala et al., 2020, Pertsch et al., 2021. Nasiriany et al., 2022, Singh et al., 2020a]) comparison Appendix C.5 on AntMaze.
5.3 Does optimistic labeling of prior data accelerate online learning?
Figure 3 shows the aggregated performance of our approach on the two state-based domains and one magee-aased domain. On AntMaze and COG domains, our optimistic reward labeling is able to outperform naïve reward labeling significantly, highlighting its effectiveness in leveraging the prior data to accelerate online learning. Interestingly, in the two state-based domains, we show that with- out access to prior rewards, our method can nearly match the performance of the oracle baseline with prior rewards. This suggests that optimistic reward labeling data almost as effectively as labeled prior data on these three domains. In particular, on the sparse Adroit relocate task, using optimistic rewards can even outpace the oracle bascline (Appendix C.2, Figure 13). In the COG domain, there is a larger performance gap between our method and the oracle. We hypothesize that this is due to the high-dimensional image observation space, caus- ing the exploration problem to be more challenging. On the Adroit domain, we find that periodic resetting of the learned reward function is necessary avoid overfitting the collected online expe- rience - with the resetting, we find that even naïve reward relabeling results in exploratory behavior, diminishing the gap when performing explicit reward uncertainty estimation with RND.
allow RL to utilize unlabeled prior
5.4 Can representation learning help obtain better reward labels?
We now investigate whether we can improve exploration by basing our UCB reward estimates on top of pre-trained representations acquired from the prior data. We chose to use representations from ICVF [Ghosh et al., 2023], a method that trains feature representations by pre-training gen- eral successor value functions on the offline data. Value-aware pre-training is natural choice for downstream reward uncertainty quantification, because the pre-training encourages representations to obey the spatiality of the environment that states close to one another have similar representa-
7

---

Oracle
Ours ICVF
Ours
Naive ICVF
Naive
Normalized Return
Pick and Place
Grasp from Closed Drawer
Grasp from Blocked Drawer 1
0.5
0.5
0.0
0.0
0.0
0.1
0.2
0.0
0.1
0.2
0.0
Environment Steps (x10%)
Figure 4: Accelerating exploration with pre-trained representations on three visual-imput COG tasks. Ours ICVF uses optimistic reward labeling with pre-trained representation initialization; Ours uses op- timistic reward labeling without the pre-trained representation initialization. The same applies for Naive and Naive ICVF. Overall, initializing the reward model and the RND network using pre-trained representations greatly increases how quickly the model Icarns.
At 25k Environment Steps
Normalized Reward
Ours + ICVF Durs
Trajectory Steps
15
20
25
45
50
Figure 5: The effect of pre-trained representations the relabeled reward. We show the normalized and unit variance) optimal prior trajectory from
relabeled rewards (by standardizing to have closed drawer task the COG environment. Using pre-trained representations to learn the optimistic rewards leads to smoother rewards over the course of an optimal trajectory. See more examples in Appendix D.
tions, while states that are difficult to reach from one another are far away. We pre-train ICVF model on the prior dataset and extract the visual encoder ficve(s), which encodes an image into low-dimensional feature vector. These pre-trained encoder parameters
used to initialize the en- coder for both the reward model re (E)( and the RND network (Es (s)). Note that these encoder parameters not shared and not frozen, meaning that once training begins, the encoder parameters for the reward model will be different from the encoder parameters for the RND model, and also diverge from the pre-trained initialization. As shown in Figure ICVF consistently improves the online learning efficiency across all image-input tasks on top of optimistic reward labeling. It is worth noting that the encoder initialization from pre-training alone is not sufficient for the learning agent to succeed online (Naive + ICVF still fails almost everywhere). On most tasks, without opti- mistic reward labeling, the agent simply makes zero progress, further highlighting the effectiveness of optimistic labeling in aiding online exploration and learning. To understand how the pre-trained representation initialization influences online exploration, we analyze its effect on the UCB reward estimate early in the training. Figure 5 shows the UCB reward estimates on an expert trajectory of the Grasp from Closed Drawer task when the reward model and the RND model are ini- tialized with the pre-trained representations (in blue) and without the pre-trained representations (in orange). When our algorithm initialized with the ICVF representations, the UCB reward estimates are increasing along the trajectory time step, labeling the images in later part of the trajectory with high reward, forming a natural curriculum for the agent to explore towards the trajectory end. When our algorithm is initialized randomly without the pre-trained representations, such clear monotonic trend disappears. We hypothesize that this monotonie trend of the reward labeling may account for the online sample efficiency improvement with better shaped reward.

---

Ours
Naive
Online RND
umaze-diverse
medium-diivere
large-diverse
1.00
Coverage
0.75
0.50
0.25
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.2
0.3
Environment Steps (x10%)
Figure 6: State coverage for 3 AntMaze tasks, estimated by counting the number of square regions that the ant has visited, normalized by the total number of square regions that can be visited. Ours optimistically labels the prior data, whereas Naïve naively labels the prior data without optimism. Optimistic laheling enables the RI. agent to achieve significantly higher coverage. Online RND also explores well the maze, with a much slower rate compared to our approach, highlighting the importance of leveraging unlabeled prior data for efficient online exploration. Full results in Appendix C.I.
Oracle
Ours Online RND
Online Only + RND Naive
Naive Online RND
Ours
Online Only
Naive BC
Normalizer lized Raturn
All
Orthogonal Transitions
Insufficient Coverage
1% Data
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
0.00
0.15
0.30
0.45
0.60
0.75
0.00
0.15
0.30
0.45 0.60
0.75
0.00
0.15
0.30
0.45 0.60
0.75
0.00
0.15 0.30
0.45 0.60 0.75
Figure 7: The normalized return on antmaze-larre-dddeeeee--- under different offline data corruptions (see visualization of the offline data for each kind in Appendix B, Figure 8).
5.5 Does optimistic reward labeling help online learning by improving online exploration?
We have shown that optimistic reward labeling is effective accelerating online learning, does actually work because of better exploration? To answer this question, evaluate the effect of our method on the state-coverage of the agent the AntMaze domain. In this domain, the agent required reach particular goal in the maze but has prior knowledge of the goal, and thus required to explore different parts of the maze in the online phase order to locate the goal. The optimal behavior that expect the agent to perform is to quickly try out every possible part of the maze until the goal is located, and then focus entirely on that goal to achieve 100% success rate.
To more quantitatively measure the quality of our exploration behavior, we divide up the 2-D maze into a grid of square regions, and count the number of regions that the Ant agent has visited (using the online replay buffer). This provides measure for the amount of state coverage that the agent has achieved the environment, and the agent that explores better should achieve higher state coverage. Figure 6 shows how optimistic reward labeling affects the state coverage, and Figure 2 shows an example visualization the state- visitation for each method on antmaze-med um. Across all six AntMaze tasks, approach achieves significantly higher state-coverage compared to naïvely label without optimism both in the carly stage training and asymptotically compared to the baseline that does not leverage the unlabcled prior data. In particular, we find that naïvely labeling reward with no optimism is doing poorly on larger mazes, highlighting the importance of optimistic reward labeling for effective online exploration.
5.6 How robust is our method in handling different offline data corruptions?
To further test the capability of our method in leveraging prior data, we corrupt one of the D4RL. offline datasets, Sastmaze-large-ae rsee and test how our method performs under the corruption. We experimented with three different corruptions (and reported results in Figure 7).

---

Orthogonal transitions: all the transitions that move in the up/right (towards the goal) are removed from the offline data. In this setting, stitching the transitions in the offline data would not lead to a trajectory that goes from the start to the goal. This is a good test for the robustness of the algorithm for exploration because the agent can not rely on the actions in the prior data and must explore to discover actions that move the ant in the correct direction.
Insufficient coverage: all the transitions around the goals are removed from the offline data. This requires the algorithm to actively explore so that it discovers the desired goal that it needs to reach.
1 1% data: A random 1% of the transitions are kept in the offline data. This tests the capability of our method in handling the limited data regime.
In addition to the main comparisons above, we consider two modifications specifically for these experiments to address the more challenging tasks.
Ours + Online RND: This is an extension of our method, with an RND bonus added to the online data in addition the prior data. Similar to how we can extend our method with representation learning, having exploration bonuses on the online data does not conflict with exploration bonuses added to the offline data.
Naïve + Online RND: This is an extension of Naive Reward Labeling. An RND bonus added to the online data, and the prior data is labeled with the unbiased reward estimate as before.
Directly using our method in the Insufficient Coverage setting yields zero success, but this is to be expected as our algorithm does not incentivize exploration beyond the offline data. We find that this can be patched by simply combining our method with an additional RND bonus on online data, allowing our method to learn even when data has poor coverage and no data near the goal is unavailable. Similarly, we have found that combining our method with online RND can also boost the performance in the Orthogonal Transitions, where no transitions move in the direction of the goal. For the 1% Data setting, our method can still largely match the oracle performance without the need for an online RND bonus.
It is also worth noting that in the Orthogonal Transitions setting, our method achieves good perfor- mance whereas the oracle fails to succeed. The failures of the oracle are not unexpected because the prior data do not contain any single trajectories, or even trajectories stitched together from different transitions, that can reach the goal. However, since our method succeeds, this indicates that explo- ration bonuses applied only to the prior data can still be utilized, despite there being no forward connection through the prior data. These additional results, while initially surprising even to us (es- pecially the "orthogonal" setting, which seems very difficult), strongly suggest that our method can lead to significant improvement even when the data is not very good - certainly such data would be woefully inadequate for regular offline RL or navve policy initialization.
Discussion
We showed how we can effectively leverage unlabeled prior data to improve online exploration by running standard off-policy RL algorithm on the data relabeled with UCB reward estimates. In practice, the UCB estimates can be approximated by combining the prediction of a reward model and an RND network, both can be trained online with little additional computational cost. We demonstrated the surprising effectiveness of this simple approach on a diverse set of domains. Our instantiation of the optimistic reward labeling idea presents number of avenues for future research. First, on the re locate Adroit task, we found that naively fitting the reward model on the online replay buffer without any regularization leads poor performance (possibly due catastrophic overfitting). While we have found a temporary workaround by periodically re-initializing the reward model, such a solution may seem ad-hoc and could be disruptive to the RL learning due to the sudden change in the learning objective (which is shaped by the reward model). Next, the UCB estimates of the aattencctiono can rapidly change, especially in the beginning of the training, which may cause learning instability in the RL algorithm. There is no mechanism in our current algorithm to tackle such a potential issue. Nevertheless, our work indicates that there exist simple ways of incorporating prior data for online RL. agents even when no reward labels exist. Scaling these mechanisms to more complex prior datasets is an exciting direction towards the promise of RI. agents that can leverage general prior data.
10

---

Acknowledgement. This work was partially done while QL was visiting student researcher at FAIR, Meta. We would like to thank Seohong Park for providing his implementation of goal- conditioned IQL. (GC-IQL.) and pre-trained GC-IQL checkpoints (which were used in producing Figure 17). We would like to thank Fangehen Liu for the acronym of the method. We would also like to thank Laura Smith, Toru Lin, Oleg Rybkin, Kuba Grudzien, Aviral Kumar, and Joey Hong for discussion on the method and feedback on the early draft the paper. We would also like to thank the members of the RAIL lab for insightful discussions on the paper. This research was partially supported by the Office of Naval Research under N00014-21-1-2836 and N00014-22-1-2773, and ARO W91 INF-21-1-0097.
References
Joshua Achiam and S. Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. ArXiv, abs/1 703.0113337 2017.
Abien Fred Agarap. Deep learning using rectified linear units (ReLU), arXiv preprint arXiv:1803.08375. 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Belle- mare. Reincarnating reinforcement learning: Reusing prior computation accelerate progress. In Neural Information Processing Systems, 2022.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline prim- itive discovery for accelerating offline reinforcement learning, arXiv preprint (0.XX::2010013111111 2020.
Anurag Ajay, Abhishek Gupta, Dibya Ghosh, Sergey Levine, and Pulkit Agrawal. Distributionally adaptive meta reinforcement learning. ArXiv, abs/2210.03104, 2022.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In International Conference Machine Learning, pages 263 272. PMLR, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Philip Ball, Laura Smith. Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. arXiv preprint 0rX2v22300..0248, 2023.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-14799 2016.
David Brandfonbrener, Stephen Tu, Avi Singh, Stefan Welker, Chad Boodoo, N. Matni, and Jacob Varley. Visual backtracking teleoperation: A data collection protocol for offline image-based reinforcement learning. ArXiv, abs/2210.02343,, 2022.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /2894, 2018.
Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning. In NIPS, 2017.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning identifiability challenges and effective data collection strategies. In Neural Information Processing Systems, 2021.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and P. Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. ArXiv, abs/ Abs//161.00279, 2016.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G Bellemare. Proto-value networks: Scaling representation learn- ing with auxiliary tasks. arXiv preprint ar3iv:2304. 12567, 2023.

---

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Dibya Ghosh, Chethan Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. arXiv preprint arXiv:2304.04788 04782, 2023.
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking re- ward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems, 15515281--12999 2022.
Botao Hao, Rahul Jain, Dengwang Tang. and Zheng Wen. Bridging imitation and online reinforce- ment learning: An optimistic tale. ArXiv, abs/2303. 11669, 2023.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing 2016.
Systems, pages
Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is Q-learning provably effi- cient? Advances in neural information processing systems, 31, 2018.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward exploration for reinforcement learning. In International Conference on Machine Learning, 2020.
Σlya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q- learning. arXiv preprint arXiv:2 10.06169, 2021.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Alhert Zhan, Kevin Lu, Catherine Cang, Ler-- rel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 10.11991, 2021.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing. Sergey Levine, and Ruslan Salakhut- dinov. Efficient exploration via state marginal matching. ArXiv, abs/1906.05274, 2019.
Seunghyun Lee, Younggyo Seo, Kimin Lee, P. Abbeel, and Jinwoo Shin. Offline-to-online re- inforcement learning via balanced replay and pessimistic q-ensemble. ArXiv, abs/2107.00591, 2021.
Qiyang Li, Yuexiang Zhai, Yi Ma, and Sergey Levine. Understanding the complexity gains of single-task RL with a curriculum. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 20412-20455 PMLR, 23-29 Jul 2023. URL https: http::////cceedeeeeeeeeee mll. press/ html.
Runji Lin, Ye Li, Xidong Feng, Zhaowei Zhang, Xian Hong Wu Fung. Haifeng Zhang, Jun Wang, Yali Du, and Yaodong Yang. Contextual transformer for offline meta reinforcement learning. ArXiv, abs/2211 0112211100016.. 2022.
Toru Lin and Allan Jabri. MIMEx: Intrinsic rewards from masked input modeling, 2023.
Hao Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. ArXiv, 00s/2103.04551, 2021.
Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions. ArXiv, abs/2303.17396. 2023.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint 20X0::2200.00030 2022.
Zhao Mandi, P. Abbeel, and Stephen James. the effectiveness of fine-tuning versus meta- reinforcement learning. ArXiv, 0bs/2206.032777 2022.
12

---

Russell Mendonca, Olch Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models. Advances in Neural Information Processing Systems, 34, 2021.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2292-6299, 2017.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets, 2021.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre-training for efficient online fine- tuning. arXiv preprint arXiv:2303.0m 00449, 2023.
Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning, 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17,2 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Kumar Gupta. Sclf-supervised exploration via dis- agreement. ArXiv, 0bss/990.04461,, 2019.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. ArXiv, abs/2010. 1944, 2020.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J Lim. Guided reinforcement learning with learned skills. arXiv preprint arXiv:2107 2021.
Vitchyr H. Pong, Ashvin Nair, Laura Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. In International Conference on Machine Learning. 2021.
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training In 6th Anmual Conference on Robot Learning. 2022. URL https: petttts://peenreeeee /wpp/reveeeeeeeeeeeeeeeeeeeeeeee net forum? -id-iid-wwe dWKKWZuugshh
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reiinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
Stefan Schaal. Learning by demonstration. In Encyclopedia of Machine Learning and Data Mining, 1996.
Ramanan Sekar. Oleh Rybkin, Kostas Danilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 5583-8592. PMLR, 2020.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Par- rot: Data-driven behavioral priors for reinforcement learning. arXiv preprint 4rX1v:2011.10024, 2020a.
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine COG: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.1455 14500, 2020b.
Yuda Song. Yifci Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hy- brid RL: Using both offline and online data can make RL efficient. In The Eleventh International Conference Learning Representations, 2023. URL https: opwnrevieeeee..ee////pe/pe net forum? id-yyBis80iUut.
A. Srinivas, Michael Laskin, and P. Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In International Conference Machine Learning, 2020.
13

---

Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.008844 2015.
Alexander L. Strehl and Michael L. Littman. An analysis model-based interval estimation for markov decision processes. J. Comput. Syst. Sci., 14:1109-1331, 2008.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schul- man, Filip DeTurrk, and Pieter Abbeel. # Exploration: study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pages 2753- 2762, 2017.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Arun Ahuja, Yee Whye Teh, and Nicolas Manfred Otto Heess. Behavior priors for efficient reinforcement learning, ArXiv, abs/2010.14274, 2020.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. arXiv preprint arXiv:2204.02332, 2022.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint trXiv:1707.088// 2017.
Ruosong Wang. Simon Shaolei Du, Lin Yang, and Ruslan Salakhutdinov. On reward-free rein- forcement learning with linear function approximation. ArXiv, abs/2006. 11274, 2020.
Tete Xiao, Ilija Radosavovie, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.0m6 062703 2022.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. In Neural Information Processing Systems, 2021.
Mengjiao Yang and Ofir Nachum. Representation matters: offline pretraining for sequential decision making. In International Conference on Machine Learning, pages 1 11784-11794, PMLR, 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, P. Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't it change the algorithm, change the data: Exploratory data for offline reinforcement learning. ArXiv, abs/2201. 2022.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning, 2022.
Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song. Dongsheng Li, and Jing Jiang. Adaptive policy learning for offline-to-onllee reinforcement learning. ArXiv, abs/2303.07699 2023.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou Learning sparse rewarded tasks from sub-
optimal demonstrations, ArXiv, abs/2004.005330 2020.
Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl. Sebastian Schulze, Yarin Gal, Katja Hof- mann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning. ArXiv, abs/1910.08348, 2019.
14

---

A Experiment Setup Details
Our codebase is based the official RLPD codebase htttps://githii com/ http/kkostrkoooooo rlpd with minor modifications. All configurations in our experiments use 10 seeds (20 seeds for all COG experiments) and we compute the standard error to plot the confidence interval in all our plots.
A.1 Architecture Details
RLPD extends standard off-policy actor-critic algorithm which trains actor network and critic network simultaneously. Our extension of the RI.PD requires training three additional networks: a reward prediction network, a termination prediction network and an RND network. For state-based domains, each of the actor, critic, reward and RND network is multilayer perceptron (MLP) with ReLU [Agarap, 2018] activation. For the COG (image-based) domain, each network uses a pixel encoder followed by MLP.
Pixel encoder architecture. For the environments in the COG domain, pixel encoder architecture, based on the pixel encoder implemented (located https: /tt::///ittttt comm i iomt/iiorriii kovk wwwp/rlp///i pddppdlll ab het//mai//// encoders / ber of frames stacked to match the dataset from the COG domain. Following the implementation from Ball et al. [2023]. between the actor and the critic, the pixel encoder parameters are shared, and only the gradients through the critic loss objective update the encoder. The RND model, reward model, and the termination model each has its own pixel encoder with no parameter sharing.
the following RLPD's official codebase nnetworkks Changes in orange are made to the observation size and num-
Parameter
Value 4
Random Crop Padding
Number of Stacked Frames
Observation Size
[48, 48,
CNN Features
[32, 64, 128, 256] 3
CNN Filter Size
CNN Padding
Valid
CNN Stride
2
Encoder Output Latent Dimension
50
Table I: Pixel Encoder Architecture
MLP architecture. In the state-based domains (AntMaze and Sparse Adroit), use 3-layer MLP with hidden dimensions of 256. In the COG domain, we use a 2-layer MLP with hidden dimensions of 256, after pixel encoder (Table 1). The RND network outputs feature size of L 256. The reward network output single scalar. For the critic network, layer norm [Ba et al., 2016] is being added to all the layers (of the MLP) with the learnable bias and scaling except the last layer.
Action parameterization. In all of our experiments, we follow RLPD's actor design where the actor outputs mean value μ4; and a log standard deviation value log σ; for each action dimension, leading to diagonal Gaussian distribution, N (j,, diag(or). This distribution then transformed through the Tanh function such that the range of the action falls in (-1,1). The log standard deviation is clipped to be between -20 and
Termination prediction network. While predicting termination is trivial for the COG domain and the Sparse Adroit domain (the episode never terminates before the maximum episode length), predicting termination in AntMaze is just as difficult predicting the reward (e.g., the episode terminates). To make our implementation generally applicable to any environments with or without terminations, we train termination prediction network which we briefly mentioned in the beginning of the section. More concretely, the termination prediction network also takes in a state-action pair and outputs scalar, which predicts the probability of the episode terminating at the current transition through a Sigmoid function. Using the termination prediction network, the termination also being labeled in the prior unlabeled data. This results in slightly different TD back-up equation, which
15

---

can be easily integrated in the RLPD algorithm:
( + UCBR(s, + 7(1 (1, a))Q((''''')
where f(8, a) is the labeled reward value and T(s,a) a) is the labeled termination prediction (with meaning termination with a probability of 1). We left the discussion of the termination prediction network out of the main body of the paper because is minor implementation detail. For com- pleteness, we present the more detailed version of Algorithm I with the addition of the termination prediction Algorithm 2.
Training of the UCB networks. The training of the reward prediction model, termination pre- diction model and the RND model starts after 10000 environment steps for the AntMaze and the Sparse Adroit domains and 5000 environment steps for the COG domain. The RND network takes 1 gradient step on the current transition (Snew; (new) according to the following objective:
- l1f (Suew, (new) f( new: (new )l
(
where L is the number of output features of the RND network which is set to be 256. Both the reward model and the termination prediction model take the same number of gradient steps as the actor/critic for COG and 20 for AntMaze and Sparse Adroit). The termination prediction model optimizes the following objective (the transition (s, a, T) is sampled from the online replay buffer D) with T {0, 1} being the binary variable that indicates whether the episode terminates at the current transition:
C(0term) Tw [T Log Sigmoid (s) + (1 T)) Log Sigmoid- -Teom (s.
and the reward prediction model optimizes the following objective:
[() (e,, [(ra(s,a) -r
Algorithm 2 EXPLORE (with termination estimates)
1: Input: prior unlabeled data Doffline
2: Initialize the UCB estimate of the reward function: UCBR(s,
3: Initialize an estimate of the termination function: f(s.
4: Online replay buffer ID x- (
5: Initialize off-policy RI. algorithm with a policy π.
6: for each environment step do
7: Collect (s, a, a.))) using the policy v and add to the replay buffer D.
8: Update the UCB estimate UCBRR(s a) using the new transition (s, a, 5
9: Relabel each transition (8, a, s²) in Dotfiine with UCBR((s, a), T = (ss, a).
10: Run off-policy RI. update on both D and the relabeled Dottiiee to improve the policy m.
11:
end for
Modifications UCB networks for the COG domain. For the COG domain, we used ICVFss to test the incorporation of representation learning methods. We wanted to experiment with the formu- lation where the UCB network learned on top of the encoder outputs of the ICVF representations, which depend only on states/observations That is, for some observation s and encoder {(s), we wished to learn a UCB reward model r{{))). However, this formulation does not incorporate ac- tions. Therefore, for all our experiments on the COG domain, the RND network, reward model, and termination prediction network all operate on observations only. More concretely, we make the following replacements to our network architecture for the COG domains: for the RND networks we replaced fø(s, a) with (s) and [(s, a) with f((),, for the reward model, we replaced ro(s, a) with re rs),,, and for the termination prediction model, we replaced T(ss, a) with f(s).
A.2 RLPD Hyperparameters
Most of the hyperparameters we used are unchanged from the original RLPD hyperparameters, listed below.
16

---

Parameter
Value
after 5000 environment steps Online batch size
128
Offline batch size
128
Discount factor y Optimizer
0.99
Adam
Learning rate
3 10-1
Critic ensemble size Random critic target subset size Gradient Steps to Online Data Ratio (UTD)
10
2 for Adroit and COG and for AntMaze
20
Network Width
256
Initial Entropy Temperature
1.0
Target Entropy
dim(2A)2
Entropy Backups
False
Start Training
Table 2: RLPD hyperparameters.
The online and offline batch sizes change when we rum baselines that do not use offline data. In that case, the offline batch size becomes O, and the online batch size becomes 256, to keep the combined batch size the same, The update the data ratio (UTD) was set in the COG domain. For the other domains, we used the default value of 20. Action repeat is hyperparameter that determines the number times an action sampled from the policy is repeated the environment. was originally 2 for the pixel-based hyperparameters in RLPD. We set it to 1 to match the prior dataset from the COG domains.
A.3 Baseline Hyperparameters and Tuning Procedure Description
Online. For the online baseline, we don't use any offline data. So the offline batch size 0 and the online batch size is 256.
Online RND. For the online with RND baseline, we added an RND bonus on top of the ground truth rewards in the online data. That is, given an online transition (s, a, r, s'), and RND feature networks fs a), f(s, a). set
a(s.a) + (lfo(ss.)) f(s
and use the transition (s, a, f. s') in the online update. The RND training is done the same way as in method where gradient step is taken every transition collected.
Naïve Reward Labeling. The naïve reward labeling baseline is simple way incorporate the unlabeled prior data by learning a reward model online and relabeling the rewards of the prior data. For some prior transition (s, a, s'), and reward model ry learned from online interactions, we set the reward of this prior transition to be f(s. a) ro(s, a), giving the relabeled offline transition (s, a, f, 8'). Then, we take batch of 50% offline data and 50% online data, and run RLPD as usual. Note that the only difference this baseline has with our method is that & does not have an RND bonus added to
Naïve + BC. Aside from performing the reward relabeling naïvely as described in the previous section, we also regularize the actor to more closely match the behavior in the prior data by adding a behavior cloning loss to the actor loss (O is the parameter of the actor in RLPD):
Lactor, be bector.be(factoll (factor) = Lactor (factor) avccE(s.a) [log Te (a]s)]
where Luctor is the original loss function used in RLPD.
To find the best loss coefficient Orbe perform sweep 0.01, 0.1} for each domain. The best coefficient is 0.01 for all the domains.
MinR. The MinR baseline is simpler version of Naïve Reward Labeling. For all prior transitions (8, a, 8'), we set the reward f(x, a) Tmins where Tiin the minimum reward for the task. For example, for sparse 0- rewards, Tmin 0. Then, we run RLPD with this relabeled offline data.
17

---

BC + JSRL. This baseline has few additional hyperparameters. One controls the number of gradient steps to pre-train the behavior cloning guide policy on the unlabeled prior data, NISRL.- Each gradient step is taken according to the BC objective (0 (olliy is the parameters for the guide policy):
Je (0gidee guide__poliyy ) = - [log poltid (als)]
The other controls the probability of starting a trajectory by first rolling out the guide policy, fJSRL- Finally, in the course of trajectory. we switch from the guide policy to the exploration policy with probability - Y where ? = 0.99 is the discount factor. To find the best papper-prameters, perform sweep over fasnI. {0.1, 0.5, 0.9) and NISRI. {5000, 20000, 100000) for both AntMaze and Sparse Adroit. For AntMaze, the best parameter set is (fJSRL. NISRI.) = (0.9. 5000). For Sparse Adroit, the best parmaeter set is (SISRL, NOSRL) (0.5, 100000). For COG domain, none of the sweep runs succeeded. We just report the results of randomly picked parameter set: (SISRL, NNSRL) (0.5, 100000). All the return evaluations are done solely on the exploration policy (without rolling out the guide policy).
Oracle. This baseline assumes that the ground truth rewards for the online task are given in the offline data, and runs RLPD with that data.
ICVF training details. To evaluate reward functions and uncertainty estimation, we run our method on top of a ICVF representation [Ghosh et al., 2023] pre-trained on the offline prior data. We use the open-source implementation from the authors at https: / thub. com/ d dibyaghoshh sevf_release, using the COG encoder as the visual backbone and the default hyperparameters otherwise from the open-source implementation. We pre-train the model for 0.75 x 10th timesteps, and use the final checkpoint as the encoder initialization for both RND and the reward model predic- tor. Notably, the representation is only ever used as an initialization, and is not frozen or regularized during reward training.
Parameter
Value
Batch size
256
Discount factor
0.99
Optimizer
Adam
Learning rate
3 x 10-4
MLP Width
256
MLP layers
2
Table 3: ICVF hyperparameters.
B
Domain Details
AntMaze (state) In the D4RL AntMaze Fu et al. [2020] environment, an 8-DoF quadrupedal "ant" agent navigates maze from a fixed starting location and towards fixed goal location. The reward is sparse: O around the goal and -1 elsewhere. The observation space is 29 dimensions, with the first dimensions indicating the position of the ant within the maze. The action space is 8 dimensional. Trajectories terminate upon reaching the goal. We tested 6 total AntMaze tasks within a total of 3 mazes of increasing size and complexity: umaze, umaze uwaze-divereeeeeee verse, medi um um-uuappppp play, medi um diverse, large-play, and large- Marge-divees diverse.
The offline data for this environment consists of trajectories with many different start and end po- sitions. The nature these positions depends on the type of environment. In the play AntMaze environments, the start and end positions handpicked whereas in the diverse AntMaze en- vironments, they are randomly chosen. Since the actual task contains only a single fixed start and goal, the offline data does not always correspond to the online task. The coverage of the offline data is relatively complete over the entire maze.
Adroit (state) The Adroit environment Nair et al. [2021] we used consisted of a 24-DoF hand robot agent and 3 dexterous manipulation tasks: pen-spinning, door-opening, and ball relocation. The data distribution for the Adroit environments is relatively narrow. The prior data for these environments consists of expert trajectories from human demonstrations, as well as many behavior- cloned trajectories on the expert data. As the prior data here is closer to the target task distribution
18

---

Figure 8: Additional visualizations on offiine datasct with four different kinds of corruptions. Top: visu- alization of the coverage of the transitions in the offline data. Bottom: visualization of the direction of the transitions in the offline data. Left: All, Middle Left: Orthogonal Transitions, Middle Right: Insufficient Coverage, Right: Data.
compared to other environments, these experiments in Adroit allow us to compare the performance of our method more critically against our baselines that are less exploration focused.
COG (vision) The environments and datasets from COG come from the repo: https: gi github. cwm/ cww/avisiiiiiiiciee.ce..c...i..a srrisinnh59999
In the COG environments Singh et al. [2020b]. the actor is simulated WidowX robot arm. The observation space is image-based with dimensions (48, 48, 3). The rewards are sparse, with +1 reward when the task is completed, and 0 reward otherwise. The horizon is infinite, but for exploration and evaluation, we set max path length to end the episode.
The first environment we call Pick and Place. For this environment, the arm is placed in front of small object and tray. The task is pick up / grasp the object and place it in the tray, where success occurs when the object is in the tray. The object initialized at different locations on the ground. The maximum episode length 40.
Visualizations of data used in the COG tasks. The left block contains the first stage of the task. and the right block contains the second stage of the task. The first is for the Pick and Place task, the second row picking out a ball from closed drawer, the Grasp from Closed Dr arawer task, and the third row is picking out ball from blocked drawer, the Grasp from Blocked Drawer task.
Figure 9:
The offline dataset for the Pick and Place consists of trajectories of grasping attempts and separate tra- jectories of placing attempts. The data for these environments comes from scripted policy that achieves roughly 40% success at grasping the object and roughly 90% success placing the object into the tray. There are 100 trajectories of grasping attempts and 50 trajectories of placing attempts. There may be overlaps in states in the grasping and placing data, but there is no trajectory that completes the entire pick and place task. For our RLPD baseline, where the rewards from the prior dataset are considered, we set reward of +1 when the object is successfully placed in the tray, and 0 for all other transitions.
We ran experiments on two drawer environments from COG. The main interaction in these environ- ments is between the WidowX robot arm and two tiered drawer. The first setup we call Grasp from Closed Drawer. In this setup, the bottom drawer is closed, and an object is inside the bottom drawer. The task is to open the bottom drawer and pick out the object. The second setup we call Grasp from Blocked Drawer 1. In this setup, the bottom drawer is closed and blocked by an open upper drawer. The task is to close the upper drawer, open the bottom drawer, and pick out the object. For both drawer environments, a success occurs when the object is taken out of the
19

---

Oracle
Online
MinR Naive
Naive BC
Ours
Online + RND
BC JSRL
Pick and Place
Grasp from Closed Drawer
Grasp from Blocked Drawer 1
Normalizen Return
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
Environment Steps (x10%)
Figure 10: The performance on COG tasks with larger offline dataset. BC+JSRL is able to match oracle performance. In the main body of the paper (Figure 3), we are using a dataset that is roughly 1% in size.
bottom drawer. The maximum episode length is 50 for Grasp from Closed Drawer, and 80 for Grasp from Blocked Drawer 1.
The offline dataset we used for the drawer tasks consists of trajectories of drawer opening attempts and separate trajectories taking the object out an open drawer. For example, the Grasp from Closed Drawer task, the drawer opening attempts consist of trajectories of opening the bottom drawer. Mcanwhile, for the Grasp from Blocked Drawer 1 task, the drawer open- ing attempts consist of trajectories of closing the top drawer and opening the bottom drawer. For both tasks, the trajectories of taking the object out of an open drawer are the same. Similar to P Pick and P Place, these trajectories
generated using randomized scripted policy described in Singh et al. [2020b] that has about 40-50% success rate at opening the drawer and about a 70% success at taking the object out of the drawer. Like the Pick and Place task, there is trajectory that completes the entire task, but there may be overlaps in states. For our RLPD baseline, set reward of +1 when the object has been successfully taken of the bottom drawer, and for all other transitions. There are 25 trajectories of drawer opening attempts and 25 trajectories of object-taking attempts for the two drawer tasks.
Initially, we experimented with a larger offline dataset with 100 times more trajectories for these three tasks and we have found that the BC+JSRL baseline to closely match the oracle performance and our method did not really an edge over that baseline (see Figure 10). This is perhaps to be expected as BC+JSRL puts strong prior following the offline data action by behavior cloning, and when offline data are good demonstration data. On the other hand, our method does not assume that the offline dataset is of good quality, and does not try to closely follow the actions. Consequently. our method is able to perform well more robustly across different settings compared to BC+JSRL shown in main results (Figure
C Additional Results
C.1 Full AntMaze results.
We include the complete set of AntMaze results in this section. Figure 11 shows the normalized return, which is the percentage success rate of the ant reaching the desired goal. Figure 12 shows the coverage metric, which is estimate of how much pereentage of the free space that the ant has explored in the maze.
20

---

Oracle
Online
MinR
Naive + BC
Ours
Online + RND
Naive
BC JSRL
umaze-diverse
medium-diverse
large-diverse
1.00
0.75
Normalized Return
0.50
0.25
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
umaze
medium-play
large-play
1.00
0.75
0.50
0.25
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x10%)
Figure 11:
The success rate for the 6 AntMaze tasks. Every method except Oracle has no access to the reward function. Our method consistently performs well and nearly matches the oracle performance.
Oracle Ours
Online
MinR
Naive + BC
Online RND
Naive
BC JSRL
umaze-diverse
medium-diverse
large-diverse
1.00
0.75
0.50
0.25
Coverage
0.00
0.0
0.1
0.0
0.1
0.2
0.0
0.1
0.2
0.3
umaze
medium-play
large-play
1.00
0.75
0.50
0.25
0.00
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Environment Steps (x10%)
Figure 12:
State coverage for the AntMaze tasks, estimated by counting the number of square regions that the has visited, normalized by the total number of square regions that visited.
C.2 Full Adroit Results
We include the complete set of Adroit results in this section. Figure 13 shows the normalized return for each method on each of the three sparse Adroit tasks. Figure studies the effect periodic resetting the reward and the RND networks the hardest task, relocate.
21

---

Oracle
Online
MinR
Naive + BC
Ours
Online + RND
Naive
BC JSRL
Normalized Return
pen-binary-vo
door-binary-v0
relocate-binary-v0
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Environment Steps (x106)
Figure 13: The normalized return for sparse Adroit tasks. Every method except Oracle has no access to the reward function. Naïve reward modeling without optimistic relabeling is sufficient for all Adroit tasks. Periodically resets (every 1000 environment steps) are used for the relocate task. See Figure 14 below for the reset ablation study.
Oracle
Naive
Ours
Naive + Reset
Ours + Reset
relocate-binary-vo
1.0
Normalized Return
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4 0.6 0.8
Environment Steps (x10%)
Figure 14: Periodically resetting reward model crucial for success relocate. experiment with different reset intervals and have found that resetting the reward model after every 20K gradient steps (IK environment steps) to work well. We hypothesize that the periodic resetting procedure might have alleviated the overfitting issue of the reward function, allowing it continuously adapt to incoming transitions.
22

---

C.3
Full COG results
Oracle
Online
MinR
Naive BC
Ours
Online + RND
Naive
BC JSRL
Pick and Place
Grasp from Closed Drawer
Grasp from Blocked Drawer 1
Normalized Return
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.1
0.2
0.0
0.1
0.2
0.0
0.1
0.2
Environment Steps (x10%)
Figure 15: The performance on COG tasks.
C.4 Implicit-Q-learning (IQL) as the backbone RL algorithm.
Conceptually, our proposed method, ExPLORe, is not specific the selected RL method, and be combined with any RL methods that take both offline and online transitions. Other than RLPD, we experimented with another RL algorithm, IQL [Kostrikov et al., 2021], in the offline online fine- tuning setting where we run IQL on an online replay buffer prepended with the reward-relabelled offline data. In experiments (Figure 16), we have found the of the RND bonus in the offline data to be essential for accelerating learning in both medium datasets (without RND bonus, none of the runs succeed).
Offline IQL Oracle
Ours
Naive
umaze
umaze-diverse
medium- play
medium- diverse
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
Normalized
0.50
0.50
0.50
0,50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.000.25
0.500.75
1.00
0.000.25
0.50 1.00
0.000.
25000.
1.00
0.000.250.000
1.000.25..50.751..
1.00
Figure 16: Results on 4 AntMaze datasets. The oracle offline IQL numbers are directly taken from the original paper. It uses the offline dataset with ground truth reward. The other curves do not have access the ground truth reward in the offline dataset. 10 seeds are used for each curve.
C.5 Pre-trained goal-conditioned (GC)-IQL skills.
To get a sense of how well skill-discovery may perform, we perform additional empirical study in antma aeeelaae -ar-aa-reeiiii sereedivereeeee
with set of unsupervised, pre-trained skills. In particular, specify the skills to be a set of pre-trained goal-conditioned (GC) policies from the offline data where the 2-D goal coordinate specifies the skill. We chose this more manually defined skill set with more inductive biases to demonstrate an approximate upper-bound bound on an approach that discovers skills in an unsupervised fashion.
We the offline data to pre-train goal-conditioned policies using GC-IQL on the offline dataset. In the online phase, we perform skill-inference on this set of skills by using RLPD to train a high-level policy that outputs actions corresponding 2D goal coordinate in the maze for the underlying skill to reach. While this combination leads to increased coverage during early exploration, we found that skill inference using RLPDD eventually gets stuck if reward is not found and ceases explore further, unless additional online RND bonuses are applied. In Figure 17, we plot the performance of this skill inference comparison (5 seeds for skill-inference, for skill pre-training, and ten seeds for the rest)- while inference in the space of skills Icads to much faster exploration than with standard
23

---

online RND, we found the best variant of skill disloveisy to be slightly worse than our method (GC- IQL+Online RLPD + Online RND vs. Ours).
In summary, we found that while skill-learning and online skill inference does lead to faster explo- ration online-only exploration, it still lags in performance compared to approach. We note that our method has several ad advantages over skill-leaning baseline, such as not needing separate pre-training phase, and not needing to specify discover a separate skill
Oracle
Online Only
GC-IQL + Online RLPD
Ours
Online Only RND
GC-IQL Online RLPD Online RND
Normalized Score
Coverage
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.0
0.1
0.2
0.3
0.1
0.2
0.3
Results AntMaze- Laaize-Larr----eeeee Diverse using pre-trained goal-conditioned IQL offline data. The a-axis is the number environment steps x 100. The y-axis is the normalized return (left) and the coverage percentage (right). The pre-trained IQL agent achieves around 50% success rate by using the correct goal.
Figure 17:
D Visualization Details for Figure
Since the prior data on COG contains no full trajectories that complete the task, to generate Fig- ure 5, successful trajectory on the Grasp from Closed Drawer environment was made by combining successful trajectory opening the drawer and successful trajectory taking the object out of the drawer from the prior dataset. Afterward, we evaluate the reward model on this successful trajectory for 20 training seeds and normalize each to have
and unit variance.
Below, we provide the same visualization as Figure 5, on other successful trajectories com- bined from the prior data on Grasp from Closed Drawer and observe similar effect.

---

Al 25k, Envirnmmnt Stops
At 25k Enviromment Steps
D.
20
Namma Aamuteddacrr
Al 25t Envionment Steps
Pente Lodi Ponzd
Al 2% Envirommennt Stegs
10
totaattd Fower1
Touectoy Stops
Norroa and Sowats
15k Envisemment Steps
Al 258 Endiremmet Stups
E Compute Description
use Tesla V100 GPU running the experiments. Each AntMaze experiment (300K envi- ronment steps) takes around 2 GPU hours. Each Adroit experiment (IM environment steps) takes around 6 GPU hours. Each COG experiment (250K environment steps) takes around GPU hours. To reproduce all the results in the main body of the paper, it takes around 10 (seeds) x 6 (tasks) x (methods)
x 2 (AntMaze) 10 (seeds) x (tasks) x 6 (methods) x (Adroit) 20 (seeds) x (tasks) x 6 (method) 3 (COG) 2880 GPU hours.
25


Reviews:
[{"review_id": "ztAi8tGAmA", "reviewer": "Reviewer_Mcz4", "summary": "This paper proposes a method to leverage unlabelled offline data to expedite online exploration. The idea is to train a reward model and random network distillation models on online data and use them to construct a UCB estimate for offline data. By training an off-policy RL algorithm on both online and relabelled offline data, the agent is encouraged to visit novel states represented in the offline data that are deemed to have high rewards. This is in contrast with most online exploration methods which only expand the exploration frontier. The authors validate their method on state-based and visual domains and demonstrate that their method achieves more efficient exploration compared to baselines. ", "questions": "- How does the quality of offline data affect performance? What happens when the offline data doesn't cover high-return trajectories? What happens if it contains orthogonal trajectories (e.g. the task is to go left, but the data goes right)? It would be helpful to see an ablation experiment with different dataset qualities vs. exploration efficiency.\n- It would be nice to include a comparison to an unsupervised skill-learning baseline. I expect UCB exploration to achieve better asymptotic performance than unsupervised skill-learning when the dataset does not cover optimal trajectories. \n", "limitations": "The authors adequately addressed the limitations of their method in the paper. They mention that sometimes naively fitting the reward to online exploration can lead to overfitting. Moreover, the UCB estimates can be volatile at the beginning of training. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method is simple and effective, providing an appealing means to leverage unlabeled offline datasets to facilitate online learning.\n- The experiments are substantiative, demonstrating the superiority of UCB rewards compared to pure online exploration as well as other exploration methods that utilize unlabelled offline data. It is nice to know that in addition to state-based domains, the method is also compatible with visual domains through pre-trained visual representation, which further expands its applicability. Moreover, the analytical experiments in Section 4.5 offer clear evidence that the reason behind improved online learning is efficient exploration.\n- The overall presentation of the paper is coherent.", "weaknesses": "- The problem setting is a bit awkward. Since the offline and online data have to be from the same domain, this method does not benefit from Internet-scale unlabeled data. It makes sense if the offline data is collected from a diverse set of tasks with the same dynamics but different underlying reward functions, and the goal is to show fast adaptation to new rewards via efficient exploration. But in that case, it would have to compare with the unsupervised skill-learning line of work, e.g. DIAYN [1], RaMP [2], and I'm not sure if exploration can outperform adaptation in terms of sample efficiency.\n- The quality of the offline dataset is important. Not only would the data need to be from the same domain, but it would also need to cover interesting trajectories for the task to really facilitate exploration. If the data is from a completely orthogonal task then it might even hinder exploration, depending on the balance between the reward estimate and the uncertainty quantification.\n\n[1] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine. Diversity Is All You Need: Learning Skills Without A Reward Function. ICLR 2019.\n[2] Boyuan Chen, Chuning Zhu, Pulkit Agrawal, Kaiqing Zhang, Abhishek Gupta. Self-Supervised Reinforcement Learning that Transfers using Random Features, ArXiv Preprint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words and thoughtful feedback about our paper. We addressed your first concern on offline data quality by conducting additional ablation experiments with different offline data distributions (e.g., poor data coverage, orthogonal trajectories) and showed that our method is also amenable and robustly well-performing in these situations. We addressed your second concern on the lack of unsupervised skill-learning comparison by adding an additional baseline that makes use of pre-trained goal-conditioned policies as skills and runs online RL to select among these skills. \n## “How does the quality of offline data affect performance?”\nFrom the additional ablation experiments, we have found that our method (when combined with online exploration bonus) can effectively utilize offline data with varying data qualities (Figure 1 in the one-page PDF). Specifically, we experimented with three different settings:\n- **Orthogonal Transitions**: all the transitions that move in the up/right (towards the goal) are removed from the offline data. \n- **Insufficient Coverage**: all the transitions around the goals are removed from the offline data. \n- **1% Data**: A random 1% of the transitions are kept in the offline data. \n\nNaively using our method in the **Insufficient Coverage** setting yields zero success, but this is to be expected as our algorithm does not incentivize exploration beyond the offline data. We find that this can be patched by simply combining our method with an additional RND bonus on online data, allowing our method to learn even when data has poor coverage and no data near the goal is unavailable. Similarly, we have found that our method can also boost the performance in the **Orthogonal Transitions**, where no transitions move in the direction of the goal. For the **1% Data** setting, our method without the online RND bonus can still largely match the oracle performance. It is also worth noting that in the **Orthogonal Transitions** setting, our method achieves good performance whereas the oracle fails to succeed. These additional results, while initially surprising even to us (especially the “orthogonal” setting, which seems very difficult), strongly suggest that our method can lead to significant improvement even when the data is not very good – certainly such data would be woefully inadequate for regular offline RL or naive policy initialization.\n\nIn summary, although our method does rely on the coverage assumption of the offline data to achieve good performance (as pointed by reviewers and evidenced in our additional experiments), it can be easily combined with an online exploration bonus to address the coverage issue and make use of different offline data distributions effectively and robustly to accelerate online exploration.\n## “It would be nice to include a comparison to an unsupervised skill-learning baseline.”\nTo get a sense of how skill-discovery may perform, we perform an additional empirical study in AntMaze-large-diverse-v2 with a set of unsupervised, pre-trained skills. In particular, we specify the skills to be a set of pre-trained goal-conditioned (GC) policies from the offline data where the 2-D goal coordinate specifies the skill. We chose this more manually defined skill set with more inductive biases to demonstrate an approximate upper-bound on an approach that discovers skills in an unsupervised fashion. \n\nWe use the offline data to pre-train goal-conditioned policies using GC-IQL on the offline dataset. In the online phase, we perform skill-inference on this set of skills by using RLPD to train a high-level policy that outputs actions corresponding to 2D goal coordinate in the maze for the underlying skill to reach. While this combination leads to increased coverage during early exploration, we found that skill inference using RLPD eventually gets stuck if reward is not found and ceases to explore further, unless additional online RND bonuses are applied. In Figure 3, we plot the performance of this skill inference comparison (5 seeds for skill-inference, 3 for skill pre-training, and ten seeds for the rest) – while inference in the space of skills leads to much faster exploration than with standard online RND, we found the best variant of skill-discovery to be slightly worse than our method (GC-IQL+Online RLPD + Online RND vs. Ours). \n\nIn summary, we found that while skill-learning and online skill inference does lead to faster exploration over online-only exploration, it still lags in performance compared to our approach. We note that our method has several advantages over a skill-learning baseline, such as not needing a separate pre-training phase, and not needing to specify or discover a separate skill set. We note that due to the rebuttal time limit, we were only able to tune a limited set of hyperparameters such as the horizon length of the skill, so it is entirely possible that further tuning of skill-discovery might yield stronger results.\n## “The problem setting is awkward.”\nThis is definitely a valid concern and making good use of Internet-scale unlabeled data is a great direction for future work!! There is however considerable interest in developing algorithms that can effectively utilize offline data from the same domain (e.g., [1, 2]). In addition, the problem setting is more suitable for tasks where large-scale data are not available (e.g., inventory management [3], nuclear fusion [4]).\n\n[1] Ajay, Anurag, et al. \"Opal: Offline primitive discovery for accelerating offline reinforcement learning.\" arXiv preprint arXiv:2010.13611 (2020).\n\n[2] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. \"Accelerating reinforcement learning with learned skill priors.\" Conference on robot learning. PMLR, 2021.\n\n[3] Madeka, Dhruv, et al. \"Deep inventory management.\" arXiv preprint arXiv:2210.03137 (2022).\n\n[4] Degrave, Jonas, et al. \"Magnetic control of tokamak plasmas through deep reinforcement learning.\" Nature 602.7897 (2022): 414-419."}, {"review_id": "JxBgfQ3jth", "reviewer": "Reviewer_egZc", "summary": "The paper propose a method exploiting unlabeled prior data (i.e. trajectories without the reward signal) for more efficient online learning. To this end, a reward model is trained (based on the online data) and the so-labeled offline data are utilised jointly with the offline data. \n", "questions": "(see above)", "limitations": "Limitations are mostly missing. In the discussion section, the authors admit that the presented approach is the 'initial instantiation' and least some problems. However, it is not clear if the presented list is comprehensive. ", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The paper tackles an important problem of using prior data towards accelerating online learning. The experimental setup covers three different domains (AntMaze, Adroit, COG) and $7$ baseline methods, both of which make a solid impression. The presentation of the method is very clean, and the paper is easy to follow. ", "weaknesses": "I find the paper's analysis somewhat shallow with respect to the design choices and analysis. Below, I list a tentative questions and suggestions:\n- the paper does not study much how does the offline data distribution affects the training. I acknowledge that different domains have different characteristics with this respect, however, it is hard to conclude something meaningful here.\n-  The conventional knowledge that the offline data should be treated with pessimism, here we assume optimism? How does these two relate? \n- Is the solution specific to the selected offline RL method? I understand that this can be done easily, but would it work? I'd be great to have some experiments or at least some arguments.\n- I'd be great to know more about the uncertainty modelling. How accurate is it? What is the best balance of the uncertainty bonus and the learned reward signal. \n\n\n\nI am aware of practical constraints, and I do not expect the authors to include all suggestion listed above. Nevertheless, I hope that including some of them would enrich the paper. \n\n### Small\n- The number of seeds (4-5) is rather small for the RL experiments, but is perhaps excusable by number of environments and baselines. \n- I am not fully convinced by Sec 4.5. It states that the method improves the coverage (a proxy of the exploration performance). However, it is not completely clear, that this yields benefits to the online training. That is, I could imagine that there is some spurious correlation here. I admit, that I do not have any particular better idea, though.\n- some hacks are needed (e.g. resetting reward model)\n- I find Sec 4.4 (representation learning) a little bit detached from the main topic of the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words and thoughtful feedback about our paper. We addressed one of your main concerns on “how the effect of offline data distribution affects training” by conducting additional ablation experiments with different offline data distributions (e.g., poor data coverage, orthogonal trajectories) and showed that our method is also amenable and robustly well-performing in these situations (Figure 1). We have also addressed another main concern on the applicability of our approach to other RL methods by showing that our method can also be applied to IQL empirically (Figure 2) and conceptually to any RL methods that can make use of both offline and online transitions.\n## “The paper does not study much how does the offline data distribution affect the training.”\nFrom the additional ablation experiments, we have found that our method (when combined with online exploration bonus) can effectively utilize offline data with varying data qualities (Figure 1 in the one-page PDF). Specifically, we experimented with three different settings:\n- **Orthogonal Transitions**: all the transitions that move in the up/right (towards the goal) are removed from the offline data. \n- **Insufficient Coverage**: all the transitions around the goals are removed from the offline data. \n- **1% Data**: A random 1% of the transitions are kept in the offline data. \n\nNaively using our method in the **Insufficient Coverage** setting yields zero success, but this is to be expected as our algorithm does not incentivize exploration beyond the offline data. We find that this can be patched by simply combining our method with an additional RND bonus on online data, allowing our method to learn even when data has poor coverage and no data near the goal is unavailable. Similarly, we have found that our method can also boost the performance in the **Orthogonal Transitions**, where no transitions move in the direction of the goal. For the **1% Data** setting, our method without the online RND bonus can still largely match the oracle performance. It is also worth noting that in the **Orthogonal Transitions** setting, our method achieves good performance whereas the oracle fails to succeed. These additional results, while initially surprising even to us (especially the “orthogonal” setting, which seems very difficult), strongly suggest that our method can lead to significant improvement even when the data is not very good – certainly such data would be woefully inadequate for regular offline RL or naive policy initialization.\n\nIn summary, although our method does rely on the coverage assumption of the offline data to achieve good performance (as pointed by reviewers and evidenced in our additional experiments), it can be easily combined with an online exploration bonus to address the coverage issue and make use of different offline data distributions effectively and robustly to accelerate online exploration.\n## “The conventional knowledge that the offline data should be treated with pessimism, here we assume optimism? How does these two relate?”\nThese ideas are mutually synergistic: the conventional wisdom says that points outside the offline data should be treated with pessimism (e.g. be pessimistic about any action or state not in the offline dataset). Our principle is that points within the offline data should be treated with optimism (e.g. be optimistic about any state that was in my offline dataset, but whose reward I don’t know yet). Pessimism outside the offline data encourages the model to stay within the support of the offline data, while optimism within the offline data encourages the model to explore all possible directions present in the offline data.\n## \"Is the solution specific to the selected offline RL method?”\nThe solution is not specific to the selected RL method. Conceptually, our method can be combined with any RL methods that take in both offline and online transitions. For the rebuttal, we experimented with the IQL fine-tuning setting where we run IQL on an online replay buffer prepended with the reward-relabelled offline data. In our experiments (see Figure 2 in the one-page PDF attached), we have found the use of the RND bonus in the offline data to be essential for accelerating learning in both medium datasets (without RND bonus, none of the runs succeed). We will add the additional experiments in our next revision.\n## “How accurate [is the uncertainty modeling]? \nWhat is the best balance of the uncertainty bonus and the learned reward signal?”\nWe measure the quality of the RND estimates by measuring the coverage and performance gains from using the reward bonus; this matches how prior works on exploration bonuses (e.g., the original RND paper) have evaluated the quality of learned uncertainty estimates. If you had in mind a particular metric for measuring the quality of learned RND uncertainty estimates, we would be happy to analyze that metric in our next revision. Regarding the weighting between reward and uncertainty bonus, we found the algorithm was relatively stable to this hyperparameter in our tasks (we use a coefficient of 1), but more careful tuning may improve further.\n\n## Small Points\n- “The number of seeds is small” – we will run more seeds (10) and include the results in our next revision.\n- Coverage as a performance proxy is not convincing – it is true that coverage does not always correlate with the performance, but having a decent amount of coverage is a prerequisite for the method to achieve good performance when the reward function is unknown (the agent needs to first discover the goal then practices reaching the goal robustly). In most of our experiments on AntMaze the baselines achieve near 0 coverage, which hinder their ability to even locate the goals.\n- “some hacks are needed” – due to the character limit, please see our response to Reviewer vsqs (the section with the heading “Why is reset really needed?”) regarding the reset."}, {"review_id": "uzmTlzrvO6", "reviewer": "Reviewer_hFD7", "summary": "The paper proposes a simple method to leverage reward-free data in order to accelerate exploration in sparse reward tasks. This is done by learning a reward model and an uncertainty model during online learning, labeling the unlabeled data with an upper confidence bound based on these models, and sampling from both the online data and the pseudo-labeled offline data during learning. It results in fast achievement of sparse-reward tasks in a number of domains.", "questions": "With respect to the first weakness above, it might be nice to have an ablation experiment that limits the frontier of the offline data so as to examine the performance in that setting . In particular this is easy with AntMaze, one can only keep transitions in the first few quadrants of the maze.", "limitations": "The discussion section (6) mentions limitations in the form of quirks in reward optimization on Adroit domain, as well as a hypothetical limitation due to stability in learning (which does not seem to manifest). But, as discussed in weaknesses, I think there is a large limitation in that this method as demonstrated requires coverage in the demonstrations in order to be feasible. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The presentation is very well done.\n- The method is extremely simple and clear to follow, simple enough to implement just from the paper with some experience in the field.\n- The coverage results in Figure 2 are very strong (if not so surprising).\n- The simple descriptions of baselines in 4.2 are helpful.\n- The results against baselines are very strong, even when compared to a topline oracle (Figure 3, 4)", "weaknesses": "- One of the core assumptions of the method is somewhat obscured, namely that the offline data must contain good coverage in order for learning to be accelerated, otherwise there would be a significant gap between the frontier of the offline data, and the reward that must eventually be achieved, which would still result in slow exploration. As a result, I'm not sure I agree with the phrasing in section 4.5 that this method has better exploration, thus it achieves higher reward. Good exploration in the offline data is a prerequisite rather in order for the method to work well. If that exploration were not achievable in the first place, we could not use the existing method to accelerate learning. In some sense, we must have a working policy in order to learn a working policy.\n- Related work comes at the end, in general this makes it difficult to contextualize the work done in the paper against prior methods, though details on the baselines help somewhat.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words and thoughtful feedback about our paper. We addressed your main concern regarding the coverage assumption of the offline data by conducting additional ablation experiments with different offline data distributions (e.g., poor data coverage, orthogonal trajectories) and showed that our method is also amenable and robustly well-performing in these situations (Figure 1 in the one-page PDF attached).\n\n## “It might be nice to have an ablation experiment that limits the frontier of the offline data”\nFrom the additional ablation experiments, we have found that our method (when combined with online exploration bonus) can effectively utilize offline data with varying data qualities (Figure 1 in the one-page PDF). Specifically, we experimented with three different settings:\n- **Orthogonal Transitions**: all the transitions that move in the up/right (towards the goal) are removed from the offline data. \n- **Insufficient Coverage**: all the transitions around the goals are removed from the offline data. \n- **1% Data**: A random 1% of the transitions are kept in the offline data. \n\nNaively using our method in the **Insufficient Coverage** setting yields zero success, but this is to be expected as our algorithm does not incentivize exploration beyond the offline data. We find that this can be patched by simply combining our method with an additional RND bonus on online data, allowing our method to learn even when data has poor coverage and no data near the goal is unavailable. Similarly, we have found that our method can also boost the performance in the **Orthogonal Transitions**, where no transitions move in the direction of the goal. For the **1% Data** setting, our method without the online RND bonus can still largely match the oracle performance. It is also worth noting that in the **Orthogonal Transitions** setting, our method achieves good performance whereas the oracle fails to succeed. These additional results, while initially surprising even to us (especially the “orthogonal” setting, which seems very difficult), strongly suggest that our method can lead to significant improvement even when the data is not very good – certainly such data would be woefully inadequate for regular offline RL or naive policy initialization.\n\nIn summary, although our method does rely on the coverage assumption of the offline data to achieve good performance (as pointed by reviewers and evidenced in our additional experiments), it can be easily combined with an online exploration bonus to address the coverage issue and make use of different offline data distributions effectively and robustly to accelerate online exploration.\n\n"}, {"review_id": "b7vhLsb6uH", "reviewer": "Reviewer_vsqs", "summary": "This work presents an off policy RL algorithm that use prior unlabeled data in a novel manner. The mechanism use an \"optimistic\" reward labeled based on  a UCB estimate, that it is practically an iterative RND full update of the unlabeled dataset based on a stepwise exploration update of the RND functions. This reward of the unlabeled data smoothly decay making the agent explore better in comparison to other methods. ", "questions": "Given the concern of the previous section, the research questions might have been extended to understand the quality of the unlabeled data.  Can the authors help me understand if this is valid concern?\n\nOther comments were done in the previous section.\n", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is clear on explaining all the methods, algorithms and theory used. Has a in depth citation of the literature. Compare the methods to relevant methods. It also characterize well were the method stand with respect to related efforts. The results clearly surpass comparative methods in sample efficiency. It also re-use state of the art results (RLPD and ICVF ) in the project. It also clearly and unambiguously answer it three research questions.\nNovel method for reward labeling prior unlabeled data.", "weaknesses": "The re-initialization of the reward model after 20k for the relocate task is puzzling, which imply that this algorithm need a more robust extension to deal with this situation. The situation is not clearly understood. Why the reset are really needed? How can the process automatic identify the needs for reset or change the interpolation dynamics of the algorithm too adapt to this situation. Seems that something is missing.\nIt seems that this approach it is inefficient  when processing or bad demonstration or subpar-demosntration in the sense of what we can derive from not able to learn from high dimensional data, and the need for ICVF to learn a compact state. Which is reasonable but also imply that it would been nice to have this analyzed in the work. The analysis of the quality and quantity of the unlabeled data, for example, as was done in (How to Leverage Unlabeled Data in Offline Reinforcement Learning. Yu et al 22) This might indicate that the method needs a certain type of unlabeled data to be successful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words and thoughtful feedback about our paper. We addressed your main concern regarding the reward resetting by providing more insights on the nature of the underlying issue (that is likely orthogonal to our main focus of the paper) as well as providing additional empirical evidence that our method is robust against offline datasets with different qualities.\n\n## “Why is reset really needed?”\nWe hypothesize that periodical reset reduces the plasticity of the reward function and allows the reward function to be fitted to new transitions well, rather than being over-fitted to the earlier transitions observed. This plasticity issue can be especially problematic in the relocate task because the success rate is almost zero throughout the first 300K environment steps and the reward function can overfit to a constant reward target 0 (since the task is of sparse reward). The plasticity phenomenon has been observed in various prior works [1, 2, 3] in the context of deep reinforcement learning especially in the high UTD (update-to-data) regime which we study. Addressing this issue is a very important research question in general for deep RL algorithms, but is orthogonal to our goal in the paper which primarily focuses on how the reward should be labeled in offline data so that deep RL algorithms can make use of the unlabeled offline data effectively.\n\n## \"Data quality analysis”\nIn addition, we provide empirical evidence that our method can be robust to different qualities of the data, which is an indirect piece of evidence that the issue underlying reward resetting might not be coming from the data qualities (Figure 1 in the one-page PDF). Specifically, we experimented with three different settings:\n- **Orthogonal Transitions**: all the transitions that move in the up/right (towards the goal) are removed from the offline data. \n- **Insufficient Coverage**: all the transitions around the goals are removed from the offline data. \n- **1% Data**: A random 1% of the transitions are kept in the offline data. \n\nNaively using our method in the **Insufficient Coverage** setting yields zero success, but this is to be expected as our algorithm does not incentivize exploration beyond the offline data. We find that this can be patched by simply combining our method with an additional RND bonus on online data, allowing our method to learn even when data has poor coverage and no data near the goal is unavailable. Similarly, we have found that our method can also boost the performance in the **Orthogonal Transitions**, where no transitions move in the direction of the goal. For the **1% Data** setting, our method without the online RND bonus can still largely match the oracle performance. It is also worth noting that in the **Orthogonal Transitions** setting, our method achieves good performance whereas the oracle fails to succeed. These additional results, while initially surprising even to us (especially the “orthogonal” setting, which seems very difficult), strongly suggest that our method can lead to significant improvement even when the data is not very good – certainly such data would be woefully inadequate for regular offline RL or naive policy initialization.\n\nIn summary, although our method does rely on the coverage assumption of the offline data to achieve good performance (as pointed by reviewers and evidenced in our additional experiments), it can be easily combined with an online exploration bonus to address the coverage issue and make use of different offline data distributions effectively and robustly to accelerate online exploration.\n\n[1] Nikishin, Evgenii, et al. \"The primacy bias in deep reinforcement learning.\" International conference on machine learning. PMLR, 2022.\n\n[2] Lyle, Clare, et al. \"Understanding plasticity in neural networks.\" arXiv preprint arXiv:2303.01486 (2023).\n\n[3] Sokar, Ghada, et al. \"The dormant neuron phenomenon in deep reinforcement learning.\" arXiv preprint arXiv:2302.12902 (2023).\n"}]

Author rebuttal:
We appreciate the feedback from all reviewers. We added three additional sets of experimental results in the one-page PDF attached to address the three shared main concerns from the reviewers --

**1. How does offline data quality affect online performance? -- Figure 1**

We ran additional ablation experiments with varying offline data distributions, and we have found that our method (when combined with online exploration bonus) can effectively utilize offline data with varying data qualities. Although our method alone does rely on the coverage assumption of the offline data to achieve good performance (as pointed by reviewers and evidenced in our additional experiments), it can be easily combined with an online exploration bonus to address the coverage issue and make use of different offline data distributions effectively and robustly to accelerate online exploration.

**2. Can our method be used in other RL algorithms? -- Figure 2**

Conceptually, our method can be combined with any RL methods that take in both offline and online transitions. To illustrate this point, we ran additional experiments where we applied our method on top of IQL where we run IQL on an online replay buffer prepended with the reward-relabelled offline data. In our experiments, we have found the use of the RND bonus in the offline data to be essential for accelerating learning in both medium datasets (without RND bonus, none of the runs succeed).

**3. Lack of unsupervised skill discovery baseline. -- Figure 3**

We include an additional baseline on the AntMaze-large-diverse-v2 with a set of unsupervised, pre-trained skills. In particular, we specify the skills to be a set of pre-trained goal-conditioned (GC) policies from the offline data where the 2-D goal coordinate specifies the skill. We chose this more manually defined skill set with more inductive biases to demonstrate an approximate upper-bound on an approach that discovers skills in an unsupervised fashion. We found that while skill-learning and online skill inference does lead to faster exploration over online-only exploration, it still lags in performance compared to our approach. 

Please see our individual responses for more details on these experiments and answers to individual questions.

Meta-review: This submission is about a method leveraging unlabelled offline data for efficient online model-free RL exploration. The proposed method is very simple and intuitive, yet showed good deal of improvement over baseline methods. Of course, the problem the submission deals with is a timely important problem too. So, the reviewers all suggested to accept the submission. However, there were a few concerns regarding limited evaluation domains and also its applicable problem setup. So, we recommend authors to further add more experiments during the camera ready preparation period if possible.
Decision: Accept (poster)