SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pixel encoder architecture. For the environments in the COG domain, we use the following pixel encoder architecture, based on the pixel encoder implemented in RLPD's official codebase (located at https://github.com/ikostrikov/rlpd/blob/main/rlpd/networks/encoders/d4pgEncoder.py). Changes in orange are made to the observation size and number of frames stacked to match the dataset from the COG domain. Following the implementation from Ball et al. [2023], between the actor and the critic, the pixel encoder parameters are shared, and only the gradients through the critic loss objective update the encoder. The RND model, reward model, and the termination model each has its own pixel encoder with no parameter sharing.
Parameter	Value
Random Crop Padding	4
Number of Stacked Frames	1
Observation Size	[48, 48, 3]
CNN Features	[32, 64, 128, 256]
CNN Filter Size	3
CNN Padding	Valid
CNN Stride	2
Encoder Output Latent Dimension	50
Table 1: Pixel Encoder Architecture
MLP architecture. In the state-based domains (AntMaze and Sparse Adroit), we use a 3-layer MLP with hidden dimensions of 256. In the COG domain, we use a 2-layer MLP with hidden dimensions of 256, after a pixel encoder (Table 1). The RND network outputs a feature size of  $L = 256$ . The reward network output a single scalar. For the critic network, layer norm [Ba et al., 2016] is being added to all the layers (of the MLP) with the learnable bias and scaling except the last layer.
Action parameterization. In all of our experiments, we follow RLPD's actor design where the actor outputs a mean value  $\mu_{i}$  and a log standard deviation value  $\log \sigma_{i}$  for each action dimension, leading to a diagonal Gaussian distribution,  $N(\pmb {\mu},\mathrm{diag}(\pmb {\sigma}))$ . This distribution is then transformed through the Tanh function such that the range of the action falls in  $(-1,1)$ . The log standard deviation is clipped to be between  $-20$  and 2.
Termination prediction network. While predicting termination is trivial for the COG domain and the Sparse Adroit domain (the episode never terminates before the maximum episode length), predicting termination in AntMaze is just as difficult as predicting the reward (e.g., the episode terminates). To make our implementation generally applicable to any environments with or without terminations, we train a termination prediction network which we briefly mentioned in the beginning of the section. More concretely, the termination prediction network also takes in a state-action pair and outputs a scalar, which predicts the probability of the episode terminating at the current transition through a Sigmoid function. Using the termination prediction network, the termination is also being labeled in the prior unlabeled data. This results in a slightly different TD back-up equation, which
can be easily integrated in the RLPD algorithm:
$$
Q (s, a) \gets \mathrm {U C B R} (s, a) + \gamma (1 - \hat {T} (s, a)) \bar {Q} (s ^ {\prime}, a ^ {\prime}),
$$
where  $\hat{r}(s, a)$  is the labeled reward value and  $\hat{T}(s, a)$  is the labeled termination prediction (with 1 meaning termination with a probability of 1). We left the discussion of the termination prediction network out of the main body of the paper because it is a minor implementation detail. For completeness, we present the more detailed version of Algorithm 1 with the addition of the termination prediction as Algorithm 2.
Training of the UCB networks. The training of the reward prediction model, termination prediction model and the RND model starts after 10000 environment steps for the AntMaze and the Sparse Adroit domains and 5000 environment steps for the COG domain. The RND network takes 1 gradient step on the current transition  $(s_{\mathrm{new}},a_{\mathrm{new}})$  according to the following objective:
$$
\mathcal {L} (\phi) = \frac {1}{L} \| f _ {\phi} (s _ {\mathrm {n e w}}, a _ {\mathrm {n e w}}) - \bar {f} (s _ {\mathrm {n e w}}, a _ {\mathrm {n e w}}) \| _ {2} ^ {2},
$$
where  $L$  is the number of output features of the RND network which is set to be 256. Both the reward model and the termination prediction model take the same number of gradient steps as the actor/critic (1 for COG and 20 for AntMaze and Sparse Adroit). The termination prediction model optimizes the following objective (the transition  $(s, a, r, T)$  is sampled from the online replay buffer  $\mathcal{D}$ ) with  $T \in \{0, 1\}$  being the binary variable that indicates whether the episode terminates at the current transition:
$$
\mathcal {L} (\theta_ {\mathrm {t e r m}}) = - \mathbb {E} _ {(s, a, T) \sim \mathcal {D}} \left[ T \cdot \operatorname {L o g \_ S i g m o i d} (\hat {T} _ {\theta_ {\mathrm {t e r m}}} (s, a)) + (1 - T) \cdot \operatorname {L o g \_ S i g m o i d} (- \hat {T} _ {\theta_ {\mathrm {t e r m}}} (s, a))) \right],
$$
and the reward prediction model optimizes the following objective:
Algorithm 2 EXPLORE (with termination estimates)
1: Input: prior unlabeled data  $\mathcal{D}_{\mathrm{offline}}$   
2: Initialize the UCB estimate of the reward function:  $\mathrm{UCBR}(s,a)$   
3: Initialize an estimate of the termination function:  $\hat{T}(s,a)$   
4: Online replay buffer  $\mathcal{D} \gets \emptyset$   
5: Initialize off-policy RL algorithm with a policy  $\pi$ .  
6: for each environment step do  
7: Collect  $(s,a,s',r)$  using the policy  $\pi$  and add it to the replay buffer  $\mathcal{D}$ .  
8: Update the UCB estimate  $\mathrm{UCBR}(s,a)$  using the new transition  $(s,a,s',r)$ .  
9: Relabel each transition  $(s,a,s')$  in  $\mathcal{D}_{\mathrm{offline}}$  with  $\hat{r} = \mathrm{UCBR}(s,a),\hat{T} = \hat{T}(s,a)$ .  
10: Run off-policy RL update on both  $\mathcal{D}$  and the relabeled  $\mathcal{D}_{\mathrm{offline}}$  to improve the policy  $\pi$ .  
11: end for
Modifications to UCB networks for the COG domain. For the COG domain, we used ICVFs to test the incorporation of representation learning methods. We wanted to experiment with the formulation where the UCB network is learned on top of the encoder outputs of the ICVF representations, which depend only on states/observations  $s$ . That is, for some observation  $s$  and encoder  $\xi(s)$ , we wished to learn a UCB reward model  $r(\xi(s))$ . However, this formulation does not incorporate actions. Therefore, for all our experiments on the COG domain, the RND network, reward model, and termination prediction network all operate on observations only. More concretely, we make the following replacements to our network architecture for the COG domains: for the RND networks we replaced  $f_{\phi}(s,a)$  with  $f_{\phi}(s)$  and  $\bar{f}(s,a)$  with  $\bar{f}(s)$ , for the reward model, we replaced  $r_{\theta}(s,a)$  with  $r_{\theta}(s)$ , and for the termination prediction model, we replaced  $\hat{T}(s,a)$  with  $\hat{T}(s)$ .
A.2 RLPD Hyperparameters
Most of the hyperparameters we used are unchanged from the original RLPD hyperparameters, listed below.
Parameter	Value
Online batch size	128
Offline batch size	128
Discount factor γ	0.99
Optimizer	Adam
Learning rate	3 × 10-4
Critic ensemble size	10
Random critic target subset size	2 for Adroit and COG and 1 for AntMaze
Gradient Steps to Online Data Ratio (UTD)	20
Network Width	256
Initial Entropy Temperature	1.0
Target Entropy	- dim(A)/2
Entropy Backups	False
Start Training	after 5000 environment steps
Table 2: RLPD hyperparameters.
The online and offline batch sizes change when we run baselines that do not use offline data. In that case, the offline batch size becomes 0, and the online batch size becomes 256, to keep the combined batch size the same. The update the data ratio (UTD) was set to 1 in the COG domain. For the other domains, we used the default value of 20. Action repeat is a hyperparameter that determines the number of times an action sampled from the policy is repeated in the environment. It was originally 2 for the pixel-based hyperparameters in RLPD. We set it to 1 to match the prior dataset from the COG domains.
A.3 Baseline Hyperparameters and Tuning Procedure Description
Online. For the online baseline, we don't use any offline data. So the offline batch size is 0 and the online batch size is 256.
Online + RND. For the online with RND baseline, we added an RND bonus on top of the ground truth rewards in the online data. That is, given an online transition  $(s,a,r,s')$ , and RND feature networks  $f_{\phi}(s,a)$ ,  $\bar{f}(s,a)$ , we set
$$
\hat {r} (s, a) \leftarrow r + \frac {1}{L} | | f _ {\phi} (s, a) - \bar {f} (s, a) | | _ {2} ^ {2}
$$
and use the transition  $(s, a, \hat{r}, s')$  in the online update. The RND training is done the same way as in our method where a gradient step is taken on every new transition collected.
Naïve Reward Labeling. The naïve reward labeling baseline is a simple way to incorporate the unlabeled prior data by learning a reward model online and relabeling the rewards of the prior data. For some prior transition  $(s, a, s')$ , and a reward model  $r_{\theta}$  learned from online interactions, we set the reward of this prior transition to be  $\hat{r}(s, a) = r_{\theta}(s, a)$ , giving the relabeled offline transition  $(s, a, \hat{r}, s')$ . Then, we take a batch of  $50\%$  offline data and  $50\%$  online data, and run RLPD as usual. Note that the only difference this baseline has with our method is that  $\hat{r}$  does not have an RND bonus added to it.
Naïve + BC. Aside from performing the reward relabeling naively as described in the previous section, we also regularize the actor to more closely match the behavior in the prior data by adding a behavior cloning loss to the actor loss ( $\theta_{\mathrm{actor}}$  is the parameter of the actor in RLPD):
$$
\mathcal {L} _ {\mathrm {a c t o r , b c}} (\theta_ {\mathrm {a c t o r}}) = \mathcal {L} _ {\mathrm {a c t o r}} (\theta_ {\mathrm {a c t o r}}) - \alpha_ {\mathrm {b c}} \mathbb {E} _ {(s, a) \sim \mathcal {D} _ {\mathrm {o f f l i n e}}} \left[ \log \pi_ {\theta_ {\mathrm {a c t o r}}} (a | s) \right],
$$
where  $\mathcal{L}_{\mathrm{actor}}$  is the original loss function used in RLPD.
To find the best loss coefficient  $\alpha_{\mathrm{bc}}$ , we perform a sweep over  $\{0., 0.01, 0.1\}$  for each domain. The best coefficient is 0.01 for all the domains.
MinR. The MinR baseline is a simpler version of Naive Reward Labeling. For all prior transitions  $(s, a, s')$ , we set the reward  $\hat{r}(s, a) = r_{\min}$ , where  $r_{\min}$  is the minimum reward for the task. For example, for sparse 0-1 rewards,  $r_{\min} = 0$ . Then, we run RLPD with this relabeled offline data.
$\mathbf{BC} + \mathbf{JSRL}$ . This baseline has a few additional hyperparameters. One controls the number of gradient steps to pre-train the behavior cloning guide policy on the unlabeled prior data,  $N_{\mathrm{JSRL}}$ . Each gradient step is taken according to the BC objective ( $\theta_{\mathrm{guide\_policy}}$  is the parameters for the guide policy):
$$
\mathcal {L} _ {\mathrm {J S R L , b c}} \left(\theta_ {\text {g u i d e - p o l i c y}}\right) = - \mathbb {E} _ {(s, a) \sim \mathcal {D} _ {\text {o f f l i n e}}} \left[ \log \pi_ {\theta_ {\text {g u i d e - p o l i c y}}} (a | s) \right].
$$
The other controls the probability of starting a trajectory by first rolling out the guide policy,  $\beta_{\mathrm{JSRL}}$ . Finally, in the course of a trajectory, we switch from the guide policy to the exploration policy with probability  $1 - \gamma$  where  $\gamma = 0.99$  is the discount factor. To find the best hyper-parameters, we perform a sweep over  $\beta_{\mathrm{JSRL}} \in \{0.1, 0.5, 0.9\}$  and  $N_{\mathrm{JSRL}} \in \{5000, 20000, 100000\}$  for both AntMaze and Sparse Adroit. For AntMaze, the best parameter set is  $(\beta_{\mathrm{JSRL}}, N_{\mathrm{JSRL}}) = (0.9, 5000)$ . For Sparse Adroit, the best para meter set is  $(\beta_{\mathrm{JSRL}}, N_{\mathrm{JSRL}}) = (0.5, 100000)$ . For COG domain, none of the sweep runs succeeded. We just report the results of a randomly picked parameter set:  $(\beta_{\mathrm{JSRL}}, N_{\mathrm{JSRL}}) = (0.5, 100000)$ . All the return evaluations are done solely on the exploration policy (without rolling out the guide policy).
Oracle. This baseline assumes that the ground truth rewards for the online task are given in the offline data, and runs RLPD with that data.
ICVF training details. To evaluate reward functions and uncertainty estimation, we run our method on top of a ICVF representation [Ghosh et al., 2023] pre-trained on the offline prior data. We use the open-source implementation from the authors at https://github.com/dibyaghosh/icvf_release, using the COG encoder as the visual backbone and the default hyperparameters otherwise from the open-source implementation. We pre-train the model for  $0.75 \times 10^{5}$  timesteps, and use the final checkpoint as the encoder initialization for both RND and the reward model predictor. Notably, the representation is only ever used as an initialization, and is not frozen or regularized during reward training.
Parameter	Value
Batch size	256
Discount factor γ	0.99
Optimizer	Adam
Learning rate	3 × 10-4
MLP Width	256
MLP layers	2
Table 3:ICVF hyperparameters.
B Domain Details
AntMaze (state) In the D4RL AntMaze Fu et al. [2020] environment, an 8-DoF quadrupedal "ant" agent navigates a maze from a fixed starting location and towards a fixed goal location. The reward is sparse: 0 around the goal and -1 elsewhere. The observation space is 29 dimensions, with the first 2 dimensions indicating the position of the ant within the maze. The action space is 8 dimensional. Trajectories terminate upon reaching the goal. We tested 6 total AntMaze tasks within a total of 3 mazes of increasing size and complexity: umaze, umaze-diverse, medium-play, medium-diverse, large-play, and large-diverse.
The offline data for this environment consists of trajectories with many different start and end positions. The nature of these positions depends on the type of environment. In the play AntMaze environments, the start and end positions are handpicked whereas in the diverse AntMaze environments, they are randomly chosen. Since the actual task contains only a single fixed start and goal, the offline data does not always correspond to the online task. The coverage of the offline data is relatively complete over the entire maze.
Adroit (state) The Adroit environment Nair et al. [2021] we used consisted of a 24-DoF hand robot agent and 3 dexterous manipulation tasks: pen-spinning, door-opening, and ball relocation. The data distribution for the Adroit environments is relatively narrow. The prior data for these environments consists of expert trajectories from human demonstrations, as well as many behavior-cloned trajectories on the expert data. As the prior data here is closer to the target task distribution
Figure 8: Additional visualizations on offline dataset with four different kinds of corruptions. Top: visualization of the coverage of the transitions in the offline data. Bottom: visualization of the direction of the transitions in the offline data. Left: All, Middle Left: Orthogonal Transitions, Middle Right: Insufficient Coverage, Right:  $1\%$  Data.
compared to other environments, these experiments in Adroit allow us to compare the performance of our method more critically against our baselines that are less exploration focused.
COG (vision) The environments and datasets from COG come from the repo: https://github.com/avisingh599/cog. In the COG environments Singh et al. [2020b], the actor is a simulated WidowX robot arm. The observation space is image-based with dimensions (48, 48, 3). The rewards are sparse, with  $+1$  reward when the task is completed, and 0 reward otherwise. The horizon is infinite, but for exploration and evaluation, we set a max path length to end the episode.
The first environment we call Pick and Place. For this environment, the arm is placed in front of a small object and tray. The task is to pick up / grasp the object and place it in the tray, where success occurs when the object is in the tray. The object is initialized at different locations on the ground. The maximum episode length is 40.
The offline dataset for the Pick and Place consists of trajectories of grasping attempts and separate trajectories of placing attempts. The data for these environments comes from a scripted policy that achieves
Figure 9: Visualizations of data used in the COG tasks. The left block contains the first stage of the task, and the right block contains the second stage of the task. The first row is for the Pick and Place task, the second row is picking out a ball from a closed drawer, the Grasp from ClosedDrawer task, and the third row is picking out a ball from a blocked drawer, the Grasp from LinkedDrawer 1 task.
roughly  $40\%$  success at grasping the object and roughly  $90\%$  success at placing the object into the tray. There are 100 trajectories of grasping attempts and 50 trajectories of placing attempts. There may be overlaps in states in the grasping and placing data, but there is no trajectory that completes the entire pick and place task. For our RLPD baseline, where the rewards from the prior dataset are considered, we set a reward of  $+1$  when the object is successfully placed in the tray, and 0 for all other transitions.
We ran experiments on two drawer environments from COG. The main interaction in these environments is between the WidowX robot arm and a two tiered drawer. The first setup we call Grasp from Closed Printer. In this setup, the bottom drawer is closed, and an object is inside the bottom drawer. The task is to open the bottom drawer and pick out the object. The second setup we call Grasp from Blocked Printer 1. In this setup, the bottom drawer is closed and blocked by an open upper drawer. The task is to close the upper drawer, open the bottom drawer, and pick out the object. For both drawer environments, a success occurs when the object is taken out of the
Figure 10: The performance on 3 COG tasks with a larger offline dataset. BC+JSRL is able to match the oracle performance. In the main body of the paper (Figure 3), we are using a dataset that is roughly  $1\%$  in size.
bottom drawer. The maximum episode length is 50 for Grasp from Closed Printer, and 80 for Grasp from Blocked Printer 1.
The offline dataset we used for the drawer tasks consists of trajectories of drawer opening attempts and separate trajectories of taking the object out of an open drawer. For example, in the Grasp from Closed Printer task, the drawer opening attempts consist of trajectories of opening the bottom drawer. Meanwhile, for the Grasp from Blocked Printer 1 task, the drawer opening attempts consist of trajectories of closing the top drawer and opening the bottom drawer. For both tasks, the trajectories of taking the object out of an open drawer are the same. Similar to Pick and Place, these trajectories were generated using a randomized scripted policy as described in Singh et al. [2020b] that has about a  $40 - 50\%$  success rate at opening the drawer and about a  $70\%$  success rate at taking the object out of the drawer. Like the Pick and Place task, there is no trajectory that completes the entire task, but there may be overlaps in states. For our RLPD baseline, we set a reward of  $+1$  when the object has been successfully taken out of the bottom drawer, and 0 for all other transitions. There are 25 trajectories of drawer opening attempts and 25 trajectories of object-taking attempts for the two drawer tasks.
Initially, we experimented with a larger offline dataset with 100 times more trajectories for these three tasks and we have found that the  $\mathbf{BC} + \mathbf{JSRL}$  baseline to closely match the oracle performance and our method did not really an edge over that baseline (see Figure 10). This is perhaps to be expected as  $\mathbf{BC} + \mathbf{JSRL}$  puts a strong prior on following the offline data action by behavior cloning, and when offline data are good demonstration data. On the other hand, our method does not assume that the offline dataset is of good quality, and does not try to closely follow the actions. Consequently, our method is able to perform well more robustly across different settings compared to  $\mathbf{BC} + \mathbf{JSRL}$  as shown in our main results (Figure 3).
C Additional Results
C.1 Full AntMaze results.
We include the complete set of AntMaze results in this section. Figure 11 shows the normalized return, which is the percentage success rate of the ant reaching the desired goal. Figure 12 shows the coverage metric, which is an estimate of how much percentage of the free space that the ant has explored in the maze.
Figure 11: The success rate for the 6 AntMaze tasks. Every method except Oracle has no access to the reward function. Our method consistently performs well and nearly matches the oracle performance.
Figure 12: State coverage for the 6 AntMaze tasks, estimated by counting the number of square regions that the ant has visited, normalized by the total number of square regions that can be visited.
C.2 Full Adroit Results
We include the complete set of Adroit results in this section. Figure 13 shows the normalized return for each method on each of the three sparse Adroit tasks. Figure 14 studies the effect of periodic resetting the reward and the RND networks on the hardest task, relocate.
Figure 13: The normalized return for the 3 sparse Adroit tasks. Every method except Oracle has no access to the reward function. Naive reward modeling without optimistic relabeling is sufficient for all Adroit tasks. Periodically resets (every 1000 environment steps) are used for the relocate task. See Figure 14 below for the reset ablation study.
Figure 14: Periodically resetting the reward model is crucial for success in relocate. We experiment with different reset intervals and have found that resetting the reward model after every  $20\mathrm{K}$  gradient steps (1K environment steps) to work well. We hypothesize that the periodic resetting procedure might have alleviated the overfitting issue of the reward function, allowing it to continuously adapt to incoming transitions.
C.3 Full COG results
Figure 15: The performance on 3 COG tasks.
C.4 Implicit-Q-learning (IQL) as the backbone RL algorithm.
Conceptually, our proposed method, ExPLORe, is not specific to the selected RL method, and can be combined with any RL methods that take in both offline and online transitions. Other than RLPD, we experimented with another RL algorithm, IQL [Kostrikov et al., 2021], in the offline to online fine-tuning setting where we run IQL on an online replay buffer prepended with the reward-relabelled offline data. In our experiments (Figure 16), we have found the use of the RND bonus in the offline data to be essential for accelerating learning in both medium datasets (without RND bonus, none of the runs succeed).
Figure 16: Results on 4 AntMaze datasets. The oracle offline IQL numbers are directly taken from the original paper. It uses the offline dataset with ground truth reward. The other curves do not have access to the ground truth reward in the offline dataset. 10 seeds are used for each curve.
C.5 Pre-trained goal-conditioned (GC)-IQL skills.
To get a sense of how well skill-discovery may perform, we perform an additional empirical study in antimaze-large-diverse-v2 with a set of unsupervised, pre-trained skills. In particular, we specify the skills to be a set of pre-trained goal-conditioned (GC) policies from the offline data where the 2-D goal coordinate specifies the skill. We chose this more manually defined skill set with more inductive biases to demonstrate an approximate upper-bound on an approach that discovers skills in an unsupervised fashion.