SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use the offline data to pre-train goal-conditioned policies using GC-IQL on the offline dataset. In the online phase, we perform skill-inference on this set of skills by using RLPD to train a high-level policy that outputs actions corresponding to 2D goal coordinate in the maze for the underlying skill to reach. While this combination leads to increased coverage during early exploration, we found that skill inference using RLPD eventually gets stuck if reward is not found and ceases to explore further, unless additional online RND bonuses are applied. In Figure 17, we plot the performance of this skill inference comparison (5 seeds for skill-inference, 3 for skill pre-training, and ten seeds for the rest) – while inference in the space of skills leads to much faster exploration than with standard
online RND, we found the best variant of skill-discovery to be slightly worse than our method (GC-IQL+Online RLPD + Online RND vs. Ours).
In summary, we found that while skill-learning and online skill inference does lead to faster exploration over online-only exploration, it still lags in performance compared to our approach. We note that our method has several advantages over a skill-learning baseline, such as not needing a separate pre-training phase, and not needing to specify or discover a separate skill set.
Figure 17: Results on AntMaze-Large-Diverse using pre-trained goal-conditioned IQL on offline data. The  $x$ -axis is the number of environment steps  $\times 10^{6}$ . The  $y$ -axis is the normalized return (left) and the coverage percentage (right). The pre-trained IQL agent achieves around  $50\%$  success rate by using the correct goal.
D Visualization Details for Figure 5
Since the prior data on COG contains no full trajectories that complete the task, to generate Figure 5, a successful trajectory on the Grasp from Closed Printer environment was made by combining a successful trajectory opening the drawer and a successful trajectory taking the object out of the drawer from the prior dataset. Afterward, we evaluate the reward model on this successful trajectory for 20 training seeds and normalize each to have zero mean and unit variance.
Below, we provide the same visualization as in Figure 5, but on other successful trajectories combined from the prior data on Grasp from Closed Printer and observe a similar effect.
E Compute Description
We use Tesla V100 GPU for running the experiments. Each AntMaze experiment (300K environment steps) takes around 2 GPU hours. Each Adroit experiment (1M environment steps) takes around 6 GPU hours. Each COG experiment (250K environment steps) takes around 3 GPU hours. To reproduce all the results in the main body of the paper, it takes around 10 (seeds)  $\times$  6 (tasks)  $\times$  6 (methods)  $\times$  2 (AntMaze) + 10 (seeds)  $\times$  3 (tasks)  $\times$  6 (methods)  $\times$  6 (Adroit) + 20 (seeds)  $\times$  3 (tasks)  $\times$  6 (method)  $\times$  3 (COG) = 2880 GPU hours.