OGBench: Benchmarking Offline Goal-Conditioned RL

## Abstract

Abstract Offline goal-conditioned reinforcement learning (GCRL) is a major problem in reinforcement learning (RL)
because it provides a simple, unsupervised, and domain-agnostic way to acquire diverse behaviors and representations from unlabeled data without rewards.
Despite the importance of this setting, we lack a standard benchmark that can systematically evaluate the capabilities of offline GCRL algorithms.
In this work, we propose OGBench, a new, high-quality benchmark for algorithms research in offline goal-conditioned RL.
OGBench consists of 8 8 8 8 types of environments, 85 85 85 85 datasets,
and reference implementations of 6 6 6 6 representative offline GCRL algorithms.
We have designed these challenging and realistic environments and datasets to directly probe different capabilities of algorithms,
such as stitching, long-horizon reasoning, and the ability to handle high-dimensional inputs and stochasticity.
While representative algorithms may rank similarly on prior benchmarks,
our experiments reveal stark strengths and weaknesses in these different capabilities,
providing a strong foundation for building new algorithms. Project page: https://seohong.me/projects/ogbench Repository: https://github.com/seohongpark/ogbench

## 1 Motivation

Why offline goal-conditioned reinforcement learning (RL)?
The enduring trend in modern machine learning is to simplify domain-specific assumptions and scale up the data.
In computer vision and natural language processing, the strongest general-purpose models are
trained via simple *unsupervised* objectives on raw, unlabeled data,
such as next-token prediction, contrastive learning, and masked auto-encoding.
What analogous paradigm could enable data-driven unsupervised learning for reinforcement learning?
Ideally, such a framework should be able to produce from data
a generalist policy that can be directly queried or adapted
to solve a variety of downstream tasks,
much like how generative language models trained via next-token prediction can be easily adapted to everyday tasks.

We posit that a natural analogy to data-driven unsupervised learning in RL is offline goal-conditioned RL (GCRL).
The objective of offline goal-conditioned RL is fully unsupervised, remarkably simple, and requires no domain knowledge:
it merely aims to learn to reach any state from any other state in the dataset in the fewest number of steps.
However, mastering this simple objective is exceptionally difficult:
the agent needs to not only acquire diverse skills to efficiently navigate the state space,
but also have a deep, complete understanding of the underlying world and dataset.
As a result, offline goal-conditioned RL yields a highly capable general-purpose multi-task policy
as well as rich, useful representations
that can be adapted to solve a variety of downstream tasks (Ghosh et al., ; Kim et al., ).
Indeed, for such simplicity and generality, interest in (offline) goal-conditioned RL has recently surged
to the extent that even a standalone workshop on goal-conditioned RL was held
at a machine learning conference.(^1^11[https://goal-conditioned-rl.github.io/2023/](https://goal-conditioned-rl.github.io/2023/))

Why a new benchmark?
Despite the importance of and increasing interest in offline goal-conditioned RL,
we currently lack a standard benchmark that can systematically assess the capabilities of offline GCRL algorithms,
such as the ability to stitch, perform long-horizon reasoning, and handle stochasticity.
Prior works in offline goal-conditioned RL (Eysenbach et al., ; Ma et al., ; Park et al., ; Myers et al., ) have mainly used
either existing datasets for standard offline RL tasks without modification (*e.g.*, D4RL (Fu et al., )),
relatively simple goal-conditioned tasks (*e.g.*, Fetch (Plappert et al., )),
or tasks tailored to demonstrate the individual abilities of the proposed methods.
This often results in limited evaluation.
Prior works often evaluate their multi-task policies only on a single task
(when using datasets not originally designed for offline GCRL),
or learn relatively simple behaviors. While there exist some prior tasks tailored to evaluate *individual* properties of offline GCRL such as stitching or generalization (Yang et al., ; Ghugare et al., ),
we lack a comprehensive, standardized benchmark
that exhaustively assesses various properties of offline GCRL algorithms with diverse, challenging tasks.

Therefore, we introduce a benchmark named Offline Goal-Conditioned RL Benchmark (OGBench) in this work (Figure 1).
The primary goals of this benchmark are
to facilitate *algorithms research* in offline goal-conditioned RL
and to provide a set of complex tasks that can unlock the potential of offline GCRL.
Our benchmark introduces $8$ types of environments and $85$ datasets across robotic locomotion, robotic manipulation, and drawing,
and provides well-tuned reference implementations of $6$ representative offline GCRL methods.
These datasets, tasks, and implementations are carefully designed.
Tasks are designed in a way that complex behaviors can naturally emerge when successfully solved,
and that they pose diverse algorithmic challenges in offline GCRL, such as goal stitching,
stochastic control, long-horizon reasoning, and more.
Dataset and task difficulties are carefully adjusted to highlight stark contrasts between algorithms across multiple criteria.
The entire benchmark is designed to minimize unnecessary computational overhead and maximize usability
such that any researcher can easily iterate and evaluate new ideas.
We believe OGBench serves as a solid foundation for
developing ideal algorithms for goal-conditioned and unsupervised RL from data.

## 2 Problem Setting

The offline goal-conditioned RL
problem is defined by a controlled Markov process ${\mathcal{M}}=({\mathcal{S}},{\mathcal{A}},\mu,p)$ (*i.e.*, a Markov decision process (MDP) without rewards)
and an unlabeled dataset ${\mathcal{D}}$,
where ${\mathcal{S}}$ denotes the state space, ${\mathcal{A}}$ denotes the action space,
$\mu({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s})\in\Delta({\mathcal{S}})$(^2^22We denote placeholder variables in gray throughout the paper.) denotes the initial state distribution,
and $p({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s^{\prime}}\mid{\color[rgb]
{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}s},{\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}a})
\colon{\mathcal{S}}\times{\mathcal{A}}\to\Delta({\mathcal{S}})$ denotes the transition dynamics function.
$\Delta({\mathcal{X}})$ denotes the set of probability distributions defined on a set ${\mathcal{X}}$.
The dataset ${\mathcal{D}}=\{\tau^{(n)}\}_{n\in\{1,2,\dots,N\}}$ consists of unlabeled trajectories
$\tau^{(n)}=(s^{(n)}_{0},a^{(n)}_{0},s^{(n)}_{1},\dots,s^{(n)}_{T_{n}})$.

The objective of offline goal-conditioned RL is to learn to reach any state from any other state in the minimum number of time steps.
Formally,
offline GCRL aims to learn a goal-conditioned policy $\pi({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}a}\mid{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}s},{\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})
\colon{\mathcal{S}}\times{\mathcal{S}}\to\Delta({\mathcal{A}})$ that maximizes the objective
$\mathbb{E}_{\tau\sim p({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor
[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}\tau}\mid g)}[\sum_
{t=0}^{T}\gamma^{t}\delta_{g}(s_{t})]$ for all $g\in{\mathcal{S}}$,
where $T\in{\mathbb{N}}$ denotes the episode horizon, $\gamma\in(0,1)$ denotes the discount factor,
$p({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}\tau}\mid g)$ denotes the distribution given by $p(\tau\mid g)=\mu(s_{0})\pi(a_{0}\mid s_{0},g)p(s_{1}\mid s_{0},a_{0})\cdots p
(s_{T}\mid s_{T-1},a_{T-1})$,
and $\delta_{g}(\cdot)$ denotes the Dirac delta “function”(^3^33In a discrete MDP, $\delta_{g}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s})$ is equal to an indicator function $\mathbf{1}_{\{g\}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[ named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s})$. In a continuous MDP, it is technically not well-defined in its current form. It can be made precise using measure-theoretic notation or the distribution theory, but we choose to avoid them for simplicity.)
at $g$.
Note that we use the entire state space as the goal space (*i.e.*, a goal is simply a full state, not part of the state like only the $x$-$y$ position of the agent).
This choice makes the objective fully unsupervised,
making it suitable for domain-agnostic training from unlabeled data.
Our goal in this paper is to propose a new benchmark in offline GCRL.
That is, formally speaking, to specify the dynamics and dataset for each task we introduce.

**Table 1: Properties of benchmark tasks. We summarize the properties of benchmark tasks commonly used in prior works (above the line) and our OGBench tasks (below the line).**
| Benchmark Task | Type^1 | Longest Task^2 | # Subtasks^3 | Test Stitching?^4 | Have Stoch. Tasks?^5 | Support Pixels?^6 | Multi-Goal?^7 | Dependency^8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| D4RL AntMaze | Loco. | $\approx 400$ | - |  |  |  |  | MuJoCo |
| D4RL Kitchen | Manip. | $\approx 250$ | $4$ |  |  |  |  | MuJoCo |
| Roboverse | Manip. | $\approx 100$ | $1$-$2$ |  |  |  |  | PyBullet |
| Fetch | Manip. | $\approx 50$ | $1$ |  |  |  |  | MuJoCo |
| PointMaze (ours) | Loco. | $\approx 600$ | - |  |  |  |  | MuJoCo |
| AntMaze (ours) | Loco. | $\approx 1000$ | - |  |  |  |  | MuJoCo |
| HumanoidMaze (ours) | Loco. | $\approx 3000$ | - |  |  |  |  | MuJoCo |
| AntSoccer (ours) | Loco. | $\approx 1000$ | - |  |  |  |  | MuJoCo |
| Cube (ours) | Manip. | $\approx 400$ | $1$-$4$ |  |  |  |  | MuJoCo |
| Scene (ours) | Manip. | $\approx 400$ | $2$-$8$ |  |  |  |  | MuJoCo |
| Puzzle (ours) | Manip. | $\approx 800$ | $2$-$24$ |  |  |  |  | MuJoCo |
| Powderworld (ours) | Draw. | $\approx 100$ | - |  |  |  |  | NumPy |

## 3 How Have Prior Works Benchmarked Offline GCRL?

While many excellent offline GCRL algorithms have been proposed so far,
the community currently lacks a standardized way to evaluate their performance,
unlike other fields in RL (Brockman et al., ; Tassa et al., ; Fu et al., ; Terry et al., ).
Tasks used by prior works in offline GCRL are often limited for providing a proper evaluation for various reasons.
For example,
many works directly use tasks in existing offline RL benchmarks (not necessarily designed for offline *goal-conditioned* RL),
such as D4RL AntMaze and Kitchen (Fu et al., ).
However, since the tasks are designed for single-task offline RL,
they often evaluate their multi-task policies only on the single, original goal when using these tasks (Eysenbach et al., ; Park et al., ; Zheng et al., ; Myers et al., ),
which results in limited evaluation.
Some employ online GCRL tasks provided by Plappert et al. () (*e.g.*, Fetch tasks) with policy-collected datasets (Ma et al., ; Yang et al., ),
but these tasks are mostly atomic (*e.g.*, single pick-and-place) and do not sufficiently address various challenges in offline GCRL,
such as long-horizon reasoning and goal stitching.
While Roboverse (Fang et al., ; Zheng et al., ) provides pixel-based manipulation tasks,
the tasks are still relatively atomic
and it requires installing multiple, fragmented dependencies,
making it less approachable for researchers.
Some prior works construct bespoke tasks and datasets to individually study
the important and specific features of their algorithms
(*e.g.*, stitching (Ghugare et al., ) and stochasticity (Myers et al., )),
but it is often not entirely clear how algorithms compare to one another
or if the new capabilities of new methods (*e.g.*, stitching)
come at the loss of other capabilities.
These limitations of previous evaluation tasks have motivated us to create a new benchmark.
In this work, we introduce a set of diverse tasks that cover various challenges in offline GCRL,
enabling a much more thorough and multi-faceted evaluation than previous tasks.
We summarize the properties of the previous tasks and our new tasks in Table 1
and refer to Appendix B for further discussion on related work.

## 4 Overview of OGBench

We now introduce our benchmark, Offline Goal-Conditioned RL Benchmark (OGBench).
The primary goal of this benchmark is to provide tasks and datasets
to unlock the full potential of offline goal-conditioned RL.
To this end, we pose diverse challenges in offline GCRL throughout the benchmark
in such a way that researchers can easily test and iterate on algorithmic ideas,
and that complex, intriguing behaviors can naturally emerge when successful.
OGBench consists of $8$ types of environments,
$85$ datasets, and reference implementations of $6$ representative offline GCRL algorithms.
In the following sections, we first describe the challenges in offline GCRL (Section 5)
and then outline our core design philosophies (Section 6).
We next introduce the tasks and datasets (Section 7)
and present the benchmarking results of the current algorithms (Section 8).

## 5 Challenges in Offline Goal-Conditioned RL

Offline goal-conditioned RL, despite its simplicity, is a challenging problem.
Here, we discuss the major challenges in offline GCRL, which will motivate the design choices in our benchmark tasks.

(1) Learning from suboptimal, unstructured data:
An ideal offline GCRL algorithm should be able to learn an effective multi-task policy from diverse and suboptimal data.
This is especially important, considering that suboptimal (yet diverse) data is much cheaper to collect than curated, expert datasets (Lynch et al., ),
and that the very use of large, diverse, unstructured data is one of the foundations for the success of modern machine learning.
Reflecting this challenge, we provide datasets with high diversity and varying suboptimality in this benchmark
to challenge the capabilities of offline GCRL algorithms.

(2) Goal stitching:
Another important challenge is
to stitch the initial and final states of different trajectories to learn more diverse behaviors.
We call this “goal stitching.”
Goal stitching is different from “regular” stitching in offline RL,
which applies only when the dataset is suboptimal.
Unlike regular stitching, goal stitching applies
*even when the dataset only consists of optimal, expert trajectories*,
because we can often acquire more diverse goal-reaching behaviors by stitching multiple trajectories together, regardless of their optimality.
For instance, an agent can stitch two atomic pick-and-place behaviors
to sequentially move two objects in a single episode,
even when the dataset does not contain any double pick-and-place behaviors.
Goal stitching is crucial to learning diverse behaviors
in many real-world applications with high behavioral diversity and large state spaces.
In our benchmark, we introduce many tasks
with large state spaces to assess the ability to perform goal stitching.

(3) Long-horizon reasoning:
Long-horizon reasoning refers to the capability of
navigating from a starting state to a goal state that is many steps apart.
This challenge is important in many real-world tasks like autonomous driving or assembly,
which may require several hours of continuous control or achieving dozens of subtasks.
To substantially challenge the long-horizon reasoning ability of offline GCRL methods,
we introduce tasks that are more than $5$ times longer than previously used ones
in terms of both the episode length and the number of subtasks (Table 1).

(4) Handling stochasticity:
Another prominent challenge in offline GCRL is the ability to deal with stochastic environments.
Correctly handling stochasticity is very important in practice, because virtually any real-world environment is stochastic due to partial observability.
Yet, many works in offline GCRL assume *deterministic* dynamics to exploit the metric structure of temporal distances (Wang et al., )
or to enable hierarchical control (Park et al., ), at the cost of being optimistically biased in stochastic environments (Wang et al., ; Park et al., ).
Correctly handling environment stochasticity while fully exploiting
the recursive subgoal structure of GCRL remains an open problem.
Since most previous tasks used to evaluate offline GCRL methods have deterministic dynamics (Table 1),
we introduce several challenging tasks with stochastic dynamics in this benchmark.

## 6 Design Principles

Next, we discuss the design principles underlying our benchmark tasks.
The tasks are intended to exercise the major challenges in offline GCRL in the previous section,
while providing a set of high-quality tasks that provide
not only a toolkit for algorithms research and evaluation,
but also a platform for vividly illustrating the capabilities of offline GCRL with compelling and complex domains.

(1) Realistic and exciting tasks:
The tasks should be realistic yet exciting enough
while posing diverse challenges in offline goal-conditioned RL.
Imagine a robot arm watching random movements of a puzzle and then solving it zero-shot at test time,
a humanoid robot navigating through a labyrinth,
or an agent painting cool pictures using different types of brushes.
In this benchmark, we design new tasks such that these kinds of exciting behaviors can naturally emerge
when an RL agent properly stitches different trajectory segments together
(up to $24$; see Table 1) or successfully generalizes.
At the same time, we make sure our tasks exhaustively cover major challenges in offline goal-conditioned RL,
such as long-horizon reasoning, stochastic control, and combinatorial generalization via goal stitching (Section 5).

(2) Appropriate difficulty:
The tasks and datasets should have appropriate levels of difficulty to properly evaluate different algorithms.
In other words, they should be of high quality *for benchmarking*.
No matter how intricate or compelling a task is, it will fail to provide a useful signal for benchmarking if it is too easy, too hard,
unsolvable from the given dataset,
or does not clearly distinguish between more or less effective methods.
In this work, we carefully curate and adjust the difficulty of each task and dataset
such that they can provide effective guidance for algorithms research.
For some tasks, we provide multiple versions with varying difficulty, all the way up to tasks that are difficult to solve with current methods,
so that the same benchmark can continuously be used to develop new methods even in the future.
Our rule of thumb is to have, for each type of tasks, at least one task
where the current state-of-the-art offline GCRL method achieves a success rate of $20$-$30\%$.
This ensures that the task is solvable from the dataset while leaving significant room for improvement.

(3) Controllable datasets:
The benchmark should provide tools to control and adjust datasets for scientific research and ablation studies.
Verifying the effectiveness of algorithms in real-world problems is surely important.
However, for algorithms research, it is equally, if not more, important to provide analysis tools
to enable a rigorous, scientific understanding of challenges and algorithms.
Hence, instead of employing fixed, human-collected data,
which does not always provide clear benchmarking signals for algorithms research,
we choose to focus on simulated environments and synthetic datasets that we have full control of,
and provide tools to reproduce and adjust them easily.
We note that many algorithmic ideas in RL that have made a major impact,
such as DQN (Mnih et al., ), PPO (Schulman et al., ), and CQL (Kumar et al., ),
were originally developed in simulated environments.
Even in natural language processing, studies on synthetic, controlled datasets
have revealed the mechanisms and limitations of language models with scientific evidence (Allen-Zhu & Li, ),
and these insights have transferred to real scenarios (Allen-Zhu & Li, ).
We demonstrate how such controllability of datasets reveals challenges and design principles in offline GCRL in Section 8.2.

(4) Minimal compute requirements:
The tasks should be designed to minimize *unnecessary* computational overhead so that
as many researchers as possible, including those from small labs and underprivileged backgrounds,
can quickly iterate on their new *algorithmic* ideas.
This does not mean that the tasks should be easy (indeed, some of our tasks are very challenging (but solvable)!);
it means the benchmark should focus mainly on algorithmic challenges (*e.g.*, not requiring high-resolution image processing).
In our benchmark, we provide both state- and pixel-based observations whenever possible,
and minimize the size of image observations (up to $64\times 64\times 3$) to reduce the computational burden.
Moreover, we carefully adjust colors, transparency, and lighting for image-based tasks
to enable pixel-based control without high-resolution images or multiple views.

(5) High code quality:
The reference implementations should be very clean and well-tuned so that researchers can directly use our implementations to build their ideas, and the benchmark should be very easy to set up.
Our benchmark environments only depend on MuJoCo (Todorov et al., ) and do not require any other dependencies (Table 1).
For reference implementations, we minimize the number of file dependencies for each algorithm,
largely following the spirit of the single-file implementations of recent RL libraries (Huang et al., ; Tarasov et al., ),
while maintaining a minimal amount of additional modularity.
We also extensively test and tune different design choices and hyperparameters for each offline GCRL algorithm,
to the degree that several methods achieve even better performances than their original performances reported on previous benchmarks (Table 3).

## 7 Environments, Tasks, and Datasets

We now introduce the environments, tasks, and datasets in our benchmark.
They can be broadly categorized into three groups: locomotion, manipulation, and drawing.
Most tasks support *both* state- and pixel-based observations,
and each task accompanies five pre-defined state-goal pairs for evaluation.
We also provide a separate validation dataset for each dataset.
Videos are available at [https://seohong.me/projects/ogbench](https://seohong.me/projects/ogbench).

### 7.1 Locomotion Tasks

We provide four types of locomotion environments,
PointMaze, AntMaze, HumanoidMaze, and AntSoccer,
with diverse variants.
These environments are designed to test the agent’s long-horizon and hierarchical reasoning abilities.
They are based on the MuJoCo simulator (Todorov et al., ).

PointMaze (pointmaze), AntMaze (antmaze), and HumanoidMaze (humanoidmaze).
Maze navigation is one of the most widely used tasks for benchmarking offline GCRL algorithms.
We provide three different types of maze navigation tasks:
PointMaze, which involves controlling a $2$-D point mass,
AntMaze, which involves controlling a quadrupedal Ant agent with $8$ degrees of freedom (DoF),
and HumanoidMaze, which involves controlling a much more complex $21$-DoF Humanoid agent.
The aim of these tasks is to control the agent
to reach a goal location in the given maze. The agent must learn both the high-level maze navigation
and low-level locomotion skills that involve high-dimensional control,
purely from diverse offline trajectories.

In our benchmark, we substantially extend the original PointMaze and AntMaze tasks proposed by D4RL (Fu et al., ).
Unlike the original D4RL tasks, which only support Point and Ant agents and do not challenge stitching(^4^44See Ghugare et al. () for the details about this point.),
stochasticity, or pixel-based control,
we support Humanoid control, pixel-based observations, and multi-goal evaluation,
while providing more challenging and diverse types of mazes and datasets.
The supported maze types are as follows:

- •
medium: This is the smallest maze, with the same layout as the original medium maze in D4RL.
- •
large: This is a larger maze, with the same layout as the original large maze in D4RL.
- •
giant: This is the largest maze, twice the size of large.
It has the same size as the previous antmaze-ultra maze by Jiang et al. (),
but its layout is more challenging and contains longer paths that require up to $3000$ environment steps (in the case of Humanoid).
This maze is designed to substantially challenge the long-horizon reasoning capability of the agent.
- •
teleport:
This maze is specially designed to challenge the agent’s ability to handle environment stochasticity.
It has the same size as large, but contains multiple *stochastic* teleporters.
If the agent enters a black hole, it is immediately sent to a randomly chosen white hole.
However, since one of the three white holes is a dead-end, there is always a risk in taking a teleporter.
The agent therefore must learn to avoid the black holes, without being optimistically biased by “lucky” outcomes.

On these mazes, we collect datasets with a low-level directional policy trained via SAC (Haarnoja et al., )
and a high-level waypoint controller.
For each maze type,
we provide three types of datasets that pose different kinds of challenges
(the figures below show example trajectories in these datasets):

- •
navigate:
This is the standard dataset, collected by a noisy expert policy that navigates the maze by repeatedly reaching randomly sampled goals.
- •
stitch:
This dataset is designed to challenge the agent’s stitching ability.
It consists of short goal-reaching trajectories,
where the length of each trajectory is at most $4$ cell units.
Hence, the agent must be able to stitch multiple trajectories (up to $8$) together to complete the tasks.
- •
explore:
This dataset is designed to test whether the agent can learn navigation skills from extremely low-quality (yet high-coverage) data.
It consists of random exploratory trajectories,
collected by commanding the low-level policy with random directions re-sampled every $10$ steps,
with a large amount of action noise.

We provide two types of observation modalities:

- •
States: This is the default setting,
where the agent has access to the full low-dimensional state representation,
including its current $x$-$y$ position.
- •
Pixels (visual): This requires pure pixel-based control, where the agent only receives
$64\times 64\times 3$ RGB images rendered from a third-person camera viewpoint.
Following Park et al. (), we color the floor to enable the agent to infer its location from the images,
obviating the need for a potentially expensive memory component.
However, unlike Park et al. (), which additionally provides proprioceptive information,
we do not provide any low-dimensional state information like joint angles;
the agent must learn *purely* from image observations.

AntSoccer (antsoccer).
To provide a more diverse type of locomotion task beyond simple maze navigation, we introduce a new locomotion task, AntSoccer.
This task involves controlling an Ant agent to dribble a soccer ball.
It is inspired by the quadruped-fetch task in the DeepMind Control suite (Tassa et al., ).
AntSoccer is significantly harder than AntMaze
because the agent must also carefully control the ball while navigating the environment.
We provide two maze types: arena, which is an open space without walls,
and medium, which is the same maze as the medium one in AntMaze.
For datasets, we provide navigate and stitch.
The navigate datasets consist of trajectories where the agent repeatedly approaches the ball and dribbles it to random locations.
The stitch datasets consist of two different types of trajectories,
maze navigation without the ball and dribbling with the ball near the agent,
so that stitching is required to complete the full task.
AntSoccer only supports state-based observations.

### 7.2 Manipulation Tasks

We provide a manipulation suite
with three types of robotic manipulation tasks,
Cube, Scene, and Puzzle,
with diverse difficulties and complexities.
They are designed to test the agent’s
object manipulation, sequential generalization, and combinatorial generalization abilities.
These environments are based on the MuJoCo simulator (Todorov et al., )
and a $6$-DoF UR5e robot arm (Zakka et al., ).
On these tasks, we provide “play”-style datasets (play) (Lynch et al., ) collected by
non-Markovian expert policies with temporally correlated noise.
To support more diverse types of research (*e.g.*, dataset ablation studies),
we additionally provide more noisy datasets (noisy)
collected by Markovian expert policies with uncorrelated Gaussian noise, which we describe in Appendix A.

For all manipulation tasks, we support *both* state-based observations and pixel-based observations with $64\times 64\times 3$ RGB camera images.
For pixel observations, we adjust colors and make the arm transparent to ensure full observability.
The transparent arm in the figure might *appear* challenging,
but the colors and transparency are carefully adjusted to minimize difficulties in visual perception,
to the extent that some methods achieve even better performance with pixels (see Table 2).

Cube (cube).
This task involves pick-and-place manipulation of cube blocks,
whose goal is to control a robot arm to arrange cubes into designated configurations.
We provide four variants, single, double, triple, and quadruple,
with different numbers ($1$-$4$) of cubes.
We provide “play”-style datasets
collected by a scripted policy that repeatedly picks a random block
and places it in other random locations or on another block.
At test time, the agent is given goal configurations that require
moving, stacking, swapping, or permuting cube blocks.
Hence, the agent must learn not only generalizable multi-object pick-and-place behaviors
from unstructured random trajectories in the dataset,
but also long-term plans to achieve the tasks
(*e.g.*, permuting blocks requires non-trivial sequential and logical reasoning).

Scene (scene).
This task is designed to challenge the sequential, long-horizon reasoning capabilities of the agent.
It involves manipulating diverse everyday objects,
such as a cube block, a window, a drawer, and two button locks,
where pressing a button toggles the lock status of the corresponding object (the drawer or window).
We provide “play”-style datasets
collected by scripted policies that randomly interact with these objects.
At test time, the agent is commanded to arrange the objects into a desired configuration.
Evaluation tasks require a significant degree of sequential reasoning:
for instance, some tasks require unlocking the drawer, opening it, putting the cube in the drawer, and closing it again (see the figure above),
and the longest task involves eight atomic behaviors.
Hence, the agent must be able to plan and sequentially combine learned manipulation skills.

Puzzle (puzzle).
This task is designed to test the combinatorial generalization abilities of the agent.
It requires solving the “Lights Out” puzzle(^5^55[https://en.wikipedia.org/wiki/Lights_Out_(game)](https://en.wikipedia.org/wiki/Lights_Out_(game)))
with a robot arm.
The puzzle consists of a two-dimensional array of buttons (*e.g.*, a $4\times 6$ grid),
where pressing a button toggles the colors of the pressed button and the buttons adjacent to it (typically four, except on the edges and corners; see [videos](https://seohong.me/projects/ogbench)).
The goal is to achieve a desired configuration of colors (*e.g.*, turning all the buttons blue)
by pressing an appropriate combination of buttons.
Since these buttons are implemented in the MuJoCo simulator, the agent must control a robot arm to *physically* press the buttons.
We provide four levels of difficulty, 3x3, 4x4, 4x5, and 4x6,
with different grid sizes.
The datasets are collected by a scripted policy that randomly presses buttons in arbitrary sequences.
Given the enormous state space of this task (with up to $2^{24}=16,777,216$ distinct button states),
the agent must achieve *combinatorial* generalization
while mastering low-level continuous control.
Some evaluation task in the hardest puzzle requires pressing more than $20$ buttons,
which also substantially challenges the long-horizon reasoning capabilities of the agent.
This might sound very challenging (and it is!), but we provide different levels and enough data
to ensure that they provide meaningful research signals and are solvable (see the results in Section 8).

### 7.3 Drawing Tasks

Powderworld (powderworld).
To provide more diverse tasks beyond robotic locomotion or manipulation,
we introduce a *drawing* task, Powderworld (Frans & Isola, )(^6^66Play here: [https://kvfrans.com/powder/](https://kvfrans.com/powder/)),
which presents unique challenges with extremely high intrinsic dimensionality.
The goal of Powderworld is to draw a target picture on a $32\times 32$ grid using different types of “powder” brushes,
where each powder brush has a distinct physical property corresponding to a unique element.
For example, the “sand” brush falls down and piles up,
and the “fire” brush burns combustible elements like “plant.”
We provide three versions of tasks, easy, medium, and hard,
with different numbers of available elements ($2$, $5$, and $8$ elements, respectively).
The datasets are collected by a random policy that keeps drawing arbitrary shapes with random brushes.
This Powderworld task poses unique challenges that are distinct from the other tasks in the benchmark.
First, the agent must deal with the high *intrinsic* dimensionality of the states,
which presents a substantial challenge in representation learning.
Second, since the transitions of powder elements are mostly stochastic and unpredictable,
the agent must be able to correctly handle environment stochasticity.
Third,
the agent must achieve a high degree of generalization and sequential reasoning through a deep understanding of the physics,
in order to complete symmetrical, orderly test-time drawing tasks from random, chaotic data.

## 8 Results

We now present and discuss the benchmarking results of existing offline goal-conditioned RL algorithms on OGBench.

### 8.1 Algorithms

We benchmark six representative offline GCRL algorithms:
goal-conditioned behavioral cloning (GCBC) (Lynch et al., ; Ghosh et al., ),
goal-conditioned implicit {V, Q}-learning (GCIVL and GCIQL) (Kostrikov et al., ; Park et al., ),
quasimetric RL (QRL) (Wang et al., ),
contrastive RL (CRL) (Eysenbach et al., ),
and hierarchical implicit Q-learning (HIQL) (Park et al., ).
GCBC is the simplest goal-conditioned behavioral cloning method.
GCIVL and GCIQL are offline GCRL algorithms that approximates the optimal value function
using an expectile regression (Newey & Powell, ).
QRL is a non-traditional GCRL method that fits a quasimetric value function with a dual objective.
CRL is a “one-step” RL algorithm
that fits a Monte Carlo value function via contrastive learning and performs one-step policy improvement.
HIQL is a hierarchical RL method that extracts a two-level hierarchical policy from a single GCIVL value function.
For benchmarking, we perform a similar amount of hyperparameter tuning for each method to ensure fair comparison.
We refer the reader to Appendices C and D for the details.

**Table 2: Full benchmark table. We report each method’s average (binary) success rate ($\%$) across the five test-time goals on each task. The results are averaged over $8$ seeds ($4$ seeds for pixel-based tasks), and we report the standard deviations after the $\pm$ sign. The numbers within $95\%$ of the best value in the row are highlighted in bold.**
| Environment | Dataset Type | Dataset | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| pointmaze | navigate | pointmaze-medium-navigate-v0 | $9$ $\pm 6$ | $63$ $\pm 6$ | $53$ $\pm 8$ | $\mathbf{82}$ $\pm 5$ | $29$ $\pm 7$ | $\mathbf{79}$ $\pm 5$ |
| pointmaze-large-navigate-v0 | $29$ $\pm 6$ | $45$ $\pm 5$ | $34$ $\pm 3$ | $\mathbf{86}$ $\pm 9$ | $39$ $\pm 7$ | $58$ $\pm 5$ |  |  |
| pointmaze-giant-navigate-v0 | $1$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{68}$ $\pm 7$ | $27$ $\pm 10$ | $46$ $\pm 9$ |  |  |
| pointmaze-teleport-navigate-v0 | $25$ $\pm 3$ | $\mathbf{45}$ $\pm 3$ | $24$ $\pm 7$ | $4$ $\pm 4$ | $24$ $\pm 6$ | $18$ $\pm 4$ |  |  |
| stitch | pointmaze-medium-stitch-v0 | $23$ $\pm 18$ | $70$ $\pm 14$ | $21$ $\pm 9$ | $\mathbf{80}$ $\pm 12$ | 0 0 $\pm 1$ | $74$ $\pm 6$ |  |
| pointmaze-large-stitch-v0 | $7$ $\pm 5$ | $12$ $\pm 6$ | $31$ $\pm 2$ | $\mathbf{84}$ $\pm 15$ | 0 0 $\pm 0$ | $13$ $\pm 6$ |  |  |
| pointmaze-giant-stitch-v0 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{50}$ $\pm 8$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |
| pointmaze-teleport-stitch-v0 | $31$ $\pm 9$ | $\mathbf{44}$ $\pm 2$ | $25$ $\pm 3$ | $9$ $\pm 5$ | $4$ $\pm 3$ | $34$ $\pm 4$ |  |  |
| antmaze | navigate | antmaze-medium-navigate-v0 | $29$ $\pm 4$ | $72$ $\pm 8$ | $71$ $\pm 4$ | $88$ $\pm 3$ | $\mathbf{95}$ $\pm 1$ | $\mathbf{96}$ $\pm 1$ |
| antmaze-large-navigate-v0 | $24$ $\pm 2$ | $16$ $\pm 5$ | $34$ $\pm 4$ | $75$ $\pm 6$ | $83$ $\pm 4$ | $\mathbf{91}$ $\pm 2$ |  |  |
| antmaze-giant-navigate-v0 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $14$ $\pm 3$ | $16$ $\pm 3$ | $\mathbf{65}$ $\pm 5$ |  |  |
| antmaze-teleport-navigate-v0 | $26$ $\pm 3$ | $39$ $\pm 3$ | $35$ $\pm 5$ | $35$ $\pm 5$ | $\mathbf{53}$ $\pm 2$ | $42$ $\pm 3$ |  |  |
| stitch | antmaze-medium-stitch-v0 | $45$ $\pm 11$ | $44$ $\pm 6$ | $29$ $\pm 6$ | $59$ $\pm 7$ | $53$ $\pm 6$ | $\mathbf{94}$ $\pm 1$ |  |
| antmaze-large-stitch-v0 | $3$ $\pm 3$ | $18$ $\pm 2$ | $7$ $\pm 2$ | $18$ $\pm 2$ | $11$ $\pm 2$ | $\mathbf{67}$ $\pm 5$ |  |  |
| antmaze-giant-stitch-v0 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ |  |  |
| antmaze-teleport-stitch-v0 | $31$ $\pm 6$ | $\mathbf{39}$ $\pm 3$ | $17$ $\pm 2$ | $24$ $\pm 5$ | $31$ $\pm 4$ | $36$ $\pm 2$ |  |  |
| explore | antmaze-medium-explore-v0 | $2$ $\pm 1$ | $19$ $\pm 3$ | $13$ $\pm 2$ | $1$ $\pm 1$ | $3$ $\pm 2$ | $\mathbf{37}$ $\pm 10$ |  |
| antmaze-large-explore-v0 | 0 0 $\pm 0$ | $\mathbf{10}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $4$ $\pm 5$ |  |  |
| antmaze-teleport-explore-v0 | $2$ $\pm 1$ | $32$ $\pm 2$ | $7$ $\pm 3$ | $2$ $\pm 2$ | $20$ $\pm 2$ | $\mathbf{34}$ $\pm 15$ |  |  |
| humanoidmaze | navigate | humanoidmaze-medium-navigate-v0 | $8$ $\pm 2$ | $24$ $\pm 2$ | $27$ $\pm 2$ | $21$ $\pm 8$ | $60$ $\pm 4$ | $\mathbf{89}$ $\pm 2$ |
| humanoidmaze-large-navigate-v0 | $1$ $\pm 0$ | $2$ $\pm 1$ | $2$ $\pm 1$ | $5$ $\pm 1$ | $24$ $\pm 4$ | $\mathbf{49}$ $\pm 4$ |  |  |
| humanoidmaze-giant-navigate-v0 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 0$ | $3$ $\pm 2$ | $\mathbf{12}$ $\pm 4$ |  |  |
| stitch | humanoidmaze-medium-stitch-v0 | $29$ $\pm 5$ | $12$ $\pm 2$ | $12$ $\pm 3$ | $18$ $\pm 2$ | $36$ $\pm 2$ | $\mathbf{88}$ $\pm 2$ |  |
| humanoidmaze-large-stitch-v0 | $6$ $\pm 3$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 1$ | $4$ $\pm 1$ | $\mathbf{28}$ $\pm 3$ |  |  |
| humanoidmaze-giant-stitch-v0 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 2$ |  |  |
| antsoccer | navigate | antsoccer-arena-navigate-v0 | $5$ $\pm 1$ | $47$ $\pm 3$ | $50$ $\pm 2$ | $8$ $\pm 2$ | $23$ $\pm 2$ | $\mathbf{58}$ $\pm 2$ |
| antsoccer-medium-navigate-v0 | $2$ $\pm 0$ | $4$ $\pm 1$ | $7$ $\pm 1$ | $2$ $\pm 2$ | $3$ $\pm 1$ | $\mathbf{13}$ $\pm 2$ |  |  |
| stitch | antsoccer-arena-stitch-v0 | $\mathbf{24}$ $\pm 8$ | $21$ $\pm 3$ | $2$ $\pm 0$ | $1$ $\pm 1$ | $1$ $\pm 0$ | $15$ $\pm 1$ |  |
| antsoccer-medium-stitch-v0 | $2$ $\pm 1$ | $1$ $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 1$ |  |  |
| visual-antmaze | navigate | visual-antmaze-medium-navigate-v0 | $11$ $\pm 2$ | $22$ $\pm 2$ | $11$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{94}$ $\pm 1$ | $\mathbf{93}$ $\pm 4$ |
| visual-antmaze-large-navigate-v0 | $4$ $\pm 0$ | $5$ $\pm 1$ | $4$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{84}$ $\pm 1$ | $53$ $\pm 9$ |  |  |
| visual-antmaze-giant-navigate-v0 | 0 0 $\pm 0$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{47}$ $\pm 2$ | $6$ $\pm 4$ |  |  |
| visual-antmaze-teleport-navigate-v0 | $5$ $\pm 1$ | $8$ $\pm 1$ | $6$ $\pm 1$ | $6$ $\pm 3$ | $\mathbf{48}$ $\pm 2$ | $37$ $\pm 2$ |  |  |
| stitch | visual-antmaze-medium-stitch-v0 | $67$ $\pm 4$ | $6$ $\pm 2$ | $2$ $\pm 0$ | 0 0 $\pm 0$ | $69$ $\pm 2$ | $\mathbf{87}$ $\pm 2$ |  |
| visual-antmaze-large-stitch-v0 | $24$ $\pm 3$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $11$ $\pm 3$ | $\mathbf{28}$ $\pm 2$ |  |  |
| visual-antmaze-giant-stitch-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| visual-antmaze-teleport-stitch-v0 | $32$ $\pm 3$ | $1$ $\pm 1$ | $1$ $\pm 0$ | $1$ $\pm 2$ | $32$ $\pm 6$ | $\mathbf{37}$ $\pm 4$ |  |  |
| explore | visual-antmaze-medium-explore-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |
| visual-antmaze-large-explore-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| visual-antmaze-teleport-explore-v0 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 0$ | $\mathbf{19}$ $\pm 8$ |  |  |
| visual-humanoidmaze | navigate | visual-humanoidmaze-medium-navigate-v0 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 0$ | 0 0 $\pm 0$ |
| visual-humanoidmaze-large-navigate-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| visual-humanoidmaze-giant-navigate-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| stitch | visual-humanoidmaze-medium-stitch-v0 | $\mathbf{1}$ $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 0$ | 0 0 $\pm 0$ |  |
| visual-humanoidmaze-large-stitch-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| visual-humanoidmaze-giant-stitch-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| cube | play | cube-single-play-v0 | $6$ $\pm 2$ | $53$ $\pm 4$ | $\mathbf{68}$ $\pm 6$ | $5$ $\pm 1$ | $19$ $\pm 2$ | $15$ $\pm 3$ |
| cube-double-play-v0 | $1$ $\pm 1$ | $36$ $\pm 3$ | $\mathbf{40}$ $\pm 5$ | $1$ $\pm 0$ | $10$ $\pm 2$ | $6$ $\pm 2$ |  |  |
| cube-triple-play-v0 | $1$ $\pm 1$ | $1$ $\pm 0$ | $3$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 1$ | $3$ $\pm 1$ |  |  |
| cube-quadruple-play-v0 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| scene | play | scene-play-v0 | $5$ $\pm 1$ | $42$ $\pm 4$ | $\mathbf{51}$ $\pm 4$ | $5$ $\pm 1$ | $19$ $\pm 2$ | $38$ $\pm 3$ |
| puzzle | play | puzzle-3x3-play-v0 | $2$ $\pm 0$ | $6$ $\pm 1$ | $\mathbf{95}$ $\pm 1$ | $1$ $\pm 0$ | $3$ $\pm 1$ | $12$ $\pm 2$ |
| puzzle-4x4-play-v0 | 0 0 $\pm 0$ | $13$ $\pm 2$ | $\mathbf{26}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $7$ $\pm 2$ |  |  |
| puzzle-4x5-play-v0 | 0 0 $\pm 0$ | $7$ $\pm 1$ | $\mathbf{14}$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 0$ | $4$ $\pm 1$ |  |  |
| puzzle-4x6-play-v0 | 0 0 $\pm 0$ | $10$ $\pm 2$ | $\mathbf{12}$ $\pm 1$ | 0 0 $\pm 0$ | $4$ $\pm 1$ | $3$ $\pm 1$ |  |  |
| visual-cube | play | visual-cube-single-play-v0 | $5$ $\pm 1$ | $60$ $\pm 5$ | $30$ $\pm 5$ | $41$ $\pm 15$ | $31$ $\pm 15$ | $\mathbf{89}$ $\pm 0$ |
| visual-cube-double-play-v0 | $1$ $\pm 1$ | $10$ $\pm 2$ | $1$ $\pm 1$ | $5$ $\pm 0$ | $2$ $\pm 1$ | $\mathbf{39}$ $\pm 2$ |  |  |
| visual-cube-triple-play-v0 | $15$ $\pm 2$ | $14$ $\pm 2$ | $15$ $\pm 1$ | $16$ $\pm 1$ | $17$ $\pm 2$ | $\mathbf{21}$ $\pm 0$ |  |  |
| visual-cube-quadruple-play-v0 | $8$ $\pm 1$ | 0 0 $\pm 0$ | $7$ $\pm 1$ | $5$ $\pm 1$ | $4$ $\pm 1$ | $\mathbf{14}$ $\pm 1$ |  |  |
| visual-scene | play | visual-scene-play-v0 | $12$ $\pm 2$ | $25$ $\pm 3$ | $12$ $\pm 2$ | $10$ $\pm 1$ | $11$ $\pm 2$ | $\mathbf{49}$ $\pm 4$ |
| visual-puzzle | play | visual-puzzle-3x3-play-v0 | 0 0 $\pm 0$ | $21$ $\pm 1$ | $1$ $\pm 2$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{73}$ $\pm 8$ |
| visual-puzzle-4x4-play-v0 | $10$ $\pm 1$ | $\mathbf{60}$ $\pm 5$ | $16$ $\pm 4$ | 0 0 $\pm 0$ | $10$ $\pm 6$ | $\mathbf{60}$ $\pm 41$ |  |  |
| visual-puzzle-4x5-play-v0 | $5$ $\pm 2$ | $\mathbf{17}$ $\pm 1$ | $7$ $\pm 2$ | 0 0 $\pm 0$ | $6$ $\pm 1$ | $13$ $\pm 9$ |  |  |
| visual-puzzle-4x6-play-v0 | $2$ $\pm 1$ | $\mathbf{15}$ $\pm 1$ | $2$ $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 1$ | $9$ $\pm 6$ |  |  |
| powderworld | play | powderworld-easy-play-v0 | 0 0 $\pm 0$ | $\mathbf{99}$ $\pm 1$ | $93$ $\pm 5$ | $12$ $\pm 2$ | $22$ $\pm 5$ | $33$ $\pm 9$ |
| powderworld-medium-play-v0 | $1$ $\pm 1$ | $\mathbf{50}$ $\pm 4$ | $16$ $\pm 5$ | $3$ $\pm 1$ | $1$ $\pm 1$ | $22$ $\pm 14$ |  |  |
| powderworld-hard-play-v0 | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ |  |  |

Figure: Figure 2: Benchmarking offline GCRL methods. We report the performances of six offline GCRL methods (GCBC, GCIVL, GCIQL, QRL, CRL, and HIQL), aggregated by different dataset categories (see Table 6 for the category list). The results are averaged over the tasks in each category, and then over $8$ seeds ($4$ seeds for pixel-based tasks). Error bars denote $95\%$ bootstrap confidence intervals. See Table 2 for the full results. In general, HIQL (a method that involves hierarchical policy extraction) tends to achieve strong performance across the board. Among the non-hierarchical methods, CRL tends to work best in locomotion tasks and GCIVL and GCIQL tend to work best in the others.
Refer to caption: x12.png

### 8.2 Benchmarking Results and Q&As

We present the full benchmarking results in Table 2.
Figure 2 summarizes the results by showing performances grouped by different task categories.
Performances are measured by average (binary) success rates on the five test-time goals of each task.
We train the agents for $1$M gradient steps ($500$K for pixel-based tasks),
and average the results over $8$ seeds ($4$ seeds for pixel-based tasks).
We discuss the results through Q&As.

Q:
Which method works best in general?

A:
While no single method dominates the others across all categories in Figure 2.
HIQL (a method that involves hierarchical policy extraction)
tends to achieve particularly strong performance among the benchmarked methods,
especially in locomotion and visual manipulation tasks.
Among the non-hierarchical methods,
CRL tends to work best in locomotion tasks and GCIQL tends to work best in manipulation tasks.
In the drawing tasks, GCIVL performs the best.

Q:
Which methods are good at goal stitching?

A:
To see this, we can compare the performances on the navigate and stitch datasets
from the same locomotion task in Table 2.
The results suggest that, as expected, full RL-based methods like HIQL
(*i.e.*, methods that fit the optimal value function $Q^{*}$)
are better at stitching than one-step RL methods like CRL
(*i.e.*, methods that fit the behavioral value function $Q^{\beta}$).
For example, in visual locomotion tasks,
the relative performance between HIQL and CRL is reversed on the stitch datasets (Figure 2).

Q:
Which methods are good at handling stochasticity?

A:
For this, we can compare the performances on the large and teleport mazes in Table 2,
where both have the same maze size, but only the latter involves stochastic transitions that incur risk.
Table 2 shows that,
value-only methods like HIQL and QRL (*i.e.*, methods that do not have a separate Q function),
which are optimistically biased in stochastic environments,
struggle relatively more in stochastic teleport tasks.
In contrast, CRL is generally robust to environment stochasticity, likely because it fits a Monte Carlo value function.

Q:
Which methods are good at handling pixel-based observations?

A:
Although state-based and pixel-based observations generally provide the same amount of information,
several methods struggle to handle image observations due to additional representational challenges.
We can understand how well a method addresses such representational challenges
by comparing the performances of corresponding state- and pixel-based tasks.
Table 2 shows that CRL is notably robust to the difference in input modalities,
likely because it is based on a pure representation learning objective.
HIQL also achieves strong performance in pixel-based tasks, especially in visual manipulation tasks.
However, these methods are still not perfect at handling image observations;
for example, HIQL achieves relatively weak performance on image drawing tasks.
We suspect this is due to the difficulty of learning low-dimensional subgoal representations from states of high intrinsic dimensionality.

**Table 3: How good are our reference implementations? Our implementations generally achieve better performance than previously reported ones.**
| D4RL antmaze-large-diverse-v2 | D4RL antmaze-large-play-v2 |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | Previously Reported Performance | Ours | Method | Previously Reported Performance | Ours |
| GCBC | $20$ (Park et al., ) | $\mathbf{41}$ $\pm 7$ | GCBC | $23$ (Park et al., ) | $\mathbf{39}$ $\pm 4$ |
| GCIVL | $51$ (Park et al., ) | $\mathbf{64}$ $\pm 8$ | GCIVL | $\mathbf{57}$ (Park et al., ) | $\mathbf{58}$ $\pm 8$ |
| GCIQL | $30$ (Zeng et al., ) | $\mathbf{64}$ $\pm 10$ | GCIQL | $40$ (Zeng et al., ) | $\mathbf{55}$ $\pm 11$ |
| QRL | $\mathbf{52}$(^7^77We note that Zheng et al. () use a different evaluation scheme based on the maximum performance over evaluation epochs. We report the average performance over the last three evaluation epochs (see Appendix D).) (Zheng et al., ) | $37$ $\pm 14$ | QRL | $\mathbf{53}$^7 (Zheng et al., ) | $38$ $\pm 8$ |
| CRL | $54$ (Eysenbach et al., ) | $\mathbf{79}$ $\pm 6$ | CRL | $49$ (Eysenbach et al., ) | $\mathbf{74}$ $\pm 4$ |
| HIQL | $\mathbf{88}$ (Park et al., ) | $\mathbf{87}$ $\pm 3$ | HIQL | $\mathbf{86}$ (Park et al., ) | $\mathbf{87}$ $\pm 2$ |

Q:
How good are our reference implementations?

A:
We compare the performance of our reference implementations with previously reported numbers
on one of the most commonly used tasks in prior work, D4RL antmaze-large (Fu et al., ).
Table 3 shows the comparison results,
with the corresponding numbers taken from the prior works (Eysenbach et al., ; Park et al., ; Zeng et al., ; Zheng et al., ).
The results suggest that our implementations generally achieve better performance than previously reported results,
sometimes significantly surpassing them (*e.g.*, CRL).

**Table 4: Do not use single-goal evaluation! Only using a single state-goal pair (a common practice when using D4RL tasks for offline GCRL) can potentially lead to inaccurate conclusions about offline GCRL methods. OGBench always uses multi-goal evaluation. See how the rank between GCIQL and QRL is reversed with multi-goal evaluation on the same antmaze-large maze.**
| Dataset | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- |
| D4RL antmaze-large-diverse-v2 (single-goal evaluation) | $41$ $\pm 7$ | $64$ $\pm 8$ | $64$ $\pm 10$ | $37$ $\pm 14$ | $79$ $\pm 6$ | $\mathbf{87}$ $\pm 3$ |
| D4RL antmaze-large-play-v2 (single-goal evaluation) | $39$ $\pm 4$ | $58$ $\pm 8$ | $55$ $\pm 11$ | $38$ $\pm 8$ | $74$ $\pm 4$ | $\mathbf{87}$ $\pm 2$ |
| OGBench antmaze-large-navigate-v0 (multi-goal evaluation, ours) | $24$ $\pm 2$ | $16$ $\pm 5$ | $34$ $\pm 4$ | $75$ $\pm 6$ | $83$ $\pm 4$ | $\mathbf{91}$ $\pm 2$ |

Q:
Why can’t I just use D4RL AntMaze instead of OGBench AntMaze for offline GCRL?

A:
D4RL AntMaze is an excellent task for benchmarking offline RL algorithms.
However, it is limited for benchmarking offline *goal-conditioned* RL algorithms
because it only involves a single, fixed state-goal pair,
and the datasets are tailored to this specific task (Fu et al., ).
In contrast, OGBench supports multi-goal evaluation
(and provides much more diverse types of tasks and datasets!).
To empirically demonstrate this difference,
we compare the benchmarking results on D4RL antmaze-large-{diverse, play} and OGBench antmaze-large-navigate
in Table 4.
The table suggests that single-goal evaluation is indeed limited, and is potentially prone to inaccurate conclusions:
for example, see how the ranking between GCIQL and QRL is reversed with multi-goal evaluation on the same antmaze-large maze.
Moreover, the performance differences between methods are more pronounced in OGBench AntMaze,
showing that OGBench provides clearer research signals.

Q:
There seem to be a lot of datasets. What should I use for my research?

A:
For general offline GCRL algorithms research,
we recommend starting with more “regular” datasets, such as
antmaze-{large, giant}-navigate, humanoidmaze-medium-navigate,
cube-{single, double}-play, scene-play, and puzzle-3x3-play.
From there, depending on the performance on these tasks,
try harder versions of them or
more challenging tasks, such as humanoidmaze-giant, antsoccer, puzzle-{4x4, 4x5, 4x6}, and powderworld.

We also provide more specialized datasets that pose specific challenges in offline GCRL (Section 5).
For stitching, try the stitch datasets in the locomotion suite
as well as complex manipulation tasks that require stitching (*e.g.*, puzzle).
For long-horizon reasoning, consider humanoidmaze-giant, which has the longest episode length,
and puzzle-4x6, which has the most semantic steps.
For stochastic control,
try antmaze-teleport, which is specifically designed to challenge optimistically biased methods,
and powderworld, which has unpredictable, stochastic dynamics.
For learning from highly suboptimal data, consider antmaze-explore as well as the noisy datasets in the manipulation suite,
which features high suboptimality and high coverage.

Q:
Have you found any insights on data collection for offline GCRL?

Figure: Figure 3: Datasets *must* be noisy enough.
Refer to caption: x13.png

A:
One of the main features of OGBench is that every task is accompanied by a reproducible and controllable data-generation script.
Here, we show one example of how this controllability
can lead to practical insights and raise open research questions.
In Figure 3, we ablate the strength of Gaussian action noise $\sigma$ added to expert actions
on two manipulation tasks (cube-single-noisy and puzzle-3x3-noisy),
and measure how this affects performance.

The results are quite remarkable:
they show that having the right amount of noise (*i.e.*, state coverage) is *extremely* important
for achieving good performance.
For example, the performance drops from $99\%$ to $6\%$ if there is no noise in expert actions,
even on the most basic cube pick-and-place task.
This suggests that, we may need to prioritize coverage much more than optimality when collecting datasets for offline GCRL in the real world as well,
and failing to do so may lead to (surprising) failures in learning.
Like action noise, we believe there are many other important properties of datasets that significantly affect performance.
We believe that our fully transparent, controllable data-generation scripts can facilitate such scientific studies.

## 9 Research Opportunities

In this section, we discuss potential research ideas and open questions.

Be the first to solve unsolved tasks!
While all environments in OGBench have at least one variant that current methods can solve to some degree,
there are still a number of challenging tasks on which no existing method achieves non-trivial performance,
such as humanoidmaze-giant, cube-triple, puzzle-4x5, powderworld-hard, and more.
We ensure that sufficient data is available for those tasks
(which is estimated from the amount needed to solve their easier versions).
We invite researchers to take on these challenges and push the limits of offline GCRL with better algorithms.

How can we develop a policy that *generalizes* well at test time?
In our experiments,
we found hierarchical RL methods (*e.g.*, HIQL) to work especially well in several tasks.
Among several potential explanations,
we hypothesize that this is mainly because
hierarchical RL reduces learning complexity by having two policies specialized in different things,
which makes both policies *generalize* better at evaluation time.
After all, test-time generalization is known to be
one of the major bottlenecks in offline RL (Park et al., ).
But, are hierarchies really necessary to achieve good test-time generalization?
Can we develop a non-hierarchical method that enjoys the same benefit
by exploiting the subgoal structure of offline GCRL?
This would be especially beneficial, not just because it is simpler,
but also because it can potentially yield better, unified *representations*
that can potentially serve as a “foundation model” for fine-tuning.

Can we develop a method that works well across all categories?
Our benchmarking results reveal that no method consistently performs best across the board.
HIQL tends to achieve strong performance
but struggles in pixel-based locomotion and state-based manipulation.
GCIQL shows strong performance in state-based manipulation, but struggles in locomotion.
CRL exhibits the opposite trend: it excels in locomotion but underperforms in manipulation.
Is there a way to combine only the strengths of these methods to develop
a single approach that achieves the best performance across all types of tasks?

More concrete research questions.
Here, we list additional, more concrete research questions that researchers may use as a starting point for research in offline GCRL:

- •
Why is PointMaze so hard?
Table 2 shows that PointMaze is surprisingly hard, sometimes even harder than AntMaze for some methods.
Why is this the case? Moreover, only in PointMaze does QRL significantly outperform the other methods.
What causes this difference, and are there any insights we can take from these results?
- •
How should we train subgoal representations?
Somewhat weirdly, HIQL struggles much more with state-based observations than pixel-based observations on manipulation tasks.
We suspect this is related to subgoal representations,
given that HIQL uses an additional learning signal from the policy loss to further train subgoal representations
*only* in pixel-based environments (which we found does not help in state-based environments).
HIQL uses a value function-based subgoal representation (Appendix C),
but is there a better, more stable way to learn subgoal representations for hierarchical RL and planning?
- •
Do we really need the full power of RL?
While learning the optimal $Q^{*}$ function is in principle better than learning the behavioral $Q^{\beta}$ function,
CRL (which fits $Q^{\beta}$) significantly outperforms GCIQL (which fits $Q^{*}$) on locomotion tasks.
Why is this the case? Is it a problem with expectile regression in GCIQL or with temporal difference learning itself?
In contrast, in manipulation environments, the result suggests the opposite: GCIQL is much better than CRL.
Does this mean we do need $Q^{*}$ in these tasks? Or can it be solvable even with $Q^{\beta}$
if we use a better behavioral value learning technique than binary NCE in CRL?
- •
Why can’t we use random goals when training policies?
When training goal-conditioned policies,
we found that it is usually better to sample (policy) goals *only* from the future state in the current trajectory
(except on stitch or explore datasets; see Table 9).
The fact that this works better even in Scene and Puzzle (which require goal stitching) is a bit surprising,
because it means that the policy can still perform goal stitching to some degree
even without being explicitly trained on the test-time state-goal pairs.
At the same time, it is rather unsatisfying because this ability to stitch goals
entirely depends on the seemingly “magical” generalization capabilities of neural networks.
Is there a way to train a goal-conditioned policy with random goals while maintaining performance,
so that it can perform goal stitching in a principled manner?
- •
How can we combine expressive policies with GCRL?
In Appendix A,
we show that current offline GCRL methods often struggle with datasets collected by non-Markovian policies in manipulation environments.
Handling non-Markovian trajectory data is indeed one of the major challenges in behavioral cloning,
for which many recent BC-based methods have been proposed (Zhao et al., ; Chi et al., ).
How can we incorporate these recent advancements in behavioral cloning into offline GCRL?

## 10 Outlook

In this work, we introduced OGBench, a new benchmark designed to advance algorithms research in offline goal-conditioned RL.
With the experimental results, we now revisit the very first question posed in this paper: “Why offline goal-conditioned RL?”

We hypothesize that offline goal-conditioned holds significant potential as a recipe for *general-purpose RL pre-training*,
which yields richer and more diverse behaviors than (arguably more prevalent) generative pre-training,
such as behavioral cloning and next-token prediction.
Our experiments, albeit preliminary, show that
even current offline GCRL algorithms can to some extent acquire effective policies for
exceptionally long-horizon tasks entirely with sparse rewards,
using data that is highly suboptimal.
These results are not limited to toy example domains,
but show up across a range of realistic simulated robotics and game-like settings in our benchmark.
While generative objectives might capture the data distribution,
offline GCRL can learn policies that actually achieve complex outcomes
(such as beating a puzzle game)
that could not be achieved simply by copying random data.

However, current offline GCRL algorithms also have limitations.
As shown in our results,
they often struggle with long-horizon, high-dimensional tasks and those that require stitching,
and no single method consistently outperforms others across all tasks.
This suggests that we have not yet found the ideal algorithm
that can fully realize the promise of offline GCRL as general-purpose RL pre-training.
The first step toward finding such an ideal algorithm is to set up a solid benchmark
that sufficiently challenges the limits of offline GCRL from diverse perspectives.
We believe OGBench provides this foundation,
and will lead to the development of performant, scalable offline GCRL algorithms
that enable building foundation models for general-purpose behaviors.

## Acknowledgments

We thank Kevin Zakka for providing the initial codebase for the manipulation environments and helping with MuJoCo implementations,
Vivek Myers for providing a Jax-based QRL implementation,
and Colin Li, along with the members of the RAIL lab, for helpful discussions.
This work was partly supported by the Korea Foundation for Advanced Studies (KFAS), National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2146752, ONR under N00014-20-1-2383 and N00014-22-1-2773, and Qualcomm.
This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.

## Reproducibility Statement

We provide the full implementation details in Appendix D.
We provide the code as well as the exact command-line flags to reproduce the entire benchmark table, datasets, and expert policies
at [https://github.com/seohongpark/ogbench](https://github.com/seohongpark/ogbench).

## Appendix A Additional Datasets

For manipulation tasks (Cube, Scene, and Puzzle),
in addition to the main play datasets,
we additionally provide noisy datasets that can potentially be useful for other types of research
(*e.g.*, ablation studies on datasets, comparing performances on non-Markovian and Markovian datasets, etc.).
The main difference is that the play datasets are collected by open-loop, non-Markovian expert policies with temporally correlated noise,
while the noisy datasets are collected by closed-loop, Markovian expert policies with larger, uncorrelated Gaussian noise.
Hence, the play datasets generally look more “natural” than the noisy datasets, but the latter has higher state coverage ([videos](https://seohong.me/projects/ogbench)).

Table 5 shows the full benchmark results on both the play and noisy datasets in the manipulation suite.
The results suggest that the performances on these two datasets are mostly similar,
but several methods struggle to handle narrower and non-Markovian trajectories in play datasets
(*e.g.*, GCIQL almost perfectly solves cube-single-noisy but struggles on cube-single-play).

## Appendix B Prior Work in Goal-Conditioned RL

The problem of reaching any goal from any state has long been considered
one of the central problems in reinforcement learning and sequential decision making (Kaelbling, ; Schaul et al., ; Andrychowicz et al., ),
owing to its unsupervised nature, simplicity, and generality.
There are many unique features of goal-conditioned RL that make it distinct from other (multi-task) RL problems,
such as the presence of recursive subgoal structures, metric structures, and probabilistic interpretations.
These intriguing properties have led to the development of diverse families of online and offline GCRL algorithms based on
hindsight relabeling (Andrychowicz et al., ),
hierarchical learning (Dayan & Hinton, ; Chane-Sane et al., ; Li et al., ; Park et al., ),
planning (Savinov et al., ; Eysenbach et al., ; Nasiriany et al., ; Huang et al., ; Hoang et al., ; Kim et al., ),
metric learning (Wang et al., ; Park et al., ; Myers et al., ),
dual optimization (Ma et al., ; ; Sikchi et al., ),
weighted behavioral cloning (Yang et al., ; ; Hejna et al., ),
generative modeling (Zeng et al., ; Hong et al., ),
and contrastive learning (Eysenbach et al., ; Zheng et al., ; ).
In this work, we also consider the problem of offline goal-conditioned RL;
however, instead of proposing a new algorithm,
we introduce a new benchmark and reference implementations
to facilitate and advance algorithms research in offline GCRL.

## Appendix C Offline GCRL Algorithms

In this section, we describe the six offline GCRL methods used for benchmarking in detail.
We first define four goal-sampling distributions
that correspond to the current state, uniform future states, geometric future states, and random states, respectively:

- •
$p^{\mathcal{D}}_{\mathrm{cur}}({\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)$ denotes the Dirac delta distribution at $s$ (*i.e.*, $g$ is always set to $s$).
- •
$p^{\mathcal{D}}_{\mathrm{traj}}({\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)$ denotes the uniform future state distribution defined as follows:
assuming $s=s_{t}$ in a trajectory $\tau=(s_{0},a_{0},s_{1},\dots,s_{T})$(^8^88If there are multiple such $(\tau,t)$ tuples in the dataset, consider the uniform mixture of them.),
we sample an index $k$ from the uniform distribution $\mathrm{Unif}(\min(t+1,T-1),T-1)$ (inclusive),
and set $g=s_{k}$.
- •
$p^{\mathcal{D}}_{\mathrm{geom}}({\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)$ denotes the truncated geometric future state distribution defined as follows:
assuming $s=s_{t}$ in a trajectory $\tau=(s_{0},a_{0},s_{1},\dots,s_{T})$,
we sample an index $k$ from the geometric distribution $\mathrm{Geom}(1-\gamma)$
(whose support starts from $1$),
and set $g=s_{\min(s+k,T-1)}$.
- •
$p^{\mathcal{D}}_{\mathrm{rand}}({\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})$ denotes the uniform state distribution over the dataset ${\mathcal{D}}$.

Additionally, $p^{\mathcal{D}}_{\mathrm{mixed}}({\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)$ denotes a mixture of these four goal-sampling distributions
with a mixture ratio defined by hyperparameters,
$p^{\mathcal{D}}(\cdot)$ simply denotes the uniform distribution over the dataset,
and we sometimes use $p^{\mathcal{D}}_{\cdot}(\cdot\mid s,a)$ instead of $p^{\mathcal{D}}_{\cdot}(\cdot\mid s)$ to denote
the distribution corresponding to the state-action pair.

Goal-conditioned behavioral cloning (GCBC).
GCBC (Lynch et al., ; Ghosh et al., )
simply performs behavioral cloning using future states in the same trajectory as goals.
GCBC maximizes the following objective to train a goal-conditioned policy $\pi({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}a}\mid{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}s},{\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})$.

$$ $\displaystyle J_{\mathrm{GCBC}}(\pi)=\mathbb{E}_{(s,a)\sim p^{\mathcal{D}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}a}),g\sim p^{\mathcal{D}}_{\mathrm{traj}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)}[\log\pi(a\mid s,g)].$ (1) $$

Goal-conditioned implicit {V, Q}-learning (GCIVL and GCIQL).
GCIVL and GCIQL are goal-conditioned variants of implicit Q-learning (IQL) (Kostrikov et al., ),
which is an offline RL algorithm that fits the optimal value functions ($V^{*}$ or $Q^{*}$)
using an expectile regression (Newey & Powell, ).
GCIQL is a straightforward goal-conditioned variant of IQL,
and GCIVL is the $V$-only variant introduced by Park et al. ().
GCIVL fits a value function $V({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}g})$ by minimizing the following loss:

$$ $\displaystyle{\mathcal{L}}_{\mathrm{GCIVL}}(V)=\mathbb{E}_{s\sim p^{\mathcal{D }}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s}),g\sim p^{\mathcal{D}}_{ \mathrm{mixed}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named] {pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)}\left[\ell_{ \kappa}^{2}\left(r(s,g)+\gamma\bar{V}(s^{\prime},g)-V(s,g)\right)\right],$ (2) $$

where $r(s,g)=\mathbf{1}_{\{g\}}(s)-1$ denotes the $(-1,0)$-sparse goal-conditioned reward function,
$\bar{V}$ denotes the target value function (Mnih et al., ),
and $\ell_{\kappa}^{2}(x)=|\kappa-\mathbf{1}_{\{\,x\,:\,x<0\,\}}(x)|x^{2}$ denotes
the expectile loss with an expectile $\kappa$.

GCIQL fits both $V({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}g})$ and $Q({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}a},{\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})$
by jointly minimizing the following losses:

$$ $\displaystyle{\mathcal{L}}_{\mathrm{GCIQL}}^{V}(V)$ $\displaystyle=\mathbb{E}_{(s,a)\sim p^{\mathcal{D}}({\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}s},{\color[rgb]{0.6015625,0.6015625,0.6015625} \definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}a}),g \sim p^{\mathcal{D}}_{\mathrm{mixed}}({\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}g}\mid s)}\left[\ell_{\kappa}^{2}\left(\bar{Q}(s ,a,g)-V(s,g)\right)\right],$ (3) $\displaystyle{\mathcal{L}}_{\mathrm{GCIQL}}^{Q}(Q)$ $\displaystyle=\mathbb{E}_{(s,a,s^{\prime})\sim p^{\mathcal{D}}({\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}s},{\color[rgb]{0.6015625,0.6015625,0.6015625} \definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}a},{ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}s^{\prime}}),g\sim p^{\mathcal{D}}_{\mathrm {mixed}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)}\left[\left(r(s,g )+\gamma V(s^{\prime},g)-Q(s,a,g)\right)^{2}\right],$ (4) $$

where $\bar{Q}$ denotes the target Q function (Mnih et al., ).
We note that GCIVL is optimistically biased in stochastic environments,
but GCIQL is unbiased (Kostrikov et al., ; Park et al., ).

To extract a policy from the learned value functions, we can use either
advantage-weighted regression (AWR) (Peters & Schaal, ; Peng et al., ) or
behavior-constrained deep deterministic policy gradient (DDPG+BC) (Fujimoto & Gu, ).
GCIVL uses the following value-only variant of the AWR objective (Park et al., ):

$$ $\displaystyle J_{\mathrm{AWR}}^{V}(\pi)=\mathbb{E}_{(s,a,s^{\prime})\sim p^{ \mathcal{D}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}a},{\color[rgb]{0.6015625,0.6015625,0.6015625} \definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s^{ \prime}}),g\sim p^{\mathcal{D}}_{\mathrm{mixed}}({\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}g}\mid s)}\left[e^{\alpha(V(s^{\prime},g)-V(s,g) )}\log\pi(a\mid s,g)\right],$ (5) $$

where $\alpha$ is the temperature hyperparameter.
In our experiments,
GCIQL mainly uses the following DDPG+BC objective
(which is known to be better than AWR (Park et al., )):

$$ $\displaystyle J_{\mathrm{DDPG+BC}}(\pi)=\mathbb{E}_{(s,a)\sim p^{\mathcal{D}}( {\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor} {rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}a}),g\sim p^{\mathcal{D}}_{\mathrm{mixed}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)}\left[Q(s,\pi^{\mu}(s,g),g)+ \alpha\log\pi(a\mid s,g)\right],$ (6) $$

where $\pi^{\mu}(s,g)=\mathbb{E}_{a\sim\pi({\color[rgb]{0.6015625,0.6015625,0.6015625
}\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}a}\mid
s
,g)}[a]$.
In discrete-action environments, GCIQL uses the following Q version of AWR:

$$ $\displaystyle J_{\mathrm{AWR}}^{Q}(\pi)=\mathbb{E}_{(s,a,s^{\prime})\sim p^{ \mathcal{D}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}a},{\color[rgb]{0.6015625,0.6015625,0.6015625} \definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s^{ \prime}}),g\sim p^{\mathcal{D}}_{\mathrm{mixed}}({\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}g}\mid s)}\left[e^{\alpha(Q(s,a,g)-V(s,g))}\log \pi(a\mid s,g)\right].$ (7) $$

In practice, we use standard double-value learning for GCIVL and GCIQL (Park et al., ),
and Q normalization for DDPG+BC (Fujimoto & Gu, ).

Quasimetric RL (QRL).
QRL (Wang et al., ) is a goal-conditioned value learning algorithm
based on quasimetric learning, where a quasimetric means an asymmetric metric.
In deterministic environments,
the shortest path length between two states $d^{*}(s,g)$
is equivalent to the optimal undiscounted goal-conditioned value function $V^{*}(s,g)$
under the $(-1,0)$-sparse reward function (Wang et al., ):
$V^{*}(s,g)=-d^{*}(s,g)$.
The main idea of QRL is to explicitly leverage the quasimetric property (*i.e.*, triangle inequality) of shortest path lengths,
namely $d^{*}(s,w)+d^{*}(w,g)\geq d^{*}(s,g)$ for any $s,w,g\in{\mathcal{S}}$,
by modeling it with a quasimetric network architecture
like MRN (Liu et al., ) or IQE (Wang & Isola, ).
Concretely, QRL maximizes the following constrained optimization objective:

$$ $\displaystyle\mathrm{maximize}\quad$ $\displaystyle\mathbb{E}_{s\sim p^{\mathcal{D}}({\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}s}),g\sim p^{\mathcal{D}}_{\mathrm{rand}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}g})}[d(s,g)]$ (8) $\displaystyle\mathrm{s.t.}\quad$ $\displaystyle\mathbb{E}_{(s,s^{\prime})\sim p^{\mathcal{D}}({\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}s},{\color[rgb]{0.6015625,0.6015625,0.6015625} \definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s^{ \prime}})}\left[(d(s,s^{\prime})-1)^{2}\right]\leq{\varepsilon}^{2},$ (9) $$

where $d({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}g})$ is a quasimetric distance function (*e.g.*, an IQE network)
and ${\varepsilon}$ is a hyperparameter that controls the strength of the constraint.

To extract a policy from the value function $V({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}g})=-d({\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}s},{\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})$,
QRL uses the value-only AWR loss Equation 5 in discrete-action MDPs.
In continuous-action MDPs,
QRL additionally fits a dynamics model $f({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}\phi(s)},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}a})\colon{\mathcal{Z}}\times{\mathcal{A}}\to{
\mathcal{Z}}$ (Wang et al., ),
where ${\mathcal{Z}}$ denotes a latent space
and $\phi({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s})\colon{\mathcal{S}}\to{
\mathcal{Z}}$ denotes the representation function used in the quasimetric distance function:
namely, $d(s,g)=\tilde{d}(\phi(s),\phi(g))$ (*e.g.*, $\phi$ is the interval representation function in IQE).
The dynamics loss is as follows:

$$ $\displaystyle{\mathcal{L}}_{\mathrm{dyn}}(f)=\mathbb{E}_{(s,a,s^{\prime})\sim p ^{\mathcal{D}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}a},{\color[rgb]{0.6015625,0.6015625,0.6015625} \definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s^{ \prime}})}\left[\tilde{d}\left(\phi(s^{\prime}),f(\phi(s),a)\right)+\tilde{d} \left(f(\phi(s),a),\phi(s^{\prime})\right)\right],$ (10) $$

where QRL jointly trains both $d$ and $f$ without stop-gradients.
Based on the dynamics model $f$, QRL maximizes the following DDPG+BC-like loss:

$$ $\displaystyle J_{\mathrm{DDPG+BC}}(\pi)=\mathbb{E}_{(s,a)\sim p^{\mathcal{D}}( {\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor} {rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}a}),g\sim p^{\mathcal{D}}_{\mathrm{mixed}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}g}\mid s)}\left[-\tilde{d}\left(f(\phi(s), \pi^{\mu}(s,g)),\phi(g)\right)+\alpha\log\pi(a\mid s,g)\right],$ (11) $$

where we use the same notation as Equation 6.

In practice, we employ a $\mathrm{softplus}$ loss shaping for the quasimetric loss
and delta prediction for the dynamics model, as in Wang et al. ().

Contrastive RL (CRL).
CRL (Eysenbach et al., ) is a “one-step” GCRL algorithm
that first trains a Monte Carlo goal-conditioned value function using contrastive learning
and performs a one-step policy improvement.
CRL maximizes the following binary NCE objective (Ma & Collins, )
with respect to $f({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}a},{\color[rgb]{0.6015625,0.6015625,0.6015625}
\definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})
\colon{\mathcal{S}}\times{\mathcal{A}}\times{\mathcal{S}}\to{\mathbb{R}}$:

$$ $\displaystyle J_{\mathrm{CRL}}(f)=\mathbb{E}_{(s,a)\sim p^{\mathcal{D}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{ 0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.6015625,0.6015625,0.6015625}a}),g\sim p^{\mathcal{D}}_{\mathrm{geom}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}g}\mid s,a),g^{-}\sim p^{\mathcal{D}}_{ \mathrm{rand}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})}[\log\sigma(f(s,a,g))+ \log(1-\sigma(f(s,a,g^{-})))],$ (12) $$

where $\sigma\colon{\mathbb{R}}\to(0,1)$ denotes the sigmoid function.
The optimal solution to the above objective is given as $f(s,a,g)=\log(p^{\mathcal{D}}_{\mathrm{geom}}(g\mid s,a)/p^{\mathcal{D}}_{
\mathrm{rand}}(g))$.
Given the equivalence between the geometric future goal distribution ($p^{\mathcal{D}}_{\mathrm{geom}}$)
and the Monte Carlo goal-conditioned Q function ($Q^{\mathrm{MC}}$) under the $(0,1)$-sparse reward function $r(s,g)=\mathbf{1}_{\{g\}}(s)$ (Eysenbach et al., ),
we get the following relation:
$f(s,a,g)=\log Q^{\mathrm{MC}}(s,a,g)+C(g)$, where $C$ is a function that only depends on $g$.
In practice, $f$ is parameterized as $f(s,a,g)=\phi(s,a)^{\top}\psi(g)/\sqrt{d}$
with $\phi\colon{\mathcal{S}}\times{\mathcal{A}}\to{\mathcal{Z}}={\mathbb{R}}^{d}$ and $\psi\colon{\mathcal{S}}\to{\mathcal{Z}}={\mathbb{R}}^{d}$
(note that this inner-product parameterization is universal (Park et al., )),
and we use the future goals from the other states in the same batch as $g^{-}$.
We also employ the double-value learning technique for $f$ (Eysenbach et al., ).

For policy extraction,
in continuous-action MDPs,
we employ DDPG+BC (Equation 6) using $f$ instead of $Q$.
In discrete-action MDPs, we use AWR (Equation 5) using $(f,f^{V})$ instead of $(Q,V)$,
where we additionally train a contrastive value function $f^{V}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{
pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}s},{\color[rgb]{
0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{
0.6015625,0.6015625,0.6015625}g})\colon{\mathcal{S}}\times{\mathcal{S}}\to{
\mathbb{R}}$ using

$$ $\displaystyle J_{\mathrm{CRL-V}}(f^{V})=\mathbb{E}_{s\sim p^{\mathcal{D}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}s}),g\sim p^{\mathcal{D}}_{\mathrm{geom}}({ \color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{pgfstrokecolor}{ rgb}{0.6015625,0.6015625,0.6015625}g}\mid s),g^{-}\sim p^{\mathcal{D}}_{ \mathrm{rand}}({\color[rgb]{0.6015625,0.6015625,0.6015625}\definecolor[named]{ pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g})}[\log\sigma(f^{V}(s,g)) +\log(1-\sigma(f^{V}(s,g^{-})))],$ (13) $$

with a similar inner product parameterization for $f^{V}$.

Hierarchical implicit Q-learning (HIQL).
HIQL (Park et al., ) is an offline GCRL algorithm that extracts two policies
from a single goal-conditioned value function.
HIQL first trains GCIVL (Equation 2)
with a parameterized value function defined as $V(s,g)=\tilde{V}(s,\phi(s,g))$,
where $\phi\colon{\mathcal{S}}\times{\mathcal{S}}\to{\mathcal{Z}}$ serves
as a (state-dependent) subgoal representation function.
Based on the GCIVL value function, HIQL extracts a high-level policy $\pi^{h}\colon{\mathcal{S}}\times{\mathcal{S}}\to\Delta({\mathcal{Z}})$
and a low-level policy $\pi^{\ell}\colon{\mathcal{S}}\times{\mathcal{Z}}\to\Delta({\mathcal{A}})$
with the following AWR-like objectives:

$$ $\displaystyle J_{\mathrm{HIQL}}^{h}(\pi^{h})$ $\displaystyle=\mathbb{E}_{(s_{t},s_{t+k})\sim p^{\mathcal{D}},g\sim p^{ \mathcal{D}}_{\mathrm{mixed}}({\color[rgb]{0.6015625,0.6015625,0.6015625} \definecolor[named]{pgfstrokecolor}{rgb}{0.6015625,0.6015625,0.6015625}g}\mid s _{t})}\left[e^{\alpha(V(s_{t+k},g)-V(s_{t},g))}\log\pi^{h}(\phi(s_{t},s_{t+k}) \mid s_{t},g)\right],$ (14) $\displaystyle J_{\mathrm{HIQL}}^{\ell}(\pi^{\ell})$ $\displaystyle=\mathbb{E}_{(s_{t},a_{t},s_{t+1},s_{t+k})\sim p^{\mathcal{D}}} \left[e^{\alpha(V(s_{t+1},s_{t+k})-V(s_{t},s_{t+k}))}\log\pi^{\ell}(a_{t}\mid s _{t},\phi(s_{t},s_{t+k}))\right],$ (15) $$

where we omit the arguments in $p^{\mathcal{D}}$, and $k$ denotes a hyperparameter corresponding to the subgoal step.
For simplicity, we ignore some edge cases in the objectives above (*e.g.*, when $t+k$ exceeds the trajectory boundary, in which case we truncate);
we refer to Park et al. () or our code for the full details.
Intuitively, the high-level policy predicts the representation of the optimal $k$-step subgoal,
and the low-level policy predicts the optimal action based on the predicted subgoal.

In practice, following Park et al. (),
we use the double-value learning technique,
normalize the output of $\phi$,
and allow gradient flows from the low-level AWR loss into $\phi$ (only) in pixel-based environments.

## Appendix D Implementation Details

We provide the full implementation details in this section.
We release the code as well as the exact command-line flags
to reproduce the entire benchmark table, datasets, and expert policies at [https://github.com/seohongpark/ogbench](https://github.com/seohongpark/ogbench).

**Table 6: Dataset categories. We list the dataset categories used to aggregate results in Figure 2. Note that some datasets or tasks (*e.g.*, PointMaze) do not belong to any of these aggregation categories (for being too simple, too special, etc.).**
| Category | Datasets |
| --- | --- |
| Locomotion (states) | antmaze-medium-navigate-v0 |
| antmaze-large-navigate-v0 |  |
| antmaze-giant-navigate-v0 |  |
| humanoidmaze-medium-navigate-v0 |  |
| humanoidmaze-large-navigate-v0 |  |
| humanoidmaze-giant-navigate-v0 |  |
| antsoccer-arena-navigate-v0 |  |
| antsoccer-medium-navigate-v0 |  |
| Locomotion (pixels) | visual-antmaze-medium-navigate-v0 |
| visual-antmaze-large-navigate-v0 |  |
| visual-antmaze-giant-navigate-v0 |  |
| visual-humanoidmaze-medium-navigate-v0 |  |
| visual-humanoidmaze-large-navigate-v0 |  |
| visual-humanoidmaze-giant-navigate-v0 |  |
| Manipulation (states) | cube-single-play-v0 |
| cube-double-play-v0 |  |
| cube-triple-play-v0 |  |
| cube-quadruple-play-v0 |  |
| scene-play-v0 |  |
| puzzle-3x3-play-v0 |  |
| puzzle-4x4-play-v0 |  |
| puzzle-4x5-play-v0 |  |
| puzzle-4x6-play-v0 |  |
| Manipulation (pixels) | visual-cube-single-play-v0 |
| visual-cube-double-play-v0 |  |
| visual-cube-triple-play-v0 |  |
| visual-cube-quadruple-play-v0 |  |
| visual-scene-play-v0 |  |
| visual-puzzle-3x3-play-v0 |  |
| visual-puzzle-4x4-play-v0 |  |
| visual-puzzle-4x5-play-v0 |  |
| visual-puzzle-4x6-play-v0 |  |
| Drawing (pixels) | powderworld-easy-play-v0 |
| powderworld-medium-play-v0 |  |
| powderworld-hard-play-v0 |  |
| Stitching (states) | antmaze-medium-stitch-v0 |
| antmaze-large-stitch-v0 |  |
| antmaze-giant-stitch-v0 |  |
| humanoidmaze-medium-stitch-v0 |  |
| humanoidmaze-large-stitch-v0 |  |
| humanoidmaze-giant-stitch-v0 |  |
| antsoccer-arena-stitch-v0 |  |
| antsoccer-medium-stitch-v0 |  |
| Stitching (pixels) | visual-antmaze-medium-stitch-v0 |
| visual-antmaze-large-stitch-v0 |  |
| visual-antmaze-giant-stitch-v0 |  |
| visual-humanoidmaze-medium-stitch-v0 |  |
| visual-humanoidmaze-large-stitch-v0 |  |
| visual-humanoidmaze-giant-stitch-v0 |  |
| Exploratory (states) | antmaze-medium-explore-v0 |
| antmaze-large-explore-v0 |  |
| Exploratory (pixels) | visual-antmaze-medium-explore-v0 |
| visual-antmaze-large-explore-v0 |  |
| Stochastic (states) | antmaze-teleport-navigate-v0 |
| Stochastic (pixels) | visual-antmaze-telpeport-navigate-v0 |

**Table 7: Environment specifications.**
| Environment Type | Environment | State Dim. | Action Dim. | Maximum Episode Length |
| --- | --- | --- | --- | --- |
| pointmaze | pointmaze-medium-v0 | $2$ | $2$ | $1000$ |
| pointmaze-large-v0 | $2$ | $2$ | $1000$ |  |
| pointmaze-giant-v0 | $2$ | $2$ | $1000$ |  |
| pointmaze-teleport-v0 | $2$ | $2$ | $1000$ |  |
| antmaze | antmaze-medium-v0 | $29$ | $8$ | $1000$ |
| antmaze-large-v0 | $29$ | $8$ | $1000$ |  |
| antmaze-giant-v0 | $29$ | $8$ | $1000$ |  |
| antmaze-teleport-v0 | $29$ | $8$ | $1000$ |  |
| humanoidmaze | humanoidmaze-medium-v0 | $69$ | $21$ | $2000$ |
| humanoidmaze-large-v0 | $69$ | $21$ | $2000$ |  |
| humanoidmaze-giant-v0 | $69$ | $21$ | $4000$ |  |
| antsoccer | antsoccer-arena-v0 | $42$ | $8$ | $1000$ |
| antsoccer-medium-v0 | $42$ | $8$ | $1000$ |  |
| visual-antmaze | visual-antmaze-medium-v0 | $64\times 64\times 3$ | $8$ | $1000$ |
| visual-antmaze-large-v0 | $64\times 64\times 3$ | $8$ | $1000$ |  |
| visual-antmaze-giant-v0 | $64\times 64\times 3$ | $8$ | $1000$ |  |
| visual-antmaze-teleport-v0 | $64\times 64\times 3$ | $8$ | $1000$ |  |
| visual-humanoidmaze | visual-humanoidmaze-medium-v0 | $64\times 64\times 3$ | $21$ | $2000$ |
| visual-humanoidmaze-large-v0 | $64\times 64\times 3$ | $21$ | $2000$ |  |
| visual-humanoidmaze-giant-v0 | $64\times 64\times 3$ | $21$ | $4000$ |  |
| cube | cube-single-v0 | $28$ | $5$ | $200$ |
| cube-double-v0 | $37$ | $5$ | $500$ |  |
| cube-triple-v0 | $46$ | $5$ | $1000$ |  |
| cube-quadruple-v0 | $55$ | $5$ | $1000$ |  |
| scene | scene-v0 | $40$ | $5$ | $750$ |
| puzzle | puzzle-3x3-v0 | $55$ | $5$ | $500$ |
| puzzle-4x4-v0 | $83$ | $5$ | $500$ |  |
| puzzle-4x5-v0 | $99$ | $5$ | $1000$ |  |
| puzzle-4x6-v0 | $115$ | $5$ | $1000$ |  |
| visual-cube | visual-cube-single-v0 | $64\times 64\times 3$ | $5$ | $200$ |
| visual-cube-double-v0 | $64\times 64\times 3$ | $5$ | $500$ |  |
| visual-cube-triple-v0 | $64\times 64\times 3$ | $5$ | $1000$ |  |
| visual-cube-quadruple-v0 | $64\times 64\times 3$ | $5$ | $1000$ |  |
| visual-scene | visual-scene-v0 | $64\times 64\times 3$ | $5$ | $750$ |
| visual-puzzle | visual-puzzle-3x3-v0 | $64\times 64\times 3$ | $5$ | $500$ |
| visual-puzzle-4x4-v0 | $64\times 64\times 3$ | $5$ | $500$ |  |
| visual-puzzle-4x5-v0 | $64\times 64\times 3$ | $5$ | $1000$ |  |
| visual-puzzle-4x6-v0 | $64\times 64\times 3$ | $5$ | $1000$ |  |
| powderworld | powderworld-easy-v0 | $32\times 32\times 6$ | $8$ (discrete) | $500$ |
| powderworld-medium-v0 | $32\times 32\times 6$ | $8$ (discrete) | $500$ |  |
| powderworld-hard-v0 | $32\times 32\times 6$ | $8$ (discrete) | $500$ |  |

**Table 8: Dataset specifications.**
| Environment Type | Dataset Type | Dataset | # Transitions | # Episodes | Data Episode Length |
| --- | --- | --- | --- | --- | --- |
| pointmaze | navigate | pointmaze-medium-navigate-v0 | $1$M | $1000$ | $1000$ |
| pointmaze-large-navigate-v0 | $1$M | $1000$ | $1000$ |  |  |
| pointmaze-giant-navigate-v0 | $1$M | $500$ | $2000$ |  |  |
| pointmaze-teleport-navigate-v0 | $1$M | $1000$ | $1000$ |  |  |
| stitch | pointmaze-medium-stitch-v0 | $1$M | $5000$ | $200$ |  |
| pointmaze-large-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| pointmaze-giant-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| pointmaze-teleport-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| antmaze | navigate | antmaze-medium-navigate-v0 | $1$M | $1000$ | $1000$ |
| antmaze-large-navigate-v0 | $1$M | $1000$ | $1000$ |  |  |
| antmaze-giant-navigate-v0 | $1$M | $500$ | $2000$ |  |  |
| antmaze-teleport-navigate-v0 | $1$M | $1000$ | $1000$ |  |  |
| stitch | antmaze-medium-stitch-v0 | $1$M | $5000$ | $200$ |  |
| antmaze-large-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| antmaze-giant-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| antmaze-teleport-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| explore | antmaze-medium-explore-v0 | $5$M | $10000$ | $500$ |  |
| antmaze-large-explore-v0 | $5$M | $10000$ | $500$ |  |  |
| antmaze-teleport-explore-v0 | $5$M | $10000$ | $500$ |  |  |
| humanoidmaze | navigate | humanoidmaze-medium-navigate-v0 | $2$M | $1000$ | $2000$ |
| humanoidmaze-large-navigate-v0 | $2$M | $1000$ | $2000$ |  |  |
| humanoidmaze-giant-navigate-v0 | $4$M | $1000$ | $4000$ |  |  |
| stitch | humanoidmaze-medium-stitch-v0 | $2$M | $5000$ | $400$ |  |
| humanoidmaze-large-stitch-v0 | $2$M | $5000$ | $400$ |  |  |
| humanoidmaze-giant-stitch-v0 | $4$M | $10000$ | $400$ |  |  |
| antsoccer | navigate | antsoccer-arena-navigate-v0 | $1$M | $1000$ | $1000$ |
| antsoccer-medium-navigate-v0 | $4$M | $4000$ | $1000$ |  |  |
| stitch | antsoccer-arena-stitch-v0 | $1$M | $5000$ | $200$ |  |
| antsoccer-medium-stitch-v0 | $4$M | $8000$ | $500$ |  |  |
| visual-antmaze | navigate | visual-antmaze-medium-navigate-v0 | $1$M | $1000$ | $1000$ |
| visual-antmaze-large-navigate-v0 | $1$M | $1000$ | $1000$ |  |  |
| visual-antmaze-giant-navigate-v0 | $1$M | $500$ | $2000$ |  |  |
| visual-antmaze-teleport-navigate-v0 | $1$M | $1000$ | $1000$ |  |  |
| stitch | visual-antmaze-medium-stitch-v0 | $1$M | $5000$ | $200$ |  |
| visual-antmaze-large-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| visual-antmaze-giant-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| visual-antmaze-teleport-stitch-v0 | $1$M | $5000$ | $200$ |  |  |
| explore | visual-antmaze-medium-explore-v0 | $5$M | $10000$ | $500$ |  |
| visual-antmaze-large-explore-v0 | $5$M | $10000$ | $500$ |  |  |
| visual-antmaze-teleport-explore-v0 | $5$M | $10000$ | $500$ |  |  |
| visual-humanoidmaze | navigate | visual-humanoidmaze-medium-navigate-v0 | $2$M | $1000$ | $2000$ |
| visual-humanoidmaze-large-navigate-v0 | $2$M | $1000$ | $2000$ |  |  |
| visual-humanoidmaze-giant-navigate-v0 | $4$M | $1000$ | $4000$ |  |  |
| stitch | visual-humanoidmaze-medium-stitch-v0 | $2$M | $5000$ | $400$ |  |
| visual-humanoidmaze-large-stitch-v0 | $2$M | $5000$ | $400$ |  |  |
| visual-humanoidmaze-giant-stitch-v0 | $4$M | $10000$ | $400$ |  |  |
| cube | play | cube-single-play-v0 | $1$M | $1000$ | $1000$ |
| cube-double-play-v0 | $1$M | $1000$ | $1000$ |  |  |
| cube-triple-play-v0 | $3$M | $3000$ | $1000$ |  |  |
| cube-quadruple-play-v0 | $5$M | $5000$ | $1000$ |  |  |
| noisy | cube-single-noisy-v0 | $1$M | $1000$ | $1000$ |  |
| cube-double-noisy-v0 | $1$M | $1000$ | $1000$ |  |  |
| cube-triple-noisy-v0 | $3$M | $3000$ | $1000$ |  |  |
| cube-quadruple-noisy-v0 | $5$M | $5000$ | $1000$ |  |  |
| scene | play | scene-play-v0 | $1$M | $1000$ | $1000$ |
| noisy | scene-noisy-v0 | $1$M | $1000$ | $1000$ |  |
| puzzle | play | puzzle-3x3-play-v0 | $1$M | $1000$ | $1000$ |
| puzzle-4x4-play-v0 | $1$M | $1000$ | $1000$ |  |  |
| puzzle-4x5-play-v0 | $3$M | $3000$ | $1000$ |  |  |
| puzzle-4x6-play-v0 | $5$M | $5000$ | $1000$ |  |  |
| noisy | puzzle-3x3-noisy-v0 | $1$M | $1000$ | $1000$ |  |
| puzzle-4x4-noisy-v0 | $1$M | $1000$ | $1000$ |  |  |
| puzzle-4x5-noisy-v0 | $3$M | $3000$ | $1000$ |  |  |
| puzzle-4x6-noisy-v0 | $5$M | $5000$ | $1000$ |  |  |
| visual-cube | play | visual-cube-single-play-v0 | $1$M | $1000$ | $1000$ |
| visual-cube-double-play-v0 | $1$M | $1000$ | $1000$ |  |  |
| visual-cube-triple-play-v0 | $3$M | $3000$ | $1000$ |  |  |
| visual-cube-quadruple-play-v0 | $5$M | $5000$ | $1000$ |  |  |
| noisy | visual-cube-single-noisy-v0 | $1$M | $1000$ | $1000$ |  |
| visual-cube-double-noisy-v0 | $1$M | $1000$ | $1000$ |  |  |
| visual-cube-triple-noisy-v0 | $3$M | $3000$ | $1000$ |  |  |
| visual-cube-quadruple-noisy-v0 | $5$M | $5000$ | $1000$ |  |  |
| visual-scene | play | visual-scene-play-v0 | $1$M | $1000$ | $1000$ |
| noisy | visual-scene-noisy-v0 | $1$M | $1000$ | $1000$ |  |
| visual-puzzle | play | visual-puzzle-3x3-play-v0 | $1$M | $1000$ | $1000$ |
| visual-puzzle-4x4-play-v0 | $1$M | $1000$ | $1000$ |  |  |
| visual-puzzle-4x5-play-v0 | $3$M | $3000$ | $1000$ |  |  |
| visual-puzzle-4x6-play-v0 | $5$M | $5000$ | $1000$ |  |  |
| noisy | visual-puzzle-3x3-noisy-v0 | $1$M | $1000$ | $1000$ |  |
| visual-puzzle-4x4-noisy-v0 | $1$M | $1000$ | $1000$ |  |  |
| visual-puzzle-4x5-noisy-v0 | $3$M | $3000$ | $1000$ |  |  |
| visual-puzzle-4x6-noisy-v0 | $5$M | $5000$ | $1000$ |  |  |
| powderworld | play | powderworld-easy-play-v0 | $1$M | $1000$ | $1000$ |
| powderworld-medium-play-v0 | $3$M | $3000$ | $1000$ |  |  |
| powderworld-hard-play-v0 | $5$M | $5000$ | $1000$ |  |  |

### D.1 Tasks and Datasets

In this section, we provide further information about our tasks and datasets.
We provide the basic specifications about the environments and datasets in Tables 7 and 8.

Locomotion tasks.
For OGBench AntMaze and AntSoccer,
we adopt the Ant model from D4RL AntMaze (Fu et al., )
(which is based on the Ant in OpenAI Gym (Brockman et al., ; Towers et al., ), but with a more restricted joint range)
and the soccer ball model from the DeepMind Control suite (Tassa et al., ).
For HumanoidMaze, we adopt the Humanoid model from the DeepMind Control suite (Tassa et al., ).

To collect datasets,
we train expert low-level directional (AntMaze and HumanoidMaze) or goal-reaching (AntSoccer) policies
using SAC with dense reward functions for $400$K (AntMaze), $40$M (HumanoidMaze), or $12$M (AntSoccer) steps.
For PointMaze, we use a scripted directional expert policy.
When collecting datasets, we add Gaussian noise with a standard deviation of
$0.5$ (pointmaze),
$1.0$ (explore), or $0.2$ (others) to expert actions.

The success criteria for the locomotion tasks are based only on the distance between the agent (or the ball in AntSoccer)
and the goal location. In particular, the tasks do not consider joint positions determining success,
as in previous works (Fu et al., ; Park et al., ).

Manipulation tasks.
We adopt the UR5e robot arm and Robotiq 2F-85 gripper models from MuJoCo Menagerie (Zakka et al., ),
and the drawer, window, and button box models from Meta-World (Yu et al., ).
The robot is end-effector controlled with a $5$-D action space,
where the dimensions correspond to the displacements in
the $x$ position, $y$ position, $z$ position, gripper yaw, and gripper opening.
In all manipulation tasks, we place invisible walls to prevent objects from moving into an area beyond the robot arm’s reach.
These invisible walls also prevent the cube objects from moving outside the camera viewpoint in pixel-based manipulation environments.
However, since some blind spots still exist in visual-scene even with the walls,
we further filter out such rare cases in trajectories prevent ambiguous camera observations completely.

In Puzzle, not every button configuration is reachable from the initial state.
While the $3\times 3$, $4\times 5$, and $4\times 6$ puzzles do have this property,
the $4\times 4$ puzzle does not.
This can be seen by computing the rank of the $nm\times nm$ button effect matrix over ${\mathbb{F}}_{2}$ (the field with two elements),
where $n$ and $m$ denote the numbers of rows and columns, respectively.
In our tasks, we ensure that every test-time goal is solvable.
Also, we note that the maximum value of the minimum number of button presses to reach a state from another state in each puzzle
is $9$ ($3\times 3$ puzzle), $7$ ($4\times 4$ puzzle), $20$ ($4\times 5$ puzzle), or $24$ ($4\times 6$ puzzle).
Each puzzle environment contains at least one evaluation goal that requires the maximum number of presses.

The play datasets are collected by open-loop, non-Markovian scripted policies,
and the noisy datasets are collected by closed-loop, Markovian scripted policies.
For the play datasets,
we add temporally correlated action noise to the expert actions to enhance state coverage.
For the noisy datasets,
we first sample the degree of action noise at the beginning of each episode,
and collect a trajectory with the chosen amount of (time-independent) Gaussian action noise.
This ensures high coverage while having a sufficient number of optimal trajectories.

The success criteria for the manipulation tasks are based only on the object configurations;
the arm pose is not considered when determining success.
For cubes, only the distances between the goal positions and their current positions are considered,
and their orientations are ignored.

Drawing tasks.
We modify the original Powderworld environment (Frans & Isola, ) to make it offline and goal-conditioned.
We also re-implement Powderworld (which was originally implemented in PyTorch) in NumPy to remove the dependency on PyTorch.
We provide three versions of Powderworld tasks:
powderworld-easy uses two elements (plant and stone),
powderworld-medium uses five elements (sand, water, fire, plant, and stone),
and powderworld-hard uses eight elements (sand, water, fire, plant, stone, gas, wood, and ice).
An action in Powderworld corresponds to drawing a $4\times 4$-sized square with a specific element brush
on the $32\times 32$-sized board.
Since naïvely implementing this atomic action requires up to $512$-dimensional discrete actions,
we split it into three *sequential* $8$-dimensional actions that correspond to
element selection, $x$-coordinate selection, and $y$-coordinate selection.
To ensure full observability,
we add three additional dimensions that contain information
about the currently selected element and $x$ coordinate to the original $32\times 32\times 3$-dimensional image,
which results in a $32\times 32\times 6$-dimensional observation space.
When the agent selects an invalid action
(which can only happen in powderworld-{easy, medium}, which has fewer than $8$ elements),
the environment instead uses a randomly sampled valid action.

The datasets are collected by a scripted policy that randomly draws squares and lines
or fills the entire board with randomly selected brushes.
With a probability of $0.5$, it performs a random action (*i.e.*, places a random element on a randomly sampled position).

For the success criterion for evaluation goals, we use the following procedure to allow for some tolerance:
For each pixel in the goal image, we check if the current image has a matching pixel that is shifted by up to one pixel in any direction.
We then compute the error as the number of pixels that do not match, and consider the task successful if the error is below a certain threshold.

### D.2 Methods

Our implementations of six offline GCRL algorithms (GCBC, GCIVL, GCIQL, QRL, CRL, and HIQL) are based on JAX (Bradbury et al., ).
In our benchmark, each run typically takes $2$-$5$ hours (state-based tasks) or $5$-$12$ hours (pixel-based tasks) on an A5000 GPU,
depending on the task and algorithm.

For benchmarking, we periodically evaluate the performance (goal success rate in percentage) of each agent on each test-time goal
with $50$ rollouts every $100$K steps,
and report the average success rate across the last three evaluation epochs
(*i.e.*, at $800$K, $900$K, and $1$M steps for state-based tasks and at $300$K, $400$K, and $500$K steps for pixel-based tasks).
That is, the performance of each agent is averaged over $750$ rollouts
($3$ evaluation epochs $\times$ $5$ test-time goals $\times$ $50$ rollouts).
While we use a relatively large number of evaluation rollouts for robustness,
researchers can adjust the number of evaluation rollouts (*e.g.*, $20$ episodes for each test-time goal)
to reduce the computational burden.

We provide the full list of common hyperparameters in Table 9.
We find that methods are more sensitive to policy extraction hyperparameters (*e.g.*, the BC coefficient in DDPG+BC) (Park et al., ),
and report these in a separate table (Table 10).
Specifically, for each method,
we use the same value learning hyperparameters across the benchmark except for the discount factor $\gamma$ (Table 9),
but individually tune the policy extraction hyperparameters
(*e.g.*, AWR $\alpha$ and DDPG+BC $\alpha$)
for each dataset category (Tables 9 and 10).

We apply layer normalization (Ba et al., ) to the value networks, but not to the policy networks.
In pixel-based environments, we use a smaller version of the IMPALA encoder (Espeholt et al., ).
We use random-crop image augmentation (with a probability of $0.5$) for pixel-based manipulation tasks, but not for pixel-based locomotion or drawing tasks,
as we find it to be helpful mainly on manipulation tasks.
In pixel-based environments, we do not apply frame stacking for simplicity,
as we find it does not necessarily improve performance on most tasks including Visual AntMaze
(although we believe the performance on Visual HumanoidMaze can further be improved with frame stacking).
For policies, we parameterize the action distribution with a unit-variance Gaussian distribution.
We find that using a fixed standard deviation is especially important for DDPG+BC.
During evaluation, we use the deterministic mean of the learned Gaussian policy.
However, in Powderworld, which has a discrete action space, we use a stochastic policy with a temperature of $0.3$
(*i.e.*, we divide the action logits by $0.3$),
as this additional stochasticity helps prevent the agent from getting stuck in certain states.

**Table 9: Common hyperparameters.**
| Hyperparameter | Value |
| --- | --- |
| Learning rate | $0.0003$ |
| Optimizer | Adam (Kingma & Ba, ) |
| # gradient steps | $1000000$ (states), $500000$ (pixels) |
| Minibatch size | $1024$ (states), $256$ (pixels) |
| MLP dimensions | $(512,512,512)$ |
| Nonlinearity | GELU (Hendrycks & Gimpel, ) |
| Target smoothing coefficient | $0.005$ |
| Discount factor $\gamma$ | $0.995$ ({antmaze, pointmaze}-giant, humanoidmaze), $0.99$ (others) |
| Image augmentation probability | $0.5$ (pixel-based manipulation), 0 0 (others) |
| GCIVL/GCIQL expectile $\kappa$ | $0.9$ |
| GCIQL expectile $\kappa$ | $0.9$ |
| QRL quasimetric | IQE (Wang & Isola, ) |
| QRL latent dimension | $512$ ($64$ components $\times$ $8$-dimensional latents) |
| QRL margin $\epsilon$ | $0.05$ |
| CRL latent dimension | $512$ |
| HIQL expectile $\kappa$ | $0.7$ |
| HIQL subgoal step $k$ | $100$ (humanoidmaze), $25$ (other locomotion), $10$ (others) |
| HIQL subgoal representation dimension | $10$ |
| Policy $(p^{\mathcal{D}}_{\mathrm{cur}},p^{\mathcal{D}}_{\mathrm{traj}},p^{\mathcal{D}<br>}_{\mathrm{geom}},p^{\mathcal{D}}_{\mathrm{rand}})$ ratio for $p^{\mathcal{D}}_{\mathrm{mixed}}$ | $(0,0.5,0,0.5)$ (stitch), $(0,0,0,1)$ (explore), $(0,1,0,0)$ (others) |
| Value $(p^{\mathcal{D}}_{\mathrm{cur}},p^{\mathcal{D}}_{\mathrm{traj}},p^{\mathcal{D}<br>}_{\mathrm{geom}},p^{\mathcal{D}}_{\mathrm{rand}})$ ratio for $p^{\mathcal{D}}_{\mathrm{mixed}}$ | $(0.2,0,0.5,0.3)$ |

**Table 10: Hyperparameters for policy extraction. Each cell indicates the policy extraction method and its $\alpha$ value (*i.e.*, the temperature (AWR) or the BC coefficient (DDPG+BC)).**
| Environment Type | Dataset Type | Dataset | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- |
| pointmaze | navigate | pointmaze-medium-navigate-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |
| pointmaze-large-navigate-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |  |  |
| pointmaze-giant-navigate-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |  |  |
| pointmaze-teleport-navigate-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |  |  |
| stitch | pointmaze-medium-stitch-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |  |
| pointmaze-large-stitch-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |  |  |
| pointmaze-giant-stitch-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |  |  |
| pointmaze-teleport-stitch-v0 | AWR $10.0$ | DDPG $0.003$ | DDPG $0.0003$ | DDPG $0.03$ | AWR $3.0$ |  |  |
| antmaze | navigate | antmaze-medium-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |
| antmaze-large-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| antmaze-giant-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| antmaze-teleport-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| stitch | antmaze-medium-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |
| antmaze-large-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| antmaze-giant-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| antmaze-teleport-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| explore | antmaze-medium-explore-v0 | AWR $10.0$ | DDPG $0.01$ | DDPG $0.001$ | DDPG $0.003$ | AWR $10.0$ |  |
| antmaze-large-explore-v0 | AWR $10.0$ | DDPG $0.01$ | DDPG $0.001$ | DDPG $0.003$ | AWR $10.0$ |  |  |
| antmaze-teleport-explore-v0 | AWR $10.0$ | DDPG $0.01$ | DDPG $0.001$ | DDPG $0.003$ | AWR $10.0$ |  |  |
| humanoidmaze | navigate | humanoidmaze-medium-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |
| humanoidmaze-large-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| humanoidmaze-giant-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| stitch | humanoidmaze-medium-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |
| humanoidmaze-large-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| humanoidmaze-giant-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| antsoccer | navigate | antsoccer-arena-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.003$ | DDPG $0.3$ | AWR $3.0$ |
| antsoccer-medium-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.003$ | DDPG $0.3$ | AWR $3.0$ |  |  |
| stitch | antsoccer-arena-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.003$ | DDPG $0.3$ | AWR $3.0$ |  |
| antsoccer-medium-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.003$ | DDPG $0.3$ | AWR $3.0$ |  |  |
| visual-antmaze | navigate | visual-antmaze-medium-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |
| visual-antmaze-large-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-antmaze-giant-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-antmaze-teleport-navigate-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| stitch | visual-antmaze-medium-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |
| visual-antmaze-large-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-antmaze-giant-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-antmaze-teleport-stitch-v0 | AWR $10.0$ | DDPG $0.3$ | DDPG $0.003$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| explore | visual-antmaze-medium-explore-v0 | AWR $10.0$ | DDPG $0.01$ | DDPG $0.001$ | DDPG $0.003$ | AWR $10.0$ |  |
| visual-antmaze-large-explore-v0 | AWR $10.0$ | DDPG $0.01$ | DDPG $0.001$ | DDPG $0.003$ | AWR $10.0$ |  |  |
| visual-antmaze-teleport-explore-v0 | AWR $10.0$ | DDPG $0.01$ | DDPG $0.001$ | DDPG $0.003$ | AWR $10.0$ |  |  |
| visual-humanoidmaze | navigate | visual-humanoidmaze-medium-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |
| visual-humanoidmaze-large-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-humanoidmaze-giant-navigate-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| stitch | visual-humanoidmaze-medium-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |
| visual-humanoidmaze-large-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-humanoidmaze-giant-stitch-v0 | AWR $10.0$ | DDPG $0.1$ | DDPG $0.001$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| cube | play | cube-single-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |
| cube-double-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| cube-triple-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| cube-quadruple-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| noisy | cube-single-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |
| cube-double-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| cube-triple-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| cube-quadruple-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| scene | play | scene-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |
| noisy | scene-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |
| puzzle | play | puzzle-3x3-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |
| puzzle-4x4-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| puzzle-4x5-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| puzzle-4x6-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| noisy | puzzle-3x3-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |
| puzzle-4x4-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| puzzle-4x5-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| puzzle-4x6-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-cube | play | visual-cube-single-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |
| visual-cube-double-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| visual-cube-triple-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| visual-cube-quadruple-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| noisy | visual-cube-single-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |
| visual-cube-double-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-cube-triple-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-cube-quadruple-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-scene | play | visual-scene-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |
| noisy | visual-scene-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |
| visual-puzzle | play | visual-puzzle-3x3-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |
| visual-puzzle-4x4-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| visual-puzzle-4x5-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| visual-puzzle-4x6-play-v0 | AWR $10.0$ | DDPG $1.0$ | DDPG $0.3$ | DDPG $3.0$ | AWR $3.0$ |  |  |
| noisy | visual-puzzle-3x3-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |
| visual-puzzle-4x4-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-puzzle-4x5-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| visual-puzzle-4x6-noisy-v0 | AWR $10.0$ | DDPG $0.03$ | DDPG $0.03$ | DDPG $0.1$ | AWR $3.0$ |  |  |
| powderworld | play | powderworld-easy-play-v0 | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ |
| powderworld-medium-play-v0 | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ |  |  |
| powderworld-hard-play-v0 | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ | AWR $3.0$ |  |  |

## Appendix E Evaluation Goals and Per-Goal Benchmarking Results

Each task in OGBench provides five evaluation goals.
We provide their full image descriptions in Figures 4, 5, 6, 7, 8, 9 and 10,
and the full per-goal evaluation results in Tables 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22 and 23,
which share the same format as Table 2.

Figure: task1
Refer to caption: extracted/5955563/figures/goal/antmaze-medium-v0_task1_goal.png

Figure: task1
Refer to caption: extracted/5955563/figures/goal/antsoccer-arena-v0_task1_goal.png

Figure: task1 horizontal
Refer to caption: extracted/5955563/figures/goal/cube-single-v0_task1_goal.png

Figure: task1 open
Refer to caption: extracted/5955563/figures/goal/scene-v0_task1_goal.png

Figure: task1
Refer to caption: extracted/5955563/figures/goal/puzzle-3x3-v0_task1_goal.png

Figure: task1
Refer to caption: extracted/5955563/figures/goal/puzzle-4x5-v0_task1_goal.png

Figure: task1 plant
Refer to caption: extracted/5955563/figures/goal/powderworld-easy-v0_task1_goal.png

**Table 11: Full results on PointMaze.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| pointmaze | navigate | pointmaze-medium-navigate-v0 | task1 | $30$ $\pm 27$ | $88$ $\pm 16$ | $\mathbf{97}$ $\pm 4$ | $\mathbf{100}$ $\pm 0$ | $20$ $\pm 6$ | $\mathbf{99}$ $\pm 1$ |
| task2 | $3$ $\pm 2$ | $\mathbf{95}$ $\pm 10$ | $76$ $\pm 29$ | $\mathbf{94}$ $\pm 17$ | $45$ $\pm 25$ | $87$ $\pm 7$ |  |  |  |
| task3 | $5$ $\pm 5$ | $37$ $\pm 28$ | $10$ $\pm 28$ | $23$ $\pm 20$ | $30$ $\pm 4$ | $\mathbf{55}$ $\pm 13$ |  |  |  |
| task4 | 0 0 $\pm 1$ | $2$ $\pm 2$ | 0 0 $\pm 0$ | $\mathbf{94}$ $\pm 14$ | $28$ $\pm 29$ | $82$ $\pm 12$ |  |  |  |
| task5 | $4$ $\pm 3$ | $92$ $\pm 7$ | $79$ $\pm 6$ | $\mathbf{97}$ $\pm 8$ | $24$ $\pm 13$ | $70$ $\pm 10$ |  |  |  |
| overall | $9$ $\pm 6$ | $63$ $\pm 6$ | $53$ $\pm 8$ | $\mathbf{82}$ $\pm 5$ | $29$ $\pm 7$ | $\mathbf{79}$ $\pm 5$ |  |  |  |
| pointmaze-large-navigate-v0 | task1 | $63$ $\pm 11$ | $76$ $\pm 23$ | $86$ $\pm 14$ | $\mathbf{95}$ $\pm 8$ | $42$ $\pm 27$ | $83$ $\pm 13$ |  |  |
| task2 | $1$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{100}$ $\pm 0$ | $31$ $\pm 24$ | $2$ $\pm 7$ |  |  |  |
| task3 | $10$ $\pm 7$ | $\mathbf{98}$ $\pm 5$ | $83$ $\pm 8$ | $40$ $\pm 50$ | $78$ $\pm 7$ | $88$ $\pm 10$ |  |  |  |
| task4 | $20$ $\pm 18$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{96}$ $\pm 7$ | $24$ $\pm 14$ | $72$ $\pm 19$ |  |  |  |
| task5 | $52$ $\pm 17$ | $53$ $\pm 20$ | 0 0 $\pm 0$ | $\mathbf{96}$ $\pm 7$ | $20$ $\pm 10$ | $46$ $\pm 16$ |  |  |  |
| overall | $29$ $\pm 6$ | $45$ $\pm 5$ | $34$ $\pm 3$ | $\mathbf{86}$ $\pm 9$ | $39$ $\pm 7$ | $58$ $\pm 5$ |  |  |  |
| pointmaze-giant-navigate-v0 | task1 | $1$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{98}$ $\pm 7$ | $6$ $\pm 15$ | 0 0 $\pm 0$ |  |  |
| task2 | $1$ $\pm 4$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{92}$ $\pm 16$ | $28$ $\pm 10$ | $72$ $\pm 17$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{68}$ $\pm 27$ | $9$ $\pm 5$ | $32$ $\pm 11$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{66}$ $\pm 20$ | $\mathbf{64}$ $\pm 17$ | $60$ $\pm 22$ |  |  |  |
| task5 | $5$ $\pm 12$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $19$ $\pm 32$ | $29$ $\pm 28$ | $\mathbf{66}$ $\pm 20$ |  |  |  |
| overall | $1$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{68}$ $\pm 7$ | $27$ $\pm 10$ | $46$ $\pm 9$ |  |  |  |
| pointmaze-teleport-navigate-v0 | task1 | $1$ $\pm 2$ | $\mathbf{33}$ $\pm 12$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 3$ | $5$ $\pm 5$ |  |  |
| task2 | $4$ $\pm 6$ | $\mathbf{49}$ $\pm 2$ | $39$ $\pm 14$ | $8$ $\pm 10$ | $30$ $\pm 23$ | $6$ $\pm 6$ |  |  |  |
| task3 | $\mathbf{50}$ $\pm 4$ | $46$ $\pm 5$ | $31$ $\pm 19$ | $2$ $\pm 5$ | $26$ $\pm 6$ | $39$ $\pm 9$ |  |  |  |
| task4 | $33$ $\pm 13$ | $\mathbf{49}$ $\pm 4$ | $42$ $\pm 13$ | $12$ $\pm 16$ | $40$ $\pm 11$ | $24$ $\pm 11$ |  |  |  |
| task5 | $38$ $\pm 6$ | $\mathbf{48}$ $\pm 4$ | $9$ $\pm 9$ | $1$ $\pm 2$ | $20$ $\pm 15$ | $17$ $\pm 8$ |  |  |  |
| overall | $25$ $\pm 3$ | $\mathbf{45}$ $\pm 3$ | $24$ $\pm 7$ | $4$ $\pm 4$ | $24$ $\pm 6$ | $18$ $\pm 4$ |  |  |  |
| stitch | pointmaze-medium-stitch-v0 | task1 | $21$ $\pm 29$ | $76$ $\pm 14$ | $56$ $\pm 24$ | $\mathbf{94}$ $\pm 13$ | 0 0 $\pm 0$ | $77$ $\pm 14$ |  |
| task2 | $32$ $\pm 35$ | $\mathbf{79}$ $\pm 23$ | $26$ $\pm 19$ | $\mathbf{81}$ $\pm 34$ | 0 0 $\pm 0$ | $61$ $\pm 23$ |  |  |  |
| task3 | $33$ $\pm 34$ | $69$ $\pm 16$ | 0 0 $\pm 0$ | $66$ $\pm 29$ | $2$ $\pm 3$ | $\mathbf{82}$ $\pm 13$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $41$ $\pm 37$ | 0 0 $\pm 0$ | $68$ $\pm 32$ | 0 0 $\pm 0$ | $\mathbf{92}$ $\pm 6$ |  |  |  |
| task5 | $29$ $\pm 37$ | $84$ $\pm 11$ | $22$ $\pm 22$ | $\mathbf{92}$ $\pm 9$ | 0 0 $\pm 0$ | $59$ $\pm 9$ |  |  |  |
| overall | $23$ $\pm 18$ | $70$ $\pm 14$ | $21$ $\pm 9$ | $\mathbf{80}$ $\pm 12$ | 0 0 $\pm 1$ | $74$ $\pm 6$ |  |  |  |
| pointmaze-large-stitch-v0 | task1 | $8$ $\pm 13$ | 0 0 $\pm 1$ | $56$ $\pm 11$ | $\mathbf{100}$ $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{74}$ $\pm 37$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| task3 | $26$ $\pm 28$ | $60$ $\pm 29$ | $\mathbf{98}$ $\pm 4$ | $74$ $\pm 23$ | 0 0 $\pm 0$ | $59$ $\pm 25$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{88}$ $\pm 32$ | 0 0 $\pm 0$ | $1$ $\pm 4$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{85}$ $\pm 22$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| overall | $7$ $\pm 5$ | $12$ $\pm 6$ | $31$ $\pm 2$ | $\mathbf{84}$ $\pm 15$ | 0 0 $\pm 0$ | $13$ $\pm 6$ |  |  |  |
| pointmaze-giant-stitch-v0 | task1 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{99}$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{80}$ $\pm 27$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 5$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{63}$ $\pm 23$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 8$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| overall | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{50}$ $\pm 8$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| pointmaze-teleport-stitch-v0 | task1 | $28$ $\pm 20$ | $\mathbf{34}$ $\pm 14$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $24$ $\pm 13$ |  |  |
| task2 | $13$ $\pm 15$ | $\mathbf{41}$ $\pm 8$ | $12$ $\pm 14$ | $7$ $\pm 7$ | 0 0 $\pm 0$ | $23$ $\pm 11$ |  |  |  |
| task3 | $\mathbf{48}$ $\pm 8$ | $\mathbf{50}$ $\pm 5$ | $47$ $\pm 2$ | $15$ $\pm 13$ | 0 0 $\pm 0$ | $46$ $\pm 9$ |  |  |  |
| task4 | $40$ $\pm 16$ | $\mathbf{50}$ $\pm 6$ | $46$ $\pm 5$ | $19$ $\pm 12$ | $8$ $\pm 8$ | $46$ $\pm 5$ |  |  |  |
| task5 | $29$ $\pm 16$ | $\mathbf{48}$ $\pm 5$ | $21$ $\pm 7$ | $1$ $\pm 3$ | $13$ $\pm 13$ | $31$ $\pm 10$ |  |  |  |
| overall | $31$ $\pm 9$ | $\mathbf{44}$ $\pm 2$ | $25$ $\pm 3$ | $9$ $\pm 5$ | $4$ $\pm 3$ | $34$ $\pm 4$ |  |  |  |

**Table 12: Full results on AntMaze.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| antmaze | navigate | antmaze-medium-navigate-v0 | task1 | $35$ $\pm 9$ | $81$ $\pm 10$ | $63$ $\pm 9$ | $\mathbf{93}$ $\pm 2$ | $\mathbf{97}$ $\pm 1$ | $\mathbf{94}$ $\pm 2$ |
| task2 | $21$ $\pm 7$ | $85$ $\pm 5$ | $78$ $\pm 8$ | $90$ $\pm 5$ | $\mathbf{95}$ $\pm 2$ | $\mathbf{97}$ $\pm 1$ |  |  |  |
| task3 | $24$ $\pm 6$ | $60$ $\pm 13$ | $71$ $\pm 8$ | $86$ $\pm 6$ | $\mathbf{92}$ $\pm 3$ | $\mathbf{96}$ $\pm 2$ |  |  |  |
| task4 | $28$ $\pm 7$ | $42$ $\pm 25$ | $59$ $\pm 12$ | $83$ $\pm 4$ | $\mathbf{94}$ $\pm 5$ | $\mathbf{96}$ $\pm 2$ |  |  |  |
| task5 | $37$ $\pm 10$ | $\mathbf{92}$ $\pm 3$ | $85$ $\pm 7$ | $88$ $\pm 8$ | $\mathbf{96}$ $\pm 2$ | $\mathbf{96}$ $\pm 2$ |  |  |  |
| overall | $29$ $\pm 4$ | $72$ $\pm 8$ | $71$ $\pm 4$ | $88$ $\pm 3$ | $\mathbf{95}$ $\pm 1$ | $\mathbf{96}$ $\pm 1$ |  |  |  |
| antmaze-large-navigate-v0 | task1 | $6$ $\pm 3$ | $16$ $\pm 12$ | $21$ $\pm 6$ | $71$ $\pm 15$ | $\mathbf{91}$ $\pm 3$ | $\mathbf{93}$ $\pm 3$ |  |  |
| task2 | $16$ $\pm 4$ | $5$ $\pm 6$ | $25$ $\pm 7$ | $\mathbf{77}$ $\pm 7$ | $62$ $\pm 14$ | $\mathbf{78}$ $\pm 9$ |  |  |  |
| task3 | $65$ $\pm 4$ | $49$ $\pm 18$ | $80$ $\pm 5$ | $\mathbf{94}$ $\pm 2$ | $91$ $\pm 2$ | $\mathbf{96}$ $\pm 2$ |  |  |  |
| task4 | $14$ $\pm 3$ | $2$ $\pm 2$ | $19$ $\pm 6$ | $64$ $\pm 8$ | $85$ $\pm 11$ | $\mathbf{94}$ $\pm 2$ |  |  |  |
| task5 | $18$ $\pm 4$ | $5$ $\pm 2$ | $26$ $\pm 9$ | $67$ $\pm 9$ | $85$ $\pm 3$ | $\mathbf{94}$ $\pm 3$ |  |  |  |
| overall | $24$ $\pm 2$ | $16$ $\pm 5$ | $34$ $\pm 4$ | $75$ $\pm 6$ | $83$ $\pm 4$ | $\mathbf{91}$ $\pm 2$ |  |  |  |
| antmaze-giant-navigate-v0 | task1 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 2$ | $2$ $\pm 2$ | $\mathbf{47}$ $\pm 10$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $17$ $\pm 5$ | $21$ $\pm 10$ | $\mathbf{74}$ $\pm 5$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $14$ $\pm 8$ | $5$ $\pm 5$ | $\mathbf{55}$ $\pm 7$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $18$ $\pm 6$ | $35$ $\pm 9$ | $\mathbf{69}$ $\pm 5$ |  |  |  |
| task5 | $1$ $\pm 1$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $18$ $\pm 5$ | $16$ $\pm 10$ | $\mathbf{82}$ $\pm 4$ |  |  |  |
| overall | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $14$ $\pm 3$ | $16$ $\pm 3$ | $\mathbf{65}$ $\pm 5$ |  |  |  |
| antmaze-teleport-navigate-v0 | task1 | $17$ $\pm 5$ | $35$ $\pm 5$ | $26$ $\pm 5$ | $31$ $\pm 6$ | $35$ $\pm 5$ | $\mathbf{37}$ $\pm 5$ |  |  |
| task2 | $51$ $\pm 5$ | $41$ $\pm 5$ | $58$ $\pm 8$ | $47$ $\pm 22$ | $\mathbf{92}$ $\pm 3$ | $66$ $\pm 8$ |  |  |  |
| task3 | $22$ $\pm 3$ | $36$ $\pm 8$ | $31$ $\pm 5$ | $35$ $\pm 6$ | $\mathbf{47}$ $\pm 4$ | $37$ $\pm 5$ |  |  |  |
| task4 | $25$ $\pm 5$ | $45$ $\pm 3$ | $33$ $\pm 5$ | $33$ $\pm 6$ | $\mathbf{50}$ $\pm 2$ | $30$ $\pm 2$ |  |  |  |
| task5 | $14$ $\pm 6$ | $38$ $\pm 6$ | $26$ $\pm 9$ | $28$ $\pm 8$ | $\mathbf{44}$ $\pm 3$ | $41$ $\pm 8$ |  |  |  |
| overall | $26$ $\pm 3$ | $39$ $\pm 3$ | $35$ $\pm 5$ | $35$ $\pm 5$ | $\mathbf{53}$ $\pm 2$ | $42$ $\pm 3$ |  |  |  |
| stitch | antmaze-medium-stitch-v0 | task1 | $70$ $\pm 33$ | $76$ $\pm 13$ | $17$ $\pm 12$ | $43$ $\pm 20$ | $43$ $\pm 10$ | $\mathbf{92}$ $\pm 2$ |  |
| task2 | $65$ $\pm 19$ | $80$ $\pm 4$ | $22$ $\pm 16$ | $61$ $\pm 12$ | $46$ $\pm 14$ | $\mathbf{94}$ $\pm 3$ |  |  |  |
| task3 | $21$ $\pm 15$ | $16$ $\pm 12$ | $41$ $\pm 9$ | $72$ $\pm 29$ | $46$ $\pm 17$ | $\mathbf{95}$ $\pm 2$ |  |  |  |
| task4 | $1$ $\pm 2$ | 0 0 $\pm 0$ | $32$ $\pm 9$ | $80$ $\pm 9$ | $53$ $\pm 19$ | $\mathbf{93}$ $\pm 2$ |  |  |  |
| task5 | $70$ $\pm 33$ | $47$ $\pm 20$ | $34$ $\pm 14$ | $41$ $\pm 18$ | $75$ $\pm 8$ | $\mathbf{95}$ $\pm 3$ |  |  |  |
| overall | $45$ $\pm 11$ | $44$ $\pm 6$ | $29$ $\pm 6$ | $59$ $\pm 7$ | $53$ $\pm 6$ | $\mathbf{94}$ $\pm 1$ |  |  |  |
| antmaze-large-stitch-v0 | task1 | $2$ $\pm 2$ | $23$ $\pm 9$ | 0 0 $\pm 0$ | $7$ $\pm 5$ | $1$ $\pm 1$ | $\mathbf{85}$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $10$ $\pm 5$ | $4$ $\pm 4$ | $\mathbf{24}$ $\pm 16$ |  |  |  |
| task3 | $15$ $\pm 14$ | $69$ $\pm 6$ | $37$ $\pm 10$ | $73$ $\pm 8$ | $43$ $\pm 11$ | $\mathbf{94}$ $\pm 3$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $5$ $\pm 5$ | $\mathbf{70}$ $\pm 8$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $1$ $\pm 2$ | $\mathbf{60}$ $\pm 9$ |  |  |  |
| overall | $3$ $\pm 3$ | $18$ $\pm 2$ | $7$ $\pm 2$ | $18$ $\pm 2$ | $11$ $\pm 2$ | $\mathbf{67}$ $\pm 5$ |  |  |  |
| antmaze-giant-stitch-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 1$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{5}$ $\pm 5$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 3$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 1$ |  |  |  |
| overall | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ |  |  |  |
| antmaze-teleport-stitch-v0 | task1 | $21$ $\pm 13$ | $39$ $\pm 7$ | $12$ $\pm 4$ | $22$ $\pm 6$ | $30$ $\pm 6$ | $\mathbf{44}$ $\pm 5$ |  |  |
| task2 | $39$ $\pm 12$ | $\mathbf{44}$ $\pm 6$ | $18$ $\pm 7$ | $22$ $\pm 6$ | $30$ $\pm 4$ | $\mathbf{42}$ $\pm 3$ |  |  |  |
| task3 | $34$ $\pm 12$ | $\mathbf{36}$ $\pm 8$ | $18$ $\pm 4$ | $25$ $\pm 7$ | $23$ $\pm 11$ | $26$ $\pm 4$ |  |  |  |
| task4 | $\mathbf{46}$ $\pm 6$ | $\mathbf{44}$ $\pm 4$ | $18$ $\pm 5$ | $24$ $\pm 9$ | $38$ $\pm 4$ | $26$ $\pm 4$ |  |  |  |
| task5 | $16$ $\pm 14$ | $33$ $\pm 6$ | $17$ $\pm 6$ | $26$ $\pm 5$ | $32$ $\pm 7$ | $\mathbf{40}$ $\pm 6$ |  |  |  |
| overall | $31$ $\pm 6$ | $\mathbf{39}$ $\pm 3$ | $17$ $\pm 2$ | $24$ $\pm 5$ | $31$ $\pm 4$ | $36$ $\pm 2$ |  |  |  |
| explore | antmaze-medium-explore-v0 | task1 | $3$ $\pm 6$ | $10$ $\pm 8$ | $12$ $\pm 6$ | $1$ $\pm 1$ | $2$ $\pm 2$ | $\mathbf{29}$ $\pm 17$ |  |
| task2 | $1$ $\pm 2$ | $74$ $\pm 9$ | $53$ $\pm 8$ | $1$ $\pm 1$ | $8$ $\pm 6$ | $\mathbf{84}$ $\pm 10$ |  |  |  |
| task3 | $1$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $3$ $\pm 5$ | $4$ $\pm 6$ | $\mathbf{18}$ $\pm 24$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $3$ $\pm 4$ | $10$ $\pm 6$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $2$ $\pm 2$ | $\mathbf{52}$ $\pm 27$ |  |  |  |
| overall | $2$ $\pm 1$ | $19$ $\pm 3$ | $13$ $\pm 2$ | $1$ $\pm 1$ | $3$ $\pm 2$ | $\mathbf{37}$ $\pm 10$ |  |  |  |
| antmaze-large-explore-v0 | task1 | 0 0 $\pm 0$ | $\mathbf{37}$ $\pm 12$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 1$ | $1$ $\pm 3$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $12$ $\pm 6$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $\mathbf{18}$ $\pm 24$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | 0 0 $\pm 0$ | $\mathbf{10}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $4$ $\pm 5$ |  |  |  |
| antmaze-teleport-explore-v0 | task1 | $2$ $\pm 2$ | $\mathbf{32}$ $\pm 4$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $\mathbf{32}$ $\pm 11$ |  |  |
| task2 | 0 0 $\pm 0$ | $2$ $\pm 4$ | $8$ $\pm 6$ | 0 0 $\pm 1$ | $5$ $\pm 4$ | $\mathbf{33}$ $\pm 17$ |  |  |  |
| task3 | $4$ $\pm 3$ | $\mathbf{48}$ $\pm 3$ | $13$ $\pm 8$ | $4$ $\pm 4$ | $\mathbf{47}$ $\pm 6$ | $34$ $\pm 16$ |  |  |  |
| task4 | $2$ $\pm 2$ | $\mathbf{47}$ $\pm 5$ | $14$ $\pm 8$ | $4$ $\pm 4$ | $16$ $\pm 12$ | $37$ $\pm 19$ |  |  |  |
| task5 | $4$ $\pm 2$ | $31$ $\pm 2$ | $2$ $\pm 1$ | $3$ $\pm 3$ | $28$ $\pm 5$ | $\mathbf{34}$ $\pm 14$ |  |  |  |
| overall | $2$ $\pm 1$ | $32$ $\pm 2$ | $7$ $\pm 3$ | $2$ $\pm 2$ | $20$ $\pm 2$ | $\mathbf{34}$ $\pm 15$ |  |  |  |

**Table 13: Full results on HumanoidMaze.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| humanoidmaze | navigate | humanoidmaze-medium-navigate-v0 | task1 | $4$ $\pm 1$ | $22$ $\pm 5$ | $23$ $\pm 6$ | $12$ $\pm 7$ | $84$ $\pm 3$ | $\mathbf{95}$ $\pm 2$ |
| task2 | $8$ $\pm 4$ | $42$ $\pm 8$ | $49$ $\pm 6$ | $25$ $\pm 8$ | $80$ $\pm 5$ | $\mathbf{96}$ $\pm 2$ |  |  |  |
| task3 | $12$ $\pm 3$ | $15$ $\pm 3$ | $12$ $\pm 6$ | $25$ $\pm 10$ | $43$ $\pm 11$ | $\mathbf{79}$ $\pm 6$ |  |  |  |
| task4 | $2$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 0$ | $16$ $\pm 7$ | $5$ $\pm 5$ | $\mathbf{75}$ $\pm 6$ |  |  |  |
| task5 | $12$ $\pm 4$ | $40$ $\pm 8$ | $51$ $\pm 8$ | $29$ $\pm 12$ | $87$ $\pm 7$ | $\mathbf{97}$ $\pm 1$ |  |  |  |
| overall | $8$ $\pm 2$ | $24$ $\pm 2$ | $27$ $\pm 2$ | $21$ $\pm 8$ | $60$ $\pm 4$ | $\mathbf{89}$ $\pm 2$ |  |  |  |
| humanoidmaze-large-navigate-v0 | task1 | $1$ $\pm 1$ | $6$ $\pm 2$ | $3$ $\pm 2$ | $3$ $\pm 2$ | $36$ $\pm 11$ | $\mathbf{67}$ $\pm 4$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 3$ |  |  |  |
| task3 | $3$ $\pm 1$ | $6$ $\pm 2$ | $5$ $\pm 2$ | $17$ $\pm 6$ | $54$ $\pm 17$ | $\mathbf{88}$ $\pm 3$ |  |  |  |
| task4 | $2$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $4$ $\pm 2$ | $23$ $\pm 11$ | $\mathbf{42}$ $\pm 11$ |  |  |  |
| task5 | $1$ $\pm 1$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $2$ $\pm 1$ | $6$ $\pm 4$ | $\mathbf{47}$ $\pm 10$ |  |  |  |
| overall | $1$ $\pm 0$ | $2$ $\pm 1$ | $2$ $\pm 1$ | $5$ $\pm 1$ | $24$ $\pm 4$ | $\mathbf{49}$ $\pm 4$ |  |  |  |
| humanoidmaze-giant-navigate-v0 | task1 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $\mathbf{13}$ $\pm 7$ |  |  |
| task2 | 0 0 $\pm 0$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $2$ $\pm 1$ | $9$ $\pm 5$ | $\mathbf{35}$ $\pm 11$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | $\mathbf{11}$ $\pm 4$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 2$ | $2$ $\pm 2$ |  |  |  |
| task5 | $1$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $\mathbf{2}$ $\pm 1$ | $1$ $\pm 1$ | $\mathbf{2}$ $\pm 2$ |  |  |  |
| overall | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 0$ | $3$ $\pm 2$ | $\mathbf{12}$ $\pm 4$ |  |  |  |
| stitch | humanoidmaze-medium-stitch-v0 | task1 | $20$ $\pm 7$ | $13$ $\pm 3$ | $12$ $\pm 3$ | $6$ $\pm 5$ | $27$ $\pm 7$ | $\mathbf{84}$ $\pm 5$ |  |
| task2 | $49$ $\pm 12$ | $7$ $\pm 2$ | $8$ $\pm 5$ | $13$ $\pm 4$ | $37$ $\pm 7$ | $\mathbf{94}$ $\pm 2$ |  |  |  |
| task3 | $24$ $\pm 8$ | $25$ $\pm 3$ | $20$ $\pm 7$ | $30$ $\pm 6$ | $40$ $\pm 4$ | $\mathbf{86}$ $\pm 4$ |  |  |  |
| task4 | $3$ $\pm 2$ | $1$ $\pm 1$ | $2$ $\pm 2$ | $18$ $\pm 5$ | $28$ $\pm 7$ | $\mathbf{86}$ $\pm 4$ |  |  |  |
| task5 | $49$ $\pm 8$ | $16$ $\pm 3$ | $18$ $\pm 7$ | $22$ $\pm 2$ | $49$ $\pm 5$ | $\mathbf{90}$ $\pm 4$ |  |  |  |
| overall | $29$ $\pm 5$ | $12$ $\pm 2$ | $12$ $\pm 3$ | $18$ $\pm 2$ | $36$ $\pm 2$ | $\mathbf{88}$ $\pm 2$ |  |  |  |
| humanoidmaze-large-stitch-v0 | task1 | $3$ $\pm 4$ | $2$ $\pm 1$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{21}$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{5}$ $\pm 2$ |  |  |  |
| task3 | $20$ $\pm 11$ | $3$ $\pm 2$ | $1$ $\pm 1$ | $16$ $\pm 7$ | $13$ $\pm 3$ | $\mathbf{84}$ $\pm 4$ |  |  |  |
| task4 | $2$ $\pm 1$ | $1$ $\pm 1$ | 0 0 $\pm 1$ | $1$ $\pm 1$ | $4$ $\pm 1$ | $\mathbf{19}$ $\pm 4$ |  |  |  |
| task5 | $2$ $\pm 2$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $3$ $\pm 1$ | $\mathbf{12}$ $\pm 2$ |  |  |  |
| overall | $6$ $\pm 3$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 1$ | $4$ $\pm 1$ | $\mathbf{28}$ $\pm 3$ |  |  |  |
| humanoidmaze-giant-stitch-v0 | task1 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 2$ |  |  |
| task2 | 0 0 $\pm 0$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{12}$ $\pm 6$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ | $\mathbf{1}$ $\pm 1$ | 0 0 $\pm 1$ | 0 0 $\pm 1$ |  |  |  |
| overall | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 2$ |  |  |  |

**Table 14: Full results on AntSoccer.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| antsoccer | navigate | antsoccer-arena-navigate-v0 | task1 | $7$ $\pm 3$ | $61$ $\pm 4$ | $60$ $\pm 6$ | $12$ $\pm 2$ | $32$ $\pm 5$ | $\mathbf{67}$ $\pm 4$ |
| task2 | $10$ $\pm 3$ | $45$ $\pm 5$ | $56$ $\pm 4$ | $8$ $\pm 4$ | $27$ $\pm 4$ | $\mathbf{59}$ $\pm 4$ |  |  |  |
| task3 | $3$ $\pm 2$ | $62$ $\pm 6$ | $63$ $\pm 6$ | $12$ $\pm 2$ | $28$ $\pm 5$ | $\mathbf{76}$ $\pm 4$ |  |  |  |
| task4 | $3$ $\pm 1$ | $23$ $\pm 5$ | $28$ $\pm 6$ | $3$ $\pm 2$ | $11$ $\pm 2$ | $\mathbf{30}$ $\pm 3$ |  |  |  |
| task5 | $3$ $\pm 1$ | $42$ $\pm 6$ | $42$ $\pm 7$ | $3$ $\pm 3$ | $15$ $\pm 4$ | $\mathbf{56}$ $\pm 4$ |  |  |  |
| overall | $5$ $\pm 1$ | $47$ $\pm 3$ | $50$ $\pm 2$ | $8$ $\pm 2$ | $23$ $\pm 2$ | $\mathbf{58}$ $\pm 2$ |  |  |  |
| antsoccer-medium-navigate-v0 | task1 | $9$ $\pm 3$ | $17$ $\pm 5$ | $29$ $\pm 5$ | $8$ $\pm 8$ | $13$ $\pm 4$ | $\mathbf{45}$ $\pm 6$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 1$ | $1$ $\pm 1$ | 0 0 $\pm 1$ | $\mathbf{2}$ $\pm 1$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $\mathbf{3}$ $\pm 1$ |  |  |  |
| task5 | $1$ $\pm 1$ | $3$ $\pm 1$ | $4$ $\pm 3$ | $1$ $\pm 2$ | $2$ $\pm 1$ | $\mathbf{13}$ $\pm 5$ |  |  |  |
| overall | $2$ $\pm 0$ | $4$ $\pm 1$ | $7$ $\pm 1$ | $2$ $\pm 2$ | $3$ $\pm 1$ | $\mathbf{13}$ $\pm 2$ |  |  |  |
| stitch | antsoccer-arena-stitch-v0 | task1 | $\mathbf{73}$ $\pm 5$ | $37$ $\pm 4$ | $6$ $\pm 2$ | $2$ $\pm 1$ | $2$ $\pm 2$ | $24$ $\pm 2$ |  |
| task2 | $\mathbf{36}$ $\pm 19$ | $13$ $\pm 4$ | $2$ $\pm 1$ | $2$ $\pm 2$ | $1$ $\pm 0$ | $14$ $\pm 4$ |  |  |  |
| task3 | $6$ $\pm 15$ | $\mathbf{34}$ $\pm 9$ | $1$ $\pm 1$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $20$ $\pm 3$ |  |  |  |
| task4 | $7$ $\pm 12$ | $\mathbf{11}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $7$ $\pm 2$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $\mathbf{12}$ $\pm 3$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{12}$ $\pm 5$ |  |  |  |
| overall | $\mathbf{24}$ $\pm 8$ | $21$ $\pm 3$ | $2$ $\pm 0$ | $1$ $\pm 1$ | $1$ $\pm 0$ | $15$ $\pm 1$ |  |  |  |
| antsoccer-medium-stitch-v0 | task1 | $10$ $\pm 7$ | $4$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{21}$ $\pm 6$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 1$ |  |  |  |
| overall | $2$ $\pm 1$ | $1$ $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 1$ |  |  |  |

**Table 15: Full results on Visual AntMaze.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| visual-antmaze | navigate | visual-antmaze-medium-navigate-v0 | task1 | $17$ $\pm 6$ | $30$ $\pm 7$ | $16$ $\pm 3$ | 0 0 $\pm 0$ | $\mathbf{92}$ $\pm 2$ | $\mathbf{90}$ $\pm 4$ |
| task2 | $8$ $\pm 2$ | $21$ $\pm 6$ | $7$ $\pm 2$ | 0 0 $\pm 0$ | $\mathbf{94}$ $\pm 2$ | $\mathbf{92}$ $\pm 7$ |  |  |  |
| task3 | $17$ $\pm 1$ | $24$ $\pm 5$ | $16$ $\pm 4$ | 0 0 $\pm 0$ | $\mathbf{98}$ $\pm 1$ | $\mathbf{94}$ $\pm 4$ |  |  |  |
| task4 | $12$ $\pm 2$ | $21$ $\pm 3$ | $9$ $\pm 2$ | 0 0 $\pm 0$ | $\mathbf{94}$ $\pm 2$ | $\mathbf{94}$ $\pm 2$ |  |  |  |
| task5 | $4$ $\pm 2$ | $16$ $\pm 5$ | $6$ $\pm 2$ | 0 0 $\pm 0$ | $\mathbf{94}$ $\pm 2$ | $\mathbf{94}$ $\pm 5$ |  |  |  |
| overall | $11$ $\pm 2$ | $22$ $\pm 2$ | $11$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{94}$ $\pm 1$ | $\mathbf{93}$ $\pm 4$ |  |  |  |
| visual-antmaze-large-navigate-v0 | task1 | $3$ $\pm 1$ | $7$ $\pm 2$ | $4$ $\pm 3$ | 0 0 $\pm 0$ | $\mathbf{78}$ $\pm 5$ | $60$ $\pm 10$ |  |  |
| task2 | $4$ $\pm 3$ | $4$ $\pm 1$ | $2$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{80}$ $\pm 3$ | $28$ $\pm 9$ |  |  |  |
| task3 | $4$ $\pm 2$ | $6$ $\pm 2$ | $4$ $\pm 1$ | $1$ $\pm 1$ | $\mathbf{90}$ $\pm 3$ | $85$ $\pm 10$ |  |  |  |
| task4 | $4$ $\pm 2$ | $5$ $\pm 3$ | $6$ $\pm 1$ | 0 0 $\pm 1$ | $\mathbf{88}$ $\pm 3$ | $46$ $\pm 7$ |  |  |  |
| task5 | $4$ $\pm 2$ | $5$ $\pm 1$ | $4$ $\pm 2$ | 0 0 $\pm 0$ | $\mathbf{83}$ $\pm 2$ | $44$ $\pm 10$ |  |  |  |
| overall | $4$ $\pm 0$ | $5$ $\pm 1$ | $4$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{84}$ $\pm 1$ | $53$ $\pm 9$ |  |  |  |
| visual-antmaze-giant-navigate-v0 | task1 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{17}$ $\pm 2$ | $2$ $\pm 1$ |  |  |
| task2 | $1$ $\pm 1$ | $2$ $\pm 1$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{73}$ $\pm 9$ | $12$ $\pm 8$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{22}$ $\pm 6$ | $2$ $\pm 3$ |  |  |  |
| task4 | 0 0 $\pm 1$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{47}$ $\pm 5$ | $4$ $\pm 2$ |  |  |  |
| task5 | $1$ $\pm 1$ | $2$ $\pm 3$ | $1$ $\pm 1$ | 0 0 $\pm 1$ | $\mathbf{77}$ $\pm 5$ | $13$ $\pm 11$ |  |  |  |
| overall | 0 0 $\pm 0$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{47}$ $\pm 2$ | $6$ $\pm 4$ |  |  |  |
| visual-antmaze-teleport-navigate-v0 | task1 | $2$ $\pm 2$ | $6$ $\pm 1$ | $2$ $\pm 1$ | $3$ $\pm 2$ | $\mathbf{32}$ $\pm 3$ | $\mathbf{32}$ $\pm 5$ |  |  |
| task2 | $6$ $\pm 3$ | $9$ $\pm 3$ | $9$ $\pm 2$ | $6$ $\pm 4$ | $\mathbf{73}$ $\pm 8$ | $40$ $\pm 6$ |  |  |  |
| task3 | $9$ $\pm 1$ | $12$ $\pm 3$ | $9$ $\pm 2$ | $10$ $\pm 4$ | $\mathbf{47}$ $\pm 3$ | $33$ $\pm 1$ |  |  |  |
| task4 | $10$ $\pm 2$ | $10$ $\pm 2$ | $8$ $\pm 3$ | $6$ $\pm 4$ | $\mathbf{50}$ $\pm 4$ | $44$ $\pm 5$ |  |  |  |
| task5 | $1$ $\pm 1$ | $3$ $\pm 1$ | $3$ $\pm 1$ | $4$ $\pm 2$ | $\mathbf{36}$ $\pm 5$ | $33$ $\pm 7$ |  |  |  |
| overall | $5$ $\pm 1$ | $8$ $\pm 1$ | $6$ $\pm 1$ | $6$ $\pm 3$ | $\mathbf{48}$ $\pm 2$ | $37$ $\pm 2$ |  |  |  |
| stitch | visual-antmaze-medium-stitch-v0 | task1 | $\mathbf{80}$ $\pm 4$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $33$ $\pm 4$ | $75$ $\pm 8$ |  |
| task2 | $\mathbf{90}$ $\pm 4$ | $1$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $69$ $\pm 5$ | $85$ $\pm 7$ |  |  |  |
| task3 | $69$ $\pm 18$ | $15$ $\pm 6$ | $8$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{88}$ $\pm 1$ | $\mathbf{92}$ $\pm 1$ |  |  |  |
| task4 | $1$ $\pm 1$ | $7$ $\pm 4$ | $3$ $\pm 1$ | 0 0 $\pm 1$ | $70$ $\pm 12$ | $\mathbf{88}$ $\pm 4$ |  |  |  |
| task5 | $\mathbf{97}$ $\pm 1$ | $6$ $\pm 3$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $85$ $\pm 5$ | $\mathbf{93}$ $\pm 1$ |  |  |  |
| overall | $67$ $\pm 4$ | $6$ $\pm 2$ | $2$ $\pm 0$ | 0 0 $\pm 0$ | $69$ $\pm 2$ | $\mathbf{87}$ $\pm 2$ |  |  |  |
| visual-antmaze-large-stitch-v0 | task1 | $26$ $\pm 11$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $6$ $\pm 1$ | $\mathbf{36}$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $\mathbf{3}$ $\pm 2$ |  |  |  |
| task3 | $73$ $\pm 14$ | $3$ $\pm 2$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | $36$ $\pm 10$ | $\mathbf{87}$ $\pm 6$ |  |  |  |
| task4 | $7$ $\pm 5$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $\mathbf{8}$ $\pm 1$ | $7$ $\pm 4$ |  |  |  |
| task5 | $\mathbf{11}$ $\pm 5$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $5$ $\pm 2$ | $6$ $\pm 1$ |  |  |  |
| overall | $24$ $\pm 3$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $11$ $\pm 3$ | $\mathbf{28}$ $\pm 2$ |  |  |  |
| visual-antmaze-giant-stitch-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{1}$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 1$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| visual-antmaze-teleport-stitch-v0 | task1 | $\mathbf{37}$ $\pm 4$ | $2$ $\pm 2$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $20$ $\pm 5$ | $\mathbf{36}$ $\pm 5$ |  |  |
| task2 | $36$ $\pm 3$ | $2$ $\pm 1$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $\mathbf{40}$ $\pm 9$ | $\mathbf{38}$ $\pm 3$ |  |  |  |
| task3 | $17$ $\pm 6$ | $2$ $\pm 1$ | $2$ $\pm 1$ | $3$ $\pm 4$ | $32$ $\pm 9$ | $\mathbf{36}$ $\pm 5$ |  |  |  |
| task4 | $39$ $\pm 9$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $2$ $\pm 3$ | $\mathbf{45}$ $\pm 7$ | $37$ $\pm 6$ |  |  |  |
| task5 | $29$ $\pm 1$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $22$ $\pm 9$ | $\mathbf{38}$ $\pm 5$ |  |  |  |
| overall | $32$ $\pm 3$ | $1$ $\pm 1$ | $1$ $\pm 0$ | $1$ $\pm 2$ | $32$ $\pm 6$ | $\mathbf{37}$ $\pm 4$ |  |  |  |
| explore | visual-antmaze-medium-explore-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 2$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| visual-antmaze-large-explore-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| visual-antmaze-teleport-explore-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 1$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 1$ | $\mathbf{38}$ $\pm 8$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{28}$ $\pm 13$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | $\mathbf{27}$ $\pm 18$ |  |  |  |
| overall | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 0$ | $\mathbf{19}$ $\pm 8$ |  |  |  |

**Table 16: Full results on Visual HumanoidMaze.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| visual-humanoidmaze | navigate | visual-humanoidmaze-medium-navigate-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 1$ | 0 0 $\pm 1$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 2$ | 0 0 $\pm 1$ |  |  |  |
| overall | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| visual-humanoidmaze-large-navigate-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| visual-humanoidmaze-giant-navigate-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| stitch | visual-humanoidmaze-medium-stitch-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{3}$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 2$ | $1$ $\pm 2$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{1}$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| overall | $\mathbf{1}$ $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| visual-humanoidmaze-large-stitch-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| visual-humanoidmaze-giant-stitch-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |

**Table 17: Full results on Cube.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| cube | play | cube-single-play-v0 | task1 | $7$ $\pm 3$ | $57$ $\pm 6$ | $\mathbf{71}$ $\pm 9$ | $6$ $\pm 2$ | $20$ $\pm 6$ | $15$ $\pm 5$ |
| task2 | $5$ $\pm 2$ | $51$ $\pm 6$ | $\mathbf{71}$ $\pm 6$ | $5$ $\pm 2$ | $20$ $\pm 4$ | $16$ $\pm 5$ |  |  |  |
| task3 | $7$ $\pm 3$ | $55$ $\pm 6$ | $\mathbf{70}$ $\pm 6$ | $4$ $\pm 1$ | $21$ $\pm 6$ | $16$ $\pm 3$ |  |  |  |
| task4 | $4$ $\pm 2$ | $50$ $\pm 4$ | $\mathbf{61}$ $\pm 8$ | $4$ $\pm 2$ | $16$ $\pm 3$ | $14$ $\pm 5$ |  |  |  |
| task5 | $4$ $\pm 2$ | $52$ $\pm 6$ | $\mathbf{67}$ $\pm 7$ | $4$ $\pm 3$ | $15$ $\pm 3$ | $13$ $\pm 4$ |  |  |  |
| overall | $6$ $\pm 2$ | $53$ $\pm 4$ | $\mathbf{68}$ $\pm 6$ | $5$ $\pm 1$ | $19$ $\pm 2$ | $15$ $\pm 3$ |  |  |  |
| cube-double-play-v0 | task1 | $6$ $\pm 3$ | $58$ $\pm 5$ | $\mathbf{74}$ $\pm 8$ | $6$ $\pm 3$ | $30$ $\pm 7$ | $22$ $\pm 6$ |  |  |
| task2 | 0 0 $\pm 0$ | $51$ $\pm 6$ | $\mathbf{55}$ $\pm 11$ | 0 0 $\pm 0$ | $9$ $\pm 2$ | $4$ $\pm 3$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $42$ $\pm 7$ | $\mathbf{45}$ $\pm 7$ | 0 0 $\pm 0$ | $6$ $\pm 1$ | $3$ $\pm 2$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $\mathbf{7}$ $\pm 2$ | $4$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $21$ $\pm 1$ | $\mathbf{23}$ $\pm 6$ | 0 0 $\pm 0$ | $3$ $\pm 1$ | $2$ $\pm 1$ |  |  |  |
| overall | $1$ $\pm 1$ | $36$ $\pm 3$ | $\mathbf{40}$ $\pm 5$ | $1$ $\pm 0$ | $10$ $\pm 2$ | $6$ $\pm 2$ |  |  |  |
| cube-triple-play-v0 | task1 | $5$ $\pm 4$ | $3$ $\pm 1$ | $13$ $\pm 3$ | $1$ $\pm 1$ | $\mathbf{19}$ $\pm 5$ | $12$ $\pm 6$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 1$ | 0 0 $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ | 0 0 $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $1$ $\pm 1$ | $1$ $\pm 0$ | $3$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 1$ | $3$ $\pm 1$ |  |  |  |
| cube-quadruple-play-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| noisy | cube-single-noisy-v0 | task1 | $5$ $\pm 3$ | $71$ $\pm 12$ | $\mathbf{100}$ $\pm 1$ | $17$ $\pm 12$ | $39$ $\pm 4$ | $48$ $\pm 6$ |  |
| task2 | $7$ $\pm 5$ | $70$ $\pm 11$ | $\mathbf{100}$ $\pm 0$ | $23$ $\pm 8$ | $39$ $\pm 7$ | $39$ $\pm 7$ |  |  |  |
| task3 | $1$ $\pm 1$ | $67$ $\pm 10$ | $\mathbf{99}$ $\pm 1$ | $4$ $\pm 3$ | $36$ $\pm 7$ | $41$ $\pm 7$ |  |  |  |
| task4 | $16$ $\pm 5$ | $76$ $\pm 10$ | $\mathbf{98}$ $\pm 2$ | $47$ $\pm 14$ | $36$ $\pm 4$ | $36$ $\pm 6$ |  |  |  |
| task5 | $12$ $\pm 5$ | $70$ $\pm 12$ | $\mathbf{100}$ $\pm 0$ | $37$ $\pm 9$ | $42$ $\pm 5$ | $44$ $\pm 9$ |  |  |  |
| overall | $8$ $\pm 3$ | $71$ $\pm 9$ | $\mathbf{99}$ $\pm 1$ | $25$ $\pm 6$ | $38$ $\pm 2$ | $41$ $\pm 6$ |  |  |  |
| cube-double-noisy-v0 | task1 | $7$ $\pm 3$ | $53$ $\pm 11$ | $\mathbf{64}$ $\pm 8$ | $16$ $\pm 5$ | $9$ $\pm 5$ | $10$ $\pm 3$ |  |  |
| task2 | 0 0 $\pm 0$ | $10$ $\pm 4$ | $\mathbf{16}$ $\pm 4$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $1$ $\pm 1$ | $\mathbf{6}$ $\pm 4$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $4$ $\pm 2$ | $\mathbf{11}$ $\pm 3$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 1$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $4$ $\pm 2$ | $\mathbf{20}$ $\pm 4$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 1$ |  |  |  |
| overall | $1$ $\pm 1$ | $14$ $\pm 3$ | $\mathbf{23}$ $\pm 3$ | $3$ $\pm 1$ | $2$ $\pm 1$ | $2$ $\pm 1$ |  |  |  |
| cube-triple-noisy-v0 | task1 | $6$ $\pm 3$ | $\mathbf{44}$ $\pm 7$ | $8$ $\pm 2$ | $5$ $\pm 2$ | $13$ $\pm 6$ | $8$ $\pm 3$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $1$ $\pm 1$ | $\mathbf{9}$ $\pm 1$ | $2$ $\pm 1$ | $1$ $\pm 0$ | $3$ $\pm 1$ | $2$ $\pm 1$ |  |  |  |
| cube-quadruple-noisy-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |

**Table 18: Full results on Scene.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| scene | play | scene-play-v0 | task1 | $18$ $\pm 7$ | $75$ $\pm 5$ | $\mathbf{93}$ $\pm 4$ | $19$ $\pm 4$ | $49$ $\pm 7$ | $40$ $\pm 4$ |
| task2 | $1$ $\pm 1$ | $62$ $\pm 8$ | $\mathbf{82}$ $\pm 8$ | $1$ $\pm 1$ | $12$ $\pm 4$ | $40$ $\pm 9$ |  |  |  |
| task3 | $2$ $\pm 1$ | $64$ $\pm 7$ | $\mathbf{72}$ $\pm 10$ | $1$ $\pm 1$ | $26$ $\pm 8$ | $36$ $\pm 5$ |  |  |  |
| task4 | $3$ $\pm 2$ | $7$ $\pm 4$ | $8$ $\pm 3$ | $5$ $\pm 2$ | $5$ $\pm 2$ | $\mathbf{55}$ $\pm 5$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $2$ $\pm 1$ | $1$ $\pm 1$ | 0 0 $\pm 1$ | $1$ $\pm 1$ | $\mathbf{20}$ $\pm 5$ |  |  |  |
| overall | $5$ $\pm 1$ | $42$ $\pm 4$ | $\mathbf{51}$ $\pm 4$ | $5$ $\pm 1$ | $19$ $\pm 2$ | $38$ $\pm 3$ |  |  |  |
| noisy | scene-noisy-v0 | task1 | $6$ $\pm 3$ | $60$ $\pm 11$ | $50$ $\pm 5$ | $39$ $\pm 10$ | $5$ $\pm 4$ | $\mathbf{68}$ $\pm 5$ |  |
| task2 | 0 0 $\pm 0$ | $42$ $\pm 11$ | $\mathbf{52}$ $\pm 13$ | $2$ $\pm 1$ | 0 0 $\pm 0$ | $29$ $\pm 6$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $\mathbf{27}$ $\pm 6$ | $\mathbf{28}$ $\pm 5$ | $1$ $\pm 1$ | 0 0 $\pm 1$ | $17$ $\pm 6$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $3$ $\pm 3$ | 0 0 $\pm 0$ | $3$ $\pm 3$ | 0 0 $\pm 0$ | $\mathbf{10}$ $\pm 8$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ |  |  |  |
| overall | $1$ $\pm 1$ | $\mathbf{26}$ $\pm 5$ | $\mathbf{26}$ $\pm 2$ | $9$ $\pm 2$ | $1$ $\pm 1$ | $\mathbf{25}$ $\pm 4$ |  |  |  |

**Table 19: Full results on Puzzle.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| puzzle | play | puzzle-3x3-play-v0 | task1 | $5$ $\pm 1$ | $17$ $\pm 4$ | $\mathbf{99}$ $\pm 2$ | $3$ $\pm 2$ | $11$ $\pm 3$ | $29$ $\pm 4$ |
| task2 | $2$ $\pm 1$ | $4$ $\pm 2$ | $\mathbf{96}$ $\pm 3$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $11$ $\pm 3$ |  |  |  |
| task3 | $1$ $\pm 1$ | $3$ $\pm 1$ | $\mathbf{95}$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $7$ $\pm 3$ |  |  |  |
| task4 | $1$ $\pm 1$ | $3$ $\pm 1$ | $\mathbf{91}$ $\pm 3$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $5$ $\pm 1$ |  |  |  |
| task5 | $1$ $\pm 0$ | $2$ $\pm 1$ | $\mathbf{94}$ $\pm 2$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $8$ $\pm 3$ |  |  |  |
| overall | $2$ $\pm 0$ | $6$ $\pm 1$ | $\mathbf{95}$ $\pm 1$ | $1$ $\pm 0$ | $3$ $\pm 1$ | $12$ $\pm 2$ |  |  |  |
| puzzle-4x4-play-v0 | task1 | 0 0 $\pm 0$ | $17$ $\pm 4$ | $\mathbf{42}$ $\pm 7$ | $1$ $\pm 1$ | $1$ $\pm 1$ | $10$ $\pm 3$ |  |  |
| task2 | 0 0 $\pm 0$ | $\mathbf{13}$ $\pm 5$ | $2$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $9$ $\pm 4$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $12$ $\pm 3$ | $\mathbf{40}$ $\pm 5$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $7$ $\pm 3$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $11$ $\pm 3$ | $\mathbf{23}$ $\pm 5$ | 0 0 $\pm 0$ | 0 0 $\pm 1$ | $6$ $\pm 2$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $10$ $\pm 4$ | $\mathbf{23}$ $\pm 5$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $5$ $\pm 2$ |  |  |  |
| overall | 0 0 $\pm 0$ | $13$ $\pm 2$ | $\mathbf{26}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $7$ $\pm 2$ |  |  |  |
| puzzle-4x5-play-v0 | task1 | $1$ $\pm 1$ | $33$ $\pm 6$ | $\mathbf{71}$ $\pm 5$ | 0 0 $\pm 1$ | $6$ $\pm 2$ | $17$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | 0 0 $\pm 0$ | $7$ $\pm 1$ | $\mathbf{14}$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 0$ | $4$ $\pm 1$ |  |  |  |
| puzzle-4x6-play-v0 | task1 | 0 0 $\pm 0$ | $43$ $\pm 8$ | $\mathbf{52}$ $\pm 6$ | 0 0 $\pm 0$ | $12$ $\pm 5$ | $12$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | $\mathbf{7}$ $\pm 4$ | $5$ $\pm 3$ | 0 0 $\pm 0$ | $\mathbf{7}$ $\pm 3$ | $2$ $\pm 1$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | 0 0 $\pm 0$ | $10$ $\pm 2$ | $\mathbf{12}$ $\pm 1$ | 0 0 $\pm 0$ | $4$ $\pm 1$ | $3$ $\pm 1$ |  |  |  |
| noisy | puzzle-3x3-noisy-v0 | task1 | $4$ $\pm 2$ | $89$ $\pm 8$ | $\mathbf{100}$ $\pm 0$ | $1$ $\pm 1$ | $76$ $\pm 8$ | $67$ $\pm 10$ |  |
| task2 | 0 0 $\pm 0$ | $42$ $\pm 29$ | $\mathbf{88}$ $\pm 9$ | 0 0 $\pm 0$ | $26$ $\pm 9$ | $54$ $\pm 11$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $26$ $\pm 20$ | $\mathbf{99}$ $\pm 1$ | 0 0 $\pm 0$ | $15$ $\pm 9$ | $43$ $\pm 12$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $23$ $\pm 20$ | $\mathbf{94}$ $\pm 3$ | 0 0 $\pm 0$ | $12$ $\pm 9$ | $41$ $\pm 13$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $31$ $\pm 19$ | $\mathbf{88}$ $\pm 5$ | 0 0 $\pm 0$ | $18$ $\pm 6$ | $47$ $\pm 15$ |  |  |  |
| overall | $1$ $\pm 0$ | $42$ $\pm 19$ | $\mathbf{94}$ $\pm 3$ | 0 0 $\pm 0$ | $30$ $\pm 6$ | $51$ $\pm 11$ |  |  |  |
| puzzle-4x4-noisy-v0 | task1 | 0 0 $\pm 0$ | $\mathbf{51}$ $\pm 10$ | $\mathbf{49}$ $\pm 9$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $19$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{16}$ $\pm 5$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $34$ $\pm 4$ | $\mathbf{61}$ $\pm 14$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $17$ $\pm 6$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $9$ $\pm 3$ | $\mathbf{23}$ $\pm 10$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $14$ $\pm 5$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $8$ $\pm 4$ | $\mathbf{14}$ $\pm 9$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $12$ $\pm 4$ |  |  |  |
| overall | 0 0 $\pm 0$ | $20$ $\pm 3$ | $\mathbf{29}$ $\pm 7$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $16$ $\pm 4$ |  |  |  |
| puzzle-4x5-noisy-v0 | task1 | 0 0 $\pm 0$ | $\mathbf{97}$ $\pm 1$ | $\mathbf{97}$ $\pm 2$ | 0 0 $\pm 0$ | $16$ $\pm 9$ | $21$ $\pm 5$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | 0 0 $\pm 0$ | $\mathbf{19}$ $\pm 0$ | $\mathbf{19}$ $\pm 0$ | 0 0 $\pm 0$ | $3$ $\pm 2$ | $5$ $\pm 1$ |  |  |  |
| puzzle-4x6-noisy-v0 | task1 | 0 0 $\pm 0$ | $80$ $\pm 8$ | $\mathbf{86}$ $\pm 7$ | 0 0 $\pm 0$ | $28$ $\pm 13$ | $8$ $\pm 4$ |  |  |
| task2 | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 2$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $1$ $\pm 1$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | 0 0 $\pm 0$ | $17$ $\pm 2$ | $\mathbf{18}$ $\pm 2$ | 0 0 $\pm 0$ | $6$ $\pm 3$ | $2$ $\pm 1$ |  |  |  |

**Table 20: Full results on Visual Cube.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| visual-cube | play | visual-cube-single-play-v0 | task1 | $12$ $\pm 4$ | $70$ $\pm 3$ | $42$ $\pm 12$ | $68$ $\pm 9$ | $47$ $\pm 20$ | $\mathbf{93}$ $\pm 1$ |
| task2 | $4$ $\pm 4$ | $65$ $\pm 9$ | $44$ $\pm 11$ | $35$ $\pm 34$ | $40$ $\pm 20$ | $\mathbf{93}$ $\pm 3$ |  |  |  |
| task3 | $6$ $\pm 5$ | $49$ $\pm 2$ | $24$ $\pm 9$ | $41$ $\pm 27$ | $33$ $\pm 17$ | $\mathbf{84}$ $\pm 5$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $60$ $\pm 13$ | $21$ $\pm 6$ | $32$ $\pm 10$ | $18$ $\pm 16$ | $\mathbf{84}$ $\pm 5$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $55$ $\pm 6$ | $20$ $\pm 7$ | $30$ $\pm 8$ | $16$ $\pm 10$ | $\mathbf{88}$ $\pm 2$ |  |  |  |
| overall | $5$ $\pm 1$ | $60$ $\pm 5$ | $30$ $\pm 5$ | $41$ $\pm 15$ | $31$ $\pm 15$ | $\mathbf{89}$ $\pm 0$ |  |  |  |
| visual-cube-double-play-v0 | task1 | $4$ $\pm 3$ | $44$ $\pm 8$ | $6$ $\pm 4$ | $20$ $\pm 3$ | $7$ $\pm 4$ | $\mathbf{91}$ $\pm 1$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | 0 0 $\pm 0$ | $\mathbf{54}$ $\pm 5$ |  |  |  |
| task3 | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $2$ $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{40}$ $\pm 6$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $4$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{11}$ $\pm 4$ |  |  |  |
| overall | $1$ $\pm 1$ | $10$ $\pm 2$ | $1$ $\pm 1$ | $5$ $\pm 0$ | $2$ $\pm 1$ | $\mathbf{39}$ $\pm 2$ |  |  |  |
| visual-cube-triple-play-v0 | task1 | $73$ $\pm 8$ | $68$ $\pm 8$ | $76$ $\pm 6$ | $81$ $\pm 3$ | $85$ $\pm 12$ | $\mathbf{98}$ $\pm 1$ |  |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{7}$ $\pm 3$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $15$ $\pm 2$ | $14$ $\pm 2$ | $15$ $\pm 1$ | $16$ $\pm 1$ | $17$ $\pm 2$ | $\mathbf{21}$ $\pm 0$ |  |  |  |
| visual-cube-quadruple-play-v0 | task1 | $42$ $\pm 4$ | $1$ $\pm 2$ | $36$ $\pm 7$ | $23$ $\pm 5$ | $20$ $\pm 4$ | $\mathbf{66}$ $\pm 7$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ |  |  |  |
| overall | $8$ $\pm 1$ | 0 0 $\pm 0$ | $7$ $\pm 1$ | $5$ $\pm 1$ | $4$ $\pm 1$ | $\mathbf{14}$ $\pm 1$ |  |  |  |
| noisy | visual-cube-single-noisy-v0 | task1 | $12$ $\pm 4$ | $89$ $\pm 4$ | $81$ $\pm 6$ | $18$ $\pm 29$ | $43$ $\pm 41$ | $\mathbf{100}$ $\pm 1$ |  |
| task2 | $17$ $\pm 6$ | $48$ $\pm 14$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $28$ $\pm 13$ | $\mathbf{100}$ $\pm 1$ |  |  |  |
| task3 | $6$ $\pm 2$ | $77$ $\pm 4$ | $90$ $\pm 3$ | $28$ $\pm 27$ | $42$ $\pm 41$ | $\mathbf{99}$ $\pm 1$ |  |  |  |
| task4 | $14$ $\pm 2$ | $82$ $\pm 2$ | $35$ $\pm 11$ | $2$ $\pm 1$ | $43$ $\pm 35$ | $\mathbf{100}$ $\pm 1$ |  |  |  |
| task5 | $20$ $\pm 10$ | $77$ $\pm 2$ | $33$ $\pm 7$ | $2$ $\pm 1$ | $37$ $\pm 27$ | $\mathbf{99}$ $\pm 1$ |  |  |  |
| overall | $14$ $\pm 3$ | $75$ $\pm 3$ | $48$ $\pm 3$ | $10$ $\pm 5$ | $39$ $\pm 30$ | $\mathbf{99}$ $\pm 0$ |  |  |  |
| visual-cube-double-noisy-v0 | task1 | $20$ $\pm 5$ | $70$ $\pm 8$ | $70$ $\pm 5$ | $27$ $\pm 9$ | $24$ $\pm 11$ | $\mathbf{98}$ $\pm 2$ |  |  |
| task2 | $2$ $\pm 2$ | $5$ $\pm 3$ | $14$ $\pm 2$ | $2$ $\pm 4$ | $3$ $\pm 3$ | $\mathbf{87}$ $\pm 9$ |  |  |  |
| task3 | $2$ $\pm 1$ | $5$ $\pm 5$ | $16$ $\pm 6$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | $\mathbf{68}$ $\pm 10$ |  |  |  |
| task4 | $1$ $\pm 1$ | $3$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{13}$ $\pm 9$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $7$ $\pm 2$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{30}$ $\pm 4$ |  |  |  |
| overall | $5$ $\pm 1$ | $17$ $\pm 4$ | $22$ $\pm 2$ | $6$ $\pm 2$ | $6$ $\pm 3$ | $\mathbf{59}$ $\pm 3$ |  |  |  |
| visual-cube-triple-noisy-v0 | task1 | $80$ $\pm 7$ | $90$ $\pm 5$ | $62$ $\pm 6$ | $44$ $\pm 21$ | $78$ $\pm 7$ | $\mathbf{99}$ $\pm 1$ |  |  |
| task2 | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ |  |  |  |
| task3 | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{14}$ $\pm 11$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 1$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $16$ $\pm 1$ | $18$ $\pm 1$ | $12$ $\pm 1$ | $9$ $\pm 4$ | $16$ $\pm 1$ | $\mathbf{23}$ $\pm 2$ |  |  |  |
| visual-cube-quadruple-noisy-v0 | task1 | $46$ $\pm 2$ | $2$ $\pm 1$ | $10$ $\pm 9$ | $2$ $\pm 2$ | $39$ $\pm 10$ | $\mathbf{60}$ $\pm 41$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $9$ $\pm 0$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | 0 0 $\pm 0$ | $8$ $\pm 2$ | $\mathbf{12}$ $\pm 8$ |  |  |  |

**Table 21: Full results on Visual Scene.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| visual-scene | play | visual-scene-play-v0 | task1 | $59$ $\pm 7$ | $\mathbf{84}$ $\pm 4$ | $56$ $\pm 4$ | $44$ $\pm 6$ | $52$ $\pm 6$ | $\mathbf{80}$ $\pm 6$ |
| task2 | 0 0 $\pm 0$ | $24$ $\pm 8$ | $1$ $\pm 1$ | $2$ $\pm 2$ | $1$ $\pm 1$ | $\mathbf{81}$ $\pm 7$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $16$ $\pm 8$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{61}$ $\pm 11$ |  |  |  |
| task4 | $2$ $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 4$ | $2$ $\pm 1$ | $1$ $\pm 1$ | $\mathbf{20}$ $\pm 8$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{3}$ $\pm 2$ |  |  |  |
| overall | $12$ $\pm 2$ | $25$ $\pm 3$ | $12$ $\pm 2$ | $10$ $\pm 1$ | $11$ $\pm 2$ | $\mathbf{49}$ $\pm 4$ |  |  |  |
| noisy | visual-scene-noisy-v0 | task1 | $64$ $\pm 9$ | $76$ $\pm 6$ | $49$ $\pm 22$ | $8$ $\pm 2$ | $70$ $\pm 9$ | $\mathbf{91}$ $\pm 4$ |  |
| task2 | 0 0 $\pm 0$ | $14$ $\pm 7$ | $2$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{69}$ $\pm 5$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $24$ $\pm 5$ | $7$ $\pm 2$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | $\mathbf{82}$ $\pm 6$ |  |  |  |
| task4 | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $\mathbf{8}$ $\pm 5$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $13$ $\pm 2$ | $23$ $\pm 2$ | $12$ $\pm 4$ | $2$ $\pm 0$ | $15$ $\pm 2$ | $\mathbf{50}$ $\pm 1$ |  |  |  |

**Table 22: Full results on Visual Puzzle.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| visual-puzzle | play | visual-puzzle-3x3-play-v0 | task1 | $1$ $\pm 1$ | $\mathbf{97}$ $\pm 2$ | $5$ $\pm 9$ | $3$ $\pm 3$ | $2$ $\pm 1$ | $\mathbf{98}$ $\pm 1$ |
| task2 | 0 0 $\pm 0$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{73}$ $\pm 10$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{64}$ $\pm 9$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $2$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{63}$ $\pm 9$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $3$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{66}$ $\pm 12$ |  |  |  |
| overall | 0 0 $\pm 0$ | $21$ $\pm 1$ | $1$ $\pm 2$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{73}$ $\pm 8$ |  |  |  |
| visual-puzzle-4x4-play-v0 | task1 | $11$ $\pm 3$ | $\mathbf{86}$ $\pm 7$ | $18$ $\pm 5$ | 0 0 $\pm 0$ | $10$ $\pm 7$ | $70$ $\pm 47$ |  |  |
| task2 | $19$ $\pm 5$ | $8$ $\pm 5$ | $28$ $\pm 12$ | 0 0 $\pm 0$ | $19$ $\pm 13$ | $\mathbf{38}$ $\pm 32$ |  |  |  |
| task3 | $8$ $\pm 3$ | $\mathbf{80}$ $\pm 4$ | $15$ $\pm 4$ | 0 0 $\pm 0$ | $8$ $\pm 7$ | $66$ $\pm 44$ |  |  |  |
| task4 | $6$ $\pm 2$ | $\mathbf{65}$ $\pm 9$ | $10$ $\pm 5$ | 0 0 $\pm 0$ | $7$ $\pm 5$ | $\mathbf{66}$ $\pm 44$ |  |  |  |
| task5 | $6$ $\pm 2$ | $\mathbf{61}$ $\pm 8$ | $9$ $\pm 3$ | 0 0 $\pm 0$ | $4$ $\pm 4$ | $\mathbf{60}$ $\pm 41$ |  |  |  |
| overall | $10$ $\pm 1$ | $\mathbf{60}$ $\pm 5$ | $16$ $\pm 4$ | 0 0 $\pm 0$ | $10$ $\pm 6$ | $\mathbf{60}$ $\pm 41$ |  |  |  |
| visual-puzzle-4x5-play-v0 | task1 | $22$ $\pm 8$ | $\mathbf{86}$ $\pm 4$ | $31$ $\pm 8$ | 0 0 $\pm 0$ | $31$ $\pm 6$ | $66$ $\pm 44$ |  |  |
| task2 | $\mathbf{1}$ $\pm 1$ | 0 0 $\pm 0$ | $\mathbf{1}$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $5$ $\pm 2$ | $\mathbf{17}$ $\pm 1$ | $7$ $\pm 2$ | 0 0 $\pm 0$ | $6$ $\pm 1$ | $13$ $\pm 9$ |  |  |  |
| visual-puzzle-4x6-play-v0 | task1 | $12$ $\pm 4$ | $\mathbf{66}$ $\pm 3$ | $10$ $\pm 3$ | 0 0 $\pm 0$ | $12$ $\pm 5$ | $34$ $\pm 23$ |  |  |
| task2 | 0 0 $\pm 1$ | $\mathbf{8}$ $\pm 2$ | $2$ $\pm 2$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $\mathbf{8}$ $\pm 6$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $2$ $\pm 1$ | $\mathbf{15}$ $\pm 1$ | $2$ $\pm 1$ | 0 0 $\pm 0$ | $3$ $\pm 1$ | $9$ $\pm 6$ |  |  |  |
| noisy | visual-puzzle-3x3-noisy-v0 | task1 | $4$ $\pm 6$ | $\mathbf{100}$ $\pm 1$ | $\mathbf{98}$ $\pm 2$ | 0 0 $\pm 0$ | $7$ $\pm 7$ | $\mathbf{100}$ $\pm 0$ |  |
| task2 | 0 0 $\pm 0$ | 0 0 $\pm 1$ | $5$ $\pm 7$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{64}$ $\pm 11$ |  |  |  |
| task3 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $2$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{55}$ $\pm 3$ |  |  |  |
| task4 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $5$ $\pm 4$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{61}$ $\pm 8$ |  |  |  |
| task5 | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $19$ $\pm 10$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{71}$ $\pm 13$ |  |  |  |
| overall | $1$ $\pm 1$ | $20$ $\pm 0$ | $26$ $\pm 4$ | 0 0 $\pm 0$ | $1$ $\pm 1$ | $\mathbf{70}$ $\pm 6$ |  |  |  |
| visual-puzzle-4x4-noisy-v0 | task1 | $6$ $\pm 2$ | $90$ $\pm 9$ | $85$ $\pm 3$ | 0 0 $\pm 0$ | $4$ $\pm 3$ | $\mathbf{98}$ $\pm 2$ |  |  |
| task2 | $16$ $\pm 7$ | $1$ $\pm 1$ | $1$ $\pm 1$ | 0 0 $\pm 0$ | $14$ $\pm 1$ | $\mathbf{44}$ $\pm 20$ |  |  |  |
| task3 | $4$ $\pm 3$ | $88$ $\pm 4$ | $77$ $\pm 6$ | 0 0 $\pm 0$ | $4$ $\pm 4$ | $\mathbf{95}$ $\pm 2$ |  |  |  |
| task4 | $2$ $\pm 1$ | $36$ $\pm 10$ | $47$ $\pm 17$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $\mathbf{91}$ $\pm 2$ |  |  |  |
| task5 | $4$ $\pm 2$ | $20$ $\pm 9$ | $34$ $\pm 13$ | 0 0 $\pm 0$ | $5$ $\pm 4$ | $\mathbf{94}$ $\pm 1$ |  |  |  |
| overall | $7$ $\pm 3$ | $47$ $\pm 3$ | $49$ $\pm 7$ | 0 0 $\pm 0$ | $6$ $\pm 2$ | $\mathbf{84}$ $\pm 4$ |  |  |  |
| visual-puzzle-4x5-noisy-v0 | task1 | $30$ $\pm 6$ | $72$ $\pm 48$ | $\mathbf{96}$ $\pm 2$ | 0 0 $\pm 0$ | $33$ $\pm 5$ | $72$ $\pm 48$ |  |  |
| task2 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $6$ $\pm 1$ | $14$ $\pm 10$ | $\mathbf{19}$ $\pm 0$ | 0 0 $\pm 0$ | $7$ $\pm 1$ | $14$ $\pm 10$ |  |  |  |
| visual-puzzle-4x6-noisy-v0 | task1 | $10$ $\pm 2$ | $61$ $\pm 41$ | $\mathbf{82}$ $\pm 4$ | 0 0 $\pm 0$ | $9$ $\pm 7$ | $56$ $\pm 7$ |  |  |
| task2 | 0 0 $\pm 0$ | $1$ $\pm 1$ | $2$ $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $\mathbf{12}$ $\pm 10$ |  |  |  |
| task3 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task4 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |  |
| overall | $2$ $\pm 1$ | $12$ $\pm 8$ | $\mathbf{17}$ $\pm 1$ | 0 0 $\pm 0$ | $2$ $\pm 1$ | $14$ $\pm 2$ |  |  |  |

**Table 23: Full results on Powderworld.**
| Environment Type | Dataset Type | Dataset | Task | GCBC | GCIVL | GCIQL | QRL | CRL | HIQL |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| powderworld | play | powderworld-easy-play-v0 | task1 | $1$ $\pm 1$ | $\mathbf{99}$ $\pm 1$ | $\mathbf{95}$ $\pm 4$ | $25$ $\pm 8$ | $40$ $\pm 6$ | $62$ $\pm 5$ |
| task2 | 0 0 $\pm 0$ | $\mathbf{96}$ $\pm 4$ | $\mathbf{93}$ $\pm 3$ | $12$ $\pm 4$ | $43$ $\pm 10$ | $54$ $\pm 10$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $\mathbf{100}$ $\pm 0$ | $94$ $\pm 8$ | $8$ $\pm 7$ | $15$ $\pm 5$ | $41$ $\pm 26$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $\mathbf{100}$ $\pm 0$ | $92$ $\pm 8$ | $8$ $\pm 5$ | $10$ $\pm 3$ | $8$ $\pm 6$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $\mathbf{100}$ $\pm 1$ | $93$ $\pm 5$ | $7$ $\pm 3$ | $2$ $\pm 3$ | $2$ $\pm 2$ |  |  |  |
| overall | 0 0 $\pm 0$ | $\mathbf{99}$ $\pm 1$ | $93$ $\pm 5$ | $12$ $\pm 2$ | $22$ $\pm 5$ | $33$ $\pm 9$ |  |  |  |
| powderworld-medium-play-v0 | task1 | 0 0 $\pm 0$ | $\mathbf{81}$ $\pm 12$ | $7$ $\pm 14$ | $2$ $\pm 3$ | 0 0 $\pm 1$ | $26$ $\pm 20$ |  |  |
| task2 | $3$ $\pm 3$ | $\mathbf{28}$ $\pm 14$ | 0 0 $\pm 0$ | $9$ $\pm 6$ | $2$ $\pm 3$ | $16$ $\pm 12$ |  |  |  |
| task3 | $2$ $\pm 3$ | $\mathbf{99}$ $\pm 2$ | $72$ $\pm 14$ | $4$ $\pm 5$ | $2$ $\pm 2$ | $60$ $\pm 37$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $\mathbf{39}$ $\pm 17$ | 0 0 $\pm 1$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $5$ $\pm 4$ |  |  |  |
| task5 | 0 0 $\pm 0$ | $\mathbf{2}$ $\pm 2$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 2$ |  |  |  |
| overall | $1$ $\pm 1$ | $\mathbf{50}$ $\pm 4$ | $16$ $\pm 5$ | $3$ $\pm 1$ | $1$ $\pm 1$ | $22$ $\pm 14$ |  |  |  |
| powderworld-hard-play-v0 | task1 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ |  |  |
| task2 | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 2$ |  |  |  |
| task3 | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 4$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $2$ $\pm 2$ |  |  |  |
| task4 | 0 0 $\pm 0$ | $\mathbf{12}$ $\pm 14$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ |  |  |  |
| task5 | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 0$ | $\mathbf{0}$ $\pm 1$ |  |  |  |
| overall | 0 0 $\pm 0$ | $\mathbf{4}$ $\pm 3$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | 0 0 $\pm 0$ | $1$ $\pm 1$ |  |  |  |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Please, also refer to the questions brought up in the "Weaknesses" section.

- **1** - Why not also collect an "exploratory" dataset for the AntSoccer task? Is it due to the lack of coverage of states where the agent is dribbling the ball?
- **2** - In the locomotion and manipulation tasks, how exactly is the goal state provided to the agent? Does it also include the final pose of the agent/robot? In that case, the task goal becomes a convolution of an environmental change (for example, a set of objects in a specific location in the Scene task) and a final pose of the agent?
- **3** - I found it quite interesting the comment at the end of Section 8.2 ("this suggests that we may need to prioritize coverage much more than optimality when collecting datasets for offline GCRL in the real world") as this seems to go against the current trends in large scale collection of interaction data in robotics, where the data are collected across multiple environments/embodiments but always by expert-level demonstrators (e.g., [1]). Can the authors make any parallelism between the case of offline GCRL and other large-scale data collection efforts?


**Reference**:
- [1] Collaboration, Open-X. Embodiment, et al. "Open X-Embodiment: Robotic learning datasets and RT-X models." arXiv preprint arXiv:2310.08864 1.2 (2023).
soundness: 4
strengths: - **Originality**:
  - Recently, there has been an extensive push towards the development of benchmarks that focus on different aspects of reinforcement learning (e.g., for offline RL [1], [2]). However, as noted by the authors, there existed no centralized evaluation suite for algorithms in offline GCRL and the community often resorted to different set of tasks (not tailored for, making difficult to access the true performance of the proposed methods. As such, the development of a benchmark that tackles specific research issues in offline GCRL is most welcomed and, to the best of my knowledge, novel.

- **Quality**:
   - The work presented here is of high quality: the proposed benchmark contains a set of diverse tasks that address specific problems in offline GCRL research and high-quality, fine-tuned implementations of recently proposed algorithms for offline GCRL (that often surpass the performance of the original versions). Furthermore, the authors collect a wide range of datasets for each task, with different levels of sub-optimality.

- **Clarity**:
   - The current version of the work is well-written, with no major typos (that I could detect). Furthermore, the document is easy to read, with self-explanatory figures and tables. I would, however, recommend some refrain when using adjectives such as "cool" (Section 6, page 4) or "exciting" (Section 6, page 4), due to their intrinsic subjectivity.

- **Significance**:
   - This work proposes a novel benchmark for offline GCRL. Given the relevance of the topic for RL research, the work can have substantial impact by providing a standardized suite for evaluation of novel algorithms.


**References**: 

- [1] Seno, Takuma, and Michita Imai. "d3rlpy: An offline deep reinforcement learning library." Journal of Machine Learning Research 23.315 (2022): 1-20.
- [2] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).
summary: In this work the authors present OGBench, a novel benchmark for offline goal-conditioned reinforcement learning (GCRL). The authors motivate the need for such benchmark, due to the lack of a standardized evaluation suite for works in offline GCRL and the increasing interest from the research community on the topic. The authors describe the main challenges tackled with OGBench: learning from sub-optimal data, goal stitching, long-horizon reasoning and stochastic environments. To tackle these challenges, the authors describe the design principles of the benchmark and consider a wide range tasks (locomotion, manipulation and drawing) across multiple environments. Additionally, the authors collect several different datasets per environment, considering different sub-optimality conditions of the behavior policy. Finally, the authors evaluate several literature-standard algorithms in OGBench, contributing their implementation as well, and discuss their performance and other findings.
weaknesses: My concerns with the current version of the work are the following (none of them particularly major):
- Currently, it is challenging to assess the varying levels of novelty present in the environments available in OGBench. Throughout Section 7, I found myself questioning whether each environment was accessible elsewhere (in another benchmark or repository) or if it was unique to this benchmark. I do understand that some tasks (e.g., AntMaze) are heavily inspired by/available in other benchmarks. It would strengthen the originality of the benchmark if the authors clarified the level of novelty in each environment.
- Currently the paper lacks any discussion about the limitations of the current benchmark and plans for future extension (for example, one potential extension could tackle robustness to data corruption as explored in [1], or integrating scenarios with multiple modalities).
- Is not clear how the five evaluation goals per task were defined or what criteria was used for their selection. Additionally, why only 5 goals? Since GC policies should be able to reach any (viable) state from any state, it is unclear why the authors selected such a set of small goals.


**References**

- [1] Yang, Rui, et al. "Towards Robust Offline Reinforcement Learning under Diverse Data Corruption." The Twelfth International Conference on Learning Representations.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: * The expert data are collected from policies trained by RL. How can we assume the policies provide sufficient optimality? Since the all the tasks are essentially planning towards a goal, would running motion planning/search algorithms with certain guarantees give better data quality?

* Can the benchmark add a study on the representation and the transferability to assess the idea of using goal-conditioned tasks as a general-purpose representation learning?
soundness: 3
strengths: * Well-motivated research efforts and a solid execution with many considerations customised to goal-conditioned offline RL.

* A good coverage of task variations and challenges to state-of-the-art algorithms based on different principles.

* Proper task difficulties demonstrated from the results. Could be promising to spur new algorithmic research.

* Decent writing with clear presentation and well organised flow of narratives.
summary: The paper proposes a new suite of tasks for benchmarking goal-conditioned offline RL algorithms. The proposed benchmarks are built with consideration specific to the goal-conditioned navigation tasks, with an emphasis on evaluating relevant task capabilities such as stitching, long-horizon reasoning, dealing with stochastic and visual input. The tasks cover a range of difficulty levels and types, including ant and humanoid locomotion, multi-step manipulation and a painting task with an high-dimension state space entailing combinatorial search. Offline policy data are also collected with various policies to emulate the data sub optimality. The benchmark is shown to be challenging for state-of-the-art and helpful in analysing strengths and limitations on investigated task capabilities. Code and policy scripted are provided and with a minimum dependencies on open-sourced mujoco and PyTorch libraries.
weaknesses: * The benchmark is motivated for the generality of goal-conditioned navigation tasks, aiming at learning transferrable representations for down-stream tasks. However, this is not reflected in the task design and experimental results. It would be better to include some functionalities to facilitate analysing and transferring the learned latent representations, with a report on a few SotA algorithms' performance on this aspect.

* The tasks seem to only challenge policies with different initial states but not the shift of transition dynamics.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: See the weakness part.
soundness: 3
strengths: * Offline GCRL is an important research direction, and this work addresses the current lack of a challenging and comprehensive benchmark.

* This benchmark evaluates GCRL algorithms across diverse aspects, including stitching, stochastic environments, high-dimensional control, and long-horizon control.

* The benchmark includes popular baseline results, highlighting some unsolved tasks and areas for improvement.
summary: This paper introduces OGBench, a comprehensive benchmark for the standardized evaluation of offline GCRL. OGBench provides a diverse range of environments and datasets, covering areas such as locomotion, manipulation, and drawing, to assess key capabilities like trajectory stitching, long-horizon reasoning, image-input handling, and managing stochasticity. The authors evaluate six offline GCRL algorithms using this benchmark, designed to highlight each algorithm's unique strengths and weaknesses across different tasks. OGBench is optimized for computational efficiency and user-friendliness, allowing researchers to easily test and refine their ideas, thereby advancing the field of offline GCRL.
weaknesses: Some benchmark designs need further discussion or improvement.

* The evaluation seems limited to five tasks. Does this mean there are only five evaluation goals? If so, this number is restricted compared to several prior goal-conditioned tasks such as the Fetch environments.

* The design of the transparent arm in pixel-based manipulation tasks seems unrealistic. It would be better to let users choose between a transparent or solid option. Additionally, demonstrating how transparency affects performance could be helpful.

* The goal information is missing in Table 6. Is the goal in image-based tasks another image provided to the agent, or is it just a transparent bot as shown in the manipulation task figures? 

* Can the benchmark support language instructions as a special type of "goal"? This would be interesting given the current research focus on LLMs.

* The stochastic task appears confined to the teleport domain. More stochastic settings that might be relevant to real-world scenarios should be explored, such as random observation perturbation during both data collection and evaluation. 

* Current baselines still use MLP networks as backbones. Could the authors consider implementing more advanced architectures, such as transformers? Are there any limitations in the current tasks that more sophisticated network architectures might address?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It would be better to retain one or two decimal places in Table 2. In benchmarks such as D4RL (Fu et al., 2021) and offline robotics (Yang et al., 2022), the success rate/score provided by decimal places is sometimes critical for comparisons. 
2. Could the authors clarify in the paper whether a rollout is considered successful if the goal is achieved at the last state, or if any state within the rollout reaches the goal? 
3. The authors should include a mention of GOPlan (Wang et al., 2024), which applies model-based planning with a generative adversarial network as the prior policy, in the related work. Additionally, GOPlan and GoFar (Ma et al., 2022) evaluate algorithm robustness in stochastic offline GCRL settings and should be discussed in Section 5.4.


**References**

Fu, J., Kumar, A., Nachum, O., Tucker, G., \& Levine, S. (2021). D4RL: Datasets for Deep Data-Driven Reinforcement Learning. 

Ma, J. Y., Yan, J., Jayaraman, D., \& Bastani, O. (2022). Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression. NeurIPS.

Wang, M., Yang, R., Chen, X., Sun, H., Fang, M., \& Montana, G. (2024). GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models. TMLR.

Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., … Zhang, C. (2022). Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ICLR.
soundness: 3
strengths: 1. They are very useful benchmarks to evaluate offline GCRL methods. 
2. The authors evaluate $6$ standard offline GCRL methods on the proposed benchmarks, establishing a solid baseline for comparison and further development.
summary: This paper contributes to offline GCRL by introducing a suite of high-quality benchmarks. These benchmarks systematically assess offline GCRL algorithms' performance in learning from suboptimal data, goal stitching, long-horizon reasoning, and handling stochasticity, providing a useful tool for advancing offline GCRL research.
weaknesses: The positioning of this paper is very similar to a previous paper - D4RL (Fu et al., 2021), which was well-known but rejected by ICLR 2021 (https://openreview.net/forum?id=px0-N3_KjA). 

This paper makes a clear contribution by providing valuable benchmarks, but it does not introduce novel ideas. However, given its potential importance to the field, I lean towards assigning a slightly positive score.

Meta-review: This paper introduces a dataset for goal-conditioned RL. Reviewers generally agree that it is an important problem and that the dataset could be useful. Reviewers also found the paper to be well-written. I agree with this sentiment and think that the dataset can complement popular RL datasets like D4RL. One concern raised was that only simple model classes (e.g., MLP and CNN) were raised. I think this is okay for a benchmark paper although it would have been great to see more common transformer-style architectures. I would urge authors to consider adding this for revision.

Overall, I recommend acceptance.

Decision: Accept (Poster)