OGBench: Benchmarking Offline Goal-Conditioned RL
Authors: Seohong Park
Kevin Frans
Benjamin Eysenbach
Sergey Levine
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
Primary area: reinforcement learning
TL;DR: A new benchmark for algorithms research in offline goal-conditioned RL.

Abstract: Offline goal-conditioned reinforcement learning (GCRL) is a major problem in reinforcement learning (RL) because it provides a simple, unsupervised, and domain-agnostic way to acquire diverse behaviors and representations from unlabeled data without rewards. Despite the importance of this setting, we lack a standard benchmark that can systematically evaluate the capabilities of offline GCRL algorithms. In this work, we propose OGBench, a new, high-quality benchmark for algorithms research in offline goal-conditioned RL. OGBench consists of 8 types of environments, 85 datasets, and reference implementations of 6 representative offline GCRL algorithms. We have designed these challenging and realistic environments and datasets to directly probe different capabilities of algorithms, such as stitching, long-horizon reasoning, and the ability to handle high-dimensional inputs and stochasticity. While representative algorithms may rank similarly on prior benchmarks, our experiments reveal stark strengths and weaknesses in these different capabilities, providing a strong foundation for building new algorithms. Project page: https://seohong.me/projects/ogbench

Published as a conference paper at ICLR 2025
OGBENCH:
BENCHMARKING OFFLINE GOAL-CONDITIONED RL
Seohong Park' Kevin Frans'
Benjamin Eysenbach² 2
Sergey Levine'
Uniiversity of California, Berkeley
Princeton University
seohongoberkeley.com edu
ABSTRACT
Offline goal-conditioned reinforcement learning (GCRL) is major problem in re- inforcement learning (RL) because it provides a simple, unsupervised, and domain- agnostic way to acquire diverse behaviors and representations from unlabeled data without rewards. Despite the importance of this setting, we lack standard bench- mark that systematically evaluate the capabilities of offline GCRL algorithms. In this work, we propose OGBench, a new, high-quality benchmark for algorithms research in offline goal-conditioned RL. OGBench consists of 8 types of envi- ronments, 85 datasets, and reference implementations of 6 representative offline GCRL algorithms. We have designed these challenging and realistic environments and datasets to directly probe different capabilities of algorithms, such as stitch- ing, long-horizon reasoning. and the ability to handle high-dimensional inputs and stochasticity. While representative algorithms may rank similarly on prior bench- marks, our experiments reveal stark strengths and weaknesses in these different capabilities, providing strong foundation for building new algorithms. Project page: https: pettps:/sseohie //ssssooongg...oo me/ ptwptt//cts////eeeeeeoeesscsssse projeccss/obbecc ogbench
A
FLAG
bunanoi
visual
cube
pazzle
powderworld
Figure 1: OGBench Overview, OGBench provides a variety of state- and paxel-based locomotion, manipulation, and drawing tasks that are designed to exercise diverse challenges in offline goal-conditioned RI, such as stitching, long- hor-horizon reasoning, and stochastic control.
MOTIVATION
Why offline goal-conditioned reinforcement learning (RL)? The enduring trend in modern ma- chine learning simplify domain-specific assumptions and scale up the data. In computer vision and natural language processing, the strongest general- purpose models are trained via simple unsu- pervised objectives on raw, unlabeled data, such as next-token prediction, contrastive learning, and masked auto-encoding. What analogous paradigm could enable data-driven unsupervised learning for reinforcement learning? Ideally, such a framework should be able to produce from data a generalist policy that can be directly queried adapted to solve variety of downstream tasks, much like how generative language models trained via next-token prediction can be casily adapted to everyday tasks.
We posit that natural analogy to data-driven unsupervised learning in RL is offfine goal-conditioned RL (GCRL). The objective of offline goal-conditioned RL is fully unsupervised, remarkably simple, and requires domain knowledge: it merely aims to learn to reach any state from any other state in the dataset in the fewest number of steps. However, mastering this simple objective is exceptionally difficult: the agent needs only acquire diverse skills to efficiently navigate the state space, but

---

Published as a conference paper at ICLR 2025
also have deep, complete understanding of the underlying world and dataset. As result, offline goal-conditioned RL yields a highly capable general-purpose multi-task policy as well as rich, useful representations that can be adapted to solve a variety of downstream tasks (Ghosh et al., 2023; Kim et al., 2024). Indeed, for such simplicity and generality, interest in (offline) goal-conditioned RI. has recently surged to the extent that even standalone workshop on goal-conditioned RI. was held at machine learning conference."
Why a new benchmark? Despite the importance of and increasing interest in offline goal-conditioned RL, we currently lack a standard benchmark that can systematically assess the capabilities of offline GCRL algorithms, such as the ability to stitch, perform oong-hoiizon reasoning, and handle stochasticity. Prior works in offline conl-condi RL (Eysenbach et al., 2022; Ma et al., 2022; Park et al., 2023; Myers et al., 2024) have mainly used either existing datasets for standard offline RL tasks without modification (e.g., D4RL (Fu et al., 2020)), relatively simple goal-conditioned tasks (e.g., Fetch (Plappert et al., 2018)), or tasks tailored to demonstrate the individual abilities of the proposed methods. This often results in limited evaluation. Prior works often evaluate their multi-task policies only on a single task (when using datasets not originally designed for offline GCRL), or learn relatively simple behaviors. While there exist some prior tasks tailored to evaluate individual properties of offline GCRL such as stitching or generalization ( (Yang et al., 2023; Ghugare al., 2024), we lack comprehensive, standardized benchmark that exhaustively assesses various properties of offline GCRL. algorithms with diverse, challenging tasks.
Therefore, we introduce benchmark named Offfine Goal-Conditioned RL Benchmark (OGBench) in this work (Figure 1). The primary goals of this benchmark are to facilitate algorithms research in offline goal-conditioned RL and to provide a set of complex tasks that can unlock the potential of offline GCRL. Our benchmark introduces 8 types of environments and 85 datasets across robotic locomotion, robotic manipulation, and drawing, and provides well-tuned reference implementations of 6 representative offline GCRL methods. These datasets, tasks, and implementations are carefully designed. Tasks are designed in way that complex behaviors can naturally emerge when successfully solved, and that they pose diverse algorithmic challenges in offline GCRL, such as goal stitching. stochastic control, long-horizon reasoning, and more. Dataset and task difficulties are carefully adjusted to highlight stark contrasts between algorithms across multiple criteria. The entire benchmark is designed to minimize unnecessary computational overhead and maximize usability such that any researcher can easily iterate and evaluate new ideas. We believe OGBench serves as solid foundation for developing ideal algorithms for goal-conditioned and unsupervised RL from data.
2 PROBLEM SETTING
The offline goal-conditioned RL problem is defined by a controlled Markov process M = (S,A, u, p.,)) (ie., Markov decision process (MDP) without rewards) and an unlabeled dataset D, where S denotes the state space, A denotes the action space, p(s) E A denotes the initial state distribution, and p(s' s, S x A + (SS) denotes the transition dynamics function. A(X) denotes the set of probability distributions defined a set X. The dataset D {[(")) (1,22,,,.. consists of
unlabeled trajectories ((n)_ (s0 40
(m)
(n) (n) ST. (m))
The objective of offline goal-conditioned RL learn to reach any state from any other state in the minimum number of time steps. Formally, offline GCRL aims to learn goal- condiiondi policy
Ta 9,9): S x S + (AA) that maximizes the objective Er~p(r19) gg) Ei= n (s)) for all g S, where T E N denotes the episode horizon,
(0, 1) denotes the discount factor, p(r g) denotes the distribution given by p(r g) = uu(s)) n (ao s0, p)( s0, ao) ... -PST (T--1, IT-1), and 6.).) denotes the Dirac delta "function"" 3* at g. Note that we use the entire state space as the goal space (i.e., a goal is simply full state, not part of the state like only the x-y position of the agent). This choice makes the objective fully unsupervised, making it suitable for domain- aagnosicc training from unlabeled data. Our goal in this paper is to propose new benchmark in offline GCRL. That is, formally speaking, to specify the dynamics and dataset for each task we introduce.
'nttpa:/ / tattpp://goal-eee conditiiiioono.. ti pioodditiinedddr github. 1o/2023/
We denote placeholder variables in gray throughout the paper.
'In discrete MDP, (s) is equal to an indicator function 1 (s). In a continuous MDP, it is technically not well-defined in its current form. It can be made precise using theserc- notation or the distribution theory, but we choose avoid them for simplicity.
2

---

Published as a conference paper at ICLR 2025
Table I: Properties of benchmark tasks. We summarize the properties of benchmark tasks commonly used in
prior works (above the line) and our OGBench tasks (below the line).
Benchmark Tosk
Type'
Longest Task' & Subtasks' Test Stitching?
Have Stoch. Tasks!" Support Pixels?" Multi-Goal" Dependency
DiRL AntMae
Loco.
=: 000
X
XOX
MulaCo
DARL Kitchhn
Manip.
250
MuloCo
Roboverse
Manip.
PyBullet
Fetch
Manip.
t: 50
X
O
MulaCo
PointMaze (ours)
Loco.
o: 600
O
O
X
O
MulaCo
AntMare (ours)
Loco.
1000
O
O
MuloCo
HuramnidMazze (ours)
MulaCo
AntSnccer (curs)
Locn.
1000
X
O
MulaCo
Cube (ouns)
Manip.
2: 400
1-4
O
MulaCo
Sccne (ours)
Manip.
22400
2-8
MuloCo
Puzzie
Manip.
2-24
MulaCo
Powderworld (ours)
Dras:
100
NumPy
Envireementt type (lugomotion, manipulation, or drawing). "The (approximalie) minimum number of enviromment sleps to solve the longest task.
The number of stomic belavions (e.g. pick-and-pllan in each munipulation task. "Does contain tisks that require goul stiiching?
contain with stochastic dynamics? "Does suppoct pixel-based observation?
? Does it multiple goals for evalustion?
The muin dependency the benchmark.
3 HOW HAVE PRIOR WORKS BENCHMARKED OFFLINE GCRL?
While many excellent offline GCRL algorithms have been proposed so far, the community currently lacks a standardized way to evaluate their performance, unlike other fields in RL (Brockman et al., 2016; Tassa et al., 2018; Fu et al., 2020; Terry et al., 2021). Tasks used by prior works in offline GCRL are often limited for providing proper evaluation for various reasons. For example, many works directly use tasks in existing offline RL benchmarks (not necessarily designed for offline goal-conditioned RL), such as D4RL AntMaze and Kitchen (Fu et al., 2020). However, since the tasks are designed for single-task offline RI, they often evaluate their multi-task policies only on the single, original goal when using these tasks (Eysenbach et al., 2022; Park et al., 2023; Zheng et al., 2024b; Myers et al., 2024), which results in limited evaluation. Some employ online GCRL tasks provided by Plappert et al. (2018) (e.g.. Fetch tasks) with policy-collected datasets (Ma et al., 2022; Yang al., 2023), but these tasks are mostly atomic (e.g., single pick-and-place and do not sufficiently address various challenges in offline GCRL, such as long-horizon reasoning and goal stitching. While Roboverse (Fang et al., 2022; Zheng et al., 2024a) provides pixel-based manipulation tasks, the tasks are still relatively atomic and it requires installing multiple, fragmented dependencies, making it less approachable for researchers. Some prior works construct bespoke tasks and datasets to individually study the important and specific features of their algorithms (e.g., stitching (Ma et al., 2022: Ghugare al., 2024; Wang et al., 2024) and stochasticity (Myers al., 2024)). but it is often not entirely clear how algorithms compare to one another or if the new capabilities of new methods (e.g, stitching) come at the loss of other capabilities. These limitations of previous evaluation tasks have motivated us to create new benchmark. In this work, we introduce set of diverse tasks that cover various challenges in offline GCRL, enabling a much more thorough and multi-faceted evaluation than previous tasks. We summarize the properties of the previous tasks and our new tasks in Table and refer to Appendix E for further discussion on related work.
4 OVERVIEW OF OGBENCH
We now introduce our benchmark, Offline Goal-Conditioned RL Benchmark (OGBench). The primary goal of this benchmark is to provide tasks and datasets unlock the full potential of offline goal-conditioned RL. To this end, we pose diverse challenges in offline GCRL throughout the benchmark in such a way that researchers can easily test and iterate on algorithmic ideas, and that complex. intriguing behaviors can naturally emerge when successful. OGBench consists of 8 types of environments, 85 datasets, and reference implementations of representative offline GCRL algorithms. In the following sections, we first describe the challenges in offline GCRL (Section 5) and then outline our core design philosophies (Section 6). We next introduce the tasks and datasets (Section 7) and present the benchmarking results of the current algorithms (Section 8).
5 CHALLENGES IN OFFLINE
RL
Offline goal-conditioned RL, despite its simplicity, is challenging problem. Here, discuss the major challenges in offline GCRL, which will motivate the design choices in our benchmark tasks.
(1) Learning from suboptimal, unstructured data: An ideal offline GCRL algorithm should be able to learn an effective multi-task policy from diverse and suboptimal data. This is especially important, considering that suboptimal (yet diverse) data is much cheaper to collect than curated,
3

---

Published as a conference paper at ICLR 2025
expert datasets (Lynch et al., 2019), and that the very use of large, diverse, unstructured data is one of the foundations for the success of modern machine learning. Reflecting this challenge, we provide datasets with high diversity and varying suboptimality in this benchmark to challenge the capabilitiess of offline GCRL algorithms.
Another important challenge is to stitch the initial and final states of different trajectories to learn more diverse behaviors. We call this "goal stitching." Goal stitching is different from "regular" stitching in offline RL, which applies only when the dataset is suboptimal. Unlike regular stitching, goal stitching applies even when the datasel only consists of optimal, expert trajectories, because we can often acquire more diverse goal-reaching behaviors by stitching multiple trajectories together, regardless of their optimality. For instance, an agent can stitch two atomic pick- and-place behaviors to sequentially move two objects in a single episode, even when the dataset does not contain any double pick-and-place behaviors. Goal stitching is crucial to learning diverse behaviors in many real-world applications with high behavioral diversity and large state spaces. In our benchmark, we introduce many tasks with large state spaces to assess the ability to stitch goals.
(2) Goal stitching:
(3) Long-horizon reasoning: Long-horizon reasoning refers to the capability of navigating from a starting state to goal state that is many steps apart. This challenge is important in many real-world tasks like autonomous driving or assembly, which may require several hours of continuous control or achieving dozens of subtasks. To substantially challenge the long-horizor reasoning ability of offline GCRL methods, we introduce tasks that are more than 5 times longer than previously used ones in terms of both the episode length and the number of subtasks (Table 1).
(4) Handling stochasticity: Another prominent challenge in offline GCRL is the ability to deal with stochastic environments. Correctly handling stochasticity is very important in practice, because virtu- ally any real-world environment is stochastic due to partial observability. Yet, many works in offline GCRL assume deterministic dynamics to exploit the metric structure of temporal distances (Wang et al., 2023) or to enable hicrarchical control (Park et al., 2023), at the cost of being optimistically biased in stochastic environments (Wang et al., 2023; Park et al., 2023). Correctly handling environ- ment stochasticity while fully exploiting the recursive subgoal structure of GCRL remains an open problem. Since most previous tasks used to evaluate offline GCRL methods have deterministic dy- namies (Table 1), we introduce several challenging tasks with stochastic dynamics in this benchmark.
6 DESIGN PRINCIPLES
Next, we discuss the design principles underlying our benchmark tasks. The tasks are intended to exercise the major challenges in offline GCRL. in the previous section, while providing a set of high- quality tasks that provide not only toolkit for algorithms research and evaluation, but also a platform for vividly illustrating the capabilities of offline GCRL with compelling and complex domains.
(1) Realistic and exciting tasks: The tasks should be realistic yet exciting enough while posing diverse challenges in offline goal-conditioned RL. Imagine a robot arm watching random movements of puzzle and then solving or an agent painting cool pictures using different types of brushes. In this benchmark, we design new tasks such that these kinds of exciting behaviors can naturally emerge when an RI. agent properly stitches different trajectory segments together (up to 24; see Table 1) or successfully generalizes. At the same time, we make sure our tasks exhaustively cover major challenges in offline goal-conditioned RL, such as long-horizon reasoning, stochastic control, and combinatorial generalization via goal stitching (Section 5).
shoo at test time, humanoid robot navigating through a labyrinth,
(2) Appropriate difficulty: The tasks and datasets should have appropriate levels of difficulty to properly evaluate different algorithms. In other words, they should be of high quality for benchmark- ing. No matter how intricate or compelling task is, it will fail to provide a useful signal for bench- marking if it too easy, too hard, unsolvable from the given dataset, or does not clearly distinguish between more or less effective methods. In this work, we carefully curate and adjust the difficulty of each task and dataset such that they can provide effective guidance for algorithms research. For some tasks, we provide multiple versions with varying difficulty, all the way up to tasks that are diffi- cult to solve with current methods, so that the same benchmark can continuously be used to develop new methods even in the future. Our rule of thumb is to have, for each type of tasks, at least one task where the current stae--of-the-as offline GCRL method achieves a success rate of 20-30%. This ensures that the task is solvable from the dataset while leaving significant room for improvement.
4

---

Published as a conference paper at ICLR 2025
(3) Controllable datasets: The benchmark should provide tools to control and adjust datasets for scientific research and ablation studies. Verifying the effectiveness of algorithms in real-world problems is surely important. However, for algorithms research, is equally, if not more, important to provide analysis tools to enable a rigorous, scientific understanding of challenges and algorithms. Hence, instead of employing fixed, human-collected data, which does not always provide clear benchmarking signals for algorithms research, we choose to focus on simulated environments and synthetic datasets that we have full control of, and provide tools reproduce and adjust them easily. We note that many algorithmic ideas in RL that have made major impact, such as DQN (Mnih et al., 2013), PPO (Schulman et al., 2017), and CQL (Kumar al., 2020), were originally developed in simulated environments. Even in natural language processing, studies on synthetic, controlled datasets have revealed the mechanisms and limitations of language models with scientific evidence (Allen- Zhu & Li, 2023a), and these insights have transferred to real scenarios (Allen-Zhu & Li, 2023b). We demonstrate how such controllability of datasets reveals challenges and design principles in offline GCRL in Section 8.2.
(4) Minimal compute requirements: The tasks should be designed to minimize unnecessary computational overhead that as many researchers possible, including those from small labs and underprivileged backgrounds, can quickly iterate on their new algorithmic ideas. This does not mean that the tasks should be easy (indeed, some of tasks very challenging (but solvable)!); means the benchmark should focus mainly on algorithmic challenges (e-g., not requiring high- resolution image processing). In our benchmark, we provide both state- and pixel-based observations whenever possible, and minimize the size of image observations (up to 64 x 64 x 3) to reduce the computational burden. Moreover, we carefully adjust colors, transparency, and lighting for image- based tasks to enable pixel-based control without high-resolution images or multiple views.
(5) High code quality: The reference implementations should be very clean and well-tuned so that researchers can directly use our implementations to build their ideas, and the benchmark should be very easy to set up. Our benchmark environments only depend on MuJoCo (Todorov et al., 2012) and do not require any other dependencies (Table 1). For reference implementations, we minimize the number of file dependencies for each algorithm, largely following the spirit of the single-file implementations of recent RL libraries (Huang al., 2022; Tarasov et al., 2023), while maintaining minimal amount of additional modularity. We also extensively test and tune different design choices and hyperparameters for each offline GCRL algorithm, the degree that several methods achieve even better performances than their original performances reported on previous benchmarks (Table 3).
7 ENVIRONMENTS, TASKS, AND DATASETS
We now introduce the environments, tasks, and datasets in our benchmark. They can be broadly categorized into three groups: locomotion, manipulation, and drawing. We provide a separate validation dataset for each dataset, and most tasks support both state- and pixel-based observations. Videos are available at https: // //sohong/././/.//////o///./ wwtpp/oects///bbeeeeeee ogbench.
Evaluation. Each task in OGBench accompanies five pre-defined state-goal pairs for evaluation (Appendix H). Performance is measured by the average success rate across the five evaluation goals. For each pre-define state-goal pair, we perform multiple rollouts with slightly randomized initial and goal states. In each evaluation episode, goal g E S (which is simply another state; see Section 2) is given the agent, and the episode immediately terminates when the agent reaches the goal. Each task has its own goal success criterion, which we describe in Appendix G.I.
7.1 LOCOMOTION TASKS
We provide four types locomotion environments, PointMaze, AntMaze, HumanoidMaze, and AntSoccer, with diverse variants. These environments are designed to test the agent's long-horizon and hierarchical reasoning abilities. They are based on the MuloCo simulator (Todorov et al., 2012).
PointMaze (pointmaze), AntMaze (antmaze), and HumanoidMaze gation is one of the most widely used tasks for bench- marking offline GCRL algorithms. We provide three different types of maze navigation tasks: Point- Maze, which involves controlling a 2-D point mass, AntMaze, which involves controlling quadrupedal
Maze navi-
pointmaze
antnaze
humano dmaze
5

---

Published as a conference paper at ICLR 2025
Ant agent with 8 degrees of freedom (DoF), and HumanoidMaze, which involves controlling much more complex 21-DoF Humamoid agent. The aim of these tasks is to control the agent to reach goal location in the given maze. The agent must learn both the high-level maze navigation and low-level locomotion skills that involve high-dimensional control, purely from diverse offline trajectories.
In our benchmark, substantially extend the original PointMaze and AntMaze tasks proposed by D4RL (Fu et al., 2020). Unlike the original D4RL tasks, which only support Point and Ant agents and do not challenge stitching', stochasticity, or pixel-based control, support Humanoid control, pixel-based observations, and multi-goal evaluation, while providing more challenging and diverse
types mazes and datasets. The supported maze types are as follows:
medium
large
giant
teleport
medium: This the smallest maze, with the same layout as the original medium maze in D4RL.
large: This is larger maze, with the same layout as the original large maze in D4RL.
giant:
This is the largest maze, twice the size of large. has same size as the previous antmaze-ultra maze by Jiang et al. (2023), but its layout is more challenging and contains longer paths that require up to 3000 environment steps (in the case of Humanoid), This maze is designed to substantially challenge the long-horizon reasoning capability of the agent.
teleport: This maze is specially designed to challenge the agent's ' ability to handle environ- ment stochasticity. It has the If the agent enters a black hole, is immediately sent to a randomly chosen white hole. How- ever, since one of the three white holes dead-end, there is always a risk in taking a teleporter. The agent therefore must learn to avoid the black holes, without being optimistically biased by "lucky" outcomes.
size large, but contains multiple stochastic teleporters.
On these mazes, collect datasets with low-level directional policy trained via SAC (Haarnoja et al., 2018) and a high-level waypoint controller. For each maze type, we provide three types of datasets that pose different kinds of challenges (the figures below show example trajectories in these datasets):
ZAa Pr
navigate
stitch
explore
navigate:
This the standard dataset, collected by a noisy expert policy that navigates the maze by repeatedly reaching randomly sampled goals.
stitch: This dataset designed to challenge the agent's stitching ability. consists of short geal-reaching trajectories, where the length of each trajectory is at most 4 cell units. Hence, the agent must be able to stitch multiple trajectories (up to 8) together to complete the tasks.
explore: This dataset is designed test whether the agent can learn navigation skills from extremely low-quality (yet high-coverage) data. It consists of random exploratory trajectories, collected by commanding the low-level policy with random directions eesaaplled every 10 steps, with a large amount of action noise.
We provide two types of observation modalities:
States: This is the default setting, where the agent has access to the full low-dimensional state representation, including its current z-y position.
TFA
Pixels (visual): This requires pure pixel-based control, where the agent only receives 64 x 64 x RGB images rendered from
colored maze
pixela
*See Ghugare al. (2024) for the details about this point.

---

Published as a conference paper at ICLR 2025
third-person camera viewpoint. Following Park et al. (2023), color the floor to enable the agent to infer its location from the images, obviating the need for a potentially expensive memory component. However, unlike Park al. (2023), which additionally provides proprioceptive information, we do not provide any low-dimensional state information like joint angles; the agent must learn purely from image observations.
AntSoccer (antsoccer). To provide more diverse type of locomo- tion task beyond simple maze navigation, we introduce a new locomo- tion task, AntSoccer. This task involves controlling Ant agent to dribble a soccer ball. It is inspired by the quadruped-fetch task in the DeepMind Control suite (Tassa et al., 2018). AntSoccer is signifi- cantly harder than AntMaze because the agent must also carefully con- trol the ball while navigating the environment. We provide types: arena, which is an open space without walls, and medium, which is the same maze the medium we provide navigate and stitch. The navigate datasets consist of trajectories where the agent repeatedly approaches the ball and drib- bles it to random locations. The stitch datasets consist of two dif- ferent types trajectories, maze navigation without the ball and drib- bling with the ball near the agent, so that stitching is required to com- plete the full task. AntSoccer only supports state-based observations.
antsoccer
in AntMaze. For datasets,
arena
medium
7.2 MANIPULATION TASKS
We provide a manipulation suite with three types robotic manipulation tasks, Cube, Scene, and Puzzle, with diverse difficulties and complexities. They are designed to test the agent's 's object manip- ulation, sequential generalization, and combinatorial generalization abilities. These environments are based on the MuJoCo simulator (Todorov et al., 2012) and 6-DoF URSe robot arm (Zakka et al., 2022). On these tasks, we provide "play"-style datasets (play) (Lynch et al., 2019) collected by non- Markovian expert policies with temporally correlated noise. To support more diverse types of re- search (e.g., dataset ablation studies), we additionally provide more noisy datasets (noisy) collected by Markovian expert policies with uncorrelated Gaussian noise, which we describe in Appendix D.
For all manipulation tasks, we support both state- state-based observations and pixel-based observations with 64 x 64 x 3 RGB camera images. For pixel observations, we adjust colors and make the arm transparent ensure full observability. The transparent appear challenging, but the colors and transparency are carefully adjusted minimize difficulties in visual perception, to the extent that some methods achieve even better performance with pixels (see Table 2).
in the figure might
states
pixels
Cube (cube). This task involves pick-and-pe place manipulation of cube blocks, whose goal to control a robot arm to arrange cubes into des- ignated configurations. We provide four variants, single, double, triple, and quadruple, with different numbers (1-4) of cubes. We provide "play"-style datasets collected by scripted policy that repeat- edly picks a random block and places it in other random locations or another block. At test time, the agent is given goal configurations that require moving, stacking, swapping, or permuting cube blocks. Hence, the agent must learn only generalizable multi-object pick- and-place behaviors from unstructured random trajectories in the dataset, but also long-tery plans achieve the tasks (e.g., permuting blocks requires non-to onitiiival sequential and logical reasoning).
single
double
triple
quadruple
7

---

Published as a conference paper at ICLR 2025
Scene (scene). This task is designed to challenge the sequential, long-horizon reasoning capabilities of the agent. It involves manip- ulating diverse everyday objects, such cube block, a window, a drawer, and two button locks, where pressing a button toggles the lock status of the corresponding object (the drawer window). We provide "play"-styla datasets collected by scripted policies that ran- domly interact with these objects. test time, the agent com- manded to arrange the objects into desired configuration. Evalua- tion tasks require significant degree of sequential reasoning: for in- stance, some tasks require unlocking the drawer, opening it, putting cube the drawer, and closing again (see the figure above), and the longest task involves eight atomic behaviors. Hence, the agent must be able to plan and sequentially combine learned manip- ulation skills.
wnlock draser
open draver
cube in dravez
close draver
scene
Puzzle (puzzle). This task designed to test the combinatorial gen- eralization abilities of the agent. It requires solving the "Lights Out" puzzle* with a robot arm. The puzzle consists of a two-dimensional array of buttons (e.g., 4 grid), where pressing button toggles the colors of the pressed button and the buttons adjacent to (typi- cally four, except on the edges and corner;; see videos). The goal is to achieve a desired configuration of colors (e.g., turning all the but- tons blue) by pressing an appropriate combination buttons. Since these buttons are implemented in the MuloCo simulator, the agent must control a robot arm to physi- cally press the buttons. We provide four levels of difficulty. 3x3, 4x4, 4x5, and 4x6, with different grid sizes. The datasets are collected by scripted policy that randomly presses buttons in arbitrary sequences. Given the enormous state space of this task (with up to 224 16,777, 776,777,1 216 distinct button states), the agent must achieve combinatorial generalization while mastering low-level continuous control. Some evaluation task the hardest puzzle requires pressing more than 20 buttons, which also substantially challenges the long- Long-horizon reasoning capabilities of the agent. This might sound very challenging (and it is!), but we provide different levels and enough data to ensure that they pro- vide meaningful research signals and are solvable (see the results in Section 8).
puzzle
7.3 DRAWING TASKS
Powderworld (powderworld). To provide more diverse tasks be- yond robotic locomotion or manipulation, we introduce drawing task, Powderworld (Frans & Isola, 2023),, which presents unique challenges with extremely high intrinsic dimensionality. The goal of Powderworld draw a target picture 32 32 grid using different types of "powder" brushes, where each powder brush has a distinct physical property corresponding to unique element. For ex- ample, the "sand" brush falls down and piles up, and the "fire" brush burns combustible elements like "plant." We provide three versions of tasks, easy, medium um, and hard, with different numbers of avail- able elements (2, 5, and 8 elements, respectively). The datasets are collected by random policy that keeps drawing arbitrary shapes with random brushes. This Powderworld task poses unique challenges that distinct from the other tasks in the benchmark. First, the agent must deal with the high intrinsic dimensionality of the states, which presents a substantial challenge in representation learning. Sec- ond, since the transitions of powder elements mostly stochastic and unpredictable, the agent must be able to correctly handle environment stochasticity. Third, the agent must achieve high degree of generalization and sequential reasoning through deep understanding of the physics, in order to complete symmetrical, orderly test-time drawing tasks from random, chaotic data.
powderworld
'nttpa://en http:::///en.iikediiaa... baakiipedia.oo//vikkke.. /Lightggghhtttt Satggttttttut.tt (game)
'Pla here: https: tattpa://kvtiee http///wwaaaaa/eeeeee wtt.:/ooccc/c///eeeee powder/
8

---

Published as a conference paper at ICLR 2025
Locomotion
Man Manipulation
Drawing pixela
Surrns (%)
states
pixela
states
pixola
Sucoess Itate (%) 1 is
Sucres Rate
Gear
octv coogt
oRt Hittt
octer
comz ao out our mul
Gene
octyt
coag
out
cut nnt
aotgt
out mug
octte
CH hugk
Stitching
Exploratory
Stochastic
Sundess Rate (%5)
Success Rate (98)
Surecss Itte (9%
eit.
ortec GO11
Gotd.
URL
ca
mag
ocTV cogg
ORL.
met
orgec
G011
cn oit.
urg.
GO1S orde
Gatd
QHL
Cu
incz
Grapt
oiu
CR
Ric
Figure 2: Benchmarking offline GCRL methods. We report the performances of six offline GCRL methods (GCBC, GCIVL, GCIQL, QRL. CRL, and HIQL), aggregated by different dataset categories (see Table for the category list). The results averaged over the tasks each category, and then over 8 seeds (4 seeds for pixel-based tasks). Error bars denote 95% bootstrap confidence intervals. See Table for the full results. In general, HIQL (a method that involves hicrarchical policy extraction) tends to achieve strong performance across the board. Among the non-hierande ao--heerrchhcal methods, CRL tends to work best in locomotion tasks and GCIVL and GCIQL tend to work best in the others.
RESULTS
We now present and discuss the benchmarking results of existing offline goal-conditioned RL algorithms on OGBench.
8.1
ALGORITHMS
We benchmark six representative offline GCRL algorithms: goal-conditioned behavioral cloning
(GCBC) (Lynch et al., 2019; Ghosh et al., 2021), goal-conditioned implicit (V. Q}-learning (GCIVL and GCIQL) (Kostrikov et al., 2022; Park et al., 2023), quasimetric RL (QRL) (Wang et al., 2023), contrastive RL (CRL) (Eysenbach et al., 2022), and hierarchical implicit Q-learning (HIQL) (Park
al., 2023). GCBC is the simplest goal-conditioned behavioral cloning method. GCIVL and GCIQL are offline GCRL algorithms that approximates the optimal value function using an expectile regression (Newey Powell, 1987). QRL
con-traditiona GCRL method that fits a quasimetric value function with dual objective. CRL is a "one-step" RL algorithm that fits a Monte Carlo value function via contrastive learning and performs one-step policy improvement. HIQL is a hierarchical RL method that extracts two-level hierarchical policy from single GCIVI value function. For benchmarking, we perform a similar amount of hyperparameter tuning for each method to ensure fair comparison. We refer the reader to Appendices F and G for the details.
8.2 BENCHMARKING RESULTS AND Q&As
We present the full benchmarking results in Table 2 in Appendix B. Figure 2 summarizes the results by showing performances grouped by different task categories. Performances are measured by average (binary) success rates on the five test-time goals of each task. We train the agents for 1M gradient steps (500K for pixel-based tasks), and average results over 8 seeds (4 seeds for pixel-based tasks). We discuss the results through Q&As (we refer to Appendix B for more Q&As).
Q: Which method works best in general?
A: While no single method dominates the others across all categories in Figure 2. HIQL (a method that involves hierarchical policy extraction) tends to achieve particularly strong performance among the benchmarked methods, especially in locomotion and visual manipulation tasks. Among the non- hicrarchical methods, CRL tends to work best in locomotion tasks and GCIQL tends to work best in manipulation tasks. In the drawing tasks, GCIVL performs the best.
Q: There seem to be a lot of datasets. What should
I use for my research?
A: For general offline GCRL algorithms research, we recommend starting with more "regu- humanoidmant
lar" datasets, such as antmaze- cube- (single, double) -play, scene -play, and puzzle-3x3 3x3-play. From there, depending on the performance on these tasks, try harder versions of them or more challenging tasks, such as humano dmaze-gi giant, antsoccer, puzzle e (4x4, 4x5, 4x6), and powderworld.
siaatt--nviige

---

Published as a conference paper at ICLR 2025
We also provide more specialized datasets that pose specific challenges in offline GCRL (Section 5). For stitching, try the stitch datasets in the locomotion suite as well as complex manipulation tasks that require stitching (e.g., puzzle). For long-horizon reasoning, consider aumanoidmaze- wwwaze-giaantttee which has the longest episode length, and puzzle-4x6, which has the most semantic steps. For stochastic control, try antmaze-teleport, which is specifically designed to challenge optimisti- cally biased methods, and powderworld, which has unpredictable, stochastic dynamics. For learn- ing from highly suboptimal data, consider antmaze-est exppleeeeepoooo oxe ore as well as the noisy datasets in the manipulation suite, which features high suboptimality and high coverage.
RESEARCH OPPORTUNITIES
In this section and Appendix C, we discuss potential research ideas and open questions.
Be the first to solve unsolved tasks! While all environments in OGBench have at least one variant that current methods solve to some degree, there are still number of challenging tasks which no existing method achieves non-trivial performance, such as humanoi dmaze- -iienggatttttt puzzle-4x55
subeetrrples and more. We ensure that sufficient data is available for those tasks (which is estimated from the amount needed to solve their easier versions). We invite researchers to take on these challenges and push the limits of offline GCRL with better algorithms.
wnnderworldhhard
How can we develop a policy that generalizes well at test time? In our experiments, we found hier- archical RL methods (e.g., HIQL) to work especially well in several tasks. Among several potential explanations, we hypothesize that this is mainly because hierarchical RL reduces learning complexity by having two policies specialized in different things, which makes both policies generalize better at evaluation time. After all, test-time generalization is known to be one of the major bottlenecks in offline RL. (Park et al., 2024a). But, are hierarchies really necessary to achieve good test-time gener- alization? Can we develop a non-hierarchical method that enjoys the same benefit by exploiting the subgoal structure of offline GCRL? This would be especially beneficial, not just because it is simpler, but also because it can potentially yield better, unified representations that potentially serve as "foundation model" for fine-tuning.
Can develop a method that works well across all categories? Our benchmarking results reveal that no method consistently performs best across the board. HIQL tends to achieve strong performance but struggles in pixel-based locomotion and state-based manipulation. GCIQL shows strong performance in state-based manipulation, but struggles in locomotion. CRL exhibits the opposite trend: it excels in locomotion but underperforms in manipulation. Is there way combine only the strengths of these methods to achieve the best performance across all types of tasks?
10 OUTLOOK
In this work, we introduced OGBench, new benchmark designed to advance algorithms research in offline goal-conditioned RL. With the experimental results, we now revisit the very first question posed in this paper: "Why offline conditondt RL?"
We hypothesize that offline goal-conditioned holds significant potential as a recipe for general-purpan RL pre-training, which yields richer and more diverse behaviors than (arguably more prevalent) generative pre-training, such as behavioral cloning and next-token prediction. Our experiments, albeit preliminary, show that even current offline GCRL algorithms can to some extent acquire effective policies for exceptionally long-horizon tasks entirely with sparse rewards, using data that is highly suboptimal. These results are not limited to toy example domains, but show up across a range of realistic simulated robotics and like settings in our benchmark. While generative objectives might capture the data distribution, offline GCRL can learn policies that actually achieve complex outcomes (such as beating a puzzle game) that could not be achieved simply by copying random data.
However, current offline GCRL algorithms also have limitations. As shown our results, they often struggle with horg-horizon, high-dimensional tasks and those that require stitching, and no single method consistently outperforms others across all tasks. This suggests that we have not yet found the ideal algorithm that can fully realize the promise of offline GCRL as ceneral-purpose RL pre-training. The first step toward finding such an ideal algorithm is to set up solid benchmark that sufficiently challenges the limits of offline GCRL from diverse perspectives. We believe OGBench provides this foundation, and will lead to the development of performant, scalable offline GCRL algorithms that enable building foundation models for general-purpose behaviors.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Kevin Zakka for providing the initial codebase for the manipulation environments and helping with MuloCo implementations, Vivek Myers for providing JAX- QRL implementation, and Colin Li, along with the members of the RAIL lab, for helpful discussions. This work was partly supported by the Korea Foundation for Advanced Studies (KFAS), National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2146752, ONR under N00014-20-1: 2383 and X00014-22-1-2777 and Qualcomm. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.
REPRODUCIBILITY STATEMENT
We provide the full implementation details in Appendix G. We provide the code well as the exact command-line flags to reproduce the entire benchmark table, datasets, and expert policies at https://github. com/ weehhngpark/ogbeeeeee
REFERENCES
Zeyuan Aluun-Zhu and Yuanzhi Li. Physics language models: Part 1, context-free grammar, ArXiv, abs/2305. 13673, 2023a.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. ArXiv, abs/2309. 14402, 2023b.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay, In Neural Information Processing Systems (NeurIPS), 2017.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, 2016.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne. and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:// thub. com/ com/jax-ml/jax
G. Brockman, Vicki Cheung, Ludwig Pettersson, J. Schneider, John Schulman, Jie Tang, and 2016.
W. Zaremba. OpenAl Gym. ArXiv,
Elliot Chane-Sane Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning (ICML), 2021.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. Roboties: Science and Systems (RSS), 2023.
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Neural Information Processing Systems (NeurIPS), 1992.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search the replay buffer: Bridging planning and reinforcement learning. Neural Information Processing Systems (NeurIPS), 2019.
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as coal-conditioned reinforeement learning. In Neural Information Processing Systems (NeurIPS), 2022.
11

---

Published as a conference paper at ICLR 2025
Kuan Fang, Patrick Yin, Ashvin Nair, and Sergey Levine. Planning practice: Efficient online fine- tuning by composing goals in latent space. In TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
Kevin Frans and Phillip Isola. Powderworld: A platform for understanding generalization via rich task distributions. In International Conference on Learning Representations (ICLR), 2023.
Justin Fu, Aviral Kumar, Ofrr Nachum, G. Tucker, and Sergey Levine. D4rl: Datasets for deep data-
driven reinforcement learning. ArXiv, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations (ICLR), 2021.
Dibya Ghosh, Chethan Bhateja, and Sergey Levine Reinforcement learning from passive data via latent intentions. In International Conference Machine Learning (ICML), 2023.
Raj Ghugare, Matthieu Geist, Glen Berseth, and Benjamin Eysenbach. Closing the gap between td learning and supervised learning-s1 generalisation point of view. In International Conference on Learning Representations (ICLR), 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018.
Joey Hejna, Jensen Gao, and Dorsa Sadigh. Distance weighted supervised learning for offline interaction data. In International Conference on Machine Learning (ICML), 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). ArXiv, 0bbs/1006.08415 2016.
Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, and Honglak Lee. Successor feature landmarks for long-horizon goal-conditioned reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused task-agnostic milestone planner. In Neural Information Processing Systems (NeurIPS), 2023.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Yc, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoAGo GM AraAšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research (JMLR), 23(274):1-18, 2022.
Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. In Neural Information Processing Systems (NeurIPS), 2019.
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yucying Li, Tim Rocktaschel, Edward Grefenstette, and Yuandong Tian. Efficient planning in compact latent action space. In International Conference on Learning Representations (ICLR), 2023.
Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference Artificial Intelligence (IJCAI), 1993.
Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guided subgoal generation in hierarchical reinforcement learning. Neural Information Processing Systems (NeurIPS), 2021.
Junsu Kim, Seohong Park, and Sergey Levine. Unsupervised-to--niie reinforcement learning. ArXiv, abs/2408. 14785, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
12

---

Published as a conference paper at ICLR 2025
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NewrIPS), 2020.
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Hierarchical planning through goal- conditioned offline reinforcement learning. IEEE Robotics and Automation Letters (RA-L), 7(4): 10216-10223, 2022.
Bo Liu, Yihao Feng. Qiang Liu, and Peter Stone. Metric residual network for sample efficient goal- conditioned reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), 2023,
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning (CoRL), 2019.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman. and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via f-advantage regression. In Neural Information Processing Systems (NeurIPS), 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In International Conference on Learning Representations (ICLR), 2023.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv,
abs/1312.5602, 2013.
Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide metric structure for decision- making. In International Conference Machine Learning (ICML), 2024.
Soroush Nasiriany, Vitchyr H. Pong. Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In Neural Information Processing Systems (NeurIPS), 2019.
Whitney Newey and James I.. Powell. Asymmetric least squares estimation and testing. Econometrica, 15:819-847, 1987.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal-conditioned with latent states actions. In Neural Information Processing Systems (NeurIPS), 2023.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? In Neural Information Processing Systems (NeurIPS), 2024a.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICML). 2024b.
Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware
abstraction. In International Conference on Learning Representations (ICLR), 2024c.
Scohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. ArXiv, abs/2502.02538, 2025.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression: Simple and scalable off- policy reinforcement learning. ArXiv, abs/1910.001777 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning (ICML), 2007.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. ArXiv, abs/ 800.09444, 2018.
13

---

Published as a conference paper at ICLR 2025
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi parametric topological memory for navigation. International Conference on Learning Representations (ICLR), 2018.
Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning (ICML), 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/ 0bbs/170,000 77777700067,,1111
Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Score models for offline goal-conditioned reinforcement learning. In Interaational Conference on Learning Representations (ICLR), 2024.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offiin reinforcement learning library. In Neural Information Processing Systems (NeurIPS), 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abhas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. ArXiv, abs/1801.006% 2018.
Jordan Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez- Vicente, al. Pettingzoo: Gym for multi-agent reinforcement learning, In Neural Information Processing Systems (NeurIPS), 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. ArXiv, 12s/2407.17032 2024.
Mianchu Wang, Rui Yang, Xi Chen, and Meng Fang. Goplan: Goal-conditioned offline reinforcement learning by planning with learned models. Transactions on Machine Learning Research (TMLR), 2024.
Tongzhou Wang and Phillip Isola. Improved representation of asymmetrical distances with interval quasimetric embeddings. ArXiv, abbs/221..15120 2022.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang, Optimal goal-reaching reinforce- ment learning via quasimetric learning. International Conference on Machine Learning (ICML), 2023.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjic Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. In International Conference on Learning Representations (ICLR), 2022.
Rui Yang. Yong Lin, Xiaoteng Ma, Haotian Hu, Chongjie Zhang, and T. Zhang. What is essential for unseen goal generalization offline goal-conditioned rl? International Conference on Machine Learning (ICML), 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors. Mujoco menagerie: A col- lection of high-quality simulation models for mujoco, 2022. URL http://githhb. com/ segplle-deemiindmmjjjee seooo_ennnneee..
Zilai Zeng. Ce Zhang, Shijie Wang, and Chen Sun. Goal conditionn predictive coding for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
14

---

Published as a conference paper at ICLR 2025
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems (RSS), 2023.
Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing contrastive rl: Techniques for offline goal reaching. In International Conference on Learning Representations (ICLR), 2024a.
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In International Conference on Learning Representations (ICLR), 2024b.
15

---

Published as a conference paper at ICLR 2025
Table 2: Full benchmark table. We report each method's average (binary) success rate (%) across five test- time goals on each task. The results are averaged over 8 seeds (4 seeds for pixel-based tasks), and we report the standard deviations after the + sign. Numbers at or above 95% of the best value in the row are highlighted in bold.
Emvirunment
Dataset Type
Dataset
GUBC
GCIVI.
GCQQL
QRI.
CRI.
HIQU.
poinrraze codu:aare-saciu--aaae www.auuatttaaaateaaaaaa
a.
63 s6
53.4
82 a5
29
79 L5
navigate
pointzaze giaeegginntttee miann--avigatee
pointzaze- Sararraaz---rr---aae
29
45 15
0.10
31.13
0.
66
86 19
39 17
27
58 15
pointzazz
poincaas--sollu-atttccrr--
23 ss
axs
stitch
poinssaze--lare--atttce
0.
0.10
12 16
0
31 12
84 115
50 16
0
1.
(1 10
13 1.
poiniraze-telepon
31
25
34
71
96
navigata
24
autaaze-glitt-nariigee
0.
0 1o
0
14 13
16 1.
05
26 23
39 an
335 an
35 15
53
42
45 pu
44
29
50
53
144
stitch
astaaze-larr-sttttdd
0n
7
31 mm
as
17.
24
31
sstaaze-rddi--cpporrr
2.
13
1:13
3.12
37
explore
astanze-large
0.
10 as
0.20
0 10
0.10
astanze-teleport
2.
32
7
2:25
20
34
Iaaanc dasaaai-ddde---ei--aaaa--
27
21.4*
80
navigate
24
humanoi draaz
0.
0.10
0.
1:10
12
hanano inazee-add un itch-vo
29
12
12
18
96
88
atitch
hunano ddaazed largge---larg-ttte tohh-00
6
1:
0.se
3:
28
hanane siatt-otttt----tiiteee
0.
0.48
navigata
astaoccer parercorrrrrnn-eiiitttt-eeee
andancoor-red gata-vo
2
4:21
7
2.
8:
23
1.3
antsoccer
24 4.
21 44
2:4
1:43
10
15
stitch
sttroccer-nediuee stitch--v
21
1 1o
0.1e
0.10
0.10
viaunl-anzze -ann---nnnraaaaeee
gatae-vO
11
22
11
0
94
93
navigate
viaual saocxaae-ggaattnavigattee
visual-aocxaze-ia
visual- -aa--aazzz----eeeeeeeee narigrte-anigi---t
6.42
6.
48 12
37
visual --attrase-radiurrtt tch-v0
67 s7
6a2
2ae
0.2
60
87
Fieusl-antnaze
stitcch
riaual-aoizaze-e
24 ax
1:
0.e
1:
11
28
ca
1:42
visual -nr
0.
0.24
explore
visual viauel-Antzee -antvaae---aag---eaaa woppere--
Dan
0
0
0
visual- waw.aaze-eeeepeeee explore-vo
0
0
0.e
0
10
19
risual www.aoidaazeeeee........... zedius- savigus----s--rrrrr
0.e
navigate
viaual-busano wwtssaaoidaazeellleee
vicual visual-humanodde humalmiidnddda
visual-2uxamo- daazee -avvigg----------
visual -uuxanoi un
0.20
0.ee
0.20
stitchh
vianal viaunl-hrraho whrrahhhhiiiaaaee laase-
0
0 an
0
0
00
viaual-buxanoi:de waw.aanooodaaaae-a----------e-
0
0
0
D
sabe-cingle-pay//
55
66
5
19
15
cabe-doable-playy
play
cibe-trigpe--layy-d
1
1:20
3
0.an
4:21
cibe-gudrrpierplaay-00
0
0 cn
0
0
0 =7
Eccae
play
ecene-play-r0
5
42 14
51 14
5
19 is
38 1.
p=male-3:--p1ay--00
6:
95
Tann
12
puzzle-xx--piay-10
0
puzzle
play
puzale-415-jlay-10
0.4
pazzle-dxx-ppay--0
10 12
12 13
0.
4.
1.
60 an
30
41
31 2m1
89
visusl-cube
play
riaual-ate-triileeppi---
Tei
14
10
1
5
2:
17
39
Looul--ute-uudrrill-p------
7.4
5.43
vieual sieusl-acane
play
12
25 as
12
10
11 12
49
21
1:42
1:43
0.am
73 =M4
viaual-puzzle
play
risual- sisual-purzle--e 1xrlll-----pp-- 1
10
60 is
7.
16
0
0
10 s0
60 4-1
15
masual-puzzle p1
2at
2:3
coedaroordd-sayypllae
0
21
03
12.
22
am
poodervor1d
play
50 ec
16
3
22 434
poudergor wowoorldd-aa d--rrd-plla---
4
0.2
0.a
0.m
1
A LIMITATIONS
While OGBench covers number of challenges in offline goal-conditioned RL, such as reasoning, goal stitching, and stochastic control, there exist other challenges that our benchmark does not address. For example, all OGBench tasks assume that the environment dynamics remain
16

---

Published as a conference paper at ICLR 2025
Table 3: How good are our reference implementations? Our implementations generally achieve better performance than previously reported ones.
D4RL -aarze---reeee-dieeeee
D4RL antasze- 1ataszz-arrge-pleyy-2
Method
Previously Reported Performance
Ours
Method
Previously Reported Performance
Ours
GCBC
20 (Park et al,, 2023)
41
GCBC
23 (Park al, 2023)
39
GCIVL
(Park et al., 2023)
64 #s
GCIVL
57
(Park et al., 2023)
ts
GCIQL
30 (Zeng al., 2023)
64
GCIQL
40 (Zeng et al., 2023)
55
QRL
62' (Zheng et al., 2024b)
37 +:4
QRL
53
(Zheng et al., 20246)
38 :6
CRL
(Eysenbach
2022)
79
CRL
(Eysenbach et al., 2022)
HIQL
88 (Park et al., 2023)
87 1:
HIQL
86 (Park et al., 2023)
87
the same between the training and evaluation environments. Also, although several OGBench tasks (e.g., Cube, Puzzle, and Powderworld) require unseen goal generalization to some degree, our tasks do not specifically test visual generalization to entirely new objects. Finally, we have made several trade-offs reduce computational cost and to focus the benchmark on algorithms research the expense of sacrificing realism to some degree (e.g., the use of the transparent arm in manipulation environments, the use of synthetic (yet fully controllable) datasets, etc.). Nonetheless, believe OGBench can spur the development of performant offline GCRL algorithms, which can then help researchers develop scalable data-driven unsupervised RL pre- pre-training methods for real-world tasks.
B ADDITIONAL RESULTS AND Q&As
We present the full benchmarking results in Table 2, and provide additional Q&As in this section.
Q: Which methods are good at goal stitching?
A: To see this, we can compare the performances on the navigate and stitch datasets from the same locomotion task in Table 2. The results suggest that, as expected, full RL-based methods like HIQL (i.e., methods that fit the optimal value function Q²) ) are better at stitching than one-step RL methods like CRL (i.e., methods that fit the behavioral value function Q²)). For example, in visual locomotion tasks, the relative performance between HIQI. and CRL reversed on the stitch datasets (Figure 2).
Q: Which methods are good at handling stochasticity?
A: For this, we can compare the performances on the large and teleport mazes in Table 2, where both have the same maze size, but only the latter involves stochastic transitions that incur risk. Table 2 shows that, value-only methods like HIQL and QRL (i.e., methods that do not have a separate Q function), which are optimistically biased in stochastic environments, struggle relatively more in stochastic teleport tasks. In contrast, CRL is generally robust to environment stochasticity, likely because it fits Monte Carlo value function.
Q: Which methods are good at handling pixel-based observations?
A: Although state-based and pixel-based observations generally provide the same amount of informa- tion, several methods struggle to handle image observations due to additional representational chal- lenges. We understand how well method addresses such representational challenges by compar- ing the performances of corresponding state- and pixel-based based tasks. Table 2 shows that CRL is notably robust to the difference in input modalities, likely because it is based on pure representation learning objective. HIQL also achieves strong performance in pixel-based tasks, especially in visual manipula- tion tasks. However, these methods are still not perfect handling image observations; for example, HIQl achieves relatively weak performance on image drawing tasks. We suspect this is due to the diffi- culty of learning low-dimensional subgoal representations from states of high intrinsic dimensionality.
Q: How good are our reference implementations?
A: We compare the performance of our reference implementations with previously reported numbers on one of the most commonly used tasks in prior work, D4RL antmaze-largs (Fu et al., 2020). Ta- ble 3 shows the comparison results, with the corresponding numbers taken from the prior works (Ey- senbach et al., 2022; Park et al., 2023; Zeng et al., 2023; Zheng et al., 2024b). The results suggest that
We note that Zheng et al. (2024b) use different evaluation scheme hased on the maximum performance over evaluation epochs. We report the average performance over the last three evaluation epochs (see Appendix G).
17

---

Published as a conference paper at ICLR 2025
Table 4: Do not use single-goal evaluation! Only using single state-ges goal pair (a common practice when using D4RL tasks for offline GCRL) can potentially lead to inaccurate conclusions about offline GCRL methods. OGBench always uses multi-ge evaluation. See how the rank between GCIQL and QRL is reversed with multi-goal evaluation on the same antmaze-large maze.
Dataset
GCBC
GCIVL
GCIQL
QRL
CRL
HIQL
D4RL Largaa---argg-iieeeeeee (single-goal evaluation)
41 ST
61 ms
61 #10
37 =14
79 =6
87
DARL saaaze-largeplaaay (single-goal evaluation)
55
OGBench antaaze-larre Sarggennvigeen navigate- (multi-goal ovaluation, ours)
21 =2
16 =5
31
75
83 m4
91
our implementations generally achieve better performance than previously reported results, some- times significantly surpassing them (e.g., CRL).
Q: Why can't I just use D4RL AntMaze instead of the OGBench one?
A: D4RL AntMaze an excellent task for benchmarking offline RL algorithms. However, limited for benchmarking offline goal-conditioned RL algorithms because it only involves single, fixed state-goal pair, and the datasets are tailored to this specific task (Fu et al., 2020). In contrast, OGBench supports multi-goal evaluation (and provides much lareee--vvageeeaiieeeeee
diverse types of tasks and datasets!). To empirically demonstrate this difference, we compare the benchmarking results on D4RL antmaze- -aarge (rigee-(diveree, play} and OGBench The table suggests that single-goal evaluation is indeed limited, and is potentially prone to inaccurate conclusions: for example, see how the ranking between GCIQL and QRL is reversed with multi- goal evaluation on the same an antmaze-lane maze. Moreover, the performance differences between methods are more pronounced in OGBench AntMaze, showing that OGBench provides clearer research signals.
in Table 4.
Q: Have you found any insights on data collection for offline GCRL?
A: One of the main features of OGBench is that every task is accompanied by reproducible and controllable data-generation script. Here, we show one example of
libel (GCIQL)
puzzs 1er3-l----a (GCIQL)
Stecoss Hone (5)
1001
how this controllability can lead to practical insights and
raise open research questions. In Figure we ablate the strength of Gaussian action noise a added to expert actions two manipulation tasks seobbe-singe--ne and puzzle muzzle-3x3 -nois),, and measure how this affects performance.
01 01 BUE
OHI 1620 I1 Noier
D3
Action Noiso
Figure 3: Datascts must be noisy enough.
The results are quite remarkable: they show that having the right amount of noise (i.e., state coverage) is very important for achieving good performance. For example, the performance drops from 99% to 6% if there is no noise in expert actions, even on the most basic cube pick-and-place task. This suggests that, we may need prioritize coverage much more than optimality when collecting datasets for offline GCRL in the real world as well, and failing to do so may lead to (surprising) failures in learning. Like action noise, we believe there are many other important properties of catasetss that significantly affect performance. We believe that our fully transparent, controllable data-generation scripts can facilitate such scientific studies.
C MORE CONCRETE RESEARCH QUESTIONS
Here, we list additional, more concrete research questions that researchers may use as a starting point for research in offline GCRL:
Why is PointMaze so hard? Table 2 shows that PointMaze is surprisingly hard, sometimes even harder than AntMaze for some methods. Why is this the case? Moreover, only in PointMaze does QRL significantly outperform the other methods. What causes this difference, and are there any insights we can take from these results?
How should we train subgoal representations? Somewhat weirdly, HIQL struggles much more with state-based observations than pixel- based observations on manipulation tasks. We suspect this is related to subgoal representations, given that HIQI. uses additional learning signal
18

---

Published as a conference paper at ICLR 2025
from the policy loss to further train subgoal representations only in pixel-based environments (which we found does not help in state-based environments). HIQL uses a value function- based subgoal representation (Appendix F). but is there better, more stable way to learn subgoal representations for hierarchical RI. and planning?
Do really need the full power of RL? While learning the optimal Q" function is in principle better than learning the behavioral Q function, CRL (which fits Q^) significantly outperforms GCIQL (which fits Q*) on locomotion tasks. Why is this the case? Is it a problem with expectile regression in GCIQL or with temporal difference learning itself? In contrast, in manipulation environments, the result suggests the opposite: GCIQL is much better than CRL. Does this mean we do need Q" in these tasks? Or can it be solvable even with Q" if we use better behavioral value learning technique than binary NCE in CRL?
Why can't we use random goals when training policies? When training goal-conditioned policies, we found that it is usually better to sample (policy) goals only from the future state in the current trajectory (except on stitch or explore datasets; see Table 10). The fact that this works better even in Scene and Puzzle (which require goal stitching) is bit surprising, because means that the policy can still perform goal stitching to some degree even without being explicitly trained on the test-time time state-goal pairs. At the same time, it is rather unsatisfying because this ability to stitch goals entirely depends on the seemingly "magical" generalization capabilities of neural networks. Is there a way to train a goal-conditioned policy with random goals while maintaining performance, that it can perform goal stitching in principled manner?
How can we combine expressive policies with GCRL? In Appendix D, we show that current offine GCRL methods often struggle with datasets collected by non-Markovian policies in manipulation environments. Handling non-Markovian trajectory data is indeed one of the major challenges in behavioral cloning, for which many recent BC-based methods have been proposed (Zhao et al., 2023; Chi et al., 2023). How can we incorporate these recent advancements in behavioral cloning into offline GCRL?
D ADDITIONAL DATASETS
For manipulation tasks (Cube, Scene, and Puzzle), in addition to the main play datasets, we additionally provide noisy datasets that can potentially be useful for other types of research (e.g., ablation studies on datasets, comparing performances on non-Markoviaa and Markovian datasets, etc.). The main difference is that the play datasets are collected by open-loop, Markoiaoo expert policies with temporally correlated noise, while the noisy datasets are collected by closed- loop, Markovian expert policies with larger, uncorrelated Gaussian noise. Hence, the play datasets generally look more "natural" than the noisy datasets, but the latter has higher state coverage (videos).
Table 5 shows the full benchmark results on both the play and noisy datasets in the manipulation suite. The results suggest that the performances on these two datasets are mostly similar, but several methods struggle handle narrower and non-Markovian trajectories in play datasets (e.g., GCIQL almost perfectly solves www-ssiinnnnisseeeeeeeeeeeeeeeee but struggles on cube-single-play).
E PRIOR WORK IN GOAL-CONDITIONED RL
The problem of reaching any goal from any state has long been considered one of the central problems in reinforcement learning and sequential decision making (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017 7), owing to its unsupervised nature, simplicity, and generality. There are many unique features of coal-conditioned RL that make it distinct from other (multi-task) RL problems, such as the presence of recursive subgoal structures, metric structures, and probabilistic interpretations. These intriguing properties have led to the development of diverse families of online and offline GCRL algorithms based on hindsight relabeling (A (Andrychowicz et al., 2017), hierarchical learning (Dayan & Hinton, 1992; Chane-Sane et al., 2021; Li et al., 2022; Park et al., 2023), planning (Savinov et al., 2018 1018 Eysenbach et al., 2019; Nasiriany et al., 2019; Huang et al., 2019; Hoang et al., ; Kim et al., 2021; Wang et al., 2024), metric learning (Wang et al., 2023; Park et al., 2024b; Myers et al., 2024), dual optimization (Ma et al., 2022; 2023; Sikchi et al., 2024), weighted behavioral cloning (Yang et al., 2022; 2023; Hejna et al., 2023), generative modeling (Zeng et al., 2023; Hong al., 2023), and contrastive learning (Eysenbach al., 2022; Zheng al.,
19

---

Published as a conference paper at ICLR 2025
Table 5: Full benchmarking results on additional noisy manipulation datasets. The table shows the performances on both the play and noisy datasets in the manipulation suite. We report each method's 's average (binary) success rate (%) across the five test- sis-time goals on each task. The results are averaged over seeds (4 seeds for pixel-based tasks), and report the standard deviations after the + sign. Numbers at ahove 95% of the best value in the are highlighted in bold.
Environment
Dutaset Type
Dataset
GCBC
GCIVL
GCIOL
QRL
CRL
HIQL
cabe-iingle-piay-0
53
68
15
abb--ouble-play--0
glay
cabe-triple-plap
1
=
D
cube
cabe-qadugie-piayy-
0:
so
0 am
cabe-single-noisy--0 cabe-6ooble-noieye
8
71
99
26
38
41
14
23
3
2
Holay
cabe-triple-nofaa
cube-quaruggenoiayy-0
0.
play
ecene-play--e
5:13
42 24
51 24
L2
35
AdanA
nolay
acmee-ntay--v
1:
26
26
26
pux21e-36S-p1ay--00
2
96
3
12
puzzle-6xi-plyy-1
0.
13
26
play
mazzle-4x5-play-vv
7.at
puzzle-4x6-g1ley-o
0an
10 27
12.s1
4:21
3.21
pazzlo
pazzle-3:3-20iey-10
1.
42 -2
94
-4
51
parzle-txd--oity--00
0.10
20
29 17
0 0.10
16
noiay
puzzle-6x6-20iey-1
19
19
3
puzzle-6x6-noiaymm0
ais
play
susuall-ubbe-oubb22
cuuull-cuee-rrpplo
1.
15
14:22
10
15 15:11
11
16
5
2 11
17 =2
21
39
visual-cube cuue
8:
0.
2
4
14
14
75
48
10
99
Boisy
16
18
12 L1
9-44
16
59
23
9
0
sz
8 :u
12
viaual-acees
viaual-acese-plaa--0
42
26 ct
12
10
49
solcy
13
28
12
15 L2
50
73
play
visual- pastullppp2eee--pppp--------
visual- pattual-puzzee----pea 1x8-play-
5.44
40
17 41
as
7.42
16
0
6
13
visual-puzzle
visual- puzzleuuzzeeeeeeee faolee-dd--ppp 1x6-play-vv
2:13
15 11
2
0_.
3
9-46
viaual spausl-puzssleee ppupsll--003-mmm ---aa--nllaa--
20
26
70
noiay
viaual-pos -p-22-e------o---ae
visual puzzzl vo
47
14
19
84
14
viaual -poooppeeeeedd--ee www.eee---naaa----e -0
2
12
17.0
0
2
14
2024a;b). In this work, we also consider the problem of offline goal-conditioned RL; however, instead of proposing a new algorithm, we introduce new benchmark and reference implementations to facilitate and advance algorithms research offline GCRL.
F OFFLINE GCRL ALGOIIHTM
In this section, we describe the six offline GCRL methods used for benchmarking in detail. We first define four goal-sampling distributions that correspond the current state, uniform future states, geometric future states, and random states, respectively:
Peur(a (g 8) denotes the Dirac delta distribution at s (i.e., g is always set to s).
Praj (g s) denotes the uniform future state distribution defined follows: assuming 8 in a trajectory T = (8o, dn, .1...., ,18....8T)))............ we sample an index l from the uniform distribution Unif(min(t + 1,T I),T 1) (inclusive), and set g 8k.
Pr (g s) denotes the truncated geometric future state distribution defined as follows: assuming s = st in a trajectory т = (so, ap $,,..., ST), we sample an index è from the geometric distribution Geom(1 7) (whose support starts from 1), and set g Smin(s+&,T-1)-
Prand (g) denotes the uniform state distribution over the dataset D.
Additionally, Piiised( (g 8) denotes a mixture of these four goal-sampling distributions with a mixture ratio defined by byperparameters, pP(-) simply denotes the uniform distribution over the dataset, and
'If there multiple such (T, tuples in the dataset, consider the uniform mixture of them.
20

---

Published as a conference paper at ICLR 2025
sometimes use p.(- s,a) instead of ( to denote the distribution corresponding to the state-action pair.
Goal-conditioned behavioral cloning (GCBC), GCBC (Lynch al., 2019; Ghosh et al., 2021) simply performs behavioral cloning using future states the same trajectory as goals. GCBC maximizes the following objective to train goal-conditioned policy x<a 8,9).
JGCBC (T) ((s.a))pPPP (s..))))=PP())))0 (a|s) [log ((a 8, 8,g]]-
(1)
Goal-conditioned implicit {V, Q}-learning (GCIVL and GCIQL). GCIVL and GCIQL are goal- conditioned variants of implicit Q-learning (IQL) (Kostrikov al., 2022), which is an offline RL algorithm that fits the optimal value functions (V* or Q²)) using an expectile regression (Newey & Powell, 1987). GCIQL a straightforward goal-conditioned variant of IQL, and GCIVL the V- only variant introduced by Park et al. (2023). GCIVI fits value function (s,gg) by minimizing the following loss:
EGCIvI(V) = " (0).gr Pi (sla)
[ (r(s,g) + - (s,,)))
(2)
where r(8, (o) (s) denotes the (-1, -sparse goal-conditioned reward function, V denotes the target value function (Mnih al., 2013), and & (z) i 1 (x:=<<) (x)]2² denotes the expectile loss with an expectile k.
GCIQL fits both g) and Q(, (1)) by jointly minimizing the following losses:
CéciQu (V) E (s.a))=""((s)))) (1.e). (ols) [e (Q(s,a,g) V(s,g))l
(3)
C (Q) = (s,.a, (r)sp,"""""p"" (s,a,s) D.g. Pa (ala) (r(s,g) + (V +V -(s, a, g)))
(4)
where denotes the target Q function (Mnih et al., 2013). We note that GCIVL optimistically
biased in stochastic environments, but GCIQL is unbiased (Kostrikov et al., 2022; Park et al., 2023).
To extract policy from the learned value functions, we can use either advantage- advantage-weightter weighted regression (AWR) (Peters & Schaal, 2007; Peng et al., 2019) or behavior-constrained deep deterministic policy gradient (DDPG+BC) (Fujimoto & Gu, 2021). GCIVL uses the following value-only variant of the AWR objective (Park et al., 2023):
(w) = E (s. (s.a,s")'p" (s.a,
( log ((a s.9)
(5)
where or is the temperature hyperparameter. In our experiments, GCIQL mainly uses the following DDPG+BC objective (which is known to be better than AWR (Park et al., 2024a)):
JDDPG+BC(T)
(s.s)) s) [Q(s, T (s, g). + a log x(a s.g)],
(6)
where (8, [ena(o)=,)) In discrete-action environments, GCIQL uses the following Q
version of AWR:
J (e,a,,") P (sle) [ (=(((s1a)) log (a s.9))]
(7)
In practice, we use standard double-value learning for GCIVL and GCIQL (Park et al., 2023), and Q normalization for DDPG+BC (Fujimoto & Gu, 2021).
Quasimetric RL (QRL). QRL (Wang et al., 2023) is a goal-conditioned value learning algorithm based quasimetric learning, where quasimetric means an asymmetric metric. In deterministic environments, the shortest path length between two states d' (s. g) is equivalent to the optimal undis- counted coal-conditioned value function V* V* g) under the (-1, 0)-spas reward function (Wang et al., 2023): (*(s,g) = -d" d² (s.g). The main idea of QRL is to explicitly leverage the quasimetric property (i.e., triangle inequality) of shortest path lengths, namely ds, w) + d' (w. g) 2 d' (s, g) for any s, w, g E S, by modeling it with quasimetric network architecture like MRN (Liu al., 2023) or IQE (Wang& & Isola, 2022). Concretely, QRL maximizes the following constrained optimization objective:
maximize E Ex (6) [d({gg)))) s.t. E(s (s,0) [(d(s - 1) < <
(8)
(9)
21

---

Published as a conference paper at ICLR 2025
where d(s,g) is a quasimetric distance function (e.g., an IQE network) and is hyperparameter that controls the strength of the constraint.
To extract policy from the value function V(s,g) = -d(s,g), g). QRL uses the value-only AWR loss Equation (5) in discrete-action MDPs. In continuous-action MDPs, QRL additionally fits a dynamics model f(((s), (f(ss),a) a): : 2 x A 2 (Wang al., 2023), where 2 denotes a latent space and ((s): S + 2 denotes the representation function used in the quasimetric distance function: namely, d(s, g) = d(((s), (()) (e.g., & is the interval representation function in IQE). The dynamics loss is
as follows:
Laya (f) (f (appa, (s,0,)))
+ (f(
(10)
where QRL.jonnlly jointly trains both d and f without stop-gradients. Based on the dynamics model f. QRL maximizes the following DDPG+BC-like loss:
JDDPG+BC(T)) = (s.a)) (gla) ("( +a log (a 8
(11)
where we use the same notation as Equation (6).
In practice, we employ a softplus loss shaping for the quasimetric loss and delta prediction for the dynamics model, as in Wang et al. (2023).
Contrastive RL (CRL). CRL (Eysenbach al., 2022) is "one-step" GCRL algorithm that first trains a Monte Carlo goal-conditioned value function using contrastive learning and performs a one- step policy improvement. CRL maximizes the following binary NCE objective (Ma & Collins, 2018)
with respect to f(s, a, 9): S x A x + R:
(f) = E (s.)))p((s)))))rr Room (als,a) up (o) Dlg (f + log(1 - off(s, 1))}
(12)
where a: R - (0. 1) denotes the sigmoid function. The optimal solution to the above objective is given as f(s a, g) = log(p goom (g) 8, ())/PPan(()) Given the equivalence between the geometric future goal distribution (ngoom) and the Monte Carlo goal-conditioned Q function (QMC) under the (0, 1)-spas reward function r(s, g) = [() (s) (Eysenbach et al., 2022), we get the following relation: f(s, a. g) = log QMC(s (s, g) g) + C(g). where C' is a function that only depends on g. In practice, f is parameterized as f(,, g) Φ(s, a) TT)g)/VV with Φ: S x A + Z I2d and wh: S + = Rd (note that this inner-product parameterization is universal (Park et al., 2024c)), and use the future goals from the other states in the same batch as g We also employ the double- value learning technique for f (Eysenbach et al., 2022).
For policy extraction, in continuous-action MDPs, employ DDPG+BC (Equation (6)) using instead of Q. In discrete-action MDPs, we use AWR (Equation (5)) using (f, f instead of (Q, V),
where we additionally train contrastive value function f (s,9): S + R using
JCRL-v(fV) = - (s) oo (gle),g up (9) [log (f (s,g)) + +log(1 - o(f (s,g))))
(13)
with similar inner product parameterization for SK
Hierarchical implicit Q-learning (HIQL). HIQL (Park et al., 2023) is an offline GCRL algorithm that extracts two policies from a single goal-conditioned value function. HIQL first trains GCIVL (Equation (2)) with parameterized value function defined as V(s.g) g) V(s, o(s. g)), where d: S x S + Z serves as (state-dependent) subgoal representation function. Based on the GCIVL value function, HIQL extracts high-level policy x²: S x S + A(Z) and a low-level policy nf: Z + ((A) with the following AWR-like objectives:
HAQQL (TT²) (( "gpppppppppp axpeisaa (oles) (1VV log nO ~ s(48) st, ss.
(14)
Jiiqu(T7) bop (aaaaa/pppaaaaaaaaaaae (erVVosniitt- log I (ar St, ((5t, sc+b,)
(15)
22

---

Published as a conference paper at ICLR 2025
where we omit the arguments in p", and i denotes a hyperparameter corresponding to the subgoal step. For simplicity, we ignore some edge cases in the objectives above (e.g., when t + k exceeds the trajectory boundary, in which case truncate); we refer to Park et al. (2023) or our code for the full details. Intuitively, the high-level policy predicts the representation of the optimal k-step subgoal, and the low-level policy predicts the optimal action based on the predicted subgoal.
In practice following Park et al. (2023), we use the double-value learning technique, normalize the output of Φ, and allow gradient flows from the low-level AWR loss into & (only) in pixel-based environments.
G IMPLEMENTATION DETAILS
We provide the full implementation details in this section. We release the code as well as the exact command-line flags to reproduce the entire benchmark table, datasets, and expert policies at https://github. wtt/ howp///ooogaaa/////////// hoo.oogggg/kooochc.h.....
G.1 TASKS AND DATASETS
In this section, we provide further information about our tasks and datasets. We provide the basic specifications about the environments and datasets in Tables 7 and 8.
Locomotion tasks. For OGBench AntMaze and AntSoccer, we adopt the Ant model from D4RL AntMaze (Fu et al., 2020) (which is based on the Ant in OpenAl Gym (Brockman et al., 2016; Towers et al., 2024), but with a more restricted joint range) and the soccer ball model from the DeepMind Control suite (Tassa et al., 2018). For HumanoidMaze, we adopt the Humanoid model from the DeepMind Control suite (Tassa et al., 2018).
To collect datasets, we train expert low-level directional (AntMaze and HumanoidMaze) or goal- reaching (AntSoccer) policies using SAC with dense reward functions for 400K (AntMaze), 40M (HumanoidMazze) or 12M (AntSoccer) steps. For PointMaze, we use a scripted directional expert pol- icy. When collecting datasets, we add Gaussian noise with standard deviation of 0.5 (pointmaze), 1.0 (explorep ore), or 0.2 (others) to expert actions.
The success criteria for the locomotion tasks are based only on the distance between the agent (or the ball in AntSoccer) and the goal location. particular, the tasks not consider joint positions determining success, in previous works (Fu et al., 2020; Park al., 2023).
Manipulation tasks. We adopt the URSe robot arm and Robotiq 2F-85 gripper models from MuJoCo Menagerie (Zakka et al., 2022), and the drawer, window, and button box models from Meta-World (Yu et al., 2019). The robot is end-effector controlled with a 5-D action space, where the dimensions correspond to the displacements in the position, y position, position, gripper yaw, and gripper opening. In all manipulation tasks, we place invisible walls to prevent objects from moving into an area beyond the robot arm's reach. These invisible walls also prevent the cube objects from moving outside the camera viewpoint in pixel-based manipulation environments. However, since some blind spots still exist in visual- -ccene even with the walls, we further filter out such rare cases in trajectories prevent ambiguous camera observations completely.
In Puzzle, not every button configuration is reachable from the initial state. While the 3 3, and 4 x 6 puzzles do have this property, the 4 x 4 puzzle does not. This can be seen by computing the rank of the nm nm button effect matrix over F2 (the field with two elements), where and m denote the numbers of rows and columns, respectively. In our tasks, we ensure that every test-time goal is solvable. Also, we note that the maximum value of the minimum number of button presses to reach state from another state in each puzzle is 9 (3 x 3 puzzle), 7 (4 x 4 puzzle). 20 (4 5 puzzle), or 24 (4 x puzzle). Each puzzle environment contains at least one evaluation goal that requires the maximum number of presses.
The play datasets are collected by open-loop, non-Markovian scripted policies, and the noisy datasets are collected by closed-loop. Markovian scripted policies. For the play datasets, we add temporally correlated action noise to the expert actions to enhance state coverage. For the noisy datasets, first sample the degree of action noise at the beginning of each episode, and collect
23

---

Published as a conference paper at ICLR 2025
trajectory with the chosen amount of (time-independent) Gaussian action noise. This ensures high coverage while having a sufficient number of optimal trajectories.
The success criteria for the manipulation tasks are based only the object configurations; the arm pose is not considered when determining success. For cubes, only the distances between the goal positions and their current positions considered, and their orientations are ignored.
Drawing tasks. We modify the original Powderworld environment (Frans Isola, 2023) to make offline and goal-conditioned. We also re-implement Powderworld (which was originally implemented in PyTorch) in NumPy to remove the dependency on PyTorch. We provide three versions of Powder- world tasks: poxderworld-easy uses two elements (plant and stone), powderwor www.wolllmdddde d-medium uses five elements (sand, water, fire, plant, and stone), and powderworddhaat water, fire, plant, stone, gas, wood, and ice). An action in Powderworld corresponds to drawing 4 x 4-sized square with a specific element brush on the 32 x 32-sized board. Since naïvely implement- ing this atomic action requires up to 312-dimensional discrete actions, we split it into three sequential 8-dimensional actions that correspond to element selection, z-coordinate selection, and y-coordinate selection. To ensure full observability, we add three additional dimensions that contain information about the currently selected element and x coordinate to the original 32 32 3-dimensional image, which results in a 32 x 32 x 6-dimensional observation space. When the agent selects an invalid ac- tion (which can only happen in powderworld- (easy,, medium). which has fewer than 8 elements), the environment instead uses a randomly sampled valid action.
uses eight elements (sand,
The datasets collected by scripted policy that randomly draws squares and lines or fills the entire board with randomly selected brushes. With a probability of 0.5, it performs a random action (i.e., places a random element on a randomly sampled position).
For the success criterion for evaluation goals, we use the following procedure to allow for some tolerance: For each pixel in the goal image, we check if the current image has matching pixel that is shifted by to one pixel in any direction. We then compute the error as the number of pixels that do not match, and consider the task successful if the error below certain threshold.
G.2 SINGLE-TASE VARIANTS
OGBench also supports standard (i.e., non-goal-conndtioned) offline RL by providing single-task variants of locomotion and manipulation tasks. To convert a goal-conditioeed task into a standard reward-maximizing task, we fix an evaluation goal and relabel the dataset with semi-sparse reward function. This semi-sparse reward function is defined as the negative of the number of unaccomplished subtasks in the current state, and the episode immediately terminates when the agent completes all subtasks of the target evaluation goal. In locomotion environments, rewards are always -1 or 0, as there are no separate subtasks. In manipulation environments, rewards range between -taak and O, where faaakk denotes the number of subtasks (e.g., in puzzle-4xx -4x6, Prask = 24 as there are 24 buttons).
Each locomotion and manipulation task in OGBench provides five single-task variants that correspond to the five evaluation goals (Appendix H), resulting in a total of 410 single-task tasks. They are named where [n]
with the suffix "singletask-taas" [n]" (e.g., denotes number between 1 and 5 (inclusive). Among the five tasks in each environment, the most representative one chosen as the "default" task, and is aliased by the suffix "singletask" without a task number. For example, in cube-double, the second task (standard double pick- and-place; see Figure 6) is set as the default task, and wttrr-dobbel-llll-a--llllleeeeee sangle--paa--sngetttee
and refer to the same task. Default tasks can be useful in various ways. For instance, one may report performance only on default tasks to reduce the computational burden, may treat default tasks as "training" task set for tuning hyperparameters while using the other four tasks as a "validation" task set. We provide the list of default tasks in Table 9.
While we do not provide separate benchmarking result the single-task environments, bench- marking table of several representative offline RL algorithms on 50 tasks can be found in the work by Park al. (2025).
24

---

Published as a conference paper at ICLR 2025
G.3 ORACLE REPRESENTATION VARIANTS
OGBench also provides oracle representation variants of locomotion and manipulation tasks, de- noted by the suffix "oraclerep" (e.g., provide low-dimensional oracle goal representations that contain only relevant information for ful- filling the goal success criterion. This corresponds to the r-y coordinates of the agent (or the ball in antsoccer) in locomotion environments, and the positions of the cubes and the states of the ob- jects in manipulation environments. The oraclerep tasks reduce the burden of goal representation learning, potentially helping diagnose the bottlenecks in goal-conditioned RL algorithms. We do not provide a separate benchmarking result for the oracle representation variants.
These tasks
G.4 METHODS
Our implementations of six offline GCRL algorithms (GCBC, GCIVL, GCIQL, QRL, CRL, and HIQL) are based on JAX (Bradbury et al., 2018). In our benchmark, each run typically takes 2-5 hours (state-based tasks) or 5-12 hours (pixel-based tasks) on an A5000 GPU, depending on the task and algorithm.
For benchmarking, we periodically evaluate the performance (goal success rate in percentage) of each agent on each test-time goal with 50 rollouts every 100K steps, and report the average success rate across the last three evaluation epochs (i.e., 800K, 900K, and 1M steps for state-based tasks and at 300K, 400K, and 500K steps for pixel-based tasks). That is, the performance of each agent is averaged over 750 rollouts (3 evaluation epochs 5 test-time goals x 50 rollouts). While we use relatively large number of evaluation rollouts for robustness, researchers can adjust the number of evaluation rollouts (e.g., 20 episodes for each test-time goal) to reduce the computational burden.
We provide the full list of common hyperparameters in Table 10. We find that methods are more sensitive to policy extraction hyperparameters (e.g.. the BC coefficient in DDPG+BC) (Park et al., 2024a), and report these in separate table (Table 11). Specifically, for each method, value learning hyperparameters across the benchmark except for the discount factor 7 (Table 10), but individually tune the policy extraction hyperparameters (e.g., AWR and DDPG+BC a:) for each dataset category (Tables 10 and 11).
the same
We apply layer normalization (Ba et al., 2016) to the value networks, but not to the policy networks.
In pixel-based environments, we use a smaller version of the IMPALA encoder (Espeholt et al., 2018). We random-crop image augmentation (with probability of 0.5) for pixel-based manipulation tasks, but not for pixel-based locomotion or drawing tasks, as we find it to be helpful mainly on manipulation tasks. In pixel-based environments, we do not apply frame stacking for simplicity, we find it does not necessarily improve performance on most tasks including Visual AntMaze (although we believe the performance on Visual HumanoidMaze can further be improved with frame stacking). For policies, we parameterize the action distribution with unit-variance Gaussian distribution. We find that using fixed standard deviation is especially important for DDPG+BC. During evaluation, we use the deterministic mean of the learned Gaussian policy. However, in Powderworld, which has discrete action space, we use a stochastic policy with a temperature of 0.3 (i.e., we divide the action logits by 0.3), as this additional stochasticity helps prevent the agent from getting stuck in certain states.
25

---

Published as a conference paper at ICLR 2025
Table 6: Dataset categories, We list the dataset categories used aggregate results in Figure 2. Note that datasets or tasks (e.g., PointMaze) do not helong to any these aggregation categories (for being too simple, too special, etc.).
Category
Datasets
antraze-nadiunn-aaziatt---
antrazze-larrge-naasaar----
Locomution (stales)
bussoiddazzeggiinn--vvvaata---
antoocsr--arrm---ariitt---
attaccar-addiu--viiitttee.
Locomotion (pixels)
cube-doubbe-ppayy--0
cube-triple-play-s0
cube-quadrrple--lley--0
Manipulation (states)
scene-play-v0
pueale-3x3-play-v00
puzale-5xé-play-00
puzzle-425-play-v0
puzzle-4x6-play-sd
Manipulation (pixels)
viaual-2ceme-play--00
vieuallppuzzlee-333-2ppyyy1
rinaal-puzzle-xxx-plaayy--
viaual-puuzale-445-pllaaa--
viaual-puuzale-xx--ppaaa---
poodersor1dd-aayy--laa---
Drawing (pixels)
poodersor110-hardd-paaa---
anfazzz-nnddua-stttttc----
aatoaze-large-sttttd---0
sataaze-giant--titddvvo
humansidazz--mddiu--tiiiiee
Stitching (states)
nttacccar-armma-atticch---dd
Stitching (pixels)
axtaaze-nediiur-erploree--
Exploratory (states)
antmaze-large-oxploreee
Exploratory (pixels)
Stochastic (states)
antmaze-telegee taiaze==ttteoort-aaviiatte-
Stochustic (pixels)
visual-aa alaual-antmsze-ttllppe -tt---ttttt-lllppppprr-aattt-eea
26

---

Published as a conference paper at ICLR 2025
Table Environment specifications. See Table 8 for the dataset specifications. Note that the episode lengths of datasets and environments can he different.
Environment Type
Environment
State Dim.
Action Dim.
Maximum Episode Length
pointaaze-medd1nm000
1000
pointmaze-large-v0
1000
pointmaze
pointnaze-giant-v0
1000
pointmaze-taleportt-vo
1000
adtaaze--edduu-10
29
1000
antmaze-large-v0
29
antnaze
antmaze-glant-vo
29
1000
antoaze-toleeort-vv
29
1000
bumanoidaaz--rodiuuuvg
69
21
2000
hunanoidmaze
humancidmaze-large-v
69
2000
humanooddnaze--gaaat--vd
69
4000
antsoccar-aremaa-10
42
1000
antaoccer
antsocceer-ted11--199
42
1000
64 64
1000
vismall-nttazzelarr--11
64.x 64 x
1000
-antnaee
viaull-anttaza-glaat-
64 64
1000
61xx 61 x
1000
r.aual-humannoddmzee----
64.x 64 x
21
2000
64 64
4000
cube-single-vd
28
200
cube-double-vo
37
500
cube
cube-triple-v0
46
cube-quadruple-t0
55
1000
acepe
scene-v0
40
750
puzzle-3r3-v0
55
500
puzzle-4x4-v0
83
500
puzzle
puzzle-4x5-v0
99
1000
puzzle-416-v0
115
1000
rieual-cube-aatggee-d
61 64 x
visual-cube
vieual-cube-double-vv
61 61 x
500
risual-oab-triille-vo
64.x 64 x
1000
rimall-uub--quddrpll---
64 64 x
1000
viaual-acene
vieual-8cene-v0
61 61 3
750
r1eual-puz2le--3x3-00
61 64 x
visual-puzzle
vieual-
LLeual-puzzlee4x44v0
61 61 x
500
visual-pusi
paauall-uzzle-445-00
64 x
1000
vieual -usual-puzele-4x6-v0
61 61 x
1000
powdersorld-easyvv0 poxdersorld-nediuuvvo posdarsorld-hard--00
32 32 x
8 (discrese) (discrete) 8 (discrete)
500
pouderworld
32 32 x
500
27

---

Published as a conference paper at ICLR 2025
Table 8: Dataset specifications. See Table for the environment specifications. Note that the episode lengths of datasets and environments can be different.
Enviroment Type
Daioset Type
Dataset
Transitions
Ejieedes
Episode Length
pot Intaaaaeeeed toe moor gig portarrr
1000
1000
poi niteere large meee gritemeo
1000
nay: gate
pointaess -a giant meet neeigptteeed
580
20000
point pointsaon
pointasze-tol wtw.aaae-ooeeeootrrrrreeeeo gatrrgate-00
1M
1000
1000
po pointaass and wit cchmvo
wti tch
pointause giant-stiishee pot nisssseetaa eepobbrat tcc-wr0
poinimese-leee large a
5000
casae mediuu nav igateem)
1000
uccanci-larg Lerge-----eeg-arrraea. gorggraggaae--- auta02c- glant saviatt-aviiae--00
nav aggtt
500
2000
an te agort may liggigee 0
1000
1000
wtitch
3073322- 4073322-1aant-6tte -t110-1
00:3000
10:00
explore
a023320-
19:00
anzanze-
uport-erploor--o
10:00
busamoidassee sndurasaeeoeedddrreeeeeeeee
2000
nawigatee
huxaao-deaaaa--arg---aaa gaterot
bananoddmaze
horanoodeaze-gh ant pate-ro)
kusanoidaaze dwwdeed
stitch
suxanoidaaze-ba tithh-v0
idewow of
nov-ga3e
1000
otitch
antencrer-areea stchhtthh-h
IM
1M
9000
5000
200
500
Visual
1M
1000
3000
nevigate
mmnn plant-nari getend
1M
1M
1000
500
2000
1000
La
1M
1000
1000
1M
5000
2000
vieuaal -actraze
stitch
viauul- maap::mmmmoaseegeeete.. sia.saaa--gn
viauul -A0:l-0--0---0000 largge largerttittt---
1M
1M
5060
5060
200
200
vigual -t0::aassstttllpe--------- tch-v0
1M
5000
200
5M
10:60
500
exploze
viaual www.aass---age-eee-
5M
10049
500
Vigual 00000ll00:3000-1100 http::00000aaa--oppp------------
5M
10009
500
2M
1000
2000
na gate
71 21m0s1-busnn wtw.-//enoooddddaaaaaarre large----rr-e---lile gate-vo0
2M
1000
2003
Vicual Satalll-nnanno
vidual- -wt.--uoooooooa---a wtw.:aaaaaa.gaatttaaaaeeeeeeee
4M
1000
4000
vimual-
2M
5000
403
stitch
vimun]-burnn --breenoddaaeeellarre large-arr------- tch-aa tch-w0
PM
5..0
103
visuk)-hurennn -bieenoiddraeeeeee stch-rt tch-w0
1M
10000
103
cube-single-plape
1M
3000
3000
play
tetp-trrppppppp----ee cube
cobe-double:ph
3M
1M
5000
1000
1000
1000)
5M
5000
1600
cibe-single-sotopp
1M
1000
1000
solay
cube-double-soing
cube-triple-soiie
3M
1M
3000
1000
3000
3000
cube-gadd:upee-od..
GM
5000
3000
5CODC
play
ccene-play-v0
1M
1000
1000
molay
vo
1000
1000
puzzle-3z2-p1ag-00
play
pozzle-4e4
1000
1000
p
1000
puzzle-332-noleyy-
Boloy
puzzle-dxd-moiay--0
puzzle--uf-aoisy--0
1000
1000
-uzzlo-xu--zoisy--
1000
1000
plag
1000
visual-cube
notay
play
boicy
1000
viausl
IM
1000
1000
1000
play
1000
visual-pizzie
vicusl coruull-przl--orte -o0--------0
5000
1000
viaus) "poscle Sed v3
1M
1060
1000
tolay
vieunl-pores poanl--prrlee-----ee
steual-puzzle-tts ----ppull--ddnooe-------
3M
3:00
1000
1000
1000
vimus) ppozsle 4edi-
5M
5000
1000
powdarvesld-ear paay------------
IM
1000
1000
pouderuarld
play
muudersorld-ode www.rerrdddoddduplllaaa
3M
3:00
1000
posthrserld hat.rrrerdd-adppplaee
5M
5000
1003
28

---

Published as a conference paper at ICLR 2025
Table Designated default tasks single-task environments. single-task (singletask) variants, each environment provides five tasks corresponding to the five evaluation goals, with the most representative one chosen as the default task.
Environment Type
Environment
Default Task
pointmaze-mediuree
task1
pointnaze
pointnaze-large-v0
taak1
pointnaze-giant-v0
taski
polntnax--tolgporttt0
task1
satnaze-redium-v0
taski
antmoze-largo-v0
anteaze
satmaze-giant-v0
taski
aatmaze-taleport-vd
hu
task1
humanoidman ddnzz
humanot
ammanotddraze-large-00
task!
humaaoo.draze-ggaatt-000
antaoccer-areea--10
taski
antsoccer--moiinn-10
rieual-anttnsze-mediun--
task1
vinual-anttnazz-llrrg--y
task!
visual-antmaze
viauallanttuaze-glaat--00
visual--attnzze---ellppor--
task!
visual-bumanoodnezz
taskl
taeki
cube-single-r0
task2
cabe-double-t0
task2
cube
cube-triple-v0
cube-quadruple-v0
task2
8cene
scese-v0
taek2
puzzle-3y3-40
task4
pazzle-4x4-v0
task4
puzzle
puzzle-4y5-c0
pizzle-4x6-v0
task2
vtaual-cube-single--0
vieual-oube
visual-cubb--doblle-00
taak2
vieual-cube-tziple--00
taek2
visual-cabe-guudruueeeyy
task2
visual-scene
visual-scene-vO
task2
-isual-puzzle--333--0
vieual-puzzle
viaual-pueg1e--x4-00
taak4
-1aua1-pu22ee-xx5-v0
taek2
visual-puzzil
iisual-puzzle-xx6-00
task2
29

---

Published as a conference paper at ICLR 2025
Table 10: Common hyperparameters.
Hyperparameterr
Value
Learning rate
0.0003
Optamizer
Adam (Kingnna & Ba, 2015)
# gradient steps
1000000 (states), 500000 (pixels) 1024 (states), 256 (pixels)
Minibatch size
MLP dimensions
(512. 2112,122,112
Nonlinearity
GELU (Hendrycks & Gimpel, 2016)
Target smoothing cocfficient
0.005
Discount facter ?
0.995 (( ((antmaze, pointmaze -giant, huranoidnaze), 0.99 (ethers)
Image augmentation probability
(pixel based manigulation), (others)
GCTVI GCIVI/GCIOL expectile
0.9
GCIQL expectile
QRL quasimetric
1QE (Wang & Isola, 2023)
QRL Istent dimension
512 (64 components dimensional latents)
QRI. margin
0.05
CRL latent dimension
512
HIQL expectile
HIQL subgual step
100 (bumanoidmaze), 25 (other locumotion). 10 (others)
HIQL subgoal representation dimeusion
Policy (yfur PPay. de Praj.Efoo p ratio for p
(0, 0.5. D, (stitchb) 0, 1) (explore). (0, 1,0,0 (others)
Value (pea PPea
Paoous praae) ratio for y Piizeed
00.2.0,0.6 0.3)
30

---

Published as a conference paper at ICLR 2025
Table 11: Hyperparameters for policy extraction. Each cell indicates the policy extraction method and its c value (i.e. the temperature (AWR) or the BC coefficient (DDPG+BC)).
Favironment Type
Type
GCIVI
GCIQr.
CRI.
HIQI.
AWR 10.0
DDPG 0.003
DDPG 0.0005
DEPG DUG3
3.0
navigate
polotaize- -aarora
polesasze-g-ante saaraaaooo----------r------
AWR 10.0
AWR 10.
DDPG 0.005
DDPG 0.003
DDPG 0.000
DDPG 0.000
DEPG OU23
DDPG 003
AWR
AWR
pointeazo
AWR 100
DDPG 0,005
DDPG 0.000
DDPG 000
3.0
medi wtitchhrr
AWR 10.0
DDPG 0.005
DDPG0.000
DORF
AWR 5.0
otitch
AWR 10.0
AWR 10.0
DDPG 0.005
DDPG 0.005
DDPG 0.00015
DDPG 0.0005
DDPG OUBS
DDRG 00S
AWB 3.0
AWR 5.0
AWR
DDPG 0,003
DDPG 0.0008
DDPG DUBS
AWB 3.0
anteace wwd. sar.uua-aaerggaeeee
AWR 10.0
DDRG 0.5
DOIN 0.0
DORG-O 01
AWR 5.0
nwvi navigate gate
usteaze
asteazi waw.:-gg-iit--aaateeeeee
asteace- wag.e-arrgaaeagnnnggg
AWR
AWR 10.0
AWR 10.0
DDRG 0.5
DDPG 0.3
DDRG 0.5
DONG FO0DDNO D00
DOOG DODEOOM O
DODG O
DOPGO
DORG 01
DORG-O 01
AWR 3.0
AWR 5.0
AWR 3.0
AWR 10.0
DONS 02
RONO 01
AWK 3.0
stiithh
anteaze gtent.wtitchhh...
anteaoe selagurt-atstthhee wtitcchrs
antaxze-1nrge warpaee--rg--tttthhhhe tchh--
AWR 10.0
AWR 10.0
AWR 10.0
DOPG
DONG 03
DNNG DOODONO 2000
000 1000
DONG
RO0G 01
AWR
AWR
AWR 3.
asteazi-18 wot.uaa--ooiuu--eoo cxplooe--0
AWR 10.0
DDPG
DONO 1M0
DOPG 2013
AWR 10.0
expLozo
AWR 10.0
AWR 10.0
DDPG 0.0L
DONG 100
20NO 100
DOPG 201
0010
AWR
AWR
busano-daaze
AWR 10.0
DOPG
DOPG 00000.00 POL
DOPG 1.000
DDPG
Sumasol
sbitch
navigate
busancidaaze barge-avigaat--e
harano large schrvn
harano sediur hodh
bumanoodaazz povigate--
bazanoddasse pisst-ttichrr schrvo
AWR
AWR 10.0
AWR 10.0
AWR 10.0
A0.
DDPG
DDPG TO
DDPG
DDPG 0.1
DOPG 0.1
DOPG 0.00
D0PE2.00
2000
0.000
100
DDPG
DDPG
DONGOL
DORGOL
XWR
AWR 3.0
AWR 5.
AWR 50
3.0
antxnocer arans navi antxnccer aed nar naripatarvd
AWR 10.0
AWR 10.0
DORG 00
DORG 0.1
DONO 0.003 1000
0000
DONGO
DONS 03
AWR 5.0
AWR 50
ANTROCOWT
ob3tch
sstoocer- -ao---c--arnn-ttied----- ustooccer watoocer--odddee -odi--------------------
AWR 10.0
AWR 100
DDRG 0.1
DDRG 0.1
DONG DONGO.00
DONG DONGORD 0.0
D0RS
DOPG 03
AWR 07
AWR
3.0
vieun] anttaaseeeddddn nact gateero0
AWR 100
DOPG 0.3
DOXO 000
ANRE 5.0
navigate
vipun] antaase -aagg nerigete
visun] antassee giant: may igaieerd
visunl antaaaze lappp maarigateee
AWR 10.0
AWR 100
AWR 10.0
DDPG 0.3
DDPG 0.3
DDPG 0.5
DONG DONDONO 20000
DOOND DONDDOO0 0000
DOIND DOIDDO00 000
DOPGO
01
0
vioual- unteaze
medi
large aastttce
AWR
AWR
DONG 0.003
0000
AWR 3.0
AWR 10.0
DDPG
1004 0,000
AWR
XWR
vicual- wauuul-attaaz--ooeee wowpaaa-oodioo-opooooeeeee
AWR 10.0
DORG
DIPGO.0
DOPG-000
AWR 10.0
oxplore
vicual- 1antll--ttzeeeae larg--l-ggg---reeeeee-----------
--attaane--te spors-neplores eoppore--e
AWR 10.0
AWR 100
DDPG 0.01
DOPG 0.01
36P40000
0010
10.0
10.0
Vicoal- Sraaadd
navigate
Viauel-bum -humazo daazze lazge igate-v0
Vicual- sastall-uuzzan Laazee glant- iaattee--0
vicual-hum www.doddod---oieeeeeem matigate-v
AWR 10.0
AWR 10.0
A0.10.
DDPG 0.1
DURG 0.1
DDPG 0.1
DDPG 00000.000 100
DUNG 0.000 100L
DOPG DOMG0000 000
DOPG 01
DOPG 01
AWX 3.0
AWB.30
3WR30
Visual- -usssuusooii cidaazze aadiuee itchh----
AWR 10.0
DDRG n.
DOPG 0.0
DORG-1
AWR 5.0
stitch
Visuel-bue -usssu waw.iiiiaz----aeaeee
Vicuol huslll-uuaaaooo sumaooddaae--geee platt----ttt-ttttcl---
AWR 10.0
AWR1000
DDRG m.
DDEG
DONG 0.000 1000
DOPG 000G0000 OON
DORS 01
DOPG
AWR 3.0
AWR 3.0
cibe-single-play
AWR 100
1000 O1
DONO 1000.00 30
AWR 30
play
mbbe trippe priplttepapapgogg
mibe-douh1e sobe-douhle-plaay palyaa--vd
AWR 100
AWR 100
D000 10
0-
DONG 53
DONG
1000.30 30
30
AWR
mabe quadrupla qualroplepplag-o
AWR 10.0
DOPG 11
DOPG
AWR AWR.59 5
seate-xingleeee snnglennttageen ag
DOPG
DOPMOOO 0.0 00
DOW0O0 00
AWR
DNNG 08M03
nodey
mabe trrplerrreeeeeee mu
AWR
0.00
siberguadruplersee
DDPG
DEPGOL
pccne-play-10
AWR
DDPG
AWR 3.0
natay
AWR 100
DDPG 0.03
DDPG DDPG000 0.05
DOPF
AWR 50
yiazle-tsd-glag-rn
AWR 100
DORG 1.0
DUPG
DOPG 30
AWR 3.0
porzle44e5
mazzte-tod-glag-oo
4edlle-4d--llyyam play
AWR. 10.0
AWR. 100
AWR. 100
DRRG 00
DORG 1.0
DOPG 0MPG-10 O1
DOMEO 3
DONE 0.3
DONG 3
D DOISSS S0
DO0S 50
DONS S0
AVA 50
50
puzsle
puzlle-30-ssoss--0
AWR 10.0
DDEG 0000
DONN 00060000 0.0
DOPG-1 01
AWR O.
nolay
puzzle-5x0-zooop-10
puzzle-5x6--o10--00
AWR 10.0
AWR 100
DORG 900
DDEG 1050
DONGO 000
D0NGO0S 00S
DOPG-OL 01
DORG.L 01
AWR 3.0
AWR
puzzlo-kx0-zotee-10
AWR 10.0
DORG 0/0
DONGOOS
01
AWR 3.0
AWR
DDPG
DDRGO3
DOPG
AWR
play
AWR 10.0
AWR 10.0
DORG
DDPG
30
AWR 3.0
AWR
vicoal-cube
AWR 10.0
DDPG
DONG 0.
DOPG DOPG30
AWR
siggl
AWR 10.0
DDPG DOPGOUD
sauull-cube-dublle-e
AWR
BDPG
0000 01
maiay
visual-cube-triplle
AWR 10.0
DOPG 11.0
BDPG
0ERGO1 01
AWR 10.0
DDPG
vixual- SOROM
vieual-ecom--lay-e play--o
AWR 10.
DOPG
DDPG
DOPO
AWR 3.0
nolcy
vicual-scessnnollyyee
AWR1000
DURG 0,00
DDPG 0.0
DDPG 0.1
XWR 3.0
visus) puose a 3rd -ad-playee
AWR. 10.0
DOPG 1.0
DONGOS 03
DOPS SO
AWR 50
play
visual- puseie--u------- 415-paay--
visual puscie 4xrile-444-- 4=4--pay-
AWR 10.0
AWR. 10.0
DORG 1.0
DORG 1.0
DOPG 0.8
DONG
DO0S S0
DOPG 50
AWR 5.0
AWR 5.
viousl-puzzle
visunl-puee poseee--tt--ppppeeeeeee
A0R.10. 10.0
DOPG 1.0
DOPG 0.3
DOPG SG
AWR 3.0
visual "puzzle 3a2-mal-- mo
AWR 10.0
DDPG 0.00
DORG 0.0
DOPGOL
AWR
visual ppseipeeeeaa m0
AWR 10.0
DDPG 0.00
DOPG 0.0
DDPG
AWR
mixx
visual wwws-p-------aaaeeeee 1a5-noiay-10
AWR 10.0
DDPG
DOPG 0.03
DDPG
AWR
AWR. 10.
DDPG 0.03
DOKG 0.0
DOPG
AWR 3
poteroor11-eas--plaeee poedersor1d x0
AWR
AWR33
poudersorid
play
AWR
postersor1d wo-.rrrddddddddlll paaa
31

---

Published as a conference paper at ICLR 2025
H EVALUATION GOALS AND PER-GOAL BENCHMARKING RESULTS
Each task in OGBench provides five evaluation goals. We provide their full image descriptions in Figures 4 to 10, and the full per-goal evaluation results Tables 12 24, which share the same format as Table 2.
taski
task2
task3
task4
task5
(pointmaze,
antmaze, humanoidmaze -atiizee--eediee -medium
US
UF
taski
task2
task3
task4
task5
(pointmaze,
antmaze, humanoidmaze)-leen -large
PEA
task1
task2
task3
task4
task5
(pointmaze,
antmaze, humanoidmaze, -giant
task1
task2
task3
task4
task5
{pointmaze, antmaze teleport
Figure 4: PointMaze, AntMaze, and HumanoidMaze goals.
32

---

Published as a conference paper at ICLR 2025
task1
task2
task3
task4
task5
antsoccer-arena
task1
task2
task3
task4
task5
antsoccer-medium
Figure 5: AntSoccer goals.
33

---

Published as a conference paper at ICLR 2025
taski horizoutal
task2
task3 Vertica12 cube-single
task4 diagonal1
task5 diagnoa12
taski
task2 double-pap1
task3 double-pnp2 cube-doube -ouu-d
task4 swap
task5 atack
aingle-pnp
task1 single-pap
task2 triple-pap
task3 pap-froa-stack cube-triple
task4 cycle
task5
task1 double-pap
task2 quadropla-pap
task3 pmp-fron-squrre
task4
task5
eycle
stack
cube- quadruuua
Figure Cube goals.
34

---

Published as a conference paper at ICLR 2025
t
taski open
task2 unlock-and and-lock
task3 rearrange- nediun
task4
task5 rearrange-bard
put-ia put-in-drawer
scene
Figure 7: Scene goals.
35

---

Published as a conference paper at ICLR 2025
task1
task2
task3
task4
task5
puzzle-3x3
t
taski
task2
task3
task4
task5
puzzle-4x4
Figure Puzzle goals.
36

---

Published as a conference paper at ICLR 2025
task1
task2
task3
task4
task5
puzzle-4x5
task1
task2
task3
task4
task5
puzzle-4x6
Figure Puzzle goals.
37

---

Published as a conference paper at ICLR 2025
task1 plant
task2 stone
task3
task4 equareq
task5 nossic
powderworld-easy
taski aquarea
task2 vater-plant
task3 aandpile
task4 two-rooma
task5 eeeentt
powderworld pnnderwwolddmeiiun
task1 bubbles
task2 firework
task3 three-roons
task4 four-equares
task5 ice-plant
powderworld-hardd
Figure 10: Powderworld goals.
38

---

Published as a conference paper at ICLR 2025
Table 12: Full results PointMaze.
Environment Type
Datnset Type
Pataset
Task
GCBC
GCIVE.
GCIQI
QRI.
CRI.
HIQI.
taskt
30
58 211
97
100
70
99
taxk?
3.2
95 211
76 20
94 mm
45 a/a
57
tac23
5.15
37
10.50
23
30
55
Leokd
Daa
2
01
94
28
82
taskE
922
79
07
24
70
cvera1l
0.as
63
82
79
taskt
63
76
BG
422
83
taaks
1.27
0
100
poistmize- Sauitaaz--argg-nav.ateeee
tackd
10:59
98
96
taok5
savigate
96
t362]
Lauk5
asx
1:42
pointasze-tel.comm politeas--tuppprr--rrippoooo
tacks
110
49 11
31 21
42 218
11
111
tacks
38 26
48 14
1.22
1:5
overall
25 s4
45
24
16
tackl
21 120
78 s.1
94 21
0.10
task2
82 sm
79 u23
28 ct
81
tasks
69 2:1
0
06
#2 2m1
taskd
41 237
0
65
0an
912
29 131
54
22
02
0.
50
overull
23
70
21
80
0
74
tack!
8
0:11
56 11
100
0.1
3.15
Laak2
Dis
01
74
0un
puintmese Largerstiteeeegg
task3
60
98
74
50
tack4
0.
0.
88
tackt
0.19
0
85
0.1
overall
7.
12
84
0.00
tack2
0
80
0.00
pointaesse itch-u
tasks
puintaesee Lagprtt--ee sepprtttttiin tchrol
texk5
1a
am
ovcrall
31
44
25
9.15
4.
34
39

---

Published as a conference paper at ICLR 2025
Table 13: Full results on AntMaze.
Environment Type
Dataset Type
Thitaset
Tisk
GCBC
GCIVL
GCTOL
QRI.
CRI
HIQI.
taski
35
81
65
93
97
94
taak0
21
78
50)
95
antrummeeentt tre-mrri getwned
tack3
28
59
83
56
taski
taak3
aargenact paatagtttgggoo
navi navigeta
overall
antraaze gl and nat--narigtte-o
tocel
0.23
0
0.20
0.29
taest
25
ovezall
01:3
0.4e
0.4
14 c/
16
65
taxel
as
as
26 c5
S1 20
55
task2
51 as
58
47 -23
92 um
(60
autause-1el
taxks
Lexie
25
22 ms
36
46
$1
35 mm
$5
53
47
60
90
37
tassl
14
38
26
28
44
41
ovezall
26
301
80
35
08
42
tasel
70 sm
70 2m
17 =11
43 :d1
92
tasks
65
801
22
61 +12
45
94
task3
21 21
16 122
41
72 150
45-11
95
Leak4
0.
$2
50
533
tasi
70
75
45
29
50
53
1:11
task2
0.20
4:24
Looks
Luakt
antmaze-gt -tt----tttttttttt----------
2.
0.
overall
0.23
0.20
0.29
0.20
0.20
2
tact2
210
18 u7
26
50
Mitazze-tcllporr-tttte
tasti
taxkd
31 219
46 10
44
18
18
24 1e
st
38
25 an
25
taxkS
16
33
17
26
$2
overall
31 ml
39 a3
17
24
$1
36
taskt
3
10
12
1:
2
29 am
Leskd
1
74
63 ms
1:tt
am
tass!
1:2
0.20
0.a9
S
18 2o4
taste
0
0
0
tastE
3
10
1
2.
52
overall
2
13
87
taski
37
1
0
LeakS
0am
0
0
explore
antzusse wepeewelloooudd
taxee
taskl
0
Laak8
epprrtreppl
2.
40

---

Published as a conference paper at ICLR 2025
Table 14: Full results HumanoidMaze.
Environment Type
Dataset Type
Dutaset
Task
GCBC
GCIVL
GCIQL
QRL
CRL
IIQ
taski
22
23
12 as
96
tastz
8
42
1s
15
96
iidwaze aad comennat cenrana gatwerd
tasts
taak3
2
12
0.
16
12
Lan
25 exx
16
43 es:
5.
71
75
tasks
12
40
51
20 a11
87 2T
97
overall
8.12
24 12
27
21
00
89
task!
1:t
6.2
S
36 c1
67
tast2
0
0:10
humano idrwonen Irngennari grtwarod
Leak3
Leakd
2
17
54
25
88
42
tasks
47
49
task
0
13
tasli
35
Nusannodaaaee -aaiigaavoo
tasks
teak4
drmanoddmazze
3
teaks
a3
overall
29
12
12
88
tank?
0.20
0.z
0.
xtitch
husano Ldeaze
tacks
tasks
2.11
20 -11
3.14
0.11
1::1
16 13
13
11
84
19
taaks
2
0
cm
12
overall
0.
0.2
3
4.21
28
taakt
0:se
0:m
0.m
0m
1.
task2
0
0:20
1
0.m
12 25
humateo weteee-ggtttttttteeee gtant-st:sch--r0
taxk3
tasks
0.
0.0
0.an
0:10
0.
0.0
0.0
0.2
0.n
0.19
1
2m
Laak5
0:
1
0.t
ovezall
0
0
3
Table 15: Full results on AntSoccer.
Environment Type
Datuset Type
Dutaset
Tack
GCBC
GCIVL
GCIQL
QRI.
CRL
ITQL
tack]
T.13
61
80
12:5
32
67
ta232
10.23
45:11
56
S:4
27
59 24
task8
622
113
12 cs
28
76
tackd
21
3-2
11:12
30
teoks
42
3
15
58
savigate
baoks
9
17
S
13
46
tac22
xatsoccer-xodin
tacas
stitch
overall
24 24
2
1
10.7
4.29
0.
satsoccer stt.:cec---edmm-ttttt-----
tack3
tackt
0.24
20
c0
10
10
20
1.
tasks
LG
0
Lo
0 40
overx1l
2
0
0
0
4
41

---

Published as a conference paper at ICLR 2025
Table 16: Full results Visual AntMaze.
Emironmenn Type
Phitaset Type
Dataset
Tank
GCBC
GCIVL
GCIQL
QRL
CRI
HIQU.
task:
17
30.45
15
0.40
takk
8.
21
7.
0.
visual rantauseered trsenar mnnigggeeedd
tack3
tackd
17
12
24:15
21
tasks
taxkd
LeckS
large-neriggn get
tackl
tas26
nevigate
tas22
vieual --antrasse- ggt ant--nniggtteeeee
tacet
0.11
0.21
0.
0.20
tacs
2:28
0.21
overall
1.
0 aa
0.00
G
tack:
2:13
6:21
2:a1
3
task2
6.43
9.
6.
visuel visual-antaaze -astaase- "attaaas-tlllorreee colgprr-zarrggaee-..
tekss
Leak4
10
10
12
6m
10
50 m4
33
taest
3:21
4:22
36 15
30
overall
6.a
37
tack:
80 24
0.
0.53
0.20
35
75
task2
90
1:4
0.40
69
85
visual-uati catenssoooddinnstttttte
tack8
Lood
R9
1
7
15
S
0.
70
88
92
ta230
6:
85
visual ancaaze
26
0.
0.
6.
tack?
0.29
2
visuel aattaaatt wwwaasss-laaaeeeeaea Laggesttt--eee-ttee----
teak
Leck
overall
Look:
tack4
LeckS
0.
am
overall
c0
0.21
0.10
20
0.31
tae32
36 23
2
la
ricual Suual--staaa---clppttttttttttt
tackd
tacks
2
2
2
32.29
Leaks
20
11
22
overall
32 ss
1:
32
Leck: cn
0.
cm
Dien
Lexk2
cm
0a
am
taes
0
0.
0:a
0.2o
tackd
0
taeid
0.
O
oewrall
0
tack:
0.
Leck2
0
visual --ntttess---aaaeeee www.erpppplc..m ooplorevv
tack
tackd
tacks
Leak2
sparts-aplore-eo
Leak
tac26
2
42

---

Published as a conference paper at ICLR 2025
Table 17: Full results on Visual HumanoidMaze.
Envirunment Type
Type
QRL
IIQQ
taski
taak2
0=n
-homanoo denow mavigetemmo
tacke
Laska
0
0
a
0en
0.
0
0.
2=
0.2
task5
Oie
a
Q.e
0.
everall
0.
1
0.o
1
0.14
taski
0.
0 -a
a
0.00
task2
0:
la
navigate
vi mnn) whimmmooi www.eeeiiddddeeeereeeae hww.eeaaee...eeeeeeeeee....aa
tacks
taak3
0:
0
0
0 aa
0
0.
0
0.59
tacks
0
0
0
0
0.2
everall
0 1z
Q..
0
0.
task1
0.
0
0 aa
0.
0
0.2
tack2
0
0
0 La
i
0.
0.10
--hmanno dtwwwwgg gettero
taak3
Laak4
0
0
0
0
0
tacks
0
0.
tack1
0:
0
tank2
vioual- hhmanoo wwwzeeaoodddaeeeeeddeeeee www.aareoooooi-ettteeeeeeeeeeeee
task3
taaké
3
visual- bww.aaoedaa----lgg--iiiaeeee
taska
taaké
0
taakS
0.
overall
21
0.2
0.
0.
0.51
am
tack?
21
0.
0.
0.21
visual hsmasoi aaatiattc-------------------
tacks
task3
1:
0
19
0
0
taak5
0.
0.
overall
0 21
0
0.2
0.
0.2
0
Table 18: Full results on Cube.
Enviroament
Dataset
Dataset
GCBC
GCIVL
GCIQL
CRL
57
0:
20
15
211
21
cube-stngleeppley--e
taszs
d
task]
tack3
ar
aa
0.se
task5
0.1e
3.1.
play
overall
96
40 15
1.
10:22
task1
5.11
13
19
tasx2
0.20
0.1e
2.21
cobertriplepplyy ppyyy
task3
0a
tas24
0
Dann
O
O
taxk5
Dan
D
O
oversll
1:
3 23
0
4.
task]
Ja
0
5a6K2
39
0
0
0
tas23
0
0
tasai
0
0
baak5
0
baak1
100
17
17
30
48
7
70
100
25
30
cabe-staglennoie--rp
36
37
task2
cube-dcube--ooty--
task3
as
tackS
nolcy
nowrsll
1
3
2
taak1
cabe-triple-ooie nooii--00
ta523
0
tadkt
20
20
task5
overall
9
3:
2
tagk1
0
0 -0
4.
task2
D
0
0
0.
eibe-ggadruph www.gidduullmmiiayee.oo momayyyo
53623
tadMt
0
0
0
0
0
0
0.
ta525
0
00
0
over3ll
0
0
0
43

---

Published as a conference paper at ICLR 2025
Table 19: Full results Scene.
Earirunment Type Datuset Type
Dutaset
Tusk
GCBC
GCIVL
GCIQL
QRL
CRL
H1QL
task:
18
75
11
19 a9
49
t3522
02
13
1.
12
40
by
nownerpl agrivd
tacM
Leck5
2.
3:
G4
8
72 ss
5:23
s:
26 s4
5:
55
36
taxks
0.
2:
1:
20
overall
5
42
51
19
38
back]
0
60
50
50 210
5:
68
ta522
0
42 L11
52
20
nceneroo
Leck3
Leak4
0-
27
3-
28
0
m3
10
17
tacko
0.
0
2.27
Table 20: Full results on Puzzle.
Eavizonment Type
Dutaset Type
Dataset
Task
GUBC
GUIVL
GCIOL
QRL
CRL
HIQL
Leak1
99
tas22
2
4-2
96
2
taxk3
S
95
2:1
94
95
42
10
puzete-fxt-play--00
23
play
a:
tasas
0.
D
overall
0:
7:11
144
taski
0:
43
52
12 15
12:13
task2
0:11
T.14
purcle<4a6-play
tas23
0
D
taxed
0
0.s
Dan
Dam
pozzle
taxt5
0.
Dan
Dann
operall
0:
10
12
0.29
3
taszi
4
89
100
1.
76 1n
67
tas22
0::1
42 :59
88
0.29
26 29
54 2:1
tas23
99
15
48
tas2i
0
21 ise
D4
12
taxes
88
18
94
taxed
51
49
0.
19
ouzio--xt-sotee-pp
S1
61
23
moiay
quzalo-us--ooeyy-e
overall
sm
19
50
27
puzale-ts--oooope
taski
tasei
0.
0.
1s
20
task5
0.
0
D
0..
overall
0
17
18
0
8
2
44

---

Published as a conference paper at ICLR 2025
Table 21: Full results on Visual Cube.
Eavirvnment Type
Datuset Type
Dutaset
Tusk
GCBC
GCIVL
GCIQL
QRL
CRL
HIQL
task:
12
88
47
93
ta522
1:11
63.15
14
35 194
30 120
933
vi mnn mombbe wwggnnggggggggggge
tase4
LeakS
6:
0.
60 2m
19
24
21 as
41 =m
52 sm
33 m17
15 216
84-
84
taxts
0.
55
20 57
50 an
16 16:21
overoll
11
60
30
41
31
89
tast1
8:23
24
0.24
20
7-s4
901
ta522
0 1.
0.11
[1
2.
54
Leck8
01
0.z
40
Leak4
tasko
4.20
0
pl
39
task
75
68
as
81
85
98
vinual-cubee trppee----pll-------0
ta5k3
ta523
cm
0.
viaual-cabe
taakS
Uss
overall
8:11
0.z
7:a1
5
4:21
taxt2
17:25
214
0n
28 211
100
visual-culen cooual-cbbe-innglll wiggee--oigeeelaeaeee
tacel
tacks
14
6:11
SQ.15
77
13
28 let
43 1:30
42 icc
09 11
100
Leak5
20
77
2
87 ext
90
overall
14 23
75
68 Aa
10 20
30 g00
99
Leck:
20 ch
70
70 ss
27
24 mn
98
tase2
2:12
14
2:24
3
87
Linbll-obbe-doulle-eee
tackd
taxts
2:a
8.
3
16 as
0.2
0
2
68 210
13
LeckS
D
80
molay
overall
17
22
69
Leck:
80
50
12
44
78
90
task2
0.s
II
ttoull-oube-trtpleeee
taxks
0.20
14
tacao
tack1
2:
2:
39
Leak2
Mioual-ptbe-uua ceteeeeuuuurrppooootttee
t3550
0.
overull
2.
0
8
Table 22: Full results on Visual Scene.
Fnvimmment Type
Type
Tisk
GCBC
GCIVI.
GCIQr.
CRL
play
Htoull-soos-plaa-00
2-4
Da
1:
ausl HEOee
12
26 mz
12
10
11
s4
taak!
64
49
8
70
Daga
2
tack3
0.
24
7:
0.20
2
BOLoy
task4
1:
8
tast6
0
0
0.1e
0.
0
ovara11
15
23 2z
12
2
15
50
45

---

Published as a conference paper at ICLR 2025
Table 23: Full results on Visual Puzzle.
Emironment Type
Type
IuQL
vioual visual-puzzies puzzie--22-paaya------
0en
0zn
3
0
0zn
everall
0.
21 11
1.14
73
tack!
27
0.
tack2
19
15
25 Lu
Lo
19 L13
1:2
vioual paneel-ppeie--ae -al-pp-------
tackd
23
80 11
15
144
taské
6as
65
10
Que
66 m44
tankd
6
61
0an
24
60
play
overall
10
60 21
16 24
0 an
10 cm
00
tanki
22
88
S1
S1 on
06
taak?
1:
0.
0 1:
0ann
0zt
vioual-puzele Baselle-----ppaa---
task3
Laaki
Ds
D
0
0 1a
0.1e
0.1e
task5
0 ss
cverall
5.
7
13
taski
12
66
10
12
34 esa
tank?
2
0.n
2
vioual sesualppzzzo--dee suxzzo-4x--pla----
0c
stmual-pozzle
vieual-puzzz puuullpuzzieeeeoi -ottlel---dd---t----oo---
66
as
s4
910
vi vteual puzzle-dri--ooo
tacki
2
100
47 Lat
77 2*
0zn
91
tacks
4.
20
34 an
0.2
04
notey
everall
47 a2
49
84 mc
tacki
10
72 am
96 1s
0.20
task2
=a
0
r.eual-puzzie--sti tetizzii---t---o----
tackd
taske
0r
0
0e
2e
0.
0.
0.
0.
tacki
0
0.
0 1n
evezall
6itt
14 am
19
7
14
tacki
10
61
82
0.1
9.2
56
task2
1
2
0.m
12
viaual vlaual-puadl pusdll-ddd--maaae 4c--miay-v0
task3
0
tacks
0r
0
0
12
17
14
Table 24: Full results on Powderworld.
Emirvnment Type
Type
QRL
pondersurrlddeerr playy-y
tuak3
tanks
0:z
gn
10
taoks
0zz
2.
2.27
overall
0.
99 11
90
12
22
33
0.z
279
7
2m0
tack2
28 Lus
0
9..6
2.
16
poodersorid
play
Luok3
2.s
90
2.
100
taoks
0
39 457
cc
taoks
0.
2:z
0unn
0ugn
1:27
overall
1.
50
16
1:
22
tack!
0 an
0
Dugh
task2
0
0.2
0.00
1:
pecdezwor1d-bard-llaae
tack3
taoks
0
0.3
12
0.3
0 c0
10
0.20
11
2
tasks
0
0.
0.m
0.1
overs]]
0
0
0g)
46


Reviews:
[{"review_id": "JE4Gj3ndF6", "reviewer": "Reviewer_jjTu", "summary": "In this work the authors present OGBench, a novel benchmark for offline goal-conditioned reinforcement learning (GCRL). The authors motivate the need for such benchmark, due to the lack of a standardized evaluation suite for works in offline GCRL and the increasing interest from the research community on the topic. The authors describe the main challenges tackled with OGBench: learning from sub-optimal data, goal stitching, long-horizon reasoning and stochastic environments. To tackle these challenges, the authors describe the design principles of the benchmark and consider a wide range tasks (locomotion, manipulation and drawing) across multiple environments. Additionally, the authors collect several different datasets per environment, considering different sub-optimality conditions of the behavior policy. Finally, the authors evaluate several literature-standard algorithms in OGBench, contributing their implementation as well, and discuss their performance and other findings.", "questions": "Please, also refer to the questions brought up in the \"Weaknesses\" section.\n\n- **1** - Why not also collect an \"exploratory\" dataset for the AntSoccer task? Is it due to the lack of coverage of states where the agent is dribbling the ball?\n- **2** - In the locomotion and manipulation tasks, how exactly is the goal state provided to the agent? Does it also include the final pose of the agent/robot? In that case, the task goal becomes a convolution of an environmental change (for example, a set of objects in a specific location in the Scene task) and a final pose of the agent?\n- **3** - I found it quite interesting the comment at the end of Section 8.2 (\"this suggests that we may need to prioritize coverage much more than optimality when collecting datasets for offline GCRL in the real world\") as this seems to go against the current trends in large scale collection of interaction data in robotics, where the data are collected across multiple environments/embodiments but always by expert-level demonstrators (e.g., [1]). Can the authors make any parallelism between the case of offline GCRL and other large-scale data collection efforts?\n\n\n**Reference**:\n- [1] Collaboration, Open-X. Embodiment, et al. \"Open X-Embodiment: Robotic learning datasets and RT-X models.\" arXiv preprint arXiv:2310.08864 1.2 (2023).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **Originality**:\n  - Recently, there has been an extensive push towards the development of benchmarks that focus on different aspects of reinforcement learning (e.g., for offline RL [1], [2]). However, as noted by the authors, there existed no centralized evaluation suite for algorithms in offline GCRL and the community often resorted to different set of tasks (not tailored for, making difficult to access the true performance of the proposed methods. As such, the development of a benchmark that tackles specific research issues in offline GCRL is most welcomed and, to the best of my knowledge, novel.\n\n- **Quality**:\n   - The work presented here is of high quality: the proposed benchmark contains a set of diverse tasks that address specific problems in offline GCRL research and high-quality, fine-tuned implementations of recently proposed algorithms for offline GCRL (that often surpass the performance of the original versions). Furthermore, the authors collect a wide range of datasets for each task, with different levels of sub-optimality.\n\n- **Clarity**:\n   - The current version of the work is well-written, with no major typos (that I could detect). Furthermore, the document is easy to read, with self-explanatory figures and tables. I would, however, recommend some refrain when using adjectives such as \"cool\" (Section 6, page 4) or \"exciting\" (Section 6, page 4), due to their intrinsic subjectivity.\n\n- **Significance**:\n   - This work proposes a novel benchmark for offline GCRL. Given the relevance of the topic for RL research, the work can have substantial impact by providing a standardized suite for evaluation of novel algorithms.\n\n\n**References**: \n\n- [1] Seno, Takuma, and Michita Imai. \"d3rlpy: An offline deep reinforcement learning library.\" Journal of Machine Learning Research 23.315 (2022): 1-20.\n- [2] Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" arXiv preprint arXiv:2004.07219 (2020).", "weaknesses": "My concerns with the current version of the work are the following (none of them particularly major):\n- Currently, it is challenging to assess the varying levels of novelty present in the environments available in OGBench. Throughout Section 7, I found myself questioning whether each environment was accessible elsewhere (in another benchmark or repository) or if it was unique to this benchmark. I do understand that some tasks (e.g., AntMaze) are heavily inspired by/available in other benchmarks. It would strengthen the originality of the benchmark if the authors clarified the level of novelty in each environment.\n- Currently the paper lacks any discussion about the limitations of the current benchmark and plans for future extension (for example, one potential extension could tackle robustness to data corruption as explored in [1], or integrating scenarios with multiple modalities).\n- Is not clear how the five evaluation goals per task were defined or what criteria was used for their selection. Additionally, why only 5 goals? Since GC policies should be able to reach any (viable) state from any state, it is unclear why the authors selected such a set of small goals.\n\n\n**References**\n\n- [1] Yang, Rui, et al. \"Towards Robust Offline Reinforcement Learning under Diverse Data Corruption.\" The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z48hE4lU6Z", "reviewer": "Reviewer_t6ud", "summary": "The paper proposes a new suite of tasks for benchmarking goal-conditioned offline RL algorithms. The proposed benchmarks are built with consideration specific to the goal-conditioned navigation tasks, with an emphasis on evaluating relevant task capabilities such as stitching, long-horizon reasoning, dealing with stochastic and visual input. The tasks cover a range of difficulty levels and types, including ant and humanoid locomotion, multi-step manipulation and a painting task with an high-dimension state space entailing combinatorial search. Offline policy data are also collected with various policies to emulate the data sub optimality. The benchmark is shown to be challenging for state-of-the-art and helpful in analysing strengths and limitations on investigated task capabilities. Code and policy scripted are provided and with a minimum dependencies on open-sourced mujoco and PyTorch libraries.", "questions": "* The expert data are collected from policies trained by RL. How can we assume the policies provide sufficient optimality? Since the all the tasks are essentially planning towards a goal, would running motion planning/search algorithms with certain guarantees give better data quality?\n\n* Can the benchmark add a study on the representation and the transferability to assess the idea of using goal-conditioned tasks as a general-purpose representation learning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Well-motivated research efforts and a solid execution with many considerations customised to goal-conditioned offline RL.\n\n* A good coverage of task variations and challenges to state-of-the-art algorithms based on different principles.\n\n* Proper task difficulties demonstrated from the results. Could be promising to spur new algorithmic research.\n\n* Decent writing with clear presentation and well organised flow of narratives.", "weaknesses": "* The benchmark is motivated for the generality of goal-conditioned navigation tasks, aiming at learning transferrable representations for down-stream tasks. However, this is not reflected in the task design and experimental results. It would be better to include some functionalities to facilitate analysing and transferring the learned latent representations, with a report on a few SotA algorithms' performance on this aspect.\n\n* The tasks seem to only challenge policies with different initial states but not the shift of transition dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDlSjcMeeS", "reviewer": "Reviewer_sCze", "summary": "This paper introduces OGBench, a comprehensive benchmark for the standardized evaluation of offline GCRL. OGBench provides a diverse range of environments and datasets, covering areas such as locomotion, manipulation, and drawing, to assess key capabilities like trajectory stitching, long-horizon reasoning, image-input handling, and managing stochasticity. The authors evaluate six offline GCRL algorithms using this benchmark, designed to highlight each algorithm's unique strengths and weaknesses across different tasks. OGBench is optimized for computational efficiency and user-friendliness, allowing researchers to easily test and refine their ideas, thereby advancing the field of offline GCRL.", "questions": "See the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Offline GCRL is an important research direction, and this work addresses the current lack of a challenging and comprehensive benchmark.\n\n* This benchmark evaluates GCRL algorithms across diverse aspects, including stitching, stochastic environments, high-dimensional control, and long-horizon control.\n\n* The benchmark includes popular baseline results, highlighting some unsolved tasks and areas for improvement.", "weaknesses": "Some benchmark designs need further discussion or improvement.\n\n* The evaluation seems limited to five tasks. Does this mean there are only five evaluation goals? If so, this number is restricted compared to several prior goal-conditioned tasks such as the Fetch environments.\n\n* The design of the transparent arm in pixel-based manipulation tasks seems unrealistic. It would be better to let users choose between a transparent or solid option. Additionally, demonstrating how transparency affects performance could be helpful.\n\n* The goal information is missing in Table 6. Is the goal in image-based tasks another image provided to the agent, or is it just a transparent bot as shown in the manipulation task figures? \n\n* Can the benchmark support language instructions as a special type of \"goal\"? This would be interesting given the current research focus on LLMs.\n\n* The stochastic task appears confined to the teleport domain. More stochastic settings that might be relevant to real-world scenarios should be explored, such as random observation perturbation during both data collection and evaluation. \n\n* Current baselines still use MLP networks as backbones. Could the authors consider implementing more advanced architectures, such as transformers? Are there any limitations in the current tasks that more sophisticated network architectures might address?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZa8BUYWbr", "reviewer": "Reviewer_BGac", "summary": "This paper contributes to offline GCRL by introducing a suite of high-quality benchmarks. These benchmarks systematically assess offline GCRL algorithms' performance in learning from suboptimal data, goal stitching, long-horizon reasoning, and handling stochasticity, providing a useful tool for advancing offline GCRL research.", "questions": "1. It would be better to retain one or two decimal places in Table 2. In benchmarks such as D4RL (Fu et al., 2021) and offline robotics (Yang et al., 2022), the success rate/score provided by decimal places is sometimes critical for comparisons. \n2. Could the authors clarify in the paper whether a rollout is considered successful if the goal is achieved at the last state, or if any state within the rollout reaches the goal? \n3. The authors should include a mention of GOPlan (Wang et al., 2024), which applies model-based planning with a generative adversarial network as the prior policy, in the related work. Additionally, GOPlan and GoFar (Ma et al., 2022) evaluate algorithm robustness in stochastic offline GCRL settings and should be discussed in Section 5.4.\n\n\n**References**\n\nFu, J., Kumar, A., Nachum, O., Tucker, G., \\& Levine, S. (2021). D4RL: Datasets for Deep Data-Driven Reinforcement Learning. \n\nMa, J. Y., Yan, J., Jayaraman, D., \\& Bastani, O. (2022). Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression. NeurIPS.\n\nWang, M., Yang, R., Chen, X., Sun, H., Fang, M., \\& Montana, G. (2024). GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models. TMLR.\n\nYang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., … Zhang, C. (2022). Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ICLR.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. They are very useful benchmarks to evaluate offline GCRL methods. \n2. The authors evaluate $6$ standard offline GCRL methods on the proposed benchmarks, establishing a solid baseline for comparison and further development.", "weaknesses": "The positioning of this paper is very similar to a previous paper - D4RL (Fu et al., 2021), which was well-known but rejected by ICLR 2021 (https://openreview.net/forum?id=px0-N3_KjA). \n\nThis paper makes a clear contribution by providing valuable benchmarks, but it does not introduce novel ideas. However, given its potential importance to the field, I lean towards assigning a slightly positive score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)