Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning
Authors: Yuanlin Duan
Guofeng Cui
He Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Goal-Conditioned Reinforcement Learning
Model-Based Reinforcement Learning
Exploration Strategies
World Models
Primary area: reinforcement_learning

Abstract: Exploring unknown environments efficiently is a fundamental challenge in unsupervised goal-conditioned reinforcement learning. While selecting exploratory goals at the frontier of previously explored states is an effective strategy, the policy during training may still have limited capability of reaching rare goals on the frontier, resulting in reduced exploratory behavior. We propose "Cluster Edge Exploration" (CE$^2$), a new goal-directed exploration algorithm that when choosing goals in sparsely explored areas of the state space gives priority to goal states that remain accessible to the agent. The key idea is clustering to group states that are easily reachable from one another by the current policy under training in a latent space, and traversing to states holding significant exploration potential on the boundary of these clusters before doing exploratory behavior. In challenging robotics environments including navigating a maze with a multi-legged ant robot, manipulating objects with a robot arm on a cluttered tabletop, and rotating objects in the palm of an anthropomorphic robotic hand, CE$^2$ demonstrates superior efficiency in exploration compared to baseline methods and ablations.

Exploring the Edges of Latent State Clusters for
Goal-Conditioned Reinforcement Learning
Yuanlin Duan
Guofeng Cui
He Zhu
Rutgers University yuanlin. duanOrutgers. edu
Rutgers University gc669êcs eG669ecs.rutgers.cm
Rutgers University hz3750cs ruttgers.
Abstract
Exploring unknown environments efficiently is a fundamental challenge in unsu- pervised goal-conditioned reinforcement learning. While selecting exploratory goals at the frontier of previously explored states is an effective strategy, the policy during training may still have limited capability of reaching rare goals the frontier, resulting reduced exploratory behavior. We propose "Cluster Edge Exploration" (CE²), a new goal-directed exploration algorithm that when choosing goals in sparsely explored areas of the state space gives priority to goal states that remain accessible the agent. The key idea is clustering group states that are easily reachable from one another by the current policy under training in a latent space and traversing states holding significant exploration potential on the boundary of these clusters before doing exploratory behavior. In challenging roboties environments including navigating maze with a multi-legged ant robot, manipulating objects with robot arm on a cluttered tabletop, and rotating ob- jects in the palm of an anthropomorphic robotic hand, CE2 demonstrates superior efficiency in exploration compared to baseline methods and ablations.
Introduction
In recent years, Goal-Conditioned Reinforcement Learning (GCRL) (Andrychowicz et al. 2017) has emerged powerful paradigm for training agents accomplish diverse tasks in complex and dynamic environments. GCRL enables agents to learn goal-directed behaviors, allowing them to achieve specific objectives in flexible and adaptive manner. However, central challenge in GCRL lies in guiding agents to effectively explore their environment during training. The exploration problem GCRL can be viewed as the task of setting goals for the agent during training to guide the agent's environment navigation to collect exploratory data that improves its learning process. In this paper, we address this critical challenge by proposing a novel strategy for selecting exploration- inducing goals in GCRL.
Because goal-conditioned policies excel at reaching states encountered frequently during training. a simple strategy is setting goals in less-visited areas of the state space to broaden the range of reachable states. However, throughout training. goal-conditioned policies may encounter difficulties in reaching arbitrary goals. For example, when instructed to navigate to an unexplored section of maze environment, a novice agent might instead revisit previously traversed area that provides low exploration value. To address this shortcoming, the environment exploration procedure must set filter out unreachable goals. A common strategy the literature is to
up additional mechanisms select goals at the frontier of previously explored states and launch an exploration phase immediately after these goals are achieved, adhering to a Go-Explore principle (Ecoffet al.) 2019). For example, Skewfit (Pong et al, 2 estimates state densities and selects goals at the fronticr from the replay buffer in inverse proportion to their density. Similarly, MEGA (Pitis et al. (020) uses kernel density estimates (KDE) of state densities and selects frontier goals with low density from the replay buffer.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, precisely identifying the frontier of known states can be challenging with these heuristics. Even once the frontier is identified, the policy during training may still have limited capability of reaching rare goals on the frontier, resulting in reduced exploratory behavior.
To address the aforementioned challenge, we propose new goal-directed exploration algorithm, CE² (short for "Cluster Edge Exploration"). When choosing goals in sparsely explored areas of the state space, CE² gives priority to goal states that remain accessible the agent. For this purpose, our key idea is clustering to group known states that are easily reachable from one another by the current policy under training, and traversing to states holding significant exploration potential on the boundary of these clusters before doing exploratory behavior. In this way, our method accounts for the capability of the current policy for exploratory goals. First, state cluster likely represents part of the state space where the training policy is familiar with. Second, given the easy accessibility of states within each cluster by the training policy, the agent's capability extends to reaching states even at cluster boundaries. Moreover, less explored regions naturally reside adjacent to the periphery of state clusters. This Go-Explore strategy enables the agent to progressively broaden the coverage of each state cluster to effectively explore novel environment. We instantiate CE² in the context of model-based GCRL, demonstrating how learned world models can facilitate clustering environment states that are easily reachable from one another by the training policy in a latent space. We validate the effectiveness of CE² in challenging robotics scenarios, including navigating a maze with a multi- legged ant robot, manipulating objects with robot arm cluttered tabletop, and rotating objects in the palm of an anthropomorphic robotic hand. In each scenario, CE² exploration results in more efficient training of adaptable GCRL policies compared to baseline methods and ablations.
Problem Setup and Background
Our work focuses on the exploration problem in unsupervised goal-conditioned reinforcement learning (GCRL) settings. In this section, we set up notation and preliminary concepts.
GCRL. A goal-conditioned Markov decision process (MDP) is defined by the tuple (S, A, G. where the state space S defines the set of all possible agent's observations into the environment, the action space A defines all possible actions that the agent can take in each state, G is the set of all possible goals that the agent may aim to achieve in the environment, and the transition function T describes the probability of transitioning from one state to another given an action. It is defined as T(s'ls, a). where s S is the next state, s E S is the current state, and a E is the action taken. S:S + G' is a tractable mapping function that maps a state to specific goal. A goal-conditioned (als, g) represents the agent's strategy for selecting actions based on states and goal commands, indicating the probability of taking action in state s given goal command g E G. In this paper, for ease of presentation, we assume G and 1) is an identify function.
Our goal is to develop agents capable of unsu- pervised exploration when dropped into an un- known environment. During the unsupervised exploration stage, there are no predefined tasks or goals. The agent sets its own goal command g E G explores the environment. Following this exploration phase, a successful agent should be able navigate to a wide range of previously unknown goal states in the environment upon goal commands.
Model
Train World Model
Train With Imagination
Aator
Actor
(1r 1. (.)5)))
Critie
Replay Buffer
nt (o.
(5
(a,1))))))
Store Trajectories
Interact
Real
Model-based GCRL. Model-based reinforce- ment learning (MBRL) is approach where an agent learns model of the environment's dynamics to prediet future states, enabling more efficient policy learning. Fig. shows the general
Figure I: Model-based GCRL Framework
MBRL framework. We use the world model structure M of Dreamer (Hafner et al. 2019a b 2020 2023) to learn real environment dynamics as a recurrent state-space model (RSSM). We provide detailed explanation of the network architecture and working principles of the RSSM in Appendix Particularly, we consider GC-Dreamer (goal-conditioned Dreamer) as a baseline. In GC-Dreamer the goal-conditioned agent G (a) [a]s,9 g) samples goal commands g E G from a given environment goal distribution Py to collect trajectories in the real world. These trajectories are used to train the world

---

model M, and subsequently, x is trained on imagined rollouts generated by M. with these two steps run in alternation. The reward function used to train S is determined by a temporal distance network Dj (see below).
Go-Explore. In unsupervised GCRL, the goal distribution Pa is only revealed test time. "Go- Explore" (Ecoffet et al., 2019 Pislar et al., 2021] Tuyls et al. [022 Hu et al. 2023) is a popular mechanism tailored for long-term GCRL scenarios that require extensive exploration. The Go- Explore methodology splits each training episode into two distinct phases: the "Go-phase" and the "Explore-phase" In the "Go-phase". the agent is guided to an "interesting" goal g (Pong al. 2019; Pitis et al. 2020) (e.g., states rarely encountered in the replay buffer) by the GCRL policy .,', reaching a final state sy. Subsequently, the "Explore phareeepaha" kicks in, with an undirected exploration policy x² taking over from ST for the remaining timesteps. This exploration policy is optimized to maximize an intrinsic exploration reward (Bellemare et al. 2016; Pathak et al. 2017] |urdda et al.) 2018 [Sekar al.) 2020) (e.g., to explore less familiar areas of the environment that the world models have not adequately learned).
Recently, Go-Explore has been integrated with model-based unsupervised GCRL (Menoonca et al. 2021 Hu et al. 2023), as depicted in Fig. [ In addition the goal-conditioned policy & [a]ss g). an exploration policy πt (s) is introduced into the model-based GCRL framework. The agent's training process involves learning the following components:
World Model:
M ((s(st-1,.(-)))) --1)
Exploration policy:
(st)
Goal Reaching policy:
C (st. g)
Exploration value:
(st)
Goal Reaching value:
r (s(.)))
where both σ and F are trained using the model-based actor-critic algorithm in Dreamer (Hafner et al. 2020). They are entirely trained with the imagined rollouts of the world model M to maximize the accumulated rewards r. and E.r respectively. The explorer reward rB encourages exploration by leveraging the Plan2Explore (Sekar et al.) 2020) disagreement objective, which motivates the agent to seek states that induce discrepancies among ensemble of world models. In contrast, the goal-reaching reward G is driven by the self-supervised temporal distance objective D, (Mendonca et al.) 202 l). which reinforces the policy to minimize the action steps required to transition from the current state s to a sampled goal state g in an imagined rollout, i.e., r (s. g) = ( (g)). The temporal distance network D, predicts the anticipated number of action steps needed to transition from s to g. It is trained by extracting pairs of states 8, and 8 from an imagined rollout generated by M and predicting the distance k as shown in Equation where H is the total length of the imagined rollout:
De (M:)s,,, ($(++)) se k/H
(I)
Here, V is a learned function for state embeddings in the world model (we assume S = G in the paper). Further details on the training procedure of D, can be found in Appendix C.2
CE2 aims to address the core challenge in the Go- Explore mechanism: how do we select an interesting
goal command g at the fronticr of known states with high exploration potential and effectively guide the agent to g?
3 State Cluster Edge Exploration
The major limitation in existing Go-Explore approaches, such as those described Pong et al. (2019); Pitis et al. (2020) is that the policy under training can struggle reach heuristically chosen goals at the frontier of known states (Hu et al., 2023). This difficulty arises because the goal commands are selected without systematic method to filter out unachievable goals for the agent, leading to diminished exploratory behavior. In CE², when choosing goals in sparsely explored areas of the state space in the "Go-phase", our method gives priority to goal states that remain accessible. For this purpose, the key idea is clustering to group states that are easily reachable from one another by the current policy under training in a latent space, and selecting states holding significant exploration potential on the boundary of these clusters as the "interesting" goals to explore. In Sec. 3.1 we discuss how to learn a latent space that can represent the reachability relationships between environment states. In Sec. 3.2 we explain how this latent space can be used to cluster states in the replay buffer that are easily reachable from one another. Sec. 3.3 we demonstrate how the agent can be brought to interesting states on the boundary of latent state clusters to effectively explore its environment.
3

---

3.1 Latent Space Learning
Typically, during the learning process of world model AM as neural network, an essential step involves encoding states from the original observation space into a latent space using an encoder. which can then be decoded back to the original observation space by decoder. This latent space is subsequently used to learn the dynamic model of the real environment (Hafner et al. 2019a 2020).
In CE², we additionally require the latent space can express the temporal distance between different states. In other words, we aim for the distances between various states in the latent space to represent the number of steps required to transition from one another in the real environment (after decoding) by the training policy. Therefore, the loss function of training the latent space in CE² comprises two components. The first component is the reconstruction loss Cree. akin to the latent space loss function in Dreamer framework (Hafner et al. 2019a 2020). It captures the association between the latent space and the re-decoding to the observation space, along with predicting dynamic transition in the latent space. We introduce a second loss term L that leverages the temporal distance network Dr in Equation | to guide the learning of the latent space structure. For any pair of states (s1, s2) sampled from the replay buffer, the Ld loss function is formulated as follows (! is learned function for state embeddings in the world model):
Lai ((1) (s2)) 2
(D.((())), (()) (s) (ss))))
(2)
Liatent = Lree + Latt
We use the loss function Liatent to supervise the training of the latent space. The trained latent space provides the agent with a deeper understanding of the real environment, where states that are easily reachable from one another in the real environment are closer proximity within the latent space.
3.2 Latent State Clustering
To identify the frontier of known states, CE2 conducts state clustering to group states in the replay buffer. States that easily reachable from one another are classified in the same cluster in the latent space by Gaussian Mixture Models (GMMs), based on the temporal distances between the encoded states. GMMs are probabilistic models that assume all data points are generated by a mixture of finite number of Gaussian Distributions. We initialize the Gaussian models in the latent space with Nº trainable latent centroids (. CN.. } and a shared variance o, where N₂ represents the desired number of clusters. These N₂ latent centroids are initialized by applying the Farthest Point Sampling (FPS) algorithm (Eldar et al.) 1997) to select a representative subset of states from a batch of data sampled from the replay buffer. We provide detailed description of the FPS algorithm in Appendix G.I. After initialization, we optimize the clustering model by maximizing the Evidence Lower Bound (ELBO) iteratively on sampled batches from the replay buffer with a uniform prior p(c) to scatter out the latent centroids (Zhang et al. 2021]:
log = (s)) 2 Bx(c(((s)) (og log (s)
(g(c)
(4)
where p and q are represented as Gaussian distributions within the GMMs. q(c) (Ts)) is the postior distribution over (the clusters) given encoded state (s)). log (TT(c)) is the distribution donating the probability of the encoded state (s) in cluster c. p(c) is the prior distribution of the weight of clusters in GMMs. For each round of optimization, we increase the probability of the sampled batches in GMMs by updating the weight of each cluster e in GMMs and the mean and variance of them.
3.3 Exploring the Boundaries of Latent State Clusters
Assuming we have already trained N₂ state clusters in the latent space, each representing part of the state space where the goal-conditioned policy under training is familiar with, how can we utilize these state clusters to plan an exploration strategy? CE² selects goal states at the edges of these latent state clusters for exploration because (1) less explored regions naturally adjacent to these boundaries, and (2) given the easy accessibility between states within each cluster by the training policy, the agent's 's capability extends to reaching states even the cluster boundaries.
We outline our exploration algorithm Algorithm [ At line 3 samples Neandidate latent states as Scondidate from GMMs. A higher sampling quantity ensures sampling from more states at the edges

---

of the clusters. We set Neandidate 1000 in CE². We compute the total probability of each latent state & Seandidate in the Gaussian mixture model, given by the formula:
p Σ 3,
In this formula, 3 are the mixture weights satisfying 3 2 O and Σ 3, - 1, sents the i-th Gaussian distribution with mean c, and the shared standard deviation ø. At line 4 we select Nedge latent states with the lowest total probability from Scandidate by Equation as set Sedge- Intuitively, these states reside the edges of the latent state clusters and, there- fore, induce a set of a goal commands Gedge = (T([f(())) E Sedge] J that may be used for the "Go-phase" for Go-Explore, where fo the state decoder and r the goal mapping function. However, randomly picking a goal com- mand from Gedge overlooks whether the policy can exactly navigate the agent to the sampled goal in the real environment. Although determining the exact outcome of the policy without execution is imprac- tical, similar to PEG (Hu et al. 2023), we can leverage the world model to pro- vide an approximation of the exploration potential p (g) a goal command g:
repre-
Algorithm I Cluster Edge Exploration(CEE)
I: Darp < {}
2: for episode to do Scandidate <- Sample Neandidate points from GMM
Gedge Nedge states in Seadidatee with the smallest total probability based Equation 5
g orgmes p through imagination with at
6:
t + GO-EXPLORR( x0.
Deep + Deep U
(ro())a)()) p(s II 81 (1-1) (a-1((--1)))))))) [
(6)
P (g) = E., Bppecroeee (V
Σ
V F (x7)
where o ~ ( (7)
K
In Equation 6, we simulate the "Go-phase" of Go-Explore over the world model M. We set each state from Gedge the goal command g for the goal-conditioned policy Nº to run over M and denote $r as the final state of the resulting imagined trajectory (here (*-()1-)(()) essentially induces the imagined trajectory distribution over the world model). In implementation, we the length of "Go-phase" to half of the maximum episode length for all environments. The time limits for both the Go and Explore phases during real environment exploration learned exploration value function V of explorer - to estimate the exploration value of s the final state of k-th imagined trajectory. We average the estimated exploration potential over K such imagined trajectories.
also set this value. We the
At line 5 in Algorithm after selecting the exploration target g with the highest exploration potential P from the latent cluster boundaries, we start the Go-Explore procedure in the real environment by executing the coal-conditioned policy TO to approach gd as closely as possible limited in T timesteps, followed by launching the explore policy πt for exploration limited in TE timesteps.
3.4 The Main Algorithm
We depict the main learning algorithm of CE2 Algorithm Recall that the learning objective is to train an agent that can achieve diverse goals revealed to it only at test time. Accordingly, in this algorithm at line 7, the data Deep collected to train the world model M generated solely by our Go- Go-Explore strategy outlined in Algorithm [ At line 6) we periodically update the cen- troids of the latent clusters again using the FPS algorithm (Fldar al.) 1997) from batch of latest trajectories from
Algorithm 2 The main training algorithm for CE²
I: Input: 7 = World Model MM, GMM, , -
2: Initialize replay butfer D
3: for i 1 to Nirain do
4:
if Should assign centroids then batch
from Desp GMM Choose N. centroids from Baep by FPS
Bex
7:
Deep + Cluster Edge Exploration(...) with Algorithm
8:
D + DU Deep
9:
Update M with D (update latent space by Cree Liatent)
10:
Update GMM with Dezp
Update nº in imagination with 1/ to maximize G
12: Update R in imagination with M to maximize F.F)
5

---

Ant Maze
Walker
3-Block Stacking
Pen Rotation
Block Rocation
Figure 2:
We conduct experiments on 6 environments: Point Maze, Ant Maze, Walker, 3-Block Stacking, Block Rotation, Pen Rotation.
the replay buffer. This ensures that the candidate goal states selected for exploration are indeed located the boundaries of key state regions. At line T0 we train the clustering model using data from the replay buffer in each round. This ensures that latent state clustering and the agent's goal-reaching capability are kept synchronized.
In our experiment, we also designed a variant of CE² in Algorithm 3] called
Algorithm 3 The main training algorithm for CE--G
CE2-G. This algorithm is given the en- vironment goal distribution Py at train- ing time. The main idea is to progres- sively expand the scope of exploration around the possible trajectories leading to the environment goals. In this al- gorithm, the replay buffer additionally includes Dege the trajectories sampled
1: Input: x² x². G. World Model M. GMM, r r
2: Initialize replay buffer D
3: Should assign centroids then
to Nuwuin do
Bege batch data from Dege G.M +- Choose N, centroids from B.ge by FPS
7:
Dexp Cluster Edge Explorationn...) with Algorithm [
8:
Dege + Rollouts of f.C using the env goal distribution Ps
9:
D + DU Dezp J Daye
by ahG conditioned on the environment goals in Pg. We only use Dege to ini- tialize and train latent state clusters. In this way, the agent encouraged to prioritize exploration starting from the edges of latent state clusters along the trajectories towards the goal states in Pg- CE2-G can be considered learning policies and world models specific given goal distribution.
10:
Update M with 1 (update latent space by Gree Listent)
11:
Update GMM with Dege
Update x in imagination with J/ maximize ,
13: Update R in imagination with M to maximize F
4 Experiments
Our experiments evaluate CE2 over goal-reaching tasks that demand significant exploration to solve. We aim to address the following questions: (1) Does CE² lead to improved exploration and goal- reaching performance? (2) How does CE² exploration qualitatively differ from those in previous goal-directed exploration methods? (3) Which components of CE² are crucial to its success?
4.1 Benchmarks
We evaluate method six hard exploration goal-conditioned RL tasks: Point-Maze, Ant-Maze, Walker, 3-Block Stacking, Block Rotation and Pen Rotation. Point-Maze: blue point is placed the bottom left of the maze and be trained to explore the structure of maze. Ant-Maze: An robot must master intricate four-legged locomotion behaviors and maneuver through narrow hallways. Walker: A 2-legged robot needs to learn how control its leg joints walk flat plane move forward or backward. In 3-Block Stacking, a robot arm with a two-fingered gripper operates on tabletop with three blocks. The goal stack the blocks into a tower configuration. The agent needs to learn pushing, picking, and stacking, as well as discovering intricate action paths to accomplish the task within the environment. Previous solutions have relied on methods like demonstrations, curriculum learning, or extensive simulator data, highlighting the task's 's difficulty. The Gymnasium Block Rotation and Pen Rotation tasks involve manipulating block and pen, respectively, to achieve a random target rotation along all axes. Pen Rotation is particularly challenging due to the pen's thinness, requiring precise control to prevent from dropping. For evaluation, we use most challenging goals, such as the farthest goal locations, in Point Maze, Ant Maze, Walker, and 3-Block Stacking. the other two environments, we utilize random goals as defined by the environment. For more settings and information about the environments, please refer to the Appendix E

---

1.00
0.80
6.00
MESSA
090
NeSA
02200
0.0
020
0.00
0.00
0/0
(a) Point Maze
(b) Ant Maze
(c) Walker
0.45
OF
0.52
12" -oog
0.15
MEGA
0.22
1442
0.22
3.5
0.55
0.35
10005
30
(d) 3-Block Stacking
(c) Block Rotation
(f) Pen Rotation
Figure 3: Experiment results comparing CE2 with the baselines over random seeds.
4.2 Baselines
In the unsupervised GCRL setting, we compared CE² with state-of-the-art methods based on the Go-Explore strategy. which has demonstrated high efficiency in this setup: PEG (Hu et al. 2023) and MEGA (Pitis et al. " MEGA commands the agent to rarely seen states at the frontier by using kernel density estimates (KDE) of state densities and chooses low-density goals from the replay buffer. PEG selects goal commands to guide an agent's goal-conditioned policy toward states with the highest exploration potential given its current level of training. This potential is defined the expected accumulated exploration reward during the Explore-phase
In scenarios where environment goal distributions with GC-Dreamer (illustrated in Sec. 2). PEG-G, MEGA-G and L3P. Similar to CE'-G, PEG-G and MEGA-G augment GC-Dreamer with the PEG and MEGA Go-Explore strategies, respectively. In these methods, the replay buffer D contains not only trajectories sampled by the goal-conditioned policy To commanded by environment goals but also exploratory trajectories sampled using the corresponding Go-Explore strategies. L.3P trains a latent space using temporal distances and performs clustering in this latent space, similar to CE--G. However, L3P does not employ Go-Explore strategy. Instead, it constructs directed graph with cluster centroids as nodes and utilizes online planning with graph search determine subgoals for task execution.
available the agents, compare C
4.3 Results
CE Results. Fig. 3 depicts the mean learning performance all the unsupervised GCRL tools in terms of the agent's s goal-reaching success rate averaged over random seeds. The evaluation goal distribution is revealed to the agent only at test time. In all tasks except PointMaze, CE² significantly outperforms PEG and MEGA in terms of both learning performance and learning speed. On PointMaze, CE² performs compara- bly with the baselines. Although MEGA can set goal commands in sparsely explored areas
S
S PEG
5
CE2
MEGA
Figure 4: Comparison of exploration goals (repre- sented red points) generated by CE2, MEGA, and PEG in the Ant Maze environment.
"Ou bosel-based MEGA baseline is borrowed from Hu et al. 2023

---

FRS.D
CC-6
FSS.0
CF-0
MEGNS
RSSc
C2-g
Crsurner
Oreamer
L33
0.40
13P
3.40
0.0
102
0.0
550
(a) Walker
(b) 3-Block Stacking
Block Rotation
(d) Pen Rotation
Figure 5: Experiment results comparing CE?-G with the baselines over 5 random seeds.
Ant Maze
5 60%
5
20%
40%
100%
Figure 6: Cluster evolution in CE as the training progresses. The red points means the goals picked by CE² to explore and other points in different colors represent the clusters CE2 learned.
of the state space to encourage exploration, unlike CE2, it lacks systematic method to filter out unachievable goals for the agent, which can result inefficient exploration. Theoretically. PEG can induce more exploration than MEGA because can sample goal commands as state within the state space initiate exploration, including those beyond the frontier of known states the replay buffer. However, because a learned world model is typically unfamiliar with rarely observed states, it may select goal commands that appear have high exploration potential in the model but perform poorly in the real environment as shown in Fig.
learning performance of all the tools in terms of the agent's goal-reaching success rate averaged over 5 random seeds when the environment goal distribution is revealed the agent training time. GC-Dreamer is the only tool that lacks Go-Explore phase, which may limit its exploration potential. Even so, it can sometimes outperform MEGA-G and PEG-G (see block rotation and pen rotation). This indicates that, without reasonably accounting the agent's 's capability to reach selected goal commands, the Go-Explore strategy does not always guarantee improved exploration. Suboptimal goal-setting during the "Go-phase" can even hinder exploration (see 3 block stacking). Notably, for the challenging 3-block stacking task, CE2-G achieves a high success rate exceeding 90%. In comparison, MEGA-G, PEG-G and GC-Dreamer only achieve less than 40% success rates. Refer to Appendix H.4 for full results of CE?.G.
CE--G Results. Fig. depicts the
4.4 Exploration Process
Fig 6 shows the evolution of state clusters (learned latent space) during the training process for Ant Maze (in different colors). The red points represent the selected goal commands used to induce exploration. We observe that the self-directed exploration goals set by CE² improve progressively the agent's capabilities increase, consistently targeting the cluster edges that require further exploration and are within the agent's reach. We compare the exploration targets generated by CR2 with those produced by the MEGA and PEG approaches throughout the training process in the Ant Maze environment in Appendix H.2
4.5 Ablation Study
In the ablation experiment, our goal determine the individual contributions of each component to our method's overall performance. The "Go-phase" of the Go-Explore procedure in CE² consists

---

UF
(E-cup
CC <oRGG
FGG
RCS
MEGA
0.33
Uritk APii
0.40
0:00
2.45
0.20
10
0.00
Ant Maze
(b) 3-Block Stacking
Block Rotation
(d) Pen Rotation
Figure 7: Ablation study on the importance of each component of CE2 over 5 random seeds.
of two main steps for selecting goal command y to initiate exploration: (a) sampling environment states at the boundaries of its trained latent state clusters, and (b) selecting the goal command with the highest exploration potential from the sampled states. Our first ablation. only performs step (a). randomly samples from the latent state clusters without considering its exploration potential. The second ablation only performs step (b) and is identical to the PEG baseline. It can sample any state within the state space for the goal command g, without the constraint of directing the agent to states the boundaries of known regions like CE² and MEGA. We also include MEGA and MEGA-wPEG as two baselines to solely compare the exploration strategy-step (a)-in CE with MEGA's strategy command the agent rarely seen states. MEGA-wPEC first uses MEGA to sample a batch of candidate goals, all with low density in the replay buffer. Then, their exploration potential is evaluated using PEG (step (b)), and the most valuable one is selected as the exploratory goal. We conduct the ablation experiments in a purely unsupervised setting without revealing any test goals training time.
Fig. 7 confirms that both step (a) and step (b) in CE² are important. CE2 significantly outperforms CE noo-noPEG and PEG in 3-block stacking and the Ant maze tasks. Notably, even without step (b), CE2 performs well across all experiments, especially in the challenging block and pen rotation tasks. This indicates that the goal commands sampled the edges of latent state clusters already possess high exploration potential and can guide the agent to traverse unseen state spaces. The superior performance of CoPE compared to both MEGA and MEGA-wPEG further reinforces this. Block Rotation is the only environment where no---nPEGG outperforms CE2 In this environment, the CE² agent often pursues states where the block falls from the palm, due to their "high" exploration potential determined by the exploration policy value functions. In contrast, CE2 Co"-noPEG agent explores the state space more evenly, gaining more in-hand manipulation skills, which is crucial for achieving the block-rotation goals revealed at test time. MEGA achieves similar or better performance compared to MEGA- -PEG, indicating that PEG's effectiveness relies on the quality of the candidate goal set. The exploratory goals sampled from the lowest-density regions in the replay buffer might beyond the agent's 's capability, leading PEG assess the true exploration potential of the candidate goals inaccurately.
We also conducted experiments in the CE² with different numbers of latent state clusters N₂ and observed that CE2 is insensitive to this hyperparameter. See Appendix H.5 for more discussion.
5 Related Work
Our method addresses the challenging and inefficient exploration problem inherent in goal- conditioned reinforcement learning (RL) settings with sparse rewards, commonly used in robotics and control fields (Ghosh al. 2019 Liu al. 2022 Plappert al. 2018 goal-conditioned RL, agents are trained to achieve various goals based on predefined commands, with rewards typically being binary, indicating positive feedback from the environment only upon reaching the specified goal. This sparse reward setting significantly complicates achieving sample efficiency and effective learning processes (Ren et al., 2019: Florensa et al. 2018 Trott et al., 2019). To mitigate this challenge, various methods have been proposed. Some reshape the sparse reward function into denser form by incorporating metrics such as distance between achieved and desired goals (Trott et al., (009)) or tem-
poral distance (Hartikainen et al. 2019; Mendonca al. 2021). Additionally, exploration strategies

---

often include rewards aimed at incentivizing visits to states with low visitation frequencies (Bellemare al. 2016 Burda et al. 2018). These approaches typically involve identifying states with infrequent occurrences within the replay buffer and targeting them for exploration, thus facilitating the discovery of unknown regions in the environment. Furthermore, some research emphasizes the ex exploration of states with high variance between ensemble predictions of future states (McCarthy et al., 202TT] Oudeyer et al. 2007 Pathak et al. 2017; Henafi
2019. |Syam et al., 2019 Sekar et al. 2020).
In addition to reshaping the exploration reward function, goal-directed exploration represents widely employed strategy that sets exploration goals distinct from the final task objective. Essentially, this approach aims to select goals that present challenges to the current policy while remaining achievable. Prior works have proposed various methods to generate goals for goal-directed exploration, [Zhang et al.] (2020) proposed to do automatic curriculum generation of goals based on the epistemic uncer- tainty of value functions. Florensa et al. (2018) use generative adversarial training to automatically generate goals, leveraging goal difficulty as a guiding factor. [Pogg et al. (2019) a et al. (2020) proposed to use the maximum entropy of achieved goal distribution to guide goal selection. Ecoffet et at (2019) introduce more efficient exploration methodology known as Go-Explore This approach initially employs the goal-conditioned policy (Go-phase), followed by the rollout of the exploration
policy from the terminal state of the goal-conditioned phase (Explore-phase), Go-Explore facilitates exploration initiation from a state area accessible by the current capabilities of the goal-conditioned policy.
PEG (Hu et 2023) proposes computing the exploration potential by simulating Go-Explore trajec- tories using world model to identify goals characterized by elevated average exploration rewards in the Explore-phase. This metric incorporates anticipated exploration rewards of the Explore- phase, providing an advantage for Go-Explore. However, the goals sampled for evaluating this exploration potential metric in PEG are drawn from a distribution updated by the MPPI method (Williams et al. 2015; Nagabandi et al. 2020) directly in the observation space. L3P (Zhang et al. 2021) employs temporal distance to train a latent space, facilitating clustering within this space to delineate key state areas based reachability. Our approach proposes exploration from the periphery of these key state regions, aiming to balance exploration of unknown territories while constraining exploration starting points to the edges of key state regions, thus avoiding meaningless exploration from widely sampled points from observation space. See Appendix for more related work discussion.
6 Conclusion
We present CB², a novel Go-Explore mechanism designed tackle hard exploration problems in unsupervised goal-conditioned reinforcement learning tasks. While CE² outperforms prior explo- ration approaches in challenging robotics scenarios, the requirement to learn state clusters identify frontier states and the reliance on world models to determine exploration potential introduce nontrivial computational costs. Exploring whether CE Go-Explore strategy can be effectively applied to model-free GCRL settings remains an interesting avenue for future work.
Reproducibility Statement
The codebase of CE² is provided on /tttps://githute http:///gghhuucc.o//-eete hhtp://U-Auttmateed-eeeeeeee inseooningGGruup/// CE2
For hyperparameter settings and the baseline pesudocode, please refer to Appendix and G.3
Acknowledgements
We thank the anonymous reviewers for their comments and suggestions. This work was supported by NSF Award #CCF-2124155.
References
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. (2017). Hindsight experience replay. Advances in neural information processing systems, 30.
10

---

Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D.. and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29.
Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. (2018). Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv: 12894.
Chaslot, G. M.-B., Winands, M. H., Szita, I., and van den Herik, H. J. (2008). Cross-entropy for monte-carlo tree search. Iega Journal, 31(3):145-156.
Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31.
Deisenroth, M. and Rasmussen, C. E. (2011). Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465-472.
Ecoffet, A., Huizinga, Lehman, J., Stanley, K. O., and Clune, (2019). Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv:1901.1099)
Eldar, Y. Lindenbaum, M., Porat, M., and Zeevi, Y. Y. (1997). The farthest point strategy for progressive image sampling. IEEE transactions on image processing. 6(9):1305-131:
Florensa, C., Held, D., Geng, X., and Abbeel, (2018). Automatic goal generation for reinforcement
learning agents. In International conference on machine learning, pages 1515-1528. PMLR.
Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S. (2019). Learning to reach goals via iterated supervised learning. arXiv preprint arXiv: 091.1992.00..0
Guan, L., Valmeckam, K., Sreedharan, S., and Kambhampati, S. (2023). Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems, (1:79081-79099)
Ha, D. and Schmidhuber, (2018). Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31.
Hafner, D., Lillicrap. T., Ba, J., and Norouzi. M. (2019a). Dream to control: Learning behaviors by latent imagination. arXiv preprint 101.:v111.001000
Hafner, D., Lillicrap, T., Fischer, I,, Villegas, R., Ha, D., Lee, H., and Davidson, J. (2019b). Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 3565-2565. PMLR.
Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. (2020). Mastering atari with discrete world models. arXiv preprint 1:Xiv22010.02193.
Hafner, D., Pasukonis, Ba, J., and Lillicrap. T. (2023). Mastering diverse domains through world models. arXiv preprint arXiv:2301.0417 04104.
Hansen, N., Lin, Y. Su, H., Wang. X., Kumar, V., and Rajeswaran, A. (2022). Modem: Accelerating visual basel-based reinforcement learning with demonstrations, arXiv preprint arX1v:2212.05698.
Hartikainen, K., Geng, X., Haarnoja, T., and Levine, (2019). Dynamical distance learning for supervipe and unsupervised skill discovery. arXiv preprint 1n0iv:1907.08000
Henaff, M. (2019). Explicit explore-exploit algorithms in continuous state spaces. Advances in Neural Information Processing Systems, 32.
Hu, F. S., Chang, R., Rybkin, O., and Jayaraman, D. (2023). Planning goals for exploration, arXiv preprint ar arX1v:2303.13002

---

Janner, M., Fu, J.. Zhang, M., and Levine, S. (2019). When to trust your model: Model-basee policy optimization. Advances in neural information processing systems, 32.
Kauvar, I,, Doyle, C., Zhou, L.,, and Haber, N. (2023). Curious replay for model-based adaptation. arXiv preprint arXiv:2306.1599
Liu, M., Zhu, M., and Zhang, W. (2022). Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint Stt.:2201.08999
Luo, Y., Xu, H.. Li. Y., Tian, Y., Darrell, T., and Ma, T. (2018). Algorithmic framework for model- based deep reinforcement learning with theoretical guarantees. arXiv preprint
McCarthy, R., Wang, Q., and Redmond, S. J. (2021). Imaginary hindsight experience replay: Curious basel learning for sparse reward tasks. arXiv preprint arXiv:2110.024144
Mendonca, R., Rybkin, O., Danilidiss K., Hafner, D., and Pathak, D. (2021). Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379 24391.
Micheli, V., Alonso, E., and Fleuret, F. (2022). Transformers are sample-efficient world models. arXiv preprint arXi::2209.00888
Nagabandi, A., Konolige, K., Levine, S., and Kumar, V. (2020). Deep dynamics models for learning
dexterous manipulation. In Conference on Robot Learming, pages 1101-1112. PMLR.
Oudeyer, P.-Y., Kaplan, F., and Hafner, V. V. (2007). Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2787.7 PMLR.
Pislar, M., Szepesvari, D., Ostrovski, G., Borsa, D., and Schaul, T. (2021). When should agents explore? arXiv preprint arXiv:2108.1 11811
Pitis, S., Chan, H., Zhao, S., Stadie, B., and Ba, J. (2020). Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR.
Plappert, M., Andrychowizz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej. M., Welinder, P., et al. (2018). Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 09464.
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. (2019). Skew-fit: State-covering sulf-super reinforcement learning. arXiv preprint arXiv:1903.03690
Ren, Z., Dong, K., Zhou, Y., Liu, Q., and Peng, J. (2019). Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. (2020). Planning to explore via self-supervised world models. In International conference on machine learning, pages 8583-8592, PMLR.
Shyam, P., Jaškowski, W., and Gomez, (2019). Model-based active exploration. In International conference on machine learning, pages 5779-5788. PMLR.
Trott, A., Zheng, S., Xiong, C., and Socher, R. (2019). Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural Information Processing Systems, 32.
Tuyls, J., Yao, S., Kakade, S., and Narasimhan, K. (2022). Multi-stage episodic control for strategic exploration in text games. arXiv preprint 20x:22200.00125).
Wagenmaker, A., Shi, G., and Jamieson, K. G. (2024). Optimal exploration for model-based rl in nonlinear systems. Advances in Neural Information Processing Systems, 36.
12

---

Williams, G., Aldrich, A., and Theodorou, E. (2015). Model predictive path integral control using covariance variable importance sampling. arXiv preprint 1XX.:0509900199
Zhang, L., Yang, G., and Stadie, B. C. (2021). World model as graph: Learning latent landmarks for planning. In International conference on machine learning, pages 12611-12620. PMLR.
Zhang, W., Wang, G., Sun, J., Yuan, Y., and Huang, G. (2024). Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36.
Zhang, Y.. Abbeel, P., and Pinto, L. (2020). Automatic curriculum learning through value disagree- ment. Advances in Neural Information Processing Systems, 33:7648-7659,
13

---

Appendix
A Discussion
Why does CE² perform better than the original Go-Explore mechanism?
Our algorithm, CE², tackles the core challenge the Go-Explore mechanism: how select an ssploation-indduinng goal command g and effectively guide the agent to g? Previous approaches, such as MEGA, set exploratory goals at rarely visited regions of the state space. However, these approaches, the policies under training may have limited capability of reaching the chosen rare goals, leading to less effective exploration. Our contribution is novel goal selection algorithm that prioritizes goal states in sparsely explored areas of the state space, provided they remain accessible to the agent. This is the key factor in why CE² outperforms the MEGA and PEG baselines in our benchmark suite in Fig. 3 As visualized in Fig. D in the appendix for the Ant Maze environment, CE enhances exploration efficiency by consistently setting exploratory goals within the current policy's capabilities. In contrast, MEGA and PEG often set goals that are unlikely to be reachable by the current agent.
Why did we not choose the original Go-Explore direct baseline?
As discussed above, the core challenge in the Go-Explore mechanism lies in selecting goal states that effectively trigger further exploration upon being reached. However, the original Go- Go-Explore method (Ecoffet et al.) 2019) does not prescribe general goal selection method, instead opting for sand-engineered novelty bonus for each task (e.g. task-specific pseudo-count tables). CE² is more related to recent instantiations of Go-Explore that automatically selects exploration-inducine goals in less-visited areas of the state space to broaden the range of reachable states, e.g. MEGA and PEG. Therefore, we compare our method with these tools instead Ecoofet al. (2019) in environments where these tools are applicable, to evaluate the strength of our goal selection method.
Why does our clustering algorithm not structure the latent space in the learning process?
While our clustering algorithm does not directly structure the latent space, requires the latent space to be organized in specific manner to be effective. In other words, the latent space learning algorithm is a key prerequisite for the latent state clustering algorithm. Specifically, our latent space learning algorithm structures the latent space such that states easily reachable from one another in the real environment (as determined by the learned temporal distance network as Equation I are also close together in the latent space. The clustering algorithm leverages this structure- cerouurepproprtty to ensure that the latent state cluster boundaries align with the frontier of previously explored states. As such, CE2 can efficiently generate exploratory goals at the frontier at training time.
B Extended Related Work
Model-based reinforcement learning (MBRL) has seen significant advancements in recent years, driven by the development of sophisticated world models and planning algorithms. One notable approach is Stochastic Ensemble Value Expansion (STEVE) (Buckman et al.) (018). which enhances sample efficiency by leveraging ensemble models to reduce overfitting and uncertainty in value estimates. Similarly, the work by CCua al. (2018) demonstrates that probabilistic dynamics models can be effectively used to achieve high performance in a small number of trials. In the realm of combining model-based and model-free methods, DDisennoth and Rasmussen (20TT) introduced PILCO, a data-efficient policy search method that uses Gaussian processes for dynamics modeling. More recent advancements include the integration of large pre-trained models for world model construction and task planning, as explored by Guan et al. (2023). The Dreamer framework by Hafner al. (2019)) utilizes latent imagination to learn behaviors directly from pixel observations, and its extensions (Hafner et al., 2020 2023) have shown impressive results in mastering diverse domains. The Recurrent World Models by Ha and Schmidhuber (2018) also contribute to this line of work by facilitating policy evolution through latent space planning. Several approaches focus on improving exploration strategies within MBRL. For instance, the use of cross-entropy methods for Monte-Carlo Tree Search (Chaslot et al., 2008) and the Curious Replay mechanism (Kauvar et al., 2023) have been proposed to enhance exploration efficiency. The work by Wagenmaker et al. (2024) further explores optimal exploration strategies in nonlinear systems. Additionally, transformers have been
14

---

leveraged for their sample efficiency in world modeling (iicheli et al. 2022 Zhang et al., 2024). demonstrating their potential in complex environments. The integration of demonstrations into visual model-based reinforcement learning, as seen in MoDem (Hansen et al. 2022). showcases another avenue for improving learning efficiency. [uo et al. (2018) provide a comprehensive framework with theoretical guarantees, while learned models.
whileJanner Janner et al. (2019) address the critical question of when to trust the
C Extended Background
C.1 Dreamer World Model
We use the world model structure M of recurrent state-space model (RSSM) of Dreamer (Hafner ctt al. b 2020 2023) to learn the dynamics. The complete model state the RSSM is the concatenation of deterministic states and stochastic states, with the latter being generated by the former. The deterministic state h, can used to get the prior state & and posterior state 2,. The & aims predict the posterior without access to the current input state 2; while the posterior state 2 is concluded by integrating the encoded information current input state 2t. The deterministic state h, updated by the recurrent transition function fo using the concatenation (h2, 24) or (he, &e) input. The world model is summarized in Fig g and the formulas of components are shown in Equation g
X
+
Dec
Dec
Gc-1
12--1
(he-2)
hr
(he, 2,)
4--1
Tt-1
Min
$
Enc
Enc
Xt-1
Xt
Figure 8: RSSM Structure
Encoder:
er = fF(ea(())
Recurrent model:
h4 1,
Representation model:
2t ~ ua(zellee, Et)
(8)
Transition predictor:
2. ~ po(Eel
Decoder:
G. ~ fo((t(ltt,
C.2 Temporal Distance Training in LEXA
The real-ng reward G is defined by the self-supervised temporal distance objective Mendoncal et al. 2021) which aims to minimize the number of action steps needed to transition from the current state to a goal state within imagined rollouts. We use by to denote the concatenate of the deterministic state ht and the posterior state at time step t.
b (h,, 21)
The temporal distance D, is trained by sampling pairs of imagined states b, betx from imagined rollouts and predicting the action steps number l between the embedding of them, with a predicted embedding é; from b, to approximate the true embedding of the observation Zt-
15

---

Tosk goal regios
CE
Normal Exploration Method
Figure 9; Illustration of differences between mothod CE²-G and other exploration methods.
Predicted embedding:
emb(b;) = ès s ct.
where
e₁ = fx(x))
(10)
Temporal distance: Dr(ê,, (x+R) 2 k/H
where = emb(br) êc4x = emb(braR)
(11)
of ((bb,bbtk) Di(ei,
(12)
D Limitations and Future Work
Our method clusters in the latent space, which necessitates a well-trained latent space. This latent space must only accurately reconstruct the original state space and facilitate dynamic prediction but also reflect the reachable distances between different states. Therefore, training this latent space requires a temporal distance predictor that can accurately estimate the number of action steps needed between two states. We utilize the temporal distance predictor network from LEXA, which constructs intrinsic coal-conditioned rewards, and this network trained using simulated trajectories. Compared to training the temporal distance predictor with real trajectories, using simulated trajectories offers greater stability. Our method requires the temporal distance predictor to reliably estimate the number of action steps needed to transition from one state to another, which is crucial prerequisite for ensuring the effectiveness of CE
Our realization of CE2 is based on Dreamer, a mhdel-based reinforcement learning (MBRL. agent known for its higher sample efficiency but greater computational demands compared to model-free alternatives. This increased resource requirement stems from the necessity to develop world model. In CE2, this world model utilized train policies and value functions through simulated trajectories. At the same time, CE² use the PEG as the filter of exploration potential, which rely world model select goals that guide exploration. Creating a model-free version of CE² would simplify both its computational and conceptual aspects, task we plan to undertake in future research.
E
Environments
E.1
3-Block Stacking
In this task, a robot is required to stack three blocks into tower. In PEG, evaluations are conducted on goals of varying difficulty levels, including 3 easy goals(picking up a single block), 6 medium goals(stacking two blocks), and hard goals(stacking three blocks). We evaluate our agent solely on 6 hard goals. At the same time, we use only 3 hard goals provided by the training environment as guiding goals for CE²-G. Relying solely the most challenging goals for training and evaluation presents a heightened challenge for both CE² and CF2-G. However, we observed that CE2 and CE2-G are capable of spontaneously discovering additional easy and medium difficulty goals through
16

---

clustering in latent space, as these serve crucial transitional states towards the hard goals. The environment features a 14-dimensional state and goal space: the first five dimensions capture the gripper's state, while the remaining nine dimensions correspond to the zyz positions of each block. The action space 4-dimensional, with three dimensions dedicated to the zyz movements of the gripper and the fourth dimension controlling the gripper's finger movement. The robot achieves success when the L2 distance between each block's zyz position and its target position is less than cm. This environment is modified version of the FetchStack3 environment from Pitis al. 20220). incorporating adjustments to better test the robot's precision in stacking.
E.2 Walker
In this environment, 2D walker robot is trained and evaluated the locomotion capabilities of on a flat surface. The environment code is sourced from Mendonca et al. (202I). In order to fully evaluate the agent's 's ability and accuracy travel to longer distances, we increased the number of 28) along the x axis
evaluation goals in PEG from 4(=7, +12) to 13 +1 119, #2, from its initial position. Noting that, in CE²-G, we only use goals at 113, 16 as the training goals returned by environments, but evaluate on all 12 goals. Success is determined by checking if the agent's position falls within a small margin of the target position. The state and goal space nine-dimensional, encompassing the walker's az positions and joint angles.
E.3 Ant Maze
This environment is adapted from the Ant Maze described in Pitis al. (2020). Like PEG, we set the goal space to be same with state space which including the ant's 's zyz positions along with joint positions and velocities, and an additional room was added in the top left to introduce a more challenging goal. In this complex environment, a high-dimensional ant robot must navigate from the bottom left to the top left of maze. passing through hallways. The task is challenging due to its long duration, with episodes lasting 500 timesteps, and the considerable distance to be traversed. Compared to evaluation on goals both in top left room and in the central hallway in PEG, our evaluation only focuses on the ant reaching the most difficult four goals in the top left room. Besides, we use all 32 goals of different positions the maze to be the training goals returned by environment for CE2-G. The maze itself measures approximately 6 8 meters. Success is determined by ensuring the L2 distance between the ant's ay position and the goal is less than 1.0 meter, roughly the width of cell in the maze. The Ant Maze environment features the highest dimensional state and goal spaces, totaling 29 dimensions. These include the ant's 's position, joint angles, and joint velocities. Specifically, the first three dimensions represent the zyz position, the next 12 dimensions correspond to the joint angles of the ant's s four limbs, and the remaining 14 dimensions capture the velocities in the zy plane and of each joint. The action space consists of 8 dimensions, controlling the hip and ankle actuators of the ant's 's limbs.
E.4
Point Maze
The 2D point agent starts at the bottom left corner of a 10 10 maze and is tasked with reaching the top right corner within 50 timesteps. The state space and action space are both two-dimensional. corresponding the agent's position and velocity on the plane. Success is determined if the L2 distance between the agent's position and the goal is less than 0.15. This environment is borrowed from Pitis et al. 2020 In CE2-G, the training goals are randomly chosen from 11 goals in different positions of the maze.
E.5
Block and Pen Rotation
The hand must manipulate both a thin pen or block to achieve target rotations. Manipulating the thin pen presents a greater challenge than manipulating the block due to the pen's tendency to slip, requiring more precise control. We utilize variant versions of the gymnasium environments: SaanddMMnippllaeeeeRRottte-- " and
BaluMMannpplaeeBokRRRttteee XYZ-v11 These environments introduce randomized target rotations for all axes of the block and x, y axes of the pen in each episode. The state space for both tasks consists of 61 dimensions, providing details on the robot's joint and object states, as well as goal information. The goal space remains consistent 7 dimensions, indicating the target pose information. During evaluation, the latest policy is evaluated across 50
17

---

episodes for each task, with each episode featuring a distinct random goal. In CE G, the training goals from environments is also randomly generated. Pen Rotation particularly challenging due to the pen's thin structure, which requires precise control to prevent from dropping. We intended to convey that this is the most difficult benchmark (with 61 observation space dimensions and 20 action space dimensions) in our test suite.
Baselines
In this section, we present the pseudocode for all baseline methods, Note that, except for L3P, each baseline employs different strategy for sampling data the real environment within this framework. Therefore, we first display the general training framework for MBRL and subsequently provide the pseudocode for each baseline's data sampling method in the real environment.
Algorithm General MBRL. Training Framework
1: Input: Policy G π Environment Goal Distribution G, World Model M, reward function C.
r
2: D < {} Initialize buffer.
3: for Episode = 1 to Niain do
4:
<< Collect trajectories(...)
5:
D
6:
Update model M with (&e, oe; 8841) ~ D
7:
Update Tº in imagination with M to maximize G
8:
Update _ in imagination with M to maximize 5
F.1 Go-Explore
To enhance the exploration area of the explorer, we adopt the Go-Explore strategy (Ecoffet et al.) 2019). This approach initially employs goal-conditioned policy, T approach specified goal as closely as possible, a process referred as the Go-phase, Following this, the explorer, n is used to further explore the environment starting from the terminal state of the Go-phase, known as the Explore-phase.
The effectiveness of the trajectories generated by the Go-Explore strategy heavily depends on the choice of the goal g during the Go-phase. Thus, establishing an efficient goal selection mechanism for the Go-phase is crucial. If the chosen goal is too easy, the explorer will not sufficiently explore the environment. Conversely, the goal g to approach it effectively. Therefore, the objective is to develop goal selection mechanism that identifies goal y capable of guiding the agent to a region with high exploration potential during Go-phase. Below, we provide the pseudocode for the Go-Expln Go-Explore strategy.
difficult, the goal-reaching policy Ω will be unable
Algorithm 5 Go Explore Framework
1: function GO-EXPLORE(g, G B)
2: 8g < env.reset()
3:
+ {
4:
for Step f = to Tep do
5:
enw.steppm ( (1))
6:
T - (st)
7:
if agent reach g then break
8:
9:
te =1
10:
for Step = La to te + Texpoore do
11:
8t - env.step(T (st-1))
12:
T T
13: return
18

---

F.2 GC-Dreamer
GC-Dreamer follows goal-conditioned approach where trajectories are collected by goal-conditioned policy n², and the goals are returned from the training environment.
Algorithm 6 GC-Dreamet Goal Sampling
function COLLECT TRAJECTORIESS...)
2: + Returned by environment
3: T Sample trajectories by n using goal g
4: refurn
F.3
PEG
PEG adopts strategy where trajectories are collected by optimizing a specific equation using the MPPI method. The optimized goal is then used to guide exploration through the GO-EXPLORE algorithm.
Algorithm 7 PEG Sampling
1: function COLLECT TRAJECTORIES(..)
2: f OOptimize Equation with MPPI
3: + --O-EXPLORE(gr x ))
4: return
F.4
PEG-G
PEG-G combines the utilization of goals from the environment with those generated by optimizing the equation using MPPI. This approach alternates between the two strategies based on the episode index.
Algorithm 8 PEG-G Sampling
1:
function COLLECT TRAJECTORIES(... if episode i%. then
2:
3:
- Optimize Equation 7 with MPPI
4:
GO-EXPLORE
5: else
6:
9
Returned by environment
7:
+ Sample trajectories by T using goal g
8:
return
E55 MEGA
For model-based MEGA, directly utilize the implementation method described in the PEG paper. This involves transplanting MEGA's KDE model and using goal-conditioned value function within the LEXA framework to filter goals based reachability. The PEG paper has demonstrated that their implementation of MEGA achieves superior performance compared to the original MEGA baseline.
Algorithm 9 MEGA Goal Sampling
1: function COLLECT TRAJECTORIES(..)
2:
mingeD p(gg
3: T G--EEPPLORE( G B.
4: return
19

---

F66 MEGA-G
Similar to PEG-G, MEGA-G alternates between using goals from the environment and MEGA goal picking strategy.
Algorithm 10 MEGA-G Goal Sampling
function COLLECC TR AJECTORIES(...)
2:
if episode 1%/ = 0 then
3:
- m p(g) GO-EXPLORE(g. T .T)
4:
5:
else
6:
f- Returned by environment
7:
<- Sample trajectories by x using goal g
8:
return
E.7 MEGA+PEG
MEGA+PEG combines the strategies of MEGA and PEG. this baseline firstly employs MEGA to sample batch of candidate goals, all of which have low density in the replay buffer. Subsequently, their exploration potential is evaluated using PEG, with the most valuable one selected as the exploration goal.
Algorithm 11 MEGA+PEG Goal Sampling
1: function COLLECT TR TRAJECTORIES(...
2: G <- Top-10 smallest p(g) for D
3: < Optimize Equation for E G
4:
GO-EXPLORE(
5: r
F8 CoPEGG
CE2-noPEG utilizes goal-picking strategy based Gaussian Mixture Model (GMM) clustering to generate goals for exploration without employing PEG optimization. The goal picked by this method sampled at the edge of our latent space clusters, which is the main contribution of our paper.
Algorithm 12 Co2-nGPEG Goal Sampling
1: function COLLECT TRAJECTORIES(..)
2: Deep +
3:
for episode i 1 to N, do
4:
Geandidate &- Sample Neandidate points from GMM
5:
Gedge f- Nedge points in Geandidate with the smallest total probability of the GMM. g +- Randomly select g from Gedge
6:
7:
GO-EXPLORE 50, RB)
8:
return
L3P
Our implementation of L.3P follows the original code provided in the L.PP paper (Zhang et al. 2021). For more details on the pseudocode and specific implementation, please refer the descriptions in their paper.
20

---

G Implementation Details
G.1 Farthest Point Sampling (FPS) Algorithm
Algorithm 13 Farthest Point Sampling (FPS)
function FPS(points, num samples)
2:
sampled_ sumpled_points--- [-
3: first_point +- random.choiccepoonns))
4: simpled__pintsse aopints.apped(((f first_ppot _ossstt_ont (frrtppoonttttnnn
5: min _in__ditteeesss t- [float( ('in""] x len(points)
6:
for each point in points do
7:
min min_distaacees(p + distance(p. first_point)
8:
for iteration to num_samples-1 do
9:
farthest
argmax(min _
10:
farthest_poing + bitiitsssfarthssspott_t_eeeeee
sumpled_pointer pwwpppttapppennaaaatteeeeeeeeeee for each point in points do
12:
min distances(p] <- min(min distiind_dttteespp... distance(p. farthest (etibbesttpintt)
14:
return
The Farthest Point Selection (FPS) algorithm, commonly employed in various applications including point cloud simplification and image sampling, initializes by creating empty list termed 'sam- pled _ond__poits'aa to retain the selected points. The process initiates by randomly selecting an initial point from the input point set, designated as 'points', and appending it to 'sampled pecappeed_poitts Subsequently, distancssn is initialized to track the minimum distance from each point to any of the sampled points, with initial values set infinity. The core procedure entails iteratively selecting points until reaching the desired number of samples. At each iteration, the algorithm identifies the point in "points' with the maximum minimum distance the previously sampled points and includes it in "sampled_ _ocoddd_po___.. Concurrently, "min_ diin_ddsttnees' is updated to reflect the recalculated minimum dis- tance of each point any of the sampled points. The algorithm incorporates two auxiliary functions: 'distanceppoint,, point2)', facilitating the computation of the Euclidean distance between two points, and (array' which returns the index of the maximum value within an array. See pseudocode Algorithm 13 for more details about FPS.
G.2 Runtime
Table 1: Runtimes per experiment.
Total Runtime (Hours)
Total Steps
Episode Length
Seconds per Episode
3-Block Stacking
60
le6
150
31.34
Walker
36
le6
150
18.62
Ant Maze
56
le6
500
96.91
Point Maze
36
le6
50
5.60
Block Rotation
58
le6
150
30.74
Pen Rotation
58
1e6
150
30.42
We conduct each experiment on GPU Nvidia A100 and require about 5GB of GPU memory. See Table I for specific running time CE2 for different task. The running time of CE2-G GG has big difference with CE2 The neural network updates of the policies and world model take most of runtime. However, CE² takes more time in goal selection compared to PEG and MEGA. This is because CE2 need evaluate the exploration potential of candidate goals sampled at the edge of latent clusters every time it picks a goal for exploration. the PEG algorithm, the MPPI parameters are only updated at fixed intervals of multiple episodes. This means that the exploration potential is assessed for a batch of data after certain number episodes have been completed. In contrast, method evaluates the exploration potential for batch of candidate goals at each episode when
21

---

sampling exploration goals. However, we can also follow the PEG by evaluating the exploration potential of batch of candidate goals after multiple episodes. Between these updates, we can select exploration goals from the previously evaluated set of candidate goals. While this may sacrifice some algorithm performance, can significantly reduce the time CH.² requires to select goals. We tried this setting and compared the computation time needed to optimize goal states for launching the Go- Go-Explore procedure among our CE and the baseline methods MEGA and PEG in the 3-Block Stacking environment. The average wall clock time are recorded in the Table 2
Table 2: Computation time needed to optimize goal states
Method
Seconds/Episode
CE
0.56
PEG
0.53
MEGA
0.47
G.3 Hyperparameters
Similar to PEG, we use the default hyperparameters of the LEXA backbone MBRL agent (e.g.. learning rate, optimizer, network architecture) and keep them consistent across all baselines. For the Gaussian Mixture MMdel we set the learning rate to be 3e-4, optimizer to be Adam. We also test result of different cluster number(10 10, 30, 50) setting as we show in ablation experiment. Every time we sample points in the GMM, we set the point number Noandidate 1000 and we set the edge point number Nedge 100. After then, we evaluate the exploration potential of these 100 points and pick the point goal state which have largest exploration potential value. In addition, to improve clustering in the latent space, we reduced the latent space dimension from 400(LEXA setting) to 50, We tested the training speed and results with both a latent space dimension of 400 and 50. We found that a latent space dimension of 50 allows for faster clustering and accelerates the training process. Meanwhile, although latent space dimension of 400 reduces the training speed little, it does not affect the final success rate.
H Additional Experiments
H.1 Space explored image for 3-Block Stacking
Space Explored
Space Explored
30
Color oradee over cimplarg time
2.0
2.0
0.5
00
0.0
0.0
0.0
XY edgee sum
XY edges sum
CE2
PEG
Figure 10: Space explored by CE² and PEG in the 3-Block Stacking environment 1M steps. X-axis: the sum of the three sides of the triangle projected on the x-y plane by the three block-connected triangles. Y-axis: sum of heights (z-coordinates) of the three blocks. Red points: evaluation goals. Other points: observations of trajectories sampled in real environment. Color from green to yellow means be sampled more recent.
22

---

In 3-Block Stacking, we innovatively designed a method based on the coordinates of three blocks to demonstrate the degree of exploration the environment, showed Fig TO Firstly, we establish connections between the coordinates of three blocks in three-dimensional space, forming spatial triangle. This spatial triangle serves to express the relative positions and distances of the three blocks within the space. Subsequently, we project this spatial triangle onto the xy-plane. The summation of the lengths of the sides of the projected triangle on the xy-plane reflects the dispersal of the three blocks within the xy-plane, while the total sum of the z-coordinates of the three blocks indicates their relative positions height Utilizing the former as the x-axis and the latter as the y-axis, we depict schematic illustration of the spatial exploration of 3-Block Stacking. We observe that CE² exhibits a more targeted and in-depth exploration around the target space (highlighted within the red box) compared to PEG. Simultancously.
observed that PEG tends conduct numerous explorations in the upper-left region of the exploration graph, which are often futile and irrelevant to the goal of completing block stacking. This highlights the advantage of CE² which benefits from the constraints imposed by clustering and avoids blindly exploring areas. Moreover, sampling at the edges of clusters ensures the profitability of exploration, enabling more efficient exploration of the vicinity of the goal space compared to PEG.
H.2
More Exploration Process
CR
MEGA
5
5 60%
PEG
10%
20%
30%
40%
Figure 11: Comparison of exploration goals generated by CE², MEGA and PEG
We present a comparison exploration targets generated by CE2, MEGA and PEG approaches over the training process in the Ant Maze environment. In the IT red points represent the generated exploration targets by different methods. We observe that the exploration targets generated by CE² are significantly superior to those generated by MEGA and PEG. Specifically, CE² consistently generates points located the forefront of agent exploration and within the agent's 's reachable capability range. In contrast, the targets generated by MEGA exhibit greater dispersion and sparsity, which are disadvantageous for concentrated exploration of forefront regions. Moreover, PEG consistently generates targets outside the Maze channels, rendering these exploration targets not only far beyond the agent's capability range also meaningless.
H.3 Centroids Visualization
By decoding the centroids of GMMs in latent space, we can visualize some centroids GMMs in
CE2, showing in Fig. 12
H.4 Full results for CE?-G
Please Fig.
13
23

---

Figure 12: some centroids visualization of GMMs in CE2
H.S More Ablation Experiments
We conducted experiments in the CE² with different numbers (10, 30, 50) of clusters to observe if the results are sensitive to the number clusters. The results are shown in Fig [4] We found that the performance of CE2 is not strongly correlated with the number of clusters; as long the number of clusters is sufficient to represent key state regions, the results tend to be stable, which demonstrates the robustness of our approach.
24

---

1.00
1.00
1.00
CE
CE G
CE G
PEG-G
MLG-
PLC-G
MESA-6
MEGA-G
GC-Draamer
GC-Dreamar
500000
0.90
0.10
0.40
20
10
5
10
Stupa
Staps
stags
(a) Point Maze
(b) Ant Maze
(c) Walker
1.00
CPPG
CE
G
MEGA-G
NP-MEAA-g::
GC-Oreermer
GC.Dreamer
UX-Ureemer
LSP
0.40
L3P
0.60
0.90
0.30
0:0
110
0.00
0.00
steps
(d) 3-Block Stacking
Block Rotation
(f) Pen Rotation
Figure 13: Full experiment Results comparing CE2-G with the baselines in six environments.
0.30
1.00
1.00
CP 10-centroodds
0.23
0.80
CE 30 centroids
CF2.50 oe-eettroid
0.80
CE centroits
50-centroids
5000255
0.60
0.00
0.15
0.30
0.00
0.00
0.00
(a) Block
(b) Ant Maze
(c) Point Maze
0.30
0.60
1.00
OP 10centints
[Fl-16erettmiis
0.25
Q3 ->-centroids
12-30-cantrolds
0.50
controlds
0.60
5102055
0.15
0.30
0.30
0.20
0.25
0.20
0.00
0.00
0.00
12606
staps
(d) Block Rotation
(c) Pen Rotation
Walker
Figure 14: Ablation Study with Different Cluster Number.
25

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly articulate the primary claims of the paper, specifically focusing on the development and implementation of the CE2 algorithm for goal-directed exploration in goal-conditioned reinforcement learning (GCRL). The introduction outlines the main contributions, including the proposal of a new exploration mechanism, the clustering strategy to prioritize accessible goals, and the validation of CE² in various challenging robotics environments. These claims are aligned with the theoretical framework and experimental results presented in the paper, ensuring they accurately reflect the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our paper clearly outlines several limitations of the proposed CE² algorithm in Appendix. Firstly, it emphasizes the dependency on a well-trained latent space, which must accurately reflect reachable distances between states and facilitate dynamic prediction. Training this latent space requires a robust temporal distance predictor, which we address by using the predictor network from LEXA, trained with simulated trajectories for stability, Additionally, we discuss that CE² may face challenges with more complex tasks such as peg insertion fluid tasks in ManiSkil12.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
26

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the Experiment section and appendix of our paper, we provide detailed descriptions of our experimental procedures and configurations. This includes elucidating the origins and modifications made to all testing environments. We also present the pseudocode and implementation methods for baseline models. Additionally, we specify the devices and memory resources utilized, well as enumerate the exact numerical values of the hyperparameters employed. Moreover, we have made our code openly accessible. For further details, please refer to the Reproducibility Statement section.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
27

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As we answer in the previous question. We have open source our code and provide detailed instructions to reproduce the main experimental results. We illustrate the benchmark source, baseline settings and CF2² implementation details.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. ccips.cc/public/gute wwtp://wwiisseeeeeeeeeee http://bbbii//uuuuddd/////eaaaas Sttps//ooooeSSSisiiiiiiiia osiocclicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed descriptions of the hyperparameters used for CR2 in Appendix, such varying cluster numbers and latent space dimensions. We also desbribe the setting of training, including learning rates, optimizers, and network architectures. This information is crucial for understanding and reproducing our experimental results.
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeated each experiment at least five times using different random seeds, and when plotting the results. As showing in the Experiment section, we displayed the experimental error. The solid line represents the average success rate, while the shaded region represents the standard deviation between the repeated experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгтor bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We clearly specifies the computer resources(Nvidia A100 GPU) and the amount of GPU memory required (approximately 5GB) in Appendix. Additionally, provides detailed information on the runtime of each experiment, including specific time metries such as episode length and seconds per episode.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
29

---

9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in our paper aligns with the NeurIPS Code of Ethics. We have reviewed the guidelines and ensured that our research adheres to ethical standards. Additionally, we have taken measures to preserve anonymity and comply with relevant laws and regulations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our study focuses solving exploration issues in the GCRL environment. At this stage, remains largely theoretical and has negligible societal implications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
30

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Our paper properly credits the creators original owners of assets used, including code, data, and models. The licenses and terms of use are explicitly respected. Specifically, cite the original papers for code packages datasets used, state the version the assets, and include URLs where possible.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have documented our code and provided detailed instructions on its usage, licenses, and permissible scope of use. Additionally, have included the documentation alongside the assets to ensure accessibility and clarity for users.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
31

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "XC57xoXBgw", "reviewer": "Reviewer_5zDz", "summary": "The authors propose a method for model-based exploration based on exploring reachable trajectories. The expand upon the Dreamer algorithm by optimizing the encoder to encode a notion of distance between state and optimizing for a likely reachable goal when training in the learned world model. The method is evaluated on a wide range of tasks, including dexterous manipulation tasks, outperforming other baselines in final performance.", "questions": "Why do you think the method is not able to achieve 100% on pen rotation?", "limitations": "Yes, on computational costs and if it can be applied in model-free settings", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clear and the algorithm well-motivated. The strong experimental performance in the complex tasks is promising and they provide nice visualizations of their algorithm versus others and over time.", "weaknesses": "One thing I would have like to see is some analysis of the additional computation time required to optimize for the goal states versus other methods. If it is prohibitively expensive, then the wall time could still be much longer than other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and constructive comments!\n\n**R1. One thing I would have like to see is some analysis of the additional computation time required to optimize for the goal states versus other methods. If it is prohibitively expensive, then the wall time could still be much longer than other methods.**\n\nWe compared the computation time needed to optimize goal states for launching the Go-Explore procedure among our tool CE$^2$ and the baseline methods MEGA and PEG in the 3-Block Stacking environment. The average wall clock time are recorded in the table below:\n\n| Method | Seconds / Episode |\n|--------|--------------------|\n| **CE$^2$**  | 0.56               |\n| **PEG**   | 0.53               |\n| **MEGA**  | 0.47               |\n\nThe results indicate that there is little difference in speed between methods. CE$^2$ adds only a minimal overhead in generating candidate goal states compared to the other goal selection baselines. We will include this discussion and the complete results for each environment in the revised paper.\n\n**R2. Why do you think the method is not able to achieve 100% on pen rotation?**\n\nPen Rotation is particularly challenging due to the pen's thin structure, which requires precise control to prevent it from dropping. We intended to convey that this is the most difficult benchmark (with 61 observation space dimensions and 20 action space dimensions) in our test suite. We will clarify this in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "psgWkMc7zh", "reviewer": "Reviewer_NgsZ", "summary": "This paper presents a method called \"Cluster Edge Exploration\" (CE2) to perform goal selection in goal-conditioned reinforcement learning and enable efficient exploration. Concretely, the method builds on the Go-Explore principle, which learns separate policies for exploration and goal-reaching.\n\nThe main idea is to learn a latent embedding space parameterized by Gaussian Mixture Models (GMMs) in the Dreamer framework. Selecting a goal is a two-step process. First, the method samples points from the GMM and only keeps a low probability collection. These points approximate the boundary of the existing explored states. Next, the algorithm uses the goal-conditioned policy to perform an imaginary rollout towards each goal candidate. That goal is selected if the rollout ends at a state with the highest exploration value. In addition, when the test goal distribution is known, the CE2-G variant learns the GMM only with trajectory data from test goals. This further provides an inductive bias to sample goals near the test distribution.\n\nThe paper presents extensive empirical studies in navigation, locomotion, and manipulation. Quantitatively, both CE2 and CE2-G outperform various baselines in their respective settings. Through visualizations of the sampled goals, CE2 bears the advantage that it samples points that are both near the frontier of the existing data and feasible to reach.", "questions": "- Equation 7 performs imaginary rollouts toward the goal candidates. How is $T$ chosen here?\n- In the ablation studies, the relative performance of CE2 and CE2-noPEG seems stochastic. Why would the inclusion of exploration value estimate hurt performance? Does this mean that the exploration value function is under-trained?", "limitations": "Yes, there's a dedicated section on page 15.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method's main idea is sound: sampling goals that are feasible but underexplored makes sense.\n- The authors did a good job explaining the algorithm and training objectives. They also made the distinction between EC2 and EC2-G clear.\n- The experiments are done in a good span of different domains. This shows that the exploration strategy is not tailored only for a certain application, such as navigation.\n- Visualizations of the goals picked by the CE2 and relevant baselines are insightful.", "weaknesses": "- It seems that some of the tasks haven't been trained to convergence. This makes it harder to draw definitive conclusions on the method's sample efficiency or final performance.\n- The method experiments appear to be done in state-based environments only. Since Dreamer is a strong algorithm for vision-based control, I'm curious how this approach performs with image observations. Would additional challenges arise for feature learning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and constructive comments!\n\n**R1. It seems that some of the tasks haven't been trained to convergence. This makes it harder to draw definitive conclusions on the method's sample efficiency or final performance.**\n\nWe thank the reviewer for the suggestion. In the global response, we have provided updated training results over extended environment interaction steps for our benchmarks in Fig. 1 of the attached PDF. Our method continues to outperform the baselines regarding both sample efficiency and final task success rates.\n\n**R2. The method experiments appear to be done in state-based environments only. Since Dreamer is a strong algorithm for vision-based control, I'm curious how this approach performs with image observations. Would additional challenges arise for feature learning?**\n\nOur method, CE$^2$, has so far only been evaluated in state space. A promising extension would be to handle image observations by optimizing goals in a compact latent space. This would likely require only minor adjustments to our code, as CE$^2$ (built upon Dreamer) already learns a latent space from state observations for goal command selection. We plan to explore this extension in future work.\n\n**R3. Equation 7 performs imaginary rollouts toward the goal candidates. How is $T$ chosen here?**\n\nIn our implementation, we set $T$ to half of the maximum episode length for all environments. The time limits for both the Go and Explore phases during real environment exploration are also set to this value. We will clarify this in the paper.\n\n**R4. In the ablation studies, the relative performance of CE2 and CE2-noPEG seems stochastic. Why would the inclusion of exploration value estimate hurt performance? Does this mean that the exploration value function is under-trained?**\n\nBlock Rotation is the only environment where CE$^2$-noPEG outperforms CE$^2$. In CE$^2$, our method selects goals by identifying states with the highest exploration value estimate sampled from the latent state cluster boundaries to initiate our Go-Explore procedure. Since we consider unsupervised exploration, the test goal distribution is not available to the CE$^2$ agent during training. In the Block Rotation environment, the CE$^2$ agent often pursues states where the block falls from the palm, due to their \"high\" exploration potential determined by the exploration policy value functions. In contrast, the CE$^2$-noPEG agent explores the state space more evenly, gaining more in-hand manipulation skills, which is crucial for achieving the block-rotation goals revealed at test time. CE$^2$ outperforms CE$^2$-noPEG in maze navigation environments (e.g., PointMaze and AntMaze) because it can use the exploration value estimates to escape the dead ends in mazes. We will incorporate this discussion into the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXBYp3HwmH", "reviewer": "Reviewer_7wGc", "summary": "This paper develops an approach for frontier exploration in the context of model based reinforcement learning. The key idea of the paper (inspired by prior work Go-Explore) is to cluster a group reachable states in the latent space and keep track of the current frontier, such that  new goals can be sampled close to the frontier. Experiments in simulated control benchmarks show that the proposed approach ism ore effective in solving hard exploration tasks compared to a few related baselines.", "questions": "Please refer to the weaknesses above.\n\n- The paper pitches the core contribution as an exploration strategy but couples it with model-based RL in the instantiation, but the exploration algorithm on its own is a small modification to Go-Explore.  Is there a reason why this instantiation is necessary?\n\n- Can the authors comment on the choice of completely orthogonal environments to those in Go-Explore (which is a direct baseline) ?\n\n- How feasible is the approach for deployment in real world control systems where building a model of the world may be much harder than learning a policy to solve a task?", "limitations": "Please refer to the weaknesses and questions above, and address limitations of the approach.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is well motivated and targets an important problem in reinforcement learning that has been studied for many years - that of effective exploration. \n\nUnder the goal-conditioned RL setting, the proposed modification to a prior work Go-Explore is novel to the best of my knowledge, and is intuitively sound in terms of choosing goals that are near the frontier and yet \"accessible\" with a high probability. \n\nThe paper is well presented, with sufficient background on prior works, and appropriate details on the algorithm and simulation environments used for experiments. \n\nThe experiments show better results than baselines on some challenging exploration tasks like in-hand manipulation and point maze navigation. The ablation experiments are good and provide into different stages of the approach like goal sampling, reaching the farthest reachable state, and exploration heuristic beyond that.", "weaknesses": "One of the main weaknesses of the paper is that the delta in terms of core algorithmic contribution beyond go-explore is limited. The paper pitches the core contribution as an exploration strategy but couples it with model-based RL in the instantiation, but the exploration algorithm on its own is a small modification to Go-Explore. \n\nA small change to an existing approach (Go-Explore) leading to massive improvements in tasks would be interesting, but the paper is not convincing in showing this. One reason for that is Go-Explore was shown to perform well in some very challenging scenarios (like Montezuma's Revenge) and the proposed approach doesn't perform direct comparisons on those environments. \n\nCombining model-based RL with exploration is a bit confusing. In many environments, a hard exploration problem needs to be solved (because of sparse rewards and long horizon tasks but learning a model of the world is an even harder task than exploring the environment to learn a policy - it seems the proposed approach will be very limiting in such scenarios)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and constructive comments!\n\n**R1. One of the main weaknesses of the paper is that the delta in terms of core algorithmic contribution beyond go-explore is limited.**\n\nOur algorithm, CE$^2$, tackles the core challenge in the Go-Explore mechanism: how to select an exploration-inducing goal command $g$ and effectively guide the agent to $g$? Previous approaches, such as MEGA (Pitis et al., 2020), set exploratory goals at rarely visited regions of the state space. However, in these approaches, the policies *under training* may have limited capability of reaching the chosen rare goals, leading to less effective exploration. Our contribution is a novel goal selection algorithm that prioritizes goal states in sparsely explored areas of the state space, **provided they remain accessible to the agent**. This is the key factor in why CE$^2$ outperforms the MEGA and PEG (Hu et al., 2023) baselines in our benchmark suite in Fig. 3. As visualized in Fig. 11 in the appendix for the Ant Maze environment, CE$^2$ enhances exploration efficiency by consistently setting exploratory goals within the current policy's capabilities. In contrast, MEGA and PEG often set goals that are unlikely to be reachable by the current agent.\n\n**R2. Can the authors comment on the choice of completely orthogonal environments to those in Go-Explore (which is a direct baseline) ?**\n\nAs discussed in R1, the core challenge in the Go-Explore mechanism lies in selecting goal states that effectively trigger further exploration upon being reached. However, the original Go-Explore method (Ecoffet et al. (2019)) does not prescribe a general goal selection method, instead opting for a hand-engineered novelty bonus for each task (e.g. task-specific pseudo-count tables). CE$^2$ is more related to recent instantiations of Go-Explore that *automatically* selects exploration-inducing goals in less-visited areas of the state space to broaden the range of reachable states, e.g. MEGA and PEG. Therefore, we compare our method with these tools instead of Ecoffet et al. (2019) in environments where these tools are applicable, to evaluate *the strength of our goal selection method*.\n\n**R3. The paper pitches the core contribution as an exploration strategy but couples it with model-based RL in the instantiation. Is there a reason why this instantiation is necessary?**\n\nOur method, CE$^2$, follows the Go-Explore mechanism by executing an exploration policy (defined in Line 109) upon reaching a chosen goal state to expand the range of reachable states. A primary reason for implementing CE$^2$ with model-based RL is to effectively train the exploration policy. We use intrinsic explorer rewards to encourage the exploration policy to reach novel states where predictions among an ensemble of world models diverge, based on the Plan2Explore disagreement objective (Sekar et al., 2020). The exploration policy is optimized purely from imagined trajectories. In our experience, this strategy results in stronger exploration capabilities compared to epsilon exploration policies, which randomly \"fill in\" sparse regions of the achieved goal space. For example, our model-based MEGA baseline uses the same exploration policy as ours and demonstrates improvement over the original MEGA implementation, which uses random epsilon exploration, in the benchmark suites we consider. We adopt this model-based approach for all the baselines to ensure the exploration policy is not the bottleneck in our experiments.\n\n**R4. How feasible is the approach for deployment in real world control systems where building a model of the world may be much harder than learning a policy to solve a task?**\n\nThe exploration strategy in CE$^2$ can be integrated with *any* model-based RL algorithm and applied to *any* tasks where model-based RL is applicable. Here, we specifically focus on the exploration problem in the unsupervised goal-conditioned reinforcement learning (GCRL) setting. During training, there are no predefined tasks or goals. A successful agent should be able to navigate to a wide range of previously unknown goal states upon receiving goal commands only revealed at test time. Intuitively, learning a policy in this setting is as challenging as learning a world model due to the vast range of possible goals. We believe this is a practical setting for deploying an agent into an unknown environment - the agent must learn about the environment and identify the feasible tasks it can perform without prior specifications. That being said, a model-free version of CE$^2$ should be conceptually simpler e.g., training the exploration policy with intrinsic rewards from an ensemble of goal-conditioned value functions instead of world models. Incorporating task-specific information, such as demonstrations and background knowledge, to guide CE$^2$'s exploration towards user-preferred areas would enhance its practicality, particularly for real-world robotics applications. We leave these extensions for future work.\n\nWe will incorporate the discussions from R1 to R4 into the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GegyQzOAQ2", "reviewer": "Reviewer_WfSq", "summary": "The paper introduces a cluster edge exploration (CE2) algorithm, which is implementing the “Go-Explore” principle in a – to the best of my knowledge – novel manner. Key idea is to use clustering of the state space latents – go to one of these clusters and then explore from there. As a main result, exploration improves and, as a consequence, success rates rise.", "questions": "Eq. 2 is not quite clear – particularly I wonder if here is not a unit problem as the first distance measure in latent state is representational while the second one depends on the movement distance estimates. Isn’t this severely restricting? \n\nEq. 4 could you elaborate slightly? How are p and q densities represented / determined? \n\nWhere does the learned value exploration function come from? \n\nWhat about environment, where sub-areas are hard to get to but do not offer themselves suitably for any further exploration.", "limitations": "An integration / automatic abstraction via shielded / gated latents would be much more appealing. \n\nThe evaluations are restricted to a standard test suite that does not really require deep reward propagation.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea builds on prior work based on go-explore, is charmingly simple, and yields positive effects as expected. \n\nThe model outperforms previous methods on standard tasks.", "weaknesses": "The evaluations are rather restricted to some standard artificial benchmark tasks. \n\nThe temporal distance network seems tedious to train additionally. \n\nThe algorithm in the end just introduces yet another method to explore the edge of the search space. \n\nThe clustering algorithm seems detached from the latent learning algorithm – it does not structure the latent state in any way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and constructive comments!\n\n**R1. The temporal distance network seems tedious to train additionally**\n\nThe temporal distance network can be trained efficiently using supervised learning from replay buffer trajectories to predict action steps from the current state to a goal state. Temporal distance networks are commonly used to create reward functions in goal-conditioned reinforcement learning e.g. [1,2]. We follow these prior works to implement this approach.\n\n[1] Discovering and Achieving Goals via World Models. NeurIPS 2021.\n\n[2] Planning Goals for Exploration. ICLR 2023.\n\n**R2. The algorithm just introduces yet another method to explore the edge of the search space**\n\nOur algorithm, CE$^2$, tackles the core challenge in the Go-Explore mechanism: how to select an exploration-inducing goal command $g$ and effectively guide the agent to $g$? Previous approaches, such as MEGA (Pitis et al., 2020), set exploratory goals at rarely visited regions of the state space. However, in these approaches, the policies *under training* may have limited capability of reaching the chosen rare goals, leading to less effective exploration. Our contribution is a novel goal selection algorithm that prioritizes goal states in sparsely explored areas of the state space, **provided they remain accessible to the agent**. This is the key factor in why CE$^2$ outperforms the MEGA and PEG (Hu et al., 2023) baselines in our benchmark suite in Fig. 3. As visualized in Fig. 11 in the appendix for the Ant Maze environment, CE$^2$ enhances exploration efficiency by consistently setting exploratory goals within the current policy's capabilities. In contrast, MEGA and PEG often set goals that are unlikely to be reachable by the current agent.\n\n**R3. The clustering algorithm seems detached from the latent learning algorithm – it does not structure the latent state in any way.**\n\nWhile our clustering algorithm does not directly structure the latent space, it requires the latent space to be organized in a specific manner to be effective. In other words, the latent space learning algorithm is a key prerequisite for the latent state clustering algorithm. Specifically, our latent space learning algorithm structures the latent space such that states easily reachable from one another in the real environment (as determined by the learned temporal distance network $D_t$ in Line 120, Equation 1) are also close together in the latent space. The clustering algorithm leverages this structure property to ensure that the latent state cluster boundaries align with the frontier of previously explored states. As such, CE$^2$ can efficiently *generate* exploratory goals at the frontier at training time.\n\n**R4. Eq. 2 is not quite clear – particularly I wonder if here is not a unit problem as the first distance measure in latent state is representational while the second one depends on the movement distance estimates**\n\nIn the implementation, to address potential scale issues, we normalize the movement distance estimate, used in the second component of Eq. 2, by dividing it by the trajectory length to ensure it is smaller than 1, ensuring it falls within a range similar to the first component. We will clarify it in the paper.\n\n**R5. Eq. 4 could you elaborate slightly? How are p and q densities represented / determined?**\n\nEq. 4 is used to optimize the Gaussian Mixture Models (GMMs) iteratively on sampled batches from the replay buffer. $p$ and $q$ are represented as Gaussian distributions within the GMMs. $q(c \\vert \\Psi(s))$ is the postior distribution over $c$ (the clusters) given an encoded state $\\Psi(s)$. $\\log p(\\Psi(s) \\vert c)$ is the distribution donating the probability of the encoded state $\\Psi(s)$ in cluster $c$. $p(c)$ is the prior distribution of the weight of clusters in GMMs. For each round of optimization, we increase the probability of the sampled batches in GMMs by updating the weight of each cluster $c$ in GMMs and the mean and variance of them.\n\n**R6. Where does the learned value exploration function come from?**\n\nThe learned exploration value function (used in Equation 7 for determining the exploration potential of a chosen goal state) is introduced in Line 109. This value function is used to guide the training of the exploration policy in our Go-Explore mechanism and is updated per learning iteration in our model-based GCRL framework (Line 12 of Algorithm 2). It encourages exploration by leveraging the Plan2Explore (Sekar et al. (2020)) disagreement objective, which motivates the agent to explore states in less familiar areas of the environment that the world models haven't adequately learned (intuitively such states often induce discrepancies among an ensemble of world models).\n\n**R7. What about environment, where sub-areas are hard to get to but do not offer themselves suitably for any further exploration**\n\nWe indeed considered such environments. We tested CE$^2$ in Point Maze and Ant Maze (Fig. 2) which contain dead ends from which exploration is doomed to fail. As explained in Equations 6 and 7 (Line 200), CE$^2$ selects goal states from latent state cluster boundaries with *the highest exploration potential*. We use the learned exploration value function (see R6) to evaluate the exploration potential of a goal state. As exploration progresses, the exploration value of states at a dead end decreases. Consequently, CE$^2$ can then select other goal states on the cluster boundaries that have not yet been explored well to escape the dead end.\n\n**R8. The evaluations are restricted to a standard test suite that does not really require deep reward propagation.**\n\nOur benchmarks include tasks with horizons of up to 500 steps (Ant Maze and Walker). The environments are also high-dimensional; for instance, Pen Rotation and Block Rotation have 61 observation space dimensions and 20 action space dimensions.\n\nWe will revise the paper to incorporate all the above discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gAMa1khEMo", "reviewer": "Reviewer_ZPaX", "summary": "This paper considers goal selection in goal-conditioned reinforcement learning. The core idea of this paper is to group states with small temporal distance into clusters, and then select goals that are on the clutter boundaries. In addition, the method gives priority to goal states that are accessible to the agent. Once the agent reaches the goal, a go-explore style exploration can be performed to potentially add new visited states to the clusters. The authors test their system across 5 different domains, and show improved performance over previous methods.", "questions": "- Figure 9 seems confusing. It seems to imply that CE2 would bias the exploration direction towards task completion, which shouldn’t be the case considering that the training phase of CE2 is task agnostic.\n- For previous methods like MEGA or PEG, couldn’t you still use the temporal distance to select “next goal to explore”? Why is the clustering necessary here?\n- Is there a more rigorous way of understanding why “less explored regions are naturally adjacent to these boundaries”? This seems intuitive in the original state space but I’m not sure if it carries over to the latent state space.", "limitations": "The limitation is briefly discussed in the conclusion section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written and easy to understand.\n- The idea of learning state representations that can help group state and identify easy-to-reach states is novel and exciting.\n- The experiment is thorough and the results are promising.", "weaknesses": "- It is worth pointing out that the method is only evaluated in relatively simple domains, with the maximum state dimensions being 29.\n- See the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and constructive comments!\n\n**R1. State Dimensionality in the Experiment Benchmarks**\n\nThe maximum state dimensions in our test suite exceed 29. Block rotation and pen rotation involve an anthropomorphic robotic hand with 24 joints. In total, the action space has 20 dimensions for absolute angular positions of the actuated joints and the observation space has 61 dimensions for information about the robot’s joint and block/pen states.\n\n**R2. Figure 9 seems confusing. It seems to imply that CE2 would bias the exploration direction towards task completion, which shouldn’t be the case considering that the training phase of CE2 is task agnostic.**\n\nWe apologize for the confusion. Figure 9 was intended to visualize environment exploration in CE$^2$-G. This paper presents two training algorithms where CE$^2$ (Algorithm 2, Line 212) is for unsupervised exploration in unknown environments and CE$^2$-G (Algorithm 3, Line 237) assumes the test goal distribution is available to the agent at training time. CE$^2$-G progressively expands the scope of exploration around the possible trajectories leading to the environment goals. We will correct the image label in Figure 9.\n\n**R3. Is there a more rigorous way of understanding why “less explored regions are naturally adjacent to these boundaries”? This seems intuitive in the original state space but I’m not sure if it carries over to the latent state space.**\n\nLess explored regions are located near the boundaries of latent state clusters due to the way we construct the latent space. Our loss function for training the latent space ($\\mathcal{L}_{dt}$ in Line 156, Equation 2) ensures that states easily reachable from one another in the real environment (determined by a learned temporal distance network $D_t$ in Line 120, Equation 1) are also close in proximity within the latent space. By enforcing the latent space to express the temporal distance between different states, $CE^2$ enables both efficient state clustering and frontier state identification.\n\n**R4. For previous methods like MEGA or PEG, couldn't you still use the temporal distance to select “next goal to explore”? Why is the clustering necessary here?**\n\nSimilar to the approach taken by MEGA, which sets frontier goals in low-density regions within the replay buffer, we could simply select goals that are furthest from the initial states, as determined by the learned temporal distance network $D_t$ (Line 120, Equation 1). However, this strategy can reduce exploratory behavior because the policy during training may still have limited capability in reaching rare goals. Instead, CE$^2$ selects the next goal to explore at the edges of latent state clusters, providing two benefits: (1) less explored regions are adjacent to these boundaries, and (2) given the easy accessibility between states within each cluster by the training policy, the agent's capability extends to reaching states even at the cluster boundaries. In other words, clustering enables CE$^2$ to precisely identify the frontier of previously explored states. For example, as visualized in Fig. 11 in the appendix for the Ant Maze environment, CE$^2$ enhances exploration efficiency by consistently setting exploratory goals within the current policy's capabilities. In contrast, MEGA and PEG often set goals that are unlikely to be reachable by the current agent.\n\nWe will update the paper to integrate all the discussions above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable feedback and suggestions from the reviewers. This global rebuttal includes a PDF file with updated training results and ablation study findings over extended training steps for our benchmarks (suggested by Reviewer NgsZ). Our method consistently outperforms the baselines in both sample efficiency and final performance.

We will address each reviewer's concerns in the individual review responses.

Meta-review: The paper Builds on Go-Explore principle and Dreamer framework using a two-step goal selection process. It introduces Cluster Edge Exploration (CE2) for goal-conditioned reinforcement learning, 
All reviewers agree that the paper is written clearly and provides an elegant and sound algorithmic idea. Further, strong experiments are provided across various domains.
Initial concerns were on limited novelty (wrt Go-Explore), thoroughness of running the experiments, and importantly limitations to state based environments. 
Further additional computational overhead from temporal distance network training is also pointed out. 
Authors point out that the method is extensible to image based environments. We recommend  that the authors consider performing initial experiments to that claim and also add related work in the space of image based exploration to the papers. In particular possible papers could be: [Lexa](https://arxiv.org/abs/2110.09514), [LEAF](https://arxiv.org/abs/2005.10934), [ExPLORe](https://github.com/facebookresearch/explore)

However, the paper presents a technically interesting idea with sufficient evaluations.  All reviewers lean accept, and the Metra-reviewer concurs.  The authors are advised to review the final comments, and update the manuscript accordingly.
Decision: Accept (poster)