Planning Goals for Exploration
Authors: Edward S. Hu
Richard Chang
Oleh Rybkin
Dinesh Jayaraman
Venue: ICLR 2023 notable top 25% (2023)
Keywords: model-based reinforcement learning
exploration
goal-conditioned reinforcement learning
planning
intrinsic motivation
reinforcement learning

Abstract: Dropped into an unknown environment, what should an agent do to quickly learn about the environment and how to accomplish diverse tasks within it? We address this question within the goal-conditioned reinforcement learning paradigm, by identifying how the agent should set its goals at training time to maximize exploration. We propose "Planning Exploratory Goals" (PEG), a method that sets goals for each training episode to directly optimize an intrinsic exploration reward. PEG first chooses goal commands such that the agent's goal-conditioned policy, at its current level of training, will end up in states with high exploration potential. It then launches an exploration policy starting at those promising states. To enable this direct optimization, PEG learns world models and adapts sampling-based planning algorithms to "plan goal commands". In challenging simulated robotics environments including a multi-legged ant robot in a maze, and a robot arm on a cluttered tabletop, PEG exploration enables more efficient and effective training of goal-conditioned policies relative to baselines and ablations. Our ant successfully navigates a long maze, and the robot arm successfully builds a stack of three blocks upon command. Website: https://sites.google.com/view/exploratory-goals

Published as a conference paper at ICLR 2023
PLANNING GOALS FOR EXPLORATION
Edward S. Hu
Richard Chang Oleh Rybkin
Dinesh Jayaraman
GRASP Lab, Department of CIS, University of Pennsylvania (hued, huangkun, oleh, dineshj @seas upenn.edu
ABSTRACT
Dropped into unknown environment, what should agent do quickly learn about the environment and how to accomplish diverse tasks within it? We address this question within the goal-conditioned reinforcement learning paradigm, by iden- tifying how the agent should set its goals at training time to maximize exploration. We propose "Planning Exploratory Goals" (PEG), method that sets goals for each training episode to directly optimize an intrinsic exploration reward. PEG first chooses goal commands such that the agent's 's goal-conditioned policy, at its current level of training, will end up in states with high exploration potential. It then launches an exploration policy starting at those promising states. To enable this di- rect optimization, PEG learns world models and adapts sampling- -based planning al- gorithms to "plan goal commands". " In challenging simulated robotics environments including multi-legged ant robot in a maze, and robot arm cluttered tabletop, PEG exploration enables more efficient and effective training of goal-conditioned policies relative baselines and ablations. Our ant successfully navigates long maze, and the robot arm successfully builds a stack of three blocks upon command. Website https: goog com/ vi ew expl oratory-goa
INTRODUCTION
Complex real-world environments such as kitchens and offices afford large number of configurations. These may be represented, for example, through the positions, orientations, and articulation states of various objects, or indeed of agent within the environment. Such configurations could plausibly represent desirable goal states for various tasks. Given this context, we seek to develop intelligent autonomous agents that, having first spent some time exploring an environment, afterwards configure it to match commanded goal states.
The goal-conditioned reinforcement learning paradigm (GCRL) (Andrychowicz et al.. 2017) offers a natural framework to train such goal-conditionss agent policies on the exploration data. Within this framework. seek to address the central problem: how should GCRL agent explore its environment during training time so that it can achieve diverse goals revealed to it only at test time? This requires efficient unsupervised exploration of the environment.
Exploration in the GCRL setting can naturally be reduced to the problem of setting goals for the agent during training time: the current GCRL policy, commanded to the right goals, will generate exploratory data to improve itself (Ecoffet et al. 2021 Nair et al. 2018b). Our question now reduces the gial-directed exploration problem: how should we choose eeploration-inducin goals at training time?
Prior works start by observing that the final GCRL policy will be most capable of reaching familiar states, encountered many times during training. Thus, the most direct approach to exploration is to set goals sparsely visited parts of the state space, to directly expand the set of these familiar states (Ecoffet et al., 2021 Pong et al. 2019 Pitis et al. 2220)). While straightforward, this approach suffers from several issues in practice. First, the GCRL policy during training is not yet proficient at reaching arbitrary goals, and regularly fails to reach commanded goals, often in uninteresting ways that have low exploration value. For example, a novice agent commanded to an unseen portion of maze environment might respond by instead reaching previously explored part of the maze, encountering no novel states. To address this, prior works (Pitis et al., 20200; [Bharadhwaj et al. 2021) set up additional mechanisms to filter unreachable goals, typically requiring additional

---

Published as a conference paper at ICLR 2023
hyperparameters. Second, recent works (Ecoffet et al. 2021 Yang et al. 2022. Pitis et al. 2020) have observed improved exploration in long- hong-horizon tasks by extending training episodes. Specifically, rather than resetting immediately after deploying the GCRL policy to goal, these methods launch a new exploration phase right afterwards, such as by selecting random actions (Pitis et al. 2020 Kamienny et al. 2022) by maximizing intrinsic motivation reward (Guo et 20.0) In this context, even successfully reaching rare state through the GCRL policy might be suboptimal; many such states might be poor launchpads for the exploration phase that follows. For example, the GCRL policy might end up in a novel dead end in the maze, from which all exploration is doomed to fail.
To avoid these shortcomings and focus exploration on the most promising parts of the environment, we pro- pose to leverage planning with world models a new goal-directed exploration algorithm, PEG (short for "Planning Exploratory Goals"). Our key idea is to optimize directly for goal commands that would induce high exploration value trajectories, cog- nizant of current shortcomings in the GCRL policy, and of the exploration phase during training. Note that this does mean merely commanding the agent to novel or rarely observed states. Instead, PEG com- mands might be previously observed state, or indeed, even to physically implausible state (see Figure Ip. PEG only cares that the command will induce the chained GCRL and exploration phases together to generate interesting training trajectories, valuable for policy improvement.
PEG
Figure PEG exploration in U-maze. Brown background dots: explored states, *: commanded goals, colored lines: resulting paths. (Left) PEG optimizes directly for exploration, even setting unseen goals, and achieving farther paths. (Right) Setting goals at the frontier of the seen state distribution yields less exploration.
Our key contributions are as follows. We propose novel paradigm for goal-directed exploration by directly optimizing goal selection to generate trajectories with high exploration value. Next, we show how learned world models permit an effective implementation of goal command planning, by adapting planning algorithms that are often used for low-level action sequence planning. We validate our approach, PEG, on challenging simulated robotics settings including a multi-legge ant robot in a maze, and a robot arm cluttered tabletop. In each environment, PEG exploration enables more efficient and effective training of generalist GCRL policies relative to baselines and ablations. Our successfully navigates a long maze, and the robot successfully builds stack of three blocks.
PROBLEM SETUP AND BACKGROUND
We wish to build agents that efficiently explore environment to autonomously acquire diverse environment-relesn capabilities. Specifically, in our problem setting, the agent is dropped into an unknown environment with no specification of the tasks that might be of interest afterwards. Over episodes of unsupervised exploration, must learn about its environment, the various "tasks" that affords the agent, and also how to perform those tasks effectively. We focus on goal state-reaching tasks. After this exploration, a successful agent would be able to reach diverse previously unknown goal states in the environment upon command.
To achieve this, our method, planning exploratory goals (PEG), focuses on improving unsupervised exploration within the condiiiniii reinforcement learning framework. We set up notation and preliminaries below, before discussing our approach in section 3
Preliminaries. We formalize the unsupervised exploration stage within a goal-conditioned Markov decision process, defined by the tuple (S, A, ().). The unsupervised goal-conditioned MDP does not contain the test-time goal distribution, nor reward function. At each time a goal-conditioned policy π (- Sc. g) in the current state 5, E S, under goal command g E G, selects an action a; E A, and transitions to the next state 81+1 with probability T(s--1 st, dz). To enable handling of the most broadly expressive goal-reaching tasks, we set the goal space to be identical to the state space: G = S, ie., every state maps plausible goal-reaching command.
In our setting, an agent must learn a noal-conditioned reinforcement learning (GCRL) policy G 8, g) well collect exploratory data train this policy on. Thus, motivated in section 1
2

---

Published as a conference paper at ICLR 2023
is critical to pick good goal commands during training because doing generates useful data to improve the policy's 's ability to achieve diverse goals.
Critical to method's design is the specific mechanism through which goal commands influence
the exploration data. We follow the recently popular "Go-explore" procedure (Ecoffet et al. 2021
Pistar et al. 2022; Tuyls al. 2022) for exploration-intensivee long-term GCRL settings. Every training episode starts with the selection of goal command g. Then, a goal-directed "Go-phase"
starts, where the GCRL policy TC is executed, conditioned on g. producing a final state sT after T steps. Immediately, an "Explore-phase" commences, with an undirected exploration policy x- (:1 taking over for the remaining Tg timesteps. For example, (- could be an RL policy trained to maximize an intrinsic exploration reward r (Bellemare et al. 2016 Pathak et al. 2017 Burda
et al. 2019 Sekar al., 2020). See [Appeddix D.I for detailed pseudocode. This structure of training episodes has been shown to result in richer exploration (Pislar et al.) (022). Intuitively,
the goal-conditioned policy efficiently brings an agent to an interesting goal, and the undirected exploration policy efficiently explores around the goal. However, Go-explore prescribes general way to select goals that induce exploration.
In this paper, we propose PEG a general goal command selection objective for Go-Explore-style un- supervised exploration. PEG uses a learned world model, making it natural to implement it alongside model-based RL approaches that already train such model. We implement PEG exploration for "latent explorer achiever" (LEXX) (Mendonca et al. (02T). goal-conditioned MBRL. framework. We briefly summarize the LEXA framework below and refer the reader to Appendix D for details. LEXA learns the following components:
World model:
a-11)
Exploration policy: T (a(())
Goal Reaching policy:
(
(1)
Exploration value:
(s))
Goal Reaching value: Ve (8t,
The world model is trained as a variational recurrent state space model following Hafner al. (2020). The explorer and goal reaching policies are trained with the model-
Algorithm 1 LEXA Training Loop
Input: x R world model T. rewards y f
2:
D (} Initialize buffer.
based on-policy actor-critic Dreamer algorithm (Hafner)
3:
for Episode i 1 to Nu do
et al. (.2T). Both policies are trained purely in imagina- tion through world model "rollouts", i.e., the world model acts as learned simulator in which to run entire training episodes for training these policies. The explorer reward
4:
5:
D- UT
Update model 7 with (st, (1.1)
Update = in imagination with 7 maximize G
Update = in imagination with 7 maximize
is the Plan2Explore (Sekar et al.] (020) disagreement objective, which incentivizes reaching states that cause an ensemble of world models to disagree amongst themselves. The goal reaching reward G is the self-supervised temporal distance objective (Mendonca et al. 2021). which rewards the policy for minimizing the number of actions between the current state and a goal state.
See Algorithm I for high level pseudocode of the training process. Highlighted in red is the goal-directed exploration strategy Algorithm 2p. LEXA explores by sampling ran- dom goals to reach or by running the exploration policy. As will see in the next section, PEG serves drop-in replacement for LEXA's random goal sampling by sam- pling goals optimized for Go-explore exploration.
Algorithm LEXA Goal Sampling
I:
function if episode i % = 0 then
2:
Sample random goal g from buffer.
4:
Collect trajectory with ( (-1)))
else
Collect trajectory with x
7: return
PLANNING GOALS FOR EXPLORATION
Having set up the preliminaries, we return to the core question of this paper: how should we pick exploration-inducin goals to help acquire diverse skills? We have argued above that the answer is not to pick goals the frontier of previously explored states, in prior work (Yamauchi) 1998 Pitis ctt al., 2020; Bhhaaddwajj et al. 202T). Instead, at the beginning of each training episode, we propose to optimize directly for goals that would lead to the highest exploration rewards during the episode's
3

---

Published as a conference paper at ICLR 2023
Explore-phase
max E pp
[V
g
T+TE
where
(ST) Egg
Σ l==
E
(3)
and where (x the distribution of terminal states generated when T. runs for T steps conditioned on goal g. V which we call exploration value, is the expected discounted intrinsic motivation return of the exploration policy, when launched from state 8T- Please refer to (ogiieeee 2 for a detailed visualization.
(-1))))
V (8y)
V ($,, Dirr
F [Er
Ite
1/2
1/2
Parett do (4)
( (vg)))
Strrent-tttte
Po do (Ry)
(11,))
1. Sample goals f
2. Predicxx ( 1-.0) trajectories with world model m(sel ((='(s.))
Choose g" by computing expected exploration for each goal.
4, Explore in real world by runningx" (lla'" an
Figure 2: PEG proposes goals for Go-explore style exploration. 1) We start by sampling goal candidates 91 -1..-9N. .N. 2) For each goal 9, execute the goal-conditioned policy n in the world model 7 generate trajectories. 3) We score each goal by seeing how useful their trajectories are for launching an exploration policy. 4) Explore using the best g.
Importantly, this objective involves computing the expected exploration value of terminal states 8T- The exploration value measures how interesting will the Explore-phase be as measured by intrinsic motivation rewards. Next, note that the expectation in |Equation (2) over the terminal states induced by the current GCRL policy. This way, it accounts for the capabilities of the current GCRL policy, regardless of whether is novice or an expert achieving the selected goal g. If we had instead sought to optimize for goals with high exploratory value, in similar spirit to prior works (Pong et al. 2019 Pitis et al.) 2020 Ecoffet et al. 202T). the objective of Eqquatin (2) would instead be replaced by V²)g), the exploration value of the goal, which is naively agnostic to the capabilities of the GCRL policy. As we have argued above, this would then require additional mechanisms to account for reachability of the specified goal etc.. bringing
hyperparameters and related overheads. Instead, PEG offers a unified and compact solution. In plain English, "first command goals that lead the goal policy states that have high future exploration potential, then explore."
The objective in Equation (2) is not straightforward to compute, because relies on the terminal state distribution induced by the goal-conditioned policy ( (sz). This distribution can change rapidly throughout training, as the policy evolves. To explore well, it is important to use an up-to-date estimate. We leverage the learned world model 7 to achieve this: P. (-1-a) (T) can be represented as the product of the transition and goal-conditionel policy distributions.
T II
(T
p(su)
(ss
18-1, #T (a.-1
9 81-1,9)
Now, may approximate the expectation over the marginal ((haa) (ST) in the objective Equa tion (2)by by sampling trajectories from the learned world model and using the last state 8y
Eg (sr)
(sr) [V (sT))
1 K
Σ
where o ~ (
(5)
This permits on fn-the-fly generation an up-to-date estimate of our objective from Equation (2)
using a recent goal-conditioned RL policy. Then, optimize goals g using sample-based
4

---

Published as a conference paper at ICLR 2023
optimization. Following Equation (5) to evaluate goal 9, condition the goal-conditioned policy on g. and roll it out for K trajectories ти within the learned world model. We then estimate the terminal state exploration value for each trajectory with the learned exploration value function VP( (Equation (1), and average the estimates.
For optimizing the goal variable g. we model predictive path integral control (Williams et al. 2015)
(MPPI, see (eppeddixx E] details). In inpppendix E] we show that the specific choice of optimizer
not critical and other optimizers like cross entropy method (CEM) (De Boer al. 2005) also
work well. More broadly, while these sampling- sampling-based based optimizers are popular for optimizing action
trajectories with dynamics models (Nagabandi et al. 2019 Chua et al. 2018 Ebert al.
2018 Hafner
et al. (020), PEG uses them instead for optimizing goal commands; this lower-dimensional search
space enables easy optimization, independent of the task horizon, permitting handling long-term tasks.
Equipped with this goal setting procedure, we can now train a goal-conditioned policy by replacing the LEXA goal sampling highlighted in red in Algorithm I with our proposed PEG goals and Go-explore episodes. AP
Algorithm PEG Goal Sampling
I: function Optimize[quation
DODOEEEEEEEXLLORTTIOOO g f Optimize
2:
Equation (5) with MPPI 7 GO-EXPLORE(g.
3:
4: return
gorithm 3 shows PEG pseudocode.
Implementation Details. We implement PEG on top of authors' code from Dreamer (Hafner et al. (22I)) LEXA (Mendonca et al. 2021), and MPPI implementation from Rybkin al. (2021). We used the default hyperparameters for training the world model, policies, value functions, and temporal reward functions. For PEG, we tried various values of K for simulating trajectories of x for each goal and found K = to be sufficient. We use the same Go-explore mechanism across all goal-setting methods: the Go and Explore phases' time limits set to half of the maximum episode length for all environments, while non-Go-explore baselines use the full episode length for exploration. See See/Appendix B for details on hyperparameters, runtime, and resource usage.
4 EXPERIMENTS
Our experiments evaluate goal-reaching tasks that require non-trivial exploration and skill acquisition solve. We aim to answer: (1) Does PEG result in better exploration and downstream goal-reaching behavior? (2) What qualitatively distinguishes PEG goals and exploration from previous goal-directed exploration methods? (3) What components of PEG contribute its success?
4.1 SETUP
We evaluate PEG and other goal-conditioned RL agents four different continuous-contru environ- ments, described below. For each environment, we define an evaluation set of goals - as a general principle, we pick evaluation goals in each environment that require extensive exploration in order for the agent to learn a successful evaluation goal reaching policy. These goals are very relevant in long-horizon tasks, as long-horizon goals tend require extensive exploration to gather the necessary data for training. Recall that we assume the fully unsupervised exploration setting; the agent does not have access to the evaluation set during training time. So, for all the agent knows at training time, any environment state could plausibly be a test goal. During evaluation, we run 10 goal-reaching episodes to account for environmental noise!" for each goal, and report the mean success rate and standard error for each environment. All methods are run with 10 seeds. The success metric d(s,, g) used for evaluation is an indicator function checking if any state in the episode is within e distance of the goal state. Figure 3 for images of each environment, and Appendix Cf more details.
Point Maze: The agent, a 2D point, is noisily initialized in the bottom left. At test time, we evaluate it reaching the right corner. The 2-D state and action space correspond to the planar position and velocity respectively. The episode length is 50 steps. The top right corner is the hardest to reach, so we set this to be the test-time goal.
In this environment from Tassa et al. 2018). a 2D 2-legged robot moves on a flat plane. The state space is 9-D and contains the walker's 's XZ position and joint angles. The action space is 6-D and represents joint torques. The episode length is 150 steps. To evaluate the ability to learn
Walker:
Our environments have deterministic transitions with noise in the initial state distribution.
5

---

Published as a conference paper at ICLR 2023
Point Maze
Walker
Ant Maze
3-Block Stack
Figure 3: We evaluate PEG across four diverse environments, selecting evaluation tasks to stress-test exploration. From left right: Point Maze requires the agent to move from the bottom left to top right. Walker evaluates if the agent can walk to different positions on the plane. In Ant Maze, the ant required move to the farthest end of the maze. Finally, in 3-Block Stack, the robot must stack the blocks into the goal configuration (colored orbs).
different locomotion behaviors of varied difficulty, the test goal commands are standing states that placed +7, and 12 meters from the initial pose.
Ant Maze: We increased exploration difficulty in the Ant Maze environment from MEGA (Pitis al. 2020) by placing one additional turn in the maze. Further, set the GCRL goal space the full 29-D ant state space, rather than reduce it to 2-D xy position as in MEGA. The ant robot must learn complex 4-legged locomotion behavior and navigate around the hallways (see Figure 3). The episode length is 500 steps. We evaluate the ant on reaching 4 hard goals that are placed in the top left room, the furthest area from the start, and intermediate goals in the middle hallway.
3-Block Stacking: Here, robot arm with two- wvo-fineeeed gripper operates on a tabletop with
blocks and boundary walls. The state space is the gripper pose and object zyz positions (14-D), and the action space is the gripper velocity and force (4-D). At evaluation time, the robot is commanded to stack three blocks into a tower configuration. To achieve this, the agent will need to have learned pushing, picking, and stacking, and discovered 3-block stacks environment. The episode length is 150 timesteps. Note that 3+oblock stacking is a classic and open exploration challenge in robot RL; to our knowledge, all prior solutions assume expensive forms supervision or computational resources to overcome the exploration challenge, such as demonstrations, curriculum learning, billions of simulator samples (Ecoffet et al. 2021 Nair et al.
possible configuration of the
2018a Lanier et al. 2019 Li et al. 2020), highlighting task difficulty.
4.2 BASELINES
PEG aims improve goal-directed exploration by improving goal selection during GCRL. To
specifically evaluate exploration, we implemented all baselines on top of LEXA (Mendonca et al.
2021). the backbone conditondt MBRL framework by replacing the LEXA exploration logic (line 4 in Algorithm 1) with each method's proposed exploration strategy. See Appendix D for detailed comparison between baselines.
Our first two baselines, Skewfit and MEGA, are alternative goal setting approaches: both follow Algorithm and replace the LEXA exploration with their own goal sampling strategy for Go-Explore style exploration (see Algorithm 7 and Algorithm 8 for pseudocode). Skewfit (Pong et al.) 2019) estimates state densities, and chooses goals from the replay buffer inversely proportional to their density, resulting in uniform state sampling. MEGA (Pitis et al. (020) similarly uses kernel density estimates (KDE) of state densities, and chooses low density goals from the replay buffer. MEGA is close to the original Go-explore approach (Ecoffet et al. 202T); rather than Go-explore's hand- designed pseudocount metric, MEGA uses KDE to choose goals. MEGA relies on a heuristic Q-function threshold to prune unachievable low density goals since many rare states may not be achievable by π This requires tuning and prior knowledge of the test goal to set.
We also run vanilla LEXA itself baseline. LEXA does not perform Go-explore style exploration, instead using an exploration policy for some exploration episodes and the goal-conditioned policy conditioned random goals for others. In addition, we evaluate against variant of LEXA that only collects data with the exploration policy. We denote this bascline Plan2Explore (P2E) after the intrinsic motivation policy it uses (Sekar al. 2020). P2E trains exploration policy maximize

---

Published as a conference paper at ICLR 2023
disagreement among an ensemble of world models. Notably, PEG and our other Go-Explore baselines all use the same P2E exploration algorithm during explore-phasee which isolates the differences in exploration to the proposed goals of each method. See Seeenppeddix Door additional details including
a side-by-side algorithm comparison, and where we verity that our MBRL. versions of MEGA and Skewfit are sound and in fact better than the original model-free versions in Pitis et al. 2020
1.00
PEG SEeVIT
MEGA
0.9
0,8
Snewii:
PEG
0.5
Snowfit
PEG
0.75
FZL
0.75
LEXA
0.6
P2E
0.4
5000055
SUCCE55
0.50
0.25
0.00
165
1.0
2,2
stape
4.0
1e5
0
0.2
steps
1.0
8800
0.2
0.4
steps
0.0
Point Maze
Walker
Ant Maze
3-Block Stack
Figure 4: Performance of agents with different goal setting strategies. All methods are run with 10
seeds. PEG outperforms all baselines, and its performance gain increases with environment difficulty.
4.3 RESULTS
Test task success rates. Figure 4] shows the evaluation performance across training time. PEG com- pares favorably to baselines both in final success rate and learning speed. PEG reaches near-optimal behavior earlier than any other baseline. PEG outperforms goal-setting baselines (MEGA and Skewfit) across all tasks, which shows that PEG goal setting is superior alternative goal-setting strategies. On the hardest task, block stacking, PEG is the only method to achieve any significant non-zero task performance: PEG achieves about 30% success rate on this challenging exploration task, all other baselines are close to 0%.
Recall that MEGA picks minimum density goals from the replay buffer. As have seen in Figure picking goals from the replay buffer may result in suboptimal goals for exploration as there could exist goals outside the data that result in more exploration. Furthermore, the MEGA objective does not reason about temporally extended exploration that begins after the goal state is reached. Skewfit, which samples goal states uniformly by their density, may often sample uninteresting goals, thus slowing down exploration. In our upcoming qualitative analysis of goal selection behavior, these conceptual differences manifest in suboptimal goal picking in comparison to PEG
Next, P2E has Go-phase, and thus goal-directed exploration, which limits its exploration horizon. Even so, it can sometimes perform better than MEGA and Skewfit (see Walker), both methods that employ the same exploration algorithm as P2E. just after Go-phase. In other words, Go-explore by itself doesn't 't always improve exploration, and suboptimal goal-setting during "Go- phase" can in fact deteriorate exploration. LEXA performs worse than simple P2E, indicating that sampling random goals from the replay buffer does not improve exploration or training.
Goal selection behaviors. Next, visualizes the goal-picking and exploration behaviors for PEG and baselines. A trend across tasks is that PEG consistently picks goals (red points) beyond the frontier of seen data. In Point Maze, the middle right region forms natural barrier for the exploration policy to overcome. The policy must find a rather tortuous path from the bottom left through the middle right hallways in order reach the top right corner. As a result, top right states beyond the middle right region are very sparsely represented. With similar states in the replay buffer (blue dots), we find that PEG generates many more goals in the top right region than MEGA, thus accelcrating exploration. Similarly, in Walker, PEG selects goals to the far left and right of the walker, where data support is the sparsest. MEGA fails to pick goals to the left.
In Ant Maze (Figure 5), we that PEG explores the most deeply into the maze of all methods, at this halfway point. Counting the total number of state visitations in the farthest lep- reaches of the maze for each method, only has ~3K. Looking at Figure 5 MEGA does pick a few "rare" goals in the top left, but T0 has not visited the top left area, either because of is not capable of reaching the rare goals and/or the
find that PEG has ~68K states whereas MEGA, the closest baseline,
7

---

Published as a conference paper at ICLR 2023
Point Maze
Walker
MEGA
PEG
MEGA
PEG
PFG
MFGA
Skewfit
P2F
Ant Maze
L
Figure 5: We plot the achieved states (blue green dots) and goals (red dots) for PEG and baselines in the Point Maze (top left), Walker (top right), and Ant Maze (bottom) halfway through training. PEG picks goals in areas (orange boxes) that lead to more exploration whereas baselines do not. Walker and Ant states are high dimensional
just plot their central zy positions.
20%
10%
60%
80%
100%
Figure 6: The evolution of achieved states (colored dots) and goals (red dots) of PEG throughout training in the Ant Maze. Lighter states are more recent. PEG samples goals to maximize exploration, sometimes picking goals that outside the seen data physically implausible.
frontier. Skewfit samples no goals in the top left region, because it does not prioritize rare states. We further examine the training evolution of PEG goals in Ant Maze to analyze how they contribute to exploration. We plot the achieved states and goals proposed by PEG periodically throughout its training in Figure PEG goals, optimized to induce the coal-conditioned policy to visit high- exploration value states, are not always predictable. PEG commonly chooses goals outside of the data distribution. For example, PEG chooses goals near the top wall for the entire training duration. We can also see it choosing physically impossible goals in the middle wall at 20%, but these goals disappear as the goal-conditioned policy and world model updated
time.
In the 3-Block Stack environment, as discussed above, PEG is the only method to train any meaning- ful policy for 3-block stacking. The space of configurations of the three blocks and the robot in this setting is extremely large. While all methods are able to explore and discover simpler skills such as picking and 2-block stacking, only PEG's exploration comprehensive enough also explore more "useful" and hard-to-reach behaviors such as 3-block stacking. Sec/Appendis F.4 for success curves on intermediate goals like picking and 2-block stacking, and the website for videos.
Ablation Analysis. What components of PEG are important to its performance? We run ab- lation studies in Ant Maze. First, ablate PEG's goal sampling: Rand-Goal PEG replaces PEG's goal sampling with randomly sampled goals. In Seen-Goal PEG, we forego optimiz- ing the goal with MPPI and simply choose goals from the replay buffer that score highly ac- cording to |Equation (5) Go-Value PEG chooses to optimize the exploration rewards achieved during the imagined Go-phase instcad of during the Explore-phase, by replacing y F in Equation (5) with ΣT
note that this still runs Explore after Go in the environment.
htttss://iiten google..oooooooooooooooooooo.. wwwp/viee/expllrrtrrryy.ooo
8

---

Published as a conference paper at ICLR 2023
0.9
Next, ablate PEG's Go-explore mechanism itself, by replacing the explorer policy with a random action pol- icy (Rand-Explore PEG). and removing Explore-phase completely (No-Explore PEG).
0.8
Band-Cool PEG Gonvalue FEG
FEG
success
No Explore PEG
Explone
0.4
Figure 7 shows the results. PEG is clearly better than all ablations, suggesting that all components of the method contribute to performance. Explore-phase is particularly important; No-Explore PEG achieves 0% success, Rand- Explore PEG performs poorly, showing the need for learned Explore-phee policy like P2E. Next, Kand-Goal PEG, Seen-Goal PEG, and Value PEG all perform worse than PEG; the specifies of the objective function for goal sampling defined in Equation (5) and its optimization procedure, are important.
0.8
8.0
0.2
0.4
0.6
0.8
1.0
steps
1e6
Figure 7: PEG ablations.
5 OTHER RELATED WORK
PEG is most closely related to the subfield of unsupervised exploration in RL, where tasks are pre-specified during training. In this setting. common approach to define an intrinsic motivation reward correlated with exploration for RL (Schmidhuber 2010; Pathak al.) 2017). Common forms of intrinsic reward incentivize visiting rare states with count- methods (Poupart et al. 2006 Bellemare al. 2016 Burda et al. 2019) finding states that maximize prediction error (Oudeyer et al. 2007] Pathak et al. 2017]] Henafi [219 Shyam et al. 2019) Sekar et al. 2020). Within PEG,
leverage intrinsic motivation policy for the Hxplore-phase.
More specifically related to PEG, goal-directed exploration a special class of unsupervised explo- ration, applicable in the challenging goal-conditioned RL (GCRL) setting. To develop generalist GCRL policies, these methods command the goal-conditioned policy towards exploratory goals at training time. Broadly, the idea is to pick goals that are difficult for the current policy, but com- pletely out of reach, i.e., within the "zone of proximal development." Within that broad framework, however, prior works propose many different metries for goal choosing, such learning progress (Baranes & Oudeyer 2013] Veeriah et al. 2018), goal difficulty (Florensa et al. 2018). "sibling rivalry" (Trott al. | 2019). or value function disagreement (Zhang al. (020)). Within this family, we have already discussed and compared against the most closely relevant methods MEGA and Skew-Fit (Pong et al. 2019 Pitis et al., 2020) (see Section 4.2 and /Section 4.3) which command the agent to rarely seen states increase the number of seen and familiar states, reasoning that this will enable broader expertise in the trained GCRL policies.
Closely related unsupervised goal-directed exploration is unsupervised skill discovery, which aims to train a skill-conditioned policy Eysenbach et al. (2019); [Sharma et al. (2019). Skill discovery objectives do not necessarily encourage exploration, and can even fall into local minima where the agent is penalized for exploration (Strouse et al. 2021 Campos et al. 2020). As such, these are usually weak baselines for exploration; for example, and MEGA, both report clearly favorable results against such methods (Eysenbach et al. 2019 Hartikainen al. 2020). See Appendix for an extended discussion of related work.
goal-directed exploration baselines, LEXA
CONCLUSIONS
We have presented PEG, approach that sets goals achieve unsupervised exploration in goal- conditioned reinforcement learning agents. PEG plans these exploratory goals through a world model. directly optimizing for the estimated exploration value of the training trajectory. While PEG performs better in our experiments than prior approaches across various environments, exploration for hong-horizon tasks remains challenging, and PEG's explicit dependence on world model rollouts may be a particular weakness in such settings. Likewise, complex environments may permit extremely large goal spaces, where purely unsupervised exploration approaches like PEG are unlikely to succeed within reasonable time, and some information about the target task space may be required. We believe that the PEG principle optimizing for goals predicted to induce highly exploratory policy behaviors will prove useful towards future efforts overcome these longstanding exploration challenges.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
This work was supported part by ONR grant number N00014-22-1-2677 and gift funding from NEC Laboratories. The authors would like to thank the anonymous reviewers for their constructive feedback, as well as Yecheng Jason Ma and the other members of the Perception, Action, and Learning (PAL) research group UPenn for useful discussions and general support.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, will release the codebase that contains our method, baselines, and environments. The supplementary contains details about hyperparameters, baselines, and architecture (see Appendixx D and Appendix E In addition, we have also provided pseudocode in il/ooormhhmm
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Adrien Baranes and Pierre-Yver Oudeyer. Active learning of inverse models with intrinsi- cally motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-77) 2013. ISSN 0921-8890. doi:
008. URL https
www. www.scccineeiire...o com
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Pro- cessing Systems, volume 29, 2016. URL htt.::ttp////ttcc.tt /tt//tttsp/pprrce.nggg neurips cc/ happppr// F431 6a66 Paa
Homanga Bharadhwaj. Animesh Garg, and Florian Shkurti. LEAF: latent exploration along the frontier. In IEEE International Conference on Robotics and Automation, ICRA 2021, Xi'an, China, May 30 Jume 5, 2021, pp. 677-684, IEEE, 2021. doi: 10.1 0BL1109/CCR4483302222..9596
URL
2021. https: org/ wtt.:////.....0....... 11t.:111111ICCR. 1tt.
9560922 20219
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https?
//openreview. net/
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi
Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Interna- tional Conference on Machine Learning, Proceedings of Machine Learning Research, 2020. URL
http://prrcceedingg mlr. wwwps//eeevvvee. wttesss/vvvv/// wttpp////////cccaaa/a////// html
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, pp. 1430-1440, PMLR, 2021.
Richard Y. Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB EXPLORATION VIA g-ENSEMBLES, 2019, URLhttpp::/opeeeeviiies
Rb
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 3759-47700 Red Hook, NY, USA, 2018, Curran Associates Inc.
Irene Cogliati Dezza, Angela J Yu, Axel Clecremans, and William Alexander. Learning the value of information and reward over time when solving exploration-cxploittiii problems. Scientific reports, 7611: 2017.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Rubinstein. tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
10

---

Published as a conference paper at ICLR 2023
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Mosel-based deep reinforcement learning for vision-bac based robotic control. arXiv preprint arXiv:1812.0003 2018.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return then explore. Nature, 590 7847:580-586, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019, URL httpppss//ppeeeeeeee..eeee net.:wnee////eem (ttp://oooummm--S063333eeeee
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. International Conference on Machine Learning. 2018.
Yijie Guo, Jongwook Choi, Marcin Moczulski, Shengyu Feng, Samy Bengio, Mohammad Norouzi, and Honglak Lee. Memory based trajectory policies for learning from sparse rewards. In Advances Neural Information Processing Systems, 2020.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020.
URL httpps://peereviiiie
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with
discrete world models. In International Conference Learning Representations, 2021. URL
net forum? https://openreviit wttp://wwwwee/oooeeeeeeeee/me forrmmidd00aabby2/00
Danijar Hafner, Kuang-Huel Lee, lan Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels, 2022. URL http:: http:///xxvvvv.....aa.. abs http:/aab/2222..... 04114
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance Iearn- ing for semi-supervised and unsupervised skill discovery. In International Conference on Learning Representations, 2020. URL /ttptppp:///penevviii wtw.eeerreiiew.eeeefrrrr..roo 1eeiiii=i1llhaaaammm 1mhaVtvr
Mikael Henaff. Explicit explore-exploi algorithms in continuous state spaces. In Advances in Neural Information Processing Systems, 2019. URL httpss::/pprccceeiinnge neur ps
C0/
Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine. Time-agnostic prediction: Predicting predictable video frames. International Conference on Learning Representations, 2019.
Pierre-Alexandre Kamienny, Jean Tarbouriech, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. In International Conference Learning Representations, 2022.
/penreviei
net/ ftst//forum211=255AAhii
John B. Lanier, Stephen McAleer, and Pierre Baldi. Curiosity-driven multi-critria hindsight experi- ence replay. CoRR, abs/1196.03770, 2019. /ttl//tttp///axx.oo/////// wwwp:xiwww.or.aeeeomoo http://bbb///..... 03710
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object manipu- lation using relational reinforcement learning. In IEEE International Conference on Robotics and Automation, pp. 4051 1051-4058. IEEE, 2020.
Russell Mendonea, Oleh Rybkin, Kostas Daniliidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals with world models. In Advances in Neural Information Processing Systems, 2021.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In International Conference on Learning Representations, 2023. URL htttts://opeereee
net feet/fooum?2 id= u1Acb0xb
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, Proceedings of Machine Learn ing Research, 2019. URL http: httLLitttt://oooooo http:://proceddings mlr.
html
11

---

Published as a conference paper at ICLR 2023
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcom- ing exploration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation, pp.
2018a.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 9209-9220, Red Hook, NY, USA, 2b.
Curran Associates Inc.
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. In Advances in Neural Information Processing Systems, 2018. URL httpss//prrcceeeingg.s neurips ctc cc/paper/2018/
Fi FLe20foe331223533333322222333333
lan Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized
value functions. Journal of Machine Learning Research, 20(124):1-62, 2019, URL http:
339. html
Pierre- Oudeyer, Frdric Kaplan, and Verena V. Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2):265-286, 2007. doi: 10.1 109/TTEVC22006.80077
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Long-horizon visual planning with gondicond hierarchical predictors. Proceedings of the 34th International Conference on Neural Information P Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829549
Miruna Pislar, David Szepesvari, Georg Ostrovski, Diana L Borsa, and Tom Schaul. When should agents explore? In International Conference on Learning Representations, 2022. URL https
//openreview.oo www.neeeoorodddddeeeee
4bca
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761. PMLR, 2020.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supervied reinforcement learning. arXiv preprint arXiv: 1903v09903.03396 2019.
Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete bayesian reinforcement learning. Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pp. 697-704, New York, NY, USA, 2006. Association for Computing Machinery. doi: 10.1145/1 143844, 1435444,4344.1439222. 143932. URL http:: http://ddooo./1...11111110 1145/
ISBN 1595933832,
1143844 1143932
Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Danilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. International Conference on Machine Learning, 2021.
Jürgen Schmidhuber.
Formal theory of creativity, fun, and intrinsic motivation (1990-201) 2010). Au- tonomous Mental Development, IÉEE Transactions on, 2:230 247, 10 2010. doi: 10.1109/TAMD. 2010.2056368.
Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via sulf-supervised world models. In International Conference on Machine Learning, pp. 8583-8592. PMLR, 2020.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: /90.00677 2019.
Pranav Shyam, Wojeiech Jaškowski, and Faustino Gomez. Model-based active exploration. In International conference on machine learning, pp. 5779-5788. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2023
Isaac J Sledge and José C Príncipe. Balancing exploration and exploitation in reinforcement learning using a value of information criterion. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2816.2820. IEEE, 2017.
RL Stratonovich and BA Grishanin. Value of information when estimated random variable is hidden. Izvestiya of USSR Academy of Sciences, Technical Cybernetics, 6(1):3-15, 1966.
DJ Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration, 2021. URL /
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 2018.
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. CoRR, abs/1911.01417, 2019.
URL http::/axxix org/ www./ss.... 1911. 01417
Jens Tuyls, Shunyu Yao, Sham M. Kakade, and Karthik R Narasimhan. Multi-stage episodic control for strategic exploration text games. International Conference on Learning Representations, 2022.
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning. ArXiv, 806.0900, 2018.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint 15X.v:1509.01 1499, 2015.
Brian Yamauchi. Frontier-based exploration using multiple robots. In Proceedings of the Second International Conference Autonomons Agents, AGENTS '98, pp. 47-53, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 0897919831, doi: 10. 10.1144/280765.287777 2807733
URL http:ppp///o...oo/00011111151111
Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, Peng Liu, and Zhen Wang, Exploration in deep reinforcement learning: a comprehensive survey, arXiv preprint 11Xi::2009006668, 2021.
Zhao Yang, Thomas M. Moerland, Mike Preuss, and Aske Plaat. When to go, and when to explore: The benefit of post-exploration in intrinsic motivation, 2022. URL https::/arxxv.... org/
abs/ 3bs/2203.r0 16311
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-766 (6569,999,,,,, 2020.
13

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
Extended Related Work
14
B Extended Limitations and Future Work
15
Environments
16
Point Maze
16
Walker
Ant Maze
16
4
3-Block Stack
17
Baselines
D.I
Go-Explore
17
D2
LEXA
17
Original MEGA codebase
D44
Masel-based MEGA
18
D.5
Model-based Skewfit
18
Plan2Explore
E
Implementation Details
19
E.1
MPP[
19
Runtime
20
E.3
Hyperparameters
20
F
Additional Experiments
20
F1
PEG with other exploration methods
20
F2 MPPI vs CEM for Goal Optimization
21
F3 Does conditioning on PEG goals maximize exploration?
21
More goals for 3-Block Stacking
22
A EXTENDED RELATED WORK
Exploration is very broad field, and encourage readers read surveys such Yang et al. (2021). Below, we summarize various subfields of exploration and reinforcement learning relevant to PEG.
Task-directed exploration. PEG reasons about the intrinsic motivation / exploration rewards be gained by commanding goal-conditioned task policy to various goals in an unsupervised RL setting, Related high level, supervised exploration approaches reason about task reward gains during exploration. (Chen et al.) 2019 Osband et al., (209)) leverage uncertainty estimates about the reward to prioritize states for task dask-directed exploration. (Stratonovich & Grishanin, 1966) proposes "value perspective. The Vol describes the improvement in supervised task rewards the current optimal action policy operating under specified limit state information. An RL agent may modulate in supervised RL settings, by tuning this information limit: a higher limit leads to lower exploration and greater exploitation (Sledge & Principe 2017 Cogliati Dezza al. 2017)
of information" (Vol) to measure exploration from information-theoree
14

---

Published as a conference paper at ICLR 2023
Unsupervised Exploration. In unsupervised exploration, information about the task (e.g. re- ward function or demonstrations) is known during training time. In this setting, a common approach is to define an intrinsic motivation reward correlated with exploration for RL (Schmidhuber 2010 Pathak et al.] 2017). Common forms of intrinsic reward incentivize visiting rare states with count- based methods Poupart al. (2006); [Bellemare al. (2016); [Burda et al. (2019) or finding states that maximize prediction error (Oudeyer et al. (207 Pathak et al., 2017] Henaff, [2019] SSyym et al. 2011
Sekar et al. 2020). Within PEG, we leverage an intrinsic motivation policy for the Explore-phase.
Goal-conditioned RL. Goal-conditioned RL (GCRL) extends RL to the multitask setting where the policy is conditioned on a goal and is expected achieve it. The goal space is commonly chosen to be the state space, although other options like desired returns or language inputs are possible. GCRL is challenging since the policy required to learn the correct behavior for achieving each goal. This introduces two challenges- thallenges-oppimiznng goal-conditioned policy, and getting adequate data to support the optimization process.
Many GCRL methods focus on improving goal-conditioned policy optimization. Hindsight Experi- ence Replay Andrychowiez et al. (2017) boosts training efficiency by relabeling failure trajectories successful trajectories, and Chane-hSan Sane et al. (2021) uses imagined subgoals guide the policy search process. However, such methods are only useful if the data distribution is diverse enough to cover the space of desired behaviors and goals, and will still suffer in hard exploration environments. Therefore, GCRL is still predominantly constrained by exploration.
Goal-directed Exploration. Goal-directed exploration, which sets exploratory goals for the goal- conditioned policy to achieve, is often the exploration mechanism of choice for GCRL, as it naturally reuses the goal-conditioned policy. It is important to note that GDE is not inherently tied GCRL, and can be used in conventional RL to solve hard exploration tasks (Fcoffet et al.) 2021 Guo et al.
2020; Hafner et al. 2022). In such cases, goals are usually picked through some combination of reward and novelty terms.
Prior works propose many different metrics for goal choosing, such as frontier-based (Yamauchi
1998). learning progress (Baranes & Oudeyer 2013) Veeriah et al. 2018). goal difficulty (Florensa
ct al. 2018), "sibling rivalry" (Trott al. 2019). or value function disagreement (Zhang et al. 2020). Within this family, we have already discussed and compared against the most closely relevant methods
MEGA and Skew-Fit (Pong et al., 2019; Pitis et al. (2020) (see Section 4.2 and (Section 4.3) which command the agent to rarely seen states to increase the number of seen and familiar states, reasoning that this will enable broader expertise in the trained GCRL policies.
Go-Explore. The Go-Explore framework from (Ecoffet et al. 2021) performs goal-directed explo- ration by first rolling out the goal-conditioned policy (Go-phase) and then rolling out the exploration policy from the terminal state of the coal-conditioned phase (Explore -hasse). Go-Explore does not prescribe a general goal setting method, instead opting for a hand-cngineered novelty bonus for each task. GDE methods like MEGA (Pitis et al. (020)) use the Go-Explore mechanism further boost exploration efficiency, but do not consider Go- Explore when designing their goal-setting objective. This results in suboptimal performance MEGA goals chooses low density goals, but this may not necessarily mean they are good states to start the exploration phase from (e.g. dead-ends in a maze). PEG 's objective already takes the Go-Explore mechanism into account, choosing goals that maximize the exploration of the Explore phase and resulting in superior exploration.
B EXTENDED LIMITATIONS AND FUTURE WORK
Model-based RL vs. Model-free RL. PEG a model-based RL agent, which is more sample- efficient and computationally expensive than model-free counterparts. MBRL agents require more computation time and resources since they learn a world model. PEG trains a world model to train policies and value functions with imagined trajectories, as well as generate goals for exploration. A model-free version of PEG should be computationally and conceptually simpler, and we leave this for future work.
Better world models. PEG depends on a learned world model to plan goals, and may suffer in prediction performance and computation time for longer prediction horizons. An interesting direction
15

---

Published as a conference paper at ICLR 2023
would be to combine PEG with temporally abstracted world models (Neitz al. 2018 Jayaraman jet al. 2019; Pertsch et al., 2020) to handle such cases. Furthermore, PEG should benefit with improved world model architectures such as Transformers (Micheli et al. 2023 and thus improving world modelling offers straightforward path for extending PEG
Exploration beyond state novelty. Further, while empirically PEG produces performant goal- conditioned policies, only explores by maximizing novelty. In harder tasks it might be necessary to explicitly consider also exploring with "practice goals" that best improve the condiiondd policy.
Next, Pislar et al. (2022) suggests that generalizing the Go-Explore paradigm itself is beneficial. In Go- Go-Explore, we assume the agent first performs "Go-phase" for the first half of the episode, and then the sisplore-phase for the remaining half. However, we can allow the agent to switch between the "Go-phase" and "Explore-phar phasee multiple times in an episode, and decide the phase lengths.
Scaling PEG to harder tasks. While we focus purely unsupervised exploration, in practice some knowledge about the task is always available. For example, human may give some rewards, few demonstrations, some background knowledge about the task. Leveraging such priors to focus exploration on the space of tasks preferred by the user would make GCRL and PEG more practical, especially in real world applications like robotics.
PEG currently is only evaluated in state space, and a potential extension would be to handle image observations by optimizing goals in compact latent space. Such extension should only require minor edits in the PEG code because PEG is extended from LEXA, an image-based RI agent that already learns such latent space.
C ENVIRONMENTS
C.I POINT MAZE
The agent, a 2D point spawned on the bottom left of the maze, is evaluated on reaching the top right corner within 50 timesteps. The two-dimensional state and action space correspond to the planar position and velocity respectively. Success is based upon the L.2 distance between the position the agent and the goal being less than 0.15. The entire maze is approximately 10 10. This environment is taken directly from Pitis et al. 2020 with no modifications.
C.2 WALKER
Next, the walker environment tests the locomotion behavior of a 2D walker robot on a flat plane. The agent is evaluated on its ability to achieve four different standing poses that are placed +7, and 122 meters in the x axis away from the initial pose. For success, we check the agent's x position is within a small threshold from the goal pose's position. The state and goal space is 9 dimensional, containing the walker's rz positions and joint angles. The environment code was taken from Mendonca et al. 2021)
C.3 ANT MAZE
A high-dimensional ant robot needs to move through hallways from the bottom left to top left (see Fig. 3). This environment is long-horizon challenge due the long episode length (500 timesteps) and traversal distance. We evaluate the ant on reaching 4 hard goals at the top left room, and intermediate goals in the middle hallway. Ant Maze has the highest dimensional state and goal spaces: 29 dimensions that correspond to the ant's 's position, joint angles, and joint velocities. The first three dimensions of the state and goal space represent the ryz position. The next 12 dimensions represent the joint angles for each of the ant's four limbs. The remaining 14 dimensions represent the velocities of the ant in the x-y direction and the velocities of each of the joints. Ant Maze also has an 8 dimensional action space. The 8 dimensions correspond to the hip and ankle actuators of each limb of the ant. Success is based upon the L2 distance between the øy position of the ant and the goal being less than 1.0 meters, which is approximately the width of a cell. The dimensions of the entire maze is approximately 6 x 8 meters. This environment is a modification from the Ant Maze environment in Pitis et al. 2002 We modified the environment that the state and goal space
16

---

Published as a conference paper at ICLR 2023
includes the ant's zya positions along with the joint positions and velocities instead of just the zy positions. We also added an additional room in the top left in order to evaluate against a harder goal.
C.4 3-BLOCK STACK
A robot has to stack three blocks into tower configuration. The state and goal space for this environment is dimensional. The first five dimensions represent the gripper state and the other nine dimensions represent the zy² positions of each block. The action space is 4 dimensional where the first three dimensions represent the zyz movement of the gripper and the last dimension represents the movement of the gripper finger. Success is based upon the L2 distance between the zyz position of all of object and the goal position of the corresponding object being less than 3cm. This environment modification from the FetchStack3 environment in Pitis et al. (2020).
D BASELINES
Algorithm 4 MBRL Training
Input: goal expl. policies x², a², world model T. rewards , ,r,
Here, we first present pseudocode for all meth- ods. We start by defining generic MBRL train- ing loop in Algorithm that alternates between data collection (instantiated by specific meth- ods) and policy and world model updates. Next, we define the pseudocode for PEG and all base- lines below. To a method, simply drop in the data collection logic specified by the method into the MBRL training loop. For exploration, LEXA and P2E do Go-explore, while PEG, MEGA, and Skewfit do.
2:
+- [ Initialize buffer.
3: for Episade i = Nirsin
GooalDineetedExplorrattn
D D DUT
6: Update model 7 with (s, 8t41) ~ D
7:
Update 70 imaainntiinn with 7 to maximize D
8: Update x² in imagination with T to maximize F
D.1 GO-EXPLORE
Go-Explore (Ecoffet et al. 2021) proposes to switch between a goal-directed policy and ex- ploration policy in single episode to tackle exploration-intensive environments. Go-explore first stores states of "interest" future goal states. Then, in a new episode, Go-explore chooses goal g. and directs goal-directed pol- icy T to achieve the goal (or upon timeout TGo). Then, for the rest of the episode, an exploration policy x is used. Intuitively, this results in frontier-reaching exploration (Yamauchi] 1998) where the goal-directed policy efficiently traverses to the "frontier" of known states before doing exploratory behavior. Of particular importance to Go-Explore is how goal states are chosen -[coffet let al. (2221) does prescribe general approach for choosing goals. Instead, they choose goals by using task specifi psuedocount tables.
Algorithm Go-cxplore
function GO-EXPLORE(g, x5 env.reset() (5o)
3:
4:
for Step 1... Tep do
5:
8, 4- env.step(s (( (8L1,g9) r TU
6:
7:
for Step t = Vigger do
8:
& enw.step( (8L-1.)))))))) TU
9:
10: return r
D.2 LEXA
Algorithm LEXA Goal Sumpling
LEXA (Mendonca et al. 2021 trains a goal- conditioned policy with model-based RI. It trains two policies, an exploration policy and goal-conditioned "achiever" policy. The original
I:
function GOXI.DIRCTEEDEEPPOOATTOO...
2:
if /%2 0 then
3:
Sample random goal y from buffer. Collect trajectory with x (.9)
LEXA codebase was designed for image-based environments, so we implemented a state- based LEXA agent by modifying DreamerV2 (Hafner ct al. 1). To do so, we added the G and n policies, goal-conditioning logic, and the temporal distance reward network.
5:
else
6:
Cullect trajectory with E
return
17

---

Published as a conference paper at ICLR 2023
D.3 ORIGINAL MEGA CODEBASE
We used the original MEGA codebase to train MEGA and Skewfit agent (Pitis et al. 2020 Pong et al. 2019) for million environmental steps to solve the Pointmaze, Ant Maze, and 5-Block Stack tasks mentioned in Section 6. We used the original MEGA implementation with the same hyperparameters except for their goal-conditioned policy training scheme.
MEGA proposes Rfaab, custom goal relabeling strategy to train the goal-conditioned policy. By default Rfaab relabels transitions with test time goals to train the goal-conditioned policy, which not possible in our unsupervised exploration setup. Therefore, we changed the Rfaab hyperparameters to rfaab 0as_0_440 0_4_0_5__ which no longer grants access to test goals.
We find that both MEGA and Skewfit agent fail to scale to the harder environments (Ant Maze and 3-Block Stack) as seen in Figuuee 8) One reason could be that the MEGA codebase uses model-free RL, which is more sample inffficient. Therefore, we reimplemented MEGA and Skewfit with LEXA, the backbone MBRI agent that PEG uses as well.
MEGA
Skewfit
Point Maze
90%
0%
Ant Maze
0%
0%
3-Block Stack
0%
0%
Figure 8: Success Rate of MEGA and Skewfit Agent
D.4 MODEL- BADDED MEGA
For model-based MEGA, simply port MEGA's KDE model. Conveniently, LEXA al- ready trains goal-conditioned value function, which we use for filtering out goals by reachabil- ity. We use the same KDE hyperparameters as the original MEGA paper. As seen in in in/uue 4 our MEGA implementation gets non-trivial suc- cess rates, improving over the original MEGA bascline.
Algorithm 7 MEGA Goal Sampling
function GOALLDRECTDEEXIOORTOOO...
2: mingeD P(g)
GO-EXPLORE(g, R at)
4: return
D.5 MODEL-BASED SKEWEIT
Algorithm 8 Skewfit Goal Sampling
For Skewfit, we port over the Skewfit base- line in MEGA's codebase. The implementation straightforward and similar to model-based MEGA. Instead of picking the minimum density goal, Skewfit picks goals inversely proportional to their density. As in in Figure 4| our Skewfit implementation gets non-trine triviai success rates.
function GOA.LDRRECEDDEXPLLRRTOO,
2:
9 ((- D)
3:
T GO-EXPLORE(S, n x
return.
D.6 PLAN2EXPLORE
Algorithm 9 Plan2Explore
Our Plan2Explore baseline uses Plan2Explore exploration policy to gather data for the world model and trains goal-conditioned policy in imagination. Our LEXA implementation al- ready uses Plan2Explore, so this variant simply requires us only rollout the exploration policy in the real world.
function GOALDDEECTEDDRPPOORATOO.N Collect trajectory with ~
3: return
18

---

Published as a conference paper at ICLR 2023
E IMPLEMENTATION DETAILS
E. MPPI
MPPI is popular optimizer for planning actions with dynamics models (Williams et al., 2015 Nagabandi et al. 2019). Instead of planning sequences of actions, which grows linearly with horizon, PEG only needs to "plan" with goal variable, making the search space independent of the horizon. First, we present a high-level summary of MPPI, and refer interested readers to Nagabandi et al. (0199) for more detailed look MPPI.
The process begins by randomly sampling goal candidates from an initial distribution (see below for more details). These candidates then evaluated described in Figure 2 generate goal-conditioned trajectories for each goal 9& through the world model, and compute the expected exploration value Vi for each goal. This exploration value acts as the "score" for the goal candidate. Once we have scores for each goal candidate, Gaussian distribution is fit according to the rule:
En ae (on) Ek
14
(6)
where y is the reward weight hyperparameter. We then sample candidates from the computed Gaussian, and repeat the process for multiple iterations.
MPPI initial distribution Throughout our paper, we initialize the MPPI optimizer with goal vectors sampled uniformly at random from the full goal space. One alternative would have been to restrict the initial candidates to in the agent's replay buffer, but we found in practice that this restriction made difference and it more convenient for implementation reasons to simply sample uniformly at random from the unrestricted goal space (same as the state space), e.g. using the joint limits of the various robot joints.
3.4
3.8
PEG-Resit CIRd
MEGA Unrest ctod
LEXA
2.0
steps
Figure 9: Performance of state space sampling variants the Ant Maze. Removing state space sampling (eG--Restii)te) does not degrade PEG performance, and adding state space sampling (lEGA--nnestrictee) does not boost performance.
we show results of rumning PEG with MPPI initialization from the replay buffer ("PEG- Restricted"), ), in the Ant Maze environment. Its performance is virtually indistinguishable from the original PEG method. This is to be expected, and is sign that the optimizer is working well.
In Figure 9
Could other methods benefit from sampling outside the replay buffer? In Figure strongest baseline MEGA to also sample goals from outside the replay buffer ("MEGA-Unnestritte) However, these methods rely on heuristic objectives for selecting goals (like "low density" of the visited state distribution) which only approximate the correct goal-setting objective for states close to the replay buffer. Therefore, AMEGA-Unrestricteed is unable to sample effective goals outside the replay buffer, and fares very poorly. In contrast, PEG directly optimizes the exploration value of goal states, and permits choosing meaningful goals even in this unrestricted setting. This is an advantage, since it permits optimizing larger space of goal candidates, which leads to better goals.
we allowed our
19

---

Published as a conference paper at ICLR 2023
E.2 RUNTIME
Total Runtime (Hours)
Episodes
Episode Length
Seconds per Episode
Point Maze
16
12000
50
4.8
Walker
16
5000
150
11.52
AntMaze
48
2000
500
86.4
3-Block Stack
48
6666
150
25.9
Figure 10: Runtimes per experiment.
The runtime and resource usage between methods did not differ significantly, everything is implemented on top of the backbone LEXA model-based RI. agent. Note that model-free agents tend to run significantly faster than model-based agents, but fail explore and optimize good policies. Sec (eepepeddix D where we found model-free variants of MEGA and Skewfit failed to learn at all. Runtime is dominated by neural network updates of the policies and world model, not the goal selection routine defined by each method. Each seed was run on GPU (Nvidia 2080ti or Nvidia 3090) and 4 CPUs, and required 11GB of GPU memory. See Figure for training time.
Seconds Episode
PEG:
0.51
MEGA:
0.48
SkewFit: 0.46
Figure 11: Runtime for goal selection.
Next, we benchmarked the goal selection procedure for PEG and the other goal selection baselines in the block stacking environment and recorded the average wall clock time in Figure II
We can see that there is little difference in speed between methods, and the overhead introduced by PEG goal selection is minimal to LEXA, and competitive with other goal selection baselines. Because LEXA does select goals, finishes up an hour earlier than goal setting methods in our experiments. As sidenote, the times reported above are amortized over episodes, since we compute a 50 goals every episodes for all methods, rather than computing goal per episode.
E.3 HYPERPARAMETERS
For parameter tuning, we used the default hyperparameters of the LEXA backbone MBRL agent (e.g. learning rate, MLP size, etc.) and kept them constant across all baselines. PEG only has I hyperparameter that requires tuning the reward weight, which affects the MPPI update distribution. The higher the weight, the more we update the goal sampling distribution towards high-reward trajectories. For each experiment, we tried weight values of (1, 2, 10) by running 1-2 seeds of PEG for each value. We used weight of 2, 2. 10 for the 4 experiments respectively PEG MPPI, sample-based optimizer, to optimize the objective. While MPPI has hyperparameters such as the number of samples and number of iteration rounds, these are easy tune, since optimizer performance increases with the amount of samples and iterations. We therefore just choose as many samples (2000 candidates) and rounds (5 optimization rounds) as we can while keeping training time reasonable.
F ADDITIONAL EXPERIMENTS
F.1 PEG WITH OTHER EXPLORATION METHODS
While chose to train exploration policy and value function with Plan2Explore, PEG can work with any exploration method as long as it provides an exploration reward and policy. Random Network Distillation (RND) is another powerful exploration method that rewards discovering new
20

---

Published as a conference paper at ICLR 2023
0.2
360
Figure 12: P2E vs. RND
states. We run both variants with 5 seeds in the Ant environment FineeennnFFiguree Figure 12 We find that the RND variant still explores the environment, albeit not well as P2E. We hypothesize P2E exploration is particularly synergistic with our model- based RL framework, since the P2E explorer explicitly seeks transitions (s,a, a, s') that improve the model. As the model accuracy increases, does the quality of the goals generated by PEG.
F.2
MPPI VS CEM FOR GOAL OPTIMIZATION
Stack
0.6
D0gE-com com
0.1
0.2
0.0
0.0
0.4
0.5
0.8
18655
:45
Figure 13: MPPI vs. CEM
We investigate the impact of the goal optimizer on PEG We choose two popular optimizers, MPPI and CEM, optimize objective, and run 5 seeds of each variant in the stacking environment. As seen Fig. [3] the two agents perform similarly, showing that the PEG objective is robust to the choice of optimizer.
F.3 DOES CONDITIONING ON PEG GOALS MAXIMIZE EXPLORATION?
PEG
Frontier
Figure 14: PEG exploration in a U-maze. Brown background dots: explored states, commanded goals: *, resulting paths: colored lines. (Left) PEG optimizes directly for exploration, even setting unseen goals, and achieving farther paths. (Right) Setting goals at the frontier of the seen state distribution yields less exploration.
We assess the importance accounting for the goal-conditioned policy's 's capabilities. To do so, we evaluate if states coming from policy conditioned on PEG goals more novel than states coming
21

---

Published as a conference paper at ICLR 2023
from the same policy conditioned other goal strategies. In Figure 14 take partially trained
agent and its replay buffer, and plot the proposed goals and resuling trajectories for PEG and MEGA (called Frontier in the figure). We find that PEG goals lead the ant to the top left more often. This figure is also used in the intro for exposition.
F.4
MORE GOALS FOR 3-BLOCK STACKING
Easy Goals
Medium Goals
Hard Goals
100
100
60
PEG MEGA Skewfit
PEG MEGA Skewfit
PEG
MEGA
80
80
Skewfit LEXA P2E
LEXA
LEXA P2E
P2E
60
60
success
40
40
20
8.0
0.2
0.4 0.6
1.0
8.0
0.4
0.6
1.0
8.0
0.2
0.6
0.8
1.0
steps
1e6
steps
1e6
steps
1e6
Figure 15: Performance of agents evaluated different types of goals in the 3-Block environment.Ar methods are run with 10 seeds. Easy goals require picking up one block, medium goals require stacking blocks, and hard goals require stacking all blocks.
We have defined three types of goals of increasing difficulty picking up a single block ("Easy"), stacking blocks ("Medium")), and stacking three blocks ("Hard")). For each goal rype, set specific evaluation goals by varying the location of the pick / stack as well as the choice and ordering of the blocks. In this way, we create distinct Easy goals, Medium goals, and Hard goals.
First, PEG remains clearly the strongest at 3-block stacking (Hard) as seen in Figure T5 The difficulty -based task grouping permits more interesting analysis: PEG performs competitively with the best approaches on the Easy goals, and its gains over baselines are larger for Medium and Hard goals where exploration is most required. On these harder categories of tasks, PEG is both quickest to achieve non-trivial performance (indicating the onset of exploration) and also achieves the highest peak performance.
An interesting side note here is that better exploration tends to create greater challenges for multi-task goal-conditioned policy function approximation with fixed capacity policy network. Around the 0.5M steps mark, when PEG starts to achieve Hard goals, it grows marginally weaker Easy and Medium goals. This kind of "forgetting" is well-studied in the continual learning literature; addressing this well may require expanding policy network capacity time, in progressive networks.
22


Reviews:
[{"review_id": "wa_sLbryPrl", "reviewer": "Reviewer_i4Jc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Existing exploration methods tend to explore at the frontier of the currently seen states. The authors of this paper propose to direct the goal-conditioned policy in a way that induces higher exploration value trajectories for the agent. Specifically they first use the goal-conditioned policy $\\pi^G$ to reach a such a state $s_T$ that maximizes the exploration value $V^E(s_T)$.\n\nExtensive experiments on control as well as robotic manipulation showcase the sample-efficiency of the method. In particular, the method achieves really good results on hard task of block-stacking which the previous works fail at.", "summary_of_the_review": "Given the clear motivation, methodology and experimentation of the paper, I lean towards accepting the paper (*6: marginally above the acceptance threshold*). However, I'd want the authors to address my concerns as well as other reviewers' comments during the rebuttal phase, and my final decision would be subject to the rebuttal.\n\n-----\n\nPost-rebuttal update:\n\nI've updated my score to 8 -- good paper, accept after going through other reviewers' comments and rebuttal. The authors' rebuttal addresses my concerns.", "strength_and_weaknesses": "**Strengths**\n\n1. The idea is novel, and the paper is clearly written, making it easy for the reader to understand the motivation as well as the method.\n\n2. Thorough experiments to validate the claim of optimizing goal commands directly.\n\n\n**Weaknesses**\n\n**Claims (clarification)**\n\n3. The authors distinguish directly optimizing for goal commands with exploration methods that aim to reach novel states, and say:\n\n    > Note that this does not mean merely commanding the agent to novel or rarely observed states. Instead, PEG commands might be to a previously observed state... PEG only cares that the command will induce the chained GCRL and exploration phases together to generate interesting training trajectories, valuable for policy improvement.\n\n    Shouldn't it be the case that the \"interesting trajectory\" would encounter some novel states -- otherwise the total exploration reward for that trajectory wouldn't be maximum?\n\n\n**Methodology**\n\n4. From equations (2) and (5) and the choice of K=1, is it correct to say the following:\n\n    $\\max_g \\mathbb{E}_{p_\\pi G(.|., g)}(s_T)[V^E(s_T)] ≈ \\frac{1}{K} \\sum_k V_\\theta ^E(s_T^k) = V_\\theta ^E(s_T)$\n\n\n\n    a) What could be the reason that K>1 doesn't have a large effect on the performance? \n\n    b) Additionally, in appendix A.5, where CEM is compared with MPPI, what were the hyperparameters for CEM? How does CEM perform with more number of trajectories?\n\n\n**Citations**\n\n5. In the Conclusion section, please also cite [1] along with [2] and [3]. Works [1] and [2] were independently published around the same time and share the same idea.\n\n------\n\n**References**\n\n[1] Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, Bernhard Schölkopf, Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models, In NeurIPS 2020.\n\n[2] Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine. Time-agnostic prediction:Predicting predictable video frames. ICLR, 2019.\n\n[3] Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. In NeurIPS 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jwTAeoyVlp", "reviewer": "Reviewer_rUvT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a method to quickly explore a state space and achieve goals that require long-horizon trajectories. They define the exploration value of a trajectory based on learned transition functions and policies for sampling trajectories, which are added to the training buffer. Their experiments on four tasks from 2D point navigation to 3D locomotion and 3-block stacking demonstrate the effectiveness of their method in comparison to two other baselines for long-horizon multi-goal RL. ", "summary_of_the_review": "This is a well-written paper on an important topic. The method has the potential to benefit robot navigation and manipulation in new environments. While the reviewer has concerns over the variance of one of their experiments, the results are overall very convincing.", "strength_and_weaknesses": "The authors contributed a useful method that speeds up exploration thus resulting in less self-exploration time for agents in a new environment to achieve goals in the furthest states. Their experiment environments are diverse and the results are strong.\n\nHowever, the test goals are selected from a specific distribution of states that require the furthest explorations. In Figure 4, the variance for PEG is very high, which makes the results less convincing given only five seeds. What are the authors' thoughts on an experimental design that could reduce the variance?\n\nFurthermore, the reviewer would like the authors to address the following questions to make the conclusions more convincing:\n\n1. About Algorithm 1, since training prioritizes exploration, how much is exploitation performance affected (e.g. the number of steps to achieve a goal sampled uniform at random in the environment) compared to other baselines? \n2. in Figure 6, many goals are sampled close to the start state (red dots on the left of the blue dot) and a few around the first corner, which increases more in quantity than those in remote states throughout training. Why might that be the case?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E3wh8CL98w", "reviewer": "Reviewer_5sEN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the \"Planning Exploratory Goals\" algorithm, which identifies goal states that are likely to result in many novel observations when used to initialize an subsequent explore-phase upon reaching a termination condition en route to the aforementioned goal. The algorithm overcomes prior issues, such as working with poor quality starting policies and unreachable (exploratory) goals, by taking the agent's learned transition (\"world\") model into account. This enables it to effectively explore in many goal-conditioned reinforcement learning scenarios.", "summary_of_the_review": "This is a strong paper that improves the state of the art in goal conditioned reinforcement learning in a wide range of problems. The algorithm is interesting and empirically validated, and I think it will inspire good follow-up work and discussion in the community.", "strength_and_weaknesses": "The paper's strengths include clear writing, interesting figures, and an effective new general purpose algorithm for goal-conditioned reinforcement learning. It effectively highlights differences between the new algorithm and baselines in (exploratory) goal setting approaches, with good explanations on how these differences ultimately contribute to better performance. The algorithm is well explained, and ablation tests validate the increased complexity of certain design choices. The description of future work and limitations of the current PEG algorithm (specifically regarding \n\nIn my opinion, the main weakness of the paper is the significance and standardization of the experimental results. The error bars on the results in figure 4 are borderline convincing that PEG is a significant step up from the baselines, particularly in the first 3 experiments. The differences highlighted in Figures 5 and 6 help to explain the marginally increased performance of PEG over baselines in those experiments, but there is an omission of good analysis into the deficiency of baseline policies relative to PEG in the 3-block stack experiment. Furthermore, the paper notes that evaluation goals were picked to \"require extensive exploration\" -- more work needs to be done to justify why this makes the test environments more representative of real-world problems, and why is does not simply bias the experimental results towards PEG (which is explicitly designed for this behaviour). I think that the \"3-block stack\" paragraph of subsection 4.3 could be improved through the use of figures and other analysis to demonstrate issues in the baselines -- I also believe that the use of the exclamation and rhetorical question in this paragraph (\"PEG experiences... throughout training!\" and \"Did the... Explore-Phase?\") are inappropriate and more passive language should be used in this section (particularly because the first statement is critical of prior works).\n\nI would also like to see some discussion of value of information/exploration methods in the related works, since this approach of identifying states with \"high-exploratory value\" seems related to other works in the areas of quantifying the similarity of states and other analysis from general exploration in POMDPS.  And, I would like a bit more rigour in the implementation details (e.g., \"optimization only takes a few seconds\" on what kind of hardware? is this significantly faster/slower than X?).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E16p9aQYQIM", "reviewer": "Reviewer_p4aC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel Reinforcement Learning based unsupervised method for the task of goal planning for agent based exploration in a non task-centric paradigm.", "summary_of_the_review": "\nAs it currently stands, the paper has several positive aspects rooting for it. Not having access to the code and finer graine details \nregarding the intrinsics and experiments is a slight turn-off, but regarding the standards of the field and the venue, it remains on the acceptable side.\n\nThe downsides however, include a lack of experimental challenge of the parameter tuning and the overal descriptive focus is not emphasized enough on the conceptual differential between the proposed method and the original prior work it builds upon, i.e., Mendonca et al. NeurIPS 2021.\n\nOverall, it is hence slightly bellow borderline but could I would be happy to revise my stance and discuss the (still/currently) missing bits.\n", "strength_and_weaknesses": "# Strengths:\n\n- (1) Readability.\n\nOverall, the readability is fair. The main ideas are well conveyed, justified and articulated while mostly providing sufficient context to the reader, albeit sometimes superficially (cf. below for the downsides). The grammar, and overall the writing, is relatively mature in its current state.\n\n- (2) Experimental Evaluation.\n\nThis one is a blend of benchmark per se, engineering to re-implement pertinent baselines and ablative study of the proposed conceptual contribution.\n\n- (3) The Performance of the proposed method.\n\n- (4) The Reproducibility.\n\nThis one is somewhat of a mixed bag. The positive aspects predominently lie in the proactive care the authors have put into explaining the experimental set-up, the re-implementation of the baselines and the description of the considered dataset environments.\nIn absence of an existing, public, standardized and unified benchmark environment, the proposed package is fair on many aspects (a few hicups though, as discussed below).\n\n# Weaknesses:\nDespite several positive aspects to the submitted paper, there are several noticeable shortcomings. In particular,\n\n\n- (1) How much does it cost?\n\nWhile the experimental evaluation which focuses on the task-oriented performance per se, alternative considerations could be discussed to help better understand the relative positioning of the proposed method and how it contrasts (favourably or not) on algorithmic and resource usage and management considerations.\n\n- (2) Related Work.\nThe section is awkwardly placed as a #5 section, right before the conclusion and is relatively short.\nIt is rather superficial by often characterizing methods at a very top-level, rather than focusing on their key differanciations and relative interconnections from a mechanistic-centric point of view. \n\nTo a lesser extent, it also ommits quite a few pertinent work in the field, e.g.,\n\n(Sub-goal oriented, and references therein)\n* CHANE-SANE, Elliot, SCHMID, Cordelia, et LAPTEV, Ivan. Goal-conditioned reinforcement learning with imagined subgoals. In : International Conference on Machine Learning. PMLR, 2021. p. 1430-1440.\n\n\n(Seminal formulation of several related concepts)\n* YAMAUCHI, Brian. Frontier-based exploration using multiple robots. In : Proceedings of the second international conference on Autonomous agents. 1998. p. 47-53.\n\n* YAMAUCHI, Brian. A frontier-based approach for autonomous exploration. In : Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97.'Towards New Computational Principles for Robotics and Automation'. IEEE, 1997. p. 146-151.\n\n- (3) The emphasis should be much stronger to explain the difference between the added contritbution and the vanilla prior work from Mendonca NeurIPS 2021.\n\nThe corresponding section (slightly more than a 1-pager) to describe it, including the original (prior work) work on which this one is based is way too shallow to support the current state of the submissions.\n\n- (4) Reproducibility:\n\nAdditionally, despite having many implementation details provided, the parameter tuning seems relatively simplistic and unchallenged.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cbtmg9MFFt", "reviewer": "Reviewer_WR3C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes planning exploratory goals (PEG) for helping an agent explore during training time to achieve diverse goals revealed at test time. PEG brings together the Go-Explore framework, where a goal-conditioned policy brings an agent to a goal, and then an undirected exploration policy explores around that goal, with Latent Explorer Achiever (LEXA), which learns a world model that is used to train an explorer and goal achiever policy. ", "summary_of_the_review": "Due to lack of clarity and issues with the method/evaluation, I think this paper is not ready for acceptance.\n\nEdit: based on additional experiments and clarifications from the authors during the rebuttal period, I have increased my score.", "strength_and_weaknesses": "Strengths:\n- The paper proposes an interesting idea in bringing together Go-Explore and LEXA and proposing a new objective for what goals to choose.\n- The visualizations are helpful for building intuition, and the 3-Block Stack results are quite nice. The ablations are also very useful. \n\nWeaknesses:\n- It is not clearly described in the main text how the goals in the algorithm are sampled. Maybe I missed this somewhere, but this is a very critical part of the method that is not well described. Can the authors clarify this?\n- Relatedly, it seems that the goals chosen by PEG are not in the replay buffer and are in fact sometimes unreachable, e.g. in Figure 5, and this is also mentioned in the paper text. It seems like none of the comparisons make the assumption of sampling from the state space. This is a pretty big difference. Can the authors clarify this? \n- All the experiments are using low-dimensional state spaces, although LEXA has been shown to work in settings with high-dimensional inputs. It would be nice to see experiments with vision inputs.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%