Discovering and Achieving Goals via World Models
Authors: Russell Mendonca
Oleh Rybkin
Kostas Daniilidis
Danijar Hafner
Deepak Pathak
Venue: NeurIPS 2021 Poster (2021)
Keywords: unsupervised goal reaching
unsupervised rl
goal-conditioned rl
exploration
model-based rl
world models

Abstract: How can artificial agents learn to solve many diverse tasks in complex visual environments without any supervision? We decompose this question into two challenges: discovering new goals and learning to reliably achieve them. Our proposed agent, Latent Explorer Achiever (LEXA), addresses both challenges by learning a world model from image inputs and using it to train an explorer and an achiever policy via imagined rollouts. Unlike prior methods that explore by reaching previously visited states, the explorer plans to discover unseen surprising states through foresight, which are then used as diverse targets for the achiever to practice. After the unsupervised phase, LEXA solves tasks specified as goal images zero-shot without any additional learning. LEXA substantially outperforms previous approaches to unsupervised goal reaching, both on prior benchmarks and on a new challenging benchmark with 40 test tasks spanning across four robotic manipulation and locomotion domains. LEXA further achieves goals that require interacting with multiple objects in sequence. Project page: https://orybkin.github.io/lexa/

Discovering and Achieving Goals via World Models
Russell Mendonca*
Oleh Rybkin* University of Pennsylvania
Carnegie Mellon University
Kostas Daniilidis
Danijar Hafner
Deepak Pathak
University of Pennsylvania
University of Toronto
Carnegie Mellon University
Abstract
How can artificial agents learn to solve many diverse tasks in complex visual environments without any supervision? We decompose this question into two challenges: discovering new goals and learning to reliably achieve them. Our proposed agent, Latent Explorer Achiever (LEXA), addresses both challenges by learning a world model from image inputs and using it to train an explorer and achiever policy via imagined rollouts. Unlike prior methods that explore by reaching previously visited states, the explorer plans to discover unseen surprising states through foresight, which are then used as diverse targets for the achiever to practice. After the unsupervised phase, LEXA solves tasks specified as goal images zero-shot without any additional learning. LEXA substantially outperforms previous approaches to unsupervised goal reaching. both on prior benchmarks and on new challenging benchmark with 40 test tasks spanning across four robotic manipulation and locomotion domains. LEXA further achieves goals that require interacting with multiple objects in sequence.
Introduction
Unsupervised Interaction
How can we build an agent that learns to solve hundreds of tasks in complex visual environ- ments, such as rearranging objects with a robot arm or completing chores in kitchen? While traditional reinforcement learning (RI.) has been successful for individual tasks, it requires sub- stantial amount of human effort for every new task. Specifying task rewards requires domain knowledge, access to object positions, is time- consuming, and prone to human errors. More- over, traditional RL would require environment interaction to explore and practice in the environ- ment for every new task. Instead, we approach learning hundreds of tasks through the paradigm of unsupervised goal-conditioned RI., where the agent learns many diverse skills in the environ- ment in the complete absence of supervision, to later solve tasks via user-specified goal images immediately without further training [2, 26, 40]. Challenges Exploring the environment and learning to solve many different tasks is sub- stantially more challenging than traditional RL with a dense reward function or learning from
Random Goal Image
Achiever
Replay Buffer
World Model
Environment without rewards
Explorer
Zero-Shot Evaluation
GOSD
User Goal Image
Achiever (frozen)
Environment
Figure 1: LEXA learns world model without any supervision, and leverages it to train two policies in imagination. The explorer finds new images and the achiever Icarns to reliably reach them. Once trained, the achiever reaches user-specified goals zero-shot without further training at test time.
Equal contribution. Ordering determined at random. Project page: https://orybliin http://orbbki..iihhb.b.b.. wwthub.io/ee........ io/lexa/
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

Robo Yoga
Side Angle
Head Stand
Arabesque
Lie Back
Lic Side
Attack
RoboBins
Reach Right
Push Front
Push Back
Push Both Back
Place Front
Place Both Front
RoboKitchen
Burner
Light
Slide
Hinge
Microwave
Kettle
Figure 2: We benchmark LEXA across four visual control environments. representative sample of the test-time goals is shown here. Robo Yoga features complex locomotion and precise control of nigh-dimensional agents, RoboBins manipulation with multiple objects, and RoboKitchen a variety of diverse tasks that require complex control strategies such as opening cabinet.
expert demonstrations. Existing methods are limited to simple tasks, such as picking or pushing a puck [13, 32, 37] controlling simple 2D robots [50]. The key challenge in improving the performance of unsupervised RL is exploration. In particular, previous approaches explore by either revisiting previously seen rare goals [14, 18, 55] or sampling goals from a generative model [32, 37]. However, in both these approaches, the policy as well as the generative model are trained on previously visited states from the replay buffer, and hence the sampled goals are either within or near the frontier of agent's experience. Ideally, we would like the agent to discover goals much beyond its frontier for efficient exploration, but how does an agent generate goals that it yet to encounter? This an open question not just for AI but for cognitive science too [42].
Approach To rectify this issue, leverage learned world model train separate explorer and achiever policy in imagination. Instead of randomly sampling or generating goals, our explorer policy discovers distant goals by first planning sequence of actions optimized imagination of the world model to find novel states with high expected information gain [30, 43, 44]. It then executes those imagined actions in the environment to discover interesting states without the need to generate them. Note these actions are likely to lead the agent to states which are several steps outside the frontier because otherwise the model wouldn't 't have had high uncertainty or information gain. Finally, these discovered states are used as diverse targets for the achiever to practice. We train the achiever from on- on-policy imagination rollouts within the world model and without relying on experience relabeling, therefore leveraging foresight over hindsight. After this unsupervised training phase, the achiever solves tasks specified as goal images zero-shot without any additional learning at deployment. Unlike in the conventional RL paradigm [31, 47], method is trained and then used achieve several tasks at test time without any supervision during training or testing.
Contributions We introduce Latent Explorer Achiever (LEXA), an unsupervised goal reaching agent that trains an explorer and an achiever within a shared world model. At training, LEXA unlocks diverse data for goal reaching in environments where exploration is nontrivial. At test time, the achiever solves challenging locomotion and manipulation tasks provided as user-specified goal images. Our contributions
summarized follows:
We propose to learn separate explorer and achiever policies as an approach to overcome the exploration problem of unsupervised goal-conditioned RI..
We show that woward-lookiin exploration by planning with a learned world model substantially outperforms previous strategies for goal exploration.
To evaluate on challenging tasks, we introduce a new goal reaching benchmark with a total of 40 diverse goal images across different robot locomotion and manipulation environments.
LEXA outperforms prior methods, being the first to show success in the Kitchen robotic manipu- lation environment, and achieves goal images where multiple objects need be moved.

---

LEXA
Explorer raaa, s))
Achiever TAaa s. ed
V
T1
V
T
V
TI
V9
1
Tt
Ve
TE
V9
1
Tt
8
World Model
..
d(s, @)
Var((s'))
enc
enc
Random Goal Image
X
Replay Buffer
X
X
Figure 3: Latent Explorer Achiever (LEXA) learns a general world model that is used to train an explorer and a goal achiever policy. The explorer (left) is trained on imagined latent state rollouts of the world model S:r to maximize the disagreement objective ry = Var(s') The goal achiever (right) conditioned on goal g and also trained on imagined rollouts minimize distance function d(st, eg). Goals are sampled randomly from replay buffer images. For training temporal distance, we use the imagined rollouts the achiever and predict the number of time steps between each two states. By combining heward-looking exploration and data-efficient training of the achiever, LEXA provides a simple and powerful solution for unsupervised reinforcement learning.
2 Latent Explorer Achiever (LEXA)
Our aim is to build an agent that can achieve arbitrary user-specified goals after learning in the environment without any supervision. This presents challenges collecting trajectories that contain diverse goals and learning to achieve these goals when specified as goal image. We introduce simple solution based The world model represents the agent's current knowledge about the environment and is used for training two policies, the explorer and the achiever. To explore novel situations, construct an estimate of which states the world model is still uncertain about. To achieve goals, we train the coal-conditioned achiever in imagination, using the images found so far as unsupervised goals. test time, the achiever is deployed to reach user-specified goals. The training procedure is in Algorithm 1.
world model and imagination training that addresses both challenges.
2.1
World Model
To efficiently predict potential outcomes of future actions in environments with high-dimensional image inputs, we leverage Recurrent State Space Model (RSSM) [23] that leamss to predict forward using compact model states that facilitate planning [7, 51J. In contrast to predicting forward image space, the model states enable efficient parallel planning with large batch size and can reduce accumulating errors [39]. The world model consists of the following components:
Encoder:
c = cnc)(1:)
Posterior:
(s( 8t-1, 0t,,,
(I)
Dynamics:
(s 1 Image decoder:
Po (o( 8t)
The model states 8 contain a deterministic component hú and stochastic component 2, with diagonal- covariance Gaussian distribution. h, is the recurrent state of Gated Recurrent Unit (GRU) [11]. The encoder and decoder are convolutional neural networks (CNNs) and the remaining components are multi-layer perceptrons (MLPs). The world model is trained end-to-end by optimizing the evidence lower bound (ELBO) via stochastic backpropagation [28, 38] with the Adam optimizer [27].
2.2 Explorer
To efficiently explore, we seek out surprising states imagined by the world model [6, 41, 43, 44, 46]. as opposed to retrospectively exploring by revisiting previously novel states [4, 5, 8, 34]. As world model can predict model states that correspond to unseen situations in the environment, the imagined trajectories contain more novel goals, compared to model-free exploration that limited to the replay buffer. To collect informative novel trajectories in the environment, we train exploration

---

Algorithm 1: Latent Explorer Achiever (LEXA)
initialize: World model M, Replay buffer D, Explorer x (at 2t), Achiever (²(a;) 21.9)
2: while exploring do
3: Train M on D
4: Train xf in imagination of M to maximize exploration rewards Σ, r
5: Train z in imagination of M to maximize Σ. (=t,g) for images g ~ DD
6: (Optional) Train d(zi, 2y) to predict distances j the imagination data from last step.
7: Deploy ne in the environment to explore and grow D.
8: Deploy 8h in the environment to achieve goal image g D to grow D.
9: end while
while evaluating do
11: given: Evaluation goal
12:
Deploy 888 in the world to reach g.
13: end while
policy Te from the model states 8, in imagination of the world model to maximize an exploration reward:
Explorer: x <'aa st) Explorer Value:
2 (st)
To explore the most informative model states, we estimate the epistemic uncertainty disagreement of an ensemble of transition functions. We train an ensemble of 1 1-step models to predict the next model state from the current model state. The ensemble model is trained alongside the world model on model states produced by the encoder qa- Because the ensemble models are initialized at random, they will differ, especially for inputs that they have not been trained on [29, 36]:
Ensemble: f(s., 0*) 2f41 for == 1..K
Leveraging the ensemble, we estimate the epistemic uncertainty as the ensemble disagreement. The exploration reward is the variance of the ensemble predictions averaged across dimension of the model state, which approximates the expected information gain [3, 43]:
(8t) -
N Σ Var(o) [f (((s,0x)).. 0k))
(4)
The explorer n" maximizes the sum of future exploration rewards rf using the Dreamer algorithm [24], which considers long-tern rewards into the future by maximizing X-returns under a learned value function. As result, the explorer is trained to seek out situations are as informative as possible from imagined latent trajectories of the world model, and is periodically deployed in the environment to add novel trajectories the replay buffer, so the world model and goal achiever policy can improve.
2.3 Achiever
To leverage the knowledge obtained by exploration for learning to reach goals, we train a goal achiever policy T that receives model state and goal as input. Our aim is to train a general policy that is capable reaching many diverse goals. To achieve this a data-efficient way, it crucial that environment trajectories that were collected with one goal in mind reused to also learn how to reach other goals. While prior work addressed this by goal relabeling which makes off-policy policy optimization necessity [2], we instead leverage past trajectories via the world model trained on them that lets us generate an unlimited amount of new imagined trajectories for training the goal achiever on-policy in imagination. This simplifies policy optimization and can improve stability, while still sharing all collected experience across many goals.
(5) To train the goal achiever, we sample a goal image Ig from the replay buffer and compute its embedding 6g = (x.) The achiever aims to maximize unsupervised goal-reaching reward r (g). We discuss different choices for this reward in Section 2.4. We again use the Dreamer algorithm [24] for training, where now the value function also receives the goal embedding as input. In addition to imagination training, it can also important to perform practice trials with the goal achiever in the true environment, so that any model inaccuracies along the goal reaching trajectories
Achiever:
(a, (t,eg) Achiever Value:
(st,

---

Goal Image
LEXA zero-shot evaluation trajectory
time
Figure 4: Successful LEXA trajectories. When given a goal image from the test set, LEXA's achiever used in the environment reach that image. On RoboKitchen, LEXA manipulates up three different objects together from single goal image (kettle, light switch, and cabinet). On RoboBins, LEXA performs temporally extended tasks such as picking and placing two objects in row.
may be corrected. To perform practice trials, sample goal from the replay buffer and execute the goal achiever policy for that goal in the environment. These trials are interleaved with exploration episodes collected by the exploration policy in equal proportion. We note that the goal achiever learning is entirely unsupervised because the practice goals are simply images the agent encountered through exploration or during previous practice trails.
2.4 Latent Distances
Training the achiever policy requires us define goal achievement reward ("(ss, Eg) that measures how close the latent state s; should be considered to the goal e.. One simple measure is the cosine distance in the latent space obtained by inputting image observations into the world-model. However, such a distance function brings visually similar states together even if they could be farther apart in temporal manner measured by actions needed reach from to other. This bias makes this suitable only to scenarios where most of pixels in the observations are directly controllable, e.g., trying to arrange robot's body certain shape, such Robo Yoga poses in Figure 2. However, many environments contain agent as well as the world, such as manipulation involves interacting with objects that are not directly controllable. The cosine distance would try matching the entire goal image, and thus places large weight on both matching the robot and object positions with the desired goal. Since the robot position directly controllable is much easier match, but this metric overly focuses on it, yielding poor policies that ignore objects. We address this is by using the number of timesteps takes to move from one image to another as a distance measure [25, 26). This ignores large changes in robot position, since these can be completed in very few steps, and will instead focus more on the objects. This temporal cost function can learned purely in imagination rollouts from world model allowing as much data needed without taking any steps in the real world.
Cosine Distance To use cosine distance with LEXA, for latent state st, and goal embedding ea, we use the latent inference network to infer 8", and define the reward as the cosine similarity [54]:
(st, Eg)
Σ Bei
where 3, = =tllselllls
By /g/ /l/lls
6
i.e. the cosine of the angle between the two vectors 8t, Sy in the N didieensional latent space.
Temporal Distance To temporal distances with LEXA, we train neural network d to predict the number of time steps between two embeddings. We train it by sampling pairs of states &,, $ from an imagined rollout of the achiever and predicting the distance k. We implement the temporal distance in terms of predicted image embeddings éc+ in order to remove extra recurrent information:
Predicted embedding: emb(s:) é e, Temporal distance:
(e,, (xtk) k/H, (7)
where the maximum distance equal to the imagination horizon. Training distance function only on imagination data from the same trajectory would cause it to predict poor distance to far away states coming from other trajectories, such as images that are impossible to reach during one episode. In order to incorporate learning signal from such far-away goals, include them by sampling images
5

---

104
10a
10a
10a
10a
10
10
burner
light
slide
hinge
microwave
kettle
slide slide+Iightt light
hinge aage++light Hight
kettle +iee++iign light
slide thinge slide kettle kettle+hinge
kettle thinge
reach left
reach right
push front
place front
push back
push both fropface both froglush both back
LEXA + temporal (Ours)
DDL
SkewFit
Figure Coincidental goal success achieved during the unsupervised exploration phase. The forward- looking explorer policy of LEXA results in substantially better coverage compared to SkewFit,
popular method for goal based exploration.
from a different trajectory. We annotate these negative samples with the maximum possible distance, so that the agent always prefers images that were seen in the same trajectory.
(st, eg) = (&,, eg), where & emb(s:), Eg - encp((r)))
The learned distance function depends on the training data policy. However, as the policy becomes more competent, the distance estimates will be closer to the optimal number of time steps to reach particular goal, and the policy converges to the optimal solution [25]. LEXA always uses the latest data to train the distance function using imagination, ensuring that the convergence is fast.
3 Experiments
Our evaluation focuses the following scientific questions:
1. Does LEXA outperform prior work on previous benchmarks and new challenging benchmark?
2. How does forward-looking exploration of goals compare previous goal exploration strategies?
3. How does the distance function affect the ability to reach goals in different types of environments?
4. Can we train one general LEXA to control different robots across visually distinct environments?
5. What components of LEXA important for performance?
We evaluate LEXA on prior benchmarks used by SkewFit [37], DISCERN [50], and Plan2Explore [43] in Section 3.3. Since these benchmarks are largely saturated, we also introduce new challenging benchmark shown in Figure 2. We evaluate LEXA on this benchmark is Section 3.2.
3.1 Experimental setup
As not many prior methods have shown success on reaching diverse goals from image inputs, we perform an apples-to-apples comparison by implementing the baselines using the same world model and policy optimization as our method:
SkewFit SkewFit [37] uses model-free hindsight experience replay and explores by sampling goals from the latent space of a variational autoencoder [28, 38). Being one of the sfate-of-the-ee agents, use the original implementation that does not use world model or explorer policy.
DDL Dynamic Distance Learning [25] trains a temporal distance function similar to our method. Following the original algorithm, DDL uses greedy exploration and trains the distance function on the replay buffer instead of imagination.
DIAYN Diversity is All You Need [15] learns a latent skill space and uses mutual information between skills and reached states as the objective. We augment DIAYN with our explorer policy and train learned skill predictor to obtain a skill for a given test image [12].
GCSL Goal-Conditioned Supervised Learning [20] trains the goal policy replay buffer goals and mimics the actions that previously led to the goal. We also augment GCSL with our explorer policy, found no learning success without it.
Our new benchmark defines goal images for diverse set of four existing environments as follows:

---

RoboYoga Walker
RoboYoga Quadruped
RoboBin
RoboKitchen
Goal Sucpess,
80
45
60
30
40
15
20
20
20
0
0
0.0
1.5
3.0
4.5
6.0
3
0
2
0.0
1.5
3.0
4.5
6.0
1e8
1e6
1e6
1e6
Environment Samples
Environment Samples
Environment Samples
Environment Samples
LEXA temporal (Ours)
DDL [26]
GCSL [20]
LEXA cosine (Ours)
DIAYN [15]
SkewFit [37]
Figure 6: Evaluation of goal reaching agents
four benchmarks. single agent is trained from images without rewards and then evaluated on reaching goal images from the test set (see Figure 1). Both LEXA agents solve many of the tasks and significantly outperform prior work. SkewFit and DLL struggle with exploration, while DIAYN and GCSL use our explorer but still are not able to learn good downstream policy, Refer table for final success percentage (averaged across tasks) for each method and benchmark domain.
Robo Yoga We use the walker and quadruped domains of the Deep.Mind Control Suite [48] to define the Yoga benchmark, consisting of 12 goal images that correspond to different body poses for each of the two environments, such as lying down, standing up, and balancing.
RoboBins Based Meta World [53], we create scene with Sawyer robotic arm, two bins, and two blocks of different colors. The goal images specify tasks that include reaching, manipulating only one block, and manipulating both blocks.
RoboKitchen The last benchmark involves the challenging kitchen environment from [22], where franka robot can interact with various objects including burner, light switch, sliding cabinet, hinge cabinet, microwave, or kettle. The goal images we include describe tasks that require interacting with only one object, well interacting with two objects.
3.2 Performance on New Benchmark
We show the results on our main benchmark in Figure and include heatmaps that show per-task success on each of the evaluation tasks from the benchmarks in the Appendix, Further, we report success averaged across tasks for each domain at the end of training in Table 1. We visualize example successful trajectory executions for tasks that require manipulating multiple objects in Fig. 4.
directly controllable since they contain other objects except the robot. We recall that for such settings we expect the cosine distance to be effective, perceptual distance quite accurate. Training is thus faster compared to using learned temporal distances, where the metric is learned from scratch. From Table and Figure we see that this is indeed the case for these environments (Walker and Quadruped), metric outperforms all prior approaches. Furthermore with temporal distances LEXA makes better progress compared to prior work a much larger number of goals can be seen from the per-task performance (Figures ??, ??), even though average success over goals looks similar to that of DDL. RoboBins This environment involves interaction with block objects, and thus is not directly control- lable, and so we expect LEXA to perform better with the temporal distance metric. From Table and
Robo Yoga The environments in this benchmark
LEXA with the cosine
Method
Kitchen
RoboBins
Quadruped
Walker
DDL
0.00
35.42
22.50
40.00
DIAYN
0.00
13.69
13.81
0.28
GCSL
0.00
7.94
15.83
1.11
SkewFit
0.23
15.77
5.52
0.01
LEXA Temporal (Ours)
37.50
69.44
31.39
36.72
LEXA Cosine (Ours)
6.02
45.83
56.11
73.06
Table 1: Performance new challenging benchmark, spanning across the four domains shown in Figure 2. The number are goal success rates, averaged over test goals within each environment.
7

---

Ours+ Temporal
0.94
0.89
0.56
0.94
0.44
0.33
Ours+ Cosine
0.5
0.5
DDL
0.83
0
0.17
DIAYN
0.62
GCSL
0.18
0.3
SkewFit
0.6
0.71
0.1
reach left
reach right
push front
place front
push back
push
place both front
push both back
both front
Figure 7: Success rates on RoboBin. In line with the prior literature, previous methods are successful at reaching and sometimes pushing. LEXA pushes the state-of oatie--ffth-errrrree by picking and placing multiple objects reach challenging goal images. Analogous heat maps for the other domains are included in the appendix.
Figure 6, we see that LEXA gets higher average success than all prior approaches. Further from the per-task performance in 7. LEXA with the temporal distance metric is the only approach that makes progress on all goals in the benchmark. The main difference in performance between using temporal and cosine distance can be seen in the tasks involving two blocks, which are the most complex tasks in this environment (the last 3 columns of the per-task plot). The best performing prior method is DDL which solves reaching, and can perform simple pushing tasks. This method performs poorly due to poor exploration, as shown in Figure 5, We see that while other prior methods make some progress reaching, they fail on harder tasks.
RoboKitchen This benchmark involves diverse objects that require different manipulation behavior. From Table and Figure and ?? we find that LEXA with temporal distance able to learn multiple RoboKitchen tasks, some of which require sequentially completing 2 tasks in the environment. All prior methods barely make progress due to the challenging nature of this benchmark, and furthermore using the cosine distance function makes very limited progress. The gap in performance between using the two distance functions is much larger in this environment compared to RoboBins since there are many more objects and they not clearly visible as the blocks.
Single Agent Across All Environments
In the previous sections we have shown that our approach can achieve diverse goals in different environments. However, we trained a new agent for every new environment, which doesn't scale well to large numbers of environments. Thus we investigate if we can train a train single agent across four environments the benchmark. From Figure ?? we see that our approach with learned temporal distance is able to make progress tasks from RoboKitchen, RoboBins Reaching, RoboBins Pick & Place and Walker, while the best prior method on the single-enviromment tasks (DDL) mainly solves walker tasks and reaching from RoboBin.
3.3 Performance on Prior benchmarks
To further verify the results obtained on our bench- mark, evaluate LEXA previously used bench- marks. We observe that LEXA significantly outper- forms prior work on these benchmarks, and is often close to the optimal policy. Additional details are provided in ??????
Table Goal distance for SkewFit goals [37].
Method
Pusher
Pickup
RIG [32]
7.7cm
3.7cm
RIG HER [2]
7.5cm
3.5cm
SkewFit Benchmark SkewFit [37] introduces robotic manipulation benchmark for unsupervised methods with simple tasks like planar pushing picking. We evaluate on this benchmark in Table 2.
Skew-Fit [37]
4.9cm
1.8cm
LEXA + Temporal
2.3cm
1.4em
Baseline results are taken from [37]. LEXA significantly outperforms prior work on these tasks.
Pushing and picking up blocks from images is largely solved and future work can focus on harder benchmarks such as those introduced in our paper.

---

DISCERN Benchmark We attempted to replicate the tasks described in [50] that are based on simple two-dimensional rohots [48]. While the original tasks are not released, we followed the procedure for gener- ating the goals described in the paper. Despite follow- ing the exact procedure, we were not able to obtain similar goals to the ones used in the original paper. Nevertheless, we show the goal completion percent- age results obtained with our reproduced evaluation compared to DISCERN results from the original pa- per. LEXA results were obtained with early stopping. In Table 3 we see that our agent solves many of the tasks in this benchmark.
Table 3: Success for DISCERN goals [50].
Task
LEXA
DISCERN
Cup
84.0%
76.5%
Cartpole
35.9%
21.3%
Finger
40.9
21.8%
Pendulum
79.1% %
75.7%
Pointmass
83.2%
49.6%
Reacher
100.0%
87.1%
Plan2Explore Benchmark We provide a compar- ison the standard seward-based DM control tasks [48] in Table 4. To compare on this benchmark, we create goal images that correspond to the reward func- tions. This setup is arguably harder for our agent, but much more practical. Note our agent never ob- serves the reward function and only observes the goal test time. Plan2Explore adapts new tasks but needs the reward function to be known at test time, while DrQV2 is an oracle agent that observes the reward at training time. Baseline results are taken from [43, 52]. LEXA results were obtained with early stopping. LEXA outperforms Plan2Fxplore on most tasks and even performs comparably to state of the art oracle agents (DrQ, DrQv2, Dreamer) that use true task rewards during training.
Table 4: Zero-shot return on P2E tasks [43].
Task
LEXA
P2E
DrQv2
Zero-Shot
/
Walker Stand
957
331
968
Hopper Stand
840
841
957
Cartpolc Balance
886
950
989
Cartpole Bal. Sparse
996
860
983
Pendulum Swing Up
788
792
837
Cup Catch
969
962
Reacher Hard
937
66
970
3.4 Analysis
Prior work Most work we compared against strug- gles with exploration, such as SkewFit and DLL meth- ods. DIAYN is augmented with explorer, but still fails to leverage the exploration data to learn a diverse set of skills. GCSL struggles to fit the ex-
80
Goal Success.
80
ploration data and produces behavior that does not solve the task, perhaps because the exploration data too diverse. We observed that all baselines make progress the simple reaching, but struggle with other tasks. We have experimented with several ver- sions and improvements the baselines and report the best obtained performance.
2
6
8
Environment Samples
Ablation of different components We ablated components of LEEXA on the RoboBins environment in Figure 8. Using separate explorer policy erucial as without it the agent does not discover the more in- teresting tasks. Without negative sampling the agent learns slower, perhaps because the distance function doesn't 't produce reasonable outputs when queried on images that are more than horizon length apart. Train- ing the distance function with real data converges to slightly lower success than using imagination data, since real data is sampled in an off-policy manner due to its limited quantity.
LEXA
No explorer
Replay buffer distance
No neg sampling
Figure 8: Ablations on RoboBins. A separate explorer is crucial for most tasks. Training temporal distance on negative samples speeds up learning, and both negative sampling and training imagination as opposed to real data important for the hardest tasks.
Exploration performance Due to importance of exploration, we further examine the diversity of the data collected during training. We log the instances where the agent coincidentally solves an evaluation task during exploration, for the RoboKitchen and RoboBins environments. In Figure see that method encounters harder tasks involving multiple objects much more often.

---

4 Related Work
Learning to Achieve Goals The problem of learning to reach many different goals has been commonly addressed with model-free methods that learn a single goal-conditioned policy [2, 26, 40]. Recent work has combined these approaches with various ways to generate training goals, such as asymmetric self-play [33, 45] or by sampling goals of intermediate difficulty [14, 18). These approaches can achieve remarkable performance in simulated robotic domains, however, they focus on the settings where the agent can directly perceive the low-dimensional environment state.
A few works have attempted scale these model-free methods to visual goals by using contrastive [50] or reconstructive [32. 37] representation learning. However, these approaches struggle to perform meaningful exploration as no clear reward signal available solving interesting tasks. Some works [10, 49] avoid this challenge by using a large dataset of interesting behaviors. Other works [37, 55] attempt to explore by generating goals similar to those that have already been seen, but do not try to explore truly novel states.
guide the agent toward
A particularly relevant set of approaches used model-based methods to achieve goals via planning [13, 17] or learning model-regularizee policies [35]. However, these approaches are limited by short planning horizons. In contrast, we learn long-horizon goal-conditioned value functions which allows us to solve more challenging tasks. More generally, most of the above approaches are limited by simplistic exploration, while our method leverages model imagination to search for novel states, which significantly improves exploration and in turn the downstream capabilities of the agent.
Learning Distance Functions A crucial challenge for visual goal reaching is the choice of the reward or the cost function for the goal achieving policy. Several approaches use representation learning to create distance in the feature space [9, 32, 50, 51J. However, this naive distance may not be most reflective of how hard a particular goal is to reach. One line of research has proposed using the mutual information between the current state and the goal as the distance metric [1, 12, 15, 21], however, it remains to be seen whether this approach can scale to more complex tasks.
Other works proposed temporal distances that measure the amount of time it takes to reach the goal. One approach is to learn the distance with approximate dynamic programming using Q-learning methods [16, 19, 26]. Our distance function is most similar to Hartikainen et al. [25], who learn a temporal distance with supervised learning on recent policy experience. In contrast to [25], we always train the distance on-policy in imagination, and we further integrate this achiever policy into our latent explorer achiever framework to discover novel goals for the achiever to practice on.
5 Conclusion
We presented Latent Explorer Achiever (LEXA), an agent for unsupervised RL that explores its environment, learns to achieve the discovered goals, and solves image-based tasks in zero-shot way. By planning for novelty in imagination, LEXA prospectively explores to discover meaningful behaviors in substantially more diverse environments than considered by prior work. Further, LEXA is able to solve challenging downstream tasks specified as images without any supervision such as rewards or demonstrations. By proposing a challenging benchmark and the first agent to achieve meaningful performance on these tasks, we hope to stimulate future research on unsupervised agents, which we believe are fundamentally more scalable than traditional agents that require human to design the tasks and rewards for learning.
Many challenges remain for building unsupervised agents. Many tasks in our benchmark are still unsolved and there remains room for progress on the algorithmic side both for the world model and policy optimization. Further, it is important demonstrate the benefits of unsupervised agents on real-world systems to verify their scalability. Finally, for widespread adoption, it is crucial to consider the problem of goal specification and design methods that act on goals that are easy to specify, such as via natural language. We believe LEXA will enable future work to tackle these goals effectively.
We thank Ben Eysenbach, Stephen Tian, Sergey Levine, Dinesh Jayaraman, Karl Pertsch, Ed Hu and the members of GRASP lab and Pathak lab for insightful discussions. We also thank Murtaza Dalal and Chuning Zhu for help with MuJoCo environments. Finally, DP would like to thank Laura Schulz, Josh Tenenbaum and Alison Gopnik for seeding the idea of goal-setting in children, and Pulkit Agrawal for several crucial discussions (over gelato) since then. OR and KD were supported by ARL DCIST CRA W91 ONR N00014-17-1-2093. and by Honda Research Institute. This work was partially supported by GoodAl Research Award and DARPA Machine Common Sense grant.
Acknowledgements
10

---

References
[1] Achiam, H. Edwards, D. Amodei, and Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 2018. 10
[2] M. Andrychowicz. F, Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint arXiv: 1707.01492
2017. 1, 4, 8, 10
[3] P. Ball, J. Parker-Holder, A. Pacchiano, K. Choromanski, and S. Roberts. Ready policy one: World building through active learning. In Interaational Conference on Machine Learning, pages 591-601. PMLR, 2020. 4
[4] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances Neural Information Processing Systems, pages 1471-1479, 2016.
[5] L. Beyer, D. Vincent, O. Teboul, S. Gelly, M. Geist, and O. Pietquin. Mulex: Disentangling exploitation from exploration in deep arXiv preprint (007,00900000066 2019.
[6] B. Bucher, K. Schmeckpeper, N. Matni, and Daniilidis. Adversarial curiosity. arXiv preprint arXiv:2003.06082 2020. 3
[7] L. Buesiing T. Weber, S. Racaniere, S. Eslami, D. Rezende, D. P. Reichert, F. Viola, F, Besse, K. Gregor, D. Hassabis, et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv: 18X.v:/802.030066 2018. 3
[8] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1e...1110...... 2018. 3
[9] V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, and J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[10] Y. Chebotar, K. Hausman, Y. Lu, Xiao, D. Kalashnikov, Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07744 2021. 10
[11] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
[12] Choi, A. Sharma, S. Levine, H. Lee, and S. S. Gu. Variational empowerment representation learning for goal-based reinforcement learning. In Deep Reinforcement Learning workshop at the Conference on Neural Information Processing Systems (DRL), 2020. 6, 10
[13] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprini arXiv:1812.00556 2018. 10
[14] A. Ecoffet, Huizinga, J. Lehman, K. O. Stanley, and J. Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv: 1901. 2019. 2, 10
[15] B. Eysenbach, A. Gupta, Ibarz, and Levine Diversity is all you need: learning skills without reward function. arXiv preprint arXiv:1802.06070, 2018. 6, 10
[16] B. Eysenbach, R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. arXiv preprint arXiv:1906.05255 2019. 10
[17] C. Finn and S. Levine. Deep visual foresight for planning robot motion. Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 2786-2793, IEEE, 2017. 10
[18] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. International conference on machine learning, pages 1515-1528. PMLR, 2018. 2. 10
11

---

[19] C. Florensa, J. Degrave. N. Heess, J. T. Springenberg. and M. Riedmiller. Self-supervised learning of image embedding for continuous control. arXiv preprint wVXi:11011.00443 2019. 10
[20] D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals without reinforcement learning. arXiv preprint vrXiiv:992.0660 2019.
[21] K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.0750, 2016. 10
[22] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy leamiing Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11955 2019. 7
[23] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 1.04551, 2018. 3
[24] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: (rXXv:1912.0060:: 2019. 4
[25] K. Hartikainen, X. Geng. T. Haarnoja, and S. Levine. Dynamical distance learning for semi- supervised and unsupervised skill discovery. ICLR, 2020. 5, 6, 10
[26] L. P. Kaclbling. Learning to achieve goals. In IJCAI, pages 1094-1099. Citeseer, 1993. 1,5, 10
[27] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arX.v:1412.6980, 2014. 3
[28] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114 2013. 3,
[29] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv: 162.0444, 2016.
[30] D. V. Lindley al. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 27(4):986-1005 1956. 2
[31] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 51 518(7540):529, 2015. 2
[32] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pages 9191-9200, 2018. 2, 8, 10
[33] O. OpenAI, M. Plappert, R. Sampedro, T. Xu, I. Akkaya, V. Kosaraju, P. Welinder, R. D'Sa, A. Petron, H. P. d. O. Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882 2021. 10
[34] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017. 3
[35] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, the IEEE conference
A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings the on computer vision and pattern recognition workshops, pages 0050-2053, 2018 10
[36] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning, pages 5062-500 5071, 2019. 4
[37] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv: 2019. 2, 6, 8, 10
[38] D. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arX1v:1401.4082, 2014. 3,
12

---

[39] V. Saxena, J. Ba, and D. Hafner. Clockwork variational autoencoders. arXiv:2102.05532. 2021.
arXiv preprint
[40] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference machine learning, pages 1312-1320. PMLR, 2015. 1, 10
[41] J. Schmidhuber. Curious model-building control systems. In [Proceedings] 1991 IEEE International Joint Conference on Neural Networks, pages 4463. IEEE, 1991. 3
[42] L.. Schulz. Finding new facts; thinking new thoughts. Advances in child development and behavior, 43:269-294, 2012.
[43] R. Sekar, O. Rybkin, K. Danillidis, P. Abbeel, Hafner, and D. Pathak. Planning explore via self--upervvsed world models. ICML, 2020. 2, 3, 4, 6,
[44] P. Shyam, W. Jaskowski, and F. Gomez. Model-based active exploration. arXiv preprint 1810. 22122, 2018. 2,3
[45] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play ICLR, 2018. 10
[46] Y. Sun, F. Gomez, and J. Schmidhuber. Planning be surprised: Optimal bayesian exploration in dynamic environments. In International Conference on Artificial General Intelligence, pages 41-51. Springer, 2011. 3
[47] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018. 2
[48] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Li, D. d. L. Casas, D. Budden, A. Abdolmaleki,
J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint 1rXivv1801.006%0 2018. 7,
9
[49] S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and Levine. Model-based visual planning with self-super functional distances. arXiv preprint arXiv:2012./5377, 2020. 10
[50] D. Warde-Farley, Van de Wiele. T. Kulkarni, C. Ioneseu, Hansen, and V. Mnih. Unsuper- vised control through non-parametric discriminative rewards. arXiv preprint arXiv:7811.1135% 2018. 2, 6, 10
[51] M. Watter, J. T. Springenherg, Boedecker, and M. Riedmiller. Embed to control: locally linear latent dynamics model for control from raw images. 2015. 3, 10
[52] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved lata-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
[53] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1 100. PMLR, 2020. 7
[54] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018. 5
[55] Y. Zhang, P. Abbeel, and L. Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33, 2020. 2,
13


Reviews:
[{"review_id": "wrrqDZdGA7", "reviewer": "Reviewer_dFJ7", "summary": "This paper proposes a model based RL setting where an agent learns a task agnostic model, learns two policies to explore and reach input goal states (specified as images) respectively and is able to operate in a goal conditional mode during evaluation time. \n\nThe key thesis is to use the world model to incentivize the agent to explore the environment by maximizing expected information gain and reach input goal states by learning to reach states from its past experiences. \n\nThere have been other model free and model based approaches studying the same problem but the claim is that its the first to combine model based exploration and reachability on new benchmark environments. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unsupervised RL is an interesting and important problem in deep and reinforcement learning. There have been several papers and baselines that have tackled this problem before. Perhaps the biggest limitation of this paper is lack of comparisons to past benchmarks. For instance the Sekar et al. paper (cited in the main paper) considers several environments and tasks that are not covered in this paper. The DISCERN (cited) model also provided benchmark results on Atari and continuous control tasks. The benchmark in this paper is new and therefore it is difficult to understand and compare the approach with past results. \n\nThe main argument that the authors present regarding past benchmarks is -- \"In contrast to common RL benchmarks such as Atari [5] that require training over 50 different agents and thus enormous computational resources, our unsupervised RL benchmark only requires training 4 agents, which are then evaluated across many tasks, allowing for faster iteration time and making the research more accessible\". It seems practical to at least try a handful of the same environments to set the right experimental standards. Otherwise it is very difficult to validate the claims and understand the effect of choices proposed in this paper. I believe the paper would become much stronger if this is addressed. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Seems adequate", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "lqT9oqVaRAQ", "reviewer": "Reviewer_LRem", "summary": "This paper introduces a framework to train policies for visual goal oriented tasks. The framework comprises two policies, the first, an explorer, which is trained to seek out novel states (where novelty is defined as the level of agreement between an ensemble of future state predictors - an approximate measure of epistemic uncertainty), while the second, an achiever, is trained to reach these states. A model based approach to train the achiever, where the achiever is trained using trajectories generated by a learned dynamics model. The paper also introduces a new benchmark for testing goal oriented polices of this form, comprising a kitchen manipulation task, a block handling task, and a yoga task. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe primary contribution of this paper is to show that an architecture explicitly balancing exploration and exploitation is also effective in high dimensional settings for goal oriented robot tasks. Existing work has focused on lower dimensional observations. I like the idea, and think its a good one, although the originality seems to be limited to scaling existing ideas to harder tasks.\n\nThe second contribution of the paper is a new suite of goal oriented tasks for robot goal tasks. These tasks seem to be more complex (in terms of manipulation, mapping of actions to goals) than existing benchmark suites, although I think some additional discussion on this may be warranted, eg.  benchmarks like robothor https://ai2thor.allenai.org/robothor/ are already widely used for goal oriented visual navigation, and have physical counterparts that are better suited for real world transfer.\n\nQuality and clarity:\n\nThe idea is solid, and experiments are clear and concise. Ablations highlight the importance of the explorer network. A number of baselines and comparisons to prior work are included.\n\nSignificance:\n\nThe results presented here are unsurprising, but solid. I think the benchmark of tasks is a nice contribution, and that the  demonstration that policy training balancing exploration and exploitation is effective in high dimensions is a useful.\n\nDetailed questions/comments:\n\nReward engineering. The introduction notes that a challenge with RL lies in reward engineering, where it can be difficult to choose effective rewards for tasks, and hence the need for unsupervised goal oriented learning. While I agree with this in principle, the results in this paper seem to me to contradict this, as it is clear that different cost functions eg. the cosine vs temporal distance measure, are still needed to train policies for different tasks. It seems to me that all that has happened here is that reward engineering has shifted to a different latent space. \n\nCould you clarify how the explorer and achiever were trained. Is this jointly, or is the explorer trained first, with the achiever then trained, by bootstrapping on data gathered by the explorer? From the text I gather the latter, but I would appreciate a bit more clarity here, eg. how much data was seen by each, etc.\n\nResults: How is success determined for each of the tasks? Is this distance to a particular simulator state? A perceptual visual metric? Some human label? \n\nIn some of the example videos, it seems that camera perspective effects is affecting the type of goals reached. How robust is this sort of approach to small changes in camera angle, etc.\n\nMinor: \n\nTypos: line 66, the introduce the\nline 129-230, grammar\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see immediate negative societal impact for this work, as it is extremely early stage research. \n\nHowever, a number of limitations were not discussed, and should be addressed in the paper.\n\nA major challenge preventing the deployment and practicality of exploratory policies in real world settings is safety. Looking at the motions followed by the policy, and given the ease with which torque controlled robots like the panda can be broken, it is clear that these policies are completely infeasible for real-world robots. I think it is important to make this clear, and reflect on the gaps between sim and real here, and the need for stringent safety constraints if this was attempted in real world settings.\n\nMoreover, the proposed approach will rely significantly on environment resetting, which is also infeasible in real world settings. Additional detail about the number of times and conditions under which these environments were reset, episode length etc. and being explicit about this would be valuable. ", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns about this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "cU3RjLq7b0H", "reviewer": "Reviewer_bDZB", "summary": "This paper proposes to jointly train two agents in a differentiable world model: an *explorer* that is rewarded to explore uncertain states and an *achiever* whose task is to achieve a given goal state. After training, the *achiever* can be deployed as a goal-reaching policy without further tuning. Authors also spent time investigating how to design the loss function for the *achiever*. In the experiments, three new benchmarks are introduced and the proposed method is shown to outperform baselines.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The main idea is novel and incorporates many sound ideas in the field including Dreamer, temporal distance, and disagreement for exploration. The related works are adequately cited and described so the contribution is clear to me. The questions asked from line 75-79 also help me understand how this work is different from previous contributions.\n\n**Quality**: Overall, the main idea kinda makes sense. However, some details still bother me:\n\n- Line 176-177, “... to incorporate learning signals from far-away goals, we include them by sampling images from a different trajectory.” -> I am not sure whether images from different trajectories are necessary *far away*. Theoretically, they can be close to each other in terms of “the number of time steps between them”. Does this implementation cause any training unstability?\n\nThe technical contributions are supported by the ablation study shown in Figure 8. Thank the authors for running this experiment, I find it pretty informative.\n\n**Clarity**: The high-level idea is presented clearly in the Introduction, Figure 1, and Figure 2. The low-level implementation details are described thoroughly in the supplementary material (which contains code). However, the mathematical notations in Section 2 are confusing and I hope authors can clean them up. I have the following questions:\n\n- What is the capital $V$ in Figure 2? Is it the same as the value $v$ in the text?\n- In section 2.1, $e_t$ shows up in equation 1's Posterior term but is not described in the text so its appearance hurts the readability.\n- In section 2.2, I think the reward function (equation 3 & 4) of the explorer should appear before equation 2 for readers to have a better understanding of what $v^e$ should look like. Currently, $v^e$ shows up abruptly in equation 2 and its relationship with $r^e$ is only described near the end of section 2.2.\n- Line 113 \"produced by the encoder $q_{\\phi}$\", so both $q_{\\phi}$ and $enc_{\\phi}$ in equation 1 are called \"encoder\"? I think authors are probably referring to the posterior estimator as an \"encoder\" following the VAE literature but it would be nice if authors can clarify its relationship with another \"encoder\" $enc_{\\phi}$ in the revised manuscript.\n- In equation 3, what’s the relationship between $\\hat{z}$ and $z$ defined at line 97?\n- Line 146 \"... to the goal $e_g$\". It would be nice if authors describe it as \"the embedding of the goal $e_g$” otherwise it confuses me as authors also denote g as goal at line 11 & 12 of Algorithm 1.\n- In equation 7, it's not clear why \\hat notation is introduced. How are variables with \\hat different from their original meaning? Also, why is $\\approx$ used instead of $=$?\n\nSome other wordings that confuse me:\n\n- Line 272-273, “real world data from the replay buffer”, it would be more clear if “real world data” is substituted with something like observed data as all the results in this paper come from the simulation.\n\nMany obvious typos appear in the text:\n\n- Line 31, \"a traditional RL would have to\" -> \"a traditional RL agent would have to\"\n- Line 61, \"between an the states\" -> \"between the states\"\n- Line 71-72, \"and thus enormous computational ...\" -> \"and thus consume enormous computational ...\"\nLine 120, “seek out situations are …” -> “seek out situations that are”.\nLine 156, “We address this is by …” -> “We address this by ...”.\nLine 250, “train a train a “ -> “train a”.\nLine 265, “i.e whether ..,” -> “i.e., whether”\nLine 267, “that is need” -> “that is needed”\n\n**Significance**: Since all the experiments are run on three *new* benchmarks introduced by authors, it’s really hard to tell whether the results are important. As the proposed idea is not limited to vision-based observations, I wonder whether authors can show results on state-based environments used by [1] and compare the performance. Experiments on state-based environments will help us understand 1) whether imagination is needed for low-dimensional observation space and 2) whether the proposed method, being more complicated, accidentally fails for simpler domains. \n\n[1] Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery, Hartikainen et al.\n\n---\n\nI am willing to raise my scores if authors addressed my concerns and above.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "25qs01REUCa", "reviewer": "Reviewer_txvK", "summary": "This paper proposes a method to learn goal-conditioned policies by combining ideas from goal-conditioned RL and model-based exploration. It operates by training an exploration policy within the model to maximize model disagreement (as proposed in previous works), as well as a goal-conditioned policy to reach states in the replay buffer. These are then both deployed to grow the replay buffer, and the process is repeated. The approach is evaluated on a newly-defined set of benchmark tasks based on the Deepmind Control suite, as well as simulated robotic manipulation. They report better performance compared to previous methods such as Skew-Fit, DIAYN and two others. They also investigate two different choices of distance functions for the goal-conditioned RL component, and show that the optimal one depends on the setting. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Update after the rebuttal: I thank the authors for having added an extensive comparison to prior work on previous environments. Since this was in my view the main limitation of this work, I have raise my score from a 4 to a 6. \n\n\n============================================================================\n\nMethods-wise, this is essentially a combination of model-based exploration methods which use the disagreement between dynamics models to incentivize exploration, and goal-conditioned RL (which is performed inside the dynamics model here). Incidentally, a piece of related work which should be mentioned is the following https://arxiv.org/pdf/1911.00617.pdf which proposed model-based exploration with multiple dynamics models and also provided some theoretical results for this class of algorithm.\n\nThe novelty of this paper is somewhat limited, in that it is a fairly straightforward combination of existing techniques. However I am not aware of this specific combination being investigated before. \n\nA big issue with the experimental setup is that the tasks are new and defined in this paper. It’s ok to define new tasks to see where existing methods fail, but it’s also important to know how the proposed method performs on tasks where existing methods are known to succeed. Also, checking that the baselines work in settings where they are known to succeed confirms the implementation, hyperparameter tuning etc is correct. Please include results for tasks from the papers where the baseline methods were proposed. \n\nFor example:\n- the simple grid-world environment from the Skew-Fit paper and/or the 2d navigation one from DIAYN. Both of those methods should work there. How do they compare to the proposed method in terms of sample complexity?\n- DIAYN uses the learned policy to initialize a policy optimizing task reward on some Mujoco tasks. Please include a comparison of the proposed method with the same setup. \n- etc\n\nIt’s also a bit unsatisfying that you need different distance metrics for different types of task (i.e. cosine for RoboYoga and temporal for the robotic manipulation). \n\nIt is unclear from the plot in Figure 4 if the other methods (DIAYN, DDL, SkewFit, GCSL) fail to learn well or if they are slower to converge. Please include all their final performance after longer training (could be a dashed line). \n\nIt is unclear why step 8 in the algorithm is necessary, since in principle (according to MAX, Plan2Explore etc) the disagreement should be sufficient for exploring the different regions of the state space. The way \\pi^g is trained is similar to Skew-Fit without the skewing step (and also training it in the model rather than the real environment), which that paper reports does not explore very fast. A useful ablation would be to try removing step 8 in Algorithm 1 and see how well it works. Does \\pi^g discover different types of states than \\pi^e? It seems like both step 7 and 8 are meant to add novel states to the buffer, but it’s not clear if they are exploring differently and are both needed.\n\nMinor:\nLines 7-8: “Unlike prior methods that explore by reaching previously visited states, our explorer…” this is not true, there are several methods (MAX, Neural-E3, Plan2Explore) which explore by planning/training a policy inside the model. \nLine 32: “A traditional RL would” -> a traditional RL agent would\nLine 66: The introduce -> We introduce \nLine 234: nvolves -> involves", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "-6ZH8Q7Mo5Q", "reviewer": "Reviewer_uhSG", "summary": "Summary: This paper proposes a model-based method with an ensemble disagreement-based exploration bonus for learning universal goal-conditioned policies. The world model and policy learning methods are adapted from DREAMER. In addition to that, the paper first introduces an explorer to first explore the state with large uncertainties, and then train the achiever to reach the proposed goals. The use of the world model enables data-efficient learning and exploration. The paper makes solid experiments and evaluates their approach on various domains. The paper also discusses two different distance metrics for learning goal-conditioned policy on images.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is well-written with convincing experiments. The proposed approach is simple yet effective. The separation of the exploration policy for goal generation and goal-conditioned policy learning looks like a general approach. The experimental results are not perfect. I notice that most tasks can't reach a 100 percent success rate. But considering the challenging image-input setup, it's great to see that it works in practice.\n- What concerns me most is the novelty of the paper. The paper's technique is a combination of image-based RL, uncertainty-based exploration, and goal-generation tasks, but I am still happy to see that one can assemble these techniques to solve challenging problems. I would be happy to see it at the Neurips conference.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim that \"Potential societal impacts of robot learning are at this stage largely hypothetical since the technology is not yet in widespread production.\" and I agree.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to learn a useful goal-conditioned policy in the absence of reward by jointly training an explorer that learns to visit uncertain states and an achiever that learns to reach a given goal state. All of the reviewers agreed that this is a novel combination of the prior work on learning a goal-conditioned policy in self-supervised learning and model-based RL, though each component is not entirely new. The only initial concern was that the proposed method was not evaluated on domains where the baselines were evaluated. However, the authors provided an additional result during the rebuttal period, and the majority of the reviewers are satisfied with it. Therefore, I recommend accepting this paper and suggest the authors to include the new result in the camera-ready version.
Decision: Accept (Poster)