Deep Hierarchical Planning from Pixels
Authors: Danijar Hafner
Kuang-Huei Lee
Ian Fischer
Pieter Abbeel
Venue: NeurIPS 2022 Accept (2022)
Keywords: Hierarchical Reinforcement Learning
World Models
Visual Control
Planning to Explore
Hierarchical Exploration
Goal-Conditioned RL

Abstract: Intelligent agents need to select long sequences of actions to solve complex tasks. While humans easily break down tasks into subgoals and reach them through millions of muscle commands, current artificial intelligence is limited to tasks with horizons of a few hundred decisions, despite large compute budgets. Research on hierarchical reinforcement learning aims to overcome this limitation but has proven to be challenging, current methods rely on manually specified goal spaces or subtasks, and no general solution exists. We introduce Director, a practical method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. The high-level policy maximizes task and exploration rewards by selecting latent goals and the low-level policy learns to achieve the goals. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels and outperforms exploration methods on tasks with very sparse rewards, including 3D maze traversal with a quadruped robot from an egocentric camera and proprioception, without access to the global position or top-down view used by prior work.

Deep Hierarchical Planning from Pixels
Danijar Hafner 123
Kuang-Huei Lee²
lan Fischer?
Pieter Abbeel
Abstract
Intelligent agents need select long sequences of actions solve complex tasks. While humans easily break down tasks into subgoals and reach them through millions of muscle commands, current artificial intelligence is limited to tasks with horizons of a few hundred decisions, despite large compute budgets. Research hierarchical reinforcement learning aims to overcome this limitation but has proven to be challenging, current methods rely on manually specified goal spaces subtasks. and no general solution exists. We introduce Director, practical method for learning hicrarchical behaviors directly from pixels by planning inside the latent space learned world model. The high-level policy maximizes task and exploration rewards by selecting latent goals and the low-level policy learns to achieve the goals. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director outperforms exploration methods on tasks with sparse rewards, including 3D maze traversal with quadruped robot from egocentric camera and proprioception, without access to global position or top-down view that was used by prior work. Director also learns successful behaviors across wide range of environments, including visual control, Atari games, and DMLab levels.
Introduction
Artificial agents have achieved remarkable performance reactive video games (Mnih et al., 2015; Badia et al., 2020) or board games that last for a few hundred moves (Silver al., 2017). However, solving complex control problems can require millions of time steps. For example, consider robot that needs to navigate along the sidewalk and cross streets to buy groceries and then return home and
Ant Maze
Episode
Goal
shown to agent)
T=0
T=40
T=80
T=120
T=160
T=200
T=240
Figure 1: Director on Ant Maze M from egocentric camera inputs. The top row shows agent inputs. The bottom row shows the internal subgoals of the agent. The goals are latent vectors that Director's world model can decode into images for human inspection. Director solves this sparse reward task by breaking down into internal goals. first targets the purple wall in the middle of the maze. Once reached, it targets the reward object, and then the brown wall behind to step onto the reward object.
'CC Berkeley
"Google Research Univerritty Toronto *Covariant
Correspondence to: Danijar Hafner <mail @ danijar.comm
Project website with videos and code: https:/ //aii jar. com/ /irrrteecoor 36th Conference Neural Information Processing Systems (NeurIPS 2022).

---

cook meal with those groceries. Manually specifying subtasks or dense rewards for such complex tasks would not only be expensive but also prone to eπrors and require tremendous effort to capture special cases (Chen et al., 2021; Ahn et al., 2022). Even training a robot to simply walk forward can require specifying ten different reward terms (Kumar al., 2021), ), making reward engineering a critical component of such systems. Humans naturally break long tasks into subgoals, each of which is easy to achieve. In contrast, most current reinforcement learning algorithms reason purely at the clock rate of their primitive actions. This poses key bottleneck of current reinforcement learning methods that could be challenging to solve by simply increasing the computational budget.
Hierarchical reinforcement learning (HRL) (Dayan and Hinton, 1992; Parr and Russell, 1997: Sutton al.. 1999) aims to automatically break long-horizon tasks into subgoals or commands that are easier to achieve, typically by learning high-level controllers that operate at more abstract time scales and provide commands low-level controllers that select primitive actions. However, most HRL. approaches require domain knowledge to break down tasks, either through manually specified subtasks (Tessler et al., 2017) or semantic goal spaces such as global XY coordinates for navigation tasks (Andrychowicz et al., 2017: Nachum al., 2018a) robot poses (Gehring et al., 2021). Attempls at learning hierarchies directly from sparse rewards have had limited success (Vezhnevets et al., 2017) and required providing task reward to the low-level controller, calling into question the benefit of their high-level controller.
In this paper, we present Director, practical method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. We observe the effectiveness of Director on ong-horizoon tasks with very sparse rewards and demonstrate its generality by learning successfully in a wide range of domains. The key insights of Director are to leverage the representa- tions of the world model, select goals in a compact discrete space to aid learning for the high-level exploration in the high-level policy.
policy, and to use a simple form of temporally-cxtendee
Contributions The key contributions of this paper are summarized as follows:
We describe practical, general, and interpretable algorithm for learning hierarchical behaviors within a world model trained from pixels, which we call Director (Section 2).
We introduce two sparse reward benchmarks that underscore the limitations of traditional flat RL approaches and find that Director solves these challenging tasks (Section 3.1).
We demonstrate that Director successfully learns in wide range of traditional RL environments, including Atari, Control Suite, DMLab, and Crafter (Section 3.2).
We visualize the latent goals that Director selects for breaking down various tasks, providing insights into its decision making (Section 3.3).
Director
Director is reinforcement learning algorithm that learns hierarchical behaviors directly from pixels. As shown in Figure 2, Director learns world model for representation learning and planning, goal autoencoder that discretizes the possible goals to make them easier for the manager to choose, manager policy that selects goals every fixed number of steps to maximize task and exploration rewards, and a worker policy that learns to reach the goals through primitive actions. All components optimized throughout the course of learning by performing one gradient step every fixed number of environment steps. The world model is trained from a replay buffer whereas the goal autoencoder trained on the world model representations and the policies are optimized from imagined rollouts. For the pseudo code of Director, refer to Appendix E.
2.1 World Model
Director learns world model that compresses the history of observations into a compact feature space and enables planning in this space (Watter et al., 2015; Zhang et al., 2019). We use the Recurrent State Space Model (RSSM) model of PlaNet (Hafner al., 2018), which we briefly review here to introduce notation. The world model consists of four neural networks that are optimized jointly:
Model representation:
reprg(5e 8t-1, (t-1,4;-1,37)
Model decoder:
reco(s:) Rs Wt.
rewa(s ~ (1)
Model dynamics:
dyng(Se 9t-1,a,-11
Reward predictor:

---

mgr
dec
7
enc
dec
wkr
a
wkrr
wkr
F
T2 T
rерr
repr
g
g
8
Figure 2: Director is based on the world model of PlaNet (Hafner et al., 2018) that predicts ahead in compact representation space. The world model is trained by reconstructing images using neural network not shown in the figure. Director then learns three additional components, which all optimized concurrently. On the left, the goal autoencoder compresses the feature vectors 8, into vectors of discrete codes ~ enc(= 8t). On the right, the manager policy mgr(= st) selects abstract actions in this discrete space every K steps, which the goal decoder then turns into feature space goals = dec(z). The worker policy wkr(a, $t,9) receives the current feature vector and goal as input to decide primitive actions that maximize the similarity rewards rr to the goal. The manager maximizes the task reward and an exploration bonus based on the autoencoder reconstruction error, implementing temporally-exee exteeded exploration.
The representation model integrates actions a, and observations 24 into the latent states 8t. The dy- namics model predicts future states without the corresponding observations. The decoder reconstructs observations to provide rich learning signal. The reward predictor later allows learning policies by planning in the compact latent space, without decoding images. The world model is optimized end-to-end on subsequences from the replay buffer by stochastic gradient descent the variational objective (Hinton and Van Camp, 1993; Kingma and Welling, 2013; Rezende et al., 2014):
T
((0)
( KL ( st-1, ac-1, || dyng(s: st-1, ax-)
+ (s)) 22112 + (rewo(se+1) - r2)²) where 81:7 repro
The variational objective encourages learning Markovian sequence of latent states with the following properties: The states should be informative of the corresponding observations and rewards, the dynamics model should predict future states accurately, and the representations should be formed such that they are easy to predict. The hyperparameter 3 trades off the predictability of the latent states with the reconstruction quality (Beattie et al., 2016; Alemi et al., 2018).
2.2 Goal Autoencoder
The world model representations
1024-dimensional continuous vectors. Selecting such representations as goals would challenging for the manager because this constitutes a very high- dimensional continuous action space. To avoid high-dimensional continuous control problem for the manager, Director compresses the representations into smaller discrete codes = using a goal autoencoder that is trained on replay buffer model states from Equation 2:
Goal Encoder: encs(z (x))
Goal Decoder: dec,(=)
Simply representing each model state by class in one large categorical vector would require roughly one category per distinct state in the environment. It would also prevent the manager from generalizing between its different outputs. Therefore, we opt for a factorized representation of multiple categoricals. Specifically, we choose the vector of categoricals approach introduced in DreamerV2 (Hafner et al., 2020a). As visualized in Figure G. I, the goal encoder takes model state as input and predicts a matrix of 8x8 logits, samples a one-hot vector from each row, and flattens the results into sparse vector with 8 out of 64 dimensions set to I and the others to O. Gradients

---

backpropagated through the sampling by braighh-through estimation (Bengio et al., 2013). The goal autoencoder is optimized end-to-end by gradient descent on the variational objective:
( = dec - ss 1|2 3 KL [enca(2 st) || p(x)] where ~~ encs(2 st)
(4)
The first term a mean squared error that encourages the encoder to compute informative codes from which the input can be reconstructed. The second term encourages the encoder to use all available codes by regularizing the distribution towards a uniform prior p(z). The autoencoder is trained at the same time as the world model but does not contribute gradients to the world model.
2.3 Manager Policy
Director learns a manager policy that selects new goal for the worker every fixed number of K 8 time steps. The manager is free to choose goals that are much further than 8 steps away from the current state, and in practice, it often learns to choose the most distant goals that the worker is able to achieve. Instead of selecting goals in the high-dimensional continuous latent space of the world model, the manager outputs abstract actions in the discrete code space of the goal autoencoder (Section 2.2). The manager actions are then decoded into world model representations before they are passed on to the worker as goals. To select actions in the code space, the manager outputs a vector of categorical distributions, analogous to the goal encoder in Section 2.2:
Manager Policy: o (t))
The objective for the manager to maximize the discounted sum of future task rewards and ex- ploration rewards. The exploration encourages the manager to choose novel goals for the worker, resulting in temporally-abstrrr so without explicit drive to expand the state distribution, it could prefer going back to previously common states it has been trained on the most, and thus hinder exploration of new states in the environment. Because the goal autoencoder is trained from the replay buffer, it tracks the current state distribution of the agent and we can reward novel states as those that have a high reconstruction error under the goal autoencoder:
exploration. This is important because the worker is goal-conditioned,
expl doco(2) st+1 || where ~~ enog(2 82+1)
(6)
Both manager and worker policies are trained from the same imagined rollouts and optimized using Dreamer (Hafner et al., 2019; 2020a), which we summarize in Appendix H. The manager learns two state-value critics for the extrinsic and exploration rewards, respectively. The critics are used to fill in rewards beyond the imagination horizon and as baseline for variance reduction (Williams. 1992). We normalize the extrinsic and exploration returns by their exponential moving standard deviations before summing them with weights extr = 1.0 and expl = 0.1. For updating the manager, the imagined trajectory is temporally abstracted by selecting every K-th model state and by summing rewards within each non-overlapping subsequence of length K. No off-policy correction (Schulman et al., 2017; Nachum et al., 2018a) is needed because the imagined rollouts are on-policy.
2.4 Worker Policy
The worker is responsible for reaching the goals chosen by the manager. Because the manager outputs codes in the discrete space of the goal autoencoder, first decode the goals into the state space of the world model g = dec(z). Conditioning the worker on decoded goals rather than the discrete codes has the benefit that its learning becomes approximately decoupled from the goal autoencoder. The worker policy is conditioned on the current state and goal, which changes every K = 8 time steps, and it produces primitive actions a, reach the feature space goal:
Worker Policy:
wkrę(a; | s(,
To reach its latent goals, we need to choose a reward function for the worker that measures the similarity between the current state &; and the current goal g, both of which are d02e-simall continuous activation vectors. Natural choices would be the negative L2 distance or the cosine similarity and their choice depends on the underlying feature space, which is difficult to reason about. Empirically, we found the cosine similarity to perform better. Cosine similarity would usually normalize both vectors, but normalizing the state vector encourages the worker to remain near the origin of the latent space, so that it can quickly achieve any goal by moving a small amount in the

---

Method
Task
Observations
Training
Evaluation
Proprioception
Random XY goals, dense L2 reward
Fixed XY goal, success rate
HIRO
Goal XY pos
Robot XY pos
Proprioception
Random XY goals, dense L2 reward
Fixed XY goal, success rate
NORL
Goal XY pos
Top view (5x5)
Proprioception First-person
End-to-end reinforcement learning, sparse reward when the robot touching the goal
This
paper
camera (64x64x3)
Figure 3: Comparison of Ant Mazes in the literature and this paper. HIRO (Nachum et al., 201 8a) provided global XY coordinates of the goal and robot position to the agent and trained with dense rewards on uniformly sampled training goals. NORL (Nachum et al., 2018b) replaced the robot XY position with global top-down view of the environments, downsampled to 5x5 pixels. this paper, tackle the global information provided to the agent, and only give a sparse reward for time steps where the robot the goal. To succeed at these tasks, agent has autonomously explore the environment and identify landmarks to localize itself and navigate the mazes.
challenging problem of learning directly from egocentric camera without
right direction. We thus incorporate the idea that the magnitude of both vectors should be similar into the cosine similarity, resulting in our max-cosine reward:
goal = (g/m) (st+2/m) where = max [Illl, ||sc+1ll)
When the state and goal vectors are of the same magnitude, the reward simplifies to the cosine similarity. When their magnitudes differ, the vectors both normalized by the larger of the two magnitudes, and thus the worker receives a down-scaled cosine similarity as the reward. As a result, the worker is incentivized to match the angle and magnitude of the goal. Unlike the L2 similarity, the reward scale of our goal reward is not affected by the scale of the underlying feature space.
The worker maximizes only the goal rewards. We make this design choice to demonstrate that the interplay between the manager and the worker successful across many environments, although we also include an ablation experiment where the worker additionally receives task reward, which further improves performance. The worker optimized by Dreamer with goal-conditioned critic. For updating the worker, we cut the imagined rollouts into distinct trajectories of length K within which the goal is constant. The state-critic estimates goal rewards beyond this horizon under the same goal, allowing the worker to learn to reach far-away goals.
value
Experiments
We evaluate Director on challenging benchmark suites with visual inputs and very sparse rewards, which we expect to be challenging to solve using a flat policy without hierarchy (Section 3.1). We further evaluate Director a wide range of standard tasks from the literature to demonstrate its generality and ensure that the hicrarchy is not harmful in simple settings (Section 3.2). We use fixed set of hyperparameters not only across tasks but also across domains, detailed in Table F.I. Finally, we offer insights into the learned hierarchical behaviors by visualizing the latent goals selected during environment episodes (Section 3.3). Ablations and additional results are included in the appendix.
Implementation We implemented Director on top of the public source code of DreamerV2 (Hafner et al., 2020a), reusing its default hyperparameters. We additionally increased the number of environ- ment instances to 4 and set the training frequency to one gradient step per 16 policy steps, which drastically reduced wall-clock time and decreased sample-efficiency mildly. Implementing Director the code base amounts to about 250 lines of code. The computation time Director is 20% longer than that of DreamerV2. Each training run used single V100 GPU with XLAA and mixed precision enabled and completed in less than 24 hours. All our agents and environments will be open sourced upon publication to facilitate future research in hierarchical reinforcement learning.
5

---

SI
Ant Maze S
Ant Maze
Ant Maze L
Ant Maze XL
3200
2400
2400
2400
2400
1800
1800
1600
1600
1200
1200
800
800
600
600
0
0
3M
6M
9M
3M
6M
9M
0
3M
6M
9M
3M
6M
9M
Director
Plan2Explore (flat)
Dreamer (flat)
Figure 4: Egocentric Ant Maze benchmark. A quadruped robot is controlled through joint torques to navigate to a fixed location a 3D maze, given only first-person camera and proprioceptive inputs. This is in contrast to prior benchmarks where the agents received their global XY coordinate or top-down view. The only reward is given at time steps where the agent touches the reward object. Plan2Explore fails in the small maze because the robot flips over much, common limitation of low-level exploration methods. Director solves all four tasks by breaking them down into manageable subgoals that the worker can reach, while learning in the end-to-end reinforcement learning setting.
Baselines To fairly compare Director to the performance of non-hierarchical methods, we compare the DreamerV2 agent on all tasks. DreamerV2 has demonstrated strong performance on Atari (Hafner et al., 2020a), Crafter (Hafner, 2021) and continuous control tasks (Yarats et al.. 2021) and outperforms top model-free methods in these domains. We kept the default hyperparameters that the authors tuned for DreamerV2 and did not change them for Director. In addition to its hierarchical policy, Director employs an exploration bonus at the top-level. To isolate the effects of hierarchy and exploration, we compare to Plan2Explore (Sekar et al., 2020), which uses ensemble disagreement of forward models as a directed exploration signal. We combined the extrinsic and exploration returns after normalizing by their exponential moving standard deviation with weights 1.0 and 0.1, in Director. We found Plan2Explore to be effective across both continuous and discrete control tasks.
3.1 Sparse Reward Benchmarks
Egocentric Ant Mazes Learning navigation tasks directly from joint-level control has been a long-standing milestone for reinforcement learning with sparse rewards, commonly studied with quadruped robots maze environments (Florensa et al., 2017: Nachum et al., 2018a). However, previous attempts typically required domain-specific inductive biases to solve such tasks, such as providing global XY coordinates to the agent, easier practice goals, and a ground-truth distance reward, summarized in Figure 3. In this paper, we instead attempt learning directly from first- person camera inputs, without privileged information, and a single sparse reward that the agent receives while in the fixed goal zone. The control frequency is 50Hz and episodes end after time limit of 3000 steps. There are no early terminations that could leak inkorrffonttoon to the agent (Laskin al., 2022). To help the agent localize itself from first-person observations, assign different colors to different walls of the maze.
As shown in Figure 4, we evaluate the agents in four mazes that span varying levels of difficulty. Because of the sparse reward, the episode returns correspond to the number of time steps for which the agent remains at the goal after reaching it. Curves show the mean and standard deviation across independent seeds. We find that the flat Dreamer agent succeeds at the smallest of the four mazes, and the flat exploration policy Plan2Explore makes some initial learning progress but fails to converge the optimal solution. Inspecting the trajectories revealed that Plan2Explore chooses too chaotic actions that often result in the robot flipping over. None of the baselines learn successful behaviors in the larger mazes, demonstrating that the benchmark pushes the limits of current approaches. In contrast, Director discovers and learns to reliably reach the goal zone at all four difficulty levels, with the larger maze taking longer to master.
Visual Pin Pads The Pin Pad suite of environments is designed to evaluate an agent's ss ability to explore and assign credit over long horizons, isolated from the complexity of 3D observations or

---

Pin Pad Three
Pin Pad Four
Pin Pad Five
Pin Pad Six
600
240
120
240
160
80
160
200
80
80
0
0
40
2M
4M
6M
2M
4M
6M
0
2M
4M
6M
2M
4M
6M
Director
Plan2Explore (fiat)
Dreamer (flat)
Figure 5: Visual Pin Pad benchmark. The agent controls the black square to move in four directions. Each environment has a different number of pads that be activated by walking and stepping on them. A single sparse reward is given when the agent activates all pads in the correct sequence. The history of previously activated pads is shown the bottom of the screen. Plan2Explore uses low-level exploration and performs well in this environment, but struggles for five and six pads, which requires more abstract exploration and longer credit assignment. Director learns successfully across all these environments, demonstrating its benefit on this long-horizon benchmark.
sophisticated joint-level control. As shown Figure 5, each environment features a moving black square that the agent control in all four directions and fixed pads of different colors that the agent can activate by walking over to the pad and stepping on it. The task requires discovering the correct sequence of activating all pads, which point the agent receives sparse reward of 10 points and the agent position randomly reset. Episodes last for 2000 steps and there are no intermediate rewards for activating the pads. To remove the orthogonal challenge of learning long-term memory (Gregor et al., 2019), the history of previously activated pads is displayed at the bottom of the image.
The agent performance is shown Figure which displays mean and standard deviation across five independent seeds. The easiest environment contains three pads, so the agent only has to decide whether to activate the pads in clockwise or counter-clockwise sequence. The flat Dreamer agent sometimes discovers the correct sequence. The exploration bonus of Plan2Explore offers significant improvement over Dreamer on this task. Dreamer fails to discover the correct sequence the harder environments that contain more pads. Plan2Explore still achieves some reward with four pads, struggles with five pads, and completely fails with six pads. contrast, Director discovers the correct sequence in all four environments, demonstrating the benefit of hierarchy over flat exploration.
3.2 Standard Benchmarks
To evaluate the robustness of Director, we train on wide range of standard benchmarks, which typically require no long-term reasoning. We choose Atari games (Bellemare et al., 2013), the Control Suite from pixels (Tassa et al., 2018), Crafter (Hafner, 2021), and tasks from DMLab (Beattie et al., 2016) to cover a spectrum challenges, including continuous and discrete actions and 2D and 3D environments. We compare two versions of Director. In its standard variant, the worker learns purely from goal rewards. This tests the ability of the manager to propose successful goals and the ability of the worker to achieve them. In the second variant, the worker learns from goal and task returns with weights 1.0 and 0.5, allowing the worker to fill in low-level details in a task-specific manner, which Director not perform
the manager may be too coarse-grained to provide. Ideally, would like worse than Dreamer on any task when giving task reward to the worker.
The results of this experiment are summarized Appendix A due to space constraints, with the full training curves for Atari and the Control Suite included in Appendices and K. We observe that Director indeed learns successfully across many environments, showing broader applicability than most prior hierarchical reinforcement learning methods. In addition, providing task reward to the worker is not as important as expected - hierarchy solves wide range tasks purely by following goals at the low level. Additionally providing task reward to the worker completely

---

Episode
Goal
T=0
T=
T=50
T=
T=100
T=125
T=150
Episode
Goal
b-19
5-19
3-197
9-19
5-67
T=0
T=25
T=50
T=75
T=100
T=125
T=150
Episode
Goal
T=0
T=25
T=50
T=75
T=100
T=125
T=150
Figure 6: Subgoals discovered by Director. For interpretability. decode the latent goals into images using the world model. Top: In DMLab, the manager chooses goals of the teleport animation that occurs every time the agent has collected the reward object, which can be seen the episode at time step 125. Middle: In Crafter, the manager first directs the worker via the inventory display to collect wood and craft pickaxe and sword, with the worker following command. The worker then suggests cave via the terrain image to help the worker find stone. As night breaks, it suggests hiding from the monsters detailed leg movements by suggesting forward leaning pose and a shifting floor pattern, with the worker successfully filling in the joint movements. Fine-grained subgoals are not required for this task, because the horizon needed for walking is short enough for the worker. Videos are available on the project website: https: : /danni//aaa//////// jar. www.cooo/iii.i.eeeeee com/director////
cave on small island. Bottom: In Walker, the manager abstracts away the
closes the gap to the state-of-the-art DreamerV2 agent. Figure K.1 the appendix further shows that Director achieves higher human-normalized median score than Dreamer on the 55 Atari games.

---

3.3 Goal Interpretations
To gain insights into the decision making of Director, we visualize the sequences of goals it selects during environment interaction. While the goals are latent vectors, the world model allows us to decode them into images for human inspection. Figures 1 and 6 show episodes with the environment frames (what the agent sees) at the top and the decoded subgoals (what the manager wants the worker to achieve) the bottom. Visualizations for additional environments
included in Appendix L.
Ant Maze M The manager chooses goals that direct the agent through the different sections of the maze until the reward object is reached. We also observed that initially, the manager chooses more intermediate subgoals whereas later during training. the worker learns to achieve further away goals and thus the manager can select fewer intermediate goals.
DMLab Goals Small The manager directs the worker to the teleport animation, which occurs every time the agent collects the reward object and gets teleported to a random location in the 3D maze. Time step T = 125 shows example of the worker reaching the animation in the environment. Unlike the Ant Maze benchmark, navigating to the goal in DMLab requires no joint-level control and is simple enough for the worker to achieve without fine-grained subgoals.
Crafter The manager requests higher inventory counts of wood and stone materials, as well as wooden tools, via the inventory display at the bottom of the screen. It also directs the worker to cave to find stone and coal but generally spends less effort on suggesting what the world around the agent should look like. As night breaks, the manager tasks the worker with finding small cave or island to hide from the monsters that are about to spawn.
Walker Walk The manager abstracts away the detail of leg movement, steering the worker using a forward leaning pose with both feet off the ground and a shifting floor pattern. While the images cannot show this, the underlying goal vectors are Markovian states that can contain velocities, so it is likely that the manager additionally requests high forward velocity. The worker fills in the details of standing up and moving the legs pass through the sequence of goals.
Across domains, Director chooses semantically meaningful goals that are appropriate for the task, despite using the same hyperparameters across all tasks and receiving no domain knowledge.
Related Work
Approaches hierarchical reinforcement learning include learning low-level policies collection of easier pre-training tasks (Heess et al., 2016; Tessler et al., 2017: Frans et al., 2017; Rao et al., 2021; Veeriah et al., 2021), discovering latent skills via mutual-information maximization (Gregor et al., 2016; Florensa et al., 2017: Hausman et al., 2018; Eysenbach et al., 2018; Achiam et al., 201 8; Merel al., 2018; Laskin al., 2022: Sharma et al., 2019; Xie et al., 2020; Hafner al., 2020b; Strouse et al., 2021), or training the low-level as a goal-conditioned policy (Andrychowicz et al., 2017; Levy al., 2017; Nachum al., 201 Sa;b; Co-Reyes al., 2018; Warde-Farles
al., 2018;
Nair et al., 2018; Pong et al., 2019; Hartikainen et al., 2019; Gehring et al., 2021; Shah et al., 2021). These approaches are described in more detail in Appendix I.
Relatively few works have demonstrated successful learning of hierarchical behaviors directly from pixels without domain-specific knowledge, such as global XY positions, manually specified pre- training tasks, or precollected diverse experience datasets. HSD-3 (Gehring et al., 2021) showed transfer benefits for low-dimensional control tasks. HAC (Levy et al., 2017) learned interpretable hierarchies but required semantic goal spaces. FuN (Vezhnevets et al., 2017) learned two-level policy where both levels maximize task reward and the lower level is regularized by goal reward but did not demonstrate clear benefits over an LSTM baseline (Hochreiter and Schmidhuber, 1997). We leverage explicit representation learning and semporally-abstraal exploration, demonstrate substantial benefits over flat policies on sparse reward tasks, and underline the generality of our method by showing successful learning without giving task reward to the worker across many domains.
Discussion
We present Director, a reinforcement learning agent that learns hierarchical behaviors from pixels by planning in the latent space of a learned world model. To simplify the control problem for the manager, we compress goal representations into compact discrete codes. Our experiments demonstrate the effectiveness of Director on two benchmark suites with very sparse rewards from pixels. We also show that Director learns successfully across a wide range of different domains without giving task

---

reward to the worker, underlining the generality of the approach. Decoding the latent goals into images using the world model makes the decision making of Director more interpretable and we observe it learning a diverse range of strategies for breaking tasks down into subgoals.
Limitations and future work We designed Director to be as simple as possible, at the expense of design choices that could restrict its performance. The manager treats its action space as a black box, without any knowledge that its actions correspond to states. For example, one could imagine regularizing the manager to choose goals that have a high value under its learned critic. Lifting the assumption of changing goals every fixed number of time steps, for example by switching based separate classifier or as soon as the previous goal has been reached, could enable the hierarchy to adapt to the environment and perform better tasks that require precise timing. Moreover, goals are points in latent space, whereas distributional goals or masks would allow the manager to only specify targets for the parts of the state that currently relevant. Learning comporally-abstrace dynamics would allow efficiently learning hierarchies of more than two levels without having to use exponentially longer batches. We suspect that these ideas will improve the long-term reasoning of the agent. Besides improving the capabilities of the agent, we see disentangling the reasons for why Director works well, beyond the ablation experiments in the appendix, as promising future work.
Acknowledgements We thank Volodymyr Mnih, Michael Laskin, Alejandro Escontrela, Nick Rhinehart, and Hao Liu for insightful discussions. We thank Kevin Murphy, Ademi Adeniji, Olivia Watkins, Paula Gradu, and Younggyo Seo for feedback on the initial draft of the paper.
References
J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. arXiv 2018
preprint
M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan,
K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01699
2022.
A. Alemi, B. Poole, I. Fischer, J. Dillon, R. A. Saurous, and K. Murphy. Fixing broken elbo. In International Conference on Machine Learning, pages 159- 168. PMLR, 2018.
M. Andrychowicz. F. Wolski, A. Ray. J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 308.3432, 2013.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv: 11..11000...9. 2018.
A. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from" -n-the-wild" human videos. arXiv preprint arXiv: 10l.:/0.3..6/7., 2021.
J. Co-Reyes, Y. Liu, A. Gupta, B. Eysenbach, P. Abbeel, and S. Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In International Conference on Machine Learning, pages 1009-1018. PMLR, 2018.
Dayan and G. Hinton. Feudal reinforcement learning. Advances neural information processing proceesssinnnoocssinn systems, 5, 1992.
10

---

B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: learning skills without reward function. arXiv preprint arXiv:1802.06070, 2018.
C. Florensa, Y. Duan, and Abbeel. Stochastic neural networks for hierarchical reinforcement learning, arXiv preprint arXiv: (a0d:::7704000222 2017.
K. Frans. J. Ho, X. Chen, P. Abbeel, and J. Schulman. Meta learning shared hierarchies. arXiv preprint arXiv:1710.09760 09767, 2017.
Gehring, G. Synnaeve, A. Krause, and N. Usunier. Hierarchical skills for efficient exploration. Advances in Neural Information Processing Systems, 34, 2021.
K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint 1rX1v.1611.07507, 2016.
K. Gregor. D. Jimenez Rezende, Besse, Y. Wu, H. Merzic, and van den Oord. Shaping belief states with generative environment models for rl. Advances in Neural Information Processing Systems, 32, 2019.
D. Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.0672 2021.
D. Hafner, T. Lillierap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 0rXiv:/811.04555 2018.
D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02/93, 2020a.
D. Hafner, A. Ortega, J. Ba, T. Parr, K. Friston, and N. Heess. Action and perception as divergence minimization. arXiv preprint arXiv:2009.01791, 2020b.
K. Hartikainen, X. Geng, T. Haarnoja, and S. Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprint arXiv:1907.0822 1907.08255. 2019.
K. Hausman, T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embedding space for transferable robot skills. International Conference Learning Representations, 2018.
N. Heess, G. Wayne, Y. Tassa, T. Lillicrap, M. Riedmiller, and D. Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv: 110iv:1610.051822, 2016.
G. E. Hinton and D. Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference Computational learning theory, pages 5-13, 1993.
Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98)) 1997.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint ar -xxx:22007.00::5 2021.
M. Laskin, H. Liu, X. B. Peng, D. Yarats, A. Rajeswaran, and P. Abbeel. Cie: Contrastive intrinsic control for unsupervised skill discovery, arXiv preprint arXiv:2202.00161. 2022.
A. Levy, G. Konidaris, R. Platt, and K. Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint 1711v:1712.00942 2017.
R. Mendonca, O. Rybkin, K. Daniliddis, D. Hafner, and D. Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 34, 2021.
Merel, Ahuja, V. Pham, S. Tunyasuvunakool, Liu, Tirumala, N. Heess, and Wayne. Hierarchical visuomotor control of humanoids. arXiv preprint arXiv:1811.09656 2018.
11

---

V. Mnih, K. Kavukcuoglu. D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland. G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. arXiv preprint ar Xrv: 180.:vv::8..00mm 08296, 2018a.
O. Nachum, S. Gu, H. Lee, and S. Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint 180.011 2018b.
A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018.
R. Parr and Russell. Reinforcement learning with hierarchies of machines. Advances in neural information processing systems, 10, 1997.
V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint 1rxiv:/903 1903.0399, 2019.
D. Rao, F. Sadeghi, I.. Hasenclever, M. Wulfmeier, M. Zambelli, G. Vezzani, D. Tirumala, Y. Aytar, Merel, N. Heess, et Learning transferable motor skills with hierarchical latent mixture policies. arXiv preprint arXiv:2 /2.050622 2021.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint 1rXiv:1401.4000 2014.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv:1707.000 1707.063477 2017.
R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning explore via self-supervised world models. arXiv preprint arXiv:2005.05960, 2020.
D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine. Rapid exploration for open-world navigation with latent goal models. arXiv preprint arXiv:2104.05800 2021.
A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills, arXiv preprini arXiv: 00000000777,..00 2019.
D. Silver, J. Schritwieser,, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):
354, 2017.
D. Strouse, K. Baumli, D. Warde-Farley, V. Mnih, and S. Hansen. Learning more skills through optimistic exploration. arXiv preprini
2021.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 11((1-2):181-111 1999.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefraneq, et al. Deepmind control suite. arXiv preprint arXiv: 1rxiiv:8011.000 006900 2018.
C. Tessler, S. Givony, T. Zahavy, D. Mankowitz, and S. Mannor. deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI Conference on Artificial Intelligence volume 31, 2017.
V. Veeriah, T. Zahavy, M. Hessel, Z. Xu, J. Oh, I. Kemaev, H. P. van Hasselt, D. Silver, and S. Singh. Discovery options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34, 2021.
A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549. PMLR, 2017.
D. Warde-Farley, Van de Wiele, T. Kulkarni, C. Jonescu, S. Hansen, and V. Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint ar arXiv: 2018.
12

---

M. Watter, J. Springenberg. J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pages 7756,2754, 2015.
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229 256, 1992.
K. Xie, H. Bharadhwaj. D. Hafner, A. Garg, and F. Shkurti. Skill transfer via partially amortized hierarchical planning. arXiv preprint arXiv:2011.13897, 2020.
D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augeetted reinforcement learning. arXiv preprint arXiv:2107.09644 2021.
M. Zhang, S. Vikram, Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: deep structured representations for model-basee reinforcement learning, In International Conference on Machine Learning, 2019.
13

---

Societal Impact
Developing hierarchical reinforcement learning methods that are useful for real-world applications will still require further research. However, in the longer term future, it has the potential to help humans automate more tasks by reducing the need to specify intermediate rewards. Learning autonomously from sparse rewards has the benefit of reduced human effort and less potential for reward hacking, but reward hacking is nonetheless a possibility that will need to be addressed once hierarchical reinforcement learning systems become more powerful and deployed around humans.
As reinforcement learning techniques become deployed in real-worid environments, they will be capable of causing direct harm. intentional or unintentional. Director does not attempt directly address such safety issues, but the ability to decode its latent goals into images for human inspection provides some transparency to the decisions the agent making. This transparency may permit auditing of failures titer-the-fact, and may additionally support interventions in some situations, if a human or automated observer in position monitor the high-level goals Director generates.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] As stated in
the text, we will provide code and training curves upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Table F.1.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Figures 4, 5, A, J.1 and K.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rxjnLkCVb38", "reviewer": "Reviewer_q27r", "summary": "Summary: Learn sub-policies by planning in the latent space of a learned dynamics model, where the low level policy performs goal-based RL, and the high level policy uses exploration and task rewards. It takes pixels as input, learns a latent world model, and then performs planning in the latent space of the world model. The world model is learned using a network dynamics modeler RSSM, from PlaNet, which is a variational pixel and rewards reconstruction algorithm. In order to reduce the dimensionality of the action space for the high-level controller, the algortihm uses a second encoder to encode into action space, which is the same as a vector of categoricals method used in Dreamer V2, which uses a variational reconstruction of the world model space. The exploration reward is based on reconstruction error (which mirrors work in curiosity rather than count-based methods), and combined with the extrinsic reward through  magic values. The worker policy uses a goal conditioned policy with a shaped max-cosine reward. The overall algorithm is evaluated on a large suite of common RL tasks including Atari, control suite, etc. against flat baselnes. ", "questions": "What fundamental insights did implementing this give towards hierarchical RL? \nHow would semantic hierarchical methods compare against this as an upper bound, or does working from pixels provide an advantage.\nWhat hindered the standard baselines from matching current flat methods?\nHow difficult was this to tune? What percentage of the development time was finding a good way to hook up the components?\n", "limitations": "This work does not describe the societal impact of an algorithm like this, though that description would be a philosophical exercise in whether better AI is a good thing. They also do not state clearly how the different components of Director could be interchanged, specifically the model-learning component, the model-based RL component, the exploration component and the low level goal reaching component, which would give more insight into the limitations and future work than describing a few edge cases.\nI think the biggest limitations were in having a clear message of what the contribution of this work is, and that it is not clear whether this method is truly hierarchical, or just a complex way to learn flat policies with model-based RL.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThis work describes a straightforward concept well, utilizing prior work in model-based RL, goal-based RL and exploration to contruct a hierarchical RL algorithm. The stated goal of creating a simple algorithm is executed well, and evidenced by the ability to test the algorithm on a wide variety of different domains. On top of that, the design choices are generally very reasonable, with the model-based being used both for exploration and to generate a pre-latent space for the mutual-information like latent skills, and the low level policy being goal-following.\nConsidering the difficulty of running hierarchical RL algorithms, which incorporate the combination of a large number of different ideas functioning simultaniously, this work is an impressive achievement engineering, with finding the right combination of methods to get generally improved performance on a wide variety of tasks. The fact that the latent space can also be visualized because of the world model is a nice perk, and helps alleviate some questions related to whether the method actually utilizes an efficient division of labor.\nWeaknesses: \nThe greatest disappointment with this paper is that it is more a feat of engineering than providing any significant insights into the way in which hierarchical RL can be run. Buried in the related work is the fact that feudal networks (Vezhnevets et. al. 2017), performed many of the same ideas with different design choices for generating the latent space and training the goal-based RL algorithm. While the design choices in this work appear to be superior, it is hard to tell directly from the experiments since there appear to be some gaps in terms of which tasks were tested on. As a result, the core insight of this paper appears to be that under these conditions we can combine feudal networks with dreamer-style model based RL. \nThis work utilizes a double-encoding which would have been interesting to investigate, at least in the appendix. While it makes sense to use a different embedding space for the world-model latent space (which needs all the information for reconstruction), and while the skill encoder finds a space that is a good size for learning, it would have been useful to get more information about some of the tradeoffs and design choices there. The ablation in the appendix only shows that it is necessary in some cases, not how the size of the goal autoencoder or other components scale. \nHyperparameters are always a significant question for hierarchical RL methods, and it isn't clear that all the (non-dreamer) hyperparameters are actually described in Appendix F, and more importantly, what the sensitivity is to those hyperparameters. It would also be nice to get the dreamer hyperparameters.\nThe Standard benchmark results (appendix A) is somewhat inconclusive. For one thing, 45M time steps is long compared to modern methods for atari games. As an example, Rainbow and PPO reach 400 reward (higher than the values given by Director and Dreamer), in 2M time steps (https://wandb.ai/tianshou/atari.benchmark/reports/Atari-Benchmark--VmlldzoxOTA1NzA5), which would equate to about 10x faster than the values given. Admittedly, Dreamer also has these issues, but as Breakout is the only atari task where Director has clear performance improvement over Dreamer, this puts into question performance in the whole domain. \nAnt-maze domains often struggle with a sort of low-variance issue, which means that a policy that explores well enough to reach the goal eventually can reach the goal again. This is not usually an issue, but because the atari and other benchmark results are somewhat inconclusive, this brings up the question of whether the performance benefit is just because of exploration (dreamer never reaches the goal to send return back from because it fails to perform good exploration). The Plan2Explore results don't really alleviate this because they never reach the goal either, and comparison with a semantic baseline, or a count-based exploration baseline, would have been interesting. It does not need to outperform, just demonstrate that it did well.\nThe qualitative subgoal results are somewhat weak, because it isn't clear in the researcher is inferring meaning onto the subgoals, or the subgoals are actually chosen such that there is significant division of labor between the low and high level controllers. For the most part, it looks like the manager just chooses one goal, and all of the actual work is carried out by the low level agent. This hypothesis is further supported by the fact that when extrinsic reward is provided to the low level agent, performance often improves. This is not always the cas , but it appears to be the case often enough to call into question whether the work is actually utilizing hierarchy. Ironically, the PIN pad domain, the toy domain that is somewhat underdescribed, is probably the best place to show this, since having subgoals that lie on each of the PIN pads would pretty much demonstrate the result. \nThis work does not compare with other hierarchical baselines, in particular feudal networks which implements largely the same algorithm. Normally, this would be a huge problem, but historically hierarchical methods are a huge pain to implement, so it's somewhat understandable. However, at least an attempt to use feudal networks would have been insightful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bdyObEErLWZ", "reviewer": "Reviewer_BArk", "summary": "This paper proposed a method for learning a hierarchical policy that operates from pixels, without pre-defined high level actions, and using model based RL\n\nThe policy is broken up into 4 key components. A world model that models environment dynamics, representation, and reward. A manger policy that selects goals in a discrete latent space. A goal autoencoder that decodes the select goal into representation space, and a worker policy that selects low level actions to take to achieve the manager's goal.\n\nThe proposed method is evaluated in two environments that stress sparsity of rewards and on standard benchmarks.\n\nIn the sparse reward benchmarks, the proposed method outperforms baselines. The gap to baselines increases as the environment complexity increases.\n\nOn the standard benchmarks, the proposed method matches Dreamer when the worker is trained with the task-specific reward in addition to the reward that encourages it to reach the manager's goal.", "questions": "See above", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe proposed method is able to learn a hierarchical policy with MBRL that does not require pre-specified high level actions. These high-level actions are simply environment states to reach, making them easily interpretable.\n\nThe proposed method matches state-of-the-art on standard benchmarks.\n\nThe proposed method works well with sparse rewards.\n\nThere are extensive ablations in the supplement.\n\nThe paper is well-written.\n\n### Weaknesses\n\nThe evaluation falls short of the initial promise in the intro. To the knowledge of the reviewer, none of the tasks studied have the length of time horizon promised in the intro. Thus it remains unclear if this method indeed works better in that promised setting. The proposed method is clearly helpful in settings with sparse rewards, so perhaps that should be the motivation and promise.\n\n### Suggestions for improvement\n\nAdding summary statistics for the main paper on the standard benchmarks would be very helpful. There is likely room for something that small. And part of Fig 6 could be moved if not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PZc4mYYIE_Z", "reviewer": "Reviewer_uwa5", "summary": "In this paper, the authors introduce a learnable RL-based planner. It is specifically design for long horizon sparse reward environment. The proposed planner consists of two major components. A world model uses an off-the-shelf encoder to represent the raw pixel input and additional goal autoencoder is trained to obtain a more sparse world representation. On the other hand, a manager policy and a worker policy are trained to predict the next goal and the next atomic action. The authors also designs exploration reward and goal reward to stimulate the agent. Experiments are conducted on public benchmarks including long horizon navigation and atari games.", "questions": "Q1: The authors may want to explain why the \"manager\" policy can be well trained with the async-trained world model decoder from a more theoretical perspective. To the best of my understanding, the autoencoder is trained by memory replay but the \"manager\" is on-policy. Given the fact the the world model is trained separately, how to guarantee that the \"z\" in world encoder share the same space with the \"z\" in the manager? The paper will be much stronger if there is a theoretical justification.\n\nQ2: There are two rewards to stimulate the agent. One is for exploration and the other one is for reaching the goals. Do these two rewards compete with each other? For example, the reward of exploring a goal state is larger than reaching it. How to avoid it?\n\nQ3: What is the magic behind changing the goal very k=8 step?\n\nQ4: Why Dreamer performs so bad in Ant Maze XL and Pin Pad Six?\n\nQ5: Any insights for the reason that the proposed method is weaker in Atari games?\n\nI've been away from this community for a while, therefore I will also consider other reviewers comments. I will rise my rating if the authors can provide insightful responses. Based on the submitted manuscript, I lean to accept.", "limitations": "See weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe overall presentation for the proposed method and experiments is friendly for audience to understand although there are some minor grammar issues. It is a challenging problem for RL agent to plan in long horizon and reward-sparse environment. According to some of the experiment results, the proposed method works very well (Ant Maze). The idea of using a more sparse representation to represent abstract action is interesting and the overall design of the proposed method is reasonable.\n\nWeaknesses:\nMost of the components used in the proposed method are off-the-shelf, but I don't think it is a big concern regarding the originality. The design of the \"manager\" mechanism is not very solid. See question below. Although the proposed method show promising results in some tasks, in general RL benchmarks it is slightly weaker than the popular baselines. It will be better to discuss more about it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept