Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors introduce a learnable RL-based planner. It is specifically design for long horizon sparse reward environment. The proposed planner consists of two major components. A world model uses an off-the-shelf encoder to represent the raw pixel input and additional goal autoencoder is trained to obtain a more sparse world representation. On the other hand, a manager policy and a worker policy are trained to predict the next goal and the next atomic action. The authors also designs exploration reward and goal reward to stimulate the agent. Experiments are conducted on public benchmarks including long horizon navigation and atari games.
strengths_and_weaknesses: Strength:
The overall presentation for the proposed method and experiments is friendly for audience to understand although there are some minor grammar issues. It is a challenging problem for RL agent to plan in long horizon and reward-sparse environment. According to some of the experiment results, the proposed method works very well (Ant Maze). The idea of using a more sparse representation to represent abstract action is interesting and the overall design of the proposed method is reasonable.

Weaknesses:
Most of the components used in the proposed method are off-the-shelf, but I don't think it is a big concern regarding the originality. The design of the "manager" mechanism is not very solid. See question below. Although the proposed method show promising results in some tasks, in general RL benchmarks it is slightly weaker than the popular baselines. It will be better to discuss more about it.
questions: Q1: The authors may want to explain why the "manager" policy can be well trained with the async-trained world model decoder from a more theoretical perspective. To the best of my understanding, the autoencoder is trained by memory replay but the "manager" is on-policy. Given the fact the the world model is trained separately, how to guarantee that the "z" in world encoder share the same space with the "z" in the manager? The paper will be much stronger if there is a theoretical justification.

Q2: There are two rewards to stimulate the agent. One is for exploration and the other one is for reaching the goals. Do these two rewards compete with each other? For example, the reward of exploring a goal state is larger than reaching it. How to avoid it?

Q3: What is the magic behind changing the goal very k=8 step?

Q4: Why Dreamer performs so bad in Ant Maze XL and Pin Pad Six?

Q5: Any insights for the reason that the proposed method is weaker in Atari games?

I've been away from this community for a while, therefore I will also consider other reviewers comments. I will rise my rating if the authors can provide insightful responses. Based on the submitted manuscript, I lean to accept.
limitations: See weakness.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a method for learning a hierarchical policy that operates from pixels, without pre-defined high level actions, and using model based RL

The policy is broken up into 4 key components. A world model that models environment dynamics, representation, and reward. A manger policy that selects goals in a discrete latent space. A goal autoencoder that decodes the select goal into representation space, and a worker policy that selects low level actions to take to achieve the manager's goal.

The proposed method is evaluated in two environments that stress sparsity of rewards and on standard benchmarks.

In the sparse reward benchmarks, the proposed method outperforms baselines. The gap to baselines increases as the environment complexity increases.

On the standard benchmarks, the proposed method matches Dreamer when the worker is trained with the task-specific reward in addition to the reward that encourages it to reach the manager's goal.
strengths_and_weaknesses: ### Strengths

The proposed method is able to learn a hierarchical policy with MBRL that does not require pre-specified high level actions. These high-level actions are simply environment states to reach, making them easily interpretable.

The proposed method matches state-of-the-art on standard benchmarks.

The proposed method works well with sparse rewards.

There are extensive ablations in the supplement.

The paper is well-written.

### Weaknesses

The evaluation falls short of the initial promise in the intro. To the knowledge of the reviewer, none of the tasks studied have the length of time horizon promised in the intro. Thus it remains unclear if this method indeed works better in that promised setting. The proposed method is clearly helpful in settings with sparse rewards, so perhaps that should be the motivation and promise.

### Suggestions for improvement

Adding summary statistics for the main paper on the standard benchmarks would be very helpful. There is likely room for something that small. And part of Fig 6 could be moved if not.
questions: See above
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Summary: Learn sub-policies by planning in the latent space of a learned dynamics model, where the low level policy performs goal-based RL, and the high level policy uses exploration and task rewards. It takes pixels as input, learns a latent world model, and then performs planning in the latent space of the world model. The world model is learned using a network dynamics modeler RSSM, from PlaNet, which is a variational pixel and rewards reconstruction algorithm. In order to reduce the dimensionality of the action space for the high-level controller, the algortihm uses a second encoder to encode into action space, which is the same as a vector of categoricals method used in Dreamer V2, which uses a variational reconstruction of the world model space. The exploration reward is based on reconstruction error (which mirrors work in curiosity rather than count-based methods), and combined with the extrinsic reward through  magic values. The worker policy uses a goal conditioned policy with a shaped max-cosine reward. The overall algorithm is evaluated on a large suite of common RL tasks including Atari, control suite, etc. against flat baselnes. 
strengths_and_weaknesses: Strengths: 
This work describes a straightforward concept well, utilizing prior work in model-based RL, goal-based RL and exploration to contruct a hierarchical RL algorithm. The stated goal of creating a simple algorithm is executed well, and evidenced by the ability to test the algorithm on a wide variety of different domains. On top of that, the design choices are generally very reasonable, with the model-based being used both for exploration and to generate a pre-latent space for the mutual-information like latent skills, and the low level policy being goal-following.
Considering the difficulty of running hierarchical RL algorithms, which incorporate the combination of a large number of different ideas functioning simultaniously, this work is an impressive achievement engineering, with finding the right combination of methods to get generally improved performance on a wide variety of tasks. The fact that the latent space can also be visualized because of the world model is a nice perk, and helps alleviate some questions related to whether the method actually utilizes an efficient division of labor.
Weaknesses: 
The greatest disappointment with this paper is that it is more a feat of engineering than providing any significant insights into the way in which hierarchical RL can be run. Buried in the related work is the fact that feudal networks (Vezhnevets et. al. 2017), performed many of the same ideas with different design choices for generating the latent space and training the goal-based RL algorithm. While the design choices in this work appear to be superior, it is hard to tell directly from the experiments since there appear to be some gaps in terms of which tasks were tested on. As a result, the core insight of this paper appears to be that under these conditions we can combine feudal networks with dreamer-style model based RL. 
This work utilizes a double-encoding which would have been interesting to investigate, at least in the appendix. While it makes sense to use a different embedding space for the world-model latent space (which needs all the information for reconstruction), and while the skill encoder finds a space that is a good size for learning, it would have been useful to get more information about some of the tradeoffs and design choices there. The ablation in the appendix only shows that it is necessary in some cases, not how the size of the goal autoencoder or other components scale. 
Hyperparameters are always a significant question for hierarchical RL methods, and it isn't clear that all the (non-dreamer) hyperparameters are actually described in Appendix F, and more importantly, what the sensitivity is to those hyperparameters. It would also be nice to get the dreamer hyperparameters.
The Standard benchmark results (appendix A) is somewhat inconclusive. For one thing, 45M time steps is long compared to modern methods for atari games. As an example, Rainbow and PPO reach 400 reward (higher than the values given by Director and Dreamer), in 2M time steps (https://wandb.ai/tianshou/atari.benchmark/reports/Atari-Benchmark--VmlldzoxOTA1NzA5), which would equate to about 10x faster than the values given. Admittedly, Dreamer also has these issues, but as Breakout is the only atari task where Director has clear performance improvement over Dreamer, this puts into question performance in the whole domain. 
Ant-maze domains often struggle with a sort of low-variance issue, which means that a policy that explores well enough to reach the goal eventually can reach the goal again. This is not usually an issue, but because the atari and other benchmark results are somewhat inconclusive, this brings up the question of whether the performance benefit is just because of exploration (dreamer never reaches the goal to send return back from because it fails to perform good exploration). The Plan2Explore results don't really alleviate this because they never reach the goal either, and comparison with a semantic baseline, or a count-based exploration baseline, would have been interesting. It does not need to outperform, just demonstrate that it did well.
The qualitative subgoal results are somewhat weak, because it isn't clear in the researcher is inferring meaning onto the subgoals, or the subgoals are actually chosen such that there is significant division of labor between the low and high level controllers. For the most part, it looks like the manager just chooses one goal, and all of the actual work is carried out by the low level agent. This hypothesis is further supported by the fact that when extrinsic reward is provided to the low level agent, performance often improves. This is not always the cas , but it appears to be the case often enough to call into question whether the work is actually utilizing hierarchy. Ironically, the PIN pad domain, the toy domain that is somewhat underdescribed, is probably the best place to show this, since having subgoals that lie on each of the PIN pads would pretty much demonstrate the result. 
This work does not compare with other hierarchical baselines, in particular feudal networks which implements largely the same algorithm. Normally, this would be a huge problem, but historically hierarchical methods are a huge pain to implement, so it's somewhat understandable. However, at least an attempt to use feudal networks would have been insightful.

questions: What fundamental insights did implementing this give towards hierarchical RL? 
How would semantic hierarchical methods compare against this as an upper bound, or does working from pixels provide an advantage.
What hindered the standard baselines from matching current flat methods?
How difficult was this to tune? What percentage of the development time was finding a good way to hook up the components?

limitations: This work does not describe the societal impact of an algorithm like this, though that description would be a philosophical exercise in whether better AI is a good thing. They also do not state clearly how the different components of Director could be interchanged, specifically the model-learning component, the model-based RL component, the exploration component and the low level goal reaching component, which would give more insight into the limitations and future work than describing a few edge cases.
I think the biggest limitations were in having a clear message of what the contribution of this work is, and that it is not clear whether this method is truly hierarchical, or just a complex way to learn flat policies with model-based RL.
review_title: null

Meta-review: metareview: This paper studies an interesting problem, and overall the reviewers agreed the exposition and validation are sufficient. We encourage the authors to consider the issues raised by the reviewers and further improve the work in the final version.
confidence: Less certain
Decision: Accept