Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes combining model-based RL with high-level skill learning and composition through hierarchical RL, into a single reinforcement learning framework. More specifically, the proposed approach leverages planning and composing skills in the low-dimensional, high-level representation, and learn low-level skills conditioned on the high-level skills. Only the low-level policies are executed in the environment to generate experiences. A mutual information objective is used to learn low-level policies conditioned on high-level skills, and this was shown to improve sample efficiency as the low-level policies do not learn to ignore the high-level skills they are conditioned on. 

The approach consists of multiple learned components working together. I wonder how stable learning of such dependent separate components is, and how much tuning is required. Can authors comment on the stability of learning, and how their system would perform under different hyperparameter values?

The experiments show LSP performing better or on par with Dreamer. On the transfer experiments, what guarantees do we have that transfer will always be beneficial? And what guarantees do we have that it will always be more beneficial than using Dreamer? I wonder if transfer was only beneficial in the 2 experiments shown? What about the other combination of tasks? 
review_title: SKILL TRANSFER VIA PARTIALLY AMORTIZED HIERARCHICAL PLANNING

final_score: rating: 7: Good paper, accept
review_content: The paper deals with learning reusable hierarchical skills in sequential decisions.
The authors introduce a partially amortized model based on four parametric functions (the representation module, observation module, latent forward dynamic, and task reward module) and an additive loss function composing these modules.
The overall algorithm consists of learning the four-modules jointly and to plan using MPC in the latent skill space.
The overall paper is well written and justified.
The experiments are done in the locomotion domain and show better or comparables results wrt to DREAM.
DADS is mentioned in the SoA section, it would have been interested to evaluate against it.
Furthermore, despite its computational complexity as mentioned in the discussion section, I would have been interested to have a comparison or discussion wrt [1].
Overall the paper is original and sufficiently empirically motivated for acceptance. 
I would request the code to be open-source for better reproducibility.

Refs:
[1] Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning, Levine and al, 2020
review_title: novel approach of transferable hierarchical planning

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose an extension of the Dreamer method (Hafner et al., 2019) to incorporate high-level planning via the cross-entropy method and temporally extended skill policies. In theory, this would allow the planning method to operate in an abstract, high-level space and delegate low-level action selection to policies trained on simulated rollouts. If I understand the paper correctly, the result of the LSP learning algorithm is the skill policy and a (fixed?) distribution over a skill variable; no planning is performed at test time. This only became clear when reading the section on further work, since the abstract states "For this, actions are produced by a policy that is learned over time while the skills it conditions on are chosen using online planning.". If there is indeed no planning during evaluation I would propose that "online planning" is not an apt description of the architecture. I general however, I like how the presented algorithm mixes MI-based skill discovery (which is typically unsupervised) with explicit supervision via long-horizon planning and would regard this a useful contribution to the current literature on hierarchical RL.

My rating is mainly influenced by the experimental section, as well as several minor issues with the overall presentation. The experiments show that the method is at least as good, and sometimes better, than Dreamer when training from scratch, and universally (although marginally) better when adapting to new tasks.  This is a good result, but some relevant details on the tasks and plots are missing, which make the achievements hard to interpret (see below). The difference to Dreamer is most pronounced on the "Quadruped Run Pixels" task, but as mentioned before it's not easy to judge the results; for example, in Figure 8 in the Dreamer paper they report scores of ~750 after 2e6 environment steps. It would also be helpful to provide reference numbers for a standard model-free algorithm on those tasks. The "Random Skills" baseline performs very poorly; does it still include the intrinsic rewards to ensure diversity of skills? What would happen if you would drop them so that you basically have Dreamer with an additional random skill input (cf. the "Switching Ensemble" baseline from [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://arxiv.org/abs/1909.10618))? Finally, the experiments don't give much insight into the nature of skills that have been discovered. In particular, what do the resulting skill distributions look like?

Several aspects of the experimental section are unclear to me and hinder the interpretation of the results:
- Several crucial details on the environments are missing, such as the reward functions. For "Quadruped Reach", is the goal constant or randomly selected for each episode?
- In Figure 3, what does the x-Axis represent?
- In Figure 3, tasks like "Quadruped Run", "WalkerRun Pixels" and "Quadruped Run Pixels" don't seem to be trained to convergence.
- There is a significant drop in performance at x=0.4 for "Quadruped Reach" for some seeds.
- In 4.4., the text for "Quadruped GetUp Walk -> Reach Goal" does not line up with the descriptions in 4.1. as it says the pretraining task would be to run as fast as possible rather than getting up and walking.
- In the same paragraph: "We see that LSP can adapt much quickly to the transfer task, achieving a reward of 500 only after 70, 000 steps, while Dreamer requires 130, 000 steps to achieve the same reward". From Figure 4 this appears to be the result for "Quadruped Walk -> Reach Goal", and in addition it's also worth noting that Dreamer catches up pretty soon afterwards (even though LSP maintains higher performance throughout training).
- For the transfer experiments, it would be useful to include the learning curves from Figure 3 so that it is apparent whether it is generally beneficial to adapt a pretrained model as opposed to learning from scratch.

A few more notes on the overall write-up:
- The site with videos linked to in the abstract is empty
- In the introduction, it's not immediately clear what "amortized" means even though it is mentioned several times. My interpretation is that it means that you use a learned model to train a policy rather than using the model during evaluation. The term could be described more explicitly; in particular I was slightly confused by the notion of "amortizing over the entire training experience".
- At the end of Section 5, regarding Dreamer: "[...], which is fast even for high-dimensional action spaces, but cannot directly transfer to new environments with different dynamics and reward functions." This is in contradiction to the transfer experiments in 4.4. which show a good amount of transfer performance?
- Figure 2b) and c): $\phi$ in text vs. $\Phi$ in figure.
- In section 3 the reference should be to Figure 2 rather than 2b)?
- In section 3, "Low level policy", it would be good to cite the Dreamer paper once more.
- In 3.1, "Behavior Learning.", the backwards skill predictor is mentioned but not introduced yet. Refer to section 3.2 for a definition?
- Section 3.2.: it would be good to relate the MI objective to previous methods. The resulting objective looks quite similar to [Variational Intrinsic Control](https://arxiv.org/abs/1611.07507).
- Section 5: DADS is mentioned, but it might be also be worth noting that they performed experiments with online planning on top of the discovered skills.
review_title: Interesting fusion of skill discovery and latent space planning with shortcomings in presentation in experimental section.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper presents a model-based RL approach that i) learns a high-level plan generator that produces a sequence of latent continuous variables as skills for a task through CEM, and ii) a low-level policy conditioned on the skills amortized policy training. The approach is shown to be more sample efficient than a recent model-base RL approach (Dreamer), and is able to learn to solve a new (but related) in simple walking tasks relatively faster as well.

Strengths:
+ The main idea (planning over low-level dimension and learning amortized low-level policy over high-dimension) is intriguing and makes a lot of sense, especially compared to alternative learning schemes such as purely CEM or purely amortized policy training.

+ The approach is well motivated and is easy to understand.

+ The results in simple tasks show promising results over existing approaches. 

+ I find the background and the discussion of the connection between existing model-based approaches and the proposed method very helpful.

Weakness:
- The high-level skill planner / generator p(z |\cdot) does not seem to rely on the physical states, which is different from typical HRL approaches where the option/subgoal at each step depends on the latest state, e.g., p(z^t | s^t). This seems to suggest that the plan of skills for a task is somewhat fixed regardless of the initial or immediate states. This may work when the environment is simple and does not vary a lot from episode to episode; but I can not see how this formulation can tackle more complex scenarios. Moreover, a fixed hyper-parameter 'K' seems to require additional tuning for new tasks and environments, which may make the approach less 'generalizable.'

- On a related note, it would be good to see more discussion on the advantages of the proposed skill generator/planner over the prior work on HRL. The related work section briefly mentions some prior work, but there is no detailed theoretical and/or empirical comparison. As of now, it is not clear to me how this is better than conventional HRL methods (e.g., learning a high-level policy).

- The main objective of the approach seems to be improving the transferability of skills learned by RL as stated in the main text and in the title. However, the transfer setting is quite simple (from 'learning to walking' to 'learning to walk to a goal location'). I would be more interested in seeing more difficult settings such as walking in new and complex environments (e.g., with obstacles). It is also not clear to me what exactly is transferred here. Is it the low-level policy? Is it the high-level skill plan? Based on my understanding, the authors may be aiming at the former. If so, a transfer setting where the low-level policies are fixed and only the high-level plans are updated / fine-tuned would be far more convincing. After all, I think that is what people would normally expect when you are claiming skill transfer, isn't? In fact, to my knowledge, some HRL approaches have attempted to reuse previously learned & fixed policies (e.g., [1,2]). I'm not saying that these approaches are better, but since the focus is skill transfer, I think a proper discussion/comparison would be critical. 

For rebuttal, please address my concerns listed above. I think the paper presents an interesting and promising idea, but I am not fully convinced that the approach has the advantages as claimed in the paper.

References:

[1] Tessler et al., A Deep Hierarchical Approach to Lifelong Learning in Minecraft, AAAI 2017.

[2] Shu et al., Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning, ICLR 2018.
review_title: Hierarchical model-base RL for skill transfer -- interesting idea but not sufficiently convincing

Meta-review: This paper proposes a unified model-based framework for high-level skill learning and composition through hierarchical RL. The proposed approach combines high-level planning in a low dimensional space with low-level skill learning, where each low-level skill is a policy conditioned on the high-level task. The low-level policies are learned by using a mutual information objective. The proposed approach is evaluated on locomotion tasks, and is shown to be overall more data efficient than alternative baselines.
The reviewers agree that this work is original and sufficiently empirically motivated for acceptance. Two reviewers were concerned by the experimental setup and the transfer setting that are somehow too simple, but the authors fixed these issues in the improved version based on the feedback.
Decision: Accept (Poster)