Latent Skill Planning for Exploration and Transfer
Authors: Kevin Xie
Homanga Bharadhwaj
Danijar Hafner
Animesh Garg
Florian Shkurti
Venue: ICLR 2021 Poster (2021)
Keywords: Model-Based Reinforcement Learning
World Models
Skill Discovery
Mutual Information
Planning
Model Predictive Control
Partial Amortization

Abstract: To quickly solve new tasks in complex environments, intelligent agents need to build up reusable knowledge. For example, a learned world model captures knowledge about the environment that applies to new tasks. Similarly, skills capture general behaviors that can apply to new tasks. In this paper, we investigate how these two approaches can be integrated into a single reinforcement learning agent. Specifically, we leverage the idea of partial amortization for fast adaptation at test time. For this, actions are produced by a policy that is learned over time while the skills it conditions on are chosen using online planning. We demonstrate the benefits of our design decisions across a suite of challenging locomotion tasks and demonstrate improved sample efficiency in single tasks as well as in transfer from one task to another, as compared to competitive baselines. Videos are available at: https://sites.google.com/view/latent-skill-planning/

Published as a conference paper at ICLR 2021
LATENT SKILL PLANNING FOR EXPLORATION AND TRANSFER
Kevin Xiel Homanga Bharadhwaj's Danijar Hafner Animesh Garg!., Florian Shkurti'
'University Toronto and Vector Institute, 'Google Brain, NNidda
Exxieccs.toront..co edu, homangaêcs toronto.
Abstract
To quickly solve new tasks in complex environments, intelligent agents need build up reusable knowledge. For example, learned world model captures knowledge about the environment that applies to new tasks. Similarly, skills capture general behaviors that can apply to new tasks. In this paper, we investigate how these two approaches Specifically, we leverage the idea of partial amortization for fast adaptation at test time. For this, actions produced by policy that is learned over time while the skills conditions on are chosen using online planning. We demonstrate the benefits of our design decisions across suite of challenging locomotion tasks and demonstrate improved sample efficiency in single tasks as well as in transfer from task another, as compared to competitive baselines. Videos are available https:: / / swtees.tssss.//eee///eees.eeeeee google. wtw.lee..coo/o/// com/viewi /tt//aaaeetttteee westentsskee skill 1ill1-a weaett-kkk--pplannn.... planning/
be integrated into single reinforcement learning agent.
INTRODUCTION
Humans can effortlessly compose skills, where skills are a se- quence of temporally correlated actions, and quickly adapt skills learned from one task to another. In order to build re-usable knowl- edge about the environment, Model-based Reinforcement Learning (MBRL) (Wang et al., 2019) provides an intuitive framework which holds the promise of training agents that generalize to different situations, and sample efficient with respect to number of envi- ronment interactions required for training. For temporally compos- ing behaviors, hicrarchical reinforcement learning (HRL) (Barto & Mahadevan, 2003) seeks to learn behaviors at different levels of abstraction explicitly.
simple approach for learning the environment dynamics is to learn world model either directly in the observation space (Chua et al., 2018; Sharma et al., 2019; Wang & Ba, 2019) or in a la- tent space (Hafner al.. 2019; 2018). World models summarize an agent's experience in the form of learned transition dynam- ics, and reward models, which are used to learn either parametric policies by amortizing over the entire training experience (Hafner al., 2019; Janner al., 2019), or perform online planning as done in Planet (Hafner et al., 2018), and PETS (Chua et al., 2018), Amortization here refers to learning a parameterized policy, whose parameters updated using samples during the training phase, and which can then be directly queried at each state to output an action, during evaluation.
Figure 1: Visual illustration of the root position of quadruped trained with LSP on an environ- ment with random obstacles and transferred this environment with obstacles aligned in line. The objective reach goal location in red.
Fully online planning methods such as PETS (Chua et al., 2018) only learn the dynamics (and reward) model and rely online
"evin and Homanga contributed equally this work.

---

Published as a conference paper at ICLR 2021
search procedure such as Cross-Entropp Method (CEM; Rubinstein, 1997) the learned models to determine which action to execute next. Since rollouts from the learned dynamics and reward models not executed in the actual environment during training, these learned models are sometimes also referred to as imagination models (Hafner et al., 2018; 2019). Fully amortized methods such as Dreamer (Hafner et al., 2019), train a reactive policy with many rollouts from the imagination model. They then execute the resulting policy in the environment.
The benefit of the amortized method is that becomes better with experience. Amortized policies are also faster. An action is computed in one forward pass of the reactive policy as opposed to the potentially expensive search procedure used in CEM. Additionally, the performance of the amortized method is more consistent as CEM relies on drawing good samples from random action distribution. On the other hand, the shortcoming of the amortized policy is generalization. When attempting novel tasks unseen during training, CEM will plan action sequences for the new task, as per the new reward function while fully amortized method would be stuck with a behaviour optimized for the training tasks. Since it is intractable to perform fully online random shooting based planning in high-dimensional action spaces (Bharadhwaj et al., 2020; Amos & Yarats, 2019), it motivates the question: can combine online search with amortized policy learning in a meaningful way learn useful and transferable skills for MBRL?
To this end, we propose partially amortized planning algorithm that temporally composes high-level skills through the Cross-Entropy Method (CEM) (Rubinstein, 1997), and uses these skills to condition low-level policy that is amortized over the agent's experience. Our world model consists of learned latent dynamics model, and learned latent reward model. We have a mutual information (MI) based intrinsic reward objective, in addition the predicted task rewards that are used train the low level-policy, while the high level skills are planned through CEM using the learned task rewards. We term our approach Learning Skills for Planning (LSP).
The key idea of LSP is that the high-level skills are able to abstract out essential information necessary for solving task, while being agnostic to irrelevant aspects of the environment, such that given a new task in similar environment, the agent will be able to meaningfully compose the learned skills with very little fine-tuning. addition, since the skill-space is low dimensional, we can leverage the benefits of online planning in skill space through CEM, without encountering intractability of using CEM for planning directly the higher dimensional action space and especially for longer time horizons (Figure 1).
In summary, our main contributions are developing a partially amortized planning approach for MBRL, demonstrating that high-level skills can be temporally composed using this scheme to condition low level policies, and experimentally demonstrating the benefit of LSP over challenging locomotion tasks that require composing different behaviors to solve the task, and benefit in terms of transfer from one quadruped locomotion task to another, with very little adaptation in the target task.
2 BACKGROUND
We discuss learning latent dynamics for MBRL, and mutual information skill discovery, that serve as the basic theoretical tools for our approach.
2.1 LEARNING LATENT DYNAMICS AND BEHAVIORS IN IMAGINATION
Latent dynamics models are special cases of world models used in MBRL, that project observations into a latent representation, amenable for planning (Hafner et al., 2019; 2018). This framework is general as it can model both partially observed environments where sensory inputs can be pixel observations, and fully observable environments, where sensory inputs can be proprioceptive state features. The latent dynamics models we consider in this work, consist of four key components, representation module po(s; ps (t-1, oz) and an observation module go(or(sT) that encode observations and actions to continuous rector-valued latent states $,, a latent forward dynamics module go( (z | 0x-1) that predicts future latent states given only the past states and actions, and a task reward module go (r, (s,) that predicts the reward from the environment given the current latent state. To learn this model, the agent interacts with the environment and maximizes the following
2

---

Published as a conference paper at ICLR 2021
?
22
X
(a) Fully Online (Planet)
(b) Fully Amortized (Dreamer)
Figure 2: (a) Online planning where actions are sampled from a distribution with parameters 9, (b) Fully amortized policy with parameters (c) [LSP] Skills are sampled from distribution with parameters 0,, and actions are sampled from a skill-conditioned policy with parameters é. Here a are actions, states and latent plan variables. 0; represents parameters of the planning distribution and & are the parameters of the policy.
expectation under the dataset of environment interactions D ={(0,, al, ac, rt]}
J = ED Σ Th + ) const
TH = In st)
Th = Inq se)
JB = KL (p(s; st-1, a--1 oz) || g(( s1-1,
For optimizing behavior under this latent dynamies model, the agent rolls out trajectories in imag- ination and estimates the value V-) of the imagined trajectories {8r, d Tr]r=t 1+H through TD(A) estimates as described by Sutton & Barto (2018); Hafner et al. (2019). The agent can either learn fully amortized policy (a(a)s as done in Dreamer, by backpropagating through the learned value network (-) or plan online through CEM, for example as in Planet.
2.2 MUTUAL INFORMATION SKILL DISCOVERY
Some methods for skill discovery have adopted a probabilistic approach that uses the mutual informa- tion between skills and future states are represented through a latent variable 2 upon which a low level policy T[ass, 2) is conditioned. Given the current state 8o, skills are sampled from some selection distribution p(=[so) The skill conditioned policy is executed under the environment dynamics Pa($e+1 1 (8,, a) resulting in a series of future states abbreviated s' := {s}.
objective (Sharma et al., 2019). In this approach, skills
Mutual information is defined as:
MI(2, (sHsp) = H(22s)) - H(2){s), sn) = H({s}}so) - H({s}]s0,
quantifies the reduction uncertainty about the future states given the skill and vice versa. By maximizing the mutual information with respect to the low level policy, the skills are encouraged to produce discernible future states.
3 PARTIAL AMORTIZATION THROUGH HIERARCHY
Our aim is learn behaviors suitable for solving complex control tasks, and amenable to transfer to different tasks, with minimal fine-tuning. To achieve this, we consider the setting of MBRL, where the agent builds up re-usable knowledge of the environment dynamics. For planning, we adopt partial amortization strategy, such that some aspects of the behavior are re-used over the entire training experience, while other aspects are learned online. We achieve partial amortization by forming high level latent plans and learning low level policy conditioned the latent plan. The three different forms of amortization in planning are described visually through probabilistic graphical models Figure and Figure 3.
We first describe the different components of our model, motivate the mutual information based auxiliary objective, and finally discuss the complete algorithm.
3

---

Published as a conference paper at ICLR 2021
max log p(z
World model. Our world model is a latent dy- namics model consisting of the components de- scribed in section 2.
2
3g
Low level policy, The low-level policy (s(eels), is used to decide which action to execute given the current latent state St and the currently active skill Dreamer (Hafner et al., 2019), we also train value model (s,) to estimate the expected rewards the action model achieves from each state 8t. We estimate value the same way in equation 6 of Dreamer, balancing bias and variance. The action model is trained maximize the estimate of the value, while the value model is trained to fit the estimate of the value that alters as the action model is updated, as done in typical actor-critic setup (Konda& Tsitsiklis, 2000).
8,
Similar to
F4
Figure 3: LSP samples kills from a distribution optimized with CEM online, and samples actions for every latent state from skill-conditioned policy. The skill distribution is updated by maximizing mu- tual information between the skills and the observed latent state distribution.
policy is updated by back- propagating value estimates based on the environment rewards through the learned latent synamics model.
High level skills. In our framework high level
skills are continuous random variables that are held for fixed number K steps. The high-level skills 2 are sampled from skill selection distribution p(=1: k/ N (N, Σ) which is optimized for task performance through CEM. Here, H denotes the planning horizon. For the sake of notational convenience we denote 21:[8/K] z. Let ()) denote the jth CEM iteration. We first sample G skills 1 ~ p(zlç()), execute G parallel imaginary rollouts of horizon H in the learned model with the skill-conditioned policy 4s (oeelsst, 2(9)). Instead of evaluating rollouts based only on the sum of rewards, we utilize the value network and compute value estimates [V1] We sort {= choose the top M values, and use the corresponding skills update the sampling distribution parameters as ((++1) = (1(i+t)), E(+!)
(m)) a (++) =Mean(1 (m) (on)MM... M=z) 20+1)_ Variance (( (m))
3.1 OVERALL ALGORITHM
Our overall algorithm consists of the three phases typical in MBRL pipeline, that are performed iteratively. The complete algorithm is shown in Algorithm 1. The sub-routine for CEM planning that gets called in Algorithm is described in Algorithm 2.
Model Learning. We sample a batch of tuples from the dataset of environment interactions {(a,, 0z, r2) 1. rt) ~ D. compute the latent states st ~ Po |t-1, Qt-1, ot), and use the re- sulting data to update the models po(s: | 8 ot). qs(s: (t-1, and go st) through the variational information bottleneck (VIB) (Tishby et al., 2000; Alemi et al., 2016) objective as in equation 13 of Hafner et al. (2019) and as described in section
Behavior Learning. Here, we roll out the low-level policy go (a 2). in the world model and use the state transitions and predicted rewards to optimize the parameters of the policy o, the skill distribution Ç. the value model vi, and the backward skill predictor x. The backward skill predictor predicts the skill given latent rollouts (s).
Environment Interaction. This step is to collect data in the actual environment for updating the model parameters. Using Model-Predictive Control (MPC), we re-sample high-level skills from the optimized Pç (2) every K steps, and execute the low-level policy 9o $t, 2), conditioned on the currently active skill Hence, the latent plan has lower temporal resolution compared to the low level policy. This helps us perform temporally abstracted exploration easily in the skill space. We store the (observation, action, reward) tuples in the dataset D.
4

---

Published as a conference paper at ICLR 2021
3.2 MUTUAL INFORMATION SKILL OBJECTIVE
Merely conditioning the low level policy qo (ay|st 2) on the skill 2 is not sufficient as it is prone to ignoring it. Hence, incorporate maximization of the mutual information (MI) between the latent skills 2 and the sequence of states {*} as an auxiliary objective.
In this paper, we make use of imagination rollouts to estimate the mutual information under the agent's learned dynamics model. We decompose the mutual information in terms of skill uncertainty reduction (Z {s} H(z]so) ())
Estimating MI {s}(su) Explicitly writing out the entropy terms, we have
MI(2, = H((=(5)) M(2](5). so) - p(2, fs}, so) log p[= 50)
p(z
In this case we need a tractable approximation to the skill posterior p(2 8¹).
(8)180) f f (+), log (<2({)).80) + log p(=l{s), s0) p(2[s0)
((=](>}, so)
Here the latter term is KL divergence and must hence be positive, providing lower bound for MI.
(8),6)))
MI(2,(5)|so) 2 p(2, (s),
80) log p(z[su)
= f p(z, {8}, 80) log (8) p(=)sa)p(so) p(2(()))))
pp Dog (=]{{)) [o)] Ex [H[p(2)80)]
We parameterize {==({s),s)) so) with x. i.e gx (=)ss), so). and call it the backward skill predictor, as it predicts the skill given latent rollouts {s}. It is trained through standard supervised learning to maximize the likelihood imagined rollouts Ex (x). [log { s.)]. This mutual information
objective is only function of the policy through the first term and hence we use it as the intrinsic reward for the agent r, log
The second term E.p [+[[[=][=)]]] is the entropy of the skill selection distribution. When skills begin to specialize, the CEM distribution will naturally decrease in entropy and so we add Gaussian noise to the mean of the CEM- -asdd skill distribution, μ f μ + where <~ A0, Io). By doing this we lower bound the entropy the skill selection distribution.
EXPERIMENTS
We perform experimental evaluation over locomotion tasks based on the DeepMind Control Suite framework (Tassa al., 2018) understand the following questions:
Does LSP learn useful skills and compose them appropriately to succeed in individual tasks?
Does LSP adapt to a target task with different environment reward functions quickly, after being pre-trained another task?
To answer these, we perform experiments on locomotion tasks, using agents with different dynamics Quadruped, Walker, Cheetah, and Hopper, and environments where either pixel observations or proprioceptive features are available to the agent. Our experiments consist of evaluation in single tasks, transfer from one task to another, ablation studies, and visualization of the learned skills.
4.1 SETUP
Baselines. We consider Dreamer (Hafner al., 2019), which is state of the art model-based RL algorithm with fully amortized policy learning, as the primary baseline, based on its open-source tensorflow2 implementation. We consider Hierarchical RL (HRL) baseline, HIRO (Nachum et al., 2018) that trains a high level amortized policy (as opposed to high level planning). For consistency, use the same intrinsic reward for HIRO method. We consider other baselines, named
5

---

Published as a conference paper at ICLR 2021
Quadruped Run
Quadruped Reach
Quadruped GetUp Walk
Cheetah Pixels
Avg. Rewards
1000
500
500
500
500
1.0
0.2
0.4
0.5
1.0
0.5
1.0
Quadruped Run Pixels
WalkerRun Pixels
WalkerRun
Hopper Hop
Avg. Rewards
400
500
500
200
500
250
250
0.3
0.6
0.3
0.6
0.5
1.0
0.5
1.0
Our method
Random Skills
Dreamer
RandSkillsinp
HIRO
Figure 4: Single task performance. x-axis represents number of enviroument interactions in terms fraction of Million. Comparison between LSP,, and the baselines Dreamer, HIRO, RandSkillsInp, and Random Skills on a suite of challenging individual locomotion tasks. Cheetah Pixels, Quadruped Run Pixels, and WalkerRun Pixels are environments with image-based observations while the rest have proprioceptive features (states) as observations to the agent. Higher better.
Random Skills that has hierarchical structure exactly as LSP but the skills are sampled randomly and there is no planning at the level of skills, and RandSkillslnit that is similar to Random Skills but does not include the intrinsic rewards (this is essentially equivalent to Dreamer with an additional random skill input) These two baselines are to help understand the utility of the learned skills. For the transfer experiments, we consider an additional baseline, variant of our method that keeps the low-level policy fixed in the transfer environment. All results are over three random seeds.
Environments. We consider challenging locomotion environments from DeepMind Control Suite (Tassa et al., 2018) for evaluation, that require learning walking, running, and hopping gaits which can be achieved by temporally composing skills. In the Quadruped GetUp Walk task, a quadruped must learn stand up from randomly initialized position that is sometimes upside down, and walk on a plane without toppling, while in Quadruped Reach, the quadruped agent must walk in order reach particular goal location. In Quadruped Run, the same quadruped agent must run as fast as possible, with higher rewards for faster speed. In the Quadruped Obstacle environments (Fig, 6a), the quadruped agent must reach goal while circumventing multiple cylindrical obstacles. In Cheetah Run, and Walker Run, the cheetah and walker agents must run as fast as possible. In Hopper Hop, one legged hopper must hop in the environment without toppling. extremely challenging to maintain stability of this agent.
4.2 SOLVING SINGLE LOCOMOTION TASKS.
In Figure we evaluate our approach LSP in comparison the fully amortized baseline Dreamer, and the Random Skills baseline on a suite of challenging locomotion tasks. Although the environments have a single task objective, in order to achieve high rewards, the agents need learn different walking gaits (Quadruped Walk, Walker Walk), running gaits (Quadruped Run, Walker Run), and hopping gaits (Hopper Hop) and compose learned skills appropriately for locomotion.
From the results, it is evident that LSP either outperforms Dreamer or is competitive to it on all the environments. This demonstrates the benefit of the hierarchical skill-based policy learning approach of LSP. In addition, we observe that LSP significantly outperforms the Random Skills and RandSkills(np baselines, indicating that learning skills and plamning over them is important to succeed in these locomotion tasks. In order to tease out the benefits of hierarchy and partial amortization separately, we consider another hierarchical RL baseline, HIRO (Nachum et al., 2018), which is a state-of-the-as the HRL algorithm that learns high level amortized policy. LSP outperforms HIRO in all the tasks suggesting the utility of temporally composing the learned skills through planning as opposed to amortizing over them with a policy. HIRO has not been shown work from images

---

Published as a conference paper at ICLR 2021
Quadruped Walk Reach
Quadruped GetUp Walk -> Reach
500
700
600
Rewards
400
500
300
400
300
200
160
100
0.25
0.5
0.2
0.4
method
Drea
HIRO
Our method (fixed policy)
Figure 5: Transfer. x-axis represents no. of environment steps in terms fraction 1 Million. Comparison between our method (LSP), and the baselines Dreamer, HIRO, and a variant of our method that keeps the policy fixed in the transfer environment. The agents need to transfer to different locomotion task, after being pretrained another task. (Higher is better).
in the original paper (Nachum et al., 2018), 8), and so we do not have learning curves for HIRO in the image-based environments, as it cannot scale directly to work with image based observations.
4.3 TRANSFER FROM ONE TASK TO ANOTHER.
In Figure 5 we show results for quadruped agent that is pre-trained on one task and must transfer to a different task with similar environment dynamics but different task description.
Quadruped GetUp Walk + Reach Goal. The quadruped agent is pre-trained on the task of standing up from a randomly initialized position that sometimes upside down, and walking on plane without toppling. The transfer task consists of walking to reach goal, and environment rewards are specified in terms of distance to goal. The agent is randomly initialized and is sometimes initialized upside down, such that it must learn get upright and then start walking towards the goal. We see that LSP can adapt much quickly to the transfer task, achieving a reward of 500 only after 70, 000 steps, while Dreamer requires 130, 000 steps achieve the same reward, indicating sample efficient transfer of learned skills.
We observe that the variant of our method LSP with fixed policy in the transfer environment performs as well as or slightly better than LSP. This suggests that while transferring from the GetUp Walk to the Reach task, low level control is useful to be directly transferred while planning over high level skills which have changed is essential. As the target task is different, so it requires composition of different skills.
Quadruped Walk - Reach Goal. The quadruped agent is randomly initialized, but it is ensured that it is upright at initialization. In this setting, after pre-training, we re-label the value of rewards in the replay buffer of both the baseline Dreamer, and LSP with the reward function of the target Quadruped Reach Goal environment. To do this, we consider each tuple in the replay buffer of imagined trajectories during pre-training, and change the reward labels the reward value obtained by querying the reward function of the target task at the corresponding state and action of the tuple. From the plot in Figure we see that LSP able to quickly bootstrap learning from the rablebd Replay Buffer and achieve better target adaptation than the baseline.
From the figure it is evident that for HIRO, the transfer task rewards converge a much lower value than our method LSP and Dreamer, suggesting that the learned skills by an amortized high level policy overfits to the source task, and cannot be efficiently adapted to the target task. Similar to the previous transfer task, also observe that the variant of our method LSP with fixed policy in the transfer environment performs as well as or slightly better than LSP. This provides further evidence that since the underlying dynamics of the Quadruped agent is similar across both the tasks, low level control is useful to be directly transferred while the high level skills need to adapted through planning.
7

---

Published as a conference paper at ICLR 2021
1000
Quadruped Getup Walk
400
200
0.10
0.13
0.23
0.30
Quadruped obstacle environments
(b) Ablation study
Figure 6: (a) Quadruped obstacle environments. Visual illustration of the environment variants we consider. For the environment the right, the agent must reach the goal while navigating across randomly located cylindrical obstacles while on the left, the agent must walk around a pillar of obstacles to reach the goal. provide details and results in the appendix. (b) Ablation study. x-axis represents no. of environment steps in terms of fraction of Million. Comparison of dreamer, LSP and the no_M/ ablation. In no_MI, we consider LSP but do not use the mutual information maximizing intrinsic reward to train the low level policy, keeping other details exactly the same. Results are on the Quadruped GetUp Walk task. Higher better.
4.4 MUTUAL INFORMATION ABLATION STUDY
In order better understand the benefit of the mutual information skill objective, we compare performance against a baseline that is equivalent to LSP but does not use the intrinsic reward to train the low level policy. We call this ablation baseline that does not have mutual information maximization between skills and states, as no_MI. We show the respective reward curves for the Quadruped GetUp Walk task in Figure 6b. Without the mutual information objective, LSP learns less quickly but still faster than the basclinn Dreamer. This emphasizes the necessity of the MI skill objective section 3.2 and suggests that merely conditioning the low-level policy on the learned skills is still effective to some extent but potentially suffers from the low-level policy learning to ignore them.
4.5 VISUALIZATION OF THE LEARNED SKILLS
In Figure 7, visualize learned skills of LSP, while transferring from the Quadruped Walk to the Quadruped Reach task. Each sub-figure (with composited images) corresponds to a different trajectory rolled out from the same initial state. It is evident that the learned skills reasonably diverse and useful in the transfer task.
5 RELATED WORK
Skill discovery. Some RL algorithms explicitly try to learn task decompositions in the form of re-usable skills, which are generally formulated as temporally abstracted actions (Sutton et al., 1999). Most recent skill discovery algorithms seek to maximize the mutual information between skills and input observations (Gregor et al., 2016; Florensa al., 2017), sometimes resulting in an unsupervised diversity maximization objective (Eysenbach et al., 2018; Sharma et al., 2019). DADS (Sharma et al., 2019) is an unsupervised skill discovery algorithm for learning diverse skills with a skill-transition dynamics model, but does not learn world model for low-level actions and observations, and hence cannot learn through imagined rollouts and instead requiring many environment rollouts with different sampled skills.
Hierarchical RL. Hierarchical RL (HRL) (Barto & Mahadevan, 2003) methods decompose a com- plex task to sub-tasks and solve each task by optimizing a certain objective function. HIRO (Nachum et al., 2018) learns a high-level policy and low-level policy and computes intrinsic rewards for training the low-level policy through sub-goals specified as part of the state-representation the agent observes. Some other algorithms follow the options framework (Sutton et al., 1999; Bacon et al., 2017), where options correspond to temporal abstractions that need specifying some termination conditions. In practice, it is difficult to learn meaningful termination conditions without additional regularization (Harb et al., 2017). These HRL approaches are inherently specific to the tasks being trained on, and do not necessarily transfer new domains, even with similar dynamics.
8

---

Published as a conference paper at ICLR 2021
Figure Qualitative results visualizing some learned behaviors of LSP for the quadruped agent while transferring from the walking to the goal reaching task. Each image depicts the agent performing a different skill for 40 time steps in the environment starting from the same initial state at the origin depicted by the red spot. We see that the learned skills are diverse, correspond to reasonable gaits, and help in performing the task. Video visualizations are in the website https://sittss www.sieeessgoo...oooooeoooooooee www.le.com//yeee.ee.o itwtii/ippatiilllmottiiitooooooo curtiiaatoonhierarchyhooomn
Transfer in RL. Multiple previous works have investigated the problem of transferring policies different environments. Progressive Networks (Rusu et al., 2016) bootstrap knowledge from previously learned tasks by avoiding catastrophic forgetting of the learned models, (Byravan et al., 2020) perform model-based value estimation for learning an amortized policy and transfer to tasks with different specified reward functions keeping the dynamics the same, while Plan2Explore (Sekar et al., 2020) first learns a global world model without task rewards through self-supervised objective, and given a user-specified reward function at test time, quickly adapts to it. In contrast to these, several meta RL approaches learn policy parameters that generalize well with little fine-tuning (often in the form of gradient updates) to target environments (Finn et al., 2017; Xu et al., 2018; Wang et al., 2016; Rakelly et al., 2019; Yu et al., 2019).
Amortization for planning. Most current MBRL approaches use some version of the 'Cross- Entropy Method (CEM) or Poddel-rredcctvee Path Integral (MPPI) for doing a random population based search of plans given the current model (Wang Ba, 2019; Hafner et al., 2018; Williams et al., 2016; Sharma et al., 2019). These online non-amortized planning approaches are typically very expensive high-dimensional action spaces. Although (Wang & Ba, 2019) introduces the idea of performing the CEM search in the parameter space of a distilled policy, it still is very costly and requires lot of samples for convergence. To mitigate these issues, some recent approaches have combined gradient-descent based planning with CEM (Bharadhwaj et al., 2020; Amos & Yarats, 2019). In contrast, (Janner et al., 2019; Hafner et al., 2019) fully amortize learned policies over the entire training experience, which is fast for high-dimensions transfer to new environments with different dynamics and reward functions. We combined the best of both approaches by using CEM plan online for high-level skills (of low dimensionality) and amortize the skill conditioned policy for low-level actions (of higher dimensionality).
action spaces, but cannot directly
6 DISCUSSION
In this paper, we analyzed the implications of partial amortization with respect to sample efficiency and overall performance on suite of locomotion and transfer tasks. We specifically focused on the setting where partial amortization is enforced through hierarchical planning model consisting of fully amortized low-level policy and fully online high level skill planner. Through experiments in both state-based and image- based environments we demonstrated the efficacy of our approach in terms of planning for useful skills and executing high-reward achieving policies conditioned on those skills, as evaluated by sample efficiency (measured by number of environment interactions) and asymptotic performance (measured by cumulative rewards and success rate).
One key limitation of our algorithm is that CEM planning is prohibitive in nigh-dimensional action spaces, and so we cannot have a very high dimensional skill-space for planning with CEM, that might be necessary for learning more expressive/comples skills in real-world robot control tasks. One potential direction of future work is to incorporate amortized learning of skill policies during training, and use CEM for online planning of skills only during inference. Another direction could be to incorporate gradient- avoiding local optima in skill planning would be a potential challenge for gradient descent planning.
based planning instead of a random search procedure as CEM, but

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENT
We thank Vector Institute Toronto for compute support. We thank Mayank Mittal, Irene Zhang, Alexandra Volokhova, Dylan Turpin, Arthur Allshire, Dhruv Sharma and other members of the UofT CS Robotics group for helpful discussions and feedback on the draft.
CONTRIBUTIONS
All the authors were involved designing the algorithm, shaping the design of experiments, and in writing the paper. Everyone participated in the weekly meetings and brainstorming sessions.
Kevin and Homanga led the project by deciding the problem to work on, setting up the coding infrastructure, figuring out details of the algorithm, rumning experiments, and logging results.
Danijar guided the setup of experiments, and helped provide detailed insights when we ran into bottlenecks and helped figure out how to navigate challenges, both with respect to implementation, and algorithm design.
Animesh and Florian provided valuable insights on what contributions to focus on, and helped us in understanding the limitations the algorithm at different stages of its development. They also motivated the students to keep going during times of uncertainty and stress induced by the pandemic.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv preprint arXiv:1612.00447 2016.
Brandon Amos and Denis Yarats. The differentiable cross-entropy method. arXiv preprint wrXiv:1909.12833 2019.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty- First AAAI Conference Artificial Intelligence, 2017.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1-2):41-77, 2003.
Homanga Bharadhwaj, Kevin Xie, and Florian Shkurti. Model-predictive planning via cross-entropy and gradient-based optimization. In Learning for Dynamics and Control, pp. 277-286, 2020.
Arunkumar Byravan, Jost Tobias Springenberg, Abbas Abdolmaleki, Roland Hafner, Michael Neunert, Thomas Lampe. Noah Siegel, Nicolas Heess, and Martin Riedmiller. Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models. In Conference on Robot Learning, pp. 566-589, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-4765, 2018.
Benjamin Eysenbach. Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1 1126-115 1136-115. JMLR. org, 2017.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. arXiv preprint arXiv: 170.00017 2017.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning hierarchical bayes. arXiv preprint arXiv: (80100893)) 2018.
10

---

Published as a conference paper at ICLR 2021
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04555 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.0160 2019.
Jean Harb, Pierre-Lu Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option: Learning options with deliberation cost. arXiv preprint 17Xx::1709.4451,, 2017.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances Neural Information Processing Systems, pp. 12498- 12509, 2019.
Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-1014, 2000.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00900 2018
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018.
Masashi Okada and Tadahiro Taniguchi. Variational inference mpe for bayesian model-based reinforcement learning. arXiv preprint 197.02190, 2019.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforreement learning via probabilistic context variables. In International conference machine learning, pp. 5331-5340, 2019.
Reuven Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1)899-112, 1997.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 04671, 2016.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. arXiv preprint arXiv:2005.0 0.5600 2020.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: arXiv:1907.00 1907.01657 2019.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):81--11, 1999.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690. 2018.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Emanuel Todorov. General duality between optimal control and estimation. 2008 47th IEEE Conference Decision and Control, pp. 4286-4292 IEFE, 2008
11

---

Published as a conference paper at ICLR 2021
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint 0rXiv:1611.0576.3 2016.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint (tt.:9900.0499 2019.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang. Pieter Abbeel, and Jimmy Ba. Benchmarking model- based reinforcement learning. arXiv preprint arXiv: 190.v0:9000000553 2019.
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggres- sive driving with model predictive path integral control. In 20/6 IEEE International Conference Robotics and Automation (ICRA), pp. 1433- 1440. IEEE, 2016.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pp. 2396-2407, 2018.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv: 111::1910.109901 2019.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 ALGORITHM
Algorithm 1: Learning Skills for Planning
Initialize dataset D with random seed episodes. Initialize neural network parameters 8, o, 1b Define skill duration K, plan horizon H, CEM population size G, MaxCEMiter, skill noise while not converged do for update step 1..C do
Model learning
Draw B data sequences ((ae, 0t, (e),+))) ~ D.
Compute model states 8t pp((s $1-1, 01). S-
Update 9 using representation learning.
// Behavior learning
(= (1, CEM(S MaxCEMiter, G. H, K, (()) Add noise μ t
Compute rewards R = (((o(r+) 8-)) with the optimized CEM distribution Ç
Compute the corresponding intrinsic rewards r'
Store the corresponding states into K sized sequences {( [H/K] (for x)
Use total rewards R + r¹ to form value estimates, and update Φ, v.
// Environment
interaction
01 - env reset () for time step t = 0..I - 1 do
// MPC in space. Resample skill every timesteps
if t%K = 0 then
Sample skill [ι:8/K]] ~ p(= Choose the first skill, 21
Compute 8, ~ p0( 8t-1, (t-1, 0t) from history, choose ot ~ go (ee 88, Tt, 0++1 +- env. step (a.).
Add experience to dataset D D U {(o, de, r.)
A.2 QUADRUPED OBSTACLE TRANSFER EXPERIMENT
We evaluate the ability of method transfer more complex tasks. Here the source task is to walk forward at a constant speed in a random obstacle environment. The policy is trained this source task for 500k steps before transferring to the target task which is pure sparse reward task. The obstacles are arranged in cove like formation where the straight line to the sparse target leads into a local minima, being stuck against the obstacles. To be able solve this task, the agent needs to be able to perform long term temporally correlated exploration. We keep all other settings the same but increase the skill length to 30 time steps and skill horizon to 120 time steps after transfer in the target task to make the skills be held for longer and let the agent plan further ahead. We see Figure 10 that the trajectories explored by Dreamer are restricted be near the initialization and does not explore beyond the obstacles. In contrast, LSP is able fully explore the environment and reach the sparse goal multiple times. By transferring skills, LSP able to explore at the level of skills and hence produces much more temporally coherent trajectories.
A.3 SETTINGS AND HYPERPARAMETERS
Our method based on the tensorflow2 implementation of Dreamer (Hafner et al., 2019) and retains most of the original hyperparameters for policy and model learning. Training is performed more frequently, in particular training update of all modules is done every 5 environment steps. This marginally improves the training performance and is used in the dreamer baseline well method. For feature- based experiments we replace the convolutional encoder and decoder with 2- MI.P networks with 256 units each and EL.U activation. Additionally, the dense decoder network
13

---

Published as a conference paper at ICLR 2021
Figure 8: Quadruped obstacle environ- ment trajectories. Visual illustration the 2D root position of the quadruped for the 1st 40k steps after transfer to the cove environment. On the left is Dreamer and on the right LSP with fixed policy. Nodes in trajectory correspond be- ginning state of new skills in the case of LSP.
outputs both the mean and log standard deviation of the Gaussian distribution over observations. The standard deviation is softly clamped between 0.1 and 1.5 described in (Chua et al., 2018).
For LSP, skill vectors are 3-dimensional and are held for K 10 steps before being updated. The iterations, proposes G 16 skills and uses the top M = 4 proposals to recompute statistics in each iteration. The additional noise e added to the CEM optimized distribution is Normal(O, 0.1).
CEM method has a planning horizon of
10, goes through MorCEMiter
The backwards skill predictor shares the same architecture and settings the feature-based decoder module described above. trained with Adam with learning rate of Se - 5.
A.4 CONTROL AS INFERENCE
The control inference framework (Todorov, 2008; Levine, 2018) provides heuristic encourage exploration. It performs inference in a surrogate probabilistic graphical model, where the likelihood of trajectory being optimal (an indicator random variable O) is hand-designed (monotonically increasing) function of the reward, log p( p(O))) f(r). The induced surrogate posterior p(r[O) places higher probability on higher reward trajectories.
Sampling from ((T is in general intractable and so common solution is to employ variational inference for obtaining an approximation go()). The objective is to minimize the KL-divergence with respect to the true posterior:
min (Lee (p((()) = max - 0
(2)
max ( (olllolp(O0011 log (a))
(3)
max Bn [f(]] + (a)
Note that this objective boils down to maximizing a function the reward and an entropy term H(.))
State of the model-free reinforcement learning approaches be interpreted instances of this framework (Haarnoja et al., 2018). The connection has also been made model-based RL (Okada & Taniguchi, 2019). Specifically they show how the ubiquitous MPPI/CEM planning methods can be derived from this framework when applying variational inference to simple Gaussian action distributions. Though they show that more sophisticated distributions can be used in the framework as well (in principle), the only other model they investigate in the paper is a mixture Gaussians distribution.
A.5 HIBRARCHICAL INFERENCE
We consider hierarchical action distributions that combine amortizing low level behaviours and online planning by temporally composing these low level behaviours. Specifically we use the following variational distribution:
T (o.0(T) (T) II 10(8+1189 at (o) =w(D)) II go, (21)
K
Here æ are latent variables defining a high level plan that modulates the behaviour of the low level policy (a (x)). k(t) is an assignment defining which of the K z's will act at time For all experiments, chose fixed size window assignment such that k<t)) (KK/T)).
14

---

Published as a conference paper at ICLR 2021
Quadruped fetch
400
dreamer
300
Return
Figure 9: Quadruped obstacle environ- ment transfer learning curves. We plot total rewards against number of transfer training time steps the sparse cove environment. Note that after 70k steps (70 episodes), the LSP agent is able reli- ably rech the target every time, whereas dreamer does not even come close to the target, stunted by the sparse reward.
200
100
10000 20000 30000
43000
50000
60000
70000
80000
Steps
Here p(T) is the task distribution which wish amortize. For example using (o) would amortize over plans starting from different initial states. Note that the minimization over & is outside the expectation which means that is amortized, whereas the minimization over 0 is inside such that it is inferred online.
min ( [min K
(5)
max LE PT)) [max (oo - log (als,, 2)ga, (z)]]
(6)
This formulation also directly relates to some meta-learning approaches that can be interpreted as hierarchical Bayes (Grant et al., 2018). There are different options for the inner and outer optimization this type of meta-optimization.
A.6 NOTE ON MUTUAL INFORMATION
In the paper used the reverse skill predictor approach estimate the mutual information objective. The alternate decomposition in terms of reduction in future state uncertainty is given by:
p(fs)12,80)
[())))s0) (({s) - p(z, (8), 50) log pl (0)))
In this case tractable approximations of the skill dynamics (2 sn) and marginal dynamics p(s'l |ss)) are required.
In theory, these can be formed exactly as compositions of the policy, skill selection distribution and dynamics model:
log logp((sHlz, sp) = log II p
log pog)(())(so) log E.'je (
The diffieulty in using this formulation in case is due to the marginal future state distribution (s0)) To form the marginal we rely on a finite number K monte carlo samples which is biased and our lower bound mutual information will be reduced by log K. This means that for every given initial state we wish to train on, we must also sample a sufficiently large number of skills from the skill selection distribution form an acceptable marginal approximation, greatly increasing the computation cost compared to using the reverse skill predictor.
A.7 MUTUAL INFORMATION SKLL PREDICTOR NOISE
Although mutual information is a popular objective for skill discovery. it is important to understand its limitations. When the underlying dynamics of the system deterministic, powerful world model may be able to make future predictions with high certainty. Consider the agent then able to communicate the skill identity to the skill predictor through smaller subspace of the state
15

---

Published as a conference paper at ICLR 2021
space. For example, imagine that in quadruped walking task the height of specific end effector at a certain future frame can very confidently predicted by the dynamics model. Whether to lift the end effector up 9cm 10cm may be largely negligible in terms of the dynamics of the motion, but the policy may elect to map the skill variable to the heights of a specific end effector at a certain future frame. When this occurs, diverse skills can become quite indistinguishable visually yet he very distinguishable by the skill predictor. In this failure case, the mutual information may be minimized without truly increasing the diversity of the motion. We use simple idea combat this by artificially adding noise to the input of the skill predictor. The higher the skill predictor input noise is, the harder is distinguish similar trajectories which forces the skills to produce more distinct outcomes.
Figure 10: Example of collapsed skill policy. Here we show skill sequences sampled from a policy where input noise is not added to the skill predictor and sub- sequently the skill policy seems per- form the (turning clockwise), despite inclusion the mutual information objective. Each row is a different latent variable.
motion for different skills
Algorithm 2: CEM subroutine for Algorithm
Function CEM (8, MaxCEMiter, G. H, K, <()));
Sample G initial skills {=1.- H/ from prior p(z)ç(0)) for j in (angeter))EM)ee) do
for g in range(G) G) do for each s; do
T t Imagine trajectories ((s,. from each st.
while r < H, +
if (T - t)-
then
Sample skill |1:[H/K] ~ p(Z
Sample action a, [s(a, 87, // Choose 21 via MPC Observe new state 9,+1 and compute new intrinsic reward r;
Compute value estimates Vg from rewards of each imagined rollout Sort [V1)g=1 and use the top M sequences to update ((t) = (n²), Σ
Sample G skills {2(0) Jg=1 from p(z(ç(i)
return (MxxCEMiter)
16


Reviews:
[{"review_id": "ROUkKMYv2Ba", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an extension of the Dreamer method (Hafner et al., 2019) to incorporate high-level planning via the cross-entropy method and temporally extended skill policies. In theory, this would allow the planning method to operate in an abstract, high-level space and delegate low-level action selection to policies trained on simulated rollouts. If I understand the paper correctly, the result of the LSP learning algorithm is the skill policy and a (fixed?) distribution over a skill variable; no planning is performed at test time. This only became clear when reading the section on further work, since the abstract states \"For this, actions are produced by a policy that is learned over time while the skills it conditions on are chosen using online planning.\". If there is indeed no planning during evaluation I would propose that \"online planning\" is not an apt description of the architecture. I general however, I like how the presented algorithm mixes MI-based skill discovery (which is typically unsupervised) with explicit supervision via long-horizon planning and would regard this a useful contribution to the current literature on hierarchical RL.\n\nMy rating is mainly influenced by the experimental section, as well as several minor issues with the overall presentation. The experiments show that the method is at least as good, and sometimes better, than Dreamer when training from scratch, and universally (although marginally) better when adapting to new tasks.  This is a good result, but some relevant details on the tasks and plots are missing, which make the achievements hard to interpret (see below). The difference to Dreamer is most pronounced on the \"Quadruped Run Pixels\" task, but as mentioned before it's not easy to judge the results; for example, in Figure 8 in the Dreamer paper they report scores of ~750 after 2e6 environment steps. It would also be helpful to provide reference numbers for a standard model-free algorithm on those tasks. The \"Random Skills\" baseline performs very poorly; does it still include the intrinsic rewards to ensure diversity of skills? What would happen if you would drop them so that you basically have Dreamer with an additional random skill input (cf. the \"Switching Ensemble\" baseline from [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://arxiv.org/abs/1909.10618))? Finally, the experiments don't give much insight into the nature of skills that have been discovered. In particular, what do the resulting skill distributions look like?\n\nSeveral aspects of the experimental section are unclear to me and hinder the interpretation of the results:\n- Several crucial details on the environments are missing, such as the reward functions. For \"Quadruped Reach\", is the goal constant or randomly selected for each episode?\n- In Figure 3, what does the x-Axis represent?\n- In Figure 3, tasks like \"Quadruped Run\", \"WalkerRun Pixels\" and \"Quadruped Run Pixels\" don't seem to be trained to convergence.\n- There is a significant drop in performance at x=0.4 for \"Quadruped Reach\" for some seeds.\n- In 4.4., the text for \"Quadruped GetUp Walk -> Reach Goal\" does not line up with the descriptions in 4.1. as it says the pretraining task would be to run as fast as possible rather than getting up and walking.\n- In the same paragraph: \"We see that LSP can adapt much quickly to the transfer task, achieving a reward of 500 only after 70, 000 steps, while Dreamer requires 130, 000 steps to achieve the same reward\". From Figure 4 this appears to be the result for \"Quadruped Walk -> Reach Goal\", and in addition it's also worth noting that Dreamer catches up pretty soon afterwards (even though LSP maintains higher performance throughout training).\n- For the transfer experiments, it would be useful to include the learning curves from Figure 3 so that it is apparent whether it is generally beneficial to adapt a pretrained model as opposed to learning from scratch.\n\nA few more notes on the overall write-up:\n- The site with videos linked to in the abstract is empty\n- In the introduction, it's not immediately clear what \"amortized\" means even though it is mentioned several times. My interpretation is that it means that you use a learned model to train a policy rather than using the model during evaluation. The term could be described more explicitly; in particular I was slightly confused by the notion of \"amortizing over the entire training experience\".\n- At the end of Section 5, regarding Dreamer: \"[...], which is fast even for high-dimensional action spaces, but cannot directly transfer to new environments with different dynamics and reward functions.\" This is in contradiction to the transfer experiments in 4.4. which show a good amount of transfer performance?\n- Figure 2b) and c): $\\phi$ in text vs. $\\Phi$ in figure.\n- In section 3 the reference should be to Figure 2 rather than 2b)?\n- In section 3, \"Low level policy\", it would be good to cite the Dreamer paper once more.\n- In 3.1, \"Behavior Learning.\", the backwards skill predictor is mentioned but not introduced yet. Refer to section 3.2 for a definition?\n- Section 3.2.: it would be good to relate the MI objective to previous methods. The resulting objective looks quite similar to [Variational Intrinsic Control](https://arxiv.org/abs/1611.07507).\n- Section 5: DADS is mentioned, but it might be also be worth noting that they performed experiments with online planning on top of the discovered skills.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8AhVerCR6UY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a model-based RL approach that i) learns a high-level plan generator that produces a sequence of latent continuous variables as skills for a task through CEM, and ii) a low-level policy conditioned on the skills amortized policy training. The approach is shown to be more sample efficient than a recent model-base RL approach (Dreamer), and is able to learn to solve a new (but related) in simple walking tasks relatively faster as well.\n\nStrengths:\n+ The main idea (planning over low-level dimension and learning amortized low-level policy over high-dimension) is intriguing and makes a lot of sense, especially compared to alternative learning schemes such as purely CEM or purely amortized policy training.\n\n+ The approach is well motivated and is easy to understand.\n\n+ The results in simple tasks show promising results over existing approaches. \n\n+ I find the background and the discussion of the connection between existing model-based approaches and the proposed method very helpful.\n\nWeakness:\n- The high-level skill planner / generator p(z |\\cdot) does not seem to rely on the physical states, which is different from typical HRL approaches where the option/subgoal at each step depends on the latest state, e.g., p(z^t | s^t). This seems to suggest that the plan of skills for a task is somewhat fixed regardless of the initial or immediate states. This may work when the environment is simple and does not vary a lot from episode to episode; but I can not see how this formulation can tackle more complex scenarios. Moreover, a fixed hyper-parameter 'K' seems to require additional tuning for new tasks and environments, which may make the approach less 'generalizable.'\n\n- On a related note, it would be good to see more discussion on the advantages of the proposed skill generator/planner over the prior work on HRL. The related work section briefly mentions some prior work, but there is no detailed theoretical and/or empirical comparison. As of now, it is not clear to me how this is better than conventional HRL methods (e.g., learning a high-level policy).\n\n- The main objective of the approach seems to be improving the transferability of skills learned by RL as stated in the main text and in the title. However, the transfer setting is quite simple (from 'learning to walking' to 'learning to walk to a goal location'). I would be more interested in seeing more difficult settings such as walking in new and complex environments (e.g., with obstacles). It is also not clear to me what exactly is transferred here. Is it the low-level policy? Is it the high-level skill plan? Based on my understanding, the authors may be aiming at the former. If so, a transfer setting where the low-level policies are fixed and only the high-level plans are updated / fine-tuned would be far more convincing. After all, I think that is what people would normally expect when you are claiming skill transfer, isn't? In fact, to my knowledge, some HRL approaches have attempted to reuse previously learned & fixed policies (e.g., [1,2]). I'm not saying that these approaches are better, but since the focus is skill transfer, I think a proper discussion/comparison would be critical. \n\nFor rebuttal, please address my concerns listed above. I think the paper presents an interesting and promising idea, but I am not fully convinced that the approach has the advantages as claimed in the paper.\n\nReferences:\n\n[1] Tessler et al., A Deep Hierarchical Approach to Lifelong Learning in Minecraft, AAAI 2017.\n\n[2] Shu et al., Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning, ICLR 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5OyRyt8WlxF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with learning reusable hierarchical skills in sequential decisions.\nThe authors introduce a partially amortized model based on four parametric functions (the representation module, observation module, latent forward dynamic, and task reward module) and an additive loss function composing these modules.\nThe overall algorithm consists of learning the four-modules jointly and to plan using MPC in the latent skill space.\nThe overall paper is well written and justified.\nThe experiments are done in the locomotion domain and show better or comparables results wrt to DREAM.\nDADS is mentioned in the SoA section, it would have been interested to evaluate against it.\nFurthermore, despite its computational complexity as mentioned in the discussion section, I would have been interested to have a comparison or discussion wrt [1].\nOverall the paper is original and sufficiently empirically motivated for acceptance. \nI would request the code to be open-source for better reproducibility.\n\nRefs:\n[1] Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning, Levine and al, 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "1dAI8ZGcfVR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes combining model-based RL with high-level skill learning and composition through hierarchical RL, into a single reinforcement learning framework. More specifically, the proposed approach leverages planning and composing skills in the low-dimensional, high-level representation, and learn low-level skills conditioned on the high-level skills. Only the low-level policies are executed in the environment to generate experiences. A mutual information objective is used to learn low-level policies conditioned on high-level skills, and this was shown to improve sample efficiency as the low-level policies do not learn to ignore the high-level skills they are conditioned on. \n\nThe approach consists of multiple learned components working together. I wonder how stable learning of such dependent separate components is, and how much tuning is required. Can authors comment on the stability of learning, and how their system would perform under different hyperparameter values?\n\nThe experiments show LSP performing better or on par with Dreamer. On the transfer experiments, what guarantees do we have that transfer will always be beneficial? And what guarantees do we have that it will always be more beneficial than using Dreamer? I wonder if transfer was only beneficial in the 2 experiments shown? What about the other combination of tasks? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a unified model-based framework for high-level skill learning and composition through hierarchical RL. The proposed approach combines high-level planning in a low dimensional space with low-level skill learning, where each low-level skill is a policy conditioned on the high-level task. The low-level policies are learned by using a mutual information objective. The proposed approach is evaluated on locomotion tasks, and is shown to be overall more data efficient than alternative baselines.
The reviewers agree that this work is original and sufficiently empirically motivated for acceptance. Two reviewers were concerned by the experimental setup and the transfer setting that are somehow too simple, but the authors fixed these issues in the improved version based on the feedback.
Decision: Accept (Poster)