Learning transferable motor skills with hierarchical latent mixture policies
Authors: Dushyant Rao
Fereshteh Sadeghi
Leonard Hasenclever
Markus Wulfmeier
Martina Zambelli
Giulia Vezzani
Dhruva Tirumala
Yusuf Aytar
Josh Merel
Nicolas Heess
raia hadsell
Venue: ICLR 2022 Spotlight (2022)
Keywords: Robotics
Reinforcement Learning
Hierarchical
Latent Variable Models
Skills
Transfer

Abstract: For robots operating in the real world, it is desirable to learn reusable abstract behaviours that can effectively be transferred across numerous tasks and scenarios.
We propose an approach to learn skills from data using a hierarchical mixture latent variable model.
Our method exploits a multi-level hierarchy of both discrete and continuous latent variables, to model a discrete set of abstract high-level behaviours while allowing for variance in how they are executed.
We demonstrate in manipulation domains that the method can effectively cluster offline data into distinct, executable behaviours, while retaining the flexibility of a continuous latent variable model.
The resulting skills can be transferred to new tasks, unseen objects, and from state to vision-based policies, yielding significantly better sample efficiency and asymptotic performance compared to existing skill- and imitation-based methods.
We also perform further analysis showing how and when the skills are most beneficial: they encourage directed exploration to cover large regions of the state space relevant to the task, making them most effective in challenging sparse-reward settings.

Published as a conference paper at ICLR 2022
LEARNING TRANSFERABLE MOTOR SKILLS WITH HIERARCHICAL LATENT MIXTURE POLICIES
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merelī Nicolas Heess, & Raia Hadsell DeepMind, London, UK
ABSTRACT
For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an approach to learn abstract motor skills from data using hierar- chical mixture latent variable model. In contrast to existing work, our method exploits a three-level hierarchy of both discrete and continuous latent variables, to capture a set of high-level behaviours while allowing for variance in how they are executed. We demonstrate manipulation domains that the method can effec- tively cluster offline data into distinct, executable behaviours, while retaining the flexibility of continuous latent variable model. The resulting skills can be trans- ferred and fine-tuned on new tasks, unseen objects, and from state to vision- based policies, yielding better sample efficiency and asymptotic performance compared to existing skill- and imitation- based methods. We further analyse how and when the skills are most beneficial: they encourage directed exploration to cover large regions of the state space relevant to the task, making them most effective in chal- lenging sparse-rewart settings.
INTRODUCTION
Reinforcement learning is powerful and flexible paradigm to train embodied agents, but relies large amounts of agent experience, computation, and time, on each individual task. Learning each task from scratch is inefficient: it is desirable to learn a set of skills that can efficiently be reused and adapted to related downstream tasks. This is particularly pertinent for real-world robots, where inter- action is expensive and data-cfficiency crucial. There are numerous existing approaches to learn transferable embodied skills, usually formulated as a two-level hierarchy with a high-level controller and lowelevel skills. These methods predominantly represent skills as being either continuous, such as conditond (Lynch et al., [019, |errtcch et al.] (0206) or latent space policies (Haarnoja jet al., 2018 Merel et al. 2019 Singh et al. (022)); ); or discrete, such as mixture or option-based
methods (Sutton et al.] 1999| Daniel et 2012 Florensa et al. [017] Wulfmeier et al. 2021). Our
goal is to combine these perspectives to leverage their complementary advantages.
We propose an approach to learn a three-level skill hierarchy from offline dataset, capturing both discrete and continuous variations at multiple levels of behavioural abstraction. The model com- prises low-level latent-conditioned controller that can learn motor primitives, set of continuous latent mid-level skills, and a discrete high-level controller that can compose and select among these abstract mid-level behaviours. Since the mid- and high-level form mixture, we call method Hierarchical Latent Mixtures of Skills (HeLMS). We demonstrate on challenging object manipula- tion tasks that our method can decompose a dataset into distinct, intuitive, and reusable behaviours. We show that these skills lead to improved sample efficiency and performance in numerous trans- fer scenarios: reusing skills for new tasks, generalising across unseen objects, and transferring from state to vision-based policies. Further analysis and ablations reveal that both continuous and discrete components are beneficial, and that the learned hierarchical skills are most useful in sparse- settings, as they encourage directed exploration of task-relevant parts of the state space.
"Corresoondigg author. Email: detphyaantdddeemminne.. Work done while DeepMind

---

Published as a conference paper at ICLR 2022
Our main contributions are follows:
We propose a novel approach to learn skills at different levels of abstraction from an offline dataset. The method captures both discrete behavioural modes and continuous variation using a hierarchical mixture latent variable model.
We present two techniques to reuse and adapt the learned skill hierarchy via reinforce- ment learning in downstream tasks, and perform extensive evaluation and benchmarking in different transfer settings: to new tasks and objects, and from state to vision-based policies.
We present detailed analysis to interpret the learned skills, understand when they are most beneficial, and evaluate the utility of both continuous and discrete skill representations.
RELATED WORK
A long-standing challenge in reinforcement learning is the ability to learn reusable motor skills that can be transferred efficiently to related settings. One way to learn such skills is via multi-task rein- forcement learning (Heess et al. 2016: James et al., 2018 Hausman al. 2018 Riedmiller al.) 2018). with the intuition that behaviors useful for a given task should aid the learning of related tasks. However, this often requires careful curation of the task set, where each skill represents a separate task. Some approaches avoid this by learning skills in an unsupervised manner using intrinsic objec- tives that often maximize the entropy of visited states while keeping skills distinguishable (Greegrr et al. 2017 Eysenbach et al. 2019. Sharma et al., 2019 Zhang et al. 22220).
A large body of work explores skills from the perspective of unsupervised segmentation of repeat- able behaviours in temporal data (Niekum & Barto [201]: Ranchod et al. [2015] Krüger et al.) 2016
Lioutikov et al., 2017 Shiarlis et al. 2018]] Kipf et al. [019]] Tanneberg et al. 2021 Other works
investigate movement motor primitives that can be selected or sequenced together to solve com-
plex manipulation or locomotion tasks (Mülling et al. 2013: Rueckert et al. 2015 Lioutikov et al.]
2013 Paraschos et al. 2018, Merel et al.] 2020; Tosatto et al.] 2021] Dalal et al.) 2021). Some
of these methods also employ mixture models to jointly model low-level motion primitives and a
high-level primitive controller (Muelling al.) 2010 Colomé & Torras 2018 Pervez & Lee 2018):
the high-level controller can also be implicit and decentralised over the low-level primitives Goyal
al. 2019).
Several existing approaches employ architectures in which the policy is comprised of two (or more) levels of hicrarchy. Typically, low-level controller represents the learned set of skills, and a high-level policy instructs the low-level controller via a latent variable or goal. Such latent vari- ables can be discrete (Florensa et al., 2017 Wulfmeier et al.) 2020) or continuous (Nachum et al. 2018 Haarnoja et al.] (2018) and regularization of the latent space is often crucial (Tirumala et al., 2019. 1. The latent variable can represent the behaviour for one timestep, for a fixed number of timesteps (Ajay et al.) 2021), or options with different durations (Sutton et al. 1999; Bacon et al.) 2017 Wulfmeier et al. 202T). One such approach that is particularly relevant (Florensa et al. 2017) learns diverse set of skills, via discrete latent variable that interacts multiplicatively with the state to enable continuous variation in a Stochastic Neural Network policy; this skill space is then trans- ferred to locomotion tasks by learning a new categorical controller. Our method differs in few key aspects: our proposed three-level hierarchical architecture explicitly models abstract discrete skills while allowing for temporal dependence and lower-level latent variation in their execution, enabling diverse object-centric behaviours in challenging manipulation tasks.
Our work is related to methods that learn robot policies from demonstrations (LfD, e.g. (Rajeswaran eet al. 2018 Shiarlis el al. 2018 Strudel et al. (0200) or more broadly from logged data (offline RL, e.g. [Wu et al. 2019 Kumar et al. 2020 Wang et al. 2020 While many of these focus on learning single-task policies, several approaches learn skills offline that can be transferred online
to new tasks (Merel et al., 2019 Lynch et al. 2019 Pertsch et al., 2020a Ajay et al., 2021 Singh et al. 2021). These all train two-level hierarchical model, with a high-level encoder that maps to a continuous latent space, and a low-level latent-conditioned controller. The high-level encoder can encode whole trajectory (Pertsch et al. 2020a 2021: Ajaa et al.] 2021); short look-ahead state sequence (Merel et al. 2019); the current and final goal state (Lynch et al., 2019); or can even be simple isotropic Gaussian noise (Singh et al. 2021) that can be flexibly transformed by a flow-based low-level controller. At transfer time, a new high-level policy is learned from scratch: this can be more efficient with skill priors (Pertsch et al. 2020a) or temporal abstraction (Ajay et al. 2021
2

---

Published as a conference paper at ICLR 2022
y
Z
a
Vt-1
2-byer
2-bayrr
2-bayo
ML LL
X
RL
x
ML
X,
(a)
(b)
Figure 1: (a) Graphical model for HeLMS, with solid lines indicating the underlying generative model (prior) and dashed lines indicating dependencies introduced by the inference model (poste- rior). (b) Network architecture, showing the high-, mid-, and low-level networks from left to right, respectively. As indicated by superscripts, different subsets of the input state x can be provided to the high level LH,,, mid level (ML), and low level (LL) peormation-asymmetry.
HeLMS builds on this large body of work by explicitly modelling both discrete and continuous behavioural structure via a three-level skill hierarchy. We use similar information asymmetry to Neural Probabilistic Motor Primitives (NPMP) (Merel et al. 2019 2020), conditioning the high- level encoder on short look-ahead trajectory. However HeL.MS explicitly captures discrete modes of behaviour via the high-level controller, and learns additional mid-level which is able to transfer abstract skills to downstream tasks, rather than learning continuous latent policy from scratch.
METHOD
This paper examines two-stage problem setup: an offline stage where hierarchical skill space learned from dataset, and online stage where these skills transferred to reinforcement learning setting. The dataset D comprises a set of trajectories, each a sequence of state-action pairs (x=:a:)==0- The model incorporates discrete latent variable y: E {1,..., K} high-level skill selector (for a fixed number skills K), and a mid-level continuous variable 2, E "R conditioned on Y, which parameterises each skill. Marginally, 2, is then latent mixture distribution represent- ing both a discrete set of skills and the variation in their execution. A sample of z; represents an abstract behaviour, which then executed low-level controller p(a, Zt, xz). The learned skill space can then be transferred to reinforcement learning agent и in a Markov Decision Process de- fined by tuple AS T, R. :): these represent the state, action, and transition distributions, reward function, and discount factor respectively. When transferring, we train a new high-level controller that acts either at the level discrete skills y, or continuous Zt, and freeze lower levels of the policy.
We explain our method in detail in the following sections.
3.1 LATENT MIXTURE SKILL SPACES FROM OFFLINE DATA
Our method employs the generative model Figure Ta As shown, the state inputs can be different for each level of the hierarchy, but keep notation uncluttered, refer to all state inputs x, and the specific input can be inferred from context. The joint distribution of actions and latents over trajectory decomposed into latent prior p(yo.T, Z1:T) and low-level controller p(a, Zr, x,):
T
p(arT Maa...No.TT Zier x1:T) = p(yo. ZTTT) II
p(as 2,, xi) (=1
T
p(yorr Birr) = p(yo) II ply: \y-r))((z, Iy))
(1)
Intuitively, the categorical variable y, can capture discrete modes behaviour, and the continuous latent 2, is conditioned this to vary the execution of each behaviour. Thus, 2, follows mixture

---

Published as a conference paper at ICLR 2022
distribution, encoding all the relevant information desired abstract behaviour for the low-level controller p(ay Zt, X²). Since each categorical latent y; is dependent on Yt-1, and 2, is only depen- dent yt. this prior can be thought of as Hidden Markov model over the sequence of Z1.T.
To perform inference over the latent variables, we introduce the variational approximation:
g(Vorr Zi.r X1:T) p(y)) II glyx yy-1,, 1 ly., X.) 1=1
(2)
Here, the selection of skill Ys q(y: Xx) is dependent on that of the previous timestep (allowing for temporal consistency), as well as the input. The mid-level skill is then parameterised by 2, g(zz| yt,X,) based on the chosen skill and current input. p(yo) and p(ye y--1) model
skill prior and skill transition prior respectively, while p(z: yt) represents a skill parameterisation prior to regularise each mid-level skill. While all of these priors can be learned in practice, we only found it necessary to learn the transition prior, with a uniform categorical for the initial skill prior and simple fixed (N0 /) prior for p(zz yy).
Training via the Evidence Lower Bound The proposed model contains number of components with trainable parameters: the prior parameters v/2 = {&aa by for the low-level controller and categorical transition prior respectively; and posterior parameters á = ( Φz for the high-level controller and mid-level skills. For a trajectory {XI.T a1:T } ~ D, we can compute the Evidence Lower Bound for the cate-conditional action distribution, ELBO S p(a1:r X1:T), follows:
ELBO B (Poer mie X1rr) log (a1:T, Vo.T, Zi.T X1.)) log go (yo.r, Zi.T Xxr)
persosmpmmnt action
percompuneet KL regulasiser
Σ eddve/xIsl log log (as 2 afree) x:) -3 KL (2: yr, Xi) pp 1x1)
yt
T
B
Σ 9(y--1 x1=-1) KL (go, (ye Nyc-1. xx) || Ip,, (yre (xe-)))
(3)
categorical regulariser
where 2/Dre) (zz X1). The coefficients By and Bg can be used to weight the KL terms, and the cumulative component probability g(ve x1x)) can be computed iteratively as q(yr [xae = x). In other words, for each timestep and each mixture
Ey. 90, I ,r,,,,,, component, we compute the latent sample and the corresponding action prg-rbbbbillty, and the KL-divergence between the component posterior and prior. This is then marginalised over all y.. with an additional KL. over the categorical transitions. For more details, see Appendix C
information-asymmerr As noted in previous work (Tirumala et al.) 2019 Galashov et al.) 2019), hierarchical approaches often benefit from information-asymmetyy with higher levels seeing addi- tional context or task-specific information. This ensures that the high-level remains responsible for abstract, task-related behaviours, while the low-level executes simpler motor primitives. We employ similar techniques in HcLMS: the low-level inputs X1.1. comprise the proprioceptive state of the em- bodied agent; the mid-level inputs I. also include the poses of objects in the environment; and the high-level X concatenates both object and proprioceptive state for short number of lookahead timesteps. The high- and low-level are similar to (Merel et al. 2019). with the low-level controller enabling motor primitives based on proprioceptive information, and the high-level using the looka- head information to provide additional context regarding behavioural intent when specifying which skill to use. The key difference is the categorical high-level and the additional mid-level, with which HeLMS can learn more object-centric skills and transfer these to downstream tasks.
Network architectures The architecture and information flow in HeLMS are shown in Figure Tb The high-level network contains a gated head, which uses the previous skill y:-1 to index into one of K categorical heads, each of which specify a distribution over y.. For a given y., the corresponding mid-level skill network is selected and used to sample latent action 2,, which is then used as input for the iieent-conditooned lowelevel controller, which parameterises the action distribution. The skill transition prior p(y: y1-1) is also learned, and is parameterised as linear softmax layer which takes in a one-hot representation of y and outputs the distribution over yr. All components are trained end-to-end via the objective in Equation 3
4

---

Published as a conference paper at ICLR 2022
3.2 REINFORCEMENT LEARNING WITH RELOADED SKILLS
Once learned, we propose two methods to transfer the hierarchical skill space to downstream tasks. Following previous work (e.g. (Merel al. 2019 Singh al. 2021)) we freeze the low-level controller p(a, Z,, Xt), and learn a policy for either the continuous (z₂) or discrete (y,) latent.
One simple and effective technique to additionally freeze the mid-level com- ponents g(2, Yt, Xt), and learn a categorical high-level controller <y,, xx) for the downstream task. The learning objective is given by:
Categorical agent
J = Es Σ of (r: KL xx) [ xx)))
(4)
where the standard discounted return objective in RL augmented by an additional term performing KL-regularisation to some prior To scaled by coefficient 77y This could be any categorical distribu- tion such as the previously learned transition prior p(y, Yr-1) ) but in this paper we regularise the uniform categorical prior to encourage diversity. While any RL algorithm could be used to optimize Ty: xt), in this paper we use MPO (Abdolmaleki et al. 2018) with a categorical action distribu- tion (see Appendix B for details). We hypothesise that this method improves sample efficiency by converting a continuous control problem into a discrete abstract action space, which may also aid in credit assignment. However, since both the mid-level components and low-level are frozen, it can limit flexibility and plasticity, and also requires that all of the mid- and low-level input states are available in the downstream task. We call this method MeLMS-cat.
Mixture agent A more flexible method of transfer is to train a latent mixture policy, aTz xxx)= = Ey. (y: xz) (Zz x,). In this case, the learning objective is given by:
J B Σ 7 7- ny ( xx) moly, xx)) m= Σ (L | Xe) |l 1y,, 1x)))))))] (5)
where in addition to the categorical prior, also regularise each mid-level skill to corresponding prior To (ze] Xt). While the priors could be any policies, we set them to be the skill posteri- g(2, X,) learned offline, ensure the mixture components remain close to the pre-learned skills. This is related to (Tirumala et al. 2019), which also applies KL-rregularisatonn at multiple levels of a hierarchy. While the high-level controller π (y: xx) learned from scratch, the mixture components can also be initialised to q(z; yt, X;),,, to allow for initial exploration over the space of skills. Alternatively, the mixture components can use different inputs, such as vision: this setup allows vision-bass skills to be learned efficiently by regularising to state based skills learned of- fline. We optimise this using RHPO (Wulfmeier al. [2020). which employs a similar underlying optimisation to MPO for mixture policies (see Appendix B for details). We call this HeL.S-min
4 EXPERIMENTS
Our experiments focus on the following questions: (1) Can we learn hierarchical latent mixture skill space of distinct, interpretable behaviours? (2) How do we best reuse this skill space to improve sample efficiency and performance downstream tasks? (3) Can the learned skills transfer effec- tively to multiple downstream scenarios: (i) different objects; (ii) different tasks; and (iii) different modalities such dision-based policies? (4) How exactly do these skills aid learning of downstream manipulation tasks? Do they aid exploration? Are they useful in sparse or dense reward scenarios?
4.1 EXPERIMENTAL SETUP
Environment and Tasks We focus on manipulation tasks, using a MuJoCo-based environment with single Sawyer arm, and three objects coloured red, green, and blue. We follow the challenging object stacking benchmark of Lee et al. (202I). which specifies five object sets (Figure 2). carefully designed to have diverse geometries and present different challenges for stacking agent. These range from simple rectangular objects (object set 4), to geometries such as slanted faces (sets and 2) that make grasping stacking the objects more challenging. This environment allows us
5

---

Published as a conference paper at ICLR 2022
(a) Set 1
(b) Set
(c) Set
(d) Set 4
(e) Set 5
Figure 2: The object sets (triplets) used for our experiments, introduced by (Lee et al.
202
0:30
500
601
0.4
6.00
8004
632
(a)
(c)
Figure 3: (a) Image sequences showing example rollouts when fixing the discrete skill (different for each row) and running the mid- and low-level controllers in the environment. Each skill exccutes different behaviour, such as lifting (top row), reach-to-red (middle row), grasping (bottom) (b)
The learned skill transition prior p(y: 1 (c) Histogram showing the use of different skills.
to systematically evaluate generalisation of manipulation behaviours for different tasks interacting with geometrically different objects. For further information, we refer the reader to Appendix D.I to (Lee et al. 2021). Details of the rewards for the different tasks are also provided in Appendix
Datasets To evaluate our approach and baselines in the manipulation settings, we use two datasets:
this data is collected by an agent trained to stack the red object on the blue object and ignore the green one, for the simplest object set, set4. htt_pparrsstackiiie similar to the previous case, but with all six pairwise stacking combinations of (red, green, blue}, and covering all of the five object sets.
red_on _a__oo__lllll
Baselines
For evaluation in transfer scenarios, we compare HeLMS with a number of baselines:
From scratch: We learn the task from scratch with MPO, without an offline learning phase. We compare against NPMP (Merel al. 2019), which is the most simi- and policy conditioning. We make some small changes to the originally proposed method, and also apply additional KL-regularisation to the latent prior: we found this to improve performance significantly in our experiments. For more details and an ablation, see Appendix
NPMP+KL:
lar skill-based approach in terms of informatioon-syymment
Behaviour Cloning (BC): We apply behaviour cloning to the dataset, and fine-tune this policy via MPO the downstream task. While the actor is initialised to the solution obtained via BC, the critic still needs to be learned from scratch.
Hierarchical BC: We evaluate a hierarchical variant of BC with a similar latent space to NPMP using a latent Gaussian high-level controller. However, rather than freezing the low-level and learning just high-level policy, Hierarchical BC fine-tunes the entire model. Asymmetric actor-critic: For state-to-vision transfer, HeLMS uses prior skills that depend on object states to learn purely vision-based policy. Thus, we also compare against variant of MPO with asymmetric actor-critic (Pinto et al. 2017) setup, which uses object states differently: to speed up learning of the critic, while still Iearning a vision-based actor.
4.2 LEARNING SKILLS FROM OFFLINE DATA
We first aim to understand whether we can learn a set of distinct and interpretable skills from data (question (1)). For this, train MeLMS the red_ wwwooonwuueeeeoeeeeoeoeeeee celll___tccl ng dataset with 5 skills.

---

Published as a conference paper at ICLR 2022
HeLMS-mix
HeLMS
NPMP+KL
Hierarchical
250
250
300
200
120
120
50
1e6
1e0
(a) Set
(h) Set 2
(c) Set
(d) Set
Figure 4: Performance when transferring to the red-on- blue stacking task using staged sparse reward with every unseen object set.
HelLffS-mix
HeLMS-cat
MPO
NPMP4KL
HeLMS-mix,
MPO. asymi
250
200
Reward
150
100
50
1.0
15
Environment
(a)
(b)
Figure 5: (a) Performance pyramid task; and (b) image sequence showing episode rollout from a learned solution this task (left-to-right, top-to-bottom).
Figure 6:
Performance for vision-based stacking.
Figure shows some example episode rollouts when the learned hierarchical agent executed in the environment, holding the high-level categorical skill constant for an episode. Each row rep- resents a different skill component, and the resulting behaviours are both distinct and diverse: for example, lifting skill (row 1) where the gripper closes and rises up, reaching skill (row 2) where the gripper moves to the red object, or a grasping skill (row 3) where the gripper lowers and closes its fingers. Furthermore, without explicitly encouraging this, the emergent skills capture temporal con- sistency: Figure 3b shows the learned prior p(y: y+-1) (visualised as a transition matrix) assigns high probability along the diagonal (remaining in the same skill). Finally, Figure 3c demonstrates that all skills are used, without degeneracy.
4.3 TRANSFER TO DOWNSTREAM TASKS
Generalising different objects We next evaluate whether the previously learned skills (i.e. trained on the simple objects in set 4) can effectively transfer to more challenging object interaction scenarios: the other four object sets proposed by (Lee et al. (02T). The task uses sparse staged reward, with reward incrementally given after completing each sub-goal of the stacking task. As shown in Figure 4 both variants of HeLMS learn significantly faster than baselines on different object sets. Compared to the strongest baseline (NPMP), HeLMS reaches better average asymptotic performance (and much lower variance) on two object sets (1 and 3), performs similarly on set 5, object set potentially highlights trade-off
and does poorer object set 2. The performance between incorporating higher-level abstract behaviours and maintaining low-level flexibility: this object set often requires reorientation of the bottom object due its slanted faces, behaviour that is not common in the offline dataset, which might require greater adaptation of mid- and low-level skills. This is an interesting investigation we leave for future work.
Compositional reuse of skills To evaluate whether the learned skills are composable for new tasks, train HeLMS the wwwppparrsss__eee _staiiinggggg. dataset with 10 skills, and transfer to pyramid task. In this setting, the agent has to place the red object adjacent to the green object, and stack the blue object top to construct pyramid. The task specified via sparse staged reward

---

Published as a conference paper at ICLR 2022
HeLMS-cat
NPMP+KL
Hierarchical
100
253
120
320
00
0.
3:0
0.2 1.00 1.3
(a) Dense reward
(b) Sparse staged reward
(c) Sparse reward
Figure 7: Transfer performance on a stacking task with different reward sparsities. Left: dense
reward, Centre: staged sparse reward, Right: sparse reward
for each stage sub-task: reaching, grasping, lifting, and placing the red object, and subsequently the blue object. In Figure[{a)) we plot the performance of both variants of our approach, as well as NPMP and MPO; omit the BC baselines this involves transferring to fundamentally different task. Both HeLMS mix and HeLMS-cat reach a higher asymptotic performance than both NPMP and MPO, indicating that the learned skills be better transferred to different task. We show an episode rollout in Figure (bb) in which the learned agent can successfully solve the task.
From state vision-based policies While our method learns skills from proprioception and ob- ject state, we evaluate whether these skills can be used to more efficiently learn vision- basoed policy. This is invaluable for practical real-world scenarios, since the agent acts from pure visual observa- tion at test time without requiring privileged and often difficult-to-obaain object state information.
We use the HeLMS mix variant to transfer skills to a vision-based policy, by reusing the low-level controller, initialising prioception as input), and KL-reaulaiiiing these to the previously learned state-based skills. While the learned policy is vision-based, this Kl.-regularisation still assumes access object states during training. For a fair comparison, we additionally compare our approach with a version of MPO using an asymmetric critic (Pinto al. 2017), which exploits object state information instead of vision in the critic, and also use this for HeLMS. As shown in Figure 6 learning a dision-based policy with MPO from scratch is very slow and computationally intensive, but an asymmetric critic significantly speeds up learning, supporting the empirical findings Pinto et al. (2077). However, HeLMS once again demonstrates better sample efficiency, and reaches slightly better asymptotic performance. We note that this uses the same offline model as for the object generalisation experiments, showing that the same state-based skill space can be reused numerous settings, even for vision-based tasks.
high-level controller and mid-level latent skills (with vision and pro-
4.4 WHERE AND HOW CAN HIERARCHICAL SKILL REUSE BE EFFECTIVE?
Sparse reward tasks We first investigate how HeLMS performs for different rewards: a dense shaped reward, the sparse staged reward from the object generalisation experiments, and fully sparse reward that only provided after the agent stacks the object. For this experiment, we use the skill space trained
red_onn salu__stacceee on object set 4. The results are shown in Figure 7 With a dense reward (and no object transfer required), all of the approaches baselines all plateau at a lower performance, with the exception of NPMP, as previously discussed. However, for the challenging fully-sparse scenario, HeLMS is the only method that achieves non- zero reward. This neatly illustrates the benefit of the proposed hierarchy of skills: it allows for directed exploration which ensures that even sparse rewards can be encountered. This is consistent with observations from prior work in hierarchical reinforcement learning (Florensa et al.) [017]] Nachum et al. 2019), and we next investigate this claim in more depth for our manipulation setting.
and transfer to the same RL task of stacking
successfully learn the task. With the sparse staged reward, the
Exploration To measure whether the proposed approach leads to more directed exploration, record the average coverage in state space at the start of RL (i.e. zero-shot transfer). This is com- puted as the variance (over episode) of the state xt, separated into three interpretable groups:
8

---

Published as a conference paper at ICLR 2022
Disortt
Continuuus only
Reward
State coverage (x10-
Method
Dense
Staged
Joints
Grasp
Objects
MPO
3.16
0.0
8.72
0.004
1.21
203
Ronur
NPMP
3.67
0.0
3.43
0.0
1.45
BC
31.68
0.004
4.22
0.05
1.52
Hier. BC
16.42
0.004
2.61
0.04
1.31
29
10
02 22
HcLMS
20.46
0.05
2.98
1.10
1.61
fromes
(a)
(b)
Table 1: Analysis of zero-shot exploration at the start of RI,, in terms of reward and state coverage (vari- ance over an episode different subsets of the agent's state). Results averaged over 1000 episodes.
Figure 8: Ablation for continuous and dis- crete components during offline learning, when transferring to the (a) easy case (ob- ject set 4) and (b) hard case (object set 3).
joints (angles and velocity), grasp (a simulated grasp sensor), and object poses. We also record the total reward (dense and sparse staged). The results are reported in Table [ While all approaches achieve some zero-shot dense reward (with BC the most effective), HeL receives a sparse staged reward an order of magnitude greater. Further, in this experiment we found it was able to achieve the fully sparse reward (stacked) in one episode. Analysing the state coverage results, while other methods are able to cover the joint space more (e.g. by randomly moving the joints), HeLMS is nearly two orders of magnitude higher for grasp states. This indicates the utility of hicrarchical skills: by acting over the space of abstract skills rather than low-level actions, HeLMS performs directed exploration and targets particular states of interest, such as grasping an object.
4.5 ABLATION STUDIES
Capturing continuous and discrete structure To evaluate the benefit of both continuous and dis- crete components, we train our method with a fixed variance of zero for each latent component (i.e. 'discrete-only') and transfer to the stacking task with sparse staged reward in an easy case (object set 4) and hard case (object set 3), as shown in Figure (igureB[a) and (b). We also evaluate the 'continuous- only' case with just single Gaussian to represent the high- and mid-level skills: this is equivalent to the IPM baseline. We observe the the discrete component alone leads to improved sam- ple efficiency in both cases, but modelling both discrete and continuous latent behaviours makes a significant difference in the hard case. In other words, when adapting to challenging objects, it is important to capture discrete skills, but allow for latent variation in how they are executed.
KL-regularisation We also perform ablation for KL-regularisation during the offline phase (via (2) and online RL (via 72), to gauge the impact on transfer; see Appendix A. for details.
5 CONCLUSION
We present HeLMS, an approach to learn transferable and reusable skills from offline data using a hierarchical mixture latent variable model. We analyse the learned skills to show that they effectively cluster data into distinct, interpretable behaviours. We demonstrate that the learned skills can be flexibly transferred to different tasks, unseen objects, and to different modalities (such from state to vision). Ablation studies indicate that it is beneficial to model both discrete modes and continuous variation in behaviour, and highlight the importance of KL-regularisation when transferring to RL and fine-tume tuning the entire mixture of skills. We also perform extensive analysis to understand where and how the proposed skill hierarchy can be most useful: we find that is particularly invaluable in sparse reward settings due to its ability to perform directed exploration.
There are a number of interesting avenues for future work. While our model demonstrated temporal consistency, it would be useful to more actively encourage and exploit this for sample-efficient transfer. would also be useful to extend this work to better fine-tune lower level behaviours, to allow for flexibility while exploiting high-level behavioural abstractions.
Note that MS-cat and Mix are identical for this analysis: at the start of reinforcement learning, both variants transfer the mid-level skills while initialising new high-level controller.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like to thank Coline Devin for detailed comments on the paper and for generating the all_pairs srs_sssccinngg dataset. We would also like to thank Alex X. Lee and Konstantinos Bousmalis for help with setting up manipulation experiments. We also grateful reviewers for their feedback.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learn- ing Representations, 2018.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline
primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Reprexentations, 2021.
Pierre-Lur Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Adrià Colomé and Carme Torras. Dimensionality reduction in learning gaussian mixture models of movement primitives for contextualized action selection and adaptation. IEEE Robotics and Automation Letters, 3(4):3922-3929) 2018.
Murtaza Dalal, Deepak Pathak, and Ruslan Salakhutdinov. Accelerating robotic reinforcement
learning via parameterized action primitives. In Thirty- Fifth Conference on Neural Information Processing Systems, 2021.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pp. 273-281. PMLR, 2012.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representa- tions (ICLR), 2019.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint arXiv: 2017.
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-regularized RL. In International Conference on Learning Representations, 2019.
Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, and Yoshua Ben- gio. Reinforcement learning with competitive ensembles of inforrmtiionconstrainne primitives. In International Conference on Learning Representations, 2019.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In Inter- national Conference on Learning Representations, 2017.
Tuomas Haamoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 1851-1860, 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference on Lear- ing Representations (ICLR), 2018.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610, 05182, 2016.
Stephen James, Michael Bloesch. and Andrew J Davison. Task-Embedded control networks for Few-Shot imitation learning. In Conference Robot Learning (CoRI.), 2018
10

---

Published as a conference paper at ICLR 2022
Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez- Gonzalez, Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference Machine Learning, pp. 3418-33188 3428..2. PMLR. 2019.
Bjöm Krüger, Anna Vögele, Tobias Willig, Angela Yao, Reinhard Klein, and Andreas Weber. Effi- cient unsupervised temporal segmentation of motion data. IEEE Transactions Multimedia, 19 (4):797-812, 2016.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint 20t.:20900.07779 2020.
Alex X. Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Jost Tobias Springenberg. Kon- stantinos Bousmalis, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, Claudio Fantacci, Jose Enrique Chen, Akhil Raju, Rae Jeong, Michael Neunert, Antoine Laurens, Stefano Saliceti, Federico Casarini, Martin Riedmiller, Raia Hadsell, and Francesco Nori. Beyond pick-and-place: Tackling robotic stacking of diverse shapes. In 5th Annual Conference Robot Learning, 2021.
Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, and Jan Peters. Probabilistic segmentation applied to an assembly task. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 533-540. IEEE, 2015.
Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, and Jan Peters. Learning movement primi- tive libraries through probabilistic segmentation. The International Journal of Robotics Research, 36(8):879-894, 2017.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning (CoRL), 2019.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations (ICLR), 2019.
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: Reusable neural controllers for vision-guided Whole-Body tasks. In SIGGRAPH 2020, 2020.
Katharina Muelling, Jens Kober, and Jan Peters. Learning table tennis with a mixture of motor primitives. In 2010 10th IEEE-RAS International Conference Humanoid Robots, pp. 411-416. IEEE, 2010.
Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3): 263-279, 2013.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In NeurlPS, 2018.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv: 10618, 2019.
Scott Niekum and Andrew G Barto. Clustering via dirichlet process mixture models for portable skill discovery. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
Alexandros Paraschos, Christian Daniel, Jan Peters, and Gerhard Neumann. Using probabilistic movement primitives in robotics. Autonomous Robots, 42(3):52--551, 2018.
Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.1 1944, 2020a.

---

Published as a conference paper at ICLR 2022
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Shenghao Zhou, Dinesh Jayaraman, Chelsea Finn, and Sergey Levine. Long-hoe Long-horizon visual planning with coal-conditioned hierarchical predictors. Ad- vances in Neural Information Processing Systems, 33, 2020b.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J Lim. Demonstration reinforcement learning with learned skills. July 2021.
Affan Pervez and Dongheui Lee. Learning pas-parraeteriized dynamic movement primitives using mixture of gmms. Intelligent Service Robolics, 11(1):61-78, 2018.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. arXiv preprint 17X0v:1710.06522, 2017.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems XIV, Robotics: Science and Systems Foundation, June 2018.
Pravesh Ranchod, Benjamin Rosman, and George Konidaris. Nonparametric bayesian reward seg- mentation for skill discovery using inverse reinforcement learning. In 2015 JEEE/RSJ Interna- tional Conference on Intelligett Robots and Systems (IROS), pp. 471 IEEE, 2015.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Proceedings of the 35th International Conference on Machine Learning, pp. 4344-4353, 2018.
Elmar Rueckert, Jan Mundo, Alexandros Paraschos, Jan Peters, and Gerhard Neumann. Extracting low-dimensional control variables for movement primitives. 2015 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 1511-1515 IEEE, 2015.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:/907 01657, 2019.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. International Conference on Machine Learning, pp. 4654-4663, PMLR, 2018.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Par- rot: Data-Driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021.
Robin A M Strudel, Alexander Pashevich, Igor Kalevatykh, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Learning combine primitive skills: A step towards versatile robotic manipulation. In International Conference on Robotics and Automation (ICRA), 2020.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligncee 112(1-2):181- 211, 1999.
Daniel Tanneberg, Kai Ploeger, Elmar Rueckert, and Jan Peters. Skid raw: Skill discovery from raw trajectories. IEEE Robotics and Automation Letters, 6(3):4696-4703, 2021.
D Tirumala, H Noh, A Galashov, I. Hasenclever, and others. Exploiting hierarchy for learning and transfer kl-regularized rl. arXiv preprint arXiv, 2019.
Samuele Tosatto, Georgia Chalvatzaki, and Jan Peters. Contextual latent-movements off-policy optimization for robotic manipulation skills. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 10815-10821. IEEE, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
12

---

Published as a conference paper at ICLR 2022
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11300 2019.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Ne- unert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin A Riedmiller. Compositional transfer in hierarchical reinforcement learning. In Robotics: Science and Systems, 2020.
Markus Wulfmeier, Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim Her- tweck, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Heess, al. Data-efficient hind- sight off-policy option learning. In International Conference on Machine Learning, pp. 11340- 11350. PMLR, 2021.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A ADDITIONAL EXPERIMENTS
A.1 ABLATIONS FOR KL- KL-REGGULARISATION
In these experiments, we investigate the effect of KL-regularisation on the mid-level components, both for the offline learning phase (regularising each component to p(z y.) = (N, 7) via coeffi- cient {2), and the online reinforcement learning stage via HeLMS mix (regularising each component to the mid-level skills learned offline, via coefficient 772)). The results are reported in Figure 9 where each plot represents different setting for offiine KL-regularisation (either regularisation to N (0. 7) with 3- = 0.01, or no regularisation with B. = 0) and a different transfer case (the easy case of transferring to object set 4, the hard case of transferring object set 3). Each plot shows the downstream performance when varying the strength of KL-regularisation during RL via coefficient M)2 The HeLMS-cat approach represents the extreme case where the skills entirely frozen (i.e. full regularisation).
The results suggest some interesting properties of the latent skill space based on regularisation. When regularising the mid-level components to the (0, 1) prior, it is important to regularise dur- ing online RL; this is especially true for the hard transfer case, where HeLMS performs much better, and the performance degrades significantly with lower regularisation values. However, when removing mid-level regularisation during offline learning. the method is insensitive to regularisation during RL over the entire range evaluated, from 0.01 to 100.0. We conjecture that with mid-level skills regularised to (0, I), the different mid-level skills are drawn closer together and occupy a more compact region in latent space, such that KL-regularisation is necessary during RL for skill to avoid drifting and overlapping with the latent distribution of other skills (i.e. skill degeneracy). In distant
contrast, without offline KI -egraaularisatton, the skills are free expand and occupy gions of the latent space, rendering further regularisation unnecessary during RL. Such latent space properties could be further analysed improve learning and transfer of skills; we leave this as an interesting direction for future work.
5- 1990
7541
200
750
Poreart
11:
2013
Rowordd
310
100
220
100
100
100
30
0
20
1.5
0.5
100
2.5
0.5
1.0
Enviromert Frmes
1+0
Environmert
framss 160
Elvironmert fiamss
140
Elvironmert frames 160
(a) B. = 0.01, easy
(b) 8g = 0.01, hard
(c) 03 0, casy
(d) 03 0, hard
Figure 9: Ablations for KL-regularisation, showing downstream performance with different degrees of online Kl-regularisation. Performance is evaluated for easy (object set 4) and hard (object set 3) transfer cases, with sparse staged rewards; when using different offline regularisation coefficient
(3.) values for the mid-level components.
A.2 NPMP ABLATION
The Neural Probabilistic Motor Primitives (NPMP) work (Merel et al. 2019) presents strong base- line approach to learning transferable motor behaviours, and we run ablations to ensure a fair com- parison to the strongest possible result. As discussed in the main text, NPMP employs a Gaussian high-level latent encoder with a AR(1) prior in the latent space. We also try a fixed N I) prior (this is equivalent to an AR(1) prior with coefficient of 0, so be considered hyperparameter choice). Since our method benefits from KL-regularisation during RL, we apply this to NPMP as well.
14

---

Published as a conference paper at ICLR 2022
find that both changes lead to substantial improvements in the manipu- lation domain, on all five object sets. Consequently, in our main experiments, we report results with the best variant, using N (0, I) prior with KL-regularisation during RL.
As shown in Figure 10
AR(1) prior
NO0.1 1) prior
NCO. prior KL-reg
130
100
50
066
300
(a) Object sct 1
(b) Object sct 2
(c) Object set 3
300
250
210
200
230
200
on
07
0.4
066
0.8
1.2
1.6
1.5
0.
0.4
0.6
0.8
1.0
12 1.5
15
Festronmen
Fistronmen
(d) Object set
(e) Object
Figure 10: Ablation for NPMP (Merel et al. 2019) using staged sparse reward, with all object sets. We find that using (0, T) prior with regguullarisatinn performs much better in the manipulation domain compared to the original AR(1) prior, so use this modified baseline.
B REINFORCEMENT LEARNING WITH MPO AND RHPO
As discussed in Section the hierarchy of skills are transferred to RL in two ways: HeLMS- cat, which learns new high-level categorical policy (yy x,) via MPO (Abdolmaleki et al., 2018); or Me-mi which learns mixture policy T (2; xx) Ey. T(y x,)=(Z, yı, Xt via RHPO (Wulfmeier et al. 2020). We describe the optimisation for both of these cases in the follow- ing subsections. For clarity of notation, we omit the additional KL-regularisstion terms introduced in Section and describe just the base methods of MPO and RHPO when applied to the RL setting
in this paper. These KL-terms are incorporated as additional loss terms in the policy improvement
stage.
B. HELMS-CAT VIA MPO
Maximum a posteriori Policy Optimisation (MPO) is an snneetaation-Maxmiisaaiiinn based algorithm that performs off-policy updates in three steps: (1) updating the critie; (2) creating non-parametric intermediate policy by weighting sampled actions using the critic; and (3) updating the parametric policy to fit the critic-reweighted non-parametric policy, with trust region constraints improve stability. We detail each of these steps below. Note that while the original MPO operates in the environment's 's action space, we use it here for the high-level controller, set the categorical variable y..
Policy evaluation First, the critic is updated via a TD(0) objective as:
min L(0) Exi. (Qr (Qr-QQ(xx,y))) Qx(x)
Here, Qr Te "Ex..1 y.-ı (s((( Y+41)] is the 1-step target with the state transition (x,, Nr, returned from the replay buffer B, and next action sampled from y+11 ~ x (-(x)))))
n' and Q' are target networks for the policy and the critic, used to stabilise learning.
15

---

Published as a conference paper at ICLR 2022
Policy improvement Next, we proceed with the first step of policy improvement by constructing an intermediate non parametrie policy g(velx)), and optimising the following constrained objective:
max (q)) = By. (Q0lxx (x)x s.t. Ex (L(((-Xx))) (x,) (-1x,)) S EB:
where eg defines bound the KL divergence between the non-parametric and parametric policies at the current learning step k. This constrained optimisation problem has the following closed-form solution:
(ye xx) x To, (ye x1) exp (Qo(x,,
(8)
In other words, this step constructs intermediate policy which reweights samples from the pre- vious policy using exponentiated temperature-scaled critic values. The temperature parameter is derived based on the dual of the Lagrangian; for futher details please refer to (Abdolmaleki al.) 2018).
Finally, we can fit parametric policy the non-parametric distribution g(y: xx) by minimising their KL-divergence, subject to a trust-region constraint on the parametric policy:
B arg min Ex,~B KL(gly: xx)) (T(x((xx)))
s.t. E KL (TT xx) ee (ye xx)) cM-
This optimisation problem be solved via Lagrangian relaxation, with the Lagrangian multiplier modulating the strength of the trust-region constraint. For further details and full derivations, please refer (Abdolmaleki al. 2018).
B.2 HELMS-MIX VIA RHPO
RHPO (Wulfmeier et al. 2020) follows similar optimisation procedure as MPO, but ex- tends to mixture policies and multi-task settings. We not exploit the multi-task capabil- ity in this work, but utilise RHPO to optimise the mixture policy in latent space, <2, xx)
Ey. Tyyel (ze Yr, xt). The Q-function Qx 2,) and parametric policy Te, (Ze Xx) use the continuous latents 2, as actions instead of the categorical yt- This is also in contrast the original formulation of RHPO, which uses the environment's action space. Compared to MPO, the policy improvement stage of the non-parametric policy is minimally adapted to take into account the new mixture policy. The key difference is in the parametric policy update step, which optimises the following:
0.41 = arg min Ex,~B (L(( xx) To (2)z))x)))
s.t. Ex
KL (LL(ToL, (yr xx) moz (ye xx))
KL. KL (Ze \1, no, (2. ye, xx)] S SeM-
(10)
ye
In other words, separate trust-region constraints are applied to sum of KL-divergences: for the high-level categorical and for each of the mixture components. Following the original RHPO, separate the single constraint into decoupled constraints that set different for the means, covari- ances, and categorical (en. e,. and Eeat. respectively). This allows the optimiser to independently modulate how much the categorical distribution, component means, and component variances can change. For further details and full derivations, please refer to (Wulfmeier et al. 2020).
16

---

Published as a conference paper at ICLR 2022
C ELBO DERIVATION AND INTUITIONS
We can compute the Evidence Lower Bound for the state-conditional action distribution, p(anT xL.T) ELBO, follows:
ELBO p(arT X1-) - KL(g(Vo.T, Zir x1.T) p Zi:Ti x1:T))
Eglyor xxrr)
p(a Norr, 21.7 X1:r) log (NVor, ZI.T X1T)
log p(ap xx) + log p(z yc) + log lyc-1) Eyuer Σ
log ( ve, logg(y: 11-11
T
Earr Σ
logp(a: xx) - KL(g(zz v1, (lp(ze))))
1=1
-KL(g(V: 11011
(11)
We note that the first two terms in the expectation depend only on timestep f, can simplify and marginalise exactly over all discrete {yı:r}' y. yz. For the final term, we note that the KL timestep t is constant with respect to (as it already marginalises over the whole distribution), and only depends on y:-1- Lastly, we will use sampling to approximate the expectation over 2t. This yields the following:
ELBO
Σ Cg lvre)))
Σ (yv.r T |x1T) [og log 2z, Xt) KL yy, 1l y:)) yoir
(KL ( pl (aosselye-))) 1Y--1))]
per-oomponent recon loss
per-component KI. regulariser
Σ glve x1x) logp(a: 2 xx) (L yr, xx)
ELBO
plzz Yr))
3
Σ a (11=1) KL(g(y: yy-1. xx) IIp(y: vx-1))
(12)
regulariser
where 2(310 qZz X1), the coefficients By and B3 be used to weight the KL terms, and the cumulative component probability 4(y,) xxxe)) can be computed iteratively as:
q(y: x1x) = Σ (1xelly-1,x))(10e X12-1)
(13)
vC-1
In other words, for each timestep and each mixture component, we compute the latent sample and the corresponding action log-probability, and the KL-divergence between the component pos- terior and prior. This then marginalised over all y., with an additional KI. over the categorical transitions.
Structuring the graphical model and ELBO in this form has a number useful properties. First, the ELBO terms include action reconstruction loss and KL term for each mixture component, scaled by the posterior probability each component given the history. For a given state, this pressures the model to assign higher posterior probability to components that have low reconstruction cost or KL, which allows different components to specialise for different parts of the state space. Second, the categorical KI. between posterior and prior categorical transition distributions is scaled by the
17

---

Published as a conference paper at ICLR 2022
posterior probability of the previous component given history g(ye-1 X1:-1): this allows the rel- ative probabilities of past skill transitions along trajectory to be considered when regularising the current skill distribution. Finally, this formulation does not require any sampling or backpropagation through the categorical variable: starting from / = 0, the terms for each timestep can be efficiently computed by recursively updating the posterior over components given history (g(y: X1:e)); and summing over all possible categorical values at each timestep.
D ENVIRONMENT PARAMETERS
As discussed earlier in the paper, all experiments take place in a MuJoCo-based object manipulation environment using Sawyer robot manipulator and three objects: red, green, and blue. The state variables in the Sawyer environment are shown in Table All state variables are stacked for 3 frames for all agents. The object states are only provided to the mid-level and high-level for HeLMS runs, and the camera images are only used by the high- and mid-level controller in the vision transfer experiments (without object states).
The action space is also shown in Table 4 Since the action dimensions vary significantly in range, they are normalised to be between [-1, for all methods during learning.
When learning via RL, we apply domain randomisation to physics (but not visual randomisation), and a randomly sampled action delay of 0-2 timesteps. This applied for all approaches, and ensures that we can learn policy that is robust to small changes in the environment.
Proprioception
State
Dims
Joint angles
7
Joint velocities
7
Joint torque
7
TCP pose
7
TCP velocity
6
Wrist angle
1
Wrist velocity
1
Wrist force
3
Wrist torque
3
Binary grasp sensor
1
Object states
State
Dims
Absolute pose (red)
7
Absolute pose (green)
7
Absolute pose (blue)
7
Distance to pinch (red)
7
Distance pinch (green)
7
Distance to pinch (blue)
7
Vision
State
Dims
Camera images
64 64 x
Table 3: Details state variables used by the agent.
Action
Dims
Range
Gripper translational velocity (x-y-z) Wrist rotation velocity
[-0.07, 0.07 m/s
- 1, rad/s
Finger speed
1-255, 255] tics/s
Table 4: Action space details for the Sawyer environment.
18

---

Published as a conference paper at ICLR 2022
Triplet
Triplet
Triplet
Triplet
Tripiet
Figure 11: The five object (triplets) used in the paper. This image has been taken directly from Lee et al. 2021) for clarity.
D.1 OBJECT SETS
As discussed in the main paper, we use the object sets defined by Lee et al. (2021) which are care- fully designed to cover different object geometries and affordances, presenting different challenges (the image has been taken directly from (Lee al. 2021 for elarity), and feature both simulated and real-world versions; in this paper we focus on the simulated versions. As discussed in detail by (Lee et al. 202I), each object set has a different degree of difficulty and presents a different challenge to the task of stacking
for object interaction tasks. The object sets are shown in Figure (th
In object set I, the red object has slanted surfaces that make it difficult to grasp, while the blue object is an octagonal prism that can roll.
In object set the blue object has slanted surfaces, such that the red object will likely slide off unless the blue object is first reoriented.
In object set 3, the red object is long and narrow, requiring precise grasp and careful placement.
Object set is the easiest case with rectangular prisms for both red and blue.
Object set 5 is also relatively easy, but the blue object has ten faces, meaning limited surface area for stacking.
For more details about the object sets and the rationale behind their design, refer the reader to (Lee et al. 2021).
E
NETWORK ARCHITECTURES AND HYPERPARAMETERS
The network architecture details and hyperparameters for HeLMS are shown in Table Parameter sweeps were performed for the 3 coefficients during offline learning and the coefficients during RI. Small sweeps were also performed for the RHPO parameters (refer to (Wulfmeier al. (020) for details), but these were found to be fairly insensitive. All other parameters were kept fixed, and used for all methods except where highlighted in the following subsections. All RL experiments with 3 seeds capture variation in each method.
For network architectures, all experiments except for vision used simple 2-layer MLPs for the high- and low-level controllers, and for each mid-level mixture component. An input representation net- work was used to encode the inputs before passing them to the networks that were learned from scratch: i.e. the high-level for state-based experiments, and both high- and mid-level for vision (re-
19

---

Published as a conference paper at ICLR 2022
call that while the Statebbsed experiments reuse the mid-level components conditioned object state, the vision-based policy learned them from scratch and KL-regularised to the offline mid-level
skills). The critic network 3-layer MLP, applied the output of another input representation network (separate to the actor, but with the same architecture) with concatenated action.
Offfine learning parameters
Name
Value 8
Latent space dimension
Number of mid-level components, K Low-level network Low-level head
5 for red on blublue data, 10 for all aal_ppaarr___aaaelllllaaalalaaaa pairs data 2-hidden layer MLP, {256, 256} units
Gaussian, tanh-on mean, fixed a 0.1
Mid-level network
2-hidden layer MLP for each component, {256, 256} units
Mid-level head
Gaussian, learned
[0.01, 1.0]
High-level network
2-hidden layer MI.P, (256, 256} units
High-level head
K K-way softmax
Activation function
elu
Encoder look-ahead duration
5 timesteps
By 32
1.0
0.1, 0.0 (object generalisation) 128
Batch size
Learning rate
10-4
Dataset trajectory length
25
Online RL parameters
Name Number of seeds
Value (all experiments)
Input normalizer layer (lincar layer with 256 units, layer-norm, and tanh-on-output)
Input representation network (state)
High-level network (state)
2-hidden layer MLP, (256, 256} units
MLP on proprio and ResNet with three layers (2, 23 blocks
Input representation network (vision)
corresponding to {32, 64, 128} channels
High-level network (vision)
2-hidden layer MLP. {256, 256} units 2- 2-hidden layer MI.P for each component, {256, 256} units 3-hidden layer MLP, (256, 256, 256} units with RNN
Mid-level network (vision)
Critic network
Activation function
elu
"/y
0.1 (vision), 0.01
(pyramid and vision), 0.01 (object generalisation)
Number of actors
1500
Batch size Trajectory length
512
10
Learning rate
2 x 10
Number of action samples
20
RHPO categorical constraint cené RHPO mean constraint 6μ- RHPO covariance constraint
1.0
5 10 33
10
Table 5: Hyperparameters and architecture details for HeLMS, for both offline training and RL.
REWARDS
Throughout the experiments, we employ different reward functions for different tasks and to study the efficacy of method in sparse versus dense reward scenarios.
Reward stages and primitive functions The reward functions for stacking and pyramid tasks use various reward primitives and staged rewards for completing sub-tasks. Each of these rewards within the range of [o, 1]
These include:
20

---

Published as a conference paper at ICLR 2022
reach (obj): shaped distance reward to bring the TCP to within certain tolerance of obj.
grasp (): a binary reward for triggering the gripper's grasp sensor.
clese_fingers () shaped distance reward to bring the fingers inwards.
1ift (obj): shaped reward for lifting the gripper sufficiently high above obj.
hover (obji, j shaped reward for holding cb jbjj above obj2.
stack obj2) sparse reward, only provided if obj1 is on top obj2 to within both horizontal and vertical tolerance.
above (obj, dtt shaped reward for being dist above obj, but anywhere horizon- tally.
pyramid (objl, obj2, obj3): sparse reward, only provided if abj3 is on top of the point midway between obj1 and obj2, within both a horizontal and vertical tolerance.
place_near (obj1, : sparse reward provided if obj1 is sufficiently near ob j2.
Dense stacking reward The dense stacking reward contains number of stages, where each stage represents a sub-task and has maximum reward of 1. The stages are:
reach (red) AND grasp (): Reach and grasp the red object.
1 ft (red) AND grasp () Lift the red object.
hover (red, blue): Hover with the red object above the blue object.
stack (red, blue): Place the red object on top of the blue one.
stack (red, blue) AND above (red) Move the gripper above after a completed stack.
At each timestep, the latest stage to receive non-zero reward is considered to be the current stage, and previous stages are assigned a reward of 1. The reward for this timestep is then obtained by summing rewards for all stages, and scaling by the number of stages, to ensure the highest possible reward on any timestep is
Sparse staged stacking reward The sparse staged stacking reward is similar to the dense reward variant, but each stage sparsified by only providing the reward for the stage once it exceeds a value of 0.95.
This scenario emulates an important real-world problem: that it may be difficult in certain cases to specify carefully shaped meaningful rewards, and it can often be easier to specify (sparsely) whether a condition (such as stacking) has been met.
Sparse stacking reward This fully sparse reward uses the stack (red, blu) function to pro- vide reward only when conditions for stacking red blue have been met.
Pyramid reward The grramid-building reward uses a staged sparse reward, where each stage represents a sub-task and has a maximum reward of 1. If a stage has dense reward, it is sparsified by only providing the reward once exceeds a value of 0.95. The stages are:
reach (red) AND grasp () Reach and grasp the red object.
lift (red) AND grasp () Lift the red object.
hover (red, green) : Hover with the red object above the green object (with a larger
horizontal tolerance, as it does not need to be directly above).
place_near (red, green): Place the red object sufficiently close the green object.
reach (blue) AND grasp () Reach and grasp the blue object.
1ift (blue) AND grasp (): Lift the blue object.
hover (blue, green) AND hover (blue, red): Hover with the blue object above the central position between red and green objects.
pyramid (blue, red, green): Place the blue object on top to make pyramid.
pyramid (blue, red, green) AND above (blue): Move the gripper above after a completed stack.
21

---

Published as a conference paper at ICLR 2022
At each timestep, the latest stage to receive non-zero reward is considered to be the current stage, and all previous stages are assigned a reward of 1. The reward for this timestep is then obtained by summing rewards for all stages, and scaling by the number of stages, ensure the highest possible reward on any timestep is 1.
22


Reviews:
[{"review_id": "s7WTFz-TyCd", "reviewer": "Reviewer_sbA9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The most interesting aspect of the proposed system is the fact that transfer by retraining only at the discrete level can be so advantageous. This is true at least as long as you stick to manipulation for which skills like moving the end-effector and opening/closing the gripper are quite generally applicable. However, since different levels may work at different dimensions of the state space, it might be possible to retrain the mid-level controller from one modality to another, something that is illustrated in the experiments. \n\nThe experimental section is quite extensive and includes comparisons to other alternative methods, as well as ablation studies. Two variants are tested for transfer learning; one that only retrains the high-level controller (HeLMS-cat) and one that also retrains the mid-level (HeLMS-mix), while the low-level controller is always kept fixed. The first option seems to be more beneficial in complex cases with sparse rewards. In fact, it is a bit surprising that HeLMS-mix rarely seems to surpass HeLMS-cat. It is easy to assume that this is due to the similarity between manipulation tasks when it comes to what skill sets are required.\n\nCompared to the other alternative methods tested, HeLMS seems to cover a larger state space during exploration, which in turn leads to higher rewards, when rewards are sparse. With the three-level hierarchy, exploration can be done at a higher level of abstraction while exploiting skills already learned.\n\nThere are some questions worth asking regarding the experimental results. It is shown that for more complicated tasks, HeLMS-mix might come to the point where the average reward starts to decrease. It would seem more reasonable if the reward had just flattened out at a lower level, which happens in most other cases. An explanation given in the paper is that in such cases spurious reward correlations might cause the skills to drift. Does this instability come as a result of including an additional hierarchical level, since none of the other tested methods seems to show anything similar? Another question is why NPMP performs better on object set 2 specifically? Is there any reasonable explanation or is it just a coincidence? \n\nWhen referring to HeLMS in the experiments, it would be good if the paper mentioned either HeLMS-cat or HeLMS-mix, unless both variants are intended. Otherwise, the reader has to go into the text to see which variant was actually used. For example, which variant is used in Table I? By the way, in this table, what is going on with the space coverage in the dimension of grasping? Since it is binary it looks as if none of the other methods really tries to explore grasping, which is odd given the nature of the task given. \n\nFigures 7 c)-d) use the wrong notation for the regularization weight, at least compared to what is written in the text. Also, instead of HeLMS-cat the label says just HeLM.\n\nOn page three, “ELBO <= p(a|x)” is written in the wrong order. It is correct in appendix B. Other than that, the paper is very easy to read and understand, with clarity in both language and notations.\n", "summary_of_the_paper": "This paper proposes a three-level hierarchy for learning motor skills. At the bottom, there is a low-level controller for action generation and at the top a high-level controller that generates sequences of skills, with continuous control signals generated by skills at the mid-level. When transferring motor skills from one task to another using reinforcement learning, it is possible to either retrain only the high-level controller or the mid-level one as well. In experiments, it is shown that for transfers to more complex manipulation tasks with sparse rewards, you are more likely to get convergence in training with positive rewards, if only retraining the high-level controller. Thus you are given more flexibility and are possibly able to learn more complex tasks, with the proposed hierarchy compared to earlier methods based on only two levels. ", "summary_of_the_review": "The proposed three-level hierarchy for learning motor skills might not be groundbreaking in itself, but it brings some interesting advantages and flexibility that is illustrated in the paper. It is interesting to see how far you can go in complicated cases by reusing an already learned skill set. At the same time, you might be able to achieve higher rewards in the end if skills are also refined. For that reason, this paper is definitely worth reading. It is possible to imagine training schemes in which you gradually reduce the regularization of the mid-level skills to make them more specific at the end of training, and thus possibly get the best of both worlds.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SjxxEU0Nwlw", "reviewer": "Reviewer_NRgW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nHierarchical policies are a great idea, and the proposed model seems like a sensible approach to integrate both discrete and continuous latent variables into a policy, in contrast to many existing approaches which use discrete latent indicator variables to trigger fixed policies.\n\nResults show the proposed architecture seems to work, and I was particularly interested in the comparison with a hierarchical behaviour cloning model.\n\nHowever, the most interesting part of this work lies around the RL phase of the work, where different levels of mixing are explored (completely freezing skills, vs allowing greater skill adaptation. This area is of particular interest as we move beyond \"look we discovered some skills and re-used them\" to more realistic online learning settings.\n\nWeaknesses:\n\nThe paper is in some need of smoothing, and I found this a difficult read, despite being familiar with the field. In particular, I believe the paper would benefit from a clearer problem formulation (ie. is the problem setting a two stage learning from demonstration one followed by an RL phase that makes use of the initial learned skills, or is it a tabular rasa setting with skills gradually incorporated and re-used) Initial sections suggest the former, but some experimental settings and baselines seem to indicate both. I understand that the architecture is very general, and the aim was to show it can be used in both settings with tweaks, but I think the paper would benefit greatly from making the different settings more clear in an expanded problem formulation. \n\nI would also greatly appreciate a more structured experiments setting, more clearly delineating the tasks and with a much more focused aim. As an example, the baselines are all designed to solve slightly different problems, and the broad set of experiments are not necessarily presented to make a cohesive argument, rather they seem to aim to answer many separate questions. A potential option would be to restructure this section around the question of how much prior do we need and how quickly we can allow skill adaptation, which is a very interesting research question.\n\nQuestions/ Comments:\n\nThere is a wealth of literature in hierachical and resuable skills discovery for RL/ continuous control that is missing from this work, in particular around switching nonlinear dynamical systems. See below for a small selection of work deserving mention. \n\nThis paper is particularly important, as it discusses a range of architectures to embed discrete latent variables for skill discovery and later re-use: Carlos Florensa, Yan Duan, Pieter Abbeel, \"Stochastic Neural Networks for Hierarchical Reinforcement Learning\" ICLR 2017 \n\nOther work on re-usable skill discovery using discrete latent variable models:\nScott Niekum, Andrew Barto, \"Clustering via Dirichlet Process Mixture Models for Portable Skill Discovery\", Neurips 2011\n\nP. Ranchod, B. Rosman and G. Konidaris, \"Nonparametric Bayesian reward segmentation for skill discovery using inverse reinforcement learning,\" 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015, pp. 471-477, doi: 10.1109/IROS.2015.7353414.\n\nKipf, Thomas, et al. \"Compile: Compositional imitation learning and execution.\" International Conference on Machine Learning. PMLR, 2019.\n\nHany Abdulsamad, Jan Peters, \"Hierarchical Decomposition of Nonlinear Dynamics and Control for System Identification and Policy Distillation\", Proceedings of the 2nd Conference on Learning for Dynamics and Control, PMLR 120:904-914, 2020.\n\nD. Tanneberg, K. Ploeger, E. Rueckert and J. Peters, \"SKID RAW: Skill Discovery From Raw Trajectories,\" in IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4696-4703, July 2021, doi: 10.1109/LRA.2021.3068891.\n\nCategorical regularisation in hierarchical models has been proposed previously in:\nMichael Burke, Yordan Hristov, Subramanian Ramamoorthy, \"Hybrid system identification using switching density networks\", Proceedings of the Conference on Robot Learning, PMLR 100:172-181, 2020.\n\nSee also:\nZhe Dong, Bryan Seybold, Kevin Murphy, Hung Bui, \"Collapsed Amortized Variational Inference for Switching Nonlinear Dynamical Systems\", Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2638-2647, 2020.\n\nMinor queries/ comments:\nI assume that the number of skills needs to be pre-specified and remains fixed?\n\nSection 2.3: \"Following most previous work, we ...\" - citation needed.\n ", "summary_of_the_paper": "This paper introduces a latent variable controller model that allows for reusable skill learning in behaviour cloning and reinforcement learning settings. The architecture comprises three stages or levels. At the highest level, input state information (eg. proprioception, visual input, object state information) is passed through an MLP to produce a discrete latent state skill selection variable, incorporating a discrete latent transition model. This skill selection prior is then used in a mid level network operating on the same input information, to produce a continuous latent state, conditioned on this discrete latent variable. Finally, this latent state, together with the input state is used by an actor network to produce actions.\n\nThe model is trained used a heuristic process depending on the deployment setting, with different elements frozen at different times, or through the use of a mixture agent that can be initialised from scratch or using previous skills (a similar approach is explored by Florensa et al., ICLR 2017). A KL regularisation on the discrete latent variable skill selector is used (also in prior work - Burke et al., CoRL 2019). Results show that this approach is effective on a range of tasks, and that skill re-use (as expected) out performs learning from scratch. Probably the most interesting result is the comparison with a hierarchical behaviour cloning network and the experiments around skills transfer. The idea seems sound, although the presentation is in need of some improvement, and at times the paper suffers from lack of clarity (in problem formulation and presentation of results). Moreover, some important work in the admittedly vast array of work on hierarchical models for RL and behaviour cloning is missed.\n\n", "summary_of_the_review": "I think this work has some very interesting experiments and results, particularly around the question of how we should trade-off leveraging existing skills against learning new ones, but these are currently not highlighted very prominently, due to a lack of clarity in problem formulation and experimental results.\n\nThere is a wealth of research around hierarchical mixture latent variable policies and skill discovery which is missing from this work, and reduces the strength of the architecture as a contribution. Refocusing on the fact that the architecture makes it easier to develop methods and strategies for transferring to new settings would help to distinguish this paper from this related work.\n\n====== Post rebuttal comments ======\n\nThank you for your response and for the hard work in addressing the feedback provided. I feel the paper is much stronger for this, so have updated my score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SVOy_p2hqX", "reviewer": "Reviewer_wAVF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written and easy to follow. It is nice to read (except for some parts of the method that are rushed)\n- The method is novel, extending prior work\n- The results show some improvements\n\nWeaknesses:\n- Structurally, I like papers that push the related work section to the back because they first present some relevant common theoretical components to understand both the presented method and the related work. This is not the case here. I would recommend placing the related work after the introduction to fully understand and compare the method section to the previous methods.\n- The experimental evaluation is scarce. The method, although explained as very general, is only applied to one domain. Some of the results are not completely clear. The paper would gain on clarity and support for conclusions if the method would be applied to other RL domains, even simple ones. Trained policies could be used to generate expert demonstrations. In its current form, the experiments fall short.\n- The comparison to other methods, especially to NPMP, is not completely clear. The results are somewhat mixed. This is probably a consequence of the limited experimental evaluation with only one domain. Right now, apart from the sparse reward setup, it is not very clear the pros and cons of the presented method compared to previous ones.\n- Could you provide an intuition of what is the different information represented by the categorical and continuous high-level latent codes? Both represent the context. What is exactly the benefit of the two levels? Temporal consistency and commitment? Semantic information encoding? Additional experiments in other domains and with other mixtures of information at different levels would help\n- What happens if there are other observations passed to the different levels? How sensitive are the results to that?\n- What happens if during training one or more of the categorical values are not allowed? Can the system recover (find the necessary skills via exploration)? \n- I’d recommend including a small figure with the objects used in the experimental evaluation to avoid having to go back and forth to know what are “set1”, “set2”...\n- The method section is a bit rushed. I’d dedicate more time to go step by step over the derivation (not completely, that is in the appendix), explaining more of why things are done instead of just describing the mathematical equations.\n", "summary_of_the_paper": "This paper presents a method to learn a three-leveled hierarchy of skills offline, from a dataset of demonstrations, that can then be applied to accelerate reinforcement learning. The three-level architecture is novel. It encodes a discrete selection, a continuous contextual variable (dependent on the discrete selection) and a low level policy dependant of the continuous contextual variable. The results show some improvements over baselines, especially in a sparse reward context where the presented method capitalizes from the offline learned strategies for exploration.", "summary_of_the_review": "In summary, the paper presents a novel approach with three hierarchical levels for skill learning. The approach is novel, the results are good, the text is well-written, and the problem is relevant. The experimental evaluation is rather limited and several questions arise with respect to the benefits over previous approaches, the robustness and generality.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5r65_ubZIoG", "reviewer": "Reviewer_cfLn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTH:\n\nS1: The paper considers a very important problem. \n\nI think that skill extrapolation and reuse it is at the core of providing more sample efficient reinforcement learning and to allow its application to real-world tasks.\n\nS2.1: I think that the proposed method is sound.\n\nThe idea of breaking the three-level of hierarchy makes a lot of sense. Many existing approaches in fact propose a similar structure, which, in higher detail, can be described as a mixture of parametric skills. \n\nThe idea of incorporating information asymmetries, as also noted by the authors, was empirically shown to be also effective. \n\nS2.2: Many works I am aware of, have limited applicability, since they do not make use of neural networks, and they do not scale well with high-dimensional input. This work does not suffer from this problem. Furthermore, the extracted skills can be easily used in many classic RL algorithms.\n\nS3. The experimental section consistently supports the statement made, showing the efficacy of the proposed method. I especially enjoyed the ablation study.\n\nWEAKNESSES\n\nW1: The readability of this paper can, in my opinion, be improved. \n\nIn particular:\n\nW1.1: the paragraph \"Training via the Evidence Lower bound\" is a bit messy. The quantities $p_{\\psi_{y,a}}$, $q_{\\phi_{y,z}}$, etc, have not been defined before. I think that these models should have been defined at the beginning of  2.1. \nFurthermore, the left-hand side of Equation 3 could have been defined explicitly, even though that the ELBO is a widely known concept. \n\nW1.2: Equations 4 and 5 present just RL objectives. I think that the authors could have spent some lines to write how MPO optimizes these objectives. Writing down the optimization process will explicitly clarify which parameters will be updated. The current form of Section 2.3. is, in my opinion, too fuzzy. \n\nW1.3: I think that many equations contain some typos. For example,\n\nEquation 1: on the right side we have p(y_0), but why on the right-hand-side $p(y_{1:T}, Z_{1:T})$ does not take in account of $p(y_0)$?\nThe same questions arise also in Equation 2. \n\nThe summations $\\sum_{y_{t-1}}$ in the derivation of Equation 3 should be better defined. \n\nIn Equations 4 and 5, $x_t$ is outside the expectation. This is wrong. Question: shouldn't the KL also be discounted with \\gamma^t? If the KL is summed for each $t$ term from $0$ to $\\infty$, then the summation will be unbounded (and, therefore, impossible to optimize).\n\nW2. \n\nThe proposed method, in my understanding, can be very much regarded as \"skills segmentation\" \"movement primitive segmentation\", [1-4]...\n\nFurthermore, the mixture of skills (or movement primitives) has been also discussed widely [5, 6, 7, 9, 10]. Notice that the probabilistic model of [6, 10] resembles closely your approach (discrete selection of parametric skills).\nThe mixture of movement primitives has also been utilized in reinforcement learning, leading to a very similar probabilistic model (with a discrete variable selecting the skill, and the continuous variable selecting the skill's parameters) [7-10].\n\nIf you dig a bit, you will find many more connections to your current work. \n\nI am also aware of this very recent work, which also uses high-level skills to perform RL tasks [11]. Their motivation and result are, in my opinion, strongly supporting the evidence of your result, although (in my opinion) your work is more complete. \n\nW3: as described above, and also as partially acknowledged in your paper, people have been working on this topic. I like how you have put things together, and I think your work is promising. I think that the contribution is enough, but not particularly strong. \n\nTo summarize, I think that the idea behind your method is good. I find it a pity that it misses a better related-work contextualization, mathematical precision, and exposition clarity. \n\nQUESTIONS:\n\n- Can the authors clarify my doubts about the equations? \n- Can the author draw a link between their work and HMM? \n\n[1] Scott Niekum, Sarah Osentoski, George Konidaris, Andrew G. Barto. Learning and Generalization of Complex Tasks\nfrom Unstructured Demonstrations. IROS 2012.\n[2] Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, Jan Peters. Probabilistic Segmentation Applied to an Assembly Task. Humanoids 2015.\n[3] Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, Jan Peters., Learning movement primitive libraries\nthrough probabilistic segmentation, IJRR 2017.\n[4] Bjorn Kruger, Anna Vogele, Tobias Willig, Angela Yao, Reinhard Klein, Efficient Unsupervised Temporal\nSegmentation of Motion Data. \n[5]  Affan Pervez,  Dongheui Lee. Learning task-parameterized dynamic movement primitives using a mixture of GMMs.\n[6] Elmar Rueckert, Jan Mundo, Alexandros Paraschos, Jan Peters, and Gerhard Neumann. Extracting Low-Dimensional Control Variables for Movement Primitives. ICRA 2015. \n[7] Katharina Muelling, Jens Kober, Jan Peters. Learning Table Tennis with a Mixture of Motor Primitives. Humanoids 2010\n[8] Katharina Mülling, Jens Kober, Oliver Kroemer and Jan Peters. Learning to select and generalize striking movements in robot table tennis. IJRR\n[9] Adria Colome, and Carme Torras, Dimensionality Reduction in Learning Gaussian Mixture Models of Movement Primitives for Contextualized Action Selection and Adaptation. RAL 2018. \n[10] Samuele Tosatto, Georgia Chalvatzaki, and Jan Peters. Contextual Latent-Movements Off-Policy Optimization for Robotic Manipulation Skills. ICRA 2021.\n[11] Dalal Murtaza, Pathak Deepak and Salakhutdinov Ruslan. Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives. NeurIPS 2021.\n", "summary_of_the_paper": "This paper presents a hierarchical model for skills extraction from an online dataset. In particular, the model can be seen as a 3-level hierarchy. The highest level of the hierarchy selects a discrete variable that corresponds to the index of a skill. The mid-level selects the continuous parameter that defines the skill. The lower level executes the skill.\n\nThe considered problem is crucial in reinforcement learning. In particular, the definition of skills to be reused across different tasks is essential to achieve better sample efficiency, and crucial to allow reinforcement learning to be applied directly on challenging real robotic tasks. \n\nWhen dealing with an off-line dataset, one wants potentially to first extract a set of skills (say, low-level policies) and then to learn via RL the selector of such policies. In this way, one both simplifies the problem (increasing sample efficiency) and allows the skills to be reused in different tasks.\n\nThe authors propose a graphical model that consists, per time step, of four variables: $x_t$, $y_t$ $z_t$ and $a_t$. $x_t$ represents the MDP's state, $y_t$ is the discrete selector of the skill, $z_t$ the (continuous) parameters of the skill, and $a_t$ is the output at time $t$ of the selected skill (and can represent, for example, the joint velocities of the robot).\n\nAfter learning via variational inference the probabilistic model, the authors propose to refine the learned behavior via MPO, learning the skill selector $p(y_t | y_{t-1},x_t)$, the skill parameters $p(z_t | y_t, x_t)$, and the skill p(a_t | z_t, y_t).\n\nFurther, the authors propose 1) to use an asymmetric information approach, where each level of the described hierarchy sees a different definition of the state. In particular, the lower layer sees the robotic state (proprioception), the mid-layer sees the position of objects in the scene, and the highest level sees both the information 2) the models use \"gated\"-heads: the categorical variable $y_t$ is fed to the last part of the neural network. \n\nOnce the hierarchy has been learned, one can tune it on the same or different tasks. In particular, the lowest layer is kept fixed, while only the highest, or the highest, and the mid-layer are learned. \n\nThe authors propose to use MPO as a reinforcement learning algorithm. They perform a few ablation studies and experiments to empirically prove the efficacy of their method. ", "summary_of_the_review": "STRENGTH\n\n1. The paper considers a very important problem (skills extraction and reuse).\n2. The proposed method is sound. Furthermore, the employment of variational inference with deep NN allows the method to be scalable and usable on high-dimensional tasks.\n3. The experiment section is well structured and supports nicely the statements made.\n\nWEAKNESSES\n\n1. I think the paper is not very clear. In particular, some passages are missing, and some equations seem not to be correct.\n2. I think that a large part of related work is missing, especially work prior to 2017.\n\nI would give 8 (Accept, Good Paper), but I really think that this paper should be organized better, and the equations are not clear. I have an issue with every single equation in the paper. May I ask the authors to clarify my doubts?\n\n\n\n================ SUMMARY OF CHANGES ======================\n\nMy main concerns were mainly about the clarity of the paper and the lack of a large body of related work. \nI think that the authors addressed both the issues in their new version of the paper.\n\nThe new paper is much clearer, and the proper discussion of related work helps in defining more rigorously what was the contribution of this paper. Therefore, I increased my score from 6 to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work propose to learn hierarchical skill representations that, as opposed to prior work, consist of both discrete and continuous latent variables. Specifically, this work proposes to learn 3 level hierarchy via a hierarchical mixture latent variable model from offline data. For test time usage and adaptation on down-stream tasks, the manuscript proposes two ways of utilizing the learned hierarchy in RL settings. 

**Strengths**
A novel method to learn hierarchical representations with mixed (discrete/continuous) latent variables is proposed
Detailed experimental evaluation, and baseline comparisons, show promising results

**Weaknesses**
There were various clarity issues as pointed out by the reviewers (fixed in rebuttal phase)
The related work was missing relevant work, and the proposed framework was not connected well to existing work (fixed during rebuttal)

**Rebuttal**
The authors significantly updated the manuscript based on the feedback of the reviewers, and improved both clarity of the manuscript (method+experiments) as well as the exposition of the proposed framework with respect to related work.

**Summary**
After the rebuttal, all reviewers agree that this is a good paper that should be accepted. Thus my recommendation is accept.
Decision: Accept (Spotlight)