SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration
Authors: Giulia Vezzani
Dhruva Tirumala
Markus Wulfmeier
Dushyant Rao
Abbas Abdolmaleki
Ben Moran
Tuomas Haarnoja
Jan Humplik
Roland Hafner
Michael Neunert
Claudio Fantacci
Tim Hertweck
Thomas Lampe
Fereshteh Sadeghi
Nicolas Heess
Martin Riedmiller
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Control
Skills
Priors
Hierarchical Reinforcement Learning

Abstract: The ability to effectively reuse prior knowledge is a key requirement when building general and flexible Reinforcement Learning (RL) agents. 
Skill reuse is one of the most common approaches, but current methods have considerable limitations. For example, fine-tuning an existing policy frequently fails, as the policy can degrade rapidly early in training, particularly in sparse reward tasks. In a similar vein, distillation of expert behavior can lead to poor results when given sub-optimal experts. 
We compare several common approaches for skill transfer on multiple domains and in several different transfer settings, including under changes in task and system dynamics. We identify how existing methods can fail and introduce an alternative approach which sidesteps some of these problems. 
Our approach learns to sequence existing temporally-abstract skills for exploration but learns the final policy directly from the raw experience. This conceptual split enables rapid adaptation and thus efficient data collection but without constraining the final solution. Our approach significantly outperforms many classical methods across a suite of evaluation tasks and we use a broad set of ablations to highlight the importance of different components of our method.

Under review as a conference paper at ICLR 2023
SKILLS: ADAPTIVE SKILL SEQUENCING FOR EFFICIENT TEMPORALLY-EXTENDEE EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
The ubility to effectively reuse prior knowledge is a key requirement when build- ing general and flexible Reinforcement Learning (RL) agents. Skill reuse is one of the most common approaches, but current methods have considerable limitations. For example, fine-tuning existing policy frequently fails, the policy de- grade rapidly early in training. In a similar vein, distillation of expert behavior can lead poor results when given sub-optimal experts. We compare several common approaches for skill transfer on multiple domains including changes task and system dynamics. We identify how existing methods fail and intro- duce an alternative approach to mitigate these problems. Our approach learns to sequence existing temporally-extende policy directly from the raw experience. This conceptual split enables rapid adap- tation and thus efficient data collection but without constraining the final solution. It significantly outperforms many classical methods across a suite of evaluation tasks and we use broad set of ablations to highlight the importance of different components of our method.
skills for exploration but learns the final
INTRODUCTION
The ability to effectively build on previous knowledge, and efficiently adapt to new tasks or con- ditions, remains a crucial problem in Reinforcement Learning (RL). It is particularly important in domains like robotics where data collection is expensive and where we often expend considerable human effort designing reward functions that allow for efficient learning.
Transferring previously learned behaviors as skills can decrease learning time through better ex- ploration and credit assignment, and enables the use of easier-to-design (e.g. sparse) rewards. As a result, skill transfer has developed into an area of active research, but existing methods remain limited in several ways. For instance, fine-tuning the parameters of policy representing an exist- ing skill is conceptually simple. However, allowing the parameters to change freely can lead a catastrophic degradation of the behavior early in learning, especially in settings with sparse rewards (Igl et 2020). An alternative class of approaches focuses on transfer via training objectives such as regularisation towards the previous skill. These approaches have been successful in various set- tings (Ross al. 2011 Galashov et al. 2019 Tirumala al.) 2020 Rana et al. 2021). but their performance is strongly dependent on hy iyprparraeeeers such as the strength of regu larisatioon. If the regularisation is too weak, the skills may not transfer. If it is too strong, learning may not be able to deviate from the transferred skill. Finally, Hierarchical Reinforcement Learning (HRL) allows the composition of existing skills via learned high-level controllers, sometimes abstraction (Sutton et al. 1999).. Constraining the space of behaviors the policy to that achievable with existing skills can dramatically improve exploration (Nachum et al.) 2019) but it can also lead to sub-optimal learning results if the skills or level of temporal abstraction are unsuitably chosen (Sutton et al. 1999. Wulfmeier et al.) 22221) As we will show later (see Section 5), these approaches demonstrably fail to learn in many transter settings.
coarser temporal
Across all of these mechanisms for skill reuse we find a shared set of desiderata. In particular, an efficient method for skill transfer should 1) reuse skills and utilise them for exploration at coarser temporal abstraction, 2) not be constrained by the quality of these skills or the used temporal ab- straction, 3) prevent early catastrophic forgetting of knowledge that could be useful later in learning.
With Skill Scheduler (SkilIS), we develop a method to satisfy these desiderata. We focus on the transfer of skills via their generated experience inspired by the Collect & Infer perspective (Ried-

---

Under review as a conference paper at ICLR 2023
miller et al. 2022 2018). Our approach takes advantage of hierarchical architectures with pretrained skills to achieve effective exploration via fast composition, but allows the final solution to deviate from the prior skills. More specifically the approach learns two separate components: First, we learn a high-level controller, which we refer to as scheduler, which learns to sequence existing skills, choosing which skill to execute and for how long. The prelearned skills and their temporally extended execution lead to effective exploration. The scheduler is further trained to maximize task reward, incentivizing it to rapidly collect rask-relevant data. Second, we distill a new policy, or skill, directly from the experience gathered by the scheduler. This policy trained off-policy with the same objective and in parallel with the scheduler. Whereas the pretrained skills in the scheduler are fixed to avoid degradation of the prior behaviors, the new skill unconstrained and can thus fully adapt to the task at hand. This addition improves over the common use of reloaded policies in hierarchical agents such as the options framework (Sutton et al. 1999) and prevents the high-level controller from being constrained by the reloaded skills.
The key contributions of this work are the following:
We propose method to fulfil the desiderata for using skills for knowledge transfer, and evaluate it on a range of embodied settings in which we may have related skills and need to transfer in a data efficient manner.
We compare our approach to transfer via vanilla fine-tuning, hierarchical methods, and imitation- based methods like DAGGER and KL-regularisation. Our method consistently performs best across all tasks.
In additional ablations, we disentangle the importance of various components: temporal abstraction for exploration; distilling the final solution into the new skill; and other aspects.
2 RELATED WORK
The study of skills in RL is an active research topic that has been studied for some time (Thrun & Schwartz 1994, Bowling & Veloso 1998 Bernstein 1999 Pickett & Barto 2002). A 'skill in
this context refers any mapping from states actions that could aid in the learning of new tasks.
These could be pre-defined motor primitives (Schaal et al. |005 Mülling et al. 2013 Ijspeert et al., cemporally-correllle
2013 Paraschos et al. 2013 Lioutikov et al., [015 Paraschos et al. 2018).
behaviors inferred from data NNickum & Barto] 20011 10 Ranchod et al. [2135) Krüger et al.] 2016
Lioutikov al.] 2017 Shiarlis et al.) 2018 Kipf al.) [209 Merel al.; 2019, Shankar et al,
2019 Tannnberg et al. 2021) or policies learnt in a multi-task setting (Heess et al. 2016 Hausman
ct al. 2018 Riedmiller et al., 2018). It is important to note that our focus in this work is not on learning skills but instead on how to to best leverage a given set of skills for transfer.
Broadly speaking, we can categorise the landscape of transferring knowledge in RL via skills into a set of classes (as illustrated in Fig. [T: direct reuse of parameters such as via fine-tuning existing
policies (Rusu et al. 2015 Parisotto et al. 2015 Schmitt et al.) 2018), direct use in Hierarchical RL
(HRL) where high-
controller tasked to combine primitive skills or options (Sutton et al.
1999
Heess et al. 2016
Bacon el al. 2017 Wulfmmier et al. 2019 Daniel et al., 2012 Peng et al., transfer via the training objective such as regularisation towards expert behavior (Ross al.
2019
201
Galashov et al. 2019 Tirumala et al.] 2020) and transfer via the data generated by executing
skills
(Riedmiller et al. 2018 Campos al.
2021 Torrey al., 2007).
Fine-tuning often underperforms because neural network policies often do not easily move away from previously learned solutions (Ash & Adams 2020 [gl et al.) 2020 Nikishin et al. 2022) and hence may not casily adapt to new settings. As a result some work has focused on using previous solutions to "kickstart" learning and improve on sub-optimal experts (Schmitt et al. 2018 Jeong cct al., 2020 Abdolmaleki et al. 2021). When given multiple skills, fine-tuning can be achieved by reloading of parameters via mixture (Daniel et al.) 2012) Wulfmeier et al. 2019) or product (Peng
et al. 2019) policy potentially with extra components be learnt.
An alternative family of approaches uses the skill as a 'behavior prior' to to generate auxiliary objectives to regularize learning (Liu al. 202T). This family of approaches have widely and successfully been applied in the offline or batch-RL setting to constrain learning (Jaques et al.
2019 Wu al., 2019 Sieeel al. 2020 Wang et al. 2020 Peng al.) 2020). When used for transfer learning though, the prior can often be too constraining and lead to sub-optimal solutions
(Rana et al. 2021
2

---

Under review as a conference paper at ICLR 2023
Takilt
Apent
Eg Regalal
shill
wkith
Collect Data
Fine-tuning
Hiararchy
Auxilery Objectives
Exploration Data
Transfer mechanisms for skills: Direct use via 1. fine-tuning or 2. hierarchical policies, 3. auxiliary training objectives, 4. exploration and data generation. Grey and green respectively denote new policy components and transferred skills.
Figure 1:
Arguably the most widely considered use case for the reuse of skills falls under the umbrella of hi- erarchical RL. These approaches consider policy that consists of two more hierarchical 'levels' where a higher level controllers modulates number of lower level skills via a latent variable or goal. This latent space
be continuous (Heess et al. 2016 Merel al. 2019; Hausman al.,
2018
Haarnoja et al., 2018, Lynch et al.] 2019 Tirumala et al. 2019; Pertsch et al.) 2020] Ajay
et al.
Singh et al. 2022 Bohez et al. (202)) 2) discrete (Florensa et al. 2017 Wulfmeier et al.,
2020
Seyde al. 2022 or combination of the two (Rao et 2002)..
A major motivation of such hierarchical approaches is the ability for different levels to operate different timescales which, it is argued, could improve exploration and ease credit assignment. The most general application of this, is the Options framework (Sutton et al.,
1999 Bacon et al.,
2017 Wulfmeier al. 2021) that operates using temporally-extended options. This allows skills to
operate at longer timescales that may be fixed (Li et al. 2019] Zhang et al. 2020 Ajay et al. 2021 or with different durations (Bacon et al. 2017] Wulffeier et al. 202 Salter al.) 2022). The
advantage of modeling such temporal correlations has been attributed to benefits of more effective exploration (Nachum al., 2019). However, constraining the policy in such way often be
detrimental to final performance (Bowling & Veloso 1998; Jong et al. 2008).
Compared to other hierarchical methods such as the options framework, unconstrained, non an-hheraarchcal solution and learn temporal abstraction as a task-specific instead of skill-specific property. In this way, we benefit from temporally correlated exploration without compromising on final asymptotic performance by distilling the final solution. Our work is closely related to transfer via data for off- off-policy multi-task RL as explored by (Riedmiller et al. 2018 Hafner al., (020)) and initial work transfer learning (Campos al. 2021 Torrey et al.
method uses final
2007)
We extend these ideas to the compositional transfer of skills.
BACKGROUND
We frame our discussion in terms the reinforcement learning problem in which an agent observes the environment, takes an action, and receives a reward in response. This can be formalized as Markov Decision Process (MDP) consisting of the state space S, the action space A, and the transition probability p(st+1 | ax) of reaching state $1+1 from state s, when executing action a.- The agent's 's behavior is given in terms of the policy π (a)))s)). In the following, the agent's policy can either be parametrised by a single neural network or further be composed of skills (aass) such that <[ass) f(a [Ts h
The agent aims to maximize the sum of discounted future rewards, denoted by:
J(T) E (oo(s (lt
Σ 1=
(1)
where y is the discount factor, = r (st, az) is the reward and po (so) is the initial state distribution. Given a policy we can define the state-action value function ((sr, a;) as the expected discounted return when taking an action or in state 9, and then following the policy क as:
Q(5r, at) r(s,, ar) +
(((+41))))((+++1 a)].
4 METHOD
The method we propose in this paper, named Skill Scheduler (SkilIS) (Fig. 2a explicitly separates the data collection and task solution inference processes in order to better accommodate the require-
3

---

Under review as a conference paper at ICLR 2023
states
Seheduller
HCMPO
Replay
t= Hewdel - mo(sio, 6) [t..k]] - (e.
1=1
t=
Boo fool ~o (alllnss
(olaa)
my (alas)
ous tr Ni, (olam)
Eivieonment
he
Aos
ag mig (a)au)
10
t 0
010 Win (a|$10)
New Skill
actions
8 kgo
Figure 2: a) The Skill Scheduler framework. Data is collected by the scheduler by sequencing selected skills among a set of pre-trained (frozen) skills and the new skill. This data used to improve the scheduler using HCMPO (see Section 4.3) and the new skill via CRR. b) Scheduler action and counter sampling. At time skill with index to is applied to the environment for kn time steps. When (t = ko). the scheduler samples new action, the counter to the new length et K2 and decreased every time step.
the scheduler samples an action 2g [šo; ko] i.c. the
ments each. The data collection performed by the scheduler, a high-level policy that chooses the best skills to maximize the task reward obtained from the environment. The task solution is learned via an off-policy learning algorithm that optimises a new skill n. (uls, , Φ) for the given task.
By avoiding assumptions about the representation and source of the underlying skills, we can use these existing behaviors agnostic to how they were generated. The agent has no access any prior information on the quality of the skills and their relevance for the new task. While most of our experiments in Section focus reusing skills that useful for the task, conduct detailed ablation study that investigates the robustness with respect to the number and quality of skills used.
4.1 SCHEDULER
The scheduler represented by a multivariate categorical policy (t(2 = [i; kl|ss, 0) that outputs an action 2 consisting of:
the index E [O] the skill to execute among the available skills = S U T (als, })}. where = {Tr N-1 are the N previously trained and frozen skills and (als,, o) is the new skill, and
the number of steps the skill is executed for k K (the skill length), where is set of available skill lengths (positive integers) with cardinality [K| = M.
Whenever new action = [i] k] chosen by the scheduler, the skill (a)a)) is executed in the environment for & timesteps. The scheduler does not sample any new action until the current skill length à is expired (see Fig. (b). Choosing k: 1 means that the scheduler selects new skill at coarser timescale than the environment timestep. This induces temporal correlations the actions, which are important for successful skill-based exploration in sparse-reward tasks (see Section
Choosing one skill for multiple timesteps affects the agent's 's future action distributions and therefore future states and rewards. We add an auxiliary counter variable to the observations to retain the Markov property. Whenever the scheduler chooses new action = = [i; k the counter is set equal to the chosen skill duration = k. It is then decreased at every time step (C++1
1) until the chosen skill duration has been reached (c 0) and the scheduler chooses new action (see Fig. 2b The counter is crucial for learning of the critic, as detailed in Section 4.33
Data is collected in the form (st, de; žt, Ct2 Tty 81+1) and is used both train the scheduler and the new skill. The scheduler is trained only using the high-level actions (Zt, Ct, Tt, 8t+1) with the accumulated task return over the episode as objective (Fq.. 1.
4

---

Under review as a conference paper at ICLR 2023
We introduce an off-policy algorithm, Hierarchical Categorical MPO (HCMPO) train the sched- uler. While any RL algorithm could be used as a starting point, HCMPO builds upon a variant of Maximum A-Posteriori Policy Optimisation (MPO) (Abdolmaleki et al. (018) with discrete action space (Neunert et al. 2020). More details are provided in Section 4.3
4.2 NEW SKILL
The new skill n x (als, 6) interacts with the environment only when chosen by the scheduler. It is trained with the data only including the low-level action (st, at, Ft, St+1) collected by the frozen skills S and the new skill itself according to the scheduler decisions. Given the strongly off-po nature of this d offline RL algorithms are most suited for learning the new skill. For this work, we use Critic Regularized Regression (Wang et al. 2020
Note that, although the scheduler and the new skill are trained with the same objective of maximizing the task reward (Eq. we consider the new skill to provide the final solution for the task at hand, as it can reach higher performance. Whereas the scheduler acts in the environment via prelearned skills fixed to avoid degradation of the prior behaviors, the new skill is unconstrained and can thus fully adapt the task at hand. In Section 5.2.3 we show how the scheduler and the new skill successfully tackle the two different problems of rapidly collecting task-relevant data (the scheduler) and slowly distilling the optimal task solution from the gathered experience (the new skill).
4.3 TRAINING THE SCHEDULER
In this section we derive the policy evaluation and improvement update rules of our proposed algo- rithm, Hierarchical Categorical MPO (HCMPO), used for training the scheduler.
Policy Evaluation In order to derive the updates, we take into account the fact that if k2 time steps have passed, i.c. the counter ct has expired, the scheduler chooses a new action 2, [āg ka] time f. keeps the previous action 2t = 21-1 = 1 1 otherwise.
C(, ke, kc) + tr + Ens (15,,,,,, k, 0) if C++1 =1 (((5++) G+11 - 1,14+1 4. ke+1 k,) otherwise.
(3)
If the scheduler has chosen a new action in the state S1+1, the critic in state s, is updated with the standard MPO policy evaluation, i.c. with the critic evaluated using actions sampled from the current scheduler policy in the next state i, ~ as,e(t; k k(s++1). In all the other states instead, the critic is updated with the critic computed using the action 2t-1 actually taken by the scheduler. which is Z(+1 = 24 (ēe k... More details on the derivations of Eq. 3 Jare available in Appendix A.
Policy Improvement The scheduler policy is improved via the following mechanism but only on transitions where+ a new action sampled by the scheduler:
[f(cll)) log(Ts(Z))) ddzds with g(2(s) x =5[((s) exp (s, = 1,2)).
T n++ = arg max
where n is the index for the learning updates.
4.4 DATA AUGMENTATION
Eq. 3 and 4 show that only subset of the available data would be actively used for learning the scheduler, i.e. only those states s where the scheduler chooses a new action z == kk: This effect is even stronger if the skill lengths A E K are >> 1, as new scheduler action is sampled less frequently. While large skill lengths induce temporal correlations that are important for skill- based exploration, it has the side effect of decreasing the data actively contributing to improve the scheduler. In order to maximize sample efficiency without sacrificing exploration, we duplicate trajectories with long skill executions by also considering the same skill being executed repetitively
1 'The new skill is initially rarely executed as it commonly provides lower rewards than reloaded skills. We present an analysis of the advantages of using offline methods like CRR against off-policy methods like MPO in Appendix D.1
5

---

Under review as a conference paper at ICLR 2023
Figure 3: Domains: The experimental domains used: (from left right) manipulation Lift, Stack, Pyramid, Triple Stack and locomotion GoalScoring and GerUpAndWall.
for shorter amount of time More details regarding the augmentation are descrihed in Appendix A.2 and we include ablations in Appendix D.2
EXPERIMENTS
Environments We empirically evaluate our method robotic manipulation and locomotion do- mains. The goal of our evaluation is to study how different skill transfer mechanisms fare in new tasks which require skill reuse. In the manipulation setting (Fig. 3 (left),, utilise simulated Sawyer robot arm equipped with Robotiq gripper and a set of three objects green (g), yellow (y) and blue (b) (Wulfmeier et al. 2020). We consider four tasks of increasing complexity: Lift g, Stack on y, building a Pyramid of on y and b, and Triple stack with g on y and y on b. Every harder task leverages previous task solutions ability to build on previous knowledge and solve increasingly complex tasks. For locomotion (Fig. 3 (right)), we consider two tasks with a simulated version of the OP3 humanoid robot (Robotis) OP3): GetUpAndWalk and GoalScoring. The GetUpAndWalk task requires the robot to compose two skills: one to get up off of the floor and one to walk. In the GoalScoring task, the robot gets a sparse reward for scoring goal, with a wall as an obstacle. The GoalScoring task uses a single skill but is transferred to setting with different environment dynamics. This allows us to extend the study beyond skill composition to skill adaptability; both of which are important reguirements when operating in the real world. All the considered transfer tasks use sparse rewards except the GelUpAndWall task where a dense walking reward is given but only if the robot is standing. We consider an off-policy distributed learning setup with a single actor and experience replay. For each setting we plot the mean performance averaged across 20 seeds with the shaded region representing one standard deviation. More details on the skills and tasks can be found in Appendix
skills: this is natural transfer setting to test agent's
Analysis details Our method SkillS consists of two learning processes, one designed quickly collect useful data (the scheduler) and one to distill the best solution for the task from the same data (the new skill). As our primary interest is in the task solution, unless otherwise specified, we focus on the performance of the new skill when executed in the environment in isolation for the entire episode. See Appendix [C.2 for details about how the new skill and the scheduler can be separately evaluated. We compare SkillS against the following skill reuse methods: Fine-tuning, RHPO, KL- regularisation (KL-reg. to Mixture) and DAGGER. Hierarchical RL via RHPO (Wulfmeier et al.) 2020) trains Gaussian mixture policies, and we adapt this method to reload skills as the components of the mixture. We can then either freeze the skills and include new learned component (RHPO) or continue training the reloaded skills (fine-tuning). KL-reg. to Mixture adapts the MPO algorithm (Abdolmaleki et al.) 2018) to include a regularisation objective to distill the reloaded skills; and DAGGER adapts Ross al. (201T) for imitation learning with previous skills. full description of our baseline methods can be tound in Appendix B
5.1 ANALYSIS: SPARSE REWARD TASKS IN LOCOMOTION AND MANIPULATION
Fig. shows the performance of skill-based approaches on manipulation domaing We observe that among all the considered methods, only SkillS can learn effectively in these domains, even
3 For example, a trajectory of a skill executed for 100 timesteps could he generated equivalently by executing it once for 100 steps or 10 times consecutively for 10 steps each.
*When the reward provides reliable learning signal (e.g. in case dense rewards) there less need to generate good exploration data using previous skills. The advantage of skill-based methods is thus primarily in
domains where defining a dense, casier-to-optimie reward is challenging. For more details see Appendix (D.3 "Given the sparse nature of the reward, most methods fail to learn all and appear flat lines in the plot.

---

Under review as a conference paper at ICLR 2023
Ski IS
RHPO
From scratch
Fina-tuning
KL-reg, Mixture
DAGGER
600
940
502
Accui LMIN Seevers
440
820
320
200
100
5.0
22
04
26.
0.5
1.0
Epnoces
1ed
Epicoees
(a) Lift
(b) Stack
(c) Pyrami
(d) Triple Stacking
Figure 4: Performance of various skill transfer mechanisms sparse-reward manipulation tasks. For each task, all approaches are given the previously trained useful skills (see Appendix for details) to learn the new task. SkillS is the only method to consistently learn across all tasks.
Skill
scratch Fire suning
KL-reg, Mixture DAGGER
Seach Nan
arm Open frrges
Reach pelios
RHPD
Clead Imgerra
1602
1000
000
1.000
600
202
-00
150
102
1000
1
104
203
(a) GetUpAndWalk
(b) GoalScoring
(c) Skill execution.
Figure 5: (a) (b) Performance on skill transfer methods on locomotion tasks. SkilIS outperforms other methods in GetUpAndWalk (a) and is worse than other methods in the single skill GoalScor- ing task (b). (c) Skill average execution (as number of timesteps) during training.
though other methods have access to the same set of skills. When given access to Lift task, all baselines still struggle to learn how to Stack. In contrast, SkillS can even learn challenging tasks like TripleStack with enough data. Importantly this illustrates that skill reuse on its own is insufficient to guarantee effective transfer, and highlights the strong benefit of temporal abstraction. While other methods such as RHPO switch between skills high frequency, the scheduler's action space encourages the execution of skills over longer horizons, considerably reducing the search space.
Fig. B compares performance in the locomotion setting. In the GetUpAndWalk task, find SkillS outperforms the other methods, confirming that temporal abstraction is fundamental when solving tasks that require sequence of skills to be learned (getting up and then walking in this task). Executing the skills for longer horizons also generates higher reward early during training. Particularly poor is the performance of regularising towards mixture of these skills as it performs worse than learning from scratch. For the GoalScoring task which requires transferring a single skill to new dynamics, SkillS is competitive and shares the strongest results. As we will demon- strate Section 5.2.4 the advantage temporally correlated exploration conferred by SkillS is less important in settings with only one skill.
5.2 ABLATIONS
In this section we analyse the robustness of SkillS and evaluate the importance of various compo- nents and design decisions. particular, we analyse (I) the scheduler skill selection during training; (2) the robustness to the number and quality of skills used; (3) the benefit and flexibility of having separate infer mechanism and (4) the utility of flexible temporal abstraction. More ablations includ- ing a study of dense and sparse rewards, the benefit of offline methods like CRR and the importance of data augmentation can be found in Appendix D
5.2. SCHEDULER SKILL SELECTION
In this section we show how the scheduler typically chooses the skills interact with the environ- ment. Fig. |c displays the skill selection during the training by means of the average consecutive execution for each skill. We assume skills available, some of them needed for solving the task
7

---

Under review as a conference paper at ICLR 2023
Uiseful skills
Distractor skills
Usaful
distractors
Usatul in many distractors
Many useful distractoos
250
0000
320
150
520
200
100
X23
100
0
0.
0.4
D:6
0.2
1.0
0.0
0.5
1.0
3.5
2.0
2.5
3.0
Stacking
(h) Pyramid
(c) Triple Stacking
Figure 6: SkillS can learn even when the number and type of skills change although performance deteriorates on harder tasks.
at hand ('reach green', 'close fingers'), while the others act distractor (e.g. 'reach yellow' or "open fingers'). ). At the beginning the scheduler quickly selects the most useful skills ( ('reach green', "close fingers') while discarding those less relevant to the task (observe the drop in executing 'reach yellow' or 'reach blue'). ) While the scheduler does not choose the poorly performing new skill early in training, eventually (after roughly 2000 episodes) it is selected more often until finally it is the most executed skill.
5.2.2 ROBUSTNESS TO SKILL NUMBER AND QUALITY
Fig. 6 evaluates SkillS the stacking, pyramid and triple-stacking domains while changing the skill set S. For this analysis we vary the numbe and the types of skills by including a variety of distractor skills not directly relevant for the task (e.g. reach 'blue' when 'red' needs to be stacked; more details in Appendix C.1.3). With only distractor skills, SkilIS understandably fails to learn anything. For the easier tasks of Stacking and Pyramid, SkillS is quite robust long there some useful skills in the set S. On the challenging triple-stacking task, increasing the number of skills has detrimental effect performance. We hypothesize that the larger skill set increases the size of the search space to a point where the exploration skill space does not easily find the task solution.
5.2.3 BENEFIT OF SEPARATE COLLECT AND INFER MECHANISMS
We conduct an experiment to study the importance of a separate inference (learning) mechanism in extension of the hierarchical agent the scheduler. Fig. 7a compares the performance of the two components of SkillS: 'Scheduler (incl. new skill)' which has access to the new skill and frozen skills and 'New skill' which is the inferred new skill. In addition, (without new skill)' where the higher level scheduler only has access to the frozen skills. As the figure shows, having separate off- off--poiiy learning process is crucial for learning as the new skill (blue) outperforms the scheduler performance, even when can execute the new skill itself (red). The gap in performance is even more evident if the scheduler cannot access the new skill (light blue).
include version 'Scheduler
5.2.4 UTILITY OF FLEXIBLE TEMPORAL ABSTRACTION
Temporally-correlate exploration Figs. 7b and 7cl illustrate the importance of temporally- correlated exploration that arises from our formulation in Section For this analysis additionally run a version of our algorithm called 'SkilIS (without temporal abstraction)' where a new skill must be chosen at each timestep (K {1}). When given single (near-)optimal skil all methods including regularisation and fine-tuning perform similarly (Fig. 7b). However, when we use more than one sub- sub-optimal skill, temporally correlated exploration is crucial for learning (Fig. 7c).
Variable temporal abstraction Fig. analyses the importance the flexible temporal abstraction of SkillS by comparing our method against a version dubbed 'SkilIS (fixed temporal abstraction X)' where each skill, once chosen, is held fixed for a given duration of X timesteps ([10, 50, 100,
'In order to augment the number of skills, we use skills trained to solve same task (e.g. reach 'blue") from different seeds and we refer to these sets in Fig. as many distractors and many useful skills.
"The optrmal skill for the Stacking task places the green object the yellow one without moving the gripper away.
8

---

Under review as a conference paper at ICLR 2023
New skill
Scheduler tinel. skill)
Scills
.eg. RHPO (relowd)
Scheduler twithout new scill)
withoou tempor abstrection
000
600
500
300
500
200
200
100
102
Tpicodes
106
10d
10 scddes
(a) Benefit of collect and infer.
(b) Single optimal skill.
(c) Multiple sub-optimal skills.
Figure 7: (a) SkilIS performs best when given access to a separately inferred new skill. (b)-(c) The main advantage of SkillS is likely due temporally correlated exploration. When given access to a single optimal skill (b) all methods can learn effectively but with multiple sub-optimal skills (c), SkillS is the only method to learn (all other methods are flat lines). All results on the Stacking task.
Skills
Skills (fixed temporal abstraction
(fxed memoraal abstraction 150) (fixed te abstraction 200)
SkiliS (fied zemporal ebstrection 10)
Skills trixed temporal abstraciion 1000
Skill5
400
250
1200
200
150
300
100
200
300
0.0
04
10
Ealoods
184
Eptcodos
Eptcades
(a) Stacking
(b) Pyramid Stacking
(c) Triple stacking
Figure 8: The variable temporal abstraction of SkillS is especially erucial for learning harder tasks.
150, 200] timesteps in a 600 step episode). We observe that the optimal fixed skill length varies from task to task (see Fig. 8a vs 8b) and, most importantly. the performance on harder tasks (especially triple-stacking) degrades when using this constraint. Thus, flexibly choosing the duration for each skill is quite important.
DISCUSSION
Transferring the knowledge of reinforcement learning agents via skills or policies has been a long- standing challenge. Here, we investigate different types of task and dynamics variations and in- troduce a new method, SkilIS, that learns using data generated by sequencing skills. This approach demonstrates strong performance in challenging simulated robot manipulation and locomotion tasks. It further overcomes limitations of fine-tuning, imitation and purely hierarchical approaches. achieves this by explicitly using two learning processes. The high-level scheduler learn quence cemporally-extendee skills and bias exploration towards useful regions of trajectory space, and the new skill can then learn optimal task solution off off-policy.
There are a number of interesting directions for future work. For instance, unlike in the options framework (Sutton et al.) 1999) which associates a termination condition with each option, in our work the duration of execution of skill is independent of the pretrained skill. While this pro- vides more flexibility during transfer, it also ignores prior knowledge of the domain, which may be preferable some settings. Similarly, the high-level controller is currently trained maximize task reward. However, it may be desirable to optimize the scheduler with respect to an objective that incentivizes exploration, e.g. reduce uncertainty with regards the value or system dynam- ics, improving its ability to collect data in harder exploration problems. Finally, our approach can naturally be combined with transfer of forms of prior knowledge apart from skills, such prior experience data, the value function (Galashov et al. 2020), or a dynamics model (Byravan et al. 2021) and we expect this provide fertile area of future research.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference Learn ing Representations, 2018.
Abbas Abdolmaleki, Sandy H Huang, Giulia Vezzani, Bobak Shahriari, Jost Tobias Springenberg, Shruti Mishra. Dhruva TB, Arunkumar Byravan, Konstantinos Bousmalis, Andras Gyorgy, et al. On multi-objeccive policy optimization as a tool for reinforcement learning. arXiv preprint arXiv:2106.08199 2021.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in Neural Information Processing Systems, 33:3884-3894 2020.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Daniel S Bernstein. Reusing old policies to accelerate learning on new mdps. Technical report, Citeseer, 1999.
Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel, Fereshteh Sadeghi, Leonard Hasenclever, Yuval Tassa, Emilio Parisotto, Jan Humplik, Tuomas Haarnoja, Roland Hafner, al. Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors. arXiv preprint arXiv:2203.17138 2022.
Mike Bowling and Manuela Veloso. Reusing learned policies between similar problems. In in Proceedings of the A/ AI-98 Workshop on New Trends in Robotics. Citeseer, 1998.
Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim, Mehdi Mirza, Alessandro Davide Ia- longo, Yuval Tassa, Jost Tobias Springenberg, Abbas Abdolmaleki, Nicolas Heess, Josh Merel, and Martin Riedmiller. Evaluating model- planning and planner amortization for continuous control, 2021. URL TRLbttps:: / org/ http://ga//////.a///////// 03363
Arunkumar Byravan, Jan Humplik, Leonard Hasenelever, Arthur Brussee, Francesco Nori, Tuo- mas Haarnoja, Ben Moran, Fereshteh Sadeghi, Bojan Vujatovic, and Nicolas Heess. NeRFSim: Sim2real transfer of vision- vision-guided bipedal motion skills using neural radiance fields. In the Inter- national Conference on Robotics and Automation (under review), 2023.
Víctor Campos, Pablo Sprechmann, Steven Stenberg Hansen, André Barreto, Steven Kapturowski, Alex Vitvitskyi, Adrià Puigdomènech Badia, and Charles Blundell. Beyond fine-tuning: Trans- ferring behavior in reinforcement learning. 2021.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pp. 273-281. PMLR, 2012.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning, arXiv preprint 1iXxv1170400002.. 2017.
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL.rregularized RL. In International Conference on Learning Representations, 2019.
Alexandre Galashov. Jakub Sygnowski, Guillaume Desjardins, Jan Humplik, Leonard Hasenclever, Rae Jeong, Yee Whye Tch, and Nicolas Heess. Importance weighted policy learning and adaption. CoRR, abs/2009.0493 04475, 2020. httttttss///axttt org/ abs/ 2009. 04875
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pp- 1851-1860, 2018.
10

---

Under review as a conference paper at ICLR 2023
Roland Hafner, Tim Hertweck, Philipp Klöppner, Michael Bloesch, Michael Neunert, Markus Wulfmeier, Saran Tunyasuvunakool, Nicolas Heess, and Martin Riedmiller. Towards general and autonomous learning of core skills: A case study in locomotion, arXiv preprini arXiv:2008. 12228, 2020.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference on Learn- ing Representations (ICLR), 2018.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610.0511 2016.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint 20Xi:20006008826, 2020.
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2): 328-373, 2013.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: Aekiv:/900.0 1907.00456, 2019.
Rae Jeong, Jost Tobias Springenberg, Jackie Kay, Daniel Zheng, Yuxiang Zhou, Alexandre Galashov, Nicolas Heess, and Francesco Nori. Learning dexterous manipulation from suboptimal experts. arXiv preprini arXiv:2010.0858/ 2020.
Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. AAMAS (1). pp. 299-306 Citeseer, 2008.
Thomas Kipf, Yujia Li, Hanjun Dai, Vinieius Zambaldi, Alvaro Manchez-Gonzaler Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. International Conference on Machine Learning, pp. 3418-3428, PMLR, 2019.
Björn Krüger, Anna Vögele, Tobias Willig, Angela Yao, Reinhard Klein, and Andreas Weber. Eff- cient unsupervised temporal segmentation of motion data. IEFE Transactions on Multimedia, 19 (4):797-812, 2016.
Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. arXiv preprint arXiv:1906.00 1906.0586, 2019.
Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, and Jan Peters. Probabilistic segmentation applied to an assembly task. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoidss, pp. 533-540. IEEE, 2015.
Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, and Jan Peters. Learning movement primi- tive libraries through probabilistic segmentation. The International Journal of Robotics Research, 36(8):879 894, 2017.
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, S. M. Ali Eslami, Daniel Hennes, Wojeiech M. Czar- necki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, Noah Y. Siegel, Leonard Hasen- clever, Luke Marris, Saran Tunyasuvunakool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarnoja, Brendan D. Tracey, Karl Tuyls, Thore Graepel, and Nicolas Heess. From mo- tor control team play in simulated humanoid football, 2021. URL httphhtts//ett. http::s////xxvvvoooo
2bbs/2005.111
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference Robot Learning (CoRL., 2019.
Sebastian Madgwick et al. An efficient orientation filter for inertial and inertial/mmaneticc sensor arrays. Report x-io and University of Bristol (UK), 25:113-118, 2010.

---

Under review as a conference paper at ICLR 2023
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations (ICLR), 2019.
Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3): 263 263-279, 2013.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv: (et:::9999.060. 10618, 2019.
Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe. Tobias Springen- berg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller. Continuous-discrete reinforcement learning for hybrid control in robotics. Conference on Robot Learning, pp. 735-751. PMLR, 2020.
Scott Niekum and Andrew G Barto. Clustering via dirichlet process mixture models for portable skill discovery. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In ICML., 2022.
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic move- ment primitives. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Alexandros Paraschos, Christian Daniel, Jan Peters, and Gerhard Neumann. Using probabilistic movement primitives in robotics. Autonomous Robots, 42(3):52--551, 2018.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint 1r 2015.
Xue Bin Peng. Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning composable hicrarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems, 32, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage weighted regression: Simple and scalable off- off-policy reinforcement learning, 2020. URL httttt////pppeeeiieee
34FvS
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.1199 19944 2020.
Mare Pickett and Andrew G Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In ICML, volume 19, pp. 506-513, 2002.
Krishan Rana, Vibhavari Dasagi, Jesse Haviland, Ben Talbot, Michael Milford, and Niko Sünderhauf. Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics. arXiv preprint arXiv:2107.09822. 2021.
Pravesh Ranchod, Benjamin Rosman, and George Konidaris. Nonparametric bayesian reward seg- mentation for skill discovery using inverse reinforcement lcarning. In 2015 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 471-477 IEEE, 2015.
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, and Raia Hadsell. Learning transferable motor skills with hierarchical latent mixture policies. International Confer- ence on Learning Representations, 2022.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Proceedings of the 35th International Conference on Machine Learning, pp. 4344-335 2018.
12

---

Under review as a conference paper at ICLR 2023
Martin Riedmiller, Jost Tobias Springenberg, Roland Hafner, and Nicolas Heess. Collect & infer-a fresh look at data-efficient reinforcement learning. Conference on Robot Learning, pp. 1736- 1744. PMLR, 2022.
Robotis OP3. introduction/
URL http:////aaaaa.....o.ooooo//////
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell, Policy distil- lation. arXiv preprint arXiv: urXiv:/511.0622. 1.06295, 2015.
Sasha Salter, Markus Wulfmeier, Dhruva Tirumala, Nicolas Manfred Otto Heess, Martin Ried- miller, Raia Hadsell, and Dushyant Rao. Mo2: Model-based offline options. 2022.
Stefan Schaal, Jan Peters, Jun Nakanishi, and Auke Ijspeert. Learning movement primitives. In Robotics research. the eleventh international symposium, 2005.
Simon Schmitt, Jonathan J. Hudson, Augustin Zidek. Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Küttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning. ArXiv, abss/100.00885, 2018.
Tim Seyde, Wilko Schwarting, Igor Gilitschenski, Markus Wulfmeier, and Daniela Rus. Strength through diversity: Robust behavior learning via mixture policies. In Conference on Robot Learn- ing, pp. 1144-1155, PMLR, 2022.
Bobak Shahriari, Abbas Abdolmaleki, Arunkumar Byravan, Abe Friesen, Siqi Liu, Jost Tobias Springenberg, Nicolas Heess, Matt Hoffman, and Martin Riedmiller. Revisiting gaussian mix- ture critics off-policy reinforcement learning: sample-based approach, 2022. URL https:
Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2019.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning, pp. 4654-4663, PMLR, 2018
Noah Y Siegel, Jost Tobias Springenberg. Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.0833% 2020.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Par- rot: Daia-Driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Daniel Tanneberg, Kai Ploeger, Elmar Rueckert, and Jan Peters. Skid raw: Skill discovery from raw trajectories. IEEE Robotics and Automation Letters, 673),466-47703, 2021.
Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. Advances in neural information processing systems, 7. 1994.
D Tirumala, H Noh, A Galashov, I. Hasenclever, and others. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprint arXiv, 2019.
13

---

Under review as a conference paper at ICLR 2023
D Tirumala, A Galashov, H Noh, L Hasenclever, and others. Behavior priors for efficient reinforce- ment learning. arXiv preprint arXiv, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033,
2012. doi: 10.1
Lisa Torrey, Jude Shavlik, Trevor Walker, and Richard Maclin. Relational macros for transfer in reinforcement learning. In International Conference on Inductive Logic Programming, pp. 254 268. Springer, 2007.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulechre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2019.
arXiv preprint arXiv:1911.111
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg. Michael Ne- unert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Com- positional transfer hierarchical reinforcement learning. arXiv preprint arXiv: /906/ //288, 2019.
Markus Wulfmeier, Abbas Abdolmalcki, Roland Hafner, Jost Tobias Springenberg, Michael Ne- unert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin A Riedmiller. Compositional transfer in hierarchical reinforcement learning. Robotics: Science and Systems, 2020.
Markus Wulfmeier, Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim Her- tweek, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Heess, et al. Data-efficient hind- sight off-policy option learning. In International Conference Machine Learning, pp. 11340- 11350. PMLR, 2021.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference Learning Representations, 2020.
APPENDIX
A SKILLS DETAILS
A. HCMPO POLICY EVALUATION
In order to derive the PE updates we write down the probability that the scheduler chooses an action 2, and the counter Ct.
This takes into account the fact that, ht time steps have passed, i.e. the counter c has expired, the scheduler chooses new action 24 [ig: ky] at time It keeps the previous action 2 2 ii kt-1] otherwise.
T(2,, Ct 18t, 2t-1, (-1)
( = B) if q-1 = =1
T(,, k2 celse, i- 12
16(hs (k-k) - kr-1) (c--1 1)) otherwise.
(5)
We can now use Eq. to derive the PE update rules reported in Eq.
A.2 DATA AUGMENTATION
We define skill trajectory T as the sequence of k. time steps where the same skill with index it is executed
((8t, 2t [i [i",k"' (1
14

---

Under review as a conference paper at ICLR 2023
T
k- 60
8
T
k Kmin 10
Figure 9: Example of data augmentation. Let's assume we have a trajectory where the skill with index i' is executed for k- = 60 time steps. The duplicated trajectory is generated by pretending the scheduler chose the same skill with index i* every kmin time steps, equal to 10 our example. This generates new trajectory F compatible with the low-level actions d, and state st actually collected during the interaction with the environment.
and kmin the smallest skill length available to the scheduler (kmin S k for & E K and kimin
For every trajectory T with skill length k' that can be written multiple of the minimum skill length kmin (mod ke, kminn 0), we generate new duplicated skill trajectory pretending it was generated by scheduler choosing the same skill i" multiple times with the smallest available skill length kmén (see Fig. The duplicated skill trajectory F
== (181, 2t = [i" J,",, Ce, F2) )t=t
is such that:
(st, žt = =1"
(8t, 2t = ii" kminn Ct 1( Kmmen -
-1) -1)
kimin r2) if mod(ks, kmin)) = O, Ct > kmin
(st, (i* kminl ve) if mod(ki, kmmn) = 0, C2 S kmin
(
(Ses 2t [i* k"), (,, r2) otherwise.
This generates a new trajectory that compatible with the data actually collected, but with more samples that are actively used for improving the scheduler and the policy, as it pretends the scheduler made decision in more states. Both т and are added to the replay and used for training the scheduler. This perspective is highly related to efficiently learning high-level controllers over sets of options in the context of intra-option learning (using data throughout option execution to train the
inter option policy) (Sutton et al. 1999 Wulfmeier et al. 2021
A.3 BIAS ON INITIAL SKILI. LENGTH
Our SkilIS formulation enables the possibility of specifying initial bias in the scheduler choices. It possible to encourage the scheduler initially choose one more skills and/or skill lengths. In all our experiments we bias the scheduler to initially choose the largest skill length. This way all the skills are executed for considerably large number of timesteps the beginning of training, helping the scheduler understand their utility for the task at hand.
B BASELINE DETAILS
In this section we describe the methods used as baselines in the main text. For all of the baselines described below, we use weighted loss that trades off the baseline with CRR using parameter a. In other words when a = 1, we use pure CRR whereas a 0 runs the pure algorithm.
B. MPO
When learning from scratch and for our baselines we largely adapt the state-of-the-art off-policy learning algorithm Maximum a-posteriori Policy Optimisation (MPO) (Abdolmaleki et al.) 2018). MPO optimises the RL objective in Eq. Uuiig an Expectation Maximisation (EM) style approach. The algorithm operates in three steps: a policy evaluation step to learn the state-action value function
15

---

Under review as a conference paper at ICLR 2023
RHPO with frozen skills
RHPO with fine-tuning
600
Accumulatted Reward
400
man of
500
200
100
0.0
0.2
0.4
0.6
0.8
1.0
Falsades
le4
Figure 10: Impact of freezing skills: Single skill transfer to stacking.
(*, a): E-step to create improved non-parametric policy using data sampled from replay buffer; and an M-step that fits a parametric policy rals) to the non-parametric estimate.
For the policy evaluation step use Mixture of Gaussian (MoG) critic as described by Shahriari ct al. (002)). MPO then optimizes the following objective in the E-step:
max | p(s)) /. q(als))((s, a) da ds 9
I. f 1 (a(as))))())) ds <e:
where g(als) an improved non non-parametric policy that is optimized for states yus) drawn from the replay buffer. The solution for q is shown to be:
a(als) x (als, 0) exp( Qo(s,a) a)
(10)
11"
Finally in the M-step, an improved parametric policy a (als) is obtained via supervised learning:
M
N
π n+1 arg Σ
max
Σ
Ti log logmo(a,)s)).
8. s. DIL. (TT (alss)](
Importantly, the M-Step also incorporates KL-constraint to the previous policy x" (a]s) create a trust-region like update that prevents overfitting to the sampled batch of data. our experiments, an extension of MPO, dubbed Regularized Hierarchical Policy Optimisation (RHPO), that learns a Mixture of Gaussian policy (Wulfmeier et al. (020)) instead a single isotropic Gaussian.
B.2 FINE-TUNING AND RHPO
For our comparison of skill reuse, reload the mixture components of an RHPO agent using the parameters of the previously trained skills. In addition we consider two scenarios: a) continuing to fine-tune the mixture components that were reloaded or b) introducing a new mixture component (similar to the new policy of SkillS) and freezing the other mixture components that were reloaded using skills. Freezing the skills is often advantageous to prevent premature over-writing of the learnt behavior an important feature which can add robustness early in training when the value function is to be learnt. We illustrate this effect in Figure 10 in the 'Stacking' setting where we transfer only single skill. As shown, when the previous skill fixed the retained knowledge can be used to bootstrap learning whereas direct fine-tuning can cause catastrophic forgetting effect where learning performance is significantly worse.
B.3 KL-RREGULARISATIO
KI-regullaristton typically involves the addition of an extra term to the RI. objective function that minimises the Kullback-Leibler (KI.) divergence between the policy and some prior distribution
16

---

Under review as a conference paper at ICLR 2023
Tirumala al., 2020 Here, define the prior distribution be uniform mixture the previously trained skills, given by:
To
W x n. (a(s)
(11)
W;
Vi
N
where (als)) are N previously trained skills.
As such, the KL objective can be incorporated into any RL algorithm. In this work we adapt MPO to include regularisation to prior. As presented above, MPO includes KL-constraints in both the E and M steps, either of which can be used to further regularise to the prior. In this work, we adapt the E-step to regularise against mixture distribution. In this setting, the non- non-parametric distribution for the E-step can obtained as:
gals)) To(als, 0) exp( 2o(s,a), "
(12)
One interpretation of this objective is that the prior serves as a proposal distribution from which actions are sampled and then re- weightttd according to the exponentiated value function. The opti- mal temperature 17" here is obtained by minimising:
g(n) T7c
f To(aass 0) exp( 2o(5,0) a)
+ 1l ((() log
(13)
where E is a hyper-parameter parameter that controls the tightness of the KL-bound. A smaller more tightly constrains the policy to be close the prior distribution whereas a looser bound allows more devia- tion from the prior and reliance on the exponentiated Q function for learning.
B.4 DAGGER
The final baseline consider is adaptation of the method DAGGER (Dataset Aggregation) introduced by Ross et al. (201T). DAGGER is an imitation learning approach wherein a student policy is trained to imitate teacher. DAGGER learns to imitate the teacher the data distribution collected under the student. In our formulation, the teacher policy is set to be a uniform mixture over skills, To as defined in Eq. 11 We train the student to minimize the cross-entropy loss this teacher distribution for a state distribution generated under the student. The loss for DAGGER is thus given by:
LDAGGER = -arg min (o((as)) log=(aal))
(14)
In addition, consider imitation + RL setting where the DAGGER loss is augmented with the standard RL objective using a weighting term ODAGGER
L aDAGGER LDAGGER + (1 aDAGGeR) LMPO-
(15)
In our experiments we conduct a hyper-parameter sweep over the parameter DDGGERR and report the best results.
C EXPERIMENT DETAILS
C.I TASKS AND REWARDS
C.1.1 MANIPULATION
In all manipulation tasks, we use a simulated version of the Sawyer robot arm developed by Rethink Robotics, equipped with Robotiq 2F85 gripper. In front of the robot, there is basket with a base size of 20x20 em and lateral slopes. The robot can interact with three cubes with side 4 cm, coloured respectively in yellow, green and blue. The robot is controlled in Cartesian velocity space
17

---

Under review as a conference paper at ICLR 2023
Entry
Dimension
Unit
End-effector Cartesian linear velocity Gripper rotational velocity Gripper finger velocity
m/s
rad/s
rad/s
Table
Robot action space manipulation tasks.
Entry
Dimension
Unit
Joint Angle
rad 7
Joint Velocity
rad/s
Pinch Pose
m, au rad rad/s n/a m, au
Finger angle
1
Finger velocity
Grasp
1
Yellow cube Pose
Yellow cube pose wrt Pinch
7
m, au
Green cube Pose
7
m, au
Green cube pose wrt Pinch
7
m, au
Blue cube Pose
7
m,
Blue cube pose wrt Pinch
7
m, au
Table 2: Observations used in simulation. In the table au stands for arbitrary units used for the quaternion representation.
with a maximum speed of 0.05 m/s. The arm control mode has four control inputs: three Cartesian linear velocities well rotational velocity of the wrist around the vertical axis. Together with the gripper opening angle, the total size of the action space five (Table TT. Observations used for experiments are shown in Table 2 For simulating the real robot setup use the physics simulator MuJoCo (ooorov et al. 2012). The simulation is run with a physics integration time step of 0.5 milliseconds, with control interval of 50 milliseconds (20 Hz) for the agent.
C.1.2 LOCOMOTION
We use a Robotis OP3 Robottis OP3) humanoid robot for all our locomotion experiments. This hobby grade platform is 510 mm tall and weighs 3.5 kg. The actuators operate in position control mode. The agent chooses the desired joint angles every 25 milliseconds (40 Hz) based on on-board sensors only. These observations include the joint angles, and the angular velocity and gravity direction of the torso. Gravity is estimated from IMU readings with Madgwick (Madgwick al.) 2010) filter that runs at higher rate. The robot has also web camera attached to the torso, but we ignore in our experiments. To account for the latency in the control loop, we stack the five observations and actions together before feeding them to the agent. Further, to encourage smoother exploration, we apply an exponential filter with a strength of 0.8 the actions before passing them to the actuator. We also assume we have access to the robot's global position and velocity via an external motion capture system. This privileged information is, however, only used evaluate the reward function, and is not visible to the agent. Although we train our policies in simulation, the model has been carefully tuned to match the physical robot, allowing policies directly from simulation to robot (Byravan et al. 2023
zero-shot transfer the
C.1.3 SKILLS
The skills used for each task are shown in Table All the plots in the paper are shown using only the skills under the column "Useful skills'. The other sets of skills are used for the ablation in Fig. 6 In particular, when in Fig. we mention 'Many Useful skills' 'Many Distractor skills' it means that we use 3 skills trained for the same task, e.g. "reach green", but from different seeds (e.g. 'Many useful skills and distractor skills' for Pyramid is total of 31 skills).
18

---

Under review as a conference paper at ICLR 2023
Task Lift
Useful skills
No.
Distractor skills
No. 4
Reach green,
Reach yellow, Reach blue, 2
(green)
Lift arm with closed fingers
Open fingers, Lift arm with open fingers Distractor skills for Lift green, Lift yellow, Lift blue
Stack (green on yellow)
Useful skills for Lift green,
Lift green.
Hover green on yellow
Pyramid (green on top)
Useful skills for Stack,
Lift arm with open fingers, Open fingers, Stack yellow on blue
3
Stack green on yellow,
Lift yellow, Lift blue
Reach yellow, Reach blue
Triple stacking
Useful skills for Stack,
Lift arm with open fingers, Open fingers, Reach blue, Lift blue
(green on yellow on blue)
Stack green on yellow,
Stack yellow on blue,
Lift yellow, Reach yellow,
Hover yellow on blue,
Table 3: Skills used for each manipulation task. Every column with title "No.' counts the number of skills in the left-hand side column, e.g. 'Lift green' has Useful skills.
C.1.4 REWARDS MANIPULATION
The skills of Table 3 are obtained either by training them from scratch with staged rewards (in particular, the basic skills such as "Open fingers') or they are the result of a SkillS experiment, Le. they represented by the New Skill and have been trained with sparse rewards. The reward functions used for this aim are the following.
Reach object- dense
iff Poripner Paripner-Pobjeen Pobject tol
S
Rreach
(16)
tanli 2
Porioper Pobient
otherwise,
zolgon
where Poripper E R is the position of the gripper. Pobject E R the position of the object, tol = [0.055, 0.055, 0.02 the tolerance in position and scaling factor.
Open fingers- dense
iff Pringer Paostved talyon
S
Ropen
(17)
1 tanh Pfimger -atee-Paaaae Pdestred tol
e)) otherwise,
where Pfinger E R is the angle of aperture of the gripper, Pdesired E R. is the angle corresponding to the open position, tolpos = -9 the tolerance in position and scaling factor.
Lift arm with open fingers dense
iff Zarm Edesired otherwise,
Rast
(18)
Zurm -urmm-uut Emiin Zdenived Zeriiss
where Zarm E R is the z-coordinate of the position of the gripper, Xmiin 0.08 E R is the minimum height the arm should lift receive non-zero reward and Zdestred 0.18 e R is the desired z-coordinate the position of the gripper should reach to get maximum reward. The reward to have both the moving up and the fingers open is given by:
Riiftopen Risf Ropen-
(19)
Lift arm with closed fingers dense
Ryrasp = 0.5 + Rgrasp.auz):
(20)
where Retosed is computed with the formula of Eq. equation but using as Pdesired the angle corresponding to closed fingers this time;
E R
{
grasp detected by the grasp sensor otherwise.
Rgrasp.auz
0
(21)
19

---

Under review as a conference paper at ICLR 2023
Then the final reward is:
Riift.looze Riift Rerasp
(22)
Lift object- sparse
Bust = {
iff Robject
2 Edesired
(23)
otherwise,
where Zobject E is the 7-coordinate the position of the object lift and Edesired 0.18 the desired 7-coordinate the object should reach to get maximum reward.
Hover object another dense
iff
Phottom oof
Rhover 1 tanh 2 Phottam - off feet
tolpos
||
otherwise
(24)
where Ptop E R. is the position of the top object, Phottom E R3 the position of the bottom object, of set [0, objectheight 0.04]. tolpos = [0.055, 0.055, 0.02) the tolerance in position and a scaling factor.
Stack top object on bottom object sparse
{ 0
iff de, (Props Poottom) I tolz,y & Id (Ptops Pbottom) - ddesiredl S. tol & Rgraspaaz =
Ratack
(25)
otherwise,
where (...) E R is the norm of the distance of 2D vectors (on and y), dz(::)) E R is the (signed) distance along the z-coordinate, Ptop R and Phottom R are respectively the position the top and bottom object, tolz,y = 0.03 and fol, = 0.01 are the tolerance used to check if the stacking is achieved, ddesired 0.04 is obtained as the average of the side length of the top and bottom cubes.
The final reward we use for stacking is:
Rytack,leave Rstack (1 Preach,sparsse
(top) (top))
(26)
where sparsch.pparsttorr the sparse version of Ryeach in Eq. equation 24 for the top object.
Pyramid sparse
iff de y (Pbotttoom Poottom, b) S tol t, & da (Ptops Photsom
< lol &
R Rpyramid
ld (Prop: Phottom) ddesired S tol &
(27)
Ryraspiaux otherwise,
where (..) R is the norm of the distance of 2D vectors (on and y). dg(:,)) R is the (signed) distance along the z-coordinate, Ptop: Pbottom,a: Pbottom,b R$ spectively the position of the top object and the 2 bottom objects, while Photiom =
Pbotcom.a a Pbutcombb E R the average position the two bottom objects, tola.y,
0.08, tol 0.03 and tol. 0.01 the tolerance used to check if the pyramid config- uration is achieved, deteeired = 0.04 is obtained as the average of the side length of the top and bottom cubes. The final reward use for our tasks is:
Regramiid,eave Rpyramidd (1 1--eeachh sporse (top)).
(28)
Triple stacking sparse The triple stacking reward given by:
Reriple stack leave = Rstack (lop, middle) -Rstack (middle, bottom) (1 111-Rreacee Rreachh.sparse (lop)).
(29)
20

---

Under review as a conference paper at ICLR 2023
C.1.5 REWARDS LOCOMOTION
In the GetUpAndWalk task, condition the robot's behavior on goal observation: a 2-d unit target orientation vector, and a target speed in m/s. In these experiments, the speed was either 0.7m/s Om/s. The targets are randomly sampled during training. Each target is kept constant for an exponential amount of time, with an average of 5 seconds between resampling.
The reward per time step is a weighted sum Rwalk with terms as described below.
While this reward is dense in the observation space, 2/3 of episodes are initialized with the robot lying prone on the ground (either front or back). The effective sparsity results from the need to first complete the challenging get-up task, before the agent can start to earn positive reward from walking.
GetUpAndWalk
Rwatk = Rorient + Reclocity + Rupright + Raction + Rpose + Rground" + 1, (30)
where Rorient is the dot product between the target orientation vector and the robot's s current heading; Rvelocity is the norm of the difference between the target velocity, and the observed planar velocity of the feet; Rapright is 1.0 when the robot's orienta- tion is close to vertical, decaying to
outside margin of 12.5% Raction penalizes the squared angular velocity of the output action controls averaged over all 20 joints,
2 Ejoint (oioint - wgcit)) Rpose regularizes the observed joint angles toward refer-
ence "standing" pose Ejoint (pioint - -0jojnt) and Rground = -2 whenever the robot brings body parts other than the feet within 4cm of the ground.
In GoalScoring, the robot is placed in a 4m x 4m walled arena with a soccer ball. It must remain in its own half of the arena. It scores goals when the ball enters goal area, 0.5m x Im, positioned against the centre the back wall in the other half. An obstacle (a short stretch of wall) is placed randomly between the robot and the goal.
GoalScoring
R gooalscoring = Rycore + Rupright + Rmaavelocity
(31)
where: Recore is 1000 on the single timestep where the ball enters the goal region (and then becomes unavailable until the ball has bounced back to the robot's own half); Rmaarvlocitty is the norm of the planar velocity of the robot's 's feet in the robot's forward direction; Rupright the same as for GelUpAndWalk. Additionally, episodes are terminated with zero reward if the robot leaves its own half, or body parts other than the feet come within 4cm of the ground.
C.2 DETAILS ON HOW WE EVALUATE PERFORMANCE
Fig.
12
show how the performance of the scheduler and the new skill can be evaluated separately.
C.3 TRAINING AND NETWORKS
For all our experiments we use Multilayer Perceptron (MLP) network torsos with a network head output that twice the size of the action dimension for the task. The network output is split into the mean and log standard deviation that is used to parameterize the output of an isotropic Gaussian distribution. We use single actor and learner, a replay buffer for off policy learning and controller that ensures a fixed ratio between actor steps and learner updates across experiments. For each method and hyper-parameter setting results are averaged across 5 random seeds. In what follows we provide a detailed description of hyper-pant parrr-parseeterss used for the various experiments.
C.3.1 MANIPULATION
The following parameters are fixed across all methods compared in the Manipulation setting (SkilIS and baselines):
21

---

Under review as a conference paper at ICLR 2023
states
states
Skill Scheduler
Environment
New Skill
Environment
Frozen skills
New Skill
actions
actions
(a)
(b)
Figure 12: During evaluation, the new skill executed the environment for the entire episode (a).
This differs from the way the data is collected at training time, where the new skill can interact with the environment only when selected by the scheduler (b).
Learner parameters
Batch size: 256
Trajectory length: 10
Learning rate: 3e-4
Min replay size to sample: 200
Samples per insert: 50
Replay size: le6
Target Actor update period: 25
Target Critic update period: 100
E-step KL. constraint e: 0.1
M-step KL. constraints: 5e-3 (mean) and 1e-5 (covariance)
Network parameters
Torso for Gaussian policies: (128, 256, 128)
Component network size for Mixture policies: (256, 128)
Categorical network size for MoG policies: (128.)
Scheduler specific parameters
Scheduler batch size: 128
Scheduler learning rate: le-4
Min replay size to sample for scheduler: 400
Samples per insert: 100
Scheduler target actor update period: 100
Scheduler target critic update period: 25
Available skill lengths: K = {n 10
Initial skill lengths biases: 0.95 for k 100, 0.005 for the other skill lengths
22

---

Under review as a conference paper at ICLR 2023
Hyper-parameter sweeps
a (MPO v/s CRR loss) - (0, 0.5, 1.0)
KL-reg: c- (0.1, 1.0, 100.)
DAGGER: &DAGGER (0, 0.5, 1.0)
RHPO: Fine-tune freeze parameters
C.3.2
LOCOMOTION
Learner parameters
Batch size: 256
Trajectory length:
Learning rate: le-4
Min replay size to sample: 128
Steps per update: 8
Replay size: le6
Target Actor update period: 25
Target Critic update period: 100
E-step KL constraint 0.1
M-step KL constraints: 5e-3 (mean) and le-5 (covariance)
Network parameters
Torso for Gaussian policies: (128, 256, 128)
Component network size for Mixture policies: (256, 128)
Categorical network size for MoG policies: (128.)
Scheduler specific parameters
Scheduler batch size: 128
Scheduler learning rate: lc-4
Min replay size to sample for scheduler: 64
Steps per update: 4
Available skill lengths: K = {n 10
Initial skill lengths biases: 0.95 for k 100, 0.005 for the other skill lengths
Hyper-parameter sweeps
a (MPO v/s CRR loss) (0, 0.5, 1.0)
KL-reg: (0.1, 1.0, 100.)
DAGGER: ODAGGER (0, 0.5, 1.0)
RHPO: Fine-tune or freeze parameters
D ADDITIONAL EXPERIMENTS
D.1 IMPACT OF TRADE-OFF BETWEEN MPO AND CRR LOSS
Fine-tuning with offline learning. As discussed in Section 5 of the main text, fine-tuning often leads to sub-optimal performance since useful information can be lost early in training. In Fig.
13 we observe that fine-tuning with offline-RI. loss like CRR improves stability and learning
23

---

Under review as a conference paper at ICLR 2023
Finetuning (w/ MPO)
Finetuning (w/ CRR)
SkillS (w/ CRR)
(w/ MPO)
1000
600
Accumulated Reward
500
Accumuleted Reward
800
400
600
300
400
200
200
100
0
0
1.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Episodes
1e4
Fpisodes
le4
Figure 13: Comparison between regular- ized fine-tuning (CRR) and vanilla fine-tuning (MPO) on the GoalScoring task. Offline learn- ing methods like CRR typically regularize to- wards the skill (via data the replay) which can improve learning stability when fine-tuning.
Figure 14:
Comparison when using CRR v/s
MPO to infer the new skill on the Stacking task.
data augmentation
augmentation
200
100
Acumm labec
200
150
100
100
0.00
50
1pisodss
164
Esisedes
ted
Epssodes
(a) Stacking
(b) Pyramid
(c) Triple stacking
Figure 15: Data Augmentation:
Data augmentation becomes crucial when addressing harder tasks.
performance. Many offline methods, including CRR, typically regularise towards the previous skill either explicitly through the loss or implicitly via the data generated by the skill (see Tirumala et al. this). We hypothesize that such regularisation could help
(2020) for more detailed discussion mitigate forgetting and hence improve learning. While this insight requires a more detailed analysis that is outside the scope of this work, however have evidence that when fine-tuning with a single skill a potentially simple algorithmic modification (from off-policy to offline RL) might already make learning more robust.
CRR for new skills. In Fig. 14 compare what happens when the new skill is inferred with an offline learning algorithm (CRR) against an off-policy algorithm (MPO) on the stacking domain from Section 5 We observe that learning is improved when using an offline algorithm like CRR. the experiments presented in the main text use CRR as the underlying skill
For this reason, inference mechanism although the method itself is somewhat robust and agnostic to this choice. For a fair comparison, we also include sweep using CRR as the underlying agent across when selecting the best performing baseline. However, since CRR implicitly regularises towards the skill, there is typically no effect when compared using MPO when regularising towards mixture.
D.2 IMPACT OF DATA AUGMENTATION
Fig. 15 shows how data augmentation improves the performance, especially for more challenging tasks.
24

---

Under review as a conference paper at ICLR 2023
scratch
KL-reg. Mixture
DAGGER
600
600
Accumnllated Reward
500
Accumulsted Reward
400
400
300
300
200
200
100
100
0.
0.0
0.4
0.6
1.0
0.0
0.2
0.6
0.8
Episodes
1ef
Esisades
1e4
(a) Lifting
Stacking
Figure 16: Performance on rewse-reward manipulation tasks, With carefully- carefully-tuned shaped rewards on simpler tasks, skill reuse is not as beneficial and all methods can learn effectively.
SkillS
SkillS
KL-reg.
SkillS RHPO
600
600
Accuniulated Reward
500
Accumiuleted Reward
500
400
400
300
300
200
200
100
100
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4,0
0.0
0.2
0.4
0.6
0.8
1.0
Eo sodes
163
Ebisodes
led
(a) Lifting
(b) Stacking
Figure 17: Performance of SkilIS when combined with other approaches to transfer, showing its flexibility.
D.3 PERFORMANCE WITH DENSE REWARDS
Fig. shows the performance with all skill transfer methods on the lift and stacking tasks when the agent has access to dense rewards. We observe that, in contrast to the sparse reward results of Fig. all methods learn effectively when provided with dense rewards.
Since the reward provides a useful and reliable signal to guide learning there is less need to generate good exploration data using previous skills. The advantage of skill-based methods is thus primarily in domains where defining a dense reward challenging but the required -ebbbehaviors are easier to specify.
D.4 COMBINATIONS WITH ADDITIONAL TRANSFER MECHANISMS
The separation of collect and infer processes means that SkilIS can be flexibly combined with other skill techniques, included the previously analysed baselines. In Figure 17 compare the per- formance of SkillS from the paper against versions of SkillS where the new skill trained using auxiliary regularisation objectives in the manipulation Lift and Stack tasks. We found no benefit of changing the underlying algorithm to infer the new skill. The main takeaway of this result that SkilIS can flexibly combined with different algorithms to run inference which could be beneficial in some settings not considered here.
25


Reviews:
[{"review_id": "uOIsS_mYKUm", "reviewer": "Reviewer_Qd5u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackle the problem of efficient transfer by leveraging both hierarchy and KL-regularisation. The authors claim that fine-tuning, hierarchical methods and imitation based approaches can all fail and therefore propose a method that combines elements from all of them. The authors perform experiments on robotic object-stacking tasks and gait learning and see consistent improvements.", "summary_of_the_review": "The authors tackle an important problem, but the proposed solution is not clearly motivated and presented. The experimental details bring major concerns in terms of reproducibility.\n\n======================================================\n\nAfter discussion with the authors, I have updated my score. The main factor is the acknowledgement that reproducibility is crucial and that the authors have attempted to add more seeds to their experiments. In general the paper still seems to add many elements to the algorithm, yet some of these choices are not well motivated. For example, the authors mention that CRR significantly helps for the off-policy problem, yet looking at Figure 14 there is no significant statistical difference. This same conclusion can also be drawn by Figure 15 concerning data augmentation. Overall, this also points towards the importance of having more seeds such that the significance of these kind of choices are clearly seen.", "strength_and_weaknesses": "# Strengths\n- The work is well situated with respect to previous work\n- The environments on which the methods are evaluated are challenging\n- The authors provide a set of ablations\n\n# Weaknesses\n- The experimental details are not clear. For example, how do you leverage previous skills across seeds? How are the skills transfered in the Lift task? Why the need for adding CRC to the mix?\n- The authors make multiple hand-wavy claims about how existing methods \"can\" fail. Any method can fail on a specific task, and saying that something \"can\" fail is not a great argument, as likely the new proposed approach \"can\" also fail.\n- The experiments are done with only 5 seeds, which to me is a deal breaker. This brings a lot of questions as to the reproducibility of the results. I also couldn't find any indication of what the shaded regions represent.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q8qfDs8vUKF", "reviewer": "Reviewer_XQxL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SkillS, a method for reusing skills in new reinforcement learning tasks that employs a learned high-level scheduler to execute skills, and a learned new skill, which is included in the set the scheduler selects. In this way, the benefit of temporally extended exploration is retained by executing previously learned skills for numerous steps at a time, catastrophic forgetting is avoided by learning an entirely new set of parameters separate from the frozen skills, and the potential suboptimal convergence of HRL is avoided by allowing the new skill to be unconstrained by previously learned behaviors. They suitably liken their approach to related works that perform “transfer via data” (i.e. the frozen skills are not used in final evaluation, but rather to generate useful training data for the new unconstrained skill). The scheduler is trained via a discretized version of MPO, and the new skill is learned using CRR. In a number of robotic manipulation and locomotion domains, the authors empirically demonstrate the performance gains achieved by SkillS relative to relevant approaches in the literature. The authors also include additional results, including a demonstration of SkillS’ ability to learn with different quantities/quality of skills, analysis of the selection trends of the scheduler, analysis of the utility of flexible temporal abstraction versus fixed, and a study of the importance of having separate collect/infer mechanisms. ", "summary_of_the_review": "* The paper is well-written and detailed, the method is well-motivated and grounded in relevant prior work, and the results are generally compelling and comprehensive. There are a few minor concerns/questions I have for the authors, but otherwise I am in favor of recommending this paper for acceptance, as I believe it will benefit the research community.", "strength_and_weaknesses": "Strengths: \n\n* The exposition is detailed and clear.\n\n* The problem is well defined and compelling and the resulting method is well-motivated. The discussion of prior work is comprehensive, and is presented in a way that effectively feeds into the motivation, and even makes the new method seem inevitable.\n\n* The design decisions are generally well-supported, are described clearly and concisely in the main body (and in even more detail in the Appendix). \n\n* The environments chosen for evaluation are challenging and diverse. \n\n* The experimental results are generally compelling and comprehensive. The expected ablations and comparisons are performed.\n\n* Useful avenues for future work are discussed in the conclusion.\n\nWeaknesses/questions: \n\n* None major.\n\n* How is the dependence on the frozen skills reduced over the course of training, if the scheduler is encouraged to maximize reward? Does this simply occur naturally? This is how I currently understand it. \n\n* Related to the previous question: did the authors try to regularize learning in any way to encourage the scheduler to choose the new skill more often? Or did this always naturally emerge? I would imagine issues with this not encouraging this; e.g. the scheduler rarely choosing the new skill in certain situations (relying on frozen skills in these), and then come evaluation, the new skill will perform poorly in these cases. Is there any intuition for why this does not (seem to) occur? \n\n* The results seem to demonstrate that variable temporal abstraction is necessary for best performance, which is an interesting finding. However, the authors appear to compare against only one constant for the fixed temporal abstraction results. How was 200 steps chosen? Can we see results with multiple other values to be convinced that the variable choice matters, and not that the fixed value is merely a poor choice? It could be helpful to compare the variable SkillS to the fixed temporal abstraction SkillS with the constant set to be the average skill length discovered by variable SkillS. This would more convincingly demonstrate the benefit of variable length skills. With my understanding of the current results, I’m not convinced.\n\n* Figure 7a: A little confused by this graph. Are these curves generated using different manually specified schedulers that were otherwise trained with the same learning procedure (i.e. “New Skill” is the performance of the scheduler only choosing the new skill after being trained the same way as the other too)? This seems to be the case, but either way, readers may benefit from a clearer explanation of this. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "epcfwohcouz", "reviewer": "Reviewer_E1qM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores an approach to using existing (learned) skills in a new task in the same domain. The approach keeps the existing skills intact (i.e., does not modify them); learns a policy over them (called the \"scheduler\") that maximises current task reward; and, at the same time, learns another policy (called the \"new skill\") that also maximises current task reward but by only using the primitive actions in the domain. Action selection in the domain is done by the scheduler. The approach is tested in two simulated robotics domains. ", "summary_of_the_review": "The paper presents an approach to knowledge transfer between tasks. While the approach is sensible, and will produce good results in some cases, the results in the paper are not particularly informative towards understanding how useful the approach will be generally and the conditions under which the approach is likely to be useful. ", "strength_and_weaknesses": "The primary strength of the paper is that it explores the important problem of knowledge transfer between tasks. In addition, the approach taken is sensible. However, I cannot claim that it is fundamentally innovative. The primary weakness is that the utility of the approach will depend on the particular task and the particular skills that are being used. While the paper gives examples of its successful use, it is not difficult to think about many cases in which the approach will not be successful. The paper does not present useful theory or analysis on the conditions under which the approach is likely to succeed. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0tk-LGYuXpQ", "reviewer": "Reviewer_C7sy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of transferring learned policies to a new target task sequentially. A new method SkillS is proposed for the problem which separates the data collection from solution inference. During data collection phase, a high-level schedule chooses the best learned skill to maximize the task reward obtained from the environment. The scheduler is trained with a proposed algorithm, Hierarchical Categorical MPO. The task solution is learned via an off-policy learning algorithm that optimists a new skill for the given task. The scheduler can learn to sequence temporally-extended skills and bias exploration towards useful regions of trajectory space, and the new skill can then learn an optimal task solution off-policy. Through experiments, the authors show the proposed method outperforms related method across all sequential tasks. ", "summary_of_the_review": "The idea in this paper makes sense to me. The proposed algorithm is shown to perform significantly better than other on certain tasks. ", "strength_and_weaknesses": "Strength: The idea of learning new policy with data generated by following sequencing policies makes sense to me. Especially for complex tasks that can be devide into subtasks, the proposed method will lead to an efficient learning strategy which gradually learn better about the entire task. Also the proposed method is shown to be significantly better than others on sparse-reward manipulation tasks. The sparsity of the reward can be handled by gradually learning subtasks. \n\nWeakness: There is no convergence analysis for the proposed HCMPO algorithm. Without any assumption on the continuity of the subtasks, it is not clear that the proposed algorithm is guaranteed  to converge to the optimal policy. It will be better if the authors can provide certain analysis on this. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject