SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration
Authors: Giulia Vezzani
Dhruva Tirumala
Markus Wulfmeier
Dushyant Rao
Abbas Abdolmaleki
Ben Moran
Tuomas Haarnoja
Jan Humplik
Roland Hafner
Michael Neunert
Claudio Fantacci
Tim Hertweck
Thomas Lampe
Fereshteh Sadeghi
Nicolas Heess
Martin Riedmiller
Venue: Accepted by TMLR ()

Abstract: The ability to effectively reuse prior knowledge is a key requirement when building general and flexible Reinforcement Learning (RL) agents. Skill reuse is one of the most common approaches, but current methods have considerable limitations. For example, fine-tuning an existing policy frequently fails, as the policy can degrade rapidly early in training. In a similar vein, distillation of expert behavior can lead to poor results when given sub-optimal experts. We compare several common approaches for skill transfer on multiple domains including changes in task and system dynamics. We identify how existing methods fail and introduce an alternative approach to mitigate these problems. Our approach learns to sequence temporally-extended skills for exploration but learns the final policy directly from the raw experience. This conceptual split enables rapid adaptation and thus efficient data collection but without constraining the final solution. It significantly outperforms many classical methods across a suite of evaluation tasks and we use a broad set of ablations to highlight the importance of different components of our method.

Published in Transactions on Machine Learning Research (11/2023)
SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extender Exploration
Giulia Vezzani* Google Mind
giulaaezaani@gogle..co
Dhruva Tirumala' Google Mend
dbrumatt@@goolee.com
Markus Wulfmeier*
mww//macierr@ggoogl..co
Google Mend
Dushyant Rao Google Mind
dushyontr&@google.com
Abbas Abdolmaleki Google Mind
aabiolmaaedhi@ggoogee.com
Ben Moran Google Derp
dermoran@google.com
Tuomas Haarnoja
tuomash aggogle.ggogg.ccom
Google Mind
Jan Humplik Google Mind
jhumplik@goglee.com
Roland Hafner
rhafner@@goole.cco
Google Mind
Michael Neunert Google Mind
neunertm@ggoolee.cm
Claudio Fantacci Google Mind
efantacci@@googee.com
Tim Hertweck Google Mind
thertweck&ggoogle.comm
Thomas Lampe Google Mind
thonaslampe@igoogle.comm
Fereshteh Sadeghi Google Deed
fsadeghi@@oogle.ccm com
Nicolas Heess Google Mind
heessggoogge.com
Martin Riedmiller Google DeepMind
riedmäller@googge.comm
Reviewed on OpenReview: https: openreview. net/ forum? id= JuCKVpR

---

Published in Transactions on Machine Learning Research (11/2023)
Abstract
The ability to effectively reuse prior knowledge is key requirement when building general and flexible Reinforcement Learning (RI.) agents. Skill reuse is one of the most common approaches, but current methods have considerable limitations. For example, fine-tuning an existing policy frequently fails, as the policy can degrade rapidly carly in training. In a similar vein, distillation of expert behavior can lead to poor results when given sul-optimal experts. We compare several common approaches for skill transfer on multiple domains including changes in task and system dynamics. We identify how existing methods fail and introduce alternative approach to mitigate these problems. Our approach learns to sequence temporally-extenddel skills for exploration but learns the final policy directly from the raw experience. This conceptual split enables rapid adaptation and thus efficient data collection but without constraining the final solution. It significantly outperforms many classical methods across a suite of evaluation tasks and we use a broad set of ablations to highlight the importance of different components of our method.
1 Introduction
The ability to effectively build on previous knowledge, and efficiently adapt to new tasks or conditions, remains a crucial problem in Reinforcement Learning (RL). It is particularly important in domains like robotics where data collection is expensive and where we often expend considerable human effort designing reward functions that allow for efficient learning.
Transferring previously learned behaviors as skills can decrease learning time through better exploration and credit assignment, and enables the use of easier-to-design (e.g. sparse) rewards. As result, skill transfer has developed into an area of active research, but existing methods remain limited in several ways. For instance, fine-tuning the parameters of a policy representing an existing skill is conceptually simple. However, allowing the paramelers to change freely can lead to a catastrophic degradation of the behavior early in learning, especially in settings with sparse rewards (Igl et al.) 2020). An alternative class of approaches focuses on transfer via training objectives such as rogularisation towards the previous skill. These approaches have been successful in various settings (Ross et al. 2011 Galashov et al., 2019: Tirunala et al., 2020 Rana al., 2021), but their performance is strongly dependent on hyperparameterss such as the strength of regularisation. If the regularisation is too weak, the skills may not transfer. If it is too strong, learning may not be able to deviate from the transferred skill. Finally, Hicrarchical Reinforcement Learning (HRL) allows the composition of existing skills via learned high-level controllers, sometimes at a coarser temporal abstraction (Sutton ctt al. 1999). Constraining the space of behaviors of the policy to that achievable with cxisting skills can dramat liaalyy improve exploration (Nachum et al.] 2019) but it can also lead to sub-optimal learning results if the skills or level of temporal abstraction are unsuitably chosen (Sutton al. 1999. Wulfmeier et al. [021]) b. As will show later (see Section 5) these approaches demonstrably fail to learn in many transfer settings.
Across all of these mechanisms for skill reuse we find shared set of desiderata. In particular, an efficient method for skill transfer should 1) reuse skills and utilise them for exploration at coarser temporal abstraction, 2) not be constrained to only these skills or the used temporal abstraction, 3) prevent early catastrophic forgetting of knowledge that could be useful later in learning.
With Skill Scheduler (SkilIS), we develop a method to satisfy these desiderata. We focus on the transfer of skills via their generated experience inspired by the Collect & Infer perspective (Riedmiller et al. 2022 2018 Our approach takes advantage of hierarchical architectures with pretrained skills to achieve effective exploration via fast composition, but allows the final solution to deviate from the prior skills. Specifically our approach learns two components: a high-level scheduler to quickly generate good exploration data and a new policy that is slowly distilled from this data. The high-level scheduler learns to sequence existing skills, choosing which skill to execute and for how long. It is trained to maximize task reward and can use the
1 Additional videos can be found under https://sites. http::///sseegggggle..e wwt.vwca./vviiiieee scheduller
2

---

Published in Transactions on Machine Learning Research (11/2023)
pre-defined skills 2 to rapidly collect task-relevant data. In parallel, distill new policy with the same objective from the experience gathered by the scheduler. While the pretrained skills in the scheduler are fixed to avoid degradation of prior behaviors, the new skills is unconstrained and fully adapt to the task at hand. This crucial addition improves over the common use of skills in hicrarchical approaches like the options framework (Sutton et al. 1999) and prevents the high-level controller from being constrained by the reloaded skills.
This can also be interpreted via the lens two learning mechanisms a fast and slow on the scheduler quickly learn to compose existing cemporally-abstrant skills and reach coarse solution, while the new policy can more slowly infer an optimal solution from the gathered data.
The key contributions of this work are the following:
We propose a method to fulfil the desiderata for using skills for knowledge transfer, and evaluate it on a range of embodied sparse-reward manipulation and locomotion lasks.
We compare our approach to transfer via vanilla fine-tuning, hicrarchical methods, and imitation- based methods like DAGGER and KT-regularisation. Our method consistently performs best across all tasks.
In additional ablations, we disentangle the importance of various components: temporal abstraction for exploration; distilling the final solution into the new skill; and robustness to the number and quality of skills used.
2
Related Work
The study of skills in RL is active rescarch topic that has been studiod for some time (Thrun & Schwartz,
1994
1999 Pickett & Barto 2002). A 'skill' in this context refers to any mapping from states to actions that could aid in the learning of new tasks. These could be pre-defined motor
Bowling & Veloso 1998
Bernstein
primitives (Schaal et al. 2005 Mülling et al. 2013 Lispeert et al. 2013 Paraschos et al. 2013 Lioutikov
ct al. 2015 Paraschos et al.
2018). temporally-coo --crelllaedd behaviors inferred from data (Nickum & Barto
2011
Ranchod et al. 2015] Krüger et al.
2016; Lioutikov et al. 2017 Shiarlis et al.] 2018 Kipf et al., 2019
Merel et al.)
2019
Shankar al. 2019
Tanneberg et al. 2021
or policies learnt in multi-task sctting It is important to note that our foeus in
(Hess et al.
2016
Hausman et al.
2018
Riedmiller et al.
2018
this work is not learning skills but instead how to to best leverage given set of skills for transfer.
Broadly speaking, we can categorise the landscape of transferring knowledge in RL via skills into set of classes (as illustrated in Fig. 1]: direct reuse of parameters such as via fine-tuning existing policies [Rusu et al. 2015 Parisotto et al. 2015 Schmitt et al. 2018), direct use in Hierarchical RL (HRL) where a high-level
controller is tasked combine primitive skills or options (Sutton et al. 1999 Heess et al. 2016 Bacon et al.,
2017 Wulfmeier et al. 2020: Daniel al. 2012 Peng al. 2209)) transfer via the training objective such as
regularisation towards expert behavior (Ross et al. 2011 Galashov et al. 2019 Tirumala et al.) 2020) and
transfer via the data generated by executing skills (Riedmiller et al. 2018 2007].
Campos et al. 2021 Torrey et al.
Fine-tuning often underperforms because neural network policies often do not easily move away from previously learned solutions (Ash & Adams 2020 [gl et al.) 2020 Nikishin et al., 2022) and hence may not easily adapt to new settings. As a result. some work has locused on using previous solutions to "kickstart' learning and improve on sub-optinal experts (Schmitt et al.) 2018 Jeong al. 2020 Abdolmaleki al.
2021). When given multiple skills, finc-tuning can be achieved by reloading of parameters via mixture
Daniel el al. 2012 Wullineier et al. 2020) or product (Peng et al. 2019 policy polentially with extra components to be learnt.
An alternative family of approaches uses the skill as a 'behavior prior' to generate auxiliary objectives to regularize learning (Liu el al. 2021 This family of approaches have widely and successfully been applied in
IIn practice, the scheduler is also given access the new skill in it's repertoire.
Tn relation to fast and slow decision making [Kahneman 2017 but focused on the optimization behind the learning process.
3

---

Published in Transactions on Machine Learning Research (11/2023)
TT skklll
Tnew
Train Agent
Knew
ST Wskill
S
Tnew
PR Regularise
7 skill
skalll
Collect Data
Fine-tuning
Hierarchy
Auxiliary Objectives
Exploration! Data
Figure 1: Transfer mechanisms for skills: Direct via. fino-tuning 2. hicrarchical policies, 3. auxiliary training objectives, 4. exploration and data generation. Grey and green respectively denote new policy components and transferred skills.
the offline or batch-RL setting to constrain learning (Jaques al. 2019 Wu et al., 2019 Siegel al. 2020
Wang et al.) 2020 Peng et al., 2020). When used for transfer learning though, the prior can often be too constraining and lead to sub-optimal solutions (Rana et al. 2021
Arguably the most widely considered use case for the reuse of skills falls under the umbrella of hierarchical RL. These approaches consider a policy that consists of two or more hierarchical 'levels' where a higher level controllers modulates number of lower level skills via a latent variable or goal. This latent space can be
continuous (Heess al.] 20016 Merel et al.
2019
Hausman
al. 2018: Haarnoja al.) 2018 Lynch et al.,
2019
Tirumala et al. 2019 Pertsch et al. 2020
Ajay et al.] 2021 Singh et al.) 2021 Bohez et al. 2022
discrete (Florensa et al. 2017 Wulfmeier et al. 2020 Seyde et al. 2022 combination of the two (Rao
ct al. 2022).
A major motivation of such hierarchical approaches is the ability for different levels to operate at different timescales which, it argued, could improve exploration and ease credit assignment. The most general application of this, is the Options framework (Sutton al. 1999 Bacon al. [017)] Wulfmeier al. 2021
Frans et al. 2017] that operates using temporally-extenndee options. This allows skills to operate at longer Ajay et al.
timescales that may be fixed (Li ct al. 2019a Zhang ct al.) 2020
2021 ) or with different durations Wulfmeier et al. 2021 Salter et al. 2022 Yang et al. 2020). Our
[acoo et al. 2017 Li et al., 2019b
approach to critic learning in this context is similar to that of Whitney et al. (211)), although we study a different problem setting. The advantage of modeling such temporal correlations has been attributed to benefits of more effective exploration (Nachum et al., 2019). However, constraining the policy in such a way can often be detrimental to final performance (Bowling Veloso 1998: Jong al. 2008 Another problem in the options framework is in the potential misalignment of termination and initial states when composing option policies (Lee et al.] 2021 Some work has addressed this issuc using transition policies (Lee et al. 2019 Byun & Perrault 2021 that slitch logether skills; role that can be fulfilled by the new skill in our approach.
Compared to other hierarchical methods such as the options framework, our method uses a final unconstrained, non-hierann non-hierarchical solution and learns temporal abstraction as a task-specific instead of skill-specific property. In this way, we benefit from temporally correlated exploration without compromising on final asymptotic performance by distilling the final solution. Our work is closely related transfer via data for off off-policy RL explored by (Fernández & Veloso 2006) for value functions in the case of discrete action and in the more general formulation for multi-task learning (Riedmiller et al.] 2018 Hafner et al. 2020 Kurenkov et al.,
2019) and initial work in transfer (Campos et al. 2021 Torrey et al. 2007). We extend these ideas to the compositional transfer of skills.
3 Background
We frame our discussion in terms of the reinforcement learning problem in which an agent obscrves the environment, takes an action, and receives a reward in response. This can be formalized as a Markov Decision Process (MDP) consisting of the state space S, the action space A, and the transition probability p(s 1ss, at)
For example, for walking task with target speed of 1.2m/s, a fixed option that can only walk m/s overly constraining.
4

---

Published in Transactions on Machine Learning Research (11/2023)
states
Scheduler
ICMMPO
Replay
f 0
f ==
=10 = kn
ko. Ts(=]
0)
[6,, kn] ko
[110 Auol #s(=]/10,
ag ~ Rig (0) ag ku
a1 ~ Wi (a|ss)
as ~M Ti, (a|s10)
Environment
c10 k10
CRR
ao Tig (a (8))
10 km
t t= 0
010 Tig
(1
Frozen skills
New Skill
actions
K10
(a)
(b)
Figure 2: a) The Skill Scheduler framework. Data is collected by the scheduler by sequencing selected skills among a sot of pre-trained (frozen) skills and the new skill. This data used to improve the scheduler using HCMPO (see Section [.3) and the skill via CRR. b) Scheduler action and counter sampling. At time t = 0 the scheduler samples an action 2p [ ko] i.e. the skill with index is applied to the environment for ko time steps. When (t (t=(a ko), the scheduler samples new action, the counter is set to the new length = kę and decreased every time step.
of reaching state 8t+1 from state 8, when executing action at- The agent's behavior given in terms of the policy x(a (st). In the following, the agent's policy x can either be parametrised by a single neural network or further be composed of N skills (als) such that r(ass) = s(al (T; Jo...N1,,
The agent aims to maximize the sum of discounted future rewards, denoted by:
(T) = (l
Σ x t=
(1)
where y is the discount factor, = r (st, az) is the reward and Po (so) is the initial state distribution. Given a policy we can define the state-action value function (s,,, a,) as the expected discounted return when taking an action ag in state 8, and then following the policy π as:
Q(s,, a.) = r(s, ar) + +1
(2)
4 Method
The method we propose this paper, named Skill Scheduler (Shill)) (Fig. 2a explicitly separates the data collection and task solution inference processcs in order better accommodato the requirements of cach. The data collection performed by the scheduler, high-level policy that chooses the best skills maximize the task reward obtained from the environment. In parallel, the task solution is learned via an off off-policy learning algorithm that optimisos new skill (als, Φ) for the given task.
By avoiding assumptions about the representation and source of the underlying skills, we can use these existing behaviors agnostic to how they were generated. The agent has no access to any prior information on the quality of the skills and their relevance for the new task. While most of our experiments Soction 5 focus on reusing skills that are useful for the task, we conduct a detailed ablation study that investigates the robustness with respect to the number and quality of skills used.
5

---

Published in Transactions on Machine Learning Research (11/2023)
4.1 Scheduler
The scheduler represented by a multivariate categorical policy 7 = [t;k|]s, 0) that outputs an action x consisting of:
the index E 10,.,, A] of the skill execute among the available skills $ = SU { Ø)} where S (=r(a)))) are the N previously trained and frozen skills and n (a) Φ) is the new skill, and
the number steps the skill executed for &s E K (the skill length), where K is a set of available skill lengths (positive integers) with cardinality [K] M.
Whenever new action [ē; k] is chosen by the scheduler, the skill Ty (alss) is executed in the enviromment for k timesteps. Note that the scheduler has access to both the fixed skills and the new skill important. deviation from the standard application of the Options framework.
The scheduler does not sample any new action until the current skill length & is expired (see Fig. 2b Choosing k > means that the scheduler selects a new skill at a coarser timescale than the environment timestep. This induces temporal correlations in the actions, which are important for successful skill-based exploration in sparsc-reward tasks (sce Section 5). The scheduler behaves similarly to policy over options Sutton al. 1999) but can additionally choose the new skill which is trained in parallel.
Choosing one skill for multiple timesteps affects the agent's 's future action distributions and therefore future states and rewards. We add an auxiliary counter variable to the observations to retain the Markov property. Whenever the scheduler chooses a new action 2 = k] the counter is set equal to the chosen skill duration c c= k. It is then decreased at every time step (C++1 = G 1) until the chosen skill duration has been reached
(c = 0) and the scheduler chooses a new action (see Fig. 2b) The counter is crucial for learning of the critic,
as detailed in Section
Data is collected in the form (st, ats 2t, Ca Pa, S1+1) and is used both to train the scheduler and the new skill. The scheduler is trained only using the high-level actions (Ze, Ct, Tr, 8041) with the accumulated task return over the episode as objective (Fq. 1
We introduce an off-policy algorithm, Hierarchical Categorical MPO (HCMPO) to train the schoduler. While any RL algorithm could be used as a starting point, HCMPO builds upon a variant of Maximum A-Postcriori Policy imisaaiin (MPO) (Abdolmaleki al. 2018) with discrete action space (Neunert et al. 2020 More details are provided in Section 4.3
4.2 New Skill
The new skill (als,, σ) interacts with the environment only when chosen by the scheduler. It is trained with the data only including the low-level action (s,, ag; Tt, 8t+1) collected by the underlying skill selected by the schoduler. Given the strongly off-policy nature of this data offline RL algorithms are most suitod for learning the new skill. For this work, we use Critic Regularized Regression (Wang el. al. 2020 [8]
Note that, although the scheduler and the new skill are trained with the same objective of maximizing the task reward (Eq. 1 we consider the new skill to provide the final solution for the task at hand, as it can reach higher performance. Whereas the scheduler acts in the environment predominantly via prelearned skills fixed to avoid degradation of learnt behaviors, the new skill is unconstruined and can thus fully adapt to the task at hand. Section 5.2.3 we show how the scheduler and the new skill can work in tandem to successfully tackle the two different problems of rapidly collecting task-relevant data (the scheduler) and slowly distilling the optimal task solution from the gathered experience (the new skill).
5he skill is initially rarcly exccuted as it commonly provides lower rewards than reloaded skills. We present an analysis of the advantages using offline methods like CRR against off-policy methods like MPO in Appendix
D.
6

---

Published in Transactions on Machine Learning Research (11/2023)
4.3 Training the Scheduler
In this section derive the policy evaluation and improvement update rules of our proposed algorithm, Hierarchical Categorical MPO (HCMPO), used for training the scheduler.
Policy Evaluation In order to derive the updates, we take into account the fact that la time steps have passed, i.e. the counter o has expired, the scheduler chooses a new action 2 = [ite ke] at time 4. It keeps the previous action 24 = 2t-1 [ - otherwise.
Qss cr, k2) ret
1Q r Q(s+t),
c++1
(3)
1,441 4, k k,) otherwise.
If the scheduler has chosen a action in the state 81+1, the critic in state 8t is updated with the standard MPO policy evaluation, ie. with the critic evaluated using actions scheduler policy in the next stale 81+1 (i.e. i Ts, k(s++1)). In all the other states instead, the critic is updated with the critic computed using the action 2++1 actually taken by the scheduler, which is 2t-1 = 2 = i;; k... More details on the derivations of Eq. 3 are available in Appendix A.1
[ē; k] sampled from the current
Policy Improvement The scheduler policy improved via the Tollowing mechanism but only on transitions where a new action is sampled by the scheduler:
n++ = arg max / (221)) log(Ts(z]s, 0)dds
(4)
with ((2(s) x (2(( exp (s, 1.
where n the index for the learning updates and ((]|s) is a non- nonpparamttric distribution computed for a batch of trajectories. Eq. a modification of the policy improvement step of the MPO algorithm (details described in Appendix B see also Abdolmaleki al. (2018)).
The pseudo code in Algorithm provides overview of the overall data collection and training procedure of SkilIS.
4.4
Data Augmentation
Eq. 3 and show that only a subset of the available data is actively used for learning the scheduler, as both the Q-function and policy of the scheduler are updated only those states where the scheduler chooses a new action z = [c; k). If the skills are executed for large number of steps (i.e. the skill lengths k E K are >> 1), the scheduler samples a new action less frequently and therefore the Q-function and policy are updated on a smaller number of states. Exccuting skills for long periods of time induces temporal corrolations that are important for skill-based exploration, but has the side effect of decreasing the data actively used for learning the scheduler. In order maximize sample efficiency without sacrificing exploration, for every trajectory T 0), we
with skill length & that can be written as multiple of the minimum skill length kmin (mod(k; generate new duplicated skill trajectory 7 pretending it was generated by schoduler choosing the same skill 4 multiple times with the smallest available skill length kmin- More details regarding the augmentation are described in Appendix A.2 and we include ablations in Appendix D.2
5 Experiments
Environments We empirically evaluate our method in robotic manipulation and locomotion domains. The goal of our evaluation is to study how different skill transfer mechanisms fare in new tasks which require skill reuse. In the manipulation setting (Fig. 3 (left)), we utilise a simulated Sawyer robot arm equipped with Robotiq gripper and set of three objects (green, yellow and blue) [Wulfmeier al. 2020).. We consider four tasks of increasing complexity: Lift the green object, Stack green on yellow, building a Pyramid of green yellow and blue, and Triple stack with green on yellow and yellow blue. Every harder task leverages
7

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 1 SkillS Training
1: Available skills: S = u { ø)}, where {a.(als)) are the N previously trained and frozen
skills and x (als, 6) is the new skill.
Scheduler policy: [2(2]s, 8).
Initialize scheduler policy (=( 0) and policy (n(als, with parameters 0- Φ0 Initialize replay buffer B.
5: for n = o.... do
6: for t 0, kn ,...,.., T - do
7: Sample scheduler action 2!" = kP) =s(als,
8: Set C = kg.
for =t.......f
do
10:
Sample skill action from selected skill ab mep S. Execute ap in environment, generating transitions (6% a 8 +1))))) Store {sp, az, zp, Cr, ++11,0) replay buffer B. Decrease counter: or Ge
11:
12:
13:
14:
end for
15:
end for
16:
Update new policy parameters tut1 with any offline RL algorithm by sampling {s] a r7, from the replay buffer B.
17:
Update scheduler parameters ga+tl via equation and equation by sampling {s] c, 8 from the replay buffer B.
18:
end for
Figure 3: Domains: The experimental domains used: (from left to right) manipulation Lift, Stack, Pyramid, Triple Stack and locomotion GoalScoring and GedUpAndWalk.
previous task solutions skills: this is natural transfer setting to test agent's ability to build on previous knowledge and solve increasingly complex tasks. For locomotion (Fig. 3 (right)), we consider two tasks with a simulated version of the OP3 humanoid robot (Robotis OP3): Wald and GoalScoring. The GelUllndWallk lask requires the robot compose two skills: one to get up off of the floor and one to walk. In the GoalScoring task, the robot gets sparse reward for scoring goal, with wall an obstacle. The GoalScoring task uses a single skill but is transferred to setting with different environment dynamics. This allows us to extend the study beyond skill composition to skill adaptability; both of which are important requirements when operating in the real world. All the considered transfer tasks use sparse reward, except the Get UpAnd task where a dense walking reward is given but only if the robot is standing. We consider an off-policy distributed learning setup with a single actor and experience replay. For each setting we plot the mean performance averaged across seeds with the shaded region representing one standard deviation. More details on the skills and tasks can be found Appendix
Analysis details Our method SkillS consists of two learning processes, one designed to quickly collect useful data (the scheduler) and one to distill the best solution for the task from the same data (the new skill). As our primary interest is in the task solution, unless otherwise specified, we focus the performance of
When the reward provides reliable learning sigual (c.g. of dense rewards) there less need generate good exploration data using previous skills. The advantage based methods thus primarily domains where defining dense, pasien-to-optimise reward challenging. For details see Appendix 13.3

---

Published in Transactions on Machine Learning Research (11/2023)
SkillS
RHPO
From scralch
Fite-ttriig
KL-reg. Mixlure
DAGGER
832
3:3
104
(a) Lift
(b) Stack
(c) Pyramid
(d) Triple Stacking
Figure 4: Performance of various skill transfer mechanisms on sparse-reward manipulation tasks. For each task, all approaches are given the previously trained useful skills (sce Appendix for details) to learn the new task. SkilIS is the only method to consistently learn across all tasks.
the new skill when executed in the environment in isolation for the entire episode. We compare SkillS against the following skill reuse methods: Fine-tuning, RHPO, KL-regularisation (KL-reg. to Mixture) and DAGGER. Hierarchical RL via RHPO (Wulfmeier et al. 2020) trains Gaussian mixture policies, and we adapt this method to reload skills as the components of the mixture. We can then either freeze the skills and inchude a new learned component (RHPO) or continue training the reloaded skills (fine-tuning.. KL-reg. to Mixture adapts the MPO algorithm (Abdolmaleki et al.] 2018 to include a regularisation objective to distill the reloaded skills; and DAGGER adapts Ross et al.J (2011) for imitation learning with previous skills. A full description of our baseline methods can be found in Appendix analysis:
B We answer the following questions with our
How does SkilIS compare to the bascline methods across the locomotion and manipulation tasks?
How robust is SkilIS to the quality and quantity of skills used?
What is the benefit of using a separate new skill to learn the final solution?
How important is the flexible temporal abstraction of met?
5.1
Analysis: Sparse Reward Tasks in Locomotion and Manipulation
Fig. Isshws the performance of skill-based approaches on manipulation domaing We observe that among all the considered methods, only SkilIS can learn effectively in these domains, even though other methods have access to the same set of skills. When given access to Lift task, all baselines still struggle to learn how to Stack. In contrast, SkilIS can even learn challenging tasks like TripleStack with enough data. Importantly this illustrats that skill reuse on its own is insufficient to guarantee effective transfer, and highlights the strong benefit of temporal abstraction. While other methods such as RHPO switch between skills at high frequency, the scheduler's action space encourages the exccution of skills over longer horizons, considerably reducing the search space.
Fig. 5 compares performance in the locomotion setting. In the GetUpAnd task, we find SkilIS oulperforms the other methods, confirming that temporal abstraction is fundamental when solving tasks that require a sequence of skills to be learned (getting up and then walking in this task). Executing the skills for longer horizons also generates higher reward carly during training. Particularly poor is the performance of regularising lowards mixture of these skills as it performs worse than learning from scratch. For the GoalScoring task which requires transferring a single skill to new dynamics, SkillS is competitive and shares the strongest results. As we will demonstrate in Section 5.2.4 the advantage of temporally correlated exploration conferred by SkilIS is less important in settings with only one skill.
5.2 Ablations
In this section we analyse the robustness of SkilIS and evaluate the importance of various components and design decisions. In particular, we analyse (1) the scheduler skill selection during training; (2) the robustness
8See Appendix C.2 for details about how the new skill and the scheduler can be separately evaluated. 9 Given the sparse nature the reward, most methods fail learn all and appear as flat lines the plot.
9

---

Published in Transactions on Machine Learning Research (11/2023)
SkilIS RHPO
From scratch
KL-reg. Mixture DARGER
Aine-runing
tingers
green
350
300
Accumlleedd
350
100
bay
-0000
0.0
0.5
3.0
2.0
25
300
3.5
(a) GelUpAndWalk
(b) GoalScoring
(c) Skill execution.
Figure 5: (a) (b) Performance on skill transfer methods on locomotion tasks. SkillS outperforms other
methods in GetUpAnd (a) and is no worse than other methods in the single skill GoalScoring task (b). (c) Skill average execution (as number of timesteps) during training.
Useful skills
Distrector skills
Usefu distractors
Useful + meny distrectors
Many useful distractors
600
400
ACO Acommulsse muisted Reward
500
-20
R30
150
300
200
100
200
100
100
50
0.0
0.2
0.8
0.0
1.5
2.0
up Sodess
104
Ep sodde
169
Epp sodes
lef
(a) Stacking
(b) Pyramid
(c) Triple Stacking
Figure 6: SkillS can learn even when the number and type of skills change although performance deteriorates on harder tasks.
to the number and quality of skills used; (3) the benefit and flexibility of having a separate infer mechanism and (4) the utility of flexible temporal abstraction. More ablations including a study of dense and sparse rewards, the benefit of oflline methods like CRR and the importance of data augmentation can be found in Appendix
5.2.1 Scheduler Skill Selection
In this section we show how the scheduler typically chooses the skills to interact with the environment. Fig. 5c displays the skill selection during the training by means of the average consecutive execution for each skill. We assume 7 skills are available, some of them needed for solving the task at hand ('reach green', 'close fingers'), while the others act as distractor (e.g. 'reach yellow' or 'open fingers'). At the beginning the scheduler quickly selects the most useful skills ('reach green', "close fingers') while discarding those less relovant to the task (observe the drop in executing 'reach yellow' or 'reach bluc'). While the scheduler does not choose the poorly performing new skill early in training, eventually (after roughly 2000 episodes) is selected more often until finally the most executed skill. An extended analysis on skill exccution across all manipulation domains is presented in Appendix D.5
5.2.2 Robustness to Skill Number and Quality
Fig. evaluates SkilIS on the stacking, pyramid and triple-stacking domains while changing the skill set S. For this analysis we vary the number and the types of skills by including variely of distractor skills not directly relevant for the task (e.g. 'reach blue* when 'red' needs to be stacked; more details in Appendix With only distractor skills, SkillS understandably fails to learn anything. For the casier tasks of
skills trained solve the same task (e.g. 'reach blue") from different. seeds and we refer these sets Fig. 0 mony distractors and many useful skills.
10 order augment the numher skills,
10

---

Published in Transactions on Machine Learning Research (11/2023)
Scheduler thout new New skill (trained endraly offine]
Skills
KL-reg RHPO (reloac)
Schoduler (inc). now skil)
Skills without temporal abstraction
602
620
Acuumuance Riswetss
5500
5500
600
400
600
200
200
200
100
1e3 0.0
0.2
0.4
0.6
0.4
0.0
0.2
0.4
Estoodos
Epuoddss
Latocoos
(a) Benefit of collect and infer.
(b) Single optimal skill.
(c) Multiple sub-optimal skills.
Figure 7: (a) SkilIS performs best when given access to separately inferred new skill, which is quite effective at lcarning offline. Results shown on the Lift task. (b)-(c) The main advantage of SkillS is likely due to temporally correlated exploration. When given access to a single optimal skill (b) all methods can learn effectively but with multiple sub-optimal skills (c), SkillS is the only method to learn (all other methods are flat lines). Results for (b) and (c) are on the Stacking task.
SkillS Skills (fxxd temporal clstraction 10)
SkillS Ifixed temporal abstraction 50) Skills tfaee temporal abstraction 100)
SkilIS (fixed te nporrl abstraction - 150) Skill (foxed roporrall abstraction 200)
400
250
Accumu ssed Revera
Accumuatedd Reward
500
300
200
400
Accumultted
150
200
200
200
100
100
50
0.0
0.2
0.4
0.6
0.8
1.0
12
14
0
0
1
Episodes
104
Fpicodes
104
Faisades
106
(a) Stacking
(b) Pyramid Stacking
(c) Triple stacking
Figure 8: The variable temporal abstraction of SkillS is especially crucial for learning harder tasks.
Stacking and Pyramid, SkillS is quite robust as long as there are some useful skills in the set S. On the challenging triple-stacking task, increasing the mumber of skills has detrimental effect performance. We hypothesize that the larger skill set increases the size of the search space to a point where the exploration in skill space does not easily find the task solution.
5.2.3 Benefit of Separate Collect and Infer Mechanisms
We conduct an experiment to study the importance of a separate inference (learning) mechanism in extension of the hierarchical agent. Fig. 7a compares SkillS 'Scheduler (inel. new skill)' which has access to the frozen skills and the new skill against 'Scheduler (without new skill)' which only has access to frozen skills. We also consider the performance of just the 'New skill' that trained off-policy former case and offline in the latter. As the figure shows, the separate off-policy learning process 'New skill' (blue) outperforms the scheduler (green) in performance even though the latter has access to this skill. The gap in performance is even greater when the scheduler does not have access to this skill at all (yellow). Interestingly, the new skill can learn a surprisingly effective policy from entirely offline data as shown by the 'New skill (trained entirely offline)' curve in red. An extension of this analysis on all manipulation domains is presented in Appendix D.5
5.2.4 Utility of Flexible Temporal Abstraction
Tenporally-correlteed exploration Figs. D and 7c illustrate the importance of temporally exploration that ariscs from our formulation in Section For this analysis we additionally run a version of our algorithm called 'SkilIS (without temporal abstraction) where a new skill must be chosen at each timestep (K = (1)). When given a single (near-)optimal skill " all methods including regularisation and
1he near-optimal skill for the Stacking task places the green object on the yellow one without moving the gripper away
11

---

Published in Transactions on Machine Learning Research (11/2023)
fine-tuning perform similarly (Fig. 7bj. However, when we use more than one sub-optimal skill, temporally correlated exploration is crucial for learning (Fig.
Variable temporal abstraction Fig. analyses the importance of the flexible temporal abstraction of SkillS by comparing our method against a version dubbed 'SkillS (fixed temporal abstraction - X)' where each skill, once chosen, held fixed for a given duration of X timesteps ([10, 50, 100, 150, 200] timesteps in a 600 step episode). We observe that the optimal fixed skill length varies from task to task (sce Fig. Sal vs 8b and, most importantly, the performance on harder tasks (especially triple-stacking) degrades when using this constraint. Thus, flexibly choosing the duration for each skill is quite important.
6 Discussion
Transferring the knowledge of reinforcement learning agents via skills policies has been a long-standing challenge. Here, we investigate different types of task and dynamics variations and introduce new method, SkilIS, that learns using data generated by sequencing skills. This approach demonstrates strong performance in challenging simulated robot manipulation and locomotion tasks. It further overcomes limitations of fine-tuning, imitation and purely hierarchical approaches. It achieves this by explicitly using two learning processes. The high-level scheduler can learn to sequence sunporall--xttedde skills and bias exploration towards useful regions of trajectory space, and the new skill can then learn optimal task solution off-policy.
There are number of interesting directions for future work. For instance, unlike in the options framework (Sutton et al.) 1999) which associates a termination condition with each option, in our work the duration of execution of a skill is independent of the pretrained skill. While this provides more flexibility during transfer, it also ignores prior knowledge of the domain, which may be preferable in some settings. Similarly, the high-level controller is currently trained to maximize task reward. However, it may be desirable to optimize the scheduler with respect to an objective that incentivizes exploration, e.g. reduce uncertainty with regards to the value system dynamics, improving its ability to collect data in harder exploration problems. Another avenue of exploration is to increase the robustness of our method to the number and quality of skills used. As Figure i shows, the robustness of SkillS to distractors reduces in more complex tasks. While any skill transfer method requires useful skills to be present in the pool, the initial exploration of the scheduler can be improved by leveraging expert trajectories or offline data. More general formulations of SkillS including vision or language-condttonee architectures could allow for sub-goal based exploration of the skill space. Finally, our approach can naturally be combined with transfer of forms of prior knowledge apart from skills, such as prior experience data, the value function (Galashov et al. 2020), or dynamics model (Byravan et al. 2021 and expect this to provide fertile area of future research.
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Ried- miller. Maximum posteriori policy optimisation. In International Conference Learning Representations, 2018.
Abbas Abdolmaleki, Sandy H Huang, Giulia Vezzani, Bobak Shahriari, Jost Tobias Springenberg, Shruti Mishra, Dhruva TB, Arunkuar Byravan, Konstantinos Bousmalis, Andras Gyorgy, et al. On multi- objective policy optimization as a tool for reinforcement learning. arXiv preprint arXiv:2106.0114 2021.
Amurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline primitive discov- ery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in Neural Information 3894, 2020.
Processing Systems,
PierreLLue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Arlificial Indelligence, volume 31, 2017.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Daniel S Bernstein. Reusing old policies to accelerate learning new mdps. Technical report, Citeseer, 1999.
Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel, Fereshteh Sadeghi, Leonard Hasenclever, Yuval Tassa, Emilio Parisotto, Jan Humplik, Tuomas Haarnoja, Roland Hafner, et al. Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors. arXiv preprint arXiv:2208.17138 2022.
Mike Bowling and Manucla Vcloso. Rousing learned policies between similar problems. in Proceedings of the AI A1-98 Workshop on New Trends in Robolics. Cileseer, 1998.
Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim, Mehdi Mirza, Alessandro Davide Ialongo, Yuval Tassa, Jost. Tobias Springenberg, Abbas Abdolmaleki, Nicolas Heess, Josh Merel, and Martin Riedmiller. Evaluating model-based planning and planner amortization for continuous control, 2021. URL https:
//arxiv. org....r//aboo. abs/ /21... 03363
Arunkumar Byravan, Jan Humplik, Leonard Hasenclever, Arthur Brussce, Francesco Nori, Tuomas Haarnoja, Ben Moran, Fereshteh Sadeghi, Bojan Vujatovic, and Nicolas Heess. NeRFSim: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields. In the International Conference on Robotics and Aulomation (under review), 2023.
Ju-Seung Byun and Andrew Perrault. Training transition policies via distribution matching for complex tasks. Xixiv preprint arXiv:21 10.0 043557 2021.
Victor Campos, Pablo Sprechmann, Steven Stenberg Hansen, André Barreto, Steven Kapturowski, Alex Vitvitskyi, Adrià Puigdomènech Badia, and Charles Blundell. Beyond fine-tuning: Transferring behavior in reinforcement learning. 2021.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Arlificial Intelligence and Statistics, pp. 773-281. PMLR, 2012.
Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in a reinforcement learning agent. In Proceedings of the fifth international joint conference on Autonomous agents and mulliagent systems, pp- 720-727,27 2006.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv: 704. 030122 2017.
Kevin Frans, Jonathan Ho, Xi Chen, P. Abbeel, and John Schulman. Meta learning shared hierarchies. ArXiv, abs/1710.09767,, 2017. URL
1369182
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenelever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M Czarnecki, Yee whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-regularized RL. In International Conference Learning Representations, 2019.
Alexandre Galashov, Jakub Sygnowski, Guillaume Desjardins, Jan Humplik, Leonard Hascnelever, Rae Jeong, Yee Whye Teh, and Nicolas Heess. Importance weighted policy learning and adaption. CoRR, Abss/2009.04877 2020. URL hattps://arse http:///xxxvvvvxaaaaaa/ org/ barg/abs//ee abs/aa09...../////// 04875
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 1851 1860, 2018.
Roland Hafner, Tim Herlweck, Philipp Klöppner, Michael Bloesch, Michael Neunert, Markus Wulfmeier, Saran Tunyasuvunakoo, Nicolas Heess, and Martin Riedmiller. Towards general and autonomous learning of core skills: A case study in locomotion. arXiv preprint ar Xiv:20008. 12228, 2020.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations (ICLR), 2018.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint or
2016.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Bochmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint arXi::006.0056: 2020.
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement. primitives: learning attractor models for motor behaviors. Neural computation, 25(2):328-3)) 2013.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arviv preprint ar Xrv: 00456, 2019.
Rac Jeong, Jost Tobias Springenberg, Jackic Kay, Danicl Zheng, Yuxiang Zhou, Alexandre Galashov, Nicolas Heess, and Francesco Nori. Learning dexterous manipulation from suboptimal experts. arv preprinl arXiv:2010.0853 2020.
Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. In AAMAS (1), pp. 199-306, Citeseer, 2008
Daniel Kahneman. Thinking, fast and slow. 2017.
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez- Gonzale,,,,,,z,, Edward Grelenstette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pp. 3418 3428. PMLR, 2019.
Björn Krüger, Anna Vögele, Tobias Willig, Angela Yao, Reinhard Klein, and Andreas Weber. Efficient: unsupervised temporal segmentation of motion data. IEEE Transactions on Multimedia, 19(4):797 812, 2016.
Andrey Kurenkov, Ajay Mandlekar, Roberto Martin Martin, Silvio Savarese, and Animesh Garg. Ac-leach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. In Conference on Robot Learning, 2019.
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S Hu, and Joseph J Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2019.
Youngwoon Loe, Joseph J Lim, Anima Anandkumar, and Yuke Zhu. Adversarial skill chaining for long-horizon robol manipulation via terminal state regularization. arXiv preprint (xXiv:2111.07999 2021.
Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. arXiv preprinl or Xin: Xii:1906.05866 05862, 2019a.
Siyuan Li, Rui Wang, Minxue Tang, and Chongjie Zhang. Hicrarchical reinforcement learning with advantage- based auxiliary rewards. Advances in Neural Information. Processing Systems, 32, 2019b.
Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, and Jan Peters. Probabilistic segmentation applied to an assembly task. In 2015 TEFE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 533 540. IEEE, 2015.
Rudolf Lioutikov, Gerhard Neumann, Guilherme Maeda, and Jan Peters. Learning movement primitive libraries through probabilistic segmentation. The International Journal of Robolics Research, 36(8):879 894, 2017.
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, S. M. Ali Eslami, Daniel Hennes, Wojeiech M. Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, Noah Y. Siegel, Leonard Hasenclever, Luke Marris, Saran Tunaasuvunkool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarnoja, Brendan D. Tracey, Karl Tuyls, Thore Graepel, and Nicolas Heess. From motor control to team play in simulated humanoid ball, 2021. URL https:: //tp///rrvvvv///////a//eaaaaiii /tt.:/b
14

---

Published in Transactions on Machine Learning Research (11/2023)
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot. Learning (CoRL., 2019.
Sebastian Madgwick et al. An efficient orientation filter for inertial and inetiaalmmageeic sensor arrays. Report x-10 and University of Bristol (UK), 25:113-118, 2010.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations (ICLR), 2019.
Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table lennis. The International Journal of Robolics Research, 32(3):263 279, 2013.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lec, and Sergey Levine. Why does hicrarchy (somelimes) work so well in reinforcement learning? arXic preprinl arXiv:1909. 10618, 2019.
Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Tobias Springenberg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller. Continuous-discred reinforcement learning for hybrid control in robotics. In Conference on Robot Learning, pp. 735-751. PMLR, 2020.
Scott Niekum and Andrew G Barto. Clustering via dirichlet process mixture models for portable skill discovery. In Workshops at the Twenty- Fifth AAAI Conference on Artificial Intelligence, 2011.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In TCML, 2022.
Alexandros Paraschos, Christian Daniel, Jan R. Peters, and Gerhard Neumann. Probabilistic movement primitives. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Alexandros Paraschos, Christian Daniel, Jan Peters, and Gerhard Neumann. Using probabilistic movement.
primitives in robotics. Autonomous Robots, 42(3):529 551, 2018.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. Xix preprint arXww:5511.0634,, 2015.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning composable hicrarchical control with multiplicative compositional policies. Advances in Newral Information Processing Systems, 32, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage weighted regression: Simple and scalable off- off-policy reinforcement learning, 2020. URL http://ppmeeriiee.neeefrrrrrrrta
Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint 2ft::0000.19444. 2020.
Mare Pickett and Andrew G Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In ICML, volume 19, pp. 506-513, 2002.
Krishan Rana, Vibhavari Dasagi, Jesse Haviland, Ben Talbot, Michael Milford, and Niko Sünderhauf. Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics. arv preprial wrXiv:2107.09822 2021.
Pravesh Ranchod, Benjamin Rosman, and George Konidaris. Nonparametric bayesian reward segmentation for skill discovery using inverse reinforcement learning. In 2015 IEFE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 471 477. IEEE, 2015.
Dushyant Rao, Fercshtch Sadeghi, Lconard Hasenelover, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, and Raia Hadsell. Learning transferable motor skills with hierarchical latent mixture policies. International Conference on Learning Representations, 2022.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Tearning by playing solving sparse reward tasks from scratch. In Proceedings of the 35th International Conference on Machine Learning, pp. 4344 4353, 2018.
Martin Riedmiller, Jost Tobias Springenbery, Roland Hafner, and Nicolas Heess. Collect & infer-a fresh look at data-efficient reinforcement learning. In Conference Robot Learning, pp. 1736-1766610 1744. PMLR, 2022.
Robotis OP3. URL
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627 635. JMLR Workshop and Conference Proceedings, 2011.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcchre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint crXfe:1511.06295, 2015.
Sasha Salter, Markus Wulfmeier, Dhruva Tirumala, Nicolas Manfred Otto Heess, Martin A. Riedmiller, Raia Hadsell, and Dushyant Rao. Mo2: Model-based offline options. 2022.
Stefan Schaal, Jan Peters, Jun Nakanishi, and Auke Ijspeert. Learning movement primitives. In Robotics research. the eleventh international symposiun, 2005.
Simon Schmitt, Jonathan J. Hudson, Augustin Zídek, Simon Osindero, Carl Docrsch, Wojeicch M. Czarnecki, Joel Z. Leibo, Heinrich Küttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning. ArXiu, 0bss/1803.0335 2018.
Tim Seyde, Wilko Schwarting, Igor Gilitschenski, Markus Wulfmeier, and Daniela Rus. Strength through diversity: Robust behavior learning via mixture policios. In Conference Robot Learning, pp. 1144-1155. PMLR, 2022.
Bobak Shahriari, Abbas Abdolmaleki, Arunkumar Byravan, Abe Fricsen, Siqi Liu, Jost Tobias Springenberg, Nicolas Heess, Matt Hoffman, and Martin Riedmiller. Revisiting gaussian mixture critics in off-policy reinforcement learning: a sample-based based approach, 2022. URL https://arxit: htttps///axxiv.oo////////////// wwgp/bs////.........///
10256
10256
Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2019.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning (ask decomposition via temporal aligument for control. In International Conference on Machine Learning, pp. 1654-4663. PMLR, 2018
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for oflline reinforcement learning. arXie preprint arXiv:2002.08300 2020.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-Driven behavioral priors for reinforcement learning. In International Conference Learning Represenlations, 2021.
Richard S Sutton, Doina. Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181 211, 1999.
Daniel Tanneberg, Kai Ploeger, Elmar Rucckert, and Jan Peters. Skid raw: Skill discovery from raw trajectories. IEEE Robolics and Aulomation Letlers, 6(3):4696 4703, 2021.
Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. Advances neural information processing systems, 7, 1994.
16

---

Published in Transactions on Machine Learning Research (11/2023)
D Tirumala, H Noh, A Galashov, L Hasenclever, and others. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXie preprint orXiv, 2019.
D Tirumala, A Galashov, H Noh, L Hasenclever, and others. Behavior priors for efficient reinforcement learning. arXiv preprint arx, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1109/ 10.1109/TROS.22 201.00..212.6380000
Lisa Torrey, Jude Shavlik, Trevor Walker, and Richard Maclin. Relational macros for transfer in reinforcement learning. In International Conference Inductive Logic Programming, pp. 254 268. Springer, 2007.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott F. Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
William Whitney, Rajat Agarwal, Kyunghyun Cho, and Abhinav Gupta. Dynamics-aware embeddings. ar preprinl ar Xiv: 09357, 2019.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv 2019.
preprint 1rXi::1911.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin A Riedmiller. Compositional transfer in hierarchical reinforcement learning. In Robotics: Science and Systems, 2020.
Markus Wulfmeier, Dushyant, Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim Hertweck, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Heess, et al. Data-efficient hindsight off off-policy option learning. In International Conference on Machine Learning, pp. 11340-11350. PMLR, 2021.
Tianpei Yang, Jianye Hao, Zhaopeng Meng, Zongzhang Zhang, Weixun Wang, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhaodong Wang, and Jiajie Peng. Efficient deep reinforcement learning through policy transfer. ArXiu, abs/2002.08007, 2020. URL
http://pp...mmanicccchhhlarrr www.laa...r//oooo Org/CorpusID:21111 2111171117
Jesse Zhang, Haonan Yu, and Wei Xu. Hicrarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations, 2020.
A Appendix
Appendix
A SkillS Details
A.1 HCMPO Policy evaluation
In order to derive the PE updates we write down the probability that the scheduler chooses an action and the counter
This takes into account the fact that, if kę time steps have passed, ie. the counter & has expired, the scheduler chooses a new action 2, = [ix; ki] at time t. keeps the previous action 2, 2t-1 [fe-1; k otherwise.
TT2,, celss1 2-,
TT(2 = | 0)0 if ff-1= =1
m(i, Ca c
(5)
5(ir - (k- - (ct-1 1)) othriwsse.
We now use Eq. to derive the PE update rules reported in Eq.
3
17

---

Published in Transactions on Machine Learning Research (11/2023)
T
k* 60
2 *
2 *
2
T
k
kmin 10
Figure 9: Example of data augmentation. Let's assume we have a trajectory where the skill with index it is executed for k" = 60 time steps. The duplicated trajectory F is generated by pretending the scheduler chose the same skill with index i" every kmin time steps, oqual to 10 in our example. This generates a now trajectory F compatible with the low-level actions a and state 8 actually collected during the interaction with the environment.
A.2 Data Augmentation
We define a skill trojectory T the sequence of k* time steps where the same skill with index i" is executed
== ((1, 24 ce,, re)
(6)
and kimin the smallest skill length available to the scheduler (kmin S k for k E K and Kmin K
For every trajectory т with skill length k* that can be written multiple of the minimum skill length kmin (ke kmmnn 0), we generate a new duplicated skill trajectory F pretending it was generated by a scheduler choosing the same skill i multiple times with the smallest available skill length kmin (see Fig. The duplicated skill trajectory
7 ((8t, Et [i*, "",, (,),,a, Te r2) re)) !Ebotkk-
is such that:
(st, 24 =i" 1,""
(st, & [i*, kmmn), Ct - [(x 1) kimin F')) if mod(ke, ke, kmmn) = 0, Cs > kmén
(st, E (i*, kmiin Ct, rx) if mod(ki, himin) = Ct I kmin
(8)
(St, 24 k"","", ke), re) otherwise.
This generates a new trajectory that is compatible with the data actually collected, but with more samples that are actively used for improving the scheduler and the policy, as it pretends the scheduler made a decision in more states. Both 7 and F are added to the replay and used for training the scheduler. This perspective is highly related to efficiently learning high-level controllers over sets of options in the context of intra-option learning (using data throughout option exccution to train the inter option policy) (Sutton al. 1999
Wulfmeier al. 2021
A.3 Bias on initial skill length
Our SkillS formulation enables the possibility of specifying initial bias in the scheduler choices. It is possible to encourage the scheduler to initially choose one or more skills and/or skill lengths. all our experiments we bias the schoduler to initially choose the largest skill length. This way all the skills are executed for a considerably large number of timesteps at the beginning training, helping the scheduler understand their utility for the task at hand.
18

---

Published in Transactions on Machine Learning Research (11/2023)
B
Baseline details
In this section we describe the methods used as baselines in the main text. For all of the baselines described below, we use a weighted loss that trades off the baseline with CRR using a parameter a. other words when a I, we use pure CRR whereas a runs the pure algorithm.
B.1 MPO
When learning from scratch and for our baselines we largely adapt the state-of the--rrtttt-rtttt off-policy learning algorithm Maximum a-posteriori Policy Optimisation (MPO) (Abdolmalcki et al.j 2018). MPQ optimises the RL objective in Eq. using an Expectation Maximisation (EM) style approach. The algorithm operates in three steps: a policy evaluation step learn the state-action value function Q a); E-step to create improved non-parametric policy using data sampled from a replay buffer; and an M-step that fits parametric policy x(ls) to the non-parametric estimate.
For the policy evaluation step we use a Mixture of Gaussian (MoG) critic as described by Shahriari et al. 2022
MPO then optimizes the following objective in the E-step:
max | pus)) /. q(als)Q(ss,a) da 9
t.
ds <
(9)
where g(als) improved non-parametric policy that is optimized for states pls) drawn from the replay buffer. The solution for is shown to be:
d(als) x (als, 0) exp( Qo(s,a)
(10)
Finally in the M-step, an improved parametric policy x (als) is obtained via supervised learning:
M
N
n arg max 36
Σ
Σ
Tha log mo(a.)))).
s. ( (a) (a|s) ((b((s)=
Importantly, the M-Step also incorporates a KL-constraint to the previous policy (als) to create a trust- region like update that prevents overfitting to the sampled batch of data. In our experiments, we use an extension of MPO, dubbed Regularized Hierarchical Policy Optimisation (RHPO), that learns a Mixture of Gaussian policy (Wulfmeier et al. 2020 instead of single isotropic Gaussian.
B.2 Fine- line-tuning and RHPO
For our comparison of skill reuse, we reload the mixture components of an RHPO agent using the parameters of the previously trained skills. In addition we consider two scenarios: a) continuing to fine-tune the mixture components that were reloaded or b) introducing new mixture component (similar to the new policy of SkilIS) and freezing the other mixture components that were reloaded using skills. Freezing the skills is often advantagoous to prevent premature over-writing of the learnt behavior an important feature which can add robustness ness early in training when the value function is to be learnt. We illustrate this effect in Figure 10 in the 'Stacking' setting where we transfer only a single skill. As shown, when the previous skill is fixed the retained knowledge can be used to bootstrap learning whercas direct fino-tuning can cause a catastrophic forgetting effect where learning performance is significantly worse
B.3 Re-glauuaiiittion
KL-regularisation typically involves the addition of an extra term to the RL objective function that minimises the Kullback-Leeillr (KT.) divergence between the policy and some prior distribution (irumala et al. 2020
19

---

Published in Transactions on Machine Learning Research (11/2023)
RHPO with frozen skills
RHPO with fine-tuning
600
Accumulates Reward
500
400
300
200
100
0.0
0.4
0.8
1.0
Episodes
164
Figure 10: Impact of freezing skills: Single skill transfer stacking.
Here, we define the prior distribution to be uniform mixture over the previously trained skills, given by:
To
wi (a(s)
(11)
1 N
Vi
where n. (a(s) are N previously trained skills.
As such, the KL objective can be incorporated into any RL algorithm. In this work we adapt MPO to include regularisation to a prior. As presented above, MPO includes KL-constraints in both the E and M steps, either of which can be used to further regularise to the prior. In this work, we adapt the E-step to regularise against mixture distribution. In this sotting, the non-parametric distribution q for the E-step can be obtained as:
g(als) (u(ass 0) exp( Qo(s.a) 17
(12)
One interpretation of this objective that the prior To serves as a proposal distribution from which actions are sampled and then re- re-weighted according to the exponentiated value function. The optimal temperature 17 here is obtained by minimising:
g(n) n/c
+ n / u(s) log / To(als, 0) exp( Qo(s,a) a)
(13)
where is Syper-parameter that controls the tightness of the KL-bound. A smaller more tightly constrains the policy to be close to the prior distribution whereas looser bound allows more deviation from the prior and reliance on the exponentiated Q function for learning.
B.4 DAGGER
The final baseline consider is adaptation of the method DAGGER (Dataset Aggregation) introduced by Ross et al. (2011). DAGGER is an imitation learning approach wherein a student policy is trained to imitate a teacher. DAGGER learns to imitate the teacher on the data distribution collected under the student. In our formulation, the teacher policy set to be a uniform mixture over skills, mo as defined in Eq. [ We train the student minimize the cross- coss-entropy loss to this teacher distribution for a state distribution generated under the student. The loss for DAGGER thus given by:
LDACGER arg min (a(ss)) log (a|s)
(14)
20

---

Published in Transactions on Machine Learning Research (11/2023)
Entry
Dimension
Unit
End-ellector Cartesian linear velocity
3
m/s
Gripper rotational velocity
rad/s
Gripper finger velocity
1 rad/s
Table 1: Robot action space in manipulation tasks.
In addition, we consider an imitation + RL setting where the DAGGER loss is augmented with the standard RL objective using a weighting term ADAGGER
L QDAGGER LDAGGER + (1 aDAGGeR) x LMrO-
(15)
In our experiments we conduct a hyper-parameter sweep over the parameter aDAGGER and report the best results.
C Experiment Details
C.1 Tasks and rewards
C.1.1 Manipulation
In all manipulation tasks, we use a simulated version of the Sawyer robot arm developed by Rethink Robotics, equipped with a Robotiq 2F85 gripper. In front of the robot, there is a basket with a base size 20x20 cm and lateral slopes. The robot can interact with three cubes with side of cm, coloured respectively in yellow, green and blue. The robot is controlled in Cartesian velocity space with a maximum speed of 0.05 m/s. The control mode has four control inputs: three Cartesian linear velocities well rotational velocity of the wrist around the vertical axis. Together with the gripper opening angle, the total size of the action space is five (Table D Observations used for our experiments are shown in Table 2 For simulating the real robot setup we use the physics simulator MuJoCo (Todorov el al. 2012). The simulation is run with a physics
integration time step of 0.5 milliseconds, with control interval of 50 milliseconds (20 Hz) for the agent.
C.1.2 Locomotion
We use a Robotis OP3 (Robotis OP3 humanoid robot for all our locomotion experiments. This hobby grade platform is 510 mm tall and weighs 3.5 kg. The actuators operate in position control mode. The agent chooses the desired joint angles every 25 milliseconds (40 Hz) based on on-board sensors only. These observations include the joint angles, and the angular velocity and gravity direction of the torso. Gravity is estimated from IMU readings with Madgwick (Madgwick et al.] 2010) filter that runs at higher rate. The robot has also a web camera attached to the torso, but we ignore it in our experiments. To account for the latency in the control loop, we stack the last five observations and actions together before feeding them to the agent. Further, to encourage smoother exploration, we apply an exponential filter with a strength of 0.8 the actions before passing them the actuator. We also assume we have access the robot's global position and velocity via external motion capture system. This privileged information is, however, only used to evaluate the reward function, and is not visible to the agent. Although train policies in simulation, the model has been carefully tuned to match the physical robot, allowing us to zero-shot transfer the policies directly from simulation to robot BBravvnn et al. 2023
C.1.3 Skills
The skills used for the manipulation tasks are shown in Table 3 only the skills under the column 'Useful skills'. The other sets of skills are used for the ablation in Fig. ( In particular, when Fig. we mention 'Many Useful skills' or 'Many Distractor skills' mcans that we use skills trained for the same task, e.g. 'reach green', but from different seeds (e.g. 'Many useful skills and distractor skills' for Pyramid is total of 31 skills.).
All the plots in the paper are shown using
21

---

m. Published in Transactions on Machine Learning Research (11/2023)
Entry
Dimension
Unit
Joint Augle
7
rad rad/s
Joint Velocity
7
Pinch Pose
7
m, au
Finger angle
1
rad
Finger velocity
1
rad/s /s n/a
Grasp
1
Yellow cube Pose
7
m, au m, an m; all
Yellow cube pose wrt Pinch
7
Green cube Pose
7
Green cube pose wrt Pinch
7
m, au
Blue cube Pose
7
m; all
Blue cube pose wrt Pinch
7
Table 2: Obsorvations used in simulation. In the table stands for arbitrary units used for the quaternion representalion.
Task
Useful skills
No.
Distractor skills
No.
Lift
Reach green,
2
Reach yellow, Reach blue,
(green)
Lift arm with closed fingers
Open fingers, Lift arm with open fingers Distractor skills for Lift green, Lift yellow, Lift blue
Stack [green on yellow)
Useful skills for Lift green, Lift green,
4
Hover green on yellow
Pyramid (green on top)
Useful skills for Stack,
Lift arm with open fingers, 9
Stack green on yellow,
Open fingers, Stack yellow on blue
Tift yellow, Lift blue
Reach yellow, Reach blue
Triple stacking (green on yellow on blue)
Useful skills for Slack,
9
Lift arm with open fingers, Open fingers, Reach blue, Lift blue
Stack green yellow,
Stack yellow blue,
Lift yellow, Reach yellow, Hover yellow on blue,
Table 3: Skills used for each manipulation task. Every column with title 'No.' counts the number of skills in the left-hand side column, c.g. 'Lift green' has 2 Uscful skills.
For the locomotion setting, consider two tasks: 'GoalScoring' and 'GetUpAndWalk' that are defined using the rewards described in Appendix C.1.5 In 'GoalScoring', consider transfer domain where the walker is randomly perturbed by applying an impulse sampled uniformly within the range (-0.05, 0.05). The 'GnlUAAndWalk' task involves two skills 'GetUp' and 'Walk' The 'GetUp' skill is trained by initializing the walker on the floor with a reward for standing upright and the Waakk' task uses the reward definition Appendix C.1.5 with no upright reward and episode termination when the walker hits the ground.
C.1.4 Rewards Manipulation
The skills of Table 3 are obtained either by training them from scratch with staged rewards (in particular, the basic skills such as 'Open fingers') or they are the result of a SkillS experiment, i.c. they are represented by the New Skill and have been trained with sparse rewards. The reward functions used for this aim are the following.
22

---

Published in Transactions on Machine Learning Research (11/2023)
Reach object- dense
iff
Portpper Pobjact tolpos
S1
Rreach
(16)
tanh"
Poripper Pobject
|| E)
otherwise,
tolpon
where Paripper E R is the position of the gripper, Pobject E R the position of the object, tol [0.055, 0.055, m.02jm the tolerance in position and a scaling factor.
Open fingers dense
iff Pfinger - Pdesered tolpos
Ropen
(17)
tanh? 2 Pfinger Pdesired
())))
otherwise,
tolpos
where Pfinger E R. is the angle of aperture of the gripper, Pdesired E R. is the angle corresponding to the open position, pol = le the tolerance in position and e a scaling factor.
Lift arm with open fingers dense
iff Zarm 2 Zdesired otherwise,
Ruft
(18)
Farm Smain Adessred -aasrred-Aman
where Zarm E R is the z-coordinate of the position of the gripper, 8min 0.08 R. is the minimum height the arm should lift to receive non-zero reward and 2desired 0.18 E R is the desired z- coordinate the position of the gripper should reach to get maximum reward. The reward to have both the arm moving up and the fingers open is given by:
Ruft,open Ruse Ropen
(19)
Lift arm with closed fingers - dense
Ryrasp = 0.5( + Rgrasp.auz):
(20)
where Relosed is computed with the formula of Eq. equation 17 but using as Piteeired E R. the angle corresponding closed fingers this time;
1 if grasp detected by the grasp sensor {
Rgrasp.owx
(21)
otherwise.
Then the final reward is:
Rlift,closed Risft Rggasp-
(22)
Lift object- sparse
Rust (1 iff Robject 2 Edessred otherwise,
(23)
where Zobject E is the z-coordinate of the position of the object lift and Zdesived 0.18 the desired z-coordinate the object should reach to get maximum reward.
Hover object on another one dense
Prop Photton -f focf faet
1
Rhover
tol
(24)
tanh2
Phop -annn-Pap Phottom fset tolpax
()
otherwise
where Prep R the position of the top object, Phottom E R the position of the bottom object, fftt 00,0, objectheightt 0.04], tolpos [0.055, 0.055. 0.02 the tolerance in position and scaling factor.
23

---

Published in Transactions on Machine Learning Research (11/2023)
Stack top object on bottom object sparse
iff (Ptop Phottom) S tolzz & Id (Prop: Pbottom) ddesired I tol & Rgrasp.aur =0
Ratack
(25)
otherwise,
where (::)) R is the norm of the distance of 2D vectors (on and y); dz(:,-) R is the (signed)
distance along the z-coordinate, Ptop R and pbottom R are respectively the position of the top
and bottom object, tolz.9 = 0.03 and tol 0.01 are the tolerance used to check if the stacking is achieved, ditesired = 0.04 is obtained as the average of the side length of the top and boltom cubes. The final reward we use for stacking is:
Reaaee Ratack (1 Preeach,.parse(top))
(26)
where Breach,spare (top) is the sparse version of Ryeach in Eq. 16 for the top object.
Pyramid sparse
iff de, (Poy(Poot as Poottom, b) S tol bottom & dz (Prop- Phottom) I tol &
Rpyromid
|d Pbottom) dieaired S tol &
(27)
Rgrasp.auz = 0
0
otherwise,
where (:),(::)) is the norm of the distance of 2D vectors (on x and y), dz(:,)) is the (signed) distance along the z-coordinate, Ptops Pbottom,, Phottom, R are respectively the position of the top object and the 2 bottom objects, while Pbottom = Pbottom, t Phottomm& E R is the average position of the two bottom objects, tol 0.08, tolz.y 0.03 and tol, = 0.01 are the tolerance usod check if the pyramid configuration achieved, ddesired 0.04 is obtained as the average of the side length of the top and bottom cubes. The final reward we for our tasks is:
2
Regramiid,eaae = Rpyremid' (1 Rreach,sporee (top)).
(28)
Triple stacking - sparse The triple stacking reward is given by:
Reviple, stacck leave Retack (top, middle). R.stack (middle, bottom) (1- Reeach..parse(top)) (top))
(29)
C.1.5 Rewards Locomotion
In the GelUpAndWalk task, we condition the robot's behavior on goal observation: 2-d unit target orientation vector, and target speed in m/s. In these experiments, the speed was either 0. 0.7m/s or Om/s. The targets are randomly sampled during training. Each target kept constant for an exponential amount. of time, with an average seconds between resampling.
The reward per time step is weighted sum Rwalk with terms as described below.
While this reward is dense in the observation space, 2/3 of episodes initialized with the robot lying prone the ground (either front or back). The effective sparsity results from the need to first complete the challenging get-up task, before the agent can start to earn positive reward from walking.
GetUpAndWalk
Bwalk = Rorient + Ruelocity + Rupright + Raction + Rpose + Rground + 1,
(30)
where Rorient the dot product between the target orientation vector and the robot's current heading; Ruelocity is the norm of the difference between the target velocity, and the observed planar
24

---

Published in Transactions on Machine Learning Research (11/2023)
states
states
Skill Scheduler
Environment
New Skill
Environment
Frozen skills
New Skill
actions
actions
(a)
(b)
Figure 12: During evaluation, the new skill is executed in the enviromment for the entire episode (a). This differs from the way the data is collected at training time, where the new skill can interact with the enviromment only when selected by the scheduler (b).
velocity of the foet; Rapright is 1.0 when the robot's orientation is close vertical, decaying to zero outside a margin of 12.5%; Raction penalizes the squared angular velocity of the output action controls averaged over all 20 joints, Ejoint (wrfoine (wofoint- -
-wfoine) Rpose regularizes the observed joint angles
toward a reference 'standing" pose Ejoint (gioint ofoine) and Rground - whenever the robot brings body parts other than the feet within 4em of the ground.
In GoalScoring, the robot is placed in a 4m 4m walled arena with soccer ball. must remain own half of the arena. scores goals when the ball enters goal area, 0.5m x Im, positioned against the centre of the back wall in the other half. An obstacle (a short stretch of wall) is placed randomly between the robot and the goal.
GoalScoring
Rgoalscoring Recore + Rupright + Rmarvelocity
(31)
where: Recore is 1000 the single timestep where the ball enters the goal region (and then becomes unavailable until the ball has bounced back to the robot's own half); Rmaasetocity is the norm of the planar velocity of the robot's feet in the robot's forward direction; Rupright is the same as for GetUpAndWalk. Additionally, episodes are terminated with zero reward the robot leaves its own half, or body parts other than the feet come within 4em of the ground.
C.2 Details on how we evaluate performance
Fig.
12 show how the performance of the scheduler and the new skill can be evaluated separately.
C.3 Training and networks
For all our experiments use Multilayer Perceptron (MLP) network torsos with network head output that is twice the size of the action dimension for the task. The network output is split into the mean and log standard deviation that is used to parameterize the output of an isotropic Gaussian distribution. We use a single actor and learner, a replay buffer for off policy learning and a controller that ensures a fixed ratio between actor steps and learner updates across experiments. For cach method and setting results are averaged across random seeds. In what follows we provide a detailed description of hyper-parameters used for the various experiments.
25

---

Published in Transactions on Machine Learning Research (11/2023)
C.3.1 Manipulation
The following parameters are fixed across all methods compared the Manipulation setting (SkilIS and baselines):
Learner parameters
Batch size: 256
Trajectory length: 10
Learning rate: 3e-4
Min replay size to sample: 200
Samples per insert: 50
Replay size: 1e6
Target Actor update period: 25
Target Critic update period: 100
E-step KL constraint <: 0.1
M-step KT. constraints: 5e-3 (mean) and 1e-5 (covariance)
Network parameters
Torso for Gaussian policies: (128, 256, 128)
Component network size for Mixture policies: (256, 128)
Categorical network size for MoG policies: (128.)
Scheduler specific parameters
Schoduler batch size: 128
Schoduler learning rate: 1c-4
Min replay size to sample for scheduler: 400
Samples per insert: 100
Scheduler larget actor update period: 100
Scheduler target critic update period: 25
Available skill lengths: K = {n 10
Initial skill lengths biases: 0.95 for k 100, 0.005 for the other skill lengths
Hyper-parameter sweeps
(MPO v/s CRR loss): (0, 0.5, 1.0)
KL-reg: c (0.1, 1.0, 100.)
DAGGER: ODAGGER (0, 0.5, 1.0)
RHPO: Fine-tune freeze parameters
26

---

Published in Transactions on Machine Learning Research (11/2023)
C.3.2 Locomotion
Learner parameters
Batch size: 256
Trajectory length: 48
Learning rate: le-4
Min replay size sample: 128
Steps per update:
Replay size: 1e6
Target Actor update period: 25
Target Critic update period: 100
E-step KL constraint e: 0.1
M-step KL constraints: 5e-3 (mean) and 1e-5 (covariance)
Network parameters
Torso for Gaussian policies: (128, 256, 128)
Component network size for Mixture policies: (256, 128)
Categorical network size for MoG policies: (128.)
Scheduler specific parameters
Scheduler batch size: 128
Scheduler learning rate: le-4
Min replay size to sample for scheduler: 64
Steps per update:
Available skill lengths: K= {n 100
Initial skill lengths biases: 0.95 for k = 100, 0.005 for the other skill lengths
Hyper-parameter sweeps
(MPO v/s CRR loss) - (0, 0.5, 1.0)
KI-reg: <- (0.1, 1.0, 100.)
DAGGER: QDAGGER (0, 0.5, 1.0)
RHPO: Fine-tune freeze parameters
27

---

Published in Transactions on Machine Learning Research (11/2023)
[w/ CRR)
SkillS (w/ MPO)
Finetuning (w/ MPO)
Finetuning (w/ CRR)
600
1000
Accumulated Reward
500
Accumulates Reward
800
400
600
300
400
200
200
100
0
0.0
0.5
1.0
1.5
2.5
3.5
0.0
0.2
0.4
0.6
0.8 Episodes
1.0
1.2
1.4
Episodes
1e4
1e4
Figure 13: Comparison between regularized fine- luning (CRR) and vanilla fine-tuning (MPO) on the GoalScoring task. Offline learning methods like CRR typically regularize towards the skill (via data in the replay) which can improve learning stability when fine-tuning.
Figure 14: Comparison when using CRR v/s MPO to infer the new skill on the Stacking task.
Additional Experiments
D.1 Impact of trade-ofi off between MPO and CRR loss
Fine-tuning with offline learning. As discussed in Section 5 of the main text, fine-tuning often leads to sub-optimal performance since useful information can be lost carly in training. In Fig. 13 observe that fine-tuning with offiin-RL loss like CRR improves stability and learning performance. Many offline methods, including CRR, typically regularise towards the previous skill either explicitly through the loss implicitly via the data generated by the skill (see Tirumala et al. (2020) for a more detailed discussion on this). We hypothesize that such regularisation could help mitigate forgetting and hence improve learning. While this insight requires a more detailed analysis that outside the scope of this work, we however have some evidence that when fine-tuning with a single skill potentially simple algorithmic modification (from off- off-policy to offline RL) might already make learning more robust.
CRR for new skills. In Fig. 14 we compare what happens when the new skill is inferred with an offline learning algorithm (CRR) against an off-policy algorithm (MPO) on the stacking domain from Section 5 We observe that lcarning improved when using an offline algorithm like CRR. For this reason, all of the experiments presented the main text use CRR as the underlying skill inference mechanism although the method itself is somewhat robust and agnostic this choice. For a fair comparison, we also include a sweep using CRR the underlying agent across when selecting the best performing bascline. However, since CRR implicitly regularises towards the skill, there is typically no effect when compared to using MPO when regularising towards a mixture.
D.2 Impact of Data Augmentation
Fig. 15 shows how data augmentation improves the performance, especially for more challenging tasks.
D.3 Performance with Dense Rewards
Fig. shows the performance with all skill transfer methods on the lift and stacking tasks when the agent has access dense rewards. We observe that, in contrast the sparse reward results of Fig, 4 all methods learn effectively when provided with dense rewards.
28

---

Published in Transactions on Machine Learning Research (11/2023)
No data augmentation
Data augmentation
Acturmu lheeatt
Accumulated Reward
Ircurmalated Swaned
400
300
200
100
200
100
100
50
8.00
0.25 0.50
0.75 1.00
1.25
1.50
1.75
2,00
Episodes
164
Episodes
164
Episodes
164
(a) Stacking
(b) Pyramid
(c) Triple stacking
Figure 15: Data Augmentation: Data augmentation becomes crucial when addressing harder tasks.
SkillS
RHPO
From scratch
KL-reg. to Mixture
DAGGER
600
600
Accumulated Reward
500
Accumulated Reward
500
400
300
300
200
200
100
100
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Episodes
1e4
Episodes
1e4
(a) Lifting
(b) Stacking
Figure 16: Performance on dense-reward manipulation tasks. With carefully-tuned shaped rewards simpler tasks, skill reuse is not beneficial and all methods can learn effectively.
Since the reward provides useful and reliable signal to guide learning there less need to generate good exploration data using previous skills. The advantage of skill-based methods thus primarily domains where defining a dense reward is challenging but the required sub-behaviors are easier specify.
D.4 Combinations with Additional Transfer Mechanisms
The separation of collect and infer processes means that SkillS can be flexibly combined with other skill techniques, included the previously analysed baselines. In Figure 17 we compare the performance of SkillS from the paper against versions of SkillS where the new skill is trained using auxiliary regularisation objectives in the manipulation Lift and Stack tasks. We found no benefit of changing the underlying algorithm to infer the new skill. The main takcaway of this result is that SkillS can flexibly combined with different algorithms to run inference which could be beneficial in some settings not considered here.
D.5 Extended results
Figures 18 and 19 extend the results of Figures 5c and 7a respectively to all of the manipulation domains. Overall, the results across the domains is consistent with the analysis presented in the main text.
For Figure 18 we show the average execution of the top-5 most chosen skills at any point during training for case of plotting. We see a consistent pattern across domains: the scheduler chooses from all skills uniformly before settling on a subset that are relevant for the task. However eventually and by the end of training, the scheduler consistently chooses the new skill most often.
29

---

Published in Transactions on Machine Learning Research (11/2023)
KL-rey.
SkillS
600
600
Accumulated Reward
500
Accumulated Reward
500
400
400
300
300
200
200
100
100
0
0.0
0.5
1.0
1.5
2.0 Episades
2.5
3.0
3.5
4,0
0.0
0.2
0.4
0.6
0.8
1.0
1e3
Episodes
1e4
(a) Lifting
(b) Stacking
Figure 17: Performance of SkilIS when combined with other approaches to transfer, showing its flexibility.
New skill
Lift green
Reach blue
Stack green on yellow
Close fingers
Open fingers
Reach green
Stack yellow blue
Lift blue
Place green on yellow
Reach yellow
350
250
Avg. Consecutiv Execution
Avq. Consecutive on
300
250
200
150
150
100
100
50
0
4
0.0
0.2
0.4
0.6
0.0
1.0
1.2
Episodes
1e3
Episodes
164
(a) Lift
(b) Stack
200
200
Avg. Consecutive Execution
Avg. Consecutive Execution
150
150
100
100
50
1.2
0.8
Episodes
165
Episodes
(c) Pyramid
(d) Triple Stacking
Figure 18: Skill average execution (as number of timesteps) during training sparso-reward manipulation tasks.
Figure 19 shows that there is a large benefit of having separate inference mechanism as part of SkilIS. In general, the performance of the new skill exceeds that of the scheduler even when the new skill trained entirely offline ie.. the scheduler is constrained to only choose from the pre-trained skills.
30

---

Published in Transactions on Machine Learning Research (11/2023)
New skill
Scheduler (without new skill)
Scheduler (incl. new skill)
New skill (trained entirely offline)
500
600
Accumu laed Reware
500
Accumulated Reward
400
400
Accumulatse lazed
300
300
200
100
100
0.0
0.2
0.4
0.6
0.0
1.0
1.2
Episodes
1e3
Episodes
164
(a) Lift
(b) Stack
600
ated Reward
500
Accumulated Reward
300
400
acd
300
200
200
100
100
0
1
2
4
5
0
2
3
7
8
Episodes
1e4
Episodes
1e4
(c) Pyranii
(d) Triple Stacking
Figure 19: Benefit of having a separately inferred new skill across all manipulation domains.
31


Reviews:
[{"review_id": "slh9flKmql", "reviewer": "Reviewer_56Lf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper presents an well motivated, intuitive approach to leveraging pre-existing skills while avoiding being constrained by them, and show strong results and extensive justification of various design choices. \n\nStrengths:\n- The proposed method exhibits strong performance in sparse reward tasks that neither baselines learning from scratch do not appear progress nor baselines constrained to following the pre-learned skills are able to make meaningful progress on.\n- The authors show very thorough ablations over various aspects of the proposed method, illustrating the benefits of specific details as well as robustness in various settings.\n\nMinor additional questions/comments:\n- Regarding including the new skill as an option for the scheduler: In principle, since the new skill changes as we jointly train the new skill and the scheduler, this could lead to instability in the scheduler, as the effective meaning of that particular action changes over time. Even beyond this instability, I'm curious what the performance of the proposed algorithm if the scheduler was limited to the frozen skills for explroation. How important is it that we get some amount of on-policy data for the newly learned skill?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors proposes a new algorithm Skill Scheduleer that combines a hierarchical reinforcement learning algorithm that learns a skill-scheduler on top of preexisting low-level policies to explore effectively and an offline RL learner trained to maximally exploit a new task given the exploratory data collected via the scheduler.\n\nTo train the upper-level scheduler, the authors propose HCMPO, a small variation on the off-policy MPO algorithm algorithm to handle different timestep offsets arising from different time horizons for the upper-level actions. The authors directly use critic regularized regression as the offline RL algorithm to train the final exploitation.\n\nThe authors show promising results on several sparse reward tasks against various baselines when evaluated with different sets of pretrained skills. They include extensive ablations on robustness to to different skill settings, the importance of variable temporal abstraction, the importance of making the newly learned skill available to the scheduler.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See additional ablation mentioned above for excluding the newly learned skill from exploration.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VQ1EVDbB8S", "reviewer": "Reviewer_y8CU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Easy to follow, with a simple and effective method design.\n2. Clear problem statement, discussing the shortcomings of various existing skill learning methods and proposing targeted solutions.\n3. On-point ablation.\n\nWeaknesses:\n1.The learning of the new skill is not detailed enough. It is better to add an algorithm pseudocode.\n2.The skill reuse in this work is more like a transfer learning mechanism. It should be compared with some transfer learning methods, such as PQRL[1], PTF[2].\n\n[1] Probabilistic policy reuse in a reinforcement learning agent\n[2] Efficient Deep Reinforcement Learning via Adaptive Policy Transfer\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work explores the problem of reusing skills in a new task. Considering that fine-tuning skills may lead to skill degradation, and not fine-tuning skills only for combination may result in suboptimal policies. Therefore, the authors propose using skills to assist exploration while simultaneously learning a new skill. Eventually, the new skill is used to solve the task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tThe proposed approach SKILLS is very similar in principle to transfer learning from multiple source policies. Maybe they did not explicitly learn a scheduler. What is SKILLS’s advantage compared to them?\n2.\tIn figure 7, the labels are confusing. Some curves have the same color.\n3.\tDoes the learning of the scheduler have instability? For example, the value corresponding to selecting actions of the new policy is changing because the new policy is also constantly updating, which may cause the early buffer data in training to be incorrect.\n\n", "broader_impact_concerns": "I believe there is no need for additional supplementation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Th0UHvLvrE", "reviewer": "Reviewer_Ht8Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The motivation of the proposed method is sufficient.\n- The investigation of related work is broad and useful. I appreciate Figure 1 which clearly categorizes existing works.\n- The experiments are conducted from diverse perspectives.\n\n&nbsp;\n\n### Weaknesses:\n\n- The presentation of the proposed method is almost clear. However, I think more details are needed for better understanding.\n    - It is not clear whether the skill scheduler and the new skill are trained concurrently. A pseudocode and some more clarifications will help a lot.\n    - The reason of choosing CRR for the learning of the new skill is not well explained or supported. It is not clear why CRR beats the alternatives like CQL[1] and IQL[2].\n- The current experimental results are not convincing enough to me.\n    - According to the sentence “unless otherwise specified, we focus on the performance of the new skill when executed in the environment in isolation for the entire episode”, does this mean that the reported results for SkillS are the results of the new skill?\n    - Among all the experiments, I am most interested in Figure 5c and Figure 7a, I think the current results are not adequate. What are the environments/tasks used for the two experiments? I expect to see the experiments on more environments. This is because I am a little bit surprised to see the poor performance of “w/ new skill” after checking the given skills in the appendix.\n    - The exploration ability of SkillS is not directly demonstrated. As the authors mentioned a few times in this paper, I would expect to see some direct analysis on this point.\n\n\n---\n### Reference\n[1] Kumar et al., **Conservative Q-Learning for Offline Reinforcement Learning.** NeurIPS 2020\n\n[2] Kostrikov et al., **Offline Reinforcement Learning with Implicit Q-Learning.** ICLR 2022\n\n[3] Whitney et al., **Dynamics-Aware Embeddings.** ICLR 2020\n\n[4] Frans et al., **Meta Learning Shared Hierarchies.** ICLR 2018\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method called SkillS for policy/skill reuse and transfer learning. SkillS has advantages over existing methods by circumventing the ineffectiveness of skill fine-tuning and reducing the dependence on expert skills.\n\nSkillS makes use of a hierarchical learning structure with a high-level skill scheduler for composing fixed skills along with a learning new skill. The main idea is to learn to sequence the fixed skills for exploration and sampling of useful data and to learn a unconstrained new skill for the entire task.\n\nSkillS is evaluated in robot manipulation and locomotion tasks with existing baselines, demonstrating the effectiveness of skill reuse and transfer learning. Analysis and ablation studies are conducted from several angles.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I think the title is improper and kind of misleading. It seems that the emphasis of current title is exploration, which may make the audience think that the paper is for hard-exploration problems. I recommend the authors to modify the title, maybe to include works like “transfer” and “reuse”.\n2. I recommend the authors to add a pseudocode to better show the training and execution process of SkIllS.\n3. I recommend the authors to conduct more experiments for Figure 5c and Figure 7a. The key is to better show how SkillS works from the lens of experiment.\n4. I recommend the authors to add the reference [3],[4]. The policy evaluation algorithm in Equation 3 of this paper which includes an additional input of the counter $c_t$ is similarly used in [3]; while [4] is a related work that leverages a similar hierarchical structure as shown in Figure 2 of this paper.\n5. The sub-captions in Figure 4 overlap with the main caption. It needs to be fixed.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is about a Hierarchical Reinforcement Learning approach able to reuse previously learned skills and simultaneously learn a new policy without being limited by the sub-optimality of the available skills.
The reviewers agree on considering this paper with acceptance: it deals with an interesting and well-motivated problem, it underlines the shortcomings of the state-of-the-art, and it provides a strong empirical evaluation that shows the effectiveness of the proposed approach.
The reviewers also raised some concerns about the presentation and asked for some improvements on the experimental part. The authors did a good job addressing these concerns, and the current version of the paper is ready for publication.
Decision: Accept (poster)