Hierarchical Skills for Efficient Exploration

1 Introduction . A promising direction for improving the sample efficiency of reinforcement learning agents in complex environments is to pre-train low-level skills that are then used to structure the exploration in downstream tasks [ 23 , 19 , 27 , 13 , 30 ] . This has been studied in particular for the control of ( simulated ) robots , where there is a natural hierarchical decomposition of the downstream tasks into low-level control of the robot ’ s actuators with a skill policy , and a high-level control signal that specifies a direction or target robot configuration with coarser temporal resolution . The large body of work on unsupervised skill or option discovery in hierarchical reinforcement learning ( HRL ) for continuous control relies , explicitly or implicitly , on prior knowledge that low-level skills should control the center of mass of the robot [ 26 , 17 , 13 , 43 , 6 ] . This nicely fits a wide range of benchmark tasks that are variants of navigation problems , but the benefit of such hierarchical setups outside this problem class is unclear . The prior knowledge embedded in a pre-trained skill defines a specific trade-off between sample efficiency and generality . Skills that severely constrain the high-level action space to elicit specific behavior ( e.g. , translation of the center of mass ) are likely to provide the largest gains in sample efficiency , but are unlikely to be useful on a diverse set of downstream tasks . Conversely , low-level skills that expose many degrees of freedom are more widely applicable but less useful for guiding exploration . There is , thus , no single universally superior pre-trained skill . Depending on the downstream task , different skills might also be useful to efficiently explore in different parts of the environment . In this paper , we aim to acquire skills that are useful for a variety of tasks while still providing strong exploration benefits . We propose to pre-train a hierarchy of skills of increasing complexity which can subsequently be composed with a high-level policy . In the context of simulated robots , each skill consists of controlling a part of the robot configuration over a short time horizon , such 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as the position of the left foot of a humanoid , or the orientation of its torso . Skills of increasing complexity are constructed by jointly controlling larger portions of the configuration . These skills are modelled with a shared policy and pre-trained in an environment without rewards and containing the robot only . Subsequently , skills are used in downstream tasks within a three-level hierarchical policy : the highest level selects the skill ( which specifies a goal space ) , the second level the target configuration within that skill ( the goal ) , and the pre-trained skill performs the low-level control to reach the goal . Compared to standard approaches involving a single static pre-trained skill [ 13 , 43 ] , our approach offers increased flexibility for structuring exploration and offloads the issue of selecting prior knowledge from pre-training to downstream task learning . As a result , our skills can be acquired once per robot and applied to many different tasks . We perform an experimental analysis of our hierarchical pre-training on a new set of challenging sparse-reward tasks with simulated bipedal robots . Our experiments show that each task is most efficiently explored by a distinct set of low-level skills , confirming that even on natural tasks , where locomotion is of primal importance , there is no overall single best pre-trained skill . We further show that dynamic selection of goal spaces with a three-level hierarchy performs equally or better than a generic skill on all tasks , and can further improve over the best single skills per task . The main contributions of our work are summarized as follows : • We propose a novel unsupervised pre-training approach that produces a hierarchy of skills based on control of variable feature sets . • We demonstrate how to automatically select between different skills of varying complexity with a three-level hierarchical policy that selects skills , goals , and native actions . • We introduce a benchmark suite of sparse-reward tasks that allows for consistent and thorough evaluation of motor skills and HRL methods beyond traditional navigation settings . • We study the implications of prior knowledge in skills experimentally and showcase the efficacy of our hierarchical skill framework on the proposed benchmark tasks , achieving superior results compared to existing skill discovery and HRL approaches [ 13 , 33 , 53 ] . 2 Related Work . The success of macro-operators and abstraction in classic planning systems [ 14 , 40 ] has inspired a large body of works on hierarchical approaches to reinforcement learning [ 8 , 46 , 3 , 50 , 33 , 39 ] . While the decomposition of control across multiple levels of abstraction provides intuitive benefits such as easier individual learning problems and long-term decision making , recent work found that a primary benefit of HRL , in particular for modern , neural-network based learning systems , stems from improved exploration capabilities [ 21 , 35 ] . From a design perspective , HRL allows for separate acquisition of low-level policies ( options ; skills ) , which can dramatically accelerate learning on downstream tasks . A variety of works propose the discovery of such low-level primitives from random walks [ 26 , 52 ] , mutual information objectives [ 17 , 13 , 43 , 6 ] , datasets of agent or expert traces [ 37 , 2 , 25 ] , motion capture data [ 36 , 31 , 42 ] , or from dedicated pre-training tasks [ 15 , 27 ] . In order to live up to their full potential , low-level skills must be useful across a large variety of downstream tasks . In practice , however , a trade-off between generality ( wide applicability ) and specificity ( benefits for specific tasks ) arises . A large portion of prior work on option discovery and HRL resolved this trade-off , explicitly or implicitly , in favor of specificity . This can be exemplified by the choice of test environments , which traditionally revolve around navigation in grid-world mazes [ 46 , 8 , 10 ] . In recent work on simulated robotics , navigation problems that are similar in spirit remain the benchmarks of choice [ 12 , 33 , 13 , 27 , 2 ] . In these settings , directed locomotion , i.e. , translation of the robot ’ s center of mass , is the main required motor skill , and the resulting algorithms require a corresponding prior . This prior is made explicit with skills partitioning the state space according the agent ’ s position [ 13 , 43 ] , or is implicitly realized by high contribution of position features to reward signals [ 33 , 34 , 2 ] ( Appendix G ) . Similarly , works that target nonnavigation environments acquire skills with tailored pre-training tasks [ 38 ] or in-domain motion capture data [ 36 , 31 , 32 ] . In contrast , our work is concerned with learning low-level skills without extra supervision from traces or pre-training task design , and which do not prescribe a fixed trade-off towards a particular type of behavior . 3 Hierarchical Skill Learning . 3.1 Overview . In this work , we propose an approach where we first acquire low-level policies that are able to carry out a useful set of skills in an unsupervised manner , i.e. , without reward signals from the main tasks of interest . We subsequently employ these skills in a hierarchical reinforcement learning setting ( Figure 1 ) . The pre-training environment consists solely of the robot . In downstream tasks , additional objects may be present besides the robot ; these may be fixed ( e.g. , obstacles ) or only indirectly controllable ( such as a ball ) . Crucially , the usefulness of a skill policy stems from providing additional structure for effective exploration . In order to provide benefits across a wide range of tasks , skills need to support both fast , directed exploration ( e.g. , locomotion ) as well as precise movements ( e.g. , lift the left foot while bending down ) . We propose to fulfill these requirements with short-horizon , goal-directed low-level policies that are trained to achieve target configurations of robot-level state features such as the position or orientation of its torso or relative limb positions . We denote this feature space with Sg . This allows the definition of a hierarchy of skills by controlling single features and their combinations , resulting in varying amounts of control exposed to high-level policies . Each skill is trained to reach goals in a goal space GF defined over a set of features F of Sg , yielding policies of the form ⇡loF : S p ⇥ G F ! A F2F with proprioceptive observations Sp ( Figure 1 , left ) . Goal-directed skill policies are trained without task-specific rewards , relying solely on state features related to the specific robot and the prescribed hierarchy of goal spaces . On downstream tasks , high-level policies operate with a combinatorial action space as the skill policy ⇡lo is conditioned on both a feature set F and a concrete goal g 2 GF . State spaces are enriched with task-specific features S+ , s.t . S = Sp [ S+ , that contain information regarding additional objects in the downstream task . This enriched state space is only available to the high-level policy . We model the high-level policy in a hierarchical fashion with two policies ⇡f : S ! F , ⇡g : S ⇥ F ! GF , that prescribe a goal space and a goal , respectively ( Figure 1 , right ) . With a task- and state-dependent policy ⇡f , it is possible to not only select the required set of skills for a given environment , but also to switch between them within an episode . In this three-level hierarchy , the higher-level policy ⇡f explores the hierarchy of goal spaces and dynamically trades off between generality and specificity . Temporal abstraction is obtained by selecting new high-level actions at regular intervals but with a reduced frequency compared to low-level actions . We term the resulting learning algorithm HSD-3 , emphasizing the three different levels of control ( goal spaces , goals and native actions ) obtained after our hierarchical skill discovery phase . 3.2 Unsupervised Pre-Training . During pre-training , skill policies act in an MDP M with proprioceptive states sp , native actions a , transition probabilities and initial states sp0 2 S p 0 drawn from a distribution P0 [ 45 , for example ] , but without an extrinsic reward function . The skill policies are trained to reach goals in a goal space defined over Sg consisting of robot-level sensor readings , which may include non-proprioceptive information . Critically , each skill ⇡loF aims to achieve goals defined over a variable-sized subset of features F of Sg . To avoid learning a combinatorial number of independent policies , we follow the principle of universal value function approximators [ 41 ] and share the parameters of each skill . We augment the policy ’ s input with goal and goal space information , and learn a single set of parameters ✓ , leading to ⇡lo✓ : S p ⇥ F ⇥ G ! A , where F : = { F : F ✓ Sg } and G : = { GF : F ✓ Sg } . For ease of modeling , F is provided to ⇡lo✓ as a bag-of-words input where a coordinate is set to 1 if its respective feature is included in F and to 0 otherwise . Also , the low-level policy receives goals relative to the current values of the sg . Thus , at each time step , the input goal to the policy is updated to reflect the progress towards the goal . As a learning signal , we provide a distance-based reward as R ( sg , a , sg 0 , F , g ) : = || ! F ( sg ) g||2 || ! F ( sg 0 ) g||2 , where ! F : Sg ! GF is a fixed transformation that selects the subset F of goal space features and applies a suitable normalization . The selection of features for Sg and transformations ! F represents the prior knowledge that we utilize for unsupervised pre-training . We discuss examples for bipedal robots in our experiments and in Appendix B . During training , features F and goals g are sampled anew for each episode in the environment . Episodes consist of several dozen steps only , reflecting the fact that we are interested in short-horizon skills . In-between episodes , we reset the simulation to states drawn from Sp0 only sporadically to encourage the resulting policies to be applicable in a wide range of states , facilitating their sequencing under variable temporal abstractions . Likewise , rewards are propagated across episodes as long as the simulation is not reset . The low-level policy parameters ✓ can be optimized with any reinforcement learning algorithm ; here , we opt for Soft Actor-Critic ( SAC ) , a state-of-the-art method tailored to continuous action spaces [ 18 ] . Pseudo-code for the pre-training algorithm and further implementation details are provided in Appendix C .

Summary: This paper presents a hierarchical framework for training locomotion policies for tasks with sparse reward. Low level policies are trained to achieve a selection of subgoals and high level policies are trained by exploring in the goal spaces. Challenging locomotion tasks are introduced to show the effectiveness of the proposed methods.