Hierarchical Skills for Efficient Exploration

1 Introduction . A promising direction for improving the sample efficiency of reinforcement learning agents in complex environments is to pre-train low-level skills that are then used to structure the exploration in downstream tasks [ 23 , 19 , 27 , 13 , 30 ] . This has been studied in particular for the control of ( simulated ) robots , where there is a natural hierarchical decomposition of the downstream tasks into low-level control of the robot ’ s actuators with a skill policy , and a high-level control signal that specifies a direction or target robot configuration with coarser temporal resolution . The large body of work on unsupervised skill or option discovery in hierarchical reinforcement learning ( HRL ) for continuous control relies , explicitly or implicitly , on prior knowledge that low-level skills should control the center of mass of the robot [ 26 , 17 , 13 , 43 , 6 ] . This nicely fits a wide range of benchmark tasks that are variants of navigation problems , but the benefit of such hierarchical setups outside this problem class is unclear . The prior knowledge embedded in a pre-trained skill defines a specific trade-off between sample efficiency and generality . Skills that severely constrain the high-level action space to elicit specific behavior ( e.g. , translation of the center of mass ) are likely to provide the largest gains in sample efficiency , but are unlikely to be useful on a diverse set of downstream tasks . Conversely , low-level skills that expose many degrees of freedom are more widely applicable but less useful for guiding exploration . There is , thus , no single universally superior pre-trained skill . Depending on the downstream task , different skills might also be useful to efficiently explore in different parts of the environment . In this paper , we aim to acquire skills that are useful for a variety of tasks while still providing strong exploration benefits . We propose to pre-train a hierarchy of skills of increasing complexity which can subsequently be composed with a high-level policy . In the context of simulated robots , each skill consists of controlling a part of the robot configuration over a short time horizon , such 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as the position of the left foot of a humanoid , or the orientation of its torso . Skills of increasing complexity are constructed by jointly controlling larger portions of the configuration . These skills are modelled with a shared policy and pre-trained in an environment without rewards and containing the robot only . Subsequently , skills are used in downstream tasks within a three-level hierarchical policy : the highest level selects the skill ( which specifies a goal space ) , the second level the target configuration within that skill ( the goal ) , and the pre-trained skill performs the low-level control to reach the goal . Compared to standard approaches involving a single static pre-trained skill [ 13 , 43 ] , our approach offers increased flexibility for structuring exploration and offloads the issue of selecting prior knowledge from pre-training to downstream task learning . As a result , our skills can be acquired once per robot and applied to many different tasks . We perform an experimental analysis of our hierarchical pre-training on a new set of challenging sparse-reward tasks with simulated bipedal robots . Our experiments show that each task is most efficiently explored by a distinct set of low-level skills , confirming that even on natural tasks , where locomotion is of primal importance , there is no overall single best pre-trained skill . We further show that dynamic selection of goal spaces with a three-level hierarchy performs equally or better than a generic skill on all tasks , and can further improve over the best single skills per task . The main contributions of our work are summarized as follows : • We propose a novel unsupervised pre-training approach that produces a hierarchy of skills based on control of variable feature sets . • We demonstrate how to automatically select between different skills of varying complexity with a three-level hierarchical policy that selects skills , goals , and native actions . • We introduce a benchmark suite of sparse-reward tasks that allows for consistent and thorough evaluation of motor skills and HRL methods beyond traditional navigation settings . • We study the implications of prior knowledge in skills experimentally and showcase the efficacy of our hierarchical skill framework on the proposed benchmark tasks , achieving superior results compared to existing skill discovery and HRL approaches [ 13 , 33 , 53 ] . 2 Related Work . The success of macro-operators and abstraction in classic planning systems [ 14 , 40 ] has inspired a large body of works on hierarchical approaches to reinforcement learning [ 8 , 46 , 3 , 50 , 33 , 39 ] . While the decomposition of control across multiple levels of abstraction provides intuitive benefits such as easier individual learning problems and long-term decision making , recent work found that a primary benefit of HRL , in particular for modern , neural-network based learning systems , stems from improved exploration capabilities [ 21 , 35 ] . From a design perspective , HRL allows for separate acquisition of low-level policies ( options ; skills ) , which can dramatically accelerate learning on downstream tasks . A variety of works propose the discovery of such low-level primitives from random walks [ 26 , 52 ] , mutual information objectives [ 17 , 13 , 43 , 6 ] , datasets of agent or expert traces [ 37 , 2 , 25 ] , motion capture data [ 36 , 31 , 42 ] , or from dedicated pre-training tasks [ 15 , 27 ] . In order to live up to their full potential , low-level skills must be useful across a large variety of downstream tasks . In practice , however , a trade-off between generality ( wide applicability ) and specificity ( benefits for specific tasks ) arises . A large portion of prior work on option discovery and HRL resolved this trade-off , explicitly or implicitly , in favor of specificity . This can be exemplified by the choice of test environments , which traditionally revolve around navigation in grid-world mazes [ 46 , 8 , 10 ] . In recent work on simulated robotics , navigation problems that are similar in spirit remain the benchmarks of choice [ 12 , 33 , 13 , 27 , 2 ] . In these settings , directed locomotion , i.e. , translation of the robot ’ s center of mass , is the main required motor skill , and the resulting algorithms require a corresponding prior . This prior is made explicit with skills partitioning the state space according the agent ’ s position [ 13 , 43 ] , or is implicitly realized by high contribution of position features to reward signals [ 33 , 34 , 2 ] ( Appendix G ) . Similarly , works that target nonnavigation environments acquire skills with tailored pre-training tasks [ 38 ] or in-domain motion capture data [ 36 , 31 , 32 ] . In contrast , our work is concerned with learning low-level skills without extra supervision from traces or pre-training task design , and which do not prescribe a fixed trade-off towards a particular type of behavior . 3 Hierarchical Skill Learning . 3.1 Overview . In this work , we propose an approach where we first acquire low-level policies that are able to carry out a useful set of skills in an unsupervised manner , i.e. , without reward signals from the main tasks of interest . We subsequently employ these skills in a hierarchical reinforcement learning setting ( Figure 1 ) . The pre-training environment consists solely of the robot . In downstream tasks , additional objects may be present besides the robot ; these may be fixed ( e.g. , obstacles ) or only indirectly controllable ( such as a ball ) . Crucially , the usefulness of a skill policy stems from providing additional structure for effective exploration . In order to provide benefits across a wide range of tasks , skills need to support both fast , directed exploration ( e.g. , locomotion ) as well as precise movements ( e.g. , lift the left foot while bending down ) . We propose to fulfill these requirements with short-horizon , goal-directed low-level policies that are trained to achieve target configurations of robot-level state features such as the position or orientation of its torso or relative limb positions . We denote this feature space with Sg . This allows the definition of a hierarchy of skills by controlling single features and their combinations , resulting in varying amounts of control exposed to high-level policies . Each skill is trained to reach goals in a goal space GF defined over a set of features F of Sg , yielding policies of the form ⇡loF : S p ⇥ G F ! A F2F with proprioceptive observations Sp ( Figure 1 , left ) . Goal-directed skill policies are trained without task-specific rewards , relying solely on state features related to the specific robot and the prescribed hierarchy of goal spaces . On downstream tasks , high-level policies operate with a combinatorial action space as the skill policy ⇡lo is conditioned on both a feature set F and a concrete goal g 2 GF . State spaces are enriched with task-specific features S+ , s.t . S = Sp [ S+ , that contain information regarding additional objects in the downstream task . This enriched state space is only available to the high-level policy . We model the high-level policy in a hierarchical fashion with two policies ⇡f : S ! F , ⇡g : S ⇥ F ! GF , that prescribe a goal space and a goal , respectively ( Figure 1 , right ) . With a task- and state-dependent policy ⇡f , it is possible to not only select the required set of skills for a given environment , but also to switch between them within an episode . In this three-level hierarchy , the higher-level policy ⇡f explores the hierarchy of goal spaces and dynamically trades off between generality and specificity . Temporal abstraction is obtained by selecting new high-level actions at regular intervals but with a reduced frequency compared to low-level actions . We term the resulting learning algorithm HSD-3 , emphasizing the three different levels of control ( goal spaces , goals and native actions ) obtained after our hierarchical skill discovery phase . 3.2 Unsupervised Pre-Training . During pre-training , skill policies act in an MDP M with proprioceptive states sp , native actions a , transition probabilities and initial states sp0 2 S p 0 drawn from a distribution P0 [ 45 , for example ] , but without an extrinsic reward function . The skill policies are trained to reach goals in a goal space defined over Sg consisting of robot-level sensor readings , which may include non-proprioceptive information . Critically , each skill ⇡loF aims to achieve goals defined over a variable-sized subset of features F of Sg . To avoid learning a combinatorial number of independent policies , we follow the principle of universal value function approximators [ 41 ] and share the parameters of each skill . We augment the policy ’ s input with goal and goal space information , and learn a single set of parameters ✓ , leading to ⇡lo✓ : S p ⇥ F ⇥ G ! A , where F : = { F : F ✓ Sg } and G : = { GF : F ✓ Sg } . For ease of modeling , F is provided to ⇡lo✓ as a bag-of-words input where a coordinate is set to 1 if its respective feature is included in F and to 0 otherwise . Also , the low-level policy receives goals relative to the current values of the sg . Thus , at each time step , the input goal to the policy is updated to reflect the progress towards the goal . As a learning signal , we provide a distance-based reward as R ( sg , a , sg 0 , F , g ) : = || ! F ( sg ) g||2 || ! F ( sg 0 ) g||2 , where ! F : Sg ! GF is a fixed transformation that selects the subset F of goal space features and applies a suitable normalization . The selection of features for Sg and transformations ! F represents the prior knowledge that we utilize for unsupervised pre-training . We discuss examples for bipedal robots in our experiments and in Appendix B . During training , features F and goals g are sampled anew for each episode in the environment . Episodes consist of several dozen steps only , reflecting the fact that we are interested in short-horizon skills . In-between episodes , we reset the simulation to states drawn from Sp0 only sporadically to encourage the resulting policies to be applicable in a wide range of states , facilitating their sequencing under variable temporal abstractions . Likewise , rewards are propagated across episodes as long as the simulation is not reset . The low-level policy parameters ✓ can be optimized with any reinforcement learning algorithm ; here , we opt for Soft Actor-Critic ( SAC ) , a state-of-the-art method tailored to continuous action spaces [ 18 ] . Pseudo-code for the pre-training algorithm and further implementation details are provided in Appendix C .

Summary: This work proposes a benchmark task with bipedal robots instead of locomotion tasks, where the bipedal robots are low dimensional and perform various movements. It also proposes a hierarchical reinforcement learning method with three levels: a policy that specifies a goal space (a set of features to operate on), a policy to specify the goal configuration given that goal space, and a low level policy to reach the desired goal configuration. It learns the low level policies through unsupervised learning, and then optimizes off policy the high level options by optimizing the value function.