Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies

1 INTRODUCTION . The notion of hierarchy is fundamental to AI systems . Effective approaches to perceptual tasks such as vision or audio rely on hierarchical neural network architectures ( e.g . convnets and wavenets ) . Hierarchical methods are also a natural approach to solving reinforcement learning problems where reward is sparse . Consider an environment where a legged robot has to navigate a maze through winding paths to one of several possible goals . The sequence of actions ( at the level of joint controls ) required for this agent to ever see a reward is too long and complex for naı̈ve reinforcement learning methods to find . Hierarchical controllers provide an intuitive solution to this problem : first learn a low-level controller that is able to handle low-level tasks such as locomotion and balance , and then learn a high-level controller that controls the inputs to that low-level controller and handles the high-level planning tasks ( e.g . which path to take in the maze ) . While conceptually straightforward , hierarchical approaches have been difficult to realize in practice for several reasons : ( i ) the lack of constraints on intermediate representations ; ( ii ) the difficulty of learning a high-level controller when the low-level policies shifts , and ( iii ) the continued problem of sample-inefficiency when training both low-level and high-level controller jointly with sparse rewards . While there has been some progress recently , there is still no general hierarchical reinforcement learning method that is robust across domains , easy to implement , and effective . The approach in this work avoids these problems by simplifying the high-level optimization problem to choosing from a set of independently trained low-level controllers . The problem then becomes how to train a variety of controllers which are generally useful for an agent acting in the world and are sufficiently diverse so that the high-level controller will always have access to the building blocks it needs to solve the high-level task . ∗Work done as an intern at Facebook AI Research . See videos and more at https : //sites.google.com/view/hrl-ep3 The basic method of learning our low-level skills is as follows . First , as with Heess et al . ( 2016 ) , we factorize the state space into components that describe the internal and external properties of the agent . The former carries information needed for proprioception ( e.g . internal joint angles ) and is exclusively used by the low-level policy and the latter is used exclusively by the high-level policy . Second , we provide a weak supervisory signal to the low-level policies that rewards them for changes in the external part of the space . We do not specify how ; the low-level agent is rewarded for any change in the perceptual features . The intuition is that a policy is useful to an agent if following it changes the external world or the agent ’ s place in it , ( e.g . by changing position or moving objects ) . Finally , we build into the agent a notion of repeated cycles of action . Biological systems are known to possess central pattern generators ( CPGs ) that produce rhythmic outputs , assisting with perception , locomotion , and self-regulation ( Marder & Bucher , 2001 ) . Taking inspiration from this , we give our agent access to a “ phase function ” and reward it for acting in consistent , cyclic ways . Using this method of training , we train a number of low-level controllers on Mujoco locomotion tasks and are able to learn a diverse and effective set of low-level policies . We then show that these policies are easily controllable by our high-level controller and sufficient for solving the final task . We show impressive results on a variety of difficult Mujoco maze and navigation tasks with sparse rewards and demonstrate that our method compares favorably to other recent hierarchical RL approaches . Finally , we demonstrate that our method is robust to choice of RL algorithm . 2 RELATED WORK . There is an extensive literature on hierarchical approaches to reinforcement learning ( Sutton et al. , 1999 ; Dayan & Hinton , 1992 ; Dietterich , 1999 ) . Options ( Sutton et al. , 1999 ) are perhaps the most standard framing for hierarchical RL , formalizing the notion of a subroutine in an MDP . Our lowlevel networks are options with a deterministic return condition . While earlier works used prespecified subroutines , and focused on learning when to call them , recent works , for example Bacon et al . ( 2017 ) ; Vezhnevets et al . ( 2017 ) ; Nachum et al . ( 2018 ) , have demonstrated success in learning multiple levels of hierarchy end-to-end from only task reward . Still , in more complex settings such as the ones discussed in this work , with high dimensional action spaces , and especially with sparse rewards , performing unsupervised pre-training , or using auxiliary tasks during learning or pre-training have led to better results ( Heess et al. , 2016 ; Frans et al. , 2017 ; Florensa et al. , 2017 ; Haarnoja et al. , 2018b ; Eysenbach et al. , 2018 ; Hausman et al. , 2018 ) . One line of work in this direction starts from a variational inference approach to intrinsic motivation and exploration ( Mohamed & Rezende , 2015 ; Gregor et al. , 2016 ; Florensa et al. , 2017 ; Eysenbach et al. , 2018 ) . They use an actor parameterized by state and a latent vector in such a way that the latent vector is predictable from a final state or a sequence of states the actor visits , but otherwise , the actions have high entropy . The latter of these works use this approach for designing hierarchical policies . In Florensa et al . ( 2017 ) , they use a pre-training task , and the prediction cost is an auxiliary reward . The main differences from our work is that we do not try to use stochastic neural networks , and we do not try to compress the one-hot representation of the low level networks into a dense vector during low level training . Instead we keep them fully separate . We also do not attempt to impose any regularization to encourage the low level networks to be diverse . These can be considered simplifications ; what we show is that the simple thing works quite well . In Eysenbach et al . ( 2018 ) the low-level policy is trained without using any environmental reward by pre-training using the unsupervised variational objective ; they then training a high-level actor to issue commands via the latent vector . This latter work is in some senses more general than ours , in that a user is not required to separate out the proprioceptive variables from the external sensory variables . In addition , the formulation directly encourages diverse and controllable low-level policies . Nevertheless , both these works were validated mostly in the same settings we use here , where the separation between variables is natural , and the reward in equation 3.2 is well motivated . In these settings our method is simpler , and performs well in comparison ( see Figure 7a ) . Our work is most closely related to Heess et al . ( 2016 ) . We operate under the same basic philosophy , separating the agent into a low-level component that only sees “ proprioceptive ” information , and works at a fine time scale , and a coarse agent that has access to “ sensory ” information . Like that work , we evaluate or method in the setting of locomotive agents . However , there are several differences between Heess et al . ( 2016 ) and this work . There , the low-level component is trained along with a provisional high-level component to solve a set of high-level tasks designed for the agent and transfer tasks , whereas in this work , the low-level controller is trained with only the objective “ change the sensory responses ” in each setting . Furthermore , our underlying architecture is different . Whereas in Heess et al . ( 2016 ) the high-level agent interacts with the low-level component via continuous control vectors , in this work , we sample one ofN low-level policies , as in Frans et al . ( 2017 ) . In addition , whereas they train their low-level actor with only the current state , our low-level actor is shown to benefit from using a pattern generator . Finally , because of the structure of our lowlevel task , we need to ensure that the low-level policies are diverse and controllable ( which in Heess et al . ( 2016 ) is given by the design of the pre-training task ) , and we do so by using multiple random instantiations of the “ change the sensory response ” policies that maximize equation 3.2 . Our work can thus be considered in some ways a simplification and generalization of Heess et al . ( 2016 ) . The use of phase information in the low-level actor recalls Holden et al . ( 2017 ) ; Sharma & Kitani ( 2018 ) . In the former , a neural network is used to control the gait of a video game character . The weights of the neural network are a function of a phase variable whose cycle corresponds to a cycle of the gait , and are trained as regressors on motion-capture data . The latter work takes this approach to RL problems that have an explicit cyclic structure , in particular the walker and hopper environments from MuJoCo . Phase has also been used in similar way in Peng et al . ( 2018 ; 2017 ; 2016 ) . While we share a similar motivation with these , our low-level actor processes the phases in a different way : instead of a phase variable directly modulating the weight of the network , we concatenate an embedded phase index to the input of the network . 3 METHODOLOGY . Section 3.1 describes the division of the state space into the proprioceptive or internal state and the external state . In section 3.2 we describe the reward function which we use to train the low-level policies . These policies take in the proprioceptive state and the reward maximizes displacement in the external state . Section 3.3 describes how we augment the low-level policy by adding an additional input that tells the agent where it is in a “ phase cycle ” and constraining the motion of the agent to match that phase cycle . Finally , in section 3.4 we describe how we learn a high-level controller by learning to choose when and which of the trained low-level policies should be used . 3.1 PROPRIOCEPTIVE AND EXTERNAL STATES . We divide the agent ’ s sensory output into two sets of variables . The proprioceptive states , sp , are the measurements of the agents own body that can be directly affected by control . The external states , se are measurements of the environment outside of the agents direct control . The proprioceptive part of the state is fed into the low-level policy network . This is the part of the state that gives the agent the information about the configuration of its own body and actuators . The low-level policy is independent of the external state as the low-level policies are intended to be generally usefully in moving in the external state independently of the exact location . The external part of the state is fed into the high-level policy network . The high-level policy is responsible for things like moving to particular coordinates in the world , and so the external state is necessary . The high-level policy does not need the low-level states such as joint locations as that is handled by the low-level policy .

Summary: This paper proposes a method to train high- and low-level controllers to tackle hierarchical RL.  The novelty is framing hierarchical RL as a problem of training a diverse set of LL controllers such that they can be used by a HL controller to solve high-level tasks.  By dividing state representation into proprioceptive and task-specific, the reward used to train LL and HL controllers are simplified.  Experimental result shows that the method is effective at solving maze environments for both ant and humanoid.