Evolution Strategies as an Alternate Learning method for Hierarchical Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has been used to create artificially intelligent agents for tasks ranging from robot locomotion ( Haarnoja et al. , 2018 ) to video games such as StarCraft ( Vinyals et al. , 2019 ) and board games such as chess and Go ( Silver et al. , 2018 ) . Many such agents use Markov Decision Process ( MDP ) based learning methods , such as Deep Q-Networks ( DQNs ) ( Mnih et al. , 2015 ) and the policy gradient family of methods ( Sutton et al. , 1999a ; Sutton & Barto , 2018 ) . MDP and gradient based RL methods are also used by hierarchical RL ( HRL ) algorithms , which are a class of RL algorithms that excel at challenging RL problems by decomposing them into sub tasks , which mimics the way we as humans build new skills on top of existing simpler skills . HRL methods have seen success in solving some of the hardest RL environments such as Montezumas revenge ( Vezhnevets et al. , 2017 ; Badia et al. , 2020 ) and generating complex robot behaviours ( Nachum et al. , 2018 ) . Another area of reinforcement learning that has enjoyed recent success is evolution strategies . These are a family of black box evolutionary optimization techniques , which Salimans et al . ( 2017 ) showed are competitive with non-hierarchical MDP based RL methods in the robot locomotion and Atari domain . The success of such approaches has lead to a wider use of ES for tackling RL problems , but it has yet to be used to solve challenging HRL problems . ES has been used as a black box optimizer for a multitude of problems such as minimizing the drag of 3D bodies ( Beyer & Schwefel , 2002 ) , optimizing designs in structural and mechanical engineering problems ( Datoussaı̈d et al. , 2006 ) , robot locomotion ( Salimans et al. , 2017 ; Conti et al. , 2017 ; Katona et al. , 2021 ) and loss function optimization ( Gonzalez & Miikkulainen , 2020 ) . There are many different flavours of ES ( Beyer & Schwefel , 2002 ) each with different selection , mutation and self-adaption properties , for example CMA-ES ( Hansen & Ostermeier , 2001 ) and ( 1+γ ) -ES ( Beyer & Schwefel , 2002 ) . However , in this work we are concerned with the version proposed by Salimans et al . ( 2017 ) , namely Scalable Evolution Strategies ( S-ES ) because of it ’ s proven performance in the domain of robot locomotion and Atari game playing . All flavours of ES follow the scheme of sample-and-evaluate , where it samples a cloud of policy variants around it ’ s current policies parameters , evaluates these sampled policies to obtain a fitness and uses fitness to inform an update to the current policy . S-ES specifically uses fitness to approximate the gradient and moves the current policy parameters in the direction that maximizes the average reward . Given that ES is both a blackbox process ( making it indifferent to temporal details ) and is a gradient free method it suffers from sub optimal sample efficiency , however Liu et al . ( 2019 ) showed promising results addressing this inefficiency using trust regions which allow for more of a monotonic improvement . S-ES obtained results comparable to MDP methods on a set of standard MuJoCo ( Todorov et al. , 2012 ) and Atari ( Mnih et al. , 2015 ) benchmarks ( Salimans et al. , 2017 ) , however there are many RL problems harder than these standard benchmarks , some of which are near impossible to solve using non-hierarchical RL ( in this paper we refer to this as flat RL ) and others that are unsolved using flat RL . These environments can range from games such as Montezuma ’ s revenge ( Mnih et al. , 2015 ) which is challenging because it requires long-term credit assignment , to robot locomotion , navigation and interaction ( Nachum et al. , 2018 ; Florensa et al. , 2017 ) which requires complex multi-level reasoning . HRL has long held the promise of solving much more complex tasks than flat RL methods . It allows policies to abstract away large amounts of complexity and focus on solving simpler sub goals . One of the first HRL methods is known as the options framework ( Sutton et al. , 1999b ) which allows the controller policy to select the most appropriate primitive policy from a pool of primitive policies , this primitive passes control back to the controller once its actions are completed and the process repeats . Another competing HRL framework is feudal-RL ( Dayan & Hinton , 1993 ) , this framework allows for communication between the controller and primitives by having the controller set goals for the primitive to complete . Recent feudal-RL methods such as FeUdal Networks for HRL ( FuN ) ( Vezhnevets et al. , 2017 ) and HRL with Off-Policy Correction ( HIRO ) ( Nachum et al. , 2018 ) have shown a lot of promise for learning sparse reward problems and hierarchies requiring complex primitives . HIRO in particular takes the approach of using a two-level hierarchy ( one controller and one primitive ) where the controller sets the goal and reward for the primitive . For example the goal can take the form of a position an agent must reach and the reward is based on the agents distance to the goal position . HIRO , FuN and most modern HRL methods use MDP based RL methods to optimize their hierarchy of policies ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ; Sutton et al. , 1999b ; Badia et al. , 2020 ) and to the best of the authors knowledge , non-MDP based RL solvers , such as ES , have not been extensively tested on hard RL problem that are typically reserved for MDP based HRL solvers . ES has multiple advantages over MDP based RL methods , but two of these advantages make ES especially suited for HRL problems . First , it is invariant to delayed rewards and second , it has a more structured exploration mechanism ( Salimans et al. , 2017 ; Conti et al. , 2017 ) relative to MDP based RL methods . Its robustness to delayed rewards is especially useful for HRL problems as much of the difficulty of these environments can come from the long term credit assignment problem . Similarly hard RL problems often have many large local minima , requiring intelligent exploration methods in order to be solved . These advantages suggest that ES and specifically S-ES should perform well on challenging HRL problems , however to the best of the authors knowledge S-ES has not yet been applied to HRL problems . In this paper we introduce a new method1 for training a two-level hierarchy of policies which are optimized using S-ES2 , namely Scalable Hierarchical Evolution Strategies ( SHES ) . It will be evaluated on two difficult environments , which require robust robot navigation , locomotion and planning . We compare our method ’ s performance to other MDP based HRL methods that have been tested on the same environments ( Nachum et al. , 2018 ; Vezhnevets et al. , 2017 ) . This paper aims to demonstrate that SHES performs well on environments that are challenging for MDP based HRL methods and that S-ES is as viable as any MDP based RL method when training hierarchies of policies . The results obtained in this work show that SHES provides the HRL space with a high task performance , highly scalable and comparably simple HRL method . Furthermore , our method achieves these results using the same hyper-parameters as its flat RL counterpart . 2 SCALABLE HIERARCHICAL EVOLUTION STRATEGIES ( SHES ) . In this section we present our framework for learning hierarchical policies using S-ES , which we call SHES . We show how current MDP based HRL methods needed to be adapted in order to work with 1Removed for review , code can be found in supplementary material ( ScalableHrlEs.jl ) 2Removed for review , code can be found in supplementary material ( ScalableEs.jl ) S-ES and explain the important design choices taken , such as choice of primitive reward function , the goal encoding and controller architecture . 2.1 POLICY HIERARCHY . SHES is a Feudal RL ( Dayan & Hinton , 1993 ) style method where a higher level policy set goals for a lower level policy . Dayan & Hinton ( 1993 ) use a multilevel feudal hierarchy , but similarly to HIRO , SHES uses a two level hierarchy consisting of a higher level controller policy µc and a lower level primitive policy µp . The controller is responsible for setting goals and can not directly perform actions in the world while the primitive directly controls the agent by taking actions in the world and attempts to reach the goals set by the controller . More formally , given a state st from the environment the controller produces a goal gt ∈ Rd ( where d depends on the goal encoding , in the case of this work d = 3 ) . The controller produces gt every c steps , in the interim the goal is transformed using a static function such that it is always relative to the current state , for example if gt is a vector from the agent to a target position , the static function will make sure that gt is always relative to the agents current position . The controller interval c is kept as a hyper-parameter since it has been observed that learning c often leads to it degenerating into the simplest cases where c becomes 1 or the maximum episode length ( Vezhnevets et al. , 2017 ) . This provides the controller with a level of temporal abstraction which , in the case of the environments tested in this work , allows the controller to plan a path without worrying about how the agent will follow this path . The primitive is passed the goal gt and the state st and tasked with reaching the goal . It samples an action at ∼ µp ( st , gt ) from its policy which is applied to the agent . The controller receives a reward from the environment , however it is also responsible for rewarding the primitive . As discussed in section 2.2 the primitive is sensitive to this reward and thus it should be chosen carefully . In HIRO , FuN and this work , the primitive reward is based on its distance to its goal gt ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ) , however all such previous work makes use of different rewards . In a strict feudal RL setting , rewards are not shared between controller and primitive . For example , if the primitive reaches the goal set by the controller , but this does not provide a high environmental reward then the primitive will receive a high reward , but the controller will not . Both this work and HIRO follow this strict style of primitive rewards . Even though FuN shares rewards between their primitives and controllers ( Vezhnevets et al. , 2017 ) , this was decided against as it introduces a hyper-parameter to balance primitive and controller reward scales , which can be challenging . Our method is similar to HIRO , though SHES differs in its lack of off-policy correction , it ’ s primitive goal encoding and the use of S-ES to train the primitive and controller . This has the effect of making SHES very simple to implement , the only difference between SHES and S-ES being that there are now two policies that co-evolve instead of a single policy . SHES stores a set of parameters for both the controller θc and primitive θp . Every generation it creates n new pairs of controllers and primitives by perturbing the parameters θc and θp . The perturbation is done by adding a small amount of noise sampled from an n-variate Gaussian to the parameters θci = θ c + c ∼ N ( 0 , σ2 ) . The primitive is perturbed similarly using a different noise vector which is sampled from the same Gaussian p ∼ N ( 0 , σ2 ) allowing for the sharing of common random numbers at no extra memory cost when compared to a single policy S-ES as SHES uses a shared noise table , which is one of the main contributions of S-ES ( Salimans et al. , 2017 ) . The sharing of common random numbers was shown by Salimans et al . ( 2017 ) to allow for near linear speedup when scaling up to 1000 CPU cores , which means that SHES should scale as well as S-ES , however it was only tested up to 240 cores3 . Each pair of perturbed controller and primitive are evaluated in the environment such that the controller is given a fitness equal to the cumulative environmental reward and the primitive is given a fitness of its cumulative reward from the controller . Both primitives and controllers are separately ranked and shaped according to their fitness using the same method as Salimans et al . ( 2017 ) for S-ES . This is used to approximate the gradients for the controller and primitive separately using the ADAM optimizer ( Kingma & Ba , 2014 ) and update the the controller ( θc ) and primitive ( θp ) parameters which is no different to how S-ES updates its policy . 3Each node has a Intel Xeon 24 core CPU at 2.6GHz , 120GB of RAM and nodes are connected by FDR InfiniBand Using this type of Feudal RL method the controller poses goals for the primitive who ’ s behaviour is constantly changing . This amounts to a non-stationary problem for the controller since while the controller is trying to learn the behaviour of the primitive its behaviour is changing , which can be challenging . Nachum et al . ( 2018 ) develop an off-policy correction method to combat the nonstationary problem and to allow for off-policy training and thus better sample efficiency . We found that we did not need a special method to combat this problem as S-ES robustness to noise makes this problem simpler to solve , since the primitives changing behaviour can be interpreted as noise by the controller , however this does come at the cost of sample efficiency . Algorithm 1 : SHES 1 Input : Learning rate α , noise standard deviation σ , rollouts n , initial policy parameters θc and θp 2 for t = 0,1,2 ... do 3 for i = 1,2 .. n do 4 Sample ci , p i ∼ N ( 0 , I ) 5 F ci , F p i = F ( θ c t + c i ∗ σ , θ p t + p i ∗ σ ) 6 end 7 θct+1 = θ c t + α 1 nσ ∑n i=1 F c i c i 8 θpt+1 = θ p t + α 1 nσ ∑n i=1 F p i p i 9 end

Summary: The paper presents the integration of evolution strategies with hierarchical reinforcement learning. From the natural characteristics of ES algorithms, it also can be highly scalable, so authors named the algorithm as SHES (Scalable Hierarchical Evolution Strategies). The algorithm is tested in two robot locomotion and navigation tasks.