Evolution Strategies as an Alternate Learning method for Hierarchical Reinforcement Learning
Authors: Sasha Abramowitz
Venue: ICLR 2022 Submitted (2022)
Keywords: Hierarchical reinforcement learning
evolution strategies
reinforcement learning

Abstract: This paper investigates the performance of Scalable Evolution Strategies (S-ES) as a Hierarchical Reinforcement Learning (HRL) approach. S-ES, named for its excellent scalability across many processors, was popularised by OpenAI when they showed its performance to be comparable to the state-of-the-art policy gradient methods. However, to date, S-ES has not been tested in conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. In this work, we introduce a novel method that merges S-ES and HRL, which allows S-ES to be applied to difficult problems such as simultaneous robot locomotion and navigation. We show that S-ES needed no (methodological or hyperparameter) modifications for it to be used in a hierarchical context and that its indifference to delayed rewards leads to it having competitive performance with state-of-the-art gradient-based HRL methods. This leads to a novel HRL method that achieves state-of-the-art performance, and is also comparably simple and highly scalable.


Under review as a conference paper at ICLR 2022
SCALABLE EVOLUTION STRATEGIES FOR IMPROVED HIERARCHICAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper investigates the performance of Scalable Evolution Strategies (S-ES) as Hierarchical Reinforcement Learning (HRL) approach. S-ES, named for its excellent scalability across many processors, was popularised by OpenAl when they showed its performance to be comparable to the state-of-the-art policy gra- dient methods. However, to date, S-ES has not been tested conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. In this work, we introduce a novel method that merges S-ES and HRL, which allows S-ES to be applied to difficult problems such as simultaneous robot locomotion and navigation. We show that S-ES needed (methodological hyperparameter) modifications for it to be used in a hier- archical context and that its indifference to delayed rewards leads to it having competitive performance with state-of-the-art gradient-based HRL methods. This leads to a novel HRL method that achieves ofate-of-the-ar performance, and is also comparably simple and highly scalable.
INTRODUCTION
Reinforcement learning (RL) has been used to create artificially intelligent agents for tasks ranging
from robot locomotion (Haarnoja et al. (018)) to video games such as StarCraft (Vinyals et al. 2019)
and board games such as chess and Go (Silver al. 2018). Many such agents Markov Decision
Process (MDP) based learning methods, such as Deep Q-Networks (DQNs) (Mnih et al.) 2015)
and the policy gradient family of methods (Sutton et al.. 1999a; Sutton & Barto 2018). MDP and gradient based RL methods are also used by hicrarchical RL (HRL) algorithms, which are a class of RL algorithms that excel at challenging RL. problems by decomposing them into sub tasks, which mimics the way we as humans build new skills on top of existing simpler skills. HRL methods have seen success in solving some of the hardest RL environments such as Montezumas revenge (Vezhnevets et al. 2017:: Badia et al. 2020) and generating complex robot behaviours (Nachum jet al. 2018). Another area of reinforcement learning that has enjoyed recent success is evolution strategies. These
family of black box evolutionary optimization techniques, which Salimans ctt al. (0117) showed are competitive with non-hierarchical MDP based RL methods in the robot locomotion and Atari domain. The success of such approaches has lead wider use of ES for tackling RL problems, but it has yet to be used to solve challenging HRI. problems.
ES has been used a black box optimizer for multitude of problems such minimizing the drag of 3D bodies (Beyer & Schwefel (2002), optimizing designs in structural and mechanical engineering problems (Datoussaid
al. 2006), robot locomotion (Salimans et al.] 2017] Conti al.) 2017 Katona et al., 2021) and loss function optimization (Gonzalez & Mükkkulainen (2020). There are many different flavours ES (Beyer & Schwefel 2002) each with different selection, mutation and self-adaption properties, for example CMA-ES (Hansen & Ostermeier (200T) and (1+ (Beyer & Schwefel 2002). However, in this work we are concerned with the version proposed by [Salimans et al. (2017). namely Scalable Evolution Strategies (S-ES) because of it's proven performance in the domain of robot locomotion and Atari game playing. All flavours of ES follow the scheme of sample-and-evaluate, where it samples cloud of policy variants around it's current policies parameters, evaluates these sampled policies to obtain a fitness and uses fitness to inform an update to the current policy. S-ES specifically uses fitness to approximate the gradient and moves the current policy parameters in the direction that maximizes the average reward. Given that ES is both a black- box process (making it indifferent temporal details) and is gradient free method it suffers from

---

Under review as a conference paper at ICLR 2022
sub optimal sample efficiency, however Liu et al. (019)) showed promising results addressing this inefficiency using trust regions which allow for more of a monotonic improvement.
S-ES obtained results comparable to MDP methods a set of standard MuJoCo (Todorov et al, 2012) and Atari (Mnih et al. 2013) benchmarks (Salimans et al. 2017). however there are many RL problems harder than these standard benchmarks, some of which are near impossible to solve using non-hierarchical RL (in this paper we refer to this as flat RL) and others that are unsolved using flat RL. These environments can range from games such as Montezuma's revenge (Mnih et al.) 2013) which is challenging because it requires long-term credit assignment, to robot locomotion, navigation and interaction (Nachum al. 2018 Florensa et al., 2017) which requires complex multi-level reasoning.
HRL has long held the promise of solving much more complex tasks than flat RL methods. It allows policies to abstract away large amounts of complexity and focus on solving simpler sub goals. One of the first HRL methods is known as the options framework (Sutton et al. 1999b) which allows the controller policy to select the most appropriate primitive policy from a pool of primitive policies, this primitive passes control back to the controller once its actions are completed and the process repeats. Another competing HRL framework is feudal-RL (Dayan & Hinton 1993). this framework allows for communication between the controller and primitives by having the controller set goals for the primitive to complete. Recent feudal-RL methods such as FeUdal Networks for HRL (FuN) (Vezhnevets et al.) 2017) and HRI. with Off-Policy Correction (HIRO) (Nachum et al.) 2018 have shown a lot of promise for learning sparse reward problems and hierarchies requiring complex primitives. HIRO in particular takes the approach of using a two-level hierarchy (one controller and one primitive) where the controller sets the goal and reward for the primitive. For example the goal can take the form of position an agent must reach and the reward is based on the agents distance to the goal position. HIRO, FuN and most modern HRL methods use MDP based RL methods to optimize their hierarchy of policies (Vezhnevets et al., 2017: Nachum et al., 2018 Sutton et al. 1999b Badia et al., 2020) and the best of the authors knowledge, non-MDP based RI solvers, such as ES, have not been extensively tested on hard RI problem that are typically reserved for MDP based HRL solvers.
ES has multiple advantages over MDP based RL methods, but two of these advantages make ES especially suited for HRL problems First, it is invariant to delayed rewards and second, it has more structured exploration mechanism (Salimans et al. 2017 Conti et al. 2017) relative to MDP based RL methods. Its robustness to delayed rewards is especially useful for HRL problems as much of the difficulty of these environments hard RL. problems often have many large local minima, requiring intelligent exploration methods in order to be solved. These advantages suggest that ES and specifically S-ES should perform well on challenging HRL problems, however to the best of the authors knowledge S-ES has not yet been applied to HRL problems.
come from the long term credit assignment problem. Similarly
In this paper we introduce a new method for training a two-level hierarchy of policies which are optimized using S S-ES") namely Scalable Hierarchical Evolution Strategies (SHES). It will be evalu- ated on two difficult environments, which require robust robot navigation, locomotion and planning. We compare our method's performance to other MDP based HRI. methods that have been tested on the same environments (Nachum et al., 2018) Vezhnevets et al., 2017). This paper aims to demon- strate that SHES performs well on environments that are challenging for MDP based HRL methods and that S-ES is as viable as any MDP based RL method when training hierarchies of policies. The results obtained this work show that SHES provides the HRL space with a high task per- formance, highly scalable and comparably simple HRL method. Furthermore, our method achieves these results using the same peeer-paraeeters as its flat RL counterpart.
SCALABLE HIERARCHICAL EVOLUTION STRATEGIES (SHES)
In this section we present framework for learning hierarchical policies using S-ES, which we call SHES. We show how current MDP based HRL methods needed to be adapted in order to work with
Removed for review, code can be found in supplementary material (ScalableHrilssji) 2 Removed for review, code can be found in supplementary material (ScalableFs.ji)
2

---

Under review as a conference paper at ICLR 2022
S-ES and explain the important design choices taken, such as choice of primitive reward function, the goal encoding and controller architecture.
2.1 POLICY HIERARCHY
SHES is a Feudal RL (Dayan & Hinton 1993) style method where a higher level policy set goals for a lower level policy. [Dayan & Hinton (1993) use a multilevel feudal hierarchy, but similarly to HIRO, SHES uses a two level hierarchy consisting of a higher level controller policy μ and a lower level primitive policy pº. The controller responsible for setting goals and cannot directly perform actions in the world while the primitive directly controls the agent by taking actions in the world and attempts to reach the goals set by the controller. More formally, given a state 8t from the environment the controller produces a goal 9. RR (where d depends on the goal encoding, in the case of this work d = 3). The controller produces Jr every e steps, in the interim the goal is transformed using static function such that it is always relative to the current state, for example if Ii is a vector from the agent to a target position, the static function will make sure that gr is always relative to the agents current position. The controller interval is kept as a prper-paramtte since it has been observed that learning c often leads to it degenerating into the simplest cases where c becomes 1 or the maximum episode length (Vezhnevets et al. 2017).
This provides the controller with a level of temporal abstraction which, in the case of the environ- ments tested in this work, allows the controller plan path without worrying about how the agent will follow this path. The primitive is passed the goal St and the state 8, and tasked with reaching the goal. It samples an action ag ~ ye (81, 9t) from its policy which is applied to the agent. The controller receives a reward from the environment, however it is also responsible for rewarding the primitive. As discussed in section 2.2 the primitive is sensitive to this reward and thus it should be chosen carefully. In HIRO, FuN and this work, the primitive reward is based on its distance to its goal 9r (Vezhnevets et al. 2017 Nachum et al. 2018), however all such previous work makes use of different rewards.
In a strict feudal RL setting, rewards are not shared between controller and primitive. For example, if the primitive reaches the goal set by the controller, but this does not provide a high environmental reward then the primitive will receive a high reward, but the controller will not. Both this work and HIRO follow this strict style of primitive rewards. Even though FuN shares rewards between their primitives and controllers (Vezhnevets et al. 2017), this was decided against as it introduces a hyper-parameter to balance primitive and controller reward scales, which can be challenging.
Our method is similar to HIRO, though SHES differs in its lack of off-policy correction, it's primitive goal encoding and the use of S-ES to train the primitive and controller. This has the effect of making SHES very simple to implement, the only difference between SHES and S-ES being that there are now two policies that co-evolve instead of a single policy. SHES stores a of parametens for both the controller d and primitive 0P. Every generation it creates n new pairs of controllers and primitives by perturbing the parameters 6f and 6P, The perturbation done by adding small amount of noise sampled from an n-variate Gaussian to the parameters 05 = 6 + c ~ (0, o²). The primitive is perturbed similarly using a different noise vector which is sampled from the same Gaussian e" ~ 'N, o²) allowing for the sharing of common random numbers at no extra memory cost when compared to a single policy S-ES as SHES uses a shared noise table, which one of the main contributions of S-ES (Salimans et al.] 2017).
The sharing of common random numbers was shown by [Salimans et al. (217)) to allow for near linear speedup when scaling up to 1000 CPU cores, which means that SHES should scale as well as S-ES, however was only tested up to 240 core Each pair of perturbed controller and primitive evaluated in the environment such that the controller is given a fitness equal to the cumulative environmental reward and the primitive is given a fitness of its cumulative reward from the controller. Both primitives and controllers are separately ranked and shaped according to their fitness using the same method as Salimans et al. (2017) for S-ES. This is used to approximate the gradients for the controller and primitive separately using the ADAM optimizer (Kingma & Ba 2014) and update the the controller (09) and primitive (0²) parameters which is no different to how S-ES updates its policy.
Faac node has Intel Xeon 24 care CPU at 2.6GHz, 120GB of RAM and nodes are connected by FDR InfiniBand
3

---

Under review as a conference paper at ICLR 2022
Using this type of Feudal RL method the controller poses goals for the primitive who's behaviour constantly changing. This amounts to a non-stationary problem for the controller since while the controller is trying to learn the behaviour of the primitive its behaviour is changing, which can be challenging. Nachum et al. (2018) develop an off-policy correction method to combat the non- stationary problem and allow for off-policy training and thus better sample efficiency. We found that we did not need a special method to combat this problem as S-ES robustness to noise makes this problem simpler solve, since the primitives changing behaviour can be interpreted as noise by the controller, however this does come at the cost of sample efficiency.
Algorithm 1: SHES
Input: Learning rate a, noise standard deviation a, rollouts n, initial policy parameters 6 and 4
for
0,1.2... do for i= 1,2.. do Sample c ~N(0,) I) FT FP = F0; + Ef o, 0f +
end
1
01
0 +
Σ Ffc.
n0
1
0
0%
+
Σ Li=1
P.P.e
n0
end
2.2 PRIMITIVE REWARD
The manner in which the primitive is rewarded can have a large impact the overall performance of SHES, where at the most basic level the reward needs to incentivise the agent to reach a target. In the literature there are many different ways that the primitive reward has been formulated (Vezhnevets) et al. 2017,Nachhn Nachum et al., 2018 Coumans et al. 2013), from this and our own informal experiments we' 'e found the main components of well performing primitive reward are incentive reach the target consistently and quickly while avoiding local minima.
HIRO uses the most simple primitive reward, by rewarding the primitive with its negative distance to the goal 9. (Nachum et al. 2018), this encourages the agent to move to the target quickly, however introduces a challenging local minima where the agent can simply die instantly thus avoiding accumulating anymore negative reward. FuN rewards its primitive based on the cosine similarity of the path the agent has taken since the goal was suggested and the straight line from the agent's 's position to the goal (Vezhnevets al., 2017). This encourages the agent be very consistent which makes more predictable for the controller, but this reward puts little emphasis on speed. Another possible reward which is used in both pyBullet (Coumans al., 2013) and MuJoCo (Todorov al.) 2012) locomotion environments is the agent's velocity towards the target, while this does encourage fast movement this often comes the cost of consistent paths, making it difficult for the controller to recommend positions. Rewarding the agent based on the percent of the total distance it covered (since the position was recommended) plus a bonus for reaching the target was found be the best performing primitive reward and is what SHES uses.
R = 1 de/de + (1if else 0)
where de is the euclidean distance between the agent and the goal 9: at timestep f, d, is the distance at timestep e (the most recent timestep at which the controller recommended a goal) and is a distance threshold (L 1 in the case of this work). This improves upon simply rewarding the primitive with the negative distance by: allowing it to be positive the primitive performs well thus avoiding local minima and normalizing the distance thus making it agnostic to target distance. Also, adding an extra reward for being close to the target incentivises the agent to reach the goal as quickly as possible in order maximize the amount of time it receives this extra reward.
4

---

Under review as a conference paper at ICLR 2022
Figure 1: Sensor example in the Ant Gather environment. Two black lines enclose area correlated with a position in the input vector. The closest object of interest between the black lines is given an intensity based on its distance to the agent and is added its input vector position. If no objects between two lines or within sensor range the value of the input vector at the related position is O.
2.3 THE GOAL
SHES manages the primitive goal similarly to HIRO in that new goal is recommended once every steps by the controller and for the next c steps this goal transformed using fixed goal transition function. In SHES the goal is the vector from the agent's 's position to the goal, thus the fixed transition function simply updates this goal each step given the agent's new position. However, the SHES goal differs from the HIRO goal in that it only recommends an x and position in space, whereas in HIRO the goal passed to the agent the entire state space, such that the primitive must attempt to match the position of all of the agents joints as well as the overall position of the agent. This approach is very general, but it limits types of primitive rewards that are usable and it is thus often more challenging for the primitive to learn this representation.
Similar to primitive rewards we found goal encoding to have a large impact on performance. The most obvious goal encoding to would be vector from the agent's 's position to the goal 9t- We found that this did not work since the values are not normalized, thus the primitive ANN performs worse because of non-normalized input data (Sola & Sevilla 1997). However, normalizing the goal vector comes with its own issue since the agent no longer has any notion of distance to the goal gr. We solve this by passing the primitive, the distance to the target which is scaled down to an appropriate range (by dividing it by 1000).
This simple normalized vector goal encoding can be improved upon, taking inspiration from py Bul- let's 's directional encoding (Coumans et al. 2013) we encode the primitive goal as the sin and cos of the angle from the agent to the goal gt. This was done by allowing the controller to output relative vector from the agent to the target and transforming this vector into angle from the agent to the target, the sin and cos of this angle the goal St that passed to the primitive. Similar to the normalized vector encoding this also provides the primitive with notion of distance to the target and as such we pass the scaled distance to the target along with this goal encoding. This angle encoding can be seen as normalization step, while also encoding more intuitive information about the direction to the goal.
2.4 ONE HOT CONTROLLER
Using a one-hot inspired encoding for the output of the controller helps simplify many problem- spaces via matching the output of the ANN controller to part of its input. To explain the one-hot inspired encoding consider the output of the controller network (n²) is vector o E Rd such that each element of corresponds to one of d equally spaced angles around the agent, this can be seen as the black lines in figure [ In the same style of one-hot encoding popularly used as the output of image classifying convolutional neural networks (Potdar et al.) 2017) the element with the highest value on, is selected and used to determine the point that the controller will recommend. To use figure [ as an example d would be 10 and each element in the output vector would correspond
5

---

Under review as a conference paper at ICLR 2022
Figure 2: Ant Gather environment (left) and Ant Maze environment (right). The two environments used to evaluate the performance of SHES. In the gather environment the agent must learn to collect as much green food as possible while avoiding the red poison (the pink dot indicates the primitive goal Ir and the blue dot is the agents starting position). In the maze environment the agent must learn to reach the red dot on the other side of the maze.
to one of the black lines. If each line is k units long, given the highest output on E [-1, the controller recommends the point оъ k units along the relevant line, where k is papra-parreeter representing the maximum distance from the agent that the controller can recommend.
This isn't 't vital part of SHES, but is especially relevant to the environments tested and can be used eectteenviromment sensory interactions. This ANN architecture choice only works well for environments with sensors, since the input to the ANN is similar to its output. For example, if the agent observed a vector representing absolute positions of objects (instead their sensor readings) this style of ANN architecture would not perform well. Only uses agent sensors and as such the one hot encoding is only used for this environment. When one-hot encoding is used the controller simply outputs a vector representing the relative (æ, y) position from the agent's ' current position that the agent should move towards.
of the two environments tested
EXPERIMENTS
SHES was evaluated on two different environments that have sparse rewards and require both robot locomotion and navigation. The first environment is called Ant Gather and the second And Maze, an example of both can be seen in figure 2
The agent given positive reward for each green food collects and a negative reward for each red poison it collects. Food and poison are generated in random positions each time the environment is initialized.
Ant Gather:
The agent must leam to move to the opposite side of the maze. During training the agent must reach randomly generated target inside the maze and rewarded based on the negative distance to this target. During testing the agent must come sufficiently close to the red dot on the ultimate step of the episode to receive score of otherwise it gets
Ant Maze:
To compare SHES we use results from Nachum et al. (2018). since this paper collected results on many different HRL methods such as HIRO (Nachum et al. 2018), FuN (Vezhnevets et al. 2017). Stochastic Neural Networks for HRL (SNN4HRL) (Florensa et al. 2017) and Variational Infor- mation Maximizing Exploration (VIME) (Houthooft et al. 2016). Results from these environments taken after the agents had trained for 10 million steps. The environments used by Nachum et al. (0188) were originally written in python, but for this work have been faithfully recreated (using the same assets and physics engine) in Julia for computational speed-up purposes.

---

Under review as a conference paper at ICLR 2022
Ant Gather
Ant Maze
SHES (100)
0.79 + 0.34
0.0 = 0.0
SHES (600)
2.88 + 0.61
0.24 + 0.10
SHES one-hot (100)
1,48 = 0.34
n/a
SHES one-hot (600)
3.68 = 0.60
n/a 0.99 + 0.01
HIRO (10)
3.04 + 1.49
FuN cos similarity (10)
0.85 + 1.17
0.16 + 0.33
SNN4HRL (10)
1.93 + 0.52
0.0 + 0.0
VIME (10)
1.42 + 0.90
0.0 + 0.0
Table 1: Task performance of comparative methods on Ant Gather and Ant Maze. Performance is the average of 10 randomly seeded trials with standard error. Values in parentheses are millions of training steps. One-ho hot encoding was not used for agents-environmen 2.4 Performance of all methods that are not SHES are taken from Nachum et al. (2018).
without sensors (section
Ant Gather: mean controller reward
Ant Gather: mean primitive reward
150
Jumulasive reward
100
Cumulaatve
410
Generation
Generation
Ant Maze: mean controller reward
Ant Maze: mean primitive reward
2000
2300
Cumulative
1200
Cumulaatvv
500
0
200
400
900
0
200
406
600
Generation
Generation
Figure 3: Cumulative training reward per generation over 10 SHES runs (mean of the mean). Shaded area shows standard error. X-axis shown generations, not total steps.
4 DISCUSSION
Results in table indicate that SHES performs suitably well, however the difference in sample ef- ficiency is also quite clear. At 600 million steps SHES needs 60 times more samples than gradient based methods. The lack of sample efficiency is not unexpected as gradient free optimization is often less efficient than gradient based methods (Sigaud & Stulp, 2019). Specifically, the original S-ES paper (Salimans al.] 2017) saw sample efficiency up to 7.88 times worse than TRPO on simple 2D locomotion environments. Thus, S-ES was already less sample efficient than on-policy gradi- ent based method, meaning that the sample efficiency difference would be more exaggerated when compared off-policy algorithm such as HIRO. The 7.88 times more samples required were measured on simple 2D locomotion environments, when moving to the more challenging 3D Ant Gather and Maze environments this only amplifies the sample efficiency difference. Nonetheless
Removed for review, code can be found in supplementary material (HriMuulCooEnssi))
7

---

Under review as a conference paper at ICLR 2022
it doesn't 't mean one should not use SHES or S-ES. In simulated learning applications, task perfor- mance, learning speed, learnt behavioral robustness and the simplicity of SHES more than makes up for its sample inefficiency.
One of the main ways that SHES makes up for it's lack of sample efficiency is it's overall task performance, having state-of-the-arr mean score Ant Gather of 3.68 when using one-hot encoding and without one-hot encoding being within one standard deviation of HIRO (the previous state-of-the-art) (Nachum et al. 2018). This clearly shows the benefit of the one-hot encoding, allowing for an easier representation for the controller to learn and improves upon the already good results from the non-one-hot version of SHES. Also, task performance on Ant Maze still achieves a respectable score given the difficulty of the problem. Results analysis indicates that learning starts quite late in the process, having mean evaluation score of at 100 million steps and 0.24 600 million steps. This is likely because during training, target positions are placed randomly and thus the agent does not get many samples of targets close to the evaluation goal after it has learned to walk around the corner. Along with its state-of-the-art performance, even after 600 million steps, SHES is still able to continue learning. This is supported by figure + showing a clear trend upwards for both the primitive and controller Ant Gather. Also, results in table 4 indicate that for SHES applied to Ant Maze, learning also starts late, thus indicating that SHES has the capacity to continue improving late into the learning process. This is in contrast to HIRO which learns and plateaus rapidly (Nachum et al. 2018).
Despite the fact that SHES requires many samples, for all tests, 600 million steps never took longer than 12 hours on a 24 core CPU and given the simplicity of SHES it is very easy to parallelize its computation across multiple CPUs in order to obtain even more speed-up. The ease of parallelization along with the fact that S-ES needed no parammee or methodological changes in order to fit into SHES shows its robustness for all types of RL tasks and frameworks. Another area where SHES is exceedingly robust is with respect to the non-stationary learning problem that the controller faces. As can be seen in 4 for both environments the controller has very few dips in task performance meaning that it easily deals with the ever changing nature of the primitive. Thus, overall, results demonstrate that SHES is a robust HRL method which is able to achieve state-of-the-art performance on HRL tasks, while maintaining excellent scalability, and being comparably simple to implement, and needing no typer-parameter modifications from its flat RL counterpart.
5 CONCLUSION
We have presented novel method (SHES) for using S-ES in two-level hierarchy which allows it to solve complex RI. tasks requiring both robot locomotion and navigation. SHES achieved compet- itive performance with state-of-the-art methods all environments tested. SHES needed no hyper- parameter tuning and demonstrated fast and scalable performance, providing viable gradient free learning method for hierarchies of policies on challenging RL tasks. To the best of the authors' knowledge, this is the first time a hierarchy of ES has been used to solve such challenging RL tasks. Future work will include combining SHES with work done by [iu et al. (2019) improve sample efficiency and testing it on more environments encapsulating a broader range of RL task types.
REFERENCES
Adri Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR,
PMLLR 2020.
Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies a comprehensive introduction. Natural Computing, 1:3-52, 2002. ISSN 155677818. doi: 10. 10.1023/::00555599284488
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-seeking agents. Advances in Neural Information Processing Systems, 2018 Deweeeeer::0027-55005 12 2017.
Erwin Coumans al. Bullet physics library. Open source: bulletphysics. org, 15(49):5, 2013.
8

---

Under review as a conference paper at ICLR 2022
Sélim Datoussaid, Guy Guerlement, and Thierry Descamps. Applications of evolutionary strategies in optimal design of mechanical systems. Foundation of Civil and Environmental Engineering, 7: 35-51, 2006.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume 5. Morgan- Kaufmann, 1993.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint arXiv:1704, 03012, 2017.
Santiago Gonzalez and Risto Miikkulainen. Improved training speed, accuracy, and data utilization through loss function optimization. In 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8. IEEE, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Nikolaus Hansen and Andreas Ostermeier: Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):199-19)) 2001.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in Neural Information Processing Systems, 29:1 109-1117. 2016.
Adam Katona, Daniel W Franks, and James Alfred Walker. Quality evolvability es: Evoly- ing individuals with distribution of well performing and diverse offspring. arXiv preprint arXiv:2103. 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: pet..:/22...00 2014.
Guoqing Liu, Li Zhao, Feidiao Yang. Jiang Bian, Tao Qin, Nenghai Yu, and Tie Yan Liu. Trust region evolution strategies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4352-4359. AAAI Press, 7 2019. ISBN 9781577358091. doi: 10.1609/aaai. v33101.33014352.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature. 5 51877500):59-5533 2015.
2015.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31:3303-3313, 2018.
Kedar Potdar, Taher S Pardawala, and Chinmay D Pai. A comparative study of categorical variable encoding techniques for neural network classifiers. International journal of computer applica- tions, 175(4):7-9, 2017.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv:/703.03380 3 2017.
Olivier Sigaud and Freek Stulp. Policy search in continuous action domains: an overview. Neural Networks, 113:28 40, 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140- 1144, 2018
Jorge Sola and Joaquin Sevilla. Importance of input data normalization for the application of neural networks to complex industrial problems. IEEE Transactions on nuclear science, 44(3): 1464- 1468, 1997.

---

Under review as a conference paper at ICLR 2022
Colin Summers, Kendall Lowrey, Aravind Rajeswaran, Siddhartha Srinivasa, and Emanuel Todorov. Lyceum: An efficient and scalable ecosystem for robot learning. 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, David McAllester, Satinder P Singh, Yishay Mansour, al. Policy gradient methods for reinforcement learning with function approximation. In NIPs, volume 99, pp. 1057- 1063. Citeseer, 1999a.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999b.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1 2...109//OSS.222..366660
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 3540-3549, PMLR, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnecki Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3550 2019.
A APPENDIX
A.1 ENVIRONMENT DETAILS
Environments were implemented using Julia's wrapper for MuJoCo, Lyceum (Summers et al. 2020).
The implementation parameters were taken from the HIRO paper appendix (Nachum et al. 2018) and from their repository Similarly to HIRO we set the simulator to have a of 0.02 and frameskip of 5.
A.1.1 ANT GATHER
This environment uses modified version of the most modern ant robot to match the one used by HIRO. In addition is allowed to observe all gpos and qvel, the current timestep and the extra sensor observations so that it can observe the food and bombs.
Episodes are terminated when the ant falls or 500 steps.
The reward is simply the number of apples collected minus the number of bombs collected.
A.1.2 ANT MAZE
The same slightly modified ant is used for this environment. It observes all gpos and quel the current timestep and the target position Ty. The agent placed in a U-shaped maze and each episode must reach randomly generated targets inside the maze. This is done slightly differently to the HIRO implementation, as that implementation positions are sampled that could potentially be unreachable by the agent, in this implementation all sampled positions are reachable. During evaluation the agent must reach specified position on the opposite side of the maze. The reward each steps corresponds to the negative 12 distance from the agent to the target.
For this environment the episode only terminates 500 steps.
http::///..iiiiiceeeooeooooeoeen efficient-hri
10

---

Under review as a conference paper at ICLR 2022
A.2 IMPLEMENTATION DETAILS
A.2.1 NETWORK STRUCTURE
The same network structure as S-ES is used, which is a simple feed forward neural network with one hidden layer of 256x256 units and tanh activation. When the one-hot encoding is not used the outputs of the controller are scaled by the controller distance hyper-parameter.
A.2.2 HYPER- PAREAE-RR
Note that for all parameters relevant to S-ESs original implementation we use the same values, namely noise table size, a, action noise standard deviation, learning rate and 12 coefficient.
episodes per policy:
policies per generation: 256
controller interval: 25
controller distance: (the max distance from the agents position that the controller can recommend goal)
noise table size: 250000000
(standard deviation of Gaussian policy perturbing noise): 0.02
action noise standard deviation: 0.01
learning rate: 0.01
12 coefficient: 0.005
A.2.3 IMPACT OF CONTROLLES ER INTERVAL.
The controller interval e or the frequency with which the controller recommends positions to primitive is sensitive parameter and can have large impact the performance of the overall system. high interval (200) does not give the controller a lot of control, but does allow the primitive to learn easily, in contrast low interval (5) gives the controller fine grained control, but is too difficult task for the primitive to learn. practice a balance must be struck between these two extremes where the controller has enough control effect the outcome and the primitive is able to learn to reach the goals set by the controller. We found 25-50 to be an ideal range for c, however Nachum nowever/Nachum et al. (2018) used c 10 which did not work for SHES. Thus the setting of this
hyperparameter cannot be generalised to any feudal RL system and our recommended range only pertains SHES.


Reviews:
[{"review_id": "tqRNAOLkho", "reviewer": "Reviewer_zZsS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- As the authors highlight, ES is invariant to the timescale of rewards. This is indeed a useful property in HRL, and one that should be exploited further.\n- As per the previous point, the paper was an interesting reading.\n- Performance on the tested environments is promising.\n- <informal on> I am a big fan of ES. :) <informal off>\n\nWeaknesses:\n- Evaluation is limited to just two environments. In particular, it would have been interesting to see a comparison of the various methods on traditional (pre-deep) HRL environments, and in general on a larger breadth of tasks.\n- Overall, the degree of novelty of the work presented seems limited, as the method presented is basically a standard Feudal RL setup optimized via ES instead of gradient-based methods.\n- Comparison of the final performance of the proposed method vs the competing methods seems a bit biased, as the competing methods were only trained for 10 million steps, while the method from the authors was trained for 600 million steps.\n   While the problem of sample efficiency of ES is a valid point, it is not clear whether the competing methods would have achieved comparable or higher performance if they were trained on the same amount of data.\n\n\n", "summary_of_the_paper": "The authors propose a method for Hierarchical Reinforcement Learning based on the OpenAI variant of Evolution Strategies.\nThe proposed method follow the HRL approach of Feudal RL, implementing two controllers at different levels of abstraction: a master controller outputs goals (every number of steps) that the slave controller learns to achieve The two controllers receive different rewards to incentivize each to learn its task.\n", "summary_of_the_review": "The main issue with the paper is that the degree of novelty seems limited, as it seems that in practice the proposed method is just Feudal RL optimized via ES instead of gradient-based methods. Also evaluation is insufficient.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "iadWG6QANxi", "reviewer": "Reviewer_BgoU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengthes:\n* Simple search strategy for hierarchical policies allowing highly parallel implementation\n\nWeaknesses:\n* Quality of the evaluation of the proposed approach is below the standard of the community. (1) The comparison in Table 1 is rather unfair. The authors compared the performances of variants of the proposed approach after 100 or 600 million steps, with the performances of existing approaches after 10 million steps. (2) The two tested environments are similar, and do not support the generality of the proposed approach. It is also unclear whether these environments are representative for the environment said to be \"hard\" by the authors repeatedly in the introduction. (3) Ablation studies are missing.\n* Novelty is questionable. The search strategy is almost the same as [Salimans et al, 2017]. The only difference is that the fitnesses for the controller policy and for the primitive policy are different. The fitness of the primitive reward is rather similar to existing approaches except for a way to normalize it. The contribution of this proposed primitive reward is not evaluated in experiments, hence unclear. \n* Clarity of the statements should be improved. (1) F in Algorithm 1 is not explained in detail. (2) The authors repeatedly says \"challenging problems\" and \"hard problems\", but it is not clearly stated what kinds of difficult tasks the authors are targeting. (3) The authors claims that ES is invariant to delayed reward. But it is not clear what is meant by it. (4) The authors say that \"hard RL problems often have many large local minima\" and ES is advantageous for this perspective. This might be true for the standard ES, but for the one used in this paper, the authors set the noise std for the parameter search to be 0.02, meaning that this approach do not take into account so much global information of the objective function.", "summary_of_the_paper": "This paper proposes a novel approach for hierarchical reinforcement learning. As the title says, the authors claims that the use of evolution strategies is useful to train hierarchical policy. Basically, the search is based on the estimation of the gradients of the controller fitness and the primitive fitness using the gaussian samples around the current parameters. It is mostly the same as [Salimans et al, 2017]. The authors also introduce the primitive reward. The performance of the proposed approach is evaluated on two test environments.", "summary_of_the_review": "Based on the above review, the novelty and the significance are questionable. The statements should be improved for clarity. The experimentation should be improved to support the claim of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fOYFEswnPuT", "reviewer": "Reviewer_xRrU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work presents the novel algorithm which applies ES to the HRL scheme. The ES part shares key concepts and features with the previous work, and the hierarchy part uses a two-level hierarchy similar to the previous works. \n\nStrengths\n- The combination of ES and HRL is novel and interesting\n- The proposed algorithm is straightforward and seems easy to implement\n- Experimental results are comparable to the previous algorithms\n\nWeaknesses\n- The core idea consists of two main parts, similar to previous works, S-ES and HRL. \n- Experiments are only conducted in two environments, some results are not compatible with previous algorithms. \n\nThis paper is well-organized and easy to follow. Their claims are reasonable but seem not fully supported by the experiments. Also, as the authors pointed out, the algorithm needs much more samples to achieve similar results. Future work will enhance the sample efficiency and test in more environments. \n\nOne of the main contributions of this paper is to show the effectiveness of ES in HRL problems. The authors provide the experimental results but I think it is not enough to show the effectiveness. They present the results in two environments, \"Ant Gather\" and \"Ant Maze\". In \"Ant Gather\", the algorithm shows comparable results with previous works, although it requires 10x to 60x training steps. However in \"Ant Maze\", the result of \"SHES one-hot\" is not available, and \"SHES\" shows low performance. Also, the closest previous work, HIRO, presents two more environments, but those environments are not shown in this work. ", "summary_of_the_paper": "The paper presents the integration of evolution strategies with hierarchical reinforcement learning. From the natural characteristics of ES algorithms, it also can be highly scalable, so authors named the algorithm as SHES (Scalable Hierarchical Evolution Strategies). The algorithm is tested in two robot locomotion and navigation tasks. \n", "summary_of_the_review": "Although the proposed methods are novel, it seems that the experimental results need to be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O0Gwj5y4K6a", "reviewer": "Reviewer_up7h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles an important research direction of reinforcement learning. While the paper is clearly written, I have a few concerns and suggestions.\n\n1) This paper misses an important prior work [Jain et al. 2019]. This prior work also applies evolution strategy (Augmented Random Search) to hierarchical learning. The prior work required less prior knowledge: the interface between the high and the low levels is learned, and the update frequency between the high and the low levels is also learned automatically. In this paper, both are specified manually. In addition, the prior work is validated on a real robot while this paper is tested in simulation. To improve the quality of this paper, a thorough discussion and quantitative comparison with [Jain et al. 2019] is needed.\n\n2) The result does not significantly outperforms the state-of-the-art (HIRO). From Table I, In Ant Gather, the proposed algorithm achieve slightly better result but uses 60 times more samples. In Ant Maze, HIRO's performance is far better, even with only 1/60 of samples needed. The paper claims that ES is more scalable. To support this claim, a direct comparison of training time (wall-clock) between HIRO and ES should be provided.\n\n3) The paper calls policy gradient methods as MDP methods multiple times in the Introduction. MDP is a problem formulation, and is not a solution. This paper is also using MDP formulation, but applies a derivative-free optimization solver. I believe what the paper intends to say is \"policy gradient methods\" that uses back-propgation to calculate gradient.\n\n4) For Figure 3, it would be nice to plot learning curves of all baselines in same figures for easy comparison. If the curves of baselines are too short due to sample efficiency, it is OK to plot the curves with wall-clock time as the x-axis since an argument in this paper is that ES algorithms are more parallelizable and thus take less wall-clock time to train.\n\nReference:\nJain et al., Hierarchical Reinforcement Learning for Quadruped Locomotion, 2019", "summary_of_the_paper": "This paper proposes to apply evolution strategy to hierarchical reinforcement learning. The high-level controller sets goals and the low-level primitive learns to move to the goal. The high-level and the low-level have different reward functions. The paper applies evolution strategy to solve such a problem. The proposed method is evaluated on two tasks: Ant Maze and Ant Gather. It outperforms the baselines in one of the experiments.", "summary_of_the_review": "This paper proposes a straightforward application of evolution strategy on hierarchical learning. The novelty is limited. It misses an important prior work [Hierarchical Reinforcement Learning for Quadruped Locomotion, Jain et al. 2019], which also uses ES (ARS) on hierarchical learning for locomotion. The prior work seems to achieve better results than this paper. In addition, comparing to HIRO, it is hard to conclude that the proposed algorithm outperforms the state-of-the-art.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents the use of scalable evolution strategies (S-ES) in hierarchical reinforcement learning. After reviewing the paper and reading the comments from the reviewers, here are my comments:
 
- The proposal is quite novel. It requires major improvements to clearly state how this proposal contributes in the field.
- The main concern is about the experimental results. There are some flaws in the comparative results. Also, they do not support the proposal.
Decision: Reject