Efficient and Stable Off-policy Training via Behavior-aware Evolutionary Learning
Authors: Maiyue Chen
Guangyi He
Venue: CoRL 2022 Poster (2022)
Keywords: Continuos control
Reinforcement learning
Evolution strategies

Abstract: Applying reinforcement learning (RL) algorithms to real-world continuos control problems faces many challenges in terms of sample efficiency, stability and exploration. Off-policy RL algorithms show great sample efficiency but can be unstable to train and require effective exploration techniques for sparse reward environments. A simple yet effective approach to address these challenges is to train a population of policies and ensemble them in certain ways. In this work, a novel population based evolutionary training framework inspired by evolution strategies (ES) called Behavior-aware Evolutionary Learning (BEL) is proposed. The main idea is to train a population of behaviorally diverse policies in parallel and conduct selection with simple linear recombination. BEL consists of two mechanisms called behavior-regularized perturbation (BRP) and behavior-targeted training (BTT) to accomplish stable and fine control of the population behavior divergence. Experimental studies showed that BEL not only has superior sample efficiency and stability compared to existing methods, but can also produce diverse agents in sparse reward environments. Due to the parallel implementation, BEL also exhibits relatively good computation efficiency, making it a practical and competitive method to train policies for real-world robots.

Efficient and Stable Off-policy Training via Behavior-aware Evolutionary Learning
Maiyue. Chen School of Aritificial Intelligence Peking University, China
Guangyi. He
Department of Medical Bioinformatics School of Basic Medical Sciences Peking University, China
Corrsponding author mychen@pku. edu.
Abstract: Applying reinforcement learning (RI.) algorithms to real-world contin- control problems faces many challenges in terms of sample efficiency, stability and exploration. Off-policy RL. algorithms show great sample efficiency but can be unstable to train and require effective exploration techniques for sparse reward environments. A simple yet effective approach to address these challenges is to train population of policies and ensemble them in certain ways. In this work, a novel population-baae -baee evolutionary training framework inspired by evolution strategies (ES) called Behavior aware Evolutionary Learning (BEL) proposed. The main idea is to train population of behaviorally diverse policies in parallel and conduct selection with simple linear recombination. BEL consists of two mechanisms called Shhavior-regularizer perturbation (BRP) and behavior-targeted training (BTT) to accomplish stable and fine control of the population behavior divergence. Experimental studies have shown that BEL not only has superior sample efficiency and stability compared to existing methods but can also produce diverse agents in sparse reward environments. Due to the parallel implementation. BEL also exhibits relatively good computation efficiency, making a practical and competitive method to train policies for real-world robots.
Keywords: Continuous control, Reinforcement learning, Evolution strategies
Introduction
Recent advances in reinforcement learning (RL) have proven that off- off-policy deep reinforcement learning (DRL) algorithms possess great potential in solving continuous control problems, especially in terms of sample efficiency [1] [2] [3]. However, off-policy algorithms are also known to be unstable or brittle [4] [5], which to some extent hinders the application of these algorithms real-world robots. On the other hand, environments with sparse rewards that are common in real-world scenarios also present challenges in terms of exploration.
Along with DRL algorithms, another line of direct policy search methods called evolutionary algo- rithms (EA) also showed significant success as a result of the improved computational efficiency of modern hardware and clever implementations [6] [7] [8]. Different from DRL algorithms that exploit the sequential structure the Markov Decision Process (MDP). EA algorithms treat policy search as black-box optimization problem and utilize population of randomly perturbed policies to search for better policies. While being less sample efficient, EA methods tend to enjoy properties such as improved stability, efficient parallelization and better diversity due to the utilization of the EA population and FA operations.
Naturally, combining those two paradigms to obtain the best of both worlds has attracted much efforts over the years [9] [10] [11] [12] [13]. The motivation behind these works to inject the gradient-trained DRL agents into the population and drive the population with policy gradient signals while enjoying the benefits of the EA population. Another perspective is that EA mutation can serve as parameter space noise and improve the exploration ability of RL agents [14]. Such a combination turned out to very successful, and the resulting hybrid algorithms can beat both of their EA and DRL. components.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

One of the key factors resulting in the success of the hybrid algorithms is that the EA population provides a way to maintain the behavioral diversity of the policies [15]. Behavioral diversity, which is usually called phenotypic diversity in Evolutionary Robotics (ER) [16] [17], refers to the differences between outputs of the policies within the population here. Intuitively, diversity helpful for environments with sparse or deceptive reward signals. providing higher chance of exploring potentially good states [18]. The improved exploration ability also helps the off-policy learning procedure by filling the replay buffer with more diverse transitions [14]. In addition, in a multi- objective scenario, higher behavioral diversity may result in better coverage of different modalities, e.g., training cheetah robot to run both forward and backward [15].
However, we identified two important problems that remain unsolved by previous EARL methods:
The first problem is how can the policy be randomly perturbed in a meaningful and controlled manner? A perturbation that is too small would result in no significant changes, while a perturbation that is too large could lead to divergent training.
The second problem is that although perturbed networks result in different policies, once policies undergo the same RL training process and sample from the sample replay buffer, they may end up being similar and reduce the overall diversity. Ideally, we would like to have a population that is not only diverse after random perturbation but also diverse after RL training.
With the output actions of a policy under different states treated as its behavior, we aim to measure the population's 's diversity as the mean behavior divergence (action space distance) between each individual policy and the center policy. The main idea of BEL is to maintain the population diversity by solving the above two problems. To solve the first problem, inspired by previous works [19] [12]. Perturbation (BRP) is proposed, which can randomly perturb a policy network within specified behavior divergence range in an online fashion. To solve the second problem, Behavior- Targeeee Training (BTT) is proposed to assign a randomly sampled target behavior diver- gence and inject it into the actor training process, which shares certain similarities with goal-directed exploration [20]. The final proposed training framework is called Behavior-aware Evolutionary Learning (BEL). In BEL, training is conducted a generational fashion that closely resembles the traditional evolution strategies (ES). In each generation, first, policies are randomly generated by applying BRP to a central mean policy. Then, those offspring policies are trained in parallel with BTT. Finally, all trained policies undergo weighted linear combination [21] and form the new center policy for the next generation.
2 Related works
Model-free off-policy RL algorithms are a class of sample efficient algorithms for continuous control tasks with relatively high dimensional action spaces [1] [22]. Built upon the actor-critic (AC) paradigm, where a pair of actor networks and critic networks are trained simultancously. the Twin Delayed Deep Deterministic (TD3) algorithm [2] and the Soft Actor-Critic (SAC) algorithm [3] showed great success, and quickly became the go-to algorithms for sample efficient RL training.
Evolutionary algorithms have also gained attention due to the fact that they prove to be competitive alternatives to MDP-based methods [6] [7]. In [6], the authors developed a simplified natural evolution strategies (NES) [23]. The resulting OpenAl ES offers massive scalability while matching the performance of MDP-based methods. In [7], it was shown that the genetic algorithm (GA) was able to evolve networks with four million parameters and achieved competitive performance compared to gradicnt- based methods.
Combining evolutionary methods and policy gradient- based methods in order to benefit from the best of two worlds soon attracted much attention when [9] first proposed to evolve a population of agents with GA and periodically inject gradient information into the population. Their resulting algorithm ERL outperformed both GA and Deep Deterministic Policy Gradient (DDPG). In [11], the authors managed to use a variant of ES called the cross entropy method (CEM) to evolve the population half of which was composed of EA agents, and the other half was composed of TD3 agents. Their hybrid algorithm CEM-RL turned out to be very competitive and served as strong baseline for derivative works. Later, [12] pointed out that traditional crossover and mutation operators widely used in GA can be detrimental in the sense that they could destroy learned behaviors. As remedy, [12] proposed

---

to conduct crossover with network distillation and mutation with SM-G-SUM [19] which proved to be able retain learned network behaviors.
3 Background
3.1 Evolution Strategies (ES)
Evolution strategies (ES) belongs the gradient-free black-box optimization algorithm family. was first proposed by Rechenberg [24], and later developed by Schwefel [25]. Mimicking the natural evolution process, ES randomly generates a population of solution vectors (usually with a Gaussian distribution) whose fitness value will be evaluated in problem-specific manner (for example episodic reward). In its canonical form, ES can be classified into two major versions: the (p, A) ES. where parents of the next generation are selected from the current À offisrrng, and the (1 + A) ES, where the selection pool contains both the current parents and offspring. The selection operation is usually simple weighted linear recombination of the population vectors according to their fitmess ranks. In this work, we adopt the simplest (1, A) ES scheme, and model the population with uniform distribution in terms of behavior divergence.
3.2 Twin Delayed Deep Deterministic Policy Gradient (TD3)
As RL algorithm, TD3 is built upon the Markov Decision Process (MDP) which is described by <. S, A, P, R. > In this formulation, S is the state space, A is the action space, P is the transition function, R is the reward function and 7 is discount factor [4]. The goal is learn an optimal policy function π to maximize the expected return J(0) = Ex-ppe.n [Ro)- TD3 solves this problem by adopting the actor-critic deterministic policy gradient [26] [1] where Q-function Qo is learned through the Bellman equation:
(s a) = r + yB,r,a' 1Q" ('', d'~ x (5')
Then the policy function Ta is optimized by the deterministic policy gradient:
VoJ(O) = E.-pt (s,"(s,a)) (s)=)))))())) Vozo(s)
(2)
For implementation, both Qo and are optimized with Monte-Carlo estimation with the help of replay buffer D, the loss function of Q₂ and mp are defined as follows:
( - (r (
(Ro(s..0)
CD3 LQ+
E (B.0)))))
max
(3)
E (s,a) ] a~00
CTD - SND
(4)
In TD3, three tricks are used to make the above learning process more stable and alleviate the overestimation bias. The first trick is to learn two Q functions and uses the smaller Q-value to form the target Q in eq. (3). The second trick is to delay the target networks updates with regard to Q network updates. The third trick to add noise to target actions to smooth out Q along changes in action [2].
4
Behavior-aware Evolutionary Learning (BEL)
4.1 BEL framework
The overall structure of BEL is outlined in fig. 1. In BEL, we maintain a center actor as the population center, and À actors as offspring. In each generation, first, all offspring actors will be initialized around the center actor with Behavior-Regularizee Regularized Perturbation smoothing (BRP), which will be introduced in detail in section 4.2. Then each offspring actor will undergo (BTT) as will be described in section 4.3. After these two processes, all offspring actors will interact with the environment and save their experiences to the replay buffer. Finally, the population selection is conducted with a weighted linear recombination of network parameters according to episodic rewards of the trained offspring actors to form the center actor for next generation. This process repeated until termination criterion is met.
Training
3

---

Exscution
Center Policy
Direction
Behavior-Rage Raguuaii-Ragularizzdd Perturbation (BRP)
Sample
Replay
Buffer
Sample
Mase Deraniton
Actor
Actor
Actor
Actor
Parallel
Parallel training
Paralllll
traning
trainingg
Critic
Crilic
Critic
Crilic
Interact
Bahavior-Targelta Training (BTT)
Weighted Linear
Recombination
Eveluit
Center Policy
Figure 1: An illustration of the BEL framework.
4.2 Dehavior-Regularizze Perturbation (BRP)
Similar to the SM-G-SUM mutation operator used in [12] and [19], BRP relies on the calculation of the so-called parameter sensitivity with regard to network outputs. Given actor network To and batch transitions BRP approximately measures how the overall output will vary with regard to small changes of the neural network's 's weights 0 through the aggregation of backward gradients of each output node i on data batch i. For each parameter in To. its sensitivity sens is calculated by:
SemIS Σ
Σ Vone (si)a
(5)
A large value of sens indicates that the corresponding parameter will lead to large change of the action output and vice versa. Denoting the overall sensitivity for all parameters as Sens, it is then used the coefficient of the following linear transformation:
6
Vec(#o)
Vec(mT)
Sens
In eq. (6), Vec(Ta) means network parameters represented as one-dimensional vector. åe is the perturbed policy network and is random vector that determines the perturbation magnitude and direction.
Unlike previous methods [27] [12] where 8 is randomly sampled from constant-scaled Gaussian distribution, the BRP instead tries to adaptively search for proper 8 within a certain magnitude that bound the behavior divergence of the perturbed network. This idea is similar the parameter noise adaptation method in [14].
To measure the behavior divergence in a continuous action space, BRP adopts the simple and widely adopted Euclidean norm as the distance metric [28] [29] [14]:
((TT(s),
Σ E, (To(s) 7o (s)))²
N
Intuitively, this metric is sensitive to large deviations and can tolerate small displacements, which is suitable for common cases. When the action output is not deterministic, such as SAC, the relative entropy measure is also applicable [30] [16].

---

Algorithm Behhavor-Reegllarzzed Perturbation
1: Input: Population center policy ио. population size X, error bound initial magnitude scalar imit. a batch states 8, and / E [0,
2: Calculate Sens for Ty according to eq. (5)
3: for prri= to À do
4: Sample target divergence A, Ui0 ) sample random direction from ó ~ N(
5: Get initial perturbed network क; with Sens according to eq. (6)
6: while (s), T. (s)) A do
7:
if (s), T. (s)) < A, then
8:
3
9:
else
10:
5 = 83
end
12: Get perturbed network क, with Sens and 8
end while
14:
end for
15:
Output: Perturbed policies { 1,... }}
Given a behavior divergence upper bound A generate one perturbed network म5 BRP first randomly samples divergence Å, and then conducts a simple iterative line search to find proper 5, detailed procedure summarized in algorithm 1. The final output is set of randomly perturbed policy networks following a uniform distribution in the behavior divergence space. Note that unlike previous implementations which only approximately calculated Sens, our implementation precisely calculated Sens with the help of Pytorch hooks. Generating five perturbed networks can be done within 0.2 seconds.
4.3 Tergeeer-Targeted Training (BTT)
BRP generates policy networks through random local perturbation, BTT on the other hand generates trained policies that are within behavior divergence range to the center policy. Consider one offspring policy n. generated by BRP, we would like its behavior divergence after training A A mained to lie in the range defined by an upper bound: A tained
[0, A B77
Negative likellhood
13
23
Behavior divergence to the center policy
Figure 2: An illustration of the Behavior-targeted training (BTT).
To achieve this goal, we gained inspiration from imitation learning. In each generation of policy gradient training, the actor network aims to optimize two objectives. The first objective is the traditional RL objective as in eq. (4), For the second objective, consider batch of states s sampled from the replay buffer, the actions of both the center policy कe and the offspring policy स; those states are calculated ag = TO(s) and = (s). Then depicted in fig. 2, we construct a behavior potential well with one dimensional Gaussian distribution force the negative higlliill of the Euclidean distance between ag and a; to stay close to the bottom of the Gaussian whose mean
5

---

is defined by a sampled and fixed A A standard deviation is defined by predefined ØBTr- This process results in the following training objective for BTT:
de (a1) -4 ad000
BIT R
TDS = 8
a aln
e
20grr
(8)
ØBTT 2m
Following eq. (8), the policy network will try to simultaneously follow the policy gradient and stay inside the behavior potential well to roughly keep A large divergence to the center policy. a is hyper-parameter balancing the two objectives. To determine A we simply sample from uniform larger
distribution as BRP: A !!rrer ~ U0. [gr] As OBrr directly controls the steepness of the Gaussian distribution, larger dærT means less restriction over the policy's 's divergence and vice versa.
Appendix 0.1 gives more details with regard to weighted linear recombination.
5
Experiments
5.1 Exploratory studies
As previous studies have shown [14], perturbing Tanh-activated networks is easier than perturbing Rel.vvacte networks. As perturbing networks with BRP straight forward and network architecture agnostic, we conducted comparative study to see how those two-types of networks respond to BRP. To be specific, we randomly sampled directions and recorded behavior divergence changes along those directions. As in Appendix Figure 1(a), where the x-axis is the percentage of the positive sign in one direction, the y-axis is the magnitude along that direction and the color-scale measures the behavior divergence (the brighter the larger), it is clear that randomly perturbing Tanh- activated networks has a larger chance of inducing significant behavior changes, which explains why networks generally favored in perturbation- based methods.
To verify that actors trained by BTT are uniformly distributed as Agrr is sampled from uniform distribution, we trained two BEL instances with 0.0 (without BTTT) and 1.0 (with BTT). From Appendix Figure (b)) is obvious that without BTT, the trained policy are quite concentrated. When BTT is applied, the behavior divergences of the population constantly follows the uniform distribution. To further verify that BTT can lead to diverse behaviors, we plotted the state visitation map the DelayedHallChhettah-> environment to visualize how the offssring policies explore different states. As is shown in Appendix Figure ((c), the BTT-trained actors (top row) showed relative different state visitation patterns compared with naively-traines actors (bottom row), which suggests that BTT could boost the diversity the population.
5.2 Ablative studies
To verify the effectiveness of BRP, tested different A settings the Walker2d-v3 environment. From Figure 3(a), it noticeable that BRP not only significantly accelerates the learning process, but also helps avoiding local optimums. As matter of fact, find that when BRP is applied, critic networks tend to constantly induce larger training loss throughout training. This phenomenon indicates that BRP indeed brings another level of behavior uncertainty, which forces critics to make better predictions.
An ablation study on the DelayedlHafCCheetah--3 environment is conducted to show that BTT is indeed helpful for exploration DelayydHaalCCheetah--> is a modified HalfChectah-v3 environment where the reward is manually delayed for 20 time steps, making it a difficult sparse reward environ- ment. The proportion of the ligliiildoood objective is tuned with a. We can observe from Figure 3(b) that when a is set to zero, which means no BTT in the training objective, BEL can't 't effectively explore. However, on the other hand, when a is too large, actors may also lose performance since their behaviors over constrained.
As [12] [19] pointed out, many operators in EA are designed for black-box optimization, and can be potentially harmful for neural networks. An experiment comparing weighted linear recombination and distillation-base recombination was designed on the DelayedHallfhhetahh V3 environment, where all parts of BEL are kept the same except for the recombination phase. In this phase, offssring

---

Walkerbdd-4 -0
DelayeditllCheetah--
MearThoedd
2:300
2302
dela=0.0
2:300
2j01
dataa-B:1
1200
1.00
data-00.2
dpre110
-1.80
2010
TeatThee (te))
Taa Tiwn Werg (11)
Teal Sape [oo]
(a)
(b)
(c)
Figure 3: (a) BRP ablation. A BRP = 0 (max delta in plot) corresponds to no BRP (b) BTT ablation.
max
0 (alpha in plot) corresponds to BTT, larger means more constrained behavior. (c)
Recombination ablation.
HeitCheelah--12
Ant-v3
Walker2d-12
14,90
0,000
9.50
1/005
5.000
3520
4,000
M0ha Resard
4.000
4.00
2,000
3.090
2/000
2.293
1,000
0.200
-3.033
Teall Diae Staps (106])
Tioe Steps (16))
Hopper-xC
Humanoidda
DekayedhanChhetahh-cc
3.50
5,000
4,03
2.30
2.500
Poower
0,000
Maan Roourd
0.000
Algorithms
(Curs)
2,000
POERI
SAG
530
1.30
03:01
00:01 07
03.01
Total Time Stepe (lrf)
Tota Time Steps (141)
Totar Time Stape (1e))
Figure 4: Learning curves on 6 MuloCo environments in one million time steps.
actors are ranked according to their latest episodic rewards, and then, offspring actors are treated as teacher networks. Their actions on N sampled observations are recorded as demonstrations. Then, center actor is treated as the student network imitate offspring actors. To give di different actors different importance, the same set of weights from weighted linear recombination is used:
N Σ (S()) (sx)ll Σ
[(TO)
Much to our surprise, although the ietillatton--basee method seems to learn slightly faster the early phase, it quickly fell into the local optimum and could hardly make its way out. This experiment showed that though naive linear recombination may break the behavior of the output network to some extent, this kind of behavior uncertainty may result in extra exploration which is beneficial.
5.3 Comparison to state-of-the-art RL and EA-RL methods
In this section, the performance of the proposed BEL. | is compared against pure RI. methods including TD3 [2]. SAC [3] and Enn-Ensemble well as other EA-RL methods including [11] and improved version of ERL which is called PDERL [12]. For TD3, CEM-RL, and PDERL, we used the code
'Sorree code for BEL: httpp::/githhbbboo https://gituub.co///ammoooo www.r/aooooddoyyoo//oooo

---

published by the original authors. For SAC, the stable baselines3 library is used. Every algorithm is run on the same machine, and the results we obtained were close what authors had claimed in the original papers. Five tasks from MuloCo continuous control benchmark are selected. Swimmer-v3 is excluded since was found that tuning the reward discount factor to 0.9999 could make all algorithms perform more or less the same, reaching approximately 350 reward. Another DelayedHalfCheettll v3 environment is constructed by delaying the reward signal for 20 time steps, making it a hard exploration task. Following the convention from other literatures, for all algorithms, their learning curves are aggregated over 10 repeated runs across one million time steps. And the evaluated policies tested for 10 times. For BEL, the population center policy is used for testing. Note that though BEL trains the population in parallel fashion, for fair comparison. the total time steps are aggregated for every policy interacting with the environment.
The learning curves are shown in Figure 4, and more detailed results organized in a table with statistical tests are shown in Appendix Table 1.
Sample efficiency It can be seen from Figure 4 that BEL turns out to be very competitive against comparing methods in terms of sample efficiency. On the one hand, it can pick up signals faster than other methods, indicating its high sample efficiency. On the other hand, its final best performance outperforms other methods except on Humanoid-v3.
BEL versus TD3-Ensemble Since trained multiple actor-critic pairs in BEL, it is natural to question if the good performance of BEL comes from the ensemble nature. To answer this question, tested the performance of TDD3ensemble where equal numbers of actor-critic pairs are trained, and all hyper parameters are kept as close as possible. From Figure 4, is clear that BEL outperforms TD3-ensemble on all tasks.
Stability As can be seen from Figure and Appendix Table I, BEL also generally has smaller standard deviations across runs, even compared to other population based evolutionary methods whose population sizes are larger, this means BEL is very stable. Another phenomenon that suggests BEL's robustness is in the Humanoid-v3 environment, where the naive TD3-Ensemble shared the same learning rate (which is larger than single instance TD3) as BEI, but failed to stably learn.
Computation efficiency Since all experiments are conducted on the same machine and all on CPUs, we also recorded the median wall-clock running time of all algorithms. TD3 the fastest algorithm as it is also the most lightweight algorithm. PDERL ranks the second because not all policies in its population are trained, great portion of its population are directly evaluated after perturbation. BEL ranks the third among all algorithms, and is generally faster than SAC and CEM-RL. We think BEL reaches good balance between sample-efficiency and computation overhead.
Limitations Though generally good performance can be expected from BEL, it still has the following limitations. First, as multiple networks are trained in parallel, a computation node with multi-core CPU and relatively large R.A.M. is required. Second, as can be seen from the Humanoid-v3 where BEL does not outperform SAC. may indicate BRP and BTT do not scale very well action space dimension grows. Further studies regarding the scalability of BEL need to be conducted.
6 Conclusion
In this work, a novel population- based ovolutionary training framework for off-policy RL algorithms called BEL is proposed. Exploratory and ablative studies show the effectiiveness of BRP and BTT. Benchmark comparisons against other methods show BEL outperforms state-of-the-art RL and EA-RL methods in terms of sample efficiency. The training pipeline is conceptually simple and offer efficient parallel implementation. Along with the improved stability and exploration ability, we believe BEL can serve as a competitive training method for waaldwrrld robot learning with off- off- policy RL algorithms.
8

---

References
[1] T. P. Lillicrap. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. URL http://arxiv.orggabbb// 02971.
[2] S. Fujimoto, H. Hoof, and D. Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the 35th International Conference on Machine Learning, pages 1587-1596. PMLR. URL www.h:tt//ppptt https::/proceeddngg..... www.rr.resse///// htt///00ffffjj/ imttoommmm11110 html.
[3] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning, pages 1861-1870. PMLR. URL https: html.
//proceedings. mlr.
[4] R. S. Sutton and A. Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning. The MIT Press, second edition edition. ISBN 278-0-262-03924-6.
[5] H. van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil. Deep Reinforcement Learning and the Deadly Triad. URL http: http://aaxiv..oraaaa//aeaee wttp://///////////a////////// 1tt///112...00 02648..8........
[6] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution Strategies Scalable Alternative to Reinforcement Learning. URL www.hhttp//aaaee wrtp
abs/ 1w0t. 038644......
[7] F. P. Such, V. Madhavan, E. Conti, Lehman, K. O. Stanley, and J. Clune. Deep Neuroevolution: Genetic Algorithms Are Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning. URL
[8] H. Mania, A. Guy, and B. Recht. Simple random search provides a competitive approach to reinforcement learning. URL http://arxiv..rrt wrtp//s/////// 07055.
[9] S. Khadka and K. Tumer. Evolution-Guided Policy Gradient in Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc. URL 85f 3 77w02229699796999999999 Abstract. html.
neurips cc/ wwtpc//ppep//////aaaeeeeee
[10] S. Khadka, Majumdar, T. Nassar, Z. Dwiel, E. Tumer, S. Miret, Liu, and K. Tumer. Collaborative Eyolutionary Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, pages 3341-3350. PMLR. utthstps::// //poceeddg...... mlr. www.ppresssv//eee http/////kaaaaaa...aa html.
[11] A. Pourchot, N. Perrin, and O. Sigaud. Importance mixing: Improving sample reuse in evolutionary policy search methods. http:////aarxi http://aaxxvv..oaab///eaaaa wwwg/ba//////////o aataaa////...// 05832.
[12] C. Bodnar, B. Day, and P. Lió. Proximal Distilled Evolutionary Reinforcement Learning. 34 (04):3283-3290. ISSN 2374-3468, 2159-5399.
[13] K. Lee. B.-U. Lee, U. Shin, and I. S. Kweon. An Efficient Asynchronous Method for Integrating Evolutionary and Gradient-based Policy Search. In Advances in Neural Information Processing Systems, volume 33, pages 10124-10135. Cur- ran Associates, Inc. URL https://proceeding 731 (00000946b224411188777o22222 -aw.:52214eb2----taaaattt Abstract. html.
[14] M. Plappert, R. Houthooft, Dhariwal, S. Sidor, R. Y. Chen, X. Chen, Asfour, Abbeel, and M. Andrychowicz. Parameter Space Noise for Exploration. URL http://arxiv abs/1706 01905.
arxiv org/
[15] J. Parker-Holder,
A. Pacchiano, K. M. Choromanski, and S. J. Roberts.
Ef-
fective Diversity Population Based Reinforcement Learning. in Neural Information Processing Systems, volume 33, pages 18050-18062. Cur- ran Associates, Inc URL https://proceedi wwwr.pp..ccpppprr//0e.... /hash/ d1 a11caa2770afe3921e97777700000000 Abstract.html
Advances

---

[16] J. Gomez. Sustaining diversity using behavioral information distance. In Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computatioon GECCO '09, page 113. ACM Press. ISBN 978-1-60558-322 325-9. doi: 10. 10.1144/11569990001 15699188.
[17] A. Cully and Y. Demiris. Quality and Diversity Optimization: A Unifying Modular Framework. 22(2):245-255 259. ISSN 1941-0026. doi: 1601111100//EVC221.2200777
[18] Z.-W. Hong. T. Shann, S.-Y. Su, Y.-H. Chang, T.-J. Fu, and C.-Y. Lee. Diversity-Driver Exploration Strategy for Deep Reinforcement Learning. In Advances in Neural Information Pro- cessing Systems, volume 31. Curran Associates, Inc. URL https: : http///pooceenn...//ee/// -eg.reeiiinnnnnunipip... neurips. cc/paper 2t1///000///aaaa/ http://00000002/222222222e444440 42802cade0664008cceel ceceeeeeeeeceeeeee--eeeeaaa Abstratt..bttt.......... html.
[19] Lehman, Chen, J. Clune, and K. O. Stanley. Safe mutations for deep and recurrent neural networks through output gradients. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '18, pages 117-124. Association for Computing Machinery. ISBN 978-1-4503-5618-3 doi: 115111//55555522255555555
[20] A. Baranes and P.-Y. Oudeyer. Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration Robots. 61(1):49-73, ISSN 09218890. doi: 10. robtt/..ob....02....0
[21] N. Hansen. The CMA Evolution Strategy: A Tutorial. URL htt.:tttaaxxii..rr///////////// 00772.
[22] B. Recht. A Tour of Reinforcement Learning: The View from Continuous Control. 2(1):
253-279. doi: 10: 46//amureeeccootro-0533301111113
[23] D. Wierstra, T. Schaul, J. Peters, and J. Schmidhuber, Natural Evolution Strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pages 3381-3387 IEEE. ISBN 978-1 178-1-42244-1 -4244-1822-0. doi: 10. 109/CEC.2008 201220088443225 4631255.
[24] I. Rechenberg. Exolutionnsstatteienn In B. Schneider and U. Ranft, editors, Simulationsmethhdee in der Medizin und Biologie, Medizinische Informatik und Statistik, pages 83-114. Springer. ISBN 0124--288--55... doi: 1oi:100.1077/78 3- 3-642-8 8 81283-5.8.
[25] H.PP Schwefel. Exolutionssstrategien für die numerische optimierung. In H.-P. Schwefel, editor, Numerische Optimierung von Computer-Modellae mittels der Evolutionsstrategie: Mir einer vergleichenden Einfühhung in die ffill-Climbing- und Zufallsstrategie, Interdisciplinary Systems Research/ / Interdisziplinäre Systemforschung, pages 123-176. Birkhäuser. ISBN 978-3-0348-5927-1
3-0348-5527-42
[26] D. Silver, G. Lever, N. Heess, T. Degris, Wierstra, and M. Riedmiller. Deterministic Policy Gradient Algorithms. In International Conference on Machine Learning, pages 387-395.
PMLR. URL attp://proceedingg mlr. wwt.pppeess////eee hwt/////ilee11// 14. .ttll.
[27] Lehman, J. Chen, Clune, and K. O. Stanley. ES is more than just a traditional finite- difference approximator. In Proceedings of the Genetic and Evolutionary Computation Confer- ence, GECCO '18 pages 450-457, Association for Computing Machinery. ISBN 978-1-4503- 5618-3. doi:10. 111111111444/225555522555
[28] J. Lehman and K. O. Stanley. Abandoning Objectives: Evolution Through the Search for Novelty Alone. 19(2):189 123-22.. ISSN 1063-6560, 1530-9304. 16221..../// a.0025.
[29] J.-B. Mouret and S. Doncieux. Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity. In Eleventh Conference on Congress on Evolutionary Computation (CEC*09), pages 1161-1168. IEEE Press. URL attps://han araaa:/ archives-ouvertes. hal ha1-00473147.
ouvertes fr/
[30] Doncieux and J.-B. Mouret. Behavioral Diversity with Multiple Behavioral Distances. In IEEE Congress on Evolutionary Computation, 2013 (CEC 2013), pages 1-8. URL. https:
/ //hal, werchiies-ouverteer wtw.uueeeeeeeraaa11/a13113. fr/hall00300070.
10


Reviews:
[{"review_id": "ddEJaot_He3", "reviewer": "Reviewer_1HqG", "summary": "The paper offers a novel population-based evolutionary training framework for off-policy RL algorithms based on the idea of combining both Evolutionary Algorithm (EA) and Deep Reinforcement Learning (DRL) into a single training framework due to the recent promising results when combining and exploiting the advantages of both EQ and DRL methods.\n\nThe paper first addresses two current limitations of hybrid approaches:\n - How to randomly perturb the policy in a meaningful and controlled manner?\n - Perturbed networks may end up being similar and reduce the overall diversity.\n\nThe paper suggests solving these two problems by proposing to combine: \nBehavior-Regularized Perturbation (BRP) to randomly perturb a policy network within a specified behavior divergence range online and \nBehavior-Targeted Training (BTT) to randomly assign a sampled target behavior divergence and inject it into the actor training process.\n\nThe proposed framework shows the benefits over baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- well-written, clear, and easy to follow\n- results are good compared to baselines\n\nWeaknesses\n- computational expensive", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Can you provide more reasons and explanations on why BRP adopts the Euclidean norm and not other distance measures?\n\nPost rebuttal update\n\nThe authors answered my question with related references, which convinced me, so I kept my score weak accept.\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- computational expensive and the author proposes parallel implementation which shows how solve the problem. ", "reviewer_expertise": "1: The reviewer's evaluation is an educated guess"}, "extra_text": {}}, {"review_id": "UGS7aPiVxi", "reviewer": "Reviewer_4o9R", "summary": "This paper presents Behavior-aware Evolutionary Learning (BEL), which combines off-policy reinforcement learning with evolutionary strategies (ES) for policy optimization. BEL consists of two parts. The first part is Behavior-Regularized Perturbation (BRP), which improves standard gaussian-based perturbations for ES algorithms by considering the sensitivity of the policy's output to parameter value changes. The second part is Behavior-Targeted Training, which ensures that the fine-tuned policies after RL still have desired diversity. With the combination of BRP and BTT, BEL outperforms baselines in achieved reward. The authors further conducted ablation studies to verify design choices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Nice incorporation of ES and gradient-based RL\n\nOverall, I think this paper presents an effective method to combine Evolutionary Strategies (ES) with gradient-based Reinforcement Learning. The use of TD3 improves the quality of the random perturbations generated by ES. Moreover, TD3 is an off-policy algorithm, which means that it does not require additional environment interaction to train on. Lastly, the incorporation of BRP and BTT makes the algorithm more effective by ensuring that perturbation happens at desired scale, and the fine-tuned policy remains at desired diversity.\n\n* Well-written and easy-to-understand\n\nThe paper is well-organized and easy to follow. Although the paper presents solid ideas with very detailed experiments, it does not feel difficult to read, and the contents flow naturally. I do have a few minor comments about certain details, but the overall writing is great.\n\n* Well-designed experiments\n\nThe experiments are well designed and justifies the performance, as well as the detailed design choices of the algorithm.\n\nWeaknesses:\n* Less related to robotics\nThe paper presents a general RL algorithm with evaluation on standard Mujoco benchmarks. While the algorithm should be easily adapted to robotics tasks, the evaluation is mostly in simulation. Overall, I think this paper is clearly relevant to robotics. However, additional robot experiments would be desired for a conference like CoRL.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think it’s a nice paper that should be accepted. The only thing that holds me back from a “strong accept” is its lack of real-robot experiments. However, even at the current shape, I think this paper shows sufficient relevance to robotics and deserves to be accepted.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I would like to point out some issues about Fig.2 in the paper. Overall, this figure is pretty difficult to understand due to small font sizes and lack of explanation. A couple of more detailed comments:\n\nFig.2(a) should probably be a standalone figure and appear earlier in the paper (close to methods section).\n\nFig.2(b) and (c): the axis labels are too small. In addition, they are referred to as 3(a) and 3(b) in the text (Line 166).\n\nFig.2(e): I’m having difficulty understanding this figure. Looks like the bottom plots (without BTT) is covering the state space better than the top plots (with BTT)? Why are they in different colors?\n\nOther than these, I do not have specific comments about the writing of the paper. It would be great if the authors could show more robotics-relevant experiments in the future as well.\n"}}, {"review_id": "LFI4YNLFKua", "reviewer": "Reviewer_gZYS", "summary": "In the paper \"Efficient and Stable Off-policy Training via Behavior-aware Evolutionary Learning\", the authors present a novel algorithm called Behavior-aware Evolutionary Learning (BEL). This algorithm enhances an evolutionary strategy algorithm with two different mutation operators that are taking into account the impact of the mutations on the \"behaviour\" of the policy and encourages diverse behaviours during each generation. \nThe paper proposes multiple experiments to visualise better the effects of their algorithm as well as three ablation studies. Finally, the authors demonstrate the higher performance of their algorithm against state-of-the-art algorithms in six different environments. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall very well written and easy to understand. The algorithm is easy to understand and is well motivated. The experimental evaluations and ablations demonstrate the benefits of the proposed implementation. \n\nI have three minor issues with the paper: \n\n- What the paper calls \"behavioural\" diversity is what we would usually call phenotypic diversity in evolutionary robotics (ER), i.e., changes in the policy outputs. Behavioural diversity in ER usually refers to changes in the robot's behaviour, which includes not only changes in the policy outputs but also how these changes interact with the environment. There is for instance a large field of ER called Quality-Diversity (QD) which focuses on promoting behavioural diversity in Policy Search. QD uses \"Behavioural descriptors\" for this purpose, and I was expecting some sort of connection between the proposed \"behaviour-aware\" aspect of the algorithm and behavioural descriptors. It was only after reading half of the paper that I realised the terminology mismatch. If this could be clarified in the paper to avoid confusion that would be great. \n\n- The figures could be significantly improved: the titles, legends, and ticks are barely readable. This should be addressed. \n\n- I was unable to find the details on how the weight linear combination is performed. I assume this is relatively trivial (i.e., the weight being proportional to the fitness), but if the paper could specify this it would be great. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Apart from the three comments raised above, which could be improved during the rebuttal, the paper is very interesting and would be a nice addition at the conference. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "FrXsDPC0O25", "reviewer": "Reviewer_XDim", "summary": "This work presents a direct policy search framework called BEL (Behavior-aware Evolutionary Learning) based on the evolution strategy (ES) approach. The main idea here is to include a policy-gradient step in the generation of the off-spring policies at each generation of the evolution strategy. To ensure that policies do not deviate too much from the parent policy distribution, a regularization step is added. The paper claims that BEL has superior sample efficiency and stability compared to existing methods, and BEL produces diverse agents in sparse reward scenarios. The paper also claims that due to the parallel nature of the ES approach, BEL is a competitive method to train policies for real-world robots. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although the paper picks up an important problem, that is, sample efficient and stable training of policies, it is hard to see how their approach is solving it. From the beginning of the paper itself, many things are either unclear or unexplained and in several places, the authors simply introduce terms/notations, etc. out of nowhere. For instance, from the title of the paper and the abstract, we get the impression that the paper is about maintaining behaviorally diverse policies during the policy search process. But nowhere in the paper do the authors explain what they call a \"behavior\". Overall the paper lacks a clear intuition behind why their approach works better than the state-of-the-art algorithms like TD3, SAC, etc in standard mujoco tasks. This is a big claim. So it needs clear justification. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper needs major revision. The paper lacks clear intuition to justify their big claims. Many things are either unclear or unexplained and in several places, the authors simply introduce terms/notations, etc. out of nowhere. Experiments are not convincing enough.\n\n### Post rebbutal update\n\nThe authors have addressed my concerns convincingly and improved relevant sections of the paper. Thus I have updated my recommendation to weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "In the abstract: \"The main idea is to train a population of behaviorally diverse policies...\" => Since behavioral diversity is a key element of the paper, spend some time to explain it clearly in the introduction itself, preferably with a example.\n\nWhy BEL is sample efficient? What is the intuition behind it? Generally evolutionary strategies are less sample efficient. \n\n- The last paragraph of the introduction is quite vague. There are many unexplained terms introduced here, e.g., behaviour divergence range, behavior, target behavior divergence etc. A concrete example will make this paragrapgh more clear.\n\n- Fig 1 is more complicated than the text. Why not use a sequntial step by step flow diagram than connecting multiple blocks to each other without any clear start or end. Also, a longer caption of a few sentences would also be useful for readers.\n\n- Eq 6: missiong subscript $\\theta$ on the left side of the equation.\n\n- Line 139: What is $\\mathbf{S}$ ? \n\n- The texts on the plots and figures are extremely small and illegible. \n\n- Fig 2(d): What is RUT in the legend? "}}]

Author rebuttal:

Decision: Accept (poster)