Genetic Soft Updates for Policy Evolution in Deep Reinforcement Learning
Authors: Enrico Marchesini
Davide Corsi
Alessandro Farinelli
Venue: ICLR 2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Evolutionary Algorithms
Formal Verification
Machine Learning for Robotics

Abstract: The combination of Evolutionary Algorithms (EAs) and Deep Reinforcement Learning (DRL) has been recently proposed to merge the benefits of both solutions. Existing mixed approaches, however, have been successfully applied only to actor-critic methods and present significant overhead. We address these issues by introducing a novel mixed framework that exploits a periodical genetic evaluation to soft update the weights of a DRL agent. The resulting approach is applicable with any DRL method and, in a worst-case scenario, it does not exhibit detrimental behaviours. Experiments in robotic applications and continuous control benchmarks demonstrate the versatility of our approach that significantly outperforms prior DRL, EAs, and mixed approaches. Finally, we employ formal verification to confirm the policy improvement, mitigating the inefficient exploration and hyper-parameter sensitivity of DRL.ment, mitigating the inefficient exploration and hyper-parameter sensitivity of DRL.

Published as a conference paper at ICLR 2021
GENETIC SOFT UPDATES FOR POLICY EVOLUTION IN DEEP REINTORCEMENT LEARNING
Enrico Marchesini; Davide Corsi, Alessandro Farinelli University of Verona, Department of Computer Science
ABSTRACT
The combination of Evolutionary Algorithms (EAs) and Deep Reinforcement Learning (DRL) has been recently proposed merge the benefits of both solu- tions. Existing mixed approaches, however, have been successfully applied only to actor-critic methods and present significant overhead. We address these is- sues by introducing novel mixed framework that exploits a periodical genetic evaluation soft update the weights of DRL agent. The resulting approach is applicable with any DRL method and, in worst-case scenario, does not ex- hibit detrimental behaviours. Experiments in robotic applications and continuous control benchmarks demonstrate the versatility of our approach that significantly outperforms prior DRL, EAs, and mixed approaches. Finally, we employ formal verification to confirm the policy improvement, mitigating the inefficient explo- ration and hyper-parameter sensitivity of DRI.
INTRODUCTION
The key a wider and successful application of DRL techniques in real scenarios is the ability to adapt to the surrounding environment by generalizing from training experiences. These solu- tions have to cope with the uncertainties of the operational environment, requiring huge number of trials to achieve good performance. Hence, devising robust learning approaches while improv- ing sample efficiency is one of the challenges for wider utilization of DRL. Despite the promising results (Tai et al., 2017 Zhang et al. 2017 Marchesini et al., 2019), DRL also suffer from conver- gence to local optima, which is mainly caused by the lack of diverse exploration when operating in sigh-dimensional spaces. Several studies address the exploration problem (c.g.. vrriosity-driven exploration (Pathak et al. 2017), count-based exploration (Ostrovski et al.) (017)), but they typi- cally rely sensitive task-specific hyper-parameters, The sensitivity such hyper-parameters is another significant issue DRL as it typically results in brittle convergence properties and poor performance in practical tasks (Haarnoja et al. 2018).
Evolutionary Algorithms (Fogel 2006 have been recently employed as a promising gradient-free optimization alternative to DRL. The redundancy of these population-base -based approaches has the ad- vantages of enabling diverse exploration and improve robustness, leading to a more stable conver- gence. In particular, Genetic Algorithms (GA) (Montana & Davis 1989) show competitive results compared to gradient-basee based DRL (Such et al.] 2017) and are characterized by low computational cost. These gradient-fres free approaches, however, struggle to solve high-dimensional problems having poor generalization skills and are significantly less sample efficient than maddeent-based methods.
An emergent research direction proposes the combination of gradient-free and gradient- methods following the physical world, where evolution and learning cooperate to assimilate the best of both solutions (Simpson) [953).. The first mixed approach, Evolutionary Rein- forcement Learning (ERL) (Khadka & Tumer 2018). relies actor-critic architecture to in- ject information in an evolutionary population while both the gradient-free and gradient-based training phases proceed in parallel. Similarly, Proximal Distilled ERL (PDERL) (Bodnar 2020) extends ERL with different evolutionary methods. CEM-RL (Pourchot 2019) brings this research direction into the family of distributed approaches, combining portfolio of TD3 (Fujimoto et al. 2018 learners with the Cross-Entropy Method (Yan Duan 2016).
"Contact author: enri coricco.o rches in @univr... it.

---

Published as a conference paper at ICLR 2021
These mixed approaches however also present several limitations, which we address through our work: (i) the parallel training phases of the DRL and EA components (Khadka & Tumer 2018 Bodnar
Grabizustose ZRL iiiniiue
Favinonment
Popuisttion
Ageit
Mutriiu
Agenr sog7
2020), or the multitude of learners (Pourchot 2019) result in sig- nificant overhead (detailed in Section 4.. (ii) The actor-critic for- malization of previous mixed approaches, allows them to be easily evaluated in continuous locomotion benchmarks (Brockman et al.) 2016 Todorov et al., 2012). However, this also hinders their combi- nation with value-basse DRL (Marchesini & Farinelli] 2020a). This
Genetic Sudi
Stoce and hamppl
Childon
Updine
Memory
Shore
Selection
Fitnox
Evelastion
Figure 1: Supe-RL overview.
is important as recent work (Matheron et al.] 2019) shows the limitation of actor-critic in determin- istic tasks, that in contrast can be effectively addressed with value-based DRL. In particular, Section 4 shows that a value-based implementation of Khadka & Tumer (2018) does not converge in our discrete robotic task. (ii) The combination strategy does not ensure better performance compared to the DRL agent as it does not prevent detrimental behaviours (e.g.. drop in performance). This is shown in the poor performance of a value-based based implementation of ERL and PDERL (Section +
We propose novel mixed framework, called Soft Updates for Policy Evolution (Supe-RL), that enables us to combine the characteristics of GAs with any DRL algorithm, addressing the limitations of previous approaches. Supe-RL. (Figure 1 benefits from the high sampling efficiency of gradient- based DRL. while incorporating gradient-free GA to generate diverse experiences and find better policies. Summarizing, Supe-RL based algorithms perform a periodical genetic evaluation applying GAs to the agent network. A selection operator uses a fitness metric to evaluate the population, choosing the best performing genome (i.e., the weights of the network) that is used update the weights of the DRL agent. In contrast to previous work, our genetic evaluation is only performed periodically, drastically reducing the overhead. Furthermore, our soft update (Section [3) allows a direct integration of GAs to any DRL algorithm as it is similar to perform a gradient step towards better policy, avoiding detrimental behaviours. As detailed in Section 3.1 this allows using value- based DRL, exploiting the variety of optimizations developed for the weol-known DQN (van Hasselt cet al. 2016;; Schaul et al./ 2016)] Wang et al. 2016 Fortunato et al.) [017 Bellemare et al. (017)). Crucially, our genetic component influences the DRL agent policy only if one of its mutated version performs better in a subset of evaluation episodes. Hence, as detailed in Section 3 with a sufficient number of episodes obtain good estimation of the overall performance of the population.
Our evaluation focuses on mapless navigation, a well-known problem in robotics and recent DRL (Zhang et al., 20177 Wahid al. 20199] Marchesini & Farinelli] 2020b). In particular, we consider two tasks developed with Unity (Juliani et 2018)) (i) discrete action space indoor scenario with obstacles for mobile robot and (ii) a continuous task for aquatic drones, with dynamic waves and physically realistic water. Besides considering standard metrics related to performance (success rate and reward), we also consider safety properties that are particularly important in these domains (e.g., the agent does not collide with obstacles). In more detail, we employ formal verification ((Corsi (et al., (0200) to compute the percentage of input cases that cause violations of these properties. This is important to confirm our claim that Supe- RL based approaches correctly bias the exploration process in the direction of more robust policy regions with higher returns.
Results show that Supe-RL. algorithms improve performance (i.e., training time, success rate, av- erage reward). stability, and safety over value-based and policy-gradient DRL (Rainbow (Hessel et al. 2018), PPO (Schulman et al. 2017)) and ERL. Finally, we performed additional comparisons of RL with: (i) PDERL (Bodnar, (2020) to evidence the poor performance of previous mixed approaches when combined with based DRL; (ii) CEM-RL (Pourchot, 2019 in the aquatic scenario, to show the differences with a multi learner approach: (iii) ERL in standard continuous benchmarks (i.e., MuJoCo locomotion (Brockman et al. 2016: Todorov al. 2012)) where results confirm the superior performance of Su.
2 BACKGROUND AND RELATED WORK
We formalize robotic navigation as RL problem, defined over a Markov Decision Process, as de- scribed in recent DRL literature (Tai et al. 2017: Zhang et al., 2017;; Wahid et all 2019). DRL for robotic navigation focus exclusively on continuous action algorithms such as actor-critic DDPG (Lillicrap al. 2015). TD3 (Fujimoto al. 2018) and PPO (Schulman al. 2017). Such methods
2

---

Published as a conference paper at ICLR 2021
have been adopted following the idea that value-based DQN (Mnih al., 2013) can not deal with nigh-dimensional action spaces. However, discrete value-basee based solutions typically result in shorter training time being more sample efficient, and show better performance even in continuous settings. In detail, Marchesini & Farinelli (2020b) shows that discrete DRI. is a more efficient alternative to continuous DRL in robotic navigation. Moreover, Tavakoli al. (2018) proposes an adaptation of Ducling DQN (Wang et al. (2016) with Double DQN (van Hasselt et al. (2016) that achieves compet- itive results in locomotion benchmarks (Brockman et al., [016 Todorov et al. 2012). More recently,
de Wiele et al. (2020) designed DQN- ---based algorithm to handle enormous discrete and continuous action spaces. These researches further motivate our contribution in the design of mixed approach that also works with value-based DRL.
Evolutionary Algorithms EA are an alternative black-box optimization characterized by three main operators (Fogell (006):: generation, alteration, and selection. In detail, Montana & Davis (1989) evolves a population of _N individuals, each one represented by the network vector parameter (genome). Each 0 (i [O, N 1)) is evaluated to produce fitness F(O.). used by the selection operator to choose the best genome. For the EA component of Supe-RL. we consider a mutation- based GA that shown competitive performance over gradient-basee DRL (Such et al. (2017).
Combining EA and DRL Following the trend of using EA as an alternative for DRL (Salimans) al. 2017). an emergent research field focuses on combining gradient-free and gradient-based so- lutions. In particular, ERL (Khadka & Tumer 2018) considers an actor-critic DDPG agent (Lillicrap) al. 2015) and a concurrent EA training that generates a population of individuals, which are mu- lated and selected based on their fitness. The DRL agent is trained in parallel from the samples generated by both the training phases and is periodically injected in the running population which is used to collect the training performance. The mutation function of ERL ensures that, in a certain number of episodes, the crudieet-based policy outperforms its evolutionary siblings, introducing the gradient-based benefits into the population. Hence, biasing the selection process of the next gen- eration and its performance. In their experiments, the authors highlight an efficient transfer of in- formation between the two families of algorithms, outperforming DDPG in
locomotion benchmarks (Brockman et al. [0166] Todorov et al. 2012). However, both the introduction of all the experiences in the buffer and forcing the DRL agent to perform better than the EA population, bias the training and can cause detrimental hehaviours. Inspired by ERI., several combinations have been proposed (Bodnar 2020) Colas et al.) 2018 Pourchot 2019 Khadka et al.) 2019). While GEP-PG (Bodnar (020) can be considered as a simplified version of mixed approach, where a curiosity- driven approach used to fill the buffer of the agent, Proximal Distilled ERL (PDERL) (Bodnar) 2020) addresses the EA component of ERL, introducing novel operators to compensate for the sim- plicity of the genetic representation (as investigated by Lehman et al. (0118), where authors address destructive behaviors of biologically-inspired variation operators applied to neural networks, which causes catastrophic forgetting). However, as detailed in Section 3.1 our genetic evaluation used to soft update the DRL agent only in the case of performance improvement, hence it does not show such catastrophic forgetting. We also mention CERL (Khadka et al. 2019) and CEM-RL (Pourchot 2019) as they are extensions of ERL for distributed training (which we do not consider here) with multiple active learners, which leads to non-negligible overhead (nonetheless, Section 4.2 reports an additional experiment in our continuous task with CEM-RL, provide more heterogeneous overview of the superior performance of Supe-RL). These works share a common baseline as they all rely on actor-critic DRL and are built on the insights of ERL, which is the most closely related to Supe-RL. Hence. we choose ERL for complete performance comparison. Section 4.2 also shows a comparison with PDERL, to further highlights the poor performance of previous approaches when combined with value-based DRI.
Finally, we use formal verification to support our claims on the beneficial effects of our genetic com- ponent into the policy. We report in Appendix Ala brief description of the considered methodology.
3 SUPE-RL
The main insight of Supe-RL is to soft update DRL agent towards better policy regions, enabling the combination with any DRL algorithm. In detail, we combine a mutation-based GA (Such et al.) 2017) with two DRL algorithm!! (i) Rainbow (Hessel et al. 2018) 8) as based algorithm for
We evaluated variety of different learning algorithms for both action domains. Among Rainhow, PPO, DDPG, and TD3, we chose the best-performing ones.
3

---

Published as a conference paper at ICLR 2021
the discrete task and (ii) PPO (Schulman al. 2017) as policy-gradies one for the continuous scenario, giving rise to two algorithms named SGRainbow and GPPO.
A typical Supe-RL RL based algorithm (Appendix B provides general pseudocode) proceeds fol- lows: the weights of a DRL agent drla (referred as genome or 9, interchangeably), are initialized with random values and, as in standard training setup, drl, starts collect experiences interacting with its environment. Such experiences are stored in a buffer R to train the network. Periodically (every GE episodes), Supe-RL makes genetic evaluation by generating a population of children, each one characterized by a different genome. Weights 0g are used to create the N individuals ap- plying noise to the parameter vector: 0a 4- mmmtt,, where n NO00, mute) and mutp the mutation probability. In contrast, the mutation function of ERL multiplies the randomly chosen weights by NO, mul,). Such mutations act in similar fashion of a dropout layer, biasing the DRL agent to perform better than the evolutionary population in the long term. In more detail, ERI. authors mu- tate 10% of the network weights in each episode (or epoch), multiplying them by (0, 0.1) (plus a mutation with standard deviation 10 reset mutation in small percentage of cases). Given that their evolutionary component is running in parallel with the gradient- agent, we noticed that the weights in the population tend to 0, hence causing detrimental behavior. The population and a copy of 0₂ are then independently tested over set of evaluation episodes which shares the same goals, to find the overall best performing individual (beat based on the fitness (the fitness definition is domain-specifit in the case of navigation, it is computed as the number of targets the agent reaches over the evaluation episodes). In this phase, we can also store portion of diverse experiences in the buffer R used by drla, to further exploit the population-based component. Finally, if the selected genome belongs to one of the children, drl, weights are updated towards the mutated version and the training phase continues with the new weights. In contrast, if the best score belongs to drla, the training phase, which was running in parallel, continues. Since the evaluation does not require any interaction among the population, we instantiate an independent copy of the environment for each N ++ 1 population component in a separate thread and test them in parallel, drastically reducing the overhead for the drla. The multi-thread nature of the Unity game engine makes this parallel testing phase straightforward and particularly efficient. This approach has both the advantage of search better-performing policy exploiting GA mutations (similar to noisy exploration (Fortunato al., 2017)) and enrich the replay buffer with new diversified experiences. As detailed in our empirical evaluation and the experiments with formal verification tools (Section B. our genetic evaluation leads to safer policies and significant reduction in training time, with Supe-RL resulting approxi- mately two times faster than ERL in the same scenario.
It is important to mention that both Supe-RL. and previous mixed approaches are especially designed for scenarios in which it is possible parallelize the learning process. Hence, when the training phase is performed on real physical systems, it is not possible in general to evaluate the population in parallel. This could significantly increase the convergence time.
Crucially, in contrast to previous mixed approaches, Supe-RL RL based algorithms are designed to improve the performance of drl, as our combination schema does not bias the choice of the better- performing children in the long term. In the worst-case scenario, the main agent is always the best genome in the population and does not improve the current policy, hence a Supe-RL, based training will match the performance of the chosen DRL. algorithm.
3.1 VALUE- -BASE AND POLICY- -RADIETT IMPLEMENTATIONS
Robotic navigation allows evaluating Supe-RL in a variety of scenarios (e.g.. discrete and continuous action spaces). Hence, two tasks present different characteristies (e.g., static and dynamic envi- ronments, sparse and dense rewards). In this section, we present the value-based and volicy-gradien implementation of Supe-RL, combined with a mutation-based GA (Such et al. 2017
3.1.1 SGRAINBOW IN DISCRETE INDOOR NAVIGATION
We consider Turtlebot indoor environment with obstacles, using discrete action space and dense reward function.
https///www.urttchor.comm/
4

---

Published as a conference paper at ICLR 2021
Environment description Target goals randomly spawn in the scenario and are guaranteed to be obstacle-free. The reward 77 is structured as two sparse value in case of reaching the target Breach = 1 within eπrоr μ 5cm from the target goal, or crashing Rea (resetting the robot to its starting position). A dense part is used during the travel: w(d,-1 de) ) where de-1, de indicate the euclidean distance between the robot and the goal at two consecutive time steps and w = 10 is a multiplicative factor.
-1 1 which terminates an episode
Network architecture: the input layer contains 19 sparse laser scans, sampled in [-90, 90] degrees in fixed angle distribution and the target position (expressed in polar coordinates); a similar setting used in Tai et (201)); [Long et al. (2018). We did explore other encodings for the problem (e.g,, adding linear velocities output) but a higher complexity of the problem causes longer training times with negligible improvements. We performed multiple trials on different network sizes and seeds (Chen & Chang. 1996) and the outcome led us to use two relu hidden layers with 64 neurons each and linear nodes to encode the output angular velocities [-90, -45, O, 45, 90] deg/s.
Methodology for SGRainbow: the genetic evaluation of the value-based agent presents challenges due to the instability of the training algorithm (which is a known drawback of DQN-based algo- rithms) and the poor scalability on high- -igeenimenii action spaces. Nonetheless, recent work shows the benefits of such solutions (Marchesini & Farinelli 2020b] Tavakoli et al.] 2018 de Wiele et al.) 2020) in these challenging setrings, further motivating the requirement for a mixed approach that works also with value-based DRI. Here we will discuss only the elements of the algorithm which are relevant SGRainbow, referring the interested reader to (Hessel et al., 20118) for further details on Rainbow. We developed different approaches to update the drl, with the genome of the better performing child. We firstly switch drl, with the child, soft updating only the target network (origi- nally developed for DDQN van Hasselt et al. (2016)) to approach the new weights. We tried different settings for the target network, but soft update with 7' 0.1 showed us better performance. This method however leads to an unusual optimizer choice, which is crucial considering the high-variance of a value-based algorithm. In particular, the widely adopted Adam optimizer (Kingma & Ba 201 4) with its delf-adaptable learning rate typically requires minimal switch of the drl, model with a better genome, a drop of performance occurs. We motivate this as the current learning rate of Adam is "balanced" for the old drl, weights and requires time adjust its value for the new unexpected and better-perfore perter-performing model. For this reason, this first application, which we refer to GRainbow, required tune an SGD optimizer for this scenario, decaying its initial learning rate from 0.1 to 0.001 based on the current success rate of the model. As shown in Section 4 this approach already outperformed both the Rainbow algorithm and the GA. The tuning for SGD requires several trials and is one of the main limitations of GRainbow. To address this, we considered Tessler et al. (2019). where authors improved DDQN stability by copying the agent model to the target when the former performs better. Hence, we flipped our approach by soft up- dating drl, towards the best genome (using r' 0.1) and switching the target network with such genome. This enabled us to use the Adam optimizer, improving the performance of GRainbow. We refer the interested reader to Appendix of the Supplementary Material for a performance compar- ison between GRainbow with SGD and Adam.
apper-parameter tuming but, after
Finally, we further improved both training performance and stability, exploiting the soft update (Lillicrap et al. 2015)) technique for both networks. We refer the resultant algorithm Soft GRainbow (SGRainbow). In particular, we soft update both the drl, agent and target models towards the weights of the best performing child to smooth the transition of the networks towards the better- mutated policy. The update rule for the drl, networks is then: 0a = 7'0, + (1 - r')@best. We tried different values for r' (results in Appendix C). obtaining the best performance with 7' 0.3. We believe that these periodical slight changes in the drl, policy, that simulate a gradient step towards better network weights, are the core mechanism that allowed Supe-RL to work and improve the performance of value-based based Supe-RL, while also reducing the variance across different runs. A useful side message is that using SGD seems better than Adam when drl, "jumps" due to target network instabilities, as Adam works better if the transition of the target network is more stable. Finally, results in Section D Fintroduce part of the experiences of the genetic evaluation into the same prioritized buffer of the drl, (Appendix contains ablation experiment on the influence of these diversified experiences with SGRainbow).
5

---

Published as a conference paper at ICLR 2021
3.1.2 GPPO IN CONTINUOUS AQUATIC NAVIGATION
We consider an aquatic drone navigation task characterized by dynamic waves, with continuous action space and a sparse reward function. Sources of the aquatic simulator as novel DRL task and a video with an overview of the environment are available at at wtt.:.c../yy/pp.
The aquatic drone is differential drive platform, where continuous action is mapped to the motor power. The drone receives sparse reward r; structured as two values in case of reaching the target Rreach 1 within error μ 5 cm from the target goal, or reaching timeout Ryau = -1 which terminates an episode (resetting the robot to its starting position).
Environment description:
Network architecture: the input layer contains the target position expressed in polar coordinates with respect to the drone and the pitch of the boat. It is possible to retrieve these values using the GPS and the compass of the boat. Our initial evaluation for the network size led to use lanh hidden layers with 32 neurons each. Finally, two lanh output nodes encode each motor velocity (multiplied by constant value to obtain our velocity limit).
Methodology for GPPO: in contrast to the value-based implementation, the stability of on-policy PPO does not present issues related to the Adam optimizer. Hence, we considered the same soft update strategy adopted with SGRainbow. Trivially we did not use experiences of the best child due to the on- on-policy nature of the algorithm. For further details on the implementation of the PPO algorithm, we refer to (Schulman et al. 2017).
EMPIRICAL EVALUATION
The goal of our empirical evaluation to investigate whether Supe-RL, approaches combine the benefits of GA with both value-based and policy-gradient DRL while maintaining minimal overhead for the training. Data are collected on an i7-9700k, using the implementation of Section 3.1 We considered the same set of hyper-parameters (reported in Appendix DD for the baselines and Supe-RL based approaches. Our ERL implementations refer to (Khadka & Tumer 2018) for the evolutionary component, while we use Rainbow and PPO for the gradient-based part. In order get reproducible and consistent results when comparing different algorithms, the random seed is fixed across a single of every algorithm (because there may exist sequence of targets that favor network initialization), while it varies in different runs. As a consequence, a certain run of every algorithm exccutes the same sequence of targets and initializes the networks with the same weights.
better
All the trained models, except for the value-based ERL (ERL-R) and the GA, are able to navigate generalizing: starting and target position, and velocity. The Turtlebot3 lidar allows navigating in unknown environments with different obstacles, while the boat maintains similar performance in different wave conditions. For each graph, we report the mean and standard deviation of ten statisti- cally independent runs, considering (i) success rate: how many successful trajectories are performed; (ii) total reward. Results are smoothed over one hundred episodes.
To collect significant data in the evaluation of navi- gation performance, we chose a set of targets reach- able by every model. Table 1 resumes the results for the discrete (Rainbow, SGRainbow) and the contin- uous (PPO, ERL, GPPO) tasks, considering only models with acceptable performance (i.e., we did not include GA and ERL-R). For clarity, in the aquatic task we collected the dense reward used in the discrete one. Hence, rewards will be similar (agents navigate towards the target, collecting positive rewards), but time and number of steps (i.e., trajectory length) differ significantly (=31% and =337.5% for SGRainbow over Rainbow; =18% and =8% for GPPO over PPO and ERL, respectively). In both tasks, Supe-RL outperforms DRL algorithms and ERL every considered metric.
Table 1: Performance in the evaluation phase
Model
Reward
Step
Time (s)
Rainbow SGRainbow
36.2
370
42
39.1
269
29
PPO
4.5
98
27
ERL
4.6
96
27
GPPO
4.8
89
22
4.1 MALUE-BASED EVALUATION
Here we compare the GA, Rainbow, ERL-R, GRainbow, and SGRainbow. As previously discussed, Figures AA B show that direct combination of ERL with the value-based algorithm can not cope
6

---

Published as a conference paper at ICLR 2021
Turidaa Somedo
Aewri Drowe
Apark Scoert
075
WP0
A3
Spece
D)
Figure 2: Left: indoor navigation (GA, Rainbow, ERL-R, GRainbow, SGRainbow). Right: aquatic
navigation (GA. PPO, ERL. GPPO). (A, C) Average success rate. (B, D) Average total reward.
15
10
15
500
200
250
Figure 3: Left: cumulative number Supe-RL genetic soft updates in continuous aquatic naviga- tion. Right: cumulative number of injection of the DRL agent using ERL in the same environment.
the issues of such DRL approaches, resulting in very poor performance. Given these results, we decided to perform additional experiment with the improved version of ERL, PDERL [Bodnar 2020). which introduces novel genetic operators to improve ERL. robustness. Nonetheless, we con- jecture that the detrimental behaviour of previous mixed approaches is related to their DRL injection pattern, rather than the simplicity of the genetic approach (the goodness of simple genetic repre- sentation is evident in our results and in [Such et al. (2017)). In contrast, our periodical genetic evaluation with a soft update strategy simulates a gradient step towards a better policy while, in a worst-case scenario, we do not update the DRL agent, avoiding detrimental behaviours. Our re- sults (detailed in Section 4.2) confirm the superior performance of PDERL over ERL in both our environments. However, PDERL still provides inferior performance compared to Supe-RL based approaches and shares the poor performance of ERL when combined with the value-based algo- rithm. In further detail, Figure 24 shows that even GRainbow with the SGD optimizer, where the best genome fully substitutes the main DRL agent, outperforms Rainbow (i.e., 80% of success rate over 60%). Moreover, the soft genetic update further improves the performance while reducing the variance and SGRainbow reaches a performance of 90% success rate in about 2000 epochs that cor- respond to 60 minutes of training (in contrast, Rainbow reached 60% success rate in similar training time). Furthermore, the standalone GA was not able to cope with the complexity of the task, where the algorithm needs to generalize the navigation while exploiting the laser values to avoid obstacles.
4.2 POLICY-GRADIEN EVALUATION
Here we compare PPO, an adapted version of ERL with PPO, and GPPO. Figures 2 D show that also in the continuous domain, Supe-RL based method offers better performances considering our evaluation metrics. In detail, GPPO reaches over 98% of average success rate in about 1300 epochs that correspond 110 minutes of training, while PPO, similarly to ERL, was able to reach = 82% of average success rate in ~ 1700 epochs (160 and 210 minutes of training, respectively). Furthermore, as reported in Table 1, GPPO uses a lower number of actions compared PPO and ERI., which translates into shorter paths and travel time for the drone towards the same target.
In this task, we also compared the efficiency of our genetic evaluation in finding better policies with respect to ERL. Figure shows the cumulative number of injections (ERL) and evaluations (Supe- RL) (x-axis), over the successful ones (y-axis) through the training, to reach similar performance. We compare injections and evaluations as they represent the approaches of ERL and Supe-RL to improve the policy, respectively. Results show that our mutation schema finds more often better- performing policies, in contrast ERL where successful network injection occurs more rarely (ERL requires 400% more injection trials, i.c., 250 over 25, to match Supe-RL performance, i.e., 98% success).
7

---

Published as a conference paper at ICLR 2021
A3
11
D3
Figure 4: Performance of Supe-RI, PPO and ERL in MuloCo benchmarks: (A) Reacher-v2; (B) HalfCheetah-v22: (C) Hopper-v2: (D) Ant-v2.
Aquaricc trone scensrto
Agaltcc Drome
TATIE
A)
1pece
100
100
250
150
D)
350
Figure 5: Comparison with CEM-RL in aquatic navigation: (A) average success rate; (B) average training time. Comparison with PDERL in the navigation tasks: (C) Value-Based implementation of PDERL in the discrete task; (D) Gradient implementation of PDERL in the continuous task.
Evaluation in Standard Benchmarks We performed additional experiments on Reacher-v2. HalfCheetah-v2, Hopper-v2, and Ant-v2 MuloCo tasks (Brockman et al., [016] Todorov et al., 2012) with GPPO and ERL. In particular, we considered the same specifics for data collection detailed in Khadka & Tumer (2018), hence report the same performance metrics instead of success rate. Our ERL implementation with PPO returned comparable results to the original ones presented in Khadka & Tumer (018);; Pourchot 2019); Khadka al. (2099) (data were collected using the same hardware, GPPO parameters, and averaged over runs). Crucially, Figure [4,, B, C, D shows that our Ru based algorithm has comparable better performance across the considered tasks. Furthermore, 4C highlights the detrimental behaviour of ERL.
Comparison with CEM-RL and PDERL To confirm that previous mixed approaches not be di- rectly combined with value-based approaches, due to their injection pattern rather than their genetic representation, we performed an additional experiment with PDERL (Bodnar 22200). Such approach addresses the poor genetic representation of ERL, improving its performance and robustness. Fig- ure 55. D shows the collected data in the discrete scenario, where we combined PDERL with our value-based baseline, and in the continuous task, where we combined it with the PPO baseline. We considered PDERL GitHub for the genetic component, and the same hyper-parameters and random seeds used in our previous experiments. Results confirm the superior performance of PDERL. over ERL in both environments (i.c., improved success rate and reduced variance). Furthermore, the two parallel training phases do not provide robust evaluation of the gradient-free population in the same set of tasks, hence, the resultant best agent does not represent an overall best policy for the task. Follows that PDERL shares the detrimental performance of previous mixed approaches when combined with value-based algorithms. Finally, as detailed in Section 2 the recent field of mixed approaches has been extended with distributed solutions that use a portfolio of active DRL learn- ers, such as CERL (Khadka et al. 2019) and CEM-RL (Pourchot 2019). This intuitively results in significant overhead in the training process. We decided to compare GPPO with the TD3 version of CEM-RL, as the authors claim it outperforms previous mixed approaches (to further confirm the overhead, Sec. 5.2.2 of CEM-RL (Pourchot (009) state that tests are on limited timesteps, due to computational demands). In contrast, Supe-RL. uses one DRL learner and the population is only used for policy evaluation. We used the GitHub provided by the authors to test CEM-TD3 in the continuous scenario (aquatic drone) and Figures 5A. B show that, as expected, CEM-RL required significantly more time (250 minutes over 110). In particular, CEM-TD3 reaches 98% success rate (the performance of Supe-RL at epoch 1200) in approximately 600 episodes. However, it required 125% more wall-clock time with respect to the time required by Supe-RL RL to reach similar perfor- mance (data were collected using the same hardware and averaged over 5 runs).
8

---

Published as a conference paper at ICLR 2021
Table 2: Verification results of: (top) indoor task; (bottom) aquatic task
Violation (%)
Time (s)
Memory (MB)
Model
1
1
1
1
11.2
θ₁,1
Rainbow
2.21
9.11
0
79.7
75.5
92.6
3.74
3,96
6.92
SGRainbow
0
4.75
66.7
74.1
80.5
2.18
2.91
4.1
Violation (%)
Time (s)
Memory (MB)
Model
A.o
AA,1
A,o
A,
A
OA.1
PPO
0.9
1.2
3,4
124
0.1
5.8
ERL
0.5
0.7
3.4
3.4
0.1
0.15
GPPO
0
0.1
3.1
3,2
0.1
0.1
4.3 ROBUSTNESS OF SUPE-RL USING FORMAL VERIFICATION
An important result in both tasks is the limited variance shown by Supe-RL. approaches, across with different network initialization seeds. Appendix shows more detailed analysis of our results, where Supe-RL based approaches seem not suffer from detrimental network initialization seeds.
To further confirm our claims on the beneficial transfer of information of mixed approaches, we employ a formal verification tool Corsi et al. (2020), to verify the behavior of our trained models with respect to series of safety properties. The idea behind this evaluation is to confirm that models trained with mixed approaches lead to more robust policies. Given the inferior performance ERI,, we also expect that Supe-RI. RL based models will present fewer configurations that cause undesirable behaviors (i.e., violates the safety property). According to the navigation scenarios, we selected the following safety properties for the / indoor and A aquatic navigation tasks:
O TurtleBot3 has obstacles too close on the right and on the front, it must turn left.
0,,1 If TurtleBot3 has obstacles too close on the left and on the front, it must turn right.
01,2: TurtleBot3 has obstacles too close on the front, it must turn in any of the other directions.
AA,0 If the aquatic drone has target on the right, it must turn right.
AA,1 If the aquatic drone has a target on the left, it must turn left.
Table shows violation percentage, computation time, and memory returned by the verification tool, to test our safety properties. In detail, models trained with mixed approaches (i.e., SGRain- bow, GPPO, and ERL) present fewer violations in every considered property. Furthermore, there is also significant improvement over the computation time and memory required by the verifier. This confirms our claims on the policy improvement of mixed approaches as they evaluate with significant difference the output values, which translates into fewer bounds re-computations for the verifier. Crucially, the superior safety performance of Supe-RL based approaches compared ERL, further motivate the introduction of our framework.
5 DISCUSSION
We presented Supe-RL. a novel mixed framework that exploits the robustness of population-base based GA to improve value-based and navigation scenarios: an indoor navigation for TurtleBot3 platform, aquatic navigation task, and locomotion benchmarks. Our empirical evaluation shows that Supe-RL significantly outperforms DRL basclines (Rainbow and PPO), the GA, and the recent ERL and PDERL, which also show poor performance when combined with value-based DRL. Crucially, Supe-RL RL is the first framework that combines GA and DRL in the field of value-based discrete methods. Furthermore, formal verification tool to confirm the beneficial effects of mixed approaches in policy improvement. This evaluation confirms the superior performance of Supe-RL based approaches, that returned safer models. This work paves the way for several interesting research directions which include exploiting complex evolutionary mechanisms and different crossover techniques to further improve our frame- work as well as the possibility of extending Supe-RL. RL to concurrently optimize the total reward and the desired safety properties.
DRL agents. We evaluate Supe-RL in two mapless
exploited

---

Published as a conference paper at ICLR 2021
REFERENCES
Marc G. Bellemare, Will Dabney, and Rémi Munos. A Distributional Perspective on Reinforcement Learning ICML, 2017.
Pietro Lio' Bodnar, Ben Day. Proximal Distilled Evolutionary Reinforcement Learning In AAAI, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym In CoRR, 2016.
Chyi-Tsong Chen and Wei-Der Chang. A feedforward neural network with function shape autotun- ing In Neural Networks, pp. 627 641, 1996.
Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms In ICML, 2018.
D. Corsi, E. Marchesini, A. Farinelli, and P. Fiorini. Formal verification for safe deep reinforcement learning in trajectory generation. IRC, 2020.
Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enor- mous action spaces via amortized approximate maximization. In arXiv, 2020.
David Fogel. Evolutionary computation toward a new philosophy of machine intelligence (3. ed.). 2006.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy Networks for Exploration ICLR, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-
Critic Methods In ICML, 2018
Tuomas Haamoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor, In ICMI., 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. In AAAI, 2018.
Arthur Juliani, Vincent- Pierre Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A General Platform for Intelligent Agents In CoRR, 2018.
Shauharda Khadka and Kagan Tumer. Evolutionary Reinforcement Learning In NIPS, 2018.
Shauharda Khadka, Somdeb Majumdar, Tarek Nassar, Zach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, and Kagan Tumer. Collaborative evolutionary reinforcement learning. In ICML, 2019.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization In ICLR, 2014.
Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O. Stanley. Safe Mutations for Deep and Recurrent In Proceedings of the Genetic and Evolutionary
Neural Networks Through Output Gradients Computation Conference, 2018.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning) In CoRR, 2015.
P. Long. T. Fanl, X. Liao, W. Liu, H. Zhang, and J. Pan. Towards Optimally Decentralized Multi- Robot Collision Avoidance via Deep Reinforcement Learning ICRA, 2018.
E. Marchesini and A. Farinelli. Genetic deep reinforcement learning for mapless navigation. In AAMAS, 2020a.
10

---

Published as a conference paper at ICLR 2021
E. Marchesini and A. Farinelli. Discrete deep reinforcement learning for mapless navigation. In ICRA, 2020b.
E. Marchesini, D. Corsi, A. Benfatti, A. Farinelli, and P. Fiorini. Double Deep Q-Network for Trajectory Generation of Commercial 7DOF Redundant Manipulator In IRC, 2019.
Guillaume Matheron, Nicolas Perrin, and Olivier Sigaud. The problem with ddpg: understanding
failures in deterministic environments with sparse rewards. In arXiv, 2019.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with Deep Reinforcement Learning In NIPS, 2013.
D. Montana and L. Davis. Training Feedforward Neural Networks Using Genetic Algorithms In IJCAI, 1989.
Ramon Edgar Moore. Interval Arithmetic and Automatic Error Analysis in Digital Computing. PhD thesis, 1963.
Georg Ostrovski, Mare Bellemare, Aaron Oord, and Remi Munos. Count-Based Exploration with Neural Density Models In ICML, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-Driveen Exploration by Self-Supervised Prediction In CVPRW, 2017.
Sigaud Pourchot. CEM-RL: Combining evolutionary and gradient-based methods for policy search
In ICLR, 2019.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution Strategies as Scalable Alternative to Reinforcement Learning arXiv, 2017.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay In ICLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms)
CoRR, 2017.
George Gaylord Simpson. The baldwin effect. In Evolution. 1953.
Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract domain for certi- fying neural networks. Proc. ACM Progrum. Lang., 2019.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep Neuroevolution: Genetic Algorithms Are Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning In CoRR, 2017.
L. Tai, G. Paolo, and M. Liu. Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation In IROS, 2017.
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action Branching Architectures for Deep Re- inforcement Learning. In AAAI, 2018.
Chen Tessler, Nadav Merlis, and Shie Mannor. Stabilizing off-policy reinforcement learning with conservative policy gradients. arXiv, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-learning In AAAI, 2016.
Ayzaan Wahid, Alexander Toshev, Marek Fiser, and Tsang- Wei Edward Lec. Long range neural navigation policies for the real world. CoRR, 2019.
Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Formal security analysis of neural networks using symbolic intervals. In 27th (USENIX) Security Symposium, 2018.
11

---

Published as a conference paper at ICLR 2021
Ziyu Wang, Nando de Freitas, and Mare Lanctot. Dueling Network Architectures for Deep Rein- forcement Learning JCML, 2016.
Rein Houthooft John Schulman Pieter Abbeel Yan Duan, Xi Chen. Benchmarking Deep Reinforce- ment Learning for Continuous Control. In ICML, 2016.
Zhang. J. Springenberg. Boedecker, and W. Burgard. Deep reinforcement learning with successor features for navigation across similar environments In IROS, 2017.
12

---

Published as a conference paper at ICLR 2021
A INTERVAL ANALYSIS FOR FORMAL VERIFICATION
We consider formal verification to support our claims on the bene- ficial effects of genetic component on the policy. We briefly de- scribe the main verification concept and refer the interested reader
the references provided. Recently, Wang et al. (2018) ( et al.) (019) propose an extension of interval analysis for the verification of neural networks. Given a set of input bounds (area) and safety property to verify, these methods propagate the input, layer by layer, to compute the bounds for each output node. Afterwards. they ex- ploit standard formalization from interval algebra (Moore (963):
Yo
Uo = [a, bj < 1/1 = [c, d] = < c to prove or deny properties. decision-making scenarios, such our navigation, property by analyzing the relationships among the computed output bounds. To simplify the understanding of this process, visualize it as 2d graph in Figure 6 which shows the possible scenarios after computing the bounds for pecision-making task: (i) there an overlap between output bounds; (ii) an output bound that sat- isfy the property is always higher than the others; (iii) an output bound corresponding to an action that violates the property is greater than the others. In the last two cases, we can conclude that the property is either verified or denied in the input set. However, the first case requires to re re-compute the output bounds in smaller input subsets (Figure 6 shows the bounds on three smaller subsets on the x-axis) to obtain more precise estimation that falls in one of the other verifiable cases.
verify
Figure 6: Bound analysis in formal verification tools.
In summary, by exploiting formal verification (and specifically Corsi et al. (0200) we can measure the number of input configurations that would cause violation of our safety properties. We measure this as the percentage of area that may cause a violation with respect to the entire input area (i.c., the violation percentage). The lower this number, the more robust is the policy (i.e., properties may be violated by a lower amount input configurations). This translates, de facto, in an overall better policy the trained network correctly evaluate with a significant difference the output values (output bounds have very different values), i.e., the action to choose. In detail, fewer bounds re-computation (i.e., fewer overlaps) means faster computation and less required memory.
B SUPE-RL PSEUDOCODE
We introduce a general pseudocode of Supe-RL. based algorithms.
Algorithm Supe-RL
1: Input: size N of the population P, periodicity GE for the genetic evaluation, seed timeout
T. Initialize network weights 0g. replay buffer R and ene with seed s.
2: for episode 1 to do
3: fort= to T do
4: Following the algorithm policy, select action a;
5: Execute action a, observe reward r, and state 8,-1, and Store (8t, 42, Y2, S24 in R
6: end for
7: env <- ReserEnnvirommen()
8: Sample random mini-batch of transitions from
9: Perform a gradient descent step, updating 0,
if episode % GE
then Start periodical genetic evaluation, in parallel
11:
12:
P. Genus + GeneraaeCChildren(( 0a. env)
13:
Score < Genetic
Genvs)
14:
ghest + Select best 0 E P based on Score
15:
0. + GeneticUpdatee(0.. ghest)
16: End periodical genetic evaluation
17: end if
18: end for
13

---

Published as a conference paper at ICLR 2021
Algorithm 2 Function GenerateChildren
procedure GenerateChiidreenMN (N, 0a. env)
2: Children + Initialize with N + copies of 0.
3: Cenos t Initialize with N + copies of env
4: for fori=1 to N do
5: for iteration to mulp * [Children do
6: Randomly sample indices e for Childrenli]
7: Children (rr c|+ ((0, mully)
8: end for
9:
end for
10: Return Children, Cenas
C ADDITIONAL SGRAINBOW EXPERIMENTS
In this section, we show the performance of GRainbow with SGD and Adam and the results of the tuning of r' parameter for SGRainbow. We also present an ablation experiment on the influence of the population experiences in the buffer of the DRL agent.
Adam and SGD GRainbow Tessler et al. 2019) improved the original DDQN evaluating the per- formance of the agent model over its target model. In particular, they show that more stable training phase can be obtained replacing the target model with the agent weights when the latter statistically perform better than the former. Inspired by this approach, when our genetic evaluation finds better child, the main agent is soft updated towards the best genome, and the latter substitutes the target model. As Figure A shows, this precaution allows use GRainbow with Adam and outperform the manually tuned GRainbow with SGD.
Tuning 7' for SGRainbow We further improved GRainbow performance, exploiting the soft up- date (Lillicrap et al., 2015) developed to improve the naive approach in DDQN which periodically substitutes the target with the agent model. We performed multiple trials with different seeds find the best value of 7 for the soft genetic update function that characterizes SGRainbow. In particular, Figure 7 clearly shows that SGRainbow outperforms GRainbow and reports the average success rate of our training phases with r' E [0.15, 0.3, 0.6].
Population Experiences in SGRainbow Finally, given our best performing algorithm (i.e., SGRainbow with 7' 0.3), we performed an ablation experiment to test the influence of the experiences collected in the genetic evaluation phase in the main agent buffer. Figure 7 shows performance improvement with the introduction of the best child experiences.
D HYPER-PARAMETERS
Genetic Evaluation We use the same genetic hyper-parameterss for both SGRainbow and GPPO for the genetic evaluation. In particular, the size N of the population P is set to 10 and the number of trials conducted in the evaluation phase to compute the fitness score ranges from 10 to 20 across tasks. The mutation probability of network weight mutp = {0.75, 0.4, 0.1] based on the current success rate of the drl, and constant mutation value mut,, 0.1.
Succes bo00)
GRAWNOW 0.15
mentrr
A)
Spach
Spech
Spoch
Figure 7: Additional value- based experiments: Average success rate for (A) SGD and Adam GRain-
bow; (B) Tuning of for SGRainbow; (C) Ablation on population experiences agent buffer.
14

---

Published as a conference paper at ICLR 2021
Rainbow, GRainbow, and SGRainbow Here discuss only the hyper- prpar--arameterss of the algo- rithmic implementations which are relevant to GRainbow and SGRainbow, referring to the original papers for further details (notice that the three approaches share the same hyper-paree parameters). Based on our experiments, we decided to use a Priority Experience Replay (Schaul et al. 2016) buffer of size 30000 with batch size of 64, a 0.6, Istart = 0.4 and Bincrement 0.0005. The soft update of the target network (Lillicrap et al., 2015) for Rainbow is performed with T = 0.01. Furthermore, given the well-structured reward function, we notice that simple with decay= = 0.99 and Cmin 0.02 led to faster training phase compared to the introduction of noisy exploration in the network (Fortunato et al., 2017). Finally, we noticed that a genetic evalua- tion collects on average 30000 total transitions, and adding a random 10% of these transitions into the priority buffer (i.e., transp = 0.1), showed the best results.
greedy exploration strategy
PPO, GPPO, and ERL First, PPO presents two versions and we choose to implement the clipped one, which was recommended in the original work (Schulman et al., 2017). to which we remind the interested reader for further details. We choose to use buffer of 256 elements with mini-batches of size 64 for the considered algorithms, while we used the original ERL (Khadka & Tumer 2018) for specific algorithm parameters.
E ROBUSTNESS OF SUPE-RL
A more detailed analysis of our results reveals that the high variance of GA, PPO, and ERL is due to a specific seed that results in a pathological performance of the approach (a similar phenomenon occurs for Rainbow). In partieular, Figure B B, C, D report the success rate and total reward of the only bad initialization seed that causes GA, Rainbow, PPO, and ERI to perform poorly, compared to the previous runs, On the other hand, GPPO, GRainbow, and SGRainbow do not present this issue, reaching similar results to the other training phases. Even if this analysis is not as relevant as previous results, as not averaged over multiple seeds, we do believe that this is an important aspect of our framework and will need further investigation. Crucially, Figure 9A, B, C, D show that SGRainbow, and GPPO perform better than other approaches even in the selected runs without such detrimental seed.
Aeedi Dewm
ma
-200
A3
B)
6p301
D)
spech
Figure 8: (A) Average success rate and (B) total reward considering only the simulation with the bad initial seed in the indoor navigation. (C) Average success rate and (D) total reward considering only the aquatic simulation with the bad initial seed.
Turtetet
WD
S0ce
1000 1100 2015
Mdd
4/5
1960
1im abo Lif 10M
Figure 9: (A) Average success rate and (B) total reward when the bad initial seed is removed from the indoor simulations. (C) Average success rate and (D) total reward when the bad initial seed is removed from the aquatic simulations.
15


Reviews:
[{"review_id": "w7KxguNA2c8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces another combination of an Evolutionary Strategy (ES) with RL, which basically consists of running an RL agent which every GE episodes alternates to run an evolutionary algorithm iteration based on the current policy as the parent of the population. If none of the individuals generated are better than the parent, the RL continues as it was before the genetic operations, otherwise the RL agent parameters are soft updated towards the value of the best individual.\nThe paper introduces a simple but effective approach which tries to obtain the benefits of both the ES and RL methods, making sure that the genetic operations do not  have a negative impact in the RL process. The contribution is interesting and fair, although the authors give considerable attention to the navigation problems used for validation,  which environments are also presented in this paper in the sections of the method and the results. Whereas in the title, abstract and first two sections the paper is completely focused only in the proposed hybrid learning method.\n- Why not focusing on the more conventional/standard tasks (approached in the appendices) for evaluating the learning methods in the main paper, and leaving the proposed navigation tasks for the appendices, for a reader it would be more intuitive when analysing results of deeply well known studied problems. With this, the paper could focus more on the method, because for instance, Section 3 is dedicated considerably to stress the navigation tasks, while the method is indeed more general and not specialized in this kind of tasks. On the other hand, the content of Appendix E would be more valuable in the main paper.\n- In Section 3, it is mentioned \"Such mutations act in a similar fashion of a dropout layer, biasing the DRL agent to perform better than the evolutionary population in the long term\". It is not clear why it is better, what is the foundation of this statement? Authors could add references or present a deeper discussion about it. If the reason is given based on the experiments of this work, this statement fits better the experiments section, or conclusions.\n- It is important to stress more that the method is especially intended for scenarios in which it is possible to parallelize the learning process. In cases of learning with physical systems, it is not possible to compute the genetic operators and the population evaluation in parallel, this could significantly increase the convergence time, which could be unfeasible. \n\n- For SGRainbow in Section 3.1.1 it is mentioned \"...as GRainbow, required to tune an SGD optimizer for this scenario, decaying its initial learning rate from 0.1 to 0.001 based on the current success rate of the model\". How is this decay computed? it seems it is a function of the success rate, however for tasks that do not have this metric, how could this be done?\n- Why the statistics are taken with only 5 runs in Section 4? this does not seem to be enough, additionally the tasks do not look extremely complex to think that running more experiments is unfeasible (this is also why analysing main results based on the standard tasks is more straightforward).\n- Regarding the results of Table 1, it is not clear what are exactly time and number of steps, since normally they represent the same information. It is said steps have to do with trajectory length,  but what does length mean? it could be either trajectory duration, or distance of the trajectory.\n- In general the results look positive and show an improvement over the baselines and previous methods combining the two worlds. It would be interesting to mark in the learning curves, the times in which the genetic operation was executed, in order to observe its effect through the convergence.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "vLckZ9jazpw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Aiming at exploiting the benefits of population based policy optimization and policy gradient, this paper proposes a novel framework that combines these two techniques. The authors claim that the previously proposed frameworks that combines evolutionary approaches and policy gradient approaches uses actor critic approaches in the policy gradient part, and that it is important to be able to incorporate value-based methods instead of actor critic approaches since the value based approach sometimes outperforms the actor critic approaches. The proposed framework is designed to be capable of combining evolutionary policy search approaches with ANY deep reinforcement learning algorithms. The framework is simple and relatively easy to combine with different deep reinforcement learning algorithms. However, in two instantiation of the proposed framework presented in this paper, the authors tune the details for each of these two. For one cases the authors obviously exploiting the structure of the combined reinforcement learning algorithm (existence of the target network and update the target network only), which is not possible in general. \n\nThe weakest point of this paper is the evaluation. The above mentioned two instantiations are evaluated only on a single task for each. It is definitely not sufficient to evaluate the generality of the framework. Moreover, the proposed approach is not compared to existing frameworks combining evolutionary approaches and policy gradients. The only existing approach is ERL, where its actor critic part is replaced so as to have the same RL approach as the proposed framework. Such a replacement is fine to evaluate the goodness of the framework itself. However, since ERL is not designed to be combined with other approaches than policy gradient, it is easy to imagine (and also mentioned in the introduction) that the performance will drop severely. To show the advantage of the proposed framework, the authors should compare the performance with the original ERL, and also more recent variants such as CEM-RL and CERL. \n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mcv946F-PuJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces Supe-RL that intermingles off-policy reinforcement learning with periodic beam search operations. The method makes a greedy selection between the rl-solution that it had and the best produced by the beam search using Polyak update to update the incumbent rl-solution if the one suggested by beam search is better. Experiments in robotics simulation benchmarks show that Supe-RL improves over prior hybrid methods. \n\nAn extensive review of related literature is included that is extremely helpful in situating the work amongst the broader field. Visuals like Figure 1 also really hep the user in understanding the core concepts presented. \n \nThe paper could be clearer with more copy-editing and technical rigor. Some technical terms are used loosely and ancillary claims made without justification which serve to confuse the reader. For example, the author claims (Page 1) that gradient-free approaches have poor generalization skills. I am not quite sure if this is correct as gradient-free EA/GA has been an industry standard as a general-purpose black-box optimizer. \n\nThe author also claims that ERL uses an ES. I have read the ERL paper in the past and from memory know this to be incorrect. Verifying with the ERL paper, it seems to use an EA, not an ES. This is crucial as an ES is a finite-difference approximation of the gradient at heart and would be rather redundant with a gradient-based learner. EA on the other hand belongs to an entirely different family of algorithms and resembles natural evolution more closely. \n\nThe author also uses the word redundancy in a confusing way. First, the author claims that redundancy in the population leads to diverse experience (page 4). I am not sure how this would work. Robustness is certainly a contributing factor to redundancy but diversity? I think the author is trying to convey that a population allows for a spread-out search across multiple promising directions/points in the search space concurrently thus helping diversity. I am not sure if redundancy is the correct word to describe this feature.\nSimilarly, the author mentions enriching the (replay) buffer with new redundant experiences (Page 4 Section 3)? That seems like an oxymoron.\nPage 5 reads “Finally, results in Section 4 considers the redundancy offered by the population to introduce part of the experiences of the genetic evaluation into the same prioritized buffer of the drla” – I am not quite sure how redundant experiences would add value?\n\nThe author refers to Polyak updates as soft updates and makes it a central component of the method’s novelty. However, these kinds of updates with a target network is a fairly standard tool in most modern policy gradient algorithms i.e., Soft-Actor Critic (SAC). The use of the word “soft update” is also a bit confusing as I thought this indicated the use of a modified Bellman backup with a Boltzmann policy representation (similar to Soft Q-learning) at first. It is unfortunate that the namespace is overloaded.\n\nIf my understanding is correct, the mutated solution (neural network policy) from a round of EA directly replaces the DRL solution? This is generally a risky thing to do. Gradient-based methods often rely on the weights to lie within specific distributions (ranges of magnitudes) for effective gradient computation and update. This is particularly true if the network uses non-linear activation functions that can easily saturate (dying ReLU since the authors use ReLU). However, EAs have no such requirement and the weights can often blow up to big numbers particularly for a non-linear function approximation. Is there a safeguard to avoid this scenario?\nI believe the nuances behind sensitivities of the optimizer (SGD and ADAM) that the author mentions (plus the fact that Polyak updates help) are symptoms of this fundamental issue (transferring policy weights with a nonlinear function approximator directly from a GA to a gradient-based learner).\n\nOverall, the authors present an interesting idea. However, the novelty is marginal compared to earlier methods and far too many technical liberties are taken with the manuscript that make it difficult to absorb as a reader. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a deep reinforcement learning algorithm Supe-RL that combines model free RL with genetic updates. The idea is to periodically mutate and evaluate the actor and greedily choose the best performing child, and incorporate it in the main actor via Polyak averaging on a target policy network. The algorithm can be in principle combined with any gradient based deep RL method. Supe-RL was demonstrated by combining it with Rainbow and PPO and evaluated in navigation tasks as well as standard MuJoCo benchmarks.

Overall, the reviewers found the idea interesting and to have value to the RL community. The reviewers raised some questions regarding technical rigor, evaluations, and the choice of base DL algorithms. As is, I find this a slightly above borderline submission, and thus recommend acceptance. However, I would encourage the authors to test their method also with a state-of-the-art off-policy algorithms, such as TD3 or SAC, in continuous domains, to better calibrate its overall performance.
Decision: Accept (Poster)