PGPS : Coupling Policy Gradient with Population-based Search
Authors: Namyong Kim
Hyunsuk Baek
Hayong Shin
Venue:  (2021)
Keywords: Reinforcement Learning
Population-based Search
Policy Gradient
Combining PG with PS

Abstract: Gradient-based policy search algorithms (such as PPO, SAC or TD3) in deep reinforcement learning (DRL) have shown successful results on a range of challenging control tasks. However, they often suffer from flat or deceptive gradient problems. As an alternative to policy gradient methods, population-based evolutionary approaches have been applied to DRL. While population-based search algorithms show more robust learning in a broader range of tasks, they are usually inefficient in the use of samples. Recently, reported are a few attempts (such as CEMRL) to combine gradient with a population in searching optimal policy. This kind of hybrid algorithm takes advantage of both camps. In this paper, we propose yet another hybrid algorithm, which more tightly couples policy gradient with the population-based search. More specifically, we use the Cross-Entropy Method (CEM) for population-based search and Twin Delayed Deep Deterministic Policy Gradient (TD3) for policy gradient. In the proposed algorithm called Coupling Policy Gradient with Population-based Search (PGPS), a single TD3 agent, which learns by a gradient from all experiences generated by population, leads a population by providing its critic function Q as a surrogate to select better performing next-generation population from candidates. On the other hand, if the TD3 agent falls behind the CEM population, then the TD3 agent is updated toward the elite member of the CEM population using loss function augmented with the distance between the TD3 and the CEM elite. Experiments in a MuJoCo environment show that PGPS is robust to deceptive gradient and also outperforms the state-of-the-art algorithms.


Under review as a conference paper at ICLR 2021
COUPLING POLICY GRADIENT WITH POPULATION- BASED SEARCH (PGPS)
Anonymous authors Paper under double-blind review
ABSTRACT
Gradient-based policy search algorithms (such as PPO, SAC, TD3) in deep rein- forcement learning (DRL) have shown successful results on range of challenging control tasks. However, they often suffer from deceptive gradient problems in flat or gentle regions of the objective function. As an alternative to policy gradient methods, population-based evolutionary approaches have been applied to DRL. While population-base based search algorithms show more robust learning in a broader range of tasks, they usually inefficient in the of samples. Recently, reported are a few attempts (such as CEMRL) to combine gradient with population in searching optimal policy. This kind of hybrid algorithm takes advantage of both camps. In this paper, we propose yet another hybrid algorithm, which more tightly couples policy gradient with the population-based search. More specifically, we use Cross Entropy Method (CEM) for population- search and Twin Delayed Deep Deterministic Policy Gradient (TD3) for policy gradient. the proposed algorithm called Coupling Policy Gradient with Population- based Search (PGPS), a single TD3 agent, which learns by a gradient from all experiences generated by population, leads population by providing its critic function Q as surrogate to select better performing next generation population from candidates. On the other hand, if the TD3 agent falls behind the CEM population, then the TD3 agent is updated toward the elite member of CEM population using loss function aug- mented with the distance between the TD3 and the CEM elite. Experiments in five challenging control tasks in a MuloCo environment show that PGPS robust to deceptive gradient and also outperforms the state-of-the-an algorithms.
INTRODUCTION
In Reinforcement Learning (RL), an agent interacts with the environment, and its goal is to find the policy that maxi- mizes the objective function, which is generally defined as a cumulative discounted reward. Recently, many researchers have worked on combining deep neural networks and a gradient-basee RL algorithm, generally known Deep Re- inforcement Learning (DRL). This approach has achieved great success not only in the discrete action domain, such as in Go (Silver et al., 2017) and Atari games (Mnih et al., 2015; 2016), but also in the continuous action domain, such as in Robot control (Fujimoto et al., 2018; Lillicrap et al., 2015; Schulman et al., 2015). However, it is difficult use the gradient-based method for the objective function (J),
((e))
1(0g)
(00,)
I(02)
Vo_
J(0,),
N(0 6
0 02
0g 0 6
Figure 1: Flat gradient and population- based search on piece-wise constant function
which includes "many wide flat regions" since the gradient (VoJ) is near zero at flat point. Figure 1 is an extreme case consisting of only flat regions, which is called piece- wise constant function. This problem remains an unsolved issue in gradient- based DRL with continuous control domains (Colas et al., 2018). The Swimmer in MuloCo environment (Todorov et al., 2012) has already been reported to be hard to use the gradient- based method (Jung et al., 2020; Liu et al., 2019). Our experiment shows that the objective function of Swimmer includes wide flat regions (Appendix A).
The population- paased Evolutionary Approach (EA), which is an alternative to the muddient-based method, has also shown successful results in various control tasks (Conti et al., 2018; Liu et al., 2019;

---

Under review as a conference paper at ICLR 2021
Salimans et al., 2017; Such et al., 2017). As a population- population-based search, the EA generates population of agents to explore policy, and the population is regenerated with improvement in each generation. direct policy search (Schmidhuber & Zhao, 1998) because it directly
The EA is also known searches by perturbing the parameter of policy. In Figure 1, the Cross-Entropy Method (CEM) as a search is simply descrihed, where the current population sampled from the target distribution is evaluated. Then the distribution is updated to the direction for generating more promising population. Not depending on the gradient, these approaches are robust to flat or deceptive gradients (Staines & Barber, 2013; Liu et al., 2019). However, the EA is sample inefficient because it requires a Monte-Carlo evaluation, and the previous results and data generally cannot be reused.
kind of population-basse
The off-policy Policy Gradient (PG) algorithm uses the data from arbitrary policies to train its actor and critic functions. It generates exciting potential by combining the EA and PG, where the data which is discarded in a standard EA is directly used to train the PG's s functions. Khadka & Tumer (2018) and Pourchot & Sigaud (2018) introduced a framework combining the EA and off-policy PG. However, the framework of (Khadka & Tumer, 2018) is less efficient to train the policy for general tasks than the PG algorithm alone, and the framework of (Pourchot & Sigaud, 2018) is unsuitable to train the policy for a task providing deceptive gradient.
In this paper, we propose another hybrid algorithm, called Policy Gradient with Population- based Search (PGPS) in which the CEM and Twin Delayed Deep Deterministie Policy Gradient (TD3) (Fujimoto et at., 2018) are combined. It is as robust to a deceptive gradient as the CEM and more efficient train the policy for general tasks than TD3. To be robust deceptive gradient, the proposed algorithm is constructed in a way similar to the one in (Khadka & Tumer, 2018), where the TD3 is trained using data from the CEM and periodically participates in the CEM population as an individual (PG guides EA). However, in this basic framework, the TD3 sometimes falls into the inferior solution and inefficiently searches. To get the TD3 out of the inferior solution, we let the EA guide the TD3 by guided policy learning (Jung et al., 2020) (EA guides PG). Furthermore, the TD3 critic contributes to generating a more promising population by filtering the set of actors sampled from CEM (Q-critic filtering). Lastly, to control the trade-off between the frequency of search and stable estimation, we used evaluation step scheduling in the process of population evaluation (Increasing evaluation steps). It carries out frequent searches when searching far from the optimal, whereas it carries out stable estimation when searching close to the optimal. These approaches bring out more synergies between the CEM and the TD3 while maintaining both the ropulation-baser search and the gradient- based search. Consequently, the proposed algorithm is not only robust to a deceptive gradient, but also produces outstanding performances with low additional computational cost.
2 RELATED WORKS
Recently, beyond the view of an alternative approach, few attempts have been proposed in the form of A supporting B. An attempt is to use EA to fill a replay buffer with diverse samples. In Colas et al. (2018), Goal Exploration Process (GEP), a kind of EA, is firstly applied to search the policy and to fill a replay buffer with the diverse samples, and then the pollcy PG algorithm is sequentially used for fine tuning the parameters of the policy. Another attempt is to combine population-basee approach and PG for efficiently searching a good policy or the good syper-parameter of an algorithm in parallel multi-learners setting. These applications generally consist of periodically evaluating the population, followed by distributing good knowledge to the other learners. To find the best architecture and hyper-parameters, Jaderberg et al. (2017) proposed a Population- -Bsee Training (PBT) method in which the current best knowledge is periodically transferred to PG learners. Gangwani & Peng (2017) developed the distilled crossover using imitation learning and mutation based on the PG. Proposed operators transfer the information on current good policies into the next population without destructive change to the neural network. Jung et al. (2020) introduced a soft-manner guided policy learning to fuse the knowledge of the best policy with other identical multiple learners while maintaining more extensive search area for the exploration.
The idea of combining the population-based EA and off- PG was recently introduced by Khadka & Tumer (2018). Their approach was called Exolutionary-Guide Reinforcement Learning (ERL) in which the Genetic Algorithm (GA) and the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) are combined. In ERL frameworks, the GA transfers the experience from evaluation into the DDPG through a replay buffer, and the DDPG transfers the knowledge learned
2

---

Under review as a conference paper at ICLR 2021
from the policy gradient into the GA by periodically injecting PG actor into the GA population. Khadka et al. (2019) expanded the PG algorithm of ERL. from a single DDPG learner to multiple TD3 learners with resource manager. Bodnar et al. (2019) revised the GA's crossover and mutation to the distilled crossover and proximal mutation inspired by (Gangwani & Peng, 2017) and (Lehman et al., 2018) to prevent the destruction of neural networks. Pourchot & Sigaud (2018) introduced another framework, which combines the CEM and the TD3. In this framework, the TD3 algorithm has only critic function trained using the experience from the CEM. In order to propagate the knowledge learned by policy gradient to the CEM, half of the population is updated to the direction indicated by the TD3 critic for fixed number of steps, followed by the evaluation. The policy gradient for half of the population not only enhances the gradient-based learning, but also deteriorate the CEM's s robustness over a deceptive gradient.
In this paper, we introduce another hybrid algorithm, in which the CEM and the TD3 are combined as in CEMRL (Pourchot & Sigaud, 2018). However, the TD3 has both actor and critic, which are trained by gradient from experiences generated by CEM. And then, the TD3 actor periodically participates in CEM population as in ERL (Khadka Tumer, 2018). This structure is an effective way to maintain direct policy search of CEM. To enhance the performance, we introduced new interactions processes between the CEM and TD3 instead of carrying out a policy gradient for numerous individual actors.
3 BACKGROUNDS
Twin Delayed Deep Deterministic Policy Gradient (TD3) RL framework has an agent interacting with environment generally defined by Markov Decision Process (MDP). At each timestep i, an agent receives the state 8t, and takes an action d₂ according to the policy R, and then receives a The goal of RL is find the policy that
reward rt and the next state st-1 at next time step 1. maximizes the discounted cumulative return R, Σx= k= f where 7 is discounted factor. Off Off-policy RL can use the data from arbitrary policies train actor and critic functions repeatedly, which is key point for improving recent gradient-based RL. Silver et al. (2014) introduced the off-policy Deterministic Policy Gradient (DPG), which has an advantage for high-dimensional action spaces. The DDPG (Lillicrap et al., 2015) network. TD3 (Fujimoto et al., 2018) is an advanced version of the DDPG, which suffers from the overestimation bias of the critic. To correct this bias, two critics are introduced, and then the critic with the lowest state-action value is taken during the critic update as in the Double Deep Q-Network (DDQN) (Van Hasselt et al., 2016). Figure 2(a) represents the architecture of the TD3.
extended from the DPG to apply it to deep neural
Environment
Generate population
(B,, Return,
8 Baltee
Replay
< Se 86- >
BaN N(I. E)
Fixed evaluation steps
Action at
Buffer
State St
[0 Returny)
(s..
CEM: (Butser Ep-update
Top
Gradient based update
Ogtice of the with highest Return
Off-policy PG (TD3)
I yk N,
Actor
Critic
(Traa)
(Qrea)
Σ M A (8, (0,a)(0 -Paa)" agl.
(a) TD3
(b) CEM
Figure 2: Architecture of TD3 and CEM. 0, is sampled from N(, E), A is a weight depending on the rank of the Return, and K is fixed number about high performing actors.
Cross Entropy Method (CEM) The Evolutionary Approach (EA) heuristic search method inspired by nature, where the current population is evaluated, and the next population is regenerated using the current evaluation result in order to produce a higher Return, which is also known as Fitness and defined as a cumulative sum of immediate reward for a fixed number of steps. The Estimation of Distribution Algorithm (EDA) is a class of the EA: updates the target distribution to generate better population. Depending the update method for the distribution, EDAs are classified as CEM (De Boer et al., 2005), a Covariance Matrix Adaptation Evolutionary Strategy (Hansen, 2016), an Evolutionary Strategy (Salimans et al., 2017), and a Natural Evolutionary Strategy (Wierstra et al., 2014). We used the CEM as one part of our proposed algorithm. As shown in Figure 2(b), the CEM procedures follows: The population is sampled from the multivariate Gaussian N N(a, Σ) and
3

---

Under review as a conference paper at ICLR 2021
evaluated, and for the top K, which is smaller than the population size (N), high performing actors are used to update a new mean (/)) and covariance (E) of the target distribution. The weight can be given to each actor according the rank of the Return (Hansen, 2016). The elite actor can be passed to the next population, which is known as elitism. The more detailed procedure reported in (De Boer al., 2005). In this paper, used diagonal covariance reduce the parameters.
Population-guider Parallel Policy Search (P3S) The Guided Policy Learning (GPL) is commonly used when the clite policy leads some sub-policies to a better solution in multiple policies setting, Teh al. (2017) introduced GPL for joint learning of numerous tasks in which common policy encourages local policies to act better. Jung et al. (2020) proposed a soft-manner GPL, called the copulation-guides Parallel Policy Search, for multiple identical learners with the same objective, where a population is evaluated periodically. Then sub-policies are trained to maximize their critic value and to concurrently minimize the distance from the elite policy for the next period. For this purpose, Augmented Loss (2) is used to train the sub- sub-policies instead of Original Loss (1).
Original Loss: L (T) Essss( Q+ TT))]
Auginented Loss: L ( Tetite: 3) = Es-ss Q- क (s)) + (1( ( [
(2)
where is a trained policy, Q is critic function depending on n, Telite is the elite policy, S$ is the set of states, and (s) Tetite [l is the Euclidean distance measure between the trained policy and the elite policy. 3 is distance weight and is controlled adaptively. In this paper, we used revised GPL inspired by P3S that the CEM elite actor guides the TD3 to better space.
COUPLING POLICY GRADIENT WITH POPULATION BASED SEARCH ALGORITHM
(5. Fu
[T Return,
11:NN
T2
(sp Tg 5p
[12 Return))
Increasing evaluation steps (T)
TTN
(S St
ITy. Returny
Off-policy PG (TD3)
Gradient based update
TN Typ3
Actor
PG guides EA
(mynn)
((s,ar,,)))
Replay Buffer
Critic (Qrna)
Top K
Q-critic filtering
TTy
2-1N
Select best actors using
State set SS ((s))
ss (s.
EA(r guides PG
FLM
Generate Population
CEM
N
T2 4- Teitter x
E), RIM (Mx)))) N(
(Nellte update
Figure 3: Architecture of PGPS. used instead of Ty for the simplification.
As shown in Figure 3, the Policy Gradient with Population-based Search (PGPS) is a coupling framework between the CEM and the TD3. In this framework, two algorithms encourage each other to be efficient. The general flow is as follows. The generated actor population is evaluated by interacting with the environment, where each actor can fail to reach the max step (2). The experience (the set of state transitions) is saved in a replay buffer, and Returns go to the CEM. The CEM updates the parameters of the population using top K (K is set half of the population) high performing actors, and then the TD3 trains its critic and actor using the mini-batche batches sampled from the replay buffer. The knowledge of TD3 is periodically transferred the CEM by copying the TD3 actor
4

---

Under review as a conference paper at ICLR 2021
into the last actor of the CEM population. The knowledge of the CEM is transferred to the TD3 by the GPL when the TD3 is presumed to falls into the inferior solution. Lastly, the next population is regenerated, where the TD3 critic is used to selects promising actors among the set sampled from (1,, Σ) so that the population will be better. A pseudocode of the PGPS is described in Appendix B.
In a standard CEM, the experience of state transitions is discarded imme- diately because they require only Returns to update the target distribution. However, the TD3 enables the discarded experience to be reused to train its actor and critic functions. Therefore, the experience is saved in the replay buffer and then is repeatedly used in gradient- based update.
Gradient-based Update
PG Guides EA In order transfer the knowledge learned in TD3 to the CEM, the TD3 actor is periodically copied to the last actor of the population. If the last actor is included in the top-perforning K actors, a multivariate Gaussian move to the direction indicated by TD3. On the other hand, if the TD3 actor is excluded from the top-performing K actors, the CEM ignores the knowledge from the TD3 and focuses on the direct policy search. High copying frequency benefits knowledge propagation from the TD3 to the CEM, but it can disturb the population-basee search.
EA Guides PG When the TD3 falls into the inferior solution, mainly due to a deceptive gradient, difficult to get out by relying solely on the gradient-basse method despite the experience from good behavior policy (Colas et al., 2018). Therefore, we use the Guided Policy Learning (GPL), where the CEM elite actor leads the TD3 to escape the inferior solution. We judge the TD3 to have fallen into the inferior solution if its actor shows a lower Return than (mean one standard deviation) of Returns of the current population. The TD3 actor that has fallen into the inferior solution is updated for the direction to minimize the Augmented Loss L,A, equation (2). Moreover, the TD3 critic is trained through the target actor indirectly guided by the elite actor, and it will fix the critic to be appropriate. The distance weight (3) in the in L is adapted several times during the GPL-based update by equation (3). It is a simplified version of P3S (Jung et al., 2020) and similar to Adaptive TRPO, which was introduced in (Schulman et al., 2017).
B x 2
if D(TTDS, Felste) Felste) > Drarget X 1.5
8
(3)
1 / 2
if D(TTDa Wetste) < Dtarget / 1.5
where the distance measure D (TTDS, Telite) is defined as Eswss (D Telite (s) [|]. SS is the set of states in the replay buffer, and Diarget as a hyper-parameter determines how close the TD3 actor and the elite actor should be. During the GPL-based update, the TD3 actor stays around the CEM elite actor while maximizing its critic value.
Q-critic Filtering It is important to generate a good population, since it not only leads N N(u, )) to be better but also encourages the TD3 to be trained well by filling the replay buffer with good experience. However, one cannot know which actor is better before the evaluation. To estimate the potential in advance, we use the Q-critic as surrogate model (Jin, 2011). 1). It can sort out promising actors from the set of actors sampled from N(p, Σ) before the evaluation.
Proposition1 If Ea (1))) 2 E Ea-trost (s) (Q a)] for all s, E. Ea (11)))Q (s, a)) 2 F Fawarpallan (QDDs(S, a))
Specific proof is described in Appendix C. We assume that a higher Eawa, (QrDs(s, a)] indicates a higher
(s, a).. In this assumption, Ex-ss Taass(QQppal (QTpa (s, (+)) is used as the relative potential of the actor (TT) where SS is the set of states in the replay buffer. The overall procedure is as follows: M (*N) actors are sampled from N (p, )), and then Q-critic fills half of the population with the actors with higher relative potential by filtering M actors. The remaining half consists of the elite actor and the actors sampled from N(u, Σ) for the exploration.
Increasing Interaction Steps In order efficiently control the trade-off between the frequency of searches and stable performance estimation of actors, we used evaluation steps scheduling, where the evaluation steps between actor and environment increase with cumulative evaluation step by equation (4).
T (Tint + 100 x modlcummulative evaluation steps, Tinter), ) T Tmax)
(4)
5

---

Under review as a conference paper at ICLR 2021
where T' is the current evaluation step, which means that each actor can maximally interact with the environment as much as T. Tisst is the initial evaluation step, Tinter is the interval for increasing evaluation step, and mazz is the maximum evaluation step depending on the task.
The evaluation step should be sufficiently long for stable performance estimation of the population, but it reduces the number of the CEM generation and delays the update of the TD3. Short evaluation steps make it possible to carry out more population-bas baaed searches and frequent TD3 updates but causes unstable performance estimation. As the arbitration approach, we used the increasing evaluation steps. This approach guarantees more searches and frequent updates at the beginning stage of learning and fine estimation the later stage of learning.
EXPERIMENTS
5.1
COMPARISON TO BASELINES
Environment The proposed algorithm (PGPS) was empirically evaluated on five games in Mujoco managed by OpenAl Gym, which is widely used as continuous control benchmarks.
Baseline Algorithms We compared the performance of PGPS with various EA, PG, and EAPG algorithms, such CEM, PPO(Schulman et al., 2017), DDPG(Lillicra) et al., 2015), TD3(Fujmmoo et al., 2018), SAC(Haarnoja et al., 2018), CERL(Khadka et al., 2019), PDERL(Boddar et al., 2019), and FEERL((Purcee provided by its authors. However, the DDPG is implemented by the code provided by the authors of the TD3. OpenAl Spinningup (Achiam, 2018) was used implement PPO. PGPS and CEM were implemented using PyTorch. Our code is available at http://github.cc http://gthuu..oom/aaaaa comm/Namiim8//GPS...
& Sigaud, 2018). Almost algorithms were implemented using the code
Hyper-parameter Setting For stable learning, we performed tuning on the architecture of the neural network and the hyper-parameters of the learning rate, population size, the period of copying TD3 actors to the population, increasing evaluation steps, and Q-critic filtering. The detailed setting described in Appendix D. Adam (Kingma & Ba, 2014) was used to train the neural networks in the TD3. The hyper-paranee parameters of baseline algorithms were set as the same as the reference code.
Performance Evaluation Metrics Each algorithm learns five times at different seeds for a task. Each learning runs for a million timesteps. That is, the total numbers of interacting all agents with the environment are a million. The evaluation tests are performed every 10,000 steps. Each evaluation test performed without any exploration behavior and reports the average reward over ten episodes, where the evaluation step of episode is 1,000. For the evaluation test of the PGPS, the current mean (μ) of (1, Σ) was used as the parameters of the evaluation policy. If the learning performed at seed 1, the evaluation test will proceed at random seed + 100. This approach is the same as the original code of the TD3 (Fujimoto et al., 2018) and applied to all baseline algorithms. The curves in Figure 4 reported over the average performance of policies trained at five random seeds from to 5.
Results In Figure 4 the performance of all baseline algorithms is similar to that reported in the orig- inal author's papers and reference papers. Some differences come from the training seed, evaluation metrics, the variance of algorithms, and the version of MuJoCo and Pytorch.
The results show that all PG algorithms suffer from deceptive gradient in Swimmer-v2. In contrast, the CEM, which is direct policy search method, yields the best result over a deceptive gradient in Swimmer-v2. The CERL and PDERL as combining algorithm between GA and PG show better performance than PG algorithms. However, their performances are lower than the CEM because the knowledge propagated from the PG to the GA disturbs the direct policy search of the GA. Despite the CEMRL combining algorithm between the CEM and TD3, it shows similar performance to the TD3. This result comes from that the gradient- based update of half of the population ruins the population-based search of the CEM.
In the remaining four tasks which are advantageous to the gradient- method, advanced PG algorithms show much better performance than the CEM. The performances of CERL and PDERL are located between an advanced PG algorithm and the CEM in most cases. Especially, CERL yields lower performance than the TD3, which is one part of CERL, This is due the failure of GA and

---

Under review as a conference paper at ICLR 2021
HalfCheetah-v/
Hopper-
Average Episode Return
10000
250
5000
200
2000
130
100
4000
1500
1000
2000
500
200000
400000
800000
B00000
200000
400000
800030
B00000
1000000
200000
400000
SC0000
B00000
Walker2d.v2
PGP5
Average Episode Return
CEM
5000
4000
TD3
3000
SAC
2000
CERL
1000
PDERL
CEMRL
c000co
600000
Total timestep
Total timestep
Figure 4: Learning curves obtained on 5 continuous control tasks.
TD3 combine effectively. CEMRL outperforms all baseline algorithms in four tasks since the gradient-based update for multiple actors amplifies the advantage of the gradient method.
PGPS carried gradient-based update the TD3, and then the TD3 actor is periodically copied to the last actor of the CEM population. keeps the computational cost lower and also minimizes the disturbance from the gradient method on direct policy search. As result, the PGPS can achieve results comparable to the CEM in Swimmer-v2 which is advantageous to a direct policy search. Furthermore, the PGPS shows an outstanding performance in the remaining four tasks, which are advantageous to the gradient-bart based method. That performance is due to the additional interaction processes for coupling two algorithms efficiently, such as mutual guidance (PG-EA), Q-critic filtering, and increasing evaluation steps.
5.2 ABLATIONS STUDIES
In this subsection, performed ablations studies to investigate the effect the final performance and computational time when a specific process is cumulatively added to the base model. The added sequence is follows: PG guides EA (P to E), EA guides PG (E P), Q-ilttring, and increasing evaluation steps. HalfCheetah-v2. Walker2d-v2. and Ant-v2 are used for the ablation studies. Each task was learned five times random seeds of 1 to 5. Each learning for million timesteps. Table reports the average over 15 (three games x five learning) runs. Evaluation test is performed using the μ of the CEM except for the Base model.
Base model The experience from CEM is saved to replay buffer. However, the TD3 and CEM are trained independently without any guidance. Two evaluation tests are performed using the TD3 and the μ of the CEM at the end of learning. A better one is selected for the final performance.
Table 1: Ablation studies on continuous control tasks.
Normalized computational time
Average normalized score
Model
TD3
Actors
Population
CEM
Sum
update
evaluation
generation
update
Base model
1.00
0.934
0.065
0.001
2 0
(I) to
3.73
0.934
0.065
0.001
R 0
33) (2) + E to P
4.26
1.107
1.041
0.065
0.001
2 0
(3) Q-filtering
4.44
1,134
1.041
0.065
0.028
2 0
5
()) Increasing
5.26
1.134
1.041
0.065
0.028
2 0
P to E shows the most noteworthy performance improvement within the proposed algorithm. A good experience is essential to train TD3's actor and critic functions well. As the TD3 actor directly guides the population be better, the population fills the replay buffer with good experience, and then it
7

---

Under review as a conference paper at ICLR 2021
makes sure the TD3 will be trained well again. For the perspective of the CEM, the TD3 actor is good exploration, which cannot be developed from a random sampling, and improves the population to be better. For the perspective of the TD3, the experience from the population is richer set of exploration behavior, which is better than PG-hased action space exploration (Plappert et al., 2017 7). The other processes also additionally contribute to performance improvement of about 4 18% The remarkable point is that these processes incur a low additional computational cost. The Q-filtering requires relatively highest computational cost, but is only about 11% of that of the TD3 alone.
The effect of EA guidance In contrast to the existing hy- brid algorithm (Bodnar et al., 2019; Khadka et al., 2019) in which the PG only guides the EA, the proposed algorithm lets the EA also guides the TD3. It is beneficial to pull the
Average Episode Return
350
CEM
300
TD3 with EA guidance
TD3 without EA guidance
250
200
TD3 agent out of the inferior solution, especially if it is
150
captured due to a deceptive gradient. Figure 5 represents the effect of EA guidance (EA guides PG) in swimmer-v2. As shown in Figure 5, the TD3 actor without EA guidance stays on the inferior solution most of the time, whereas the TD3 actor with EA guidance quickly gets out of the inferior solution and goes on searching for better space. The proposed algorithm will also perform robust learning
100
50
200000
400000 600000
Learning curves of TD3 with(out) EA guidance on Swimmer-v2 even both deceptive and general gradient occurs in the environment since It lets CEM lead TD3 when deceptive gradient occurs and lets TD3 lead CEM when the general gradient occurs.
Figure 5:
HalfCheetah-v2
The effect of Q-critic filtering The proposed algorithm selects half of the population using Q-critic (Q-filter), and the remaining half is sampled from multivariate Gaussian (Random). Figure 6 shows the average Return of Q-filter and Random in HalfCheetah-v2. Q-filter shows a 7.7% higher average performance than Random. (Because the evaluation steps were controlled, the average Return in Figure 6 is lower than the test episode return in Figure 4.) This result is additional evidence that Q-critic filtering is meaningful for sorting out promising actors. The remaining concern is the effect of Q-critic filtering on the exploration. If Q-critic selects similar actors, it might deteriorate the
10000
Average Return
8000
6000
Q-lilter Random
2000
200000
400000
600000
800000
Figure 6: Average Return of filtered ac- tors and sampled actors during running exploration behavior for both the EA and the PG. To further improve exploration, the distance-based criteria introduced in (Bodnar et al., 2019) can also be used with Q-critic filtering.
6 CONCLUSIONS
In this paper, we proposed another hybrid algorithm coupling the Cross-Entrops Method (CEM) and the Twin Delayed Deep Deterministic Policy Gradient (TD3), which is called Policy Gradient with Population- based Search (PGPS). Proposed algorithm is not only robust to a deceptive gradient, which is difficult to be learned by the TD3 alone, but it also achieves higher sample efficiency, which is deficient in the EA alone. To be robust to a deceptive gradient with low additional computational cost, revised the existing hybrid algorithm framework make it an improved structure. Also, to enhance the performance, we introduced new interaction processes such as mutual guidance (PG+EAA), Q-critic filtering, and increasing evaluation step. First, mutual guidance the most crucial process, where the TD3 guides the CEM to make it better, and the CEM also guides the TD3 that has fallen into the inferior solution to make it better. Second, Q-critic helps the population to consist of more promising actors by filtering the set of actors sampled from a multivariate Gaussian. Lastly, the increasing evaluation step controls the trade-off between the frequency of searches and stable estimation. It takes the frequent searches and updates when searching on a coarse policy space far from the optimal at the beginning stage of learning and fine estimation close to the optimal at the later stage of learning. Our experiments a MuJoCo confirmed that the proposed algorithm outperforms state-of-tt ffate-of-the-as art PG and hybrid algorithms, while its computational cost is kept at about 13.5% 5% higher than the TD3 algorithm.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam. Openai spinning up. GitHub, GitHub repository, 2018.
Cristian Bodnar, Ben Day, and Pietro Lió. Proximal distilled evolutionary reinforcement learning. arXiv preprint arXiv:1906.09800 2019.
Cédric Colas, Olivier Sigaud, and Pierre-Yvel Oudeyer. Gep-pg: Decoupling exploration and
exploitation in deep reinforeement learning algorithms. arXiv preprint arXiv: 2018.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-secking agents. In Advances in neural information processing systems, pp. 5027-5038, 2018.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 2018.
Tanmay Gangwani and Jian Peng. Policy optimization by genetic distillation. arXiv preprint arXiv:1711.01012. 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:8801.01290. 2018.
Nikolaus Hansen. The cma evolution strategy: A tutorial. arXiv preprint arXiv: 1rXiv:1604.0077. 2016.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: /ggthtgttthh///gg com/ 2018.
hill-al
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09844 2017.
Yaochu Jin. assistd evolutionary computation: Recent advances and future challenges. Swarm and Evolutionary Computation, 1(2):61-70, 2011.
Whiyoung Jung, Giseung Park, and Youngchul Sung. Population-guider parallel policy search for reinforcement learning. arXiv preprint arXiv:2001.02907, 2020.
Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1188-12 1200, 2018
Shauharda Khadka, Somdeb Majumdar, Santiago Miret, Evren Tumer, Tarek Nassar, Zach Dwiel, Yinyin Liu, and Kagan Tumer. Collaborative evolutionary reinforcement learning, arXiv preprint 19X.v:190..00976 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.69800 2014.
Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O Stanley. Safe mutations for deep and recurrent neural networks through output gradients. In Proceedings of the Genetic und Evolutionary Computation Conference, pp. 117-124, 2018.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 00t./5099900229 2015.

---

Under review as a conference paper at ICLR 2021
Guoqing Liu, Li Zhao, Feidiao Yang, Jiang Bian, Tao Qin, Nenghai Yu, and Tie- Yan Liu. Trust region evolution strategies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4352 2352-4359, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 7777.09069990 2017.
Alois Pourchot and Olivier Sigaud. Cem-rl: Combining evolutionary and gradient-based methods for policy search. arXiv preprint arXiv:/810.0000 2018
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, arXiv preprint arXiv:1703.03866 2017.
Juergen Schmidhuber and Jieyu Zhao. Direct policy search and uncertain policy evaluation. In Aaai spring symposium on search under uncertain and incomplete information, stanford univ, pp. 119-124, 1998.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. 2014.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of without human knowledge. Nature, 550(7676)3354-333 359,, 2017.
Joe Staines and David Barber. Optimization by variational bounding. In ESANN, 2013.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth Stanley, and Jeff competitive alternative for training deep 2017.
Clune. Deep neuroevolution: Genetic algorithms neural networks for reinforcement learning. arXiv preprint
Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496 4506, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
George Uhlenbeck and Leonard S Ornstein. On the theory of the brownian motion. Physical review, 36(5):823, 1930.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. Thirtieth AAAI conference on artificial intelligence, 2016.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949-99) 980, 2014.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX A. THE SHAPE OF SWIMMER-V2
[c]
Treel
(d)
(e)
(f)
Rdut
Figure 7: The graphs represent the change of average episodic discounted reward average episodic return of linear policy actor by the value change of single parameter in the Swimmer, where the other parameters were fixed and we evaluated thirty times at different seeds for each policy.
Average Episode Discounted Reward (- Ei 7(r
E
Σ Ee 7'r
7'rg
Average Episode Return Ex(1s) E t= ril
999 30 E t= r?
where 7 0.99, and the architecture of the linear policy [state dim, action dim] -> tanh
To find the reason why the previous sfate-of-the-re policy gradient methods, such as TRPO, PPO, SAC, and TD3, were hard to solve the Swimmer in the Mujoco Enviromment, performed an interesting two-steps experiment. In the first step, we sat an actor, following a simple linear policy, and executed the CEM algorithm find optimal policy parameters 0 in the Swimmer. When the Return value of the actor reached over 200, we saved the policy parameters 800. In the next step, while changing one parameter value, we evaluated the changed policy thirty times with different seeds and recorded the all Discounted Rewards (J (S))) and Returns. We represents the some cases of the experiment in Figure 1. As you can find in the Figure 1, we can discover interesting facts: I) on the graphs (a), (d), and (f) in Figure 1, we can find wide regions, where the gradients ()J((,)) are near zero; 2) except for (b) in Figure at particular parameter value, the gradient of J J(So) and Return is steep enough to be seen
piece-wise; and 3) on the graphs (c). (d) and (e), the graphs are shaped like valleys neat at those steep points. We suspected the above facts as the cause of the deceptive gradient problem of the Swimmer, and raised the question in the Introduction. Finally, by considering those issucs, we propose the combined algorithm with TD3 and CEM.

---

Under review as a conference paper at ICLR 2021
B APPENDIX B. PSEUDOCODE OF PGPS ALGORITHM
Algorithm 1 Coupling Policy Gradient with Population based Search Algorithm
Set hyper-parameters: TD3: Iractor Faaaarrrrrrritee oriitie Y; and w; CEM: pop-size N, top K, Dinst: Lend,
and Mutual guidance: PTD3-CEM Binit, and Diarget Q-critic filtering: - and SR: Increasing interaction steps: Tanzy Timst; and Tinterval
and
1: Initialize the mean of the multivariate Gaussian of the CEM
2: Initialize the TD3 actor D3 and TD3 critic QTD3
3: initialize replay buffer R
4: total_steps- =
5: for generation=1:00 do
6: if total steps 2 sttart-Q then
7: pop Q-critic Filtering(N, Telite, M1, E, QTDS, R)
8: else
9: pop[1] <- Tetite: N] are sampled from N(,, E)
10: end if
11: if generation mod FrDS-CEM then
12: pop[N] + TrDs
13:
end if
14:
T= min((imit 100 mod(tolal_stepps TeeoppssTneeee Tintervat): Tmax)
interaction peeppiiooonstsssseeeee 0 for i=1:pop_siize size N do
16:
Set the current actor и as pop[i].
18:
Return;, (St, 0z
(ttealltt < Evaluate(1t π,, T)
19:
Fill replay buffer R with (sy, dz, Tt, $141)==1:6md
interaction steps-ssppp-pp interaction steps + fend
21:
end for
22:
total steps total_ stel_stepss> + interaction_stepy
23:
Update (Telite μt 2) with the top-K Return actors
24:
num _nd_upeteee = interaction_
25:
if generation mod FYD3-CEM 0 and Returny MENNGReuurrs)))TT((etrraa then
26:
for i=1:5 do
27:
Sampled states (SS5) drawn from R
Update 8 8 x 2 if Eg E. - T Telite (a)1 (elite()))
> D Diarget X 1.5
28:
2 if Eg (TDD((s Tetite (s) Dtarget / 1.5
29:
Train Q for num gum_update mini-batches from R using standard TD3 algorithm Train TT for num_update mini-batches from R to minimize [AA(TTDS D3, Welite, (3)
30:
31:
end for
32:
else
33:
for
i=1:5 do
34:
Train Qr for update mini-batches from R using standard TD3 algorithm Train TrDS for num son_updtte mini-batchee from to minimize L (TTDS)
35:
36:
end for
37:
end if
end for
In contrast to standard TD3(Fuimooto al., 2018) performing one step interaction with environment and then one update repeatedly, our TD3 carries out many updates as the sum of the evaluation steps of the current generation after the evaluation. In proposed algorithm, the total updates steps is divided into 5 iterations. At each iteration, critic firstly trained for fixed number of steps followed by an actor trained for same steps. If total update steps are 10,000, at each iteration, critics is firstly trained for 2,000 minibatches, and then actor is trained for 2,000 minibatches for the direction
12

---

Under review as a conference paper at ICLR 2021
to maximize critic. For stabilizing the volatility of critic, it is widely used in the implementation (Achiam, 2018; Hill et al., 2018; Pourchot & Sigaud, 2018). L (TTDS) is original TD3 Loss in equation (1). L (TT Retite: 3) is augmented loss for guided policy learning in equation (2).
Algorithm 2 Function Q-critic Filtering
1: procedure Q-critic Filtering(N, Telites ft Σ, QTD3, R)
2: pop[1] - Welites pop[2 N/2] are sampled from N NJ,, Σ)
3: (/=1:M(=SR+N) are sampled from (u, Σ)
4: Sampled states (SS,) are drawn from replay buffer
5: for j=1:M do
6: P3 = Esoss, (a x,)l
7: end for
8: pop[NN/2 + 1, M] < Select policies (TS) with higher Pj among Tj=1:M
9: Return pop
10: end procedure
Algorithm 3 Function Evaluate
1: procedure Evaluate(Taditte: 7)
2: returns, buffer (BF) 0, 0
3: Reset environment and get imitial state
4: while
not done and
do
5: Select action a₂ (T(ss)
6: Execute action a, and receive reward ri and next state 8141
7: Fill BF with stat transition (st, (z, Tt, 8141)
8: relurns relurns + Tt and 141
9: end while
10: Return returns, BF
11:
end procedure
In standard TD3 algorithm, Gaussian noise or noise are added the action o, for exploration. It usually known action space noise. Pourchot & Sigaud (2018) empirically showed that action space noise does not contribute the performance improvement. We also cannot find any evidence for the advantage about action space noise. Therefore, proposed algorithm does not use action space noise.
& Ornstein, 1930)
13

---

Under review as a conference paper at ICLR 2021
C APPENDIX C. PROOF OF PROPOSITION
In this section, we prove Proposition
Propositionl If (o 2 (ol a)] for all (1a)) (8, a)] 2 (awarda((1e) QTDs3 a)
Proof. For arbitrary
VarDa (8t)
(1) (80, at)]
S Fa (aT 0.)
Eu 17 Ba (ea (441) 0+11)l
(an) (r? + F (acesoa (Q (8+11,
Ea I7 rr2t + 7² E 1anarssl 2) ( (1 044)]
Ea +rrl + 72 1or 0142)
$ ( Ii +r74 + + 7, 13
E Ek=l?? 2
= Eawns(-ss) (s,
(s.)
We assumed that higher policy ı with higher 1== (Q (s, a)] better. We used sampled state (SS) from repay buffer to estimate the performance of policy. That is, deuss|QrrDsl [s)))] is estimator for (e)aaaa a)] for all s. Sum up, the policy with higher Es-ss (Q (x))) is better policy.
a)] means higher En 1r, a)]. Therefore, the
14

---

Under review as a conference paper at ICLR 2021
D APPENDIX D. DETAILED HYPERPARAMETERS SETTING
Table 2 includes the architecture of the neural networks. Table represents the hyperparameters that
kept constant across all tasks. Table 4 describes the hyperparameters that vary with the task.
Table 2: The architecture of the neural networks.
Actor
Critic
[state dim, 400]
[state dim + action dim, 400]
elu [400, 300] elu [300, action dim] tanh
clu [400, 300] elu [300, I]
Table 3: Hyperparameters constant across all tasks.
Hyperparameter
Value
Target weight (T)
0.005
TD3 Actor learning rate (Ir
2e-3
TD3 Critic learning rate (Ireiitie)
le-3
Discount factor (7)
0.99
Replay buffer size
le-6
Batch size (x)
256
CEM initial covariance (Eiit)
7.5e-3
CEM limit covariance (Simat)
le-5
Initial distance weight (Sinit)
1
Target distance (Drorget)
0.05
Sampled states (SS)) size
512
Q-fittring start step (7 (Tstart-Q Q)
150000
Multiple sample ratio (SR)
50
Sampled states (SS,) size
64
Max interaction (Tmax)
1000
Initial interaction steps (T imit)
400
Interval for increasing step (T (Tinter)
100000
Table 4: Hyperparameters varying across tasks.
Task
Swimmer
HalfCheetah
Hopper
Walker2d
Ant
Population size (N)
10
10
6
6
6
top K
5
Frequency of TD3 to CEM (FTD3-CEM)
3
Decaying covariance constant (7x)
0.01
0.03
0.03
0.03
0.03
15


Reviews:
[{"review_id": "xlWfvP0uVF", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new method combining evolutionary methods and RL. In particular, the authors combine CEM and TD3 in PGPS. PGPS maintains a population of policies, which interact with the environment to collect data filling the replay buffer. The data in replay buffer is then used to train TD3. PGPS enables information flow in both directions: when the TD3 policy performs poorly, the elite policy from the population is used to guide TD3 by an imitation learning loss; The TD3 critic helps select top policies in the population and the TD3 actor is also included in the population. The experiments on simple Mujoco domains demonstrate the utility of PGPS and the ablation study analyzes the utility of each part of PGPS.\n\nMy major concern is that the experiments are not convincing enough. \n(1) I think TRES (Liu et al., 2019), Salimans et al. (2017), and PBT (Jaderberg et al., 2017) are important baselines that the authors need to compare with.\n(2) The reported results are based on only 5 random seeds which I don't think are enough. \n(3) The experiments have only 5 domains, I think at least Humanoid should be included. Moreover, as PBT does demonstrate advantages in video games (e.g., Atari, DMLab), I'd like to see if PGPS can scale up to video games as well.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pUjTt7a49x8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This a nicely written paper.  The authors propose an algorithm that combines population-based search with policy gradient approach to obtain optimal policies in reinforcement learning. The proposed combination leads to a yet another algorithm aiming at exploitation (TD3) and exploration (CEM). The idea of using such combinations is not new (a point also acknowledged by the authors). The results are promising, though these results are obtained with some additional computational cost. I believe this manuscript should be considered as an implementation work, where the novelty of the idea is not clear. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "nVwyhJoRgD5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nRecently, several researchers have been trying to combine the goodnesses of direct policy search approaches (mostly based on evolutionary computation approaches)  and those of policy gradient approaches in control tasks. This paper proposes a novel combination of an evolutionary direct policy search and an actor-critic approach. The authors combines a cross-entropy method, which directly samples parameters of actor network (policy) from a Gaussian distribution that is trained during the search process, and the twin delayed deep deterministic policy gradient (TD3), which is an off-policy actor-critic approach. \n\nIn existing combination of an evolutionary direct policy search and an actor critic approach, the evolutionary search part contributes to fill the replay buffer with diverse experiences, whereas the actor critic approach leads the population of the evolutionary search by replacing subset of the population with the policy of the actor critic agent, which is often expected to be superior to the population of the evolutionary search. Differently from these existing researches, the proposed approach also make an elite population to guide the actor critic agent when it is stacked at an area where the policy gradient almost vanishes. This idea comes from the recently proposed P3S-TD3, another population based (multi-policy) TD3 approaches.\n\nComparison has been conduced on 5 MuJoCo environments. In Swimmer-v2 environment, the proposed approach reached the performance between the pure cross entropy method (best) and other combinations of evolutionary approaches and policy gradient approaches. In the other four environments, the proposed approaches reached the competitive or superior average performance. \n\nCriticism:\n\nCompared to the results presented in P3S-TD3 paper (ICLR 2020), which used a slightly different environment and therefore the direct comparisons may not be valid, differences observed in this paper between the proposed approach and the pure TD3 is rather similar to the differences between P3S-TD3 and TD3. Since one of the main contributions of this paper, if I understand correctly, is the introduction of the component borrowed from P3S-TD3, P3S-TD3 should be included in the baseline approaches in the comparison. Is there any advantage of the proposed approach over P3S-TD3 on HalfCheetah, Hopper, Walker2d and Ant? \n\nSwimmer environment is the only tested environment where CEM works significantly better than TD3. On this environment the proposed approach was shown to be as effective as CEM. Does it generalize to other environment where CEM works significantly better than TD3? The current evaluation lacks the evaluation of the generality of this approach. \n\nIn the ablation study (Table 1) the effect of Equation (4) was evaluated, and shows a non-negligible impact. Equation (4), which controls the number of interaction steps performed in each MC evaluation in CEM, can be incorporated to other approaches such as CEMRL, and if the same impact appears in the existing approaches, it is ambiguous whether the other components really contribute to improve the performance over existing approaches. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ERJxryzAkO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** The authors propose PGPS as a minor variation on [CEM-RL](https://arxiv.org/abs/1810.01222). \n\n**Quality:** The quality of the paper is overall very low: misleading and unsustained claims, poor and incorrect literature review, cherry-picking results.\n\n**Clarity:** While the exposition overall follows a standard template, the writing is in dire need of intensive copy efforts prior to being considered for publication.\n\n**Originality:** The authors present a minor augmentation of CEM-RL while claiming some of its results as their own.\n\n**Pros:**\n- The experimental part seems to have received a lot of work\n- The setup and hyperparameter description is very complete, coupled with open sourced code, potentially allowing full experiment reproduction\n\n**Cons:**\n- The first sentence of the Conclusions claims the pairing of CEM and TD3 integrating policy gradient and population-based search as a novelty and contribution, while it was introduced in the (cited!) CEM-RL.\n- There is no clear, explicit definition of the paper's contributions, which given the circumstances is unacceptable.\n- Some results (particularly regarding the Q-critic filtering) are sketchy and not statistically meaningful.\n- The Related Work study in Section 2 is riddled with incorrectness in the evolutionary part. For example: stating EA stands for Evolutionary Approach rather than Algorithms; referring to CMA-ES by citing a 2016 tutorial instead of the [2001 paper](https://scholar.google.ch/scholar?q=completely+derandomized); equating EA to direct policy search; stating that previous-generation data cannot be reused; and many more. This overall shows a less than perfunctory understanding of the field. The parallels with the CEM-RL paper are once again obvious and misquoted.\n- The paper writing requires a thorough overhaul.\n\n**Comments:**\nThe contributions are not made clear, and a quick read of the original CEM-RL paper shows an exact correspondence but for 1. the double-feedback loop and 2. the Q-critic filtering. These two contributions however are very minimal, and most importantly they are not tested in isolation: end of Section 5, 1. \"The effect of EA guidance\" does not compare with CEM-RL (or other hybrid methods), and 2. \"The effect of Q-critic filtering\" drops the variance information in favor of what looks like single runs with statistically indistinguishable results.\n\n**Final remarks:**\nThis work is a barely-augmented, error-riddled rewriting of a 2019 paper, and its state is nowhere near publishable. I strongly advise the authors to decide if their contributions are sufficient for publication, and if so to write a paper dedicated to proving their point to the scientific community. \nFinally, the un-anonymized GitHub link on Section 5.1 breaks the code of conduct; the fact that the author (NamKim88) is less known does not excuse the breach of anonymity.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a hybrid algorithm that combines RL and population-based search. The work is interesting and well-written. But, the contribution of the work is very limited, in comparison with the state-of-the-art. 
Decision: Reject