PGPS : Coupling Policy Gradient with Population-based Search

1 INTRODUCTION In Reinforcement Learning ( RL ) , an agent interacts with the environment , and its goal is to find the policy that maximizes the objective function , which is generally defined as a cumulative discounted reward . Recently , many researchers have worked on combining deep neural networks and a gradient-based RL algorithm , generally known as Deep Reinforcement Learning ( DRL ) . This approach has achieved great success not only in the discrete action domain , such as in Go ( Silver et al. , 2017 ) and Atari games ( Mnih et al. , 2015 ; 2016 ) , but also in the continuous action domain , such as in Robot control ( Fujimoto et al. , 2018 ; Lillicrap et al. , 2015 ; Schulman et al. , 2015 ) . However , it is difficult to use the gradient-based method for the objective function ( J ) , which includes “ many wide flat regions ” since the gradient ( ∇θJ ) is near zero at a flat point . Figure 1 is an extreme case consisting of only flat regions , which is called a piece-wise constant function . This problem remains an unsolved issue in gradient-based DRL with continuous control domains ( Colas et al. , 2018 ) . The Swimmer in a MuJoCo environment ( Todorov et al. , 2012 ) has already been reported to be hard to use the gradient-based method ( Jung et al. , 2020 ; Liu et al. , 2019 ) . Our experiment shows that the objective function of Swimmer includes wide flat regions ( Appendix A ) . The population-based Evolutionary Approach ( EA ) , which is an alternative to the gradient-based method , has also shown successful results in various control tasks ( Conti et al. , 2018 ; Liu et al. , 2019 ; Salimans et al. , 2017 ; Such et al. , 2017 ) . As a population-based search , the EA generates a population of agents to explore policy , and the population is regenerated with improvement in each generation . The EA is also known as a direct policy search ( Schmidhuber & Zhao , 1998 ) because it directly searches by perturbing the parameter of policy . In Figure 1 , the Cross-Entropy Method ( CEM ) as a kind of population-based search is simply described , where the current population sampled from the target distribution is evaluated . Then the distribution is updated to the direction for generating a more promising population . Not depending on the gradient , these approaches are robust to flat or deceptive gradients ( Staines & Barber , 2013 ; Liu et al. , 2019 ) . However , the EA is sample inefficient because it requires a Monte-Carlo evaluation , and the previous results and data generally can not be reused . The off-policy Policy Gradient ( PG ) algorithm uses the data from arbitrary policies to train its actor and critic functions . It generates exciting potential by combining the EA and PG , where the data which is discarded in a standard EA is directly used to train the PG ’ s functions . Khadka & Tumer ( 2018 ) and Pourchot & Sigaud ( 2018 ) introduced a framework combining the EA and off-policy PG . However , the framework of ( Khadka & Tumer , 2018 ) is less efficient to train the policy for general tasks than the PG algorithm alone , and the framework of ( Pourchot & Sigaud , 2018 ) is unsuitable to train the policy for a task providing a deceptive gradient . In this paper , we propose another hybrid algorithm , called Policy Gradient with Population-based Search ( PGPS ) in which the CEM and Twin Delayed Deep Deterministic Policy Gradient ( TD3 ) ( Fujimoto et al. , 2018 ) are combined . It is as robust to a deceptive gradient as the CEM and more efficient to train the policy for general tasks than TD3 . To be robust to a deceptive gradient , the proposed algorithm is constructed in a way similar to the one in ( Khadka & Tumer , 2018 ) , where the TD3 is trained using data from the CEM and periodically participates in the CEM population as an individual ( PG guides EA ) . However , in this basic framework , the TD3 sometimes falls into the inferior solution and inefficiently searches . To get the TD3 out of the inferior solution , we let the EA guide the TD3 by guided policy learning ( Jung et al. , 2020 ) ( EA guides PG ) . Furthermore , the TD3 critic contributes to generating a more promising population by filtering the set of actors sampled from CEM ( Q-critic filtering ) . Lastly , to control the trade-off between the frequency of search and stable estimation , we used evaluation step scheduling in the process of population evaluation ( Increasing evaluation steps ) . It carries out frequent searches when searching far from the optimal , whereas it carries out stable estimation when searching close to the optimal . These approaches bring out more synergies between the CEM and the TD3 while maintaining both the population-based search and the gradient-based search . Consequently , the proposed algorithm is not only robust to a deceptive gradient , but also produces outstanding performances with a low additional computational cost . 2 RELATED WORKS . Recently , beyond the view of an alternative approach , few attempts have been proposed in the form of A supporting B . An attempt is to use EA to fill a replay buffer with diverse samples . In Colas et al . ( 2018 ) , a Goal Exploration Process ( GEP ) , a kind of EA , is firstly applied to search the policy and to fill a replay buffer with the diverse samples , and then the off-policy PG algorithm is sequentially used for fine tuning the parameters of the policy . Another attempt is to combine a population-based approach and PG for efficiently searching a good policy or the good hyper-parameters of an algorithm in parallel multi-learners setting . These applications generally consist of periodically evaluating the population , followed by distributing good knowledge to the other learners . To find the best architecture and hyper-parameters , Jaderberg et al . ( 2017 ) proposed a Population-Based Training ( PBT ) method in which the current best knowledge is periodically transferred to PG learners . Gangwani & Peng ( 2017 ) developed the distilled crossover using imitation learning and mutation based on the PG . Proposed operators transfer the information on current good policies into the next population without destructive change to the neural network . Jung et al . ( 2020 ) introduced a soft-manner guided policy learning to fuse the knowledge of the best policy with other identical multiple learners while maintaining a more extensive search area for the exploration . The idea of combining the population-based EA and off-policy PG was recently introduced by Khadka & Tumer ( 2018 ) . Their approach was called Evolutionary-Guided Reinforcement Learning ( ERL ) in which the Genetic Algorithm ( GA ) and the Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) are combined . In ERL frameworks , the GA transfers the experience from evaluation into the DDPG through a replay buffer , and the DDPG transfers the knowledge learned from the policy gradient into the GA by periodically injecting a PG actor into the GA population . Khadka et al . ( 2019 ) expanded the PG algorithm of ERL from a single DDPG learner to multiple TD3 learners with a resource manager . Bodnar et al . ( 2019 ) revised the GA ’ s crossover and mutation to the distilled crossover and proximal mutation inspired by ( Gangwani & Peng , 2017 ) and ( Lehman et al. , 2018 ) to prevent the destruction of neural networks . Pourchot & Sigaud ( 2018 ) introduced another framework , which combines the CEM and the TD3 . In this framework , the TD3 algorithm has only a critic function trained using the experience from the CEM . In order to propagate the knowledge learned by policy gradient to the CEM , half of the population is updated to the direction indicated by the TD3 critic for a fixed number of steps , followed by the evaluation . The policy gradient for half of the population not only enhances the gradient-based learning , but also deteriorate the CEM ’ s robustness over a deceptive gradient . In this paper , we introduce another hybrid algorithm , in which the CEM and the TD3 are combined as in CEMRL ( Pourchot & Sigaud , 2018 ) . However , the TD3 has both actor and critic , which are trained by a gradient from experiences generated by CEM . And then , the TD3 actor periodically participates in CEM population as in ERL ( Khadka & Tumer , 2018 ) . This structure is an effective way to maintain a direct policy search of CEM . To enhance the performance , we introduced new interactions processes between the CEM and TD3 instead of carrying out a policy gradient for numerous individual actors . 3 BACKGROUNDS . Twin Delayed Deep Deterministic Policy Gradient ( TD3 ) RL framework has an agent interacting with an environment generally defined by a Markov Decision Process ( MDP ) . At each timestep t , an agent receives the state st , and takes an action at according to the policy π , and then receives a reward rt and the next state st+1 at next time step t + 1 . The goal of RL is to find the policy that maximizes the discounted cumulative return Rt = ∑∞ k=t γ k−trk where γ is a discounted factor . Off-policy RL can use the data from arbitrary policies to train its actor and critic functions repeatedly , which is a key point for improving recent gradient-based RL . Silver et al . ( 2014 ) introduced the off-policy Deterministic Policy Gradient ( DPG ) , which has an advantage for high-dimensional action spaces . The DDPG ( Lillicrap et al. , 2015 ) was extended from the DPG to apply it to a deep neural network . TD3 ( Fujimoto et al. , 2018 ) is an advanced version of the DDPG , which suffers from the overestimation bias of the critic . To correct this bias , two critics are introduced , and then the critic with the lowest state-action value is taken during the critic update as in the Double Deep Q-Network ( DDQN ) ( Van Hasselt et al. , 2016 ) . Figure 2 ( a ) represents the architecture of the TD3 . Cross Entropy Method ( CEM ) The Evolutionary Approach ( EA ) is a heuristic search method inspired by nature , where the current population is evaluated , and the next population is regenerated using the current evaluation result in order to produce a higher Return , which is also known as Fitness and defined as a cumulative sum of immediate reward for a fixed number of steps . The Estimation of Distribution Algorithm ( EDA ) is a class of the EA : It updates the target distribution to generate a better population . Depending on the update method for the distribution , EDAs are classified as a CEM ( De Boer et al. , 2005 ) , a Covariance Matrix Adaptation Evolutionary Strategy ( Hansen , 2016 ) , an Evolutionary Strategy ( Salimans et al. , 2017 ) , and a Natural Evolutionary Strategy ( Wierstra et al. , 2014 ) . We used the CEM as one part of our proposed algorithm . As shown in Figure 2 ( b ) , the CEM procedures are as follows : The population is sampled from the multivariate Gaussian N ( µ , Σ ) and evaluated , and for the top K , which is smaller than the population size ( N ) , high performing actors are used to update a new mean ( µ ) and covariance ( Σ ) of the target distribution . The weight can be given to each actor according to the rank of the Return ( Hansen , 2016 ) . The elite actor can be passed to the next population , which is known as elitism . The more detailed procedure is reported in ( De Boer et al. , 2005 ) . In this paper , we used a diagonal covariance to reduce the parameters . Population-guided Parallel Policy Search ( P3S ) The Guided Policy Learning ( GPL ) is commonly used when the elite policy leads some sub-policies to a better solution in multiple policies setting . Teh et al . ( 2017 ) introduced GPL for joint learning of numerous tasks in which a common policy encourages local policies to act better . Jung et al . ( 2020 ) proposed a soft-manner GPL , called the Population-guided Parallel Policy Search , for multiple identical learners with the same objective , where a population is evaluated periodically . Then sub-policies are trained to maximize their critic value and to concurrently minimize the distance from the elite policy for the next period . For this purpose , Augmented Loss ( 2 ) is used to train the sub-policies instead of Original Loss ( 1 ) . Original Loss : LO ( π ) = Es∼SS [ −Qπ ( s , π ( s ) ) ] ( 1 ) Augmented Loss : LA ( π , πelite , β ) = Es∼SS [ −Qπ ( s , π ( s ) ) + β||π ( s ) − πelite ( s ) ||22 ] ( 2 ) where π is a trained policy , Qπ is a critic function depending on π , πelite is the elite policy , SS is the set of states , and ||π ( s ) − πelite ( s ) ||22 is the Euclidean distance measure between the trained policy and the elite policy . β is a distance weight and is controlled adaptively . In this paper , we used a revised GPL inspired by P3S so that the CEM elite actor guides the TD3 to better space .

Summary: The paper proposes a new method combining evolutionary methods and RL. In particular, the authors combine CEM and TD3 in PGPS. PGPS maintains a population of policies, which interact with the environment to collect data filling the replay buffer. The data in replay buffer is then used to train TD3. PGPS enables information flow in both directions: when the TD3 policy performs poorly, the elite policy from the population is used to guide TD3 by an imitation learning loss; The TD3 critic helps select top policies in the population and the TD3 actor is also included in the population. The experiments on simple Mujoco domains demonstrate the utility of PGPS and the ablation study analyzes the utility of each part of PGPS.