Population-Guided Parallel Policy Search for Reinforcement Learning

1 INTRODUCTION . RL is an active research field and has been applied successfully to games , simulations , and actual environments . With the success of RL in relatively easy tasks , more challenging tasks such as sparse reward environments ( Oh et al . ( 2018 ) ; Zheng et al . ( 2018 ) ; Burda et al . ( 2019 ) ) are emerging , and developing good RL algorithms for such challenging tasks is of great importance from both theoretical and practical perspectives . In this paper , we consider parallel learning , which is an important line of RL research to enhance the learning performance by having multiple learners for the same environment . Parallelism in learning has been investigated widely in distributed RL ( Nair et al . ( 2015 ) ; Mnih et al . ( 2016 ) ; Horgan et al . ( 2018 ) ; Barth-Maron et al . ( 2018 ) ; Espeholt et al . ( 2018 ) ) , evolutionary algorithms ( Salimans et al . ( 2017 ) ; Choromanski et al . ( 2018 ) ; Khadka & Tumer ( 2018 ) ; Pourchot & Sigaud ( 2019 ) ) , concurrent RL ( Silver et al . ( 2013 ) ; Guo & Brunskill ( 2015 ) ; Dimakopoulou & Van Roy ( 2018 ) ; Dimakopoulou et al . ( 2018 ) ) and population-based training ( PBT ) ( Jaderberg et al . ( 2017 ; 2018 ) ; Conti et al . ( 2018 ) ) . In this paper , in order to enhance the learning performance , we apply parallelism to RL based on a population of policies , but the usage is different from the previous methods . One of the advantages of using a population is the capability to evaluate policies in the population . Once all policies in the population are evaluated , we can use information of the best policy to enhance the performance . One simple way to exploit the best policy information is that we reset the policy parameter of each learner with that of the best learner at the beginning of the next M time steps ; make each learner perform learning from this initial point for the next M time steps ; select the best learner again at the end of the next M time steps ; and repeat this procedure every M time steps in a similar way that PBT does ( Jaderberg et al . ( 2017 ) ) . We will refer to this method as the resetting method in this paper . However , this resetting method has the problem that the search area covered by all N policies in the population collapses to one point at the time of parameter copying and thus the search area can be narrow around the previous best policy point . To overcome such disadvantage , instead of resetting the policy parameter with the best policy parameter periodically , ∗Corresponding author we propose using the best policy information in a soft manner . In the proposed scheme , the shared best policy information is used only to guide other learners ’ policies for searching a better policy . The chief periodically determines the best policy among all learners and distributes the best policy parameter to all learners so that the learners search for better policies with the guidance of the previous best policy . The chief also enforces that the N policies are spread in the policy space with a given distance from the previous best policy point so that the search area by allN learners maintains a wide area and does not collapse into a narrow region . The proposed Population-guided Parallel Policy Search ( P3S ) learning method can be applied to any off-policy RL algorithms and implementation is easy . Furthermore , monotone improvement of the expected cumulative return by the P3S scheme is theoretically proved . We apply our P3S scheme to the TD3 algorithm , which is a state-of-the-art off-policy algorithm , as our base algorithm . Numerical result shows that the P3S-TD3 algorithm outperforms the baseline algorithms both in the speed of convergence and in the final steady-state performance . 2 BACKGROUND AND RELATED WORKS . Distributed RL Distributed RL is an efficient way of taking advantage of parallelism to achieve fast training for large complex tasks ( Nair et al . ( 2015 ) ) . Most of the works in distributed RL assume a common structure composed of multiple actors interacting with multiple copies of the same environment and a central system which stores and optimizes the common Q-function parameter or the policy parameter shared by all actors . The focus of distributed RL is to optimize the Q-function parameter or the policy parameter fast by generating more samples for the same wall clock time with multiple actors . For this goal , researchers investigated various techniques for distributed RL , e.g. , asynchronous update of parameters ( Mnih et al . ( 2016 ) ; Babaeizadeh et al . ( 2017 ) ) , sharing an experience replay buffer ( Horgan et al . ( 2018 ) ) , GPU-based parallel computation ( Babaeizadeh et al . ( 2017 ) ; Clemente et al . ( 2017 ) ) , GPU-based simulation ( Liang et al . ( 2018 ) ) and V-trace in case of on-policy algorithms ( Espeholt et al . ( 2018 ) ) . Distributed RL yields performance improvement in terms of the wall clock time but it does not consider the possible enhancement by interaction among a population of policies of all learners like in PBT or our P3S . The proposed P3S uses a similar structure to that in ( Nair et al . ( 2015 ) ; Espeholt et al . ( 2018 ) ) : that is , P3S is composed of multiple learners and a chief . The difference is that each learner in P3S has its own Q or value function parameter and policy parameter , and optimizes the parameters in parallel to search in the policy space . Population-Based Training Parallelism is also exploited in finding optimal parameters and hyperparameters of training algorithms in PBT ( Jaderberg et al . ( 2017 ; 2018 ) ; Conti et al . ( 2018 ) ) . PBT trains neural networks , using a population with different parameters and hyper-parameters in parallel at multiple learners . During the training , in order to take advantage of the population , it evaluates the performance of networks with parameters and hyper-parameters in the population periodically . Then , PBT selects the best hyper-parameters , distributes the best hyper-parameters and the corresponding parameters to other learners , and continues the training of neural networks . Recently , PBT is applied to competitive multi-agent RL ( Jaderberg et al . ( 2018 ) ) and novelty search algorithms ( Conti et al . ( 2018 ) ) . The proposed P3S uses a population to search a better policy by exploiting the best policy information similarly to PBT , but the way of using the best policy information is different . In P3S , the parameter of the best learner is not copied but used in a soft manner to guide the population for better search in the policy space . Guided Policy Search Our P3S method is also related to guided policy search ( Levine & Koltun ( 2013 ) ; Levine et al . ( 2016 ) ; Teh et al . ( 2017 ) ; Ghosh et al . ( 2018 ) ) . Teh et al . ( 2017 ) proposed a guided policy search method for joint training of multiple tasks in which a common policy is used to guide local policies and the common policy is distilled from the local policies . Here , the local policies ’ parameters are updated to maximize the performance and minimize the KL divergence between the local policies and the common distilled policy . The proposed P3S is related to guided policy search in the sense that multiple policies are guided by a common policy . However , the difference is that the goal of P3S is not learning multiple tasks but learning optimal parameter for a common task as in PBT . Hence , the guiding policy is not distilled from multiple local policies but chosen as the best performing policy among multiple learners . Exploiting Best Information Exploiting best information has been considered in the previous works ( White & Sofge ( 1992 ) ; Oh et al . ( 2018 ) ; Gangwani et al . ( 2019 ) ) . In particular , Oh et al . ( 2018 ) ; Gangwani et al . ( 2019 ) exploited past good experiences to obtain a better policy , whereas P3S exploits the current good policy among multiple policies to obtain a better policy . 3 POPULATION-GUIDED PARALLEL POLICY SEARCH . The overall structure of the proposed P3S scheme is described in Fig . 1 . We haveN identical parallel learners with a shared common experience replay buffer D , and all N identical learners employ a common base algorithm which can be any off-policy RL algorithm . The execution is in parallel . The i-th learner has its own environment E i , which is a copy of the common environment E , and has its own value function ( e.g. , Q-function ) parameter θi and policy parameter φi . The i-th learner interacts with the environment copy E i with additional interaction with the chief , as shown in Fig . 1 . At each time step , the i-th learner performs an action ait to its environment copy E i by using its own policy πφi , stores its experience ( sit , a i t , r i t , s i t+1 ) to the shared common replay buffer D for all i = 1 , 2 , · · · , N . Then , each learner updates its value function parameter and policy parameter once by drawing a mini-batch of size B from the shared common replay buffer D by minimizing its own value loss function and policy loss function , respectively . Due to parallel update of parameters , the policies of all learners compose a population ofN different policies . In order to take advantage of this population , we exploit the policy information from the best learner periodically during the training like in PBT ( Jaderberg et al . ( 2017 ) ) . Suppose that the Q-function parameter and policy parameter of each learner are initialized and learning is performed as described above for M time steps . At the end of the M time steps , we determine who is the best learner based on the average of the most recent Er episodic rewards for each learner . Let the index of the best learner be b . Then , the policy parameter information φb of the best learner can be used to enhance the learning of other learners for the next M time steps . Instead of copying φb to other learners like in PBT , we propose using the information φb in a soft manner . That is , during the next M time steps , while we set the loss function L̃ ( θi ) for the Q-function to be the same as the loss L ( θi ) of the base algorithm , we set the loss function L̃ ( φi ) for the policy parameter φi of the i-th learner as the following augmented version : L̃ ( φi ) = L ( φi ) + 1 { i 6=b } βEs∼D [ D ( πφi , πφb ) ] ( 1 ) where L ( φi ) is the policy loss function of the base algorithm , 1 { · } denotes the indicator function , β ( > 0 ) is a weighting factor , D ( π , π′ ) be some distance measure between two policies π and π′ .

Summary: This work presents a distributed framework for off-policy RL consisting of multiple agents that are trained in parallel while also being regularized to be similar to the current best policy. Maintaining a population of agents can help mitigate issues due to convergences to local optima. The method is evaluated on standard continuous control tasks, and shows some performance improvements over methods that train just a single agent. Ablation experiments are also conducted to evaluate the effects of different design decisions.