Interactive Parallel Exploration for Reinforcement Learning in Continuous Action Spaces

1 INTRODUCTION . Reinforcement learning ( RL ) for continuous action control is an active research field . In RL , an agent learns a policy through interaction with the environment to maximize the cumulative reward . One of the key issues in RL is the trade-off between exploitation and exploration . Exploitation is to make a best decision based on the already collected information , whereas exploration is to collect more new information about the environment . The balance between the two is important for good RL algorithms . For example , DQN ( Mnih et al . ( 2015 ) ) balances exploitation and exploration by taking actions based on the ǫ-greedy approach . Deep deterministic policy gradient ( DDPG ) ( Lillicrap et al . ( 2015 ) ) and twin delayed deep deterministic ( TD3 ) ( Fujimoto et al . ( 2018 ) ) policy gradient algorithms promote exploration by adding Ornstein-Uhlenbeck noise and Gaussian noise to the best decision action , respectively . Soft actor-critic ( SAC ) ( Haarnoja et al . ( 2018 ) ) performs balancing by using a maximum entropy objective . However , most of the previous works focus on exploration to obtain unobserved states or actions . In this paper , we consider exploration in the policy parameter space by using parallel identical learners for the same environment . By having multiple identical learners for the same environment , we can have increased search capability for a better policy . Parallelism in learning has been investigated widely in distributed RL ( Nair et al . ( 2015 ) , Mnih et al . ( 2016 ) , Horgan et al . ( 2018 ) , Barth-Maron et al . ( 2018 ) , Espeholt et al . ( 2018 ) ) , evolutional strategies ( Salimans et al . ( 2017 ) , Choromanski et al . ( 2018 ) ) , and recently in population based training ( PBT ) ( Jaderberg et al . ( 2017 ) , Jaderberg et al . ( 2018 ) , Conti et al . ( 2017 ) ) for faster and better search for parameters and/or hyperparameters . In this paper , we also apply parallelism to RL in order to enhance the learning performance but in a slightly different way as compared to the previous methods . The proposed algorithm is intended for any off-policy RL algorithms and is composed of a chief , N environment copies of the same environment , and N identical learners with a shared common experience replay buffer and a common base algorithm , as shown in Fig . 1 . Each learner has its own value function ( s ) and policy , and trains its own policy by interacting with its own environment copy with some additional interaction with the chief , as shown in Fig . 1 . At each time step , each learner takes an action to its environment copy by using its own policy , stores its experience to the shared common experience replay buffer . Then , each learner updates its value function parameter and policy parameter by drawing mini-batches from the shared common replay buffer by minimizing its own value loss function and policy loss function , respectively . One way to implement parallel learning under the above setup is to run N fully independent parallel learning without interaction among the learners except sharing their experiences until the end of time steps and to choose the policy from the learner with the maximum accumulated reward at the end for future use . We will refer to this method as the experience-sharing-only method . However , this method ignores possible benefit from mutual interaction among the learners during training . In order to harness the benefit of mutual interaction among the learners in parallel learning , we exploit the information from the best learner among all learners periodically during training like in PBT ( Jaderberg et al . ( 2017 ) , Jaderberg et al . ( 2018 ) ) . Suppose that the value and policy parameters of each learner are initialized and learning is performed as described above forM time steps . At the end of M time steps , we can determine who is the best learner based on the average of the most recent Er episodic rewards for each learner . Then , the policy parameter information of the best learner can be used to enhance the learning of other learners for the next M time steps . This information can help learners stuck in local minima escape from the local minima and guide other learners for better direction . One simple way to exploit this best policy parameter information is that we reset the policy parameter of each learner with the policy parameter of the best learner at the beginning of the next M time steps , make each learner perform learning from this initial point in the policy parameter space for the next M time steps , select the best learner again at the end of the next M time steps , and repeat this procedure every M time steps in a similar way that PBT ( Jaderberg et al . ( 2017 ) ) copies the best learner ’ s parameters and hyperparameters to other learners . We will refer to this method as the reloading method in this paper . However , this reloading method has the problem that the search area covered by all N searching policies collapses to one point at the time of parameter copying and thus the search area can be narrow around the previous best policy point . In order to overcome such disadvantage , instead of resetting the policy parameter with the best policy parameter every M time steps , we here propose using the policy parameter information of the best learner in a soft manner to enhance the performance of the overall parallel learning . In the proposed scheme , the shared best policy information is used only to guide the policies of other learners . The policy of each learner is updated by improving the performance around a certain distance from the shared guiding policy . The chief periodically determines the best policy among the policies of all learners and distributes the best policy parameter to all learners so that the learners search for better policies around the previous best policy . The chief also enforces that the N searching policies are spread in the policy parameter space with a given distance from the previous best policy point so that the search area in the policy space by all N learners maintains a wide area and does not collapse into a narrow region . The proposed interactive parallel exploration ( IPE ) learning method can be applied to any off-policy RL algorithms and implementation is easy . Furthermore , the proposed method can be extended directly to distributed or multi-agent RL systems . We apply our IPE scheme to the TD3 algorithm and the SAC algorithm , which are state-of-the-art off-policy algorithms , as our base algorithms , and the new algorithms are named IPE-TD3 and IPE-SAC algorithms , respectively . Numerical result shows that the IPE-enhanced algorithms outperform the baseline algorithms both in the speed of convergence and in the final steady-state performance . The gain by IPE 2 BACKGROUND AND RELATED WORKS . 2.1 DISTRIBUTED RL . Distributed RL is an efficient way that takes advantage of parallelism to achieve fast training for large complex tasks ( Nair et al . ( 2015 ) ) . Most of the works in distributed RL assume a common structure composed of multiple actors interacting with multiple copies of the same environment and a central system which stores and optimizes the common Q-function parameter or the policy parameter shared by all actors . The focus of distributed RL is to optimize the Q-function parameter or the policy parameter fast by generating more samples for the same wall clock time with multiple actors . In order to achieve this goal , researchers investigated various techniques for distributed RL , e.g. , asynchronous update of parameters ( Mnih et al . ( 2016 ) , Babaeizadeh et al . ( 2017 ) ) , sharing an experience replay buffer ( Horgan et al . ( 2018 ) ) , GPU-based parallel computation ( Babaeizadeh et al . ( 2017 ) , Clemente et al . ( 2017 ) ) , GPU-based simulation ( Liang et al . ( 2018 ) ) and V-trace in case of on-policy algorithms ( Espeholt et al . ( 2018 ) ) . Distributed RL yields significant performance improvement in terms of the wall clock time but it does not consider the possible enhancement by interaction among multiple learners like in IPE and PBT . The proposed IPE uses a similar structure to that in ( Nair et al . ( 2015 ) , Espeholt et al . ( 2018 ) ) : that is , IPE is composed of multiple learners and a central system called chief . The difference is that each learner has its own Q or value function parameter and policy parameter and optimizes the parameters in parallel with interactions . 2.2 POPULATION BASED TRAINING . Parallelism is also exploited in finding optimal parameters and hyperparameters of training algorithms for neural networks in PBT ( Jaderberg et al . ( 2017 ) , Jaderberg et al . ( 2018 ) , Conti et al . ( 2017 ) ) . PBT ( Jaderberg et al . ( 2017 ) ) first chooses multiple sets of hyperparameters and parameters for a common base algorithm , and runs the base algorithm separatively in parallel at multiple learners to train their neural networks using those parameters and hyperparameters . Each learner updates the neural network parameters by perturbing the assigned hyperparameters . During training , in principle PBT evaluates the performance of multiple learners periodically , and selects the best performing hyperparameters , and then distributes the best performing hyperparameters and the corresponding parameters to other learners , although implementation details can be changed . Recently , PBT is applied to competitive multi-agent RL ( Jaderberg et al . ( 2018 ) ) and novelty search algorithms ( Conti et al . ( 2017 ) ) . Although PBT is mainly developed to tune hyperparamters , the philosophy of PBT can be applied to find optimal parameters for given hyperparameters by multiple learners . In this case , multiple learners update their parameters in parallel , their performance is measured periodically , the parameters of the best performing learner are copied to other learners , other learners independently update their parameters from the copied parameters as their new initialization , and this process is repeated . The proposed IPE is similar to PBT in the sense that it exploits the parameters of the best performing learner among multiple parallel learners . However , IPE is different from the PBT-derived method in the way how IPE uses the parameters of the best learner . In the PBT-derived method , the parameters of the best learner are copied to other learners and other learners ’ parameters are reset to the parameters of the best performing learner . Then , the parameters of each learner are updated by stochastic gradient descent ( SGD ) . However , in IPE the parameters of the best performing learner are not copied but used in a soft manner as a guiding direction . Copying means that the parameters of all learners collapse to a single point in the parameter space . Furthermore , unlike PBT , IPE uses a common experience replay buffer to store all experiences from multiple learners with different parameters to exploit the diverse experiences of multiple learners with different parameters . As mentioned in Section 1 , we refer to the PBT-derived method with a common experience replay buffer as the reloading method of which performance will be given in ablation study in Section 4 . Although IPE is considered only for parallel parameter search in this paper , combining the soft way of using the parameters of the best performing learner with hyperparameter search is an interesting future work .

Summary: The paper present interactive parallel exploration (IPE), a reinforcement learning method based on an ensemble of policies and a shared experience pool. Periodically, the highest-return achieving policy is selected, towards which the other policies are updated in a sense of some distance metric. IPE is applicable to any off-policy reinforcement learning algorithm. The experiments demonstrate some improvement over TD3 on four MuJoCo benchmark tasks.