Title: INTERACTIVE PARALLEL EXPLORATION FOR REIN-FORCEMENT LEARNING IN CONTINUOUS ACTION SPACES

Abstract: In this paper, a new interactive parallel learning scheme is proposed to enhance the performance of off-policy continuous-action reinforcement learning. In the proposed interactive parallel learning scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search space by the multiple learners. The guidance by the previous best policy and the enlarged search space by the proposed interactive parallel learning scheme enable faster and better policy search in the policy parameter space. Working algorithms are constructed by applying the proposed interactive parallel learning scheme to several off-policy reinforcement learning algorithms such as the twin delayed deep deterministic (TD3) policy gradient algorithm and the soft actor-critic (SAC) algorithm, and numerical results show that the constructed IPE-enhanced algorithms outperform most of the current state-of-the-art reinforcement learning algorithms for continuous action control.

1. INTRODUCTION:

Reinforcement learning (RL) for continuous action control is an active research field. In RL, an agent learns a policy through interaction with the environment to maximize the cumulative reward. One of the key issues in RL is the trade-off between exploitation and exploration. Exploitation is to make a best decision based on the already collected information, whereas exploration is to collect more new information about the environment. The balance between the two is important for good RL algorithms. For example, DQN (Mnih et al. (2015) ) balances exploitation and exploration by taking actions based on the ǫ-greedy approach. Deep deterministic policy gradient (DDPG) (Lillicrap et al. (2015) ) and twin delayed deep deterministic (TD3) (Fujimoto et al. (2018) ) policy gradient algorithms promote exploration by adding Ornstein-Uhlenbeck noise and Gaussian noise to the best decision action, respectively. Soft actor-critic (SAC) (Haarnoja et al. (2018) ) performs balancing by using a maximum entropy objective. However, most of the previous works focus on exploration to obtain unobserved states or actions. In this paper, we consider exploration in the policy parameter space by using parallel identical learners for the same environment. By having multiple identical learners for the same environment, we can have increased search capability for a better policy. Parallelism in learning has been investigated widely in distributed RL (Nair et al. (2015) , Mnih et al. (2016) , Horgan et al. (2018) , Barth-Maron et al. (2018) , Espeholt et al. (2018) ), evolutional strategies (Salimans et al. (2017) , Choromanski et al. (2018) ), and recently in population based training (PBT) (Jaderberg et al. (2017) , Jaderberg et al. (2018) , Conti et al. (2017) ) for faster and better search for parameters and/or hyperparameters. In this paper, we also apply parallelism to RL in order to enhance the learning performance but in a slightly different way as compared to the previous methods.

The proposed algorithm is intended for any off-policy RL algorithms and is composed of a chief, N environment copies of the same environment, and N identical learners with a shared common experience replay buffer and a common base algorithm, as shown in Fig. 1 . Each learner has its own value function(s) and policy, and trains its own policy by interacting with its own environment copy with some additional interaction with the chief, as shown in Fig. 1 . At each time step, each learner takes an action to its environment copy by using its own policy, stores its experience to the shared common experience replay buffer. Then, each learner updates its value function parameter and policy parameter by drawing mini-batches from the shared common replay buffer by minimizing its own value loss function and policy loss function, respectively.

One way to implement parallel learning under the above setup is to run N fully independent parallel learning without interaction among the learners except sharing their experiences until the end of time steps and to choose the policy from the learner with the maximum accumulated reward at the end for future use. We will refer to this method as the experience-sharing-only method. However, this method ignores possible benefit from mutual interaction among the learners during training. In order to harness the benefit of mutual interaction among the learners in parallel learning, we exploit the information from the best learner among all learners periodically during training like in PBT (Jaderberg et al. (2017) , Jaderberg et al. (2018) ). Suppose that the value and policy parameters of each learner are initialized and learning is performed as described above for M time steps. At the end of M time steps, we can determine who is the best learner based on the average of the most recent E r episodic rewards for each learner. Then, the policy parameter information of the best learner can be used to enhance the learning of other learners for the next M time steps. This information can help learners stuck in local minima escape from the local minima and guide other learners for better direction.

One simple way to exploit this best policy parameter information is that we reset the policy parameter of each learner with the policy parameter of the best learner at the beginning of the next M time steps, make each learner perform learning from this initial point in the policy parameter space for the next M time steps, select the best learner again at the end of the next M time steps, and repeat this procedure every M time steps in a similar way that PBT (Jaderberg et al. (2017) ) copies the best learner's parameters and hyperparameters to other learners. We will refer to this method as the reloading method in this paper. However, this reloading method has the problem that the search area covered by all N searching policies collapses to one point at the time of parameter copying and thus the search area can be narrow around the previous best policy point. In order to overcome such disadvantage, instead of resetting the policy parameter with the best policy parameter every M time steps, we here propose using the policy parameter information of the best learner in a soft manner to enhance the performance of the overall parallel learning. In the proposed scheme, the shared best policy information is used only to guide the policies of other learners. The policy of each learner is updated by improving the performance around a certain distance from the shared guiding policy. The chief periodically determines the best policy among the policies of all learners and distributes the best policy parameter to all learners so that the learners search for better policies around the previous best policy. The chief also enforces that the N searching policies are spread in the policy parameter space with a given distance from the previous best policy point so that the search area in the policy space by all N learners maintains a wide area and does not collapse into a narrow region.

The proposed interactive parallel exploration (IPE) learning method can be applied to any off-policy RL algorithms and implementation is easy. Furthermore, the proposed method can be extended directly to distributed or multi-agent RL systems. We apply our IPE scheme to the TD3 algorithm and the SAC algorithm, which are state-of-the-art off-policy algorithms, as our base algorithms, and the new algorithms are named IPE-TD3 and IPE-SAC algorithms, respectively. Numerical result shows that the IPE-enhanced algorithms outperform the baseline algorithms both in the speed of convergence and in the final steady-state performance. The gain by IPE 2 BACKGROUND AND RELATED WORKS 2.1 DISTRIBUTED RL Distributed RL is an efficient way that takes advantage of parallelism to achieve fast training for large complex tasks (Nair et al. (2015) ). Most of the works in distributed RL assume a common structure composed of multiple actors interacting with multiple copies of the same environment and a central system which stores and optimizes the common Q-function parameter or the policy parameter shared by all actors. The focus of distributed RL is to optimize the Q-function parameter or the policy parameter fast by generating more samples for the same wall clock time with multiple actors. In order to achieve this goal, researchers investigated various techniques for distributed RL, e.g., 2017)), GPU-based simulation (Liang et al. (2018) ) and V-trace in case of on-policy algorithms (Espeholt et al. (2018) ). Distributed RL yields significant performance improvement in terms of the wall clock time but it does not consider the possible enhancement by interaction among multiple learners like in IPE and PBT. The proposed IPE uses a similar structure to that in (Nair et al. (2015) , Espeholt et al. (2018) ): that is, IPE is composed of multiple learners and a central system called chief. The difference is that each learner has its own Q or value function parameter and policy parameter and optimizes the parameters in parallel with interactions.

Learners:



2.2. POPULATION BASED TRAINING:

Parallelism is also exploited in finding optimal parameters and hyperparameters of training algorithms for neural networks in PBT (Jaderberg et al. (2017) , Jaderberg et al. (2018) , Conti et al. (2017) ). PBT (Jaderberg et al. (2017) ) first chooses multiple sets of hyperparameters and parameters for a common base algorithm, and runs the base algorithm separatively in parallel at multiple learners to train their neural networks using those parameters and hyperparameters. Each learner updates the neural network parameters by perturbing the assigned hyperparameters. During training, in principle PBT evaluates the performance of multiple learners periodically, and selects the best performing hyperparameters, and then distributes the best performing hyperparameters and the corresponding parameters to other learners, although implementation details can be changed. Recently, PBT is applied to competitive multi-agent RL (Jaderberg et al. (2018) ) and novelty search algorithms (Conti et al. (2017) ).

Although PBT is mainly developed to tune hyperparamters, the philosophy of PBT can be applied to find optimal parameters for given hyperparameters by multiple learners. In this case, multiple learners update their parameters in parallel, their performance is measured periodically, the parameters of the best performing learner are copied to other learners, other learners independently update their parameters from the copied parameters as their new initialization, and this process is repeated. The proposed IPE is similar to PBT in the sense that it exploits the parameters of the best performing learner among multiple parallel learners. However, IPE is different from the PBT-derived method in the way how IPE uses the parameters of the best learner. In the PBT-derived method, the parameters of the best learner are copied to other learners and other learners' parameters are reset to the parameters of the best performing learner. Then, the parameters of each learner are updated by stochastic gradient descent (SGD). However, in IPE the parameters of the best performing learner are not copied but used in a soft manner as a guiding direction. Copying means that the parameters of all learners collapse to a single point in the parameter space. Furthermore, unlike PBT, IPE uses a common experience replay buffer to store all experiences from multiple learners with different parameters to exploit the diverse experiences of multiple learners with different parameters.

As mentioned in Section 1, we refer to the PBT-derived method with a common experience replay buffer as the reloading method of which performance will be given in ablation study in Section 4.

Although IPE is considered only for parallel parameter search in this paper, combining the soft way of using the parameters of the best performing learner with hyperparameter search is an interesting future work.

2.3. GUIDED POLICY SEARCH:

Our IPE method is also related to guided policy search (Levine & Koltun (2013) , Levine et al. (2016) , Teh et al. (2017) , Ghosh et al. (2018) ). Recently, Teh et al. (2017) proposed a guided policy search method for joint training of multiple tasks in which a common policy is used to guide local policies and the common policy is distilled from the local policies. Here, the local policies' parameters are updated to maximize the performance and minimize the KL divergence between the local policies and the common distilled policy. The proposed IPE is related to guided policy search in the sense that multiple policies are guided by a common policy. However, the difference is that the goal of IPE is not learning multiple tasks but learning optimal parameters for a common task as in PBT and hence the guiding policy is not distilled from multiple local policies but chosen as the best performing policy among multiple learners.

2.4. EXPLORATION:

Improving exploration has been one of the key issues in RL and many different ways have been developed to improve exploration through maximum entropy objectives (Haarnoja et al. (2017; 2018) ), noise in networks (Fortunato et 2018)). In particular, Oh et al. (2018) exploited past good experiences to explore the sample space, whereas IPE exploit the current good policy among multiple policies to explore the policy space.

2.5. THE SET UP: PARALLEL LEARNING FOR A COMMON ENVIRONMENT:

The considered parallel learning setting consists of the environment E and N policies {π

1 , • • • , π N }.

The environment E is described as a Markov decision process (MDP) defined by the tuple S, A, T , r , where S is the state space, A is the action space, T : S × A × S → [0, 1] is the state transition probability, and r : S × A → R is the reward function. There exist N copies

{E 1 , • • • , E N } of the environment E, i.e., E 1 = • • • = E N = E,

and the N environment copies may have different random initial seeds. The policy π i interacts with its corresponding environment copy E i and builds up its trajectory {(

s i t , a i t , r i t ), t = 1, 2, • • • } for each i = 1, • • • , N . At time step t, the environment copy E i has a state s i t ∈ S.

The policy π i interacts with the environment copy E i by taking an action a i t according to π i given the current state s i t . Then, the environment copy E i yields the reward r i t = r(s i t , a i t ) and makes transition to the next state s i t+1 according to T . In this paper, in order to account for the actual amount of interaction with the environment, we define environment steps as the total number of interactions by all N parallel policies with all N environment copies. Suppose that all N policies generate their trajectories simultaneously in parallel, and suppose that M time steps have elapsed. Then, although the number of elapsed time steps is M , the number of environment steps is N M .

3. INTERACTIVE PARALLEL POLICY EXPLORATION:

We now present the proposed IPE scheme with the parallel environment learning setting described in Section 2.5, and the overall structure is described in Fig. 1 .

We have N identical parallel learners with a shared common experience replay buffer D, and all N identical learners employ a common base algorithm, which can be any off-policy RL algorithm. The execution is in parallel. The i-th learner has its own environment E i , which is a copy of the common environment E, and has its own value function (e.g., Q-function) parameter θ i and policy parameter φ i . The i-th learner interacts with the environment copy E i with some additional interaction with the chief, as shown in Fig. 1 . At each time step, the i-th learner performs an action a i t to its environment copy E i by using its own policy π φ i , stores its experience (s i t , a i t , r i t , s i t+1 ) to the shared common experience replay buffer D for all i = 1, 2, • • • , N . Note that one time step corresponds to N environment steps. Then, at each time step, each learner updates its value function parameter and policy parameter for N times by drawing N mini-batches of size B from the shared common replay buffer D by minimizing its own value loss function and policy loss function, respectively. The N time updates for each learner for each time step is to exploit the samples provided by other N -1 learners stored in the shared common replay buffer.

In order to harness the benefit of mutual interaction among the learners in parallel learning, we exploit the information from the best learner periodically during training like in PBT (Jaderberg et al. (2017) ). Suppose that the Q-function parameter and the policy parameter of each learner are initialized and learning is performed as described above for M time steps. At the end of the M time steps, we determine who is the best learner based on the average of the most recent E r episodic rewards for each learner. Let the index of the best learner be b. Then, the policy parameter information φ b of the best learner can be used to enhance the learning of other learners for the next M time steps. Here, instead of copying φ b to other learners, we propose using the information of φ b in a soft manner to enhance the performance of the overall parallel learning. That is, during the next M time steps, whereas we set the loss function L(θ i ) for the Q-function to be the same as the loss L(θ i ) of the base algorithm, we set the loss function L(φ i ) for the policy parameter φ i of the i-th learner as the following augmented version:

EQUATION

where L(φ i ) is the policy loss function of the base algorithm, 1 {•} denotes the indicator function, β(> 0) is a weighting factor, D(π, π ′ ) be some distance measure between two policies π and π ′ , and Ês∼D denotes the sample expectation based on mini-batch drawn randomly from the experience replay buffer D. The augmented loss function L(φ i ) in ( 1) is composed of two terms L(φ i ) and 1 {i =b} β Ês∼D D(π φ i , π φ b ) . Thus, for the non-best learners in the previous M time steps, the gradient of L(φ i ) is the mixture of two directions: one is to maximize the return by itself and the other is to follow the previously best learner's policy. The second term in the right-hand side (RHS) of (1) guides non-best learners towards a good direction in addition to each leaner's self search.

3.1. DESIGN OF THE WEIGHTING FACTOR β:

In (1), the weighting factor β is common to all N learners and should be determined judiciously to balance between improving its performance by each learner itself and going towards the previous best policy among the N learners. We adopt an adaptive method to determine the value of β as follows:

β = β ← 2β if Dbest ≥ max{ρ Db change , d search } × 1.5 β ← β/2 if Dbest < max{ρ Db change , d search }/1.5 (2)

where Dbest is the estimated distance between π φ i and π φ b averaged over all N -1 non-best learners, and Db change is the estimated distance between π φ b updated (i.e., the policy of the current best learner at the end of the current M time steps) and π φ b (i.e, the policy of the current best learner at the end of the previous M time steps), given respectively by

Dbest = 1 N -1 i∈I -b Ês∼D D(π φ i , π φ b ) and Db change = Ês∼D D(π φ b updated , π φ b ) . (3)

Here, I -b = {1, . . . , N } \ {b}, and d search and ρ are predetermined hyperparameters.

This adaptation method is similar to that used in proximal policy optimization (PPO) (Schulman et al. (2017) ). The update of β is done every M time steps and the updated β is used for the next M time steps. First, suppose that we do not have the first term ρ Db change in the maximum of the condition in (2). Then, when the estimated average distance Dbest from the best policy to the remaining policies is smaller than d search /1.5, the parameter β is decreased by half. Hence, the movement in the gradient direction of the second term in the RHS of ( 1) is diminished and the independent movement into the optimization direction for L(φ i ) becomes more important. So, each policy gradually diverges from the previous best policy serving as the reference point due to internal exploration mechanism such as added noise in action. On the other hand, when Dbest is larger than 1.5d search , the parameter β increases by factor two and the movement towards the previous best policy becomes more important. As time steps elapse, β is settled down so that Dbest is around d search . Hence, the proposed IPE scheme searches a wide area with rough radius d search around the best policy in the policy parameter space, as shown in Fig. 2(a) . Furthermore, with the first term ρ Db change in the maximum of the condition in (2), we can control the speed of tracking the best policy. Db change measures the speed of change in the best policy parameter. When the best policy parameter change scaled by ρ, i.e., ρ Db change is less than d search , the term is invisible by the maximum operation in (2). On the other hand, when ρ Db change > d search , the term is active and it means that the best policy parameter changes fast. Thus, the tracking speed should be controlled. If Dbest > ρ Db change , i.e., the distance from π φ i to π φ b is larger than ρ times the distance from π φ b updated to π φ b , then this means that the speed of tracking the best policy is slow. Hence, we increase β by factor two. Otherwise, we decrease β by half. When the search for the current M time steps is finished, the new best learner is selected and a new search for a wide area around the new best learner's policy π φ b is performed, as illustrated in Fig. 2(b ). The policy parameter information φ b of the best learner can be changed before the next best learner selection. Now, the overall procedure for the proposed IPE scheme is explained with the diagram in Fig. 1 . The value function parameter and policy parameter of each learner are initialized. The chief distributes the parameter β and the reference policy parameter φ b , which is the policy information of the best learner over the previous M time steps, to all learners. At each time step, each learner interacts with its own environment copy by taking its action and receiving the reward and the next state, and stores its experience to the shared common replay buffer D. Then, the i-th learner updates its value function parameter θ i by minimizing its own value loss function L(θ i ) which is the same as that of the base algorithm, and updates the policy parameter φ i by minimizing the augmented loss function L(φ i ) in (1) for N times by drawing N mini-batches from the shared common replay buffer D. Whenever an episode ends for a learner, the learner reports the episodic reward to the chief. The i-th learner reports Ês∼D D(π φ i , π φ b ) to the chief for computation of Dbest in (3). At every M time steps, the chief updates β according to (2), determines the best learner over the most recent M time steps based on the collected episodic rewards from each learner. Once the best learner is determined, the chief obtains the policy parameter information φ b from the determined best learner, and distributes the new β and the reference policy parameter φ b to all N learners. This procedure repeats until the time steps reaches the predefined maximum. When the parallel learning based IPE reaches a steady state, we can choose any of the N learners' policies and use the chosen policy for the environment E in future since it is expected that at the steady-state the performance of all N policies is more or less similar due to their distance property.

• • •:

(a) (b) π φ b π φ 1 π φ 2 π φ 3 π φ N d search

3.2. IPE-ENHANCED ALGORITHMS:

The proposed IPE method can be applied to any off-policy RL algorithms regardless of whether the base RL algorithms have continuous actions or discrete actions. Here, we consider the application of IPE to the TD3 algorithm as the base algorithm and the constructed algorithm is named the IPE-TD3 algorithm. The details of baseline TD3 are explained in Appendix A. With TD3 as the base algorithm, each learner has its own parameters θ i 1 , θ i 2 , and φ i for its two Q-functions and policy. Furthermore, it has (θ i 1 ) ′ , (θ i 2 ) ′ , and (φ i ) ′ which are the parameters of the corresponding target networks. For the distance measure between two policies, we use the mean square difference given by

EQUATION

For the i-th learner, as in TD3, the parameters θ i j , j = 1, 2 are updated every time step by minimizing

EQUATION

where

y = r + γ min j=1,2 Q (θ i j ) ′ (s ′ , π (φ i ) ′ (s ′ ) + ǫ), ǫ ∼ clip(N (0, σ2 ), -c, c).

The parameter φ i is updated every d time steps by minimizing the following augmented loss function:

EQUATION

For the first T initial timesteps for initial exploration we use a random policy and do not update all policies over the initial exploration period. With these loss functions, the reference policy, and the initial exploration policy, all procedure is the same as the general IPE procedure described previously. The pseudocode of the IPE-TD3 algorithm is provided in Appendix B.

The application of IPE to other algorithms such as SAC and DQN is also provided in Appendices.

4. EXPERIMENTS:

In this section, we provide the numerical results on the performance of the proposed IPE-TD3 and current state-of-the-art on-policy and off-policy baseline algorithms on several MuJoCo environments (Todorov et al. (2012) ). The baseline algorithms are Proximal Policy Optimization (PPO) (Schulman et al. (2017) ), Actor Critic using Kronecker-Factored Trust Region (ACKTR) (Wu et al. ( 2017)), Soft Q-learning (SQL) (Haarnoja et al. (2017) ), Soft Actor-Critic (SAC) (Haarnoja et al. (2018) ), and TD3 (Fujimoto et al. (2018) ). More numerical result on IPE applied to SAC is provided in Appendices.

4.1. PARAMETER SETTING:

All hyperparameters we used for evaluation are the same as those in the original papers (Schulman et al. (2017) ; Haarnoja et al. (2018); Fujimoto et al. (2018) ). Here, we provide the hyperparameters of TD3 and IPE-TD3 only.

TD3:

The networks for two Q-functions and the policy have 2 hidden layers. The first and second layers have sizes 400 and 300, respectively. The non-linearity function of the hidden layers is ReLU, and the activation functions of the last layers of the Q-functions and the policy are linear and hyperbolic tangent, respectively. We used the Adam optimizer with learning rate 10 -3 , discount factor γ = 0.99, target smoothing factor τ = 5 × 10 -3 , the period d = 2 for updating the policy. The experience replay buffer size is 10 6 , and the mini-batch size B is 100. The standard deviation for exploration noise σ and target noise σ are 0.1 and 0.2, respectively, and the noise clipping factor c is 0.5.

IPE-TD3:

In addition to the parameters for TD3, we used N = 4 learners, the period M = 250 of updating the best policy and β, the number of recent episodes E r = 10 for determining the best 

4.2. COMPARISON TO BASELINES:

In order to have sample-wise fair comparison among the considered algorithms, we obtain the performance with respect to environment steps (not time steps), which is defined as the total number of interactions with the environment by the agent. This comparison makes sense because the performance at the same environment steps means that all algorithms use the same number of samples obtained from the environment. The performance is obtained through the evaluation method that is similar to those in (Haarnoja et al. (2018); Fujimoto et al. (2018) ). Evaluation of the policies are conducted every R eval = 4000 environment steps for all algorithms. At each evaluation instant, the agent (or learner) fixes its policy as the one at the evaluation instant, and interacts with the same environment separate for the evaluation purpose with the fixed policy to obtain 10 episodic rewards. The average of these 10 episodic rewards is the performance at the evaluation instant. In the case of IPE-TD3 and other parallel learning schemes, each of the N learners fixes its policy as the one at the evaluation instant, and interacts with the environment with the fixed policy to obtain 10 episodic rewards. First, the 10 episodic rewards are averaged for each learner and then the maximum of the 10-episode-average rewards of the N learners is taken as the performance at that evaluation instant. We performed this operation for five different random seeds, and the mean and variance of the learning curve are obtained from these five simulations. The policies used for evaluation are stochastic for PPO and deterministic for the others.

Fig. 3 shows the learning curves over one million environment steps for several MuJoCo tasks: Hopper-v1, Walker2d-v1, HalfCheetah-v1, and Ant-v1. First, it is observed that the performance of TD3 here is similar to that in the original TD3 paper (Fujimoto et al. (2018) ), and the performance of other baseline algorithms is also similar to that in the original papers (Schulman et al. (2017) ; Haarnoja et al. (2018) ). It is seen that the IPE-TD3 algorithm outperforms the state-of-the-art RL algorithms in terms of both the speed of convergence with respect to environment steps and the final steady-state performance (except in Walker2d-v1, the initial convergence is a bit slower than TD3.) Especially, in the cases of Hopper-v1 and Ant-v1, TD3 has large variance and this means that the performance of TD3 is quite dependent on the initial condition of the environment and it is not easy for TD3 to escape out of bad local minima in certain environments. However, it is seen that IPE-TD3 yields much less variance as compared to TD3. This implies that the wide area search by IPE in the policy parameter space helps the learners escape out of bad local optima. It is seen that the wide area search around the previous best policy point in the policy parameter space by IPE yields faster and better policy search. IPE-TD3 has several components to improve the performance based on parallelism: 1) sharing experiences from multiple policies, 2) using the best policy information, and 3) fusing the best policy information in a soft manner based on the augmented loss function. Thus, we investigated the impact of each component on the performance improvement. For comparison we considered the following parallel policy exploration methods gradually incorporating more techniques:

1. TD3 The original TD3 with one learner 2. Distributed RL TD3 (DRL-TD3) N actors obtain samples from N environment copies.

The policy and the experience replay buffer are shared by all N actors.

3. Experience-Sharing-Only TD3 (ESO-TD3) N learners interact with N environment copies and update their own policies using experiences drawn from the shared experience replay buffer.

4. Reloading TD3 (Re-TD3) At every M ′ timesteps, the best policy is determined and all policies are initialized as the best policy, i.e., the best learner's policy parameter is copied to all other learners. The rest of the procedure is the same as experience-sharing-only TD3.

At every M timesteps, the best policy information is determined and this policy is used in a soft manner based on the augmented loss function.

Note that Re-TD3 exploits the best policy information from N learners. The main difference between IPE-TD3 and Re-TD3 is the way how the best learner's policy parameter is used. Re-TD3 initializes all policies with the best policy parameter every M ′ timesteps like in PBT (Jaderberg et al. (2017) ), whereas IPE-TD3 uses the best learner's policy parameter information determined every M timesteps to construct an augmented loss function. For fair comparison, M and M ′ are determined independently and optimally for IPE-TD3 and Re-TD3, respectively, since the optimal period can be different for the two methods. Thus, M ′ = 5000 is determined for Re-TD3 by tuning, whereas M = 250 is used for IPE-TD3. Since all N policies collapse as one point in Re-TD3 at the beginning of each period, we expect that a larger period is required for Re-TD3 to have sufficiently spread policies at the end of the best policy selection period.

Fig. 4 shows the learning curves of the considered parallel exploration methods for the Ant-v1 task and Table 1 shows the final (steady-state) performance of the considered parallel exploration methods for four MuJoCo tasks, respectively. It is seen that IPE-TD3 outperforms other parallel methods: DRL-TD3, ESO-TD3 and Re-TD3 except the case that ESO-TD3 outperforms all other parallel schemes in Hopper-v1. Both Re-TD3 and IPE-TD3 have better final (steady-state) performance than TD3 and ESO-TD3 for all tasks except Hopper-v1 for which ESO-TD3 performs best. Note that ESO-TD3 obtains most diverse experiences since the N learners shares the experience replay buffer but there is no interaction among the N learners until the end of training. So, it seems that this diverse experience is beneficial to Hopper-v1. The final performances of Re-TD3 and IPE-TD3 are more or less the same for HalfCheetah-v1 but the final performance of IPE-TD3 is noticeably better than that of Re-TD3 in other cases.

5. CONCLUSION:

In this paper, we have proposed a new interactive parallel learning scheme, IPE, to enhance the performance of off-policy RL systems. In the proposed IPE scheme, multiple identical learners with their own value-functions and policies sharing a common experience replay buffer search a good policy with the guidance of the best policy information in the previous search interval. The information of the best policy parameter of the previous search interval is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search space by the multiple learners. The guidance by the previous best policy and the enlarged search space by IPE enables faster and better policy search in the policy parameter space. The IPE-enhanced algorithms constructed by applying the proposed IPE scheme to TD3 or SAC outperforms most of the current state-of-the-art continuous-action RL algorithms. Although we mainly considered continuous-action off-policy algorithms in this paper, the proposed IPE method can also be applied to RL with discrete actions, as seen in Appendix E. In the case of continuous action control, the gain by IPE can be beneficial for recent trend of fast computer-based prototyping of complex robotics systems or autonomous cars, whereas in the discrete-action case better policy parameters can be searched for more challenging tasks by IPE.

Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Advances in Neural Information Processing Systems, pp. 5279-5288, 2017.

Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. arXiv preprint arXiv:1804.06459, 2018.

:



Figures and Tables:

al. (2018); Plappert et al. (2018)), and intrinsically motivated approaches (Bellemare et al. (2016); Ostrovski et al. (2017); Pathak et al. (2017); Achiam & Sastry (2017); Zheng et al. (2018)). The proposed IPE also enhances exploration. Specifically, IPE uses exploitation for exploration. Exploitation for exploration has been considered in the previous works(White & Sofge (1992),Oh et al. (

Figure 2: (a) the conceptual search coverage in the policy parameter space by parallel learners (the proper individual search area by each learner may be larger than that in the figure) and (b) an illustration of search comparison: single search (blue) versus the proposed interactive parallel search guided by the policy of best learner at each search interval (pink -policies of best learners during search)

Figure 3: Performance for PPO (brown), ACKTR (orange), SQL (purple), SAC (red), TD3 (green), and IPE-TD3 (proposed method, blue) on MuJoCo tasks.

Figure 4: Ablation study: Different parallel methods for Ant-v1 task

Steady state performance of different parallel exploration methods: IPE-TD3, Re-TD3, ESO-TD3, DRL-TD3 and TD3

Table content: <table><tr><td>Environment</td><td colspan="4">IPE-TD3 Re-TD3 ESO-TD3 DRL-TD3</td><td>TD3</td></tr><tr><td>Hopper-v1</td><td>3729.92</td><td>3619.04</td><td>3745.77</td><td colspan="2">3456.30 2555.85</td></tr><tr><td>Walker2d-v1</td><td>5029.22</td><td>4921.51</td><td>4677.01</td><td colspan="2">4813.72 4455.51</td></tr><tr><td colspan="3">HalfCheetah-v1 11418.50 11549.11</td><td>11086.88</td><td colspan="2">11159.92 9695.92</td></tr><tr><td>Ant-v1</td><td>5501.87</td><td>5393.09</td><td>4847.34</td><td colspan="2">4885.74 3760.50</td></tr></table>

Peer review (iclr2019):
 This paper describes a new architecture for parallelizing off-policy reinforcement learning systems with a pool of independent learners trained on identical, but independent instances of the environment with a scheme for periodically synchronizing the the policy knowledge across the pool. The paper provides demonstrations in several continuous control domains. I think this paper should be rejected because: 1) the approach is not well justified or placed within the large literature on parallel training architectures and population-based training methods, (2) the results are competitive with the best in each domain, but there are many missing details. Since the contribution is entirely support by empirical evidence, these issues need to be clarified. I look forward to the author response, as I will pose several questions below and my final score will carefully take the answers into account. Justification of decision. There are numerous papers on parallel architectures for training deep RL systems [1,2, 6] and you cited a few, and while not all of them focus on continuous control there are design decisions and insights in those works must be relevant to your efforts. You should make those connections clear in the paper. One line of the paper is not nearly enough. The stated focus of the paper is exploration-exploitation yet there is little to no discussion of other ideas including noisy networks, intrinsic motivation, or count-based exploration methods. The paper is missing a lot of key connections to the literature. I am certainly not a fan of architectures that assume access to many instances of the environment. In this case that assumption seems worse because of the target application: continuous control domains. These domains are simulations of physical control systems; on a robot the agent receives only one stream of experience and thus these architectures would not work well. Though there is some work on applying these multi-environment architectures to farms of robot arms; the reality of the real-world is that the arms end up being very different due to wear and tear, and engineers must constantly fix the hardware because these multi-environment architectures do not work when the environments are different. We cannot loose sight of the goal here—maximizing these simulation environments is not of interest itself, its a stepping stone—architectures that only work on simulations that afford multiple identical environments but fail in the real world have very limited application. I think this paper needs to motivate why parallel training in this way in these robotics inspired domains is interesting and worthwhile. The main area of concern with this paper is the experiment section. There are several issues/questions I would like the authors to address: 1) You built on top of TD3, however you just used the parameter settings of TD3 as published and didn’t tune them. This is a problem because it could just be that the existing parameter choices for TD3 were just better for the new approach. You have to take additional effort in this case to ensure your method is actually better than just using TD3. Additional parameter tuning of TD3 is required here. 2) I think its an odd choice for TD3 to have an infinite buffer, as recent work has show at least for DQN that large buffers can hurt performance [7]. Can you justify this choice beyond “the authors of TD3 did it that way”? 3) Why is R_eval different for each method? 4) Why did you not compare to TD3 on the same set of domains as used in the TD3 paper? Why a subset? Why these particular domains? 5) In 2 of the 4 domains the proposed method ties or is worse than the baselines. In half-cheetah it looks close to significant, and in the ant domain the result is unlikely to be significant because the error-bars overlap and the error-bars of TD3 are wider than the other methods so a simple visual inspection is not enough. There does not seem to be a strong case for the new method here. I may be misunderstanding the results. Help me see the significance. 6) The paper claims improvement in variance, but this requires additional analysis in the form of an F-test of better. 7) Why these baselines (e.g., SAC) and not others? Why did you not include D4PG [6]? Soft Q-learning? A population-based training method [3,4,5,8] to name a few? [1] GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning [2] IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures [3] Human-level performance in first-person multiplayer games with population-based deep reinforcement learning [4] Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents [5] Structured Evolution with Compact Architectures for Scalable Policy Optimization [6] Distributed Distributional Deterministic Policy Gradients [7] A Deeper Look at Experience Replay [8] Evolution Strategies as a Scalable Alternative to Reinforcement Learning Small things that did not impact the score: 1) references to “search interval” in the abstract are confusing because the reader has not read the paper yet 2) Description of the method in abstract is too specific 3) P1 intro, not a topic sentence for what follows 4) “performs an action to its environment” >> grammar 5) “One way to parallel learning…” >> grammar 6) “that the value parameter and” >> grammar 7) “pitfalls” >> minima 8) Did you try combining you method with other base off-policy methods? how did it work? 9) GAE undefined? 10) “among the baseline and”>>grammar…there are many grammar errors