Population-Guided Parallel Policy Search for Reinforcement Learning
Authors: Whiyoung Jung
Giseung Park
Youngchul Sung
Venue:  (2020)
Keywords: Reinforcement Learning
Parallel Learning
Population Based Learning

Abstract: In this paper, a new population-guided parallel learning scheme is proposed to enhance the performance of off-policy reinforcement learning (RL). In the proposed scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The key point is that the information of the best policy  is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search region by the multiple learners. The guidance by the previous best policy and the enlarged  range enable faster and better policy search, and monotone improvement of the expected cumulative return by the proposed scheme is proved theoretically. Working algorithms are constructed by applying the proposed scheme to the twin delayed deep deterministic (TD3) policy gradient algorithm, and numerical results show that the constructed P3S-TD3 outperforms most of the current state-of-the-art RL algorithms, and the gain is significant in the case of sparse reward environment.

Published as a conference paper at ICLR 2020
POPULATION-GUIDER PARALLEL POLICY SEARCH FOR REINFORCEMENT LEARNING
Whiyoung Jung, Giseung Park, Youngchul Sung' School of Electrical Engineering
Korea Advanced Institute of Science and Technology {wy- jung, gs. gark, ycsung]ekat wtt.ett...................aaee.e st ac.
ABSTRACT
In this paper, a new population eppulation-guided parallel learning scheme is proposed to en- hance the performance off-policy reinforcement learning (RI.). In the proposed scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collabo- ration with the guidance of the best policy information. The key point is that the information of the best policy is fused augmented loss function for policy update enlarge the overall search region by the multiple learners. The guidance by the previous best policy and the enlarged range enable faster and better policy search. Monotone improvement of the ex- pected cumulative return by the proposed scheme is proved theoretically. Working algorithms are constructed by applying the proposed scheme to the twin delayed deep deterministic (TD3) policy gradient algorithm. Numerical results show that the constructed algorithm outperforms most of the current state-of-the-art art RL al- gorithms, and the gain is significant in the case of sparse reward environment.
soft manner by constructing an
INTRODUCTION
RL is an active research field and has been applied successfully to games, simulations, and ac- tual environments. With the success of RL in relatively easy tasks, more challenging tasks such as sparse reward environments (Oh et al. (2018); Zheng et al. 2018); Burda et al.) (2019) are emerg- ing, and developing good RL algorithms for such challenging tasks is of great importance from both theoretical and practical perspectives. In this paper, we consider parallel learning, which is an important line of RI research enhance the learning performance by having multiple learners for the same environment. Parallelism in learning has been investigated widely in distributed RL (Nair al. (2015): Mnih al. (2016); Horgan et al. (2018): Barth-Maron et al. (2018); [speholt
et al. (2018), )) evolutionary algorithms (Salimans et al. (2017), Choromanski et al. (2018) Khadka
& Tumer pp; Pourchot& & Sigaud (2019)), concurrent RL (Silver al. (2013); Guo Brunskill
Dinakooolnooooloo & Van Roy (2088) Dimakopoulou et al. (2018)) and population- train-
ing (PBT) (Jaderberg et al. (201 2018); Conti al. (2018)) this paper, in order to enhance
learning performance, we apply parallelism to RI. based on population of policies, but the usage is different from the previous methods.
One of the advantages of using population is the capability to evaluate policies in the population. Once all policies the population are evaluated, we can use information the best policy to en- hance the performance. One simple way to exploit the best policy information is that policy parameter of each learner with that of the best learner at the beginning of the next M time steps; make each learner perform learning from this initial point for the next M time steps; select the best learner again at the end of the next M time steps; and repeat this procedure every M time steps in a similar way that PBT does (Jaderberg et al. (2017). We will refer this method as the resetting method in this paper. However, this resetting method has the problem that the search area covered by all N policies the population collapses to one point at the time of parameter copying and thus the search area can be narrow around the previous best policy point. To overcome such disadvantage, instead of resetting the policy parameter with the best policy parameter periodically,
reset the
"Corresponding author

---

Published as a conference paper at ICLR 2020
propose using the best policy information in a soft manner. In the proposed scheme, the shared best policy information is used only to guide other learners' policies for searching a better policy. The chief periodically determines the best policy among all learners and distributes the best policy parameter to all learners so that the learners search for better policies with the guidance of the pre- vious best policy. The chief also enforces that the N policies are spread in the policy space with given distance from the previous best policy point so that the search area by all N learners maintains a wide area and does not collapse into a narrow region.
The proposed Population-guided Parallel Policy Search (P3S) learning method can be applied to any off- off-po RL algorithms and implementation easy. Furthermore, monotone improvement of the expected cumulative return by the P3S scheme is theoretically proved. We apply our P3S scheme to the TD3 algorithm, which is state-of-the-art off-policy algorithm, as our base algorithm. Numerical result shows that the P3S-TD3 algorithm outperforms the baseline algorithms both in the speed of convergence and in the final steady-state performance.
BACKGROUND AND RELATED WORKS
Distributed RL Distributed RL is an efficient way of taking advantage of parallelism to achieve fast training for large complex tasks (Nair al. (2015). Most of the works in distributed RL assume common structure composed of multiple actors interacting with multiple copies of the same environment and central system which stores and optimizes the common Q-function parameter or the policy parameter shared by all actors. The focus of distributed RL is to optimize the Q-function parameter or the policy parameter fast by generating more samples for the same wall clock time with multiple actors. For this goal, researchers investigated various techniques for distributed RL, e.g., asynchronous update of parameters (Mnih et al. (2016); Babaeizadeh et al. (2017)) ), sharing an experience replay buffer (Horgan et al. (2018)) GPU- based parallel computation (Babaeizadch et al. (017): Clemente et al. (2017)), GaU-b simulation (Liang et al. (0088) and V-trace in case of on-policy algorithms (Espeholt et al. (2018). Distributed RL yields performance improvement in terms of the wall clock time but it does not consider the possible enhancement by interaction among a population of policies of all learners like in PBT or our P3S. The proposed P3S uses a similar structure to that in (Nair et al. (2015); [spehhot et al. (2018)) that is, P3S is composed of multiple learners and chief. The difference is that each learner in P3S has its own Q or value function parameter and policy parameter, and optimizes the parameters in parallel to search in the policy space.
Population-Base Based Training Parallelism is also exploited in finding optimal parameters and hyper- parameters of training algorithms in PBT (Jaderberg et al.] (2017) 2018); Conti et al. (2018). PBT trains neural networks, using population with different parameters and hyper-parameters in paral- lel at multiple learners. During the training, in order to take advantage of the population, it evaluates the performance of networks with parameters and hyper-parameters in the population periodically. Then, PBT selects the best peper-parameters distributes the best syper-parameters and the corre- sponding parameters to other learners, and continues the training of neural networks. Recently, PBT is applied to competitive multi-agent RL. (Jaderberg et al. (2018) and novelty search algorithms search a better policy by exploiting
(Conti et al. (2018)) The proposed P3S uses a population the best policy information similarly to PBT, but the way of using the best policy information is different. In P3S, the parameter of the best learner is not copied but used in a soft manner to guide the population for better search in the policy space.
Guided Policy Search Our P3S method also related to guided policy search (Levine Koltun (013); Levine et al.) (2016); Teh et al. (2017); Ghosh et al. (2018)) Teh et al. (207)) proposed a guided policy search method for joint training multiple tasks in which a common policy used to guide local policies and the common policy is distilled from the local policies. Here, the local policies' parameters are updated to maximize the performance and minimize the KL divergence between the local policies and the common distilled policy. The proposed P3S is related to guided policy search in the sense that multiple policies are guided by a common policy. However, the difference is that the goal of P3S is not learning multiple tasks but learning optimal parameter for common task as in PBT. Hence, the guiding policy is not distilled from multiple local policies but chosen the best performing policy among multiple learners.
2

---

Published as a conference paper at ICLR 2020
Exploiting Best Information Exploiting best information has been considered in the previous works (White & Sofge (1992): Oh et al. (2018): Gangwani et al. (2019). In particular, Oh et al. (2018);
Gangwani et al. 2019) exploited past good experiences to obtain a better policy, whereas P3S
ploits the current good policy among multiple policies to obtain a better policy.
3 POPULATION-GUDDE PARALLEL POLICY SEARCH
The overall structure of the proposed P3S scheme is described in Fig. [ We have N iden- tical parallel learners with a shared common experience replay buffer D, and all N identi- cal learners employ a common base algorithm which can be any off-policy RL algorithm. The execution is in parallel. The i-th learner has copy of
Learners
Networks @%P%
Expert eaeeen
nowurd Uila Belanciin
Numdrr
Update R.4'
its own environment Eú, which the common environment E, and has its own value function (e.g., Q-function) parameter 8* and policy parameter ø²_ The i-th learner in- teracts with the environment copy E with ad- ditional interaction with the chief, as shown in Fig. I At each time step, the i-th learner performs an action aj to its environment copy E by using its own policy ggs, stores its experience (s), at, r/, (1+1) the shared common replay buffer D for all i = 1.2,.... .N. Then, each learner updates its value function parameter and policy parameter once by drawing mini-batch of size B from the shared common replay buffer 'D by minimizing its own value loss function and policy loss function, respectively.
Replay Buffer P
Parameters p.
Chief
Figure 1:
The overall structure of P3S
Due to parallel update of parameters, the policies of all learners compose population of N different policies. In order to take advantage of this population, we exploit the policy information from the best learner periodically during the training like in PBT (Jaderberg et al. (2017). Suppose that the Q-function parameter and policy parameter of each learner are initialized and learning is performed as described above for M time steps. At the end of the M time steps, we determine who the best learner based on the average of the most recent F,r episodic rewards for each learner. Let the index of the best learner b. Then, the policy parameter information Ø of the best learner can be used to enhance the learning of other learners for the next M time steps. Instead of copying d² to other learners like in PBT, we propose using the information &² in a soft manner. That is, during the next M time steps, while we set the loss function ((O²) for the Q-function to be the same as the loss L(O') of the base algorithm, we set the loss function [(di) learner as the following augmented version:
) for the policy parameter ø² of the i-th
ī(o')= = L(d')
[TT )
(I)
where L(ø') is the policy loss function of the base algorithm. (g) denotes the indicator function, 3(> 0) is weighting factor, D(x,
(1x π²) be some distance measure between two policies π and x'.
3.1 THEORETICAL GUARANTEE OF MONOTONE IMPROVEMENT OF EXPECTED CUMULATIVE RETURN
In this section, we analyze the performance of the proposed soft-fusion approach theoretically and show the effectiveness of the proposed soft-fusion approach. Consider the current update period and its previous update period. Let fod be the policy of the i-th learner at the end of the previous "
update period and let Tha be the best policy among all policies sald 1,..., N. ,. Now, consider any learner who is not the best in the previous update period. Let the policy of learner in the current update period be denoted by Tai, and let the policy loss function of the base algorithm be denoted as L(TT))). order to analyze the performance, we consider L(Tx:) in the form of
L(TT:) )
Eg-D,arT (-1s)
-" (s,a).. The reason behind this choice that most actor- the the policy is
critic methods update the value (or Q- ffnnctio and the policy iteratively. That is, for given gold a
Q-function is first updated approximate Q old
0' Then, with the approximation "T old
3

---

Published as a conference paper at ICLR 2020
updated yield an updated policy " This procedure is repeated iteratively. Such loss function is used in many RL algorithms such as SAC and TD3 (Haarnoja et al. (2018) Fujimoto et al. 2018 8018). For the distance measure D(T, x') between two policies и and T', we consider the KL divergence (I) the augmented loss function for non-best learner at the
KL(T (')) for analysis. Then, by eq- current update period is expressed as
[(TT:)) = BBNDDaNN
( Q nold (8, (s, + B.B.D (1as
(2)
[al(a]))
=
EDDD Bawr (11b)
(11b)
x=*** (s, + 3 log TTh (als)
(3)
Let " be solution that minimizes the augmented loss function eq. conditions.
We assume the following
Assumption 1. For all 8,
Eavn (+16) 10t00 (s 2 E (awrrttrs)ss) 1
vold (s)
(A1)
Assumption 2. For some p, d > 0,
Kl. ( new TT (+18)]| (-18) 2 max p KL. (T (-10"(-1))) (1150) (16)) .d). .d) Vs.
(A2)
Assumption I means that if we draw the first time step action from T& and the following actions from n²g then this yields better performance on average than the case that we draw all actions including the first time step action from zold This makes sense because the definition of w Assumption 2 is about the distance relationship among the policies to ensure a certain level of spreading of the policies for the proposed soft-fusion approach. With the two assumptions above, we have the following theorem regarding the proposed soft-fusion parallel learning scheme:
Theorem 1. Under Assumptions
andd the following inequality holds:
(a) a), 2 TT BE 18
(6) QT (s,a) V(s,a), Vi #bb..
Q
(SK))
Improvement gop
(4)
where
A(s) KL TT Tn (17)(()))) max {pmax KL
(1.1 (11s)(()s) (111)))))))))) .d
(5)
Here, inequality (a) requires Assumption only and inequality (b) requires Assumption
Proof. See Appendix A.
Theorem states that the new solution -new for the current update period with the augmented loss function ylelds better performance (in the expected reward sense) than the best policy T of the previous update period for any non-best learner of the previous update period. Hence, the proposed parallel learning scheme yields monotone improvement of expected cumulative return.
3.2
IMPLEMENTATION
The proposed P3S method can be applied to any off- off-policy base RL algorithms whether the base RL algorithms have discrete or continuous actions. For implementation, we assume that the best policy update period consists of M time steps. We determine the best learner at the end each update period based on the average of the most recent E, episodic rewards of each learner. The key point in implementation is adaptation of B3 so that the improvement gap SE 51 wup _ >*=AA(sx) in becomes non-negative and is maximized for given p and d. The gradient of the im- provement gap with respect to is given by À == E Be [E AA='A(s))), and A the average (with forgetting) of A(Sk) by using samples from n. Hence, if A 0, i.e.,
4

---

Published as a conference paper at ICLR 2020
d on average, then / should be increased maximize the performance gain. Otherwise, 8 should be decreased. There- fore, we adopt the following adaptation rule for 33 which is common for all non-best learners:
KL To Th (1s))] > max p KL (r (1 TT
28 if D Depread max max{pDerange dmnin) 1.5 8/2
B
(6)
if D Depread < max &max{p [pD change: dmnin) /1.5
N-I EiEr n BAND plage" .Tpe ] is the estimated distance between T and Tg. and D Dohange N-T Es LOND D(mgr mg)dt) is the estimated distance between x new
Here, D Dapread
and rold averaged over all N non-best learners, where dmnn and p predetermined hyper- parameters. D Dspread and m dmin } are our practical implementations of the left-hand side (LHS) and the right-hand side (RHS) of eq. (22.. respectively. This adaptation method is similar that used in PPO (Schulman et al. (2017).
The update (6) of B3 is done every M time steps and the updated / is used for the next M time steps. As time steps elapse, is settled down so that Dspread around dsearch and this implements Assumption 2 with equality. Hence, the pro- posed P3S scheme searches a spread area with rough radius deearch around the best policy in the policy space, as illustrated in Fig. 2 The search radius dsearch is determined proportionally Dehange that represents the speed of change in each learner's policy. In the case of being stuck in local optima, the change Dehange can be small, making the search area narrow. Hence, we set minimum search radius dmin to encourage escaping out of local optima.
TT dagieveh Tg
T
maxapDehange: dminn
The conceptual search coverage in the policy space by parallel learners
Figure 2:
We applied P3S to TD3 as the base algorithm. The constructed algorithm is named P3S-TD3. The details of TD3 is explained in Appendix G. We used the mean square difference given by D(T(s), (s)) 1/2 [|](s) (s))) the distance measure between two policies for P3S-TD3. Note that if we consider two deterministic policies as two stochastic policies with same standard deviation, the KL divergence between the two stochastic policies is the same the mean square difference. For initial exploration P3S-TD3 uses a uniform random policy and does not update all policies over the first Tinitial time steps. The pseudocode of the P3S-TD3 given in Appendix H. The implementation code for P3S-TD3 is available at ttthttp::/// /ttp:////tttttt httt:::://gttuubb.omm/////yp/ppp wwn.o///wwj0g022220
p3s
EXPERIMENTS
4.1
COMPARISON TO BASELINES
In this section, provide numerical results on performance comparison between the proposed P3S-TD3 algorithm and current state-of-the-ard on-policy and off-policy haseline algorithms on sev- eral MuJoCo environments (Todorov et al. (2012)). The baseline algorithms are Proximal Policy Optimization (PPO) (Schulman et al. (2017)), Actor Critic using CroneckerFacctore Trust Region (ACKTR) (Wu et al. (2017). Soft Q-learning (SQL) (Haarnoja et al. (2017)) (clipped double Q) Soft Actor-Critic (SAC) (Haarooj et al. (2018) and TD3 (Fujimoto et al. (2018)).
Hyper-parameter setting All papprrrrprattersss we used for evaluation are the same as those in the original papers (Schulman et al. (2017); :): Wuu et al. (2017);] Haarnoja et al.] (2017) (018);; Fujimoto al. (2088). Here, we provide the hyper-parameters of the P3S-TD3 algorithm only, while details for
of the lyper-paraametes for TD3 are provided in Appendix I. On top of the hyper-parameter the base algorithm TD3, we used = learners for P3S-TD3. To update the best policy and 3, the period M 250 is used. The number of recent episodes E,- 10 was used to determine the best learner b. For the search range, we used the parameter = 2, and tuned dman among dmin {0.02, 0.05) for all environments. Details dmin for each environment is shown in Appendix The time steps for initial exploration Tiitiia is set as 250 for Hopper-v1 and Walker2d- vl and as 2500 for HalfCheetah-vl and Ant-v1.
5

---

Published as a conference paper at ICLR 2020
(a) Hopper-vl
(b) Walker2d-vl
(c) HalfCheetah--v
(d) Ant-v1
Figure 3: Performance for PPO (red), ACKTR (purple), SQL (brown), (clipped double Q) SAC
(orange), TD3 (green), and P3S-TD3 (proposed method, blue) on MuJoCo tasks.
Evaluation method Fig. shows the learning curves over one million time steps for several MuJoCo tasks: Hopper-v1, Walker2d-v1, HalfCheetah-vl, and Ant-v1. In order to have sample-wise fair comparison among the considered algorithms, the time steps in the z-axis in Fig. for P3S-TD3 is the sum of time steps of all N users. For example, in the case that N 4 and each learner performs 100 time steps in P3S-TD3, the corresponding æ-axis value is 400 time steps. Since each learner performs parameter update once with interaction with environment per each time step in P3S-TD3, the total number of parameter updates at the same r-axis value in Fig. 3 is the same for all algorithms including P3S-TD3, and the total number of interactions with environment the same r-axis value in Fig. 3 is also the same for all algorithms including P3S-TD3. Here, the performance obtained through the evaluation method which is similar to those in Haarnoja et al. (2018); Fujimoto et al. (2018). Evaluation of the policies is conducted every Reeat = 4000 time steps for all algorithms. At each evaluation instant, the agent (or learner) fixes its policy as the one at the evaluation instant, and interacts with the same environment separate for the evaluation purpose with the fixed policy to obtain 10 episodic rewards. The average of these 10 episodic rewards the performance at the evaluation instant. In the case of P3S-TD3 and other parallel learning schemes, each of the N learners fixes its policy as the one at the evaluation instant, and interacts with the environment with the fixed policy to obtain 10 episodic rewards. First, the 10 episodic rewards are averaged for each learner and then the maximum of the 10-episode-avcrage rewards of the N learners is taken the performance at that evaluation instant. We performed this operation for five different random seeds, and the mean and variance of the learning curve are obtained from these five simulations. The policies used for evaluation are stochastic for PPO and ACKTR, and deterministic for the others.
Performance on MuJoCo environments In Fig. 3 it is observed that all baseline algorithms is similar to that in the original papers (Schulman et al. 2017): Haarnoja et al. (2018); Fujimooo et al. (2018)) With this verification, we proceed compare P3S TD3 with the baseline algorithms. It is seen that the P3S-TD3 algorithm outperforms the statc-of-the the speed of convergence with respect to time steps and the final steady-state performance (except in Walker2d-v1, the initial convergence is bit slower than TD3.) Especially, in the cases of Hopper-vl and Ant-v1, TD3 has large variance and this implies that the performance TD3 is quite dependent on the initialization and it is not easy for TD3 to escape out of bad local minima resulting from bad initialization certain environments. However, it is seen that P3S-TD3 yields much smaller variance than TD3. This implies that the wide area search by P3S in the policy space helps the learners escape out of bad local optima.
RL algorithms in terms of both
4.2 COMPARISON WITH OTHER PARALLEL LEARNING SCHEMES AND ABLATION STUDY
In the previous subsection, we observed that P3S enhances the performance and reduces dependence on initialization as compared to the single learner case with the same complexity. In fact, this should parallel learning scheme. Now, in order to demonstrate
be accomplished by any groperly-designee the true advantage of P3S, we compare P3S with other parallel learning schemes. P3S has several components to improve the performance based on parallelism: I) sharing experiences from multiple policies, 2) using the best policy information, and 3) soft fusion of the best policy information for wide search area. We investigated the impact of each component on the performance improvement. For comparison we considered the following parallel policy search methods gradually incorporating more techniques:

---

Published as a conference paper at ICLR 2020
(a) Hopper-vl
(b) Walker2d-vl
(c) HalfCheetah-vy
(d) Ant-v1
(c) Del. Hopper-v1
(f) Del. Walker2d-v1
(g) Del. HalfChoctah-vl
(h) Del. Ant-vl
Figure 4: Performance of different parallel learning methods on MuJoCo environments (up), on delayed MuJoCo environments (down)
1. Original Algorithm The original algorithm (TD3) with one learner
2. Distributed RL (DRL) N actors obtain samples from N environment copies. The com- mon policy and the experience replay buffer are shared by all N actors.
3. Experience-Sharingg Only (ESO) N learners interact with N environment copies and up- date their own policies using experiences drawn from the shared experience replay buffer.
4. Resetting (Re) At every M' time steps, the best policy is determined and all policies are initialized as the best policy, i.e., the best learner's policy parameter is copied to all other learners. The rest of the procedure is the same as experience-sharing--aa
algorithm.
5. P3S At every M time steps, the best policy information is determined and this policy is used in soft manner based on the augmented loss function.
Note that the resetting method also exploits the best policy information from N learners. The main difference between P3S and the resetting method is the way the best learner's policy parameter is used. The resetting method initializes all policies with the best policy parameter every M' time steps like in PBT (Jaderberg al. (2017)). whereas P3S algorithm uses the best learner's policy parameter information determined every M time steps to construct an augmented loss function. For fair comparison, M and M' are determined independently and optimally for P3S and Resetting, respectively, since the optimal period can be different for the two methods. We tuned M' among (2000, 5000, 10000) (MuloCo environments) and (10000, 20000, 50000) (Delayed MuloCo envi- ronments) for Re-TD3, whereas M = 250 was used for P3S-TDD TD3. The specific parameters used for Re-TD3 are in Appendix Since all N policies collapse one point in the resetting method the beginning of each period, we expect that a larger period is required for resetting to have sufficiently spread policies at the end of each best policy selection period. We compared the performance of the aforementioned parallel learning methods combined with TD3 on two classes of tasks; MuJoCo environments, and Delayed sparse reward MuJoCo environments.
Performance on MuJoCo environments The upper part of Fig, 4shows the learning curves of the considered parallel learning methods combined with TD3 for the four tasks (Hopper-vl, Walkerd-v1, HalfCheetah-vl and Ant vl). It is seen that P3S-TD3 outperforms other parallel methods: DRL- TD3, ESO-TD3 and Re-TD3 except the case that ESO-TD3 or Re-TD3 slightly outperforms P3S- TD3 in Hopper-vl and Walker2d-v1. In the case of Hopper-vl and Walker2d-v1, ESO-TD3 has better final (steady-state) performance than all other parallel methods. Note that ESO-TD3 obtains most diverse experiences since the N learners shares the experience replay buffer but there is no interaction among the N learners until the end of training. So, it seems that this diverse experience beneficial to Hopper-vl and Walker2d-v1.

---

Published as a conference paper at ICLR 2020
[0m
60
(a)
(b)
(c)
Figure 5: Ablation study of P3S-TD3 on Delayed Ant-vl: (a) Performance and / (1 seed) with dmin 0.05, (b) Distance measures with dnainn 0.05, and (c) Comparison with different dinin = 0.02, 0.05
Performance on Delayed MuJoCo environments Sparse reward environments especially require more search to obtain a good policy. To see the performance of P3S in sparse reward environments, we performed experiments on Delayed MuloCo environments. Delayed MuJoCo environments are reward-sparsifee versions of MuloCo environments and used in /Zeng et al. (218). Delayed Mu- JoCo environments give non-zero rewards periodically with frequency freward or only at the end of episodes. That is, in delayed MuJoCo environment, the environment accumulates rewards given by the corresponding MuJoCo environment while providing zero reward to the agent, and gives the accumulated reward the agent. We evaluated the performance the four delayed environments with freward = 20: Delayed Hopper-v1, Delayed Walker2d-v1, Delayed HalfCheetah-v/ and De- layed Ant-v1.
The lower part of Fig. shows the learning curves of the different parallel learning methods for the four delayed MuloCo environments. It is seen that P3S outperforms all other considered parallel learning schemes on all environments except on delayed Hopper-v1. It seems that the enforced wide-area policy search with the soft-fusion approach in P3S is beneficial improve performance in sparse reward environments.
Benifits of P3S Delayed Ant-v1 is case of sparse reward environment in which P3S shows signif- icant improvement as compared to other parallel schemes. As shown in Fig. 4h the performance of TD3 drops below zero initially and converges zero as time goes. Similar behavior is shown for other parallel methods except P3S. This is because in Delayed Ant-v1 with zero padding rewards between actual rewards, initial random actions do not generate significant positive speed to a for- ward direction, so it does not receive positive rewards but receives negative actual rewards due the control cost. Once its performance less than 0, learners start learning doing nothing to reach zero reward (no positive reward and no negative reward due to no control cost). Learning beyond this seems difficult without any direction information for parameter update. This is the interpretation of the behavior of other algorithms in Fig. 4h However, it seems that P3S escapes from this local optimum by following the best policy. This is evident in Fig. 5a showing that after few time steps, is increased to follow the best policy more. Note that at the early stage of learning. the perfor-
mance difference among the learners large as seen in the large D Depread values in Fig. 5bb As time elapses, all learners continue learning, the performance improves, and the spreadness among the learners' policies shrinks. However, the spreadness among the learners' policies is kept at a certain level for wide policy search by dmiin as seen in Fig. [b Fig. 5c shows the performance of P3S with dmnn = 0.05 and 0.02. It shows that a wide area policy search is beneficial as compared to a narrow area policy search. However, it may be detrimental to set too large value for dmin due to too large statistics discrepancy among samples from different learners' policies.
5 CONCLUSION
In this paper, we have proposed a new population-guie guidedd parallel learning scheme, P3S, to enhance the performance of off-policy RL. In the proposed P3S scheme, multiple identical learners with their own
and policies sharing a common experience replay buffer search a good policy with the guidance of the best policy information in the previous search interval. The information of the best policy parameter of the previous search interval is fused in a soft manner by constructing an augmented loss function for policy update enlarge the overall search region by the multiple
8

---

Published as a conference paper at ICLR 2020
learners. The guidance by the previous best policy and the enlarged search region by P3S enables faster and better search the policy space, and monotone improvement of expected cumulative return by P3S is theoretically proved. The P3S-TD3 algorithm constructed by applying the proposed P3S scheme to TD3 outperforms most of the current state-of-the-art RI. algorithms. Furthermore, the performance gain by P3S over other parallel learning schemes is significant harder environments especially on sparse reward environments by searching wide range in policy space.
ACKNOWLEDGMENTS
This work was supported in part by the ICT R&D program of MSIP/IITP (2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Com- panion) and in part by the National Research Foundation of Korea(NRF) grant funded by the Korea covernment(Minnssty of Science and ICT) 1NRF2077R1E1A1A0000000
REFERENCES
Mohammad Babacizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforcement learning through asynchronous advantage actor-critic on a GPU. International Conference on Learning Represeniations, Apr 2017.
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. International Conference on Learning Representations, Apr 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. May 2019.
Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard E. Turner, and Adrian Weller. Structured evolution with compact architectures for scalable policy optimization. In Proceedings of the 35th International Conference on Machine Learning, pp. 970-978, Jul 2018.
Alfredo V. Clemente, Humberto N. Castejón, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint arXiv:1705.04880 2017.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. pp. 5027-5038, Dec 2018.
Maria Dimakopoulou and Benjamin Van Roy. Coordinated exploration in concurrent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1271-1279, Jul 2018.
Maria Dimakopoulou, Ian Osband, and Benjamin Van Roy. Scalable coordinated exploration in concurrent reinforcement learning. In Advances in Neural Information Processing Systems, pp- 4223-4232, Dec 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed Reep-RL. with importance weighted actor-learner architectures. In Proceed- ings of the 35th International Conference on Machine Learning, pp. 1407-1416, Jul 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference Machine Learning, pp. 1587-1596, Jul 2018.
Tanmay Gangwani, Qiang Liu, and Jian Peng. Learning self self-imitating diverse policies. In Interna- tional Conference on Learning Representations, May 2019.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran. Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. In International Conference on Learning Representations, Apr 2018.

---

Published as a conference paper at ICLR 2020
Zhaohan Guo and Emma Brunskill. Concurrent PAC RL. In AAAI Conference on Artificial Intelli- gence, pp. Jan 2015.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning. pp. 1352-1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, pp. 1861-1870, Jul 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Has- selt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations, Apr 2018.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. arXiv preprint arXiv:1711.09846. 2017.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, David Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-baseee deep reinforcement learning. arXiv preprint /807.01288 2018.
Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1196- 1208, Dec 2018.
Sergey Levine and Vladlen Koltun. Guided policy search. In Proceedings of the 30th International Conference on Machine Learning, pp. 1-9, 2013.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):13341373,, 2016.
Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. GPU-accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning, pp. 270-282, 2018.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 10Xi::1509002990 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, pp. 1928- 1937, 2016.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattic, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuogiu, and David Silver. Massively parallel methods for deep reinforcement learning. arXiv preprint 1507.04299 2015.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Proceedings of the 35th International Conference on Machine Learning, pp. 3878-3887, 2018.
Aloïs Pourchot and Olivier Sigaud. CEM-RL: Combining evolutionary and gradient- baaee methods for policy search. In International Conference on Learning Representations, 2019.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative reinforcement learning, arXiv preprint 1X03::7733.03666 2017.
10

---

Published as a conference paper at ICLR 2020
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, pp. 18899-1997, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.06347, 2017.
David Silver, Leonard Newnham, David Barker, Suzanne Weller, and Jason McFall. Concurrent re- inforcement learning from customer interactions. International Conference on Machine Learn- ing, volume 28, pp. 924-932, Jun 2013.
Yee Whye Teh, Victor Bapst, Wojeiech M. Czarnekki, John Quan, James Kirkpatrick, Raia Had- sell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4499 4509, Dec 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, 5026- 5033. IEEE, Oct 2012.
David A. White and Donald A. Sofge. The role of exploration in learning control. Handbook of Intelligent Control: Neural, Fuzzy and Adaptive Approaches, pp. 1-27, 1992.
Yuhuai Wu, Elman Mansimov, Roger Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using fronecker-factored approximation. In Advances in Neural Information Processing Systems, pp. 5279-5288, Dec 2017.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp. 4649-4659 Dec 2018.

---

Published as a conference paper at ICLR 2020
APPENDIX A. PROOF OF THEOREM 1
In this section, we prove Theorem 1. Let " be the policy of the i-th learner the end of the previous update period and let Nge be the best policy among all policies 2 = 1, ... N. Now, consider any learner who is not the best in the previous update period. Let the policy of learner in the current update period be denoted by Tgi, and let the policy loss function of the base algorithm be denoted as L(TT:)) ), given in the form of
L(T = E BawDDa (-1s) [-Q (s.
(7)
The reason behind this choice is that most of actor-critic methods update the value (or Q- (-))unntion and the policy iteratively. That is, for given nold the Q-function is first updated so as to approximate
Q Q"o" Then, with the approximation Q" old the policy is updated to yield updated policy T and this procedure is repeated iteratively. Such loss function is used in many RI algorithms such as SAC and TD3 (Haarnoja et al.] (2018); Fujimoto et al. (2018)). SAC updates its policy by mini- mizing E aa 1-Q Rord (s. + log TT' (a(s) over TD3 updates its policy by minimizing Bg>D,a=x'(s) --Q as) (s,
With the loss function eq. 7 and the KI. divergence KL(TT||²) the distance measure D(T, x') between two policies и and as stated in the main paper, the augmented loss function for non-best learner at the current update period is expressed as
[(xx))) = ESnD (((l))
- Q-00 ot (s, (s, + SE (T (T))
(s,
(8)
T (a|s)
= EsDD
E Bawnge(la)
-Qaat (8, + 3 log
(9)
Tah
Let 2 be solution that minimizes the augmented loss function eq. (9) Assumption 1. For all 8,
Ea (c16) @Q (s, as 2 E (owrot((1)) (-1b) 1 (s)
(10)
Assumption 2. For some p. >
KL (118)l (-18)) 2 max maxx KL Inew (= (-1o))))) .d). Vs.
new T
(11)
For simplicity of notations, we use the following notations from here on.
- for Tg.
ata for n
Thew for new
h for Tgk.
KLmax (Tieu (Tad) for max, KL ( (-Is")) (-1s))
A.1. 1. A PRELIMINARY STEP
Lemma 1. Let TO Thew be the solution of the augmented loss function eq. 9
Then, with Assumption
have the following:
Eawni (Qriua(s, an) 2 Banaiig(1))) (Qr
(12)
for all
12

---

Published as a conference paper at ICLR 2020
Proof. For all
E (a-
[- Qran
2 E a (4)
(s,
(13)
x (als)
(s, + Blog
(14)
xa|ss)
(n) + Blog T (als)
E Ba hew (118)
(15)
(alss)
E E (-1s)
(s,
(16)
where Step (a) holds by Assumption divergence is always non-negative.
1
(b) holds by the definition of कं neww and (c) holds since KL
With Lemma]
I
we prove the following preliminary result before Theorem 1:
Proposition 1. With Assumption 7 the following inequality holds for all and a:
nnww (s, a) 2 1 (s,a).
(17)
Proof of Proposition For arbitrary 8 and ar.
To (st, (z)
Bartinauue (Q (8411,
r(s, at) + +-7 Bn
(18)
r(t, at) + (a)
E Eastiwaten
(19)
E, Bis
r(s,, at) + 77(8441, a(41) + Bartono B
(20)
E Be 1 (b)
<
(((), + (810011,0041) + 7² Barzzrate
(21)
(22)
E B mi hew
(sk,, as)
(23)
=Qriew (st, at),
(24)
where p(-1st, at) in eq. 18 is the environment transition probability, and 81+1 8t+2 ~ Thew in eq. 20 means that the trajectory from St-1 $142 is generated by T together with the environment transition probability p(-1st, oz). (Since the use of p(-1se, at) is obvious, we omitted p(-1st, 0z) for
notational simplicity.) Steps (a) and (b) hold due to Lemma 1
A.2. PROOF OF THEOREM I
Proposition states that for a non-best learner i, the updated policy Thew with the augmented loss function yields better performance than its previous policy Thas but Theorem states that for a non- best learner i, the updated policy Thew with the augmented loss function yields better performance than even the previous best policy T..
To prove Theorem 1, we need further lemmas: We take Definition I and Lemma directly from reference (Schulman et al. (2015)).
Definition 1 (From Schulman and N' are a-coupled if Pr(a # a') S a, (a, a') ~ (m(-1s), T' (+Is)) for all s. Lemma (From Schulman et al. (2015)). Given a-coupled policies and n', for all s.
(2155). Consider two policies x and n'. The two policies
Eawyy [A a)]l < 2a max A (s a)), 3,8
(25)
where ATd (s, a) is the advantage function.
13

---

Published as a conference paper at ICLR 2020
Proof. (From Schulman et al. (2015)
Baaw [A" (s, a)ll
Bal [A a')] - Bans [A a)l
(26)
((aaa)) (A a') A (s,
(27)
Pr(a = a (aaa"")))
(s,
+ Pr(a & (a))
(s, a') (²
(28)
Pr(a & (')) Maska (A a') (s,
(29)
S 2a max a));
(30)
where Step (a) holds since Eawr [A (s, a)] for all by the property of advantage function.
By modifying the result on the state value function in Schulman et al. (2015). we have the following lemma on the Q-function: Lemma 3. Given two policies a and x', the following equality holds for arbitrary so and og:
( au) Q" ao) + TB
1-11A AT an)
(31)
where Erra' is expectation over trajectory which start from state drawn from the transition probability p(-150, an) of the environment.
Proof. Note that
("o ao) = r + 7T,, Ppppppppleease
[s()]
(32)
(su, ao) = + p V ($1)
(33)
Hence, it is sufficient to show the following equality:
EToNY
Σ Etd11 1-1 Am (st, AT at)
E Belrp(-1bo.aa)
[ (s1) -a s0,00) [s3(s))
(34)
Note that
A (st, az) (apppp r VV V=
(35)
Then, substituting eq.
35 into the LHS of eq. 344. we have
Eywrr
Σ
A AT (st, ar)
= Eyon
(re +
(36)
= EE
r
- -Errp(()ooaa))) (s)))
(37)
t==
E. (s0) [ (01) - Essrp(( [V (s1)l.
(38)
where eq.
37 is valid since
Bywr!
Ee == 111 (TV" (8141) - V**
-E,, Spt -p(- ag) [V* [V*()))]
Since the RHS of eq. 38 is the same as the RHS of eq.
34
the claim holds.
Then, can prove the following lemma regarding the difference between the Q-functions of two a-coupled policies π and n':
Lemma 4. Let and n' be a-coupled policies. Then.
(s. (s. a) & 20y where maxx.a 1A (s) and C 1-
max 2 1/C},
(39)
14

---

Published as a conference paper at ICLR 2020
Proof. From Lemma
we have
q (8o, an) (80, = 7B
Σ 7 1-1 A (8c,
(40)
Then, from eq. 44) we have
Q (s0, 00) - Q~0 (s0,,a))
1 8t, (1)
(41)
0
$
2-1
EE [A (A (s),
(42)
1=1
00
5 Σ
1/
14-1aams
2a max |A (s" 5,8
(43)
(=1
EY
2a
(44)
1-7
5 67
(Ca² 1/C)
(45)
7
$ ?
2 max (Ca?: 1/C},
(46)
where maxs,a ( (s, a) and C > 0. Here, eq. (43 is valid due Lemma eq. 45 is valid
since Ca + 1/C 2a = C (a )²²² 2 0, and eq. 46 is valid since the sum of two terms is less
than or equal two times the maximum of the two terms.
Up now, considered some results valid for given two a-coupled policies and n². On the other hand, it is shown in Schulman et al. (2015) that for arbitrary policies π and n', if we take a as the maximum (over s) of the total variation divergence max, Drv (-)(())(=(((((s)) between rT|ss) and x' (-1s), then the two policies are a-coupled with the value of a = max, Drv( ((=(())
Applying the above facts, we have the following result regarding क and T Lemma 5. For some constants d >> O,
nhe (s,a) S Qu (s, a) 3 max {oKLmox (Tiew Trta) dd for all a and where KLmav (π|(TT) max, KL (T((1s)]
(47)
Proof. For Thew and wold take a as the maximum of the total variation divergence between Thew Let this a value be denoted by à. Then, by the result of Schulman et al. (2015) mentioned the above, Thew and Tata are â-coupled with max, DrV (Teew (.1s) atd (8)). Since
and ootd i.e., a = max, Drv (xt ( (-1s))))))))
Drv Drv new (1++) (())) < (1Is) (sd()
(48)
by the relationship between the total variation divergence and the KL divergence, we have
a² max (K ( (=1=)]]==de(11))).
(49)
Now, substituting क = neww n' aadd and a â into eq. 39 and applying eq. 449 we have
@uhee (8,a) Quae (s, < [9 max { {pKLmax (Tiem (
(50)
Here, proper scaling due to the introduction of is absorbed into p 200C and can be set as and d. That is, p can be set as 3[I-5)
for some p.
20q 3(-7)0 Then, by Proposition
@ (s, - Quria(s,0) in the LHS of eq. 500 becomes @Thew (s, Quéna (s, a)
Q new (s, a) ( a). Hence, from this fact and eq. (50). we have eq. 47 This concludes proof.
15

---

Published as a conference paper at ICLR 2020
Proposition 2.
With Assumption
have
Banwhh (11s) Q (s, 2 (a @ (s,a)) + BA(s).
(51)
where
(ss) = KL (new (-1s)[==(((1s)) - max pKLmax (T Il (oid) dd}
(52)
Proof.
(exxx((-1)) [- 1-tnow (s)
(53)
Q Toa (s, - max {pKLmaz (Thewll (itaa) d}
(54)
(a)
a) 3llog T& (a|s)) max {pKImaz Wreewllrree d}
(1
@r (s, (s,
(55)
T (a|ss) 8)
new (a|s)
Bawnhou
Q raild (s, + log (alss)
33 max {pKLman (Theall (l .d
(56)
(6)
Bawshen (110)
@Rodd (s, BKL (new (11)]l== max {pKLmax (T
d.)
(57)
Bawthen (11x)
(uoia(s, a))
+ [KL (T ( max {pKLman (Tnew (
,d}]
(58)
-Bawshen (168)
(ma a) +BA(s)
(59)
Bar (11)
Q nee
ss a))
+s),s),
(60)
where step (a) is valid due to Lemma valid due to Proposition
step (b) is valid due to the definition of x and step (c) is
Finally, we prove Theorem
Theorem 1. Under Assumptions and 2, the following inequality holds:
Quhew(s,0) 2 (s, s
* 2 (s,(s, v(s,a), Vi 4b.
(61)
where
(ss) [KL (Tee (())))(=((-(s)) max { {pKLmaz (Thewll T .d]]
(62)
Proof of Theorem 7 Proof of Theorem i by recursive application of Proposition 8; and az.
For arbitrary
Q Qthew (st, 0t)
a() + - Ba [Qriew (S1+1,
(63)
2 r(st, at) 7 Beeuinhh (Qrhen (4441,041)) 30
(64)
(a)
: ((s))))) ar) + 77(5041 +
2 BuB @ (8442)
+ BE 11 h7(44+1)
(65)
E, 1:81 g (()))))) + (7(5441, + 2B, a [Qrine (8442) + 3 (
(6)
BE Be (((84+1)
(66)
16

---

Published as a conference paper at ICLR 2020
20 nh r(2, as)
+ 11 ~pp
A
(67)
k=l+1
= =Q (st, + s4
k- (ss))
(68)
where steps (a) and (b) hold because of Proposition 2 Assumption 2 ensures
A ()) KL (The [+|s]] (-|s)) max {
pKLmar (T II ,dH 2 0, Vs.
(69)
Hence, the second term in 688 is non-negative. Therefore, we have
00
Q nhw (st,
Q² (sy, az) + BB
Σ k=
7 k A(s))
(70)
2 Q (st,at)
(71)
17

---

Published as a conference paper at ICLR 2020
APPENDIX B. INTUITION OF THE IMPLEMENTATION OF & ADAPTATION
Due to Theorem 1, we have
20
@uaen (st, a1) 2 (st, (s))0a) + S8,, 11000 was
Σ
A(K))
(72)
kmt+1
Improvement gap
where
(ss) = KL new [[])][="((111) max {pKLmax
..]] 0, Vs. (73)
In deriving eqs. 722 and 773 we only used Assumption 1. When we have Assumption 2, the improvement gap term in becomes non-negative and we have
Quaen (s,, at) 2 Q² (sc.
(74)
as desired. However, in practice, Assumption should be implemented so that the im- provement gap term becomes non oon-negatvve and we have the desired result (74). The imple- mentation of the condition provement gap 3I 1 wnt [k=t+17 1 As=))( in (72) for given p and d. Let us denote 1 EEam-1 (Assk) by A. Then, the improvement gap is given by 3A. Note that A the average (with forgetting) of A (sk) ) by using samples from Th. The gradient of the improvement gap with respect
through adaptation
B._ We adapt to maximize the im-
given by
va(AA)= = Å.
(75)
Thus, if Å > 0, i.e., KL (Thew (1hee (-1ss) > max {pKLman (
(IT()) d) on average, then 3 should be increased to maximize the performance gain. On the other hand, if A < 0, ie., KL (Thew (-Is)((=((())) < max (pKLmaz (ThewllTT d} on average, then & should be de- creased. Therefore, we adapt as follows:
8 28
Depread > max change: dmin 1.5
B
(76)
B 8 3/2
if D Depread <
dmin 1.5
where D Dapread and Klaaa (New (odd)) respectively.
D Dohange
are implementations of KL (Tew (15)][ and
18

---

Published as a conference paper at ICLR 2020
APPENDIX C. COMPARISON TO BASELINES ON DELAYED MUJoCo ENVIRONMENTS
In this section, provide the learning curves of the state-of-the-art single-learner baselines delayed MuloCo environments. Fig. 6 shows the learning curves of P3S-TD3 algorithm and the single-learner baselines on the four delayed MuJoCo environments: Delayed Hopper-vl, Delayed Walker2d-v1, I, Delayed HallCheetaahvl I, and Delayed Ant-vl. It is observed that in the delayed Ant- vl environment, ACKTR outperforms the P3S-TD3 algorithm. It is also observed that P3S-TD3 significantly outperforms all baselines on all other environments than the delayed Ant-vl environ- ment.
SAC
SAC
904
504
3000
2000
1000
(le))
Time Stepe (lee))
(a) Delayed Hopper-v1
(b) Delayed Walker2d-vl
SAC
SAC
0900
PP0
700
0930
500
250
2020
0.3
3.4
0.6
1.0
6.0
02
644
3.6
1.
Time Stapo (ictl
(c) Delayed HalfChectah-vl
(d) Delayed Ant-v1
Figure 6: Performance for PPO (brown), ACKTR (purple), (clipped double Q SAC (orange), TD3 (green), and P3S-TD3 (proposed method, blue) on the four delayed MuJoCo tasks with freward = 20.
19

---

the next page Published as a conference paper at ICLR 2020
APPENDIX D. COMPARISON TO CEM-TD3
In this section, we compare the performance of TD3 and P3S-TD3 with CEM-TD3 (Pourchot & Sigaud (2019), which state-of-the-art evolutionary algorithm. CEM-TD3 uses population search a better policy as other evolutionary algorithms do. The operation of CEM-TD3 is described as follows:
1. It first samples N policies by drawing policy parameters from Gaussian distribution.
2. It randomly selects half of the population. The selected policies and common Q function are updated based on minibatches drawn from a common replay buffer.
3. Both the updated selected policies and the unselected policies are evaluated and the expe- riences during the evaluation are stored in the common replay buffer.
4. After evaluation of all N policies, takes the best N/2 policies and updates the mean and variance of the policy parameter distribution as the mean and variance of the parameters of the best 2/2 policies.
5. Steps are iterated until the time steps reach maximum.
For the performance comparision, we used the implementation of CEM-TD3 in the original pa- (Pourchot & Sigaud (2099) with the hyper-parameters provided therein. Table shows the steady state performance on MuloCo and delayed MuJoCo environments. Fig.
shows the learning curves on MuloCo and delayed MuloCo environments. It is seen that P3S-TD3 outperforms CEM-TD3 on all environments except delayed HalfCheetah-vl. Notice that P3S-TD3 significantly outperforms CEM-TD3 on delayed Walker2d-vl and delayed Ant-vl.
Table I: Steady state performance of P3S-TD3, CEM-TD3, and TD3
Environment
P3S-TD3
CEM-TD3
TD3
Hopper-vl
3705.92
3686.08
2555.85
Walker2d-v1
4953.00
4819.40
4455.51
HalfCheetah-vl
11961.44
11417.73
9695.92
Ant-v1
5339.66
4379.73
3760.50
Delayed Hopper-v1
3355.53
3117.20
1866.02
Delayed Walker2d-vl
4058.85
1925.63
2016.48
Delayed HalfCheetah-vl
5754.80
6389.40
3684.28
Delayed Ant-vl
724.50
70.44
-7.45
'The code is available http://htthhbcomm/ppprccCCCCCC
20

---

Published as a conference paper at ICLR 2020
POL TB1
3979
2030
1530
1000
103
990
CES T05
Steas
(a) Hopper-vl
(b) Delayed Hopper-vl
735-1D3
7S5-T03
3920
Rowurt
2030
0030
2000
2073
TetarTime Stero Sttt
Time Staro Dleft
Walker2d-vl
(d) Delayed Walker2d-vl
12300
F36
PSS-TDO
TC3
7000
10003
5993
6202
3000
4302
Total Time Stepe Diet
Teln Time Stepe (led)
(e) HalfCheetah--vl
Delayed HalsCheetah-vl
4030
Averrge
1030
-000
0.5
0.2
3.6
TstalTieee Sters Timt
Total Time Stpo Stps- 1icht
(g) Ant-v1
(h) Delayed Ant-v1
Figure 7: Performance of P3S-TD3, CEM-TD3, and TD3
21

---

Published as a conference paper at ICLR 2020
APPENDIX E. COMPARISON TO METHOD USING CENTER POLICY
In this section, we consider a variant of the proposed P3S- TD3 algorithm, named Center-TIDS This variant uses center policy like in Distral (Teh al. 2017)) and Divide-and-Conquer (Ghosh al. )2018). Center-TD3 has N policies and a center policy π in addition. The value and policy parameter update procedure is the same the original TD3 algorithm but the loss functions are newly defined. That is, the Q function loss is the same as the original TD3 algorithm, but the parameters of the N policies are updated based on the following loss:
L(0') EgDD -oof (s, Tas (s) + 8 2 (T) Tee (s)
(77)
The parameter loss function of Center-TD3 is obtained by replacing the best policy with the center policy in the parameter loss function of P3S-TD3. The center policy is updated in every M time steps to the direction of minimizing the following loss
B
REND
Tas (8) ne (s)
(78)
2
Center-TD3 follows the spirit of Distral (Teh al. (2017) and Divide-and-Conquer (Ghosh et al. 2018 algorithms.
We tuned and selected the syper-parametery for Center-TD3 from the sets E 10) and M {2, 20, 40, 100, 200, 500}. We then measured the performance of Center- TD3 by using the selected nyper-parameter
1, .M 40. Fig. [8 in the next page and Table 2 show the learning curves and the steady-state performance on MuJoCo and delayed MuloCo environments, respectively. It is seen that P3S-TD3 outperforms Center-Tie TD3 on all environments except Delayed HalfCheetah-vl.
Table 2: Steady state performance of P3S-TD3, Center-TD3: and TD3
Environment
P3S-TD3
Center-TD3
TD3
Hopper-v1
3705.92
3675.28
2555.85
Walker2d-v1
4953.00
4689.34
4455.51
HalfCheetah-vl
11961.44
10620.84
9695.92
Ant-v1
5339.66
4616.82
3760.50
Delayed Hopper-vl
3355.53
3271.50
1866.02
Delayed Walker2d-v1
4058.85
2878.85
2016.48
Delayed HalfCheetah-vl
5754.80
6047.47
3684.28
Delayed Ant-vl
724.50
688.96
-7.45
22

---

Published as a conference paper at ICLR 2020
3020
PIA TD1
3000
3979
2520
2030
2000
1530
1500
1000
F35-TD5
530
Conter 108
Steas
(a) Hopper-vl
(b) Delayed Hopper-vl
7SS-TD3
1SS-TD0
9332
2030
2000
2033
TetarTime Stemo Bett
TetarTime Time Staro Deef
Walker2d-vl
(d) Delayed Walker2d-vl
12300
FSs
F35-7D0
TC3
7000
10003
5933
6203
3000
4302
Total Time Stepe (ied)
Teln Time Stepe (Led)
(e) HalfCheetah-vl
Delayed HalfCheetah-vvl
103
2030
Avergge
0.5
-000
0.2
3.6
TstalTivee Steas Tomt
Total Time Stpo Stps- 1icht
(g) Ant-v1
(h) Delayed Ant-v1
Figure 8: Performance of P3S-TD3, Center-TD3, and TD3
23

---

Published as a conference paper at ICLR 2020
APPENDIX F. RESULT ON SWIMMER-V v1
Khadka & Tumer 2018): Pourchot & Sigaud (019) noticed that most deep RL. methods suffer from deceptive gradient problem the Swimmer-vl task, and most RL methods could not learn effectively on the Swimmer-vl task. Unfortunately, we observed that the proposed P3S-TD3 algo- rithm could solve the deceptive gradient problem the Swimmer-vl task either. Fig. shows the learning curves of P3S-TDS and TD3 algorithm. In Khadka & Tumer (2018). the authors pro- posed an effective evolutionary algorithm named ERL solve the deceptive gradient problem on the Swimmer-vl task, yielding the good performance on Swimmer-vl, falls short of the performance ERI. on Swimmer-vl. However, it is known that CEM-TD3 dis- cussed in Appendix D outperforms ERL other tasks (Pourchot & Sigaud (2019)). Furthermore, we observed that P3S-TD3 outperforms CEM-TD3 on most environments in Appendix D.
shown in Fig. 9 P3S-TD3
350
300
250
Average Rewards
200
ERL (Khadka Tumer (2018))
P35-TD3
TD3
150
100
50
0.2
0.4 Total Time Steps (1e6)
0.6
0.8
1.0
Figure 9: Performance Swimmer-vl of P3S-TD3 (blue), TD3 (orange), and the final performance of evolutionary RI. (Khadka & Tumer (2018) green dashed line).
24

---

Published as a conference paper at ICLR 2020
APPENDIX G. THE TWIN DELAYED DEEP DETERMINISTIC POLICY GRADIENT (TD3) ALGORITHM
The TD3 algorithm is current state-of-the-art off-policy algorithm and is variant of the deep deterministic policy gradient (DDPG) algorithm (Lillicrap et al. (2015). The TD3 algorithm tries resolve two problems in typical actor-critic algorithms: 1) overestimation bias and high vari- ance in the approximation of the Q-function. In order to reduce the bias, the TD3 considers two Q-functions and uses the minimum of the two Q-function values to compute the target value, while in order to reduce the variance in the gradient, the policy is updated less frequently than the Q- functions. Specifically, let Qa. . Qa and Te be two current Q-functions and the current deterministic policy, respectively, and let Qo;: Qe, and way be the target networks of QA Qa, and TΦ. respec- tively. The target networks are initialized by the same networks as the current networks. At time step the TD3 algorithm takes action a, with exploration noise ae (s, + where is zero-mean Gaussian noise with variance a², i.e., ~ [00 rd). Then, the environment returns re- ward rt and the state is switched to 8:411. The TD3 algorithm stores the experience (st, aes Yty 8t4-1 at the experience replay buffer D. After storing the experience, the Q-function parameters 0 and 02 are updated by gradient descent of the following loss functions:
L(0,) (( [(v - Qo, (s. a)) j == 1,2
(79)
where ((saan"')> denotes the sample expectation with an uniform random mini-batch size B drawn from the replay buffer D, and the target value y is given by
y y=rt
min Qo; (s', (s') + +c,, e~ clip(W
(80)
Here, for the computation of the target value, the minimum of the two target Q-functions is used to reduce the bias. The procedure of action taking and gradient descent for 01 and 8g are repeated for times (d = 2), and then the policy and target networks are updated. The policy parameter á is updated by gradient descent by minimizing the loss function for d:
L(d) EEND (Qa, (8, n (x)))
(81)
and the target network parameters 0g and are updated as
8' - (1- (1- + o' - + TT.
(82)
The networks are trained until the number of time steps reaches a predefined maximum.
25

---

Published as a conference paper at ICLR 2020
APPENDIX H. PSEUDOCODE OF THE P3S-TD3 ALGORITHM
Algorithm The Population-Guil Guideed Parallel Policy Search TD3 (P3S-TD3) Algorithm
Require: N: number of learners, Tinstial initial exploration time steps, T: maximum time steps, M : the best-policy update period, B: size of mini batch, d: update interval for policy and target networks.
1: Initialize 6 -.... Φ0 Φ², 0 0g. = 1,2, randomly.
2: Initialize 3 = 111,
3: while < T do
4: < 1+ (one time step)
5: for 1... N in parallel do
6: if' Tinitial then
7:
Take uniform random action at to environment copy E
8:
else
9:
Take an action a; = n² (s!) + (N to environment copy E end if
10:
Store experience (s), ai, r/, s(41) the shared common experience replay D
12:
end for
if't Tinitall then
14:
continue (i.e., go to the beginning of the while loop)
end if
16:
for i 1,2, 1......,. N in parallel do
17:
Sample mini-batch B = {(s,, On Tty St,
from D
18:
Update 0j, 1, 2, by gradient descent for minimizing L(03) 83 with
19:
t 0(mod d) then
20:
Update ø² by gradient descent for minimizing L(6') ) in 84 with B
Update the target networks: (0j) + (1 ())) T8 (o') end if end for
( TO¹
22:
24:
if = 0(mod M) then
Select the best learner 26: Adapt B
25:
27:
end if
28:
end while
In P3S-TD3, the i-th learner has its own parameters 0f. 0z and oi for its two Q-functions and policy. Furthermore, it has (61)', (0 and (ø+y which are the parameters of the corresponding target networks. For the distance measure between two policies, use the mean square difference, given by (iT(s),, (s)) (s) #(s)ll- For the i-th learner, in TD3, the parameters 8), j = 1, are updated every time step by minimizing
[(0)) = Es [wv 2o (s,
(83)
where y + 112 Qoyy (s', (s) + c), ~ clip(A a²), -c, c). The parameter o updated every d time steps by minimizing the following augmented loss function:
B | -Qp; (s, (g) (126)) || - ne (s)
L(d') - BBND
(84)
For the first Tiiitial time steps for initial exploration all policies over the initial exploration period. With these loss functions, the reference policy, and the initial exploration policy, all procedure is the same the general P3S procedure described in Section 3. The pseudocode of the P3S-TD3 algorithm is shown above.
random policy and do not update
26

---

Published as a conference paper at ICLR 2020
APPENDIX I. HYPER-PARAMETERS
TD3 The networks for two Q-functions and the policy have hidden layers. The first and second layers have sizes 400 and 300, respectively. The non-linearity function of the hidden layers is ReLU, and the activation functions of the last layers of the Q-functions and the policy are linear and hyper- bolic tangent, respectively. We used the Adam optimizer with learning rate 10-3, discount factor 0.99, target smoothing factor т 5 x 10 -
the period d = 2 for updating the policy. The experience replay buffer size is 10 and the mini-batch size is 100. The standard deviation for exploration noise and target noise ē are 0.1 and 0.2, respectively, and the noise clipping factor 0.5.
P3S-TD3 In addition to the hyper- hyper- parameters for TD3, we used N = 4 learners, the period M 250 of updating the best policy and B, the number recent episodes E, = 10 for determining the best learner b. The parameter dmii chosen among {0.02, 0.05} for each environment, and the chosen parameter was 0.02 1Walker2d-v1, Ant-v1, Delayed Hopper-vl, Delayed Walker2d-v1, Delayed HalfCheetah-vI) 1), and 0.05 (Hopper-vl. HalfCheetahh-l, Delayed Ant-vl). The parameter for the exploration range was 2 for all environments. The time steps for initial exploration Tinitial was set as 250 for Hopper-v1 and Walker2d-v1 and 2500 for HalfCheetah-vl and Ant-v1.
Re-TD3 The period M' was chosen among {2000, 5000, 10000) (MuJoCo environments) and {10000, 20000, 50000} (Delayed MuJoCo environments) by tuning for each environment. The cho- sen period M was 2000 (Ant-v1), 5000 (Hopper-vl, Walker2d-v1, 1, HalfCheetah-vll I), 10000 (De- layed HalfCheetah-vl, Delayed Ant-vl), and 20000 (Delayed Hopper-v1, Delayed Walker2d-v1).
27


Reviews:
[{"review_id": "ryxMDGOKur", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a distributed framework for off-policy RL consisting of multiple agents that are trained in parallel while also being regularized to be similar to the current best policy. Maintaining a population of agents can help mitigate issues due to convergences to local optima. The method is evaluated on standard continuous control tasks, and shows some performance improvements over methods that train just a single agent. Ablation experiments are also conducted to evaluate the effects of different design decisions.\n\nThe overall method is sensible and performance looks promising. But the technical innovation is fairly modest. As the authors pointed out, the proximal constraint has been used in previous distributed RL framework, and the main difference in this work is enforcing a trust region penalty against the best policy, as opposed to an average policy. The proof of guaranteed improvement largely follows the proof from other trust region methods, by replacing the old policy with the best policy from the previous iteration. The experiments did compare with a number of previous algorithms, but the comparisons are all to algorithms that train a single agent at a time, and no comparisons are made to other distributed RL algorithms. Including some comparisons to other distributed methods can help strengthen the claims in favour P3S. In particular, how does regularizing using the best policy compare to using an average policy like in Distral [The et al., 2017]? That being said, the performance improvements on most environments appear to be fairly modest, and it is not clear if the improvements are indeed significant. Including additional experiments on more challenging tasks could be helpful here. Overall, the contribution of this work is pretty incremental, and I think it is not quite at the standards for ICLR at this time. But with more thorough evaluation and polishing, I think this work can make for a strong submission.\n\nMore specific notes:\n\nThere are a fair bit of awkward phrasing and grammatical errors in the writing, which hurts the clarity of the exposition.\n\nOne of the advantages of a distributed framework is faster wall-clock time. The current learning curves only compare the sample count. It might also be informative to compare the wall-clock time of the different methods, as well as including comparisons to other distributed frameworks.\n\nSome experiments to show how performance scales with the number of learners can also be helpful. Since one of the possible factors that improve performance for P3S is having multiple agents, providing some insight on how performance varies with different numbers of agents can be valuable in this regard.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1lL6hgdKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose another method of doing population-based training of RL policies. During the training process, there are N workers running in N copies of the environment, each with different parameter settings for the policies and value networks. Each worker pushes data to a shared replay buffer of experience. The paper claims that a natural approach is to have a chief job periodically poll for the best worker, then replace the weights of each worker with the best one. Whenever this occurs, this reduces the diversity within the population.\n\nIn its place, the authors propose a soft-update in the chief. At every merging cycle, the chief queries which worker performs best. If that worker is worker B, it emits pi_B's parameters to each of the other workers. Instead of replacing the parameters exactly, worker i's loss is then augmented by beta * D(pi_i, pi_B), where D is some distance measure that is measured over states sampled from the replay buffer. The \"soft\" update encourages individual workers to match pi_B without directly replacing their parameters, which maintains diversity in the population. In this work, pi is always represented by a deterministic policy and D is the mean-squared-error in action space (this is argued as equivalent to the KL divergence between the two policies if the policies were represented by Gaussian with the same, constant standard deviation). The beta parameter is updated online using heuristics based on how D(pi_i, pi_B) compares to D(pi_i, old_pi_i). Using TD3 as a base algorithm, the population-based version performs better, and there are ablations for various parts of the population algorithm.\n\nI thought this paper was interesting, but thought it was strange that there were very few comparisons to other population / ensemble-based training methods. In particular they mention the copying problem as a downside of population-based training (PBT), but do not compare against PBT at all. Additionally, my understanding of PBT is that when they replace bad agents with the best agent, they only replace the worst performing agents (not all of them), and they additionally add some random perturbations to their hyperparameter settings. This goes counter to the claim that they collapse the population to a single point- by my reading the exploration step avoids this collapse.\n\nAn experiment I'd like to see is trying PBT, where different workers do in fact use different hyperparameters. My understanding is that in P3S-TD3 there is a single hyperparameter setting shared across all workers (plus some hyperparameters deciding the soft update).\n\nI'd also like to see ablations for the Resetting variant (Re-TD3), where only the bottom half or 2/3rds of the workers are reset. This would give empirical evidence for the \"population collapse\" intuition - we should expect to see some improvements if we avoid totally collapsing the population, while still copying enough to partially exploit the current best setting.\n\nMany inequalities in the paper are argued by compare the expectation of negative Q of one policy to the negative Q of another - I believe the derivations would be much easier to follow if the authors simply multiplied all sides by -1 and adjusted inequalities accordingly. It is much easier to think about Q-value-1 > Q-value-2 rather than -Q-value-1 < -Q-value-2 when trying to interpret what the equation is saying.\n\nFor related work, papers on evolutionary strategies and the various self-play-in-a-population papers seem relevant, since these often take the form of having each worker i do a different perturbation that is later merged by a chief.\n\nIn Figure 4 it feels weird that results are the regular Mujoco envs for 2 problems and the delayed envs for the other 2 problems. When looking at the appendix, it's rather clearly cherry picked to show the best results in favor of PS3-TD3. I would prefer the Delayed MuJoCo experiments be in a separate figure, or to include the TD3/SAC/ACKTR/PPO/etc. results for the delayed envs as well (these don't appear to be in the appendix)\n\nOn the theoretical results: the 1st assumption seems very strong. The first assumption argues that pi_B is always 1-step better than pi_old for every state. That assumption already takes you very far towards arguing \"updating pi_old to pi_B is good\". The 2nd assumption is more reasonable but I'm confused how rho and d play into the theoretical results. Do they play any role in how much the policy is expected to improve, or do the constants just need to exist?\n\nThe last comment on the theory side is that I still don't understand the intuition for why we want to learn beta such that \n\nKL(pi_new || pi_b) = max {rho * KL_max(pi_new || pi_old), d}\n\nIn the practical algorithm, beta is updated online to increase / decrease the importance of the \"match pi_B\" term if the ratio between the two strays too far from 1 (with the threshold set to [1/1.5, 1.5] in a manner similar to PPO's approach). But why should it be important for the two values to be close to one another? Let me write out the derivation continuing from Eqn (57) in the appendix.\n\nWith a substitution that doesn't use (c) to drop the beta * (KL - KL) term, we get\n\nE_{pi_b}[-Q_new] >= E_{pi_new}[-Q_new] + beta * (KL - KL)\n-->\nE_{pi_new}[Q_new] >= E_{pi_b}[Q_new] + beta * (KL - KL)\n\nThen, in Theorem 1, we recursively apply this inequality, accumulating a number of beta * (KL - KL) terms. In the end we get\n\nQ_new >= (discounted sum rewards from pi_b) + (discounted sum of beta * (KL - KL) with expectation over states from pi_b)\n= Q_pi_b + (sum of beta *(KL - KL) terms)\n\nBy my reading, shouldn't this mean we want KL(pi_new || pi_b) - max {rho * KL_max(pi_new || pi_old), d} to be as large as possible, rather than 0? The more positive this term is, the more improvement we get between Q_new and Q_pi_b.\n\n--------------------------\n\nOverall, this feels like a good paper, but I'm not too familiar with prior empirical results for population-based RL methods. The ablations suggested that pretty much any reasonable population-based method outperformed using a single worker, and because of this it seems especially important to have ablations to other population-based prior work, rather than just variants of its own method. \n\nI would be okay with this paper as-is despite some of its flaws, but think it could be better pending rebuttal.\n\nEdit: I have read the author reply and other reviews. I do not plan to change my rating but do think the paper is improved by the added baselines and better explanation of what the beta adaptation rule is doing. I would ask the authors to make sure this description is as clear as possible, as the argued improvement gap seems central to the work.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1e8Q9IiuH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The  paper proposes a new approach to multi-actor RL, which ensure diversity and performance of the population of actors, by distilling the policy of the best performing agent in a soft way and maintaining some distance between the agents. The authors show improved performance over several state-of-the-art mono-actor algorithms and over several other multi-actor RL algorithms.\n\nI'm in favor of accepting the paper despite a few serious weaknesses described below.\n\nA good point of the paper is the related work section which provides a good and concise survey of various multi-actor RL approaches: distributed RL, population-based training and Guided Policy Search (the last part about exploiting best information looks less relevant).\n\nHere a set of random  remarks:\n\nAbout  population-based training, there is quite a lot of repetition between the introduction and the related work  section.\n\nThe P3S approach described in the beginning of Section 3 and the end of Section 3.2 seems to rely on some arbitrary choices and a few hyperparameters whose impact is not much studied.\n\nIn the theoretical study (Section 3.1), a KL divergence is used as a distance between policies, which implies stochastic policies. But P3S is used on top of TD3, where policies are deterministic. This should definitely be discussed.\n\nThe role of \\beta in the theory is not put forward in a way to make the point of Section 3.2 clear. The theory should be clarified in this respect.\n\nThe top of p5 is made of a unique sentence over 7 lines which is completely obfuscating. This part must be rewritten and much clarified.\n\nI would be glad to see the performance on Swimmer, as this benchmark is known to suffer from  a deceiptive gradient.\n\nThe fact that a negative cost of action in Ant-v1 can result in no action in this environment is reminiscent of the same effect shown in the simpler Continuous Mountain Car environment in the Gep-PG paper (Colas, Oudeyer and Sigaud, ICML 2018). Actually, moving to simpler benchmarks would make it possible to provide more detailed empirical studies of the inner mechanisms of the P3S approach.\n\n p6: \"The policies used for evaluation are stochastic for PPO and ACKTR, and deterministic for the others.\" Do you mean you used a deterministic policy for SAC? This would be unusual, as SAC with a deterministic policy is very close to TD3.\n\n p6: \"In Fig.   3,  it is first observed that the performance of TD3 here is similar to that in the originalTD3 paper (Fujimoto et al. (2018)), and the performance of other baseline algorithms is also similar to that in the original papers (Schulman et al. (2017); Haarnoja et al. (2018))\". This sentence can be much compressed: \"In Fig.   3,  it is observed that all other baseline algorithms is also similar to that in the original papers (Fujimoto et al. (2018),Schulman et al. (2017); Haarnoja et al. (2018))\". \n\nWhy did you use the v1 versions of the benchmarks, and not the v2?\n\nI did not check the proofs in appendix.\n\ntypos:\n\np2: hyperparamters\np7: is the way how the best =>  is the way the best...\np7: other all parallel => all other parallel\np22: environmet", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The  paper proposes a new approach to multi-actor RL, which ensure diversity and performance of the population of actors, by distilling the policy of the best performing agent in a soft way and maintaining some distance between the agents. The authors show improved performance over several state-of-the-art mono-actor algorithms and over several other multi-actor RL algorithms.  Initially, reviewers were concerned with magnitude of the contribution/novelty, as well as some technical issues (e.g. the beta update), and relative lack of baseline comparisons.  However, after discussion the reviewers largely agree that their main concerns have been addressed.  Therefore, I recommend this paper for acceptance.
Decision: Accept (Poster)