Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose another method of doing population-based training of RL policies. During the training process, there are N workers running in N copies of the environment, each with different parameter settings for the policies and value networks. Each worker pushes data to a shared replay buffer of experience. The paper claims that a natural approach is to have a chief job periodically poll for the best worker, then replace the weights of each worker with the best one. Whenever this occurs, this reduces the diversity within the population.

In its place, the authors propose a soft-update in the chief. At every merging cycle, the chief queries which worker performs best. If that worker is worker B, it emits pi_B's parameters to each of the other workers. Instead of replacing the parameters exactly, worker i's loss is then augmented by beta * D(pi_i, pi_B), where D is some distance measure that is measured over states sampled from the replay buffer. The "soft" update encourages individual workers to match pi_B without directly replacing their parameters, which maintains diversity in the population. In this work, pi is always represented by a deterministic policy and D is the mean-squared-error in action space (this is argued as equivalent to the KL divergence between the two policies if the policies were represented by Gaussian with the same, constant standard deviation). The beta parameter is updated online using heuristics based on how D(pi_i, pi_B) compares to D(pi_i, old_pi_i). Using TD3 as a base algorithm, the population-based version performs better, and there are ablations for various parts of the population algorithm.

I thought this paper was interesting, but thought it was strange that there were very few comparisons to other population / ensemble-based training methods. In particular they mention the copying problem as a downside of population-based training (PBT), but do not compare against PBT at all. Additionally, my understanding of PBT is that when they replace bad agents with the best agent, they only replace the worst performing agents (not all of them), and they additionally add some random perturbations to their hyperparameter settings. This goes counter to the claim that they collapse the population to a single point- by my reading the exploration step avoids this collapse.

An experiment I'd like to see is trying PBT, where different workers do in fact use different hyperparameters. My understanding is that in P3S-TD3 there is a single hyperparameter setting shared across all workers (plus some hyperparameters deciding the soft update).

I'd also like to see ablations for the Resetting variant (Re-TD3), where only the bottom half or 2/3rds of the workers are reset. This would give empirical evidence for the "population collapse" intuition - we should expect to see some improvements if we avoid totally collapsing the population, while still copying enough to partially exploit the current best setting.

Many inequalities in the paper are argued by compare the expectation of negative Q of one policy to the negative Q of another - I believe the derivations would be much easier to follow if the authors simply multiplied all sides by -1 and adjusted inequalities accordingly. It is much easier to think about Q-value-1 > Q-value-2 rather than -Q-value-1 < -Q-value-2 when trying to interpret what the equation is saying.

For related work, papers on evolutionary strategies and the various self-play-in-a-population papers seem relevant, since these often take the form of having each worker i do a different perturbation that is later merged by a chief.

In Figure 4 it feels weird that results are the regular Mujoco envs for 2 problems and the delayed envs for the other 2 problems. When looking at the appendix, it's rather clearly cherry picked to show the best results in favor of PS3-TD3. I would prefer the Delayed MuJoCo experiments be in a separate figure, or to include the TD3/SAC/ACKTR/PPO/etc. results for the delayed envs as well (these don't appear to be in the appendix)

On the theoretical results: the 1st assumption seems very strong. The first assumption argues that pi_B is always 1-step better than pi_old for every state. That assumption already takes you very far towards arguing "updating pi_old to pi_B is good". The 2nd assumption is more reasonable but I'm confused how rho and d play into the theoretical results. Do they play any role in how much the policy is expected to improve, or do the constants just need to exist?

The last comment on the theory side is that I still don't understand the intuition for why we want to learn beta such that 

KL(pi_new || pi_b) = max {rho * KL_max(pi_new || pi_old), d}

In the practical algorithm, beta is updated online to increase / decrease the importance of the "match pi_B" term if the ratio between the two strays too far from 1 (with the threshold set to [1/1.5, 1.5] in a manner similar to PPO's approach). But why should it be important for the two values to be close to one another? Let me write out the derivation continuing from Eqn (57) in the appendix.

With a substitution that doesn't use (c) to drop the beta * (KL - KL) term, we get

E_{pi_b}[-Q_new] >= E_{pi_new}[-Q_new] + beta * (KL - KL)
-->
E_{pi_new}[Q_new] >= E_{pi_b}[Q_new] + beta * (KL - KL)

Then, in Theorem 1, we recursively apply this inequality, accumulating a number of beta * (KL - KL) terms. In the end we get

Q_new >= (discounted sum rewards from pi_b) + (discounted sum of beta * (KL - KL) with expectation over states from pi_b)
= Q_pi_b + (sum of beta *(KL - KL) terms)

By my reading, shouldn't this mean we want KL(pi_new || pi_b) - max {rho * KL_max(pi_new || pi_old), d} to be as large as possible, rather than 0? The more positive this term is, the more improvement we get between Q_new and Q_pi_b.

--------------------------

Overall, this feels like a good paper, but I'm not too familiar with prior empirical results for population-based RL methods. The ablations suggested that pretty much any reasonable population-based method outperformed using a single worker, and because of this it seems especially important to have ablations to other population-based prior work, rather than just variants of its own method. 

I would be okay with this paper as-is despite some of its flaws, but think it could be better pending rebuttal.

Edit: I have read the author reply and other reviews. I do not plan to change my rating but do think the paper is improved by the added baselines and better explanation of what the beta adaptation rule is doing. I would ask the authors to make sure this description is as clear as possible, as the argued improvement gap seems central to the work.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The  paper proposes a new approach to multi-actor RL, which ensure diversity and performance of the population of actors, by distilling the policy of the best performing agent in a soft way and maintaining some distance between the agents. The authors show improved performance over several state-of-the-art mono-actor algorithms and over several other multi-actor RL algorithms.

I'm in favor of accepting the paper despite a few serious weaknesses described below.

A good point of the paper is the related work section which provides a good and concise survey of various multi-actor RL approaches: distributed RL, population-based training and Guided Policy Search (the last part about exploiting best information looks less relevant).

Here a set of random  remarks:

About  population-based training, there is quite a lot of repetition between the introduction and the related work  section.

The P3S approach described in the beginning of Section 3 and the end of Section 3.2 seems to rely on some arbitrary choices and a few hyperparameters whose impact is not much studied.

In the theoretical study (Section 3.1), a KL divergence is used as a distance between policies, which implies stochastic policies. But P3S is used on top of TD3, where policies are deterministic. This should definitely be discussed.

The role of \beta in the theory is not put forward in a way to make the point of Section 3.2 clear. The theory should be clarified in this respect.

The top of p5 is made of a unique sentence over 7 lines which is completely obfuscating. This part must be rewritten and much clarified.

I would be glad to see the performance on Swimmer, as this benchmark is known to suffer from  a deceiptive gradient.

The fact that a negative cost of action in Ant-v1 can result in no action in this environment is reminiscent of the same effect shown in the simpler Continuous Mountain Car environment in the Gep-PG paper (Colas, Oudeyer and Sigaud, ICML 2018). Actually, moving to simpler benchmarks would make it possible to provide more detailed empirical studies of the inner mechanisms of the P3S approach.

 p6: "The policies used for evaluation are stochastic for PPO and ACKTR, and deterministic for the others." Do you mean you used a deterministic policy for SAC? This would be unusual, as SAC with a deterministic policy is very close to TD3.

 p6: "In Fig.   3,  it is first observed that the performance of TD3 here is similar to that in the originalTD3 paper (Fujimoto et al. (2018)), and the performance of other baseline algorithms is also similar to that in the original papers (Schulman et al. (2017); Haarnoja et al. (2018))". This sentence can be much compressed: "In Fig.   3,  it is observed that all other baseline algorithms is also similar to that in the original papers (Fujimoto et al. (2018),Schulman et al. (2017); Haarnoja et al. (2018))". 

Why did you use the v1 versions of the benchmarks, and not the v2?

I did not check the proofs in appendix.

typos:

p2: hyperparamters
p7: is the way how the best =>  is the way the best...
p7: other all parallel => all other parallel
p22: environmet
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work presents a distributed framework for off-policy RL consisting of multiple agents that are trained in parallel while also being regularized to be similar to the current best policy. Maintaining a population of agents can help mitigate issues due to convergences to local optima. The method is evaluated on standard continuous control tasks, and shows some performance improvements over methods that train just a single agent. Ablation experiments are also conducted to evaluate the effects of different design decisions.

The overall method is sensible and performance looks promising. But the technical innovation is fairly modest. As the authors pointed out, the proximal constraint has been used in previous distributed RL framework, and the main difference in this work is enforcing a trust region penalty against the best policy, as opposed to an average policy. The proof of guaranteed improvement largely follows the proof from other trust region methods, by replacing the old policy with the best policy from the previous iteration. The experiments did compare with a number of previous algorithms, but the comparisons are all to algorithms that train a single agent at a time, and no comparisons are made to other distributed RL algorithms. Including some comparisons to other distributed methods can help strengthen the claims in favour P3S. In particular, how does regularizing using the best policy compare to using an average policy like in Distral [The et al., 2017]? That being said, the performance improvements on most environments appear to be fairly modest, and it is not clear if the improvements are indeed significant. Including additional experiments on more challenging tasks could be helpful here. Overall, the contribution of this work is pretty incremental, and I think it is not quite at the standards for ICLR at this time. But with more thorough evaluation and polishing, I think this work can make for a strong submission.

More specific notes:

There are a fair bit of awkward phrasing and grammatical errors in the writing, which hurts the clarity of the exposition.

One of the advantages of a distributed framework is faster wall-clock time. The current learning curves only compare the sample count. It might also be informative to compare the wall-clock time of the different methods, as well as including comparisons to other distributed frameworks.

Some experiments to show how performance scales with the number of learners can also be helpful. Since one of the possible factors that improve performance for P3S is having multiple agents, providing some insight on how performance varies with different numbers of agents can be valuable in this regard.

review_title: Official Blind Review #2

Meta-review: The  paper proposes a new approach to multi-actor RL, which ensure diversity and performance of the population of actors, by distilling the policy of the best performing agent in a soft way and maintaining some distance between the agents. The authors show improved performance over several state-of-the-art mono-actor algorithms and over several other multi-actor RL algorithms.  Initially, reviewers were concerned with magnitude of the contribution/novelty, as well as some technical issues (e.g. the beta update), and relative lack of baseline comparisons.  However, after discussion the reviewers largely agree that their main concerns have been addressed.  Therefore, I recommend this paper for acceptance.
Decision: Accept (Poster)