PGPS : Coupling Policy Gradient with Population-based Search

Abstract: Gradient-based policy search algorithms (such as PPO, SAC or TD3) in deep reinforcement learning (DRL) have shown successful results on a range of challenging control tasks. However, they often suffer from flat or deceptive gradient problems. As an alternative to policy gradient methods, population-based evolutionary approaches have been applied to DRL. While population-based search algorithms show more robust learning in a broader range of tasks, they are usually inefficient in the use of samples. Recently, reported are a few attempts (such as CEMRL) to combine gradient with a population in searching optimal policy. This kind of hybrid algorithm takes advantage of both camps. In this paper, we propose yet another hybrid algorithm, which more tightly couples policy gradient with the population-based search. More specifically, we use the Cross-Entropy Method (CEM) for population-based search and Twin Delayed Deep Deterministic Policy Gradient (TD3) for policy gradient. In the proposed algorithm called Coupling Policy Gradient with Population-based Search (PGPS), a single TD3 agent, which learns by a gradient from all experiences generated by population, leads a population by providing its critic function Q as a surrogate to select better performing next-generation population from candidates. On the other hand, if the TD3 agent falls behind the CEM population, then the TD3 agent is updated toward the elite member of the CEM population using loss function augmented with the distance between the TD3 and the CEM elite. Experiments in a MuJoCo environment show that PGPS is robust to deceptive gradient and also outperforms the state-of-the-art algorithms.
      

Reviews and author responses:

This a nicely written paper.  The authors propose an algorithm that combines population-based search with policy gradient approach to obtain optimal policies in reinforcement learning. The proposed combination leads to a yet another algorithm aiming at exploitation (TD3) and exploration (CEM). The idea of using such combinations is not new (a point also acknowledged by the authors). The results are promising, though these results are obtained with some additional computational cost. I believe this manuscript should be considered as an implementation work, where the novelty of the idea is not clear. 

Summary: 

Recently, several researchers have been trying to combine the goodnesses of direct policy search approaches (mostly based on evolutionary computation approaches)  and those of policy gradient approaches in control tasks. This paper proposes a novel combination of an evolutionary direct policy search and an actor-critic approach. The authors combines a cross-entropy method, which directly samples parameters of actor network (policy) from a Gaussian distribution that is trained during the search process, and the twin delayed deep deterministic policy gradient (TD3), which is an off-policy actor-critic approach. 

In existing combination of an evolutionary direct policy search and an actor critic approach, the evolutionary search part contributes to fill the replay buffer with diverse experiences, whereas the actor critic approach leads the population of the evolutionary search by replacing subset of the population with the policy of the actor critic agent, which is often expected to be superior to the population of the evolutionary search. Differently from these existing researches, the proposed approach also make an elite population to guide the actor critic agent when it is stacked at an area where the policy gradient almost vanishes. This idea comes from the recently proposed P3S-TD3, another population based (multi-policy) TD3 approaches.

Comparison has been conduced on 5 MuJoCo environments. In Swimmer-v2 environment, the proposed approach reached the performance between the pure cross entropy method (best) and other combinations of evolutionary approaches and policy gradient approaches. In the other four environments, the proposed approaches reached the competitive or superior average performance. 

Criticism:

Compared to the results presented in P3S-TD3 paper (ICLR 2020), which used a slightly different environment and therefore the direct comparisons may not be valid, differences observed in this paper between the proposed approach and the pure TD3 is rather similar to the differences between P3S-TD3 and TD3. Since one of the main contributions of this paper, if I understand correctly, is the introduction of the component borrowed from P3S-TD3, P3S-TD3 should be included in the baseline approaches in the comparison. Is there any advantage of the proposed approach over P3S-TD3 on HalfCheetah, Hopper, Walker2d and Ant? 

Swimmer environment is the only tested environment where CEM works significantly better than TD3. On this environment the proposed approach was shown to be as effective as CEM. Does it generalize to other environment where CEM works significantly better than TD3? The current evaluation lacks the evaluation of the generality of this approach. 

In the ablation study (Table 1) the effect of Equation (4) was evaluated, and shows a non-negligible impact. Equation (4), which controls the number of interaction steps performed in each MC evaluation in CEM, can be incorporated to other approaches such as CEMRL, and if the same impact appears in the existing approaches, it is ambiguous whether the other components really contribute to improve the performance over existing approaches. 

**Summary:** The authors propose PGPS as a minor variation on [CEM-RL](https://arxiv.org/abs/1810.01222). 

**Quality:** The quality of the paper is overall very low: misleading and unsustained claims, poor and incorrect literature review, cherry-picking results.

**Clarity:** While the exposition overall follows a standard template, the writing is in dire need of intensive copy efforts prior to being considered for publication.

**Originality:** The authors present a minor augmentation of CEM-RL while claiming some of its results as their own.

**Pros:**
- The experimental part seems to have received a lot of work
- The setup and hyperparameter description is very complete, coupled with open sourced code, potentially allowing full experiment reproduction

**Cons:**
- The first sentence of the Conclusions claims the pairing of CEM and TD3 integrating policy gradient and population-based search as a novelty and contribution, while it was introduced in the (cited!) CEM-RL.
- There is no clear, explicit definition of the paper's contributions, which given the circumstances is unacceptable.
- Some results (particularly regarding the Q-critic filtering) are sketchy and not statistically meaningful.
- The Related Work study in Section 2 is riddled with incorrectness in the evolutionary part. For example: stating EA stands for Evolutionary Approach rather than Algorithms; referring to CMA-ES by citing a 2016 tutorial instead of the [2001 paper](https://scholar.google.ch/scholar?q=completely+derandomized); equating EA to direct policy search; stating that previous-generation data cannot be reused; and many more. This overall shows a less than perfunctory understanding of the field. The parallels with the CEM-RL paper are once again obvious and misquoted.
- The paper writing requires a thorough overhaul.

**Comments:**
The contributions are not made clear, and a quick read of the original CEM-RL paper shows an exact correspondence but for 1. the double-feedback loop and 2. the Q-critic filtering. These two contributions however are very minimal, and most importantly they are not tested in isolation: end of Section 5, 1. "The effect of EA guidance" does not compare with CEM-RL (or other hybrid methods), and 2. "The effect of Q-critic filtering" drops the variance information in favor of what looks like single runs with statistically indistinguishable results.

**Final remarks:**
This work is a barely-augmented, error-riddled rewriting of a 2019 paper, and its state is nowhere near publishable. I strongly advise the authors to decide if their contributions are sufficient for publication, and if so to write a paper dedicated to proving their point to the scientific community. 
Finally, the un-anonymized GitHub link on Section 5.1 breaks the code of conduct; the fact that the author (NamKim88) is less known does not excuse the breach of anonymity.

The paper proposes a new method combining evolutionary methods and RL. In particular, the authors combine CEM and TD3 in PGPS. PGPS maintains a population of policies, which interact with the environment to collect data filling the replay buffer. The data in replay buffer is then used to train TD3. PGPS enables information flow in both directions: when the TD3 policy performs poorly, the elite policy from the population is used to guide TD3 by an imitation learning loss; The TD3 critic helps select top policies in the population and the TD3 actor is also included in the population. The experiments on simple Mujoco domains demonstrate the utility of PGPS and the ablation study analyzes the utility of each part of PGPS.

My major concern is that the experiments are not convincing enough. 
(1) I think TRES (Liu et al., 2019), Salimans et al. (2017), and PBT (Jaderberg et al., 2017) are important baselines that the authors need to compare with.
(2) The reported results are based on only 5 random seeds which I don't think are enough. 
(3) The experiments have only 5 domains, I think at least Humanoid should be included. Moreover, as PBT does demonstrate advantages in video games (e.g., Atari, DMLab), I'd like to see if PGPS can scale up to video games as well.

Meta-review: This paper proposes a hybrid algorithm that combines RL and population-based search. The work is interesting and well-written. But, the contribution of the work is very limited, in comparison with the state-of-the-art. 

Decision: withdrawn-rejected-submissions