Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes a genetic algorithm inspired policy optimization method, which mimics the mutation and the crossover operators over policy networks.

The title and the motivation about the genetic algorithm are missing leading and improper. The genetic algorithm is a black-box optimization method, however, the proposed method has nothing to do with black-box optimization. 

The mutation is a method to sample individual independence of the objective function, which is very different with the gradient step. Mimicking the mutation by a gradient step is very unreasonable. 

The crossover operator is the policy mixing method employed in game context (e.g., Deep Reinforcement Learning from Self-Play in Imperfect-Information Games, https://arxiv.org/abs/1603.01121 ). It is straightforward if two policies are to be mixed. Although the mixing method is more reasonable than the genetic crossover operator, it is strange to compare with that operator in a method far away from the genetic algorithm.

It is highly suggested that the method is called as population-based method as a set of networks is maintained, instead of as "genetic" method.

Another drawback, perhaps resulted from the "genetic algorithm" motivation is that the proposed method has not been well explained. The only explanation is that this method mimics the genetic algorithm. However, this explanation reveals nothing about why the method could work well -- a random exploration could also waste a lot of samples with a very high probability.

The baseline methods result in rewards much lower than those in previous experimental papers. It is problemistic that if the baselines have bad parameters.
1. Benchmarking Deep Reinforcement Learning for Continuous Control
2. Deep Reinforcement Learning that Matters
review_title: The title and motivation is somehow missing leading. The proposed method has no sound explanation. The experiment does not support the method well.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This is a highly interesting paper that proposes a set of methods that combine ideas from imitation learning, evolutionary computation and reinforcement learning in a novel way. It combines the following ingredients:
a) a population-based setup for RL
b) a pair-selection and crossover operator
c) a policy-gradient based “mutation” operator
d) filtering data by high-reward trajectories
e) two-stage policy distillation

In its current shape it has a couple of major flaws (but those can be fixed during the revision/rebuttal period):

(1) Related work. It is presented in a somewhat ahistoric fashion. In fact, ideas for evolutionary methods applied to RL tasks have been widely studied, and there is an entire research field called “neuroevolution” that specifically looks into which mutation and crossover operators work well for neural networks. I’m listing a small selection of relevant papers below, but I’d encourage the authors to read a bit more broadly, and relate their work to the myriad of related older methods. Ideally, a more reasonable form of parameter-crossover (see references) could be compared to -- the naive one is too much of a straw man in my opinion. To clarify: I think the proposed method is genuinely novel, but a bit of context would help the reader understand which aspects are and which aspects aren’t.

(2) Ablations. The proposed method has multiple ingredients, and some of these could be beneficial in isolation: for example a population of size 1 with an interleaved distillation phase where only the high-reward trajectories are preserved could be a good algorithm on its own. Or conversely, GPO without high-reward filtering during crossover. Or a simpler genetic algorithm that just preserves the kills off the worst members of the population, and replaces them by (mutated) clones of better ones, etc. 

(3) Reproducibility. There are a lot of details missing; the setup is quite complex, but only partially described. Examples of missing details are: how are the high-reward trajectories filtered? What is the total computation time of the different variants and baselines? The x-axis on plots, does it include the data required for crossover/Dagger? What are do the shaded regions on plots indicate? The loss on \pi_S should be made explicit. An open-source release would be ideal.

Minor points:
- naively, the selection algorithm might not scale well with the population size (exhaustively comparing all pairs), maybe discuss that?
- the filtering of high-reward trajectories is what estimation of distribution algorithms [2] do as well, and they have a known failure mode of premature convergence because diversity/variance shrinks too fast. Did you investigate this?
- for Figure 2a it would be clearer to normalize such that 1 is the best and 0 is the random policy, instead of 0 being score 0.
- the language at the end of section 3 is very vague and noncommittal -- maybe just state what you did, and separately give future work suggestions?
- there are multiple distinct metrics that could be used on the x-axis of plots, namely: wallclock time, sample complexity, number of updates. I suspect that the results will look different when plotted in different ways, and would enjoy some extra plots in the appendix. For example the ordering in Figure 6 would be inverted if plotting as a function of sample complexity?
- the A2C results are much worse, presumably because batchsizes are different? So I’m not sure how to interpret them: should they have been run for longer? Maybe they could be relegated to the appendix?

References:
[1] Gomez, F. J., & Miikkulainen, R. (1999). Solving non-Markovian control tasks with neuroevolution.
[2] Larranaga, P. (2002). A review on estimation of distribution algorithms.
[3] Stanley, K. O., & Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. 
[4] Igel, C. (2003). Neuroevolution for reinforcement learning using evolution strategies.
[5] Hausknecht, M., Lehman, J., Miikkulainen, R., & Stone, P. (2014). A neuroevolution approach to general atari game playing.
[6] Gomez, F., Schmidhuber, J., & Miikkulainen, R. (2006). Efficient nonlinear control through neuroevolution.


Pros:
- results
- novelty of idea
- crossover visualization, analysis
- scalability

Cons:
- missing background
- missing ablations
- missing details

[after rebuttal: revised the score from 7 to 8]
review_title: novel but incomplete

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present an algorithm for training ensembles of policy networks that regularly mixes different policies in the ensemble together by distilling a mixture of two policies into a single policy network, adding it to the ensemble and selecting the strongest networks to remain (under certain definitions of a "strong" network). The experiments compare favorably against PPO and A2C baselines on a variety of MuJoCo tasks, although I would appreciate a wall-time comparison as well, as training the "crossover" network is presumably time-consuming.

It seems that for much of the paper, the authors could dispense with the genetic terminology altogether - and I mean that as a compliment. There are few if any valuable ideas in the field of evolutionary computing and I am glad to see the authors use sensible gradient-based learning for GPO, even if it makes it depart from what many in the field would consider "evolutionary" computing. Another point on terminology that is important to emphasize - the method for training the crossover network by direct supervised learning from expert trajectories is technically not imitation learning but behavioral cloning. I would perhaps even call this a distillation network rather than a crossover network. In many robotics tasks behavioral cloning is known for overfitting to expert trajectories, but that may not be a problem in this setting as "expert" trajectories can be generated in unlimited quantities.
review_title: Could equally well be titled "crossover by distillation"

Meta-review: At least two of the reviewers found the proposed approach novel and interesting and worthy of publication at ICLR. The reviewers raised concerns regarding the paper's terminology, which may lead to some misunderstanding. I agree that upon a quick skim, a reader may think that the paper performs the crossover operation outlined at the bottom right of Figure 1. Please consider improving the figure and the caption to prevent such a misunderstanding. You can even slightly change the title to reflect the policy distillation operation rather than naive crossover. Finally, including some more complex baselines benefits the paper. I am curious whether performing policy gradient on an ensemble of 8 policies + periodic removal of the bottom half of the policies will provide similar gains.
Decision: Accept (Poster)