Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes an integration of neuroevolution and gradient-based learning for reinforcement learning applications. The evolutionary algorithm focuses on sparse reward and multiagent/team optimization, while the gradient-based learning is used to inject selectively improved genotypes in the population.
This work addresses a very hot topic, i.e. the integration of NE and DRL, and the proposed method offers the positive side of both without introducing major downsides. The presented results come from a relatively simple but useful multiagent benchmark which has broad adoption. The paper is well written, presents several contributions that can be extended and ported to other work, and the results are statistically significant.

There is one notable piece missing which forces me to bridle my enthusiasm: a discussion of the genotype and of its interpretation into the network phenotype. The form taken by the actual agent is not explicitly stated; following the adoption of TD3 I would expect a policy and two critics, for a grand total of three neural networks, but this remains unverified. And if each agent is composed of three neural networks, and each individual represents a team, does this mean that each genotype is a concatenation of three (flattened) weight matrices per each agent in the team? What is the actual genotype size? It sounds huge, I would expect to be at least several hundred weights; but then this would clash with the proposed minuscule population size of 10 (recent deep neuroevolution work from Uber uses populations THREE orders of magnitude larger). Has the population size been proportionated to the genotype dimensionality? Would it be possible to reference the widely adopted defaults of industry standard CMA-ES? Speaking of algorithms, where is the chosen EA implementation discussed? The overview seems to describe a textbook genetic algorithm, but that has been overtaken as state-of-the-art since decades, constituting a poor match for TD3.

Omitting such a chapter severely limits not only the reproducibility of the work but its full understanding. For example, does the EA have sufficient population size to contribute significantly to the process, or is it just performing as a fancy version of Random Weight Guessing? Could you actually quickly run RWG with direct policy search (rather than random action selection) to establish the effective complexity of the task? My final rating after rebuttal will vary wildly depending on the ability to cover such an important piece of information. 

A few minor points, because I think that the paper appearance deserves to match the quality of the content:
- The images are consistently too small and hard to read. I understand the need to fit in the page limit by the deadline, but for the camera ready version it will be necessary to trim the text and rescale all images.
- The text is well written but often slowing down the pace for no added value, such as by dedicating a whole page to discussing a series of previously published environments.
- The hyperparameters of the evolutionary algorithm look completely unoptimized. I would expect a definite improvement in performance with minimal tuning.
- The "standard neuroevolutionary algorithm" from 2006 presented as baseline has not been state-of-the-art for over a decade. I would understand its usage as a baseline if that is indeed the underlying evolutionary setup, but otherwise I see no use for such a baseline.

-----------------------------------------------------------------------------------------------
# Update following the rebuttal phase
-----------------------------------------------------------------------------------------------

Thank you for your work and for the extended experimentation. I am confident the quality of the work is overall increased.

The core research question behind my original doubt however remains unaddressed: does the EC part of the algorithm sensibly support the gradient-descent part, or is the algorithm basically behaving as a (noisy) multi-agent TD3?
Such a contribution by itself would be undoubtedly important. Submitting it as a principled unification of EC and DL however would be more than a simple misnomer: it could mislead further research in what is an extremely promising area.

The scientific approach to clarify this point would be to design an experiment showcasing the performance of MARL using a range of sensible population sizes. To understand what "sensible" means in this context, I refer to a classic:
http://www.cmap.polytechnique.fr/~nikolaus.hansen/cec2005ipopcmaes.pdf
A lower bound for the population size with simple / unimodal fitness functions would be $4+floor(3*log(10'000)) = 31$. With such a complex, multimodal fitness though, no contribution from the EA can be expected (based on common practice in the EC field) without at least doubling or tripling that number. The upper bound does not need to be as high as with the recent Uber AI work (10k), but certainly showing the performance with a population of a few hundreds would be the minimum necessary to support your claim. A population size of 10 represents a proper lower bound for a genotype of up to 10 parameters; it is by no means within a reasonable range with your dimensionality of 10'000 parameters, and no researcher with experience in EC would expect anything but noise from such results -- with non-decreasing performance uniquely due to elitism.
The new runs in Appendice C only vary the population size for the ES algorithm, proposed as a baseline. No performance of MARL using a sensible population size is presented.

The fundamental claim is thereby unsustainable by current results. The idea is extremely intriguing and very promising, easily leading to supportive enthusiasm; it is my personal belief however that accepting this work in such a premature stage (and with an incorrect claim) could stunt further research in this direction.

[By the way, the reference Python CMA-ES implementation runs with tens of thousands of parameters and a population size of 60 in a few seconds per generation on a recent laptop: the claim of performance limitations as an excuse for not investigating a core claim suggests that more work would be better invested prior to acceptance.]

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper proposes to use a two-level optimization process to solve the challenge of optimizing the team reward and the agent's reward simultaneously, which are often not aligned. It applies the evolutionary algorithm to optimize the sparse team reward, while using RL (TD3) to optimize the agent's dense reward. In this way, there is no need to combine these two rewards into a scalar that often requires extensive manual tuning.

I vote for accepting this paper, because it tackles a practical problem in multi-agent learning. The presentation is clear, the algorithm is simple and sensible, the evaluation is thorough, and the results are better than the state-of-the-art.

The only question that I have is the sharing of replay-buffer for the same agent in different teams (Figure 2). For example, since the second agents in team1 and in team2 might learn to serve different roles in the task and may have completely different policies. I am not sure what is the purpose of this sharing. Considering the two extreme cases of replay buffer sharing, we could share all the data in a single replay buffer, or we could keep a separate replay buffer for each individual agent in each team, the paper chose the compromise between these two extremes. I wonder whether there is any theoretical or practical reasons to make this design choice. Is it important? I hope that the paper could have a deeper discussion if it is an important design decision.

----------------------Update after rebuttal----------------------

Thanks for the detailed response and the additional experiments. The response addressed my questions. Thus I will keep my original recommendation of acceptance.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes an algorithm to learn coordination strategies for multi-agent reinforcement learning. It combines gradient-based optimization (Actor-critic) with Neuroevolution (genetic algorithms style). Specifically, Actor-critic is used to train an ensemble of agents (referred to as “team”) using a manually designed agent-specific reward. Coordination within a team is then learned with Neuroevolution. The overall design accommodates sharing of data between Actor-critic and Neuroevolution, and migration of policies. Evaluation is done using the multi-particle environments (Lowe et. al. 2017) and a Rover domain task.


I have the following questions:

1.	Comparison with PBT-MARL (Liu et al, 2019): PBT-MARL also proposed using gradient-methods (Retrace-SVG0) for learning with dense, shaped (local) rewards, and then using Neuroevolution to optimize the agents for coordination. I don’t think the description in Section 2 characterizes PBT-MARL well enough – reading that gives the impression that it only evolves the scalarization coefficients. PBT-MARL combines rewards with different discount factors (yielding it much more representation power than simple scalarization), and the discount factors are also evolved. Furthermore, the agent network weights are evolved based on team-reward, to learn coordination strategies. 

At a qualitative level, this paper seems to be using a similar approach, albeit the specifics of Neuroevolution and policy-gradients are different. I would like the authors to shed light on the scenarios where they argue their method holds inherent advantage(s) compared to PBT-MARL.

2.	If I understand correctly, the individual agents in the team trained with DDPG would converge to similar policies – this is because each individual is trained with the same (agent-specific) reward, all agents in team use the same shared critic Q, and there is no flow of team from Neuroevolution to DDPG phase. This in itself is not a problem, because MADDPG should do the same if all agents are trained with the same team-reward function. The issue is that there are crucial differences in the architectures -- while the MADDPG paper had a separate Q network and policy network for each agent, MERL shares policy parameters (lower layers) between agents and uses a single, shared Q network. Have the authors done ablations with more aligned architectures, so that the improvements due to the main algorithmic contributions can be clearer?  

3.	The notation defined in the background section is “s” for completed state (all agents) and “o” for individual observations. In the for-loop in Algorithm 1, are correct variables being used at all places, e.g. for pi, Q? In other words, which functions depend on the complete state of all agents, and which are decentralized? Additionally, should lines 23/24 be inside the for-loop?

4.	Experiments – In section 4, could the authors provide the values of N, L, K used in different environments, as applicable? The MADDPG paper claims good performance for multi-particle environments, which contradicts Figure 4 and 5. For example, MADDPG claims about 16 touches per episode with 30% faster prey, but Figure 4 has it converging to about 5. This makes me wonder if the parameters (N, L, K) are different in the two evaluations. Also, physical-deception task performance is reported in success% in the MADDPG paper, but the authors use a different distance metric. A standardized way of comparing performance would help the community. 

5.	In MADDPG paper, an ensemble-based version is reported to perform much better. Since MERL is an ensemble method, is that not a more direct comparison?

Minor points:

1.	In environments (expect Rover), did the authors observe any benefits from adding the agent-specific reward to supplement the team-rewards (i.e. mixed reward setting) for the baselines?

2.	In Figure 8, why is the right-most POI lit when the red trajectory is far away from it? If the authors could provide a video of the MERL-trained policy for this domain, it’d be really cool.



----------------------Post-rebuttal update----------------------


I am satisfied with the author response and changes to the paper. I have increased my rating accordingly.
review_title: Official Blind Review #2

Meta-review: This work has a lot of promise; however, the author response was not sufficient to address the concerns expressed by reviewer 1, leading to an aggregate rating that is just not sufficient to justify an acceptance recommendation. The AC recommends rejection.
Decision: Reject