Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a kind of curriculum for large-scale multi-agent learning. The related work section mentions some obvious points of comparison (note: see also https://science.sciencemag.org/content/364/6443/859.abstract). However, the authors do not compare with ANY of this work (either in terms of algorithm design or performance). It is therefore difficult to evaluate the contribution. 

In more detail, the paper combines RL, multi-agent learning and evolution. This is an extremely challenging domain, with many moving parts. How does this approach relate to the work of Salimans et al, Jaderberg et al, Houthooft et al, etc? Without detailed discussion and experiments it is impossible to tell if this is an advance. Improving on the baselines is a useful sanity check. Showing the work is an actual contribution requires comparing against other algorithms in the same space. 

---- ----
After reading the rebuttal and other reviews and comments, I've modified my score to weak accept. The paper makes an interesting contribution that is distinct from other approaches.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposes a new method of scaling multi-agent reinforcement learning to a larger number of agents using evolution. Specifically, the procedures (EPC) involves starting with a small number of agents, training multiple sets in parallel, and doing crossover to find the set of agents that generalize best to a larger number of agents. This is motivated by the intuition that agents that perform best in small groups may not be the ones that perform best in larger groups. These claims are empirically verified in three games based on the particle world set of environments. 

I’m a fan of ‘automatic curriculum learning’-style methods designed to gradually add complexity to improve final agent performance, and this paper is no exception. The proposed method is simple, but it makes sense. I like the fact that it is both RL algorithm agnostic, and that it can be largely executed in parallel, which means that it introduces only small training time overhead. I also like the proposed method of making the Q function agnostic to the number of agents and entities using attention (although whether these policies incorporate information from previous time steps, or if they can be made to do so). The experimental results are thorough, comparing to MADDPG, a simpler version of their curriculum without evolution, and a recently proposed method for scaling up MADDPG, showing that EPC consistently outperforms all of them, and is more stable. I think the complexity of the environments is also suitable for this style of paper, although it would be nice to see results in a more open and complex domain such as the recent NeuralMMO game.

Overall, I think this is a good paper and I recommend acceptance.  


Small fixes:
-	‘asThank yoTha’ -> not sure what this means
-	N1 agents for of role 1 -> N1 agents of role 1
-	“We adopt the decentralized learning framework” --- even if each agent has their own Q function, if that Q function is centralized (uses the observation of all agents) then training is still centralized 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Review Update: Thank you for the detailed response, it raised my opinion of the paper as it reduced my concerns on the rigor of the experiments performed. I believe the changes increase the significance of the contribution and may help it reach a broader audience.

--
This paper proposes evolving curriculums of increasing populations of agents to improve multi-agent reinforcement learning with large number of agents. The topic is of relevance to the ICLR community and the results are tending towards publishable contributions, but I have some concerns that I would like the authors to discuss in their rebuttal.

The method is evaluated on a good range of suitably challenging environments. However, why did the authors propose new challenges within the particle environments and not those used in the original publication? This change makes it harder to compare results across publications, whilst not seeming to add a significant change in the learning problem beyond what was present in the original benchmark tasks. The food collection task sounds like it may be equivalent to simple spread. Will the new environments be released as open source for others to build upon this work? 

The method is compared against a good range of baseline methods and ablations of the proposed method. However, without grounding the results in a known environment it is hard to place whether the implementations of MADDPG and Mean-Field are fair reproductions. Presuming the authors are using the open source implementation of MADDPG (please confirm) this is most significant for Mean-Field particularly given its poor performance in Section 5.4. Please provide evidence that this is a fair comparison.

To evaluate the resultant agents in the competitive environments, all methods were placed into games against the authors proposed EPC method. Was this the same EPC opponents the evaluated EPC team were trained against? If so, this is an unfair advantage to the EPC team as it has time to optimize against this opponent whilst the other methods have not. Even if it is an EPC team from a different training run, there may be outstanding biases in the joint policies EPC tends towards that benefit the EPC team evaluated. This could be overcome by evaluating all methods in competition with all other methods.

For all experimental results, please quantify variance in performance as well as average value (currently only done in Figure 9 a and b). How many repeats of evaluation and training were performed for each? Without these details the claim on page 9 that "EPC is always the best among all the approaches with a clear margin" is too strong. Are these differences statistically significant? Additionally, please also include the maximum scores (where normalized score = 1.0) for all experiments, as presenting results with only normalized scores unnecessarily reduces the reproducibility of the work. 

Finally, in Appendix B, the authors provide a list of hyperparameter settings without discussion of how these were chosen. Were they optimised for one specific method or set to defaults from the literature? In particular, as the performance of Att-MADDPG is still improving at the end of the plot in Figure 9e, I am concerned that the #episodes was chosen to optimise the performance of EPC.

Overall, this is an interesting approach with promising initial results. I believe the contribution would be significantly improved by addressing the issues above and look forward to the authors responses which could increase my rating to acceptance.

Things that could improve the paper but did not impact the score:
1) On page 5 it is noted that the authors do not share parameters between the Q-function and policy. It would improve the paper to justify why this choice was made.
2) Page 5: "N_1 agents for of the role" -> N_1 agents of the role
3) Page 5: "as follows to evolved these K parallel sets" -> evolve
4) Page 7: "resources asThank yoTha green landmarks"
5) Page 8: "understand how the trained sheep behavior in the game" -> how the trained sheep behave in the game
6) Page 14: There are repeated grammatical issues in Appendix A e.g. "is more closer to grass / sheep / other agents" -> is closer to grass / sheep / other agents and "will less negative reward" -> will receive less negative rewards
review_title: Official Blind Review #2

Meta-review: The paper proposes a curriculum approach to increasing the number of agents (and hence complexity) in MARL.

The reviewers mostly agreed that this is a simple and useful idea to the MARL community. There was some initial disagreement about relationships with other RL + evolution approaches, but it got resolved in the rebuttal. Another concern was the slight differences in the environments considered by the paper compared to the literature, but the authors added an experiment with the unmodified version.

Given the positive assessment and the successful rebuttal, I recommend acceptance.
Decision: Accept (Poster)