Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 1. Summary

The authors report on an empirical study of emergent behavior of multiple RL agents learning to play hide-and-seek (a sparse reward task). The main point of this paper is that RL agents learning at scale (large number of samples, batch-size 64000). can learn to solve tasks with strategies that are human-interpretable (e.g., using ramps, boxes). Scale also requires various simplifications (e.g., keeping the learning setup as close as possible to a single-agent problem as possible).

Agents are grouped in 2 teams (seekers, hiders). Each agent receives a team reward, e.g., it can be punished for events that it did not participate in, e.g., if a team-mate is seen by an opponent. If hiders are hidden, seekers also automatically see reward. The first 40% of the episode there is no reward to let hiders hide.

There is one actor model, all agents share weights. Hence this is self-play: hiders and seekers use the same agent model. Also, all agents use a central value function that can see the entire state (decentralized execution, centralized learning). This makes the setting basically a single-agent problem, with the only decentralized aspect being each actor model only receiving its own observation. Note that a large body of multi-agent RL work in fact uses agents that do not share weights, etc.

Other features described:
- Auto-curricula: e.g. agents find new strategies (using ramps, boxes) that other agents have to counteract.
- Human-relevant skills: They report that the agent model learns multiple ways to interact with (objects in) the environment that are semantically interesting (resembles something humans might do).
- Authors compare with policies learning via intrinsic motivation.
- Evaluation through transfer learning shows some benefit of transfer of hide-seek agents to auxiliary tasks. However, it is not so clear how this evaluation informs future work on transfer learning (e.g., how would you pick evaluation tasks for a given train-task?) 

1. Decision (accept or reject) with one or two key reasons for this choice.

Reject.

The main point of the paper is empirical RL at scale. Although the learned behaviors are human-interpretable, this does not seem surprising given the fact that in many (large-scale) RL applications (Atari games, Go, DotA 2, Starcraft), it has been observed that RL agents can learn to manipulate and use their environment (which includes other agents!) in unexpected ways / find creative ways to exploit the reward function (see e.g. demos in https://www.alexirpan.com/2018/02/14/rl-hard.html). There has also been work on object-level RL [Agnew, Domingos 2018], which involves agents interacting with objects in the environment. Compared to this, the observation that RL agents learn human-interpretable uses of objects does not seem surprising.

The paper also does not give new insights in how to make large-scale RL ``'work'. For instance, there are no significant differences in algorithm / model structure from DotA / Starcraft agents that can inform future large-scale experiments.

The paper also does not introduce new concrete evaluation metrics that can apply to other tasks / RL problems, skill detection / segmentation methods to learn the structure of auto-curricula. Furthermore, the setup is very close to a single-agent problem (see above), and is far simpler in the multi-agent assumptions from other decentralized multi-agent work (Foerster 2018, Jacques 2019, etc).
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Authors in introduce a new competitive/cooperative physics-based environment in which different teams of agents compete in a visual concealment and search task with visibility-based team-based rewards (although There are no explicit incentives for agents to interact with objects in the environment). They show that, complex behaviour emerge as the episode progresses and agents are able to learn 6 emergent skills/(counter-)strategies (including tool use), where agents intentionally change their environment to suit their needs. Agents trained using self-play 

In my opinion, this is an excellent paper which main contribution is to provide experimental evidence that relevant and complex skills and strategies can emerge from multi-agent RL competing scenarios.

Minor comments:

- Hide&seek rules and safety issues: is it not supposed that hiders and the seekers could not get together (i.e., hiders cannot push seekers or as we can see in some videos)? Furthermore, it is surprising (one would say worrying) that hiders identified the barriers as an impediment to the seeker (not only as a way to hide). I wouldn’t say that this is a “ human-relevant strategies and skills “ as the authors claim. Hider agents even double walled seekers!  

- Have the authors thought about joining the Animal-AI Olympics (http://animalaiolympics.com/) competition? It would be a great opportunity to to test the skills of your agents in a further general testing scenario. They provide an arena (test-bed) which contains 300 different intelligent tests for testing the cognitive abilities of RL agents (https://www.mdcrosby.com/blog/animalaiprizes1.html) which have to interact with the environment. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: # Review ICLR20, Emergent Tool Use...

This review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.

I apologize in advance for being reviewer 2.

## Overall

**Summary**

The article introduces a new multi-agent physics environment called "hide-and-seek". The authors trained agents in this environment and studied the emergence of and changes in strategies. The authors also study the performance of these same agents in new "targeted intelligence tests" compared to training from scratch and compared to agents trained with curiosity.

**Overall Opinion**

I think the environment is very appealing and the paper is overall well-structured and demonstrates novel work. Therefore I'd recommend this paper to be accepted. That being said, there are glaring issues with some of the writing that need to be addressed before I think this work conforms to the standards of ICLR. However, if these issues are addressed, I have no issue increasing my review score.

Main problems:

- The majority of the paper presents essentially a case study of what happened during a single seed of policy training. For RL literature that's very uncommon and I think it's consensus that DRL is very sensitive to random seeds. I know that you do have additional seeds in the appendix, but why didn't you mention those in the main body of the paper? You seem to have found some robustness against multiple seeds, so why not show it? And also the fact that Figure 1 & 3 only apply to 1 seed is not mentioned. I think this is easy enough to fix - I suggest since you're already at 10 pages, to just bring in the additional seeds from the appendix and average over their performance in Fig.1&3.
- The contributions section is overselling the work: (1) states that autocurricula lead to changes in agent strategy - Maybe I'm mistaken here but that sounds like a tautology. In other words, "a self-generated sequence of challenges" ("Autocurricula", according to [Leibo et al., 2019][1]) lead to changes in strategy. And (3) advertises "a proposed framework for evaluating agents in open-ended environments" and also "a suite of targeted intelligence tests for our domain". The former of those two is either not in the paper or you mean your section "6.2 Transfer and Fine-Tuning as Evaluation", which isn't novel (see e.g. [Alain & Bengio, 2016][2])
- Your acknowledgments should be anonymized until publication. Otherwise, reviewers might draw conclusions which group published this work, thus violating the double-blind review procedure.

[1]: https://arxiv.org/pdf/1903.00742.pdf
[2]: https://arxiv.org/pdf/1610.01644.pdf

Like I mentioned above, I think these are all easy to address, which should allow acceptance of this work. Here are some additional questions, comments, and nitpicks:

## Specific comments and questions

### Abstract

- "evidence that ... competition may scale better with increasing environment complexity" - that's only shown in the appendix

### Intro

- You mention TD-Gammon as a game, but I think it's an algorithm for the game Backgammon, similarly to how "Go" is the game and "AlphaGo" is an algorithm for playing.

### Rel. Work

- all good

### Hide And Seek

- Arena boundaries: What's the penalty and what's " too far outside the play area"? And in all depictions, it looks like the geometry of the arena is elevated around the edges and the agents don't have a jump action, so how would they ever go out of borders? After watching the videos: Apparently, the jagged-looking arena boundary in the videos is purely cosmetic and agents can still access that space. This is unclear from just the paper and the renderings in Figure 1.

### Policy Optimization

- Policy network and fusion are underspecified: How do you deal with the varying number of agents, boxes, obstacles? Do you just set the x/v of the missing pieces to zero or is the observation actually of a different shape in case there are more/fewer objects or agents? How's the embedding done that is depicted in Figure 2? Also, I didn't see the embedding being mentioned in the text - any reason for that?
- Figure 2 - This diagram is visually appealing but confusing and needs to be improved. Why does the agent's embedding say "1"? Why do the other agents' embeddings have a "-1" at the end of the orange box and the others don't? Is the agent's embedding concatenated with the other embeddings? If so, why and how (concat, sum, multiply, conditional batch norm, etc.)? In the center and on the right you use a blue block to indicate the agent's embedding and then at the bottom right you seem to use it as a network component or something (between the "LSTM")? If you're trying to signal that this is the agent's perception at different stages in the network, I'd use a different color to separate it from the agent's lidar and pos/vel. You don't mention that "x,v" stands for "position, velocity".
 
### Auto-curriculum and Emergent Behavior

- Figure 3: "environment-specific" (add dash). Draw skill development boundaries like in Fig.1.
- How exactly does the "surfing" work? The seekers step (not jump, right, since there is no jumping?) onto the boxes and then what? The momentum propels the box forward? Do other seekers push the box? Their movement on top of the box somehow moves the box (this seems to be the case judging by the videos but this is the least physically plausible)? This is a super interesting adaptation but I'd suspect the physics simulation to have a bug/glitch that's being exploited here.
- You mention in footnote 3 that the developmental stage and changes in reward aren't necessarily correlated. The same seems to be true for the metrics in Fig.3, which raises the question how did you come up with those boundaries for the different developmental stages in Fig.1? Did someone look at rollouts from the trained policy every couple of million steps? And do all agents learn new skills at the same time or is there a delay? From my understanding, they are all using the same policy and critic networks but maybe dependent on the proximity of an agent to an object/obstacle, it's easier or harder to execute.

### Evaluation

- clear and well-written, slightly too much content in the appendix and not enough in the main paper. Weird appendix numbering - A.6 appears in the main paper pages after A.7

### Discussion and Future Work

- all good

### Appendix

- I appreciate the TOC. I did not look into Appendix B-D because it's another 10 pages on top of the 10 pages of the article.

All in all an interesting work. Good luck with the rebuttal/discussion.
review_title: Official Blind Review #2

Meta-review: This paper describes how multi-agent reinforcement learning at scale leads to the evolution of complex behaviors. Actually, "at scale" may be an understatement - a lot of computing power was used here. But the amount of compute used is not the point, rather the point is that complex and fascinating behavior can emerge from a long co-evolutionary process (though gradient-based RL is used here, the principle is the same) where the arms race forms an implicit curriculum. This is the existence proof that people in artificial life and adaptive behavior have been looking for for so long. 

Two reviewers were positive about the paper, with a third being negative because the paper does not give any new insights about how to do RL at scale. But that was not the stated aim of the paper, as the authors clarify in a response.

This paper will draw quite some attention and deserves an oral presentation.
Decision: Accept (Spotlight)