Venue: ICLR 2018 conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Understanding how-and-why complex motion skills emerge is an complex and interesting problem.
The method and results of this paper demonstrate some good progress on this problem, and focus on
the key point that competition introduces a natural learning curriculum.
Multi-agent competitive learning has seen some previous work in setting involving physics-based skills
or actual robots. However, the results in this paper are compelling in taking this another good step forward.
Overall the paper is clearly written and I believe that it will have impact.

list of pros & cons
+ informative and unique experiments that demonstrate emergent complexity coming from the natural curriculum
  provided by competitive play, for physics-based settings
+ likely to be of broad interest
- likely large compute resources needed to replicate or build on the results
- paper is not anonymous to this reviewer, given the advance publicity for this work when it was released
==> overall this paper will have impact and advances the state of the art, particular wrt to curriculums
    In many ways, it is what one might expect. But executing on the idea is very much non-trivial.

other comments

Can you comment on the difficulty of designing the "games" themselves?
It is often difficult to decide apriori when a game is balanced; game designers of any kind
spend significant time on this. Perhaps it is easier for some of the types of games investigated in
this paper, but if you did have any issues with games becoming unbalanced, that would be worthwhile commenting on.
Game design is also the next level of learning in many ways.  :-)

The opponent sampling strategy is one of the key results of the paper.
It could be brought to the fore earlier, i.e., in the abstract.

How much do the exploration rewards matter?
If two classes of agents are bootstrapped with different flavours of exploration rewards, how much would it matter?

It would be generally interesting to describe when during the learning various "strategies" emerged,
and in what order.

Adding sensory delays might enable richer decoy strategies.

The paper could comment on the further additional complexity that might result from situations
that allow for collaboration as well as competition. (ok, I now see that this is mentioned in the conclusions)

The Robocup tournaments for robot soccer (real and simulated) have for a long time provided
a path to growing skills and complexity, although under different constraints, and perhaps less interesting
in terms of one-on-one movement skills.

Section 2, "Notation"
why are the actions described as being discrete here, when the paper uses continuous actions?
Also, "$\pi_{\theta}$ can be Gaussian":   better to say that it *is* Gaussian in this paper.

"lead to different algorithm*s*"

Are there torque limits, and if so, what are they?

sec 4: "We do multiple rollouts for each agent *pair*" (?)

"Such rewards have been previously researched for simple tasks like walking forward and standing up"
Given the rather low visual quality and overly-powerful humanoids of the many of the published "solutions", 
perhaps "simple" is the wrong qualifer.

Figure 2:  curve legend?

"exiting work" (sic)

4.2 Opponent sampling:
"simultaneously training"  should add "in opponent pairs" (?)

5.1 "We use both MLP and LSTM"
should be "We compare MLP and LSTM ..." (?)

For "kick and defend" and "you shall not pass", are there separate attack and defend policies?
It seems that these are unique in that the goals are not symmetric, whereas for the other tasks they are.
Would be worthwhile to comment on this aspect.

episodic length T, eqn (1)
It's not clear at this point in the paper if T is constant or not.

Observations: "we also give the centre-of-mass based inertia *tensor*" (?)

"distance from the edge of the ring"
How is this defined?

"none of the agents observe the complete global state"
Does this really make much of a difference?  Most of the state seems visible.

"But these movement strategies" -> "These movement strategies ..."

sec 5.4  suggest to use $\mathrm{Uniform}(...)$

"looses by default" (sic)

review_title: Impressive results; insights on opponent sampling

final_score: rating: 3: Clear rejection
review_content: In this paper, the authors produced quite cool videos showing the acquisition of highly complex skills, and they are happy about it. If you read the conclusion, this is the only message they put forward, and to me this is not a scientific message.

A more classical summary is that the authors use PPO, a state-of-the-art deep RL method, in a context where two agents are trained to perform competitive games against each other. They reuse a very recent "dense reward" technique to bootstrap the agent skills, and then anneal it to zero so that the competitive rewards obtained from defeating the opponent takes the lead. They study the effect of this annealing process (considered as a curriculum) and of various strategies for sampling the opponents. The main outcome is the acquisition of a large variety of useful skills, just observed from videos of the competitions.

The main issue with this paper is the lack of scientific analysis of the results, together with many local issues in the presentation of these results.
Below, I talk directly to the authors.

---------------------------------

The related work subsection is just a list of works, it should explain how the proposed work position itself with respect to these works.


In Section 5.2, you are just describing "cool" behaviors observed from your videos.
Science is about producing quantitative results, analyzing them and discussing them.
I would be glad to read more science about these cool behaviors. Can you define a repertoire of such behaviors?
Determine how often they are discovered? Study how the are represented in the networks?
Anything beyond "look, that's great!" would make the paper better...

By the end of Section 5.2, you allude to transfer learning phenomena.
It would be nice to study these transfer effects in your results with a quantitative methodology.

Section 5.3 is more scientific, but it has serious issues.

In all subfigures in Figure 3, the performance of opponents should be symmetric around 50%. This is not the case for subfigures (b) and (c-1). Why?
Do they correspond to non-zero sum game? The x-label is "version". Don't you mean "number of epochs", or something like this? Why do the last 2 images
share the same caption?

I had a hard time understanding the message from Table 1. It really needs a line before the last row and a more explicative caption.

Still in 5.3, "These results echo"...: can you characterize this echo? What is the relationship to this other work?

Again, "These results shed further light": further with respect to what? Can you be more explicit about what we learn?

Also, I find that annealing a kind of reward with respect to another is a weak form of curriculum learning. This should be further discussed.

In Section 5.4, the idea of using many opponents from many stages of learning in not new.
If I'm correct, the same was done in evolutionary method to escape the "arms race" dead-end in prey-predator races quite a while ago  (see e.g. "Coevolving predator and prey robots: Do “arms races” arise in artificial evolution?" Nolfi and Floreano, 1998)

Section 5.5.1 would deserve a more quantitative presentation of the effect of randomization.
Actually, in Fig5: the axes are not labelled. I don't believe it shows a win-rate. So probably the caption (or the image) is wrong.

In Section 5.5.2, you "suspect this is because...".
The role of a scientific paper is to clearly establish results and explanation from solid quantitative analysis. 

-------------------------------------------
More local comments:

Abstract:

"Normally, the complexity of the trained agent is closely related to the complexity of the environment." Here you could cite Herbert Simon (1962).

"In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself."
Well, for an agent, the other agent(s) are part of its environment, aren't they? So I don't like this perspective that the environment itself is "simple".

Intro:

"RL is exciting because good RL exists." I don't believe this is a strong argument. There are many good things that exist which are not exciting.

"In general, training an agent to perform a highly complex task requires a highly complex environment, and these can be difficult to create." Well, the standard perspective is the other way round: in general, you face a complex problem, then you need to design a complex agent to solve it, and this is difficult. 

"This happens because no matter how weak or strong an agent is, an environment populated with other agents of comparable strength provides the right challenge to the agent, facilitating maximally rapid learning and avoiding getting stuck." This is not always true. The literature is full of examples where two-players competition end-up with oscillations between to solutions rather than ever-increasing skill performance. See the prey-predator literature pointed above.

"in the domain of continuous control, where balance, dexterity, and manipulation are the key skills." In robotics, dexterity, and manipulation usually refer to using the robot's hand(s), a capability which is not shown here.

In preliminaries, notation, what you describe corresponds to the framework of Dec-POMDPs, you should position yourself with respect to this framework (see e.g. Memory-Bounded Dynamic Programming for DEC-POMDPs. S Seuken, S Zilberstein)

In PPO description : Let l_t(\theta) ... denote the likelihood ratio: of what?

p5:
would train on the dense reward for about 10-15% of the trainig epochs. So how much is \alpha_t? How did you tune it? Was it hard?

p6:

you give to the agent the mass: does the mass change over time???

In observations: Are both agents given different observations? Could you specify which is given what?

In Algorithms parameters: why do you have to anneal longer for kick-and-defend? What is the underlying phenomenon?

In Section 5, the text mentions Fig5 before Fig4.

-------------------------------------------------
Typos:

p4:
research(Andrychowicz => missing space
straight forward => straightforward

p5:
agent like humanoid(s)
from exi(s)ting work

p6:
eq. 1 => Eq. (1) (you should use \eqref{})
In section 4.1 => In Section 4.1 (same p7 for Section 4.2)

"One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore the benefit of this exploration reward. As already argued, we found the exploration reward to be crucial for learning as otherwise the agents are unable to explore the sparse competition reward." => One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore its benefit. As already argued, we found it to be crucial for learning as otherwise the agents are unable to explore the sparse competition reward.

p8:
in a local minima => minimum

p9:
in references, you have Jakob Foerster and Jakob N Foerster => try to be more consistent.

p10, In Laetitia Matignon et al.  ... markov => Markov

p11, I would rename C_{alive} as C_{standing}
review_title: This paper showing nice results lacks a serious scientific analysis and contains several issues

final_score: rating: 7: Good paper, accept
review_content: This paper demonstrates that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself and such environments come with a natural curriculum by introducing several multi-agent tasks with competing goals in a 3D world with simulated physics. It utilizes a decentralized training approach and use distributed implementation of PPO for very large scale multiagent training. This paper addresses the challenges in applying distributed PPO to train multiple competitive agents, including the problem of exploration with sparse reward by using full roll-outs and use the dense exploration reward which is gradually annealed to zero in favor of the sparse competition reward. It makes training more stable by selecting random old parameters for the opponent. 
 
Although the technical contributions seem to be not quite significant, this paper is well written and introduces a few new domains which are useful for studying problems in multiagent reinforcement learning. The paper also makes it clear regarding the connections and distinctions to many existing work. 

Minor issues:

E[Loss] in table 1 is undefined.

In the notation section, the observation model is missing, and the policy is restricted to be reactive.
 
Uniform (v, \deta v) -> Uniform (\deta v, v)

review_title: Review for Emergent Complexity via Multi-Agent Competition

final_score: rating: null
review_content: There is a strong disagreement among reviewers (top paper according to one, clear rejection according to me), so I believe some discussion is necessary.

I completely agree that the videos are awesome, but in itself a nice video does not make a strong scientific paper. Engineering (and most of time a good deal of science) is about producing new interesting phenomena, but science is also about rigorously analyzing these results (rigorously generally means in a quantitative way) and extracting scientific messages from these analyses.

Quoting this excellent blog post 
http://www.inference.vc/my-thoughts-on-alchemy/
"It's Okay to use non-rigorous methods, but... it's not okay to use non-rigorous evaluation".

Let me just take 3 examples to show you that this paper lacks rigor everywhere (the first two are admittedly minor points, but they feed my general feeling about this paper. Most other points are present in my review below, and the authors did not make the effort to address all of them):

Taken from their reply to my review:

1) "“In all subfigures in Figure 3, the performance of opponents should be symmetric around 50%. This is not the case for subfigures (b) and (c-1). Why? Do they correspond to non-zero sum game?”
Reply:
The games are not zero-sum, there is some chance of a draw as described in Section 3."

Well, in case of a draw, both players should get 0, so this cannot be a correct explanation of why the game is not zero sum. If I cannot trust the answer, can I trust the results themselves?

2) 4. "“Why do the last 2 images share the same caption?”
Reply:
Because the kick-and-defend game is asymmetric, so there are two plots -- one where keeper is trained with curriculum and another where kicker is trained with curriculum. "

Fine. I checked in the new version. The only sentence about Fig3 is "We plot the average win-rates over 800 games at various intervals during training in Fig. 3, for the sumo and kick-and-defend environments." Not a word about the above fact. The reader has to guess that. By the way, why don't we get any result for the "run to goal" case? What are the "various intervals"? Etc.

3) This one is much more serious, about transfer learning. Any rigorous transfer learning paper will precisely define a source domain and a target domain and will measure (quantitatively) the performance in the target domain with and without training first in the source domain. Here we just get "The agent trained in a non competitive setting falls over immediately (with episodes lasting less than 10 steps on average), whereas the competitive agent is able to withstand strong forces for hundreds of steps. This difference (in terms of length of episode till agent falls over) can be quantified easily over many random episodes, we only included the qualitative results as the difference is huge and easily visible in the videos."
OK, this is awesome. But to me, the evaluation methodology cannot just be "visual". This would be perfectly okay for the "scientific american" magazine, but this is not OK for a strong scientific conference. The evaluation itself has to be performed by the computer, because this makes it necessary to rigorously define all the conditions of this evaluation (define fall and stand, define the initial posture, define the number of steps you evaluate, define the wind variations, define everything). The computer will also be able to perform a statistical analysis, which is completely lacking here: did this phenomenon appear every time? If not, what are the chances? For instance, given the lack of quantitative analysis, one cannot determine if another method to come will perform better or worse. 

To me, this paper is representative of what Ali Rahimi's talk was about (the talk was given after I wrote my review). So sorry guys, but though I like the videos and I must admit that my review is a little too harsh because the paper irritated me, I'm still considering that this paper should be rejected.





review_title: Lack of evaluation rigor

Meta-review: This paper received divergent reviews (7, 3, 9). The main contributions of the paper -- that multi-agent competition serves as a natural curriculum, opponent sampling strategies, and the characterization of emergent complex strategies -- are certainly of broad interest (although the first is essentially the same observation as AlphaZero, but the different environment makes this of broader interest).

In the discussion between R2 and the authors, I am sympathetic to (a subset of) both viewpoints.

To be fair to the authors, discovery (in this case, characterization of emergent behavior) can be often difficult to quantify. R2's initial review was unnecessary harsh and combative. The points presented by R2 as evidence of poor evaluation have clear answers by the authors. It would have been better to provide suggestions for what the authors could try, rather than raise philosophical objections that the authors cannot experimentally rebut.

On the other hand, I am disappointed that the authors were asked a reasonable, specific, quantifiable request by R2 --
"By the end of Section 5.2, you allude to transfer learning phenomena. It would be nice to study these transfer effects in your results with a quantitative methodology.”
-- and they chose to respond with informal and qualitative assessments. It doesn't matter if the results are obvious visually, why not provide quantitative evaluation when it is specifically asked?

Overall, we recommend this paper for acceptance, and ask the authors to incorporate feedback from R2. 
Decision: Accept (Poster)