Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper looks at the question of emergent communication amongst self-interested learning agents. The paper finds that "selfish" (ie. self-interested) agents can learn to communicate using a cheap talk channel as long as the objective is partially cooperative. 
The paper makes states that this is is a novel finding that contradicts the previous understanding of emergent communication in the literature (side point: at least some of the papers referenced for this claim did not at all make the claim). 

I believe there is a major miss-understanding here: As noted in the paper, self-interested agents can learn to communicate in settings in which the reward function is cooperative. Furthermore, it is also known that in 2 player zero-sum there is no incentive to learn a communication protocol. 
This clearly shows that talking about whether or not "selfish" agents can learn to communicate only ever makes sense within the context of a specific game / reward structure. 

With this in mind, the main finding, agents learn to somewhat communicate with each other in a simple toy setting, with more communication happening when the payouts are more cooperative, is not very interesting. 

This doesn't mean that there isn't a good paper to be written here, in principle. Finding simple settings in which SOTA multi-agent learning "fails", ie. doesn't find Nash policies, understanding why it fails and then finding ways to mend things is generally a good research direction. However, this would require a few things which are currently lacking from the paper: (1) clear understanding of the Nash policies for the different reward settings (2) Implementation of SOTA methods for MARL which are appropriate for this setting (3) In depth analysis of learning successes and failures, ideally in settings which have previously been studied in literature (given how task-specific this analysis necessarily is).

Regarding 2: General sum games will generally have mixed-strategies as Nash equilibria (just think 'rock-paper-scissors'). With this in mind, using a deterministic policy for the receiver is inappropriate for making any claims about learning in general sum games. 
Furthermore, it is well known that independent gradient descent (IGD) is not generally going to converge in general sum games (consider the loss functions X * Y and - X *Y or matching pennies). So looking at the outcome of IGD without checking for convergence means the results could be just about anything. Indeed, we don't have to go all the way to writing about emergent communication or complex "sequential social dilemma" to study this, those issues can easily be found in (iterated) matrix games. 

This gets us to the second major point of the paper. To the authors' credit,  LOLA [1] has been shown to help with convergence in general sum settings and to lead to the emergence of cooperation and reciprocity in iterated games. 

However, the key point for the ‘cooperation’ part is iterated. In a single shot setting (which is explored in this paper), there is simply no way for the agents to reciprocate with each other. So in short, I do not believe the authors' interpretation that agents learn to cooperate with each other because of LOLA, but I do believe that LOLA can help with the learning of mixed strategies (at least for the sender, given that the receiver is deterministic) and with stabilizing convergence. Lastly, the part of the experimental section is dominated by large error bars and graphs that are difficult to interpret.


Other points:
-"..but train agents to emerge their own." (and many other instances). AFAIK "to emerge something" is grammatically wrong (and also sounds really odd). 
-"Since the loss is differentiable with respect to the receiver, it is trained directly with gradient descent, so we are training in the style of a stochastic computation graph (Schulman et al., 2015).". This is a weird statement. You don't need SCGs for training a supervised objective. Also, note that the loss is also differentiable with respect to the action of the 1st agent. It is trivial in this setting to compute the true expected return, if that is what you are after. Note my point above about deterministic policies
-"We perform a hyperparameter search to over both agents’" -> spurious "to"
-"We investigate a similar scenario but concern ourselves with learning agents as opposed to fully-rational agents that have full knowledge of the structure of the game, and we do not assume that agents use an existing language, but train agents to emerge their own" .This would be interesting, if the game was complex.
- L_1 vs L - these symbols are used inconsistently, with the subscript _1 sometimes being applied and sometimes not.
-"we can look to extant results" - s/extant/extent?
-"We use the L2 metric only on hyperparameter search and keep L1 as our game’s loss to maintain a constant-sum game for the fully competitive case." - A few points: (a) the game is not in general constant sum (b) By doing this hyperparameter search the evaluation is strongly biased towards 'fair' attributions. This seems highly problematic. 
-"We report our results in Figure ??" -> Broken reference. 
-"We do not test b = 180◦ because the game is constant-sum and therefore trivially Ls1 + Lr1 = 180◦." -> So? It would still be interesting to see what learning agents do in this setting. 

[1]: "Learning with Opponent Learning Awareness", Foerster et al. 

[update: I have updated the score based on the discussion with the authors]. While the paper lacks execution and conceptual clarity, I believe the game itself is interesting and could serve as a starting point for more thorough investigation.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This ICLR submission deals with a problem of whether selfish agents can learn to use an emergent communication channel, using a sender-receiver game as a case study. It is found that communication can emerge in partially-competitive scenarios, and conditions in which this can happen are investigated.
This review is delivered with the caveat that I am not an expert in this particulat field.
The investigation seems relevant and the paper is well written and structured, being within the scope of the conference. Proofs in the appendix are sound to the best of my understanding.
The literature review is up to date and seems overall relevant.
This study should be understood as a proof of concept, given that the setting seems rather restrictive, so I am unsure that he results could be generalized.
They seem anyhow promising and partially challenge the current understanding of the problem.
Minor issues:
All acronyms in the text should be defined the first time they appear in the text.
LaTex problem with Fig. reference at the beginning of section 5.1.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: Summary:
This paper introduces a new sender-receiver game to study emergent communication in partially-competitive scenarios. The authors find that communication can also emerge in partially-competitive scenarios and demonstrate how to encourage communication: 1) selfish communication is proportional to cooperation, and it naturally occurs for situations that are more cooperative than competitive, 2) stability and performance are improved by using LOLA, and 3) discrete protocols are better than continuous ones.

Strengths:
- This is an interesting paper that is well written and motivated.
- They have justified a new sender-receiver game that can be tuned for various levels of competition which then allows them to analyze the effects of various levels of cooperation and competition.
- They perform sufficient experimental analysis to show that LOLA outperforms standard methods like REINFORCE in these settings and that discrete communication lends to cooperative communication.
- Evaluation is good in the sense that they repeat their experiments multiple times across different random seeds.

Weaknesses:
- Given that cheap talk is an extremely well-studied topic in economics, I feel that the authors should have devoted more time to explain the difference in setting between their work in classic pieces like those of Sobel and Crawford. The authors should properly define what they mean by learning agents versus fully rational agents, and the key differences between the two. Furthermore, nowhere do the authors in the cheap talk paper assert that agents use an existing language: the equilibrium itself assigns meaning to each sender’s message; this is not part of the problem definition per se. In fact, the work of Sobel and Crawford does not even constrain the size of the vocabulary (as was done in this paper): one of its key contributions is to show that in strictly non-cooperative settings, all equilibria must be partition equilibria, with only a finite number of messages used.
- Section 4.2: “initial random mapping of targets to messages.” The authors made the assumption that this mapping has to be deterministic. Absent a proof or a citation, I find this difficult to accept. This is especially so since mixed strategies are a crucial component of games of imperfect information.
- The introduction of the circular game is suspect. There already exist numerous games involving cheap talk, one of them from the Sobel and Crawford paper. Why is there a need for this new benchmark?
-  The description of the game is given as an algorithm in the appendix. This comes across as counterintuitive: why are the gradient steps being included as part of the game description? A game’s specification and the algorithm which is being used to solve it are two different things.
- It is difficult for me to assess the significance of these results since the authors have not presented real-world scenarios and experiments that demonstrate the importance of selfish communication. For cooperative communication we see it a lot in examples like grounded language learning, visual dialog, multi-agent communication etc. But I am concerned that the new setting proposed in this paper seems like a 'toy setting' to investigate if emergent communication would happen.
- Are the communicated symbols (discrete or continuous) semantically meaningful? It was shown in Kottur et al. (2017) that for emergent communication to occur and generalize to unseen test instances, it was crucial that the communication protocol was grounded i.e. one symbol learning to represent the color, one representing the shape, one representing the size. What is the final communication protocol learned in this case, and it is useful/interpretable in a similar sense?
- Typo: 'Figure ??' in line 3 of section 5.1
review_title: Official Blind Review #1

Meta-review: There has been a long discussion on the paper, especially between the authors and the 2nd reviewer. While the authors' comments and paper modifications have improved the paper, the overall opinion on this paper is that it is below par in its current form. The main issue is that the significance of the results is insufficiently clear.  While the sender-receiver game introduced is interesting, a more thorough investigation would improve the paper a lot (for example, by looking if theoretical statements can be made).
Decision: Reject