Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary:** This paper studies multi-agent communication with an aim to mimic conditions for the emergence of language in society. They argue that the emergence of language requires co-operation. However, people can cheat and make a profit by lying, and eventually, people have no incentive to co-operate. 

A proxy multi-agent setting is created to study this which proceeds in a series of interaction between two (randomly sampled) agents. In each interaction, there is a market demand for N items, and both agents have a fixed quantity of these items and their own hidden utility. An agent makes a proposal to another agent for a set of goods which consists of quantities of each item that they are willing to give. The other agent has to fulfil the remaining quantity to match the market demand. The other agent can accept the proposal at which point both agents get reward based on the remaining items and their respective utilities. Otherwise, the bargaining will continue and the other agent will make a proposal. There is a timeout after which both get a reward of 0.  There are certain agents who are chosen as "liars" who can make a proposal but refused to follow through leading to the remaining market gap being split equally.  Lying can, therefore, result in profit. A credit mechanism is also introduced to measure the performance of agents. Liars can use credit scores to decide whether to lie or not. A neural network is used to generate proposals and REINFORCE style training is performed to optimize reward. Authors also study "natural selection" where low-ranking agents get eliminated after a certain time, and "genetic evolution" where high-ranking agents produce clones.

Three different settings are studied: (i) where there are no measures taken to stop liars, (ii) where each truth-teller can refuse to play with an agent with negative credit score, (iii) where global measures are taken including reward penalty for lying or liar agents being banned for a fixed time. It is found that in both (i) and (ii); liars take over society. Individual counter-measures slow down the proliferation of liars but don't stop them. However, global measures in (iii) are able to stop liars. Authors argue that global measures in human society (reciprocal altruism) is the reason why conditions were right for language to evolve.

**Strength:**
- The setting is interesting. In general, understanding the dynamics of language evolution is a very interesting topic. This can shed light on human history, and maybe potentially be helpful for developing language understanding agents. 

**Weakness:**
- The language part of the story is confusing since there is no language used in the experiments. In fact, what is really being asked here is if we can have a society without cooperation? Since, if everyone ends up being a liar then we don't have a society. Therefore, the language part feels like a red-herring. 

- Implications of the results are unclear as it is not obvious that the setup captures all the important nuances present in human society. E.g., in a real-world, people who are speaking the truth can help each other by sharing technology that can give them a survivor edge over liars. This happens even if liars are not directly punished for their deeds. Then there is the question if the way natural selection and genetic evolution are modelled, reflect the real-world. I am not an expert on evolution but the paper doesn't say enough to justify the choice.

**Questions:**

- In the case where agent "i" lies, the other agent will still end up paying its share of the accepted proposal plus 50% of what agent "i" promised. Is this right? 

- Can one pay more items than they possess? If not, do we always have $d_n \ge q^i_n + q^j_n$ for every $i$ and $j$, as otherwise there are not enough items to satisfy the market?

- In the setting of individual resistance, once an agent ends up with a negative credit score then the truth-tellers will refuse to play with them resulting in an addition of $1 - \sqrt{e} < 0$ to their credit score. So can an agent never increase their total reward once their credit score becomes negative? Or, perhaps liars will still play with them and so there is a chance for their total reward to increase.

- How would the behaviour be if the agent also had access to (i) total reward earned by the other agent, (ii) percentage of cases the other agent lied. E.g., if an agent saw that the other agent lie then they are likely to either not play with them or demand a more lucrative offer before committing. This can allow for more interesting policies to evolve.
review_title: Interesting Setting but Implications of Results are Unclear or Speculative

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Overall review

This paper attempts to address a question in the emergent communication literature: what preserves / maintains the stability of emerged communication protocols.  The authors manipulate the prevalence of lying behavior in a community of agents playing a variant of a Nash bargaining game.  The main take-away is that explicit punishment, from the environment and from truth-tellers not wanting to communicate with liars, can prevent the spread of exploitative lying behavior in the community.

My main worry about the paper is that the conceptual motivation is a bit unclear.  The authors present the paper as addressing when emergent communication can be stable.  And, after identifying a condition under which a penalty for lying removes liars from the population, they conclude that "Only under such a environment, language is finally possible."  But in all experiments, the communication protocol is fixed: mappings from market needs to proposals.  In other words, what changes is less the communication protocol itself, and more the prevalence of lying / truth-telling.  It thus is not clear how these results are to be interpreted as being about the emergence of communication itself, and not alternative uses of a pre-existing protocol.

Pros:
* Addresses a foundational question in emergent communication.
* Explicitly models a community of agents, instead of just dyads.
* Identifies explicit factors influencing their outcome of interest.

Cons:
* Conceptual motivation a bit unclear (see above).
* Communication protocol is fixed, not emergent.
* Many interesting results / sub-experiments are presented as parentheticals / foot-notes, without full details in appendices.


# Small comments

* p 3, choice to not do simultaneous proposals.  More details about these experiments should probably be provided in an appendix.

* Natural selection mechanism: the authors remove the lowest-performing single agent.  This is relatively different from the standard interpretation, which is that the number of offspring in the next generation is proportional to an agent's fitness.  Did the authors experiment with that approach, or can they say more to motivate theirs?

* The y-axis in the main results are "average scaled reward".  How exactly is the scaling done?  This would help in interpreting the results, as would comparison with a random baseline and/or an optimal strategy.

* "As the natural selection approaches, there is less and less need to pretend for liars ..."  I had some difficulty following the reasoning here.  Are the 10 tournaments _before_ a single episode of natural selection?  Or did natural selection occur in the middle?  If the former, it would help to know if these are averaged across "generations" in order to support the authors' interpretation.


# Minor typographic comments

* abstract: "From the perspective of Darwinian, ..." needs to be finished after "Darwinian".  Same at the end of page 1.

* p 1, "some intrigue properties" should be "some intriguing properties"

* p 2, "lairs" should be "liars"; this one recurs a few times in the text.

* p 8, "artificial intelligent" should be "artificial intelligence"
review_title: Interesting, but needs clearer motivation and detail

final_score: rating: 3: Clear rejection
review_content: ##### Summary ##### 
This paper investigates conditions under which communities of cooperative agents are stable. Communities in multi-round bargaining games with evolutionary dynamics are evaluated in three main setups. The first imposes no restrictions on the agents' behavior and is shown to be easily invaded by deceitful agents. The second enables agents to refuse to bargain with deceitful agents. Nevertheless, such communities are shown to be invadable. Finally, in the third setup, a global punishment system is shown to be able to drive out deceitful invaders. The main take-home message is that, when lying is an option, agents(' communities) need to be prepared for it.  

##### Reasons for score #####
I vote for rejecting this submission. The main reason, further detailed in "Cons" below, is that I am not convinced that the problem this manuscript addresses is not an artifact of the rather strong assumption of selfish individuals and communities driven solely by functional pressure. I agree with the authors that we need to start looking at more naturalistic setups (e.g., communities instead of two-player games). However, this also relates to the evolutionary dynamics we take into consideration, and to a careful motivation of the setups we analyze.     

##### Pros #####

+ Language emergence is a topic that has drawn renewed interest from multiple disciplines. Studying the conditions under which agents' acquired behaviors are (not) stable in a dynamic setting is important to understand their underpinning.
+ The setups are well explained and easy to follow   

##### Cons ######
- Much of the motivation of this paper relies on the argument that there are strategies that functionally dominate cooperation. However, the view that functional pressure is all there is to evolution is rather outdated. If the authors wish to stick to this line of argumentation, I encourage them to give strong arguments for why we should disregard other factors, such as transmittability/acquisition (e.g., Kirby et al. 2015, "Compression and Communication in the Cultural Evolution of Linguistic Structure", or Brochhagen et al. 2018, "Coevolution of Lexical Meaning and Pragmatic Use") or neutral models (e.g., Reali & Griffiths 2011, "Words as alleles: connecting language evolution with Bayesian learners to models of genetic drift.", or Perfors & Navarro 2014, "Language Evolution Can Be Shaped by the Structure of the World"). Along the same lines, the view that functional pressure needs to apply to only individuals, and that these are fully selfish, also requires evidence or needs to be clearly marked as a rather strong assumption. The claim that "[...] in order to ensure the survival advantage of many competitors, animals are selfish in nature" flies in the face of the whole line of research on animal alarm calls (e.g., Zuberbuehler 2009, "Survivor Signals: The Biology and Psychology of Animal Alarm Calling"); and, for that matter, the existence of any social animal. In sum, as it stands, I am not convinced that the problem this manuscript addresses is not an artifact of the assumption of selfish individuals and communities driven solely by functional pressure.
- Terminology is very sloppy at times. What is "successful evolution" (p. 1) or "naive natural selection" (p.8)?
- Writing could be greatly improved. There are a lot of typos and borderline grammatical sentences
- References are wrong (e.g., Lewis' work on convention is not from 2008 but from 1969. The same goes for all the references on seminal work in game theory)
- Many assumptions in the setup are not motivated. For instance, is there a reason to use a discrete dynamic over a continous one? (p. 3) Why use a multi-round bargaining game to model language evolution by contrast to, say, an established signaling game-style setup with conflict of interests? How is the "credit mechanism" and the "market" motivated/grounded in light of this being an investigation about language emergence? (p. 3)
- The lines in Figure 3a and Figure 3b are very hard/impossible to read. Particularly Figure 3b.
- The y-axis in the plots in Figure 3 are not aligned, rendering a comparison across conditions hard (and visually misleading)

review_title: This work investigates conditions under which communities of cooperative agents are stable. While the experiments are interesting, there's a lack of motivation behind fundamental assumptions made in this work

Meta-review: The authors present a study on what maintains the stability of emerged communication protocols. To study this question the authors design experiments in bargaining communities of agents in 3 setups,  a) no punishment of restriction of liar agents b) allowing individual agents to refuse bargaining with  liar agents and c) introducing a global punishment system for liar agents.

Overall the reviewers agree that the design of the study is interesting, but also point that motivation and take-home messages of this study are unclear. Having read the paper, I share the same opinion. The authors discuss on a very abstract level about the implications of this study for the field of AI, but this study is quite specific and clearly does not capture all the complexities or real societies. From the scale of results and study, I think it would be more valuable to draw some concrete proposals/implications about perhaps multi-agent modelling or environment design in general. 

All in all, this is an interesting study but some more work needs to be done around research framing.

Decision: Reject