What Preserves the Emergence of Language?
Authors: Ziluo Ding
Tiejun Huang
Zongqing Lu
Venue:  (2021)
Keywords: emergence of language
reinforcement learning

Abstract: The emergence of language is a mystery. One dominant theory is that cooperation boosts language to emerge. However, as a means of giving out information, language seems not to be an evolutionarily stable strategy. To ensure the survival advantage of many competitors, animals are selfish in nature. From the perspective of Darwinian, if an individual can obtain a higher benefit by deceiving the other party, why not deceive? For those who are cheated, once bitten and twice shy, cooperation will no longer be a good option. As a result, motivation for communication, as well as the emergence of language would perish. Then, what preserves the emergence of language? We aim to answer this question in a brand new framework of agent community, reinforcement learning, and natural selection. Empirically, we reveal that lying indeed dispels cooperation. Even with individual resistance to lying behaviors, liars can easily defeat truth tellers and survive during natural selection. However, social resistance eventually constrains lying and makes the emergence of language possible.

Under review as a conference paper at ICLR 2021
WHAT PRESERVES THE EMERGENCE OF LANGUAGE?
Anonymous authors Paper under double-blind review
ABSTRACT
The emergence of language is a mystery. One dominant theory is that coopera- tion boosts language to emerge. However, as means of giving out information, language seems not to be an evolutionarily stable strategy. To ensure the survival advantage of many competitors, animals are selfish in nature. From the perspec- tive of Darwinian, if an individual can obtain a higher benefit by deceiving the other party, why not deceive? For those who are cheated, once bitten and twice shy, cooperation will no longer be a good option. As a result, motivation for communication, as well as the emergence language would perish. Then, what preserves the emergence of language? We aim to answer this question in brand new framework of agent community, reinforcement learning, and natural selee- tion. Empirically, we reveal that lying indeed dispels cooperation. Even with individual resistance to lying behaviors, liars can easily defeat truth tellers and survive during natural selection. However, social resistance eventually constrains lying and makes the emergence of language possible.
INTRODUCTION
Unveiling the principles behind the emergence and evolution of language is attractive and appealing to all. It is believed that this research field is of great significance for promoting the development of enabling agents to evolve an efficient communication protocol (Nowak & Krakauer, 1999; Kottur al., 2017: Chaabouni al., 2019) or acquire existing one (Li & Bowling, 2019). especially when interacting with humans. Previously, many studies have investigated some intriguing properties of language and their effects on the emergence of language (Andreas & Klein, 2017: Lazaridou al., 2018: Mordatch & Abbeel, 2018). The motivation behind these is that human language is considered as a remarkable degree of structure and complexity (Givon, 2013) and each character is the result of evolution, thus they believe that understanding the language itself is an indispensable step to take. Unlike existing work, we, from a different perspective, focus on a fundamental question that what made the emergence of language possible during evolution.
One of the dominant theories in the community of emergent communication is: cooperation boosts language to emerge (Nowak & Krakauer, 1999; Cao et al., 2018). Hence, there has been surge of work investigating this field in cooperative multi-agent (mostly two agents) referential games (Lazaridou & Peysakhovich, 2017: Kottur et al., 2017: Das et al., 2017; Evtimova et al., 2018; Lazaridou et al., 2018), a variant of the Lewis signaling game (David, 1969). However, they seem to miss some basic elements in the human language. On one hand, human language emerges from the community, not just two persons, after all, language is learnable and can spread from one place other (Dagan et al., 2020). Studying language in two-player games is like looking at the world through a keyhole. On the other hand, many works make an agreement that prior to the emergence of language some pre-adaptations occurred in the hominid lineage, and one of the candidates is the ability to use symbols (Deacon, 2003; Davidson, 2003; Christiansen & Kirby, 2003). It seems understanding the emergence of symbolic signals is the key to approach the truth of the origin of language (Deacon, 1998). However, chimpanzees have demonstrated a degree of language capacity by using arbitrary symbols as well as the ability for the cross-modal association, abstract thought, and displacement of thought in time (Meddin, 1979). So why don't they have a language like us? One of the theory is selfishness has kept animal communication at a minimum (Ulbaek, 1998). In more detail, if an individual can obtain higher benefit by deceiving the other party in the cooper- ation, why not deceive? Once deception emerges, mistrust among individuals will haunt. For those who are cheated, once bitten and twice shy, cooperation will no longer be good option. As a result,

---

Under review as a conference paper at ICLR 2021
motivation for communication, as well demands of the emergence of language will perish. But human beings are so special since we have overcome this kind of obstacle and evolved language. Then, what preserves the emergence of language? ? We aim to answer this question in a brand new framework of agent community, reinforcement learning (RL), and natural selection. We believe this process should occur in the pre-language period since lying is possible as long as agents can com- municate. Therefore, our investigating communication protocol uses symbols to transmit meaning based on social convention or implicit agreement.
In this paper, we introduce several agents to form one community and allow natural evolution and elimination among them. Both liars (agents tell lies) and truth tellers (agents always tell truth) exist in the community. Each tournament, every agent is supposed to play a non-cooperative game (Nash Jr, 1950; Nash, 1951; Schelling. 1958; Binmore et al., 1986; Von Neumann & Morgenstern, 2007) with others. In our multi-round bargaining game, agents are required to reach an agreement about how many items to give out so that the total quantity can satisfy the market's demand and they can keep their loss to the minimum in the meantime. We believe this is perfect fit for the nature of human beings and more common in the hominid lineage compared with the cooperation game. Importantly, during the process of natural selection, the fraction of liars and truth tellers may change from time to time and this allows us to observe what factor imposes influence on the motivation of communication, which is the prerequisite of the emergence of language. It is worthy of note that pre-language communication was subject to the constraints of Darwinian evolution. While linguistic change, which began in the post-language communicative era of hominid evolution, by and large tied to society and culture (Givón & Malle, 2002; Li & Hombert, 2002). Thus, we disregard the factors related to linguistic change since we are investigating motivation for communication from which language evolved.
Moreover, apart from the normal setting mentioned above, we add up two more rules further dig out. Firstly, we introduce a credit mechanism for truth tellers. In other words, we make sure truth tellers know the existence of liars which is one step the evolution process. Specifically, every liar has credit in the mind of truth teller, and the credit varies with the profit of truth teller. Cooperation would be impossible between two agents as soon as the credit drops to negative. Secondly, an addi- tional penalty will be brought in as price of lying, and we consider it social pressure for resisting lying behaviors. All in all, we want to make a thorough investigation about how the individual or social resistance lying affects communication.
Empirically, we show that in normal settings, two truth tellers can make a fair agreement, and liars can achieve huge advantage over truth teller by telling lies. As for two liars, there is always better liar that gains relative more than the other. In the credit setting, liars can learn a sophisticated lying strategy that deceives the credit mechanism and makes more profits meanwhile. In both settings, as time goes on, truth tellers seem not to show enough competition against liars and thus die out. In the society setting, we find out liars are afraid of lying if punishment sufficiently large. This again proves the theory (Ulbaek, 1998): in the human lineage, social cooperation based on obligatory reciprocal altruism (Trivers, 1971) as well as a system which punishes people morally and physically for cheating has evolved. In such an environment language is finally possible.
2
EXPERIMENTAL FRAMEWORK
2.1 GAME SETTINGS
We explore emergent language in the context of multi-round non-cooperative bargaining game (Nash Jr, 1950; Nash, 1951) as illustrated Figure 1. The core is binding cooperative strategy with the highest profit is impossible whereas selfish strategy sometimes can. In this case, the behav- ior of telling lies can be meaningful since undermines cooperation and grabs more benefits from others.
In the game, two agents i.j bargain over how to satisfy the demand of market. Agents are presented with N different items. They possess fixed number of quantity for each item (19), J.Y {9) J.M_)) and have their own hidden utilities for N items (fw,, ( [ug 1N 1)= Agents move sequentially. Suppose at bargaining round t, it is turn for agent to be proposer and it makes proposal {pi... ) about how many to give out to the market, which means the other agent j should contribute the rest (d,, p I. - to the market, where d, is the market demand for item Then agent would
2

---

Under review as a conference paper at ICLR 2021
Hisden Latisiee
Hidde Latissn
Tel Les
[AN
Citta
Propocal [1.24]]
Rool 10,10,10
ham
[18,10,10
Agreed
Agent
Giop RA-I
Gie Bottong
Agent
[10,10188)
Callect compaisory &8
Cattedt Compateury 14.1
O I
Market
Overview of the bargaining game. Two agents learn to bargain over how to satisfy the demand of market. The reward of each agent is the value of remaining items. If the demand of market is not satisfied, the difference will collected from both agents evenly a mandatory way.
Figure 1:
choose to accept the proposal or not. The game will be terminated when they make an agreement about the proposal or the number of bargaining rounds reaches the upper limit maxx, and agents and j will receive rewards Σx= (g) p(.n) x ui, and E (1) d, + p(,)) x ud, respectively or get when agreement has been made in the end. In order to make the reward comparable, it will be further normalized by the maximum reward achievable for each agent. Both agents want to keep as more items possible rather than giving out. Therefore, agents are supposed to seek tradeoff between keeping more items and satisfying the market demand.
In this part, we illustrate how lying mechanism works, where lying means the proposal will not be followed by actions. Suppose agents i, satisfy the market demand at round 4, and agent chooses to tell lies about the proposal and then gives nothing to the market. This leads that the market will have demand gap which is {pi,,) J and the market would force each agent to hand over half of the gap for remedy. It is noted that liars are allowed to tell lies about any items. In our settings, we conform to the principle of high risk and high return. To illustrate, when dj = 10 agent j tells lies and keeps offering p21 = 9, agent i definitely delightful and more likely to take di - F = 1. Although agent j can easily put others on the hook, it cannot gain large advantage since the final result is not much better than evenly contributing, On the contrary, if agent; j takes v 1, it allows agent j keeping more items and obtaining more profits. However, this proposal is less appealing to agent i. The key point is want link attractive lies to relatively lower profits.
As agents take turns to play, first-mover has absolute advantages since the other one is compelled to accept unfair proposal get nothing. We once tried to convert our bargaining game from moving alternately to moving simultaneously to solve this problem. In detail, two agents make proposals at the same time and their total quantity should surpass the demand. However, turns out that agents only learn to evenly divide the market demand. In this setting, no agent can guarantee an agreement, therefore the proposal made by an agent could be more conservative, meaning is hard for the agent to infer others' utilities during bargaining. Ultimately, we adopt the strategy (Cao et al., 2018) by randomly sampling T Taax between 4 and 10 mitigate the first-mover effect.
2.2 GAME RULES
Natural Selection. We have claimed that it is more reasonable to study the emergence of language in a community since natural selection played an essential role in the acquisition of new ability. Twins, for instance, sometimes can develop cryptophasia, which is language no one can understand except the twins. However, such a language is an incomplete work and will be eliminated by nature. In the community, there are M agents: some of them are liars and the rest are truth tellers. In each tournament, each agent will play the game with all other agents in the community. Natural selection occurs after every K tournaments. To be specific, natural elimination is reflected by that the agent with the lowest profit will be weeded out from the community. As for genetic evolution, a new agent that evolved from the elite will fill the opening. The new agent will be trained to learn its own policy
3

---

Under review as a conference paper at ICLR 2021
d
< a
Excoding het
3 @ctellynn Net
Enoosing
hv
14
A'
Prooosal
8/.
Themltadice
hf
1f'
1
Terristion Na
Figure Network architecture.
by playing with the remaining agents, but it is initialized by the parameters of the agent with the highest profit. At the meantime, the remaining agents are also allowed to be trained for the purpose of adapting the new agent.
Every truth teller has a credit record for each liar. At the beginning, truth tellers consider all other agents are trustful, and credit records are initialized as zero. The credit record will be altered every time after a truth teller plays with a liar. To illustrate, truth teller j plays a game with liar i and gets reward r², then the credit record of for i will add a value of - exp(0.5 rr) In this setting, a truth teller cannot tolerate that the profit is less than half of the maximum, and it will realize that it might be cheated since cooperation can definitely achieve more. Also, it is much easier for a liar to lose trust than acquiring We want reflect the reality that one lie can destroy a person no matter how much truth he told before. In addition, the liar can observe its credit records from truth tellers when making decisions.
Credit Mechanism.
2.3 NETWORK ARCHITECTURE AND TRAINING
At each round t, suppose it is the turn for agent to make a proposal, first, agent obtains a partial observation oj. which consists of four parts: market demand d, its own utility u', proposal pl_ made in the previous round t - by its opponent j. and ID of j. An encoding network (-)) which realized by a two-layer feedforward neural network, maps the observation 0j to an encoded vector hi. Then, termination network Tm (-), which is realized by single-layer feedforward neural network, takes hü as input and outputs a binary action of that determines whether agent accepts the proposal p- If it accepts, the game ends and both agents receive their rewards based on the agreement; otherwise, agent needs give an alternative proposal. Truth teller and liar are different in generating a proposal. If agent i is a liar, it sends h; into a local lying network The (-) to output st that indicates whether to tell lies for each item. Currently, the local lying network has a separate two-layer feedforward neural network for each item, however, it can also be instantiated by single network that outputs decisions for all the items. Next, agent feeds h and st into a proposal network Tprop()) (-) to produce the proposal pi. The proposal network also has separate feedforward neural network for each item n, which outputs a distribution over (0, ...; 9 The proposal is sampled from the distributions. If agent is truth teller, st is set 0 by default. After that, it is the turn for agent to move and it will repeat the procedure. Note that when the game ends, the liar determines whether to make deception about each item according to the output of its local lying network. The network architecture illustrated in Figure 2.
When introducing credit mechanism, liar i additionally has global lying network The that takes inputs as credit records of and ID of its opponent j. The global lying network acts like gate for local lying network Tie and outputs binary variable f at the beginning of each game that determines whether to tell lies at the subsequent bargaining round. Unlike Tfe that controls the agent at each round and focuses on how to achieve more profits in just aims to deceive the credit mechanism of truth tellers in order to get more profits in a long run. considers questions follow: what would happen if it lies in one game? would others realize that it is a liar?
game, ThE takes the whole picture and
During training, there three kinds of policies, Teems Tprops and Tie (if existed) be updated for each agent, which are parameterized by 6 frums 0prop and tie respectively. For each game, the policies are updated towards maximizing its own expected reward. Suppose agent plays with agent
4

---

Under review as a conference paper at ICLR 2021
the objective of agent i
J(O,:) = (
where 0z: 10 Orjma mig π² {Tiemm ppopp mie. and (T) the reward that agent receives from trajectory т, playing with agent j. The trajectory T of one bargaining game is defined as (or, {Ut, 8t, Pt J: 11 r (Williams, 1992) and can be further derived as
Then, the gradient of the policies is computed by REINFORCE
T Σ
Vo [(xx:) =B and (T)(T)
[(Vo
log Fiem ( (vilo) (oi)
Vo log tie (6) (/)) + Ve
log m (piloi, s()) + AVo,. H(
*jog
where are hyper-parameters and H. is the entropy regularizer to encourage exploration. As for Tim. the policy gradient computed as
K
M Σ
Ve
J(0 sin.
Ve-t
log
EiE
k=1
where G) is the return of agent playing with agent j started from tournament after previous natural selection and Gj = Ei Rj,
EXPERIMENTS
3.1
CAN LANGUAGE SURVIVE WITH NO RESTRICTIONS IN ONE COMMUNITY?
Experimental setting. In this experiment, are investigating what would happen when there is nothing to counter lying behaviors in the community. There are 8 agents living in a community and trying to make profits by playing bargaining games with others. Half of them are truth tellers and they do not have credit records for liars, which means they are unable to make any response to lying behaviors. The rest are liars which possess local lying network but not the global one. agent has to play game with everyone else in each tournament. Every 10 tournaments, the community updated according to natural selection. We have two training phases. First we train 4 truth tellers for 200k episodes. In each episode, two random truth tellers are sampled to play the game. After all truth tellers have learned how to infer intention of others and make proper proposals, we begin to train liars for 200k episodes by sampling two random agents from the community to play the game in each episode. Note that at least one sampled agent is a liar and the models truth tellers would be frozen in this phase. Each episode corresponds to batch of 128 games.
For game settings, we have N = different items. Market demand (dn \ and items in possession {gm J.W are set 10. Utilities {u,, J.W are sampled uniformly between 0 and 5 for each agent. For each game, Timxx is uniformly sampled from to 10.
Results and analysis. Figure 3a shows the learning curves of all truth tellers in terms of reward in the first training phase. And we can find out truth tellers converge to a similar level (around 0.64), demonstrating that they have learnt make a fair proposal that can be accepted by both party. To some extent, their proposal shows cooperation and is a win-win strategy since they can obtain more
024
126 340 124
1.5 250
17. 25
(a) truth teller vs. truth teller
(b) lia vs. truth teller
(c) liar uw. Har
Figure 3: Learning curves terms of rewards different games.
5

---

Under review as a conference paper at ICLR 2021
Hodon US
Haden DE BINE
Hodee
Lyke Anture
Hobee Vniitee
Prosued 2013
Proposal (ull)
toa 10
(1A In
(a) truth teller vs. truth teller
(b) truth teller vs. liar
Figure 4: Examples of bargaining between agents
than 0.5 (i.e., the profit achieved by dividing items equally). Figure 3b and 3c show the learning curves of all agents in terms of reward in the second training phase. Figure 3b presents the games between liar and truth teller, and Figure 3c presents the games between liars. From Figure 3b, 4 liars perform better in the game than truth tellers and reach above 0.75, while truth tellers only get around 0.45 that is lower than the equally dividing strategy. It shows lying behaviors indeed help liars to take a dominant position over truth tellers. The performance of liars showed in Figure is between the win-win strategy and equally dividing strategy. It turns out there exits a liar that outperforms all other liars when comes to liar vs. liar games. Figure 4 visualizes two examples of bargaining using learned policies between truth tellers and between truth teller and liar.
We further analyze the change of the community un- der natural selection. Figure 5 shows natural se-
Agent Ageess
Aganta
4 Agents Agento
Agent
Agents
lection goes, truth tellers are gradually wiped out from the community and eventually liars take over the whole community. Liars are able to obtain more profits by bargaining with truth tellers that they can fill the reward gap caused by liar vs. liar games. When liars are the only kind in the community, their average rewards begin to decline since they are un- able to squeeze the profit from truth tellers. The results suggest cooperation is finally impossible as long as lying behavior is possible in the community. Liars would never try to cooperate with others, and they undermine the motivation of communication (lower and lower rewards). As many hypothe- ses state cooperation is prerequisite for the emergence of language (Cao et al., 2018; Nowak & Krakauer, 1999), we believe more sophisticated language is hard to evolve in such a community. Our finding also corroborates the hypothesis that if it is better to lie than to tell truth, why is all this elaborate coding of thoughts into speech necessary whereas the effective strategy is just not to listen (Ulbaek, 1998), as evidenced by that some liars get nearly the same as equally dividing and hence communication is not necessary, not to mention it incurs extra cost in reality.
M5.1
Las
1003
c5o Laso
Jsw
N53
Les
Figure S: Evolution of the community. The num- ber the average reward during a natural selec- tion period.
3.2 How INDIVIDUAL RESISTANCE TO LYING AFFECTS THE EMERGENCE OF LANGUAGE?
Experimental setting. In the second experiment, we turn to the scenario where truth tellers would counter lying behaviors on their own by refusing to bargain with agents they consider liars. We are investigating that whether individual countermeasure is able to suppress lying behaviors and is a necessity for language emerge. In this settings, we still have 4 truth tellers and 4 liars in the commu- nity. For truth tellers, they have credit records for all the liars. They would not play following games with those whose credit is negative until next natural selection. And both parties get reward 0 for the games where one party is absent. Credit mechanism is designed to help truth tellers to discern liars and punish them within their own ability. However, it is still possible for one agent to get low reward even if no one lies due to the random utilities. To reduce such re- ward variance, truth tellers update credit records based they have the global lying network apart from local one, which gives them opportunities of seek- ing strategies to deceive credit mechanism and keep their advantages. In addition, the global lying
Tum
Figure Learning curves in terms of rewards in games of liar vs. truth teller.
batch of games (the size is 16). For liars,

---

Under review as a conference paper at ICLR 2021
network is trained for 3k episodes with batch size of 16. As for other networks, use pre- trained models from the first experiment. The game settings are the same with the first experiment.
Results and analysis. Figure 6 illustrates example of learning curves of liar truth teller in terms of average reward in one natural selection period. We can see that the liar converges to about 0.67 and the truth teller is able obtain above-average reward of 0.57. Comparing to the first experiment, the difference of rewards between truth teller and liar narrows down. This is the consequence of the compromise between the credit mechanism and global lying network. It turns out the global lying network assists the liar to deceive the credit mech- anism in some way since it can achieve more than that it always tells truth (around 0.64). However, the liar still needs to sacrifice some benefits in order to gain enough trusts from its opponent. As observed in the experiment, the liar has learned an effective strategy to accomplish the mission. Figure 7 shows the lying frequency in each tournament during the natural selection period. We can find out that in the first 4 tournaments, the lying frequency is relatively low since lirrs need to earn trust so that truth tellers would not identity them and refuse to play games at an early stage. With the accumulation of enough trust, liars start tell lies more often to make more profits. As the natural selection approaches, there is less and less need to pretend for liars since the punishment imposed by truth tellers is meaningless at the end of natural selection period. This explains that the lying frequency gradually increases and is relatively high after 4 tournaments.
Figure 7:
Lying frequency.
The competition becomes more fierce since the gap of average rewards between agents is closer than in the previous experiment. However, similar phe- nomenon in natural selection can be observed in Fig-
Aguess
Ageets agerts Agonts Ageets Agents
Agont?
Agulta
M50
ure 8, Le., liars gradually taking over the community, though the process is slower than the previous ex-
periment. The process is slower, because the truth
teller is not always ranked at the bottom of the list in terms of rewards. However, as weaker liars are replaced by the elite liar by natural selection, truth tellers progressively decline to the bottom and are eventually wiped out from the community. Once all agents in the community are liars, similar conclusion for the emergence of language can be drawn as the first experiment. In addition, it is indicated that the credit mechanism effective to protect truth tellers to some extent, but liars can still find their way to overcome it. Furthermore, individuals are unable to impose enough force to suppress lying behaviors. Therefore, we claim that is not easy to re- sist lying behaviors relying solely on individuals and more compelling force is needed to create harmonious cooperative environment.
65.4
M55
Figure 8: Evolution of the community with indi- vidual resistance to lying behaviors, where each grid is formed by the results of three runs and hence fraction appears.
3.3 How SOCIAL RESISTANCE TO LYING AFFECTS THE EMERGENCE OF LANGUAGE?
Experimental setting. In the final experiment, we seek for more severe punishment to resist lying behaviors. It reminds us that so- cial organization, e.g., enterprise or government, would penalize anyone who breaks rules, e.g., breach of contract, by imposing a fine or putting into prison. Inspire by this, we introduce two kinds of artificial pressure on liars to counterbalance their lying behav- iors, mimicking two types of aforementioned punishment. One is designed for local lying mechanism, and a penalty a x |lies] is di- rectly added to reward function to disfavour lying behaviors. a is a hyperparameter which is considered as the punishment strength in reality and |lies| is the number of items a liar tells lies about in one game. Another is for global lying mechanism, every time the gap of rewards between two agents exceeds over a threshold (i.e., 0.2).
Figure Learning curves in terms of lying frequency of dif- ferent a.
the system believes lying behavior has occurred and the liar will be banned for next I Txprison games
7

---

Under review as a conference paper at ICLR 2021
1.00
6.4
300
154
175
150
140
( truth teller
Figure Learning curves truth teller and liar terms of rewards with different a.
Figure 11: Lying frequency of dif- ferent Lyeison.
with the same player as punishment. However, in order not damage the interest of truth tellers, the system will enforce equally dividing for canceled games and both sides get reward 0.5 instead. As for training procedure and game settings, they are the same as the first experiment.
Results and analysis. Figure 9 shows the learning curves different a in terms of lying frequency. We find out that when is under the threshold of 0.02, it has little influence resisting lying behaviors. When the value of a exceeds 0.02 and continues to increase (i.e., more and more fines), the lying frequency gradually decreases zero. And the rewards of both truth tellers and liars converges to a similar level, showing they begin to cooperate, as illustrated in Figure 10. In addition, similar pattern also observed in Figure 11. If the prison time Lpiison is long enough, the global lying network tends to always output zero. It shows liars would turn into truth tellers as long as the system exerts sufficiently severe punishment.
In this experiment, our results show that artificial penalties inspired by existing social rules can restrain lying behaviors fundamentally compared with individual forces. With the help of strict social pressure, agents in the community are willing to cooperate with each other honestly. This is the premise of hypothesis that cooperation boosts language to emerge (Nowak & Krakauer, 1999; Cao et al., 2018). Only under such environment, language is finally possible.
4 DISCUSSION
We found that language is impossible be preserved when some agents are entitled lying ability in community with natural selection. When two agents solve a simple non-cooperative task in bargaining way, agents which are not prepared for lying behaviors easily fall into a very unfavor- able situation. And most of the agents in the community adapt to lying during a long evolutionary process based on our simulations, leading to that cooperation is no longer a good option and there no intention to communicate. This motivates us to unearth what preserves language from lying behaviors. According to existing human social behaviors and rules, we design the credit mechanism and artificial penalties, reflecting individual and social resistance lying behaviors, in reality, to in- vestigate how they affect the emergence of language. Follow-up experiments suggest the limitations of individual resistance. Individuals are unable obtain full information determine whether their opponents are lying and hence this is leading to more conservative sanctions that do not effectively suppress lying behaviors. On the contrary, the social system has the right to access all. Thus, it can impose more accurate and severe punishment directly to lying behavior and retain a system with obligatory reciprocal altruism, which provides the soil for the evolution of language. Based on the results above, we hypothesize that social resistance is the key to preserving the language other than individual resistance. Besides, each agent is realized by neural networks. The results are stable across hyper-yaramene parameters and different kinds of bargaining games since we also try settings in (Cao al., 2018).
From the perspective of artificial intelligence, our results stress the importance of controlling the external environment language emergence. It is well known that enabling intelligent agents to communicate with humans is always one of the ultimate goals for the AI community. If we want to achieve this, is not enough just to teach agents about human language. Unlike the unemotional machine, the human lying and playing jokes. We want to emphasize that is impossible to evolve language if agents are unaware of such behaviors of human beings and existing language can be vulnerable facing them.
convey meaning different from the literal one, reflected in acts such as
8

---

Under review as a conference paper at ICLR 2021
Creating environment conducive cooperation seems particularly significant. Also, present two proof-of-concepl examples of how to maintain a favorable environment for the emergence of language by directly penalizing lying behaviors in reward functions or banning liars from games. In future work, we would look for less ad hoc ways for providing good soil for language to thrive and investigate how agents develop such resisting mechanism naturally in more precise way.
REFERENCES
Jacob Andreas and Dan Klein. Analogs of linguistic structure in deep representations. In EMNLP, 2017.
Ken Binmore, Ariel Rubinstein, and Asher Wolinsky. The nash bargaining solution in economic modelling. The RAND Journal of Economics, pp. 176-188, 1986.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In ICLR, 2018.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient en- coding emergent communication. NeurIPS, 2019.
Morten H Christiansen and Simon Kirby. Language evolution: Consensus and controversies. Trends in cognilive sciences, 7(7):300-307, 2003.
Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Co-evolution language and agents in referential games. arXiv preprint arXiv:2001.0333 003366661,000,,, 2020.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. In CVPR, 2017.
Lewis David. Convention: philosophical study, 1969.
lain Davidson. The archaeological evidence language origins: States of art. STUDIES IN THE EVOLUTION OF LANGUAGE, 3:140-157, 2003.
Terrence W Deacon. Universal grammar and semiotic constraints. STUDIES IN THE EVOLUTION OF LANGUAGE, 3:111-139, 2003.
Terrence William Deacon. The symbolic species: The co-evolution of language and the brain. Number 202. WW Norton Company, 1998.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in multi-modal, multi-step referential game. In ICLR, 2018.
T Givon. Function, structure and language acquisition. The crosslinguistic study of language acqui- sition, 2:1005-1028, 2013.
Talmy Givón and Bertram Malle. The evolution of language out of pre-language, volume 53, John Benjamins Publishing, 2002.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge' naturally' in multi-agent dialog. In EMNLP, 2017.
Angeliki Lazaridou and Alex Peysakhovich. Multi-agent cooperation and the emergence of (natural) language. ICLR, 2017.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In ICLR, 2018
Charles N Li and Fram-Marie Hombert. On the evolutionary origin of language. Advances in Con- sciousness Research, 42:175-206, 2002.

---

Under review as a conference paper at ICLR 2021
Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent commu- nication. In NeurIPS, 2019.
Jay Meddin. Chimpanzees, symbols, and the reflective self. Social Psychology Quarterly, pp. 99- 109, 1979.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAAI, 2018.
2018.
John Nash. Non-cooperative games. Annals of mathematics, pp. 286-295, 1951.
John F Nash Jr. The bargaining problem. Econometrica: Journal of the econometric society, pp- 155-162, 1950.
Martin A Nowak and David C Krakauer. The evolution of language. Proceedings of the National Academy of Sciences, 96(14):8028 8033, 1999.
Thomas Schelling. The strategy of conflict. prospectus for a reorientation of game theory. Journal of Conflict Resolution, 2(3):203-264, 1958.
Robert L Trivers. The evolution of reciprocal altruism. The Quarterly review of biology, 46(1): 35-57, 1971.
lb Ulbaek. The origin of language and cognition. Approaches to the Evolution of Language, pp- 30-43, 1998.
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior (commemo- rative edition). Princeton university press, 2007.
Ronald J Williams. Simple statistical folliin algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255: 1992.
A IMPLEMENTATION DETAILS
Embedding sizes, as well all neural network hidden states, have dimension 100. And we ADAM optimizer (Kingma & Ba, 2014) with default settings to optimize the model of each agent. For entropy regularisation hyperparameters X, we have different value for different policies, i..., 0.05 for Ter.- 0.005 for Tpoop and 0.05 for In addition, during two phase training, we use an exponentially smoothed mean baseline for reducing reward variance as (Cao al., 2018), which is defined as bnew 0.7 botd 0.3 R. As for the training phase of genetic evolution, we train agents with 200k episodes. In more details, if the new agent evolved from truth teller, training happens in all scenarios randomly. Otherwise, training happens in liar vs. liar and truth teller liar. And agents are selected randomly according to different scenarios.
An encoding network (.).) is realized by two-layer feedforward neural network. Termination network (-) is realized by single-layer feedforward neural network followed by a sigmoid function. The local lying network T-i(() has separate two-layer feedforward neural network for each item and each followed by a sigmoid function, however, it can also be instantiated by a single network that outputs decisions for all the items. And the global lying network TLEE() also has two- layer feedforward neural network followed by a sigmoid function. As for π prop- it is parametrised by 3 separate single-layer feedforward neural networks followed by a softmax function.
10


Reviews:
[{"review_id": "a0iw7YBEPt1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Overall review\n\nThis paper attempts to address a question in the emergent communication literature: what preserves / maintains the stability of emerged communication protocols.  The authors manipulate the prevalence of lying behavior in a community of agents playing a variant of a Nash bargaining game.  The main take-away is that explicit punishment, from the environment and from truth-tellers not wanting to communicate with liars, can prevent the spread of exploitative lying behavior in the community.\n\nMy main worry about the paper is that the conceptual motivation is a bit unclear.  The authors present the paper as addressing when emergent communication can be stable.  And, after identifying a condition under which a penalty for lying removes liars from the population, they conclude that \"Only under such a environment, language is finally possible.\"  But in all experiments, the communication protocol is fixed: mappings from market needs to proposals.  In other words, what changes is less the communication protocol itself, and more the prevalence of lying / truth-telling.  It thus is not clear how these results are to be interpreted as being about the emergence of communication itself, and not alternative uses of a pre-existing protocol.\n\nPros:\n* Addresses a foundational question in emergent communication.\n* Explicitly models a community of agents, instead of just dyads.\n* Identifies explicit factors influencing their outcome of interest.\n\nCons:\n* Conceptual motivation a bit unclear (see above).\n* Communication protocol is fixed, not emergent.\n* Many interesting results / sub-experiments are presented as parentheticals / foot-notes, without full details in appendices.\n\n\n# Small comments\n\n* p 3, choice to not do simultaneous proposals.  More details about these experiments should probably be provided in an appendix.\n\n* Natural selection mechanism: the authors remove the lowest-performing single agent.  This is relatively different from the standard interpretation, which is that the number of offspring in the next generation is proportional to an agent's fitness.  Did the authors experiment with that approach, or can they say more to motivate theirs?\n\n* The y-axis in the main results are \"average scaled reward\".  How exactly is the scaling done?  This would help in interpreting the results, as would comparison with a random baseline and/or an optimal strategy.\n\n* \"As the natural selection approaches, there is less and less need to pretend for liars ...\"  I had some difficulty following the reasoning here.  Are the 10 tournaments _before_ a single episode of natural selection?  Or did natural selection occur in the middle?  If the former, it would help to know if these are averaged across \"generations\" in order to support the authors' interpretation.\n\n\n# Minor typographic comments\n\n* abstract: \"From the perspective of Darwinian, ...\" needs to be finished after \"Darwinian\".  Same at the end of page 1.\n\n* p 1, \"some intrigue properties\" should be \"some intriguing properties\"\n\n* p 2, \"lairs\" should be \"liars\"; this one recurs a few times in the text.\n\n* p 8, \"artificial intelligent\" should be \"artificial intelligence\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "RstOOu5fQ4n", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary ##### \nThis paper investigates conditions under which communities of cooperative agents are stable. Communities in multi-round bargaining games with evolutionary dynamics are evaluated in three main setups. The first imposes no restrictions on the agents' behavior and is shown to be easily invaded by deceitful agents. The second enables agents to refuse to bargain with deceitful agents. Nevertheless, such communities are shown to be invadable. Finally, in the third setup, a global punishment system is shown to be able to drive out deceitful invaders. The main take-home message is that, when lying is an option, agents(' communities) need to be prepared for it.  \n\n##### Reasons for score #####\nI vote for rejecting this submission. The main reason, further detailed in \"Cons\" below, is that I am not convinced that the problem this manuscript addresses is not an artifact of the rather strong assumption of selfish individuals and communities driven solely by functional pressure. I agree with the authors that we need to start looking at more naturalistic setups (e.g., communities instead of two-player games). However, this also relates to the evolutionary dynamics we take into consideration, and to a careful motivation of the setups we analyze.     \n\n##### Pros #####\n\n+ Language emergence is a topic that has drawn renewed interest from multiple disciplines. Studying the conditions under which agents' acquired behaviors are (not) stable in a dynamic setting is important to understand their underpinning.\n+ The setups are well explained and easy to follow   \n\n##### Cons ######\n- Much of the motivation of this paper relies on the argument that there are strategies that functionally dominate cooperation. However, the view that functional pressure is all there is to evolution is rather outdated. If the authors wish to stick to this line of argumentation, I encourage them to give strong arguments for why we should disregard other factors, such as transmittability/acquisition (e.g., Kirby et al. 2015, \"Compression and Communication in the Cultural Evolution of Linguistic Structure\", or Brochhagen et al. 2018, \"Coevolution of Lexical Meaning and Pragmatic Use\") or neutral models (e.g., Reali & Griffiths 2011, \"Words as alleles: connecting language evolution with Bayesian learners to models of genetic drift.\", or Perfors & Navarro 2014, \"Language Evolution Can Be Shaped by the Structure of the World\"). Along the same lines, the view that functional pressure needs to apply to only individuals, and that these are fully selfish, also requires evidence or needs to be clearly marked as a rather strong assumption. The claim that \"[...] in order to ensure the survival advantage of many competitors, animals are selfish in nature\" flies in the face of the whole line of research on animal alarm calls (e.g., Zuberbuehler 2009, \"Survivor Signals: The Biology and Psychology of Animal Alarm Calling\"); and, for that matter, the existence of any social animal. In sum, as it stands, I am not convinced that the problem this manuscript addresses is not an artifact of the assumption of selfish individuals and communities driven solely by functional pressure.\n- Terminology is very sloppy at times. What is \"successful evolution\" (p. 1) or \"naive natural selection\" (p.8)?\n- Writing could be greatly improved. There are a lot of typos and borderline grammatical sentences\n- References are wrong (e.g., Lewis' work on convention is not from 2008 but from 1969. The same goes for all the references on seminal work in game theory)\n- Many assumptions in the setup are not motivated. For instance, is there a reason to use a discrete dynamic over a continous one? (p. 3) Why use a multi-round bargaining game to model language evolution by contrast to, say, an established signaling game-style setup with conflict of interests? How is the \"credit mechanism\" and the \"market\" motivated/grounded in light of this being an investigation about language emergence? (p. 3)\n- The lines in Figure 3a and Figure 3b are very hard/impossible to read. Particularly Figure 3b.\n- The y-axis in the plots in Figure 3 are not aligned, rendering a comparison across conditions hard (and visually misleading)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BL7EiOyDE0x", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** This paper studies multi-agent communication with an aim to mimic conditions for the emergence of language in society. They argue that the emergence of language requires co-operation. However, people can cheat and make a profit by lying, and eventually, people have no incentive to co-operate. \n\nA proxy multi-agent setting is created to study this which proceeds in a series of interaction between two (randomly sampled) agents. In each interaction, there is a market demand for N items, and both agents have a fixed quantity of these items and their own hidden utility. An agent makes a proposal to another agent for a set of goods which consists of quantities of each item that they are willing to give. The other agent has to fulfil the remaining quantity to match the market demand. The other agent can accept the proposal at which point both agents get reward based on the remaining items and their respective utilities. Otherwise, the bargaining will continue and the other agent will make a proposal. There is a timeout after which both get a reward of 0.  There are certain agents who are chosen as \"liars\" who can make a proposal but refused to follow through leading to the remaining market gap being split equally.  Lying can, therefore, result in profit. A credit mechanism is also introduced to measure the performance of agents. Liars can use credit scores to decide whether to lie or not. A neural network is used to generate proposals and REINFORCE style training is performed to optimize reward. Authors also study \"natural selection\" where low-ranking agents get eliminated after a certain time, and \"genetic evolution\" where high-ranking agents produce clones.\n\nThree different settings are studied: (i) where there are no measures taken to stop liars, (ii) where each truth-teller can refuse to play with an agent with negative credit score, (iii) where global measures are taken including reward penalty for lying or liar agents being banned for a fixed time. It is found that in both (i) and (ii); liars take over society. Individual counter-measures slow down the proliferation of liars but don't stop them. However, global measures in (iii) are able to stop liars. Authors argue that global measures in human society (reciprocal altruism) is the reason why conditions were right for language to evolve.\n\n**Strength:**\n- The setting is interesting. In general, understanding the dynamics of language evolution is a very interesting topic. This can shed light on human history, and maybe potentially be helpful for developing language understanding agents. \n\n**Weakness:**\n- The language part of the story is confusing since there is no language used in the experiments. In fact, what is really being asked here is if we can have a society without cooperation? Since, if everyone ends up being a liar then we don't have a society. Therefore, the language part feels like a red-herring. \n\n- Implications of the results are unclear as it is not obvious that the setup captures all the important nuances present in human society. E.g., in a real-world, people who are speaking the truth can help each other by sharing technology that can give them a survivor edge over liars. This happens even if liars are not directly punished for their deeds. Then there is the question if the way natural selection and genetic evolution are modelled, reflect the real-world. I am not an expert on evolution but the paper doesn't say enough to justify the choice.\n\n**Questions:**\n\n- In the case where agent \"i\" lies, the other agent will still end up paying its share of the accepted proposal plus 50% of what agent \"i\" promised. Is this right? \n\n- Can one pay more items than they possess? If not, do we always have $d_n \\ge q^i_n + q^j_n$ for every $i$ and $j$, as otherwise there are not enough items to satisfy the market?\n\n- In the setting of individual resistance, once an agent ends up with a negative credit score then the truth-tellers will refuse to play with them resulting in an addition of $1 - \\sqrt{e} < 0$ to their credit score. So can an agent never increase their total reward once their credit score becomes negative? Or, perhaps liars will still play with them and so there is a chance for their total reward to increase.\n\n- How would the behaviour be if the agent also had access to (i) total reward earned by the other agent, (ii) percentage of cases the other agent lied. E.g., if an agent saw that the other agent lie then they are likely to either not play with them or demand a more lucrative offer before committing. This can allow for more interesting policies to evolve.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a study on what maintains the stability of emerged communication protocols. To study this question the authors design experiments in bargaining communities of agents in 3 setups,  a) no punishment of restriction of liar agents b) allowing individual agents to refuse bargaining with  liar agents and c) introducing a global punishment system for liar agents.

Overall the reviewers agree that the design of the study is interesting, but also point that motivation and take-home messages of this study are unclear. Having read the paper, I share the same opinion. The authors discuss on a very abstract level about the implications of this study for the field of AI, but this study is quite specific and clearly does not capture all the complexities or real societies. From the scale of results and study, I think it would be more valuable to draw some concrete proposals/implications about perhaps multi-agent modelling or environment design in general. 

All in all, this is an interesting study but some more work needs to be done around research framing.

Decision: Reject