Selfish Emergent Communication
Authors: Michael Noukhovitch
Travis LaCroix
Aaron Courville
Venue:  (2020)
Keywords: multi agent reinforcement learning
emergent communication
game theory

Abstract: Current literature in machine learning holds that unaligned, self-interested agents do not learn to use an emergent communication channel. We introduce a new sender-receiver game to study emergent communication for this spectrum of partially-competitive scenarios and put special care into evaluation. We find that communication can indeed emerge in partially-competitive scenarios, and we discover three things that are tied to improving it. First, that selfish communication is proportional to cooperation, and it naturally occurs for situations that are more cooperative than competitive. Second, that stability and performance are improved by using LOLA (Foerster et al, 2018), especially in more competitive scenarios. And third, that discrete protocols lend themselves better to learning cooperative communication than continuous ones. 

Under review as a conference paper at ICLR 2020
SELFISH EMERGENT COMMUNICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Current literature in machine learning holds that unaligned, self-interested agents do not learn to use an emergent communication channel. We introduce new sender-receiver game to study emergent communication for this spectrum purtially-ompettitee scenarios and put special care into evaluation. We find that communication can indeed emerge in partially-competinng cover three things that are tied improving First, that selfish communication is proportional to cooperation, and it naturally occurs for situations that are more co- operative than competitive. Second, that stability and performance are improved by using LOLA (Foerster et al. (2018a), especially in more competitive scenarios. And third, that discrete protocols lend themselves better to learning cooperative communication than continuous ones.
scenarios, and we dis-
INTRODUCTION
The principles involved in the evolution of effective communication are essential for artificial intel- ligence research since they may lead to innovative communication methods for use by interacting AI agents and multi-robot systems. AI agents need a common language coordinate with one another and to communicate successfully with humans [Wagner et al. 2003). The emergence of communi- cation protocols between learning agents has seen surge of interest in recent years, but most work tends to study nullycoooperative agents that share a reward (Foerster et al. 2016 Havrylov & Titov 2017 Lazaridou et al.) 2016). Work on selfish agents, that separately optimise their own reward, has been limited, with results suggesting that selfish agents do not learn to use communication channel effectively (Cao et al., 2018 Jaques et al., 2018). This has contributed to a perspective in the field that emergent communication is a purely cooperative pursuit Lanctot et al. (2017). This is in contrast to theoretical results in game theory that show it is possible to use an existing cheap-talk
communication channel effectively to resolve situations of partial conflict (Farrell & Rabin 1996).
We aim to reconcile these different findings and establish the degree of cooperation necessary for useful communication to emerge.
To study this problem in detail, we look at the simplest case of communication: a sender-receiver game (Lewis) 1969). This is a game of incomplete information, wherein a sender obtains private knowledge and communicates this to receiver via a signal, or message. The receiver uses the message to inform its action the environment. Though messages are arbitrary, and initially mean- ingless, the players can coordinate upon a conventional meaning for the signal (Skyrms 2010)) While the classic game fully cooperative, we introduce an arbitrary level of conflict between our two players and investigate whether communication can emerge for each level of conflict. Contrary to current literature machine learning, we find evidence that communication emerges in compet- itive scenarios scenarios-provided that the agents' interests are at least partially aligned. We further find that using LOLA (Foerster et al. 2018a) an effective strategy for resolving social dilemmas to explicitly model opponent learning yields more effective communication protocols. Finally, we sider the difference between continuous and discrete emergent communication and find that discrete communication lends itself better cooperative communication.

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
2.1 FULLY COMPETITIVE/COOERATII MARL
Multi-agent RL is used for many fully-competitive, zero-sum games (Silver et al. 2017 Vinyals ct al. 2019 OpenAT 2018) with the objective of finding a single, best learned agent (or team) at Test time. In these games, "best" agent is one that can outplay all opponents (Balduzzi et al. 2018). We cannot evaluate its play by looking at the reward that agent achieves against given opponent because the reward received will be function of the choice of opponent In contrast, MARL in fully cooperative games has usually been able to make the assumption that we get to pick our team, and so can use self-play to try and achieve maximal possible reward. (Foerster et al. 20188). In this way, cooperccoeee MARL compares the maximum performance between learning algorithms and architectures by optimising the joint reward of the team.
2.2 GENERAL- SUM MARL
We investigate the space of partially competitive games known as general-sum games, where there is some amount of common interest and some amount of conflict. In this case, care must be taken in defining the "best" agent: it not necessarily the agent that does as well or better than all opponents because, to achieve the highest possible reward in general-sum game, agents might have to coop- erate to some extent (Leibo et al. 2017). For example, consider an agent playing iterated prisoner's dilemma. The agent that always defects will never have a reward worse than its opponent, but, when playing against a tit-for-tat agent, will also not achieve the total reward of an agent that is also playing tit-for-tat strategy.
Since the maximum expected reward may only possible by cooperating, agents must learn how to coordinate with each other. This can be done by training together or learning to understand opponent intentions at test time by observing their actions. The latter allows for ad-hoc comparison of learned agents at test time; the former requires comparing learning algorithms trained together. The latter should then require a sequential/iterative game, so there is time to infer opponent intentions before acting (Fujimoto & Kaneko 2019). It may also require meta-learning other modifications to un- derstand and adapt to these intentions as it seems current self-play methods are insufficient to adapt ad-hoc play even against different versions of their own architecture (Bard et al. 2019). Work in general-sum MARL has mostly worked on analysing learning algorithms'' ability to cooperate and resolve social dilemmas (Foerster et al. 201 Sa Letcher et al. 2019
Lerer & Peysakhovich 2017)
2.3 EMERGENT COMMUNICATION
To date, investigations of emergent communication have remained mostly in the realm of fully- cooperative games (Lazaridou et al. 2018; Das et al. 2019 Evtimova et al., (017). For contin- uous communication, Singh al. (2018) claimed to learn mixed cooperative-competttve see- narios. However, their setup uses parameter sharing between opponents; their "mixed" case is non-competitive (and implicitly cooperative); their competitive game is actually two stages one fully cooperative and one fully competitive; and, their results in the competitive scenario are simply to mask out all communication. Previous attempts to learn discrete emergent protocols by selfish agents in competitive games have failed (Cao et al. 2018) unless additional, more complex learning rules are adopted (Jaques et al., (018).. In the latter case, they compare learning algorithms trained together as opposed to learned agents at test time to avoid a significant issue of comparing emer- gent communication agents- i.c., different protocols. Since an emergent protocol only has meaning between the agents that learned it together, comparing two learned agents at test time would require that they infer each others' protocols without training with them. This seems impossible, and it is more reasonable to frame it as meta-learning problem where agents can have a brief adaptation period to synchronise their protocols. Meta-learning is beyond the scope of this paper, so we follow Jaques et al. (2018) in comparing learning algorithms trained together. We also take a more prin- cipled approach than previous work by guaranteeing no communication through the action space,
For multi-agent RL, we must define the space of possible agents that we play against because our aim is not merely to beat one opponent but all potential opponents e.g., in non-transitive zero-sum games (Balduzzi et al. (019). Following the "Al agenda" Shoham et al. (2003), we believe it is reasonable to consider space of leamnngg agents similar to our own that are trained with gradient descent.
2

---

Under review as a conference paper at ICLR 2020
using more rigorous, quantitative criteria for evaluating communication, and precisely setting the levels of competitiveness. Notably, this work does not propose new architectures or learning rules but aims to take critical look at existing beliefs and draw important distinctions as Kottur et al. (217)) did for natural language emergence.
2.4 SENDER GENDER-RECEIVER RECEIVER GAMES
Our experiments use the simple emergent-commuuniation framework known as a sender-receiver game (or "referential game" [Lazaridou et 2016)), which finds extensive use in economics (Riley) 200T] and philosophy (Lewis) 1969, SSyrms) (0TO) among others. In the classic game, the sender given target value to be communicated to the receiver via message. The receiver receives the sender's message and must decode it to predict the target value. Both players are rewarded according to the negative of the receiver's prediction error. In this fully-cooperative setting, players often coordinate a protocol to transfer information as effectively as possible (Skyrms) 2010). The messages between the sender and the receiver can be categorised "cheap talk": messages are costless, non-binding, non-verifiable and may affect the receiver's ' beliefs (Farrell & Rabin 1996).
Our work benefits greatly from Crawford & Sobel (1982) seminal work in classical game theory. They study possible fixed communication equilibria under competition by giving the sender and receiver different targets and creating a conflict of interest. They perform static analysis and prove the existence of a Nash equilibrium where the amount of information communicated is proportional to the alignment between the players' interests; however no informative equilibrium exists when interest diverge too greatly. In contrast, we do not look for existence of an equilibrium but do dynamic analysis and show the feasibility of communication using standard learning rules in RL. We do not explicitly aim for equilibria (Shoham et al., (003) but look at the information transfer of communication protocols in flux (and therefore out of equilibrium). This is more in line with previous work in emergent communication (Jaques et al. (2018) as well as evolutionary signalling (Skrms 2010).
3 THE CIRCULAR, BIASED HENDER-RECEIVER GAME
To investigate a range of competitive scenarios, we introduce a modified sender-receiver game with continuous-bias variable, b, that represents the agents' conflict of interest, ranging from fully cooperative to fully competitive. The two players-the Sender (S) and the Receiver have corresponding targets (T, and T,), which are represented by angles on circle that are b degrees apart: T, = (T, + b) mod 360
The game starts with the sender's target being sampled uniformly from the circle T. Uniform (0, 360). The sender is given its target as input and outputs a message, m S(T.), con- sisting of a single, discrete token from vocabulary m E V. The receiver given the message and outputs scalar action a R(m). The goal of each agent is to make the receiver's 's action as close as possible its own target value. After the receiver acts, both players get a loss between the action and their respective targets, L' = L(a, T,). By using L1 loss between the angle of the target and action L (T,, a) = min[[[. - a),
360 - - -a), it evident that game with b 0 fully coop- erative (LΙ Li) and game with the maximum bias b 180 is fully competitive constant-sum (a generalisation of zero-sum, see Appendix I for proof). All values in-between, E (0%, 180%), represent the spectrum of partially cooperaatie/cooppttiive general-sum games. Figure Ta gives an instance of this game; the game's 's algorithm is given in Algorithm i Appendix [ This can be seen as the game from Crawford & Sobel (1982) modified to cover the range of cooperative/competiive games.
3.1 TRAINING DETAILS
Both agents are implemented as MLPs with two hidden layers and ReLU (Nair & Hinton 2010 nonlinearities between all layers. The targets are sampled from the circle, the sender takes its tar- get, T,, as input and outputs a categorical distribution over a vocabulary from which we sample a message-in its output. The receiver takes the message as input and deterministically outputs its tion, a. Errors are calculated using the L, loss on the circle. The sender estimates its loss using the score function estimator (Ful 2006)- also known as REINFORCE (Williams) 1992 and has
3

---

Under review as a conference paper at ICLR 2020
T
4
action
bias
4
T,
(a) The circular, biased sender-receiver game
Figure 1: Circular Biased Receiver Game has both agents given targets TR, T' that are b apart and choose an action to receive the L1 losses L L
an added entropy regularisation term. Since the loss is differentiable with respect to the receiver, is trained directly with gradient descent, so we are training in the style of a stochastic computation graph (Schulman al. 2013).
We train for 30 epochs of 250 batches, with batch size 64, and set the circumference of our circle to 36 (so that loss of 90 an error 9). Both agents are trained using Adam (Kingma & Ba 2014). To evaluate, we use fixed test set of 100 equidistant points E , 1800) and take the arg max of output distributions instead sampling. We do hyperparameter searches with Orfon (Bouthillier et al. 2019), using random search with fixed budget of (100) searches. We perform hyperparam- eter search to over both agents' learning rates, hidden layer sizes, the vocabulary size, and entropy regularisation (when used). We always report results for given hyperparameters averaged over 5 ran- dom seeds, and we average our metric for hyperparameter search over the last 10 epochs to capture some level of stability well performance. All hyperparameter search spaces available in the config files of the code repository.
4 COMMUNICATION: COOPERATION OR MANIPULATION
4.1 EVALUATING INFORMATION TRANSFER
To evaluate the communication emerged with a cheap-talk channel, we can simply look to the sum of agents' T, losses. Under noncoummunia (or uninformative communication), we know that the receiver will just guess point at random, and the average loss for both players is the expected value of the loss-given that drawn uniformly Ex=U (T,, =]] 90 Therefore, any error for either agent below 90º is evidence of information transfer (Lowe et al. 2019). Furthermore, since there is no other action space for agents to communicate in, the information transfer must be happening in the emergent communication space (Mordatch & Abbeel 2017). Therefore, the lower LT + Li is, the more informative the learned protocol is; and, the most informative protocol will have the lowest loss minge (0.360) L + L b. To show this comparison, we always plot the loss under uninformative communication (909) and the loss for each agent if they were to both fairly split the bias (b/2).
(T,,,)
4.2 INFORMATION TRANSFER VS COMMUNICATION
While we have found evidence of information transfer, does that necessarily mean our agents have learned to communicate? For example, our hyperparameter search could find minimal learning rate for the sender, such that it is essentially static, and a normal configuration for the receiver. The game would then become not one of learning protocol between two agents, but rather just a receiver learning the sender's s initial random mapping of targets to messages. The receiver could then dominate the sender by always choosing a T,, which would yield L₁ + Lí b; namely, the optimal sum of losses and, therefore, optimal information transfer. This situation is clearly not what we are looking for, but it would be permissible, potentially even encouraged, under
4

---

Under review as a conference paper at ICLR 2020
an information-tranttn objective (as measured by the sum of agents' L1 losses). It is, therefore, necessary to delineate the differences in communication; here, we can look to extant results in signalling (Skyrms 2010).
4.3 COOPERATION VS. MANIPULATION
One perspective on information transfer is that of manipulation of receivers by senders (Dawkins) & Krebs 1978) or vice-versa (Hinde) [981]; this manifests as the domination one agent over the other. We note that these situations modelled as cue-reading or sensory manipulation, respec- tively, and are distinct from signalling-is ingly, communication requires both agents to receive net benefit (Krebs & Dawkins 1984). which implies some degree of cooperation (Lewis) 1969). For the cully-cooperative case, previous metrics of joint reward (Lowe al. 2019). or even influence of communication (Jaques et al. 2018),
communication (Barrett & Skyrms 2017). Accord-
sufficient to drive the hyperparameter search. But for competitive scenarios, neither of these can distinguish between manipulation and cooperation (Skymms & Barrett 2018).
Since our focus is on the emergence of cooperative communicationn, we are looking for settings where both agents perform better than either their fully-exploited losses (Li < 6 and 4i 6) or the loss under non-communication (Lf < 90 and I, < 90%). With this goal in mind, we choose the sum of squared losses ((I3)² + (L5)²) as our hyperparameter-sean partially competitive scenario having a common-interes/ loss (180 b), in which both agents are fully cooperative, and a conflict-of-interes loss (b), in which both agents are fully competitive. The sum of L1 losses optimises only for the common interest, whereas La prefers more fair division of the conflict-of intterrstt loss in addition to optimising common interest (see proof in Appendix[A2 We use the L2 metric only on hyperparameter search and keep Lı game's 's loss to maintain constant-sum game for the fully competitive case.
metric. We can view our
5 EMERGING COOPERATIVE SELFISH COMMUNICATION
5.1
COMMUNICATION IS PROPORTIONAL To COOPERATION
We use six equidistant values E [0, 30, 60% 90% 120%, 150"] and for each one, we do a hyper- parameter search to find the lowest achievable Lē + L. We do not usually test b 180 because the game is constant-sum and therefore trivially Li + LĀ = 180%, but for completeness you can see the results of a hyperparameter search with b 180 in Appendix Figure [6] We report our results in Figure 2 and find that agents do learn to cooperatively communicate without any special learning that the performance decreases proportionately the
rules contrary current literature. We bias, meaning the sender less informative with messages, forcing the receiver to be less accurate its own guesses. This matches the theoretical results of Crawfford & Sobel (1982); information transfer with communication is inversely proportional to the conflict of interest. Plots for each b are in shown in Appendix Figure 6 For the curve, we still plot the L₁ losses to maintain consistency and to make clearer the comparison to the no-communication baseline and the optimal information transfer (common interest maximisation).
We find that our results are basically unchanged between the different hyperparameter metrics; relatively fair and useful protocol learned by the agents, but this deteriorates in more competitive scenarios. This is clear when comparing the stability and relative efficacy of protocols in b = 30,, 60", shown in Figures 2b 2c. and that of 6 = 90" shown in Figure 2d We can understand this through the lens of honest communication, which can be taken advantage of in highly competitive scenarios. If. for example, the sender communicates, with complete honesty, its own coordinates, then the receiver can take advantage of this and choose its location exactly so that L, 0 and L. = b. Comparing this situation to non-communication (L, La = 90%), it is clear that even fully-exploited communication is strictly dominant strategy for 90" (i.e, when the game is more cooperative than competitive).
5.2 IMPROVING COMPETITIVE COMMUNICATION WITH LOLA-DICE
For more competitive cases, exllotbell communication is no longer dominant, and active communication now requires both agents to cautiously cooperate. To achieve this cooperation,
5

---

Under review as a conference paper at ICLR 2020
sender
150
80
receiver
Test +
Test L loss
100
60
40
50
no REINFORCE- CeDDRCE-Dettrminisiii
25
50
75 bias
100
125
150
5
10
15 epoch
20
25
30
(a) Best hyperparameters found for each bias
(b) bias 30
sender receiver
100
receiver
100
Test L1 loss
Test L, loss
80
60
60
40
5
10
15
20
25
30
0
5
10
15 epoch
20
25
30
epoch
(c) bias 60
(d) bias 90"
Figure 2: All results are shown in Figure 2a we plot the lowest L + L test loss found with hyperparameter search for E [0, 10.300pp00 -00,0000 120", 150%]. demonstrating that informative com- munication (below the dashed line) is indeed learned by selfish agents. Note that performance even in the fully cooperative For [30", 60% |90] we show the training curve of the best hyperparameters found in 2b 2c 2d We plot the test loss over training epochs and showing the mean and standard deviation over 5 seeds, finding that for b < 90" we find stable and relatively fair communication is naturally learned
is not optimal because the bottleneck discrete communication.
propose using LOLA (Foerster et al. 2018a)-4 learning rule, resembling theory-of-mind, that lows backpropogate through steps of the opponent's learning. LOLA able to emerge cooperative behaviour in an iterated prisoner's 's dilemma, so a prime candidate for resolving game in similar situation. We experiment with LOLA in three configurations--LLLL. sender, LOLA on the receiver, LOLA on both-and do a similar hyperparameter search, with the added search space of the LOLA learning rate. Per the improvements made by FFerster al. (2019), we replace the receiver's s score function estimate with the DiCE estimator, and we backpropogate through exact copies of opponents opposed using opponent modelling. We show our results in Figure Ba with extended plots in Figures & 10 in Appendix
the
We find that LOLA on the sender ineffective, but LOLA on the receiver and on both agents does indeed lead to better performance. This implies that emerging communication in competitive scenarios necessitates cooperation and that this cooperation can found through explicit opponent modelling. Furthermore, comparing the curves of basic agents (Figure [d] with those of LOLA agents (Figure Beb shows that gains performance are not from one agent dominating the other, but from both agents improving and increasing stability. We also look at the performance of n-step LOLA, which backpropogates through n > steps of opponent learning. Figure 3b demonstrates that 2-step LOLA slightly outperforms 1 1-step LOLA, but 3-step LOLA does provide any increase over 2-step. We see from Figure 3d that the increase comes mostly from stability of learning and slight improvement the part of the sender.

---

Under review as a conference paper at ICLR 2020
150
Test L, 44
+ L
Test Li
100
100
REINFORCE Deterministic
REINFORCE-Determinstii SenderLOLA-Determiiste
50
LOLA1-LOLA1
DICE-ReceiverLOLA
50
LOLA2-LOLA2
SenderrLOLA-RecciverrLLL..
LOLA3-LOLA3
25
50
75
100
125
150
bias
25
100
150
bias
(a) Comparing the original sctup, LOLA on the sender, LOLA on the receiver, and LOLA on both
(b) Comparing 1, 2, and 3-step LOLA both agents
100
100
90
90
Test L, loss
80
Test loss
80
60
sender
60
sender
50
receiver
50
receiver
no communication
no communication
40
40
fair split
fair split
10
20
5
10
20
30
epoch
epoch
(c) 1-step LOLA on Sender and Receiver for b 90 (d) 2-step LOOLA on Sender and Receiver for = 90
Figure 3: LOLA improves learning communicate and it is especially visible b 90 where original setup does very poorly. 3b shows that higher step LOLA improves slightly further but not past 2-step. Best feasible communication protocols found for 90 using 1-step (3c) and 2-step LOLA 3d on both agents demonstrates that the gains in performance over the basic setup shown in Figure 2d are not just from one agent doing better (though the sender doing better) but both agents improving in performance and stability. Shaded area is standard deviation over seeds
5.3 DISCRETE VS CONTINUOUS COMMUNICATION
Another axis to consider is whether discrete or continuous communication lends itself better to learning with selfish agents. To compare, we make the sender's message real-valued scalar and ap- propriately change output distribution be a Gaussian, for which it learns the mean and variance (concretely described in Algorithms Appendix B We, again, hyperparameter searches, and we consider training our baseline training with a REINFORCE Sender and deterministic receiver well training both agents with 1 1-step LOLA. Our results in Figure 4a suggest that the learned protocols for continuous communication are all highly informative and near optimal. However, in all cases, the receiver is learning to manipulate the sender, and there is little evidence of cooperative communication. Indeed, we found no cases of both agents having a net benefit (I,,, T.g < 90%) in any of the hyperparameter and we only two cases of net benefit for LOLA-1 agents. Comparing this to discrete communica- tion with the same LOLA-1 agents in Figure 41 we can clearly see that they have preference for more cooperative behaviour. Thus, we find that discrete messages are important component in emerging cooperative elf-interested communication.
for continuous REINFORCFFdetemministi agents past 90".
CONCLUSION
First and foremost, we show evidence against the current notion that selfish agents do not learn communicate, and hope findings encourage more research into communication under

---

Under review as a conference paper at ICLR 2020
125
Test -.
Test &y loss:
Discrete
50
Secrette LOLAL
30
Carkinuzus
8
communicatoo
Communiccoon
Tair spot
farr spil
25
90
100
125
150
Bas
(a) Comparing discrete and continu- communication
(b) Continuous Game for b = 90
(c) Continuous Game for b = 120
80
80
Receiver L
60
Receiver L1
60
glop
40
2ag
40
continuous-LOLA discrete-LOLA
continuous-LOLA
discrete-LOLA
30
40
50
60
70
80
90
50
60
70
80
90
100
Sender
Sender L1
(d) L, vs. L, for all 100 searches for 30
(c) L. vs. Lp for all 100 scarches for b 60
continuous-LOLA discrete-LOLA
80
80
Receiver
Receiver
40
40
20
continuous-LOLA
dIscrete-LOLA
70
75
80
85
90
95
90
100
110
120
Sender
Sender L1
(f) vs. L, for all 100 searches
90
(g) vs. L. for all 100 searches for 120
Figure 4: The comparison between discrete and continuous communication for both the REINFORCE-detrminiticc setup as well as I-step LOLA agents shown in Figure 4a We see that though overall continuous communication can achieve highest information transfer, the gains in performance seem to mostly from manipulation of the sender by the receiver. Two examples are shown for REINFORCE agents in Figures 4b|cc| To find trend, we plot all 100 hyperparame- ter runs for b E [3 13,6,99 12 between continuous and discrete communication using 1-step LOLA agents in Figures munication though individual cooperative points can sometimes be found. general, continuous communication does not lend itself to cooperative communication
We find that manipulation is the common result in continuous com-
competition. We have shown three important properties of communication. First, game being more cooperative than competitive is sufficient to naturally emerge communication. Second, we' ve clarified the distinction between information transfer, communication. and manipulation, provid- ing motivation for a better quantitative metric to measure emergent communication in competitive environments. Next, we' ve found that LOLA improves effective selfish communication and, using our metric, we find it does so by improving both agents' performance and stability. Finally, we ve shown that using discrete communication channel encourages the learning of cooperative commu-
8

---

Under review as a conference paper at ICLR 2020
nication in contrast to the continuous communication channel setting, where find little evidence
of cooperation.
In coooy-cooeeeative emergent communication, both agents fully trust each other, cooperatively learning a protocol mutually beneficial. competitive MARL, the task is using an existing protocol (or action space) compete with each other. However, selfish emergent communication combines these two since the inherent competitiveness the inherent cooperativeness of learning it; without somewhat agreeing meanings, agents cannot use those meanings to compete (Searcy & Nowicki 2003 Skyrms & Barreti 2018). Thus, the agents must both learn a protocol and use that protocol simultaneously. In this way, even while competing, selfish agents emerging a communication protocol must learn to cooperate.
using the protocol to win is tempered by
REFERENCES
David Balduzzi, Karl Tuyls, Julien Pérolat, and Thore Graepel. Re-evaluating evaluation. In NeurIPS, 2018.
David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojeiech M Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In JCML, 2019.
Nolan Bard, Jakob N. Foerster, A. P. Sarath Chandar, Neil Burch, Marc Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Marc G. Bellemare, and Michael H. Bowling. The hanabi challenge: A new frontier for ai research. ArXiv, abs/1902.00506, 2019.
Jeffrey A. Barrett and Brian Skyrms. Self Self-Assembling Games. The British Journal for the Philoso- phy of Science, 68(2):329-353, 2017.
Xavier Bouthillier, Christos Tsirigotis, François Corneau-Tremblay Pierre Delaunay, Michael Noukhovitch, Reyhane Askari, Peter Henderson, Dendi Suhubdy, Frédéric Bastien, and Pas- Lamblin. Orion Asynchronous Distributed Hyperparameter Optimization. https:// github. com wwcm//pisstmiie /atimmoooooion September 2019.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z. Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. CoRR, 0bs/1804.03980, 2018. URI. http:
Vincent P. Crawford and Joel Sobel. Strategic information transmission. Econometrica, 50(6):
1431-1451, 1982. URL lhtpp:: //op:::////i.ooo// ttg//.t..o....oooo 2t11110.0000111390001
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. TarMAC: Targeted multi-agent communication. In Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1538-1546, Long Beach, Cali- fornia, USA. 09-15 Jun 2019. PMLR. URL http: /ttp://prroeee http:///rcceeeiigg... ml htt.eeisng.si.....s/////.s das 1 1.ahtmml
Richard Dawkins and John Krebs. Animal signals: Information or Manipulation? In J. R. Krebs and N. B. Davies (eds.), Behavioural Ecology, pp. 282-309, Blackwell Scientific Publications, Oxford, 1978.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in a mudti-modal, multi-step referential game. arXiv preprint arXiv: 1705. 10369, 2017.
Joseph Farrell and Matthew Rabin. 103-118, 1996.
Cheap talk. The Journal of Economic Perspectives, 10(3):
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.

---

Under review as a conference paper at ICLR 2020
Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In AAMAS, 2018a.
Jakob N. Foerster, H. Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew M Botvinick, and Michael H. Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. ArXiv, abs/18 181 11118111.01144 11.01558, 2018b.
Jakob N. Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric P. Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte-carlo estimator. ICML, 2019.
Michael Fu, Chapter 19 gradient estimation. Handbooks in Operations Research and Management Science, 13, 12 2006. doi: (0.1016650027-0000000)0110111110
Yuma Fujimoto and Kunihiko Kaneko. Functional dynamic by intention recognition in iterated games. New Journal of Physics, 21(2)0223025, 2019.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to com- municate with sequences of symbols. In Advances in neural information processing systems, pp. 2149-2159, 2017.
Robert A. Hinde. Animal Signals: Ethological and Games-Theory Approaches are Not Incompati- ble. Animal Behavior, 29:535-544 542, 1981.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Çaglar Güllçehre Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, and Nando de Freitas. Intrinsic social motivation via causal influence in multi-agent RL. CoRR, abs/1810.08647, 2018. URL
08647
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge "naturally" multi-agent dialog. CoRR, abs/1706.08502, 2017. URL /ttt:://axxiv.orrr abs/1706. 08502
John R. Krebs and Richard Dawkins. Animal Signals: Mind Reading and Manipulation. In J. R. Krebs and N. B. Davies (eds.), Behavioural Ecology, pp. 380-402. Sinauer Associates, Sunder- land, MA, 1984.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pero- lat, David Silver, and Thore Graepel. A Unified Theoreie Approach to Multiagent Rein- forcement Learning. In NIPS, 2017.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint 11Xi0:161 2016.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. CoRR, abss/18040033
2018. URL BtRLtttt:///xxtt orq/ httg/ab////e 1804. C3984
Joel Z. Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS '17, pp. 464-473, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems. URL http::/dl. acm. wwt.or////aoe hwtp/iin.iiiiinno etm
cfm? wtt.:mimmmmmm1mmmeeem
3091194]
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. CoRR, abs/1707.01063 2017. /ttpppt////axxi.....
abs/1707.01063
Alistair Letcher. Jakob N. Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson, Stable opponent shaping in differentiable games. In ICLR, 2019.
David Lewis. Convention: Philosophical Study. Wiley-Blackwell, Oxford, 1969.
10

---

Under review as a conference paper at ICLR 2020
Ryan Lowe, Jakob N. Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. In AAMAS, 2019.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. CoRR,
700300498, 2017. JRLhtp:///arx: org. www../og////...e.........eoseeae /tobs/1703. 04 908
Vinod Nair and Geoffrey Hinton. Rectified linear units improve restricted boltzmann machines vinod nair. In ICML, volume 27, pp. 807-814, 06 2010.
OpenAl. Openai five. https: http://ooooeeeaa...coeeeeeaaaaaa com -tveeeennniiiieeeeeeeeeeeieeieer 2018.
John G Riley. Silver signals: Twenty-five years of screening and signaling. Journal of Economic literature, 39(2):432 478, 2001.
John Schulman, Nicolas Manfred Otto Heess, Theophane Weber, and Pieter Abbeel. Gradient esti- mation using stochastic computation graphs. In NIPS, 2015.
William A Searcy and Stephen Nowicki. The evolution of animal communication: reliability and deception in signaling systems. Princeton University Press, Princcton, NJ, 2005.
Yoav Shoham, Rob Powers, and Trond Grenager. Multi-agent reinforcement learning:a critical survey. 2003.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lynette R. Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lill- icrap. Fan Hui, Laurent Sifre, George den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-3599 2017.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In ICLR, 2018.
Brian Skyrms. Signals: Evolution, Learning, Information. Oxford University Press, Oxford, 2010.
Brian Skyrms and Jeffrey A. Barrett. Propositional content in signals, June 2018. URL http:
//philsci-archive aittev...taiii............. pitp edu/14747)
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojeiech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mas- tering the real-time strategy game starcraft ii DeepMind Blog, 2019.
Kyle Wagner, James A Reggia, Juan Uriagereka, and Gerald S Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-69, 2003.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
A PROOFS
A.1 PROOF OF FULLY COOPERATIVE/FULLY COMPETITIVE GAME
For b = 0, T, = T, so trivially L, L,, and the game is fully cooperative.
For 180",, T, T, 180 mod 360 we provide visual demonstration in Figure 5 that the sum is always L, + L, = 180 and therefore the game is constant-sum and fully competitive. We can also think of this as moving the actions distance d towards one agent's target means moving distance d away from the other agent's target.
Assume without loss of generality T, < Tr so T, T, + 180 and T. 5 180 T. < 360
< T,, T.,a < 360
Lg + L, = a) (
=min([T, 360 [T, -al) minn(TT, al, 360 IT, -al)

---

Under review as a conference paper at ICLR 2020
T
4
action
) 180
L,
Figure 5: The game with maximal bias 180 showing the sum of Ln losses L, L, 180
case 1: [T, al < 360 - |T, al
min([T, - al. 360
[Tg al) T, a T. 180
subcase a: T, >=
T. a T,
: T. T, + 180
T,
180
:. minin[T, - al, 360
(T, - al) 360 (T, a)
L, + L, T, 360
(Tr -aa) aa)
T, - a + 360
T - 180 + a
180
subcase b: T2 <
T, T a T, < 180
T + 180 a T,
Tr al T,
T, = T, + 180
T,
180
T, - a < 180
2(T, a) 360
т, - a < 360
(T a)
min((T, a/. 360
IT. al) T,
L, + L (a T.) + (T, - a)
T, To
= 180
We can extend the proof by symmetry (on the circle) for [T, - a| 2 360 - [T, - al. so the sum of losses J, Tay always equals 180" so the game is constant-sum and therefore fully competitive.
12

---

Under review as a conference paper at ICLR 2020
A.2 PROOF OF L2 FAIRNESS
Assume without loss of generality Ta < Tr, we are minimizing the sum of L2 losses
min Lg + Ly min(T, a)² (T, a)² min(I', a)² + (I, b a)²
let T. x
min(x a)² 2 (a + b a)²
min 2ox + a² + x² + 2bx + b² -aar 2ob a²
min 2(x 2ax + a² + br ab + /2/44 - -22/2
min 2(x b/2)² 12
+ b/2
Sum of L2 losses is minimized when the action is T, bb2 or halfway between both agents' targets.
B ALGORITHMS
Algorithm Circular Biased Sender-Receiver Game
procedure TRAINING BATCU(b)
T. ~ Uniform(0, 360)
T, T, +
m (ST)))
< R(m)
L, (T, a) = min 360 T, al)
L, < (T, = min min al, a/,360 360
TT, a)
R is updated with SGD
updated with REINFORCE or DiCE
Algorithm 2 Continuous Circular Biased Reedev-e Game
procedure TRAINING BATCH(b)
T. Uniform(O, 360)
T, - +
,4, < S(T.)
m ~ Gaussian (H,
R(m)
L. - La min 360 T. a))
L, (Tr,aa min (T, 360 360
TT a))
R is updated with SGD
updated with REINFORCE or DICE
C Plots
13

---

Under review as a conference paper at ICLR 2020
Test La 1os
Hest C. loss
sender mecevvr
communication
Tair split
sender meniier
commun cation
nO communnccatio
Fair
30
20
30
30
20
25
30
epoch
epoch
opach
(a) bias b 0%
(b) bias b 30
(c) bias 60
11
Senderr
cedonn
1e4r ines
10
lest C. loss
10
retelver
Falr spilt
C.
sender
communication
render
no commun cation
split
10
15
20
25
30
30
20
30
30
25
200
25
30
spoch
opach
opach
(d) bias 90
(c) bias 120
(f) bias b 150
Figure 6: REINFORCE Sender, Deterministic Receiver, L1 hyperparameter search
100
Teed 7, 10SS
20
sadkar receiver
80
Test
20
#20ch
opach
(a) hias b 0
(b) bias b 30
(c) hias 60
120
100
recerver
recetver
110
receiver
Tasst L, loss
Test loos
100
L4 loss
80
80
30
(d) bias b 90
(c) bias b 120
(f) bias b 150
Figure 7: REINFORCE Sender, Deterministic Receiver, L2 hyperparameter search. Note these are identical Figure 6 exxept for b 150
14

---

Under review as a conference paper at ICLR 2020
80
Test La loss
L.
60
20
no Communicatoon
communicctiin
communiccaton spat
20
fair sprit
0
10
15 apoch
20
25
70
20
10
15 spoch
30
25
30
(a) bias
0%
(h) bias b 30
(c) bias
60
120
Taiz 1, 10:55
Tost loss
1.0
100
70
50 'dlr
Schder
90
Sonder
recetver
Ieceiver
Iecesver
camerunicahon
communication
70
communcaboon
30
15 epoch
20
25
30
15 esoch
20
30
10
15 epach
20
25
30
(d) bias = 90
(c) bias 120
(f) bias b 150
Figure DICE Sender, LOLA-1 Receiver
200
100
Teet loSs
80
Test C. loss
30
60
80
no communiication
no continniattionn Fair spiit
no communication
Far agiln
20
Bur spia
20
opoch
opach
bias 0
(b) bias b
-3
bias b 60
120
120
1LD
Test 1001
Test La look
1:00
120
80
60
no communication TUI oplit
can municcation
communccation tair sole
1hrr sple
50
35
20
30
15 #poch
20
30
10
20
spach
epach
(d) bias b 90
(c) bias b
120
(f) bias b 150
Figure 9: LOLA-1 Sender, Deterministic Receiver
15

---

Under review as a conference paper at ICLR 2020
Tel IASS
Test ItSS
sarrder
commurication
spit
communnccctin
recetver
Pair spin
spoch
(a) bias b 0
(b) bias b 30
(c) bias b 60
sonder
no communication
Sonder
.... communication
Tea 18S5
Fest loos
L.
split
(d) bias b 90
(c) bias b 120
(f) bias b 150
Figure 10: LOLA-1 Sender, LOLA-1 Receiver
sender recelver
no communicatin
sender
communication
fall split
Test 1,22,,100 loss
Test C3 1O00
Merzder
commurication
20
30
15 opoch
20
25
opoch
(a) bias 0
(b) bias b
30
(c) bias b 60"
sender
..... no communicatin
sender
.... no communication
11
sender
.... no commun
TPST InSS
1eSt L. toss
(d) bias b 90
(c) bias b 120
(f) bias b 150"
Figure 11: LOLA-2 Sender, LOLA-2 Receiver
16

---

Under review as a conference paper at ICLR 2020
Tegr BASS
Tes 1. BoXs
communicaton far spit
conmunicaatio Fair split
IEcRIver
recerver
sender
commruiication
20
70
20
10
15
23
aporth
opoch
(a) bias 0%
(b) bias b
30
(c) bias 60
30
serrider AECoIver
no communccaton Tail spii
sender
RO Commmnuniaaton
21
receiver
Tail spår
10
20
30
30
15 opoch
20
25
opoch
(d) bias b = 90
(c) bias b 120
(f) bias b 150
Figure 12: LOLA-3 Sender, LOLA-3 Receiver
100
Tast loss
Test loss
80
G0
2G
sa senderr
sender
sender
recelver
sccelver
receiver
communication
20
20
opoch
epoch
epach
(a) bias
0
(b) bias b = 30
(c) bias
60%
100
00
Batt. boss
120
60
recotvor
Test
focoiver spiit
windr
80
80
rocoiver
winder
ogint
15
20
70
spach
epoch
epach
(d) bias h = 90
(c) bias b = 120
(f) bias b 150
Figure 13: LOLA-4 Sender, LOLA-4 Receiver
17

---

Under review as a conference paper at ICLR 2020
B0
80
Todd 1008
lest L. 1085
40
L,
40
no mommmnaciion fair spit
rocoiver
no communicaiio fair spitt
20
communication
Tail spit
10
20
30
20
30
30
15 opech
20
25
spoch
opoch
(a) bias
(b) bias b 30
(c) bias
100
80
Text L, lost
100
sender
Test
sender
50
sender
recelver
suceiver
receivrr
comeunscattin
75
communicaaion
communicaiion
15
20
25
30
15 eboch
20
30
10
15
20
25
30
opoch
epach
(d) bias 90
(c) bias 120
(f) bias b 150
Figure 14: Gaussian Sender. Deterministic Receiver playing the continuous game
80
1pe InVS
Test L Ioss
60
Lg
40
20
no communicatoon
communiccaton
communicatinn spit
ferr
0
10
15 apoch
20
25
70
30
20
25
30
30
15 spach
30
25
30
#poch
(a) bias
0"
(b) bias b 30
(c) bias
60
200
120
100
Taiz 10:5
100
50 10tr
Tost
20
Sehder
sonder necoveer
recetver
Necevvrr
20
communicaticn
15 epoch
20
25
30
15 esoch
20
30
10
15
20
25
30
epach
(d) bias 90"
(c) bias 120
(f) bias b 150"
Figure 15: Gaussian LOLA Sender. LOLA Receiver playing the continuous game
95
Receiver
93
bies 18
92
90
Sender L
Figure 16: REINFORCE Sender vs Deterministic Receiver enrors for all hyperparameter runs with b = 1880/egred shows that agents are mostly fair in the fully competitive constant-sum game
18


Reviews:
[{"review_id": "r1xb36iCFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces a new sender-receiver game to study emergent communication in partially-competitive scenarios. The authors find that communication can also emerge in partially-competitive scenarios and demonstrate how to encourage communication: 1) selfish communication is proportional to cooperation, and it naturally occurs for situations that are more cooperative than competitive, 2) stability and performance are improved by using LOLA, and 3) discrete protocols are better than continuous ones.\n\nStrengths:\n- This is an interesting paper that is well written and motivated.\n- They have justified a new sender-receiver game that can be tuned for various levels of competition which then allows them to analyze the effects of various levels of cooperation and competition.\n- They perform sufficient experimental analysis to show that LOLA outperforms standard methods like REINFORCE in these settings and that discrete communication lends to cooperative communication.\n- Evaluation is good in the sense that they repeat their experiments multiple times across different random seeds.\n\nWeaknesses:\n- Given that cheap talk is an extremely well-studied topic in economics, I feel that the authors should have devoted more time to explain the difference in setting between their work in classic pieces like those of Sobel and Crawford. The authors should properly define what they mean by learning agents versus fully rational agents, and the key differences between the two. Furthermore, nowhere do the authors in the cheap talk paper assert that agents use an existing language: the equilibrium itself assigns meaning to each sender’s message; this is not part of the problem definition per se. In fact, the work of Sobel and Crawford does not even constrain the size of the vocabulary (as was done in this paper): one of its key contributions is to show that in strictly non-cooperative settings, all equilibria must be partition equilibria, with only a finite number of messages used.\n- Section 4.2: “initial random mapping of targets to messages.” The authors made the assumption that this mapping has to be deterministic. Absent a proof or a citation, I find this difficult to accept. This is especially so since mixed strategies are a crucial component of games of imperfect information.\n- The introduction of the circular game is suspect. There already exist numerous games involving cheap talk, one of them from the Sobel and Crawford paper. Why is there a need for this new benchmark?\n-  The description of the game is given as an algorithm in the appendix. This comes across as counterintuitive: why are the gradient steps being included as part of the game description? A game’s specification and the algorithm which is being used to solve it are two different things.\n- It is difficult for me to assess the significance of these results since the authors have not presented real-world scenarios and experiments that demonstrate the importance of selfish communication. For cooperative communication we see it a lot in examples like grounded language learning, visual dialog, multi-agent communication etc. But I am concerned that the new setting proposed in this paper seems like a 'toy setting' to investigate if emergent communication would happen.\n- Are the communicated symbols (discrete or continuous) semantically meaningful? It was shown in Kottur et al. (2017) that for emergent communication to occur and generalize to unseen test instances, it was crucial that the communication protocol was grounded i.e. one symbol learning to represent the color, one representing the shape, one representing the size. What is the final communication protocol learned in this case, and it is useful/interpretable in a similar sense?\n- Typo: 'Figure ??' in line 3 of section 5.1", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkleHtCd9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper looks at the question of emergent communication amongst self-interested learning agents. The paper finds that \"selfish\" (ie. self-interested) agents can learn to communicate using a cheap talk channel as long as the objective is partially cooperative. \nThe paper makes states that this is is a novel finding that contradicts the previous understanding of emergent communication in the literature (side point: at least some of the papers referenced for this claim did not at all make the claim). \n\nI believe there is a major miss-understanding here: As noted in the paper, self-interested agents can learn to communicate in settings in which the reward function is cooperative. Furthermore, it is also known that in 2 player zero-sum there is no incentive to learn a communication protocol. \nThis clearly shows that talking about whether or not \"selfish\" agents can learn to communicate only ever makes sense within the context of a specific game / reward structure. \n\nWith this in mind, the main finding, agents learn to somewhat communicate with each other in a simple toy setting, with more communication happening when the payouts are more cooperative, is not very interesting. \n\nThis doesn't mean that there isn't a good paper to be written here, in principle. Finding simple settings in which SOTA multi-agent learning \"fails\", ie. doesn't find Nash policies, understanding why it fails and then finding ways to mend things is generally a good research direction. However, this would require a few things which are currently lacking from the paper: (1) clear understanding of the Nash policies for the different reward settings (2) Implementation of SOTA methods for MARL which are appropriate for this setting (3) In depth analysis of learning successes and failures, ideally in settings which have previously been studied in literature (given how task-specific this analysis necessarily is).\n\nRegarding 2: General sum games will generally have mixed-strategies as Nash equilibria (just think 'rock-paper-scissors'). With this in mind, using a deterministic policy for the receiver is inappropriate for making any claims about learning in general sum games. \nFurthermore, it is well known that independent gradient descent (IGD) is not generally going to converge in general sum games (consider the loss functions X * Y and - X *Y or matching pennies). So looking at the outcome of IGD without checking for convergence means the results could be just about anything. Indeed, we don't have to go all the way to writing about emergent communication or complex \"sequential social dilemma\" to study this, those issues can easily be found in (iterated) matrix games. \n\nThis gets us to the second major point of the paper. To the authors' credit,  LOLA [1] has been shown to help with convergence in general sum settings and to lead to the emergence of cooperation and reciprocity in iterated games. \n\nHowever, the key point for the ‘cooperation’ part is iterated. In a single shot setting (which is explored in this paper), there is simply no way for the agents to reciprocate with each other. So in short, I do not believe the authors' interpretation that agents learn to cooperate with each other because of LOLA, but I do believe that LOLA can help with the learning of mixed strategies (at least for the sender, given that the receiver is deterministic) and with stabilizing convergence. Lastly, the part of the experimental section is dominated by large error bars and graphs that are difficult to interpret.\n\n\nOther points:\n-\"..but train agents to emerge their own.\" (and many other instances). AFAIK \"to emerge something\" is grammatically wrong (and also sounds really odd). \n-\"Since the loss is differentiable with respect to the receiver, it is trained directly with gradient descent, so we are training in the style of a stochastic computation graph (Schulman et al., 2015).\". This is a weird statement. You don't need SCGs for training a supervised objective. Also, note that the loss is also differentiable with respect to the action of the 1st agent. It is trivial in this setting to compute the true expected return, if that is what you are after. Note my point above about deterministic policies\n-\"We perform a hyperparameter search to over both agents’\" -> spurious \"to\"\n-\"We investigate a similar scenario but concern ourselves with learning agents as opposed to fully-rational agents that have full knowledge of the structure of the game, and we do not assume that agents use an existing language, but train agents to emerge their own\" .This would be interesting, if the game was complex.\n- L_1 vs L - these symbols are used inconsistently, with the subscript _1 sometimes being applied and sometimes not.\n-\"we can look to extant results\" - s/extant/extent?\n-\"We use the L2 metric only on hyperparameter search and keep L1 as our game’s loss to maintain a constant-sum game for the fully competitive case.\" - A few points: (a) the game is not in general constant sum (b) By doing this hyperparameter search the evaluation is strongly biased towards 'fair' attributions. This seems highly problematic. \n-\"We report our results in Figure ??\" -> Broken reference. \n-\"We do not test b = 180◦ because the game is constant-sum and therefore trivially Ls1 + Lr1 = 180◦.\" -> So? It would still be interesting to see what learning agents do in this setting. \n\n[1]: \"Learning with Opponent Learning Awareness\", Foerster et al. \n\n[update: I have updated the score based on the discussion with the authors]. While the paper lacks execution and conceptual clarity, I believe the game itself is interesting and could serve as a starting point for more thorough investigation.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1eW6F5J9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This ICLR submission deals with a problem of whether selfish agents can learn to use an emergent communication channel, using a sender-receiver game as a case study. It is found that communication can emerge in partially-competitive scenarios, and conditions in which this can happen are investigated.\nThis review is delivered with the caveat that I am not an expert in this particulat field.\nThe investigation seems relevant and the paper is well written and structured, being within the scope of the conference. Proofs in the appendix are sound to the best of my understanding.\nThe literature review is up to date and seems overall relevant.\nThis study should be understood as a proof of concept, given that the setting seems rather restrictive, so I am unsure that he results could be generalized.\nThey seem anyhow promising and partially challenge the current understanding of the problem.\nMinor issues:\nAll acronyms in the text should be defined the first time they appear in the text.\nLaTex problem with Fig. reference at the beginning of section 5.1.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There has been a long discussion on the paper, especially between the authors and the 2nd reviewer. While the authors' comments and paper modifications have improved the paper, the overall opinion on this paper is that it is below par in its current form. The main issue is that the significance of the results is insufficiently clear.  While the sender-receiver game introduced is interesting, a more thorough investigation would improve the paper a lot (for example, by looking if theoretical statements can be made).
Decision: Reject