Revisiting Populations in multi-agent Communication
Authors: Paul Michel
Mathieu Rita
Kory Wallace Mathewson
Olivier Tieleman
Angeliki Lazaridou
Venue: ICLR 2023 poster (2023)
Keywords: emergent communication

Abstract: Despite evidence from cognitive sciences that larger groups of speakers tend to develop more structured languages in human communication, scaling up to populations has failed to yield significant benefits in emergent multi-agent communication. In this paper we advocate for an alternate population-level training paradigm for referential games based on the idea of "partitioning" the agents into sender-receiver pairs and limiting co-adaptation across pairs. We show that this results in optimizing a different objective at the population level, where agents maximize (1) their respective "internal" communication accuracy and (2) some measure of alignment between agents. In experiments, we find that this leads to the emergence of languages that are significantly more compositional. Moreover, when agents are trained in populations that are not fully connected (ie. not all agent pairs interact at training time), this approach reduces multi-linguality and improves zero-shot communication with new agents (ie. agents are able to communicate successfully with other agents outside their training partners).

Published as a conference paper at ICLR 2023
REVISITING POPULATIONS IN MULTI-AGENT COMMUNICATION
Paul Michel ENS-PSL
Mathieu Rita INRIA, Paris
Kory Mathewson DeepMind
Olivier Tieleman DeepMind
Angeliki Lazaridou DeepMind
ABSTRACT
Despite evidence from sociolinguistics that larger groups of speakers tend to develop more structured languages, the use of populations has failed to yield significant benefits in emergent multi-agent communication. In this paper we reassess the validity of the standard training protocol and illustrate its limitations. Specifically, we analyze population-level communication at the equilibrium in sender-receiver Lewis games. We find that receivers co-adapt to senders they are interacting with, which limits the effect of the population. Informed by this analysis, we propose an alternative training protocol based on "partitioning" agents. Partitioning isolates sender-receiver pairs, limits co-adaptation, and results in new global optimization objective where agents maximize (1) their respective "internal" communication accuracy and (2) their alignment with other agents. In experiments, we find that agents trained in partitioned populations are able to communicate successfully with develop shared language. Moreover, we observe that larger populations develop languages that are more compositional. Our findings suggest that scaling up populations in multi-agent communication can be beneficial, but that it matters how we scale up
agents which they have never interacted with and tend
INTRODUCTION
Uncovering the mechanisms that underlie our ability to communicate using language is important stepping stone towards developing machine learning models that are capable of coordinating and interacting via natural language. Over the last few years, there has been increasing interest in simulating the emergence of language using artificial agents trained with reinforcement learning to communicate achieve cooperative task (Lazaridou & Baroni 2020). Typically, agents trained to perform a variant of the Lewis signaling game (Lewis) 1969; Skyrms 2010) wherein a sender emits a message describing an object and receiver attempts reconstruct the object based on the description. This line of work has applications to semi- coni-supervised learning. For example, agents that develop languages exhibiting universal properties of natural languages may be used as useful initialization for downstream tasks such as image captioning (Lazaridou et al., (2200) or representation learning (Dessi et al. 2021).
Most previous research has focused on communication between single pair of agents. However, there is mounting evidence that the communication protocols developed this restricted setting become highly specialized and exhibit properties that are at odds with those found in human languages (Bouchacourt & Baroni 2018 Chaabouni et al. 2019): for example agents are able to solve the task successfully while using languages that are not compositional (Kottur et al. 2017 Chaabouni et al.) 2020). These idiosyncrasies of the emergent languages can preclude their use in practical applications (Lazaridou et al. (020). As possible solution, growing body of work is advocating for scaling up the emergent communication literature to populations of more than two agents communicating simultaneously (Harding Graesser al., 2019 Kim & Oh 2021 Rita et al.) 2022a Chaabouni cet al., 20022. Indeed, there is substantial evidence within the language sciences that population dynamics shape the language structure Raviv al. (2019); Nölle et al. (2020). In spite this fact,
several negative results have been obtained, showing that training agents in population yield marginal
Now at DeepMind. Contact at pmi che 31 1155gmm il.aill.ooo com.
Code reproduce our experiments can be found at https: / / thub. www.b...om/mmm.oo che httmm/pmmm.ee11141///c/c.

---

Published as a conference paper at ICLR 2023
benefits without explicit pressure towards e.g. population diversity (Rita al. 2022a) emulation mechanisms (Chaabouni et al. 2022).
In this paper, we call into question the way such populations are trained. By studying simple referential game, we evaluate populations on two desirable features observed in natural language:
Agents are able to communicate with new partners within the same population (Gupta et al. 2021
Larger populations tend to develop more structured languages (Nölle al. 2020).
We provide evidence that populations of artificial agents do not always possess these features (as also attested by previous work, e.g. Kim & Ohj (202I); Chaabouni al. (002)). To shed light on this phenomenon, we analyze the behaviour of agents in a population at the equilibrium ($2). We find that with the standard training procedure, the functional form of the objective is the same as that of single pair of agents, due to receivers co-adapting to their training partners. As our main contribution, we propose an alternative training procedure which partitions sender-receiver pairs and limits o--adattation of receiver agents ($3). We show that this new training paradigm maximizes a different objective at the population level. In particular, it explicitly promotes mutual-intellie -itlliiiillligibilityy across different agents.
In experiments, we find that agents trained in partitioned populations are able to communicate successfully with new communication partners with which they have never interacted during training, and that languages spoken by various agents tend to be similar to one another ($5). In addition, we observe that (1) languages developed in partitioned populations tend to be more compositional and (2) there is a population size effect whereby larger populations develop more structured languages (.6). Our results show that there are multiple ways generalize from single agent pairs larger populations, and that these design choices matter when it comes to studying the emergent language.
COMMUNICATION GAME
We study communication in referential games, a variant of the Lewis signaling game (Lewis) 1969) proposed by Lazaridou et al. (2117). The game proceeds as follows: during each round, sender agent т observes an object x E X' (e.g.. an arbitrary categorical entity, or natural images) sampled from input space X' according to distribution p and generates message ~ .[- x). Messages consist of variable length sequences of tokens picked from a discrete vocabulary V. Note that the tokens themselves
arbitrary and meaningless (typically they are represented numbers from to IVD. A receiver agent then observes message m. and must predict the original object from among a set of candidates C = {ιz, V1.... 3/--1 containing x and [C] distractors, where each distractor is sampled uniformly without replacement from the input space excluding the original object, X {a). Concretely, this is implemented by calculating score fyy m) for each candidate y cf())
and defining the probability of a candidate conditioned on the message p(- m. C) as Evee F(amm
Based on the receiver's success, the sender agent receives a reward R(x, p(- m, C)).
In practice, both senders and receivers are implemented as neural networks T. and Py with parameters 0 and vú estimated by gradient descent. The sender is trained to maximize its expected reward using the REINFORCE algorithm (Williams 1992). while the receiver maximizes the expected log likelihood of identifying the original object, log (a m. C) (also known as the InfoNCE objective; Oord jet al. (2018)) Denoting as Expp the expectation over x sampled from p. the corresponding training objectives are:
(0) = (e Bepp R(x, P: m, m,,)) (((b) Enn Em ( Ec-p log (x m, C)
2.1 POPULATION LEVEL TRAINING
The two-player referential game can be generalized to larger populations of agents (Mordatch & Abbeel 2018 Chaabouni et al.) 2022). In the most general case, we consider a population of N. senders and N, receivers that are linked by a bipartite communication graph G defining connections between senders and receiver (To Pe,) (Harding Graesser et al., 2019 Kim & Oh 2021). At training time, sender-receiver pairs repeatedly sampled and tramned to pertorm round of the game. Importantly, only agent pairs that are connected in the communication graph are sampled.
Throughout this paper, will refer to this type of training Standard training.
2

---

Published as a conference paper at ICLR 2023
With this training procedure, agents are trained to maximize their communicative success with all their neighbors in the communication graph. Let (V(() refer to the neighbors of the i-th node in the graph, and J, (respectively Jr,é-z) denote the objective of no, (respectively Pey )) in the pairwise communication from sender i to receiver j- We can write the overall objective for sender i (and receiver j. respectively) as:
(s,) [Wa(1)
Σ jENc(()
(0.) and Je5(D,) [No() iENc())
Σ
Jn.-s((1))
(3)
At test time, the population is evaluated by averaging the referential accuracy across all possible sender-receiver pairings. Following previous work, in this paper we focus on populations with an equal number N == N₂ = N, of senders and receivers, meaning that there are up to N2 possible pairings.
2.2
WHAT DOES POPULATION-LEVEI TRAINING OPTIMIZE?
To shed light on the differences between training a single agent pair and training a population of agents, analyze the objective optimized by the population. Inspired by (Rita al., 2022b)'s analysis the two-player case, we study the behaviour of the population at the optimum, that is when senders and receivers have reached a Nash equilibrium (Osborne & Rubinstein 1994).
In this section, we make the simplifying assumption that C = X. In other words, receiver agents must pick the correct candidate out of all possible objects in X. This allows us remove the conditioning on C. and write (x m. C") Pv (z m). We make this simplification to reduce clutter in notations. for a detailed discussion). Nevertheless, our key observations still hold for C # X' (see Appendix C
At a Nash equilibrium, the optimal receiver parameters v satisfy
Pe: arg max (ry( = arg max Vi
Σ iENo
(1)))
[aai
$
Assuming that receiver Pv, has high enough capacity, and training is able to reach the global optimum, the solution of the optimization problem in Equation 4 has analytical solution Pez which can be
written as a function "N))) (m x)) == [Ne(()) EieNeG) To: (m x)) the mixture of all senders communicating with receiver
Pag (I m) = (x m) Noo (m x))(()
Na(s (m v)
In other words, Paj is the posterior associated with *Na()) (full derivation in appendix B
An important implication of this result is that when the population graph is fully connected (all senders are connected to all receivers), each receiver converges to the same optimum x (x m) =
"i Na, (m] (m))))(=))))))))))))))))))))))))
Eynp Ei= #9, (m]z)* the posterior of the mixture of all senders in the population. Plugging this back
into each sender's objective, we have
(0,)(0t) Emostt (-1x) R(x,, m" m))
Summing across all senders, we can rewrite the global objective optimized by the senders as max PesppEman R(x, 77( m)).
(5)
9+
In other words, at the equilibrium, the population maximizes the expected reward of the "sender ensemble" n", rather than that of individual agents TØ; : the objective of population N agents is functionally the same irrespective of N. We postulate that this indifference to the population size may account for the surprising lack of effect of larger populations observed in some previous work (Rita et al.) 2022a; Chaabouni et al., 2022). Differences in behaviour must be attributed to effects stemming from training dynamics (e.g. it becomes more difficult for receivers to learn the posterior T <xx m)), or be imposed through extrancous modifications of the population objective (for example explicit imitation components: Chaabouni al, (2022)

---

Published as a conference paper at ICLR 2023
A second observation is that there is direct pressure for agents that communicate at training time to develop the same language. Indeed, it is entirely possible that all senders develop different but non-overlapping languages: it suffices that two senders communicating with shared receiver use the same message m to describe a different object. In this case receivers can simply learn their neighhoring sender's languages and there no need for the senders converge to unified language.
PARTITIONING AGENTS
A key difference between the usual population setting and populations of humans in laboratory experiments is that agents not usually split into "senders" and "receivers". Rather, each par- ticipant in the experiment assumes both sender and receiver role (Galke et al., 2022). Our hy- pothesis is that, counter what customary in the emergent communication literature, tying senders and receivers is key in surfacing useful population-level dynamics in multi-agent com- munication.
Sunder gradien ( Pa
JFL
Sender
Receser
groden (8 Assuf!
Figure 1:
In the standard setting (left), both receivers (in blue) are trained by maximizing their discrimination objective with respect both senders. With partitioning, receiver Pet (resp. Pva)) only trained maximize its communica- tion objective with sender ne; (resp. TT,)
To operationalize this sender-receiver coupling, we identify an "agent" as a sender-receiver pair. During training, only train receiver Pas with its associated sender πe., In other words, (+;) - (t1)). ). In doing so, "parti- tion" the agents by preventing receiver from co-adapting to other senders j * i. This procedure is illustrated in Figure [ Note that senders nevertheless still train with rewards from neighboring receivers, and so communication across agents can still emerge. Importantly, partitioning prevents receivers from learning recognize multiple languages, as they now only trained messages emitted by single sender.
Following a similar analysis Section 2.2, derive that the optimum, receiver Pv; (a m) now
takes the form of the posterior associated with its respective sender, Top (x m) *ep (mla))p()) (derivation in Appendix B). We can thus write the population-leve objective at the equilibrium
Eg-u Tily
E Exng E Eawno (.1)) R(I, noz m))
E EzopEmwap R(x) Tor m)) jEN
(6)
N
Internal communication
Mutual intelligibility
Note that the functional form of the objective can now be decomposed into two parts: an "internal communication" objective which takes the same form that of a single pair of agents, and "mutual intelligibility" objective which enforces that neighboring agents are able to communicate successfully. In experiments, show that this explicit pressure towards mutual intelligibility promotes the emergence of a single language within the population, which in turn enables agents to communicate with new partners outside of their training neighborhood.
EXPERIMENTAL SETTING
4.1 DATASETS
We perform experiments on two datasets: a simple, synthetic "attribute/valess" dataset and more realistic image dataset.
Attribute/Values In this dataset, each object represent by a collection of abstract "attributes". Specifically, each input is vector of 4 attributes, each of which can take 10 total values. This results in 10 total attribute/value combinations (Kottur et al. 2017 Chaabouni et al. 2020). In each setting we hold out 1, 000 combinations to be used validation, and 1, 000 more for use as a test set. We can thus ensure that we are evaluating the agents' ability to generalize to unseen combinations of attributes.
4

---

Published as a conference paper at ICLR 2023
ImageNet In addition toy objects, perform experiments with referential games based more realistic objects. Following Chaabouni et al. (2022). we use the ImageNet (Deng et al. 2009) dataset of natural images. The dataset consists of about 1.4M training images collected on the internet and annotated for 1,000 labels from the WordNet database (Miller 1995). Images are first encoded as 2048-sized real-valued vectors with (frozen) ResNet pre-trained with BYOL (Grill et al. 2020 before being passed to sender and receivers.
4.2 GAME ARCHITECTURE
Both sender and receiver agents are based on layer LSTMs (Hochreiter & Schmidhuber 1997) with embedding and hidden dimensions of size 256. Specifically, the sender first encodes the object into a vector of size 256, which is concatenated to the input of the LSTM. At each step, the output of the LSTM cell is passed through a fully connected layer to produce logits of size [V]. A softmax function then applied to obtain normalized probabilities over the vocabulary. During training. messages are generated by sampling from the distribution whereas at test time generate messages deterministically via greedy decoding. In both cases, generation stops whenever a special "<BOS>" generated, or when the number of tokens reaches a fixed limit L.
The receiver encodes the message with an LSTM encoder, the output of which is the fed into a fully connected layer to yield a vector of size 512. The candidate objects C are then scored by computing the dot product of this vector with 512-dimensional encoding of each candidate. The conditional distribution over candidates is then obtained by taking a softmax. We set the reward function for the sender to the log-likelihood assigned by the receiver to the correct candidate, R(2, (- m)) log (p m).
Throughout all experiments, we set the vocabulary size [V] to 20 and the maximum length of the messages, L, to 10. This means that the communication channel used by the agents has a capacity of about 20 which ensures that there is communication bottleneck (the size of the channel is several orders of magnitude larger than the size of our datasets). Our implementation, based on the EGG toolkit (Kharitonov et al. 2021). will be open-soureed upon de-anonymization.
4.3 POPULATION TRAINING
We train populations following the procedure outlined by Chaabouni et al. (2022): for each minibatch of data, we sample K pairs from the population (uniformly among the pairs linked in the communication graph). Each pair plays an episode of the game, and the agents are updated simultancously following the gradients of their respective objectives. We take = max max(10, .) to ensure that each agent plays the game least once every step aver- age. This procedure needs to be modified for partitioned populations: since receiver j is only with its respective sender instead of with all of its neighbors, there is now only Neti]] chance that receiver will be updated every step (the probability that the pair (j.j) is sampled), For larger populations, especially those that are fully-connected, this dramatically slows down training as receivers are updated very infrequently. To address this issue, we modify the procedure as follows: for every sampled agent pair (TT. Pv, ). we calculate both Je,i-j and Jr,s4s and update both me, and Pv: Note that this necessitates calculating both Prey (x m. C) and pe: (x: m,CC and therefore we incur a small computational overhead. However we only observe a 5% increase training time due to the fact that are mack-propagating through only of the two receivers, Pv. (z m, C). With this modification, we recover the property that each agent (sender or receiver) is updated once every step average.
(a) Fully-connected
(b) Circular
Figure 2: Example of communication graphs used in this paper
In all experiments we train with a batch size of 1024 with the Adam optimizer Kingma & Ba 2014) using learning rate of 0.001 for the attribute/value dataset and 0.0001 for Imagenet. The other parameters are set to 31 0.9, B2 = 0.999 and e = 10 10-8 We apply & regularization with coefficient of 10-3
5

---

Published as a conference paper at ICLR 2023
Table Accuracies with training partners and new partners both datasets. Numbers are reported with standard deviation across all pairs for independent experiments
ImageNet
Attribute/Valie Values
Standard
Partitioned
Standard
Partitioned
Training partners
97.09 1.10
99.75 + 0.08
99.88 + 0.15
99.81 + 0.22
New partners
5.41 13355
96.24 = 3.25
7.81 18.28
40.37 = 29.44
Table Language similarity between training partners and new partners on both datasets. Numbers are reported with standard deviation across all pairs for 3 independent experiments
ImageNet
Attribute/Valle
Standard
Partitioned
Standard
Partitioned
Training partners
0.28 + 0.07
0.40 + 0.02
0.28
0.05
0.36 + 0.01
New partners
0.22 + 0.19 0.37 + 0.15
0.23 + 0.19 0.31 + 0.17
We systematically augment the sender objectives with an entropy maximizing term, which has been found to encourage exploration (Williams & Peng. 1991). The coefficient for this entropy term is 0.1 in all experiments. To reduce the variance of the policy gradient in REINFORCE, substract a baseline computed by taking the average reward within a mini-batch for each pair (Sutton) al. 1999). We evaluate the population every epoch (every 5 epochs for the Attribute/Valu dataset) on the validation set. We only evaluate on up to 100 unique pairs sampled uniformly within the population, this time without consideration for the communication graph. We train for fixed number of epochs, selecting the best model based on the average validation accuracy across all evaluation pairs. Although convergence not guaranteed this general-sum communication game, all reported experiments find that agents converge reliably with our choice of ppper-paramtters
COMMUNICATION WITH NEW PARTNERS 5
In our first set of experiments, we evaluate the ability of agents trained in populations to communicate with partners they haven't 't interacted with during training.
5.1
CIRCULAR POPULATIONS
Specifically, study "circular" populations of agents arranged ring lattice. Each agent (sender- receiver pair) is only trained with neighhoring agents i 1,..., + 1 and the graph is cyclical (see Figure 6a)) We choose this type of population because is extreme case of a population where each agent has the same, minimal amount of neighbors (two), yet there is still path between any two agents. In this context, training partners are sender-receiver pairs that connected in the graph and have interacted during the training phase whereas new partners refers to pairs that have not interacted during training. Experiments with other population types are reported in Appendix D
We report results along two metrics:
Communication Accuracy of sender/receiver pairs evaluation set. This measures how successful the pair is in communicating.
Language Similarity between senders. This metric (also called synchronization Rita et al.) (2022a)) is calculated as 1 8i.j. where d.,j is the normalized edit distance between messages output by two senders, averaged across all objects in our evaluation set.
We report these metrics for both training partners and new partners. Note that high communication accuracy does not always entail similar languages: accuracy despite all senders sending different messages for any given object (it is only necessary for a given message to unambiguously refer to one object across senders).
possible for the receivers achieve high
5.2
PARTITIONING ENABLES SUCCESSFUL ZERO-SHOT COMMUNICATION
In Table and report accuracies and similarities for circular populations of 20 sender-receiver pairs trained on ImageNet and the Attribte/Vallue dataset. All metrics are calculated on the test set and averaged across 3 independent experiments.

---

Published as a conference paper at ICLR 2023
0.4
80
partitioned
Accuracy
Similarity
60
standard partitioned
0.3
40
20
0.2
7
0
2
3
7
8
10
0
1
2
4
5 6
7
8
9
10
Distance
Distance
(a) Accuracy
(b) Similarity
Figure 3: Accuracy and language similarity as a function of the distance between two agents in the communication graph.
We observe that in populations following the standard training paradigm (Standard), there is stark discrepancy between training and new partners. Indeed, on both datasets the accuracy with training partners reaches very high value, above 95%. Yet, the accuracy when agents communicate with new partners drops down to less than 10%. On the other hand, in Partitioned populations, agents reach much higher accuracy with non-neighbors, up 96% ImageNet and 40%. A similar trend holds for language similarity.
Note that all metries on new partners exhibit high standard deviation. An explanation is that among non-neighboring pairs there is a different behaviour depending on how far the two agents are in the population. This is verified in Figure which displays breakdown between two agents in the communication graph (on ImageNet). We find that without partitioning, accuracy drops off sharply close to 0 for agents distance 2 whereas decreases almost linearly with the distance in the partitioned case, down to about 95% for the most distant agents.
function of the distance
5.3 TRAINING DYNAMICS
We further investigate the evolution of accuracies during training. In Figure 4 we plot the evaluation accuracies of both standard and partitioned populations broken down by distance between pairs, focusing the ImageNet dataset. Note that there epoch #s 10, the accuracy for all training pairs increases, after which agents over-fit to their training partners (distances 0 and 1) and the accuracy other pairs decreases plateau.
training phases in the standard case. Up to
On the other hand, Figure 4b illustrates the pressure for nutual-intelligibility in partitioned popula- tions: accuracy between training pairs reaches close 99% accuracy (around epoch 20), accuracies across distant pairs increases rapidly before plateauing above 90%. In fact, our results show that the most distant accuracies are still increasing after 150 epochs, albeit very slowly.
PARTITIONED POPULATION DEVELOP MORE COMPOSITIONAL LANGUAGES
In this section, investigate the effect of partitioning the structure of the language, with focus on compositionality.
6.1 MEASURING COMPOSITIONALITY
language is said be compositional (Szabó) 2020) when the meaning of whole utterance can be systematically deduced from the meaning of its components (i.e. words). It has been argued that compositionality makes languages easier to learn (Davidson, 1965). Consequently, emergent communication protocols that are compositional may ultimately be casier to understand by humans.
common metric for measuring compositionality in emergent languages is the topographic similarity (Brighton & Kirby 2006, Lazaridou et al. 2018). Topographic similarity captures the intuition that compositional language will map similar "meanings" to similar messages: the phrase "a red bird" is more similar to the phrase "a blue bird" than to "a powerful computer". In practice, the topographic similarity is computed by measuring the Spearman rank correlation coefficient (Spearman 1904) between (1) the pairwise distances across all objects and (2) the pairwise distance across all messages.
By construction, the similarity of sender with itself (corresponding to distance of () is always one. We omit this value from the figure better illustrate the trends for distance > 1.
7

---

Published as a conference paper at ICLR 2023
100
100
80
60
40
125
100
epoch
egoch
(a) Standard
(b) Partitioned
Figure 4: Evolution of validation accuracy during training across agent pairs at various distances in the communication graph. Results are aggregated over all agent pairs and 3 populations.
6.2 EFFECT OF POPULATION SIZE ON COMPOSITIONALITY
We run experiments our Attribute/Valuee dataset, with both standard and partitioned populations that are fully-connected (see Figure (d)). Population sizes range from 2 to 25 sender-receiver pairs. We compute topographic similarity using the Hamming distance in the object space (the distance between two objects is the number of attributes in which they differ) and the normalized edit distance between messages. In Figure 5a we observe that while standard population-level training does increase the topographic similarity of the language overall, population size has very little effect: populations of sizes 3 and 20 both reach about the same value of 30 average. On the other hand, partitioning greatly increases the effect of population size on compositionality: populations of size 20 have significantly higher topographic similarity than populations of size 5, with 10 points difference.
6.3 CO-ADAPTATION IS RESPONSIBLE FOR THE DECREASE IN COMPOSITIONALITY
Up until this point, we have described partitioning (or lack thereof) binary choice. However, it is possible to partition a population only partially, by allowing receiver j to train with senders i + occasionally with probability associated with a mixture between me; (m x) and xt 7(m x)) (see Appendix B for the derivation). If 0 a 1, receivers are now optimizing for different objective (as in partitioned populations), but some amount of co-adaptation is still allowed. We perform this experiment on the Attribute/ Values dataset with fully connected population of size 10. varying the degree of co-adaptation ranging in {0, 0.1, 0.5, 0.9, 1}. a = corresponds to partitioned training whereas a = is equivalent to standard training. All populations converge to > 99% accuracy. However, in Figure 5b) we find that topographic similarity drops as soon as we introduce minimal amounts of co-adaptation (a 0.1) and decreases steadily to the level of standard populations as a grows to This further corroborates our hypothesis that reducing co-adaptation promotes the emergence of more structured language, and that eliminating it altogether (in a partitioned population) yields the best results.
O. In doing so, the optimal receiver now becomes the posterior
6.4 IMPORTANCE OF MUTUAL INTELLIGIBILITY
Recall that the objective a partitioned population at the equilibrium (Equation can be decomposed in two terms: an "internal communication" corresponding to the single agent pair objective and "mutual intelligibility" term which encourages senders to align their languages. Importantly, the latter is the only element that separates a partitioned population from collection of isolated agents. To measure its effect on the compositionality populations of size 10 and decrease the relative weight of the mutual intelligibility term. This is implemented by making the pair (Te, Pe.) more likely to be sampled than other pairs (mo. Pe,) j i by factor x 1-8 isolated sender-receiver pairs). In Figure 5c we find that emergent languages retain high topographic similarity even at small 8, and the sharp drop-off occurs only when 33 is very close to O. This confirms that the mutual intelligibility term exerts strong pressure towards compositionality. We investigate the evolution of the two terms during training in Appendix
the emergent language, we train fully connected
B. We let 3 range from 0.5 (partitioned population) to 0.0 (collection of
8

---

Published as a conference paper at ICLR 2023
Topagraphic similbrit
Topographh similarity
Topograpang similartyy 24
40
15
35
36
dard
34
1
25
Sort
Single
30
2345
10
25
0.6 Co-adaptation probability
0.0
0.1
Population
Mutual Intelliibiiity relative weight g
(a) Topographic similarity as a func-
(b) Topographic similarity with (c) Topographic similarity when ab-
tion of population size on an at- varying degrees of partitioning (pop-
lating the mutual intelligiibilliy term (populations of size 10).
tribute/value communication game. ulations of size 10).
Figure 5: Influence of partitioning on the topographic similarity of the emergent languages.
7 RELATED WORK
There is a rich history of modeling the emergence of language as the solution to a cooperative game
that can be traced back to functional theories of language (Wittgenstein, 1953 Austin 1962] Clark
1996). With a regain of interest for the study of language evolution (Crawford & Sobel 1982
Christiansen & Kirby] 2003), rich literature has developed around computational simulations of the emergence of language based on simple language games (Lewis) 1969, Skyrms, 2010 Batali
1998; Cangelosi & Parisi 2002). Examples include studying evolutionary models the emergence
of grammar (Nowak & Komarova 2001), the influence of cultural transmission (Brighton & Kirby
2006), game theoretical considerations (Huttegger et al. 2014) or linguistic diversity (Livingstone &
Fyfe 1999) among others.
The recent success of deep learning in natural language processing has spurred interest in studying
signaling games between deep neural network trained with reinforcement learning solve signaling
game (Lazaridou et al. 2017 Foerster et al., 2016). Several follow-ups have taken this idea further
by extending it to
complex games environment (Sukhbaatar et al. 2016 Havrylov & Titov
[017]Jaques et al., 2019 Das et al. 2019) or by adding an element of competition (Singh et 2018
Noukhovitch et al. 202 or negotiation (Cao al. (018)) or even explicit pressure towards certain
desirable properties (Kottur et al.) 2017 Choi et al. 2018 Li & Bowling 2019 Ren et al. 2019).
In parallel, several efforts have been made to understand the properties of the emergent languages
(Bouchacourt & Baroni 2018 Chaabouni et al. 2019 2020).
Within this growing literature, many authors have explicitly studied the use of populations of more than two agents. Various works have argued for augmenting populations with explicit pressure towards more structure languages, via e.g. generational transmission (Cogswell et al. 2019). adversarial regularization (Tieleman et al. 2019), varying learning speeds (Rita et al. (2022a) or imitation learning and voting (Chaabouni et al., 2022) Although the focus is often on fully-connected populations, some authors have explored more complex communication graphs, for the purpose of modeling contact linguistics (Harding Graesser et al., 2019) or the effect of social network structure on the own: the authors study the effect of population size and connectivity in the standard training paradigm. The purpose of this paper is to highlight the impact of the training procedure on these very effects.
language (Dubova et al.) (020). Recent work from Kim & Oh (202T) 1 is perhaps closest
CONCLUSION
Empirical findings in socio-linguistics suggest that population dynamics should help in simple sender-receiver communication games. In this paper, we observed that populations trained by naively extending the simple 1-1 protocol to N x N agent pairs fail to exhibit some of the properties that are observed in human populations. Motivated by an analysis of populations at the equilibrium, we described alternative training paradigm, based agents partitioning to reduce co-adaptation. Empirically, we find that partitioning enables us to recover some of the aforementioned properties.
Our findings call attention to the fact that there is more than one way to generalize two single to many agents, and simple design choices can have a great impact on the training dynamics and ultimately the effect of population on the emergent language. Beyond emergent communication, hope that this observation can inspire similar work in other cooperative multi-agent problems where co-adaptation between agents may counteract population effects.

---

Published as a conference paper at ICLR 2023
REFERENCES
John Langshaw Austin. How to do things with words. Harvard University Press, Cambridge, MA, 1962.
John Batali.
Computational simulations of the emergence of grammar. Approaches to the evolution of language: Social and cognitive bases, pp. 405, 1998.
Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in emergent language game. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 981-985, 2018.
Henry Brighton and Simon Kirby, Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life, 12(2):229-22) 242, 2006.
Angelo Cangelosi and Domenico Parisi. Computer simulation: A new scientific approach to the study of language evolution. In Simulating the evolution of language, pp. 3-28. Springer, 2002.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z. Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding emergent communication. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 8th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4427-4442, 2020.
Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent munication at scale. In Proceedings of the International Conference on Learning Representations
(ICLR), 2022.
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Compositional obverter communication learning from raw visual input. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Morten H Christiansen and Simon Ed Kirby. Language evolution. Oxford University Press, 2003.
Hebert Clark. Using Language. Cambridge University Press, Cambridge, UK, 1996.
Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruy Batra. Emergence of compositional language with deep generational transmission. arXiv preprint arXiv:1904.00066 2019.
Vincent P Crawford and Joel Sobel. Strategic information transmission. Econometrica: Journal of the Econometric Society, pp. 1431-1451, 1982.
Abhishek Das, Théophile Gervet, Joshua Romoff. Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 1538-1546. PMLR, 2019.
Donald Davidson. Theories of meaning and learnable languages. In Yehoshua Bar-Hillel (ed.), Proceedings of the 1964 International Congress for Logic, Methodology, and Philosophy of Science, pp. 383-394. Amsterdam: North-Holland, 1965.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the 22nd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Roberto Dessi, Eugene Kharitonov, and Marco Baroni. Interpretable agent communication from scratch (with a generic visual processor emerging on the side). arXiv preprint SEti:2106.04252: 2021.
10

---

Published as a conference paper at ICLR 2023
Marina Dubova, Arsenii Kirillovich Moskvichev, and Rob Goldstone. Reinforcement communication learning in different social network structures. In Language in Reinforcement Learning Workshop 2020, 2020.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Annual Conference on Neural Information Processing Systems (NIPS), volume 29, 2016.
Lukas Galke, Yoav Ram, and Limor Raviv. Emergent communication for understanding human language evolution: What's 's missing? In Emergent Communication Workshop at ICLR 2022, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Proceedings of the 34th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp. 21 21271-21284, 2020.
Abhinav Gupta, Mare Lanctot, and Angeliki Lazaridou. Dynamic population-base based meta-learning for multi-agent communication with natural language. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, 2021.
Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3700-3710, 10, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/11/11/1/ 13844
URL https
//cclannholooy.orrgg//
1384
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games:
Learning to
communicate with sequences of symbols. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS), volume 30, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1788 1780, 1997.
Simon Huttegger, Brian Skyrms, Pierre Tarres, and Elliott Wagner. Some dynamics of signaling games. Proceedings the
the National Academy of Sciences, 11 111(Supplement 10:10873-0880 2014.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In Proceedings of the 36th International Conference Machine Learning (ICML), pp. 3040-3045 PMLR, 2019.
Eugene Kharitonov, Roberto Dessi, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: a toolkit for research on Emergence of lanGuage Games. https://githini com/
facebookresearch/ EGG 2021.
Jooyeon Kim and Alice Oh. Emergent communication under varying sizes and connectivities. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Satwik Kottur, José Moura. Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2962-29967. 2017.
Angeliki Lazaridou and Marco Baroni. Emergent multi- agent communication in the deep learning era. arXiv preprint 1txXiv:2006.02411 2020.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
11

---

Published as a conference paper at ICLR 2023
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls. and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In Proceedings of the International Conference Learning Representations (ICLR), 2018.
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 8th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7663-7674, 2020.
David Lewis. Convention: philosophical study. John Wiley & Sons, 1969.
Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communi- cation. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Daniel Livingstone and Colin Fyfe. Modelling the evolution of linguistic diversity. In European Conference on Artificial Life, pp. 704-708. Springer, 1999.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39 41, 1995.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the 32nd Meeting of the Association for Advancement of Artificial Intelligence (AAAI), volume 32, 2018.
Jonas Nölle, Riccardo Fusaroli, Gregory Mills, and Kristian Tylén. Language shaped by the environment: linguistic construal a collaborative spatial task. Palgrave Communications, 6(1): 1-10, 2020.
Michael Noukhovitch, Travis LaCroix, Angeliki Lazaridou, and Aaron Courville. Emergent commu- nication under competition. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 974-982, 2021.
Martin A Nowak and Natalia L Komarova. Towards an evolutionary theory of language. Trends in cognitive sciences, 5(7):288 295, 2001.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint / 2018.
Martin J. Osborne and Ariel Rubinstein. Course in Game Theory. The MIT Press, 1994. ISBN 0262150417.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Larger communities create more systematic languages. Proceedings of the Royal Society B. 286 286(190))20191222, 2019.
Yi Ren, Shangmin Guo, Serhii Havrylov, Shay Cohen, and Simon Kirby. Enhance the compositionality of emergent language by iterated learning. In 3rd NeurlPS Workshop on Emergent Communication (EmeCome@ NeurlPS 2019). URI. https://papers. nips. cefbook/advances http://ddaannne----inee--------a inter-rrnnnuunlnnfooiiii 32-2019, 2019.
Mathieu Rita, Florian Strub, nan--Batiien Grill, Olivier Pietquin, and Emmanuel Dupoux. On the role of population heterogeneity in emergent communication. In Proceedings of the International Conference Learning Representations (ICLR), 2022a.
Mathieu Rita, Corentin Tallec, Paul Michel, beatibastien Grill, and Florain Strub. Emergent com- munication: Generalization and overfitting in lewis games. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS). 2022b.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbuatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
Brian Skyrms. Signals: Evolution, learning, and information. OUP Oxford, 2010.
12

---

Published as a conference paper at ICLR 2023
C Spearman. The proof and measurement of association between two things. The American Journal of Psychology. 1904.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS), volume 29, 2016.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Proceedings of the 1 13th Annual Conference on Neural Information Processing Systems (NIPS), 12, 1999.
Zoltán Gendler Szabó. Compositionaliiyy In Edward N. Zalta (cd.). The Stanford Encyclopedia of Philosophy. 2020. URL https:t//ppppttttee stanford. edu sedu/arre acchi ves/fa f20020 cettriisscompossoooo onaallanll ity/ Accessed: 2022-05-13.
Olivier Tieleman, Angeliki Lazaridou, Shibl Mourad, Charles Blundell, and Doina Precup. Commu- nity size effect artificial learning systems. In ViGIL@ NeurIPS, 2019.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-23) 2566, 1992.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268 1991.
Ludwig Wittgenstein. Philosophical Investigations. Blackwell, Oxford, UK, 1953. Translated by G.E.M. Anscombe.
A APPENDIX
B DERIVATION OF THE OPTIMAL RECEIVER
We first prove a more general result from which the optimal receiver both in the standard and partitioned can be derived.
B. GENERAL CASE
Consider receiver j trained maximize
Jns(1;))
Σ
a.Jr,
Esenderss
where O;=11... are arbitrary weights for the senders (we assume that the a, are positive and sum to one). We can rewrite the objective as:
Jnb(:by))
Σ iEsenders
Σ iEsenders
agEEmnto (11z) log Pays (x m)
Note that by linearity of expectation we can pass the a weighted average over the senders inside of the expectation and rewrite the second expectation in terms of the mixture t- | x) Eiewnders 0; (m a:):
(1i() = Percop Em Exemno as (m]z) log Pvy (x m)
Ba (112) log Pv. (x m)
13

---

Published as a conference paper at ICLR 2023
With slight abuse of notation, let us denote by Ta (m) Be | x)) the marginal
distribution over messages and To (x m) == "5 (m)a)p(p)) the associated posterior. Notice that rewrite the double expectation Berwp.mwxx (-1x) as Emway (-) (owag by inverting the order summation. We can therefore rewrite
(m)
since by definition n/ (m. <))p(a)) x. (ar m)m (m),
(r = E Em () ( m) (- m))
where Hi(p, q) denotes the cross-entropy E,, | log of two distributions p and q. Importantly the cross-entropy is non-negative and Hp, g) = 0 if and only if p
Consequently, the receiver Po will be optimal (v, 0) if and only if for all
T. (m x))((z)
Pvg (z m) xz(x m)
(8)
By (m y)
B.2 OPTIMAL RECEIVER IN STANDARD POPULATIONS
Recall that in standard populations, the training objective for receiver j is:
Jr,((15)) INa(1 kENol)
Σ (
Note that this is special case of Equation with
Necil f
otherwise
Consequently, the derivation in Section B. tells us that the optimal receiver is
Pez (x m) = TNoC)( (x m) Tio (m x))(z)
y)
Where T (ao x) = [Nooill EvENaGo (m; 1
1
B.3 OPTIMAL RECEIVER IN PARTITIONED POPULATIONS
In partitioned populations, the training objective for receiver is:
Jo)(Wn) = (1)))
This is also special case of Equation with
[ ifi=j
0r 0 otherwise
The derivation in Section B.1 thus yields the optimal receiver
Tī (m [ x)p(x)
Pug (z m) nz (x m)
(10)
Ey=p (m g)
"oore accurately, if the message space is not finite then the condition holds not for all m, but almost surely. However throughout the paper we experimenting with finite (albeit large) message spaces.
14

---

Published as a conference paper at ICLR 2023
(a) =1
(b) k
(c) k 3
(d)
Figure 6: Example of ring populations with connectivities ranging from h (circular population) n (fully connected population)
B.4 OPTIMAL RECEIVER IN PARTIALLY PARTITIONED POPULATIONS
In the partially partitioned populations used in Section 6.3 each receiver's objective is mixture
between the standard and partitioned objective. This can also be rewritten special case of Equation with
a + [Naui
if
if e \ (i)
NeUi
otherwise
The optimal receiver can then be rewritten the posterior distribution associated with the mixture sender
a x + x (x m)
C THE CASE OF REFERENTIAL GAMES
In the analysis from Section 2.2 onward, we assumed C X' simplify notation. We relax this assumption without changing our key observation that all receivers are the same at the optimum.
Indeed, in this case the receiver's objective in standard population is:
(v,)
Σ
Ja)))((b))
iENo()
Σ
Ez Exaple E Bmma (12) Bemp Pog m, C)
[Woli)) vENa()
-Eewy Eawp Bernn (o Beep log Pey (x m,C)
This objective, called InfoNCE (Oord et al. 2018) also has an analytical solution that can be expressed function of Na)))) of the form:
Pug (x m, C)
p(z)
(1I)
Ti ( (vlm)
Σ
YEC
ply)
Despite the more complicated form of the optimal receiver, the key ingredients to analysis Sections 2.2 and 3 are preserved: at the optimum, each receiver function the posterior TNa (x vn associated with the communication partners to which it co-adapts. A similar analysis in partitioned populations shows that the optimum for receiver j then only depends on the posterior associated with its respective sender Tez instead.
D EXPERIMENTS WITH RING POPULATIONS WITH GREATER CONNECTIVITY
The circular population described in Section 5 is an extreme case of a connected population where each agent has the minimal amount of neighbors (2). To verify that findings apply to more general
15

---

Published as a conference paper at ICLR 2023
Table 3: Accuracies with training partners and new partners on the Attribute/Values dataset for ring populations. Numbers are reported with standard deviation across all pairs for 3 independent experiments
k 3
Standard
Partitioned
Standard
Partitioned
Training partners
99.288 0.36
99.32 =0.37 99.71 + 0.33
99.23 + 0.39
99.26 = 0.40
New partners
79.67 200.78
98.08 + 4.17
99.89 = 0.12
B
B 0.001
B 0.01 Topsim
B 0.1 Topsim
B 0.5 Topsim
Topsim
Topsim
Loss Mutual inteliiibiiity
les5
Internal
50 spacc
opoch
epach
opoch
epoch
Figure 7: Evolution of internal communication and mutual intelligibility terms with different weight- ings 3 (populations of size 10).
(non connecco populations, we perform the same experiment on ring populations with higher connectivities. We define a ring graph with connectivity it as a circular graph where each vertex is connected to its k-nearest neighbors on each side (ic agent i is connected to agents i k...., i + k). Examples are shown in 6 allows us to interpolate between circular populations (A: = 1) and fully connected populations (k n/2) while jointly (1) increasing the number of communication partner given agent and (2) decreasing the path length between any two agents, two properties of the graph that are arguably most impactful the mutual intelligibility between agents.
perform the same experiment as Section with graphs of connectivities and 6 training partners for each agent) the Attributc/values dataset. We report accuracues in Table 3 We observe that across the board, populations with greater connectivity exhibit higher accuracy in both conditions. However, we still find that partitioned populations still perform best.
and k = 3 (resp.
E FURTHER ANALYSIS OF THE EFFECT OF MUTUAL INTELLIGIBILITY
find that languages stay highly compositional until the mutual intelligibility weight 3 is decreased to almost O. Our hypothesis that even with small amounts of mutual intelligibility, agents will eventually have optimize this part of the objective after they have maximized their respective internal communication to the point where the main contributor to the training gradient is the mutual intelligibility term.
In Section 6.4
To verify this hypothesis, in Figure 7 we report the evolution of both internal communication and mutual intelligibility losses during training for various values of the mutual intelligibility weight 3. As expected, we observe that for all but very small values of B, the mutual intelligibility loss eventually decreases (although it decreases faster for high 9).
16


Reviews:
[{"review_id": "h6PMpMyHC6", "reviewer": "Reviewer_uuc8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper revisits the population-based training method in emergent multi-agent communication. The authors analyze the standard population-based training protocol and point out that the overall objective is the same as that of a single-pair training protocol. The co-adaptation of receivers to senders in the standard method is believed to limit the effect of population. To better make use of population, the paper proposes to partition agents into pairs and one receiver is only trained with its associated sender. The authors claim that such partitioning enhances zero-shot communication with new partners as well as the compositionality of the emergent language. Empirical results in two referential games verify the statements and the effectiveness of the proposed method.", "summary_of_the_review": "The paper is interesting and solid. But the experiments are not convincing enough, requiring comparing with different patterns of graphs, e.g., random graphs. Besides, demonstrating the feasibility of extending the method to a sequential decision-making setting will help the method be more general and impactful.", "strength_and_weaknesses": "Strength：\n- This paper is well-written and the figures are demonstrative, making it easy to follow. \n- The proposed method is quite simple yet effective. I appreciate the intuition of reducing co-adaptation to enhance the compositionality of the emergent language. \n- The main results and the ablation study clearly show the effectiveness of the method and the influence of all the components and factors.\n\nWeakness:\n- Two communication graphs are adopted in the experiments, as shown in figure 2. However, the fully-connected graph and the circular graph are the two extreme cases. Maybe it is necessary to add a randomly sampled graph as a reference.\n- The experiments are all conducted on the referential games, where the receiver only needs to pick a candidate from a set as the prediction of the original object. What if we extend the game into a sequential decision problem? For example, the sender may send messages as the instructions, and the receiver needs to achieve some goals (such as navigation to the goals) by following such instructions. I wonder whether the proposed method can be applied to these games.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gcw6lBK1CV", "reviewer": "Reviewer_2JSn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The contribution of the paper is twofold: The paper explores if the “standard training protocol” in multi agent communication exhibits features observed in natural language, namely, if agents are able to communicate with new partners within the same population, and if lager populations tend to develop more structured languages. Authors show that, it is not the case. Moreover, they show that objective function of the standard training approach for multi agents is the same as that of a single pair of agents due to receivers co-adapting to their training partners. Second, authors propose an alternative training approach which partitions sender-receiver pairs which limits co-adaptation of receiver agents. They show that this alternative approach maximizes an objective function that involves two parts:\none function to maximize the “internal communication” between pair of agents and another function to maximize the “mutual intelligibility” across agents. They empirically show that this partitioning approach increases the effect of population size on compositionality and allows emergence of a more structured language, thus allowing to exhibit those features in natural language.\n\n\n", "summary_of_the_review": "I believe the authors have obtained some interesting results to explore and answer an interesting problem. However the flow and the explanations could be improved. In my opinion, how the results can be connected at large could be discussed more.\nI have made some suggestions to improve the understandability of the paper. After revision paper can be accepted for publication.\n", "strength_and_weaknesses": "\nThe authors study an interesting and motivating problem. Analysis is done to explore the objective functions in both cases: standard and  partitioned. They have performed number of experiments to analyze different hypothesis and to showcase the claims and the results are compelling. Provides how the work is related to other work.\n\nI like the organization of the paper (sections containing individual task.) However, the structure can be improved. Here are some suggestions.\n\nIn the introduction, adding what you do in each section. This will help the readers to better understand the flow of the paper and your results.\nAdding a discussion explaining your results and what the broader connections are. You have explained that \"standard procedure\" do not showcase the features of the reality. You could consider elaborating on that and explain more on how partitioning allows those features (of course you have explained them- you could consider elaborating and being more precise ), how scaling up to populations can be beneficial and how the way it is done matters. What are the implications and broader connections to natural language? Consider explaining how \"simple design choices can have a great impact on the training dynamics and ultimately the effect of population on the emergent language\"\n\nConclusion looks a bit vague. \" exhibit some of the properties\",\"Motivated by an analysis of populations at the equilibrium\"-be a bit explicit.\n\nAdd future work.\n\nAll the points made above are suggestions, and I would defer to the authors who best know their work and are in the best position to decide how to make their work more accessible.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "334iE7qEZo", "reviewer": "Reviewer_ULLE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of using population-based training to learn more \"natural\" languages for communicating between agents.  When two agents (a \"sender\" and a \"receiver\") are trained to solve a communication task together, the communication protocol, or language, that they learn often lacks structure, and simply maps arbitrary strings of symbols to different objects.  Surprisingly, training populations agents, and randomly pairing senders and receivers, does not lead to the agents converging to a more significantly structured language than training a single pair of agents.  This paper argues that this is due to the fact that no matter how large the population, the population-level training objective can be maximized by a single receiver policy that simply memorizes the (arbitrary) languages of each sender.\n\nTo address this issue, this work modifies the population-based training procedure so that while senders are rewarded for being able to communicate with multiple receivers, each receiver is only required to understand communication from a single sender.  This prevents receivers from directly optimizing their behavior with respect to the average behavior of the senders, while requiring that the senders be interpretable by many distinct receivers.  The experimental results demonstrate that the under this \"partitioned\" training paradigm, receiver agents are able to correctly interpret communication from senders that they were never paired with during training (which is not observed without partitioning).  Perhaps more importantly, the languages learned by the senders demonstrate significantly more structure (as measured by the topological similarity between the language and the underlying object space), than those learned without this partitioning mechanism.", "summary_of_the_review": "Overall the work provides useful, and somewhat surprising, insights that have not been previously published.  It also demonstrates that a simple modification to population-based training can substantially change the nature of the solutions learned in cooperative communication tasks, which raises the question of whether similar changes might be useful for population based training in other multi-agent settings.", "strength_and_weaknesses": "The main contribution of this work is that it provides a convincing explanation for the somewhat counterintuitive phenomenon that population-based training does not more structured and less arbitrary synthetic languages in cooperative communication tasks.  They also present a straightforward solution to this issue through a simple modification of the training process, and demonstrate that this recovers the behavior that was originally expected from population-based training.  There are potential applications of these ideas to other topics such as zero-shot coordination in RL or human-AI interaction more generally.\n\nThe main weakness of the work is that the value of learning more \"structured\" languages in this setting is not fully articulated.  Practical use cases for this training setup, even if only hypothetical, would be helpful in understanding how we should evaluate emergent languages relative to one another.  One example might be to use an emergent language to evaluate to train an agent designed to adapt to existing languages (eg. a particular human language) in an online fashion.  Another might be to design visual communication protocols for user interfaces that are structure in a way that makes it easy for humans to interpret them.  The lack of such motivation means that we cannot determine how useful metrics such as topological similarity really are as proxies for performance in realistic applications.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster