Revisiting Populations in Multi-Agent Communication
Authors: Paul Michel
Mathieu Rita
Kory Wallace Mathewson
Olivier Tieleman
Angeliki Lazaridou
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Emergent Communication
Multi-agent Communication
Populations

Abstract: Despite evidence from sociolinguistics that larger groups of speakers tend to develop more structured languages, the use of populations has failed to yield significant benefits in emergent multi-agent communication. In this paper we reassess the validity of the standard training protocol and illustrate its limitations. Specifically, we analyze population-level communication at the equilibrium in sender-receiver Lewis games. We find that receivers co-adapt to senders they are interacting with, which limits the effect of the population. Informed by this analysis, we propose an alternative training protocol based on ``partitioning'' agents. Partitioning isolates sender-receiver pairs, limits co-adaptation, and results in a new global optimization objective where agents maximize (1) their respective "internal" communication accuracy and (2) their alignment with other agents. In experiments, we find that agents trained in partitioned populations are able to communicate successfully with new agents which they have never interacted with and tend to develop a shared language. Moreover, we observe that larger populations develop languages that are more compositional. Our findings suggest that scaling up to populations in multi-agent can be beneficial, but that it matters how we scale up.

Revisiting Populations in Multi-Agent Communication
Anonymous Author(s) Affiliation Address
email
Abstract
Despite evidence from sociolinguistics that larger groups of speakers tend develop more structured languages, the use of populations has failed to yield significant benefits in emergent multi-agent communication. In this paper we reassess the validity of the standard training protocol and illustrate its limitations. Specifically, we analyze population-level communication sender-receiver Lewis games. We find that receivers co-adapt to senders they are interacting with, which limits the effect of the population. Informed by this analysis, we propose an alternative training protocol based on "partitioning" agents. Partitioning isolates sender-receiver pairs, limits co-adaptation, and results new global optimization objective where agents maximize (1) their respective "internal" communication accuracy and (2) their alignment with other agents. In experiments, we find that agents trained in partitioned populations are able to communicate successfully with new agents which they have never interacted with and tend develop shared language. Moreover, we observe that larger populations develop languages that are more compositional. Our findings suggest that scaling up to populations in multi-agent communication can be beneficial, but that it matters how we scale up.
the equilibrium in
10
11
13
14
15
17
Introduction
Uncovering the mechanisms that underlie ability communicate using language is important stepping stone towards developing machine learning models that are capable of coordinating and interacting via natural language. Over the last few years, there has been increasing interest in simulating the emergence of language using artificial agents trained with reinforcement learning to communicate achieve cooperative task (33] Typically, agents trained perform variant of the Lewis signaling game [37] 5 wherein a sender emits a message describing an object and receiver attempts to reconstruct the object based on the description. This line of work has applications
20
22
23
24
25
26
to semi-supervised learning applied to concrete tasks such as image captioning representation learning [36]]]]]]] [8].
27
28
Most previous research has focused on communication between a single pair of agents. However, there is mounting evidence that the communication protocols developed in this restricted setting hecome highly specialized and exhibit properties that are at odds with those found in human languages for example agents are able to solve the task successfully while using languages that not compositional [32
30
31
:
32
9 As possible solution, a growing body of work is advocating for scaling agents communicating 49 10). Indeed, there is substantial evidence within the language sciences that population dynamics shape the language structure 47 42 In spite of this fact, several negative
33
up the emergent communication literature populations of more than
34
simultaneously 244
30
35
36 results have been obtained, showing that training agents in population yield marginal benefits without explicit pressure towards e.g. population diversity 49 or emulation mechanisms [10]
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38 In this paper, we call into question the way such populations are trained. By studying a simple
39 referential game, we evaluate populations on two desirable features observed in natural language:
Agents are able to communicate with partners within the same population 23
41
Larger populations tend to develop more structured languages 422
We provide evidence that populations of artificial agents do not always possess these features (as also attested by previous work, e.g. Kim and Oh [30], Chaabouni et al. [TO). To shed light on this phenomenon,
43
analyze the behaviour of agents in a population at the equilibrium. We find that with the standard training procedure, the functional form the objective is the same as that of a single pair of agents, due to receivers co-adapting their training partners. As main contribution, we propose an alternative training procedure which partitions sender-receiver pairs and limits co-adaptation of receiver agents. We show that this new training paradigm maximizes different objective at the population level. In particular, it explicitly promotes mutual--ntellggiiiltt across different agents.
45
47
48
49
In experiments, find that agents trained in partitioned populations are able to communicate 51 successfully with new communication partners with which they have never interacted during training, and that languages spoken by various agents tend to be similar to one another. In addition, we observe 53 that (1) languages developed in partitioned populations tend to be more compositional and (2) there a population size effect whereby larger populations develop more structured languages. Our results
55 show that there are multiple ways to generalize from single agent pairs to larger populations, and that these design choices matter when it comes to studying the emergent language.
56
57
2
Communication Game
58
We study communication in referential games, a variant of the Lewis signaling game [37 proposed (34). The game proceeds follows: during each round, sender agent observes an object x E X' (e.g.. an arbitrary categorical entity, or a natural images) sampled from input
59 by Lazaridou et al.
60
61 space X' according distribution p and generates a message m ~ T- 2:). Messages consist of
82 variable length sequences of tokens picked from a discrete vocabulary V. Note that the tokens themselves are arbitrary and meaningless (typically they are represented as numbers from 1 [VD. A receiver agent then observes message m and must predict the original object from among a set of candidates C (a, //..... //.....//--11 containing and [C] distractors, where each distractor y is sampled uniformly without replacement from the input space excluding the original object, X' {x)) Concretely, this is implemented by caleulating score f(y, m) for each candidate y and defining ef(..))
63
64
65
67
68
the probability of candidate conditioned on the message p(- m, C) as Evec Fy.mj Based on the receiver's success, the sender agent receives reward R(x,, p(- m, C)).
69
70 In practice, both senders and receivers are implemented as neural networks To and Py with parameters 0 and va estimated by gradient descent. The sender is trained to maximize its expected reward using the REINFORCE algorithm [57], while the receiver maximizes the expected log-likelihood of identifying the original object, log Ps (a m, C) (also known the InfoNCE objective; Oord et al.
71
72
74 45).. Denoting as Ezpp the expectation over x sampled from p, the corresponding training objectives
are:
(0) 1ap Bm (-1x) Ec-p R(x, p mm,C) C)) J+(+)) Ezzp Em Eeep log (1 m.
(2)
76
2.1
Population Level Training
The two-player referential game can be generalized to larger populations of agents [4] 10.. In the most general case, we consider a population of N, senders and N, receivers that are linked by bipartite communication graph G defining connections between senders and receiver (TT, Peş)
78
80
24 30] At training time, sender-receiver pairs are repeatedly sampled and trained to perform a round of the game. Importantly, only agent pairs that are connected in the communication graph
81
82
sampled. Throughout this paper, we will refer to this type of training as Standard training.
85 With this training procedure, agents are trained to maximize their communicative success with all
84 their neighbors the communication graph. Let Wo(i) refer to the neighbors of the i-th node in the graph, and Jajioy (respectively Jr,44) denote the objective of тe, (respectively Pey )) in the

---

86
pairwise communication from sender i to receiver j. We can write the overall objective for sender i (and receiver j. respectively) as:
87
(8,) [Wa(1)) jENo(()
E (O,) and Jr,(W,z) (Na(i)) iENo()
E (1))) (3)
At test time, the population is evaluated by averaging the referential accuracy across all possible sender-receiver pairings. Following previous work, in this paper we focus on populations with an equal number N = N₂ = N, of senders and receivers, meaning that there are up N² possible
89
90
81 pairings.
92
2.2
What does Population-level Training Optimize?
93
To shed light on the differences between training a single agent pair and training a population of agents, we analyze the objective optimized by the population. Inspired by M's analysis in the two-player case, we study the behaviour of the population at the optimum, that is when senders and receivers have reached a Nash equilibrium [6
94
95
96
97 In this section, we make the simplifying assumption that C = X. In other words, receiver agents must pick the correct candidate out of all possible objects in X. This allows us remove the conditioning on C and write ( m. C) p P (ær m.) We make this simplification to reduce clutter in notations. for detailed discussion).
98
89
100
Nevertheless, our key observations still hold for C # X' (see Appendix B
101
At a Nash equilibrium, the optimal receiver parameters 36 satisfy
Pej arg max Jr (bb) arg max
1
E (ENGO)
Jr
(4)
[Na
102
Assuming that receiver Pe, has high enough capacity, and training is able to reach the global optimum, the solution of the optimization problem in Equation 4 has an analytical solution Pej which can be
103
written function of a (m x)) [Nati LieNo() Top (m xr), the mixture of all senders
105
communicating with receiver
T TNo (m x))(x))
Pay(x m) TNo()) (a m) Eysp Noo (m v)
106
In other words, P0; the posterior associated with x}o(() (full derivation in appendix A
An important implication of this result is that when the population graph is fully connected (all senders are connected to all receivers), each receiver converges to the same optimum x (x m)) =
108
Ee me, (m]=))p(r))
109
Eyop Diz Te, (m|z] the posterior of the mixture of all senders in the population. Plugging this back into each sender's objective, we have Jz,; = Enpp Em (+1x) R(x, nt
110
m))
Summing across all senders, can rewrite the global objective optimized by the senders as
max Ezcp EmNnT R(x, x (m)m)..
(5)
112
In other words, at the equilibrium, the population maximizes the expected reward of the "sender ensemble" n", rather than that of individual agents z the objective of population N agents functionally the same irrespective of N. We postulate that this indifference to the population size may account for the surprising lack of effect of larger populations observed in some previous work 10]. Differences in behaviour must be attributed to effects stemming from training dynamics (e.g. it becomes more difficult for receivers to learn the posterior π (x m)), or be imposed through extraneous modifications of the population objective (for example explicit imitation components; Chaabouni et al. (10).
113
115
49
117
119
120
A second observation is that there is no direct pressure for agents that communicate at training time to develop the same language. Indeed, it is entirely possible that all senders develop different but non-overlapping languages: it suffices that no two senders communicating with a shared receiver use the same message m to describe a different object. this case receivers can simply learn their neighboring sender's languages and there is no need for the senders to converge to unified language.
121
123
3

---

Sender gradiant (V
of
n
Sander
A
Recaiver
Receiver gradient (V (2
Ps
PP
TTO
Figure I: the standard setting (left hand side), both receivers (in blue) are trained by maximizing their discrimination objective with respect to both senders. With partitioning, receiver pv, (resp. Pv.) is only trained to maximize its communication objective with sender тe, (resp. Te,)
Partitioning Agents
125
A key difference between the usual population setting and populations of humans in laboratory experiments that agents are not usually split into "senders" and "receivers". Rather, each participant in the experiment assumes both sender and receiver role [21]. Our hypothesis is that, counter to what customary in the emergent communication literature, tying senders and receivers is key in surfacing useful population-Iewe dynamics in multi-agent communication.
127
129
131
To operationalize this sender-receiver coupling, we identify an "agent" as sender-receiver pair. During training, we only train receiver Pe, with its associated sender To,- In other words, (r.( (t):). doing so, we "partition" the agents by preventing receiver from co-adapting to other senders j * i. This procedure is illustrated in Figure [ Note that senders can nevertheless still train with rewards from neighboring receivers, and so communication across agents can still emerge. Importantly, partitioning prevents receivers from learning recognize multiple languages, they are now only trained on messages emitted by a single sender.
132
133
135
137
138
Following a similar analysis as Section 2.2, we derive that at the optimum, receiver Pvr (x m) now
takes the form of the posterior associated with its respective sender, Te: (x m) zaz (mlx))(() (derivation in Appendix A). We can thus write the population-level objective at the equilibrium as
139
Eysp Toix
Bzop Bo (-))) R(x, To, m)) +
Bzep Em (1a) R(x, Toy (: m))
(6)
N
jEVa(()
Internal
intelligibillity
Note that the functional form of the objective can now be decomposed into two parts: "internal communication" objective which takes the same form as that a single pair of agents, and a "mutual able communicate successfully.
142
intelligibility" objective which enforces that neighboring agents In experiments, we show that this explicit pressure towards mutual intelligibility promotes the emergence of single language within the population, which in turn enables agents to communicate with new partners outside their training neighborhood.
144
146
Experimental Setting
147
4.1
Datasets
We perform experiments on two datasets: a simple, synthetic "attribute/values" dataset and more realistic image dataset.
150
In this dataset, each object is represent by a collection of abstract "attributes", Specifically, each input x is vector attributes, each of which can take 10 total values. This results in a0a total attribute/value combinations 32 9. In each setting we hold out 1, 000 combinations 154 to used as validation, and 1, 000 more for use as test set. We can thus ensure that we are evaluating the agents' ability to generalize to unseen combinations of attributes.
Attribute/Values
152

---

156
ImageNet In addition to toy objects, we perform experiments with referential games based on more realistic objects. Following Chaabouni et al. [TO), we use the ImageNet [17] dataset of natural images. The dataset consists of about 1.4M training images collected on the internet and annotated for 1 1,000 labels from the WordNet database [40]. Images are first encoded as 2048-sized real-valued vectors with (frozen) ResNet pre-trained with BYO [2 before being passed to sender and receivers.
157
158
159
160
4.2
Game Architecture
Both sender and receiver agents are based on layer LSTMs 226 with embedding and hidden dimensions of size 256. Specifically, the sender first encodes the object x into a vector of size 256, which is concatenated the input of the LSTM. At each step, the output of the LSTM cell is passed through a fully connected layer to produce logits of size IV. A softmax function is then applied to obtain normalized probabilities over the vocabulary. During training, messages are generated by sampling from the distribution whereas at test time we generate messages deterministically via greedy decoding. In both cases, generation stops whenever special "<EOS>" is generated, or when the number of tokens reaches fixed limit L.
163
165
166
167
168
189
The receiver encodes the message with an LSTM encoder, the output of which is the fed into fully connected layer to yield a vector of size 512. The candidate objects are then scored by computing the dot product of this vector with 512-dimensional encoding of each candidate. The conditional distribution over candidates is then obtained by taking a softmax. We set the reward function for the sender to the log-liieldhood assigned by the receiver to the correct candidate, R(x, ( m)) = log Ap [ m).
171
172
173
175
176
Throughout all experiments, we set the vocabulary size [V] to 20 and the maximum length of the messages, T., to 10. This means that the communication channel used by the agents has a capacity of about 20t0 which ensures that there is communication bottleneck (the size of the channel is several orders of magnitude larger than the size of our datasets). Our implementation, based on the will be open-sourced upon de-anonymization.
177
179
EGG toolkit 29
181
4.3
Population training
182
We train populations following the procedure outlined by Chaabouni et al. [10] for each minibatch of data, we sample K pairs from the population (uniformly among the pairs linked in the communication graph). Each pair plays an episode of the game, and the agents are updated simultaneously following the gradients of their respective objectives. We take K = max (10, N) to ensure that each agent plays the game at least once at every step on aver- age. This procedure needs to be modified for partitioned populations: since receiver j is only with its respective sender instead of with all of its neighbors, there is now only Ne(d) chance that receiver j will be updated every step (the probability that the pair (j.j) is sampled). For larger populations, especially those that are fullycconnected, this dramatically slows down training as receivers are updated very infrequently. To address this issue, we modify the procedure as follows: for every sampled agent pair (To, Pv.). we calculate both Je, and Jr,i+4 and update both Te and Pv. Note that this necessitates calculating both Pe; (ar m, C) and p (x: (,,)) and therefore we incur a small computational overhead. However we only observe a 5% increase in training time due to the fact that we are back- mack-propagating through only one of the two receivers, Pvs (r m, C).
183
184
185
186
187
189
191
(a) cully-coneected
(b) Circular
Figure 2: Example of communication graphs used in this paper
193
194
195
197
198
199
200
201
With this modification, we recover the property that each agent (sender or receiver) updated once
202
every step on average.
train with batch size of 1024 with the Adam optimizer 31 rate of 0.001 for the attribute/value dataset and 0.0001 for Imagenet. The other parameters are set to
In all experiments
using learning
204
31 0.9, 82 0.999 and 10-8 We apply & regularization with coefficient of 10-5
205
206 We systematically augment the sender objectives with an entropy maximizing term, which has been found to encourage exploration [58). The coefficient for this entropy term is set to 0.1 in all
5

---

Table Accuracies with training partners and new partners on both datasets. Numbers are reported with standard deviation across all pairs for independent experiments
ImageNet
Attribute/Valuee Values
Standard
Partitioned
Standard
Partitioned
Training partners
97.09 11.10
99.75 + 0.08
99.88 1 0.15
99.81 0.22
New partners
5.41 13.57
96.24 = 3.25
7.81 = 18.28
40.37 = 29.44
Table 2: Language similarity between training partners and new partners on both datasets. Numbers are reported with standard deviation across all pairs for 3 independent experiments
ImageNet
Attribute/Values
Standard
Partitioned
Standard
Partitioned
Training partners
0.28 + 0.07 0.22 + 0.19 0.37 0.15
0.40 0.02
0.28 = 0.05
0.36 + 0.01
New partners
0.23 = 0.19
0.31 0.17
208
experiments. To reduce the variance of the policy gradient in REINFORCE, we substract baseline computed by taking the average reward within given mini-batch for each pair [54].
We evaluate the population every epoch (every 5 epochs for the Attribute/Value dataset) the validation set. We only evaluate on up to 100 unique pairs sampled uniformly within the population, this time without consideration for the communication graph. We train for fixed number of epochs, selecting the best model based on the average validation accuracy across all evaluation pairs.
211
213
5
Communication with New Partners
215 In our first set of experiments, we evaluate the ability of agents trained in populations to communicate with partners they haven't interacted with during training.
216
217
5.1
Circular Populations
Specifically, study "circular" populations of agents arranged ring lattice. Each agent (sender- receiver pair) is only trained with neighboring agents i 1,... i 1 and the graph is cyclical (see Figure 2b). We choose this type of population because is extreme case of a population where each agent has the same, minimal amount of neighbors (two), yet there is still path between any two agents. In this context, training partners sender-receiver pairs that connected in the graph and have interacted during the training phase whereas new partners refers to pairs that have not interacted during training.
219
221
223
224
225
We report results along two metrics:
Communication Accuracy of sender/receiver pairs on an evaluation set. This measures how successful the pair is in communicating.
227
228
Language Similarity between senders. This metric (also called synchronization in Rita et al. (49] is calculated two senders, averaged across all objects in our evaluation set.
229
ös.5 where 6,,, the normalized edit distance between messages output by
231 We report these metrics for both training partners and new partners. Note that high communication accuracy does not always entail similar languages: it is possible for the receivers achieve high accuracy despite all senders sending different messages for any given object (it is only necessary for given message unambiguously refer object across senders).
233
235
5.2
Partitioning Enables Successful Zero-Shot Communication
we report accuracies and similarities for circular populations of 20 sender-receiver pairs trained on ImageNet and the Atribute/Valun dataset. All metrics are calculated on the test set and averaged across 3 independent experiments.
In Table T and 2
237

---

100
standard partitioned
Accuracy
Similarity
60
standard partitioned
0.3
1
0.2
0
1
2 3
5
6
7
10
0
2
4
6
8
9
10
Distance
Distance
(a) Accuracy
(b) Similarity
Figure 3: Accuracy and language similarity as a function the distance between two agents in the communication graph.
100
80
Distance
60
40
10
0
25
50
75 epoch
100
125
150
140
epach
(a) Standard
(b) Partitioned
Figure 4: Evolution of validation accuracy during training across agent pairs at various distances in the communication graph. Results aggregated over all agent pairs and populations.
We observe that in populations following the standard training paradigm (Standard), there is stark discrepancy between training and new partners. Indeed, on both datasets the accuracy with training partners reaches very high value, above 95%. Yet, the accuracy when agents communicate with new partners drops down less than 10%. On other hand, in Partitioned populations, agents reach much higher accuracy with non-neighbors, up 96% ImageNet and 40%. A similar trend holds for language similarity.
240
242
244
245
Note that all metrics on new partners exhibit high standard deviation. An explanation is that among non neighborigg pairs there a different behaviour depending on how far the two agents are the population. This is verified in Figure which displays breakdown as a function of the distance between two agents in the communication graph (on ImageNet). We find that without partitioning, accuracy drops off sharply to close to () for agents at distance > 2, whereas it decreases almost linearly with the distance in the partitioned case, down about 95% for the most distant agents.
246
247
248
250
5.3
Training dynamics
252
We further investigate the evolution of accuracies during training. In Figure 4 we plot the evaluation accuracies of both standard and partitioned populations broken down by distance between pairs, focusing on the ImageNet dataset. Note that there are two training phases in the standard case. Up to epoch & 10, the accuracy for all training pairs increases, after which agents over-fit to their training partners (distances 0 and I) and the accuracy on other pairs decreases to a plateau.
254
255
296
257
On the other hand, Figure 4b illustrates the pressure for mutual-intelligibiltty in partitioned popula- tions: accuracy between training pairs reaches close 99% accuracy (around epoch 20). accuracies across distant pairs increases rapidly before plateauing above 90%. In fact. our results show that the most distant accuracies are still increasing after 150 epochs, albeit very slowly.
258
259
280
By construction, the similarity of sender with itself (corresponding to distance of 0) is always one. We omit this value from the figure better illustrate the trends for distance 2 1.
7

---

Topographil similorty
Tapographic similariy
40
Topographi simillrity
40
38
35
36
24
sandard
23
Hared
2.45
10
25
0.4 0.6 Co-adaptation probability
0.0
0.1
0.2
0.3
0.4 0.5
Mutual Intelligibiliy relative weight
(a) Topographic similarity as fune- (b) Topographic similarity with (c) Topographic similarity when ab- tion of population size on an at- varying degrees of partitioning (pop-
lating the mutual intelliiilll term (populations of size 10).
tribute/value communication game. ulations of size 10).
Figure 5; Influence partitioning on the topographic similarity of the emergent languages.
6 Partitioned Population Develop More Compositional Languages
262
In this section, we investigate the effect of partitioning on the structure of the language, with a focus on compositionality
283
284
6.1
Measuring Compositionality
285
A language is said to be compositional when the meaning of a whole utterance can be systematically deduced from the meaning of its components (i.e. words). The notion of compositionality is widely
266
267
construed to underlay the near infinite productivity of human languages [55]
common metric for measuring compositionality in emergent languages is the topographic similarity Topographic similarity captures the intuition that a compositional language will map similar "meanings" similar messages: the phrase "a red bird" is more similar to the phrase "a blue bird"
269
5
35
271
than to "a powerful computer". In practice, the topographic similarity is computed by measuring the Spearman rank correlation coefficient (52) between (I) the pairwise distances across all objects and
272
273
(2) the pairwise distance across all messages.
274
6.2
Effect of Population Size on Compositionality
275
We run experiments on our Attribute/Values dataset, with both standard and partitioned populations that are fully-connected (see FigureRai Population sizes range from 2 to 25 sender-receiver pairs.
277
We compute topographic similarity using the Hamming distance in the object space (i.e. the distance between two objects is the number of attributes in which they differ) and the normalized edit distance between messages.
279
280
In Figure Sa we observe that while standard population-level training does increase the topographic similarity of the language overall, population size has very little effect: populations of sizes 3 and 20 both reach about the same value of 30 on average. On the other hand, partitioning greatly increases the effect of population size on compositionality; populations of size 20 have a significantly higher topographic similarity than populations of size 5, with a zo 10 points difference.
281
282
283
6.3
Co-adaptation is Responsible for the Decrease in Compositionality
286
Up until this point, we have described partitioning (or lack thereof) as binary choice. However, possible to partition population only partially, by allowing receiver j to train with senders i
288
occasionally with probability a > 0. In doing so, the optimal receiver now becomes the posterior associated with mixture between πe; (m æ) and nT" (m 2) (see Appendix A for the derivation). If < a < 1, receivers are now optimizing for a different objective (as in partitioned populations), but some amount of co-adaptation is still allowed.
290
292 We perform this experiment on the Attribute/Values dataset with fully connected population of
size 10, varying the degree of co- co-adaptation ranging in {0, 0.1, 0.5. 0.9, 1}. 0 corresponds
294
to partitioned training whereas a = equivalent to standard training. All populations converge
to > 99% accuracy. However, in Figure 5b we find that topographic similarity drops as soon we
296
introduce minimal amounts of co-adaptation (a 0.1) and decreases steadily to the level of standard
populations as grows to 1. This further corroborates hypothesis that reducing co-adaptation

---

298
promotes the emergence of a more structured language, and that eliminating it altogether (in a partitioned population) yields the best results.
299
300
6.4
Importance of Mutual Intelligibility
301
Recall that the objective of a partitioned population at the equilibrium (Equation can be decomposed in two terms: an "internal communication" corresponding to the single agent pair objective and "mutual intelligibility" term which encourages senders to align their languages. Importantly, the latter
302
303
304
is the only element that separates a partitioned population from a collection of isolated agents.
305
To measure its effect the compositionality of the emergent language, we train fully connected populations of size 10 and decrease the relative weight of the mutual intelligibility term. This is implemented by making the pair (To,, (xo,.Po.) more likely be sampled than other pairs (Fo,, PO;),
306
307
308 j * i by a factor x 1-8
D We let 33 range from 0.5 (partitioned population) to 0.0 (collection of isolated sender-receiver pairs). In Figure 5c, we find that emergent languages retain high topographic
309
310
similarity even at small 3. and the sharp drop-off occurs only when is very close to O. This confirms
311
that the mutual intelligibility term exerts a strong pressure towards compositionality. We investigate the evolution of the two terms during training in Appendix C
312
Related Work
313
There is rich history of modeling the emergence of language the solution to cooperative game
315
that can be traced back to functional theories of language [59 2 13]. With a regain of interest for the study of language evolution [15 12,, a rich literature has developed around computational
317
simulations of the emergence of language based on simple language games [37 5 3 6. Examples
318
include studying evolutionary models of the emergence of grammar [44), the influence of cultural transmission
[5] game theoretical considerations (27 or linguistic diversity (39 among others.
319
The recent success of deep learning in natural language processing has spurred interest in studying signaling games between deep neural network trained with reinforcement learning to solve a signaling game [34] 20). Several upp have taken this idea further by extending it to more complex games
321
323
or environment [53] 25| 28 [6] or by adding an element of competition [50 43) or negotiation [7] or
even explicit pressure towards certain desirable properties (32]
48 In parallel, several efforts [1 38
325
have been made to understand the properties of the emergent languages
4 8.
Within this growing literature, multiple authors have explicitly studied the use of populations of more than two agents. Various works have argued for augmenting populations with an explicit pressure towards more structure languages, via e.g. generational transmission [1], adversarial regularization
327
329
(6), varying learning speeds [49] or imitation learning and voting [TO]. Although the focus is often on fully-connected populations, some authors have also explored more complex communication graphs, for the purpose of modeling contact linguistics (24) or the effect of social network structure on the language [19]. Recent work from Kim and Oh [30] is perhaps closest to our own: the authors study the effect of population size and connectivity in the standard training paradigm. In contrast, the purpose of this paper is highlight the impact of the training procedure these very effects.
331
333
334
Conclusion
335
Empirical findings in socio-linguistics suggest that population dynamics should help in simple sender-receiver communication games. In this paper, we observed that populations trained by naively extending the simple 1-1 protocol to N X N agent pairs fail to exhibit some of the properties that are observed in human populations. Motivated by an analysis of populations at the equilibrium, we described an alternative training paradigm, based agents partitioning to reduce co-adaptation. Empirically, we find that partitioning enables us to recover some of the aforementioned properties.
337
339
341
342
Our findings call attention to the fact that there is more than one way to generalize two single to many agents, and simple design choices can have a great impact on the training dynamics and ultimately the effect of population on the emergent language. Beyond emergent communication, we hope that this observation can inspire similar work in other cooperative multi-agent problems where co-adaptation between agents may counteract population effects.
343
345

---

References
347
348
[1] Anonymous. Emergent communication: Generalization and overfitting in lewis games. Under Review, 2022.
349
[2] John Langshaw Austin. How to do things with words. Harvard University Press, Cambridge,
351 MA, 1962.
352
[3] John Batali. Computational simulations of the emergence of grammar. Approaches to the evolution of language: Social and cognitive bases, page 405, 1998.
353
354
[4] Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an emergent language game. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 98 1 1985, 2018.
355
356
357
[5] Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life, 12(2):29-2422, 2006.
358
[6] Angelo Cangelosi and Domenico Parisi. Computer simulation: A new scientific approach the study of language evolution. In Simulating the evolution of language, pages 3-28. Springer, 2002.
360
77] Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. Proceedings of the International Conference on Learning Representations (ICLR), 2018.
363
385
[8] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficien encoding in emergent communication. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
366
388
[9]
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 8th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4427-4442, 2020.
389
370
371
372
[10]
Rahma Chaabouni. Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
373
374
375
376
[1]] Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Compositional obverter communica- tion learning from raw visual input. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
377
378
379
[12] Morten H Christiansen and Simon Ed Kirby. Language evolution. Oxford University Press, 2003.
[13] Hebert Clark. Using Language. Cambridge University Press, Cambridge, UK, 1996.
382
[14] Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of compositional language with deep generational transmission. arXiv preprint 07X7v.1904.09067, 2019.
383
385
[15]
Vincent P Crawford and Joel Sobel. Strategic information transmission. Econometrica: Journal of the Econometric Society, pages 1431-1451, 1982.
306
387
[16]
Abhishek Das. Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 1538-1546. PMLR, 2019.
388
389
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the 22nd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. 2009.
391
382
[18] Roberto Dessi, Eugene Kharitonov, and Marco Baroni. Interpretable agent communication from scratch (with a generic visual processor emerging on the side). arXiv preprint arXiv:2106.04258, 2021.
394
[19] Marina Dubova, Arsenii Kirillovich Moskvichev, and Rob Goldstone. Reinforcement communi- cation learning in different social network structures. In Language in Reinforcement Learning Workshop 2020, 2020.
397
10

---

389
[20]
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learning. In Annual Conference on Neural Information Processing Systems (NIPS), volume 29, 2016.
400
402
121
Lukas Galke, Yoav Ram, and Limor Raviv. Emergent communication for understanding human language evolution: What's missing? In Emergent Communication Workshop at ICLR 2022, 2022.
403
404
405
[22]
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Proceedings of the 34th Anmual Conference on Neural Information Processing Systems (NeurIPS), volume 33, pages 21271-21284 2020.
406
407
409
410
[23]
Abhinav Gupta, Mare Lanctot, and Angeliki Lazaridou. Dynamic population based meta- learning for multi-agent communication with natural language. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, 2021.
411
412
[24]
Laura Harding Graesser, Kyunghyun Cho, and Douwe Kicla. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3700-3710, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19. 1384. URL Estps://aclanthoolle
414
416
Dirg/D191133 1384
418
[25]
Serhii Havrylov and Ivan Titov. Emergence of language with multi games: Learning to communicate with sequences of symbols. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS), volume 30, 2017.
419
420
421
[66]
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
422
423
[27]
Simon Huttegger, Brian Skyrms, Pierre Tarres, and Elliott Wagner, Some dynamics of signaling games. Proceedings of the National Academy of Sciences, 1 111(Supplement 1):10873-10888 2014.
424
426
[28]
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse. Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In Proceedings of the 36th International Confer- ence on Machine Learning (ICML), pages 3040-3049. PMLR, 2019.
427
428
429
[29]
Eugene Kharitonov, Roberto Dessi, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: a toolkit for research on Emergence of lanGuage in Games. suttps://githie com/
431
432
facebookressercch/E60 2021.
433
[30]
Jooyeon Kim and Alice Oh. Emergent communication under varying sizes and connectivities. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, 2021.
434
435
436
[31]
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference Learning Representaatins (ICLR), 2014.
438
[32] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge naturally''n in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2962-29621 2017.
439
440
441
[33] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint ar arXiv:2006.02419, 2020.
442
443
[34] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
444
446
[35] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
447
448
449
[36] Anellkii Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agen Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 8th Annual Meeting of the Association for Computtational Linguistics (ACL),
451
pages 7663-7674 2020.
11

---

453
[37]
David Lewis. Convention: A philosophical study. John Wiley & Sons, 1969.
454
[38] Fushan Li and Michael Bowling, Ease-of of tfachchiinaagggggee and language structure from emergent com- munication. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
455
456
457
[39] Daniel Livingstone and Colin Fyfe. Modelling the evolution of linguistic diversity. In European Conference on Artificial Life, pages 704-708. Springer, 1999.
458
459
[40] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):
39-41, 1995.
[41]
Igor Mordatch and Picter Abbeel. Emergence of grounded compositional language in multi- agent populations. In Proceedings of the 32nd Meeting of the Association for Advancement of Artificial Intelligence (AAAI), volume 32, 2018.
462
463
[42] Jonas Nölle, Riccardo Fusaroli, Gregory J Mills, and Kristian Tylén. Language as shaped by the environment: linguistic construal in a collaborative spatial task. Palgrave Communications,
485
6(1):1100, 2020.
467
[43] Michael Noukhovitch, Travis LaCroix, Angeliki Lazaridou, and Aaron Courville. Emergent communication under competition. In Proceedings of the 20th International Conference on Autonomous Agents and MulliAgent Systems, pages 974-982, 2021.
468
469
470
[44] Martin A Nowak and Natalia L Komarova. Towards evolutionary theory of language. Trends in cognitive sciences, 5(7):288-295, 2001.
471
2001.
472
[45] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1807.03 2018
473
474
[46] Martin J. Osborne and Ariel Rubinstein. Course in Game Theory. The MIT Press, 1994. ISBN 0262150417.
475
476
[47] Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Larger communities create more systematic languages. Proceedings of the Royal Society B, 286 (86(1907)20111126, 2019.
477
478
[48] Yi Ren, Shangmin Guo, Serhii Havrylov, Shay Cohen, and Simon Kirby. Enhance the composi- tionality of emergent language by iterated learning. In 3rd NeurIPS Workshop on Emergent Communication (EmeCom@ NeurlPS 2019). URL https://pupers. nips. cc/book/udvacess-in crunalaiffommtiionnnnneeesssssss 92221. 2019.
480
[49] Mathieu Rita, Florian Strub, Jean-Bastien Grill, Olivier Pietquin, and Emmanuel Dupoux. On the role of population heterogeneity in emergent communication. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
483
[50]
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
486
487
488
[51]
Brian Skyrms. Signals: Evolution, learning, and information. OUP Oxford, 2010.
[52] C Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 1904.
490
491
[53] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backprop- agation. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS), volume 29, 2016.
492
483
484
[54] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Proceedings of the 13th Annual Conference on Neural Information Processing Systems (NIPS), 12, 1999.
495
496
497
[55] Zoltán Gendler Szabó. Compositionality. In Edward N. Zalta, editor, The Stanford Encyclo- pedia of Philosophy. 2020. URL https://plato. stanf ord. ward.edd/arccae adtt/archives/aa1120000
496
cuttries/compositiion i Accessed: 2022-05-13.
500
[56] Olivier Tieleman, Angeliki I Lazaridou, Shibl Mourad, Charles Blundell, and Doina Precup. Community size effect in artificial learning systems. In ViGIL@@ NeurIPS, 2019.
501
502
[57] Ronald J Williams. Simple statistical soadient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-256 1992.
12

---

504
[58] Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241 1991.
505
506
[59] Ludwig Wittgenstein. Philosophical Investigutions. Blackwell, Oxford, UK, 1953. Translated by G.E.M. Anscombe.
508 Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
511
(b) Did you describe the limitations of your work? [Yes] The marginal computational
513
overhead incurred by training partitioned populations is described details in Section
515
(c) Did you discuss any potential negative societal impacts of your work? [No] As our work primarily focuses on artificial languages developed by simple agents, we do not expect any immediate negative societal impact.
516
517
518
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
520
2. If you are including theoretical results...
521
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
522
(b) Did you include complete proofs of all theoretical results? [Yes] Derivations are provided in appendices
523
524
3. If you ran experiments..
525
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [No] Code to reproduce our experiments will be released upon deanonymization.
526
527
528
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
529
530
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
532
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] Our work was carried out GPUs located on an institutional cluster. Each experiment runs on a single V100-32G GPU
534
535
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] The ImageNet paper was cited
536
537
538
(b) Did you mention the license of the assets? [No] We were not able to find the license of
ImageNet
540
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
541
542
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
543
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
545
546
5. If you used crowdsourcing or conducted research with human subjects...
547
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
548
549
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
551
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13

---

553
A
Derivation of the Optimal Receiver
We first prove general result from which the optimal receiver both in the standard and partitioned can be derived.
555
S56
General Case
557
Consider receiver j trained to maximize
Jr.(1;z) Σ
iEsenders
558
where Oi=1I... are arbitrary weights for the senders (we assume that the o; are positive and sum to one). We can rewrite the objective as:
559
Jng((b))
Σ aidrasny(15))
=E iEsenders
04 Ro (112) log (x m)
Note that by linearity of expectation can pass the oş weighted average the senders inside 561 of the expectation and rewrite the second expectation in terms the mixture n. (m I x) ==
582 Σ Liesendors TgTT; (m x):
(15()))) = Expo Em Bi=Eiicento 00 (m//2) log Pwù (o m)
Exxp Em (12z log Puty (2 m)
With slight abuse of notation, let us denote by T. (m) Ezsp (m x) the marginal distribution over messages and T (x m) = a (m(a)pp(()) the associated posterior. Notice that
564
mm)
rewrite the double expectation Egwp Emmwz (-la) as Emwaz()) (-) Bg (-Im) by inverting the order of summation. We can therefore rewrite
since by definition xf (m. | x))(x)) m (ar m)r
566
(W)) = Emwage) H(m [m) mm)
where H(p, q) denotes the cross-entropy E, |- log of two distributions p and g. Importantly the cross-entropy is non-negative and Hp g) if and only if p = q.
569
570
Consequently, the receiver Ps will be optimal (Jr.( = 0) if and only if for all m
T." (m x))(xx)
Pey (x m) (x m)
(8)
E Eyso a (m 9)
572
A.2
Optimal Receiver in Standard Populations
573
Recall that in standard populations, the training objective for receiver j is:
Jnn( [No() (ENc6)
Σ (1in))
574
Note that this a special case of Equation
7 with
e
Neti
a
otherwise
MMore accurately, if the message space is not finite then the condition holds not for all m. but almost surely. However throughout the paper experimenting with finite (albeit large) message spaces.
14

---

575
Consequently, the derivation in Section A.1 tells us that the optimal receiver is
TNoO) (m x))(())
Pag (I m) = T (N(()) (o m)
(9)
Bypp TNou) (m v)
Where (NaC) x) = [NGoll EieNot) Taz (m x))
576
577
A.3
Optimal Receiver in Partitioned Populations
578
In partitioned populations, the training objective for receiver is:
(+)) = (n5))
579
This is also special case Equation with
[ i
a; 0 otherwise
The derivation in Section A. thus yields the optimal receiver
Tz m x (2)
Pey (x m) = x (a m)
(10)
Ey-p x (m y)
A.4
Optimal Receiver in Partially Partitioned Populations
582
In the partially partitioned populations used in Section 6.3 each receiver's objective is mixture
583
between the standard and partitioned objective. This can also be rewritten as special case of
584
Equation| with
a + [etiil
if if N (t)
|NGuil
otherwise
585
The optimal receiver can then be rewritten as the posterior distribution associated with the mixture sender
586
x +(1-a) x (5((1 m)
587
B
The Case of Referential Games
In the analysis from Section 2.2 onward, we assumed C = X' to simplify notation. We can relax this assumption without changing our key observation that all receivers are the same the optimum.
589
590
Indeed, in this case the receiver's objective a standard population is:
Jro(W;)
Σ
(15))(11;))
[Wc())
iENo(1)
Σ [o6(i iENol)
Exnp E Emnsy (.1z) Be log Pet, (x m, O)
= Eoxp Bmmiiclt Eeep log Pay, (x m, C)
591 This objective, called InfoNCE [45 also has an analytical solution that can be expressed as function of π NoG) of the form:
Pug (x m, C)
p(2)
(11)
Tia (vlm)
Eyec
ply)
593 Despite the more complicated form of the optimal receiver, the key ingredients to our analysis function the posterior (N (x m) associated with the communication partners to which it co-adapts. A similar analysis partitioned populations shows that the optimum for receiver j then only depends on the posterior associated with its respective sender Thz instead.
594 in Sections 2.2 and are preserved: at the optimum, each receiver
596
15

---

B= Topsim = 22.7
0=0.01 Topsim = 33.3
8= Topsim = 33.2
Topsim = 25.6
Topsim 38.1
80
Mutual inteligibiiit
60
1055
Internal communication
40
20
100
50
100
50
100
100
50 epoch
100
epoch
epoch
epoch
epoch
Figure 6: Evolution of internal communication and mutual intelligibiliity terms with different weight-
ings 3 (populations of size 10).
C Further Analysis of the Effect of Mutual Intelligibility
599
In Section 6.4 we find that languages stay highly compositional until the mutual intelligibiliity weight is decreased to almost O. Our hypothesis that even with small amounts of mutual intelligibiliiy, agents will eventually have to optimize this part of the objective after they have maximized their respective internal communication the point where the main contributor the training gradient is the mutual intelligibility term.
601
603
To verify this hypothesis, in Figure 6 we report the evolution of both internal communication and mutual intelligibility losses during training for various values of the mutual intelligibility weight
605
8. As expected, we observe that for all but very small values of B, the mutual intelligibility loss eventually decreases (although it decreases faster for high 5).
607
16


Reviews:
[{"review_id": "rW30O2Y5mkC", "reviewer": "Reviewer_cLW8", "summary": "This paper proposes a partitioned way of training communication in receiver-sender games in a network of receiver-sender agents. A theoretical analysis is carried and the optimality of popoulation training is discussed. The results show that a partitioned approach performs better when compared to a standard fully connected communication approach. In addiiton, the quality of the language learned by the agents is analysed. ", "questions": "- according to line 246, the high standard deviations are due to the fact that different behaviour may show up between agents that are far away in the population. Does it mean that agents are then limited to communicate with their neighbours if they would be placed in a new setting? Based on the results with the distances between pairs (Fig 3) I would say that the distance does not seem to be a big problem.\n- in section 6.4 and Figure 5c, the values remain stable until very low values of beta and only drop when mutual intelligibility is almost non-existent, raising questions regarding the importance of a lot of mutual intelligibility. It seems true that the existence of mutual intelligibility is needed for better compositionality, but why is there almost no difference from 0.1 to 0.5? \n- in line 197, if for every sampled pair both objectives for sender and receiver i are calculated, from my understanding then it means that the neighbour senders of receiver i will also be updated (because it receives the message from the other receiver as in Fig 1) without updating their tied receiver. Will this have an impact on the language if many times only the sender of the pair will be updated, as it was initially mentioned to be a problem in line 190?", "limitations": "In the results, the compositionality demonstration could be improved; although the plots in Fig 5a give an idea of the argument, it would be interesting to visualise in some way the learned languages to a more clear perception of the compositionality. \n\n\nSome minor mistakes/misspelings:\n- in line 66, blank space at the end, seems that something is missing.\n- line 294: “Figure 5c” -> “Figure 5b”\n- line 79: “an” -> “a”\n- line 94: “analyses” -> “analysis”.\n- line 65: C is defined as a set of length |C| including x; thus “containing x and |C| distractors” is incorrect since the number of distractors should be |C|-1.\n- line 192: “neighbor” -> “neighbors”\n\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Despite some grammar mistakes/misspelings (I document some below) this paper reads well.\n\nStrengths:\n-\nThe paper analyses theoretically a sender receiver system where there is exchange of messages and builds up to achieve the proposed partitioned setup. In addition, an interesting analysis of components such as language similarity/topographic similarity is carried.\n\nWeaknesses:\n-\n- in line 74: “sampled from p”; p should be a distribution (or the input space X in this case); according to line 61, p is a probability value. Same problem in most equations throughout the paper, when writing for example “$E_{x\\sim p}$”.\n- in the objective of the sender in equation before line 111, why is $\\pi$ conditioned on m in the second expectation $E_{m\\sim\\pi_{\\theta_i^*(.|m)}}$? In all the previous definitions $\\pi_i$ is always conditioned in the input, i.e., x.\n- in line 138, the same derivation from section 2.2 is assumed to be valid to the partitioned setting. While I understand the logic for the standard setting (fully connected) I have concerns about whether the derivations still hold for the partitioned setting. I believe this requires some deeper explanation instead of simply stating it. For instance, in section 2 the objective of the sender is given as the average of the objectives of the neighbour senders of the sender i. But according to Fig 1, in the partitioned setting only the receivers have neighbour senders since the senders are only connected to receivers. Same question for receivers, since the neighbours of receivers are all senders according to Fig 1. I believe this should be made more clear.\n- in line 104, from my understanding it is stated that the policy of the receiver is equal to the average policy of all the neighbour senders. It is unclear to me how this equality holds even when the receiver has enough capacity. At the optimal level I agree that the policy of the receiver could indeed be some mix of the policies of the neighbour senders but it is unclear to me why this mix is the average. Is this an assumption?\n\nGenerally, I have some concerns that I have outlined in this section and some questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "if4eMb2ePU", "reviewer": "Reviewer_z9wN", "summary": "The research reevaluates the standard training procedure, highlights its shortcomings, and studied the communication of multi-agents at the population level. \nThe primary contributions: the paper propose an alternative training procedure that partitions sender-receiver pairs, restricts co-adaptation of receiver agents, and explicitly promotes mutual understanding between various agents. Moreover, the article discovers that languages created in partitioned populations are more compositional, and that population size has an effect, with bigger populations developing more structured languages.\n", "questions": "Can you make your code public?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nIn this study, multi-agent communication is examined from the viewpoint of the population. the idea is relatively creative, Moreover, population-level communication in sender-receiver Lewis games is analyzed, and a novel protocol is proposed, experiments also demonstrate the efficacy of the proposed protocol. \nWeaknesses: The analysis of 6.3 may be unclear. It would not be able to understand it without reading the supplementary materials, but in 6.3 it does not say that supplementary materials are needed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SqOg9_WLQA3", "reviewer": "Reviewer_sLjb", "summary": "The authors propose a modified training procedure for population-based training for emergent communication in which specific speaker-listener pairs are constructed, and replace the \"free for all\" mixing of pairs present in traditional methods.  Constraining the mixing in this way limits co-adaptation, and thus improves generalization and topographical similarity.  In addition, this solution also solves some more long-standing issues, like why scaling to larger populations in traditional training methods has not typically met with significant improvements in emergent structure or task accuracy, and the authors demonstrate how this originates in the loss of the standard training algorithm.\n", "questions": "Questions are mostly implied by 'weaknesses', but to be clear: how does your proposed method relate to neural iterated learning, cultural transmission, and ease-of-teaching based approaches which study similar dynamics and reach similar conclusions -- typically higher task accuracy and topographic similarity?", "limitations": "I did not see any explicit discussion of limitations or societal impact, but it is hard to imagine such small scale experiments in this topic having important negative societal impact.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nOverall I really enjoyed this paper.  It's very well written, and the way the authors narrow their focus to this one specific issue helps the argument structure come across very clearly.  I thought their argument was convincing, especially when the scope and argument applies to training dynamics in toy scenarios.  The paper contributes important theoretical understanding to population-based optimization for EC, and the effects of controlling population size, a co-adaptation \"factor\", and mutual intelligibilty.\n\nWeaknesses:\nI don't have any major criticism of the work within the scope the authors layout.  However, working against this paper is that the authors make only a passing effort to connect their work to previous related work, some of which may have very similar interpretations.  Two strands of work that I think should be discussed in much more detail and direct comparison are are neural iterated learning and cultural transmission models.  Both approaches are mentioned in the related work, but no direct comparison is made, despite that both approaches apply similar constraints to the population, essentially limiting who is talking to who, and likely reducing co-adaptation.  Considering the improved performance on communicating with novel partners, one would think some comparison to iterated learning, or the Easy-of-teaching (Li 2019) would have been warranted here.\n\nI think by the phrasing of the last paragraph of the related work (L332/333) the authors are trying to position themselves outside of this work, but I don't find it a convincing excuse.  The motivations are similar, the solutions are quite similar, and it would be appropriate to connect to this research, both in discussion and in experimentation.  Ideally I would like to see existing work as baselines in direct comparison to this work, exploring the role of different partitioning patterns have.\n\nBecause in the end, I'm unsure of what these results would mean in terms of pushing the field forward.  If iterated learning is more effective on all of the proposed benchmarks, is this just a footnote in the progression of EC research or only relevant to those who are for some reason unwilling to consider known approaches to training that work better than the standard population-based training scheme?  If the authors more explicitly compared their partitioning approach to other population-based approaches, it would be clearer what the impact of the research would mean in a more practical sense.\n\nIntuitively, to me, the other approaches seem more a bit more plausible from a sociolinguistic point-of-view.  Of course there is no need for the partitionings to exist in this circular pattern, but I'm not sure how these results relate to existing work when more realistic communication cliques are used.  Some discussion of partitioning as it relates to a plausible force in shaping human language development would be appreciated.\n\nMy review score reflects that I believe this paper has a lot of merit, but comes up short in connecting to existing research and acknowledging extent and similarities of those contributions to a sufficient extent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject