Emergent Communication under Varying Sizes and Connectivities
Authors: Jooyeon Kim
Alice Oh
Venue: NeurIPS 2021 Poster (2021)
Keywords: Emergent communication
emergence of language
group communication
communication graph
graph optimization

Abstract: Recent advances in deep neural networks allowed artificial agents to derive their own emergent languages that promote interaction, coordination, and collaboration within a group. Just as we humans have succeeded in creating a shared language that allows us to interact within a large group, can the emergent communication within an artificial group converge to a shared, agreed language? This research provides an analytical study of the shared emergent language within the group communication settings of different sizes and connectivities. As the group size increases up to hundreds, agents start to speak dissimilar languages, but the rate at which they successfully communicate is maintained. We observe the emergence of different dialects when we restrict the group communication to have local connectivities only. Finally, we provide optimization results of group communication graphs when the number of agents one can communicate with is restricted or when we penalize communication between distant agent pairs. The optimized communication graphs show superior communication success rates compared to graphs with same number of links as well as the emergence of hub nodes and scale-free networks. 

Emergent Communication under Varying Group Sizes and Connectivities
Jooyeon Kim
Alice Oh KAIST
thingsflow Seoul, Korea
Daejeon, Korea alice. oh@kaist. edu
jyscardioid@gmail.co
Abstract
Recent advances in deep neural networks allowed artificial agents to derive their own emergent languages that promote interaction, coordination, and collaboration within group. Just as we humans have succeeded in creating a shared language that allows to interact within a large group, can the emergent communication within artificial group converge to a shared, agreed language? This research provides an analytical study of the shared emergent language within the group communication settings of different sizes and connectivities. As the group size increases up to hundreds, agents start to speak dissimilar languages, but the rate at which they successfully communicate is maintained. We observe the emergence of different dialects when we restrict group communication to have local connectivities only. Finally, we provide optimization results of group communication graphs when the number agents one can communicate with is restricted or when we penalize communication between distant agent pairs. The optimized communication graphs show superior communication success rates compared to graphs with the same number of links, well as the emergence of hub nodes and scale-frec free networks.
Introduction
Communication plays vital role for humans interact with one another [Asttin 1975 Allwood 1976 Linel] 2009]. We acquire new information about the environment and other people around us through communication, which further facilitates collaboration competition and allows to coordinate our future decisions and actions better.
Over the past decades, numerous research outcomes have succeeded in mimicking human behav- ior and making artificial agents equipped with interactive communication abilities [Batali
1998
Cangelosi and Parisi] 2002 Christiansen and Kirby 2003 Steels, 2003 Wagner et al. 2003 The
communication in the artificial system is emergent in way that there are no pre-specified usage rules, semantics, or syntax, all of which are formed by the artificial agents upon their necessities [azaridou сet al. 2017]. Not only does empowering agents with their own communicative abilities set an impor- tant milestone in interactive Al systems, also brings steps closer to the world where machines cooperate with humans Mikolovv et al. 2016 Crandall et al. 2018 Lazaridou and Baroni 2020
In human societies, hundreds of millions of people use the shared, agreed communication protocols to interact. the group becomes more complex with increased size and connectivity, communication becomes more and more vital for us to coordinate with one another. Fortunately, we humans excel at Lupoan Bergen 2016]. However, in the AI system counterpart, the question of how emergent communication will function under different group settings understudied. As the group becomes more complex, will emergent communication become more effective for the system in promoting group coordination. as in human society, or will it lose its functionality? Will the emergent communication still converge
using languages for large-group coordination and problem-solving [Tomasello, 2010 Lupyan and
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Color-only
Shape-only agents
Listener oo e)
Speaker (c
agents
BOUBAI
Agent
Communication Channel
S
OR% O%)
Tokenize
Concar
B
10%, ONj
C2 O
S
"Blue circle"
"Blue circle
KIKI!
Descritize)
Left
<Find a blue circles-
Figure 1: Left: A scenario in which two agents speak different languages to describe the same object. Middle: Bi-partite group communication using variant of the referential game, involving color-only agents c and shape-only agents s. Right: Communication model architecture between speaker and listener agents. Both color-only agent and shape-only agent can be speakers and listeners.
shared, agreed language? In the worst case, efficient interaction among agents through emergent communication can be hampered because different agents will start to use different languages to describe the same observation (the left-most panel of Figure
This research provides an analytical study on emergent communication in artificial groups of different sizes and connectivities. Our goal is to investigate the effect of emergent communication and analyze how agents' general agreement on the communication protocol changes under varying group settings. To study this question, we construct settings within which communication among agents indispensable for achieving their shared goals. Communication is their only means to interact and compensate for their limited access to information from an environment. We divide agents into two groups and allow an agent to communicate with another agent in a different group through bi bi-partite communication graph.
To quantitatively evaluate the functional aspect of communication, we look at (a) the language similarity message agreement of agents within the same group when describing the same referents and (b) the rate at which an arbitrary pair of agents in the bi-partite communication group succeed in communication and achieve the shared goal.
To this end, our contributions and findings are as follows: (1) We find that as the size of the communication graph scales up to hundreds, the language used in the group becomes less similar amongst agents. However, the communication success rate is maintained. (2) We observe that when agents only get to communicate with nearby agents, a particular form of communication peculiar to agents' locality emerges. At the same time, distant pairs of agents that have not interacted during training can still communicate when a certain threshold for the maximal communication distance is reached. (3) We present and analyze the optimized communication graph in terms of overall communication success rate when (a) the node outdegrees
fixed when (b) we penalize the formation of communication links between a distant pair of agents. Compared to the fully-connected all-to-all graph, the link density decreases by 81.25%, but the communication success rate drops just by 5.89%. With random, unoptimized communication links, the success rate drops by over 40% with the same link density. Throughout the optimization process, we observe the emergence of hub nodes taking up large share of the total communication channels and connecting the majority of other agent nodes, as we see the real-world ceale-free social networks [Barabási and Albert 1999
2 Related work
There have been successive attempts to let agents develop their own communication protocols through
emergent communication [Batali 1998 Cangelosi and Parisi 2002 Christiansen and Kirby 2003
Steels 2003
Wagner et al.] 2003]. J. followed by more recent approaches that are based on deep
neural
Hausknecht 2016 Havrylov and 2017]. The idea is applied and extended to variety of collaborative [Lowe et al.) 2020
networks Foerster et al.] 2016, Sukhbaatar et al.] 2016
Titov
and Foerster 2020 Jaques et al. 2019, Kajic et al. 2020 Mordatch and Abbeel| 2018 and
competitive [Singh al.) 2019 Jaques et al. 2019 environments. Recently, advances have been made for the emergent communication to operationalized at scale; to prevent agents from being "lazy" and let them generate meaningful signals (Sunehag et al. 2018 to effectively schedule
'The work by Lazaridou and Baroni 2020 surveys the nascent literature.

---

the communication in a limited bandwidth [im et al. 2019 to foster conditions that encourage compositional and easier-to-teace languages Li and Bowling 2019 to understand other agents' intention, belief, and point of view, i.e., theory of mind [Huu and Foerster 2020]; to win a combat against another group [Singh et al. 2019]. Orthogonally, there has been attempts to make the interactive communication more dynamic and more human-like by increasing the complexity of the environment (Daques et al. 2019 Das et al. 2019], making the communication span multiple turns [Eytimova et al.) 2018 Jorge et al., 2016). These lines of research focus on increasing the efficacy functionality of the emergent communication by suggesting more advanced perceptual, navigational, and generative Al capabilities.
On the other hand, our work falls into the category of an analytical study of understanding the emergent communication from various perspectives. Recent studies find properties of emergent language that sets itself aside from [Chaabouni et al., 2019 Kottur et al. 2017] or are shared with [Resnick et al. 2020 Baroni, 2020] natural human languages and seek for different possibilities to benefit existing NLP models Lee el. 2018 Lazaridou et al.) 2020 |ouchacourt and Baroni
[2019] and Cao et al.
2018 studies various conditions under which the emerged language can be
consonant and agreed as apposed to giving rise to idiolects or agent identity playing a crucial factor in interactive systems.
The work of Graesser et al. [019 is especially similar to ours in that also studies the linguistic phenomena of emergent communication at a group (community) level. Specifically, the authors formulate one, two, or five artificial communities each of which comprising agents up to size 30. By changing the inter and intra-community connection densities, the authors reproduce linguistic behaviors that can be observed in human communities. One of the findings the emergence of the "linguistic continuum", which to a great degree resembles our finding of "dialects" in section 5.4 Our work is distinctive in a way that we do not assign agents into hard clusters of communities. Therefore, we observe finer granularities of linguistic continuums (dialects) at agent-levels rather than at community-levels. Finally, Dubova and Moskvichev 2220)) Dubova et al. 2020 Tieleman et al. 2019) also studies emergent communications in group settings.
3 Game design
We design a communication game based on partial information. The purpose of the game is similar to that of the referential game [Lazaridou et al., 2017], which is a variant of the signaling game [Lewis] 1969]; we test whether artificial agent can perceive given observation, speak through the emergent protocol about the observation according to the instruction, listen and make the right decision. Throughout the game, the message from the speaker agent should convey crucial information about the observation for the game to succeed. Compared to the referential game, the difference is: 1) agents only get to have partial, mutually exclusive information of the observations, so the communication should solely be about describing what information each agent has from their partial, limited observation, 2) The instruction information is symmetric for both listener and speaker: both agents have access to the instruction.
In each game trial, two agents are presented with multiple pixel-based objects: a target, distractors, and instruction. The goal is to choose the correct target object for the description in the instruction. Crucially, communication is indispensable for the agents correctly find the target since each of the two agents has its own limitation in the perception: One agent can only recognize different shapes, and the other can only differentiate colors. Therefore, the information of the object retrieved by the agents from their observations is mutually exclusive, but at the same time, is guaranteed to recover the complete state information when aggregated. Under this setting, the agents communicate based on their limited observations to compensate for their deficiencies and correctly choose the target object described in the instruction. In section 5.3 referential game as well.
we derive similar findings using the more conventional
Agent and action Each of the two non- non-embodied agents chosen from two distinctive groups. We denote agents that can only perceive colors (i.e., the color-only agents) by c, and the corresponding group by Go such that e E g Similarly, the shape-only agents and the corresponding group are denoted by 8 and G². The number of agents of both groups is set to be the same, i.e., IG | = N. We consider a situated environment, where the action space A is divided into the disjoint communication action space Th and decision making action space A such that A U A A and
3

---

Ath 0 = 0. Of color-only and shape-only agents, one is randomly selected and act as speaker; it takes its observations, i.e., objects and instruction, as inputs and sends message m E A the other agent an output, which can be either discrete or continuous. Next, the other agent is randomly selected and act as a listener. This agent takes the message along with the observations and makes a decision d E . The decision is about choosing the position of the correct target object. Thus, the size of the decision making action space | is equal to the number of objects in the observation. Finally, the communication between agents (c, s) is successful if the decision d finds the target object. In the referential games, both speaker and listener agents get to observe the full information about the objects, i.e., both shapes and colors, but only the speaker agent has access to the instruction and the orders of the objects are shuffled for the listener agent.
State and observation A state is 3-tuple (or, Oa, w) with the target object or, the distractor objects Oz, and the instruction w. Each object is of different colors and shapes and the instruction tells of choosing an object with certain color and shape. Therefore, for and m different colors and shapes, there can be n x m varying instructions. We represent the instruction using a fixed-sized vector by concatenating n and m-diimensional one-hot vectors. Observations comprise objects and instructions. In section 5.3] we also conduct experiments using real-world images [Krizhevsky] 2009 instead of the artificial shape and color observations.
Communication graph A bipartite communication graph G is formed between G and G* as multiple pairs (c, s) communicate, i.e., form a set of directed communication links (c, s), (s, E E and establish communication protocols.
Message agreement and communication success rate We introduce message agreement and communication success rates used to evaluate language convergence. First, let and a' be two agents randomly sampled from the same color-only or shape-only group, i.e., a, a' G or a, E G and let m and m² be the messages generated by the agents from the same observation o e O. The message agreement of certain group, which is either color-only or shape-only, in communication graph is measured by the expected similarity between two messages m² and m²: Eu [f (mmm,, m² [o]. where f(---) -) is a similarity measure and G' E (G*, Next, the communication success rate of a communication graph is measured as: 1e "o--" aooor(c, 8, o)] with being an identity function that returns if agents successfully find the target object.
4 Group communication model
We describe how agents communicate with others within G their speaking and listening networks and explain the training procedure. In a communication graph, the selected color-only agent e G and shape-only agent E G with (c, s) E E or (s, c) E & communicate via fixed-sized messages and make collective decisions. The network architecture is illustrated in the right panel of Figure The communication algorithm that summarizes the overall procedure can be referenced in Appendix B
Speaker In each trial of the game, one color-only or shape-only agent c G or E 9" is selected from its group and serves as a speaker, generating message m. The speaker agent starts by processing a raw pixel-based observation input, consisting of multiple objects with different shapes or colors, into fixed-sized encoding using variational autoencoder (VAE) [Kingma and Welling Each encoded vector then concatenated with the instruction vector and is used as an input to the speaker network. The speaker network is feedforward neural network and outputs a fixed-sized message m.
2014 Rezende et al. 2014 $
The listener agent processes the observation in the same manner as the speaker network. However, rather than outputting the message, the listener network's ' intermediate layer is concatenated by the message generated from the speaker agent. The expanded intermediate layer is then used as an input for the upper layers of the network. It outputs the decision action d of size [A*, i.c., predicted position of the target object.
Listener
2 We also differentiate the size, rotation, and angle of objects and vary the color codes within the same color to diversify the state-space, as further detailed Appendix A Each agent trains VAE separately that no information shared this stage.

---

Training During training, we iteratively sample communication links between two agents and s, and apply backpropagation, with binary cross-entropy as the loss function over the training batch. We adopt centralized training and decentralized execution regime [Foerster et al.) 2016 Therefore, when two agents e and communicate during training, the gradients flow through the communication channel from one agent's s listening network to another speaking network. However, agents do not have access to the actual inner architecture, the parameter values of other agents, and no gradient flow during execution. Also, agents within G and G are completely decentralized. Message discretization achieved by the Gumbel-softmax relaxation [Jang et al., 2017, Maddison et al., 2017]. Finally, ensure that agents and s communicate enough to learn the communication protocol and be fairly compared but not too much to prevent the generated messages from overfitting the training data by adopting the early stopping rule with the validation set. Refer to Appendix C for the detailed architectural settings and training results.
Experiments
We investigate how different configurations of communication graph affect language convergence in group communication. Specifically, we answer the following questions: 1. Group size: In all-to-all communication, i.e., when all agents communicate with each other, how does the size of the communication group N = | = [ affect the message agreement and communication success rate? 2. Group connectivity: In weighbor-to-ncigheee
communication, i.c., when agents only communicate with nearby agents, how successful can a distant pair of agents who has not exchanged messages during training communicate? How similar do their languages become? 3. Optimization: Can we structuralize the optimal communication graph in terms of communication success rate when agents only establish a limited number of communication links?
5.1 Experimental setup
Game configuration Agents observe 10 objects with I target object and distractors. The size of the decision-action space |A45| is set to 10. For both discrete and continuous messages, we set the dimensionality of E A to 10. Discrete messages are binary vectors using the Gumbel-softmax relaxation. Continuous messages are real-valued vectors. Objects observations are RGB pixel- based images with sizes 32 x 320. Each image contains TO objects with different colors (red, green, blue, cyan, magenta, yellow) and 5 different shapes (ellipse, triangle, quadrilateral, pentagon, hexagon). Instruction is an I-dimensional vector that concatenates and 5-dimensional one-hot vectors for shapes and colors. Example object observations are in Appendix A
Data We generate 128,000 observations for training set and 12.800 observations for and validation sets. The early stopping rule is applied for the validation set, and the reported results are calculated using the test set. Finally, the model parameter settings are detailed in Appendix C
5.2 Preliminary results and message categorization
In the absence of communication, the success rate of agents correctly identifying the target object is 0.236. After the munication, the success rate increases to 0.972, the maximal success rate the agents can achieve, as there are cases of having one or more distractors with the same shape and color as the target object. This result confirms that the messages transmitted from a speaker agent convey critical information the speaker agents have but the listener agents do not. Therefore, we can formally calegorize all messages transmitted by the speaker agents based on their instructions and observations.
G. (343
&. CK 4)
I. (3.30
6.)
Figure 2: Message categories
Definition 5.1. (message categorization). Let P..........) be the unique positions of the objects that have the colors (shapes) described in the instruction of color-only agent e (shape-only agent s), with < p. < | The category of the message transmitted from the agent (s) is defined as
'We can straightforwardly restrict the range of the message values by e.g., applying the softmax or sigmoid function to the message. In our experiments, findings and statements made are consistent with and without such restrictions.
5

---

(c.....Pa...... or (s.f....PPu. ......................... 1).
Figure 2 validifies out message categorization. Here, we randomly sample 21,00 2000 messages across from 32 color-only agents with 6 different categories and visualize them using (-SNE. Then, messages is each category are highlighted by different markers and colors. From the figure, we confirm that the messages in same categories are clustered in the latent space and exhibit high topographic similarity.
(a)
1.00
Color-otty Cert.
1.50
Shape only spents, Cont,
Untener: Coleertyy Cent.
1.00
Litemer: Shape-anty Cont.
6.00
3:50
328
0.20
1.6
6.70
16
128
(c)
(g)
1.0
Color-caty
Shape Bac
1.20
Latener: Ditarary: Dac
1.00
Ubitner: Shupzorly, Dac.
6.
6.7
3.3
0.70
6.70
324
176
16
128
(
(k)
Figure 3: All-to-all communication. (a)-(d): t-SNE plots of messages. (b) and (d) are the magnified portions (a) and (c) respectively. is the number of agents and each point is a single commu- nication message. Colors represent different, randomly selected message categories, and markers represent different agents. As grows, what is the message becomes more relevant than who the speaker is for identifying the message. (e)-(I): Similarities of messages sent from different agents with same observation and communication success rates of the communication graph with respect to the number of agents N in the group. (e)-(h) and (i)-(I) are the results when agents use continuous and discrete messages, respectively. As N grows, the messages sent by different agents become less similar, but the communication success rate is maintained.
5.3
Group size
We answer the first question regarding the all-to-all communication, where all agents in go commu- nicate with all agents in go. The top panels of Figure B iillustrate the t-SNE [Mateen and Hinton 2008) plots of the messages generated, with N being the size of g and g (g 2N). We only show the color-only agents' results with continuous message setting. Each dot represents single message, with different markers representing the messages generated by different agents. Different colors represent different, randomly selected message categories. i As the group size increases, the messages tend to gather by their respective categories rather than agent identities (panels (a) and (c)), and within each cluster, the identities of the agents who transmit the messages become more difficult to be separated (panels (b) ad (d)). As N increases, what is entailed in the message becomes more relevant when identifying the message than who sending the message.
SSimilar results can he ohtained from the other configurations, c.g., shape-only agents with discrete messages. 'For the visualization purpose, only display three categories.

---

Referential: object, CIFAR-10
Referential: object. CIFAR-10
0.90
Comm. success rate
Cosine sim.
0.85
0.95
Object
0.80
CIFAR-10
0.90
16 32 Agents (a)
128
2
4
8
16
32
64
128
Agents
(b)
Figure 4: All-to-all communication with the referential game setting and continuous messages. The blue solid lines are the results from the color-shape-ohjet dataset and the orange dashed lines the results from the CIFAR-10 dataset. (a): message similarity decreases as the number of agents within communication graph increases. (b): communication success rate is maintained as the communication group size grows.
Second, from the subfigures (e), (f), (i), (i) of figure
3 the message agreement calculated by the cosine similarity decreases as the group size N increases, for both color-only and shape-only groups and both continuous and discrete-messart
settings. On the other hand, in the subfigures (g), (h), (k), ()) of figure 3 the communication success rate does not increase nor decrease as N increases. Even though m and m² become dissimilar N increases, the rate of the color-only agents and d successfully communicating with the shape-only agents Vs G does not decrease.
In Figure we observe similar trends from a different game and datasets. We analyze (a) the message similarity and (b) the communication success rate using the referential game Lazaridou et al. [017]. where agents have access the full observation and one agent is randomly selected and act as the sender (speaker), with an access to the instruction while the counterpart receiver (listener) agent does not. Furthermore, aside from the artificial shape-color object datasets, we also experiment with the real-world CIFAR- 10 images with ten classes (Krizhessyy 20099]
Overall, N increases, messages in the categories exhibit less topographic similarities by agents, while maintaining the communication success rate. Furthermore, agents' languages becoming more similar does not directly translate the increased communication success rate.
5.4 Group connectivity
In neighbor-to-ncighbor communication the color-only agents G = [ci,..., CN) and the shape-only
agents G {s1,.... .... SN communicate with nearby-agents only, i.c., c, only communicates with {8i-j, Si+j } and vice versa, with being the maximum communication range. Thus, j = 0 signifies one-to-one communication.
Figure iilustrates the teighbor-to-neighbor communication results with N set to 64 and varying the maximum communication range j. Each dot in the top panels represents single message. Here, all messages are a same category and different colors represent different localities of the agents; as i increases in g = {(I, ..., . Ci ..., CN)) ), the color becomes darker, from light green to black.
When j is small (j = 1), multiple separate clusters form, and the dots of the same colors are scattered, suggesting that the agents' communication patterns are distinctive and do not depend too much on the locality. As j increases (j = 16), the groupings of messages emerge according to agents' locality; particular form of communication is shared by agents at the top, middle, and bottom of the graph. Simultaneously, as j grows, the messages of the pairs of color-only agents that are far away from each other become more similar (the middle panels, off off-diagonal parts; the row and column indices indicate the identities of color-only agents). Moreover, the communication success rate between pair of agents that has never communicated during training increases (the bottom panels, dff-ddaa parts; the row indices indicate the localities of the color-only agents and the columns are the localities of the shape-only agents).
7

---

Maximum
Maximum
Moximum distance
(a)
(b)
(c)
(d)
(e)
Nean wwwilartty D:315
Nean simillrtty, o 200
Viean stmilartyy 3.503
Moan simdiariyy 0.000
Mean amilarty
06
(h)
i
Nean rate: 2.400
R1
10 24 32
24 50
28 12
40
(k)
(m)
(n)
communication. (a)-(e): Color represents locality: lighter colors at the top of the bipartite graph and darker colors at the bottom. When the maximal communication distance ranges from 1 (a) to 16 (d), groupings of communications that are peculiar their locality, e.g., dialects are formed. However, when the maximal communication distance is small (a), dialect clusters are relatively small, and such clusters from distant localities often exhibit high topographic similarity. In contrast, when the maximal communication distance is large (d), the topographic similarity of dialects is clearly reflected by localities. the same time, their messages become more similar ((f)-()) and the communication success rate increases ((k)-oo)), even between non- neighbors (off-diagonal parts).
Figure 5: Neighbor-to-neighhee
As the maximum communication range increases, particular form of communication peculiar to agents' locality, which is analogous to a dialect in human languages, emerges, and the agents' overall language gradually converges.
Node count
80
Node count
15
Node count
15
Node count
10
10
20
10
5
12
Indegree
Indegree
Indegree
Indegree
(a) All-to-one
(b) N-to-N
(c) Random
(d) Optimized
Figure Different communication graphs and the respective in-degree distributions.
5.5 Communication graph optimization
In the previous subsection, we observe that in the sparse setting pair of agents located far away struggles to communicate successfully and become speaking dissimilar languages. this regard, we aim to find the optimal communication graph that maximizes the communication success rate. We formulate two settings: The first setting is the hard-constraint setting, with all agents share the same limited number of outgoing communication links. With N 16, we restrict the number of outgoing

---

Table 1: Communication graph optimization. We restrict the number of outgoing edges for each agents. Reported numbers are the success rates with one standard error over five different trials.
No Link
ole-to-one
N-to-N
Rand. Links
Optimal
Success Rate
0.236
0.384 + 0.031 0.454 + 0.017 0.578 + 0.154 0.894 = 0.021
links to 3. A color (shape)-only agent can form communication links with any shape (color)-only agents but the total number of outgoing links from the agent always has to be 3. Therefore, the graph density remains constant at O. 1815 (3/16) throughout optimization. The next setting is the soft-constraint setting, where we do not restrict the number of outgoing links but rather penalize the formation of communication links between distant agents. Specifically, we maximize the communication success rate of the communication graph subtracted by a penalty p k Eee i - + (e,,e,, with the parameter k controlling the density of the communication links, and j j -j+1 1| favoring links with shorter communication distances. Therefore, there is cost entailed when setting communication link, and the cost increases when the distance between a pair of agents increase. We set = 0.0012 so that the objective value becomes zero the all-to-all communication, i.e., fully connected graph. For both settings, we optimize the link structure using covariance matrix adaptation evolution strategy (CMA-FS) [Hansen and Ostermeier 2001 Hansen 2006 2016) with population size 32. For each link structure setting, we use the averaged score from 4 different initializations to stabilize the optimization process, Refer to Appendix D for additional details regarding the optimization process.
Table shows the communication success rate of the optimized and comparison communication graphs in the hard-constraint setting. All except for no link have the same graph density of 0.1875. The visualizations of the graph structures along with their indegree histograms are shown in Figure 6 two extreme cases. In All-to-one graph, only 3 randomly selected agents in 9 and G have the in-degrees of N = 16, while the rest of the agents have no incoming links. On the contrary, in neighbor-to-neighee
All-to-one and neighbor- neighbor (N-to-N)
graph, all agents have the same in-degrees and out-degrees of 3. The communication success rate is less than 0.5 for both graphs. Also, when the links are formed randomly, the mean communication success rate is 0.578. Remarkably, we are able to achieve the communication success rate of 0.894 after the optimization. In all-to-all communication, the communication success rate is 0.950. The communication success rate decreases just by 5.89% while the communication density is dropped by 81.25% The panel (d) of Figure 6 show the visualization of the optimized communication graph; while most agents have in-degrees ranging from 1 to 4, one agent in each group 9 and G that has in in-degree greater than 12. These agents act as the hub nodes, increasing the overall communication success rates. In random graphs (panel (c) of Figure G. there is no such node with high centrality value.
Table 2: Soft-constraints: comparison with baselines.
20
Node count
# Links
Penalty
Success Rate
15
No Link
0
0.236
10
1 1-to-All
31
0.094
0.251 + 0.091
N-to-N
74
0.050
0.454
0.031
Indegree
3
Rand. Links
19
0.053
0.513 + 0.106
Figure 7: Optimization with soft-constraints.
Optimal
46
0.054
0.877 + 0.026
Table shows the communication success rate of the optimized graph under the soft-constraint setting along with the comparison settings. The comparison graphs other than 1-to-All and No-link are constructed in a way that has the similar penalty value as the optimal graph. Similar to the hard-constraint setting, the success rate of the optimized graph is much greater than those of the comparison graphs with higher or similar penalty values. The left panel of Figure 7 shows the illustration of the optimized graph along with its in-degree distribution. Unlike the hard-constraint setting, we observe scale-free graph with majority of nodes having the in-degree of one and the number of nodes drastically decreasing as the in-degree increases. The right-panel of Figure 7 shows the optimization result for the communication graph, where the blue dots the thh row and j"h column indicate the link formation between shape-only agent s; and color-only agent cj. Since there is

---

larger penalty for the links between distant agents, we observe that most links are formed by nearby agents. However, we sparingly observe links being formed between distant agents (off- diagonal dots in the figure), hinting that these links are crucial for maintaining the overall communication success rate despite their large costs.
Conclusion & discussion
In this work, we investigated whether the emergent language converge in different configurations of the communication groups. We observed that when the number of group size increases, the language spoken by different agents becomes success rate is maintained. This is intriguing since the agents speaking similar languages does not necessarily translate them communicating successfully.
discordant while the overall communication
The analyses on temporally extended and complex environment with embodied agents would be desirable next step. Hill et al. [2020] focused on various environmental factors that lead to generalization an embodied system. Overcoming the existing problem of -ugti- RL, such as
the lazy-agent problem (Sunchag et al., 2018). the message scheduling issue [Kim et al. 2019 and
joint exploration problem Lowe et al.
2017 would also be an interesting future direction.
Understanding whether or to what extent neural agents exhibit properties that are analogous to human linguistic evolution is an interesting open question. Further, we believe that there might be some clues given from the systematicity in human linguistic evolution [Raviv et al. 2019
Building clear distinction between structural (grammatical) systematicity and input variability is another interesting future research topic. With this distinction, we believe that our finding (partially) aligns with what the human language studies have found. From result with neural agents, find that: "for larger groups, different agents start to use less similar languages to describe the same observation". This
direct correspondence the "linguistic variability" in human languages [Raviv ct al. 20191. In fact, human linguistic evolution studies [Gomez] 2002 Lev-Ari 2016 Rost and McMurra 2009 Perry al., 2010 suggest this "input variability" as one of the factors that promote "structural systemization".
Our finding with neural agents that show "maintained communication success rate" despite increased input variability might suggest the emergence of structural systematicity as the group size increases. However, we hesitate make definitive statement this without additional experimental analyses, e.g., the actual measurement of grammatical systematicity (Raviv et al. 2019) also measure the communication success rate as one of the metrics for such systematicity).
To summarize, linguistic variability suggested to the key factor that promotes structural system- aticity in human language studies. Our paper finds higher linguistic variability in larger, artificial neural systems and a weak sign structural systematicity from the maintained communication success rates.
Acknowledgments and Disclosure of Funding
We would like to thank the reviewers for taking the time and effort to review this paper and providing invaluable, constructive comments. This work supported by the National Research Foundation of Korea(NRF) grant funded by the Korean government (MSIT) 2019RIAAC10855771,
10

---

References
Jens Allwood. Linguistic communication as action and cooperation. University of Göteborg. Department of Linguistics, 1976.
John Langshaw Austin. How to do things with words, volume 88. Oxford university press, 1975.
Albert-LászIó Barabási and Réka Albert. Emergence of scaling in random networks. Science, 286 (5439):5095512, 1999.
Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks Philosophical Transactions of the Royal Society B, 375(1791)220190307, 2020.
John Batali. Computational simulations of the emergence of grammar. Approach to the Evolution of Language, pages 405 426,, 1998.
Diane Bouchacourt and Marco Baroni. Miss tools and Mr fruit: Emergent communication in agents learning about object affordances. In ACL. 2019.
Angelo Cangelosi and Domenico Parisi. Simulating the evolution of language. Springer, 2002.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In ICLR, 2018
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. In NeurIPS, 2019.
Morten Christiansen and Simon Kirby. Language evolution. Oxford University Press, 2003.
Jacob Crandall, Mayada Oudah, Fatimah Ishowo-Oloko, Sherief Abdallah, Jean Jean-François Bonnefon, Manuel Cebrian, Azim Shariff, Michael Goodrich, Iyad Rahwan, et al. Cooperating with machines. Nature communications, 9(1):1-12, 2018
Abhishek Das, Théophile Gervet, Joshua Romoff. Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In ICMI, 2019.
Marina Dubova and Arseny Moskvichev. Effects of supervision, population size, and self-play celf-play multi-agent reinforcement learning to communicate. In Artificial Life Conference Proceedings, pages
MIT Press, 2020.
Marina Dubova, Arseny Moskvichev, and Robert Goldstone, Reinforcement communication learning in different social network structures. arXiv preprint arXiv:2007.09822 2020.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in a multi-modal, multi-step referential game. In ICLR, 2018.
Jakob Foerster, Yannis Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In NeurIPS, 2016.
Rebecca Gomez. Variability and detection of invariant structure. Psychological Science, 13(5): 431-436, 2002.
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In EMNLP, 2019.
Nikolaus Hansen. The CMA evolution strategy: a comparing review. In Towards a new evolutionary computation, pages 752102. Springer, 2006.
Nikolaus Hansen. The CMA evolution strategy: a tutorial. arXiv preprint arXiv:1604.00772 2016.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):19-1155 2001.
Matthew Hausknecht. Cooperation and communication in multiagent deep reinforcement learning. PhD thesis, University of Texas at Austin, 2016.

---

Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In NeurlPS, 2017.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James McClel- land, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. In ICLR, 2020.
Hengyuan Hu and Jakob Foerster. Simplified action decoder for deep multi multi-agent reinforcement learning, In ICLR, 2020.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9()): 90-95, 2007.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax, In ICLR, 2017.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, Dj Strouse, Joel Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In ICML,
Emilio Jorge, Mikael Kågebäck, Fredrik Johansson, and Emil Gustavsson. Learning to play guess who? and inventing grounded language as a consequence. In NeurIPS Workshop Track, 2016.
Ivana Kajić, Eser Aygün, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. arXiv preprint arXiv:2004.0100 2020.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. In ICLR, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In ICIRR 2014.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does emerge 'naturally't in multi-agent dialog. In EMNLP, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02411 2020.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In ICLR, 2017.
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. ACL, 2020.
Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. Emergent translation in multi-agent communication. In ICLR, 2018.
Shiri Lev-Ari. How the size of our social network influences our semantic skills. Cognitive science, 40(8):2050-200 2064, 2016.
David Lewis. Convention: a philosophical study, 1969.
Fushan Li and Michael Bowling. Fase-of-teaching and language structure from emergent communi- cation. In NeurlPS, 2019.
Per Linell. Rethinking language, mind, and world dialogically. Information Age Publishers, 2009.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettivee environments. In NeurIPS, 2017.
Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. the interaction between supervision and self-play in emergent communication. In ICLR, 2020.
12

---

Gary Lupyan and Benjamin Bergen. How language programs the mind. Topics in cognitive science, 8(2):408 424, 2016.
Andrew Maas, Awni Hannun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML, 2013.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 9(Nov): 2579-2605, 2008.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: continuous relaxation of discrete random variables. In ICLR, 2017.
Tomas Mikolov, Armand Joulin, and Marco Baroni. A roadmap towards machine intelligence. In CICLing, 2016.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAAI, 2018.
Vinod Nair and Geoffrey Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, al. Py- torch: An imperative style, high-performance deep learning library. In NewrIPS, pages 8024-8035. Curran Associates, Inc., 2019. URL http::/paperrs wetppeerreeeuuipppp. neurips cc/paper/ wal1--pytocchaa--e htw.ap/aeettttttteeeeeeeeeeereer
pdf
Lynn Perry, Larissa Samuelson, Lisa Malloy, and Ryan Schiffer. Learn locally, think globally:
Exemplar variability supports higher-order generalization and word learning. Psychological science, 21(12):1894-1900 2010.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Larger communities create more systematic languages. Proceedings of the Royal Society B, 286( (9661) 2019.
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew Dai, and Kyunghyun Cho. Capacity, bandwidth, and compositionality in emergent language learning. In AAMAS, 2020.
Danilo Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In ICML, 2014.
Gwyneth Rost and Bob McMurray. Speaker variability augments phonological processing in early word learning. Developmental science, 12(2):399-349, 2009.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In JCLR, 2019.
Luc Steels. Evolving grounded communication for robots. Trends in cognitive sciences, 7(7):308-31,, 2003.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NeurlPS, 2016.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS, 2018.
Olivier Tieleman, Angeliki Lazaridou, Shibl Mourad, Charles Blundell, and Doina Precup. Shaping representations through communication: community size effect in artificial learning systems. arXiv preprint /9/2.060 06208, 2019.
Michael Tomasello. Origins of human communication. MIT press, 2010.
Kyle Wagner, James Reggia, Juan Uriagereka, and Gerald Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-69, 2003.
13


Reviews:
[{"review_id": "utii8doIkck", "reviewer": "Reviewer_8o6e", "summary": "This paper studies population size and connectivity graph effects in emergent communication in a population of agents. By limiting the communication partners in specific ways, i.e. deleting edges from the fully connected graph, the authors show that dialects can evolve: communication protocols that are specific to sub-groups even when those groups are not completely isolated. Furthermore, by experimenting with different connectivity patterns that share a constant overall graph sparsity, they find that a pattern with few highly connected nodes (hubs) leads to significantly higher communication success than random or uniform patterns. Such hubs are a common feature of complex networks found in many natural systems. In terms of communicative success, the authors find that more complex tasks are associated with better generalization to unseen data (a smaller drop in success relative to the training data)\n\nThe paper also introduces a variant of the widely used referential game where the speaker and listener observe non-overlapping aspects of the images they communicate about, and where agents act as both listener and speaker, depending on how they are sampled at the start of a round in the game.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The signalling game variant proposed in the paper is interesting, in that it has different information requirements than the more common fully observed one. The speaker does not know the target exactly, and has to communicate to the listener which images are candidates. One aspect that did not become obvious to me while reading the paper is whether the listener also receives the instruction, or whether the speaker has to communicate that too.\n\nTo better understand the contribution of the partial observation, it would be good to ablate it by replicating some of the experiments in a fully observed signalling game, and investigating whether the resulting language patterns are at all different. A priori I don’t see a very clear reason why they should be, but I would be happy to be proven wrong on this point.\n\nThe graph configuration effects are nice results. The emergence of dialects even in the absence of clear subgroups (with higher internal connectivity than external) like in figures 4b and 4c is even somewhat surprising. The optimal graph question is a valuable contribution the field of multi-agent communication, as it has the potential to enable a significant saving in the computational cost of such experiments.\n\nOne question that is left open, but would be very interesting and probably relatively straightforward to answer, is what happens to the optimal graph structure when the constraint that each node has at least three outgoing vertices is dropped. The result found here, with the hub nodes, follows one pattern that often occurs in natural complex systems, but goes against another - that of the power law distribution. I would have expected to see the power law here too; can you comment on why it does not emerge?\n\nThe results you find are mostly about communicative success, but within the population that the agents are trained with. I would really like to know whether the languages that are formed in such non-uniform connectivity graphs have the same properties as those from uniform graphs when it comes to compositionality and learnability. E.g. analysing the topographic similarity of the messages to the ground truth feature vectors could provide some insights here, as could training newly initialized evaluation agents against members of a fully trained population.\n\nA third way to make the paper stronger would be the inclusion of more standard datasets, to enable easier comparisons to existing work.\n\nAll in all, I believe this paper takes on a very interesting and relevant question, but is a little light on the results presented.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, section 6 and appendix G.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "STj9VDKn25V", "reviewer": "Reviewer_z6vN", "summary": "The paper studies the effect of population size and graph connectivity on the emergence of communication. The main questions studied in the paper were 1) how does the size of agents' population affect task success? and 2) when agents don't get to train directly, but through some other agents, how well they can communicate with each other when they put in contact? The game that was used in this paper to study the problem, was a referential game, and the task was discrimination task. The agents can take 2 roles of speaker and listener, such that speaker gets some observation and send a message to listener, and listener gets some other observation along with speaker's message, and it has to make a decision in a discriminative setup. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and well-organised. \n\nThe problem understudy is an interesting and can be an impactful problem to study. The idea of the paper is not novel and it was studied in previous papers. Also, the game that was designed is a referential game that you can find very similar games in other emergent communication papers.\n\nI found the methodology used to solve the problem, i.e. train agents, not a very good approach! There are some weaknesses when it comes to the used methodology. One of them is centralized training! In multi-agent systems, using centralized training violates a fundamental assumption that information should not flow between agents. The other weakness is related to using a discrimination loss and treat the problem as a supervised learning rather than reinforcement learning. The discrimination loss is a very strong signal such that training such agents will be an easy task. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have 3 main concerns:\n1. Centralized training: Although we see in the litrature that centralized training may be an acceptable approach in some cases, I believe for the problem understudied in this paper, we need to train agents independently. The authors argued that only gradients flow from listener agent to the speaker agent, and it doesn't contain information about parameter values of the other agent. I believe gradients contain information about inner state of the listener agents such that without having this information, training speaker agents may be hard! The nature of emergent communication research require independent training as the message sent by speaker shouldn't be tuned based on inner state of listener, but only by the task reward.\n2. Supervised training: did you try to train agents using reinforcement learning as well?\n3. Some of the conclusions are not clear to me. For example, the authors showed communication success graphs in Figure 2 and 3. Is it the results after you trained the agents and took the best performing ones? I don't know how to interpret the results shown as I believe the entire training should be shown (at each timestep). In Figure 2, the fact that population size doesn't affect communication success rate may mean all the agents in the population are copies of each other! Is that the case?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "BrGvxFa5d-1", "reviewer": "Reviewer_c94V", "summary": "This paper proposes a number of experiments within a novel referential game where agents must learn to communicate among a population of agents. Authors design a novel referential game played in an artificial dataset of shapes and colors, where speaker and listener have access to a description of the true shape and color but are only able to either perceive shape or color. Thus, an agents' message must fill in the missing information of their corresponding listener, such that the listener can identify the correct target image.\n\nWith this setup, the authors design an experiment where there are a group of agents, and agents either are jointly trained with all other agents, or in more isolated connectivity settings (e.g. agents only speak to one or two other agents). Their aim is to see how the group size and connectivity of the group affects the learned communication protocol along a variety of axes:\n\n- Q: how similar are the agents' utterances as the group size increases? (A: agents maintain communicative success, but messages become *more diverse* as the group size increases.)\n- Q: Are agents able to generalize to new partners within a population, and to unseen tasks? (A: As agents learn from larger numbers of agents, zero-shot generalization to other agents also increases; agents do show some generalizability to unseen tasks, though it seems to vary based on group size and the complexity of the task distribution.)\n- What is the optimal communicative graph which results in the least number of links between agents, but most homogenous communication?", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a fairly well-written paper with a number of interesting findings on how group size effects learned communication protocols in multi-agent environments. However, I also see several weaknesses related to the odd experimental design, imprecise evaluation, lack of (discussion of) broader applicability of these experiments and comparisons to related work. Thus, right now, I'm a bit on the fence about this paper, and am hesitant to unequivocally recommend acceptance.\n\n## Strengths\n\nThis paper has some interesting findings:\n\n1. Perhaps the most potentially interesting is the finding that larger community sizes increase diversity of the languages (L243-244), yet maintain linguistic success. This is perhaps the opposite of what the human linguistic evolution literature might suggest, and similar papers (e.g. Tieleman et al. 2019).\n2. The experiment for identifying the optimal communication graph to maintain intelligibility among the agents is quite interesting. I imagine this might have some interesting downstream uses, but it's not quite clear to me. I'd be interested in the authors thoughts on this.\n\n## Weaknesses\n\n### Unusual experimental design (reference game) limits comparability to past approaches.\n\nAuthors design a variant of the reference game where agents play both the role of the speaker and listener, and are divided into agents that can only perceive *shape* and those that can only perceive *color*. In the game, unlike the typical referential game, both speaker and listener agents receive an instruction which denotes the features of the target shape, though since speaker and listener are unable to identify one of the features (shape/color) this is insufficient information to solve the task from the instruction alone. Therefore presumably the speaker must convey a message that encodes which objects have the target feature it can recognize. This is less of a referential game and more of a partial-missing-information game.\n\nI feel like this design choice is somewhat odd and insufficiently motivated. Authors claim that this setup encourages *symmetry* of the game setting, in that agents do an equal degree of speaking and listening; but this doesn't seem unique to this experimental design, and can indeed be implemented in standard Lewis referential games (we simply endow an agent with both language understanding and generation modules). Agents with dual speaker/listener roles have also been implemented in Graesser et al., 2019, Choi et al., 2018.\n\nIn the absence of any distinct advantages of the currently presented referential game setup, I have a strong prior towards using as standard of an experimental setup as possible; this concern generally falls under the broader concern about the extent to which the findings found in this paper can be generalized to other agents, environments, and datasets.\n\n### Imprecise evaluation\n\nSome of the evaluation in the paper is seems vague and ad-hoc, and I would have liked more standardized/systematic ways of evaluating the emergent language. A clear example is L231, the authors group messages according to criteria identified themselves, e.g. \"the target object is \"blue\" and it is at the 3rd position in the observation.\" They display only 3 types of messages, but do not state what kinds of messages were identified and vaguely state that \"ther are more than three types of messages we can identify.\" in Footnote 6.\n\nThis is all quite confusing: it's hard to understand what criteria the authors used to categorize the messages, and how the meaning of such messages was verified (how do we know that a message means \"3rd position in the observation?\"). Furthermore, the authors use a t-SNE plot to illustrate that it gets harder to identify which agent transmitted what messages as the group size increases. I find it really hard to examine parts of a t-SNE plot, with 3 messages labeled according to categories that are unknown to me, and conclude based on the closeness of some of the dots in the plot that \"what is entailed in the message becomes more relevant when identifying the message than who is sending the message\" (and even this line is a bit inscrutable to me).\n\nSimilarly the t-SNE plots in Section 5.4, Figure 4a-e apparently plot messages for color-only speakers that \"have similar observations\", but again this is quite vague, and the conclusion is a bit unclear to me (how does the scattered-ness of 4e illustrate the rise of \"dialects\"?) \n\nFinally, in L248, details of how the dataset tests generalization is crucial to the conclusions of this section. It's hard to reconstruct what is happening from the few sentences here (\"diversify colors\"). These details shouldn't be left for the Appendix.\n\n### Artificial experiments only\n\nOnly one artificial shape dataset is explored, and the data and experimental design (as mentioned above) limit the generalizability of the present results. The dataset uses only artificial colors and shapes. Moreover, the experimental design requires cleanly dividing agents by the dataset characteristics into one of two features (color/shape)—how commonly will we encounter such a setting in practice? Do the conclusions about the regularity and systematicity of the languages based on population size extend to more tasks where it is harder to extract the relevant features (e.g. more realistic image-based reference games) or for tasks besides reference games entirely? The \"simple\" and \"complex\" tasks explored at the end of section 5.3 lack calibration - we obtain \"simple tasks\" by reducing the colors and shapes by one or two possibilities, but perhaps these tasks are both extremely simple in the context of other environments we might deploy our agents in. Thus, I'm hesitant to make broad conclusions that the authors do, e.g. \"in complex environments, the emergent protocol generalizes better as the group size increases.\"\n\n### Could use more discussion about broader impacts\n\nThere are several experimental results presented in this study, where many parameters of group communication are verified, but aside from being interesting experiments, it's somewhat unclear what the takeaways are for either people interested in linguistic evolution or people who want to build better multi-agent systems.\n\nWhat are some examples we might face in practice, in multi-agent communication settings, where the communication graph of a population is not fixed, is bipartite w/ a clean division in information accessibility, etc?\n\nThe optimized communication graph experiment in section 5.5 is interesting, but it's quite unclear to me why we would want to do such a thing. For example, we find that one node emerges as \"hub nodes\" which regularize the communication protocol across the population. Are we meant to take away from this something about human linguistic evolution? When would we want a non-fully-connected communication graph in a multi-agent setting?\n\n### Some overlaps and lack of comparison to related work\n\n- Tieleman et al. Shaping representations through communication: community size effect in artificial learning systems. NeurIPS ViGIL 2019. The communication experiments with continuous messages in the authors' paper are very similar to the experiments here. Some of the conclusions of the authors should be compared/contrasted with Tieleman et al. Seems like authors conclude that languages become more discordant (but still successful) when group size increases, while Tieleman et al. almost observe the opposite: larger group sizes result in *less idosyncratic, better* learned representations.\n- The Graesser et al. paper is particularly relevant; it's mentioned that there are similarities in L106, but not at all referred to for the rest of the paper, though I feel the similarities warrant more discussion. The Graesser paper is dismissed as irrelevant beacuse it \"assigns agents into hard clusters of communities\"  but this is a bit unclear to me - from what I recall Graesser et al. also explore communicative graphs with loosely-defined groups where there are more in-links than out-links.\n\n## Missing references\n\n- Tieleman et al. Shaping representations through communication: community size effect in artificial learning systems. NeurIPS ViGIL 2019\n- Raviv et al. Larger communities create more systematic languages. Proc Roy Soc B, 2019. (In general I'd like some comparisons to the human linguistic evolution literature).\n\n## Minor\n\n- I think \"varying group sizes and connectivities\" would be a helpful edit to the title, as sizes/connectivities is ambiguous - I first interpreted it as size/connectivities of the architectures of speaker/listener implementations\n- How exactly are the models limited to only either perceiving color and/or shape? I feel like I missed an actual discussion of this in the paper?\n- L124 instruction is not a pixel-based object, I assume?\n- L172 How is the VAE trained?\n- L325 \"...evidences the benefit of emergent communication involving larger number of participants rather than concerns.\" unclear sentence - what does concerns mean here? Do you mean it's better to have larger group sizes than smaller group sizes?\n- [David, 1969] citation should be [Lewis, 1969]", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "BhiTagwtsl8", "reviewer": "Reviewer_e6Vg", "summary": "The paper investigates the results of Emergent Communication learning in different settings (number of agents, communication connectivity, task difficulty). The authors use a reference game variant where the communicating parties are limited in their perceptual abilities and hence need to communicate in order to solve the task (select a specific object among a number of alternatives). The authors obtain a number of interesting observations (for example, observing that larger groups of agents result in more diverse communication protocols, which (in certain settings) still allow to achieve high communication success rate).\n\n## Strengths \n- The authors tackle a highly relevant problem\n- The experimental setup aims to investigate the impact of a number of factors that may indeed be crucial in determining the properties of emergent communication\n- The paper has a large number of illustrations and is generally well-written, which promotes understanding.\n\n## Weaknesses\n- The decision to use early stopping, while motivated by valid concerns, does not actually address these concerns. A universally controlled amount of training in terms of total number of interaction per agent would be much easier to interpret. Ideally, more than one schedule should be tested: 1-training until convergence, 2-training for a set number of interactions for any given agent.\n- Literature review overlooks a number of relevant works and thus inadvertently presents the experimental setup as more novel than it is. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality, novelty & relevance\n\nThe paper addresses a relevant problem by providing a new take on a well-studied referential game setup.\nAlthough the setup explored in the paper is quite close to a number of existing works, I believe that the paper expands previously known results and modifies the setup enough to be considered sufficiently novel.\n\n## Clarity \n\nThe paper is generally well written and is a pleasure to read. The authors also provide a number of high-quality visualizations that further promote understanding and improve the reading experience.\n\n## Literature review\n\nThe literature review and, generally, a number of branding decisions severely limit (in my opinion) the quality of the paper.\n\nFirstly, some quotations are not very precise. For example, authors say that they adapted a symmetric setting since it \"allows all agents to function as both speakers and listeners and suppresses potential unfavored bias that might arise in the asymmetrical setting that potentially distracts our analyses on the convergence of a shared language\", quoting Bouchacourt and Baroni, 2019 (https://arxiv.org/pdf/1905.11871.pdf). But the quoted material does not seem to support the claim. Here is a quote from the cited paper: \"However, we also find that even perfectly symmetric agents converge to distinct idiolects instead of developing a single, shared code.\"\n\nSecondly, certain phrasing decisions sound like the authors claim to introduce things that are, in fact, already thoroughly explored. For example, the authors say \"we construct a unique setting within which communication among agents is indispensable for achieving their shared goals\", which makes a confusing impression, since such a description is true for most (if not all) referential game settings. Similarly, on line 59, the authors phrase the symmetric nature of their game as a novelty, while it was investigated in numerous existing works.\n\nPerhaps, the most concerning thing is that the literature review, while giving a broad overview of the field, does not discuss in enough detail the works most closely resembling the paper.\n\nFor example, the effect of population size on emergent communication outcomes was considered in\nhttps://direct.mit.edu/isal/proceedings/isal2020/32/678/98420 \n\nIn a similar vein, the authors say \"Our work is distinctive in a way that we do not assign agents into hard clusters of communities, and therefore, we observe finer granularities of linguistic continuums at agent-levels rather than at community-levels\", but https://arxiv.org/abs/2007.09820 studies the effects of connectivity structure on emergent communication without hard clustering. \n\nFurthermore, the authors \"introduce message agreement and communication success rates used to evaluate language convergence\", without mentioning https://arxiv.org/abs/1903.05168 or https://direct.mit.edu/isal/proceedings/isal2020/32/678/98420 which both pay particular attention towards discussing metrics that can/should be used to study emergent communication in similar setups.\n\nFinally, the paper repeatedly mentions that it is not injecting architectural bias. E.g. \"without injecting architectural bias into the environment or agents that could favor or hamper communication, our goal is to investigate the effect of emergent communication and analyze how agents’ general agreement on the communication protocol changes under varying group settings.\" This claim is vague, and it seems to slightly misrepresent the situation. For example, one of the limitations of this work is that the agents are trained in a centralized manner (i.e. they \"merge\" during training to propagate gradients). While this decision is understandable and common, it is unrealistic for modeling real-world communication system evolution, and is an example of a bias that fosters communication learning.\n\nOverall, the literature review helps to place the work in the global context of the field, but omits a number of works that closely resemble the present contribution. Additionally, a number of phrasing decisions are sub-optimal, giving the paper an impression of making unwarranted claims.\n\n## Experiments\n\nMost of the experiments are reasonable and provide insight into the problem in question. There are, however, limitations.\n\nOne of the questionable decisions is using the Early Stopping criterion. While the authors mention the problem in the appendix C3, their solution does not seem satisfactory. For example, under their schedule, when N=2, the agents stop training after 1000 epochs. At the same time, for N=128, 60000 epochs is required. On the first glance it seems that each agent in the group adds ~500 epochs to training and that each agent gets the same amount of exposure to training stimuli. But judging by the code, (multiagent_communication.py), the epoch size itself depends on the population size. As a result, agents in larger communities get exposed to a larger set of stimuli, which is crucial for generalization.\n\nThis limitation could explain the results in Figure 3. For complex tasks, agents in small communities simply learn too fast to get exposed to a large enough pool of stimuli, hence they don't generalize well.\n\nThis seems to contradict what authors wrote in lines 252-260. In particular, they say that \"number of observations provided to agents during training is maintained or even decreases depending on the early stopping criteria\". It's not clear how this conclusion was reached. I would appreciate a clarification on that matter.\n\nLastly, some experiments have unclear interpretations. For example, figure 4 claims that dialects only form in the cases with relatively large maximal communication distance. At the same time, it seems that dialects do form in all conditions, but the dialect groups are very small. Additionally, the experiment seems to contradict the claim on line 69-70. The authors state \"At the same time, distant pairs of agents that have not interacted during training can still communicate.\", while figure 4k strongly suggests that distant agents actually fail to communicate.\n\n## Conclusion\n\nThe paper tackles an important problem and provides a number of interesting results. \n\nMany of the key experimental results, however, are either weakened by the early stopping problem or are not as clearly interpreted as authors present them (for example, the qualitative evaluation of what is happening on figure 4). \n\nThe literature review does not (in my opinion) properly represent the paper's relation to previous work and does not properly separate novel contributions from previously known results.\n\nAt present, I select \"marginally below acceptance\", but I will be open to read other reviewers' opinions and the authors' rebuttal. \n\nI would be especially glad to be wrong when it comes to the early stopping problem (I described my concern in the \"Experiments\" section of the review).\n\n## Questions\nI would appreciate a comment on my concerns in the Experiments and Literature review sections.\n\nAdditionally, for how many epochs were the agents trained to obtain results in Figure 4? Was the \"early stopping\" criterion applied as well?\n\n## On authors' response\n\nI have read the responses and increased my score by 1 point. The reason I can not select \"full accept\" is because although authors plan to rewrite parts of the paper and tone down the claims, I can not check the quality of the final product.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations were partially addressed, but I believe that the paper would benefit from expanding that section, especially when it comes to the potential confounding effects of early stopping.\n\nThere is no discussion of societal impacts, but I think it is reasonable since the paper is not addressing a question of immediate practical application.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the effect of population size and connectivity of the communication topology of those populations in emergent communication and referential games. While population size has received some (very limited) attention in the literature, how the topology of the communicating agents is organized has been quite understudied. The authors report both communicative accuracy as a function of populations/connectivity but also present analysis of the resulting languages.

Overall, this has been among the most productive authors-reviewers interaction I've experienced. Authors have worked with reviewers to clarify questions and address concerns. Some new content has been generated during the discussion period and so I would ask the authors to update their manuscript respectively to reflect these discussions.
Decision: Accept (Poster)