Multi-Agent Language Learning: Symbolic Mapping
Authors: Yicheng Feng
Zongqing Lu
Venue: ICLR 2022 Submitted (2022)
Keywords: emergent communication
multi-agent reinforcement learning

Abstract: The study of emergent communication has long been devoted to coax neural network agents to learn a language sharing similar properties with human language. In this paper, we try to find a natural way to help agents learn a compositional and symmetric language in complex settings like dialog games. Inspired by the theory that human language was originated from simple interactions, we hypothesize that language may evolve from simple tasks to difficult tasks. We propose a novel architecture called symbolic mapping as a basic component of the communication system of agent. We find that symbolic mapping learned in simple referential games can notably promote language learning in difficult tasks. Further, we explore vocabulary expansion, and show that with the help of symbolic mapping, agents can easily learn to use new symbols when the environment becomes more complex. All in all, we probe into how symbolic mapping helps language learning and find that a process from simplicity to complexity can serve as a natural way to help multi-agent language learning. 

Under review as a conference paper at ICLR 2022
MULTI-AGENT LANGUAGE LEARNING: MAPPING
SYMBOLIC
Anonymous authors Paper under double-blind review
ABSTRACT
The study of emergent communication has long been devoted to coax neural network agents to leam a language sharing similar properties with human language. In this paper, we try to find 'natural' way to help agents learn compositional and symmetric language complex settings like dialog games. Inspired by the theory that human language was originated from simple interactions, we hypothesize that language may evolve from simple tasks to difficult tasks. We propose a novel architecture called symbolic mapping as basic component of the communication system of agent. We find that symbolic mapping learned in simple referential games can notably promote language learning in difficult tasks. Further, explore vocabulary expansion, and show that with the help of symbolic mapping, agents can easily learn use new symbols when the environment becomes more complex. All in all, we probe into how symbolic mapping helps language learning and find that process from simplicity to complexity can serve natural way to help multi-agent language learning.
INTRODUCTION
Agent communication has been a popular research field in the context of multi-agent reinforcement learning (Foerster et al., 2016; Sukhbaatar al., 2016; Jiang & Lu, 2018; Eccles et al., 2019). Recent work has focused on the emergence of language in cooperative tasks where neural network agents learn communication protocol from scratch solve problems together (Lazaridou et al., 2017: Das et al., 2017; Havrylov & Titov, 2017; Kottur et al., 2017; Li & Bowling, 2019; Ren et al., 2020). An array of work has empirically shown that agents can make use of their developed language to successfully complete the tasks. Beyond that, some work probed into the process of language emergence, and tried to figure out whether the learned language could share similar properties with human language like compositionality (Mordatch & Abbeel, 2018; Resnick al., 2020; Chaabouni et al., 2020; Choi et al., 2018) and symmetry (Graesser et al., 2019; Dubova & Moskvichev, 2020; Dubova al., 2020).
Most of these studies on emergent communication are based on referential games (Lewis, 1969) and have shown that compositionality can be induced by adding suitable environmental pressures. Some have explored the influential factors on symmetry of communication protocols among group of agents. However, tasks in these studies are often simple, and some of these methods are hard to implement in complex settings like dialog games. Kottur et al. (2017) found that in a two-agent multi-round dialog game, language with compositionality does not naturally emerge, unless strict conditions are imposed to agents, such as deprivation of memory.
Language emergence only in simple tasks obviously not satisfactory. In this paper, we tend find a new way to make compositional and symmetric language emerge 'naturally" in complex settings. Psychological studies suggest that human language was originated from simple gestures like pointing and pantomiming (Tomasello, 2010). This may explain why simple referential games are suitable for emergent language studies: these games are similar to 'pointing' in pragmatic process. However, from another perspective, the theory may also imply that communication protocols like human language cannot be formed directly from complex interactions. Instead, a natural process is probably that language is first formed in simple tasks, and then applied in more complex tasks. meanwhile it evolves to become more complicated and complete, similar to the concept of curriculum learning (Bengio et al., 2009). This is reasonable because a well-structured communication protocol is hard

---

Under review as a conference paper at ICLR 2022
to form, and complex setting difficult task exacerbates the problem. On the other hand, an important characteristic of language is that it is a general capacity. Once learned, it should be helpful in all kinds of tasks that need communication. Hence, we design two games, including a two-player referential game and a multi-round dialog game involving a group of agents, and investigate whether the same trend also arises on the communication protocol learned by neural network agents, so that language could evolve from simple tasks to difficult tasks and this process, which we call as task transfer, could help language learning complex settings.
In order to implement this process, we need find a way to enable agents to learn a common function for communication. because the speaking and listening policy can be different across tasks and thus should not be transferred directly. We propose novel architecture called symbolic mapping, which maps the input to related symbols, as a basic component of communication system of agent. The intuition is that when presented with the same input, we always associate it with the same pile of words and concepts, and this kind of association is consistent across tasks. This association does not determine the communication protocol, but it can encode language properties and be shared for communication at all time. Our experiments show that by implementing symbolic mapping, agents can achieve higher success rate in diffieult tasks after training in simple referential games, and the learned communication protocol presents better language properties.
As explore the learning process of agents from simple tasks to difficult tasks, are also curious about how the language becomes more complicated when old conventions are not enough in new environments. Language learning should not be accomplished overnight, and sometimes agents cannot learn language well if the environment is complex at the beginning. In more natural scene, agents should first learn a simple language in a simple initial environment, and when they enter a more complicated environment, they will learn something new and the language develops. We conduct the experiment about vocabulary expansion, and find that agents with symbolic mapping can learn to communicate using new symbols well in new environments. And through vocabulary expansion agents can accomplish tasks in complex environments where they would fail if they are asked to learn a language directly. This result once again reveals that process from simplicity to complexity is crucial for multi-agent language learning.
2 RELATED WORK
Cooperative games. Different kinds of cooperative games have been proposed in emergent commu- nication literature. A popular one is referential game (Lewis, 1969), where one agent, often noted as the speaker, has to send message describing a target (e.g., an image) which it has just observed to the other agent. Then the other agent, often noted as the listener, must select the target from several candidates containing the target and some distractors, after receiving the message (Lazaridou et al., 2017; Havrylov& & Titov, 2017). We design variant of referential game to serve as the simple task in our experiments. The game in Chaabouni et al. (2020) is most similar to our simple task, where the listener should reconstruct the target instead of picking out the target from a pool of candidates. The main difference is that in our task, the listener model is trained by reinforcement learning, while they use the entss-yntropy loss to train the listener.
Our difficult task is inspired by the Task & Talk game proposed by Kottur et al. (2017), which is a multi-round dialog game. In the Task & Talk game, there are two agents, one always asks questions while the other always answers questions. However, our task involves a group of homogeneous agents who do not play specific roles in the game. Besides, agents in our task can choose to end the dialog any time before the number of dialog rounds reaches the upper limit. But in the Task & Talk game, the number of dialog rounds is fixed. Other studies (Mordatch & Abbeel, 2018: Graesser et al., 2019; Fitzgerald, 2019) also concern emergent language in group of agents, and Evtimova et al. (2018) proposed a multi-step referential game. However, no game in these studies is similar to ours.
Properties of communication protocols. A mainstream research direction in emergent communica- tion is to find out whether neural network agents can produce communication protocols which exhibit some properties of human language. The most extensively studied property is compositionality. Many studies (Lazaridou et al., 2018; Li & Bowling, 2019; Ren et al., 2020; Resnick et al., 2020) have found that in referential games, once given appropriate environmental pressures. like changing learning environments, communication capacities or agents' model capacities, compositionality could be improved. Kottur al. (2017) found that compositionality does not emerge naturally in dialog
2

---

Under review as a conference paper at ICLR 2022
pair
label
AA
same
A
A
color
A
P
shape
B
reconstruction triangle, green
1
O
style
color
same
(a) pair dataset
(b)
discrimination game
(c) description game
Figure 1: Datasct and games.
games, which is also verified by our experiment. In the studies where group of agents learn their languages together, another important communication property is symmetry. That means an agent community should converge on shared communication protocol. Dubova et al. (2020) investigated the impact of different social network structures on language symmetry, while Dubova & Moskvichev (2020) explored some other factors including supervision, population size and self-play, In this paper, we focus on improving the two properties through a process from simplicity to complexity, and we propose an architecture called symbolic mapping implement the process.
Evolution of communication. Recent studies, inspired by linguistic theories, have brought evolution into the research of emergent communication. Cogswell et al. (2019) investigated the benefit from cultural transmission, while Dagan et al. (2021) integrated both cultural evolution and genetic evolution into emergent communication. Ren al. (2020) proposed neural iterated learning algorithm, where agents in a new generation are partially exposed to the language emerged from the previous generation. Li & Bowling (2019) let the speaker interact with new listeners periodically, while Graesser et al. (2019) analyzed how the learned language evolves when different linguistic communities come contact with each other. Most similar to our approach, Korbak et al. (2019) explored language learning across games of varying complexity by template transfer. Different from their work where hard task decomposed into several parts and the transferred agent the listener, explore language transfer from simple interactions to different tasks involving more complex communication forms, and the speaker is not reinitialized so that the language evolution is consistent. And we also explore the expansion of vocabulary.
Symbolic representation. Previous studies have explored symbolic representation in the deep reinforcement learning (RL) framework (Garnelo al., 2016; Garnelo & Shanahan, 2019), and found that a compositionally-struuiii in the deep RL systems. Symbolic mapping can be seen kind of symbolic representation in its function. Different from prior work, symbolic mapping is learned and constructed through emergent communication instead of representation learning techniques and is trained end-to-end by RL. That means agents form the symbolic representation when learning to communicate.
representation could help address several shortcomings inherent
3
EXPERIMENTAL FRAMEWORK
3.1 GAME SETTINGS
Discrimination game. We explore emergent communication in the context of a multi-round dialog game as the difficult game, illustrated in Figure 1b. Our game includes group of homogeneous agents, which we call community. Each game episode involves two agents i and which are randomly sampled from the community. They are presented with object o; and o, respectively. The object pair (o;, 0;) is sampled from pair dataset P. Each pair in P contains two objects selected from an object dataset D. Each object D comprises n attributes. For each attribute there are m(a) possible values. For a given n and tuple of value numbers we note the corresponding object dataset as Da,ms and the number
EE (1,2....,, (n)), m!))..... m = (m .m(2) of different objects will be | IT=", ml(") Given an object dataset D, the pair dataset P, as illustrated in Figure la, is then constructed for each pair (o,, where o,, oj E D, o, 0j or o and 0j have only one different attribute. If the objects are selected from Do,m. we note the pair dataset as P Pa. Note that different orders of o; and oj mean different pairs, since o, will be observed by agent i who will speak first in game episode. Moreover, each pair p = (o,, 0,) E P has label 1p. If o; oj. then I,, = O; otherwise if o, and 0, are different in attribute a, then I,, a.
....),....
After observing their respective objects, two agents start the dialog. At each time step t, the speaking agent should choose a symbol 8, from shared vocabulary V and send it to the other agent. Any
3

---

Under review as a conference paper at ICLR 2022
Input object
Word bark
A-
Symbolic Masping
Model both games
hess
hr
Listaning model deser iption game
LSTM
h2
Speaking
Reconstruction
hr
Network
Decision
softmax
Reconstruction Network
triangle
$
hws
LSTM
Reconstruction Network
Figure 2: Agent architecture.
agent, after receiving a symbol, can choose to continue or terminate the dialog. If the choice is to continue, then the receiving agent becomes the speaker at the next time step, and the players take turns to speak until the dialog is terminated. Suppose agent j chooses to end the dialog, then must answer whether and 0j are the same; if not, then which attribute is the different one. In other words, it must pick the true label I, for the pair (or,),). If the answer is correct, then both agents succeed and get reward r = 1. Otherwise, they fail and get no reward (r 0). If the number of dialog rounds reaches the upper limit T max. the agents also fail.
Description game. We also design variant of referential game called description game, depicted in Figure Ic. The game proceeds as follows. First, an agent é receives an input object o, from Da Next, it chooses fixed-length (n) sequence of symbols from vocabulary V to describe oN, and sends it to listener j. After that, j consumes all symbols and outputs ó, If o; o, which means j successfully reconstructs what agent is talking about, the agents succeed. Otherwise, they fail. The reward for speaker is according to the game result, namely = 1 if they succeed or r 0 if they fail. The reward for listener j according to its reconstruction of each attribute. In our setting, the listener has a reconstruction model for each attribute, Each reconstruction model gets r = 1 if its corresponding attribute is reconstructed correctly, and gets r 0 otherwise.
3.2 AGENT ARCHITECTURE
We propose a novel agent architecture which contains symbolic mapping, shown Figure 2. First represent it illustrate how symbolic mapping works. Concretely, for each attribute represented by the Na-dimension one-hot vector, where N₂ = m!()) An input object from Dn,m is concatenation of all its attributes. Symbolic mapping map(-). realized by a linear layer followed by a sigmoid function, maps the input object o; vector with dimension [V], and each element of the vector corresponds to the degree of relevance between a symbol and the object. Several symbols are sampled using the Bernoulli distribution for each element of the vector according the probability given by the output of the sigmoid function, and then stored as the agent's 's word bank. The number of sampled symbols, namely the size of the word bank, is not predefined limited.
In discrimination game, an LSTM f(() serves as an agent's memory. We initialize the hidden state ho as zero vector, and each time symbol is transmitted in the dialog, the symbol is fed into f-)). We encode symbols into one-hot embeddings. To differentiate the speaker of each symbol, we concatenate a flag the embeddings. the speaker the agent itself, the flag is zero; if the speaker is another agent, the flag is one. Note that the agent does not know the identity of its partner. Suppose at time f, agent is ready to speak, and agent j is its partner. Each symbol s in the word bank of agent i will be encoded into a one-hot embedding. Then each embedding is concatenated to the agent's s hidden state hi; and the speaking network 9%p (-), realized by a 2-layer MIP, takes each of them as input and outputs score for each symbol s. All symbols in the word bank get scores by the shared speaking network (-), and all scores will be passed through a softmax function to get a probability distribution = (-) over the word bank wi, and a symbol 8; will be sampled from TPp (-). The symbol is then fed into both agents' LSTM f'(() and (+))). At the next time step & 1, agent j passes the concatenation of its hidden state h/ and input object oj into a decision network Tyec (-), realized by a 2-layer MLP, and outputs an action 1 If the action is to continue, then it is time for agent j to speak. Otherwise, the action means the answer, and both agents get the corresponding reward.
4

---

Under review as a conference paper at ICLR 2022
In description game, the architecture of the speaker agent is the same above, except that the decision network is not used. We fix the message length to n, corresponding to one symbol per attribute. To do this, after the speaker produces symbol 8 at time step t, the symbol is fed into its memory network f()), and the next symbol 8++1 is sampled at time step t + 1. This process proceeds until the fixed message length is reached. The listener is instantiated by n linear layers, which are called reconstruction networks. The message sent by the speaker is represented by the bag-of-words model and consumed by the listener. Then each of its reconstruction network outputs an action to predict the value of each attribute of the object.
In both games, we use REINFORCE (Williams, 1992) to train each agent end end. We apply entropy regularization in the loss function to encourage exploration, and use the Adam optimizer with learning rate 0.001 in all settings. We run all our experiments three times with different random seeds and present the mean and standard deviation of the results.
3.3 THE BENEFIT OF SYMBOLIC MAPPING
In this section we illustrate the benefit of symbolic mapping briefly. One advantage brought by symbolic mapping is the capability to easily transfer between different tasks. While neural network has limited generalising capabilities to new tasks, the symbolic association should represent more abstract concepts which can help task transfer. Besides, though the mapping is simple, it can encode the basic component of a language, so can help maintain language properties across tasks. For example, compositional structure of the symbolic mapping can help maintain compositionality. Another benefit is that symbolic mapping is suitable for vocabulary expansion.Sii Since our speaking network chooses symbols from the word bank rather than sampling from a fixed-dimension distribution, we can explore vocabulary expansion just by adding outputs to symbolic mapping so that agents can associate input objects with more symbols. We will verify these benefits in our experiments.
3.4 METRICS
Compositionality. In our setting, the evaluation criterion of compositionality is whether agents can communicate different attributes independently. Note that compositionalty in natural language has more complicated forms, but we only consider the juxtaposition of independent symbols to represent an overall meaning because we hypothesize that compositionality was rather simple when language was formed in the early stage and thus the proposed form is adequate for our research. Inspired by positional disentanglement in Chaabouni et al. (2020), we propose a metric called referential disentanglement (refdis), which measures whether specific symbol refers to a specific attribute. We ignore the positional information because we need a language suitable for different kinds of interactions, and if symbols' positions informative, the language is hard transfer to dialogs.
For each symbol s in the vocabulary, we denote af the attribute that has the lowest conditional entropy given af arg min, (a|s). Similarly, we denote az arg min H(a)s), which has the second lowest conditional entropy. Then define refdis as:
refdis Σ H(az)s) H(ajls) H((s)) H(a))
k(s),
(I)
where k(s) is the firequency of occurrence of symbol The intuition of equation I is that each symbol should only be informative about one attribute. The best case is when one attribute is determined but all other attributes are totally uncertain given any specific symbol, with refdis being 1, and in the worst case the refdis is O. Context centexx-independdeee (CI) proposed in Bogin et al. (2018) shares similar concept with refdis, but refdis evaluates compositionality according to symbols while CI focuses the alignment between symbols and concepts.
Symmetry. We evaluate the symmetry of the learned communication protocol by computing the Jensen-Shannon divergence between pairs of agents' distributions of different values of attributes, given a specific symbol. For pair of agents and j. we define referential divergence (refdiv) as:
refdiv-
V ΣΣ JSD (p(mç (a, s)lpp(mfla, s)),
(2)
where p(mç (a, 8) is the value distribution of attribute a of agent given symbol s. The value of refdiv is also between O and 1, and perfectly symmetric communication protocol will get refdiv 0.
5

---

Under review as a conference paper at ICLR 2022
Table 1: The performance of the agent community playing discrimination game dataset (3. LSTM
refers to vanilla LSTM based agents, while refers to LSTM agents trained with iterated learning. The first and second column shows the success rate in training set and testing set respectively.
Training (%) Testing (%) refdis
refdiv 4
LSTM 47. 47.62(2.54)
8.42(1.27)
0.07
0.87(o.11)
IL
45.67(0.66)
13.47(1.27)
0.06m.001)
0.87(0.02)
4 EXPERIMENTS
4.1 LANGUAGE LEARNING IN DISCRIMINATION GAME
We first examine the performance of neural network agents leaning language in discrimination game, which is a multi-round dialog game involving a group of agents, and the round number is not fixed. We test two methods: vanilla LSTM, which is aimed to show the performance of simple LSTM- based agents in the game without particular training methods, and iterated learning (IL), which is framework proposed by evolutionary linguists to simulate the language evolution process, and is believed to help compositional languages emerge (Kirby et al., 2014). To apply iterated learning in setup, modify the neural iterated learning algorithm (NIL) proposed by Ren et al. (2020). The implementation details of LSTM and II. can be found in appendix. We use dataset (3. and refer to the attributes color, shape and style, and each of them has 3 values (i.e., red, green, blue, triangle, square, circle, solid, dotted, filled). We also split the dataset into the training set and the testing set to explore the generalization ability of the learned languages, which can also reflect the compositionality. We set agent number to 3, and the vocabulary size is set to 9. The upper limit for the number of dialog rounds is Tmax (each agent has three turns speak).
Table I shows the results, where refdiv is averaged over all pairs of agents. We can find that both two methods get poor performance in discrimination game. The success rates reveal that agents encounter difficulties in learning a good policy to accomplish the game, and their learned communication protocols are overfitting the training set, which implies that the language is not compositional. The low refdis also verifies this. Besides, the results of refdiv show that the agents do not converge on symmetric communication protocols. These results confirm that the multi-round dialog game is challenging for good language to emerge. And methods like iterated learning may not work well in complex settings, though the IL agents achieve relatively higher success rate in the testing set.
We conjecture that the difficulty may come from the following reasons. For compositionality, agents need to express a complex object with multiple symbols each referring to a component element. In ppeaker-listened game, agents free to send all relevant symbols at time, and the structure of language can be determined by themselves. But in a dialog, an agent cannot predict what its partner will say in the next round, and when the dialog will be terminated. On one hand, the language structure must be flexible enough to respond to different coming messages, which is harder to be explored. On the other hand, this kind of instability may push the agents to convey more information each time (e.g., using one symbol to express both two attributes), ending up in a non-compositional communication protocol. For language symmetry, in an agent group, different partners may decode a same message in different ways, and as a result the communication will be unstable and hard to converge on a shared communication protocol. On the other hand, agents cannot get high success rate if they speak differently, so the demand to converge on symmetric language makes language learning more difficult. Therefore, learning language directly in discrimination game is hard. However, as aforementioned, natural process is probably that a language is first formed in simple tasks, and we should not train agents in the complex settings to learn a language from scratch.
4.2 FROM SIMPLE TASKS TO DIFFICULT TASKS
In this section, we want to verify our hypothesis that language can evolve from simple tasks to difficult tasks, and this process, which we call as task transfer, helps language learning in difficult tasks. To do this, we first carry out deseription game on the agent community, and then train the learned speakers to play discrimination game. And we want to investigate whether our proposed symbolic mapping architecture can indeed promote this process, so we use LSTM and IL introduced in the previous section to serve our baselines.

---

Under review as a conference paper at ICLR 2022
Table 2: The performance of the agent community playing with shared listener in description game dataset (3,33,3,3) SM refers to agents with the proposed architecture. The two metrics are calculated on both symbolic mapping and communication protocol for SM agents.
Success Rate (%) refdis f
refdiv +
LSTM
100.00ga.0ny
0.48(0.07)
0.06(0.04)
100.00(0.00)
0.71
0.19(0.03)
protocol mapping
0.89:0.06)
0,12().33)
SM
100.00(0.00)
0.71(0.20)
0.04(0.04)
Table 3: The performance of the agent community playing discrimination game after they have learned to accomplish description game.
Training(%)
Testing(%)
refdis f refdiv 4
LSTM IL
85.80 82)
51. (1.01
0.34(0.05)
(.28(0.08)
51.13(4.87)
15.66(5.82)
0.05(0.03) 0.75
protocol mapping
0.62(0.08)
0.18().06)
SM
94.17(4.98)
(8.35
0.37(0.09)
0.06(0.01)
4.2.1 LANGUAGE LEARNING IN DESCRIPTION GAME
To conduct a Srraker-listener game in an agent community, most studies make each agent both speaker and listener to simulate human community (Dubova & Moskvichev, 2020; Dubova et al., 2020). However, we argue that agent community performs differently from human community so that this way makes language learning more difficult. When human beings learn expressions from each other, they tend to imitate them and speak in the same way (Garrod & Doherty, 1994), so their speaking and listening are tied together. But for neural network agents, it is a different story since their speaking policy and listening policy are separate from each other. Concretely, after agent learns from agent by listening, may speak to agent & in another way. From this perspective, the setting where each agent is both speaker and listener can be seen multiple speaking models speaking to multiple listening models, making the learning unstable and hard converge.
Therefore, instead of giving each agent a listening model to interact with all other agents, we choose to use a shared listener to simplify and stabilize the language learning. And as mentioned above, different partners decoding same message in different ways makes symmetric language hard to be learned, but the shared listener can solve this problem. As we need each agent in the agent community play the speaker role to train the speaking policy, we additionally introduce another agent to play the listener role.
We use dataset (3.33,3.) and set agent number in the community to 3 and vocabulary size to 9, the same as in Section 4.1. The message length is set to 3. The results are shown in Table 2. SM refers to agents with the proposed architecture in Section 3.2, and for SM agents we calculate the two metrics on both symbolic mapping (which symbols are stored into word bank) and the actual communication protocol (which words are sent to another agent) explore their relationship. All methods can learn to accomplish the game perfectly, and results of refdiv show that agents can converge on symmetric languages more easily in this simple referential game. Besides, the languages that emerge in this game present much higher compositionality compared with language learned in discrimination game, confirming that simple tasks are more suitable for agents learn language with good properties.
Among the three methods, LSTM agents achieve relatively poor compositionality, showing that agents cannot learn compositionality so well without any environmental pressure, in line with conclusions in other studies. IL. agents perform much better in terms of compositionalty, so the method can indeed help in this simpler game. The relatively poor symmetry may be caused by the supervised learning phase in iterated learning, where each new agent learns language from different agents in the past generation. Languages learned by SM agents present best compositionality. This may be because that the symbolic mapping naturally promotes compositionality, since the association between input and symbols can be easily disentangled. High refdis and low refdiv calculated on symbolic mapping also indicate that after language learning, the mapping can encode good language properties.
7

---

Under review as a conference paper at ICLR 2022
4.2.2 TASK TRANSFER
After the agents have successfully learned to accomplish description game, we then train the speakers to play discrimination game. For LSTM agents, use the learned model directly in the new task. For IL agents, we use the learned model to perform task transfer in the first generation. For SM agents, we load the learned symbolic mapping to reinitialized models, and we do not fix the symbolic mapping so that it can continue to evolve. The experiment settings are the same as in Section 4.1.
The results are shown in Table 3. The performance improvement of LSTM and IL compared with that in Table I proves our hypothesis that making languages formed through simple interactions first and then applied in more complex tasks be natural way for agents learn good language in complex settings. Further, the best performance of SM agents confirms the benefit of our proposed architecture. In different kinds of games, agents need different speaking policies, so LSTM and IL agents, who transfer the speaking policies directly, cannot generalize so well to the new game. IL agents perform relatively bad in task transfer probably because that in the last few generations when training in the simple game, they reinforce the successful policy again and again, and they learn the policy for the simple game so firmly that the generalization to new task becomes more difficult.
In contrast, SM agents learn new speaking policy from scratch in the new game, while the symbolic mapping provides knowledge about the learned language implicitly. The association from input to symbols encodes the information and properties of a language which is not tied to specific tasks, so SM agents can transfer the language more easily, and maintain the properties like compositionality and symmetry better. We note that refdis caleulated reason is that there are several redundant symbols associated, which may reflect that the symbolic mapping becomes more conservative in the difficult game. However, the speaking policy is not constrained by it, and SM agents can make use of the symbolic mapping to find a compositional language.
symbolic mapping here is relatively low. The
4.3 VOCABULARY EXPANSION
We have empirically shown that our agents' language can evolve in task transfer, and in this section we explore whether language can evolve when the environment becomes more complex. In natural language, it is common that vocabulary changes continually over time and new words are created endlessly, so we hope language emerged by agents can also develop. Besides, the emergence of language should not be accomplished overnight, and a natural process is to form the language step by step.
We explore this question by conducting the experiment called vocabulary expansion. We first carry out description game using LSTM and SM agents dataset (3,(4.444 which contains 64 objects. We set agent number to 3 and vocabulary size to 12. The results are shown in Table 4. It is surprising that in this bigger dataset, both methods fail in the simple task. LSTM agents learn only to speak a single word all the time, and the symbolic mapping learned by SM agents is nearly random. The reason is probably that when the object number in the environment is big, the chance to succeed is very small at the beginning, e.g., 1/64 in this setting. Then the reward is very sparse and reinforcement agents will find it hard learn.
Now we try to make agents learn the language from simpler start. We first train the agents on a smaller dataset D2.(4.4). and then we introduce new attribute into the environment and train them on 4.,44, This simulates a setting where agents do not care about objects' styles at first so they only learn to communicate about colors and shapes, but when they find that there are also many kinds of styles of objects, they start to learn to communicate about them too.
When training the description game on D2,(4.4) we use zero pero-padding to object representations and symbol embeddings to encode the new attribute and new symbols, and we set message length to 2. The vocabulary size is set to at first. For LSTM agents, the output number of the speaker network is set to 12, but we mask 4 of them in the first training. When training the three attribute game, the message length is added to 3, and the vocabulary size is expanded to 12. We use the learned model directly for LSTM agents. For SM agents, we reinitialize the agents' symbolic mapping layer with output dimension dim = 12 and set the weights to be zero. Then we load the parameters of the learned symbolic mapping into it. We also try to reinitialize the speaker network and the LSTM
linear
8

---

Under review as a conference paper at ICLR 2022
Table 4: The performance of the agent community playing with a shared listener in description game on dataset (4,44,4.))
Table 5: The performance of the agent community playing with a shared listener description game on dataset (2,(4,4)
Success Rate (%) refdis f refdiv +
Success Rate (%) refdis f refdiv +
LSTM
1.56(0.00)
0.00 1.00r0.00)
LSTM
100.00t0.00)
0.64(0.12) 0.1 (
protocol 2.77(0.60)
0.09/0.06) 0.75(0.07)
protocol mapping
0.84(0.0)) 0. (0.02)
SM
SM
100.00.0.00)
mapping
0.03(0.01) 0. (.
0.59 18) (.05
Table 6: The performance of the agent community playing with a shared listener in description game on dataset (4.44.44) after vocabulary expansion. SM-reinitialized means the speaker network and the LSTM network of SM agents reinitialized.
Success Rate (%) refdis f
refdiv
LSTM
83.85(22.65)
0.47(0.25)
0.14(0.05)
protocol
(.91(0.03)
(.10())02)
SM
100.00(0.00)
mapping
0.73(0.10)
0.05(0.01)
protocol 100.00ç0.00)
(.91(0.01)
0.12(0.04)
SM- reiitiiiti
mapping
0.72(0.04)
0.06(8.02)
SM
SM-reinitialize
LSTM
1.0
1.0
frequency
frequency
troquency
0.75
0.5
0.5
0.50
0.25
attribute values
Figure 3: The frequencies of attribute values observed by LSTM and SM agents corresponding to four new symbols in the vocabulary expansion experiment. The four colors of bars correspond to four new symbols respectively. The label abbreviations attribute values, and the last four are values the new attribute.
network of SM agents, only retaining the symbolic mapping, to investigate the effect of symbolic mapping in vocabulary expansion.
Table 5 and Table 6 show the results of the two experiments. While agents can learn good language in the small environment, they can also achieve good performance the bigger environment now via vocabulary expansion. This demonstrates that language evolve to become more complicated as the environment develops, and this process is crucial for agents to learn language in complex environments. The results also reveal that SM agents not only express new attributes with the help of new symbols, thus achieving higher success rate, but also use the symbols more compositionally. Note that the reinitialized model performs close the not reinitialized model, showing that symbolic mapping plays an deterministic role in vocabulary expansion. From this perspective, symbolic mapping is good for language development.
better vocabulary expansion they can
We present an example of the frequencies of different attribute values observed by LSTM and SM agents corresponding to four new symbols in Figure 3. SM agents mainly use the new symbols to express values of the new attribute, showing good compositionality. In contrast, LSTM agents fail to use the new symbols to express accurate meanings after vocabulary expansion.
CONCLUSION
In this paper, we argue that process from simplicity to complexity is natural way to help multi- agent language learning. We have proposed symbolic mapping as basic component of an agent's s communication system, and implemented it in LSTM-based agents. This architecture can be applied in different kinds of interactions, that it can help realize language transfer across different tasks. We conduct experiments about task transfer and vocabulary expansion, and the results show that learning from simplicity to complexity indeed helps, while symbolic mapping greatly promotes the effect of these two processes. We conclude that symbolic mapping is not only good for language transfer, but also good for language development.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML,, 2009.
Ben Bogin, Mor Geva, and Jonathan Berant. Emergence of communication an interactive world with consistent speakers. arXiv preprint arXiv: (80,000499 2018.
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. ACL, 2020.
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent compositional communication learning from raw visual input. In ICLR, 2018.
Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of compositional language with deep generational transmission. arXiv preprint arXiv: 2019.
Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Co-evolution of language and agents in referential games. In EACL, 2021.
Abhishek Das, Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. In ICCV, 2017.
Marina Dubova and Arseny Moskvichev. Effects of supervision, population size, and self-play on multi-agent reinforcement learning communicate. ALIFE, 2020.
Marina Dubova, Arseny Moskvichev, and Robert Goldstone. Reinforcement communication learning in different social network structures. arXiv preprint arXiv:2007.09820) 2020.
Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication in multi-agent reinforcement learning. In NeurlPS, 2019.
Katrina Eytimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in mudti-modal, multi-step referential game. In ICLR, 2018.
Nicole Fitzgerald. To populate is to regulate. arXiv preprint arXiv:1911.04362 2011,
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In NeurIPS, 2016.
Marta Garnelo and Murray Shanahan. Reconciling deep learning with symbolic artificial intelligence: representing objects and relations. Current Opinion in Behavioral Sciences, 2019.
Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. arXiv preprint arXiv:1609.0551| 2016.
Simon Garrod and Gwyneth Doherty. Conversation, co-ordination and convention: An empirical investigation of how groups establish linguistic conventions. Cognition, 1994.
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In EMNLP, 2019.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In NeurIPS, 2017.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In NewrlPS, 2018
Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution of language. Current opinion in neurobiology, 28: 28:108-114, 2014.
Tomasz Korbak, Julian Zubek, Lukasz Kucinski, Piotr Milos, and Joanna Raczaszek-Leonatde Developmentally motivated emergence of compositional communication via template transfer. arXiv preprint arXiv: 1910 2019.
10

---

Under review as a conference paper at ICLR 2022
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In EMNLP, 2017.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. ICLR, 2017.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In ICLR, 2018.
David K. Lewis. Convention: A Philosophical Study. Wiley-Blackwell, 1969.
Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communi- cation. In NeurIPS, 2019.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAAI, 2018.
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional languages emerge in neural iterated learning model. In ICLR, 2020.
Cinjon Resnick, Abhinav Gupta, Jakob N. Foerster, Andrew M. Dai, and Kyunghyun Cho. Capacity, bandwidth, and compositionality in emergent language learning. In AAMAS, 2020.
Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning multiagent communication with
backpropagation. In NeurIPS, 2016.
Michael Tomasello. Origins of human communication. MIT press, 2010.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229- 1992.

---

Under review as a conference paper at ICLR 2022
A TRAINING AND IMPLEMENTATION DETAILS
In all of our experiments, each agent's LSTM has a hidden state of size 50, the dimensions of the hidden layers of all MLPs are the same as their input size, and the entropy regularization parameter Aн is set to 0.05. We train LSTM and SM agents for 10000 epochs in description game and 20000 epochs in discrimination game, unless the agents achieve 100% success rate ahead of time.
The LSTM agents are implemented as LSTM networks with hidden states of size 50. When an LSTM agent observes an object, linear layer maps the input embedding into the agent's initial hidden state hg. When speaking, we map the agent's 's hidden state into a probability distribution over the whole vocabulary with an MLP and softmax function, and we randomly sample a symbol from the probability distribution. The generated symbol will then be fed back into the LSTM. The decision network is the same as SM agents.
We modify the neural iterated learning algorithm to apply iterated learning in our setup. The IL agents' architecture are the same as LSTM agents. The algorithm runs for several generations, and there are three phases in each generation: learning phase, interacting phase and transmitting phase. At the beginning of each generation, all agents are randomly initialized. When training description game, in the learning phase, each agent in the community learns from data collected in the previous generation with cross-entropy, and the shared listener is pre-trained with REINFORCE by interacting with the pre-trained agent community. In the interacting phase, the agent community plays description game with the shared listener and they are trained the same way as LSTM agents. In the transmitting phase, all objects are fed to each speaking agent, and the corresponding messages generated are stored in a dataset for the next generation. When training discrimination game, in the learning phase, two agents are randomly sampled to learn dialogs with supervised learning from data collected in the previous generation, and the rest agent is pre-trained with REINFORCE by interacting with the pre-trained other two agents. In the interacting phase, the agent community plays discrimination game and they are trained the same way as LSTM agents. In the transmitting phase, two agents are randomly sampled, and the whole training set is fed to them to collect the generated dialogs into dataset for the next generation. In description game training, we set generation number to 20, pre-train iteration number to 2000 for supervised learning and 3000 for reinforcement learning. We train agents for 2000 epochs in the interacting phase. In discrimination game training, we set generation number to 10, pre-train iteration number to 40000 for supervised learning and 100000 for reinforcement learning. We train agents for 4000 epochs in the interacting phase. We tried set of hyperparameters and use the ones with the best performance.
B EXAMPLES OF THE LEARNED SYMBOLIC MAPPING AND COMMUNICATION PROTOCOL
To show what symbolic mapping learns and how it helps task transfer, we conduct the task transfer experiment smaller dataset (3.33.3) and present here some examples. We refer to the attributes as color and shape, and each of them has 3 values (i.e., red, green, blue, triangle, square, circle). The vocabulary size is set to 6, the message length is set to 2 in description game and the upper limit for the number of dialog rounds in discrimination game is T Tmaxx 2.
Examples of the learned symbolic mapping in the agent community is shown in Table 7 and Table 8. They verify that symbolic mapping is not changed greatly across two tasks, so the learned language can be transferred. In both games, all agents associate symbol 'O' with attribute 'green', 'I' with 'circle', '2' with 'blue' and 5 with 'square", which presents good compositionality and symmetry.
Table 7: The learned symbolic mapping of the three agents in the community when playing with shared listener in description game dataset D2,(a.a).
red
green
blue
red
green
blue
red
green
blue
triangle
3,4
0,3,4
2,3,4
triangle
3,4
0,3.4
2,3,4
triangle
3,4
0,3,4
2,3,4
square
0,5
2,5"
square
5
0,5
2,5
square
5
0,5
2,5
circle
0,1
1,2
circle
1,4 0,1
1,2
circle
1
0,1
1,2
12

---

Under review as a conference paper at ICLR 2022
Table The learned symbolic mapping of the three agents in the community when playing discrimination game after they have learned accomplish description game.
red
green
blue
red
green
blue
red
green
blue
triangle
3,4
0,3.4
2,3,4
triangle
3,4
0,3,4
2,4
triangle
3,4
0,3,4
2,3,4 2,3,5
square
4.5
0,3,4,5
2,3,5
square
4,5
0,5
2,5
square
3,4.5
0,3.5
circle
1,4
0,1,4
1,2
circle
1,4
0,1,4
1,2
circle
1,3,4
0,1,3
1,2,3
Table The learned communication protocols the three agents in the community when playing with shared listener in description game on dataset D2,(3,3)
red
green
blue
red
green
blue
red
green
blue
triangle 3,4
0.3
2,3
triangle
4,4
0.4
4,2
triangle
4,4
0.4
2,4
square
5,5
5,0
5,2
square
5,5
5,0
5,2
square
5,5
5,0
5,2
circle
1,1 1,0
1,2
circle 1,1 1,0
1,2
circle 1,1
1,0
1,2
Table 10: The learned communication protocols of the three agents in the community when playing discrimination game after they have learned accomplish description game.
red
green
blue
red
green
blue
red
green
blue
triangle
4
2
triangle
3
0,4
2,4
triangle 3
2
square circle
4.5
2,5
square
4.5
0,5
2,5
square
3.5
0,5
2,5
1,2
circle
1,4
0,1
1,2
circle
1,3
0,1
1,2
Symbol '3' and '4' have relatively ambiguous meanings, which is changed between two tasks, but they mainly cover the attributes 'red" and 'triangle' which cannot be expressed by other symbols. So agents can form compositional structure in symbolic mapping through emergent communication. and the properties like compositionality and symmetry shown in symbolic mapping can explain why symbolic mapping helps language learning through task transfer and why the learned language properties in simple tasks can be maintained complex tasks by SM agents.
We also present the corresponding communication protocols learned by the agents in the experiment Table and Table 10. As discrimination game can be terminated at any time, agents may not have chance express complete information. So in Table 10 only present all symbols that the agent has spoken in different games after observing a specific object discrimination game.
Compared with Table and Table 8. the communication protocols make of the compositional words in symbolic mapping faithfully in both games, so the language is indeed transferred across tasks. Besides, good compositionality and symmetry exhibited in description game also transferred, which helps success rate in discrimination game.
may seem odd that the first agent only speaks symbol '0' after observing all green objects in discrimination game. We point out that results from its game policy: always expresses 'green' and wait the other agent communicate about the shape. That may explain why we think speaking policy should not be transferred directly like LSTM agents: policies can be specific to tasks, while only more basic components like symbolic mapping can carry general information about a language.
We should also point out that though the third agent associates symbol '3' with all objects in discrimination game in symbolic mapping, it only speaks it when presented with red objects. This may explain why refdis can be higher in protocol compared with mapping.
C DETAILED ILLUSTRATION OF SYMBOLIC MAPPING
For clarity, we present here a more detailed illustration of symbolic mapping in Figure
13

---

Under review as a conference paper at ICLR 2022
Input object
Symbolic
Mapping
|
MV|
[V
Input object
B000000 0101
Word bank
sample
Linear
OOD
s
Figure 4: The architecture of symbolic mapping.
14


Reviews:
[{"review_id": "lHnjIIQPAj", "reviewer": "Reviewer_rYPh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI have a number of larger issues with the paper. Some of this may stem from me not being able to understand the paper. The writing isn't very clear. Concepts are seemingly introduced in a random order. The authors move around between architectures, tasks, training regimes - which made it very hard to follow for me.\n\nThe paper is trying to make too many different points including a \"new\" architecture, a \"new\" game, a \"new\" curriculum approach. Some of these seem trivial or known in prior work. \n\nIt's quite known in RL more generally and in language emergence more broadly that curriculums are helpful. I don't think the paper really adds anything interesting to that discussion.\n\nUnfortunately, I did not understand well how the agent architecture works. It's not clear how the word bank works and what the speaking networks and reconstruction networks are. But from what I understand - the symbolic mapping is basically an MLP that looks up words for objects. It's these words then that are fed to the recurrent sentence generator. This seems like a prior that works in the particular setting here. However, it also seems like a trivial extension.\n\n\nImportant baselins such as IL are not explained in the paper.\n\nSome major details are missing. E.g. the different MLPs such as the reconstruction network - how many layers etc\n\nOther Comments and Questions\n\nwas originated -> originated\n\nThe definitions around objects in 3.1 don't seem to make sense. If a \\in {1,2,...,n} and there are n attribues in each object - are you saying that all objects are the same? What's m? It seems to be the number of values per attribute. but you already said that the value of a is one of these {1,2,...,n}. That's quite confusing\nAre you saying that all attributes have the same number of possible values in your experiments?\n\nIt's not clear to me what is the word bank and what does it consist of. \n\nWhat is a reconstrution network? The term is used once and never reappears. Is it the rec networks in your figure? Are they per attribute?\n\nWhat aren the dimensions for your different MLPs\n\nWhy do you use $s$ and $w_i$ for the word?\n\n\"then both agents success\" -> \"then both agents succeed\"?\n\nI think it's not useful to use the term discrimination game and then suggest that it's more difficult since it requires dialogue. Discrimination games have been introduce in the 90s and they are referential and they do not necessarily require dialogue\n\nSimilarly people have used the term \"decscription game\" in emergent communciation research since the 90s and it is precisely not a referential game in the classical sense.", "summary_of_the_paper": "The paper proposes symbolic mapping. The main point of this method is to associate objects with words in processing before making pragmatic decisions about what should be said. THis is then part of a larger recurrent architecture that produces words to describe or discrimate objects. The architecture is trained using REINFORCE - in different scenarios. Another point the paper is trying to make is that training the symbolic mapping on a simple scenario and the explanding complexity helps. The method is evaluated for success but also in terms of its ability to create compositional and symmetric language.", "summary_of_the_review": "\nThe paper unfortunately suffers from some unclear writing as well as some lengthy unimportant descriptions and some important aspects that are missing\n\nPRO\n* interesting domain with lots of problems\n* main claim of language emergence from simple to complex is illustrated by one example. But this is such a general claim - it would require a lot more experiments\n* the main claim of the paper is not really a technical claim \n\nCONS\n* some key explanations are unclear.\n* many less important explanations are lengthy and convoluted making it difficult to follow\n* somewhat trivial domain (that is pretty common though in this line of research)\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "gXuhRes2ght", "reviewer": "Reviewer_ZcF5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Papers like this are hard for me to evaluate. There has been quite a bit of previous work on looking at what environmental pressures and inductive biases lead to the emergence of compositional communication. In these papers, the environments considered are so simple that the systems developed are unlikely to be practical for any near-term applications. The contributions in these papers are instead scientific -- do the experiments help us understand something new about how simulated languages develop? Are the experiments rigorously performed?\n\nI would say this is not a bad paper. The paper uses an environment that, while simple, is in fact a bit more complicated than environments in previous emergent communication papers. I would lump their 'task transfer' and 'vocabulary expansion' results under the general banner of curriculum learning -- if you first train on a simpler task, then train on the more complicated task, you will do better at the more complicated task. This is pretty intuitive, I'm a bit surprised that no previous paper to my knowledge has made this specific point (though it may exist), but I think this is a positive contribution of the paper.\n\nThe paper also proposes a fairly specific LSTM-based architecture (SM) that further improves performance. I don't find this result particularly compelling -- given that it is mostly an engineering contribution, I'd want to see it tried on a broader range of tasks that we care about. One of the claimed benefits of SM from section 3.3 is that the architecture allows us to do 'vocabulary expansion', but the results indicate that vocabulary expansion seems to help the base LSTM policy just as much as the SM policy. I don't find the remaining arguments and intuitions in 3.3 general or compelling, but I may be missing something.\n\nUltimately, if we take the main contribution of the paper to be 'forms of curriculum learning help compositional languages to emerge', I don't think this paper presents experiments that are quite extensive enough for the reader to really understand *how* curriculum learning helps. I'd want to see more ablations, e.g. looking at multi-step curricula (the paper only considers 2 steps), effect of initial task difficulty, etc.. \n\nOverall, I think the paper is not quite at the level of acceptance, and would currently recommend rejection.", "summary_of_the_paper": "This paper investigates methods for emergent communication. Specifically, it looks at inductive biases that can contribute to the emergence of compositional and symmetric languages. The main environment the paper studies is a multi-turn dialog game, where two agents are shown objects that differ by 0 or 1 attribute and have to agree on the differing attribute.\n\nThe paper makes 2 main contributions:\n\n1) It investigates the role of curriculum learning, and shows that first training on a simpler task -- a referential description game (task transfer), or the discrimination game with fewer properties (vocabulary expansion) -- helps agents learn compositonal languages on the broader discrimination game.\n\n2) It proposes a new agent architecture, 'symbolic mapping', which can help agent performance and increase compositionality.", "summary_of_the_review": "While the paper shows that curriculum learning can help compositional language emergence, which is interesting, I'd want to see more extensive experiments and ablations to recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XXRgyuJkztU", "reviewer": "Reviewer_CC9y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Post-rebuttal Edit\nIn light of the added clarity regarding symbolic mapping, I am upgrading my review from 5 to 6 and correctness from 1 to 3.\n\n## Strengths\n- It presents an intuitive yet novel approach to emergent language: transfer from a simpler to more complex task.\n- The work is complete in itself but clearly points the way towards future directions.\n- The structure of experiments provides the right kind of empirical support for the premise.\n\n## Weaknesses\n- The explanation of the core method, symbolic mapping, is unclear, which makes a difficult to evaluate exactly what is happening with the task transfer.\n- The experiments use a small number of trials (3) in very small environments which raises concerns about the robustness of the observed effects.\n- It seems as if refdis is selected as the metric for compositionality and then discarded in Section 4.2.2 Paragraph 3 with the claim that the communication protocol is compositional anyway despite the low refdis.\n\n## Recommendation\nI recommend \"reject\" in the current form of the paper because even with careful reading, I am not able to clearly understand what symbolic mapping is, a key component of the paper.\nI would be happy to change my recommendation with revisions which clarify the explanation of symbolic mapping.\nLess important, but running a greater number of trials and giving a rigorous qualitative explanation when compositionality does not line up with refdis would strengthen the paper.\n\n## Justification\nThe paper up to Section 3.2 is well written and motivated, but Sections 3.2 and 3.3 (and Figure 2) are critical yet unclear.\nAn understanding of symbolic mapping bridges the conceptual gap from task transfer as a motivated method of doing emergent language to concrete instantiation of the agents in neural networks.\nAs a result, I cannot determine if the experimental results are trivial or informative.\nDirect references to the location of issues in the paper are given in `Additional Comments`.\n\nSpecific criteria:\n- Correctness: 1\n    - Although I cannot judge the experimental design fully until I understand symbolic mapping, I believe that the experiments largely support the claims (i.e., 3).\n    - A larger number of trials and a clearer analysis of refdis (in the context of the experiments -- it's formulation is clear) would be needed in addition for a 4.\n- Technical Novelty and Significance: 3\n- Empirical Novelty and Significance: 2\n    - As mentioned above, a larger number of trials is needed.\n    - Additionally, the empirical significance is lessened by the fact that only the only environments that work are of the scale $(3,3,3)$ and $(4,4)$.\n\n## Questions\n- What exactly is symbolic mapping? Equations and examples can always make things clearer.\n- In what way is refdis an inadequate metric (if at all)? Is there a better alternative that could be used for this paper?\n\n## Additional Comments\n- `s1`: Contributions should explicitly stated in the introduction\n- `s1`: \"curriculum learning\" should at least be mentioned somewhere since it is a common concept in machine learning\n- `s3.1 discrimination game p1`: \"is comprised of\" -> \"comprises\"\n- `s3.1 discrimination game p2`: use $m_a$ or $m^{(a)}$ instead of $m^a$ -- it took me a minute to realize it was indexing and not an exponent\n- `s3.1 description game p1`: Is there any reason the simpler game is second? It would seem to make a bit more sense going first.\n- `Figure 2`: This figure is very unclear insofar as the most important details are left unspecified.\n    - Do the triangle and the symbolic mapping box have any relation? Or do they just happen to be adjacent?\n    - How is the green triangle represented numerically, since it is presumably being concatenated with a vector?\n    - What is the symbolic mapping box? What is coming out of it?\n    - What are the inputs and outputs to the word bank?\n    - Is the word bank generating multiple $s$'s? Why is that?\n    - Why are there three copies of the speaking network and why do they appear to feed into the same softmax?\n    - What is the purpose of the decision network and what $v_t$?\n    - Is the robot face in the top right corner supposed to represent anything?\n    - What is the rec network?\n    - What is $m$ in the listening model?\n- `s3.2 p1`: What does it mean to \"sample symbols according to the probability given by the output of the sigmoid function\"?\n    Please be clear with what the symbols and the word bank are because they are critical to understanding symbolic mapping.\n    What are the values in the vector and why do they correspond to \"relevance\"?\n    What is the probability distribution being used here, is it a Bernoulli distribution for each element of the vector or a categorical distribution over the whole vector?\n- `s3.2 p2`: Initializing with Gaussian noise seems a bit more typical and could maybe aid in robustness slightly. Just speculation on my part, though.\n- `s3.2 p2`: \"id\" -> \"identity\"\n- `s3.2 p3`: What does it mean for a symbol to be encoded into a one-hot embedding?\n- `s3.2 p3`: What is the distribution $pi^i_\\text{sp}$?\n- `s3.2 p3`: What is the action $v_{t+1}$?\n- `s3.2 p3`: \"an one-hot\" -> \"a one-hot\"\n- `s3.2 p4`: \"In description game\" -> \"In the description game\"\n- `s3.2 p4`: What does it mean for the speaker to be represented by the \"bag-of-words model\"?\n    I am familiar with BoW in the context of text features, but I am not sure how it is applied here.\n    Does that just mean that the listener receives a sum of vectors which the speaker has generate?\n- `s3.2 p5`: 3 independent runs seems very small. On the order of 10s seems like a minimum while 100s is even better.\n- `s3.4 Compositionality`: I see how positional disentanglement is inappropriate for this task, but BoW disentanglement from the same paper might work just fine -- it is very similar to referential disentanglement as given here. Furthermore, 15 minutes of notebook scribbles leads me to think that it might not suffer from the issue of redundant symbols for a given attribute (i.e., bosdis is still high unlike refdis).\n- `s4.1 p1`: \"unfixed\" -> \"not fixed\"\n- `s4.2.1 p2`: If the experiments are using a shared listener in the dialog setting, wouldn't this undermine the concept of refdis measuring _symmetry_? It seems like it would be measuring something more like consensus or inter-speaker agreement since the agents are not properly symmetric (i.e., independent speaking and hearing models).\n- `Table 2`: What is the meaning of \"protocol\" and \"mapping\"?\n- `s4.2.2 p3`: It seems like the point is being made that compositionality is still present despite the low refdis.\n    This undermines the usefulness of the metric.\n    If the claim is that the metric does not capture compositionality in this case, a well-illustrated qualitative argument should be made to this end.\n", "summary_of_the_paper": "This paper models a new aspect of emergent language: developing language to complete a simple task before transferring the emergent language to a new, more complex task.\nIt does this by creating curriculum of a simple referential game followed by a slightly harder dialog task.\nA traditional architecture does not perform well on this task transfer so the paper introduces _symbolic mapping_ in order to encourage a compositional language which generalizes well.\n\nThe following empirical evaluations are performed:\n- discrimination (dialog) game only\n- description (referential) game then discrimination game with and without symbolic mapping\n- expanding vocabulary by transferring to a new task with new attributes to describe\n\nThe contributions of the paper are:\n- Introducing experiments which look at the progression from a simpler game to a more complex one\n- Developing a novel architectural component called _symbolic mapping_ which aids in both transferring to new tasks and in developing a compositional communication protocol\n- Introducing a metric referential disentanglement based off of positional disentanglement\n", "summary_of_the_review": "The paper presents a novel and motivated setting of task transfer and has the right start for experiments.\nOn the other hand, the explanation of symbolic mapping, central to linking the setting and experiments, is very unclear.\nI intend to change my \"reject\" to an \"accept\" if the symbolic mapping is sufficiently clarified.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EoPZsJzRUNE", "reviewer": "Reviewer_Litw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Weaknesses:**\n1. Unclear that SM, refdis/refdiv, and the reinforcement learning tasks are sufficiently original to stand alone.\n2. Missing significant literature.\n3. General issues with clarity \n\n**Strengths:**\n1. Well-motivated and well-designed experiments. \n2. Results make a strong case for further work in intermediate symbolic representations and support prior research!\n\n**Recommendation:**\nBased on the above reasons, especially due to lack of citations/originality, I vote to reject the paper in its current form. However, since there are strong empirical contributions, if the authors revise the paper to include the relevant prior work (details below), I'd willing to change my vote to accept the paper.\n\n**Questions and Additional Comments:**\n(organized by originality, technical quality, and clarity)\n\n*Originality============================================================*\n\nIn general, the paper can benefit from a more thorough discussion on how the new research differs from past work. There are several major areas in the paper where the previous literature is inadequately cited (see points 1-4). It's unclear that the authors' modelling contribution and reinforcement learning tasks are sufficiently original to stand alone. However, the experiments showing that SM is beneficial in task transfer are important in that they support prior findings, and the experiments on vocabulary expansion are novel, to my knowledge.\n\n(1) The main idea of symbolic mapping, which is mapping raw input to an intermediate symbolic representation, is not new and should be discussed in relevant work. See work such as \"Towards Deep Symbolic Reinforcement Learning\" (Garnelo, et al 2016) and \"Reconciling deep learning with symbolic artificial intelligence: representing objects and relations\" (Garnelo, et al 2019) for a starting point.  \n\n(2) The idea of bootstrapping in emergent communication from simpler tasks to harder tasks is not new and should be properly cited in section 2 (I suggest removing the statement \"orthogonal to existing work\"). See \"Developmentally motivated emergence of compositional communication via template transfer\" by Korbak, et al (2019) and its references for a starting point.\n\n(3) The literature in Section 1 on language evolving from simple to complex tasks can be greatly expanded. See Deacon's \"The Symbolic Species\" and other references in Korbak, et al (2019) for a starting point.\n\n(4) The concept of refdis is very similar to context independence introduced in \"Emergence of Communication in an Interactive World with Consistent Speakers\" by Borgin et al (2018) and discussed in \"On the Pitfalls of Measuring Emergent Communication\" by Lowe et al (2019). It would be good to differentiate context independence from refdis, discuss with a citation in section 3.4.\n\nNit:\n(5) Rewording the explanation of Eq. (1) to be less similar to the explanation of Eq. (1) in Chaabouni, et al (2020). \n\n*Technical details ============================================================*\n\nOverall, the authors explain technical details and modelling choices thoroughly, and well-motivate the training scheme/experiments. In particular, the LSTM ablation results are strong in that they suggest SM produces a good intermediate object representation. While the big-picture is technically sound, I disagree with some of the claims in the paper and would like to see a deeper discussion of referential disentanglement. I recommend editing the verbiage to be more task-specific instead of general (see point 1).\n\n(1) Section 3.3: \"symbolic association which is nearly orthogonal to tasks…\" I disagree. For example, in grounded language games, symbolic association is closely linked to the task. It is important to specify that this claim applies to *this specific setting* rather than in general tasks as implied.\n\n(2) The discussion of compositionality in Section 3.4 needs further development. Why does referential disentanglement, which is similar to context independence, capture compositionality in this setting? What are the limitations of referential disentanglement? Note that referential disentanglement works in this setting because the semantic composition rule of the attributes is intersective and not order-dependent. \n\n(3) Section 4.1: \"agent cannot predict… when the dialog will be terminated.\" Is this problem solved when fixing the number of turns instead of letting agents terminate the game?\n\n(4) Section 4.3: was NIL tested in vocab expansion?\n\n*Clarity ============================================================*\n\nThe implementation of the model and experiments are mostly clear. However, there are some statements that I found vague, some claims that can be better argued, and some adjustments that can improve the flow of the paper. \n\n(1) In Section 3.1, it's unclear what the turn taking structure is until Section 3.2. It would help to include a line saying players explicitly take turns unless the game is terminated.\n\n(2) The first two lines on pg. 4 are hard for me to understand. Why does detecting the difference between objects imply the agents may not be clear what their partner's symbols mean? I would think that it's never guaranteed that the agent understands the meaning of their partner's symbols.\n\n(3) In Section 3.1, Description Game, reconstruction model is used first and defined later. I would include a line explaining what a reconstruction model is. \n\n(4) Section 3.3: \"though the mapping is simple… help maintain language properties across tasks.\" What is intended by \"language properties\"?\n\n(5) Section 3.3: in the explanation of vocabulary expansion, the word \"vocabulary\" is overloaded. It's not clear upon initial reading that vocabulary expansion is increasing the size of the word bank rather than the base vocabulary.\n\nNit:\n(6) We fix the message length to n (unclear upon first reading why it's n) -> We fix the message length to n, corresponding to one symbol per attribute.\n\n(7) Please state both bounds for both refdis and refdiv.\n\n(8) Table 1: second and third column -> first and second column\n\n(9) Section 4.3: \"We have empirically proved that language can evolve from simple tasks to difficult tasks\" is too general. Change to \"We have shown that our agents' language/communication protocol evolved in task transfer,..\" or something more specific.\n\n(10) Figure 3: it's hard to tell blue/green and red/orange apart.\n\n(Not important for this review, but please clean up grammatical mistakes for future iterations.)\n", "summary_of_the_paper": "This paper has several main contributions: (1) A novel architecture, symbolic mapping (SM), (2) Showing SM outperforms vanilla LSTM and NIL in terms of success rate, degree of compositionality, and degree of symmetry on the description game and discrimination game after bootstrapping,(3) Showing SM aids vocabulary expansion moving from a smaller to a larger vocabulary and attribute space. \n\nSM, a linear module that maps an object to a subset of the symbol vocabulary, is intuitively an agent's internal object representation or \"word bank\". A recurrent speaking module then selects the speaker agent's utterance from their word bank. SM associates object to symbols in a task and speaker-agnostic way. This not only facilitates the transfer of these object-symbol mappings to different tasks but also expanded vocabularies.\n\nTheories of language emergence posit that human communication evolved to be more complex by adapting to increasingly complex settings. Following this hypothesis, the authors define two emergent communication games, a description game which is easier and discrimination game which is harder, and use SM to facilitate the transition between the two. They quantify performance using training and test success rate, as well as refdis (compositionality) and refdiv (symmetry). They find that SM outperforms a vanilla LSTM and NIL in the description game in all metrics. They show that all three models perform poorly trained from scratch in the discrimination game, then that their performance improves after pre-training speaker agents on the description game. Notably, when bootstrapping from the description game, SM achieves a higher success rate while also exhibiting a higher degree of compositionality and symmetry than the other two models. This supports the effectiveness of SM as an internal object representation for the agent.\n\nFinally, the authors conduct a vocabulary expansion experiment. They first show that both an LSTM and SM agent learn poorly  from scratch on a large attribute space and vocabulary size. Then, they show that all agents perform well on the same task after bootstrapping from a smaller attribute space and vocabulary size. In particular, SM (whether or not reinitialized) outperforms LSTM after transfer.  ", "summary_of_the_review": "The current version of the paper greatly overstates the novelty of its technical contributions (SM, refdis, task transfer). To me the primary contribution is therefore the synthesis of prior technical contributions, the strong experimental result, and the case for further research into symbolic representations. Only on the condition that the authors substantially revise their discussions of SM, refdis, and task transfer to credit prior work, I would recommend the paper for acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2l4azLGN8KY", "reviewer": "Reviewer_Z2BE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated and is easy to understand. The authors perform an extensive search over the relevant metrics used for computing compositionality and systematic generalization in an emergent communication setting. Although, the authors show promising results on the small 'toyish' dataset, there is no evidence or discussion on whether the findings would scale to more complex domains. The two different games used in the paper still only use symbolic data with only a limited number of training samples.\n\nThe task transfer experiment is not clearly stated. The authors state that symbolic mapping provides knowledge about the learned language implicitly. It would be better to demonstrate this with a small example on what exactly transfers from learning the simple solution to solving a more complex task. Moreover, the authors refer to increasing the complexity of the task to increasing the number of types of an attribute. How would this approach scale if we add a new dimension of complexity? Would it require fewer game interactions or samples than training an agent from scratch? More empirical evidence is needed to support the claim that the proposed module would scale to any complex task, or state how such a complex task should be designed.\n\nThe authors state that training agents in the discrimination game is harder than expected and the effect of symbolic mapping is not really pronounced in this game. A qualitative example would be helpful to highlight the possible issues with this game as compared to the description game.\n\n", "summary_of_the_paper": "The paper talks about introducing a new learning paradigm to train compositional and symmetric emergent languages while the tasks increase in complexity as the training progresses. The main motivation comes from works in psychology that show that human languages are also evolved from a simpler task to gradually increasing the task complexity. The proposed symbolic mapping module is shown to perform better than training agents directly on the complex task, thereby showing the importance of using past strategies to learn a solution for the new complex task.", "summary_of_the_review": "The authors propose a novel module that helps in training better compositional and symmetric language using a curriculum from simple to complex tasks. The paper is well-written and the experiments show that the proposed method indeed supports the hypothesis although the experiments are only performed on a simple environment. The complexity of the tasks can be varied along different axis but only a few options are explored to extend this.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript presents a novel approach to learning a shared language between multiple agents.

In general, reviewers had difficulty understanding the symbolic mapping component. For such a critical part of the manuscript, questions by multiple reviewers were extremely basic, asking what symbolic mapping even is. Authors did clarify this in the discussion and updated the manuscript, but further improvements to the manuscript are warranted.

Reviewers had concerns about the novelty of the approach. Including being confused about whether this is just an application of curriculum learning. Reviewers were also concerned about the lack of ablations.

Reviewers also had concerns about the fact that this is a toy domain. Symbolic mapping as defined in the manuscript appears to be possible only for such toy domains. It fundamentally wouldn't scale to simple language games with real images. This significantly limits the scope of the work. More broadly, reviewers wanted to see symbolic mapping exercised much more. If this is a useful idea, they wanted to see the authors apply it to other domains.

Reviewers were confused about many other details in the manuscript. For example, about the fact that refdis is later discarded as a metric, which the authors answered is due to redundant symbols ("the symbolic mapping is not a highly compositional representation here because of the redundant symbols"). Why redundant symbols lead to less compositional representations seem unclear.

With significant additional improvements to the clarity of the manuscript, a demonstration of how symbolic mapping is useful in another domain, and additional experiments suggested by multiple reviewers this could be a strong submission in the future.
Decision: Reject