Expressivity of Emergent Languages is a Trade-off between Contextual Complexity and Unpredictability
Authors: Shangmin Guo
Yi Ren
Kory Wallace Mathewson
Simon Kirby
Stefano V Albrecht
Kenny Smith
Venue: ICLR 2022 Poster (2022)
Keywords: Emergent Language
Expressivity

Abstract: Researchers are using deep learning models to explore the emergence of language in various language games, where agents interact and develop an emergent language to solve tasks. We focus on the factors that determine the expressivity of emergent languages, which reflects the amount of information about input spaces those languages are capable of encoding. We measure the expressivity of emergent languages based on the generalisation performance across different games, and demonstrate that the expressivity of emergent languages is a trade-off between the complexity and unpredictability of the context those languages emerged from. Another contribution of this work is the discovery of message type collapse, i.e. the number of unique messages is lower than that of inputs. We also show that using the contrastive loss proposed by Chen et al. (2020) can alleviate this problem.

Published as a conference paper at ICLR 2022
EXPRESSIVITY OF EMERGENT LANGUAGES IS A TRADE-OFF BETWEEN CONTEXTUAL COMPLEXITY AND UNPREDICTABILITY
Shangmin G Ren", Kory Mathewson Simon Kteeano fUniversity of Edinburgh, ;University of British Columbia, &DeepMind
Stefano V. Albrecht', Kenny Smith
ABSTRACT
Researchers are using deep learning models to explore the emergence of language in various language games, where agents interact and develop an emergent language to solve tasks. We focus on the factors that determine the expressivity of emergent languages, which reflects the amount of information about input spaces those languages are capable of encoding. We measure the expressivity of emergent languages based on the generalisation performance across different games, and demonstrate that the expressivity of emergent languages is trade-off between the complexity and unpredictability of the context those languages emerged from. Another contribution of this work is the discovery of message type collapse, i.c. the number of unique messages is less than that of inputs. We also show that using the contrastive loss proposed by Chen et al. (020) can alleviate this problem.
INTRODUCTION
Language games were first introduced by Wittgenstein 1954)
explore the meanings of language
utterances. Instantiating this concept with the signalling game design from Lewis| (969) enables
linguists to explore the emergence of linguistic structure (Kirby 2001 Kirby & Hurford 2002 where
artificial languages are represented as symbolic systems. The success of deep learning (DL) models
on complicated cognitive tasks (Krizhevsky et al. 2012 LeCun al.
2015 Silver et al. 2016 then
inspired researchers to apply DL-based models to language games to investigate the agents' ability to
invent communication protocols without preset rules (e.g. Lee et al. 2018
Lazariouu et 2018).
In the existing works (c.g. Lee et al. 20188) Lazaridou et al. 2018 Graesser et al. 2019 Zhang et al.
2019. Yuan et al. 2020), there are usually two types of agents, speakers that emit messages based on their observations (i.e. input target objects) and listeners that receive messages and act accordingly. as illustrated in Figure ! Based on the goals for listeners, we can categorise most the games into the following three types: 1) referential games in which listeners need to select the target object observed by the speaker among a set of candidate objects (candidates for short) (e.g. Lazaridou et al.) 2017:Rep et al. 2020 2) reconstruction games in which listeners need to reconstruct the speaker's observation (c.g. Chaabouni et al., 2020 Kharitonov et al. 2020): and 3) navigation games in which listeners need to go the location specified by speakers (e.g. Lowe et al. 2017 Kajié et al. 2020). We focus on referential games, illustrated in Figure Tal which have been well investigated in both the linguistic community (c.g. Kirby et al., 2014: Winters al. 2018 and emergent communication community (e.g. Lazaridou et al. 2018: Li & Bowling 2019).
It is reasonable assume that listeners in referential games can differentiate the target object from the other distractors in the context as long as the speaker's 's message encodes some unique feature of the target. Therefore, the speaker's 's messages must convey a different amount of information for listeners to complete their task in games where candidates are similar in different degrees. For example, to distinguish two very similar candidates, e.g. blue wide-stripped shirt and blue narrow-stripped shirt, it requires more information about the target to be encoded by the speaker's 's message than to distinguish a shirt from a cup. Experiments with human participants show that emergent languages are sensitive to the requirements of the communicative tasks for which they are used, with languages developing in which only the necessary discriminatory information is encoded (Winters et al. 2018).
& "Correspondence author, e-mail address: s. www.ed..c.cc.oooo .c..uk

---

Published as a conference paper at ICLR 2022
00, 0, 0, 1, 0,,00,
Empus
Spraker Mediclos
coa Desdor
Spedeer Messeg Cere (Lmm)
Message
Messagrat to
Lirzach Manage Zodde LSTO
Modills
Speaker
Listener
Coaldiar (4L7))
Condidato sxt
1. 1.0.0,0,,,1,,,,00000 0,0.0,11111,,,,00000000
AkApal wupabiiattiaieee pidia
(a) Illustration of the referential game we investi- gate in this work. Observations are concatenations of two 5-digits-long one-hot vectors. The objective of the listener is select among a set of candidate inputs (the context).
(b) Illustration of the agent architecture we implement for referential games. Blue modules are components of the speaker, and green modules helong to the listener. Orange and grey indicate the vectors generated by agents and input samples. respectively.
Figure 1: Diagram of referential game and the architecture of our agents.
Following Kirby et al. (2015), we refer such ability to encode information about input space (e.g. structure. dimensionality, distribution of samples) as expressivity. In this work, we explore the factors that influence expressivity in the framework of DL-based referential games, and we argue that it is important to understand these factors such that agents could develop languages that are effective enough for completing various tasks.
Our contribution in this work is threefold. First, we propose and verify a hypothesis about the determining factors of expressivity of emergent languages under the DL-based framework. Following Winters et al. (2018), we show that context size (i.e. the number of candidates the listener must select among when interpreting the speaker's message) influence both the complexity and unpredictability of that context. Complexity refers to the similarity between the items in the context: in order for the listener to select the target, more information needs to be encoded in the speaker's 's signal if the context contains more similar distractors for given target. Unpredictability refers to the extent to which the information that needs to be encoded by the speaker is stable across contexts in successive episodes of communication: in games where the information required by the listener differs from trial (training epoch for DL agents) to trial, the speaker needs to encode more information on every trial in order to be sure to encode the information the listener needs ! complexity and unpredictability are affected differently by the context size: as context size increases, complexity increases but unpredictability decreases. Therefore, we propose and verify the following hypothesis about the determining factors of expressivity:
As show in Section 3.1
[HyPOTHESIS) The expressivity of emergent languages trade-off between the complexity and the umpredictability of context in language games.
Our second contribution is a novel measure of expressivity based on partial ordering of languages in terms of their generalisation performances across tasks. Although expressivity is related to the amount of information encoded in a language, we illustrate that mutual information (MJ) is not an appropriate measurement for expressivity in Section 3.2 Considering that one of our objectives is to facilitate a universally useful emergent language, we propose to measure expressivity of a language based on the generalisation performance of listening agents trained with that language across different games. Since it is more challenging to quantitatively and directly measure expressivity, we focus on the partial ordering between languages in this first step towards understanding expressivity.
Our final contribution is the discovery of message type collapse, i.c. the number of unique messages significantly lower than the size of input space relatively easy games, which can lead to highly ambiguous emergent languages. To overcome the technical limitations imposed by GPU memory
size on large-scale referential games, we introduce the contrastive loss proposed by Chen et al.) 2020) in referential games. While comparing the behaviour of the contrastive loss and the loss function
used in previous works (e.g Havrylov & Titov 2017 Li & Bowling, 2019 Ren et al. 2020 Guo al. 2020), we find that this contrastive loss can greatly alleviate the collapse of message types, leading to more disambiguous and potentially more expressive emergent languages.
'Context, complexity, and unpredictability are defined and discussed in depth in Section
2

---

Published as a conference paper at ICLR 2022
2
GAMES, AGENTS AND LEARNING METHODS
2.1 GAME ENVIRONMENT AND ARCHITECTURE OF AGENTS
Environment Components As shown in Figure [b formally speaking, the environment in our referential games consists of: 1) input space X' from which the speaker's observations are drawn, which consists of 10, 000 possible inpute] where each æ X' is a concatenation of 4 one-hot vectors whose length is 10; 2) message space M consists of 10t 6-tokens-long sequences m (called as a message type) where the size of the token inventory 10; note that the tokens are initially meaningless, and meaning emerges through the negotiation of a communication protocol between speakers and listeners; 3) candidate set C which contains a target input 2; and |C| 1 distractors {xd, Idz...... ) sampled uniformly at random from X without replacement.
Agents The architecture of the agents we implemented is also illustrated in Figure [bb] 4) speaker $ consists of both a multi layer perceptron (MLP) for encoding the 40-digits-long target input æ; onto a 256-digits-lone
embedding h and a decoder for generating a message m based on long-short term memory (LSTM) network (Hochreiter & Schmidhuber 1997). 5) listener L is constituted by an LSTM- based sequence encoder which encodes m into a 256-digits- embedding h hmm), and an MLP for encoding the inputs which can be denoted by ())
Definition of an emergent language Following e.g. Ren et al. (2020). we also define an emergent language L as mapping function from the input space X' to the message space M.e. L: X - M, thus it can be represented as {( [(x;))\Var & } Note that L could be a non-injective mapping function, i.e. it is possible for two different inputs to be mapped an identical message thus the number of message types might be less than the number of unique inputs.
2.2 LEARNING METHODS OF AGENTS
In the previous works (e.g. Havrylov & Titov 2017). the action module of L is usually a softmax function which produces categorical distribution over the candidates based on the distance between hmmm and embeddings of each candidate (f (x;)) obtained through listener's 's candidate encoder (-). By comparing the loss function used in Havrylov & Titov 2017) and the contrastive loss function proposed by Hadsell et al. (2006), we find that the aim both functions is the same, i.e. to make the distance between positive pairs (æ; and m) closer while keeping the negative pairs (Ir and m) as far apart as possible. To effectively validate our hypothesis, we need to make the whole input space X' as candidate set C in which case the batch is too large to be stored in GPU memory (since the space complexity is (()))))) Therefore, to overcome this issue, we adapt the contrastive learning method proposed by Chen et al, (2020) referential games. Suppose that we have a batch of input samples B = (I1, L2, Ti.... Taal and the corresponding messages from the speaker are (MB, M. m....... mB b (note that it is possible that m. = mn for different i, which would be further encoded into (hiren, .hinn...... h/n,,.. :.... h hmm by the
listener L. Then the loss function for sample x; is the defined on a softmax function:
exp (h.. T. f(e)o(
& - log
(I)
Σ 1B] In this way, the number of candidates |C] exactly the batch size [B] during training. More details about the comparison between referential loss and this contrastive loss can be found in Appendix A
== exp (htn. Slene(ar))
As for updating the parameters, we use the Adam algorithm introduced by Kingma Ba 2015). and the learning rate is set to 10-4. To allow the gradients being propagated through the discrete channel to overcome the sampling issue of messages, we apply the Gumbel-Softman trick proposed by Jang et al. (2000), and the temperature hyper-parameter 7 is set to 1.0.
Our implementatior of games is based on the framework EGG developed by Kharitonov et al. 2211)).
The experiments across 6 random seeds, 18 source games, 11 target games took 4, 216 hours in total, on Nvidia Tesla P100. Definition of "source/target games" " are given in the following Section 4.2
AAcorrding to the results from Chaahouni et al. (020): an input space containing 10, 000 possible inputs is large enough for neural models to achieve > 99% generalisation accuracy.
Codess released at tttp://gghh...coomooeeee///////p wiw.rii--o---meeeeee----ananniie

---

Published as a conference paper at ICLR 2022
3 CONTEXT AND EXPRESSIVITY
3.1 CONTEXT, AND ITS COMPLEXITY AND UNPREDICTABILITY IN DIFFERENT GAMES
Context Communication between humans always takes place w.r.t some context (Winters et al. 2018). For humans, that context includes: 1) features of the environment that can be directly perceived by the dialogue participants and used for interpreting
utterance (e.g. "cup" is interpreted as an object in the immediate physical context); and 2) information that cannot be directly perceived but can affect the interpretation of utterances (e.g. using and interpreting "cup" assumes some shared knowledge of what counts as a "cup"; once particular cup has been established as salient in particular dialogue, further utterances of 'the cup" "it" will be interpreted as referring to that specific cup).
Inspired by this conclusion from pragmatics, in the DL-based language games, we define context as the supervision information involved in the calculation of losses, i.e. the context of a specific target denoted as C(zz). is the space of samples involved in the calculation of loss. As for in referential games, the cross entropy loss calculated based on the distance between the message embedding
h h (m.)) and candidate embedding (x:)) where o;; E C, thus C(æ,)) C = X in referential games.
Complexity of context We assume that the goal of communication is to distinguish target object from other possibilities in the context (as defined above). It therefore follows that the similarity of distractors in the context to the target influences the communicative precision required, and that greater precision is required to distinguish the target from a more similar distractor. For example, is relatively easy in natural language to distinguish e.g. cat from table (a relatively general label like "cat" or "animal" would suffice), but harder to make fine-grained distinctions between very similar objects e.g. Scottish Fold Cat and American Shorthair Cat (a specialist vocabulary lengthy description is necessary).
Following the assumption (described verbally above) that context which contains more similar objects makes the game harder because there are fewer unique features that suffice to distinguish the target from the distractors, we first define neighbour set in k-th degree of a given target æ, as (z,) = { d(x;, . & k} where d is a distance function that can properly capture the similarity between inputs, e.g Hamming distance in our setting. The complexity of C(x,) is then defined as the expectation of the probability that C(ar;) contains an object from N2 (ar)), i.e.
1. if Dæ;; e C(x,) s.t. a E Na(()))
Ex, [(() ,where g ((()))
(2)
0. otherwise
In our referential games, since the sampling procedure is independent Bernoulli without replacement, the value of the above expectation is then 1- 1 (w.))
[C]
which is a monotonically increasing
-1
function w.r.t |C'l and fixed k. That said, larger contexts are more complex since they are more likely to include items which are very similar to the target.
Unpredictabilityy of context Our notion of unpredictability comes from experimental work with human participants, e.g. Winters et al. (2018). Suppose the aim is to distinguish a striped t-shirt from a number of distractors, and there are two sequences context: 1) three runs of games where distractors are all cups; 2) three runs of games where distractors are a cup, a plain t-shirt, and pencil. In the first sequence of games, participants would become certain that "t-shirt" is enough for distinguishing the target, whereas in the second sequence participants would Iearn that more overspecified utterance (c.g. "striped t-shirt") is necessary to guarantee comprehension after a failure on the trial involving the plain t-shirt distractor. That is, the context in the first sequence is more predictable than the second. WWinters et al. (2018) show that human participants are sensitive to this kind of unpredictability, and adapt their communicative strategies accordingly.
In DL-based games, we refer to the context at the e-th trial, i.c. the e-th training epoch, of a target 20, as C (=x)). Following the above example, the unpredictability of context is then defined as the
4

---

Published as a conference paper at ICLR 2022
proportion of (*+1(())) that are not from (a;)), i...
Ex,
(ax, (xx)))
(x++ my (m))
In our referential games, since the sampling procedure is independent Bernoulli without replacement, the proportion of objects not from (24) in C*+1 (x+) is then simply [c"();) |C|
X (i.e.
|x]
smaller contexts are more unpredictable, since contexts on successive trials are more likely differ in their composition).
More details about the above discussion are provided in Appendix
3.2 MEASURING EXPRESSIVITY IN LANGUAGE GAMES
Following the intuition that the different types of games require messages convey different amount of information about input space X, MI may seem to be good candidate measure for expressivity. However, we found that even for two emergent languages that both deterministic and bijective mapping functions, i.e. they have the same MI value log( (l.))) their expressivity can still be different. Further discussion about MI can be found in Appendix D
Since it is more challenging to directly quantify the expressivity of emergent languages, we instead focus on the partial orders between them in this work. Considering that the purpose of the emergent communication is to complete tasks, we therefore propose to measure the partial order between expressivity based on game performance. This leads us to a common practice in the transfer learning community (e.g. Pan & Yang 2009), i.e. train models source domain but evaluate them different target domains. Inspired by this practice, here we propose definition of the partial order between the expressivity of two emergent languages LA and LB, i.e. CCA, and Czn:
Definition 3.1 (E) > E Let g be an instance of language game, and G a set of language game instances. Then, given G. if the generalisation performance of La is better than L on some language games G' with statistically significant difference, i.c. Vg E G', P2, > F2 where and } denotes the generalisation performance, while there is no statisticallyy significant
@ c G' S difference between the converged generalisation performance of LA and La the remaining games, B% Vg' Gi we then say that expressivity of L is higher than LB on G, i.c. cA eg The metric of generalisation performance could be varied according to different types of games.
Le.
33% CA
games, if one language performs better than the other on some games while they perform approximately the same the remaining games, we then say the expressivity of that language is higher than the other. Following this definition, there are several possible relationships
Briefly speaking, given
between the expressivity of two languages LA and LB: 1) eE > eg defined above; 2)
C. =c if = F2 Lg Vgg E G; 3) CB > 2a defined as above; 4) Co is incomparable to Cº (Ec, both true.
(C E..)) if propositions ágí e G, s.t. 32 > 33% and 3g" E G. s.t. T2 > T2 are
PREDICTIONS AND EXPERIMENT DESIGNS
4.1 PREDICTIONS FROM OUR HYPOTHESIS
As we illustrated in Section 3.1 higher complexity indicates more similar distractors to targets, thus requires messages conveying more fine-grained features of inputs. This leads to the following prediction:
1.
higher complexity in games with same contextual unpredictability can facilitate emergent languages having higher expressivity;
Note that in this work means that there is no statistically significant difference between means of two sequences.
5

---

Published as a conference paper at ICLR 2022
Meanwhile, following the conclusion from Winters et al. (2018) that higher contextual unpredictability leads to higher signal autonomy, i.. more comprehensive information about imput space is encoded in the emergent languages, then make the following prediction:
2. higher unpredictability in games having same contextual complexity can facilitate emergent languages having higher expressivity;
Following the above two predictions, both higher complexity and higher unpredictability can unilater- ally lead to better expressivity. However, with an input space whose size is finite, changing context size always affects the two factors in an inverse way, which leads our Hypothesis I following prediction:
and gives the
3.
if the size of X' fixed across games, a language which emerges in game with moderate complexity and unpredictability should have the highest expressivity.
4.2 LANGUAGE TRANSFER EXPERIMENT
To verify the above three predictions and thus our hypothesis we evaluate the generalisation perfor- mance of languages emerging in various games following the Procedure Briefly speaking, we first train a pair of speaker and listener to complete referential game, i.c. a "source game" g.. and obtain an emergent language. We then train new listener in another referential game, i.e. "target game" It, with only 90% of the imput-message pairs, and use the game performance of this listener with the remaining 10% pairs as the generalisation performance of gs on gt-
Procedure 1: Procedure for the language emergence and transfer experiment
Input: A set of source game G., set of target game Gr for every game g¹, in G. do
initialise a new speaker and listener for g', and train them to play 9 with the whole Xt;
2. after the agents converge on g.. record L = {(x, m) E X3; randomly shuffle and split L into 2 disjoint sets Lirain and Liest s.t. |trainl = 90% [C];
4. for every game of in G. do
1. initialise a new listener for gf:
2. train the listener with Lerain to complete gt: record the accuracy of listener on Ltest as the generalisation performance of g on g/:
end end
Since our hypothesis concerns the size of C, we propose to experiment on a series of referential games with varying candidate set size |C| ("referX"" where X= (C(). Mcanwhile, as these referential games have both different complexity and unpredictability, we also implement series of referential games with different |C| but the batches are fixed during training such that there is no unpredictability of the context ("referXf" where X= |C|). To sum up, in our experiment, 9, ("refer2/f" "refer10/f"" "refer100/1" 1000/"" "refer2500/f", "refer5000/1" "refer7500/f" "refer "refer10000"" 10000 Since lan- guage games with fixed context are not good simulation of human communication. we keep only the ones with varying batches from G. in G..
As a naming convention to allow us to refer casily to the factors that influence a language's 'ss expressivity, we refer to emergent languages by the parameters of their source game, e.g. "refer10f" refperoo@ssseeeentss represents a language which emerges in referential games whose |C| = 10 and context varies across epochs. More details about the naming of games and languages be found in Appendix E.1
5 RESULTS & DISCUSSION
To alleviate the effect from randomness as much as possible, all the following results are obtained with 6 different random seeds. All values are averaged across the seeds, and the corresponding standard deviation (SD) are also provided (either in this section or appendices).
There is no "refer 1000F"" since no unpredictability is possible when / = |C| = 10, 000 "referX" game while 'referX" emergent language from it.
|

---

Published as a conference paper at ICLR 2022
5.1 HIGHER COMPLEXITY SAME UNPREDICTABILITY HIGHER EXPRESSIVITY
To verify the Prediction 1 compare the generalisation performance of languages from fixed context, i.. the unpredictability of source games' context are the same, on Gt. and the results shown in Figure 2 Languages which emerge in source games whose contexts have higher complexity consistently perform better on all games than those which emerge in source games with lower-complexity contexts, since all lines keep growing with the increase of values æ-axis (i.e. complexity). That is, the generalisation performance is a monotonically increasing function of |C* if there is no unpredictability in the context, which matches with the prediction of our hypothesis.
2.00
5.95
Ga
refer2
Acoursyy
refer1
2.65
refer000
metr3500
refe19000
reeri5000
refer10000
0.75
2.70
1ef:201
refer2001
Fefer22001 TelS 1505
5ele10001 1efer20000
Source Games
Figure 2: Generalisation performance of languages from fixed context with varying complexity. The r-axis indicates the source games, and is the accuracy. Lines in different colours represent the generalisation performance on different target games. We plot only the means of the converged performance to keep the figure readable (SD values are provided in Table 3 in Appendix (E.2 and remove 'refer7500f" from x-axis since is mix [C] = 7,500 and = 2, 500.
5.2 HIGHER UNPREDICTABILITY SAME COMPLEXITY HIGHER EXPRESSIVITY
To verify the Prediction we compare the difference between the generalisation performance of language with varying context ('referX") and their corresponding fixed-context twins ('referXf"). As indicated by that all points in Figure 3 are positive, languages from context with varying objects always perform better on all target games. Since all "referXf" has lower unpredictability of context than the corresponding "referX" games, this indicates that higher unpredictability unilaterally lead to better generalisation performance of emergent language across various target games, which matches with the prediction of hypothesis.
0.4
0.3
Accuraty Difference
0.2
0.3
-0.1
refer2
refer10
refer
refer1000
refer2500
referSoco
refer2500
Source Games
Figure 3: Generalisation performance gaps between languages from varying context and their corresponding fixed-context twins on all target games. The r-axis indicates the types of source games, and the y-axis indicates the distribution of difference between the generalisation performance of 'referX" and 'referXf" on all target games (i.e. 'referX" "referXf"). Each point represent the difference on specific kind of target game, and the boxes show the corresponding values of medians, Sith-percentile, and 75-th percentile. "refer1 is not plotted since it doesn't have fixed-context twin game.
7

---

Published as a conference paper at ICLR 2022
5.3 EXPRESSIVITY TRADE-OFF OFF BETWEEN CONTEXTUAL COMPLEXITY AND UNPREDICTABILITY
To verify the Prediction 3
compare performance for all source games with varying context on all target games, and the results are shown in Figure 4 (see also Appendix E.3 for more details).
1.00
2.85
larget Games refe?2
0.90
Accuracy
refer10
refe/1000
2.85
refer1000
2-fe-2000
refer5000
refer19000
8.75
0.70
Te/eTDD 10fer100
1ele11000 TWIN2500 referS000 1efer7500 TWIeT10300
Source Games
Figure 4: Generalisation performance of language from different source games all target games. Each line represents a specific kind of target game. The z-axis indicates the type of source games, and the y-axis is the accuracy on target games. To keep the figure readable, and instead provide the corresponding SD values in Table We limit the range of y-axis to [0.7, 1.0] emphasise the difference between languages on complex games, refer to Figure 8 for more values.
remove the error bars,
As shown in Figure on whichever type of game, the peaks of all curves are always among 1000", "refer2500", or "refer5000", which means that the best expressivity always emerges in these 3 types of games. Considering that the complexity of context in referential game is monotonically increasing function of the [C] while the contextual unpredictability is a monotonically X
decreasing of it, the results shown Figure support our hypothesis, i.e. the expressivity is indeed a trade-off between the contextual complexity and unpredictability.
To be specific, although context "refer2", "refer10", and "refer100" have high unpredictability, their complexity is low thus the languages have lower expressivity. On the other hand, context of "refer7500" and "refer 10000" have high complexity, but expressivity of neither language is the highest due to the high predictability of the context in these games. This supports our argument that unpredictability of context incentivises the agents to encode more information into the messages such that the messages could overcome the uncertainty of contexts and be interpreted in various contexts. Therefore, the three languages ('refer 100'r, 'refer2500', and 'refer5000") from context with both moderate complexity and unpredictability perform better than the others. To make it more straightforward to compare the expressivity between these languages, we also provide another view of the above results in Figure 9 in Appendix E.3 which shows that 'refer1000" "refer2000' x 'refer5000' > 'refer7500' ~ 'referT0000" > 'refer100' 'refer10' > 'refer2'.
To eliminate the possibility that the above phenomena caused by the hyperparameters used in our games, we also run the language transfer experiment with varying communication channel capacity and agent capacity (i.e. the size of hidden layers). As shown in the figures in Appendix F both 'refer 000' and 'refer2500" perform better than 'refer 10000' across varying hyperparameters, which shows that the trade-off phenomenon consistent over game settings and agent capacity.
5.4 THE COLLAPSE OF MESSAGE TYPES
As contrastive loss instead of the conventional referential loss applied by (c.g. Havrylov Titov| 2017 Ren et al. 2020), we also compare the behaviours of both loss functions on 9; 10" "00fer100"). We discover that the number of message types is some times greatly less than the size of input space. We refer such phenomenon as "message type collapse", and the mappings from several different inputs to same message type "degeneracy components".
9
"refer ["refer2"
While comparing the two loss functions, we observe that contrastive loss is more efficient on removing degenerate components in large-scale games (where |C| > 100), and can better avoid the collapse of
8

---

Published as a conference paper at ICLR 2022
the number of message types. To be specific, shown in Figure 5a the numbers of message types collapses to less than 500 on games with conventional referential loss. They, however, would only decrease roughly 5, 000 on 'refer10' and 'refer1000" with contrastive loss as shown in Figure [5b] The contrastive loss can also make the training of referential game more stable, as illustrated by the smaller variance in Figure 5b
Our results indicate that it is necessary to establish a referential game with candidate sets containing enough many distractors guarantee that the emergent language will be relatively unambiguous. As far as we know, the existing works do not investigate the effects of the size of candidate sets, e.g. Lee al. (2018) set [C| 2. Li & Bowling (2019) set [C] = 5, and |C| < 20 is used by Lazaridou et al. (2018). Given our results, our heuristic is that C*l should be greater than either 1, 000 for large ipput spaces or 1 |X] for smaller ones.
We also explore more about the degeneracy degree (as well structure degree) of the mappings constituting different languages, and the results and discussion are provided in Appendix C
10030
13030
Scarce
refer2
B333
reffer 103
9020
biszgga
Number of Mhassages
$030
8030
mirrr
reler 50
4230
660
200
120
200
400
300
Spects
Epects
(a) With conventional referential loss
(b) With contrastive loss
Figure 5: How number of message types change over training epochs with different loss functions. The lines are the means over 6 runs and shadow areas indicate the corresponding variance.
6 CONCLUSION & FUTURE WORK
To our knowledge, this is the first work exploring the expressivity of emergent languages from language games played by DL-based agents that have different context size. We first proposed a hypothesis about the factors influencing the expressivity of emergent languages, and defined partial ordering between expressivity of different emergent languages based on the generalisation performance across set of games. Through series of experiments on referential games with varying size of context, we then validated the three predictions of our hypothesis, which furthers indicates that expressivity is indeed a trade-off between the complexity and unpredictability of context in language games. overcome the memory inefficiency issue caused by the conventional referential loss function, introduced contrastive loss into implementation. In the comparison between this contrastive loss and the referential loss, we found the message type collapse problem, and also showed that using this contrastive loss alleviate it.
Since this is a first step towards defining and understanding the expressivity of emergent languages, there still some questions unanswered by this work. For example, some properties of our emergent languages are surprising given the simplicity of some of our referential games. In the simplest setting (CC) 2), we see emergent languages with more than 2, 000 message types, whereas an optimal strategy requires language consists of only 40 unique messages (since there are only 40 different values for all properties in X' and two items in the context always differ on least one property). Thus, it shows that further investigation about how redundancy helps during the communication between agents is necessary. More importantly, we lack understanding the structure of mappings constituting a language, especially when the languages are generated or encoded by neural networks. We believe that the expressivity of languages still needs further exploration in order to facilitate languages that are universally effective across various tasks.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This project has also received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme under grant agreement No. 681942, held by K. S.
REFERENCES
Jacob Andreas. Measuring compositionality in representation learning. In International Confer- ence on Learning Representations, 2019. URL https:://ppnreeveew..ees hwtp://wwwwe....oe..eeeeeeeeeeee www.eeeoommmiee
HJz05o0qk7
Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life, 12(2):229-26) 2006.
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Association for Computational Linguistics. doi: 10. 10611661112220.000 acl- m07. URL https
Online, July 2020.
7/aclanthologye org. acl aww-maainn.......... maiin.477
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 3700-3710, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D191111 1384.
org/D19-1384
Jean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural Information Processing Systems, 33:21271 228222242, 2020.
Shangmin Guo, Yi Ren, Serhii Havrylov, Stella Frank, Ivan Titov, and Kenny Smith. The emergence of compositional languages for numeric concepts through iterated learning in neural agents. The 13th International Conference on the Evolution of Language, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEFE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '66), volume 2. pp. 1735-1742. IEEE, 2006.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in Neural Information Processing Systems, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2020. URL [tL[ittttp////eeeeevee net/
Ivana Kajić, Eser Aygün, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. arXiv preprint arXiv:2004001097 2020.
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: toolkit for research on Emergence lanGuage in Games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference
10

---

Published as a conference paper at ICLR 2022
Natural Language Processing (EMNLP-IJCNLP)) System Demonstrations. Association for Computational Linguistics, 2019, doi: 10.186653/1/D119-300 CRLLhttss:// wttp://ww....ww..............
sttp/anthhooooy/1133300
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimization in emergent languages. In International Conference on Machine Learning, pp. 5220-5230. PMLR,
2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015. URL httphst/oopeeeeiieeet htwp:/rwwwwwww.eeeoeoeeeeeeeeeee forum?
(d=8gmWw]FyLL
Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learning model of the emer- gence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, 5(2): 102-110, 2001.
Simon Kirby and James Hurford. The emergence of linguistic structure: overview of the iterated learning model. In Simulating the evolution language, pp. 121-147. Springer, 2002.
Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution of language.
Current opinion in neurobiology, 28:108-114 2014.
Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. Compression and communication in the cultural evolution of linguistic structure. Cognition, 141:87-102, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference on Learning Representations, 2017.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Confer- ence on Learning Representations, 2018. URL httpss////////..t.. /tR.ittss:///ppeeee ew. net. fo Forum?? i
HJGv1Z-AM
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 2015.
Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. Emergent translation in multi-agent communication. In International Conference on Learning Representations, 2018.
David Lewis. Convention: philosophical study. John Wiley & Sons, 1969.
Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent com- munication. In H. Wallach, H. Larrocelle,, A. Beygelzimer, F. l'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran As- sociates. Inc., 2019, URL BRLL/tttss///proeeeettt hwwp://rw....ciiio..e.......o.. hwtp:esenniriiccresesse http:///.....///aa/////////////. 0e1.884779999bbb28 9b2 225.
pdf
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL
mww.rrpps.cco/ppppr///////////.. Paper.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-1359 2009.
Yi Ren, Shangmin Guo, Matthieu Labeau. Shay B. Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In International Conference on Learning
Representations, 2020. http::pppp...eeeeeeee.eeeeeeeeee
11

---

Published as a conference paper at ICLR 2022
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844455 2016.
James Winters, Simon Kirby, and Kenny Smith. Contextual predictability shapes signal autonomy. Cognition, 176:15-30, 2018.
Ludwig Wittgenstein. Philosophical investigations. John Wiley & Sons, 1954.
Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chun Zhu. Emergence of pragmatics from referential game between theory mind agents. arXiv preprint arXiv:2001.07752, 2020.
Sai Qian Zhang, Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2022
A MORE DETAILS ABOUT COMPARISON BETWEEN DIFFERENT LOSS FUNCTIONS
In this section, we give more details about the comparison between the referential loss used by previous works and the contrastive loss we use Section 2.2
Following the same notations used in Section 2.2 the referential loss used by (e.g. Havrylov & Titov 2017) can written as:
p(c] m)x exp E(h m: foonl( exp
(hm -eenno
(4)
where h is the embedding for the message obtained from the sequence encoder of listener L, and (.) is the candidate encoding function of L. From the above equation, we can see that if a candidate has an embedding closer message m, the probability mass for predicting would then be higher. Hence, the predicated probability mass is inversely proportional to the distances, e.g. the dot product in the above equation, between the embeddings of m and all candidates.
The contrastive loss function proposed by Chen et al. (2020) with 7 = 1 is given as follow:
exp (sim Z,, ..y))
L Σ. [[zz exp (sim( (2,, (x,))
(5)
where IT is indicator function. By comparing the above loss function with Equation 4 find
that the aim of them is the same, i.c. make the distance between positive pairs (targets and their corresponding messages) closer while keeping the embeddings for negative pairs (distractors and the messages) as far apart as possible. While negative sampling has been deeply investigated in the sele-vupe learning (SSL) community, (e.g. Chen al. 2020 Grill et al. 2020). has been discussed in the emergent communication community yet. thus we simply use all randomly sampled distractors as negative examples. As mentioned in Section 2.2 we need to make the whole input space candidate set, thus the above contrastive loss which has lower space complexity is preferred.
In the following, we give a bit further derivation to show that Equation ! iis computationally equivalent see that:
to Equation 5 By substituting k in the denominator of Equation 5 with
exp (sim( x,, æ.))
l = - log Σ, [[ exp (sim atx))
Σ, [im exp (sim (z;, .x))
log
exp (sim (2, 2zy))
Σ, [[i= exp (sim (Z,, 27))
exp (sim xz,) exp (sim( az,)
log
log
exp (sim(2; a2, ..y))
(6)
Σg exp (sim (z,, 2;,))
log
exp (sim( 3., 2z,)
exp (sim( (2, 2z))
log
Σ, exp (sim (x,, x,,)
B MORE DETAILS ABOUT CONTEXTUAL COMPLEXIYY/UNPEEDCCAABUTTT
In this section, we give more details about the definition context and its complexityllnpredicabilttt
B. CONTEXT
As defined in Section 3.1 the context for target object is the space of samples involved in the calculation of its loss. We know that the prediction of the listener is calculated based on the distance between the message received (m((:)) which is a function of the target input (æx), i.e. softmax (d1 (m(x:). æx)) where xy is a distractor and d2 is a distance function for embeddings. Cross entropy loss is then calculated based on softmax function. Thus, function as [(m( x.), { } 0). As shown in the equation, the calculation is based on samples from candidate set C. Therefore, in referential game, C(x.) (æt) (ma) C = X = O.
also write the loss
13

---

Published as a conference paper at ICLR 2022
B.2 COMPLEXITY OF CONTEXT
As defined in Section 3.1 the k-th degree neighbour set is (ax):)) {xr d(ar,, mz & k) where k E Z+. As described in Section 2.1 there are 4 features in our synthetic data set and each of them can take 10 different values. Therefore, for specific target æt, the k in the definition of neighbour set can take only the following 4 values:
k = 1: samples in (xr) differ only in 1 feature from Tt. For the only different feature, there are only 9 possible values distractor can take. Thus, [W] 4 9 = 36.
k = 2: samples from (zx) and samples that differ at exactly features from Tt. For the later case, there are first (t) = 12 possible combinations of such two features, and there are 9 9 81 possible samples for each combination. Thus, [N2(ax:) 12 x 81 36 1008.
h = 3: samples from (2,) and samples that differ at exactly 3 features from 24. For the later case, there are first (t) = possible combinations of such three features, and there are
729 possible samples for each combination. Thus, (aa(z)) 4 x 729 + 1008 3924.
k = 4: all samples in X' except æ; are in this neighbour set, thus [Na(2z)) 10000 1 9999
Since we use independent Bernuulii sampling, for each draw, the probability to get sample from No (zx) is simply TXI-I where |X] 1 is due to excluding æ, from X. Meanwhile, since we define
(ax)
C(axt) as the candidate set in referential games. Therefore, for [C(x))] = |C| draws, the probability
of having at one sample from No. (3x) is:
(i( [
IC1
| [Wi(()))
ICI
= =1
X -1
B.3 UNPREDICTABILITY OF CONTEXT
In this section, first derive measure of unpredictability following Winters al. (2018). In their experiments, the imput space consists of 16 referents. There are 2 different attributes, colour and shape, and each attribute has 4 different values. By setting the size of candidate set to 4, they define the following two types of context:
shape-different: all the 3 different distractors have identical colour to the target, but different shapes. Since there are only shapes and candidate set size is also 4, in this setting, the distractors for a given target will never change during the games. This is exactly the same as our "fixed-batch" games defined in Appendix E.
mixed: all the 3 different distractors are sampled uniformly at random from the input space, thus they may have different shapes as well as different colours to the target. this case, the distractors for given target vary across different trials during the games.
By comparing the two above cases, it is straightforward to see that it is more possible to see different distractors for a given target in the games with mixed context. Following this heuristic, we proposed measure the unpredictability of context by the expected chance of observing different distractors for targets across training epochs, which further leads us to the definition given in Equation 3 Note that there is also complexity (under our definition) involved in the above two types of context, we focus only on the variability part of the context in our definition of unpredictability since the complexity part has been taken into consideration by the neighbourhood set defined in Section 3.1
We then give further explanation about Equation 3 and its simplified form. Since we use Bernoulli sampling for each individual object in candidate set, the overall sampling of candidate set is thus binomial procedure. As the probability of getting a sample from (xx) in X' is [C"(,,)] the overall
|
proportion of objects from C (2x) in C*+1 (xx) is the same, i.c.
[0"(2,) Therefore, the probability of
|
the complementary event, i.c. getting object not from (*xx) ) in C (xx), is [c"(w.))
1x
14

---

Published as a conference paper at ICLR 2022
C MORE RESULTS AND DISCUSSION ON EXPRESSIVITY
In this section, we provide two more intermediate properties of emergent languages that are also influenced by complexity and unpredictability of context, and thua may influence the expressivity.
C.I DEGREE OF DEGENERACY AFFECTS BUT DOES NOT DETERMINE THE EXPRESSIVITY
Inspired by Kirby al. 2015). the first intermediate factor we explore is the degree of degeneracy (i.e. the degree of ambiguity) of the languages. As language is defined as a mapping function L : X' M, the degree of its degeneracy could be measured by the number of distinct message types.
As shown in the [ in referential languages, the degeneracy degree decreases rapidly with the increase of the contextual complexity of the source games, and reaches a ceiling around [D] > 1, 000. There is a positive correlation between the number of message types and expressivity for the languages with low complexity, which indicates that degree of degeneracy could be an intermediate factor that correlated with the expressivity of these languages. However, the fact that the average number of message types on refer10000 is higher than refer1000, while the expressivity of 'refer1000' is higher than 'refer I 0000', i.e. emergent languages having lower level of degeneracy does not necessarily have higher expressivity, indicates that degeneracy degree of one language cannot determine its expressivity.
Game
refer2
refer10
refer
refer 100o
refer2500
refer5000
refer7500
refer
2872.18
8565.69
9547.58
9801.60
9839.33
9841,27
9840,34
9856,96
25th percentile
2629.0
8325.0
9501.5
9778.75
9825.0
9837.5
9819.0
9845.0
75th Fiil-ppeeentiee
3195.0
8795.5
9639.25
9823.25
9X54.25
9864.25
9866.0
9871.
Table 1: Mean, 25th beth-percentile, and 75th-percentile of the distributions of message types numbers from different source game.
C.2 STRUCTURE DEGREE OF MAPPINGS OF DIFFERENT LANGUAGES
Since the degenerate degree is not the only factor determining expressivity, another factor to be considered is how the mappings of a language L are structured after the training procedure of agents on source games. According to the cardinality of the mappings of L, we can divide them into two disjoint sets: i) one-to-one mappings, i.e. an input is mapped to a unique message; and ii) many-to-one mappings (a.k.a degenerate components), i.e. several inputs are mapped the same message type. Between this two types, the structure of one-to-one mappings has been explored under the view of compositionality in the previous works, e.g. (Brighton Kirby| 2006 Andreas 2019). However, as illustrated by Chaabouni et al. (2020), for emergent languages from DL. agents, good generalisability does not require a good compositionality, which means that measurements for compositionality are not good predictors for expressivity either. In this work, we focus only the many-to-one mappings.
Given the objectives of games, for the degenerate components in an emergent language, message being mapped to several closely-related inputs could imply that captures common features among these inputs. Suppose that several inputs Xx {=1...., .x are mapped the same message m in a converged language L. we hypothesise that the average distances between every pair of inputs, i.e. 1 _i.je(,,.... d(x;, 2, would become smaller after training, compared with their X
random initial values. Therefore, we compare the averaged distances between meanings of degenerate components as well as the frequency of degenerate components at the beginning and end of training. To make the plots more readable, we only plot the statisties about the top-10 most frequent ambiguous message types, where the frequency of a message type is defined as the number of inputs that are mapped to it in a languages. Results on 'refer1000" (one the most expressive language in our experiments) are shown in Figure 6
By comparing the data of beginning and end of training in Figure 6b averaged distance decreased throughout the training, which indicates that the degenerate components actually became more although not fully structured. Another observation we have limited under- standing of is that 'refer1 1000 becomes more ambiguous after training (shown by that the red bars
we can easily observe that the
15

---

Published as a conference paper at ICLR 2022
bndimin
bag m0
03
00
(a) Frequencies degeneracy components
(b) Distance between corresponding inputs
Figure 6: Statistics about frequency of Top-10 most frequent message type and the average Hamming distances between their meanings at the beginning and end training on 'refer1000". The bars indicate the frequency and the Hamming distance, and the black lines indicate the corresponding SD.
representing the statistics the end of training are lower than the bars representing the statistics the beginning of training), although its accuracy does improve to almost 100%. However, limited by the scope of this work, will leave further exploration of future works.
D MUTUAL INFORMATION IS INSUFFICIENT
In Section we mentioned that MI cannot completely reflect the expressivity of an emergent language. Here, we provide more detailed explanations. On one hand, a higher MI value does not necessarily mean more task-relevant information, e.g. encoding colour in messages is unnecessary for tasks involving only the shapes inputs although it provides more information. On the other hand, not only the determinacy of the mappings matters for generalising them, but also the structures, e.g. it would be easier for humans to extrapolate compositional languages to novel concepts than holistic ones although they are both unambiguous and therefore have the same MI. In the following paragraphs, we will give more reasoning and empirical evidence to support our claim.
Following the mapping function definition of emergent languages illustrated in Section 2.1 suppose that we get a message type collection M c .M in some training epoch, the calculation of the mutual information between M and X' is given follows:
I(M
p(x, m)
Σ
Σ
p(x, m)) m) log
p(x) m
De mEM
p(æ, m2)
Σ
log p(z,
X
p(x)p((m) m)
DEX
ME M'
p(æ, m))
Σ
log
X
p(x))p(m)
xEX
MEM
1
Σ
Σ
(logp(æ, m) (og)(()) - log p(m)
X me
p(#) क
Σ
log
log
log p( (m)
X'
X
XEX MEM
Σ
Σ
(- log p(m))
TEY mE
Σ logp(m)
meM
16

---

Published as a conference paper at ICLR 2022
[(m)
Σ log
[M
Σ (log [[(m]] log (M
mEM
Σ (log |M| - log [m(m)
where p(x) = [
as a follows uniform distribution over the input space, p(z, m) = since every is mapped to message, and [mm) is the appearance number of m in one language.
90000
80000
Mutual Information
70000
Source Games
refer10
refer100
60000
refer1000
refer2500
refer5000
50000
refer7500
refer10000
500
Epochs
Figure 7: Mutual information curves over training epochs in different types of games. The converged values are provided Table 2
By observing the above equation, it straightforward to see that the MI value is decided by the number of message types, i.e. the degeneracy degree one language. Therefore, if two emergent languages have approximately the same number of messages types (as shown in Table and degeneracy degree, the MI values for these two languages should also be the same. However, as we illustrate in Section C that degree of degeneracy cannot determine the expressivity of emergent languages, can then deduce that MI cannot fully measure the expressivity emergent languages.
To provide more empirical evidence to support our claim, we tracked the MI values over training epochs on a subset of G., and the results are shown in the following Figure 7 As we can see, there is no statistically significant difference between the MI values of 'referX' (where X> 1, 0000 while it has been shown in Section 5.3 that their expressivity is different.
Game
refer10
refer100
refer1000
refer2500
refer5000
refer7500
refer10000
Mcan
78269.22
86823.66
89823.16 16
90447.69
90504.40
90378.89
90533.06
SD
3760.05
1464,69
443.12
213.76
412.37
306.85
139.42
Table 2: Converged MI from different source games. Both mean and standard deviation are given.
E LANGUAGE TRANSFER EXPERIMENT RESULTS
E. NAMING OF GAMES AND LANGUAGES
In this section, we give more details about the naming of games and corresponding languages we used in Section 5 as follows:
17

---

Published as a conference paper at ICLR 2022
"referX": this refers to the referential games with candidate sets whose size is X (which itself is an integer), note that we use double quotation marks to indicate that this is a specific type of game. For example, "refer2500" refers refers to referential games with 2, 500 candidates
for listeners.
'referX': : this refers emergent languages from "referX",, note that we use apostrophes here in order to differentiate it from the notations of the corresponding game. For example, "refer2500" refers to emergent languages from referential games with 2, 500 candidates for listeners.
"referXf": this refers to the referential games with candidate sets whose size is X, but the partition of batches is identical over epochs. thus giving the game fixed context. For example, "refer2500f" refers to referential games with 2, 500 candidates for listeners, but the batches of data are kept identical through the whole training procedure.
'referXf": this refers to the emergent languages from "referXf". For example, 'refer2500f" refers to emergent languages from referential games with 2, 500 candidates for listeners with fixed context. Regarding the meaning of tixed-batch" or "fixed context", example below where the whole input space is {a, b, c, d} and batch size is 2:
give an
varying batch/context: {a, b), {c, d) the partition for the 1st epoch; then {a, c), (b, d] is used for the 2nd epoch; and random partitions are used in the following epochs; whichever epoch it is, {a, b). {c, d) is always used as the partition
fixed batch/context:
of context;
"referX/f": this equals "referX" and "referXf", both of them have same complexity but different unpredictability. To be more specific, contextual unpredictability of "referX" is higher than "referXf".
Gy: the universal set of source games which is defined ("refer2", "refer10", "refer100", "refer1000" "refer2500", "refer5000", "refer7500", "refer10000", "refer2f", "refer10f", "refer100f", "refer1 "refer2500f", "refer5000f", "refer7500f",
Gr: the universal set of target games which is defined as ("refer2", "refer10", "refer100", "r0fer1000"; "refer2500", "refer5000", "refer7500", "refer100000).
E.2 RAW RESULTS OF COMPLEXITY EXPERIMENT
In this section, we provide the raw data of Figure in the following Table 3 Not that although 100ff performs the best on "refer10" than all other languages, we argue this outlier doesn't influence conclusion of our prediction considering the simplicity of "refer10".
Source
Statistics
refer2
refer10
refer100
refer1000
refer2500
refer5000
refer7500
refer100000
refer2f
mean
0.9596
0.7423
0.2244
0.0322
0.0168
0.0163
0.0156
0.0163
0.0236
0.1493
0.1308
0.0203
0.0103
0.0105
0.0111
0.0119
refer
mean
0.9979
0.9823
0.8175
0.3259
0.2028
0.1918
0.1909
or
0.0007
0.0044
0.0404
0.0518
0.0461
0.0446
0.0500
0.0418
refer100r
moan
0.9983
0.9958
0.9655
0.7428
0,5988
0.3933
0.5946
0,5871
0.0010
0.0029
0.0106
0.0573
0.0738
0.0685
0,0719
0.0663
refer1000r
0.9989
0.9977
0.9881
0.8794
0.7849
0.7793
0.7565
0.7686
0.0008
0.0006
0.0032
0.0163
0.0250
0.0242
0.0664
0.0237
refer2500r
mean
0.9992
0.9987
0,9892
0.8871
0.8044
0.7960
0.7788
0,7880
0.0008
0.0005
0.0032
0.0199
0.0259
0.0239
0.0267
0.0264
refer5000r
0.9988
0.9980
0.9897
0.8916
0.8094
0.798D
0.7941
0.7928
0.0007
0.0023
0.0099
0.0143
0.0127
100e
mean
0.9993
0.9980
0.9897
0.8864
0.8095
0.7991
0.7943
0.7950
0.0005
0.0007
0.0021
0.0084
0.0142
0.0095
0.0123
0.0107
Table 3: Mean and standard deviation a of generalisation performance of languages from fixed-context on various target games.
18

---

Published as a conference paper at ICLR 2022
E.3 MORE RESULTS OF TRADE-OFF EXPERIMENT
In this section, we first provide the complete version of Figure i.e. Figure Since the performance of 'refer2', 'refer10', and 'refer100" are too bad on complex target games, is very clear that their expressivity are much lower that the rest. Therefore, to emphasise the difference between the other languages, we limit the range y-axis [0.7, 1.0] Figure
1.0
0.8
Target Games
refer2
refer10
Accuracy
refer100 00
refer1000
refer2500
0.4
refer5000
refer7500
refer10000
0.2
refer2
refer10
refer100
refer1000
refer2500
refer5000
refer7500 refer10000
Source Games
Figure 8: Same to Figure but with wider range for y-axis.
We also illustrate the partial order between the expressivity different languages in Figure 9 very clear that the 'refer2', 'refer10', and refer100 performs worse than the others referential games with more candidates than their sources. Combining this fact with the fact that 'refer7500' and 1000) perform worse than 'refer1000", 'refer2500", and 'refer5000" shows that increasing the size of candidate set |C] would not always benefit the expressivity. As for 'refer1000", 'refer2500', and 'refer5000", there is statistically significant difference between their performance the simpler referential games, and they intersect on some games, thus their expressivity are either incomparable approximately the same.
It
At last, we give the precise values of both Figure 9 and in the following Table 4 All the results are obtained by multiple runs, thus both mean and standard deviation (a) are given.
To better support our claim, we also provide p-values of testing the statistical significance between the generalisation performance of two different source games across all types of target games in the following Table 5
F VARYING THE CAPACITY OF CHANNEL AND NETWORK
To verify that our findings robust to the capacity of the communication channel and agents, also run some extra experiments on the following different configurations of referential games:
larger channel capacity: set the length messages to and size of token inventory to 20, thus the size of message space becomes 20
2. larger agent capacity: set the hidden layer size to 512, thus the capacity of both speaker and listener become larger than the original setting (256);
3. larger channel&agent capacity: do the above two changes at the same time.
Since our key findings are about the generalisation performance between refer1000, re- fer2500, and refer10000, ('refer1000* 'refer2500', 'refer10000") with multiple runs. The results configuration 1, 2, and given in Figure 10 11 and respectively. It can be observed in all figures that 'refer1000"
just these extra experiments
game set Gentro
19

---

Published as a conference paper at ICLR 2022
1.00
0.90
Saurce Games
refer2
refer10
Accuracy
0.85
refer100
refer1000
0.80
refer2500
refer5000
refer7500
0.75
refer10000
0.70
0.65
refer2
refer10
refer100
refer1000 Target Games
refer2500 refer5000 refer7500
source game different target Figure 9: Generalisation performance of language from the
games, where each line represents source game. Since the plot would become not readable if we plot error bars here, provide SD in Table The x-axis indicates the type of target games, and the y-axis is the accuracy on target games. If a line lies above the other over all the target games, means that the corresponding source language has higher expressivity than the other. On the other hand, if two lines intersect, e.g. 'refer 1000' and 'refer2500" above, then the expressivity of these languages is incomparable.
Source
Statisties
refor2
refer10
refer100
refer1000
refer2500
refer5000
refer7500
refer10000
refer2
mean
0.9936
0.9543
0.6043
0.1203
0.0668
0.0616
0.0592
0.0565
0.0014
0.0077
0.0394
0.0146
0.0105
0.0078
0.0090
0.0047
0.9980
0.9909
0.9076
0.4896
0.3434
0.3318
0.3282
0.3188
refer10
0.0007
refer100
mean
0.9986
0.9978
0.9812
0.8298
0.7162
0.7121
0.7078
0.7001
0.0007
0.0009
0.0035
0.0171
0.0299
0.0312
0.0293
0,0350
refer1000
mean
0.9989
0.9979
0.9881
0.9003
0.8256
0.8139
0,8153
0.8033
0.0007
0.0009
0.0015
0.0143
0.0151
0.0165
0.0226
0.0183
refer2500
moan
0.9988
0.9981
0.9892
0.9036
0.8235
0.8178
0.8160
0,8059
0.0006
0.0009
0.0012
0.0121
0.0099
0.0158
0.0148
0.0197
refer5000
moan
0.9990
0.9975
0,9890
0.8991
0,8212
0.8141
0,8125
0.8013
0.0004
0.0004
0.0022
0.0050
0.0140
0.0095
0.0177
0.0140
refer7500
meam
0.7963
0.7895
0.7906
0.0005
0.0004
0.0034
0.0156
0.0206
0.0185
0.0223
0,0215
100er10000
mean
0.9988
0.9980
0.9887
0.8916
0.8044
0.7930
0.7940
0.7828
0.0007
0.0005
0.0023
0.0077
0.0099
0.0143
0.0127
0.0122
Table 4: Mean and standard deviation of generalisation performance of language games each other.
'refer2500' always perform better than 'refer10000' on all kinds of target games, i.e. 'refer 100' or 'refe25000' both have higher expressivity than 'refer10000" Therefore, we believe the key observation from Figure Sngure@hole holld across configurations of communication channel and capacity of agents, which means that our conclusions robust the configuration of language games.
20

---

Published as a conference paper at ICLR 2022
Sources
reler2
referl0
reler100
reler1000
reter2500
reler5000
refer7500
reler10000
reler2 vs refer10
2.73e-06
5.27e-MM
163e-06
1.27e-04
5.166-414
5,92e-04
4.51e444
5.56e-04
refer2 refor100
7,44c-06
3,82e-04
6.70c-06
1.57c-13
3.78c-09
2.12c-08
6.69c-07
1,00c007
refer2 vs refer (000
4.62e-06
3.47e-04
6.21e-06
5.75c-16
7.36e-14
2.71c-12
2.51e-10
4.47c-10
refer2 refer2500
6.46e-06
3,50c-04
6.04e-06
7.39c-15
1.08c-14
5.08c-10
1.0e-100
5.59c-08
refer2 refer5000
2.40c-06
3,42e-04
6.1le
1.13c-13
5.47e-18
3.57c-15
1,43c-18
2.62e-11
reler2 vs reler7500
1.53e-06
3.
6.14e-06
6.99c-16
7,59c-12
9.73e-11
3.19e-10
1.98e-09
relor2 vs 1000
7.69e-06
3.42e-04
6.11e-06
2.45c-11
5.66e-17
2.63e-14
2.72e-14
4.85c-12
refer10 100
1.71.-03
2.24c-04
5.99c-04
1.58c-04
4,44c-05
3.71e-05
164c405
1.83-05
refer10 vs 1000
3.336-04
7.23c-05
3.90c-04
8.50c-05
2.47e-05
2.59g-05
1.33e-05
1,51e005
refer10 vs refer2.00
3.23e.04
9.75c-05
0.54c-04
8.36--05
3.14.45
2.29c-05
2.04-005
1.15c-05
1e/er10 vs 5000
1666-044
4.92e-05
3.65-004
9.31e-05
2186-415
3.16e-05
2.65e415
2.03e-05
refer10 vs re0er7500
181c-04
9.8e-005
3.70c-04
9,65c-05
2.70c-05
2.60e-05
1.77e405
155e-00
reler10 vs 10000
7.58c-04
5.61e-05
3.64c-04
1.09c-04
4,14c-05
3.76e-05
3.01e-05
2.73c-05
refor100 vs refor 1000
2.88c-02
7.18-404
100c-04
1.37c-04
1.10--04
2.77c-04
0.360-03
4.50c-04
refer100 vs refer2500
1.70e-03
4.17e-04
1.78--05
1.08c-04
1.92e-04
1.89-004
1,49c433
3.87e-04
reler100 vs r0fer5000
0.
1.85c-33
2.43e-05
2.12e-04
1.70.004
3.78c-04
1.99.403
6.50k-04
10fer100 vs rofo
4.87e-02
1.39c-03
4.27c-05
4.11c-04
8,38c-04
1.09-003
039
1.97c-03
1eTer100 va 10000
5.08c-02
5.88c-04
2.51e-05
7.31c-04
7.31c-04
1.21e-03
4.03c-43
2.14c-03
refer1000 vs refer2500
4936-01
2.70g-01
2.78c-01
6.41c-01
9.84c-01
5,97c-01
9.076-01
8.31c-01
100e vs refer5000
1.85 01
9.86c
1.14e 01
7.32e 01
7.58e 01
9.28e 01
8.53e.01
8.39e 01
1e0er1000 vs reler7500
0198--0
0.87e-01
3.86e-01
1.90e-01
4174
9.23e-02
1.13e401
1.12e-01
1e0er1000 10000
01
3.33e 01
1.15e.01
1.79 01
4.96e-012
6,98c 02
1.08e 01
7.06e 02
reJer2500 vs reter5000
5.03e-02
2.92c-01
9.26e-01
8.44c-01
7,35c-01
6.04e-01
6.95c-01
6.82c-01
refer2500 vs refer7500
1556001
3.66e-01
7.53c-01
7.06c-02
2.85-402
4.47c-02
6.60e-02
1.04-001
refer2500 vs refer 10000
I 1550
9.86x
7.56e 01
5.12e 02
199c-412
3.31e-02
4.02e402
8.05e 02
refer5000 vs refer7500
5.2
2.87c-01
6.05e-01
6.59e-02
4.95c-02
5.91e-02
8.63e-02
1.19e-01
1eler5000 vs 10000
6.08e 01
3.53e
7.58c 01
2.25e-02
0.91e-02
2.73e-02
3.71e-02
5.61e-02
refer7500 reter10000
9.23--11
5.01c-02
4.89c-01
7.82c-01
4.31c-01
8.47c-01
7.67c401
8.96-01
Table 5: p-values of testing the statistical significance between two different types of source languages across various target games. We take 0.05 as the threshold for p-values, and bold all the significant results. For the comparison between { 'rrefer000'', 'refer2500', 'refer5000') and ('refer7500', 'refer10000", ). we also make the significant results be in red colour.
refer1000
0.96
refer2500
refer10000
0.94
0.92
Performance
0.90
0.88
0.86
0.84
0.82
0.80
refer1000
refer2500
refer10000
Target Games
Figure 10: Results of language transfer with larger channel capacity. Error bars indicate the standard deviation of the mean values.
21

---

Published as a conference paper at ICLR 2022
0.94
refer2500 refer10000
0.90
Performance
0.88
0.86
0.84
0.80
refer1000
refer2500
refer1 DD00
Target
Figure 11: Results of language transfer with larger agent capacity. Error bars indicate the standard deviation of the mean values.
1.00
refer1000
refer2500
refer10000
0.96
0.94
Performance
0.90
0.88
0.86
0.84
refer1000
refer2500
refer10000
Target Games
Figure 12: Results of language transfer with larger channel and agent capacity. Error bars indicate the standard deviation of the mean values.
22


Reviews:
[{"review_id": "vzr_Rz5DiYW", "reviewer": "Reviewer_jyzs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strength\n- This paper provides novel and interesting insights to the language emergence analysis by introducing the concepts of expressivity, context complexity and unpredictability. These concepts provide a deeper understanding of the behaviors of emergent languages from (referential) language games.\n\n- The hypotheses are clearly stated, reasonable, and well supported by the experiment results.\n\n- Besides the main results, the paper also proposes a simple update on training loss, contrastive loss, which improved the model behavior.\n\n\n#### Weakness\n- The introduced concepts (context complexity and unpredictability) are finally simplified to be only dependent on the candidate set size $|C|$. Also, experiments and analysis design are focused on $|C|$. The scope might be narrow (but also specific, so not necessarily a weakness).\n\n- Although the hypotheses are clearly stated and supported, they are not very surprising based on intuition.\n\n- Regarding the proposed training loss, to my understanding, it simply replaces the original candidate set with the batch. In this way, the contribution does not look significant.\n\n\n#### Questions & Suggestions\n- The definition of key concepts, context complexity and unpredictability, do not seem fully accurate. In text, the context complexity is defined as the expectation of the probability that \"$C(x_t)$ includes an object from $N_k(x_t)$\". However, equation 2 means the expectation of the probability that \"a random object from $C(x_t)$ is in $N_k(x_t)$\". Nonetheless, the simplified form $1-(\\frac{|X|-|N_k(x_t)}{|X|})^{|C|}$ is according with the text definition. Maybe equation 2 needs to be modified. For unpredictability, the text definition is the expectation of \"the probability that $C^{e+1}(x_t)$ contains an object that is not from $C^e(x_t)$\", which seems wrong because this is clearly not monotonically decreasing on |C|. Based on the simplified form, I guess it should be the expectation of \"the proportion of $C^{e+1}(x_t)$ that are not from $C^e(x_t)$\".", "summary_of_the_paper": "This paper studies the \"expressivity\" of emergent language in language games. To my understanding, \"expressivity\" is empirically the transferring ability of the language to unseen data. In this paper, the authors propose two factors of the underlying language game that can affect the expressivity of emergent language: context complexity and unpredicability. In referential games, the context is the distractor candidates for a sample. Complexity denotes how likely a \"close\" distractor will be included in the candidates. Unpredictability is how likely the context for a single sample will be different among epochs. This paper proposed hypotheses that both context complexity and unpredicability can improve language expressivity, but these two factors are \"contradictory\" and we need to have a trade-off between them. The hypotheses are supported by empirical experiment results.", "summary_of_the_review": "This paper studies \"expressivity\" of emergent language, which is empirically the transferring ability of the language. The paper also defines \"context complexity\" and \"unpredictability\" of the underlying language game. The study shows that these two factors both contribute to the language expressivity, but they are contradictory on the candidate set size $|C|$ so a trade-off is needed. Generally, the analysis is novel and interesting. Although I have some concerns about the scope and significancy, the work still provides useful insights for language emergence study. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "u2MpAIfYFU", "reviewer": "Reviewer_MdRY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the overall idea of the paper on exploring the relationships between expressivity, complexity and unpredictability in emergent languages. \n\n**Strengths:**\n\n- The paper properly motivates with examples, the study of expressivity, complexity and unpredictability in the framework of deep learning based language games as a way to gain further insights into natural languages which I find really interesting.\n- The paper introduces a novel measure of expressivity for emergent languages that is defined through its generalization performance across different tasks.\n- The paper studies the usage of contrastive loss in referential games and this loss helps alleviate the collapse of message types in language games which is interesting and novel.\n- The experiments are extensive which is great to see.\n- Further discussions in the appendix is also nice. Particularly, those around mutual information.\n\n**Weaknesses:**\n\nI have included actions that can be taken to strengthen this paper. If these actions are addressed, I am willing to increase my score.\n\n- **Section 3.1/4.1:**\n    - **Complexity/unpredictability:** I don't fully understand what equations 2, 3 mean, both are of the form $f(x_t) = E_{x_t}[...]$. So, $x_t$ is both an argument to the function and the expectation. How is the expectation over $x_t$?\n        - **Action:** Clarify equations 2 and 3.\n    - **Complexity/unpredictability:** I am a little confused at how the closed form solutions for equations 2 and 3 were arrived at. Equation 2 for example, might be correct only if it is sampled with replacement, not without. Both are of the following form: the probability that a set A contains an object from set B where A, B are sampled from X without replacement. The solution to this is 1 - (|X| - |B|  \\choose |A|) / (|X| \\choose |A|).\n        - **Action:** Correct or clarify solutions to equations 2 and 3.\n- **Section 3.2:**\n    - The paper does not fully motivate why Definition 3.1 is a good quantifiable metric for studying expressivity in emergent languages. How does generalization that is studied in emergent languages like [2] relate to this particular definition of expressivity?\n        - **Action:** Compare expressivity with the notion of generalization studied in past works.\n- **Section 4.1:**\n    - Predictions in section 4.1 depends on the expressions being accurate in Section 3.1. It is also not clear to me how the definitions of unpredictability and complexity leads to predictions over expressivity.\n        - **Action:** Clarify how we can expect these predictions to be true from the definitions provided.\n- **Section 5.3:**\n    - Looking at Figure 4 and the corresponding Table 4 (in the appendix), it does not seem that referY is statistically worse than referX for Y > X as claimed in this section.\n        - **Action:** Including these comparisons with a statistical significance test would be needed to draw these conclusions.\n\n**Comments:**\n\n- **Section 2.2:**\n    - What are source and target games? They are defined in a later section and pointing the readers to section 4.2 would make it easier to follow.\n    - Section 4.2 is a little difficult to follow. This section defines ReferX and ReferX/f which is used in the rest of the paper. The definition for ReferX/f is particular is not very clear. The following statement is also not very clear: \"since language games with fixed context are not good simulation of human communication , we keep only the ones having varying batches from $G_s$ in $G_t$.\"\n        - **Action:** Include further discussions into the definitions for ReferX, ReferX/f, $G_s$ and $G_t$.\n\n[1] Kirby, S., Tamariz, M., Cornish, H., & Smith, K. (2015). Compression and communication in the cultural evolution of linguistic structure. *Cognition*, *141*, 87-102.\n\n[2] Chaabouni, R., Kharitonov, E., Bouchacourt, D., Dupoux, E., & Baroni, M. (2020). Compositionality and generalization in emergent languages. *arXiv preprint arXiv:2004.09124*.", "summary_of_the_paper": "This paper studies the expressivity, complexity and unpredictability of emergent languages in referential games. The authors demonstrate that the expressivity of emergent languages is a trade-off between the complexity and unpredictability of the context that the languages are used in. They also introduce a contrastive loss based training method for referential games that alleviates the collapse of message types often seen when using other standard training methods.", "summary_of_the_review": "The paper properly motivates the study of expressivity, complexity and unpredictability in the framework of deep learning based language games as a way to gain further insights into natural languages. The paper also studies the usage of contrastive loss in referential games and this loss helps alleviate the collapse of message types in language games. The key contributions of the paper is predicated on the definition and closed form expressions for unpredictability and complexity. However, I do not believe that those expressions are sound. Although, there are extensive experiments, some conclusions are not well supported. Therefore, as it stands, I would not recommend the paper to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "YCrkOqCZIDU", "reviewer": "Reviewer_27ev", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors attempt to tackle the important issue of how functional pressures affect the properties of resulting emergent communication protocols. In particular, the are interested in quantifying the 'expressivity' of the resulting communication protocol, and give an operational definition of expressivity in terms of learnability by a randomly initialised listener. The authors note that mutual information is a poor measure for expressivity, and I like their proposed metric. It reminds me a little bit of various definitions of probing in the NLP literature, and perhaps a connection can be drawn here.\n\nThe resulting partial order in language transferability is described in text on page 8, but I think it would be better to draw the poset as a diagram, to make it easier to see what is comparable to what. In addition, I think the target games should be the x-axis of Figure 4, and the colours should be used for the source games. This way, we would be able to read off expressivity dominance immediately from the plot, as which curve was higher overall than the other. Further, it's unclear whether the generalisation performance is measured using a held-out set of language games, or whether the same games are used to train and evaluate the meta-suite of models.\n\nThe authors investigate the effect of two factors on language expressivity: scenario complexity and scenario unpredictability. Scenario complexity is defined as how many similar items each batch of data contains, on the assumption that items close in perceptual space are more difficult to distinguish. Scenario unpredictability is defined as the probability that the next batch of data contains an item not contained in the current batch.\n\nI feel like complexity as the authors have defined is a very sensible metric. However, I'm not convinced by the unpredictability measure. It only considers bigram transitions, so considering a game with 4 inputs a, b, c, d and cycling the inputs endlessly as (a, b), (c, d) would be maximally unpredictable under this metric, when clearly it shouldn't be. I feel like the right notion of unpredictable should consider some kind of online learning scenario, such as prequential prediction. In addition, the human experiments used to motivate unpredictability rely on the fact that humans can remember past contexts when playing the game. In comparison, the models considered in the paper are memoryless, and the only effect of data presentation order is implicit in the optimisation dynamics. I understand that the intuition is that unpredictable environments force the speaker to be maximally informative in its utterances, but it's not clear that the proposed metric is the right way to capture this intuition.\n\nAnother contingent issue is that, at least as presented, complexity and informativeness are dependent variables, and the true controlled variable is batch size (indeed, the x axis of all the plots is batch size, and the formulae used to define complexity and unpredictability are not used anywhere else in the paper). This is because data points for a batch are sampled i.i.d. This means that the two axes of variation are not independent, which leads to conclusions which can't tease apart the effect of either measured variable, and reduces the overall impact of the paper as the empirical results suffer from confounding. I believe that alternative data sampling schemes would help tease apart the effect of both variables.\n\nOne final thing is that the 'contrastive' loss presented in equation 1 is contrasted with a 'referential' loss which is never explicitly given in the paper. To make the exposition self-contained, it would at least help to see how the 'referential' loss is computed, so that the reader can make a better comparison. \n\n(As an aside: Wittgenstein did use the phrase 'language game', but his concern was not acquisition, but rather how meaning is modulated by context. The first sentence of the introduction therefore misrepresents his statements, and I feel should be removed.)\n\n=============\n\nPost author response:\n\n> Memory of human participants also decay, thus the context is always limited in a finite number of time steps. Defining unpredictability on a longer history simply changes the power of Equation 3 in our setting, but doesn’t influence the growth relationship between unpredictability and context size. Therefore, it won’t influence our conclusion if we extend Equation 3 to a longer history.\n\nThis is only assuming a uniform bigram/trigram/n-gram distribution. Often in the real world, sequential data has long-term structure which isn't well captured by any n-gram transition model (which is why RNN/Transformer language models so dramatically outperform count-based ones), and so it's not obvious that using more sophisticated transition schemes would result in similar findings.\n\nTo be honest, I find the unpredictability measure more opaque after author response. Section 5.1 varies complexity while keeping unpredictability constant using something called a 'fixed batch', but it's not very well explained what this is, and why it makes minimises unpredictability. Again, I wonder why predictability of the future given the past is important for these communicating agents, considering they are memoryless across episodes. I brought up this point in my original review, and it is not addressed in the author response.\n\n> We would be happy to include more useful definition of unpredictability if there’s other helpful definition you have in mind. We’ll also be happy to see more exploration on how unpredictability of context implicitly influence the training dynamics of agents.\n\nI gave a definition: online learnability using prequential coding (see, e.g. The Description Length of Deep Learning Models, Blier and Ollivier 2018 for a description). There is a large literature relating unpredictability and online learning, and I believe that many ideas in that field will be applicable here.\n\n> We would be happy to include a new data sampling scheme, if there’s some other useful method you have in mind.\n\nAny kind of non-independent sampling would decouple complexity from unpredictability, such as stratified sampling of contexts according to similarity given a target. I believe that there is much scope to precisely control complexity and unpredictability using such schemes, which I believe would make a future revision of this paper much more solid.\n\nAfter seeing the author resposne, I don't feel that many of the issues I raised in my review have been substantially addressed, and I have not changed my score. Most notably, the issue of the definition of unpredictability and how it affects the communication are still unclear, and I believe this needs to be clarified substantially in future revisions. ", "summary_of_the_paper": "The authors investigate what aspects of scenario design affect the resultant communication protocol in emergent communication. Specifically, the authors investigate whether scenario complexity (in terms of having many similar distractors) and scenario unpredictability (in terms of future batches not containing items to previous batches) affect the expressivity (measured in terms of learnability) of the resulting protocol.\n\nFurther, the authors show that defining a softmax loss over the entire batch of possible references (dubbed the 'contrastive' loss) outperforms the 'referential' loss used in previous works. ", "summary_of_the_review": "Some interesting ideas in this paper (measuring the effect of the environment on the effectiveness of the language protocol, introducing a new and sensible definition of effectiveness), but unfortunately I think there are issues outlined in the main review which preclude acceptance in the current state. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "39jyNt_e48Q", "reviewer": "Reviewer_B1Wi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, the biggest strength of the paper is the empirical evaluation conducted to support the claims. The experimental design and setup look sound. The definitions and different notions introduced in the paper seem natural and reasonable. At the same time, given the notions of complexity and unpredictability, a number of conclusions seem very natural based on intuition.\n \nThe paper is very well written and the arguments are backed up with reasonable experiments.  \n\nThe proposed metric seems to work well in the given setting. Isn't the metric is dependent on the transfer learning abilities of the model to a certain extent? Will the partial ordering be consistent across different types of models?  \n\nThe idea of applying contrastive loss is not particularly novel. It seems like to address the issue of space efficiency, they simply confined the previous method to batches rather than the entire set. ", "summary_of_the_paper": "The paper studies the properties of emergent languages in DL-based language games. In particular, they look at referential games where speakers emit messages and listeners need to identify the target object observed by the speaker from some set of candidate objects.  \n\nThey define the expressivity of an emergent language as the amount of discriminatory information required to encode the inputs so that a listener can correctly decode them. In this setting, complexity refers to the similarity between different objects in a given context. In a context with higher complexity, more similar objects will be present, and more discriminatory information will have to be encoded so that a listener is capable of identifying the correct target. In their definition, the notion of unpredictability can be thought of as how stable the information necessary for encoding is across different trials.  \n\nTheir primary contribution is to support the hypothesis that the expressivity of emergent language is determined by (and a trade-off between) the complexity and unpredictability of context in language games. The authors introduce a new measure to evaluate expressivity based on partial ordering between languages in terms of their generalization across tasks. They argue that mutual information is not the most appropriate measure to evaluate the expressivity of languages. They propose a contrastive loss which they show helps mitigate the issue of the collapse of message types. ", "summary_of_the_review": "Overall, it seems like a strong paper with a clear hypothesis and brings in a novel perspective which could be a worthwhile contribution in the subarea of emergent languages. I see no obvious issues with the paper.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the expressivity, complexity and unpredictability of emergent languages in referential games. The authors defined measures of complexity and unpredictability and empirically showed that the expressivity of emergent languages is a trade-off between the complexity and unpredictability of the context that the languages are used in. They introduced a contrastive loss based training method that alleviates the collapse of message types seen using standard referential loss functions.

The paper is controversial among the reviewers. On the positive side, most liked how the paper has a clearly stated hypothesis and extensive evaluations which makes a clear contribution to the field of emergent languages. On the negative side, the paper only shows the results in an artificial setting where the key variables are highly simplified (e.g. size of candidates). The main negative review argue the authors used an inappropriate definition of unpredictability and that the batch size is actually the key independent variable instead of what is claimed. The paper does somewhat equate batch size with the candidate size that is so important to their results (after eq (1)), but they seem to measure candidate size in the key figures. Perhaps an experiment controlling for batch size independently of the candidates size can address this issue. On the point of defining unpredictability, the other reviewers and I find the given definition to be reasonable and at least defensible. However, the reviewer remained unconvinced. More generally, the paper relies on one definition of the concepts measured in one setting to make a general claim, which is at risk of missing other important variables. Overall, most reviewers found the scope to be sufficient, and two improved their scores after the discussion.

Recommendation: accept
Decision: Accept (Poster)