Emergent Communication: Generalization and Overfitting in Lewis Games
Authors: Mathieu Rita
Corentin Tallec
Paul Michel
Jean-Bastien Grill
Olivier Pietquin
Emmanuel Dupoux
Florian Strub
Venue: NeurIPS 2022 Accept (2022)
Keywords: Emergent communication
Lewis games
Signaling games
generalization
compositionality

Abstract: Lewis signaling games are a class of simple communication games for simulating the emergence of language. In these games, two agents must agree on a communication protocol in order to solve a cooperative task. Previous work has shown that agents trained to play this game with reinforcement learning tend to develop languages that display undesirable properties from a linguistic point of view (lack of generalization, lack of compositionality, etc). In this paper, we aim to provide better understanding of this phenomenon by analytically studying the learning problem in Lewis games. As a core contribution, we demonstrate that the standard objective in Lewis games can be decomposed in two components: a co-adaptation loss and an information loss. This decomposition enables us to surface two potential sources of overfitting, which we show may undermine the emergence of a structured communication protocol. In particular, when we control for overfitting on the co-adaptation loss, we recover desired properties in the emergent languages: they are more compositional and  generalize better.

Emergent Communication:
Generalization and Overfitting in Lewis Games
Mathieu Rita INRIA, Paris mathieu. ritiiiuurita@inia..f.
Corentin Tallec
Paul Michel' Jean-Bastien Grill DeepMind
[corentint, pawpliitt.aammiimaaaam che,, j @deepmind. com
Olivier Pietquin Google Research, Brain Team pietquin@google.comm
Emmanuel Dupoux
Florian Strub
DeepMind
Meta AI Research
fstrub@deepmind.comm com
emmanuel. hat.nuuellduuuuuxmmal...oooo
Abstract
Lewis signaling games are class of simple communication games for simulating the emergence of language. In these games, two agents must agree on a commu- nication protocol in order to solve cooperative task. Previous work has shown that agents trained to play this game with reinforcement learning tend to develop languages that display undesirable properties from linguistic point of view (lack of generalization, lack of compositionality, etc). In this paper, we aim to provide better understanding of this phenomenon by analytically studying the learning problem in Lewis games. As a core contribution, we demonstrate that the standard objective in Lewis games be decomposed in components: co-adaptation loss and an information loss. This decomposition enables us to surface two po- tential sources of overfitting, which we show may undermine the emergence of a structured communication protocol. In particular, when we control for overfitting the co-adaptation loss, we recover desired properties in the emergent languages: they are more compositional and generalize better.
Introduction
Understanding the dynamics of language evolution has been challenging if not controversial research topic in the language sciences [32, 13]. Given that the very first human language cannot be unearthed from fossils [5], computational models have been designed simulate the emergence of structured language within a controlled environment. In this line of work, Lewis signaling games [55] are among the most widespread playground environments to model language emergence: they inherently simple, yet they exhibit a rich set of communication behaviors [17, 70]. Therefore, understanding Lewis games dynamics may shed light on the prerequisites of language emergence.
In their original form, Lewis signaling games involve two agents: speaker and a listener. The speaker observes random state from its environment, e.g. image, and sends signal the listener. The listener then undertakes an action based on this signal. Finally, both agents are equally rewarded based on the outcome of the listener's action. The resolution of this cooperative two-player game requires the emergence of a shared protocol between the agents [55, 17]. One way to model the emergence of such protocol is give the agents the capacity learn. The agents, and therefore, the communication protocol, are shaped by a sequence of trials and errors over multiple games [81, 44, 75, 70J. This learning-centric approach allows for fine analysis of the language emergence dynamics [70, 36).
This work performed when Paul Michel was affiliated with Ecole Normale Supérieure PSL.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

also raises challenging learning spprciicc questions: What are the inductive biases present in the agent architecture and loss function that shape the emergent language [43]? How do agents generalize from their training set? Is the resulting language compositional [8]? What is the impact of overfitting [53]?
Recently, there has been resurgence of interest for such learning-based approaches following advances in machine learning [51]. In these approaches, the speakers and listeners are modeled as deep reinforcement learning agents optimized to solve instances of the Lewis games [53, 33, 65, 56. 28]. The vast majority of these works explore Lewis games from an empirical perspective. However, some of the recent experimental results are odds with experimental findings from the linguistics literature. For instance, the emergent protocols lack interpretability [48], generalization does not always correlate with language compositionality [10], successful strategies are not naturally adopted in populations [68, 12]. and anti-efficient communication may even emerge [9]. It is unclear whether those empirical observations result from learning failure, e.g. optimization problems, overfitting, or whether they are symptomatic of more fundamental limitations of Lewis games for modeling language emergence, e.g. lack of embodiment [31, 4, 63, 37]. Overall, is crucial to establish new analytical insight to analyze Lewis games in the learning setting.
In this paper, we introduce such an analytical framework diagnose the learning dynamics of deep reinforcement learning agents in Lewis signaling games. As a core contribution, we demonstrate under mild assumptions that the loss of the speaker and listener can be decomposed into two components when resolving Lewis signaling games: (i) an information loss that maximizes the mutual information between the observed states and speaker messages: (ii) a co-adaptation loss that aligns the speaker and listener's ' interpretation of the messages (Section 2). Based on this decomposition, we empirically examine the evolution of these two losses during the learning process (Section 5). In particular, we identify an overfitting problem in the co-adaptation loss between the agents which undermines the emergence of structured language. We then show that the standard setup used in the deep language emergence literature consistently suffers from this overfitting issue (Section 5.1). This realization explains some of the contradictory observations [10] and experimental choices from past works [65, 56, 68]. Finally, we explore regularization methods to tackle this co-adaptation overfitting. We observe that reducing the co-adaptation overfitting allows for developing a more structured communication protocol (Section 5.2).
All in all, our contributions are three-fold: (i) we provide a formal description of Lewis games from a learning standpoint (Section 2.3); (ii) we apply this framework in experiments to show that degenerate results are primarily due to overfitting in the co-adaptation component of the game (Section 5.1): : (ii) we propose natural ways of tackling this overfitting issue and show that, when we control the receiver's s level of convergence, we obtain a well-structured emergent protocol (Section 5.2).
2 Analyzing Lewis Games
We show that Lewis games' objective decomposes into two terms: (i) an information loss that measures whether each message refers to a unique input; (ii) a co-adaptation loss that quantifies the alignment the speaker's and listener's interpretation the messages.
For simplicity and to ease the reader's intuition, we focus on the reconstruction variant of Lewis games with agents optimizing the reconstruction log-likelihood in the main paper. In Appendix A, we show that our analysis extends to broader of Lewis signaling games, e.g. discrimination games [12. 60, 21, 30, 65, 53, 52, 33, 56, 58], and general form of reward that covers the rewards commonly used in emergent communication, c.g. log-likelihood [9, 10, 40, 68, 67, 1 11], accuracy reward [65, 53, 52, 48, 56, 28, 23].
2.1 Background: Lewis Reconstruction Games
Game formalism In reconstruction Lewis games, a speaker observes random object of its envi- ronment. The speaker then sends a deseriptive message, which a second agent, the listener, uses to reconstruct the object. The success of the game is quantified by how well the original object reconstructed [40, 9, 67, 68). Formally, the speaker parameterized by and the listener is parameterized by Φb The observed object denoted by is selected from a set of objects denoted by X. We denote by X the random variable characterizing æ, sampled from distribution p. The intermediate message sent by the speaker m belongs to the set of all potential messages M. The speaker follows

---

policy which samples message m with probability (m(m)z) conditioned on object z. We denote by Mg the random variable characterizing the message m, sampled from (- We denote by no (m) = E. Na (m])))((r) the marginal probability of message given policy xp- Given message m, the listener outputs a probability distribution over inputs pol (m), and the probability of reconstructing the entire object x given m is thus Po (z/mm)
Game objectives reconstruction games, the speaker and listener minimize the negative log likelihood of the reconstructed object. Both agents thus optimize the objective:
La.s -B==p,m=-ot(t [lo Pa (2/mm),
where optimizing the speaker a reinforcement learning problem whose parameters are optimized using policy gradient [76] and optimizing the listener is supervised learning problem whose parameters @ are optimized with gradient descent. In our theoretical analysis, we consider that agents not regularized. In practice, regularizations, e.g. entropy regularization [59], may be added the game objective but does not alter our main conclusions.
2.2 Building Intuition on the Lewis Reconstruction Game Learning Dynamics
To get a better intuition of the dynamic of Lewis reconstruction games, we can analyze the form taken by the optimal listener, given speaker Ta. In what follows, we use sub-script denote an explicit dependency of the policy on parameters 0, e.g. policy parameterized with neural network. Conversely, the use of super-script (0) corresponds
implicit dependency of the policy parameters 0. As shown in Appendix 1.1.1, given a message m, the optimal listener's distribution p=(0) m)) can be written in closed-form:
p(z))o((m(z)
(flm): Ev'ex p(='))T(m)(x')
Here, (0(0) does not depend on Φ. but implicitly depends on 0, as the optimal listener given policy parameterized by At each update, the listener ps gets closer its optimum we suppose that the listener perfectly fits (0()) (m) at any moment, the loss becomes:
Co.s -Bx-p.mm llog p (x|m)
((X; + H(X)
where H(X[Ma) is the conditional entropy of X conditioned on MM and [(X; Ma) is the mutual information between X and Mg. Thus, if the listener is optimal at every point in time, the speaker's task merely becomes the construction of a message protocol that maximizes the mutual information between objects and messages, i.e. the construction of unambiguous message protocol.
In practice, the listener never perfectly fits the optimum. In the following, we elucidate the effect of this gap between the listener and its optimum on the dynamics of the game.
2.3 Analytical Result: The Lewis Games Loss Decomposition
In cooperative Lewis games, the agents' loss can be decomposed into two terms:
L0. Linto + Ladapt.
An information term Linto quantifies the degree of ambiguity of the language protocol. It minimal when each message refers to a unique object;
A co-adaptation term Ladapt quantifies the gap between the listener and its optimum: the speaker's 's posterior distribution. This co-adaptive term is optimized both by the speaker and the listener. When the listener is optimal, this co-adaptation objective is zeroed.
In particular, the decomposition takes the following form in the Lewis reconstruction game:
Ca.s = H(X + Em (x (0 (1mm)lol (m)),
(5)
Linto
Ladaps
3

---

The proof of the decomposition is provided in Appendix A. Appendix A provides the proof for the reconstruction -ikelliihooo reward and extends to broader class of Lewis signaling games, e-g. discrimination games, and general cooperative rewards covering usual emergent communication rewards, e-g. the accuracy reward. This decomposition gives us insights on the game dynamies and the constraints that shape languages in the game with neural agents:
The information loss Linfo captures the speaker's intrinsic objective: develop an unambiguous protocol. Linto is minimal, equals to 0, when the communication protocol is unambiguous, i.e. every message from the speaker's policy Ty refers to unique object. Conversely, Linto maximal, equal to H(X), when the message protocol is fully ambiguous, and X and M3 are independent variables.
The co-adaptation loss Ladapt is specific to learning agents. This loss measures how far the listener Ps is from its optimum p=(0), If Ladaps O, the listener and its optimum coincide. Cadapt has the particularity to be optimized by the two agents, From the listener's ' side, it merely corresponds to the optimization of its supervised task. From the speaker's 's side, brings out that the speaker must adapt its language to the listener in addition to build an unambiguous message protocol. In other words, the co-adaptation loss pushes the speaker develop a language that can be easily recognized by listeners. This pressure diminishes as the listener approaches its optimum.
From a practical perspective, Equation (5) yields the following individual gradients:
VeCo VX + VgEman (KL(p (-1m)]lp((-mm)) do =
(6)
VoEmon, ( (0) (m) (m)),
where the listener only receives gradients from the co-adaptation term, and the speaker receives gradients from both terms.
This loss decomposition also finds echoes in the cognitive science literature in the form of an expressivity vs. learnability trade-off [72]; see Section 6 for a detailed discussion.
2.4 Generalization Gaps in Lewis Reconstruction Games
We explore another facet of the loss decomposition that arises from learning. As agents are trained on partial views of their environment, it opens questions of overfitting and generalization to unseen objects. As is customary in machine learning, we consider agents trained on a fixed, finite sample from the data distribution: the training set. Let us denote by Pursin the empirical object distribution over the training set and Xtrain an object sampled from Purain- Similarly let Mg denote a message sampled from ( train), m merain (m) = En Te (m (m|a))) Ptrainn (x) the marginal probability Ptesin (w) (m)aa)
of message the training set, and Perain mm) Lxex Parsin (m)]) the speaker's posterior
(0)
distribution with respect to the prior distribution Ptrain- The training loss can be written as follow:
Ctrain - E -n--aiiii mo (11x) [log (=(m))) 0,
H(X train [M train
+ Em D KL (or (-Im))lp(((Im)
Ctrain
Closia
Decomposing the gap between Ctrain and La.s uncovers two sources of overfitting: L0
etrain Lg
La. +
Liato
Linto
adapt
Ladapt
information overlitting
co-adapiation overlitting
Intuitively, information overfitting occurs when the speaker only develops an unambiguous language on the training set, but ambiguities remain on the total dataset. Co-adaptation overfitting occurs when the two agents agree on a common communication protocol on the training data, but not on all data.
3 Method
This section gathers the methodological tools required to empirically study the loss decomposition.

---

Lewis game
Probing
Train until stopping cniterion
Speaker
Freeze
Speaker
Probe listener
loss information
( Linto )
Freeze
Listener
Listener
loss
loss co-adoptation
( B
( (uudpp
Troin
Paris
Figure I: Probing method: (1) the speaker and listener are frozen and the probe listener is initialized. (2) the probe listener is trained on Purain (resp. p) with the speaker's messages until convergence; (3) The speaker takes imputs from Ptrain (resp. Ptest) and messages the probe listener and the listener. The resulting loss of the probe listener is Linfo, and the loss of the listener is used estimate Ludapt-
3.1 Probing the Information and Co-adaptation Losses
Computing Linto and Ladapt directly necessitates estimating the posterior distribution of the speaker, p=(0) m). Doing so requires summing over all X' which is intractable. Fortunately, deep models are large enough so that they can perfectly solve their task on their train set. We can leverage this fact to compute empirical estimates Linfo and Eudape of Linto and Ladapt respectively by using an auxiliary listener trained to optimality.
We here detail an empirical probing mechanism obtain estimates Linto and Ladapt given speaker Ty and listener Pa. As noted in Equation 2, the posterior p (0) also corresponds to the optimal listener. Therefore, we obtain an estimate of the posterior by training a listener to optimality, and use this optimal listener decompose the loss. practice, obtain this optimal listener, we freeze speaker Ty and listener Pg and initialize a new, auxiliary listener from scratch, which we refer to as the probe listener. As illustrated in Figure 1, the probe listener trained to reconstruct object æ from message with drawn from distribution p or Ptrain and m sampled according to the frozen speaker policy (-))), until a stopping criterion is met. We then distinguish between train and test estimates:
Atrain Erprperrin Perain m (m) [log train (xx m]
(8)
Atrain = E perain my log [m]
Atrain info
adapt
and,
Atest
Expopeant mom (- Dlg Pwr (ar [m]]
adapp
Atest
E Exeepeest momm o (-1x))
llog Ps (z( mm]] info
where plaain and Pw- are the probe listeners trained over distributions Ptrain and p respectively." Note that this probing mechanism, while tractable, is computationally costly as it necessitates training new probe listener to convergence, and so we only use it as a valuable diagnosis tool.
3.2 Ballaccigg the Information and Co-adaptation Terms
As explained in Section 2.3, the information loss alone is sufficient for the speaker to develop an unambiguous language. This begets the question: does the no-adaptation loss have any bearing the emergent language at all? We elucidate this question by balancing the weight the co-adaptation term in the decomposition. By using the probing method described above, build the following training loss:
Co(a) = (1 a) Atrai + a
Atrain
where
[0: 0.5]
(10)
adapt
"The estimate is trained on the fill distribution of objects, and not Preat. Training on Preat would result in optimal listener overfitting the test set, which would results in bad estimates of the mutual information.
5

---

Hence, balances the two speaker objectives (up to an approximation error). When a = 0.5, the loss falls back the classic setting. When a 0, the co-adaptation term is removed on the speaker side; note that the Lewis game can still be solved since the listener still optimizes the co-adaptation term. We experimentally analyse the effect of a on resulting languages in Section 5.1. Appendix B, we describe how we build the balanced loss and explain why a should be bounded by 0.5.
3.3 Controlling the Listener's Co-adaptation Loss Level of Convergence
As mentioned in 2.2, the influence of Ladupt on the co-adaptation term in the speaker's loss is modulated by the listener's level of convergence to its optimum. To understand the effect of this co-adaptation, we decouple the speaker and listener training and train the listener via three procedures:
Continuous listener The listener is continuously trained, jointly with the speaker. This is the standard setting in the emergent communication literature, and serves to report the baseline behavior.
Partial listener The listener is re-initialized after each of the speaker's 's update and trained on the training set for Natep before updating the speaker again. This baseline enables fine-grained analysis of the influence of under-training (low Nstep) and over-training (large Nstep) the listener.
Early stopping listener The listener is also re-initialized after each of the speaker's ss update but is now trained until an early stopping criterion is met on the validation set. This allows us to get the best estimate of the posterior p=(1) (.))) each update. This can be seen variant of the partial listener with an adaptive number of steps Natep-
Experimental settings
4.1 Game description
Unless specified, all our experiments are run on the reconstruction game defined in Section 2.1. Experiments are run over 6 seeds and reach > 99% training reconstruction scores unless otherwise stated. Our implementation is based on the EGG toolkit [39] and the code is available at https: / //github. www.///aatheeeettttaaalllllaaaia
Environment We consider objects x =; (II,..,,x)) E =x= X Ax characterized by K attributes where attribute may take X different values. By design, this synthetic environment allows us to test the ability of agents to refer to unseen objects by communicating their attributes [3, 48]. Each object is the concatenation of one-hot representations of the attributes (Iz)asisk. Objects have K = 6 attributes, each taking 10 different values, for a total of 1 million objects. Training, validation and test sets are randomly drawn from this pool of objects (uniformly and without overlap), and are respectively composed of 4000, 1000 and 1000 elements. Thus, the agents only have access to small fraction (< 1%) of the environment, making the generalization problem challenging.
M Communication channel Messages m =: (my)}= M =: V" are sequences of I' tokens where each token is taken from a finite vocabulary V. finishing by hard-coded end-of-sentence token EoS. In our experiments, messages have maximum length T = 10 and symbols are taken from a vocabulary of size |V| = 10 to prevent bottleneck in the communication channel.
Speaker model The speaker follows a recurrent policy: given an input object æ. it samples for all e [] token m, with probability
(m/mm. 2). 2). The speaker takes in the object x as vector of size K x [x[ and passes it through a linear layer of size 128 to obtain an object embedding, used to initialize a L.STM [35] of size 128 with layer normalization [2]. At each time step, the LSTM's output is fed into a linear layer of size Vl. followed by a softmax, to produce Tel (m, (m 2)
Listener model Given message m (m.,..., mT), the listener outputs for each attribute A probability distribution over the | values: (-mm) The probability of reconstructing the entire object v given m then (x(x) m) := 1I 15..p./..0. (=*(m) (m). The listener passes each message m, through an embedding layer of dimension 128 followed by LSTM with layernorm of size 128. The final recurrent state h is passed through K linear projections of size |x],, each followed by a softmax, providing K independent probability distributions of sizes | predict each attribute of æ.
Optimization The agents are optimized using Adam [42] with learning rate 10-4, 3 = 0.9 and B2 = 0.999 and batch size of 1024. For the speaker use policy gradient [76], with bascline

---

computed as the average reward within the minibatch, and an entropy regularization of 0.01 to the speaker's loss [82]. In all experiments, we select the best models by early stopping.
4.2 Evaluating emergent languages properties
Generalization We measure generalization by computing the average test reconstruction score over all the attributes of probe listener trained on the training set using an early stopping criterion on the validation set. Indeed, the trained listener Pe may overfit to the training set, and so using it may under-estimate. Using separate listener removes this bias.
Compositionality Compositionalliy is a fundamental feature of natural language often seen precondition to generalize [6, 77. 79]. We assess the compositionality by computing the topographic similarity [8, 53]. defined as the Spearman correlation [47, 80] between the distance input space, i.e. the average number of common attributes, and the distance in message space, i.e. the edit-distance between the corresponding messages [54]. As we here deal with large object space and stochastic policies, we use bootstrapped estimate of topographic similarity as in [46] to get reliable numbers. We sub-sample 1000 elements x from the object space X, and sample the corresponding message from the speaker's policy To (-(zz). We compute the topographic similarity for this batch of 1000 pairs (x, m). We repeat this protocol 100 times and take the mean to measure compositionality.
Empirical results
5.1
Visualizing the loss decomposition dynamies
Losses (at convergence)
Info. (train)
6.0
0.60
Losses (a=1]
10
Co-adapt. (train)
accuracy
Co-adapt, (test)
4.0
Information (test)
Co-adaptation (test)
5
0.50
2.0
0.45
0.0
2000
4000
6000
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1 0.2 0.3 0.4 0.5
Iteration
Co-adaptation weight a
Co-adaptation weight
Figure 2: (a)Training dynamics (a = 0.5). (b,c)Agents score as function of co-adaptation weight a.
We here visualize the loss decomposition dynamics. Following the protocol of 3.2, we control Ltrain adapt in speaker's loss with weight a to understand the influence of the co-adaptation term on the language.
The co-adaptation task overfits rapidly We plot information and co-adaptation training dynamics in the standard setting (a 0.5). Note that both train and test information losses quickly converge to 0, in other words the speaker succeeds in developing protocol that is unambiguous on both the training set and the overall distribution. On the other hand, the test co-adaptation loss diverges while the train co-adaptation keeps disminishing, highlighting a clear overfitting problem.
The co-adaptation task promotes generalization We then display in Figure the evolution of the information and co-adaptation losses for different co-adaptation weight a. We observe that down-weighting Lirain tends enforce both information and co-adaptation overfitting. Thus, even though the co-adaptation loss is not inherently necessary for the speaker develop an unambiguous language, it is important to encourage the speaker to build a better language. This is confirmed when looking generalization accuracies. From generalization. In conclusion, we note that (i) balancing the loss in favor of cerain has a negative impact on generalization, (ii) the co- ro-adaptation loss Ctrain that generalizes better.
0
0.5, there gain of 15 points of
adapt pushes the speaker to develop language
These experiments highlight two key findings: (i) co-adaptation is crucial for generalization ; (ii) in standard settings, the co-adaptation loss overfits substantially, whereas the information loss does not.
5.2 Countering co-adaptation overfitting
We here investigate whether limiting overfitting in the co-adaptation loss may push towards languages that generalize better and are more stuctured. As described in 3.3, we compare three control baselines: Continuous listener, Partial listener with varying levels of convergence, and Early stopping listener.
7

---

1.0
Top. similarity aryyy
0.4
Top. similarity
Accuracy
0.8
0.6
0.3
0.3
0.4
0.2
0
200
400
600
0.2
200
400
600
Nstep
Nstep
Test accuracy
Partial (train)
e Partial (test)
750
Continuous
Early stopping
Figure 3: (a,b) Evolution of generalization and top.sim with Partial listener's number of learning steps Natep ; (c) Top. sim VS. generalization. The color level of orange dots increases with N Nstep- Blue (resp. green) lines and points refer to the Continuous listener (resp. Early stopping listener).
Countering co-adaptation overfitting improves generalization In Figure 3, we observe that the level of convergence of the Partial listener between each speaker's update (controlled by Nstep) has a strong impact on the generalization of the emergent protocol. Overall, we recover classic machine learning trends when varying Natep: when Natep 50, both train and test accuracy are low the agents underfit. When 50 < Natep < 250, the train and test accuracy are almost optimal the agents are good training regime. Finally, when Natep 250, the train accuracy is maximal while the test accuracy collapses - the agents overfit. These observations reveal that the level of convergence of the listener has substantial impact on the final emergent language capacity to generalize. Recall that, in these experiments, the direct effect of the listener's s overfitting is mitigated, as we measure generalization using an auxiliary listener that is early stopped, and should therefore not overfit as noted in Section 4. The listener's overfitting impacts the speaker's 's update through the co-adaptation loss, which, by inducing poorer final language leads to a degradation in generalization. Additionally, Figure 3 shows that the continuous listener, standard in the Lewis games literature, provides generalization performance similar to the worst overfitting listeners.
Controlling the listener's co-adaptation level appears crucial to let the speaker develop a language that generalizes well; this effect may have been underestimated in the standard Lewis learning dynamic.
Countering co-adaptation overfitting improves compositionality Figure reveals that composi- tionality follows the same pattern. In the underfitting regime, the topographic similarity is low but still outperforms the Continuous listener. Similarly, it is also low in the overfitting regime. In In-between the two which corresponds to high generalization in Figure 3 the topographic similarity reaches high values, which suggests that more compositional languages emerge. This indicates that the listener's lack of co-adaptation overfitting promotes structured languages.
Compositionality correlates with generalization In Figure 3, we plot the correlation between generalization and compositionality. As opposed to [10], observe a strong correlation between generalization and topographic similarity when varying the Partial listener's level of convergence. In particular, identify two correlation branches: the second to the overfitting regime. Together, they retrace the evolution of generalization and compositionaliiy with respect to N Nstep. We see that Continuous listeners belong the end of this trajectory, in the overfitting regime. Note that the blue rectangle which delineates the range of values reached with the Continuous listener corresponds the classic learning setting in the literature. As this range is tight, it may explain the initial negative results reported by [10].
belonging to the underfitting regime and
In conclusion, the listener exerts necessary pressure on the speaker to develop structured language that generalizes better. This pressure can controlled by limiting the listener's level overfitting. which is inevitably too high when the listener is trained continuously is usually done.
Comparison with standard regularization methods In practice, re-initializing the listener as done with the Partial or Early stopping listener is costly. We thus test whether performances comparable to Figure can be obtained by controlling the listener's level of overfitting with standard regularization methods. In Table 1, we report the influence of applying common regularization methods to the listener on various metrics of the language. We find that regularization consistently results in noticeable improvements. Moreover, once again, gains of generalization correlate with gains of
8

---

Gen. t
Compo.
arilaps +
Generalizaatin
Continuous
0.58+o.00 40.06
0.22+0.02
4.64+1.22
CelebA
I/1
1/20
1/100
Dropout
0.G4-2o.08
0.2440.01
4.86+
Continuous
0.9440.01
0.67+ na
0.39
No LN.
0.70
0.2420.02
0.68
Early stopping
0.97+0.01
0.80-0.00
0.69+0.04
Weight decay
0. 0.72
0.2551
4.29
No
WD
0.87 #0.07
0.30
2.1240.67
ImageNet
1/1
1/20
1/100
Early stopping Top Partial
0.96++0.0
0.39+0.04
1.10+a.69
Continuous
0.964
0.77 01
0.51
0.95
0.42
0.97.2.55
Early stopping
0. 0.98+0.01
0.81 -0.01
0.64440.0
Table I: (left) Performance comparisons between Continuous listener, Partial listener, Early stopping listener and classic listener regularization, e.g- weight decay [34, 491, Dropout [74] and layernorm [2]. Regularization parameters were tuned and are detailed in Appendix C. (right) Generalization scores for continuous baselines and Early stopping listener visual Lewis Games. 1/1, 1 1/20 and 1 1/100
refer to the subset ratios of the dataset.
compositionality. These trends corroborate our hypothesis that controlling the listener's learning is key to encourage the speaker to develop more structured languages, However, those methods remain under the upper bound reached by the Early stopping listener, which suggests that further research on regularization in cooperative games is warranted.
We complement this analysis in Appendix C.2 by studying the impact of regularization the speaker's ' side, and show that such regularization does not result in similar improvements. This indicates that the listener is the main contributor to the co-adaptation overfitting.
5.3 Scaling to the Image Discrimination Games
To validate our empirical findings beyond synthetic games, we scale our approach to complex games with natural images as advocated by [12]. We thus train our agents on a discriminative game on top of the CelebA [57] and ImageNet [69, 19] datasets while applying previous protocol. We work on 3 sizes of training set with increasing generalization difficulty. We provide all the training details and game settings in Appendix D.1 and report our results in Table 1. While agents generalize well when trained on the entire training set, generalization issues occur on smaller training sets and performances can indeed be improved by controlling the listener's ' level of convergence. However, Appendix D.2 shows that gain of generalization does not correlate with gain of topographic similarity, supporting that agents' language structure is not captured by the topographic similarity in image based settings [12, 1].
Related work
The decomposition of the loss function in the Lewis Game that we introduced finds echos in the cognitive science literature. According to Skyrms [70], communicative organisms or systems are confronted with two types of information: about the environmental states shared by the agents (called objective information), and about how an agent would react to a signal (called subjective information). Communication protocols emerge as a trade-off between constraints related to those two types of information [44, 45]: the sender should be expressive [25, 24] and transcribe the information available in the world with as little ambiguity as possible, which has been described as bias against ambiguity [73] ; sender and receiver should agree on the same referring system, which has been described as conceptual pact [7]. The latter has been shown to impose compressibility and Icarnability pressures promoting structure [78, 72, 84]. This analysis resonates well with our analytical decomposition of the loss function in the Lewis game.
The first term of the decomposition, which we called the information loss, has been addressed by previous work that assumed that linguistic structure and generalization emerge from the requirement of creating an unambiguous language. In this line of work, studies have either manipulated the complexity of the environment [12, 30, 71, 60), restricted the bandwidth of the communication channel [48, 66], or added noise to the message [50, 85). In our main experiment, we do not apply such information constraints to better focus the second term of the decomposition, the co-adaptation constraint, less studied within machine learning approach. Previous work have assumed that the co-adaptive dynamics encourage speakers to develop more structured language for

---

learnability reasons [56]. Support for this hypothesis can be found directly via the implementation of neural variant of Iterated Learning [65] or the introduction of learning speed heterogeneities [68] and indirectly via the restriction of agents capacity [66], the variation of the communication-gren -graphh in populations [28, 41] or the addition of newborn agents [15]. In our paper, we demonstrate that co- adaptation term is always present in standard agents optimization protocols and show that controlling co-adaptation overfitting enhances language properties. The existence an overfitting regime found under the default setting (continuous training) may explain the counter-intuitive lack of relationship between compositionality and generalization previously reported with neural agents [51, 10, 38, 20].
7 Conclusion
In this paper, we propose methodological approach better understand the dynamics in Lewis signaling games for language emergence. It allows us to surface two components of the training: (i) an information loss, (ii) co-adaptation loss. We shed light that the agents tend to overfit this co-adaptation term during training, which hinders the learning dynamic and degrades the resulting language. soon as this overfitting is controlled, agents develop compositional languages that better generalize. Remarkably, this emergent compositionality does not result from environmental factors, e.g. communication bottleneck [43], parametrarattinaaiioonn [48, 26]. population dynamics [12, 68], memory restriction [15, 16] or inductive biases [67]. but only through trial-and-rroor process. Therefore, we advocate for better comprehension of the optimization and machine learning issues. As illustrated in this paper, such understanding may unveil contradictions between computational models and language empirical observations and better expose the existing synergies between learning dynamics and environmental factors [27, 83, 64, 14, 18. 22].
Acknowledgments
Authors would like to thank Rahma Chaabouni, Marco Baroni, Paul Smolensky, Bilal Piot and Karl Tuyls for helpful discussions and the anonymous reviewers to their relevant comments. M.R would also like to thank Michael Sander and Maureen de Seyssel for last minute feedbacks. M.R. was supported by the MSR-Inria joint lab and granted access to the HPC resources of IDRIS under the allocation 2021-AD01 1012278 made by GENCI. P.M. was supported by the ENS-CFM Data Science Chair. E.D. was funded in his EHESS role by the European Research Council BBRC-2011-ACGG 295810 BOOTPHON), the Agence Nationale pour la Recherche (ANR-17-EURE-001 Frontcog, MAR--0-IDEE0001-00 PSL*, ANR-19-P3IA 1-1-199--A-0000 PRAIRIE 31A Institute) and grants from CIFAR (Learning in Machines and Brains) and Meta AI Research (Research Grant).
10

---

References
[1] Jacob Andreas. Measuring compositionality in representation learning. In Proc. of International Conference on Learning Representationss (ICLR), 2019.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint 1xxvv1667.006450. 2016.
[3] Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. Philosophical Transactions the Royal Society B: Biological Sciences, 375((791):20100377 2020. doi: 10. (..109//stb.2019.0000
[4] Lawrence W Barsalou. Grounded cognition. Annu. Rev. Psychol, 59:617-645, 2008.
[5] Derek Bickerton. Language evolution: A brief guide for linguists. Lingua, 117(3):10--26,, 2007.
[6] Derek Bickerton. More than nature needs. In More than Nature Needs. Harvard University Press, 2014.
[7] Susan E Brennan and Herbert H Clark. Conceptual pacts and lexical choice in conversation. Journal of experimental psychology: Learning, memory, and cognition, 22(6):1482, 1996.
[8] Henry Brighton and Simon Kirby. Understanding Linguistic Evolution by Visualizing the Emergence of Topographic Mappings. Artificial Life, 12(2):229 242, 04 2006. ISSN 1064-5462. doi: 10.1 100/aat.2.00 12.2.229.
[9] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Proc. of Advances Neural Information Processing Systems (NeurIPS). 2019.
[10] Rahma Chaabouni, Eugene Kharitonov. Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proc. of the Association for Computational Linguistics (ACL), 2020.
[11] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences, 118(12), 2021.
[12] Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In Proc. International Conference Learning Representations (ICLR), 2022.
[13] Morten H Christiansen and Simon Kirby. Language evolution: Consensus and controversies. Trends in cognitive sciences, 7(7):300-30) 30,7070000, 2003.
[14] Michael Clyne. Linguistic and sociolinguistic aspects of language contact, maintenance and loss. Mainte- nance and loss of minority languages, 1:17, 1992.
[15] Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra, Emergence of compositional language with deep generational transmission. arXiv preprint arXiv: 2019,
[16] Hannah Cornish, Rick Dale, Simon Kirby, and Morten H Christiansen. Sequence memory constraints give rise to language-like structure through iterated learning. PloS one, 12(1):0168533,, 2017.
[17] Vincent Crawford Joel Sohel. Strategic information transmission. Econometrica: Journal of the Econometric Society, pages 1431-1451, 1982.
[18] General Cultural Intelligence Team, Avishkar Bhoopchand, Bethanie Brownfield, Adrian Collister, Agustin Dal Lago, Ashley Edwards, Richard Everett, Alexandre Frechette, Yanko Gitahy Oliveira, Edward Hughes, et al. Learning robust real-time cultural transmission without human data. arXiv preprint 2rXiw:2203.00715. 2022.
[19] Jia Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[20] Roberto Dessi and Marco Baroni. Cans found to jump around more skillfully than rnns: Compositional generalization in seq2seq convolutional networks. arXiv preprini 19055
2019.
[21] Roberto Dessi, Eugene Kharitonov, and Baroni Marco. Interpretable agent communication from scratch (with a generic visual processor emerging on the side). Advances in Neural Information Processing Systems.
34, 2021.
11

---

[22] Nick C Ellis. The dynamics of second language emergence: Cycles of language use, language change, and language acquisition. The modern language journal, 92(2):232-249, 2008.
[23] Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in a multi-modal, multi-step referential game. In Proc. of International Conference on Learning Representations (ICI.R), 2018.
[24] Nicolas Fay and T Mark Ellison. The cultural evolution of human communication systems in different sized populations: usability trumps learnability. PloS one, 808),717811 2013.
[25] Bruno Galantucci and Simon Garrod. Experimental semiotics: a review. Frontiers in human neuroscience, 5: 2011.
[26] Lukas Galke, Yoav Ram, and Limor Raviv. Emergent communication for understanding human language evolution: What's 's missing? In Emergent Communication Workshop at ICLR, 2022.
[27] Rick Dale Gary Lupyan. Language structure is partly determined by social structure. PLDS ONE 5, 1, 2010.
[28] Laura Graesser, Kyunghyun Cho, and Douwe Kicla. Emergent linguistic phenomena in agent communication games. In Proc. of Empirical Methods in Natural Language Processing (EMNLP), 2019.
[29] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires. Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 21221221122288 2020.
[30] Shangmin Guo, Yi Ren, Kory Mathewson, Simon Kirby, Stefano Alhrecht, and Kenny Smith. Expressiv- ity of emergent language is trade-off between contextual complexity and unpredictability. In Proc. of International Conference on Leurning Representations (ICLR), 2021.
[31] Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3)333--466, 1990.
[32] Steven Harnad, Horst Steklis, and Jane Ed Lancaster. Origins and evolution of language and speech. Annals of the New York Academy of Sciences, 1976.
[33] Serhii Havrylov and Ivan Titow. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. Pruc. of Advances in Neural Information Processing Syxtems (NeurIPS). 2017.
[34] Geoffrey Hinton.
Learning translation invariant recognition in a massively parallel networks. In International Conference on Parallel Architectures and Languages Europe, pages 1-13. Springer, 1987.
[35] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):1735 1997.
[36] Simon Huttegger, Brian Skymms, Pierre Tarres, and Elliott Wagner. Some dynamics of signaling games. Proceedings of National Academy of Sciences, 11 IS(pppppem 3):10873- 2014.
[37] Aleksandra Kalinowska, Elnaz Davoodi, Florian Strub, Kory Mathewson, Todd Murphey, and Patrick Pilarski. Situated communication: A solution over-communication between artificial agents. In Emergent Commanication Workshop at ICLR 2022, 2022.
[38] Fugene Kharitonov and Marco Baroni, Emergent language generalization and acquisition speed are not. tied to compositionality. In Proc. of the BlackhoxNI.P Workshop on Analyzing and Interpreting Neural Networks for NLP. 2020.
[39] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Egg: toolkit for research emergence of language in games. In Proc. of Empirical Methods in Natural Language Processing (EMNLP), 2019.
[40] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimization in emergent languages. In Proc. of International Conference on Machine Learning (ICML), 2020.
[41] Jonyeon Kim and Alice Oh. Emergent communication under varying sizes and connectivities. Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2021.
[42] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of International Conference on Learning Representations (ICLR), 2015.
12

---

[43] Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learning model of the emergence of regularity and irregularity. IEEE Transactions on Evolatioary Computation. 5(2):102-110, 2001.
[44] Simon Kirby and James R Hurford. The emergence of linguistic structure: An overview of the iterated learning model. Simmlating the evolution of language, pages 121-147, 2002.
[45] Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. Compression and communication in the cultural evolution of linguistic structure. Cognition, 141:87- 102, 2015.
[46] Philipp Kochn. Statistical significance tests for machine translation evaluation. In Proc. of Empirical Methods in natural Longuage Processing (EMNLP), 2004.
[47] Stephen Kokoska and Daniel Zwillinger. CRC standard probability and statistics tables and formulae. Cre Press, 2000.
[48] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proc. of Empirical Methods in Natural Language Processing (EMNLP), 2017.
[49] Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
[50] Lukasz Kuciński, Tomasz Korbak, Pawel Kolodziej, and Piotr Milos. Catalytic role of noise and necessity of inductive biases the emergence of compositional communication. Proc. of Advances in Neural Information Procexsing Systems (NeurIPS), 2021.
[51] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.000 02419, 2020.
[52] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi- agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07188 2016.
[53] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In Proc. of International Conference on Learning Representations (ICLR), 2018.
[54] Vladimir Levenshtein et al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10:8, pages 707-710. Soviet Union, 1966.
[55] David Kellogg Lewis. Coneentin: Philosophical Study. Cambridge, MA, USA:
1969.
[56] Fushan Li and Michael Bowling. Base-of-teaching and language structure from emergent communication. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2019.
[57] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep leamingg face attributes in the wild. In Proc. of the International Conference on Computer Vision (ICCV), 2015.
[58] Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication. In Proc. of International Conference Learning Representations (ICLR), 2020.
[59] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, In Maria Florina Balcan and Kilian Q. Weinberger, editors, Procediiggs of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL httpp:://proceedinnnn mniha16. 6. html.
.wt.mmlr.eerr..... wwt..emer......eeeeeeeeeeeeeeeee press//48/
[60] Jesse Mu and Noah Goodman. Emergent communication of generalizations. Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2021.
[61] Michael Noukhovitch, Travis LaCroix, Angeliki Lazaridou, and Aaron Courville. Emergent communication under competition. In Proc. of International Conference Autonomous Agents and MultiAgent Systems (AAMAS). 2021.
[62] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1rXiv:1807.037488 2018.
[63] Marie Ossenkopf. Kevin Sebastian Luck, and Kory Wallace Mathewson. Which language evolves between heterogeneous agents? communicating movement instructions with widely different time scopes. In Emergent Communication Workshop at ICLR 2022, 2022.
13

---

[64] Limor Raviv. Antje Meyer, and Shiri Lev-Ari. Compositional structure can emerge without gen- erational transmission. Cognition. 182:151-164, 2019. ISSN 0010-0277. doi: https://doi.or//10. bol6ff.cognition2118.00000 $001002771830246..
www.ncedieecct..c.c..... cot////iiieee/////r//leeaiiiiii URL https//www..cienceiieet.com
[65] Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional languages emerge in neural iterated learning model. In Proc. of International Conference on Learning Representa- tions (ICLR), 2020.
[66] Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M. Dai, and Kyunghyun Cho. Capacity, band- width, and compositionality in emergent language learning. Proc. of Autonomous Agents and Multiagent Systems (AAMAS), 2020.
[67] Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. "Lazlmpa": Lazy and impatient neural agents learn to communicate efficiintly. In Proc. of the Conference on Computational Natural Language Learning (CoNLL), 2020.
[68] Mathieu Rita, Florian Strub, Jean-Bastien Grill, Olivier Pietquin, and Emmanuel Dupoux. On the role of population heterogeneity in emergent communication. In Proc. of International Conference on Learning Representations (ICLR), 2022.
[69] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Audrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision (ICV)., 115(3):211 1-22,, 2015.
[70] Brian Skyrms. Signals: Evolution, learning, and information. OUP Oxford, 2010.
[71] Agnieszka Stowik, Abhinav Gupta, William I. Hamilton, Mateja Jamnik, Sean Holden, and Christopher Pal. Exploring structural inductive biases emergent communication. arXiv preprint arXiv:2002.01333, 2002.
[72] Kenny Smith, Simon Kirby, and Henry Brighton. Iterated learning: A framework for the emergence of language. Artificial life, 94):37113366 2003.
[73] Matthew Spike, Kevin Stadler, Simon Kirby, and Kenny Smith. Minimal requirements for the emergence of learned signaling. Cognitive science, 41(3):623-658, 2017.
[74] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56); 1929-1958, 2014. URL http://jmlx www.oogg/g.....eo..oo pappps/ppes/ss// ppppers/v1/ssriaatteteeeeeeeeeee stsss/riistttaa1.. a html.
[75] Luc Steels. The synthetic modeling of language origins. Evolution of communication, 1(1):1 34, 1997.
[76] Richard S Sutton. David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proc. of Advances in Neural Information Processing Systems (NIPS), 2000.
[77] Zoltán Gendler Szabó. Compositionality, In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2020 edition, 2020.
[78] Mónica Tamariz and Simon Kirby. Culture: copying, compression, and conventionality. Cognitive science, 39(1):171-183, 2015.
[79] Simon W Townsend, Sabrina Engesser, Sabine Stoll, Klaus Zuberbühler, and Balthasar Bickel. Composi- tionality in animals and humans. PLoS Biology, 16(8):e2006425, 2018
[80] Pauli Virtanen, Ralf Gommers, Travis F. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas. Denis Laxalde. Josef Perktold. Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-274 2722, 2020. doi: 10. 000044411122---006888900
[81] Kyle Wagner, James A Reggia, Juan Uriagereka, and Gerald S Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-69, 2003.
14

---

[82] Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
[83] Alison Wray and George W. Grace. The consequences of talking strangers: Evolutionary corollaries of socio-cultural influences linguistic form. Lingua, 117(3):543 577878888-88888 2007. ISSN 0024-3841, The Evolution of Language.
[84] Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. Efficient compression in color naming and its evolution. Proc. of the National Academy Sciences, 115(31):9337-9442 7942, 2018. doi: 10.1073/ pnas. 18005100052111. URL https: /www. pnas org.oog//oooooo hoo//o///aba /abs//ab//aa a...ab...
10. 10.3/07/paas htt.://pp....1111111.111 1800521115.
[85] Lukasz Kuciński, Kolodziej, and Piotr Milos. Emergence compositional language communication through noisy channel. In Language in Reinforcement Learning (LaReL), ICML Workshop, 2020.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We mentioned the Partial and
Early Stopping listener training are costly opening a discussion on regularization methods (Section 5.2)
(c) Did you discuss any potential negative societal impacts of your work? [No] We run experiments that have no negative societal impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]] We took large part of our analytical part to detail the set of assumptions. Further discussions to more general cases are discussed in Appendix A
(b) Did you include complete proofs of all theoretical results? [Yes] If not in the main
paper, all derivations are brought in the Appendices
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.? [No] Code will be released upon deanonymization.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [No] Our work was carried out GPUs an institutional cluster. It is will be mentionned upon deanonymization Each experiment runs on a single V100-32G GPU
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The ImageNet and CelebA papers were cited
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "yMOmUKeSuff", "reviewer": "Reviewer_dJK1", "summary": "Teaching agents to play lewis signalling games is still one of the most common ways of studying emergent communication, and has historically been a test bed for our understanding of many EC dynamics.  In this paper, the authors propose to view the standard lewis game loss as a composition of two terms -- one that essentially promotes a unique message for each object, and one that promotes mutual understandability between speaker and listener.  It is this second term that is the main focus here, deemed as co-adaptation, as the authors explore the effect that artificially controlling this factor has on the learned languages.  They find that when trained in the de facto way, this term often overfits, and this overfitting has a detrimental effect on compositionality.\n", "questions": "\nI think the authors did a convincing job at showing how this co-adaptability effect gets to the heart of effective communication protocols failing to exhibit compositional structure and generalization, so as a take-away I think that is constant, but I'm unsure from a practical perspective the extent to which it matters to regularize the term explicitly vs. obtain regularization from aspects of the game/environment.\n\nTo this end, to what extent do these results persist as the task changes?  I was a bit disappointed that in scaling it up to the vision task, the size of the task was deliberately kept low.  Is it difficult to run on the full set?  Could the authors present some results more in-line with what other EC work is focusing on?  Perhaps it is not important to consider ad hoc regularization of the loss function if the environment is sufficiently rich or the task sufficiently hard.\n\nAlbeit the decomposition and analysis are novel, but can the authors clarify how their focus on the co-adaptability factor and early stopping regularization is different from the goal and solution of neural iterated learning and ease of teaching (both cited)?  It seems that these papers have identified this dynamic as a point where compositionality and generalization fail.", "limitations": "In terms of limitations, not sufficiently.  In terms of societal impact, it should not be applicable.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nFirst off, many thanks to the authors for presenting such a clear paper.  It was generally a joy to read.  The proposed decomposition makes a lot of sense, and maybe more than that just formalizing the loss in this way may help other researchers to structure the hypotheses of their own work.  In particular I liked the formalization of l^train_adapt and its role in the loss function since this gets quite close I think to how one should discuss generalization to novel object-attribute combinations through compositionality.\n\nIn terms of the results, the authors make a compelling case for the relationship between co-adaptation overfitting hindering compositionality in languages.  Both experiments with really hands-on early stopping regularization, and with more traditional ML regularization on the co-adaptation term, yield better generalization.\n\n\nWeaknesses:\n\nAlthough I think this paper is mostly ready for publication in its current state, and I do think it is beneficial to the EC community, I'm not sure the scope of the contribution is significant enough to warrant publication at NeurIPS, both in terms of what benefit it brings to the EC community, and in terms of it's interest to the larger set of research at the venue.  I think the contribution is quite niche (okay) and maybe rehashing a lot of what the EC community seems to know.\n\nDoes this work really shine new light on these issues, or does it just reinforce the lessons learned by the slew of other EC work?  I think it comes down to this: other (cited) EC work tends to also tackle the same problem of lack of compositionality stemming from co-adaptation, but focus more on how solutions may emerge from population dynamics and which sets of agents are communicating with each other.  These will ultimately find themselves expressed in gradients.  In this work, the authors cut straight through to the optimization itself.  In that sense, it helps point research towards more desirable properties in the loss function, but it is still up to the research community to find plausible ways for these pressures to originate.  Because presumably as a model of language development we don't want strong assumptions governing the loss itself over the actual agent/environment dynamics?  And the particular method of regularizing as applied here (primarily the early stopping) seems hacky and more for the purpose of probing than for simulation, whereas other proposed methods like neural iterated learning apply similar constraints in a more plausible way.\n\nSo in that extent, the closing argument (L342) seems somewhat backwards to me -- researchers seem aware that co-adaptation and lack of compositionality is undesirable, so they design realistic and intuitive pressures which ultimately have similar effects on the loss.  It is not surprising if a more hands-on approach goes directly to the loss to accomplish the same effect - as the authors say, \"remarkably, this emergent compositionality does not result from [things]...\" but it is instead created forcefully, and not in some natural way that would be remarkable IMO.\n\nIn conclusion, like a good literature review, I would like to see it published - it could be useful to EC researchers.  But I find it difficult to see this particular analysis as sufficiently novel or important to future EC research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "vkVJaC9ylrV", "reviewer": "Reviewer_VUeC", "summary": "The authors propose a framework for studying the generalization and compositionality of communication games playing Lewis signaling games. In short, there are two agents a speaker and a listener and \"the speaker observes a random state from its environment, e.g. an image, and sends a signal to the listener. The listener then undertakes an action based on this signal. Finally, both agents are equally rewarded based on the outcome of the listener’s action.\" (Line 24-26) The authors demonstrate how the agent's loss can be decomposed into an information term that quantifies the degree of ambiguity of the language protocol and a co-adaptation term that quantifies the gap between the listener and its optimum. \n\nThis observation allows studying the source of overfitting (or lack of generalization) in the communication agents especially when communication is in an unseen state from the training set. The main idea is to decrease the importance of the co-adaptation term in the loss so as to avoid the speaker to overfit the listener. For doing so, the authors experimented with using different training strategies Continuous listener, Partial listener, and Early stopping listener (Section 3.3). \n\nThe authors experimented using, the EGG toolkit with an environment in which every object has K = 6 attributes, each taking 10 different values, for a total of 1 million objects, and a non-overlapping train/val/test split is used. The experiments confirm the hypothesis of the loss decomposition by showing the importance of the co-adaptation term while evaluating an unseen set of objects (from the test set). Moreover, the authors show the effectiveness of the proposed method on more complex games CelebA and ImageNet, showing a much stronger generalization compared to a \"standard\" Continuous listener.", "questions": "I have no particular question. The paper is very clear and self-explanatory, I appreciated the read.", "limitations": "Not reported by the authors.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed loss decomposition is novel, to the best of my knowledge. The paper is formal and rigorous without being too verbose or hard to read in its presentation.\n- The experimental results are conclusive and well-conducted. Adding the final experiments on more complex tasks adds validation to the proposed hypothesis. \n- The paper is very well written and easy to follow. The content is well organized and the plots add to the presentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_pwOkH3cLuW", "reviewer": "Reviewer_mSoN", "summary": "The paper provides a theoretical analysis of the objective of Lewis communication games, that reveal that the objective can be decomposed into two components: (1) mutual-information component, that encourages the speaker to generate unambiguous (highly informative) messages on its input; and (2) adaptation component, that encourages the distribution of the speaker's posterior to match the distribution of the listener. The latter is directly optimized by both players, and is zeroed when the listener achieves an optimal distribution over objects given a message. Following this theoretical analysis, the authors hypothesize that overfitting issues in the adaptation game result in some previously-reported problems in the emergent communication, such as low accuracy and lack of compositionality.\n\nTo test this hypothesis, the co adaption term is estimated with a probe -- a listener that is trained to convergence over the frozen speaker. By modifying the weight of the probe's loss and the original loss in the reward of the speaker, the influence of mutual information component vs. the adaption component is tested. it is shown that (1) the adaptation term severely overfits, while the information term does not (2) by replacing the continuous co-training of the speaker and the listener with an alternate training where the listener is trained for N steps, it is possible to find a setting where the emergent language presents much superior properties. ", "questions": "- is it accurate to say that the proposed weighting scheme can be equivalently describes as follows? (1) we know the overall value of the loss L, and we can estimate the information term L_info (which doesn't require marginalization) with a probe; (2) Then the adaptation term is approximated as L_adapt = L - L_info; (3) Instead of training with L, we train with alpha*L_adapt + (1-alpha) * L_info.  In case this description is correct, I think it's a more intuitive way to describe the weighting experiment.\n\n- Why does the adaptation term strongly overfit, while the other term does not?\n\n- It would be beneficial to briefly explain why the speaker's posterior is the optimal distribution over objects for the listener. Intuitively, why can't the listener make up for some ambiguity in the speaker's messages by changing its distribution?", "limitations": "Yes. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the paper is an excellent example of a simple and highly elegant theoretical observation that leads to significant empirical gains. It is largely clearly written (apart from some issues, see below), and the empirical results (especially figure 2 and 3) are persuasive and surprising -- they point out to an inherent limitation that arises from optimization and may explain many of the previously reported issues with neural Lewis games. As such, the analysis in this paper has the potential of substantially influencing the study of emergent communication, and I strongly support its acceptance. \n\nWeaknesses: while the presentation is largely clear, it would be beneficial to improve, in the camera ready version, the presentation of the experiment that involves the weighting of the probe's loss and the original loss. In equations 10-11 the constant alpha is presented as a way to *upweight* the original loss, while, in practice, it is used to *downweight* the influence of the adaptation term. While the two are equivalent, these two conflicting views make it harder to understand the discussion in lines 237-244 (I think this paragraph in particular has to be improved. particularly the sentence at line 240). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept