Entropy Minimization In Emergent Languages
Authors: Eugene Kharitonov
Rahma Chaabouni
Diane Bouchacourt
Marco Baroni
Venue:  (2020)
Keywords: language emergence

Abstract: There is a growing interest in studying the languages emerging when neural agents are jointly trained to solve tasks requiring communication through a discrete channel.  We investigate here the information-theoretic complexity of such languages, focusing on the basic two-agent, one-exchange setup. We find that, under common training procedures, the emergent languages are subject to an entropy minimization pressure that has also been detected in human language, whereby the mutual information between the communicating agent's inputs and the messages is minimized, within the range afforded by the need for successful communication. This pressure is amplified as we increase communication channel discreteness. Further, we observe that stronger discrete-channel-driven entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks. We conclude by discussing the implications of our findings for the study of natural and artificial communication systems.

Under review as a conference paper at ICLR 2020
ENTROPY MINIMIZATION IN EMERGENT LANGUAGES
Anonymous authors
Paper under double-blind review
ABSTRACT
There is a growing interest in studying the languages emerging when neural agents are jointly trained to solve tasks requiring communication through a discrete chan- nel. We investigate here the coormatiionthhoreee complexity of such languages, focusing on the basic two-agent, one-exchange setup. We find that, under common training procedures, the emergent languages are subject to an entropy minimiza- tion pressure that has also been detected in human language, whereby the mutual information between the communicating agent's 's inputs and the messages is mini- mized, within the range afforded by the need for successful communication. This pressure is amplified as we increase communication channel discreteness. Further, we observe that stronger discrete-channel -h--nnn--n---eeeen entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks. We conclude by discussing the implications our findings for the study of natural and artificial communication systems.
INTRODUCTION
There has recently been much interest in the analysis of the communication systems arising when deep network agents that interact to accomplish goal allowed to exchange language-like discrete
messages (Lazaridou et al. 2016; Havrylov & Titov 2017 Choi et al. [018 Lazaridou et al.) [2018). Understanding the emergent protocol important we want to eventually develop agents capable of interacting with each other and with us through language (Mikolov et al. 2016) Cheealier- al. 2019). The pursuit might also provide comparative evidence about how core properties of human language itself have evolved (Kirby 2002 Hurford [2014] Graesser et al., 2019). While earlier studies reported ways in which deep agent protocols radically depart from human language (Kottur al. 2017 Bouchacourt & Baroni, 2018 Chaabouni et al.) 20199] Lowe et al., 2019), we show here that emergent communication shares an important property the latter, namely tendency towards entropy minimization.
Converging evidence indicates that efficiency pressures are at work in language and other biological communication systems (Ferrer Cancho et al. 2013 Gibson et al. 2019). One particular aspect of communicative efficiency, that has been robustly observed across many semantic domains, is a tendency to minimize lexicon entropy, to the extent allowed by the counteracting need for accuracy (Zaslavsky et al. 2018; 2019). For example, while most languages distinguish grandmothers from grandfathers, very few have separate words for mother- and father-side grandmothers, as the latter distinction would make communication only slightly more accurate at the cost of an increase in lexicon complexity (Kemp & Regier, 2012). We show here, two separate games designed to precisely measure such property, that the protocol evolved by interacting deep agents is subject to the same complexity minimization pressure.
Entropy minimization in natural language has been connected to the Information Bottleneck princi- ple (Tisbby et al. 1999). In turn, complexity reduction due to the Information Bottleneck provides beneficial regularization effect on the leamned representations (Fischer 2019 Alemi et al. 2016 Achille & Soatto, 2018: 8a b). It is difficult to experimentally verify the presence of such effect in look for it in emergent language simulations. We confirm that, when relaxing channel discreteness, the entropy minimization property no longer holds, and the system becomes less robust against overfitting and adversarial noise.
human languages, but

---

Under review as a conference paper at ICLR 2020
2 GENERAL FRAMEWORK
We establish our results in the context of signaling games (Lewis) 1969). as introduced to the current language emergence literature by] Lazaridou et al. (2016) and adopted in several later studies (Havrylov & Titov 2017; Bouchacourt & Baroni 2018 Lazaridou et al. 20188) There are two agents, Sender and Receiver, provided with individual inputs at the beginning of each episode. Sender sends single message to Receiver, and Receiver has to perform an action based on its own input and the received message. Importantly, there is no direct supervision on the message protocol. We consider agents that are deterministic functions of their inputs (after training).
As an example, consider the task of communicating n-bit number, sampled uniformly at random from 0...2 - 1. The full number is shown to Sender, and its & (0 I k S n)) least-significant bits are also revealed to Receiver. Receiver has to output the full number, based on the message from Sender and its own input. Would the Sender transmit the entire number through its message? In this case, the protocol would be "complex," encoding n bits. Alternatively, Sender could only encode the bits that Receiver does not know, and let Receiver fill in the rest by itself. This emergent protocol would be "simple." " encoding less information about the number. We find experimentally that, once the agents successfully trained to jointly solve the task, the emergent protocol minimizes the entropy of the messages or, equivalently in our setup, the mutual information between Sender's 's input and messages. In other words, the agents consistently approximate the simplest successful protocol (in the current example, the one transmitting ~ n k bits).
After training, we can connect the entropies of Sender and Receiver inputs i, and é, messages m S(i.). Receiver's 's output (the chosen action) o R(m, iy), and ground-truth outputs I by using standard inequalities (Cover Thomas 2012):
(1) (Note that, since agents are deterministic after training, H(m) = I(i.; We can then use these quantities interchangeably.) Our empirical measurements indicate that the entropy of the messages m in the emergent protocol tends to approach the lower bound: (mm + (I),,,, even if the upper-bound H (iz) is far.
H(a.)= 2 (SS(i))) (m) 2 Hm)),,) 2 H(R(m, (.) H(oli,) ~ H((,))
In our experiments, we observe that when the amount of information that Receiver needs is reduced, without changing other parameters, the emergent protocol becomes simpler (lower entropy). In other words, the emergent protocol adapts to minimize the information that passes through it.
We will release the code for our experiments upon acceptance.
3 METHODOLOGGY
3.1 GAMES
We study two signaling games. In Guess Number, the agents are trained to recover an integer- representing vector with uniform Berroull components. This simple setup gives us full control over the amount of information needed to solve the task. The second game, Image Classification, uses more naturalistic data, as the agents are jointly trained to classify pairs of hand-written MNIST digits (LeCun et al. 1998b).
Guess Number We draw an 8-bit integer 2, 5= S 255 uniformly at random, by sampling its 8 bits independently from the uniform Bernoulli distribution. All bits are revealed to Sender as 8-dimensional binary vector ig. The last bits are revealed Receiver (0 < k < 8) as its input é, Sender outputs a single-symbol message m to Receiver. In turn, Receiver outputs a vector that recovers all the bits of and should be equal to i..
In this game, Sender has linear layer that maps the input vector i, to a hidden representation of size 10, followed by a leaky ReLU activation. Next a linear layer followed by a softmax over the vocabulary. Receiver linearly maps both its input i, and the message to 10-dimensional vectors, concatenates them, applies a fully connected layer with output size 20, followed by a leaky ReLU. Finally, another linear layer and sigmoid nonlinearity are applied. When training with REINFORCE and the Stochastic Computation graph approach (see Section 3.2), we increase the hidden layer sizes threefold, as this leads to more robust convergence.
2

---

Under review as a conference paper at ICLR 2020
Image Classification In this game, the agents are jointly trained to classify 28x56 images of two MNIST digits, stacked side-by-side (more details in Appendix). Unlike Guess Number, Receiver has no side imput. Instead, control the informational complexity of Receiver's task by controlling the size of its output space, i.e., the number of labels we assign to the images. To do so, we group all two-digit sequences 00..99 into N/ {2, 4, 10, 20, 25, 50, 100} equally-sized classes.
In Sender, input images are embedded a LeNet-1 instance (LeCun et al. 1990) into 400-dimensional vectors. These embedded vectors are passed to a fully connected layer, followed by a softmax selecting a vocabulary symbol. Receiver embeds the received messages into 400-dimensional vectors, passed to a fully connected layer with softmax activation returning the class probabilities.
We report hyperparameter grids in Appendix. In the following experiments, we fix vocabulary to 1024 symbols (experiments with other vocabulary sizes, multi-symbol messages, and larger architectures are reported in Appendix). No parts of the agents are pre-trained or shared. The optimized loss depends on the gradient estimation method used (see Section 3.2). We denote C(o, l). and it is a function of Receiver's s output and the ground-truth output l. When training in Guess Number with REINFORCE, we use a 0/1 loss: the agents get 0 only if all bits of z were correctly recovered. When training with Gumbel-Softmax relaxation or the Stochastic Computation Graph approach, we use binary cross-entropy (Guess Number) and negative -ikeliillhood (Image Classification).
3.2 TRAINING WITH DISCRETE CHANNEL
Training to communicate with discrete messages is non-trivial, as we cannot back-propagate through the messages. Current language emergence work mostly uses Gumbel-Softmax relaxation (e.g.
(Havrylov & Titov 2017)) or REINFORCE (e.g. (Lazaridou al., 2016) to get gradient estimates. We also explore the Stochastic Computation Graph optimization approach. We plug the obtained gradient estimates into the Adam optimizer (Kingma Ba 2014].
Gumbel-Softmax relaxation Samples from the Gumbel-Softmax (Maddison et al. 2016 Jang et al. 2016) distribution (a) are reperameterizable, hence allow gradient- based training, and (b) approximate samples from the corresponding Categorical distribution. To get sample that approximates an n--imennionnl Categorical distribution with probabilities pi, we draw n i.i.d. samples gi from Gu.bel(0.1) and use them to calculate vector y with components:
exp [(g, + log pt)/T]
Vi Σ, exp [gj + log P5)/T]
(2)
where 7 is the temperature hyperparameter. As T tends to O. the samples y get closer to one-hot samples; as J ~ +00, the components y; become uniform. During training, we use these relaxed samples as messages from Sender, making the entire Sender/Receiver setup differentiable.
REINFORCE by Williams (1992) is a standard reinforcement learning algorithm. our setup, estimates the gradient of the expectation of the loss L(o, 1) w.r.t. the parameter vector f as follows:
Ei Bm Stt,) (t.)), o. [[C(o; 1) - V log Po (m..o)
(3)
The expectations estimated by sampling m from Sender and, after that, sampling from Receiver. We use the running mean baseline b (Greensmith et al. 2004 Williams) 1992) as a control variate.
We adopt the common trick to add entropy regularization term (Williams & Peng 1991 Mnih
et al. 2016) that favors higher entropy. We impose entropy regularization on the outputs of the agents with coefficients A, (Sender) and X, (Receiver).
Stochastic Computation Graph In our setup, the gradient estimate approach of Schulman et al. (2015) reduces to computing the gradient of the following surrogate function:
F a E (t,, [L(o: + stop_gradienl (L(o: t) b) log (om))
Here, we do not sample Receiver actions: Its parameter gradients are obtained with standard back- propagation (the first term in Eq. 4 Sender's messages are sampled, and its gradient calculated akin to REINFORCE (the second term in Eq. 4). As in REINFORCE, we apply entropy-favoring regularization Sender's output (with coefficient X₂) and use the mean baseline b.
Role of entropy regularization As we mentioned above, when training with REINFORCE and Stochastic Computation Graph, we include (standard) entropy regularization term in the loss which
3

---

Under review as a conference paper at ICLR 2020
Gumbel Saftmax
Binary digits hidden
Binary digits hiddon
All three training approaches.
Training with Gumhel-Softman relaxation.
Him)
J. 5.01
Huml
A.
4 0.025
A. 0.05
3.4 0.03
0.075
0.075
0.1
Binary digits hidden
Binary digits hidden
(c) Training with Stochastic Compu- tation Graph.
(d) Training with REINFORCE.
Figure 1: Guess Number: entropy of the messages m. Shaded regions mark standard deviation.
explicitly maximizes entropy of Sender's output. Clearly. this term is at odds with the entropy minimization effect we observe. In our experiments, we found that high values of X prevent communication success; the other hand, small non-zero is crucial for successful training. In Section 4 we investigate the effect of X on entropy minimization.
3.3 EXPERIMENTAL PROTOCOL
In Guess Number, we use all 2 possible inputs for training, early stopping and analysis. In Image Classification, we train on random image pairs from the MNIST training data, and use image pairs from the MNIST held-out set for validation. We select the runs that achieved high level of performance (training accuracy above 0.99 for Guess Number and validation accuracy above 0.98 for Image Classification), thus studying typical agent behavior provided they succeeded the game.
At test time, we select the Sender's message symbol greedily, hence the messages are discrete and Sender represents (deterministic) function S of its input é, S(i). Calculating the entropy H (m) of the distribution of discrete messages m straightforward. In Guess Number, we enumerate all 256 possible values of inputs, the messages from Sender and calculate entropy H H(m). For Image Classification, we sample image pairs from the MNIST hold-out set.
The upper bound H (m) is follow: H 8 bits (bounded by H(i.)) in Guess Number, and Hmaz = 10 bits (bounded by vocabulary size) in Image Classification. Its lower bound is equal bits for Guess number. Image Classification, communication can only succeed if H (m) is not less than H (t), i.c., Hmin H (t) log, Ni, with Nj the number of equally-sized classes we split the images into.
Hmin ((),,,)
4 EXPERIMENTS
4.1 ENTROPY MINIMIZATION
Guess Number In Figure I the horizontal axes span the number of bits of & that Receiver lacks, 8 l.. The vertical axis reports the information content of the protocol, measured by messages entropy
4

---

Under review as a conference paper at ICLR 2020
Hew
Heb
3
2
log
log.M
Successful pooled together.
(b) Successful runs grouped by tem-
perature.
Figure 2: Image Classification: entropy of the messages m in function of log number of target classes, Ni. Shaded regions mark standard deviation.
(m). Each integer on the horizontal axis corresponds to game configuration, and for each such configuration we aggregate multiple (successful) runs with different hyperparameters and random seeds. H indicates the minimal amount of bits Sender has to send in particular configuration for the task to be solvable. The upper bound (not shown) is maaz = 8 bits.
Consider first the configurations where Receiver's input is insufficient to answer correctly (at least one binary digit hidden, k S 7). From Figure Ia we observe that the transmitted information is strictly monotonically increasing with the number of binary digits hidden from Receiver. Thus, Sender sees the very same input in all configurations, a more nuanced protocol is only developed when it is necessary. Moreover, the entropy H (m) (equivalently: the transmitted information) stays close to the lower bound. This entropy minimization property holds for all the considered training approaches across all configurations.
Consider next the configuration where Receiver is getting the whole integer = as its input (k = 8, the leftmost configuration in Figure corresponding one would expect that the protocol would approach zero entropy in this case (as no information needs be transmitted). However, the measurements indicate that the protocol is encoding considerably more information. It turns out that this information is entirely ignored by Receiver. To demonstrate this, we fed all possible distinct inputs to Sender, retrieved the corresponding messages, and shuffled them to destroy any information about the inputs they might carry. The shuffled messages were then passed to Receiver alongside with its own (un-shuffled) inputs. The overall performance was not affected by this manipulation, confirming the hypothesis that Receiver ignores messages. We conclude that in this case there is no apparent entropy minimization pressure on Sender simply because there is no communication. The full experiment is reported in Appendix.
x axis). Based on the observations above,
We further consider the effect of various hyperparameters. In Figure 1b
we split the results obtained with -ooftelSSfttax by relaxation temperature. As discussed in Section 3.2 lower temperatures more closely approximate discrete communication, hence providing a convenient control the level of discreteness imposed during training (recall that at time we select the symbol greedily). The figure shows that lower temperatures consistently lead to lower H (m) values. This implies that, as increase the "level of discreteness" at training, get stronger entropy minimization pressures.
In Figures Ic & 1d we report H (m) when training with Stochastic Graph Optimization and RE- INFORCE across degrees of entropy regularization. We report curves corresponding to A values which converged in more than three configurations. With REINFORCE, we see weak tendency for a higher X, to trigger higher entropy the protocol (only violated at X = 0.5). However, message entropy stays generally close to the lower bound even in presence of strong exploration, which favors higher entropy in Sender's output distribution.
Image Classification As the models are more complex, only had consistent success when training with Soffhaa- Figure 2a we aggregate all successful runs. The information encoded by the protocol grows as Receiver's 's output requires more information. However, in all configurations, the transmitted information stays well below the 10-bit upper bound and tends to be close to H
'The parameter Xy that controls the strength of Receiver's entropy regularization turned out not play role in the observed effect.
5

---

Under review as a conference paper at ICLR 2020
1.0
U5. r=1.0
55, c:10.
Train accuracy
Test accuracy
SM, 1-1.0
Unser
0.6
65. 1=1.0
0.6
1=10.0
0.4
0.4
0.0
0.0
100
150
200
50
100
150
200
Epochs
Epochs
All train labels shuffled.
(h) All train labels shuffled.
1.0
1.0
0.8
0.8
Train accurac)
Test accurrccy
0.4
0.4
65, t=1.0
55, c=1.0
0.2
65, t-10.0
02
65. t-10.0
Lineer
Unwer
0.0
100 Epochs
200
0.0
200
Epochs
(c) Half of train labels are shuffled.
(d) Half of train labels are shuffled.
bafore c=1.0 after
0.8
0.8
Train accuracy
Train accuracy
0.6
0.6
0.4
0.2
18.0, refore channe
0.2
T- 10.0, after channel
0.0
0.0
400
600
800
200
400
1000
(c) All labels shuffled; Additional layer before channel vs. after channel
(f) All labels shuffled; Additional layer before channel vs. after channel
Figure 3: Learning in presence of random labels. GS (SM) indicates models trained with Gumbel- Softmax (Softmax) channel. Linear are models with the channel removed.
A natural interpretation is that Sender prefers to take charge of image classification and directly pass information about the output label, rather than sending along a presumably more information heavy description of the input. In Figure 26 we split the runs by temperature. Again, temperatures consistently lead to stronger entropy minimization pressures.
that lower
Summarizing, when communicating through discrete channel, there consistent pressure for the emergent protocol to encode as little information as necessary. This holds across games, training methods and hyperparameters. When training with Gumbel-Softmax, temperature controls the strength of this pressure, confirming the relation between entropy minimization and discreteness.
4.2 REPRESENTATION DISCRETENESS AND ROBUSTNESS
The entropy minimization effect we established in Section 4.1 indicates that a discrete representation will only store as much information necessary to solve the task. This emergent behavior respects the "information bottleneck" principle (Tishby et al. 1999; Achille & Soatto, 2018a). The fact -imen time temperatures in Gumbel- discreteness and tighter bottleneck (see Section 4.1) makes us further conjecture that discreteness is causally connected to the emergent bottleneck.
that lower training-
optimization correlate with both higher

---

Under review as a conference paper at ICLR 2020
Interestingly, the same conoommtioo--otleeneeee principle has also been claimed to govern entropy minimization in natural language (Zaslavsky et al. |018) 2019). The in neural agents and natural language might be due to the same cause, namely discreteness of the communication. Further, we hypothesize that the emergent discrete information-bottlenekk might have useful properties, since existing (continuous) architectures that explicitly impose bottleneck pressure to provide form of beneficial regularization are more robust to overfitting (Fischer) 2019) and adversarial attacks (Alemi et al, 2016; Fischer 2019). We test here whether the expected regularization properties also emerge in our computational simulations (without any explicit pressure imposed through the cost function), and whether they are correlated with the degree of discreteness of the communication channel. If this connection exists, this also suggests that discreteness might be "beneficial" to human languages for the same reasons.
citraaiion-ottleneckk effects
To assess our hypotheses, we consider the Image Classification game (N₂ 10) in presence of randomly-shuffled training labels (the test set is untouched) (Zhang et al. 2016). This task allows us to explore whether the discrete communication bottleneck is associated to robustness overfitting, and whether the latter depends on the level of discreteness (controlled by the temperature T of the Gumbel-Softmas relaxation) H
We use the same architecture as before. The agents are trained with Humbel-Softmas relaxation. However, for practicality, we do not switch to fully discrete communication at test time, only removing the noise component, thus effectively reducing Sender's ' output to softmax with temperature. We refer this architecture GS.
We also consider two baseline architectures without relaxed discrete channel. In Linear, the fully connected output layer of Sender is directly connected the linear embedding input of Receiver. Softmax (SM) places a softmax activation (with temperature) after Sender's output layer and passes the result to Receiver. At test time, SM coincides with GS with the same temperature, but there was no discrete-sampling approximation during SM training.
We vary temperature and proportion of training examples with shuffled labels. We use temperatures 7 = 1.0 and r = 10.0 (the agents reach test accuracy of 0.98 when trained with these temperatures on the original training set). SM with 7 1.0 and T 10.0 behave similarly, hence we only report SM with 7 1.0.
In Figure Sa we report training accuracy when all labels are shuffled. Linear and SM fit the random labels almost perfectly within the first 150 epochs. With r = 10.0, GS achieves 0.8 accuracy within 200 epochs. When GS with r 1.0 is considered, the agents only start to improve over random guessing after 150 epochs, and accuracy is well below 0.2 after 200 epochs. As expected, test set performance is at chance level (Figure 36). In the next experiment, we shuffle labels for a randomly selected half of the training instances. Train and test accuracies are shown in Figures Bc and 3d respectively. All models initially fit the true-label examples (train accuracy Rs 0.5, test accuracy R 0.97). With more training, the baselines and GS with labeled examples, too: train accuracy grows, while test accuracy falls. In contrast, GS with 7 = 1.0 does not fit random labels, and its test accuracy stays high. Note that SM patterns with Linear and high-temperature GS, showing that the training-time discretization noise in GS is instrumental for robustness to over-fitting.
10.0 start (over)fitting the randomly
We interpret the results as follows. To fully exploit their joint capacity for "successful" over-fitting, the agents need to coordinate label memorization. This requires passing large amounts of information through the channel. With a low temperature (more closely approximating a discrete channel), this is hard, due to stronger entropy minimization pressure. To test the hypothesis, we ran an experiment where all labels are shuffled and layer of size 400x400 is either added to Sender (just before the channel) or to Receiver (just after the channel). We predict that, with higher r (less discrete, less entropy minimization pressure), the training curves will be close, as the extra capacity can be used for memorization equally easy in both cases. With lower т (better discrete approximation, more pressure), the accuracy curves will be more distant, as the extra capacity can only be successfully exploited for memorization when placed before the channel. Figures Be & 3f borne out the prediction.
Thhe appendix reports experiments showing that agents trained with a low-temperature channel also display increased robustmess against adversarial attacks.
7

---

Under review as a conference paper at ICLR 2020
These experiments showed that increased channel discreteness makes it harder to pass large amounts of information through, and leads to increased robustness against overfitting. This supports our hypotheses that discreteness brings about bottleneck that in turn has some beneficial properties, which might ultimately provide a motivation for why an emergent communication system should evolve towards discreteness.
RELATED WORK
We briefly reviewed studies of emergent deep agent communication and entropy minimization in human language in the introduction. We are not aware of carlier work that looks for this property in emergent communication, although Evtimova al. (2018) used information theory to study protocol development during learning. and, closer to us, Kágebäck et al. (2018) studied the effect of explicitly adding complexity minimization term the cost function on an emergent color-naming system.
Discrete representations are explored in many places (e.g., van den Oord et al., 2017 Jang et al. 2016, Rolfe 2016). However, these works focus ways learn discrete representations, rather than analyzing the properties of representations that are independently emerging on the side. Furthermore, our study also extends to the agents communicating with variable-lengee messages, produced and consumed by GRU (Cho et al. 2014) and Transformer (Vaswani et al. 2017) cells (see Appendix C.3). The sequential setup is specific to language, clearly distinguished from the settings studied in generic reareeerepresnntation work.
Other studies, inspired by the informational bottleneck principle, control the complexity of neural representations by regulating their information content (Strouse & Schwab, 2017 Fischer 2019) Alemi al.] 2016) Achille & Soatto, 2018a b). While they externally impose the bottleneck, we observe that it an intrinsic feature when learning to communicate through discrete channel.
6 DISCUSSION
Entropy minimization is pervasive in human language, where it constitutes specific facet of the more general pressure towards communication efficiency. We found that the same property consistently characterizes the protocol emerging in simulations where two neural networks learn to solve task jointly through a discrete communication code.
In comparative perspective, our results suggest that entropy minimization is general property of discrete communication systems, independent of specific biological constraints humans are subject to. In particular, our analysis tentatively establishes a link between this property and the inherent difficulty of encoding information in discrete form (cf. the effect of adding a layer before or after the communication bottleneck in the overfitting experiment above).
Exploring entropy minimization in computational simulations provides a flexibility we lack when studying humans. For example, uncovered here initial evidence that the communication bottleneck is acting as a good regularizer, making the joint agent system more robust to noise. This leads to an intriguing conjecture
the origin of language. Its discrete nature is often traced back the fact that it allows us to produce an infinite number of expressions by combining a finite set of primitives (e.g.. Berwick & Chomsky 2016). However, an infinite number of concepts could have provided the initial pressure to develop a discrete code. More probably, once such code independently emerged, it laid the conditions develop an infinitely expressive language (Bickerton) 2014 Collier et al. 2014). Our work suggests that, because of its inherent regularizing effect, discrete coding is advantageous already when communication is about a limited number of concepts, providing an alternative explanation for its origin.
far from clear that the need communicate
In the future, we would like to study more continuous domains, such as color maps, where perfect accuracy is not easily attainable, nor desirable. Will the networks find an accuracy/complexity trade-off similar to those attested in human languages? Will other core language properties claimed to be related to this trade-off, such as Zipfian frequency distributions (Ferrer i Cancho & Diaz-Guilera 2007), concurrently emerge? We would also like to compare the performance of human subjects equipped with novel continuous vs. discrete communication protocols, adopting the methods of experimental semiotics (Galantucci 2009). We expect discrete protocols to favor generalization and robustness.
8

---

Under review as a conference paper at ICLR 2020
Our results have implications for the efforts to evolve agents interacting with each other and with humans through discrete channel. First, because of entropy minimization, we should not expect the agents to develop richer protocol than the simplest one that will ensure accurate communication. For example, |Bouchacourt & Baroni (2018) found that agents trained to discriminate pairs natural images depicting instances of about 500 high-level categories, such cats and dogs, developed lexicon that does not denote such categories, but low-level properties of the image themselves. This makes sense from an entropy- minimiziiinniiiiiinnnn perspective, as talking about the 500 high-level categories demands log, 500 bits of information, whereas many low-level strategies (e.g., discriminating average pixel intensity in the images) will only require transmitting a few bits. To have agents developing rich linguistic protocols, we must face them with varied challenges that truly demand them.
Second, the focus on a discrete protocol typically motivated by the goal to develop machines eventually able to communicate with humans. Indeed, discrete messages are not required in multi- agent scenarios where no human in the loop is foreseen (Sukhbaatar al.) 2016). Our results suggest that, long before agents reach the level of complexity necessary to converse with humans, there are independent reasons to encourage discreteness, as it provides a source of robustness in a noisy world. An exciting direction for future applied work will be to test, in more practical settings, the effectiveness of discrete communication as a general form of representation learning.
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE TPAMI, (2112)):897-29055 2018a.
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):144-1190, 2018b.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
Robert Berwick and Noam Chomsky. Why Only Us: Language and Evolution. MIT Press, Cambridge, MA, 2016.
Derek Bickerton. More than Nature Needs: Language, Mind, and Evolution. Harvard University Press, Cambridge, MA, 2014.
Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an emergent language game. In EMNLP, 2018.
Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv: 1607.0460,. 2016.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti enti-efficient encoding in emergent communication. In Proceedings of NeurlPS, Vancouver, Canada, 2019. In
press.
Maxime Chevalier-ooisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby AI: platform to study the sample efficiency of grounded language learning. In Proceedings of ICLR, New Orleans. LA, 2019. Published online: net group -idd-id-iii ICLR. CCC.C//.cc//011/ooeeeeeeee
/ttpp::/opperevieee
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint urXiv: /406. 1078, 2014.
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Compositional obverter communication learning from raw visual input. arXiv preprint 1804.0800 2018.
Katie Collier, Balthasar Bickel, Carel van Schaik, Marta Manser, and Simon Townsend. Language evolution: Syntax before phonology? Proceedings of the Royal Society B: Biological Sciences, 281(1788):177, 2014.
Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2012.

---

Under review as a conference paper at ICLR 2020
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in a modal, multi-step referential game. In Proceedings of ICLR Conference Track, Vancou- ver, Canada, 2018. Published online: https: http::::///ppeeeiiieeee net wwtp://wwweetgrooooooooooo. cc/ 201 8 ////n//oooeeeeceeee
Ramon Ferrer Cancho and Albert Díaz-Guilera. The global minima of the communicative energy of natural communication systems. Journal of Statistical Mechanics: Theory and Experiment, 2007 (06):P06009, 2007.
Ramon Ferrer Cancho, Antoni Hernández-Fermánddzz, David Lusseau, Govindasamy Agoramoorthy, Minna Hsu, and Stuart Semple. Compression universal principle of animal behavior. Cognitive Science, 37(8):1565-1555 2013.
Ian Fischer. The conditional entropy bottleneck, 2019. URL https::/openreevierg forum?id=rkkoxhaa
Bruno Galantucei. Experimental semiotics: new approach for studying communication as form of joint action. Topics in Cognitive Science, 1(2):393-410, 2009.
Edward Gibson, Richard Futrell Steven Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. How efficiency shapes human language. Trends Cognitive Science, 2019, In press.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena multi-agent 1911 08706 2019.
communication games.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. JMIRR, 147111344 2004.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In NIPS, 2017.
James Hurford. The Origins of Language. Oxford University Press, Oxford, UK, 2014.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-SSoftax, arXiv
preprint arXiv: 1.0/ 144, 2016.
Mikael Kågebäck, Devdatt Dubhashi, and Asad Sayeed. DeepColor: Reinforcement learning optimizes information efficiency and well-formmennes in color name partitioning. In Proceedings of CogSci, pp. 1895-199000 1900, Austin, TX, 2018.
Charles Kemp and Terry Regier. Kinship categories across languages reflect general communicative principles. Science, 136(6084):1049-110 2012.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 141.691412.6980, 2014.
Simon Kirby. Natural language from artificial life. Artificial life,
Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Natural language does not comerge'naturally in multi-agent dialog. arXiv preprint arXiv: (ee::776.008511 2017.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint
2016.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of lin- guistic communication from referential games with symbolic and pixel input. arXiv preprint arXiv:1804.03984, 2018
Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a proppagatinn network. In NIPS, 1990.
10

---

Under review as a conference paper at ICLR 2020
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2 -2224,, 1998a.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 96(11):2278-2322 1998b.
David Lewis. Convention harvard university press. Cambridge, MA, 1969.
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. In Proceedings of AAMAS, pp. 693-701, Montreal, Canada, 2019.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: continuous relaxation of discrete random variables. arXiv preprint arXiv:6611.0071., 2016.
Tomas Mikolov, Armand Joulin, and Marco Baroni. A roadmap towards machine intelligence. In International Conference on Intelligent Text Processing and Computational Linguistics, pp. 29-61. Springer, 2016.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016.
Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint arXiv: 2016.
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. In NIPS, 2015.
DJ Strouse and David J Schwab. The deterministic information bottleneck. Neural computation, 29 (6):1611-133) 2017.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NIPS. 2016.
N. Tishby, Pereira, and W. Bialek. The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing. University of Illinois Press, 1999.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NIPS, pp. Long Beach, CA, 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 1992.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. Efficient compression color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31);7937-7942, 2018.
Noga Zaslavsky, Terry Regier, Naftali Tishby, and Charles Kemp. Semantic categories of artifacts and animals reflect efficient coding. In Proceedings of CogSci, pp. 1254-1260, Montreal, Canada, 2019.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 2016.
11

---

Under review as a conference paper at ICLR 2020
1.0
1.0
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.00
0.05
0:10
0.15
0.20
0.25
0.00
0.05
0.10
0.15
0.20
0.25
Perturbation norm
Perturbation norm,
Robustness temp. T.
(b) Comparison the baselines.
Figure 4: Robustness to adversarial examples: higher accuracy given fixed implies more robustness.
1.0
Gumbel Softmax
computation
0.8
REINFORCE
upper bound
Accuracy
0.6
0.4
0.2
0.0
Binary digits hidden
Figure 5: Guess Number: Receiver's dependence on messages, measured as performance drop under message intervention.
A ROBUSTNESS TO ADVERSARIAL ATTACKS
In this Section, we study robustness of the agents equipped with relaxed discrete channel against adversarial attacks. We use the same architectures as Section the main paper. Specifically, by GS we indicate the architecture where agents are trained with Gumbel-Softmax relaxation, which at test-time replaced by (noiseless) softmax with the same temperature. SM is an architecture where the communication channel is replaced by a Softmax layer with temperature. The Linear baseline has no "channel": the output of Sender is directly plugged as input to the Receiver.
We train agents with different random seeds and implement white-box attacks on the trained models, varying temperature 7 and the allowed perturbation norm, e. We use the standard Fast Gradient Sign Method (FGSM) of Goodfellow et al. (2014). The original image i, is perturbed to i along the direction that maximizes the loss of Receiver's s output o R(S((.)) w.r.t ground-trus -tuut class l:
f = clip [i, + sign (V, ((),
(5)
where controls the Tox norm of the perturbation. Under an attack with a fixed e, more robust method would have smaller accuracy drop. To avoid numerical stability issues akin those reported done in 64-bit floats.
by Carlini & Wagner (0166), all computations
As earlier in Section 4.2 we observed that SM behaves similarly with different temperatures (we experimented with E 10 1.0, 10.0)), report only results with - = 1.0. Figure 4a shows that, as the relaxation temperature decreases, the accuracy drop also decreases. The highest robustmess achieved with - 0.1. Comparison with the baselines (Figure 4b) confirms that relaxed discrete training with 5 = 0.1 improves robustness. However, this robustness comes at the cost of harder training: 2 out of 5 random seeds did not reach the desired performance level (0.98) after 200 epochs.
12

---

Under review as a conference paper at ICLR 2020
B How MUCH DOES RECEIVER RELY ON MESSAGES IN GUESS NUMBER?
We supplement the experiments of Section 3 of the main text by studying the degree to which Receiver relies on messages in Guess Number. In particular, we show that when Receiver has the full input (i, = i,), it ignores the messages.
We measure the degree to which Receiver relies on the messages from Sender by constructing a setup where we break communication, but still let Receiver rely on its own input. More precisely, we first enumerate all test inputs for Sender i, and Receiver ,, We obtain messages that correspond to Sender's s inputs, and shuffle them. Next, we feed the shuffled messages alongside Receiver's 's own (un-shuffled) inputs and compute accuracy, as a measure of Receiver's 's dependence on the messages. This procedure preserves the marginal distribution of the Receiver input messages, but destroys all the information Sender transmits.
Without messages, Receiver, given k input bits, can only reach accuracy of 2 k. In Figure report results aggregated by training method. Receiver is extremely close to the accuracy's 's higher bound in all configurations. Moreover, when Receiver gets the entire input, the drop in accuracy after shuffling is tiny, proving that Receiver's reliance on the message is minimal in that setting.
C INFLUENCE OF ARCHITECTURE CHOICES
C.I DOES VOCABULARY SIZE AFFECT THE RESULTS?
We repeat the same experiments in Section 3 of the main text while varying vocabulary size. Note that, to make Guess Number solvable across each configuration, the vocabulary has to contain at least 256 symbols. Similarly, for Image Classification, vocabulary size must be of at least 100. We tried vocabulary sizes of 256, 1024, 4096 for Guess Number, and 512, 1024, 2048 for Image Classification. The results reported in Figures 6 (Guess Number) and (Image Classification). We observe that there is little qualitative variation over vocabulary size, hence the conclusions we had in Section 3 of the main paper robust to variations of this parameter.
C.2 DOES RECEIVER'S 'S CAPACITY AFFECT THE RESULTS?
One potential confounding variable is the capacity of Receiver. Indeed, if Receiver is very simple, then, for the task to be solved, Sender would have to calculate the answer itself and feed it to Receiver. To investigate this, we repeat the Image Classification experiment from Section of the main paper while controlling the power of Receiver's architecture: we put two additional fully-connected 400x400 hidden layers between the input embedding and the output layer, while Section 4 Receiver had single hidden layer.
In Figure g we compare the results obtained with these two variations of Receiver. The reported entropy minimization effect holds; even in presence of additional layers, the entropy of messages H (m) is far from the bound H Hmaz = 10 bits and closely follows the lower bound, Hman = log, N₁. Thus, again, more nuanced protocol only appears when it is needed. Finally, we that results for both architectures are close, although in three out of seven task setups (the number of classes N2 is 2, 10, and 20) deeper model results in slightly higher entropy of the protocol, on average. Overall, we conclude that Receiver's is capacity does not play a major role in the entropy minimization effect and the latter also takes place with more powerful Receiver.
C.3 WHAT IF COMMUNICATION TAKES PLACE THROUGH SEQUENCES OF SYMBOLS?
We also experiment with Guess Number in a setup where the agents communicate via variable-length messages. The general architecture of the agents is same in Section 3.1 however the output of Sender is used as the initial hidden state of a GRU cell (Cho et al., 2014). In turn, this GRU is unrolled to generate the message. The message is produced until the GRU outputs special <eos> token or until the maximal length is reached. In the latter case, <eos> is appended to the message. The produced message is consumed by Receiver's 's GRU unit and the hidden state corresponding to <eos> is used by Receiver as input to further processing. We use the Stochastic Computation Graph estimator described in Section 3.2 it provided fastest convergence.
13

---

Under review as a conference paper at ICLR 2020
Hom, atts
10ml 215
H4m.
1.24
1225
Birary digis hidden
Binay dight hidden
Binayy dighs hidden
(a) Vocab.
size:
256.
(b) Vocab.
size:
1024,
(c)
Vocab.
size:
4096,
Gumbel-Softmax
Gumbel-Softmax
Gumbel-Softma
6-
0.
0.05
A
0.
Birory digts Moden
Binory digits hoden
Bihory dight hidden
(d) Vocab. size: 256.
(c)
Vocab.
size:
1024.
(I) Vocab,
size: 4096.
Stoch. Computation Graph
Stoch. Computation Graph approach
Stoch. Computation Graph approach
approach
0.025
0.
Birwry dight hidden
Binwry dights hidden
Biinayy dight hidder
(g) Vocab. size: 256, REIN- FORCE
(h) Vocab. size: 1024, REIN- FORCE
(i) Vocab. size: 4096, REIN- FORCE
Figure 6: Guess Number: Entropy of the messages m, depending on vocabulary size, training method, and relaxation temperature (when trained with Gumbel-Softmax) or Sender's entropy regularization coefficient Ag. Shaded regions mark standard deviation.
We consider the entire variable-length message as the realization random variable m when calculating the entropy of the messages, H(m). The results are reported in Figure 9 arranged in function of maximal message length and vocabulary size. As before, we aggregate the successful runs according the entropy regularization coefficient X, applied Sender's output layer.
From Figure 9 we observe that the results are in line with those obtained in the one-symbol scenario. Entropy minimization still holds: more nuanced (high-entropy) protocol only develops when more digits are hidden from Receiver, which hence requires more information to perform the task. The approximation to the lower bound is however less tight the overall number of possible messages grows (higher maximum length and/or vocabulary size). There also a weak tendency for lower X, encourage tighter bottleneck.
In preliminary experiments, we have similar results when the variable-length communication is performed via Transformer cells (Vaswani et al., 2017) instead of GRUs (not reported here).
D TWO-DIGIT MNIST DATASET
As discussed in Section t
ensure high output informational complexity in the Image Classification
task, we use two-digit variant of the MNIST dataset (LeCun et al.) 1998a). We construct it as follows. When iterating the original MNIST dataset, take batch and (a) select the first
14

---

Under review as a conference paper at ICLR 2020
H40m
100%
103.55
(a) Vocab. size: 512
(b) Vocab, size: 1024
(c) Vocab. size: 2048
Figure 7: Image Classification: entropy of the messages H (m) across vocabulary sizes. Successful runs are pooled together. Shaded regions mark standard deviation.
7
4-
layer
2
Three hidden layers
1
Hun
logg
Figure 8: Image Classification: entropy the messages H (m) across Receiver model sizes. Suc- cessful runs are pooled together. Shaded regions mark standard deviation.
Htm), bits
A00.00
Hop
Nom
Binary digits hidden
Binary digits
(a) Max length: 5, vocabulary size:
(b) Max length: 10, vocabulary size:
16
16
bits
Htm), bits
Nom
Binary digits hidden
Binary digits hidden
(c) Max length: 5, vocabulary size:
(d) Max length: 5, vocabulary size:
64
64
Figure 9; Guess Number: Entropy the emergent protocol when communication is performed with variable-length messages. Shaded regions mark standard deviation.
15

---

Under review as a conference paper at ICLR 2020
161/2 2 and last /b 22 images, refer to them b1 and b2, respectively; (b) create new batch where the ath image from b1 is placed to the left of the ith image from b2 and then vice versa. As a result, we obtain new stream of images, where each MNIST digit is seen twice, the left and on the right side. Note that not all possible pairwise combinations of the original images are generated (there 60000 of those in the training alone) and the exact combinations change across epochs. As labels, we use the depicted two-digit number modulo Nr. where N₁ is the required number of classes. All pixels are scaled into [0, 1]. We use the same process generate training and test sets, based on the training and test images of the original MNIST dataset, respectively.
E HYPERPARAMETERS
In our experiments, we used the following hyperparameter grids.
Guess Number (Gumbel-Softmax) Vocab. size: [256, 1024, 4096]; temperature, т: [0.5, 0.75, 1.0, 1.25, 1.5]; learning rate: [0.001, 0.0001]; max. number of epochs: 250; random seeds: [0, 2, batch size: 8; early stopping thr.: 0.99; bits shown to Receiver: [0, 1, 3, 4, 5, 6, 7, 8).
Guess Number (REINFORCE) Vocab. size: [256, 1024, 4096]; Sender entropy regularization coef., As: [0.01, 0.05, 0.025, 0.1, 0.5, 1.0]; Receiver entropy regularization coef., A,: [0.01, 0.1, 0.5, 1.0]; learning rate: [0.0001, 0.001, 0.01]; max. number of epochs: 1000; random seeds: [0. 1, 2, 3]; batch size: 2048; early stopping thr.: 0.99; bits shown to Receiver: [0, 2, 4, 5, 6, 7, 8].
Guess Number (Stochastic Compulation Graph approach): Vocab. size: [256, 1024, 4096]; Sender entropy regularization coef., Ag: [0.01, 0.025, 0.05, 0,075, 0.1, 0.25]; learning rate: [0.0001, 0.001]: max. number of epochs: 1000; random seeds: [0, 1, 3]: batch size: 2048; early stopping
thr:: 0.99; bits shown to Receiver: [0. 2, 3, 4, 5, 6, 7, 8].
Image Classification experiments Vocab. size: [512, 1024, 2048]: temperature, [0.5, 0.75, 1.0, 1.5, 2.0]: learning rate: [0.01, 0.001, 0.0001]. max. number of epochs: 100; random seeds: [0, 1, 2]; batch size: 32; early stopping thr.: 0.98; number of classes: [2, 4, 10, 20, 25, 50, 100].
Fitting random labels experiments Vocab. size: 1024; temperature, т: [1.0, 10.0]; learning rate: le-4, max. number of epochs: 200; random seeds: [0, 1, 2, 3, 4]; batch size: 32; early stopping thr.:
0o; prob. of label corruption: [0.0, 0.5, 1.0].
learning rate: le-4, max. number of epochs: 200; random seeds: [0. 1, 2, 3, 4]; batch size: 32; early stopping thr.: 0.98.
Adversarial attack experiments Vocab. size: 1024; temperature, r: [0.1, 1.0, 10.0];
F EVOLUTION OF MESSAGE ENTROPY DURING TRAINING
In this Section, we aim gain additional insight into development of the communication protocol by measuring its entropy during training. We concentrate on Guess Number and use the same experimental runs summarized in Figure the main text.
For each game configuration (that is, number of bits hidden from Receiver), we randomly select one successful run and plot the evolution of Sender message entropy and accuracy over training epochs We also plot entropy and accuracy curves for randomly selected failed run, to verify to what extent entropy development depends on task success.
We report results for runs where training was performed with Gumbel-Softmax relaxation and with the Stochastic Graph Computation approach in Figures 10 and TT respectively. The reported entropy and accuracy values calculated in evaluation mode, where Sender's 's output is selected greedily, without sampling. higher entropy of such deterministic Sender indicates that the latter can encode more information about inputs in its messages.
From these results, we firstly observe that the initial entropy of Sender's messages (before training) can be both higher than required for communication success (Figures 10a and I 11a and lower (the
rest). When it starts higher than needed, it generally falls closer to the minimum level required for the
We exclude the configuration in which Receiver sees the entire input, as it is a degenerate case of non- communication, as discussed in Section 4
16

---

Under review as a conference paper at ICLR 2020
solution. When the initial value is low, it increases during training. The failed runs can have message entropy above (Figures 10a TOb & [Ta) and below (e.g. Figures [0c] 10d & (Td) successful runs, suggesting that there is systematic relation between degree of entropy and task success.
The fact that the entropy can be reduced with no decrease in accuracy or even with accuracy growth (e.g. Figure [0a red line, epochs 5..30) indicates that the tendency discover new messages (increasing entropy) is counter-balanced by the complexity of mutual coordination with Receiver when entropy is larger. In interpretation, is this interplay that serves source of the natural bottleneck.
Finally, while in some the entropy is effectively increased w.r.t. its initialization level, the resulting protocol's entropy is at, or slightly above the lower bound of what the task allows. In this sense, we argue that the reported effect can be correctly denoted as a "minimization" result.
17

---

Under review as a conference paper at ICLR 2020
2.5
1.0
2.0
0.8
H(m), bits
1.5
Accuracy
0.6
1.0
0.4
0.5
0.2
0.0
0.0
50
100
150
200
250
50
100
150
200
250
epoch
epoch
(a) Binary digits hidden: 2
5
1.0
4-
0.8
H(m), bits
Accuracy
3-
0.6
1-
0.2
0.0
50
100
150
200
250
50
100
150
200
250
epoch
epoch
(b) Binary digits hidden: 4
6
1.0
0.8
H(m), bits
4-
Accuracy
0.6
0.4
0.2
0.0
50
100
150
200
250
50
100
150
200
250
eooch
eboch
(c) Binary digits hidden: 6
8
1.0
0.8
6-
H(m). bits
Accuracy
0.6
4-
0.4
0.2
2
50
100
150
200
250
50
100
150
200
250
epoch
epoch
(d) Binary digits hidden: 8
Figure 10: Evolution of (mm) over training epochs. Gumbel Softmax-based optimization, Guess Number. For each game configuration, specified by the number of bits Receiver lacks, sample one successful (black line) and one failed (red line) training trajectory. The blue line marks mins minimal entropy for successful solution.
18

---

Under review as a conference paper at ICLR 2020
1.0
6
0.8-
H(m), bits
4
Accuracy
0.6
0.4
0.2
50
100
150
200
250
50
100
150
200
250
epoch
epoch
(a) Binary digits hidden: 2
1.0
6
0.8
H(m), bits
Accuracy
4-
0.6
0.4
0.2
M Ma uMM Myy
0.0
50
100
150
200
250
50
100
150 epoch
200
250
epoch
(b) Binary digits hidden: 4
1.0
0.8
1ui bits
5
Accuracy
0.6
4
0.4
3
0.2
www.wlll--d........ www Man
0.0
50
100
150
200
250
50
100
150
200
250
eooch
eboch
(c) Binary digits hidden: 6
8
1.0
0.8
H(m), bits
6
Accuracy
0.6
0.4
0.2
0.0
50
100
150
200
250
50
100
150 epoch
200
250
epoch
(d) Binary digits hidden: 8
Figure 11: Evolution of H Hm) over training epochs. Stochastic Computation Graph-based optimiza- tion, Guess Number. For each game configuration, specified by the number of bits Receiver lacks, sample one successful (black line) and one failed (red line) training trajectory. The blue line marks Il Hmin minimal entropy for a successful solution.
19


Reviews:
[{"review_id": "SJgZchUCFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the phenomenon of entropy minimization in emergent languages. The paper studies two simple speaker-listener prediction tasks where the amount of information that’s needed to be encoded in the speaker’s message can be manipulated. They find, in both games and across training methods, that speaker agents generally learn to minimize the amount of information conveyed in their messages (i.e. minimize the entropy), such that they are still able to solve the task. The paper then conducts some experiments measuring the robustness to overfitting, and find that more discrete channels leads to better generalization when training on partially shuffled labels.\n\nOverall, this paper is very well written. I think the main result, that speaker agents learn to only convey as much information as is needed to solve the task, is interesting and insightful. I do, however, have a slight concern about the novelty of this result. Given that the environments are very simple (consisting of a single message sent by a speaker and a prediction made by a listener), the line between ‘multi-agent emergent communication’ and ‘neural network with discrete representations’ is very blurred. This is addressed only briefly in the related work section, with a handful of post-2016 papers cited. I’d personally want to see a more expanded related work section that goes into some more depth into some of these papers, to be able to better judge the novelty of this contribution. \n\nI found the second result, that making the representations more discrete (by lowering the temperature in Gumbel-Softmax) leads to increased robustness to overfitting, to be less clearly explained, especially how it relates to the first result. It seems obvious to me that, if you can’t pass enough information from the speaker to the listener (when there’s a low temperature), then you won’t be able to solve the task at training time if a lot of information needs to be conveyed (as in the case of randomly shuffled labels). The authors describe this by saying: “With a low temperature (more closely approximating a discrete channel), this is hard, due to stronger entropy minimization pressure.” But this seems misleading to me, since it’s very different from the ‘entropy minimization pressure’ that was discovered in the first result (which comes about when both agents are able to solve the task, but the listener has redundant information). Thus, I don’t see how this result is either surprising or connected to the first result. \n\nFurther, the main claim to novelty of this second result is that it tests the information bottleneck principle in a ‘language learning’ set-up. However, since now the communication is continuous, this setup resembles the classic ‘neural network prediction’ even more closely, and calling it a language learning setup seems down to semantics. Given this, I’m unsatisfied with the comparison to previous work on the information bottleneck (which is not my area of expertise). \n\nGiven the above points, I’d say the paper is borderline it its current form, with a tendency towards rejection. However, I’d be willing to increase my score if the authors can clarify some of my confusion around the second experiment. \n\nUPDATE: I've increased my review to a 6 after the author rebuttal, although I still feel the paper is borderline. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gTnJOuqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the specific question/problem tackled by the paper?\n\nThe paper studies whether discrete communication channels between agents are low-entropy. The claim is that agents that try to solve a prediction task subject to a communication bottleneck will exchange low entropy messages, even if these messages are not explicitly encouraged to have low entropy.\n\nIs the approach well motivated, including being well-placed in the literature?\n\nThe paper is well motivated, though I am not an expert in the area.\n\nDoes the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.\n\nThe support for the claims is almost adequate. The entropy of the messages was only analyzed for the runs where agents have successfully learned to communicate in order to solve the task. Part of the paper's conclusion is that an entropy constraint on messages is not necessary, but maybe it still is necessary to increase the frequency of successful runs, or help faster learning.\n\nThis means that successful runs lead to low-entropy messages, but what about the unsuccessful runs? Do messages have low entropy as well? \n\nI am also somewhat confused by the second set of experiments. The discussion seems to suggest that setting higher temperature in GS creates pressure for lower-entropy messages. Buf if that's the case, then there's a controllable parameter that implicitly controls an entropy constraint and it's no longer clear to me that low-entropy is emerging.\n\nSummarize what the paper claims to do/contribute. Be positive and generous.\n\nI think the paper does an interesting analysis and makes an interesting point about the problem being studied. I am a bit confused by how the experimental setup supports the claims and their consequences. In particular, I have some doubts about the claim that entropy regularization is unnecessary.\n\nClearly state your decision (accept or reject) with one or two key reasons for this choice.\n\nI am voting for acceptance.\n\nProvide supporting arguments for the reasons for the decision.\n\nThe paper sets up a clear problem to study and focuses on increasing our understanding around the issue. After reading the paper a few times I am a bit confused about how the experimental setup supports the claims & conclusions. I think the results in Figs. 1-2 adequately support the claim, but the results in Fig. 3 make it unclear whether the temperature parameter is implicitly controlling entropy. The fact that unsuccessful runs were discarded for the first set of experiments limits the implications of the main claim that low entropy emerges, because an entropy regularization might still meaningfully improve the frequency of successful runs. \n\nProvide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n\nI think if the paper will be improved if it resolves the lack of clarity around the temperature in GS being an implicit entropy-regularization parameter. Perhaps an entropy-regularized setup is a useful comparison to show that it provides marginal benefit over the setup studied, and this might resolve the lack of clarity around the implications of the claims made from the first set of experiments.\n\nApart from these issues I am happy with the choice of topic and execution of the paper. I also appreciate that due care has been taken to present the work as understanding a phenomenon, to avoid any misconceptions about a new method being proposed. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HkebXr-r9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper sets up a couple discrete communication games in which agents must communicate a discrete message in order to solve a classification task.  The paper claims that networks trained to perform this case show a tendency to use as little information as necessary to solve the task.\n\nI vote to reject this paper.\n\nThe experiments are not very interesting and I don't at all agree with the assertion of the paper.  The paper claims the networks use only the entropy necessary to solve the task, but there are two main problems with this assertion.  (1) their own experiments don't support this all that strongly, as in the limit of few hidden bits (left half of the x axis in Figure 1), the networks all had noticeable excess information, and (2) and perhaps most damning the paper applies entropy regularization on the sender during training?  Could it perhaps instead be the fact that the entropy of the sender was penalized as an explicit regularization term that the entropy of the senders messages tended to be small?\n\nI also find the experimental design puzzling.  Why both reinforce and the 'stochastic computation graph' approach?  Treating the receiver's output as binary and stochastic without using the log loss of the bernoulli observation model is just giving up on a good gradient as far as the receiver is concerned.\n\nThe experiments done are much to simple and the protocol flawed.\n\nThe second set of experiments in Figure 3 were not left to converge, so I'm not sure how we can derive a great deal of insight.  Additionally, relaxing the gumbel softmax channel to being continuous rather than discrete technically ruins any argument that there is an entropy bottleneck present anymore, as theoretically even a single dimensional continuous signal could store an arbitrary amount of information.  If the paper wanted to, it could have upper bounded the mutual information between the input and the message using a variational upper bound.\n\n--------------- Response to Response ---------------------------------\n\nI'm editing here in light of continuing to look at the paper and the responses from the author below.\n\nI have to still argue for a rejection of this paper.  \n\nI thank the authors for addressing my comments and I admit that at first I thought the paper was minimizing the entropy during training which would have been particularly bad.  While I was mistaken on that point, I still believe the paper is deeply flawed.\n\nIn particular, the paper makes a very bold claim, namely that \"We find that, under common\ntraining procedures, the emergent languages are subject to an entropy minimization pressure that has also been detected in human language, whereby the mutual\ninformation between the communicating agent’s inputs and the messages is minimized, within the range afforded by the need for successful communication.\"  But if we are being honest here, the experiments are very lacking to support such a bold claim.\n\nIn particular there was one thing I was worried about upon reading the paper again, and is similar to the point raised by the other reviewers.  In Figure 1, we are shown the entropy only of those networks that have succeeded.  Naturally to succeed, the entropy i the message must be large enough to accomodate the size of the remaining bits we are trying to reconstruct.  That is why Figure 1 includes the dotted line, since the networks must be above that line to have good performance.  And the main evidence for the main claim of the paper is that the trained networks are above that line and arguably close to it.\n\nNow, we know that there are clearly solutions to these tasks (in particular the Guess Number task) which could achieve good performance at noticeably higher entropy.  For instance we could take any minimal solution and simply split up each message into 8x different buckets, each of which had exactly the same behavior from the decoder.  This would give us a +3 in the entropy of our message space while having no effect whatsoever on the loss.  The claim of the paper is that under normal training procedures it seems like we don't find those solutions and instead seem to find minimal ones.\n\nBut after implementing a simplified version of the experiment in the paper (Notebook available here: https://nbviewer.jupyter.org/urls/pastebin.com/raw/ZF7g34GN ) I suspect something much simpler is going on.  The reason the solutions look minimal in Figure 1 is probably because the initialization chosen for the encoder they used in the paper tended to start at low entropies.  Imagine if all of the networks started out with an initial message entropy of around 3 bits.  Then Figure 1 could be explained by the problems with hidden bits ~< 3 bits simply preserved their entropy, which in order to solve the task with higher numbers of digits hidden we know requires some minimal budget, so they get sort of pushed up.  This could explain the figure, but we wouldn't claim this explains why we observe small entropy for the high number of hidden digits case.\n\nIn particular, if we initialized the encoders with higher entropy, we might expect that we fail to see this phenomenon.  That is exactly what I was able to show for myself in that notebook.  If you simply initialize the encoder to have high entropy, all of the solutions have high entropy and the observed effect goes away.\n\nOverall, the paper as I said is low quality.  Several choices were made that don't make a lot of sense.  With the experiments being as small scale as they were, why not explicitly marginalize out the message (as I did in the notebook)?  Why use single layer neural networks to predict 256 x 1024 parameters? Why not just learn them directly?   If the paper aimed to mimic more standard setups and show that under those setups we observe this kind of minimal message entropy, then it would have to much better tease out the effects of all of these choices. \n\nWhy does the decoder use a mean field sigmoid bernoulli observation model to try to predict something is in one of ~32 states?  The missing digits are not independent given the message, why model them as so?  Is that part of the purported reason why these models show minimal entropy (cause it isn't discussed).  \n\nFor such a simple problem, you could presumably analytically compute the gradient with respect to the loss and study whether that correlates with the gradient of the entropy.  There are several things I could imagine checking, none of which are checked in the paper.\n\nThe primary question the paper addresses is an interesting one.  But this paper does very little to carefully investigate that question.  I maintain my vote to reject.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1exhXP6KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a series of simple experiments to characterize the objective that emergent languages are optimizing and why we see various behaviors (not aligned with natural language) when training them to play these language games.  The paper is clearly written, the tasks are simple and minimalist (in a good way) and the experiments are often followed by the exact ablation I was just noting that I wanted to see. \n\nQuestions:\n1. What happens if the vocabulary is too small to completely communicate the space?  Do models accept that some concepts are not expressible, discover a random behavior that overloads a lexical item, or fail to learn entirely?\n\n2. Why are values for lamda_r not investigated or plotted?\n\n3. How many runs are averaged for each experiment? Should I assume one per seed? why are there different numbers of seeds for each experiment?\n\n4. Page 5 -- \"We conclude that in this case there is no apparent entropy minimization pressure on Sender simply because there is no communication\" -- Was any analysis performed on the gradients to see if they are completely random or what kind of signal they are getting?\n\n5. How important is the architectural design to these experiments? e.g. number of hidden layers, etc?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the information-theoretic complexity for emergent languages when pairs of neural networks are trained to solve a two player communication game. One of the primary claims of the paper was that under common training protocols, networks were biased towards low entropy solutions. During the discussion period, one reviewer shared an ipython notebook investigating the experiments shown in Figure 1. There it was discovered that low entropy solutions were only obtained for networks which were themselves initialized at low entropy configurations. When networks are initialized at high entropy configurations, the converged solution would remain high entropy. This experiment raises questions about the validity of the claim that there was "pressure" towards low entropy solutions to the task. Therefore, a more careful analysis of the phenomenon is required. 
Decision: Reject