Trading off Utility, Informativeness, and Complexity in Emergent Communication
Authors: Mycal Tucker
Roger P. Levy
Julie Shah
Noga Zaslavsky
Venue: NeurIPS 2022 Accept (2022)
Keywords: emergent communication
information bottleneck
multi-agent reinforcement learning

Abstract: Emergent communication (EC) research often focuses on optimizing task-specific utility as a driver for communication. However, there is increasing evidence that human languages are shaped by task-general communicative constraints and evolve under pressure to optimize the Information Bottleneck (IB) tradeoff between the informativeness and complexity of the lexicon. Here, we integrate these two approaches by trading off utility, informativeness, and complexity in EC. To this end, we propose Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to encode inputs into discrete signals embedded in a continuous space. We evaluate our approach in multi-agent reinforcement learning settings and in color reference games and show that: (1) VQ-VIB agents can continuously adapt to changing communicative needs and, in the color domain, align with human languages; (2) the emergent VQ-VIB embedding spaces are semantically meaningful and perceptually grounded; and (3) encouraging informativeness leads to faster convergence rates and improved utility, both in VQ-VIB and in prior neural architectures for symbolic EC, with VQ-VIB achieving higher utility for any given complexity. This work offers a new framework for EC that is grounded in information-theoretic principles that are believed to characterize human language evolution and that may facilitate human-agent interaction.

Trading off Utility, Informativeness, and Complexity in Emergent Communication
Mycal Tucker
Roger Levy M.L.T rplevyemit.edd
M.I.T. mycal@mit.ce edu
Julie Shah
Noga Zaslavsky M.I.T. nogazsEmit edu
M.I.T. julie_a a_ saa.s_sshahhsailll mit.eduu
Abstract
Emergent communication (EC) research often focuses on optimizing task-specific utility a driver for communication. However, there is increasing evidence that human languages are shaped by task-general communicative constraints and evolve under pressure to optimize the Information Bottleneck (IB) tradeoff between the informativeness and complexity of the lexicon. Here, we integrate these two approaches by trading off utility, informativeness, and complexity in EC. To this end, we propose Vector- Quantized Variational Information Bottleneck (VQ-VIB), method for training neural agents to encode inputs into discrete signals embedded in continuous space. We evaluate our approach in multi-agent reinforcement learning settings and in color reference games and show that: (1) VQ- VIB agents can continuously adapt to changing communicative needs and, the color domain, align with human languages; (2) the emergent VQ-VIB embedding spaces are semantically meaningful and perceptually grounded; and (3) encouraging informativeness leads faster convergence rates and improved utility, both in VQ- VIB and in prior neural architectures for symbolic EC, with VQ- VQBVIB achieving higher utility for any given complexity. This work offers new framework for EC that grounded in iiformation-theorede principles that are believed to characterize human language evolution and that may facilitate human-agent interaction.
Introduction
Good communication is critical component of successful teams of humans and artificial agents, but differing notions of "good" makes training agents to develop such communication challenging. One view of communication focuses utility by framing languages successful to the extent they enable high task performance. Emergent communication (EC) literature, wherein agents learn to communicate while optimizing the utility associated with task-specific goal, such reaching a target landmark, emphasizes this view [1-6]. However, training via utility alone may lead to undesirable properties of the EC system, such violations of linguistic universals [7], poor human interpretability [8], slow convergence rates [9], or high sensitivity to environment design [10]. This implies that some constraints may be needed to guide agents toward human-like communication.
Another view of communication. often considered the cognitive science literature, focuses on task-agnostic communicative objectives major forces shaping human languages [11, 12]. Most relevant to our work a recent line of work [13-17] that argues that languages evolve under pressure to efficiently compress meanings into communicative signals by optimizing the Information Bottleneck (IB) principle [18]. In this context, IB can be interpreted as a tradeoff between the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(
Au
(b)
Informatieenes
for humans (Ac=0)
conver convergence (A-=0) (D)
Too specific, slow
Too comple
Task
specific
comm.
description
Silence
Complexity
21
MC
Utility
Task agnostic (Au=0)
Figure 1: (a) In our theoretical framework, agents evolve communicate with each other by optimizing a tradcoff' between utility, informativeness, and complexity, weighted by Au, Az, and Ac respectively. (b) Basic setup and agent architecture. speaker, S, maps an input, æ, signal, c. A listener, L, takes an action, y, and reconstructs r using a decoder D.
complexity and informativeness of the lexicon [13]. Intuitively, complexity corresponds to the number of bits allocated for communication, and informativeness corresponds to how well listener can understand a speaker regardless of any task-specific goal (c.g.. humans understand the term "blue" in "avoid the blue box" and "find blue cup"). This information-theoree semantic categories has been gaining broad empirical support across hundreds of languages and multiple semantic domains [13-16], and has been directly tied language change over time [17]. However, the actual process that may drive a population of speakers - whether human or artificial toward IB-efficient communication systems in given environment remains largely unknown.
framework for grounded
In this work, we integrate these two views of communication by training teams of interactive neural agents to optimize tradeoff between maximizing utilityy maximizing informativeness, and minimiz- ing complexity, as depicted in Figure 1. We hypothesize that taking into account complexity and informativeness, which are cognitively-motivate to task-specific utility, may help guide artificial agents toward communication systems that can be more naturally aligned with and adapted to human languages. To this end, building prior work [19, 20), we propose Vector-Quantinn Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to encode inputs into signals, while maintaining discrete (symbolic) signal representation that is embedded in a continuous space, inspired by word embeddings [21-23]. We evaluate our approach in two multi-agent reinforcement learning (MARL) environments and in two variants of color reference game. We show that: (1) by annealing a single tradcoff parameter, VQ- VIB agents can continuously adapt to changing communicative pressures and, in the color domain, align with human languages, without changing the agents' architecture or the characteristies of the training environment; (2) the emergent VQ-VIB embedding spaces capture semantically meaningful structures, grounded in the perceived environment; and (3) encouraging agents to be informative generally leads to faster convergence rates and improved utility, with VQ-VIB achieving higher utility for any given complexity compared to previous EC approaches that also support symbolic communication. Taken together, this work offers a new cetemattion-thhoreee framework for EC that is grounded in general optimization principles that are believed to characterize human language evolution, and that has the potential of improving human-agent interaction.
task-general communicative facctors, in addition
2 Related work
While EC studies usually focuses on task utility, some studies also consider communicative constraints that are closely related to our measures of complexity and informativeness (defined in Section 3). For example, numerous works consider discretization, environmental noise, or penalty on large vocabulary sizes as proxies for reducing complexity [23-25]. In contrast, our notion of complexity grounded in rate-distortion theory [26], and has been shown capture the degree of semantic compression seen across languages [13]. Wang et al. [27] use the same IB complexity measure, but in different communicative settings that involve a centralized message processor. Other notions of linguistic complexity often focus on grammar, word forms, or compositionality [28-31]; these are beyond the scope of this work but offer promising directions for future work. Apart from complexity, some works consider auxiliary reconstruction losses in EC [9, 32]. In particular, Lin et al. [33] use an autoencoding loss which closely related our notion of informativeness. However, they do not penalize complexity and note that their method may fall short in cases where "agents need to be more

---

selective of the information communicated." Lastly, there is a body of work on reinforcement learning with informational constraints similar to ours [e.g., 34-38J, however not in the context of EC.
To evaluate our method, we consider particle-world environments that agents must navigate [3, 23}, well reference games (related to Lewis signalling games [39]), both of which popular types of EC environments. For reference games, we focus on the domain of color for several reasons: First, it has been central test-case for cognitive theories of semantic categorization and language evolution [1, 40, 41]. Second, it enables comparison with the World Color Survey [WCS; 42], a dataset of color naming systems from 110 languages of -on-industrialized societies. Third, it has been focus of numerous studies in EC and machine learning more generally [1. 43-45, 10]. In particular, Chaabouni et al. [10] show that color signaling systems in artificial agents may reach the IB efficiency bound from [13] and capture some of the variation seen across human languages. However, to achieve this, they hand-craft varying characteristics of the training environment by selecting dissimilar distractor colors or by adding noise to the agents" communication. Rather than focusing on environment- hased factors, we adopt a principled approach by directly optimizing agents with respect to a variational IB principle in a given environment.
3 Information-theerrtic emergent communication
Here, we propose an information-theoreti EC framework that integrates task-general communicative objectives with task-specific utility maximization. We begin by reviewing the theoretical framework on which we build (Section 1.1.1) and the set of variational inference tools that we use in order to ex- tend that framework to EC settings (Sections 3.1.2 and 3.1.3). We then define integrative objective function for training EC agents (Section 3.2) and present Vector-Quantizer Quantized Variational Information Bottleneck (VQ-VIB, Section 3.3). a method for training agents with respect to our objective function while maintaining structured symbolic representations that can be used for communication. The experimental setup and results for evaluating this new method are presented in Sections 4 and 5.
3.1 Preliminaries
3.1.1 The Information Bottleneck framework for grounded semantic systems
We build on and extend Zaslavsky et al. [13]'s information-theoreee framework for grounded semantic systems. In this framework, speakers and listeners jointly optimize the IB tradeoff [18] between the complexity and informativeness of their communication system. A speaker is characterized as probabilistic encoder S(c|x) that, given an input z p(x), mentally represents it as m, and generates a communication signal, c. mg is the speaker's belief state, which is grounded in the agents' representation of the world and captures any uncertainly the speaker may have about the input. More formally, m, is defined by a probability distribution over a set of features u E ll that characterize the input space, i.e, my (u) = p(u(m)) For example, may be target color and my Gaussian distribution over the CIELAB perceptual color space (Figure 3c), capturing the speaker's 's perceptual uncertainly. Given e, the listener infers the speaker's belief state by performing Bayesian inference, i.e., it constructs an estimator ( = Σ. (x)c))m.(() using the posterior S(x c) xx ((c(x))(z))
Complexity is measured by the mutual information between the speaker's 's inputs and signals. [s(z, c). which, as in rate-distortion theory [26, 46], is roughly the number of bits that are needed for communication [18]. Informativeness corresponds to the similarity between the speaker's 's and listener's belief states. Because these states are defined by probability distributions, maximizing informativeness amounts minimizing the expected Kullback- them, i.c., Es [KL [m ||m] Taken together, in this view, for any given tradcoff parameter 3 2 0 the speaker and listener should jointly minimize
(KL) divergence between
Is(x, c) + fBs (DLs [ms
which is equivalent to optimizing the IB principle [see 13, 47, for derivation). For B S 1, the optimal systems are non- non-informative and minimally complex (Is(æ, e) = 0), yielding no communication. Gradually annealing 3 yields optimal systems with greater complexity. and as 3 + oo, the optimal systems become maximally informative but also maximally complex. Zaslavsky et al. [13] and col- leagues have shown that human languages achieve near-optimal IB tradeoff's across multiple semantic domains (see Figure 6a for example), and that the continuous evolution of the optimal IB systems via annealing the tradeoff parameter /3 captures important aspects of language evolution [13, 17].
3

---

3.1.2 Variational Information Bottleneck
Variational Information Bottleneck [VIB, 19] is variational approximation method for IB with continuous representations. In its typical form, a neural encoder go maps an input x to parameters of d-dimensional Gaussian distribution, su(x)) and E(x), from which a latent representation z I&d is sampled. Given 2, VIB reconstructs a target feature u using gul (u(z), parameterized by soft-max function. VIB optimizes variational bound on the IB objective function, based on the fact that
qo (z: 2) - E D go (z (=]=)]][(=)]]
for any distribution r(2), which is often taken to be (0, Ia). VIB also bounds the IB informativeness term. However, because we consider here different instantiation of IB, based on unsupervised rather than supervised learning, we use a different informativeness bound, as explained in Section 3.3.
3.1.3 Vector-Quantized Variational Autoencoder
While VIB learns continuous representations, Vector-Quantized Variational Autoencoder [VQ-VAE, 20] Iearns discrete, but potentially highly complex, representations in an unsupervised manner. More specifically, VQ-VAE defines latent embedding space of K vectors 4 E [R ,,..., K. A deterministic encoder maps an input x to a discrete variable i by first passing æ through deterministic network obtain latent representation 2(x) E Rd and then discretizing it to the index of the nearest embedding vector, i.e., i = argmin, [1= - Gill- The decoder receives the corresponding embedding vector, (((x), and passes it through another deterministic network to reconstruct the input. The set of trainable parameters O consists of the embedding vectors and the parameters of the encoder and decoder networks. Because gradients cannot be passed through the argmin operator, a ththggh estimator [48] is used. Similar VAE [49], VQ- VA-V'''s training loss (Eq. (3)) uses the evidence lower bound (ELBO), which is reduced in this case to the first term in Eq. (3). VQ- adds to it two VQ terms, designed for learning the embedding space, yielding the overall training loss
lvQ log (()) θ) + [|s][>())] ((=))]1 + all=(x)) sg(((()))1
where sg stands for the stopgradi ent operator, and following [20], 0.25.
3.2 Integrating utility, informativeness, and complexity
The theoretical framework and variational tools reviewed in section 3.1 focus on general informational constraints and autoencoding reconstruction losses. However, EC settings are often based on multi- agent reinforcement learning (MARL), where agents are trained to maximize an expected reward, or utility, for given task. In basic cooperative setting, there are two agents: speaker and listener. The speaker receives an input, E X, which is sampled from a distribution p(x) and contains information about the task's 's goal which is hidden from the listener. The speaker's s actions are restricted communicating a vector e E to the listener. The listener receives e and partial observation o E O about the world state, and takes an action y ) We denote the speaker's 's policy by S(cla) and the listener's policy by L(ylc, o). each episode the agents receive a utility U U(x; v) based on their actions and aim to learn policies that maximize the expected utility Es. [U (ar, v)).
Note that this setting similar to that of Section 3.1.1, with the key difference being that here the listener takes an action in the world instead of reconstructing the speaker's s belief state. To integrate the two settings, we add to the basic EC setting a decoder, D, as shown in Figure lb. If the speaker has no uncertainty about the input, then the decoder maps to a reconstruction a.. Otherwise, the decoder reconstructs a belief state, me. as explained in Section 1. A similar approach can also be applied to more complex EC settings, which may involve sequential actions with discounted rewards, additional agents, or agents that can simultaneously communicate and act. In Sections 4 and 5, we consider navigation tasks with sequential actions, and reference games which correspond to the basic setting. We leave the exploration of other multi-agent settings to future work.
Our proposed objective function for training EC agents in these settings combines utility maximization with the IB objective from Eq. (I), yielding
LX[S, L, D] = (U v)] A/Es, ( [m - Az)
(4)
such that agents should jointly maximize [A L., D). The non-negative tradeoff parameters, or Lagrange multiplies, X = A,, control the relative weights of utility, informativeness

---

z ~ 2),
argmin(Ilz - <;ll²) 4
The VQ-VIB encoder architecture. The speaker encodes input, æ;, as parameters Gaussian distribution, from which a vector is sampled and then discretized by mapping to the nearest encoding, to produce a communication vector, c.
Figure 2:
(negatively related the second term), and complexity. Intuitively, these three weights form simplex (Figure la) that spans the range of communication systems that may emerge in our framework, from uninformative communication (minimizing complexity alone), task-specific communication that may not generalize (maximizing utility alone), to complete descriptions (maximizing informativeness alone) that may be unnecessarily complex for many tasks and too complex for alignment with humans.
3.3 Vector-Quantized Variational Information Bottleneck
As in IB, directly maximizing Eq. (4) is intractable in large domains. To address this we take variational approximation approach, where we assume parameterized agents and maximize variational lower bound on Eq. (4). Additionally, we consider an encoder (speaker) architecture that learns symbolic representations that are embedded in continuous space, inspired by word embeddings [22, and see also [23] for a related prototype-based EC approach]. We refer to this new method as Vector- Quantized Variational Information Bottleneck (VQ-VIB). Before defining our method precisely, we note that VQ-VIB builds on VIB and VQ-VAE, but at the same time differs from them in important ways. VQ- IQ-VIB learns a stochastic encoder, contrast VQ- and symbolic representations, in contrast to VIB. Additionally, while VQ-VIB's training objective borrows from VIB and VAE, considers different variational IB bound than VIB, uses that bound instead of the BLBO in VQ-, and adds to that symmetry -ereakrr-breaking term." These difference are particularly crucial for human-like communication, as human languages employ symbolic representations (c.g., words), human speakers often stochastic, and discussed in Section 1.1.1, there is substantial empirical evidence that human languages are shaped by the IB principle.
3.3.1 VQ-VIB architecture
The VQ-VIB encoder (illustrated in Figure 2) is composed of components: a codebook of K embedding vectors, = {Si,,... [K] C Rd similar to VQ-VAE, and a VIB-like stochastic encoder (=(xx)) Given an input æ, the encoder qo is used sample a latent representation E RR. followed by quantization layer that discretizes into the nearest embedding vector. This encoding process generates a vector
argmin,ccc | <lI which our settings will be used for communication. We instantiate go (=(xr by a feedforward network that maps æ to parameters of Gaussian distribution over the latent space. Overall, the VQ-VIB encoder, speaker, is given by
So (p argmin || <1 |1 C,
ÇEC
Finally, as shown in Figure 1b, we pair the VQ-VIB encoder with a simple decoder network D (&lc) that reconstructs the speaker's input given the communication vector (see Section 3.3.2 for justifica- tion), and with a task-specific policy network Lyle,, o) from which the listener selects its actions. These two networks can also be paired with other speaker (encoder) architectures, as do in subsequent sections in order to evaluate the VQ-VIB method.
3.3.2 Learning objective
To train VQ-VIB agents, we first define a variational bound on Eq. (4). We bound the complexity term (third term in Eq. (4)) by noting that Is, (æ; Lgo (z; 2) < [K] [ge (2](<=/=)]]r(((] where the first inequality folows from the data processing inequality, and the second one follows from Eq. (2).
VIB and VQ- VAE also do not optimize the utility term in Eq. (4), but that can be easily added them.
5

---

(a) MARL: points
(b) MARL: Uniform
(c) Reference game: 330 WCS colors
960
O
Representation in CIELAB coordinates
Figure 3: Test environments. In the MARL settings (a-b) speaker (clear triangle) observes target location drawn either from a fixed set of 9 points (a) or uniformly in a continuous space (b), communicates it to listener (gray triangle), which then needs to navigate that location (adapted from [23]). In the color reference game (c), a speaker observes a target color from the World Color Survey [WCS, 42] palette, represented in the 3D CIELAB perceptual color space, and communicates it to the listener, who then needs to identify the target color.
To bound the informativeness loss (second term in Eq. (4)), we first assume that given I E R", the speaker's uncertainty is captured by Gaussian noise and therefore its belief states are given by my = -xx Σ). Under this assumption. it holds that E[ [m [me]]] S [[[| - -f f + const
(see Appendix for proof), allowing to approximate the ideal decoder by simplified decoder DΦ C + R." that outputs ã... This yields the following lower bound on Eq. (4):
L0 o, Cb Àu Es.. (xx y)] - Es [||= = Aug (DKL [Pkk
(6)
To train the embedding vectors, we include the two VQ terms from Eq. (3), and based on [50], we also add small regularization term, =(0, C). designed to break the symmetry between equally good
solutions (see Appendix B). Taken together, we obtain the following objective function for VQ-VIB:
LVQ-VIBAX o. s. C] Ex10, C] Es. [[|sg[2] - cll2 + all= sg[e]][ =(O.C), (7)
which should be jointly maximized by the agents for any given A. Following [19], we fix r(2)
N(0, Ia). and following [20], we fix a 0.25. To optimize this objective function, we use the VAE trrriggh-tthrouge estimator to bypass the con-differentiablity of the argmin operator, and the reparameterizatioon trick [49] for backpropogating through 2. While the latter is internal to the speaker, the straight-through estimator is used to pass gradients from the decoder D& and the listener's policy L into the speaker Sg. Here, we allow gradients to pass from the decoder, but take more careful approach to passing gradients from the listener by comparing the straigh--thruugh estimator approach to methods that do not pass gradients, as explained in the next section.
4 Experimental setup
To evaluate the emergent VQ-VIB communication systems and how they are shaped by X, we consider three test environments (Figure 3) and compare VQ-VIB with strong baselines, as described below. Full details of the architectures, hyperparameters, and environments are provided in Appendix D.2
Trials and baselines We consider three types of baseline speaker architectures that are derived from prior work: (I) a continuous speaker [c.g., 51], which unlike humans, communicates using continuous vectors without maintaining symbolic representations; (2) a one-hot speaker [e.g.. 3], which communicates with discrete but unstructured vectors; and (3) a prototype- based speaker [Proto, 23], which maps one-hot vectors to embedding space, and in that sense is similar to VQ-VIB, although its internal mechanism is different. We also considered a variant of the VQ-VIB speaker ('VQ-VIB After') which is more similar Proto, but because it does perform well VQ-VIB we leave its discussion to Appendix C. For a fair comparison, we train all speakers with the same decoder and listener architectures, allow them be stochastic similar to the VQ-VIB speaker, and
Code is available https: hogp:////tthuue www.:uu...oommmaallalaacaaaaa http://wwwwaauutttt//////.oeoeoa newrip200er0s2222222222222

---

adjusted their training objective according to Eq. (6) to allow them to consider the same informational constraints as VQ-VIB. In all domains, agents are trained from scratch using random seeds.
Particle worlds We train agents in the two partiele world environments shown in Figure 3, which adapted from Tucker et al. [23]. Both environments consist of two-dimensional worlds in which speaker observes the exact target location that listener needs to reach but cannot directly observe. In the 9 points environment, there are landmarks at fixed locations, one of which is chosen at random as the target for each episode. In the Uniform environment, two landmarks are spawned in randomly drawn locations in the continuous space and one of them is specified as the target. Both agents receive reward equal to the negative distance from the listener to the target and are trained using Multi-Agent Deep Deterministic Policy Gradient [3], which is a standard policy-gradien method without backpropagating from one agent into another. The listener cannot observe the target location, and the speaker can only communicate at the first timestep by broadcasting vector that is observable at all future timesteps; thus, an optimal team policy consists of the speaker communicating about the target at the first timestep, and the listener navigating to the target based on the message.
Color reference games We consider two variants of a color reference game, a basic game and a common ground game (see Figure D.I). In the basic game, speaker observes a target color and outputs a communication vector. A listener observes the speaker's ' communication vector and two colors- the target and a distractor, without knowing which the target - and tries to guess the target. In the common ground game, the speaker also observes the distractor, in addition to the target, which in principle allows the distractor may influence the listener's s choices [44, 52]. Apart from that, the two variants are the same. Following [41, 13), colors are represented in the 3D CIELAB color space (Figure 3c), in which small Euclidean distances between colors are correlated with human perceptual dissimilarities. To capture perceptual noise, the speaker's 's observations are corrupted by e ~ (O, o²1) with o² = 64, as motivated by human perception [53]. The target and distractor are sampled independently from the 330 color chips of the World Color Survey [WCS, 42] shown in Figure 3c, according to a human- based prior from [13].² Agents are trained with respect to the binary crossentropy loss computed on the listener's 's output. We consider two training methods: (i) using truight-thhouge gradient estimators to backpropogate from the listener into the speaker, which is common in EC and supports efficient training, although it is not biologically plausible; and (ii) using the REINFORCE algorithm [54], which does not pass gradients between agents but is known to suffer from slow convergence rates.
communicate in a more pragmatic way by taking into account how
5 Results
5.1
Particle worlds
We begin with exploring the influence of À on the emergent agent behavior in the two particle world environments. Without loss of generality, we take Au unless stated otherwise, and then evaluate the influence of Åo and \ separately. Starting with Aç. we simulate the evolution of the emergent VQ-VIB communication systems by fixing ', = and varying Åc using deterministic annealing [55] (see Appendix D for detail), much like the evolution of the IB representations in [50]. Figure 4 shows lower- and higher-complexity solutions for the two environments (see Movie I for the continuous evolution of the systems). The top panels (Figure 4a and c) show how the speaker partitions the environment into communication signals, together with the listener's trajectories in response to the speaker's communication. The lower panels (Figure 4b and d) show the emergent VQ-VIB embedding space C. In both environments, lower complexity solutions form a coarser representation of the space, with fewer communication vectors (signals) used by the speaker and, respectively, fewer final positions reached by the listener. We wish to emphasize that the number of signals available to the speaker is fixed throughout these experiments (K = 9 in the 9 points environment; K = 100 in the uniform environment), as well as the agents' architectures and all environment parameters. The structural changes in the effective number of signals used by the speaker result from annealing Ac which dynamically controls the communicative bandwidth that the agents coordinate during self-play.
Remarkably, the emergent structure of the VQ-VIB embedding spaces (Figure 4b and d) captures semantically meaningful information that is grounded in the environment. That is, the embedding vectors roughly encode distances between "named" regions. We note that Tucker et al. [23]'s
'The WCS data are available at www1 icsi, berkeley edu/wcs and the human- based color prior is available https: / // giggthu........ www.nogazs/ibbbbboooo wwwabb--ooooonammmaaeee
7

---

400
100
680
750
300
300
230
100
100
201
200
100
342
(b)
Lower
Higher
(d)
Lower
Higher
Figure 4: VQ-VIB solutions for the 9 points (a-b) and uniform (c-d) environments, with lower (left panels) and higher (right panels) complexity (see also Movie 1). Each communication vector is associated with the mean color the positions can refer to, based on the speaker's 's encoder and the colors in Figure 3. (a) Listener's trajectories (black curves) and final positions (dots) in response to speaker's communication vector, indicated by the color of the final dot. (b) The emergent VQ- VIB communication vectors (trained with d 2). (c) Same (a) but for the uniform environment. Background colors show the modal communication vector for each target position the continuous space. (d) 2D PCA of the VQ-VIB communication vectors (trained with d 3).
(a)
(b)
-100
-00
-70
-200
Utility
Utility
-80
VO-V8 100
Cont
-90
VQVV18 9
4400
-100
Proto. 9
300
-110
Onehot
0
1000 2000
3000
a30o
5000
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Training episode
Complexity (bits)
Figure 5: Uniform environment results. (a) Expected team utility during training of VQ- IQ-VIB agents with varying informativeness pressures. (b) Utility cs.pllllyy for the continuous architecture and the symbolic EC architectures with = 9 (dashed curves and stars) and = 100 (solid curves and dots) signals, excluding few points below -120 for visualization only. Each curve corresponds to cubic best to the data (stars/dots) generated across random seeds for each configuration.
prototype-based approach is also capable of learning such grounded representations. although without informational constraints, thus resulting in maximally complex and task-specific communication Therefore, as explained Section 4, we compare VQ-VIB with an adjusted version of Tucker et al. [23]'s approach (Proto), in addition to a one-hot and continuous architecture. While the latter human-like we consider it here as an upper bound on the achievable performance in these environments. The results for the uniform environment are shown in Figure 5b, and similar trends also hold the points environment (Appendix E). can seen that VQ-VIB achieves higher utility for any given complexity and codebook size K compared to the two baselines that support discrete (symbolic) communication, i.c., Proto and one-hot. Additionally, increasing K from 9 to 100 substantially improves VQ-VIB, bringing close to the performance of the continuous architecture, while the same increase in yields much smaller improvements for Proto and one-hot.
Next, we evaluate the influence of the informativeness. weight, Y₁- While informativeness and utility may seem similar, and they are indeed closely related, they are also importantly different: maximizing utility encourages agents to coordinate on a task-specific communication system, whereas maximizing informativeness encourages agents to develop shared understanding regardless of any task-specific needs. Therefore, intuitively, informativeness may help agents coordinate faster and converge to more effective communication systems. test this intuition, we eliminate the complexity constraint by setting Ào 0 and vary λ Figure 5a shows that, indeed, increasing x leads to higher utility and

---

(c)
VQ-VIB embeddings
(e)
VQ-VIB embeddings
Informativeness (bits)
50-
3-
(c-f)
15-
25
2
(c-d)
50-
Theoretica limit
1
VQ-V B
-0-
-62
WCS languages
-25
-10
-50
25
56
28
00
75
Complexity (bits)
(b)
(d)
VQ-VIB mode map
VQ-VIB mode map
0.34
0.92
Ubilty
0.90
Human mode map (Kwerba)
Human mode map (Kalam)
VQ-V onehat
0.46
0.42
Complexity (bits)
Figure 6: Color reference game results (basic setting, see also Movie 2). (a) VQ-VIB agents (orange dots) reach the theoretical bound from [13] (black curve) and capture the variation observed across the WCS languages (teal dots). (b) VQ-VIB achieves greater utility, for the same complexity, than one-hot. (c-f) The emergent VQ-VIB color naming systems and closest (min gNID) human systems in the WCS data, for agents with lower (c-d) and higher (e-f) complexity, as shown in (a). Each communication vector associated with the mean color it can refer to. (c, e) 2D PCA of the VQ-VIB communication vectors, where the size of each point reflects its relative category size. (d, f) Mode maps, where each color chip from Figure 3c is colored according to its modal communication vector.
faster convergence rates. The results shown here are only for VQ-VIB in the uniform environment, but similar trends appear for all architectures in both environments (Appendix E).
5.2 Color reference games
Our results for the two types of color reference games (basic and common ground), and for the two types of training methods (REINFORCE and straight-through estimator) are similar, with REINFORCE being slightly worse (consistent with [10]). We therefore focus here on the basic game with straight-through estimator and provide the full set of results in Appendix F. In addition, because agents' communication systems with actual human color naming systems,
wish to compare we focus only on VQ-VIB and one-hot. The continuous architecture cannot generate discrete color categories, and based on the previous section, Proto performs similarly to one-hot.
As before, we fix \ color chips) for all agents, which allows full expressivity. although that would require maximal complexity and is unlike attested human languages which typically use only a few color terms [40]. We therefore focus on the range of Xo that produces the variation in complexity observed across languages. Following Zaslavsky et al. [13], we use the color naming data from the World Color Survey IWCS, 42] to estimate encoder (c((rr for each WCS language 1. Figure 6a replicates Zaslavsky et al. [13]'s finding that languages are ncar-optimal with respect to the IB bound, and further shows that VQ-VIB agents also achieve that bound via self-play. Visualizing the emergent VQ- communication systems (Figure 6c-f) shows, as before, the structural changes in the effective codebook size as complexity increases (see Movie 2 for the continuous evolution of these systems), as well as the similarity between the VQ-VIB speaker color-to-signa mapping and the nearest WCS language (Figure 6d and f). This recapitulates the theoretical evolutionary process derived by [13], while additionally taking into account local agent interactions and a learned embedding space.
and train agents while annealing Àc- We set K 330 (the number of
This embedding space is particularly noteworthy, as it is grounded in human color perception and captures semantically meaningful information about the perceptual color space. At low complexity (Figure 6c), the embeddings reflect the irregularity of the CIELAB space due to the strong perceptual

---

Table 1: Quantitative evaluation of the color communication systems of annealed agents (VQ-VIB. One-hot), utility-only VQ-VIB agents, and the IB -B-opiimal systems from [13]. Human- gNID is the average dissimilarity (+STD) between human color naming systems and nearest agent system. Agent efficiency loss is the average deviation from optimality (+STDD of the agents' systems, measured with respect to the theoretical bound from [13]. For both measures, lower values are better.
VQ-VIB
One-hot
VQ-VIB (utility-only)
IB-optimal
Human-agent gNID
0.16 (0.01)
0.16 (0.00)
0.42 (0.02)
0.18 (0.10) N/A, by definition
Efficiency loss
0.03 (0.00)
0.05 (0.00)
0.11 (0.01)
salience of yellow (Figure 3c). At higher complexity (Figure 6e), the embeddings form huc circle, with achromatic colors (black and white) in the middle, as in CIELAB. Intriguingly, Shepard and Cooper [56] found that human similarity judgments for color words also recapitulate the hue circle, suggesting that VIB may potentially provide a possible mechanistic explanation for the emergence of grounded semantic representations both in humans and in artificial agents.
Finally, we measure the agents' deviation from the IB bound (efficiency loss, as in [13]) and their structural similarity to the nearest WCS languages (using gNID [13], dissimilarity measure for encoders) for both VQ-VIB and one-hot agents. Table shows that both VQ- (I- and one-hot agents learn human-like systems across the complexity spectrum, confirming the patterns in Figure 6. In addition, both are slightly better (lower human-agent gNID) compared to the IB-optimal systems from [13] that do not take into account utility. It is not surprising that both stochastic one-hot encoder and a VQ-VIB encoder can perform well within the IB framework, because the traditional IB formulation with discrete representations does not assume any typology the representation space. However, when assessed with respect to utility (Figure 6b), VQ- VQ-VIB outperforms the one-hot architecture, as in the particle world domains, suggesting that embedding space may be particularly useful for downstream decision making. Training only to maximize utility (A, = X = 0). however, causes VQ-VIB agents to converge diversity of human languages (Table 1, VQ- (utility-only); training utility-only one-hot agents led to similar findings; see Appendix F). This suggests that neither communicative efficiency alone, at least as captured by IB, nor utility alone, but rather their combination may be important for the emergence of human-like communication.
narrow range of communication that fails capture the
6 Conclusions
In this work, we have proposed an cntemattion-theorede framework for emergent communication that integrates two views: task-specific utility maximization, and general communicative constraints derived from the Information Bottleneck (IB) principle [18. 13]. To train agents within this framework, we have introduce a new method, VQ-VIB, that builds on VIB [19] and VQ-VAE [20], but also differs from them in ways that are particularly important for modeling human languages (Section 3.3). We have shown, across three types of test environments, that VQ- VIB agents can continuously evolve and adapt changing communicative pressures via annealing single tradeoff parameter, while learning grounded embedding spaces for communication that capture semantically meaningful information about the perceived environment, reminiscent of word embeddings. VQ-VIB also outperforms prior methods that support discrete (symbolic) communication, and in the domain of color, it can be aligned with human languages and possibly also with human similarity judgments. Natural extensions of our work include more complex domains and challenging tasks, as well as human-agent interaction experiments, which may further demonstrate advantages of VQ-VIB. More broadly, believe our framework could be useful for further developing and improving both emergent communication in artificial agents and models of human language evolution.
Acknowledgments and Disclosure of Funding
M.T. was supported by an Amazon Science Hub Fellowship. N.Z. was supported by a K. Lisa Yang Integrative Computational Neuroscience Postdoctoral Fellowship. We thank Rahma Chaabouni for useful discussions on their prior work, and our anonymous reviewers for their helpful feedback.
10

---

References
[1] Luc Steels and Tony Belpaeme. Coordinating perceptually grounded categories through lan- guage: A case study for colour. Behavioral and Brain Sciences, 28(4):469-89, 2005.
[2] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pages 2145- 2153, Red Hook, NY, USA, 2016. Curran Associates Inc.
[3] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettiv environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pages 6382-6393, Red Hook. NY, USA, 2017. Curran Associates Inc.
[4] Igor Mordatch and Picter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the Thirty-Second AAAI Conference Ar- tificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Con- ference and Eighth AAAI Symposium Educational Advances in Artificial Intelligence, AAAI' 18/IAAI 18/EAAI 18. AAAI Press, 2018.
[5] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[6] Angeliki Lazaridou and Marco Baroni. Emergent -gget communication in the deep learning arXiv preprint wwxXi::0006.22111 2020.
[7] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficiee encoding in emergent communication. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buee Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2019.
[8] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally" in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962-2967, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 18553
[9] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication in multi-agent reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Ahhé-BBue, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[10] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient namiing systems. Proceedings of the National Academy of Sciences, 118(12)ee0 (0(1)))22016669 (160066561111 18, 2021. doi: 10. 1..0777/paa.2211656666
[11] Edward Gibson, Richard Futrell, Steven Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Philip Levy. How efficiency shapes human language. Trends in Cognitive Sciences, 23:389-407, 2019.
[12] Charles Kemp, Yang Xu, and Terry Regier. Semantic typology and efficient com- munication. Annual Review of Linguistics, 4:109-128, 01 2018. doi: 10.1146.
[13] Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31): 7937-7942, 2018. doi: 10. 10.1073/pnas. 11005

---

[14] Noga Zaslavsky, Terry Regier, Naftali Tishby, and Charles Kemp. Semantic categories of artifacts and animals reflect efficient coding. In Proceedings of the 41st Annual Meeting of the Cognitive Science Society, 2019.
[15] Noga Zaslavsky, Mora Maldonado, and Jennifer Culbertson, Let's talk (efficiently) about us: Person systems achieve near-optimal compression. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society, 2021.
[16] Francis Mollica, Geoff Bacon, Noga Zaslavsky, Yang Xu, Terry Regier, and Charles Kemp. The forms and meanings of grammatical markers support efficient communica- tion. Proceedings of the National Academy of Sciences, 118(49):22025993111 2021. doi: 10. 1t.1077//naas22255991111 2025993118.
[17] Noga Zaslavsky, Karee Garvin, Charles Kemp, Naftali Tishby, and Terry Regier. The evolution of color naming reflects pressure for efficiency: Evidence from the recent past. Journal of Language Evolution, 04 2022. ISSN 2058-458X. doi: 10. 10.1099/jole/zzcc001. Izac001.
[18] Naftali Tishby, Fernando C. Pereira, and William Bialek. The Information Bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, 1999.
[19] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[20] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the st International Conference on Neural Information Processing Systems, NIPS' 17, pages 6309-6318, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978 9781510860964.
[21] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[22] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, meeting of SIGDAT, 2014. doi: 1/3115//1/d144111
a Special Interest Group of the ACL, pages
[23] Mycal Tucker, Huao Li, Siddharth Agrawal, Dana Hughes, Katia P. Sycara, Michael Lewis, and Julie A. Shah. Emergent discrete communication in semantic spaces. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 641 2021, virtual, pages 10574-10586, 2021.
[24] ukasz Kuciński, Tomasz Korbak, PawelKol odziej, and Piotr Mil oš. Catalytic role of noise and necessity of inductive biases in the emergence of compositional communication. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances Neural Curran Associates, Inc.,
Information Processing Systems, volume 34, pages 2021.
[25] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
[26] Claude E. Shannon. Coding theorems for discrete source with fidelity criterion. IRE Nat. Conv. Rec, 1(142-163):1, 1959.
[27] Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient auett- communication: An information bottleneck approach. In ICMI., pages 9908-9918. 2020.
12

---

[28] Matti Miestamo. On the feasibility of complexity metrics. In FinEst linguisties, proceedings of the annual Finnish and Estonian conference of linguisties, Tallinn, pages 11-26, 2004.
[29] Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. "LazImpa": Lazy and impatient neural agents learn communicate efficiently. In Proceedings of the 24th Conference Com- putational Natural Language Learning, pages 335-343, Online, November 2020. Association for Computational Linguistics. doi: 00018653/11/222..coom 1.26.
[30] Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of compositional language with deep generational transmission. CoRR, abs/ (bs/1994.00067, 2019.
[31] Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In International Conference on Learning Representations, 2020.
[32] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning, In International Conference Machine Learning, pages 3040-3049, PMLR, 2019.
[33] Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning to ground multi-agent communication with autoencoders. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15230-15242. Curran Associates, Inc., 2021.
[34] Naftali Tishby and Daniel Polani. Information theory of decisions and actions. In Vassilis Cutsuridis, Amir Hussain, and John G. Taylor, editors, Perception-Action Cycle: Models, Architectures, and Hardware, pages 601-636. Springer New York, New York, NY, 2011.
[35] Susanne Still and Doina Precup. An information theorettc approach to curiosity-drive rein- forcement learning. Theory in Biosciences, 131(3):13--148, 2012.
[36] Pedro A. Ortega and Daniel A. Braun. Thermodynamies as a theory of decision-making with information-procesiini costs. Proceedings of the Royal Society A: Mathematical, Physical and Fngineering Sciences, 469((153))200 20683, 2013. doi: 10. 10.10098/spa..211..63...
[37] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. In 32nd Conference on Uncertainty in Artificial Intelligence (UAI), 2016.
[38] Anirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Sergey Levine, and Yoshua Bengio. Transfer and exploration via the informa- tion bottleneck. In International Conference on Learning Representations, 2019.
[39] David Lewis. Convention: philosophical study. John Wiley & Sons, 2008
[40] Brent Berlin and Paul Kay. Basic Color Terms: Their Universality and Evolution. University of California Press, Berkeley and Los Angeles, 1969.
[41] Terry Regier, Paul Kay, and Naveen Khetarpal. Color naming reflects optimal partitions of color space. Proceedings of the National Academy of Sciences, 104(4):1436 1441, 2007. doi: 104.
10.
[42] Paul Kay, Brent Berlin, Luisa Maffi, William R. Merrifield, and Richard Cook. The World Color Survey. Stanford: Center for the Study of Language and Information, 2009.
[43] Mike Dowman. Explaining color term typology with an evolutionary model. Cognitive Science, 31(1):99-132, 2007.
[44] Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325 1388. 2017.
[45] Mikael Kägebäck, Emil Carlsson, Devdatt Dubhashi, and Asad Sayeed. A reinforcement- learning approach to efficient communication. PLOS ONE, 15(7):1-26. 07 2020.
13

---

[46] Toby Berger. Rate distortion theory; mathematical basis for data compression. Prentice-Har Englewood Cliffs, NJ, 1971.
[47] Peter Harremoës and Naftali Tishby. The Information Bottleneck revisited or how to choose good distortion measure. In IEEE International Symposium on Information Theory, pages 566-571 571, June 2007.
[48] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.
[49] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB. Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[50] Noga Zaslavsky. Information-Theune Principles in the Evolution of Semantic Systems. Ph.D. Thesis, The Hebrew University of Jerusalem, 2020.
[51] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pages 2252-2260, Red Hook, NY, USA, 2016. Curran Associates Inc.
[52] Jennifer Hu, Roger Levy, and Noga Zaslavsky. Scalable pragmatic communication via self- supervision. In ICML Workshop on Self Self-Supeevised Learning for Reasoning and Perception, 2021.
[53] WS Mokrzycki and M Tatol. Colour difference AE a survey, Machine Graphic and Vision, 8, 2012.
[54] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist re- inforcement learning. Mach. Learn., 8(3-4))229-256, may 1992. ISSN 0885-6125. doi: 10. 10.10077BF000992666
[55] Kenneth Rose. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proc. IEEE, 86(11):2210-22) 2239, 1998. doi: 10.1109/5. 726788.
[56] Roger N. Shepard and Lynn A. Cooper. Representation of colors in the blind, color-blind, and normally sighted. Psychological Science, 3(2):97-104, 1992. ISSN 09567976, 14679280.
[57] Eric Jang, Shixiang Gu, and Ben Poole.
Categorical reparameterization with gumbel-softmat In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings, 2017.
[58] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A contin- uous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See abstract, introduction, and findings
(b) Did you describe the limitations of your work? [Yes] See Section 6, in which we note that tested in limited domains and Appendix B, in which note that approximation we made may be improved upon in future work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section which we discuss how readers may misconstrue quantitative measures of languages assigning value one over the other.
14

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See footnote for link to code
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] See error bars plots and Table 1, Section All scatter plots directly included the raw results from all 5 random trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Training times and computational resources are listed in Appendix D.
4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the World Color Survey (WCS) data, well as the publicly-availate code by Zaslavsky et al.
[13]. Both sources are cited.
(b) Did you mention the license of the assets? [Yes] We note that both are publicly-availaby and link to them in Footnote 1.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] Code is linked to in footnote.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A] We only used the existing WCS dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We only used the existing WCS dataset.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "jYqe6kZEV40", "reviewer": "Reviewer_1FYj", "summary": "In this work, the authors introduced a new discretization method for endowing neural agents with a communication channel in a cooperative multi-agent framework. By adding auxiliary losses, each penalizing the utility, informativeness, and complexity of the generated language, testing on two MARL benchmark and a referential task, the authors study the conditions that led the communication protocol to be successful and to resemble properties of natural language as described by the IB (information bottleneck) principle.", "questions": "Lin et a. 2021 reports how difficult it can be to train fully decentralized agents for tasks that involve learning a communication protocol. Sharing gradients between agents is a form of communication that could not used in a fully decentralized setups. Have you considered or tried training your agents with some form of RL algorithms that would not involve any gradient passing between agents? Note that this is not to be intended as a missing experiment for this work but rather as something to be intended as future work.", "limitations": "The authors have adequately addressed limitations and negative societal impact of their work", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe the main strengths of this paper are in the introduction of VQ-IB and the analysis of the contribution of the additional losses in terms of complexity and informativeness.  The experimental setup is solid and the results support the idea that the introduced terms can steer the emergent languages to different complexity classes and make them similar to natural languages.\n\nI could not find any major weaknesses in this work. Useful addition to the paper would be a qualitative analysis for the navigation environments. The quantitative results are informative and support the main claim of the paper but a qualitative analysis might help shed a light on better understanding the language protocol. Interesting questions would be of this sort: \"are there specific messages the systematically mapped to actions of the environment e.g. go right, the target is in the top right corner, go there etc.\".  \n\nWhile I believe that the tested environments correctly corroborates the introduced hypothesis, it would be interesting to know whether the discretization method can scale to larger scale setup involving complex input like natural images. Havrylov and Titov 2017 and Dessi et al. 2021 showed that agents can learn a protocol describing natural images when trained from pixel input and without any supervised pretraining. VQ-IB is a drop in replacement in their methods (they both used gumbel softmax sampling) so it would interesting to see wether your results hold at that scale. Given that the way \"complexity\" is penalized in previous work is usually by changing vocabulary sizes it could be useful to test whether an auxiliary loss term would be more effective instead.\n\n\nFinally, I consider the introduced method and the presented results interesting, the analysis to be thorough and informative and believe this paper should be included in NeurIPS since it could be beneficial for the emergent communication community.\n\n-----\n\nMinor:\n\n- A brief description of the Proto architecture would be useful to understand its differences with the other architectures\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MNspLE303uT", "reviewer": "Reviewer_3YrT", "summary": "This paper analyzes the influence of three factors on languages emerging in communication games between simple VQ-VAE based agents. Specifically, the authors contrast the effect of promoting utility and \"informativeness\" versus reducing the complexity of the language. The authors demonstrate that informativeness generally improves success at the game, but at the detriment of complexity. An adequate tradeoff between these two factors seem to lead to \"human-like\" languages, at least as measured by a comparison to color-naming schemes.", "questions": "- What is semantic communication? Or rather what would the opposite of semantic communication be? Communication without meaning?\n- What is the purpose of VQ-VIB after? It seems to me that randomness in VQ-VIB after emulates a perturbation in the environment (on which the agent has no control), whereas randomness in VQ-VIB before models uncertainty from the agent (via a reparametrization trick). It is not clear to me why the former is interesting in the context of this work (in fact it is used much less than VQ-VIB before in the experiments.\n- What are $m$ and $\\hat m$ in section 3.1? Are they typos for $x$ and $\\hat x$?", "limitations": "I think there could be at the very least a more thorough discussion of the notion of complexity of a language (and the difficulty of defining a metric)", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the paper touches on interesting issues (the interplay between informativeness and complexity), I am having a hard time understanding what I should make out of it. The extreme simplicity of the tasks and languages considered, coupled with a debatable definition of complexity make it hard to understand the relevance of the results.\n\n\n### Strengths\n\n1. **Clear exposition**: the paper is rather clear throughout. The writing reads well and a number of well designed figures illustrate the author's points (in particular figure 1)\n2. **Well laid out experimental section**. I enjoyed the structure of the experiments section, first spelling out hypotheses, experimental settings and then discussing results.\n\n### Weaknesses\n1. **Notion of complexity**: In the paper, the complexity of a language is defined as the mutual information between \"inputs\" (what is communicated about) and signals (what is communicated). I am not entirely convinced by this notion of complexity for several reasons\n    - It seems to me that a language can have a high mutual information by virtue of being unambiguous: indeed if we write $I(X;C) = H(X) - H(X\\mid C)$, then for a fixed distribution over inputs, the mutual information is maximal for languages where $H(X\\mid C)=0$, in other words when any given message $c$ refers unambiguously to a single input $x$. To me this seems unrelated to complexity. For example emergent languages might be equally unambiguous, yet more \"complex\" in a different sense (eg. less compositional, using longer words...).\n    - More generally this is an odd definition in the context of human languages. Much ink has been spilled in linguistics on the topic of language complexity: syntactical complexity vs morphological complexity, are all human languages equally complex, is there a meaningful measure of language complexity, etc... There is very little mention of this literature which is a bit of a shame given that one of the objectives of the paper is to \"[demonstrate] how fundamental principles that are believed to characterize human language evolution may inform emergent communication in artificial agents.\" I recommend \"On the Feasibility of Complexity Metrics\" by Miestamo (2004) as a starting point for a discussion. It should also be mentioned that previous work in emergent communication has looked at other metrics which are related to complexity: length of the messages (\"“LazImpa”: Lazy and Impatient neural agents learn to communicate efficiently\", Rita et al. 2020), ease of cultural transmission (\"Emergence of Compositional Language with Deep Generational Transmission\", Cogwell et al. 2019, \"Compositional languages emerge in a neural iterated learning model\", Ren et al. 2020).\n\n    Overall it seems to me that the notion of complexity advocated by the author is more or less the entropy of the distribution of the messages, which I understand to be a proxy for the size of the lexicon (the number of utterances in the language). I am not sure if this is an interesting metric: arguably a key feature of human languages is that they allow for generating a potentially infinite number of utterances, and what differentiate them is how they are able to do so by the use of finite means (phonemes, words...)\n2. **Unclear takeaway**: What do the results of the paper mean? The two takeaways I read from the papers are that (1) \"principles that characterize language evolution can inform emergent communication\" and (2) \"penalizing complexity is necessary to avoid idiosyncratic languages\". I would contest that (1) is a novel insight because there is already plenty of work porting ideas from cognitive science and language evolution to emergent communication (see work on iterated learning, zipf's law in emergent languages, etc...). As for (2) I am not entirely convinced due to my issues with the definition of complexity used in the paper (outlined above).\n3. **Informativeness vs utility**: The distinction between informativeness and utility does not seem very useful in the context of the paper: indeed the two are not really contrasted in experiments. In a lewis signaling game (such as the color naming experiment) they are essentially equivalent. It would have been interesting to also vary along this dimension in experiments. Alternatively, I think the key claims of the papers could have been made by focusing on the opposition between informativeness and complexity (and leaving utility out of the picture).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5qsHUVQDqg", "reviewer": "Reviewer_kucm", "summary": "This work proposes to learn to communicate while balancing three different losses. The effectiveness of communication (utility as usual), the informativeness of communication as measured by autoencoding loss, and the complexity as measured by the entropy of the communication distribution. Furthermore, the authors propose a new algorithm VQ-VIB which integrates the information bottleneck with a discrete gradient estimator VQ-VAE. The authors demonstrate how to learn to communicate with their novel method and how to integrate all three losses into their method. \n\nThe authors test the three losses and their novel algorithm on two environments, a particle world where a sender must give a location to a receiver and the World Colour Survey, following Chaabouni et al, where agents must learn a vocabulary to communicate colours. In the former, the use MADDPG and compare to regular discrete, continuous, and proto communication. In the latter they compare to just discrete communication using the gumbel-softmax estimator. They find that the autoencoding loss improves performance, and the complexity loss can be used to reduce complexity. In the particle world, VQ-VIB can outpeform a regular discrete method for the same vocabulary size but not a continuous communication method. In the WCS, agents can learn an optimal language (complexity for effectiveness) as in Chaabouni et al and the complexity loss can vary across the range of complexities. In some configuration, VQ-VIB learns a language that better corresponds to the space of colours than regular gumbel-softmax.", "questions": "why use VQ-VIB after if it generates a continuous message and is worse than simple continuous messages?\n\nfor 5.1, you haven't demonstrated that VQ-VIB is better than one-hot, please test vocab size > 100\n- if performance increases as vocab size increases, why not increase vocab size more?\n- you can make the argument that VQ-VIB outperforms one-hot *at the same vocab size* but not at the sample complexity. If one-hot can reach the same performance/complexity at a larger vocab size then you can't say it completely outperforms it\n- furthermore, neither agent is reaching optimal performance!\n- if you can show than VQ-VIB outperforms one-hot accuracy/complexity for *any* one-hot hyperparameters, then your method becomes much more interesting and your results are stronger\n\nrelated to the above, Fig 4b shows that continuous communication (cont) clearly outperforms discrete communication (one-hot) but the only difference between the two channels should be the bandwidth\n- for a fairer comparison, you should increase the discrete vocabulary size (or use a channel of length > 1)\n\nwhy does Fig 4b show unconverged protocols? \n- ``complexity and reward measured at each increment of 1,000 training episodes from 3,000 to 10,000''\n- why are you measuring unconverged protocols during training? they are obviously going to be sub-optimal? \n\nfor 5.2, please test REINFORCE for discrete communication\n    - Chaabouni et al found that REINFORCE learns less complex protocols than Gumbel-Softmax which is what you use\n    - REINFORCE is as common (or more common) than GS for emergent communication, so it would be the better baseline for complexity\n\nFig 5a is suspicious\n- either 5a is too small and therefore we can't see the variance of results\n- or the learned protocols are at the exact optimal accuracy / complexity tradeoff and $lambda_I = 1$ is sufficient to achieve this (despite Chaabouni et al's results which show a variance when $lambda_I = 0$)\n- these results could imply the choice of a vocabulary size (or other hyperparameters) that guaranteed optimality (see Natural Language does not Emerge Naturally (Kottur et al)) and this graph should preferably me more zoomed in and likely should stick with the original vocabulary size 1024 used in Chaabouni et al\n\n\n### citation suggestions\n\nyour auxiliary loss for informativeness seems the same as AEComm (Learning to Ground Communication with Autoencoders, Lin et al, 2021) and may be worth a citation\n- notably, their experiments found that adding informativeness (AE) to utility (RIAL) reduces performance so your results are interesting and novel from that perspective!\n\nplease cite Concrete Distribution (Maddison et al, 2016) on top of Jang et al for the Gumbel-Softmax estimator (they are concurrent work)\n\n\n### clarification\nis the entropy regularization done in place of $\\lambda_c I(X,C)$ or on top of that loss? (I understood it to be in place of that loss)\n\nis the entropy regularization in one-hot equivalent to regular entropy regularization done in previous RL works (e.g. Mnih et al 2016, Lazaridou et al, 2018) or is it the opposite? Are you penalizing entropy or encouraging entropy?\n\n\nin 5.2.1 you meantion that VQ-VIB learn more complex communication, did you mean the opposite since we are trying to learn less complex, more effective communication", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe paper is both interesting and well-written. It contains many novel insights and experiments but overall, it is not well organized and the overall message and novelty are not clear. If the authors could add some experiments listed in the questions section and perhaps formulate their contribution more clearly, I would be likely to recommend acceptance.\n\nTo begin, the paper proposes two separate ideas. The first is the three axes (or two auxiliary losses) utility, informativeness, and complexity. Utility is a given in emergent communication, and the latter two have been investigated separately before (informativeness is autoencoding in AEComm, complexity has been penalized as far back as Mordatch and Abbeel, 2017). The second idea is VQ-VIB which is a clever combination of VQ-VAE and VIB and does seem quite interesting. The issue is that these two contributions are quite separate as far as I can tell and neither contribution is very strong.\n\nFor VQ-VIB, although the algorithm is interesting it is not empirically verified in depth. The authors claim it achieves greater utility and informativeness for the same complexity but this is only true in the particle envs and not in the WCS (and could depend on hyperparamters). Furthermore, they do not do an exhaustive comparison of baselines (REINFORCE instead of GS, trying a wider range of vocabulary sizes, etc..). Where the results are better, they are not hugely better and for a novel algorithm it should be investigated on more environments or more in depth.\n\nFor the tradeoff of utility, informativeness, and complexity, the analysis is quite good and the results are interesting. Sadly, the novelty is not quite there. Demonstrating that a complexity loss can reduce complexity is not very novel and does not require so much attention (both Fig 4b and 5). Chaabouni et al's work on WCS was novel because it showed that the natural inductive bias of neural networks was similar to that of humans (low complexity, high informativeness). This work explicitly adds those factors as losses and shows it can find the optimal curve, which is again not very novel. Still, the auto-encoding loss is shown to be effective and this is quite exciting but it is not investigated in depth and no previous benchmarks are improved upon (e.g. MADDPG's particle envs)\n\nSimply showing that these losses work as intended is not particularly interesting and as a reviewer, I am not fully convinced of the value of VQ-VIB. Because of the quantity of new ideas here, none of them seem to be investigated in enough depth either and many results are relegated to the appendix with only one or two sentence summaries. This is not very convincing and the overall paper feels like it brushes over key contributions. \n\nOverall, a stronger contribution would be to focus on one of the two proposed ideas or demonstrate that both ideas are crucially necessary towards some goal. E.g. if the authors were working towards human-AI communication and demonstrate that both techniques (VQ-VIB, and losses) improve zero-shot human understanding of self-play messages. In that case, contributions feel like two separate but important tools towards a single goal. In the current case, there is an interesting investigation but none of the results are sufficiently convincing or novel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept