On the Correspondence between Compositionality and Imitation in Emergent Neural Communication
Year: 2023

Abstract:
Compositionality is a hallmark of human language that not only enables linguistic generalization, but also potentially facilitates acquisition. When simulating language emergence with neural networks, compositionality has been shown to improve communication performance; however, its impact on imitation learning has yet to be investigated. Our work explores the link between compositionality and imitation in a Lewis game played by deep neural agents. Our contributions are twofold: first, we show that the learning algorithm used to imitate is crucial: supervised learning tends to produce more average languages, while reinforcement learning introduces a selection pressure toward more compositional languages. Second, our study reveals that compositional languages are easier to imitate, which may induce the pressure toward compositional languages in RL imitation settings.

heading: 1 Introduction
text: Compositionality, a key feature of human language, makes it possible to derive the meaning of a complex expression from the combination of its constituents (Szabo, 2020). It has been suggested that more compositional languages are easier to acquire for both humans and artificial agents (Raviv et al., 2021; Li and Bowling, 2019; Ren et al., 2020; Chaabouni et al., 2020). Therefore, to better understand the factors underlying language transmission, it is crucial to understand the relationship between ease-of-acquisition and compositionality.
We study the link between compositionality and ease-of-acquisition in the context of emergent communication. In this setting, two deep artificial agents with asymmetric information, a Sender and a Receiver, must develop communication from scratch in order to succeed at a cooperative game (Havrylov and Titov, 2017; Lazaridou
∗Work done while visiting LATTICE at the CNRS / École Normale Supérieure.
et al., 2017; Lazaridou and Baroni, 2020). We will refer to this mode of language learning, in which agents develop language via feedback from mutual interaction, as communication-based learning.
Several studies have linked compositionality to ease-of-acquisition in communication-based learning. Chaabouni et al., 2020 show compositionality predicts efficient linguistic transmission from Senders to new Receivers. Conversely, Li and Bowling, 2019 re-pair a Sender periodically with new Receivers, and show this ease-of-teaching pressure improves compositionality.
Communication-based learning is not the only possibility for language learning, however. Humans also crucially acquire language through imitation-based learning, in which they learn by observing other humans’ language use (Kymissis and Poulson, 1990). Ren et al., 2020 and Chaabouni et al., 2020 employ imitation learning, where in the first study, agents undergo a supervised imitation stage before communication-based learning, and where in the second, agents alternate between communication-based learning and imitating the best Sender. However, the dynamics of imitation are not the focus in either study. For such an important vehicle of language acquisition, imitationbased learning thus remains under-explored in the emergent communication literature.
We extend these lines of inquiry to systematically investigate compositionality in imitationbased learning.1 Our contributions are as follows: (1) We show that imitation can automatically select for compositional languages under a reinforcement learning objective; and (2) that this is likely due to ease-of-learning of compositional languages.

heading: 2 Setup
text: We study imitation in the context of referential communication games (Lewis, 1969). In this setting, a
1...for artificial agents. We do not test theories of human imitation learning.
12432
Sender agent observes an object x and transmits a message m to a second Receiver agent. Using this message, the Receiver performs an action for which both agents receive a reward. Over the course of the game, agents converge to a referential system (x,m), which we refer to as an emergent language.
Measuring Compositionality Evaluating compositionality in emergent languages is not straightforward given their grammars are a-priori unknown. Therefore, we quantify compositionality using topographic similarity (topsim) (Kirby and Brighton, 2006), a grammar-agnostic metric widely applied to emergent languages in the literature. Topsim is defined as the Spearman correlation ρ between Euclidean distances in the input space and Levenstein distances in the message space– that is, it captures the intuition that nearby inputs should be described with similar messages. While we consider other compositionality metrics such as positional disentanglement (Chaabouni et al., 2020), we focus on topsim due to its high correlation with generalization accuracy (ρ = 0.83) (Rita et al., 2022b). See appendix A.3 for extended experiments on compositionality metrics and generalization.

heading: 2.1 Imitation Task
text: To investigate whether compositional languages are selected for in imitation, we posit an imitation task where one new Imitator Sender or Receiver simultaneously imitates several Expert Senders or Receivers with varying topsims. Both Sender and Receiver agents are parameterized by single-layer GRUs (Cho et al., 2014) that are deterministic after training (see appendix B for implementation).2 While we explore imitation for both agents, we focus on Sender imitation in the main text, and extend to Receiver imitation in appendix E. A minimal example of imitation learning with only one Expert Sender-Receiver pair is shown in fig. 1.
The Sender imitation task is as follows: given a set of k Expert Senders, we train an identical, newly initialized Sender on the Experts’ inputs and outputs (x,m). That is, for each round of training, all k Experts as well as the Imitator Sender receive input x and output m(1) · · ·m(k) and mI , respectively. The Imitator is then tasked to minimize the difference between their output and a uniform mixture of the k Expert outputs.
2Experiments are implemented using EGG (Kharitonov et al., 2021). Code may be found at https://github.com/chengemily/EGG/tree/imitation.
Dataset Data in the imitation task consists of inputs and outputs of pairs of Expert agents trained to convergence on a communication game– in our case, the two-agent reconstruction task of Kottur et al. (2017). To generate the Experts, we pre-train N = 30 Sender-Receiver pairs on this reconstruction task to high validation accuracy (0.99± 0.01) (task and training details in appendix A).
Expert training produces the following data: 1) inputs x; 2) messages m corresponding to Expert Senders’ encodings of x; and 3) outputs x̂, the Expert Receivers’ reconstructions of x given m.
Each input x denotes an object in an “attributevalue world", where the object has natt attributes, and each attribute takes nval possible values. We represent x by a concatenation of natt one-hot vectors, each of dimension nval. On the other hand, messages m are discrete sequences of fixed length L, consisting of symbols taken from a vocabulary V . We set natt = 6, nval = 10, |V | = 10, and L = 10, corresponding to a relatively large attribute-value setting in the literature (Table 1 of Galke et al. (2022)).
We split the input data (n = 106) into a training set and two holdout sets. Similar to Chaabouni et al. (2020), we define two types of holdouts: a zero-shot generalization set (n = 354294), where one value is held-out during training, and an indistribution generalization set (n = 531441). The training set, on which we both train and validate, represents 1% of in-distribution data (n = 5315). These data splits are used in Expert training and are
inherited by the imitation task (see appendix B.2 for details on generating the data split).
Imitation learning algorithms While imitation is classically implemented as supervised learning, we test two imitation learning procedures: 1) supervised learning (SV) with respect to the cross-entropy loss between Imitator and Expert outputs; and 2) reinforcement learning with the REINFORCE algorithm (RF) (Williams, 1992), using per-symbol accuracy as a reward. When using REINFORCE, we additionally include an entropy regularization term weighted by λ to encourage exploration, and subtract a running mean baseline from the reward to improve training stability (Williams and Peng, 1991). See appendix D for loss functions and B.2 for detailed hyperparameter settings.

heading: 2.2 Evaluation
text: To evaluate properties of imitation learning, we identify three descriptors of interest: validation accuracy, ease-of-imitation, and selection of compositional languages.
Accuracy We evaluate imitation performance between an Imitator and Expert by the average persymbol accuracy between their messages given an input. When using REINFORCE, training accuracy is computed using the Imitators’ sampled output while validation accuracy is computed using the Imitators’ argmax-ed output.
Ease-of-imitation We evaluate ease-of-imitation of a language two ways. First, imitation sample complexity (T ): the number of epochs needed to reach 99% validation accuracy, and second, imitation speed-of-learning (SOLI ): the area under the validation accuracy curve, cut-off at t epochs chosen by visual inspection of convergence.
Selection of compositional languages Sender imitation consists of learning one-to-one input-tomessage mappings from a sea of one-to-many Expert mappings. Then, the Imitator’s language will consist of a mixture of Expert languages, where the mixture weights reveal the extent of selection.
In this mixture, we proxy the Imitator’s learned weight for an Expert as the proportion of messages in the training set for which Imitator accuracy on the Expert message is the highest. Note that the coefficients may not add to one: if the highest Expert accuracy for a message does not exceed chance (10%), we consider the message unmatched.
To quantify selection, we use the intuition that selection corresponds jointly to peakedness and asymmetry in the learned distribution over Expert languages sorted by topsim. We evaluate peakedness using the Shannon entropy and asymmetry using Fisher’s moment coefficient of skew of Expert weights. Formally, let there be k Experts, where Experts are sorted in ascending order of topsim (Expert i=1 is the least and i=k is the most compositional, respectively). The Imitator learns a mixture of the Expert languages with weights W := (wi)1≤i≤k (normalized). Given W , we evaluate peakedness with:
H(W ) = − k∑
i=1
wi log(wi). (1)
To quantify asymmetry of expert weights, we estimate the Fisher’s moment coefficient of skew:
µ̃(W ) = 1
k
k∑
i=1
( wi − µ
σ
)3 , (2)
where µ is the mean and σ is the standard deviation of W . A skew of 0 implies perfect symmetry, positive skew corresponds to a right-tailed distribution, and negative skew corresponds to a left-tailed distribution. Intuitively, the more negative the skew of the Expert weights, the more weight lies on the right side of the distribution, hence the greater “compositional selection effect".
We proxy selection, then, by a negative skew (more weight assigned to high-topsim Experts) and low entropy (peakedness) in the Expert weight distribution.

heading: 3 Imitation and Selection of Compositionality
text: We present results for imitation on mixtures of k = 2-5 Expert Senders. First, we generate 30 Expert languages from the referential task, initially considering Expert distributions corresponding to evenly-spaced percentiles of topsim, including the minimum and maximum (0.26, 0.43). For example, when k = 3, we take the lowest, 50th percentile, and highest-topsim languages. All results are aggregated over 5 random seeds after 2000 training epochs.
We find that (1) whether Imitators prefer compositional Experts depends crucially on the learning algorithm: imitation by reinforcement results in marked compositional selection compared to
supervision; and (2) compositional selection also depends on variance of expert topsims, λ entropy regularization coefficient, and number of Experts.
The distribution of learned Expert weights in fig. 2, as well as imitation validation accuracy curves in fig. C.2, evidence that in imitation by supervision, the empirical mixture is closer to uniform than when imitating by reinforcement. Otherwise, when optimizing using reinforcement, the Imitator selects more compositional languages.
The shape of the Expert weight distribution is tempered by the entropy regularization coefficient λ: smaller λ results in greater compositional selec-
tion (that is, lower entropy and more negative skew) of the weight distribution (fig. 3). At the limit, imitation by supervision results in the highest entropy and the skew that is closest to zero.
We then test the effect of Expert topsim distribution asymmetry on the learned weights. To do so, for each k > 2, we generate 10 Expert topsim distributions with varying skew, following the procedure outlined in appendix D.2 (when k = 2, skew is mechanically 0). We find that for both REINFORCE and supervision, holding k equal, the skew and entropy of the learned Expert weight distribution are robust (i.e., not correlated) to the skew of the underlying Expert topsim distribution (fig. D.2). This is desirable when imitating by reinforcement and undesirable when imitating by supervision: for example, consider Expert topsim distributions [low high high] (skew< 0) and [low low high] (skew> 0). In both cases, REINFORCE will select a high-topsim Expert, and supervision will weight all Experts equally, that is, supervision is unable to de-select poor topsims.
Using all Expert topsim distributions generated so far (those where topsim ranks are evenly spaced, and those exhibiting varying skews), we investigate the effect of topsim distribution spread, quantified by standard deviation, on the learned weights. In fig. 4, we note a significant negative effect of Expert topsim standard deviation on the degree of compositional selection. That is, the more disperse the Expert topsims, the more the Imitator can differentiate between and select compositional Experts (shown by a more negative skew in learned Expert weights). Though this correlation is highly statistically significant for both REINFORCE and supervision, the effect is ∼ 8x greater for REINFORCE, demonstrating that the spread between expert compositionalities plays a more important
role in the degree of selection by reinforcement. Finally, selection is less salient as the number of Experts increases, seen by the increasing entropies and skews of Expert weights (figs. 3 and D.3). Results for k > 3 may be found in appendix D.
Understanding why REINFORCE selects for compositional languages The different results between the optimization algorithms correspond to inherent differences in learning objective. Successful imitation minimizes the Kullback-Leibler divergence between the Imitator πI and the Expert policies πE ; supervision is classically known to minimize the forward KL divergence DKL(πE ||πI), while reinforcement minimizes the reverse KL divergence DKL(πI ||πE) with respect to πI . That is, imitation by supervision is mean-fitting while imitation by reinforcement is mode-fitting– the former learns a uniform mixture of Expert languages (see appendix D.4 for proof), and the latter selects the best Expert language.

heading: 4 Speed-of-Imitation May Explain Compositional Selection
text: Thus far, we have seen that imitation by reinforcement selects compositional languages. This is likely because higher topsim languages are easier to imitate. We establish a positive and statistically significant relationship between topsim and ease-ofimitation, expanding the explorations in Ren et al. (2020); Li and Bowling (2019); Chaabouni et al. (2020) (see appendix C for experimental details).
We evaluate ease-of-imitation using k = 1, after t = 500 (SV) and 2000 epochs (RF), where t is chosen based on validation accuracy convergence. Correlations between topsims of 30 Expert languages and Imitator performance (averaged over three random seeds) are shown in table 1. We find that for both imitation by supervision and reinforcement, topsim is (1) significantly negatively correlated to imitation sample complexity T ; (2) significantly positively correlated to speed-of-imitation SOL.
Moreover, correlations between topsim and easeof-imitation are stronger than those between Expert validation accuracy and ease-of-imitation (table C.1). This suggests that the positive relationship between compositionality and ease-ofimitation is not due to a confound of high validation accuracy.

heading: 5 Discussion
text: Having (1) demonstrated a selection of compositional languages in imitation by reinforcement; (2) established a significant correlation between topsim and ease-of-imitation; we offer the following explanation for compositional selection: mode-seeking behavior in reinforcement learning exploits easeof-learning of compositional languages, resulting in a selection of compositionality.
While both imitation and ease-of-learning of compositional languages have been instrumentalized in population training, they are engineered in a top-down way: in Chaabouni et al. (2022), agents imitate the best-accuracy agent, who is algorithmically designated as the teacher; in Ren et al. (2020), imitation is stopped early to temporally select compositional features.3 Our work, using basic RL principles, proposes an alternative mechanism that selects compositional languages while requiring minimal engineering and assumptions.
Selection by RL imitation, using the same easeof-learning argument, applies to not only compositionality but also potentially to other traits, e.g., language entropy or message length. That is, RL imitation naturally promotes any learnability advantage among candidate languages without manual intervention, while agnostic to the signaling system. This may then be leveraged alongside communication-based learning in population-based emergent communication, where imitation would persist easy-to-learn linguistic features.

heading: Limitations
text: There are several limitations to our work. First, although we choose the attribute-value dataset due to its high degree of interpretability and control, we acknowledge that its simplicity limits the impact of our findings. Though imitation
3We did not succeed in replicating results in Ren et al. (2020) (see appendix C).
by reinforcement is a data-agnostic mechanism, we have yet to explore how it behaves in more complex settings, such as using naturalistic image inputs or embodied communication. We defer to Chaabouni et al. (2022); Galke et al. (2022) for further discussion on scaling up communication settings.
A second limitation of our results is that we do not explore how imitation-based learning scales to k > 5 Experts. In particular, our hyperparameter regime handles up to around k = 5 Experts– very preliminary analyses on k ≥ 10 Experts suggest a need to also scale up hyperparameters such as agent size and communication channel capacity. When training agents to imitate, one must therefore consider feasibility of the learning problem– for example, as a function of the imitation network topology, communication channel size, agent size, etc– in order for training to converge.
Finally, although our work is inspired by imitation learning in humans, the extent to which simulations explain human linguistic phenomena is not clear. We intend for our work to only serve as a testbed to understand communication from a theoretical perspective.

heading: Ethics Statement
text: Because our work uses synthetic data, it has little immediate ethical impact. However, our work may enable large populations of communicating agents down the line, which could have a range of civilian or military purposes.

heading: Acknowledgements
text: We would like to greatly thank Marco Baroni for feedback on experiments and manuscript; Paul Michel and Rahma Chaabouni for early feedback on research direction; the three anonymous reviewers, Jeanne Bruneau-Bongard, Roberto Dessi, Victor Chomel, Lucas Weber and members of COLT UPF for comments on the manuscript. M.R. also would like to thank Olivier Pietquin, Emmanuel Dupoux and Florian Strub.
This work was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir" program, reference ANR-19-P3IA0001 (PRAIRIE 3IA Institute), and by the ALiEN (Autonomous Linguistic Emergence in Neural Networks) European Research Council project no. 101019291. Experiments were conducted using HPC resources from TGCC-GENCI (grant
2022-AD011013547). M.R. was supported by the MSR-Inria joint lab and granted access to the HPC resources of IDRIS under the allocation 2021- AD011012278 made by GENCI.

heading: A Expert Training

heading: A.1 Reconstruction Task
text: In the reconstruction task, a Sender observes an object with several attributes, encoding it in a message to the Receiver, and the Receiver decodes this message to reconstruct the object. Formally, 1. The Sender network receives a vector input x
and constructs a message m of fixed length L. Each symbol is taken from the vocabulary V = {s1, s2, · · · , s|V |}.
2. The Receiver network receives m and outputs x̂, a reconstruction of x.
3. Agents are successful if x̂ = x.
Optimization In the reconstruction task, the cross-entropy loss is computed between x and x̂, and backpropagated directly to the Receiver. The same loss is propagated to the Sender via REINFORCE. When training with REINFORCE, we also employ an entropy regularization coefficient λ and subtract a running mean baseline from the reward to improve training stability.
Let the Sender policy be πS and the Receiver be πR. Let xi ∈ {0, 1}nval refer to the one-hot vector in x indexed by i, which corresponds to one attribute. Then, the Receiver’s supervised loss LR is as follows:
LR(m,x) = 1 natt
natt∑
i=1
CE(xi, π R(m)i). (3)
Let the Sender reward at time t be rt = −LR(πS(x), x), and let µt be a running mean of rt. Then, the Sender’s REINFORCE policy loss LS at time t is as follows:
LS(x) = (−rt − µt) log πS(x)− λH(πS(x)). (4)
Finally, loss is optimized by Adam’s default parameters (β = 0.9, 0.999), with a learning rate of 0.005.

heading: A.2 Experimental Details
text: We train 30 Expert pairs on the reconstruction task over 1000 epochs. Expert pairs converge to high validation accuracy and generalize to the indistribution set well (statistics in table A.2).

heading: A.3 Expert Compositionality Distributions
text: We considered using topsim, positional disentanglement (posdis) (Chaabouni et al., 2020), bagof-symbols disentanglement (bosdis) (Chaabouni et al., 2020), and context independence (ci) (Bogin et al., 2018) for our experiments (see fig. A.1 for distributions). However, as a fundamental reason we care about compositionality is due to its link to linguistic generalization, we focus on topsim, which we found has the highest correlation with generalization accuracy on the reconstruction task (table A.1).

heading: Topographic similarity and generalization
text: Similar to Rita et al. (2022a); Auersperger and Pecina (2022) and in contrast to Chaabouni et al. (2020); Kharitonov and Baroni (2020), we find that correlations between topsim and both indistribution and zero-shot generalization on the reconstruction task are high, and highly significant (α = 1e-2): Spearman’s ρ = 0.83 and Pearson’s R2 = 0.81 for in-distribution generalization, and ρ = 0.81, R2 = 0.78 for zero-shot generalization. This correlation is stronger than that between generalization and validation accuracy, where ρ = 0.75 for in-distribution generalization and ρ = 0.73 for zero-shot generalization (α = 1e-2). Furthermore, the correlation between topsim and validation accuracy is only ρ = 0.57 (α = 1e-2) suggesting that the relationship between generalization and compositionality is not explained by high validation accuracy.4 Our results support the stance in Auersperger and Pecina (2022) that compositionality, when evaluated on a suitably large dataset, indeed predicts generalization.
B Implementation Details

heading: B.1 Model Architecture
text: Both agents are single-layer recurrent neural networks that are deterministic after training.
4We do not report the Pearson R2 for Expert validation accuracy as the its distribution violates normality assumptions according to a Shapiro-Wilk non-normality test (α = 1e-3)).
The Sender is a single-layer GRU (Cho et al., 2014) containing a fully-connected (FC) layer that maps the input x to its first hidden state (dim=128). A symbol is then generated by applying an FC layer to its hidden state, and sampling from a categorical distribution parameterized by the output. We include LayerNorm (Ba et al., 2016) after the hidden state to improve training stability. Then, the next input to the GRU is the previous output, which is sampled during training and argmax-ed during evaluation. This input is fed through an embedding module (dim=128), which then gets fed into the GRU. The first input is the [SOS] token, and the Sender is unrolled for L = 10 timesteps to output symbols comprising a message. Only in imitation training, when unrolling the Imitator Sender, we take the Expert Sender’s previous output to be the Imitator’s next input so that the Imitator learns an autoregressive model of the Expert language.
The Receiver has a similar architecture to the Sender. It consists of an FC symbol-embedding layer, a GRU with LayerNorm (hidden dim=128), and an FC output head. The first hidden state is initialized to all zeros, then the FC-embedded symbols of the Sender message are sequentially fed into the GRU for L = 10 timesteps. We pass the GRU’s final output through a final FC layer and compute the Receiver’s distribution over objects on the result, which we interpret as a concatenation of natt probability vectors each of length nval.

heading: B.2 Hyperparameter settings
text: Hyperparameters tested may be found in table B.1. These hold for all experiments unless explicitly stated otherwise.
Dataset splits Of the n = nnvalatt = 106 datapoints in the entire dataset, the in-distribution set has size 106 ∗ (0.9)6 = 531441, and we randomly sample 1% to be the training set, (n = 5315), and
delegate the rest to the generalization set (n = 526126). Finally, the zero-shot generalization set consists of inputs where one attribute assumes the held-out value, and other attributes take on seen values (n = 354294).
B.3 Implementation Details
Experiments were implemented using PyTorch and the EGG toolkit (Kharitonov et al., 2021). They were carried out on a high-performance cluster equipped with NVIDIA GPUs. The number of GPU-hours to run all experiments is estimated to be between 50 and 100.

heading: C Supplementary Material: Ease-of-Imitation
text: In the compositionality vs. ease-of-imitation experiments, we train newly initialized Imitator pairs
on each Expert pair over 500 epochs for supervision and 2000 epochs for reinforcement, aggregating over 3 random seeds. The number of training epochs is chosen by visual inspection of validation accuracy convergence. We note that, when imitating by both reinforcement and supervision, there is no initial increase in topsim followed by a convergence to Expert topsim (fig. C.1), contrary to what is observed in (Ren et al., 2020).
For imitation by reinforcement, we use an entropy coefficient of λ = 0.1 for both Sender and Receiver. Comparing SOL and T for both Sender and Receiver to other compositionality metrics (table C.1), we see that topsim is generally most correlated with sample complexity and speed-oflearning. For the opposite reason, we did not move ahead with, e.g., experiments on positional disentanglement.

heading: D Supplementary: Imitators Select Compositional Languages to Learn

heading: D.1 Optimization
text: In the imitation task, we test both direct supervision and REINFORCE. Importantly, when doing a forward pass for the Sender during training, we feed it the Expert symbol from the previous timestep as input so that the Sender learns an autoregressive model of language. Hence, define the Imitator policy πIj as in appendix D.4.
In the direct supervision setting, for the Sender producing a distribution over messages πI given x and a given Expert i producing message m(i), where mj is the jth symbol of m(i), the overall loss for a uniform mixture of k Expert Senders is the following cross-entropy loss:
LISV (x) = k∑
i=1
L∑
j=1
CE ( m
(i) j , π I j
) . (5)
In the REINFORCE setting, we use accuracy persymbol as a reward for the Sender, with entropy regularization and mean baseline.5 For Expert i, this corresponds to a reward r(i) of
5We also tried REINFORCE using negative cross-entropy loss as a reward, but found training to be unstable.
r(i) = 1
L
L∑
j=1
Acc ( m
(i) j , π I j
) (6)
and a policy loss of
LI,(i)RF (x) = (−r(i) − µt) log πI(x)− λH(πI(x)), (7)
per Expert, which is averaged over Experts to produce the mixture-policy loss. This is optimized by Adam with a learning rate of 0.005.

heading: D.2 Sampling Sender Expert Distributions
text: To test the effect of the shape (skew, standard deviation) of the Expert topsims on imitation, we define a set of 10 distributions for each setting of k > 2 Experts, noting that when k = 2, the skew is mechanically equal to 0.
For interpretability, we hold the endpoints of the distributions equal at the minimum and maximum possible topsims (0.26, 0.43) for all distributions and values of k. Then, we sample the median M of the 10 distributions evenly from 0.26 to 0.43. Then, we fill the other k − 3 points to create a uniform distribution with mean M. If the median is less than the average topsim, then the left endpoint of this uniform distribution is the minimum topsim. Otherwise, it is the maximum topsim.
D.3 Effect of population size on learned Expert Sender weights
We find that the selection effect decreases as the number of Experts increases, i.e. the Expert weight distribution looks increasingly uniform (fig. D.2). We offer two possible explanations: (1) the harder learnability of this problem given our hyperparameter regime– this is suggested by the lower maximum validation accuracy achieved on any one Expert in RF– or (2) the (mechanically) smaller variance between values, holding endpoints equal, as we increase the number of agents. Notably, the purpose of this work is not to scale up to imitation in large populations of agents; we delegate the problem of operationalizing RL imitation at scale to future work.
D.4 Learning a uniform mixture of policies Claim A Sender that imitates a uniform mixture of k Expert Senders will output a uniform mixture of the k Expert languages.
Proof Let π(1) · · ·π(k) be k Expert Senders and let πI be the Imitator Sender. For each position in a message, agents produce a probability distribution over all possible symbols in V . Recall that the Expert Senders are deterministic at evaluation time. Given an input x, we write m(i)j as the value of the jth position in the message m(i) produced by Expert i.
For the Imitator Sender, we write
πIj := π I ( m (i) j−1 ; x ) ∈ [0, 1]|V |
as the probability distribution over possible symbols in position j of a message produced by the Imitator agent, given the previous output symbol m
(i) j−1 of Expert i. The k th index of πIj , or π I j [k], gives the Imitator agent’s probability of symbol k at position j in the message.
The ideal Imitator πI∗ minimizes the crossentropy objective between its messages and that
of a uniform mixture of k Expert Senders. Formally,
πI∗ = min πI
k∑
i=1
L∑
j=1
CE ( m
(i) j , π I j
)
= min πI
k∑
i=1
L∑
j=1
− log πIj [m(i)j ]
= max πI
k∑
i=1
L∑
j=1
log πIj [m (i) j ]
= max πI
k∏
i=1
L∏
j=1
πIj [m (i) j ]
subj. to ∑
k∈V πIj [k] = 1.
whose unique solution is πIj [m (i) j ] = π I j [m (l) j ] ∀j ∈ NL, ∀i ̸= l ∈ Nk, i.e. a uniform distribution over Expert languages. 2

heading: E Receiver imitation

heading: E.1 Setup
text: The Receiver imitation task is as follows: given a set of k Expert Receivers and their corresponding Senders, we train an identical, newly initialized Receiver on the Experts’ inputs and outputs (m, x̂). That is, for each round of training, all k Experts as well as the Imitator Receiver receive input m, or the output of Expert Sender given x, and output x̂(1) · · · x̂(k) and x̂I , respectively. Imitators are then tasked to minimize the difference between their output and a uniform mixture of the k Expert outputs.
The architecture for the Receiver agent may be found in B.1.
Optimization Similar to in Sender imitation, we test a supervised learning and a reinforcement imitation learning setting. For supervised learning, the Receiver imitation loss is equal to the crossentropy loss between its output and the Expert Receiver’s output given the same corresponding Expert Sender’s message m. Then, the loss over the entire mixture is the average cross-entropy loss per Receiver, aggregated across Expert Receivers.
For REINFORCE, the Receiver reward is similar to the Sender reward– analogous to the per-symbol accuracy, it is the per-attribute accuracy. We compute the corresponding policy loss (using a mean baseline per Expert and λ defined in table B.1), and
average over all Experts to get the overall policy loss for the Receiver.
E.2 Imitation and Selection of Compositionality
With the large communication channel size typical of emergent communication games, we can expect little Expert message collision. Then, in this setting, Receiver imitation consists of learning a many-toone mapping of messages to outputs, obviating a real need for selection if the goal is to maximize eventual communication accuracy. Indeed, we find that Imitator Receivers learn to be multilingual, achieving high validation accuracy on all Experts, and especially in the supervised setting.
Figure E.1: For the Receiver imitation, the entropy (left) and skew (right) (±1 standard deviation) of learned Expert weights for k = 2–5 Experts. We plot one row per Expert for legibility as the entropy ranges are quite different for different values of k. Values generally increase as exploration (λ) increases, attaining maxima in the supervision setting (SV).
Figure E.2: Receiver Imitator’s validation accuracies per-Expert on k = 2-5 (top to bottom) languages, where the topsim of Expert languages ranges evenly from 0.26 to 0.43. Imitation by supervision results in the highest and most uniform validation accuracies, and increasing the entropy coefficient λ in imitation by reinforcement increases the uniformity of validation accuracies. The y-axis is cut below at 0.4 for legibility.
We do note, however, greater differentiation in validation accuracy, as well as speed-of-learning, between Experts of varying compositionality when using reinforcement compared to supervision, and again influenced by the entropy coefficient λ (figs. E.1 to E.3).
How one operationalizes Receiver imitation then depends on one’s goal: for example, if the goal is to maximize communication accuracy in a population of communicating agents, then we want to have “tolerant" Receivers, and imitation by supervision allows the Receiver to achieve the highest validation accuracy on all languages. However, if we want to bottleneck the compositionality of the language in the population, we want to have more “selective" Receivers, and imitation by reinforcement may be more appropriate.

heading: E.3 Speed-of-Imitation May Explain Compositional Selection
text: Results for the Sender also hold for the Receiver; see section 4 for the analogous comments.
Figure E.3: For k = 2 (top left), k = 3 (top right), k = 4 (bottom left) and k = 5 (bottom right) Experts, validation accuracy curves of the Imitator Receiver on Expert messages over 7000 epochs of imitation training. For each setting of k, reinforcement is shown on the left and supervision on the right. Note (1) the greater dispersion of validation accuracy when training by reinforcement compared to be supervision; (2) lower validation accuracy on the poorest topsim Expert in RF compared to SV.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section
3 A2. Did you discuss any potential risks of your work? Ethics section, limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section B
3 B1. Did you cite the creators of artifacts you used? Section 2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section B
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix
C 3 Did you run computational experiments? All sections
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 2, Sections A-D
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 3, A-D
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? An environment.yml file will be release with the code upon de-anonymization.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Notably, the purpose of this work is not to scale up to imitation in large populations of agents; we delegate the problem of operationalizing RL imitation at scale to future work. D. 4 Learning a uniform mixture of policies Claim A Sender that imitates a uniform mixture of k Expert Senders will output a uniform mixture of the k Expert languages. Proof Let π(1) · · ·π(k) be k Expert Senders and let πI be the Imitator Sender. For each position in a message, agents produce a probability distribution over all possible symbols in V. Recall that the Expert Senders are deterministic at evaluation time. Given an input x, we write m(i)j as the value of the jth position in the message m(i) produced by Expert i. For the Imitator Sender, we write
πIj := π I ( m (i) j−1 ; x ) ∈ [0, 1]|V |
as the probability distribution over possible symbols in position j of a message produced by the Imitator agent, given the previous output symbol m
(i) j−1 of Expert i. The k th index of πIj , or π I j [k], gives the Imitator agent’s probability of symbol k at position j in the message. The ideal Imitator πI∗ minimizes the crossentropy objective between its messages and that
of a uniform mixture of k Expert Senders. Formally,
πI∗ = min πI
k∑
i=1
L∑
j=1
CE ( m
(i) j , π I j
)
= min πI
k∑
i=1
L∑
j=1
− log πIj [m(i)j ]
= max πI
k∑
i=1
L∑
j=1
log πIj [m (i) j ]
= max πI
k∏
i=1
L∏
j=1
πIj [m (i) j ]
subj. to ∑
k∈V πIj [k] = 1. whose unique solution is πIj [m (i) j ] = π I j [m (l) j ] ∀j ∈ NL, ∀i ̸= l ∈ Nk, i. e. a uniform distribution over Expert languages. 2