Venue: NAACL 2024

# XferBench: a Data-Driven Benchmark for Emergent Language

Brendon Boldt, David Mortensen

Language Technologies Institute

Carnegie Mellon University

Pittsburgh, PA 15213, USA {bboldt,dmortens}@cs.cmu.edu

###### Abstract

In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language--the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1475–1489

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Neural language models learn many things in pretraining, but research suggests Artetxe et al. (2020) that a substantial part of that knowledge is not simply knowledge of a particular language or domain, but rather knowledge of "how to language." We currently teach models to "language" using vast quantities of text dredged from the dark recesses of the Web--text that is full of bias, toxicity, and potential intellectual property violations. Ideally, we would be able to teach models to "language" without such compromises through the use of synthetic data, but mainstream approaches to synthesizing data produce outputs that do not have the same structural and social properties as human language.

Emergent communication (EC), also called emergent language (EL), is a potential solution to this problem Yao et al. (2022); Downey et al. (2023); Mu et al. (2023). Emergent languages are communication systems developed _de novo_ among multiple agents in a reinforcement learning simulation. Because the conditions under which they develop mirror, reductively, the conditions under which languages develop among humans, there is reason to believe that ELs will ultimately be more like human language than other sources of synthetic data. However, up to this point, there is no way of quantifying--in a holistic way--how much like human languages any particular EL really is, or to what extent it may provide useful pretraining signals.

Research on deep learning-based emergent communication has seen the introduction of many metrics to measure various aspects of the language. These metrics sturthy2006learning, Lazaridou et al. (2018), expressivity Guo et al. (2023), ease-of-teaching Li and Bowling (2019), and zero-shot transfer Bullard et al. (2020), to name a few. Despite this proliferation of metrics, emergent language largely lacks _evaluation_ metrics. An evaluation metric is specifically one that measures the _overall quality of an emergent language_ and not simply a particular property. Thus, we introduce XferBench, a data-driven benchmark for evaluating the overall quality of emergent languages using transfer learning with deep neural models.

Evaluation metrics are critical in gauging progress in technical fields since they quantify otherwise vague notions of improvement over time. Benchmarks, in particular, pair evaluation metrics with specific data and evaluation procedures to compare various systems on common ground. Benchmarks and shared tasks have been critical to the development of NLP from the Penn Treebank Marcus et al. (1993) to the WMT datasets Bojar et al. (2014) to GLUE Wang et al. (2018).

In the field of emergent communication specifically, Yao et al. (2022) introduced the idea of using _corpus transfer_ as means of practically applying emergent communication to deep learning-based NLP via transfer learning. In corpus transfer, a language model is pretrained on a corpus of emergent language utterances before being tuned on real data for a human language-based downstream task. As a corollary, they suggest that the effectiveness of this transfer can serve as a means of evaluating the quality of the emergent in a more general sense. This is based on the intuition that the more similar two language are, the better transfer learning works from one to the other (observed in Zoph et al. (2016), for example).

This paper takes the transfer learning-as-an-evaluation metric idea from Yao et al. (2022) and expands it into a full benchmark, XferBench, for emergent languages (illustrated in Figure 1). An evaluation metric for emergent languages in a benchmark format is the first of its kind. Additionally, XferBench is unique within emergent communication for being primarily data-driven instead of relying on particular handcrafted

algorithms for quantifying a given phenomenon. This means that XferBench can be easily scaled up in the future as the field of emergent communication advances and requires expanded means of evaluating emergent languages. Finally, XferBench is distributed as a user-friendly Python package, allowing researchers from across the field of emergent communication to apply XferBench to their own work on emergent communication.

ContributionsThis paper makes the following contributions: (1) Introduces XferBench, a data-driven benchmark for evaluating the overall quality of an emergent language, the first of its kind in emergent communication. (2) Provides a analysis of the quality human, synthetic, and emergent language according to XferBench. (3) Provides an easy-to-use Python implementation of XferBench.

## 2 Related Work

Emergent CommunicationThis paper is situated in the field of emergent communication (a.k.a. emergent language) which is generally covered by the review Lazaridou and Baroni (2020). The field centers around the invention of language by deep neural networks typically using multi-agent reinforcement learning techniques. The study of emergent communication is intended to (1) shed light on the origin and nature of the human language LaCroix (2019); Moulin-Frier and Oudeyer (2020); Galke et al. (2022) and (2) provide an alternative approach to problems in NLP and multi-agent reinforcement learning which relies on constructing language from the ground up and not just pre-existing (human) languages alone Li et al. (2020); Yao et al. (2022); Mu et al. (2023); Downey et al. (2023).

Transfer LearningTransfer learning for deep neural networks is a key component of XferBench and follows in general tradition of Zoph et al. (2016). Specifically, this paper draws heavily from Yao et al. (2022) (see also Papadimitrou and Jurafsky (2020); Artetxe et al. (2020)) which introduce the technique of _corpus transfer_ for emergent language, that is, pretraining a neural model on an emergent language corpus before tuning it on a downstream human language task. In particular, this paper takes Yao et al. (2022)'s idea of using corpus transfer as a metric and adapts it into a benchmark pipeline which can easily be applied to new emergent languages.

BenchmarksWork such as Guo et al. (2023) and Perkins (2022) have looked at benchmarking particular aspects of emergent languages, but XferBench is the first of its kind in benchmarking the overall quality of an emergent language. Yao et al. (2022) also explicitly provide a metric for emergent language quality, but this metric is restrictive in that it can only be applied to emergent languages derived from a model that takes images (that have captions available) as input; this conflicts with the design goals of XferBench discussed below.

Outside of emergent communication, XferBench is more analogous to benchmarks for generative models (e.g., Frechet Inception Distance Heusel et al. (2017) for image generation) than more traditional NLP benchmarks like GLUE Wang et al. (2018) or SQuAD Rajpurkar et al. (2016). This is because emergent communication is a generative enterprise, where one of the main goals is to create samples (emergent languages) which resemble a target distribution (human languages) either generally or in some particular respect. Furthermore, metrics like FID are primarily self-supervised, data-driven measures of similarity in the same vein as XferBench. This is in contrast to more traditional NLP benchmarks which combine data-driven methods with many human judgments (i.e., through labeled examples).

## 3 XferBench

### Design Goals

We frame the primary design goals of the benchmark as three desiderata:

* Quantitatively capture a meaningful notion of the overall quality1 of an emergent language from a data-driven perspective. Footnote 1: We are aiming for \(a\) meaningful notion of overall quality: we are not claiming that this is the only meaningful notion nor that it is the best among all possible notions of “quality”.
* Be applicable to as wide a variety of emergent languages as possible, not restricted to a specific game, environment, or agent architecture.
* Be relevant and accessible to the broader EC/EL community, by being: (a) easy to interpret, (b) minimally biased with regards to language typology, (c) runnable with minimal coding experience, and (d) runnable on modest hardware.

While there are other consideration in the benchmark, these form the bulk of the motivation. In the following

Figure 1: Illustration of the architecture of XferBench.

 

paragraphs we expand upon the motivation for each design goal.

D1: Quantifying qualityD1 is the core of what a benchmark seeks to do: to quantify a desirable property of a given system such that it can be compared directly to other systems (i.e., be an _evaluation_ metric). There are two distinct senses in which XferBench strives towards this goal. First, XferBench measures how good an emergent language is from a specifically machine learning perspective; that is, it addresses the question, "How useful would this emergent language be for practical machine learning tasks?" The second sense is more general: XferBench addresses the question, "How similar is an emergent language to human language according to how deep neural networks process language?" That is, it uses data-driven techniques to quantify the similarity between emergent language and human language in some general sense.

D2: Wide applicabilityD2 is intended to make XferBench practically applicable to a wide range of EC research. The field of EC has an especially diverse set of possible approaches, environments, agents, games, etc. Thus, it is especially salient that the benchmark be designed with interoperability in mind, having minimal assumptions as to the nature of the EC system being evaluated.

The influence of this design goal is primarily seen through the use of a textual corpus as the sole input to the benchmark: the vast majority of EC systems generate utterances which can be represented as sequences of discrete tokens.2 EC presents the opportunity for much richer representations of its language: leveraging the grounded semantics of the communication, incorporating non-verbal behavior, and even directly interacting with the agents themselves. Yet such richer representations also limit the range of EC systems to which XferBench could apply. Even if it is possible to define some universal EC interface that could allow for richer representations, the implementation cost for each and every EC system to be tested is significant compared to the ease of producing a corpus of utterances from the emergent language.

Footnote 2: In the minority case, there are EC methods which use communication channels that are, for example, continuous [6] or even pictorial [30].

D3: Easy-to-useD3 is critical to the success of XferBench as a practical tool for diverse field of researchers--a benchmark is expressly _for_ the broader research community, and, as such, should be widely accessible. In particular, D3a demands that XferBench be conceptually simple with results that can easily be reported, compared, and incorporated into a research program. D3b is relevant to both aspects of D1. First, if XferBench is to gauge an EL's practical use in machine learning, it should seek to use a typologically diverse set of human languages in the downstream tasks. Second, since XferBench is trying to capture a notion of "similarity to human language generally", it is important to test this against a wide range of language typologies so as not to unnecessarily narrow the criteria for "similar to human language". D3c is particularly important for incorporating interdisciplinary researchers into the field of EC who might not have a background in computer programming. Finally, D3d ensures that XferBench is accessible not only to labs and researchers with fewer financial resources but also makes it much easier to incorporate into the fast-paced research and development cycles prevalent in contemporary ML research.

### Methods

The following procedure describes the benchmark (illustrated in Figure 1):

1. Initialize a causal language model.
2. Train the model on the corpus of utterances from the EL being evaluated.
3. Re-initialize the input and output (i.e., language modelling head) embedding layers; this is the _base model_.
4. For each downstream human language: 1. Train the base model on the human language data. 2. Evaluate the cross-entropy on a held-out test set of the human language.
5. Average the cross-entropies across the downstream human languages; this is the corpus's score on the benchmark (lower is better).

The structure of the benchmark is derived from the _corpus transfer_ method presented in Yao et al. (2022).

TaskFor XferBench's evaluation task, we choose causal language modeling for a few different reasons. In principle, language modeling is a component of a wide variety of NLP tasks, especially generative tasks; the prevalence of language modeling is in line with the benchmark providing a very general notion of quality that will be familiar to anyone acquainted with NLP. On a practical level, language modeling is easy to acquire data for--especially helpful for evaluating against low-resource languages--and there are fewer hyperparameters and confounding variables compared to other downstream tasks like machine translation or questionnaire. The main limitation from using language modeling is that it itself is not a widespread downstream task and so cannot guarantee direct correlation with metrics on more concrete downstream tasks (e.g., accuracy on a QA task).

For the pretraining task we also use causal language modeling. Due to requiring a wide applicability across emergent languages (Design Goal 2), we select causal language modeling for our pretraining task since it requires only a corpus without any additional annotations or stipulations.

DataThe data for the transfer learning targets (vizhuman languages) comes from Wikipedia dumps (Wikimedia Foundation) (under the GFDL and CC-BY-SA 

3.0 License) hosted by Hugging Face3. This dataset provides a diverse set of languages each with sufficient amounts of data. For our downstream human languages, we use the same \(10\) languages presented in Yao et al. (2022), namely: Basque, Danish, Finnish, Hebrew, Indonesian, Japanese, Kazakh, Persian, Romanian, and Urdu. Having a variety of languages reduces the likelihood that XferBench will be biased toward specific typologies of human language (Design Goal 3b).

Footnote 3: [https://huggingface.co/datasets/wikimedia/wikipedia/tree/973235edeffictofabid786bedd0788e4abd24b03](https://huggingface.co/datasets/wikimedia/wikipedia/tree/973235edeffictofabid786bedd0788e4abd24b03)

We use \(15\) and \(2\) million tokens for the pretraining and fine tuning phases, respectively following Yao et al. (2022). Datasets are always repeated or truncated to fit the required size so that the number of training steps stays constant.

TokenizationFor tokenization we use byte pair encoding (BPE) Gage (1994) with a vocabulary size of \(30\,000\) for all human languages. Using BPE across all human languages is done primarily to simplify the implementation and keep tokenization methods consistent across all of the selected human languages. Emergent languages are generally considered to be pre-tokenized since most communication channels consist of one-hot vectors; thus, no additional tokenization or preprocessing is applied.4

Footnote 4: Whether the tokens of an EL should be treated as words or subword units is an open question, although tokens as words is more common (but see Ueda et al. (2023) for tokens as subword units). Practically speaking, many emergent languages are small enough that applying a \(30\,000\)-item BPE model would severely reduce the corpus size.

ModelFor our model, we use a small configuration of GPT-2 Radford et al. (2019), similar to that used in Yao et al. (2022): \(6\) attention heads, \(6\) layers, context length of \(256\), and hidden size of \(768\) with the remainder of the model parameters being the same as the defaults in the Hugging Face Transformers implementation.5 This yields \(65\) million parameters in total. We kept the model on the smaller size to better suit it for the generally small amounts of data emergent languages corpora provide as well as to be more accessible (Design Goal 3d). Further details are listed in Appendix A.1.

MetricGiven the use of language modeling for our evaluation task, we use token-level cross-entropy as the evaluation metric on the downstream task. This is a very common metric, making the outputs easy to interpret (Design Goal 3a). Although perplexity is more common as an evaluation of language models, the exponential nature of perplexity leads to more circuitous analyses and interpretation in our case, whereas cross-entropy is comparatively linear and additive (loosely speaking).6 For the final score of the benchmark, we take the arithmetic mean of the cross-entropy across the \(10\) downstream human languages. That is, we define the benchmark's score for a given source language \(s\) as as \(h_{s}\):

Footnote 5: [https://huggingface.co/docs/transformers/v4.36.1/en/model](https://huggingface.co/docs/transformers/v4.36.1/en/model) ̇doc/pptz#transformser.GPTZConfig

\[h_{s}=\underset{t\in T}{\text{mean}}\left(h_{s,t}\right) \tag{1}\]

where \(h_{s,t}\) is the test cross-entropy of a model trained on source language \(s\) and finetuned and tested on target language \(t\); \(T\) is the set of target languages. Since the score is based on cross-entropy, a lower score means better performance.

### Implementation

XferBench is implemented as a small Python codebase which relies primarily on Hugging Face Transformers Wolf et al. (2019) (Apache-2.0 license) and PyTorch Paszke et al. (2019) (BSD-3Clause license) libraries. To run the benchmark, all that is required is to install the environment with either pip or conda, and run python -m xferbench path/to/corpus.jsonl (Design Goal 3c). The input corpus is simply formatted as a newline-separated list of integer arrays, specifically in the JSON Lines format (see Appendix B for an example); a Hugging Face dataset (backed by Apache Arrow) can also be used for larger input corpora. The script executes all of the steps of the benchmark and yields a single floating point number which is that corpus's score on XferBench (the benchmark also saves the individual score across target languages for further analysis). Finer-grained functionalities are available and documented in the codebase. The benchmark takes about \(5.5\) hours to run on a single NVIDIA GeForce RTX 2080 Ti: \(90\) minutes to train the base model and \(30\) minutes for tuning and testing on each of the target languages (Design Goal 3d). Since the model is tuned independently on each target language, it is easy to parallelize this step and drastically shorten the wall-clock time of XferBench.

The implementation is available at [https://github.com/brendon-boldt/xferbench](https://github.com/brendon-boldt/xferbench) under the MIT license.

## 4 Experiments

### Procedures

XferBenchThe causal language modeling experiment is simply running XferBench as described in Section 3.2 on the reference and emergent languages discussed in Sections 4.2 and 4.3.

Machine translationThe machine translation experiment is structured similarly to XferBench except with the downstream task being English-to-French translation (using the WMT 2014 dataset Bojar et al. (2014)). The primary purpose of this experiment is to determine how well XferBench correlates with a more concrete downstream task (especially one that incorporates language modeling). We choose this language pair in part to gauge the relative differences between the task languages and the baseline human languages (in contrast to XferBench which we want to be largely agnostic 

to human languages). Looking at our reference human languages, we have: French, the target language itself; Spanish, closely related to French; Russian and Hindi, distantly related to French; and Chinese, Korean and Arabic, not related to French. Instead of using a GPT-2-based model, we use a BART-based model since MT is a conditional generation task (see Appendix A.2 for details). The pretraining dataset size is increased to \(100\) million due to the increased difficulty of this task compared to language modeling. We evaluate the translation performance with chrF Popovic (2015) and BLEU Papineni et al. (2002) using the default Hugging Face Evaluate metrics (derived from sacreBLEU Post (2018)). Evaluation is performed with beam sizes of \(1\), \(3\), and \(5\), and the resulting values are averaged.

We present three settings for this experiment. The first is _Full_ which tunes on \(50\) million source tokens at a higher learning rate (\(1\cdot 10^{-4}\) for training and \(2\cdot 10^{-4}\) for the AdamW optimizer Kingma and Ba (2015)), which we found empirically to lead to the best performance. The second is _rozen_, in which we use the same configuration as _Full_ but freeze all but the embedding layers before tuning the model for translation (as in Papadimitriou and Jurafsky (2020); Artetxe et al. (2020)). Finally, we also present _Reduced_ which uses a smaller tuning dataset of \(10\) million tokens and lower learning learning (\(2\cdot 10^{-5}\)); the lower rate helped the random baselines converge better as well as showed better distinction between languages.

### Reference languages

The following reference languages serve as a way to contextualize the results of XferBench as well as to validate that it is capturing some notion of the quality of the emergent languages (cf. Section 4.4).

Human languagesFor our baseline line human languages, we selected French, Spanish, Russian, Chinese, Korean, Arabic, and Hindi.7 Like the evaluation languages, the data is derived from Wikipedia articles (same source as the target languages).

Footnote 7: The main reason for choosing the high-resource language is due to the higher data requirements of machine translation experiment discussed below.

Synthetic languagesFor synthetic languages, we follow Yao et al. (2022) and use Zipfian parentheses" from Papadimitriou and Jurafsky (2020). This synthetic dataset--referred to as _Paren, real_--is hierarchically balanced "parentheses" where each parenthesis is the token ID sampled from the unigram distribution of a human language (hence "Zipfan"). This datasets mimics both the unigram distribution of a human language as well as the basic recursive hierarchical structure. This yields a reasonably strong yet simple baseline for synthetic data.

We also test a fully synthetic dataset (_Paren, synth_) which uses the same hierarchical parenthesis generation script from Papadimitriou and Jurafsky (2020), replacing the data-derived unigram distribution with Zipf-Mandelbrot distribution:

\[f(w_{i})=\frac{1}{\left(i+\beta\right)^{\alpha}} \tag{2}\]

where \(f(w_{i})\) is non-normalized probability weight of word \(w\) with \(1\)-based index (rank) \(i\), \(\alpha=1\), \(\beta=2.7\) Mandelbrot et al. (1953); Piantdosi (2014).

Random baselinesWe use two random baselines. The first is simply a uniform unigram distribution across the whole vocabulary with no additional structure (referred to as _Random_). This baseline sheds light on whether the optimization itself, no matter training data, primes the network in some way for transfer learning. The second "random" baseline is no pretraining at all (_No pretrain_); that is, a network which has been freshly initialized at the tuning stage. This baseline helps establish whether or not pretraining on other languages has any impact beyond tuning alone.

### Emergent languages

We present a summary of the key hyperparameters of emergent languages in Table 1. The emergent language corpora below come from reproductions from existing codebases with the exception of Yao et al. (2022), whose emergent language corpus is available for download. Emergent languages which have a corpus size smaller than the required size are simply repeated and shuffled as many times as necessary so that the model receives the same number of optimization steps.

Generic signalling gameThe first set of emergent languages we test are generic versions of the of the signalling game (reference game) as implemented in EGG Kharitonov et al. (2019) (MIT license). These games use one-hot vectors to represent attribute-value observations, that is, observations are elements of the set \(V^{|A|}\) where \(V\) is the set of values and \(|A|\) is the number of attributes. The signalling game is one of the simplest and most used games in emergent communication research.

The first two language are _Disc, small_ and _Disc, large_ which are two configurations of the discrimination version of the signalling game. Here, the sender makes an observation and sends a message; then, the receiver must

\begin{table}
\begin{tabular}{l l r r r} \hline \hline Setting & Observ. & \(|V|\) & \(|M|\) & \(|C|\) \\ \hline Disc, small & one-hot & \(6\) & \(11\) & \(700\) \\ Disc, large & one-hot & \(100\) & \(31\) & \(100\) M \\ Recon, large & one-hot & \(100\) & \(31\) & \(31\) M \\ Mu+, CUB & embed & \(20\) & \(10\) & \(1.3\) M \\ Mu+, SW & embed & \(14\) & \(7\) & \(1.2\) M \\ Yao+ & embed & \(4028\) & \(15\) & \(43\) M \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of key hyperparameters in the tested emergent languages. Observations are either one-hot vectors or embeddings. \(|V|\), \(|M|\), and \(|C|\) refer to the vocabulary, message, and corpus size respectively.

 

select the corresponding observation from a small set of potential observations (like a multiple-choice question). The _small_ configuration consists of attributes and values with a small vocabulary size and medium message length; this setting is intended to represent a toy environment that one might find in an emergent communication paper. The _large_ configuration consists of attributes and values with a larger vocabulary and longer message length. Both environments show distractor distractors to the receiver (i.e., -way multiple choice). Both settings converge to a success rate compared to a random baseline of .

The _Recon, large_ environment is based on the reconstruction version of the signalling game. In this version, the receiver does not make any observations and instead must recreate the sender's observation based on the message alone (similar to an autoencoder). The observation space has attributes and values with other settings identical to that of _Disc, large_. Since the reconstruction game considerably harder, the game does not converge but does reach an overall accuracy of compared and per-attribute accuracy of compared to a random baseline of and respond. For details, see Appendix A.3.

Mu and Goodman (2021)present the second pair of emergent languages which we test XferBench on (code under MIT license). The emergent communication game is also a discriminative signalling game but with (1) richer observations and (2) more abstract information needing to be communicated. In one setting, the observations are images from ShapeWorld Kuhnle Copestake (_Mu+_, SW), a synthetic data of various geometric shapes, and the other setting is CUB Wah et al. (2011) Mu+, _CUB_) which contains labeled images of birds; both settings encode features with a CNN which is the passed to the sender and receiver. In the basic discriminative game, the observation made by the sender is the exact same one seen by the receiver. Mu and Goodman (2021) instead uses a "concept game" where the sender must communicate some abstract concept shared by a set of input images which the receiver will then have to a pick out from a different set of images, some sharing the same concept (e.g., isolating the concept of "triangle" or "bird size"). The ShapeWorld and CUB games had test accuracies of and respectively compared to a random baseline of comparable to the reported values in the paper. All messages were taken from observations seen in training.

Yao et al. (2022)present a standard discrimination game which uses natural images Conceptual Captions Sharma et al. (2018) (images only)) as inputs to the sender and receiver (code unlicensed but distributed on GitHub with paper). The accuracy for the particular emergent language corpus is not reported in the paper, but comparable experiments from the paper would suggest that it converged to an accuracy of compared to a baseline of (i.e., distractors).

### Hypotheses

The following hypotheses are directly relate to determining whether or not XferBench is quantifying some meaningful notion of the quality of a language (i.e., Design Goal 1).

(H1) Human languages will perform best, followed by the synthetic and emergent languages, followed by the random baselines.

(H2) Human languages will have similar performance on XferBench (also key for Design Goal 3b); the intuition here is that human languages share deep structural similarities. This hypothesis is supported, in part, by Artetxe et al. (2020). For the MT experiment, we expect to see the following order of performance based on language relatedness: {_French_}, {_Spanish_}, {_Russian, Hindi_}, {_Chinese_, _Korean_, _Arabic_}.

(H3) Languages with a larger vocabulary, longer message length, and larger corpora will perform better. In particular, we expect _Disc, large_ will perform better than _Disc, small_ since the former is a more "complex" version of the latter. This hypothesis (for vocabulary size and message length) is supported by some experiments in Yao et al. (2022, app. B.4).

(H4) XferBench will correlate well with scores on the machine translation task (i.e., cross-entropy will correlate negatively with chrF).

## 5 Results

### XferBench

In Figure 2 we show the results of the benchmark (i.e., causal language modeling) on the various baselines. Each mean is displayed with error bars showing the confidence interval of mean as calculated with bootstrapping (details in Appendix E). For reference, the cross-entropies range from about to corresponding to complexities of to .

The human languages show the best score (lowest cross-entropy) on the benchmark with _Chinese_, _Ko-rean_, and _Arabic_ performing the best in one cluster and _French_, _Spanish_, _Russian_, and _Hindi_ performing slightly worse in their own cluster (based on confidence intervals). The synthetic and emergent languages all show similar performance with only small variations with the exception of the _Disc, large_ language which is better than the rest of the emergent languages but still worse than the human languages. Finally, the random baselines perform worse than the rest of the tested languages. _No pretrain_'s performance is worse than the cluster of synthetic and emergent languages but better than the fully random language (_Random_).

### Machine Translation

The chrF scores of the machine translation experiment are given in Table 2 (BLEU scores in Appendix D.1). Additionally, we give Pearson correlation coefficients between each setting and the scores generated by XferBench (scatter plots shown in Appendix D.4). In all 

settings, we see that XferBench is strongly correlated with the results of the machine translation experiment.

For the _Full_ setting, the results are somewhat inconclusive. Human languages perform the best and similarly to each other. _Paren, real, Paren, syn, Disc, large_, and _Mu+, CUB_ all match the performance of human languages as well. The rest of the language perform significantly worse than the aforementioned languages, especially _Yao+_ and _Mu+, SW_ (see Appendix F for sample outputs). In the case of _Random_, the training loss did not decrease during training likely due to the high learning rate.

In _Frozen_, we see the best correlation with the hypothesis regarding human languages (as well as with XferBench itself). _Disc, large_ performs comparably to the worst human languages and better than the rest of the languages. The remainder of the synthetic and emergent languages perform worse than the human languages but better than the random baselines.

Finally, _Reduced_ (i.e., lower learning rate and tuning data) displays better separation than _Full_, but not as significant as _Frozen_. Human languages still perform the best, although they are matched by the _Parent_ languages. _Disc, large_ underperforms the human languages but still outperforms all other emergent languages. All emergent languages, apart from _Disc., large_ underperform the _No pretrain_ baseline. The better half of languages performed better (compared to themselves) with a higher learning rate while the lower half performed better with a reduced learning rate.

## 6 Discussion

### Experiments

The basic ordering of the language by XferBench follows basic _a priori_ assumptions: random baselines perform the worst, human languages perform the best, and emergent and synthetic languages are bounded above and below by these (supporting Hypothesis 1). Human languages cluster together in XferBench although there is still variation with non-overlapping confidence intervals (partially supporting Hypothesis 2).

Intra-EL differencesGenerally speaking, there is very little variation shown by XferBench on the emergent languages; nevertheless, we can still draw a handful of conclusions. First, _Disc, large_ outperforms _Disc, small_ while sharing the same codebase, task, etc, and differing only in message length, vocabulary size, observation space, and corpus size (supporting Hypothesis 3). This result matches the trend seen in Yao et al. (2022) that larger vocabularies and message lengths in an emergent language lead to better performance on downstream data. On the other hand, _Disc, small_ performs similarly to other languages with larger vocabularies and longer message lengths (contradicting Hypothesis 3).

\begin{table}
\begin{tabular}{l r r r} \hline \hline Source & Full & Frozen & Reduced \\ \hline French & 47.8 & 31.4 & 35.8 \\ Spanish & 48.0 & 27.9 & 34.8 \\ Russian & 47.6 & 29.0 & 37.2 \\ Chinese & 47.5 & 22.2 & 35.2 \\ Korean & 47.7 & 23.3 & 35.6 \\ Arabic & 47.8 & 27.6 & 36.6 \\ Hindi & 47.5 & 26.0 & 31.7 \\ Paren, real & 47.5 & 10.5 & 35.0 \\ Paren, synth & 48.2 & 12.0 & 34.3 \\ Disc, large & 47.7 & 24.7 & 30.7 \\ Disc, small & 14.3 & 16.2 & 17.3 \\ Rec, large & 22.5 & 18.4 & 25.4 \\ Yao+ & 4.0 & 20.1 & 25.6 \\ Mu+, SW & 3.3 & 18.4 & 23.3 \\ Mu+, CUB & 47.6 & 21.6 & 24.6 \\ Random & 1.8 & 3.0 & 19.7 \\ No pretrain & 11.4 & 4.3 & 28.1 \\ \hline Correl. with & \multirow{2}{*}{\(-0.75\)} & \multirow{2}{*}{\(-0.84\)} & \multirow{2}{*}{\(-0.79\)} \\ XferBench & & & \\ \end{tabular}
\end{table}
Table 2: chrF scores across three English-to-French machine translation settings. Correlation measured with the Pearson correlation coefficient. Colors normalized by column.

Figure 2: Average cross-entropy on target language datasets for each source language. Lower is better. Error bars represent \(95\%\) confidence intervals.

 

Second, it seems that the underlying complexity of the emergent communication game does not directly correlate with XferBench score: the abstract visual reasoning of _Mu+_, \(SW\) and _Mu+_, _CUB_ does not lead to it outperform _Disc_, _small_. Additionally, the richer observations (i.e., image embeddings) of _Mu+_, _CUB_ and _Yao+_ also do not, by their mere presence, confer an advantage to the emergent language with respect to XferBench.

Finally, _Disc_, _large_ and _Recon_, _large_ both share hyperparameters in terms of the vocabulary size, message length, and corpus size, yet _Disc_, _large_ shows significantly better performance on XferBench. This indicates that XferBench is not _solely_ concerned with surface-level features as we see that the nature of the game (e.g., discrimination versus reconstruction, success rate) is relevant as well.

Correlation with MTThe results from the machine translation experiment show strong, though not perfect, (negative) correlation with XferBench (supporting Hypothesis 4). For example, in all cases, _Disc_, _large_ outperforms all other emergent languages. This strongly supports the notion that XferBench performance is predictive of downstream performance on more concrete NLP tasks.

The results from the _Full_ setting of the MT experiment do show some correlation with XferBench but fail to show expected trends in other ways. For example, there is no clear ordering among the human languages (e.g., _French_ does _not_ outperform _Arabic_). Additionally _Yao+_ and _Mu+_, \(SW\) drastically underperform the other emergent languages and the _No pretrain_ baseline. We suspect that these aberrations from expected results come in part due to the high learning rate which cause unstable training or generation. On the other hand, the _Frozen_ setting gives us the clearest ordering of human languages that matches with _a priori_ expectations; this setting also has the strongest correlation with XferBench scores. The _Reduced_ setting shows better correlation than _Full_ but is not as clear as _Frozen_.

Random baselinesIn all of our experiments, the pretraining on random tokens (_Random_) performed notably worse than not pretraining at all (_No pretrain_), suggesting that ill-conditioning the neural network can be a significant hindrance to performing well on XferBench. This is important to note in light of the fact that a perfectly one-to-one compositional language describing uniformly sampled attribute-value vectors would yield a corpus with a uniformly random unigram distribution. This is to say, a fully compositional language, which is often seen as desirable in emergent communication research, could make for a very poor source of pretraining data as shown by _Random_'s performance on XferBench.

This fact along with the observations about sensitivity to learning rate indicates that performance on XferBench is not simply a function of the particular features of the emergent language in relation to the downstream human languages but also a function of the dynamics of optimization (i.e., priming the model to adapt well). Although this increases the difficulty of developing and interpreting a tool like XferBench, it is almost an unavoidable part of deep learning methods.

### Future work

We identify three main directions for future work with XferBench. The first direction is determining what XferBench is measuring and how its scores correlate with the different factors of emergent languages. Yao et al. (2022, app. B.4) pursued this on a small scale with factors like vocabulary size and message length, but there exist a host of other factors worth exploring: speaker model size, game design, language entropy, observation modality, etc.

The second direction is more extensively investigating the correlation of XferBench with downstream tasks. We would expect that tasks that rely heavily on a language model--such as automatic speech recognition, abstractive summarization, and generative question-answering--to correlate well with XferBench. On the other hand, tasks that are more focused on classification--such as named entity recognition, sentiment analysis, and multiple choice question-answering-- might not correlate as well.

Finally, XferBench would benefit greatly from improved compute efficiency. For example, if the results of XferBench could be replicated with a fraction of the training steps, it could (1) allow for a larger number of downstream languages to be tested which would reduce the size of the confidence intervals, allowing more more precise scoring. And (2), it would open the door to using larger models which would better capture the deeper structures of language and likely correlate better with realistic downstream tasks.

## 7 Conclusion

In this paper we have introduced XferBench, a first-of-its-kind benchmark for evaluating the quality of an emergent language corpus based on its transfer learning performance on human languages. This approach to evaluating emergent language scales with data and compute as opposed to requiring increasingly complex handcrafted rules to measure the desirable qualities of emergent language. We provide empirical results of XferBench across human, synthetic, and emergent languages and demonstrate that these results correlate with downstream performance on a machine translation task. XferBench is implemented as an easy-to-use Python package that will permit researchers in the field to easily apply XferBench to new emergent languages.

## 8 Limitations

The first limitation of XferBench is that it relies on a restricted interface with the emergent communication system. With emergent communication we have access not only to the grounding of all of the utterances of the emergent language but also full access to the 

agents themselves. Language is fundamentally a contextual phenomenon, so only a small part of it can be understood from looking at corpora in isolation. Thus, although XferBench is much more broadly applicable because of this restricted interface, it is also quite limited in what it can detect from a theoretical point of view.

The other set of limitations we will discuss have to do with the model and data size. First, the model and data size (\(60\) M parameters and \(15\) M tokens) are quite small by contemporary standards, limiting the direct applicability of results from XferBench to relevant downstream tasks involving large language models, for example. On the other hand, scaling up the models, data, and methods of XferBench comes with its own difficulties. First, it would start to bias the benchmark towards high-resource languages, as only those could provide the necessary data to accommodate larger models. Second, it would make XferBench, which is already relatively slow as a metric (\(6\) GPU-hours) even slower. This would decrease the speed of the iterative design process of emergent communication systems and, thus, the utility of the metric as a whole.

## References

* Aretxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4623-4637, Online. Association for Computational Linguistics.
* Bojar et al. (2014) Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Brighton and Kirby (2006) Henry Brighton and Simon Kirby. 2006. Understanding linguistic evolution by visualizing the emergence of topographic mappings. _Artificial Life_, 12:229-242.
* Bullard et al. (2020) Kalesha Bullard, Franziska Meier, Douwe Kiela, Joelle Pineau, and Jakob N. Foerster. 2020. Exploring zero-shot emergent communication in embodied multi-agent populations. _ArXiv_, abs/2010.15896.
* Downey et al. (2023) C.m. Downey, Xuhui Zhou, Zeyu Liu, and Shane Steinert-Threlkeld. 2023. Learning to translate by learning to communicate. In _Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL)_, pages 218-238, Singapore. Association for Computational Linguistics.
* Eloff et al. (2021) Kevin Eloff, Arnu Pretorius, Okko Johannes Rasanen, Herman Arnold Engelbrecht, and Herman Kamper. 2021. Towards learning to speak and hear through multi-agent communication over a continuous acoustic channel. _ArXiv_, abs/2111.02827.
* Gage (1994) Philip Gage. 1994. A new algorithm for data compression. _The C Users Journal archive_, 12:23-38.
* Galke et al. (2022) Lukas Galke, Yoav Ram, and Limor Raviv. 2022. Emergent communication for understanding human language evolution: What's missing? _ArXiv_, abs/2204.10590.
* Guo et al. (2023) Yukuan Guo, Yifan Hao, Rui Zhang, Enshuai Zhou, Zidong Du, Xishan Zhang, Xinkai Song, Yuanbo Wen, Yongwei Zhao, Xuehai Zhou, Jiaming Guo, Qi Yi, Shaohui Peng, Di Huang, Ruizhi Chen, Qi Guo, and Yunji Chen. 2023. Emergent communication for rules reasoning. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Kharitonov et al. (2019) Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. 2019. EGG: a toolkit for research on emergence of lanGuage in games. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_, pages 55-60, Hong Kong, China. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* a new test methodology for multimodal language understanding. _ArXiv_, abs/1704.04517.
* LaCroix (2019) Travis LaCroix. 2019. Biology and compositionality: Empirical considerations for emergent-communication protocols. _CoRR_, abs/1911.11668.
* Lazaridou and Baroni (2020) Angeliki Lazaridou and Marco Baroni. 2020. Emergent multi-agent communication in the deep learning era. _ArXiv_, abs/2006.02419.
* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. 2018. Emergence of linguistic communication from referential games with symbolic and pixel input. _ArXiv_, abs/1804.03984.
* Li and Bowling (2019) Fushan Li and Michael Bowling. 2019. _Ease-of-Teaching and Language Structure from Emergent Communication_. Curran Associates Inc., Red Hook, NY, USA.
* Li et al. (2020) Yaoyiran Li, Edoardo Maria Ponti, Ivan Vulic, and Anna Korhonen. 2020. Emergent communication pretraining for few-shot machine translation. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 4716-4731, Barcelona, 

Spain (Online). International Committee on Computational Linguistics.
* Mandelbrot et al. (1953) Benoit Mandelbrot et al. 1953. An informational theory of the statistical structure of language. _Communication theory_, 84:486-502.
* Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. _Comput. Linquistics_, 19:313-330.
* Mihai and Hare (2021) Daniela Mihai and Jonathon S. Hare. 2021. Learning to draw: Emergent communication through sketching. In _Neural Information Processing Systems_.
* Moulin-Frier and Oudeyer (2020) Clement Moulin-Frier and Pierre-Yves Oudeyer. 2020. Multi-agent reinforcement learning as a computational tool for language evolution research: Historical context and future challenges. _ArXiv_, abs/2002.08878.
* Mu and Goodman (2021) Jesse Mu and Noah Goodman. 2021. Emergent communication of generalizations. In _Advances in Neural Information Processing Systems_, volume 34, pages 17994-18007. Curran Associates, Inc.
* Mu et al. (2023) Yao Mu, Shunyu Yao, Mingyu Ding, Ping Luo, and Chuang Gan. 2023. Ec2: Emergent communication for embodied control. In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 6704-6714.
* Papadimitriou and Jurafsky (2020) Isabel Papadimitriou and Dan Jurafsky. 2020. Learning music helps you read: Using transfer to study linguistic structure in language models. In _Conference on Empirical Methods in Natural Language Processing_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Perkins (2022) Hugh Perkins. 2022. Icy: A benchmark for measuring compositional inductive bias of emergent communication models.
* Piantadosi (2014) Steven T Piantadosi. 2014. Zipf's word frequency law in natural language: a critical review and future directions. _Psychon. Bull. Rev._, 21(5):1112-1130.
* Popovic (2015) Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Twenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,00+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypermymed, image alt-text dataset for automatic image captioning. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2556-2565, Melbourne, Australia. Association for Computational Linguistics.
* Ueda et al. (2023) Ryo Ueda, Taiga Ishii, and Yusuke Miyaao. 2023. On the word boundaries of emergent languages based on harris's articulation scheme. In _The Eleventh International Conference on Learning Representations_.
* Wah et al. (2011) C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011. The caltech-UCSD birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In _BlackboxNLP@EMNLP_.
* Foundation (2019) Wikimedia Foundation. Wikimedia downloads.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Fantowicz, and Jamie Brew. 2019. Huuggingface's transformers: State-of-the-art natural language processing. _ArXiv_, abs/1910.03771.
* Yao et al. (2022) Shunyu Yao, Mo Yu, Yang Zhang, Karthik R Narasimhan, Joshua B. Tenenbaum, and Chuang Gan. 2022. Linking emergent and natural languages via corpus transfer. In _International Conference on Learning Representations_.
* Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1568-1575, Austin, Texas. Association for Computational Linguistics 

Hyperparameters

### Causal language modeling

For values not listed, see Hugging Face Transformers' defaults at [https://huggingface.co/docs/transformers/v4.36.1/en//model_doc/gpt2#transformers.GPI2Config](https://huggingface.co/docs/transformers/v4.36.1/en//model_doc/gpt2#transformers.GPI2Config).

* Model: GPT-2
* Tokenizer: Byte pair encoding Hidden size: \(768\) (default) Vocabulary size: \(30\,000\)
* Context length: \(256\)
* Number of layers: \(6\)
* Number of attention heads: \(6\)
* Learning rate: \(1\cdot 10^{-4}\)
* Optimizer: AdamW
* Weight decay: \(0.01\)
* Learning rate schedule: linear (to \(0\))
* Batch size: \(32\)
* Train dataset size: \(15\cdot 10^{6}\) tokens
* Train epochs: \(5\)
* Tune dataset size: \(2\cdot 10^{6}\) tokens
* Train epochs: \(10\)

### Machine translation

For values not listed, see Hugging Face Transformers' defaults at [https://huggingface.co/docs/transformers/v4.36.1/en/model_doc/bart#transforms.BartConfig](https://huggingface.co/docs/transformers/v4.36.1/en/model_doc/bart#transforms.BartConfig). The following is for the _Full_ setting.

* Model: BART
* Training objective: text infilling only (see note below)
* Tokenzer: Byte pair encoding
* Hidden size: \(512\) Vocabulary size: \(30\,000\)
* Context length: \(512\)
* Number of encoder layers: \(6\)
* Number of decoder layers: \(6\)
* Number of encoder attention heads: \(8\)
* Number of decoder attention heads: \(8\)
* Encoder feedforward dimension: \(2048\)
* Decoder feedforward dimension: \(2048\)
* Train learning rate: \(1\cdot 10^{-4}\)
* Tune learning rate: \(2\cdot 10^{-4}\)
* Optimizer: AdamW
* Weight decay: \(0.01\)
* Learning rate schedule: linear (to \(0\))
* Batch size: \(32\)
* Train dataset size: \(100\cdot 10^{6}\) tokens
* Train epochs: \(5\)
* Tune dataset size: \(50\cdot 10^{6}\) tokens
* Train epochs: \(3\) Test beam size: \(1,3,5\) (final metric averaged across each size)
* Test context size: \(128\)

The objective used to pretrain BART was text infilling _only_; we cannot use the sentence permutation objective because we do not know _a priori_ what constitutes a sentence in an emergent language, hence we do not use it for any settings. For the _Frozen_ setting, all is as above, but all non-embedding layers are frozen for the duration of tuning. For the _Reduced_ setting, all is as above except for the following:

* Tune learning rate: \(1\cdot 10^{-5}\)
* Time dataset size: \(10\cdot 10^{6}\)

### Generic signalling game

We use the following hyperparameters for the _Disc, small_ emergent language.

* Game (from EGG): egg.zoo.basic_games.play
* Message optimization: Gumbel-softmax (as opposed to REINFORCE)
* Game type: discrimination
* Number of attributes: \(4\)
* Number of values: \(4\)
* Number of distractors: \(5\)
* Vocabulary size: \(6\)
* Max message length: \(10\)
* Number of examples: \(32\,768\)
* Batch size; \(1024\)
* Number of epochs: \(10\)
* Sender hidden size: \(256\)
* Receiver hidden size: \(512\)
* Sender embedding size: \(32\)
* Receiver embedding size: \(32\)
* Sender network type: GRU
* Receiver network type: GRU
* Learning rate: \(0.001\)

The _Disc, large_ setting uses the same hyperparameters as above with the exception of the following.

* Number of attributes: \(12\)
* Number of values: \(8\)
* Number of distractors: \(5\)
* Number of examples: \(3.5\cdot 10^{6}\)
* Max message length: \(30\)
* Vocabulary size: \(100\)
* Number of epochs: \(100\)

The _Recon, large_ setting is as in _Disc, large_ with the following changes.

* Game type: reconstruction
* Number of attributes: \(8\)
* Number of distractors: N/A
* Number of examples: \(1\cdot 10^{6}\)
* Number of epochs: \(10\)

## Appendix B Example of benchmark input format

The input format for the benchmark is simple: integer arrays in a JSON format separated by newlines (i.e., JSON Lines, JSONL, *.jsonl). The following is an example of file contents in this format:

[3, l, 4, l, 5, 9, 2]  [6, 5, 3, 5, 8, 9, 7, 9, 3]  [2, 3, 8, 4]  [6, 2, 6, 4, 3, 3]  [8, 3, 2, 7, 9, 5, 0, 2, 8, 8, 4]

## Appendix C Computing resources used

See Table 3 for rough estimates of the compute used in writing this paper. Most experiments were run on a shared cluster comprising approximately \(150\) NVIDIA A6000 (or comparable) GPUs 



## Appendix D Additional results

### BLEU scores for machine translation

See Table 4.

### Raw cross-entropies on XferBench

See Table 5.

### Writing system matrix for normalized XferBench scores

See Tables 7 and 8. Scores for reach writing system are aggregated by taking the mean. Table 6 gives the writing system classification for the languages used in the experiments. Although the class imbalance makes it impossible to draw any definitive claims, the preliminary results do not suggest any correlation in XferBench between the writing systems of the source and target languages.

### Scatter plots for XferBench and MT See Figure 3.

## E Cross-entropy confidence interval computation

Let \(s\in S\) and \(t\in T\) represent source and target languages, respectively. \(h_{s,t}\) represents the test cross-entropy of a model pretrained on \(s\) and evaluated on \(t\). As stated in Equation (1), the score on XferBench is the mean cross-entropy across all target languages:

\[h_{s}=\underset{t^{\prime}\in T}{\text{mean}}\left(h_{s,t^{\prime}}\right). \tag{3}\]

We would like to calculate a confidence interval (i.e., \(h_{s}^{-}\) and \(h_{s}^{+}\)) for a source language's mean cross-entropy using the different cross-entropies on the target languages (i.e., \(h_{s,t}\) for \(t\in T\)), yet these samples are not i.i.d., since the mean of cross-entropy each _target_ language can vary. Thus, if we would like to use bootstrapping to calculate confidence intervals, we must first normalize the cross-entropies. Let \(\hat{h}_{s,t}\) be the normalized score:

\[\hat{h}_{s,t}=\frac{h_{s,t}-\text{mean}_{s^{\prime}\in S}\left(h_{s^{\prime} ,t}\right)}{\text{stdev}_{s^{\prime}\in S}\left(h_{s^{\prime},t}\right)}. \tag{4}\]

Given the normalized scores, we can now bootstrap in order to compute confidence intervals for \(\hat{h}_{s}\) (i.e., in the normalized space).8 Let \(\hat{h}_{s}^{+}\) and \(\hat{h}_{s}^{-}\) be the upper and lower bounds of the confidence interval computed using bootstrapping in the normalized space. We can now translate these back into the raw cross-entropy space using the means and standard deviations from before:

Footnote 8: This is not intended to be statistically rigorous. Our cross-entropies are unlikely to be normally distributed, but this still be helpful for generally gauging uncertainty.

\[h_{s}^{+} =\hat{h}_{s}^{+}+\underset{s^{\prime}\in S}{\text{stage}}\left( h_{s^{\prime},t}\right)+\underset{s^{\prime}\in S}{\text{mean}}\left(h_{s^{\prime}, t}\right) \tag{5}\] \[h_{s}^{-} =\hat{h}_{s}^{-}\underset{s^{\prime}\in S}{\text{stage}}\left(h_{ s^{\prime},t}\right)+\underset{s^{\prime}\in S}{\text{mean}}\left(h_{s^{\prime}, t}\right). \tag{6}\]

\begin{table}
\begin{tabular}{l r r r} \hline \hline Source & Full & Frozen & Reduced \\ \hline French & 12.93 & 5.33 & 6.61 \\ Spanish & 13.32 & 4.52 & 6.35 \\ Russian & 12.93 & 4.37 & 7.02 \\ Chinese & 12.71 & 3.04 & 6.03 \\ Korean & 12.83 & 2.95 & 6.36 \\ Arabic & 13.12 & 4.16 & 6.74 \\ Hindi & 12.72 & 3.20 & 5.24 \\ Paren, real & 12.60 & 0.65 & 6.26 \\ Paren, synth & 13.19 & 0.82 & 6.15 \\ Disc, large & 12.93 & 2.08 & 4.44 \\ Disc, small & 0.17 & 0.19 & 0.38 \\ Rec, large & 1.92 & 0.86 & 2.50 \\ Yao+ & 0.01 & 1.04 & 2.57 \\ Mu+, SW & 0.00 & 1.05 & 1.86 \\ Mu+, CUB & 12.71 & 1.45 & 2.35 \\ Random & 0.00 & 0.00 & 1.02 \\ No pretrain & 0.10 & 0.06 & 3.43 \\ \hline \hline \end{tabular}
\end{table}
Table 4: BLEU scores for machine translation experiment. Colors normalized by column.

\begin{table}
\begin{tabular}{l r r r} \hline \hline Item & Base GH & \(n\) items & Total \\ \hline XferBench & \(6\) & \(45\) & \(270\) \\ MT & \(8\) & \(50\) & \(400\) \\ Other experiments & \(2\) & \(50\) & \(100\) \\ \hline Total & & & \(770\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Estimate of compute used for this paper in GPU-hours (specifically NVIDIA RTX 2080 Ti–hours).





\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline Source & Danish & Basque & Persian & Finnish & Hebrew & Indonesian & Japanese & Kazakh & Romanian & Urdu & Mean \\ \hline French & 4.93 & 6.03 & 5.04 & 5.62 & 5.48 & 4.87 & 5.23 & 5.46 & 5.15 & 4.43 & 5.22 \\ Spanish & 4.92 & 6.06 & 5.03 & 5.61 & 5.47 & 4.82 & 5.25 & 5.46 & 5.12 & 4.42 & 5.22 \\ Russian & 4.94 & 6.04 & 5.04 & 5.65 & 5.48 & 4.88 & 5.27 & 5.48 & 5.14 & 4.45 & 5.24 \\ Chinese & 4.89 & 6.02 & 5.01 & 5.58 & 5.43 & 4.76 & 5.18 & 5.44 & 5.12 & 4.39 & 5.18 \\ Korean & 4.89 & 6.01 & 5.02 & 5.57 & 5.44 & 4.78 & 5.20 & 5.45 & 5.12 & 4.38 & 5.19 \\ Arabic & 4.90 & 6.02 & 5.02 & 5.59 & 5.45 & 4.81 & 5.22 & 5.44 & 5.13 & 4.40 & 5.20 \\ Hindi & 4.94 & 6.06 & 5.08 & 5.65 & 5.47 & 4.83 & 5.29 & 5.52 & 5.20 & 4.46 & 5.25 \\ Paren, real & 5.07 & 6.11 & 5.11 & 5.75 & 5.59 & 5.06 & 5.38 & 5.57 & 5.22 & 4.56 & 5.34 \\ Parent, synth & 5.08 & 6.13 & 5.14 & 5.74 & 5.58 & 5.09 & 5.43 & 5.58 & 5.26 & 4.57 & 5.36 \\ Disc, large & 5.00 & 6.06 & 5.11 & 5.71 & 5.52 & 4.92 & 5.34 & 5.56 & 5.25 & 4.49 & 5.30 \\ Disc, small & 5.09 & 6.06 & 5.17 & 5.80 & 5.59 & 5.05 & 5.41 & 5.65 & 5.31 & 4.56 & 5.37 \\ Rec, large & 5.09 & 6.06 & 5.16 & 5.79 & 5.57 & 5.04 & 5.41 & 5.64 & 5.30 & 4.55 & 5.36 \\ Yao+ & 5.07 & 6.03 & 5.17 & 5.79 & 5.56 & 5.03 & 5.41 & 5.65 & 5.31 & 4.56 & 5.36 \\ Mu+, SW & 5.09 & 6.10 & 5.18 & 5.80 & 5.58 & 5.05 & 5.42 & 5.65 & 5.33 & 4.58 & 5.38 \\ Mu+, CUB & 5.08 & 6.06 & 5.18 & 5.79 & 5.58 & 5.05 & 5.42 & 5.65 & 5.32 & 4.56 & 5.37 \\ Random & 5.23 & 6.17 & 5.31 & 5.92 & 5.71 & 5.22 & 5.55 & 5.76 & 5.45 & 4.72 & 5.50 \\ No pretrain & 5.17 & 6.10 & 5.23 & 5.85 & 5.66 & 5.14 & 5.47 & 5.68 & 5.38 & 4.65 & 5.43 \\ _Mean_ & 5.02 & 6.07 & 5.12 & 5.72 & 5.54 & 4.96 & 5.35 & 5.57 & 5.24 & 4.51 & 5.31 \\ \hline \hline  & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 5: Cross-entropies across all source and target languages. Colors normalized by column.

Figure 3: Scatter plots showing XferBench score versus machine translation score.





## Appendix F Error analysis

In the _Full_ setting of the machine translation task, the _Yao+_ and _Mu+_, _SW_ settings perform worse than expected (_a priori_ and compared to the other results in the settings). Validation loss converged while chrF and BLEU scores remained near zero. We provide a couple examples (taken from the predefined test set of WMT 2014) of model output to provide some insight into the reason for this. No post processing used, generation is capped at \(50\) tokens, and "u0000" represent single non-printable characters.

**Example 1**: _Input_: "And while Congress can't agree on whether to proceed, several states are not waiting."

_Reference_: "Et tandis que les membres du Congres n'arrivent pas a se mettre d'accord pour savoir s'il faut continuer, plusieurs Etats n'ont pas attendu."

[Model pretrained on] _French_: "#E lators que le Congres ne peut pas conviner de pouvsurve, plusieurs Etats ne sont gas en attente. >> (traduction libre) Le Parlement eeuropoen. Le Parlementen europeen est d'avis que la Complement europeen doit etre en mesure de faire preuvel#"

_Disc, large_: "#Et bien que le Congres ne puisse pas convenir de la marche a suivre, plusieurs Etats ne sonst pas entente.uz028ua208[Traduir par la Redacation]u2028(Traduit par la Redaction)u2028(Trauf"

_Yao+_: "#Annexe II, par.xa02.2.





\begin{table}
\begin{tabular}{l r r r r r} \hline \hline Source & Arabic & Cyrillic & Hebrew & Japanese & Latin \\ \hline Arabic & \(-0.65\) & \(-0.81\) & \(-0.42\) & \(-0.35\) & \(-0.56\) \\ Chinese & \(-1.05\) & \(-0.93\) & \(-1.60\) & \(-1.46\) & \(-1.00\) \\ Cyrillic & \(0.63\) & \(0.68\) & \(1.07\) & \(0.90\) & \(0.78\) \\ Devanagari & \(1.62\) & \(1.93\) & \(0.39\) & \(1.47\) & \(1.23\) \\ Hangul & \(-0.98\) & \(-0.41\) & \(-0.89\) & \(-0.80\) & \(-1.04\) \\ Latin & \(0.22\) & \(-0.22\) & \(0.72\) & \(0.12\) & \(0.29\) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Normalized XferBench scores by writing system (lower is better). Color is normalized across all values.

\begin{table}
\begin{tabular}{l|r r r} \hline \hline Source & Abjad & Alphabet & Mixed \\ \hline Abjad & \(-0.57\) & \(-0.60\) & \(-0.35\) \\ Abugida & \(1.21\) & \(1.35\) & \(1.47\) \\ Alphabet & \(0.15\) & \(0.06\) & \(0.09\) \\ Logographic & \(-1.23\) & \(-0.99\) & \(-1.46\) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Normalized XferBench scores by writing system type (lower is better). Color is normalized across all values.