Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM’s Translation Capability
Year: 2023

Abstract:
Large, multilingual language models exhibit surprisingly good zeroor few-shot machine translation capabilities, despite having never seen the intentionally-included translation examples provided to typical neural translation systems. We investigate the role of incidental bilingualism—the unintentional consumption of bilingual signals, including translation examples—in explaining the translation capabilities of large language models, taking the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method approach to measure and understand incidental bilingualism at scale. We show that PaLM is exposed to over 30 million translation pairs across at least 44 languages. Furthermore, the amount of incidental bilingual content is highly correlated with the amount of monolingual in-language content for non-English languages. We relate incidental bilingual content to zero-shot prompts and show that it can be used to mine new prompts to improve PaLM’s out-of-English zero-shot translation quality. Finally, in a series of small-scale ablations, we show that its presence has a substantial impact on translation capabilities, although this impact diminishes with model scale.

heading: 1 Introduction
text: Recent work has shown that large language models (LLMs) exhibit impressive capabilities in performing various natural language generation tasks, even in the zero-shot paradigm. In particular, such models have shown interesting machine translation (MT) capabilities (Brown et al., 2020; Chowdhery et al., 2022; Vilar et al., 2022)—especially when translating into English, despite never having been explicitly and intentionally exposed to translation data in the way their supervised counterparts are. This raises the question: where do these translation capabilities come from?
We hypothesize that the translation capabilities of LLMs connect to incidental bilingualism: the
unintentional consumption of bilingual text within a single training instance. To test this hypothesis, we take PaLM (Chowdhery et al., 2022)—a 540- billion parameter Transformer language model—as a case study. We first conduct a large-scale analysis of its training data in order to characterize the nature and quantity of bilingual text, then perform experiments to assess the impact of this text on translation performance.
To measure incidental bilingualism at scale, we develop a processing pipeline that alternates between quantitative and qualitative analysis (§3): first detect bilingual versus monolingual text using a language tagger, then qualitatively analyze the nature of bilingual text, and finally measure the amount of translation data within bilingual instances. Our analysis spans 44 languages, for which we study bilingualism paired with English. Our findings are:
• In all, 1.4% of PALM’s training instances are detected as bilingual, while 0.34% contain at least one translated sentence pair. We were able to mine such pairs across all languages studied; therefore, none of these languages is truly zero-shot in the context of translation.
• The number of monolingual instances in a language is predictive of the number of instances containing bilingual or translation content for that language (paired with English).
After establishing that both bilingual and translation content are incidentally consumed during PaLM’s training, we study how they connect to its MT capabilities (§4). We run a series of training and prompting experiments and found that:
• Prompting the full PaLM model with alternative, data-driven prompts improves outof-English zero-shot translation by 14 chrF points on average across languages, indicating
9432
that its zero-shot translation capabilities were underestimated due to sub-optimal prompts.
• Ablating detected translation pairs with smaller versions of PaLM has a dramatic effect on the translation capabilities of 1Bparameter models for high-resource languages, reducing average into-English zeroshot results by 7.4 BLEU and 5-shot results by 5.9 BLEU. The effect falls off but remains notable (+2-3 BLEU across several conditions) as we scale to 8B-parameter models.

heading: 2 Related Work
text: Translation Capabilities of LLMs Large-scale generative language models, such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and XGLM (Lin et al., 2021) have been shown to exhibit translation capabilities, despite not being explicitly trained to translate. These capabilities are surprisingly strong, particularly when translating into English with few-shot examples. One explanation for this behavior is that it results from incidental multitask learning (Radford et al., 2018; Sanh et al., 2021). This hypothesis has not been explored for MT, where recent work has mostly focused on improving LLM translation capabilities by optimizing few-shot prompting strategies (Vilar et al., 2022; Agrawal et al., 2022). Rather than trying to improve translation quality for LLMs, our goal is to understand where their translation abilities stem from by tracing them back to the properties of the pretraining data.
Large-Scale Data Analysis LLMs rely on massive amounts of unlabeled corpora for training. These corpora are primarily acquired by combining heterogeneous online resources (e.g., Wikipedia, Web forums, Common Crawl, etc.)—whose properties are usually unknown. Recent work on largescale analysis has shed some light: Dodge et al. (2021) analyze C4 (Raffel et al., 2019)—a dataset created from a snapshot of Common Crawl—and show that it contains machine generated texts as well as evaluation samples from commonly used NLP benchmarks; Kreutzer et al. (2022) manually audit the quality of multilingual datasets and find systematic quality issues amongst popular pretraining datasets. Most related to our work, Blevins and Zettlemoyer (2022) show that popular corpora routinely used for training English-only LLMs contain a non-negligible amount of non-English text,
which helps explain their cross-lingual capabilities. Their manual analysis of corpus subsamples covers several bilingual categories, including a translation category. But where analysis of bilingualism is a side result of their work, it is our primary contribution. We extend their work by proposing automatic tools to quantify bilingualism at scale and directly relate it to LLM translation performance.
Eliciting Knowledge from LLMs Prompting language models to elicit knowledge acquired during pre-training has received a lot of research interest. Petroni et al. (2019) show that LLMs can recall factual knowledge by answering queries structured as cloze statements. Jiang et al. (2020) further show that query-based prompts outperform manually created cloze statements, suggesting that the latter provide a lower bound estimate on the actual abilities of LLMs. Follow-up work confirms those findings by suggesting better prompts with automatic generation methods (Shin et al., 2020) or prompt engineering (Reynolds and McDonell, 2021). We similarly explore how to extract translation knowledge from LLMs using data-driven prompts.

heading: 3 Measuring & Understanding Incidental Bilingualism
text: We introduce a mixed-method approach (Creswell and Clark, 2017; Shorten and Smith, 2017) to measure and understand incidental bilingualism—the unintentional consumption of bilingual signals— at scale. Since we expect bilingual signals to be rare, we explore the huge data space by alternating between quantitative and qualitative steps, with results from each step complementing and informing one another (Figure 1). The quantitative steps play the role of inducing a smaller-scale focus space to study, while the qualitative steps provide insights into the nature of bilingual signals.
Preliminaries PaLM’s pretraining dataset consists of 780 billion tokens from a mixture of multilingual sources (social media conversations (50%), filtered webpages (27%), and Wikipedia (4%)), presumably English sources (books (13%) and news articles (1%)), and source code (5%). PaLM was trained on 2,048-subword-token examples formed by concatenating and truncating documents. As PaLM is a multi-source LM, a document may be a web page, a book, or a conversation, depending on the source. Our primary units for data analysis are instances we created by splitting training
examples along document boundaries. As such, each instance is either a complete document or a contiguous fragment of one, up to 2,048 tokens in length. A more detailed discussion of instances is given in Appendix A.
We study bilingualism between English and 44 other languages. We choose language pairs that: a) are supported by our language identification models, and b) have FLORES-101 (Goyal et al., 2022) evaluation data. We divide languages into high, medium, and low-resource groups according to their monolingual instance counts, as shown below:
HIGH FR, DE, ES, IT MEDIUM PT, RU, ZH, JA, AR, ID, KO, VI, FA, SR, UK LOW PS, HY, IW, BG, KK, BE, HI, UR, EL, TH,
MK, KY, BN, KA, TG, SD, NE, TA, MN, PA, TE, ML, MR, AM, MY, KN, KM, GU, LO

heading: 3.1 Detecting Bilingual Instances
text: Our first goal is to automatically detect all training instances that contain bilingual text without presupposing a specific granularity for bilingualism. To that end, we use CMX (Zhang et al., 2018)—a language identification model for codemixed texts—to produce a sequence of token-level language tags for each training instance. An instance is labeled as bilingual if it contains at least two contiguous segments in different languages, each consisting of at least N consecutive identical language tags. Instances with more than two languages are interpreted as bilingual, as discussed in Appendix B. One of the two languages must always be English, both to simplify our analysis and to work within the limits of the CMX tool.
Findings Figure 2 presents the per-language monolingual and bilingual instance counts. We include raw counts per language in Table 7. We
observe that across the languages studied, PaLM consumes bilingual instances that, in total, account for 1.4% of its training instances.

heading: 3.2 Characterizing Bilingual Instances
text: Next, we turn to understanding the nature of bilingual instances detected by the above procedure. To make manual analysis easier, we used the KnowYourData tool1 to highlight spans of the less frequent language in each bilingual instance.
Findings Our qualitative analysis of a sample of 100 English-French bilingual instances reveals that bilingualism manifests in various cross-lingual phenomena (examples of bilingual instances are presented in Table 8 of Appendix E). Our detection approach is reasonably accurate: only 5% of instances correspond to errors mostly attributed to language identification issues (i.e., the detected instances are indeed bilingual, but at least one of the two languages is not English or French). Each correctly detected bilingual instance is annotated as belonging to one of five categories, with the typology shown in Figure 3.
Most bilingual instances (55%) fall under the broader class of “Not Translations” and cover cases
1https://knowyourdata.withgoogle.com
where the two languages encode information that does not correspond to translation content. This class is further decomposed into three sub-classes. First, we found a few instances (10%) of codeswitching where one or two speakers alternate between two languages in the context of a single conversation. As expected, most code-switching instances were spotted in social media conversations, as it is primarily used within multilingual communities in informal communication. Second, we observed that many bilingual instances (21%) are attributed to references, where named entities or bibliography entries are cited in their native language, such as instances drawn from Wikipedia. Third, we also found a considerable number of bilingual instances (24%) that include completely unrelated content in the two languages that just happened to co-exist within the same web page.
The remaining bilingual instances are evenly distributed (20%) across two categories that fall loosely under the rubric of “Translations”. Here, we distinguish between cases where some amount of the text expresses a typical translation relation and cases where content across languages is semantically related, but not exactly by translation. The latter involves a rich spectrum of cross-lingual semantic relations, including cross-lingual entailment, summarization, and paraphrasing, mainly noticed within books in the genre of literary criticism and interpretation. We also spotted a few cases of forum discussions around explanations of translation or stylistic manipulation of translations.

heading: 3.3 Detecting Translation Pairs
text: Our manual analysis exposed an opportunity to automatically extract and count translated sentence pairs (translation pairs for short). We cast the
problem of within-instance translation detection as a local mining task following recent advances in parallel text acquisition. Concretely, for each bilingual instance from §3.1, we run a sentence breaker and extract two pools of candidate sentences x and y in the two languages. The language of each sentence is inferred by majority voting over token-level language tags. Whichever language has fewer sentences is labeled the embedded language and the other becomes the primary. Each candidate sentence is then encoded to a vector representation using the LABSE (Feng et al., 2022) cross-lingual sentence encoder. Translation pairs are extracted by finding the most similar primary sentence for each embedded sentence and then checking whether the cosine distance of their representations falls below a threshold. We choose a threshold of 0.6 on the cosine distance to mine plausible translation pairs, following Feng et al. (2022). We also apply a series of length-and-language-based heuristic data quality filters, adapted from Alibaba’s WMT Data Filtering submissions (Lu et al., 2018, 2020), described in Appendix C.
Note that this extraction process is oblivious to document structure: the instance may be formatted as parallel sentences, paragraphs, documents, or as a free-form discussion that happens to mention both a sentence and its translation. Our extraction is also incapable of detecting translation relations below the sentence level. If we can extract at least one translation pair from an instance, then we label it as a translation instance.
Findings We find that 0.34% of PaLM’s training instances contain at least one translation pair. Note that this number provides a lower bound on the amount of incidental bilingualism and translation that PaLM consumes, as we are restricted to a specific set of language pairs, and we only study bilingualism with English. Figure 4 presents the number of translation pairs we mined within PaLM’s training instances between English and each language. At a minimum, PaLM consumes thousands of parallel texts for all language pairs studied, while for high-resource languages it sees more than a million translation pairs.
Furthermore, we investigate the correlation between the number of monolingual instances in each language and their bilingual and translation counterparts. Our results in Figure 5 indicate that, surprisingly, the monolingual counts in each language correlate strongly with the bilingual (r=0.944) and
translation (r=0.938) counts. This strong correlation implies that, when working at scale, we can predict the bilingual and translation sizes for a given language (within an error rate) by simply counting monolingual instances.

heading: 3.4 Discovering Natural Prompts
text: After identifying a smaller-scale set consisting of training instances that contain translation pairs, we further manually inspect them to understand how the translation task is naturally modeled by PaLM. We find that sentence-level translations are presented within a training instance in three ways. The majority of them appear across paragraphs and do not follow a canonical pattern. Among the remainder, we noticed two canonical patterns: translation pairs that belong to stacked translated paragraphs (e.g., {x1, x2, y1, y2}) and interleaved translations where a sentence and each translation are adjacent to each other (e.g., {x1, y1, x2, y2}). Among the latter, we saw an opportunity to extract natural prompts automatically. We do so by analyzing the prefixes of the translation pairs mined in §3.3. Drawing on our manual observations, we mine the most frequent prefixes per language pair that follow a simple colon prompt format: any sequence of non-whitespace characters followed by a colon. Finally, we manually filter the automatically mined
prefix lists to look for consistent natural prompt patterns across languages.
Findings Table 1 presents the results of our prompt discovery module followed by manual filtering to extract plausible translation prefixes. First, we found empirically that one of the most frequent translation prompts that naturally arises in the data is the default prompt adopted by most MT research with LLMs: source and target language names in English followed by a colon (e.g., “French:”). We also found three alternative prompts that are frequently presented within incidental translation pairs: i) code: source and target ISO language codes (e.g., “FR:”), ii) native: source and target language names in their respective languages (e.g.,
“Français:”), iii) translation: source language in English, and the word “translation” in the target language (e.g., “Traduction:”). Interestingly, prompt types are not evenly distributed across our language groups: language codes appear primarily with highresource languages, while low-resource languages favor prompts written in their native language. We include a complete list of prompt counts per language in Figure 6 of Appendix E.

heading: 4 Analyzing the Impact of Bilingualism
text: We analyze the impact of bilingualism on the translation capabilities of PaLM with a series of MT experiments on the FLORES-101 (Goyal et al., 2022) evaluation set, which provides translations of a common set of English Wikipedia sentences into all of our 44 languages. We report results on the 1,012 sentence devtest set. We use the 997 sentence dev set primarily as a source of randomlydrawn exemplars when reporting 5-shot results. We report BLEU (Papineni et al., 2002) for intoEnglish translation and chrF (Popović, 2015) for out-of-English translation, both computed by Sacrebleu (Post, 2018) with default settings. For LLMbased translation, we follow the template from Vilar et al. (2022) unless stated otherwise:
[source]: [X]
[target]:
where [source], and [target] are the source and target language names (in English) and [X] is the source text. When present, few-shot exemplars are provided above the template in the same format, as detailed in Appendix D.

heading: 4.1 Prompting PaLM with Natural Prompts
text: We prompt the original 540B parameter PaLM model with templates that use naturally-occurring
prefixes of incidental translations, as discussed in §3.4. In our template, we replace [source] and [target] with each alternative, data-driven prompt. We experiment with zero-shot and 5-shot prompting.
Findings Table 2 presents average translation quality results for different prompts across high, medium, and low resource settings. We present the complete, per language results in Table 9 of Appendix E. When translating into English (XX→EN), the default prompt yields the best results, while alternative prompts result in a small degradation in quality; overall, translating into English seems to be robust across different prompts supported by our data. On the other hand, PaLM’s translation quality is surprisingly sensitive to the choice of prompt when translating out of English (EN→XX): simply changing the default prompt to its native variant improves quality by 14 chrF points, with most of the improvement reported in medium and low-resource languages. The “translation” prompt also yields consistent improvements over the default. Finally, prompting with language codes only improves translation out of English for the highresource group—this is expected as this prompt was only present for a few high-resource languages. Further analysis of out-of-English results reveals that native prompts trigger text in the desired language, while the default prompt results in high rates of generating the wrong target language (see gray percentages in Table 2). The output’s target language is determined by a sequence-level languageidentification tool (Botha et al., 2017).
Finally, although choosing natural prompts that arise from the data can help us better understand PaLM’s zero-shot capabilities, large differences between prompts do not carry over to the few-shot setting (right-most columns of Table 2).

heading: 4.2 Extrinsic Evaluation of Translation Pairs
text: It is one thing to report counts of translation pairs mined from bilingual instances, but is the resulting bitext of high quality? We adopt the parallel text quality evaluation framework of the WMT Shared Task on Parallel Corpus Filtering and Alignment (Koehn et al., 2020) and train supervised neural machine translation models from scratch on the mined translations. This allows us to jointly assess the quality of PaLM’s translation content and our extraction heuristics. We focus this analysis on FR→EN, PaLM’s highest-resource language pair.
Data For PaLM translation pairs, we explore a number of thresholds on the LABSE distance. To put our results in perspective, we additionally train a model on all pairs from the WMT14 FR→EN task (Bojar et al., 2014) and on random samples thereof to establish fair data comparison points at notable LABSE thresholds. Sentence counts for all conditions are shown in Table 3.
Architecture We adopt the 6-layer encoderdecoder Transformer Base (Vaswani et al., 2017) architecture, with minimal hyper-parameter tuning. Shared sentence piece (Kudo and Richardson, 2018) vocabularies with 32K tokens are constructed from bitext for each scenario. Dropout is set to 0.3 for all systems except for the full WMT system, which uses 0.1. Systems are trained up to 450K steps with a batch size of 1,024. Checkpoints are selected by FLORES dev BLEU.
Findings Table 3 presents the results of our analysis. In general, the mined translation pairs from our analysis pipeline provide useful signal for training supervised MT systems with reasonable translation quality (i.e., 37 to 38 BLEU across various thresholds, compared to 41 that we achieve using 40M translations from available WMT parallel corpora). Moreover, these results confirm that 0.6 seems to be the right threshold for detecting translation pairs that are useful, or at least not harmful in the presence of other positive signals (i.e., at 0.6 we are within 1 BLEU point of a system trained on the same amounts of WMT parallel text).

heading: 4.3 Ablating Incidental Bilingualism
text: We now explore the impact of bilingualism on the translation capabilities of PaLM. To do so, we conduct smaller-scale experiments by training 1B and 8B parameter models on different training samples
to measure the effect of removing various types of multilingual data.
Architecture Our 1B and 8B models are scaleddown versions of PaLM with small changes. Like PaLM, each is a decoder-only model trained with a causal language modeling objective, using a dense transformer architecture and a sentence piece tokenizer (Kudo and Richardson, 2018) that retains spacing information. Unlike PaLM, we do not share key and value tensors across attention heads (Shazeer, 2019), which should affect only decoding speed. We include a hyper-parameter summary in Table 6 in Appendix E. Also, we use a smaller vocabulary size of 128K tokens compared to PaLM’s 256K tokens, a concession to fit the models onto available hardware. Both 1B and 8B train on examples of 2,048 tokens with a batch size of 512 for 100K steps. Note that using the same number of examples for both scales means that the 8B models are likely under-trained; however, holding data quantity constant is useful for directly measuring the effect of model scale.
Data To simulate PaLM’s data conditions with smaller models, we begin by partitioning PaLM’s training instances into four non-overlapping groups: ENG: English instances, NEN: non-English (excluding bilingual) instances, BIL: bilingual (excluding translation) instances, and TRA: translation instances. We then merge instances within their groups into 2,048 token examples. Counting examples from each group allows us to determine the full data’s implicit mixture of these groups: ENG: 84.4%; NEN: 14.1%; BIL: 1.0%; TRA: 0.5%. These should not match the instance-
level proportions reported earlier, as these count examples, which are merged instances. Also, they will not match the multilinguality proportions reported by Chowdhery et al. (2022), as we have removed non-natural-language (code) data and any non-English text not in our 44-language set. We can now sample examples from our partitions to create a smaller training set with the same proportions of incidental bilingualism. No attempt is made to retain PaLM’s original proportions for other aspects like data source or language. Counts for this sample are shown as FULL in Table 5.
We ablate each group in the following order: TRA, BIL and then NEN. At each step, we replace ablated examples with examples from the next group in the chain. The counts for all ablation conditions are shown in Table 5. The -NEN setting corresponds to the English-only setting studied by Blevins and Zettlemoyer (2022), but as they show, this will contain some non-English content due to language-identification errors. Analogous provisos exist for each ablation, as all our automatic tools make errors. We aim to measure the effect of removing most of a type of content, not all of it.
Findings Table 4 presents the results of our ablation—the complete, per language, results are in Table 10 of Appendix E. Focusing on our 1B model, we note that examples containing translation pairs (TRA) have an outsized impact on translation quality for being only 0.5% of the training data. In the high-resource XX→EN, zero-shot scenario, replac-
ing TRA examples with BIL results in a drop of 7.4 BLEU. With TRA removed, the additional impact of removing the remaining bilingual instances (BIL) is much smaller: 1.2 BLEU. One might expect the utility of translation data to fall off as we add 5-shot examples at inference time, but TRA is still quite important, with its removal resulting in a reduction of 5.9 BLEU. The importance of TRA holds throughout our 1B experiments, to the extent that the system cannot translate at all, i.e. for 5-shot versions of XX→EN MEDIUM and EN→XX HIGH.
Turning to our 8B model, we see that translation content continues to have a substantial impact on translation quality, though the absolute score differences have diminished, hovering between 2-3 BLEU or 3-4 chrF, depending on the scenario. This result, where a 4x increase in parameters leads to a roughly 2x reduction in the absolute impact of TRA suggests that it would be interesting to build scaling laws to study the impact of incidental translation data, which we leave to future work. Also, for 5-shot scenarios, there is no longer such a big difference between the impact of BIL and TRA data. Given exemplars, the larger model seems to be able to make better use of weaker bilingual signals.
Surprisingly, the 8B model that does not have access to multilingual content (-NEN), exhibits some translation capabilities for XX→EN HIGH (i.e., 17.3 and 25.9 BLEU for zero- and few-shot, respectively). A closer look at the per-language breakdown (see Table 10) reveals that those capabilities are restricted to languages written in Latin script. This adds evidence for larger models being better equipped to leverage either sparse signals (i.e., language-identification failures during ablation) and weak signals (i.e., language similarities from shared scripts). As expected, non-English content is critical for translation out of English.

heading: 5 Conclusion
text: We explore the role of incidental bilingualism—the unintentional consumption of bilingual signals— in PaLM’s translation capabilities. We introduce a mixed-method approach that alternates between quantitative and qualitative analyses to measure and understand incidental bilingualism at scale by processing 780 billion tokens. Our work shows that PaLM consumes a significant amount of bilingual text: 1.4% of training instances in natural language are bilingual. At the same time, it is naturally exposed to translation signals, having seen more than 30 million translation pairs in 44 languages paired with English. Furthermore, we extrinsically evaluate the quality of these translations, showing that they can be used to train supervised models that roughly match the quality of equal amounts of WMT data. Finally, we show that incidental bilingualism connects to the machine translation capabilities of PaLM. First, we show that data-driven prompts extracted from incidental translations can improve the zero-shot abilities of PaLM when translating out of English by 14 chrF on average. Second, we provide empirical evidence that bilingual and translation signals can partially explain the translation capabilities of smaller-scale LLMs.
Limitations
Our findings should be interpreted considering a series of problem definitions and design choices. First, our quantitative results on measuring incidental bilingualism at scale are subject to language identification, sentence splitting, and mining errors. Our qualitative analysis for the English-French language pair revealed that those errors are reasonably small (see §3.2). However, we expect the accuracy of our tools to vary across languages and, crucially, exhibit unanticipated failure modes on web text and low-resource languages (Caswell et al., 2020). Second, our findings are restricted to quantifying bilingualism and translations within a limited set of language pairs and only paired with English. Thus, by problem definition, we are limited to computing a lower-bound estimate on incidental bilingualism of PaLM. The above limitations should also be taken into consideration when interpreting our ablation results. Although we attempted to remove most bilingual signals in our series of MT experiments, it is still possible that bilingualism slips through due to either model errors or due to bilin-
gual signals beyond our focus set of languages. Finally, any results and findings of our work are restricted to PaLM; the single LLM studied in this work. However, our finer-grained analysis (see Table 11 of Appendix E) reveals that incidental bilingualism, including translation signals, is observed across various data sources (e.g., webpages, books, etc.) that are commonly included in the training data of other popular LLMs.

heading: Acknowledgements
text: We thank Jiaming Luo, Julia Kreutzer, Orhan Firat, Xavier Garcia, Markus Freitag, Sweta Agrawal, Marine Carpuat, Elijah Rippeth, and the anonymous reviewers for their helpful and constructive comments.

heading: A Units of Analysis of Training text
text: Throughout this paper we have adopted special meanings for the common (often interchangeable) terms document, example and instance. Here we make those terms concrete and justify our use of the instance as our primary unit of analysis.
Document A document is a logical unit of text from one of our source corpora: a web page or wiki page from a web-crawled corpus, a conversation from a chat or forum corpus, or a book from a books corpus.
Example Each PaLM training example is exactly 2,048 subword tokens. These are assembled by concatenating and/or splitting documents to the appropriate length. As such, an example may contain several short documents, and a long document may be spread over several examples. Multiple documents concatenated into a single example are separated by special document-boundary tokens. The relevant features of examples that make them more useful for analysis than documents are:
• We know exactly which examples PaLM saw during training.
• Examples reflect when co-occurring textual information (for example, a translation pair) was lost due to a document being split into multiple examples.
However, examples can also introduce spurious co-occurrences from merged documents. We assume that a language model can and will ignore any merge-induced co-occurrences due to the presence of document separator tokens; therefore, we should ignore them as well. This leads us to our next and final unit.
Instance Instances are created by splitting examples according to document-separator tokens. Therefore, each instance is either a complete document or a fragment of a single document, and is up to 2,048 tokens in length. Instances have all of the advantages of examples, without introducing spurious co-occurrences, hence why they are our primary unit of analysis.

heading: B Bilingual Detection Pipeline Details
text: CodeMixer Model Details We use the CMX (CodeMixer) model (Zhang et al., 2018)—a tokenlevel language identification model, to detect bilingual instances. CMX is a simple feed-forward
model that takes as input a set of character and word-level features and produces a distribution over a set of languages for each token. The entire sequence of language tags is obtained using constrained decoding over a pre-defined set of permitted languages. The model is trained on a combination of synthetic and real-world translation data (both monolingual and code-mixed with English) for 100 languages. Note that CMX predicts codemixing between a pair of languages, as a result, it does not reliably predict language tags for multilingual instances involving more than two languages. For example, if an instance actually contains English, French, and German text, with German being the least frequent, it will be tagged as containing only English and French; all German words will be mislabeled as one of the other two languages or as “undefined.”
Algorithmic Description of Bilingual Detection Given a training instance t = {ti}ni=1, a focus set L of the 44 studied languages, and a threshold N , we detect bilingual instances based on the following steps: (i) We start by extracting a sequence of language tags, using the CMX model. (ii) We mark the most frequent language as the primary language, and the other (if exists) as the embedded. (iii) If the primary and the embedded languages do not fall under our focus set L, we exclude it from our analysis. (iv) If a training instance contains more than 10% of “undefined” predictions (e.g., resulting from non-linguistic content), it is not annotated as bilingual. (v) Finally, if a training instance contains at least two contiguous segments—consisting of at least N consecutive identical language tags—in different languages, it is annotated as bilingual.
Given that the CMX model is known to overpredict English tags, we employ a stricter threshold on defining contiguous segments for English (N = 10) compared to the rest of the languages (N = 5). For all languages we operate at the tokenlevel, with the exception of Chinese, Japanese, and Korean for which we apply the above algorithm at the character-level.

heading: C Heuristic Translation Pair Filters
text: When extracting translation pairs found within a bilingual instance, our primary quality signal is from the cosine distance between cross-lingual LABSE sentence embeddings. However, we also apply a suite of heuristic filters which help catch non-translations that slip through this primary fil-
ter. These filters are adapted from Alibaba’s WMT Data Filtering submissions (Lu et al., 2018, 2020). When a tokenization is required for token counts or edit distance, we use tokens from the mBERT tokenizer (Devlin et al., 2019). The filters are as follows: 1. both sentences must respect a min (3) and max (200) token length; 2. we enforce a max length ratio (2x) between sentences; 3. we enforce a min edit distance (2) and a min edit distance ratio (0.1) between sentences; 4. we apply a secondary, sequence-level language-identification tool (Botha et al., 2017) to re-identify each side of the pair and ensure that the two halves are written in different languages. When extracting sentences to train Transformer Base MT systems in §4.2, the different-language check is replaced by a check to ensure that the translation pair respects the language pair being studied, i.e.: one sentence is in English and the other is in French.

heading: D Prompting Details
text: For 5-shot prompting experiments we used the following format (e.g., for French to English translation):
French: [X1] English: [Y1 ] ...
French: [X5] English: [Y5 ] French: [X ]
English:
Each slot (Xi, Yi) is filled with five translation examples that are randomly sampled from the devtest split of the FLORES dataset, while the final slot X , is filled with the source text that comes from the test split of FLORES.
E Additional Tables and Figures
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
"Limitations” section.
7 A2. Did you discuss any potential risks of your work? Our work aims at understanding LLMs better; this analysis is unlikely to lead to any technology that would cause harm beyond the harms that are already widely known for LLMs.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.
3 B1. Did you cite the creators of artifacts you used? Section 3.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? This paper analyzes data that was previously collected to train PaLM, and which is described in detail in the PaLM paper (Chowdhery et al., 2022; cited prominently in our paper). We did not discuss the licenses for that data in this paper, but we verified that our use of the data was permitted for research purposes. We are not distributing artifacts.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We are not producing any new data artifacts. The quantities of Personally Identifiable Information in the data that we study are discussed in detail in Appendix C of Chowdhery et al., 2022.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sections 3, 4, and Appendix C.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix D.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 and Appendices B & C.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This result, where a 4x increase in parameters leads to a roughly 2x reduction in the absolute impact of TRA suggests that it would be interesting to build scaling laws to study the impact of incidental translation data, which we leave to future work. Also, for 5-shot scenarios, there is no longer such a big difference between the impact of BIL and TRA data. Given exemplars, the larger model seems to be able to make better use of weaker bilingual signals. Surprisingly, the 8B model that does not have access to multilingual content (-NEN), exhibits some translation capabilities for XX→EN HIGH (i. e. , 17. 3 and 25. 9 BLEU for zero- and few-shot, respectively). A closer look at the per-language breakdown (see Table 10) reveals that those capabilities are restricted to languages written in Latin script. This adds evidence for larger models being better equipped to leverage either sparse signals (i. e. , language-identification failures during ablation) and weak signals (i. e. , language similarities from shared scripts). As expected, non-English content is critical for translation out of English