<figure><img src="image_1.png" /></figure>

HAL
open science

# Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM

Rachel Bawden, François Yvon

<figure><img src="image_2.png" /></figure>

To cite this version:

Rachel Bawden, François Yvon. Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM. EAMT 2023 - 24th Annual Conference of the European Association for Machine Translation, Jun 2023, Tampere, Finland. 10.48550/ARXIV.2303.01911. [hal-04015863v2](https://hal.inria.fr/hal-04015863v2)

HAL Id: hal-04015863

https://inria.hal.science/hal-04015863v2

Submitted on 9 May 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM

**Rachel Bawden**

Inria, Paris, France

rachel.bawden@inria.fr

**François Yvon**

Université Paris-Saclay, CNRS, LISN

francois.yvon@cnrs.fr

## Abstract

The NLP community recently saw the release of a new large open-access multilingual language model, BLOOM (BigScience et al., 2022) covering 46 languages. We focus on BLOOM's multilingual ability by evaluating its machine translation performance across several datasets (WMT, Flores-101 and DiaBLa) and language pairs (high- and low-resourced). Our results show that 0-shot performance suffers from over-generation and generating in the wrong language, but this is greatly improved in the few-shot setting, with very good results for a number of language pairs. We study several aspects including prompt design, model sizes, cross-lingual transfer and the use of discursive context.

## 1 Introduction

Large language models (LLMs) trained at scale with simple objectives have been found to achieve results that match dedicated systems on numerous NLP tasks (Radford et al., 2019), as long as tasks are formulated as text generation though “prompting” (Liu et al., 2023). LLMs’ multi-task performance can even be improved with “instruction” fine-tuning (Sanh et al., 2022; Muennighoff et al., 2022), few-shot priming, and better strategies to select or learn prompts (Petroni et al., 2019; Shin et al., 2020; Schick and Schütze, 2021; Lester et al., 2021; Wei et al., 2022). In multilingual settings, their performance on machine translation (MT) tasks, as measured by automatic scores, is often close to state of the art, even when mostly trained on monolingual data (Brown et al., 2020). Moreover, prompting-based MT offers the prospect of better control of outputs, e.g. in terms of quality, style and dialect (Garcia and Firat, 2022). However, these abilities remain poorly understood, as LLM analyses primarily focus on their multitask rather than multilingual ability (see however (Vilar et al., 2022; Zhang et al., 2023; Moslem et al., 2023), which we discuss in Section 2).

In this work, we focus on the MT performance of BLOOM (BigScience et al., 2022), a (family of) open-access multilingual LLM(s), designed and trained by the collaborative BigScience project.<fnref n="1" /> Our main aims are to (i) evaluate BLOOM’s zero- and multi-shot behaviour, (ii) study the effect of prompt design, (iii) evaluate a diverse set of language pairs and (iv) assess its ability to use linguistic context. Our main conclusions, which extend those in (BigScience et al., 2022), are (i) 0-shot ability is blighted by overgeneration and generating in the wrong language, (ii) using few-shot improves both issues, with results much closer to state of the art across datasets and language pairs, (iii) there are clear transfer effects, with high scores for languages not officially seen in training, and successful transfer across language pairs via few-shot examples and (iv) although linguistic context does not lead to higher scores, there is evidence that BLOOM’s translations are influenced by it. We release our code and translation outputs.²

## 2 Related work

Since the early attempts to use language models (LMs) as multi-task learners (McCann et al., 2018), MT has been a task of choice to gauge LMs’ multilingual ability. Results for the zero- and few-shot ability of LMs were discussed for both GPT-2 and GPT-3 (Radford et al., 2019; Brown et al., 2020), which is especially intriguing as they were trained primarily on monolingual (English) data. These results have since been confirmed for other monolingual LMs such as T5 (Raffel et al., 2020) and multilingual LMs such as XGLM (Lin et al., 2022), PALM (Chowdhery et al., 2022), and ALEX-ATM (Soltan et al., 2022). However, the focus has mainly been on global multi-task performance; often only a small part of the discussion is devoted to

<footnote n="1">¹https://hf.co/bigscience/bloom
²https://github.com/rbawden/mt-bigscience</footnote>
MT. Moreover, results are often only reported for
a few well-resourced language pairs (e.g. English-
French and English-German), and the scores re-
ported (mostly BLEU), are hard to compare due
to a non-systematic use of standardised evaluation
protocols and metrics. ³

There are however some in-depth analyses of
MT performance of LLMs, each focusing on a spe-
cific LM’s performance in a true multilingual set-
ting with respect to prompt design and number of
few-shots. For instance, Vilar et al. (2022) reevalu-
ate the MT performance of the multilingual PALM
(Chowdhery et al., 2022), focusing notably on the
selection of few-shot examples. Consistent with
our findings, they determine that prompt choice
becomes unimportant in few-shot settings and that
using few-shot examples increases performance
with diminishing returns for *k* > 5 examples, using
BLEURT and BLEU scores, as well as the results
of a human evaluation. They find that the quality
of few-shot examples has a large impact on perfor-
mance. However, even with good prompts, PALM
lags a couple of points behind state-of-the-art MT
systems, especially when translating from English,
notable due to adequacy problems. Zhang et al.
(2023) focus on the evaluation of GLM-130B, a
bilingual (Chinese and English) LLM (Zeng et al.,
2022). Their main conclusions are also consistent
with ours: (a) zero-shot performance varies greatly
across different prompts, (b) increasing the number
of prompts from 0 to 20 yields consistent improve-
ments in performance, again with variance across
instructions, and (c) finding the best few-shot ex-
ample selection policy is difficult. It seems that
having good and long examples, for instance, may
help, even though none of the criteria explored in
this study seem to provide any systematic improve-
ment. A last point worth mentioning is that prompt-
ing with monolingual data hurts performance, but
that using pseudo-parallel data obtained with back-
translation (Bojar and Tamchyna, 2011) is an effec-
tive workaround.

Moslem et al. (2023) evaluate OpenAI's GPT-3 (Brown et al., 2020)⁴ with sampling-based decoding and a prompt resembling our own `xglm-source+target` prompt. They report strong zero-shot behaviour using multiple metrics,

plus clear improvements with an increased number
of shots for the well-resourced languages, less so
for the only low-resource language in their lot (Kin-
yarwanda). The main novelty of this study is to use
prompting as a vehicle to perform local adaptation
and to ensure terminological consistency. For this,
they use fuzzy matches from a translation memory
as well as MT outputs to build their prompts, yield-
ing results that both outperform their zero-shot sys-
tem, but also their initial MT engine. Additionally
inserting terms and their translation in the instruc-
tion yields supplementary improvements.

Finally note the preliminary evaluation of CHAT-GPT in (Jiao et al., 2023), which reports interesting insights regarding the multilingual abilities of this model, as well as proposing innovative techniques to generate (artificial) prompts and to use pivoting in prompting. Similar to ours, this study considers multiple test domains such as news (WMT) and Wikipedia (Flores). A more in-depth analysis of the same model can be found in (Hendy et al., 2023), which confirms CHATGPT's strong translation abilities, at least for "well-resourced"⁵ language pairs. Document-level scores are also reported, as well as human evaluations and qualitative analyses.

Multilingual MT is also the subject of dedicated
(monotask) architectures and training regimes.
Originally introduced in (Dong et al., 2015; Fi-
rat et al., 2016; Luong et al., 2016) with limited
language coverage, the latest versions of these ap-
proaches are able to handle hundreds of languages,
including very low-resource language pairs (Fan
et al., 2021; Bapna et al., 2022; Costa-jussà et al.,
2022). Although we found that BLOOM is able to
match this performance, given sufficient training
data, we also see that it still lags behind for many
languages pairs that are under-represented in its
training data.

3 BLOOM Language Model

BLOOM is a large open-access multilingual model
trained on 46 natural languages developed within
the BigScience project (BigScience et al., 2022).
It is an auto-regressive language model designed
to generate text to complete a user-entered text
prefix, known as a prompt. It can be used for
multiple tasks, including MT, question answer-
ing, etc. BLOOM was trained on 1.6TB of text

³See the discussion at http://blog.benjaminmarie.com/2/comparing-uncomparable.html of these differences, and an attempt to reconstruct consistent scores.

⁴Version: text-davinci-003 model.

⁵A rather slippery concept in this context, as the content
of the training data is not fully known and seems to mostly
comprise English texts.
<table><caption>Table 1: Seven MT prompts for the WMT'14 dataset (Bojar et al., 2014). All prompts specify the target language (L2). Each prompt exists in a 'target-only' version (-target), where only the target language is specified, and two prompts also exist in a second -source+target version, where the source language (in red and in brackets) is explicit in the instruction.</caption><thead><tr><th></th><th>Prompt name</th><th>Prompt</th><th>Target</th></tr></thead><tbody><tr><td>1-2</td><td>a.good translation</td><td>Given the following source text (in L1): [source sentence], a good L2 translation is:</td><td>[target sentence]</td></tr><tr><td>3</td><td>version</td><td>If the original version says [source sentence] then the L2 version should say:</td><td>[target sentence]</td></tr><tr><td>4</td><td>gpt3</td><td>What is the L2 translation of the sentence: [source sentence]?</td><td>[target sentence]</td></tr><tr><td>5-6</td><td>xg1m</td><td>(L1:) [source sentence] = L2:</td><td>[target sentence]</td></tr><tr><td>7</td><td>translate_as</td><td>[source sentence] translates into L2 as:</td><td>[target sentence]</td></tr></tbody></table>

(of which 30% English), from various sources, although 38% of the data, known as the ROOTS corpus (Laurençon et al., 2022),<fnref n="6" /> is from Oscar web data (Ortiz Suárez et al., 2019). The model is openly released on HuggingFace in multiple sizes, ranging from 560M to 176B parameters.<fnref n="7" />

# 4 Evaluating BLOOM on the MT task

## 4.1 MT Datasets Used

We experiment with three datasets, chosen to test different aspects of BLOOM for MT: WMT (Bojar et al., 2014), Flores-101 (Goyal et al., 2022) and DiaBLa (Bawden et al., 2021). We use the WMT 2014 news test sets for English↔French and English↔Hindi, which we take as representative high- and lower-resource language pairs with respect to BLOOM's training data.<fnref n="8" /> These test sets are somewhat outdated (Garcia et al., 2023), but have been used repeatedly in past LLM evaluations and are included as standard benchmarks for comparison. Flores-101 is a multi-parallel dataset in 101 languages, translated from original English sentences. In fact, evaluations into English are bound to yield overly good results (e.g. (Toral et al., 2018)) and between other languages may mostly reflect their similarity with the original English. We use it to test and compare BLOOM's multilinguality, including for low-resource languages.<fnref n="9" /> DiaBLa is a bilingual test set of spontaneous written dialogues between English and French speakers, mediated by MT. We use this as a test of MT in an informal

domain and the impact of (cross-lingual) linguistic context in MT.

## 4.2 Experimental setup

We evaluate and compare BLOOM (and its variants) using the Language Model Evaluation Harness (Gao et al., 2021) in 0-shot and few-shot settings. For few-shot, *k* examples are prefixed to the prompt and separated with ### as shown in Example 1 (1-shot example is underlined).

(1) **Input:** French: je m'ennuie = English: I'm bored. ###
English: Is that your dog that's just wandered in over there? = French:
**Reference:** Est-ce que c'est votre chien qui vient de rentrer par là ?

Results are reported on the datasets' test splits. Few-shot examples are randomly taken from the data splits according to availability (train for WMT, dev for Flores-101 and test for DiaBLa). We evaluate using BLEU (Papineni et al., 2002) as implemented in SacreBLEU (Post, 2018), using as tokenisation 13a for WMT and DiaBLa and spm for Flores-101 as recommended (Costa-jussà et al., 2022).<fnref n="10" /> BLEU has many shortcomings but is good enough to provide quantitative comparisons for most systems used in this study. We additionally use COMET (Rei et al., 2020) for finer grained comparisons when the scores are closer.

### 4.2.1 Comparative models

In our cross-dataset comparison (Section 5.1), we compare BLOOM to other LLMs: (i) two task-fine-tuned models: T0¹¹ (Sanh et al., 2022), trained on English texts, and MT0-XXL<fnref n="12" /> (Muennighoff et al., 2022), the multilingual version, and (ii) OPT<fnref n="13" /> (Zhang et al., 2022), an English generative LM. We evaluate all models on the

<footnote n="6">⁶The ROOTS corpus can now be queried using the dedicated search tool https://hf.co/spaces/bigscience-data/roots-search.</footnote>

<footnote n="7">⁷https://hf.co/bigscience/bloom</footnote>

<footnote n="8">⁸English, French and Hindi make up 30%, 12.9% and 0.7% of the training data respectively (Laurençon et al., 2022).</footnote>

<footnote n="9">⁹An extended version, Flores-200, has been recently released (Costa-jussà et al., 2022), which is larger and covers approximately twice as many languages. As this new version was released late in our evaluation process and had only been used in one paper, we decided to stick to Flores-101.</footnote>

<footnote n="10">¹⁰BLEU+case:mixed+smooth.exp+{13a,spm}+version.2.2.1</footnote>

<footnote n="11">¹¹https://hf.co/bigscience/T0</footnote>

<footnote n="12">¹²https://hf.co/bigscience/mt0-xxl</footnote>

<footnote n="13">¹³https://hf.co/facebook/opt-66b</footnote>
same prompt `xglm-source+target`. To evaluate multiple language pairs with Flores-101, we compare (as a topline) to the supervised 615M-parameter MT model M2M-100 (Fan et al., 2021), using the scores computed by Goyal et al. (2022).

## 4.2.2 Prompts

We use several prompts, designed to illustrate different sources of variation: (i) the inclusion (or not) of the source language name, (ii) the relative order of source and target language names, (iii) the position of the source sentence (beginning or end of the prompt) and (iv) the prompt's verbosity. These prompts, available in PromptSource (Bach et al., 2022), are shown in Table 1. The first three are inspired by previous work:<fnref n="14" /> (Brown et al., 2020) for `gpt3`,<fnref n="15" /> (Lin et al., 2022) for `xglm` and (Wei et al., 2022) for `translate_as`, which also resembles Raffel et al. (2020)'s prompt (*Translate English to German: "[source text]": [target sentence]*), also used in (Wei et al., 2022; Garcia and Firat, 2022).

Considering the entries in Table 1, we can see that “prompting” in fact refers to two distinct aspects of the input: (i) the formulation of the task in natural language and (ii) the presentation of related examples (for few-shot setups) interleaved with language tags (perhaps more clearly referred to as *priming* by Pham et al. (2020)). As illustrated by the `xglm` prompt for example, the instruction part can reduced to one single word. As our results below suggest, the instruction mostly matters in 0-shot setups, but can almost be dispensed with in few-shot scenarios. The authors of (Brown et al., 2020) and (Hendy et al., 2023) also use a verbose, instruction-like prompt in their zero-shot setup, and a much more compact one for few shots experiments. Also note that InstructGPT's prompt combines both an instruction and language tags (Ouyang et al., 2022, p. 49).

# 5 Evaluation results

Our evaluation of `BLOOM` starts with a comparison across the three datasets and detection of major MT errors with a focus on WMT (Section 5.1) and then we present more in-depth analyses of particular

aspects: (i) using WMT, a comparative study of `BLOOM` model sizes (Section 5.2) and prompts (Section 5.3), (ii) using Flores-101 an evaluation of more language pairs and cross-lingual few-shot transfer (Section 5.4), and (iii) using DiaBLa, a study of the use of linguistic context (Section 5.5).

## 5.1 Comparison across datasets

<table><thead><tr><th></th><th colspan="5">0-shot</th><th colspan="5">1-shot</th></tr><tr><th></th><th>BLOOM</th><th>T0</th><th>mT0</th><th>OPT</th><th>BLOOM</th><th>T0</th><th>mT0</th><th>OPT</th><th>BLOOM</th><th>T0</th><th>mT0</th><th>OPT</th></tr></thead><tbody><tr><th colspan="14">WMT 2014</th></tr><tr><td>en→fr</td><td>14.9</td><td>1.2</td><td>29.3</td><td>12.9</td><td>27.8</td><td>1.4</td><td>25.2</td><td>21.9</td><td>en→fr</td><td>14.9</td><td>1.2</td><td>29.3</td><td>12.9</td></tr><tr><td>fr→en</td><td>15.5</td><td>25.8</td><td>32.9</td><td>15.5</td><td>34.6</td><td>21.0</td><td>30.0</td><td>24.6</td><td>fr→en</td><td>15.5</td><td>25.8</td><td>32.9</td><td>15.5</td></tr><tr><td>en→hi</td><td>6.8</td><td>0.2</td><td>11.2</td><td>0.1</td><td>13.6</td><td>0.1</td><td>9.5</td><td>0.1</td><td>en→hi</td><td>6.8</td><td>0.2</td><td>11.2</td><td>0.1</td></tr><tr><td>hi→en</td><td>12.1</td><td>0.0</td><td>26.1</td><td>0.4</td><td>25.0</td><td>0.0</td><td>20.1</td><td>0.6</td><td>hi→en</td><td>12.1</td><td>0.0</td><td>26.1</td><td>0.4</td></tr><tr><th colspan="14">DiaBLa</th></tr><tr><td>en→fr</td><td>0.9</td><td>0.5</td><td>28.4</td><td>0.5</td><td>5.7</td><td>0.6</td><td>21.0</td><td>15.5</td><td>en→fr</td><td>0.9</td><td>0.5</td><td>28.4</td><td>0.5</td></tr><tr><td>fr→en</td><td>0.8</td><td>25.5</td><td>35.0</td><td>0.8</td><td>12.1</td><td>20.6</td><td>26.9</td><td>12.1</td><td>fr→en</td><td>0.8</td><td>25.5</td><td>35.0</td><td>0.8</td></tr><tr><th colspan="14">Flores-101</th></tr><tr><td>en→fr</td><td>2.8</td><td>1.9</td><td>55.5</td><td>2.8</td><td>45.0</td><td>2.1</td><td>53.5</td><td>24.4</td><td>en→fr</td><td>2.8</td><td>1.9</td><td>55.5</td><td>2.8</td></tr><tr><td>fr→en</td><td>2.7</td><td>31.9</td><td>60.1</td><td>2.6</td><td>45.6</td><td>24.9</td><td>58.2</td><td>16.7</td><td>fr→en</td><td>2.7</td><td>31.9</td><td>60.1</td><td>2.6</td></tr><tr><td>en→hi</td><td>1.3</td><td>0.1</td><td>67.7</td><td>0.1</td><td>27.2</td><td>0.1</td><td>54.7</td><td>0.1</td><td>en→hi</td><td>1.3</td><td>0.1</td><td>67.7</td><td>0.1</td></tr><tr><td>hi→en</td><td>3.4</td><td>0.0</td><td>59.5</td><td>0.1</td><td>35.1</td><td>0.2</td><td>57.3</td><td>0.5</td><td>hi→en</td><td>3.4</td><td>0.0</td><td>59.5</td><td>0.1</td></tr></tbody></table>

<table><caption>(a) Original predictions</caption><thead><tr><th></th><th colspan="5">0-shot</th><th colspan="5">1-shot</th></tr><tr><th></th><th>BLOOM</th><th>T0</th><th>mT0</th><th>OPT</th><th>BLOOM</th><th>T0</th><th>mT0</th><th>OPT</th><th>BLOOM</th><th>T0</th><th>mT0</th><th>OPT</th></tr></thead><tbody><tr><th colspan="14">WMT 2014</th></tr><tr><td>en→fr</td><td>32.2</td><td>1.2</td><td>29.2</td><td>18.9</td><td>36.3</td><td>1.4</td><td>25.2</td><td>22.3</td><td>en→fr</td><td>32.2</td><td>1.2</td><td>29.2</td><td>18.9</td></tr><tr><td>fr→en</td><td>37.2</td><td>25.8</td><td>32.9</td><td>33.2</td><td>38.2</td><td>21.1</td><td>29.9</td><td>33.2</td><td>fr→en</td><td>37.2</td><td>25.8</td><td>32.9</td><td>33.2</td></tr><tr><td>en→hi</td><td>12.1</td><td>0.2</td><td>11.2</td><td>0.1</td><td>15.7</td><td>0.1</td><td>9.5</td><td>0.1</td><td>en→hi</td><td>12.1</td><td>0.2</td><td>11.2</td><td>0.1</td></tr><tr><td>hi→en</td><td>24.3</td><td>0.0</td><td>26.1</td><td>0.5</td><td>25.0</td><td>0.0</td><td>20.1</td><td>0.6</td><td>hi→en</td><td>24.3</td><td>0.0</td><td>26.1</td><td>0.5</td></tr><tr><th colspan="14">DiaBLa</th></tr><tr><td>en→fr</td><td>24.2</td><td>0.5</td><td>28.4</td><td>17.4</td><td>37.6</td><td>0.6</td><td>21.9</td><td>20.7</td><td>en→fr</td><td>24.2</td><td>0.5</td><td>28.4</td><td>17.4</td></tr><tr><td>fr→en</td><td>22.9</td><td>25.5</td><td>34.9</td><td>36.8</td><td>41.4</td><td>21.1</td><td>27.2</td><td>37.6</td><td>fr→en</td><td>22.9</td><td>25.5</td><td>34.9</td><td>36.8</td></tr><tr><th colspan="14">Flores-101</th></tr><tr><td>en→fr</td><td>26.9</td><td>1.9</td><td>55.3</td><td>21.4</td><td>49.3</td><td>2.1</td><td>53.4</td><td>28.4</td><td>en→fr</td><td>26.9</td><td>1.9</td><td>55.3</td><td>21.4</td></tr><tr><td>fr→en</td><td>40.3</td><td>31.9</td><td>60.0</td><td>39.4</td><td>47.2</td><td>25.2</td><td>58.2</td><td>39.8</td><td>fr→en</td><td>40.3</td><td>31.9</td><td>60.0</td><td>39.4</td></tr><tr><td>en→hi</td><td>7.7</td><td>0.1</td><td>67.7</td><td>0.1</td><td>29.5</td><td>0.1</td><td>54.7</td><td>0.1</td><td>en→hi</td><td>7.7</td><td>0.1</td><td>67.7</td><td>0.1</td></tr><tr><td>hi→en</td><td>30.2</td><td>0.0</td><td>59.5</td><td>0.2</td><td>35.1</td><td>0.2</td><td>57.3</td><td>0.5</td><td>hi→en</td><td>30.2</td><td>0.0</td><td>59.5</td><td>0.2</td></tr></tbody></table>

(b) Truncated predictions

<caption>Table 2: Cross-dataset comparison of BLEU scores (spBLEU for Flores-101) using the `xglm-source+target` prompt.</caption>

We first prompt `BLOOM` and the comparative models using the same prompt across datasets, restricting the directions tested to `en↔fr` and to `en↔hi`. We choose to systematically use the `xglm-source+target` prompt (Table 1), which corresponds to the following template:

$$ (2) \quad L1: \ [source sentence] = L2: $$

where `L1` and `L2` refer to the source and target languages respectively (e.g. English and French for `en→fr`) and `[source sentence]` is replaced by a given source sentence.

BLEU scores are in Table 2a for both 0-shot and 1-shot (results with COMET are given in Appendix A). There are issues for 0-shot MT for all

<footnote n="14">¹⁴This was not always straightforward due to incomplete documentation concerning (a) prompts tested, and (b) those actually used in each experiment (e.g. different ones for 0-shot and few-shot runs (Chowdhery et al., 2022)).</footnote>

<footnote n="15">¹⁵Used only it seems, for zero-shot learning in the form “Q: what is the L2 translation of sentence [source sentence]. A:”, where special tokens Q and A are the query and the answer texts (cf. Figure G.36, pp 59).</footnote>
directions, particularly when translating into non-English languages, (BLEU scores are systematically lower than into English). Even into English, the scores remain low with respect to state of the art (e.g. 2.7 BLEU for Flores-101 fr→en BLOOM vs. 60.1 for MT0-XXL).<fnref n="16" /> MT0-XXL's good results illustrate the benefits of prompt fine-tuning. Results are considerably improved in the 1-shot setting (except for T0 and MT0-XXL, both fine-tuned) and are comparable to other LLM MT results reported in the literature; they are markedly better for BLOOM than for OPT, which was mostly trained on English data. These general observations apply for WMT and Flores-101, while issues remain for DiaBLa.

**What are the main issues?** We identify two main issues (particularly for 0-shot): (i) over-generation and (ii) generating in the wrong language (an obvious problem for MT). Examples 3 and 4 illustrate some instances of these problems. The first example shows a very common problem, with repetition of the prompt in multiple languages until the maximum generation length is reached.

(3) **Input:** French: Il dit avoir conçu une sonnette de porte Wi-Fi. = English:
BLOOM: He says he conceived a Wi-Fi doorbell. = Spanish: Dice que concibió un timbre de puerta Wi-Fi. = German: Er sagt, er hat einen WLAN-Türglocken gegründet. = Italian: Dice di aver concepito un chiuso Wi-Fi [...]

(4) **Input:** French: Le tigre fait partie de la même famille (genre Panthera) que les lions, les léopards et les jaguars. Ces quatre félins sont les seuls capables de rugir. = English:
BLOOM: The tiger is a member of the same family (genus Panthera) as lions, leopards and jaguars. These four cats are the only ones capable of roaring. Le tigre est un grand félin qui vit en Asie. Il est le plus grand des félins. Il est aussi le plus gros carnivore terrestre. Il est très rapide et peut courir à plus de 60 km/h. [...]

**Separating MT quality from overgeneration**

Overgeneration as seen in Example 3 is a separate issue from BLOOM's capacity to translate into another language. We therefore devise a custom truncating method for this type of overgeneration such that only the first translation in a prediction is kept, i.e. anything after a newline or the regular expression pattern = .+?: is discarded.

Results after truncation (Table 2b) show that for all three datasets, 0-shot and 1-shot scores are significantly improved (e.g. 1-shot DiaBLa fr→en increases from 12.05 to 41.36 and 0-shot Flores-101

hi→en increases from 3.40 to 30.19). BLOOM is capable of performing good MT but has a problem knowing when to stop generating. We use the same truncation elsewhere too and indicate when we show results for original or truncated outputs.

<table><caption>Table 3: The number of outputs (after truncation) classified as being in the (correct) target language, the source language, or another language for 0-shot and 1-shot setups (for WMT).</caption><thead><tr><th rowspan="2"></th><th colspan="2">en→fr</th><th colspan="2">fr→en</th><th colspan="2">en→hi</th><th colspan="2">hi→en</th></tr><tr><th>0</th><th>1</th><th>0</th><th>1</th><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><th>Target</th><td>2814</td><td>2959</td><td>2954</td><td>2979</td><td>1998</td><td>2431</td><td>2469</td><td>2499</td></tr><tr><th>Source</th><td>181</td><td>32</td><td>47</td><td>22</td><td>476</td><td>48</td><td>29</td><td>2</td></tr><tr><th>Other</th><td>8</td><td>12</td><td>2</td><td>2</td><td>33</td><td>28</td><td>9</td><td>6</td></tr><tr><th>Total</th><td>3003</td><td>3003</td><td>3003</td><td>3003</td><td>2507</td><td>2507</td><td>2507</td><td>2507</td></tr></tbody></table>

**Detecting generation in the wrong language**

We automatically detect the language of predictions using fasttext langid<fnref n="17" /> (Joulin et al., 2017). Table 3 shows the number of translations identified as being in the correct target language, or alternatively in the source or another language for 0-shot and 1-shot setups after truncation.<fnref n="18" />,<fnref n="19" /> The number of sentences in the correct target language increases from 0- to 1-shot, particularly for the two non-English target languages. When translating into Hindi (0-shot), 1/5 (509) of predictions are not detected as Hindi; the 1-shot largely mitigates the issue (only 76 outputs are in the wrong language).

<figure><img src="image_4.png" /><figcaption>Figure 1: BLEU scores for WMT 2014 en↔fr and the xglm prompt, with an increasing number of few-shot examples.</figcaption></figure>

**Increasing the number of few-shot examples**

Both problems improve significantly in the 1-shot

<footnote n="16">¹⁶For comparison, (Bi et al., 2020) reports state-of-the art BLEU scores for supervised MT as 45.6 and 45.4 for WMT14 en→fr and fr→en respectively.</footnote>

<footnote n="17">¹⁷https://fasttext.cc/docs/en/language-identification.html, using the compressed version lid.176.ftz.</footnote>

<footnote n="18">¹⁸Raw tables can be found in Tables 12 and 13 in Appendix B.</footnote>

<footnote n="19">¹⁹These numbers are better than the initial ones reported in (BigScience et al., 2022), as we use a different prompt and truncation. See below for a detailed analysis per prompt.</footnote>
setup, a trend that continues as the number of few-shot examples increases, resulting in higher BLEU scores, as can be seen in Figure 1 for WMT en↔fr. However, we see diminishing returns, particularly visible between 2 to 5 examples, suggesting that gains beyond 5-shot would be more marginal.

## 5.2 BLOOM model size

Several versions of BLOOM exist, with differing numbers of parameters. To test how size impacts performance, we report average scores and ranges for WMT across the seven prompts. Table 4 shows that as the size decreases (from 176B to 560M parameters), the performance also decreases significantly. We see substantial gains for all models when moving from 0-shot to 1-shot, the smaller models (e.g. BLOOM-7b1, BLOOM-3b) slightly closing the gap with the largest one. As the ranges in Table 4 are computed across prompts, we see that different prompts yield markedly different BLEU scores in the 0-shot setup; for 1-shot, we still see variations of 6-8 BLEU points between the best and the worst prompt. Similar analyses performed with post-processing and also for English↔Hindi (Appendix C) confirm that (i) truncation improves scores for all model sizes and prompts and (ii) the choice of a bad prompt can result in catastrophic MT performance as compared to a good one.

<table><thead><tr><th>Model</th><th>en→fr</th><th>fr→en</th></tr></thead><tbody><tr><td>BLOOM</td><td>11.2</td><td>3.0-22.0</td><td>15.4</td><td>10.3-26.8</td></tr><tr><td>BLOOM-7b1</td><td>6.5</td><td>1.5-12.1</td><td>12.8</td><td>4.8-25.1</td></tr><tr><td>BLOOM-3b</td><td>3.6</td><td>1.2-9.6</td><td>10.6</td><td>2.8-19.3</td></tr><tr><td>BLOOM-1b1</td><td>1.7</td><td>0.5-3.9</td><td>7.1</td><td>0.7-11.4</td></tr><tr><td>BLOOM-560m</td><td>0.6</td><td>0.4-0.9</td><td>3.7</td><td>1.4-5.4</td></tr></tbody></table>

<table><caption>(b) 1-shot</caption><caption>(a) 0-shot</caption><thead><tr><th>Model</th><th>en→fr</th><th>fr→en</th></tr></thead><tbody><tr><td>BLOOM</td><td>32.6</td><td>27.8-36.4</td><td>34.9</td><td>33.1-36.6</td></tr><tr><td>BLOOM-7b1</td><td>25.9</td><td>20.8-29.9</td><td>29.1</td><td>25.4-32.5</td></tr><tr><td>BLOOM-3b</td><td>21.6</td><td>16.7-26.8</td><td>25.7</td><td>18.6-29.6</td></tr><tr><td>BLOOM-1b1</td><td>10.1</td><td>6.3-13.2</td><td>16.1</td><td>12.2-19.9</td></tr><tr><td>BLOOM-560m</td><td>3.6</td><td>2.2-4.4</td><td>8.6</td><td>5.8-12.1</td></tr></tbody></table>

<caption>Table 4: Average BLEU scores and ranges across the seven prompts for decreasing sizes of BLOOM (original outputs).</caption>

## 5.3 Per-prompt analysis

Looking at average WMT results computed with respect to prompt choice (using the prompts in Table 1) allows us to further investigate cross-prompt variability.

**Which prompt works best?** This variability is illustrated in Tables 5 and 6 report performance across prompts for en↔{fr,hi}, averaged over the five BLOOM models from Section 5.2.<fnref n="20" /> The corresponding tables for truncated outputs are in Appendix D. version and a good translation (source+target) get the highest average (and maximum) scores. Both prompts are more verbose (instruction-like), but the performance gap in the 1-shot setting between these prompts and the simpler, ‘priming-style’ prompts (e.g. xglm) narrows. The worst results are seen for gpt3. With this prompt, translating into French after a text that only contains English seems particularly difficult: half of the 0-shot translations for gpt3 are classified as non-French by langid (most of them are English). When translating into Hindi, only 10 outputs are detected as being in Hindi.

**Does it help to specify the source language in the prompt?** We compare the two versions (-target and -source+target) of a good translation and xglm. Results in Tables 5 and 6 are inconclusive. For these language directions and prompts, we see small differences for 1-shot, which may be due to variance between runs. For 0-shot, it clearly helps xglm to indicate the source language, but for the more verbose a good translation, it helps one direction and hurts the other. This question would need to be further explored to draw more solid conclusions, including with non-English prompts.

## 5.4 Evaluating more language directions

We further explore more language directions in the 1-shot setting using Flores-101. As in Section 5.1, we use the xglm-source+target prompt.<fnref n="21" />

### 5.4.1 Per-language results

To optimise computational resources, instead of running all language combinations, we concentrate on: (i) high-resource language pairs, (ii) high→mid-resource language pairs, (iii) low-resource language pairs and (iv) related languages (specifically Romance languages). Results are shown in Tables 7 and 8 for original outputs, given that overgeneration is less problematic for 1-shot.

<footnote n="20">²⁰For a given prompt, the range mainly reflects the performance of the different sizes of BLOOM model.</footnote>

<footnote n="21">²¹It behaved well on average in the previous experiments and is one of the least verbose, making it more suitable in a multilingual setting.</footnote>
<table><caption>Table 5: Average, min and max BLEU scores by prompt for en↔fr (original outputs). Best average result per setting in bold.</caption><thead><tr><th rowspan="2">Prompt / Few-shot #</th><th colspan="2">en→fr</th><th colspan="2">fr→en</th></tr><tr><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>a.good translation-source+target</td><td>6.7 0.6-15.4</td><td>18.7 4.1-36.4</td><td>11.0 5.4-14.2</td><td>25.8 11.6-36.6</td></tr><tr><td>a.good translation-target</td><td>3.1 0.4-10.1</td><td>20.3 3.2-35.5</td><td>12.1 5.1-16.8</td><td><b>25.9</b> 12.1-36.2</td></tr><tr><td>gpt3-target</td><td>2.5 0.5-7.9</td><td>16.6 2.2-32.5</td><td>4.5 0.7-12.7</td><td>19.3 5.8-33.1</td></tr><tr><td>translate_as-target</td><td>3.3 0.4-5.0</td><td>17.1 3.2-32.7</td><td>6.9 2.1-11.3</td><td>21.6 7.6-35.1</td></tr><tr><td>version-target</td><td>7.5 0.6-22.0</td><td><b>21.4</b> 4.3-34.2</td><td><b>17.1</b> 3.9-26.8</td><td>24.9 7.8-35.4</td></tr><tr><td>xglm-source+target</td><td><b>8.3</b> 0.9-14.9</td><td>17.5 3.3-27.8</td><td>11.8 5.0-15.5</td><td>22.1 7.8-34.6</td></tr><tr><td>xglm-target</td><td>1.6 0.7-3.0</td><td>16.7 4.4-29.0</td><td>6.2 2.6-10.3</td><td>20.7 7.5-33.3</td></tr></tbody></table>

<table><caption>Table 6: Average, min and max BLEU scores per prompt for en↔hi (original outputs). Best average result per setting in bold.</caption><thead><tr><th rowspan="2">Prompt / Few-shot #</th><th colspan="2">en→hi</th><th colspan="2">hi→en</th></tr><tr><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>a.good translation-source+target</td><td>0.7 0.1-1.9</td><td><b>5.8</b> 0.3-14.5</td><td>4.8 0.9-10.2</td><td>13.1 2.8-24.6</td></tr><tr><td>a.good translation-target</td><td>0.2 0.1-0.8</td><td><b>5.5</b> 0.3-14.1</td><td>6.3 1.1-13.0</td><td>13.2 2.8-24.8</td></tr><tr><td>gpt3-target</td><td>0.1 0.0-0.3</td><td>1.4 0.0-6.5</td><td>0.2 0.0-0.7</td><td>2.2 0.0-10.0</td></tr><tr><td>version-target</td><td>0.7 0.1-2.0</td><td><b>5.6</b> 0.2-14.0</td><td><b>6.8</b> 1.7-11.5</td><td><b>13.3</b> 2.4-25.8</td></tr><tr><td>xglm-source+target</td><td><b>2.1</b> 0.1-6.8</td><td><b>6.9</b> 0.3-13.6</td><td>4.4 0.6-12.1</td><td>11.9 1.7-25.0</td></tr><tr><td>xglm-target</td><td>0.2 0.0-0.6</td><td>5.1 0.1-14.6</td><td>1.6 0.2-4.1</td><td>6.6 0.5-13.2</td></tr></tbody></table>

**High-resource and high→mid-resource** The results for high-resource and high→mid-resource language directions are generally good, surpassing M2M scores for high-resource, except for es→fr.<fnref n="22" /> This suggests that BLOOM has good multilingual capacity, even across scripts (between (extended) Latin, Chinese, Arabic and Devanagari scripts).

**Low-resource** For low-resource languages, the results are more variable; some language directions see better results than M2M, notably most into-English directions, but others are less good (e.g. into Hindi and Swahili). Results for the lowest-resourced languages tested (sw↔yo and en↔yo) are particularly disappointing because the scores indicate that the resulting translations are meaningless, even though Yoruba and Swahili are present (although under-represented) in BLOOM's training data (<50k tokens each).

**Romance languages** This contrasts with the results between Romance languages, where results are good across-the-board, including from and into Italian (it) and Galician (gl), which are not officially in the training data. Note that Galician shares many similarities with the other Romance languages, in

particular with Portuguese (pt). These contrasted results show the performance of an LLM not only depends on the amount of training data, but also largely on the similarity with seen languages. To be complete, these analyses should also take into account the possibility of mislabellings in the training data,<fnref n="23" /> which have been found to explain a great deal of cross-lingual abilities of LLMs (Blevins and Zettlemoyer, 2022).

### 5.4.2 Cross-lingual transfer

1-shot results are positive for many of the language directions tested (including low-resource), provided they are sufficiently represented in the ROOTS corpus. To better understand how cross-lingual BLOOM is and how the 1-shot mechanism functions, we vary the language direction of the few-shot examples, taking Bengali→English (bn→en) translation as our case study. Taking random 1-shot dev set examples,<fnref n="24" /> we compare the use of 1-shot examples from (i) the same direction (bn→en), (ii) the opposite direction (en→bn), (iii) a language direction whereby the source languages are related (hi→en), (iv) the same related direction but from a different dataset (the WMT dev set) (v) a high-resource direction into the same

<footnote n="22">²²French and Spanish, although related and comparably represented in ROOTS, have very different scores. Our preliminary analysis suggests that this is due to the Spanish references being less literal than the French and structurally more different from the original English. See Appendix E for some examples.</footnote>

<footnote n="23">²³In a personal communication, N. Muennighoff estimates that Italian accounts for ~0.33% of the ROOTS corpus, slightly below the proportion of Hindi texts (0.47%).</footnote>

<footnote n="24">²⁴The random seed is kept the same for all runs.</footnote>
<table><thead><tr><th>Src ↓</th><th>Trg →</th><th>ar</th><th>en</th><th>es</th><th>fr</th><th>zh</th></tr></thead><tbody><tr><td rowspan="2">ar</td><td>BLOOM</td><td>-</td><td>40.3</td><td>23.3</td><td>33.1</td><td>17.7</td></tr><tr><td>M2M</td><td>-</td><td>25.5</td><td>16.7</td><td>25.7</td><td>13.1</td></tr><tr><td rowspan="2">en</td><td>BLOOM</td><td>28.2</td><td>-</td><td>29.4</td><td>45.0</td><td>26.7</td></tr><tr><td>M2M</td><td>17.9</td><td>-</td><td>25.6</td><td>42.0</td><td>19.3</td></tr><tr><td rowspan="2">es</td><td>BLOOM</td><td>18.8</td><td>32.7</td><td>-</td><td>24.8</td><td>20.9</td></tr><tr><td>M2M</td><td>12.1</td><td>25.1</td><td>-</td><td>29.3</td><td>14.9</td></tr><tr><td rowspan="2">fr</td><td>BLOOM</td><td>23.4</td><td>45.6</td><td>27.5</td><td>-</td><td>23.2</td></tr><tr><td>M2M</td><td>15.4</td><td>37.2</td><td>25.6</td><td>-</td><td>17.6</td></tr><tr><td rowspan="2">zh</td><td>BLOOM</td><td>15.0</td><td>30.5</td><td>20.5</td><td>26.0</td><td>-</td></tr><tr><td>M2M</td><td>11.6</td><td>20.9</td><td>16.9</td><td>24.3</td><td>-</td></tr></tbody></table>

(a) High-resource language pairs.

<table><thead><tr><th>Src ↓</th><th>Trg →</th><th>en</th><th>fr</th><th>hi</th><th>id</th><th>vi</th></tr></thead><tbody><tr><td rowspan="2">en</td><td>BLOOM</td><td>-</td><td>45.0</td><td>27.2</td><td>39.0</td><td>28.5</td></tr><tr><td>M2M</td><td>-</td><td>42.0</td><td>28.1</td><td>37.3</td><td>35.1</td></tr><tr><td rowspan="2">fr</td><td>BLOOM</td><td>45.6</td><td>-</td><td>18.5</td><td>31.4</td><td>32.8</td></tr><tr><td>M2M</td><td>37.2</td><td>-</td><td>22.9</td><td>29.1</td><td>30.3</td></tr><tr><td rowspan="2">hi</td><td>BLOOM</td><td>35.1</td><td>27.6</td><td>-</td><td>-</td><td>-</td></tr><tr><td>M2M</td><td>27.9</td><td>25.9</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan="2">id</td><td>BLOOM</td><td>43.2</td><td>30.4</td><td>-</td><td>-</td><td>-</td></tr><tr><td>M2M</td><td>33.7</td><td>30.8</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan="2">vi</td><td>BLOOM</td><td>38.7</td><td>26.8</td><td>-</td><td>-</td><td>-</td></tr><tr><td>M2M</td><td>29.5</td><td>25.8</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

(b) High→mid-resource language pairs.

<table><caption>Table 7: 1-shot MT results (spBLEU) on the FLORES-101 devtest set (original outputs).</caption><thead><tr><th>Src↓</th><th>Trg→</th><th>en</th><th>bn</th><th>hi</th><th>sw</th><th>yo</th></tr></thead><tbody><tr><td rowspan="2">en</td><td>BLOOM</td><td>-</td><td>24.6</td><td>27.2</td><td>20.5</td><td>2.6</td></tr><tr><td>M2M</td><td>-</td><td>23.0</td><td>28.1</td><td>26.9</td><td>2.2</td></tr><tr><td rowspan="2">bn</td><td>BLOOM</td><td>29.9</td><td>-</td><td>16.3</td><td>-</td><td>-</td></tr><tr><td>M2M</td><td>22.9</td><td>-</td><td>21.8</td><td>-</td><td>-</td></tr><tr><td rowspan="2">hi</td><td>BLOOM</td><td>35.1</td><td>23.8</td><td>-</td><td>-</td><td>-</td></tr><tr><td>M2M</td><td>27.9</td><td>21.8</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan="2">sw</td><td>BLOOM</td><td>37.4</td><td>-</td><td>-</td><td>-</td><td>1.3</td></tr><tr><td>M2M</td><td>30.4</td><td>-</td><td>-</td><td>-</td><td>1.3</td></tr><tr><td rowspan="2">yo</td><td>BLOOM</td><td>4.1</td><td>-</td><td>-</td><td>0.9</td><td>-</td></tr><tr><td>M2M</td><td>4.2</td><td>-</td><td>-</td><td>1.9</td><td>-</td></tr></tbody></table>

(a) Low-resource languages

<table><thead><tr><th>Src↓</th><th>Trg→</th><th>ca</th><th>es</th><th>fr</th><th>gl</th><th>it</th><th>pt</th></tr></thead><tbody><tr><td rowspan="2">ca</td><td>BLOOM</td><td>-</td><td>28.9</td><td>33.8</td><td>19.2</td><td>19.8</td><td>33.0</td></tr><tr><td>M2M</td><td>-</td><td>25.2</td><td>35.1</td><td>33.4</td><td>25.5</td><td>35.2</td></tr><tr><td rowspan="2">es</td><td>BLOOM</td><td>31.2</td><td>-</td><td>24.8</td><td>23.3</td><td>16.5</td><td>29.1</td></tr><tr><td>M2M</td><td>23.1</td><td>-</td><td>29.3</td><td>27.5</td><td>23.9</td><td>28.1</td></tr><tr><td rowspan="2">fr</td><td>BLOOM</td><td>37.2</td><td>27.5</td><td>-</td><td>24.9</td><td>24.0</td><td>38.9</td></tr><tr><td>M2M</td><td>28.7</td><td>25.6</td><td>-</td><td>32.8</td><td>28.6</td><td>37.8</td></tr><tr><td rowspan="2">gl</td><td>BLOOM</td><td>37.5</td><td>27.1</td><td>33.8</td><td>-</td><td>18.3</td><td>32.2</td></tr><tr><td>M2M</td><td>30.1</td><td>27.6</td><td>37.1</td><td>-</td><td>26.9</td><td>34.8</td></tr><tr><td rowspan="2">it</td><td>BLOOM</td><td>31.0</td><td>25.4</td><td>31.4</td><td>20.2</td><td>-</td><td>29.2</td></tr><tr><td>M2M</td><td>25.2</td><td>29.2</td><td>34.4</td><td>29.2</td><td>-</td><td>31.5</td></tr><tr><td rowspan="2">pt</td><td>BLOOM</td><td>39.6</td><td>28.1</td><td>40.3</td><td>27.1</td><td>20.1</td><td>-</td></tr><tr><td>M2M</td><td>30.7</td><td>26.9</td><td>40.2</td><td>33.8</td><td>28.1</td><td>-</td></tr></tbody></table>

(b) Romance languages

<table><caption>Table 8: 1-shot MT results (spBLEU) on the Flores-101 devtest set (original outputs).</caption><thead><tr><th rowspan="2">1-shot example direction type</th><th colspan="2">Original</th><th colspan="2">Truncated</th></tr><tr><th>spBLEU</th><th>COMET</th><th>spBLEU</th><th>COMET</th></tr></thead><tbody><tr><td>Same</td><td>bn→en</td><td>0.444</td><td>29.9</td><td>0.444</td></tr><tr><td>Opposite</td><td>en→bn</td><td>0.313</td><td>29.4</td><td>0.414</td></tr><tr><td>Related src</td><td>hi→en</td><td>0.449</td><td>30.5</td><td>0.460</td></tr><tr><td>Related src (WMT)</td><td>hi→en</td><td>0.422</td><td>29.1</td><td>0.427</td></tr><tr><td>HR unrelated src</td><td>fr→en</td><td>0.315</td><td>29.7</td><td>0.396</td></tr><tr><td>HR unrelated src</td><td>fr→ar</td><td>-0.102</td><td>28.0</td><td>0.322</td></tr></tbody></table>

<caption>Table 9: 1-shot results for Flores bn→en when varying the language direction of 1-shot examples. HR=high-resource.</caption>

target language (fr→en) and (vi) a high-resource unrelated language direction (fr→ar).

The results (Table 9) show that cross-lingual transfer is possible, but using a different language direction can impact overgeneration and translation quality. The unrelated direction fr→ar gives the worst results, with most overgeneration (see the score difference between original and truncated), but also the worst quality after truncation, suggesting that language relatedness does play a role. Overgeneration is still a problem (although less so) when using the opposite direction (en→bn) or the same target language (fr→en). Using a related (higher-resource) source language (hi→en) reduces overgeneration and also gives the best MT results. However, better results are seen when using Flores-101 rather than WMT examples, suggesting that in-domain examples are best.

## 5.5 Use of Linguistic Context

<table><caption>Table 10: Comparison of 1-shot results (BLEU) for DiaBLa when using the previous/random sentence for the 1-shot example (using the xglm-source+target prompt). In bold are the best results for each language direction.</caption><thead><tr><th rowspan="2">1-shot example<br>Origin</th><th rowspan="2">Dir.</th><th rowspan="2">Trunc.</th><th colspan="2">en→fr</th><th colspan="2">fr→en</th></tr><tr><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th></tr></thead><tbody><tr><td rowspan="2">Rand.</td><td rowspan="2">rand.</td><td>×</td><td>5.7</td><td>0.342</td><td>12.1</td><td>0.614</td></tr><tr><td>✓</td><td>37.6</td><td>0.634</td><td>41.4</td><td>0.758</td></tr><tr><td rowspan="2">Prev.</td><td rowspan="2">rand.</td><td>×</td><td>6.1</td><td>0.328</td><td>12.3</td><td>0.617</td></tr><tr><td>✓</td><td>38.5</td><td>0.614</td><td>41.6</td><td>0.751</td></tr><tr><td rowspan="2">Prev.</td><td rowspan="2">same</td><td>×</td><td>19.3</td><td>0.597</td><td>20.7</td><td>0.719</td></tr><tr><td>✓</td><td><b>39.0</b></td><td><b>0.632</b></td><td><b>42.1</b></td><td><b>0.761</b></td></tr><tr><td rowspan="2">Prev.</td><td rowspan="2">opp.</td><td>×</td><td>3.6</td><td>0.064</td><td>8.6</td><td>0.518</td></tr><tr><td>✓</td><td>37.8</td><td>0.590</td><td>41.2</td><td>0.742</td></tr></tbody></table>

There has been a considerable amount of research on linguistic context in MT, e.g. to disambiguate lexically ambiguous texts or when additional information is necessary for the output to be well-formed (e.g. translating anaphoric pronouns into a language that requires agreement with a coreferent) (Hardmeier, 2012; Libovický and Helcl,
2017; Bawden et al., 2018; Voita et al., 2018; Lopes
et al., 2020; Nayak et al., 2022).

We test the usefulness of linguistic context
in DiaBLa in the 1-shot setting (again using
xglm-source+target) by changing the ori-
gin of 1-shot examples: (i) a random example vs.
(ii) the previous dialogue utterance. If linguistic
context is useful, we would expect there to be an
improvement for (ii). We also vary the language
direction of the 1-shot example. By default, given
that the dataset is bilingual, the direction of 1-shot
examples is en→fr or fr→en, independent of the
current example's direction. Given the results in
Section 5.4.2 and the poor 0-shot results in Table 2a,
it is important to account for this to provide a fair
comparison. We therefore compare each type of
context (random/previous) with (i) the same ran-
dom directions, and (ii-iii) the same (and opposite)
language directions as the current example. We
show results for original and truncated outputs.

Results are shown in Table 10. Truncation helps considerably; even for 1-shot, BLOOM struggles not to overgenerate and this is considerably reduced when the same rather than the opposite language direction is used for the 1-shot example. It is unclear whether using previous rather than random context helps: BLEU is higher (38.5 vs. 37.6), whereas COMET is lower (0.328 vs. 0.342). These differences could be the result of randomness in 1-shot example selection, and different results could be obtained with a different random seed. Despite these inconclusive results, it is clear that using previous context influences the translation, for better or worse. For evidence of this, see Table 19 in Appendix F, which provides three such examples: (i) an unlucky negative influence on the translation of an ambiguous word *glace* 'ice cream or mirror' from the previous context, resulting in the wrong sense being chosen, (ii) the use of a coreferent *instrument* 'instrument' from the previous sentence and (iii) the correct gender agreement of the pronoun *they* into French (*elles* 'they (fem.)' as opposed to *ils* 'they (masc.)') to correspond to the feminine coreferent *filles* 'girls'.

6 Conclusion

We have evaluated BLOOM's MT performance
across three datasets and multiple language pairs.
While there remain problems of overgeneration and
generating in the wrong language (particularly for
0-shot MT), MT quality is significantly improved

in few-shot settings, closer to state-of-the-art re-
sults. Low-resource MT remains challenging for
some language pairs, despite the languages being
in the training data, questioning what it means to be
a BLOOM language. However, we see evidence for
cross-lingual transfer for non-BLOOM languages
and when using few-shot examples from other lan-
guage pairs. Finally, although using linguistic con-
text does not give improvements with automatic
metrics, there is evidence that discursive phenom-
ena are taken into account.

Acknowledgements

This work was made possible with the collec-
tive efforts of the BigScience community, who
designed, developed and prepared the tools and
datasets used to train BLOOM. Special mention
to evaluation working group members and espe-
cially to Niklas Muenninghoff and Pawan Sasanka
Ammanamanchi for producing some of our results.

This work was granted access to the HPC
resources of Institut du développement et
des ressources en informatique scientifique
(IDRIS) du Centre national de la recherche
scientifique (CNRS) under the allocations
2021-AD011011717R1, AD011012254R2, 2021-
A0101012475 and 2022-AD010614012 made
by Grand équipement national de calcul intensif
(GENCI). R. Bawden's participation was partly
funded by her chair position in the PRAIRIE insti-
tute, funded by the French national agency ANR as
part of the "Investissements d'avenir" programme
under the reference ANR-19-P3IA-0001, and by
her Emergence project, DadaNMT, funded by
Sorbonne Université.

References

Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert
Webson, Colin Raffel, Nihal V. Nayak, Abheesht
Sharma, Taewoon Kim, M Saiful Bari, Thibault
Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli,
Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gun-
jan Chhablani, Han Wang, Jason Fries, Maged Al-
shaibani, Shanya Sharma, Urmish Thakker, Khalid
Almubarak, Xiangru Tang, Dragomir Radev, Mike
Tian-jian Jiang, and Alexander Rush. 2022. Prompt-
Source: An integrated development environment and
repository for natural language prompts. In *Proceed-
ings of the 60th Annual Meeting of the Association
for Computational Linguistics: System Demonstra-
tions*, pages 93–104, Dublin, Ireland. Association for
Computational Linguistics.
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Fı-
rat, Daan van Esch, Aditya Siddhant, Mengmeng Niu,
Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey,
Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan
Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun,
Pidong Wang, Alexander Gutkin, Apurva Shah, Yan-
ping Huang, Zhifeng Chen, Yonghui Wu, and Mac-
duff Hughes. 2022. *Building machine translation
systems for the next thousand languages.* CoRR,
abs/2205.03983.

Rachel Bawden, Eric Bilinski, Thomas Lavergne, and Sophie Rosset. 2021. DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation. *Language Resources and Evaluation*, 55(3):635–660.

Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*, pages 1304–1313, New Orleans, Louisiana. Association for Computational Linguistics.

Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2020. PALM: Pre-training an autoencoding&autoregressive language model for context-conditioned generation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 8681–8691, Online. Association for Computational Linguistics.

Workshop BigScience, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Vilanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponsferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Gíada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško,

Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Moham-mad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Pe-ter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, SOMaieh Nikpoor, Stanislav Silber-berg, Suhas Pai, Sydney Zink, Tiago Timponi Tor-rent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Ta-lat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajy-oti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Sai-ful Bari, Maged S. Al-shaibani, Matteo Manica, Ni-hal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Ur-mish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, San-chit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramanian, Aurélie Névéol, Charles Lover-ing, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bog-danov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Na-joung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Al-ice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdol-lahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Car-los Muñoz Ferrandis, Danish Contractor, David Lan-sky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fa-tima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman,
Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihajcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2022. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100.

Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In *Proceedings of the Ninth Workshop on Statistical Machine Translation*, pages 12–58, Baltimore, Maryland, USA. Association for Computational Linguistics.

Ondřej Bojar and Aleš Tamchyna. 2011. Improving translation model by monolingual data. In *Proceedings of the Sixth Workshop on Statistical Machine Translation*, pages 330–336, Edinburgh, Scotland. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Matěusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In *Advances in Neural Information Processing Systems*, volume 33, pages 1877–1901. Curran Associates, Inc.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pilai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311.

Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. CoRR, abs/2207.04672.

Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In *Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 1723–1732, Beijing, China. Association for Computational Linguistics.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021. *Beyond English-Centric multilingual machine translation. Journal of Machine Learning Research*, 22(107):1–48.

Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. *Multi-way, multilingual neural machine translation with a shared attention mechanism*. In *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 866–875, San Diego, California. Association for Computational Linguistics.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.

Xavier Garcia, Yamini Bansal, Colin Cherry, George F. Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. CoRR, abs/2302.01398.

Xavier Garcia and Orhan Firat. 2022. *Using natural language prompts for machine translation*. CoRR, abs/2202.11822.

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. *Transactions of the Association for Computational Linguistics*, 10:522–538.

Christian Hardmeier. 2012. Discourse in statistical machine translation: A survey and a case study. *Dis-cours - Revue de linguistique, psycholinguistique et informatique*, (11).

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hasan Awadalla. 2023. How good are GPT models at machine translation? a comprehensive evaluation. CoRR, abs/2302.09210.

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a Good Translator? A Preliminary Study. CoRR, abs/2301.08745.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. *Bag of tricks for efficient text classification*. In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, pages 427–431, Valencia, Spain. Association for Computational Linguistics.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Luccioni, and Yacine Jernite. 2022. *The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset*. In *Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track*.

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Jindřich Libovický and Jindřich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 196–202, Vancouver, Canada. Association for Computational Linguistics.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Nathan Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9).

António Lopes, M. Amin Farajian, Rachel Bawden, Michael Zhang, and André F. T. Martins. 2020. Document-level neural MT: A systematic comparison. In *Proceedings of the 22nd Annual Conference for Computational Linguistics: Volume 2, Short Papers*, pages 427–431, Valencia, Spain. Association for Computational Linguistics.
of the European Association for Machine Translation, pages 225–234, Lisboa, Portugal. European Association for Machine Translation.

Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In *Proceedings of the 4th International Conference on Learning Representations*, San Juan, Puerto Rico.

Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. *CoRR*, abs/1806.08730.

Yasmin Moslem, Rejwanul Haque, and Andy Way. 2023. Adaptive machine translation with large language models. *CoRR*, abs/2301.13294.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai-ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Al-banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning. *CoRR*, abs/2211.01786.

Prashanth Nayak, Rejwanul Haque, John D Kelleher, and Andy Way. 2022. Investigating Contextual Influence in Document-Level Translation. *Information*, 13(5):249.

Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In *Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7)*, pages 9–16, Mannheim. Leibniz-Institut für Deutsche Sprache.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. *CoRR*, abs/2203.02155.

Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In *Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics*, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.

Minh Quang Pham, Jitao Xu, Josep Crego, François Yvon, and Jean Senellart. 2020. Priming neural machine translation. In *Proceedings of the Fifth Conference on Machine Translation*, pages 516–527, Online. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. *Technical report*.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. *Journal of Machine Learning Research*, 21(140):1–67.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2685–2702, Online. Association for Computational Linguistics.

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multitask prompted training enables zero-shot task generalization. In *Proceedings of the International Conference on Learning Representations*.

Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learners. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 2339–2352, Online. Association for Computational Linguistics.

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
*Language Processing (EMNLP)*, pages 4222–4235, Online. Association for Computational Linguistics.

Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gökhan Tür, and Prem Natarajan. 2022. *Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model*. CoRR, abs/2208.01448.

Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. *Attaining the unattainable? reassessing claims of human parity in neural machine translation*. In *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 113–123, Brussels, Belgium. Association for Computational Linguistics.

David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George F. Foster. 2022. *Prompting PaLM for Translation: Assessing Strategies and Performance*. CoRR, abs/2211.09102.

Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. *Context-aware neural machine translation learns anaphora resolution*. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1264–1274, Melbourne, Australia. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. *Chain of thought prompting elicits reasoning in large language models*. In *Proceedings of the 36th Annual Conference on Neural Information Processing Systems*, New Orleans, Louisiana, USA.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2022. *GLM-130B: an open bilingual pre-trained model*. CoRR, abs/2210.02414.

Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. *Prompting large language model for machine translation: A case study*. CoRR, abs/2301.07069.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. *OPT: Open pre-trained transformer language models*. CoRR, abs/2205.01068.
## A COMET Results for Main Comparison

Table 11 shows the COMET scores for the cross-dataset and model comparison. The conclusions drawn for the Table 2 with BLEU scores hold here.

<table><thead><tr><th rowspan="2"></th><th rowspan="2">BLOOM</th><th colspan="3">0-shot</th><th colspan="3">1-shot</th></tr><tr><th>T0</th><th>mT0</th><th>OPT</th><th>T0</th><th>mT0</th><th>OPT</th></tr></thead><tbody><tr><td colspan="8"><strong>WMT 2014</strong></td></tr><tr><td>en→fr</td><td>-0.985</td><td>-0.700</td><td>0.453</td><td>-0.919</td><td>0.085</td><td>-1.035</td><td>-0.015</td><td>-0.165</td></tr><tr><td>fr→en</td><td>-0.675</td><td>0.337</td><td>0.567</td><td>-0.493</td><td>0.448</td><td>-0.087</td><td>0.250</td><td>0.039</td></tr><tr><td>en→hi</td><td>-0.482</td><td>-1.819</td><td>0.484</td><td>-1.525</td><td>0.288</td><td>-1.733</td><td>0.026</td><td>-1.460</td></tr><tr><td>hi→en</td><td>-0.387</td><td>-1.346</td><td>0.514</td><td>-1.200</td><td>0.378</td><td>-1.624</td><td>-0.019</td><td>-1.290</td></tr><tr><td colspan="8"><strong>DiaBLa</strong></td></tr><tr><td>en→fr</td><td>-1.573</td><td>-0.528</td><td>0.380</td><td>-1.762</td><td>0.342</td><td>-0.585</td><td>-0.018</td><td>0.123</td></tr><tr><td>fr→en</td><td>-1.581</td><td>0.228</td><td>0.534</td><td>-1.507</td><td>0.614</td><td>-0.032</td><td>0.365</td><td>0.389</td></tr><tr><td colspan="8"><strong>Flores-101</strong></td></tr><tr><td>en→fr</td><td>-1.469</td><td>-0.682</td><td>0.797</td><td>-1.438</td><td>0.602</td><td>-0.983</td><td>0.605</td><td>0.130</td></tr><tr><td>fr→en</td><td>-1.143</td><td>0.499</td><td>0.833</td><td>-1.008</td><td>0.687</td><td>-0.081</td><td>0.706</td><td>0.404</td></tr><tr><td>en→hi</td><td>-0.972</td><td>-1.848</td><td>1.025</td><td>-1.699</td><td>0.454</td><td>-1.795</td><td>0.718</td><td>-1.622</td></tr><tr><td>hi→en</td><td>-0.339</td><td>-1.391</td><td>0.797</td><td>-1.493</td><td>0.538</td><td>-1.264</td><td>0.667</td><td>-1.263</td></tr></tbody></table>

(a) Original predictions

<table><thead><tr><th rowspan="2"></th><th rowspan="2">BLOOM</th><th colspan="3">0-shot</th><th colspan="3">1-shot</th></tr><tr><th>T0</th><th>mT0</th><th>OPT</th><th>T0</th><th>mT0</th><th>OPT</th></tr></thead><tbody><tr><td colspan="8"><strong>WMT 2014</strong></td></tr><tr><td>en→fr</td><td>0.434</td><td>-0.700</td><td>0.452</td><td>0.034</td><td>0.424</td><td>-1.035</td><td>-0.017</td><td>-0.000</td></tr><tr><td>fr→en</td><td>0.604</td><td>0.336</td><td>0.566</td><td>0.534</td><td>0.532</td><td>-0.090</td><td>0.247</td><td>0.449</td></tr><tr><td>en→hi</td><td>0.053</td><td>-1.819</td><td>0.483</td><td>-1.491</td><td>0.448</td><td>-1.733</td><td>0.026</td><td>-1.460</td></tr><tr><td>hi→en</td><td>0.445</td><td>-1.346</td><td>0.511</td><td>-1.113</td><td>0.386</td><td>-1.624</td><td>-0.022</td><td>-1.274</td></tr><tr><td colspan="8"><strong>DiaBLa</strong></td></tr><tr><td>en→fr</td><td>0.433</td><td>-0.528</td><td>0.380</td><td>-0.002</td><td>0.634</td><td>-0.585</td><td>-0.023</td><td>0.192</td></tr><tr><td>fr→en</td><td>0.567</td><td>0.228</td><td>0.534</td><td>0.554</td><td>0.758</td><td>-0.039</td><td>0.356</td><td>0.639</td></tr><tr><td colspan="8"><strong>Flores-101</strong></td></tr><tr><td>en→fr</td><td>0.182</td><td>-0.683</td><td>0.793</td><td>0.027</td><td>0.622</td><td>-0.984</td><td>0.601</td><td>0.180</td></tr><tr><td>fr→en</td><td>0.697</td><td>0.499</td><td>0.831</td><td>0.689</td><td>0.690</td><td>-0.086</td><td>0.702</td><td>0.594</td></tr><tr><td>en→hi</td><td>-0.608</td><td>-1.849</td><td>1.025</td><td>-1.638</td><td>0.461</td><td>-1.795</td><td>0.718</td><td>-1.622</td></tr><tr><td>hi→en</td><td>0.509</td><td>-1.391</td><td>0.797</td><td>-1.166</td><td>0.538</td><td>-1.264</td><td>0.666</td><td>-1.251</td></tr></tbody></table>

(b) Truncated predictions

Table 11: Comparison of COMET scores across the three datasets using the xglm-source+target prompt.

## B Wrong language prediction and over-generation

As described in Section 5.1, one problem identified with BLOOM, particularly for 0-shot translation, is generating in the wrong language. Tables 12 and 13 give the full analysis including raw figures for language identification for WMT14 fr↔en and hi↔en translation directions. For 0-5 few-shot examples, we indicate the number of truncated outputs identified as being from each language (indicated by the rows), the correct language (the target) being indicated in green, and the source language (therefore incorrect) being indicated in red. We also provide the average length difference (Δ) between BLOOM's outputs and the reference translations (negative numbers indicate that the prediction is longer than the reference).

For 0-shot translation, a significant number of examples are classed as being in the source language for en→fr, and even more so for en→hi (almost

<table><thead><tr><th></th><th colspan="2">0-shot</th><th colspan="2">1-shot</th><th colspan="2">2-shot</th><th colspan="2">5-shot</th></tr><tr><th></th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th></tr></thead><tbody><tr><td>cs</td><td>1</td><td>408</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>de</td><td>1</td><td>3</td><td>2</td><td>146</td><td>2</td><td>-12.5</td><td>1</td><td>2</td></tr><tr><td>en</td><td>181</td><td>16</td><td>32</td><td>57</td><td>10</td><td>73.8</td><td>8</td><td>92.2</td></tr><tr><td>es</td><td>1</td><td>12</td><td>3</td><td>89.3</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>fr</td><td>2814</td><td>7.9</td><td>2959</td><td>2.1</td><td>2989</td><td>1.5</td><td>2992</td><td>1.6</td></tr><tr><td>ht</td><td>1</td><td>57</td><td>1</td><td>89</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>it</td><td>2</td><td>4.5</td><td>3</td><td>13.3</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>nl</td><td>1</td><td>131</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>pt</td><td>1</td><td>146</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ms</td><td>-</td><td>-</td><td>1</td><td>28</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ru</td><td>-</td><td>-</td><td>1</td><td>16</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>zh</td><td>-</td><td>-</td><td>1</td><td>10</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ca</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>198</td><td>1</td><td>18</td></tr><tr><td>uk</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>3</td><td>1</td><td>3</td></tr></tbody></table>

(a) en→fr

<table><thead><tr><th></th><th colspan="2">0-shot</th><th colspan="2">1-shot</th><th colspan="2">2-shot</th><th colspan="2">5-shot</th></tr><tr><th></th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th></tr></thead><tbody><tr><td>en</td><td>2954</td><td>1</td><td>2979</td><td>0.8</td><td>2988</td><td>1</td><td>2987</td><td>1.3</td></tr><tr><td>fr</td><td>47</td><td>-23.4</td><td>22</td><td>-1.4</td><td>13</td><td>1.3</td><td>13</td><td>-2.2</td></tr><tr><td>it</td><td>1</td><td>3</td><td>-</td><td>-</td><td>2</td><td>6</td><td>3</td><td>5.3</td></tr><tr><td>tr</td><td>1</td><td>-1</td><td>1</td><td>-1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>es</td><td>-</td><td>-</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

(b) fr→en

Table 12: Raw figures for language identification and length differences of outputs compared to the reference translation for WMT2014 en→fr using the xglm-source+target prompt. For 0-5 few-shot examples, N is the number of sentences identified as being in each language (the target language's row (correct) is indicated in green and the source language's row (one of the many incorrect options) in red) and Δ is the length difference in number of characters (N.B. it is negative when the prediction is longer than the reference).

one fifth of the outputs are in the wrong language). As we increase the number of few-shot examples used, both of these problems are significantly reduced, and almost disappear for all language pairs and directions with 5 examples.

## C Analysis per model

In this section, we complete the results of Section 5.2 with Tables 14 and 15, respectively for French↔English and Hindi↔English, reporting results without truncation. As expected, the systems are ranked according to their size. For French-English we see that decent performance can already be obtained with the second largest model BLOOM-7b1, using 1-shot. Using this model, or even a model half this size can provide good indication of the performance of prompts, and be reliably used as test beds. We obtain less satisfactory results with English↔Hindi, even with the large BLOOM; for this language pair, we even observe a large variation across prompts (looking at the range of scores)
in the 1-shot setting for all models.

## D Analysis per prompt

In this section, we replicate the analysis of Section 5.3 and report results per prompt with truncated outputs in Tables 16 and 17. The conclusions are overall consistent with what we report for non-truncated outputs in the main text. We note that after truncating the outputs, `xglm-source+target` yields very good results across the board, outperforming its closest contenders `a good translation-source+target` and `version-target` in almost all configurations. However, the choice of the prompt seems to matter more (a) in the zero-shot setting, (b) when translating out of English. Conversely our more stable results are for `fr-en`, 1-shot.

## E Translation divergences in Flores 101

A striking observation reported in the main text (Section 5.4.1) is the difference between French and Spanish for the Flores-101 experiments. This is unexpected, as both languages are well represented in the training data. Yet, when translating from and into English the difference in spBLEU score is huge; and there is a clear gap with the other Romance languages as well. A related question is the poor translation between French and Spanish, not much better than for French→Arabic. Looking at some sample outputs, this seems to be due to the peculiarities of the Spanish translations, which appear to be less literal than their French counterparts, but which yield equally good translations into English. This can be seen when we compare translations back into English for these languages (see a random subset in Table 18). The last example illustrates this very clearly: we see “34 percent” in both the original English and in the translation from French, while translation from Spanish starts with “one third”.

## F DiaBLa context-use examples

Table 19 contains examples where the preceding context in 1-shot examples has a positive, negative or neutral influence on the current prediction, showing that the choice of the 1-shot example is important and is taken into account by the model. Some details of these experiments are found in the accompanying Section 5.5 in the main text.

<table><thead><tr><th></th><th colspan="2">0-shot</th><th colspan="2">1-shot</th><th colspan="2">2-shot</th><th colspan="2">5-shot</th></tr><tr><th></th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th></tr></thead><tbody><tr><td>ceb</td><td>1</td><td>-150</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>en</td><td>476</td><td>10.5</td><td>48</td><td>12.4</td><td>71</td><td>13.9</td><td>26</td><td>18.8</td></tr><tr><td>eo</td><td>1</td><td>-134</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>fi</td><td>1</td><td>19</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>fr</td><td>2</td><td>94.5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>gom</td><td>2</td><td>6.5</td><td>1</td><td>4</td><td>-</td><td>-</td><td>1</td><td>0</td></tr><tr><td>hi</td><td>1998</td><td>9.3</td><td>2431</td><td>6</td><td>2403</td><td>5.5</td><td>2457</td><td>5.5</td></tr><tr><td>hsb</td><td>1</td><td>98</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ht</td><td>2</td><td>147</td><td>6</td><td>257.5</td><td>11</td><td>135.3</td><td>1</td><td>158</td></tr><tr><td>hu</td><td>1</td><td>71</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>lv</td><td>3</td><td>63.3</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>mr</td><td>5</td><td>64.4</td><td>11</td><td>14.6</td><td>17</td><td>11.7</td><td>19</td><td>6</td></tr><tr><td>ne</td><td>5</td><td>7.6</td><td>9</td><td>28.2</td><td>4</td><td>16.8</td><td>3</td><td>8.3</td></tr><tr><td>nl</td><td>2</td><td>-13.5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>pt</td><td>1</td><td>24</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>sa</td><td>1</td><td>-25</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>sw</td><td>1</td><td>12</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>tl</td><td>1</td><td>24</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>war</td><td>3</td><td>3</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>vec</td><td>-</td><td>-</td><td>1</td><td>-38</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>new</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>25</td><td>-</td><td>-</td></tr></tbody></table>

<table><thead><tr><th></th><th colspan="6">(a) en→hi</th></tr><tr><th></th><th colspan="2">0-shot</th><th colspan="2">1-shot</th><th colspan="2">2-shot</th><th colspan="2">5-shot</th></tr><tr><th></th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th><th>N</th><th>Δ</th></tr></thead><tbody><tr><td>en</td><td>2469</td><td>4</td><td>2499</td><td>5.1</td><td>2503</td><td>3.8</td><td>2498</td><td>3</td></tr><tr><td>fr</td><td>1</td><td>151</td><td>1</td><td>-5</td><td>-</td><td>-</td><td>1</td><td>8</td></tr><tr><td>hi</td><td>29</td><td>3.3</td><td>2</td><td>0</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ht</td><td>6</td><td>199.8</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>it</td><td>1</td><td>139</td><td>-</td><td>-</td><td>1</td><td>-18</td><td>3</td><td>4.3</td></tr><tr><td>nl</td><td>1</td><td>9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2</td><td>-3</td></tr><tr><td>id</td><td>-</td><td>-</td><td>1</td><td>-6</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>nds</td><td>-</td><td>-</td><td>1</td><td>16</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>pl</td><td>-</td><td>-</td><td>1</td><td>-14</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>tr</td><td>-</td><td>-</td><td>1</td><td>-15</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>war</td><td>-</td><td>-</td><td>1</td><td>344</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>de</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>-15</td><td>1</td><td>188</td></tr><tr><td>es</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>2</td><td>-</td><td>-</td></tr><tr><td>la</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>17</td><td>-</td><td>-</td></tr><tr><td>fi</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>-1</td></tr><tr><td>pt</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>1</td></tr></tbody></table>

(b) hi→en

<caption>Table 13: Raw figures for language identification and length differences of outputs compared to the reference translation for WMT2014 en→hi using the `xglm-source+target` prompt. For 0-5 few-shot examples, N is the number of sentences identified as being in each language (the target language's row (correct) is indicated in green and the source language's row (one of the many incorrect options) in red) and Δ is the length difference in number of characters (N.B. it is negative when the prediction is longer than the reference).</caption>
<table><caption>Table 14: Average, min and max BLEU scores per model of increasing size, for WMT14 en↔fr (original outputs). Best average result per setting in bold.</caption><thead><tr><th rowspan="2">Model / Direction</th><th colspan="2">0-shot</th><th colspan="2">1-shot</th></tr><tr><th>en→fr</th><th>fr→en</th><th>en→fr</th><th>fr→en</th></tr></thead><tbody><tr><td>BLOOM</td><td><strong>11.2</strong> 3.0-22.0</td><td><strong>15.4</strong> 10.3-26.8</td><td><strong>32.6</strong> 27.8-36.4</td><td><strong>34.9</strong> 33.1-36.6</td></tr><tr><td>BLOOM-7b1</td><td>6.5 1.5-12.1</td><td>12.8 4.8-25.1</td><td>25.9 20.8-29.9</td><td>29.1 25.4-32.5</td></tr><tr><td>BLOOM-3b</td><td>3.6 1.2-9.6</td><td>10.6 2.8-19.3</td><td>21.6 16.7-26.8</td><td>25.7 18.6-29.6</td></tr><tr><td>BLOOM-1b1</td><td>1.7 0.5-3.9</td><td>7.1 0.7-11.4</td><td>10.1 6.3-13.2</td><td>16.1 12.2-19.9</td></tr><tr><td>BLOOM-560m</td><td>0.6 0.4-0.9</td><td>3.7 1.4-5.4</td><td>3.6 2.2-4.4</td><td>8.6 5.8-12.1</td></tr></tbody></table>

<table><caption>Table 15: Average, min and max BLEU scores per model of decreasing size, for WMT14 en↔hi (original outputs). Best average result per setting in bold.</caption><thead><tr><th rowspan="2">Model / Direction</th><th colspan="2">0-shot</th><th colspan="2">1-shot</th></tr><tr><th>en→hi</th><th>hi→en</th><th>en→hi</th><th>hi→en</th></tr></thead><tbody><tr><td>BLOOM</td><td><strong>2.1</strong> 0.3-6.8</td><td><strong>8.3</strong> 0.7-13.0</td><td><strong>12.9</strong> 6.5-14.6</td><td><strong>19.8</strong> 10.0-25.8</td></tr><tr><td>BLOOM-7b1</td><td>0.1 0.1-3.0</td><td>5.7 0.3-9.5</td><td>5.9 0.3-10.4</td><td>12.4 1.0-17.5</td></tr><tr><td>BLOOM-3b</td><td>0.2 0.0-0.5</td><td>3.6 0.0-7.0</td><td>4.9 0.2-7.2</td><td>8.9 0.1-13.5</td></tr><tr><td>BLOOM-1b1</td><td>0.1 0.0-0.1</td><td>1.5 0.0-4.5</td><td>1.4 0.1-3.1</td><td>4.6 0.00-8.2</td></tr><tr><td>BLOOM-560m</td><td>0.1 0.0-0.1</td><td>0.8 0.0-1.7</td><td>0.2 0.0-0.3</td><td>1.5 0.1-2.8</td></tr></tbody></table>

<table><caption>Table 16: Average, min and max BLEU scores per prompt for WMT14 en↔fr (truncated outputs). Best average result per setting in bold.</caption><thead><tr><th rowspan="2">Prompt / Few-shot #</th><th colspan="2">en→fr</th><th colspan="2">fr→en</th></tr><tr><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>a Good translation-source+target</td><td>8.5 0.7-17.0</td><td>19.1 4.32-37.12</td><td>16.4 7.5-22.2</td><td>26.0 12.0-37.0</td></tr><tr><td>a Good translation-target</td><td>4.6 0.6-13.9</td><td>20.9 3.4-36.8</td><td>21.7 6.6-35.2</td><td>26.31 12.5-36.9</td></tr><tr><td>gpt3-target</td><td>4.0 0.7-14.0</td><td>18.7 3.0-36.4</td><td>8.3 1.3-25.7</td><td>21.6 7.2-37.2</td></tr><tr><td>translate_as-target</td><td>6.4 0.6-10.1</td><td>18.1 3.5-33.1</td><td>11.5 2.3-20.4</td><td>22.9 8.2-35.7</td></tr><tr><td>version-target</td><td>9.7 0.7-30.3</td><td>21.9 4.4-36.7</td><td>22.2 4.7-35.2</td><td>25.3 8.0-37.2</td></tr><tr><td>xGLM-source+target</td><td><strong>17.2</strong> 1.33-32.2</td><td><strong>23.2</strong> 5.0-36.3</td><td><strong>25.6</strong> 8.3-37.2</td><td><strong>26.7</strong> 11.1-38.2</td></tr><tr><td>xGLM-target</td><td>2.5 1.1-4.6</td><td>20.1 6.8-33.1</td><td>11.0 4.5-17.6</td><td>23.1 10.4-36.4</td></tr></tbody></table>

<table><caption>Table 17: Average, min and max BLEU scores per prompt for WMT14 en↔hi (truncated outputs). Best average result per setting in bold.</caption><thead><tr><th rowspan="2">Prompt / Few-shot #</th><th colspan="2">en→hi</th><th colspan="2">hi→en</th></tr><tr><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>a Good translation-source+target</td><td>1.2 0.1-3.3</td><td>5.8 0.3-14.5</td><td>6.2 1.0-12.7</td><td>13.0 2.6-24.4</td></tr><tr><td>a Good translation-target</td><td>0.4 0.1-1.3</td><td>5.5 0.3-14.1</td><td>10.8 1.1-25.4</td><td>13.2 2.7-24.7</td></tr><tr><td>gpt3-target</td><td>0.0 0.0-0.1</td><td>1.6 0.0-7.6</td><td>0.0 0.0-0.0</td><td>2.5 0.0-11.4</td></tr><tr><td>version-target</td><td>1.0 0.1-3.0</td><td>5.5 0.2-13.9</td><td><strong>11.3</strong> 2.4-21.4</td><td><strong>13.5</strong> 2.7-25.7</td></tr><tr><td>xGLM-source+target</td><td><strong>3.9</strong> 0.1-12.1</td><td><strong>7.3</strong> 0.2-15.8</td><td>8.8 0.9-24.3</td><td>12.4 1.2-25.0</td></tr><tr><td>xGLM-target</td><td>0.3 0.0-1.0</td><td>5.1 0.0-14.5</td><td>2.1 0.3-5.8</td><td>6.5 0.1-13.0</td></tr></tbody></table>
<table><caption>Table 18: A random subset of Flores-101 examples translated using BLOOM into English from French and Spanish (N.B. English was the original language of the sentences). Each block of three sentences contains the original English and the automatic French→English and Spanish→English translations.</caption><tr><td>en</td><td>They are cooler than the surrounding surface in the day and warmer at night.</td></tr><tr><td>fr→en</td><td>"They are cooler than the surrounding surface during the day and warmer at night".</td></tr><tr><td>es→en</td><td>During the day, its temperature is lower than that of the surrounding surface, and at night, higher.</td></tr><tr><td>en</td><td>"This is not going to be goodbye. This is the closing of one chapter and the opening of a new one."</td></tr><tr><td>fr→en</td><td>"It's not goodbye. It's a page that is turning, and another that is opening."</td></tr><tr><td>es→en</td><td>"This will not be a farewell; it is just the end of one chapter and the beginning of another".</td></tr><tr><td>en</td><td>"We now have 4-month-old mice that are non-diabetic that used to be diabetic," he added.</td></tr><tr><td>fr→en</td><td>"We now have mice that are four months old and are not diabetic, whereas they were before", he added.</td></tr><tr><td>es→en</td><td>"Currently, we have mice that are four months old and used to be diabetic, but they are no longer diabetic", he added.</td></tr><tr><td>en</td><td>"We will endeavour to cut carbon dioxide emissions per unit of GDP by a notable margin by 2020 from the 2005 level," Hu said.</td></tr><tr><td>fr→en</td><td>"We will strive to significantly reduce carbon dioxide emissions per unit of GDP by 2020 compared to the 2005 level," said Mr. Hu.</td></tr><tr><td>es→en</td><td>Hu said, "We will work hard to reduce the level of carbon dioxide emitted per unit of GDP by 2020, so that the difference is significant compared to 2005."</td></tr><tr><td>en</td><td>Scientists say this animal's plumage was chestnut-brown on top with a pale or carotenoid-colored underside.</td></tr><tr><td>fr→en</td><td>Scientists say that the plumage of this animal was chestnut brown on top and pale or carotenoid on the underside.</td></tr><tr><td>es→en</td><td>According to the experts, this animal has a brown plumage on the upper part and a pale or carotenoid color on the lower part.</td></tr><tr><td>en</td><td>34 per cent of those in the poll share this view, wanting Queen Elizabeth II to be Australia's last monarch.</td></tr><tr><td>fr→en</td><td>34 % of the people surveyed share this view, and want Queen Elizabeth II to be the last monarch to rule Australia.</td></tr><tr><td>es→en</td><td>One third of the respondents share this view and want the last queen to be Queen Elizabeth II.</td></tr></table>

<table><caption>Table 19: Ambiguous DiaBLa examples with different 1-shot contexts. Words that are relevant to the ambiguity are underlined, and incorrect translations are marked with an asterisk.</caption><thead><tr><th>1-shot origin</th><th>Context</th><th>Reference</th><th>Prediction</th></tr></thead><tbody><tr><td>Prev. same</td><td>French: Pensez vous en permanence à la glace qui se mange ? = English: Do you constantly think about the <u>ice-cream</u> that's eaten? ### French: ou bien à une <u>glace</u> pour se regarder ? = English:</td><td></td><td>*or to <u>an ice-cream</u> to look at?</td></tr><tr><td>Prev. opposite</td><td>English: Do you constantly think about the <u>ice-cream</u> that's eaten? = French: Pensez vous en permanence à la <u>glace</u> qui se mange ? ### French: ou bien à une <u>glace</u> pour se regarder ? = English:</td><td>Or a <u>mirror</u> to look into?</td><td>or to a <u>mirror</u> to look at?</td></tr><tr><td>Random</td><td>French: N'empêche, on vit une époque folle, folle! = English: Still, what a crazy, crazy time we're living in! ### French: ou bien à une <u>glace</u> pour se regarder ? = English:</td><td></td><td>or to a <u>mirror</u> to look at yourself?</td></tr><tr><td>Prev. same</td><td>English: What kind of <u>instrument</u> were you thinking of? = French: Tu penses à quelle sorte d' <u>instrument</u> ? ### English: A wooden one I suppose... = French:</td><td></td><td><u>Un instrument en bois, je suppose...</u></td></tr><tr><td>Prev. opposite</td><td>French: Tu penses à quelle sorte d' <u>instrument</u> ? = English: What kind of <u>instrument</u> were you thinking of? ### English: A wooden one I suppose... = French:</td><td><u>Un instrument en bois, je suppose.</u></td><td><u>Un instrument en bois, je suppose...</u></td></tr><tr><td>Random</td><td>French: Ils vont vous changer les idées après votre dure journée! = English: They'll help you take your mind off things after your hard day! ### English: A wooden one I suppose... = French:</td><td></td><td><u>Un en bois, je suppose...</u></td></tr><tr><td>Prev. same</td><td>English: He showed me how it works, but if I get stuck the <u>girls</u> in here will always help me. = French: Il m'a montré comment cela marchait, mais si je n'y arrive pas, les <u>filles</u> ici m'aideront sans problème. ### English: They are very kind. = French:</td><td></td><td><u>Elles sont très gentilles.</u></td></tr><tr><td>Prev. opposite</td><td>French: Il m'a montré comment cela marchait, mais si je n'y arrive pas, les <u>filles</u> ici m'aideront sans problème. = English: He showed me how it works, but if I get stuck the <u>girls</u> in here will always help me. ### English: They are very kind. = French:</td><td><u>Elles sont très gentilles.</u></td><td><u>Elles sont très gentilles.</u></td></tr><tr><td>Random</td><td>English: I don't know about <u>loans</u>. = French: Je ne sais pas pour les <u>prêts</u>. ### English: They are very kind. = French:</td><td></td><td>*<u>Elles sont très gentils.</u></td></tr></tbody></table>