<figure><img src="image_1.png" /></figure>

HAL
open science

# Topic-guided Example Selection for Domain Adaptation in LLM-based Machine Translation

Seth Aycock, Rachel Bawden

► To cite this version:

Seth Aycock, Rachel Bawden. Topic-guided Example Selection for Domain Adaptation in LLM-based Machine Translation. 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, Association for Computational Linguistics, Mar 2024, St. Julians, Malta. hal-04502291

HAL Id: hal-04502291

https://hal.science/hal-04502291v1

Submitted on 13 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Topic-guided Example Selection for Domain Adaptation
in LLM-based Machine Translation

Seth Aycock¹,²*

Rachel Bawden³

¹Institute for Logic, Language and Computation, University of Amsterdam

²Language Technology Lab, University of Amsterdam

³Inria, Paris, France

s.aycock@uva.nl

rachel.bawden@inria.fr

**Abstract**

Current machine translation (MT) systems perform well in the domains on which they were trained, but adaptation to unseen domains remains a challenge. Rather than fine-tuning on domain data or modifying the architecture for training, an alternative approach exploits large language models (LLMs), which are performant across NLP tasks especially when presented with in-context examples. We focus on adapting a pre-trained LLM to a domain at inference through in-context example selection. For MT, examples are usually randomly selected from a development set. Some more recent methods though select using the more intuitive basis of test source similarity. We employ topic models to select examples based on abstract semantic relationships below the level of a domain. We test the relevance of these statistical models and use them to select informative examples even for out-of-domain inputs, experimenting on 7 diverse domains and 11 language pairs of differing resourcedness. Our method outperforms baselines on challenging multilingual out-of-domain tests, though it does not match performance with strong baselines for the in-language setting. We find that adding few-shot examples and related keywords consistently improves translation quality, that example diversity must be balanced with source similarity, and that our pipeline is overly restrictive for example selection when a targeted development set is available.¹

# 1 Introduction

Adaptation of neural Machine Translation (MT) models to unseen domains remains a difficult problem because it requires handling out-of-distribution data at inference (Koehn and Knowles, 2017). Large language models (LLMs) offer an alternative method to the standard approach of fine-tuning an

MT model or selected layers therein (Luong and Manning, 2015; Bapna and Firat, 2019). Openly available models such as Llama-2 (Touvron et al., 2023) and explicitly multilingual models including BLOOM (BigScience Workshop et al., 2023) and XGLM (Lin et al., 2022) perform well cross-lingually in classification and generation tasks, including many-to-many translation despite lacking explicit MT training.

However, regardless of the choice of LLM, some domains and vocabulary will remain under-exposed or unseen, especially for low-resource languages. Additionally, the optimal use of LLMs at inference to enhance translation quality remains underexplored. Domain adaptation of LLM-based translation is therefore an open and persistent challenge. Translation with LLMs requires prompting to elicit outputs in the desired language and domain, either via a zero-shot instruction or more effectively with in-context examples (Zhang et al., 2023a). In this work, we address the problem of domain adaptation at inference by exploring in-context example selection. Selecting lexically, semantically or grammatically relevant translation examples for prompting LLMs is arguably more important when translating out-of-domain texts, to help fill gaps in domain vocabulary or demonstrate different styles.

Many works select examples randomly from a development set (Brown et al., 2020; Chowdhery et al., 2022; Bawden and Yvon, 2023). However, other strategies have been developed. While some works show example diversity helps task performance (Zhang et al., 2022), intuitively we expect that examples showing translations of words in or related to the test source will improve output quality. Prior work has selected relevant examples based on *n*-gram overlap (Agrawal et al., 2023), feature matching (Kumar et al., 2023) or embedding similarity to the test source (Liu et al., 2022). Here we test a method that exploits more abstract semantic relationships that are also more

<footnote>*This work was primarily carried out while at Inria.
¹Our code, topic models, and data splits are available at www.github.com/Sethjsa/LLM-Dom-Ad.</footnote>
fine-grained than domain categories. For this we use a topic modelling pipeline (Grootendorst, 2022) that predicts a topic for a source sentence, and selects examples from this topic for translating in the given domain. The motivation is two-fold: first, we aim to test the continuing relevance of these simple models to complement LLMs for MT; and second, we aim to study the importance of semantic similarity for domain adaptation at a more abstract granularity than prior work, providing an alternative method for example selection.

In practice we test Llama-2-13B, a state-of-the-art LLM, on MT in varied domains including medical, legal, educational, religious, and entertainment texts. We test across several high and low-resource languages from and into English: French, German, Czech, Romanian, Finnish, Lithuanian, and Tamil. We compare two uses of topic models for domain adaptation: topic-guided few-shot example selection and adding topic keywords. We test these against random baselines, information retrieval and embedding similarity-based selection, as well as simply adding domain labels. Our standard method uses multilingual topic models to select examples or keywords from seen domains across all tested languages. We show that our topic-guided method is robust to unseen domains and outperforms strong baselines in this setting, but is too restrictive to achieve competitive results against baselines for simpler in-language tests, suggesting a trade-off between similarity and diversity of examples.

## 2 Related Work

Domain adaptation methods for MT can be categorised as either data- or model-centric (Saunders, 2022). Data-centric approaches include fine-tuning models on in-domain parallel data (Dakwale and Monz, 2017) or synthetic backtranslated in-domain data (Sennrich et al., 2016; Jin et al., 2020), which is effective but costly in multilingual settings; or fine-tuning with labels encoding domain-specific information (Kobus et al., 2017; Stergiadis et al., 2021), which restricts prediction to seen domains. Model-centric approaches may use specialised architectures (Park et al., 2022) or different training methods such as curriculum or meta-learning (Zhang et al., 2019; Sharaf et al., 2020). Alternatively, adapters (Bapna and Firat, 2019) may be inserted into pre-trained models, with past work using separate domain and language adapters, or hierarchical domain adapters (Cooper Stickland

et al., 2021a; Chronopoulou et al., 2022). Contrary to these approaches, we focus on domain adaptation of a pre-trained LLM at inference through in-context example selection, which requires no additional data manipulation or fine-tuning.

Recent work explores using pre-trained LLMs as a form of unsupervised transfer learning (Chronopoulou et al., 2020; Cooper Stickland et al., 2021b). Many LLMs are competent in multilingual translation despite lacking explicit MT training (Alves et al., 2023; Bawden and Yvon, 2023; Hendy et al., 2023; Peng et al., 2023), though LLMs often struggle in low-resource settings (Zhu et al., 2023). For zero-shot translation, prompt design is key, with prior work improving translation with instructions (Li et al., 2023), dictionary hints (Ghazvininejad et al., 2023), chained bilingual dictionary entries (Lu et al., 2023), or chain-of-thought prompting to predict keywords, topics, and relevant examples (He et al., 2023). In-context learning, i.e. providing few-shot task examples, is effective for LLM prompting (Brown et al., 2020), and various aspects of examples have been shown to impact translation quality: Vilar et al. (2023) find example quality outweighs domain provenance or source similarity, while Zhang et al. (2023a) show semantic similarity correlates with improved performance, and Zhang et al. (2022) show example diversity helps task performance more generally. Prior example selection methods include using *n*-gram-based BM25 retrieval plus a reranking model (Agrawal et al., 2023), a regression model with manually defined features to score retrieved prompts (Kumar et al., 2023), training a dense retrieval model (Rubin et al., 2022), or selecting based on proximity to the test source in a pre-trained LLM's embedding space (Liu et al., 2022). In this work we intend to achieve similar results using an alternative topic-guided selection method, permitting more abstract semantic relationships than *n*-gram overlap or embedding similarity.

Topic models are statistical tools that model latent semantic structure in texts (Blei et al., 2010), and while not state-of-the-art, these methods remain relevant for neural NLP. Prior work has integrated topic models into neural MT architectures to improve translation performance (Zhang et al., 2016; Wang et al., 2021), or fused external topic knowledge to improve domain robustness (Xezonaki et al., 2023). Aharoni and Goldberg (2020) study in-domain training data selection methods using unsupervised clustering methods based on pre-
<table><caption>Table 1: An example illustrating our different prompting methods: domain labels, topic keywords, and a 1-shot example for an English-Romanian example from EMEA, with predicted and target outputs for the Keywords-10 prompt. Information in square brackets is not included in the prompt.</caption><tr><td>[Label]</td><td>Domain: EU biomedical texts.</td></tr><tr><td>[Keywords-10]</td><td>Related keywords: stabilité, stabilumas, stability, lämpötilassa, temperatura, raumtemperatur, tempéra-<br/>ture, teplotě, temperatūrai, conservée.</td></tr><tr><td>[Fewshot (1)]</td><td>English: the lower operating value of ambient air temperature is minus 45 ° C; = Romanian: valoarea<br/>inferioară a temperaturii de functionare a aerului ambiant – minus 45 °C;</td></tr><tr><td>[Source]</td><td>English: Keep your Humalog Mix50 Pen in use at room temperature (below 30°C) for up to 28<br/>days. = Romanian:</td></tr><tr><td>[Prediction]</td><td>Stergeti penul Humalog Mix50 din uz la temperaturi de cameră (sub 30°C) pentru 28 de zile.</td></tr><tr><td>[Target]</td><td>Țineți Humalog Mix50 Pen în curs de utilizare la temperatura camerei (sub 30°C) timp de până la 28<br/>zile.</td></tr></table>

trained language model embeddings, while Groo-
tendorst (2022) introduces a neural topic modelling
pipeline that clusters SBERT embeddings. We
build on these works and train multilingual topic
models to select relevant in-context examples for
prompting at inference. Our work tests the contin-
uing relevance of topic models in the context of
example selection against information retrieval and
embedding similarity baselines. The intuition is
that topic models identify semantic relationships
below the level of a domain but more abstract than
embedding similarity or n-gram overlap. We ex-
pect this intermediate level of semantic abstraction
will aid domain example selection for NMT.

# 3 Domain Adaptation Approach using Topic Modelling

## 3.1 Defining a domain

We employ topic models as the core mechanism
of our domain adaptation approach. Traditionally
in MT, a domain is defined as being a different
source text, i.e. each corpus is taken as a differ-
ent domain (Koehn and Knowles, 2017). Other
definitions are more nuanced: Joshi et al. (2013)
consider domains as consisting of multiple meta-
data attributes; van der Wees et al. (2015) subdivide
domains into topic and genre characteristics; and
Aharoni and Goldberg (2020) take a data-driven
approach to defining domains, letting statistical
models elucidate fine-grained cross-corpus associa-
tions and sub-domains within corpora. Building on
the above, we suggest that domains can intuitively
be defined by sets of distinctive words, forming a
domain's vocabulary. We expect these words to
be somewhat infrequent and pose a greater chal-
lenge for MT systems, suggesting this vocabulary
should be prioritised for adaptation. Our definition,
in addition to the data-driven approach, motivates

using topic models for domain adaptation since
they represent the latent semantic sets in a corpus.

## 3.2 Integrating domain information

Topic models find salient semantic relations be-
tween words or phrases in a corpus, representing
these relations with a small number of abstract
topics. Although typically modelled via the proba-
bilistic latent Dirichlet allocation (Blei et al., 2003),
we employ a different method which uses neural
text representations as the basis for topics (Groo-
tendorst, 2022). In these models, sentences are
converted to contextual embeddings which are clus-
tered based on similarity, then topic representations
are extracted from these clusters of sentences using
TF-IDF. Concretely, these topics consist of a set
of associated vocabulary, and a set of representa-
tive sentences from the training corpus containing
this vocabulary. We train multilingual topic models
over data from seen domains in all languages on
test. Once trained, any given input sentence can be
embedded and assigned to the closest topic in the
model. We therefore have three sources of addi-
tional information for each source sentence which
we integrate into the translation prompt: a corpus-
based domain label (Label), a list of keywords
from the closest topic (Keywords), and a topic-
guided set of representative examples (Fewshot).
We illustrate our methods for integrating multilin-
gual domain information in Table 1, with further
examples in Appendix B.

**Domain labels** We add a descriptive domain label for a source sentence based on the corpus it comes from (i.e. following the standard definition of a domain as a corpus), referred to in results as Label. We avoid using the corpus name as these are not uniformly informative and instead use a short description (as shown in Table 3). We expect
this to slightly improve translation performance for
given domains by conditioning the model to adapt
to an expected style and topic.

**Related Keywords** For a given source sentence, we predict the closest topic from our model and use the 10 related keywords from that topic, referred to as Keywords-10. We hypothesise that adding keywords will marginally improve performance by both conditioning the model's generation context on the current domain by introducing distinctive domain vocabulary, and by acting as a stochastic proxy for a multilingual lexicon, often providing translations in other languages given the multilingual nature of the topic model.

**Fewshot Examples** Finally, we select the top *n* representative examples from a topic for a given source to use as in-context examples in the prompt, known in testing as Fewshot (*n*). We describe the variations of fewshot examples that are tested in Section 4. We expect topic-guided fewshot examples to result in larger performance improvements for these domains by showing semantically relevant vocabulary, grammatical sentence-level translations, and examples of the expected target domain style and output format. We also expect more examples to improve performance by giving further explicit translations. The topic model's ability to select semantically similar examples within domains, as opposed to random or *n*-gram matched examples, may allow the LLM to observe translations of domain-distinctive vocabulary, improving translation quality.

# 4 Experiments

**Data and preprocessing** We select several diverse high and low-resource languages: Czech (cs), German (de), English (en), French (fr), Finnish (fi), Lithuanian (lt), Romanian (ro), and Tamil (ta), both into and out of English. These languages vary from group 3 to 5 in Joshi et al.'s (2020) taxonomy of language resourcedness. We test on 7 domains with data in most languages: medical European Medicines Agency texts (EMEA), transcribed TED Talks (Reimers and Gurevych, 2020), localisation files for KDE4 software, educational video transcripts from QCRI (Abdelali et al., 2014) (QED), Quran translations (Tanzil), EU legal texts (JRC), and transcripts of TV and films from OpenSubtitles<fnref n="2" /> (Lison et al., 2018) (Subs). All data was

obtained from OPUS (Tiedemann, 2012), and we
release our data splits, topic models, and code to
aid reproducibility and future research.<fnref n="3" />

Our preprocessing involves removing newlines and sentences over 175 tokens with Moses scripts (Koehn et al., 2007);<fnref n="4" /> removing sentences with over 50% punctuation; correct language identification using FastText (Joulin et al., 2017);<fnref n="5" /> and sentence-level deduplication. For each domain-language pair, our development and test sets consist of 5000 and 500 sentences respectively (N.B. Tanzil Tamil-English has only 4800 dev set sentences). Development sets are used to train topic models and as sources for example selection.

**Models** We used the HuggingFace (Wolf et al., 2020) implementation of Llama-2-13B,<fnref n="6" /> with greedy decoding up to 256 tokens. This model is mainly English with substantial multilingual capabilities, and is a state-of-the-art open-source LLM. We note however that the training data is not published, so our experiments are potentially at risk of data contamination. Our results therefore can only be considered in the context of this specific model.

While Llama-2 is not as performant on translation tasks as significantly larger models such as GPT-3.5 (Hendy et al., 2023; Xu et al., 2023), we chose Llama-2 for our experiments because it outperforms similarly-sized, explicitly multilingual LLMs including XGLM and BLOOMZ models (Zhang et al., 2023b). Llama-2 is also more robust to translation prompt perturbations than BLOOM models (Chitale et al., 2024). Further, Llama-2's permissive licence and open-source weights are a significant benefit against API-only models, leading to substantial research interest such as Llama-2-based translation models including ALMA (Xu et al., 2023). Our translation research on Llama-2 is therefore robustly motivated.

We implement our topic modelling pipeline with BERTopic (Grootendorst, 2022).<fnref n="7" /> Our topic models are trained on parallel development sets; we focus on multilingual seen domain models, and also test in-language all-domain models. The multilingual setting is challenging and tests generalisation: without full domain coverage, methods must compensate, perhaps

<footnote n="2">²www.opensubtitles.org</footnote>

<footnote n="3">³www.github.com/Sethjsa/LLM-Dom-Ad</footnote>

<footnote n="4">⁴www.github.com/moses-smt/mosesdecoder</footnote>

<footnote n="5">⁵www.fasttext.cc/docs/en/
language-identification.html</footnote>

<footnote n="6">⁶www.huggingface.co/meta-llama/Llama-2-13b</footnote>

<footnote n="7">⁷www.github.com/MaartenGr/BERTopic</footnote>
<table><caption>Table 2: Prompt templates for our experiments. In each prompt, both the source and target language are specified to aid in the zero-shot setting. The bold **L1** and **L2** are replaced with full language names e.g. English or Lithuanian, and [source sentence] is replaced by a given L1 sentence. The Fewshot prompt includes *n* example pairs.</caption><thead><tr><th>Template</th><th>Prompt</th></tr></thead><tbody><tr><td>Base</td><td><b>L1:</b> [source sentence] = <b>L2:</b></td></tr><tr><td>Verbose</td><td>Given the following source text in <b>L1:</b> [source sentence], a good <b>L2</b> translation is:</td></tr><tr><td>Label</td><td>Domain: [domain description] \n <b>L1:</b> [source sentence] = <b>L2:</b></td></tr><tr><td>Keywords-10</td><td>Related keywords: [keyword list] \n <b>L1:</b> [source sentence] = <b>L2:</b></td></tr><tr><td>Fewshot (<i>n</i>)</td><td><b>L1:</b> [example source] = <b>L2:</b> [example target] \n <b>L1:</b> [source sentence] = <b>L2:</b></td></tr></tbody></table>

<table><caption>Table 3: Domain Label descriptions, providing similar information across domains. Bold domains are treated as *seen* in experiments.</caption><thead><tr><th>Domain</th><th>Description</th></tr></thead><tbody><tr><td><b>EMEA</b></td><td>EU biomedical texts</td></tr><tr><td><b>JRC</b></td><td>EU legislative texts</td></tr><tr><td><b>KDE4</b></td><td>Software localization files</td></tr><tr><td><b>OpenSubtitles</b></td><td>TV and movie subtitles</td></tr><tr><td>QED</td><td>Educational video transcripts</td></tr><tr><td>Tanzil</td><td>Religious Quran text</td></tr><tr><td>TED</td><td>Public speaking transcripts</td></tr></tbody></table>

by using related examples from non-target languages. We expect the abstract cross-lingual semantic relationships identified by topic models to show robustness across domains. All models use embeddings from the 100-language paraphrase-multilingual-MiniLM-L12-v2 model from SentenceTransformers (Reimers and Gurevych, 2019), with UMAP dimensionality reduction (McInnes et al., 2018) and HDBSCAN clustering (Malzer and Baum, 2020). Our standard topic model has 500 topics trained on multilingual seen domains. Further details and hyperparameters are available in Appendix C.

**Prompt design** Our baseline experiments use the XGLM translation prompt (Lin et al., 2022), denoted as Base, and we also test a more verbose prompt, the two best performing MT prompts from Bawden and Yvon (2023) for BLOOM. Table 2 shows the format for our baselines and Label (see Table 3), Keywords-10, and Fewshot settings. Fewshot examples are selected from development sets of *seen* domains in all languages, unless otherwise specified; we expect this setting to be conducive to cross-lingual transfer.

**Baselines** Our Base setup uses a zero-shot XGLM-style prompt format. We implement two simple example selection techniques: BM25, an unsupervised information retrieval (Retrieval) tech-

nique based on *n*-gram matching; and sentence-level embedding similarity (Similarity), finding the closest sentences by cosine distance using the same SentenceTransformers model. For each baseline, we select from seen-domain multilingual development sets, and we additionally test in-language data against in-language topic models. Further baselines and ablations are described in Section 5. Finally, to contextualise our experiments we include topline results for NLLB-200-1.3B (Costa-jussà et al., 2022), a specialised translation model.

**Evaluation** We measure COMET scores using the wmt22-comet-da model (Rei et al., 2022), and BLEU (Papineni et al., 2002) with SacreBLEU<fnref n="8" /> (Post, 2018), included for interpretability despite poorer correlation with human judgments (Mathur et al., 2020). We build on the lm-evaluation-harness<fnref n="9" /> (Gao et al., 2022) for evaluation.

**Postprocessing** During initial tests we found the model often repeated outputs or provided translations in other languages, as found by Bawden and Yvon (2023). We therefore used a regular expression to capture the first L2 translation, discarding output after (.+?:) or a newline. Translation quality for trimmed results dramatically improves, showing the extent of Llama-2's overgeneration issues, and all results presented are trimmed outputs. In Appendix D, we show changes in length and correct language output for raw and trimmed results, plus Base-raw COMET and BLEU scores in Appendix E, confirming that trimming helps disentangle translation quality from overgeneration.

<footnote n="8">⁸Signature: nrefs:1|case:mixed|eff:no|tok:13a| smooth:exp|version:2.3.1</footnote>

<footnote n="9">⁹www.github.com/EleutherAI/ lm-evaluation-harness</footnote>
<table><caption>Table 4: COMET results for main experiments including domain labels (Label), 10 topic-guided keywords (Keywords-10), topic-guided 3-shot (Fewshot), from multilingual seen domains (Seen), and for topline NLLB model. Prompts are zero-shot unless specified; best performing mean results in **bold**; and a reference for experiment names is found in Appendix A.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="4">NLLB-1.3B</td><td>en-de</td><td>82.5</td><td>86.5</td><td>78.9</td><td>79.5</td><td>80.6</td><td>76.9</td><td>84.3</td></tr><tr><td>en-ro</td><td>86.0</td><td>90.5</td><td>80.6</td><td>83.7</td><td>82.9</td><td>79.7</td><td>86.8</td></tr><tr><td>lt-en</td><td>82.4</td><td>86.4</td><td>75.7</td><td>79.8</td><td>80.4</td><td>-</td><td>83.5</td></tr><tr><td>mean</td><td>84.9</td><td>88.7</td><td>79.1</td><td>81.7</td><td>81.3</td><td>77.7</td><td>85.3</td></tr><tr><td rowspan="4">Base</td><td>en-de</td><td>71.8</td><td>74.0</td><td>72.1</td><td>75.0</td><td>74.4</td><td>68.5</td><td>78.2</td></tr><tr><td>en-ro</td><td>66.5</td><td>79.2</td><td>70.9</td><td>74.1</td><td>72.5</td><td>65.7</td><td>77.1</td></tr><tr><td>lt-en</td><td>62.4</td><td>62.5</td><td>55.6</td><td>54.3</td><td>56.7</td><td>-</td><td>58.6</td></tr><tr><td>mean</td><td>70.8</td><td>73.5</td><td>67.4</td><td>68.2</td><td>67.2</td><td>62.6</td><td>70.0</td></tr><tr><td rowspan="4">Label</td><td>en-de</td><td>76.2</td><td>80.6</td><td>73.8</td><td>75.9</td><td>77.1</td><td>70.4</td><td>79.8</td></tr><tr><td>en-ro</td><td>73.9</td><td>84.2</td><td>71.2</td><td>74.9</td><td>75.2</td><td>67.2</td><td>79.6</td></tr><tr><td>lt-en</td><td>65.5</td><td>65.0</td><td>63.4</td><td>54.2</td><td>58.1</td><td>-</td><td>59.7</td></tr><tr><td>mean</td><td>73.9</td><td>76.9</td><td>70.6</td><td>69.4</td><td>69.2</td><td><strong>64.1</strong></td><td>71.7</td></tr><tr><td rowspan="4">Keywords-10 (Seen)</td><td>en-de</td><td>77.6</td><td>81.2</td><td>75.8</td><td>75.9</td><td>77.4</td><td>70.1</td><td>80.0</td></tr><tr><td>en-ro</td><td>76.7</td><td>84.3</td><td>74.8</td><td>75.9</td><td>76.5</td><td>67.2</td><td>80.4</td></tr><tr><td>lt-en</td><td>67.4</td><td>67.1</td><td>63.7</td><td>56.9</td><td>59.4</td><td>-</td><td>60.4</td></tr><tr><td>mean</td><td>75.3</td><td>77.7</td><td>71.3</td><td>69.6</td><td><strong>69.7</strong></td><td>63.6</td><td>72.1</td></tr><tr><td rowspan="4">Fewshot (3, Seen)</td><td>en-de</td><td>79.1</td><td>82.5</td><td>77.6</td><td>75.6</td><td>77.2</td><td>69.8</td><td>80.7</td></tr><tr><td>en-ro</td><td>80.1</td><td>85.9</td><td>77.8</td><td>75.9</td><td>76.6</td><td>68.2</td><td>80.7</td></tr><tr><td>lt-en</td><td>70.2</td><td>70.7</td><td>66.2</td><td>58.2</td><td>58.3</td><td>-</td><td>60.1</td></tr><tr><td>mean</td><td><strong>77.0</strong></td><td><strong>79.4</strong></td><td><strong>74.5</strong></td><td><strong>70.3</strong></td><td>69.6</td><td>63.8</td><td><strong>72.5</strong></td></tr></tbody></table>

# 5 Results

**Main Experiments** We start by comparing our three approaches for integrating domain information into prompts, domain labels (Label), topic-keywords (Keywords-10), and topic-guided 3-shot examples (Fewshot), against our zero-shot prompt (Base) and the topline NLLB model. We also tested a verbose prompt to validate previous claims (Bawden and Yvon, 2023) and provide a point of reference, but since they are not central to our method, we present these results in Appendix E.

Table 4 shows COMET scores for a selection of representative high and low-resource language pairs plus mean results over all pairs.<fnref n="10" />

We first note that the baseline zero-shot Llama-2 model shows substantially reduced performance compared to the NLLB model, especially on the lower-resource languages of Romanian and Lithuanian, which is to be expected from a specialist MT model explicitly trained on these languages. NLLB is thus a useful topline for our experiments. However, fewshot prompting helps Llama-2 begin to approach the scores achieved by the NLLB model.

Experiments with a domain label in the translation

<table><caption>Table 5: COMET scores for Label prompts against Base zero-shot and Random Label prompts.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="4">Base</td><td>en-de</td><td>71.8</td><td>74.0</td><td>72.1</td><td>75.0</td><td>74.4</td><td>68.5</td><td>78.2</td></tr><tr><td>en-ro</td><td>66.5</td><td>79.2</td><td>70.9</td><td>74.1</td><td>72.5</td><td>65.7</td><td>77.1</td></tr><tr><td>lt-en</td><td>62.4</td><td>62.5</td><td>55.6</td><td>54.3</td><td>56.7</td><td>-</td><td>58.6</td></tr><tr><td>mean</td><td>70.8</td><td>73.5</td><td>67.4</td><td>68.2</td><td>67.2</td><td>62.6</td><td>70.0</td></tr><tr><td rowspan="4">Label</td><td>en-de</td><td>76.2</td><td>80.6</td><td>73.8</td><td>75.9</td><td>77.1</td><td>70.4</td><td>79.8</td></tr><tr><td>en-ro</td><td>73.9</td><td>84.2</td><td>71.2</td><td>74.9</td><td>75.2</td><td>67.2</td><td>79.6</td></tr><tr><td>lt-en</td><td>65.5</td><td>65.0</td><td>63.4</td><td>54.2</td><td>58.1</td><td>-</td><td>59.7</td></tr><tr><td>mean</td><td><strong>73.9</strong></td><td><strong>76.9</strong></td><td><strong>70.6</strong></td><td><strong>69.4</strong></td><td><strong>69.2</strong></td><td><strong>64.1</strong></td><td><strong>71.7</strong></td></tr><tr><td rowspan="4">Random Label</td><td>en-de</td><td>75.6</td><td>80.4</td><td>73.4</td><td>75.7</td><td>76.9</td><td>70.2</td><td>79.8</td></tr><tr><td>en-ro</td><td>72.1</td><td>82.9</td><td>71.6</td><td>74.4</td><td>75.5</td><td>66.5</td><td>80.1</td></tr><tr><td>lt-en</td><td>64.0</td><td>62.7</td><td>60.7</td><td>54.6</td><td>57.0</td><td>-</td><td>58.7</td></tr><tr><td>mean</td><td>73.3</td><td>76.4</td><td>70.0</td><td>69.1</td><td>68.9</td><td>63.4</td><td><strong>71.7</strong></td></tr></tbody></table>

tion prompt show increases in COMET scores of up to 3 points over the baseline model, though the effect is smaller for OpenSubtitles, perhaps due to its heterogeneity, and greater for Tanzil, for the opposite reason. This suggests the model is able to use this minimal domain information to condition the output style and improve translation quality in highly restrictive domains such as Tanzil.

Results for prompting with 10 related keywords show average improvements of 2-4 COMET over the baseline, and up to 1 point over domain label tests, except for the unseen Tanzil domain. This suggests that the topic model-predicted keywords are useful for the model, providing lexical information beyond a domain description, and acting as a proxy for a bilingual lexicon. We would expect a handmade bilingual lexicon to improve results further (Waldendorf et al., 2022) but we note that quality lexicons are rare and thus keywords from topic models are a useful approximation.

Topic-guided 3-shot examples provide the largest performance boost of up to 6 COMET points, outperforming keywords on all domains. Gains are smaller again for OpenSubtitles; and the unseen Tanzil and QED domain results are marginally outperformed by Label and Keywords results respectively, though performance remains competitive. This shows the difficulty of selecting relevant out-of-domain examples for domains with more distinctive vocabulary or styles. Overall these results support our hypothesis that while domain labels and keywords provide useful domain and lexical information, especially in restrictive domains, few-shot examples help the model to better mimic the task and produce the desired output format.

Our main results show substantial improvements in translation quality using our example selection

<footnote n="10">¹⁰For full COMET and BLEU results over all 11 language pairs, see Appendix E. We note BLEU scores follow patterns in COMET results.</footnote>
<table><caption>Table 6: COMET scores for topic-guided Keywords-10 and Keywords-30, random topic Keywords-10, and Random Keywords-10 from multilingual seen domains.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="4">Base</td><td>en-de</td><td>71.8</td><td>74.0</td><td>72.1</td><td>75.0</td><td>74.4</td><td>68.5</td><td>78.2</td></tr><tr><td>en-ro</td><td>66.5</td><td>79.2</td><td>70.9</td><td>74.1</td><td>72.5</td><td>65.7</td><td>77.1</td></tr><tr><td>lt-en</td><td>62.4</td><td>62.5</td><td>55.6</td><td>54.3</td><td>56.7</td><td>-</td><td>58.6</td></tr><tr><td>mean</td><td>70.8</td><td>73.5</td><td>67.4</td><td>68.2</td><td>67.2</td><td>62.6</td><td>70.0</td></tr><tr><td rowspan="4">Keywords-10<br>(Seen)</td><td>en-de</td><td>77.6</td><td>81.2</td><td>75.8</td><td>75.9</td><td>77.4</td><td>70.1</td><td>80.0</td></tr><tr><td>en-ro</td><td>76.7</td><td>84.3</td><td>74.8</td><td>75.9</td><td>76.5</td><td>67.2</td><td>80.4</td></tr><tr><td>lt-en</td><td>67.4</td><td>67.1</td><td>63.7</td><td>56.9</td><td>59.4</td><td>-</td><td>60.4</td></tr><tr><td>mean</td><td><strong>75.3</strong></td><td><strong>77.7</strong></td><td><strong>71.3</strong></td><td>69.6</td><td><strong>69.7</strong></td><td>63.6</td><td>72.1</td></tr><tr><td rowspan="4">Keywords-30<br>(Seen)</td><td>en-de</td><td>77.5</td><td>81.3</td><td>76.3</td><td>75.8</td><td>77.7</td><td>70.2</td><td>80.1</td></tr><tr><td>en-ro</td><td>76.0</td><td>84.4</td><td>72.4</td><td>75.4</td><td>76.7</td><td>66.8</td><td>80.2</td></tr><tr><td>lt-en</td><td>67.8</td><td>67.4</td><td>64.0</td><td>57.5</td><td>59.7</td><td>-</td><td>60.1</td></tr><tr><td>mean</td><td><strong>75.3</strong></td><td><strong>77.7</strong></td><td>71.1</td><td>69.5</td><td><strong>69.7</strong></td><td>63.4</td><td>72.1</td></tr><tr><td rowspan="4">Keywords-10<br>(Seen,<br>Random Topic)</td><td>en-de</td><td>76.5</td><td>81.3</td><td>75.4</td><td>76.0</td><td>77.3</td><td>70.2</td><td>80.1</td></tr><tr><td>en-ro</td><td>75.4</td><td>84.7</td><td>73.0</td><td>75.6</td><td>76.4</td><td>67.3</td><td>80.0</td></tr><tr><td>lt-en</td><td>64.4</td><td>64.8</td><td>61.8</td><td>54.9</td><td>57.6</td><td>-</td><td>59.2</td></tr><tr><td>mean</td><td>74.2</td><td>77.4</td><td>70.7</td><td>69.5</td><td>69.5</td><td>63.5</td><td>71.9</td></tr><tr><td rowspan="4">Random<br>Keywords-10<br>(Seen)</td><td>en-de</td><td>76.9</td><td>81.2</td><td>75.6</td><td>76.2</td><td>77.2</td><td>70.4</td><td>80.2</td></tr><tr><td>en-ro</td><td>75.4</td><td>84.3</td><td>73.7</td><td>75.9</td><td>76.4</td><td>67.4</td><td>80.1</td></tr><tr><td>lt-en</td><td>66.3</td><td>65.3</td><td>61.9</td><td>56.1</td><td>58.8</td><td>-</td><td>60.7</td></tr><tr><td>mean</td><td>74.4</td><td>77.4</td><td>71.0</td><td><strong>69.8</strong></td><td>69.6</td><td><strong>63.8</strong></td><td><strong>72.2</strong></td></tr></tbody></table>

method. We now ablate each method against various baselines to further understand the source of these improvements.

**Domain Labels** While adding domain labels improves translation quality, we now test with randomised labels from the set of 7 labels to understand the source of improvements. The results in Table 5 show that, while prompting with the true domain label leads to overall better quality outputs across languages and domains, the random domain label tests produce similar improvements over the baseline, trailing the true label results by approximately 0.5 COMET points for most domains. This suggests that the presence of any additional structured information conditions the model to focus on the translation task, whether or not that information is directly useful for the current sentence.

**Related Keywords** We test various ablations of the Keywords prompt in Table 6. The Keywords-30 prompt is constructed using 10 keywords each from the top 3 predicted topics. Here we see equivalent or marginally lower quality compared to the standard Keywords-10 setting (less than 1 COMET point difference), suggesting that most gains stem from the first few keywords. The random topic setting adds keywords from a randomly selected topic from seen domains, and results are consistently lower than topic keywords, with results from Lithuanian to English showing substantial degradation across domains (up to -3

<table><caption>Table 7: COMET scores for Fewshot examples predicted by the multilingual seen-domain topic model, from one random topic, and random examples from seen domains across languages.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="4">Base</td><td>en-de</td><td>71.8</td><td>74.0</td><td>72.1</td><td>75.0</td><td>74.4</td><td>68.5</td><td>78.2</td></tr><tr><td>en-ro</td><td>66.5</td><td>79.2</td><td>70.9</td><td>74.1</td><td>72.5</td><td>65.7</td><td>77.1</td></tr><tr><td>lt-en</td><td>62.4</td><td>62.5</td><td>55.6</td><td>54.3</td><td>56.7</td><td>-</td><td>58.6</td></tr><tr><td>mean</td><td>70.8</td><td>73.5</td><td>67.4</td><td>68.2</td><td>67.2</td><td>62.6</td><td>70.0</td></tr><tr><td rowspan="4">Fewshot<br>(3, Seen)</td><td>en-de</td><td>79.1</td><td>82.5</td><td>77.6</td><td>75.6</td><td>77.2</td><td>69.8</td><td>80.7</td></tr><tr><td>en-ro</td><td>80.1</td><td>85.9</td><td>77.8</td><td>75.9</td><td>76.6</td><td>68.2</td><td>80.7</td></tr><tr><td>lt-en</td><td>70.2</td><td>70.7</td><td>66.2</td><td>58.2</td><td>58.3</td><td>-</td><td>60.1</td></tr><tr><td>mean</td><td><strong>77.0</strong></td><td><strong>79.4</strong></td><td><strong>74.5</strong></td><td><strong>70.3</strong></td><td><strong>69.6</strong></td><td><strong>63.8</strong></td><td><strong>72.5</strong></td></tr><tr><td rowspan="4">Fewshot<br>(3, Seen,<br>Random Topic)</td><td>en-de</td><td>79.0</td><td>81.5</td><td>76.9</td><td>75.3</td><td>77.1</td><td>69.8</td><td>80.1</td></tr><tr><td>en-ro</td><td>79.2</td><td>85.0</td><td>76.0</td><td>75.5</td><td>76.9</td><td>67.6</td><td>80.8</td></tr><tr><td>lt-en</td><td>59.8</td><td>60.7</td><td>52.3</td><td>49.5</td><td>51.5</td><td>-</td><td>53.5</td></tr><tr><td>mean</td><td>75.1</td><td>77.1</td><td>70.9</td><td>68.4</td><td>68.5</td><td>63.5</td><td>71.2</td></tr><tr><td rowspan="4">Random<br>Fewshot<br>(3, Seen)</td><td>en-de</td><td>77.0</td><td>80.5</td><td>75.2</td><td>75.3</td><td>76.7</td><td>69.8</td><td>79.4</td></tr><tr><td>en-ro</td><td>78.1</td><td>84.9</td><td>75.7</td><td>76.2</td><td>76.9</td><td>68.1</td><td>80.4</td></tr><tr><td>lt-en</td><td>70.9</td><td>71.6</td><td>69.7</td><td>69.1</td><td>68.1</td><td>-</td><td>72.3</td></tr><tr><td>mean</td><td>73.9</td><td>75.6</td><td>71.5</td><td>69.7</td><td>68.7</td><td>61.1</td><td>71.3</td></tr></tbody></table>

COMET points). This indicates related keywords may provide more utility in low-resource settings. Finally, Random Keywords-10 selects individual words randomly from multilingual seen domains, i.e. the topic model's training set. This setting is competitive with and on some domains outperforms the topic keyword prompts; while topic keywords provide semantically relevant words, and random topic keywords provide irrelevant but semantically consistent keywords, this setting provides genuinely diverse keywords, which appears to help performance. This suggests there is a trade-off between semantic relevance (through topic modelling) and information diversity in the prompt. In sum, only marginal gains can be attributed to the topic-guided method, suggesting the choice of keywords has less of an effect than the presence of keywords themselves.

**Fewshot Examples** We test topic-guided fewshot examples against random baselines, all 3-shot: few-shot examples from one random topic (Fewshot (Random Topic)), and random fewshot examples from seen multilingual data (Random Fewshot). The results in Table 7 show that while Fewshot (Random Topic) and Random Fewshot improve on the Base setting, the best results by 1-4 COMET points are achieved by the topic-guided example selection. This suggests that although there are gains to be had from simply adding random examples, the semantic relevance of these examples can lead to further improvements in translation performance. We expect this is due to a combination of both in-
<table><caption>Table 8: COMET scores for Fewshot topic-guided examples, Retrieval selected examples, and embedding similarity selected examples (Similarity), all from seen domains across languages.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="3">Fewshot<br>(3, Seen)</td><td>en-de</td><td>79.1</td><td>82.5</td><td>77.6</td><td>75.6</td><td>77.2</td><td>69.8</td><td>80.7</td></tr><tr><td>en-ro</td><td>80.1</td><td>85.9</td><td>77.8</td><td>75.9</td><td>76.6</td><td>68.2</td><td>80.7</td></tr><tr><td>lt-en<br>mean</td><td>70.2<br><strong>77.0</strong></td><td>70.7<br><strong>79.4</strong></td><td>66.2<br><strong>74.5</strong></td><td>58.2<br><strong>70.3</strong></td><td>58.3<br><strong>69.6</strong></td><td>-</td><td>60.1<br><strong>63.8</strong></td><td><strong>72.5</strong></td></tr><tr><td rowspan="3">Retrieval<br>(3, Seen)</td><td>en-de</td><td>76.4</td><td>79.8</td><td>74.6</td><td>73.8</td><td>76.0</td><td>68.9</td><td>78.5</td></tr><tr><td>en-ro</td><td>76.7</td><td>83.2</td><td>74.4</td><td>74.5</td><td>74.8</td><td>66.3</td><td>79.3</td></tr><tr><td>lt-en<br>mean</td><td>70.8<br><strong>73.2</strong></td><td>71.7<br><strong>74.9</strong></td><td>69.5<br><strong>70.6</strong></td><td>69.6<br><strong>68.8</strong></td><td>68.1<br><strong>67.9</strong></td><td>-</td><td>72.3<br><strong>60.5</strong></td><td><strong>70.6</strong></td></tr><tr><td rowspan="3">Similarity<br>(3, Seen)</td><td>en-de</td><td>76.3</td><td>80.6</td><td>76.2</td><td>74.8</td><td>75.9</td><td>68.8</td><td>79.0</td></tr><tr><td>en-ro</td><td>77.5</td><td>84.7</td><td>76.4</td><td>75.2</td><td>75.9</td><td>67.3</td><td>80.4</td></tr><tr><td>lt-en<br>mean</td><td>70.9<br><strong>73.9</strong></td><td>71.6<br><strong>76.0</strong></td><td>69.7<br><strong>71.9</strong></td><td>69.4<br><strong>69.5</strong></td><td>68.1<br><strong>68.5</strong></td><td>-</td><td>72.4<br><strong>61.2</strong></td><td><strong>71.1</strong></td></tr></tbody></table>

creased embedding similarity and *n*-gram overlap. The topic model selects a topic probabilistically, and although there may be noise within the topic— for example, the representative sentence pairs are not always in the same language or the correct target language—the semantic cohesiveness of these sentences outweighs the noise present in randomly selected examples. We can therefore attribute a small amount of quality improvements to the proposed topic-guided method.

We also test our topic-guided selection method against strong baselines inspired by prior work: a Retrieval method using BM25, and an embedding Similarity approach. For both we select examples from multilingual seen domains to control the data available for selection, since our standard topic model was tested in this challenging set-up.

The results in Table 8 show competitive performance for Retrieval and Similarity baselines against each other. However, our topic-guided fewshot method achieves the best results across domains by up to 3 COMET points. We also see slightly larger improvements for tests on unseen domains (QED, Tanzil and TED). Our method is more robust to all three unseen domains since it relies on an intermediate level of semantic relations, more complex than *n*-gram overlap, more abstract than raw embedding similarity, and finer-grained than domain-level selection. The Retrieval baseline especially suffers in the unseen domains, underperforming or matching the baseline zero-shot setting for QED and Tanzil, we expect because with lower or zero vocabulary overlap, *n*-gram matching fails where embeddings can exploit contextual information.

<table><caption>Table 9: COMET scores for increasing Fewshot examples, from 1-shot to 5-shot, using our standard multilingual seen-domain 500 topic model.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="3">Fewshot<br>(1, Seen)</td><td>en-de</td><td>78.1</td><td>81.3</td><td>75.9</td><td>74.8</td><td>76.3</td><td>69.0</td><td>79.8</td></tr><tr><td>en-ro</td><td>78.5</td><td>84.6</td><td>75.5</td><td>74.7</td><td>75.5</td><td>67.6</td><td>80.1</td></tr><tr><td>lt-en<br>mean</td><td>64.0<br><strong>74.8</strong></td><td>68.3<br><strong>78.5</strong></td><td>62.4<br><strong>72.3</strong></td><td>55.8<br><strong>69.3</strong></td><td>57.3<br><strong>68.9</strong></td><td>-</td><td>58.1<br><strong>63.5</strong></td><td><strong>71.6</strong></td></tr><tr><td rowspan="3">Fewshot<br>(3, Seen)</td><td>en-de</td><td>79.1</td><td>82.5</td><td>77.6</td><td>75.6</td><td>77.2</td><td>69.8</td><td>80.7</td></tr><tr><td>en-ro</td><td>80.1</td><td>85.9</td><td>77.8</td><td>75.9</td><td>76.6</td><td>68.2</td><td>80.7</td></tr><tr><td>lt-en<br>mean</td><td>70.2<br><strong>77.0</strong></td><td>70.7<br><strong>79.4</strong></td><td>66.2<br><strong>74.5</strong></td><td>58.2<br><strong>70.3</strong></td><td>58.3<br><strong>69.6</strong></td><td>-</td><td>60.1<br><strong>63.8</strong></td><td><strong>72.5</strong></td></tr><tr><td rowspan="3">Fewshot<br>(5, Seen)</td><td>en-de</td><td>79.5</td><td>82.4</td><td>78.2</td><td>75.8</td><td>77.4</td><td>70.3</td><td>80.1</td></tr><tr><td>en-ro</td><td>80.4</td><td>85.8</td><td>78.3</td><td>76.1</td><td>77.1</td><td>68.9</td><td>81.2</td></tr><tr><td>lt-en<br>mean</td><td>70.9<br><strong>77.3</strong></td><td>71.2<br><strong>79.6</strong></td><td>67.9<br><strong>74.9</strong></td><td>58.1<br><strong>70.5</strong></td><td>59.0<br><strong>70.0</strong></td><td>-</td><td>60.3<br><strong>64.1</strong></td><td><strong>72.6</strong></td></tr></tbody></table>

**Number of Examples** We present topic-guided fewshot results in Table 9 for 1, 3, and 5-shot settings. The results show gains of circa 1 COMET point from 1-shot to 3-shot, and even smaller gains of approximately 0.3 COMET in overall performance from 3 to 5-shot. This suggests that 3 examples are sufficient to provide substantial translation improvements over a zero-shot baseline, with diminishing returns for adding extra examples, corroborating results for BLOOM (Bawden and Yvon, 2023). Lithuanian-English results show low-resource languages, especially those not in the model's training data, may benefit more from additional examples; here we see continued improvements from 1 to 3 to 5-shot.

**Going Further** We also test various topic model sizes. While most experiments use the multilingual seen-domain 500-topic model (trained on the devsets of 4 domains totalling 140,000 parallel sentences), we also experiment with 200- and 1000-topic models. The results in Table 10 are mixed; some domains exhibit improved performance with larger models, but the improvements for the 1000-topic model are small or negligible over the 500-topic model. This is unexpected; a larger topic model implies more semantic variety and thus a wider choice sentences to select for a given test source. However, we observed that in the Fewshot (3, Seen, 500 topic) setting across languages and domains, a 'general' catch-all topic is selected for 3.1% of tests, and the top 5 topics make up 16% of selected topics, when a uniform distribution would result in each topic having a 0.2% selection rate. Therefore the overselection of certain topics, and consequent reduction of sentences available for selection, is likely to reduce performance. We also
<table><caption>Table 10: COMET scores for Fewshot examples from 200, 500, and 1000-topic seen-domain multilingual models.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="3">Fewshot<br>(3, Seen,<br>200 topic)</td><td>en-de</td><td>79.4</td><td>81.9</td><td>76.3</td><td>75.7</td><td>76.6</td><td>69.9</td><td>79.8</td></tr><tr><td>en-ro</td><td>79.8</td><td>85.7</td><td>76.4</td><td>76.0</td><td>76.8</td><td>68.6</td><td>80.7</td></tr><tr><td>lt-en</td><td>71.0</td><td>65.7</td><td>64.6</td><td>54.9</td><td>56.5</td><td>-</td><td>56.9</td></tr><tr><td></td><td>mean</td><td>77.2</td><td>78.3</td><td>72.9</td><td>69.6</td><td>69.4</td><td>63.9</td><td>71.7</td></tr><tr><td rowspan="3">Fewshot<br>(3, Seen,<br>500 topic)</td><td>en-de</td><td>79.1</td><td>82.5</td><td>77.6</td><td>75.6</td><td>77.2</td><td>69.8</td><td>80.7</td></tr><tr><td>en-ro</td><td>80.1</td><td>85.9</td><td>77.8</td><td>75.9</td><td>76.6</td><td>68.2</td><td>80.7</td></tr><tr><td>lt-en</td><td>70.2</td><td>70.7</td><td>66.2</td><td>58.2</td><td>58.3</td><td>-</td><td>60.1</td></tr><tr><td></td><td>mean</td><td>77.0</td><td>79.4</td><td><strong>74.5</strong></td><td><strong>70.3</strong></td><td><strong>69.6</strong></td><td>63.8</td><td><strong>72.5</strong></td></tr><tr><td rowspan="3">Fewshot<br>(3, Seen,<br>1000 topic)</td><td>en-de</td><td>79.0</td><td>82.3</td><td>78.1</td><td>75.6</td><td>76.6</td><td>70.4</td><td>79.7</td></tr><tr><td>en-ro</td><td>79.8</td><td>85.8</td><td>77.0</td><td>75.9</td><td>76.6</td><td>68.4</td><td>80.1</td></tr><tr><td>lt-en</td><td>71.2</td><td>70.9</td><td>64.6</td><td>56.3</td><td>57.8</td><td>-</td><td>59.7</td></tr><tr><td></td><td>mean</td><td><strong>77.4</strong></td><td><strong>79.5</strong></td><td>74.1</td><td>69.9</td><td><strong>69.6</strong></td><td><strong>63.9</strong></td><td>72.0</td></tr></tbody></table>

note that the homogeneity of examples within topics is likely to degrade performance since example diversity helps for other tasks (Zhang et al., 2022). Therefore issues remain with the restrictiveness of this method, which we leave open to future work.

**In-language example selection** Finally, we test language-specific 500-topic models to test whether the above results hold in a more restrictive scenario assuming the availability of in-language datasets in all domains, which is not always possible. Note here there are no unseen domains, though we include the multilingual seen-domain model for comparison. The results in Table 11 show improvements of 1-2 COMET for the all-domain language-specific models against the seen-domain multilingual model. We also provide language-specific Retrieval and Similarity results, which show even greater improvements compared to in-language topic-guided fewshot examples. Both baselines outperform the in-language topic model by 2 to 5 COMET, approaching the topline NLLB results in Table 4. This suggests that in the more restrictive scenario where we have full domain coverage in the target language pair, Retrieval and Similarity methods are very strong baselines because there is a greater probability of similar vocabulary, semantics, and syntax, while the topic model’s noise and highly homogeneous examples may hinder performance. However, in the more challenging scenario of the main results with unseen multilingual domains, our topic-guided method is more robust to domain shift.

## 6 Conclusion

We investigate the use of topic models for translation prompt construction and in-context example

<table><caption>Table 11: COMET scores for language-specific fewshot settings against our standard multilingual seen-domain Fewshot setup.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="3">Fewshot<br>(3, Seen)</td><td>en-de</td><td>79.1</td><td>82.5</td><td>77.6</td><td>75.6</td><td>77.2</td><td>69.8</td><td>80.7</td></tr><tr><td>en-ro</td><td>80.1</td><td>85.9</td><td>77.8</td><td>75.9</td><td>76.6</td><td>68.2</td><td>80.7</td></tr><tr><td>lt-en</td><td>70.2</td><td>70.7</td><td>66.2</td><td>58.2</td><td>58.3</td><td>-</td><td>60.1</td></tr><tr><td></td><td>mean</td><td>77.0</td><td>79.4</td><td>74.5</td><td>70.3</td><td>69.6</td><td>63.8</td><td>72.5</td></tr><tr><td rowspan="3">Fewshot<br>(3, Language)</td><td>en-de</td><td>79.7</td><td>82.7</td><td>78.7</td><td>76.3</td><td>77.6</td><td>71.4</td><td>80.9</td></tr><tr><td>en-ro</td><td>81.3</td><td>86.9</td><td>79.1</td><td>77.2</td><td>78.2</td><td>71.1</td><td>81.9</td></tr><tr><td>lt-en</td><td>71.8</td><td>71.6</td><td>67.4</td><td>61.0</td><td>62.9</td><td>-</td><td>65.6</td></tr><tr><td></td><td>mean</td><td>77.9</td><td>80.2</td><td>75.4</td><td>71.0</td><td>71.3</td><td>65.2</td><td>73.4</td></tr><tr><td rowspan="3">Retrieval<br>(3, Language)</td><td>en-de</td><td>81.5</td><td>84.0</td><td>79.8</td><td>75.8</td><td>76.9</td><td>73.4</td><td>80.3</td></tr><tr><td>en-ro</td><td>83.7</td><td>87.7</td><td>80.4</td><td>77.2</td><td>78.1</td><td>75.2</td><td>82.3</td></tr><tr><td>lt-en</td><td>77.0</td><td>75.3</td><td>74.9</td><td>61.8</td><td>65.8</td><td>-</td><td>67.9</td></tr><tr><td></td><td>mean</td><td><strong>81.2</strong></td><td><strong>82.5</strong></td><td><strong>78.8</strong></td><td>72.4</td><td><strong>73.4</strong></td><td><strong>71.0</strong></td><td><strong>75.3</strong></td></tr><tr><td rowspan="3">Similarity<br>(3, Language)</td><td>en-de</td><td>80.5</td><td>83.2</td><td>79.3</td><td>75.9</td><td>77.9</td><td>73.8</td><td>80.2</td></tr><tr><td>en-ro</td><td>83.5</td><td>87.6</td><td>79.9</td><td>78.2</td><td>78.3</td><td>74.0</td><td>82.7</td></tr><tr><td>lt-en</td><td>76.8</td><td>75.9</td><td>73.2</td><td>64.9</td><td>67.3</td><td>-</td><td>69.2</td></tr><tr><td></td><td>mean</td><td>80.6</td><td>82.2</td><td>77.2</td><td><strong>72.6</strong></td><td>72.9</td><td>67.5</td><td>74.5</td></tr></tbody></table>

selection to aid domain adaptation for LLM-based MT. We train a multilingual topic model which, in a challenging multilingual seen-domain setting, outperforms random and statistical baselines, showing the importance of semantically similar examples. Our method offers a lightweight, robust solution for when no parallel data is available for a new domain. However if suitable (in-domain and in-language) development data is available then information retrieval and embedding similarity-based methods are more performant, simpler solutions. In future work, we intend to assess the transferability of our method to LLM-based translation more generally by testing across various LLMs, including more explicitly multilingual models. With this work we show an example of how statistical models can complement the performance of Llama-2, an English-centric LLM, in translation tasks to and from English.

## Limitations

We recognise our work has limitations including: 1) We experimented with only one pre-trained LLM, Llama-2-13B. Further investigation is required to understand how our results and prompts would vary across a) different model families and b) varying model scales. We note therefore that our results are not generalisable to other LLMs, pending further work. 2) While we consider a variety of high- and low-resource pairs, all our tests are into or out of English. Further work is required to test other pairs, including both high-high and low-low resource pairs. We note that this may be more
difficult due to the reduced availability of good quality in-domain parallel data. 3) Our conclusions must be understood with the caveat that we do not know our chosen model's training data, including datasets and language distributions, beyond the basic information provided by Touvron et al. (2023).

## Acknowledgements

Both authors' contributions were funded by Rachel Bawden's Emergence project, DadaNMT, funded by Sorbonne Université. Rachel Bawden's participation was also funded by her chair position in the PRAIRIE institute funded by the French national agency ANR as part of the "Investissements d'avenir" programme under the reference ANR-19-P3IA-0001. The authors are grateful for the feedback provided by the anonymous reviewers.

## References

Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA Corpus: Building Parallel Language Resources for the Educational Domain. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)*, pages 1856–1862, Reykjavik, Iceland. European Language Resources Association (ELRA).

Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In-context Examples Selection for Machine Translation. In *Findings of the Association for Computational Linguistics: ACL 2023*, pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.

Roee Aharoni and Yoav Goldberg. 2020. Unsupervised Domain Clusters in Pretrained Language Models. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7747–7763, Online. Association for Computational Linguistics.

Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, and Andre Martins. 2023. Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning. In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 11127–11148, Singapore. Association for Computational Linguistics.

Ankur Bapna and Orhan Firat. 2019. Simple, Scalable Adaptation for Neural Machine Translation. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 1538–1548, Hong Kong, China. Association for Computational Linguistics.

Rachel Bawden and François Yvon. 2023. Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM. In *Proceedings of the 24th Annual Conference of the European Association for Machine Translation*, pages 157–170, Tampere, Finland. European Association for Machine Translation.

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Vilanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Gíada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Tatal, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M. Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-
Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, San-chit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lover-ing, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bog-danov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Na-joung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Al-ice Rueda, Amanda Pestana, Amir Feizpour, Am-mar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Are-zoo Abdollahi, Aycha Tammour, Azadeh HajiHos-seini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Danish Contrac-tor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ez-inanne Ozoani, Fatima Mirza, Frankline Onon-iwu, Habib Rezanejad, Hessie Jones, Indrani Bhat-tacharya, Irene Solaiman, Irina Sedenko, Isar Ne-jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim El-badri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Ra-jani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Al-izadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjava-cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Ranga-sai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A. Castillo, Mari-anna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myung-sun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale

Fung, Patrick Haller, Ramya Chandrasekhar, Re-nata Eisenberg, Robert Martin, Rodrigo Canalli, Ros-aline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S. Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Sr-ishti Kumar, Stefan Schweter, Sushil Bharati, Tan-may Laud, Théo Gigant, Tomoya Kainuma, Wo-jciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023. BLOOM: A 176B-Parameter Open-Access Multilingual Lan-guage Model. ArXiv:2211.05100 [cs].

David Blei, Lawrence Carin, and David Dunson. 2010. Probabilistic Topic Models. *IEEE Signal Processing Magazine*, 27(6):55-65. Conference Name: IEEE Signal Processing Magazine.

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. *Journal of Machine Learning Research*, 3(Jan):993-1022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma-teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In *Advances in Neural Information Processing Systems*, volume 33, pages 1877-1901. Curran Associates, Inc.

Pranjal A. Chitale, Jay Gala, Varun Gumma, Mitesh M. Khapra, and Raj Dabre. 2024. An Empirical Analysis of In-context Learning Abilities of LLMs for MT. ArXiv:2401.12097 [cs].

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin-odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An-drew M. Dai, Thanumalayan Sankaranarayana Pil-lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov,
and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. ArXiv:2204.02311 [cs].

Alexandra Chronopoulou, Matthew Peters, and Jesse Dodge. 2022. Efficient Hierarchical Domain Adaptation for Pretrained Language Models. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 1336–1351, Seattle, United States. Association for Computational Linguistics.

Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2020. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2703–2711, Online. Association for Computational Linguistics.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8440–8451, Online. Association for Computational Linguistics.

Asa Cooper Stickland, Alexandre Berard, and Vassilina Nikoulina. 2021a. Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters. In *Proceedings of the Sixth Conference on Machine Translation*, pages 578–598, Online. Association for Computational Linguistics.

Asa Cooper Stickland, Xian Li, and Marjan Ghazvininejad. 2021b. Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation. In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*, pages 3440–3453, Online. Association for Computational Linguistics.

Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. ArXiv:2207.04672 [cs].

Praveen Dakwale and Christof Monz. 2017. Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data.

In *Proceedings of Machine Translation Summit XVI: Research Track*, pages 156–169, Nagoya Japan.

Leo Gao, Jonathan Tow, Stella Biderman, Charles Lovering, Jason Phang, Anish Thite, Niklas Muennighoff, Thomas Wang, Zdeněk Kasner, Khalid Almubarak, Jeffrey Hsu, Pawan Sasanka Ammanamchi, Dirk Groeneveld, Eric Tang, Charles Foster, Andy Zou, Ben Wang, Jordan Clive, Kevin Wang, Nicholas Kross, and Fabrizio Milo. 2022. A framework for few-shot language model evaluation.

Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation. ArXiv:2302.07856 [cs].

Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. ArXiv:2203.05794 [cs].

Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023. Exploring Human-Like Translation Strategy with Large Language Models. ArXiv:2305.04118 [cs].

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. ArXiv:2302.09210 [cs].

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. A Simple Baseline to Semi-Supervised Domain Adaptation for Machine Translation. ArXiv:2001.08140 [cs].

Mahesh Joshi, Mark Dredze, William W. Cohen, and Carolyn P. Rosé. 2013. What's in a Domain? Multi-Domain Learning for Multi-Attribute Data. In *Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 685–690, Atlanta, Georgia. Association for Computational Linguistics.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6282–6293, Online. Association for Computational Linguistics.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, pages 427–431, Valencia, Spain. Association for Computational Linguistics.

Catherine Kobus, Josep Crego, and Jean Senellart. 2017. Domain Control for Neural Machine Translation. In
*Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017*, pages 372–378, Varna, Bulgaria. INCOMA Ltd.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In *Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions*, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.

Philipp Koehn and Rebecca Knowles. 2017. Six Challenges for Neural Machine Translation. In *Proceedings of the First Workshop on Neural Machine Translation*, pages 28–39, Vancouver. Association for Computational Linguistics.

Aswanth Kumar, Ratish Puduppully, Raj Dabre, and Anoop Kunchukuttan. 2023. CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation. In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 7736–7752, Singapore. Association for Computational Linguistics.

Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Cheng, and Jiajun Chen. 2023. Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions. ArXiv:2305.15083 [cs].

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Nathan Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot Learning with Multilingual Generative Language Models. In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan. European Language Resources Association (ELRA).

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3? In *Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures*, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.

Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Hao-ran Yang, Wai Lam, and Furu Wei. 2023. Chain-of-Dictionary Prompting Elicits Translation in Large Language Models. ArXiv:2305.06575 [cs].

Minh-Thang Luong and Christopher Manning. 2015. Stanford neural machine translation systems for spoken language domains. In *Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign*, pages 76–79, Da Nang, Vietnam.

Claudia Malzer and Marcus Baum. 2020. A Hybrid Approach To Hierarchical Density-based Cluster Selection. In *2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)*, pages 223–228.

Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4984–4997, Online. Association for Computational Linguistics.

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. UMAP: Uniform Manifold Approximation and Projection. *Journal of Open Source Software*, 3(29):861.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In *Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics*, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Cheonbok Park, Hantae Kim, Ioan Calapodescu, Hyun Chang Cho, and Vassilina Nikoulina. 2022. DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translation. In *Findings of the Association for Computational Linguistics: ACL 2022*, pages 1789–1807, Dublin, Ireland. Association for Computational Linguistics.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. *Journal of Machine Learning Research*, 12(85):2825–2830.

Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards Making the Most of ChatGPT for Machine Translation. In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 5622–5633, Singapore. Association for Computational Linguistics.

Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In *Proceedings of the Third Conference on*
*Machine Translation: Research Papers*, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbel-IST 2022 Submission for the Metrics Shared Task. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2020. Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 4512–4525, Online. Association for Computational Linguistics.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning To Retrieve Prompts for In-Context Learning. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 2655–2671, Seattle, United States. Association for Computational Linguistics.

Danielle Saunders. 2022. Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey. *Journal of Artificial Intelligence Research*, 75:351–424.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Models with Monolingual Data. In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 86–96, Berlin, Germany. Association for Computational Linguistics.

Amr Sharaf, Hany Hassan, and Hal Daumé III. 2020. Meta-Learning for Few-Shot NMT Adaptation. In *Proceedings of the Fourth Workshop on Neural Generation and Translation*, pages 43–53, Online. Association for Computational Linguistics.

Emmanouil Stergiadis, Satendra Kumar, Fedor Kovalev, and Pavel Levin. 2021. Multi-Domain Adaptation in Neural Machine Translation Through Multidimensional Tagging. In *Proceedings of Machine Translation Summit XVIII: Users and Providers Track*, pages 396–420, Virtual. Association for Machine Translation in the Americas.

Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In *Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)*, pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv:2307.09288 [cs].

Marlies van der Wees, Arianna Bisazza, Wouter Weerkamp, and Christof Monz. 2015. What's in a Domain? Analyzing Genre and Topic Differences in Statistical Machine Translation. In *Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)*, pages 560–566, Beijing, China. Association for Computational Linguistics.

David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting PaLM for Translation: Assessing Strategies and Performance. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 15406–15427, Toronto, Canada. Association for Computational Linguistics.

Jonas Waldendorf, Alexandra Birch, Barry Hadow, and Antonio Valerio Micele Barone. 2022. Improving Translation of Out Of Vocabulary Words using Bilingual Lexicon Induction in Low-Resource Machine Translation. In *Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)*, pages 144–156, Orlando, USA. Association for Machine Translation in the Americas.

Weixuan Wang, Wei Peng, Meng Zhang, and Qun Liu. 2021. Neural Machine Translation with Heterogeneous Topic Knowledge Embeddings. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 3197–3202,
Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In *Advances in Neural Information Processing Systems*, volume 33, pages 5776–5788. Curran Associates, Inc.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 38–45, Online. Association for Computational Linguistics.

Danai Xezonaki, Talaat Khalil, David Stap, and Brandon Denis. 2023. Improving Domain Robustness in Neural Machine Translation with Fused Topic Knowledge Embeddings. In *Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track*, pages 209–221, Macau SAR, China. Asia-Pacific Association for Machine Translation.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hasan Awadalla. 2023. A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ArXiv:2309.11674 [cs].

Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting Large Language Model for Machine Translation: A Case Study. ArXiv:2301.07069 [cs].

Jian Zhang, Liangyou Li, Andy Way, and Qun Liu. 2016. Topic-Informed Neural Machine Translation. In *Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers*, pages 1807–1817, Osaka, Japan. The COLING 2016 Organizing Committee.

Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehn. 2023b. Machine Translation with Large Language Models: Prompting, Few-shot Learning, and Fine-tuning with QLoRA. In *Proceedings of the Eighth Conference on Machine Translation*, pages 468–481, Singapore. Association for Computational Linguistics.

Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019. Curriculum Learning for Domain Adaptation in Neural Machine Translation. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 1903–1915, Minneapolis, Minnesota. Association for Computational Linguistics.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic Chain of Thought Prompting in Large Language Models. In *Proceedings of the Eleventh International Conference on Learning Representations*, Kigali, Rwanda.

Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. ArXiv:2304.04675 [cs].

## A Results legend

In Table 12 we provide a brief reference for the experimental naming used in results tables.

<table><caption>Table 12: Reference for experimental terminology.</caption><thead><tr><th>Label</th><th>Description</th></tr></thead><tbody><tr><td>Base</td><td>Zero-shot XGLM-style prompt</td></tr><tr><td>Label</td><td>Descriptive domain label + Base prompt</td></tr><tr><td>Keywords-10</td><td>10 related keywords selected from topic + Base prompt</td></tr><tr><td>Fewshot</td><td>Example source-target pairs selected by topic model + Base prompt</td></tr><tr><td>(1, Seen)</td><td>1-shot; selected from multilingual seen domains</td></tr><tr><td>(3, Language)</td><td>3-shot; selected from all-domain in-language data</td></tr><tr><td>(200 topic)</td><td>Selected using a 200-topic model.</td></tr><tr><td>(Random Topic)</td><td>Examples/keywords selected from one random topic.</td></tr></tbody></table>

## B Further examples

In Table 13 we provide further examples of the prompt format, and predicted outputs, for Label, Keywords-10 (Seen), and Fewshot (3, Seen) settings.

## C Topic Model Hyperparameters

We use UMAP and HDBSCAN implementations from cuML.<fnref n="11" /> Our embedding model is paraphrase-multilingual-MiniLM-L12-v2 from SentencePiece. This language model is a MiniLM model (Wang et al., 2020) distilled from XLM-R (Conneau et al., 2020) and thus is expected to have some knowledge of the 100 lan-

<footnote n="11">¹¹www.github.com/rapidsai/cuml</footnote>
<table><caption>Table 13: Three examples illustrating our different prompting methods: domain labels, topic keywords, and a 3-shot topic-guided example from seen domains. We show examples for Lithuanian-English, English-German, and English-French in OpenSubtitles, Tanzil, and JRC domains respectively, with predicted and target outputs below the example prompts.</caption><tr><td>[Label]</td><td>Domain: TV and movie subtitles.</td></tr><tr><td>[Source]</td><td><b>Lithuanian:</b> Šis miestas, ir viskas jame... = English:</td></tr><tr><td>[Prediction]</td><td>This city, and all of it...</td></tr><tr><td>[Target]</td><td>This city, everyone in it...</td></tr><tr><td>[Keywords-10]</td><td>Related keywords: juice, nápojů, grapefruitsaft, štáva, grapefruit, drinks, greipfrutų, vartoti, pomerančová, frucht.</td></tr><tr><td>[Source]</td><td><b>English:</b> fruits — and they will be held in honour, = German:</td></tr><tr><td>[Prediction]</td><td>früchte — und sie werden in Ehren gehalten werden.</td></tr><tr><td>[Target]</td><td>Früchte, und sie werden geehrt</td></tr><tr><td>[Fewshot]</td><td>English: (c) With effect from 1 July 1972 the text of Article 4 (2) and (3) shall be replaced by the following: = French: c) Le texte de l'article 4 paragraphes 2 et 3 est remplacé par le texte suivant, avec effet au 1er juillet 1972:<br>English: 9. Article 28 shall be replaced by the following: = French: 9) L'article 28, est remplacé par le texte suivant:<br>English: (h) the text of Part L. PORTUGAL shall be replaced by the following: = French: h) Le texte de la partie L. PORTUGAL est remplacé par le texte suivant:</td></tr><tr><td>[Source]</td><td><b>English:</b> a) in paragraph 1 the following subparagraph shall be added: = French:</td></tr><tr><td>[Prediction]</td><td>a) au paragraphe 1, le sous-alinéa suivant est ajouté:</td></tr><tr><td>[Target]</td><td>a) au paragraphe 1, l'alinéa suivant est ajouté:</td></tr></table>

guages used in training. After dimensionality reduction and clustering, the inputs are tokenized using CountVectorizer and weighted with cTF-IDF (Pedregosa et al., 2011). The standard BERTopic hyperparameters are as follows: 500 topics, 10 keywords/topic, with stopwords removed for the vectorisation step; no stopwords were available for Lithuanian, so we used the top 100 most frequent words from our multi-domain development set.

Other parameters follow the standard implementation of BERTopic: UMAP: number of components = 5, number of neighbours = 15, metric = cosine distance; HBDSCAN: minimum samples=10. Finally we use the KeyBERT-inspired implementation to select the best 10 keywords to represent a topic, which avoids the repetitive selection of function words and stopwords. We also note here that predicting the nearest topic for a given input does not significantly slow down the inference process, with a rate of 80-100 iterations per second.

## D Length and Language ID results

We present raw and trimmed results in Tables 14 and 15 for length and correct language identification respectively, for a selection of settings (Base, Verbose and Fewshot (3, Seen)). These results illustrate how the trimming procedure vastly reduces the length and improves the correct language identification of the outputs; note especially the high

sentence lengths and low correct language identification for Base-raw experiments.

## E Full COMET and BLEU results

We present full COMET and BLEU results in Tables 16–23, which follow the same patterns presented in Section 5.
<table><caption>Table 14: Average length measured in space-tokenized words for selected settings.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="6">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="12">Base-raw</td><td>en-cs</td><td>91.8</td><td>85.7</td><td>76.0</td><td>20.7</td><td>41.0</td><td>73.4</td><td>34.2</td></tr><tr><td>en-de</td><td>51.5</td><td>95.2</td><td>69.2</td><td>15.1</td><td>32.4</td><td>32.2</td><td>28.9</td></tr><tr><td>en-fi</td><td>94.5</td><td>89.3</td><td>77.7</td><td>29.7</td><td>40.1</td><td>-</td><td>44.0</td></tr><tr><td>en-fr</td><td>58.3</td><td>82.8</td><td>74.6</td><td>24.2</td><td>44.3</td><td>63.5</td><td>34.2</td></tr><tr><td>en-lt</td><td>99.2</td><td>84.6</td><td>69.9</td><td>25.2</td><td>53.4</td><td>-</td><td>57.0</td></tr><tr><td>en-ro</td><td>83.7</td><td>87.8</td><td>65.7</td><td>19.6</td><td>36.4</td><td>57.7</td><td>35.5</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>57.6</td><td>14.1</td><td>50.5</td><td>27.3</td><td>22.8</td></tr><tr><td>cs-en</td><td>27.8</td><td>43.9</td><td>28.3</td><td>7.9</td><td>14.8</td><td>31.3</td><td>16.0</td></tr><tr><td>fr-en</td><td>28.5</td><td>45.8</td><td>32.2</td><td>8.8</td><td>17.0</td><td>25.3</td><td>18.2</td></tr><tr><td>lt-en</td><td>29.9</td><td>52.8</td><td>31.4</td><td>7.8</td><td>18.0</td><td>-</td><td>19.1</td></tr><tr><td>ro-en</td><td>28.7</td><td>44.9</td><td>31.1</td><td>9.8</td><td>15.9</td><td>19.1</td><td>17.5</td></tr><tr><td>mean</td><td>59.4</td><td>71.3</td><td>55.8</td><td>16.6</td><td>33.1</td><td>41.2</td><td>29.8</td></tr><tr><td rowspan="12">Base</td><td>en-cs</td><td>16.7</td><td>25.0</td><td>6.4</td><td>7.8</td><td>12.0</td><td>17.3</td><td>14.4</td></tr><tr><td>en-de</td><td>15.6</td><td>28.1</td><td>11.6</td><td>7.7</td><td>13.8</td><td>16.5</td><td>15.8</td></tr><tr><td>en-fi</td><td>17.3</td><td>19.5</td><td>6.5</td><td>7.2</td><td>11.8</td><td>-</td><td>14.3</td></tr><tr><td>en-fr</td><td>18.3</td><td>29.7</td><td>13.6</td><td>8.4</td><td>14.6</td><td>16.7</td><td>17.2</td></tr><tr><td>en-lt</td><td>16.6</td><td>30.1</td><td>6.9</td><td>8.6</td><td>18.0</td><td>-</td><td>22.1</td></tr><tr><td>en-ro</td><td>17.9</td><td>28.4</td><td>7.2</td><td>8.9</td><td>14.3</td><td>18.0</td><td>16.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>27.9</td><td>11.5</td><td>27.9</td><td>24.3</td><td>19.9</td></tr><tr><td>cs-en</td><td>19.2</td><td>31.5</td><td>9.9</td><td>7.6</td><td>13.5</td><td>24.3</td><td>15.9</td></tr><tr><td>fr-en</td><td>17.3</td><td>31.4</td><td>15.5</td><td>7.9</td><td>14.6</td><td>17.1</td><td>16.8</td></tr><tr><td>lt-en</td><td>22.4</td><td>40.6</td><td>14.4</td><td>6.8</td><td>16.3</td><td>-</td><td>17.1</td></tr><tr><td>ro-en</td><td>19.0</td><td>31.5</td><td>9.7</td><td>8.0</td><td>14.1</td><td>16.8</td><td>16.0</td></tr><tr><td>mean</td><td>18.0</td><td>29.6</td><td>11.8</td><td>8.2</td><td>15.5</td><td>18.9</td><td>16.9</td></tr><tr><td rowspan="12">Verbose-raw</td><td>en-cs</td><td>16.7</td><td>26.4</td><td>6.7</td><td>8.7</td><td>13.4</td><td>17.7</td><td>14.9</td></tr><tr><td>en-de</td><td>15.9</td><td>28.1</td><td>11.3</td><td>8.2</td><td>14.2</td><td>17.6</td><td>16.8</td></tr><tr><td>en-fi</td><td>13.8</td><td>19.0</td><td>5.8</td><td>7.4</td><td>11.2</td><td>-</td><td>12.6</td></tr><tr><td>en-fr</td><td>19.1</td><td>30.0</td><td>14.9</td><td>9.2</td><td>16.6</td><td>18.4</td><td>18.9</td></tr><tr><td>en-lt</td><td>19.7</td><td>33.5</td><td>6.7</td><td>11.1</td><td>25.0</td><td>-</td><td>27.4</td></tr><tr><td>en-ro</td><td>18.1</td><td>27.7</td><td>7.3</td><td>8.1</td><td>14.6</td><td>17.5</td><td>16.7</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>9.4</td><td>14.0</td><td>24.4</td><td>22.4</td><td>21.1</td></tr><tr><td>cs-en</td><td>26.4</td><td>40.2</td><td>11.7</td><td>11.5</td><td>19.3</td><td>30.4</td><td>22.4</td></tr><tr><td>fr-en</td><td>17.3</td><td>29.7</td><td>15.1</td><td>7.9</td><td>14.3</td><td>17.7</td><td>16.5</td></tr><tr><td>lt-en</td><td>20.6</td><td>35.3</td><td>8.2</td><td>7.2</td><td>16.9</td><td>-</td><td>18.4</td></tr><tr><td>ro-en</td><td>18.5</td><td>30.3</td><td>7.8</td><td>8.2</td><td>14.8</td><td>17.7</td><td>16.4</td></tr><tr><td>mean</td><td>18.6</td><td>30.0</td><td>9.5</td><td>9.2</td><td>16.8</td><td>19.9</td><td>18.4</td></tr><tr><td rowspan="12">Verbose</td><td>en-cs</td><td>16.6</td><td>26.4</td><td>6.6</td><td>8.7</td><td>13.4</td><td>17.7</td><td>14.9</td></tr><tr><td>en-de</td><td>15.8</td><td>28.1</td><td>11.2</td><td>8.2</td><td>14.2</td><td>17.6</td><td>16.8</td></tr><tr><td>en-fi</td><td>13.6</td><td>19.0</td><td>5.8</td><td>7.4</td><td>11.2</td><td>-</td><td>12.6</td></tr><tr><td>en-fr</td><td>19.0</td><td>30.0</td><td>14.9</td><td>9.2</td><td>16.6</td><td>18.4</td><td>18.9</td></tr><tr><td>en-lt</td><td>19.7</td><td>33.5</td><td>6.7</td><td>11.1</td><td>25.0</td><td>-</td><td>27.4</td></tr><tr><td>en-ro</td><td>18.1</td><td>27.7</td><td>7.3</td><td>8.1</td><td>14.5</td><td>17.5</td><td>16.7</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>9.4</td><td>14.0</td><td>24.4</td><td>22.4</td><td>21.1</td></tr><tr><td>cs-en</td><td>26.4</td><td>40.2</td><td>11.7</td><td>11.5</td><td>19.3</td><td>30.4</td><td>22.4</td></tr><tr><td>fr-en</td><td>17.2</td><td>29.7</td><td>15.0</td><td>7.9</td><td>14.3</td><td>17.7</td><td>16.5</td></tr><tr><td>lt-en</td><td>20.5</td><td>35.3</td><td>8.2</td><td>7.2</td><td>16.9</td><td>-</td><td>18.4</td></tr><tr><td>ro-en</td><td>18.5</td><td>30.3</td><td>7.8</td><td>8.2</td><td>14.8</td><td>17.7</td><td>16.4</td></tr><tr><td>mean</td><td>18.5</td><td>30.0</td><td>9.5</td><td>9.2</td><td>16.8</td><td>19.9</td><td>18.4</td></tr><tr><td rowspan="12">Fewshot-raw (3, Seen)</td><td>en-cs</td><td>17.5</td><td>26.0</td><td>7.0</td><td>7.0</td><td>12.4</td><td>16.6</td><td>14.1</td></tr><tr><td>en-de</td><td>18.1</td><td>27.0</td><td>13.3</td><td>7.8</td><td>14.1</td><td>17.5</td><td>16.4</td></tr><tr><td>en-fi</td><td>14.6</td><td>19.7</td><td>5.9</td><td>6.5</td><td>12.5</td><td>-</td><td>12.3</td></tr><tr><td>en-fr</td><td>20.1</td><td>29.1</td><td>16.2</td><td>9.0</td><td>16.2</td><td>17.8</td><td>18.0</td></tr><tr><td>en-lt</td><td>18.4</td><td>29.9</td><td>6.1</td><td>8.9</td><td>20.9</td><td>-</td><td>23.2</td></tr><tr><td>en-ro</td><td>19.3</td><td>27.6</td><td>7.1</td><td>8.0</td><td>14.4</td><td>17.1</td><td>16.3</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>8.3</td><td>10.0</td><td>23.1</td><td>21.4</td><td>18.1</td></tr><tr><td>cs-en</td><td>18.2</td><td>30.4</td><td>7.2</td><td>7.4</td><td>12.7</td><td>21.7</td><td>15.8</td></tr><tr><td>fr-en</td><td>17.3</td><td>30.0</td><td>14.4</td><td>7.7</td><td>14.4</td><td>17.3</td><td>16.4</td></tr><tr><td>lt-en</td><td>17.5</td><td>31.4</td><td>7.2</td><td>6.5</td><td>13.8</td><td>-</td><td>15.3</td></tr><tr><td>ro-en</td><td>18.5</td><td>29.7</td><td>8.0</td><td>7.8</td><td>13.9</td><td>16.7</td><td>15.8</td></tr><tr><td>mean</td><td>18.0</td><td>28.1</td><td>9.2</td><td>7.9</td><td>15.3</td><td>18.3</td><td>16.5</td></tr><tr><td rowspan="12">Fewshot (3, Seen)</td><td>en-cs</td><td>14.7</td><td>24.3</td><td>6.1</td><td>6.8</td><td>12.0</td><td>16.6</td><td>13.9</td></tr><tr><td>en-de</td><td>15.1</td><td>26.2</td><td>11.2</td><td>7.7</td><td>13.6</td><td>16.6</td><td>16.3</td></tr><tr><td>en-fi</td><td>12.8</td><td>17.8</td><td>5.2</td><td>6.5</td><td>11.0</td><td>-</td><td>12.1</td></tr><tr><td>en-fr</td><td>18.1</td><td>28.9</td><td>14.9</td><td>8.6</td><td>15.6</td><td>17.3</td><td>17.9</td></tr><tr><td>en-lt</td><td>17.1</td><td>29.4</td><td>5.8</td><td>8.8</td><td>20.7</td><td>-</td><td>23.1</td></tr><tr><td>en-ro</td><td>17.4</td><td>27.3</td><td>6.8</td><td>8.0</td><td>14.0</td><td>17.1</td><td>16.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>8.0</td><td>9.9</td><td>22.9</td><td>21.4</td><td>18.1</td></tr><tr><td>cs-en</td><td>18.1</td><td>30.4</td><td>7.1</td><td>7.4</td><td>12.7</td><td>21.7</td><td>15.8</td></tr><tr><td>fr-en</td><td>16.5</td><td>29.8</td><td>14.3</td><td>7.7</td><td>14.4</td><td>17.2</td><td>16.4</td></tr><tr><td>lt-en</td><td>17.2</td><td>31.0</td><td>7.1</td><td>6.5</td><td>13.8</td><td>-</td><td>15.3</td></tr><tr><td>ro-en</td><td>17.9</td><td>29.5</td><td>7.3</td><td>7.8</td><td>13.9</td><td>16.7</td><td>15.8</td></tr><tr><td>mean</td><td>16.5</td><td>27.5</td><td>8.5</td><td>7.8</td><td>15.0</td><td>18.1</td><td>16.4</td></tr></tbody></table>

<table><caption>Table 15: Average correct language identification (%) measured with FastText's language ID tool.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="6">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="12">Base-raw</td><td>en-cs</td><td>30.0</td><td>62.0</td><td>23.0</td><td>76.8</td><td>70.8</td><td>66.2</td><td>79.2</td></tr><tr><td>en-de</td><td>57.2</td><td>58.4</td><td>40.6</td><td>92.0</td><td>86.0</td><td>87.6</td><td>88.6</td></tr><tr><td>en-fi</td><td>12.8</td><td>31.6</td><td>11.4</td><td>56.8</td><td>49.2</td><td>-</td><td>53.0</td></tr><tr><td>en-fr</td><td>44.4</td><td>47.0</td><td>21.2</td><td>73.8</td><td>67.6</td><td>55.6</td><td>80.0</td></tr><tr><td>en-lt</td><td>13.6</td><td>42.6</td><td>23.4</td><td>67.0</td><td>48.8</td><td>-</td><td>52.2</td></tr><tr><td>en-ro</td><td>24.6</td><td>60.8</td><td>28.8</td><td>80.4</td><td>79.8</td><td>70.4</td><td>81.6</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>48.4</td><td>78.4</td><td>43.0</td><td>87.6</td><td>79.0</td></tr><tr><td>cs-en</td><td>91.8</td><td>91.6</td><td>89.2</td><td>97.8</td><td>98.4</td><td>97.4</td><td>99.6</td></tr><tr><td>fr-en</td><td>95.4</td><td>95.2</td><td>95.0</td><td>99.2</td><td>99.2</td><td>99.0</td><td>99.8</td></tr><tr><td>lt-en</td><td>88.4</td><td>91.0</td><td>86.0</td><td>94.6</td><td>96.2</td><td>-</td><td>95.4</td></tr><tr><td>ro-en</td><td>92.4</td><td>91.4</td><td>89.6</td><td>97.6</td><td>98.0</td><td>98.6</td><td>99.2</td></tr><tr><td>mean</td><td>55.1</td><td>67.2</td><td>50.6</td><td>83.1</td><td>76.1</td><td>82.8</td><td>82.5</td></tr><tr><td rowspan="12">Base</td><td>en-cs</td><td>41.4</td><td>79.8</td><td>59.8</td><td>82.8</td><td>78.4</td><td>90.0</td><td>83.4</td></tr><tr><td>en-de</td><td>67.2</td><td>76.0</td><td>64.8</td><td>94.8</td><td>89.2</td><td>97.0</td><td>93.0</td></tr><tr><td>en-fi</td><td>22.2</td><td>61.0</td><td>36.2</td><td>65.4</td><td>57.6</td><td>-</td><td>58.0</td></tr><tr><td>en-fr</td><td>62.8</td><td>81.4</td><td>64.4</td><td>80.4</td><td>79.6</td><td>94.0</td><td>88.4</td></tr><tr><td>en-lt</td><td>19.0</td><td>59.4</td><td>43.2</td><td>72.8</td><td>56.2</td><td>-</td><td>57.8</td></tr><tr><td>en-ro</td><td>41.0</td><td>82.8</td><td>56.4</td><td>86.0</td><td>85.6</td><td>96.6</td><td>85.2</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>51.0</td><td>79.4</td><td>43.4</td><td>87.4</td><td>79.8</td></tr><tr><td>cs-en</td><td>96.6</td><td>97.8</td><td>94.0</td><td>98.0</td><td>99.2</td><td>99.4</td><td>99.6</td></tr><tr><td>fr-en</td><td>96.8</td><td>98.4</td><td>96.2</td><td>99.2</td><td>99.6</td><td>99.8</td><td>99.6</td></tr><tr><td>lt-en</td><td>91.2</td><td>94.8</td><td>89.4</td><td>94.6</td><td>96.4</td><td>-</td><td>96.0</td></tr><tr><td>ro-en</td><td>96.6</td><td>98.6</td><td>93.4</td><td>97.4</td><td>98.6</td><td>99.6</td><td>99.6</td></tr><tr><td>mean</td><td>63.5</td><td>83.0</td><td>68.1</td><td>86.4</td><td>80.3</td><td>95.5</td><td>85.5</td></tr><tr><td rowspan="12">Verbose</td><td>en-cs</td><td>95.0</td><td>98.8</td><td>89.8</td><td>93.0</td><td>96.0</td><td>97.8</td><td>98.2</td></tr><tr><td>en-de</td><td>96.8</td><td>98.2</td><td>94.0</td><td>98.0</td><td>97.4</td><td>98.6</td><td>98.8</td></tr><tr><td>en-fi</td><td>95.6</td><td>96.6</td><td>92.4</td><td>95.2</td><td>96.8</td><td>-</td><td>98.4</td></tr><tr><td>en-fr</td><td>95.4</td><td>98.6</td><td>91.6</td><td>93.0</td><td>95.2</td><td>99.0</td><td>97.4</td></tr><tr><td>en-lt</td><td>91.8</td><td>92.6</td><td>88.0</td><td>90.2</td><td>94.0</td><td>-</td><td>95.2</td></tr><tr><td>en-ro</td><td>96.6</td><td>99.0</td><td>87.2</td><td>93.0</td><td>97.0</td><td>99.0</td><td>98.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>85.6</td><td>92.0</td><td>95.6</td><td>93.6</td><td>94.6</td></tr><tr><td>cs-en</td><td>97.8</td><td>97.4</td><td>92.0</td><td>97.8</td><td>99.4</td><td>99.2</td><td>99.8</td></tr><tr><td>fr-en</td><td>97.0</td><td>98.6</td><td>93.8</td><td>98.8</td><td>99.2</td><td>100.0</td><td>99.4</td></tr><tr><td>lt-en</td><td>92.8</td><td>95.8</td><td>84.8</td><td>95.6</td><td>96.8</td><td>-</td><td>96.2</td></tr><tr><td>ro-en</td><td>97.0</td><td>98.8</td><td>88.4</td><td>97.6</td><td>99.2</td><td>100.0</td><td>99.6</td></tr><tr><td>mean</td><td>95.6</td><td>97.4</td><td>89.8</td><td>94.9</td><td>97.0</td><td>98.4</td><td>97.8</td></tr><tr><td rowspan="12">Verbose</td><td>en-cs</td><td>94.8</td><td>98.6</td><td>89.6</td><td>93.0</td><td>96.0</td><td>97.8</td><td>98.2</td></tr><tr><td>en-de</td><td>96.8</td><td>98.2</td><td>94.0</td><td>98.0</td><td>97.4</td><td>98.6</td><td>98.8</td></tr><tr><td>en-fi</td><td>95.4</td><td>96.6</td><td>92.4</td><td>95.2</td><td>96.8</td><td>-</td><td>98.4</td></tr><tr><td>en-fr</td><td>95.2</td><td>98.6</td><td>91.4</td><td>93.0</td><td>95.2</td><td>99.0</td><td>97.4</td></tr><tr><td>en-lt</td><td>91.8</td><td>92.6</td><td>88.0</td><td>90.2</td><td>94.0</td><td>-</td><td>95.2</td></tr><tr><td>en-ro</td><td>96.6</td><td>99.0</td><td>87.2</td><td>93.0</td><td>96.8</td><td>99.0</td><td>98.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>85.6</td><td>92.0</td><td>95.6</td><td>93.6</td><td>94.6</td></tr><tr><td>cs-en</td><td>97.8</td><td>97.4</td><td>92.0</td><td>97.8</td><td>99.4</td><td>99.2</td><td>99.8</td></tr><tr><td>fr-en</td><td>96.8</td><td>98.6</td><td>93.6</td><td>98.8</td><td>99.2</td><td>100.0</td><td>99.4</td></tr><tr><td>lt-en</td><td>92.8</td><td>95.8</td><td>84.8</td><td>95.6</td><td>96.8</td><td>-</td><td>96.2</td></tr><tr><td>ro-en</td><td>97.0</td><td>98.8</td><td>88.4</td><td>97.6</td><td>99.2</td><td>100.0</td><td>99.6</td></tr><tr><td>mean</td><td>95.5</td><td>97.4</td><td>89.7</td><td>94.9</td><td>96.9</td><td>98.4</td><td>97.8</td></tr><tr><td rowspan="12">Fewshot-raw (3, Seen)</td><td>en-cs</td><td>95.4</td><td>97.6</td><td>91.4</td><td>91.8</td><td>94.2</td><td>98.0</td><td>97.6</td></tr><tr><td>en-de</td><td>95.4</td><td>99.2</td><td>92.6</td><td>99.4</td><td>98.4</td><td>97.8</td><td>99.4</td></tr><tr><td>en-fi</td><td>95.4</td><td>96.0</td><td>88.4</td><td>96.4</td><td>95.2</td><td>-</td><td>98.4</td></tr><tr><td>en-fr</td><td>94.8</td><td>99.0</td><td>92.8</td><td>92.4</td><td>94.4</td><td>98.4</td><td>98.0</td></tr><tr><td>en-lt</td><td>93.8</td><td>97.2</td><td>86.4</td><td>90.2</td><td>93.8</td><td>-</td><td>94.8</td></tr><tr><td>en-ro</td><td>96.4</td><td>99.4</td><td>85.8</td><td>92.8</td><td>95.4</td><td>99.0</td><td>98.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>95.2</td><td>98.0</td><td>96.0</td><td>99.8</td><td>98.8</td></tr><tr><td>cs-en</td><td>98.2</td><td>99.4</td><td>94.2</td><td>98.2</td><td>99.4</td><td>100.0</td><td>99.6</td></tr><tr><td>fr-en</td><td>98.2</td><td>99.8</td><td>96.6</td><td>98.8</td><td>99.6</td><td>99.8</td><td>99.6</td></tr><tr><td>lt-en</td><td>96.2</td><td>98.4</td><td>90.0</td><td>95.8</td><td>98.0</td><td>-</td><td>98.6</td></tr><tr><td>ro-en</td><td>97.4</td><td>99.4</td><td>91.4</td><td>98.2</td><td>98.6</td><td>99.4</td><td>99.6</td></tr><tr><td>mean</td><td>96.1</td><td>98.5</td><td>91.3</td><td>95.6</td><td>96.6</td><td>99.0</td><td>98.4</td></tr><tr><td rowspan="12">Fewshot (3, Seen)</td><td>en-cs</td><td>96.0</td><td>98.2</td><td>92.0</td><td>92.0</td><td>94.8</td><td>98.2</td><td>97.6</td></tr><tr><td>en-de</td><td>97.4</td><td>99.8</td><td>93.8</td><td>99.6</td><td>99.0</td><td>99.8</td><td>99.6</td></tr><tr><td>en-fi</td><td>97.2</td><td>98.0</td><td>89.0</td><td>96.4</td><td>96.6</td><td>-</td><td>98.4</td></tr><tr><td>en-fr</td><td>96.4</td><td>99.6</td><td>93.4</td><td>94.2</td><td>96.4</td><td>99.6</td><td>98.0</td></tr><tr><td>en-lt</td><td>95.0</td><td>97.6</td><td>86.8</td><td>90.2</td><td>94.2</td><td>-</td><td>94.8</td></tr><tr><td>en-ro</td><td>98.2</td><td>99.8</td><td>86.2</td><td>92.8</td><td>95.4</td><td>99.2</td><td>98.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>95.8</td><td>98.0</td><td>95.4</td><td>99.8</td><td>98.8</td></tr><tr><td>cs-en</td><td>98.4</td><td>99.4</td><td>94.4</td><td>98.2</td><td>99.4</td><td>100.0</td><td>99.6</td></tr><tr><td>fr-en</td><td>98.4</td><td>99.8</td><td>96.4</td><td>98.8</td><td>99.6</td><td>99.8</td><td>99.6</td></tr><tr><td>lt-en</td><td>96.4</td><td>98.6</td><td>90.0</td><td>95.8</td><td>98.0</td><td>-</td><td>98.6</td></tr><tr><td>ro-en</td><td>97.6</td><td>99.6</td><td>91.2</td><td>98.2</td><td>98.6</td><td>99.4</td><td>99.6</td></tr><tr><td>mean</td><td>97.1</td><td>99.0</td><td>91.7</td><td>95.8</td><td>97.0</td><td>99.5</td><td>98.5</td></tr></tbody></table>
<table><caption>Table 16: COMET scores for various zero-shot translation prompts, and for zero-shot NLLB tests.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="8">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">NLLB-1.3B</td><td>en-cs</td><td>88</td><td>91.8</td><td>81.8</td><td>84.1</td><td>83.4</td><td>77.6</td><td>86.4</td></tr><tr><td>en-de</td><td>82.5</td><td>86.5</td><td>78.9</td><td>79.5</td><td>80.6</td><td>76.9</td><td>84.3</td></tr><tr><td>en-fi</td><td>87</td><td>90.9</td><td>80.9</td><td>84.2</td><td>84.9</td><td>-</td><td>87.7</td></tr><tr><td>en-fr</td><td>83.7</td><td>88.6</td><td>78</td><td>79.1</td><td>80.2</td><td>78.2</td><td>84</td></tr><tr><td>en-lt</td><td>84.7</td><td>90.6</td><td>78.8</td><td>81.4</td><td>81.3</td><td>-</td><td>85.6</td></tr><tr><td>en-ro</td><td>86</td><td>90.5</td><td>80.6</td><td>83.7</td><td>82.9</td><td>79.7</td><td>86.8</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>76.9</td><td>80.1</td><td>73.2</td><td>85.2</td><td>83.9</td></tr><tr><td>cs-en</td><td>86</td><td>86.1</td><td>79.9</td><td>81.6</td><td>81.5</td><td>74.3</td><td>84.2</td></tr><tr><td>fr-en</td><td>84.6</td><td>87.5</td><td>79.4</td><td>80.7</td><td>82.8</td><td>75.5</td><td>85.9</td></tr><tr><td>lt-en</td><td>82.4</td><td>86.4</td><td>75.7</td><td>79.8</td><td>80.4</td><td>-</td><td>83.5</td></tr><tr><td>ro-en</td><td>84.5</td><td>87.6</td><td>79.2</td><td>84</td><td>83.2</td><td>73.9</td><td>86.4</td></tr><tr><td></td><td>mean</td><td>84.9</td><td>88.7</td><td>79.1</td><td>81.7</td><td>81.3</td><td>77.7</td><td>85.3</td></tr><tr><td rowspan="10">Base</td><td>en-cs</td><td>65.7</td><td>76.6</td><td>71.6</td><td>70.8</td><td>69.6</td><td>63.2</td><td>72.1</td></tr><tr><td>en-de</td><td>71.8</td><td>74.0</td><td>72.1</td><td>75.0</td><td>74.4</td><td>68.5</td><td>78.2</td></tr><tr><td>en-fi</td><td>65.4</td><td>68.8</td><td>68.4</td><td>70.3</td><td>70.3</td><td>-</td><td>71.3</td></tr><tr><td>en-fr</td><td>75.2</td><td>79.6</td><td>72.9</td><td>72.6</td><td>73.9</td><td>68.7</td><td>78.9</td></tr><tr><td>en-lt</td><td>54.7</td><td>45.9</td><td>59.9</td><td>52.0</td><td>47.8</td><td>-</td><td>47.3</td></tr><tr><td>en-ro</td><td>66.5</td><td>79.2</td><td>70.9</td><td>74.1</td><td>72.5</td><td>65.7</td><td>77.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>39.5</td><td>46.9</td><td>35.9</td><td>28.9</td><td>37.9</td></tr><tr><td>cs-en</td><td>82.4</td><td>81.3</td><td>77.3</td><td>76.3</td><td>77.8</td><td>66.6</td><td>80.7</td></tr><tr><td>fr-en</td><td>82.7</td><td>83.9</td><td>78.0</td><td>79.2</td><td>81.2</td><td>73.2</td><td>84.8</td></tr><tr><td>lt-en</td><td>62.4</td><td>62.5</td><td>55.6</td><td>54.3</td><td>56.7</td><td>-</td><td>58.6</td></tr><tr><td>ro-en</td><td>81.4</td><td>83.7</td><td>75.7</td><td>78.3</td><td>79.0</td><td>66.2</td><td>83.6</td></tr><tr><td></td><td>mean</td><td>70.8</td><td>73.5</td><td>67.4</td><td>68.2</td><td>67.2</td><td>62.6</td><td>70.0</td></tr><tr><td rowspan="10">Base-raw</td><td>en-cs</td><td>49.8</td><td>52.4</td><td>43.6</td><td>62.3</td><td>60.3</td><td>47.0</td><td>65.2</td></tr><tr><td>en-de</td><td>61.7</td><td>56.9</td><td>51.6</td><td>68.0</td><td>67.3</td><td>63.3</td><td>72.3</td></tr><tr><td>en-fi</td><td>50.4</td><td>46.6</td><td>44.2</td><td>61.1</td><td>61.5</td><td>-</td><td>63.8</td></tr><tr><td>en-fr</td><td>61.3</td><td>60.8</td><td>48.0</td><td>65.4</td><td>64.6</td><td>52.4</td><td>72.3</td></tr><tr><td>en-lt</td><td>42.9</td><td>38.5</td><td>41.4</td><td>46.2</td><td>42.5</td><td>-</td><td>43.0</td></tr><tr><td>en-ro</td><td>48.9</td><td>55.5</td><td>45.4</td><td>66.0</td><td>64.0</td><td>50.6</td><td>69.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>34.2</td><td>44.4</td><td>34.7</td><td>31.4</td><td>37.7</td></tr><tr><td>cs-en</td><td>75.7</td><td>75.3</td><td>64.1</td><td>70.5</td><td>72.5</td><td>64.0</td><td>76.6</td></tr><tr><td>fr-en</td><td>75.3</td><td>77.5</td><td>67.6</td><td>73.3</td><td>75.8</td><td>69.4</td><td>80.3</td></tr><tr><td>lt-en</td><td>58.4</td><td>58.9</td><td>48.7</td><td>51.3</td><td>54.4</td><td>-</td><td>56.6</td></tr><tr><td>ro-en</td><td>74.6</td><td>76.9</td><td>62.9</td><td>72.4</td><td>74.6</td><td>64.4</td><td>79.2</td></tr><tr><td></td><td>mean</td><td>59.9</td><td>59.9</td><td>50.2</td><td>61.9</td><td>61.1</td><td>55.3</td><td>65.1</td></tr><tr><td rowspan="10">Verbose</td><td>en-cs</td><td>77.9</td><td>82.1</td><td>75.2</td><td>71.5</td><td>74.4</td><td>66.8</td><td>77.7</td></tr><tr><td>en-de</td><td>77.3</td><td>79.2</td><td>73.5</td><td>75.5</td><td>76.6</td><td>70.6</td><td>80.0</td></tr><tr><td>en-fi</td><td>77.8</td><td>79.6</td><td>73.5</td><td>75.6</td><td>78.2</td><td>-</td><td>80.9</td></tr><tr><td>en-fr</td><td>78.4</td><td>83.0</td><td>73.5</td><td>73.7</td><td>76.2</td><td>70.9</td><td>80.7</td></tr><tr><td>en-lt</td><td>48.7</td><td>44.4</td><td>60.7</td><td>52.0</td><td>46.2</td><td>-</td><td>44.9</td></tr><tr><td>en-ro</td><td>78.4</td><td>83.9</td><td>72.7</td><td>75.7</td><td>76.7</td><td>69.8</td><td>81.9</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>48.9</td><td>46.1</td><td>36.1</td><td>32.2</td><td>38.7</td></tr><tr><td>cs-en</td><td>72.5</td><td>73.6</td><td>72.4</td><td>72.0</td><td>71.7</td><td>62.6</td><td>73.1</td></tr><tr><td>fr-en</td><td>80.4</td><td>83.2</td><td>75.6</td><td>78.0</td><td>79.2</td><td>73.2</td><td>82.9</td></tr><tr><td>lt-en</td><td>65.9</td><td>67.1</td><td>60.1</td><td>59.5</td><td>61.4</td><td>-</td><td>63.7</td></tr><tr><td>ro-en</td><td>80.4</td><td>82.8</td><td>73.9</td><td>78.9</td><td>79.7</td><td>67.2</td><td>83.7</td></tr><tr><td></td><td>mean</td><td>73.8</td><td>75.9</td><td>69.1</td><td>69.0</td><td>68.8</td><td>64.2</td><td>71.7</td></tr><tr><td rowspan="10">Label</td><td>en-cs</td><td>74.1</td><td>81.0</td><td>72.0</td><td>72.3</td><td>70.9</td><td>65.6</td><td>75.1</td></tr><tr><td>en-de</td><td>76.2</td><td>80.6</td><td>73.8</td><td>75.9</td><td>77.1</td><td>70.4</td><td>79.8</td></tr><tr><td>en-fi</td><td>72.1</td><td>77.3</td><td>70.9</td><td>75.0</td><td>76.4</td><td>-</td><td>78.3</td></tr><tr><td>en-fr</td><td>79.2</td><td>83.7</td><td>73.7</td><td>75.2</td><td>76.7</td><td>69.7</td><td>81.2</td></tr><tr><td>en-lt</td><td>49.7</td><td>45.5</td><td>60.1</td><td>51.7</td><td>46.8</td><td>-</td><td>44.0</td></tr><tr><td>en-ro</td><td>73.9</td><td>84.2</td><td>71.2</td><td>74.9</td><td>75.2</td><td>67.2</td><td>79.6</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>49.3</td><td>47.9</td><td>38.5</td><td>29.3</td><td>39.6</td></tr><tr><td>cs-en</td><td>83.1</td><td>82.4</td><td>81.3</td><td>77.4</td><td>79.0</td><td>68.6</td><td>81.9</td></tr><tr><td>fr-en</td><td>83.1</td><td>85.1</td><td>80.7</td><td>79.7</td><td>81.8</td><td>73.9</td><td>85.3</td></tr><tr><td>lt-en</td><td>65.5</td><td>65.0</td><td>63.4</td><td>54.2</td><td>58.1</td><td>-</td><td>59.7</td></tr><tr><td>ro-en</td><td>82.1</td><td>84.0</td><td>80.0</td><td>79.3</td><td>80.3</td><td>68.1</td><td>84.4</td></tr><tr><td></td><td>mean</td><td>73.9</td><td>76.9</td><td>70.6</td><td>69.4</td><td>69.2</td><td>64.1</td><td>71.7</td></tr><tr><td rowspan="10">Label-R</td><td>en-cs</td><td>72.6</td><td>80.5</td><td>72.9</td><td>72.0</td><td>70.9</td><td>65.7</td><td>74.5</td></tr><tr><td>en-de</td><td>75.6</td><td>80.4</td><td>73.4</td><td>75.7</td><td>76.9</td><td>70.2</td><td>79.8</td></tr><tr><td>en-fi</td><td>72.0</td><td>77.6</td><td>71.4</td><td>74.2</td><td>75.7</td><td>-</td><td>79.2</td></tr><tr><td>en-fr</td><td>78.8</td><td>83.3</td><td>73.9</td><td>74.8</td><td>76.7</td><td>69.4</td><td>81.0</td></tr><tr><td>en-lt</td><td>49.6</td><td>45.8</td><td>59.4</td><td>51.9</td><td>46.3</td><td>-</td><td>45.4</td></tr><tr><td>en-ro</td><td>72.1</td><td>82.9</td><td>71.6</td><td>74.4</td><td>75.5</td><td>66.5</td><td>80.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>47.9</td><td>47.4</td><td>38.4</td><td>30.1</td><td>39.3</td></tr><tr><td>cs-en</td><td>82.9</td><td>82.5</td><td>80.1</td><td>77.2</td><td>78.8</td><td>66.0</td><td>81.6</td></tr><tr><td>fr-en</td><td>83.0</td><td>84.6</td><td>79.8</td><td>79.4</td><td>81.8</td><td>73.0</td><td>85.2</td></tr><tr><td>lt-en</td><td>64.0</td><td>62.7</td><td>60.7</td><td>54.6</td><td>57.0</td><td>-</td><td>58.7</td></tr><tr><td>ro-en</td><td>82.0</td><td>83.6</td><td>78.8</td><td>78.8</td><td>80.2</td><td>66.2</td><td>84.0</td></tr><tr><td></td><td>mean</td><td>73.3</td><td>76.4</td><td>70.0</td><td>69.1</td><td>68.9</td><td>63.4</td><td>71.7</td></tr></tbody></table>

<table><caption>Table 17: COMET scores for topic-guided and random keyword prompts.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="8">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">Keywords-10 (Seen)</td><td>en-cs</td><td>78.6</td><td>83.3</td><td>74.8</td><td>73.0</td><td>74.8</td><td>65.8</td><td>76.9</td></tr><tr><td>en-de</td><td>77.6</td><td>81.2</td><td>75.8</td><td>75.9</td><td>77.4</td><td>70.1</td><td>80.0</td></tr><tr><td>en-fi</td><td>75.5</td><td>79.5</td><td>73.8</td><td>73.8</td><td>76.8</td><td>-</td><td>80.5</td></tr><tr><td>en-fr</td><td>80.4</td><td>84.2</td><td>76.1</td><td>75.7</td><td>77.3</td><td>69.8</td><td>81.7</td></tr><tr><td>en-lt</td><td>48.8</td><td>44.9</td><td>61.2</td><td>51.1</td><td>45.4</td><td>-</td><td>43.8</td></tr><tr><td>en-ro</td><td>76.7</td><td>84.3</td><td>74.8</td><td>75.9</td><td>76.5</td><td>67.2</td><td>80.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>46.1</td><td>47.1</td><td>37.3</td><td>30.8</td><td>37.9</td></tr><tr><td>cs-en</td><td>83.2</td><td>82.9</td><td>79.5</td><td>77.5</td><td>79.3</td><td>66.3</td><td>81.6</td></tr><tr><td>fr-en</td><td>83.3</td><td>85.0</td><td>80.1</td><td>79.3</td><td>81.7</td><td>72.9</td><td>85.2</td></tr><tr><td>lt-en</td><td>67.4</td><td>67.1</td><td>63.7</td><td>56.9</td><td>59.4</td><td>-</td><td>60.4</td></tr><tr><td>ro-en</td><td>81.9</td><td>84.2</td><td>78.5</td><td>79.6</td><td>80.5</td><td>65.7</td><td>84.2</td></tr><tr><td></td><td>mean</td><td>75.3</td><td>77.7</td><td>71.3</td><td>69.6</td><td>69.7</td><td>63.6</td><td>72.1</td></tr><tr><td rowspan="10">Keywords-30 (Seen)</td><td>en-cs</td><td>78.2</td><td>83.0</td><td>75.6</td><td>72.7</td><td>73.9</td><td>65.9</td><td>76.9</td></tr><tr><td>en-de</td><td>77.5</td><td>81.3</td><td>76.3</td><td>75.8</td><td>77.7</td><td>70.2</td><td>80.1</td></tr><tr><td>en-fi</td><td>75.9</td><td>79.2</td><td>73.0</td><td>74.3</td><td>77.0</td><td>-</td><td>80.7</td></tr><tr><td>en-fr</td><td>80.2</td><td>84.6</td><td>75.7</td><td>75.7</td><td>77.6</td><td>69.8</td><td>81.4</td></tr><tr><td>en-lt</td><td>48.4</td><td>44.6</td><td>60.1</td><td>50.4</td><td>44.9</td><td>-</td><td>43.5</td></tr><tr><td>en-ro</td><td>77.1</td><td>84.9</td><td>74.7</td><td>75.7</td><td>77.2</td><td>68.1</td><td>81.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>45.5</td><td>46.0</td><td>36.8</td><td>30.3</td><td>38.1</td></tr><tr><td>cs-en</td><td>83.1</td><td>82.9</td><td>80.0</td><td>77.5</td><td>79.2</td><td>64.9</td><td>81.9</td></tr><tr><td>fr-en</td><td>83.1</td><td>85.0</td><td>79.4</td><td>79.3</td><td>81.6</td><td>72.9</td><td>85.0</td></tr><tr><td>lt-en</td><td>67.8</td><td>67.4</td><td>64.0</td><td>57.5</td><td>59.7</td><td>-</td><td>60.1</td></tr><tr><td>ro-en</td><td>82.0</td><td>84.5</td><td>77.8</td><td>79.7</td><td>80.6</td><td>65.4</td><td>84.4</td></tr><tr><td></td><td>mean</td><td>75.3</td><td>77.7</td><td>71.1</td><td>69.5</td><td>69.7</td><td>63.4</td><td>72.1</td></tr><tr><td rowspan="10">Keywords-10 (Seen, Random Topic)</td><td>en-cs</td><td>76.6</td><td>83.5</td><td>74.0</td><td>73.3</td><td>73.9</td><td>65.4</td><td>76.0</td></tr><tr><td>en-de</td><td>76.5</td><td>81.3</td><td>75.4</td><td>76.0</td><td>77.3</td><td>70.2</td><td>80.1</td></tr><tr><td>en-fi</td><td>73.9</td><td>79.3</td><td>72.8</td><td>74.2</td><td>77.2</td><td>-</td><td>80.4</td></tr><tr><td>en-fr</td><td>79.7</td><td>84.0</td><td>75.8</td><td>75.5</td><td>77.3</td><td>69.8</td><td>81.6</td></tr><tr><td>en-lt</td><td>47.8</td><td>44.4</td><td>60.6</td><td>51.7</td><td>46.0</td><td>-</td><td>44.0</td></tr><tr><td>en-ro</td><td>75.4</td><td>84.7</td><td>73.0</td><td>75.6</td><td>76.4</td><td>67.3</td><td>80.0</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>45.7</td><td>47.3</td><td>37.9</td><td>30.5</td><td>38.3</td></tr><tr><td>cs-en</td><td>83.1</td><td>82.6</td><td>80.2</td><td>77.4</td><td>78.7</td><td>66.2</td><td>81.7</td></tr><tr><td>fr-en</td><td>83.0</td><td>85.0</td><td>79.8</td><td>79.4</td><td>81.6</td><td>72.9</td><td>85.0</td></tr><tr><td>lt-en</td><td>64.4</td><td>64.8</td><td>61.8</td><td>54.9</td><td>57.6</td><td>-</td><td>59.2</td></tr><tr><td>ro-en</td><td>82.1</td><td>84.2</td><td>78.6</td><td>79.7</td><td>80.2</td><td>65.4</td><td>84.3</td></tr><tr><td></td><td>mean</td><td>74.2</td><td>77.4</td><td>70.7</td><td>69.5</td><td>69.5</td><td>63.5</td><td>71.9</td></tr><tr><td rowspan="10">Random Keywords-10 (Seen)</td><td>en-cs</td><td>76.7</td><td>83.0</td><td>75.8</td><td>73.5</td><td>74.2</td><td>66.3</td><td>77.0</td></tr><tr><td>en-de</td><td>76.9</td><td>81.2</td><td>75.6</td><td>76.2</td><td>77.2</td><td>70.4</td><td>80.2</td></tr><tr><td>en-fi</td><td>73.4</td><td>79.3</td><td>74.0</td><td>75.2</td><td>77.2</td><td>-</td><td>80.9</td></tr><tr><td>en-fr</td><td>80.1</td><td>84.3</td><td>75.4</td><td>75.4</td><td>77.4</td><td>69.9</td><td>81.7</td></tr><tr><td>en-lt</td><td>46.9</td><td>44.6</td><td>60.8</td><td>51.5</td><td>45.3</td><td>-</td><td>43.9</td></tr><tr><td>en-ro</td><td>75.4</td><td>84.3</td><td>73.7</td><td>75.9</td><td>76.4</td><td>67.4</td><td>80.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>46.1</td><td>47.5</td><td>37.9</td><td>30.3</td><td>37.8</td></tr><tr><td>cs-en</td><td>83.0</td><td>82.6</td><td>79.8</td><td>77.5</td><td>79.2</td><td>66.6</td><td>81.9</td></tr><tr><td>fr-en</td><td>83.3</td><td>85.1</td><td>79.3</td><td>79.3</td><td>81.7</td><td>73.3</td><td>85.2</td></tr><tr><td>lt-en</td><td>66.3</td><td>65.3</td><td>61.9</td><td>56.1</td><td>58.8</td><td>-</td><td>60.7</td></tr><tr><td>ro-en</td><td>82.3</td><td>84.1</td><td>78.5</td><td>79.6</td><td>80.5</td><td>66.4</td><td>84.5</td></tr><tr><td></td><td>mean</td><td>74.4</td><td>77.4</td><td>71.0</td><td>69.8</td><td>69.6</td><td>63.8</td><td>72.2</td></tr></tbody></table>
<table><caption>Table 18: COMET scores for various topic-guided few-shot example experiments.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">Fewshot (1, Seen, 500 topics)</td><td>en-cs</td><td>79.0</td><td>84.1</td><td>77.7</td><td>71.6</td><td>73.6</td><td>65.0</td><td>76.7</td></tr><tr><td>en-de</td><td>78.1</td><td>81.3</td><td>75.9</td><td>74.8</td><td>76.3</td><td>69.0</td><td>79.8</td></tr><tr><td>en-fi</td><td>76.1</td><td>81.4</td><td>75.8</td><td>74.7</td><td>77.4</td><td>-</td><td>79.7</td></tr><tr><td>en-fr</td><td>80.1</td><td>85.2</td><td>75.4</td><td>75.2</td><td>76.1</td><td>69.8</td><td>80.6</td></tr><tr><td>en-lt</td><td>47.8</td><td>46.0</td><td>62.0</td><td>51.7</td><td>45.2</td><td>-</td><td>42.8</td></tr><tr><td>en-ro</td><td>78.5</td><td>84.6</td><td>75.5</td><td>74.7</td><td>75.5</td><td>67.6</td><td>80.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>49.6</td><td>47.9</td><td>36.1</td><td>30.2</td><td>39.0</td></tr><tr><td>cs-en</td><td>82.2</td><td>83.5</td><td>80.3</td><td>77.3</td><td>79.1</td><td>66.9</td><td>81.5</td></tr><tr><td>fr-en</td><td>81.5</td><td>85.6</td><td>80.3</td><td>79.4</td><td>81.7</td><td>73.1</td><td>85.3</td></tr><tr><td>lt-en</td><td>64.0</td><td>68.3</td><td>62.4</td><td>55.8</td><td>57.3</td><td>-</td><td>58.1</td></tr><tr><td>ro-en</td><td>81.2</td><td>85.1</td><td>80.0</td><td>79.4</td><td>80.0</td><td>66.1</td><td>83.8</td></tr><tr><td></td><td>mean</td><td>74.8</td><td>78.5</td><td>72.3</td><td>69.3</td><td>68.9</td><td>63.5</td><td>71.6</td></tr><tr><td rowspan="10">Fewshot (3, Seen, 500 topics)</td><td>en-cs</td><td>80.5</td><td>84.9</td><td>80.8</td><td>73.7</td><td>74.9</td><td>66.4</td><td>77.2</td></tr><tr><td>en-de</td><td>79.1</td><td>82.5</td><td>77.6</td><td>75.6</td><td>77.2</td><td>69.8</td><td>80.7</td></tr><tr><td>en-fi</td><td>79.3</td><td>82.5</td><td>78.3</td><td>76.0</td><td>78.5</td><td>-</td><td>81.5</td></tr><tr><td>en-fr</td><td>81.2</td><td>85.4</td><td>77.9</td><td>75.9</td><td>77.3</td><td>70.2</td><td>81.3</td></tr><tr><td>en-lt</td><td>49.4</td><td>46.5</td><td>64.6</td><td>52.2</td><td>45.3</td><td>-</td><td>44.8</td></tr><tr><td>en-ro</td><td>80.1</td><td>85.9</td><td>77.8</td><td>75.9</td><td>76.6</td><td>68.2</td><td>80.7</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>51.8</td><td>48.7</td><td>36.5</td><td>31.3</td><td>39.4</td></tr><tr><td>cs-en</td><td>83.9</td><td>84.3</td><td>82.2</td><td>77.9</td><td>79.2</td><td>66.7</td><td>81.8</td></tr><tr><td>fr-en</td><td>83.9</td><td>86.2</td><td>81.5</td><td>79.8</td><td>82.0</td><td>72.8</td><td>85.3</td></tr><tr><td>lt-en</td><td>70.2</td><td>70.7</td><td>66.2</td><td>58.2</td><td>58.3</td><td>-</td><td>60.1</td></tr><tr><td>ro-en</td><td>82.6</td><td>85.4</td><td>80.9</td><td>79.3</td><td>80.1</td><td>65.3</td><td>84.3</td></tr><tr><td></td><td>mean</td><td>77.0</td><td>79.4</td><td>74.5</td><td>70.3</td><td>69.6</td><td>63.8</td><td>72.5</td></tr><tr><td rowspan="10">Fewshot (5, Seen, 500 topics)</td><td>en-cs</td><td>80.7</td><td>85.1</td><td>81.1</td><td>74.0</td><td>75.6</td><td>66.9</td><td>78.1</td></tr><tr><td>en-de</td><td>79.5</td><td>82.4</td><td>78.2</td><td>75.8</td><td>77.4</td><td>70.3</td><td>80.1</td></tr><tr><td>en-fi</td><td>79.7</td><td>82.6</td><td>79.0</td><td>77.3</td><td>79.0</td><td>-</td><td>81.5</td></tr><tr><td>en-fr</td><td>80.9</td><td>85.5</td><td>78.2</td><td>75.9</td><td>77.4</td><td>70.3</td><td>81.9</td></tr><tr><td>en-lt</td><td>50.4</td><td>46.9</td><td>64.5</td><td>52.0</td><td>45.5</td><td>-</td><td>44.7</td></tr><tr><td>en-ro</td><td>80.4</td><td>85.8</td><td>78.3</td><td>76.1</td><td>77.1</td><td>68.9</td><td>81.2</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>51.4</td><td>49.1</td><td>37.2</td><td>31.6</td><td>39.0</td></tr><tr><td>cs-en</td><td>84.2</td><td>84.3</td><td>82.3</td><td>77.8</td><td>79.3</td><td>66.1</td><td>81.7</td></tr><tr><td>fr-en</td><td>83.9</td><td>86.4</td><td>81.8</td><td>79.9</td><td>82.0</td><td>72.9</td><td>85.4</td></tr><tr><td>lt-en</td><td>70.9</td><td>71.2</td><td>67.9</td><td>58.1</td><td>59.0</td><td>-</td><td>60.3</td></tr><tr><td>ro-en</td><td>82.6</td><td>85.5</td><td>81.2</td><td>79.7</td><td>80.4</td><td>65.6</td><td>84.3</td></tr><tr><td></td><td>mean</td><td>77.3</td><td>79.6</td><td>74.9</td><td>70.5</td><td>70.0</td><td>64.1</td><td>72.6</td></tr><tr><td rowspan="10">Fewshot (3, Seen, 200 topics)</td><td>en-cs</td><td>80.6</td><td>83.0</td><td>79.8</td><td>73.5</td><td>74.3</td><td>67.2</td><td>76.7</td></tr><tr><td>en-de</td><td>79.4</td><td>81.9</td><td>76.3</td><td>75.7</td><td>76.6</td><td>69.9</td><td>79.8</td></tr><tr><td>en-fi</td><td>79.4</td><td>82.4</td><td>77.3</td><td>76.0</td><td>79.0</td><td>-</td><td>80.8</td></tr><tr><td>en-fr</td><td>80.6</td><td>85.2</td><td>76.5</td><td>74.8</td><td>77.6</td><td>70.2</td><td>80.8</td></tr><tr><td>en-lt</td><td>50.9</td><td>46.9</td><td>63.8</td><td>52.5</td><td>45.4</td><td>-</td><td>44.2</td></tr><tr><td>en-ro</td><td>79.8</td><td>85.7</td><td>76.4</td><td>76.0</td><td>76.8</td><td>68.6</td><td>80.7</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>51.2</td><td>48.4</td><td>37.2</td><td>30.7</td><td>38.8</td></tr><tr><td>cs-en</td><td>83.9</td><td>82.7</td><td>77.9</td><td>77.1</td><td>79.1</td><td>65.8</td><td>81.4</td></tr><tr><td>fr-en</td><td>83.5</td><td>85.3</td><td>81.3</td><td>78.9</td><td>81.4</td><td>73.1</td><td>85.0</td></tr><tr><td>lt-en</td><td>71.0</td><td>65.7</td><td>64.6</td><td>54.9</td><td>56.5</td><td>-</td><td>56.9</td></tr><tr><td>ro-en</td><td>82.9</td><td>84.6</td><td>76.4</td><td>78.3</td><td>79.4</td><td>65.4</td><td>83.9</td></tr><tr><td></td><td>mean</td><td>77.2</td><td>78.3</td><td>72.9</td><td>69.6</td><td>69.4</td><td>63.9</td><td>71.7</td></tr><tr><td rowspan="10">Fewshot (3, Seen, 1000 topics)</td><td>en-cs</td><td>80.9</td><td>85.0</td><td>80.2</td><td>73.0</td><td>74.4</td><td>66.2</td><td>77.6</td></tr><tr><td>en-de</td><td>79.0</td><td>82.3</td><td>78.1</td><td>75.6</td><td>76.6</td><td>70.4</td><td>79.7</td></tr><tr><td>en-fi</td><td>79.7</td><td>82.3</td><td>78.3</td><td>75.9</td><td>78.8</td><td>-</td><td>80.8</td></tr><tr><td>en-fr</td><td>80.8</td><td>85.4</td><td>77.5</td><td>75.4</td><td>77.4</td><td>70.1</td><td>81.0</td></tr><tr><td>en-lt</td><td>51.3</td><td>46.9</td><td>64.6</td><td>51.7</td><td>45.6</td><td>-</td><td>43.6</td></tr><tr><td>en-ro</td><td>79.8</td><td>85.8</td><td>77.0</td><td>75.9</td><td>76.6</td><td>68.4</td><td>80.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>51.6</td><td>48.5</td><td>36.7</td><td>30.9</td><td>38.4</td></tr><tr><td>cs-en</td><td>84.1</td><td>84.2</td><td>81.9</td><td>77.6</td><td>79.5</td><td>66.4</td><td>81.6</td></tr><tr><td>fr-en</td><td>83.6</td><td>86.3</td><td>81.3</td><td>79.6</td><td>81.7</td><td>72.9</td><td>85.2</td></tr><tr><td>lt-en</td><td>71.2</td><td>70.9</td><td>64.6</td><td>56.3</td><td>57.8</td><td>-</td><td>59.7</td></tr><tr><td>ro-en</td><td>83.3</td><td>85.7</td><td>80.3</td><td>79.4</td><td>80.0</td><td>65.6</td><td>83.9</td></tr><tr><td></td><td>mean</td><td>77.4</td><td>79.5</td><td>74.1</td><td>69.9</td><td>69.6</td><td>63.9</td><td>72.0</td></tr><tr><td rowspan="10">Fewshot (3, Seen, Random Topic, 500 topics)</td><td>en-cs</td><td>80.2</td><td>83.9</td><td>78.6</td><td>73.1</td><td>74.5</td><td>65.9</td><td>77.2</td></tr><tr><td>en-de</td><td>79.0</td><td>81.5</td><td>76.9</td><td>75.3</td><td>77.1</td><td>69.8</td><td>80.1</td></tr><tr><td>en-fi</td><td>78.6</td><td>80.1</td><td>77.4</td><td>75.4</td><td>78.3</td><td>-</td><td>80.2</td></tr><tr><td>en-fr</td><td>80.5</td><td>84.9</td><td>75.7</td><td>75.1</td><td>76.7</td><td>69.7</td><td>81.4</td></tr><tr><td>en-lt</td><td>49.1</td><td>45.7</td><td>62.2</td><td>50.7</td><td>45.2</td><td>-</td><td>43.2</td></tr><tr><td>en-ro</td><td>79.2</td><td>85.0</td><td>76.0</td><td>75.5</td><td>76.9</td><td>67.6</td><td>80.8</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>49.7</td><td>47.6</td><td>36.2</td><td>30.8</td><td>38.7</td></tr><tr><td>cs-en</td><td>82.2</td><td>81.7</td><td>76.9</td><td>75.7</td><td>77.3</td><td>65.8</td><td>80.2</td></tr><tr><td>fr-en</td><td>82.4</td><td>84.8</td><td>79.1</td><td>78.3</td><td>80.9</td><td>73.1</td><td>84.6</td></tr><tr><td>lt-en</td><td>59.8</td><td>60.7</td><td>52.3</td><td>49.5</td><td>51.5</td><td>-</td><td>53.5</td></tr><tr><td>ro-en</td><td>80.5</td><td>82.5</td><td>75.5</td><td>76.6</td><td>78.6</td><td>65.4</td><td>83.0</td></tr><tr><td></td><td>mean</td><td>75.1</td><td>77.1</td><td>70.9</td><td>68.4</td><td>68.5</td><td>63.5</td><td>71.2</td></tr></tbody></table>

<table><caption>Table 19: COMET scores for few-shot baseline experiments.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">Fewshot (3, Seen, 500 topics)</td><td>en-cs</td><td>78.3</td><td>83.1</td><td>77.8</td><td>73.1</td><td>74.3</td><td>65.3</td><td>76.0</td></tr><tr><td>en-de</td><td>77.0</td><td>80.5</td><td>75.2</td><td>75.3</td><td>76.7</td><td>69.8</td><td>79.4</td></tr><tr><td>en-fi</td><td>76.4</td><td>77.8</td><td>76.7</td><td>75.0</td><td>78.2</td><td>-</td><td>80.5</td></tr><tr><td>en-fr</td><td>80.0</td><td>84.5</td><td>76.0</td><td>75.2</td><td>76.8</td><td>69.5</td><td>81.0</td></tr><tr><td>en-lt</td><td>50.7</td><td>47.3</td><td>63.9</td><td>52.6</td><td>47.6</td><td>-</td><td>45.7</td></tr><tr><td>en-ro</td><td>78.1</td><td>84.9</td><td>75.7</td><td>76.2</td><td>76.9</td><td>68.1</td><td>80.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>51.5</td><td>48.7</td><td>39.5</td><td>31.2</td><td>39.7</td></tr><tr><td>cs-en</td><td>76.4</td><td>73.7</td><td>74.3</td><td>74.0</td><td>71.8</td><td>57.8</td><td>75.2</td></tr><tr><td>fr-en</td><td>76.2</td><td>77.1</td><td>71.7</td><td>73.0</td><td>73.2</td><td>65.7</td><td>77.1</td></tr><tr><td>lt-en</td><td>70.9</td><td>71.6</td><td>69.7</td><td>69.1</td><td>68.1</td><td>-</td><td>72.3</td></tr><tr><td>ro-en</td><td>74.6</td><td>75.0</td><td>73.6</td><td>74.2</td><td>73.0</td><td>61.4</td><td>76.7</td></tr><tr><td></td><td>mean</td><td>73.9</td><td>75.6</td><td>71.5</td><td>69.7</td><td>68.7</td><td>61.1</td><td>71.3</td></tr><tr><td rowspan="10">Fewshot (3, Seen, 500 topics)</td><td>en-cs</td><td>81.9</td><td>86.2</td><td>81.5</td><td>74.1</td><td>76.6</td><td>66.3</td><td>78.7</td></tr><tr><td>en-de</td><td>79.7</td><td>82.7</td><td>78.7</td><td>76.3</td><td>77.6</td><td>71.4</td><td>80.9</td></tr><tr><td>en-fi</td><td>80.4</td><td>82.6</td><td>79.4</td><td>77.1</td><td>79.6</td><td>-</td><td>81.8</td></tr><tr><td>en-fr</td><td>81.3</td><td>86.2</td><td>78.7</td><td>75.8</td><td>77.5</td><td>71.5</td><td>81.6</td></tr><tr><td>en-lt</td><td>51.5</td><td>49.6</td><td>67.3</td><td>52.8</td><td>47.4</td><td>-</td><td>45.0</td></tr><tr><td>en-ro</td><td>81.3</td><td>86.9</td><td>79.1</td><td>77.2</td><td>78.2</td><td>71.1</td><td>81.9</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>53.2</td><td>48.9</td><td>42.3</td><td>33.7</td><td>39.7</td></tr><tr><td>cs-en</td><td>84.4</td><td>84.3</td><td>81.2</td><td>78.1</td><td>79.9</td><td>67.0</td><td>82.3</td></tr><tr><td>fr-en</td><td>83.6</td><td>86.6</td><td>81.7</td><td>79.5</td><td>81.7</td><td>73.7</td><td>85.4</td></tr><tr><td>lt-en</td><td>71.8</td><td>71.6</td><td>67.4</td><td>61.0</td><td>62.9</td><td>-</td><td>65.6</td></tr><tr><td>ro-en</td><td>83.2</td><td>85.8</td><td>81.2</td><td>80.2</td><td>81.1</td><td>67.3</td><td>84.5</td></tr><tr><td></td><td>mean</td><td>77.9</td><td>80.2</td><td>75.4</td><td>71.0</td><td>71.3</td><td>65.2</td><td>73.4</td></tr><tr><td rowspan="10">Fewshot (3, Seen, 500 topics)</td><td>en-cs</td><td>83.9</td><td>87.2</td><td>83.2</td><td>74.6</td><td>78.0</td><td>71.0</td><td>79.2</td></tr><tr><td>en-de</td><td>80.5</td><td>83.2</td><td>79.3</td><td>75.9</td><td>77.9</td><td>73.8</td><td>80.2</td></tr><tr><td>en-fi</td><td>82.3</td><td>86.7</td><td>80.7</td><td>76.9</td><td>81.3</td><td>-</td><td>82.5</td></tr><tr><td>en-fr</td><td>82.3</td><td>86.5</td><td>79.3</td><td>75.6</td><td>77.5</td><td>74.7</td><td>81.7</td></tr><tr><td>en-lt</td><td>63.2</td><td>56.9</td><td>70.4</td><td>58.1</td><td>53.0</td><td>-</td><td>52.1</td></tr><tr><td>en-ro</td><td>83.5</td><td>87.6</td><td>79.9</td><td>78.2</td><td>78.3</td><td>74.0</td><td>82.7</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>52.7</td><td>56.0</td><td>45.3</td><td>36.2</td><td>39.0</td></tr><tr><td>cs-en</td><td>85.0</td><td>84.7</td><td>84.0</td><td>78.2</td><td>79.9</td><td>68.9</td><td>82.5</td></tr><tr><td>fr-en</td><td>84.3</td><td>86.8</td><td>83.3</td><td>79.6</td><td>82.0</td><td>72.3</td><td>85.4</td></tr><tr><td>lt-en</td><td>76.8</td><td>75.9</td><td>73.2</td><td>64.9</td><td>67.3</td><td>-</td><td>69.2</td></tr><tr><td>ro-en</td><td>84.2</td><td>86.3</td><td>82.8</td><td>80.6</td><td>81.1</td><td>69.1</td><td>85.4</td></tr><tr><td></td><td>mean</td><td>80.6</td><td>82.2</td><td>77.2</td><td>72.6</td><td>72.9</td><td>67.5</td><td>74.5</td></tr><tr><td rowspan="10">Fewshot (3, Seen, 500 topics)</td><td>en-cs</td><td>76.2</td><td>82.0</td><td>77.8</td><td>71.9</td><td>73.6</td><td>64.7</td><td>74.6</td></tr><tr><td>en-de</td><td>76.3</td><td>80.6</td><td>76.2</td><td>74.8</td><td>75.9</td><td>68.8</td><td>79.0</td></tr><tr><td>en-fi</td><td>76.3</td><td>78.1</td><td>77.0</td><td>74.5</td><td>77.5</td><td>-</td><td>80.0</td></tr><tr><td>en-fr</td><td>79.6</td><td>85.0</td><td>76.4</td><td>74.5</td><td>76.3</td><td>69.2</td><td>80.3</td></tr><tr><td>en-lt</td><td>55.0</td><td>51.5</td><td>65.2</td><td>54.2</td><td>48.2</td><td>-</td><td>46.5</td></tr><tr><td>en-ro</td><td>77.5</td><td>84.7</td><td>76.4</td><td>75.2</td><td>75.9</td><td>67.3</td><td>80.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>52.9</td><td>49.8</td><td>39.2</td><td>33.2</td><td>40.2</td></tr><tr><td>cs-en</td><td>76.5</td><td>73.7</td><td>73.9</td><td>73.7</td><td>71.8</td><td>58.5</td><td>75.3</td></tr><tr><td>fr-en</td><td>76.1</td><td>77.3</td><td>71.7</td><td>72.6</td><td>73.4</td><td>66.2</td><td>77.2</td></tr><tr><td>lt-en</td><td>70.9</td><td>71.6</td><td>69.7</td><td>69.4</td><td>68.1</td><td>-</td><td>72.4</td></tr><tr><td>ro-en</td><td>74.6</td><td>75.0</td><td>73.5</td><td>74.0</td><td>73.1</td><td>61.7</td><td>76.4</td></tr><tr><td></td><td>mean</td><td>73.9</td><td>76.0</td><td>71.9</td><td>69.5</td><td>68.5</td><td>61.2</td><td>71.1</td></tr><tr><td rowspan="10">Fewshot (3, Seen, 500 topics)</td><td>en-cs</td><td>85.1</td><td>87.3</td><td>83.2</td><td>74.9</td><td>77.6</td><td>72.7</td><td>79.2</td></tr><tr><td>en-de</td><td>81.5</td><td>84.0</td><td>79.8</td><td>75.8</td><td>76.9</td><td>73.4</td><td>80.3</td></tr><tr><td>en-fi</td><td>83.2</td><td>87.4</td><td>81.7</td><td>77.9</td><td>80.9</td><td>-</td><td>82.8</td></tr><tr><td>en-fr</td><td>82.4</td><td>87.0</td><td>79.1</td><td>75.9</td><td>77.9</td><td>75.6</td><td>81.5</td></tr><tr><td>en-lt</td><td>66.1</td><td>58.0</td><td>73.0</td><td>57.6</td><td>54.6</td><td>-</td><td>52.3</td></tr><tr><td>en-ro</td><td>83.7</td><td>87.7</td><td>80.4</td><td>78.1</td><td>78.1</td><td>75.2</td><td>82.3</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>63.3</td><td>58.0</td><td>52.0</td><td>60.4</td><td>48.6</td></tr><tr><td>cs-en</td><td>84.8</td><td>85.0</td><td>84.6</td><td>77.9</td><td>80.3</td><td>69.3</td><td>82.8</td></tr><tr><td>fr-en</td><td>84.6</td><td>86.8</td><td>84.3</td><td>79.6</td><td>81.8</td><td>72.4</td><td>85.2</td></tr><tr><td>lt-en</td><td>77.0</td><td>75.3</td><td>74.9</td><td>61.8</td><td>65.8</td><td>-</td><td>67.9</td></tr><tr><td>ro-en</td><td>83.9</td><td>86.2</td><td>82.8</td><td>80.3</td><td>81.2</td><td>69.3</td><td>85.5</td></tr><tr><td></td><td>mean</td><td>81.2</td><td>82.5</td><td>78.8</td><td>72.4</td><td>73.4</td><td>71.0</td><td>75.3</td></tr><tr><td rowspan="10">Fewshot (3, Seen, Random Topic, 500 topics)</td><td>en-cs</td><td>75.8</td><td>81.0</td><td>75.8</td><td>69.2</td><td>72.0</td><td>63.9</td><td>72.6</td></tr><tr><td>en-de</td><td>76.4</td><td>79.8</td><td>74.6</td><td>73.8</td><td>76.0</td><td>68.9</td><td>78.5</td></tr><tr><td>en-fi</td><td>75.0</td><td>76.9</td><td>75.0</td><td>74.0</td><td>75.7</td><td>-</td><td>79.0</td></tr><tr><td>en-fr</td><td>78.9</td><td>84.0</td><td>74.5</td><td>74.4</td><td>76.3</td><td>68.6</td><td>80.1</td></tr><tr><td>en-lt</td><td>51.3</td><td>47.1</td><td>62.7</td><td>51.8</td><td>46.9</td><td>-</td><td>46.0</td></tr><tr><td>en-ro</td><td>76.7</td><td>83.2</td><td>74.4</td><td>74.5</td><td>74.8</td><td>66.3</td><td>79.3</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>51.5</td><td>48.8</td><td>39.1</td><td>31.6</td><td>40.2</td></tr><tr><td>cs-en</td><td>76.3</td><td>73.2</td><td>73.9</td><td>73.7</td><td>71.7</td><td>57.8</td><td>75.2</td></tr><tr><td>fr-en</td><td>76.0</td><td>76.8</td><td>71.5</td><td>72.4</td><td>72.9</td><td>65.7</td><td>76.9</td></tr><tr><td>lt-en</td><td>70.8</td><td>71.7</td><td>69.5</td><td>69.6</td><td>68.1</td><td>-</td><td>72.3</td></tr><tr><td>ro-en</td><td>74.6</td><td>74.9</td><td>73.4</td><td>74.2</td><td>72.9</td><td>61.3</td><td>76.5</td></tr><tr><td></td><td>mean</td><td>73.2</td><td>74.9</td><td>70.6</td><td>68.8</td><td>67.9</td><td>60.5</td><td>70.6</td></tr></tbody></table>
<table><caption>Table 20: BLEU scores for various zero-shot translation prompts, and for zero-shot NLLB tests.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="8">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">NLLB-1.3B</td><td rowspan="10">en-cs</td><td>32.6</td><td>42.2</td><td>25.6</td><td>25.8</td><td>24.4</td><td>18.1</td><td>24.2</td></tr><tr><td>34.6</td><td>39.8</td><td>24.9</td><td>25.8</td><td>28.3</td><td>30.7</td><td>30.1</td></tr><tr><td>27</td><td>24.8</td><td>22.9</td><td>19.4</td><td>22.8</td><td>-</td><td>21.9</td></tr><tr><td>40.4</td><td>52.1</td><td>36.8</td><td>30</td><td>37.9</td><td>30</td><td>38</td></tr><tr><td>25.3</td><td>37.9</td><td>16.8</td><td>17</td><td>22.5</td><td>-</td><td>21.3</td></tr><tr><td>41.3</td><td>35.5</td><td>26.8</td><td>30.7</td><td>27.4</td><td>23.8</td><td>30.7</td></tr><tr><td>-</td><td>-</td><td>12</td><td>16.5</td><td>4.7</td><td>9.7</td><td>6.9</td></tr><tr><td>46.7</td><td>50</td><td>31.8</td><td>35.2</td><td>31.6</td><td>25.4</td><td>34.1</td></tr><tr><td>46.6</td><td>56.1</td><td>38.3</td><td>33</td><td>38</td><td>24.3</td><td>38.7</td></tr><tr><td>36</td><td>50.2</td><td>25</td><td>30.1</td><td>32.4</td><td>-</td><td>33.3</td></tr><tr><td>49.4</td><td>54.4</td><td>33.9</td><td>41.9</td><td>38.4</td><td>22.7</td><td>42</td></tr><tr><td>mean</td><td>38</td><td>44.3</td><td>26.8</td><td>27.8</td><td>28</td><td>23.1</td><td>29.2</td></tr><tr><td rowspan="10">Base</td><td rowspan="10">en-cs</td><td>8.6</td><td>15.4</td><td>17.0</td><td>8.9</td><td>10.0</td><td>4.0</td><td>11.5</td></tr><tr><td>18.5</td><td>20.2</td><td>16.0</td><td>18.2</td><td>19.6</td><td>11.1</td><td>21.0</td></tr><tr><td>5.3</td><td>5.0</td><td>13.7</td><td>5.0</td><td>6.4</td><td>-</td><td>6.0</td></tr><tr><td>20.8</td><td>28.3</td><td>25.4</td><td>20.5</td><td>25.3</td><td>12.1</td><td>27.2</td></tr><tr><td>4.0</td><td>3.2</td><td>4.4</td><td>1.1</td><td>0.9</td><td>-</td><td>0.9</td></tr><tr><td>9.0</td><td>18.7</td><td>14.6</td><td>11.2</td><td>13.6</td><td>4.9</td><td>17.5</td></tr><tr><td>-</td><td>-</td><td>0.1</td><td>0.6</td><td>0.8</td><td>0.1</td><td>0.1</td></tr><tr><td>29.7</td><td>32.5</td><td>21.6</td><td>24.5</td><td>24.4</td><td>10.5</td><td>27.0</td></tr><tr><td>37.2</td><td>40.2</td><td>34.9</td><td>27.6</td><td>34.9</td><td>16.8</td><td>34.2</td></tr><tr><td>8.6</td><td>11.5</td><td>4.4</td><td>4.2</td><td>7.2</td><td>-</td><td>8.6</td></tr><tr><td>35.6</td><td>37.9</td><td>22.2</td><td>29.6</td><td>29.8</td><td>9.9</td><td>34.2</td></tr><tr><td>mean</td><td>17.7</td><td>21.3</td><td>15.8</td><td>13.8</td><td>15.7</td><td>8.7</td><td>17.1</td></tr><tr><td rowspan="10">Base-raw</td><td rowspan="10">en-cs</td><td>1.5</td><td>4.4</td><td>1.2</td><td>3.0</td><td>2.7</td><td>0.9</td><td>4.6</td></tr><tr><td>5.3</td><td>5.8</td><td>2.4</td><td>8.0</td><td>7.7</td><td>6.0</td><td>10.7</td></tr><tr><td>1.0</td><td>1.1</td><td>1.0</td><td>1.0</td><td>1.8</td><td>-</td><td>1.9</td></tr><tr><td>6.4</td><td>9.6</td><td>4.8</td><td>5.9</td><td>7.7</td><td>2.9</td><td>12.8</td></tr><tr><td>0.7</td><td>1.2</td><td>0.4</td><td>0.3</td><td>0.3</td><td>-</td><td>0.3</td></tr><tr><td>1.9</td><td>5.9</td><td>1.3</td><td>4.7</td><td>5.0</td><td>1.4</td><td>7.3</td></tr><tr><td>-</td><td>-</td><td>0.1</td><td>0.5</td><td>0.6</td><td>0.1</td><td>0.1</td></tr><tr><td>19.9</td><td>22.8</td><td>6.3</td><td>23.2</td><td>22.0</td><td>7.9</td><td>27.0</td></tr><tr><td>21.6</td><td>26.2</td><td>14.8</td><td>25.4</td><td>29.3</td><td>11.5</td><td>31.8</td></tr><tr><td>6.3</td><td>8.6</td><td>1.8</td><td>3.6</td><td>6.3</td><td>-</td><td>7.4</td></tr><tr><td>22.5</td><td>25.5</td><td>5.6</td><td>23.2</td><td>27.0</td><td>9.7</td><td>32.4</td></tr><tr><td>mean</td><td>8.7</td><td>11.1</td><td>3.6</td><td>9.0</td><td>10.0</td><td>5.1</td><td>12.4</td></tr><tr><td rowspan="10">Verbose</td><td rowspan="10">en-cs</td><td>15.7</td><td>18.6</td><td>15.6</td><td>7.6</td><td>10.7</td><td>4.0</td><td>13.0</td></tr><tr><td>22.2</td><td>24.0</td><td>18.2</td><td>14.8</td><td>19.8</td><td>11.6</td><td>20.9</td></tr><tr><td>11.2</td><td>7.4</td><td>14.0</td><td>5.2</td><td>10.4</td><td>-</td><td>10.8</td></tr><tr><td>28.1</td><td>33.3</td><td>32.8</td><td>17.7</td><td>25.6</td><td>11.3</td><td>27.3</td></tr><tr><td>3.9</td><td>4.2</td><td>5.0</td><td>0.8</td><td>1.1</td><td>-</td><td>1.2</td></tr><tr><td>20.7</td><td>21.2</td><td>17.0</td><td>12.2</td><td>15.3</td><td>4.5</td><td>19.1</td></tr><tr><td>-</td><td>-</td><td>0.6</td><td>0.5</td><td>0.4</td><td>0.0</td><td>0.2</td></tr><tr><td>20.6</td><td>25.6</td><td>14.5</td><td>13.6</td><td>16.6</td><td>7.3</td><td>19.1</td></tr><tr><td>33.1</td><td>37.6</td><td>31.6</td><td>22.9</td><td>31.2</td><td>15.2</td><td>31.2</td></tr><tr><td>10.7</td><td>13.6</td><td>7.4</td><td>4.9</td><td>8.3</td><td>-</td><td>9.0</td></tr><tr><td>33.4</td><td>34.9</td><td>21.7</td><td>24.9</td><td>27.3</td><td>10.7</td><td>32.6</td></tr><tr><td>mean</td><td>20.0</td><td>22.0</td><td>16.2</td><td>11.4</td><td>15.2</td><td>8.1</td><td>16.8</td></tr><tr><td rowspan="10">Label</td><td rowspan="10">en-cs</td><td>13.8</td><td>19.4</td><td>20.5</td><td>12.1</td><td>11.2</td><td>4.4</td><td>13.7</td></tr><tr><td>22.8</td><td>25.5</td><td>19.0</td><td>18.7</td><td>22.1</td><td>12.5</td><td>21.7</td></tr><tr><td>9.0</td><td>7.5</td><td>16.6</td><td>7.1</td><td>10.1</td><td>-</td><td>11.6</td></tr><tr><td>28.8</td><td>33.7</td><td>31.7</td><td>22.7</td><td>29.7</td><td>13.2</td><td>29.2</td></tr><tr><td>4.2</td><td>4.2</td><td>5.3</td><td>1.1</td><td>0.9</td><td>-</td><td>1.2</td></tr><tr><td>16.3</td><td>22.7</td><td>15.5</td><td>13.8</td><td>16.4</td><td>5.2</td><td>20.3</td></tr><tr><td>-</td><td>-</td><td>1.4</td><td>0.6</td><td>0.6</td><td>0.0</td><td>0.1</td></tr><tr><td>33.2</td><td>37.3</td><td>31.4</td><td>26.1</td><td>26.2</td><td>14.8</td><td>28.1</td></tr><tr><td>39.2</td><td>43.4</td><td>37.3</td><td>28.8</td><td>35.9</td><td>18.3</td><td>35.6</td></tr><tr><td>12.7</td><td>14.7</td><td>11.7</td><td>5.1</td><td>8.3</td><td>-</td><td>8.7</td></tr><tr><td>39.0</td><td>42.1</td><td>31.1</td><td>31.7</td><td>30.9</td><td>12.2</td><td>34.7</td></tr><tr><td>mean</td><td>21.9</td><td>25.1</td><td>20.1</td><td>15.3</td><td>17.5</td><td>10.1</td><td>18.6</td></tr><tr><td rowspan="10">Label-R</td><td rowspan="10">en-cs</td><td>12.9</td><td>18.9</td><td>19.4</td><td>11.2</td><td>12.3</td><td>4.3</td><td>12.5</td></tr><tr><td>22.1</td><td>25.5</td><td>18.1</td><td>18.1</td><td>21.1</td><td>12.3</td><td>22.4</td></tr><tr><td>9.1</td><td>7.9</td><td>17.3</td><td>6.5</td><td>10.0</td><td>-</td><td>11.1</td></tr><tr><td>28.0</td><td>33.3</td><td>30.8</td><td>21.5</td><td>28.9</td><td>12.8</td><td>30.1</td></tr><tr><td>4.3</td><td>4.2</td><td>5.2</td><td>0.7</td><td>0.9</td><td>-</td><td>1.4</td></tr><tr><td>15.1</td><td>21.5</td><td>16.2</td><td>13.7</td><td>16.6</td><td>5.0</td><td>19.5</td></tr><tr><td>-</td><td>-</td><td>0.7</td><td>0.6</td><td>0.5</td><td>0.0</td><td>0.1</td></tr><tr><td>33.1</td><td>35.3</td><td>29.1</td><td>25.3</td><td>25.4</td><td>9.6</td><td>27.2</td></tr><tr><td>38.5</td><td>41.0</td><td>37.8</td><td>28.1</td><td>35.6</td><td>16.3</td><td>34.8</td></tr><tr><td>11.2</td><td>13.5</td><td>11.2</td><td>4.8</td><td>7.6</td><td>-</td><td>8.5</td></tr><tr><td>38.3</td><td>39.0</td><td>27.5</td><td>30.8</td><td>30.6</td><td>9.5</td><td>34.9</td></tr><tr><td>mean</td><td>21.3</td><td>24.0</td><td>19.4</td><td>14.7</td><td>17.2</td><td>8.7</td><td>18.4</td></tr></tbody></table>

<table><caption>Table 21: BLEU scores for topic-guided and random keyword prompts.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="8">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">Keywords-10<br>(Seen)</td><td rowspan="10">en-cs</td><td>16.4</td><td>21.8</td><td>13.0</td><td>11.3</td><td>14.2</td><td>4.2</td><td>15.2</td></tr><tr><td>23.7</td><td>26.4</td><td>18.7</td><td>18.4</td><td>21.6</td><td>12.3</td><td>22.4</td></tr><tr><td>10.6</td><td>8.0</td><td>13.4</td><td>6.3</td><td>11.6</td><td>-</td><td>12.6</td></tr><tr><td>31.1</td><td>34.8</td><td>33.9</td><td>22.8</td><td>29.4</td><td>13.0</td><td>30.8</td></tr><tr><td>3.4</td><td>4.3</td><td>5.4</td><td>1.0</td><td>0.9</td><td>-</td><td>1.3</td></tr><tr><td>20.9</td><td>23.5</td><td>21.0</td><td>14.5</td><td>17.3</td><td>5.1</td><td>20.2</td></tr><tr><td>-</td><td>-</td><td>0.5</td><td>0.8</td><td>0.5</td><td>0.1</td><td>0.1</td></tr><tr><td>32.7</td><td>35.9</td><td>22.6</td><td>26.2</td><td>26.3</td><td>10.1</td><td>27.2</td></tr><tr><td>38.8</td><td>42.0</td><td>35.1</td><td>28.2</td><td>36.1</td><td>16.5</td><td>35.1</td></tr><tr><td>11.3</td><td>16.1</td><td>8.9</td><td>5.7</td><td>8.6</td><td>-</td><td>9.1</td></tr><tr><td>38.9</td><td>41.1</td><td>27.9</td><td>32.6</td><td>31.0</td><td>10.2</td><td>35.7</td></tr><tr><td>mean</td><td>22.8</td><td>25.4</td><td>18.2</td><td>15.3</td><td>18.0</td><td>8.9</td><td>19.1</td></tr><tr><td rowspan="10">Keywords-30<br>(Seen)</td><td rowspan="10">en-cs</td><td>16.5</td><td>21.2</td><td>13.7</td><td>12.0</td><td>13.2</td><td>4.3</td><td>15.8</td></tr><tr><td>22.9</td><td>26.4</td><td>19.4</td><td>18.5</td><td>23.5</td><td>12.1</td><td>23.1</td></tr><tr><td>11.2</td><td>7.9</td><td>9.4</td><td>6.5</td><td>10.8</td><td>-</td><td>12.9</td></tr><tr><td>30.8</td><td>35.2</td><td>34.0</td><td>22.9</td><td>30.5</td><td>12.7</td><td>30.2</td></tr><tr><td>3.6</td><td>3.9</td><td>4.0</td><td>0.7</td><td>0.9</td><td>-</td><td>1.1</td></tr><tr><td>21.0</td><td>24.1</td><td>20.5</td><td>14.0</td><td>19.4</td><td>5.3</td><td>21.0</td></tr><tr><td>-</td><td>-</td><td>0.6</td><td>0.5</td><td>0.5</td><td>0.0</td><td>0.2</td></tr><tr><td>32.9</td><td>37.5</td><td>23.8</td><td>25.5</td><td>26.1</td><td>9.3</td><td>27.2</td></tr><tr><td>37.6</td><td>41.3</td><td>34.4</td><td>28.4</td><td>35.6</td><td>16.4</td><td>34.9</td></tr><tr><td>11.5</td><td>14.6</td><td>6.4</td><td>5.9</td><td>9.1</td><td>-</td><td>9.2</td></tr><tr><td>39.9</td><td>40.6</td><td>21.6</td><td>32.1</td><td>30.9</td><td>9.8</td><td>35.1</td></tr><tr><td>mean</td><td>22.8</td><td>25.3</td><td>17.1</td><td>15.2</td><td>18.2</td><td>8.7</td><td>19.2</td></tr><tr><td rowspan="10">Keywords-10<br>(Seen,<br>Random Topic)</td><td rowspan="10">en-cs</td><td>15.4</td><td>21.9</td><td>13.7</td><td>12.3</td><td>12.1</td><td>4.4</td><td>13.9</td></tr><tr><td>22.1</td><td>26.3</td><td>18.4</td><td>18.6</td><td>22.3</td><td>12.0</td><td>22.1</td></tr><tr><td>9.8</td><td>8.3</td><td>14.5</td><td>6.5</td><td>11.5</td><td>-</td><td>13.1</td></tr><tr><td>29.2</td><td>34.7</td><td>33.3</td><td>24.0</td><td>30.5</td><td>12.5</td><td>31.5</td></tr><tr><td>3.9</td><td>4.2</td><td>5.7</td><td>1.2</td><td>1.1</td><td>-</td><td>1.3</td></tr><tr><td>19.5</td><td>23.3</td><td>17.9</td><td>14.8</td><td>17.6</td><td>5.1</td><td>19.8</td></tr><tr><td>-</td><td>-</td><td>0.5</td><td>0.7</td><td>0.5</td><td>0.0</td><td>0.1</td></tr><tr><td>32.8</td><td>35.7</td><td>22.8</td><td>25.9</td><td>26.1</td><td>9.9</td><td>27.7</td></tr><tr><td>38.6</td><td>41.9</td><td>33.9</td><td>28.3</td><td>35.8</td><td>16.2</td><td>35.3</td></tr><tr><td>10.7</td><td>14.0</td><td>6.4</td><td>4.9</td><td>8.4</td><td>-</td><td>8.6</td></tr><tr><td>38.5</td><td>40.0</td><td>28.0</td><td>32.2</td><td>30.9</td><td>9.2</td><td>34.8</td></tr><tr><td>mean</td><td>22.0</td><td>25.0</td><td>17.7</td><td>15.4</td><td>17.9</td><td>8.7</td><td>18.9</td></tr><tr><td rowspan="10">Random<br>Keywords-10<br>(Seen)</td><td rowspan="10">en-cs</td><td>15.5</td><td>21.7</td><td>19.1</td><td>11.6</td><td>13.8</td><td>4.6</td><td>14.0</td></tr><tr><td>23.5</td><td>26.3</td><td>20.2</td><td>18.5</td><td>22.5</td><td>12.3</td><td>22.2</td></tr><tr><td>10.2</td><td>7.9</td><td>18.6</td><td>6.4</td><td>11.2</td><td>-</td><td>14.2</td></tr><tr><td>30.0</td><td>35.0</td><td>32.6</td><td>23.5</td><td>30.5</td><td>12.8</td><td>31.0</td></tr><tr><td>2.9</td><td>3.9</td><td>4.7</td><td>1.1</td><td>1.1</td><td>-</td><td>1.5</td></tr><tr><td>19.5</td><td>23.6</td><td>19.3</td><td>15.9</td><td>17.8</td><td>5.0</td><td>20.2</td></tr><tr><td>-</td><td>-</td><td>0.4</td><td>0.6</td><td>0.5</td><td>0.0</td><td>0.2</td></tr><tr><td>33.0</td><td>36.0</td><td>27.5</td><td>25.1</td><td>26.7</td><td>10.2</td><td>27.9</td></tr><tr><td>37.9</td><td>42.4</td><td>31.4</td><td>28.0</td><td>35.9</td><td>17.0</td><td>35.1</td></tr><tr><td>12.5</td><td>13.6</td><td>7.1</td><td>5.7</td><td>7.6</td><td>-</td><td>9.7</td></tr><tr><td>38.9</td><td>39.8</td><td>25.9</td><td>32.0</td><td>31.3</td><td>10.0</td><td>35.2</td></tr><tr><td>mean</td><td>22.4</td><td>25.0</td><td>18.8</td><td>15.3</td><td>18.1</td><td>9.0</td><td>19.2</td></tr></tbody></table>
<table><caption>Table 22: BLEU scores for various topic-guided fewshot example experiments.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">Fewshot<br>(1, Seen,<br>500 topics)</td><td>en-cs</td><td>16.9</td><td>21.5</td><td>23.0</td><td>11.0</td><td>13.6</td><td>4.3</td><td>13.6</td></tr><tr><td>en-de</td><td>24.7</td><td>25.7</td><td>20.7</td><td>17.6</td><td>21.2</td><td>11.3</td><td>22.0</td></tr><tr><td>en-fi</td><td>11.3</td><td>9.0</td><td>17.3</td><td>6.6</td><td>12.8</td><td>-</td><td>13.1</td></tr><tr><td>en-fr</td><td>30.3</td><td>35.3</td><td>33.5</td><td>23.1</td><td>29.2</td><td>12.9</td><td>29.0</td></tr><tr><td>en-lt</td><td>4.8</td><td>4.9</td><td>6.6</td><td>0.8</td><td>1.0</td><td>-</td><td>1.1</td></tr><tr><td>en-ro</td><td>22.8</td><td>23.3</td><td>21.4</td><td>14.3</td><td>17.0</td><td>5.0</td><td>20.2</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>0.7</td><td>0.7</td><td>0.3</td><td>0.0</td><td>0.2</td></tr><tr><td>cs-en</td><td>33.7</td><td>38.4</td><td>27.8</td><td>26.1</td><td>25.8</td><td>10.9</td><td>27.5</td></tr><tr><td>fr-en</td><td>37.6</td><td>44.6</td><td>36.7</td><td>28.7</td><td>35.8</td><td>16.5</td><td>35.3</td></tr><tr><td>lt-en</td><td>12.7</td><td>17.5</td><td>11.0</td><td>5.2</td><td>8.1</td><td>-</td><td>7.8</td></tr><tr><td>ro-en</td><td>40.3</td><td>43.7</td><td>32.0</td><td>31.6</td><td>30.1</td><td>10.1</td><td>34.9</td></tr><tr><td></td><td>mean</td><td>23.5</td><td>26.4</td><td>21.0</td><td>15.1</td><td>17.7</td><td>8.9</td><td>18.6</td></tr><tr><td rowspan="10">Fewshot<br>(3, Seen,<br>500 topics)</td><td>en-cs</td><td>18.1</td><td>22.3</td><td>24.8</td><td>12.2</td><td>13.5</td><td>4.2</td><td>15.4</td></tr><tr><td>en-de</td><td>25.9</td><td>27.0</td><td>21.7</td><td>19.1</td><td>23.4</td><td>11.9</td><td>24.0</td></tr><tr><td>en-fi</td><td>13.2</td><td>9.3</td><td>20.9</td><td>7.5</td><td>13.4</td><td>-</td><td>13.4</td></tr><tr><td>en-fr</td><td>32.0</td><td>36.7</td><td>35.2</td><td>23.8</td><td>29.2</td><td>12.6</td><td>30.8</td></tr><tr><td>en-lt</td><td>5.2</td><td>4.9</td><td>7.8</td><td>1.2</td><td>1.3</td><td>-</td><td>1.7</td></tr><tr><td>en-ro</td><td>25.0</td><td>26.4</td><td>23.5</td><td>15.4</td><td>17.1</td><td>5.5</td><td>21.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.2</td><td>1.0</td><td>0.5</td><td>0.0</td><td>0.3</td></tr><tr><td>cs-en</td><td>36.5</td><td>39.6</td><td>32.5</td><td>27.4</td><td>25.5</td><td>10.6</td><td>28.0</td></tr><tr><td>fr-en</td><td>41.7</td><td>46.4</td><td>39.7</td><td>29.7</td><td>35.7</td><td>16.7</td><td>35.4</td></tr><tr><td>lt-en</td><td>16.5</td><td>19.2</td><td>13.1</td><td>6.7</td><td>8.4</td><td>-</td><td>9.2</td></tr><tr><td>ro-en</td><td>41.0</td><td>45.3</td><td>32.9</td><td>31.9</td><td>30.8</td><td>9.6</td><td>35.6</td></tr><tr><td></td><td>mean</td><td>25.5</td><td>27.7</td><td>23.0</td><td>16.0</td><td>18.1</td><td>8.9</td><td>19.6</td></tr><tr><td rowspan="10">Fewshot<br>(5, Seen,<br>500 topics)</td><td>en-cs</td><td>20.4</td><td>23.2</td><td>23.6</td><td>11.9</td><td>14.5</td><td>4.4</td><td>15.3</td></tr><tr><td>en-de</td><td>26.6</td><td>27.7</td><td>22.4</td><td>19.0</td><td>23.5</td><td>12.2</td><td>23.9</td></tr><tr><td>en-fi</td><td>12.7</td><td>10.4</td><td>21.4</td><td>8.1</td><td>13.3</td><td>-</td><td>13.0</td></tr><tr><td>en-fr</td><td>32.1</td><td>36.1</td><td>35.2</td><td>24.1</td><td>29.6</td><td>12.5</td><td>30.5</td></tr><tr><td>en-lt</td><td>5.4</td><td>5.4</td><td>7.1</td><td>1.0</td><td>1.1</td><td>-</td><td>1.6</td></tr><tr><td>en-ro</td><td>25.8</td><td>25.8</td><td>23.4</td><td>16.8</td><td>18.2</td><td>5.7</td><td>20.9</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.0</td><td>1.0</td><td>0.6</td><td>0.0</td><td>0.2</td></tr><tr><td>cs-en</td><td>38.1</td><td>41.1</td><td>33.0</td><td>27.5</td><td>25.6</td><td>10.1</td><td>27.8</td></tr><tr><td>fr-en</td><td>42.4</td><td>47.9</td><td>40.5</td><td>29.5</td><td>35.9</td><td>16.2</td><td>35.6</td></tr><tr><td>lt-en</td><td>16.6</td><td>20.3</td><td>13.5</td><td>6.1</td><td>9.1</td><td>-</td><td>9.2</td></tr><tr><td>ro-en</td><td>41.6</td><td>45.5</td><td>34.0</td><td>33.1</td><td>31.0</td><td>9.4</td><td>35.5</td></tr><tr><td></td><td>mean</td><td>26.2</td><td>28.3</td><td>23.2</td><td>16.2</td><td>18.4</td><td>8.8</td><td>19.4</td></tr><tr><td rowspan="10">Fewshot<br>(3, Seen,<br>200 topics)</td><td>en-cs</td><td>17.7</td><td>20.6</td><td>24.8</td><td>12.4</td><td>12.9</td><td>4.8</td><td>14.5</td></tr><tr><td>en-de</td><td>25.8</td><td>26.0</td><td>22.4</td><td>18.3</td><td>22.9</td><td>11.7</td><td>22.7</td></tr><tr><td>en-fi</td><td>12.4</td><td>9.8</td><td>19.6</td><td>6.4</td><td>13.7</td><td>-</td><td>12.4</td></tr><tr><td>en-fr</td><td>31.7</td><td>36.0</td><td>34.4</td><td>22.9</td><td>30.1</td><td>12.8</td><td>30.5</td></tr><tr><td>en-lt</td><td>6.3</td><td>6.1</td><td>6.9</td><td>1.3</td><td>1.1</td><td>-</td><td>1.3</td></tr><tr><td>en-ro</td><td>24.5</td><td>25.2</td><td>22.7</td><td>16.4</td><td>18.8</td><td>5.6</td><td>21.8</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.3</td><td>0.6</td><td>0.5</td><td>0.0</td><td>0.1</td></tr><tr><td>cs-en</td><td>38.4</td><td>38.0</td><td>28.2</td><td>25.8</td><td>25.6</td><td>9.6</td><td>27.6</td></tr><tr><td>fr-en</td><td>41.8</td><td>45.1</td><td>40.1</td><td>28.8</td><td>35.0</td><td>17.1</td><td>34.9</td></tr><tr><td>lt-en</td><td>17.3</td><td>16.0</td><td>11.5</td><td>4.8</td><td>7.8</td><td>-</td><td>8.3</td></tr><tr><td>ro-en</td><td>41.4</td><td>44.1</td><td>25.8</td><td>30.6</td><td>30.8</td><td>9.2</td><td>34.4</td></tr><tr><td></td><td>mean</td><td>25.7</td><td>26.7</td><td>21.6</td><td>15.3</td><td>18.1</td><td>8.8</td><td>19.0</td></tr><tr><td rowspan="10">Fewshot<br>(3, Seen,<br>1000 topics)</td><td>en-cs</td><td>18.5</td><td>22.8</td><td>23.7</td><td>11.2</td><td>13.5</td><td>4.3</td><td>13.9</td></tr><tr><td>en-de</td><td>26.0</td><td>27.1</td><td>21.3</td><td>17.4</td><td>23.0</td><td>12.0</td><td>22.8</td></tr><tr><td>en-fi</td><td>12.9</td><td>10.9</td><td>20.9</td><td>6.9</td><td>13.1</td><td>-</td><td>12.9</td></tr><tr><td>en-fr</td><td>31.5</td><td>35.7</td><td>35.0</td><td>23.4</td><td>30.5</td><td>12.8</td><td>30.7</td></tr><tr><td>en-lt</td><td>5.4</td><td>5.3</td><td>6.7</td><td>1.0</td><td>1.2</td><td>-</td><td>1.2</td></tr><tr><td>en-ro</td><td>25.0</td><td>25.5</td><td>23.5</td><td>16.9</td><td>18.3</td><td>5.4</td><td>20.3</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.1</td><td>1.0</td><td>0.4</td><td>0.0</td><td>0.2</td></tr><tr><td>cs-en</td><td>38.2</td><td>40.4</td><td>32.4</td><td>26.3</td><td>25.9</td><td>9.7</td><td>27.5</td></tr><tr><td>fr-en</td><td>41.4</td><td>46.8</td><td>40.0</td><td>29.2</td><td>35.2</td><td>15.6</td><td>35.1</td></tr><tr><td>lt-en</td><td>17.6</td><td>21.1</td><td>11.9</td><td>5.0</td><td>8.1</td><td>-</td><td>8.8</td></tr><tr><td>ro-en</td><td>43.0</td><td>45.6</td><td>32.5</td><td>32.4</td><td>31.6</td><td>9.7</td><td>35.2</td></tr><tr><td></td><td>mean</td><td>25.9</td><td>28.1</td><td>22.6</td><td>15.5</td><td>18.3</td><td>8.7</td><td>19.0</td></tr><tr><td rowspan="10">Fewshot<br>(3, Seen,<br>Random Topic,<br>500 topics)</td><td>en-cs</td><td>17.8</td><td>21.6</td><td>23.6</td><td>12.6</td><td>13.2</td><td>4.5</td><td>14.7</td></tr><tr><td>en-de</td><td>25.4</td><td>26.7</td><td>21.5</td><td>17.6</td><td>22.5</td><td>12.1</td><td>23.2</td></tr><tr><td>en-fi</td><td>12.3</td><td>8.4</td><td>19.7</td><td>6.9</td><td>12.7</td><td>-</td><td>13.1</td></tr><tr><td>en-fr</td><td>31.4</td><td>35.3</td><td>33.7</td><td>23.9</td><td>29.2</td><td>13.1</td><td>31.2</td></tr><tr><td>en-lt</td><td>4.6</td><td>4.5</td><td>5.1</td><td>1.0</td><td>1.1</td><td>-</td><td>1.4</td></tr><tr><td>en-ro</td><td>22.6</td><td>24.2</td><td>22.2</td><td>16.2</td><td>18.1</td><td>5.3</td><td>21.6</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>0.6</td><td>0.8</td><td>0.4</td><td>0.0</td><td>0.1</td></tr><tr><td>cs-en</td><td>33.4</td><td>35.5</td><td>26.4</td><td>23.6</td><td>24.3</td><td>9.5</td><td>26.5</td></tr><tr><td>fr-en</td><td>38.5</td><td>42.6</td><td>37.6</td><td>27.6</td><td>35.3</td><td>17.1</td><td>34.9</td></tr><tr><td>lt-en</td><td>9.8</td><td>13.7</td><td>3.4</td><td>2.2</td><td>5.0</td><td>-</td><td>6.4</td></tr><tr><td>ro-en</td><td>38.7</td><td>39.7</td><td>24.0</td><td>29.2</td><td>29.6</td><td>9.2</td><td>33.2</td></tr><tr><td></td><td>mean</td><td>23.4</td><td>25.2</td><td>19.8</td><td>14.7</td><td>17.4</td><td>8.8</td><td>18.8</td></tr></tbody></table>

<table><caption>Table 23: BLEU scores for few-shot baseline experiments.</caption><thead><tr><th rowspan="2">Prompt</th><th rowspan="2">Pair</th><th colspan="7">Dataset</th></tr><tr><th>EMEA</th><th>JRC</th><th>KDE4</th><th>Subs</th><th>QED</th><th>Tanzil</th><th>TED</th></tr></thead><tbody><tr><td rowspan="10">Random Fewshot<br>(3, Seen)</td><td>en-cs</td><td>16.1</td><td>21.6</td><td>22.2</td><td>13.0</td><td>14.5</td><td>4.4</td><td>13.3</td></tr><tr><td>en-de</td><td>23.0</td><td>25.8</td><td>19.3</td><td>18.1</td><td>21.6</td><td>11.8</td><td>23.3</td></tr><tr><td>en-fi</td><td>11.6</td><td>7.6</td><td>19.3</td><td>7.4</td><td>12.4</td><td>-</td><td>12.6</td></tr><tr><td>en-fr</td><td>30.3</td><td>34.6</td><td>33.1</td><td>23.6</td><td>28.4</td><td>12.2</td><td>29.9</td></tr><tr><td>en-lt</td><td>4.9</td><td>4.8</td><td>7.8</td><td>1.1</td><td>1.3</td><td>-</td><td>1.8</td></tr><tr><td>en-ro</td><td>20.9</td><td>24.0</td><td>20.9</td><td>16.0</td><td>18.7</td><td>5.1</td><td>20.8</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.4</td><td>1.0</td><td>0.8</td><td>0.0</td><td>0.2</td></tr><tr><td>cs-en</td><td>5.3</td><td>6.3</td><td>8.1</td><td>3.4</td><td>1.9</td><td>0.2</td><td>1.4</td></tr><tr><td>fr-en</td><td>6.3</td><td>6.6</td><td>7.1</td><td>3.7</td><td>2.8</td><td>0.5</td><td>2.7</td></tr><tr><td>lt-en</td><td>4.4</td><td>3.5</td><td>5.1</td><td>0.8</td><td>1.2</td><td>-</td><td>0.9</td></tr><tr><td>ro-en</td><td>5.8</td><td>5.5</td><td>7.0</td><td>2.3</td><td>2.6</td><td>0.1</td><td>2.8</td></tr><tr><td></td><td>mean</td><td>12.9</td><td>14.0</td><td>13.8</td><td>8.2</td><td>9.7</td><td>4.3</td><td>10.0</td></tr><tr><td rowspan="10">Fewshot<br>(3, Language)</td><td>en-cs</td><td>19.4</td><td>24.5</td><td>24.6</td><td>12.6</td><td>16.2</td><td>5.4</td><td>15.5</td></tr><tr><td>en-de</td><td>26.9</td><td>27.4</td><td>23.0</td><td>18.4</td><td>23.4</td><td>13.1</td><td>23.8</td></tr><tr><td>en-fi</td><td>13.3</td><td>10.2</td><td>23.0</td><td>7.2</td><td>14.8</td><td>-</td><td>14.8</td></tr><tr><td>en-fr</td><td>34.1</td><td>38.0</td><td>36.3</td><td>24.2</td><td>30.3</td><td>15.3</td><td>30.1</td></tr><tr><td>en-lt</td><td>6.0</td><td>7.6</td><td>8.5</td><td>1.5</td><td>1.5</td><td>-</td><td>1.8</td></tr><tr><td>en-ro</td><td>28.5</td><td>29.1</td><td>24.1</td><td>18.6</td><td>19.7</td><td>7.6</td><td>21.7</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.2</td><td>0.8</td><td>1.2</td><td>0.4</td><td>0.4</td></tr><tr><td>cs-en</td><td>38.2</td><td>40.3</td><td>31.0</td><td>26.5</td><td>27.8</td><td>14.0</td><td>28.7</td></tr><tr><td>fr-en</td><td>41.6</td><td>47.5</td><td>39.8</td><td>29.1</td><td>35.9</td><td>18.4</td><td>35.6</td></tr><tr><td>lt-en</td><td>17.5</td><td>22.5</td><td>14.5</td><td>7.3</td><td>10.9</td><td>-</td><td>12.8</td></tr><tr><td>ro-en</td><td>42.0</td><td>45.8</td><td>34.1</td><td>32.4</td><td>32.0</td><td>12.8</td><td>36.7</td></tr><tr><td></td><td>mean</td><td>26.8</td><td>29.3</td><td>23.6</td><td>16.2</td><td>19.4</td><td>10.9</td><td>20.2</td></tr><tr><td rowspan="10">Similarity<br>(3, Language)</td><td>en-cs</td><td>29.6</td><td>28.9</td><td>27.8</td><td>13.0</td><td>16.1</td><td>19.0</td><td>16.6</td></tr><tr><td>en-de</td><td>30.6</td><td>32.8</td><td>24.9</td><td>17.8</td><td>24.8</td><td>23.2</td><td>23.0</td></tr><tr><td>en-fi</td><td>21.1</td><td>22.8</td><td>23.6</td><td>8.4</td><td>19.3</td><td>-</td><td>16.6</td></tr><tr><td>en-fr</td><td>38.3</td><td>42.7</td><td>36.9</td><td>23.0</td><td>29.1</td><td>31.4</td><td>32.0</td></tr><tr><td>en-lt</td><td>12.4</td><td>12.3</td><td>17.4</td><td>2.5</td><td>5.7</td><td>-</td><td>6.2</td></tr><tr><td>en-ro</td><td>35.2</td><td>34.2</td><td>28.0</td><td>19.8</td><td>19.3</td><td>24.8</td><td>23.3</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.5</td><td>2.3</td><td>2.0</td><td>0.5</td><td>0.2</td></tr><tr><td>cs-en</td><td>45.2</td><td>43.2</td><td>36.8</td><td>26.7</td><td>28.2</td><td>16.4</td><td>30.0</td></tr><tr><td>fr-en</td><td>46.8</td><td>49.6</td><td>43.0</td><td>28.7</td><td>36.4</td><td>18.0</td><td>36.5</td></tr><tr><td>lt-en</td><td>28.3</td><td>28.2</td><td>23.6</td><td>10.9</td><td>17.3</td><td>-</td><td>18.9</td></tr><tr><td>ro-en</td><td>48.4</td><td>49.9</td><td>35.7</td><td>33.6</td><td>32.8</td><td>13.9</td><td>37.8</td></tr><tr><td></td><td>mean</td><td>33.6</td><td>34.5</td><td>27.2</td><td>17.0</td><td>21.0</td><td>18.4</td><td>21.9</td></tr><tr><td rowspan="10">Fewshot<br>(3, Seen)</td><td>en-cs</td><td>17.5</td><td>22.9</td><td>22.9</td><td>11.1</td><td>12.5</td><td>3.9</td><td>12.9</td></tr><tr><td>en-de</td><td>24.3</td><td>27.0</td><td>20.4</td><td>17.4</td><td>20.9</td><td>11.5</td><td>21.8</td></tr><tr><td>en-fi</td><td>12.5</td><td>12.1</td><td>21.3</td><td>6.3</td><td>11.9</td><td>-</td><td>12.7</td></tr><tr><td>en-fr</td><td>30.1</td><td>37.0</td><td>33.3</td><td>22.0</td><td>28.5</td><td>11.9</td><td>29.3</td></tr><tr><td>en-lt</td><td>6.9</td><td>7.5</td><td>12.4</td><td>1.9</td><td>1.3</td><td>-</td><td>1.4</td></tr><tr><td>en-ro</td><td>24.0</td><td>25.6</td><td>22.0</td><td>15.7</td><td>17.0</td><td>4.9</td><td>20.1</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.8</td><td>1.1</td><td>0.8</td><td>0.0</td><td>0.1</td></tr><tr><td>cs-en</td><td>5.4</td><td>6.1</td><td>7.8</td><td>3.0</td><td>1.8</td><td>1.6</td><td>1.5</td></tr><tr><td>fr-en</td><td>6.4</td><td>6.9</td><td>8.0</td><td>2.8</td><td>3.8</td><td>1.3</td><td>4.3</td></tr><tr><td>lt-en</td><td>4.4</td><td>3.8</td><td>5.1</td><td>0.7</td><td>1.2</td><td>-</td><td>0.9</td></tr><tr><td>ro-en</td><td>5.8</td><td>5.1</td><td>6.6</td><td>2.2</td><td>3.0</td><td>1.0</td><td>2.3</td></tr><tr><td></td><td>mean</td><td>13.7</td><td>15.4</td><td>14.7</td><td>7.7</td><td>9.3</td><td>4.5</td><td>9.8</td></tr><tr><td rowspan="10">Retrieval<br>(3, Language)</td><td>en-cs</td><td>31.4</td><td>30.9</td><td>29.1</td><td>12.5</td><td>17.9</td><td>21.4</td><td>18.0</td></tr><tr><td>en-de</td><td>34.0</td><td>35.3</td><td>26.4</td><td>18.8</td><td>23.2</td><td>23.1</td><td>23.5</td></tr><tr><td>en-fi</td><td>24.0</td><td>25.5</td><td>27.0</td><td>9.2</td><td>19.3</td><td>-</td><td>18.3</td></tr><tr><td>en-fr</td><td>39.3</td><td>43.9</td><td>39.7</td><td>23.0</td><td>31.4</td><td>34.3</td><td>30.8</td></tr><tr><td>en-lt</td><td>15.8</td><td>14.0</td><td>18.8</td><td>2.7</td><td>6.5</td><td>-</td><td>7.4</td></tr><tr><td>en-ro</td><td>36.7</td><td>35.6</td><td>30.1</td><td>19.9</td><td>19.5</td><td>28.9</td><td>22.4</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>6.6</td><td>3.9</td><td>4.5</td><td>25.1</td><td>6.2</td></tr><tr><td>cs-en</td><td>44.7</td><td>44.7</td><td>38.7</td><td>26.6</td><td>28.9</td><td>16.9</td><td>30.0</td></tr><tr><td>fr-en</td><td>47.3</td><td>50.3</td><td>45.2</td><td>29.3</td><td>36.0</td><td>17.7</td><td>36.8</td></tr><tr><td>lt-en</td><td>27.9</td><td>28.5</td><td>27.4</td><td>9.8</td><td>16.8</td><td>-</td><td>18.9</td></tr><tr><td>ro-en</td><td>48.3</td><td>49.9</td><td>40.6</td><td>33.0</td><td>32.8</td><td>14.3</td><td>38.4</td></tr><tr><td></td><td>mean</td><td>34.9</td><td>35.9</td><td>30.0</td><td>17.2</td><td>21.5</td><td>22.7</td><td>22.8</td></tr><tr><td rowspan="10">Retrieval<br>(3, Seen)</td><td>en-cs</td><td>15.0</td><td>20.4</td><td>19.3</td><td>9.7</td><td>11.6</td><td>4.3</td><td>10.9</td></tr><tr><td>en-de</td><td>23.7</td><td>25.5</td><td>19.9</td><td>15.9</td><td>21.3</td><td>11.7</td><td>22.1</td></tr><tr><td>en-fi</td><td>10.8</td><td>7.6</td><td>19.4</td><td>6.7</td><td>9.0</td><td>-</td><td>12.3</td></tr><tr><td>en-fr</td><td>29.2</td><td>35.1</td><td>32.8</td><td>21.7</td><td>29.2</td><td>12.1</td><td>29.8</td></tr><tr><td>en-lt</td><td>4.9</td><td>4.4</td><td>6.9</td><td>0.7</td><td>1.3</td><td>-</td><td>1.5</td></tr><tr><td>en-ro</td><td>20.2</td><td>22.4</td><td>20.2</td><td>15.0</td><td>16.2</td><td>4.9</td><td>20.6</td></tr><tr><td>en-ta</td><td>-</td><td>-</td><td>1.4</td><td>0.9</td><td>1.1</td><td>0.0</td><td>0.2</td></tr><tr><td>cs-en</td><td>5.1</td><td>3.5</td><td>7.2</td><td>1.7</td><td>1.1</td><td>0.1</td><td>1.4</td></tr><tr><td>fr-en</td><td>5.7</td><td>5.2</td><td>6.4</td><td>1.9</td><td>2.0</td><td>0.1</td><td>1.6</td></tr><tr><td>lt-en</td><td>4.4</td><td>3.6</td><td>5.1</td><td>1.0</td><td>0.9</td><td>-</td><td>0.9</td></tr><tr><td>ro-en</td><td>5.2</td><td>4.3</td><td>5.9</td><td>3.5</td><td>2.1</td><td>0.1</td><td>2.0</td></tr><tr><td></td><td>mean</td><td>12.4</td><td>13.2</td><td>13.1</td><td>7.2</td><td>8.7</td><td>4.2</td><td>9.4</td></tr></tbody></table>