In-context Examples Selection for Machine Translation
Year: 2023

Abstract:
Large-scale generative models show an impressive ability to perform a wide range of Natural Language Processing (NLP) tasks using in-context learning, where a few examples are used to describe a task to the model. For Machine Translation (MT), these examples are typically randomly sampled from the development dataset with a similar distribution as the evaluation set. However, it is unclear how the choice of these in-context examples and their ordering impacts the output translation quality. In this work, we aim to understand the properties of good in-context examples for MT in both in-domain and out-of-domain settings. We show that the translation quality and the domain of the in-context examples matter and that 1-shot noisy unrelated example can have a catastrophic impact on output quality. While concatenating multiple random examples reduces the effect of noise, a single good prompt optimized to maximize translation quality on the development dataset can elicit learned information from the pre-trained language model. Adding similar examples based on an n-gram overlap with the test source significantly and consistently improves the translation quality of the outputs, outperforming a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets.

heading: 1 Introduction
text: In-context learning (Brown et al., 2020) has recently received a lot of attention from the NLP research community due to its remarkable ability to utilize only a few input-output examples to perform many NLP tasks (Liu et al., 2021). For example, Lin et al. (2021) demonstrate that a 7.5B multilingual generative model, XGLM, outperforms a supervised sequence-to-sequence baseline in 45 translation directions on the FLORES-101 machine translation benchmark (Goyal et al., 2022) using just 32 randomly sampled translation examples
∗ Work done during internship at Meta AI Research.
as demonstrations. While these results are compelling, recent work has also shown that the performance and capability of a pre-trained language model (PLM) can be highly sensitive to many factors, such as the choice of in-context examples (Liu et al., 2022b), their ordering (Lu et al., 2022) and the template (Jiang et al., 2020).
Typically, in-context learning for MT uses examples that are randomly sampled from a small development set that resembles the domain of the test dataset. The effect of the aforementioned factors (such as the choice of the examples) on the translation quality of the PLM hence remains unclear and unexplored. Yet another crucial gap in using in-context learning for MT in the current literature is the effect of the domain of in-context examples on translation quality since out-of-domain generalization is a known and important challenge in MT (Koehn and Knowles, 2017).
In this work, we systematically analyze how factors such as the choice and the number of few-shot in-context examples and their ordering impact MT output quality. We show that while noisy unrelated 1-shot example can have a significantly adverse effect on translation quality, a single prompt optimized to maximize the translation quality on a development set can sufficiently elicit task-based information from the PLM. Our analysis thus demonstrates the importance of selecting good examples for MT and raises the question: What are the properties of good in-context examples for MT? In that direction, our findings suggest that a well-formed meaning-equivalent translation example results in higher quality translation than randomly selected in-context examples.
Motivated by the use of Translation Memory in Computer-Aided Translation (Yamada, 2011) and its usage in computational approaches to Machine Translation (Somers, 1999; Koehn and Senellart, 2010; Khandelwal et al., 2020, inter alia), we retrieve similar examples to the test source from
8857
a datastore that includes pairs of the source text and their corresponding translations via BM25, an unsupervised efficient retriever to provide additional context to the model. We propose a novel incontext example selection and re-ranking strategy to maximize the coverage of the source n-grams in the retrieved examples. Experiments on WMT’19 English↔German and English↔Russian datasets show that our proposed strategy can consistently improve the translation quality over the outputs generated using BM25 retrieved examples. Combining optimized 1-shot task-level with examplespecific in-context examples using a simple concatenation strategy further improves translation quality, outperforming state-of-the-art inferenceadapted nearest-neighbor MT models (kNN-MT) on two out-of-domain datasets (Medical and IT) while being memory and compute efficient as our approach does not require constructing and querying a dense token-level datastore.

heading: 2 Background: In-context Learning
text: Generating translations from large-scale multilingual language models like mGPT (Shliazhko et al., 2022), XGLM (Lin et al., 2021) or AlexaTM 20B (Soltan et al., 2022) requires conditioning the decoder-only language model with in-context parallel examples. These examples serve two purposes: a) providing the model with the format and knowledge of the task (task-level) and b) guiding the output generation via providing useful information about the unseen source sentence (example-specific). This is different from the standard sequence-to-sequence models, where the task is always known, and the model learns generalizable patterns from the input-output examples to perform the task (in this case, translation) for the unseen source text.
Formally, given k in-context examples {xi, yi}k1 the prefix input or the prompt, xpj , is generated by concatenating the demonstration examples {(xi, yi)}k1 to the test input, xsj according to a template, P (see Table 1). The output, ŷ, is then generated via the PLM with parameters θ via greedy decoding as follows:
ŷj,t = argmax y′j,t
PPLM(y ′ j,t|xpj , ŷj,<t; θ) (1)

heading: 3 Prompt Selection
text: Ideally, good in-context examples can trigger the pre-trained language model to generate the desired output and also elicit the information learned during pre-training (Jiang et al., 2020). Min et al. (2022) show that, for classification tasks, the incontext examples provide information about the task (the distribution of the input text, the label space, and the format of the task) and that the model does not rely on these examples to generate the final output. However, their analysis is limited to a) classification tasks and 2) randomly sampled in-context examples. Prior work has also shown that the order of these in-context examples can also lead to high variance in downstream performance (Zhang et al., 2022). However, less is understood about how these factors impact text generation tasks like MT. Do we need multiple incontext examples? What makes good in-context examples for MT? How sensitive is the model to the order of the prompts?
In this work, we aim to better understand the impact of prompt selection on the translation quality of the outputs. Given a training dataset consisting of n parallel examples D = {xi, yi}ni=1, and a test source xj , we select a subset of m informative samples to form a prompt which either provides task-level and/or example-specific information as discussed below.

heading: 3.1 Task-level In-context Examples
text: A good task-level in-context example should be able to elicit information learned during pretraining from the PLM. One way to measure the efficacy of an example as a prompt is via computing the translation quality of the outputs generated when prompting the PLM given an example. Hence, we select the task-level prompt as follow: For a given example sampled from the training dataset, (xi, yi) ∈ DS , we create a prompt, xpi by concatenating the example {(xi, yi)} to each source in the
development set. The system outputs are then generated using equation 1. We then rank examples from DS as task-level prompts based on the BLEU of the generated outputs against the references on this held-out development set, Ddev = {X,Y }:
(xs, ys) = argmax (x,y)∈DS BLEU(Y, Ŷ ) (2)

heading: 3.2 Example-specific In-context Examples
text: Prior work on retrieving good in-context examplespecific prompts for tasks other than MT (like question answering or knowledge retrieval) either trains a dense-retriever (Rubin et al., 2021) or utilizes samples that are closer to the test source in the embedding space of a PLM like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), or XLNET models (Liu et al., 2022b). While contextual models can generate a global sentence representation, they overlook rare lexicons which can be important for generating translations in unseen domains like medical or IT (Wrzalik and Krechel, 2021).
However, for MT, overlapping n-grams between the source and the retrieved sentences ensures informativeness as the target associated with the retrieved sentence is likely to include partial translations of the source. We can thus use BM25 as an efficient unsupervised retrieval method to retrieve similar examples. However, as the examples are scored independently and BM25 favors rare word matches (Robertson et al., 2009), the top retrieved candidates might not cover all the terms in the source text (Figure 1). Given that the context window of the PLM is usually limited (∼ 3096 tokens, 16− 20 examples), maximizing the coverage of all
the terms found in the test input might be favorable. Hence, we propose to re-rank the top 100 candidates retrieved from BM25 using our algorithm outlined in 1. We extract all the word n-grams, and their counts from the test source, xsj and source of the BM25 retrieved examples, {Pj(xi)}k1 (lines 2-4). Let S and Q denote the set of the source n-grams and the n-grams from a BM25 retrieved example, respectively. We compute a recall-based (R) n-gram overlap score (line 7):
Rn =
∑ ngram∈S∩Q Countmatched(ngram)∑
ngram∈S CountS(ngram) (3)
Score = exp( 1
n
∑
n
log(Rn)) (4)
The example with the maximum score is then added to the set of selected prompts, and the found n-grams from the test source are then downweighted by a factor, λ, for the next iteration of selection (line 14). For example, setting λ = 0 will select the example that covers the n-grams from the test source in the subsequent iteration that has not already been encountered. This process is then repeated over the retrieved pool until a set threshold of the score is reached.
Figure 1 shows the top-100 candidates retrieved via BM25 for the input: “Welche Risiken sind mit Poulvac FluFend H5N3 RG verbunden?”. The top few candidates provide the same information to the PLM, i.e., translation of the phrase “Poulvac FluFend H5N3 RG”. The examples including the other terms (“Welche Risiken sind mit verbunden ?”) from the input text, are ranked lower. On the
Algorithm 1: An N-gram Recall-based Strategy to Re-rank In-context Examples Input: Prompts {Pj(xi, yi)}k1 for the test source xsj , λ, Threshold Output :Ordered Selected Prompts {T = Pj(xi, yi)}s1, s ≤ k
1 T ← Empty Ordered List 2 S ← EXTRACT_WORD_NGRAMS_WITH_COUNTS (xsj) 3 for i ∈ {1..k} do 4 Q[i]← EXTRACT_WORD_NGRAMS_WITH_COUNTS (P kj (xi)) 5 while True do 6 for i ∈ {1..k} do 7 Score[i]← NGRAM_OVERLAP_SCORE (S,Q[i]) 8 if max(Score) < Threshold then 9 break
10 T.append(Pargmax(Score)) 11 matched_ngrams← S ∩Q[argmax(Score)] 12 Q[argmax(Score)]← ∅ 13 for ngram ∈ matched_ngrams do 14 CountS(ngram)× = λ
15 Return T
other hand, our proposed re-ranking strategy can cover all the terms from the input text, in this case, with just the top-2 examples.

heading: 4 Evaluation Settings

heading: 4.1 Datasets and Evaluation Metric
text: We perform our in-domain evaluation on the WMT19 German (de) ⇔ English (en) and WMT-19 Russian (ru) ⇔ English (en) datasets (Barrault et al., 2019). For the out-of-domain evaluation, we use the multi-domain dataset from Aharoni and Goldberg (2020) for the following domains: Medical, Law, IT, and Koran. The dataset statistics are reported in the Appendix (Table 8). Following Ng et al. (2019), we normalize punctuation using Moses (Koehn et al., 2007) and remove sentences longer than 250 tokens and sentence pairs with a source/target length ratio exceeding 1.5 from the in-domain datasets. The detokenized length truncated model-generated outputs are evaluated using sacreBLEU (Papineni et al., 2002; Post, 2018).1 The PLM outputs are truncated to twice the source length, as preliminary analysis suggested degeneration in a few (∼10-20) examples.

heading: 4.2 Experimental Conditions
text: Language Model We use the publicly available checkpoint of the XGLM7.5B, a decoder-only multilingual language model (Lin et al., 2021) for all
1https://github.com/mjpost/sacrebleu We also report Comet (Rei et al., 2020) scores for evaluating translation quality in Appendix Tables 14 and 15.
our experiments, which has 32 layers and a hidden dimension of 4096.
Baselines and Comparisons We consider the following comparisons:
• Random: p random few-shot examples sampled from the training dataset (number of trials=3).
• Task-level: top-p examples that achieve the highest BLEU on the development set (§ 3.1).
• Retrieved In-context (BM25): qmax examples retrieved via BM25, since, unlike task-level examples, there is no guarantee that exactly q similar examples will be found in the training dataset for each input.
• Retrieved Re-ranked In-context (R-BM25): qmax re-ranked examples using our proposed approach as detailed in § 3.2.
We also compare our results with the state-ofthe-art nearest neighbor-based approach for out-ofdomain evaluation, kNN-MT (Khandelwal et al., 2020). We use λ = 0.1, threshold=1.0 and order the examples according to their similarity to the source, with the most similar examples on the left in all our experiments (Appendix Tables 9,10).

heading: 5 Results
text: Table 2 and 3 summarize the main results for the in-domain and the out-of-domain evaluations.

heading: 5.1 In-domain Evaluation

heading: A single task-level prompt is competitive with
text: 16 random few-shot examples. Our experiment suggests that it is possible to elicit the task-level knowledge from the large-scale language model using a single prompt as opposed to using 16 random few-shot examples when translating into English (Table 2). Using a single task-level prompt (optimized on the development set) improves BLEU over using 16 random few-shot examples for 2 out of 4 translation directions (De-En, Ru-En). We hypothesize that when translating out of English, the model still benefits from getting exposed to multiple and diverse random few-shot examples as the target language model is relatively weaker.
Multiple example-specific prompts are required to improve translation quality over a single tasklevel prompt. Using a single task-level (p = 1) prompt attains higher BLEU over using a single example-specific prompt (q = 1; BM25, RBM25) across the board. By contrast, using up to 16 BM25 prompts (qmax = 16) significantly improves output quality over using task-level prompts, with an average gain of 1.41 in BLEU.
Re-ranking BM25 retreived examples improves BLEU. Our proposed re-ranking strategy consistently improves BLEU across the board over BM25 for both values of qmax = {1, 16} showing that both the order and the choice of the in-context examples matters.
Both task-level and R-BM25 examples provide complementary advantages, as combining them us-
ing a simple concatenation strategy improves output quality over task-level or R-BM25 examples. We leave the exploration of optimizing the number and the joint order of task-level and examplespecific prompts to future work.

heading: 5.2 Out-of-domain Evaluation
text: As XGLM is trained on monolingual Common Crawl snapshots, translation in any domain and language could be considered an out-of-domain task. However, we hypothesize that translation in specific domains like medical, law, or IT could still be challenging for the PLM as the model is less likely to have observed sufficient monolingual datasets for these specialized domains, in contrast to the news text found in WMT. Examples from these domains will require translating rare terminology and carry domain-specific idiosyncrasies, which is known to pose a challenge even for a well-trained supervised neural MT model (Koehn and Knowles, 2017). Hence, we also evaluate PLM under these specialized out-of-domain scenarios.
Domain of few-shot in-context examples matter. Task-level in-context examples drawn from the domain of evaluation, i.e., domain-specific, obtain on an average higher BLEU scores across the board than using examples from a distant WMT corpus as expected (Table 3) in both 1-shot (p = 1: +1.4) and 16-shot (p = 16: +2.7) settings.
Example-specific prompts significantly improve translation quality over task-level prompts. Unlike the in-domain evaluation, retrieved and reranked example-specific prompts (R-BM25) im-
Method Corpus p+ qmax MEDICAL LAW IT KORAN Avg.
prove the translation quality significantly across the board with up to 23 BLEU gain in the Law domain using just a single example as a prompt over a task-level prompt. This can be attributed to the high lexical overlap in the examples retrieved from the training data for these domains (Table 6).
Task-level and R-BM25 prompts are complementary. Both task-level and R-BM25 provide supporting information for a given test source sentence as concatenating these set of prompts improves output quality over using these methods independently, outperforming a strong kNN-MT baseline on 2 out of 4 domains (Medical and IT). Where kNN-MT utilizes token-level nearestneighbor inference with representations extracted for bitext using and in combination with a strong supervised MT model to reach the reported translation quality, our approach only uses a sentencelevel unsupervised retrieval (BM25) to provide additional context to the unseen source with a multilingual PLM that has not been trained with any known parallel supervision to reach better or comparable translation quality. Hence, our results provide support for further analysis of the translation abilities of retrieval-augmented PLM on new domains and language pairs.
Our manual analysis suggests that the higher gain obtained in the IT domain (+0.86) with both task-level and example-specific prompts can be explained by the observation that for 100 test source sentences, there are no training examples with any lexical overlap with the test source. The task-level prompt can still elicit learned information from the PLM over using no examples for these inputs.

heading: 6 Analysis

heading: 6.1 Task-level Example Selection
text: Choice of Few-shot Examples We show the distribution of output quality as measured by BLEU when using 100 different examples as prompts in Figure 2. Across all four language pairs, there is a large variation in BLEU scores (up to 20 BLEU), where noisy or unrelated prompts can lead to significantly worse output quality. Given that most existing parallel corpora are web-crawled and the quality of bitext can vary significantly across different language pairs (Kreutzer et al., 2022), randomly sampled examples can under-estimate the translation quality attainable by prompting the PLM.
Impact of Pool Size on Task-level Prompt Selection We select the best task-level prompt based on the translation quality on the development set from a random sample of 100 examples (pool) as detailed in Section 3.1. However, one concern regarding selecting the best task-level prompt in this
fashion could be that we might still be underestimating the PLM (s) performance, as a larger pool size could result in better output quality. We study the impact of using a larger pool size in Table 4 where increasing the number of examples from 100 to 1000 only leads to a gain of 0.5 points in the maximum BLEU. From the same table, we can also observe that for any subset of random 100 fewshot examples, we can extract a task-level prompt (BLEU: 36) with a small standard deviation in overall output quality (0.18).
Properties of good Task-level prompts Our manual analysis on the best task-level prompts suggests that any well-formed and meaning-equivalent translation (Vyas et al., 2018; Briakou and Carpuat, 2020) could make a good task-level prompt (see
examples in Appendix Table 11). To quantify the meaning equivalence of the 1-best task-level prompt against random 1-shot examples, we report the percentage of aligned words between the source and reference translation (“% Aligned words”) using fastAlign (Dyer et al., 2013) and the log probability of generating the reference translation conditioned on the source using a pre-trained multilingual NMT model, Prism-src (Thompson and Post, 2020; Agrawal et al., 2021) in Table 5.2 Across all language pairs and both metrics, task-level examples achieve higher semantic similarity scores than random 1-shot examples suggesting that task-level examples are relatively more equivalent in meaning than random examples.
Impact of Ordering To investigate the sensitivity of the few-shot prompts ordering on MT quality, we use all possible order permutations of four randomly sampled examples and the top four task-level examples as prompts and report BLEU in Table 7. Task-level prompts are less sensitive to prompt order, as suggested by the lower standard deviation achieved in all settings, and result in higher translation quality than randomly selected examples. Across the three different runs of randomly sampled examples, there is a significant difference in BLEU, further corroborating that the
2https://github.com/clab/fast_align, https: //github.com/thompsonb/prism
choice of in-context examples and their ordering matters.

heading: 6.2 Informativeness of BM25 Examples
text: To understand the benefit of retrieved examples in the out-of-domain evaluation, we measure the lexical overlap between the test input (x, y) and the prompts (Ix, Iy) using BLEU (Avg. BLEU (Ix, x), Avg. BLEU (Iy, y)), where Ix and Iy are the sources and target translations of the retrieved incontext examples. We also report the correlation against the output translation quality BLEU(ŷ, y). Table 6 shows that the source lexical overlap is a good indicator of the informativeness of a prompt for 3 out of 4 domains, with Koran as an exception. For Koran, while the retrieved sentences have a high overlap with the source (36.03), the target associated with the prompts (Iy) does not get high BLEU with the reference (10.36) compared to other domains. We hypothesize that this might be due to a bias in the reference translations towards a particular output style. We provide examples of this phenomenon in the Appendix Section F.

heading: 6.3 Size of the Datastore
text: Figure 3 shows BLEU when varying the size of the datastore used to retrieve similar in-context examples using BM25 on the Medical dataset. As the size of the datastore increases, the likelihood of retrieving a more similar example increases. However, similar output quality in BLEU can be achieved by using multiple in-context examples when a smaller in-domain datastore is available as
multiple examples can provide better coverage of the source terms — BLEU @q=16 with a datastore size of 100k is equivalent to BLEU @q=1 with twice as many examples (200k).

heading: 7 Related Work
text: The selection of in-context examples and their impact on downstream NLP task performance has been studied in prior work for tasks other than MT (Liu et al., 2022b; Lu et al., 2022; Jiang et al., 2020; Min et al., 2022; Zemlyanskiy et al., 2022; Rubin et al., 2021; Liu et al., 2022a). Garcia and Firat (2022) use natural language prompts to control the target language in multilingual MT and investigate the effect of scale, number of languages, and their similarity for this phenomena. Wang et al. (2022) utilize BM25 retrieved training examples in a supervised fashion to learn from similar examples during training. Contrary to prior work, we utilize similar examples to form a textual prompt which is used to guide the generation of a translation during inference.
Prior work on domain adaptation for MT uses domain-specific bilingual or monolingual datasets to improve the translation quality of a neural sequence-to-sequence MT model either during training (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016; Wang et al., 2017) or inference
(Zheng et al., 2021; Khandelwal et al., 2020; Martins et al., 2022). Similar to past work, our work utilizes out-of-domain bitext during inference but instead adapts a PLM on unseen domains. However, our approach does not rely on creating a domainspecific token-level datastore, hence is more compute and memory efficient.
Several concurrent works investigate in-context learning for MT: Zhang et al. (2023) study prompting strategies for MT and examine several factors that could impact translation quality. Garcia et al. (2023) show the effectiveness of using few-shot examples to control translation formality and also corroborates our finding that the quality of the fewshot in-context examples matter. Ghazvininejad et al. (2023) provide control hints to large language models via bilingual dictionaries to improve the translation of rare words. Our work provides both supporting and complementary pieces of evidence to these studies by a) contributing a systematic analysis showing that the impact of the ordering of the demonstration examples on translation quality is dependent upon the nature and the quality of the examples and b) proposing a novel recall-based reranking approach that overcomes the limitations of BM25-based retrieval for in-context examples selection and optimizes for the selection of multiple prompts for MT. To the best of our knowledge, ours is the first work to jointly optimize the selection of multiple prompts for MT either via combining task-level and example-specific prompts or via directly optimizing the joint utility of multiple example-specific prompts by maximizing the coverage of the selected n-grams.

heading: 8 Conclusion
text: We investigate the choice of in-context examples selection for MT in both in-domain and out-ofdomain settings. We propose a novel recall-based re-ranking approach to utilize similar training examples as prompts and show their efficacy across multiple datasets and domains. Our findings show that task-level prompts can provide a complementary advantage to example-specific prompts, outperforming a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets while being memory and compute efficient. Our manual analysis of the generated outputs reveals that the PLM can mimic the style of the in-context examples provided and can be used for template-based translation synthesis. These results allow future research to evaluate the
potential of generating diverse and style-specific outputs for MT.

heading: 9 Limitations
text: We note a few limitations of our work: a) while we systematically investigate the choice of in-context examples for both in- and out-of-domain settings for higher-resource language pairs (EnglishGerman, English-Russian), it is unclear how this in-context ability of the PLM varies for the lowerresourced language pairs; b) We only experimented with one pre-trained language model, XGLM. Our preliminary experiments suggested XGLM-7.5B to result in better translation quality than Bloom7B (Scao et al., 2022) under the same settings. However, further investigation is required to understand how these results vary across different model scales; c) We analyze different orderings for the few-shot task-level prompts but only examine limited sets of ordering (most similar to the left or right) for the example-specific prompts. As the PLM is shown to be sensitive to the ordering of these in-context examples, it remains an open question to study how to best combine the information from multiple example-specific prompts, with prompt ensembling being a viable option, which we leave to future work.

heading: A Statistics of Datasets
text: Table 8 includes statistics of training, development and test sets used for the experiments discussed in the paper.

heading: B Compute Infrastructure & Run time
text: Each experiment is run on a single Nvidia Tesla V100 Volta GPU machine with 32G Ram. A single inference experiment on 2000 test examples using XGLM with 16 in-context examples takes around 3-4 hrs to complete.

heading: C Results using Second Metric: Comet
text: We report translation quality using Comet (Rei et al., 2020) in Tables 14 and 15. We use the eamt22-cometinho-da model (Rei et al., 2022) to generate the scores as it was shown to achieve higher correlations with human judgments than lexical overlap metrics while being computationally efficient. Our re-ranking strategy (with qmax = 16) consistently performs the best across the board except for Koran, outperforming strong kNN-MT baselines on the multi-domain test set in 3 out of 4 settings. Adding a task-level prompt to 16 RBM25 prompts via concatenation further improves quality in 5 out of 8 settings.

heading: D Hyperparameter Search
text: D.1 Order of BM25 Retrieved Examples
We report BLEU when using two different orderings of example-specific prompts on the development set for the medical domain. Ordering the examples with the most similar examples on the left attains higher BLEU than the right-to-left order. We note that the trend could vary depending on the noise in the training dataset, the degree of similarity, and the number of retrieved examples. We leave
the exploration of the ordering of example-specific prompts to future work.
D.2 Choice of λ, Threshold
Table 10 shows the BLEU and the average number of in-context examples selected when varying λ and the threshold described in Section 3.2. We select λ = 0.1 and threshold value of 1.0 as it achieves the best BLEU on the Medical development set as shown below:

heading: E Example Task-Level Prompts
text: Table 11 shows the best task-level in-context example selected by our method described in § 3.1 and the respective BLEU scores on the development set for the German-English and Russian-English tasks.

heading: F Output Analysis
text: We report two interesting findings when prompting PLM with task-level and example-specific prompts:
Stylistic Outputs One advantage of using a single task-level in-context example to prompt the PLM is that it allows us to systematically study how the choice of prompt influences the style of the generated translation. Table 12 illustrates one
such example: we can observe that as the prompt includes a contraction (“we are” vs. “we’re”), the outputs generated by the PLM also include contractions and can be incorrectly penalized by BLEU while being meaning equivalent.
Template-based MT Template-based translation in medical, legal, it, or e-commerce domain can be preferable as they reduce the risk of generating errors in automatically generated translations. We present some examples in Table 13 on how PLM can seamlessly use retrieved prompts to synthesize a translation from the template provided.
Method Corpus p+ qmax MEDICAL LAW IT KORAN
Results from Jiang et al. (2022) Vanilla kNN-MT - - 0.548 0.662 0.531 -0.014 Their model - - 0.578 0.703 0.585 0.047
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 9

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 9 (Limitation 1). In our work, we study and improve the translation ability of large-scale language models for higher resource language pairs only. It still remains an open question on how these abilities transfer to the lower-resourced language pairs. Furthermore, getting reliable and consistent outputs from generative language models is a known problem: https://openreview.net/forum?id=98p5x51L5af. Furthermore,

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes, our abstract summarizes the main results and takeaways.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? While large-scale generative models are not directly intended to be used for machine translation and many other downstream NLP tasks, they have been shown to be able to utilize very few examples to perform these tasks. Our work studies this phenomenon and provides analysis and modifications to improve this capability.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix Table A, B and Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4 and Appendix Table B.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.2 and Appendix Table B.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 and Appendix Table D.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 4, 5 and 6 and Appendix Table C

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1 (Footnote 1).
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We leave
the exploration of the ordering of example-specific prompts to future work. D. 2 Choice of λ, Threshold
Table 10 shows the BLEU and the average number of in-context examples selected when varying λ and the threshold described in Section 3. 2. We select λ = 0. 1 and threshold value of 1. 0 as it achieves the best BLEU on the Medical development set as shown below: