An Empirical Study of In-context Learning in LLMs for Machine Translation (2024)

Abstract: Recent interest has surged in employing Large Language Models (LLMs) for machine translation (MT) via in-context learning (ICL) (Vilar et al., 2023). Most prior studies primarily focus on optimizing translation quality, with limited attention to understanding the specific aspects of ICL that influence the said quality. To this end, we perform the first of its kind, an exhaustive study of in-context learning for machine translation. We first establish that ICL is primarily example-driven and not instructiondriven. Following this, we conduct an extensive exploration of various aspects of the examples to understand their influence on downstream performance. Our analysis includes factors such as quality and quantity of demonstrations, spatial proximity, and source versus target originality. Further, we also investigate challenging scenarios involving indirectness and misalignment of examples to understand the limits of ICL. While we establish the significance of the quality of the target distribution over the source distribution of demonstrations, we further observe that perturbations sometimes act as regularizers, resulting in performance improvements. Surprisingly, ICL does not necessitate examples from the same task, and a related task with the same target distribution proves sufficient. We hope that our study acts as a guiding resource for considerations in utilizing ICL for MT. Our code is available on https://github.com/PranjalChitale/ in-context-mt-analysis.

heading: 1 Introduction
text: Large Language Models leverage In-Context Learning to effectively solve diverse downstream tasks (Brown et al., 2020; Dong et al., 2023; Liu et al., 2023; OpenAI et al., 2023; Chowdhery et al., 2022). This inference-only method involves conditioning
* Equal contribution † Work done at Nilekani Centre at AI4Bharat
the model with task-specific demonstrations consisting of input-output pairs within the prompt before the actual test example. Most recently, MT via ICL has become popular (Vilar et al., 2023), however, there is still a limited understanding of how ICL works and its aspects in this context. Raunak et al. (2023) make some initial explorations on partial test sets, but there are still open questions pertaining to whether ICL is example-driven or instruction-driven, whether all examples contribute equally or not, and its potential to enforce control over the generation to regulate the pre-training biases observed in models. To answer these questions, in this paper, we conduct experiments by perturbing either the instructions or the demonstrations while keeping the other clean to assess the model’s ability to perform tasks appropriately. This allows us to understand if clear instructions can guide the model effectively even when examples are perturbed and, conversely, if clean examples can steer the model to perform the appropriate task despite the instruction being misleading. We assess 6 models across 2 language families, spanning 12 language pairs on complete test sets across various noise thresholds by considering realistic perturbation attacks.
When perturbing examples, we study 1) whether ICL treats all examples equally or favors those in spatial proximity to the test example, 2) whether utilizing in-context examples from related tasks can effectively guide models in performing another task to simulate the unavailability of demonstrations for some language pairs, 3) whether various choices of auxiliary source language for the demonstrations influence downstream MT, 4) whether models can implicitly make associations based on contextual information via a transitive MT setup and finally, 5) whether smaller models are susceptible to contextual misinformation or if their pre-training biases act as safeguards against it. Our work aims to holistically understand ICL and its driving factors for
7384
MT tasks with experiments spanning over 25K runs across various models and language directions. We observe that examples significantly influence ICL, while instructions have a limited impact. Notably, target distribution plays a more significant role than source distribution in ICL demonstrations. While perturbations might seem potentially harmful, in some cases, they can act as a form of regularization, particularly evident in the Llama 2 7B model. Spatial proximity emerges as a critical factor, implying that clean examples should be placed closer and noisy ones farther during in-context example selection for optimal downstream performance. Furthermore, our findings suggest that examples from related tasks can suffice for ICL in MT, with the constraint that the target language of the demonstrations matches the language of the test example while the choice of the source language is inconsequential. The directionality of the demonstrations has minimal impact, with both target-original and source-original demonstrations being equally effective. ICL has the potential to override semantic priors and may be exploited to misguide models or generate misinformation. We believe our study offers valuable insights for practitioners and would be widely generalizable.

heading: 2 Related Works
text: In-context Learning: Large Language Models have demonstrated strong performance in various downstream tasks through supervised fine-tuning on task-specific data (Devlin et al., 2019; Raffel et al., 2020; Lewis et al., 2020; Radford et al., 2019). ICL (Brown et al., 2020; Dong et al., 2023; Liu et al., 2023; OpenAI et al., 2023; Chowdhery et al., 2022) is a training-free method that has emerged as a cost-effective approach given the increasing size of LLMs (≥ 7B parameters). ICL aims to implicitly learn the task-solving abilities through a few hand-crafted demonstrations. These emergent capabilities (Wei et al., 2022; Lu et al., 2023) are driven by the distribution of the pretraining data (Chan et al., 2022; Briakou et al., 2023). Intuitively, ICL can be considered analogous to performing a gradient descent implicitly during the inference (Akyürek et al., 2023; Li et al., 2023; Zhang et al., 2023b). Vilar et al. (2023) explored few-shot prompting strategies for MT task on PaLM (Chowdhery et al., 2022) suggesting that the downstream MT performance is largely correlated with the quality of demonstrations. Subsequently, several recent works have also explored ICL capabilities of LLMs for the MT task (Robinson et al., 2023; Zhang et al., 2023a; Zhu et al., 2023; Kumar et al., 2023). However, none of these works delve into a fine-grained examination of the underlying factors influencing the model performance; a gap that we fill with our study. Factors impacting in-context learning: Several aspects of the demonstrations, such as input distribution, output distribution, or input-output alignment, can play a significant role in the ICL performance of LLMs. While Min et al. (2022) observed limited significance of label correctness in demonstrations for classification tasks, however, Yoo et al. (2022); Kossen et al. (2023) argued that ICL is sensitive to input-label mapping and can affect downstream performance. Furthermore, Wei et al. (2023) showed that ICL can override semantic priors, being influenced even by semantically unrelated input-label mappings, particularly in larger models. While prior works (Min et al., 2022; Wei et al., 2023; Kossen et al., 2023) have explored various aspects of ICL for NLU tasks, it is crucial to examine its applicability to NLG tasks. Raunak et al. (2023), which is most related to our work, found that syntactic perturbations to demonstrations affect MT performance, with target text distribution having more impact than source text distribution. However, their experiments were limited to GPT-3 (Brown et al., 2020) with only 100 test examples per direction. Our study substantially expands on this by extensively experimenting across the multiple axes mentioned earlier. Perturbation and Robustness: Text perturbation attacks (Xu et al., 2021; Moradi and Samwald, 2021; Zhang et al., 2022; Niu et al., 2020; Salinas and Morstatter, 2024) are frequently used to assess the resilience of models to input alterations, distinguishing them from adversarial attacks (Michel et al., 2019; Garg and Ramakrishnan, 2020; Zhang et al., 2021) with their intent to simulate realworld noise rather than to explicitly deceive models. These attacks, applied at either word level (Zhao et al., 2018; Cheng et al., 2020) or character level (Belinkov and Bisk, 2018; Ebrahimi et al., 2018; Formento et al., 2023), assess NMT model robustness through strategies like introducing random or linguistically aware perturbations, such as frequently confused characters. Additionally, perturbations may also involve random removal or addition of punctuations (Formento et al., 2023). Closest to our work is the work by Moradi and Samwald
(2021), however, our approach differs from theirs as we only perturb demonstrations while leaving test examples as is.

heading: 3 Methodology
text: Our experimentation first focuses on understanding whether in-context learning for MT is instructiondriven or example-driven, and then expands on the latter, exploring various aspects of examples. We categorize our experimentation along two principal axes: instruction perturbations and demonstration perturbations.

heading: 3.1 Instruction Variations
text: LLMs require extensive task descriptions for controlled generations and aligned outputs (Sondos Mahmoud Bsharat, 2023). We intend to understand if instructions are really necessary or not and whether in-context exemplars can guide the model to perform the MT task in cases of suboptimal or confounding instructions. Keeping the demonstrations fixed, we experiment with the task instructions described below:
Standard instruction: The baseline condition, a default instruction, indicates the MT task is used.
No instruction: Only in-context exemplars are used without any explicit instructions to see if the models can infer the task implicitly.
Generic instruction: A non-task-specific generic instruction is used. This is similar to the previous setting, however, we add a generic instruction to infer the task from the exemplars.
Constrastive instruction: An instruction with the opposite translation direction than the in-context exemplars is used to assess the model’s susceptibility to off-target translations due to instruction changes.
Random instruction: This includes a random instruction from a pool of non-translation tasks, while the in-context exemplars still reflect the translation task, to see if the model is misled into performing a different task.

heading: 3.2 Demonstration Perturbations
text: In this set of experiments, we intend to investigate the impact of perturbing in-context demonstrations on the downstream MT performance, keeping instructions fixed. Specifically, our objective is to
ascertain whether the model can effectively follow clear instructions and perform the tasks or whether the inclusion of suboptimal in-context exemplars adversely influences the subsequent downstream MT performance. To achieve this, we homogeneously either perturb the source or target distribution of in-context demonstrations by introducing different types of errors. Our perturbation methods are limited by a noise budget and influence the lexical (alterations to individual words), syntactic (alterations to structure or ordering of words), and semantic (alterations to meaning) properties of the in-context demonstrations and are listed below:
Span Noise: Random contiguous segments of the original text are modified by string operations such as deletion or replacement of characters within the selected spans similar to Maurya et al. (2023) and Joshi et al. (2020). We consider the number of characters to be perturbed by uniformly selecting 1-3 gram spans and uniformly choosing to delete or replace with a single random character until the chosen budget is exhausted.
OCR: Words from the original text are uniformly selected, and operations, such as fusion with adjacent words or splitting into two words to simulate noise, are performed, which might commonly be introduced in the pipelines involving OCR systems. The noise percentage determines the degree of perturbation, affecting how many words are manipulated with a uniform probability of fusing consecutive words together or splitting a word into two parts.
Word Ordering: The original word order of some portions of the original text is disrupted. It is important to note that this perturbation can have different implications across different languages due to differing levels of word order flexibility. We uniformly sample a set of words from the original text based on noise budget for reordering by generating a new order for these words.
Word Duplication: Redundancy is added to the original text without altering its semantics by duplicating certain words. We uniformly choose a set of words for duplication based on noise budget.
Punctuation: The syntactic cues and rhythm of the original text are altered by either inserting or deleting existing punctuations present in the original text. We uniformly add punctuation to words lacking it in addition and uniformly delete from
the original text in case of deletion, given a noise budget.
Table 3 in Appendix E provides a categorization of various perturbation methods considered as a part of this study with regard to the properties it impacts.

heading: 3.3 Role of Demonstration Attributes
text: While perturbations influence exemplar quality, there are other important attributes such as source distribution, target distribution, spatial proximity of the demonstration, and alignment between incontext demonstrations and test examples. We focus on these to further delve deeper into which attributes of an in-context demonstration play a critical role in downstream performance.
Heterogeneous Perturbations: We perform heterogeneous perturbations to investigate if spatial proximity to the test example affects the downstream performance. Specifically, we consider cases involving a mix of clean and noisy demonstrations: {(k − 1 c, 1 n), (1 c, k − 1 n), (1 n, k − 1 c), (k − 1 n, 1 c)} where c and n indicate clean and noisy and k denotes number of shots.
Directionality: While the influence of test set directionality and translationese effects on traditional MT is well-explored (Zhang and Toral, 2019; Federmann et al., 2022), its impact on the performance of LLMs using in-context examples remains largely unexplored (Raunak et al., 2023). We explore this by choosing in-context exemplars from source-original and target-original sets to analyze whether leveraging target-original in-context examples is a superior choice than source-original in-context examples.
Demonstrations from Allied Tasks as Proxy: When demonstrations for the specific task are not available during inference, a key question arises: Can demonstrations from a related task serve as a proxy for the model? This is particularly relevant in MT, where obtaining demonstrations for every low-resource language pair might not be feasible. In this scenario, if the translation direction at test time is from language X to Y, we provide exemplars from language A to Y to see if these are sufficient to guide the model in generating language Y effectively. Furthermore, we also experiment with various auxiliary languages as the source for these in-context exemplars to determine if the
choice of auxiliary language affects downstream performance. Specifically, we consider whether the auxiliary language a) matches the test time source language (baseline), b) matches the script of the test time source language and is not the test time source language, c) is English, d) is a randomly selected non-English language that differs in script from the test time source language.
Misalignment: Effective in-context learners should extract pertinent information from the context and provide precise responses. However, this trait can be exploited by intentionally injecting misinformation into the examples. Therefore, we aim to understand if models are susceptible to such attacks. This differs from prior perturbation experiments (see Section 5.2.1), where attacks targeted either source or target distribution without introducing misinformation through alignment alterations. Our experiment emulates a pivot translation scenario. We present two in-context examples: the first in language X with its English translation and the second demonstrating the translation of the English sentence into language Y. During testing, the model is tasked with translating from language X to Y, resembling a multi-hop reasoning process. Here, we consider 4 alignment types: a) all aligned, b) pivot misaligned, c) pivot and target misaligned, d) target misaligned. Types a) and b) are unperturbed cases, while c) and d) present perturbations or misinformation.

heading: 4 Experimental setup
text: This section describes the different pre-trained models, evaluation benchmarks, in-context prompts, and decoding hyperparameters used for our current study.
Models: We conduct our experimentation on offthe-shelf open-source pre-trained LLMs like Llama 2 (Touvron et al., 2023), BLOOM (Fan et al., 2022) and their corresponding instruction-tuned variants, ALMA (Xu et al., 2023) and BLOOMZ (Muennighoff et al., 2023) respectively. Firstly, models like Llama 2 (Touvron et al., 2023) are predominantly trained with English or European languages and the tokenizer has poor fertility on Indic languages. Further, Ahuja et al. (2023) show that Llama 2 70B (Touvron et al., 2023) exhibits poor in-context MT performance on Indian languages. Hence, we evaluate Llama 2 (Touvron et al., 2023) and ALMA (Xu et al., 2023) only on European
languages and evaluate BLOOM (Fan et al., 2022) and BLOOMZ (Muennighoff et al., 2023) on the remaining three Indian languages. For consistency in our experiments, we add a task-specific finetuned baseline (BLOOM 7B FT) by fine-tuning the BLOOM 7B model on a subset of BPCC-seed data (Gala et al., 2023) using the same prompt structure as ALMA (Xu et al., 2023). The fine-tuning hyperparameters are outlined in Appendix A. Due to computational constraints, we restrict our experimentation to 7B models for a fair comparison across the base and instruction-tuned variants. We adopt a greedy decoding strategy without sampling to generate a maximum of 256 tokens with early stopping to ensure reproducibility in our results.
Languages: Broadly, our experimentation consists of 3 Indian languages like Bengali, Hindi, and Tamil, and 3 European languages like Czech, German, and Russian. However, we consider different sets of languages for certain experiments due to the nature of the experiment or the availability of the data. Table 4 in Appendix F provides the languages considered, and the test set for each experiment.
Benchmarks: FLORES-200 (Goyal et al., 2022; Costa-jussà et al., 2022) is a N-way parallel benchmark that covers 200 languages, including various low-resource languages. However, it is important to note that data from FLORES-200 (Goyal et al., 2022; Costa-jussà et al., 2022) has been consumed in the multitask fine-tuning xP3 mixture of BLOOMZ (Muennighoff et al., 2023), suggesting data contamination. Therefore, we do not consider FLORES-200 for the evaluation of Indian languages and instead, we use the newly released IN22-Gen (Gala et al., 2023). IN22-Gen is a 22- way parallel benchmark for Indian languages that focuses on demography-specific content and serves as an unbiased evaluation set for both BLOOM and BLOOMZ models.
Prompt Details: In order to maintain consistency in terms of evaluation, we follow the same prompt that was used for fine-tuning ALMA (Xu et al., 2023) across all different models considered in this work (see Appendix C). However, we find Llama 2 Chat models do not perform well with the above standard prompt in the monolithic format, which is similar to Sclar et al. (2023). Therefore, we resort to passing each demonstration as a user-assistant turn and task description in the system prompt in the chat format. We use uniformly sampled high-
contrastive generic no_instruct random standard 0
10
20
30
40
50
En-XX
contrastive generic no_instruct random standard 0
10 20 30 40 50 60
XX-En
Instruction Variations Ch
rF ++
sc or
e
BLOOM-7B BLOOMZ-7B
BLOOM-7B-FT Llama-2-7B
Llama-2-7B-Chat ALMA-7B
Figure 1: Comparison of ChrF++ scores for En-XX (top) and XX-En (bottom) across different instruction types for BLOOM and Llama 2 model families (averaged across different languages and shots).
quality translation pairs from the FLORES-200 dev set to compose the in-context demonstrations with k = {1, 4, 8} across various noise scales with δ = {0, 0.1, 0.25, 0.5, 0.75} for evaluating LLMs.
Evaluation: Following Sai B et al. (2023), we use the ChrF++ metric (Popović, 2017), computed using SacreBLEU1 (Post, 2018), as the primary metric for our analysis due to its strongest correlation with human judgments among automatic metrics.

heading: 5 Results and Analysis
text: We now describe results that attempt to reveal the various factors that influence ICL for MT.

heading: 5.1 Sensitivity to Instruction Variations
text: Figure 1 illustrates that base LLMs such as BLOOM-7B and Llama 2 7B exhibit less variability to instruction perturbations averaged across different shots. Furthermore, Figure 12 provides detailed results depicting performance variation across different shots subjected to instruction perturbation. We observe that chat models like BLOOMZ-7B and Llama 2 7B Chat show some sensitivity to instructions, particularly in a
1sacreBLEU ChrF++ signature: nrefs:1|case:mixed|eff:no|tok:flores200| smooth:exp|version:2.4.0
1-shot setting, but this sensitivity diminishes as the number of shots increases. On the other hand, task-specific fine-tuned models like ALMA and BLOOM-7B-FT might be expected to be highly sensitive to instructions due to their training on data with specific instruction formats. Surprisingly, these models remain largely unaffected by perturbed instructions, even in a 1-shot setting. Although contrastive or random instructions aim to induce off-target or irrelevant generations, we observe that all the models consistently perform the translation task in the appropriate language, albeit with slightly reduced quality, highlighting that incontext examples primarily influence task recognition and learning.

heading: 5.2 Example Perturbations
text: Having established that models are mostly immune to instruction perturbations and are influenced more by examples, we focus on the latter.

heading: 5.2.1 Homogeneous Perturbations
text: Figure 2 indicates that perturbing in-context examples appear to adversely impact performance despite clear instructions. It is important to note that these instructions do not suffice as substitutes for noisy in-context examples. Generally, perturbations to the target distribution have a more severe impact than those to the source distribution, however, we observe instances where the impact of the source distribution cannot be disregarded. For instance, the BLOOMZ-7B model and ALMA-7B are susceptible to source-side perturbations, necessitating dedicated experimentation to ascertain the significance of the source distribution. We explore this aspect further in Section 5.2.3. Span noise emerges as the most detrimental perturbation across different model families, with the exception of BLOOM-7B-FT and Llama 2 7B models.
Across various perturbation attacks, we observe that the task-specific fine-tuned BLOOM-7B-FT model is most robust, followed by the BLOOM-7B model, while the multitask fine-tuned BLOOMZ7B model is highly vulnerable. Llama 2 model generally demonstrates strong robustness to different attacks, except for the task-specific fine-tuned ALMA model, which is susceptible to span noise attack on the target side. Despite the relatively mild nature of word duplication attacks, the BLOOM7B models surprisingly show high susceptibility. BLOOMZ-7B is robust to all attacks in XX-En translation, except for the span noise attack. Sur-
prisingly, the Llama 2 7B model shows significant performance improvements when subjected to various attacks compared to the clean baseline across both translation and perturbation directions.
The current experiments primarily focus on homogeneous perturbations, indicating a detrimental impact on performance when in-context examples are perturbed uniformly. To investigate the potential influence of spatial proximity between perturbed in-context examples and the test examples, we delve into heterogeneous perturbations in Section 5.2.2. To limit the number of experiments due to computational constraints, we choose to exclude punctuation add and drop attacks, as the number of punctuations in a sentence is limited.

heading: 5.2.2 Heterogeneous Perturbations
text: We consider four cases categorized into two pairs based on the degree of noise perturbation: {(k − 1 c, 1 n), (1 n, k − 1 c) and {(1 n, k − 1 c), (k − 1 n, 1 c)} where c and n indicates clean and noisy. We find that placing noisy examples closer to the test example has a more detrimental impact than placing them farther away (see Figure 3). Among these cases, {(1 c, k−1 n)} is identified as the most detrimental. Consistent with prior findings, we note an improvement in Llama 2 7B performance when exposed to noise, with the most significant improvement occurring when the noisy examples are furthest from the test example (optimal conditions at (k − 1 c, 1 n)). Llama 2 7B Chat is fairly robust and demonstrates improvement upon perturbation, except for the most detrimental case. ALMA shows slight susceptibility across all variants, with the most severe impact observed in the most detrimental case. BLOOM-7B demonstrates robustness against source-side perturbations but is heavily affected by target-side perturbations, particularly when k − 1 noisy examples are closer to the test example. BLOOMZ-7B is generally affected across all settings, with the most severe impact in the most detrimental case. Conversely, BLOOM7B-FT also demonstrates a fair amount of robustness. Our experimental results provide empirical evidence to practitioners that in settings involving a mixed pool of noisy and clean in-context examples, the optimal approach is to position noisy examples at the beginning and cleaner examples closer to the test example to maximize performance.

heading: 5.2.3 Demonstrations from Allied Tasks
text: Prior experiments (see Sections 5.1 and 5.2.1) indicate that examples are more influential than instructions, and the target distribution of the example has broadly more importance than the source distribution. However, our focus has been limited to in-context examples of the same task during testing. To explore the limits of example-based learning, we aim to investigate if in-context examples from a different source language but the same target language suffice as a proxy for guiding the model in translating into the desired target language, thus probing the limits of example-based learning.
Our initial experiments, selecting an auxiliary language, showed that using in-context examples from a related task can generally serve as a suitable proxy for the target task, barring a few exceptions. To delve deeper, we considered 2 additional settings: using the same script as the test time source language and using English, the predominant language models were trained on, as the auxiliary source. Figure 4 shows that the source distribution of in-context examples has a marginal effect on downstream MT performance. This suggests that any auxiliary source language can generally guide the model adequately, although there are a few exceptions, such as Llama 2 7B where using the same script as the auxiliary language outperforms English, and BLOOMZ-7B where a randomly chosen auxiliary language surpasses even using the same language. This suggests the limited impact of the choice of source language (source distribution). We conclude that in the absence of demonstrations
X-Y
BLOOM-7B BLOOMZ-7B
BLOOM-7B-FT Llama-2-7B
Llama-2-7B-Chat ALMA-7B
from the exact pair as the test examples, demonstrations from any alternative source language can serve as a viable substitute, yielding comparable performance to that of the same source language baseline.

heading: 5.3 Misalignment
text: In this experimental setup, we aim to understand if semantic priors in models can protect against susceptibility to misinformation presented in context, rendering them effective and robust in-context learners. It is evident that most models, except Llama 2 7B Chat, can form transitive associations to varying degrees (see Figure 5). Models from the BLOOM family and ALMA are particularly susceptible to extensive copy-pasting, suggesting they can be easily misled by in-context demonstrations, especially in cases of the target being misaligned. Notably, the ALMA model heavily relies on the English sentence as a pivot for these associations, and perturbing the English example also severely impacts performance despite the target translation being present in the context. Llama 2 7B exhibits poor translation quality, being unable to deduce answers from the context using transitive relations in aligned cases, and the performance further degrades in misaligned cases, indicating vulnerability to perturbations. In contrast, Llama 2 7B Chat remains largely unaffected by misinformation-
based perturbations, showing robustness but poor downstream performance, indicating inadequacy in building transitive associations in the case of aligned examples. An ideal model should utilize its inherent biases to regulate its reliance on the information presented in context. Current 7B LLMs fall short in this regard, highlighting the necessity for future research to develop models that excel as in-context learners while minimizing susceptibility to misinformation within the context.

heading: 5.4 Directionality of ICL Demonstrations
text: Sections 5.2.1 and 5.2.3 shows the significance of target distribution; therefore, we further study if target-original demonstrations are superior to source-original demonstrations, depending on the translation direction. Focusing on Indic languages, we utilize an in-house multi-domain Indic Original set for sourcing Indic Original demonstrations and FLORES-200 dev set (Goyal et al., 2022; Costajussà et al., 2022) for English original ones. IN22Gen (Gala et al., 2023), which is English-original, serves as the test set. The Indic Original set2 serves as the target original set for En-XX direction and the source original set for XX-En, while FLORES200 dev set (Goyal et al., 2022; Costa-jussà et al., 2022) acts as the target original set for XX-En direction and source original set for En-XX direction.
From Figure 6, across both En-XX and XX-En
2Will be released later as a held-out set of https://www2. statmt.org/wmt24/multiindicmt-task.html
translation directions, we observe a minimal difference in the performance when demonstrations are chosen from the respective source original and target-original sets for all BLOOM family models considered. This suggests that the directionality of demonstrations may not have much significance on the downstream performance. However, it’s important to acknowledge that our findings are limited to experiments conducted for Indic languages and using automatic metric-based evaluations, which may not sufficiently capture whether using targetoriginal in-context examples has any impact on the fluency of the generation or not. Further investigation through human evaluation is necessary to ascertain this, an aspect we leave for future work.

heading: 6 Conclusion
text: Although ICL has demonstrated good performance and is widely employed across different tasks, there is limited understanding of the aspects that influence the downstream performance of ICL. In this work, we aim to bridge this gap for MT by investigating the central question of whether ICL for MT is example-driven or instruction-driven and subsequently examining the role of different aspects of ICL. We find that: (1) ICL is primarily example-driven and choice of instruction has limited impact, with target distribution of the demonstrations being the most influential (2) Perturbation to demonstrations is generally detrimental but can have a regularization effect in some cases (3) Spatial proximity to the test example is an important factor in ICL (4) Demonstrations having the same target language can serve as a reliable proxy and choice of the source language of the demonstration is inconsequential (5) The directionality of the demonstrations has minimal impact. (6) ICL can potentially be exploited to mislead even smaller models, highlighting the need for further research to make models robust in-context learners.

heading: 7 Limitations
text: Our perturbation experiments focused on only 5 non-linguistic perturbations, which involved random textual attacks that were mostly uniform and language-agnostic in nature. However, it would also be interesting to explore linguistically aware perturbations such as causality alternation, entity replacement, and number replacement, similar to Chen et al. (2023). This would test if the model is susceptible to more fine-grained and subtle errors.
Additionally, our investigation was limited to the 7B model scale due to computational constraints, and exploring whether our current findings generalize across larger model scales would provide us a more comprehensive understanding of whether ICL capabilities models vary with scale. Furthermore, while the focus of this study was on the MT task, subsequent research should examine the transferability of these insights to diverse natural language generation tasks.

heading: 8 Ethical Considerations
text: Potential toxic and hateful outputs: Our work focuses on the robustness of models to various factors influencing MT via ICL. One aspect of our exploration is via perturbation which may lead to hallucinations as well as generate toxic and hateful content as a result of the model’s distributions being perturbed. This may even be applicable to other generative tasks but that is not the intention of our work. Safety Circumvention and Jailbreaking: Adversarial perturbations might be able to circumvent a model’s safety parameters and enable the generation of biased and harmful outputs. We plan to release our perturbation scripts and resultant perturbed data for research, and we do not intend it to be used to perform adversarial attacks in practice intended to undo the security features, also known as jailbreaking, of a model.

heading: 9 Acknowledgements
text: We sincerely thank Varun Gumma (SCAI Fellow, Microsoft Research) for the insightful discussions and his generous review of this draft, which helped us organize our ideas more effectively and improve its overall readability.

heading: A Fine-tuning BLOOM 7B on MT
text: We fine-tune BLOOM (Fan et al., 2022) 7B model on a human-annotated subset of BPCC (Gala et al., 2023), namely BPCC-seed and this forms the taskspecific fine-tuned baseline for the BLOOM family. Table 1 and Table 2 list the data scales and hyperparameters used for fine-tuning respectively.

heading: B Use of model-based metrics
text: Model-based metrics such as COMET (Rei et al., 2022) have demonstrated a good correlation with human judgments across several languages (Kocmi et al., 2021). However, these metrics may not be well-calibrated for all the languages considered in this study (Gala et al., 2023). Furthermore, the scale of our experiments makes it impractical to compute model-based metrics like COMET due to the additional computational overhead needed. Consequently, we use the best string-based metric, ChrF++ (Popović, 2017) as outlined in (Sai B et al., 2023). For completeness, we also compute the COMET scores using the wmt22-comet-da3 model for one of our primary experimental setups 3https://huggingface.co/Unbabel/ wmt22-comet-da
contrastive generic no_instruct random standard 0.0
0.2
0.4
0.6
0.8
En-XX
contrastive generic no_instruct random standard 0.0
0.2
0.4
0.6
0.8
XX-En
Instruction Variations CO
M ET
sc or
e
BLOOM-7B BLOOMZ-7B
BLOOM-7B-FT Llama-2-7B
Llama-2-7B-Chat ALMA-7B
Figure 7: Comparison of COMET scores for En-XX (top) and XX-En (bottom) across different instruction types for BLOOM and Llama 2 model families (averaged across different languages and shots).
(instruction perturbation) and found the trends to be consistent with the ChrF++ metrics. This suggests that ChrF++ can serve as a reliable proxy, indicative of the overall trends. Figure 7 shows the trends in COMET scores for the instruction perturbation experiment, clearly mirroring the trends in ChrF++ scores presented in Figure 1.

heading: C Prompt Templates
text: We use the standard prompt template (Figure 8) for most of our experiments except for experiments related to Sections 5.2.3 and 5.3. In the case of experiments in Section 5.2.3, we do not specify the source language in the instruction similar to the prompt illustrated in Figure 9. For the experiments in Section 5.3, we use the prompt template specified in Figure 10.
D Instruction Templates
We outline the different types of instructions considered in the instruction perturbation experiments mentioned in section 5.1. For each type of instruction, an example is provided in Appendix C. Additionally, for the random instruction type, any one of the prefix instructions is selected at random.

heading: E Examples of Perturbation Variants
text: Table 3 categorizes the different perturbations used in this study based on attributes they impact, along with an example.

heading: F Languages and Directions Considered
text: Table 4 describes the languages considered and respective benchmarks used for each experiment
conducted as a part of this study.

heading: G Additional results
text: Figures 12 to 18 illustrate fine-grained details such as shot-wise and case-wise trends for the aspects outlined in Section 3. The overall trends have been described in Section 5.

heading: I am hesitating between two options. Help me choose the more likely cause or effect.

heading: Future Work
text: Further investigation through human evaluation is necessary to ascertain this, an aspect we leave for future work