The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis (2024)

Abstract: August 11-16, 2024 ©2024 Association for Computational Linguistics The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis Miaoran Zhang1 Vagrant Gautam1 Mingyang Wang2,3,4 Jesujoba O. Alabi1 Xiaoyu Shen5 Dietrich Klakow1 Marius Mosbach6 1Saarland University, Saarland Informatic Campus 2Bosch Center for AI 3LMU Munich 4Munich Center for Machine Learning (MCML) 5Eastern Institute of Technology, Ningbo 6Mila, McGill University {mzhang,vgautam,jalabi,dietrich.klakow}@lsv.uni-saarland.de mingyang@cis.lmu.de xyshen@eitech.edu.cn marius.mosbach@mila.quebec

heading: 1 Introduction
text: An intriguing property of large language models (LLMs) is their ability to perform in-context learning (Brown et al., 2020), i.e., solve a task conditioned on a few demonstrations at inference time, without updating the model parameters. It has been shown to be an efficient alternative to finetuning when adapting models to diverse tasks and domains (Dong et al., 2022; Min et al., 2022b; Si et al., 2023, inter alia). In light of the success of incontext learning, there has been increased interest
∗Corresponding author. 1We release our code publicly at https://github.com/
uds-lsv/multilingual-icl-analysis.
in better understanding the factors that influence its success, such as demonstration selection (Liu et al., 2022; Rubin et al., 2022; Wang et al., 2023c), prompt design (Min et al., 2022a; Wei et al., 2022), and more generally on understanding how and why in-context learning works (Xie et al., 2022; Bansal et al., 2023; Hendel et al., 2023; Pan et al., 2023; Wang et al., 2023b).
However, most recent work on in-context learning predominantly focuses on English, and the exploration of multilingual in-context learning generally lags behind. This is problematic, as results that apply to English might not hold for other languages, especially those that are less represented in LLM training data. While there have been a few studies on in-context learning that go beyond English, they either focus on benchmarking LLMs on multilingual tasks without in-depth exploration, e.g., MEGA (Ahuja et al., 2023) and BUFFET (Asai et al., 2023), or zoom in on specific capabilities such as mathematical reasoning (Shi et al., 2023b), machine translation (Zhu et al., 2023; Agrawal et al., 2023), or code-switching (Zhang et al., 2023).
In this work, we take a multidimensional approach (Ruder et al., 2022) that unifies these strands of research and comprehensively evaluate the multilingual in-context learning abilities of LLMs. We focus on dissecting the actual impact of in-context demonstrations, which is crucial for understanding model behaviour. Our research covers various models, tasks, and languages, and we seek to answer the following research questions:
1. Does multilingual performance benefit from demonstrations? (§4)
2. Does demonstration quality matter? (§5) 3. What is the interplay between demonstrations
and templates? (§6) 4. How do the answers to these questions vary
across languages and models? (§4, §5, §6)
7342
Specifically, we address our research questions by evaluating 5 LLMs including base models that are only pre-trained on unlabeled text corpora (XGLM and Llama 2), and chat models that are further refined with instruction tuning and reinforcement learning (Llama 2-Chat, GPT-3.5, and GPT-4). We evaluate on 9 multilingual datasets that include both classification and generation tasks, covering 56 typologically different languages.
Our main findings are: (1) The effectiveness of demonstrations varies widely depending on the model, task, and language used. For base models, in-context learning barely outperforms zeroshot learning on many tasks. In general, in-context learning matters more for generation tasks with loosely-specified prompts; (2) Even with sophisticated demonstration selection methods, in-context learning is not always beneficial and can sometimes be worse than using no demonstrations at all; (3) Chat models are less sensitive to seeing correctly-labeled demonstrations than base models, suggesting that for the former, demonstrations primarily help the model understand the task format, while for the latter, demonstrations also impart taskspecific knowledge; (4) Using a formatting-focused template can even eliminate the need for demonstrations with chat models. The relative significance of demonstrations versus prompt templates varies based on inherent model capabilities.
In sum, we suggest that the benefits of adding demonstrations may be overestimated. Future work on in-context learning should carefully compare their results with zero-shot learning and on multiple
templates to faithfully represent its effectiveness. Given the vast variance across models, tasks, and languages, it is also important to cautiously frame claims about in-context learning.

heading: 2 Preliminaries

heading: 2.1 In-context learning
text: In-context learning (ICL) is a popular inference strategy where models solve2 a task without any parameter updates (Brown et al., 2020). Instead, the model performs the task by conditioning on labeled demonstrations. Demonstrations are typically formatted using “pattern-verbalizer pairs,” as this has been shown to be effective in eliciting good task performance (Schick and Schütze, 2021; Bach et al., 2022). Here, a pattern is used to format the input for the model, and a verbalizer maps the label to a textual representation. Additionally for instruction-tuned LLMs, a task instruction is often added to provide information about the task beyond individual demonstrations (Mishra et al., 2022b; Wang et al., 2022; Ouyang et al., 2022).
Formally, given a test sample xt, k demonstrations {(xi, yi)}ki=1, a pattern P , a verbalizer V and a task instruction I, the model (parameterized by θ) makes its prediction as follows:
yt ∼ pθ(y|I, {(P(xi),V(yi))}ki=1,P(xt)). (1) 2The extent to which models actually “solve” tasks is an open question as ICL, similar to fine-tuning, has generalization issues despite its impressive results (Mosbach et al., 2023). Regardless, we use the word “solve” in the rest of this paper for simplicity.
Taken together, the pattern, the verbalizer and the optional task instruction comprise the template with which demonstrations and the test sample are formatted as the input prompt for model inference. The effectiveness of demonstrations is thus linked with the template used to present them to the model.

heading: 2.2 Multilingual prompting
text: Previous studies highlight that the selection of demonstrations and prompt templates can significantly influence model performance (Liu et al., 2022; Fu et al., 2023b; Sclar et al., 2024). In multilingual in-context learning, the variation in input prompts is further complicated by the language of demonstrations, templates and test samples, all of which are important design choices.
For the template language, Lin et al. (2022) and Ahuja et al. (2023) found that English templates generally perform better than native language templates, possibly due to superior instructionfollowing abilities on existing LLMs on English compared to other languages. Following this, we use English templates in our study.
For the language of few-shot demonstrations and test samples, there are three popular settings. Given a test sample in a certain language, the most straightforward approach is to use demonstrations in the same language (referred to as in-language demonstrations). This setting directly measures the model’s inherent ability to solve problems in that language. Another choice is to use English demonstrations regardless of the language of the test sample. This is a cross-lingual transfer setup, where the goal is to transfer knowledge from a pivot language to a target language via in-context learning. As highlighted in Shi et al. (2023b) and Ahuja et al. (2023), in-language demonstrations often outperform English demonstrations on diverse multilingual tasks. Yet another option is to translate the test sample into English – an approach called translate-test, where the demonstrations are also in English. While translate-test leads to strong performance (Ahuja et al., 2023), this approach heavily relies on a translation system for data processing and centers the English proficiency of LLMs. In this work, we are interested in dissecting the intrinsic multilingual capabilities of LLMs, therefore we choose to use in-language demonstrations.
All these design choices are represented visually in Figure 1, which gives an overview of multilingual in-context learning. Detailed setup information is provided in the next section.

heading: 3 Experimental setup
text: Models. We evaluate two types of LLMs: pretrained base models and chat models. Our base models include XGLM (Lin et al., 2022) and Llama 2 (Touvron et al., 2023). Our chat models are Llama 2-Chat, GPT-3.5 (Ouyang et al., 2022) and GPT-4 (OpenAI et al., 2023). Specifically, we use xglm-7.5B, Llama-2-13b, and Llama-2-13b-chat on Huggingface (Wolf et al., 2020), and we access gpt-3.5-turbo-16k and gpt-4-32k APIs via Microsoft Azure.3
Tasks and datasets. We experiment on a diverse range of multilingual classification and generation tasks, using 9 datasets covering 56 languages in total. Our dataset selection largely follows MEGA (Ahuja et al., 2023), but we add datasets for extremely under-represented African languages. Our classification tasks include natural language inference (NLI), paraphrase identification, commonsense reasoning and sentiment analysis, with the following datasets: XNLI (Conneau et al., 2018), IndicXNLI (Aggarwal et al., 2022), PAWS-X (Yang et al., 2019), XCOPA (Ponti et al., 2020), XStoryCloze (Lin et al., 2022) and AfriSenti (Muhammad et al., 2023). Our generation tasks are extractive question answering (QA) and machine translation (MT), for which we use XQuAD (Artetxe et al., 2020), TyDiQAGoldP (Clark et al., 2020), and MAFAND (Adelani et al., 2022). See Appendix A.1 for more details.
In-context learning. For each test sample, we select k ∈ {0, 2, 4, 8}4 different demonstrations, which are randomly sampled unless otherwise specified. All demonstrations are in the same language as the test sample, and all templates are in English. We employ appropriate task-specific templates for different model types. All templates and data splits are shown in Appendix A.2.
Metrics. For classification tasks, we report the rank classification accuracy5 for open-source base models (Muennighoff et al., 2023; Lin et al., 2022).
3We also experiment with BLOOMZ and mT0 (Muennighoff et al., 2023). Results in Appendix B.1 show that their zero-shot performance significantly surpasses few-shot performance, which we ascribe to their training scheme.
4For QA datasets, we select a maximum of 4 demonstrations due to context size limitations.
5The scoring function is the average of per-token log probabilities (ignoring the common prefix of different candidates). The candidate with the highest score is chosen as the prediction.
For chat models, we measure the exact match between generated outputs6 and verbalized labels (Ahuja et al., 2023). As for generation tasks, we use the F1 score for QA datasets and ChrF++ score (Popović, 2017) for MAFAND. Implementation details for our evaluation are provided in Appendix A.3.

heading: 4 Do (more) demonstrations benefit multilingual performance?
text: In this section, we systematically compare ICL and zero-shot learning as this question is underexplored in previous studies of multilingual ICL (Ahuja et al., 2023; Asai et al., 2023). We examine model performance on diverse multilingual tasks while varying the number of demonstrations, and show the results for classification tasks and generation tasks in Figure 2.
We begin with the overall trends across models and datasets. OpenAI’s GPT-3.5 and GPT-4 models achieve the best multilingual in-context learning performance on all our datasets, which is unsurprising as they are currently the state-of-the-art on a large suite of NLP benchmarks. The next best models are Llama 2 and Llama 2-Chat, which demonstrate comparable or superior performance to the multilingual XGLM model despite being trained primarily on English corpora (Touvron et al., 2023). This indicates that their task-solving abilities can transfer across languages. Regardless of the model, however, performance on the AfriSenti
6We extract verbalized labels from the generated outputs using regular expressions before calculating the exact match.
and MAFAND datasets, particularly when translating English to African languages, lags significantly behind other tasks, showing that language discrepancies remain even in the best models.
An important pattern across datasets and models is that in-context learning does not always improve over zero-shot learning – in particular, it helps with generation tasks, but results on classification tasks are mixed. For the AfriSenti dataset, many models show noticeable improvements with ICL. However, with other tasks such as IndicXNLI, XNLI and PAWS-X, the same models, especially base models, perform much worse compared to the zero-shot setting. We also see marginal improvements in some cases, e.g., XGLM and Llama 2 on XCOPA. In comparison to chat models, the addition of demonstrations typically reduces the performance of base models across many tasks.
When examining the cases where ICL improves performance, we see that improvements saturate quickly with 2 to 4 demonstrations. This aligns with Chen et al. (2023), who found that reducing the number of demonstrations to one does not significantly deteriorate chain-of-thought reasoning.
Looking at the improvements over zero-shot performance (for all models and languages combined) across tasks in Figure 3, we observe that there are large fluctuations between individual languages that are not captured by the average. The PAWS-X dataset in particular shows an average degradation, but in fact some languages benefit from ICL while others degrade. For a more nuanced understanding of language-specific differences within a task, we zoom into this dataset in Figure 4 to inspect these language-specific differences.7 We see that
7Plots for other datasets are provided in Appendix B.2.
languages and models can behave very differently even on just one dataset, and a pattern which holds for one language with one model does not necessarily apply to a different language. For example, the ICL performance of Llama 2 outperforms its zeroshot performance by 2.3 points on Japanese and 1.3 points on Korean. However, demonstrations degrade performance for other languages, e.g., English performance degrades by 10.3 points. In sum, the effectiveness of demonstrations varies widely depending on the model, task, and language.

heading: 5 Does demonstration quality matter?
text: Our previous experiments evaluated ICL using randomly selected demonstrations. To ablate for the effects of demonstration quality, this section experiments with the choice of demonstrations as well as the importance of ground truth labels, i.e., the inputlabel mapping. Inspired by work on demonstration selection (Liu et al., 2022; Rubin et al., 2022) and input-label mapping (Min et al., 2022c; Yoo et al., 2022) in English, we compare the following three types of demonstrations:
• RANDOM: demonstrations are randomly selected from clean data
• TOP-K: the k most semantically similar8 examples to a given test sample are selected (Liu
8We quantify semantic similarity using LaBSE (Feng et al., 2022), a multilingual sentence embedding model trained on 109+ languages.
et al., 2022)
• RANDOM-CORRUPTED: demonstrations are randomly selected but the labels are corrupted by replacement with random labels9 (Min et al., 2022c)
Table 1 shows that top-k selection performs better than random selection in many cases, especially for the base models XGLM and Llama 2. For chat models, the largest improvements are on generation tasks. For example, GPT-3.5 achieves a 2.98-point improvement on TyDiQA. Nevertheless, top-k selection often degrades performance on many other tasks, e.g., GPT-3.5 is 2.07 points worse on PAWSX compared to random selection. When compared to zero-shot performance, ICL with top-k selection is even worse in some cases, such as XGLM on PAWS-X and XStoryCloze. In cases where random selection performs worse than zero-shot, even top-k selection gives only marginal improvements (see detailed numbers in Table 5 in Appendix C.1). These findings indicate that sophisticated demonstration selection methods are not always beneficial and can sometimes be worse than using no demonstrations at all.
Exploring this further, in Table 2, we compare randomly selected demonstrations with ground truth labels and corrupted labels. We find that using corrupted labels does not hurt performance on multilingual classification tasks much, which is consistent with previous research on English (Min
9For classification tasks, we randomly choose a label from the fixed label set. For generation tasks, we randomly choose a label from the label space of the entire demonstration data.
et al., 2022c). On generation tasks, however, all models perform worse with corrupted labels, but to vastly different extents. XGLM and Llama 2 perform significantly worse with corrupted labels, especially on the machine translation task, whereas chat models do not rely as much on correct labels. This might be explained by ICL helping the model understand the task format and activating prior knowledge acquired by the model, rather than the model learning the task from demonstrations. The observed model insensitivity to correct labels on certain tasks implies that random labels can serve as a strong baseline for demonstration generation before exploring more complex methods (Lyu et al., 2023; Wan et al., 2023).
To investigate how these patterns split up across languages, Figure 5 shows language-specific results on AfriSenti and XQuAD with Llama 2 and GPT-3.5.10 On AfriSenti, top-k selection outperforms random selection with Llama 2 across most languages; however, in the case of Swahili and Tsonga, there is a performance drop of 3.2 and 1.2 points, respectively. With GPT-3.5, top-k selection does not help across most languages, but it does help with Mozambican Portuguese and Twi. Similarly, the impact of corrupted labels varies. Llama 2 is affected dramatically by corrupted labels on all languages in XQuAD, whereas GPT-3.5 is much less affected, although to varying degrees across different languages. We urge NLP practitioners to attend to these discrepancies when creating language-specific applications, and leave it to future work to explore where they come from.
10See Appendix C.2 for other models and datasets.

heading: 6 Better templates further reduce the benefits of demonstrations
text: In-context learning performance depends not only on the demonstrations, which we have varied so far, but also on how they are formatted using templates. Previous work (Gonen et al., 2023; Mizrahi et al., 2024) has shown that modifying the template changes task performance. This section thus seeks to examine the interplay between template choice and demonstrations.
Template design. In the zero-shot setting, we observe that chat models tend to generate verbose responses (e.g., “Sure! I can help you with that”) or explanations (e.g., “The reason is that ...”) that pose a challenge for automatic evaluation. We observe a reduction in this behaviour with ICL, which leads us to question whether demonstrations are merely a means to format model responses. To see if we can achieve the same effect with minor template engineering, we augment the original templates with instructions that focus on output formatting. We call these formatting-focused templates which are shown in Table 9.
In this section, we focus on XCOPA, AfriSenti, XQuAD, and TyDiQA, as these are the classification and generation tasks that seem to benefit most from in-context demonstrations (see Section 4). However, as Figure 6 shows, the performance gap between zero-shot and in-context learning diminishes with formatting-focused templates. The gap reduction is more substantial for QA datasets (i.e., the generation tasks) than for XCOPA and AfriSenti (i.e., the classification tasks). We speculate that it is simpler for the model to generate label words for classification tasks with a pre-defined label space than to answer questions in a way that is easy to evaluate automatically. In the latter case, formatting-focused templates can teach output styling, largely eliminating the benefits of demonstrations.
Compared to GPT-3.5 and GPT-4, Llama 2-Chat performs worse in both zero-shot and few-shot settings, and formatting-focused templates have a less pronounced impact. On QA datasets, GPT-3.5 and GPT-4 even achieve better zero-shot performance with formatting-focused templates than ICL with original templates, a pattern that is not observed with Llama 2-Chat. This suggests that the relative significance of demonstrations and templates varies based on the inherent abilities of models at solving tasks and following instructions.
With our new formatting-focused templates, we revisit the impact of the input-label mapping discussed in Section 5. As Table 3 shows, all models perform worse with corrupted labels, but formatting-focused templates largely mitigate this degradation. Notably, GPT-4 using corrupted labels performs on par with ground truth labels. This strengthens our finding that the correct inputlabel mapping is not that important, while also highlighting the crucial role that templates play in in-context learning.
Figure 7 shows the language-specific effects of formatting-focused templates on XQuAD (results for other tasks are in Appendix D.1). For Llama 2-Chat, demonstrations remain essential even with
a formatting-focused template for most languages, but not Greek and Hindi. GPT-3.5 and GPT-4 also show variance across languages. Moreover, for most languages, zero-shot learning with minor template engineering can match and even exceed in-context learning performance, aligning with previous work on GPT-3 (Reynolds and McDonell, 2021). The fact that we can achieve the same effects through template engineering or demonstrations reinforces our hypothesis that models are not actually learning tasks on the fly. Instead, some combination of demonstrations and templates serves to activate prior knowledge of a task and encourage a consistent output format for automatic evaluation.

heading: 7 Discussion
text: Our systematic study provides strong evidence that the importance of in-context demonstrations on existing multilingual datasets might be overestimated, as it highly depends on the model, task, and language used. For strong instruction-following models, the effect of demonstrations is superficial and can be eliminated with minor template engineering. These findings open up new questions, which we discuss below.
Understanding the failures of ICL. There has been a surge of research interest in understanding
the underlying mechanisms of ICL (Xie et al., 2022; Von Oswald et al., 2023; Wang et al., 2023b; Hendel et al., 2023), motivated by its successes. Our results show that ICL is not always effective, and that its performance changes depending on multiple factors including the choice of model, task and language. The failures of ICL need as much scrutiny as its successes for a more fundamental understanding of the learning mechanisms of LLMs.
Optimizing demonstrations or templates. With the increasing popularity of research on demonstration selection (Liu et al., 2022; Rubin et al., 2022; Li et al., 2023b) and prompt engineering (Mishra et al., 2022a; White et al., 2023; Khattab et al., 2023), it is important to understand the interplay of the two. We show that good demonstrations help base models perform better on certain tasks, but that formatting-focused prompting has a much bigger impact on chat models. These results show that the impact of demonstrations cannot be fairly evaluated in isolation from the choice of prompt. These findings have implications both for researchers interested in fairly evaluating ICL, and for practitioners to choose to spend time optimizing demonstrations, templates or both.
Evaluating multilingual ICL. Compared to the extensive research on ICL in English (Zhao et al., 2021; Dong et al., 2022; Min et al., 2022b; Mosbach et al., 2023), multilingual ICL remains underexplored. There is no widely accepted setup to robustly evaluate the effectiveness of ICL across languages, since the choice of multilingual models and tasks is limited. Based on our findings, we have some recommendations for the nascent field of multilingual ICL. First, critical evaluation is important. We need to compare ICL strategies to zero-shot learning, and ablate them with multiple templates. Second, as there is so much variance across models, tasks and languages, it is important to carefully scope claims about ICL. Last but not least, every language is different, so granular per-language analysis is a must in multilingual research.

heading: 8 Related Work
text: Multilingual in-context learning. Most multilingual in-context learning studies focus on benchmarking LLMs on diverse tasks and comparing them with smaller fine-tuned models (Ahuja et al., 2023; Asai et al., 2023; Zhang et al., 2023; Zhu et al., 2023). As these works focus on benchmark-
ing, their analysis of the role of demonstrations is limited. Ahuja et al. (2023) explore different prompting strategies by adjusting the language of templates and demonstrations. Zhang et al. (2023) find that demonstrations sometimes do not contribute to or even degrade model performance on code-switching. Zhu et al. (2023) look at machine translation and analyze the effects of template and demonstration selection with XGLM. In the context of cross-lingual transfer, Shi et al. (2022), Tanwar et al. (2023), and Agrawal et al. (2023) investigate demonstration selection for specific applications. In contrast, we take a much broader perspective and investigate the actual impact of demonstrations across a wide range of models, tasks and languages.
English-centric demonstration analysis. Most of the current demonstration analysis literature focuses on English: Lu et al. (2022) analyze the sensitivity of ICL to the order of demonstrations, Min et al. (2022c) and Yoo et al. (2022) explore whether the ground truth labels matter for classification tasks, and Wei et al. (2023) investigate the sensitivity of various model families to different input-label mappings. Similarly, Pan et al. (2023) disentangle task recognition and task learning by manipulating the label space. Beyond this, Shi et al. (2023a) and Wang et al. (2023a) modify the validity of chain-of-thought (CoT) reasoning steps in demonstrations and explore the impact of this modification on mathematical reasoning. Also focusing on CoT, Chen et al. (2023) investigate how varying the number of demonstrations affects performance.

heading: 9 Conclusion
text: In this paper, we conduct an in-depth multidimensional analysis on the impact of demonstrations in multilingual in-context learning. We find that the use of demonstrations does not always provide benefits compared to zero-shot learning, and that there is a large variance in performance across models, datasets and languages. While the quality of demonstrations influences the performance of base LLMs on certain tasks, the impact is significantly reduced for LLMs tuned with alignment techniques. We also examine the interplay between demonstrations and templates, finding that a carefully crafted template can further decrease the benefits of demonstrations. Our granular analysis contributes novel insights with nuance and paves the way for a more thoughtful multilingual ICL evaluation.
Limitations
Data contamination. Since LLMs are trained with a vast amount of data scraped from the internet, this might result in data contamination, i.e., when the training data includes test datasets. Ahuja et al. (2023) suspect that many multilingual datasets appear in the training data of GPT-4, which might lead to an overestimation of the model’s capabilities. In the context of our work, our prompt might just be reminding LLMs of a task they have already seen, whereas on an unseen task, the impact of demonstrations might be different. We do not examine the impact of potential data contamination in our paper and leave an exploration of this to future work.
Other demonstration choices. In this work, we choose to use demonstrations that are in the same languages as the test sample, due to our focus on evaluating inherent multilingual abilities of LLMs, as explained in Section 2.2. However, using English demonstrations for cross-lingual transfer or translating test samples into English has its own practical value for NLP applications. Additionally, it is worth exploring selecting demonstrations from a mixture of languages. Expanding our study to more setups would provide additional insights into multilingual and cross-lingual LLM abilities.
Other prompting methods. In Section 6, we only experiment with manually augmented templates to illustrate how the choice of template can reduce the effectiveness of demonstrations. There is a broad literature on prompt engineering and prompt sensitivity (White et al., 2023; Gonen et al., 2023), suggesting that it is plausible that another prompt could reduce the gap between few-shot and zero-shot performance even further. Chain-ofthought (CoT) prompting is another approach with promising multilingual abilities (Shi et al., 2023b; Huang et al., 2023) that might affect our findings. Our manually-augmented templates are intended only as a starting point for further analysis, which we leave to future work.
Beyond automatic evaluation. When examining model responses, we noticed some cases where a correct answer as evaluated by a human was not fully captured by automatic evaluation metrics. Human evaluation is time-consuming, expensive, and hard to source for the wide range of languages that we explore in our work. Another option is LLM
evaluation, which is becoming increasingly popular (Fu et al., 2023a; Chan et al., 2024), but is also an expensive approach. More importantly, we have no guarantees about LLMs’ multilingual capabilities. As a trade-off between cost and evaluation quality, we stick to automatic evaluation in our work for all tasks and languages.

heading: Acknowledgments
text: We thank Matan Eyal for his valuable feedback. Our use of Microsoft Azure is sponsored by the Microsoft Accelerating Foundation Models Research (AFMR) program. Miaoran Zhang and Marius Mosbach received funding from the DFG (German Research Foundation) under project 232722074, SFB 1102. Vagrant Gautam and Jesujoba O. Alabi were supported by the BMBF’s (German Federal Ministry of Education and Research) SLIK project under the grant 01IS22015C.

heading: A Experimental details
text: A.1 Tasks and datasets We conduct experiments on 9 multilingual datasets with a wide coverage of tasks and languages, as shown in Table 4. All datasets are public research datasets and our experiments are consistent with their intended use, i.e., NLP evaluation. For the machine translation dataset MAFAND, English serves as the pivot language and there are two translation directions: en-xx (i.e., translating from English to another language) and xx-en (i.e., translating from another language to English). As the black-box training data of OpenAI APIs that we used is up to September 2021, we include the dataset release date in the table which can be taken as a clue to the severity of dataset contamination.
A.2 In-context learning We sample few-shot demonstrations from the validation set and evaluate the test set. For datasets without a test data split (XStoryCloze and TyDiQA), we sample few-shots from the train set and evaluate the validation set. Since XQuAD only has a validation data split, we utilize it for both demonstration sampling and evaluation, ensuring that the test sample itself is not included in its demonstrations. For chat models (Llama 2-Chat, GPT-3.5, and GPT-4), we limit the test sample size to a maximum of 200 in order to reduce inference expenses and ensure a fair comparison.
We use GPT-3 style prompting templates for XGLM and Llama 2 as shown in Table 6. The templates for BLOOMZ and mT0 are shown in Table 7. For Llama 2-Chat, GPT-3.5 and GPT-4, default templates are shown in Table 8 and task instructions are used to assign a system role to the model. Inspired by Lai et al. (2023) and Li et al. (2023a), where emotional stimuli are able to enhance LLM understanding, we design formatting-focused templates (discussed in Section 6) to reinforce LLM to generate formatted outputs that are easier to evaluate automatically, as shown in Table 9.
A.3 Implementation Our codebase is adapted from OpenICL (Wu et al., 2023). We use int8bit model quantization11 for all models except OpenAI models. Experiments are conducted using a single NVIDIA A100-80GB GPU. As models have a maximum context length, we preserve complete demonstrations that can fit within the context window. We employ greedy decoding for model generation. For chat models, the maximum new token is set to 50, while for machine translation, it is set to 100. For other models, the maximum
11In our preliminary experiments, we found that int8 quantization led to a performance degradation of 1-2% on a few classification datasets with Llama 2 and XGLM. Since this degradation is consistent across different setups, we believe that it would not affect our overall findings.
new token is set to 20, while for machine translation, it is set to 50. We use three seeds (0, 33, 42) in our experiments, and the single-seed results for BLOOMZ and mT0 are obtained with the seed 0.

heading: B More results for varying numbers of demonstrations
text: In this section, we provide supplemental results for Section 4.
B.1 Results for BLOOMZ and mT0 In addition to the 5 models (base models and chat models) we discussed in the main content, we also experiment with two instruction-tuned models: BLOOMZ and mT0 (Muennighoff et al., 2023). The results for varying numbers of random demonstrations are shown in Figure 8. In line with findings from Asai et al. (2023), we observe significant performance degradation when using demonstrations compared to zero-shot learning in all cases. This decline can be attributed to their training scheme, where models are fine-tuned on a large collection of existing datasets in a zero-shot manner. In contrast, several studies (Chen et al., 2022; Wang et al., 2022) focus on enhancing the in-context learning ability of LLMs by incorporating demonstrations into their training process. This suggests that we should be careful in model selection for in-context learning research and take the model training process into consideration.
B.2 Results for individual languages The language-specific results for each task are shown in Figure 9. The order of languages follows their data ratio in the CommonCrawl corpus12 from high-resource to low-resource. We observe large variations in model performance across different languages. For instance, there exists a large performance disparity between English and Urdu in XNLI. In XCOPA, the performance of Quechua is significantly worse compared to other languages.

heading: C More results for ablating the quality of demonstrations
text: In this section, we provide supplemental results for Section 5.
C.1 Performance of different types of demonstrations In Table 5, we show the model performance of three types of demonstrations, as well as the zero-shot performance for comparative analysis. As we notice, top-k selection may not always be the optimal choice, given the considerable effort in optimizing demonstrations. For QA, XGLM and Llama 2’s abilities in solving this task almost collapse with corrupted labels. However, for chat models, demonstrations with corrupted labels can achieve comparable performance with ground truth labels and largely improve the
12http://commoncrawl.org
zero-shot performance. Overall, the base models are more sensitive to the type of demonstrations than chat models.
C.2 Results for individual languages In Figure 10, we show the language-specific results for each task, in which we can see language discrepancies with different types of demonstrations.

heading: D The interplay between demonstrations and templates
text: In this section, we provide supplemental results for Section 6.
D.1 Results for individual languages We examine the effect of templates and show language-specific results for XCOPA, AfriSenti, XQuAD and TyDiQA in Figure 11. In a few cases, we found that formatting-focused templates lead to a decline in performance compared to original templates (e.g., Igbo and Mozambican Portuguese in AfriSenti with GPT-3.5). This can be attributed to the model’s sensitivity to prompts, highlighting the potential of automatic prompt engineering. Still, formatting-focused template can largely narrow the performance gap between 0-shot and 4-shot in a broad context.
0 2 4 8
40
60 80 A cc ur ac y
English
0 2 4 8
40
60
80
German
0 2 4 8
40
60
80
Russian
0 2 4 8
40
60
80
French
0 2 4 8
40
60
80
A cc
ur ac
y
Spanish
0 2 4 8
40
60
80
Chinese
0 2 4 8
40
60
80
Vietnamese
0 2 4 8
40
60
80
Turkish
0 2 4 8
40
60
80
A cc
ur ac
y
Arabic
0 2 4 8
40
60
80
Greek
0 2 4 8
40
60
80
Thai
0 2 4 8
40
60
80
Bulgarian
0 2 4 8
40
60
80
A cc
ur ac
y
Hindi
0 2 4 8
40
60
80
Urdu
0 2 4 8
40
60
80
Swahili XGLM
Llama 2
Llama 2-Chat
GPT-3.5
GPT-4
(a) XNLI
0 2 4 8
30
40
50
60
70
80
A cc
ur ac
y
Hindi
0 2 4 8
30
40
50
60
70
80
Bengali
0 2 4 8
30
40
50
60
70
80
Tamil
0 2 4 8
30
40
50
60
70
80
Marathi
0 2 4 8
30
40
50
60
70
80
A cc
ur ac
y
Malayalam
0 2 4 8
30
40
50
60
70
80
Telugu
0 2 4 8
30
40
50
60
70
80
Kannada
0 2 4 8
30
40
50
60
70
80
Punjabi
0 2 4 8
30
40
50
60
70
80
A cc
ur ac
y
Oriya
0 2 4 8
30
40
50
60
70
80
Assamese
0 2 4 8
30
40
50
60
70
80
Gujarati XGLM
Llama 2
Llama 2-Chat
GPT-3.5
GPT-4
(b) IndicXNLI
0 2 4 8
50
60
70
80
A cc
ur ac
y
English
0 2 4 8
50
60
70
80
German
0 2 4 8
50
60
70
80
Japanese
0 2 4 8
50
60
70
80
French
0 2 4 8
50
60
70
80
A cc
ur ac
y
Spanish
0 2 4 8
50
60
70
80
Chinese
0 2 4 8
50
60
70
80
Korean XGLM
Llama 2
Llama 2-Chat
GPT-3.5
GPT-4
(c) PAWS-X
0 2 4 8
20
40
60
80
100
A cc
ur ac
y
Chinese
0 2 4 8
20
40
60
80
100
Italian
0 2 4 8
20
40
60
80
100
Vietnamese
0 2 4 8
20
40
60
80
100
Indonesian
0 2 4 8
20
40
60
80
100
A cc
ur ac
y
Turkish
0 2 4 8
20
40
60
80
100
Thai
0 2 4 8
20
40
60
80
100
Estonian
0 2 4 8
20
40
60
80
100
Tamil
0 2 4 8
20
40
60
80
100
A cc
ur ac
y
Swahili
0 2 4 8
20
40
60
80
100
Haitian
0 2 4 8
20
40
60
80
100
Quechua XGLM
Llama 2
Llama 2-Chat
GPT-3.5
GPT-4
(d) XCOPA
0 2 4 8
20
40
60
80 100 A cc ur ac y
English
0 2 4 8
20
40
60
80
100
Russian
0 2 4 8
20
40
60
80
100
Spanish
0 2 4 8
20
40
60
80
100
Chinese
0 2 4 8
20
40
60
80
100
A cc
ur ac
y
Indonesian
0 2 4 8
20
40
60
80
100
Arabic
0 2 4 8
20
40
60
80
100
Hindi
0 2 4 8
20
40
60
80
100
Basque
0 2 4 8
20
40
60
80
100
A cc
ur ac
y
Telugu
0 2 4 8
20
40
60
80
100
Burmese
0 2 4 8
20
40
60
80
100
Swahili XGLM
Llama 2
Llama 2-Chat
GPT-3.5
GPT-4
(e) XStoryCloze
0 2 4 8
20
40
60
80
A cc
ur ac
y
Swahili
0 2 4 8
20
40
60
80 Amharic
0 2 4 8
20
40
60
80 Hausa
0 2 4 8
20
40
60
80 Kinyarwanda
0 2 4 8
20
40
60
80
A cc
ur ac
y
Yoruba
0 2 4 8
20
40
60
80 Tigrinya
0 2 4 8
20
40
60
80 Igbo
0 2 4 8
20
40
60
80 Oromo
0 2 4 8
20
40
60
80
A cc
ur ac
y
Moroccan Arabic
0 2 4 8
20
40
60
80 Algerian Arabic
0 2 4 8
20
40
60
80 Nigerian Pidgin
0 2 4 8
20
40
60
80 Mozambican Portuguese
0 2 4 8
20
40
60
80
A cc
ur ac
y
Tsonga
0 2 4 8
20
40
60
80 Twi
XGLM
Llama 2
Llama 2-Chat
GPT-3.5
GPT-4
(f) AfriSenti
0 2 4 0
20
40
60
80
F1
English
0 2 4 0
20
40
60
80
German
0 2 4 0
20
40
60
80
Russian
0 2 4 0
20
40
60
80
Spanish
0 2 4 0
20
40
60
80
F1
Chinese
0 2 4 0
20
40
60
80
Vietnamese
0 2 4 0
20
40
60
80
Turkish
0 2 4 0
20
40
60
80
Arabic
0 2 4 0
20
40
60
80
F1
Greek
0 2 4 0
20
40
60
80
Romanian
0 2 4 0
20
40
60
80
Thai
0 2 4 0
20
40
60
80
Hindi
(g) XQuAD
0 2 4 0
20
40
60
80
F1
English
0 2 4 0
20
40
60
80
Russian
0 2 4 0
20
40
60
80
Indonesian
0 2 4 0
20
40
60
80
Korean
0 2 4 0
20
40
60
80
F1
Arabic
0 2 4 0
20
40
60
80
Finnish
0 2 4 0
20
40
60
80
Bengali
0 2 4 0
20
40
60
80
Telugu
0 2 4 0
20
40
60
80
F1
Swahili XGLM
Llama 2
Llama 2-Chat
GPT-3.5
GPT-4
(h) TyDiQA
0 25
50
75
A cc
ur ac
y
XGLM
Zero Top-k Random Random-corrupted
0
25
50
75
A cc
ur ac
y
Llama 2
0
25
50
75
A cc
ur ac
y
Llama 2-Chat
0
25
50
75
A cc
ur ac
y
GPT-3.5
E ng
lis h
G er
m an
R us
si an
Fr en
ch
S pa
ni sh
C hi
ne se
V ie
tn am
es e
Tu rk
is h
A ra
bi c
G re
ek Th ai
B ul
ga ria
n
H in
di
U rd
u
S w
ah ili
0
25
50
75
A cc
ur ac
y
GPT-4
(a) XNLI
0
25
50
75
A cc
ur ac
y
XGLM
Zero Top-k Random Random-corrupted
0
25
50
75
A cc
ur ac
y
Llama 2
0
25
50
75
A cc
ur ac
y
Llama 2-Chat
0
25
50
75
A cc
ur ac
y
GPT-3.5
H in
di
B en
ga li
Ta m
il
M ar
at hi
M al
ay al
am
Te lu
gu
K an
na da
P un
ja bi
O riy
a
A ss
am es
e
G uj
ar at
i0
25
50
75
A cc
ur ac
y GPT-4
(b) IndicXNLI
0
25
50
75
A cc
ur ac
y
XGLM
Zero Top-k Random Random-corrupted
0
25
50
75
A cc
ur ac
y
Llama 2
0
25
50
75
A cc
ur ac
y
Llama 2-Chat
0
25
50
75
A cc
ur ac
y
GPT-3.5
E ng
lis h
G er
m an
Ja pa
ne se
Fr en
ch
S pa
ni sh
C hi
ne se
K or
ea n
0
25
50
75
A cc
ur ac
y
GPT-4
(c) PAWS-X
0
50
100
A cc
ur ac
y
XGLM
Zero Top-k Random Random-corrupted
0
50
100
A cc
ur ac
y
Llama 2
0
50
100
A cc
ur ac
y
Llama 2-Chat
0
50
100
A cc
ur ac
y
GPT-3.5
C hi
ne se
Ita lia
n
V ie
tn am
es e
In do
ne si
an
Tu rk
is h
Th ai
E st
on ia
n
Ta m
il
S w
ah ili
H ai
tia n
Q ue
ch ua
0
50
100
A cc
ur ac
y
GPT-4
(d) XCOPA
0 50
100
A cc
ur ac
y
XGLM
Zero Top-k Random Random-corrupted
0
50
100
A cc
ur ac
y
Llama 2
0
50
100
A cc
ur ac
y
Llama 2-Chat
0
50
100
A cc
ur ac
y
GPT-3.5
E ng
lis h
R us
si an
S pa
ni sh
C hi
ne se
In do
ne si
an
A ra
bi c
H in
di
B as
qu e
Te lu
gu
B ur
m es
e
S w
ah ili
0
50
100
A cc
ur ac
y
GPT-4
(e) XStoryCloze
0
25
50
75
A cc
ur ac
y
XGLM
Zero Top-k Random Random-corrupted
0
25
50
75
A cc
ur ac
y
Llama 2
0
25
50
75
A cc
ur ac
y
Llama 2-Chat
0
25
50
75
A cc
ur ac
y
GPT-3.5
S w
ah ili
A m
ha ric
H au
sa
K in
ya rw
an da
Yo ru
ba
Ti gr
in ya Ig bo
O ro
m o
M o.
A ra
bi c
A l.
A ra
bi c
N i.
P id
gi n
Po rt
ug ue
se
Ts on
ga Tw i0
25
50
75
A cc
ur ac
y GPT-4
(f) AfriSenti
0
25
50
75
F1
XGLM
Zero Top-k Random Random-corrupted
0
25
50
75
F1
Llama 2
0
25
50
75
F1
Llama 2-Chat
0
25
50
75
F1
GPT-3.5
E ng
lis h
G er
m an
R us
si an
S pa
ni sh
C hi
ne se
V ie
tn am
es e
Tu rk
is h
A ra
bi c
G re
ek
R om
an ia
n Th ai
H in
di
0
25
50
75
F1
GPT-4
(g) XQuAD
0
25
50
75
F1
XGLM
Zero Top-k Random Random-corrupted
0
25
50
75
F1
Llama 2
0
25
50
75
F1
Llama 2-Chat
0
25
50
75
F1
GPT-3.5
E ng
lis h
R us
si an
In do
ne si
an
K or
ea n
A ra
bi c
Fi nn
is h
B en
ga li
Te lu
gu
S w
ah ili
0
25
50
75
F1
GPT-4
(h) TyDiQA
0 25
50
75
100
A cc
ur ac
y
Llama 2-Chat
0-shot w/ original template 4-shot w/ original template
0-shot w/ formatting-focused template 4-shot w/ formatting-focused template
0
25
50
75
100
A cc
ur ac
y
GPT-3.5
C hi
ne se
Ita lia
n
V ie
tn am
es e
In do
ne si
an
Tu rk
is h
Th ai
E st
on ia
n
Ta m
il
S w
ah ili
H ai
tia n
Q ue
ch ua
0
25
50
75
100
A cc
ur ac
y
GPT-4
(a) XCOPA
0
20
40
60
80
A cc
ur ac
y
Llama 2-Chat
0-shot w/ original template 4-shot w/ original template
0-shot w/ formatting-focused template 4-shot w/ formatting-focused template
0
20
40
60
80
A cc
ur ac
y
GPT-3.5
S w
ah ili
A m
ha ric
H au
sa
K in
ya rw
an da
Yo ru
ba
Ti gr
in ya
Ig bo
O ro
m o
M o.
A ra
bi c
A l.
A ra
bi c
N i.
P id
gi n
Po rt
ug ue
se
Ts on
ga Tw i0
20
40
60
80
A cc
ur ac
y
GPT-4
(b) AfriSenti

heading: Future Work
text: We do not examine the impact of potential data contamination in our paper and leave an exploration of this to future work. Other demonstration choices. In this work, we choose to use demonstrations that are in the same languages as the test sample, due to our focus on evaluating inherent multilingual abilities of LLMs, as explained in Section 2. 2. However, using English demonstrations for cross-lingual transfer or translating test samples into English has its own practical value for NLP applications. Additionally, it is worth exploring selecting demonstrations from a mixture of languages. Expanding our study to more setups would provide additional insights into multilingual and cross-lingual LLM abilities. Other prompting methods. In Section 6, we only experiment with manually augmented templates to illustrate how the choice of template can reduce the effectiveness of demonstrations. There is a broad literature on prompt engineering and prompt sensitivity (White et al. , 2023; Gonen et al. , 2023), suggesting that it is plausible that another prompt could reduce the gap between few-shot and zero-shot performance even further. Chain-ofthought (CoT) prompting is another approach with promising multilingual abilities (Shi et al. , 2023b; Huang et al. , 2023) that might affect our findings. Our manually-augmented templates are intended only as a starting point for further analysis, which we leave to future work. Beyond automatic evaluation. When examining model responses, we noticed some cases where a correct answer as evaluated by a human was not fully captured by automatic evaluation metrics. Human evaluation is time-consuming, expensive, and hard to source for the wide range of languages that we explore in our work. Another option is LLM
evaluation, which is becoming increasingly popular (Fu et al. , 2023a; Chan et al. , 2024), but is also an expensive approach. More importantly, we have no guarantees about LLMs’ multilingual capabilities. As a trade-off between cost and evaluation quality, we stick to automatic evaluation in our work for all tasks and languages