MEGA: Multilingual Evaluation of Generative AI
Authors: Kabir Ahuja
Harshita Diddee
Rishav Hada
Millicent Ochieng
Krithika Ramesh
Prachi Jain
Akshay Nambi
Tanuja Ganu
Sameer Segal
Mohamed Ahmed
Kalika Bali
Sunayana Sitaram
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Multilinguality
Evaluation
Low Resource Languages
Benchmarking

Abstract: Generative AI models have shown impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. An important question being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. We compare the performance of generative LLMs including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and tasks and discuss challenges in improving the performance of generative LLMs on low-resource languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

MEGA: Multilingual Evaluation of Generative AI
Kabir Ahuja Harshita D
Rishav Hada
Millicent Ochieng
Krithika Ramesh Prachi Jain'
Akshay Nambit
Tanuja Ganu
Sameer Segal
Maxamed Axmed
Kalika Bali
Sunayana Sitaram
University of Washington
Carnegie Mellon University
iMicrosoft Corporation
Johns Hopkins University
kahuja@cs. Baahuja@cs.washiggtonn edu,
sunayana si aaraammiiicimm wtwp:emiiooooooooooomoo
Abstract
(OpenAl, 2023) was evaluated on the MMLU mul- tiple choice questions benchmark by automatically translating it into 26 languages, and the results for some low-resource languages in the Latin script were found to be quite promising.
Generative Al models have shown impressive performance many Natural Language Pro- cessing tasks such as language understanding, reasoning, and language generation. An impor- tant question being asked by the Al commu- nity today is about the capabilities and limits of these models, and it clear that evaluating generative AI is very challenging. Most stud- ies on generative LLMs have been restricted English and it is unclear how capahle these models are at understanding and generating text other languages. We present the first com- prehensive benchmarking of generative LLMs MEGA, which evaluates models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. We compare the performance of generative L.LMs including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous gen- eration of LLMs. We present thorough anal- ysis of the performance models across lan- guages and tasks and discuss challenges in im- proving the performance of generative L.MSs on low-resource languages. We create frame- work for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.
The multilingual capabilities of these models can be traced to their pre-training data, where even the predominantly English large-scale corpora con- tain hundreds of millions of non-Engish tokens (Blevins and Zettlemoyer, 2022). For GPT-3 un- labeled pre- pre-training data has been documented to contain 1 119 languages (Brown et al., 2020), where roughly 93% of the tokens are in English Other LLMs like BLOOM (Scao et al., 2022) and PaLM (Chowdhery al., 2022) have better multilingual representation with 60% and 18% non-English data respectively for pre-training. While these models have been trained on multiple languages with vary- ing distributions in the pre-training data, clear how well they perform relative to each other across diverse tasks and languages due to lack of comprehensive analysis across all models with the same experimental setup.
not
Recently, there has been a lot of interest in evalu- ating the different capabilities of LLMs, with com- prehensive studies like HELM (Liang et al., 2022) that evaluate these models on a wide variety of capabilities. However, such studies are largely per- formed on English language data and there is a lack of such large-scale evaluation of LLMs for their multilingual capabilities. Given the current pace at which new language technologies are be- ing developed that use LLMs, the importance of such an evaluation cannot be understated as the cases of inequalities in the performance of previous- generation models across languages have been well- documented (Blasi et al., 2022).
1 Introduction
Large Large Models (LLMs) such as ChatGPT and GPT-4 have created lot of interest in the Al com- munity and beyond, due to the step jump in their capabilities, such as maintaining context over con- versations, fluency of generation, and reasoning. Many users have reported having tested these sys- tems on languages other than English, with varying results, and recent demos these models (Warren, 2023) have been shown in multiple (albeit high- resource) languages. Recently, the GPT-4 model
In work, present the first large-scale Multilingual Evaluation of Generative Al mod-
hhttps://githhe http:///hhhbbccmm//////////gggt/ http:ppppaaaagggotao///////a/aaa 3/blob/master/ aatast_sttaissticc//mnggggssss http:////aaaaueeeeeeeooooooooouo count..cvv
Work done when the author was at Microsoft.

---

Instruction
Language Family Distribution in MEGA
are assistant shose jurrpsse solve Naturat
grablemn comeiitieenentt
CUa (30)
Fee:Shot Exameles
Mertifiietee.
Question: B1
4nzil
Teep lenplatized Input
C3
Slewe Falau, failler?
Spu Profiesie
Telogh
Label
No S00o
Iedo
Autto
oti Falie, Weither?
ings-age family
(a) Tasks and Datasets included in MEGA.
(b) Language Family Distribution
(c) Example multilingual prompting
Figure 1: An overview of our benchmarking exercise: Multilingual Evaluation of Generative Al (MEGA), Numbers parentheses in Figure contain the number of languages supported the dataset.
els (MEGA), spanning 16 different datasets, 70 topologically diverse languages, and four LLMs i.c. GPT-3.5 models text-davinci-003 and 3pt-3.5-turbo, 5-turbo, GPT-4 (gpt-4-32k) and BLOOMZ (Muennighoff et al., 2022). We also compare these models with the models fine-tuned on these datasets like TULRv6 (Patra et al., 2022) and MuRIL (Khanuja et al.. 2021), which are SoTA on different multilingual benchmarks.
for strategies that can be used for building systems using generative AI for multilingual users. We also release our code for the community scale up the multilingual evaluation of generative models.
MEGA
In this section, we discuss different components of our benchmarking exercise to measure the multilin- gual capabilities of LLMs. We start by discussing different NLP tasks and datasets that we evaluate these models on, along with their linguistic diver- sity. We provide an overview of the models we evaluate, basclines for comparison, and describe our evaluation scheme and prompting strategies.
Through our evaluation, we aim answer three research questions. ()), how well do LLMs fare on multilingual benchmarks compared to fine-tune tuned SOTA models? (2), what languages do these mod- els perform well in, and can we explain the trends in performance for these models across languages? (3), what prompting strategies should be used for using L.Mss for non-English languages?
2.1 Datasets and Languages
We broadly consider five families NLP tasks in our experiments covering 16 different datasets: Classification Tasks. Here, we further have four different sub-tasks, Natural Language In- ference (classify if hypothesis is entailed in the premise, contradicts or neither), which in- cludes XNLI (Conneau et al., 2018), Indic-XNLI (Aggarwal et al., 2022) (version of XNLI trans- lated to 11 Indian languages), and GLUECos NLI(Khanuja et al., 2020b) for English-Hindi code- mixed data; ii) Commonsense Reasoning datasets including causal commonsense reasoning bench- mark XCOPA (Ponti al., 2020) and XStoryCloze (Lin et al., 2022a), where the correct ending of a story with four sentences is to be predicted; iii) Paraphrase Identification task PAWS-X (Yang et al., 2019a), where given two sentences, the
Our study highlights that there is a signifi- cant disparity between the performance of LLMs in English vs non-English languages, especially low-resource languages with non-Latin scripts for which fine-tuned models perform significantly bet- ter. While GPT-4 bridges this gap to some extent, the discrepancy still exists. Further, we find that for these languages is often difficult to do bet- ter than simply machine translating the input in a target language to English and then sending it to the LLM for prediction (translate-test). We also discuss how different pesmpt choices like prompt-tuning, use of cxplanations, and number of few-shot examples impact multilingual perfor- mance. Finally, we perform some initial analysis to the test the possibility of test data contamination in LLMs that we evaluate and discuss its implications on our findings. Our work provides a blueprint
MS/MEGA

---

model must predict if the two have the same mean- ing: iv) EN-ES-CS dataset for Sentiment Analysis on English-Spanish code-mixed tweets.
lines, which are fine-tuned on task-specific training data. For the former we consider BLOOMZ (Muen- nighoff et al., 2022), a multi-task fine-tuned version of the BLOOM (Scao et al., 2022) model, which is a 1 76 billion parameter model trained on 46 natu- ral languages and 13 programming languages. For fine-tuned baselines, we consider TULRv6 (Patra et al., 2022) (the current SoTA on XTREME bench- mark), XLMR (Conneau et al., 2020), multilingual BERT (Devlin et al., 2019), and mT5 (Xue et al., 2021). For Indic-datasets we also compare with MuRIL((Kanuja et al., 2021), a multilingual BERT model trained on 16 Indic languages that obtains SOTA performance on many Indic benchmarks. All of these models (excluding mT5 for the XL.- Sum and XCOPA), were fine-tuned with English data and then evaluated in zero-cross- croo-croos---luuaall lingual fash- ion on other target languages.
Question Answering (QA), For QA we consider Span-Prediction tasks, where the answer to ques- tion is to be predicted within piece of context provided. We evaluate on XQuAD (Artetxe et al.. 2020), MLQA (Lewis et al., 2020), TyDiQA-GoldP (Clark et al., 2020), and IndicQA (Doddapaneni et al., 2022).
Sequence Labeling. This task involves classify- ing each token in a piece of text and we consider Named Entity Recognition dataset PAN-X (Pan et al., 2017) (also called WikiANN) and UDPOS (Nivre ct al., 2018) for Part of Speech Tagging. Natural Language Generation (NLG). For NLG we consider the multilingual Abstractive Summa- rization dataset XL-Sum.
Responsible AI (RAD). We consider the multilin- gual Toxicity Prediction dataset Jigsaw(Kivlichan et al., 2020), and Wino-MT to measure Gender Bias in MT systems.
2.3 Evaluation Methodology
LLMs exhibit two remarkable properties that make them effective solving variety of NLP tasks. The first is in-context learning (Brown et al., 2020), where the model learns to solve a task through the few input-output examples provided as part of the context without any weight updates. Secondly, the ability to follow instructions (Mishra et al., 2022; Wei et al., 2021; ; Ouyang et al., 2022) which is property of instruction-tumee LLMs, where the models can be prompted solve new-tasks based on the textual instructions provided in context.
All the datasets with the number of languages they include are listed in Figure la. These 16 datasets encompass a total of 70 languages cov- ering 21 different language families, with Indo- Aryan and Afro-Asiatic languages in the majority (see Figure 1b). Note that for tasks with > 30 lan- guages i.e. UDPOS, PAN-X, and XL-Sum, evaluations on the first 1000 examples of the test sets. For tasks where no public test sets are avail- able (like XQUAD, TyDiQA-Golde and IndicQA), we evaluate on validation data. Refer to Appendix $A.1 for a detailed description of all the datasets.
run
We adopt these two techniques together to test the capabilities of LLMs to solve variety of tasks in different languages. We define five main com- ponents define the prompts: i) a test example Ptest for which the predictions are to be made; ii) k few-shot exemplars { (Tx y1) H that are used to provide in-context supervision to the model; iii) a task instruction I which describes the instruction in text for the task to L.M;; iv) a prompt template Stemp (a:) which turns dataset input example into text format that can be used for prompting; and v) an answer verbalizer (y) that maps the label y to a textual representation. In our evalua- tion framework we often consider the instruction, template, and verbalizer as a single entity, and from now on will denote the template to encapsulate the three unless specified separately.
2.2 Models
OpenAI Models. We conduct all bench- marking experiments on the GPT-3.5 mod- els text-davinci-e03 (denoted as DV003 in the paper) and gpt-3.5-turbo (Ouyang et al., 2022) (GPT-3.5-Turbo) well on the GPT- 4 model gpt-4-32k (OpenAl, 2023). The text-davinci-e00 model has a maximum con- text size of 4096 tokens, while 20t-3.5-urbb and gpt-4-32k support context sizes of 16k and 32k respectively.
Baselines. We compare the performance of Ope- nAI models with two classes of baselines, i) Prompt-Bas Based baselines, which like the OpenAT models are evaluated by prompting the model di- rectly for solving a task, and ii) Fine-tuned Base-
Given these components, the final prompt fprompt (Z,, { II fremp- fverb) or Fpromps (Iust) for short for a test input Tuest can

---

be defined as:
Prompt Tuning. We use PromptSource (Bach et al., 2022) for a database of existing prompts to use for our experiments. In order to select the best prompt for a dataset (to appropriately mea- sure the capabilities of these models), we evaluate the performance of available English templates on PromptSource on the English validation set and select the prompt that gives the best performance. This prompt template is then used to evaluate mod- els on the test sets for all languages and prompt strategies. While it would be ideal to tune the prompts separately for each language, the scale of our experiments and the computational costs of these models make it prohibitive. We investigate the impact this choice has on our results in $4.1. 1. We perform separate prompt-tuning for DV003 and GPT-3.5-Turee models, and to keep the costs in check, we use the prompts obtained for the latter for GPT-4 as well. Final prompts selected are in- cluded Appendix $A.4.
fprompt (Frest) II |1:=1 {frmp(ay (2i) fverd (yi)
ftemp (Frest)
where || denotes the string concatenation opera- tor. The prompt can then be provided as input to the LLM P(:: 0) to obtain the prediction 2test arg max P(z) (Ztess))Ztes)) 0), where Z is the space of possible answers, which in all of our experiments taken to be the entirety of the lan- guage as modeled by the LLM. We approximate the arg max by sampling from the probability dis- tribution predicted by the LLM.
2.3.1 Multilingual Prompting Strategies
The choice of prompt significantly influences the performance of LLMs and these models have been shown to be brittle to simple prompting variations, such as the choice of prompt template and the train- ing examples or even the ordering of examples (Zhao et al., 2021). For multilingual setups as high- lighted in Lin et al. (2022a) and Shi et al. (2022), some additional variations to consider include, the choice of the language of the few-shot examples, the language of the prompt template, and the lan- guage of the test examples.
Choice of Few-Shot Examples. In all our exper- iments, we choose few-shot examples randomly from the training or validation set (depending on what's available) of a dataset. For most datasets, we use few-shot examples, excluding tasks with longer contexts like QA and summarization tasks where we use k 4.
In this work, we evaluate models using three types of prompting strategies: Monolingual Prompting: In this setup, the k randomly selected examples are of the same language as the test exam- ples. Zero-Shot Cross-Lingual Here, we evaluate generative models' zero-shot cross-lingual transfer ability during in-context learning. We use k-shot examples from a pivot language (always English in our experiments) which is different from the language of the test example. Translate-Testt In this setup also, the few-shot examples are sampled from English data. However, the test example it- self modified by translating it into English. We use Bing Translator to translate the test examples into English. We do not perform evaluations with Translate- prompting for QA and Sequence La- belling tasks where there is no trivial alignment between the labels in the translated text with native language text. To preserve costs, for GPT-4 we only run evaluations with the Monolingual prompt- ing strategy except for couple of datasets, which we explicitly discuss later in $3. Irrespective of the prompting strategy, we use the prompt templates written in English (see Appendix $A.7 for the im- pact of this choice).
3 Results and Analysis
In this section, we analyze the results of our benchmarking exercise across tasks and languages. Broadly, we cover the comparison between the ef- fectivness of various prompting strategies $3.1 fol- lowed by the performance comparison of GPT-3.5 and GPT-4 models with appropriate baselines $3.2. We conclude with an examination of the factors that affects the performance of these models $3.3.
3.1 Comparing different prompting strategies
In Figure 2, we compare the performance of the three prompting strategies. We find that translate- test often improves the performance over the mono- lingual strategy, especially so in the case of DV003. We also find that for datasets, which include many low resource and non-latin script languages like In- dicXNLI and XStoryCloze, the gains with translate- test are even more substantial for both the mod- els. In Figure 3, we present the average (over dif- ferent tasks) relative improvement by Translate- Test over Monolingual on GPT-3.5-Turbo for dif- ferent languages and observe for languages like

---

Comooring Promping Strategles
0.
0.2
013
TyDIAA lals?
XXLI
16A7-X
XCOFA
Zertany Cate
Outand
Figure 2: Comparing different prompting strategies discussed in $2.3.1 on DV003 and GPT-3.5-Turbe The y-axis denotes the task-wise performance metric, e.g. Accuracy for XNLI and FI-Score for TyDiQA-GoldP. A list of metrics for all the tasks is provided in Table 1.
Translate- Test Relative Improvement Monolingual for GPT-3.5 Turbo
Improvement
Langinge
Burmer
Tanti
Teloga
Malayalm
Onio
Baque
Marahi
Punjbii
Bengit
Kanrad-
Haltinn Crett
Undu
Aosomeee
The
Hind
Bulga lan
Vetmamose
Goeak
Garman-
Russian
Italion
Indometion
Frunch
Spanish
Chinas
Esooian
Swihil
Twdksh
Koren
Japanese-
Language
Figure 3: Relative percentage improvement over Monolingual prompting when using Translate- Test for GPT-3.5- Turbo. The bars are color-coded based on the class taxonomy provided in (Joshi et al., 2020)
Burmese, Tamil, and Telugu the relative improve- ment can be > 30%! In general, we see that for low-resource languages, the translate-test results in substantial improvement in performance, while for high-resource languages the two perform simi- larly. While do not evaluate GPT-4 Translate- Test exhaustively for all tasks, we do run the tests for XStoryCloze and XCOPA datasets. Based on these two, we observe that GPT-4's Monolingual prompting performance is often much more on-par with Translate Test and many times even better. However, for low-resource languages we again see Translate-Test to perform much better. e.g., in XS- toryCloze GPT-4's accuracy on Burmese is 77.6% 93.2% for Monolingual and Translate-Test re- spectively Figures 10b and 10d Appendix). Note that while Translate-Test substantially im- proves performance low-resource languages, compared to the performance of these models in English, the gap even after Translate-Tess Test is sig- nificantly high. For example, using translate-test with GPT-3.5-Turbo for Urdu in XNLI results in 5.1% accuracy compared to 19.19 1..1% % for monolingual. However, this contrasts with the 76.2% accuracy that the same model achieves in English.
often performs on par with Monolingual but for GPT-3.5-Turbo, there is drop in performance, es- pecially so for tasks like XCOPA which have some extremely low resource languages: Quechua and Haitian Creole. For these languages, observed that when provided few-shot examples in English, GPT-3.5-Turbe would often resort to predicting outputs like "I'm sorry, but the premise is not in a language that I understand." ". However, by provid- ing examples in the language, we are able to ground the model to these languages and we almost never observe such predictions in that case.
3.2 Comparing different models
The aggregated results comparing different mod- els and prompting strategies are provided Ta- ble and Table (for Indic Datasets). Exclud- ing the commonsense reasoning tasks XCOPA and XStoryCloze, the OpenAl models generally lag behind the fine-tuned baseline TULRvó for most tasks often by significant margin and often only slightly better than some of the smaller fine- tuned multilingual models i.e. mBERT and mT5- base. Between OpenAl models and BLOOMZ, the former models tend to outperform the latter (despite having a larger proportion of multilingual
Zero-Sho Cross Lingual prompting for DV003

---

Classiliction PAWS-X
Question Answering
Sequence Labelling
Summarization
Model
XNLI
XCOPA XStoryCloze
XQUAD
TyDiQA-Gole
MLQA
UDPOS
PAN-X
XLSumm
Metrics
Acc.
Acc.
Acc.
Ace.
FI/EM
FI EM
FI FI/EM
FI
ROUGE-L
Fine-tuned Baselines
MBEKT
65.4
81.9
56.1
64.5
49.7
61.4/ 44.2
71.9
62.2
--TS-Base
75.4
86.4
49.9
47.0/49.0
57.2/41.2
64.6/45.0
55.7
28_11
XLM-R M. Large
79.2
86.4
69.2
76.6/ 60.8
65.1744.0
53.2/
76.2
65.2
Tul.Rv6 XXI.
88.N
93.21
82.21
86/72.91
84.6/738
81/63.9
83.0
84.21
Promps Dased Buselines
BLOOMZ
54.2
(82.2)
60.4
76.2
(70.7 58.8)
(752/6322)
Open Models
text-davinci-083
59.27
67.08
75.2
74.7
40.5 28.0
49.7/38.3
14.0/28.8
text-dacinci-805 (TT)
67.0
68.5
83.8
94.8
X
54.97 34.6
X
got-3, 5-turbo
62.1
70.0
79.1
87.7
60.4/
60.1 38.4
56.1 /33.8
60.2
40.3
18.8
(TT))
64.3
67.2
81.9
93.8
X
X
46.3/27.00
x
X
16.0"
20t-4-32k
75.4
89.71
96.5"
646.6
50,9
67.2/43.3
66.6
55.51
19.71
Table I: Average performance across languages in each of the different datasets included in MEGA. TT suffix refers to the translate-test prompting strategy discussed in Section 2.3.1, without any suffix we refer to the monolingual strategy by default (except for XQuAD and IndicQA where refers cross-lingual setup). Numbers in bold with symbol indicate best performing fine-tuned model and the ones with & refer to the best prompt-based generative model. The best overall numbers are underlined. For BLOOMZ. the values in parenthesis indicate that the model fine-tuned the task during mulli-task. training. Missing values corresponding to the 'x' symbol denote experiments that were not applicable and the ones with were the ones deprioritized due to limited compute. gpt-3. b- (TT) on XL-Sum was only evaluated on 29 languages which are supported by Bing Translator.
pre-training data), except for datasets like PAWS-X, XQUAD, and TyDiQA-GoldP, where BLOOMZ performs better. However, it must be noted that all these three datasets were present in the multi- task fine-tuning stage for BLOOMZ, especially for XQUAD and TyDiQA-GoldP for which the valida- tion data that we use for evaluation is also likely to be included in the fine-tuning d
results spanning all models, tasks, and languages. please refer to Appendix $A.8.
3.3 Factors Explaining Performance Trends
In this section, we try to understand what factors influence our observed trends in multilingual LLM capabilities. We begin by investigating the Fertility of the tokenizers used by different models, which is defined the average number of sub-words pro- duced per tokenized word (higher means worse quality), as that has been shown to critically impact the downstream task performance of pre-trained multilingual models (Rust et al., 2021). In Figure 4, we plot the tokenizer fertility of different mod- els. We observe that the tokenizers for the OpenAl models are substantially worse for low-resource. non-latin script languages: where the fertility for languages like Malayalam and Tamil is so high (~ 10) that the tokenizer essentially operates as a byte-level tokenizer for these languages. Note that this means that for low-resource languages, substantially larger number of tokens are needed to encode the inputs as well as for generation, which results in significant additional API costs. Ahia et al. (2023) discusses how this phenomenon leads to large socio-economic disparities for speakers of underrepresented languages. We study if these discrepancies in the tokenizer's quality across lan- guages have any effect on the performance. As can be seen in Figure 5, for six tasks we observe statis-
Between the OpenAl models, generally DV003 and GPT-3.5-Turbo perform on par, with Translate- Test performance of DV003 being generally better than GPT-3.5-Turbo, and the other way around for Monolingual performance. However, we do ob- serve a notable exception to this, which for the QA tasks where GPT-3.5-Turbo performs substan- tially better than DV003, especially so for IndicQA. We attribute this to the fact that in order to fit the prompt in the 4096 context size for DV003, we had to resort to retrive-then prompt strategy and im- perfect retrieval for low-resource languages leads worse performance. Please check $A.5 of Ap- pendix for more details on this. For GPT-4 on the other hand, we consistently observe substan- tial improvements, with it being Pareto Optimal (Choudhury and Deshpande, 2021) compared to the two GPT-3.5 models for all datasets with an exception of XL-Sum, where for some languages Gurrb performs better. For the detailed
"Note that this can be a possibility for OpenAl models as well and we discuss this in more detail in $4.2.

---

Tolesiae
Enghh
Frundh
Jepanoo
Sponch
Indonsaan
Coman
Diggrion
Rowon
Koren
Grec
koloplon
Tanl
Bo
Langsage
M.0MM
Figure 4: Tokenizer Fertility for OpenAl models, mBERT, and BLOOM for different languages
GPT-b.5-Turbo
related with each other. However, we do see that using pre-trate pre-training data we are able to explain some trends that are not explained by tokenizer fertility alonc. For example, even though the OpenAl mod- els have similar tokenizer fertilities for both French and Japanese, these models perform much better in French than they do for Japanese (72. % accuracy vs 67% accuracy for GPT-3.5-Turbo) for PAWS- X. However, when we take into consideration the amount of pre-training data for these languages: roughly 3.5 B French tokens in the pre-training data versus 214M for Japanese, we can partially explain this discrepancy.
Stane
0.1
0.2
0.0
Takemizer Foriiiity
IxliGA, pm
XKU pm 0.70
XCORAA 3.00
XIVAD -077
XLSum, 0.82
Xitery Clozs
0.75
Figure 5: Correlation between the performance of GPT- 3.5-Turbo with the tokenizer fertility. We report curves for the cases where the person coefficient > 0.7 with a p-value of 0.05. We have combined Indic- XNLI and XNLI for a better coverage of languages. Similar plots for GPT-4 can be found in Figure 7b of Appendix.
However, we must note that these two factors correlate well with only subset of the tasks and what we are measuring is the correlation which might not imply causation. Investigating different factors that together more holistically explain mul- tilingual capabilities is an important direction that we leave for future work. Please check Appendix $A.6 for detailed results from this section.
tically significant (negative) correlations between the tokenizer's fertility and datasct-specific perfor- mance i.e. the models obtain worse performance on languages for which the tokenizer is of poor quality, and vice-versa.
4 Challenges in Multilingual Evaluation
In this section, we examine some of the challenges and consequent limitations of a large-scale multi- lingual evaluation like ours.
We also study the effect that the amount of data available for each language during pre-training (Wu and Dredze, 2020; Lauscher al., 2020) has on the multilingual performance of these models. We mea- sure the correlations between the language-wist number of tokens present in the pre-training data with language-wiss performance on each dataset. While the exact language-wise pre-training data distribution for GPT-3.5 and GPT-4 models is not available, we use the GPT-3's language-win proxy. We observe
4.1 A Kaleidoscope of Choices.
There are various moving parts when evaluating LLMs using prompting-ban approaches, includ- ing the choice of prompt templates, instructions, and few-shot examples (Liu et al., 2022; Lu et al., 2022; Zhao et al., 2021), different prompting strate- gies (Wei et al., 2023; Nye et al., 2021; Ye and Durrett, 2022a), using external tools (Schick et al., 2023), the language of prompts (Shi et al., 2022: Lin et al., 2022a), well different decoding spe- cific pyper-parameters (Shih al., 2023), which can have varying degrees of impact on the perfor- mance, sometimes unexpected ways. Holisti-
pretraining distribution that for four tasks (PAWS-X, XNLI, XCOPA, and XQuAD) statistically significant positive correla- tions between the pre-training data size and perfor- mance. Note that, the amount of pre- pre-training data and tokenizer fertility are highly likely to be cor-

---

Effect of Shott Sec
Eket of SowShat Saz
2023). Accordingly, we attempt to verify if the performances we observed are fact, representa- tive of the capabilities of these models or merely a result of memorization. Given the lack of trans- parency in the training distribution of recent models like GPT-4, we perform some preliminary inves- tigations against this phenomenon. Specifically, we consider three factors: i) LLM's knowledge of the dataset, ii) availability of test datasets on the internet, and ii) dataset release date.
4.20
0.32
312
12 Toweee campar
Sevelot osrole)
(a)
(b)
Dl of Toie
To measure the LLM's (we do this for GPT-4) memory of the dataset, we prompt it to fill the dataset cards for each of MEGA's datasets (denoted as Card Fill). This involves filling templatic infor- mation like the task's supported languages, input- output structure and description. If the model fills dataset card correctly (Full), we note this as sus- picion of contamination. If fills the card partially correct (Partial) i.c. detecting either the correct task structure or correct set of languages, we mark it as partial evidence, and if it succeeds in neither, we mark it as no suspicion (None). For test dataset availability, we check if the test dataset can be cessed online directly without downloading either as part of the official release from the authors or via other sources such as Hugging Face dataset viewer (Data Acc. w/o Down.). For relcase date, we check if the dataset was made public after the cut-off date of September 2021.
Leggung
(c)
(d)
Figure 6: Analysing the effect GPT-3.5-Turbo's per- formance given different evaluation factors. To obtain explanations we use Super-Natural Instructions (Wang et al., 2022).
cally exploring these choices for all the datasets and languages can quickly get out of hand, espe- cially given the excessive computational cost of running these models. In order to understand the sensitivity of our observations to the choices we make in $3, we re-evaluate our setups on a sub- set of datasets and languages for a varying set of parameters. Our findings are summarized in Fig- ure 6, where we see that having a large few-shot size generally helps improve performance, how- ever, the performance is often stable beyond k- 8. Further, language-specific fine-tuning can help im- prove the performance like we see for Haitian Cre- ole in XCOPA, but for Tamil we actually observe the accuracy to go down which might be attributed to the small size of the validation set (100 in the case of XCOPA). Finally, on XStoryClozi dataset (also for XCOPA), we see using explanations to prompt the models have negligible impact on the performance. Overall, these experiments indicate that the existing prompting approaches might not be sufficient to address the performance gap that ex- ists for non-English languages (cspecially mid-to- low resource languages) and there is an imminent need to propose new methods as well as improve the representation of different languages in these model's 's pre-training (and instruction-tuning data.
The overall results from this analysis are pro- vided in Table 2. We see that for majority of datasets, GPT-4 can fill in the dataset card cor- rectly; On the more recent datasets like XLSum and XStoryCloze it is only partially successful, while on Jigsaw and code-mixing datasets it fails to cor- rectly fill the cards. Note that except XStoryCloze, Jigsaw and the Code-mixing datasets, evaluation sets for all other datasets directly accessible online. Collectively, this connotes that for tasks like XStoryCloze and IndicQA there is a weak sus- picion against contamination. While all other tasks are highly likely contaminated (except Jigsaw, and Code-Mixed datasets).
Implications. Our analysis implies notable chance of the test data appearing in the training datasets of these LLMs. The contamination of test datasets serious problem for works centered around LLM evaluation (including ours), as they might lead to an overestimation of the capabili- ties of these models. However, we would like to highlight that despite the possibility of contamina-
4.2
Test data contamination
Given the massive amount of online data that LLMs are trained with, it is critical to factor in the possi- bility of contamination of test datasets (Sainz et al.,

---

Datasct
Card Fill
Data Acc, w/o Down.
Release Date
marks have mainly focused on pre-train then fine- tune kinds of setups. Particularly for prompting style evaluation, Bang et al. (2023) evaluates the multilingual capabilities of ChatGPT and shows that it fails to generalize to low-resource languages with non-latin scripts. However, multilingual eval- uation is performed only on a few tasks, and a subset of 50-100 examples are used for testing the model. Hendy al. (2023) evaluate the translation abilities of GPT-3.5 models and find that these mod- els, while perform well translating high-resource languages, their capabilities for low-resource lan- guages are limited. Concurrent work BUFFET (Asai et al., 2023) and Lai et al. (2023) also per- form multilingual benchmarking of large language models, however, they evaluate the performance of ChatGPT and BLOOMZ in their work while our evaluation also spans GPT-4.
XNLI
Full
Yes
September 2019
Indic XNLI
Full
Yes
April 2022
Full
August 2019
XCOPA
Partial
Yes
April 2120
XStoryCloze
Partial
No
May 2023
XQuAD
Full
Yes
Octoher 2019
QA
Full
Octoher
TyDiQA-GoldP
Full
Yes
Fehruary 2020
IndicQA
Partial
Yes
September 2022
PAN-X
Full
Yes
July 2017
UDPOS
Yes
March 2020
XLSum
Partial
Yes
June 2021
Jigsaw
None
No
February 2020
GLUECos NLI
None
No June 2020
EN-ES-CS
None
No May 2016
Table Contamination analysis for the datasets that we consider in MEGA. We use red color when there is a strong suspicion of contamination based on these three metrics, green for suspicion, and yellow for partial.
tion, LLMs still vastly underperform on (especially low-resource) non-English languages. These ob- servations about data contamination indicate that the disparity in performance between English and non-English languages might be even greater than what we observe in our work.
While most work on prompting or in-context learning in LLMs focuses on English data, recently, there has been some in- terest in prompting them with non-English data. Zhao and Schütze (2021), for instance, use discrete and soft prompting techniques evaluate XLM- RoBERTa and show that prompting can be more effective compared to fine-tuning when the amount of labeled data is limited. Lin et al. (2022a) show that English prompts perform better than prompts written in the target language (both hand-written and translated). Finally, (Shi et al., 2022) show chain-of of ttothhttht (CoT) prompting results leads to striking multilingual reasoning capabilities in LLMs, even in under-represented languages espe- cially when prompted when English COT.
Multilingual Prompting:
5 Related Work
Evaluation of LLMs. A growing interest in the evaluation of LLMs has harbingered several efforts towards the holistic evaluation of their capabilities. While work like BIG-bench Srivastava et al. (2023) cover diverse range of tasks, the non-English tasks are mostly translation cunisattoo-orientedd the more general task based inferences that for such an evaluation. Similarly, Liang et al. (2022) pro- pose taxonomy of scenarios and metrics in Holis- tie Evaluation of Language Models (HELM) to define the space of LLM evaluation, and evaluate 30 language models on 42 scenarios and 7 metrics. However, all the scenarios are focused on datasets in standard English or its dialects.
oriented which limit
Conclusion
In this work, we conduct an evaluation across dif- ferent prompting strategies, models, tasks, and lan- guages to investigate the multilingual capabilities of LLMs. We also investigate underlying properties like tokenizer quality and size of pretraining data to explain the trends in performance that we observe. Our investigation shows the consistent performance gap between high-resource, Latin script, and under- resourced languages in addition to highlighting the efficacy, yet limited sufficiency of methods like translate-test prompting. Through evaluation, we present evidence of the need prioritize auto- matic benchmarking and human evaluation across as many languages as possible. We hope that this work spurs research in meeting this goal.
Multilingual Benchmarks and Evaluation. Benchmarks for multilingual evaluation, such as XTREME (Hu et al., 2020), XTREME-R (Ruder et al., 2021) and XGLUE (Liang et al., 2020) have been proposed to measure cross- lingull transfer in pre-trained language models. Following their pop- ularity, there has been the development of bench- marks covering specific language families, such IndicXTREME (Doddapaneni et al., 2022) for Indian languages, Adelani al. (2022) for African Languages, and Wilie et al. (2020) for Indonesian languages, as well. The evaluations on these bench-

---

Limitations
Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Although we compare the evaluation results of GPT-3.5 and GPT-4 with BLOOMZ and SOTA models, we could not evaluate other closed models such as PALM, which also contains training data in many languages. A limitation of our study is that we do not evaluate on all the multilingual datasets that are available, and we plan to scale up our evalu- ation in future versions of the study with the help of the research community. Even if we do evaluate all available multilingual datasets, they do not cover many typologically diverse and under-resourced languages, which is a fundamental limitation of try- ing scale up multilingual evaluation today. For example, there is very little representation from African languages, Indigenous languages of the Americas etc. in any of the evaluation benchmarks available today. Finally, we restrict ourselves to the performance metrics and to some extent gender bias dimension of evaluation for this study how- ever, we plan to include evaluation of calibration, toxicity, bias, robustness, etc. in future work.
Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan, 2022. Indicxnli: Evaluating multilin- gual inference for indian languages. arXiv preprint arXiv:224.00877.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yu- lia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language mod- els. ArXiv, abs/2305. 13707.
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association Computational Linguistics, pages 4623-46377.
Akari Asai, Sneha Kudugunta. Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023. Buffet: Benchmarking large language mod- cls for few-shot cross-lingual transfer. arXiv cs. CL 14857.
Stephen Bach, Victor Sanh, Zheng Xin Yong. Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gun- jan Chhablani, Han Wang. Jason Fries, Maged Al- shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang. Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. Prompt- Source: An integrated development environment and repository for natural language prompts. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstra- tions, pages 93-104, Dublin, Ireland. Association for Computational Linguisties.
Acknowledgments
The authors would like to thank Barun Patra and Vishrav Chaudhary for their help with TULR eval- uation results. We also thank the anonymous re- viewers for their helpful feedback, which helped us improve the quality of our paper.
References
David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani. Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor. Fatoumata Ka- bore. Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh- Buabeng, Victoire Memdjokam Koagne, Allah- sera Auguste Tapo, Tebogo Macuewa, Vukosi Mari- vate, Mboning Tchiaze Elvis, Tajuddeen Gwad- abe, Tosin Adewumi, Orevaoghene Ahia, Joyce takatumba-Nabende Neo Lerato Mokono, Ig- natius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo. Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empiricul Methods Nat- ural Language Processing, pages 4488-4508, Abu
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, ballucination, and interactivity. arXiv preprint arXiv:2302.04022
Damian Blasi, Antonios Anastasopoulos, and Gra- ham Neubig. 2022. Systematic inequalities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1486-5505, Dublin, Ireland. Association for Computational Linguistics.
Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. Prompting language models for linguistic structure. arXiv cs. CL 2211007330
Terra Blevins and Luke Zeutlemoyer 2022. Language contamination helps explains the cross-lingual capa- bilities of English pretrained models. In Proceedings

---

the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3563-3574, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
A. Seza Dogruöz, Sunayana Sitaram, Barbara E. Bul- lock, and Almeida Jacqueline Toribio. 2021. A sur- vey of code-switching: Linguistic and social per- spectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), ), pages 1654-1666, Online. Association for Computational Linguisties.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemerr, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are sho- learners. CoRR, abs/2005.1411 14165.
Tahmid Hasan, Abhik Bhattacharjee. Md Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang. M Sohel Rahman, and Rifat Shahriyar. 2021a. XI- sum: Large-scal multilingual abstractive summariza- tion for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693 4703.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021b. XI.- sum: Large-scale multilingual abstractive summariza- tion for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693-4703 Online, Association for Computa- tional Linguistics.
Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair are multilingual pre-trained lan- guage models? In AAA--21. AAAL, AAAI.
Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? comprehensive evaluation. arXiv preprimt arXiv:2302.092110
Jonathan H Clark, Eunsol Choi, Michael Collins. Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qu: A benchmark for information-seekinn question answering in typo- logically diverse languages. Transactions the As- sociation for Computational Linguistics, 8:454-470.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross- cross-lingual generalisa- tion. In International Conference on Machine Learn- ing, pages 4411-4421. PMLR.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Compulational Linguistics, pages 8440- 8451.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury, 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282 6293, Online. Association for Computational Linguisties.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 XNLI: Evaluating cross- lingual sentence representations. In Proceedings of EMNLP 2018, pages 2475-2485.
Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal. Rajiv Teja Nagipogu, Shachi Dave, et al. 2021. Muril: Multi- lingual representations for indian languages. arXiv preprint arXiv:2103.100 107300
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of naacL-HLT, pages 4171- 4186.
Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2020a. A new dataset for natural language inference from code- mixed conversations. In Proceedings of the The 4th Workshop on Computational Approaches to Code Switching, pages 9-16.
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2022. Indicx- treme: A multi-task benchmark for evaluating indic languages. arXiv preprint urXiv:2212.05440
Simran Khanuja, Sandipan Dandapat, Anirudh Srini- vasan, Sunayana Sitaram, and Monojit Choudhury.

---

2020b. Gluecos: An evaluation benchmark for code- switched nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 3575-3585.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-1 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
lan Kivlichan, Jeffrey Sorensen, Julia Elliott, Lucy Vasserman, Martin Görner, and Phil Culliton. 2020. Jigsaw multilingual toxic comment classification.
Viet Dac Lai. Nghia Trung Ngo, Amir Pouran Ben Vey- seh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards comprehensive evaluation of large lan- guage models in multilingual learning.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086-80999 Dublin, Ireland. Association for Compu- tational Linguistics.
Anne Lauscher. Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4483-4499, On- line. Association for Computational Linguistics.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 3470-3487, Dublin, Ireland. Association for Computational Linguisties.
Patrick Lewis. Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. Mlqa: Eval- uating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 7315- 7330.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguis- tics.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.0911
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fen- fei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: new benchmark dataset for cross-lingual pre- training, understanding and generation. arXiv preprint 20Xhv:2004.04401.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Ls- dsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse- Semantics. pages 46-51.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary. Brian Horo, Jeff Wang, Luke Zettle- moyer, Zoritsaa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022a. Few- learning with multilingual generative language models. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019-9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Niklas Muennighoff. Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman. Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hai- ley Schoelkopf. Xiangru Tang. Dragomir Radev, Al- ham Fikri Aji, Khalid Almubarak, Samuel Albanie. Zaid Alyafeai, Alhert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning.
Akshay Nambi, Vaibhav Balloli, Mercy Ranjit, Tanuja Ganu, Kabir Ahuja. Sunayana Sitaram, and Kalika Bali. 2023. Breaking language barriers with leap: Learning strategies for polyglot 1lms. arXiv CL 2775.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du et al. 20226. Few-shot learning with multilingual generative lan- guage models. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 9019 9019-9052.
Joakim Nivre, Mitchell Abrams, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018. Universal dependen- cies 2.2.

---

Maxwell I. Nye, Anders Johan Andreassen, Guy Gur- Ari. Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyzz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for inter- mediate computation with language models. CoRR, abs/21 14.
Rachel Rudinger, Jason Naradowsky, Brian Leonard. and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics.
OpenAl. 2023. Gpt4 technical report.
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? on the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th Internutional Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 31 18-3135, Online. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida. Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Xiaoman Pan, Boliang Zhang, Jonathan May. Joel Noth- man, Kevin Knight, and Heng Ji. 2017. lingull name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958.
Oscar Sainz, Jon Ander Campos, Iker Garefa-Ferres Julen Etxaniz, and Eneko Agirre. 2023. Did chatgpt cheat your test?
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv prepriat urXiv:2211.05100.
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong. Furu Wei, Vishrav Chaudhary, and Xia Song. 2022. Beyond english-centric bitexts for better multilingual language representation learning, arXiv preprint arxiv:2210. 14867.
Timo Schick, Jane Dwivedi-Yo Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli. Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulié, and Anna Korhonen. 2020. Xcopa: multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 2372-2376.
tools.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder. Denny Zhou, Di- panjan Das, and Jason Wei. 2022. Language models are multilingual chain-of-thought reasoners. CoRR, abs/22 10.03057.
Pranav Rajpurkar, Jian Zhang, Konstantia Lopyrev, and Perey Liang. 2016a. SQuAD; 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. 2023. Long horizon temperature scaling.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Kr- ishna Rallabandi, and Alan W Black. 2019. A survey of code-switched speech and language processing. arXiv preprint arXiv: 1eX.v:9944000 1904.00784.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyzz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek. Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane. Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, An- drew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang. Angelica Chen, Anh Vuong. Animesh Gupta, Anna Gottardi, Antonio Norelli,
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew Gordon. 20 Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Sid- dhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, et al. 2021. Xtreme-r: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 1021 100445.

---

Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Cather- ine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng. Chitta Baral, Chiyu Wu, Chris Callison-Burde Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Asheraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Free- man, Daniel Khashabi, Daniel Levy, Daniel Moseguf González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Do- han, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang. Hong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Elcanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, EI- lie Pavlick, Emanuele Rodola, Emma Lam, Frie Chu, Eric Tang. Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice En- gefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed, Francesca Happé. Francois Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski, Giambatista Parascandolo, Gior- gio Mariani, Gloria Wang. Gonzalo Jaimovitch- López, Gregor Betz, Guy Gur-Ari, Hana Galijase- vic, Hannah Kim, Hannah Rashkin, Hannaneh Ha- jishirzi. Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schüitze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kermion, Jacob Hilton, Jac- hoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel. Jesujoba/ Alabi, Jiacheng Xu, Ji- aming Song, Jillian Tang, Joan Waweru, John Bur- den, John Miller, John U. Balis, Jonathan Batchelder. Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hemandez--Orlloo Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gim- pel. Kevin Omondi, Kory Mathewson, Kristen Chi- afullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis- Philippe Morency, Luea Moschella, Lucas I Lam, Lucy Noble, Ludwig Schmidt, Luheng
He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Mareo Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew I.. Leavitt, Mauthis Hagen, Mátýás Schu- bert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Co- hen, Michael Gu, Michael Ivanitskiy, Michael Star- ritt, Michael Strube, Michat Swedrowski. Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts. Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang. Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao. Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Mitkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Fita Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras. Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhut- dinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Techan, Rylan Yang, Sahib Singh, Saif M. Moham- mad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bow- man, Samuel S. Schoenholz, Sanghyun Han, San- jeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sehastian Bischoff, Sehastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan. Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee. Spencer Torene. Sriharsha Hatwar, Stanislas De- haene, Stefan Divic, Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi. Stuart M. Shieber, Summer Misherghi, Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes. Theodore Rothschild, Thomas Phan. Tianle Wang. Tiberius Nkinyili, Timo Schick, Timofci Kornev, Titus Tunduny, Tobias Ger- stenberg. Trenton Chang. Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmaku- mar, Vivek Srikumar, William Fedus, William Saun- ders, William Zhang, Wout Vossen, Xiang Ren, Xi-

---

aoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen. Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai. Zachary Seid, Zhuoye Zhao, Zi- jian Wang, Zijie J. Wang. Zirui Wang, and Ziyi Wu. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, et al. 2020. Indonlu: Benchmark and resources for evaluating indonesian natural language understanding. arXiv preprint 2009:2000005333
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 130-130, Online. Association for Com- putational Linguistics.
Gabriel Stanovsky, Noah A Smith, and Luke Zettle- moyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Adiyya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498.
Mike Thelwall. 2017. The heart and soul of the web? sentiment strength detection in the social web with sentistrength. Cyberemotions: Collective emotions in cyberspace. pages 119-134.
David Vilares, Miguel A Alonso, and Carlos Gómez- Rodríguez. 2016. En-es-cs: An english-spamisk code- switching twitter corpus for multilingual sentiment analysis. In Proceedings of the Tenth International Conference on Language Resources and Evaluation 16), pages 4149 4153.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019a. PAWS-X: A cross-lingual adver- sarial dataset for paraphrase identification. In Pro- ceedings of EMNLP 2019, pages 3685-3690.
Yinfei Yang. Yuan Zhang, Chris Tar, and Jason Baldridge. 2019b. Paws-x: A cross-lingual adver- sarial dataset for paraphrase identification. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (FMNL.P-JCCNIPP) pages 3687-3692.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018. page 353.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel. Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Necraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Summnnt Patro, Tanay Dixit, and Xudong Shen. 2022. aagerraNaurlllnsttttttoo tion via declarative instructions on 1600+ NLP tasks. Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 5085-5109 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Xi Ye and Greg Durrett. 2022a. Can explanations useful for calibrating black box models? In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pu- pers), pages 6199 6199-6212, Dublin, Ireland. Association for Computational Linguistics.
Xi Ye and Greg Durrett. 2022b. The unreliability of explanations in few-shot prompting for textual rea- soning. In Advances in Neural Information Process- ing Systems, volume 35, pages (0378-3039) Curran Associates, Inc.
Generaliza-
Daniel Zeman, Joakim Nivre Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Hamid Aghaci, and R Ziane. 2020. Universal dependencies 2.5. LIN- DAT/CIARIATICT digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University, url: http://hhll handle.
Tom Warren. 2023. Microsoft's chatgpt event live blog.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Lc. 2021. Finetuned language models are zero-shot learners. CoRR, abs/2109.00 109.0652.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Fdd Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. arXiv 2201. 1001,11903 cx. CL.
Jieyu Zhao, Tianlu Wang. Mark Yatskar, Vicente Or- donez, and Kai- Wei Chang. 2018 Gender bias in coreference resolution: Evaluation and debiasing

---

methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 15-20, New Orleans, Louisiana. Association for Computational Linguistics.
Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547-8555, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Zihao 7hao, Fric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 12706. PMIR.

---

A Appendix
2. Paraphrase Identification: PAWS-X (Yang et al., 2019b) is a paraphrase identification dataset professionally translated from the PAWS (Zhang et al., 2019) dataset into six typologically diverse languages.
A.1 Tasks and Datasets
In our experiments, we consider 16 tasks spanning the following task types classification, sequence to sequence labeling and generation. Below we review the experimental setups and datasets used for benchmarking for these two tasks. list of all the datasets with the languages covered by them can be found in Table 3.
3. Commonsense Reasoning: XCOPA (Ponti et al., 2020) is commonsense reasoning dataset, which is translation of the COPA (Roemmele et al., 201 dataset into 11 typologically diverse languages, including very low-rcsource languages such as Eastern Apurímac Quechua and Haitian Creole.
A.1.1 Classification
These tasks involve classifying a single sentence a group of sentences into a finite number of discrete labels. For each dataset, we measure the performance of different models in terms of clas- sification accuracy. For prompt-based models in particular, since add no constraint on the output space of the L.L we compute the exact match be- tween the generated output and a verbalized label to determine if the example was classified correctly. We run experiments for all the prompting strategies that we discussed in the previous sections for each dataset. The details of each dataset that we use for benchmarking are given below:
XStoryCloze (Lin et al., 2022b) is created by translating the English StoryCloze (Mostafazadeh et al., 2017) dataset using professional translators into 10 typologically diverse languages.
A.1.2 Question Answering
We focus on Span Prediction type of Question Answering (QA) tasks in our experiments, where given context and question the task is to predict the answer within the context. One major challenge that we come across for multilingual evaluation of QA tasks is that for many languages we often can- not fit the context and question pairs for the few- shot and text examples the maximum context size of 4096 for the DV003 model. This is mainly attributed to the poor performance of GPT's tok- enizer many non-latin script languages which results in over-tokenizing the words in these lan- guages.
Dataset
Task
Languages
XNLI
Natural Language Inference
15
Indie-XNLI
Natural Language Inference
11
GLUBCOS
Natural Language Inference
2
PAW'S-X
Paraphrase Identification
7
XCOPA
Commonsense Reasoning
10
XStoryCloze
Commonsense Reasoning
11
To overcome this issue we follow two steps. First, for the few-shot examples we only provide the line within the paragraph containing the answer as the context. Second, for the test example, we index the chunks of the context using the embed- dings from the text-embedding -aapdddiiinnnd--022222- ada-002 model. Given the question, the closest chunk in the full context is retrieved and used in the prompt for the test example. We use a maximum chunk size of 100 in our experiments and use the implementation for retrieval provided in the Lang Chain" library. By doing this,we minimize the space taken by the context tokens in our prompt.
TyDiQA-Gan
Question Answering
9
MLQA
Question Answering
6
XQuAD
Question Answering
11
IndicQA
Question Answering
10
UDPOS
Part of Speech Tagging
38
PANX
NER
48
WinoMT
Gender Bius
8
GLUECOS
Sentiment Analysis
Jigsaw
Toxicity Classificatiin
XLSum
Summarization
44
Table 3: Datasets and Language coverage of the datasets that MEGA presents evaluation for.
1. Natural Language Inference: XNLI (Conneau et al., 2018) is a dataset for cross-lingual Natural Language Inference, which consists of professional translations of the MNLI (Wang et al., 2018) corpus into 14 languages. We also consider IndicXNLI (Aggarwal et al., 2022) that translates the XNLI dataset into 11 Indic languages by using Machine Translation, followed by validation by native speak-
Note that, for newer GPT models i.e. GPT-3.5- Turbo and GPT-4 which support longer context lengths, we do not use this retrieval strategy for QA tasks and prompt the models to obtain the an- swers directly. For each task, we calculate the Exact Match and F1 score defined in Rajpurkar
ers.
*ht httphh/wcc.//ecccaaaa/aaaeeeeeee

---

al. (2016a). For our experiments we consider the following four tasks:
system.
A.2.1 Part of Speech Tagging
1. TyDiQA (Clark et al., 2020) is a QA dataset covering 11 typologically diverse languages. The task consists of two sub-tasks passage selection and minimum answer span (Gold-P). For our ex- periments, we consider the Gold-P task and eval- uate Monolingual and Zero-Shot Cross-Lingual prompting strategies. Since the labels do not di- rectly transfer one-to-one across translation for QA tasks as they do for classification and require the use of alignment algorithms, we skip translate-test prompting for this task.
UDPOS (Zeman et al., 2020) is dataset for Part of Speech Tagging taken from the Universal Depen- dencies 2.5 from the XTREME (Hu et al., 2020) benchmark. We benchmark subset of the lan- guages available UDPOS.
A.2.2 Named Entity Recognition
PANX (Pan al., 2017) WikiANN is Named Entity Recognition dataset consisting of Wikipedia sentences tagged with Person, Organization and Location.
2. MLQA (Lewis et al., 2020) an extractive QA dataset translated into 7 languages by profes- sional translators. The task has two variants, the first where the question, context, and answer are all in the same language; and the second, where the question is in different language than the con- text and answer. We consider the former variant of the task in our experiments. For MLQA, translate- test splits are also available, where each language's test data has been translated into English with an- swers aligned using the attention scores. There is no training data available for MLQA, and we use SQuAD's Rajjuurkr et al. 62)16a) training data for selecting few-shot examples in English and vali- dation data for MI.QA in other languages to get their few-shot examples. This way, we are able to evaluate for all three prompting setups.
For both tasks we use the linguistic structure prompting approach of Blevins et al. (2022) to de- fine the prompts. The exact prompts used can be found in $A.4. Given the nature of both tasks, which would involve token alignment across the translation, we do not evaluate the translate-test prompting strategies for these setups. Also, since both tasks involve > 30 languages, to make the best use of the compute resources we only evaluate GPT-3.5-Turbo in a monolingual setup for these two tasks. Finally, we evaluate the first 1000 exam- ples for each language for these datasets given the large number of languages. We have recomputed all baselines with this specification as well.
A.3
Generation
A.3.1 Summarization
3. XQuAD (Artetxe et al., 2020) consists of pro- fessional translations of subset of the SQuaD dataset (Rajpurkar et al., 20) into 10 languages. XQuAD only has validation datasets available pub- licly, hence we evaluate the models on them. Like MLQA we use English SQuAD data for few-shot examples and since we cannot use validation data in other languages for few-shot, we only evaluate for zero-shot cross-lingual setup for this task.
The XLSum (Hasan et al., 2021a) dataset contains article-summary pairs across 44 typologically di- verse languages, ranging from high to very low- resource.
For a similar reason as the tagging datasets, we only evaluate on first 1000 examples of the test sets in different languages and recompute the baselines on the same testset using the weights of the XI-- SUM pretrained model, opensourced by the authors (Hasan et al., 2021b).
4. IndicQA (Doddapaneni al., 2022) man- ually curated cloze-style reading comprehension dataset that can be used for evaluating question- answering models in 11 Indic languages. The con- text paragraphs are chosen from Wikipedia articles whose topics are closely related to Indic culture, history.etc. The publicly available test set has about 2000 sentences that we carry out our evaluation on.
A.3.2 Code-switching datasets
All the datasets we consider so far are monolin- gual, however, a majority of the world's population speaks more than one language, leading to lan- guage contact phenomena such as code-switching (Doğruöz et al., 2021; Sitaram et al., 2019). We include two code-switching datasets in MEGA benchmark the performance of generative models. GLUECoS-NLA (Khanuja et al., 2020a) is a code-mixed NLI dataset in Findi-Englice consist-
A.2 Sequences Labeling
the sequence labeling task, a sequence of tokens (such as words) to be labeled are provided to the

---

ing of Bollywood (Hindi) movie conversations as premises, with manually created hypotheses.
gender can be finally compared against the gold annotations for English.
The EN-ES- EI-ES-CS CS Sentiment Analysis dataset (Vi- lares et al., 2016). part of the GLUECoS benchmark (Khanuja et al., 2020b) is a code-mixed dataset con- sisting of SpaannhhS Tweets annotated with SentiStrength (Thelwall, 2017) scores.
A.4 Prompts
A.4.1 XNLI, IndicXNLI, GLUECoS NLI
Models -GP--35-TTrbo, GPT-4
Task Instruction Z: You are an NLP assistant whose purpose to solve Natural Language Inference (NLI) problems. NLI is the task of determining the inference relation between two (short, ordered) texts: entailment, contradiction, or neutral. Answer as concisely as possible in the same format the examples below:
A.3.3 RAI datasets
We include datasets that measure the Respon- sible AI (RAT) dimensions of fairness and toxic- ity Jigsaw for toxic comment classification and WinoMT for gender bias.
The Jigsaw dataset contains online comments sourced from Wikipedia. The training data, which is in English, contains labels pertaining the tox- icity of the comment and any relevant identity mentions contained in the comment. We use the test dataset, which contains these comments for languages as illustrated in Table for evaluation. The test dataset contains binary label indicating whether not the comment is toxic. Our objective is assess the performance of these models across multiple languages and observe the disparity in this performance that could arise due number of factors, prominent one being the source data that these models are trained on. Using English prompts from PromptSource for the original mono- lingual Jigsaw task, we task the model with classi- fying comment as toxic or non-toxic. We perform crosslingual few-shot prompting and translate-test experiments for the test sets of all languages, and report the results excluding content violations in Table 21.
Template ftempi
{premise}
Question: (hypothesis) True, False, Neither?
Verbalizer fverb:
Entailment True,
Contradiction: False, Neutral: Neither
Models DV003
Template ftemp- {premise} Based on previous passage is it true that {hypothesis}- Yes, No, Maybe?
Verbalizer foerb:
Entailment Yes,
Contradiction: No. Neutral: Maybe
A.4.2 PAWS-X
Models GPT-3.5-Turbo, GPT-4
The WinoMT dataset (Stanovsky et al., 2019) is created by concatenating the WinoGender (Rudinger al., 201 8) and WinoBias (Zhao al., 2018) datasets. WinoMT dataset consists of 3888 English sentences with equal distribution of Male and Female genders. also equally balanced between stereotypical and non-stcreotypical gen- der role assignments. We follow the method as reported by (Stanovsky et al., 2019) in their paper. We perform zero-shot monolingual prompting of all sentences in the dataset to translate them in 8 target languages. Further using fast fast_align we map the English entity its translation. Finally, we ex- tract the sadget-side entity's using off the shelf tools for each target language. The extracted translated
Task Instruction I: You are an NLP assistant whose purpose is perform Paraphrase Identification. The goal of Paraphrase Identification is deter- mine whether a pair of sentences have the same meaning. Answer as concisely as possible the same format as the examples below:
Template ftempi
{sentencel}
Question: (sentence2)
True or False?
Models DV003
Template frempi Sentence 1:
(sentence1) Sentence 2:
(sentence2) Question: Does
Sentence
paraphrase Sentence ? Yes or No?

---

Verbalizer feerbi
{context)
Positive: Yes
Q: [question]
Negalive: No
Referring to the passage above, the correct answer to the given question is: {answer)
A.4.3 XCOPA
A.4.5 IndicQA
Models : GGP-3.5-Turbb GPT-4
Models GPT-3.5-Turbo, GPT-4
Task Instruction I: You are an AI assistant whose purpose to perform open-domane commonsense causal reasoning. You will be provided a premise and two alternatives, where the task is to select the alternative that more plausibly has a causal relation with the premise. Answer as concisely as possible in the same format as the examples below:
Task Instruction I: You are an NLP assistant whose purpose to solve reading comprehension prob- lems. You will provided questions on a set passages and you will need to provide the answer as it appears in the passage. The answer should be in the same language the question and the passage.
Template ftempi
Template ftomp'
{ premise }
{% if question == "cause" x} This happened because...
{context} Q: [question]
Referring the passage above, the correct answer to the given question is? If you can't find the an- swer, please respond "unanswerable", [answer)
{% else x} As consequence.. (% endif %} Help pick the more plausible option: (choicel) [choice2)
Models DV003
Models DV003
Template ftempi
Template ftemp:
{context)
{ premise }
Q: (question]
{% if question == "cause" x} This happened because...
Referring to the passage above, the correct answer to the given question is: (answer)
(% else %} a consequence... (% endif %} Help me pick the more plausible option: choicel: (choice1), ), choice2: (choice2)
A.4.6 XStoryCloze
Models DV003, GPT-3.5-Turbo, GPT-4
Verbalizer feerb:
Template ftompi {input_sentence_1] (input_sentence_2
choicel: (choice1)
choice2: (choice2)
(input_sentence_3) (input_sentence_4
What is possible continuation for the story given
A.4.4 XQUAD, TyDiQA, MLQA
the following options
Models : -GPT-3.5-urbb, GPT-4
Optionl: {sentence_quiz1) (sentence_quiz)
Option2:
Task Instruction I: You NLP assistant whose purpose is solve reading comprehension prob- lems. You will be provided questions on a set of passages and you will need to provide the answer as it appears in the passage. The answer should be in the same language as the question and the passage.
Verbalizer fvert: peetencce_qizzle Option1, $sentence_quiz2) Option2
A.4.7 PANX
Models GPT-335-Turbo, GPT-4
Template ftemp:
Task Instruction I: You are an NLP assistant whose purpose is to perform Named Entity Recognition (NER). NER involves identifying and classifying named entities in a text into predefined categories such person names, organizations, locations, and others. You will need use the tags defined below: means the word doesn't i correspond to any entity. B-PER/I-PER means the word corresponds to the
(context)
Q: {question}
Referring to the passage above, the correct answer to the given question is: (answer)
Models DV003
Template frempi

---

beginning of/is inside person entity. B-ORG/- ORG means the word corresponds the beginning of/is inside an organization entity. B-LOC/1LLCC means the word corresponds to the beginning of/is inside a location entity. Do not try to answer the question! Just tag each token in the sentence.
A.4.9 GLUECOS Sentiment Analysis
Models GPT-3.5-Turbo, GPT-4
Task Instruction Z: You are an NLP assistant whose purpose is to solve Sentiment Analysis prob- lems. Sentiment Analysis is the task of determin- ing whether the sentiment, opinion emotion pressed in a textual data is: positive, negative, neutral. Answer as concisely as possible in the same format as the examples below:
Template Stemp: [token_1
token_2
token_n3
Verbalizer feerb:
{tag_1} [tag_2] ... (tag_n):
Template ftempi Does the following sentence have positive, negative or neutral sentiment? (text)
[ocken_1]_(tagp (token_2)_(tan) (tag__ ("tokennn__(ag__)
Models DV003
A.4.8 UDPOS
Template ftempi Does the following sentence have positive, negative or neutral sentiment? (text)
Models : GPP-3.5-Turbo GPT-4
Task Instruction I: You NLP assistant whose purpose is to perform Part Speech (PoS) Tagging. PoS tagging the process of marking up a word in a text (corpus) as corresponding to a particular part of speech, based on both its definition and its context. You will need to use the tags defined below:
A.4.10 XLSum
Models GGPT-3.5-Turb,, GPT-4
Task Instruction I: You are an NLP assistant whose purpose to summarize any given article. You should summarize all important information con- cisely in the same language in which you have been provided the document. Following the examples provided below:
1. ADJ: adjective
2. ADP: adposition
Template frempi {document)
3. ADV: adverb
4. AUX: auxiliary
5. CCONJ: coordinating-conjume conjunction
Write summary of the text above
6. DET: determiner
A.4.11 Jigsaw
7. INTJ: interjection
Models GGPT-3.-TTurbo, GPT-4
8. NOUN: noun
Task Instruction I: You are an NLP assistant whose purpose is to solve the Jigsaw Toxicity Prompt problem. Toxicity is defined anything rude, dis- respectful otherwise likely to make someone leave discussion. The objective is to identify toxic comments. Answer as concisely as possible in the same format as the examples below:
9. NUM: numeral
10. PART: particle
11. PRON: pronoun
12. PROPN: proper-noun
13. PUNCT: punctuation
Template ftempi (text) Should this online com- ment be removed for its toxicity? Yes No?
14. SCONJ: subordinating-coonjuttiin
15. SYM: symbol
Models DV003
16. VERB: verb
Template fremp: {text} Someone posted this com- ment online. Do you find it toxic? Yes or No?
17. X: other
Template
ftempi token_n}
[token_1
token_2
A.4.12 WinoMT
Models :GPT-3.5-Turbo GPT-4
Verbalizer foerbi
(tag_1} 2} ... (tag_n):
Template fromp: Translate the following English text to (target_ languugee_ (sentence)
(token_1)_(tag__} (token_2)_(tag_3 (token_n)-(tag_n)

---

Language
Retrieval Acc.
A.6 Factors Explaining Multilingual Capabilities of LLMs
0.858
0.492
bn
0.141
We provide correlation plots in Figures 7 (between performance and fertility) and 8 (between perfor- mance and pre-training size) for both GPT-3.5- Turbo and GPT-4. The exact values of the correla- tions for tasks and the two models is provided in 5.
0.756
0.680
0.760
0,453
0.056
0,421
Table 4: Retrieval accuracy on TyDiQA dataset for
chunk size 100.
A.7 Challenges in Multilingual Evaluation
Effect of number of in-context examples Our main experiments were conducted with k = 8 or k = 4, depending on the task. Here, we evaluate what effect different numbers of in-context exam- ples have XNLI and XCOPA for three languages in Figures 6a and 6b. We observe while the perfor- mance increases sharply while moving from 2-4 examples, the exception Haitian Creole in XCOPA, where continues to improve.
A.5 Handling Long Contexts
As discussed in $2.2, the models we study have limited context lengths and for QA tasks in par- ticular, fitting the entire prompt containing the few-shot examples is often not fcasible for low- resource languages where the tokenizers of these models are found over-tokenize the text (nearly resulting in byte level tokens). To overcome this issuc we follow two steps. First, for the few- shot examples we only provide the line within the paragraph containing the answer as the con- text. Second, for the test example, we index the chunks of the context using the embeddings from the text waw.bedinng-dd-aaa-e question, the closest chunk in the full context is retrieved and used in the prompt for the test ex- ample. We use a maximum chunk size of 100 in our experiments and use the implementation for retrieval provided in the LangChain library. By doing this,we minimize the space taken by the con- text tokens in our prompt. Note that, for newer GPT models i.c. G and GPT-4 which support longer context lengths, hence only use this retrieval strategy for DV003 QA tasks.
fairly stable after k 2 8, with
Effect of language-speeific prompt tuning. As discussed in $2.3.1, we use English validation data for prompt selection in each dataset that we use for all languages. Here, we explore whether sepa- rately tuning the prompts for each language helps. For XNLI, run this experiment on Urdu and Swahili, tuning over different prompt templates from Prompt-Source but find that the same prompt that was tuned for English gets picked up for these two languages as well. For XCOPA however, dif- ferent prompts are chosen when tuned on Haitian Crcole and Tamil. This leads to an improvement in the test performance for Haitian Creole (from 72% to 75.6%, see Figure 6c). Interestingly for Tamil, the test performance actually drops slightly compared the accuracy obtained with prompt se- lected on English data, which we conjecture might be due to the fact that the validation sets in XCOPA have only 100 examples that may not be sufficient for selecting optimal prompts.
model. Given the
We attribute the significantly worse performance of DV003 on IndicQA to imperfect retrieval in the case of DV003, while for GPT-3.5-Turbo do not rely on retrieval due to the larger context size. We provide the retrieval accuracies for DV003 (i.e. if the retrieved chunk contains the answer) in Appendix Table 4, where we clearly see for low- resource languages like Telugu, the accuracies can be as low as 5%. While beyond the scope of this work, alternate retrieval strategies like using better embeddings from multilingual models for retrieval be explored to close this gap (Nambi al., 2023).
Effect of Explanations. Ye and Durrett (2022b), showed for sel-dddvviiiee@ee that prompting the model with explanations before the outputs Mixplain-then-Predee in the in-context examples can help improve few-shot performance substan- tially English language datasets. Hence, here we evaluate if they help improve the multilingual per- formance of the GPT-3.5-Turie model as well. We perform experiments on XStoryCloze and XCOPA datasets and use the explanations available in Super-
'https:/ http://gggtttttteee wttg:/twww..cm///cccha//////eeee

---

GP -urbbo
1.0
0.8
0.6
English
Score
0.4
0.2
0.0
4
6
8
10
Tokenizer Fertility
IndicQA, P -099
XNLI, P = - 0.79
XCOPA, = 0.98
XQUAD, = 0.77
.
XLSum, P -0.82
XstoryCloze, P 0.75
(a) Correlation between tokenizer fertility and performance for GPT-3.5-Turho
GPT-4
1.0
English
0.8
Score
0.6
2
6 Tokenizer Fertility
4
8
10
IndicQA, p 0.86
XQuAD, p - 0.71
XCOPA, 00.96
XstoryCloze, p -092
XNLI, p= -= -0.8
Correlation between tokenizer fertility and performance for GPT-4
Figure 7: Correlation between the performance of GPT-3.5-Tuufe and GPT-4 with the tokenizer fertility.

---

GPT-3.5-Turbe
1.2
1.0
Score
English
Oriya
0.8
0.6
0.4
4
6
8
10
12
log (PretrainSize)
PAWS-X, / = 0.85
XNLI, = 0.9
XCOPA, p -= 0.7
XQuAD, 0.76
(a) Correlation between pre-training size and performance for GPT-3.5-Turbo.
GPT-4
0.8
English
Score
Oriva
0.6
0.4
0.2
4
8
10
12
log PretrainSize)
IndicQA, p 0.69
XNLI, p 0.84
MLQA, p 0.8
XQuAD, 0.71
PAWS-X, = 0.92
(b) Correlation hetween pre-training size and performance for GPT-4
Figure 8: Correlation between the performance of GPT-3.5-Turbo and GPT-4 with the pre-training size.

---

Tokenizer Fertility
Pre-training Size
GPT-3.5- burbo
GPT4
3..--35-
GPT-4
Task
P-value
p
P-value
p
P-value
P-value
XNLI+IdicNNLI
-0.784
6.9e 05
-0.803
3.4c 05
0.893
4.1e - 09
0.836
3.5e 07
XCOPA
-0.982
7.9e
-0.957
0.70
0.035
0.489
0,181
XstoryCloze
-0.745
0.033
-0918
0.001
0.603
0.064
0.407
0.242
PAWS-X
0.587
0.219
0.61
0.198
0.85
0.031
0.94
0,005
MLQA
-0,451
0.368
0.674
0.141
0.71
0.085
0.808
0.051
TyDiQA-GoldP
0.543
0.163
0.049
0.907
00.644
0.207
-0.159
0,682
XQuAD
-0.65
0.00
-0818
0.002
0.782
0.004
0.736
0.009
IndieQA
0.960
0.002
0.856
0.029
0.628
0.051
0.690
0.027
WinoMT
0.36
0.379
0.249
0.589
Jigsaw
0.306
0.554
0.674
0.141
PAN-X 00.45
0.003
-0.42
0.004
0.41
0.006
0.326
0.032
UDPOS
00226
0.198
-0.30
0.066
0.29
0.095
0.359
0.036
XLSum
-0.821
4.8e
-0.578
0.002
0.448
0.011
0.49
0.005
Table 5: Pearson Correlation coefficient between performance and tokenizer fertility and performance and pre-training data size for different datasets and models. We also provide the p-values, which correlations are statistically significant.
NaturalInstructions (San) al., 2022)?. All the explanations that we used were written in En- glish. For XStoryCloze, the results are plotted in Figure 6d, and we observe that while there is a slight gain upon using explanations for Telugu, for all other languages the performance remains largely unchanged if not slightly worse. Interestingly, upon manual inspection of the model's prediction, observe that the model often first translates the problem English and then proceeds with the ex- planation, without having prompted to do so. We have similar observations for the XCOPA dataset as well, where adding explanations doesn't t help improve performance and ends up hurting the per- formance by slight margin (Figure 9)
Effect of Explanations XCOPA
0.7
0.6-
0.5
Apcuracy
0.4-
0.3
0.2
Effect of the language of the prompt templates. While all our experiments were run using prompt templates written in English, we initially evaluated DV003 on aative-Language--eeplaa which were obtained by translating English tem- plates using Bing-Transsltor. As can be in Table the performance much worse when us- ing templates the native language compared to English. This is consistent with the results in Muen- nighoff et al. (2022) for BLOOMZ and Lin et al.
0.1
Haitian Creole
Tamil
as well,
Language
Figure 9: Effect of using explanations in XCOPA dataset. Blue bars explanations in the prompt and orange bars correspond to prompting with explana- tions.
Att the time of writing this paper, XStoryCloze wasn't included in SNL, hence we use the few-shot examples and explanations available for StoryClaze datasct(Mostafe Mosttaaradd et al., 2016), making the prompting setup Zcro-Shot Cross- Lingual.

---

Task
English- English-Template
Nutve-Lunguuge-eepluue
formance when using prompt templates in English.
XNLI
58.3
54.4
Indic-XNLI
49.6
38.7
PAWS-X
67.1
64.2
A.8 Detailed Results
XCOPA
77.6
73.1
The results across all tasks, languages and mod- els included in our benchmarking exercise can are provided in Figures 10 (for Classification tasks), 11 (for QA Tasks), 12 (for XLSum), 14 (for PAN-X), 13 (for UDPOS), 15 (for Jigsaw), and finally 16 (for Wino-MTT. The results for the Indic Datasets and the two code-mixed datasets GLUECOS NLI and En-ES-CS are provided in Tables 8 respectively.
Table 6: Average performance non-English lan- guages with the Monolingual Prompting strategy us- ing English-Template and Native-I aaggagge-Templle prompts for the classification tasks for DV003.
Model
IndicXNI.I
IndicQA
MuRII.
72.4
47.7
text-davinci-003
49.6
8.45
text-dawinci-000 (TT)
62.4
3pt-3.5-turbe
50.7
38.6
5-turbo (TT) gDt-4-32k
59.7
x
66.8
55.0
Model
NLI En-Hi
Sentiment En-Es
Table 7:
performance of text-davinci-009, gpt-3.5-turbel and gpt-4-32k with fine-tuned baseline MuRIL Indic datasets (Doddapaneni al., 2022). For IndicXNI.I we report Accuracy and Fl-score for IndicQA.
Comparing
mBERT
63.1
69.31
text-davinci-00
72.1
68.8
5pt-3.
78.8
68.0
Table 8: Performance of GPT-3.5 models code- mixing datasets from (Khanuja et al., 2020b). For both tasks, metric reported accuracy.
(2022a) for XGLM, which also show better per-

---

Acunzy
06
11.4
0.2
0.0
Frofas
Redgarion Rumian
Hindi
Thai
Swahill
Undu
Language
1000 Werd ag
FANEA Truasar
Tacles Mixilegall
Tarle
Monilagall
ThE
(a)
XitaryClaze
0.8
Accuney
11.9
0.2
0.0
Faglas
Sparish
Indonmaant
Rywan
Anhr Language
Hind
Ronega
Soahi
Tolugi
Rurmen
(Mull Mendong
DVbn Trenkrttee
GPT So Taato Menwinge
Toree
GPI4 Trendett Toml
R ROW
(b)
IndicXNLI
0.5
0.5-
A
0.2
0.0
Hind
Bangal
Panjabi
Karnada
Gujarati
Malayslam Language
Marath
Telagu
Tamil
Only
Assamose
EX0O Mortiingal
DVDCD Thonsbri
GPTT-3. Turbs Meunlinga
Tuato Trand
Mandagga
MARI
(c)
XCOPA
1.0
0.5-
7o5-
0.44
0.2
Isfoemion
FoRon Crakk
Languge
(d)
Figure 10: Comparing the language-wis wise performance of different models on the classification tasks.

---

XQuAD
0.5
21
0.4
0.22
English
Sparish
Vietnameee
Gorman
Aralic
Turkish
Chinsee
Hindi
Russian
Greek
Language
Teas Canss
GPT.4 Skar
Skir
(a)
MLQA
0.5-
SSone
0.4
English
Spanish
angguage
Dvass Mondinga
De060 Turakee Tox
GPT-5S Tuaks Mur ciingel
GFT-S5-Tata Tramins Tei
GPT4 Maniinga
TULRS
(b)
TyDiQA GoldP
0.8
87 Store
0.0
English
Swahli
Russion
Language
DV000 Mo wil agai
S- Turbo Movelingal
144 Maroingee
TUL.Ro
ELOONZ
(c)
IndicQA
0.66
F1 Score
0.1
0,0
Hindi
Bengol
Marothi
Gujaroti
Assamese
Telugi Languags
Taml
Punjabi
Kanrado
Malayslom
Oriyo
daw Shos- Cran Uigal
GPI-355 lurb Aur--boet
Liquu
Lar-2in Com Uiguur
BLOUML
(d)
Figure 11: Comparing the language- wise performance of different models the QA tasks.

---

XLSum
0.3
ROJGE-
0.2
English
Chinaon
[S] Chionin (T)
Hausa
Hind
French
Arabic
Bengali
Azer bajanii
Gujarati
Amhar
Burnese
Language
(PPT Tarko
CPT.4 Maniiga
XLSum
0.3
ROUGEEL
0.1
Japanise
Pidgin
Iabo
Kytgyz
Language
GPT-36 6.2.6TTo Mondingal
CPT.4 Maniigga
wITS
(b)
XLSum
ROUGE-L
0.2
0.1
0.0
Portaguese
Punjabi
Seottish_gpeli
Spanish
Cyrilll
Teluga
Language
CPT-35- Turoo Mondingal
CPT4 Meckiggd
miTS
(c)
XLSum
0.3
RO00E-
0.2
0.2
0.0
Urd
Welsh
Vietnomese
Ulrainion
Yorubo
Uzbek
Language
Mondingal
(d)
Figure 12: Comparing performance of different models on XLSUM.

---

UDPOS
1.0
0.5
0.6
English
German
Dutch
Bulgarian
French
Mrikans
Estonian
Finnish
Basque
Greek
Chinsse
Arabic
Language
GPT Manelng
(a)
UDPOS
0..5
82 SSone
0.2
Lihhannan
Hungarian
Marathi
Kazakh
Hcbrow
Japanase
Language
CPTS5 Tasbe Moedingal
GPT4 Mandinga
HOBET
(b)
UDPOS
SSure
0.6
0.2
0.0
Portuguese
Telugu
Russion
Romanion
Tagalog
Spanch
Language
GPT-355- Turbo Mondingal
GPT- Mank qua
XU/SR
(c)
UDPOS
0.4-
F1 Score
0,0
Yorubo
Wolsf
Language
GPI-355 lurs Meesingal
GF1-
nobxi
(d)
Figure 13: Comparing performance of different models UDPOS

---

PAN-X
0.5
F1 Score
0.4
0.24
English
Dutch
Estonin
Finrish
Bulganin
Mflkans
Bengali
Azorbajjani
Dasque
Arabic
Burnese
Chirese
Language
GPT Mariing
(a)
PAN-X
01.5
0.5
F1 10orre
0.4
0.2
Cerman
Hungarian
French
Javanise
Crock
Indonssian
Hrbrow
Goorglan
Gujarati
Japanase
Language
CPT.35 Tasbe Mondingal
GFT4 Mandinga
(b)
PAN-X
Sture
0.4
0.2
0.0
Polsh
Lituuonian
Portuguese
Romonian
Mala
Quechua
Koreon
Malajolam
Punjabi
Language
GPT-35- Turbo Mondiiggal
GPT- Mank ua
MISENT
(c)
PAN
0.8
F1 Score
0.24
0.0
Turksh
Spanish
Swahii
Tagalog
Vietnamese
Ukramian
Russin
Yoruba
Urdu
Tamil
Telugu
Thai
Language
GPI-S furbe Meesingall
GF1-4 Mano yua
XUIMI
(d)
Figure 14: Comparing performance of different models on PAN-X

---

0.8
Actrrcc
34
0.1
34
0.0
Portuguese
French
Language
Ov00s TrralacToo
DV333 Zeo-Siet Cros Lingad
TTT-3--aabo TrenkitsTor
GPT-1S-Tuta Zeto-She Groe Lirgxl
PaLM Fewe
Figure 15: Comparing performance of different models on the Jigsaw datasct.
WinoMT
0-5-
Accuracy
0.1
0.2
0.00
Arabic
Hebrow
Ukrainien
Russian
Language
GPT--S-Terte Mensingual
BLOONZ
Bing Tremilttor
Googl: Translator
Figure 16: Comparing performance of different models on the WinoMT dataset.
Model
en
ar
bg
de
es
fr
hi
TU
SW
th
tr
Wr
Vi
zh
avg
Pine-tuned Baselines
MBERT
80.8
64.3
68.0
70.0
73.5
73.4
58.9
67.8
49.7
54.1
60.9
57.2
67.8
65.4
mTS-Base
84.7
73.3
78.6
77.4
77.1
80.3
79.1
70.8
77.1
69.4
73.2
72.8
68.3
74.2
74.1
75.4
XLM-R Large
88.7
77.2
83.0
82.5
80.8
83.7
82.2
75.6
79.1
71.2
77.4
78.0
71.7
78.2
79.2
TuLRv6 XXL
89.0
90.6
90.7
86.2
89.2
85.5
87.5
88.4
82.7
Prompt- Based Baselines
BLOOMZ.
67.5
60.7
46.5
54.0
47.4
61.2
61.4
56.8
53.3
50.4
43.8
42.7
50.0
61.0
56.7
54.2
XGLM
52.6
46.4
48.9
45.6
48.7
45.8
49.4
46.8
48.6
44.5
46.6
45.4
43.4
48.5
48.8
47.3
Open Models
gpt-3.5-turde
76.2
65.1
67.7
55.5
62.5
56.3
54.0
62.6
62.1
8pt-3.5-turbo (TT)
76.2
63.7
55.8
59.6
text-davinci-003
79.5
52.2
65.7
47.6
62.2
50.2
51.1
57.9
50.0
56.4
text-davinci-ee (TT)
79.5
65.1
70.8
71.7
72.2
71.8
63.3
67.3
57.3
62.0
67.6
55.1
66.9
67.1
apt-4-32k
84.9
73.1
79.0
78.8
79.5
72.0
74.3
70.9
68.8
76.3
68.1
74.6
Table 9: Comparing performance of different models on languages in XNLL Metrie: Accuracy.
Model
bn gu
hi
kn
ml
Or
ta
te
avg
Fine-tuned Baselines
MuRII.
76.0
75.0
77.0
77.0
77.0
79.0
74.0
76.0
77.0
77.0
74.0
76.0
Open Al Models
gpt-3. 5-turbo
53.6
50.6
55.5
53.9
48.4
49.9
47.4
53.6
48.2
47.4
50.7
gpt-3.5-turbo. (TT)
54.3
61.6
61.8
59.6
60.8
59.9
58.7
58.5
62.3
58.3
60.8
59.7
text-davinci-000
48.6
52.6
51.2
56.9
49.1
48.2
49.4
46.4
50.4
45.5
47.2
49.6
text-davinci-002 (TT)
56.0
66.0
64.7
62.6
63.9
61.8
60.9
60.8
64.7
61.8
63.1
62.4
gpt-4-32k
63.5
72.2
66.9
71.7
69.0
64.3
66.2
61.1
71.1
63.7
64.8
66.8
Table 10: Comparing performance of different models all languages in IndicXNLL Metric: Accuracy.

---

Model
en
de
es
fr
ja
ko
zh
avg
Fine-tuned Baselines
mBERT
94.0
85.7
87.4
87.0
73.0
69.6
77.0
81.9
mT5-Base
95.4
89.4
89.6
91.2
79.8
78.5
81.1
86.4
XLM-R Large
94.7
89.7
90.1
90.4
78.7
79.0
82.3
86.4
TuLRv6 XXL
97.2
95.1
94.8
95.6
89.4
90.4
90.4
93.2
Prompt-Based Baselines
BLOOMZ
89.8
84.3
88.9
87.5
74.4
85.8
65.2
82.3
Open AI Models
3pt-3.5-turbe.
72.4
70.6
72.0
72.1
67.2
66.5
69.2
70.0
gpt-3. 5-turbo (TT)
72.4
70.8
69.7
70.1
61.9
62.5
63.1
67.2
text-davinci-003
72.5
70.6
72.7
70.7
60.6
61.8
60.8
67.1
text-davinci-out (TT)
72.5
69.8
70.1
71.3
65.4
65.8
65.2
68.6
gpt-4-32k
76.2
74.0
74.1
72.6
71.5
69.9
72.6
73.0
Table II: Comparing performance of different models on all languages in PAWS-X, Metrie: Accuracy.
Model
cn
ct
ht
id
it
qu
SW
ta
th
tr
avg
Fine-tuned Baselines
mT5-Base
50.3
49.9
49.2
49.6
50.5
50.4
49.2
50.7
49.5
49.9
TuLRv6 XXL
77.4
78.0
92.6
96.0
61.0
69.4
85.4
87.2
92.8
74.0
Prompt- Based Baselines
BLOOMZ
88.0
48.0
55.0
86.0
74.0
50.0
60.0
67.0
50.0
54.0
63.2
XGLM
65,9
58.9
68.9
69.2
47.1
62.9
56.3
62.0
58.5
61.1
Open Al Models
gpt-3.5-turbo
97.8
90.6
72.0
90.4
95.2
54.6
82.0
59.0
77.6
91.0
81.0
gpt-3.5-turbo (TT)
97.8
88.2
79.4
90.8
94.4
50.0
77.6
87.0
82.2
87.8
83.5
text-davinci-003
98.2
87.8
75.0
91.4
96.0
54.8
63.6
53.8
66.6
87.8
77.5
text-davinci-e00 (TT)
98.2
89.6
82.8
93.0
94.6
50.0
82.8
87.0
84.8
89.8
85.3
gpt-4-32k
99.6
98.8
93.2
97.6
99.8
58.6
94.4
79.6
87.8
97.4
90.7
8pt-4-32k (TT)
99.6
94.4
85.8
96.0
98.2
85.8
83.4
91.4
87.8
92.2
90.6
Table 12: Comparing performance of different models on all languages in XCOPA, Metrie: Accuracy.

---

Model
avg
Finr-noue Rasefinry
iBEER
83.5/722
61.5745.1
70.57 54.0
62.67499
75.5/5699
59.214400
71.31
427/55
554/401
/9.5
58.07 48.3
64.5/494
mT5-Base Large
846/7117
46.3
73./54.5
35.6
748/56.1
60.31 43.4
57.674577
6797482
66.1/441
67.0/490
86.5/757
68.61490
8044633
718/61.7
820/639
76.7/59.7
759/593
79.1
593/500
1o6- XXL.
85.4/ 69.6
86.1/70,4
78.4
87.6/1.9
85.9/70.5
868/7322
87./8111
84.37718
87.6/11.
79.2/73.2
86.072.9
Prompp- Bassd Bovelines
16.37604
75.7/58.8
Open Mudelly
got-3. Suturbo
59.6/5511 58.3
70.67 46.6
40.0 22.8
70.3 70.3/408
54.07 39.
58.0/313
419 10.4
618/350
50.4 75.8/49.7
69.1
604/38. 00.0/575
text-davinci got-6-32K
832/65.6
67,8/424
71.1 48.7
62.31 36.6
77.750.7
63.93667
63.8/35.8
51./420
46.6
68.3/466
Table 13: Comparing performance of different models all languages in XQuAD. Metric: F1 Score / Exact Match.
Model
En
bn
8i
id
ko
avg
Pine-tun fne-ttned Raselinet
MBEKT
63.3/
62.2/42.8
49.3//3277
59.7 455.3
64.8/458
58.8/50.0
60.0/3888
51.5/379
49.6 38.4
59.7/43.9
mi[S-Base
71.8/60.9
67.1/504
20.7/22.1
67.0/522
71.3/54.5
/9,5
$4.93266
40.6/31.1 60.4/439
58.1/42.8
XLM-R Large
71.5//568
67.6 40.4
64.0/47.8
70.5/53.
714/619
30.9/
67.0/42.1
66. 16.1 /48.1
70.1
65.1 /45.0
Tul Rv6 XXL.
85.4/76.4
84.1 70..4
86.9.79.5
8388/228
88.8/77.9
78.5/67.8
68.6
87.2/79.6
85.2/71.6
84.6/
Prompt-Basee Based Bavelines
BLOOMZ
72.4/70.9
81.9/62.2
87.8/82.3
43.6/
85.0/71.0
52.3/43.1
67.4/51.5
860/772
73.2/63.2
Open Models
apt-3.5-turbo 5-turbo
$4.8/30.7
50.9/24.2
60.7/32.7
66.6/49.0
63.2/43.4
59.7 /45.3
45.8/20.0
64.3/47.7
70.9 53.1
60.1 38.4
text-davinci-488
59.1
56.2 338.7
16.1 10.6
70.3 58.8
68.6/51.2
40.6/32.2
42.3 /28.9
/
5.8 3.0
49.8/38.3 38.3
spt-4-32k
52.9/114
60.8/32.7
68.0/42.5
75.4/57.7
80.8/6111
69.7 /5.7
61.4/30.5
81.8 68.7
72.5/54.9
/
Table 14: Comparing performance of different models on all languages in TyDiQA. Metric: Score / Exact Match.
Model
en
de
Vi
avg
Fine-tuned Baselines
mBERT
67.2
52.3/
59.0/43.8
47.4/49.2
50.2/35.3
61.2 40.7
59.6/38.6
41.4/44.2
mTS-Base
81.7 66.9
57.1/36.9
/43.2
67.1
65.9/44.1 53.9
61.6/38.6
64.4/45.0
XLM-R Large
83.5 / 70.6
66.67 47.1
70.1 54.9
74.1/56.6
70.6/53.1
74.0 52.9
62.1
71.6/ 53.2
TuLRv6 XXL
86.6/ 74.4
76.2/56.5
80.2 67.0
81.7/65.J
82.2/ 64.8
82.3 / 63.2
7
81.0 63.9
Open Models
8pt-3.5-turbo
72.8 / 53.2
48.5 /3.9
51.0/ 29.6
29.44
50.7 28.
58.9 35.1
29.7
56.1 /2.1
5pt-3.5-turbo (TT)
72.8 / 53.2
37.8/18.
44.3/ 26.
54.1/31.
37.3/20.0
12.5
36.5/
46.4 /
text-davinci-083
74.8 / 59.0
38.4/217
57.7/38.
24.9/ 62,9/3788
47.7
32.3/3177
48.4/33.1
text-davinci-003 (TT)
74.8 59.0
48.2/25.6
53.5/33.9
49.2/28.7 /4.9
$1.0/30.4
24.2
55.0/ 34.7
gpt-4-32k
80.3 62.8
59.1/33.5
/4.7/44.4
70.0/45.9
57.3/35.6
72.2/49.0
/3../
67.2 44.2
Table 15: Comparing performance different models on all languages in MLQA. Metrie; F1 Score Exact Match.
Model
16
ba
g2
122
OC
p0
12
1e
A0g
Finr-nused Batetines
BLOOMM
4h.73317
42.9/
440/44.11
378/266
30.5/284
392/333
25///201
264/335 30./355
34.
366/325
Open Models
Sat-u_S-turbo S-turbo
35.3/214
/
40.5/25.5
$5.91 39.3
35.3/204
30.0/192
30.033200
22./112
358/15.1
323//116
32.9 19.7
38.2/234
toxt-davinci-003
6.73322
10.3/58
10.7441
42/25
8./48
got-4-32%
5K8/404
67.1/474
39.4/42.4
75.26222
47.1/316
48.3/33.7
60.7/4.1
29.91667
56.1/411
$4.8/39.7
47.9 27.8
55.03881
Table 16: Comparing performance of different models all languages in IndicQA Metrie: Fl Score / Exact Match.

---

Model
bg
de
61
X.
Fine-tuned Rovelines
70.4
XL.-R Large
97.0
89.2
63.0
88.3
91.2
N6.5
89.2
87.3
74.9
70.88
N2.7
N6.7
67.5
75.2
83.4
75.7
89.3
29.3
78.3
Open AI Models
gpt-3. bot-a.s-
78.5
74.3
38.3
79.1
80.7
47.1
34.8
76.0
72.0
46.7
79.5
78.0
53.8
50.7
65.4
63.6
75.4
47.4
64.8
upt-4-12k
84.1
77.6
42.0
83.1
86.3
49.8
68.4
80.2
79.3
46.4
82..7
85..4
60.4
52.2
68.3
68.6
84.1
60.2
71.8
pl
y0
BV2
Fine taned Baselines
iBERT Lurge
51.7
78.8
68.7
88.6
80.7
88.0
71.5
N2.4
58.5
75.2
41.3
80.5
70.5
M0.6
56.6
55.6
0.0
56.6
59.6
71.9
Open Af Models
got-3. got-4-37k
39.0
21.3
57.9
78.3
81.7
76.7
66.7
69.9
32.6
79.8
25.5
34.3
77.2
58.9
39.9
57.7
50.4
7.0
57.2
60.2
73.7
81.81
80.7
81.0
66.31
74.7
34.7
84.6
31.21
58.4"
59.1
63.5
66.6
Table 17: Comparing performance different models on languages POS. Metric: Score. (All numbers are Monolingual results except the ones marked with symbol which indicate Zero-Shot Cross- results (due to the absence of training data in those languages)
Flu-stood Boneiies
XIM-R Large
85.
Open AT Modele
apt- tutbbo
43.2
43.8
45.4
42.1
51.6
40.3
52.7
41.0
60.2
58.7
31.5
50.3
50.1
50.7
184
34.3
45.5
53.7
58.4
60.0
$1.7
25.1
30.9
20t-4-324
19.7
55.9
59.4
59.6
62.6
$2.7
69-2
$4.4
68.6
74,4
57.5
67.6
71.1
68.5
23.8
48.0
59.4
71.9
72.7
72.8
68.8
31.6
45.3
Ko
Parnduned Rarelmee
Open
gpt 1.sttrbb
22.0
51.
32.0
3pt-4-12k
31.4
113
35.0
47.4
64.1
16.3
07.9
29.8
70.3
64.5
69.8
59.0
64.8
68.9
36.9
3300
61.9
72.9
58.0
58.4
73.9
18.5
55.5
Table 18: Comparing performance of different models on all languages in PAN-X. Metric: F1 Score.
Model
en
0S
hi
id
rU
SW
te
zh
avg
Prompt-Based Baselines
BLOOMZ
79.7
95.7
87.3
70.5
79.9
85.6
49.9
67.3
65.3
67.4
90.0
76.2
XGLM
59.8
75.9
69.2
63.8
62.5
70.8
61.2
72.4
65.2
63.4
67.7
66.5
Open Al Models
gpt-3.5-turbo 5-turbo
92.5
96.8
95.8
78.4
91.1
95.0
57.2
96.6
92.3
73.1
95.6
87.7
5-t-rbo (TT)
94.3
96.8
96.1
92.5
94.7
95.2
88.6
96.2
88.7
93.6
95.6
93.9
text-davinci-0es
87.4
98.3
97.6
78.1
77.8
96.4
47.4
94.2
78.1
57.6
95.0
82.5
text-davinci- (TT)
98.3
96.2
94.1
95.1
95.9
90.1
96.9
90.7
94.3
96.2
gpt-4-32k
99.1
99.6
99.5
97.6
98.8
99.0
77.6
99.1
98.4
93.4
99.2
96.5
gpt-4-32k (TT)
97.7
99.6
98.7
96.8
97.9
98.1
93.2
99.2
93.6
96.4
98.3
97.0
Table Comparing performance of different models on all languages XStoryCloze, Metric: Accuracy.

---

Google
Microsoft
Amazon
Systran
GPT Turbo 3,5
Bloomz
Acc
Ag
A
Acc
Ag
As
Acc
Ag
As
Ace
Ag
As
Acc
Ac
As
Acc
As
50.9
23.2
20.9
45
36.5
22.9
57.2
15.3
21.7
42.5
46.2
15.6
54.9
22.7
26.2
55.6
17.2
32.5
61.6
22.3
44.5
34.2
15.8
54.2
16.4
15
43.4
41.8
-0.1
52.7
21.4
26.1
17.8
24.6
it
38.6
32,9
18.6
38.8
41.8
10.5
40.2
26.8
14.7
38.1
47.3
6.3
45.1
21.9
26.7
45.7
9
18.5
37.8
36,7
11.4
36.9
42
8.4
39.8
34.8
9.4
37.3
44.1
9.2
41
31.6
10.2
5.9
INV
0
38.4
43.5
10.7
41.3
46.8
11.9
28.9
22.4
12.9
42.9
34.2
12.1
16.8
22.7
2.2
50.8
11.7
35.5
44
29.8
48
13.6
45.9
43.1
26.9
23.1
57.5
7.6
40.8
27.5
31.4
ar
45.8
42.5
16.2
45
47.1
14.2
48.3
37.8
18.8
45.6
49.4
4.1
61.1
13.9
27.9
48.1
23
25.6
de
59.4
12.5
12.6
74.1
8.8
62.4
12
16.7
48.5
34.5
10
57.5
19.5
14.2
47.6
56.2
6.6
Table 20: Performance of commercial MT systems and LLMs are categorized by language family. Acc indicates overall gender accuracy (% of instances the translation had the correct gender). AG denotes the difference in performance (F1 score) between masculine and feminine scores, and Ag is the difference in performance (FI score) between tro-stereotypical and anti- aati-stereotypical gender role assignments (higher numbers the two latter metrics indicate stronger biases). Numbers in bold indicate best accuracy for the language across all systems. Notes: [l. For Google, Microsoft, Amazon, and Systran we use the translations provided by (Stanovsky et al., 2019). Some values differ from the original paper due to updated Spcay modules. For Ru in Bloomz, Precision in male predictions is 0 lcading to Invalid (INV) in Ag]
WinoMT corpus on target languages. Results
Model
es
fr
pt
r1
tr
avg
LLM Baselines
PALM (0-Shot)
79.83
78.99
77.58
80.35
84.1
80.17
PALM (10-Shot Monolingual)
91.23
86.16
90.99
92.47
84.5
89.07
PALM-2 (O-Shot)
88.6
84.11
87.68
90.5
93.42
88.86
PALM-2 (10-Shot Monolingual)
89.68
87.94
92.05
94.25
94.34
91.65
OpenAl Models
3pt-3.5-turbo (Crosslingual)
77.27
73.64
80.05
81.16
74.99
85.65
78.79
gpt-3.5-turbe (TT)
74.20
70.09
76.67
72.66
73.68
82.99
75.05
text-davinci-003 (Crosslingual)
79
74.55
81.11
81.63
79.13
93.55
81.50
text-davinci-003 (TT)
79.06
72.93
78.93
75.18
80.48
93.22
79.97
Table 21: Comparing performance of different models on all languages in Jigsaw. Metric: Accuracy.


Reviews:
[{"review_id": "KNAh7rXcnt", "reviewer": "Reviewer_1g3u", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for carefully going through our work and providing their valuable feedback. \n\nRegarding the statistical analysis of the trends that we observe, we would like to kindly clarify that we do not merely just eye-ball the results, but do provide a concrete analysis, especially in Section 3.3 of the paper. We measure the Pearson correlation coefficients between the observed performance (for different tasks and languages) and factors such as tokenizer fertility and pre-training size of a language. While in the main paper, we only show the results for tokenizer fertility for the tasks where the correlation is statistically significant, we do report the full results with p-values in Table 21 of the Appendix. We use phrases like *“we can partially explain this discrepancy”* or *”this measurement might not imply causation”*, to underscore the nuanced nature of the problem and to acknowledge the limitations inherent in the techniques we employ. \n\nWe agree that an in-depth study of the factors that influence multilingual performance in LLMs is extremely important. However, given the breadth of topics covered in our paper, delving deeply into this specific aspect falls beyond the scope of our current work. Once again, we sincerely thank the reviewer for their insightful comments and will incorporate their feedback regarding the presentation of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "To the best of my knowledge, this is the first paper that aims at systematically benchmark the performance of LLMs on such a large variety of tasks and typologically diverse pool of languages.\nAuthors also conclude by investigating the challenges of testing LLMs and explore test data contamination, showing that there is a strong suspicion of contamination of test data for OpenAI model (indicating that their relatively good performances might only be due to having been trained on the data).", "reasons_to_reject": "Authors could strengthen their conclusions by using additional appropriate statistical analyses to more systematically analyse the causes that explain discrepancies in the result.  Another factor of analysis (which is currently left out, but which also falls out of the scope of the paper) would be to study the the influence of the writing system.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces MEGA (Multilingual Evaluation of Generative AI) which aims at evaluating the performances of generative LLMs in a wide variety of tasks (NLI, QA, seq. labelling, summarisation, etc.) in a wide variety of languages (70 topologically diverse languages). A wide range of models is tested, ranging from closed sourced/data models (Chat-GPT, GPT3.5, GPT4) to open models (BLOOM-like model) in a prompt-based setting, to fine-tuned-based models (mBERT, mT5, etc.).\n\nModels are evaluated using monolingual prompting (same examples as the test language), 0-shot X-lingual prompting (example in a pivot language different from the test language), and translate-text (where test sentences are translated to English).\n\nOverall, this very dense paper presents some nice contributions. Authors show that translating sentences from a low-resourced (LR) language into English works best (rather than leaving it in the original language). Open-AI models generally fall behind fine-tuned based models with the exception of GPT-4 which is on par with the others. Authors show performance in negatively correlated to tokenizer fertility (the latter being linked to the writing system) and to the amount of non-English data in the training data fed to the LLMs.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be better to keep the bar-order in histograms in the same order, as they would be easier to compare."}, "extra_text": {}}, {"review_id": "bcCCqhMLZt", "reviewer": "Reviewer_eRUE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments. Below we try to answer their questions and concerns.\n\n**How many runs is the average score?**: We assume that here the reviewer is talking about average scores in Tables 1 and 2, and Figure 2. Here, we have averaged the test data performance across languages available in a given dataset. We consider only a single run for evaluation, as the scale of our experiments i.e. the number of datasets and languages as well as the size of the models that we deal with, makes it computationally prohibitive to perform multiple runs for each evaluation. To ensure reproducibility we use a temperature value of 0 and we share the prompts that we use for each task and model in Section A.4 of the Appendix. We agree that some of these details are missing in the present manuscript and we will provide the same in the final version.\n\n**What parameters were used for fine-tuning the baseline models on the benchmark data?**: In all the fine-tuning baselines, we fine-tune all the parameters of the models, following other benchmarking studies like XTREME [1] that cover these models. Regarding hyperparameters used for fine-tuning, we use the hyperparameters reported in previous work [1,2]. More commonly, we used a learning rate of 3e-5, an effective batch size of 32, and 3 epochs, with some modifications depending on different models and datasets. We will list down all the hyperparameters used for fine-tuning in the final version.\n\n**Does the mathematical expression in the paper add in more to the explainability? Perhaps the figure 1c is enough?**: We firmly believe that the mathematical expression is important in understanding the inference strategy that we use for evaluating the models through prompting, especially for readers who are not well versed with the literature on prompting methods. We are able to explain different components of the prompt through this notation, which also help motivate different multilingual prompting strategies (Section 2.3.1) that we use in our work. However, we agree that in the current form this particular section is a bit long and we will make the writing more precise in the final version to make room for other experimental details.\n\n**Language Coverage**: We are not sure if we follow the phrase *“number of languages covered by the paper models”*. We provide the number of languages covered in each dataset that we evaluate in Figure 1a, where the number in parenthesis next to the name of the dataset contains the number of languages in that dataset. Further, in Figure 1b, we provide the language family distribution of the languages covered in different datasets included in MEGA. We discuss this in Lines 162-167 as well. **These are the languages that we evaluate in our work**. Due to the paucity of space we couldn’t provide the names of each language included in all of the 16 datasets in the main paper, but this information is available in Figures 7-13 or Tables 8-20 of the Appendix.\n\n**Connection between investigation questions and conclusion**: We believe that all of the three research questions that we motivate in Section 1 (Lines 89-96) are explored and answered in the paper in sufficient detail. \n- Q1. *How well do LLMs fare on multilingual benchmarks compared to fine-tuned SOTA models?*: We cover this in Section 3.2, where we answer this question by showing that LLMs lag behind fine-tuning baselines for most of the datasets and languages (373-380)\n- Q2. *What languages do these models perform well in, and can we explain the trends in performance for these models across languages?*: We discuss this in Section 3.3 of the paper, where we show that languages with little pre-training data or poor tokenizer qualities are the ones on which LLMs struggle the most to perform well. Further, in Section 3.1, we compare which languages perform well under different prompting strategies, where we show low-resource non-latin script languages benefit the most from translate-test prompting (Lines 334-338) and extremely low-resource languages suffer with zero-shot cross lingual prompting (Lines 359-366).\n- Q3. *What prompting strategies should be used for using LLMs for non-English languages?*: We cover this in Section 3.1, where we show that for low-resource and non-latin script languages translate-test works the best.\n\n We agree that in the current manuscript, these questions are not explicitly answered in the Conclusion Section and with an additional page for the final version, we will restate these results in the Conclusion.\n\n[1] XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. Hu et al. 2020.\n[2] mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Xue et al. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "> For the languages and models provided, the paper provides good discussion and insights into the performance of these models\n> Experimental prompts strategy is explained well, despite the added explanatory complexity/redundancies, and they also investigate the models for test data contamination as part of their model analysis", "reasons_to_reject": "> The experimental section of the paper writing can be improved as it has bad or complicated writing at some parts \"like ours\", \"overally\", incomplete or jump sentences \"we have the ability to follow instructions\".\n> The experimental section has missing information, and may not lead to easily reproducible results: How many runs is the average score? What parameters were used for fine-tuning the baseline models on the benchmark data? Does the mathematical expression in the paper add in more to the explainability? Perhaps the figure 1c is enough? Please consider these questions\n> The number of languages covered by the paper models and the number of languages evaluated/described in the experimental section are two different things. The experimental section is lacking in language coverage to say this is a multilingual model benchmark paper. With more language variety considered, this can be addressed.\n> The connection between the investigation questions and the paper conclusion is not very clear\n\nFor what the paper wants to represent, I believe more work needs to be done and the paper flow could be improved, by removing unnecessary complex explanation of simple concepts for instance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents benchmarking results for multilingual generative models and investigates three questions: \n1) how well do LLMs fare on multilingual benchmarks compared to fine-tuned SOTA models?\n2) What languages do these models perform well in, and can we explain the trends in performance for these models across languages?\n3) What prompting strategies should be used for using LLMs for non-English languages?", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "(1, 30) Engish\n(3, 214) We have the ability to follow instructions ???\n(7, 472) Overally"}, "extra_text": {}}, {"review_id": "teoh3eLUwk", "reviewer": "Reviewer_svC2", "summary": "", "questions": "Q1: How the language-specific LMs are compared with other models?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments and helpful feedback. Below we try to answer their questions and concerns.\n\n**Comparison with Language-Specific Pre-trained models**: The reason we focus on multilingual pre-trained LMs only and not language-specific LMs is due to the zero-shot cross lingual properties of the multilingual models i.e. fine-tuning only with English data and then evaluating zero-shot on other languages. This is important because for most of the datasets that we study in our work, there are no language-specific training datasets that can be used for fine-tuning. Hence, in most cases these baselines are fine-tuned on English data only (Lines 204-207) and it becomes infeasible to use language specific pretrained LMs. Further, there is also evidence in prior work showing that large enough pre-trained multilingual models can outperform monolingual models pre-trained on the languages to be evaluated for [1]. \n\n[1] Are the Multilingual Models Better? Improving Czech Sentiment with Transformers (Přibáň & Steinberger, RANLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experiments are conducted on a wide range of tasks and languages to lead to the conclusion that the LLMs still underperform SOTA fine-tuned models. \n- The analysis of different prompting strategies and their effects in the multilingual setting can offer solid baselines for other works.\n- Authors carefully tune prompts. ", "reasons_to_reject": "- Authors only compare LLMs with fine-tuned multilingual pre-trained LM, it leaves open questions about other language-specific pre-trained models and how they might perform. The additional comparative analysis could be beneficial.", "questions_to_authors": "Q1: How the language-specific LMs are compared with other models?", "paper_topic_and_main_contributions": "The paper evaluates the performance of large language models (LLMs) on various multilingual tasks and languages. The paper uses 16 tasks and 70 languages to compare LLMs with fine-tuned models. The paper also explores different ways of using prompts to elicit responses from LLMs, i.e., monolingual prompting, zero-shot cross-lingual prompting, and translate-test prompting. Through experiments, the authors find that fine-tuned models outperform large language models, especially, in low-resource settings, though techniques like translate-test prompting help bridge the gap. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c91IQpSA9n", "reviewer": "Reviewer_ppmZ", "summary": "", "questions": "A: There is a fourth research question that you did not investigate that is : how much English is used internally really? You acknowledge it (lines 104-108) but do not handle it. Why is that, and why don’t you think that this could undermine your entire enterprise?\nCan you propose tasks that would discover translations from English?\n\nB: Figure 2 is not clear, what is the y-axis?\n\nC: I find your results very interesting, especially those on memorisation, could you extend the discussion? For example, what does it mean given that you demonstrate that data is contaminated? How serious is the problem and how does it relate to the multilingual aspect? What should be done to avoid that? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments and helpful feedback. Below we try to answer their questions and concerns.\n\n**How much English is used internally really?**: We are not sure if we understood the reviewer’s question here, but let us try to clarify some potential mis-understandings about the lines 104-108. In these lines, we explain that for low-resource languages, translate-test strategy for prompting i.e. first translating an example in non-English language to English (using some MT tool like google-translate or bing) and then feeding the translated example to the LLM for prediction gives the best performance. These observations are discussed in detail in Section 3.1 of the manuscript. **We didn’t mean to say that LLMs do a poor job at translation, but instead that for most cases, the best performance on non-English languages for different tasks is obtained by first translating them to English.** We agree our wording in the Lines 104-108 can be a source of confusion for the readers and we will rephrase them to convey our point properly. To the best of our understanding of the reviewer’s question, these lines do not undermine our premise and in-fact convey one of the core findings from our work. We hope this answers the reviewer's question and if we have missed something, we will be happy to discuss this further during the discussion period.\n\n**Size of Figure 1**: We agree that in the current manuscript the size of Figure 1 is small. We will ensure that with an additional page for the final version, the figure is readable at the default scale.\n\n**Y-axis in Figure 2**: We apologize for the lack of clarity here. We label the y-axis as “Score” which denotes the values for the evaluation metrics specific to each task. For eg. for XNLI the “Score” is classification accuracy and for TyDiQA-GoldP it is the F1-Score. These metrics are listed in the second row of Table 1. But we agree that since Figure 2 appears before Table 1, we should have listed them earlier and we will correct this in the final version positively. Thank you for bringing this to our attention. \n\n**Findings on Memorization**: We thank the reviewer for asking this question. Given the lack of transparency about the training details of the LLMs that we evaluate, finding concrete evidence of memorization or data contamination becomes very difficult. What we have in our work are some indicators like the model’s understanding of the task structure and accessibility and release dates of the test datasets, which convey the possibility (qualitatively) of the test data appearing in the training datasets of these LLMs. We believe that the contamination of test datasets is a serious problem for works centered around LLM evaluation (including ours), as they might lead to an overestimation of the capabilities of these models. However, we would like to highlight that despite the possibility of contamination, LLMs still vastly underperform on (especially low-resource) non-English languages . The findings on contamination would have been more troubling if our results indicated superior performance on these languages, which is not the case. In the context of our work, these observations (about data contamination) indicate that the disparity in performance between English and non-English languages (Lines 97-102), might be even greater than what we observe in our work. About the practices that can be adopted to avoid the possibility of contamination, we believe that when we construct new datasets, we can avoid the test datasets to be directly available on the web and instead could be available via some manual download policy or other licenses. We once again thank the reviewer for their questions regarding this topic and we will include this discussion in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic: The problem of honestly evaluating LLMs and separating the real advances from the hype is a central one. Multilingual evaluation should be a must  and foremost evaluation standard.\n\nThe thoroughness of the analyses.\n\nThe very interesting results, especially about data contamination.\n\n", "reasons_to_reject": "There is a fourth question that is : how much English is used internally really? so tasks must be used that uncover translations from English. The authors themselves acknowledge this problem (lines 104-108), which to me seems pretty undermining of their whole enterprise, but do not address it in practice. \n\nFigure 1 is far far too small, even with 200% enlargement it is still too small to read, which means the paper is more than 8 pages long. So this should be corrected.\n\n\n", "questions_to_authors": "A: There is a fourth research question that you did not investigate that is : how much English is used internally really? You acknowledge it (lines 104-108) but do not handle it. Why is that, and why don’t you think that this could undermine your entire enterprise?\nCan you propose tasks that would discover translations from English?\n\nB: Figure 2 is not clear, what is the y-axis?\n\nC: I find your results very interesting, especially those on memorisation, could you extend the discussion? For example, what does it mean given that you demonstrate that data is contaminated? How serious is the problem and how does it relate to the multilingual aspect? What should be done to avoid that? \n", "paper_topic_and_main_contributions": "The paper argues that Generative LLMs should be evaluated multilingually and provides a comprehensive benchmarking. \n\nThe precise research questions are: (1), how well do LLMs fare on multilingual benchmarks compared to fine-tuned  SOTA models? (2), what languages do these models perform well in, and can we explain the trends in performance for these models across languages? (3), what prompting strategies should be used for  using LLMs for non-English languages?\n\n\nI think this paper provides a convincing contribution (in terms of models tested and language diversity of their 70 languages)  to a core problem that could have real impact.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 335 upto -> up to \n\nLine 401 retrive -> retrieve\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents the first large scale evaluation framework for multilingual NLP tasks. Reviewers see merit in the proposal and mostly raise minor issues. Multilingual evaluation is a highly important topic in the field and the paper presents a sound account of the issue.
Decision: Accept-Main