XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
Authors: Sebastian Ruder
Jonathan H. Clark
Alexander Gutkin
Mihir Kale
Min Ma
Massimo Nicosia
Shruti Rijhwani
Parker Riley
Jean Michel Amath Sarr
Xinyi Wang
John Frederick Wieting
Nitish Gupta
Anna Katanova
Christo Kirov
Dana L Dickinson
Brian Roark
Bidisha Samanta
Connie Tao
David Ifeoluwa Adelani
Vera Axelrod
Isaac Rayburn Caswell
Colin Cherry
Dan Garrette
Reeve Ingle
Melvin Johnson
Dmitry Panteleev
Partha Talukdar
Venue: EMNLP 2023 Findings (2023)
Keywords: nlp
low-resource
under-represented
scarce data
benchmark
TL;DR: XTREME-UP is a new benchmark focused on the scarce-data scenario for user-centric tasks for under-represented languages.

Abstract: Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs) --- languages for which NLP research is particularly far behind in meeting user needs --- it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks --- tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario tends to be most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text only, multi-modal (vision, audio, and text), supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark. We release all code and scripts to train and evaluate models.

XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
Sebastian Ruder' Jonathan H. Clark'
Alexander Gutkin
Mihir Kale Min Ma Massimo Nicosia Shruti Rijhwani
Parker Riley
Jean-Michel A. Sarr Xinyi Wang John Wieting
Nitish Gupta
Anna Katanova Christo Kirov
Dana L. Dickinson
Brian Roark
Bidisha Samanta Connie Tao
David I. Adelani
Vera Axelrod Isaac Caswell Colin Cherry Dan Garrette
Reeve Ingle
Melvin Johnson Dmitry Panteleev Partha Talukdar
Google *University College London
Abstract
Input
Information ac0888
Output
Spawch Recoag lilion
GA
Data scarcity is crucial issue for the devel- opment of highly multilingual NLP systems. Yet for many under-represented languages (anga_-as for which NLP research is particularly far behind in meeting user needs- is feasible to annotate small amounts of data. Motivated by this, propose XTREME-UP UP, benchmark defined by: its focus on the scarce- data scenario rather than zero-shot; its focus on user-centric tion by speakers of high-resource languages; and its focus under-represented languages where this scarce-data scenario is most realis- tic. XTREME-UD UP evaluates the capabilities of language models across 88 under-represented languages over key user-centric technolo- gies including ASR, OCR, MT. and informa- tion access tasks that are of general utility. We create new datasets for OCR, autocom- plete, question answering, semantic parsing, and transliteration, and build on and refine ex- isting datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text-only, multi- modal (vision, audio, and text), supervised pa- rameter tuning, and in-context learning.' We evaluate commonly used models on the bench- mark.? 2
Dooumentt OCR
Retiieval for OA
Machee Trndarior
Autocomplete
NER
Transitoration
Semantic Farsing
The tasks in XTREME-UF UP and their role in language technology. Left: enabling access to language technology; middle: facilitating information access as part of larger systems (question answering, information extraction, virtual assistants); right: making informa- tion accessible in the speaker's language.
Figure 1:
with broad adop-
datasets are often informed by established research directions the NLP community (de Marneffe et al., 2021). While linguistic tasks such as syntac- tic parsing have become less practically relevant (Glavaš and Vulić, 2021), other impactful capabili- ties such as question answering virtual assistants (Asai et al., 2021), often depend on ancillary tech- nologies such as language ID, data filtering, auto- matic speech recognition (ASR), or optical char- acter recognition (OCR) that are typically under- performing or unavailable for ULs (Caswell et al., 2020; Bapna et al., 2022; Kreutzer et al., 2022; Ri- jhwani et al., 2021; Khare et al., 2021). As result, speakers of ULs are unable to reap the benefits, even if the development of models is successful.
In order to make progress on NLP for ULs, we should thus focus on evaluating models on tasks that are most likely to benefit speakers of those languages." 3 To this end, we propose XTREME- UP and User-Centric with Paucal' Data), benchmark focusing on evaluation of mul- tilingual models on user-centric tasks in scarce-
1 Introduction
The development of natural language processing (NLP) technology that serves most of world's lan- guages is hindered by the stark lack of data for most languages (Joshi al., 2020). While there is increasing interest in developing datasets and mod- els for under-representeed languages (ULs), existing
2 SSeaakers of ULs have needs ranging from standard NLP technology language documentation and revitalization (Bird, 2022). Our focus is on standardized, institutional, and contact languages including dialects and non-standard lan- guage varieties spoken by large speaker populations.
"Equal contribution. We list detailed contributions in $AA. 'Our results in $4 indicate that few-shot in-context learning is less effective than fine-tuning on 100s of examples for ULs. We advocate for comparing such approaches directly as the community explores XTREME-US Phttps: / thub. com/
WWe horrow the term paucal-meaning few from linguis- tics, to emphasize the scarce-data nature of XTREME- UPP

---

data setting.
et al., 2023) and BUFFET (Asai al., 2023) eval- uate in-context learning on existing multilingual tasks. contrast, XTREME-UD UP focuses on under- represented languages, user-centric tasks, a more realistic scarce-data setting, and introduces new tasks and datasets.
We focus on tasks that technology users en- counter in their daily lives: i) information access tasks reflecting generally useful NLP capabilities; and ii) input/output tasks that enable other technolo- gies. We show the corresponding tasks and their role in interactions with language technology in Figure Moving away from the cross-lingual zero- shot setting (Hu et al., 2020; Ruder et al., 2021), we standardize multilingual in-language fine-tuning based on the amount data that can realistically be annotated within 8h for a language. Our re- sults highlight the limitations of current models on ULs, demonstrate the potential of language models (LMs) to improve user-centric applications, and show the benefit of byte-based approaches.
Multilingual evaluation The choice of the ex- perimental setting and aggregation metric are im- portant considerations in multilingual evaluation. Prior work focused on zero-shot cross-lingual transfer (Hu al., 2020), which- being compelling from a scientific perspective (Artetxe et al., 2020)-is is less practically useful. While in-language fine-tuning has been explored before (Lauscher et al., 2020; Hedderich et al., 2020), XTREME- UP is the first to standardize the setting acToss tasks based on realistic annotation costs. Dif- ferent frameworks aggregate performance in dif- ferent ways across languages. Blasi al. (2022) assess the utility task by weighting model per- formance based on the size of the speaker popu- lation while Khanuja et al. (2023) introduce the Gini coefficient to quantify performance disparity across languages. XTREME- UP opts for simple average over ULs. emphasizing intuitiveness and accessibility of the results.
In this work, we contribute the first massively- multilingual few-example benchmark including: a) newly created data for QA, OCR, autocomplete, se- mantic parsing, and sentence-level transliteration; b) new task setups for named entity recognition (NER) enabling evaluation on nathral-rather than tokenized- text; and for QA and retrieval provid- ing more interesting setting than the gold passage (GoldP) setup while offering lower barrier-to- entry than the full TyDi QA (Clark et al., 2020) XOR (Asai et al., 2021) tasks; e) carefully- designed experimental setups, standardizing in- language fine-tuning and in-context learning and focusing on the information access scenario for ULs for ASR and MT; d) baseline results with commonly used subword and byte-based models.
3 STREME-UP
3.1 Design Principles
XTREME-UP is motivated by the following design principles:
Under-representeed languages Following Joshi et al. (2020) we select languages in categories 1-3 (c.g., Amharic, Estonian, Kinyarwanda) under-represented, leaving categories 4-5 as high- resource languages (e.g., English, German, Hindi), We focus on tasks with existing data in ULs and tasks where we can efficiently collect data at scale (see Appendix B for an overview of ULs in XTREME-UP).
2 Related Work
Multilingual benchmarks Some studies em- ploy highly multilingual individual datasets for the evaluation of multilingual models, including Uni- versal Dependencies (de Marneffe et al., 2021) or XL-Sum (Hasan al., 2021). At the same time, there is increasing work on datasets in ULs for a variety of applications (Niyongabo et al., 2020; Winata et al., 2023; Muhammad et al., 2023). Due to their rapidly growing capabilities, NLP models are increasingly evaluated on a suite of datasets. Existing multi-task multilingual benchmarks such XTREME (Hu et al., 2020), XGLUE (Liang al., 2020), and XTREME-R (Ruder et al., 2021) cover 20-50 mainly high-resource languages and prioritize tasks with available data, regardless of their utility to speakers. Recently, MEGA (Kabir
User-centric tasks We focus on widely adopted user-facing tasks beneňting speakers of high- resouree languages. We further break these down into two major groups: 1) input/output tasks; and 2) information access tasks (see Figure 1).
Scarce data We focus on a realistic scenario where a small amount of data is available each UL. Mirroring reality, we do not restrict the amount

---

Train sum over HLL
Train arg. per UL
Validation sum across ULs
Test sum across ULs
oo ULs
Metric
Annolation Cost minutevexuppl
Task
(t HL.)
Input Outpur
Speoch Recognition
274,514
2,647
26,556
60,118
77(23)
CER
0.2"
Document OCR
60
g*
447
452
700)
CER
44.5
Aulocomplete
44,554
1,850
13,080
14,747
2020
Acc
0.3
Transliteration
7,360
120
28,000
28,000
12(1)
CER
2.7z
Machine Translation
19,877
120
34,860
70,000
70k23)
ChrF
4.0
in-lang.
59,550
426
3,656
3,688
6(3
Span F1
3.0
QA
Information
cross-lang.
22,544
361
8,199
12,720
21(6)
Span F1
3.0
in-lang.
29.683
320
1,830
1,846
6(3
MRR
3.0
Access
Retrieval for QA
cross- Langs
13,270
265
6,183
10,704
21(6)
MRR
3.0
NER
28,023
1,401
14,250
18.192
20x0)
0.3
Semantic Parsing
0,373
273
2,533
39,253
9(11)
2.0
Table 1: The tasks in XTREME-UP For each task, we show both the sum of training examples across all languages- give some insight into training scale and the average number training examples for each under- represented language-
highlight the challenge of the scarce-data learning scenario. XTREME- UP does not limit supervised training data in high-resource languages (HL.)) while each under-represented language (UL) has a max- imum of 8 hours annotation effort in its training split; see last column for estimated annotation effort. We also show the sum of validation and test examples across ULs as XTREME-UP UP evaluates only on ULs.
Average time for read speech. Based on mean lyping speed (Dhakal et al., 2018) and average sentence length in Universal Dependencies (de Marneffe et al., 2021). "An annotated example can be used for both task directions. FFrr document OCR, each example is whole page, for example, 9 pages of training images per language corresponds to approximately 290 lines of output text per language.
of training data available in high-resource lan- guages, but rather provide only many labeled training examples as can be annotated in realistic amount of time for ULs (see Section 3.2).
long it takes to annotate single example for trained annotator:" We base our estimates on prior work and our own annotation efforts. 6 We show the data annotation time estimates Table 1. For tasks with larger training datasets, we sub-sample the available data accordingly. Table I shows the sub-sampled data sizes. We show the input and output format of each task in Table 2. We provide example instance of each task in Appendix C.
Efficiency We focus on massively multilingual evaluation settings that still be run efficiently with modest amount of compute.
Text-centric, yet multi-modal We focus on tasks that can be tackled using textual data alone and provide baseline systems that do so. We frame multi-modal tasks (OCR and ASR) that natively multi-modal models can be evaluated fairly along- side text-only models. We accomplish this by re- leasing original audio, image, and text model inputs while also providing baseline system output that can be fed to second-stage text-only systems. We hope to see fully multi-modal models take up this challenge over the coming years.
3.3 Input/ / Output Tasks
Automatic speech recognition (ASR; D.1) The goal of ASR is to transcribe speech into human- readable text. It thus serves as fundamental step for enabling natural language understanding appli- cations on speech input. In many contexts, users may strongly prefer to speak rather than type and so high-quality ASR is an enabling factor for such interactions. We employ the FLEURS dataset (Con- neau et al., 2023) consisting of recordings in 102 languages for sentences from FLORES-101 (Goyal et al., 2022), which were translated from English Wikipedia to 101 languages. We evaluate on 77 under-represented languages.
We provide overview of the tasks in XTREME- UP in Table 1. We discuss motivation and high- level information in the next section and provide more details for each task in Appendix D.
3.2 How much data?
Optical character recognition (OCR; D.2) OCR, the process of converting text from images
To ensure a realistic amount of training data, we limit the training data in each task per language to the number of examples that can be annotated in 8 hours. We believe this reflects the real difficulty of annotating training and evaluation data for a very large number of languages. In this way, we design for the task first. For each task, we estimate how
'For simplicity, we estimate the annotation time for label- ing only, ignoring factors such as training annotators, data processing, data validation, interface design, etc. We note that unlabeled data may not be available for certain ULs (Nekoto al.. 2020) and its creation may require tools such as key- boards, which may not available in all languages.
'oor autocomplete, we calculate average writing time.

---

Task
Inpat
Output
that users have come to expect and rely on for in- put in high-resource languages. The standard next word prediction task (Sundermeyer et al., 2012) does not accurately reflect this practical setting as it relies on predicting entire units (words, sub- words, or characters); similarly, perplexity-basee evaluation makes comparisons across segmenta- tions and languages difficult (Mielke, 2019) and ignores threshold effects associated with top-k pre- dictions in a user interface (Tam and Wells, 2009).
Speech Recognition
Spesch audio
Transeription
Document OCR
Page image
Transeription
Autocomplete
Sentence prefix
Completed word
Transliteration
Source script
Target script transliteration
sentence
Source langunge sentence
Target language translation
Machine Translation
In-language Retrieval for QA
Source language question
Source language answer passage
Source langunge question, title,
Source language
In-language QA
answer span (or
and passage
"No answer") )
To fill this gap, we introduce a new autocomplete task that unifies character, subword, and token- "word" as the predictive unit. Models are required to complete the next word based on left context of N words and an optional character n-gram prefix. We use ac- curacy @3 for evaluation to reflect the requirement of displaying a limited number of candidates to the user. We process high-quality natural language data from Universal Dependeneies (de Marneffe et al., 2021), which we deduplicate against mC4 (Xue et al., 2021), the most common multilingual pre- training corpus in order to test models' predictive rather than memorization capabilities.
Source language question
English answer passage
Cross-lnguuge Retrieval for QA
level LM settings by focusing
Source language
English answer span (or "No answer")
Cross-language QA
question, English
title and passage
Eniiy mentions
NER
Sentence
Structured
Semantic Parsing
Assistant query
with intent, slats
Table 2: The input and output format of each task in XTREME-UP. Tasks are generally text-in, text-out with a few exceptions. See Appendix C for task examples.
into machine-readable formats, is used in a wide range of applications, from extracting data only available in paper books (Rijhwani et al., 2020) and imaging legal documents (Singh et al., 2012). to improving accessibility for people with low vi- sion (Mowar al., 2022). is especially important for reder-represented languages, where both train- ing data and content that users may wish to access may not be available as digital text on the web.
Transliteration (D.4) Transliteration is the con- version of text between writing systems (Wellisch, 1978). Unlike translation, it does not change con- tent but only script. Transliteration is important because it allows users to type in their preferred script (e.g., Latin script) even if it is different than their preferred display script (e.g. Devanagari) and is used internally by many machine translation sys- tems to rewrite names from different scripts.
We create a dataset that aims to fill the gaps in previous work in OCR for ULs (see Appendix D.2) by focusing on larger-scale, typologically diverse, and user-centric data. Our dataset con- tains transcriptions for books in seven languages: Amharic (am), Bengali (bn), Kannada (kn), Myan- mar (Burmese; my), Sanskrit (sa), Sinhala (si), and Swahili (sw). The books domain is the pri- mary use-case for a large number of downstream users. but is one of the most challenging for OCR models (Rigaud et al., 2019). The dataset consists of transcriptions of entire pages and thus enables leveraging the full context understanding capabili- ties of large language models.
We extend the Dakshina dataset (Roark et al., 2020), which provides romanizations of Wikipedia sentences written in the native scripts of 12 South Asian languages, with: a) romanizations of native script Wikipedia for one new language (Ambaric); and b) transliteration to a third script (Shahmukhi) for one already covered language (Punjabi). The re- sulting task covers 13 languages for which translit- eration occurs from the Latin script the native script, and vice versa, and between Shahmukhi, Gurmukhi, and Latin for Punjabi.
Machine translation (MT; App. D.S) MT is important technology for users of ULs wishing read text written in a different language. However, most current approaches require large amounts of parallel training data to achieve good perfor- mance, which are often not available for ULs (Had-
Autocomplete (D.3) Autocomplete (or predic- tive text), i.e., predicting the rest of word user is typing, a useful technology that speeds up human-computer interaction (Anson et al., 2006). As such, autocomplete has become a technology

---

dow et al., 2022). We focus on the information dissemination scenario where content from high- resource languages (including from tasks such as cross-lingual QA) is translated to enable informa- tion access by common users; as such, XTREME- UP includes translations from English into 93 lan- guages, covering wide range of high-resource and UL languages. Only 39 ULs are used for evalua- tion; the high-resource languages are included to allow for transfer learning." The dataset is adapted from FLORES- 101 (Goyal et al., 2022), repurpos- ing half of the dataset's original development set as a training set. See $6 for a detailed discussion of how we distinguish freely-available unsupervised data versus aunoose-d supervised data in XTREMB-UP.
end, we generalize the gold passage (Clark et al., 2020) setting, augmenting it with negative exam- ples. These negatives are obtained from (a) pas- sages within the same article as a passage contain- ing the answer and (b) question-answer pairs from the full TyDi QA dataset where no answer was found in the candidate Wikipedia article. The data is split into training, validation, and test splits in such way to avoid deduplication and overlap of splits, even across our various QA tasks.
Retrieval for QA (D.6) Within the information- seeking QA scenario, the above core QA task as- sumes answer candidate passages practice, a passage retrieval system for question- answering allows for the extraction of relevant text from a vast text corpus. The retrieved passages can then be used by a question-answerng extract or generate an answer to the user's question. In XTREME-UP, we separate retrieval into two distinct tasks, in-language retrieval and cross- language retrieval. For in-language retrieval, both the questions and passages are in the same lan- guage. The preparation of negatives, deduplication, and splits are identical to the QA task above. For validation and test, we create an index of 271k in- language passages (447k English passages for the cross-language task) making for a small enough in- dex for efficient experimentation, while containing distractors that make for a challenging task, since these distractors are drawn from the same articles containing the target passages.
input. In
3.4 Information Access Tasks
system to
Question Answering (D.6) Question answering enables responding to natural language questions with answers found in text. We focus on the information-seeking scenario (Kwiatkowski et al., 2019) where questions are asked without knowing the answer. Information-sseeing question- reeswr pairs tend to exhibit less lexical and morphosyntac- tic overlap between the question and answer since they are written separately.
We include two variants of the task: In in- language QA, both question and passage in the same language. We obtain original questions and passages from TyDi QA (Clark et al., 2020). For cross-language QA, the question is in the user's native language while passage and answer are in a language with large amount of answer con- tent available (English). We use examples from TyDi XOR (Asai et al., 2021) in 7 languages. We additionally collect new data in 23 new Indic languages for cross-lingual QA by professionally translating questions and answers from existing Indic languages in XOR QA. This methodology mitigates the issue of translating Western-centric English data to locales with different topical inter- ests. Cross-lingual QA is especially important for ULs since they lack plentiful in-language answer content the web.
Named entity recognition (NER; D.7) NER is an important capability for information access sys- tems that users depend on with applications rang- ing from recognizing requests for entity lookups to performing information extraction to populate the knowledge graphs that handle those requests. NER is also a capability needed in spell-checking and localization systems (Li et al., 2020). 10 Iden- tifying entities in ULs poses challenges due to the use of different scripts, lack of capitaliza- tion, different numerical representations, etc. We build on MasakhaNER (Adelani et al., 2021) and MasakhaNER 2.0 (Adelani al., 2022), two large
In XTREME-UPP UP's QA task, a system is given a question, title, and passage and must provide the answer-if ony-or otherwise return that the question has "no answer" in the passage." To this
"This turns out non-trivial given different splits strategies across the various datasets and our decision to create train, validation, and test set even where only validation set were previously available for public download.
and
We emphasize the word capability here since we recog- nize that stand-alone NER systems may not be strictly neces- sary in the long run; however, the capability of recognizing and properly handling entities will remain.
"Our baseline results were trained only on the 39 UL pairs for efficiency.
"This follows SQuAD v2 (Rajpurkar et al., 2018).

---

NER datasets in African languages, which pro- vide data in the standard CoNLL tokenized for-
Eval
of
Vocab units
% of non-en
Model
setring
params
pre pre-train data
mat (Tjong Kim Sang and De Meulder, 2003). In order to enable evaluation a setting that is closer to the real world, we automatically map
mT5--aae
FT
580M
Subwords
94.3
ByT5-Base TSse
FT
580M
Bytes
94.3
Man-PaLLM
ICL
62B
Subwords
22.0
the annotated spans to the original raw text. The combined data with byte-level span annotations termed MasakhaNER-XXOO0 --oovrrs 20 languages. !I
Table 3: Additional information on baseline models including the setting in which we evaluate them (fine- tuning in-context learning), their size, their vocabu- lary, and the fraction of non-English pre-training data.
Semantic parsing (D.8) Semantic parsing is the task of mapping natural language utterance to logical form or structured interpretation that can be executed by system such as a virtual assistant. This task is especially timely as users will increas- ingly want to turn their interactions with assistants and chat-like dialog systems into actions on exter- nal systems, which require API calls; this capability is what the semantic parsing task evaluates.
scores of all tasks. For each task, we only aver- age performance over ULs (discussed in $3.1). For metrics such as character error rate (CER) where lower is better, we invert the scores before averag- ing scores across tasks. For mean reciprocal rank (MRR), which is in the 0.0-1.0 range, we renormal- ize to the 0-100 range before averaging. While this scalar provides a quick overall impression of a system's 's quality across a broad range of tasks, it is not a substitute for analyzing performance on individual tasks, languages, or types of examples.
We adapt the test split of MTOP12 (Li et al., 2021) with professional translators/annotatorr to 15 Amharic, Belarusian, Bengali, Brazil- ian Portuguese, Finnish, German, Hausa, Hun- garian, Japanese, Russian. Swahili, Tamil, Turk- ish, Yoruba, and Zulu. Together with the original MTOP languages, the new MTOP++ dataset covers total of 20 languages. Differently from MTOP, collect localized data (i.e., Westem-centric entities are replaced with more culturally relevant entities for the target language), following recent trends in multilingual benchmarking (Lin et al., 2021; Ding et al., 2022; Majewska et al., 2023).
languages:
4 Experiments
4.1 Experimental setting
Multilingual fine-tuning In contrast to prior benchmarks that focus on zero-shot cross-lingual transfer from English, XTREME-UP UP focuses on the more realistic scenario of fine-tuning on a small amount of data in the target language. To make this scenario sealable in massively multilingual setting, XTREMB-UP fine-tunes single model on the combined training data across the available lan- guages for each task. The data for cach language is sub-sa to emulate data sizes that can be real- istically annotated within a reasonable time frame (see $3.2).
We also extend MTOP to three widely spoken but under-represented Indic languages in code- switching setting: Hindi-Engliss Bengali-English and Fnglish. We automatically convert the test-split of MTOP to code-mixed utterances using PaLM (Chowdhery et al., 2022) and run human verification on such utterances.
3.5 Overall Evaluation
For each task, we evaluate model performance by computing a task-specific score. We employ character-level metrics such character error rate (CER) and character n-gram F-score (chrl:; Popovié, 2015) rather than their word-level coun- terparts as they enable more fine-grained evaluation and are better suited to morphologically rich lan- guages. We obtain a final score by averaging the
In-language in-context learning We also pro- vide a 5-shot in-context learning setting where model is provided with English instruction and exemplars the target language in order to eval- uate the progress on few-shot learning with large models for ULs. We provide the instruction for each task in Appendix E.! 13
11 We remove the Fon and Hausa subsets of MasakhaNER 2.0 due to quality issues in annotated data.
12 "Other datascts (see App. D.8) were not yet available at the start of the project and do not focus on ULs.

---

4.2 Baselines
Infrastructure Models were trained using seqio and t5x (Roberts et al.. 2022) on TPUs (Kumar et al., 2019; Pope et al., 2022).
We provide results handful of baseline systems that have already been developed by the research community. Given that our focus in this paper is on the dataset and task setup rather than system build- ing, we do not focus on offering novel modeling types nor do we exhaustively evaluate all possible models; rather we view these results estimat- ing a starting point from some well-know model- ing approaches and seeding contributions from the broader research community." 14
4.3 Results
We show the baseline results in Table 4.
Byte-based models outperform subword- on ULs. The byte-based ByT5 outperforms the subword-based mT55 across most tasks. Gains are particularly pronounced for tasks that require deal- ing with information the character level such as autocomplete and transliteration and for predict- ing information on the word level such as for NER and semantic parsing. These results demonstrate that as we train and evaluate our models on under- represented languages, standard modeling choices such as subword representations fall short.
Multilingual fine-tuning baselines For the main experimental setting of multilingual fine- tuning, we provide the following baselines: mT5- base (Xue al., 2021) and subword-base multi- lingual encoder-decoder model: By B--5-base (Xue al., 2022), byte-based multilingual encoder- decoder model.
In-context learning underperforms fine-tuning on limited data. The Flan-Pal.M model gener- ally performs worse than fine-tuned models, de- spite being much larger. Nevertheless, it achieves reasonable performance on machine translation, which is likely reflected in the pre- pre-training data. On other tasks. however, it fails to reliably apply its English-centric knowledge ULs. Despite fine- tuned models performing relatively well on NER, the in-context learning model unable to consis- tently generalize to the task a few-shot setting in under-represented languages. On semannic parsing, the model fails to generalize to the large number of domain-speciffe intents and slots using standard prompting in ULs.. 16 The autocomplete tasks in par- ticular demonstrate the lack of robust cross-lingual information in the English-centric PaLM model: it struggles to complete a sentence given a character prefix and fails to reliably convert between differ- ent scripts in the same language. XTREME-UP UP thus provides strong challenge to test the general- ization abilities of in-context learning methods to ULs.
In-context learning baseline For the in-context learning setting, we employ Flan-PaLM (Chung al., 2022), an instruction-tuned version of PaLM (Chowdhery et al., 2022). We provide additional information on the baseline systems in Table
To offer baseline systems that allow experimen- tation with text-only models, we use upstream mod- els to provide initial output for ASR and OCR, and present text-based baselines that use these in- puts. We expect these baselines to give way fully multi-modal models as research progresses. These initial ASR and OCR outputs should be seen as part of baseline system, nol part of the XTREME-US P benchmark iteself. For ASR, we augment the data with predictions of the state-offthhe-at Maestro-U (Chen et al., 2023) and then model improve the outputs (Bassil and Alwani, 2012). Similarly, for OCR, we use the off-the-sheif Google Vision OCR to get first-pass outputs, and train language models to improve them (Dong and Smith, 2018; 8 Rijhwani et al., 2020).
downstream text
The choice of prompt and exemplars can have signifi- cant impact performance (Zhao et al., 2021a.b). We provide single instruction and set of exemplars per task and language for replicability and leave the search for better instructions and exemplars future work.
There is a lot of headroom left to improve per- formance on ULS. Overall, across all tasks there is still considerable amount of headroom left. For ASR, OCR and transliteration, around 10% of characters are still incorrectly predicted. On auto- complete, models only make the correct prediction in about one fourth of all cases. For MT, on average
1 "XTreme--U UP offers public results tracker
tracking the community's 's progress on XTREME- UP. We con- ceptualize these results not as competition, but as offering insights about different models and their trade-off's each justi- fying and explaining how it should be compared to others and how it informs the research landscape. Submissions can made self-service git pull requests. 15 "https: / //w./dd google. wwwccoooiiiieeee watssiiooooo//orrr/e docs/ocr
We leave the exploration of multilingual adaptive prompt- ing and dynamic exemplar selection (Drozdov et al., 2023) methods to future work.

---

Input Output Tasks
Information Access Tasks
Semantic
ASR
OCR
Autocomplete
Transliteration
MT
QA
Retrieval MRR1
NER
Parsing
CER1
CER
CER
chrFt
FIf
F1
EM1
Multilingual fine-runing
mTS-Base
8.5
(11.1)
12.7
37.0 6
22.5
59.7 (749/446)
0.23 (0.11 1007)
74.0
21.8
ByT5-Base
8.2
(11.1)*
27.6
14.6
26.9
71.4 (82.36055)
0.29 (0.45 70.8)
84.0
37.5
In-context learning (5-shot)
Plan-PaLM-621B
23.2
0.0
77.4
32.1
22.9 (20.9/24.9)
12.9
0.1
Table 4: Overall results of baselines across all XTREME- UP v1.0 tasks for the test split. Scores on XTREME-UP UP average over evaluation scores of represseeeee languages. QA and retrieval performance the average of in-language and cross-language settings (indicated in brackets in-language cross-language). For OCR, do not apply any additional models (mT5 nor ByT5) on top of the baseline OCR system; we show these results in parentheses. We not attempt in-context learning (ICL) results for retrieval since ICL is typically only used for text-in, text-out cases. For OCR, use the Google OCR API. 'For autocomplete, while observe reasonable performance on English completions, we find the model typically does a very poor job outside English.
only about third of n-grams in the hypothesis also present in the reference, and vice versa. For QA and retrieval, there are large performance dif- ferences between in-language and cross-language settings and much headroom still left. On NER, models perform relatively well but are still far from perfect performance on the task. Finally, on se- mantic parsing models are only able to produce the correct output in around third of all cases.
6 Recommendations
Use of splits XTREME-UP UP offers a train, valida- tion, and test split for each task. We recommend using the training split for learning the parameters of your model or as exemplars for in-context learn- ing while iteratively checking your progress on the validation (i.e. development) split. The test split should not be used for itcrative evaluation of your models or other sorts of hill-climbing: instead, it should be reserved for reporting your results and comparing after you have finished development on your models. Experiments that follow this custom- ary scientific rigor should expect show better generalization and less overfitting the test split.
5 Analyses
Lowest-performing languages Models gener- ally perform poorly African languages. On transliteration, models perform relatively worst on the newly added Amharic language. On NER, which covers only African languages, performance is lowest for Amharic Amharic-likely due to its differ- ent script-and the extremely under-represented Ghomálá'. Similarly, translation models underper- form in Amharic and Yoruba. On ASR, the lowest- performing languages are Yoruba but models also struggle with other languages such as Gaelic, and many South Asian languages such as Lao, Khmer, and Burmese.
Use of additional pre-training data One po- tential confounder for results along different pre- trained models is the variation pre-training data; where this data overlaps with the targets (outputs) in X XTREME-UP validation and test splits, results can be artificially inflated, providing a sense that results are better than they are in reality-if the val- idation or test data leaked into the pre-training data via contamination during large-scale data scraping, then it's unlikely that the system would truly per- form as well for new unseen inputs. Therefore, we recommend that when researchers modify the pre-training data for a model, they explicitly re- port overlap (contamination) between the targets of the XTREME- Up validation/test splits and their pre-training corpus." 17
Task-specific observations ByT5 provides the best performance while the size of the model does not seem to impact performance much. Several aspects of the data lead to higher error rates in transliteration: the model struggles with input in the Perso-Arabic script and to produce output in Latin based on a different script. For autocomplete (sce Appendix D.3), our analyses indicate that mod- els perform better on text that uses the Latin script.
We recognize that this is a very large-scale undertaking. requiring fairly large amount of compute. As such, we suggest that it's may only he needed when making claims that compare systems (c.g. that the system with possibly-

---

Use of additional supervised data is entirely possible that the community will find creative ways to improve models based on supervised data not included with XTREME-UL UP. However, researchers should bear in mind how this might affect the com- parability of their results with other models. The following axes should be considered:
above all, researches should make a well-reasoned argument that their use of data (i) does not artifi- cially inflate evaluation scores and (ii) reflects a real-world scenario of finding and applying data.
7 Conclusion
1. Any additional data from high resource lan- guages is always allowed in the XTREME- UP setting.
We have presented XTREME-UP, multilingual benchmark distinguished by its being (i) scarce- data, (ii) user-centric, and (iii) focused on under- represented languages. The benchmark contains input modalities of text, images, and audio while still allowing experimentation with text-only mod- els. We hope this benchmark will be useful in accelerating research that is useful to speakers of under-represented languages and in highlighting both the progress and limitations of current models of language.
2. Supervised data (e.g. parallel data for MT) harvested from the web, religious, books, and other opportunistic sources will typically be out-of-domain and is therefore admissible; conversely, supervised data from ULs from highly similar tasks or domains should gen- erally be considered against the spirit of the XTREME- UP benchmark.
3. Monolingual data from UL is admissible with the caveat that one should measure overlap with targets, as discussed above.
Limitations
Avoid off-the-shelff MT systems Data augmen- tation via automatically translating high-resource supervised data to languages with less supervised data has proven very effective means of improv- ing system quality. However, it is not necessar- ily realistic use a pre-existing MT system (e.g. an API or an open-source model) since those sys- tems have typically been trained on a large amount of parallel data-or at least unknown data. This means that additional supervised data would then be leaking into the experimental setup, which is otherwise intended to reflect the reality that most under-representer languages have very little super- vised data. If data augmentation via translation is used, we encourage researchers to report the parallel data sources used and argue why this ex- perimental setup is realistic- or to clearly point out such usage in their experiments as an unavoidable confound and discuss the limitations this sets on what conclusions can be drawn about how results will extrapolate to the breadth of under--epresented languages.
The dataset presented in this work does not rep- resent all of the world's languages nor all of the under-represented languages. While we have made efforts include languages and dialects across a broad variety of geographic regions and language families, it was not feasible to locate or create data in the same set of languages across all tasks. Since this is a data-focused paper, we present modeling results on a few strong modern models; this is not exhaustive exploration of how all current mod- els may perform on this dataset. We look forward to exploring more under-represented languages as more data becomes available.
Acknowledgements
We thank Slav Petrov, Jason Riesa, Raphael Hoff- mann, Dipanjan Das, Clara Rivera, Chris Alberti, Machel Reid, and Timothy Dozat for helpful discus- sions and feedback. We are grateful to Noah Con- stant for a review of a draft of the paper. We also gratefully acknowledge the contributions of the re- searchers who built the datasets that have gone into -TREME-UP; we recommend that all component datasets be cited individually when using XTREME- UP in a paper such that dataset authors (many of whom are authors of this article) receive credit for their work and so that those original sources remain casily discoverable in the literature.
In all cases, researchers should rigorously re- port what additional data was used and how; each use case comes with its own considerations and,
contaminated pre-training data is equivalent, better, or almost as good us some other system). Note, this analysis only needs to be done once for each pre-training corpus (e.g.. once for mC4) and it is very likely that organizations with enough compute to pre-train a new model on a now corpus would also have sufficient compute to calculate overlap.

---

References
for more rigor in unsupervised cross-lingual leam- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7375-7388, Online. Association for Compu- tational Linguistics.
David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Roowei- ther Mabuya, Bonaventure F. P. Dossou, Bless- ing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye. Amelia Tay- lor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gi- tau, Edwin Munkoh-Buaheng, Victoire Memd- jokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe. Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuncke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Ahdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4488 4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question answer- ing. In Proceedings of the 2021 Conference of the North American Chapier of the Association for Com- pulcational Linguisties: Human Language Technolo- gies, pages 547-564, Online. Association for Com- putational Linguistics.
Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu. Terra Blevins, Hila Gonen, Machel Reid, Yu- lia Tsvetkov, Sebastian Ruder, and Hannaneh Ha- jishirzi. 2023. BUFFET: Benchmarking large lan- guage models for few-shot cross-lingual transfer. arXiv preprint 23X5v:2305.1455
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Fi- rat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey. Theresa Breiner. Vera Axelrod. Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang. Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2022. Building machine translation systems for the next thousand languages. arXiv preprint arXiv:2205.0399
David Ifeoluwa Adelani, Jade Abbott, Graham Neu- big, Daniel D' Dosuuaa Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen May- hew, Israel Abebe Azime, Shamsuddeen H. Muham- mad, Chris Chinenye Emezue. Joyce Nakatumba- Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Al- abi. Seid Muhie Yimam. Tajuddeen Rabiu Gwad- abe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Troro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu. Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Batcesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deb- orah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebrey- ohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore- vaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thiemn Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Sa- lomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
Youssef Bassil and Mohammad Alwani. 2012. Post- Editing Error Correction Algorithm For Speech Recognition using Bing Spelling Suggestion. (LJACSA) International Journal of Advanced Computer Science and Applications. 3(2):95-011.
Steven Bird. 2022. Local languages, third spaces, and other high-resouree scenarios. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 7817-7829, Dublin, Ireland. Association for Com- putational Linguistics.
Damian Blasi, Antonios Anastasopoulos, and Gra- ham Neubig. 2022. Systematic inequalities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages Ireland. Association for Computational Linguistics.
Dublin,
Isaac Caswell, Theresa Breiner. Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unexpected challenges on the path to a thousand- language web text corpus. In Proceedings of the 28th International Conference on Computa- tional Linguistics, pages 6588-6608, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Denis Anson, Penni Moist, Mary Przywara, Heather Wells. Heather Saylor, and Hantz Maxime. 2006. The effects of word completion and word prediction on typing rates using on-screen keyboards. Assistive Technology, 18(2):16-1154
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020. A call
Zhehuai Chen, Ankur Bapna, Andrew Rosenberg. Yu Zhang, Bhuvana Ramabhadran, Pedro Moreno,

---

and Nanxin Chen. 2023. Maestro-U: Leverag- ing joint speech-text representation learning for zero supervised speech ASR. In Proceedings of 2022 IEEE Spoken Language Technology Workshop (SLT), pages 68-75, Doha, Qatar. IEEE.
Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, and Antti Oulasvirta. 2018. Observations on typing from 136 million keystrokes. In Proceedings of the 2018 CHI Conference on Human Factors in Comput- ing Systems, pages 1-12, Montréal, Canada. Associ- ation for Computing Machinery (ACM).
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebustian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya. Sanjay Ghe- mawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fe- dus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thamumalayan Sankara- narayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira. Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsseem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling language modeling with Pathways. arXiv preprint 2rXiv:220..03311.
Bosheng Ding, Junjie Hu, Lidong Bing, Mahani Alju- nied, Shafiq Joty, Luo Si, and Chunyan Miao. 2022. GlobalWoZ: Globalizing MultiWoZ to develop mul- tilingual task-oriented dialogue systems. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume Long Papers), pages 1639-1657. Dublin, Ireland. Associ- ation for Computational Linguistics.
Rui Dong and David Smith. 2018. Multi-input atten- tion for unsupervised OCR correction. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 2363-2372, Melbourne, Australia. As- sociation for Computational Linguistics.
Andrew Drozdov, Nathamacl Schärli, Ekin Akyürck, Nathan Scales, Xinying Song. Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2023. Compositional semantic parsing with large language models. In Proceedings of the Eleventh International Confer- ence on Learning Representations (ICLR) 2023, Ki- gali, Rwanda.
Jack FitzGerald, Christopher Hench, Charith Peris, and Kay Rottmann. 2022. Massively multilingual natural language understanding 2022 (MMNLU-22) workshop and competition. In Proceedings of the Massively Multilingual Natural Language Un- derstanding Workshop (MMNLJU-22), pages 83-87, Abu Dhabi, United Arab Emirates (Hybrid). Associ- ation for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robin- son, Dasha Valter. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. An- drew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le. and Jason Wei. 2022. Scal- ing caaructiionfinetunel language models. arXiv preprint arXiv: 10.1 1476.
Goran Glavaš and Ivan Vulié. 2021. Is supervised syn- tactic parsing beneficial for language understanding tasks? an empirical investigation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisticx: Main Volume, pages 3090-3104, Online. Association for Computational Linguistics.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A bench- mark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454- 470.
Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang, HyunJcong Choe, David Greene, Kyle He, Rattima Nitisaroj, Anna Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, and Zhou Yu. 2023. PRESTO: A mul- tilingual dataset for parsing realistic task-oriented di- alogs. arXiv preprint arXiv:2303 333.00994..
Alexis Conneau, Min Ma. Simran Khanuja, Yu Zhang. Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. FLEURS: Few- shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technol- agy Workshop (SLT), pages 798-805, Doha, Qatar. IEEE
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores- 101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538
Marie-Catherine de Marneffe, Christopher D. Man- ning, Joakim Nivre, and Daniel Zeman. 2021. Uni- versal Dependencies. Computational Linguistics, 47(2):255-308
Alexander Gutkin, Cibu Johny, Raiomond Doctor, Brian Roark, and Richard Sproat 2022. Beyond

---

Arabic: Software for Perso-Arabic script manipu- lation. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)), pages 381-387, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguis- tics.
world. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 6282-6293, Online. Association for Computa- tional Linguistics.
Ahuja Kabir, Diddee Harshita, Hada Rishav, Ochieng Millicent, Ramesh Krithika, Jain Prachi, Nambi Ak- shay, Gamu Tanuja, Segal Sameer, Axmed Maxamed, Bali Kalika, and Sitaram Sunayana. 2023. MEGA: Multilingual evaluation of generative AL. arXiv preprint arXiv:2303.12522
Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindrich Helel, and Alexandra Birch. 2022. Survey of low-resource machine translation. Computational Linguistics, 48(3):673-732.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL- sum: Large-scale multilingual abstractive summa- rization for 44 languages. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021, pages 4693 4700, Online, Association for Computational Linguistics.
Simran Khanuja, Sebastian Ruder, and Partha Talukdar. 2023. Evaluating Inclusivity, Equity, and Accessi- bility of NLP Technology: A Case Study for Indian Languages. In EACL 2023 Findings, Dubrovnik, Croatia. Association for Computational Linguistics. To appear.
Shreya Khare, Ashish R Mittal, Anuj Diwan, Sunita Sarawagi, Preethi Jyothi, and Samarth Bharadwaj. 2021. Low resource ASR: The surprising effective- ness of high resource transliteration. In Proceedings of I'ieespeecc,, pages 1529-1533, Brno, Czechia. In- ternational Speech Communication Association.
Michael A. Hedderich, David Adelani, Dawei Zhu, Je- sujoba Alabi, Udia Markus, and Dietrich Klakow, 2020. Transfer learning and distant supervision for multilingual transformer models: A study on African languages. Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2580-2591, Online. As- sociation for Computational Linguistics.
Julia Kreutzer, Isaac Caswell, Lisa Wang. Ahsan Wa- hab, Daan van Esch, Nasanbayar Ulzii-rshikh, Al- lahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheak- mungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Sa- lomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo. Toan Q. Nguyen, Mathias Müller, André Müller, Sham- suddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapi- wanashe Matangira, Colin Leong, Nze Lawson, Sncha Kudugunta. Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çahuk Ball, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime. Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofe- toluwa Adeyemi. 2022. Quality at glance: An au- dit of web-crawled multilingual datasets. Transac- tions of the Association for Computational Linguis- tics, 10:50-72.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 4411 141114421, Online, PMLR.
Oana Ignat, Jean Maillard, Vishrav Chaudhary, and Francisco Guzmán. 2022. OCR improves machine translation for low-resource languages. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1164-1174, Dublin, Ireland. As- sociation for Computational Linguistics.
ISO. 2004. ISO 15924: Codes for the representation of names of scripts. International Organization for Standardization, Geneva, Switzerland.
Paul Jaccard. 1908. Nouvelles recherches sur la distri- bution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles, 44(163):223-270. In French.
Sameer Kumar, Victor Bittorf, Dehao Chen, Chiachen Chou, Blake Hechtman, HyoukJoong Lee, Naveen Kumar, Peter Mattson, Shibo Wang, Tao Wang. Yuanzhong Xu, and Zongwei Zhou. 2019. Scale MLPerf-0,6 models on Google TPU-v3 pods. In Proceedings of Workshop Systems for ML at NewrlPS 2019, Vancouver, Canada.
Cibu Johny, Lawrence Wolf-Sonkin, Alexander Gutkin, and Brian Roark. 2021. Finite-state script normal- ization and processing utilities: The Nisaba Brahmic library. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: System Demonstrations, pages 14-23, Online. Association for Computational Lin- guistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Ilaia Polosukhin, Jacob De- vlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP

---

Natural questions: A benchmark for question an-
for Natural Language Processing- ConsILR, pages 7-16, Online. Romanian Association of Computa- tional Linguisties.
swering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš, 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 4483 4483-4499, On- line. Association for Computational Linguistics.
Nikita Moghe, Exgeniia Razumovskaia, Liane Guillou, Ivan Vulić, Anna Korhonen, and Alexandra Birch. 2022. MULTIöNLU++: A multilingual, multi- intent, multi-domani dataset for natural language understanding in task-oriented dialogue. arXiv preprint arXiv:2212.10433
Peya Mowar, Tanuja Ganu, and Saikat Guha. 2022. Towards optimizing OCR for accessibility. arXiv preprint arXivv2206. 10254. Extended abstract for poster session Accessibility, Vision, and Auton- omy Meet (CVPR 2022 Workshop).
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual forentree semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2950-2962, Online. Association for Computational Linguistics.
Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ife- oluwa Adelani, Seid Muhie Yimam. Ibrahim Sa'id Ahmad, Meriem Beloucif, Saif Mohammad, Sebas- tian Ruder, Oumaima Hourrane, Pavel Brazdil, Fel- ermino Dário Mário António Ali, Davis Davis, Sa- lomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Be- lay, Wendimu Baye Messelle, Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael. Bernard Opoku, and Steven Arthur. 2023. AfriSenti: A Twitter sentiment analysis benchmark for African languages. arXiv preprint ar Xiv:2302.00855
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering. 34(1):50-70.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fen- fei Guo. Weizhen Qi. Ming Gong, Linjun Shou, Daxin Jiang. Guihong Cao. Xiaodong Fan. Ruofei Zhang. Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Win- nie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-linguts looss-lingual pre- training, understanding and generation. In Proceed- ings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng. Dimosthenis Karatzas, Zhenbo Luo, Uma- pada Pal, Christophe Rigaud, Joseph Chazalon, et al. 2017. ICDAR2017 robust reading challenge on multi- lingual scene text detection and script identi- fication RRC-MIT. In 2017 14th IAPR Interna- tional Conference on Document Analysis and Recog- nition (ICDAR), volume 1, pages 1454-1459, Kyoto. Japan. IEEE
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang. Yuxiang Hu, Chen Shi, and Pascale Fung, 2021. BiToD: A bilingual multi- domain dataset for task-oriented dialogue model- ing. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Datasets and Benchmarks Track (Round 1), volume
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muham- mad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo,, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu. Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife. Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper. Hady Elsahar, Good- ness Duru, Ghollah Kioko, Murhabazi Espoir, Fllan van Biljon, Daniel Whitenack, Christopher Onyefuluchi. Chris Chinenye Emezue. Bonaventure F. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir. 2020. Participatory research for low-resourced machine translation: A case study in African languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2144-2166
Olga Majewska, Edoardo Maria Ponti, Ivan Vulié. and Anna Korhonen. 2023. Cross-lingual dialogue dataset creation via outline- based generation. Transactions of the Association for Computational Linguistics, 11:139-156.
Evgeniia Razumovskaia,
Sabrina J. Miclke. 2019. Can you compare perplexity across different segmentations? Available in: http: //sjmielke.
//sjmielke..omm/cmmprrrttt www.pmppaaapppppeeeeeeeeeeeeeeee ties. htm. Blog post.
Verginica Barbu Mititelu and Maria Mitrofan. 2020. The Romanian medical treebank- SiMoNERo. In Proceedings of the The 15th Edition of the Interna- tional Conference on Linguistic Resources and Tools
Online. Association for Computational Linguistics.

---

Massimo Nicosia and Francesco Piccinno. 2022. Byte- level massively multilingual semantic parsing. In Proceedings of the Massively Mulrilingual Natural Language Understanding Workshop (MMNLU-22), pages 25-34, Abu Dhabi, United Arab Emirates (Hy- brid). Association for Computational Linguisties.
Shruti Rijhwani, Antonios Anastasopoulos, and Gra- ham Neubig. 2020. OCR Post Correction for Endan- gered Language Texts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 5931-5942, On- line. Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajie, Christopher D. Manning, Sampo Pyysalo, Sehastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034-4043, Mar- seille, France. European Language Resources Asso- ciation.
Shruti Rijhwani, Daisy Rosenblum, Antonios Anas- tasopoulos, and Graham Neubig. 2021. Lexi- cally aware semi-supervised learning for OCR post- correction. Transactions of the Association for Com- putcational Linguistics, 9:1285-1302.
Brian Roark, Lawrence Wolf-Sonkin. Christo Kirov, Sabrina J. Mielke, Cibu Johny, Isin Demirsahin, and Keith Hall. 2020. Processing South Asian languages written in the Latin script: the Dakshina dataset. In Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 2413-2425 Mar- seille, France. European Language Resources Asso- ciation.
Rubungo Andre Niyongabo, Qu Hong, Julia Kreutzer, and Li Huang. 2020. KINNEWS and KIRNEWS: Benchmarking cross-lingual text classification for Kinyarwanda and Kirundi. In Proceedings of the 28th International Conference Computa- tional Linguistics, pages 5507-5521, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang. Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez. Alex Salcianu, Mare van Zee, Jacob Austin, Sebas- tian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jas- mijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee- Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin- Shepard, Noah Fiedel, Mark Omernick. Brennan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203. 17189.
Sebastian Nordhoff and Harald Hammarström. 2011. Glottolog/Langdoc: Defining dialects, languages, and language families as collections of resources. In Proceedings of the First International Workshop on Linked Science 2011. In conjunction with the In- ternational Semantic Web Conference (ISWC 2011). volume 783, Bonn, Germany. Paper 7.
Addison Phillips and Mark Davis. 2009. BCP 47- Tags for Identifying Languages. IET Trust. RFC 5646.
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury. Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022. Efficiently scaling transformer in- ference. arXiv preprint arXiv:2211.05112
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Jun- jic Hu. Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R Towards more chal- lenging and nuanced multilingual evaluation. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing. pages 10215- 10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Maja Popović. 2015. chrF: character n n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Amarjot Singh, Ketan Bacchuwar, and Akshay Bhasin. 2012. A survey of OCR applications. Interna- tional Journal of Machine Learning and Computing. 33):314-318..
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784- 789, Melboume,, Australia. Association for Compu- tational L. Linguistics.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language model- ing. In Proceedings of Interspeech, pages 194-197, Portland, OR, USA. International Speech Communi- cation Association.
Christophe Rigaud, Antoine Doucet, Mickaël Cous- taty, and Lean-Philippe Moreux. 2019. ICDAR 2019 competition on post-OCR text correction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1588-1593, Sydney, Australia. IEEE
Cynthia Tam and David Wells. 2009. Evaluating the benefits of displaying word prediction lists on a per- sonal digital assistant the keyboard level. Assis- live Technology, 2 1153)105-114.

---

Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: I anguagg-inddeeedentt named entity recognition. In Proceedings of the Seventh Conference on Natu- ral Language Learning at HLT-NAACL 2003, pages 142-147.
A Contributions
In this section, we provide more detail about the contributions of each author.
General overview
Project leads Sebastian Ruder, Jonathan Clark
Hans H. Wellisch. 1978. The Conversion of Scripts: Its Nature, History, and Utilization. Information Sci- ences Series. John Wiley Sons, New York.
Primary contributors and task owners Alexander Gutkin, Mihir Kale. Min Ma, Massimo Nicosia, Shruti Rijhwani. Parker Riley, Jean-Michel Sarr, Xinyi Wang, John Wieting
Major contributors Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark, Bidisha Samanta, Con- nie Tao
Ken Whistler. 2021.
Unicode normalization forms. Technical Report TR15-51, Unicode Consortium. Version 14.0.0.
Supporting contributors David Adclani, Vera Axelrad, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Pantelcev, Partha Talukdar
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyaw- jjyaa, Rahmad Mahendra, Fajri Koto, Ade Ro- madhony, Kemal Kurniawan, David Moeljadi, Ra- dityo Eko Prasojo, Pascale Fung, Timothy Bald- win, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. NusaX: Multilingual Parallel Senti- ment Dataset for 10 Indonesian Local Languages. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics (EACL 2023), Dubrovnik, Croatia. Associa- tion for Computational Linguistics. To appear.
By Task ASR Min Ma
Autocomplete Jean-Michel Samr Vera Axelrod, Colin Cherry. Schastian Ruder, Jonathan Clark
MT Parker Riley, Isaac Caswell, Colin Cherry, Jonathan Clark
NER Sebastian Ruder, David Adelani, Dan Garrette
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards token-free future with pre-trained byte-to-byte models. Trans- actions of the Association for Computational Lin- guistics, 10:29 1-306.
OCR Shruti Rijhwani, Dana Dickinson, Reeve Ingle, Dmitry Panteleev, Sebastian Ruder
QA Mihir Kale, John Wieting, Nitish Gupta, Partha Talukdar. Jonathan Clark
Retrieval John Wieting
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre- traitra text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 483-498, Online, Association for Computa- tional Linguistics.
Semantic parsing Massimo Nicosia, Bidisha Samanta, Partha Talukdar
Transliteration Alexander Gutkin, Anna Katanova, Christo Kirov, Brian Roark
Evaluation framework and public results tracker Xinyi Wang Program management Connie Tao
Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulié, Roi Reichart, Anna Korhonen, and Hinrich Schütze. 2021a. A closer look few-shot crosslingual trans- fer: The choice shots matters. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Popers), pages 5751-5767 Online. Association for Computational Linguistics.
Fine tuning and modeling Alexander Gutkin, Mihir Kale. Min Ma. Massimo Nicosia, Shruti Rijhwani. Parker Riley. Jean-Michel Sarr. John Wieting, Sebastian Ruder. Jonathan Clark
Data processing Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Mech-M Sarr. Xinyi Wang. John Wieting. David Adelani. Vera Axel- rod, Isaae Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Dmitry Panteleev, Sebastian Ruder, Jonathan Clark
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 2021b. Calibrate before use: Im- proving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pages 12697- Online. PMLR.
Data collection Massimo Nicosia, Bidisha Samanta, Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark
In-context learning
Sebastian Ruder
Benchmark design Jonathan Clark, Schastian Ruder, Melvin Johnson

---

B Language Coverage
D.1.3 Data structure
We followed the data split of train, dev, and test sets in FLEURS, and filtered out the examples where Maestro-U prediction is empty (i.e., all the deletion errors). The pairs of transcript and ground truth are saved in jsonl and tsv format.
We provide an overview of the under-represented languages in XTREME-UP in Table 5. For each language, we indicate a) the ISO 639-1 code (or ISO 639-3 code if the former is unavailable); b) its language family according to Glottolog (Nord- hoff and Hammarström, 201 1); c) the number of datasets in XTREME-UU UP including the language; d) its resource level based on the taxonomy of Joshi al. (2020) (0 is least and 5 is highest-resourced);; and e) which tasks include the language.
The individual language datasets are mostly dis- tinguished by the language and region BCP-47 codes, e.g., the kam_ke code represents Kamba language spoken in Kenya. In some cases, when multiple writing systems are available for lan- guage, the ISO 15924 script code is used as well, as the case with the code sd_arab___ in that de- notes Sindhi as spoken India and recorded using Arabic script, as opposed to its Pakistani counter- part. 18
C Task Examples
We provide an example instance of each task in
T 6.
D.1.4 Data statistics
D Data cards
The FLEURS dataset contains about 1.4k hours of audio in total for 102 languages. The training data contains 27 2,444 examples across 102 languages, average length per utterance is about 20 tokens. There are 34,661 examples in the validation (dev) set, and 77,943 examples in the test set.
D.1
ASR
D.1.1 Task description
Automatic speech recognition (ASR) transcribes speech inputs into human- tuananre text, serving as fundamental step for various speech language understanding applications. The transcripts are often calibrated with some pre-trained language models to produce the final outputs, In this paper, we build the ASR benchmark in this way: first, transcribe input audio into text with a pre-trained speech recognition model; then calibrate the tran- scripts by fine-tuning pre-trained language models on paired transcripts and ground truths.
D.1.5 Experiments and Discussion
We compared fine-tuned mT5-base and ByT5 baselines, which were built on TPU. In addition, we explored the compute efficient fine-tuning on GPU, using a m'55-malll model as pre-trained model, The three models took 4500, 6500 and 4000 steps to converge, respectively. We report the character error rate for the predicted transcripts by the fine- tuned models against the one for the Maestro-U baseline, which is 8.28% on average for 102 lan- guages - quite strict baseline. We observed small gains through fine-tuning with different pre-trained models, as shown in Table 7.
D.1.2 Data creation
Experimented on the FLEURS corpus (Conneau et al., 2023), we use Maestro-U (Chen et al., 2023) to generate the ASR transcripts. For the pre-trained language models, we choose mT5-base (Xue et al., 2021) and ByT5-base (Xue et al., 2022) models. We paired the ASR transcripts with the ground truths to fine-tune the mT5 or ByT5 models. The average character eтror rate (CER) of Maestro-U is 8.28% across 102 languages, providing a strong baseline. Therefore, we build the ASR benchmark in selective way: first, we compare the Maestro- U baseline CER the dev set with the CER ob- tained by fine-tuned mT5 or fine-tuned ByT5. If the fine-tuned result is better, we choose the fine- tuned model for the language rescore its test set; otherwise, we keep the bascline Maestro-U U results for the test.
It is observed that ByT5 yields better fine-tuned results than mT5, indicating that byte is better modeling unit when it comes to textual data of various writing systems. By calculating the average CER for 24 high-resourced language group and 78 low-resourced language group respectively, we find that both mT5 and ByT5 fine-tuned models can reduce CER from 6.40% baseline to 6.36% for high-resourced languages, while ByT5 on its own can further improve CERs for low-resourced languages from 8.86% baseline to 8.80%.
Fine-tuned B5T5 also generalized well on lan- guages which were not seen in the pre-training
The es_419 code represents Latin American Spanish.

---

phase. With a limited amount of fine-tuning data, ByT5 can improve baseline on the group of unseen languages, especially on Umbundu (umb_ao, -14% CER Relative). Even though only Romanized Chi- nese is used to pre-train ByT5, the fine-tuned ByT5 outperformed basclines for both Mandarin (in sim- plified Chinese, com__aan__cn) and Cantonese (in traditional Chinese, cmn cmn_hant_hk).
natural language processing (NLP) task, current LM evaluation does not address the practical con- straints of human-computer interaction and current LMs are not directly useful for autocomplete in under-represented languages.
In order to evaluate multilingual models on an evaluation setting as close possible to the real- world usage of autocomplete, we curated the Uni- versal Dependencies (UD) dataset (Nivre al., 2020; de Marneffe al., 2021) according to set of high level principles that we describe in the section below.
D.2
Optical character recognition (OCR)
D.2.1
General information
Dataset title UL-OCR
D.2.2 Related work
D.3.2 Data creation
While most existing datasets focus on higher- resourced languages (Nayef et al.. 2017: Rigaud et al., 2019), there has been recent interest in de- veloping OCR for ULs. This includes the creation of dataset for OCR endangered languages (Ri- jhwani et al., 2020) and synthetic dataset for 60 languages (Ignat et al., 2022).
The original UD dataset was filtered to better fit the user centric paradigm proposed. We removed a) treebanks using only ancient data, for example litur- gical text written in Latin, Ancient Greek or San- skrit; b) languages with fewer than 100 speakers like Akuntsú; c) signed languages like the Swedish Sign Language; d) highly domain-specific content like for instance SiMoNERo (Mititelu and Mitro- fan, 2020) which contains texts from three medical subdomains: cardiology, diabetes, endocrinology; e) languages that are "high resource" by XTREME- UP standards with the exception of English which we kept for prototyping: f) languages that do not have all three of: training, validation and test sets: g) languages with fewer than 1000 examples when combining training and validation set.
D.2.3 Data creation
We retrieve books that are in the public domain on Google Books. These are historic books, where the copyright has expired, as well as more recent and public-domain books, used in this dataset with approval from their publishers. We focus on lan- guages with diverse scripts, for which no existing OCR dataset currently available. We observe that many public-domain books in such languages are religious or linguistic in nature and were created for missionary purposes. In order to identify a di- verse set of high-quality books, we first conduct an annotation task where we ask annotators to look at pages of book and assign whether it is not in the target language, b) religious, c) consisting mainly of tables/other structured formatting, d) linguistic (e.g., dictionary or grammar book), e) not intelli- gible, or f) low quality. Based on this annotation, we filtered out some languages that did not have a sufficient amount of high-quality public-domain books available. After filtering, the dataset contains annotated documents in seven under-represenned languages- described in detail in Section 3.3.
The resulting corpus features 23 languages: Basque, Belarusian, Bulgarian, Danish, Eastern Armenian, English, Estonian, Galician, Scottish Gaelic Greek, Hebrew, Icelandic, Indonesian, Irish, Latvian, Lithuanian, Nigerian Pidgin, Romanian, Slovak, Slovenian, Ukrainian, Urdu, and Uyghur.
D.3.3 Data structure
A data instance has two fields, input and target, for instance (input: "en_-We look fS", target: "for- ward"). The input field is composed of prefix "en__" -" indicates the language to the model and a context sentence: "We look f$". The target field is the word to predict. We normalize all text with Unicode NFKC normalization (Whistler, 2021).
Annotation process In the following, we de- scribe how the example described above is gen- erated from the source data. The original sentence is "We look forward your active participation make this forum an exciting meeting place for like minded individuals." The steps are: a) The context sentence including the target can have at most 10
D.3 Autocomplete
D.3.1
Task description
Autocomplete (or predictive text), i.e., predicting the rest of a word a user is typing, is a useful tech- nology that speeds up human-computer interaction. However, while language modeling (LM) is a core

---

words. A random word of more than 5 characters is chosen to be the target. b) A target context is sampled from the target and added to the context. In this example it is the character "f"" The sample rule is to select a number of characters that can vary between 0 to the number of characters in the target minus three. In our example, the target "forward" could be sampled from to "forw". c) A specific token "$" is added just after the target context.
same words? c) Do some languages yield higher scores because they have a smaller vocabulary in their corpora? d) Does similarity to the Latin alpha- bet impact models' performance? We test several hypotheses below, considering a relationship to be significant when the p-value is under 0.05.
Do mT5 and ByT5 have the same cross-lingual generalization pattern? mT5 and ByT5 have the same cross-lingual generalization pattern if the difficulty to generalize to new language is the same for both models relatively to other languages. In other words, if models' performance are ranked similarly, they share the same cross-lingual gener- alization pattern. To evaluate this hypothesis we computed the Spearman's 's rank correlation between mT5 and ByT5 Acc@3. We got a Spearman's rank correlation of 0.69 with p-value < 0.001. This means that the two models have a high degree of relative agreement, in other words, if new lan- guage is added, there is high chance that the language is going to be challenging or not for both mT5 and ByTs.
D.3.4 Data statisties
We sampled up to 2,000 examples from each lan- guage's training set, 1,000 examples from valida- tion, and 1,000 examples from test. This prevents the languages from having disproportionately more data; where the original sets were smaller than these targets, we used all available data. We dis- play the language statisties in Table 8. Note that these experiments are done on a preliminary dataset and not the final release version of XTREME-UP.
D.3.5 Experiment
We compared mT5 (Xue et al., 2021) and ByT5 (Xue et al., 2022), two statc-of-the-an art multilingual pre-trained LMs that are based on subwords and bytes respectively. The models were fine-tuned for 10 epochs on autocomplete training set, more- over. We used two metries: top-3 word accu- racy (Ace@3) and chrF: character n-gram F-score (Popović, 2015).
Do some languages yield higher scores because autocompletion guesses the same words? If our dataset in given language over-represents word to predict, then the model might have mis- leadingly good performance by always predicting the same word. This would mean that the dataset is not balanced with regards to the diversity of target words. A common way to model the diversity of a distribution of words is to compute its entropy, so we computed the the Pearson correlation between the entropy of the test set's target word distribution in each language and mT5 and ByT5 Acc@3, The entropy of a distribution of word is maximal ev- ery word different, and minimal if it consist on a single word. mT5 and ByT5 displayed corre- lation coefficients of 0.16 and 0.13 respectively with p-value of 0.45 and 0.53 respectively. These results show that there is insufficient evidence to conclude that there is a significant linear relation- ship between target words diversity and model per- formance because the p-value is far above the 0.05 significance threshold. Hence, target word diver- sity is not a good predictor of model performance variability across languages.
D.3.6 Results
We observe that B5T5 achieve better performance than mT5 for both Acc@3 3 and chrF on the auto- complete task as it is displayed in Table 9. Also ByT5 require less than half the time to fine-tune on the training set (45 minutes) compared to mT5 (1 hours and 30 minutes).
D.3.7 Analyses
Based on Ace@3 and chrf, the most challenging languages for mT5 are Eastern Armenian ((hy) and Uyghur (ug) respectively. Whereas Nigerian Pidgin is the (pcm) and Scottish Gaelic are the eas- iest languages. For ByT5, whether we consider Acc@3 3 or chrF, the most challenging language is Uyghur, and the easiest language is Galician (gl). Yet. these extremes only offer a qualitative com- parison of mT5 and ByT5. Next, we investigate four questions around model performance: a) Do mT5 and ByT5 have the same cross-lingual gen- cralization pattern? b) Do some languages yield higher scores because autocompletion guesses the
Do some languages yield higher scores because they have a smaller vocabulary in their cor- pora? We expect that languages with smaller cor- pora will be easier to tne-tune on because of a

---

smaller prediction space, To test that hypothesis, we computed the Pearson correlation between test set's vocabulary size and mT3 and ByT5's Ace @3 for each language. mT5 and ByT5 displayed corre- lation coefficients of - 0.29 and 0.13 respectively with p-value of 0.17 and 0.54 respectively. Thus there is insufficient evidence to conclude that there is a significant linear relationship between vocab- ulary size and model performance because the p- value is above the 0.05 significance threshold.
tences, not just isolated terms, from one script another.
D.4.2 Data Creation and Annotation process Most of the data for the task comes from the ro- manized full-string subset of the Dakshina dataset (Roark et al., 2020), in which 10,000 Wikipedia sentences written in the native scripts of the 12 languages were human- romanized by native speak- ers, resulting in parallel sentences in the native and Latin scripts. Two 10,000 sentence additions were made this data for the current transliteration task: Amharic Wikipedia sentences were similarly manually romanized by native speakers; and the Punjabi sentences from the Dakshina dataset, orig- inally written in the Gurmukhi (Brahmic) script, were manually transliterated by native speakers to the Shahmukhi (Perso-Arabic) script.
Does similarity to the Latin alphabet impact models' performance? We verify this hypoth- esis quantitatively by computing the similarity be- tween a) a Latin alphabet composed of the 26 let- ters of the alphabet in lower and upper case and b) the alphabet of each language corresponding to all the characters in the test set except punctu- ation and special characters. The similarity was computed with the Jaccard similarity coefficient (Jaccard, 1908), i.e. the ratio of number of unique items in the intersection of both alphabets and the number of unique items in the union of both al- phabets. Moreover we used the same methodology as before and computed the Pearson correlation between the Jaccard similarity index and chrF as this metric is more granular in models' character level performance. We observed correlation of 0.56 and 0.75 for mT5 and ByT5 respectively with p-values 0.01 respectively. It indicates that the similarity between the Latin alphabet and each lan- guage alphabet significantly correlated to mT5 and ByT5 chrFF.
D.4.3 Data Preparation
The resulting collection allows for overall 30 tasks converting between various scripts. These are sum- marised in Table 10 where, for each language in- dicated by the BCP-47 code (Phillips and Davis, 2009), the corresponding transliteration tasks are shown for scripts indicated by their IS0-15924 codes (ISO, 2004).
All the native script data was normalized using Unicode NFC (Whistler, 2021). The data was then further transformed using sanguage--speiffe visual normalization for Brahmic and Perso-Arabee writ- ing systems using the Nisaba script normalization library (Johny et al., 2021; Gutkin et al., 2022). Both NFC and visual normalization operations pre- serve visual invariance of the input text, with visual normalization handling many ambiguous cases that fall outside the scope of standard NFC.
D.3.8 Evaluation and Discussion
Whether we used a word level metric like @ or a character level metric like chrI,, ByT5 is more accurate at autocomplete than mT5. We also ob- serve that these models generalize more easily to languages written in an alphabet closer to the Latin alphabet, ByT5 being more sensitive to the alpha- bet of the input language.
D.4.4 Data Statisties
For each task, we establish 2,000 training sen- tences, 2,000 development set sentences, and close to 6,000 test sentences. Training data for any pre- trained models used in the task cannot include the Dakshina dataset. Since this is contextual few- shot transliteration benchmark, we do not provide the romanization lexicons that were released in the Dakshina dataset along with the full sentence ro- manizations.
D.4 Transliteration
D.4.1 Task description
Transliteration is the conversion of text in one writ- ing system to another writing system, e.g., text written in the Devanagari seript to the Latin script. It differs from translation in that it does not change the language content of the text, just the script. Many languages are written in multiple scripts, and the current task involves transliterating whole sen-
Our few-shot contextual transliteration task covers 13 languages from 3 language families
In the Dakshina distribution, the parallel sentences can romanized. LANG. where LANG is a BCP-47 language code.
he found in files named
rejoined. tsv,

---

Alyao-Aryan, Dravidian and Semitic), all but one (Amharic) from South Asia.
size of 128. We used Google TPU-v3 accelera- tors (Kumar et al., 2019) for fine-tuning all the configurations apart from ByT5 Large for which a more powerful TPU-v4 (Pope et al., 2022) was necessary.
D.4.5
Directionality and Evaluation Ambiguity
One difference between romanization in these lan- guages and transliteration in the opposite direction (from the Latin script to the native script) is that none of the languages in the benchmark have an or- thography in the Latin script, i.e., there is no single correct spelling in the Latin script for these lan- guages. Rather, individuals tend to provide rough phonetic transcription of the sentences using the Latin script. As result, word identity may be dif- ficult to achieve (hence high word-error rate), but string similarity should be relatively high between quality romanizations hence we use character-error rate to evaluate the transliterations. The ability to produce romanizations automatically has several key use cases, including simulation of parallel data from mono-script language samples, and for mul- tilingual modeling of languages that use different scripts. For that reason, we include both directions in the benchmark.
D.4.7 Evaluation and Discussion
The evaluation results of the full fine-tuning setup described above are provided in Table 11, which shows the character error rate (CER) for each of the 30 transliteration tasks in six configurations, along with the corresponding averages over all the tasks.
Some general trends are observable in these base- line results. The ByTS error rates are generally substantially better than mT5, and, while the size of the configuration matters for mT5, it does not seem to matter much for ByT5. Overall, roman- ization harder, i.e., transliterating into the Latin script yields higher error rates than transliterating out of it, perhaps due to the fact that there is no set orthography in the Latin script in those languages. For the best performing configuration - 9 out of 10 of the tasks with the lowest CER are from Latin script to native script. All of the tasks with the highest CER are into either the Latin Perso-Arabic scripts, and all of the tasks transliter- ating Perso-Arabic input have worse-than-media CER. In other words: Perso-Arabic input is hard; Latin output is hard; and Perso-Arabic to Latin is particularly hard.
D.4.6 Experimental Setup
Previously Xue al. (2022) performed ByT5 fine- tuming and evaluation of transliteration and ro- manization directions separately on single-word, rather than full-sentence, data from vanilla Dak- shina dataset. In this benchmark we remove the separation into transliteration and romanization by requiring all tasks to be fine-tuned jointly. In order achieve this, during all stages of training, devel- opment and testing a special code is prepended to the input fcature strings for each task. This task code indicates that the input features correspond to the conversion from writing system Source to writing system Target for language lang. It is For
D.5 Machine Translation
D.5.1 Data Card
Basic Info
1. Original datset name: FLORES-101
https://githuit com/ facbborrreeerr////oorr//////aai flores200
Repository:
encoded string Laang_Source_aaaaa example, for Punjabi (pa) conversion from Shah- mukhi (Arab) to Gurmukhi (Guru) writing systems, _u__uu""
the task code is
3. Paper:
Goyal et al. (2022)
In the full fine-tuning setup2 we jointly fine-tune the 30 transliteration tasks using mT5 and ByT5 models in Small, Base and Large configurations that correspond to around 300M, 582M and 1.2B parameters, respectively (Xue et al., 2021, 2022). Fine-tuning uses 10K training steps with batch
4. Point of Contact (original version): NLLB Team (flores @ fb.com)
Why is this dataset part of XTREME-UP? Ma- chine translation is an important tool for expanding language coverage for natural language process- ing tools. FLORES-101 is a high-quality, highly- multilingual dataset.
This full fine-tuning setup contrasts with an efficient fine- tuning setup relying on fewer fine-tuning steps and smaller batch sizes, the results of which are reported in Table 4.

---

(pa), Shona (sn), Sindhi (sd), Somali (so). Swahili Data Fields
(Chichewa) (ny), Oromo (om), Pashto (ps), Punjabi
1. input: the source sentence, which is always English (string)
(sw), Tajik (tg), Telugu (te), Welsh (cy), Xhosa (xh), Yoruba (yo), Zulu (zu).
2. target: the larget-language translation of the source sentence (string)
Data Statisties 50% of the FLORES- 101 dev split was reserved for training and the remainder for validation. The original devtest split was unchanged and reserved for testing. This results in 499/498/1012 sentence pairs for train/valida- tion/test, respectively.
Data Example ("input": "<2xh> Local media reports an airport fire vehicle rolled over while responding.", "target": "Oonondaba basekuhlaleni bxele ukuba isithuthi somlilo sesitishi senqwelomoya siye saphethuka sisazama ukunceda.")
The original dataset was cu- rated by the NLLB (No Language Left Behind) Team (hores@fb.com). The version included in XTREME-UP was curated by Parker Riley (prkriley @ google.com) and Isaac Caswell (icas- well googlegoo...mmm
Dataset Curators
Languages Included in XTREME-UP release (93): Afrikaans (af), Amharic (am), Arabic (ar),
(Eastern) Armenian (hy), Assamese (as), (North) Azerbaijani (az), Belarusian (be), Bengali (bn),
Curation Rationale The original FLORES-101 dataset was created to be able to evaluate machine translation models in many languages. The version released in XTREME-UP was created to focus on low-resource languages and provide an in-domain train split along with validation and test splits, all of sizes in line with other tasks in XTREME-UP.
Bosnian (bs), Bulgarian (bg), Burmese (my), Cata- lan (ca), Cebuano (ceb), Central Kurdish (ckb), Chinese (zh), Croatian (hr), Czech (cs), Dan- ish (da), Dutch (nl), Estonian (et), Finnish (fi), French (fr), Fula (ff), Galician (gl), Georgian
(ka), German (de), Greek (el), Gujarati (gu), Hausa (ha), Hebrew (he), Hindi (hi), Hungarian
(hu), Icelandic (is), Igbo (ig), Indonesian (id), Irish (ga), Italian (it), Japanese (ja), Javanese (iv), Kannada (kn), Kazakh (kk), Khmer (km), Ko- rean (ko), Kyrgyz (ky), Lao (lo), Latvian (1v), Lin-
Data Sources The source data (selected by the NLLB Team) comes from Wikinews, Wikijunior, and Wikivoyage.
gala (ln), Lithuanian (1t), (Lu)Ganda (lg), Lux- embourgish (1b), Macedonian (mk), Malay (ms),
Dataset Creation Details of the creation of the original dataset are available in the original publi- cation (Goyal et al., 2022).
Malayalam (m1), Maltese (m1), Maori (mi), Marathi (mr), Mongolian (mn), Nepali (ne), Pedi (Sepedi) (Northern Sotho) (nso), Norwegian (no), Nyanja (Chichewa) (ny), Oriya (or), Oromo (om), Pashto (ps), Persian (fa), Polish (pl), Portuguese (pt), Punjabi (pa), Romanian (ro), Russian (ru), Serbian (sr), Shona (sn), Sindhi (sd), Slovak (sk), Slove- nian (sl), Somali (so), Spanish (es), Swahili (sw), Swedish (sv), Tagalog (t1), Tajik (tg), Tamil (ta),
Changes to the Original Dataset for XTREME- UP The version of the dataset in XTREME-UP UP only has the source and target strings, removing additional metadata. We also include 93 of the original 100 non-English languages (the subset sup- ported by Google Translate). Of these, only 39 are used for official evaluation.
Telugu (te), Thai (th), Turkish (tr), Ukrainian
D.6 Question Answering and Retrieval
(uk), Urdu (ur), Uzbek (uz), Vietnamese (vi),
D.6.1 Data Card
Welsh (cy), Xhosa (xh), Yoruba (yo), Zulu (zu).
Basic Info
Evaluated in benchmark (39): Amharic (am), (Eastern) Armenian (hy), Assamese (as), (North)
Original datset names: TyDi QA, XOR-TyDi
Azerbaijani (az), Burmese (my), Central Kurdish
QA
(ckb), Gujarati (gu), Hausa (ha), Icelandic (is),
2. Additional cross-lingual data was collected part of XTREME-UR following similar methodology
Igbo (ig), Irish (ga), Javanese (jv), Kannada (kn). Khmer (km), Kyrgyz (ky), Lao (1o), Lingala (1n). Gan)aanda (1g), Luxembourgish (1b), Macedo- nian (mk), Malayalam (ml), Mongolian (mn). Nepali (ne), Pedi (Sepedi) (Northern Sotho) (nso), Nyanja
Why is this dataset part of XTREME-UP? Question answering enables information access.

---

Data Fields
Query
Start playing Rihanna
latest
album
1. question: question in the target language (string)
Parse
[IN: MUSIC MUSICU _
+ Rihanna] [SL: MUSIC_TYPP album]
2. title: the title the evidence passage target language for in-language setting, English for cross-language setting (string)
Linearized Query
Start
playing SL: MUSMC_ _T_NE * Rihanna)'s MUSIC TYPE (allm))
3. passage: the evidence passage, which might contain answer to the question target language for in-language setting, English for cross-language setting (string)
Figure 2: Creation of linearized query from the actual query and its parse for semantic parsing.
Changes to the original datasets for XTREME- UP The original MasakhaNER datasets are pro- vided in CoNLL format where each input sentence is already tokenized. This makes it difficult to evaluate NER models on natural text where tok- enization may often be messy and introduces bias towards word and subword- based models. To pro- vide a level playing field and enable evaluation of NER models on natural data data, we process the data order to align the token-level annotations with byte-level spans in the original pre-tokenized text. For the NER task, we provide the original pre- tokenized text as input to the model. Hausa and Fon subsets of the original data were excluded matching with the unlabeled source data revealed annotation artefacts in both language subsets.
4. answer: the answer (if any) to the question (string)
Data Example
See Table 6.
Languages
See Table 5.
Data Statistics
See Table 1.
Data Sources Evidence text was sourced from Wikipedia.
Dataset Creation Details of the creation of the original dataset are available in the original TyDi QA and XOR QA publications.
D.7 Named Entity Recognition (NER)
Dataset and task description The dataset con- tains processed data from MasakhaNER (Adelani al., 2021) and MasakhaNER 2.0 (Adelani et al., 2022). Both datasets were created by Masakhane
D.8 Semantic parsing
D.8.1 Task description
Semantic parsing is the task mapping a natural language utterance to a logical form or structured interpretation that can be executed by a system such as virtual assistant. For XTREMB-UR we adapted the MTOP (Li et al., 2021) test dataset to 15 lan- guages, and to code-switched Indic languages. The original MTOP data was published by Face- book and covers languages across 11 domains, 117 intents and 78 slots.
Why is this dataset part of XTREME-UP? Named entity recognition is a fundamental task in natural language processing. The Mas datasets are high-quality multilingual datasets that provide data in 20 African languages. The data is human-annotated annotated and thus higher quality than automatically collected NER datasets.
D.8.2 Data creation
Languages and ISO 639-3 codes Bambara (bam), Ghomálá' (bbj), Éwé (ewe), Igbo (ibo), Kin- yarwanda (kin), Luganda (lug), Luo (luo), Mossi (mos), Naija (pem), Chichewa (nya), chiShona (sna), Kiswahili (swa), Setswana (tsn), Akan/Twi (twi), Wolof (wol), isiXhosa (xho), Yorùbá (yor), isiZulu (zul)
In this section, we describe the two processes used to extend the MTOP instances: the first involves translation and localization with professional trans- lators and the second code-switching using lan- guage model and verification by human annotators.
In both processes, we perform a linearization step of the query and parse. Given an English utterance from the MTOP English test set and the
2'https:/ //www. www.wmawaaaaaaaaaammmmm io/

---

corresponding slot information (slot names each with start and end bytes), we add slot tags around corresponding tokens the query (Figure 2).
Parallel sentence prompting. In this case. the prompt contains (a) the task description. (b) the original English linearized query, and also (c) the target translated query (obtained with Google translate) and (d) the correspond- ing code-mixed query.
Motivation Recently, researchers published more multilingual semantic parsing datasets that focus on virtual assistant domains (Li ct al., 2021; FitzGerald et al., 2022; Moghe et al., 2022; Goel et al., 2023). We extend a portion of an existing se- mantic parsing dataset to new languages targeting the following features: a) high-quality utterances produced by professional translators; b) wide range of domains and intents; e) inclusion of differ- ent language families and some underrepresented languages; d) sentences with culturally relevant en- tities; and e) code-mixed sentences, i.e., multiple language within the same sentence-a common phenomenon in multilingual societies.
Parallel reordered sentence prompting, Similar to the previous, however, target trans- lated queries are human written.
We observed that the Parallel sentence prompt- ing was producing higher quality utterances, with 7/11 correct conversions for Findi-Engliss 6/1 for Bengali-English, and 8/11 for Tamil-English. We used this strategy design prompts with the help of native speakers of those languages. We selected 21 sentences from the training split for creating corresponding exemplars for the prompts. With the latter, we performed few-shot prompting with the 64b PaL.M model and converted the test split of MTOP to code-switched corpora.
Translating MTOP to 15 languages: We take the bracketed versions of the slot-tagged English sentences from MTOP and we create translations and localization tasks to be carried out by profes- sional translators. We ran two pilots on small sample of the data to gather feedback and improve the annotation guidelines. The translators had to translate the original utterances to given target language, while keeping the brackets around slot value translations and localizing those where possi- ble. Once the pilots were completed without issues, we scaled the tasks to the full test set.
Human annotators then had to check the PaLM generated data for the presence of code-mixing and for the labeling be consistent between the original query and the code-mixed version. The annotators were instructed to fix the automatically generated data whenever they found such issues.
D.8.3 Data structure and statistics
To create the training, validation and testing splits for MTOP, we start from the English test set and remove intents with less than 10 examples. This leaves us with 53 intents and a maximum of 4,223 examples for each language (some original MTOP languages may have less examples, while our code- switched data may have more due to multiple para- phrases).
We carried out manual inspections on samples of the data to check if translation and localization was happening correctly, and a set automatic checks on the full data to ensure that slots were matching between original and translated utterances. Data was sent back to annotators until all the issues were fixed.
For each intent, randomly select training amples such that each slot is covered by at least example, for minimum of examples. We end up with training, development and test sets containing respectively a maximum of 285, 239, and 3,669 instances for each language.
Code-switching MTOP to 3 Indic languages: We use PaLM to convert the lincarized query into code-mixed query using few-shot prompting. We experimented with different discrete prompt de- sign strategies and selected the best prompts after a qualitative evaluation on a small held-out set (1) examples) covering all 1 domains. Specifically we experimented with three designs.
D.8.4 Experiments
We fine-tune mT5 (Xue et al., 2021) and ByT5 (Xue et al., 2022) in their base and large configura- tions on the multilingual training data collected. Table 12 contains the Exact Match accuracies of multilingual model trained on data from all lan- guages but the code-switched sets. Table 13 con- tains the results of a model that includes the code-
Naive prompting. The prompt contains (a) the task description followed by set of ex- amples consisting of (b) the original English linearized query and (c) the corresponding code-mixed version.

---

switched sets. From both tables, can see that ByTS-base more accurate then the other models, even compared with the larger ones. This surpris- ing result confirms similar findings on word-level tasks reported by Xue et al. (2022) and Nicosia and Piccinno (2022). We expect mT5 to catch up with ByT5 at larger sizes.
E In-context learning examples
We show in-context learning examples for selee- tion tasks in Table 14. Each example consists of a general instruction and prefixes for the input and target, which are repeated for each exemplar.

---

ISO code
Language
# of datasets
Resource level
Semantic parsing
Translit- eration
Auto complete
Language
Family
QA
Retrieval
NER
MT
ASR
OCR
Indo-European
Ambaric
sm
Afro Asiatic
2
Asameese
Inko-Furopaan
Asturion
Buropean
Azerbaijani
Turkie az
3
Ghomálá'
Adlasti-Conp
0
Belanssinn
be
Indo-Furopaan
Bulgarian
Mande bg
Indo- Inde-European
3
Bambura
Bengili
SSSSSS
Bosnian
bs
Indo Bare-eur
2
Cebuano
Austronesian
Central Kurdish
ckb
Indo Baropean
Welsh
cy
Imko-Brrepan
da
Indo-Buropean
Fwe
en
Atlantie-Oongo
Greek
el
Indo Buropean
Estonian
et
Uralic
3
Fula
Ff
Atlantic Congo
2
Filipino
Irish
ga
mnde-Brrpeean
Galician
g1
Indo- Indo-Burepean
3
Gujarati
qu
European
Hausa
ha
Afro-Asiatic
Hebrew
he
Afro-Asiatic
Armenian
hy
Indo-Furopean
Indomesian
id
Austromesian
5
Atlantic-Conge
Icolandic
15
Indo Baropen
Javanese
IV
Austronesian
3
Gevrgian
Kartvelian
Kamba
kan
Atlantic Congo
Kabuvverdamm
kea
Imko-Furopzan
2
Khmeer
lon
Austronsiatic
Kannada
kn
Dravidian
Kyngy
ky
Loxembourgish
16
Inde-European
3
(LuXiinda
Atlansie-Conno
Lingala
1n
Atlantic-Cong
3
Lao
10
Tai Rad-Kaaai
2
Indo-Buropean
Luo
luo
Nilotic
Latviam
1v
Imdo-Bunopaan
Indo-Buropean
Malayalum
m1
Draidian
Mongolian
Mongolic Khitan
Mosii (Mouré)
nos
Atlantic- Congo
Marathi
mr
Indo European
3
2
11555551 SS
Malay
Austromsium
Maltose
nt
Afro-Asiatic
Burmese
Sino-Tibetan
Nepali
ne
European
Norwegian
no
ndd-BBaroeaan
2
Norther Sotho
Atlantic Congo
Atlantie-Congo
3
Nyanja (Chichewa)
ny
Atlantic-Congo
Ouziton
on
Inko-Faropaan
2
Afto-Asiatc
Oriya
or
mnd--uropean
Punjabi
pa
Indo European
2
Nigerian Pidgin
Indo-Europpan
Pashto
p5
Indo- Bur--arepeann
Romanian
Indo-Furopean
Kinyarwanda
Atlansic Congo Congo
Sanskrit
5a
Indo-Earopean
Sindhi
Indo European
Indo-Fanopean
Slovak
sk
Indo-European
3
Slovenian
Indo-Earopean
Shona
5n
Adlanti-Congo
Somali
50
Asto-Asiatic
Congo
Tamil
Dravidian ta
Telogo
Le
Dravidiam
TTjik
tx
Indo-Buropean
Thai
th
Tai-Kndai
Tswana (Setswana)
Atlantic-Comp
Twi
tw
Allatti-Coogg
Uyghur
ug
Turkic
Ukasinian
Indo-Faropaan
Umbundu
unb
Adlazti-Conge
2
Unda
ur
Inde-Eurepean
Wolof
MD
Atlantic-Conge
Congo
Xhosa
xh
Atbasti-Congo
yo
AtlanticCCong
Congo
Zuhu
zu AllntieCoonge
Table Overview of under-represented languages covered in XTREME-UP.

---

Task
Language
Input
Output
marekebisho maombi yoyole laz- yafuatwe kupitia wakula kusafiri kwanza na si moja kwa muja na hoteli
Speech Recognition
Swahili
Soo Roódo oo eg6:
Segpnboosupppreeey o Eijooogoo oge= ogonn soyoolnggoo molfpnösenqppupe *éi rofanborrbeegi Guponóro Schapiso@eraajf qeoqbuakes uefrionó
ogamn agcodragoor googorreingrggn aS onStRoonSoogi Gg os odcqu Softae @ çeug Guaße: oo
Document OCR
Burmese
Aulocomplete
Nigerian
make just dey back to my papa hou
house
Pidgin
Transliteration
Marathi
surguja bhagatale rahivahi.
उमाा पााार रतरिायीी
Machine Translation
Xhosa
was developed by John Smith
Yeenziwa nguJohn Smith kwiminyyka yee- 1970 ukunceda iifolda ezingena- mava okanye ezo zinobuchule bemoto obulinganiselweyo.
1970s to help inexperienced folders or those with limited motor skills.
pirk
Pessage: Sduee ongr eogg 1x25 grons Andrdibrr iobota asseet cang 2dad engilf avag who seiiião a.d big ghaddo dhoban wogfff aacinee sherdds adtar stownt audeeigg 660s wne cbassnn grout 28mdd.111
In-language Retrieval for QA
Telugu
eogetya
tags abmddma aldd??
Question:
uegelpa iag dbaszmo walt? ddurö cmprlg))
Passage: 2 of eegale bas bard de pags 265a wogefs dld mrad sae sdiaaio 4ab arg ad esd and grivdo drodld. uogdhe asagse argede ad down Sudeeaa addd and dhardi srean 2errth.11).
coumpies)
In-language QA
Telugu
Title: Satyavati
Context: Daughter of the Chedi king, Vasu (also known as Uparichara Vasu) and cursed "apsara" (celestial nymph) who turned into fish called Adrika, Satyavati was brought up as commoner...
[ che a9%0e1 Nan endowa QIaI asß?
Cross-language Retrieval for QA
Oriya
Question:
og & deser on erdewe ae??
Title: Satyavati
Context: Daughter of the Chedi king, Vasu (also known Uparichara Vasu) cursed "apsara" (celestial nymph) who was turned into fish called Adrika, Satyavati was brought up commoner...
Uparichara Vasu
Cross-language QA
Oriya
tor No Ansver" for some examples)
LOC: Ërob LOC: Farās
NER
Wolof
Dafa di, nag, Erob rawatina Farās, dañuy xeex ak a bunduxatual tuu- taaf6oni waaso yi.
[IN:GE WNN:GETEEVENT SL:AATRIBBTE__EE yasendaweni] :SL:DATE_TIME kuleli sonto]
Semantic Parsing
Zulu
Ingabe ikhona imicimbi yasendaweni eqhubekayo kuleli sonto
Table 6: Examples each task in XTREME-US UP. The tasks are generally text-in, text-out with few exceptions. On the output side, autocomplete requires generating the top-3 outputs and retrieval outputs document identifiers- current systems tend to implement retrieval by mapping both inputs and candidate outputs to vector and performing nearest neighhor lookup. On the input side, speech recognition has audio input and document OCR has image outputs: our initial baseline systems use external systems to map this to text as a preprocessing step, though we hope to see multi-modal systems eliminate this step the near future.

---

Maestro-U
mT5
ByT5
Maestro-U
m
By
Task Code
Task Code
Base
Small
Base
Base
Small
Base
af
4.19
4.19
4.24
4.19
continued
am_et
8.60
8.60
8.66
8.6
mt_
11.48
11.57
11.56
11.57
ar_eg
6.00
6.00
6.03
6.00
14.70
14.70
14.87
14.66
as_in
8.49
8.49
8.56
8.49
nb_no
4.14
4.14
4.21
4.11
ast_es
4.49
4.49
4.66
4.49
ne_np
9.22
9.26
9.25
9.22
az_azz
5.67
5.67
5.7
5.67
nl_n1
3.15
3.15
3.27
3.15
be_by
3.34
3.34
3.42
3.34
7.13
7.13
7.17
7.13
bn_in
6.16
6.16
6.20
6.16
ny_nw
7.08
7.08
7.08
6.95
bs_ba
2.93
2.93
3.05
2.93
oc_fr
7.68
7.68
7.84
7.68
2.72
2.72
2.76
2.72
14.36
14.36
14.52
14.36
ceb
4.36
4.36
4.49
4.36
or_in
7.42
7.42
8.70
7.42
8.70
8.70
8.77
8.70
pa_in
7.35
7.35
7.38
7.35
cmn_hans_en
16.48
16.06
16.12
16.06
pl_pl
2.49
2.19
2.52
2.49
con_hant_hk
34.82
34.23
34.24
34.13
ps_af
16.82
16.82
16.86
16.82
cs_cz
3.30
3.30
3.35
3.30
pt_br
2.87
2.87
3.05
2.87
cy-gb
7.11
7.11
7.17
7.11
3.58
3.58
3.64
3.58
da_dk
6.30
6.30
6.35
6.30
rup_bg
2.72
2.72
2.86
2.72
de_de
2.39
2.39
2.46
2.39
3.05
2.86
3.09
2.87
el_gr
4.73
4.77
4.73
sd_arab_in
9.22
9.22
9.65
10.08
en_us
9.02
9.11
9.02
2.39
2.39
2.43
2.39
es_419
1.81
1.85
sl_si
4.58
4.58
4.60
4.17
et_ee
2.24
2.28
2.24
sn_zw
9.45
9.45
9.48
9.45
fa_ir
4.96
5.87
4.96
$0_s0
13.73
13.81
13.73
ff_sn
21.22
21.22
21.42
$r_rs
9.93
9.93
9.95
9.95
fi_fi
2.02
2.02
2.05
2.02
sv_se
4.21
4.21
4.30
4.21
fil_ph
3.59
3.59
3.63
3.59
sw_ke
12.62
12.62
12.76
12.62
fr_fr
4.57
4.57
4.63
4.57
ta_in
12.35
11.55
15.06
12.35
ga_ie
29.75
29.75
29.78
29.79
te_in
7.48
7.48
7.56
7.48
gl_es
2.55
2.55
2.58
2.55
tg_tj
4.56
4.56
4.60
4.56
gu_in
5.75
5.75
5.9
5.75
th_th
11.89
11.89
11.92
11.48
ha_ng
7.70
7.70
9.23
6.90
tr_tr
4.28
4.28
4.34
4.28
he_il
18.36
18.36
18.40
18.36
ua_
5.44
5.44
5.47
5.44
hi_in
5.59
3.59
5.63
5.59
umb_ao
17.46
17.09
17.47
14.98
hr_hr
4.46
4.46
4.56
4.46
ur _r_pk
7.61
7.61
7.63
7.61
hu
7.05
7.05
7.10
7.05
UZ
7.40
7.40
7.42
7.40
hy_am
4.93
6.25
4.94
4.93
vi_vn
11.80
11.80
11.83
11.80
id_id
3.14
3.14
3.16
3.14
wn
15.26
15.26
15.29
15.26
ig_ng
14.06
14.06
14.29
14.07
xh_za
16.65
16.65
16.69
16.68
is_is
6.23
6.23
6.25
6.23
yo_ng
19.84
19.84
19.93
19.84
it_it
1.39
1.39
1.44
1.39
zu_za
5.56
5.56
5.62
5.56
ja_jp
25.74
25.49
25.51
25.43
Micro-Average
8.28
8.27
8.40
8.22
jv_id
4.66
4.66
4.72
4.52
ka_ge
10.09
10.09
10.16
10.09
kam_ke
11.74
11.69
11.78
11.74
kea_cv
4.11
4.11
4.17
4.11
kk_kz
3.58
3.58
3.66
3.58
km_kh
20.15
20.15
20.15
20.15
kn_in
5.13
5.13
5.40
5.13
kr_kr
14.29
14:29
14:22
14.29
ky_kg
4.53
4.53
4.56
1.44
lb_lu
13.54
13.54
13.64
13.54
lg_ug
8.99
8.99
9.13
8.99
In_cd
4.61
4.61
4.76
4.61
lo_la
22.80
22.80
22.84
23.25
It_lt
4.51
4.51
4.55
4.51
luo_ke
5.64
5.73
5.64
Iv_lv
2.18
2.21
2.18
mi_nz
9.51
8.68
mk_rk
3.60
3.66
3.60
ml_in
5.45
5.07
mn_m
8.43
8.46
8.43
mr mr_in
7.37
7.37
7.48
7.37
ms_my
3.89
3.89
3.92
3.89
Table 7: ASR tasks evaluated using CER metric at 4K steps of fine-tuning mT5 and ByT5 Small and Base models.

---

Training
Task Code
mT3 Base
ByTS
Small
Large
Small
Base
Large
min
mean
max
context length
28
96
an_Ethi_Latn
16.72
16.20
15.77
10.80
11.61
11.85
target length
32
an_Latn_Ethi
17.61
14.25
13.08
8.58
8.19
8.94
bn_Beng_Latn
14.94
12.78
12.06
8.58
8.70
8.88
Validation set
bn_Latn_Beng
17.07
12.90
11 11.26
5.67
5.44
6.13
min
max
gu_Gujr_Late
15.58
14.19
12.37
gu_Latn__uujr
10.49
7.92
6.94
2.86
2.83
3.21
context length
1
28
86
hi hi_Deva_Late
10.59
9.63
9.49
7.24
7.71
7.70
target length
5
23
hi_Latn_Dest
12.09
9.11
8.08
5.00
4.37
4.98
kn_Knda_Latmm
11.35
8.05
4.37
4.53
Test set
kn_Latn_Knda
14.28
10.95
9.57
3.72
3.62
3.88
min
mean
m1 ol_Latn_Miym
21.10
16.41
14.25
5.66
5.31
5.95
context length
1
28
86
mI_ ml_Miym_Latn
15.54
13:27
12:29
8.18
8.43
8.46
larget length
5
28
mr mr_Deva_Latn
11.40
9.70
9.25
5.77
6.04
6.07
mr__aannDeva
13.68
10.92
4.94
4.73
5.40
pa_Arab_Guru
13.53
11.49
10.99
7.67
7.72
8.14
Table 8: Context and target character length statistics averaged over the 23 languages of autocomplete.
pa_Arab_Latn
21.89
19.62
19.44
16.03
16.03
16.59
pa_Guru_Arab
14.10
13.82
10.76
11.06
11.32
pa_Guru_Lath
15.81
15.21
15.13
12.36
12.78
pa_Latn_Arab
19.55
17.91
16.96
12,90
13.06
13.77
pa_Latn_Guru
12.78
10.38
9.42
5.47
5.11
5.72
sd_Arab_Latn
18.27
17.24
17.03
14.20
14.66
14.45
Acc@3
chrF
sd_Latn_Arab
14.00
8.65
9.98
6.70
Language
mT5
ByT5
mT5
ByT5
si_Latn_Sinh
14.79
12.34
11:31
6.06
5.78
6.24
si_Sinh_Latn
10.76
9.43
9.06
6.36
6.79
6.62
be
12.4
8.99
22.12
26.02
ta_Latn_Taml
18.92
14.57
12.89
5.15
4.70
5.06
bg
14.45
15.38
23.27
29.92
ta_Taml_Latn
15.83
13.01
12.58
8.35
9.01
da
13.36
26.95
22.33
35.03
Te_Latn_Tellu
18.16
13.60
11.70
5.01
4.82
5.27
el
21.74
14.91
25.3
32.81
te_Telu_Latn
14.50
11.26
10.51
7.25
7.30
7.31
19.2
40.1
23.08
39.84
ur_Arab_Latn
16.73
15.01
14.28
11.88
11.58
12.17
23.2
20.3
31.85
ur_Latn_Arab
13.01
10.07
8.30
5.23
5.59
22
26.36
34.52
Average
7.92
8.16
ga
11.8
23.06
22.08
30.66
gd
26.28
31.85
32.79
37.65
Table 11: Transliteration tasks evaluated using CER metric 10K steps of fine-tuning the mT5 and ByT5 models in Small, Base and Large configurations.
g1
19
41.4
26.53
45.76
he
13.87
18.76
23.25
hy
3.49
14.98
24.4
18.59
37.92
28.67
41.34
is
26.37
32.45
27.96
32.39
mT5
ByT5
8.41
21.03
21.44
31.39
lv
9.6
19
22.72
31.87
Language
base
large
base
large
pcm
30.3
36.42 42
31.32
39
am
20.01
26.17
33.41
25.60
ro
11.9
21.4
22.79
32.39
be
27.82
37.52
46.72
37.36
sk
17.49
26.23
24.56
33.75
bn
29.06
37.66
45.07
35.69
s1
14.1
25.6
23.54
33.41
de
33.71
39.96
45.34
37.93
ug
3.41
0.23
15.72
14.88
de (loc)
33.31
40.58
45.81
38,20
uk
11.3
13.04
23.36
29.78
34.09
40.39
49.50
39.52
ur 21.96
23.33
23.57
29.22
34.95
41.52
48 73
39.29
Average
14.85
22.84
23.63
32.22
fi
26.63
35.74
46.80
37.17
fr
33.72
40.29
48.97
39.84
Table 9: mT5 and ByT5 performance averaged on the languages test sets after 10 epochs of fine-tuning on ablation dataset.
ha
21.84
27.60
42.07
29.98
hi
27.89
37.59
42.26
35.42
hu
25.87
33.47
43.82
35.98
ja
28.71
33.68
45.23
35.90
pt_br 33.98
39.12
47.90
39.50
ru 34.44
41.36
48.58
42.80
Sw
24.06
30.25
39.96
32.09
Lang.
Tasks
Lang.
Tasks
ta
25.03
33.20
43.31
31.41
th
23.81
34.35
43.80
35.30
Ethin
GuruesLatn
tr
27.44
36.41
44.58
36.63
Beng++Latn
pa
Arabe>Latn
yo 14.52
16.30
30.39
18.44
gu
Gujr+>Latn
GuruesArab
zu 18.73
26.79
36.96
27.49
Devat>Latn
ArabesLatn
Kndar+LLtn
si Sinhe>Latn
Average
27.6
34.78
43.77
34.84
Mlyme>Latn
ta TamlesLatn
Devar-LLatn te Telux>Latn
Table 12: Semantic Parsing: Exact Match (EM) accura- cies of mT5 and ByT5 models of different sizes trained multilingually on few-shot data. We report accuracies on all languages.
ur Arabe-Latn
Table 10: Summary of the tramsliteration tasks.

---

mT5
By
Language
base
large
large
bn
10.72
11.78
22.69
16.25
hi
16.05
18.48
25.03
17.69
16.98
19.71
26.21
19.27
Table 13: Semantic Parsing: Exact Match (EM) ac- curacies of mT5 and ByT5 models of different sizes trained multilingually on few-shot data. Here the multi- lingual training data includes three code-switched Indic languages and we report EM for such languages.
Task
In-context leamning example
Translate between English and Afrikuans.
Translation
English: [INPUT]
Afrikaans: [TARGET]]
Correct the ASR output in Afrikaans.
ASR
ASR Afrikaans output: [INPUT]
Corrected: [TARGET]
Tag the named entities the Swahili text person (PER), organization (ORG), Jocation (LOC), and date (DATE). $$
NER
delimiter.
Swahili text: [INPUT]
Named entities: [TARGET]
Complete Urdu sentence. Write the
word or finish last word is incomplete.
Autocomplete
Urdu sentence: [INPUT]
Completion: [TARGET]
Table 14: In-context learning examples.


Reviews:
[{"review_id": "LLuJen3w4P", "reviewer": "Reviewer_jMj2", "summary": "", "questions": "1.What is the point of adding tasks and datasets directly from other sources directly to Up? I am referring to the tasks like ASR and MT.\n\n2.How did you process the original MasakhaNER datasets? In appendix, you mentioned that \"we process the data in order to align the token-level annotations with byte-level spans in the original pre-tokenized text.\" How did you do that?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and for taking the time to review this work\n\n> 1.Including some datasets directly from other sources unnecessariy like ASR, MT\n\nWe note that UP is primarily a benchmark (like SuperGLUE, XTREME, etc.); to accomplish its purpose of evaluating coverage of well-established language technologies in under-represented languages, we view the inclusion of these tasks as necessary. We included the best available resources we could find where possible, creating new resources where needed. That isBoth ASR and MT are such key technologies that have a wide range of use cases and make information much more accessible, which is crucial for under-represented languages. For both tasks, existing datasets already cover a large number of under-represented languages so we chose to prioritize tasks with sparser language coverage in our data collection efforts.\n\n> Statistics of the datasets based on the categories, coverage of the tasks from the categories, presenting results by categories is missing\n\nYou can already view the language coverage of each task with each language’s resource level in Table 5 in the Appendix. We will add summary statistics per resource level to make inspecting this table easier. We will also present results per resource level to provide a more fine-grained breakdown. \n\n> 1.What is the point of adding tasks and datasets directly from other sources directly to UP?\n\nAs mentioned above, the inclusion of MT and ASR is important for a user-centric evaluation and both tasks already have datasets with a large language coverage. We would also like to highlight that using existing datasets has been a common practice for prior benchmarks (see GLUE, SuperGLUE, XTREME, BUFFET, SUPERB, etc). We go significantly beyond prior benchmarks by creating new datasets for a wide range of applications. \n\n> 2.How did you process the original MasakhaNER datasets?\n\nWe reproduce the preprocessing of the MasakhaNER datasets, which we re-apply to the original unlabeled source data. We then match each sentence in the unlabeled source data with its annotated and tokenized counterpart based on regular expressions and string matching. After finding a match, we map the annotated token-level spans to their corresponding spans in the original source text via regex matching. This automatic process finds matches for around 98% of cases. We resolve instances where no match is found through manual inspection and by consulting with the original dataset creators.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper, includes: \n\n1. Large-scale evaluation: The UP benchmark evaluates the capabilites of language models accross 88 under-represented languages over 9 tasks. \n\n2. New datasets: Creation of new datasets for OCR, auto-complete, question answering, semantic parsing, and transliteration", "reasons_to_reject": "The limitations includes,\n\n1.Including some datasets directly from other sources unnecessariy like Automatic speech recognition, Machine Translation. I do not get the merit of adding others datasets in UP without any contribution.\n\n2.Following Joshi et al. (2020) they select languages in categories 1–3 (Line#132-135). However, there is no mention of these categories later. Catergory#1 is much low-resourced compared to category#3. Statistics of the datasets based on the categories, coverage of the tasks from the categories, presenting results by category-wise instead of just averaging would be more insightful, which are missing.\n", "questions_to_authors": "1.What is the point of adding tasks and datasets directly from other sources directly to Up? I am referring to the tasks like ASR and MT.\n\n2.How did you process the original MasakhaNER datasets? In appendix, you mentioned that \"we process the data in order to align the token-level annotations with byte-level spans in the original pre-tokenized text.\" How did you do that?", "paper_topic_and_main_contributions": "This paper presents the UP benchmark, which is a user-centric scarce-data benchmark for under-represented languages. The main problem addressed by this paper is the lack of resources and evaluation benchmarks for under-represented languages, which makes it difficult to develop and evaluate natural language processing (NLP) models for these languages. The UP benchmark aims to fill this gap by providing a standardized evaluation framework for NLP models across nine tasks. \n\nThe main contributions of this paper are: \n\n1. The development of a multilingual few-example benchmark. \n2. Newly created data for five tasks.\n3. Baseline results.\n\nOverall, this paper makes contributions towards addressing the lack of resources and evaluation benchmarks for under-represented languages, and provides resources for researchers and practitioners working in this area.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "df83FPVVGW", "reviewer": "Reviewer_tT4X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback. We will aim to incorporate the suggestions and to clarify the under-specified settings. \n\n> 1. The notion of a user-centric task is not so clear\n\nThank you for the suggestion. We would like to highlight that quantifying the overall usage or impact an NLP technology has is not straightforward. Jin et al. (2021; https://aclanthology.org/2021.findings-acl.273/) propose to measure the impact of applicable tools (e.g., MT) via the impact of Stage-4 technologies that build on them, such as Google Translate, Amazon Echo, etc., but do not provide any actual numbers. Blasi et al. (2022; https://aclanthology.org/2022.acl-long.376) similarly study “tasks that technology users interact with directly in their everyday life” including QA and MT but without any further quantification of the user base of each task.\nAs information of all products that use a certain NLP technology is not available, we could use an estimate of the user base of the most widely used product where such information is available as a proxy. Note that also these numbers should be taken with a grain of salt. For MT, Google Translate has 1B users (https://blog.google/products/translate/one-billion-installs/). For QA and retrieval, Google Search processes 3.5B searches per day (https://www.internetlivestats.com/google-search-statistics/) with 1B+ users. We will attempt to provide a similar estimate of the usage of the other tasks in UP.\n\n> 2. The estimation of annotation effort, which is central to the paper, is underspecified\n\nWe quantify and compare the annotation costs for each task in Table 2. We will provide further details regarding the specific measurements in a revised version of the paper. Regarding the selection of 8 hours as annotation time: We sought to select a realistic amount of annotation time that can be easily scaled up to extend to additional languages while still keeping the setting challenging for future research. In addition, some of the employed datasets do not provide much more than 8 hours of data per language, which limits the timeframes that can be used in a general setting.\n\n> 3. The UP benchmark is very rich in the coverage of tasks, but the description of the generation of datasets for each of these tasks is very small\n\nWe had to limit the information on the creation of the new datasets due to space limitations in the submission. We will provide additional information on the data annotation of the new datasets in the revised version; many of these details are currently in the appendix and could be moved forward given additional space\n\n> 4. The paper focuses on few-shot learning and mentions the sensitivity of prompt design, but looks at single instructions per task and leaves the search for better designs to future work\n\nWe use English as the prompt language, which has been shown to be the best source language for prompting current generative models in prior work (Winata et al., 2021, https://aclanthology.org/2021.mrl-1.1/; Lin et al., 2022, https://aclanthology.org/2022.emnlp-main.616/; Shi et al., 2023, https://arxiv.org/abs/2210.03057). These prior works similarly use a single instruction per task. This instruction may be machine-translated to other languages. Given the poor performance of MT for many of the languages in our study, we do not observe good performance with a machine-translated instruction. For this reason, we focus on the best-performing setting in accordance with prior work.\n\n> 5. The evaluation in section 4 is not very helpful in understanding the usefulness of the benchmark\n\nThe fact that our models are fine-tuned in a joint multilingual setting means that while the data for each language may be low given the 8-hour annotation timeframe, we would expect that models that have learned strong multilingual representations would be able to benefit from positive cross-lingual transfer. In addition, recent LLMs have demonstrated exceptional performance even in 1-shot or 0-shot settings on many tasks; in contrast to this, we would expect much better performance even with 8 hours of annotated data.\n\nOur results clearly demonstrate that byte-based modeling is particularly important for under-represented languages with rich morphology while fine-tuning even on limited data outperforms in-context learning. We will provide further analysis regarding the performance of models across different resource groups in order to show a more differentiated picture of model performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a new benchmark with a special focus on underrepresented languages. This is very important because data for these languages is rare, while it is very important to develop NLP systems for all these languages. In this way, the benchmark can help to encourage such developments and to shift the focus from high-resource languages to low-resource languages. \n2. The paper develops its benchmark for a wide range of different tasks, using a lot of data, augmenting it or generating new data (but with a focus on a certain anotation time limit). This broadens the applicability of the benchmark and makes it more likely to find followers who use this dataset for their NLP developments.\n3. Successes in NLP and language modeling are probably due to the availability of sufficient data. It is interesting to see how this paper focuses on this task under its special time constraint (eight hours). More research is needed on what data is needed to at least provide a valuable resource that can be used for research on underrepresented languages, and how this is affected by annotation time limits. The paper provides some initial insights into this scenario. \n", "reasons_to_reject": "1. The notion of a user-centric task (something like 'We focus on widely used, user-oriented tasks that benefit speakers of high-resource languages. resource languages') is not so clear. The authors presuppose a definition; a quantification would be more valid here.\n2. The estimation of annotation effort, which is central to the paper, is underspecified: it is based on previous work and the authors' own 'annotation effort'. However, both of these reference points are not quantified, and especially not in a comparative way that would allow measurement in an inter-annotator estimation scenario. Here I would expect a more experimental setup to measure this important time, and I would also expect a deeper justification for the eight hours, which is certainly not that much, but why this time limit - just because it works sufficiently?\n3. The UP benchmark is very rich in the coverage of tasks, but the description of the generation of datasets for each of these tasks is very small, so that a larger set of annotation efforts and data additions are noted, which would require more details to understand what was actually done. Each of these additions requires annotations (see, for example, lines 402-408), which in turn need to be evaluated in order to know something about the quality of the generated data and data additions. However, such evaluations (e.g. IAA coefficients) are not provided. Thus, the description of the tasks in Section 3 is manifold, but lacks a bit more information that would help to assess the success of the authors more reliably. This is all the more striking as the evaluation of the models in section 4 is not described as central (lines 452-454): apparently the evaluation of the data set extensions would have been more central. As it is, Section 3 is a very nice enumeration of selected tasks, but it is difficult for me to judge the exact results of data generation. The appendices could be more helpful in this respect too.\n4. The paper focuses on few-shot learning and mentions the sensitivity of prompt design, but looks at single instructions per task and leaves the search for better designs to future work: this is a way around the underlying problem, but not a solution. Because of the centrality of few-shot learning to the paper, as opposed to zero-shot learning, one would expect a more sophisticated technique at this point. \n5. The evaluation in section 4 is not very helpful in understanding the usefulness of the benchmark: is it just to say that there is now a new set of datasets with which we can show that some standard models perform poorly when trained with the \"scarce\" data from UP for some tasks - this is not so surprising since this data is by definition limited by the 8 hour annotation time (which is remarkably low). Or do we learn more about what it helps to use the UP data to improve systems that would otherwise not perform as well as they do when using this data from UP? Here the description is a bit short and does not fit perfectly into the overall picture of the paper. This has also to be criticized in the context of the lack of information on the annotation effort for each resource: if I don't know the quality of the 8-hour limited training splits, how do I know how fine-tuning based on this data affects the benchmarking of existing models? Again, the paper can be more informative in this respect. The discussion/analysis in Section 5 is also not so informative in this respect.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper develops a benchmark called UP to support NLP for underrepresented languages. More specifically, it focuses on the evaluation of multilingual models in a few-shot setting (rather than a zero-shot setting) with respect to so-called user-centric tasks. These are tasks that users may encounter in their daily lives in the area of computer use. This concerns information access or the linking of different technologies via their data. The benchmark includes data for five tasks (\"QA, OCR, autocomplete, semantic parsing and sentence-level transliteration\"), two additional task setups (NER and retrieval), experimental setups focusing on ASR and MT, and baseline results contrasting subword and byte-based models. The focus of the benchmark is on scarce data rather than small data in a multilingual setting, focusing on under-represented languages and specific (\"user-centric\") tasks. In this way, it contrasts with related work. A key reference point for the development of the benchmark is the annotation cost. This relates to the amount of data that can be annotated in a given period of time (8 hours): it defines a lower boundary of the amount of annotation data to be considered for further processing. A second reference point is the efficiency in a multilingual computing scenario.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- \"UP provides methodology\" -> \"UP provides a methodology\"\n- \"QA Clark et al. (2020)\" -> \"QA (Clark et al. 2020)\"\n- \"We focus on widely adopted user-facing tasks beneﬁting speakers of high-resource languages\" -> \"We focus on widely used, user-oriented tasks that benefit speakers of high-resource languages\"\n- \"UP does not limit supervised training data in high-resource languages (HLs) while each under-represented language (UL) has a maximum of 8 hours of annotation effort in its training split;\" -> \"UP does not limit supervised training data in High Resource Languages (HLs), while a maximum of 8 hours of annotation effort is provided for each training split for each Underrepresented Language (UL);\" - I am unsure about the quantification in the sentence; I make a proposal that is better understandable for me; here the authors should check their wording. It is also unclear that UP is limiting HLs while only  evaluating ULs.\n- \"In this way, we design for the task ﬁrst. \" -> This sentence does not make sense for me. Do you mean \"In this way, we first design the task.\"? \n- \"that aims ﬁll gaps\" -> \"that aims to ﬁll gaps\"\n"}}, {"review_id": "6zdbWHVKSm", "reviewer": "Reviewer_qsF6", "summary": "", "questions": "You can see the Reasons To Reject section.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review.\n\n> 1. The choice and the number of baselines is not justified.\n\nWe select three representative methods, each from a different model family: one subword-based method, one byte-based method, and one in-context learning method. We focused on optimizing the experimental settings for the open-source models so that they can be easily reproduced (rather than evaluating a larger number of under-tuned methods).\n\n> 2. The experimental setting is a bit flawed in my opinion (from a literature perspective). The setups for ASR is not what is generally used. A lot of the tasks like NER and QA is not generally not treated as a sequence-to-sequence task commonly and more of a token classification task.\n\nWe included multi-modal tasks such as ASR and OCR as they are widely used technologies and are thus important to be made available in a broader set of languages. To make them accessible for the current text-only models and to enable NLP researchers to improve performance for certain languages while staying within the text modality, we provide intermediate outputs of ASR and OCR systems. This post-correction setting has been studied in prior work (e.g., Dutta et al., 2022, https://arxiv.org/abs/2202.01157;  Rijhwani et al., 2020, https://aclanthology.org/2020.emnlp-main.478/). Nevertheless, we expect multi-modal models to be available soon that can tackle these and the other tasks in UP in an end-to-end fashion.\n\nNER and QA have been treated as token classification tasks with the previous generation of encoder-only language models such as BERT. With the recent generation of decoder-only and encoder-decoder generative LLMs, a sequence-to-sequence encoding has become much more commonplace, even for traditional NLP tasks such as NER. We highlight a few studies here that study these tasks in a seq2seq setting. For QA: Khahabi et al. (2020; https://aclanthology.org/2020.findings-emnlp.171/); Asai et al. (2021; https://arxiv.org/abs/2107.11976). For NER: Yan et al. (2021; https://aclanthology.org/2021.acl-long.451/); Lu et al. (2022; https://aclanthology.org/2022.acl-long.395/); Zhang et al. (2022; https://aclanthology.org/2022.acl-long.59/); Shen et al. (2023; https://aclanthology.org/2023.acl-long.698/). In addition, all studies we are aware of that evaluate recent state-of-the-art models such as GPT-3, ChatGPT, GPT-4, PaLM, PaLM 2, LlaMa, and LlaMa-2 follow a sequence-to-sequence format. For these reasons, we adopt the sequence-to-sequence format as the unified format across tasks. We note that datasets can be easily converted back to their standard formats (e.g., span extraction, sequence labeling) to evaluate encoder-only models if desired.\n\n> 3. The analyses section needs more in-depth intuitions.\n\nWe observe a correlation between the amount of data of a language during pre-training and a model’s downstream performance on that language. For instance, there is little data available in African languages in common pre-training sources and performance is consequently low for African languages. Similarly, we observe a correlation with the resource category of a language where performance is lower the lower-resourced a language is overall. Finally, we observe the script to be a major factor (see, for instance, the analysis on the impact of the script on the Auto-complete task in the appendix). In particular, performance is generally higher for languages which employ a Latin script due to additional lexical overlap. We highlight additional studies that perform similar analyses on a more limited set of languages: Rust et al. (2021; https://aclanthology.org/2021.acl-long.243/); Hu et al. (2020; https://arxiv.org/abs/2003.11080); Adelani et al. (2022; https://aclanthology.org/2022.emnlp-main.298/). We will extend the analysis justification and add additional justifications in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Well-written benchmark paper. Benchmark collection rationale and process has been clearly explained. The diagrams and the tables are self explanatory.\n2) Important topic of UL languages.\n3) New dataset will benefit the community.\n4) Benchmark will serve as a good contribution for evaluation of multi-lingual LLMs.", "reasons_to_reject": "1) The choice and the number of baselines is not justified. I understand when the authors say: \"Given that our focus in this paper is on\nthe dataset and task setup rather than system building, we do not focus on offering novel modeling types nor do we exhaustively evaluate all possible models;\". Though I agree that proposing a novel method is beyond scope, but just mT5 and ByT5 is concerning.\n\n2) The experimental setting is a bit flawed in my opinion (from a literature perspective). The setups for ASR is not what is generally used. A lot of the tasks like NER and QA is not generally not treated as a sequence-to-sequence task commonly and more of a token classification task. Again, I am aware of papers that do treat it as a seq-to-seq task, but it is not the traditional method to evaluate these tasks.\n\n3) The analyses section needs more in-depth intuitions. Why do models suffer in African languages? What is the reason for low vs high performance on some languages? Is there a previous reference to the reason atleast?", "questions_to_authors": "You can see the Reasons To Reject section.", "paper_topic_and_main_contributions": "The paper proposes a benchmark for data-scarce, user-centric and under-represented languages. It evaluates the capability of language models across 88 under-represented languages over 9 tasks. Additionally, several novel datasets have been proposed.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this large-scale evaluation benchmark, which includes data and tasks for many under-resourced languages, will be useful for the multilingual NLP community as the current data and task coverage for these languages are scarce. In the rebuttal, it is made clearer what the notion of user-centric means (i.e., downstream tasks with a large number of end users), which justifies the selection of tasks including ASR and MT in the benchmark. The experimental setup, including the transformation of tasks into a seq-to-seq format and the few-shot prompting method, is also based on the recent standard practice of prompting language models. While one point of contention is about the choice of baselines, given the task format and the inclusion of multilingual models of different scales and different tokenization scheme, I believe the setup is sufficient. The authors also make salient their exact contributions to existing datasets by providing the table breakdown of their data contributions during the post-rebuttal discussion period, which hopefully will be included in the next revised version of the paper. One common weakness of the current paper raised by reviewers is the lack of in-depth analysis in Section 5; for instance, how the quality of the training splits of collected data (in a constrained setting of 8 hours) impacts the benchmarking evaluation and what is the reason for the poor performance of African languages.
Decision: Accept-Findings