GlobalBench: A Benchmark for Global Progress in Natural Language Processing
Authors: Yueqi Song
Simran Khanuja
Pengfei Liu
Fahim Faisal
Alissa Ostapenko
Genta Indra Winata
Alham Fikri Aji
Samuel Cahyawijaya
Yulia Tsvetkov
Antonios Anastasopoulos
Graham Neubig
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual benchmark
Leaderboard

Abstract: Despite the major advances in NLP, significant disparities in NLP system performance across languages still exist. Arguably, these are due to uneven resource allocation and sub-optimal incentives to work on less resourced languages. To track and further incentivize the global development of equitable language technology, we introduce GlobalBench. Prior multilingual benchmarks are static and have focused on a limited number of tasks and languages. In contrast, GlobalBench is an ever-expanding collection that aims to dynamically track progress on all NLP datasets in all languages. Rather than solely measuring accuracy, GlobalBench also tracks the estimated per-speaker utility and equity of technology across all languages, providing a multi-faceted view of how language technology is serving people of the world. Furthermore, GlobalBench is designed to identify the most under-served languages, and rewards research efforts directed towards those languages. At present, the most under-served languages are the ones with a relatively high population, but nonetheless overlooked by composite multilingual benchmarks (like Punjabi, Portuguese, and Wu Chinese). Currently, GlobalBench covers 966 datasets in 190 languages, and has 1,128 system submissions spanning 62 languages.

GlobalBench: A Benchmark for Global Progress in Natural Language Processing
Yueqi Song', Catherine Cui', Simran Khanuja', Pengfei Liu', Fahim Faisal',
Alissa Ostapenko', Genta Indra Winata Alham Fikri Aji', Samuel Cahyawijaya', Yulia Tsvetkov 1.7 Antonios A Anastasopoulos", Graham Neubig'
CCareegie Mellon University "Harvard University SShanghai Jiaotong University Thhe Hong Kong University of Science and Technology MBZUAI
*George Mason University
"University of Washington
Abstract
All System Results
Despite the major advances NI.P, significant disparities system performance across lan- guages still exist. Arguably. these are due to un- even resource allocation and sub-optimal incen- tives to work on less resourced languages. track and further incentivize the global develop- ment equitable language technology. we in- troduce GlobalBench. Prior multilingual bench- marks are static and have focused on a limited number of tasks and languages. In contrast, Glob- alBench is an ever-expanding collection that aims to dynamically track progress on all NLP datasets in all languages. Rather than solely measuring ac- curacy, GlobalBench also tracks the estimated per- speaker unility and equity of technology across all languages, providing a multi-faccted view of how language technology is serving people of the world. Furthermore, GlobalBench is designed to identify the most under-served languages, and rewards research efforts directed towards those languages. At present, the most seree- lan- guages are the ones with relatively high pop- ulation, but nonetheless overlooked by compos- ite multilingual benchmarks (like Punjabi, Por- tuguese, and Wu Chinese). Currently, Global- Bench covers 966 datasets in 190 languages and includes 1,128 system submissions spanning 62 languages.'
MT
NER
MT
MT
MT eng
NER NER NER
cmn
spa
can
spa
eng
GlobalBench MT
GlobalBench NER
(other tasks)
aee Equity
Demo.
Ling.
Equity
Demo. Avg.
Ling.
Avg.
Avg.
Avg.
GlobalBench
GlobalBench NER
Figure I: GlobalBench Design: A leaderboard for each task is separately maintained. Each leaderboard con- tains a multi-faceted evaluation of submitted systems, along with ranking of the most under-served languages. More details can be found in Section 2.
(as measured by GDP of the countries where they are spoken) see more research and resource devel- opment, leading to more performant systems.
1 Introduction
Advances in multilingual natural language process- ing (NLP) technologies (Dabre et al., 2020; Hed- derich et al., 2021) have raised the enticing possibil- ities of NLP systems that benefit all people around the world. However, at the same time, studies into the state of multilingual NLP have demonstrated stark differences in the amount of resources avail- able (Joshi et al., 2020; Yu et al., 2022) and perfor- mance of existing NLP systems (Blasi al., 2022; Khanuja et al., 2023; Ahia et al., 2023).
In this paper, we propose GlobalBench, a new benchmark and leaderboard that is designed to specifically incentivize the global development of equitable language technologies that serve speak- ers of all languages throughout the world. Glob- alBench follows the footsteps of other successful multilingual benchmarks such as XTREME (Hu et al., 2020) and XGLUE (Liang et al., 2020), which aggregate results of systems across several tasks to provide a general idea of progress being made in the field of multilingual NLP. However, these benchmarks, by design, are static and lack the goal be an all-inclusive, ever-expanding collec- tion of datasets. Additionally, they mainly focus on average accuracy over all languages in the dataset, and thus say little about the downstream utility and
Why do these disparities exist? The causes of these disparities are multifarious, but Blasi et al. (2022) argue that one major factor is a problem of incentives and resource allocation. For instance, languages associated with larger economic might
Globbalennh is available at neulab/globallbench.

---

Inclusive Dataset Selection: We aim to be able evaluate all datasets in all languages for all tasks, making it possible (in theory) cover any language for which a dataset exists. Pillar 1: Dataset Inclusivity
Pillar 2: Multi-Faccteed Evaluation
Pillar Reward Improvement
960 datasets in 190 languages
1,128 system submissions
Dataset & model contributions
Change Rankings of Most Under-Served Languages
Tasks
Datasets
Languages
Systems
Languages
Metrics
Datasct
System,
1 and 2
NER (before)
NER (after)
Accuracy/ F1 / BLEU chrf/-. /..
NER
Dataset
2
System,
2 and 1
Dataset,
3
Utility
L
Equity
Tasks: tasks supported by GlobalBench. Datasets: datasets submitted NER task. Languages; languages datasets,
Systems: Systems submitted for task.
Submitting new datasets new systems may change the rankings most under-served languages.
Languages: languages supported by each system.
Metrics; Metrics to evaluate systems.
GlobalBench aims to be all-inclusive with respect to tasks, datasets, and languages.
GlobalBench provides factt-f view of how well language technologies serve people worldwide.
GlobalBench ranks most under- served languages for each task, encouraging improvements.
Figure 2: GlobalBenck's 's Philosophy: First, we aim to inclusively gather datasets for all tasks and languages. Second, we present multi-faceted evaluation of systems, going beyond average accuracies across languages, keep track of the utility and equity of these systems. Third, GlobalBench ranks the most to least under-served languages on the leaderboard of each task, and rewards improvement in utility, which can be achieved through both dataset and model contributions. The lower the ranking of language the hetter since lower ranking indicates it is less under-served. In the above example, the addition of Datasetz improves the measured utility of L3 in the NER leaderboard, and the addition of Systema improves the measured utility of L2 and L3 for the NER leaderboard.
equity of submitted systems across languages. Hence, in designing GlobalBench, we make a number of intentional design decisions to explicitly promote the improvement of language technology for all of the world's citizens:
In the remainder of this paper, we detail Glob- alBench's 's design principles ($2), how interested research community members can participate ($3), the currently covered tasks and systems ($4), anal- ysis of the current state of NLP viewed through the lens of GlobalBench ($5), related work ($6) and our expectations for the path forward ($7).
All in all, we believe that improving the quality and equity of language technologies for all speak- ers in the world is one of the paramount challenges of NLP today. In the famous mantra from Peter Drucker, what you cannot measure, you cannot im- prove; GlobalBench is first step in this direction.
Multi-Faceted Evaluation: As shown in Fig- ure 1, GlobalBench explicitly considers per- speaker utility and equity ($2), measuring how close NLP systems have come to equitably covering all speakers in the world, instead of just those in our existing data.
GlobalBench Design Principles
Philosophy: A working example of our guiding philosophy is shown in Figure 2. Our unique re- ward system incentivizes model builders to not only improve system performance, but also build datasets for new languages. To illustrate the for- mer, let's assume that there were NER datasets for L2 and La on GlobalBench, and now researchers build new system for NER (System2) which is Mfte-of-the-ar for languages Lz and Lg- This
Reward Data and Model Contributions:
GlobalBench encourages improvements in the state-of-the-ait (instead of just measuring the state-of-ther thee-of- itself), by identifying under- served languages and rewarding progress on them, both in terms dataset and model con- tributions.

---

increases utility for both languages (Equation 3), which is attributed to System2 on our NER leader- board. For the latter, let's change our assumption to that there was no NER dataset for L3 before, but a pre-trained model (System²) supports this language. Hence, the introduction of Datasets helps realize a sharp improvement in utility for Lg which was previously immeasurable (hence, for all practical purposes, zero). Thus, the increase in util- ity for L3 on the NER leaderboard is attributed to Datasetz. Additionally, we rank the most to least under-served languages. To illustrate how this rank- ing helps, let's consider the case for NER in Figure 2. Before the introduction of Dalasels. L3 was most under-served, followed by 11. After inclusion of Dalaselz in the leaderboard, the measured util- ity for Ly increases. Now, even though the utility value of L1 remains unchanged. L1 might become the most under-served language following the in- crease in utility of L3. This would act as a positive feedback for the community to direct their efforts towards languages needing most work (here, Lt, specified by the most under-served languages rankings), and drive progress for global equity in a positive cause-effect feedback loop.
managing include the majority of the about 7000 spoken languages around the world (Austin and Sallabank, 20 At present, named entity recogni- tion is the task with the highest coverage of world speaker population (59.34%), but GlobalBench hopes to continually evolve with time.
2.2 Multi-Faceted Evaluation: Utility and Equity
Utility Blasi et al. (2022) introduce utility m of system for task and language to be its per- formance normalized by the best possible perfor- mance (typically, human-level performance, but if it's unattainable, we use the empirical maximum as an estimate) afforded by the task:
performance,
Ul theoretical max performance
While the above helps estimate system perfor- mance relative the ideal scenario, the final utility provided also depends the systems' demand, which is the second term used by Blasi al. (2022) in their analysis. Demand d is characterized by tak- ing into consideration demographic and linguistic perspectives. Under the demographic perspective, the demand for technology in a language is esti- mated to be proportional to the number of speakers of the language itself (dl x m)). Under the lin- guistic perspective, the demand across languages is identical (dj < 1). These two alternatives, well as any intermediate combination of them, are parameterized through a single exponent r:
Design: GlobalBench maintains separate leader- board for cach of the covered tasks, as shown in Figure 1. Each leaderboard details the con- stituent datasets, system submissions and the fol- lowing evaluation metrics (further details in $2.2 and $2.3): a) Performance (F1, accuracy, BLEU, etc.); b) System-by-Systee Urility (linguistic and demographic); c) Global Average Utility (linguistic and demographic); d) Equity; e) Most under-served Languages; f) Score by Language. For more details about GlobalBench UI, please refer to SA.3.
n
(2)
Ere. nf
where 5 = corresponds to a demographic notion of demand and = 0 to linguistic one. Using the above, Blasi et al. (2022) define a global metric follows:
2.1 Dataset Selection: Inclusivity
The first pillar of GlobalBench's 's approach is that we attempt to be all-inclusive with respect to tasks, datasets, and languages. On the dataset front, Glob- alBench currently includes 966 datasets spanning 190 languages. On the modeling front, it has 1.128 system outputs spanning 6 NLP tasks and 62 languages (note that not every dataset integrated within GlobalBench has system outputs submitted the leaderboard at present). Overall, Global- Bench has support to accept dataset and system submissions for 17 NLP tasks in 6671 languages?,
M, = Σ alt) uj lLL
(3)
In essence, M, - 0 means that no user bene- fits from language technology and M, = 1 cor- responds to cach language user enjoying perfect technology.
In GlobalBench, we provide the demographic weighted (T= 1) and the linguistic weighted (T = 0) utilities for all languages in each task. For each language, we take the maximum utility scores
"The source of metadata of these languages is "Ethnologue: Languages of the World."

---

Text Clasiiiicaionn
Text Clasifiction
This
Positive, Negative, Neutral
Named Eatity Recognition
Amy was born in 2020 #> AmyyLennonn) Person] was born in 2020trime).
Sequence Labeling
Word Sogmentation
8 77 15 I
Chanking
Amy will close the window => [NP Amy| [VP will clase] [NP the window]
Multiple Choice
The capiral of France
Paris, b) London, c) Rome, di Madrid
Generative
The capiral of France
+> The capiral France Paris.
Classficattio
Pair Classfication
51: Amy doesn't live for away from me; 52: Amy lives close me Entaliment
Span Text Classiecation
Aspoct-based Sentiment
happy the delivery, dissarissled with product delivery
Text Editing
Grammatical Error Corr.
There is apple. => There is an apple.
Extractive QA
Text: "Any and Bob formed a group", " Qu How mony people are in the group? -p [Text abovej Q: How many people
Two.
Quaston Answring
Multiple QA
group? a) 1,
Open Domain
Q: What is the capitai of Fronce? -o A. Paris.
Machine Trandatioo
[Prench Japanese) Bonjour => Blaik
Couditional Gencration
Summmrtirttion
Story 'Beauty the Beast" Summory "Beauty the Beast"
Code Generation
Tharate through words of file Pyltion words opann"myllle)eeooddd-
Knowledge Graph Prodiction
KG link tail predlcction
Given the true head of a KG, predict the tail entify of the KG.
Language Modeling
Language Modeling
weothrr _----
beautiful
Figure 3: Overview of Tasks: All tasks currently supported by GlobalBench are shown above. This is expected to constantly evolve with time. Refer to Section 4 for details.
of all systems submitted to GlobalBench. To ob- tain task global averages, as shown in Table 2, we average across utility values for all languages.
the system performance to be zero. The global equity values for each task are in Table 2.
Apart from the above, we keep track of system performances (FV/Accurayy/BLEU etc.). For each task, we take the system output with the highest performance among all system outputs with the same language, and provide a ranking of languages with the highest system performances. We also maintain ranking of most under-served languages (sorted in increasing order of utility), for reasons detailed below.
Equity While utility paints the picture of how far from ideal we are in serving NLP technol- ogy each user, equity helps measure how uni- form the technologies we serve are, across lan- guages. Khanuja al. (2023) recently proposed that amongst other measures of statistical disper- sion, the Gini coefficient (Dorfman, 1979) best captures this uniformity. Hence, we use the same as a measure of equity in our work. Intuitively, a lower value of G indicates that languages are closer to uniform distribution. Considering either extremes, when all languages have the same perfor- mance. G 0, and when there is support for only one language, G of language for a task is yi. (i 1... where is the total number of languages), and is indexed in non-decreasing order (y: < 1). then the Gini coefficient (G) can be calculated as:
2.3 Incentivization: Reward Improvement
We can estimate current global progress in lan- guage technologies using the demographic and lin- guistic weighted global averages, and the global equity values. In GlobalBench, development of systems that improve upon these metrics. We accomplish this in two ways:
also encourage
Formally, if performance
First, we identify areas with the greatest potential for improvement, i.e., we identify the most under- served languages. Blasi et al. (2022), demon- strates that variety of values provide different prioritizations over demographic- and linguistic- weighted utility. Following the footsteps of Blasi et al. (2022), we chose parameter of 7 0.4 largely qualitatively, observing the results with dif- ferent values and taking one that seemed to strike a reasonable balance between higher- and lower-
G - I + (
Dn 1
(n + 1 ilyi
(4)
E Vi
For each task, we obtain equity values, calcu- lated using the maximum performance of submitted systems for each language in a task. For languages that are not supported by any dataset, we assume

---

resource languages, such that some of each ap- peared near the top of the languages prioritize. Languages farthest from the ideal r-weighted util- ity are expected to be most under-served. Hence, (1 (1-T-wed T-weighted utiliry) of a language gives us this measure. We sort each of the 6671 languages supported in GlobalBench according to this mea- sure. In the end, we obtain a ranking of languages with relatively high populations and relatively low scores, broken down by task.
all datasets that GlobalBench sees system submis- sions in $A.1. While we support 17 tasks covering 966 datasets, we don't have system outputs for all as now. We make note of system outputs avail- able for each task in Table 1, to inform our results and analyses.
Results and Analysis
GlobalBench allows to conduct series of anal- yses to assess global progress in NLP technologies. While GlobalBench intended be evolving gauge the continued improvements in performance of NLP systems, we examine the current state of systems that have been submitted, to demonstrate how GlobalBench can guide and incentivize future participation and improvement in language tech- nology.
Second, submission's rank on our leaderboard is determined by how much it contributes to in- creasing the overall r-weighted utility across lan- guages. This can be achieved in two ways: i) Data Efforts: By contributing datasets for previously unsupported languages, their utility, which was previously immeasurable (hence, for all practical purposes, zero), sees sharp rise; it) Improved Systems: By submitting new systems which im- prove upon the state-of-the-art, we improve utility by definition (Equation 3).
5.1 How inclusive is GlobalBench?
The inclusive design of GlobalBench elucidated in Section 2 means that it can (in theory) support any NLP task and dataset. GlobalBench currently sup- ports dataset submissions of 6671 languages and 17 tasks. GlobalBench now covers 966 datasets in 190 languages. We have 1,128 system outputs at the time of writing, spanning 6 NLP tasks: named entity recognition (NER), text pair classification, text classification, extractive QA, machine transla- lion, and KG link tail prediction; over a total of 62 languages. With the existing systems included in GlobalBench, we already cover 4. ..72% to 59.34% of the first languages of people in the world depend- ing on the task, detailed in Table 2. We focus on analyzing system submissions for these six tasks below
3 Implementation and Participation Details for GlobalBench
GlobalBench accepts open submissions of datasets and systems.
For datasets that are already part of Global- Bench, system results for them can submitted to GlobalBench for evaluation. The submission process of system results is simple: one can submit system results to GlobalBench through github di- rectly. Participants don't need to separately submit anything else.
In addition, if people want to submit new datasets to benchmark, one can follow the dataset submission process provided in our github reposi- tory. After doing so, corresponding system results can be submitted as per the above instructions.
5.2 What is our Current Progress as Measured by GlobalBench?
Next, we discuss the current state of NLP progress through the lens of GlobalBench. To do so, we display the demographic- and linguistic-eiighted global average scores in Table 2.
4 Datasets and Tasks
In GlobalBench, we currently support 17 tasks that fall into 10 distinet categories. These tasks rep- resent diverse set of NLP technologies, ranging from those that are highly applicative and user- facing (question answering, machine translation etc.), to those that aren't directly applied, but are nonetheless fundamental to NLP (language mod- eling, etc.). We briefly summarize the tasks and provide one example for each task in Figure 3, and we provide a more detailed description and list of
Variance in estimated utility across tasks: In Table 2, we observe that NER has the highest es- timated overall demographic and linguistic global average. Additionally, NER and MT have the high- est language coverage (60 languages). This is be- cause these tasks have been subject extensive and impressive multilingual dataset development and evaluation efforts by authors of FLORES (Goyal et al., 2022), MasakhaNER (Adelani et al., 2021),
Phttps: http:://ggggtuuu.ooo/ioii/ta wwwp//wwww./oloaallee.ooo

---

Number
Category
Task
Metrie
Datasets
Languages
System Outputs
Text Classification
Text Classification
127
12
Accuracy
Named Entity Recognition
78
60
450
F1
Sequence Labeling
Word Segmentation
Chunking
2
F1
Generative
10
CorrectCount
Cloze
Multiple Choice
2
Accuracy
Text Pair Classification
Text Pair Classification
30
96
Accuracy
Span Text Classification
Span Text Classificatiin
Accuracy
Text Editing
Grammatical Firor Correction
10
CogroreeCCo
Extractive
80
18
185
F1
Question Answering
Multiple Choice
Acc.
Open Domain
ExactMatch
Machine Translation
242
60
170
Bleu
Conditional Generation
Summarization
55
Bleu
Code Generation
5
Bleu
KG Prediction
Prediction
Hits
Language Modeling
Language Modeling
Perplexity
Table 1:
Overall Statistics of supported tasks, datasets and system outputs. We currently have 1128 system outputs across 6 tasks and 62 languages. Refer to $5 for detailed analysis.
and NusaCrowd (Cahyawijaya al., 2023), which are all included in GlobalBench. In contrast, the estimated demographic- and linguistic- weighted utility for tasks like KG link prediction or multiple- choice QA are low. These are tasks where intensive data creation efforts have traditionally focused on English. The multilingual datasets that do exist are less widely used and/or not yet included in Glob- alBench. However, GlobalBench can help iden- tify these coverage failures and improve accuracy ($5.3).
Tusk NER
Demo. Avg.
Ling. Avg.
Gini +
% Pop.
0.4489
0.0067
0.9920
59.34%
Extractive QA
0,3460
0.0020
0.9975
44.46%
Text Pair Classif.
0.3466
0.0019
0.9976
39.02%
MT
0.0485
0.0002
0.9987
10.58%
Text Classif. 0.0477
0.0001
0.9997
4.72%
KG Prediction 0.0221
0,0001
0.9998
4.72%
Table 2: Demographic and linguistic global averages, equity values (Gini), and percentage of world population covered by current submissions of system results to GlobalBench. Only showing tasks with at least one system submission.
Linguistic demographic utility: In addition, we observe that overall linguistic utility scores across all tasks are very low, and substantially lower than the corresponding demographic utility scores. For instance, the demographic utility score for NER is 0.4489, while the linguistic utility score for the same is only 0.0067. This makes clear that the systems submitted to GlobalBench are currently doing better job of covering widely-spoken lan- guages, but are doing less well covering all of the languages in the world.
and MT having the same language coverage, MT has a higher Gini value, indicating that amongst the languages supported, NER has a more uniform distribution of performance as compared to MT.
Variation across languages per task: We also maintain ranking of system performance for each lunguage as described in $2.2. For each task, we take the highest performance amongst all systems, to represent the performance for a (task, language) pair. Next, we rank all system performances across languages for each task. For example, suppose we have 4 systems with performances P1, Pz, P3. and Pa for task Ti, where P1 and P2 language L1. and P3 and P4 are in language L2. If P: Pz, then P1 used represent the performance of T, in L,; similarly, if P3 > PI, then Pa is used to represent the performance of T₁ in L2. We then rank the
Equity of systems across languages: Since calculate the Gini coefficient accounting for all 6671 languages, all values are extremely high (near- ing 1). Text classification and KG Prediction only have datasets for English, hence the Gini values almost equal 1. We also note that, despite NER

---

1.0
Score
0.6
0.4
0.2
ben
era
Cinh
other
100%
Percent Population
0.1
(a) NER
1.0
(a) Demographic global average
Score
0.6
0.007
0.4
0.008
0.2
0.005
0.004
othier
100%
0.003
Percent Population
0.002
(b) KG Link Tail Prediction
0.001
Figure 4: Variations across languages for task: Rank- ing of system performance for each language in NER (above) and KG Prediction (below).
Linguistic global average
Figure 5: A snaphsot capturing the increase in global averages for NER in the recent past, with the addition of new datasets.
system performances of L1 and L2 under task Ti, i.c., compare P1 and P3 to which language sees higher system performance. Figure 4a shows chromatic visualization of system analysis across language populations for NER. GlobalBench sup- ports 78 datasets and 60 languages for this task, and sees 450 system submissions of this task. There- fore, we can see high system performances for many languages. However, KG Link Tail Predic- tion does not have many submitted systems (28), with monolingual coverage and low performance, shown Figure 4b. For more comprehensive set of chromatic analysis figures for each task with system outputs, please refer to Appendix $A.2.
tively low. In the second datapoint, systems for African languages from the MasakhaNER dataset were added (Adelani et al., 2021), significantly rais- ing the linguistic average. In the third datapoint, systems from the XTREME benehmark (Hu al., 2020) were added, covering a more populous set of languages, significantly raising the demographic average.
5.3.2 Where
improve?
The variety of analysis in the previous section is, in sense, backward-lookinn progress that has already been made. In order instead get better idea of how we may improve our systems in the future, we use the methodology in Section 2.3 to identify most under-served lan- guages in GlobalBench, which are the languages with relatively high populations and relatively low scores for each of the tasks. To display in Global- Bench, we choose parameter of T 0.4, which allows moderate between considerations of serving all speakers in the world and serving all languages in the world.
it looks back at the
5.3 Measuring improvement with GlobalBench
Another major focus of GlobalBench is to mea- sure and encourage improvement in the quality of language technology systems.
5.3.1 How have we improved?
GlobalBench keeps track of when each submission was made, making it possible to examine global progress in NLP over time. To give one example of this, in Figure 5 we show how dataset submis- sions have helped increase the global averages for NER in the recent past. [Switch to use publication date of datasets/systems to address reviewer:'s question -YS] Specifically, the carliest time point only cov- ered English datasets, leaving both averages rela-
We show the three most under-served languages for each task in Table 3.4 From these statistics we observe some interesting trends. First, for tasks where the most widely spoken non-English lan-
"Note that for the purpose of these statistics, we use the source language for the machine translation task.

---

Task
Lang Lang? 2 Lang 3
collection and benchmarking, which currently sup- ports ten tasks. Notably, none of these benchmarks provide utilittyequityy measures, or a reward struc- ture that incentivizes progress towards the most under-served languages.
Named Entity Recognition
pub
Extractive QA
por
Jpn
urd
Text Pair Classification
ben
por
ind
Machine Translation
cmn
spa
ara
Text Classification
cmn spa
ara
KG Prediction
emn
spa
ara
Moving beyond English While the aforemen- tioned benchmarks have driven progress in NLP for English, there have been several recent efforts made towards other languages as well. Multilin- gual composite benchmarks such as XTREME (Hu et al., 2020), XTREME-R (Ruder et al., 2021), and XGLUE (Liang et al., 2020) are a collection of datasets in variety of tasks and languages. XTREME includes 9 tasks across 40 languages, XTREME-R includes 10 tasks across 50 languages with 198 datasets, and X-GLUE improves GLUE (Wang al., 2018) by including 1 cross-lingual tasks. However, all of these are static and lack the goal to be an all-inclusivv, ever-expanding collec- tion of datasets. Beyond these, there have been directed efforts towards dataset curation, especially for the low [text] resource. MasakhaNER (Adelani et al., 2021) supports large dataset for NER task of 10 African languages. IndoNLU (Wilie et al., 2020) the first vast resource Indonesian bench- mark, and the KLUE benehmark (Park et al., 2021) focuses on diverse tasks in Korean.
Table 3: Most under-served languages for each task (by ISO 639-3 language code).
guages in the world, Mandarin Chinese (cmn), Spanish (spa), and Arabic (ara) are not covered, these are selected the most under-served languages. However, for the tasks with better language cov- erage such as NER, extractive QA, and text pair classification, the most under-served languages are ones with relatively high population that are nonetheless not covered well by existing multilin- gual datasets that have been included in Global- Bench. This indicates a need for more creation or incorporation of datascts for major languages such Punjabi, Wu Chinese, and Portuguese, which have been overlooked by existing composite bench- marks.
6 Related Work
From datasets to benchmarks Given the ubiq- uitous use of NLP technology in applications, it is imperative to track and maintain progress across a variety of NLP tasks. Evaluating and compar- ing systems on a single task can also be problem- atic; past work has identified issues with standard datasets (Artetxe et al., 2020; Gururangan et al., 2018). As the field has progressed, several bench- marks have been released to spur the development of generalizable NL. systems. GLUE (Wang et al., 2018) was one such benchmark with collection of 9 diverse NLU tasks (sentiment analysis (Socher et al., 2013), natural language inference (Williams al., 2018), etc.), contrary to prior benchmarks that focused on datasets for a single category of tasks (Conneau and Kiela, 2018). SuperGI.UF (Wang et al., 2019) updates GLUE by introducing a new set of harder tasks like commonsense rea- soning (Zhang et al., 2018) and question answer- ing (Khashabi et al., 2018). The recently released BIG-bench (Srivastava et al., 2022) consists of diverse set of 204 tasks, aimed specifically evalu- ating the capabilities and limitations of large LMs. Finally, Dynabench (Kiela et al., 2021) is a human- umd-model-ine mooee-ii--te-loog llop platform, for dynamic data
In sum, while all of the above efforts have been impactful, none have had the goal to track global progress made by us research community, and design reward system that incentivizes both data and model work, especially for the under-served languages. With GlobalBench, we propose a first step to bridge this gap and move towards inclusive, multi-faceted measurement of progress.
7 Conclusion
In this paper, we introduce GlobalBench, ever- expanding collection of datasets and models span- ning all tasks and languages within NLP. Our aim is for the community to move towards a common goal of building NLP technology that equitably serves all of the world's citizens. To achieve this, we design a leaderboard resting upon three founda- tional principles: i) inclusivity: we track progress across all tasks and datasets for 6671 languages; ii) multi-faceted evaluation: our evaluation measures the per-speaker utility and equity of submitted sys- tems, and also maintains list of most under-served languages; iii) reward improvement: we reward data and modeling efforts that help improve utility

---

across languages, rather than simply maintaining the best-performing systems. Analysing the 1,128 system outputs already integrated within Global- Bench, reveals that NER has the highest utility at present, but is also least equitable. The combina- tion of high demographic and low linguistic utility underscores that efforts have been mostly limited to populous languages. Finally, we identify that the most under-served languages vary across tasks, but are primarily the ones with relatively high speaker population, but nonetheless low coverage our datasets. All all, we believe that GlobalBench is one step towards measurable progress in improving the global quality and equity of languages tech- nologies for all speakers in the world, and we hope the rest of the research community will join us in pursuit of this goal.
number of speakers of these varieties existed, such information could be incorporated within Glob- alBench at a future date. But the current results reported in this paper do not consider this informa- tion.
Reliance on Performance and Population- Demand Measures:
Currently, GlobalBench re- lies standard performance measures such curacy, F1, and BLEU approximate the utility that would be provided by system a poten- tial user. However, in reality there not so direct of a connection between model performance and whether it is actually serving speakers of a partic- ular language well. In addition, we use the first- language speaking population as an approximation for the demand for language technology in par- ticular language. However, this disregards second- language speakers, and cannot take into account the case where there may be differing demand for particular pieces of technology by speakers dif- ferent languages.
8 Limitations
GlobalBench is a broad-reaching effort that has the ambitious goal of measuring performance across all languages in the world. However, to even take the first steps towards this goal we needed to make number of upproximations, which are also inherent limitations of the current work.
Possible Bad Actors: Bad actor is a common concern for benchmarks working on such incen- tives like GlobalBench. While there may be differ- ent varieties of bad actors those who explicitly cheat, or those who don't 't explicitly cheat but tempt to get good score on the benchmark without solving the underlying problem. For the time being, we intend to manually identify and disqualify the former, and we believe that our broad-spanning approach benchmarking approach may be a sufficient deterrent for the latter.
Comparability: Inclusivity across datasets doesn't come without its downsides. With the goal of covering all datasets, we lose some measure of control to GlobalBench. When evalu- ating global progress of language technology for particular tasks, GlobalBench uses multilingual datasets that may come from distinct sources or have dissimilar genres, causing the difficulty of each dataset to vary. Since GlobalBench doesn't 't take into consideration the differences in diffi- culty among datasets of different languages, dis- tinct datasets across different languages might not be directly comparable. However, this com- mon practice for previous benchmarks such as XTREME (Hu et al.. 2020) and Universal Depen- dencies (Nivre et al., 2016). Furthermore, we ex- pect the law of averages to even out this issue as we keep collecting diverse datasets across domains for each language.
Inclusivity
and
Acknowledgements
We gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, the Alfred P. Sloan Foundation Fellowship, and NSF grants No. IIS2125201, IIS2203097, and NSF-FAI 2040926.
References
David Ifeoluwa Adelani, Jade Abbott, Graham Neu- big, Daniel D' souza, Julia Kreutzer, Constantine Lig- nos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Is- rael Abebe Azime, Shamsuddeen H. Muhammad. Chris Chinenye Emezue, Joyce Nakatumba-Nahende Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yi- mam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani,
Languages vs. Language Varieties: In addition, while GlobalBench relies heavily on distinctions between languages, language boundaries are neb- ulous, and many dialects or language varieties ex- ist. If datasets annotated with language varicties, as well as demographic information regarding the

---

Rubungo Andre Niyongabo, Jonathan Mukiibi, Ver- rah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chuk- wuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akin- ode, Deborah Nabagereka, Maurice Katusimme, Ayo- dele Awokoya, Mouhamadane MBOUP, Dibora Ge- breyohannes, Henok Tilaye, Kelechi Nwaike, De- gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore- vaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thiemo Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Sa- lomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji. Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moel- jadi, Cahya Wirawan, Frederikus Hudi, Muham- mad Satrio Wicaksono, Ivan Parmonangan, Ika Al- fina, mhaa Firdausi Putra, Samsul Rahmadani, Yu- lianti Oenang. Ali Septiandri, James Jaya, Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindyatama Nityasya, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Haryo Wibowo, Cuk Tho, Ich- wanul Karo Karo, Tirana Fatyanosa, Ziwei Ji, Gra- ham Neubig, Timothy Baldwin. Sebastian Ruder, Pas- cale Fung. Herry Sujaini, Sakriani Sakti, and Ayu Pur- warianti. 2023. NusaCrowd: Open source initiative for Indonesian NLP resources. In Findings of the As- sociation for Computational Linguistics: ACL 2023, pages 13745-13818, Toronto, Canada, Association for Computational Linguistics.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tok- enization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jonathan H. Clark, Eunsol Choi, Michael Collins. Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki, 2020. TyDi QA: A benchmark for information-sseking question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross- lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4631 Online. Association for Computational I Linguisties.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Mikel Artetxe and Holger Schwenk. 2019, Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610. 6100
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Peter K Austin and Julia Sallabank. 2011. The Cam- bridge handbook of endangered languages. Cam- bridge University Press.
Damian Blasi, Antonios Anastasopoulos, and Gra- ham Neubig, 2022. Systematic incqualities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 5486-5505, Dublin, Ireland. Association for Computational Linguistics.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine trans- lation. ACM Computing Surveys (CSUR), 53)): 38.
Robert Dorfman. 1979. A formula for the gini coeffi- cient. The review of economics and statistics, pages 146.149.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. Advances in neural information pro- cessing systems, 26.
Fahim Faisal, Sharlina Keshava, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2021. SD-QA: Spo- ken dialectal question answering for the real world. In Findings of the Asxociation for Computational Linguistics: EMNLP 2021, pages 3296-3315, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015, A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare Aurelio Ranzato, Francisco Guzmán,

---

and Angela Fan. 2022. The Flores-101 evaluation benchmark for low- fow-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguisties, 10:522-538.
gen, Grusha Prasad, Amanpreet Singh, Pratik Ring- shia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel. Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4110 4124, Online. Association for Computa- tional Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computa- tional Linguisties.
Patrick Lewis. Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evalu- ating cross- mooss-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7315- 7330, Online. Association for Computational Lin- guistics.
Michael A. Hedderich, Lukas Lange, Heike Adel, Jan- nik Strötgen, and Dietrich Klakow, 2021. A survey on recent approaches for natural language process- ing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Xin and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Human Language Technologies. pages Online. Association for Computational Linguistics.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenféi Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang. Rahul Agrawal, Edward Cui, Siming Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark datasetfor cross-lingual pre-training, un- derstanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008-6018 Online, Association for Computational Linguistics.
Charles T. Hemphill. John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference on Machine Learn- ing, pages 4411 1-4421. PMLR.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bemardi, and Roberto Zam- parelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14). pages 216-223, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293 6223, Online. Association for Computational Linguistics.
Simran Khanuja, Sebastian Ruder, and Partha Talukdar. 2023. Evaluating the diversity, equity, and inclu- sion of NLP technology: A case study for Indian languages. In Findings of the Association for Compu- lational Linguistics: EACL 2023, pages 1763-1777, Dubrovnik, Croatia. Association for Computational Linguistics.
Joakim Nivre, ter, Yoav Goldberg, Jan Hajič, Christopher D. Man- ning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies vI: A multilingual treebank collection. Proceedings of the Tenth In- ternational Conference on Language Resources and Evaluation 16), pages 1659-1666.. Portorož. Slovenia. European Language Resources Association (ELRA).
de Marneffe, Filip Gin-
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay. and Dan Roth. 2018 Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pa- pers), pages 252 2522 New Orleans, Louisiana. As- sociation for Computational Linguistics.
Alissa Ostapenko, Shuly Wintner, Melinda Fricke, and Yulia Tsvetkov. 2022. Speaker information can guide models to better inductive biases: A case study on predicting code-switching. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3853-3867, Dublin, Ireland. Association for Compu- tational Linguistics.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vid-

---

Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Anmual Meeting of the Association for Compu- lational Linguistics (ACL' '05), pages 1 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purposs language under- standing systems. Advances in neural information processing systems, 32.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Jun- seong Kim, Youngsook Song. Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee. Sangwoo Sco, Dongjun Lee. Hyunwoo Kim, Myeonghwa Lee. Scongbo Jang, Seungwon Do. Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Sconghyun Kim, Lucy Park, Lucy Park, Alice Oh, Jung-Woo Woo Ha (NAVER AI Lab), Kyunghyun Cho, and Kyunghyun Cho. 2021. Klue: Korean language understanding eval- uation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks, volume I. Curran.
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Lan- guage Processing, pages 843-857 857, Suzhou, China. Association for Computational Linguisties.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Sid- dhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Grabam Neubig, and Melvin John- son. 2021. XTREME-R R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P Processing, pages 10215-10245, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguisties.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 12221122, New Orleans. Louisiana. Association for Computational Linguis- tics.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-end structure- aware convolutional networks for knowledge base completion. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3060- 3067.
Xinyan Yu, Trina Chatterjee, Akari Asai, Junjie Hu, and Eunsol Choi. 2022. Beyond counting datasets: A survey of multilingual dataset construction and necessary resources. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3725-3743, 3343, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642 Seattle, Washington, USA. Association for Computational Linguistics.
Weizhe Yuan and Pengfei Liu. 2022. restructured pre- training. arXiv preprint arXiv:2206.1 11147.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Dub, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. arXiv preprint arXiv: 1810. 12885.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0445 04611.
A Appendix
A.1
Tasks, Datasets and System Outputs
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In
We summarize the tasks and all datasets for which GlobalBench sees submissions of system outputs. The datasets cover a wide range of languages and domains, and vary in terms of size and complexity.
I Language-indepenee Proceedings of the Seventh Conference Natural Language Learning at HLT-NAACL 2003, pages 142- 147.

---

Text Classification This task involves classify- ing text into one or more predefined categories (one label associated with each sequence). An exam- ple of this would be classifying the sentiment of a movie review to be either positive, negative or neu- tral. GlobalBench covers the following datasets for this task: the QC (Question Classification) dataset (Li and Roth, 2002), the ATIS (Airline Travel In- formation Systems) dataset (Hemphill al., 1990), the MR (Movie Review) dataset (Pang and Lee, 2005), the SST-2 (Stanford Sentiment Treebank) Corpus (Socher et al., 2013), datasets from GLUE (the General Language Understanding Evaluation) benchmark (Wang et al., 2018), and the Code- Switching Corpus (Ostapenko et al., 2022).
et al., 2018), the Stanford Natural Language Infer- ence (SNLI) corpus (Bowman et al., 2015), and the Sentences Involving Compositional Knowldedge (SICK) dataset (Marelli et al., 2014).
Span Text Classification This task involves clas- sifying a span of text within a larger piece of text, rather than the whole text. Here, we include aspect- based sentiment classification as a task, which in- volves predicting the sentiment of certain features within sequence.
Text Editing tions or improvements requirements, such fixing grammar or spelling errors. We include the task of grammatical error correction here.
This task involves making correc-
piece of text given some
Sequence Labeling This task involves label- ing each word or token in a sequence with a specific tag, requiring contextual understand- ing at the token-level. We cover three tasks (NER), Word-
Question Answering This task involves gener- ating answers to questions either given a context, or in an open-ended fashion. We focus on three tasks: extractive QA: extracting the answer span from a given context, multiple choice QA: answer- ing question based on a set of given options, and open-domain QA: answering questions from an open domain, such as general knowledge current events. GlobalBench covers the following datasets for this task: XQuAD (Artetxe al., 2020), Ty- DiQA (Clark et al., 2020), SD-QA (Faisal et al., 2021), and MLQA (Lewis et al., 2020).
here: Named-Eniity-Recogeee Segmentation, and Chunking. NER aims to iden- tify and classify named entities, such as person names, organizations, and locations, in a piece of text. Word-Segmentation involves identifying the word boundaries of certain languages like Chi- nese, Japanese, and Thai. Chunking involves divid- ing text into syntactically and semantically coher- ent chunks or phrases for languages like Chinese. GlobalBench covers the following datasets for this task: the MasakhaNER Corpus (Adelani et al., 2021), the CoNLL-2003 dataset (Tjong Kim Sang and De Meulder, 2003), and the PAN-X dataset (Artetxe and Schwenk, 2019).
Conditional Generation This task involves gen- erating text based on certain conditions or con- straints. We focus on three tasks: machine trans- lation (MT), summarization, and code generation. Machine translation involves translating text from one source language to another where possible MT pairs are counted by considering every pos- sible pair of languages, sunmarization involves generating a concise summary of articles or doc- uments, and code generation involves generating a program in a programming language given an input command in natural language. GlobalBench covers the following datasets for this task: datasets from the Fifth Conference on Machine Transla- tion (WMT20) shared tasks and datasets from the Gaokao benchmark (Yuan and Liu, 2022).
Cloze This task involves filling in missing words or phrases in text, based on the context provided by the surrounding words. We focus on multiple- choice cloze and generative cloze. The former in- volves filling in the blanks with one of several op- tions, while the latter involves filling in the blanks for given prompt.
Text Pair Classification This task involves clas- sifying the relationship of pairs of texts, such as determining whether two sentences are paraphrases of each other, or contradict each other. A widely known example is that of natural language in- ference, where the task is to predict whether given hypothesis can be entailed from or contra- dicts a premise. GlobalBench covers the follow- ing datasets for this task: the Cross-lingual Natu- ral Language Inference (XNLI) corpus (Conneau
Knowledge Graph Prediction This task in- volves reasoning over knowledge graphs (KG), for example, predicting missing or unknown facts in KG, based on information contained in the graph.
Phttss/wwww.stumm.nnn/mmmz

---

We include the KG link tail prediction task, which aims to predict the tail entity of missing links in knowledge graphs. GlobalBench covers the follow- ing datasets for this task: WordNet18RF (Shang et al., 2019), FB 15K-237 (Bordes et al., 2013).
Language Modeling This task involves predict- ing the next token in sequence, given the con- text of previous tokens. It is fundamental task used to pre-train decoder models, which are further adapted for downstream applications.
A.2 Visualization of System Performance across Language populations
We visualize system performances across language populations for each task with at least one system output, shown in Figure 6.
A.3 GlobalBench UI
Figure 7 shows the User Interface of GlobalBench Text Pair Classification task. On the top left of the webpage, there is a brief description of the task. Participants will able to see the statistics of all analyses. For instance, under the Demographic- Weighted Global Average analysis, there is the overall Demographic Average of this task and the diachronic figure representing how the the over- all Demographic Average of this task has changed over time.

---

Score
0.2
c10
100%
c1g
100%
Percent Population
Percent Population
(a) KG
Prediction
(b) Text Classification
1.0
1.0
0.8
5core
105 mmg other
100%
wanm
other
100%
Percent Population
Percent Population
(c) Machine Translation
(d) Text Pair Classification
1.0
0.8
56014
cmn
ben
other
100%
eng
cmn
other
100%
Percent Population
Percent Population
(e) Extractive QA
(f) Named Entity Recognition
Figure Visualization of System Performance across Language Population
Global Text-Pair
Desciatio: Teo Clasi
wirdonatoo
whather
whetee they cantraic wid other.
Contect
Deferrecee Severmal
Cowured Delenst:
the Lemparn
Cavere Telo:
bndiaaiiiedaaeee Kibls susiil Ar ardee anioiion inchariies
Updetedet 0--31-2003 2532
Contibund inderbuett
Coneituunt Tasts
yuar
Figure 7: GlobalBench Text Pair Classification task UI


Reviews:
[{"review_id": "iqRM6CM1s7", "reviewer": "Reviewer_bbS1", "summary": "", "questions": "- How does the \"reward\" mechanism work? Are the most important systems listed or highlighted in some way?\n\n- What happens to systems that are submitted but do not beat the current state of the art for the systems? How do you measure if those systems are beneficial in other ways, like being simpler, consuming less resources, etc.\n\n- How are the possible MT pairs counted? Is it only translation from and into English, does it try to consider every possible pair (ideally), or something else?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback! We are encouraged that you are interested in our incentivization approach. We are grateful that you like our inclusion of the relative number of speakers as a measure of impact while at the same time striking a balance with fair distribution across languages.\n\nWe attempt to address the following concerns.\n\n+ Regarding the problem of \"*racing to create systems that satisfy the benchmarks regardless of their utility in real scenarios*\": We fully understand the reviewer's concern, but would argue that this is an issue with all benchmarks, and historically the NLP community has successfully made gradual improvements to adjust to the issues of benchmarks not being aligned with real-world use cases. For example, in question answering, we started with Squad, which evolved into Squad 2.0 adding unanswerable questions, then Natural Questions, which added naturally asked questions, then TyDiQA, which was multilingual. We hope that GlobalBench is a starting point, not an ending point, for development of inclusive NLP benchmarks. In addition, we'd like to highlight that a system can ideally \"satisfy our benchmark\", only if it contributes to increasing the utility of NLP technology for a language or task. The inclusion of rewarding data contributions is unprecedented and previous benchmarks have only focused on rewarding absolute performance. Hence, we believe that our reward system might better align with utility in the real world. In the future, we'd like to make our reward system multi-dimensional, where systems that are resource-efficient, scalable, and deployable, can also be rewarded.\n\n+ Regarding the problem of \"*the platform does not contain many of the common datasets and benchmarks*\": Currently we have the largest coverage in terms of datasets, tasks and languages (to the best of our knowledge). GlobalBench is also constantly expanding, continually incorporating more datasets, benchmarks, tasks, and languages, with the effort from all participants. With the opportunity to present at EMNLP and thus gain greater visibility, we strongly believe that we can achieve our goal of securing GlobalBench's continuous expansion.\n\n+ Regarding \"*How does the \"reward\" mechanism work? Are the most important systems listed or highlighted in some way?*\": We maintain separate leaderboards for all systems submitted for each dataset in each task on ExplainaBoard, the benchmarking platform where GlobalBench is built on top of. As shown in figure 7, the first two pages (\"demo. avg. and ling. avg.\") simply track the average demographic and linguistic utility across all languages for the task. The third leaderboard (\"Gini\") ranks systems based on how equitable their performance is across all languages supported, and the fourth and fifth (\"demo. avg. systems\" and \"ling. avg. systems\"), rank systems based on their demographic and ling utility resp. The \"reward\" system is implemented through the next page on \"most under-served languages\". Here, we rank \"most under-served languages\" by ranking how far the \"$\\tau$-weighted utility\" of language is away from 1 (ideal utility). We are also considering highlighting recent contributions, so that new contributors can also be recognized.\n\n+ Response to the question of \"*systems that are beneficial in other ways*\": This is an excellent idea! We do not have metrics of computational efficiency or other beneficial traits in the leaderboard, but we definitely want to consider ways to incorporate them. One idea might be that we make it possible to view the leaderboard limited to systems that are under a certain parameter size (a similar feature was recently added to the Hugging Face Open LM leaderboard).\n\n+ Response to the question of \"*MT pairs*\": Yes, possible MT pairs are counted by considering every possible pair. Translation is not limited to English: for instance, translation from Indonesian into Turkish is considered a possible MT pair. There are obviously some issues with this – the demographic demand (as calculated and used in section 2.2) for translation may not be equivalent for each pair of languages. We will clarify this in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "If it catches on, this platform could help to incentivize both the creation of datasets for underserved languages and the improvement of the state of the art for existing ones.\n\nI liked the idea of taking in consideration the relative number of speakers to measure the impact, but also balancing with a more equitable distribution across languages, trying not to leave languages with few speakers behind.\n", "reasons_to_reject": "In a way, this could be seen as an attempt to \"gamify\" the process of creating datasets and NLP systems, which could attract attention but at the same time could also worsen the problem of racing to create systems that satisfy the benchmarks regardless of their utility in real scenarios. At the same time, according to the definitions in the paper, it seems a system that performs relatively well but does not beat any existing benchmark would not get any recognition, which could be hindering the intention of the system.\n\nIt seems that the current version of the platform does not contain many of the common datasets and benchmarks that are used for the different languages. For example, sentiment analysis datasets certainly exist for many languages.\n", "questions_to_authors": "- How does the \"reward\" mechanism work? Are the most important systems listed or highlighted in some way?\n\n- What happens to systems that are submitted but do not beat the current state of the art for the systems? How do you measure if those systems are beneficial in other ways, like being simpler, consuming less resources, etc.\n\n- How are the possible MT pairs counted? Is it only translation from and into English, does it try to consider every possible pair (ideally), or something else?\n", "paper_topic_and_main_contributions": "The paper proposes to create a benchmark tool for keeping record of NLP state of the art for many tasks across many languages. The system keeps track of task datasets and submissions to those datasets, calculating metrics for the tasks, and the authors define a set of equations for calculating the utility and equity of the submitted datasets and systems. The approach is interesting in that it would incentivize the creation of datasets for underserved languages and also to improve the state of the art in general. It keeps track of many usual NLP tasks like NER, MT, QA and sentiment analysis, and it could be expanded to include new tasks. However, it seems that so far it has very few datasets for those tasks, e.g. sentiment analysis seems to be covered only for English, although there are datasets for many other languages. It is not clear what language pairs are considered for MT, as it would grow quadratically if we consider all possible combinations. As the platform in its current format relies on using a definition of language that does not count dialects or varieties, it could be a problem as many communities could be left out, this is correctly acknowledged in the limitations section.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c6Zbuyyu5N", "reviewer": "Reviewer_kyHY", "summary": "", "questions": "A. How do the authors propose to ensure or validate the quality of the datasets submitted to the benchmark? Although this is briefly mentioned, this seems like a major risk for the reliability of the benchmarks\nB. In addition to the quality, have the authors considered how the size of submitted datasets can be taken into account when calculating utility? Although a new dataset for a language that previously did not have any is surely an improvement, but 10,000 words with NER annotated is certainly not the same as a 300,000 word annotated dataset. This may cause misrepresentations of the utility and equity of languages.\nC. Given that the benchmark aims to work on the principal of incentives, have the authors considered how bad actors may manipulate or undermine these incentives, and how their platform will manage these types of submissions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback! We are glad that you find our work well argued with a clear delineation of GlobalBench. We appreciate your acknowledgement that our approach offers a more grounded way to identify the technological and data requirements for various languages. \n\nWe attempt to address the following concerns. \n\n+ Regarding the problem of \"*figures and tables*\": We will definitely follow your advice and reformat our figures and tables to make them closer to the text they are referenced in.\n\n+ Response to question A (\"*How do the authors propose to ensure or validate the quality of the datasets submitted to the benchmark?*\"): \nThis is a common concern for benchmarks with dynamic datasets, we agree and hence listed it as a potential limitation that we might lose some measure of control. We are considering including a dataset's status of peer-reviewed conferences in the dataset submission process, which may provide some credibility of the dataset. We would very much appreciate any other suggestions on how to mitigate this limitation.\n\n+ Response to question B (\"*have the authors considered how the size of submitted datasets can be taken into account when calculating utility?*\"): This is a very good point and we will certainly think about this more. Note that GlobalBench is not a \"leaderboard of datasets\" per say, but a leaderboard of systems that also incentivizes dataset creation (as dataset creators will often also be the first to submit a system for that task and language). Because of this, we opt to not explicitly make a judgment on the quality of the submitted datasets, although we may omit datasets that we deem to be of particularly low quality.\n\n+ Response to question C (\"*how bad actors may manipulate or undermine these incentives and how GlobalBench will manage them*\"): This is a valid concern, and we will include this in the limitations section in the final version. Bad actors is a common concern for benchmarks working on such incentives. While there may be different varieties of bad actors – those who explicitly cheat, or those who don't explicitly cheat but attempt to get a good score on the benchmark without solving the underlying problem. For the time being, we intend to manually identify and disqualify the former, and we believe that our multi-faceted and broad-spanning approach benchmarking approach may be a sufficient deterrent for the latter.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In general the submission is very well argued and written with a clear delineation of aims, philosophy, and design of the benchmark. The notions of utility also provides a more grounded way of determining what the technology and data needs are for different languages.\nThis is a very exciting endeavour and I applaud the effort thus far and look forward to see further developments and releases on the platform.\n", "reasons_to_reject": "To me there aren't really any clear reasons to reject the submission. \nAs stated in the questions below addressing the issues of dataset size and quality, as well as considering bad actors in an incentive based platform will further improve the submission.\n\nFor ease of reading, the figures and tables should be closer to where they are referenced in the text. E.g. Figure 1 is on p1, while only mentioned on p3, while Fig 2 is already discussed on p2.\n", "questions_to_authors": "A. How do the authors propose to ensure or validate the quality of the datasets submitted to the benchmark? Although this is briefly mentioned, this seems like a major risk for the reliability of the benchmarks\nB. In addition to the quality, have the authors considered how the size of submitted datasets can be taken into account when calculating utility? Although a new dataset for a language that previously did not have any is surely an improvement, but 10,000 words with NER annotated is certainly not the same as a 300,000 word annotated dataset. This may cause misrepresentations of the utility and equity of languages.\nC. Given that the benchmark aims to work on the principal of incentives, have the authors considered how bad actors may manipulate or undermine these incentives, and how their platform will manage these types of submissions?", "paper_topic_and_main_contributions": "This submission describes the design principals and initial tasks and datasets of a multilingual benchmark and leaderboard platform, GlobalBench.\nThe platform aims to improve the availability of language technologies and datasets across all* of the world's languages, following on from the work of similar multilingual benchmarks. The major contribution of this work is in the fact that it provides specific breakdowns of dataset and technology needs for particular languages, while also providing indications of where developments may have the largest impact in terms of the language community a particular technology would serve.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wWsbQ93Gu3", "reviewer": "Reviewer_8Lhb", "summary": "", "questions": "Question A: Line 244. How do you choose the parameter value 0.4?\n\nQuestion B: Section 5.3.1 and Figure 5. The MasakhaNER dataset was added (2021) as the second dataset, and the XTREME dataset (2020) was added as the third dataset. It is very interesting to see how the performance changes over time. Is there a particular order in how these datasets and systems are added to the GlobalBench? Is it ordered by the date the datasets and systems were released? Or is it more random? I am worried that this randomness will bring artefacts to the analysis.\n\nQuestion C: Table 3. I find it a bit unintuitive that Mandarin Chinese is considered under-served. The research community usually believes Chinese is one of the resource-rich languages. Do you have any explanation for why your analysis shows a different result?\n\nQuestion D: In line 441. You have mentioned, “spoken non-English languages”. How do you distinguish between spoken and written languages in GlobalBench? Most of the datasets used to train NLP systems are based on written text. How do you determine two spoken languages that may share the same or very similar written scripts? For example, Mandarin Chinese and Wu Chinese?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback! We appreciate your recognition of our work as a valuable initiative to identify the most underserved languages and GlobalBench's unique role in assessing contributions of new datasets. We share your excitement for GlobalBench's role as an NLP status indicator and global scores tracker.\n\nWe attempt to address the following concerns. \n\n+ Regarding the problem of \"*how metrics correlate to real benefit*\": This is an excellent question, and it is actually on our roadmap for future research work. For a limited number of tasks there are actually research studies demonstrating the relationship between accuracy and user utility (e.g. for translation and speech recognition). For other tasks the relationship is not entirely clear, and thus examining this correlation becomes a first order of business. This is a very interesting topic but due to its scope we, hopefully understandably, would like to pursue it in future work rather than in this paper.\n\n+ Regarding the problem of \"*directly comparing metrics across different datasets*\": As mentioned in the limitations section, we are aware that this is not ideal but most works so far also compare metrics across different datasets. \n\n+ Regarding the problem of \"*measuring progress of each task by taking the average performance of all datasets*\": This is a valid concern, and hence we listed it as a potential limitation. However, as GlobalBench sees more use, we expect multiple datasets being available for each task and language. Aggregating different datasets can help even out discrepancies across domains. In addition, this practice is also established in other benchmarks like XTREME.\n\n+ Regarding the problem of \"*findings are not very exciting*\", we acknowledge that parts of our analysis are based on snapshots, but this is primarily because GlobalBench is a relatively new benchmark and it is still in the process of evolving. We are dedicated to actively managing and updating GlobalBench with new datasets, benchmarks, tasks, and languages. In addition, with the opportunity to present at EMNLP and thus gain greater visibility, we strongly believe that with GlobalBench's continuous expansion, future analysis will reveal more exciting findings and insights such as the changes of global utility and equity scores over time.\n\n+ Response to question A (\"*How do you choose the parameter value 0.4*\"): Blasi et al., which our work is based on demonstrates that a variety of $\\tau$ values provide different prioritizations over linguistic and demographic utility. We chose 0.4 largely qualitatively, observing the results with different $\\tau$ values and taking one that seemed to strike a reasonable balance between higher- and lower-resource languages (such that some of each appeared near the top of the languages to prioritize). We will explain more about this hyper-parameter choice in the paper. \n\n+ Response to question B (\"*particular order in how these datasets and systems are added to the GlobalBench*\"): This is an excellent point! We are currently using the order of the dataset/system's submission time on GlobalBench in tracking the change in performance over time. We will change this to use the publication date of datasets/systems in the final version.\n\n+ Response to question C (\"*I find it a bit unintuitive that Mandarin Chinese is considered under-served*\"): In measuring most underserved languages, we balance between demographic-weighted (for considerations of serving all speakers) and linguistic-weighted utility (for considerations of serving all languages) as discussed in Section 2.3. Therefore, languages with a lower utility and a larger population would be considered more underserved. In the current nascent incarnation of GlobalBench, there are actually several tasks where there are, as of yet, no Chinese submissions. GlobalBench's population-sensitive utility metrics uncover this, and ideally this will encourage further submissions to fill this current gap.\n\n+ Response to question D (\"*distinguish between spoken and written languages in GlobalBench*\"): This is a valid concern, and hence we listed this problem in the limitations section. GlobalBench depends heavily on distinctions between languages, where language boundaries might be unclear, with many dialects and language varieties, such as the example of Mandarin Chinese and Wu Chinese you mentioned. If we can find relevant information regarding dialects and language varieties, we will include this in GlobalBench. For the current version of GlobalBench, participants submitting datasets are required to identify the language(s) of the submitted dataset.\n\n+ Regarding \"*presentation improvements*\": We will definitely follow your advice of indentation and simplify Figure 2 in the final version. We will also add more explanations to what the arrows are in Figure 2. For the redundant information, we will try to be more succinct and reduce repetitive clarifications in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In addition to automatic evaluation metrics, GlobalBench also evaluates the downstream utility and equity across languages. It is useful to identify languages that may be marginalised or underserved in the NLP research.\n\nIt also provides an opportunity to evaluate how a new dataset contributes to more inclusive and equitable language technology, which previous benchmarks are unable to do.", "reasons_to_reject": "The definitions of utility and equity are all based on automatic evaluation metrics. As mentioned in the Limitation section, it is unclear how these metrics collaborate to the “real” benefit, utility, and satisfaction when users interacting with the system. Alao, directly comparing these automatic metrics across different datasets is also less convincing. However, I acknowledge that how to properly evaluate many NLP systems is still an open research question. \n\nMeasuring the progress for a particular task by simply taking the averaged performance or utility of all available datasets is a bit probabilistic, in my opinion. This issue is particularly profound when we are training and evaluating systems on a dynamic set of datasets. For example, introducing a high-quality but difficult dataset will result in a lower total utility value, which is unintuitive. This will not be an issue if we have a fixed set of evaluation dataset.\n\nThis benchmark is dynamic, in which new datasets and systems are being added. It is exciting to see how the global utility and equity scores change over time. However, the analysis of this paper is largely based on a single snapshot of the benchmark. Consequently, most of the findings are not very exciting and sound very familiar to previous works. (e.g., https://aclanthology.org/2022.acl-long.376.pdf).", "questions_to_authors": "Question A: Line 244. How do you choose the parameter value 0.4?\n\nQuestion B: Section 5.3.1 and Figure 5. The MasakhaNER dataset was added (2021) as the second dataset, and the XTREME dataset (2020) was added as the third dataset. It is very interesting to see how the performance changes over time. Is there a particular order in how these datasets and systems are added to the GlobalBench? Is it ordered by the date the datasets and systems were released? Or is it more random? I am worried that this randomness will bring artefacts to the analysis.\n\nQuestion C: Table 3. I find it a bit unintuitive that Mandarin Chinese is considered under-served. The research community usually believes Chinese is one of the resource-rich languages. Do you have any explanation for why your analysis shows a different result?\n\nQuestion D: In line 441. You have mentioned, “spoken non-English languages”. How do you distinguish between spoken and written languages in GlobalBench? Most of the datasets used to train NLP systems are based on written text. How do you determine two spoken languages that may share the same or very similar written scripts? For example, Mandarin Chinese and Wu Chinese?", "paper_topic_and_main_contributions": "This paper introduces a multilingual benchmark named GlobalBench. Compared to previous works, GlobalBench is: \n\n1) ever-expanding: More datasets can be added in the future.\n2) multi-faceted: It evaluates not only the automatic evaluation metrics but also the utility and equity (please see Section 2.2) of both systems and datasets for each language.\n3) large: It tries to include \"all\" datasets. The community is encouraged to upload datasets to this benchmark. Theoretically, it can include all the datasets.\n\nBased on the current version of GlobalBench, the paper analyses the performance disparities among different languages and tasks.\n\nThis paper describes an ambitious plan to summarise the current progress of multilingual NLP with a single benchmark, which will require collaborative efforts from the community. The technical contribution of this paper is limited. However, I see this paper as a valuable initiative to identify and develop resources for the most underserved languages. \n\nIf the authors are committed to actively maintaining and updating this benchmark for a period of time, this benchmark can be a useful indicator of the status quo of the multilingual NLP. In this case, I look forward to seeing this paper making a meaningful contribution to the community.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 2. No need for indentation for the abstract.\n\nFigure 2. I find Figure 2 is not very intuitive. It looks a bit repetitive. For example, in Pillar 2, the top and bottom graphs are almost the same. Pillar 3 also has the same issue. It is also unclear what these dashed arrows ----> are. It would be nice to provide more explanation in the captioning.\n\nSome sentences are a bit repetitive. It would be nice to remove these repetitions. For example, there are many occurrences of “966 datasets”, “17 tasks”, and “a parameter of τ = 0.4”."}}]

Author rebuttal:

Meta-review: This paper presents GlobalBench: a benchmark that measures global progress in NLP. 
GlobalBench improved upon existing multilingual benchmarks such as XTREME or XGLUE. It supports 6671 languages and 17 tasks. 
Instead of aggregating accuracy metrics like it's done in most benchmarks, GlobalBench provides a multi-view task-specific dashboard based on three main metrics:
- Performance at a given task (e.g. BLEU for Machine Translation)
- Utility that tracks the downstream utility of a system based on how many speakers can benefit from it and how many languages can benefit from it. 
- Equity: Based on the Gini index, it measures how equitable, in terms of language coverage, state-of-the-art NLP technologies is

With Utility, Globalbench provides a leaderboard that incentivizes utility. It ranks all the 6671 supported languages based on a measure of utility that balances demographic and linguistic utility, identifying the languages for which progress is the most needed. 
Not only does utility incentivize performance improvement at existing task and dataset, but it also incentivizes the addition of new dataset and new task. 

Reasons to accept: 
- This paper provides a usable resource to promote measured empirical progress of NLP systems on potentially all languages worldwide (R1, R2 and R3).
- The design choices for Globalbench are novels and thoroughly described  
- Specifically, the integration of utility and equity metrics provides a clear incentive to make progress on under-served languages

Reasons to Reject: 
- A testable anonymous GlobalBench would have been ideal to make the contribution even more concrete.
Decision: Accept-Main