(WIP)
GlobalBench: A Benchmark for Global
Progress in Natural Language Processing
Graham Neubig
w/ Catherine Cui, Pengfei Liu, Fahim Faisal, Alissa Ostapenko,
Yulia Tsvetkov, Antonios Anastasopoulos

and You!
https://github.com/neulab/globalbenchWhy do we Build Language Technology?

Why do we Build Language Technology?
お手洗いはそちらの青い建物にあります😰

Why do we Build Language Technology?
お手洗いはそちらの青い建物にあります😰

お手洗いはそちらの青い建物にありますThe bathroom is in theblue building over there😃

Why do we Build Language Technology?
お手洗いはそちらの青い建物にあります😰

お手洗いはそちらの青い建物にあります• We want to make the world a better place!

The bathroom is in theblue building over there😃

Why do we Build Language Technology?
お手洗いはそちらの青い建物にあります😰

お手洗いはそちらの青い建物にあります• We want to make the world a better place!
• How do we quantify "better"?

The bathroom is in theblue building over there😃

Why do we Build Language Technology?
お手洗いはそちらの青い建物にあります😰

お手洗いはそちらの青い建物にありますThe bathroom is in theblue building over there• We want to make the world a better place!
• How do we quantify "better"?
•Utility (economics): the total satisfaction received fromconsuming a good or service.

😃

How Much Utility Do We Derive from Language Technology?

Image Credit: Wikipedia

How Much Utility Do We Derive from Language Technology?
•American English Speaker: Use Google assistant, car navigation
system, translate text, bene t from good search technologyfiImage Credit: Wikipedia

How Much Utility Do We Derive from Language Technology?
•

American English Speaker: Use Google assistant, car navigation
system, translate text, bene t from good search technology•Japanese Speaker: Use the above technology, maybe with fewerfeatures, maybe a bit worsefiImage Credit: Wikipedia

How Much Utility Do We Derive from Language Technology?
•American English Speaker: Use Google assistant, car navigation
system, translate text, bene t from good search technology•Japanese Speaker: Use the above technology, maybe with fewerfeatures, maybe a bit worse•

Marshalese Speaker: Don't use the above technology, or be forced
to use it in a second language

fiImage Credit: Wikipedia

How Much Utility Do We Derive from Language Technology?
•American English Speaker: Use Google assistant, car navigation
system, translate text, bene t from good search technology•Japanese Speaker: Use the above technology, maybe with fewerfeatures, maybe a bit worse•

Marshalese Speaker: Don't use the above technology, or be forced
to use it in a second language•

English Speaker with a Greek accent: Use Google assistant, carnavigation system, translate text, bene t from good search technology

fifi

Image Credit: Wikipedia

A Recipe for Progress?

A Recipe for Progress?
Acc= 97.1•

Quantify disparities in language
technology performance

94.2

75.1

A Recipe for Progress?
Acc= 97.1••94.2

75.1

Quantify disparities in language
technology performance

Link performance to derived utility

Accuracy=94.2

😃?
😰?

A Recipe for Progress?
Acc= 97.1
••

94.2

75.1

Quantify disparities in language
technology performance

Link performance to derived utility

Accuracy=94.2

😃?
😰?

•

Mitigate disparities through researchcommunity incentives

😃

😃

😃

Multilingual Leaderboards

Multilingual Leaderboards...

Multilingual Leaderboards...

Multilingual Leaderboards...

Multilingual Leaderboards...

Multilingual Leaderboards...

•Very popular! But what does “multilingual” mean?

Multilingual Leaderboards...

•

Very popular! But what does “multilingual” mean?

•

XTREME: "availability of monolingual data, and typological diversity"

Multilingual Leaderboards...

•

Very popular! But what does “multilingual” mean?

•

XTREME: "availability of monolingual data, and typological diversity"•

XGLUE: ??? (as far as I can tell, not stated)

Multilingual Leaderboards...

•

Very popular! But what does “multilingual” mean?

•

XTREME: "availability of monolingual data, and typological diversity"•XGLUE: ??? (as far as I can tell, not stated)
But what about people?!

Global Utility Metrics(Blasi et al. 2022)

Global Utility Metrics(Blasi et al. 2022)•A language technology should be measured by the utility it provides to every
person in the worldGlobal Utility Metrics(Blasi et al. 2022)•A language technology should be measured by the utility it provides to every
person in the world
X

M=uii

Global Utility Metrics(Blasi et al. 2022)•

A language technology should be measured by the utility it provides to every
person in the world
X

M=•

Two problems:uii

Global Utility Metrics(Blasi et al. 2022)•A language technology should be measured by the utility it provides to every
person in the world
XM=•Two problems:•uiiProblem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)Global Utility Metrics(Blasi et al. 2022)
A language technology should be measured by the utility it provides to every
person in the world
X•

M=Two problems:i•Problem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)•Problem 2: how to consider different utility provided to every person in the world?
→ Measure over subgroups (here, languages), weighted by demand + coef cient τ.

fi•uiGlobal Utility Metrics(Blasi et al. 2022)
A language technology should be measured by the utility it provides to every
person in the world
X•

M=Two problems:i•

Problem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)•Problem 2: how to consider different utility provided to every person in the world?
→ Measure over subgroups (here, languages), weighted by demand + coef cient τ.

X (⌧ )
M⌧ =dl · ull2Lfi•uiGlobal Utility Metrics(Blasi et al. 2022)
A language technology should be measured by the utility it provides to everyperson in the world
X•

M=

Two problems:i•

Problem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)•Problem 2: how to consider different utility provided to every person in the world?
→ Measure over subgroups (here, languages), weighted by demand + coef cient τ.

X (⌧ )
M⌧ =dl · ull2L"normalized
demand"fi•ui

Global Utility Metrics(Blasi et al. 2022)
A language technology should be measured by the utility it provides to every
person in the world
X•

M=

Two problems:i•

Problem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)•Problem 2: how to consider different utility provided to every person in the world?
→ Measure over subgroups (here, languages), weighted by demand + coef cient τ.

X (⌧ )M⌧ =dl · ull2L"normalized"utility"
demand"fi•ui

Global Utility Metrics(Blasi et al. 2022)
A language technology should be measured by the utility it provides to every
person in the world
X•

M=

Two problems:i•

Problem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)•Problem 2: how to consider different utility provided to every person in the world?
→ Measure over subgroups (here, languages), weighted by demand + coef cient τ.

X (⌧ )M⌧ =dl · ull2L"normalized"utility"demand"(⌧ )
dl =

⌧nlP

⌧
n0l 2L l0fi•

uiGlobal Utility Metrics(Blasi et al. 2022)
A language technology should be measured by the utility it provides to every
person in the worldX•

M=Two problems:i•Problem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)•Problem 2: how to consider different utility provided to every person in the world?
→ Measure over subgroups (here, languages), weighted by demand + coef cient τ.

X (⌧ )M⌧ =dl · ull2L"normalized"utility"demand"

(⌧ )
dl =

⌧nlP

⌧
n0l 2L l0

τ=1 : every person equal("demographic-average utility")

fi

•ui

Global Utility Metrics(Blasi et al. 2022)
A language technology should be measured by the utility it provides to every
person in the worldX•M=

Two problems:i•Problem 1: how to measure utility of an NLP system?
→ Very hard, use standard accuracy metrics as a proxy now (happy to discuss more!)•Problem 2: how to consider different utility provided to every person in the world?
→ Measure over subgroups (here, languages), weighted by demand + coef cient τ.

X (⌧ )M⌧ =dl · ull2L"normalized"utility"demand"(⌧ )dl =⌧nlP⌧n0l 2L l0

τ=1 : every person equal("demographic-average utility")
τ=0 : every subgroup equal("linguistic-average utility")fi•uiZooming In
Dependency Parsing
Question Answering

Machine Translation to English

Zooming In
Dependency Parsing
Question Answering

Machine Translation to English

Demographic

Financial

Academic/professional

incentives

incentives

incentives

Language technologies R&D

Demographic
incentives

Financialincentives≈ number language users≈ approximate GDP
associated with language

Which one better predicts the total numberof papers on a given language?

fiWhile GDP and number of language users are correlated,
GDP predicts better the total number of papers publishedon any speci c language

How Can We Change our Incentives?

•

GlobalBench: a benchmark to measure research community progress onequitable language technology
https://explainaboard.inspiredco.ai/benchmark?id=globalbenchGlobalBench May Help By•

Identifying which languages are notcovered yet, or under-performing•

Rewarding submissions of system
results for under-served languages•

Rewarding submissions of datasets
for under-served languages

We Want You!
•Please submit datasets!

•Please submit systems!

•Please propose tasks!

•Please give us other ideas!

https://www.github.com/neulab/globalbench