Challenges and Opportunities
in Multilingual Evaluation
Sebastian Ruder
Eval4NLP Workshop at EMNLP 2021

Challenges and Opportunities in Multilingual Evaluation

Language Diversity in NLP●

There are more than 7,000 languages spoken around the world●In NLP research, models are typically developed to work
well only for English
Two big implications:○ Lack of technological inclusion of more than 3 billion
speakers○ Overﬁtting to English●Performance on SQuAD 1.1 Question Answering dataset(Papers with Code)2

http://langscape.umd.edu/map.php

Challenges and Opportunities in Multilingual Evaluation

Agenda01

Resources*

02

Evaluation Setting

3*: Main focus is on datasets for tasks other than MT01

Resources
aBiases and limitationsb

Scaling to Many Languages

Challenges and Opportunities in Multilingual Evaluation

Translation-based Bias●

Many multilingual datasets (XNLI, XQuAD, PAWS-X, etc)
are based on translations●“Translationese”: translations differ in many aspects fromnatural language [Volansky et al., 2015]
Inherits artefacts from existing datasets○ Train-test overlap for answers in NQ [Lewis et al., 2020]○ Language-speciﬁc replications may improve uponannotation methodology [Watarai & Tsuchiya, 2020]
Leads to new artefacts○ Bias towards models trained on translations in XNLI[Artetxe et al., 2020]
Translated text is different from text “naturally” generated
text by speakers of different languages→ English and Western-centric bias

●

●

●

English examples in SQuAD and translationsin XQuAD [Artetxe et al., 2020]

Challenges and Opportunities in Multilingual Evaluation

English and Western-centric Bias●●

Crowd-sourced content is biased towards an English andWestern-centric viewpointCultures differ in what type of content is relevant to them
○ Speakers outside the US probably don’t care about famous
American football and baseball players○ In COPA [Roemmele et al., 2011], many referents have nolanguage-speciﬁc terms in some languages, e.g. bowling ball,hamburger, lottery [Ponti et al., 2020]○ Concepts in ImageNet are Western-centric [Liu et al., 2021]○ Commonsense knowledge, social norms, taboo topics, social
distance, etc are culture-dependent [Thomas, 1983]

Opportunity:

Coverage (% of examples) of countries
across examples with people in QA
datasets [Gor et al., 2021]

A Tamil example in MaRVL [Liu et al., 2021]

Create datasets that capture knowledge and information natively in different languages

Challenges and Opportunities in Multilingual Evaluation

Scaling to Many Languages●●

Labelled data for evaluation is only available
in a small number of languages
Annotation or generation of data in manylanguages is expensive

Distribution of resources across languages [Joshi et al., 2020]

Challenges and Opportunities in Multilingual Evaluation

Scaling to Many Languages●●●●●Labelled data for evaluation is only available
in a small number of languages
Annotation or generation of data in manylanguages is expensive
How can we ﬁll in the gaps and eﬃciently assess
performance in many languages?
Translation is comparatively cheap but introduces
biasesAlternatives:○ Generate targeted evaluation datasets○ Create few-shot datasets○ Cross-lingual performance prediction
Language coverage of tasks in XTREME-R[Ruder et al., 2021]

Challenges and Opportunities in Multilingual Evaluation

Creating Targeted Evaluation Data●●●●

Create template-based test cases, e.g. using CheckList [Ribeiro et al., 2020]
A small number of templates can cover many different model capabilities
Scaling across languages still requires native speaker expertise ortranslation
So far have been used for evaluating reading comprehension [Ruder et al.,2021] and closed-book QA [Jiang et al., 2020; Kassner et al., 2021]

Templates and generated tests for different capabilities in English, Hebrew, Arabic, and Bengali (top) and Multilingual CheckList evaluation ofXLM-R (right) [Ruder et al., 2021]

Challenges and Opportunities in Multilingual Evaluation

Creating Targeted Evaluation Data●●●●Create template-based test cases, e.g. using CheckList [Ribeiro et al., 2020]
A small number of templates can cover many different model capabilities
Scaling across languages still requires native speaker expertise ortranslation
So far have been used for evaluating reading comprehension [Ruder et al.,2021] and closed-book QA [Jiang et al., 2020; Kassner et al., 2021]

Opportunity:

Create targeted evaluation datasets in many languages

Challenges and Opportunities in Multilingual Evaluation

Few-shot Learning●●●For many tasks, ﬁne-tuning on a small number of examples
can signiﬁcantly improve performance in the target language[Hu et al., 2020; Hedderich et al., 2020; Lauscher et al., 2020]
compared to zero-shot transfer
Fine-tuning on many languages is better than on a few, with thesame number of examples [Debnath et al., 2021]
Caveat: More examples are needed for more challenging tasks[Kirstain et al., 2021]
Dependency parsing results across different
numbers of examples [Lauscher et al., 2020]Opportunity:

Focus on creating small training datasets with larger test sets in many languages

Challenges and Opportunities in Multilingual Evaluation

Cross-lingual Performance Prediction
●
●

●
●
●

Instead of creating labelled data, extrapolate performance tolanguages without data [Ye et al., 2021]
Evaluating models on many languages is expensive; could savecosts by only evaluating on a representative subset of languages[Xia et al., 2020]
Could also inform on which languages to focus annotation efforts
Performance prediction methods have been evaluated onlanguages in existing datasets such as UD
Chicken-and-egg problem: need labelled data in order to evaluate
beneﬁt of performance prediction for unseen languages

Opportunity:

Density of WALS typological features of theworld’s languages. Red dots are languages
in UD [Ponti et al., 2021]

Develop performance prediction methods that generalize to unseen languages with different linguistic characteristics

Challenges and Opportunities in Multilingual Evaluation

Participatory Research●In order to create high-quality, naturalistic data inunder-represented languages, we need to work with language
communities directly●Involvement of native speakers is beneﬁcial beyond annotation:○ Ensures that a task formulation is realistic andbeneﬁcial for a language community○ Prevents biases, e.g., a Western-centric viewpoint○ Enables covering language varieties such as dialects ordifferent styles○ See also Steven Bird’s EMNLP 2021 keynote talkOpportunity:

Participatory research with grassroots communities such as Masakhane

Different stakeholders involved in the MTprocess [∀ et al., 2020]

02

Evaluation SettingaEvaluation Protocolb

Evaluation Metrics

Challenges and Opportunities in Multilingual Evaluation

Evaluation Protocol●●●

For cross-lingual transfer, there is a bias towards the source language (often English)○ Favours languages similar to the source language○ Other source languages often perform better [Lin et al., 2019; Anastasopoulos & Neubig, 2020;
Turc et al., 2021]
Training on translations helps particularly for some translated tests sets such as XNLI [Artetxe et al,2020]
Evaluation across many source languages enables a more ﬁne-grained evaluation (but is also moreexpensive)

Opportunity:

Consider the evaluation protocol and associated biases

15

Challenges and Opportunities in Multilingual Evaluation

Evaluation Metrics●●●Token-based metrics (e.g., F1, EM in QA tasks) are not appropriate for languages without
whitespace separation (e.g. Japanese, Thai, Chinese)
Require a language-speciﬁc segmentation method, which introduces a dependence on theevaluation
Metrics based on string matching such as BLEU are not appropriate for morphologically richlanguages
English:Her village is large.
Translations in Shipibo (spoken in Peru) [Valenzuela, 2003]:
Jawen jemara

ani iki.

Jawen jemaronki ani iki.

16Credit: Rachael Tatman

Challenges and Opportunities in Multilingual Evaluation

Evaluation Metrics●●●Token-based metrics (e.g., F1, EM in QA tasks) are not appropriate for languages without
whitespace separation (e.g. Japanese, Thai, Chinese)
Require a language-speciﬁc segmentation method, which introduces a dependence on theevaluation
Metrics based on string matching such as BLEU are not appropriate for morphologically richlanguages

Opportunity:

Consider the bias of your evaluation metric; use character-level evaluation, e.g. chrF [Popović, 2015]17Challenges and Opportunities in Multilingual Evaluation

Aggregating Evaluation Metrics●●●

Averaging performance only overlanguages in existing datasets provides a
distorted view of progress
Most languages in existing datasets are
high-resource
If we consider all languages of the worldor languages with large speaker
populations, the outlook is much more
pessimistic [Blasi et al., 2021]
Linguistic vs demographic global utility ofdifferent NLP applications [Blasi et al., 2021]Opportunity:

Be cautious with aggregating performance; highlight performance on different language families, etc

Challenges and Opportunities in Multilingual Evaluation

Languages are diverse, not only linguistically but culturally
EnglishFascinating language!
...but no credit!

Arabic‫ﻛﺗب‬Bengaliসেফদা ফল খেত কমন
Finnishjälleenrakennustöihin
Kiswahili inayozungumzwa katika
Korean우리 모두가 만들어가는
Indonesian kecilkecilan
Japanese 24時間でのサーキット周回数
RussianмикроскопическийTeluguఖండాలలో అతిపెద్దThaiเรือยิงตอรปโดยูโกสลาเวีย ที5

FinnishRussianEnglishKoreanArabicBengaliJapaneseThai

Kiswahili
TeluguIndonesianCredit: Jon ClarkChallenges and Opportunities in Multilingual Evaluation

Takeaways
●
●
●

●

●
●
●

Be aware of biases in existing multilingual datasets
Aim to create datasets that depart from a Western-centric viewpoint
To scale evaluation to many languages, we can…
○ Create targeted evaluation datasets○ Create small datasets across many languages for few-shot learning○ Develop better cross-lingual performance prediction methods
Participatory research with native speaker communities can help to generate more high-quality,
naturalistic data on tasks that native speakers care aboutConsider biases in the evaluation protocol and in the evaluation metricBe cautious with aggregating performance
Highlight performance across different language families and geographies

20Thank You