Title: The BELEBELE Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
Benchmark: BELEBELE
Summary: BELEBELE is a multiple-choice machine reading comprehension benchmark designed to evaluate language models' multilingual capabilities. It's covering diverse languages and scripts from high-, medium-, to low-resource languages. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: reading comprehension, multilingual 
Phenomenon defined by authors: Yes
Phenomenon definition: Multilingual reading comprehension as the ability to understand text passages and correctly answer multiple-choice questions about those passages across different languages.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Multiple-choice reading comprehension: LMs read a passage and answer a question about it by selecting the correct option from four possible answers.
Task item definition: passage from FLORES-200, question about the passage, four multiple-choice answers
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task source detail: Adapted FLORES-200 machine translation dataset. Questions and answers were created in English by professional annotators.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 67.5k training samples, 3.7k development samples
Dataset metadata available: Yes
Dataset metadata detail: language, script, language family
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean accuracy
Metric subscores: Yes
Metric subscores detail: Per-language scores,
Scores by language family,
Scores by script type
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Statistical t-tests to ensure the distribution of features; Training a logistic regression;
Comparison with human performance; Cross-correlation with established benchmarks
Taxonomy: NLP / Understanding
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding