Title: MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks
Benchmark: MEGAVERSE
Summary:  This study aims to perform a thorough evaluation of the non-English capabilities of SoTA LLMs (GPT-3.5-Turbo, GPT-4, PaLM2, Gemini-Pro, Mistral, Llama2, and Gemma) by comparing them on the same set of multilingual datasets. This benchmark comprises 22 datasets covering 83 languages, including low-resource African languages. We also include two multimodal datasets in the benchmark and compare the performance of LLaVA models, GPT-4-Vision and Gemini-Pro-Vision. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: various capabilities (classification, text generation, reasoning, etc.) of LLMs in 83 languages using 22 datasets and cross-lingual transfer in LLMs
Phenomenon defined by authors: Yes
Phenomenon definition: There are multiple phenomena are addressed in this paper.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The tasks are included in the paper: Visual Reasoning, Multiple Choice Reading Comprehension, Commonsense Reasoning, POS Tagging, NER, NLI, Paraphrase Detection, Image Captioning, Translation, Summarization, QA, Task-Oriented Dialouge 
Task item definition: It's a multiple tasks/datasets benchmark 
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: Yes
Dataset metadata detail: language 
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Multilinguality