Title: Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
Summary: In this work, the authors aim to improve the NLU capabilities of Indic languages by making contributions along 3 important axes (i) monolingual corpora (ii) NLU testsets (iii) multilingual LLMs focusing on Indic languages. Specifically, they curate the largest monolingual corpora, IndicCorp, with 20.9B tokens covering 24 languages from 4 language families - a 2.3x increase over prior work, while supporting 12 additional languages. Next, they create a humansupervised benchmark, IndicXTREME, consisting of nine diverse NLU tasks covering 20 languages. Across languages and tasks, IndicXTREME contains a total of 105 evaluation sets, of which 52 are new contributions to the literature. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Natural Language Understanding for Indic Languages
Phenomenon defined by authors: No
Phenomenon definition: It's not defined.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: There are 9 different tasks across 20 languages. These include five classification
tasks, two structure prediction tasks, one QA task, and one text retrieval task.
Task item definition: There are 9 different asks across 20 languages. These include five classification
tasks, two structure prediction tasks, one QA task, and one text retrieval task. So, depends on each task the item is different.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: Dev: ~15k, Test: ~17k
Dataset metadata available: Yes
Dataset metadata detail: Domain
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean, standard deviation
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding