Title: IndicXNLI: Evaluating Multilingual Inference for Indian Languages
Benchmark: IndicXNLI
Summary: The paper introduces INDICXNLI, an NLI dataset for 11 Indic languages. It has been created by high-quality machine translation of the original English XNLI dataset and the analysis
attests to the quality of INDICXNLI. By finetuning different pre-trained LMs on this INDICXNLI, they analyze various cross-lingual transfer techniques with respect to the impact of the choice of language models, languages, multi-linguality, mix-language input,
etc.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Natural Language Inference (NLI)
Phenomenon defined by authors: Yes
Phenomenon definition: NLI assesses if a premise entails, negates, or is neutral towards the hypothesis statement.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Natural Language Inference (NLI) assesses if a premise entails, negates, or is neutral towards the hypothesis statement.
Task item definition: Each item is a pair of sentences.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: for each language: Train 393k, Validation:2.49k, Test: 5.1k
Dataset metadata available: Yes
Dataset metadata detail: Language
Sampling method: Unknown
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple Mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding