IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task
  Language Understanding
Authors: name: Sankalp KJ, name: Ashutosh Kumar, name: Laxmaan Balaji, name: Nikunj Kotecha, name: Vinija Jain, name: Aman Chadha, name: Sreyoshi Bhaduri

Known by more than 1.5 billion people in the Indian subcontinent, Indic
languages present unique challenges and opportunities for natural language
processing (NLP) research due to their rich cultural heritage, linguistic
diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark
designed to evaluate Large Language Models (LLMs) across Indic languages,
building upon the MMLU Pro (Massive Multitask Language Understanding)
framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi,
Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique
challenges and opportunities presented by the linguistic diversity of the
Indian subcontinent. This benchmark encompasses a wide range of tasks in
language comprehension, reasoning, and generation, meticulously crafted to
capture the intricacies of Indian languages. IndicMMLU-Pro provides a
standardized evaluation framework to push the research boundaries in Indic
language AI, facilitating the development of more accurate, efficient, and
culturally sensitive models. This paper outlines the benchmarks' design
principles, task taxonomy, and data collection methodology, and presents
baseline results from state-of-the-art multilingual models.