BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic
  Domains
Authors: name: Vijay Devane; name: Mohd Nauman; name: Bhargav Patel; name: Aniket Mahendra Wakchoure; name: Yogeshkumar Sant; name: Shyam Pawar; name: Viraj Thakur; name: Ananya Godse; name: Sunil Patra; name: Neha Maurya; name: Suraj Racha; name: Nitish Kamal Singh; name: Ajay Nagpal; name: Piyush Sawarkar; name: Kundeshwar Vijayrao Pundalik; name: Rohit Saluja; name: Ganesh Ramakrishnan
Published: 2025-10-29 11:27:08

Abstract: The rapid advancement of large language models(LLMs) has intensified the need
for domain and culture specific evaluation. Existing benchmarks are largely
Anglocentric and domain-agnostic, limiting their applicability to India-centric
contexts. To address this gap, we introduce BhashaBench V1, the first
domain-specific, multi-task, bilingual benchmark focusing on critical Indic
knowledge systems. BhashaBench V1 contains 74,166 meticulously curated
question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from
authentic government and domain-specific exams. It spans four major domains:
Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and
covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs
reveals significant domain and language specific performance gaps, with
especially large disparities in low-resource domains. For instance, GPT-4o
achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models
consistently perform better on English content compared to Hindi across all
domains. Subdomain-level analysis shows that areas such as Cyber Law,
International Finance perform relatively well, while Panchakarma, Seed Science,
and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive
dataset for evaluating large language models across India's diverse knowledge
domains. It enables assessment of models' ability to integrate domain-specific
knowledge with bilingual understanding. All code, benchmarks, and resources are
publicly available to support open research.