Title: MILA (MULTILINGUAL INDIC LANGUAGE ARCHIVE): A DATASET FOR EQUITABLE MULTILINGUAL LLMS
Primary area: datasets and benchmarks
Keywords: Large Language Models (LLMs), Multilingual NLP, Low-resource languages, Indic languages, Data curation, OCR for Indic scripts, Synthetic data generation, Translation pipelines, Data distillation, Benchmarking (Indic-MMLU), Domain-specific corpora, Continual pretraining, Equitable multilingual modeling, Inclusive language technologies

Abstract: Large Language Models (LLMs) are predominantly trained on high-resource languages such as English, leaving low-resource languages marginalized. This imbalance is particularly acute in India, where most Indic languages lack clean, large-scale digitized corpora despite having hundreds of millions of speakers. Building equitable representation for these languages requires not only greater data volume, but also novel pipelines for acquisition, curation, and validation. We present ***MILA***, the largest curated Indic multilingual dataset to date, spanning 7.5 trillion tokens across 16 of India’s 22 official languages. The dataset is constructed through a multi-stage process combining large-scale crawling, OCR pipelines tailored to Indic scripts, LLM post-corrected translations, synthetic augmentation via the **Indic-Persona Hub**, data distillation, and rigorous filtering. Each stage is validated by expert linguists, ensuring both linguistic fidelity and cultural authenticity. Alongside, we release \textbf{Indic-MMLU}, a translation and verification of MMLU into 16 Indian languages, providing the first large-scale multilingual benchmark for evaluation. We further introduce multiple *domain-specific taxonomies* (finance, Ayurveda, agriculture, law, and catering) to enable creation of targeted pre-training and post-training corpora for Indic use cases. To assess the impact of these resources, we conduct extensive experiments across translation pipelines, OCR incorporation, synthetic supervised fine-tuning (SFT) data generation, and continual pretraining analyses. Across all tasks, models trained on MILA demonstrate stronger performance on Indic-MMLU and achieve improved parity with English, underscoring the value of curated pipelines for equitable multilingual modeling. By bridging resource gaps at scale and validating through expert and synthetic strategies, MILA promises to be a foundational archive for inclusive large-scale language modeling in the Indic context.