Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
Year: 2023

Abstract:
Building Natural Language Understanding (NLU) capabilities for Indic languages, which have a collective speaker base of more than one billion speakers is absolutely crucial. In this work, we aim to improve the NLU capabilities of Indic languages by making contributions along 3 important axes (i) monolingual corpora (ii) NLU testsets (iii) multilingual LLMs focusing on Indic languages. Specifically, we curate the largest monolingual corpora, IndicCorp, with 20.9B tokens covering 24 languages from 4 language families a 2.3x increase over prior work, while supporting 12 additional languages. Next, we create a humansupervised benchmark, IndicXTREME, consisting of nine diverse NLU tasks covering 20 languages. Across languages and tasks, IndicXTREME contains a total of 105 evaluation sets, of which 52 are new contributions to the literature. To the best of our knowledge, this is the first effort towards creating a standard benchmark for Indic languages that aims to test the multilingual zero-shot capabilities of pretrained language models. Finally, we train IndicBERT v2, a state-of-the-art model supporting all the languages. Averaged across languages and tasks, the model achieves an absolute improvement of 2 points over a strong baseline. The data and models are available at https:// github.com/AI4Bharat/IndicBERT.

heading: 1 Introduction
text: Recent advances in Natural Language Understanding are largely driven by pretrained multilingual models (Conneau et al., 2020; Xue et al., 2021; Doddapaneni et al., 2021). One of the advantages of such models is that they can potentially reduce the performance gap between high and lowresource languages through zero-shot knowledge transfer (Hu et al., 2020; Liang et al., 2020). However, in practice, the benefits of such models are
∗ Corresponding authors: Sumanth Doddapaneni (dsumanth17@gmail.com), Mitesh M. Khapra (miteshk@cse.iitm.ac.in)
still skewed towards high-resource languages due to 3 main reason as outlined below.
First, current multilingual models often have a poor representation of low-resource languages. For example, out of the 22 languages listed in the 8th schedule of the Indian constitution, only 15 languages are supported by the popular XLM-R model (Conneau et al., 2020). This is mainly due to the non-availability of pretraining data for languages like Bodo, Dogri, Kashmiri, etc. in large multilingual corpora such as CC-100 (Conneau et al., 2020), or mC4 (Xue et al., 2021). Hence, dedicated efforts towards collecting pretraining data for these languages by discovering and crawling language-specific sources are needed.
Second, even for low-resource languages supported by existing multilingual models, the size of pretraining data is much smaller than that of
1Note that while the number of tokens in mC4 is larger than that in IndicCorp, recent studies (Kreutzer et al., 2022b) have shown that mC4 contains a significant amount of offensive and pornographic content. Further, it is often the case that the content does not belong to the designated language. This is mainly because the data is not crawled from verified URLs. In contrast, in IndicCorp we make a conscious choice to crawl content only from human-verified URLs.
12402
English and other resource-rich languages (Xue et al., 2021). Due to this disparity, low-resource languages get a very poor share of the model’s capacity and vocabulary, and thus the performance on these languages is poor (Conneau et al., 2020). Indeed, a few recent efforts (Kakwani et al., 2020; Khanuja et al., 2021; Dabre et al., 2022; Reid et al., 2021) show that multilingual models trained using pretraining data from a smaller set of related languages leads to better performance on downstream tasks than large scale models which support many languages. Hence, there is a need for training language models only on Indic languages thereby ensuring that the model capacity is not dominated by unrelated high-resource languages.
The third reason is the poor representation of these languages in existing evaluation benchmarks. For example, in the XTREME-R (Ruder et al., 2021) benchmark, out of the 10 tasks only three contain evaluation data for more than two Indic languages. Further, the maximum number of Indic languages for any task is just seven. In effect, 15 of the 22 constitutionally recognized Indic languages have no representation in XTREME-R for any task. Thus, a human supervised evaluation benchmark tailored for Indic, and other low-resource language families is essential for furthering inclusivity and equity in NLP research (Khanuja et al., 2022).
In this work, we make contributions toward addressing all the three challenges. We focus on the 22 languages listed in the 8th schedule of the Indian constitution spanning 4 language families and spoken by over a billion speakers (8 of these languages being amongst the top-20 most spoken languages globally). Some of these languages are also widely spoken and/or are official languages in neighbouring countries viz., Bangladesh, Nepal and Pakistan. Our first contribution towards serving these languages is to release IndicCorp v2, the largest collection of corpora for languages spanning 4 Indic language families with 20.9 Billion tokens and 1.1 Billion sentences. Table 1 shows a comparison of IndicCorp v2 with existing collections of monolingual corpora. As is clear, IndicCorp not only supports more Indic languages but also improves upon the data for languages supported in existing collections (e.g., ×2.3 improvement over IndicCorp v1 with 12B new tokens). Our second contribution is IndicBERT v2, a multilingual LM pretrained on IndicCorp v2 and supporting the largest number of Indic languages compared to existing models such
as XLM-R, MuRIL, and IndicBERT v1. Our third, and perhaps, the most important contribution is IndicXTREME, a human supervised benchmark containing evaluation sets for nine diverse tasks with each task covering 7-18 Indic languages per task. These include five classification tasks, two structure prediction tasks, one QA task, and one text retrieval task. Of the total 105 evaluation sets, summed across languages and tasks, 52 have been newly created as a part of this benchmark. All the newly added evaluation sets have been created manually with the help of in-house language experts with several years of experience in language annotation and translation. The datasets for three tasks, viz., NER, QA, and paraphrase detection were created from scratch without any translation from English sources. We consciously make an effort to include languages spanning all the classes from the inclusion taxonomy introduced in Joshi et al. (2020). According to their classification (Table 14), nine languages in IndicXTREME are the so-called “Left-Behinds”, the most ignored, with exceptionally minimal resources. Only three are “Winners”, the high-resource languages, which have a dominant online presence with industry and government investments.
Using IndicXTREME, we evaluate IndicBERT and show that it outperforms strong baselines on 7/9 evaluation tasks. We also do a series of ablation tests to show that (i) the translation language modeling (TLM) objective slightly improves zero-shot performance when high-quality parallel data is used, (ii) using noisy parallel data during pretraining leads to sub-optimal zeroshot performance, (iii) using in-language-family development sets allows better model selection, and (iv) zero-shot transfer via Hindi, as opposed to English, leads to better performance. All the datasets, code, and models developed as a part of this work will be open-sourced. All the datasets and models developed as a part of this work are available at https://ai4bharat.iitm. ac.in/language-understanding.

heading: 2 Related Work
text: The ability of multilingual models to do zero-shot transfer is often limited to typological cousins inside language families (Ponti et al., 2021, Section 2). This has spurred coordinated research efforts for underrepresented languages, such as Indic languages. Recent works in this domain can be
broadly classified into the following three broad areas.

heading: 2.1 Resources
text: The data resource used most often for pretraining models in Indic languages is Wikipedia. Though it has high-quality text, Indic Wikis are sparsely populated2. Corpora derived from CommonCrawl like CC100 (Conneau et al., 2020) and mC4 (Xue et al., 2021) are a popular source for major Indian languages. However, this text is often noisy and contains offensive content (Kreutzer et al., 2022a). IndicCorp v1 (Kakwani et al., 2020) is the first effort to curate a pretraining corpus exclusively for Indic languages. In this work, we build upon IndicCorp v1 to include more languages as well as crawl more data for existing languages.

heading: 2.2 Models
text: Most multilingual pretrained language models and their variants like mBERT (Devlin et al., 2019), mT5 (Xue et al., 2021), and XLM (Conneau and Lample, 2019) are trained on major Indic languages. However, it is difficult to get optimum performance from these models on Indic tasks as they have to compete for model capacity with other highresource languages (Conneau et al., 2020; Khanuja et al., 2022). Indic family-specific models like MuRIL (Khanuja et al., 2021) and IndicBERT v1 (Kakwani et al., 2020) do much better on such tasks than the aforementioned models.
2Apart from Hindi, which has 153,000 articles as of November 2022 all others have few thousand articles.

heading: 2.3 Benchmarks
text: Benchmarks like GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) have driven research on multitask models for English and IndicGULE (Kakwani et al., 2020) has been created to benchmark performance on Indic languages. Similarly, there have been multiple efforts to drive research on crosslingual, multitask models. Important among them are XGLUE (Liang et al., 2020), XTREME (Hu et al., 2020), and XTREME-R (Ruder et al., 2021). In order to accommodate a diverse set of languages, these benchmarks have a limited representation of Indic languages. Also, most evaluation sets are automatically translated or generated which is known to have problems (Vanmassenhove et al., 2021). In this work, we aim to fill this gap by presenting an Indic family-specific evaluation benchmark consisting of 9 tasks with human-created or human-translated test sets.

heading: 3 IndicXTREME
text: The IndicXTREME benchmark includes 9 tasks that can be broadly grouped into sentence classification (5), structure prediction (2), question answering (1), and sentence retrieval (1). Since the benchmark is designed to evaluate models in a zeroshot setting, we only create test sets. Table 2 gives a summary of the testsets in IndicXTREME.

heading: 3.1 New Contributions
text: IndicCOPA We manually translate the COPA (Roemmele et al., 2011) test set into 18 Indic languages to create IndicCOPA. The premise and the
choices from the original dataset are randomized and assigned to translators to avoid any bias. Once translated, the sentences are re-grouped. For finetuning, we use the English Social IQA dataset (Sap et al., 2019).
IndicQA We introduce IndicQA, a manually curated cloze-style reading comprehension dataset that can be used for evaluating question-answering models in 11 Indic languages. The context paragraphs are chosen from Wikipedia articles whose topics are closely related to Indic culture, history, etc. The dataset consists of 18,579 questions out of which 13,283 are answerable. A language-wise breakdown of the numbers can be seen in Table 7 in Appendix E. For more details about the collection process and annotation guidelines, see Appendix E.3. For fine-tuning of baseline models, we use the English SQuAD (Rajpurkar et al., 2016) dataset.
IndicXParaphrase We take 1001 English sentences from Kumar et al. (2022) with a mean sentence length of 17 words. We auto-translate these sentences into 10 languages using the IndicTrans translation model (Ramesh et al., 2022). Human annotators then verify (and correct, if required) these translations. Next, the annotators manually create paraphrases and non-paraphrases for each translated sentence. This results in 1001-way parallel <sentence, paraphrase, non-paraphrase> triplet in each of the 10 languages, where the sentences are shared across languages. The annotators are provided with strict guidelines to ensure the quality of the (non-)paraphrases. See Appendix F for more details about the annotation process. Contrary to prior works like Yang et al. (2019), we do not use back-translation or other noisy alignment methods to create non-paraphrases. For fine-tuning, we use the English part of the PAWS-X (Yang et al., 2019).
IndicSentiment In general, product reviews are one-dimensional and a vast majority of the reviews are highly polarized which makes classification easy. This results in models performing poorly on nuanced reviews. Therefore in this dataset, we ask annotators to create synthetic reviews for real products. We curate a list of aspects for each product category and ask the annotators to write reviews that talk about a subset of those aspects. All the reviews are first written in English and then manually translated to 13 Indic languages, thus making it a 13-way parallel dataset. More information about annotation guidelines can be found in Appendix
G. For fine-tuning, we use the English Amazon Multilingual Reviews dataset (Keung et al., 2020).

heading: 3.2 Other Datasets
text: IndicXNLI This dataset, already proposed in (Aggarwal et al., 2022) released an automatically translated version of XNLI (Conneau et al., 2018) in 11 Indic languages. Though the translations are generally good, there are certain quality issues that are a result of the dataset containing text that is a transcription of spoken language. This results in the translations being structurally and semantically incorrect. In this work, we manually verify the translations of some parts of the test set and make changes where necessary. Due to cost and time constraints, we could not verify the entire test set. Please see Table 9 in Appendix I to see the number of instances that were manually verified and corrected across languages. We plan to continue this effort and correct/verify the entire test set over a period of six months. For fine-tuning, we use the MultiNLI dataset (Williams et al., 2018).
Naamapadam This NER dataset was proposed in Mhaske et al. (2022)3 with manually curated testsets for nine Indic languages. The testsets have been created using the following process: (i) for an English-Indic language parallel sentence pair, the English sentence was NER tagged using an off-theshelf model, (ii) the NER tags were automatically projected to the Indic language sentence via word alignments, and (iii) the tags in the Indic sentence were verified and corrected by annotators. The annotations follow the standard IOB2 format. For training and validation, we use the CoNLL-2003 dataset (Tjong Kim Sang and De Meulder, 2003).
FLORES To evaluate the retrieval capabilities of models, we include the Indic parts of the FLORES101/200 dataset (Goyal et al., 2022; Costa-jussà et al., 2022) to IndicXTREME. This is an n-way parallel dataset containing 1012 sentences manually translated into 18 Indic languages. We do not perform any fine-tuning and use mean-pooled representations from the final layer of the models as sentence embeddings.
MASSIVE This intent classification and slotfilling dataset proposed by FitzGerald et al. (2022) is created using user queries collected by Amazon Alexa. The dataset contains 60 intents and 55 slot
3https://huggingface.co/datasets/ai4bharat/ naamapadam
types and is available in 51 languages. We take a subset of it consisting of seven Indic languages to be part of IndicXTREME. We use the English train and validation sets for training baseline models.
We reemphasise that ALL the evaluation sets included in IndicXTREME were created with human supervision. In other words, they were either translated or post-edited or created or verified by humans.

heading: 4 IndicCorp v2
text: In this section, we describe the process followed to build IndicCorp v2, the largest collection of texts for Indic languages consisting of 20.9 billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.5B tokens of Indian English content curated from Indian websites. Table 3 shows the size of the de-duplicated corpus across languages. The current corpus (24 languages) is 2.3× compared to IndicCorp v1 (12 languages) with the largest increase in Hindi (3.3×). The corpus contains 1.08 billion tokens from the bottom 11 low-resource languages.

heading: 4.1 Data
text: With the goal of creating a clean and diverse corpus, we choose news articles as our primary sources. In addition to the sources already discovered by Kakwani et al. (2020), we identify new sources for more languages through news repositories and automatic web searches. In particular, we determine the most frequent words that occur in a language and use these as queries for automated web searches. We identify URLs of sources that potentially contain content in those languages from the retrieved results. An analysis of the retrieved URLs shows that some of them are noisy with offensive content or machine-generated content. We, therefore, add a filtering stage wherein we ask human annotators to manually verify the URLs. Specifically, each annotator is asked to visit the URL and verify that it is a genuine website containing clean data in the language of interest. Across languages, we find that 1-33% of the URLs are noisy and we discard them. We then used the open-source toolkit webcorpus4 to crawl the shortlisted URLs.

heading: 4.2 Post-processing
text: We process the crawled dumps to produce clean text. We see that the crawls often contain data from
4https://gitlab.com/AI4Bharat/NLP/webcorpus
other languages. In order to remove such undesired text, we perform language detection-based (LID) filtering at paragraph level using cld35 and langdetect6 and discard text that is not in the language of interest. Note that low-resource languages like bd and dg are not supported by the libraries and hence we do not perform LID-based filtering for these languages.
Previous works suggest that data crawled from the web often contains offensive text (Kreutzer et al., 2022a). To remove such text from our corpus, we create a list of offensive words and phrases in 17 languages with the help of in-house annotators. In a parallel approach, a similar list of offensive words was released for 209 languages by Costajussà et al. (2022). We merge these two lists to create a comprehensive blacklist of words for all languages in the corpus. This list is used to filter text containing offensive content reducing the corpus size from 23.1 billion to 20.9 billion tokens. Following Kakwani et al. (2020), we add data from Wikipedia and OSCAR (Suarez et al., 2019) to our final corpus.

heading: 5 IndicBERT v2
text: This section describes the various aspects of training IndicBERT, a language model trained on IndicCorp and evaluated on IndicXTREME. In our experiments, we train with BERT architecture and ablate on objective functions and training data. Compared to IndicBERT v1 (Kakwani et al., 2020), trained on the smaller ALBERT (Lan et al., 2020) architecture, this version has ∼7.5x more param-
5https://github.com/google/cld3 6https://github.com/shuyo/language-detection
eters and is able to transfer across languages in zero-shot settings. The model has 278M parameters and supports all 24 languages in IndicCorp.
Training Objectives We experiment with two objective functions: Masked Language Modeling (Devlin et al., 2019, MLM) and Translation Language Modeling (Conneau and Lample, 2019, TLM). We use the document-level data created as part of IndicCorp for MLM objective training. Pretraining hyperparameters are listed in Appendix C.
Data As mentioned in Section 4.2, we merge data from IndicCorp v2 with Indic language data from Wikipedia and OSCAR. For MLM, we use these monolingual corpora spanning 24 languages, 5 language families, and 13 scripts. For TLM, we use language-parallel data from two sources: mined data from Samanantar corpus (Ramesh et al., 2022), and machine-generated English translations of the entire IndicCorp. We use IndicTrans (Ramesh et al., 2022) for all translations. We are limited in our ability to generate parallel sentences since IndicTrans supports only 11 of the 24 languages in IndicCorp. We perform ablations by training models on various subsets of this data as discussed in Section 6.2. Since data distribution across languages is skewed (Fig. 1 in Appendix B), we follow Khanuja et al. (2021) to upsample the underrepresented languages with 0.3 temperature coefficient.
Vocabulary We learn a WordPiece (Wu et al., 2016) vocabulary from a uniformly sampled fraction of the upsampled data. We also add special <lang-id> tokens to the vocabulary since Ramesh et al. (2022) have shown that training multilingual models with language tokens improve performance. These tokens are prepended to input documents during pretraining. Given that our model supports 24 languages and 13 scripts, we use a vocabulary size to 250K tokens. See Appendix K for more details.

heading: 6 Experiments
text: We compare IndicBERT v2 with the following LMs - IndicBERT v1 (Kakwani et al., 2020), mBERT (Devlin et al., 2019), XLMR (Conneau et al., 2020) and MuRIL (Khanuja et al., 2021). We describe our choice of baseline models, and their similarities and differences in Appendix D. We then briefly introduce our fine-tuning details and the various ablation studies conducted.

heading: 6.1 Fine-Tuning
text: The pre-trained LM is independently fine-tuned for each task in IndicXTREME. We perform zero-shot evaluation by fine-tuning the model on English and testing on the available Indic test sets. The best configuration of the model is chosen based on its performance on the English development set. While most works in literature (Khanuja et al., 2021; Conneau et al., 2020) use the same hyperparameters for fine-tuning models on various tasks, we find that task-specific hyperparameter-tuning improves performance. For a fair comparsion, we perform hyperparamter-tuning for all the models that we compare with. Our choice of hyperparameters for each task can be found in Tables 12, and 13 in the Appendix N. Models are fine-tuned for every task except for the retrieval task, where we directly use the mean pooled sentence representation from the last layer of the pretrained models.

heading: 6.2 IndicBERT v2 Ablations
text: We train four flavors of IndicBERT v2 to understand the role of parallel data and its quality in improving crosslingual performance. The first model is a vanilla BERT style model trained on IndicCorp v2 with the MLM objective. In the other two ablations, we include TLM as an additional objective with different sets of parallel data. In one ablation, we include parallel data from the Samanantar dataset.7 This corpus contains high-quality translations mined from various sources and supports 11 Indic languages. These models are denoted by (+Samanantar) in the results. Third, we translate the whole IndicCorp v2 to English using IndicTrans and use it as additional parallel data (+Back-Trans in results). Empirically, the quality of these translated parallel data is lower than those of Samanantar, especially for very low-resource languages like Assamese. Finally, to encourage better lexical sharing among languages we convert the scripts from Indic languages to Devanagari (IndicBERT-SS). All Indian languages are derived from the Brahmi script and there exists a 1-1 mapping between characters across different scripts. We convert all the supported languages to Devanagari script using IndicNLP Library (Kunchukuttan, 2020).

heading: 7 Results
text: The results for each task in IndicXTREME averaged across languages are shown in Table 4.
Massively Multilingual vs Indic Models It is clear that there is no single best model on the benchmark. However, IndicBERT v2 family of models beat the baselines in 7/9 tasks. The languagespecific results for all experiments can be found in Appendix O. When averaged across tasks (see Table 24), IndicBERT v2 performs the best on 17/20 languages. On average, the IndicBERT v2 family of models, outperform other models.
The results show that models trained only on Indic languages perform better since languages do not have to compete for model capacity. We see that IndicBERT v2 trained only on MLM, by itself performs much better than the standard baselines. The only exception to this is that MuRIL outperforms IndicBERT v2 in the paraphrase detection and NER tasks. We also see that adding the
7Samanantar data is sentence-level parallel and is not ideal. But document-level parallel data for Indic languages are scarce.
TLM objective with (i) high-quality parallel data increases the model performance across the board, and (ii) machine-translated data hurts performance.
Effect of Monolingual Corpora Table 4 compares the results for IndicBERT trained on IndicCorp v1 and v2. We can clearly see that model trained on the much larger v2 corpora performs better than model trained with v1 (see v1-data in Table 4), thereby establishing the utility of the larger monolingual corpora which we release as a part of this work.
Utilizing language similarity All models in Table 4 are optimized using English development sets. We can get better performance from these models if we have access to in-language development sets. This is not always possible since it may involve expensive and time-consuming human annotations. An alternate approach is to use machine-translated developments sets. For some languages, getting these translations is also impossible. In such cases, we might be able to use a surrogate development set from a different language that has similar linguistic properties. Often, this condition is satisfied by a
sibling language from the same family subtree. To test this hypothesis, we fine-tune models with in-language development sets if available, and compare their performance with those fine-tuned with in-family development sets. We use Hindi and Tamil development sets to select the best models for Indo-European and Dravidian languages respectively and the results are shown in Table 5. We see that models fine-tuned with in-family development sets generally perform on par with those fine-tuned with in-language sets, and give better performance than that obtained using English validation sets.
Shared Script Prior works Ramesh et al. (2022); Khemchandani et al. (2021) established that having a shared script model helps in lexical sharing leading to better performance. Taking inspiration from this, we train IndicBERT-SS. Largely the performance of IndicBERT-SS is comparable to models without script sharing, however, it does improve the performance of low resource languages written in Devanagari, see Tables 17, 23 in Appendix.
Transfer Languages We use English as the transfer language given the availability of sufficient training data for most tasks, but it might not be the best choice and another similar “related” language might be a better transfer language (Lauscher et al., 2020; Lin et al., 2019). We conduct a preliminary experiment to verify this observation on the Naamapadam and MASSIVE datasets for Indic languages (which contains both training and development sets in multiple languages). Here, we compare Hindi (a “related” language) with English as the transfer language (Table 6, gold zero-shot). We also compare this across models (Table 8). For NER we see a significant jump of 3.5 points when finetuning with Hindi. Similarly, for MASSIVE we see gains of 3.1 and 10.6 for Intent classification and slot filling respectively. These results suggest that it is useful to leverage training data in a related language. Prior work also suggests that finetuning with data translated to the transfer language
(Turc et al., 2021) or the target language (Aggarwal et al., 2022; Hu et al., 2020) (translate-train method) can perform better than when English is used as a transfer language. We plan to do further experiments with more tasks to investigate these observations broadly for Indic language settings. We call upon the community to create and share more in-language data, either through human annotation or (semi-)automated techniques.
Silver zero-shot To further test the hypothesis that zero-shot with “related” language results in better performance, we surrogate the English training data with translated data. Specifically, we translate the English training data for tasks to Hindi (w/ (Ramesh et al., 2022)) and use this for zero-shot transfer. For QA, we use the translation released by authors of Lewis et al. (2020). The results are shown in Table 4. We see that zero-shot with silver translation leads to much better performance than with English. The COPA task is generally described as a much harder task and even small perturbations in the data leads to bad performance. Similarly, translating QA datasets by preserving the answers spans is typically error prone, so we see a slight drop in performance for QA task.
“Winners” vs. “Left-Behinds” Table 24 presents language-wise results which are averaged across tasks. We can see a clear performance drop for extremely low-resource languages (those below the 10th percentile in Table 3). For example, Santhali and Sindhi performance on IndicXCOPA is 25.9% & 17.7% less than that for Hindi. Apart from lacking pretraining data, there are two other important reasons for this drop: (i) no shared script among languages, and (ii) no linguistic cousin in the corpus to act as a bridge for effective transfer. It is to be noted that IndicXTREME can only evaluate 19 of the 24 languages present in IndicCorp. There is an urgent need to build datasets for these “left-behind” languages.

heading: 8 Conclusion
text: Through this work, we distinctively contribute towards all the fundamental requirements of developing Indic language technologies; These include IndicCorp v2, the largest pretraining corpus for 24 Indic languages, IndicBERT v2 a language model pretrained on IndicCorp v2 and a holistic crosslingual NLU benchmark, IndicXTREME, for 20 Indic languages. We provide empirical evidence for our design decisions and show that pretraining models only on Indic languages result in much better performance on IndicXTREME.

heading: Acknowledgements
text: We would like to thank the Ministry of Electronics and Information Technology8 of the Government of India for their generous grant through the Digital India Bhashini project9. We also thank the Centre for Development of Advanced Computing10 for providing compute time on the Param Siddhi Supercomputer. We also thank Nilekani Philanthropies for their generous grant towards building datasets, models, tools and resources for Indic languages. We also thank Microsoft for their grant to support research on Indic languages. We also thank Google’s TPU Research Cloud (TRC) for giving us free access to their v3-128 TPUs for pretraining our models. We would like to thank Janki Nawale, Anupama Sujatha, and Krishnan Karunganni for their help in coordinating the annotation work. Most importantly we would like to thank all the annotators who spent their time helping create the IndicXTREME benchmark. We also thank Raghavan AK for helpful discussions on corpus cleaning and Harshita Diddee for insightful discussions on model pretraining.
Limitations
To create a clean and diverse corpus, we have chosen to crawl news articles as our primary data sources. Since all the articles are crawled from public domains, the data could potentially encompass the biases which propagate in public channels. Currently, the models trained on such data sources could model the inherent biases present within the data. In the current work, we do not perform any debiasing techniques and leave that for future work.
8https://www.meity.gov.in/ 9https://www.bhashini.gov.in/
10https://www.cdac.in/index.aspx?id=pune
Language Identification (LID) tools are restricted to a limited number of languages and unavailable for some of the very low-resource languages like Bodo, Dogri, Khasi, etc. We made our best effort to clean the corpus using Unicode spans, but it is possible that the data sources could have some issues. We leave developing LID tools for low-resource languages as part of future work.
From our ablation studies, we see that models are benefited by using in-language training and/or development sets. We call upon the community to work together to create more in-language data resources. Finally, there is still work required in terms of building datasets for hundreds of extremely low-resource languages not represented in this work.
Ethics Statement
Annotators who participated in the annotation and/or verification task are paid a competitive monthly salary to help with the tasks. The salaries were determined based on the qualification and the prior experience working on similar tasks and adhering to the norms of the government of our country. All the annotators were native speakers of the respective languages and from the Indian subcontinent. The annotators were made aware that the datasets will be publicly released. The annotated datasets have no personally identifying information. The annotated data and the crawled corpus have been checked for any offensive data and discarded if present.
The released code and models will have an MIT License11. The dataset will be released under a CC-0 License12.

heading: A Environmental Impact
text: IndicBERT and its variants are trained on 20.9 billion tokens encompassing 24 Indic languages. The models are trained on v3-128 TPUs.13 Each model takes 11 days to complete 1 million training steps and we estimate it to consume 9,562.9 kWh of energy with a carbon footprint of 5.4 MTCO2e. All models are further fine-tuned before downstream evaluation. These experiments are carried out on NVIDIA A100 GPUs and we estimate a total usage of 72 kWh of energy which is equivalent to 41.04 kg of CO2e. To limit the pretraining of such models from scratch, and to enable further research, we release all models trained as part of this work.

heading: B Data Distribution

heading: C Pretraining Hyperparameters
text: We use the default hyperparameters of BERT-Base with 12 encoder layers, and a maximum sequence length of 512. With 12 attention heads, a hidden dimension of 768, and feedforward network width of 3072, the model has 278 million parameters. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with α = 0.9 and β = 0.999. We use an initial learning rate of 5e-4 with a warm-up of 50,000 steps and linearly decay the learning rate till we reach the 1M steps. We use a global batch size of 4096 examples and train the model on v3128 TPUs. The models take 11 days to train. More details about environmental impact can be found in Appendix A.
13The TPUs reside in the Google Cloud Platform which is carbon neutral: https://cloud.google.com/ sustainability

heading: D Baseline language models
text: mBERT (Devlin et al., 2019) is one of the first massively multilingual models trained on 104 languages (11 Indic). It is trained on Wikipedia with exponentially smoothed weighting to rectify corpus imbalance. The model has 12 encoder layers with 768-dimensional embeddings and is trained with the MLM objective. It has a vocabulary size of 119,000 and 172 million parameters.
XLM-R (Conneau et al., 2020) is the multilingual version of RoBERTa (Liu et al., 2019) that is trained on the CC-100 dataset with 100 languages (15 Indic). The model has the same architecture as BERT but has an optimized hyperparameter set. It drops the next-sentence prediction (NSP) objective from the original BERT implementation and uses a combination of MLM and TLM objectives for training. It has a vocabulary size of 250,000, and 278 million parameters.
IndicBERT v1 (Kakwani et al., 2020) is a multilingual ALBERT (Lan et al., 2020) model trained on IndicCorp v1. The model supports 11 Indic languages. It is smaller than most multilingual models with 33 million parameters.14 It has a vocabulary size of 200,000 and uses temperature sampling to balance the data across languages. It is trained with the MLM objective, a smaller maximum sequence length of 128, and on sentences instead of the standard practice of training on whole documents.
MuRIL (Khanuja et al., 2021) is a multilingual BERT model trained exclusively on 16 Indic languages, with data taken from Wikipedia, OSCAR, PMI corpus (Haddow and Kirefu, 2020), and the Dakshina dataset (Roark et al., 2020). While it follows standard hyperparameter settings and corpus balancing tricks, it stands out by using silvertranslated and transliterated data, along with their gold counterparts. It has a vocabulary of 197,000 tokens, 237 million parameters and is trained with both MLM and TLM objectives.
E IndicQA
E.1 Article Selection A list of topics related to Indic history, monuments, authors, politicians, festivals, etc., was manually collected. The topics were then ranked by the number of Indic language Wikipedias they appeared
14Given its small size, we do not perform extensive ablations on this model.
in after discarding those that had less than 10 sentences (on average) in their articles. Finally, the articles of the top-ranking topics were used to create the QA pairs.
E.2 Annotation Process
From the shortlisted articles, paragraphs containing 8-10 sentences were used as context.15 Previous works have shown that annotators often create questions that have a high lexical overlap with the context paragraphs. To avoid this, we divide the collection process into two phases.
In Phase one, each context is first split into two parts where the second part is smaller, usually containing 2-3 sentences. Both these context paragraphs are then translated into English with Google Translate 16. The annotators are asked to create questions (in an Indic language) from these translated context paragraphs. This intermediate translation step ensures that the lexical overlap is reduced since the annotators cannot copy a sentence and turn it into a question by prepending a wh word.
In Phase two, the first part of the original context paragraph (in an Indic language) is presented to a different annotator and is asked to mark the answer spans for the questions created previously. Since the second part of the context is not provided, the questions created from them become unanswerable.
On average there were 2-3 annotators per language and all the annotations were done on Haystack tool17.
E.3 Annotation Guidelines
The annotators were given a set of detailed guidelines to avoid problems seen in previous QA
15Smaller paragraphs were merged. 16http://translate.google.com 17https://github.com/deepset-ai/haystack
datasets. The list of guidelines for question creation is as follows: (i) Create a minimum of two questions from each paragraph, (ii) The answers should not have a span of more than five continuous words, (iii) The questions should be unambiguous and understandable even if the context is not provided, (iv) Try to minimize phrase overlapping between the context paragraph and question, and (v) Create questions in such a way that the answer span is contained within a single sentence of the paragraph. The list of guidelines for answer marking is as follows: (i) The answer should always be a continuous span whose length is not more than five words, (ii) An entire sentence cannot be marked as an answer, (iii) The answer cannot be a pronoun, and (iv) If the context paragraph contains multiple occurrences of the answer string, always mark the one which is most relevant to the question.
F IndicXParaphrase
We randomly choose 1001 English sentences from the dataset introduced in Kumar et al. (2022), such that each sentence is at least 10 words long. Next, we machine-translate these sentences into the required languages using the IndicTrans model. Following this, we ask annotators across languages to (i) verify and correct the translations, if required, and (ii) create one paraphrase and a non-paraphrase for each sentence. The instructions to the annotators are as follows: (i) minimize word overlap between the sentence and the paraphrase, (ii) use temporal phrase swapping where ever possible, e.g., he fell and got hurt → he got hurt when he fell (iii) swap active and passive voice, (iv) use synonyms liberally.
For creating sentences that are not paraphrased, the annotators are instructed to swap named entities, pronouns, adjectives, adverbs, etc. where possible. An example for named entity swapping: John drove Jane to the market → Jane drove John to the market. They are also instructed to restrict the use of negation and antonyms unless necessary. There were 2 annotator per language and the whole task has been carried out on Google Sheets.
G IndicSentiment
We curate a list of products from 15 popular categories from online marketplaces like Amazon18,
18https://amazon.in
Flipkart19 and Myntra20. For each product, we first ask annotators to list aspects of the product that they deem important. We then ask a different set of annotators to write reviews for the products, based on the aspects provided in the previous step. We encourage annotators to be natural and draw from their experiences of using the same, or a similar product. We instruct annotators not to use offensive language in the reviews. For example, for the product category dress, we ask the annotators to write both positive and negative reviews by concentrating on one or more of the following aspects: material, color, and sleeves. The reviews are initially written in English and then manually translated into other languages. There were 2 annotator per language and the whole task has been carried out on Google Sheets.

heading: H Naamapadam
text: Results for NER task using Hindi data from Naamdapadam. We perform ablations comparing zeroshot transfer via English and Hindi.
I IndicXNLI
Our effort to manually correct all the translations in the IndicXNLI (Aggarwal et al., 2022) dataset is currently ongoing. Table 9 & Table 10 shows the current status of the project & current scores, respectively, across all 11 Indic languages. Once the complete test set is verified and cleaned, we plan to update IndicXTREME with the additional data.
J IndicCorp Data Cleaning
Since most of our data come from Indic news websites, we discover source URLs through online
19https://flipkart.com 20https://myntra.com
newspaper directories (e.g., w3newspaper21) and through automated web searches using hand-picked terms in various languages. We manually identify spam websites from the list of sources and remove them.
Language Identification We use cld322 and langdetect23 to detect the language of an article. We use both in parallel since cld3 does not identify Assamese and Oriya.
Script-based cleaning Often sentences contain transliterations and phrases in other languages, especially English. Therefore, we use Unicode-based verification to determine if sentences are in their native script. We remove a sentence from the corpus if the ratio of the number of characters in the native script to the total number of characters is less than 0.75.
Punctuation-based cleaning We strip punctuation from sentences and if the length of the stripped document is less than 10 words, then we remove the document from the corpus.
21https://www.w3newspapers.com/ 22https://github.com/google/cld3 23https://github.com/shuyo/language-detection
Offensive word filtering We collect an exhaustive list of offensive words/phrases from online sources, and native speakers.24 On average, we curated close to 90 words/phrases per language. When suggested by native speakers, we also add ambiguous words to the list, which are not offensive on their own but can be used in offensive contexts.
Sentences containing at least one word from the list are removed from the corpus. In the case of offensive phrases, we remove a sentence only if the whole phrase appears in the sentence.

heading: K Tokenizers
text: Fig. 2 compares the fertility scores (Ács, 2019) of the IndicBERT tokenizer with that of mBERT, XLM-R, and MuRIL. We see that the IndicBERT tokenizer has consistently lower fertility scores across languages which suggests that its vocabulary contains a larger fraction of tokenized words that do not need to be split into subwords. Fertility ratio is higher for mni due to script mismatch between FLORES (Bengali) and IndicCorp (Meitei).

heading: L Language Identification
text: Since IndicBERT is pretrained with prepended <lang-id> tags, we evaluate its language identification ability without any fine-tuning. We use the FLORES devtest split for this evaluation. We pass the input sentences by prepending the [MASK] token and expect the model to replace it with the appropriate <lang-id>. For this experiment, we only consider top-1 accuracy. See Table 11 for results. Apart from Manipuri, IndicBERT identifies
24The words/phrases obtained from online sources were manually verified by native speakers.
all other languages with high accuracy. It cannot identify Manipuri since FLORES uses the Bengali script for Manipuri, whereas IndicCorp uses Meitei.
M Impact of pre-training data size
As expected, we can see from Fig.3 that as the size of pretraining data increases, there is an increase in downstream performance as well. This holds for all tasks across languages, except for IndicXParaphrase. It just holds for Naamapadam (NER) albeit with a high variance. As mentioned in Section 7, we hypothesize that this could be due to the model’s inability to learn good representations for noun phrases which play a major role in resolving named entities and paraphrase detection.

heading: N Fine-tuning Hyperparamters
text: We perform a grid search over learning rates [1e-5, 3e-5, 5e-6] and weight decay [0, 0.01] to choose the best model across tasks and languages. We report the best hyperparameters for English, in-language, and in-family validation sets. Table 12 shows the best configuration for IndicCOPA and IndicXParaphrase for which only English validation sets are available. Table 13 shows the best configurations for all other tasks for which both in-language and in-family validation sets are available.
For intent classification and slot-filling tasks, we use the same hyperparameter setting since they come from the same underlying data. We use a learning rate of 1e-5, weight decay of 0.1, and batch size of 256. For all the best models, unless otherwise mentioned we use a batch size of 32, and
train with an initial warmup of 10%. All the models are fine-tuned with half-precision on NVIDIA A100 GPUs.

heading: O Language-wise Results
text: Tables 15, 16, 17, 18, 19, 20, 21, 22, 23 show the language-wise results for IndicSentiment, IndicXNLI, IndicCOPA, IndicXParaphrase, MASSIVE Intent Classification, Naamapadam, MASSIVE Slot-filling, IndicQA, and FLORES sentence retrieval tasks respectively.

heading: P Language Classes
text: Table 14 contains more information about each language in IndicCorp. We want to emphasize the diversity present in the corpus, and the differences in the size of resources available across languages through the classes to which they are assigned by Joshi et al. (2020).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations (after the conclusion)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations (after the conclusion)

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes, we create datasets and models whose descriptions are available in Section 3 and Section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We use existing datasets and pertained models and cited them in Section 3.2 and Section 6

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We discuss this in the Ethics Statement (after Limitations)

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discuss the intended use of our dataset in Section 1.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We discuss this is Section 4

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We discuss this in Section 1 and Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We discuss this in Section 3 and Section 4
C 3 Did you run computational experiments? Section 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In Appendix Sections C, N
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In Section 7 and Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 7 and Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix E, F, G, H, I

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? In Ethics Section

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? In Ethics Section

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: In Ethics Section

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: In Ethics Section

heading: Future Work
text: In the current work, we do not perform any debiasing techniques and leave that for future work. 8https://www. meity. gov. in/ 9https://www. bhashini. gov. in/
10https://www. cdac. in/index. aspx?id=pune
Language Identification (LID) tools are restricted to a limited number of languages and unavailable for some of the very low-resource languages like Bodo, Dogri, Khasi, etc. We made our best effort to clean the corpus using Unicode spans, but it is possible that the data sources could have some issues. We leave developing LID tools for low-resource languages as part of future work. From our ablation studies, we see that models are benefited by using in-language training and/or development sets. We call upon the community to work together to create more in-language data resources. Finally, there is still work required in terms of building datasets for hundreds of extremely low-resource languages not represented in this work