IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languages (2024)

Abstract: In this paper, we introduce Neural Information Retrieval resources for 11 widely spoken Indian Languages (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu) from two major Indian language families (Indo-Aryan and Dravidian). These resources include (a) INDICMARCO, a multilingual version of the MS MARCO dataset in 11 Indian Languages created using Machine Translation, and (b) IndicColBERT, a collection of 11 distinct Monolingual Neural Information Retrieval models, each trained on one of the 11 languages in the INDIC-MARCO dataset. To the best of our knowledge, IndicIRSuite is the first attempt at building large-scale Neural Information Retrieval resources for a large number of Indian languages, and we hope that it will help accelerate research in Neural IR for Indian Languages. Experiments demonstrate that Indic-ColBERT achieves 47.47% improvement in the MRR@10 score averaged over the INDIC-MARCO baselines for all 11 Indian languages except Oriya, 12.26% improvement in the NDCG@10 score averaged over the MIRACL Bengali and Hindi Language baselines, and 20% improvement in the MRR@100 Score over the Mr. Tydi Bengali Language baseline. IndicIRSuite is available at github.com/saifulhaq95/IndicIRSuite.

heading: 1 Introduction
text: Information Retrieval (IR) models process user queries and search the document corpus to retrieve a ranked list of relevant documents ordered by a relevance score. Classical IR models, like BM25 (Robertson et al., 2009), retrieve documents that have lexical overlap with the query tokens. Recently, there has been a notable upsurge in adopting Neural IR models utilizing language models such as BERT (Devlin et al., 2018), which enable semantic matching of queries and documents. This shift
has proven highly effective in retrieving and reranking documents. ColBERTv2 (Santhanam et al., 2021), one of the state-of-art neural IR models, has shown 18.5 points improvement in NDCG@10 Score over the BM25 model baseline on the MS MARCO dataset (Thakur et al., 2021).
The importance of dataset size outweighs domain-matching in training neural IR models (Zhang et al., 2022a). Due to the scarcity of largescale domain-specific datasets, Neural IR models are first trained on the MS MARCO passage ranking dataset (Nguyen et al., 2016), and they are subsequently evaluated on domain-specific datasets in a zero-shot manner. MS MARCO dataset contains 39 million training triplets (q, +d, -d) where q is an actual query from the Bing search engine, +d is a human-labeled passage answering the query, and -d is sampled from unlabelled passages retrieved by the BM25 model. The MS MARCO dataset is in English, implying that neural IR models trained on it are effective only with English queries and passages.
Monolingual IR for non-English languages (Zhang et al., 2022b) (Zhang et al., 2021), Multilingual IR (Lawrie et al., 2023), and Cross-lingual IR (Lin et al., 2023; Sun and Duh, 2020) extend the English IR paradigm to support diverse languages. In Monolingual IR for non-English languages, the query and passages are in the same language, which is not English. In cross-lingual IR, the query is used to create a ranked list of documents such that each document is in the same language, which is different from the query language. In Multilingual IR, the query is used to create a ranked list of documents such that each document is in one of the several languages, which can be the same or different from the query language. In this work, we focus on Monolingual IR for non-English languages.
Monolingual IR for non-English languages involves training an encoder like mBERT (Devlin et al., 2018), on a large-scale general-domain
501
monolingual dataset for non-English languages to minimize the pairwise softmax cross-entropy loss. The trained models are subsequently finetuned or used in a zero-shot manner on small-scale domainspecific datasets. However, there is a notable lack of large-scale datasets like mMARCO (Bonifacio et al., 2021) for training monolingual neural IR models on many low-resource Indian languages. We introduce neural IR resources to address this scarcity and facilitate Monolingual neural IR across 11 Indian languages. Our contributions are:
• INDIC-MARCO, a multilingual dataset for training neural IR models in 11 Indian Languages (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil and Telugu). For every language in INDIC-MARCO, there exists 8.8 Million passages, 1 Million queries, 39 million training triplets (query, relevant document, irrelevant document), and approximately one relevant document per query. To the best of our knowledge, this is the first large-scale dataset for training a neural IR system on 11 widely spoken Indian languages.
• Indic-ColBERT, a collection of 11 distinct Monolingual Neural Information Retrieval models, each trained on one of the 11 languages in the INDIC-MARCO dataset. IndicColBERT achieves 47.47% improvement in the MRR @10 score averaged over the INDICMARCO baseline for all 11 Indian languages except Oriya, 12.26% improvement in the NDCG @10 score averaged over the MIRACL Bengali and Hindi Language baselines, and 20% improvement in the MRR@100 Score over the Mr. Tydi Bengali Language baseline. To the best of our knowledge, this is the first effort for a neural IR dataset and models on 11 major Indian languages, thereby providing a benchmark for Indian language IR.

heading: 2 Related work
text: The size of datasets holds greater importance than ensuring domain matching in the training of neural IR models (Zhang et al., 2022a). In terms of size and domain, mMARCO (Bonifacio et al., 2021) is the most similar to our work as it introduces a largescale machine-translated version of MS MARCO in many languages, Hindi being the only Indian language. MIRACL (Zhang et al., 2022b) and Mr.
Tydi (Zhang et al., 2021) also introduce datasets and models for Monolingual Neural IR in Hindi, Bengali, and Telugu.
FIRE1 was the most active initiative from 2008 to 2012 for Multilingual IR in Indian languages. FIRE developed datasets for Multilingual IR in six Indian Languages (Bengali, Gujarati, Hindi, Marathi, Oriya, and Tamil). However, the size of these datasets is not large enough to train neural IR systems based on transformer models like mBERT (Devlin et al., 2018) and XLM (Lample and Conneau, 2019). In addition, the text in the FIRE dataset comes from newspaper articles (Palchowdhury et al., 2013), which is domain-specific; hence, the models trained on such datasets cannot generalize well to other domains. Due to the lack of large-scale datasets, Cross-lingual knowledge transfer via Distillation has become popular for neural IR in low-resource languages (Huang et al., 2023a) (Huang et al., 2023b).
The key distinction in our work from the earlier approaches is that we introduce monolingual datasets and neural IR models in 11 major Indian Languages (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil and Telugu), that can also benefit Crosslingual and Multilingual IR models from the crosslingual transfer effects when trained on a large number of Indian Languages (Zhang et al., 2022a).

heading: 3 Datasets

heading: 3.1 INDIC-MARCO
text: We introduce the INDIC-MARCO dataset, a multilingual version of the MS MARCO dataset. We translate the queries and passages in the MS MARCO passage ranking dataset into 11 widely spoken Indian languages (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil and Telugu) originating from two major language families (Indo-Aryan and Dravidian). The translation process utilizes the int-8 quantized version of the NLLB-1.3B-Distilled Model (Costa-jussà et al., 2022), available at CTranslate22 (Klein et al., 2020). We chose int-8 quantized version of NLLB-1.3B-Distilled Model for two reasons: (a) it has shown remarkable performance in terms of BLEU scores for many Indian languages as compared to IndicBART (Dabre et al., 2021)
1http://fire.irsi.res.in/fire/static/data 2https://forum.opennmt.net/t/nllb-200-with-
ctranslate2/5090
and IndicTrans (Ramesh et al., 2022) (b) Quantization (Klein et al., 2020) enables faster inference with less computing power and little or no drop in translation quality. The machine translation process employs specific hyper-parameters: a beam width of 4, a maximum decoding sequence length of 200 tokens, a batch size of 64, and a batch type equal to ‘examples’. Passages from the MS MARCO dataset are split into multiple sentences using the Moses SentenceSplitter3, ensuring that each sentence serves as a translation unit in a batch of 64 sentences. In contrast, queries with an average length of 5.96 words (Thakur et al., 2021) are not sentence-split before translation. We also translate the MS MARCO Dev-Set(Small)4 containing 6,390 queries (1.1 qrels/query) to obtain INDIC-MARCO Dev-set(Small). The translation process on an Nvidia A100 GPU with 80 GB VRAM takes approximately 1584 hours for passages in MS MARCO, 55 hours for queries in MS MARCO, and 1.5 hours for queries in MS MARCO Dev-Set(Small). Upon translation, the resulting INDIC-MARCO dataset comprises around 8.8 million passages, 530k queries, and 39 Million training triplets in 11 Indian languages. This dataset allows for training monolingual neural IR models for each language in the INDIC-MARCO dataset.

heading: 4 Models

heading: 4.1 Baselines
text: BM25 (Robertson et al., 2009) serves as a strong baseline as it performs better than many neural IR models on domain-specific datasets with exceptions (Thakur et al., 2021). It does not require any training. BM25 retrieves documents containing query tokens and assigns them a score for reranking based on the frequency of query tokens appearing in them and the document length. In this work, we use the BM25 implementation provided by Pyserini5 with values for parameters k1=0.82 and b=0.68 for evaluation on INDIC-MARCO DevSet obtained after machine translation. We use Whitespace Analyzers to tokenize queries and documents during indexing and searching for all Indian languages except Hindi, Bengali, and Telugu, for which we use language-specific analyzers provided in Pyserini. BM25-tuned (BM25-T) presented in
3https://pypi.org/project/mosestokenizer/ 4https://ir-datasets.com/MS MARCO-passage.html 5https://github.com/castorini/pyserini
Mr. Tydi (Zhang et al., 2021) is optimized to maximize the MRR@100 score on the Mr. Tydi test-set using a grid search over the range [0.1, 0.6] for k1 and [0.1, 1] for b.
Multilingual Dense Passage Retriever (mDPR) is presented in both Mr. Tydi and MIRACL by replacing the BERT encoder in Dense Passage Retriever(DPR) (Karpukhin et al., 2020) with an mBERT encoder. In Mr. Tydi, mDPR is trained on English QA dataset (Kwiatkowski et al., 2019) and used in a zero-shot manner for indexing and retrieval of documents. In MIRACL, mDPR is trained on the MS MARCO dataset and used in a zero-shot manner for indexing and retrieving documents. Multilingual ColBERT (mCol) is introduced in MIRACL by replacing the BERT encoder in ColBERT (Santhanam et al., 2021) with an mBERT encoder. mCol is trained on the MS MARCO dataset and used in a zero-shot manner for indexing and retrieval of documents.

heading: 4.2 Indic-ColBERT
text: Indic-ColBERT (iCol) is based on ColBERT (Khattab and Zaharia, 2020) for training and ColBERTv2 (Santhanam et al., 2021) for compression and inference. There are some distinctions: it uses mBERT as query-document encoder, and is trained on INDIC-MARCO. Model architecture comprises (a) a query encoder, (b) a document encoder, and (c) max-sim function (same as ColBERTv2). Given a query with q tokens and a document with d tokens, the Query encoder outputs q fix-sized token embeddings, and the document encoder outputs d fix-sized token embeddings. The maximum input sequence length for the query, qmax, and, for the document, dmax, is set before giving them to the respective encoders. If q is less than qmax, we append qmax − q [MASK] tokens to the input query, and if q is greater than qmax, q is truncated to qmax. If d is less than dmax, then d is neither truncated nor padded. If d is greater than dmax, d is truncated to dmax. The max-sim function is used to obtain the relevance score of a document for a query using the encoded representations.

heading: 5 Experiment Setup
text: We train 11 distinct Indic-ColBERT (iCol) models separately for 50k iterations with a batch size of 128 on the first 6.4 million training triplets from the INDIC-MARCO dataset to optimize the pairwise softmax cross entropy loss function, where each
triplet contains a query, a relevant passage and an irrelevant passage in one of the 11 languages on which the model is trained. The mBERT encoder is finetuned from the official "bert-base-multilingualuncased" checkpoint, and the remaining parameters are trained from scratch.

heading: 6 Results
text: Indic-ColBERT (iCol) outperforms baseline models (BM25, BM25-T, mDPR, mCol) by 20%, in MRR@100 Score and on Mr. Tydi test-set (Refer Table 2) for Bengali Language. For Telugu, Indic-ColBERT (iCol) outperforms 3 (BM25, mDPR, mCol) out of 4 baselines in terms of MRR@100 scores. Indic-ColBERT (iCol) outperforms baseline models (BM25, mDPR, mCol) by 19.29% in Bengali and 5.4% in Hindi, in NDCG@10 Score on MIRACL dev-set(Refer Table 2). For Telugu, Indic-ColBERT (iCol) outperforms 2 (mDPR, mCol) out of 3 baselines in terms of NDCG@10 scores. Indic-ColBERT (iCol) outperforms baseline models (BM25, mCol) by 47.47% in MRR@10 Score on INDIC-MARCO
Dev-Set(Small) (Refer Table 1) averaged over all 11 Indian languages (excluding Oriya).
We do not see any improvements for Oriya because mBERT used in Indic-ColBERT is not pretrained on Oriya and Assamese. Assamese demonstrates a 125% MRR@10 improvement over the BM25 baseline, attributed to its linguistic similarity with Bengali (indicated by the mColBERT model outperforming BM25 by 21% in MRR@10 Score) and the high-quality data in INDIC-MARCO, further enhancing the MRR@10 score by 104%, making INDIC-MARCO a significant contributor to the advancement for a low-resource language like Assamese which mBERT does not support.

heading: 7 Ablation Study
text: In this section, we perform ablation study with three different machine translation models and two different document splitting schemes. We compare the NDCG@10 scores of Indic-ColBERT models trained on machine translated MS-MARCO data using NLLB-600M, NLLB-1.3B and IndicTrans2. As shown in Table 4, the impact of translation quality
on retrieval effectiveness follows a different trend for each language. In terms of chrF++ score, IndicTrans2 performs better than NLLB-1.3B which performs better than NLLB-600M on Flores-200 devtest (Gala et al., 2023) (Costa-jussà et al., 2022). For Telugu, we observe a negative correlation between translation quality and retrieval effectiveness, where the Indic-Colbert trained on data translated using NLLB-600M model, which has the lowest chrF++ score among the three machine translation models, gives the best retrieval effectiveness. For Hindi, we observe a positive correlation between the translation quality and retrieval effectiveness. For Bengali, we don’t observe any correlation between translation quality and retrieval effectiveness.
Each document in MS-MARCO dataset is first split into sentences, each sentence is translated by the machine translation model and finally the translated sentences are merged back into the document. We experimented with two different document splitting schemes. We compare the NDCG@10 scores for Indic-ColBERT models trained on machine translated MS-MARCO dataset using NLLB-1.3B model on sentences obtained from Moses Splitting and Full-stop Splitting schemes. As shown in Table 4, we can observe "NLLB-1.3B + Full-Stop Splitting" outperforms "NLLB-1.3B + Moses Splitting" for Hindi and Bengali Languages.

heading: 8 Summary, conclusion, and future work
text: We present IndicIRSuite, featuring INDICMARCO, a multilingual neural IR dataset in 11 Indian languages, and Indic-ColBERT, comprising 11 monolingual neural IR models based on ColBERTv2. Our results demonstrate performance enhancements over baselines in Mr. Tydi, MIRACL, and INDIC-MARCO, particularly benefiting low-resource languages like Assamese. INDICMARCO proves valuable for such languages, not supported by models like mBERT but linguistically akin to Bengali. We also perform an ablation to find the impact of translation quality and sentence splitting on retrieval effectiveness. Future work includes expanding IndicIRSuite to Multilingual and Crosslingual IR.
Limitations
The primary limitation of our study is the absence of a comprehensive comparison of the trained IR models across out-of-domain datasets beyond MIRACL and Mr. Tydi. It is imperative to delve deeper into the translation quality, specifically assessing whether it exhibits pronounced "translationese." A more exhaustive examination is warranted, particularly in cases where the proposed models, such as Indic-ColBERT, demonstrate subpar performance compared to baseline models, as observed in the instance where Indic-ColBERT lags behind the BM25 Baseline for the Telugu Language in Mr. Tydi test-set and MIRACL Dev-set.
Ethics Statement
We want to emphasize our commitment to upholding ethical practices throughout this work. This work publishes a large-scale machine-translated dataset for neural information retrieval in 11 Indian languages - Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu. MS MARCO passage ranking Dataset in the English language used as a Source dataset for translation is publicly available, and no annotators were employed for data collection. We have cited the datasets and relevant works used in this study.

heading: A Examples
text: Snapshots from the INDIC-MARCO dataset are shown in Figure 1, Figure 2 and Figure 3.

heading: Future Work
text: Future work includes expanding IndicIRSuite to Multilingual and Crosslingual IR. Limitations
The primary limitation of our study is the absence of a comprehensive comparison of the trained IR models across out-of-domain datasets beyond MIRACL and Mr. Tydi. It is imperative to delve deeper into the translation quality, specifically assessing whether it exhibits pronounced "translationese. " A more exhaustive examination is warranted, particularly in cases where the proposed models, such as Indic-ColBERT, demonstrate subpar performance compared to baseline models, as observed in the instance where Indic-ColBERT lags behind the BM25 Baseline for the Telugu Language in Mr. Tydi test-set and MIRACL Dev-set