Venue: NAACL 2025

# Benchmarking and Building Zero-Shot Hindi Retrieval Model with

Hindi-BEIR and NLLB-E5

Arkadeep Acharya1 , Rudra Murthy2, Vishwajeet Kumar2, Jaydeep Sen2,

Work done as an intern at IBM Research

###### Abstract

Given the large number of Hindi speakers worldwide, there is a pressing need for robust and efficient information retrieval systems for Hindi. Despite ongoing research, comprehensive benchmarks for evaluating retrieval models in Hindi are lacking. To address this gap, we introduce the **Hindi-BEIR** benchmark, comprising 15 datasets across seven distinct tasks. We evaluate state-of-the-art multilingual retrieval models on the Hindi-BEIR benchmark, identifying task and domain-specific challenges that impact Hindi retrieval performance. Building on the insights from these results, we introduce **NLLB-E5**, a novel multilingual retrieval model that leverages a zero-shot approach to support Hindi without the need for Hindi training data. We believe our contributions, which include the release of the **Hindi-BEIR** benchmark and the **NLLB-E5** model, will prove to be a valuable resource for researchers and promote advancements in multilingual retrieval models. The datasets from Hindi-BEIR are publicly available at **Hindi-BEIR**.The training and evaluation code for the NLLB-E5 model can be found on GitHub: **NLLB-E5 Repository**.

## 1 Introduction

Information retrieval (IR) models are indispensable in our digital age, enabling swift and accurate extraction of relevant data from vast amounts of information. These models enable quick and accurate retrieval of relevant data, thereby facilitating informed decision-making across various downstream applications and domains. In this modern era of LLMs, retrievers have become even more relevant and almost necessary in curbing hallucinations from Large Language Model (LLM) generations by means of much popular retrieval augmented generation (RAG) Lewis et al. (2021) methods. However, the focus of information retrieval (IR) research has predominantly centered on the English language, which can be attributed to the availability of comprehensive evaluation benchmarks, such as the BEIR Thakur et al. (2021). Although, more recently, there have been efforts to build multi-lingual retrievers and associated evaluation benchmarks Zhang et al. (2023, 2021) on non-English languages, multi-lingual expansion of robust benchmarks is still a work in progress, with many languages like Hindi lacking a BEIR like a comprehensive benchmark.

Hindi is one of the official languages of India1, a language of immense importance and global reach, with over half a billion speakers worldwide and ranking as the third most widely spoken language2. However, from NLP research point of view, Hindi can be categorized as a low-resource language, lacking comprehensive resource and benchmarks to advance scientific research and this includes lack of a good retrieval benchmark too. Though there has been some efforts in recent time for the development of Indic language dataset for various NLP task like Question answering Sabane et al. (2024), Doddapaneni et al. (2023a)) and Summarization Datta et al. (2023), Ghosh et al. (2024a), Ghosh et al. (2024b)), the evolution of Hindi or, in general, Indic language retrieval datasets has been rather ad-hoc, restricted to a small subset of domains, languages and tasks. Although a recent effort attempts to incorporate multiple Indic languages Haq et al. (2023) for a retrieval benchmark, the absence of diversity in domains and tasks remains a major limitation, affecting the robust evaluation of the current state-of-the-art retriever models in practical applications, which often involve varying tasks and domains.

Footnote 1: [https://en.wikipedia.org/wiki/Langaugages_of_India](https://en.wikipedia.org/wiki/Langaugages_of_India)

In this work, we posit that creating a BEIR-like diversified and robust benchmark is the necessary

first step to track and advance any retrieval research effectively in a new language. Such a benchmark will be more useful than ad-hoc datasets with multiple languages. Thus, we choose the most widely spoken language among Indic languages, Hindi, as our pivot language to pioneer a large-scale diversified retrieval benchmark **Hindi-BEIR**, spanning 15 diverse datasets across \(7\) tasks. **Hindi-BEIR** aims for two key objectives:(1) Establish a standardized retrieval benchmark to assess, compare, and advance the state-of-the-art retriever models and (2) Provide workable insights into the potential research directions on building retrieval models for Hindi. With multiple tasks and domains in **Hindi-BEIR**, we posit that **Hindi-BEIR** will provide a more realistic and accurate assessment of retriever models for zero-shot usage in unseen domains and applications.

On the choice of Hindi language it is important to note here, besides being a language of high usage, a retrieval benchmark in Hindi offers some unique challenges too such as:

\(\bullet\)**Script Difference:** Hindi uses the Devanagari script, which is fundamentally different from the Latin script used in English. This affects character encoding, text normalization, and processing. Existing tokenizers trained on English data may not handle Hindi text well, thus needing to evaluate various tokenization strategies too.

\(\bullet\)**Grammatical Structure:** Hindi predominantly uses Subject-Object-Verb (SOV) word-order as opposed to the Subject-Verb-Object (SVO) word-order in English. Hindi words often include more inflections and agglutinations, affecting word tokenization and testing the robustness of retrieval models.

\(\bullet\)**Ambiguity:** In Hindi, some proper nouns can also function as common nouns. For instance, the name _Lata_, a common female name, can also refer to _creeper_, a common noun. A query like _lata ko kaise saaph karen_ (how to clean a creeper) can easily mislead lexicon-based retrieval systems and test a model's ability for word sense disambiguation.

The introduction of **Hindi-BEIR** benchmark lets us assess and compare the existing multi-lingual retriever models more accurately for their zero-shot performance in Hindi across diverse domains and tasks covered in the benchmark. Based on this performance comparison, we notice that a key limitation of existing models is the need for a substantial amount of language-specific training data, which is often not available for low-resource languages such as Hindi. To address this problem, we propose **NLLB-E5**, a multilingual retrieval model, inspired by recent works like Langbridge Yoon et al. (2024) and NLLB-LLM2Vec Schmidt et al. (2024), which distills information from a monolingual retrieval model into a multilingual retrieval model in a zero shot setup via a multi-lingual encoder. The key strength of **NLLB-E5** lies in its ability to perform distillation without reliance on multilingual training data, thereby eliminating the necessity for supervised multilingual datasets.

We summarize our contributions as follows:

* We introduce **Hindi-BEIR**, the first comprehensive retrieval benchmark in the Hindi language, encompassing 15 diverse datasets from 7 distinct tasks. This was accomplished through the harmonization of high-quality quality verified translated data from BEIR, the conversion of existing Hindi datasets for retrieval purposes, and the introduction of strategically generated synthetic data.
* We develop **NLLB-E5**, which combines a multilingual encoder with a monolingual retrieval model through knowledge distillation to develop a multilingual retrieval model without the need for any multilingual training data.
* We evaluate **NLLB-E5** on Hindi-BEIR and compare against existing multilingual models, which have been trained on multilingual data. We empirically demonstrate that **NLLB-E5** achieves strong performance on Hindi-BEIR, outperforming existing multilingual models without the need for multilingual training data.

## 2 Related Works

One of the known sources of testing retriever performance on Indic languages was to test on the Indic language subset contained in multi-lingual IR datasets such as MIRAC Zhang et al. (2022), Mr.TyDi Zhang et al. (2021) having instances from Indic languages such as Hindi, Bengali, and Telugu. However, two major drawbacks here were: (1) all of these subsets were specific to a domain, hence couldn't really be used as a robust benchmark, and (2) the Indic language-specific subsets of instances didn't contain enough data points to facilitate building neural retrieval models for Indic languages either.

More recently, a relatively large-scale retrieval benchmark MS MARCO Nguyen et al. (2016) 

got translated into multiple languages to produce mMARCO (Bonifacio et al., 2022), including Hindi from the set of Indic languages. mMARCO was further extended to more Indic languages by IndicIRSuite (Haq et al., 2023), which is the most recent retrieval benchmark in Indic languages. However, the common problem for both mMARCO and IndicMarco is that these benchmarks, too, are tied to a single type of retrieval task and web domain as was originally in MS MARCO and, therefore, do not provide a robust benchmark.

In terms of Language specific benchmarks, Snegirev et al. (2024) introduce the ruMTEB Benchmark, an embedding model evaluation benchmark in Russian with Retrieval being a subtask, Valentini et al. (2024) introduce the Spanish IR dataset while Wojtasik et al. (2024) release the IR Benchmark in the Polish Language. In Indian languages such as Hindi, the Forum for Information Retrieval Evaluation (FIRE)3 has released numerous sub-tasks and datasets over the years to facilitate the benchmarking and development of information retrieval models. However, these datasets have predominantly been sourced from various news outlets and are not freely or openly accessible to the public.

Footnote 3: [https://fire.irsi.org.in/fire/2024/home](https://fire.irsi.org.in/fire/2024/home)

To the best of our knowledge, **Hindi-BEIR** presents the first comprehensive IR benchmark, which spans diverse domains and tasks and therefore, provides the first BEIR (Thakur et al., 2021) equivalent comprehensive retrieval benchmark in Hindi.

For multilingual retrieval model development, the most promising works are from Chen et al. (2024) and Wang et al. (2024c), but both these models are heavily dependent on multilingual data and do not scale for languages and domains where data is not readily available in the target language.

Yoon et al. (2024) proposed an approach to equip existing LLMs with multilingual capability without any multilingual supervised data. They vertically stack a multilingual encoder and an LLM, adding a projection/alignment layer between them. The input first passes through the multilingual encoder, the representation from the multilingual encoder passes through the alignment layer and then the output from the alignment layer is passed as input to the LLM. They use English-labeled data to fine-tune the overall model and update only the embedding layers and the alignment layer. The model showed better multilingual capabilities. Schmidt et al. (2024) extended the LLM2vec model (BehnamGhader et al., 2024) by vertically stacking a NLLB model (Team et al., 2022) and LLM2vec model. Similar to LangBridge (Yoon et al., 2024), they add a projection/alignment layer between the two and add LoRA parameters on the LLM. Different from LangBridge, they have a teacher model and a student model. They use LLM2vec as the teacher model and the NLLB-LLM2vec as the student model. The input is passed through both the teacher and the student models. They minimize the mean squared error between the final representations obtained from both the teacher and the student model. They train the model in two stages, the first stage being general alignment between the teacher and the student model, and, the second stage being task alignment. They show that the NLLB-LLM2vec model is able to obtain superior performance on various natural language understanding tasks.

Our **NLLB-E5** model is inspired by Yoon et al. (2024) who propose LangBridge, an approach which enables multilingual reasoning without multilingual supervised training and Schmidt et al. (2024) who opt for a distillation based approach and introduce NLLB-LLM2Vec capable of producing robust multilingual embeddings extending from NLLB encoder.

## 3 Hindi-BEIR Retrieval Benchmark

Table 1 summarises the various datasets included in the **Hindi-BEIR** benchmark, along with the number of documents, queries, and domain information.

To ensure that the **Hindi-BEIR** Benchmark is

\begin{table}
\begin{tabular}{l c r r r l} \hline \multirow{2}{*}{**Dataset Name**} & \multicolumn{1}{r}{**Tasks**} & \multirow{2}{*}{**\#Corpus**} & \multirow{2}{*}{**\# Queries**} & \multicolumn{2}{c}{**Average \# words**} \\ \cline{6-7}  & & & & ** Corpus** & **Query** \\ \hline ArgusAna & Argument Retrieval & 7763 & 1194 & 159.20 & 178.20 \\ FPAJ03-IE & Question Answering & 48178 & 5924 & 118.02 & 15.23 \\ TRRC-COVID & Bio-Medical IR & 78492 & 49 & 159.77 & 14.39 \\ SCIDOCS & Citation-Prediction & 20200 & 850 & 169.88 & 12.82 \\ SoftText & Fact-Checking & 2849 & 109 & 164.37 & 19.44 \\ Roache-2ND3 & Argument Retrieval & 385273 & 49 & 351.47 & 5.88 \\ NO & Question Answering & 559856 & 52985 & 892.83 & 9.63 \\ FEVER & Fact Checking & 5536276 & 120975 & 88.86 & 9.48 \\ Climate-FEVER & Fact Checking & 5362911 & 1499 & 88.86 & 24.40 \\ \hline CC News Retrieval & News Article Retrieval & 5005438 & 4699 & 272.20 & 9.30 \\ Snapfir-IR & Question Answering & 350000 & 9744 & 308.13 & 30.91 \\ \hline MIRACI & Passage Retrieval & 5005264 & 350 & 66.49 & 10.42 \\ IndioQARetrived & Question Answering & 261 & 15ata & 480.86 & 10.88 \\ ManRoC & Prange Retrieval & 5841823 & 6909 & 64.40 & 6.46 \\ WikiaProkitalRetroid & Question Answering & 13500 & 1500 & 77.19 & 9.54 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the Dataset in the Hindi-BEIR Benchmark showing the tasks which each dataset covers and the number of corpus and query in the evaluation set of each dataset in the Hindi-BEIR Benchmark.

 

comprehensive, challenging, and accessible to the public for future research and evaluation, we adhered to the following objectives:**1) **Diverse Domains and Tasks:** We include datasets from diverse domains like Wikipedia and news articles to niche domains like scientific publications, finance to test the robustness and generalization ability of the retrieval models. Additionally, a good retrieval model should handle documents and queries of varying lengths equally well. We have ensured that the datasets included in the Hindi-BEIR benchmark exhibit a wide range of query and document lengths among the datasets. **2) Difficulty Level:** We ensure that the datasets are challenging and systems relying solely on lexical overlap have a hard time retrieving the correct document. **3) Public Availability:** All datasets curated in **Hindi-BEIR** Benchmark have user-friendly licenses and are publicly available at Hindi-BEIR

We discuss the **Hindi-BEIR** benchmark creation in the following subsections. Please refer to Appendix A.2 for a more detailed analysis of each dataset.

### Translating English Datasets to Hindi:

We translate a subset of the existing English datasets from the BEIR benchmark Thakur et al. (2021) into Hindi. This approach was necessitated by the need to ensure comprehensive coverage across multiple domains and to maintain a high level of data quality and complexity. We utilized the Indic-Trans2 model 4Gala et al. (2023), a multilingual NMT model supporting translations across all 22 scheduled Indic languages (including English). We employ the back-translation technique to retain good translations. Specifically, given an English query/document we translate it to Hindi. This Hindi-translated query/document is then translated back to English. We calculate the Chrf(++) score Popovic (2017) between the original English query/document and the backtranslated English query/document. We retained only those translations with a Chrf(++) score exceeding a threshold. We empirically set the threshold to 50 after manually verifying the translation quality of texts obtained from different thresholds.

Footnote 4: Our choice for IndicTrans2 over other translation models has been discussed in A.3

All entries that fell below this threshold were removed. If an entry was part of the corpus and had an associated query, the query was also discarded. Likewise, any corresponding records in the relevancy mapping file were also deleted.

This strategy enables us to leverage the wealth of existing high-quality datasets in English while making them accessible and useful for Hindi language information retrieval tasks. Nine out of the fifteen datasets, which include Arguana, FiQA-2018, TREC-COVID, SCIDOCS, SciFact, Touche-2020, NQ, FEVER, and Climate-FEVER were created by this method.

### Using existing datasets for Retrieval task

We create Hindi CC News Retrieval and Sangraha-IR datasets from existing datasets. We detail the process we followed for each of the datasets as follows:

**Hindi CC News Retrieval :** This is a cross-lingual dataset derived from the Hindi subset of the Multilingual CC News dataset5, which contains 7,444,584 data points comprising Hindi news articles and their corresponding titles. The article text, which is in Hindi, serves as the corpus to be retrieved. At the same time, the English news titles act as queries, representing a practical and common real-world scenario that needs the models to have language-agnostic knowledge across both Hindi and English.

Footnote 5: [https://huggingface.co/datasets/intfloat/multilingual_cc_news](https://huggingface.co/datasets/intfloat/multilingual_cc_news)

We selected only those title-article pairs where the title never appears in the article text, thus reducing the dataset size to 451,803. This ensures no lexical overlap between the title and the article, making the retrieval task challenging as the model needs to rely on cross-lingual cues. From these filtered data points, we randomly selected 49,699 queries to form the test split, which has been incorporated into the Hindi-BEIR benchmark. Additionally, we release training and validation splits consisting of 301,578 and 100,526 query-corpus pairs, respectively. The train, test, and validation splits collectively encompass all data points obtained after the second filtering step.

**Sangraha-IR :** This dataset builds on top of the Sangraha Dataset Khan et al. (2024). We handpicked 350,000 entries from the Sangraha-verified subset, focusing on topics that truly represent Indian culture, such as agriculture, Bollywood, cricket, and other similar areas that capture the heart of India. We specifically chose the Sangraha-verified corpus as it consists of scraped data from "human-verified" Websites, OCR-extracted data from high-quality Indic language PDFs, transcribed 

data from various Indic language videos, podcasts, movies, courses, etc, and truly captures the essence of the language. To construct a set of relevant queries, we randomly selected 10,000 data points and used the Google Gemini-1.5-flash model to generate corresponding queries 6 for these documents. Queries that did not adhere to the provided instructions or deviated from the desired format were excluded, resulting in a final dataset of 9,744 queries and 350K candidate documents.

Footnote 6: Please refer to the Appendix for the prompt used

### Subset from existing Multilingual Datasets

We also include 5 publicly available datasets, namely MIRACL (dev split)Zhang et al. (2021), MLDR Chen et al. (2024), mMarco Bonifacio et al. (2022), IndicQARetrieval (developed by modifying the IndicQA Dataset Doddapaneni et al. (2023b)) and WikiPediaRetrieval 7.

Footnote 7: [https://huggingface.co/datasets/ellamind/wikipedia-2023-11-retrieval-multilingual-queries](https://huggingface.co/datasets/ellamind/wikipedia-2023-11-retrieval-multilingual-queries)

For translation, we specifically selected a subset of BEIR that focuses on factual and general knowledge content, avoiding topics with strong cultural or language-specific nuances that might be lost in translation. However, to ensure the inclusion of real-world data that captures the linguistic intricacies of Hindi, we carefully incorporate datasets such as Sangraha-IR, CC News Retrieval, MIRACL, IndicQA Retrieval, and Wikipedia Retrieval. These datasets, sourced from manually curated or web-scraped content, effectively preserve the cultural and language-specific nuances essential for robust evaluation.

## 4 NLLB-E5: multilingual retriever

In this section, we build a strong multilingual retrieval model based on the approach proposed by Yoon et al. (2024) and Schmidt et al. (2024).

Figure 1 shows the architecture of **NLLB-E5** with the different components and data flow between them. We build **NLLB-E5** on the back of two components (1) A multilingual encoder NLLB Team et al. (2022) and (2) a monolingual retriever E5 Wang et al. (2024a). We bank on the multilingual encoder to project semantically similar sentences across languages to a shared representation space. We hypothesize that embeddings coming from such a language-agnostic shared representation space can act as a strong prior that can be easily aligned for producing multilingual task-specific embeddings. Moreover, given the multilingual nature of the embedding coming directly from the multilingual encoder, the aforementioned task-specific alignment is language agnostic and, therefore, can be done with task-specific data in a high-resource language, too.

Figure 1 demonstrates how we follow the above intuition in designing **NLLB-E5**. We use NLLB Team et al. (2022) as the strong multilingual encoder that produces robust multilingual embeddings owing to its translation capabilities for over 200 languages. To align these embeddings

Figure 1: A comprehensive overview of our training methodology. The portion above the dotted lines represents the teacher model (enclosed by a blue dotted box), while the portion below represents our proposed NLLB-E5 student model (enclosed by a red dotted box). We use NLLB as a multilingual encoder and train the Linear projection layer and the Lora adapters by forcing its outputs to match that of the teacher model via the Mean Squared Error (MSE).

 

for the retrieval task, we learn a projection layer from the multilingual NLLB encoder to a monolingual retriever E5. The alignment happens by learning the projection layer parameters between NLLB and E5 through distillation from E5 embeddings, which are specifically tuned for retrieval tasks. Since this alignment to the retrieval task is agnostic to a specific language, we use English data and monolingual retriever E5 to do the distillation, eliminating the need for Hindi training data. We chose E5-large as the teacher and student model coupled with NLLB owing to its satisfactory retrieval performance on the BEIR benchmark.

## 5 Training Methodology

The **NLLB-E5** model is constructed by integrating the monolingual E5 model Wang et al. (2024) atop the multilingual encoder from NLLB Team et al. (2022), connected via a learnable projection layer. The parameters of both the monolingual and multilingual base models remain frozen throughout training. To enable adaptation, we introduce \(W\in R^{n\times d}\), a linear projection layer mapping the \(n\)-dimensional output space of the multilingual encoder to the \(d\)-dimensional input space expected by the E5 model.

We employ knowledge distillation Hinton et al. (2015) to transfer the capabilities of the monolingual E5 model to the **NLLB-E5** model, facilitating multilingual sentence representation learning. Specifically, the original E5 model serves as the _teacher_, producing high-quality English embeddings, while the **NLLB-E5** model, which includes the learnable Linear projection \(W\), acts as the _student_. By minimizing the discrepancy between the teacher and student embeddings, the student model learns to approximate the E5 model's performance across Hindi and other languages supported by NLLB. The frozen multilingual encoder ensures effective cross-lingual transfer, enabling generalization across all NLLB supported languages. 8

Footnote 8: We also experiment with LoRA adapters added on top of the E5 model. Details have been discussed in the Appendix A.5

### Training Process

The training procedure consists of the following steps:

**Get the Teacher Model Embeddings:** Given an input English sentence \(S\), we first tokenized it using the E5 tokenizer, yielding a sequence of tokens \(\{t_{1},t_{2},\dots,t_{n}\}\). These tokens are passed through the teacher E5 model, with frozen parameters, denoted as \(\mathcal{M}_{T}\). The model produces a sequence of token-level representations for the tokens:

\[\{X_{1},X_{2},\dots,X_{n}\}=\mathcal{M}_{T}(\{t_{1},t_{2},\dots,t_{n}\})\]

where \(X_{i}\in R^{d}\) represents the \(d\)-dimensional embedding of the token \(t_{i}\) produced by the model \(\mathcal{M}_{T}\). To obtain the final sentence-level embedding \(\vec{X}_{T}\), we apply average pooling over the token-level representations, which act as the final embedding for the sentence \(S\) obtained from the _teacher model_.

**Get Student Model Embeddings:** The input sentence is tokenized using the NLLB tokenizer, producing a sequence of tokens \(\{t_{1},t_{2},\dots,t_{m}\}\). The frozen multilingual encoder processes this token sequence, generating token-level representations in the multilingual semantic space \(\{H_{1},H_{2},\dots,H_{m}\}\). To align with the input structure expected by the E5 model, the token sequence is modified by prepending a token embedding corresponding to either [CLS] query: (for queries) or [CLS] passage: (for corpus), obtained from the E5 embedding layer (referred from here on as \(H_{\textit{prefix}}\)), along with the representation of [SEP](referred from here on as \(H_{\textit{posuffix}}\)) which is appended at the end. This is projected through a learnable layer \(W\) as \(\{H^{\prime}_{1},H^{\prime}_{2},\dots,H^{\prime}_{m}\}=W\{H_{\textit{prefix}}, H_{1},H_{2},\dots,H_{m},H_{\textit{Posuffix}}\}\) This projected sequence is then passed through the student E5 model, which outputs the token-level representations \(\{X^{\prime}_{1},X^{\prime}_{2},\dots,X^{\prime}_{m}\}\) Finally, average pooling is applied over these token-level representations to obtain the final sentence-level embedding \(\vec{X}_{S}\) from the student model.

**Minimizing the Loss Function.** To align the student model with the teacher, we minimize the Mean Squared Error (MSE) between the teacher and student embeddings, \(\vec{X}_{T}\) and \(\vec{X}_{S}\), respectively, for a batch of sentences.

This optimization encourages the student to generate multilingual embeddings that are aligned with the high-quality English embeddings produced by the teacher.

## 6 Experiments

This section provides a detailed overview of our experiments on **Hindi-BIER** datasets, including our experimental setup and the baseline models used in our experimentation 



### Benchmark and Evaluation Metric

We use **Hindi-BEIR** as the target benchmark for all our experiments. We evaluate the models across all \(15\) datasets in **Hindi-BEIR** and compare their performance at the dataset level too for finer insights.

Following BEIR Thakur et al. (2021) benchmark standard, we use Normalised Discounted Cumulative Gain (NDCG) Jarvelin and Kekalainen (2002), more specifically NDCG@10 as our evaluation metric. NDCG is known to be a more robust metric than simple recall in the IR community because it is purposefully designed to be sensitive towards ranking of the retrieved results.

### Implementation Details of NLLB-E5

To make **NLLB-E5** produce robust, generalizable embeddings across different kind of retrieval tasks and domains, we rely on using diverse data to learn NLLB and E5 alignment during training. We followed a similar approach as taken by generic text-embedding models and used a subset of data from the huge pool of 1B diverse data curated for training sentence transformer models SIMOULIN (2021). We randomly selected 100,000 samples from each dataset to create the final training set. The datasets chosen are i) Arxiv ii) Natural Questions (NOJ iii HotpotQA iv) Stack Exchange - Duplicate questions (titles) v) Stack Exchange - (Title, Body) pairs vi) Stack Exchange - (Title+Body, Answer) pairs vii) Stack Exchange - (Title, Answer) pairs viii) Stack Exchange - Duplicate questions (bodies) ix) Stack Exchange duplicate questions (titles+bodies) x) SQuAD 2.0 xi) S2ORC Citation - pairs (Abstracts) xi) S2ORC (Title, Abstract) xiii) SearchQA xiii) PubMed (Title, Abstract) six) SPECTER xv) FEVER xvi) Wikipedia Sections xvii) Stack Exchange Math xviii) Stack Overflow Posts xix) Wikipedia

We train the **NLLB-E5** model for 10 epochs, with a learning rate of 2e-4 and a linear scheduler for learning rate adjustment with FP16 precision. We chose the best checkpoint based on the evaluation result on the MsMARCO dev set. We ran all our experiments using Nvidia A100 80GB GPUs. Evaluating the **NLLB-E5** model using 4 A100 80GB GPUs on the **Hindi-BEIR** benchmark took 24 hours.

### Baseline Models

We evaluate our Hindi-BEIR dataset on the following models:

1. The **NLLB-E5 variations:** We develop and evaluate NLLB-E5 using 600M, 1.3B and 3.3B parameter versions of the NLLB model and compare their performance9 Footnote 9: [https://huggingface.co/facebook/nllb-200-distilled-1.3B](https://huggingface.co/facebook/nllb-200-distilled-1.3B), [https://huggingface.co/facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M), [https://huggingface.co/facebook/nllb-200-3.3B](https://huggingface.co/facebook/nllb-200-3.3B)
2. The **BOGE-M3**Chen et al. (2024) model was pre-trained on a large multilingual and cross-lingual unsupervised data and subsequently fine-tuned on high-quality multilingual retrieval datasets using a custom loss function based on the InfoNCE loss function. BGE-M3 supports a context length of 8,192.
3. **Multilingual E5 (mE5) Wang et al. (2024c)** was developed by continually pre-training the E5 model Wang et al. (2024b) on a large multilingual corpus using a weakly supervised contrastive pretraining method with InfoNCE contrastive loss van den Oord et al. (2019). It was then fine-tuned on high-quality labelled multilingual datasets for retrieval tasks. mE5 has a context length of 512 tokens.
4. **LASER**Artetxe and Schwenk (2019) focuses on universal language agnostic sentence embeddings across 93 languages. LASER uses a language-agnostic BiLSTM encoder architecture trained on parallel corpora from different languages without any retrieval-specific finetuning.
5. **LaBSE**Feng et al. (2022) uses a dual encoder model using BERT to obtain language-agnostic sentence embedding without any retrieval-specific fine-tuning. It has a maximum context length of 256 tokens.
6. **BM-25**Amati (2009) is a ranking function in information retrieval based on estimating relevance by combining term frequency and document length normalization.

We extend the MTEB benchmark code repository10 to include Hindi-BEIR and evaluate the models discussed above.





## 7 Results and Analysis

In this section, we compare the performance of the aforementioned baselines with our proposed **NLLB-E5** model on **Hindi-BEIR** benchmark and highlight our key findings from the results.11

Footnote 11: We also evaluated NLLB Encoder trained using distillation approach as used for NLLB-E5. Details has been discussed in A.6

**NLLB-E5 outperforms multilingual retrievers on Hindi-BEIR:** Table 2 shows NLLB\({}_{1.3B}\)-E5 model achieves better average compared to all other baselines on Hindi-BEIR. This is significant because we had state-of-the-art multilingual retriever models such as BGE-M3 and mE5 as our baselines in addition to LASER and LaBSE. It is important to note here that all these multilingual embedding models have seen a huge amount of multilingual data in their training, including Hindi. In contrast, **NLLB-E5** is tuned only on English language data for learning retrieval alignment multilingual embeddings from NLLB encoder.

As we can see in Table 2 **NLLB-E5** performs the best on almost all of BEIR subsets of Hindi-BEIR. More specifically, the **NLLB-E5** model attains the highest scores on the Hindi versions of arguAna, FiQA-2018, NQ, SCIDOCS and SciFact, FEVER, Climate-FEVER and mMARCO and performs 2nd best on Hindi TREC-COVID dataset. In comparison to its direct competitor, mE5-Large, the NLLB-E5 model demonstrates a substantial edge in fact-checking datasets such as FEVER, Climate-FFVER and SciFact, resulting in a significant boost of 82.47 %, 212 % and 15.06 % on the respective datasets. For non-BIER subsets too, NLLB\({}_{1.3B}\)-E5 easily outperforms LASER and LaBSE and performs competitively with mE5 and BGE-M3.

**NLLB-E5 outperforms BM25 on Hindi-BEIR:** BM25 is known to be a strong baseline on BEIR Thakur et al. (2021) benchmark. In fact, as seen in Table 2, BM25 indeed acts as a better baseline than LASER and LaBSE too. However, **NLLB-E5** outperforms BM25 on 13/15 datasets in the Hindi-BEIR benchmark, achieving a gain of \(~{}+37\%\) on average. This gain justifies that we

\begin{table}
\begin{tabular}{l r r r r r r r} \hline \hline
**Dataset Name** & \multicolumn{1}{c}{\multirow{2}{*}{\begin{tabular}{c} **BGE-M3** \\ (56T)M \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{\begin{tabular}{c} **mE5-Base** \\ (110M) \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{\begin{tabular}{c} **mE5-Large** \\ (335M) \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{LASER}} & \multicolumn{1}{c}{\multirow{2}{*}{
\begin{tabular}{c} **LaBSE** \\ (471M) \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{**BM-25**}} & \multicolumn{1}{c}{\multirow{2}{*}{**NLLB-E5 (3.3B)**}} \\ \cline{1-1}  & & & & & & & \\ \hline ArguAna & 53.81 & 49.96 & 54.77 & 11.27 & 32.90 & 43.75 & **57.68** \\ FiQA-2018 & 25.89 & 22.38 & 27.33 & 1.58 & 7.23 & 16.57 & **34.41** \\ TREC-COVID & 64.60 & 62.42 & **70.80** & 3.95 & 29.94 & 52.30 & 70.12 \\ SICDOCS & 14.24 & 10.42 & 11.32 & 0.59 & 6.95 & 11.40 & **17.82** \\ SeiFact & 52.39 & 51.50 & 55.92 & 5.37 & 33.42 & 60.80 & **64.34** \\ Touche-2020 & 26.68 & 27.44 & 26.89 & 1.06 & 6.82 & **33.59** & 25.35 \\ NJG & 39.15 & 44.10 & 44.10 & 0.49 & 9.36 & 16.79 & **53.04** \\ FEVER & 66.91 & 32.87 & 39.36 & 0.19 & 8.27 & 40.57 & **71.82** \\ Climate-FEVER & 23.71 & 5.93 & 8.22 & 0.28 & 3.72 & 14.00 & **25.66** \\ \hline CC News Retrieval & 34.40 & 20.81 & **35.00** & 0.52 & 5.63 & 0.01 & 31.73 \\ Sangraha-IR & 44.85 & 41.12 & **47.91** & 3.76 & 21.76 & 30.92 & 42.36 \\ \hline MIRACL & **59.34** & 58.11 & 59.24 & 0.69 & 13.76 & 40.98 & 53.56 \\ IndicQARTetival & **69.92** & 67.11 & 67.11 & 21.28 & 46.85 & 74.02 & 61.94 \\ mMARCO & 29.49 & 29.94 & 30.92 & 0.48 & 6.98 & 16.53 & **33.03** \\ WikiPediaRetrieval & **87.38** & 84.40 & 86.71 & 0.03 & 61.28 & 82.54 & 85.65 \\ \hline
**Average** & 46.18 & 40.57 & 44.37 & 3.44 & 19.66 & 35.65 & **48.57** \\ \hline \hline \end{tabular}
\end{table}
Table 2: NDCG@10 scores of the existing multilingual model and best-performing NLLB-E5 on Hindi-BEIR datasets. The best-performing model has been highlighted as **bold** while the second-best model has been underlined.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset Name** & \multicolumn{1}{c}{\multirow{2}{*}{\begin{tabular}{c} **600M** \\ **(distilled)** \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{\begin{tabular}{c} **1.3B** \\ **(distilled)** \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{\begin{tabular}{c} **1.3B** \\ **(distilled)** \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{\begin{tabular}{c} **3.3B** \\ **(33.92)** \\ \end{tabular} }} & \multicolumn{1}{c}{\multirow{2}{*}{
\begin{tabular}{c} **3.3B** \\ **3.41** \\ \end{tabular} }} \\ \cline{1-1} Arguana & 55.23 & 57.24 & 56.50 & 57.68 \\ FQua-2018 & 32.83 & 34.19 & 33.92 & 34.41 \\ TREC-COVID & 70.07 & 70.53 & 70.54 & 70.12 \\ SCIIOocs & 17.53 & 18.23 & 17.67 & 17.82 \\ SciFact & 63.93 & 64.36 & 64.35 & 64.34 \\ Touché-2020 & 25.89 & 25.21 & 26.13 & 25.35 \\ NQ & 50.14 & 53.38 & 52.77 & 53.04 \\ FEYER & 66.79 & 71.04 & 69.02 & 71.82 \\ Climate-FEVER & 25.12 & 22.51 & 23.79 & 25.66 \\ \hline CC News Retrieval & 29.11 & 31.91 & 30.93 & 31.73 \\ Sangraha-IR & 40.51 & 43.30 & 42.84 & 42.36 \\ \hline MIRACL & 50.38 & 52.96 & 52.88 & 53.56 \\ IndicQARTetieval & 60.47 & 62.10 & 61.69 & 61.94 \\ mMARCO & 31.63 & 34.03 & 33.15 & 33.03 \\ WikiPediaRetrieval & 84.89 & 86.20 & 86.22 & 85.65 \\ \hline
**Average** & 46.97 & 48.48 & 48.16 & 48.57 \\ \hline \hline \end{tabular}
\end{table}
Table 3: NDCG@10 scores of NLLB-E5 models using NLLB encoders of different parameter sizes on **Hindi-BEIR** benchmark

can treat **NLLB-E5** as a robust neural model much better than lexical retrievers such as BM25.

BM25 has very poor performance on CC News Retrieval datasets only. The CC News Retrieval dataset was intentionally designed to test the ability of multilingual dense retrieval models to learn language-agnostic embeddings for retrieval. This dataset presents a scenario where the query is in English, and the corpus is in Hindi, which needs to move beyond token-level matching. This explains the poor performance of BM25 due to the lack of overlap between English and Hindi tokens.

**Variation in Performance Across Different Tasks and Domains:** One of the important aspects of building Hindi-BEIR was to see how a retriever adapts to different domains and tasks. As we can see, mE5 specifically struggles in fact-checking datasets, while BGE-M3 and NLLB-E5, although not great, still fare better for fact-checking tasks. All the models perform quite poorly for the citation prediction tasks on SCIDOCS. Also, for niche domains such as Finance and Climate, we see a sharp drop for all the models, prompting the need for focused research in these areas.

**Effect of Encoder Param Size:** Table 3 presents the NDCG@10 scores for the NLLB model's encoder across different parameter sizes. As the table shows, performance generally improves as the parameter size increases, though minor anomalies exist across specific datasets. Notably, the performance gain plateaus beyond the 1.3B parameter model, with only a marginal improvement of 0.09% when comparing the NLLB-E5 model's 1.3B (distilled) encoder to that of the 3.3B parameter version. This possibly indicates that **NLLB-E5** is not lacking in model capacity yet and thus can possibly be improved further by showing more volumes of data from diverse tasks and domains.

## 8 Conclusion and Future Work

In this work, we introduced **Hindi-BEIR**, a comprehensive benchmark for Hindi language information retrieval. The benchmark consists of 15 datasets, with a corpus size exceeding 27 million and a query size of over 200K, covering a diverse range of 7 tasks. This provides the first-of-its-kind robust benchmark for assessing the performance of retriever models in Hindi. Our experimental results offer insights into the strengths and limitations of current multilingual retrievers on **Hindi-BEIR**, indicating the pressing need for further research in this domain. Additionally, we also proposed the **NLLB-E5** model, a novel architecture for multilingual retriever tuning without needing multilingual training data, capable of handling data from over 200 languages, including Hindi, which outperformed existing multilingual models on Hindi-BEIR.

Future work will focus on expanding the Hindi-BEIR benchmark to include more diversity by curating additional domains such as Law and Medicine. We also plan to extend this benchmark to cover languages beyond Hindi and explore alternative multilingual encoders to optimize representation. We believe our work will have a lasting impact in developing inclusive and scalable information retrieval systems across diverse languages.

## 9 Limitations

**The queries are AI generated etc.** While the **Hindi-BEIR** Benchmark and **NLLB-E5** model provides valuable advancements, we acknowledge several limitations. One limitation of the **Hindi-BEIR** Benchmark is that it may not fully capture the breadth of tasks where retrieval models are essential. The current scope, while extensive, lacks coverage in critical domains such as Law and Medicine, which are key areas we plan to include in future expansions. Additionally, although designed primarily for Hindi, the benchmark's extension to other languages is an area that remains unexplored in this work.

The **NLLB-E5** model, while showing promise, exhibits sub-optimal performance on certain English tasks. This limitation indicates a need to explore alternative multilingual encoders to improve language performance. Additionally, we realise that the current model would architecture struggle with long texts due to a context length limitation of 508 tokens, which poses challenges in tasks requiring extended context handling. Further, we would also like to extend and validate this idea for other low-resource languages in the future.

## References

* Amati [2009] Giambattista Amati. 2009. _BM25_, pages 257-260. Springer US, Boston, MA.
* Artetxe and Schwenk [2019] Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. _Transactions 

of the Association for Computational Linguistics_, 7:597-610.
* BehnamGhader et al. (2024) Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2vec: Large language models are secretly powerful text encoders. In _First Conference on Language Modeling_.
* Bonifacio et al. (2022) Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2022. mmarco: A multilingual version of the ms marco passage ranking dataset. _Preprint_, arXiv:2108.13897.
* Chen et al. (2024) Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. _Preprint_, arXiv:2402.03216.
* Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2270-2282, Online. Association for Computational Linguistics.
* Datta et al. (2023) Debtanu Datta, Shubham Soni, Rajdeep Mukherjee, and Saptarshi Ghosh. 2023. MILDSum: A novel benchmark dataset for multilingual summarization of Indian legal case judgments. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 5291-5302, Singapore. Association for Computational Linguistics.
* Doddapaneni et al. (2023a) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023a. Towards leaving no indic langauge behind: Building monolingual corpora, benchmark and models for indic languages. _Preprint_, arXiv:2212.05409.
* Doddapaneni et al. (2022b) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khappa, Anoop Kunchukuttan, and Pratyush Kumar. 2023b. Towards leaving no indic langauge behind: Building monolingual corpora, benchmark and models for indic languages. _Preprint_, arXiv:2212.05409.
* Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. _Preprint_, arXiv:2007.01852.
* Gala et al. (2023) Jay Gala, Pranjal A Chitale, A K Raghavan, Varun Guamma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Pudupply, Vivetk Raghavan, Pratyush Kumar, Mitesh M Khappa, Raj Dabre, and Anoop Kunchukuttan. 2023. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. _Transactions on Machine Learning Research_.
* Ghosh et al. (2024) Akash Ghosh, Arkadeep Acharya, Prince Jha, Sriparna Saha, Aniket Gaudgauld, Rajdeep Majumdar, Aman Chadha, Raghaw Jain, Settu Sinha, and Shirani Agarwal. 2024a. Medsumm: A multimodal approach to summarizing code-mixed hindi-english clinical queries. In _European Conference on Information Retrieval_, pages 106-120. Springer.
* Ghosh et al. (2024) Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Gaurav Pandey, Dinesh Raghu, and Settu Sinha. 2024b. HealthAlignSum : Utilizing alignment for multimodal summarization of code-mixed healthcare dialogues. In _Findings of the Association for Computational Linguistics: EMNLP 2024_, pages 11546-11560, Miami, Florida, USA. Association for Computational Linguistics.
* Haq et al. (2023) Saiful Haq, Ashutosh Sharma, and Pushpak Bhattacharyya. 2023. Indicirusite: Multilingual dataset and neural information models for indian languages. _Preprint_, arXiv:2312.09508.
* Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. _Preprint_, arXiv:1503.02531.
* Jarvelin and Keskalainen (2002) Kalervo Jarvelin and Jaana Keskalainen. 2002. Cumulated gain-based evaluation of ir techniques. _ACM Trans. Inf. Syst._, 20:422-446.
* Khan et al. (2024) Mohammed Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad B, Varun G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh Khappa. 2024. IndictLMSuite: A blueprint for creating pre-training and fine-tuning datasets for Indian languages. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15831-15879, Bangok, Thailand. Association for Computational Linguistics.
* Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Pittus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Preprint_, arXiv:2005.11401.
* Maia et al. (2018) Macedo Maia, Siegfried Handschuh, Andre Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www'18 open challenge: Financial opinion mining and question answering. In _Companion Proceedings of the The Web Conference 2018_, WWW '18, page 1941-1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. _CoRR_, abs/1611.09268 

* Popovic (2017) Maja Popovic. 2017. chrF++: words helping character n-grams. In _Proceedings of the Second Conference on Machine Translation_, pages 612-618, Copenhagen, Denmark. Association for Computational Linguistics.
* Sabane et al. (2024) Maithili Sabane, Onkar Litake, and Aman Chadha. 2024. Breaking language barriers: A question answering dataset for hindi and marathi. _Preprint_, arXiv:2308.09862.
* Schmidt et al. (2024) Fabian David Schmidt, Philipp Borchert, Ivan Vulic, and Goran Glavas. 2024. Self-distillation for model stacking unlocks cross-lingual nlu in 200+ languages. _Preprint_, arXiv:2406.12739.
* SIMOULIN (2021) Antoine SIMOULIN. 2021. Train a sentence embedding model with l billion training pairs.
* Snegirev et al. (2024) Artem Snegirev, Maria Tikhonova, Anna Maksimova, Alena Fenogenova, and Alexander Abramov. 2024. The russian-focused embedders' exploration: rumteb benchmark and russian embedding model design. _arXiv preprint arXiv:2408.12503_.
* Team et al. (2021) NLLB Team, Marta R. Costa-jussas, James Cross, Onur Celebi, Maha Elbayad, Kenneth Hefield, Kevin Hefernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Meja Gonzalez, Prangthip Hansanti, John Hoffman, Semarely Jarrett, Kaushik Ram Sadagopam, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necio Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzman, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. _Preprint_, arXiv:2207.04672.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. _Preprint_, arXiv:2104.08663.
* Valentini et al. (2024) Francisco Valentini, Viviana Cotik, Damian Furman, Ivan Bercovich, Edgar Altskzyler, and Juan Manuel Perez. 2024. Messive: A large-scale spanish information retrieval dataset. _Preprint_, arXiv:2409.05994.
* van den Oord et al. (2019) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding. _Preprint_, arXiv:1807.03748.
* Voorhees et al. (2021) Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Tree-covid: constructing a pandemic information retrieval test collection. _SIGIR Forum_, 54(1).
* Wachsmuth et al. (2017) Henning Wachsmuth, Martin Potthast, Khalid Alkhatib, Yamen Ajour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. 2017. Building an argument search engine for the web. In _Proceedings of the 4th Workshop on Argument Mining_, pages 49-59, Copenhagen, Denmark. Association for Computational Linguistics.
* Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 241-251, Melbourne, Australia. Association for Computational Linguistics.
* Wang et al. (2024a) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Davin Jiang, Rangan Majumder, and Furu Wei. 2024a. Text embeddings by weakly-supervised contrastive pre-training. _Preprint_, arXiv:2212.03533.
* Wang et al. (2024b) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Dexin Jiang, Rangan Majumder, and Furu Wei. 2024b. Text embeddings by weakly-supervised contrastive pre-training. _Preprint_, arXiv:2212.03533.
* Wang et al. (2024c) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024c. Multilingual e5 text embeddings: A technical report. _Preprint_, arXiv:2402.05672.
* Wojtasik et al. (2024) Konrad Wojtasik, Vadim Shishkin, Kacper Wolowiec, Arkadiusz Janz, and Maciej Piasecki. 2024. Beir-pl: Zero shot information retrieval benchmark for the polish language. _Preprint_, arXiv:2305.19840.
* Yoon et al. (2024) Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. 2024. Langbridge: Multilingual reasoning without multilingual supervision. _Preprint_, arXiv:2401.10695.
* Zhang et al. (2021) Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. In _Proceedings of the 1st Workshop on Multilingual Representation Learning_, pages 127-137, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Zhang et al. (2022) Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezaghollizadeh, and Jimmy Lin. 2022. Making a miracl: Multilingual information retrieval across a continuum of languages. _Preprint_, arXiv:2210.09984.
* Zhang et al. (2023) Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qin Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. _Transactions of the Association for Computational Linguistics_, 11:1114-1131 

Appendix

### Dataset License

The datasets will be publicly released with the same license as the parent dataset and will be made available for research purposes.

### Dataset Description

#### a.2.1 ArguAna

* **Task definition:** Derived from the work by Wachsmuth et al. (2018) , the task is to retrieve the best counterargument, given an argument. Translations of arguments and countermeasures from online debates constitute the corpus, while the translations of arguments in the original test split, after going through the filtration process based on Chrf++ scores (refer to 3.1), constitute the queries.
* **Domain :** Misc.

An example of a query with its corresponding golden corpus has been provided in Figure 2 posts under the Investment topic from 2009-2017, after passing through filteration processes mentioned in 3.1, acts as the corpus. While translations from the original training split acts as the queries.

## 2 Domain : Finance

An example of query with its corresponding golden corpus from the FiQA-2018 dataset has been provided in Figure 5

Distribution of the number of words in the corpus and queries in the ArguAna dataset has been shown in Figure 4 and Figure 3, respectively.

#### a.2.2 Fiqa-2018

1. **Task Definition:** It deals with Opinion-Based Question answering. Based on the works of Maia et al. (2018),translation of the financial data extracted by crawling StackExchange posts under the Investment topic from 2009-2017, after passing through filteration processes mentioned in 3.1, acts as the corpus. While translations from the original training split acts as the queries.
2. **Domain :** Finance

An example of query with its corresponding golden corpus from the FiQA-2018 dataset has been provided in Figure 5

Distribution of the number of words in the corpus and queries in the ArguAna dataset has been shown in Figure 6 and Figure 7, respectively.

#### a.2.3 Fiqa-2018

1. **Task Definition:** It deals with Opinion-Based Question Answering. Based on the works of Maia et al. (2018),translation of the financial data extracted by crawling StackExchange posts under the Investment topic from 2009-2017, after passing through filteration processes mentioned in 3.1, acts as the corpus. While translations from the original training split acts as the queries.
2. **Domain :** Finance

An example of query with its corresponding golden corpus from the FiQA-2018 dataset has been provided in Figure 5

Figure 4: Distribution of Number of Words in the corpus of ArguAna Dataset

Figure 5: An example of a query with its corresponding golden corpus from the FiQA-2018 Dataset

Figure 3: Distribution of the number of words in the queries of ArguAna Dataset

Figure 2: An example of a query with its corresponding golden corpus from the ArguAna Dataset



pus and queries in FiQA-2018 dataset has been shown in Figure 7 and Figure 6 respectively.

#### a.2.3 Trec-Covid

1. **Task Definition:** Voorhees et al. (2021) introduced the original TREC-COVID dataset which is an ad-hoc each challenge based on CORD-19 dataset containing articles about the COVID-19 Pandemic. The translated and filtered version of the CORD-19 Dataset constitutes the corpus while the final cumulative judgements with query descriptions from the original task are the queries in the TREC-COVID Dataset.
2. **Domain :** Bio-Medical An example of a query with its corresponding golden corpus from the TREC-COVID dataset has been provided in Figure 8 Distribution of the number of words in the corpus and queries in TREC-COVID dataset has been shown in Figure 10 and Figure 9 respectively.

 

An example of query with its corresponding golden corpus from the SCIDOCS dataset has been provided in Figure 11

Distribution of the number of words in the corpus and queries in the SCIDOCS dataset has been shown in Figure 13 and Figure 12, respectively.

#### a.2.5 SciFact

1. **Task Definition:** This task involves the verification of scientific claims given the abstract of scientific articles from recent literature. For this task, the model is expected to retrieve relevant abstracts with a given claim as input.
2. **Domain :** Scientific

An example of query with its corresponding golden corpus from the SciFact dataset has been provided in Figure 14

Distribution of the number of words in the corpus and queries in the SCIDOCS dataset has been shown in Figure 13 and Figure 12, respectively.

Distribution of the number of words in the corpus and queries in the SciFact dataset has been shown in Figure 16 and Figure 15, respectively.

Distribution of the number of words in the corpus and queries in the SciFact dataset has been shown in Figure 16 and Figure 15, respectively.

#### a.2.6 Touche-2020

1. **Task Definition:** Similar to ArguAna this task deals with this task deals with the retrieval of conversational arguments. The translated and filtered conclusion forms the title and premise for arguments (Wachsmuth et al.,

Figure 11: An example of a query with its corresponding golden corpus from the SCIDOCS Dataset

Figure 14: An example of a query with its corresponding golden corpus from the SciFact Dataset

Figure 12: Distribution of the number of words in the queries of SCIDOCS Dataset

Figure 13: Distribution of Number of Words in the corpus of SCIDOCS Dataset



2017) constitutes the corpus. The translations of the Touche-2020 task data are the queries.
3. **Domain :** Miscellaneous
An example of query with its corresponding golden corpus from the Touche-2020 dataset has been provided in Figure 17

Distribution of the number of words in the corpus and queries in the Touche-2020 dataset has been shown in Figure 19 and Figure 18 respectively.

#### a.2.7 Nq

1. **Task Definition:** The task here is to retrieve the correct answer to a given question from a broad corpus of candidate answers. The NQ dataset in the Hindi-BEIR Benchmark is the translated and filtered version of the NQ dataset released by Thakur et al. (2021), which contains Google search queries and documents with paragraphs and answer spans within Wikipedia articles as the corpus.
2. **Domain :** WikiPedia

An example of a query with its corresponding golden corpus from the NQ dataset has been provided in Figure 20

Distribution of the number of words in the corpus and queries in the NQ dataset has been shown in Figure 22 and Figure 21, respectively.

Distribution of the number of words in the corpus and queries in the NQ dataset has been shown in Figure 22 and Figure 21, respectively.

#### a.2.8 Fever

1. **Task Definition:** Similar to SciFact, here, the task is to retrieve relevant documents that

Figure 16: Distribution of Number of Words in the corpus of SciFact Dataset

Figure 19: Distribution of Number of Words in the corpus of Touché-2020 Dataset

Figure 17: An example of a query with its corresponding golden corpus from the Touché-2020 Dataset

Figure 20: An example of a query with its corresponding golden corpus from the NQ Dataset



claim a given fact (query). We translate and filter the test split of the FEVER dataset as proposed by Thakur et al. (2021) and include it in the Hindi-BEIR Benchmark.
2. **Domain :** Wikipedia

An example of a query with its corresponding golden corpus from the FEVER dataset has been provided in Figure 23

Distribution of the number of words in the corpus and queries in the FEVER dataset has been shown in Figure 25 and Figure 24, respectively.

#### a.2.9 Climate-FEVER

1. **Task Definition:** Similar to FEVER, Climate-FEVER is a dataset for the verification of real-world climate claims. We translate and filter the test split of the Climate-FEVER dataset as proposed by Thakur et al. (2021) and include it in the Hindi-BEIR Benchmark.
2. **Domain :** Wikipedia

An example of query with its corresponding golden corpus from the Climate-FEVER dataset has been provided in Figure 26

Distribution of the number of words in the corpora and queries in the Climate-FEVER dataset has been shown in Figure 28 and Figure 27, respectively.

a.2.10 CC News Retrieval Introduces the task to retrieving relevant news articles given a news title. Subsection 3.2 talks in detail about the data creation process.

Figure 23: An example of a query with its corresponding golden corpus from the FEVER Dataset

Figure 22: Distribution of Number of Words in the corpus of NQ Dataset

Figure 21: Distribution of the number of words in the queries of NQ Dataset

Figure 24: Distribution of the number of words in the queries of FEVER Dataset

Figure 25: Distribution of Number of Words in the corpus of FEVER Dataset



An example of a query with its corresponding golden corpus from the CC News Retrieval dataset has been provided in Figure 29

Distribution of the number of words in the corpus and queries in CC News Retrieval dataset has been shown in Figure 31 and Figure 30 respectively.

#### a.2.11 Sangraha-IR

1. **Task Definition:** Sangraha-IR introduces the task to retrieving relevant text documents given a question in Hindi. Subsection 3.2 talks in detail about the data creation process.
2. **Domain :** Miscellaneous

Figure 31: Distribution of Number of Words in corpus of CC News Retrieval Dataset

Figure 28: Distribution of Number of Words in Corpus of Climate-FEVER Dataset

Figure 27: An example of a query with its corresponding golden corpus from the Climate-FEVER Dataset

Figure 26: An example of a query with its corresponding golden corpus from the Climate-FEVER Dataset

Figure 30: Distribution of the number of words in the queries of CC News Retrieval Dataset



An example of a query with its corresponding golden corpus from the Sangraha-IR dataset has been provided in Figure 32

Distribution of the number of words in the corpus and queries in the Sangraha-IR dataset has been shown in Figure 34 and Figure 33, respectively.

#### a.2.12 Miracl

1. **Task Definition:** MIRACL has been created from the Wikipedia dump of each language. Only plain text is considered, while images, etc., are omitted. The plain text is split up into multiple paragraphs, which act as the corpus. We consider the dev split of the Hindi subset of the original MIRACL in the Hindi-BEIR Benchmark.
2. **Domain :** Wikipedia An example of query with its corresponding golden corpus from the MIRACL dataset has been provided in Figure 35

Distribution of the number of words in the corpus and queries in the MIRACL dataset has been shown in Figure 36 and Figure 37 respectively.

Distribution of the number of words in the corpus and queries in the MIRACL dataset has been shown in Figure 38 and Figure 39 respectively.

Figure 30: Distribution of the number of words in the queries of MIRACL Dataset

Figure 31: An example of a query with its corresponding golden corpus from the Sangraha-IR Dataset

Figure 32: An example of a query with its corresponding golden corpus from the Sangraha-IR Dataset

Figure 33: Distribution of the number of words in the queries of Sangraha-IR Dataset





#### a.2.13 IndicQARetrieval

1. **Task Definition:** It is created by transforming the IndicQA dataset 12 to a retrieval dataset. The task is similar to NQ, where the model is expected to return the paragraph that contains the answer to the question when given a question. Footnote 12: [https://huggingface.co/datasets/ai4bharat/IndicQA](https://huggingface.co/datasets/ai4bharat/IndicQA)
2. **Domain :** Wikipedia An example of query with its corresponding golden corpus from the IndicQARetrieval dataset has been provided in Figure 38

Footnote 12: [https://huggingface.co/datasets/ai4bharat/IndicQA](https://huggingface.co/datasets/ai4bharat/IndicQA)

Distribution of the number of words in the corpus and queries in the IndicQARetrieval dataset has been shown in Figure 40 and Figure 39 respectively.

#### a.2.14 mMarco

1. **Task Definition:** It is a multilingual version of the MSMARCO dataset. The dataset contains the translation of queries from Bing search logs with one text passage from various web sources annotated as relevant.
2. **Domain :** Miscellaneous An example of a query with its corresponding golden corpus from the mMARCO dataset has been provided in Figure 41

Figure 41: An example of a query with its corresponding golden corpus from the mMARCO Dataset

Figure 37: Distribution of Number of Words in the corpus of MIRACL Dataset

Figure 38: An example of a query with its corresponding golden corpus from the IndicQARetrieval Dataset

Figure 39: Distribution of the number of words in the queries of IndicQARetrieval Dataset

Distribution of the number of words in the corpus and queries in the mMARCO dataset has been shown in Figure 43 and Figure 42 respectively.

#### a.2.15 WikiPediaRetreival

1. **Task Definition:** It is similar to a question-answering task dataset like NQ, where given a query, the model is expected to retrieve a relevant article which answers the question. We have included the Hindi subset of WikiPediaRetrieval dataset 13 Footnote 13: [https://huggingface.co/collections/ellamind/mmteb-6661723dc229eIda8e837cdf](https://huggingface.co/collections/ellamind/mmteb-6661723dc229eIda8e837cdf)
2. **Domain :** Wikipedia An example of a query with its corresponding golden corpus from the WikiPediaRetreival dataset has been provided in Figure 44 Distribution of the number of words in the corpus and queries in the WikiPediaRetrieval dataset has been shown in Figure 46 and Figure 45 respectively.

### Why choose IndicTrans2 over other available translation models?

**Ans:**Gala et al. (2023) clearly illustrate the superior performance of IndicTrans2 over other models and systems like NLLB and Google Translate for English to Hindi tasks. In our preliminary analysis on a subset of the BEIR datasets (Chrf scores show in Table 4 ), we also observed that IndicTrans2 outperformed alternative models, such as NLLB, in terms of translation quality.

### Prompts used for Sangraha-IR Query Generation

The prompt that was used to obtain answers from the Gemini-Flash-1.5 Models is as follows

_Given the above text, generate a question in

Figure 44: An example of a query with its corresponding golden corpus from the WikiPediaRetreival Dataset

Figure 43: Distribution of Number of Words in the corpus of mMARCO Dataset

Figure 45: Distribution of the number of words in the queries of WikiPediaRetrieval Dataset

Figure 42: Distribution of the number of words in the queries of mMARCO Dataset



Hindi which can only be answered by the above passage. The question should be difficult with only semantic similarity and should not contain any lexical overlap; that is, do not directly use the exact phrases as shown in the above passage. You can use synonyms and make the question as complex as possible. Only return the relevant question and nothing else in the format given below : \(<\)QUESTION\(>\): ...question in hindi.... \(</\)QUESTION\(>\)

### Ablation of LoRA parameters

Similar to Schmidt et al. (2024), we also experiment with the addition of LoRA adapters during training on top of the E5 model. The LoRA adapters were applied to the "query", "key", and "value" layers of the retrieval model head, with a rank of 32, an alpha parameter of 64, and a dropout rate of 0.05.

Table 5 presents a side-by-side comparison of the NLLB-E5 model with and without LoRA adapters applied to the E5 head. The results indicate a slight performance boost on certain datasets, particularly those derived from the original BEIR benchmark, which closely resembles the training data. However, it is important to note a significant drop in performance on non-BEIR datasets, such as CC-News Retrieval, MIRACL, IndicQA Retrieval, mMARCO, and Wikipedia Retrieval. We hypothesize that this drop could be due to the LoRA adapters causing the model to overfit on datasets in the style of Sentence Transformers, thereby reducing its generalization capabilities. The overall stronger performance of the model without LoRA adapters is further supported by its higher average NDCG@10 score, and thus, we decided to proceed with the "without LoRA" model setup.

### Performance of NLLB Encoder

We trained the NLLB-Encoder using the same knowledge distillation approach as NLLB-E5 and evaluated its performance on subsets of both English and Hindi datasets from BEIR and Hindi-BEIR. The results, presented in Table 6, reveal a significant performance gap between NLLB-Encoder and both NLLB-E5 and mE5-Large in the retrieval task. These findings indicate that NLLB-Encoder alone is insufficient to act as a retrieval model and requires additional components, thereby motivating the development of NLLB-E5.