Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages (2024)

Abstract: Large language models (LLMs) as listwise rerankers have shown impressive zero-shot capabilities in various passage ranking tasks. Despite their success, there is still a gap in existing literature on their effectiveness in reranking low-resource languages. To address this, we investigate how LLMs function as listwise rerankers in cross-lingual information retrieval (CLIR) systems with queries in English and passages in four African languages: Hausa, Somali, Swahili, and Yoruba. We analyze and compare the effectiveness of monolingual reranking using either query or document translations. We also evaluate the effectiveness of LLMs when leveraging their own generated translations. To grasp the general picture, we examine the effectiveness of multiple LLMs—the proprietary models RankGPT4 and RankGPT3.5, along with the open-source model RankZephyr. While the document translation setting, i.e., both queries and documents are in English, leads to the best reranking effectiveness, our results indicate that for specific LLMs, reranking in the African language setting achieves competitive effectiveness with the cross-lingual setting, and even performs better when using the LLM’s own translations.

heading: 1 Introduction
text: Several studies have shown that large language models (LLMs) excel in various NLP tasks (Zhou et al., 2022; Zhu et al., 2023; Wang et al., 2023). In text ranking, LLMs have been used effectively as retrievers (Ma et al., 2023a) and in both pointwise and listwise reranking. In reranking, models may generate an ordered list directly (Sun et al., 2023; Ma et al., 2023b; Pradeep et al., 2023a; Tamber et al., 2023) or sort based on token probabilities (Ma et al., 2023b). The large context size of LLMs makes listwise approaches particularly attractive because the model attends to multiple documents to produce a relative ordering.
Cross-lingual retrieval aims to provide information in a language different from that of the search query. This is especially relevant when the required information is not available or prevalent in the query’s language, as is the case for most lowresource languages. Previous work has examined sparse and multilingual dense retrieval models in cross-lingual settings for these languages (Zhang et al., 2023b; Ogundepo et al., 2022). However, studies on the effectiveness of LLMs as crosslingual retrievers or rerankers for low-resource languages are few to non-existent.
In this study, we examine the effectiveness of proprietary and open-source models for listwise reranking in low-resource African languages. Our investigation is guided by the following research questions: (1) How well do LLMs fare as listwise rerankers for low-resource languages? (2) How effectively do LLMs perform listwise reranking in cross-lingual scenarios compared to monolingual (English or low-resource language) scenarios? (3) When we leverage translation, is reranking more effective when translation uses the same LLM used for zero-shot reranking?
We answer these questions through an extensive investigation of the effectiveness of RankGPT (Sun et al., 2023) and RankZephyr (Pradeep et al., 2023b) in cross-lingual and monolingual retrieval settings. We use CIRAL (Adeyemi et al., 2023), a cross-lingual information retrieval dataset covering four African languages with queries in English and passages in African languages, and construct monolingual retrieval scenarios through document and query translations.
Our results show that cross-lingual reranking with these LLMs is generally more effective compared to reranking in the African languages, underscoring that they are better tuned to English than low-resource languages. Across all languages, we achieve our best results when reranking entirely in English using retrieval results obtained by doc-
650
ument translation. In this setting, we see up to 7 points improvement in nDCG@20 over crosslingual reranking using RankGPT4, and up to 9 points over reranking in African languages. We specifically notice improvements with RankGPT4 when using its query translations for reranking in African languages.

heading: 2 Background and Related Work
text: Given a corpus C = {D1, D2, ..., Dn} and a query q, information retrieval (IR) systems aim to return the k most relevant documents. Modern IR pipelines typically feature a multi-stage architecture in which a first-stage retriever returns a list of candidate documents that a reranker reorders for improved quality (Asadi and Lin, 2013; Nogueira et al., 2019; Zhuang et al., 2023).
More recently, the effectiveness of decoder models as rerankers (dubbed “prompt decoders”) has been explored in some depth. Researchers have fine-tuned GPT-like models in the standard contrastive learning framework (Neelakantan et al., 2022; Muennighoff, 2022; Zhang et al., 2023a) and studied different approaches to reranking using both open-source LLMs and proprietary GPT models. Sun et al. (2023) evaluated the effectiveness of OpenAI models on multiple IR benchmarks using permutation generation approaches, while Ma et al. (2023b) demonstrate the effectiveness of GPT-3 as a zero-shot listwise reranker and the superiority of listwise over pointwise approaches.
While these papers focus on reranking with LLMs, they only cover two African languages— Swahili and Yoruba. For both languages, GPT-3 improves over BM25 significantly but still falls behind supervised reranking baselines. In this work, we examine the effectiveness of these LLMs as components of IR systems for African languages. Specifically, we study the effectiveness of opensource and proprietary LLMs as listwise rerankers for four African languages (Hausa, Somali, Swahili, and Yoruba) using the CIRAL cross-lingual IR test collection (Adeyemi et al., 2023).
To be more precise, cross-lingual information retrieval (CLIR) is a variant of the standard retrieval task in which the queries qi are in a different language from the documents in the corpus C. Popular approaches to CLIR include query translation, document translation, and language-independent representations (Lin et al., 2023). As the focus of this work is on the effectiveness of LLMs as listwise
rerankers in cross-lingual settings, we primarily explore document and query translation approaches in this study.

heading: 3 Methods
text: Listwise Reranking. In listwise reranking, LLMs compare and attribute relevance over multiple documents in a single prompt. As this approach has been proven to be more effective than pointwise and pairwise reranking (Ma et al., 2023b; Pradeep et al., 2023a), we solely employ listwise reranking in this work. For each query q, a list of provided documents D1, ..., Dn is reranked by the LLM, where n denotes the number of documents that are inserted into the prompt.
Prompt Design. We adopt RankGPT’s (Sun et al., 2023) listwise prompt design as modified by Pradeep et al. (2023a). The input prompt and generated completion are presented in Figure 1.
LLM Zero-Shot Translations. We examine the effectiveness of LLMs in using their translations in crossing the language barrier. For a given LLM, we generate zero-shot translations of queries from English to African languages and implement reranking with the LLM using its translations. With this approach, we are able to examine the ranking effec-
tiveness of the LLM solely in African languages, and examine the correlation between its translation quality and reranking. The prompt design for generating the query translation is shown in Figure 2.

heading: 4 Experimental Setup
text: Models. We implement zero-shot reranking for African languages with three models. These include proprietary reranking LLMs: RankGPT4 and RankGPT3.5, using the gpt-4 and gpt-3.5-turbo models, respectively, from Azure’s OpenAI API. To examine the effectiveness of open-source LLMs, we rerank with RankZephyr (Pradeep et al., 2023b), an open-source reranking LLM obtained by instruction-fine-tuning Zephyrβ (Tunstall et al., 2023) to achieve competitive effectiveness with RankGPT models.
Baselines. We compare the reranking effectiveness of the LLMs using already established models as baselines. Our baselines include two crossencoder models, the multilingual T5 (mT5) (Xue et al., 2021) and AfrimT5 (Adelani et al., 2022), which is mT5 with continued pre-training on African corpora. The mT51 and AfrimT52 rerankers were obtained from fine-tuning the base versions of both models on the MS MARCO passage collection (Bajaj et al., 2016) for 100k iterations, with a batch size of 128.
Test Collection. Models are evaluated on CIRAL (Adeyemi et al., 2023), a CLIR test collection consisting of four African languages: Hausa, Somali, Swahili, and Yoruba. Queries in CIRAL are natural language factoid questions in English while passages are in the respective African languages. Each language comprises between 80 and 100 queries, and evaluations are done using the
1https://huggingface.co/castorini/ mt5-base-ft-msmarco 2https://huggingface.co/castorini/ afrimt5-base-ft-msmarco
pooled judgments obtained from CIRAL’s passage retrieval task.3 We also make use of CIRAL’s translated passage collection4 in our document translation scenario. The test collection’s documents were translated from the African languages to English using the NLLB machine translation model (Costajussà et al., 2022).
We report nDCG@20 scores following the test collection’s standard, and MRR@100.
Configurations. First-stage retrieval uses BM25 (Robertson and Zaragoza, 2009) in the open-source Pyserini toolkit (Lin et al., 2021). We use whitespace tokenization for passages in native languages and the default English tokenizer for the translated passages. Our BM25 retrieval is implemented using document (BM25-DT) and query (BM25-QT) translations. For BM25-QT, queries are translated with Google Machine Translation (GMT).
We rerank the top 100 passages retrieved by BM25 using the sliding window technique by Sun et al. (2023) with a window of 20 and a stride of 10. Experiments were conducted using the RankLLM toolkit.5 We use a context size of 4,096 tokens for RankGPT3.5 and RankZephyr, and 8,192 tokens for RankGPT4. These context sizes are also maintained for the zero-shot LLM translation experiments. For each model, translation is performed over three iterations and we vary the model’s temperatures from 0 to 0.6 to allow variation in the translations. Translations are only obtained for the GPT models since RankZephyr is suited only for reranking. Reranking results are reported over a single run, except with the LLM translations where we take the Reciprocal Rank Fusion (RRF) (Cormack et al., 2009) of results from the three iterations.

heading: 5 Results and Discussion

heading: 5.1 Cross-Lingual vs. Monolingual Reranking
text: Table 1 compares results for cross-lingual reranking using CIRAL’s queries and passages unmodified, and also the English reranking scenario. Row (1) reports scores for the two first-stage retrievers, BM25 with query translation (BM25-QT) and document translation (BM25-DT). Cross-lingual reranking scores for the different LLMs are presented in Row (2), and we employ BM25-DT for first-stage retrieval given it is more effective. 3https://ciralproject.github.io/ 4https://huggingface.co/datasets/CIRAL/ ciral-corpus#translated-dataset 5https://github.com/castorini/rank_llm
Scores for reranking in English are reported in Row (3), and results show this to be the more effective scenario across the LLMs and languages. However, the cross-encoder T5 baselines have better reranking effectiveness in the cross-lingual scenario.
Improved reranking effectiveness with English translations is expected, given that LLMs, despite being multilingual, are more attuned to English. The results obtained from reranking solely with African languages further probe the effectiveness of LLMs in low-resource language scenarios. We report scores using query translations in Table 2, with BM25-DT also as the first-stage retriever for a fair comparison. In comparing results from the query translation scenario to the cross-lingual results in Row (2) of Table 1, we generally observe better effectiveness with cross-lingual. However, RankGPT4 obtains higher scores for Somali, Swahili, and Yoruba in the African language scenario, especially with its query translations, comparing Rows (2a) in Table 1 and 2.

heading: 5.2 LLM Reranking Effectiveness
text: We compare the effectiveness of the different LLMs across the reranking scenarios. RankGPT4 generally achieves better reranking among the 3 LLMs, as presented in Tables 1 and 2. In the crosslingual and English reranking scenarios, the opensource LLM RankZephyr (Pradeep et al., 2023b) achieves better reranking scores in comparison with RankGPT3.5 as reported in Rows (*b) and (*c) in Table 1. RankZephyr also achieves comparable scores with RankGPT4 in the English reranking scenario, and even a higher MRR for Somali as
reported in Row (3c) of Table 1. These results establish the growing effectiveness of open-source LLMs for language tasks considering the limited availability of proprietary LLMs, but with room for improvement in low-resource languages.
In comparing the reranking effectiveness of LLMs with that of the baseline models, scores vary depending on the scenario and specific LLM. Reranking scores of the cross-encoder T5 baselines are reported in Rows (*d) and (*e) of Tables 1 and 2. As seen in Rows (2d) and (2e) of Table 1, the crossencoder multilingual T5 baselines achieve higher reranking scores compared to all three LLMs. However, RankGPT4 outperforms both baselines in the English reranking scenario and using its query translations in the African language reranking scenario. We can attribute the higher effectiveness of the baselines to being fine-tuned for reranking as compared to the LLMs where reranking is carried out in a zero-shot fashion.

heading: 5.3 LLM Translations and Reranking
text: Given that RankGPT4 achieves better reranking effectiveness using its query translations in the monolingual setting, we further examine the effectiveness of this scenario. Row (2) in Table 2 reports results using LLMs translations, and we compare these to results obtained using translations from GMT. Compared to results obtained with GMT translations, RankGPT4 does achieve better monolingual reranking effectiveness in the African language using its query translations. RankGPT3.5 on the other hand achieves less competitive scores on average using its query translations when com-
pared to translations from the GMT model, with the exception of Yoruba where it has much higher scores using its translations.
Considering the effect of translation quality on reranking, we evaluate the LLMs’ translations and report results in Table 3. Evaluation is done against CIRAL’s human query translations using the BLEU metric. We observe better translations with GPT4 compared to GPT3.5, with GMT achieving the best quality. However, RankGPT4 still performs better using its query translations, indicating a correlation in the model’s understanding of the African languages.

heading: 6 Conclusion
text: In this work, we evaluate zero-shot cross-lingual reranking with large language models (LLMs) on African languages. Our suite covered three forms of LLM-based reranking: RankGPT4, RankGPT3.5 and RankZephyr. Using the listwise reranking method, our results demonstrate that reranking in English via translation is the most optimal. We examine the effectiveness of LLMs in reranking for low-resource languages in the cross-lingual and African language monolingual scenarios and find that LLMs have comparable effectiveness in both scenarios but with better results in cross-lingual. In the process, we also establish that good translations obtained from the LLMs do improve their rerank-
ing effectiveness in the African language reranking scenario as discovered with RankGPT4.
Additionally, while open-source models showcase slightly lower effectiveness than RankGPT4, they still largely improve over other proprietary models like RankGPT3.5, an important step towards the development of effective listwise rerankers for low-resource languages.

heading: 7 Limitations
text: While we provide valuable insights into the application of LLMs for reranking tasks in low-resource settings, our work is not without limitations. One constraint is the reliance on translations for achieving good reranking effectiveness, which inherently introduces dependencies on the quality of translation models and their compatibility with the target languages. Additionally, the scope of languages and models evaluated in this study, covering only a small spectrum of African languages and a mix of proprietary and open-source LLMs, remains limited in the broader context of low-resource language research.
Future research directions could address these limitations by exploring a wider array of lowresource languages and incorporating more diverse LLMs, including those specifically trained or finetuned on low-resource language datasets. Investigating alternative reranking pipelines that reduce reliance on translation or enhance the multilingual capabilities of LLMs directly could also offer new avenues for improving retrieval effectiveness in low-resource language settings.

heading: Acknowledgements
text: This research was supported in part by the Natural Sciences and Engineering Research Council
(NSERC) of Canada and Huawei Technologies Canada. Thanks to Microsoft for providing access to OpenAI LLMs on Azure via the Accelerating Foundation Models Research program. We also thank the anonymous reviewers for their constructive suggestions.

heading: Future Work
text: Future research directions could address these limitations by exploring a wider array of lowresource languages and incorporating more diverse LLMs, including those specifically trained or finetuned on low-resource language datasets. Investigating alternative reranking pipelines that reduce reliance on translation or enhance the multilingual capabilities of LLMs directly could also offer new avenues for improving retrieval effectiveness in low-resource language settings