Introduction
Large Language Models (LLMs) have recently made remarkable progress, demonstrating human-level performance across a wide range of tasks (Adiwardana et al., 2020 ; Wang et al., 2019 ) . However, despite these advancements, most LLMs, such as LLaMA-3 Dubey et al. ( 2024 ) , LLaMA-2 Touvron et al. ( 2023 ) , and Qwen Yang et al. ( 2024 ) , are predominantly pre-trained on English texts, leading to significant performance disparities when applied to low-resource, non-English languages (Ahuja et al., 2023 ) . The scarcity of ground-truth paired data in many low-resource languages makes text generation particularly challenging, as fine-tuning LLMs becomes infeasible in such settings. This issue is especially pronounced in lesser-known Indic languages such as Tibetan which has only around 5000 Wikipedia articles compared to 6.6 M+ Wikipedia articles in English, despite having approximately 6 M speakers. In this work, we focus on downstream generation tasks with low-resource Indic languages, a critical challenge for making LLMs more widely accessible.

When downstream tasks have limited labeled data, few-shot learning or in-context learning (ICL) has emerged as a powerful and practical approach for text generation (Tanwar et al., 2023 ; Zhang et al., 2021 ; Winata et al., 2021 ; Huang et al., 2023 ; Etxaniz et al., 2023 ) . ICL operates by providing the LLM with a prompt that consists of task-specific instructions and a set of input-output examples (demonstrations) to guide the model’s output generation for a specific input query. While ICL is computationally efficient (it requires no parameter updates), it faces two critical challenges when labeled data is scarce, particularly for low-resource Indic languages.

( Relevance ) First, as in any learning task with limited data, the small size of the available example pool can result in a lack of relevant examples to guide the model effectively. In the context of ICL, this scarcity can severely degrade performance, as the quality of examples is crucial. Poor example selection can lead to performance worse than zero-shot scenarios, while optimal selection can achieve near state-of-the-art results (Liu et al., 2021 ) . (Diversity) Second, existing example selection techniques, such as random selection or retrieving semantically similar examples Zhang et al. ( 2021 ); Winata et al. ( 2021 ); Tanwar et al. ( 2023 ) , often struggle to account for diversity among the selected examples, which can further limit performance. An appropriate diverse set of selected examples in the prompt can improve generalization significantly.

Proposed Approach. In this work, we focus on an ICL approach for enhancing LLM performance in low-resource Indic languages. Specifically, our goal is to identify an optimal subset of guiding examples to be inserted into the prompt, without modifying the LLM parameters. Rather, we train retrievers to extract a near optimal set of examples (demonstrations) based on the input query. To this end, we propose a novel trainable approach PromptRefine that combines examples from auxiliary example banks associated with the related task. While PromptRefine is broadly applicable across different language families and even low-resource task clusters, in this study we focus on Indic languages Singh et al. ( 2024 ) , to enhance performance specifically on low-resource Indic languages through improved example selection.

Given a specific input text generation task in a low-resource Indic language, we use the following set of key ideas to enhance model performance (see Algorithm 1 ): 1) First, we utilize example banks from closely-related relatively high-resource 1 1 1 High resource language refers to a language in which a relatively large volume of pre-training data for training an LLM compiled from web-text is available. Indic languages to provide relevant guidance, improving LLM performance on low-resource tasks. Therefore, the first step in our algorithmic approach given an input low-resource task is to select relevant high-resource languages (see Algorithm 2 ) with associated example banks. 2) Second, merging these example banks is non-trivial, as each language-specific retriever is trained on a unique representation space. To address this, we adapt data-scarce multi-task learning techniques (Thekumparampil et al., 2021 ) to align the individual retrievers into a shared representation space, enabling cross-language retrieval. 3) Third, we incorporate diversity in the selected examples to improve generalization and reduce bias during generation.

To empirically validate the effectiveness of PromptRefine , we evaluate its performance across four distinct text generation tasks on a subset of languages outlined in Singh et al. ( 2024 ) : Cross-Lingual Question Answering (XorQA-In), Multilingual Question-Answering (XQuAD-In), Machine Translation (Flores-In), and Cross-Lingual Summarization (CrossSum-In), using recent LLMs such as LLAMA-3.1-8B (Dubey et al., 2024 ) , LLAMA-2-7B (Touvron et al., 2023 ) , Qwen-2-7B (Yang et al., 2024 ) , and Qwen-2.5-7B (Team, 2024 ) . Our results demonstrate that PromptRefine significantly improves generation performance in all tasks compared to baseline approaches for example selection. Specifically, using LLAMA-3.1-8B as the LLM, PromptRefine achieves a Token-F1 improvement of +\mathbf{16.07} and +\mathbf{8.26} over zero-shot prompting and the current state-of-the-art retriever (Ye et al., 2023 ) , respectively, in the cross-lingual QA task. Similarly, using Qwen-2-7B, we observe an improvement in chrF1 of up to +\mathbf{7.77} on the machine translation task compared to the baseline of selecting semantically similar examples. To ensure a comprehensive evaluation, we also test PromptRefine on proprietary LLMs such as GPT-3.5 and GPT-4 OpenAI et al. ( 2024 ) , where PromptRefine outperforms baseline retrievers on translation task, aligning with our previous findings.