RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
In-Context Learning
In-context learning (ICL) leverages the intrinsic abilities of language models to learn and infer new tasks without the need for parameter updates. Formally, let \pi_{\text{LM}} be a language model with a vocabulary \mathcal{V} .
Consider a downstream generation task with input space \mathcal{X} and output space \mathcal{Y} .
For a given test query \mathbf{x}_{\text{test}}\in\mathcal{X} and a retrieved subset of K input-output pairs \{(\mathbf{x}_{i},\mathbf{y}_{i})\}_{i=1}^{K}\in\mathcal{X}\times\mathcal{Y} describing the intended task, ICL generates the output \mathbf{y}_{\text{test}}\in\mathcal{Y} as follows:

\mathbf{y}_{test}\sim\pi_{LM}(\cdot|\mathbf{x}_{1},\mathbf{y}_{1},\mathbf{x}_{2},\mathbf{y}_{2},\dots,\mathbf{x}_{K},\mathbf{y}_{K},\mathbf{x}_{test})

Here, \sim represents the sampling techniques commonly used in the literature, such as Greedy Sampling, Top-p Sampling (Holtzman et al., 2019 ) , Top-k Sampling (Fan et al., 2018 ) , and Beam Search (Freitag and Al-Onaizan, 2017 ) . Each in-context example a_{i}=(x_{i},y_{i})\in\mathcal{X}\times\mathcal{Y} is drawn from a training set example bank \mathcal{D}=\left\{(\mathbf{x}_{i},\mathbf{y}_{i})\right\}_{i=1}^{N} of input-output sequences.
Example Retrieval for Few-shot learning
Our main goal is to train a retriever \mathcal{R}_{\phi}(\mathbf{x}_{\text{test}},\mathcal{D}) model parameterized by \phi , that retrieves a set of in-context examples \{a_{i}\}_{i=1}^{K}\subset\mathcal{D} given a test sample \mathbf{x}_{\text{test}} where typically K\ll N . Usually, \phi:(\mathcal{X}\cup\mathcal{Y})^{\star}\rightarrow\mathbb{R}^{d} represents 2 2 2 \mathcal{A}^{\star} denotes the smallest superset of \mathcal{A} closed under set concatenation - known as Kleene operator. In this case (\mathcal{X}\cup\mathcal{Y})^{\star} corresponds to the concatenation of strings in \mathcal{X} and \mathcal{Y} . the embedding function that maps the text in the input space \mathcal{X} into a d -dimensional vector representation. Such representations are subsequently
used to measure similarity between samples. Previous works have explored various retrieval strategies, ranging from random example selection to using off-the-shelf retrieval models (Liu et al., 2021 ; Wu et al., 2022 ) , as well as fine-tuning the retriever’s embeddings (Ye et al., 2023 ; Rubin et al., 2021 ) .
Determinantal Point Processes
In this subsection, we introduce a recently studied framework Determinantal Point Processes (DPPs) (Ye et al., 2023 ) for example selection that ranks subsets of examples rather than individual ones.
Introduced in Macchi ( 1975 ) , DPP’s are elegant probabilistic models that have been extensively used in the literature (Ye et al., 2023 ; Borodin and Olshanski, 2000 ; Benard and Macchi, 1973 ; Kulesza et al., 2012 ; Liu et al., 2022 ) to capture negative correlation among items.

We leverage the DPP framework to promote diversity within the set of retrieved in-context examples. Formally, a point process \mathcal{P} is called a DPP if, for any random subset Y drawn according to \mathcal{P} , the probability that a subset S is contained within Y is given by:

\mathcal{P}(S\subseteq Y)=det(\mathbf{Z}_{S})

where \mathbf{Z}\in\mathbb{R}^{n\times n} is a PSD similarity matrix, and \mathbf{Z}_{S} denotes the submatrix of K corresponding to the rows and columns indexed by S .
For in-context learning, given a test sample \mathbf{x}_{\text{test}} , the input dependent similarity of any two examples \mathbf{a}_{i},\mathbf{a}_{j} is denoted by \mathbf{Z}_{ij} - formally, we model \log\mathbf{Z}_{ij} as

\sum_{k\in\{i,j\}}\log\phi(\mathbf{a}_{k})^{T}\phi(\mathbf{x}_{test})+\log{\phi}(\mathbf{a}_{i})^{\top}\phi(\mathbf{a}_{j})

where \phi(\mathbf{a}_{i})^{T}\phi(\mathbf{x}_{\text{test}})\in\mathbb{R}^{+} measures the relevance of \mathbf{a}_{i} to input \mathbf{x}_{\text{test}} and \phi(\mathbf{a}_{i})^{T}\phi(\mathbf{a}_{j}) measures the similarity between the i^{\text{th}} and j^{\text{th}} example.
Proposed Framework
Problem Setup. Despite the growing focus on evaluating the multilingual capabilities of large language models (LLMs), there remains a substantial performance gap between high-resource languages and those with limited web resources (Ahuja et al., 2023 ; Singh et al., 2024 ) . In this study, we aim to address this gap by enhancing the few-shot performance of low-resource Indic languages. To this end, we adopt the recently released IndicGen Benchmark (Singh et al., 2024 ) , targeting a subset of Indic languages for our analysis. Specifically, we focus on the following low-resource languages:

Additionally, we assume the availability of data from relatively high-resource Indic languages, which we refer to as the auxiliary dataset. This assumption is justified by the fact that these languages have relatively ample web-text resources (Singh et al., 2024 ) .
Our Approach: PromptRefine
To enhance the performance of language models on low-resource languages, we introduce PromptRefine , a three-step framework that: 1) identifies closely related high-resource Indic languages and leverages associated example banks (Section 4.1.1 ), 2) iteratively refines retriever embeddings \mathcal{R}_{\phi} (Section 4.1.2 ), and 3) incorporates diversity-based finetuning of retriever \mathcal{R}_{\phi} to rank subsets of in-context examples for a given input query (Section 4.1.3 ). The complete approach is outlined in Algorithm 1 . In what follows, we provide an in-depth overview of each component.
Auxiliary Dataset Selection
Despite the advent of numerous LLMs, low-resource Indic languages constitute only a negligible portion of their pre-training corpora, resulting in a suboptimal performance for generation tasks in these languages. To address this, we propose using relatively high-resource Indic languages, such as Hindi and Bengali, as auxiliary datasets.
Our approach involves selecting auxiliary languages that are closely related to the target Indic low-resource language. Specifically, for each low-resource language, we compute the cosine similarity between the embeddings and those of the auxiliary languages. An auxiliary language is included if its similarity score surpasses a threshold parameter \delta ,
as outlined in Algorithm 2 (Appendix H ). The underlying motivation for this approach is that for an input query in a low-resource Indic language, relevant examples that can guide the LLM in generation might not be present in the associated example bank - therefore,
by incorporating examples from the related high-resource languages, we provide the LLM additional context or rather relevant guidance that can help improve the model’s performance on the input query. However, the following critical challenge is now raised: "How can we effectively integrate the diverse information from auxiliary datasets for optimal performance?"
Iterative Prompt Refinement
We denote the example bank of the low-resource target language \mathcal{T} as \mathcal{D}^{\mathcal{T}}=\left\{(\mathbf{x}_{i},\mathbf{y}_{i})\right\}_{i=1}^{N} , the selected set of auxiliary languages (using Alg. 2 ) as \mathcal{H}=\{\mathcal{H}_{1},\cdots,\mathcal{H}_{M}\} and the auxiliary example banks as \mathcal{D}^{\text{aux}}=\{\mathcal{D}^{\mathcal{H}_{1}},\cdots,\mathcal{D}^{\mathcal{H}_{M}}\} . Note that the number of high-resource auxiliary example banks M is determined by the threshold parameter \delta , as defined in Section 4.1.1 . Our goal is to train a single retriever \mathcal{R}_{\rho}(\mathbf{x}_{\text{test}},\mathcal{D}^{\text{aux}}\cup\mathcal{D}^{\mathcal{T}}) - however the challenge is both the shared representation space for all example banks combined and the parameter weights \phi are unknown. At the same time, the retriever must capture the specific traits of each individual language. Balancing these requirements requires an alternating optimization procedure.

To maximize the information gain from the auxiliary example banks, we propose an Alternating Minimization (AM) framework that alternately performs the following two steps successively until convergence 1) (Specialize) Fine-tune relevance-based retrievers \{\mathcal{R}_{\phi_{i}}\}_{i} on each of several selected languages by retraining only on the example bank associated with the corresponding language (Step 7 in Alg. 1 ). Intuitively, the goal of this step is to allow a particular retriever to acquire task / language-specific knowledge associated with the corresponding language.
Note that each of the individual retrievers \{\mathcal{R}_{\phi_{i}}\}_{i} is initialized with pre-trained multilingual BERT encoder weights at the beginning of first iteration - in subsequent iterations, all the individual retrievers are initialized with shared parameter weights \rho computed in the next step 2) (Merge) merges the individual retrievers \{\mathcal{R}_{\phi_{i}}\}_{i} into a single retriever \mathcal{R}_{\rho} by simple parameter averaging to obtain a shared representation space enabling cross-language retrieval (Step 10 in Alg. 1 ). Intuitively, the shared retriever \mathcal{R}_{\rho} encapsulates the diverse knowledge learnt by each individual retriever.

In essence, our alternating minimization algorithm (Alg. 1 ) alternately finetunes the individual retriever on language-specific example bank and creates a merged retriever enabling a shared representation space for \mathcal{I} iterations.
We denote the model with the highest validation accuracy on the target language \mathcal{T} after \mathcal{I} iterations as \rho^{*} (Step 11 in Alg. 1 ).
Divsersity-induced finetuning
A major limitation of relevance-based finetuning is that the in-context examples are retrieved solely based on relevance, thereby ignoring diversity and inter-relationship among the selected examples (Ye et al., 2023 ) . To overcome this challenge, we leverage the DPP framework to enhance diversity within the retrieved in-context examples. Specifically, we obtain the final retriever model by fine-tuning \rho^{*} on the merged dataset \widetilde{\mathcal{D}}=\mathcal{D}^{\mathcal{T}}\cup\mathcal{D}^{\text{aux}} . Due to the technically involved training procedure of the DPP framework via contrastive learning, we delegate the training details to the Appendix D .
Note that during inference, we use the retriever model further fine-tuned in the DPP framework to retrieve both diverse and relevant samples (Ye et al., 2023 ) . Specifically, we perform MAP inference using the model within the DPP framework - however exact MAP inference is NP-Hard (Ko et al., 1995 ) since it involves an exact search over an exponentially large collection of subsets of examples. However, in our setting, we can use a greedy but computationally efficient procedure to build the subset by inserting one example at a time (Chen et al., 2018 ) .
Main Results
Evaluation on open-source LLMs. We present the evaluation results on state-of-art open-source models, including LLAMA-3.1-8B, LLAMA-2-7B, Qwen-2-7B, and Qwen-2.5-7B, for Cross-Lingual QA task (Table 1 ), Machine Translation task (Table 2 ), Cross-lingual Summarization task (Table 3 ), and Multi-lingual QA task (Table 4 ). Our empirical findings yield several key insights: (1) Across all diverse tasks, PromptRefine consistently outperforms both finetuning-based and finetuning-free baselines, with a improvement of up to 2.09x compared to current state-of-art. Notably, for Cross-lingual QA task, for Manipuri, PromptRefine improves the Token-F1 by +10.21 over CEIL (Ye et al., 2023 ) . (2) Compared to fine-tuning free baselines such as Top-K, PromptRefine provides a staggering improvement of upto 3.38x on low-resource language such as Bodo. Further, when compared to finetuning-based retrievers like EPR Rubin et al. ( 2021 ) and CEIL, PromptRefine consistently delivers superior performance across all tasks, as exemplified by a +3.21 Token-F1 improvement on Maithili. (3) Interestingly, even when auxiliary datasets are available, other finetuning-based retrievers such as EPR Rubin et al. ( 2021 ) and CEIL Ye et al. ( 2023 ) do not exhibit significant improvements. This finding highlights the importance of learning a better representation space to effectively integrate and leverage auxiliary data. In contrast, PromptRefine successfully utilizes auxiliary data to generate a richer set of in-context examples, as reflected by its substantial performance gains across multiple tasks.

Evaluation on proprietary LLMs. To ensure a comprehensive evaluation, we also test PromptRefine on proprietary LLMs such as GPT-3.5/4 OpenAI et al. ( 2024 ) . However, since we do not have access to the output logits in proprietary models, required for scoring candidate examples, we use a different open-source LLM architecture for scoring. To be specific, for this experiment, we employ LLAMA-3.1-8B as the scorer model \pi_{\text{Scorer}} . The results for the translation task are reported in Table 5 (Appendix). Consistent with our findings using open-source LLMs, PromptRefine significantly improves generation quality, as measured by chrF1, compared to other retrieval methods.
Discussion
Note, ablation studies on threshold parameter \delta for selecting related languages and number of ICL examples are deferred to Appendix C .

Why is choosing a closely related auxiliary example bank important? In Section 4.1.1 , we discussed the approach of selecting a closely related high-resource Indic language as an auxiliary example bank for a target low-resource language. To highlight importance of selecting the right set of related languages, we present ablation studies in this section.

We evaluate three setups: (1) Our method (Alg. 2 ) of selecting the most closely related language as the auxiliary example bank, (2) selecting a random language, and (3) selecting the most unrelated language. We show the few-shot performance on cross-lingual QA task in Figure 1 , with retrievers trained under each setup. Our findings show that: (1) As expected, using the most closely related language as the auxiliary example bank yields the best performance, as it provides relevant guidance to the LLM, and (2) Selecting a random or unrelated language results in little to no improvement, with performance remaining close to the relevance-based EPR baseline Rubin et al. ( 2021 ) that only uses target-language specific data.

Importance of Diversity-Induced Fine-Tuning. A key factor contributing to the improved performance of PromptRefine is the retrieval of a diverse set of in-context examples. Merging auxiliary example banks imply a larger overall example bank and therefore incorporating diversity during example selection becomes important to improve generalization. In this section, we empirically demonstrate the significance of diversity-induced fine-tuning (Section 4.1.3 ).
Figure 2 compares the text generation quality in Santali-to-English translation task, where examples are selected using two retriever models namely our approach PromptRefine with and with-out diversity-induced fine-tuning - in the latter, we skip line 15 in Alg. 1 . Results clearly show that incorporating diversity training leads to performance improvements across different LLMs, underscoring its effectiveness.

Importance of Alternating-Minimization approach. In Section 4.1.2 , we proposed an alternating minimization approach to alternately fine-tune the retriever on language-specific data and merge the parameters in order to learn a shared representation space that captures the knowledge associated with both the target language \mathcal{T} and the auxiliary languages \mathcal{H} . In Figure 3 , we plot the validation accuracy across different iterations for the merged retriever \rho and the target language retriever \phi_{\mathcal{T}} . The results show that merging the retriever embeddings provides a significant boost in validation accuracy over the iterations, thereby justifying the effectiveness of the parameter-averaging strategy.
Conlusion
The pre-training data for state-of-the-art LLMs Dubey et al. ( 2024 ); Yang et al. ( 2024 ) predominantly comprises English text, resulting in suboptimal performance on low-resource Indic languages Singh et al. ( 2024 ) . To address this, we propose a novel Alternating Minimization approach PromptRefine for example selection that improves ICL performance on low-resource Indic languages. Our approach follows a three-step framework: (1) identifying closely related high-resource Indic languages and utilizing their example banks, (2) iteratively refining retriever embeddings, and (3) employing diversity-based finetuning to rank subsets of in-context examples for a given input query. Comprehensive testing across various tasks and LLMs demonstrates that PromptRefine significantly enhances text generation quality on low-resource Indic languages.
Limitations
In Section 5.1 , we demonstrated the effectiveness of our proposed framework across diverse tasks. However, PromptRefine has the following limitations:

To improve text generation quality in low-resource Indic languages, PromptRefine depends on the availability of example banks from closely related, relatively high-resource Indic languages to provide additional context. We believe this is a reasonable assumption, as publicly available web-text resources are accessible for high-resource languages.

As explained in Algorithm 1 , our proposed approach involves three key steps: (1) identifying closely related example banks, (2) iterative refinement of retriever embeddings, and (3) diversity-based finetuning. These steps could have been arranged in several different configurations. However, we empirically tested several alternatives and found that our proposed approach consistently delivers the best performance improvements.
Figure 1: Token-F1 evaluation on the cross-lingual QA task in Manipuri, with retrievers trained using different auxiliary high-resource example banks: (1) Closely related language, (2) Random language, and (3) Unrelated language.; Token-F1 evaluation on the cross-lingual QA task in Manipuri, with retrievers trained using different auxiliary high-resource example banks: (1) Closely related language, (2) Random language, and (3) Unrelated language.
Figure 2: To highlight the importance of DPP training, we compare the quality of responses generated with and without diversity-induced fine-tuning on the task of translating from Santali to English.; To highlight the importance of DPP training, we compare the quality of responses generated with and without diversity-induced fine-tuning on the task of translating from Santali to English.
Figure 3: We plot the validation accuracy (measured by Token-F1) obtained by individual and merged retrievers over different iterations of finetuning.; We plot the validation accuracy (measured by Token-F1) obtained by individual and merged retrievers over different iterations of finetuning.
Figure 4: We vary the threshold parameter \delta to evaluate its impact on model performance. Our results show that setting \delta to the 95th percentile of cosine similarity values between target language embeddings and high-resource languages yields optimal performance. The evaluation task for this experiment is cross-lingual QA on Bodo.; We vary the threshold parameter \delta to evaluate its impact on model performance. Our results show that setting \delta to the 95th percentile of cosine similarity values between target language embeddings and high-resource languages yields optimal performance. The evaluation task for this experiment is cross-lingual QA on Bodo.
Figure 5: We plot the chrF-1 score for different values of K\in\{1,2,4,8,16\} , where K represents the number of in-context examples. The task for this experiment is translation from Rajasthani to English and LLM is LLAMA-3.1-8B.; We plot the chrF-1 score for different values of K\in\{1,2,4,8,16\} , where K represents the number of in-context examples. The task for this experiment is translation from Rajasthani to English and LLM is LLAMA-3.1-8B.
Evaluation on XorQA-In. We report Token-F1 results for performance on Bodo, Manipuri and Maithili. The evaluation includes three LLMs: Qwen-2-7B (QW-2-7B), Qwen-2.5-7B (QW-2.5-7B), and LLAMA-3.1-8B (LM-3.1-8B).
Evaluation on Flores-In. We report chrF1 results for translation performance from three low-resource languages to English. The evaluation includes three LLMs: LLAMA-2-7B (LM-2-7B), LLAMA-3.1-8B (LM-3.1-8B), and Qwen-2-7B (QW-2-7B).
Evaluation on CrossSum-In. We report chrF1 results for the summarization performance of an English text into four low-resource languages. The evaluation includes two recent LLMs: LLAMA-3.1-8B (LM-3.1-8B), and Qwen-2-7B (QW-2-7B).
Evaluation on Xquad-In. We report Token-F1 results for QA performance from Odia to English. The evaluation includes three LLMs: Qwen-2-7B (QW-2-7B), Qwen-2.5-7B (QW-2.5-7B), and LLAMA-3.1-8B (LM-3.1-8B).
Evaluation on Proprietary LLMs. We report chrF1 results for translation performance from Santali (sat) and Manipuri (mni) to English on GPT-3.5/GPT-4. For this evaluation, we used LLAMA-3.1-8B as the scorer LLM.
Prompt template used for various datasets.