Venue: NAACL 2024

# MDR: Model-Specific Demonstration Retrieval at Inference Time for

In-Context Learning

Huazheng Wang1, Jinming Wu2, Haifeng Sun2, Zixuan Xia, Daixuan Cheng,

Jingyu Wang, Qi Qi3, Jianxin Liao

State Key Laboratory of Networking and Switching Technology

Beijing University of Posts and Telecommunications

{wanghz,wjm_18,hfsun,zjjs2019xzx,wangjingyu,qiqi8266}@bupt.edu.cn {daixuancheng6,jxlbput}@gmail.com

Equal Contribution.Corresponding Author.

Footnote 1: Equal Contribution.

###### Abstract

Recently, retrieval-based in-context learning (ICL) methods for selecting demonstrations have been widely investigated. Existing methods train a dense retriever to retrieve the most appropriate demonstrations for a given test query, which improves ICL performance. However, we find that distinct LLMs exhibit different biases for "what is a good demonstration" since they possess differences in training data, model architectures and training methods. As a result, a demonstration suitable for one LLM may not be appropriate for others. Previous approaches ignore the model bias and fail to retrieve the most appropriate demonstrations for different inference LLMs, resulting in a degradation of ICL performance. To address this problem, we propose a simple yet effective metric to evaluate the appropriateness of demonstrations for a specific inference LLM. Furthermore, we introduce a **A**odel-specific **D**emonstration **R**etrieval (MDR) method for ICL at inference time, which considers the biases of different LLMs. We test MDR on seen and unseen tasks with multi-scale inference LLMs, such as GPT-Neo-2.7B, LLaMA-7B and Vicuna-13B. Experiments on 23 datasets across 11 data domains highlight the remarkable effectiveness of MDR, showcasing improvements of up to 41.2% in comparison to methods that neglect model biases. Our code will be publicly available at: [https://github.com/kiming-ng/MDR](https://github.com/kiming-ng/MDR).

+
Footnote †: footnotemark: [ENDFOOTNOTE]


simultaneously minimizing it for negative demonstrations. During inference, the demonstrations exhibiting the highest similarity are retrieved and concatenated with the test query to form the model input. This approach has shown better performance in specific task domains, such as question answering [46] and semantic parsing [25]. Recent efforts have shifted towards cross-task prompt retrieval [30] and unified multi-task retrieval [18] by incorporating signals from various tasks across multiple data domains to train one retriever, which has demonstrated impressive improvements.

However, distinct LLMs exhibit different biases [9, 7] for "what is a good demonstration" since they possess differences in training data, model architectures and training methods. As a result, a demonstration suitable for one LLM may not be appropriate for others. We conclude that existing methods fail to retrieve the most appropriate demonstrations for different inference LLMs. There are two key factors: (i) **Bias Introduced in Training Phase:** Previous methods solely rely on one scoring LM to identify positive and negative demonstrations. The inherent bias of the scoring LM can impact this identification process, potentially introducing noise to the retriever training. Consequently, this may lead to the retrieval of bad demonstrations when inference on different LLMs. (ii) **Bias Neglected in Inference Phase:** A high-quality demonstration should not only exhibit similarity to the test query, but also align with the bias of the inference model. Previous methods employ the similarity-based extraction of demonstrations, which greatly neglect the inference model's bias. To test the bias issue, we train 3 tailored retrievers on RTE dataset using three different scoring LMs, including GPT-Neo-2.7B, LLMA-7B and Vicuna-13B. Then we test the performance of each retriever on SST2 dataset under different inference LLMs, respectively. As shown in Fig 1, the one-to-one corresponding scoring LM and inference LLM perform the best while performances on different inference LLMs can be inferior, which illustrates that using a single scoring LM for various inference LLMs limits the performance of ICL. Since training multiple retrievers for multiple inference LLMs is costly, retrieving demonstrations that meet inference model's bias at inference stage is crucial to improve the ICL performance. However, directly estimating and evaluating the model bias is impractical, the evaluation of a demonstration's appropriateness for a specific inference LLM remains a challenging and under-explored aspect in ICL.

To address this limitation, we propose a simple yet effective eigenvalue-based evaluation metric to assess the appropriateness of demonstrations for a specific inference LLM. Based on this metric, we further propose MDR, a Model-specific Demonstration Retrieval (MDR) method for ICL at inference time. Given a pre-trained demonstration retriever, MDR retrieves the most appropriate demonstrations from a pre-constructed demonstration pool for a given test query and a specific inference LLM. Results in Fig 1 verify that MDR is effective of mitigating the bias issue by reducing the performance gap between the corresponding and the non-corresponding scoring-inference LLMs.

To evaluate MDR's generalization across diverse models, we test MDR on a variety of seen and unseen tasks with different LLMs at inference time. The outcomes highlight the effectiveness of MDR as a simple and straightforward method to select the most appropriate demonstrations for a specific LLM without any additional training or human annotation.

Our contributions are summarized as follows:

* We introduce a novel evaluation metric to measure the appropriateness of demonstrations for a specific inference LLM.
* We propose MDR, a simple and effective framework to retrieve the most appropriate demonstrations for different inference LLMs, without any training cost.
* Experiments on 23 datasets across 11 data domains under various LLMs (\(2.7{\rm B}\sim 13{\rm B}\)) show the validity and scalability of MDR.

## 2 Task Definition

We aim to improve the performance of ICL by retrieving the most appropriate demonstrations for any given task input. Specifically, given a pre-trained demonstration retriever, a test query \(x_{test}\) and an inference model \({\rm LM}\), we retrieve the top-\(K\) most appropriate demonstrations \(\{p_{j}\}^{K}_{j=1}\) from a pre-constructed demonstration pool \(\mathbb{P}\). These demonstrations are concatenated with the test query in ICL fashion. Our objective is to optimize the model prediction \(y^{*}_{test}\) to align with the ground truth \(y_{test}\), where \(y^{*}_{test}\) can be denoted by:

\[y^{*}_{test}=P_{{\rm LM}}(y^{*}_{test}|p_{K}\oplus...\oplus p_{1}\oplus x_{ test}).\] (1) 

Preliminaries

We give a detailed introduction to the framework of previous retrieval-based methods in this section. These methods rely on a retriever to retrieve demonstrations from a demonstration pool. The retriever is based on a bi-encoder architecture (Rahman and Ng, 2012) containing two encoders \(E_{q}\) and \(E_{p}\), where \(E_{q}\) encodes the test query and \(E_{p}\) encodes demonstrations. The training and inference phases of the retriever are outlined as follows.

### Training Phase

Existing works usually use the feedback signal of a LM to distinguish positive and negative samples, and train the retriever with contrastive learning. The details are as follows.

To obtain the training data, a demonstration pool \(\mathbb{P}\) is first constructed by selecting a certain number of data examples from multiple datasets covering various task types including Reading Comprehension, Closed-book QA, Paraphrase Detection, etc. Each data example, consisting of the input text and its task label, can be denoted as \(e=(e_{x},e_{y})\).

For each data example \(e\in\mathbb{P}\), a set of demonstration candidates \(\{p_{i}\}_{i=1}^{L}\) is constructed by randomly selecting \(L\) data examples from \(\mathbb{P}\). Then a specific model \(\mathrm{G}\), namely the scoring LM, is used to identify demonstration candidates as positive or negative. For each candidate \(p_{i}\in\{p_{i}\}_{i=1}^{L}\), \(e_{x}\) is concatenated with \(p_{i}\) to form the input of \(\mathrm{G}\). According to the task type that \(e\) belongs to, the score of \(p_{i}\) respecting to \(e\) is calculated as follows:

\[\operatorname*{score}\left(p_{i},e_{x}\right)=\operatorname*{metric}\left(e_{ y},e_{y}^{*}\right), \tag{2}\]

where \(e_{y}^{*}=P_{\mathrm{G}}\left(e_{y}^{*}|p_{i}\oplus e_{x}\right)\) is the prediction of the scoring LM and \(\operatorname*{metric}()\) is the specific task metric, such as F1 or Rouge for generation tasks and accuracy for classification tasks. The demonstration candidate with the highest score is identified as the positive, denoted as \(p^{+}\), while others are constructed into a negative examples set, denoted as \(\{p_{j}^{-}\}_{j=1}^{L-1}\).

Based on the training data, previous works utilize InfoNCE loss (van den Oord et al., 2018) to maximize the similarity score between data example \(e\) and its positive demonstrations, while minimize it for negative demonstrations:

\[L(e_{x},p^{+},p_{1}^{-},\ldots p_{L-1}^{-})\] \[= -\log\frac{e^{\text{sim}(e_{x},p^{+})}}{e^{\text{sim}(e_{x},p^{+} )}+\sum_{j=1}^{L-1}e^{\text{sim}(e_{x},p_{j}^{-})}}.\]

The similarity is calculated by:

\[\text{sim}(e_{x},p)=E_{q}(e_{x})^{\top}E_{p}(p) \tag{4}\]

where \(E_{q}\) encodes \(e_{x}\), \(E_{p}\) encodes the demonstration \(p\), and \(p\in\{p^{+}\}\cup\{p_{j}^{-}\}_{j=1}^{L-1}\).

### Inference Phase

Given a test query \(x_{test}\), the trained retriever is used to encode \(x_{test}\) and all demonstrations in \(\mathbb{P}\), then generate their embeddings. Previous works use Maximum Inner-Product or FAISS Johnson et al. (2021) to calculate the similarity between the embeddings of \(x_{test}\) and all demonstrations. As a result, top-\(K\) similar demonstrations are retrieved and then concatenated with the test query to form the input of inference models.

We consider that a high-quality demonstrations should not only exhibit similarity to the test query, but also align with the model bias. However, previous works greatly neglect this impact and fail to retrieve the most appropriate demonstrations for a specific inference LLM, resulting in a generalization degradation.

## 4 Methodology

Provided by a frozen pre-trained demonstration retriever \(\mathcal{R}\) trained on a demonstration pool \(\mathbb{P}\), and various inference models with their parameters, we aim to retrieve the most appropriate demonstrations from \(\mathbb{P}\) for a test query to improve the ICL performance without any retriever parameters updating. To achieve this, in this section, we first introduce a novel evaluation metric to evaluate the appropriateness of a demonstration to a specific inference LLM. Subsequently, we propose an inference framework named MDR, to re-rank demonstrations with our proposed evaluation metric.

### Evaluation Metric

Based on ICL, LLMs are able to perform various tasks conditioned on a few input-output demonstrations (Brown et al., 2020). Consequently, the quality of these demonstrations plays a crucial role in guiding LLMs to achieve the best generalization performance effectively (Liu et al., 2022). In this paper, we assume that a high-quality demonstration should not only exhibit similarity to the test query but also align with the preferences of the specific inference model, which is overlooked by previous methods. Given a demonstration \(p=(x,y)\), one general indicator capable of reflecting the model's 

preference for \(p\) is the loss \({\cal J}(y,y^{*})\) computed between the ground truth \(y\) and the prediction \(y^{*}\) generated by the inference model when provided with \(x\) as input. We adopt the per-token negative log-likelihood as the loss function.

\[{\cal J}(y,y^{*})=-\mathbb{E}_{y|x}[log~{}p_{\theta}(y|x)] \tag{5}\]

\[y^{*}=P_{\rm LM}(y^{*}|x). \tag{6}\]

While a smaller loss may suggest that the demonstration is better comprehended by the inference model, it is not conclusive evidence that it can serve as a better demonstration in ICL. As claimed in Zhuo et al. (2023) and Zheng and Saparov (2023), LLMs are vulnerable to perturbed prompts, such as minor typos, synonyms, or other common lexical perturbations. If a demonstration has minor typos, it is itself not robust to the model. Then, when it is fed to the model as part of the ICL prompt, it may cause LLMs to shift their focus towards the perturbed elements, thus producing wrong responses Zhu et al. (2023). On the other hand, given that Language Generation Models generate words based on the previous context, an ICL demonstration carries a blend of contextual information across layers Ferrando et al. (2023). If the demonstration is not robust to the model, its meaning space may be distorted by semantic perturbations. Such pollution potentially hampers the natural language understanding capabilities of the inference model Zheng and Saparov (2023), resulting in inferior model performance. Therefore, a model-preferred demonstration should not only exhibit a small loss but also demonstrate robustness to the perturbation. Due to the lack of labeled data describing the robustness of a demonstration to a specific inference model, the unsupervised evaluation of robustness can be extremely challenging.

To solve this, we are inspired by Zhao et al. (2019), who adopt the Fisher Information Matrix (FIM) of the input sample as a metric tensor to measure the robustness of deep learning models in adversarial attack task. Borrowing from this idea, we define a novel FIM-based matrix **H** to characterize the vulnerability of a demonstration to the perturbation in its feature space for a specific model \(\rm LM\). The matrix \(\textbf{H}\in\mathbb{R}^{n\times n}\) is defined as:

\[\textbf{H}=\nabla_{x}{\cal J}(y,y^{*})^{\top}\nabla_{x}{\cal J}(y,y^{*}), \tag{7}\]

where \(n\) is the hidden dimension of \(\rm LM\) and \(\nabla_{x}{\cal J}(y,y^{*})\) is the partial differential of \({\cal J}(y,y^{*})\) respecting to \(x\).

Similar to the conclusion of Zhao et al. (2019), which quantifies the vulnerability of deep learning models, we deduce that the maximum eigenvalue of **H**, denoted as \(\lambda_{max}\), reflects the vulnerability of demonstrations to \(\rm LM\). A smaller \(\lambda_{max}\) indicates a more robust demonstration with higher resilience to the perturbation. The expression for \(\lambda_{max}\) can be written as:

\[\lambda_{max}\!=\!\frac{1}{m}\sum_{i=1}^{m}(\nabla_{x_{i}}{\cal J}(y,y^{*}))^ {2}, \tag{8}\]

Figure 2: The inference framework of MDR. Given a test query, we first use the trained retriever to select top \(K^{\prime}\) demonstrations from the demonstration pool based on Maximum Inner-Product. Then we score and re-rank these demonstrations with our proposed evaluation metric given a specific inference LLM. Finally, the top \(K\) demonstrations are concatenated with the test query to guide the inference model in making predictions.

 

where \(m\) is the sequence length of \(x\) and \(x_{i}\) is the \(i\)th token.

We deduce four distinct scenarios based on the interplay between the loss and \(\lambda_{max}\). A demonstration with a large loss and a large \(\lambda_{max}\) indicates a misfit with the model and should be discarded. Conversely, a demonstration with a large loss but a small \(\lambda_{max}\) suggests a potential misalignment between the model and the demonstration, possibly within a region of flatter gradient, or when the model is on the verge of a state where the gradient vanishes at this specific demonstration. On the other hand, a demonstration exhibiting a small loss but a large \(\lambda_{max}\) indicates a well-fitted model for this demonstration. But this data may be outliers or atypical samples. The model can exhibit high sensitivity to such demonstration, leading to significant output variants even with a slight change in the input space. Therefore, demonstrations with both a small loss and a small \(\lambda_{max}\) are considered as appropriate for \(\mathrm{LM}\). Subsequently, we propose a tuning coefficient \(C\) to integrate these two considerations. Given a demonstration \(p=(x,y)\), we present the final expression of the evaluation metric, denoted as \(\mathrm{Metric}(p)\):

\[\mathrm{Metric}(p)=C\cdot\lambda_{max}+(1-C)\cdot\mathcal{J}(y,y^{*}). \tag{9}\]

In general, using the aforementioned derivation, a demonstration \(p=(x,y)\) can be easily evaluated by three steps. Firstly, input \(x\) to the inference model and calculate the loss \(J(y,y^{*})\) using \(y\) and the model prediction \(y^{*}\). Secondly, compute the maximum eigenvalue \(\lambda_{max}\) using Eq 8. Finally, \(\mathrm{Metric}(p)\) is derived using \(J(y,y^{*})\) and \(\lambda_{max}\) according to Eq 9. demonstrations with a small \(\mathrm{Metric}(p)\) are considered as appropriate for \(\mathrm{LM}\).

The significance of the introduced evaluation metric is that it remains invariant as long as the loss function keeps unchanged, ensuring stability and necessitating calculation only once. Moreover, the computation of \(\mathrm{Metric}(p)\) is simple and convenient without any human or machine annotation.

### Inference Framework

Based on the evaluation metric, we further propose a novel framework, MDR, to evaluate and retrieve the most appropriate demonstrations for a specific inference LLM. The overall inference framework is shown in Figure 2.

Given a test query \(x_{test}\) and a specific inference model \(\mathrm{LM}\), a high-quality demonstration should be appropriate to both \(x_{test}\) and \(\mathrm{LM}\). For the first consideration, in MDR retrieval stage, we use \(E_{q}\) to encode test query \(x_{test}\) and \(E_{p}\) to encode each demonstration in the demonstration pool \(\mathbb{P}\), then retrieve \(K^{\prime}\) demonstrations using Maximum InnerProduct similarity to construct a set of demonstration candidates \(\{p_{i}\}_{i=1}^{K^{\prime}}\):

\[\{p_{i}\}_{i=1}^{K^{\prime}}=top\text{-}K^{\prime}_{p_{i}\in\mathbb{P}}\sin( x_{test},p_{i}). \tag{10}\]

For the second consideration, we calculate \(\mathrm{Metric}(p_{i})\) for demonstration \(p_{i}=(x_{i},y_{i})\), \(i=(1,2,...,K^{\prime})\) using Eq 9. Then the \(K^{\prime}\) demonstrations are re-ranked according to \(\mathrm{Metric}(p)\) in ascending order and the top \(K\) (\(K\leq K^{\prime}\)) demonstrations are selected:

\[\{p_{j}\}_{j=1}^{K}=top\text{-}K_{p_{j}\in\{p_{i}\}_{i=1}^{K^{\prime}}}~{} \mathrm{Metric}(p_{j}). \tag{11}\]

As a result, demonstrations with relatively lower similarity to the test query may have higher ranks due to their better appropriateness with \(\mathrm{LM}\). We then concatenate the top \(K\) demonstrations with the test query to form the model input \(p_{K}\oplus...\oplus p_{1}\oplus x_{\text{test}}\).

## 5 Experiment Settings

### Implementation Details

In our experiment, for the demonstration retriever, we adopt the pre-trained one from UPRISE Cheng et al. (2023), which has been trained on a diverse set of tasks. The reason we choose UPRISE is that only UPRISE releases the checkpoint of the pre-trained retriever. Furthermore, CEIL Ye et al. (2023), UDR Li et al. (2023), and UPRISE are all built upon the framework introduced by EPR Rubin et al. (2022). Given that UPRISE has been tested in a more comprehensive scenario, involving both seen and unseen datasets across 5 models, we have selected it as our primary baseline method. The training data of UPRISE contains 30 datasets, resulting in a demonstration pool of \(224k\) demonstrations. At inference stage, we evaluate the performance of MDR under three inference LLMs with different scales, including GPT-Neo-2.7B, LAMA-7B, and Vicuna-13B. We set the number \(K^{\prime}\) of the demonstrations retrieved by similarity to be 20 and \(K\) of the concatenated demonstrations to be 3. For hyper-parameter \(C\), we set it to be 0.9 



### Datasets

We choose 8 seen datasets from the demonstration pool and 15 unseen datasets across 8 task types to test MDR. All datasets are transformed into natural language instructions using randomly selected instruction templates from FLAN Wei et al. (2022) or UDR Li et al. (2023). For each dataset, we report metrics on test set if available, falling back to the validation set otherwise. For detailed information, please refer to A.1.

### Baselines

We compare MDR with previous methods. **0-Shot**: The input to the model is the concatenation of the task instruction and the test query only. **Random**: We random select the demonstrations. **BM25 Robertson and Zaragoza (2009)**: For each test query, we use BM25 to retrieve the most similar demonstrations. **SBERT Reimers and Gurevych (2019)**: We use Sentence-Bert to retrieve the most similar demonstrations. **UPRISE**Cheng et al. (2023): UPRISE is a recently proposed representa

tive method for demonstration retrieval. For fair comparison, we test all baselines on each task under the same experimental settings.

## 6 Main Results

### Overall

Performance on Unseen DatasetsWe show the performance comparison on unseen datasets where the demonstrations and the test query belong to different data domains. As shown in Table 1, MDR outperforms baselines significantly on most tasks across different inference LLMs. Specially, compared with 0-Shot, the accuracy of MDR is improved by up to 44.13% on classification task (MR) using Vicuna-13B. F1 score is increased by up to 39.74 points on generation task (SQuADv2) using GPT-Neo-2.7B. Moreover, when compared with UPRISE, the performance of MDR is improved by up to 10.8% and 41.2% for classification task (WIC) and generation task (SamSum) using LLAMA-7B, respectively. The results verify the effectiveness of MDR. However, in text generation tasks, especially in QA, the performance of the proposed solution does not achieve as high a level compared to the baselines as in other evaluation scenarios. We speculate that the task's inherent nature may contribute to this difference, and we intend to investigate this aspect further in future research.

Performance on Seen DatasetsTo make a comprehensive comparison of MDR on seen datasets where the demonstrations and the test query belong to the same data domain, we select 8 representative datasets covering multiple task types. As shown in Table 2, MDR exhibits a considerable advantage on most datasets. Specifically, MDR has an absolute improvement of up to 34.98% when compared with random selection on SQuADv1 dataset using GPT-Neo-2.7B, and a relative improvement of up to 6.3% when compared with UPRISE on COPA dataset using LLAMA-7B. The results prove the effectiveness of MDR.

### Ablation Study

We perform an ablation study to assess the influence of various hyper-parameters, including: the number of top demonstrations retrieved by similarity, denoted by \(K^{\prime}\); the number of concatenated demonstrations, denoted by \(K\); and the hyperparameter \(C\).

#### 6.2.1 The impact of \(K\).

To assess the impact of the concatenated demonstrations number, denoted by \(K\), we conduct a comparative analysis involving random selection, BM25, SBERT, and UPRISE. This evaluation is performed on one classification task, i.e., WIC, and one generation task, i.e., SamSum, using LLAMA-7B while keeping other parameters unchanged. Specifically, \(K^{\prime}\) is set to be 20. The results are shown in Figure 3. As \(K\) increases, the accuracy rises consistently, indicating that more examples can help the model do better ICL. Moreover, MDR demonstrates substantial advantages over the baseline methods, irrespective of the value of \(K\).

#### 6.2.2 The impact of \(K^{\prime}\).

The quantity of top demonstrations, initially selected based on similarity and subsequently evaluated through MDR, significantly influences model performance. To test the impact of \(K^{\prime}\), we conduct experiments on four distinct datasets: WSC273 and RT for classification task, Trivia QA and SQuADv2 for generation task using GPT-Neo-2.7B while maintaining consistent parameter settings. Specifically, \(K\) is set to be 3. The results are shown in Figure 4. Notably, the highest accuracy is achieved at \(K^{\prime}=20\). Consistent with the previous analysis, increasing the value of \(K^{\prime}\) provides more opportunities for demonstrations that are less similar to the test query but are more compatible with the inference LLM. However, the performance of the model does not keep increasing as \(K\) increases. It is explainable since the demonstrations exhibiting low similarity to the test query may rank higher after re-ranking. Though these demonstrations may be more appropriate to the inference model, they are not appropriate to the test query. Therefore, it's essential to keep the balance between the similarity to the test query and the appropriateness to the model.

Figure 3: Ablation study on concatenated demonstrations number \(K\). The experiments are conducted on WIC and SamSum with LLAMA-7B.

 



#### 6.2.3 The joint impact of \(K\) and \(K^{\prime}\).

As \(K\) and \(K^{\prime}\) jointly impact the final experimental results, we evaluate the model's performance on the WIC dataset using GPT-Neo-2.7B to assess how increasing both \(K\) and \(K^{\prime}\) simultaneously on the same dataset affects the results. All other parameters remain unchanged. As observed in A.2 Table 7, increasing \(K\) from 4 to 10 only brings marginal improvement, aligning with findings in Li et al. (2023) and Chen et al. (2023). When \(K\leq 10\), the optimal \(K^{\prime}\) falls within the range of 15 to 20. Consequently, selecting \(K^{\prime}=20\) proves to be the optimal choice for the retrieval-based ICL.

Additionally, we perform experiments by setting \(K=K^{\prime}\) on WSC273 dataset using GPT-Neo-2.7B (i.e., MDR only re-ranks the order of demonstrations). As shown in A.2 Table 4, MDR consistently outperforms baselines as \(K^{\prime}\) increases, providing further validation of the effectiveness of MDR in evaluating and retrieving model-specific appropriate demonstrations for ICL.

#### 6.2.4 The impact of \(C\).

We apply parameter tuning method to explore the impact of \(C\) for different inference LLMs. Experiments are conducted with \(C\) from 0.0 to 1.0 on one classification dataset, i.e., WSC273, and one generation dataset, i.e., SQuADv2, while keeping other parameters unchanged. As shown in Fig 5, the best accuracy occurs when \(C=0.9\), indicating the importance of considering the appropriateness to the inference model when evaluating the quality of a demonstration. Additionally, models perform badly when \(C=0.0\) and \(C=1.0\), demonstrating that solely relying on the loss or the eigenvalue-based metric is inadequate to measure the appropri

Figure 4: Ablation study on the number \(K^{\prime}\) of demonstrations initially selected by similarity. The experiments are conducted on WSC273, Trivia QA, RT and SQuADv2 with GPT-Neo-2.7B.

\begin{table}
\begin{tabular}{c|l|r r|r r|r r|r r r} \hline \hline \multicolumn{1}{c|}{**Task Dataset**} & \multicolumn{1}{c|}{_NLI_} & \multicolumn{1}{c|}{_Comm Reasoning_} & \multicolumn{1}{c|}{_Para Detection_} & \multicolumn{1}{c|}{_Sentiment CLF_} & \multicolumn{1}{c}{_Reading Comprehension_} \\ \cline{3-11}  & \multicolumn{1}{c|}{RTE} & COPA & PIQA & MRPC & SST2 & YELP & OBQA & SQuADv1 \\
**Metric** & acc\(\uparrow\) & acc\(\uparrow\) & acc\(\uparrow\) & acc\(\uparrow\) & f1\(\uparrow\) & acc\(\uparrow\) & acc\(\uparrow\) & acc\(\uparrow\) & em\(\uparrow\) & f1\(\uparrow\) \\ \hline \multirow{8}{*}{**GPT-Neo (2.7B)**} & 0-Shot & 35.74 & 67.00 & 68.60 & 44.11 & 42.71 & 51.60 & 79.00 & 44.00 & 0.20 & 4.82 \\  & Random & 34.29 & 68.00 & 72.60 & 60.53 & 71.70 & 55.40 & 77.40 & 45.00 & 8.00 & 16.91 \\  & UPRISE & 62.81 & 71.00 & 72.80 & 73.28 & 80.28 & 78.60 & 93.20 & 49.20 & 38.20 & 49.51 \\  & **MDR** & **66.06** & **75.00** & **75.40** & **76.47** & **83.61** & **81.00** & **93.80** & **49.80** & **41.80** & **51.89** \\ \hline \multirow{8}{*}{**LLaMA (7B)**} & 0-Shot & 49.45 & 67.00 & 74.80 & 35.04 & 15.33 & 51.00 & 83.20 & 47.59 & 0.00 & 5.95 \\  & Random & 47.29 & 73.00 & 75.80 & 58.82 & 70.62 & 52.60 & 79.00 & 52.00 & 0.00 & 4.90 \\  & UPRISE & 62.81 & 79.00 & 77.60 & 72.54 & 79.41 & 83.60 & 96.60 & 56.80 & **0.20** & 7.21 \\  & **MDR** & **65.34** & **84.00** & **78.40** & **76.43** & **83.67** & **84.20** & **97.39** & **58.80** & 0.00 & **7.62** \\ \hline \multirow{8}{*}{**Vicuna (13B)**} & 0-Shot & 53.42 & 74.00 & 77.60 & 51.35 & 66.01 & 60.19 & 78.80 & 46.00 & 0.60 & 9.79 \\  & Random & 49.45 & 77.00 & 77.00 & 58.08 & 69.84 & 77.00 & 83.00 & 49.60 & 0.60 & 9.27 \\ \cline{1-1}  & UPRISE & 53.06 & 86.00 & 79.00 & 69.36 & 77.63 & 89.20 & 94.80 & 61.60 & 0.60 & 9.47 \\ \cline{1-1}  & **MDR** & **55.23** & **90.00** & **81.80** & **70.58** & **78.57** & **90.40** & **95.60** & **62.40** & **0.60** & **10.70** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Overall experimental results on seen datasets.



ateness of a demonstration for a specific model.

We discover an erratic pattern when \(C\) is small, indicating that loss takes a large proportion in our metric. Given that loss itself is an uncertainty factor, the re-rank process can be influenced. This influence causes the quality of demonstrations to depend more on similarity in the first step. In such cases, the performance across different datasets can suffer from fluctuations since various datasets retrieve demonstrations with different extents of similarity. On the other hand, when \(C\) increases to a certain extent (such as between 0.7 and 1.0), the accuracy on both datasets across all three models consistently demonstrates a stable improvement. This illustrates that enhancing the proportion of robustness can make the re-rank process less sensitive to similarity and become more effective. In such a scenario, loss, robustness, and similarity can collaboratively work well. To further validate the pattern of \(C\), we conduct experiments on two additional datasets using GPT-Neo-2.7B and LLaMA-7B. As shown in A.2 Table 8, the accuracy reaches a peak at 0.8 or 0.9, indicating that increasing \(C\) to a larger value reduces more unstable factors.

## 7 Related Work

In this section, we introduce previous works on exploring different strategies for selecting in-context demonstrations for LLMs. Two primary types of retrievers for ICL are commonly used. One is off-the-shelf retrievers, such as fine-tuned BERT Liu et al. (2022), BM25 Robertson and Zaragoza (2009) or SBERT Reimers and Gurevych (2019). The other approaches train a task-specific retriever using designed signals, such as question answering Das et al. (2021), code generation Poesia et al. (2022) and dialogue state tracking Hu et al. (2022). In particular, Rubin et al. (2022) introduce Efficient Prompt Retriever (EPR), which employs a LM to score demonstrations, and trains a dense retriever using contrastive learning. Additionally, Ye et al. (2023) propose CEIL to select diverse and helpful ICL demonstrations by using determinantal point processes. Recently, Li et al. (2023) propose UDR, a unified retriever designed for a wide range of tasks. Unlike UDR, which focuses on testing on seen tasks, Cheng et al. (2023) utilize a single retriever for cross-task and cross-model scenario on unseen tasks. However, previous studies neglect the biases of different inference LLMs. In this paper, we fully consider LLMs biases and strive to retrieve the most appropriate demonstrations for a specific inference model during inference stage.

## 8 Conclusion

In this paper, we introduce MDR, a simple yet effective method to evaluate and retrieve the model-specific appropriate demonstrations for ICL without re-training the retriever. MDR offers the feasibility of evaluating the appropriateness of a demonstration for LLMs without supervision. Moreover, given a trained retriever, MDR makes it more applicable to use it on various larger inference models.

In summary, MDR offers a promising insight for enhancing LLMs performance by leveraging their biases, presenting a notable perspective over existing retrieval-based methods and even prompt engineering.

### Ethical Considerations

We believe that this study contributes intellectual value to the dependable application of retrieval-based in-context learning in the field of NLP, with potential broader implications for tasks in other areas. It is noteworthy that there are no direct societal consequences, and all experiments are conducted on open datasets in this work.

### Limitations

Given the constraints of computing power, incorporating language models with larger scales poses a challenge for us. Despite selecting numerous and diverse tasks in this paper, the chosen set remains limited. Moreover, while results using automatic metrics provide a fair assessment of task performance, we aim to conduct a human evaluation in the near future.

#### Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 62201072, Grant U23B2001, Grant 62171057, Grant 62101064, Grant 62001054, and Grant 62071067; in part by the Ministry of Education and China Mobile Joint Fund under Grant MCM20200202 and Grant MCM20180101; in part by the BUPT-China Mobile Research Institute Joint Innovation Center 



## References

* Bentivogli et al. (2009) Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC. NIST.
* Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In AAAI, pp. 7432-7439. AAAI Press.
* Bojar et al. (2016) Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphaael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation.
* Bojar et al. (2014) Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Pratulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
* Chen et al. (2023) Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. How many demonstrations do you need for in-context learning? In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 11149-11159. Association for Computational linguistics.
* Cheng et al. (2023) Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, and Qi Zhang. 2023. UPRISE: universal prompt retrieval for improving zero-shot evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 12318-12337. Association for Computational Linguistics.
* Das et al. (2021) Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godoble, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazeros Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 9594-9611. Association for Computational Linguistics.
* Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In IWP@_IJCNLP_. Asian Federation of Natural Language Processing.
* Ferrando et al. (2023) Javier Ferrando, Gerard I. Gallego, Ioannis Tsiamas, and Marta R. Costa-jussa. 2023. Explaining how transformers use context to build predictions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 5486-5513. Association for Computational Linguistics.
* Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization.
* Hu et al. (2022) Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. Context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2627-2643. Association for Computational Linguistics.
* Huang et al. (2019) Liftu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
* Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Herve Jegou. 2021. Billion-scale similarity search with gpus. IEEE Trans. Big Data7(3):535-547.
* Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, page arXiv:1705.03551.
* Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP (1), pages 3045-3059. Association for Computational Linguistics.
* Levesque et al. (2012) Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In KR. AAAI Press.
* Li et al. (2023) Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for incontext learning. In ACL 2023, pages 4644-4668. Association for Computational Linguistics

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In _DecialO@ACL_, pages 100-114. Association for Computational Linguistics.
* Liu et al. (2021) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. _CoRR_, abs/2103.10385.
* Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.
* Mao et al. (2023) Rui Mao, Qian Liu, Kai He, Wei Li, and Erik Cambria. 2023. The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection. _IEEE Trans. Affect. Comput._, 14(3):1743-1753.
* Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In _EMNLP_, pages 2381-2391. Association for Computational Linguistics.
* Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 11048-11064. Association for Computational Linguistics.
* Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Pang and Lee (2005) Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In _Proceedings of the ACL_.
* Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 1267-1273. Association for Computational Linguistics.
* Poesia et al. (2022) Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In _ICLR 2022_. OpenReview.net.
* Rahman and Ng (2012) Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The winograd schema challenge. In _EMNLP-CoNLL_, pages 777-789. ACL.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In _EMNLP_, pages 2383-2392. The Association for Computational Linguistics.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. _arXiv e-prints_, page arXiv:1606.05250.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _EMNLP/IJCNLP (1)_, pages 3980-3990. Association for Computational Linguistics.
* Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. _Found. Trends Inf. Retr._, 3(4):333-389.
* Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_. AAAI.
* Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In _NAACL-HLT_, pages 2655-2671. Association for Computational Linguistics.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _EMNLP_, pages 1631-1642. ACL.
* van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _CoRR_, abs/1807.03748.
* Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In _ICLR_. OpenReview.net.
* Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. _CoRR_, abs/2201.11903.
* Ye et al. (2023) Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. _CoRR_, abs/2302.05698.
* Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _NIPS_, pages 649-657 

* [Chenxiao Zhao et al.2019] Chenxiao Zhao, P. Thomas Fletcher, Mixue Yu, Yaxin Peng, Guixu Zhang, and Chaomin Shen. 2019. The adversarial attack and detection under the fisher information metric. In _AAAI2019, IAAI 2019, EAAI 2019_, pages 5869-5876. AAAI Press.
* [Zheng and Saparov2023] Hongyi Zheng and Abulhair Saparov. 2023. Noisy exemplars make large language models more robust: A domain-agnostic behavioral analysis. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 4560-4568. Association for Computational Linguistics.
* [Zhu et al.2023] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, and Xing Xie. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. _CoRR_, abs/2306.04528.
* [Zhuo et al.2023] Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, and YuanFang Li. 2023. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023_, pages 1090-1102. Association for Computational Linguistics.

## Appendix A Appendix

### Datasets

We give detailed split statistics and evaluation metric for each test datasets. The datasets are collected following FLAN Wei et al. (2022) and UDR Li et al. (2023). we select 15 unseen datasets across 8 task types and various data domains from FLAN Wei et al. (2022) and UDR Li et al. (2023) to evaluate the performance of MDR , including:

* **Word Sense Disambiguation:** WIC Pilhwar and Camacho-Collados (2019).
* **Natural Language Inference:** WNLI Wang et al. (2019) and ANLI Nie et al. (2020).
* **Multi Choice:** WSC273 Levesque et al. (2012) and Cosmos QA Huang et al. (2019).
* **Sentiment Classification:** Amazon Li et al. (2023), MR Li et al. (2023) and Rotten tomatoes Pang and Lee (2005).
* **Question Answering:** SQuADv2 Rajpurkar et al. (2016) and Trivia QA Joshi et al. (2017).
* **Translation:** WMT14 Bojar et al. (2014) and WMT16 Bojar et al. (2016).
* **Text Summarization:** Samsum Gliwa et al. (2019).
* **Code Summarization:** Java Li et al. (2023) and Python Li et al. (2023).

For test efficiency, we limit the test set size of each dataset. Specifically, we randomly sample a 2000 subset for whose test set size is \(\geq 2000\) when testing on GPT-Neo-2.7B and 500 for whose test set size is \(\geq 500\) when testing on Vicuna-13B. We randomly sample a 2000 subset for classification tasks and a 500 subset for generation tasks when testing on LLaMA-7B. The detailed split statistics and evaluation metric are shown in Table 3.

Additionally, we select 8 representative datasets from the demonstration pool as the seen datasets, including:

* **Natural Language Inference:** RTE Bentivogli et al. (2009).
* **Commonsense Reasoning:** COPA Roemmele et al. (2011) and PIQA Bisk et al. (2020).
* **Paraphrase Detection:** MRPC Dolan and Brockett (2005).
* **Sentiment Classification:** SST2 Socher et al. (2013) and YELP Zhang et al. (2015).
* **Reading Comprehension:** OBQA Mihaylov et al. (2018) and SQuADv1 Rajpurkar et al. (2016).

Similarly, we limit the test set size of each dataset to be 500. The detailed split statistics and evaluation metric are shown in Table 5.

We set the maximum sequence generation length to be 100. The test process of GPT-Neo-2.7B can be implemented with only one 3090-24GB. The memory occupied by LLaMA-7B and Vicuna-13B is below 70GB, which means that all tests can be completed with a single A100-80GB.

### Ablation Study

We present the experimental results in this section.

### Case Study

To further demonstrate the effectiveness of MDR, we present the rankings of demonstrations retrieved using the Maximum Inner-Product method and the rankings after re-ordering by 

MDR (Table 6). The results demonstrate a significant influence of MDR on rankings. Firstly, demonstration No.144251, initially ranked second using similarity-based retrieval, shifts to the sixth position after re-ranking when applied to GPT-Neo-2.7B, and further drops beyond the seventh rank on LLaMA-7B. Secondly, the rankings of the top demonstrations exhibit notable variations. These results emphasize that demonstrations with high similarity to the test query do not necessarily align with the model bias. On the contrary, demonstrations with lower similarity still show the potential to enhance their rankings after re-ranking by MDR. Consequently, it becomes crucial to consider model biases during demonstrations retrieval. MDR furnishes a direct insight into this aspect of the field.

More specifically, we present the results for a task query tested on GPT-Neo-2.7B and LLaMA-7B using UPRISE and MDR, respectively. As shown in Table 9, for a given test query, UPRISE retrieves the same demonstrations for different inference LLMs, resulting in varying predictions. In contrast, when using MDR, as shown in Table 10, MDR retrieves different demonstrations for different inference models, all of which yield correct answers. This confirms the effectiveness of MDR.

 



\begin{table}
\begin{tabular}{c|c|c c c c c c c c c} \hline
**Inter** & **Retrieval** & **Prompt** & **Rank#1** & **Rank#2** & **Rank#3** & **Rank#4** & **Rank#5** & **Rank#6** & **Rank#7** \\
**Model** & **Phase** & **Attribute** & **Prompt** & **Prompt** & **Prompt** & **Prompt** & **Prompt** & **Prompt** & **Prompt** & **Prompt** & **Prompt** \\ \hline  & Similarity-based & id & 160353 & 144251 & 160717 & 159436 & 147132 & 147879 & 148630 \\  & Search & task & yelp & sst2 & sylp & spelp & sst2 & sst2 & sst2 \\ \hline   **GPT-Neo** & Eigenvalue-based & id & 145331 & 147894 & 146698 & 145985 & 147132 & 144251 & 148200 \\
**(2.7B)** & Re-rank & task & sst2 & sst2 & sst2 & sst2 & sst2 & sst2 & sst2 \\  & score & 3.2532 & 3.2605 & 3.2899 & 3.2945 & 3.3019 & 3.3052 & 3.3152 \\ \hline   **LLaMA** & Eigenvalue-based & id & 147844 & 145985 & 146698 & 151484 & 147894 & 149971 & 147132 \\
**(7B)** & Re-rank & task & sst2 & sst2 & sst2 & sst2 & sst2 & sst2 \\  & score & 0.3886 & 0.3929 & 0.398 & 0.3996 & 0.4005 & 0.4042 & 0.4069 \\ \hline \end{tabular}
\end{table}
Table 6: The first line is the top-7 demonstrations extracted by Maximum Inner-Product. The second and third lines represent the top-7 demonstrations after re-ranking with our proposed eigenvalue-based metric under GPT-Neo-2.7B and LLaMA-7B.

\begin{table}
\begin{tabular}{c|c|c c c c c c c c c} \hline
**Model** & **C** & **0.0** & **0.1** & **0.2** & **0.3** & **0.4** & **0.5** & **0.6** & **0.7** & **0.8** & **0.9** & **1.0** \\ \hline
**GPT-Neeo** & **WIC** & 51.00 & 50.52 & 50.30 & 50.78 & 51.07 & 51.30 & 51.40 & 51.97 & 52.23 & **53.29** & 53.12 \\
**(2.7B)** & **RT** & 70.39 & 70.36 & 70.31 & 70.22 & 70.31 & 70.44 & 70.57 & 70.61 & **71.10** & 71.01 & 70.97 \\ \hline
**LLaMA** & **WCI** & 51.72 & 51.75 & 51.94 & 51.78 & 51.82 & 51.88 & 51.94 & 52.02 & 52.31 & **52.66** & 52.62 \\
**(7B)** & **RT** & 75.32 & 76.00 & 75.80 & 75.40 & 75.80 & 76.50 & 76.40 & 76.60 & 77.10 & **77.20** & 76.90 \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 7: The joint ablation study of parameters \(K\) and \(K^{\prime}\) on WIC dataset using GPT-Neo-2.7B.

\begin{table}
\begin{tabular}{c|c|c c c c c c c c} \hline
**Cluster** & **Task** & \multicolumn{2}{c}{**Report**} & \multicolumn{2}{c}{**TestSize**} & \multirow{2}{*}{**Metric**} \\  & & & **Split** & **(GPT-Neo-2.7B)** & & & \\ \hline _NLLI_ & RTE & validation & 500 & 500 & acc \\ \hline _Commonsense_ & COPA & validation & 500 & 500 & acc \\ _Reasoning_ & PIQA & validation & 500 & 500 & acc \\ \hline _Paraphrase_ & MRPC & validation & 500 & 500 & acc \\ _Detection_ & SST2 & validation & 500 & 500 & acc \\ _Classification_ & YELP & test & 500 & 500 & acc \\ \hline _Reading_ & OBQA & test & 500 & 500 & acc \\ _Comprehension_ & & & & & & & & \\ \hline \end{tabular}
\end{table}
Table 5: The statistics, split and evaluation metrics of seen datasets.





**UPRISE**

Dataset: Amazon

Retrieved example numbers: 3

_Test Query_

Sentiment of the sentence: The first time I used this product it was fabulous. My hair turned out beautifully. When I tried another day to get the same results, the curling iron wouldn't heat up. It is

_Ground Truth_

terrible

_Retrieved demonstrations_

\({\bf Q}\): What is the sentiment of the following review? "Their Chocolate is divine; so creamy and smooooth! Whether you get a Pure Chocolate or a Blended Mocha you will be in Chocolate Heaven. I have even bought the chocolate powder for home and it never disappoints. It is so nice to shop the district with a mocha or pure chocolate in hand. I cannot walk by without stopping in :-)"

\({\bf A}\): Positive

\({\bf Q}\): "can only love the players it brings to the fore for the gifted but no-nonsense human beings they are and for the still-inestimable contribution they have made to our shared history." How would the sentiment of this sentence be perceived?

\({\bf A}\): Positive

\({\bf Q}\): Is the following review positive or negative? "Sara is so talented. She new exactly what I wanted and she gave me the best hair cut ever. I will use her from now on."

\({\bf A}\): Positive

_Prediction of GPT-Neo-2.7B:_

terrible

_Prediction of LLaMA-7B:_

great

Table 9: The results of the case study for a task query tested on GPT-Neo-2. 7B and LLaMA-7B when using UPRISE.





**MDR**

Dataset: Amazon

Retrieved example numbers: 3

_Test Query_

Sentiment of the sentence: The first time I used this product it was fabulous. My hair turned out beautifully. When I tried another day to get the same results, the curling iron wouldn't heat up. It is

_Ground Truth_

terrible

**GPT-Neo-2.7B**

_Retrieved demonstrations_

**Q**: "both heartbreaking and heartwarming ... just a simple fable done in an artless sytle , but it 's tremendously moving." How would the sentiment of this sentence be perceived?

**A**: Positive

**Q**: "filling nearly every minute ... with a lighthearted glow , some impudent snickers , and a glorious dose of humankind 's liberating ability." How would the sentiment of this sentence be perceived?

**A**: Positive

**Q**: "a journey spanning nearly three decades of bittersweet camaraderie and history , in which we feel that we truly know what makes holly and marina tick , and our hearts go out to them as both continue to negotiate their imperfect , love-hate relationship." How would the sentiment of this sentence be perceived?

**A**: Positive

_Prediction of GPT-Neo-2.7B:_

terrible

\begin{table}
\begin{tabular}{l} \hline \hline
**MDR** \\ \hline Dataset: Amazon \\ Retrieved example numbers: 3 \\
**_Test Query_** \\ Sentiment of the sentence: The first time I used this product it was fabulous. My hair turned out beautifully. When I tried another day to get the same results, the curling iron wouldn’t heat up. It is \\
**_Ground Truth_** \\ terrible \\ \hline
**GPT-Neo-2.7B** \\ \hline \hline \end{tabular}
\end{table}
Table 10: The results of the case study for a task query tested on GPT-Neo-2.7B and LLaMA–7B when using MDR.