Venue: NAACL 2025

# Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented

In-Context Learning for Long-Tail Knowledge in LLMs

Shuyang Yu\({}^{1}\), Runxue Bao\({}^{2}\) , Parminder Bhatia\({}^{2}\), Taha Kass-Hout\({}^{2}\), Jiayu Zhou\({}^{3}\), Cao Xiao\({}^{2}\)\({}^{\dagger}\)

Work was done during the internship at GE Healthcare.Correspondence to: Runxue Bao, Cao Xiao c{runxue.bao, caao.xiao}@gehealthcare.com>

###### Abstract

Large language models (LLMs) can learn vast amounts of knowledge from diverse domains during pre-training. However, long-tail knowledge from specialized domains is often scarce and underrepresented, rarely appearing in the models' memorization. Prior work has shown that in-context learning (ICL) with retriever augmentation can help LLMs better capture long-tail knowledge, reducing their reliance on pre-trained data. Despite these advances, we observe that LLM predictions for long-tail questions remain uncertain to variations in retrieved samples. To take advantage of the uncertainty in ICL for guiding LLM predictions toward correct answers on long-tail samples, we propose a reinforcement learning-based dynamic uncertainty ranking method for retrieval-augmented ICL that accounts for the varying impact of each retrieved sample on LLM predictions. Our approach prioritizes more informative and stable samples while demoting misleading ones, updating rankings based on the feedback from the LLM w.r.t. each retrieved sample. To enhance training efficiency and reduce query costs, we introduce a learnable dynamic ranking threshold, adjusted when the model encounters negative prediction shifts. Experimental results on various question-answering datasets from different domains show that our method outperforms the best baseline by \(2.76\%\), with a notable \(5.96\%\) boost in accuracy on long-tail questions that elude zero-shot inference. Our code is available at [https://github.com/Yu-shbuyan/uncertain_ranker](https://github.com/Yu-shbuyan/uncertain_ranker).

## 1 Introduction

Pretrained large language models Brown (2020); Touvron et al. (2023); Almazrouei et al. (2023) have achieved remarkable success across various natural language processing (NLP) tasks, such as summarization Zhang et al. (2019); Van Veen et al. (2024), question answering Jiang et al. (2021); Wang et al. (2024a), and code generation Li et al. (2023b); Wang et al. (2024b). These impressive results are largely due to their pre-training on vast, web-sourced datasets spanning multiple domains. However, these real-world datasets often follow a long-tail distribution Liu et al. (2019); Mallen et al. (2022); Dai et al. (2023); Sun et al. (2023a), where knowledge from less frequent domains is underrepresented. Consequently, certain domain-specific information may be rarely or even never included in the LLMs' memorization Kandpal et al. (2023). As a result, LLMs struggle to provide accurate responses to queries drawn from these long-tail distributions, since the pre-training process fails to capture this sparse information.

In-context learning (ICL) Brown (2020) is a few-shot learning method that queries LLMs by concatenating relevant samples with the test query, without updating the model's parameters. Kandpal et al. (2023) found that ICL, when combined with retriever augmentation, can reduce LLMs' reliance on pre-training knowledge by retrieving relevant examples related to long-tail queries during inference. Common retrieval methods used to select augmentation examples for ICL include random selection Wei et al. (2022); Wang et al. (2022), off-the-shelf retrievers (e.g., BM25 Robertson et al. (2009)), and fine-tuned retrizevers (e.g., PromptPG Lu et al. (2022)). However, prior works Zhao et al. (2021); Liu et al. (2021); Lu et al. (2021); Chen et al. (2023) have shown that ICL with different selection and ordering of the retrieved samples could lead to unstable predictions of LLMs. In our experiments, we observed a similar pattern: when utilizing existing methods to retrieve relevant samples for ICL, the model's predictions for long-tail questions--those not captured by zero-shot inference--exhibited particularly high uncertainty. In some cases, a subset of the retrieved samples led

to correct predictions, while the full set misled the model, even with the same retrieval method.

In this paper, to enhance the retrieval augmentation for long-tail samples regarding LLM's uncertainty, we propose a reinforcement learning-based dynamic uncertainty ranking method motivated by reinforcement learning's capacity to search for optimal retrieved samples based on the LLM's feedback (Lu et al., 2022). Specifically, our approach trains a retriever to prioritize informative and stable samples while down-ranking misleading ones, enhancing performance on both head and tail distributions. We build on the BERT-based retriever architecture (Devlin, 2018) with an appended linear layer. During the training of the retriever, only the linear layer is fine-tuned. Initially, BM25 (Robertson et al., 2009) is used for pre-selection, and the retriever is trained using policy gradients (Sutton et al., 1999), guided by feedback from the LLM for each retrieved sample. To improve efficiency, we introduce a learnable dynamic threshold as a budget controller for retrieval, selecting only samples with high-ranking scores above this threshold, which adjusts whenever the LLM experiences a negative prediction change, i.e., the prediction changes from true to false. To evaluate the proposed approach, we compared our method with the state-of-the-art methods across both multi-choice and open-ended question-answering (QA) datasets from different domains. The experimental results show that our method outperforms the best baseline by \(2.76\%\). Long-tail questions failed to be captured by a zero-shot inference benefit particularly from our proposed method. The accuracy of long-tail questions of our method surpasses previous methods with a large margin of up to \(5.96\%\).

We summarize our key contributions as follows:

* We investigate the limitations of existing retrieval-augmented ICL approaches for handling long-tail questions, highlighting how variations in retrieved samples contribute to prediction uncertainty.
* We propose a reinforcement learning-based dynamic uncertainty ranking method with a budget controller that considers the dynamic impact of each retrieved sample on the LLM's prediction, which selectively elevates informative retrieved samples and suppresses misleading ones with minimal query costs.
* Extensive experiments demonstrate that our method consistently outperforms the state-of-art method on multiple QA datasets from different domains, achieving nearly a \(6\%\) improvement in accuracy for long-tail questions.

## 2 Related Work

**In-context learning (ICL).** ICLR (Brown, 2020) queries the LLMs with a concatenation of related samples and the test query without parameter updating. To improve the quality of ICL, retrievers have been proposed to select related samples, which can

Figure 1: Training framework of the proposed method. After pre-selection using BM25 for each validation sample \(p_{i}\), we conduct from \(0\)-shot to \(k_{i}\)-shot inference and update retriever \(S_{\theta}\) according to the dynamic impacts of each sample on LLMs based on the reward from LLM. To reduce the query cost, we update the threshold \(\sigma\) when the LLM experiences a negative prediction change. The query time \(k_{i}\) is decided by retriever score \(S_{\theta}\) and threshold \(\sigma\).

 

be categorized into sparse retrievers (e.g. Robertson et al., 2009)) and dense retrievers (e.g. Liu et al., 2021)). To further improve the effectiveness of the off-the-shelf retrievers, strategies for fine-tuning retrievers on specific target domains have been proposed such as PromptPG Lu et al. (2022), UDR Li et al. (2023c), and LLM-R Wang et al. (2023), etc. Some works also adopt GPT to help retrieve and rerank samples by providing special prompts and related samples, such as Rerank Sun et al. (2023b), SuRe Kim et al. (2024), etc.

**Long-tail knowledge learning for ICL.**Kandpal et al. (2023) is the first to explore the influence of the long-tail distribution in pre-training data on LLM memorization. They find retrieval augmentation as a promising approach to significantly reduce the LLM's dependence on pre-training knowledge. Several subsequent works have built on this retrieval augmentation approach to address the long-tail problem in LLMs. For example, Dai et al. (2023) propose a retrieve-then-rerank framework leveraging knowledge distillation (KD) from the LLM to tackle long-tail QA. However, their method involves tuning the language model, which is computationally expensive and impractical for black-box LLMs such as GPT-4 Achiam et al. (2023). Another line of research focuses on augmenting the training set using GPT Saad-Falcon et al. (2023); Cloutier and Japkowicz (2023); Li et al. (2023a), followed by fine-tuning the retriever to enhance its performance. Nonetheless, determining which samples should be augmented remains challenging. Augmenting the training set based on seed sentences often introduces repetitive rather than diverse information, and incurs significant costs due to GPT queries. Therefore, in this paper, rather than augmenting the training set for fine-tuning the retriever, we aim to train an effective retriever capable of selecting the most informative samples to augment the test query during inference.

## 3 Problem Formulation

In this paper, we target in-context learning (ICL) for QA tasks including multiple-choice QA and open-ended QA from different domains. Suppose we have a training set \({\mathcal{T}}=\{(x_{i},y_{i})\}^{N}_{i=1}\) related to the query domain, where \(x\) is the question and \(y\) is the answer. Given a query problem \(p_{i}\) from a test set \({\mathcal{P}}\) and a \(K\)-shot inference budget, we will retrieve \(K\) related samples \({\cal E}_{i}=\{e^{k}_{i}=(x_{i},y_{i})|e^{k}_{i}\in{\mathcal{T}}\}^{K}_{k=1}\) and construct a prompt \(P({\cal E}_{i},p_{i})\) as input to feed into the LLM:

\[P({\cal E}_{i},p_{i})=\pi(e^{1}_{i})\oplus\cdots\oplus\pi(e^{ K}_{i})\oplus\pi(p_{i},\cdot), \tag{1}\]

where \(\pi\) is the template for each sample. The predicted answer from the LLM for question \(p_{i}\) is given by:

\[\hat{a}_{i}=\text{LLM}(P({\cal E}_{i},p_{i})). \tag{2}\]

## 4 Motivation: Uncertainty of In-context Learning

Due to the lack of knowledge of some specific domains during the pre-training stage, there exists long-tail knowledge that failed to be captured by the LLMs Kandpal et al. (2023). We define easy samples as queries that have been captured during the LLM's pre-training stage and are stored in its memorization. In contrast, hard samples refer to queries that the LLM failed to capture, which are more likely to represent long-tail data. We classify easy and hard samples using the zero-shot testing results \(\hat{a}_{i}=\text{LLM}_{\text{0-shot}}(p_{i})\):

\[{\mathcal{P}}_{\text{easy}} =\left\{(p_{i},a_{i})\in{\mathcal{P}}|\mathbbm{1}(\hat{a}_{i},a_ {i})=1\right\},\] \[{\mathcal{P}}_{\text{hard}} =\left\{(p_{i},a_{i})\in{\mathcal{P}}|\mathbbm{1}(\hat{a}_{i},a_ {i})=-1\right\}, \tag{3}\]

where the indicator function \(\mathbbm{1}(\cdot)\) returns \(1\) if the predicted answer \(\hat{a}_{i}\) aligns with the ground truth answer \(a_{i}\), otherwise it returns \(-1\). According to Kandpal et al. (2023), retrieval augmentation methods help alleviate the long-tail problem, as when a

Figure 3: Uncertain sample ratios.

Figure 2: Case study for uncertainty of LCL.

 

retriever succeeds in finding the most relevant samples from the training set \({\cal T}\), it reduces the LLM's needs to have a large amount of related knowledge in its memorization. However, our experiments revealed that the LLMs exhibit higher uncertainty when presented with hard samples, regardless of the retrieval augmentation applied. Fig.3 shows the uncertain sample ratios that experienced a prediction change on five datasets. Given a certain inference budget \(K=5\), \(21.84\%\) of queries experience a prediction change when we increase from \(0\)-shot to \(5\)-shot. Among these uncertain queries, \(87.18\%\) are hard samples and \(12.82\%\) samples are easy samples using BM25 retrieval Robertson et al. (2009). For hard samples, even a tiny variation in retrieved set \({\cal E}\) can mislead the LLM's prediction. One case study for hard sample queries from T-REx Elshar et al. (2018) is shown in Fig.2. In this case, LLM gives a correct answer with the first two informative samples in \({\cal E}\), effectively compensating for the LLM's long-tail knowledge. However, the answer gets wrong when a third sample is added to the prompt, which indicates the newly added knowledge is misleading. Other cases to show the uncertain prediction of LLM can be found in Fig.7 in Section6.4 and Table6 in Appendix.

Given the uncertainty of in-context learning, our goal is to improve the prediction accuracy of hard samples while maintaining the prediction stability on easy samples. During testing, we lack prior knowledge to determine whether a query falls into the easy or hard category. The primary challenge, therefore, is to prevent the inclusion of misleading information in the retrieved set \({\cal E}\), which could lead to incorrect predictions. Simultaneously, we must ensure that the retrieved samples are sufficiently informative to address long-tail knowledge gaps and guide the LLM toward the correct answer.

## 5 In-context Learning with Dynamic Uncertainty Ranking

In this section, we introduce a dynamic uncertainty ranking method built on a reinforcement learning-based retriever. This method adjusts the retriever by applying a dynamic threshold, lowering the rankings of misleading samples while elevating the rankings of informative and stable ones.

### Retrieved Sample Selection

The original training set \({\cal T}\) is randomly divided into a validation set \({\cal V}\), and a candidate pool \({\cal C}\), from which the retrieved sample set \({\cal E}\) is selected. Following Lu et al. (2022), the retriever structure is built upon BERT Devlin (2018) with a linear layer appended to the final pooling layer of the BERT model. During training, the BERT is frozen, and only the parameter \(\theta=({\bf W},{\bf b})\) of the linear layer is fine-tuned. Given a query \(p_{i}\) from the validation set \({\cal V}\) and a retrieved sample \(e_{i}\) from \({\cal C}\), the ranking score of the retriever is achieved by the hidden logical similarity shared among samples:

\[S_{\theta}(e_{i}|p_{i})=\frac{\exp[{\bf h}(e_{i})\cdot{\bf h}(p_{ i})]}{\sum_{e^{\prime}_{i}\in{\cal E}}\exp[{\bf h}(e^{\prime}_{i})\cdot{\bf h }(p_{i})]}, \tag{4}\]

where \({\bf h}(\cdot)={\bf W}(\text{BERT}(\cdot))+{\bf b}\) is the output of the linear layer.

To ensure the diversity and similarity of retrieved samples, and reduce the computational cost, we first adopt an off-the-shelf retriever BM25 Robertson et al. (2009) to pre-select a small candidate set \({\cal C}^{\prime}_{i}\) from the large candidate pool \({\cal C}\) following Rubin et al. (2021); Sun et al. (2023b); Kim et al. (2024).

Suppose the shot number is \(k\), by selecting samples with the Top-\(k\) highest ranking score using our retriever \(S_{\theta}\), we can achieve the retrieved sample set \({\cal E}_{i}\) for \(p_{i}\) from candidate pool \({\cal C}^{\prime}_{i}\) as follows:

\[{\cal E}_{i}=\{e^{k}_{i}\sim\mbox{Top-$k$$(S_{\theta}(e^{k}_{i}| p_{i}))|e^{k}_{i}\in{\cal C}^{\prime}_{i}\}. \tag{5}\]

The retriever selection process for testing is the same as the training, the only difference is the validation set \({\cal V}\) will be replaced with the test set \({\cal P}\).

### Retriever Training

Motivated by the exploration in Section4, to improve retrieval augmentation for both hard and easy samples, we introduce a dynamic ranking method that updates the retriever using feedback from the LLM, driven by its varying responses to each retrieved sample.

**Decide maximum shot number.** Before training, we first decide the maximum shot number for each validation sample \(p_{i}\in{\cal V}\). To achieve this, we define a maximum shot number budget \(K\) and a dynamic budget controller \(\sigma\) initialized as \(0\) for ranking scores \(S_{\theta}\). Only samples with ranking scores above the threshold \(\sigma\) will be selected to update the retriever. The maximum shot number \(k_{i}\) for \(p_{i}\) is:

\[k_{i}=\min(K,N^{\max}_{i}), \tag{6}\]

where \(N^{\max}_{i}=|\{e^{k}_{i}\sim S_{\theta}(e^{k}_{i}|p_{i})|e^{k}_{i}\in{\cal C }^{\prime}_{i},S_{\theta}(e^{k}_{i}|p_{i})>\sigma\}|\).

 



**Training process.** Given the maximum shot number \(k_{i}\), we then conduct inference for \(p_{i}\) from \(0\)-shot to \(k_{i}\)-shot to capture the effect of each retrieved sample on the LLM. The \(0\)-shot inference on \(p_{i}\) can be considered as a means of long-tail sample detection as defined in Eq.3. If the model's answer is incorrect, the sample is classified as a hard sample (i.e., long-tail sample), and the retrieved set should provide informative augmentation. Conversely, if the model produces the correct answer, the sample is classified as an easy sample, and the retrieved set should avoid introducing any misleading samples. We define the retrieved sample set for the \(j\)-shot inference as the top-\(j\) highest ranking score selected from candidate pool \(\mathcal{C}^{\prime}_{t}\):

\[\mathcal{E}^{j}_{i}=\{e^{k}_{i}\sim\text{Top-$j$}(S_{\theta}(e^{ k}_{i}|p_{i}))|e^{k}_{i}\in\mathcal{C}^{\prime}_{i}\}, \tag{7}\] \[\text{where}\quad j=\{0,1,\cdots,k_{i}\}.\]

The prediction from LLM based on \(\mathcal{E}^{j}_{i}\) and \(p_{i}\) is generated according to Eq.2 as \(\hat{a}^{j}_{i}=\text{LLM}(P(\mathcal{E}^{j}_{i},p_{i}))\). The retrieved sample's impact on the prediction is reflected by the reward function \(R(\hat{a}^{j}_{i},a_{i})=\mathbbm{1}(\hat{a}^{j}_{i},a_{i})\), where \(a_{i}\) is the ground truth answer for \(p_{i}\), \(\mathbbm{1}(\cdot)\) is the indicator function.

Our training goal is to maximize the expected reward w.r.t. the parameters of the retriever using the Policy Gradient method (Sutton et al., 1999). Since the expected reward cannot be computed in closed form, following Lu et al. (2022), we compute an unbiased estimation with Monte Carlo Sampling:

\[\mathbb{E}_{e_{i}\sim S_{\theta}(e_{i}|p_{i})}[(\hat{a}_{i},a_{ i})]\approx\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{k_{i}}R(\hat{a}^{j}_{i},a_{i}), \tag{8}\]

where \(N\) is the batch number yielded from \(\mathcal{V}\). Following the REINFORCE policy gradient (Williams, 1992), we update the retriever using:

\[\nabla\mathbb{E}_{e_{i}\sim S_{\theta}(e_{i}|p_{i})}[R(\hat{a}_{ i},a_{i})]\] \[= \mathbb{E}_{e_{i}\sim S_{\theta}(e_{i}|p_{i})}\nabla_{\theta}\log (S_{\theta}(e_{i}|p_{i}))R(\hat{a}_{i},a_{i})\] \[\approx \frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{k_{i}}\nabla_{\theta}\log(S _{\theta}(e^{j}_{i}|p_{i}))R(\hat{a}^{j}_{i},a_{i}), \tag{9}\]

where \(e^{j}_{i}=\mathcal{E}^{j}_{i}-\mathcal{E}^{j-1}_{i}\) is the difference between the retrieved sets for \(j\)-shot and \((j-1)\)-shot. This approach incorporates the dynamic influence of each retrieved sample on the LLM, providing a better handling of uncertainty in ICL. Specifically, retrieved samples that yield correct predictions (\(R(\cdot)=1\)) are treated as informative and contribute to augmenting long-tail knowledge, thus receiving a higher ranking. Conversely, retrieved samples that lead to incorrect predictions (\(R(\cdot)=-1\)) are considered misleading and are ranked lower.

```
1:Input: Retriever \(S_{\theta}\), training set \(\mathcal{T}\), maximum shot number \(K\).
2:Output: Trained retriever \(S_{\theta}\) .
3:Randomly split \(\mathcal{T}\) into \(\mathcal{V}\) and \(\mathcal{C}\).
4:Initialize \(\theta\leftarrow\theta_{0}\), threshold \(\sigma\gets 0\).
5:for\(\mathcal{V}_{\text{batch}}\in\mathcal{V}\)do
6: Initialize batch loss \(L\gets 0\).
7:fore each validation sample \(p_{i}\in\mathcal{V}_{\text{batch}}\)do
8: Pre-select \(\mathcal{C}^{\prime}_{i}\) from \(\mathcal{C}\) using BM25 for \(p_{i}\).
9: Calculate the maximum shot number \(k_{i}\) based on \(\sigma\) using Eq.6.
10:for\(j=0,1,\cdots,k_{i}\)do
11: Get the retrieved set \(\mathcal{E}^{j}_{i}\) using Eq.7.
12: Get prediction \(\hat{a}^{j}_{i}=\text{LLM}(P(\mathcal{E}^{j}_{i},p_{i}))\).
13: Get reward \(R(\hat{a}^{j}_{i},a_{i})=\mathbbm{1}(\hat{a}^{j}_{i},a_{i})\).
14:\(L\gets L-R(\hat{a}^{j}_{i},a_{i})\cdot\log(S_{\theta}(e^{j}_{i}|p_{i}))\).
15:if\(R(\hat{a}^{j}_{i},a_{i})=-1,R(\hat{a}^{j-1}_{i},a_{i})=1\)then
16: Update \(\sigma\) using Eq.10.
17:endif
18:endfor
19:endfor
20: Optimize \(L\) w.r.t. \(\theta\) using Eq.9.
21:endfor
```

**Algorithm 1** ICL with dynamic uncertainty ranking

**Update budget controller \(\sigma\).** In order to increase training efficiency and reduce the cost of querying the LLM, we also update the threshold \(\sigma\) that served as a budget controller at the turning point for prediction change to decrease the inference times while maintaining the effect of our training strategy. Specifically, we focus on a special case: when the LLM experiences a prediction change from true to false , i.e., \(R(\hat{a}^{j-1}_{i},a_{i})=1\) and \(R(\hat{a}^{j}_{i},a_{i})=-1\). In this case, the first \((j-1)\)-th samples have a positive impact on the inference of LLM, while the \(j\)-th sample has a negative impact. Thus, we update the threshold \(\sigma\) as the maximum value of the ranking score for unselected samples in \(\mathcal{E}^{k_{i}}_{i}\) for the \((j-1)\)-shot round as follows:

\[\sigma=\max(S_{\theta}(e^{k}_{i}|p_{i})),\quad e^{k}_{i}\in\mathcal{ E}^{k_{i}}_{i}-\mathcal{E}^{j-1}_{i}. \tag{10}\]

Since we only select samples with ranking scores larger than \(\sigma\) as shown in Eq.6, the retrieved sam

ples that serve as a good compensation for long-tail knowledge will be ranked higher, and be used for updating the retriever more frequently. Note that updating \(\sigma\) will not wipe out the updating of misleading samples, as the turning point for prediction change is different for each validation sample. Without affecting our original training strategy, we improve the efficiency and deduct the querying cost. Our algorithm is summarized in Algorithm1.

## 6 Experiments

In this section, we first introduce the experiment setup and then show the effectiveness of our method through various empirical results.

### Experimental Setup

**Datasets:** We conduct the experiments on QA datasets from different domains, including three multi-choice datasets: biomedical dataset _Pubmedqa_[30], speech detection dataset _ethos-national_[25], climate change dataset _eval-climate_[49], and two open-ended QA dataset: _T-REx_[78] and NaturalQuestions (_NatQA_)[26]. More datasets details can be found in AppendixA.1.

**Baselines:** We compare our method with six baselines, including _0-shot_ inference and five few-shot retrieval augmentation methods. The retrieval augmentation methods are as follows: 1) _Random sampling_: selecting ICL samples from the candidate set, a widely adopted practice in many ICL studies [24; 25]; 2) _BM25_[21]: an off-the-shelf sparse retriever; 3) SuRe [31]: first use GPT to summarize the retrieved passages from BM25 for multiple answer candidates, then determines the most plausible answer by evaluating and ranking the generated summaries; 4) Rerank [32]: use GPT to rerank samples retrieved by BM25; 5) _PromptPG_[30]: a BERT-based dense retriever trained using reinforcement learning based on the feedback from GPT.

**Evaluation:** For multi-choice QA, we use accuracy for evaluation. For open-ended QA, we use Normalized Exact Match (NEM), which evaluates whether the normalized string output by the inference LLM is identical to the reference string.

**Implementation:** The LLM used in our experiment is GPT-4 [1]. Due to the limited data size in _tweet_eval-stance_climate_, the training set is split into \(50\) candidate samples and \(150\) validation samples. For the other datasets, we use \(1000\) samples in the candidate pool and \(200\) samples in the validation set. All methods share the same train-test split. The number of pre-selected samples in \({\cal C}^{\prime}\) is set to \(20\) by default for both the training and testing stages. For the few-shot case, the shot number is set to \(5\), unless otherwise specified. During the training of our method, the maximum shot number budget \(K\) is also set to \(5\). The batch size is set to \(20\). Experiments for all test datasets are repeated \(3\) times with different seeds, and the average accuracy is reported in the results.

Figure 4: Accuracy on easy and hard samples for proposed method and baselines.

 



### Main Results

Table 1 presents the mean and standard deviation (std) of accuracy for our proposed method and the baselines across five QA datasets. Our approach outperformed all baselines across tasks, with an average improvement of ranging from to over the best baseline. The trained retriever PromptPG gives the most uncertain prediction with a std of . Although our method is based on PromptPG, by giving informative and stable samples higher ranks, we not only improve the overall accuracy but also decrease std to comparable to 0-shot inference.

We further investigate the accuracy of easy and hard samples in Fig. 4. As illustrated in Eq. (3), the easy/hard sample classification is decided by the 0-shot inference results, and the hard samples can be considered as long-tail questions of GPT-4. First, we observe a similar pattern to Kandpal et al. (2023) that retrieval augmentation greatly improves the accuracy of long-tail samples. This could come from various aspects of augmented samples--such as label space, input text distribution, and sequence format--that collectively improve final predictions Min et al. (2022). Compared with 0-shot inference, even random sampling improves accuracy on hard samples from to . However, retrieval augmentation is highly dependent on the quality of the retrieval set. By retrieving the most similar samples, BM25 achieves an accuracy of . Rerank further improves the accuracy to . Our method includes the most informative samples based on the sample-wise feedback from LLM, and improves the accuracy on hard samples to , which surpasses the best baseline with a large average margin of ranging from to , while maintaining the accuracy on easy samples.

### Ablation Studies

**Effects of different components.** We verify the effectiveness of two components of our proposed method: uncertainty rank and pre-selection in Table 2. We first compared the uncertainty rank (UR) strategy with another trained retriever PromptPG which shared the same retriever architecture as ours. We improve the accuracy by and for two different datasets. PromptPG adjusts the ranking of candidate samples based on the feedback on the entire retrieved set for the validation samples, while UR raises the ranks for informative and stable samples and lowers the ranks for misleading samples based on the sample-wise feedback from LLMs. UR avoids the condition when misleading samples are included and negatively changes the answer from true to false. In this way, UR greatly enhances the retrieved sample set for augmentation.

The second component pre-selection (PS) improves the results of both PromptPG and UR by selecting more diverse and similar related samples in the candidate set . Then the second step retrieval can select samples from a smaller candidate pool of higher quality. By combining these two components together, we can achieve an overall improvement of and for two different datasets. The improvement on ethos-national is more significant than Pubmedqa because the predicted answer on ethos-national is more uncertain given different combinations of retrieved samples.

**Effects of different number of shots.** We show the effects of different shot numbers for two datasets in Fig. 5 where o

outperforms other baselines. For NatQA, the accuracy of random sampling and PromptPG retrieval does not monotonically increase with shot number due to low-quality, misleading samples, which can degrade performance. In contrast, our method prioritizes high-quality samples, and as the number of shots increases, the advantages of our algorithm become more pronounced, resulting in improved accuracy.

**Effects of different number of pre-selection samples.** In Fig. 6, we investigate how the number of pre-seletion samples impacts our algorithm. For both datasets, the accuracy first increases and then decreases. If too few samples are selected, the candidate pool \({\cal C}^{\prime}\) for our reinforcement learning -based ranking stage lacks diversity, limiting the policy gradient strategy's action space. Consequently, the learned retriever struggles to find the most informative samples. If the number is too large, \({\cal C}^{\prime}\) includes many irrelevant samples, making it difficult for the policy gradient strategy to learn an optimal solution in the large search space Lu et al. (2022). This can lead the retriever to capture irrelevant or misleading information.

### Case Study

To intuitively show the effectiveness of our proposed method on hard samples, we show one case on Pubmedqa by comparing the retrieved samples of PromptPG retriever and our retriever in Fig. 7. According to this case, the two retrieved sets even have three overlap samples (marked as the same color), but the prediction is completely different. PromptPG gives a wrong prediction answer, while our method delivers the right answer. This result verifies that GPT-4 gives uncertain predictions on long-tail samples. Since \(0\)-shot inference gives a wrong prediction answer on this query question, the informative augmented information can be contained in the retrieved set of our method (see right column), while for PromptPG, misleading information can be contained in the two samples that do not intersect with our retriever set (see left column), which shifts the predicted answer from true to false. Compared with PromptPG, our retriever ranks the three overlapped samples higher and gives two more informative samples. With the combination effect of these two, our method gives the correct prediction. More cases on hard samples from other datasets can be found in Table 7 in the appendix.

### Efficiency Analysis

**Query cost.** We set threshold \(\sigma\) as the budget controller to reduce the cost of the querying GPT-4. Since the query cost depends on token length, we compare the query costs of our method and PromptPG (both trained based on GPT-4) in Fig. 8a. Specifically, we calculate the total number of shots included in each query during training for each batch within one epoch for both methods. The blue dash line shows the total shot number of PromptPG for all datasets, since the batch size is \(20\), and the shot number is fixed at \(5\), the total shot number is fixed at \(100\) for each batch. According to the results, only batch \(0\) of our method surpasses PromptPG with a total shot count of \(300\). For subsequent batches, as the threshold \(\sigma\) is adjusted based on changes in the LLM's pre

Figure 7: Case study for retrieved samples of hard samples.

 

dictions, the query shot count drops significantly, resulting in the total shot count consistently being lower than that of PromptPG. Aggregating the shot numbers across \(10\) batches, our method achieves only \(33.8\%\), \(65.2\%\), and \(35.3\%\) of the shot count of PromptPG on Pubmedqa, ethos-national, and NatQA, respectively as shown in Fig. 7(b). Thus, in conjunction with the accuracy comparison presented in Table 1, our approach not only enhances query accuracy but also reduces the overall query cost.

**Convergence speed.** We empirically demonstrate the convergence speed by showing training loss curves in Fig. 9. According to the results, the training loss quickly converges to a small value close to \(0\) within \(15\) batchs, which verify the high computational efficiency of our method.

### Transferability Analysis

We investigate the transferability of our retriever in Table 3. We use our retriever trained on dataset _ethos-national_, and evaluate its cross-domain effectiveness across the rest of the four datasets. Although the cross-domain results are still slightly inferior to the in-domain results, the performance gap is minimal, averaging only \(0.98\%\). Furthermore, the cross-domain results outperform the best baseline. These findings indicate that our trained ranking strategy is transferable to other datasets, providing a cost-effective alternative to retraining.

## 7 Conclusion

In this paper, to improve the uncertain prediction of LLMs on long-tail knowledge, we propose a reinforcement learning-based dynamic uncertainty ranking method for retrieval-augmented ICL with a budget controller. Specifically, it considers the dynamic impact of each retrieved sample based on the LLM's feedback. Our ranking system system raises the ranks of more informative and stable samples and lower the ranks of misleading samples efficiently. Evaluations of various QA datasets from different domains show that our proposed method outperformed all the baselines, and especially improve the LLM's prediction on the long-tail questions.

## 8 Limitations

There are several limitations of our work.

First, our method do not consider the effect of different orders within the retrieved set and rank the retrieved samples according to their ranking scores. Future works can be extended based on our work by considering different inner order within the retrieved set and their effect on the prediction results.

Second, although our experimental results show that our method greatly improves the prediction accuracy on long-tail samples, our method cannot handle query cases with no related knowledge either in the pre-training set or candidate pool.

Third, our method focused on QA tasks using LLM. For future work, our method can be extended to other tasks such as summarization, translation, and recommendation as follows. Since our method is to train a reranker based on the reward signal from LLM, to adapt to other tasks, we can modify the evaluation score that is used to determine the reward. If the accuracy of the LLM's predicted answer is unavailable, alternative metrics such as BLEU and ROUGE can be used to assess the consistency between the prediction and the ground truth. A threshold can then be set for these scores, where values exceeding the threshold yield a positive reward, while lower values result in a negative reward.

Figure 8: Efficiency analysis.

\begin{table}
\begin{tabular}{c c c c c}  & Pubmedqa & eval-climate & NatQA & T-REx & Avg \\ \hline Best baseline & 78.93 & 83.22 & 55.00 & 62.13 & 69.82 \\ Ours: cross-domain & 79.60 & 83.33 & 57.20 & 64.50 & 71.16 \\ Ours: in-domain & 80.60 & 85.37 & 57.60 & 65.00 & 72.14 \\ \hline \end{tabular}
\end{table}
Table 3: Transferability of our method.

Figure 9: Training loss w.r.t. batch.

 



## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsmi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The Falcon series of open language models. _arXiv preprint arXiv:2311.16867_.
* Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, and Luis Espinosa-Anke. 2020. Tweet-eval: Unified benchmark and comparative evaluation for tweet classification. _arXiv preprint arXiv:2010.12421_.
* Brown (2020) Tom B Brown. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Chen et al. (2023) Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. How many demonstrations do you need for in-context learning? _arXiv preprint arXiv:2303.08119_.
* Cloutier and Japkowicz (2023) Nicolas Antonio Cloutier and Nathalie Japkowicz. 2023. Fine-tuned generative llm oversampling can improve performance over traditional techniques on multiclass imbalanced text classification. In _2023 IEEE International Conference on Big Data (BigData)_, pages 5181-5186. IEEE.
* Dai et al. (2023) Yi Dai, Hao Lang, Yinhe Zheng, Fei Huang, and Yongbin Li. 2023. Long-tailed question answering in an open world. _arXiv preprint arXiv:2305.06557_.
* Devlin (2018) Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Elsahar et al. (2018) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lacforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
* Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. _Transactions of the Association for Computational Linguistics_, 9:962-977.
* Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. _arXiv preprint arXiv:1909.06146_.
* Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In _International Conference on Machine Learning_, pages 15696-15707. PMLR.
* Kim et al. (2024) Jaeyhyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Summarizing retrievals using answer candidates for open-domain qa of llvms. _arXiv preprint arXiv:2404.13081_.
* Kwiatkowski et al. (2023) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:453-466.
* Li et al. (2023a) Huihan Li, Yuting Ning, Zeyi Liao, Siyuan Wang, Xiang Lorraine Li, Ximing Lu, Faeze Brahman, Wenting Zhao, Yejin Choi, and Xiang Ren. 2023a. In search of the long-tail: Systematic generation of longitudinal knowledge via logical rule guided search. _arXiv preprint arXiv:2311.07237_.
* Li et al. (2023b) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Koectkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023b. Starcoder: may the source be with you! _arXiv preprint arXiv:2305.06161_.
* Li et al. (2024) Xiang Li, Haoran Tang, Siyu Chen, Ziwei Wang, Ryan Chen, and Marcin Abram. 2024. Why does in-context learning fail sometimes? evaluating in-context learning on open and closed questions. _arXiv preprint arXiv:2407.02028_.
* Li et al. (2023c) Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023c. Unified demonstration retriever for context learning. _arXiv preprint arXiv:2305.04320_.
* Liu et al. (2021) Jiaching Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? _arXiv preprint arXiv:2101.06804_.
* Liu et al. (2019) Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. 2019. Large-scale long-tailed recognition in an open world. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 2537-2546.
* Lu et al. (2022) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2022. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. _arXiv preprint arXiv:2209.14610_.
* Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. _arXiv preprint arXiv:2104.08786_.
* Mallen et al. (2022) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. _arXiv preprint arXiv:2212.10511_

* Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanesh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11048-11064.
* Mollas et al. (2022) Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2022. Ethnos: a multi-label hate speech detection dataset. _Complex & Intelligent Systems_, 8(6):4663-4678.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends@ in Information Retrieval_, 3(4):333-389.
* Rubin et al. (2021) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. _arXiv preprint arXiv:2112.08633_.
* Saad-Falcon et al. (2023) Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avivip Sil, Md Arafat Sultan, and Christopher Potts. 2023. Udapdr: unsupervised domain adaptation via lm prompting and distillation of rerankers. _arXiv preprint arXiv:2303.00807_.
* Sun et al. (2023a) Kai Sun, Yifan Ethan Xu, Hanwen Zhao, Yue Liu, and Xin Luna Dong. 2023a. Head-to-tail: How knowledgeable are large language models (lllm). _AKA will llms replace knowledge graphs_.
* Sun et al. (2023b) Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023b. Is chatgpt good at search? investigating large language models as re-ranking agents. _arXiv preprint arXiv:2304.09542_.
* Sutton et al. (1999) Richard S Sutton, Andrew G Barto, et al. 1999. Reinforcement learning. _Journal of Cognitive Neuroscience_, 11(1):126-134.
* Tourron et al. (2023) Hugo Tourron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Van Veen et al. (2024) Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoti Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerova, et al. 2024. Adapted large language models can outperform medical experts in clinical text summarization. _Nature medicine_, 30(4):1134-1142.
* Wang et al. (2024a) Fali Wang, Runxue Bao, Suhang Wang, Wenchao Yu, Yanchi Liu, Wei Cheng, and Haifeng Chen. 2024a. Infuserkl: Enhancing large language models with knowledge graphs via infuser-guided knowledge integration. In _Findings of the Association for Computational Linguistics: EMNLP 2024_, pages 3675-3688.
* Wang et al. (2023) Liang Wang, Nan Yang, and Furu Wei. 2023. Learning to retrieve in-context examples for large language models. _arXiv preprint arXiv:2307.07164_.
* Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. _arXiv preprint arXiv:2203.11171_.
* Wang et al. (2024b) Zhenpeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, and Yanfu Zhang. 2024b. Unlocking memorization in large language models with dynamic soft prompting. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 9782-9796.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8:229-256.
* Zhang et al. (2019) Haoyu Zhang, Jianjun Xu, and Ji Wang. 2019. Pretraining-based natural language generation for text summarization. _arXiv preprint arXiv:1902.09243_.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _International conference on machine learning_, pages 12697-12706. PMLR.

 

Appendix

### Experiment Details

**Dataset Details.** In this paper, we evaluate across five QA datasets from different domains including multi-choice QA and open-ended QA. The detailed statistics of these datasets and the prompt format we used are shown in Table 4 and Table 5. We conduct the train-test split for the last four datasets following Li et al. (2024). We randomly sample \(1000\) samples from the training dataset if the training set size exceeds \(1000\) to simulate the scenario where only a limited number of samples can be collected.

### Extended Experimental Results

**More case study on the uncertainty of ICL.** According to Table 6 on the healthcare dataset Pubmedqa, LLM can achieve correct prediction with the first two retrieved samples but gives a wrong prediction when the third sample is added to the prompt, which indicates that the third sample is misleading.

**More case study on hard samples.**Table 7 shows another case for retrieved samples of hard samples on T-REx. According to the results, the query question asks about the instance of a subject, while the prompt retriever retrieved samples about the questions related to the locations, which mislead the final prediction. For our retriever, all the retrieved samples are related to the questions related to the instance of the subject, and provide informative augmentations for the inference.

 



\begin{table}
\begin{tabular}{c|l l} \hline \multicolumn{1}{c|}{Query question} & \multicolumn{2}{c}{Outlaw [SEP] instance of.} \\ \cline{2-3} Retriever & PromptPG retriever & Our retriever \\ \hline  & Question: Hingani Dam [SEP] country, Answer: The answer is India. & Question: Schleich [SEP] instance of. Answer: The answer is municipality of Germany. \\ \cline{2-3} Retrieval samples & Question: Maryland State Archives [SEP] applies to jurisdiction. Answer: The answer is Maryland. & Question: Chevy-sous-le-Bignon [SEP] instance of. Answer: The answer is commune of France. \\ \cline{2-3} Retrieval samples & Question: Silvia Panguana [SEP] country of citizensh. Answer: The answer is Mozambique. & Question: The Listed Hotel [SEP] instance of. Answer: The answer is hotel. \\ \cline{2-3}  & Question: New Palvavyi [SEP] located in the administrative territorial entity. Answer: The answer is Andra Pradesh. & Question: Westona [SEP] instance of. Answer: The answer is railway station. \\ \cline{2-3}  & Question: The ’59 Sound [SEP] country of origin. Answer: The answer is United States of America. & Question: Secu [SEP] instance of. Answer: The answer is commune of Romania. \\ \hline Prediction & film. (✗) & wooden roller coaster. (✓) \\ \hline \end{tabular}
\end{table}
Table 6: Extended case study for the uncertainty of ICL on Pubmedqa.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Dataset & Type & Domain & Training & Test & Prompt format \\ \hline Pubmedqa & Multi-choice & Healthcare & 1000 & 500 & SQO-A \\ ethos-national & Multi-choice & Speech detection & 476 & 298 & QO-A \\ eval-climate & Multi-choice & climate change & 288 & 180 & QO-A \\ T-REx & Open-ended & Wikipedia & 20128 & 5032 & Q-A \\ NatQA & Open-ended & Wikipedia & 11476 & 2869 & Q-A \\ \hline \hline \end{tabular}
\end{table}
Table 4: The statistics of the datasets used in this paper.

\begin{table}
\begin{tabular}{c p{113.8pt} p{113.8pt}} \hline Notation & Retrieval sample format & Query sample format \\ \hline Q-A & Question: <question> Answer: The answer is ¡answer? & Question: <question> Answer: \\ \hline QO-A & Question: <question> Options: (A) <option A> (B) < (C) <options C>... Answer: The answer is ¡answer? & Question: <question> Options: (A) <option B> (C) <options C>... Answer: \\ \hline SQO-A & Statement: <context> Question: <questions Options: (A) caption A> (B) option B> (C) <options C>... Answer: The answer is ¡answer? & Question: <question> Options: (A) <option A> (B) <option B> (C) options C>... Answer: \\ \hline \hline \end{tabular}
\end{table}
Table 4: The statistics of the datasets used in this paper.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt}|p{113.8pt}} \hline \hline \multicolumn{1}{c|}{**Query:** Statement: Lymphedema may be identified by.. Question: Can a practicing surgeon detect early lymphedema reliably?} \\ \hline \multicolumn{1}{c|}{Retrieved samples} & Prediction \\ \hline Retrieved sample 1 & Statement: Minority patients with cancer experience.... Question: Can patient coaching reduce racial/ethnic disparities in cancer pain control? Answer: Yes. & Maybe (✓) \\ \hline Retrieved sample 1 + sample 2 & Statement: Minority patients with cancer experience.... Question: Can patient coaching reduce racial/ethnic disparities in cancer pain control?Answer: Yes. Statement: The potential effects of binge drinking during pregnancy. & Maybe (✓) \\ \hline Retrieved sample 1 + sample 2 + sample 3 & Statement: Minority patients with cancer experience.... Question: Can patient coaching reduce racial/ethnic disparities in cancer pain control?Answer: Yes. Statement: The potential effects of binge drinking during pregnancy.. Question: Does binge drinking during early pregnancy increase the risk of psychomotor deficits?Answer: No. & No (✓) \\ \hline Retrieved sample 1 + sample 2 + sample 3 & Statement: Minority patients with cancer experience.... Question: Can patient coaching reduce racial/ethnic disparities in cancer pain control?Answer: Yes. Statement: The potential effects of binge drinking during pregnancy.. Question: Does binge drinking during early pregnancy increase the risk of psychomotor deficits?Answer: No. Statement: Despite the advantages from using aromatase inhibitors.... Question: Do adjuvant aromatase inhibitors increase the cardiovascular risk in postmenopausal women with early breast cancer?Answer: Yes. & No (✓) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Prompt Format notations.