RARe: Retrieval Augmented Retrieval with In-Context Examples

## Abstract

Abstract We investigate whether in-context examples, widely used in decoder-only language models (LLMs), can improve embedding model performance in retrieval tasks. Unlike in LLMs, naively prepending in-context examples (query-document pairs) to the target query at inference time does not work out of the box. We introduce a simple approach to enable retrievers to use in-context examples. Our approach, RARe , finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query. This can be applied to adapt various base architectures (i.e., decoder-only language models, retriever models) and consistently achieves performance gains of up to +2.72% nDCG across various open-domain retrieval datasets (BeIR, RAR-b). In particular, we find RARe exhibits stronger out-of-domain generalization compared to models using queries without in-context examples, similar to what is seen for in-context learning in LLMs. We further provide analysis on the design choices of in-context example augmentation and lay the foundation for future work in this space.

## 1 Introduction

In-context learning (ICL) (Brown et al., ) has emerged as a powerful paradigm enabling diverse applications without parameter updates in large language models (LLMs). By conditioning on input-output examples that demonstrate a specific task, LLMs can generate predictions while maintaining fixed parameters. While in-context learning has been extensively studied for LLMs (Xu et al., , Min et al., , Dong et al., ), its potential for retriever models remains unexplored.

We study how in-context examples can be effectively leveraged to enhance performance in retriever models. Unlike in decoder-only LLMs where in-context examples expand model capacity at generation time, in-context examples may primarily provide task-relevant information rather than increasing model capacity. Specifically, we study injecting in-context examples to build a dense retriever model (Karpukhin et al., ) which embeds queries and documents into a shared representational space for efficient search over a large corpus. Text retrieval is a core component of many natural language processing (NLP) tasks, serving as a key component for retrieval-augmented language language models (Lewis et al., ). State-of-the-art retriever models started to leverage decoder-only models as a backbone (Wang et al., , BehnamGhader et al., , Muennighoff et al., , Meng et al., , Lee et al., ), further motivating our study of applying in-context examples.

We begin by naively prepending in-context examples to the target query and provide it to existing retriever models (BehnamGhader et al., , Wang et al., , Meng et al., ), observing that this leads to significant performance drop. We propose a new approach to construct retrieval models that can leverage in-context examples, which we name as RARe: Retrieval Augmented Retrieval with In-Context Examples. Our approach modifies the query format of retrieval systems by providing in-context examples whose query is semantically similar to the target query. Then, we apply standard continued fine-tuning with contrastive loss. We conduct a comprehensive evaluation of new query format across various experimental settings, initializing from both decoder-only checkpoints and pre-trained retriever model checkpoints. We demonstrate that RARe outperforms baseline models across multiple tasks, achieving up to +1.41% nDCG@10 on standard retrieval benchmarks (Thakur et al., ) and demonstrating even larger gains (+2.72%) on reasoning-oriented retrieval tasks (Xiao et al., ).

Our contributions can be summarized as follows:

- •
We introduce RARe, an approach that adapts pre-trained checkpoints to use in-context examples for retrieval.
- •
We demonstrate that this recipe can improve the performance of various base architectures, including decoder-only models and existing retriever models.
- •
We provide detailed analyses on how the quality, quantity, and selection of in-context examples affect performance, contextualizing the sources of our experimental gains.

All our code and model checkpoints are publicly released.(^1^11Code is available at: [https://github.com/atutej/RARe](https://github.com/atutej/RARe))

Figure: Figure 1: Overview – Prior work augments a task-specific instruction to a given query as input to the Retriever. In RARe, we further leverage a set of in-context exemplars that contain pairs of queries and relevant documents. These in-context examples are augmented with the original query as input to the retriever along with the instruction.
Refer to caption: x1.png

## 2 Setup & Existing Approaches

##### Standard Retrieval Setup

We consider a standard dense retriever  (Karpukhin et al., ), where input queries $q$ and documents $d$ are encoded with an embedder $E(\cdot)$ into a fixed-dimensional embedding. The embedder $E(\cdot)$ is trained on a training set $\mathcal{D}$ which consists of multiple retrieval tasks $\{\mathcal{D}_{1},\mathcal{D}_{2},\cdots,\mathcal{D}_{T}\}$, where each task contains training examples of the form $(q,d^{+},d^{-})$ (Wang et al., , BehnamGhader et al., ). Here, $q$ is the input query, $d^{+}$ is a positive (relevant) document, and $d^{-}$ is a hard-negative (irrelevant) document, which allows for a contrastive-loss based training.

The evaluation task $\mathcal{D_{\mathrm{test}}}$ consists of a corpus of documents $C$, as well as test pairs $(q,R^{+})$, where $R^{+}=\{d^{+}_{1},d^{+}_{2},...,d^{+}_{m}\}\subset C$ is a set of relevant document(s) for the query (Thakur et al., ). The aim is to retrieve these relevant documents $R^{+}$ from the corpus $C$ using the embedder $E(\cdot)$. Specifically, an index $C_{\bm{e}}$ of the corpus with document embeddings $E(d),\forall d\in C$ is created. Then, the embedding $E(q)$ of a test query $q$ is used to retrieve the documents $d$ whose embedding $E(d)$ is closest to $E(q)$, typically with the cosine (cos) similarity function.

##### Existing Methods

Current architectures (Asai et al., , BehnamGhader et al., ) prepend task-specific instruction $t_{i}$, $i\in[1,2,\cdots,T]$ to the query to contextualize the task:

$$ $q^{\text{inst}}=\text{Instruct: }\{t_{i}\};\text{ Query: }\{q\},\quad q\in \mathcal{D}_{i}$ (1) $$

Then, the embedder $E(\cdot)$ is trained with a standard contrastive loss (Izacard et al., , Karpukhin et al., ), incorporating $q^{\text{inst}}$, and $d^{+},d^{-}\in\mathcal{D}_{i}$, along in-batch negatives $n\in{\mathbb{N}}$, where ${\mathbb{N}}$ represents the set of in-batch negatives,

$$ $\displaystyle{\bm{e}}_{q^{\text{inst}}}=E(q^{\text{inst}});\quad{\bm{e}}_{d^{+ }}=E(d^{+});\quad{\bm{e}}_{d^{-}}=E(d^{-});\quad{\bm{e}}_{n}=E(n)$ (2) $\displaystyle\mathcal{L}=-\log\frac{\exp(\text{cos}({\bm{e}}_{q^{\text{inst}}} ,{\bm{e}}_{d^{+}}))}{\exp(\text{cos}({\bm{e}}_{q^{\text{inst}}},{\bm{e}}_{d^{+ }}))+\exp(\text{cos}({\bm{e}}_{q^{\text{inst}}},{\bm{e}}_{d^{-}}))+\sum\limits _{n\in{\mathbb{N}}}\exp(\text{cos}({\bm{e}}_{q^{\text{inst}}},{\bm{e}}_{n}))}$ (3) $$

During evaluation on $\mathcal{D_{\mathrm{test}}}$, each test query is augmented with task-specific instruction $t_{\text{test}}$.

Figure: Algorithm 1 RARe - Training

## 3 Our Method – RARe

RARe consists of two main components – (a) We enhance the query representation by incorporating in-context examples, which provide additional query-specific guidance to the model, (b) We fine-tune $E(\cdot)$ on $\mathcal{D}$ to learn to leverage these in-context examples.

Given a query $q$, we use BM25 (Robertson & Zaragoza, ), a sparse retrieval technique that ranks documents based on keyword matching, and find $k$ closest queries $q_{j}$ from $\mathcal{D}_{i}\in\mathcal{D}$ to obtain in-context examples $\mathcal{D}^{\text{ic}}_{i}=\{(q^{\text{ic}}_{1},d^{\text{ic}+}_{1}),(q^{\text
{ic}}_{2},d^{\text{ic}+}_{2}),\cdots,(q^{\text{ic}}_{k},d^{\text{ic}+}_{k})\}$. As shown in Figure 1, we then augment these examples to the original query $q^{\text{inst}}$ to obtain the final query $q^{\text{inst+ic}}$,

$$ $q^{\text{inst+ic}}=\text{Instruct: }\{t_{i}\};\text{ Query: }\{q^{\text{ic}}_{ 1}\};\text{ Document: }\{d^{\text{ic}+}_{1}\}\,\cdots;\text{ Query: }\{q\}$ (4) $$

We then train embedder $E(\cdot)$ with the same loss as Equation 3, but with $q^{\text{inst+ic}}$ instead of $q^{\text{inst}}$,

$$ $\displaystyle\mathcal{L}_{\text{{RARe}}}=-\log\frac{\exp(\text{cos}({\bm{e}}_{ q^{\text{inst+ic}}},{\bm{e}}_{d^{+}}))}{\exp(\text{cos}({\bm{e}}_{q^{\text{ inst+ic}}},{\bm{e}}_{d^{+}}))+\exp(\text{cos}({\bm{e}}_{q^{\text{inst+ic}}},{ \bm{e}}_{d^{-}}))+\sum\limits_{n\in{\mathbb{N}}}\exp(\text{cos}({\bm{e}}_{q^{ \text{inst+ic}}},{\bm{e}}_{n}))}$ (5) $$

Algorithm 1 presents our training procedure in detail. At inference time, we similarly perform a search to find nearest in-context examples to form an augmented query. Algorithm 2 in the Appendix provides an overview of the inference procedure.

## 4 Experimental Setup

### 4.1 Fine-Tuning

##### Base Models

We explore two training setups: fine-tuning decoder-only models for retrieval, and fine-tuning existing retriever models. For the first setup, we train the Llama-3 family of models, following the training methodology outlined by Ma et al. (), Weller et al. (). For the second setup, we use two high-performing publicly available embedding models that were trained with task-specific instructions: LLM2Vec-Llama-3-8b-Supervised (BehnamGhader et al., ) and E5-Mistral-Instruct (Wang et al., ). We chose these two models because, unlike some other strong performers (Meng et al., , de Souza P. Moreira et al., ), they were not trained on most of the datasets used in our downstream benchmarks. The LLM2Vec-Llama-3-8b-Supervised model is initially trained using an unsupervised text reconstruction objective and then fine-tuned with supervised contrastive learning on a public subset of the E5 dataset, which incorporates various supervised training datasets (Gao et al., , Nguyen et al., , Kwiatkowski et al., ). In contrast, E5-Mistral-Instruct undergoes further training on synthetic data that is not publicly available. These models are chosen to assess the impact of additional supervised training on an existing retriever model versus training a generative model for retrieval from scratch.

##### Training Data

For fine-tuning existing retriever models, we follow prior work (BehnamGhader et al., ) and train on a publicly available portion of E5 dataset (Springer et al., , Wang et al., ), which contains MS-MARCO (Nguyen et al., ) NLI (Gao et al., ), ELI5 (Fan et al., ), FEVER (Thorne et al., ), HotpotQA (Yang et al., ), NQ (Kwiatkowski et al., ), SQuAD (Rajpurkar et al., ), Quora Duplication Questions (DataCanary et al., ). For fine-tuning decoder-only models from scratch, we use the MS-MARCO (Nguyen et al., ) passage ranking dataset and train without a task-specific instruction prefix, following Ma et al. ().

##### Constructing In-Context Examples

During training, we provide each training example with five in-context examples from the dataset that it belongs to ($k$=5). Specifically, the set of examples $\mathcal{D}^{\text{ic}}_{i}$ for each task is drawn from the training set $\mathcal{D}_{i}$, $q\notin\mathcal{D}^{\text{ic}}_{i}$.

### 4.2 Evaluation

##### Datasets

We evaluate on the widely used BeIR retrieval benchmark (Thakur et al., ). For ablative experiments, we follow prior work and focus on low-resource datasets (Wang et al., ) that potentially benefit more from few-shot examples. Since the BeIR benchmark contains a few datasets whose training sets are in the E5 dataset mixutre, we categorize them as in-domain and out-of-domain i.e. not seen during training. See Table 7 in the Appendix for a list of in-domain and out-of-domain datasets from BeIR.
We also evaluate on a subset of the RAR-b (Xiao et al., ) benchmark, which requires complex reasoning for retrievers. Specifically, we evaluate on HellaSwag (Zellers et al., ), PIQA (Bisk et al., ), ARC-C (Clark et al., ), TempReason-L1 (Tan et al., ), WinoGrande (Sakaguchi et al., ), $\alpha$-NLI (Bhagavatula et al., ), SiQA (Sap et al., ), and Quail (Rogers et al., ). Unlike BeIR, some RAR-b queries are composed of sentences with (multiple) indicators (e.g., Start:, End:). Each dataset is associated with a task-specific instruction, following prior work (Muennighoff et al., , Wang et al., , BehnamGhader et al., ). We provide additional preprocessing details in Appendix A.

##### Constructing In-Context Examples

We construct $\mathcal{D}^{\text{ic}}_{\text{test}}$ from the training/development set of each datasets. For datasets on BeIR that do not have either of these, we use a synthetically generated collection of document-query pairs (GenQ) from Thakur et al. (). For all experiments, we use $k$=5 in-context examples.

##### Metrics

We use standard metrics for retrieval benchmarks. Following Thakur et al. (), we report nDCG@10, which measures the ranking quality of the top 10 retrieved documents, taking into account both the relevance and position of each retrieved document (Wang et al., ).

## 5 Results

Figure: Figure 2: Inference-only modification does not work Performance after adding in-context examples to the query without updating model parameters. We see that embedding models are not able to leverage in-context examples out of the box, as opposed to decoder-only models.
Refer to caption: x2.png

We evaluate in-context example augmented queries in three settings. First, we evaluate the performance after inference-only modification, where we take existing pre-trained retrievers and simply provide in-context examples at inference time (Section 5). Second, we evaluate training retriever with in-context examples from an LLM (decoder-only) backbone (Section 5.1). Third, we compare training retriever models with in-context examples from a pre-trained retriever (Section 5.2).

##### Inference-only Modification

Figure 2 illustrates the impact of incorporating in-context examples at inference time. Here, we simply modify the query format with retrieved in-context examples (i.e. $q^{\text{inst+ic}}$, Eq. 4) at inference time and compare its performance with the query format that does not have retrieved in-context examples (i.e. $q^{\text{inst}}$, Eq. 1). We evaluate the performance on three retriever models: SFR-Embedding-2-R (Meng et al., ), LLM2Vec-Llama-3-8B-Supervised (BehnamGhader et al., ), and E5-Mistral-7B-Instruct (Wang et al., ). Unlike in autoregressive LLMs, these embedding models generally exhibit decreased performance when in-context examples are added, with LLM2Vec-Llama-3-8B-Supervised showing the largest drops in performance, except on one dataset (SciFact), where 2 out of 3 models show marginal gains over providing only instructions. Our experiments, which include adding more in-context examples and using nearest-neighbor examples, extend the findings of Muennighoff et al. (), where in-context examples led to decrease in performance on the GritLM models.

**Table 1: Training from decoder-only (LLM) checkpoint. Performance is measured by nDCG@10. RARe shows up to +2.72% absolute gain on average over Promptriever, demonstrating that starting from an existing embedding model is not a requirement. We provide a breakdown of In-Domain (ID) and Out-of-Domain (OOD) performance.**
| Method | Base model | Training Data | ID | OOD | Average |  |
| --- | --- | --- | --- | --- | --- | --- |
| MS-MARCO | BeIR | RAR-b |  |  |  |  |
| RepLLaMA | Llama-2 | MS-MARCO | 42.00 | 53.69 | 20.23 | 38.64 |
| RepLLaMA | Llama-3 | MS-MARCO | 43.56 | 53.99 | 18.50 | 38.68 |
| RARe | Llama-3 | MS-MARCO | 44.77 | 55.87 | 22.34 | 40.99 |
| RepLLaMA | Llama-3.1-Instruct | MS-MARCO | 43.67 | 54.34 | 19.20 | 39.07 |
| Promtpriever | Llama-3.1-Instruct | MS-MARCO + Synthetic | 42.70 | 56.10 | 20.95 | 39.94 |
| RARe | Llama-3.1-Instruct | MS-MARCO | 42.93 | 56.05 | 23.67 | 40.88 |

### 5.1 Training from LLM Checkpoints

Next, we present the results of applying our approach when training from LLM checkpoint. This might preserve in-context learning capacity of the LLM, which can be lost during standard IR training, which compresses query and passage into a fixed dimensional vector. We experiment with three LLM checkpoints (Llama-2 (Touvron et al., ), Llama-3 (Dubey et al., ), Llama-3.1-Instruct) to enable comparison with prior work Ma et al. (), Weller et al. ().

##### Comparison Systems

We compare training with our in-context example augmented query with two baselines. The first baseline is vanilla query (Eq. 1), which was explored in RepLLaMA (Ma et al., ). The second baseline is Promptriever (Weller et al., ) which augments query-specific instructions using a synthetically generated training set from MS-MARCO. In all these systems, the task-specific instruction is a null string (Ma et al., ) as we train on a single task (MS-MARCO).

##### Results

Table 1 presents the performance on downstream benchmarks when training from LLM checkpoints. Comparing within the same base LLM checkpoint, our apporach outperforms both baselines (RepLLaMA and Promptriever). Our performance is competitive to that of Promptriever (Weller et al., ), without incorporating synthetic data during training. Specifically, RARe achieves an absolute gain of +2.7% over Promptriever on the RAR-b benchmark.

### 5.2 Training from Retriever Checkpoints

**Table 2: Training from retriever checkpoint. Performance (nDCG@10) on BeIR (Thakur et al., ) and RAR-b (Xiao et al., ) benchmarks when fine-tuning retriever model on E5 dataset. We report a breakdown of performance on In-Domain (ID) and Out-of-Domain (OOD) tasks on BeIR. We consider all RAR-b tasks as OOD.**
| Method | LLM2Vec-Llama-3-8b-Supervised | E5-Mistral-Instruct |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BeIR | RAR-b | BeIR | RAR-b |  |  |  |  |  |
| ID | OOD | All | ID | OOD | All |  |  |  |
| Base | 71.31 | 49.28 | 56.63 | 21.55 | 71.95 | 49.33 | 56.87 | 22.17 |
| Instruct | 70.46 | 47.79 | 55.35 | 23.44 | 72.91 | 48.98 | 56.96 | 24.12 |
| RARe | 71.67 | 49.30 | 56.76 | 23.10 | 72.98 | 50.93 | 58.28 | 25.79 |

Lastly, we continue training retriever models – LLM2Vec-Llama-3-8B-Supervised (BehnamGhader et al., ), E5-Mistral-Instruct (Wang et al., ) on a training set where queries are augmented with in-context examples. As these initial checkpoints have already been trained on the training dataset, the extent that retrievers adapt to new query format can be limited.

##### Comparison Systems

We first report the initial retriever performance (Base) without any modification. Then, we compare continued fine-tuning with the task-specific instruction query format (Eq. 1) which only prepends the task specific instruction (Instruct, $q^{\text{inst}}$) to our in-context example augmented query format (Eq. 4).

##### Results

Table 2 reports experimental results in this setting. Overall, both fine-tuning approaches provides gains over the base checkpoints. Comparing two settings, Instruct ($q^{\text{inst}}$) vs. RARe ($q^{\text{inst+ic}}$), our method achieves notable improvement with E5-Mistral-Instruct base model (1.95% over Instruct on out-of-domain tasks, and 1.32% overall). Our method performs similar to Instruct ($q^{\text{inst}}$) setting when trained with the LLM2Vec base model. It is hard to attribute why experimental results varies based on the base retriever checkpoint, but we note the following differences between the two models. LLM2Vec-Llama-3-8b-Supervised is the only model in our experiments where further fine-tuning with only instructions led to a decrease in in-domain performance. E5-Mistral-Instruct employs causal attention with last token pooling, and trains on a proprietary synthetic dataset, LLM2Vec-Llama-3-8b-Supervised uses bidirectional attention with mean pooling, training only on the E5 public portion. The effectiveness of learning with in-context examples may depend on the model architecture or data setting, and further investigation can be explored in future work.

Figure: Figure 3: Retrieved vs. Random In-context Examples. Change in performance ($\Delta$nDCG@10) on E5-Mistral-Instruct with RARe ($q^{\text{inst+ic}}$) from the baseline setting ($q^{\text{inst}}$ both during training and evaluation time). Using retrieved examples during training and inference enhance model performance in most benchmark datasets.
Refer to caption: x3.png

## 6 Discussions and Analysis

### 6.1 Choice of In-context Examples

##### Retrieved (Similar) vs. Random In-Context Examples

In Figure 3, we study the impact of retrieving the nearest neighbor query-document pairs as examples against randomly chosen examples during training and evaluation. We observe that using retrieved examples during both training and evaluation (Retrieved, Retrieved) consistently outperforms other configurations across most datasets. (Random, Retrieved) shows second best overall performance, and generally outperforms (Random, Random), suggesting retrieved examples during evaluation is advantageous even when trained with randomly paired in-context examples. Our findings align with prior work in in-context learning – that the incorporation of semantically similar examples is beneficial (Agrawal et al., , Rubin et al., ).

Figure: Figure 4: Change in performance ($\Delta$nDCG@10) from the base model (E5-Mistral-Instruct) for varying similarity between the closest in-context example query and target query (Score@Top-1).
Refer to caption: x4.png

##### Does Having Semantically Relevant In-Context Example Help?

For some test examples, augmented in-context examples are very relevant, and for others, much less so. In this section, we group the evaluation examples by the maximum similarity of in-context query and the test query measured by an off-the-shelf sentence embedding model (Score@Top-1).(^2^22[https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)) and plot the performances for each group. Figure 4 presents the performance of our system (RARe) and baseline (Instruct). On NFCorpus and SciFact datasets, we observe that when the closest in-context example has a high similarity with the target query, RARe demonstrates over 10% gains compared to the base model. On the other hand, fine-tuning exhibits relatively lower performance gains with increasing similarity thresholds. For ArguAna, and FiQA2018, RARe’s gains with increasing Score@Top-1 are less pronounced, but generally matches the performance of the base model.

##### How Many In-Context Examples Are Sufficient?

We analyze the performance of RARe when varying the number of in-context examples provided during training and inference. Table 3 shows that increasing the number of in-context examples generally enhances performance. On ArguAna, we observe that 0 examples are optimal, which is likely due to the mismatch in the lengths of the queries used as in-context examples(^3^33[https://huggingface.co/datasets/BeIR/arguana-generated-queries](https://huggingface.co/datasets/BeIR/arguana-generated-queries)) (which are significantly shorter) versus the actual test queries. However, the impact is not uniformly positive across all datasets, suggesting that the optimal number of in-context examples may be dataset-dependent. We observe similar trends when we fix the number of in-context examples to five during training and vary the number of examples provided during inference, which are provided in Table 12 in the Appendix.

**Table 3: Impact of the number of in-context examples ($k$) during training and evaluation. All results are on E5-Mistral-Instruct. In general, performance increases when increasing the number of examples, and the optimal number of examples depends on the task.**
| $k$ | Arguana | CQADupStack | FiQA2018 | NFCorpus | SciFact | Touche2020 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Instruct (0) | 61.19 | 44.82 | 57.39 | 40.99 | 77.28 | 29.35 | 51.84 |
| 1 | 60.47 | 46.76 | 56.07 | 40.67 | 81.47 | 29.78 | 52.54 |
| 3 | 62.98 | 47.12 | 57.08 | 40.77 | 83.71 | 27.12 | 53.13 |
| 5 | 60.87 | 48.46 | 57.31 | 42.28 | 84.79 | 28.70 | 53.74 |
| 10 | 58.85 | 48.92 | 57.03 | 42.24 | 87.61 | 28.29 | 53.82 |

**Table 4: In-Context Format Comparing variants of in-context example format on E5-Mistral-Instruct. Instruct refers to the baseline which does not use any in-context examples.**
| Fine-Tuning | IC Eval Setting | ArguAna | CQA | FiQA2018 | NFCorpus | SciFact | Touche2020 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Instruct | - | 61.19 | 44.82 | 57.39 | 40.99 | 77.28 | 29.35 | 51.83 |
| RARe | Queries-Only | 58.88 | 46.66 | 54.44 | 41.42 | 78.84 | 28.09 | 51.39 |
| Doc-Only | 57.54 | 48.28 | 56.02 | 41.62 | 79.80 | 29.01 | 52.05 |  |
| Shuffle-NC | 60.17 | 45.78 | 54.25 | 41.17 | 80.70 | 29.18 | 51.88 |  |
| Shuffle-C | 58.97 | 47.97 | 55.98 | 41.78 | 80.51 | 28.97 | 52.36 |  |
| Regular | 60.87 | 48.46 | 57.31 | 42.28 | 84.79 | 28.70 | 53.74 |  |

##### Ablating Content and Format of In-context Examples

One can view in-context examples as a form of query expansion (Lv & Zhai, , Wang et al., ), providing useful keywords to improve the performance. In Table 4, we analyze the impact of various formats of in-context examples. All models are trained with the same format that they are evaluated on. Query-Only and Doc-Only contain only queries and documents of in-context examples, respectively. For Shuffle-C, we randomly shuffle the mapping between $q$ and $d$. On the other hand, for Shuffle-NC, we do not assume any structure, meaning that a query can be followed by a query as well as a document. First, we observe that Query-Only shows a larger performance drop over Doc-Only, suggesting in-context documents might contain more useful contents than in-context queries. Second, we observe that shuffling the pairings (Shuffle-C) marginally hurts in-context learning in RARe, as opposed to Shuffle-NC. Our findings align with prior study in decoder-only models (Min et al., ) which showed strict correspondence between $q$ and $d$ is not required for performance gains from in-context examples. We observe similar trends on keeping the training format fixed (Regular), and varying only the evaluation format – see Table 14 in the Appendix.

**Table 5: Impact of adding negative documents in the in-context prompt. All results are on E5-Mistral-Instruct. Negative documents $(d^{-})$ in the prompt do not enhance performance.**
| Training / Eval Setting | ArguAna | CQA | FiQA2018 | NFCorpus | SciFact | Touche2020 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| RARe-$q^{\text{inst+ic}}$ | 60.87 | 48.46 | 57.31 | 42.28 | 84.79 | 28.70 | 53.74 |
| RARe-$q^{\text{inst+ic+neg}}$ | 61.19 | 48.09 | 56.89 | 41.58 | 82.37 | 30.51 | 53.44 |

##### Negative Documents in the Query

So far, we have used $(q,d^{+})$ i.e (Query, Positive Document) pairs as the in-context prompt. Therefore, we study the impact of including negative documents. Specifically, the augmented query $q^{\text{inst+ic+neg}}$ includes examples of the form $(q,d^{+},d^{-})$, where the documents are prefixed with the term “Positive Document: ” and “Negative Document: ” respectively. Table 5 presents the downstream performance comparison between RARe variants trained solely on positive examples and those trained with augmented negative documents. The results indicate no performance gains from including negative documents. In fact, training with negative examples led to a slight decrease in performance.

**Table 6: Latency breakdown (in seconds) of each stage in the retrieval pipeline for $q^{\text{inst}}$ and $q^{\text{inst+ic}}$ evaluation settings. # Corpus denote the number of documents and Avg Q len. denote the average number of query tokens split by whitespace. Table 11 in the Appendix provides numbers on additional datasets.**
| Dataset | # Corpus | Eval Setting | Avg Q len. | NN | Query | Search | Total | Inc. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NFCorpus | 3633 | $q^{\text{inst}}$ | 3.3 | 0 | 3.30 | 0.54 | 3.84 | - |
| $q^{\text{inst+ic}}$ | 866.0 | 0.20 | 152.99 | 0.57 | 153.76 | 40.04$\times$ |  |  |
| FiQA2018 | 57638 | $q^{\text{inst}}$ | 10.9 | 0 | 6.92 | 7.92 | 14.84 | - |
| $q^{\text{inst+ic}}$ | 1016.6 | 0.45 | 278.62 | 8.83 | 287.90 | 19.40$\times$ |  |  |
| TRECCOVID | 171332 | $q^{\text{inst}}$ | 10.6 | 0 | 1.83 | 4.08 | 5.91 | - |
| $q^{\text{inst+ic}}$ | 722.54 | 0.31 | 21.78 | 4.30 | 26.39 | 4.47$\times$ |  |  |
| Touche2020 | 382545 | $q^{\text{inst}}$ | 6.6 | 0 | 1.42 | 9.29 | 10.71 | - |
| $q^{\text{inst+ic}}$ | 1287.8 | 0.20 | 40.32 | 10.50 | 51.02 | 4.76$\times$ |  |  |
| Quora | 522931 | $q^{\text{inst}}$ | 9.5 | 0 | 113.93 | 986.42 | 1100.35 | - |
| $q^{\text{inst+ic}}$ | 129.5 | 3.19 | 530.33 | 982.61 | 1516.13 | 1.38$\times$ |  |  |
| DBPedia | 4635922 | $q^{\text{inst}}$ | 5.5 | 0 | 36.93 | 588.38 | 625.31 | - |
| $q^{\text{inst+ic}}$ | 158.2 | 0.19 | 46.21 | 709.27 | 755.67 | 1.21$\times$ |  |  |

### 6.2 Efficiency Analysis

In Table 6, we present a breakdown of the latency of each stage of the retrieval pipeline for both baseline ($q^{\text{inst}}$) and in-context ($q^{\text{inst+ic}}$) settings. We measure the total time required to obtain nearest-neighbour in-context examples (NN) from BM25, compute query embeddings (Query), and perform search with FAISS (Douze et al., ) with encoded query embeddings on the pre-computed document index (Search). We observe that the largest contributing factors to latency are the average length of input queries (Avg Q len.), and the size of the index (# Corpus). For large query length and small corpus sizes, the in-context setting demonstrates a significant increase in total latency (19.40-40.04$\times$ for FiQA2018 and NFCorpus, respectively). However, for smaller average query lengths, this latency diminishes, as seen for Quora ($1.38\times$) and DBPedia ($1.21\times$). Moreover, the added latency due to the in-context setting also diminishes when the corpus size grows, as the time required for search outweighs the time to encode the query. For example, on Touche2020 with a larger corpus of 380K documents, the increase in latency is $4.76\times$ compared to FiQA2018 ($19.40\times$) for similar query lengths.

## 7 Related Work

##### In-context learning

ICL (Brown et al., ) allows models to adapt to new tasks in a few-shot manner by conditioning on the input data and the context provided at inference time. ICL has been effectively applied to a wide range of tasks such as classification (Milios et al., ), translation (Zhu et al., ), mathematical reasoning (Wei et al., , Zhou et al., ), and code generation (Li et al., ). Recent advancements have enhanced the ICL capabilities of language models through additional training procedures (Huang et al., , Gu et al., , Shi et al., ). Min et al. () and Chen et al. () perform meta-learning with in-context examples on a wide collection of tasks, with the goal of adapting to a new task at inference time through few-shot in-context examples. Other works have explored improving performance through more principled approaches to select in-context examples during inference (Zhang et al., , Sorensen et al., , Wang et al., , Qin et al., , Lee et al., ). A simple and popular approach is to retrieve examples that are most similar to the input (Liu et al., , Rubin et al., , Li et al., ). Providing in-context examples to re-ranking models has been studied in prior work (Drozdov et al., ), but the potential of augmenting retrievers themselves by leveraging in-context examples remains unexplored. Muennighoff et al. () explored providing an in-context example out-of-the-box, but showed an overall decrease in performance compared to zero-shot inference.

##### Retrieval

Large language models pre-trained with autoregressive setups (Jiang et al., , Dubey et al., ) have shown remarkable performance when adapted to retrieval tasks (Wang et al., , BehnamGhader et al., ), outperforming encoder-style retrievers (Izacard et al., , Wang et al., ). Despite these advancements, a challenge that remains is the ability to tailor retrieval systems to specific tasks or queries. To address this, a recent line of work explores incorporating instructions into retrieval by training models to use task-specific instructions along with the query (Su et al., , Asai et al., ). Oh et al. () and Weller et al. () further propose using instructions that are specific to each query. Another well-established technique in retrieval is query expansion (Jagerman et al., , Li et al., , Chen et al., ), where the query is augmented with additional terms to enrich the embedding as a form of relevance feedback (Lv & Zhai, ). Recent efforts have focused on applying LLMs to expand the original query before retrieval (Wang et al., , Shen et al., ). These techniques are not mutually exclusive, and can be integrated with our approach.

## 8 Conclusion

In this paper, we explored augmenting in-context examples to retrieval models. Building on the limitations of existing retriever models in following in-context examples, we introduced RARe, a simple strategy that equips retrievers with the ability to leverage in-context examples by training with semantically similar in-context examples. Through detailed experiments and analyses, we demonstrated that RARe consistently improves performance across various architectures and downstream retrieval tasks, demonstrating the effectiveness of in-context learning for retriever models.

## 9 Limitations and Future Work

Similar to in-context settings in autoregressive models, a limitation of our approach is the requirement for a set of in-context examples in the form of $(q,d^{+})$ pairs at inference time. RARe also introduces additional latency at inference time due to the encoding of in-context examples in the augmented query. This latency becomes more pronounced with longer documents, resulting in correspondingly extended queries. While the overhead is particularly significant for small indexes, it diminishes as the size of the index grows. To address these challenges, future research could explore several avenues, such as using efficient long-context retrievers (Saad-Falcon et al., , Zhang et al., ) as a backbone, or developing extractive and/or abstractive compression techniques on in-context documents to reduce query length. In this work, we used BM25 due to its lightweight nature to retrieve nearest neighbour examples. Future work could explore stronger models and approaches to reduce latency. Our current experiments are limited to English-language tasks, with potential to expand the scope to multilingual settings. Future work could explore curating synthetic data, an increasingly popular area of study for embedding models (Lee et al., , Wang et al., , Weller et al., ), but for training with in-context examples. Future work could also explore developing new contrastive objectives to provide better signals during training with in-context examples.

#### Acknowledgments

The work is partially supported by NSF grant IIS-2312948. We also thank Fangyuan Xu, Michael Zhang, Anuj Diwan, and other members of the UT NLP community for insightful feedback.

## Appendix

The appendix is organized as follows:

- •
In Appendix A, we present details on additional data preprocessing and other training details.
- •
In Appendix B, we present additional results and experiments.

## Appendix A Experimental Details

### A.1 Training Details

##### Hyperparameters

For fine-tuning Llama-3-8B, we follow the setting outlined in Ma et al. (). We train on 4 H100 GPUs with per-device batch size 8 and gradient accumulation steps 4. We apply LoRA (Hu et al., ) with $r$=32, temperature of 0.01, learning rate 1e-4 with 100 warmup steps. We use a sequence length of 512 for documents and 1024 for queries as in-context augmented queries are longer. For RARe we use a mixture of 70% examples with in-context examples and 30% without Table 16.

When fine-tuning existing retriever models (E5-Mistral-Instruct, LLM2Vec-Llama-3-8B-Supervised), we follow a setting similar to BehnamGhader et al. (). We train on 8 H100 GPUs with a largest possible per-device batch size of 32 along with 2 gradient accumulation steps. We consider a random subset of $100K$ examples from the public E5 dataset mixture (Springer et al., , Wang et al., ). We use a learning rate of 2e-4, maximum sequence length 1024, warmup ratio 0.1 for 1 epoch. For E5-Mistral-Instruct, we apply LoRA (Hu et al., ) $r$=16, and $r$=4 for LLM2Vec-Llama-3-8B-Supervised since a higher rank was leading to severe overfitting on the instruction baseline.

### A.2 Data Processing

##### RAR-b

Since RAR-b benchmark provides only test split, we parse the original training data for each dataset to use as in-context examples. We exclude datasets without any training splits and 2 datasets that were a mixture of multiple tasks or datasets, thereby being difficult to parse. This results in 8 datasets to evaluate on. We preprocess the training split to match the format of RAR-b test split, without excluding any instances. An exception is made for $\alpha$-NLI, where there were multiple identical instances. Therefore, we removed such duplicates, resulting in 72,046 in-context candidates. Furthermore, some RAR-b queries are composed of sentences with (multiple) indicators (e.g., Start:, End:). To address this, we make a minor modification in formatting, enclosing the queries in brackets. The final query representation is $q^{\text{inst+ic}}=\text{Instruct: }\{t\};\text{ Query: }[\{q^{ic}_{1}\}];
\text{ Document: }\{d^{ic+}_{1}\}\,\cdots;\text{ Query: }[\{q\}]$.

##### Inference Algorithm

Algorithm 2 provides a detailed outline of inference with RARe.

##### Promptriever

Promptriever(Weller et al., ) employs 10 different prompts and reports the highest score for each dataset. We apply the prompt that works the best (outperforms 5/15 datasets), which is as follows: A document that meets these criteria is considered relevant, while a document that does not meet these criteria is considered non-relevant.

Figure: Algorithm 2 RARe - Inference

## Appendix B Additional Experiments

**Table 7: Performance (nDCG@10) on BeIR (Thakur et al., ) when fine-tuning retriever model on E5 dataset. We report a breakdown of performance on In-Domain (ID) and Out-of-Domain (OOD) tasks on BeIR.**
|  | LLM2Vec-Llama-3-8b-Supervised | E5-Mistral-Instruct |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Category | Dataset | Base | Instruct | RARe | Base | Instruct | RARe |  |  |
|  |  | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ |
| ID | FEVER | 90.20 | 88.12 | 88.43 | 86.62 | 87.84 | 91.50 | 90.18 | 90.48 |
| HotpotQA | 71.76 | 72.50 | 73.83 | 79.09 | 75.72 | 73.91 | 72.18 | 75.95 |  |
| NQ | 64.21 | 63.63 | 65.00 | 66.13 | 63.53 | 67.44 | 68.15 | 67.66 |  |
| QuoraRetrieval | 87.16 | 87.85 | 87.88 | 87.63 | 89.61 | 89.82 | 89.59 | 88.95 |  |
| MSMARCO | 43.24 | 40.19 | 40.77 | 38.88 | 43.06 | 41.89 | 41.88 | 41.88 |  |
| OOD | ArguAna | 62.78 | 60.51 | 59.54 | 57.05 | 61.65 | 61.19 | 62.90 | 60.87 |
| ClimateFEVER | 34.27 | 34.49 | 34.67 | 34.73 | 38.35 | 39.03 | 38.99 | 37.50 |  |
| CQADupStack | 48.25 | 49.76 | 49.10 | 49.93 | 42.97 | 44.82 | 45.57 | 48.46 |  |
| DBPedia | 48.34 | 48.61 | 48.41 | 49.09 | 48.89 | 48.92 | 49.24 | 49.65 |  |
| FiQA2018 | 55.33 | 52.99 | 54.26 | 52.82 | 56.81 | 57.39 | 56.33 | 57.31 |  |
| NFCorpus | 41.83 | 41.92 | 41.61 | 41.84 | 38.58 | 40.99 | 41.19 | 42.28 |  |
| SCIDOCS | 22.96 | 23.97 | 22.92 | 23.35 | 16.32 | 17.94 | 18.71 | 20.19 |  |
| SciFact | 78.22 | 76.89 | 77.70 | 81.77 | 76.42 | 77.28 | 77.11 | 84.79 |  |
| Touche2020 | 20.50 | 22.11 | 22.71 | 19.54 | 26.27 | 29.35 | 27.56 | 28.7 |  |
| TRECCOVID | 80.34 | 68.37 | 78.55 | 82.78 | 87.03 | 72.89 | 77.03 | 79.58 |  |
|  | Average | 56.63 | 55.35 | 56.36 | 56.76 | 56.87 | 56.96 | 57.11 | 58.28 |

**Table 8: Performance on reasoning-focused IR benchmark RAR-b (Xiao et al., ) when fine-tuning existing retriever models.**
|  | LLM2Vec-Llama-3-8b-Supervised | E5-Mistral-Instruct |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Base | Instruct | RARe | Base | Instruct | RARe |  |  |
|  | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ |
| ARC-C | 18.81 | 18.77 | 18.28 | 17.02 | 19.00 | 20.37 | 22.72 | 26.44 |
| $\alpha$-NLI | 26.59 | 27.29 | 25.25 | 23.66 | 26.04 | 25.70 | 24.19 | 23.23 |
| HellaSwag | 34.32 | 34.19 | 34.19 | 33.29 | 35.38 | 35.99 | 35.07 | 36.29 |
| PIQA | 33.57 | 37.07 | 38.12 | 39.72 | 39.80 | 39.35 | 37.22 | 41.35 |
| Quail | 6.83 | 6.06 | 5.57 | 4.25 | 8.40 | 10.94 | 15.34 | 14.69 |
| SiQA | 6.99 | 5.34 | 4.39 | 4.55 | 5.66 | 5.45 | 5.75 | 6.15 |
| TempReason-L1 | 5.24 | 5.89 | 5.55 | 7.87 | 3.60 | 4.71 | 4.55 | 4.67 |
| WinoGrande | 40.02 | 52.88 | 48.47 | 54.44 | 39.48 | 50.41 | 44.26 | 53.50 |
| Average | 21.55 | 23.44 | 22.48 | 23.10 | 22.17 | 24.12 | 23.64 | 25.79 |

**Table 9: Performance (nDCG@10) on BeIR when training decoder-only models.**
| Dataset | Llama2 | Llama3 | Llama-3.1-Instruct |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| RepLLaMA | RepLLaMA | RARe | RepLLaMA | Promptreiver | RARe |  |  |
| $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ |  |
| ArguAna | 48.60 | 52.83 | 49.48 | 51.38 | 58.90 | 54.77 | 52.83 |
| ClimateFEVER | 29.30 | 32.52 | 32.12 | 33.13 | 29.80 | 35.91 | 34.24 |
| CQADupStack | 37.91 | 42.59 | 42.96 | 41.58 | 42.18 | 42.55 | 43.31 |
| DBPedia | 44.80 | 45.62 | 45.79 | 44.73 | 46.00 | 45.87 | 45.95 |
| FEVER | 82.90 | 81.79 | 83.66 | 79.22 | 85.50 | 80.05 | 81.84 |
| FiQA2018 | 45.00 | 44.31 | 47.13 | 44.50 | 47.20 | 44.36 | 46.20 |
| HotpotQA | 68.80 | 72.24 | 72.72 | 70.90 | 71.70 | 70.55 | 74.01 |
| MSMARCO | 42.00 | 43.56 | 44.77 | 43.67 | 42.70 | 41.65 | 42.93 |
| NFCorpus | 36.00 | 37.73 | 39.34 | 38.77 | 38.50 | 38.16 | 39.74 |
| NQ | 63.00 | 62.70 | 65.96 | 61.09 | 63.80 | 60.92 | 65.20 |
| Quora | 86.00 | 88.34 | 87.65 | 86.84 | 87.30 | 87.95 | 87.65 |
| SCIDOCS | 16.10 | 19.66 | 19.45 | 19.26 | 20.80 | 20.02 | 19.52 |
| SciFact | 75.30 | 75.02 | 77.20 | 75.38 | 77.50 | 74.59 | 76.54 |
| TRECCOVID | 83.90 | 83.15 | 85.76 | 83.15 | 84.50 | 77.52 | 85.30 |
| Touche2020 | 34.10 | 27.84 | 32.89 | 30.77 | 31.70 | 25.47 | 32.38 |
| Average | 52.91 | 53.99 | 55.13 | 53.62 | 55.21 | 53.36 | 55.18 |

**Table 10: Performance (nDCG@10) on datasets from RAR-b when training decoder-only models.**
| Dataset | Llama2 | Llama3 | Llama-3.1-Instruct |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| RepLLaMA | RepLLaMA | RARe | RepLLaMA | Promptreiver | RARe |  |  |
| $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst}}$ | $q^{\text{inst+ic}}$ |  |
| ARC-C | 11.79 | 11.65 | 13.48 | 11.68 | 14.63 | 13.24 | 15.02 |
| $\alpha$-NLI | 25.40 | 24.35 | 30.38 | 24.96 | 24.70 | 27.34 | 31.58 |
| HellaSwag | 30.83 | 31.47 | 30.27 | 31.03 | 32.57 | 31.42 | 28.81 |
| PIQA | 31.56 | 32.84 | 34.12 | 33.42 | 34.80 | 34.23 | 35.59 |
| Quail | 6.40 | 6.21 | 5.98 | 5.71 | 7.80 | 6.92 | 6.91 |
| SiQA | 2.82 | 2.61 | 3.87 | 2.75 | 3.53 | 2.18 | 3.14 |
| TempReason-L1 | 1.49 | 1.75 | 3.61 | 2.05 | 4.32 | 4.84 | 6.59 |
| WinoGrande | 51.58 | 37.11 | 57.01 | 42.01 | 45.25 | 44.72 | 61.69 |
| Average | 20.23 | 18.50 | 22.34 | 19.20 | 20.95 | 20.61 | 23.67 |

### B.1 Performance on BeIR and RAR-b

Table 7 and Table 8 provide detailed numbers on each dataset from BeIR and RAR-b respectively when training from retriever checkpoints. Table 9 and Table 10 provide detailed numbers on each dataset from BeIR and RAR-b respectively when training from decoder-only LLMs.

**Table 11: Latency breakdown (in seconds) of each stage in the retrieval pipeline for $q^{\text{inst}}$ and $q^{\text{inst+ic}}$ evaluation settings. # Corpus denote the number of documents and Avg Q len. denote the average number of query tokens split by whitespace.**
| Dataset | # Corpus | Eval Setting | Avg Q len. | NN | Query | Search | Total | Inc. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SciFact | 5183 | $q^{\text{inst}}$ | 12.5 | 0 | 4.52 | 0.61 | 5.13 | - |
| $q^{\text{inst+ic}}$ | 1250.7 | 0.25 | 212.35 | 0.61 | 213.21 | 41.56$\times$ |  |  |
| SCIDOCS | 25657 | $q^{\text{inst}}$ | 9.4 | 0 | 11.29 | 5.74 | 17.03 | - |
| $q^{\text{inst+ic}}$ | 901.1 | 0.67 | 354.82 | 5.79 | 361.28 | 21.21$\times$ |  |  |
| CQADupStack | 38100 | $q^{\text{inst}}$ | 8.6 | 0 | 10.80 | 10.20 | 21.01 | - |
| $q^{\text{inst+ic}}$ | 678.2 | 1.33 | 570.04 | 10.30 | 581.67 | 27.69$\times$ |  |  |
| ClimateFEVER | 5416593 | $q^{\text{inst}}$ | 20.2 | 0 | 26.06 | 1725.90 | 1751.96 | - |
| $q^{\text{inst+ic}}$ | 831.3 | 3.55 | 651.76 | 1723.84 | 2379.15 | 1.36$\times$ |  |  |

### B.2 Efficiency Evaluation

Table 11 provides a breakdown of latency on additional datasets.

**Table 12: Impact of the number of in-context examples ($k$) at inference time. $k=5$ during training. All results are on E5-Mistral-Instruct. In general, performance increases when increasing the number of examples, and the optimal number of examples can vary by task.**
|  |  |  |  | # Examples |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | Instruct (0) | 0 | 1 | 3 | 5 | 10 |
| ArguAna | 61.19 | 62.90 | 61.24 | 60.99 | 61.18 | 60.37 |
| ClimateFEVER | 39.03 | 38.99 | 38.27 | 37.97 | 37.50 | 37.67 |
| CQADupStack | 44.82 | 45.57 | 47.49 | 48.33 | 48.46 | 48.48 |
| DBPedia | 48.92 | 49.24 | 49.79 | 48.34 | 49.65 | 49.82 |
| FiQA2018 | 57.39 | 56.33 | 57.61 | 57.42 | 57.31 | 57.38 |
| NFCorpus | 40.99 | 41.19 | 41.48 | 42.10 | 42.28 | 42.29 |
| SCIDOCS | 17.94 | 18.71 | 19.83 | 20.17 | 20.19 | 20.20 |
| SciFact | 77.28 | 77.11 | 83.56 | 84.45 | 84.79 | 85.12 |
| Touche2020 | 29.35 | 27.56 | 27.53 | 27.70 | 28.70 | 30.77 |
| TRECCOVID | 72.89 | 77.03 | 76.96 | 78.99 | 79.58 | 78.77 |
| Average | 48.98 | 49.46 | 50.38 | 50.65 | 50.96 | 51.09 |

**Table 13: Impact of the number of in-context examples ($k$) during training and inference. All results are on E5-Mistral-Instruct. In general, performance increases when increasing the number of examples, and the optimal number of in-context examples can vary by task.**
|  |  |  |  | # Examples |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | Instruct (0) | 0 | 1 | 3 | 5 | 10 |
| Arguana | 61.19 | 62.90 | 60.47 | 62.98 | 60.87 | 58.85 |
| ClimateFEVER | 39.03 | 38.99 | 37.94 | 36.45 | 37.50 | 36.54 |
| CQADupStack | 44.82 | 45.57 | 46.76 | 47.12 | 48.46 | 48.92 |
| DBPedia | 48.92 | 49.24 | 47.70 | 49.05 | 49.65 | 47.95 |
| FiQA2018 | 57.39 | 56.33 | 56.07 | 57.08 | 57.31 | 57.03 |
| NFCorpus | 40.99 | 41.19 | 40.67 | 40.77 | 42.28 | 42.24 |
| SCIDOCS | 17.94 | 18.71 | 20.01 | 19.28 | 20.19 | 21.54 |
| SciFact | 77.28 | 77.11 | 81.47 | 83.71 | 84.79 | 87.61 |
| Touche2020 | 29.35 | 27.56 | 29.78 | 27.12 | 28.70 | 28.29 |
| TRECCOVID | 72.89 | 77.03 | 78.95 | 73.25 | 79.58 | 86.11 |
| Average | 48.98 | 49.46 | 50.18 | 48.83 | 51.11 | 53.16 |

**Table 14: In-Context Format Comparing variants of in-context example format on E5-Mistral-Instruct during inference only. Training is done with the Regular format. Instruct refers to the baseline which does not use any in-context examples.**
|  | Instruct | RARe |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | - | Query-Only | Doc-only | Shuffle-NC | Shuffle-C | Regular |
| ArguAna | 61.19 | 57.36 | 60.35 | 55.64 | 60.49 | 60.87 |
| ClimateFEVER | 39.03 | 38.35 | 38.32 | 37.44 | 37.84 | 37.50 |
| CQADupStack | 44.82 | 39.56 | 48.43 | 47.70 | 48.27 | 48.46 |
| DBPedia | 48.92 | 49.14 | 49.69 | 49.72 | 50.04 | 49.65 |
| FiQA2018 | 57.39 | 55.67 | 56.85 | 56.64 | 57.41 | 57.31 |
| NFCorpus | 40.99 | 41.00 | 42.09 | 42.02 | 41.92 | 42.28 |
| SCIDOCS | 17.94 | 19.06 | 20.06 | 19.98 | 20.25 | 20.19 |
| SciFact | 77.28 | 77.46 | 81.88 | 81.51 | 82.20 | 84.79 |
| Touche2020 | 29.35 | 27.04 | 29.02 | 28.60 | 29.31 | 28.70 |
| TRECCOVID | 72.89 | 75.11 | 79.97 | 79.07 | 80.03 | 79.58 |
| Average | 48.98 | 47.98 | 50.67 | 49.83 | 50.78 | 50.93 |

**Table 15: In-Context Format Comparing variants of in-context example format on E5-Mistral-Instruct. Instruct refers to the baseline which does not use any in-context examples.**
|  | Instruct | RARe |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | - | Query-Only | Doc-Only | Shuffle-NC | Shuffle-C | Regular |
| ArguAna | 61.19 | 58.88 | 57.54 | 60.17 | 58.97 | 60.87 |
| ClimateFEVER | 39.03 | 36.21 | 35.59 | 30.83 | 35.71 | 37.50 |
| CQADupStack | 44.82 | 46.66 | 48.28 | 45.78 | 47.97 | 48.46 |
| DBPedia | 48.92 | 49.98 | 49.08 | 50.93 | 50.24 | 49.65 |
| FiQA2018 | 57.39 | 54.44 | 56.02 | 54.25 | 55.98 | 57.31 |
| NFCorpus | 40.99 | 41.42 | 41.62 | 41.17 | 41.78 | 42.28 |
| SCIDOCS | 17.94 | 20.04 | 20.12 | 20.35 | 20.11 | 20.19 |
| SciFact | 77.28 | 78.84 | 79.80 | 80.70 | 80.51 | 84.79 |
| Touche2020 | 29.35 | 28.09 | 29.01 | 29.18 | 28.97 | 28.70 |
| TRECCOVID | 72.89 | 79.54 | 83.29 | 82.14 | 82.97 | 79.58 |
| Average | 48.98 | 49.41 | 50.04 | 49.55 | 50.32 | 50.93 |

### B.3 Choice of In-Context Examples

Table 13 provides detailed numbers for varying in-context examples on all OOD BeIR tasks. Table 15 provides detailed numbers for various prompt formats on all OOD BeIR tasks.

**Table 16: Performance (nDCG@10) on datasets from the BeIR benchmark Thakur et al., when training decoder-only model (Llama3). Applying RARe with only in-context examples can lead to degradation of performance in the zero-shot setting ($q^{\text{inst}}$), but this is easily mitigated my including a mixture of $q^{\text{inst}}$ and $q^{\text{inst+ic}}$ data (30% and 70%) respectively.**
| Training | Eval | NQ | Quora | NFCorpus | SciFact | SCIDOCS | FiQA2018 | CQA | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RepLLaMA-$q^{\text{inst}}$ | $q^{\text{inst}}$ | 62.70 | 88.34 | 37.73 | 75.02 | 19.66 | 44.31 | 42.59 | 52.91 |
|  | $q^{\text{inst}}$ | 39.64 | 88.39 | 35.42 | 74.52 | 21.04 | 30.44 | 37.74 | 46.74 |
| RARe-$q^{\text{inst+ic}}$ | $q^{\text{inst+ic}}$ | 65.19 | 86.79 | 38.87 | 78.41 | 19.70 | 46.58 | 43.75 | 54.18 |
|  | $q^{\text{inst}}$ | 63.68 | 87.84 | 38.06 | 76.07 | 20.11 | 46.02 | 42.99 | 53.54 |
| RARe-$q^{\text{inst}}$ + $q^{\text{inst+ic}}$ | $q^{\text{inst+ic}}$ | 65.96 | 87.65 | 39.34 | 77.20 | 19.45 | 47.13 | 42.96 | 54.24 |

### B.4 Mixture of Training Data

In Table 16, we analyze the impact of training with only in-context examples when starting from decoder-only LLMs. As opposed to starting from existing retriever models, which have been trained without in-context examples, we observe that performance drops in the instruction-only setting. This can be largely mitigated by considering a mixture of in-context and instruction-only queries.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - While in-context learning is not new, its application to retriever models in this specific manner is new. The paper creatively adapts this technique, showing potential new directions for retrieval model improvements.
summary: The paper explores the application of in-context learning to improve the performance of retriever models in information retrieval tasks. The authors introduce RARe, a method where in-context examples (query-document pairs related to the target query) are used during the fine-tuning of retriever models. This approach differs from direct application in LLMs where in-context examples are prepended at inference time. RARe enhances retrieval performance by up to 2.72% in nDCG across various datasets.
weaknesses: - While the application of in-context learning to retrievers is new, it might not strike everyone as a groundbreaking shift.
- The approach, as well as the performance gain, looks a bit incremental. There might also be a tradeoff between efficiency, accuracy, and ease of use of the method.
- Discussion on how RARe might scale or face challenges in real-world applications beyond the benchmarks used could be expanded.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1) What are the actual trends of NDCG in Figure 4?
2) Can you explain why ArguAna’s Retrieved/Random setup is worse than Random/Random in Figure 3?
soundness: 3
strengths: 1. The paper is well-structured and easy to follow.
2. Extensive experiments were conducted on recent base models and popular datasets, such as MS-MARCO, BeIR, and RAR-b, demonstrating that the proposed RARe method enhances baseline models, including Llama and other LLM-based retrievers.
3. Detailed ablation studies investigate critical questions, such as the impact of retrieved vs random in-context examples and whether semantically closer in-context examples are more beneficial.
summary: This paper employs BM25 to retrieve top-k relevant queries and their associated documents as in-context examples, enhancing query representation when using an LLM as the retrieval encoder. Extensive experiments, including comprehensive ablation studies, were conducted to demonstrate the effectiveness of the proposed method.
weaknesses: 1. There are no statistical significance tests to confirm that the improvements over baselines in Tables 1 and 2 are meaningful.
2. Only a basic retriever, BM25, was applied.
3. In Figure 3, the performance of ArguAna’s Retrieved/Random setup is worse than Random/Random, which is inconsistent with other datasets and lacks an explanation.
4. Figure 4 appears to contradict the paper’s premise, which relies on similar queries and their associated documents to enhance query representation. When score@Top-1 improves, relative improvement drops are observed on NFCorpus and FiQA2018, without further clarification. Additionally, only relative results are reported, making it difficult to discern the actual trend of NDCG against score@Top-1.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Why the search time for DBPedia is much lower than Quora even if DBPedia has much larger corpora in Table 6?
soundness: 3
strengths: 1. Studying adding in context examples for the query is a under explored topic for retrieval community.
2. The results are positive and the ablations are quite extensive.
summary: This paper explores adding in context examples in the query for retriever. It first retrieves related pairs using BM25 and then explores different methods, such as LLM based methods as well as based on existing well trained retriever methods. The idea is quite natural to explore and it is a good plus for the retrieval community with its positive results as well as its extensive ablation study.
weaknesses: 1 The baselines are a bit weak where I am not sure how much value will the less than 2% improvement add. There are other leading models on BEIR benchmark and how will the proposed methods compare to those and will those method get improved after adding in context example?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In Figure 3, only part of OOD test sets from BEIR are selected. Could you supplement with the complete set to observe trends, especially since these selected test sets are relatively short?
2. In Table 4, the performance with doc-only is also quite good for several test sets, such as CQA, NFCorpus, and Touche2020. Given that examples can be viewed as a form of query expansion, how do you explain the improvement brought by doc-only, and what is its relationship with ICL training? Can training with doc-only also bring performance gain?
3. To truly demonstrate the ICL capabilities, evaluating more embedding tasks and scenarios, such as classification, reranking, and clustering, might be necessary. Have you considered adding other tasks from METB that can better test the ICL ability?
soundness: 2
strengths: 1. The paper explores in-context examples for retrieval, which are widely employed in decoder-only LLMs, to enable the model to incorporate the semantic and pattern information of examples for adjusting its output embeddings across various retrieval tasks with differing user intents. This approach could potentially be a step forward from instruction-based retrieval models to ICL-based retrieval models.
2. Despite the simplicity of the training method, the paper provides an extensive analysis of how the quality, quantity, and selection of in-context examples impact the model's performance.
summary: This paper investigates whether training with in-context examples can enhance the in-domain performance of LLM-based retriever models and improve their generalization to different unseen tasks. It introduces a simple training strategy that identifies examples using BM25 during training and performs inference with in-context examples. Experiments were conducted on the BEIR benchmark and a reasoning-intensive retrieval benchmark, RAR-b.
weaknesses: 1. It is unclear whether the in-context examples truly contribute during training: Since all examples are retrieved using BM25, it raises the question of whether they merely act as a form of query expansion. Deeper experimentation is needed to address this, such as retrieving only similar passages and incorporating them into the training process. As shown in Table 4, using only documents during inference sometimes yields good results; thus, what happens if doc-only is used during training as well?
2. This paper lacks an in-depth analysis of the generated embeddings. For instance, why does adding examples lead to performance declines on some test sets, such as ArguAna and ClimateFEVER? Further attribution analysis, like exploring the impact of examples on attention patterns, could deepen the study and enhance its contributions to the research community.
3. The training strategy is too simplistic: A more sophisticated training strategy could be explored. The current approach is too basic. For example, the impact of using in-batch negatives, a commonly employed technique in retrieval training, remains unexplored. Specifically, if in-batch negatives come from different tasks, could they help in better training the ICL capabilities due to differing examples and retrieval intents?
4. The analytical experiments should include results on the complete BEIR benchmark or at least all out-of-domain (OOD) results.

Meta-review: This paper proposes a novel framework, RARe, which augments the input with in-context learning examples for retrieving relevant documents. Different from existing ICL which helps LLMs during the inference/generation process, RARe explores the benefit of ICL in producing semantic embeddings to assist retrieval. By using BM25 to select ICL examples and with contrastive training, the retrieval model shows better performances on benchmark datasets, compared with existing baselines.

Strengths:
- The idea of exploring ICL to enhance retrieval is novel and interesting.
- The experiments showcase the advantage of the proposed method compared with existing baselines.
- Additional experiments such as varying in-context formats and significance test are provided during the rebuttal phase which is helpful.

Weaknesses:
- The benefit of this approach and the potential implications in real scenarios remains uncertain. The performance improvement may not be profound, especially when compared with the increased latency. This poses the question whether it is worth spending extra efforts when doing retrieval.
- The scope of this method is a bit limited. The ICL encoding strategy could potentially be applied to more tasks besides retrieval.
- The performance could be sensitive to the ICL retrieval method which is not fully explored.

Decision: Reject