RARe: Retrieval Augmented Retrieval with In-Context Examples

Authors: Atula Tejaswi, Yoonsang Lee, Sujay Sanghavi, Eunsol Choi
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Retrieval, Embedding models, In-Context Learning

Abstract:
We investigate whether in-context examples, widely used in decoder-only language models (LLMs), can improve embedding models for retrieval. Unlike in LLMs, naively prepending in-context examples (query-document pairs) to the target query at inference time does not work out of the box. We introduce a simple approach to enable retrievers to use in-context examples. Our approach, \texttt{RARe}, fine-tunes a pre-trained model with in-context examples whose query is semantically similar to the target query. This can be applied to adapt various base architectures (i.e., decoder-only language models, retriever models) and consistently achieves performance gains of up to +2.72\% nDCG across various open-domain retrieval datasets (BeIR, RAR-b). Particularly, we find \texttt{RARe} exhibits stronger out-of-domain generalization compared to models using queries without in-context examples, similar to what is seen for in-context learning in LLMs. While our approach incurs additional computational cost to encode lengthier queries, the impact is less pronounced in large-corpus scenarios. We further provide analysis on the design choices of in-context example augmentation and lay the foundation for future work in this space.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730258738499

thread_type:
official_review

<thread_object>
creator: reviewer_uz4y

<content>
summary:
This paper investigates whether training with in-context examples can enhance the in-domain performance of LLM-based retriever models and improve their generalization to different unseen tasks. It introduces a simple training strategy that identifies examples using BM25 during training and performs inference with in-context examples. Experiments were conducted on the BEIR benchmark and a reasoning-intensive retrieval benchmark, RAR-b.

strengths:
1. The paper explores in-context examples for retrieval, which are widely employed in decoder-only LLMs, to enable the model to incorporate the semantic and pattern information of examples for adjusting its output embeddings across various retrieval tasks with differing user intents. This approach could potentially be a step forward from instruction-based retrieval models to ICL-based retrieval models.
2. Despite the simplicity of the training method, the paper provides an extensive analysis of how the quality, quantity, and selection of in-context examples impact the model's performance.

weaknesses:
1. It is unclear whether the in-context examples truly contribute during training: Since all examples are retrieved using BM25, it raises the question of whether they merely act as a form of query expansion. Deeper experimentation is needed to address this, such as retrieving only similar passages and incorporating them into the training process. As shown in Table 4, using only documents during inference sometimes yields good results; thus, what happens if doc-only is used during training as well?
2. This paper lacks an in-depth analysis of the generated embeddings. For instance, why does adding examples lead to performance declines on some test sets, such as ArguAna and ClimateFEVER? Further attribution analysis, like exploring the impact of examples on attention patterns, could deepen the study and enhance its contributions to the research community.
3. The training strategy is too simplistic: A more sophisticated training strategy could be explored. The current approach is too basic. For example, the impact of using in-batch negatives, a commonly employed technique in retrieval training, remains unexplored. Specifically, if in-batch negatives come from different tasks, could they help in better training the ICL capabilities due to differing examples and retrieval intents?
4. The analytical experiments should include results on the complete BEIR benchmark or at least all out-of-domain (OOD) results.

questions:
1. In Figure 3, only part of OOD test sets from BEIR are selected. Could you supplement with the complete set to observe trends, especially since these selected test sets are relatively short?
2. In Table 4, the performance with doc-only is also quite good for several test sets, such as CQA, NFCorpus, and Touche2020. Given that examples can be viewed as a form of query expansion, how do you explain the improvement brought by doc-only, and what is its relationship with ICL training? Can training with doc-only also bring performance gain?
3. To truly demonstrate the ICL capabilities, evaluating more embedding tasks and scenarios, such as classification, reranking, and clustering, might be necessary. Have you considered adding other tasks from METB that can better test the ICL ability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670935356

thread_type:
official_review

<thread_object>
creator: reviewer_svxx

<content>
summary:
This paper explores adding in context examples in the query for retriever. It first retrieves related pairs using BM25 and then explores different methods, such as LLM based methods as well as based on existing well trained retriever methods. The idea is quite natural to explore and it is a good plus for the retrieval community with its positive results as well as its extensive ablation study.

strengths:
1. Studying adding in context examples for the query is a under explored topic for retrieval community.
2. The results are positive and the ablations are quite extensive.

weaknesses:
1 The baselines are a bit weak where I am not sure how much value will the less than 2% improvement add. There are other leading models on BEIR benchmark and how will the proposed methods compare to those and will those method get improved after adding in context example?

questions:
Why the search time for DBPedia is much lower than Quora even if DBPedia has much larger corpora in Table 6?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704391819

thread_type:
official_review

<thread_object>
creator: reviewer_pxah

<content>
summary:
This paper employs BM25 to retrieve top-k relevant queries and their associated documents as in-context examples, enhancing query representation when using an LLM as the retrieval encoder. Extensive experiments, including comprehensive ablation studies, were conducted to demonstrate the effectiveness of the proposed method.

strengths:
1. The paper is well-structured and easy to follow.
2. Extensive experiments were conducted on recent base models and popular datasets, such as MS-MARCO, BeIR, and RAR-b, demonstrating that the proposed RARe method enhances baseline models, including Llama and other LLM-based retrievers.
3. Detailed ablation studies investigate critical questions, such as the impact of retrieved vs random in-context examples and whether semantically closer in-context examples are more beneficial.

weaknesses:
1. There are no statistical significance tests to confirm that the improvements over baselines in Tables 1 and 2 are meaningful.
2. Only a basic retriever, BM25, was applied.
3. In Figure 3, the performance of ArguAna’s Retrieved/Random setup is worse than Random/Random, which is inconsistent with other datasets and lacks an explanation.
4. Figure 4 appears to contradict the paper’s premise, which relies on similar queries and their associated documents to enhance query representation. When score@Top-1 improves, relative improvement drops are observed on NFCorpus and FiQA2018, without further clarification. Additionally, only relative results are reported, making it difficult to discern the actual trend of NDCG against score@Top-1.

questions:
1) What are the actual trends of NDCG in Figure 4?
2) Can you explain why ArguAna’s Retrieved/Random setup is worse than Random/Random in Figure 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716984906

thread_type:
official_review

<thread_object>
creator: reviewer_3gic

<content>
summary:
The paper explores the application of in-context learning to improve the performance of retriever models in information retrieval tasks. The authors introduce RARe, a method where in-context examples (query-document pairs related to the target query) are used during the fine-tuning of retriever models. This approach differs from direct application in LLMs where in-context examples are prepended at inference time. RARe enhances retrieval performance by up to 2.72% in nDCG across various datasets.

strengths:
- While in-context learning is not new, its application to retriever models in this specific manner is new. The paper creatively adapts this technique, showing potential new directions for retrieval model improvements.

weaknesses:
- While the application of in-context learning to retrievers is new, it might not strike everyone as a groundbreaking shift.
- The approach, as well as the performance gain, looks a bit incremental. There might also be a tradeoff between efficiency, accuracy, and ease of use of the method.
- Discussion on how RARe might scale or face challenges in real-world applications beyond the benchmarks used could be expanded.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732165307760

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate the reviewers for their valuable comments and feedback. We are encouraged to hear from the reviewers that our paper studies an under-explored topic in retrieval models (**SvxX**,  **3Gic**), and presents a step forward towards new research directions in this space  (**Uz4Y, 3Gic**) by creatively adapting this technique (**3Gic**). All reviewers highlight the extensive (**Uz4Y, SvxX, 3Gic**) and critical (**PxAH**) nature of our experiments and ablation studies, spanning multiple datasets and base architectures. 

All changes in the paper are highlighted with blue font in the updated PDF. We address common concerns below:

**i) Significance of performance gain**: An overall improvement of 2% nDCG is statistically significant, given that the aggregate benchmarks encompass over 50,000 queries and 31 million documents. Prior work [1,2,3,4] in this domain also show relatively modest gains in this aggregate benchmark (0.3% to 1.6% on average over their respective baselines). While the improvements might seem modest in absolute terms, it is noteworthy in the context of benchmarks like BEIR.

In the updated draft, **we provide statistical significance tests** on each dataset for retriever and decoder-only checkpoints, respectively in Tables 7-10. In the retriever-checkpoint setting, RARe (E5-Mistral) and RARe (LLM2Vec) are statistically significant (p < 0.05) compared to their Instruct counterparts on the BEIR dataset on average. RARe (E5-Mistral) is statistically significant compared to Instruct on RAR-b on average. In the LLM-checkpoint setting, RARe (Llama-3.1-8B-Instruct) is statistically significant compared to Instruct on BEIR, and statistically significant compared to Promptriever on RAR-b.   

We also emphasize that our focus is not solely on achieving state-of-the-art performance but on developing a conceptual/empirical understanding of the potential of incorporating in-context examples into retrievers. 

**ii) Low performance on ArguAna:** We think this could be caused as the in-context examples in Arguana dataset are synthetically generated from prior work [6]. This leads to mismatch between queries in in-context examples with that of the original test queries (L403/footnote 1) i.e. the queries provided in the in-context examples are significantly shorter than the test queries. Similar behavior of format-sensitivity has been observed in LLMs [5].

**References**

[1] Adversarial Retriever-Ranker for Dense Text Retrieval  (Zhang et al., ICLR 2022)

[2] Unsupervised Dense Information Retrieval with Contrastive Learning (Izacard et al., TMLR 2022)

[3] Rethinking the Role of Token Retrieval in Multi-Vector Retrieval (Lee et al., NeurIPS 2023)

[4] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (BehnamGhader et al., COLM 2024)

[5] Reframing Instructional Prompts to GPTk’s Language (Mishra et al., ACL 2022)

[6] BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (Thakur et al., NeurIPS 2021 Datasets and Benchmark)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732766843912

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow up and additional clarifications

comment:
Dear Reviewers **PxAH, SvxX**, and **3Gic**,

Thank you once again for your valuable feedback and comments which has helped us improve the paper. We wanted to follow up to confirm that we have adequately addressed your concerns and to kindly request if you would consider reevaluating your assessment.

We would like to further clarify and contextualize the significance of our work (**3Gic**). We do believe that demonstrating the distinction of ICL between generative models & retrievers, and proposing a method to adapt ICL for retrievers represents a meaningful contribution. Notably, we found a concurrent work that is also under review at ICLR 2025:  
[1] https://openreview.net/forum?id=wfLuiDjQ0u 

Similar to us, [1] studies the incorporation of in-context examples to enhance text representations (showing a performance improvement of 0.97% on retrieval and 1.25% on other representation tasks on average over zero-shot fine-tuning), highlighting the relevance of our work.

Additionally, our work has some key differences, mentioned below:
* Our method incorporates retrieved in-context examples (obtained from BM25), as opposed to randomly selected in-context examples.
* We experiment with multiple architectures/checkpoints and training setups (training from decoder LLM vs retriever checkpoint, i.e. Table 1 & Table 2), while [1] evaluates on multiple types of tasks.

We also conduct extensive analysis on quantity, number, and format of in-context examples, some of which were highlighted as limitations by the reviewers of [1]. In summary, we:
* Demonstrate that providing in-context examples does not work out-of-the-box on retriever models, even when using nearest-neighbor examples (Figure 2).
* Show that even in the absence of multi-task learning or task-specific instruction, in-context learning helps out-of-domain performance (Table 1, training from decoder LLM checkpoint on MS-MARCO).
* Examine alternative formats of examples, such as plain query expansion with documents (Table 4).
* Study the role in-context example similarity on performance (Figure 3, Figure 4).
* Analyze the impact of adding negative-examples in the prompt (Table 5).
* Quantify the efficiency-performance tradeoff of adding in-context examples (Table 6).

We kindly request the reviewers to take these points into consideration and reassess the scores assigned to our paper, especially in light of the evaluations and scores provided to the concurrent work. We are confident that our work makes a notable contribution, and we greatly appreciate your thoughtful assessment of our submission.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel framework, RARe, which augments the input with in-context learning examples for retrieving relevant documents. Different from existing ICL which helps LLMs during the inference/generation process, RARe explores the benefit of ICL in producing semantic embeddings to assist retrieval. By using BM25 to select ICL examples and with contrastive training, the retrieval model shows better performances on benchmark datasets, compared with existing baselines.

Strengths:
- The idea of exploring ICL to enhance retrieval is novel and interesting.
- The experiments showcase the advantage of the proposed method compared with existing baselines.
- Additional experiments such as varying in-context formats and significance test are provided during the rebuttal phase which is helpful.

Weaknesses:
- The benefit of this approach and the potential implications in real scenarios remains uncertain. The performance improvement may not be profound, especially when compared with the increased latency. This poses the question whether it is worth spending extra efforts when doing retrieval.
- The scope of this method is a bit limited. The ICL encoding strategy could potentially be applied to more tasks besides retrieval.
- The performance could be sensitive to the ICL retrieval method which is not fully explored.