Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

## Abstract

Abstract Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources.
The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs.
It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance.
However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases.
This paper investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" as a key contributor.
To mitigate this and enhance the robustness of long-context LLM-based RAG, we propose both training-free and training-based approaches.
We first showcase the effectiveness of retrieval reordering as a simple yet powerful training-free optimization.
Furthermore, we explore training-based methods, specifically RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning with intermediate reasoning, demonstrating their capacity for substantial performance gains.
Finally, we conduct a systematic analysis of design choices for these training-based methods, including data distribution, retriever selection, and training context length.

## 1 Introduction

Retrieval-augmented generation (RAG) (Gao et al., ) empowers large language models (LLMs) to utilize external information sources by selecting the most relevant pieces from a large corpus (Zhao et al., ), thereby enhancing their effectiveness, customizability and efficiency in complex problem-solving. RAG can also mitigate issues such as factual inaccuracies (Augenstein et al., ) and hallucinations (Huang et al., ), which LLMs often exhibit when confronted with knowledge-intensive tasks. RAG systems typically employ a retriever to identify relevant information from a corpus, which is then presented in the context of an LLM as the generator.

Recent advances in computational resources and methodological innovations have enabled the development of LLMs that support increasingly longer context (Reid et al., ; Dubey et al., ).
This has even opened up new avenues for directly inputting entire corpora or knowledge bases into the LLMs.
Yet, it would still not be feasible for large corpora (e.g., Wikipedia) and can incur higher computational costs.
Despite extensive research on RAG (Xu et al., ; Li et al., ; Lee et al., ), the interplay with long-context LLMs, particularly how to optimally design RAG systems using them effectively, remains under-explored.
Existing works (Lin et al., ; Asai et al., ; Yoran et al., ) propose tuning LLMs for RAG, but predominantly focus on a limited number of retrieved passages (fewer than 10).
Intuitively, longer context would allow for the inclusion of more retrieved passages, leading to higher recall and potentially improved performance.
However, our findings reveal that this does not always hold true and highlight the need for a careful re-evaluation of standard RAG designs when utilizing long-context LLMs.
We demonstrate that achieving optimal performance in such systems and to fully utilize the opportunities provided by the LLMs require a holistic rethinking and effective novel approaches to the unique challenges.

This paper presents comprehensive analyses on long-context LLMs in RAG systems.
Contrary to the suggestions of previous work (Xu et al., ; Li et al., ), our research reveals that increasing the number of retrieved passages does not consistently improve performance with long-context LLMs (Section 3.1).
Instead, we observe that the generative modeling performance initially increases and then declines – simply providing more retrieved passages does not guarantee better outcomes.
Using stronger retrievers is also not a mitigation mechanism – indeed the performance degradation can even be more severe with them.
For deeper understanding of the phenomenon, we conduct further investigations, which reveal that increasing the number of retrieved passages can introduce irrelevant information (“noise”) that can mislead the LLM generation (Section 3.2).
We also examine the impact of “hard negatives” of different retrievers on the LLMs, and show that there are scenarios where the ‘hard negatives’ from stronger retrievers might confuse the LLM generation even more than those from weaker retrievers (Section 3.3).

To address the challenges identified in our analyses, we propose three methods, encompassing both training-free and training-based approaches, to enhance the performance of long-context LLMs in RAG applications:
(1) Retrieval reordering: recognizing the "lost-in-the-middle" phenomenon observed for long-context LLMs (Liu et al., ), we propose reordering retrieved documents based on their retrieval scores.
By prioritizing documents with higher scores at the beginning and end of the input sequences, we guide the LLMs’ attention towards more relevant information and mitigate the impact of hard negatives.
(2) Implicit robustness fine-tuning: given the ability to handle noisy retrieved context is not explicitly acquired during standard LLM training, we propose tuning the LLMs with the data comprising queries and retrieved documents, including those with potential noise.
This encourages the LLMs to implicitly learn robustness to hard negatives.
(3) Explicit relevance fine-tuning: while the previous method implicitly enhances robustness, it does not explicitly teach the LLMs to identify relevant documents.
Therefore, we propose augmenting the LLM tuning with an intermediate reasoning step, where the LLMs are trained to analyze the retrieved documents and explicitly identify relevant information before generating the final output.
This approach aims to improve the LLMs’ ability to discern relevant information from noise within the retrieved context.

Overall, the main contributions can be summarized as follows:

- •
Systematic analysis of long-context RAG: we systematically analyze the use of long-context LLMs in RAG systems, specifically examining the impact of retrieved "hard negatives" on performance.
- •
Novel methods for robust RAG: we propose three methods to improve the robustness of long-context LLMs in RAG: (1) a training-free method based on retrieval reordering, (2) implicit tuning for robustness to hard negatives and (3) explicit tuning with intermediate reasoning for relevance identification. Overall, our proposed approaches show significant accuracy and robustness improvements on long-context RAG performance.
- •
Comprehensive study of RAG-specific LLM tuning: we conduct a thorough investigation into various factors influencing the effectiveness of RAG-specific tuning, including data distribution, the employed retriever, and training context length.

## 2 Related Work

Large language models (LLMs) can be prone to hallucinations especially at knowledge-intensive tasks (Zhao et al., ; Huang et al., ; Augenstein et al., ).
Retrieval-augmented generation (RAG) addresses this by incorporating external knowledge sources to provide accurate and relevant information (Gao et al., ).
Traditional RAG systems comprise a retriever to identify relevant information and a generator to synthesize the answer (Zhao et al., ; Zhu et al., ).
While previous research focused on improving either the retriever (Karpukhin et al., ; Izacard et al., ; Wang et al., ) or the generator (Dong et al., ; Liu et al., ; Agarwal et al., ) in isolation, we take a holistic approach.
Conducting comprehensive analyses of the entire RAG system, we focus on the challenges and opportunities presented by using long-context LLMs as generators. We propose novel solutions to better employ them in long-context RAG.

Increased computational resources and advancements in efficient training methods have pushed LLMs supporting longer inputs (Wang et al., ; Zhou et al., ).
While long-context LLMs (Reid et al., ) demonstrated impressive performance on benchmarks like "needle-in-the-haystack" (Kamradt, ) and RULER (Hsieh et al., ), these benchmarks often rely on random negative examples and do not accurately reflect the challenges posed by the "hard negatives" encountered in real-world RAG scenarios (Cuconasu et al., ).
Furthermore, existing studies on long-context LLMs in multi-document settings (Liu et al., ; Shi et al., ) often assume a single "golden" document and random negatives, which differs from the RAG context where multiple relevant passages and hard negatives may exist (Hsieh et al., ; Cuconasu et al., ).
Although some research has explored the relationship between RAG and long-context LLMs (Xu et al., ; Li et al., ; Lee et al., ), these works take different perspectives.
They mainly focus on studying the (1) trade-offs between RAG and long-context LLMs (Xu et al., ), (2) routers to manage RAG and long-context LLMs (Li et al., ), (3) and the potential for LLMs to replace retrieval entirely (Lee et al., ), while leaving long-context LLMs as generators in RAG under-explored.
We delve deeper into the potential benefits of long-context LLMs for RAG and investigate how to optimize these LLMs specifically for this application.

Previous research has explored adapting LLMs for RAG using instruction tuning (Zhang et al., ).
RetRobust (Yoran et al., ) fine-tunes LLMs with 1 retrieved relevant passage or random negative passage to make it robust to irrelevant passage.
RA-DIT (Lin et al., ) conducts dual instruction tuning to make the LLM more effectively leverage retrieved information and retriever provide results more aligned with LLM preference.
Self-RAG (Asai et al., ) introduces a framework to train a LM that dynamically retrieves passages, generates content, and evaluates the retrieved passages for improved performance.
RAFT (Zhang et al., ) trains the LLMs to improve their ability to answer questions in “open-book” in-domain settings.
More recently, RankRAG (Yu et al., ) tunes a LLM for the dual purpose of context ranking and answer generation in RAG.
InstructRAG (Wei et al., ) finetunes the LLM to generate self-synthesized rationales rather than directly answering the question.
However, these existing efforts primarily focus on tuning with a limited number of retrieved passages (typically fewer than 10) and do not fully leverage the potential of long-context LLMs.
This work aims to address this gap by specifically investigating how to optimize long-context LLMs for large-scale RAG, where the number of retrieved passages can be significantly higher.

## 3 Challenges of Long context LLMs in RAG

We present a systematic investigation into the challenges of utilizing long-context LLMs in RAG.
Each subsection focuses on a specific research question, outlining corresponding experiments and analyzing the results on the key challenges.
These insights inform the development of targeted solutions for improving RAG performance with long-context LLMs, which are presented in subsequent sections.

### 3.1 The Effect of retrieved context size on RAG performance

This subsection investigates the relationship between the number of retrieved passages and the performance of long-context LLMs in RAG systems.

Research question.
Long-context LLMs offer the potential to incorporate more retrieved passages into RAG systems. This raises a crucial question:
Does a larger volume of retrieved context consistently translate to better performance when using long-context LLMs in RAG?

Experimental setting.
We evaluate the performance of RAG systems on the Natural Questions (NQ) (Kwiatkowski et al., ) dataset using two different retrievers (BM25 (Robertson et al., ) and e5 (Wang et al., ), where e5 exhibits higher performance on NQ ($\text{Recall}@40$ is 0.90 with e5 and 0.73 with BM25)) and four long-context LLMs (Gemma-7B-Chat (Team et al., ), Gemma-2-9B-Chat (Team et al., ), Mistral-Nemo-12B-Instruct (Jiang et al., ) and Gemini-1.5-pro (Reid et al., )).
We systematically vary the number of passages retrieved by each retriever.

Figure: (a) RAG performance with e5 retriever
Refer to caption: x1.png

Observations.
Figure 1 presents the following key observations:
1) Strong Retriever (e5): Across all LLMs, increasing the number of retrieved passages initially improves performance, but then leads to a sharp decline or plateau.
2) Weak Retriever (BM25): Performance generally exhibits a continuous increase or a slight decrease as the number of retrieved passages increases.
While these observations may appear counter-intuitive - given that one might expect monotonic improvements due to higher recall (i.e., a greater chance of retrieving relevant information) - the inclusion of additional documents can reduce precision, with irrelevant or misleading passages detracting LLMs from overall performance.
Comparison of different retrievers and the results on other datasets are shown in Appendix A and B.1.

Insights.
The effectiveness of increasing retrieved context size in RAG depends on the strength of the retriever.
With a strong retriever, performance exhibits an “inverted-U pattern”, while a weak retriever shows more consistent, albeit potentially limited, improvement.
This suggests that factors beyond simply the amount of retrieved information are at play.

### 3.2 The interplay of retrieval quality and LLM capabilities

This subsection delves into the factors hindering the performance of long-context LLMs in RAG, aiming to discern whether limitations arise from retrieval quality or the LLM’s ability to process the retrieved information.

Research question. Do the observed performance bottlenecks originate from limitations in the retriever’s ability to identify relevant information, or from the long-context LLM’s capacity to effectively utilize the retrieved context?

Experimental setting.
We analyze the relationship between RAG performance and retrieval quality, specifically recall and precision, using the Gemma-2-9B-Chat LLM with both e5 and BM25 retrievers (Figure 2).
Recall@k measures the presence of relevant passages within the top-k retrieved passages, while precision@k quantifies the proportion of relevant passages among them.

Figure: (a) Retrieval with e5 retriever
Refer to caption: x3.png

Observations.
Increasing the number of retrieved passages consistently leads to higher recall but lower precision, irrespective of the retriever used.
Crucially, the overall accuracy of the RAG system falls below the recall across all retrieval sizes.
This indicates that even when relevant information is present in the retrieved context, the LLM may fail to generate the correct answer.
This demonstrates that the irrelevant retrieved passages can sometimes mislead the LLM.
Furthermore, despite exhibiting higher precision, the e5 retriever leads to a more pronounced performance degradation as the number of retrieved passages increases compared to BM25.

Insights.
These observations yield two key insights:
(1) Influence of irrelevant passages: The discrepancy between retrieval recall and RAG accuracy underscores the detrimental effect of irrelevant retrieved passages ("hard negatives") on the LLMs’ performance.
Even when relevant information is available, the presence of hard negatives can mislead the LLMs and hinder their ability to generate accurate answers.
(2) Limitations of precision as a metric: The contrasting performance trends observed with e5 and BM25, despite the former’s higher precision, reveal that precision alone is an inadequate measure of retrieval quality in this context, when the end-to-end performance is considered.
The specific characteristics of the irrelevant passages, rather than just their quantity, significantly impact the LLMs’ performance. Retrievers might significantly differ in their way of priorization of them, and that might not be fully captured in metrics like precision.
In this scenario, it is observed that “hard negatives” retrieved by a stronger retriever (e5) might even more detrimental to the LLM than those retrieved by a weaker one (BM25).

### 3.3 The importance of hard negatives for long-context LLM evaluation

This subsection investigates the impact of "hard negatives" on the performance of long-context LLMs in RAG, highlighting the need for more robust evaluation methodologies.

Research question.
In long-context RAG scenarios, where a vast knowledge source necessitates retrieving numerous passages, the likelihood of including relevant information (i.e. obtaining high recall) increases.
However, this also elevates the risk of introducing hard negatives.
This raises two critical questions: (1) How robust are current long-context LLMs to these hard negatives? and (2) Does the impact of hard negatives vary with the retriever used?

Experimental setting.
This study investigates the effect of hard negative passages on long-context LLM performance in a controlled setting.
We tasked three LLMs (Gemma2-7B-Chat, Mistral-Nemo-12B-Instruct, and Gemini-1.5-Pro) with answering queries based on a context comprising a single golden passage and a varying number of hard negative passages retrieved using different methods (e5, Contriever, BM25, and random sampling).
This synthetic experiment, detailed in Figure 3, isolates the impact of hard negatives by holding the golden passage constant and intentionally excluding scenarios with multiple golden passages, which are common in real-world RAG systems.
See Appendix C for a complete illustration of the experimental setup.

Figure: (a) Retrievers
Refer to caption: x5.png

Observations.
(1) Sensitivity to hard negatives: Across all LLMs, increasing the number of hard negative passages generally leads to a decline in RAG answer accuracy.
(2) Retriever strength and hard negative difficulty: The strength of the retriever directly correlates with the difficulty of the retrieved hard negatives. LLMs struggle more with hard negatives from stronger retrievers (e.g., e5) compared to those from weaker retrievers (e.g., BM25) or random sampling.
(3) Distinguishing random and hard negatives: While Gemini-1.5-Pro demonstrates robustness to random negatives, it remains susceptible to the influence of hard negatives.
More results on other datasets and qualitative studies can be found in Appendix B.2 and D.

Insights.
Existing benchmarks for evaluating long-context LLMs, such as "needle-in-the-haystack" (Kamradt, ) and RULER (Hsieh et al., ), predominantly utilize random negatives.
Our findings demonstrate that such benchmarks may not adequately capture the challenges posed by hard negatives, which are prevalent in real-world RAG applications. Their takeaways would have limitations.
The need for new evaluation methodologies that incorporate hard negatives (specific to the employed retrievers) is highlighted, to provide a more comprehensive and realistic assessment of long-context LLM performance in RAG.

## 4 Simple and effective training-free RAG improvement

Building upon the analyses in Section 3 on the detrimental impact of hard negatives on long-context LLMs in RAG, we focus on the training-free solution, retrieval reordering.
This method leverages the inherent "lost-in-the-middle" phenomenon observed in LLMs to mitigate the negative effects of hard negatives.
As highlighted by Liu et al. (), LLMs exhibit a tendency to prioritize information presented at the beginning and end of an input sequence, while paying less attention to the middle.

Exploiting this "lost-in-the-middle" behavior, we consider a simple and effective strategy: reordering the retrieved passages based on their relevance scores calculated by the retriever.
Given a query $q$ and a set of retrieved passages $d_{1}$, $d_{2}$, …, $d_{k}$ with decreasing relevance scores, the standard input sequence construction for an LLM with instruction $I$ would be: $[I,d_{1},d_{2},...,d_{k-1},d_{k},q]$.
Retrieval reordering modifies this to prioritize passages with higher scores at the beginning and end: $[I,d_{1},d_{3},...,d_{4},d_{2},q]$
where the position of passage $d_{i}$ is determined by

$$ $\textit{Order}(d_{i})=\begin{cases}\frac{i+1}{2}&\text{if mod($i$, 2) = 1}\\ (k+1)-\frac{i}{2}&\text{if mod($i$, 2) = 0}\end{cases}$ (1) $$

This reordering strategy aims to guide the LLM’s attention towards the most relevant passages, thereby reducing the influence of hard negatives positioned in the middle of the sequence. The pseudo-code for retrieval reordering can be found in Appendix E.

Figure: (a) NQ: Gemma2+e5
Refer to caption: x9.png

Retrieval reordering significantly improves RAG performance, particularly with larger numbers of retrieved passages.
To assess the effectiveness of retrieval reordering, we conduct experiments with two retrievers (e5 and BM25), two long-context LLMs (Gemma-2-9B-Chat and Mistral-Nemo-12B-Instruct), and two datasets (NQ and PopQA).
As illustrated in Figure 4, retrieval reordering yields negligible improvements with smaller retrieval sets, but significantly and consistently outperforms the original ordering when the number of retrieved passages is large.
This behavior is attributed to the interplay of two factors that become increasingly significant with larger retrieval sets: (1) the amplified "lost-in-the-middle" phenomenon, where LLMs prioritize information at the beginning and end of the input sequence, and (2) the increased prevalence of hard negatives, which can hinder accurate answer generation.
By strategically placing passages, retrieval reordering mitigates these issues, highlighting the potential of position engineering as a complementary technique to prompt engineering for optimizing long-context LLMs in RAG.

## 5 Improving Robustness for RAG via Data-Augmented Fine-Tuning

### 5.1 Implicitly improving LLM robustness through fine-tuning

While the retrieval reordering strategy presented in Section 4 mitigates the detrimental impact of hard negatives, it does not inherently enhance the LLM’s ability to handle such irrelevant information within the context.
To address this, we conduct a systematic investigation into RAG-specific tuning as a means of improving long-context LLMs for RAG applications.

Our tuning paradigm involves training LLM to generate the correct answer ($a$) given a comprehensive input comprising an instruction ($I$), a query ($q$), and a set of retrieved passages $(d_{1},d_{2},...,d_{k})$:

$$ $\text{Input:}\ [I,d_{1},d_{2},...,d_{k-1},d_{k},q]\longrightarrow\text{Output: }\ a.$ (2) $$

This approach aims to implicitly enhance the LLM’s robustness to hard negatives by exposing it to a diverse range of retrieved contexts during fine-tuning, thus enabling it to learn to effectively identify and utilize relevant information even in the presence of noise.

Figure: (a) TriviaQA
Refer to caption: x17.png

To assess the generalization capabilities of RAG-tuned LLMs, we fine-tune Gemma-2-9B-Base, Mistral-Nemo-12B-Base and Gemini-1.0-Pro using a diverse dataset comprising NQ, WoW, Fever, and MMLU.
We then evaluate on a range of unseen datasets, including TriviaQA, PopQA, HotpotQA, 2wikimultihopqa, Webquestions, Bamboogle, ASQA, T-REx, and zsRE.
We compare the performance of the RAG-tuned model (RAG FT) with two types of baselines: (1) Chat model with retrieval augmentation: the Gemma-2-9B-Chat/Mistral-Nemo-12B-Instruct/Gemini-1.0-Pro w. RAG; (2) Direct SFT: the ones fine-tuned with standard supervised fine-tuning (SFT) on question-answer pairs without retrieved context (Direct FT w/o RAG). Further details regarding the datasets and experimental setup can be found in Appendix F and G.

Figure 5 shows the three key observations:
(1) Consistent improvement over baselines: RAG FT consistently outperforms the chat model w. RAG and the Direct FT model across all evaluated datasets.
(2) Robustness to hard negatives: the curve of RAG FT is generally flatter than that of the chat model, which demonstrates that our finetuned LLM is more robust to the hard negatives as the number of retrieved passages increases.
(3) Superiority over direct fine-tuning: In most cases, RAG FT demonstrates superior performance compared to Direct FT. This indicates that RAG FT not only enables the LLM to "memorize" knowledge during training but also equips it with the ability to effectively "extract" relevant information from retrieved context during inference.
These findings highlight the effectiveness of RAG-specific tuning in enhancing the generalization capabilities of LLMs for knowledge-intensive tasks.
Separate results on those three LLMs are shown in Appendix J, K and L.
Qualitative studies can be found in Appendix I.

### 5.2 Enhancing relevance identification through reasoning augmentation

While the fine-tuning approach described in Section 5.1 implicitly enhances the LLM’s robustness to hard negatives, it does not explicitly train the model to differentiate between relevant and irrelevant passages within the retrieved context.
To address this, we investigate the effectiveness of incorporating an intermediate reasoning step into the fine-tuning process.

This modified paradigm involves training the LLM to generate both a reasoning paragraph ($r$) that explicitly identifies the relevant passages for the given query ($q$) and the final answer ($a$):

$$ $\text{Input:}\ [I,d_{1},d_{2},...,d_{k-1},d_{k},q]\longrightarrow\text{Output: }\ [r,a],$ (3) $$

During training, the LLMs are provided with labeled reasoning paragraphs to guide its learning process.
During inference, the LLMs are instructed to first generate the reasoning paragraph and then utilize this analysis to produce the answer.
This approach aims to explicitly enhance the LLMs’ ability to discern relevant information from noise within the retrieved context, thereby improving its overall performance in RAG.

We utilize the same training data mixture as in Section 5.1 and augment it with reasoning labels generated by Gemini-1.5-Pro for each question-passage pair. These labels provide explicit guidance on identifying relevant passages. Further details of the experimental setup and the generation of reasoning labels can be found in Appendix H.

Figure: (a) TriviaQA
Refer to caption: x26.png

Figure 6 demonstrates the effectiveness of this approach. The LLM fine-tuned with explicit intermediate reasoning consistently outperforms training with implicit RAG data. This improvement can be attributed to two key factors:
(1) Explicit relevance training: Providing intermediate reasoning labels during training explicitly teaches the LLM to differentiate between relevant and irrelevant passages, enhancing its ability to discern crucial information from noise.
(2) Structured reasoning for enhanced understanding: Generating a reasoning paragraph before answering introduces a structured approach to processing the retrieved context. This step, akin to chain-of-thought reasoning (Wei et al., ), helps decouple the complex information and facilitates a more focused analysis, ultimately leading to improved performance.
These highlight the value of incorporating explicit reasoning mechanisms in RAG tuning to enhance the LLM’s ability to effectively utilize retrieved context.
More results on Gemma-2-9B models, Mistral-Nemo-12B models and Gemini-1.0-Pro models are shown in Appendix J, K and L.
Qualitative studies can be found in Appendix I.

## 6 Data-Centric Perspectives on Fine-tuning LLMs for RAG

Impact of training data distribution on generalization.
We first examine how the distribution of training data affects the generalization of the fine-tuned LLM.
We train LLMs on five different data distributions, each with 50k samples: (1) a mixed dataset comprising NQ, WoW, Fever, and MMLU (12.5k samples from each); (2) NQ only; (3) WoW only; (4) Fever only; and (5) MMLU only.

Figure 7(a) demonstrates that a mixed distribution of training data leads to superior generalization performance on unseen RAG tasks compared to training on a single data source.
This highlights the importance of data diversity in enhancing the adaptability of LLMs to new RAG scenarios.

Influence of retrievers on generalization.
In real-world RAG deployments, LLMs might be paired with different retrievers depending on specific external knowledge corpus and retrievers’ capabilities. To investigate the impact of different retrievers on fine-tuning, we explore three adaptation scenarios on NQ: fine-tuning with (1) passages retrieved by BM25 (FT w. BM25); (2) passages retrieved by e5 (FT w. e5); and (3) mixture of passages retrieved by both BM25 and e5 (FT w. mix).
We evaluate the performance of these fine-tuned LLMs using both retrievers seen during training (BM25 and e5) and unseen retrievers (Contriever (Izacard et al., ) and BGE (Chen et al., )).

Figure 7(b) presents the results, revealing two key findings:
(1) Superiority of mixed retriever training: Fine-tuning with the data corresponding to a mix of retrievers consistently yields the best performance across both seen and unseen retrievers during inference. This suggests that training on a diverse set of retrieved passages enhances the LLMs’ ability to adapt to different retrieval strategies and knowledge sources.
(2) Retriever similarity and generalization: The generalization ability of an LLM fine-tuned with a specific retriever is influenced by the similarity between the training retriever and the inference retriever. For instance, an LLM trained with BM25 generalizes better to Contriever, while an LLM trained with e5 generalizes better to BGE. This observation suggests that "hard negatives" exhibit different characteristics depending on the employed retriever, and training with a specific retriever implicitly equips the LLM to better handle similar types of hard negatives. See Appendix A for a detailed analysis of retriever similarity.

Figure: (a) Analysis of training data distribution. (Test: HotpotQA)
Refer to caption: x32.png

Optimizing training for variable retrieval sizes.
In real-world RAG systems, the number of retrieved passages can vary depending on the specific knowledge source and user requirements.
Therefore, it is essential to determine the optimal training strategy for LLMs to ensure robust performance across different retrieval sizes during inference.
We investigate this aspect with the Gemma-2-9B-Base model, which has a maximum input sequence length of 8192 tokens (corresponding to approximately 40 passages).
We evaluate five different training configurations:
(1) Fixed 10 retrieved passages (25% max).
(2) Fixed 20 retrieved passages (50% max).
(3) Fixed 40 retrieved passages (maximum input capacity) (100% max).
(4) Dynamic 0-40 retrieved passages (0-100% max).
(5) Dynamic 20-40 retrieved passages (50-100% max).

Figure 7(c) presents the results on NQ, demonstrating that fine-tuning with the maximum number of retrieved passages (100% max) consistently yields the best performance across various retrieval sizes during inference.
This suggests that training with the full context capacity enhances the LLM’s ability to effectively handle varying amounts of retrieved information, leading to improved generalization and robustness.
More analyses of RAG-specific tuning can be found in in Appendix M and N.

## 7 Conclusions

This paper investigates the impact of increasing the number of retrieved passages on the performance of long-context LLMs in retrieval-augmented generation (RAG) systems.
Contrary to expectations, we observe that performance initially improve but then degrade as more passages are included.
This phenomenon is attributed to the detrimental influence of retrieved "hard negatives".
To mitigate this issue, we propose and evaluate three solutions: training-free retrieval reordering, RAG-specific implicit LLM fine-tuning, and RAG-oriented LLM fine-tuning with intermediate reasoning.
A systematic analysis of the training-based methods explores the effects of data distribution, retriever for training, and training context length.
Interesting future directions include exploring (automated) position optimization with more advanced retrieval ordering methods, and fine-tuning the LLMs for RAG with more fine-grained and multi-step reasoning chains.

*

## Appendix

## Appendix A Retriever performance and similarity

We analyze the performance and similarity of four retrievers (BM25, contriever, e5 and bge) on the NQ dataset shown in Figure 8.
Each data point corresponds to a retrieval (recall, precision) pair for a specific number of retrieved passages.
The overall retrieval performances on NQ are observed as e5 > bge > contriever > bm25, with contriever having a similar performance with BM25 and bge having a similar performance with e5 (as their curves are closer).

Figure: Figure 8: Retriever performance on NQ. (1) Retrieval performance: e5 > bge > contriever > BM25; (2) Contriever is more similar to BM25, while bge is more similar to e5 (since their curves are closer respectively).
Refer to caption: x35.png

## Appendix B Long context LLMs in RAG analysis on other datasets

In addition to the analysis presented on the NQ dataset in Section 3, we conduct further studies on the PopQA dataset to underscore the generality of our findings.

### B.1 The Effect of retrieved context size on RAG performance

Figure: (a) RAG performance with e5 retriever
Refer to caption: x36.png

Observations.
Figure 9 presents the following key observations similar to that in Section 3.1:
1) Strong Retriever (e5): Across all LLMs, increasing the number of retrieved passages initially enhances performance, but subsequently results in either a sharp decline or a plateau.
2) Weak Retriever (BM25): Performance generally shows a continuous improvement or a slighter decrease as the number of retrieved passages increases.
While these observations may appear counter-intuitive - given that one might expect monotonic improvements due to higher recall (i.e., a greater chance of retrieving relevant information) - the inclusion of additional documents can reduce precision, with irrelevant or misleading passages detracting LLMs from overall performance.

### B.2 The importance of hard negatives for long-context LLM evaluation

Figure: (a) Retrievers
Refer to caption: x38.png

Observations. Figure 10 shows the following observations similar to that in Section 3.3:
(1) Sensitivity to hard negatives: Across all LLMs, increasing the number of hard negative passages generally results in a decline in RAG answer accuracy.
(2) Retriever strength and hard negative difficulty: The strength of the retriever is directly correlated with the difficulty of the retrieved hard negatives. LLMs struggle more with hard negatives generated by stronger retrievers (e.g., e5) compared to those produced by weaker retrievers (e.g., BM25) or through random sampling.
(3) Distinguishing random and hard negatives: While all the LLMs demonstrates robustness to random negatives, it remains susceptible to the influence of hard negatives.

## Appendix C Illustration of Section 3.3 : Hard negative study

Figure: Algorithm 1 Data Construction for Hard Negative Study

## Appendix D Hard negatives case study

In this section, we provide a case study to compare the hard negatives returned by different retrievers.
We classify the negative passages into two types: (1) Related but Irrelevant: passages related to some entities mentioned in the question but not containing the ground truth answer; (2) Not Related: passages not related to the question at all.
Note that Related but Irrelevant passages are harder and more misleading to the LLMs compared with Not Related passages.
We show the top-5 negatives from each retriever for a random sampled question as below.
From the case study, we can find that the negatives retrieved by e5 contain more Related but Irrelevant passages compared with those retrieved by bm25, while those retrieved by bm25 still have more Related but Irrelevant passages than random sampling.
This qualitatively demonstrates that the hardness of the negatives from different retrievers as e5 > bm25 > random.

| Question | The south west wind blows across Nigeria between? |
| --- | --- |
| Ground Truth | Till September |
| Retrieved Hard Negative Passages (high retrieval score but lacking ground truth answer) |  |
| w. e5 | Doc 1 (Title: "Geography of Nigeria") … south atlantic ocean, locally known as the south western wind, or by its main name, The Tropical Maritime (MT) airmass. These two major wind systems in Nigeria are known as the trade winds. The tropical maritime airmass (MT) is responsible for Nigeria’s rainy season. This wind (the tropical maritime airmass) invades the country from February in the southern part of Nigeria while it takes longer for the wind to fully cover the whole of the country, reaching the northern part of Nigeria in June. Its invasion is as a result of the northward retreat, … [Related but Irrelevant]<br>Doc 2 (Title: "Onikwu") … The dry season is accompanied by a dust laden airmass from the Sahara Desert, locally known as Harmattan, or by its main name, The Tropical Continental (CT) airmass, while the rainy season is heavily influenced by an airmass originating from the south Atlantic Ocean, locally known as the south west wind, or by its main name, The Tropical Maritime (MT) airmass. These two major wind systems in Nigeria are known as the trade winds. The region Onikwu/Ndoni is flood prone communities, this is because the inland part of Rivers state consists of tropical rainforest … [Related but Irrelevant]<br>Doc 3 (Title: "Geography of Nigeria") … northern end is south of the 15 degrees line at about 14 degrees. Nigeria’s location in the wetter part of the easterly waves south of the 15 degree line creates wetter climatic conditions for Nigeria especially during the monsoons. The Tropical Continental Airmass (CT) locally known as the harmattan, is a wind originating from North Africa which crosses the Sahara Desert into west Africa to Nigeria. This airmass dominates Nigeria’s climate during the dry season from December to March. The Tropical continental airmass is dusty and creates a haze within the atmosphere of west Africa and Nigeria when it predominates. … [Related but Irrelevant]<br>Doc 4 (Title: "Nigeria") … Niger, Chad, Cameroon, and has a coastline of at least s. Nigeria lies between latitudes 4 and 14N, and longitudes 2 and 15E. The highest point in Nigeria is Chappal Waddi at . The main rivers are the Niger and the Benue, which converge and empty into the Niger Delta. This is one of the world’s largest river deltas, and the location of a large area of Central African mangroves. Nigeria has a varied landscape. The far south is defined by its tropical rainforest climate, where annual rainfall is a year. In the southeast stands the … [Related but Irrelevant] |

|  | Doc 5 (Title: "Geography of Nigeria") … Nigeria, has a temperature range of to , and an annual rainfall of about with a single rainfall maxima in September. The single Dry season experienced in this climate, the tropical savanna climate in central Nigeria beginning from December to march, is hot and dry with the Harmattan wind, a continental tropical (CT) airmass laden with dust from the Sahara Desert prevailing throughout this period. With the Intertropical Convergence Zone (ITCZ) swinging northward over West Africa from the Southern Hemisphere in April, heavy showers coming from pre-monsoonal convective clouds mainly in the form of … [Related but Irrelevant] |
| --- | --- |
| w. bm25 | Doc 1 (Title: "Oron people") … Civil War. Oron is found in the flood plain of South Eastern Nigeria, with the land mainly intersected by numerous streams and tributaries flowing into Cross River. The entire coastline stretches from Uya Oron to Udung Uko. Oron is in the tropical region and has a uniformly high temperature all the year round. The two main seasons are the dry which spans between October and April and wet season which starts around May and ends in September. There are also two prevailing winds – the South-West onshore winds which brings heavy rains and the … [Not Related]<br>Doc 2 (Title: "South Equatorial Current") … is driven directly by the trade winds which blow from east to west. In the Indian Ocean, the westward-flowing South Equatorial Current is well-developed only south of the equator. Directly on the equator, the winds reverse twice a year due to the monsoons, and so the surface current can be either eastward or westward. South Equatorial Current Ocean current in the Pacific, Atlantic, and Indian Ocean that flows east-to-west between the equator and about 20 degrees south. In the Pacific and Atlantic Oceans, it extends across the equator to about 5 degrees north. Within the southern hemisphere, the South Equatorial … [Related but Irrelevant]<br>Doc 3 (Title: "Wind direction") … Wind direction Wind direction is reported by the direction from which it originates. For example, a ""northerly"" wind blows from the north to the south. Wind direction is usually reported in cardinal directions or in azimuth degrees. Wind direction is measured in degrees clockwise from due north. Consequently, a wind blowing from the north has a wind direction of 0; a wind blowing from the east has a wind direction of 90; a wind blowing from the south has a wind direction of 180; and a wind blowing from the west has a wind direction of 270 … [Related but Irrelevant]<br>Doc 4 (Title: "Gulf Stream") … this current interacts with the northeastern coast of South America, the current forks into two branches. One passes into the Caribbean Sea, while a second, the Antilles Current, flows north and east of the West Indies. These two branches rejoin north of the Straits of Florida. The trade winds blow westward in the tropics, and the westerlies blow eastward at mid-latitudes. This wind pattern applies a stress to the subtropical ocean surface with negative curl across the north Atlantic Ocean. The resulting Sverdrup transport is equatorward. Because of conservation of potential vorticity caused by the northward-moving winds on the subtropical … [Not Related]<br>Doc 5 (Title: "Climate of the United Kingdom") … climate that western parts of the UK experience. The high latitude and proximity to a large ocean to the west means that the United Kingdom experiences strong winds. The prevailing wind is from the south-west, but it may blow from any direction for sustained periods of time. Winds are strongest near westerly facing coasts and exposed headlands. Gales — which are defined as winds with speeds of — are strongly associated with the passage of deep depressions across the country. The Hebrides experience on average 35 days of gale a year (a day where there are gale-force winds) while inland … [Not Related] |
| w. random sampling | Doc 1 (Title: "Queen of Peace, Bray") … of Bugisi in Tanzania for a number of years. The parish has a very close relationship with St Cronan’s B.N.S., Scoil Chualann and Gaelscoil Uí Chéadaigh. The parish provides the sacraments of Communion and Confirmation to the children in the schools. They also help to raise funds for the twin parish of Bugisi. Queen of Peace, Bray The Queen of Peace is a Catholic church situated at the junction of the Putland Road and the Vevay Road in Bray, Co. Wicklow, Ireland. The present church was built in 1946 by TJ Macken of St Patrick’s Street, Dún Laoghaire, … [Not Related] |

|  | Doc 2 (Title: "Cordova Congressional Internship Program") … Puerto Rico’s Constitutional Convention from 1951 to 1952. By 2012, over 670 students from colleges and universities in Puerto Rico had enjoyed internships under the program, and the Spring 2009 class included a record 24 members. A private sector committee, recently headed by Univision Puerto Rico president Larry Sands, provides private funds to supplement the 350,000 annual grant provided by the Puerto Rico Legislative Assembly. Under the auspices of TWC, seventeen states have since established similar legislative-funded Congressional internship programs. The Center established in 2008 the McClintock Award to the State Legislator of the Year … [Not Related]<br>Doc 3 (Title: "V bomber") … Puerto Rico’s Constitutional Convention from 1951 to 1952. By 2012, over 670 students from colleges and universities in Puerto Rico had enjoyed internships under the program, and the Spring 2009 class included a record 24 members. A private sector committee, recently headed by Univision Puerto Rico president Larry Sands, provides private funds to supplement the 350,000 annual grant provided by the Puerto Rico Legislative Assembly. Under the auspices of TWC, seventeen states have since established similar legislative-funded Congressional internship programs. The Center established in 2008 the McClintock Award to the … [Not Related]<br>Doc 4 (Title: "Defence Materials and Stores Research and Development Establishment") … materials for the Indian Armed Forces. DMSRDE has developed Nuclear Shielding Pad, Boot Anti Mine, Blast Protection Suit, Bullet Proof Jackets, etc.. ""The Defence Material and Stores Research Development Establishment in Kanpur has developed a new NBC suit that would be proved effective against any kind of dangerous weapons or chemicals and protect soldiers from any sort of attack,"" DMSRDE Director Arvind Kumar Saxena was quoted by media-persons. 40,000 pieces of NBC suits costing about Rs 30,000 had been requested by Indian army. ""the further progress on the other two suits are going on."" further … [Not Related]<br>Doc 5 (Title: "Chess title") … retain the title of Candidate Master, if it was earned according to criteria above). This is in contrast to international titles awarded by FIDE, which are awarded for life. In European countries the term of ""expert"" is not used. Instead, players of that level are called ""Candidate Masters"", although the FIDE Candidate Master title generally requires a higher rating (2200 FIDE). It is possible (and common), however, for players in the United States to have a rating that places them in the ’expert’ category while still retaining the title of ’Life Master’ or ’National Master’ … [Not Related] |
| --- | --- |

## Appendix E Retrieval reordering

Figure: Algorithm 2 Retrieval Reordering Algorithm

## Appendix F Datasets

In this section, we discuss the datasets for RAG-specific LLM training and evaluation.

### F.1 Training datasets

**Table 1: Training data statistics.**
| Dataset | the number of instances |
| --- | --- |
| Natural Question | 12,500 |
| Wizard of Wikipedia | 12,500 |
| FEVER | 12,500 |
| MMLU | 12,500 |

We select a series of fine-tuning data designed to enhance the model’s robustness to hard negatives in the retrieval context and improve its contextual awareness in generating predictions.
The training data are from four sources with different answer types: Natural Question (short-form), Wizard of Wikipedia (long-form), FEVER (true/false), and MMLU (close-set).
The statistics of the training data mix can be found in Table 1.

### F.2 Testing datasets

To comprehensively evaluate our methods, we select testing datasets across different tasks including: (1) Question-answering: TriviaQA, PopQA, WebQuestions; (2) Multi-hop tasks: HotpotQA, 2WikiMultiHopQA, Bamboogle; (3) Long-form tasks: ASQA; (4) Slot filling: T-REx, Zero-shot RE.
The statistics of all the datasets can be found in Table 2.

**Table 2: Testing data statistics.**
| Dataset | Task | the number of instances |
| --- | --- | --- |
| TriviaQA | QA | 11,313 |
| PopQA | QA | 14,267 |
| WebQuestions | QA | 2,032 |
| HotpotQA | Multi-Hop QA | 7,405 |
| 2WikiMultiHopQA | Multi-Hop QA | 12,576 |
| Bamboogle | Multi-Hop QA | 125 |
| ASQA | Long-form QA | 948 |
| T-REx | Slot filling | 5,000 |
| Zero-shot RE | Slot filling | 3,724 |

### F.3 Retrieval corpus

Following Karpukhin et al. (), we use the text chunks from 2018 Wikipedia dump as the retrieval corpus.
The articles are split by section, where long sections are further split into text chunks of equal sizes and contain less than 100 words, leading to a total of 21M text chunks.

## Appendix G Implicit RAG Fine-Tuning Experimental Setting

### G.1 Training settings

Hyperparameters.
We use the top-40 retrieved text chunks for a given example to generate the fine-tuning samples and use e5 as the retriever for the main results.
We fine-tune both Gemma-2-9B-Base and Mistral-Nemo-12B-Base using 8 H100 GPUs.
For both models, we use the chat template corresponding to Gemma-2-9B-Chat and Mistral-Nemo-12B-Instruct respectively when tuning the models.
We use the axolotl(^1^11[https://github.com/axolotl-ai-cloud/axolotl](https://github.com/axolotl-ai-cloud/axolotl)) codebase for their tuning.
For Gemini-1.0-Pro tuning, we use the Google Cloud Tuning API(^2^22[https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/tuning](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/tuning)) with the default settings.
The hyperparameters can be found in Table 3.

**Table 3: Implicit RAG finetuning hyperparameters.**
| Model | peak lr | lr scheduler | warm up | # epoch | batch size | Flash Att |
| --- | --- | --- | --- | --- | --- | --- |
| Gemma-2-9B-Base | 1e-6 | cosine | 5% | 4 | 64 | False |
| Mistral-Nemo-12B-Base | 1e-6 | cosine | 5% | 4 | 64 | True |
| Gemini-1.0-Pro | default | default | default | 1 | default | default |

Training RAG instruction templates. The RAG instruction templates for different training datasets can be found in Table 4.

**Table 4: Training instruction templates for implicit RAG tuning.**
| Task | Instruction Templates |
| --- | --- |
| NQ | Answer the question based on the given document. |
|  | Only give me the answer and do not output any other words. |
|  | The following are given documents. |
|  | {reference} |
| Wizard of Wikipedia | Provide a response to the conversation based on the given document. |
|  | The following are given documents. |
|  | {reference} |
| FEVER | Verify a fact based on the given documents |
|  | The following are given documents. |
|  | {reference} |
| MMLU | Given a question, choose the answer from the options based on the given documents. |
|  | The following are given documents. |
|  | {reference} |

Training RAG answer templates. The RAG answer templates for different training datasets can be found in Table 5.

**Table 5: Training answer templates for implicit RAG tuning.**
| Task | Answer Templates |
| --- | --- |
| NQ | Question: {question} |
|  | Answer: |
| Wizard of Wikipedia | Conversation: {question} |
|  | Response: |
| FEVER | Return SUPPORTS if it is correct and return REFUTES if it is not correct. |
|  | Fact: {question} |
|  | Response: |
| MMLU | Question: {question} |
|  | Options: {choices} |
|  | Answer: |

### G.2 Evaluation Settings

Hyperparameters. For all the compared LLMs, we conduct top-p sampling (p = 1) and the maximum number of generated token is set to be 32.
For Gemma-2 series models, we use the huggingface inference pipeline(^3^33[https://huggingface.co/docs/transformers/](https://huggingface.co/docs/transformers/)).
For Gemini series models, we use Google Cloud Inference API(^4^44[https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference)).
While for other series of LLMs, we utilize vLLM(^5^55[https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)) codebase for efficient generation.

Evaluation RAG instruction templates. The RAG instruction templates for different testing datasets can be found in Table 6.

**Table 6: Testing instruction templates for implicit RAG tuning.**
| Task | Instruction Templates |
| --- | --- |
| QA | Answer the question based on the given document. |
|  | Only give me the answer and do not output any other words. |
|  | The following are given documents. |
|  | {reference} |
| Multi-hop | Answer the question based on the given document. |
|  | Only give me the answer and do not output any other words. |
|  | The following are given documents. |
|  | {reference} |
| Long-form | Answer the question based on the given document. |
|  | Please give in-depth explanation and avoid only returning the answer. |
|  | The following are given documents. |
|  | {reference} |
| Slot filling | Given a question, choose the answer from the options based on the given documents. |
|  | The following are given documents. |
|  | {reference} |

Evaluation RAG answer templates. The RAG answer templates for different testing datasets are all: "Question: {question}. Answer:"

## Appendix H RAG Finetuning with Intermediate Reasoning Experimental Setting

### H.1 Training settings

Hyperparameters. The hyperparameter setting is the same to that in Appendix G.1.

Training RAG instruction templates. The RAG instruction templates with intermediate reasoning for different training datasets can be found in Table 7.

**Table 7: Training instruction templates for RAG tuning with intermediate reasoning.**
| Task | Instruction Templates |
| --- | --- |
| NQ | Answer the question based on the given documents. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to answer the question. |
|  | Then output a concise answer to the question based on the analysis. |
|  | The following are given documents. |
|  | {reference} |
| Wizard of Wikipedia | Provide a response to the conversation based on the given documents. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to provide the response. |
|  | Then output a concise response to the question based on the analysis. |
|  | The following are given documents. |
|  | {reference} |
| FEVER | Verify a fact based on the given documents. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to verify the fact. |
|  | Then output a concise answer (SUPPORTS or REFUTES) based on the analysis. |
|  | The following are given documents. |
|  | {reference} |
| MMLU | Given a question, choose the answer from the options based on the given documents. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to answer the question |
|  | Then output a concise answer to the question based on the analysis. |
|  | The following are given documents. |
|  | {reference} |

Training RAG Answer templates. The RAG answer templates for different training datasets can be found in Table 8.

**Table 8: Training answer templates for RAG tuning with intermediate reasoning.**
| Task | Answer Templates |
| --- | --- |
| NQ | Question: {question} |
|  | Answer: |
| Wizard of Wikipedia | Conversation: {question} |
|  | Response: |
| FEVER | Return SUPPORTS if it is correct and return REFUTES if it is not correct. |
|  | Fact: {question} |
|  | Response: |
| MMLU | Question: {question} |
|  | Options: {choices} |
|  | Answer: |

Instructions to generate intermediate reasoning from Gemini-1.5-pro. The prompt that guides Gemini-1.5-pro for intermediate reasoning generation can be found in Table 9.

**Table 9: Prompts to guide Gemini-1.5-pro for intermediate reasoning generation.**
| Task | Prompts |
| --- | --- |
| NQ | Read the following documents relevant to the given question: {question} |
|  | {reference} |
|  | Please identify documents that are useful to answer the given question: {question}, |
|  | and explain how the contents lead to the answer: {answers}. |
|  | If none of the documents is aligned with the answer, |
|  | in that case, you have to explain the answer only based on your own knowledge, |
|  | without referring to the provided information. |
|  | Note that the question may be compositional and require intermediate analysis to deduce the final answer. |
|  | Make sure your response is grounded and provides clear reasoning details followed by a concise conclusion. |
| Wizard of Wikipedia | Read the following documents relevant to the given conversation: {question} |
|  | {reference} |
|  | Please identify documents that are useful to provide a response to a conversation: {question} |
|  | and explain how the contents lead to the response: {answers}. |
|  | If none of the documents is aligned with the response, |
|  | in that case, you have to explain the response only based on your own knowledge, |
|  | without referring to the provided information. |
|  | Make sure your response is grounded and provides clear reasoning details followed by a concise conclusion. |
| FEVER | Read the following documents relevant to the given question: {question} |
|  | {reference} |
|  | Please identify documents that are useful to verify a fact: {question} |
|  | (Return SUPPORTS if it is correct and return REFUTES if it is not correct.), |
|  | and explain how the contents lead to the answer: {answers}. |
|  | If none of the documents is aligned with the answer, |
|  | in that case, you have to explain the answer only based on your own knowledge |
|  | without referring to the provided information. |
|  | Make sure your response is grounded and provides clear reasoning details followed by a concise conclusion. |
| MMLU | Read the following documents relevant to the given question: {question} |
|  | {reference} |
|  | Please identify documents that are useful to answer the given question: {question} with options: {choices}, |
|  | and explain how the contents lead to the answer: {answers}. |
|  | If none of the documents is aligned with the answer, |
|  | in that case, you have to explain the answer only based on your own knowledge, |
|  | without referring to the provided information. |
|  | Make sure your response is grounded and provides clear reasoning details followed by a concise conclusion. |

### H.2 Evaluation settings

Hyperparameters. For all the compared LLMs, we conduct top-p sampling (p = 1) and the maximum number of generated token is set to be 256. For Gemma-2 series models, we use the huggingface inference pipeline. While for other series of LLMs, we utilize vLLM codebase for efficient generation.

Evaluation RAG instruction templates. The RAG instruction templates for different testing datasets can be found in Table 10.

**Table 10: Testing instruction templates for RAG tuning with intermediate reasoning.**
| Task | Instruction Templates |
| --- | --- |
| QA | Answer the question based on the given documents. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to answer the question |
|  | Then output a concise answer to the question based on the analysis. |
|  | The following are given documents. |
|  | {reference} |
| Multi-hop | Answer the question based on the given documents. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to answer the question |
|  | Then output a concise answer to the question based on the analysis. |
|  | The following are given documents. |
|  | {reference} |
| Long-form | Answer the question based on the given document. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to answer the question |
|  | Then provide an in-depth long-form answer for the question (avoid only returning the answer) based on the analysis. |
|  | The following are given documents. |
|  | {reference} |
| Slot filling | Provided an answer to the given slot filling question based on the given document. |
|  | In the question, the words before and after [SEP] correspond to the head entity and relation respectively. |
|  | You are asked to output the tail entity corresponded to the given head entity and relation. |
|  | Please first provide an analysis with clear reasoning details of which documents are relevant to answer the question. |
|  | Then output a concise answer to the question based on the analysis. |
|  | The following are given documents. |
|  | {reference} |

Evaluation RAG answer templates. The RAG answer templates for different testing datasets are all: "Question: {question}. Answer:"

## Appendix I Data-Augmented RAG Case Studies

| Question | Which film features the Dawes Tomes Mousley Grubbs Fidelity Fiduciary Bank? |
| --- | --- |
| Ground Truth | Mary Poppins |
| Retrieved Passages | Doc 1 (Title: "Fidelity Fiduciary Bank") Fidelity Fiduciary Bank ""Fidelity Fiduciary Bank"" is a song from Walt Disney’s film ""Mary Poppins"", and it is composed by Richard M. Sherman and Robert B. Sherman. The song sung by the stodgy old bankers at the ""Dawes, Tomes, Mousely, Grubbs Fidelity Fiduciary Bank"", led by the ""Elder Mr. Dawes"" (Nackvid Keyed), to George Banks’s two children, Jane and Michael, in an attempt to get Michael Banks to invest his tuppence in the bank. As the song continues the pressure is on George, a junior clerk at the bank, to sway Michael. When Michael finally, and ever so slightly, opens …<br>Doc 2 (Title: "Fidelity Fiduciary Bank") convince Michael to invest his money goes like this: has as its origins an essay by C. C. Turner titled ’Money London’ in the book edited by G. R. Sims called ""Living London"" (London: 1903): Fidelity Fiduciary Bank ""Fidelity Fiduciary Bank"" is a song from Walt Disney’s film ""Mary Poppins"", and it is composed by Richard M. Sherman and Robert B. Sherman. The song sung by the stodgy old bankers at the ""Dawes, Tomes, Mousely, Grubbs Fidelity Fiduciary Bank"", led by the ""Elder Mr. Dawes"" (Nackvid Keyed), to George Banks’s two children, Jane and Michael, in an attempt to get …<br>…<br>Doc 40 (Title: "The Firm (soundtrack)") The Firm (soundtrack) The Firm is an album by American pianist Dave Grusin released in 1993, recorded for the GRP label. This album is a soundtrack to the Motion Picture ""The Firm"" directed by Sidney Pollack. It reached No. 131 on ""Billboard""’s Contemporary Jazz chart. This soundtrack was a 1993 Academy Award nominee for Best Original Score. All tracks composed by Dave Grusin; except where indicated In 2015 La-La Land Records issued a remastered and expanded edition, featuring the film score on disc one and the 1993 soundtrack album and bonus tracks on disc two. Italicised tracks on disc one … |
| Model Predictions |  |
| Gemma-2-9B-Chat | Fidelity Fiduciary Bank |
| Implicit RAG FT | Mary Poppins (1964 film) |
| RAG FT with Reasoning | Documents 1 and 2 are useful to answer the question.<br>Both documents explicitly state that the song "Fidelity Fiduciary Bank" from the musical film *Mary Poppins* features the bank "Dawes, Tomes, Mousely, Grubbs Fidelity Fiduciary Bank".<br>Therefore, the film that features the mentioned bank is Mary Poppins. |

| Question | Which English chemist discovered the most elements? |
| --- | --- |
| Ground Truth | Humphry Davy |
| Retrieved Passages | Doc 1 (Title: Chemistry) J.J. Thomson of Cambridge University discovered the electron and soon after the French scientist Becquerel as well as the couple Pierre and Marie Curie investigated the phenomenon of radioactivity. In a series of pioneering scattering experiments Ernest Rutherford at the University of Manchester discovered the internal structure of the atom and the existence of the proton, classified and explained the different types of radioactivity and successfully transmuted the first element by bombarding nitrogen with alpha particles. His work on atomic structure was improved on by his students, the Danish physicist Niels Bohr and Henry Moseley. The electronic theory of chemical …<br>Doc 2 (Title: Chemistry) weights. The development of the electrochemical theory of chemical combinations occurred in the early 19th century as the result of the work of two scientists in particular, J.J. Berzelius and Humphry Davy, made possible by the prior invention of the voltaic pile by Alessandro Volta. Davy discovered nine new elements including the alkali metals by extracting them from their oxides with electric current. British William Prout first proposed ordering all the elements by their atomic weight as all atoms had a weight that was an exact multiple of the atomic weight of hydrogen. J.A.R. Newlands devised an early table of …<br>…<br>Doc 40 (Title: "History of chemistry") many had already applied the concept of atomism to chemistry. A major example was the ion theory of Svante Arrhenius which anticipated ideas about atomic substructure that did not fully develop until the 20th century. Michael Faraday was another early worker, whose major contribution to chemistry was electrochemistry, in which (among other things) a certain quantity of electricity during electrolysis or electrodeposition of metals was shown to be associated with certain quantities of chemical elements, and fixed quantities of the elements therefore with each other, in specific ratios. These findings, like those of Dalton’s combining ratios, were early clues … |
| Model Predictions |  |
| Gemma-2-9B-Chat | William Hyde Wollaston |
| Implicit RAG FT | William Ramsay |
| RAG FT with Reasoning | The best document to answer this question is **Doc 2(Chemistry)**.<br>This document states that **Humphry Davy** "discovered nine new elements including the alkali metals by extracting them from their oxides with electric current."<br>Therefore, the answer is Humphry Davy.<br>While other documents mention chemists and their discoveries, none of them contradict or provide a higher number of discovered elements than Davy’s nine. This makes Doc 2 the most useful in identifying the English chemist who discovered the most elements.<br>It’s important to note that this answer is based on Doc 2 alone. If other documents mentioned an English chemist who discovered more than nine elements, they would supersede the information in Doc 2. However, without that information, we can confidently say that based on these documents, Humphry Davy discovered the most elements among the English chemists mentioned.<br>**Conclusion:** Based on the given documents, Humphry Davy is the English chemist who discovered the most elements. |

## Appendix J Data-Augmented RAG Finetuning on Gemma-2-9B

In Figure 5, we illustrate the performance of implicit RAG finetuning on eight datasets with three different base models due to the space limitation. The whole results with Gemma-2-9B models can be found in Figure 11.

Figure: (a) TriviaQA
Refer to caption: x42.png

In Figure 6, we show the power of RAG finetuning with intermediate reasoning on five datasets because of the space limitation. The whole results on all the nine datasets with Gemma-2-9B models can be found in Figure 12.
Note that due to the computational complexity of inference with reasoning augmentation, results are shown for 1000 randomly-sampled queries for each dataset.

Figure: (a) TriviaQA
Refer to caption: x51.png

## Appendix K Data-Augmented RAG Finetuning on Mistral-Nemo-12B

In addition to the comprehensive data-augmented RAG fine-tuning results with three different base LLMs reported in Section 5, we also would like to show the results specifically with the Mistral-Nemo-12B models in Figure 13.

Figure: (a) TriviaQA
Refer to caption: x60.png

## Appendix L Data-Augmented RAG Finetuning on Gemini-1.0-Pro

In addition to the comprehensive data-augmented RAG fine-tuning results with three different base LLMs reported in Section 5, we also would like to show the results specifically with the Gemini-1.0-Pro models in Figure 14.
Due to the Gemini-1.0-Pro API call credit limitation, we random sample 1000 queries for each dataset.

Figure: (a) TriviaQA
Refer to caption: x69.png

## Appendix M Training data scaling and RAG performance.

**Table 11: Impact of RAG-specific training data scale on LLM performance in RAG.**
| Number of Retrieval Passages | 5k | 20k | 50k | 200k |
| --- | --- | --- | --- | --- |
| 10 | 0.5942 | 0.5925 | 0.6058 | 0.6277 |
| 20 | 0.5909 | 0.5925 | 0.6078 | 0.6294 |
| 30 | 0.5787 | 0.5792 | 0.6072 | 0.6150 |
| 40 | 0.5582 | 0.5582 | 0.5859 | 0.5983 |
| Avg. | 0.5805 | 0.5806 | 0.6017 | 0.6176 |

To investigate the influence of the size of the training data on the effectiveness of RAG-specific tuning, we fine-tune the Gemma-2-9B-Base model using varying amounts (5k to 200k samples) of mixed training data from NQ, WoW, Fever, and MMLU.
Table 11 presents the evaluation results on the NQ dataset, demonstrating a clear positive correlation between the scale of training data and the performance of the resulting LLM in RAG.
Increasing the amount of training data consistently leads to improved accuracy, highlighting the benefits of leveraging larger datasets for fine-tuning LLMs in RAG applications.

## Appendix N RAG-specific tuning data inside SFT mixtures

**Table 12: Combining RAG-specific data with general SFT data for enhanced LLM performance in RAG.**
| Dataset | base | SFT only | SFT + RAG-FT |
| --- | --- | --- | --- |
| MT-Bench | 2.3125 | 5.8969 | 5.6031 |
| NQ | 0.2105 | 0.5687 | 0.6033 |
| TriviaQA | 0.4940 | 0.7155 | 0.7481 |

Having established the effectiveness of RAG-specific fine-tuning for improving LLM performance in RAG tasks, we now investigate whether combining RAG-specific data with general SFT data can further enhance performance while preserving the LLM’s general capabilities (e.g., reasoning and long-form generation), as a way to assess the potential of the proposed tuning methods to be useful for construction of foundation models.
We train the Gemma-2-9B model using two different strategies:
(1) SFT data only: The LLM is trained solely on general SFT data (Ultrachat 200k).
(2) SFT data + RAG-specific data: The LLM is trained on a combination of Ultrachat 200k and 50k RAG-specific data (the same data used in Figure 5).
We evaluate the resulting models on MT-Bench to assess their general language capabilities and on NQ and TriviaQA to measure their RAG performance.

Table 12 presents the results, demonstrating that incorporating RAG-specific data into the SFT process can significantly improve the LLM’s performance on RAG tasks while maintaining its performance on general language tasks.
This finding suggests that combining task-specific and general-purpose data during fine-tuning can be a viable strategy for enhancing LLMs in specialized applications without compromising their overall capabilities.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: - Fig 1 also has an interesting observation where the Gemini-1.5-pro is quite robust to increasing number of retrieved passages and doesn't decline in performance. Is there a correlation between the context windows of these long-context LLMs and their ability to be robust to increasing number of retrieved passages? And if so, would "infinite"-context window LLMs have most robustness? It would be interesting to perform additional study around this.
- The above is especially the case when answering the subsequent questions in the paper regarding the hard negatives. Since LLMs are not perfect processing blocks. Just because the correct one is present doesn't mean the LLM will get it right, and it depends on how good the LLM is at processing the given information.
- Fig. 7b indicates that there might not be much impact on the choice of the retriever. More information on that would be helpful.
soundness: 3
strengths: The paper focuses on answering a question that hasn't received rigorous attention yet: understanding if retrieval optimizations are not needed anymore in RAGs system if we have long-context LLMs. They approached it quite methodically by first trying to answer the question by looking at multiple LLMs (each with its own context length limit) and multiple retrievers and seeing the impact of the RAG system with increasing retrieved passages. The problem has lot of moving parts: LLM, context length limit, # of retrieved passages, choice retriever, evaluation and training datasets, to name a few. The authors have done a reasonably well job to balance these different parts to answer the question.
summary: This paper studies how RAG systems are impacted by the advancement in long-context LLMs. The study empirically shows how the increase in the context-length (which allows more passages to be used) doesn't necessarily imply improved performance of the system. They show that the performance first improves before declining with increases number of retrieved passages. This is attributed to the presence of "hard negatives" which confuses the LLMs. The authors next propose 3 ways to counter the "hard-negatives" issue. The first is the reordering of the retrieved passages and it is shown that it can be an effective non-training based method. The other two approaches are training-based: finetuning with explicit hard-negatives, and finetuning with an additional reasoning step to train the LLM to learn to reject hard negatives. They show how these approaches help improve performance. A well-performed set of ablations help see the impact of data distribution match, retriever choice, and context length of LLMs during training.
weaknesses: There are a few places where the observations can be better represented and new hypotheses can be developed and evaluated. Some examples are below (and in the "questions" section below):
- In Fig 1, the authors claim that for strong retriever, the performance either reduces or reaches a plateau with increasing number of retrieved passages whereas it is different for weak retriever. However, I don't see any difference from the figure. Even for BM25 (weak retriever) the performance has inverted U shape similar to e5 (strong retriever).
- The reordering algorithm assumes that hard negatives are typically lower in relevance and hence reordering allows them to be in the middle, therefore allowing the LLM to not get distracted by them. However, that's not necessarily true. They most likely would be at the top and hence are called "hard" negatives.
- It was unclear until we look at the appendix on what was the knowledge base used for the experiments. It is also not clear (maybe my mistake) on what was the retriever used for experiments in Sec. 5.
- A key issue in the experiments is that for all the datasets used, the knowledge base is wiki which is already a part of the LLMs pretraining data. So some of the observations made could be impacted by how much of the knowledge for answering the query is already in the LLM's weights and how much is from the retrieved passages. Since the answers to the query are most probably in the model weights due to pretraining data, the observations from the experiments could be flawed and we might not be able to make fair conclusions. It would be good if we can run these experiments with a knowledge base use for retrieval be something that is NOT a part of the LLM's pretraining data.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - What are the key properties of "hard negatives" that make them particularly challenging? Is it semantic similarity, structural patterns, or something else? How does your analysis generalize to other types of retrievers not tested in the paper?
- How does your approach compare to recent methods like RetRobust (Yoran et al. 2023 - Making Retrieval-Augmented Language Models Robust to Irrelevant Context), or RA-DIT: Retrieval-Augmented Dual Instruction Tuning (Lin et al. 2023) that also aim to improve RAG robustness?
- Have you considered comparing against hierarchical retrieval methods that might naturally mitigate the hard negatives problem?
- Could you provide a cost-benefit analysis comparing your methods against simpler approaches like reducing retrieval size or basic reranking?
- Are there specific types of queries or documents where hard negatives are particularly problematic?
- Why did you choose the specific reordering strategy for the training-free approach? Have you explored other ordering patterns?
- How sensitive is the intermediate reasoning approach to the quality and style of reasoning labels?
soundness: 2
strengths: - The paper presents a technically sophisticated analysis of long-context LLMs in RAG systems, with comprehensive experiments across multiple models, retrievers, and datasets. The empirical validation is thorough and well-documented.
- The identification of the "hard negatives" problem in long-context RAG represents a major contribution to understanding LLM behavior in retrieval contexts.
- The proposed methods, particularly the training-free retrieval reordering approach, offer immediate practical value for improving RAG systems. The solutions are well-motivated by the empirical findings.
summary: This paper investigates the challenges and opportunities of using long-context LLMs in RAG systems. The main contributions include:
- A systematic analysis demonstrating that increasing retrieved passages doesn't always improve performance with long-context LLMs
- Identification of the detrimental impact of "hard negatives" on LLM performance
- Three approaches to improve RAG performance: (1) a training-free retrieval reordering method, (2) Implicit robustness fine-tuning, (3) explicit relevance fine-tuning with intermediate reasoning.
- Comprehensive analysis of design choices for RAG-specific LLM tuning.
weaknesses: - The paper lacks to explain why long-context LLMs behave differently with hard negatives, which may limit the generalizability of the findings and makes it harder to predict when the proposed solutions might fail.
- The experimental evaluation doesn't compare against recent competing methods for improving RAG performance, such as reranking approaches or hierarchical retrieval methods.
- The paper lacks an analysis of failure cases or limitations. When do these methods break down? What types of queries or documents are particularly challenging?
- The computational requirements of the intermediate reasoning approach requiring Gemini-1.5-Pro are not thoroughly discussed. This raises questions about the practical applicability in resource-constrained environments.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: See Questions
soundness: 3
strengths: 1. The phenomenon where the quality of LLM-generated outputs first improves and then declines with an increasing number of retrieved segments is intriguing, and the authors have conducted thorough experiments to explore this.
2. The importance of hard negatives for long-context LLM evaluation is well-studied, and it is observed that increasing the number of hard negatives typically leads to a decrease in RAG answer accuracy. The strength of the retriever is directly related to the difficulty of the hard negatives.
3. The three methods proposed by the authors (Retrieval reordering, Implicit robustness fine-tuning, Explicit relevance fine-tuning) all enhance the performance of long-context LLMs in RAG applications.
4. Overall, the paper is well-written and easy to follow.
summary: This paper primarily explores the phenomenon of long inputs in Retrieval-Augmented Generation (RAG): for many long-context large language models (LLMs), the quality of generated output initially improves but then declines as the number of retrieved passages increases. This observation contradicts the previous assumption that a larger retrieval set, containing more relevant information, would enhance performance. The study identifies the detrimental impact of retrieved "hard negatives" as a significant factor. To mitigate this effect and enhance the robustness of long-context LLMs in RAG, the paper proposes three methods: (1) an untrained method based on retrieval reordering, (2) implicit robustness fine-tuning to enhance resistance to hard negatives, and (3) explicit relevance fine-tuning with intermediate reasoning to identify relevance.
weaknesses: 1. Figure 1 shows that while other LLMs exhibit an initial improvement in performance followed by a decline with an increasing number of retrieved passages, Gemini-1.5-Pro does not exhibit this trend, and the authors do not explain this phenomenon.
2. From the experiments, it appears that LLMs ranging from 7B to 12B are susceptible to the influence of hard negatives. It remains unclear whether this phenomenon can still be observed in LLMs larger than 30B. I suspect that larger LLMs might automatically identify these hard negatives.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Please see Weaknesses above.
soundness: 3
strengths: 1. The writing is clear and easy to follow

2. Experimental results show the effectiveness of the proposed methods.

3. This paper makes significant contribution in the context of the rapid development of long-context LLM
summary: This paper investigates the effect of increasing the length of the retrieved document list on the performance of RAG. They find that simply increasing the number of retrieved documents does not consistently improve the performance of RAG, and both training-free and training-based methods are proposed to solve this problem.
weaknesses: In this rebuttal period, the authors address most of my concerns below

1. **Wrong research basis**: In Abstract and Introduction Sections, this paper's description of the current research status of the RAG community is wrong, which directly leads to the fact that the research basis of this paper is wrong, which is unacceptable. Specifically, in Lines 14-16 of Abstract "It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance" and Lines 46-47 of Introduction "Intuitively, longer context would allow for the inclusion of more retrieved passages, leading to higher recall and potentially improved performance". Both claims misrepresent the current state of research in the RAG community. Many studies in RAG have shown that increasing the number of retrieved documents cannot consistently improve the RAG performance due to the noise or incorrectness of the retrieved texts [1, 2 ,3, 4, 5]. This paper wrongly uses a problem that has been clarified in the RAG community as the motivation, which is unacceptable.

2. **Over-claiming of contributions**: As weakness 1 indicates that the robustness of RAG has been fully-studied in RAG [1,2,3,4,5] and shows irrelevant retrieved documents may interference the performance of RAG, the main contribution claimed in this paper that "we systematically analyze the use of long-context LLMs in RAG systems, specifically examining the impact of retrieved "hard negatives" on performance" cannot be seen as the contribution of this paper.

3. **Limited technical contribution**：This paper claims that they propose two methods to enhance the robustness of RAG on "hard negatives". One method is a training-free method based on retrieval reordering, this method is incremental based on Lost-in-the-Middle [6]. The other method is explicit tuning with intermediate reasoning for relevance identification, this is similar to RetRobust [1] without distinguishable points.

[1] Making Retrieval-Augmented Language Models Robust to Irrelevant Context

[2] Chain-of-note: Enhancing robustness in retrieval-augmented language models

[3] Corrective retrieval augmented generation

[4] Evaluation of Retrieval-Augmented Generation: A Survey

[5] Benchmarking large language models in retrieval-augmented generation

[6] Lost in the Middle: How Language Models Use Long Contexts

Meta-review: This paper explores an important aspect of long-context large language models (LLMs), specifically how the quality of generated output initially improves but then declines as the number of retrieved passages increases. The study proposes a training-free method through retrieval reordering and two training-based methods: RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning. Experiments conducted across multiple models and datasets provide insightful findings.

Decision: Accept (Poster)