Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Authors: Bowen Jin
Jinsung Yoon
Jiawei Han
Sercan O Arik
Venue: ICLR 2025 Poster (2025)
Keywords: retrieval-augmented generation
large language models
long-context
Primary area: foundation or frontier models, including LLMs

Abstract: Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs.  From a long-context LLM perspective, it assumes that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance. However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases. This paper investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" as a key contributor. To mitigate this and enhance the robustness of long-context LLM-based RAG, we propose both training-free and training-based approaches. We first showcase the effectiveness of retrieval reordering as a simple yet powerful training-free optimization. Furthermore, we explore training-based methods, specifically RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning with intermediate reasoning, demonstrating their capacity for substantial performance gains. Finally, we conduct a systematic analysis of design choices for these training-based methods, including data distribution, retriever selection, and training context length.

Published as a conference paper at ICLR 2025
LONG-CONTEXT LLMs MEET RAG: OVERCOMING CHALLENGES FOR LONG INPUTS IN RAG
Bowen Jin's JJinsung Yoon², Jiawei Han's Sercan Ö. Arik?
University of Illinois Urbana-Chaman Champa-hn Goooge Cloud
(bowenj4, (anj)eeillan edu, ( iiinuuggyonn, www.::oooo.....rrreoeoooo.......
ABSTRACT
Retrieval-augmeented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer imput sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs. From long-context L.M perspective, it assumes that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance. However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines the number of retrieved passages increases. This paper investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" key contributor. To mitigate this and enhance the robustness of long-context LLM-bbased RAG, we propose both training- free and training-based approaches. We first showcase the effectiveness of retrieval reordering as simple yet powerful training-fire optimization. Furthermore, we explore training-based methods, specifically RAG- specific implicit LLM fine-tuning and RAG-oriented reasoning. demonstrating their capacity for substantial performance gains. Finally, we conduct systematic analysis of design choices for these maining-based methods, including data distribution, retriever selection, and training context length.
with intermediate
INTRODUCTION
Retrieval-augmenne generation (RAG) (Gao et al., 2023) empowers large language models (LLMs) to utilize external information sources by selecting the relevant pieces from a large corpus (Zhao et al., 2023), thereby enhancing their effectiveness, customizability and efficiency in complex problem- solving (Yu et al., 2024a; Chen et al., 2024b). RAG can also mitigate issues such as factual inaccura- cies (Augenstein et al., 2023) and hallucinations (Huang et al., 2023), which LLMs often exhibit when confronted with knowledge-intensive tasks. RAG systems typically employ retriever to identify relevant information from corpus, which is then presented in the context of an LLM as the generator.
Recent advances in computational resources and methodological innovations have enabled the development of LLMs that support increasingly longer context (Reid et al., 2024; Dubey et al., 2024). This has even opened up new avenues for directly inputting entire corpora or knowledge bases into the I.LMs. Yet, it would still not be feasible for large corpora (e.g., Wikipedia) and can incur higher computational costs. Despite extensive research on RAG (Xu et al., 2023; Li et al., 2024; Lee et al., 2024), the interplay with long-context LLMs, particularly how to optimally design RAG systems using them effectively, remains under-explored. Existing works (Lin et al., 2024; Asai et al., 2024; Yoran et al., 2024) propose tuning LLMs for RAG, but predominantly focus on limited number of retrieved passages (< 10). From the long-contens LLM perspective (Reid et al., 2024; Dubey et al., 2024), longer context would allow for the inclusion of more retrieved passages, leading to higher recall and potentially improved performance. However, our findings reveal that this does not always hold true and highlight the need for a careful re-evaluation of standard RAG designs when utilizing long-context LLMs. We demonstrate that achieving optimal performance in such systems and to fully utilize the opportunities provided by the LLMSs require a holistic rethinking and effective novel approaches to the unique challenges.
Work was done while Bowen student researcher Google Cloud.

---

Published as a conference paper at ICLR 2025
Though there are some studies evaluating long-context LLMs in synthetic benchmarks such as
(Kamradt, 2023), their capability in real long-context RAG scenarios is underexplored. This paper presents comprehensive analyses long- long-context LLMs in RAG systems. Contrary to the suggestions of previous work (Xu et al., 2023; Li et al., 2024), our research reveals that increasing the number of retrieved passages does not consistently improve performance with long- context LLMs (Section 3.1). Instead, we observe that the generative modeling performance initially increases and then declines simply providing more retrieved passages does not guarantee better outcomes. Using stronger retrievers is also not mitigation mechanism indeed the performance degradation can even be more severe with them. For deeper understanding of the phenomenon, we conduct further investigations, which reveal that increasing the number of retrieved passages can introduce irrelevant information ("noise") that can mislead the LLM generation (Section 3.2). We also examine the impact of "hard negatives" of different retrievers on the L.Mss, and show that there are scenarios where the 'hard negatives' from stronger retrievers might confuse the LLM generation even more than those from weaker retrievers (Section 3.3).
To address the challenges identified in our analyses, we propose three methods, encompassing both training-free -ree and training- based approaches, to enhance the performance of long-context LLMs phenomenon
in RAG applications: (1) Retrieval reordering: recognizing the post-in-the-midee observed for long-context LLMS (Liu et al., 2024), we propose reordering retrieved documents based on their retrieval scores. By prioritizing documents with higher scores at the beginning and end of the input sequences, we guide the LLMs' attention towards more relevant information and mitigate the impact of hard negatives. (2) Implicit robustness fine-tuming: given the ability to handle noisy retrieved context is not explicitly acquired during standard LLM training, we propose tuning the LLMs with the data comprising queries and retrieved documents, including those with potential noise. This encourages the LLMs to implicitly learn robustmess to hard negatives. (3) Explicit relevance fine-tuning: while the previous method implicitly enhances robustness, it does not explicitly teach the LLMs to identify relevant documents. Therefore, we propose augmenting the LLM tuning with an intermediate reasoning step, where the LLMs trained to analyze the retrieved documents and explicitly identify relevant information before generating the final output. This approach aims to improve the LLMs' ability to discern relevant information from noise within the retrieved context.
Overall, the main contributions can be summarized as follows:
we systematically analyze the use of long context LLMs in RAG systems, specifically (1) examining the conflict impact of retrieved "relevant documents" and "hard negatives" on RAG performance when scaling up the number of retrieved passages and (2) exploring the correlation between retriever strength and hard negative difficulty.
Systematic analysis of long-context RAG:
Novel methods for robust RAG: we propose three methods to improve the robustness of long- context LLMs in RAG: (1) a training- method based on retrieval reordering, (2) implicit tuning for robustness to hard negatives and (3) explicit tuning with intermediate reasoning for relevance identification. Overall, our proposed approaches show significant accuracy and robustness improvements on long-context RAG performance.
Comprehensive study of RAG-specific LLM tuning: we conduct a thorough investigation into various factors influencing the effectiveness of RAG-specific tuning. including data distribution, the employed retriever, and training context length.
2 RELATED WORK
Large language models (LLMs) can be prone hallucinations especially at knowledge-intensing tasks (Zhao et al., 2023; Huang et al., 2023; Augenstein et al., 2023). Retrieval-augmente generation (RAG) addresses this by incorporating external knowledge sources to provide accurate and relevant information (Gao et al., 2023). Traditional RAG systems comprise a retriever to identify relevant information and a generator to synthesize the answer (Zhao et al., 2024; Zhu et al., 2021). While previous research focused on improving either the retriever (Karpukhin et al., 2020; Izacard et al., 2021; Wang et al., 2022) or the generator (Dong et al., 2022; Liu et al., 2024; Agarwal et al., 2024) in isolation, we take a holistic approach. Conducting comprehensive analyses of the entire RAG system, we focus on the challenges and opportunities presented by using long-context LLMs as generators. We propose novel solutions to better employ them in long-context RAG.
2

---

Published as a conference paper at ICLR 2025
Increased computational resources and advancements in efficient training methods have pushed LLMs supporting longer inputs (Wang et al., 2024; Zhou et al., 2024). While long-contex: LLMs (Reid et al., 2024) demonstrated impressive performance 2023) and RULER (Hsieh et al., 2024a), these benchmarks often rely on random negative examples and do not accurately reflect the challenges posed by the "hard negatives" encountered in real-world RAG scenarios (Cuconasu et al., 2024). Furthermore, existing studies on long-context LLMs in multi-document settings (Liu al., 2024; Shi al., 2023) often assume a single "golden" document and random negatives, which differs from the RAG context where multiple relevant passages and hard negatives may exist (Hsieh et al., 2024b; Cuconasu al., 2024). Although some research has explored the relationship between RAG and long-context LLMs (Xu et al., 2023; Li et al., 2024; Lee et al., 2024), these works take different perspectives. They mainly focus on studying the (1) trade-offs between RAG and long-contexs LLMs (Xu et al., 2023), (2) routers to manage RAG and long-context LLMs (Li et al., 2024), (3) and the potential for LLMs to replace retrieval entirely (Lee et al., 2024), while leaving long-context LLMs as generators in RAG under-explored. Some existing work (Yu et al., 2023; Yan et al., 2024) identify the harm of irrelevant retrieved information in RAG, but they mainly focus on synthetic scenario where no more relevant information is added, which is different from the real-world RAG systems with large-scale retrieval. We delve deeper into the potential benefits of long-context LLMs for RAG and investigate how optimize these LLMs specifically for this application.
benchmarks like weeedle-n-thh-haystee
(Kamradt,
Previous research has explored adapting LLMs for RAG using instruction tuning (Zhang et al., 2023). RetRobust (Yoran et al., 2024) fine-tunes LLMs with I retrieved relevant passage or random negative passage make it robust to irrelevant passage. RA-DIT (Lin et al., 2024) conducts dual instruction tuning to make the LLM more effectively leverage retrieved information and retriever provide results more aligned with LLM preference. Relf-RAG (Asai et al., 2024) introduces framework to train a LM that dynamically retrieves passages, generates content, and evaluates the retrieved passages for improved performance. RAFT (Zhang et al., 2024) trains the LLMs to improve their ability to answer questions in Hopen-book' in-domain settings. More recently, RankRAG (Yu et al., 2024b) tunes LLM for the dual purpose of context ranking and answer generation in RAG. InstructRAG (Wei et al., 2024) finetunes the LLM to generate self-synthesized rationales rather than directly answering the question. However, these existing efforts primarily focus on tuning with a limited number of retrieved passages (typically fewer than 10) and do not fully leverage the potential of long-context LLMs. This work aims to address this gap by specifically investigating how to optimize long-contens LLMs for large-scale RAG, where the number of retrieved passages can be significantly higher.
CHALLENGES OF LONG CONTEXT LLMS IN RAG
We
present systematic investigation into the challenges of utilizing long-context L.LSs in RAG. Each subsection focuses on a specific research question, outlining corresponding experiments and analyzing the results on the key challenges. These insights inform the development of targeted solutions for improving RAG performance with long-context LLMs, which are presented in subsequent sections.
3.1 THE EFFECT OF RETRIEVED CONTEXT SIZE ON RAG PERFORMANCE
This subsection investigates the relationship between the number of retrieved passages and the performance of long-context LLMS in RAG systems.
Research question. Long-context LLMs offer the potential to incorporate more retrieved passages into RAG systems. This raises a crucial question: Does a larger volume of retrieved context consistently translate to better performance when using long-context LLMs in RAG?
Experimental setting. We evaluate the performance of RAG systems on the Natural Questions (NQ) (K (Kwiatkowski et al., 2019) dataset using two different retrievers (BM25 (Robertson et al., 2009) and c5 (Wang et al., 2022), where e5 exhibits higher performance on NQ (Recall@40 is 0.90 with e5 and 0.73 with BM25)) and four long-context LLMs (Gemma-7B-Chee (Team et al., 2024a), Gemma-2- 9B-Chat (Team et al., 2024b), Mistral-Nemo- 12B-Instruct (Jiang et al., 2023) and Gemini- 1.5-pro (Reid et al., 2024)). We systematically vary the number of passages retrieved by each retriever.
Observations. Figure 1 presents the following key observations: 1) Strong Retriever (e5): Across all LLMss, increasing the number of retrieved passages initially improves performance, but then
3

---

Published as a conference paper at ICLR 2025
0.55
0.55
0.50
Accuracy
RAG Accurao
0.40
0.40
0.35
0.35
Gemma- 78- Chat
Gemma-TB-C- Chat
0.25
0.5
Gemini-1.8 1.5mm
Cemini-1.--9h
0.20
10a
10
10"
16
020
10P
162
1if
16"
Retrieved Passages
Retrieved Passages
RAG performance with e5 retriever
(b) RAG performance with BM25 retriever
Figure 1: Impact of retrieved context size on RAG performance with different LLMs on NQ- Increasing the number of retrieved passages initially improves performance but then leads to decline. This degradation is more pronounced using a retriever (e5) that exhibits higher recall @k on NQ compared BM25 (Recall @40 is 0.90 with e5 and 0.73 with BM25). The maximum number of retrieved passages varies across LLMs due to differences in their maximum token limits.
leads to a sharp decline or plateau. 2) Weak Retriever (BM25)): Performance generally exhibits a continuous increase slight decrease the number of retrieved passages increases. While these observations may appear counter-intuitivy higher recall (i.e., greater chance of retrieving relevant information) the inclusion of additional documents can reduce precision, with irrelevant or misleading passages detracting L.MSs from overall performance. Comparison of different retrievers, the results on other datasets and case studies shown in Appendix A, 1/1/R and O.
given that one might expect monotonic improvements due
Insights. The effectiveness of increasing retrieved context size in RAG depends on the strength of the retriever. With both retrievers, performance exhibits an "inverted-U pattemn". However, the curve with strong retriever reaches the performance peak earlier and the decreases faster than that with weak retriever. This suggests that factors beyond the amount of retrieved information are at play.
3.2 THE INTERPLAY OF RETRIEVAL QUALITY AND LLM CAPABILITIES
This subsection delves into the factors hindering the performance of long-context LLMs in RAG, aiming discern whether limitations arise from retrieval quality the LLM's ability to process the retrieved information.
Research question. Do the observed performance bottlenecks originate from limitations in the retriever's ability to identify relevant information, or from the long-context LLM's capacity to effectively utilize the retrieved context?
Experimental setting. We analyze the relationship between RAG performance and retrieval quality, specifically recall and precision, using the Gemma-2-9B-Char LLM with both e5 and BM25 retrievers (Figure 2). Recall @ measures the presence of relevant passages within the top-k retrieved passages, while precision @k quantifies the proportion of relevant passages among them.
Increasing the number of retrieved passages consistently leads to higher recall but lower precision, irrespective of the retriever used. Crucially, the overall accuracy of the RAG system falls below the recall across all retrieval sizes. This indicates that even when relevant information is present in the retrieved context, the LLM may fail generate the correct answer. This demonstrates that the irrelevant retrieved passages can sometimes mislead the L.M.. Furthermore, despite exhibiting higher precision, the retriever leads to more pronounced performance degradation as the number of retrieved passages increases compared BM25.
Observations.
Insights. These observations yield two key insights: (1) Influence of irrelevant passages: The discrepancy between retrieval recall and RAG accuracy underscores the detrimental effect of irrelevant retrieved passages ("hard negatives") on the LLMs' performance. Even when relevant information is available, the presence of hard negatives accurate answers. (2) Limitations of precision as a metric: The contrasting performance trends observed with e5 and BM25, despite the former's higher precision, reveal that precision alone is
mislead the LLMs and hinder their ability generate
4

---

Published as a conference paper at ICLR 2025
1.0
0.54
RAG acouracy
0.425
FAG accuracy
<i. Rccall
Precision
0.400
Rocal Precision
accurac
RAG accuracy
+0.00
0.350
0.50
0.325
0.49
0.48
Recall
0.25
Prodition
20
20
# Retrieved Passages
# Retrieved Passages
(a) Retrieval with c5 retriever
(b) Retrieval with BM25 retriever
Figure 2: Analyzing the relationship between RAG performance and retrieval quality (recall/rrecsioo) using Gemma-2-98 2------9----- with e5 and BM25 retrievers. (1) Accuracy vs. Recall: RAG accuracy consistently falls below retrieval recall for both retrievers, indicating that the presence of relevant information does not guarantee correct answers. This highlights the detrimental impact of irrelevant passages on LLM performance. (2) Precision and hard negatives: Despite higher precision with e5, the performance degradation with increasing retrieval size is more pronounced compared to BM25. This demonstrates that precision alone an insufficient metric for assessing the impact of "hard negatives," as the nature of irrelevant information significantly influences LLM performance.
inadequate measure of retrieval quality in this context, when the end-to-end performance is considered. The specific characteristics of the irrelevant passages, rather than just their quantity, significantly impact the L.MMs performance. Retrievers might significantly differ in their way of priorization of them, and that might not be fully captured in metrics like precision. In this scenario, it is observed that "hard negatives" retrieved by a stronger retriever (e5) might even more detrimental to the LLM than those retrieved by weaker one (BM25).
3.3 THE IMPORTANCE OF HARD NEGATIVES FOR LONG-CONTEXT LLM EVALUATION
This subsection investigates the impact of "hard negatives" on the performance of long-context LLMss in RAG. highlighting the need for more robust evaluation methodologies.
Research question. In long-context RAG scenarios, where a vast knowledge source necessitates retrieving numerous passages, the likelihood of including relevant information (i.e. obtaining high recall) increases. However, this also elevates the risk of introducing hard negatives. This raises two critical questions: (1) How robust current long-con conteex LLMs to these hard negatives? and (2) Does the impact of hard negatives vary with the relriever used?
Experimental setting. This study investigates the effect of hard negative passages long-context LLM performance in a controlled setting. We tasked three L.MSs (Gemma2-7B-Chhl Mistral-Nemo- 12B-Instruct, and Gemini-1.5-P
with answering queries based on a context comprising a single golden passage and varying number of hard negative passages retrieved using different methods (e5, Contriever, BM25, and random sampling). This synthetic experiment, detailed in Figure isolates the impact of hard negatives by holding the golden passage constant and intentionally excluding scenarios with multiple golden passages, which are common in real-world RAG systems. See Appendix C for complete illustration of the experimental setup.
Observations. (1) Sensitivity hard negatives: Across all L.Ms, increasing the number of hard negative passages generally leads to a decline in RAG answer accuracy. (2) Retriever strength and hard negative difficulty: The strength of the retriever directly correlates with the difficulty of the retrieved hard negatives. LLMs struggle more with hard negatives from stronger retrievers (e.g. e5) compared to those from weaker retrievers (e.g., BM25) or random sampling. (3) Distinguishing random and hard negatives: While Gemini- 1.5-Pro demonstrates robustness random negatives, remains susceptible to the influence of hard negatives. More results on other datasets and qualitative studies be found in Appendix B.2 and D.
1emnini-1.5-Pa
5

---

Published as a conference paper at ICLR 2025
e5
0.5
BM25
0.75
randam
0.75
0.80
Precision
0.4
contriever
RAG Accuracy
0.70
BM25
contriever
RAG Accuracy
0.70
contriever
BM25
RAG Accuracy
0.75
contriever
BM25
0.65
0.3
0.65
0.70
0.60
0.60
0.65
0.2
0.50
0.60
0.30.40.50 60.70.80.9
0.45
10 15
20
25 30 35
0.50
10 15 25 30 35
0.55
10 15 20 25 30 # Passages
Recall
# Passages
# Passages
(a) Retrievers
(b) Gemma2-9B-CMm
(c) -2n-structt
(d) 1.5-Po
Figure 3: Evaluating the impact of hard negatives on long-context LLMs. (a) The retriever perfor- mance on NQ dataset: e5 > contriever > BM25. (b)(c)d) For each query, a single golden passage (containing the correct answer) combined with varying numbers of hard negative passages retrieved by different methods: e5, Contriever, BM25, and random sampling. The LLMs are then tasked with answering the query based on this context. This setup allows to assess the robustness of LLMs to hard negatives and the influence of retriever characteristics on their overall impact.
Insights. Existing benchmarks for evaluating long-context L.MMs, such "needle-in- "teeelliintthe--aastak""" (Kamradt, 2023) and RULER (Hsieh et al., 2024a), predominantly utilize random negatives. Our findings demonstrate that such benchmarks may not adequately capture the challenges posed by hard negatives, which are prevalent in real-world RAG applications. Their takeaways would have limitations. The need for new evaluation methodologies that incorporate hard negatives (specific to the employed retrievers) is highlighted, to provide a more comprehensive and realistic assessment of long-context LLM performance RAG.
SIMPLE AND EFFECTIVE TRAINING-FREE RAG IMPROVEMENT
Building upon the analyses in Section on the detrimental impact of hard negatives on long-conters LLMs in RAG, we focus on the training-free solution, retrieval reordering. This method leverages the inherent fost-in-the-middll phenomenon observed in LLMS mitigate the negative effects of hard negatives. As highlighted by Liu et al. (2024). LLMs exhibit tendency to prioritize information presented at the beginning and end of an input sequence, while paying less attention the middle.
Exploiting this "lost-in-the-midde the retrieved passages based on their relevance scores calculated by the retriever. Given query and a set of retrieved passages d1, d2, ... de with decreasing relevance scores, the standard input sequence construction for an L.M with instruction I would be: [T,d] d22 d de, 4). Retrieval reordering modifies this prioritize passages with higher scores at the beginning and end: [T,d,, da...,, d4, d2, 4] where the position of passage de is determined by
behavior, we consider simple and effective strategy: reordering
[1+] 2
if mod(i, 2)
Order(d:)
-(Kk + 1) - if mod(i, 2) 0
This reordering strategy aims to guide the LLM's attention towards the most relevant passages, thereby reducing the influence of hard negatives positioned in the middle of the sequence. The pseudo-code and intuition for retrieval reordering can be found in Appendix E and Q.
Retrieval reordering significantly improves RAG performance, particularly with larger num- bers of retrieved passages. To assess the effectiveness of retrieval reordering, we conduct ex- periments with two retrievers (e5 and BM25), two long-context LLMs (Gemma-2-9B-Char and Mistral-Nemo- 12B- I2s-rustuuc),, and two datasets (NQ and PopQA). As illustrated in Figure 4, retrieval reordering yields negligible improvements with smaller retrieval sets, but significantly and consis- tently outperforms the original ordering when the number of retrieved passages is large. This behavior attributed to the interplay of two factors that become increasingly significant with larger retrieval sets: (1) the amplified "lost-in-the-middle phenomenon, where LLMs prioritize information at the beginning and end of the input sequence, and (2) the increased prevalence of hard negatives, which can hinder accurate answer generation. By strategically placing passages, retrieval reordering miti- gates these issues, highlighting the potential of position engineering as a complementary technique to prompt engineering for optimizing long-context L.LMs in RAG.

---

Published as a conference paper at ICLR 2025
0.44
2.40
Accurao
0.535
0.42
0.52
0.40
0325
0.530
RAG
0.40
RAG
0.46
2:50
2.49
RAG
2.42
0.60
0.44
0ng
onginal
0.30
6.42
6.10
16
10
25 10
to 126151151750
10 1822910010
# Retrieved Passages
Retrieved Passages
Retrieved Passages
Retrieved Passages
(a) NQ: Gemma2+e5
(h) NQ: Gemma2+BM25
(c) NQ: Mistral+e5
(d) NQ: Mis
0.500
RAG Accurasy
0.33
0.39
0.500
0.525
woodewtng
BAG Accurary
0.40
0.52
0.37
3
0415
0440
0.38
0.51
9m ender
24G
0.35
mortedin
nigiral omser
0.375
0:10
04:0
0425
6.32
orgiall order
merreriig
15
10 1307511117221
Retrieved Passages
Retrieved Passages
Retrieved Passages
Passages
(c) PQA: Gemma2+e5
(f) PQA: Gemma2+BM25
(g) PQA: Mistral+e5
(h) PQA: Mistral+BM25
Figure 4: Evaluating the effectiveness of retrieval reordering in various RAG contigurations. Results demonstrate that reordering retrieved passages consistently enhances performance, particularly when the number of retrieved passages is large. (Retrievers: e5, BM25; LLMs: Gemma2 9b-Chat, Mistral-
Nemo-12B 12B-IIntructtt Datasets: NQ, PopQA)
IMPROVING ROBUSTNESS FOR RAG VIA DATA- AATA-AUGMENTED TUNING
5.1 IMPLICITLY IMPROVING LLM ROBUSTNESS THROUGH FINE-TUNING
While the retrieval reordering strategy presented in Section 4 mitigates the detrimental impact of hard negatives, it does not inherently enhance the LLM's ability to handle such irrelevant information within the context. To address this, we conduct systematic investigation into RAG-specific tuning as a means of improving long-context LLMs for RAG applications.
Our tuning paradigm involves training LLM to generate the correct answer (a) given a comprehensive input comprising an instruction (4), a query (q), and a set of retrieved passages (d1, .2...... dz):
Input: [I, d1, d2 ..., dx de.
Output: a.
(2)
This approach aims to implicitly enhance the LLM's robustness to hard negatives by exposing it diverse range of retrieved contexts during fine-tuning, thus enabling it to learn to effectively identify and utilize relevant information even in the presence of noise.
To assess the generalization capabilities of RAG-tuned LLMs, we fine-tune Gemma-2 --9-n---------s- Mistral-Nemo-1225
and Gemini-1.0-PP using diverse dataset comprising NQ, WoW, Fever, and MMLU. We then evaluate on a range of unseen datasets, including TriviaQA, PopQA, HotpoiQA, 2wikimultihopaa, Webquestions, Bamboogle, ASQA, T-REx, and zsRE. We e5 the retriever and Wikipedia- 18 (Karpukhin et al., 2020) as the corpus. We compare the performance of the RAG- tuned model (RAG FT) with two types of baselines: (1) Chat model with retrieval augmentation: the Gemma-2-9B Laatmaa---B--haall Chat//Mistral- fine-tuned with standard supervised fine-tuning (SFT) on question-answer pairs without retrieved context (Direct FT w/o RAG). Further details regarding the datasets and experimental setup can be found in Appendix and G.
228-Instruct/Giemini .0-Pro w. RAG; (2) Direct SFT: the ones
Figure 5 shows the three key observations: (1) Consistent improvement over baselines: RAG FT consistently outperforms the chat model w. RAG and the Direct FT model across all evaluated datasets. (2) Robustness to hard negatives: the curve of RAG FT is generally flatter than that of the chat model, which demonstrates that our finetuned LLM is more robust the hard negatives as the number of retrieved passages increases. (3) Superiority over direct fine-tuning: In most cases, RAG FT demonstrates superior performance compared to Direct FT. This indicates that RAG FT not only enables the LLM to "memorize" knowledge during training but also equips it with the ability to effectively "extract" relevant information from retrieved context during inference. These findings highlight the effectiveness of RAG-specific tuning in enhancing the generalization capabilities of LLMs for knowledge-intensin tasks. Separate results on those three LLMs are shown Appendix J, K and I. Qualitative studies be found in Appendix
7

---

Published as a conference paper at ICLR 2025
0.775
0.60
0.50
RAG Accuracy
0.750
0.55
0.45
0.725
0.50
0.40
0.700
0.45
0.40
0.35
0.650
RAG
0.35
RAG
0.30
0.625
0.600
025
10 15 25
35
10
20
25
35
10 15
25 30
40
# Retrieved Passages
Retrieved Passages
# Retrieved Passages
(a) TriviaQA
(b) PopQA
(c) HotpotQA
0.40
0.7
RAG Accuracy
0.400
RAG Accuracy
RAG Accuracy
0.375
0.30
0.350
0.25
0.5
0.300
0.4
0.15
0.275
0.250
0.225
5
10
15
20
25
30
35
40
5
10
15
20
25
30
35
10 15 20 25 30 35 40
#
Retrieved
Passages
Retrieved
Passages
# Retrieved Passages
(d) 2wikimultihopqa
(c) Bamboogle
(I) ASQA
0.70
RAG Accuracy
0.60
0.65
RAG Accuracy
Gemma-2-906
RAG
0.55
0.5
Gemma-2 Direct FT
0.50
Mistral-Namo-12 B-
0.45
Misoral-Nemo RAG
Mistral-Namo Direct
0.40
0.2%
Gemini-1.0PP
Gemini RAG IT
5 10
13
20
25
30
35
40
0.1
5
10
15
20
25
30
35
Gemini Direct
# Retrieved Passages
#
Retrieved Passages
(g) T-REx
(h) zsRE
(i) Legend
Figure 5: Generalization ability LLMs fine-tuned with RAG-specific data (RAG FT). RAG FT consistently outperforms the chat LLM w. RAG and the model fine-tuned directly pairs (Direct FT). This demonstrates the effectiveness of RAG FT in enabling the LLM to effectively extract knowledge from retrieved context unseen tasks. Note that Direct FT is evaluated without retrieval to align with its training paradigm and all others are evaluated with retrieval augmentation. (LLMs: Gemma-2-9B-Baan Mistral-Nemo- 12B-Base, Gemini-1.0-Prat
crestion-answer
5.2 ENHANCING RELEVANCE IDENTIFICATION THROUGH REASONING AUGMENTATION
While the fine-tuning approach described in Section 5.1 implicitly enhances the LLM's robustness to hard negatives, it does not explicitly train the model to differentiate between relevant and irrelevant passages within the retrieved context. address this, we investigate the effectiveness incorporating an intermediate reasoning step into the fine-tuning process.
This modified paradigm involves training the L.LM to generate both a reasoning paragraph (r) that explicitly identifies the relevant passages for the given query (g) and the final answer (a):
Input: d1, d2, d2-1- dR, q] Output: a].
During training, the LLMs are provided with labeled reasoning paragraphs to guide its learning process. During inference, the I.MSs are instructed to first generate the reasoning paragraph and then utilize this analysis produce the answer. This approach aims to explicitly enhance the LLMs' ability to discern relevant information from noise within the retrieved context, thereby improving its overall performance in RAG.
We utilize the same training data mixture as in Section 5.1 and augment with reasoning labels generated by Gemini- - Pro for each question-passagee pair. These labels provide explicit guidance on identifying relevant passages. We use e5 as the retriever and Wiki- 18 as the corpus. Further details of the experimental setup and the generation of reasoning labels can be found in Appendix H.
8

---

Published as a conference paper at ICLR 2025
0.80
RAG Accuract Accuract
0.6
0.775
RAG Accuracy
0.5
0.70
0.4
0.4
0.35
0.60
0.30
025
0.50
0.2
020
10 5
15 20
30 35
10 5
15
25 30
35
25 30 # Retrieved Passages
10 15
20
40
#
Retrieved Passages
# Retrieved Passages
(a) TriviaQA
(b) PopQA
(c) HotpotQA
0.0
RAG Accuracy
RAG Accuracy
0.7
Gemma-2-98 Saamma---9BCha-
0.50
0.6
RAG
0.45
Gemma-2 RAG FT w. Int.
0.5
Direct Gemini-1.0-Prg
0.35
0.4
0.30
0.3
Gemini RAG FT w. Int
0.25
0
5
10
15
20
25
30
35
40
5
10
15
20
25
30
35
# Retrieved Passages
# Retrieved Passages
(d) 2wikimultihopqa
(e) ASQA
(1) Legend
Figure 6: Evaluating the impact of intermediate reasoning on the performance RAG-tuned LLMs. Results demonstrate that fine-tuning with an intermediate reasoning step (RAG FT w. Int) leads to further improvements compared to implicit RAG fine-tuning (RAG FT) and direct fine-tuning (Direct FT). Direct FT is evaluated without retrieval to align with its training and all others are evaluated with retrieval augmentation. Due to the computational complexity of inference with reasoning augmentation, results shown for 1000 randomly-sampe queries from each dataset. (LLMs: Gemma-2-9B-BB 2-9B-Base and Gemini- 1.0-Pr, more results Appendix J, K and L)
Figure demonstrates the effectiveness of this approach. The LLM fine-tuned with explicit interme- diate reasoning consistently outperforms training with implicit RAG data. This improvement can be attributed two key factors: (1) Explicit relevance training: Providing intermediate reasoning labels during training explicitly teaches the LLM to differentiate between relevant and irrelevant passages, enhancing its ability discern crucial information from noise. (2) Structured reasoning for enhanced understanding: Generating a reasoning paragraph before answering introduces structured approach processing the retrieved context. This step, akin chain-of-thouge reasoning (Wei al., 2022), helps decouple the complex information and facilitates a more focused analysis, ultimately leading to improved performance. These highlight the value of incorporating explicit reasoning mechanisms in RAG tuning to enhance the LLM's ability to effectively utilize retrieved context. More results on Gemma-2-90 models, Mistral-Nemo- 12B models and Gemini-1. 1.0-Pro models are shown in Appendix J, K and L. Qualitative studies and cost-benefit analysis can be found in Appendix and P.
PERSPECTIVES ON FINE-TUNING LLMS FOR RAG
Impact of training data distribution on generalization. We first examine how the distribution of training data affects the generalization of the fine-tuned LLM. We train LLMs on five different data distributions, each with 50k samples: (1) mixed dataset comprising NQ, WoW, Fever, and MMLU (12.5k samples from each); (2) NQ only; (3) WoW only; (4) Fever only; and (5) MMLU only.
Figure 7(a) demonstrates that mixed distribution of training data leads to superior generalization performance on unseen RAG tasks compared to training on a single data source. This highlights the importance of data diversity enhancing the adaptability of L.L.Ms to new RAG scenarios.
Influence of retrievers on generalization. In real-world RAG deployments, LLMs might be paired with different retrievers depending on specific external knowledge corpus and retrievers' capabilities. To investigate the impact of different retrievers fine-tuning, we explore three adaptation scenarios on NQ: finettuning with (1) passages retrieved by BM25 (FT w. BM25); (2) passages retrieved by e5 (FT e5); and (3) mixture of passages retrieved both BM25 and e5 (FT mix). We evaluate the performance of these fine- tuned LLMs using both retrievers seen during training (BM25 and c5) and unseen retrievers (Contriever (Izacard et al., 2021) and BGE (Chen al., 2024a)).

---

Published as a conference paper at ICLR 2025
0.8
0.61
No FT
RAG Accuracy
0.45
RAG Accuracy
0.7
Accuracy
0.600
FT W. c5
0.35
Mixed data
0.6-
FT mix
0.59
25% max
NO only
0.58-
50% max
only
0.5
0.57
100% max
0.25
Fever only
0.100% max
Miniu only
0.56
50- 50-100%
5
10
15
20
25
30
35 40
B122 0U1 192 Inference retriever
Arg
10 15
20 Retrieved Passages
25 30 35 40
# Retrieved Passages
(a) Analysis of training data distri- (b) Influence of retriever variations (c) Investigation of the optimal bution. (Test: HotpotQA)
on fine-tuning effectiveness. (NQ) number of passages for training.
Figure 7: (a) Impact of training data distribution: A diverse mix of training data sources enhances the generalization ability of the LLM. (b) Influence of the retriever choice: Fine-tuning with data retrieved from multiple retrievers improves generalization to unseen retrievers during inference. (c) Effect of training context length: Fine-tuning with the maximum context length yields optimal performance across varying numbers of retrieved passages during inference. (LLM: Gemma-2 2-t-na------Baaa 9B-Base)
Figure 7(b) presents the results, revealing three key findings: (1) Superiority of mixed retriever training: Fine-tuning with the data corresponding to mix of retrievers consistently yields the best performance across both seen and unseen retrievers during inference. This suggests that training on a diverse set of retrieved passages enhances the LLMs' ability to adapt to different retrieval strategies and knowledge sources. (2) Retriever-agee agnossic improvement: Our proposed RAG-speciff tuning consistently enhances LLM performance across all retrievers, demonstrating the generalizability of our approach. (3) Retriever similarity and generalization: The generalization ability of an LLM fine-tuned with a specific retriever is influenced by the similarity between the training retriever and the inference retriever. For instance, an LLM trained with BM25 generalizes better to Contriever, while an LLM trained with e5 generalizes better to BGE. This observation suggests that "hard negatives" exhibit different characteristics depending on the employed retriever, and training with a specific retriever implicitly equips the LLM better handle similar types of hard negatives. See Appendix for a detailed analysis of retriever similarity.
Optimizing training for variable retrieval sizes. In real-world RAG systems, the number of retrieved passages can vary depending on the specific knowledge source and user requirements. Therefore, essential to determine the optimal training strategy for LLMs to ensure robust performance across different retrieval sizes during inference. We investigate this aspect with the Gemma-2-9B-BBl model, which has a maximum input sequence length of 8192 tokens (correspond- ing to approximately 40 passages). We evaluate five different training configurations: (1) Fixed 10 retrieved passages (25% max). (2) Fixed 20 retrieved passages (50% max). (3) Fixed 40 retrieved passages (maximum input capacity) (100% max). (4) Dynamic 0-40 retrieved passages (0-100% max). (5) Dynamic 20-40 retrieved passages (50-100% max).
Figure 7(c) presents the results on NQ, demonstrating that fine-tuning with the maximum number of retrieved passages (100% max) consistently yields the best performance across various retrieval sizes during inference. This suggests that training with the full context capacity enhances the LLM's ability to effectively handle varying amounts of retrieved information, leading to improved generalization and robustness. More analyses of RAG-specific tuning can be found in in Appendix M and N.
7 CONCLUSIONS
This paper investigates the impact of increasing the number of retrieved passages on the performance of long-context LLMs in retrieval-augmenter generation (RAG) systems. Contrary to expectations. we observe that performance initially improve but then degrade as more passages are included. This phenomenon attributed to the detrimental influence of retrieved "hard negatives". To mitigate this issue, we propose and evaluate three solutions: training- free retrieval reordering, RAG-specific implicit LLM fine-tuning, and RAG-oriented LLM fine-tuning with intermediate reasoning. A systematic analysis of the training-based methods explores the effects of data distribution, retriever for training, and training context length. Interesting future directions include exploring (automated) position optimization with more advanced retrieval ordering methods, and fine-tuning the LLMs for RAG with more ane-grained and multi-step reasoning chains. More discussion on the limitation and future works can be found in Appendix S.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Rishabh Agarwal, Avi Singh, Lei M Zhaan, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint urXiv:2404.11012 2024. 2
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. The Twelfth International Conference on Learning Representations, 2024. 3,
Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, al. Factu- ality challenges in the era of large language models. arXiv preprint arXiv: 201.05//89 2023. 1,
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionaliiyy multi-granularity text embeddings through self-knowledge distillation. arXiv preprint urXiv:2402.03211 2024a.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 17754-17762, 2024b.
Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 719-729, 2024. 3
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:230..02334 2022. 2
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. estrieval-augmentrr generation for large language models: A survey. arXiv preprint arXiv:2312. 10997, 2023. 1,2
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint 14Xx:22404.066544 2024a. 3,
Cheng- Yu Hsich, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yo Bien-Yulo Lee, Ranjay Krishna, al. Found the middle: Calibrat- ing positional attention bias improves long context utilization. arXiv preprint arXiv:2406. 1600. 2024b.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023. 1,2
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXhv:2112.091188 2021. 2,
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 3
Gregory Kamradt. Needle in haystack pressure testing 1lms, 2023. URL https: : :/t::t///iitii thub. com/ EkkamradtLLLMTesee Sckaanaettt.tttttteeeeeeeeaaaaar /aain Accessed: 2024-09-10. 10.10.01100000
3,6 6
11

---

Published as a conference paper at ICLR 2025
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04990 04900, 2020. 7, 7,
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019. 3
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprini arXiv:2406. 13121, 2024. 1,
Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. Making large language models a better foundation for dense retrieval. arXiv preprint arXiv:2312.15505 2023. 37
Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval augmented generation or long-context Ilms? a comprehensive study and hybrid approach. arXiv preprint arXiv:2407.16833, 2024. 2.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmentee dual instruction tuning. In The Twelfth International Conference on Learning Representations, 2024.
3
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. 2, 3,
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024. 1.3
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4)3333-38, 2009. 3
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference Machine Learning, pp. 31220-31122 PMLR, 2023. 3
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08295, 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 20880:2408.00118 2024b.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervise contrastive pre-training. arXiv preprint arXiv:2212.03353,, 2022. 2,3
Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. Beyond the limits: A survey of techniques to extend the context length in large language models. arxiv preprint arXiv:2402.02244, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 25:24824-24837, 2022.
Zhepei Wei. Wei-Lin Chen, and Yu Meng. Instructrag: Instructing retrieval-augmentte generation with explicit denoising. arXiv preprini arXiv:2406. 13629, 2024.
12

---

Published as a conference paper at ICLR 2025
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking Serreval-augmente generation for medicine. arXiv preprint arXiv:2402.13118, 2024. 38
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025, 2023. 2.
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling, Corrective retrieval augmented generation. arXiv preprint arXiv:2401./5884 2024. 3
Ori Yoran, Tomer Wolfson, Oni Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Repre- sentations, 2024. 3,3
Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. Evaluation of retrieval- augmented generation: A survey, arXiv preprint arXiv:2405.07453 2024a.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210, 2023.
Yue Yu, Wei Ping. Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retrieval-augmeet generation in llms. arXiv preprint arXiv:2407.022. 002485, 2024b. 3
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey, arXiv preprint ar0ivv2300. 10792, 2023. 3
Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, lon Stoica, and Joseph E Gonzalez. Raft: Adapting language model to domain specific rag. arXiv preprint arXiv:2403. 10131, 2024. 3
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.
2023. 1.2
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42(4):1-60, 2024. 2
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024. 3
Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.000 007704074 2021. 2
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A RETRIEVER PERFORMANCE AND SIMILARITY
We analyze the performance and similarity of four retrievers (BM25, contriever, e5 and bge) on the NQ dataset shown Figure 8. Each data point corresponds to retrieval (recall, precision) pair for specific number of retrieved passages. The overall retrieval performances on NQ are observed as e5 > bge contriever bm25, with contriever having similar performance with BM25 and bge having similar performance with e5 (as their curves are closer).
contriever
bge
Precston
0.2
0.1
0.3
0.4
Figure 8: Retriever performance on NQ. (1) Retrieval performance: e5 bge contriever BM25; (2) Contriever is more similar to BM25, while bge is more similar to e5 (since their curves are closer respectively).
B LONG CONTEXT LLMs IN RAG ANALYSIS ON OTHER DATASETS
In addition to the analysis presented on the NQ dataset in Section 3, we conduct further studies on the PopQA dataset to underscore the generality of our findings.
B.1
THE EFFECT RETRIEVED CONTEXT SIZE RAG PERFORMANCE
0.7
0.7
Gemma- 78- Chat
Gemme-2-98-Chon
0.6
0.6
Mistral--eme-128-4e Nemn--------2--ttnn
Gemiri-1.5.Poo
Accuracy
RAG Accuracy
0.5
0.3
Gemma-T0h
0.3
Matral-Namoo-28--rsaie
02
Gemiri-1.5.PP Pro
0.2
16
Passages
Retrieved Passages
(a) RAG performance with retriever
(b) RAG performance with BM25 retriever
Figure 9: Impact of retrieved context size on RAG performance (on PopQA) with 4 different LLMs. Increasing the number of retrieved passages initially improves performance but then leads to decline. This degradation is more pronounced using retriever (e5) that exhibits higher recall on PopQA compared to BM25 (Recall @40 is 0.85 with e5 and 0.57 with BM25). The maximum number of retrieved passages varies across LLMs due to differences their maximum token limits.
14

---

Published as a conference paper at ICLR 2025
Observations. Figure 9 presents the following key observations similar that in Section 3.1: 1) Strong Retriever (e5): Across all LLMs, increasing the number of retrieved passages initially enhances performance, but subsequently results in either sharp decline or plateau. 2) Weak Retriever (BM25): Performance generally shows a continuous improvement or a slighter decrease as the number of retrieved passages increases. While these observations may appear counter-intuitive given that one might expect monotonic improvements due to higher recall (i.e., greater chance of retrieving relevant information) the inclusion of additional passages can reduce precision, with irrelevant or misleading passages detracting LLMs from overall performance.
B.2
THE IMPORTANCE OF HARD NEGATIVES FOR LONG-CONTEXT LLM EVALUATION
0.35
0.775
0,800
Precision
0.30
e5
RAG Accuracy
0.725
0.750
UM25
RAG Accuracy
0.775
0,750
8AG Accuract
0.80
0.78
0.25
0.725
BM25
0.76
BM25
contriever
0.700
contriever
contriever
0.74
0.650
0.72
0.15
0.625
0.650
0.600
0.625
25
10 25
15 30 35 40
Recall
Retrieved Passages
Retrieeed Passages
Retrieved Passages
(a) Retrievers
(b) Gemma2-9B-- -B--hhtt
(c) Diistal-12B-Instruue
(d) Gemini- 1.5ini--..5-Pro
Figure 10: Evaluating the impact hard negatives on long-context LLMs. (a) The retriever perfor- mance on PopQA dataset: e5 > contriever > BM25. (b) For each query, single golden passage (containing the correct answer) combined with varying numbers of hard negative passages retrieved by different methods (e5, Contriever, BM25, and random sampling). The LLMs are then tasked with answering the query based on this context. This setup allows us to assess the robustness of LLMs to hard negatives and the influence of retriever strength their impact.
Observations. Figure 10 shows the following observations similar to that in Section 3.3: (1) Sensitivity to hard negatives: Across all LLMs, increasing the number of hard negative passages generally results in decline in RAG answer accuracy. (2) Retriever strength and hard negative difficulty: The strength of the retriever is directly correlated with the difficulty of the retrieved hard negatives. LLMs struggle more with hard negatives generated by stronger retrievers (e.g. e5) compared to those produced by weaker retrievers (e.g., BM25) or through random sampling. (3) Distinguishing random and hard negatives: While all the LLMs demonstrates robustness random negatives, it remains susceptible to the influence of hard negatives.
15

---

Published as a conference paper at ICLR 2025
C ILLUSTRATION OF SECTION 3.3: HARD NEGATIVE STUDY
Algorithm Data Construction for Hard Negative Study
Require: Query q. instruction I, golden passage dgold golden answer a, retrieved passages da..d...... dx] with decreasing retriever relevance scores, desired number of passages K (K & N).
Ensure: Input sequence S. 1: Initialize list S - [dgota]]
2: for each passage in do
3: if d; * dgoud and not in di then
4: Append d, S
5: end if
6: if [S] = K then
7: break
8: end if
9: end for
10: Randomly shuffle S.
11: Construct input sequence
12: return The input sequence [1,5[1].
16

---

Published as a conference paper at ICLR 2025
D HARD NEGATIVES CASE STUDY
In this section, we provide a case study to compare the hard negatives returned by different retrievers. We classify the negative passages into two types: (1) Related but Irrelevant: passages related to some entities mentioned in the question but not containing the ground truth answer; (2) Not Related: passages not related the question all. Note that Related but Irrelevant passages are harder and more misleading to the LLMs compared with Not Related passages. We show the top-5 negatives from each retriever for a random sampled question as below:
Question
The south west wind blows across Nigeria between?
Ground Truth
Till September
Retrieved Hard Negative Passages (high retrieval score but lacking ground truth answer)
W. c5
Doc 1 (Title: "Geography of Nigeria") south atlantic ocean, locally known the south western wind, or by its main name, The Tropical Maritime (MT) airmass. These two major wind systems Nigeria are known as the trade winds, The tropical maritime airmass (MT) is responsible for Nigeria's rainy season. This wind (the tropical maritime airmass) invades the country from February in the southern part of Nigeria while it takes longer for the wind to fully cover the whole of the country, reaching the northern part of Nigeria in June. Its invasion as a result the northward retreat, ... [Related but Irrelevant]
Doc 2 (Title: "Onikwu")... ... The dry season is accompanied by dust laden airmass from the Sahara Desert, locally known Harmattan, by its main name, The Tropical Continental (CT) airmass, while the rainy scason is heavily influenced by an airmass originating from the south Atlantic Ocean, locally known as the south west wind, or by its main name, The Tropical Maritime (MT) airmass. These two major wind systems in Nigeria are known the trade winds, The region Onikwu/Ndoni is flood prone communities, this is because the inland part of Rivers state consists of tropical rainforest ... [Related but Irrelevant]
Doc 3 (Title: "Geography of Nigeria") ... northern end is south of the 15 degrees line about 14 degrees. Nigeria's 's location in the wetter part of the easterly waves south of the 15 degree line creates wetter climatic conditions for Nigeria especially during the monsoons. The Tropical Continental Airmass (CT) locally known as the harmattan, is wind originating from North Africa which crosses the Sahara Desert into west Africa to Nigeria. This airmass dominates Nigeria's 's climate during the dry season from December March. The Tropical continental airmass dusty and creates haze within the atmosphere of west Africa and Nigeria when it predominates.. [Related but Irrelevant]
Doc 4 (Title: "Nigeria") ... Niger, Chad, Cameroon, and has coastline of least Nigeria lies hetween latitudes 4 and 14N, and longitudes 2 and 15E. The highest point in Nigeria is Chappal Waddi at The main rivers are the Niger and the Benue, which converge and empty into the Niger Delta. This is one of the world's largest river deltas, and the location of a large area of Central African mangroves. Nigeria has varied landscape. The far south is defined by its tropical rainforest climate, where annual rainfall is year. the southeast stands the /Related but Irrelevant]
Doc 5 (Title: "Geography of Nigeria") ... Nigeria, has temperature range of to and annual rainfall of about with a single rainfall maxima in September. The single Dry season experienced this climate, the tropical savanna climate central Nigeria beginning from December to march, is hot and dry with the Harmattan wind, a continental tropical (CT) airmass laden with dust from the Sahara Desert prevailing throughout this period. With the Intertropical Convergence Zone (ITCZ) swinging northward over West Africa from the Southem Hemisphere in April, heavy showers coming from pre- mre-monnooaal convective clouds mainly in the form of [Related but Irrelevant]
17

---

Published as a conference paper at ICLR 2025
bm25
"Oron people")... Civil War. Oron is found in the flood plain of South Eastern Nigeria, with the land mainly intersected by numerous streams and tributaries flowing into Cross River. The entire coastline stretches from Uya Oron to Udung Uko. Oron is in the tropical region and has uniformly high temperature all the year round. The two main seasons are the dry which spans between October and April and wet September. There are also two prevailing winds the South- West onshore winds which brings heavy rains and the [Not Related]
Doc (Title:
which starts around May and ends in
Doe (Title: "South Equatorial Current").. ... is driven directly by the trade winds which blow from South Equatorial Current is well-developed only south the equator. Directly on the equator, the winds reverse twice year due to the monsoons, and so the surface current can be either eastward or westward. South Equatorial Current Ocean current the Pacific, Atlantic, and Indian Ocean that flows east-to-west Pacific Atlantic
west. In the Indian Ocean, the westward-flooe flowing
between equator about 20 degrees south. Occans, it extends across the equator to about 5 degrees north. Within the southern hemisphere, the South Equatorial /Related but Irrelevant]
"Wind direction") ... Wind direction Wind direction is reported the direction from which originates. For example, ""northerly"" "" wind blows from the north to the south. Wind direction is usually reported in cardinal directions or in azimuth degrees. Wind direction is measured in degrees clockwise from due north. Consequently, wind blowing from the north has wind direction of 0; wind blowing from the east wind blowing from the south has a wind direction of 180; and a wind blowing from the west has wind direction of 270 /Related but Irrelevant]
Doc (Title:
wind direction of 90;
Doc 4 (Title: "Gulf Stream") ... this current interacts with the northeastern coast South America, the current forks into two branches. One passes into the Caribbean Sea, while a second, the Antilles Current, flows north and east of the West Indies. These two branches rejoin north of the Straits of Florida. The trade winds blow westward in the tropics, and the westerlies blow eastward mid-late mid-latitudes. This wind pattern applies stress to subtropical ocean surface with negative curl across the north Atlantic Ocean. The resulting Sverdrup transport is equatorward. Because of conservation of potential vorticity caused by the northward-movin winds the subtropical... [Not Related)
Doc (Title: "Climate of the United Kingdom") ... climate that western parts the UK experience. The high latitude and proximity to large ocean to the west means that the United Kingdom experiences strong winds. The prevailing wind is from the south-west but may blow from any direction for sustained periods time. Winds are strongest near westerly facing coasts and exposed headlands. Gales which are defined as winds with speeds of are strongly associated with the passage of deep depressions across the country. The Hebrides experience on average 35 days of gale year (a day where there are gale-force winds) while inland ... [Not Related]
18

---

Published as a conference paper at ICLR 2025
random sampling
Doc 1 (Title: "Queen of Peace, Bray") of Bugisi in Tanzania for number of years. The parish has very close relationship with St Cronan's 's B.N.S., Scoil Chualann and Gaclscoil Ui Chéadaigh. The parish provides the sacraments of Communion and Confirmation to the children in the schools. They also help to raise funds for the twin parish of Bugisi. Queen of Peace, Bray The Queen of Peace is Catholic church situated the junction of the Putland Road and the Vevay Road in Bray, Co. Wicklow, Ireland. The present church was built 1946 by TJ Macken of St Patrick's Street, Dún Laoghaire, [Not Related]
Doc (Title: "Cordova Congressional Internship Program") Puerto Rico's Constitutional Convention from 1951 to 1952. By 2012, over 670 students from colleges and universities Puerto Rico had enjoyed internships under the program, and the Spring 2009 class included record 24 members. A private sector committee, recently headed by Univision Puerto Rico president Larry Sands, provides private funds supplement the 350,000 annual grant provided by the Puerto Rico Legislative Assembly. Under the auspices of TWC. seventeen states have since established similar legislative-fundee internship programs. The Center established in 2008 the McClintock Award to the State Legislator of the Year.. (Not Related]
Congressional
Doe (Title: "V bomber")... ... Puerto Rico's 's Constitutional Convention from 1951 to 1952. By 2012, over 670 students from colleges and universities in Puerto Rico had enjoyed internships under the program, and the Spring 2009 class included record 24 members. private sector committee, recently headed by Univision Puerto Rico president Larry Sands. provides private funds to supplement the 350,000 annual grant provided by the Puerto Rico Legislative Assembly. Under the auspices of TWC, seventecn states have since established similar rggislative-fuddes Congressional internship programs. The Center established in 2008 the McClintock Award to the ... [Not Related]
Doc 4 (Title: "Defence Materials and Stores Research and Development Establishment") materials for the Indian Armed Forces. DMSRDE has developed Nuclear Shiclding Pad, Boot Anti Mine, Blast Protection Suit, Bullet Proof Jackets, etc.. ""The Defence Material Stores Research Development Establishment Kanpur has developed new NBC suit that would proved effective against any kind of dangerous weapons or chemicals and protect soldiers from any sort of attack," DMSRDE Director Arvind Kumar Saxena was quoted by media-persons. 40,000 pieces of NBC suits costing about Rs 30,000 had been requested by Indian army. ""the further progress on the other two suits are going on. "" further (Not Related]
Doe 5 (Title: "Chess title") retain the title of Candidate Master, if carned according to criteria above). This is in contrast to international titles awarded by FIDE, which are awarded for life. In European countrics the term of ""expert"" is not used. Instead, players of that level are called ""Candidate Masters"". although the FIDE Candidate Master title generally requires higher rating (2200 FIDE). It possible (and common), however, for players in the United States to have a rating that places them the 'expert' category while still retaining the title of LLif Master" or 'National Master' [Not Related]
From the case study, we can find that the negatives retrieved by e5 contain more Related but Irrelevant passages compared with those retrieved by bm25, while those retrieved by bm25 still have more Related but Irrelevant passages than random sampling. This qualitatively demonstrates that the hardness of the negatives from different retrievers as e5 > bm25 > random.
19

---

Published as a conference paper at ICLR 2025
E RETRIEVAL REORDERING
Algorithm 2 Retrieval Reordering Algorithm
Require: Query instruction 7, retrieved passages D [d], d2. ..,, dy} with decreasing retriever
relevance scores.
Ensure: Reordered sequence S.
Initialize an empty list S of length k.
2: for = 1 to & do
3: if' mood(i, 2) = then
+
4: Order(d; d:) - 2
is odd}
5: else
6: Order(d, (d;) - k + 1 1
(i is even)
2
7: end if'
8: Place d, position Order(d:) in
9: end for
Construct input sequence [T] S[k], 4).
11: return The reordered sequence [I,], S|2], S[k].g].
F DATASETS
In this section, we discuss the datasets for RAG-specific LLM training and evaluation.
F.1 TRAINING DATASETS
Dataset
the number instances
Natural Question
12,500
Wizard of Wikipedia
12,500
FEVER MMLU
12,500
12,500
Table 2: Training data statistics.
We select a series of fine-tuning data designed to enhance the model's robustness to hard negatives in the retrieval context and improve its contextual awareness in generating predictions. The training data are from four sources with different answer types: Natural Question (short-form). Wizard of Wikipedia (long-form), FEVER (true/false), and MMLU (close-set). The statistics of the training data mix can be found in Table 2.
F.2 TESTING DATASETS
To comprehensively evaluate our methods, we select testing datasets across different tasks including: (1) Question-anssering: TriviaQA, PopQA, WebQuestions; (2) Multi-hop tasks: HotpotQA, 2Wiki- MultiHopQA, Bamboogle; (3) Long-form tasks: ASQA; (4) Slot filling: T-REx, Zero-shot RE. The statistics of all the datasets can be found in Table 3.
F.3 RETRIEVAL CORPUS
Following Karpukhin et al. (2020), we use the text chunks from 2018 Wikipedia dump the retrieval corpus. The articles are split by section, where long sections are further split into chunks equal sizes and contain less than 100 words, leading to total of 21M text chunks.
20

---

Published as a conference paper at ICLR 2025
Datasct
Task
the number of instances
TriviaQA
QA
11,313
PopQA
14,267
WebQuestions
QA
2,032
HotpotQA
Multi-Hop QA
7,405
2WikiMultiHopQA
Multi-Hop QA
12,576
Bamboogle
Multi-Hop QA
125
ASQA
Long-form QA
T-REx
Slot filling
5,000
Zero-shot shot RE
Slot filling
3,724
Table 3: Testing data statisties.
G IMPLICIT RAG FINE-TUNING EXPERIMENTAL SETTING
G.1 TRAINING SETTINGS
Hyperparameters. We use the top-40 retrieved text chunks for given example to generate the fine-tuning samples and use e5 as the retriever for the main results. We fine-tune both Gemma-2-9B- Base and -eii--eeeo----B-aee
using 8 H100 GPUs. For both models, use the chat template corresponding to Gemma-2-99 2-9hhhhtt and Distral-Nemo- 1omno-1BB-oe 1-B- Instructt respectively when tuning the models. We use the axolot codebase for their tuning. For Gemini-1.0-PP tuning, we the Google Cloud Tuning API with the default settings. The hyperparameters can be found in Table 4.
Model
peak Ir
Ir scheduler
warm up
# epoch
batch size
Flash Att
Gemma-2-99 2--------------
Ic-6
cosine cosine
4
64
False
Mistral-Nemo-12B-- 12B-Bee
le-6
5%
4
64
True default
Gemini-1.0-Pra
default
default
default
1
default
Table Implicit RAG finetuning hyperparameters.
Training RAG instruction templates. The RAG instruction templates for different training datasets can be found in Table 5.
Task
Instruction Templates
NQ
Answer the question based on the given document. Only give me the answer and do not output any other words. The following given documents. (reference)
Wizard of Wikipedia
Provide response the conversation based the given document. The following given documents. (reference)
FEVER
Verify fact based given documents The following are given documents. (reference)
MMLU
Given a question, choose the answer from options based on the given documents.
The following are given documents. (reference)
Table 5: Training instruction templates for implicit RAG tuning.
Phttps://gi wtlp:tccoooooooooollll--o-o---- pottps://clot http://w..ccmm/eertaaaa/////e/ea / /occs /ooeel reference/ tuning
21

---

Published as a conference paper at ICLR 2025
Training RAG answer templates. The RAG answer templates for different training datasets be found in Table 6.
Task
Answer Templates
NQ
Question:
{question]
Answer:
Wizard of Wikipedia
Conversation:
[question]
Response:
FEVER
Return SUPPORTS if it is correct and return REFUTES if it is not correct.
Fact: [question]
Response:
MMLU
Question:
[question]
Options: {choices)
Answer:
Table 6: Training answer templates for implicit RAG tuning.
G.2 EVALUATION SETTINGS
Hyperparameters. For all the compared LLMs, we conduct top-p sampling (p 1) and the maximum number of generated token is set to be 32. For Gemma-2 series models, we use the huggingface inference pipeline For Gemini series models, we use Google Cloud Inference API3, While for other series of LLMs, utilize VLLM codebase for efficient generation.
Evaluation RAG instruction templates. The RAG instruction templates for different testing datasets can be found in Table 7.
Task
Instruction Templates
QA
Answer the question based the given document. Only give me answer and not output any other words. The following are given documents. [reference]
Multi-hop
Answer the question based on the given document. Only give me the answer and do not output any other words. The following given documents. [reference]
Long-form
Answer the question based the given document. Please give in-depth explanation and avoid only returning answer. The following are given documents. [reference]
Slot filling
Given question, choose the answer from the options based on the given documents.
The following are given documents. [reference)
Table 7: Testing instruction templates for implicit RAG tuning.
Evaluation RAG answer templates. The RAG answer templates for different testing datasets are all: "Question: {question). Answer:"
httppee///oggiiraaa..oooooooetrr hhttps://clo oud. htt.ood..oooo.....oeeeeeeeeeeeee wot.c///eeee-aaaageeeeeeee ve- htt///oc///maee /odeel reference/ inference
'https://gg //gi thgt. com ccom///l1mprrjjec//ee
22

---

Published as a conference paper at ICLR 2025
H RAG FINETUNING WITH INTERMEDIATE REASONING EXPERIMENTAL SETTING
H.1 TRAINING SETTINGS
Hyperparameters.
The hyperparameter setting is the same to that in Appendix G.I.
Training RAG instruction templates.
The RAG instruction templates with intermediate reasoning for different training datasets can be found in Table 8.
Task
Instraction Templates
NQ
Anewer the question hased given decuments.
Pleuse lirst provide an unalysis with cleur reasoning details which documents are relevant to answer the question. Then output concise The following are given documents. [reference]
to the question based the analysis.
Wikipedin
given documents. Please first provide un analysis with cleur reasoning detuils of which documents are relevant to provide the response. output response question based on analysis.
response conversation
The following are given documentss. [reference)
FEVER
Verify a fact based on the given documents.
Please first provide analysis with clear reasoning details of which documents relevant verify the fact. Then output a concise answer (SUPPORTS or REFUTES) based on the analysis. The following given documentts. [reference]
MMLU
Given question, choose Please first provide analysis with clear reasoning details which documents are relevant to answer question Then outpui a concise answer to the question bused on the analysis. The following given documents. [reference)
from the
based on the documents.
Table 8: Training instruction templates for RAG tuning with intermediate reasoning.
Training RAG Answer templates. be found in Table 9.
The RAG answer templates for different training datasets can
Task
Answer Templates
NQ
Question:
{question]
Answer:
Wizard Wikipedia
Conversation:
(question)
Response:
FEVER
Return SUPPORTS it is correct and return REFUTES it not correct.
Fact: {question)
Response:
MMLU
Question:
{question]
Options: {choices)
Answer:
Table 9: Training answer templates for RAG tuning with intermediate reasoning.
23

---

Published as a conference paper at ICLR 2025
Instructions to generate intermediate reasoning from Gemini-1.5-pro. The prompt that guides Gemini-1.5-pro for intermediate reasoning generation can be found in Table 10.
Task NQ
Prompts
Read the following documents relevant the given question: [question)
rreference)
Please identify documents that useful and explain how the contents lead to the answer: [answers).
the given question: [question).
If none in that case, have to explaio the answer only based on your own knowledge.
documents aligned with the answer.
without referring the provided information.
Note that the question he compositionl and require intermediate analysis dedace the final Make sure your response is grounded and provides clear reasoning details followed by a concise conclusion.
Wizard of Wikipedia
Read the following documents relevant the given conversation: (question)
[reference]
Please identify documents that are useful to provide response to conversation: (question)
and explain how the contents lead the response: [answers].
If none of the documents is aligned with the response. case, you have to explain response only based on your knowledge, without referring the provided information.
Make sure your response grounded and provides clear reasoning details followed by a concise conclusion.
FEVER
Read the following documents relevant given question: [question)
identify documents [reference)
useful verifyy fact: [question]
(Retum SUPPORTS if it is cerrect and return REFUTES if it is not correct.). and explain how the contents lead If none the documents is aligned with the in that case, you have to explain the answer only based on your own knowledge without referring
answer: [answers).
provided information.
Make sure response is grounded and provides clear reasoning details followed by concise conclusion.
MMLU
Read the following documents relevant to the given question: [question) [reference]
Please identify documents that are useful to answer the given question: [question) with options: (choices). answer: [answers).
and explain If none of the documents is aligned with the answer,
contents
in that case, you have to explain the answer only based on your own knowledge, without referring Make sure your response is grounded and provides clear reasoning details followed by a concise conclusion.
provided information.
Table 10: Prompts guide Gemini-1, 1.5-i- for intermediate reasoning generation.
24

---

Published as a conference paper at ICLR 2025
H.2 EVALUATION SETTINGS
Hyperparameters. For the compared LLMs, we conduct top-p sampling (p 1) and the maximum the huggingface
number of generated token is be 256. For Gemma-2 series models, inference pipeline. While for other series of LLMs, we utilize vLLM codebase for efficient generation.
Evaluation RAG instruction templates. The RAG instruction templates for different testing datasets can be found in Table 11.
Task
Instruction Templates
QA
Answer the question based on the given documents. Please first provide analysis clear reusoning details of which documents relevant question Then output concise answer to the question based on the analysis. The following are given documents. [rfference]
Multi-hop
Answer the question based on the given documents.
Please first provide an analysis with clear reasoning details of which documents are relevant to answer question Then output concise answer the question based the analysis. The following are given documents.
(reference] ]
Long-form
Answer the question based
given document.
Please first provide an analysis with clear reasoning details of which documents are relevant to answer the question Then provide an in-depth long- form answer question (avoid only returning answer) on the analysis, The following are given documents. rreference
filling
Provided an answer to the given slot filling question based on the given document. In the question, the words before and after [SEP) correspond the head entity and relation respectively. You are asked to output the tail entity corresponded to the given head entity and relation. relevant to the question.
Please first provide analysis with clear reasoning details of which documents Then output concise answer to the question based on the analysis.
The following are given documents. (reference)
Table 11: Testing instruction templates for RAG tuning with intermediate reasoning.
Evaluation RAG answer templates. The RAG answer templates for different testing datasets are all: "Question: (question). Answer:"
25

---

Published as a conference paper at ICLR 2025
1 DATA-AUGMENTED RAG CASE STUDIES
Question
Which film features the Dawes Tomes Mousley Grubbs Fidelity Fiduciary Bank?
Ground Truth
Mary Poppins
Retrieved Passages
Doe (Title: "Fidelity Fiduciary Bank") Fidelity Fiduciary Bank ""Fidelity Fiduciary Bank"" is song from Walt Disney's 's film ""Mary Poppins"", "", and is composed by Richard M. Sherman and Robert B. Sherman. The song sung by the stodgy old bankers the ""Dawes, Tomes, Mousely, Grubbs Fidelity Dawes"" "M (Nackvid Keyed). to George
Fiduciary Bank "". by ""Elder Banks's s two children, Jane and Michael, in an attempt to get Michael Banks to invest his tuppence in the bank. As the song continues the pressure is on George, junior clerk the bank, to sway Michael. When Michael finally, and ever slightly, opens
Doc 2 (Title: "Fidelity Fiduciary Bank") convince Michael to invest his money goes like this: has as its origins an essay by C. C. Turner titled 'Money London' the book edited by G. R. Sims called ""Living London"" (London: 1903): Fidelity Fiduciary Bank ""Fidelity Fiduciary Bank"" is song from Walt Disney's 's film ""Mary Poppins"" "" and composed by Richard M. Sherman and Robert B. Sherman. The song sung by the stodgy old bankers at the "'Dawes, Tomes, Mousely, Grubbs Fidelity Fiduciary Bank"", led by the ""Elder Mr. Dawes"" (Nackvid Keyed), to George Banks's 's two children, Jane Michael, attempt to get...
Doc 40 (Title: "The Firm (soundtrack)") The Firm (soundtrack) The Firm is an album American pianist Dave Grusin released in 1993, recorded for the GRP label. This album soundtrack the Motion Picture "'The Firm" "" directed by Sidney Pollack. It reached No. on ""Billbarrd"'ss Contemporary Jazz chart. This soundtrack was a 1993 Academy Award nominee for Best Original Score. All tracks composed by Dave Grusin; except where indicated In 2015 La-La Land Records issued remastered and expanded edition, featuring the film score on disc one the 1993 soundtrack album bonus tracks on disc two. Italicised tracks on disc one
Model Predictions
Gemma-2-9R-Char
Fidelity Fiduciary Bank
Implicit RAG FT
Mary Poppins (1964 film)
RAG FT with Reasoning
Documents 1 and 2 are useful to answer the question.
Both documents explicitly state that the song "Fidelity Fiduciary Bank" from the musical film "Mary Poppins features the bank "Dawes, Tomes, Mousely, Grubbs Fidelity Fiduciary Bank".
Therefore, the film that features the mentioned bank is Mary Poppins.
26

---

Published as a conference paper at ICLR 2025
Question
Which English chemist discovered the most elements?
Ground Truth
Humphry Davy
Retrieved Passages
Doc 1 (Title: Chemistry) J.J. Thomson of Cambridge University discovered the electron and soon after the French scientist Becquerel well the couple Pierre and Marie Curie investigated
phenomenon radioactivity. a series of pioneering scattering experiments Ernest Rutherford at the University of Manchester discovered the internal structure of the atom and the existence of the proton, classified and explained the different types of radioactivity and successfully transmuted the first element by bombarding nitrogen with alpha particles. His work on atomic structure was improved on by his students, the Danish physicist Niels Bohr and Henry Moseley. The electronic theory of chemical
Doe (Title: Chemistry) weights. The development the electrochemical theory of chemical combinations occurred the carly 19th century as the result of the work of two scientists in particular, J.J. Berzelius and Humphry Davy, made possible by the prior invention of the voltaic pile by Alessandro Volta. Davy discovered nine elements including the alkali metals by extracting them from their oxides with electric current. British William Prout first proposed ordering all the elements by their atomic weight as all atoms had a weight that was exact multiple of the atomic weight of hydrogen. J.A.R. Newlands devised early table of...
Doe 40 (Title: "History of chemistry") many had already applied the concept of atomism to chemistry. A major example was the ion thcory of Svante Arrhenius which anticipated ideas about atomic substructure that did not fully develop until the 20th century. Michael Faraday was another early worker, whose major contribution to chemistry was electrochemistry. which (among other things) a certain quantity of electricity during electrolysis or electrodeposition of metals was shown to be associated with certain quantities of chemical elements, and fixed quantities of the elements therefore with each other, in specific ratios. These findings, like those of Dalton's 's combining ratios, were early clues ...
Model Predictions
Gemma-2-9B-Cnn
William Hyde Wollaston
Implicit RAG FT
William Ramsay
RAG FT with Reasoning
The best document to answer this question is Doc 2(Chemistry)ee, This document states that **Humphry Davy** "discovered nine new elements including the alkali metals by extracting them from their oxides with electric current."
Therefore, the answer Humphry Davy.
While other documents mention chemists and their discoveries, none of them contradict or provide a higher number of discovered elements than Davy's nine. This makes Doc the most useful in identifying the English chemist who discovered most elements.
It's important to note that this answer is based on Doc 2 alone. If other documents mentioned an English chemist who discovered more than nine elements, they would supersede the information in Doc 2. However, without that information, we can confidently say that based these documents, Humphry Davy discovered the most elements among English chemists mentioned. ##Cooluuion:: Based on the given documents, Humphry Davy is the English chemist who discovered the most elements.
27

---

Published as a conference paper at ICLR 2025
J DATA-AUGMENTED RAG FINETUNING ON GEMMA-2-9B
In Figure 5, we illustrate the performance of implicit RAG finetuning on eight datasets with three different base models due the space limitation. The whole results with Gemma-2-9B models can be found in Figure 11.
0.78
RAG Accuracy
0.76-
0.55
RAG Accuracy
0.45
0.74
0.50
0.70
0.72
RAG IT
Gemma-2-00
0.45
RAG
Gemma- (aamma---35-coo Chat
0.4
0.40
0.68
Gemma-2-98-CMn
0.66
0.3
RAG FT
0.64
0.25
0.5
5
15 20
30
15
5
10
20
25 35
40
25 30 35 40 Retrieved Passages
10
Retrieved Passages
Retrieved Passages
(a) TriviaQA
(b) PopQA
(c) HotpotQA
0.50
0.40
RAG Accuracy
RAG Accuracy
0.35
0.36
0.38
0.46
RAG FT
Dommma-2-9Bccom
0.0
RAG
Gemma.2-88 2aama----9B-hat
0.34
0.25
0.32
Gemma-2-98 Chat
0.42
RAG FT
Direct
0.15
35
25 Retrieved Passages
Retrieved Passages
Passages
(d) 2wikimultihopqa
(c) Webquestions
(1) Bamboogle
0.70
0.7
RAG Accuracy
0.65
RAG Accuracy
0.60
0.6
0.60
Gemma-2-98.com
0.55
0.50
0.4
RAG FT
Gemme-2-98-Chal
0.45
Gem-ma-2-6--hat
RAG
0.3
0.45
Direct
Direct FT
0.40
13
Retrieved Passages
Retrieved Passages
Retrieved Passages
ASQA
(h) T-REx
(i) zsRE
Figure 11: Generalization ability LLMs fine-tuned with RAG-specific data (RAG FT). RAG FT consistently outperforms the chat LLM RAG and the model fine-tuned directly question question-answer answer pairs (Direct FT). This demonstrates the effectiveness of RAG FT in enabling the LLM to effectively extract knowledge from retrieved context unseen tasks. Note that Direct FT is evaluated without retrieval to align with its training paradigm and all others are evaluated with retrieval augmentation. (LLM: Gemma-2-9B-- 2-9--Base)
Figure 6, we show the power RAG finetuning with intermediate reasoning on five datasets because of the space limitation. The whole results all the nine datasets with Gemma-2-9p models can be found in Figure 12. Note that due to the computational complexity of inference with reasoning augmentation, results shown for 1000 randomly- sampled queries for each dataset.
28

---

Published as a conference paper at ICLR 2025
0.55
0.80
RAG Accuract
0.6
0.775
RAG Accuracy
0.70
040
Gemma-2-9B-Chhn
0.4
ma-2-9B--hat
0.35
Gemma-2-98-Cha
0.60
RAG FT
0.3
RAG FT
RAG
0.30
RAG FT
025
0.50
Direct FT
0.2
Direct
020
Direct FT
5 15 20
30 35 40
5 10 15 25 30 35
40
10 15 25 30 40 # Retrieved Passages
# Retrieved Passages
# Retrieved Passages
(a) TriviaQA
(b) PopQA
(c) HotpotQA
Gemma-2-966
0.40
RAG Accuracy
RAG FT
RAG Accuracy
0.35
0.50
0.50
0.45
Direct FT
0.45
0.30
0.5
0.40
Semma-2-98-Ch)
Gemma 2 98-Chtat
0.35
RAG
0.0
RAG
0.30
RAG FT Int
0.15
RAG w. Int
0.25
0.25
Direct
0.10
Direct FT
0
5
10
15
20
25
30
35
40
5 10
20 25 30 35 40 Retrieved Passages
15
10 15 20 25 30 35 40 # Retrieved Passages
# Retrieved Passages
(d) 2wikimultihopqa
(c) Webquestions
(1) Bamboogle
0.8
0.7
RAG Co
0.7
RAG Accuracy
RAG Accuracy
Gemma-2-98-Chai
0.6
RAG
Gemma-2-98-CNM
0.5
Gemma-2-96-CNn
0.3
0.4
Direct
RAG w. Int
0.4
RAG FT w. Int
RAG FT Int
02
0.3
Direct
0.1
0
5 10
15
20 25 30 35
40
5
30 35 40 Retrieved Passages
10 15
20 25
10
15
20
25
30
35
40
# Retrieved Passages
#
# Retrieved Passages
(g) ASQA
(h) T-REx
(i) zsRE
Figure 12: Evaluating the impact of intermediate reasoning the performance of RAG-tuned LLMs. Results demonstrate that fine-tuning with an intermediate reasoning step (RAG FT w. Int) leads to further improvements compared implicit RAG fine-tuning (RAG FT) and direct tuning (Direct FT). Direct FT is evaluated without retrieval to align with its training paradigm and all others evaluated with retrieval augmentation. Due to the computational complexity of inference with reasoning augmentation, results are shown for 1000 mandomly-sampless queries from each dataset. (LLM: 1aamma---9B-Bann
29

---

Published as a conference paper at ICLR 2025
K DATA-AUGMENTE RAG FINETUNING ON MISTRAL-NEMO-
In addition to the comprehensive data-augmented RAG fine-tuning results with three different base LLMs reported in Section 5, we also would like show the results specifically with the Mistral- Nemo-12B models in Figure 13.
0.80
0.60
RAG Accuracy
0.75
0.50
0.55
0.45
0.70
0.40
0.65
Mistral-Nermo-1 2h-Chet
0.40
Mistral-Nemo- 1ilemo--28--har Ch6-Chat
0.35
Mistral-Nemo-1286 12
0.35
RAG
RAG FT Int
RAG FT W. Int
025
RAG FT Int
35 40 020
Direct
15 20
30 35
5 10
20 25
10
25 30 35 40
Retrieved Passages
Retrieved Passages
Retrieved Passages
(a) TriviaQA
(b) PopQA
(c) HotpotQA
0.60
Mistral-Nemmo-1200
RAG Accuracy
0.50
RAG FT
RAG Accuracy
0.35
0.45
0.30
0.40
Direct FT
0.45
025
0.35
RAG
0.40
Mistra-Nemoo-28-c00
RAG FT
0.15
Mistral-Nmmo-1288 RAG FT
RAG FT Int
RAG FT
0.25
Direct FT
0.10
Direct
0.05
35
25 Retrieved Passages
Retrieved Passages
Passages
(d) 2wikimultihoqqa
(c) Webquestions
(1) Bamboogle
0.70
0.65
0.7
RAG Accuracy
0.60
Mistral-Nemo-128
0.5
RAG FT
0.50
0.4
0.5
RAG FT
0.45
Mistra-Nemm-122-Chhal
RAG FT
RAG
Direct
02
Direct FT
Direct FT
0.30
30
15
Retrieved Passages
Retrieved Passages
Retrieved Passages
(g) ASQA
(h) T-REx
zsRE
Figure 13: Evaluating RAG-specific tuning with Mistral-Nemo-11 12B models. Results demonstrate that fine-tuming with an intermediate reasoning step (RAG FT w. Int) leads further improvements compared to implicit RAG fine-tuning (RAG FT). while implicit RAG fine-tuning outperforms LLMs without RAG-specific tuning (Mistral-Nemo- 12B-1BBBB (2BCCht) and direct fine-tuning (Direct FT). Direct FT is evaluated without retrieval to align with its training paradigm and all others are evaluated with retrieval augmentation. (LLMM: Mistral-Nemo- - 12B-Base)
30

---

Published as a conference paper at ICLR 2025
L DATA-AUGMENTED RAG FINETUNING ON GEMINI-1.0-PRG
In addition to the comprehensive luta-aug RAG fine-tuning results with three different base LLMs reported in Section also would like show the results specifically with the Gemini-1.0 .0- Pro models in Figure 14. Due to the Gemini-1 1emini--,OO-Pro .0-Prr API call credit limitation, we random sample 1000 queries for each dataset.
RAG Accuracy
0.80
Accuracy
0.60
RAG Accuracy
0.50
0.55
0.45
0.76
0.40
Gemini-1.0-PM
0.45
Gemini-11.-PP
Gemini-1.0:PP
0.74
RAG Int
RAG
0.35
RAG
RAG
RAG w.
Direct FT
0.35
Direct FT
0.30
Direct FT
5 10 15 20 25 30 35 40 # Retrieved Passages
5 10 20 25 as 40 # Retrieved Passages
10 15 20 25 30 40 # Retrieved Passages
(a) TriviaQA
(b) PopQA
(c) HotpotQA
0.375
0.575
Gemini-1.0-PO
RAG Accuracy
0.50
RAG FT
0.350
0.45
RAG FT
1-0-Pr--1.O-Pro
0.500
0.525
Direct FT
0.3325
0.40
Int
0.475
Gemini-110-Pro
Direct
RAG
0.425
RAG w.
0.30
0.400
0.200
Direct
0.375
5
10
15
20
25
30
35
40
5 10
15
20
25
30
35
40
10 15
20
25
30 35 40
Retrieved Passages
Retrieved Passages
Retrieved Passages
(d) 2wikimultihopqa
(c) Wehquestions
(I) Bamboogle
0.75
0.7
RAG Accurac)
0.70
RAG Accuracy
0.6
0.6
Gemini-1.0--P
Gemini-1.O-Poo RAG W. Int Direct
RAG
0.5
RAG FT w.
0.55
Gemini-1.0PPo
0.4
0.4
Direct FT
RAG
0.50
RAG w. Int
RAG
0.3
0.3
02
$
10
15 20 25 30 25
40
5 10 15 20 25 30 35 40
10
15
20 25
30 as 40
# Retrieved Passages
Retrieved Passages
# Retrieved Passages
(g) ASQA
(h) T-REx
(i) zsRE
Figure 14: Evaluating RAG-specifi tuning with Gemini-1.0 1.0-Pro models. Results demonstrate that fine-tuning with an intermediate reasoning step (RAG FT w. Int) leads to further improvements compared to implicit RAG fine-tuning (RAG FT), while implicit RAG fine-tuning outperforms LLMs without RAG-specific tuning (Gemini-1.. 1.0Prr)) and direct fine-tuning (Direct FT). Direct FT is evaluated without retrieval to align with its training paradigm and all others evaluated with retrieval augmentation. Due to the Gemini- 1.0-Pr API call credit limitation, results are shown for 1000 aadomml-samplle queries from each dataset. (LLM: Gemini-1.00 1.0-Pro)
31

---

Published as a conference paper at ICLR 2025
M
TRAINING DATA SCALING AND RAG PERFORMANCE.
Number Retricval Passages
5k
20k
50k
200k
0.5942
0.5925
0.6058
0.6277
20
0,5909
0.5925
0.6078
0.6294
30
0,5787
0,5792
0.6072
0.6150
40
0.5582
0.5582
0.5859
0.5983
Avg.
0.5805
0.5806
0.6017
0.6176
Table 14: Impact of RAG-specific training data scale on LLM performance in RAG.
To investigate the influence of the size of the training data on the effectiveness of RAG-specific tuning, we fine-tune the Gemma-2-9B 1attmma-2-9BBBaat 9-BBse model using varying amounts (5k to 200k samples) of mixed training data from NQ. WoW, Fever, and MMLU, Table 14 presents the evaluation results on the NQ dataset, demonstrating a clear positive correlation between the scale of training data and the performance of the resulting LLM in RAG. Increasing the amount of training data consistently leads to improved accuracy, highlighting the benefits of leveraging larger datasets for fine-tuning LLMs in RAG applications.
N RAG-SPECIFIC TUNING DATA INSIDE SFT MIXTURES
Dataset
base
SFT only
SFT RAG-FT
MT Bench
2.3125
5.8969
5.6031
NQ
0.2105
0.5687 0.7155
0.6033 0.7481
TriviaQA
0.4940
Table 15: Combining RAG-specific data with general SFT data for enhanced LLM performance in RAG.
Having established the effectiveness of RAG-specific fine-tuning for improving LLM performance in RAG tasks, we now investigate whether combining RAG-specific data with general SFT data can further enhance performance while preserving the LLM's general capabilities (e.g., reasoning and long-form generation), as a way to assess the potential the proposed tuning methods be useful for construction of foundation models. We train the Gemma-2-9B model using two different strategies: (1) SFT data only: The LLM trained solely on general SFT data (Ultrachat 200k). (2) SFT data + RAG-specific data: The LLM is trained on combination of Ultrachat 200k and 50k RAG-specific data (the same data used in Figure 5). We evaluate the resulting models on MT Bench to assess their general language capabilities and on NQ and TriviaQA measure their RAG performance.
Table 15 presents the results, demonstrating that incorporating RAG-specific data into the SFT process significantly improve the LLM's performance on RAG tasks while maintaining its performance on general language tasks. This finding suggests that combining task-specific and general- data during fine-tuning can be viable strategy for enhancing LLMs specialized applications without compromising their overall capabilities.
32

---

Published as a conference paper at ICLR 2025
O CASE STUDIES OF FAILURE QUERIES
In this section, we conduct case studies for queries which can be correctly answered by RAG with small number of retrieved passage (1), but wrongly answered by RAG with larger number of retrieved passages (5).
Case 1:
Question
When do primary ossification centers appear in an embryo?
Ground Truth
Prenatal development
RAG with retrieved passage
Prediction
Prenatal development
Retrieved Passages
Doc (Title: "Ossification center") of the same row also becomes calcified, and transverse bars of calcified substance stretch across from calcareous column another. Thus there are longitudinal groups the cartilage cells enclosed oblong cavities, the walls of which are formed of calcified matrix which cuts off all nutrition from the cells; the cells, in consequence, atrophy, leaving spaces called the primary arcolæ. There two types of ossification centers primary and secondary. primary ossification center is the first of bone start ossifying. usually appears during prenatal development central part of each developing....
RAG with retrieved passages
Prediction
Fetal development
33

---

Published as a conference paper at ICLR 2025
Retrieved Passages
Doc 1 (Title: "Ossification center") of the same also becomes calcified. and transverse bars calcified substance stretch across from one calcarcous column to another. Thus there are longitudinal groups of the cartilage cells enclosed in oblong cavities, the walls of which are formed of calcified matrix which cuts off all nutrition from the cells; the cells, in consequence, atrophy, leaving spaces called the primary arcolæ. There secondary. primary ossification center is the first area of bone to start ossifying. usually appears during prenatal development in the central part of each developing
types of ossification centers primary
Doe 2 (Title: "Ossification center") bone. In long bones the primary centers occur the diaphysis/shaft and in irregular bones the primary centers occur usually in the body of the bonc. Most bones have only one primary center (e.g. all long bones) but some irregular bones such the os coxa (hip) and vertebrae have multiple primary centers. A secondary ossification center the area of ossification that appears after the primary ossification center has already appeared most of which appear during the postnatal and adolescent years. Most bones have more than one secondary ossification center. In long bones, the sccondary centres appear in
Doe (Title: "Primary bone") )) Primary bone Primary bone the first hone tissue that appears in embryonic development and in fracture repair. is characterized by its random position of collagen fibers. In most places in adults this tissue is replaced by secondary bone tissue except, for example. calvara or tooth sockets. The secondary bones have lower amounts of osteocytes so primary bone is much more casily penetrated by x-ray. Primary bone or the primary ossification center is the beginning of the bone building process during the first trimester. Calcificed cartilage is basophilic and new bone being made is more
the sutures of
Doc 4 (Title: Ossification) small opening in the diaphysis. It invades the primary center of ossification, bringing osteogenic cells (osteoblasts on the outside, osteoclasts the inside.) The canal of the nutrient foramen directed away from more active end of bone when one end grows more than the other. When bone grows same rate at both ends, the nutrient artery is perpendicular the bone. Most other hones (c.g. vertehrac) also have primary ossification centers, and bone is laid down in similar manner. Secondary centers The secondary centers generally appear the epiphysis. Secondary ossification mostly occurs after birth (except for...
Doc 5 (Title: Bone) development of the periosteum. Endochondral ossification occurs in long bones and most other bones in the body; it involves the development of bone from cartilage. This process includes the development of cartilage model, its growth and development, development of the primary and secondary ossification centers, and the formation of articular cartilage and the epiphyseal plates. Endochondral ossification begins with points in the cartilage called primary ossification centers." "" They mostly appear during fetal development, though a few short bones begin their primary ossification after birth. They are responsible for the formation of the diaphyses of long bones, short bones and
Case 2:
Question
What is the final season of downton abhey?
Ground Truth
The sixth series
RAG with I retrieved passage
Prediction
The sixth series
34

---

Published as a conference paper at ICLR 2025
Retrieved Passages
Doc 1 (Title: "Downton Abbey") both ITV and PBS, and subsequently became the most successful British costume drama series since the 1981 television serial of '"Biieshhcad Revisited"". On 26 March 2015, Carnival Films and ITV announced that the sixth series would be the last. It aired on ITV between 20 September 2015 and 8 November 2015. The final episode, serving as the annual Christmas special,
broadcast on 25 December 2015. A film adaptation was confirmed on July 2018. The first series, comprising seven episodes, explores the lives of the fictional Crawley family, the hereditary Earls of Grantham, and their domestic servants. The storyline...
RAG with retrieved passages
Prediction
Between 20 September 2015 and 8 November 2015
Retrieved Passages
"Downton Abbey") both ITV and PBS, and subsequently became the most successful British costume drama series since the 1981 television serial of Briideshheadd Revisited"". On 26 March 2015, Carnival Films and ITV announced that the sixth series would be the last. It aired ITV between 20 September 2015 and November 2015. The final episode, serving as the annual Christmas special, was broadcast on 25 December 2015. film adaptation was confirmed on 13 July 2018. The first series, comprising seven episodes, explores the lives of the fictional Crawley family, the hereditary Earls of Grantham, and their domestic servants. The storyline...
Doc (Title:
Doe (Title: "Downton Abhey (film)") Downton Abhey (film) Downton Abbey is an upcoming British historical period drama film, written by Julian Fellowes and directed by Michael Engler. It is continuation of the television series of the name, created film scheduled to be released on September 2019 the United Kingdom, and one week later in the United States on 20 September. The film is follow-up to the television series of the same name, which ended its original run in December 2015 after 52 episodes. In April 2016, it was
Fellowes, that
ITV from 2010 2015. The
Doe (Title: "Downton Abhey") ") Focus Features and Universal Pictures International. The film is scheduled for a UK release on 13 September 2019, with the US following one week later on 20 September 2019. Downton Abbey Downton Abbey is British historical period drama television series set in the early 20th century, created by Julian Fellowes. The series first aired on ITV the United Kingdom on 26 September 2010, and in the United States on PBS, which supported production of the series as part of its ""Masterpiece Classic"" anthology, on January 2011 1. The series, set the fictional Yorkshire country estate of Downton
"Downton Abbey") Downton Abbey Downton Abbey is a British historical period drama television series set in the early 20th century, created by Julian Fellowes. The series first aired ITV in the United Kingdom September 2010, and in the United States PBS, which supported production of the series as part of its ""Maaterpiecee Classic" anthology, on 9 January 2011 1. The series, set in the fictional Yorkshire country estate of Downton Abbey between 1912 and 1926, depicts the lives of the aristocratic Crawley family and their domestic servants in the post-Edwardhig era-with the great events in history having an effect ...
Doc 4 (Title:
26
Doc 5 (Title: "Downton Abbey") Mary, but she puts him off when Lady Grantham becomes pregnant. Cora miscarries after O' Brien, believing she be fired, retaliates by kicking broken bar of soap near the bathtub steps, making Cora slip and fall, killing her unborn child which had been a boy. The series ends just after the assassination of Archduke Franz Ferdinand and the outbreak of the First World War in July 1914. The second series comprises eight episodes and runs from the Battle of the Somme in 1916 to the 1918 Spanish flu pandemic. During the war, Downton Abbey is temporarily converted into
Case 3:
Question
When does the movie jeepers creepers come out?
35

---

Published as a conference paper at ICLR 2025
Ground Truth
September 26, 2017
RAG with 1 retrieved passage
Prediction
September 26, 2017
Retrieved Passages
Doc 1 (Title: "Jeepers Creepers 3") Jeepers Creepers 3 Jeepers Creepers 3 is a 2017 American horror film written and directed by Victor Salva and the third ""Jeepers Creepers"" film, taking place in between ""Jeepers Creepers"" and ""Jeepers Creepers 2. n. Jonathan Breck reprises his role as the Creeper. Gina Philips returns in cameo as Trish Jenner, her first return to the series since the original film. The film was shown in theaters on September 26, 2017 in what was originally announced as one-night-only showing and was then shown again on October 4, 2017. A shuriken flies through the air and hits wooden post
RAG with retrieved passages
Prediction
Between 20 September 2015 and 8 November 2015
Retrieved Passages
"Jeepers Creepers 3") Jeepers Creepers 3 Jeepers Creepers 3 is a 2017 American horror film written and directed by Victor Salva and the third ""Jeepers Creepers" "M film, taking place between ""Jeepers Creepers"" and ""Jeepers Creepers 2. "" Jonathan Breck reprises his role as the Creeper. Gina Philips returns in cameo as Trish Jenner, her first return to the series since the original film. The film was shown in theaters September 26, 2017 in what originally announced as one-night-only showing and then shown again on October 4, 2017. A shuriken flies through the air and hits a wooden
Doc 1 (Title:
Doe 2 (Title: "Jeepers Creepers 2")) spot still held by ""Jeapers Creepers"".
Allowing for films that had been released prior to Labor Day, '"Jeepers Creepers 2"" holds the 9 spot after the 2015 Labor Day four-day weekend. In September 2015, '"Jeepers Creepers 3 was officially greenlit. The film was slated to begin filming in April 2016 until production was halted when Victor Salva was boycotted from filming in Canada for his criminal past The film was eventually released in one- mee-iight-oll showing on September 26, 2017, 14 years after the release "" ""Jeepers Creepers 2"", grossed $2.3 million theaters. Jeepers Creepers 2 Jeepers Creepers...
Doe 3 (Title: "Jeepers Creepers 3") been planned. This more closely replicates the Florida setting of the original film. Principal photography hegan February 15, 2017 in Baton Rouge, Louisiana. On April 4, 2017, one of the cameramen of the production revealed on their social networks that filming had by then been completed. During an interview for the Edmond Sun, Justin Hall specifically revealed that the film would be released September 2017, but nothing was confirmed by Salva or the studio. On August 16, 2017, the AMC Theatres website stated that ""Jeepers Creepers 3 would open on September 26, 2017. On August 29, 2017.
Doe 4 (Title: "Jeepers Creepers (2001 film)") McCoy"" AKA ""Roach"" who is a car thief and regular the Poho County jail. In the second film, he portrays ""Coach Dwayne Barnes"". . On September 11, 2015, ""Jeepers Creepers 3"" was officially greenlit, with planned 2017 release. Victor Salva returns as director. Jonathan Breck returns The Creeper. and Gina Philips returns Trish Jenner. her first sereen role in five years. Production was halted in 2016 until it resumed February 2017, 7, and completed April. The film opened for what was said would be only a onc-night showing on September 26, 2017; it was then shown
Doc 5 (Title: "Jeepers Creepers (2001 film)") Jeepers Creepers (2001 film) Jeepers Creepers a 2001 American- Gerrann horror film written and directed by Victor Salva. The film takes its name from the 1938 song '"Jeepers Creepers"", which is featured in the film. Francis Ford Coppola executive produced, and the film stars Gina Philips. Justin Long, Jonathan Breck, and Eileen Brennan. Philips and Long play older siblings who hecome the targets of demonic creature (Breck) in nural Florida. Trish Jenner (Philips) and her brother Darry (I (Long) are traveling home from college for spring break. As they drive through the Florida countryside, an old rusty truck...
36

---

Published as a conference paper at ICLR 2025
Our analysis reveals no clear pattern distinguishing queries that can be effectively answered with small number of retrieved passages but fail when the retrieval set size increases. We hypothesize that these failure cases arise from complex interactions between the query, retriever, and knowledge corpus. Specifically, for a given query, if the retriever successfully identifies relevant information with small retrieval top-k, but increasing ls primarily introduces noise, the likelihood of failure in answering the query correctly with a larger retrieval set increases significantly.
P COST-BENEHIT ANALYSIS OF BASIC RERANKING AND OUR PROPOSED METHODS
To provide comprehensive assessment, we analyze the cost-benefit trade-offs between basic rerank- ing, retrieval reordering (Ours), and RAG-specific LLM tuning (Ours).
Basic reranking. Cost: Reranking involves processing every query in each testing dataset with a computationally expensive ross-encoder model. This process can be time- cone-consumnng, especially for large datasets. In our experiments, reranking the top 200 passages on the TriviaQA dataset (over 11k samples) took over 1.5 hours (with bge-reranker-large (Li et al., 2023)) using PataPlrallel and 8 H100s (while retrieval only (without reranking) with e5 and gpu-based approximated nearest neighbor search faiss-gpu only takes 10 mins). This cost scales linearly with dataset size, making it significant bottleneck for large-scale applications. Benefit: Reranking can improve retrieval accuracy, leading to better overall RAG performance, as demonstrated in our results above.
Retrieval reordering (Ours). Cost: This method is designed to be plug-and-play and can be seam- lessly integrated with any existing RAG pipeline (retriever, reranker, and generator) without incurring additional computational costs; Benefit:It consistently improves RAG performance, particularly when using& larger number of retrieved passages. Importantly, it is both retriever-agnostic and agrosti-,ggnoticc, demonstrating its versatility and broad applicability (as shown in Figure 4).
RAG-specific LLM tuning (Ours). Cost: Training an LLM requires computational resources and time. In our experiments, achieving a well-trained checkpoint took approximately 2 hours using DeepSpeed zero3 and H100s. Benefit: This method yields substantial and consistent improvements in the LLM's ability handle RAG scenarios across different numbers of retrieved passages (as shown in the table above). Crucially, this training is one-time cost; the resulting L.M can be deployed for any testing dataset without incurring further computational overhead during inference (retrieval with e5 and gpu-based approximated nearest neighbor search faiss-gpu only takes 10 mins on TriviaQA, while reranking with bge-reranker-large (Li et al., 2023) takes more than 1.5 hours).
It's important to highlight that these three approaches are orthogonal and address different aspects of the RAG system: Basic reranking focuses improving the quality of retrieved passages. Retrieval reordering optimizes the interaction between the retriever and the LLM generator. RAG-specific LLM tuning enhances the generator's 's capabilities for handling retrieved information. This inherent complementarity suggests that combining these techniques can lead to further performance gains in RAG systems.
INTUITION BEHIND RETRIEVAL REORDERING
The intuition behind retrieval reordering is that, average, hard negatives are ranked lower than relevant passages. To illustrate this, we conducted experiments on the NQ dataset with both e5 and BM25 retrievers, analyzing the top-A retrieved passages (k 5/10/20050/1000 100). We calculated the average ranking of both relevant passages and hard negatives, first averaging at the query level and then across all queries. The results are shown in Table 19 and Table 20.
Our findings consistently show that the average ranking of relevant passages is higher than that of hard negatives for both retrievers and across all top-k values. This observation supports the rationale behind our retrieval reordering strategy: by prioritizing higher-ranked passages, we increase the likelihood that the L.M focuses relevant information.
37

---

Published as a conference paper at ICLR 2025
# Retrieved Passages (Top-k)
10
20
50
100
Avg. Relevant Passage Ranking
2.497
4.171
7.406
16.638
30.826
Avg. Hard Negative Ranking
3.426
6.157
11.429
27.044
52.801
Table 19: Average rankings of relevant passages and hard negatives on NQ for different numbers of retrieved passages (retriever: e5).
# Retrieved Passages (Top-k)
10
20
50
100
Avg. Relevant Passage Ranking
2.696
4.808 5.784
8.609
19.615 26.262
36.252
Avg. Hard Negative Ramking
3.189
10.931
51.719
Table 20: Average rankings of relevant passages and hard negatives on NQ for different numbers of retrieved passages (retriever: BM25).
Furthermore, it's important to emphasize that the benefits of reordering extend beyond mitigating the impact of hard negatives. By placing higher-relevance passages in more prominent positions, we simultaneously reduce the influence of lower-relevance passages, leading to a more effective utilization of the retrieved information by the LLM.
R EXPERIMENTS ON MEDICINE DOMAIN RAG
We have expanded our analysis include RAG experiments on PubMedQA and BioASQ (Xiong al., 2024), utilizing PubMed as the retrieval corpus.
Retrieved Passages (Top-k)
20
PubMedQA
0.314
0.592
0.41 0.7767
0.43
0.46 0.7330
BioASQ
0.7524
0.7638
0.7735
# Retrieved Passages (Top-k)
30
40
50
100
150
PubMedQA
0,416
0.38
0.394
0.32
0.354
BioASQ
0.7120
0.6974
0.6505
0.6845
0.6942
Table Performance metries for PubMedQA and BioASQ datasets across different numbers of retrieved passages. (LLM: Histral-Nemo-122 Inn-sttuu
These new experiments confirm the presence of the "inverted U-curve phenomenon" with mistral- nemo-instruction as the generator across both datasets (with PubMed as the knowledge base and e5 as the retriever), further reinforeing our observations.
LIMITATIONS AND FUTURE WORK
In this paper, we first identify the inverse-U performance curve exhibited by Retrieval-Augmented Generation (RAG) systems as the number of retrieved passages increases. We then conduct systematic analysis understand the underlying causes of this phenomenon and propose three approaches to enhance the performance of long-context large language models (LLMs) in RAG applications. However, due to page limitations, we acknowledge that certain problems and solutions are only briefly addressed and warrant deeper exploration.
First, the robustness Gemini--.55PP to random negatives and lack an observable inverse-U curve remain insufficiently explained. We hypothesize that the quality and characteristics of its pretraining and post-training datasets play a critical role. Unfortunately. the absence publicly available information regarding the training data for these LLMs precludes definitive conclusions at this time. We recognize this limitation and encourage further research into the impact training data on retrieval robustness an important future direction.
hhups///uubmed.nhhii nlm nlm.nih.gov
38

---

Published as a conference paper at ICLR 2025
Second, propose straightforward yet effective solution: retrieval reordering. By strategically positioning high-scoring passages at the beginning and end of the input sequence, we leverage the known positional biases of LLMs, which tend to prioritize information in these regions. This introduces the concept of "position engineering" in RAG systems, highlighting the critical role of passage ordering. While this work focuses demonstrating the efficacy of this approach, we acknowledge the potential for exploring alternative reordering strategies and suggest this as promising avenue for future research.
Finally, we present two tuning-based methods to enhance the robustness of LLMs in RAG applications: (1) implicit RAG-specific tuning. and (2) RAG-specific tuning with intermediate reasoning steps. These methods aim to improve the alignment of LLMs with RAG tasks and their ability to handle noisy or redundant retrievals. Future work could explore more fine-grained and multi-step reasoning chains to further refine RAG performance and robustness.
39


Reviews:
[{"review_id": "hEAjwaieBL", "reviewer": "Reviewer_RnDx", "summary": "This paper studies how RAG systems are impacted by the advancement in long-context LLMs. The study empirically shows how the increase in the context-length (which allows more passages to be used) doesn't necessarily imply improved performance of the system. They show that the performance first improves before declining with increases number of retrieved passages. This is attributed to the presence of \"hard negatives\" which confuses the LLMs. The authors next propose 3 ways to counter the \"hard-negatives\" issue. The first is the reordering of the retrieved passages and it is shown that it can be an effective non-training based method. The other two approaches are training-based: finetuning with explicit hard-negatives, and finetuning with an additional reasoning step to train the LLM to learn to reject hard negatives. They show how these approaches help improve performance. A well-performed set of ablations help see the impact of data distribution match, retriever choice, and context length of LLMs during training.", "questions": "- Fig 1 also has an interesting observation where the Gemini-1.5-pro is quite robust to increasing number of retrieved passages and doesn't decline in performance. Is there a correlation between the context windows of these long-context LLMs and their ability to be robust to increasing number of retrieved passages? And if so, would \"infinite\"-context window LLMs have most robustness? It would be interesting to perform additional study around this.\n- The above is especially the case when answering the subsequent questions in the paper regarding the hard negatives. Since LLMs are not perfect processing blocks. Just because the correct one is present doesn't mean the LLM will get it right, and it depends on how good the LLM is at processing the given information.\n- Fig. 7b indicates that there might not be much impact on the choice of the retriever. More information on that would be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper focuses on answering a question that hasn't received rigorous attention yet: understanding if retrieval optimizations are not needed anymore in RAGs system if we have long-context LLMs. They approached it quite methodically by first trying to answer the question by looking at multiple LLMs (each with its own context length limit) and multiple retrievers and seeing the impact of the RAG system with increasing retrieved passages. The problem has lot of moving parts: LLM, context length limit, # of retrieved passages, choice retriever, evaluation and training datasets, to name a few. The authors have done a reasonably well job to balance these different parts to answer the question.", "weaknesses": "There are a few places where the observations can be better represented and new hypotheses can be developed and evaluated. Some examples are below (and in the \"questions\" section below):\n- In Fig 1, the authors claim that for strong retriever, the performance either reduces or reaches a plateau with increasing number of retrieved passages whereas it is different for weak retriever. However, I don't see any difference from the figure. Even for BM25 (weak retriever) the performance has inverted U shape similar to e5 (strong retriever).\n- The reordering algorithm assumes that hard negatives are typically lower in relevance and hence reordering allows them to be in the middle, therefore allowing the LLM to not get distracted by them. However, that's not necessarily true. They most likely would be at the top and hence are called \"hard\" negatives.\n- It was unclear until we look at the appendix on what was the knowledge base used for the experiments. It is also not clear (maybe my mistake) on what was the retriever used for experiments in Sec. 5.\n- A key issue in the experiments is that for all the datasets used, the knowledge base is wiki which is already a part of the LLMs pretraining data. So some of the observations made could be impacted by how much of the knowledge for answering the query is already in the LLM's weights and how much is from the retrieved passages. Since the answers to the query are most probably in the model weights due to pretraining data, the observations from the experiments could be flawed and we might not be able to make fair conclusions. It would be good if we can run these experiments with a knowledge base use for retrieval be something that is NOT a part of the LLM's pretraining data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tr0nUzI9eX", "reviewer": "Reviewer_vMLV", "summary": "This paper investigates the challenges and opportunities of using long-context LLMs in RAG systems. The main contributions include:\n- A systematic analysis demonstrating that increasing retrieved passages doesn't always improve performance with long-context LLMs\n- Identification of the detrimental impact of \"hard negatives\" on LLM performance\n- Three approaches to improve RAG performance: (1) a training-free retrieval reordering method, (2) Implicit robustness fine-tuning, (3) explicit relevance fine-tuning with intermediate reasoning.\n- Comprehensive analysis of design choices for RAG-specific LLM tuning.", "questions": "- What are the key properties of \"hard negatives\" that make them particularly challenging? Is it semantic similarity, structural patterns, or something else? How does your analysis generalize to other types of retrievers not tested in the paper?\n- How does your approach compare to recent methods like RetRobust (Yoran et al. 2023 - Making Retrieval-Augmented Language Models Robust to Irrelevant Context), or RA-DIT: Retrieval-Augmented Dual Instruction Tuning (Lin et al. 2023) that also aim to improve RAG robustness?\n- Have you considered comparing against hierarchical retrieval methods that might naturally mitigate the hard negatives problem?\n- Could you provide a cost-benefit analysis comparing your methods against simpler approaches like reducing retrieval size or basic reranking?\n- Are there specific types of queries or documents where hard negatives are particularly problematic?\n- Why did you choose the specific reordering strategy for the training-free approach? Have you explored other ordering patterns?\n- How sensitive is the intermediate reasoning approach to the quality and style of reasoning labels?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a technically sophisticated analysis of long-context LLMs in RAG systems, with comprehensive experiments across multiple models, retrievers, and datasets. The empirical validation is thorough and well-documented.\n- The identification of the \"hard negatives\" problem in long-context RAG represents a major contribution to understanding LLM behavior in retrieval contexts.\n- The proposed methods, particularly the training-free retrieval reordering approach, offer immediate practical value for improving RAG systems. The solutions are well-motivated by the empirical findings.", "weaknesses": "- The paper lacks to explain why long-context LLMs behave differently with hard negatives, which may limit the generalizability of the findings and makes it harder to predict when the proposed solutions might fail.\n- The experimental evaluation doesn't compare against recent competing methods for improving RAG performance, such as reranking approaches or hierarchical retrieval methods.\n- The paper lacks an analysis of failure cases or limitations. When do these methods break down? What types of queries or documents are particularly challenging?\n- The computational requirements of the intermediate reasoning approach requiring Gemini-1.5-Pro are not thoroughly discussed. This raises questions about the practical applicability in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtTNWsVD1I", "reviewer": "Reviewer_k49Q", "summary": "This paper primarily explores the phenomenon of long inputs in Retrieval-Augmented Generation (RAG): for many long-context large language models (LLMs), the quality of generated output initially improves but then declines as the number of retrieved passages increases. This observation contradicts the previous assumption that a larger retrieval set, containing more relevant information, would enhance performance. The study identifies the detrimental impact of retrieved \"hard negatives\" as a significant factor. To mitigate this effect and enhance the robustness of long-context LLMs in RAG, the paper proposes three methods: (1) an untrained method based on retrieval reordering, (2) implicit robustness fine-tuning to enhance resistance to hard negatives, and (3) explicit relevance fine-tuning with intermediate reasoning to identify relevance.", "questions": "See Questions", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The phenomenon where the quality of LLM-generated outputs first improves and then declines with an increasing number of retrieved segments is intriguing, and the authors have conducted thorough experiments to explore this.\n2. The importance of hard negatives for long-context LLM evaluation is well-studied, and it is observed that increasing the number of hard negatives typically leads to a decrease in RAG answer accuracy. The strength of the retriever is directly related to the difficulty of the hard negatives.\n3. The three methods proposed by the authors (Retrieval reordering, Implicit robustness fine-tuning, Explicit relevance fine-tuning) all enhance the performance of long-context LLMs in RAG applications.\n4. Overall, the paper is well-written and easy to follow.", "weaknesses": "1. Figure 1 shows that while other LLMs exhibit an initial improvement in performance followed by a decline with an increasing number of retrieved passages, Gemini-1.5-Pro does not exhibit this trend, and the authors do not explain this phenomenon.\n2. From the experiments, it appears that LLMs ranging from 7B to 12B are susceptible to the influence of hard negatives. It remains unclear whether this phenomenon can still be observed in LLMs larger than 30B. I suspect that larger LLMs might automatically identify these hard negatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hsx8Dj1gDj", "reviewer": "Reviewer_ua3n", "summary": "This paper investigates the effect of increasing the length of the retrieved document list on the performance of RAG. They find that simply increasing the number of retrieved documents does not consistently improve the performance of RAG, and both training-free and training-based methods are proposed to solve this problem.", "questions": "Please see Weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and easy to follow\n\n2. Experimental results show the effectiveness of the proposed methods.\n\n3. This paper makes significant contribution in the context of the rapid development of long-context LLM", "weaknesses": "In this rebuttal period, the authors address most of my concerns below\n\n1. **Wrong research basis**: In Abstract and Introduction Sections, this paper's description of the current research status of the RAG community is wrong, which directly leads to the fact that the research basis of this paper is wrong, which is unacceptable. Specifically, in Lines 14-16 of Abstract \"It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance\" and Lines 46-47 of Introduction \"Intuitively, longer context would allow for the inclusion of more retrieved passages, leading to higher recall and potentially improved performance\". Both claims misrepresent the current state of research in the RAG community. Many studies in RAG have shown that increasing the number of retrieved documents cannot consistently improve the RAG performance due to the noise or incorrectness of the retrieved texts [1, 2 ,3, 4, 5]. This paper wrongly uses a problem that has been clarified in the RAG community as the motivation, which is unacceptable.\n\n2. **Over-claiming of contributions**: As weakness 1 indicates that the robustness of RAG has been fully-studied in RAG [1,2,3,4,5] and shows irrelevant retrieved documents may interference the performance of RAG, the main contribution claimed in this paper that \"we systematically analyze the use of long-context LLMs in RAG systems, specifically examining the impact of retrieved \"hard negatives\" on performance\" cannot be seen as the contribution of this paper.\n\n3. **Limited technical contribution**：This paper claims that they propose two methods to enhance the robustness of RAG on \"hard negatives\". One method is a training-free method based on retrieval reordering, this method is incremental based on Lost-in-the-Middle [6]. The other method is explicit tuning with intermediate reasoning for relevance identification, this is similar to RetRobust [1] without distinguishable points.\n\n[1] Making Retrieval-Augmented Language Models Robust to Irrelevant Context\n\n[2] Chain-of-note: Enhancing robustness in retrieval-augmented language models\n\n[3] Corrective retrieval augmented generation\n\n[4] Evaluation of Retrieval-Augmented Generation: A Survey\n\n[5] Benchmarking large language models in retrieval-augmented generation\n\n[6] Lost in the Middle: How Language Models Use Long Contexts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)