LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing
Authors: Kuan Li
Liwen Zhang
Yong Jiang
Pengjun Xie
Fei Huang
Shuai Wang
Minhao Cheng
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Retrieval Augmented Generation
Benchmark
Long-context
Primary area: applications->language_speech_and_dialog

Abstract: As Large Language Model (LLM) context windows expand, the necessity of Retrieval-Augmented Generation (RAG) for integrating external knowledge is debated. Existing RAG vs. long-context (LC) LLM comparisons are often inconclusive due to benchmark limitations. We introduce LaRA, a novel benchmark with 2326 test cases across four QA tasks and three long context types, for rigorous evaluation. Our analysis of eleven LLMs reveals the optimal choice between RAG and LC depends on a complex interplay of model capabilities, context length, task type, and retrieval characteristics, offering actionable guidelines for practitioners. Our code and dataset is provided at:https://github.com/Alibaba-NLP/LaRA

LaRA: Benchmarking Retrieval-Augmenter Generation and Long-Context LLMs - No Silver Bullet for LC or RAG Routing
Kuan Li
Liwen Zhang²
Yong Jiang?
Pengjun Xie
Fei Huang? Shuai Wang' Minhao Cheng
Abstract
documents, knowledge bases, or websites. However, recent advancements in LLMs, such as GPT-4o (OpenAI, 2024), Llama 3.2 (Meta, 2024a), Claude 3.5 (Anthropic, 2024), and Qwen 2.5 (Yang et al., 2024), now support input lengths of up to 128k tokens, offering an alternative by directly feeding the full context of relevant information into the model. This raises questions about the continued necessity of RAG, which was initially crucial for handling long texts, since these models can now potentially access and process the necessary information directly. Therefore, it is essential to systematically compare the strengths and weaknesses of RAG and long-context (LC) LLMs.
As Large Language Model (LLM) context windows expand, the necessity of Retrieval- Augmented Generation (RAG) for integrating external knowledge is debated. Existing RAG vs. long-context (LC) LLM comparisons are often inconclusive due to benchmark limitations. We introduce LaRA, novel benchmark with 2326 test cases across four QA tasks and three long context types, for rigorous evaluation. Our analysis of eleven LLMs reveals the optimal choice between RAG and LC depends on a complex interplay of model capabilities, context length, task type, and retrieval characteristics, offering actionable guidelines for practitioners. Our code and dataset is provided at: https://github.com///li libaa- NLP/LaRA.
Numerous studies have investigated the performance differences between RAG and providing LLMs with full long contexts. For example, Xu et al. find that RAG outperforms LCC on several traditional QA datasets. More recently. Li et al. argued that LC consistently outperforms RAG in almost all settings. However, Yu et al. subsequently conducted experiments demonstrating that RAG not inherently weaker than LC. This lack of consensus likely stems from several flaws in the evaluation pipeline design of existing benchmarks, including issues with the corpus (e.g.. excessively short text lengths, failed replacements), evaluation metrics, and impractical task design.
1. Introducion
While large language models (LLMs) excel across various domains, the dynamic nature of information poses significant challenges to their ability to acquire new knowledge effectively. Current studies reveal several limitations of LLMs, including high computational costs when processing long texts, a tendency to produce factual errors and hallucinations, difficulty adapting to specialized domains, and a propensity for generating overly generic responses (Gao al., 2023). To address these limitations, generation (RAG) (Guu et al., 2020; Lewis et al., 2020). RAG enables LLMs to efficiently utilize external knowledge by retrieving the most relevant fragments from uploaded
To address these issues and facilitate a robust comparison between RAG and LCC, we propose LaRA, a benchmark for evaluating Long-context LLMs competing againt RAG, In constructing LaRA, we adhere the following criteria: (I) Context length should be maximized within the LLMs' input limits to avoid truncation that could obscure the true capabilities of the models. (2) The context should consist of naturally occurring long text, rather than artificially constructed examples, to reflect real-world usage scenarios. (3) Answering questions should require information from the provided context, ensuring the LLM cannot answer them based solely its internal knowledge. (4) Questions should have definitive answers to facilitate an accurate evaluation using LLMs. (5) The questions should reflect practical queries that humans are likely to ask in real-world L.M usage scenarios. LaRA designed to serve as a guidebook for designing effective RAG-or-LC planning systems, therefore, it is crucial that all included questions are meaningful,
researchers have explored eetrieval-augmentt
The project was done during Kuan Li's internship at Tongyi Lab, Alibaba Group. 'The Hong Kong University of Science and Technology 'Tongyi Lab, Alibaba Group PPenn State University. Correspondence to: Yong Jiang <yongjiang.jy @alibaba- inc.com>, Shuai Wang <shuaiw Ashuaiw@cse.ust.ie usc.ust.hk>. Minhao Cheng <mme7149@psu.c
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmented Generation
relevant, and reflect real-world information needs.
2. Revisiting RAG vs. Long Context
Benchmarks
Specifically, LaRA includes three of the most common types of long contexts-novee academic papers, and financial Statements-seleee to represent diverse writing styles and information densities. LaRA features four types of question- answering tasks designed to assess essential capabilities for handling long contexts: Locating specific information, Comparing different parts of the text, Reasoning about the content, and Detecting hallucinations. The QA pairs are constructed using a combination of human annotation and LLM-assisted generation, where we iteratively refine seed examples and prompts until a predefined pass rate achieved for the generated pairs. To ensure accurate and reliable evaluation, we employ GPT-4o as judge to determine the correctness of predictions and further validate the results by computing Cohen's Kappa coefficient between LLM and human evaluations, confirming a high level of consistency.
Desipte a lot of benchmarks has been used to compare RAG with feeding lanuage model with long context, there lacks a clear guidelines and conclusion on when and where the RAG will be a better choice than long context. Xu et al. and Bai et al. draw opposing conclusions on whether RAG or LC performs better on traditional QA datasets. Recently, Li et al. argued that L.C consistently outperforms RAG in almost all settings, and Yu et al. subsequently claim RAG can defeat LC on the same benchmark. In this section, we conduct detailed analysis on the existing benchmark and analysis, and find the key issues are stem from some significant flaws with their evaluation pipeline. For simplicity, our analysis is mainly limited to question answering tasks based on long contexts.
Our extensive experiments on LaRA demonstrate that
2.1. Issues with External Contexts
the choice between RAG and LC is not trivial, as it varies significantly depending on factors such as model size, query type, type of tasks, context length, context type, and number of retrieved chunks. If we can pinpoint the scenarios in which RAG outperforms LC, we can better design workflows to route each query through RAG or LC, thereby optimizing for both cost and performance, leading to more efficient and effective LLM applications. Our key findings are as follows:
Insufficent context lengths. As LLM base models continue to evolve, the definition of "long context" has also shifted, expanding from the early limit of 4k tokens to the now commonly supported 128k context length. Some early work utilize datasets such as Qasper (QASP) (Dasigi et al., 2021), NarrativeQA (NQA) (Kociský et al., 2018), and QuALITY (QLTY) (Pang et al., 2022) to compare RAG and L.C. For instance, Xu et al. conduct experiments on these datasets and find that RAG can strengthen large models, such Llama2-70B and GPT-43B. Similarly, Lee et al. combine these datasets to create new benchmark for further evaluations. However, such datasets no longer align with the current definition of long context. For example, QASP and QLTY have average context lengths of only 4912 and 6592 tokens, respectively, which are far below the context length capabilities of modern LLMs. Moreover, RAG typically uses chunk sizes of 300-600 tokens, and with 5-20 retrieved chunks, the total context length in RAG becomes comparable to that of full-context input, reducing the distinction between the two approaches.
Model Strength: RAG provides more significant improvements for weaker models. Our analysis indicates correlation between model strength and RAG's effectiveness: the weaker the model, the greater the improvement from RAG. For instance, with 128k context length, RAG outperformed LCC by 6.48% and and
38. 12% in accuracy on Llama-3.2-3B-Insttee Mistral-Nemo- 12B, respectively. For models with strong long-text capabilities, such GPT-4o and Claude-3.5-sonnet, LC generally outperforms RAG, demonstrating the effectiveness of these models in directly processing extensive contexts.
Context Length: RAG's advantages become more pronounced as context length increases. With 32k context length, LCC achieved an average accuracy 2.4% higher than RAG across all models. However, with 128k context length, this trend reversed, with RAG outperforming LCC by 3.68%.
Data Leakage. Since LLMs use more and more datasets in the training procedure, the problem of data leakage becomes more serious. At the same time, it is challenging to verify whether these early datasets were part of the training data for LLMs, potentially causing the models to memorize the answers. For example, although NarrativeQA has an average context length of 84,770 tokens, Gemini 1.5pro achieves 100% accuracy on this dataset (Lee al., 2024), indicating that either the dataset itself or its contexts were likely included in the model's 's training process.
Task Performance: RAG demonstrates similar performance to LC in single-location tasks and offers a significant advantage in identifying hallucinations. In contrast, LC excels in reasoning tasks and comparison tasks.
Inappropriate Contexts Handling. co-bench (Zhang
2

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmented Generation
datasets are not as difficult as they appear.
Table 1. Results on En.MC, EN.QA, and Zh.QA in oc-hench.
Task
GPT-40
Qwen 2.5-7B
Qwen 2.5-7B (vote)
Mn
76.0
67.7
87.9
No dedicated benchmarks. Over the past year, several long-context benchmarks have been introduced, e.g.. ZeroSCROLLS (Shaham et al., 2023), LongBench (Bai et al., 2024a); BAMBOO (Dong et al., 2024), LooGLE (Li et al., 2024a), Ruler (Hsieh et al., 2024), co-bench (Zhang et al., 2024a), and LongBench-V? (Bai et al., 2024b), with text lengths progressively increasing from 20k to 200k tokens. However, these benchmarks focus primarily on testing the ability of models to handle long texts. Although some of these benchmarks include experiments related to RAG, they lack a more systematic comparison between RAG and LC. Furthermore, existing RAG benchmarks typically utilize context lengths under 10k tokens (Chen et al., 2024a; Zhang et al., 2024c; Stolfo, 2024; Lyu et al., 2024; Saad-Falcon et al., 2024), failing to adequately address the challenges of long contexts. Although concurrent work like LONG"RAG(Q et al., 2024) explores long-context RAG, its average text length remains smaller than LaRA, and its focus lies on evaluating long- formformmue responses. Similarly, Loong (Wang al., 2024a) introduces tasks for comparing RAG and LC in multi- document QA but suffers from homogeneity in queries. For example, the clustering task only involves determining citation relationships between papers, and reasoning questions merely require providing citation chains. These queries, disconnected from specific content and applicable to any similar text, fail to capture the generalization capabilities of LLMs and RAG across diverse scenarios and context distributions.
En.QA
31.5
20.3
31.2
En.QA (L.LM)
82,9
78.1
85.5
Zh.QA
26.3
18.4
25.6
Zh.QA (LLLM)
79.9
72.5
85.2
et al., 2024a), recent benchmark widely used for comparing RAG and LC (Li et al., 2024b; Yu et al., 2024), includes two tasks, En.QA and En.M. with average context lengths of 192.6k and 184.4k tokens, respectively. However, their method of handling the excessive context length is problematic. When the context exceeds the LLM's maximum context length, the middle portion of the context truncated. Given that most models have limit of 128k tokens or even less, is highly likely that the answer to question is removed during truncation. In such cases, failure to answer reflects the truncation issue rather than the model's ' capabilities. To verify this, we split the excessively long cases in En. Gn.QA and Mn.MC into several parts, each within the context length limit. These parts are then paired with the query and fed into the LLM separately. The LLM is instructed to first determine whether the question is answerable; if not, it would decline to provide an answer. After obtaining multiple answers, a voting mechanism is used to get the final answer. The results are list in Table I. We observe that after adopting the segmented input and voting mechanism, Qwen-2.5-77 can even outperform GPT-4o.
Moreover, to prevent overlap with data seen during LLM training, key entity replacement is employed as a countermeasure in oc-bench. However, upon closer inspection, we find that some replacements are unsuccessful. For example, some entities mentioned in the questions do not exist in the provided context and vice versa!.
3. LaRA
In this section, we introduce the construction of LaRA and how it addresses the issues present in previous benchmarks, mentioned in Section 3. The statisties of LaRA provided in Appendix A.
2.2. Inaccurate Evaluation
3.1. Long Context Data Collection
Unreasonable metrics. Many previous evaluations use automated metrics such as Fl-score and exact match (EM) (Chen al., 2024a; Zhang et al., 2024c), which not reliable for NLG (Novikova al., 2017). For example, if the ground truth answer is "Allyson Kalia" and the model's response is "Allyson Kalia is convicted of the murder of Kiran's younger brother, "osetta." the prediction is clearly correct. However, it would only achieve an Fl-score of 0.29. This is also why the scores on the En. QA task in oo-bench (Zhang et al., 2024a) tend to be very low. We use L.M to re-evaluate, and the results are shown in Table 1. The accuracy becomes much higher, indicating that these
In context selection process, adhere the following principles: (1) Timeliness: We select recent high-qualic long contexts to prevent data leakage issues, ensuring that they are less likely to have been included in the LLM's s training data. (2) Appropriate Length: Considering that mainstream commercial and open-weight models typically support context length of 32k and 128k, we choose contexts that are as close these window sizes as possible without exceeding them. (3) Naturalness: The chosen contexts naturally occurring long documents, rather than artificially constructed or assembled from unrelated short texts, to ensure the benchmark reflects the complexity and diversity of real-world use. (4) Authoritativenes:: All contexts are considered reliable and credible sources of information due
httpps:/ http://gtttuuuuu. www.tttbb...oeeeee com//penMM/ InfiniteBench/ /isues////ssss
3

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmented Generation
expertise, reputation, and qualifications of the authors or institutions behind them.
basic comprehension and information retrieval capabilities within long context.
To ensure a diverse range of contexts, we select novels", financial statements', and academic papers* as the context. For novels, we choose the txt format of novelettes and novels to serve as the 32k and 128k contexts, respectively. Financial statements include the latest quarterly reports (32k) and annual reports (128k) from publicly listed companies in the United States for the year 2024. To create contexts of appropriate length for academic papers, we combine several papers published on arXiv in 2024 that are related through citations.
Reasoning. The reasoning task in LaRA involve questions that require logical deduction, inference, calculation based the information provided in the long context. Instead of directly extracting answers from the text, these tasks demand a deeper understanding and processing of the information to derive the correct answer, such as inferring the relationship between two characters or calculating relevant data in financial statements. These tasks evaluate the ability of LC and RAG to handle complex questions, particularly in scenarios where the long context contains a significant amount of noise irrelevant to the question. The specific questions vary significantly depending on the type of context involved. Instead of explicitly defining sub-task types, we adopt different seed questions tailored to specific text types. These seed questions are used to generate similar QA pairs through in-context learning. For example, in financial statements, which contain a significant amount of statistical data, we focus on computational questions, and for novels, the questions involve reasoning about the plot or character traits.
Entity Replacement. To mitigate the risk of data leakage from novels, which are likely present in LLMs' training data, we perform entity replacement. Previous work has employed similar strategies (Zhang et al., 2024a; Li et al., 2022), but we find that many entity replacements were incorrect or inconsistent, leading to inaccurate evaluations. To address this, use GPT-4o to accurately identify and replace character entities as well as formulating questions targeting the replaced entities, ensuring consistency between the novel text and the questions. Details are provided in Appendix B.
The comparison task in LaRA evaluates the ability of RAG and LC to synthesize information from multiple parts of a long context, comparing their content or numerical values to arrive at the final answer. Crucially, the comparison task also involves manually designing different seed questions tailored to various text types. This approach ensures that the generated questions are not only relevant but also reflect the nuances and complexities of the specific context. For instance, in academic papers, the questions may focus on comparing different explanations of the same phenomenon, while in novels, they may compare changes in a character's traits or appearance over time. This task is essential for assessing an LLM's ability extract information from different parts.
Comparison.
3.2. Tasks in LaRA
To comprehensively evaluate the capabilities of LC LLMs and RAG, LaRA includes four major task categories: location, reasoning, comparison, and hallucination detection, which are designed assess distinct aspects of LLM performance, motivated by the need to assess both the strengths and weaknesses of RAG and LC in handling complex, real-world information needs. Below, we will introduce each task in detail and further elaborate on the motivation behind them. Examples of each task are provided in Appendix F.
Location. The location task, the most fundamental task in LaRA, evaluates an LLM's ability to locate specific information within a long context. In this task, the answer resides in single sentence or paragraph within a long context, and no additional reasoning computation is required to formulate correct response, such as identifying character's name specific value mentioned in the text. It is worth noting that the location task differs from the "Needle in a Haystack" problem (Kamradt., 2023), which focuses on verbatim matching. In contrast, the location task allows for paraphrasing, as long as the underlying meaning preserved. This task is crucial for assessing an LLM's
Hallucination detection. Hallucination, a common issue in LLMs,, occurs when the model generates inaccurate irrelevant information (Huang et al., 2023). The hallucination detection task aims to test the model's ability to decline answering questions that are not mentioned in the given context. Although the questions appear to be answerable using the context, the required information is not actually mentioned in the text. Consequently, such questions have a uniform answer: "XXX is not mentioned in the provided context." " The ability to refuse to answer is crucial in practical applications of RAG and LC,, particularly in domains where accuracy and reliability are paramount, as users cannot always guarantee that their questions have answers within the provided context. For example, a user might pose a seemingly relevant question about a paper,
"https://www.r annua wwwp:aa reports. "https://arxiv.rr
4

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmentee Generation
and if the model hallucinates and generates an incorrect response, it could be highly detrimental.
the ground-truth answer, and the prediction, enabling it to assess the correctness of the response (details and prompt are provided in Appendix C). Since LaRA consists solely of QA pairs with clearly defined answers and no open- ended questions, using L.M as a judge is highly effective in ensuring accuracy and consistency in the evaluation process.
3.3. Data Annotation
The annotation process for different tasks follows similar framework, starting with the manual creation of seed questions and answers. We then utilize GPT-4o to generate new QA pairs through in-context learning. A subset of newly generated QAs is sampled for manual validation to ensure correctness and practicality. If the pass rate does not meet a predefined threshold (we sample 40 generated QAs for each context type and task, and the threshold is set to 90%), the seed QAs and prompts are refined, followed by re-generation and re- velviidation.. We provide the annotation prompt in Appendix E.
Manual Verification. To ensure the quality and reliability of LaRA, we incorporate manual verification at two stages of the construction process. First, during the generation process, we employ a pipeline involving sampling, prompt refinement, and seed QA selection as manual adjustments. We find that the choice of seed questions has the most significant impact, possibly because LLMs perform much better in in-context learning compared to zero-shot question generation, demonstrating the importance of providing relevant examples for guiding the generation process (For details, see Appendix E). Second, we calculate the Cohen's Kappa coefficient between the evaluation from LLM and human to quantitatively assess the agreement between the LLM and human evaluations, ensuring consistency and reliability in the judgment process. We provide the details in Appendix C.
Annotating long texts presents a unique challenge due to the inherent difficulty of long context processing. One effective approach to improve generation quality is to convert annotations for long texts into annotations for shorter texts. To achieve this, we employ various strategies tailored to different context types and tasks. Specifically, for location and reasoning tasks, we split the long context into multiple segments, each approximately 10k tokens in length, and input them individually into GPT-4o to generate QAs. This approach serves multiple purposes: First, it reduces the cognitive load on the annotator (GPT-40 here) and improves the focus and accuracy of the generated QA pairs. Second, it ensures that the answers are evenly distributed across the entire context, as we observe that providing the full context to the LLM often results in answers being concentrated at the beginming and end of the context. Third, it allows us to examine the relationship between answer accuracy and answer location, enabling us investigate whether the LLM suffers from the "lost in the middle" issue, where performance declines for information located in the middle sections of long documents (Liu et al., 2024a). For the comparison task, we split the context into smaller segments and then sample two segments to generate comparison questions similar to the seed questions. Meanwhile, our segmentation strategies are tailored to the specific context type to preserve the inherent structure and coherence of the documents. For research papers, we separate concatenated papers to maintain the integrity of each individual paper. For novels and financial statements, we directly split the text into multiple segments based on token count.
4. Experiments
4.1. Experimental Settings
Baselines. To investigate the impact of various factors RAG and LC performance, we evaluate a diverse set of 11 LLMs, encompassing both open-source and proprictary models. This includes seven open-source LLMs: Llama-3.2- 3B-Instruct (Meta, 2024a), 10ama--.1-8B-Insttete 2024b), 1.3ma-3.3-700 70B-Instruct (Meta, 2024c), Llama-3.3- 70B- Instruct-Q8 (Meta, 2024c) (utilizing FP8 quantization), 2ren-2.557B-Imm
(Meta,
(Yang et al., 2024), Qwen-2.5-77 Instruct (Yang et al., 2024), and Mistral-Nemo- 12B (AI, 2024). We also evaluate four advanced proprictary LLMs: GPT-4o (OpenAI, 2024), GPT-40-mini (OpenAI, 2024), Claude-3.5-Sonme (Anthropic, 2024), and Gemini-1.5. Pro (Reid et al., 2024).
Implementation of RAG. We employ a common configuration with a chunk size of 600 tokens, 5 chunks per document, and an overlap of 100 tokens between chunks. We utilize GTE-large-en-vl.. (Zhang et al., 2024b; Li et al., 2023) for embedding extraction and reranking and employ hybrid search strategy (Wang et al., 2024b) combining embedding similarity and BM25 (Robertson al., 2009). Before conducting system experiments, we attempted to replace the embedding model and rerank model, such as with bge-m3 (Chen et al., 2024b), and also tried adding a summarization module (Xu et al., 2023). However, we found that these changes had minimal impact on the
3.4. Evaluation
Metrics. Automated evaluation metrics, such as F1-score and ROUGE, can often produce lower scores, while L.LM evaluations for QA tasks with definitive answers have demonstrated high precision (Chiang & Lee, 2023; Liu al., 2023). Therefore, we provide GPT-4o with the query,
5

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmennte Generation
Table The accuracy of haselines evaluated by GPT-4o LaRA (%). We evaluate performance for context lengths of 32k and 128k tokens separately, with the gray background representing RAG and the white background representing L.C. The highest-performing open-source and proprietary LLMs for each task are highlighted in bold. "Avg GAP" refers to the difference between the average accuracy of LC and RAG across models for specific task (calculated LC minus RAG). Blue text indicates that LC performs better, while red text indicates that RAG is better.
Model
Location
Reasoning
Comparison
Hallucination
Overall
Open-source LLMs
(32k)
Bamma-3.1-8BInssee
77.60
73.85
51.65
46.59
58.29
38.06
78.80
83.73
66.58
60.56
69.11
71.08
36.60
34.88
31.10
25.02
65.13
79.71
50.48
52.67
77.50
79.83
66.43
57.23
65.33
53.13
73.59
86.62
70.71
69.20
78.24
77.86
61.12
58.55
64.83
47.90
74.09
83.95
69.57
67.06
Owen-2.5-275 II--nrttttt
78.10
73.70
51.30
48.63
62.35
43.79
76.42
84.36
67.04
62.62
Bwen-2.5-72BB-Instte
83.29
81.63
74.06
65.24
68.89
49.24
85.08
83.76
77.83
69.97
Distral-Nemo-I
54.45
72.38
29.81
45.74
29.02
43.63
35.27
71.70
37.14
58.36
Proprietary
LLMs
(32k)
GPT-4o
86.33
82.55
75.48
68.51
79.66
46.56
72.30
78.39
78.44
69.00
GPT-4o-mini
79.92
77.69
77.65
62.23
73.91
54.48
52.82
67.17
71.08
65.39
Claude-3.5-some
85.82
83.10
70.42
66.81
66.28
51.38
86.12
90.99
77.16
73.07
Gemini-1.5-pro
78.94
74.71
64.68
53.50
77.34
56.43
84.58
88.03
76.39
68.17
Avg GAP
0.08
4.66
15.22
-10.38
2.40
Open-source
I.LMs
(128k)
1aama-31-8B-Instteee
72.64
72.65
48.48
47.73
40.87
22.27
58.40
78.36
55.10
55.25
-lama-3.2-3B-Insttee
60.96
68.96
33.99
38.86
24.54
20.20
52.06
69.45
42.89
49.37
74.54
78.11
52.98
59.05
43.09
26.69
44.00
77.67
53.65
60.38
72.44
77.99
53.97
58.32
45.32
29.61
38.18
76.87
52.48
60.70
Owen-2.5-7B-Insite
68.94
74.08
44.69
50.93
39.51
29.17
42.52
71.02
48.91
56.30
Qwen-2.5572B-IInstre
76.10
78.92
65.25
64.37
54.62
36.10
64.45
71.32
65.11
62.68
Mistral-Nemo-122
23.44
72.29
14.87
50.53
7.77
32.59
14.77
57.92
15.21
53.33
Proprietary
LLMs
(128k)
GPT-4o
87.70
82.22
79.35
70.26
64.74
40.84
56.34
67.33
72.03
65.16
GPT-4o-mini
79.86
80.43
67.80
65.15
65.31
41.29
32.08
59.31
61.26
61.55
Claude-3.5-some
85.44
80.94
73.79
64.81
58.35
33.18
77.64
84.73
73.81
65.92
Gemini-1.55 1.5-pro
82.15
75.41
67.97
48.47
61.67
36.54
66.16
78.60
69.49
59.75
Avg GAP
-5.25
-1.39
14.30
-22.36
-3.68
performance and could potentially introduce additional computational overhead.
where RAG demonstrates superior performance across most models. In contrast, proprictary models consistently favor I.C both context lengths, likely due to their larger parameter sizes and enhanced ability to process long- context inputs. The inherent self-attention mechanism in these models appears more effective at handling extended contexts compared to the sparse attention employed in RAG. Notably, at a 128k context length, the top three performing models (GPT-4o, Gemini- 1.5-Pr,, and Claude- 3.5- 3.5-Sonnet) all utilize LC, while the bottom three (Llama- 12-3B-IIntren Qwen-2.5-1 2.5--B--nsBtttt and Mistral-Nema 12B) are also LC-based. This observation underscores the absence of a universal "winner" between RAG and LC, as performance is highly dependent on the specific LLM and context length.
4.2. Main Results and Analysis
Overall performance. Table presents the evaluation results, comparing the performance of LC and RAG across various LLMs. The table uses white background for LC results and gray background for RAG results, with the tighest-performing open-source and proprietary LLMs for each task highlighted in bold. Our analysis reveals complex relationship between model architecture, context length, and performance. For open-source models at a 32k context length, LC generally outperforms RAG, with the exception of Llama-3, 3.2-aBBBBBBBBBB ---trutntt and Mistral-Nemo- 12B. However, this trend reverses at 12 128k context length,
6

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmented Generation
Owend
calculate the average performance gap (avg gap) for each task, representing the mean difference in accuracy between LC and RAG across all evaluated models.
Liama82-4BB
Llarad 1:88
Liaradl 5-708
Book-LC
Book-LC
Paper LC
An LC
Paper LC
An LC
Location. The location task proves to be the easiest among the four, with both RAG and LCC achieving high accuracy. The average performance gap between RAG and LC is 0.08% at a 32k context length and -5.25% at 128k, indicating a slight advantage for LC with shorter contexts and a more pronounced advantage for RAG with longer contexts. For open-source models at 32k, the performance difference between RAG and LC is minimal, while at 128k, RAG demonstrates superior performance. This suggests that when models struggle to process long texts, retrieval acts as a valuable tool for location-based questions. Conversely, for proprictary models, LC consistently outperforms RAG, indicating that with sufficient model capacity, LLMs can outperform RAG in handling such simple tasks on their own.
PaperRAG
PaperRAG
Book RAG
Book RAG
(a) 32k
(b) 128k
Figure 1. The average accuracy across different context types. The left figure (a) represents context length of 32k, while the right figure (b) represents context length of 128k.
Scaling law holds in LC. Our experimental results confirm the established scaling law in LC (Kaplan et al., 2020): larger models consistently outperform smaller counterparts. For example, GPT-4o and Qwen-2.5-72BB Instruct show significant performance gains ranging from 7.35% to 16.2% over their smaller versions, GPT-4o-mini and B.e.a--.5----Inttteee
Reasoning. The performance trends for reasoning tasks basically mirror those observed in the location tasks, particularly for smaller models. With a 32k context, RAG exhibits slightly lower accuracy compared to I.C, while at 128k, this trend reverses. However, for larger models like GPT-4o and Claude-3.5-Sonnee the advantage of LC becomes more pronounced. At 128k context length, outperform RAG by 9.09%
respectively. This advantage is further amplified a 128k context length. While all models experience a performance decline with longer contexts, the drop is more pronounced for smaller models, highlighting their limitations in processing extensive textual input. This observation challenges the purported ability of smaller models to handle extremely long contexts effectively.
GPT-4o and and 8.98% in accuracy, respectively. We speculate that while reasoning tasks often rely on specific text segments for answers, other parts of the document may contain supplementary information that aids in inference. Models with stronger long-context capabilities are better equipped to leverage this global knowledge. leading to improved performance in reasoning tasks.
RAG empowers models to handle extremely long context. At a 128k context length, RAG consistently outperforms LC across nearly all open-source models. For example, Llama-3.1-8B-Instt Instruct demonstrate improvements of 0.15% and 7.39%, respectively, when using RAG instead of LC. All models exhibit performance decline at a 128k context length compared to 32k. However, LC experiences a more significant drop than RAG, indicating that as context length approaches its limit, RAG is less affected by the increase in context length. Furthermore, RAG enables models with weaker long-context capabilities, such as Mistral-Nemo- 13B and Llama-3.2-23 Innttuutt to achieve performance comparable to other models. These findings highlight that while larger models excel at long-context processing, RAG offers an effective alternative for smaller or weaker models, ensuring competitive performance even with extended context lengths.
and Qwen-2.5-7B-
Comparison. Comparative tasks pose the greatest challenge for RAG, showing the largest performance gap compared to LC. The average gap reaches 15.22% at 32k and 14.30% 128k. A deeper analysis of problematic cases reveals two primary reasons for the struggle of RAG. First, some comparative questions emphasize one aspect of the comparison while providing limited information about the other, making it difficult for RAG to retrieve both necessary chunks for a correct answer. Second, certain queries describe the comparison in abstract terms rather than concrete details, hindering the effectiveness of similarity- based retrieval. This abstraction makes it challenging for RAG to locate all the relevant chunks for comparison. In contrast to location tasks, which involve pinpointing single piece of information, comparative tasks require the accurate retrieval and comparison of multiple distinct chunks, significantly increasing the complexity for RAG.
4.3. Task Analysis
Table 2 presents a detailed breakdown of LC and RAG performance across four distinct tasks. To provide a clear overview of the relative strengths of each task, we also

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmentee Generation
80
75
Accuracy (%6)
25
Accuracy (%6)
Accuracy (%))
-3
70
05
Accurasy
70-
60
03
15
55
50
60
50
Devm 73
Owe Naraa
10
15
20
25
10
15
20
25
30
200
600
600
RD0
1000
2000
700
480
600
000
1000
2000
of Chunk
# of Chunk
Chunk Size
Chunk Size
(a) Chunk quantity (32k)
(b) Chunk quantity (128k)
(c) Chunk size (32k)
(d) Chunk size (128k)
Figure 2. The accuracy of 2.i---2.-5-----sss-- and Qwen-2.5-78 Instrucct with different chunk quantity and size on LaRA.
Hallucination detection. This is the only task where RAG demonstrates a clear advantage in both small and large models. LC tends to generate more hallucinated or incorrect answers, likely due to the increased noise introduced by feeding the entire text to the model. This makes the model more susceptible to errors and distractions, leading to fabricated responses. Interestingly, larger models do not exhibit evident advantage in this task. While GPT-4o achieves top performance on other tasks, it only attains accuracy of 56.34% on hallucination detection. This suggests that even models capable of handling long contexts can be overwhelmed by the sheer volume information and generate flawed conclusions. In contrast, RAG's selective retrieval of relevant information helps mitigate this issue, enabling both small and large models to better identify situations where they lack sufficient knowledge to provide an accurate answer.
number of chunks and the size of each chunk. We conduct experiments on Qwen-2.5-72B Insstrucct and Qwen-2.5-7B Instruct to observe the impact of chunk size and quantity on both large and small models. As shown in Figure 2, for the 72B model, performance improves consistently as the number of retrieved chunks increases, benefiting from its stronger long-context processing capability. In contrast, the 7B model exhibits a peak performance at an intermediate chunk quantity, after which excessive retrieval introduces noise that outweighs the information gain. Regarding chunk size, both excessively large and small chunks lead to performance degradation. Within a reasonable range, increasing chunk size provides some improvement, though its effect less significant than increasing the number of chunks.
4.6. Lost in The Middle
By controlling the location of the information required to answer the questions, we find that LC LLMs exhibit decrease in accuracy when the answer is closer to the center of the context, indicating susceptibility to the "lost in the middle" phenomenon. In contrast, we do not observe a clear correlation between RAG performance and the position of the answer, suggesting that RAG models are more robust to this issue. These findings highlight a potential advantage of RAG models in handling long contexts, particularly for tasks that require accessing information from various parts of the document. For detailed results, see Appendix D.
4.4. Context Type Analysis
We present the influence of different context types the performance of RAG and LC in Figure 1. Novel-related questions pose the greatest challenge. while paper-related questions are the easiest. This disparity likely stems from the repetitive sentence structures common in novels, which can hinder precise information localization. Conversely, academic papers typically exhibit a stronger logical flow and higher information density, facilitating easier distinction between questions and answers. At 32k context length, LC outperforms RAG for nearly all models. However, at 1 128k, weaker models demonstrate better performance with RAG. Interestingly, the performance gap between RAG and LC is smaller for novel-related tasks compared to paper-related or financial statements tasks, regardless of context length. This suggests that for less structured contexts, RAG presents viable alternative for reducing computational cost. On the other hand, for highly structured texts like academic papers and financial statements, LC demonstrates a clear advantage.
5. Conclusion
This study addressed the critical question of whether RAG or LC is more effective for incorporating external knowledge into LLMs.. Through the development and evaluation of LaRA, novel benchmark, demonstrated that the optimal choice depends complex interplay of factors, including model size, context length, and task type. Our findings challenge previous inconclusive comparisons and offer actionable guidelines for practitioners. LaRA serves as a valuable resource for evaluating and comparing RAG and LC models, facilitating further research in this rapidly evolving field.
4.5. Impact of Chunk Quantity and Size
We explore the impact of the length of retrieved information length on RAG performance through two aspects:
the
8

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmennted Generation
Impact Statement
Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 15607-15631. Association for Computational Linguisties, 2023. doi: 10. 10.1863//1/22233LLLLL0 870. URL /ttp://////o...oo//00000003///
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
2023 1023.acl-
- acl- waa.clloog..111 870.
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., and Gardner, M. A dataset of information-secking questions and answers anchored in research papers. In Toutanova, K., Rumshisky, A., Zettlemoyer, L.,, Hakkani-Tür, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-IILT 2021, Online, June 6-11, 2021, pp. 4599-4610. Association for Computational Linguistics, 2021. doi: 10.
References
AI, M. Mistral large, 2024. URL ai htt////wwwmmsssallleeeeeaa
Anthropic. Claude 3.5 sonnet, 2024.
URL
https: /:::://wwww wwwwwwwtwwwp.......woot..t. com/news/////eeoeeooomoooooomo/
claude - 3-5-sonnet/.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang. Z
Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang. J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. In Ku, I.., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok. Thailand, August 11 11-16, 2024, pp. 3119-3137. Association for Computational Linguistics, 2024a. doi: 10.18653/VI/ 2024.ACL-LONG.172 URL https:://di.. org/ 10.18653/v1/220 acl -ow.oo.....1111..11711..
2021.NAACL-MMINN.65 URL https: //do:. org// 10.18653/v1/2022
Dong, Z., Tang, T., Li, J., Zhao, W. X., and Wen, J. BAMBOO: A comprehensive benchmark for evaluating long text modeling capacities of large language models. In Calzolari, N., Kan, M.. Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 2086- 2099. ELRA and ICCL, 2024. URL https:/ // aclanthology org org.orgg///22 2024.224..00 wwwe-rrriinnnnnm 188.
Bai. Y.. Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao. S., Xu, J. Hou, L., Dong, Y., Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprini arXiv:2412. 2024b.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Guo, Q. Wang, M., and Wang, H. Retrieval- augmented generation for large language models: survey. CoRR, abs/2312.109997, 2023. doi: 10.48550/ ARXIV.2312.10999. URL https: //ts::///oo... org/10. 48550/arXis 2312 10997.
Chen, J., Lin, H., Han, X., and Sun, I.. Benchmarking large language models in retrieval-augmentdd generation. In Wooldridge, M. J., Dy. J. G., and Natarajan, S. (eds.). Thirty-Eighti AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 17754-17762. AAAI Press, 2024a. doi: (111609/AAAI.V331811.. //d....0//1...110//////// aaai. wttp:///....../...........1.....
Guu, K., Lee, K., Tung, Z., Pasupat, P... and Chang, M. Retrieval augmented language model pre-training. Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3929-3933 PMLR, 2020. URL http:://rtoceeeeee guu20a 20 html.
URL ht https:
mlr www..press//ee
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embedding: Multi-lingual, multi- nulti-fuuctionnaliy. multi-granulariyg text embeddings through self- eoflleowledge distillation. arXiv preprint arXiv:2402.032.6. 2024b.
Hsich, C., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: what's the real context size of your long-context language models? CoRR, abs/2404.06654, 2024. doi: 10.48550/ARXIV. 2404.06654. URL https: http://///o...o/..//ooo//// arXiv.2404. 2404. 06654.
Chiang, D. C. and Lee, H. Can large language models be an alternative to human evaluations? In L., and Okazaki, N.
Rogers, A., Boyd-Graber, (eds.). Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I:
Huang, L., Yu. W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q. Peng, W., Feng, X., Qin, B., and Liu,

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmennted Generation
T. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. CoRR, abs/231 2023. doi: 10.48550/ARXIV. org 10. 48550/
I: Long Papers). ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp- 16304-16333. Association for Computational Linguistics, 2024a. doi: 10. 18.58/1//V1/ 2024.ACL-LONG.89, URL https: /tpps://ddo //ooi.doi//// orq/ .all:244cclllll. 1occc-1-pp ong.. 859.
2311.05232. URL 2311.
Kamradt., G. Needle in haystack pressure testing llms., 2023. URL https://githal. com/ gkamradt/ JLMMestt_NeedleInAfiee
Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint S.Kt::2308.0328.. 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. CoRR, 0bss22011088360 2020. URL https::// htttps://ariv.. hrg/abs//an 20020001000000 08361.
Li, Z., Li, C., Zhang, M. Mei, Q. and Bendersky, M. Retrieval augmented generation or long-context Ilms? comprehensive study and hybrid approach. In Dernoncourt, F., Preotiuc-
D., and Shimorina, A. (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing: EMNLP 2024 Industry Track, Miami, Florida, USA, November 12- 16, 2024, pp. 881-893. Association for Computational Linguistics, 2024b. URL http://///cnaahhoooyy... /tt.o//tttt////caaaaaaloooooo org hrgg2024.eo 2ww.2244emmmmmmm emn emnle-mnnm--mmm industry........ 66
Kocisky, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. The narrativeqa reading comprehension challenge. Trans. Assoc. Compul. Linguistiess, 6:317-328, 2018. doi: 10.1162/TACL 00023. URL http:: /ttp///oiiioaa http://t..........///////// a_00023.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12:157-173, 2024a. doi: 10.1162/TACL) _A,006638 URL https: //t::///ooiii/ http://oo..oogg//////// /1662/11/1/ tacl_a_00638.
Lee, J., Chen, A., Dai, Z., Dua, D., Sachan, D. S., Boratko, M., Luan, Y., Arnold, S. M. R., Perot, V., Dalmia, S., Hu, H., Lin, X., Pasupat, P., Amini, A., Cole. J. R., Riedel, S., Naim, I., Chang, M., and Guu, K. Can long-contens language models subsume retrieval, rag, sql, and more? CoRR, abs/2406. 13121, 2024. doi: 10.48550/ARXIV. 1102. URL hwtph:ttp///// http::///ooooooo.o.oooooooooo 0r5510.48550/ 48550/ arXiv. 2406 13121.
Liu, Y., Iter, D., Xu, Y.. Wang, S., Xu, R., and Zhu, C. G-eval: NLG evaluation using gpt-4 with better human aligmment. In Bouamor, H., Pino, J. and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 2511-2522. Association for Computational Linguistics, 2023. doi: 10. 0018653/1120033.MM.LPPMMM
Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmeeneed generation for knowledge- intensive NLP tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: /ttp:///eepp/.....//////
153. URL http:////db..oor/1.1116////1// 2023. emnlp- main.
Liu, Y., Yang, T., Huang, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., and Zhang, Q. Calibrating llm-based evaluator. In Calzolari, N., Kan, M., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguisties, Language Resources and Evaluation, LREC/COLING 2024, 20- 25 May, 2024, Torino, Italy, pp. 2638-2656. ELRA and ICCL, 2024b. URL https: aclanthooott.y. org/ htt./22a..rrccmmiii..
neurips heurips.cc/pappeeeppppp hctp://p.ccpppppe/////aaaaattt 6b4932 html.
Li, J., Sun, A., Han, J., and Li, C. A survey on deep learning for named entity recognition. IEEE Trans. Knowl. Data Eng., 34(1):50-70, 2022. doi: 109/ PKDE.2020.2981314 URL 1109/TKDE. 2020 981314.
Li, J., Wang, M., Zheng, Z., and Zhang, M. Loogle: Can long-context language models understand long contexts? In Ku, L.,, Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume
Long, L., Wang, R., Xiao, R., Zhao, J., Ding, X., Chen, G., and Wang, H. On lims-driven synthetic data generation, curation, and evaluation: A survey. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok,
10

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmenntel Generation
Thailand and virtual meeting, August 11-16, 2024, pp. 11065-11082. Association for Computational Linguistics,
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillierap, T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J.. Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sottiaux, T., Lee, B., Viola, F,, Reynolds, M., Xu, Y., Molloy, J., Chen, J., Isard, M., Barham, P., Hennigan, T., Mcllroy, R., Johnson, M., Schalkwyk, J., Collins, E., Rutherford, E., Moreira, E., Ayoub, K., Goel, M., Meyer, C., Thornton, G., Yang, Z., Michalewski, H., Abbas, Z.., Schucher, N., Anand, A., Ives, R., Keeling, J., Lene, K., Haykal, S., Shakeri, S., Shyam, P., Chowdhery, A., Ring, R., Spencer, S., Sezener, E., and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403 05530, 2024. doi: 10445550ARRXIV24400.05555 URL https: //doi. org/10 www.4/4800....222. 05530.
2024. doi: 10. (0.5665/W1/222FFFNDNNSSAGCLL
658. URL https:/ ott.o/..........//.1.
2024. findingsnnainnnneeee acl. .658.
Lyu, Y., Li, Z,, Niu, S., Xiong, F., Tang, B., Wang. W., Wu, H., Liu, H., Xu, T., and Chen, E. CRUD- RAG: A comprehensive chinese benchmark for retrieval- augmented generation of large language models. CoRR, abs/2401.17043, 2024. doi: 10.48550ARRIV:2411 17043. URL
arXiv. 2401.17043.
Meta. Llama 3.2, 2024a. https: httta://ai.meta.
com.
Meta.
Llama 3.1, 2024b. URL
htmm/blog/metee aalamma-l11 3-1/.
Robertson, S., Zaragoza, H., et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389 2009.
Meta.
Llama 3.3, 2024c.
URL
ht https: a1ama..a...mmmmmm com/ docs/ model-caardd-and-prrm-------- 1lama3_3/.
Saad-Falcon, J., Khattab, O., Potts, C., and Zaharia, M. ARES: an automated evaluation framework for retrieval-augmented generation systems. In Duh, K., Gómez-Adorno, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 338-354. Association for Computational Linguistics, 2024. doi: 10. 18.18653/V1/ 2024NAACL-LLONG..2 URL. htw.t: //attpa://oo/ //doi /tt.torg////oooooooooooooo 10.18653/v1/200 nww.l-accllaaaaa long. 20.
Novikova, J., Dusek, O., Curry, A. C., and Rieser, Why we need new evaluation metrics for NLG. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 2241-2252, Association for Computational Linguistics, 2017. doi: 10.18653/V1/ D17-1238. URL https: v1/d17-1238.
OpenAI. Gpt-4o, 2024. URL Lndex/hello-gpt--e
Shaham, U., Ivgi, M., Efrat, A., Berant, J., and Levy, O. Zeroscrolls: A zero-shot benchmark for long text understanding.
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang.
J. Chen, A., Padmakumar, V., Ma, J., Thompson, J. He. H., and Bowman, S. R. Quality: Question answering with long input texts, yes! In Carpuat, M., de Marneffe, M., and Ruíz, I. V. M. (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp- 5336-5358. Association for Computational Linguistics, 2022. doi: 10.
In Bouamor, H.. Pino, J., and Bali, K. (eds.), Findings of the Association for Computattional Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 7977-
7989. Association for Computational Linguistics,
2023. doi: 10.1865/1/20233FFDDDSSSSS-
536. URL
2023. findings-emnllp........... 536.
391. URL 10.1663/W1//0222AAACLLMMM
Stolfo, A. Groundedness retrievallaugmented long- form generation: An empirical study. In Duh, K., Gómez-Adorno, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp.
http:////oo..0.1..11811585 2022. naaclnaacl-man 391.
Qi, 7., Xu, R., Guo, Z., Wang, C., Zhang, H., and Xu, W. Long2rag: Evaluating long-context & long- form augmentl generation with key point recall. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4872, 2024.
1537-1552, Association for Computational Linguistics,
2024. doi: 10.1855/N1220AAFFNDDINGSS NAACL.
100. URL
2024. f fi iid-

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmentedd Generation
Wang, M., Chen, L.., Cheng, F., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L.,, Luo, R., Li, Y. Yang. M., Huang, F,, and Li, Y. Leave no document behind: Benchmarking long-contens lims with extended multi-doc QA. In Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 5627-5646. Association for Computational Linguistics, 2024a. URL org/ 2r2/ emn ammnp-maa mamin.3.. 3222.
2024. doi: 1044550AAR2XX..2400006666 URL https: 6666.
/
Zhang. X., Chen, Y., Hu, S., Xu, Z., Chen, J.. Hao, M. K., Han, X., Thai, Z. L.., Wang. S., Liu, Z., and Sun, M. inftybench: Extending long context evaluation beyond 100k tokens. In Ku, l.., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 15262-15277. Association for Computational Linguistics, 2024a. doi: 10. 18.13////V1/ 202A.ACL-LONG.84, URL https: /tt///di..// org/ 10. 10.18653/v1/2022 accl llo.lnnnn........ 8114
Wang. X., Jiingg Y., Bach, N., Wang, T., Huang, Z., Huang, F, and Tu, K. Improving named entity recognition by external context retrieving and cooperative learning. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the Itth International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1800-1812, Online, August 2021. Association for Computational Linguistics. URL https: http//aaaannntoyy...oooo org/ 2021 -olellllogg 142.
Zhang, X., Zhang. Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F, et al. mgte: Generalized long- con-ccontext text representation and reranking models for multilingual text retrieval. arXiv 2024b.
preprint
Zhang, Z., Fang. M., and Chen, L. Retrievalqa: Assessing adaptive retrieval- -uggmett generation for short-form open-domain question answering. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 1 11-16, 2024, pp. 6963-6975. Association for Computational Linguistics, 2024c. doi: 10.1665//122244FEDDIISSS..
Wang, X., Shen, Y., Cai, J., Wang. T., Wang, X., Xie, P., Huang, F., Lu, W., Zhuang, Y., Tu, K., Lu, W., and Jiang. Y. DAMO-NLP at SemEval-2022 task 11: A knowledge- system for multilingual named entity recognition. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp- 1457-1468, Seattle, United States, July 2022. Association for Computational Linguistics. URL https: aclanthology. org/2022. www.wal-1.2000
415. URL
2024. findings-acl. 415.
Wang, X., Wang. Z., Gao, X., Zhang, F., Wu, Y., Xu, Z., Shi, T., Wang, Z., Li, S., Qian, Q., et al. Searching for best practices in retrieval-au -aggeettd- generation. arXiv preprint ar Xiv:24407.00/219 2024b.
Xu, F. Shi, W., and Choi, E. Recomp: Improving attieval-augmeen lms with compression and selective augmentation. arXiv preprint arXiv:2310.044000 2023.
Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro, B. Retrieval meets long context large language models. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenReview.net, 2024. URL https: //openreviiieeeeeeee ew net forum? d=xw55xFFM11.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 241 2024.
Yu. T., Xu, A., and Akkiraju, R. In defense of RAG in the era of long-context language models. CoRR, abx/2409.0166
12

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmenteed Generation
A. Statistics of LaRA
LaRA consists of approximately 2300 test cases, encompassing three context types and four task categories. To ensure that the token count of the context as close as possible 32k and 128k without exceeding these limits, the primary token ranges for these two lengths are 20-30k and 80-120K, respectively. The average token counts and numbers of tasks provided in Table 3.
Table 3. Statistics of LaRA.
Context
Location
Reasoning
Comparison
Hallucination
32k length
Novel
25673
25908
25681
25433
Financial
27548
27531
27546
27527
Paper
28078
28088
27708
28081
# of Cases
276
230
151
230
128k length
Novel
96452
96226
95903
96182
Financial
92684
92831
92830
92812
Paper 93911
93818
94731
93890
# of Cases
489
374
198
378
B. Named Entity Recognition and Replacement for Novels
We initially tried using some traditional Named Entity Recognition (NER) methods (Wang al., 2021; 2022), but found that their performance was poor. First, traditional sequence labeling models struggle with long-text processing due to fixed-length context windows, failing to capture cross cross- passarapa entity associations. Second, performance degrades significantly on out-of-distributtio data, particularly when handling domain-specific or stylistically unique texts (Li et al., 2022). These constraints prove especially problematic for literary analysis, where novels exhibit both long-range narrative dependencies and rich variations entity references (e.g., honorifies, epithets, and contextual substitutions).
The emergence of LLM presents opportunities to overcome these limitations through their superior contextual understanding and robust generalization capabilities. Hence, we leverage GPT-4o to perform entity extraction and replacement in full-length novels through three-stage pipeline:
1. We partition input texts into coherent segments averaging 500 tokens, preserving complete sentence/paragraahh boundaries. This chunk size optimizes the balance between L.LMs' context window constraints and narrative continuity requirements.
2. Each text chunk undergoes parallel entity recognition through GPT-4o processing. Our extraction protocol captures both original names and contextual variants (e.g.. "The Dark Lord" "Voldemort" in Harry Potter). After feeding each chunk for extraction, merge all the entities and remove duplicates to obtain final list of entities while preserving legitimate aliases.
3. For alias disambiguation, we prompt GPT-4o with the entity list and the novel's 's title to determine if multiple names indicate the same character, and if they do, we replace them with the same fake name.
Figure and Figure detail our carefully engineered prompts for entity extraction and substitution respectively. Note that the novels we select are classical literary works, which, along with extensive related discussions, are included in the LLMs' training data. As a result, we find that simply providing the name of the novel is sufficient to accurately map multiple aliases the same character.
13

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmennte Generation
Example prompt used for Named Entity Extraction
Your task is to extract named entities(only person) from the given paragraph. Response with a list of entities, and strings the list should be enclosed in double quote. Below is an example, and you need to use style consistent with the example.
Task:
Example:
<puragraph>>Fily became pensive. "Clearly," the Time Traveller proceeded, "any real body must have extension in four directions: it must have Length, Breadth, Thickness, and und-Duration. But through natural infirmity of the flesh, which will explain to you in a moment, we incline to overlook this fact. There are really four dimensions, three which we call the three planes of Space, and fourth, Time. There is, however, a tendency to draw an unreal distinction between the former three dimensions and the latter, because happens that our consciousness moves intermittently in one direction along the latter from the beginning the end of lives." "That," said very young man, making spasmodic efforts to relight his cigar over the lamp; "that... very clear indeed."
"Now, it is very remarkable that this is so extensively overlooked," continued the Time Traveller, with slight accession of cheerfulness. "Really this is what meant by the Fourth Dimension, though some people who talk about the Fourth Dimension do not know they mean it. It is only another way of looking at Time. There is no difference between Time and any of the three dimensions of Space except that our consciousness moves along But some foolish people have got hold of the wrong side of that idea. You have all heard what they have say about this Fourth Dimension?"
"I have not," said the Provincial Mayor. Mayar_//paragraph
Entities: ["Filby", "Time Traveller", "Provincial Mayor"]
Entities:
Figure 3. The prompt for extracting named entities from novel chunks. The chunk that need to be extracted is highlighted in red text.
C. LLM as A Judge
Given the unreliability of rule-based evaluations and the high costs associated with human evaluation, the use of LLM for assessment has gained increasing popularity (Liu et al., 2024b; Wang et al., 2024a). In LaRA, we prompt GPT-4o to determine whether model correctly answer a question, using the query, the ground-tron truth answer, and the model's prediction as inputs. The specific prompt is shown Figure 5.
To verify the consistency between GPT-4o evaluation and human evaluation, we compute the Cohen's Kappa coefficient, which is defined as:
P. Pa K 1 - Pe
where Po represents the proportion of agreement between the two evaluators on the positive and negative classes, and Pa denotes the probability of agreement between evaluators under the assumption of independent and random classification.:
TP+TN Pa TP + FP + TN + FN
(2)
P. CPP+FNIIPP + FP) (TN FN)(TN FP)
(3)
TPP + FP - TN + FN)2
where TP and TN refer to cases where both the human evaluator and the model agree that the answer is correct, while FP and FN refer to cases where only one of them judges the answer as correct.
14

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmentted Generation
Example prompt used for Name Replacement
Task: have extracted the names of all the characters in the novel (novel). as shown in the list below. Your task is to replace the names in novel with fictitious ones.
Firsdl, You need to determine based on the novel's content whether any these names refer to the same character. If they refer the same person, they should be assigned the same fictitious name. You should respond with Python dict only, with the keys being the names from the novel and the values being the new fictitious names. Strings in the dict should be enclosed in double quote
Name list: [name_list
Figure 4. The prompt for replacing the names with fictitious ones.
Example prompt used for Evaluation
Task: You are discriminator that judges whether the predictions to questions are correct.
I will provide you with a question and its Ground-trud answer, as well as an answer from an Al assistant. You need to judge whether the assistant's answer is correct based on the Ground-truth answer. it correct, you should only output True; if is incorrect, only output False.
[Query] {query}
[Ground-truth Answer] [label}
[AI Assistant's Answer] {pred}
Your judgment:
Figure 5. The prompt for evaluation.
Cohen's Kappa coefficient greater than zero indicates consistency in the evaluation, with values approaching indicating stronger agreement. We sample 100 predictions from each task type 128k context length from GPT-4o (L.C) and Qwen-2.5-7B (LC) for manual evaluation and then compute the Cohen's Kappa coefficient. The results provided in Table 4. The values for both models are close to 1, demonstrating that evaluations from GPT-4o are highly consistent with human evaluation, whether applied to large or small models.
Table Cohen's Kappa coefficient for GPT-4o (I.C) and Qwen-2.5-7B (L.C).
Model
Location
Reasoning
Comparison
Hallucination
Qwen-2.5-7B GPT-4o
0.9737
0.9184
0.8955
0.9800
0.9509
0.9254
0.9776
0.9604
D. Lost in The Middle
To assess the presence of "lost in the middle" our LaRA benchmark, we ensure a uniform distribution of answers across different positions within the context during the annotation location and reasoning tasks. Specifically, for location tasks
15

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Auggennted Generation
Qwen 7u-
GPI-4o-0 GPT-io fill
80
90
90
Accuracy 1%6)
80
Accuracy (%6)
70
Accuracy (%6)
70
Accuracy
80
80
60
70
50
60-
Answer Position
Answer Position (b) 128k location
Answer Position
Answer Position
32k location
32k reasoning
(d) 128k reasoning
Figure 6. The accuracy of the location and reasoning tasks when the answer appears at different positions within the context. 32k-length and 128k-length contexts split into 5 and 10 segments, respectively. The contexts here novels and financial statements, split papers in another method metioned in Section 3.3.
with 32k and 128k context lengths, we divide the context into 5 and 10 segments, respectively, and for reasoning tasks, we use and segments for 32k and 128k contexts, respectively.
As illustrated in Figure 6, experiments reveal that LC models indeed suffer from the "lost in the middle" phenomenon. This issue is particularly pronounced in weaker models like Qwen-2.5-7B. In contrast, RAG demonstrates consistent accuracy regardless of answer position, showcasing its robustness against this phenomenon. This highlights a key advantage of RAG: its ability to effectively retrieve and utilize relevant information regardless of its location within the context.
E. Annotation Details
Recent advancements in LLMs have demonstrated their ability to generate high-quality text comparable to human output, positioning synthetic data generated by LLMs as viable alternative or complement to numan-generater data (Long et al., 2024). Benefiting from this, we use GPT-4o to generate highly diverse question-answer pairs across multiple scenarios and tasks.
Specifically, for each context type and task category, we design different prompts and seed questions based on distinct guiding principles. To better understand how we generate QA pairs in LaRA, we present the prompts used for generating financial statement QAs as an example. The prompts for the Location, Reasoning, Comparison, and Hallucination tasks shown in Figures 8, 9, and 10, respectively.
Each task's 's annotations share some common principles, such as ensuring that questions are unanswerable without the provided context and keeping responses concise to avoid lengthy elaborations. Additionally, there are specific requirements to more precisely define the scope of the questions. For example, the location task, we require the answer to be directly extracted from the context and prohibit any interpretive phrasing. In contrast, for the reasoning task, the answer must be impossible obtain through simple lookup. Finally, seed example questions are provided for in-context learning, both as reference and ensure clear output formatting for easier subsequent processing. During fine-tuning prompts and seed questions, find that once the problem types precisely defined, the seed examples have significant impact on the quality of the generated QA pairs. The higher the diversity of seed examples, the less uniform the generated questions become. Therefore, we strive to ensure that the seed questions cover various aspects and reflect the practical questions that users might realistically ask in real-world scenarios. For each task, we provide 3-5 seed examples.
F. Task Examples
For each task type, we provide three test examples, for each type of context, specifically demonstrate queries in LaRA that are close to real-world scenarios (Figure 11, 12, 13, and 14). Although we do not explicitly define lower-level sub-tasks within the four main task categories, it is evident from the examples that the emphasis of the same task varies across different context types. For example, financial statements are well-suited for mathematical calculations, while research papers are ideal for comparing different viewpoints and data.
We believe the most essential principle that all questions must adhere to is practicality that is, the questions should reflect
16

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Auggennted Generation
Example prompt used for generating Location QAs in financial statements
Task: You are a financial question designer. Based on the provided financial statement, create a factual question and its corresponding answer that strictly satisfies these criteria:
Design Requirements:
1. Direct Extraction:
The answer must be explicitly stated in single contiguous segment of the document
Require pinpoint localization (e.g.. specific value, exact date, named section)
Answerability Constraints:
Unanswerable without the provided context
Must reference concrete elements (numerical values, named metrics, verbatim terms)
3. Response Specifications:
Maximum answer length: 20 words
Prohibit any interpretive phrasing
Example QAs:
(Q: "What the amount of impairment changes recorded to goodwill during the three months ended December 31, 2023?", A:2$62.8 million.))
[Q: "How many reportable business segments does 2U, Inc. have?", 2 Inc. has two reportable business segments."
Financial Statement: {Context)
Figure 7. The prompt for generating QA pairs.
those that person could realistically ask based on the given context. While questions designed solely to challenge LLMs have academic value, they do not address the considerations necessary for designing real-world RAG or LC planning system. Our work aims to explore how to make more optimal choices under a realistic query distribution.
17

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmentee Generation
Example prompt used for generating Reasoning QAs financial statements
Task: You are financial analyst. Design a question requiring mathemattica//ogicall derivation from the financial statement, adhering to:
Design Requirements:
1. Require multi-step processing of:
Numerical calculations (e.g., ratios, growth rates)
Temporal comparisons Reasonable inferences
2. Answerability Constraints:
Unanswerable without the provided context Impossible to answer through simple lookup
Response Specifications:
Maximum answer length: avoid too long
Forbid speculative probabilistic responses
Example QAs:
[Q: "Calculate the percentage increase in total assets from December 31, 2023, to March 31, 2024, and explain what this increase indicates about the company's 's financial position during this period.", A:oToal assets increased from $22,008,7 739 on December 31, 2023, to $36,852,097 on March 31, 2024, which is an increase of $14,843,358 The percentage increase is (14.843.358 22,008, 7299) 100, which equals upproximately 67.5%. This significant increase in total assets indicates improved financial strength, potentially from additional capital inflows asset acquisitions during the period.")
Financial Statement: [Context)
Figure 8. The prompt for generating QA pairs.
18

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmentee Generation
Example prompt used for generating Comparison QAs financial statements
Task: You are financial analyst. Create a comparative question requiring synthesis of two distinct sections from financial statement, following these principles:
Design Requirements:
1. Must reference:
Disparate metrics (c.g., departmental budgets vs regional sales)
Chronological differences (quarterly/annual comparisons)
Contrasting categories (actual projected figures)
2. Dependency Rules:
Each segment provides unique essential information
No overlapping data between required sections
3. Answerability Constraints:
Unanswerable without the provided context
Answer must demonstrate relational understanding
Require explicit mention of both referenced sections
4. Response Specifications:
Maximum answer length: avoid too long
Comparisons must use contextually appropriate units
Example QAs:
(Q: "How did the average revenue per Full Course Equivalent (FCE) enrollment change in the Degree Program Segment compared the Alternative Credential Segment from 2022 2023?", A: average revenue per FCE enrollment increased by 17.8% in the Degree Program Segment from $2,447 in 2022 to $2,883 in 2023, while it decreased by 6% in the Alternative Credential Segment from $3,897 in 2022 to $3,662 in 2023.")
Financial Statement: {Context)
Figure 9, The prompt for generating QA pairs.
19

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmented Generation
Example prompt used for generating Hallucination Detection QAs financial statements
Task: You are a financial analyst. Design a pseudo-relevant question that appears answerable but actually lacks sufficient basis in the financial statement, following these principles:
Design Requirements:
1. Surface-level Relevance:
Use doocument-specffii terminology
Reference actual sections/metries as distractors
2. Unanswerability Guarantees:
Absolutely not mentioned in the context
Missing critical data points required for resolution
No inferential path from provided information
Confirm absence of:
Direct mentions
Implied values
Comparable proxies
Example QAs:
(Q: "What measures are being taken to mitigate foreign currency risk?", The document does not specify any measures being taken to mitigate foreign currency risk."}
(Q: "What is the company's market share in the non-combustible nicotine-related products sector?", The document does not provide specific percentage for the company's market share in the non-combustible nicotine- related products sector.")
Financial Statement: [Context)
Figure 10. The prompt for generating QA pairs.
20

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmentee Generation
Examples of location task
Novel
[Query] What does Mrs. Fitzgerald think of Violet and her family? (Ground-truth Answer] She believes they are very common.
Academic Paper
[Query] According paper 2, what impact does self-critiquing have the plan generation performance of LLMs in comparison using external verifier?
[Ground-truth Answer] Self-critiquing degrades the plan generation performance compared using an external, sound verifier.
Financial Statements
[Query] How much cash was provided by financing activities during the three months ended March 31, 2024? [Ground-trute Answer] S6.3 million.
Figure 11. The examples location task.
Examples of reasoning task
Novel
[Query] Why did Richard know by the second meeting Karen didn't like sweets? [Ground-trude Answer] Because during their first meeting over a meal, Richard noticed that Karen quietly disposed of the dessert on her plate.
Academic Paper
[Query] Based on paper why does the I.LM+LLM hackprompting system underperform compared the L.M+VAL. system plan generation?
[Ground-truth Answer] The LLM+LLM backprompting system underperforms because verifier LLM produces significant number of false positives, declaring incorrect plans as valid, which undermines the reliability of the system. In contrast, the LLM+VAL system uses an external sound verifier, VAL, which provides accurate validation, lcading to better overall performance due to fewer verification error..
Financial Statements
[Query] What percentage of the total fair value of the liabilities assumed during the merger with DHC warrant liabilities?
(Ground-tuuth Answer] To calculate the percentage of warrant liabilities in the merger with DHC, sum the warrant liabilitys value, $1,913 913,7,, and divide it by the total net liabilities assumed, $9,863,190 196. The percentage is found by (1,913,777 9,863,196 100, which equals approximately 19.4%.
Figure 12. The examples of reasoning task.
21

---

LaRA: Benchmarking LLMs with Long-Context and Retrieval-Augmented Generation
Examples of comparison task
Novel
[Query] How does the narrator's 's attitude towards Turkey Nippers compare to attitude towards Bartleby? (Ground-truth Answer] The narrator is willing to overlook Turkey's 's Nippers' flaws due to their usefulness, just as he tolerates Bartleby's 's peculiar behavior because of his steady work and presence.
Academic Paper
[Query] How do the approaches for incorporating context into ranking functions differ between the method proposed paper and in paper 1?
[Ground-truth AnswerJ Paper incorporates context using features comparing neighboring items, while paper uses self-attention mechanism account for interactions hetween items during both training and inference.
Financial Statements
[Query] How did the total current assets and total current liabilities change for CISO Global, Inc. 31, 2023, to March 31, 2024? (Ground-truth Answer] CISO Global's total current assets decreased from $10,957,81 December 31, 2023, $9,276,063 on March 31, 2024, while total current liabilities increased from $26,071,102 $32,604, 126 in the same period.
from December
Figure 13. The examples of comparison task.
Examples of hallucination detection task
Novel
[Query] What type of flower did Alexander place Violet's grave the cemetery? (Ground-truth Answer] TThe text does mention Alexander placing any type of flower on Violet's grave the cemetery.
Academic Paper
[Query] In paper 2, what are the implications Al-enhanced NMR processing on the prediction of chemical reaction pathways? [Ground-truth Answer] Paper 2 does not discuss the implications of Al-enhanced NMR processing on the prediction of chemical reaction pathways.
Financial Statements
[Query] What were the results of the company's 's environmental sustainability initiatives? [Ground-truth Answer] The financial statement does mention or provide any details about the results of the company's environmental sustainability initiatives.
Figure 14. The examples of hallucination detection task.
22


Reviews:
[{"review_id": "K76qNIXWXp", "reviewer": "Reviewer_WGRn", "summary": "This paper studies the problem of benchmarking RAG and long-context LLMs. The authors first revisit the existing benchmarks to compare RAG and long-context LLMs. They further construct a dataset called LaRA, which contains location-related question, reasoning-related question, comparison-related questions and hallucination detection questions. They conduct experiments with seven open-source LLMs and four proprietary LLMs and systematically analysis the comparison between RAG and long-context LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive review! Please see our responses below.\n\n>**Discuss with [1]**\n\nThanks for pointing out this important related work. [1] mainly investigated the phenomenon that increasing the number of retrieved passages does not consistently improve the performance of LLMs. As the amount of retrieved information increases, performance first increases and then decreases. Some of our experimental results align with the conclusions in [1]. In Figure 2, we observe that as the number of retrieved chunks and chunk size increase, LLM performance first improves and then declines. Furthermore, weaker models compared to stronger ones, such as Qwen-7B versus Qwen-72B, begin to show performance degradation earlier, indicating that weaker models are more susceptible to the influence of large amounts of irrelevant noise in the retrieved information. We will add the discussion with [1] in the revision. \n\nWe further explored the impact of increasing the number of chunks to 100 on Qwen-2.5-7b-instruct and Qwen-2.5-72b-instruct with 128k context and find that too many chunks can cause RAG's performance to decrease.\n\n|Model\\\\#Chunk|5|10|15|20|25|30|40|50|80|100|LC|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Qwen-2.5-7B|56.30|60.15|62.80|61.65|59.76|59.62|59.22|58.47|57.39|55.61|48.91|\n|Qwen-2.5-72B|62.68|63.48|64.05|64.06|65.08|67.78|68.18|67.57|66.89|65.87|65.11|\n\n>**I would recommend the authors add another “RAG” and “LC” column in Table 2 to make things clearer.**\n\nThanks for this advice, and we will change it in our revision!\n\n>**Typos**\n\nThanks for pointing out these typos. We have fixed them in the revision and will keep polishing our paper.\n\n\n### References\n\n[1] Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG. ICLR 2025.", "claims_and_evidence": "I think the claims are well-supported.", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental designs make sense to me.", "relation_to_broader_scientific_literature": "Yes, could be interesting the a broader community.", "essential_references_not_discussed": "I would encourage the author to discuss [1] which is also a paper comparing RAG and long-context LLMs.\n\n[1] Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG. ICLR 2025.", "other_strengths_and_weaknesses": "- Line 68, “lanuage” typo\n- Line 156, Section “3” link is not effective.\n- I would recommend the authors add another “RAG” and “LC” column in Table 2 to make things clearer.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "scfnUiPHcs", "reviewer": "Reviewer_M1DH", "summary": "This paper introduces a new benchmark called LaRA, which is designed to systematically compare Retrieval-Augmented Generation (RAG) and long-context (LC) large language models (LLMs). It evaluates 11 models on 2,326 test cases across four key tasks (information retrieval, reasoning, comparison, and hallucination detection) using naturally occurring long texts. The study finds that neither approach is universally superior. The choice depends on model size, task type, and context length. RAG benefits weaker models and excels in hallucination detection, while LC performs better in reasoning and structured text processing, particularly for stronger models with extensive context capabilities. The findings offer practical guidelines for optimizing LLM applications through strategic use of RAG and LC.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to Thank you for your constructive review, as well as your positive feedback! Let us answer your question below.\n\n>**Some implementation details are missing from the main paper---How many chunks were used in the setting of RAG experiments?**\n\nIn Section 4, paragraph “Implementation of RAG (line 269)”, we provide the details that \"Our evaluation employs a standardized configuration with a chunk size of 600 tokens, 5 chunks per document, and an overlap of 100 tokens between chunks.\" In the main results, we use a chunk size of 600 and 5 chunks. Additionally, we further explore the impact of more chunks in Figure 2. \n\n>**Some parts (with 2024 knowledge) of the benchmark may not be less useful for future LLMs.**\n\nFor future LLMs, all existing benchmarks may become outdated due to information leakage when they are used to train newer LLMs. However, on one hand, we have already used nearly the most recent corpus in our data selection, with almost all contexts being released in the second half of 2024, except for novels. On the other hand, in addition to providing datasets for evaluation, we also provide concrete pipeline for creating new data, which can be used to generate new datasets with more updated contexts.\n\n>**Is hallucination detection a valid task? How do we know the performance change is not purely because of the change in context length?**\n\nHallucination remains a significant challenge for LLMs. While RAG can potentially mitigate this issue, we provide a quantitative assessment of RAG's effectiveness in reducing hallucinations specifically in long-context scenarios. Our evaluation spans models of various sizes, including both proprietary and open-source LLMs, comparing RAG against standard long-context input. Our experimental design focuses on measuring models' ability to abstain from answering when presented with unanswerable questions—defining abstention as correct behavior and hallucination as incorrect.\n\nThe experimental results yield three key findings: (1) LC LLMs are substantially more susceptible to hallucinations compared to RAG; (2) Model strength does not correlate with reduced hallucination rates that stronger LLMs do not demonstrate fewer hallucinations; (3) Increasing context length corresponds to higher hallucination probability. Two of three findings are independent of the increase in context length, therefore we believe this is an effective task that provides strong support for the study of hallucinations in RAG and LC LLM.\n\n>**In Line 367, are the results for 128k and 32k contexts directly comparable? Do they share the same input?**\n\nYes, they are directly comparable. At Line 244, we mention that generating QA pairs for long texts is inherently a long-context generation problem. To improve generation quality, we divide long contexts into several shorter segments and generated QA pairs based on these individual segments. This means that for both 32K and 128K contexts, GPT-4o generated QA pairs using the same prompts and similarly sized segments. Therefore, the distribution of these QA pairs can be considered approximately equivalent across different context lengths. \n\nSpecifically, in lines 249-252 we wrote \"we split the long context into multiple segments, each approximately 10k tokens in length, and input them individually into GPT-4o to generate QAs.\" In lines 267-273, we clarified that we have different segmentation strategies for different types of contexts. We will clarify this point more clearly in the final version.\n\n>**Are there task-specific scores for the results in Figure 2? Do different tasks share the same trend?**\n\nWe find that the trends across different tasks are generally similar, but there are some exceptions. Below we provide the results of Qwen-2.5-72B at 32k length. We find that for the location task, novels perform worse than other context types, possibly because novels contain more similar content, making it difficult to locate answers. For the reasoning task, papers performed best, which we speculate is because papers have stronger logical structure, lower information redundancy, and are more conducive to reasoning. We will add a systematic analysis of performance across different tasks on various context types in the appendix.\n\n| |location|Reasoning|Comparison|Hallu|\n|-|-|-|-|-|\n|Novel|72.00|76.27|71.11|88.14|\n|Financial|89.19|61.02|72.41|82.20|\n|Paper|88.68|84.91|63.16|84.91|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper contributes by providing a new benchmark for RAG vs. LC comparison with four carefully designed key tasks. It also provide a comprehensive analysis of different aspects.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The authors provide a novel benchmark featuring four types of questions.\n2. Comprehensive experiments are performed on the benchmark with multiple LLMs.\n3. Practical insights are provided based on the experimental results.\n\nWeaknesses:\n1. Some implementation details are missing from the main paper.\n2. Some parts (with 2024 knowledge) of the benchmark may not be less useful for future LLMs.\n3. More analyses could be done on the experiments (see Questions).", "other_comments_or_suggestions": "1. The year information should be included when adding in-text citations.", "questions_for_authors": "1. How many chunks were used in the setting of RAG experiments?\n2. Is hallucination detection a valid task? How do we know the performance change is not purely because of the change in context length?\n3. In Line 367, are the results for 128k and 32k contexts directly comparable? Do they share the same input?\n4. Are there task-specific scores for the results in Figure 2? Do different tasks share the same trend?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYGnWp7dsG", "reviewer": "Reviewer_cyju", "summary": "The paper proposes LaRA, a benchmark that attempts to answer if RAG is still necessary compared with long-context LLMs.\n\nThe LaRA dataset is constructed from novels, academic papers, and financial statements with four tasks: locating specific information, comparing different parts of the text, reasoning about the content, and detecting hallucinations (questions that are not answerable from the provided context).\n\nResults show that the choice between RAG and LC is not trivial, as it varies significantly depending on factors such as model size, query type, type of tasks, context length, context type, and number of retrieved chunks.\n\nNotably, the proprietary LLMs tends to perform better in the long context configuration, except for the hallucination category.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive feedback, and we are very happy that the reviewer liked how our dataset is structured!  Let us answer your question below.\n\n>**RAG retrieval setup: the potential unfairness in the RAG setup, as it relies on a limited number of chunks without further experimentation**\n\nYes, in the main results, each query retrieves 5 chunks. The specific method is to search for 5 chunks based on similarity using the GTE embedding model and search for 5 chunks using BM25, then take their intersection. If fewer than 5 chunks are found, the remaining chunks are equally selected from the two retrieval methods. We adopt this hybrid search strategy with reference to [1], which is a strong method.\n\nIn Figure 2 of the draft, we provide experimental results with different chunk numbers and sizes, comparing the impact of the amount of retrieved information on smaller models (Qwen-2.5-7B) and larger models (Qwen-2.5-72B). We gradually increase the number of chunks from 5 to 30. Since 30 retrieved chunks means the total input length has already reached 30 * 600 = 18,000, which is on the same order of magnitude as long-context, further increasing the number of chunks would cause RAG to lose its significant efficiency advantage. We set 5 as the default number of chunks and 600 as the chunk size primarily by referencing previous works [1, 2].\n\nWe further explore the impact of increasing the number of chunks to 100 on these two models with 128k context, and the results are provided below. We find that too many chunks can cause RAG's performance to decrease rather than increase.\n\n|Model\\#Chunk|5|10|15|20|25|30|40|50|80|100|LC|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Qwen-2.5-7B|56.30|60.15|62.80|61.65|59.76|59.62|59.22|58.47|57.39|55.61|48.91|\n|Qwen-2.5-72B|62.68|63.48|64.05|64.06|65.08|67.78|68.18|67.57|66.89|65.87|65.11|\n\n>**The definition of \"long context\" needs further clarification, as the experiments are capped at 128k context. What happens beyond this limit, and does RAG remain relevant in such scenarios?**\n\n**This is one of the key designs of our benchmark.** Under extremely long context, RAG has an overly obvious advantage. An extreme example is knowledge base-based QA, where LC LLM cannot process the entire knowledge base and struggles to rely on external knowledge to answer correctly.\n\nAs emphasized in the paper, when collecting contexts, we choose texts that are as close as possible to the LLM's limit without exceeding it. This allows for a fair comparison of the actual capabilities between RAG and LC LLMs. If the context exceeds the LLM's input limit, we need to employ tricks like truncation, which could result in the answer to a question not being present in the LLM's input, thus failing to reflect the LLM's true capabilities. In Table 1, we conduct relevant experiments to verify the impact of such excessively long contexts. In lines 127-149, we specifically analyze why these overly long contexts cannot be appropriately used to compare RAG and LC LLMs in long-context QA scenario. \n\nWe will clarify this point more clearly in the final version.\n\n>**The framing of the \"Hallucination detection\" task**\n\nThanks for this suggestion. We will rename \"hallucination detection\" to the more appropriate term \"hallucination occurrence\" to express whether RAG and LC LLM produce hallucinations.\n\n### References\n\n[1] Wang, Xiaohua, et al. Searching for best practices in retrieval-augmented generation. EMNLP 2024.\n\n[2] Li, Zhuowan, et al. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. EMNLP 2024.", "claims_and_evidence": "This is a dataset/benchmarking paper. The main claims are in experiment findings and are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Overall, I liked how the dataset is structured for evaluation, featuring recent data across various domains and question types, with both 32k and 128k context configurations.  \n\nHowever, the evaluation setup is somewhat restrictive, as all data is designed to fit within the LLM's context window. This raises a key question: what happens when content exceeds this limit? Should RAG be used, or is the context ensemble approach described Section 2 more suitable? A clearer definition of the scope of long-context may be needed.", "theoretical_claims": "Not Applicable.", "experimental_designs_or_analyses": "My main concern is the RAG retrieval setup. The paper mentions \"5 chunks per document\" and a \"hybrid search combining embedding similarity and BM25.\" However, it’s unclear how many chunks are retrieved per question—possibly 5. If the RAG setup retrieves only a small number of chunks without further experimentation, it might be unfair, as important context could be missed due to retrieval inaccuracies.", "relation_to_broader_scientific_literature": "Existing literature provided conflicting evidence in terms of whether RAG is still necessary given LLMs. This paper attempts to answer this question by resolving limitations of existing approaches (Insufficent context lengths, Data Leakage, Inappropriate Contexts Handling). Findings in this paper show that the choice between RAG and LC is not trivial, as it varies significantly depending on factors such as model size, query type, type of tasks, context length, context type, and number of retrieved chunks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths\n1. The paper is well-written, easy to follow, and tackles an important question by comparing RAG with long-context LLMs.\n2. I liked how the dataset is structured for evaluation, featuring recent data across various domains and question types, with both 32k and 128k context configurations.  \n3. The interpretation of results is well-explained, even though no definitive solution due to \"no silver bullet.\"\n\n### Weaknesses\n1. As noted in the *Experimental Designs or Analyses* section, my main concern is the potential unfairness in the RAG setup, as it relies on a limited number of chunks without further experimentation. This could lead to retrieval failures, affecting performance.\n2. The definition of \"long context\" needs further clarification, as the experiments are capped at 128k context. What happens beyond this limit, and does RAG remain relevant in such scenarios?", "other_comments_or_suggestions": "1. I think the framing of the \"Hallucination detection\" task could be improved. Typically, \"hallucination\" refers to model outputs, whereas in this paper, it is used to describe a question that cannot be answered based on the given context. Referring to a question as a \"hallucination\" may be somewhat misleading.\n2. I think more analysis on the retrieval size could strength the claims of this paper.", "questions_for_authors": "1. Please elaborate the RAG retrieval setup.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b9C7QQkwkU", "reviewer": "Reviewer_dbyM", "summary": "The paper investigates whether Retrieval Augmented Generation (RAG) or Long Context (LC) generation is superior to answer questions with LLMs. For this, it first identifies shortcomings of evaluations in current studies and then introduces its own dataset (LaRA) that aims to mitigate these shortcomings. In an extensive evaluation with different models, the authors find that choosing RAG over LC or vice versa depends on various factors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive feedback, which helps enhance the clarity of our paper! Let us answer your question below.\n\n>**More details about the QA generation process and answer to question “Could it be that the generation produces wrong answers, and the final LC or RAG answers are actually correct?”**\n\nThanks for this advice. We conducted human inspection after the generation. After constructing prompts and example QAs, we sample 40 generated QAs for each context type and task, which are then manually verified for validity. We only stop modifying the prompts and in-context examples for a given context type and task when the accuracy reaches above 90%. Additionally, larger models compared to smaller ones, and stronger proprietary models compared to open-source models, consistently demonstrate higher accuracy, which further validates the overall correctness of the QAs. If the correctness of QAs were not guaranteed, we would likely observe random results. While this process cannot ensure that all QAs are 100% correct, it does guarantee a very high accuracy rate, making them effective for evaluation purposes. We will include these details in our revision.\n\n>**The impact of retrieval strategies and answer to question “What role does retrieval play in comparing LC vs. RAG?”**\n\nWe agree that comparing RAG and LC is influenced by many factors, and a complete advanced RAG system can be more complex and have more modules, including query rewrite, different retrieval strategies, reranking, summarization, etc. However, we would like to highlight that this does not affect the value of our work from two perspectives. First, [5] conduct a systematic analysis of RAG implementations, and we adopt their advice to use a hybrid search strategy combining vector search and BM25. We choose gte-1.5 [4], a very strong embedding model released in late 2024, for search and reranking, ensuring that our RAG implementation already employs a strong strategy. On the other hand, our benchmark itself is one of the core contributions, providing effective support for future systematic exploration of the impact of different RAG modules in long-context QA.\n\nBelow are the experimental results of replacing gte-1.5 with bge-m3 [6] and adding Recomp [7] as a summarization module (also suggested in [5]). As can be seen, further complicating the RAG process does not bring significant additional gains, but instead makes retrieval a computationally-intensive process.\n\n| |ours (32k)|bge-m3 (32k)|Recomp (32k)|ours (128k)|bge-m3 (128k)|Recomp (128k)|\n|-|-|-|-|-|-|-|\n|Qwen-2.5-7B|62.62|61.78|62.45|56.30|55.81|56.22|\n|Qwen-2.5-72B|69.97|70.11|70.34|62.68|61.88|63.09|\n\n\n>**Is it possible that GPT-4o creates more complex questions than GPT-4o can answer?**\n\nYes, this holds true under our generation process, as we employ a short-context generation method for creating QAs. At Line 244, we mention that generating QA pairs for long texts is inherently a long-context generation problem. To improve generation quality, we divide long contexts into several short segments and generated QA pairs based on these individual segments. This means that GPT-4o only needs to process a short context when generating QAs, but when answering, it needs to find answers from the complete context, which is far more challenging than generating them.\n\n>**What is the stance of prior literature on your results?**\n\nWhile [1] observes that longer contexts pose challenges for RAG, our work provides a more nuanced analysis, demonstrating that RAG performance is comparable to LC LLMs on tasks such as location and hallucination detection. [2] states that open-source models perform poorly on QA, so we discuss the results of open-source and proprietary models separately in our paper.  [3] finds that more chunks can increase performance of RAG; we also conduct related experiments in Figure 2 to verify the impact of using more chunks. Detailed discussions of these connections to prior work will be added to the appendix in the final version.\n\n### References\n\n[1] Leng, Quinn, et al. \"Long context rag performance of large language models.\" arXiv 2024.\n\n[2] Schimanski, Tobias, et al. Towards faithful and robust llm specialists for evidence-based question-answering. arXiv 2024.\n\n[3] Li, Zhuowan, et al. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. EMNLP 2024.\n\n[4] Zhang, Xin, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. ACL 2024.\n\n[5] Wang, Xiaohua, et al. Searching for best practices in retrieval-augmented generation. EMNLP 2024.\n\n[6] Chen, Jianlv, et al. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv 2024.\n\n[7] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. ICLR 2024.", "claims_and_evidence": "Yes, the majority of the claims are backed by clear evidence. The authors did a very good job of outlining the shortcomings of existing studies. The experimental results are interpreted well. Also, the paper is well-written. Thus, it is simple to follow.", "methods_and_evaluation_criteria": "The paper uses a synthetic pipeline to generate QA pairs. The dataset itself is central to the paper. However, the authors remain vague when describing the exact generation process. For me, it is unclear by which criteria the in-context samples are selected and what the authors mean by refining the prompts when \"the pass rate does not meet a predefined threshold\" (lines 240-241). Also, while human evaluation is performed when evaluating the LLM's judgements of the answer quality, no such evaluation is performed in the actual question generation process. Thus, as a reader, I just need to \"trust\" that this went well. Could it be that the generation produces wrong answers, and the final LC or RAG answers are actually correct?\n\nAlso, it remains unclear what the synthetic question generation process means for the task complexities. Is it possible that GPT-4o creates more complex questions than GPT-4o can answer?\n\nIn my eyes, the one core contribution of the paper revolves around a good dataset. The authors describe very well that current literature has significant shortcomings but then do an uncareful job when defining their own data.", "theoretical_claims": "This is not really a theory paper, so I have no comments.", "experimental_designs_or_analyses": "In my eyes, the comparison between RAG and LC is a bit more tricky than presented. Essentially, what the investigations may mean is that you compare two different retrieval strategies. If the model stays the same, then the question becomes whether retrieval works better in LC (within the model) or in RAG (with an external retriever). This notion does not find space. While models are alternated, retrievers are not. There is literature proposing advanced retrieval strategies (https://arxiv.org/abs/2406.14162, https://arxiv.org/abs/2311.09476). This could significantly enrich the observations.\n\nBesides, there could be more alignment of the findings with prior literature. For instance, Leng et al. (https://arxiv.org/pdf/2411.03538) find similar results that longer contexts are harder. Schimanski et al. (https://arxiv.org/abs/2402.08277) find that open-source models are generally lacking QA capabilities. Li et al. (https://arxiv.org/abs/2407.16833) find that more chunks increase the performance. For me, the paper would become more credible if these results were reflected on.", "relation_to_broader_scientific_literature": "As stated, relating to existing insights in the results and broader literature in information retrieval may be helpful.", "essential_references_not_discussed": "I have pasted some examples above. I think nothing entirely critical is missed out.", "other_strengths_and_weaknesses": "As stated above.", "other_comments_or_suggestions": "While I like the extensive motivation of the paper, I feel like this takes too much space overall in the paper. The appendix is relatively short. More experiments and human data investigations would serve the soundness of the paper well.", "questions_for_authors": "As stated above:\n- Could it be that the generation produces wrong answers, and the final LC or RAG answers are actually correct?\n- Is it possible that GPT-4o creates more complex questions than GPT-4o can answer?\n- What role does retrieval play in comparing LC vs. RAG?\n- What is the stance of prior literature on your results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces LaRA, a benchmark for RAG vs. LC (long context) comparison. They evaluated 11 different LLMs (7 open-source, 4 proprietary) on LaRA. Their key finding is that there's no universally superior approach; the optimal choice (RAG vs. LC) depends on a variety of factors including the LLM's parameter size, its inherent long-context processing ability, the specific context length used, the type of QA task, and the properties of the chunks retrieved by RAG. The paper aims to provide practical guidelines based on these findings and address limitations of prior work.

The reviewers raised several important points, however, overall, these were adequately addressed in an excellent rebuttal in which the authors' thorough responses significantly improved the rigor of the work. I would like to see theses discussions make it into the next version of the paper. 

These discussions primarily covered data quality, where the authors clarified their dataset generation process and filled in details about their human validation. They explained their method (sampling QAs, manually checking against a >90% accuracy target, refining prompts), and they also shared quantitative error data (~2.5%) building confidence in the benchmark quality. They also alleviated concerns about the fairness of their RAG setup, explaining the hybrid retrieval approach, referencing existing experiments, and importantly, providing new results that showed performance could actually drop with too many retrieved chunks or different RAG components. Finally, smaller issues around references, terminology, and specific implementation details were also handled well, with the authors agreeing to make the needed changes.
Decision: Accept (poster)