RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Authors: Jennifer Hsia
Afreen Shaikh
Zora Zhiruo Wang
Graham Neubig
Venue: ICML 2025 poster (2025)
Keywords: Retrieval-Augmented Generation
RAG
Evaluation
Information Retrieval
Question Answering
Primary area: general_machine_learning->evaluation
TL;DR: We show that retrieval-augmented generation systems depend more on the reader’s ability to handle noise than on retrieval quality, and introduce a framework (RAGGED) to systematically evaluate their stability and scalability.

Abstract: Retrieval-augmented generation (RAG) enhances language models by integrating external knowledge, but its effectiveness is highly dependent on system configuration. Improper retrieval settings can degrade performance, making RAG less reliable than closed-book generation. In this work, we introduce RAGGED, a framework for systematically evaluating RAG systems across diverse retriever-reader configurations, retrieval depths, and datasets. Our analysis reveals that reader robustness to noise is the key determinant of RAG stability and scalability. Some readers benefit from increased retrieval depth, while others degrade due to their sensitivity to distracting content. Through large-scale experiments on open-domain, multi-hop, and specialized-domain datasets, we show that retrievers, rerankers, and prompts influence performance but do not fundamentally alter these reader-driven trends. By providing a principled framework and new metrics to assess RAG stability and scalability, RAGGED enables systematic evaluation of retrieval-augmented generation systems, guiding future research on optimizing retrieval depth and model robustness.

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Jennifer Hsia Afreen Shaikh 2
Zhiruo Wang²
Graham Neubig
Abstract
R3: Robustness to irrelevant information $6
R4: Impact of retriever choice $7
generation (RAG) enhances language models by integrating external knowl- edge, but its effectiveness highly dependent on system configuration. Improper retrieval settings can degrade performance, making RAG less reli- able than closed-boot generation. In this work, introduce RAGGED, framework for system- atically evaluating RAG systems across diverse retriever-reader configurations, retrieval depths, and datasets. Our analysis reveals that reader ro- bustness to noise is the key determinant of RAG stability and scalability. Some readers benefit from increased retrieval depth, while others de- grade due to their sensitivity to distracting content. Through large-scale experiments on open-domain, multi-hop, and specialized -orizend datasets, we show that retrievers, rerankers, and prompts in- fluence performance but do not fundamentally alter these reader-driven trends. By providing a principled framework and new metrics assess RAG stability and scalability, RAGGED enables systematic evaluation of retrieval-augmenter gen- eration systems, guiding future research on opti- mizing retrieval depth and model robustness.
Documents
Retriever
Reader
R2: Impaat increasing context documents. $5
R1: When does RAG improve over closed-book generation? $4
Roadmap of what our framework RAGGED anal- yses across the RAG pipeline.
Figure 1:
tion. Understanding when and why RAG helps or harms is critical for optimizing system design.
Most prior work evaluates RAG under controlled conditions and curated contexts (Liu et al. (2033 Cuconaaau al. (024), which fail to reflect real-world retrieval challenges. In prac- tice, retrieved contexts contain both relevant and irrelevant information, making the reader model's ability to filter noise a critical factor in RAG success. Additionally, prior studies provide conflicting findings on retrieval depth (k)-- while some suggest increasing k improves performance (Izacard) & Grave 2021), others observe diminishing returns (Liu et al. 2023 even degradation high k (Cuconasu et al. 2024 Jiang et al. 2024). This lack of consensus leaves practitioners without clear guidance on how to configure RAG systems for different tasks.
1. Introduction
Retrieval-augnee -augmeneed generation (RAG) (Chen et al. 2017 Lewis et al., 2020) enhances large language models (LLMs) by retrieving relevant external contexts, enabling more spe- cific and factually grounded responses. However, despite its promise, RAG's effectiveness improper configurations degrade model performance, leading to outputs that are worse than closed-book genera-
To address these challenges, we introduce RAGGED [Retrieval-Augmented Generation Generalized Evalu- ation Device), framework for systematically evaluating RAG performance across retrieval depths, model architec- tures, and retrieval conditions. Unlike prior work, which often relies on synthetic or manual retrieval modifications, RAGGED assesses models under realistic retrieval scenar- ios - analyzing performance on naturally retrieved top-k contexts rather than manually curated, oracle-aware con- texts.
guaranteed. In fact,
Equaal contribution 'Macine Learning Department, Carnegie Mellon University, Pittsburgh, USA "The Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA. Correspon- dence to: Jennifer Hsia <jhssia @ andrew.cmu.cdu.>
Our study reveals that reader robustness to noise is the pri- mary factor driving RAG stability and scalability, rather than retriever quality alone. To quantify this, we intro- duce two new metrics: the RAG Stability Score (RSS) and RAG Scalability Coefficient (RSC), providing princi-
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Code and data for the RAGGED framework available at https//gtthuk.co///ellbbaaagggg

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
datasets, and reader models. Prior evaluations often rely oracle-aware curation of contexts fixed retrieval con- figurations, thereby failing to capture how RAG systems behave under real-world retrieval conditions. These limi- tations obscure the key factors that determine RAG effec- tiveness, particularly in terms of stability (consistency near the retrieval depth that yields optimal performance) and scalability (sustained performance gains as retrieval depth increases).
CUIDING
0A pert
use more docs
AAnGRNN nelsy does
robust more. notey does
# of cix passages
ctx passages
Figure 2: While some readers exhibit 'peak-thent -henl-eelene (left), others exhibit "improve-then-plane -paatea" behavior (right) with increasing number of contexts.
To address these gaps, we introduce RAGGED, a system- atic framework for evaluating RAG systems across diverse retrieval settings. RAGGED provides a principled approach to assessing model behavior and optimizing retrieval depth for robust performance.
pled framework for evaluating retrieval effectiveness across diverse configurations.
Using RAGGED, conduct a large-scale empirical study to answer four key questions (Figure I), each corresponding to core section of our paper:
Objectives of RAGGED: RAGGED provides a structured evaluation of RAG effectiivnnsss, enabling:
Identifying whether increas- ing k improves or harms model performance.
Retrieval-depth analysis:
1. Under What Conditions Does Retrieval Outperform Closed-Book Generation? ($4) We analyze when retrieval improves performance and identify that some readers frequently benefit from RAG, particularly at large k while others degrade due to noise sensitivity.
Stability assessment: Measuring how consistently models maintain performance near their optimal re- trieval depth, avoiding sharp performance drops.
2. How Does Retrieval Depth Impact Stability and Scalability? ($5) We identify two distinct reader be- haviors: puprove-then-platee models, which scale effectively, and peak-then-decline models, which de- grade at higher i (Figure 2).
Scalability evaluation:
Determining whether a model continues to benefit from increasing retrieval depth or experiences diminishing returns.
Reproducible benchmarking: Standardizing evalua- tion across different RAG implementations.
3. How Do Readers Handle Noisy Retrieval, and Is Prompting Reliable Fix? ($6) We evaluate RAG performance under realistic retrieval conditions, show- ing that noise sensitivity-rather rather than retriever quality alone determinss downstream effectiveness. We also assess whether instructing readers to focus on relevant content mitigates noise sensitivity.
To operationalize these goals, we introduce two new metrics:
RAG Stability Score (RSS) The RSS metric quantifies how consistently a model maintains performance around its op- timal retrieval depth (k*). A stable model should exhibit minimal performance fluctuation within small retrieval window, while an unstable model's performance will de- grade noticeably when retrieval depth varies slightly. RSS is formally defined as:
4. When Does Better Retriever Actually Lead to Better Performance? ($7) While retriever choice shifts overall performance. it does not alter fundamen- tal reader behaviors, thus highlighting the reader as the key driver of stability and scalability.
Performance at k
ke(k*
RSS
(1)
Performance at k:
where he is the retrieval depth that yields peak model per- formance, and A defines a local window (e.g.. A² = 5) to assess stability. The numerator captures the minimum per- formance in this range, excluding k*, effectively measuring the worst-case volatility.
By introducing a structured and reproducible evaluation framework, our study provides foundational insights into the dynamics of RAG systems and guides future research toward optimizing retreeeal-augmented generation for real-
world applications.
A higher RSS ( 1.0) indicates that performance remains stcady across nearby retrieval depths, suggesting robustness to retrieval variations. A lower RSS (RSS < 1.0) signals sharp fluctuations near k*, implying sensitivity to retrieval depth choices and reduced stability.
2. The RAGGED Framework
Evaluating surreval-auugenneed generation (RAG) remains challenging due inconsistencies across retrieval depths,
2

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
While RSS currently uses a symmetric window evaluate performance variation around the optimal retrieval depth, we acknowledge that retrieval effects can be directionally asymmetric - adding irrelevant context (right side) may de- grade performance differently than omitting high-quality content (left side). However, our empirical analysis shows that this asymmetry is not consistent across models (e.g., LLAMA2 vs. LLAMA3), thus supporting the use of sym- metric window general-purpost diagnostic. That said, directional extensions of RSS could offer deeper insight into model fragility under under- or over-retrieval, which we leave to future work.
behavior. They help developers understand not only how well system performs, but also how reliably and efficiently it does across changing retrieval conditions.
Hyperparameter Justification and Metric Stability We set for RSC to the 70th percentile of all positive AF values across models and retrieval depths, and = 5 for RSS. This percentile rule balances sensitivity to meaningful gains with robustness against small fluctuations, ensuring discriminative power across models. We verified that model rankings remain unchanged when varying between the 65th and 75th percentile and å from # to 10, indicating that both RSS and RSC are robust to reasonable parameter shifts and provide consistent comparative signals across models.
RAG Scalability Coefficient (RSC) The RSC metric cap- tures the total accumulated benefit a model gains as retrieval depth increases before performance plateaus or declines. A model with high scalability should continue to improve with additional retrieved contexts, while a low-scalability model will either plateau early or exhibit only minimal improve- ment. The RSC is defined as:
3. Experimental Setup
We implement the RAGGED framework by evaluating re- trievers and readers across multiple retrieval depths, ana- lyzing how models respond to increasing context sizes and retrieval noise.
dis gai
RSC Σ
(ki+1 (F15 + Fis41)
3.1. Retrievers
We evaluate three retrievers with different retrieval paradigms: (1) BM25 (Robertson et al., 2009), sparse
where kiuss guin is the last retrieval depth before performance plateaus or declines. This is determined as the last k where:
lexical retriever based on term matching. (2) ColBERT
(Santhanam al. (02T), ). a neural retriever using contextual- ized late interaction. (3) Contriever (Izacard et al. 2022), an
FIR - FIR-1 2
for a predefined threshold e.
unsupervised dense retriever emphasizing document-level semantic similarity.
A higher RSC reflects sustained improvements over a broader range of retrieval depths, demonstrating strong seal- ability. Conversely, a lower RSC suggests the model stops improving early, indicating limited scalability.
3.2. Readers
We analyze both closed-source and open-source reader mod-
cls: Open-source: We use FLANT5-XXL (Chung et al.)
By providing structured evaluation across diverse retriever- reader configurations, RAGGED enables systematic com- parisons different RAG systems and offers insights into optimizing retrieval depth for robust performance.
2022 (11B parameters) and UL2 (Ta et al. 2023)
(20B). ), as well as LLAMA2 (Tourron et al. 2023 (7B, 70B) and LLAMA3 (8B, 70B).
Closed-source: We evaluate GPT-3.5-turoo (16k context length) and CLAUDE-3-HAIKU (200k). A subset of experi- ments includes GPT-40 with a 128k context window. We include both open- and closed-source models to ensure that our findings generalize across different architectures and training paradigms.
In particular, the metrics RSS and RSC serve comple- mentary diagnostics rather than substitutes for task-specific metrics like F;: RSS quantifies how brittle robust model is to changes in retrieval depth near its peak performance, while RSC assesses how much a model continues to henefit from additional retrieval. A model may achieve high per- formance at single retrieval depth but still have low RSS (unstable across depths) or low RSC (unable to scale with more information). Conversely, a model with slightly lower peak performance but high RSS/RSC may be more robust and easier to deploy in real-world settings where retrieval conditions fluctuate.
3.3. Datasets
We evaluate RAG performance across three datasets spanning different reasoning complexities and domain- specificity (Table 2 Table 3):
Together with task specific performance scores, these diag- nostic metries provide more complete picture of model
Natural Questions (NQ) (kwiakkoskii et al. 2019): Wikipedia- single-hop QA with real user queries.
3

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Wikipedia-based, multi- hop QA requiring reasoning over multiple passages.
HotpotQA Yang al. 2018):
sistently achieve gain with RAG, suggesting that they can ef- fectively extract useful information from retrieved passages while discarding irrelevant details. However, just because a reader consistently gains, does not mean the gain amount is significant. For example, across datasets, FI.ANT5 achieves an average gain of 16-30 Fi points whereas GPT-3.5 achieves an average gain of to 9 Fi points in compari- son closed-book generation.
BioASQ (Task 11B) (Krithara et al. 2023 J PubMed-
based biomedical QA for specialized domains.
These datasets allow us to assess RAG performance across general knowledge (NQ), complex reasoning (HotpotQA), and domain-specific retrieval (BioASQ).
In contrast, some models degrade with retrieval, some- times performing worse than their no-context baseline. Mod- cls like LLAMA and CLAUDE struggle with filtering noisy retrievals, which results in lower accuracy when using RAG. Instead of leveraging additional knowledge, these models become more susceptible to incorrect or distracting pas- sages.
3.4. Metriis
We evaluate both retrieval and reader performance following best practices from Petroni et al. 2021)
Retriever Performance: We report recall@k, which mea- sures the fraction of ground-truth passages present in the top- k retrieved results. Higher recall indicates better retrieval coverage but does not guarantee better reader performance.
This suggests that retrieval is not inherently beneficial, but instead depends on how well a reader can balance the trade- off between extracting useful knowledge and avoiding re- trieval noise.
Reader Performance: We compute unigram Fj, which measures lexical overlap between model predictions and gold answers. Each query is evaluated against all gold answers, and the highest score is reported. To further assess correctness, we validate key results using an LLM-based semantic correctness metric (Kim et al., 2024) subset of responses (Appendix J).
While retrieval can provide additional signal (relevant knowledge), it also introduces noise (irrelevant passages), The models that benefit most from retrieval tend to be those that can effectively distinguish between high-value and low- value context, whereas poise-senssiive models treat all re- trieved passages equally, leading to instability. We discuss some hypothesis (Appendix Elreader architecture and train- ing details to trends.
To analyze retrieval-depth stability and scalability, we eval- uate the RAG Stability Score (RSS) and RAG Scalability Coefficient (RSC) as defined in section 2] These metrics provide a principled way to assess RAG systems beyond traditional retrieval and reader accuracy measures.
4.2. Task-Specific Retrieval Trends
We observe that retrieval effectiveness is not uniform across tasks and domains.
4. Under What Conditions Does Retrieval Outperform Closed-Book Generation?
Multi-hop questions benefit more from retrieval than single-hop questions. Since multi-hop reasoning requires synthesizing multiple pieces of information, and can not be tackled simply by retrieving a short fact learned from pretraining. Thus, retrieval can be particularly helpful for multi-hop settings.
eetrieval-augmenter generation (RAG) is widely assumed to enhance model performance by providing external knowl- edge, but our findings reveal that its effectiveness is highly model-dependent. While some models benefit from re- trieved context, others perform worse than when no retrieval is used at all. This section investigates when retrieval ac- tually helps, which models are most affected by retrieval noise, and how retrieval effectiveness varies across tasks and domains.
Key Takeaways
Our results show that retrieval is not inherently helpful. Its effectiveness depends the model's ability to handle noisy information. While some models consistently benefit from retrieval, others degrade due to over-reliance on irrelevant or misleading content. This highlights the need for retrieval- aware reading mechanisms that allow models to selectively integrate useful passages rather than treating all retrieved content equally.
4.1. When Does Retrieval Help?
Retrieval effectiveness is primarily determined by the model's ability selectively use relevant information while ignoring misleading redundant content. We observe two distinct behaviors:
First, some models benefit consistently from retrieval, showing significant performance improvements when re- trieval enabled. Models such as FLAN and GPT-3.5 con-

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
ColBERT BM25
ColBERT
BM25
Dataset
Dataset
GPT-3.5-turbo
Claude-3-haiku
NQ
only for 5
x
NQ HotporQA BioASQ
X
Ho
/
only for à: <
/
BioASQ
/
X
FlanTS
FlanUL2
NQ Ho
NQ HotporQA
only for > 3
BioASQ
/
BioASQ
Llama2 7B
Llama2 70B
NQ
only for < 10
NQ
only for is < 20
HotpotQA
HotpotQA
BioASQ
BioASQ
only for hs <, 20
only for l < 20
Llama3 8B
Llama3 70B
NQ HotpotQA
only for = 2
x
NQ
X
only for lo S 5
only for le < 5
HotpotQA BioASQ
only for l: < 5 only for k:
only for < 5
BioASQ
only for &s S 2
only for A: < 2
X
Table 1: / means the particular reader-retriever combination performs better than closed-book generation for all k's. On the other hand, X signifies that the particular reader-retriever combination consistently performs worse than closed-book generation, regardless of k. Otherwise, we describe the k-condition for which the retriever-reader combination performs better than closed-bood generation.
5. How Does Retrieval Depth Impact Stability and Scalability?
minimize performance drops.
Prior work reports conflicting effects of increasing retrieval depth (k)): some studies find that performance saturates at high & (Liu et al. 2023), while others observe degradation (Cuconasu et al.] 2024 Jiang et al.) 2024). Although these findings appear contradictory, we argue that they are actually complementary, as each study focuses on a limited range of retrievers, readers, and datasets. Our experiments, which span wider variety of retrievers, readers, and datasets, demonstrate that both saturation and degradation behaviors can occur with the determining factor being the choice of reader model.
60
Claude Haiku
FlanTS
40
Reader F1
20
LLaMa2 708
NQ
LLaMa3
LLaMa3 708
(# of context passages)
Figure 3: Reader performance on the NQ dataset as k, the number of contexts retrieved by CoIBERT, varies. Col- ored circles indicate reader performance at the optimal k*, Similar trends hold across retrievers (BM25, ColBERT, Con- triever) and datasets (NQ, HotpotQA, BioASQ) in Figure 14 and Figure 15
Specifically, we observe two distinct trends in reader perfor- mance (Figure 3):
Improve-then-Platean Models Models such FL.AN and GPT-3.5 improve & increases and plateau around k = 10. For these models, increasing i maximizes per- formance without significant risk of degradation.
Why This Matters A model's response increasing affects not only its peak performance but also its stability and scalability.
Beek-then-Dellnee Models In contrast, models like LLAMA and CLAUDE-33 3-HDD peak at small (around 5) but degrade as & increases due to their sensitivity retrieval noise. For these models, small i is optimal to
A well-designed RAG system should be scalable, meaning it should benefit from increasing Multi-hop reasoning tasks, in particular, require synthesizing multiple pieces of
5

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
information, making large k essential. Peak-then-Decline models struggle in such cases.
decline models, they decline steadily. This reminds that we really should aspire to optimize for both metries, not just one.
We should also strive for stable model that maintains consis- tent performance near the optimal k", so that retrieval depth tuning is practical. weak-then--ecling models can exhibit a sharp performance drops even when A is only off from k*, making them harder tune and unreliable in practice.
To assess how well our findings generalize beyond tradi- tional QA datasets, we conduct a preliminary evaluation on CRAG, a newer and more challenging RAG benchmark (Yang et al. (024). We observe that the same reader-specifhe retrieval-depth trends hold: LLAMA2 exhibits early degra- dation, while FLANTS remains stable across increasing k. Full results are provided in Appendix F
To quantify these trends, we compute the RAG Scalability Coefficient (RSC) Figure 4 and the RAG Stability Score (RSS) Figure 5) We note that improve-then-plateau mod- els have high RSC and RSS, which aligns with the intu- ition. Models like FLANTS and GPT-3.5 exhibit high RSS scores, indicating strong performance stability across re- trieval depths. In partieular, FLANTS achieves an RSS of 0.99, reflecting near-constant performance around its opti- mal k. We confirmed that this is not due to input truncation masking additional retrieved content, where we provide supporting analysis in Appendix G
Key Takeaways More retrieval is not always bet- ter-some models improve with increasing k, while others degrade due to noise sensitivity. More importantly, the kind of improvement future research should strive for is better scalability and stability. RAGGED provides principled way to assess these two critical aspects, helping practitioners determine optimal retrieval depth per model.
6. How Do Readers Handle Noisy Retrieval, and Is Prompting a Reliable Fix?
RAG Scalability Confficient (RSC)
RSC Scores Across Different Models
200-
225.7
236.2
2004
Real-world RAG systems retrieve a mix of relevant and irrelevant content, making reader robustness to noise key factor in performance. This section evaluates reader behav- ior when (1) at least one gold passage is present and (2) no gold passage retrieved. The former setting represents good scenario when there is sufficient signal to answer the question, the latter setting represents the worst-case sce- nario where there is not enough information to answer the question.
:30
160-
50.
Heet
Lamas far
Lutto 40
Throughout this section, we define noise as naturally re- trieved, non-gold passages from actual retrievers. These are not artificially injected distractors but instead reflect real-world retrieval failures, such topically related but misleading or irrelevant content.
Figure 4: Ragged scalability coefficient for NQ, retriever colbert.
RAG Scalability Coefficlent (RSS)
RSS Scores Across Different Models
1.0
0.99
0.97
0.99
0.95
0.8
0.86
6.1. With Gold Passages
0.6
0.70
0.64
We compare three conditions: (1) Top-k: full retrieved set, (2) Top-gold: only gold passages within the top-k, and (3) No-context: no retrieval. This evaluates how distracting noise compared to the signal (Figure 6).
0.2
Reader Robustness Determines Gains from Retrieval While robust models (e.g., FLAN, GPT) consistently im- prove with retrieval, noise-sensitive models (e.g.. LLAMA, CLAUDe)) degrade below their no-context baselines. This suggests that some models are not good at performing the second-step filtering. leaving them vulnerable to noise when irrelevant passages are included. Future work should ex- plore fine-tuning readers on diverse, noisy retrieval settings to improve robustness.
0.0
PlanTS
FlanUL2
GPT-3.5
LiaMaz 708
UaMa277 78
Claude Haiku
UaMM3 708
LaaMa3 98
Figure 5: Ragged stability score for NQ, retriever colbert.
What is more interesting is that while a scalable model is often more stable, a stable model does not have to be a scalable one. For example, LLAMA2 models are not as scalable but are still stable. Although they are peak then
Multi-hop Questions Mitigate Noise Effects
In Hot-
6

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
top-gold
top-k
no-ctx
below their no-context baseline (Figure 7p.
This is
80
80
prising since these models instructed to the context at hand, which has insufficient information.
60
GPT-3.5
Reader Performance (F1)
40
Claude
What is more interesting is that FLAN models outperform no-context baselines even without gold passages. They seem better than other readers at processing partial clues from these non-gold passages. For example, on NQ with the FLAN models achieve 20% accuracy when no gold paragraphs retrieved but paragraphs from the gold Wikipedia pages are present. Although such paragraphs are not sufficient themselves, they are highly related to the right information, thus providing some contextual clues.
Halku
FlanTS
30
FlanUL2
LLaMa2
80
LLaMa2 70B
60
40
40
LLaMa3
20
LLaMa3 708
30
k (# of retrieved passages)
6.3. Can Prompting Improve Noise Filtering?
Figure 6: NQ results when at least one gold passage in the top-k. 'Top-gold' includes only gold passages.
We test whether explicit relevance instructions improve noise filtering (Figure 8). We pick one noise-robust model (FLANT5) and one noise-sensitive model (LLAMA2 7B).
potQA, models maintain accuracy above no-context longer than in NQ. We hypothesize that multi-hop signals force the model rely on more signal anchors, reducing reliance on single-passage heuristies and making models more resilient to noise.
FlanT5
LLaMa2 7B
rerank + relevant
base
rerank
relevant
60
Domain-Specific Jargon Strengthens Retrieval In BioASQ, the gap between top-gold and top-k is smaller than in open-domain datasets, indicating that comain--peciffe ter- minology provides stronger retrieval cues. However, noise- sensitive models (e.g., CLAUDE-3 3 --HAIKU and LLAMA3) still fall below no-context performance, suggesting that trieval alone is insufficient fine-tuning domain-specific noisy retrievals may be necessary.
50
Reader
20
0
10
20
30
k (# of context passages)
6.2. Without Gold Passages
Figure 8: Effects of applying reranking and instructing the model to focus on the relevant passages ("relevant"). These results are for when the retriever is ColBERT and the dataset is NQ. Results for HotpotQA and BioASQ are at Appendix
top-k
no-ctx
GPT-3.5
Reader Performance (F1)
20
20
Claude
Prompting Has Mixed Effects and Does Not Improve Stability Across models, reranking consistently im- proves retrieval, while prompting has inconsistent effects. Prompting helps the noise-sensitive model but has no im- pact on the noise-robust model, which may already have the pretrained ability to pay attention to relevant passages. In- terestingly, prompting does not improve performance when reranking is already applied, reinforeing that prompting cannot compensate for poor retrieval, and when retrieval strong, prompting is redundant.
Haiku
FlanTS
0
10
20
30
0
10
20
30
FlanUL2
LLaMa2 7B
40
40
LLaMa2 70B
LLaMa3
20
20
LLaMa3 70B
0
30
20
30
(# of retrieved passages)
Prompting Can Harm Performance in Specialized Do- mains In BioASQ, prompting degrades performance likely because the reader not pretrained on enough domain-specifie knowledge to have and context and rea- son what relevant not.
Figure 7: NQ results when no gold passages are retrieved.
When no gold passages are retrieved, most models degrade

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Key Takeaways Future RAG research should focus on fine-tuning readers for noise resilience rather than relying on retrieval-side interventions. While prompting can help, it is not a universal fix. RAGGED provides structured way to assess noise robustness, guiding both retrieval adaptation and reader optimization.
One possible explanation is that domain-specific terminol- ogy provides stronger retrieval cues, allowing retrievers to separate relevant from irrelevant content more effectively, This reduces the need for reader-level noise filtering, making retrieval improvements directly beneficial.
7.2. Does Reranking Improve Retriever-Reader Alignment?
7. When Does a Better Retriever Actually Improve RAG Performance?
Reranking Helps More in Open-Domain QA Than in Specialized Domains Reranking improves performance in open-domain datasets (NQ. HotpotQA), particularly for noise-sensitive models like LLAMA [Appendxx O How- ever, in BioASQ, reranking fails to provide consistent gains and in some cases degrades performance. In open-domain QA, retrieval errors often involve partially relevant passages, meaning reranking can improve performance by clevating the most useful documents. However, in Nomain--spccife tasks like BioASQ, where documents contain dense techni- cal content, reranking may prioritize semantically similar but not specific enough passages, leading to worse perfor- mance.
The reader robustness trends from Section persist across retrievers and even with reranking. That is not to say re- triever choice has no impact. Retriever choice still affects retrieval efficiency, computational cost, and domain-specific performance. Below, we compare ColBERT (a neural re- triever) and BM25 (a lexical retriever), analyze their impact on different reader types, and assess reranking as a retrieval- side intervention.
7.1. When Does a Stronger Retriever Improve RAG Performance?
We evaluate retriever effects using two metrics in Table 8) (1) Average Difference: Mean F, difference between Col-
Reranking Outperforms Prompting, But Gains Do Not Stack Across datasets, reranking consistently outper- forms prompting as a noise-filtering strategy. However, applying both does not yield additional gains. Once re- trieval quality improved via reranking, prompting has little residual effect. Since reranking, in some sense, fil- ters input before it reaches the model, prompting has no additional noise to filter out.
BERT and BM25 across 1 to 50. (2) Optimal F1 Difference:
The peak performance difference between Col- BERT and BM25 at each reader's optimal-A: These met- rics assess whether a stronger retriever consistently benefits downstream readers (Table 8).
Retriever Quality Improves Recall, But Not Always Reader Performance While COIBERT consistently achieves higher recall @k than BM25, its downstream gains vary by reader type. For peak thee-then- models (LLAMA, -LAUDE-3-HAIKU ColBERT performs worse than BM25 at large k. despite better retrieval quality. This suggests that some readers are highly sensitive to the nature of the retrieval noise. One possible explanation is that Col- BERT retrieves more semantically similar passages, which can be more distracting and misleading than a less relevant passage retrieved from BM25.
Key Takeaways Stronger retrievers do not always lead to better RAG performance, and reader robustness to noise remains the key bottleneck. While dense retrievers improve recall, their benefits depend on how well the reader inte- grates retrieved information. Reranking, although beneficial, depends on domain-speciff retrieval quality and does not fundamentally change a reader's 's stability and scalability.
8. Related Work
Retriever Improvements Have Modest Gains in Open- Domain QA Although ColBERT improves recall sig- nificantly in NQ (+21.3 recall k) and HotpotQA (+14.6 recall@k) k), the corresponding reader gains are much smaller (+5.2 and +1.9 F,, respectively). The low ratio of reader gain to retrieval gain (0.13 in HotporQA) suggests that better retrieval alone does not guarantee proportionately substan- tial reader improvement, especially in open-domain settings.
Retrieval Depth and Performance Prior work offers mixed conclusions on increasing retrieval depth (k). Some studies report consistent improvements (Izacard & Grave 2021), while others find diminishing returns (Liu et al.
2023 or even performance degradation at high k (Cuconasu
et al.
2024 Jiang et al. 2024).
Rather than contradictory, we find these trends depend on the reader's robustness to noise. Our work systematically evaluates retrieval depth effects across diverse readers, dis- tinguishing improve-thert wwwtttheetteeeee plateau haviors as key factors in retrieval effectiveness.
Specialized Domains Benefit More from Stronger Re- trieval In contrast, in specialized domains (BioASQ), even small retrieval improvements (+0.7 recall @k) yield substantial reader gains (+2.08 F.).
men-thhen-declie be-
8

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Domain-Speeific RAG Effectiveness RAG's impact varies by domain, particularly for long-tail knowledge. Some studies suggest retrieval is beneficial (Kandpal et al.] 2023). while others find it unnecessary or even harmful for common knowledge (Mallen et al. 2023).
formalizing how evaluate reader robustness and retrieval utility, RAGGED lays foundation for building more reli- able, adaptive, and efficient retrieval-augmenned generation systems.
We show that domain effects are not inherently stronger or weaker but depend on the retriever-reader interaction, highlighting the need optimize retrieval strategies per task rather than assuming domain specificity guarantees improvements.
Impact Statement
Our work contributes to improving the evaluation and op- timization of RAG systems, which are increasingly used in chowledge-intensiie tasks such question answering, fact-checking, and scientific information retrieval. By intro- ducing a systematic framework for assessing RAG stability and scalability, our study provides actionable insights for building more reliable and robust Al systems.
Retriever Choice and Reader Performance Dense re-
trievers often improve retrieval quality (Lewis et al., 2020). but their downstream impact is not always positive. Fi nardi et al. (2024) report a correlation between retriever and reader performance in specialized settings, yet our results reveal that stronger retrievers do not always yield better RAG outputs, especially for noise-sensitive readers.
Ethical Considerations: tual accuracy by incorporating external knowledge, they also introduce risks such information distortion when retrieval is noisy or biased. Our findings highlight the im- portance of reader robustness to retrieval noise, suggesting that deployments of RAG models should include safeguards against misleading or incorrect retrieved content.
While RAG systems enhance fac-
While retriever quality enhances recall, reader robustness dictates final performance, with ppecializedddomat tasks benefiting disproportionately from even minor retrieval im- provements. This underscores the need for domain-aware retrieval strategies rather than assuming higher retrieval ac- curacy guarantees better generation.
Future Societal Impact: As RAG-based models become integral to decision-making in fields like healthcare, law, and education, ensuring their stability and reliability is crucial. The RAGGED framework provides a principled way to measure and improve retrieval robustness via RSS and RSC, which could help mitigate misinformation risks in high- stakes applications.
9. Conclusion
Retrieval-augmented generation (RAG) systems are widely used to enhance language models, but their performance hinges not just on retrieval quality, but on the reader's ability to handle noise and uncertainty. Our study demonstrates that retrieval depth must be dynamically tuned for each model, and that reader robustness, not retriever strength, is the key driver of scalable and stable RAG performance.
While our study focuses on evaluation, its insights can in- form the development of more scalable and stable RAG systems.
A Acknowledgements
Special thanks to Alex Cabrera, Alex Bäuerle, Jun Araki, Md Rizwan Parvez for providing Zeno support for analysis visualization. Our appreciation extends to Hao Zhu, Jacob Springer, and Vijay Viswanathan for providing feedback for our paper. This paper was supported in part by gift from Bosch research.
To support this insight, we introduce RAGGED, a modu- lar evaluation framework that systematically analyzes re- trieval depth, noise sensitivity, and reader-retriever dynam- ics. Through two new metrics- RAG Stability Score (RSS) and RAG Scalability Coefficient (RSC) RAGGED of- fers principled way to assess how reliably and efficiently models use retrieved information across configurations and domains.
References
Chen, D., Fisch, A., Weston, J., and Bordes, A. Reading Wikipedia to answer open-domain questions. In Barzi- lay, R. and Kan, M.-Y. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Compu- tational Linguistics. doi: 10. 10.18655/11/P17-1171 URL https//accamtholog..rr///1
These findings challenge the assumption that retrieval qual- ity alone governs RAG success, and highlight the impor- tance of tuning retrieval strategies around reader behavior. As models continue evolve, RAGGED remains appli- cable as model-agnostic harness for measuring retrieval sensitivity and guiding deployment decisions.
Looking ahead, expanding RAGGED to adversarial, out- dated, or temporally shifting noise scenarios will further enhance its relevance to high-stakes, real- world settings. By
Chung, H. W., Hou, L.., Longpre, S., Zoph, B., Tay, Y.,

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang. S., Mishra, G., Yu, A., Zhao, V., Huang, Y. Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, Scaling cnttrucctionfinettune language models. arXiv preprint arXiv:2210.1144 2022.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H.,, Lewis, M., Yih, W.-., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459-9444 94444 2020.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F,, and Liang, P. Lost in the middle: How language models use long contexts, 2023.
Cuconasu, F., Trappolini, G. Siciliano, F., Filice, S., Cam- pagnano, C., Maarek, Y., Tonellotto, N., and Silvestri, F. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 719-729, 2024.
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Inves- tigating effectiveness of parametric and non non-parametric memories, 2023.
of Medicine, N. L. Pubmed baseline 2023 repository, 2023. URL https://hnccb.nlm.n http:///w.......goo/////oomaiioo goviii/nfoorrmaiin///// MBR.html
Finardi, P., Avila, L., Castaldoni, R., Gengo, P., Larcher, C., Piau, M., Costa, P., and Caridá, V. The chronicles of rag: The retriever, the chunk and the generator. arXiv preprint vrXiv:2411.07888 2024.
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., Cao, N. D., Thome, J., Jernite, Y., Karpukhin, V., Maillard, J., Plachouras, V., Rocktüschel, T., and Riedel, S. Kilt: a benchmark for knowledge intensive language tasks, 2021.
Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering, 2021.
Robertson, S., Zaragoza, H., et al. The probabilistic rele- vance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389, 2009.
Izacard, G., Caron, M., Hosseini, L., Riedel, S.,, Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense infor- mation retrieval with contrastive learning, 2022. URL
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., and Zaharia, M. Colbertv2: Effective and efficient re- trieval via lightweight late interaction. arXiv preprint arXiv: 2 12.01488, 2021.
Jiang, Z., Ma, X., and Chen, W. Longrag: Enhancing aetrieval-augment generation with long-context Ilms. arXiv preprint ar arXiv:2406. 15319, 2024.
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., Zheng, H. S., Zhou, D., Houlsby, N., and Metzler,
Kandpal, N., Deng, H., Roberts, A., Wallace, E., and Raffel,
C. Large language models struggle to learn long-tail knowledge, 2023.
D. U12: Unifying language learning paradigms. arXiv preprint arXiv:2205.0513/ 2023.
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. Prometheus 2: An open source language model special- ized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- plc, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. / 2023.
Krithara, A., Nentidis, A., Bougiatiotis, K., and Paliouras, G. -Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10:170, 2023. URL http://bbooorr///0.0000666111119
Yang, X., Sun, K., Xin, H., Sun, Y., Bhalla, N., Chen, X., Choudhary. S., Gui, R. D., Jiang, Z. W., Jiang, Z., Kong, L., Moran, B., Wang, J., Xu, Y. E., Yan, A., Yang, C., Yuan, E., Zha, H., Tang, N., Chen, L.. Scheffer, N., Liu, Y., Shah, N., Wanga, R., Kumar, A., tau Yih, W., and Dong, X. L. Crag comprehensive rag benchmark. arXiv preprint arXiv:2406004744 2024. URL https://arxiv.org Abs/2406.04744
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., De- vlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for question answering research. Transactions of the As- sociation for Computational Linguistics, 2019. URL hutps:/accaattology..o//111111
Yang. Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W. Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering, 2018
10

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
A. Reader Implementation Details
Corpus
# of par
of doc
Avg of doc
We truncate the context to make sure the the rest of the prompt still fits within a reader's context limit. Specifi- cally, when using FLANT5 and FLANUL2 readers, we use T5Tokenizer to truncate sequences to up to 2k tokens; when using LLAMA models, we apply the Tokenize and truncate sequences by 4k tokens for LLAMA2 and 8k for LLAMA3. For closed-source models, we spent around S300. Subsequently, we incorporate a concise question-and -ndiiooonnn------ answer format that segments the query using "Question:" and cues the model's response with "Answer:", ensuring precise and targeted answers.
Wikipedia Medline
111M 58M
SM 34M
18.9 1.7
Table Retrieval corpus information
Dataset
# of Queries
NQ
2837
HotpotQA BioASQ 3837
5600
Table 3: Dataset information
For our reader decoding strategy. we used greedy decoding with a beam size of I and temperature of 1, selecting the most probable next word at each step without sampling. The output generation was configured to produce responses with 10 tokens. The experiments were conducted on NVIDIA A6000 GPUs, supported by an environment with 60GB RAM. The average response time was ~1.1s per query when processing with a batch size of 50.
D. Comparison with No-Context Performance
We include additional reader results comparing ColBERT and BM25 Table 4 and Table 5
E. Relating Reader Trends to Reader Architectures and Training Details
There are two primary types of readers observed in our experiments:
B. Prompt
For all experiments, we use the following prompt:
Peak-then Dhen-De Behavior: Models including those from the LLAMA and CLAUDE families show sen- sitivity to noisy documents, Icading to performance degradation as the number of retrieved passages (k) increases beyond a certain point.
Instruction: Give simple short one phrase answers for the questions based on the context
Context: [passage1, passage2. .... passage Question:
[the question of the current example]
Answer:
Improve-then- Plettttaan Behavior: Models including those from the GPT and FLAN families are more ro- bust to noise, continuing to benefit from additional context until performance plateaus.
For the "relevant" prompt, we swap the instruction for "Give simple short one phrase answers for the questions based only the parts of the context that are relevant to the question."
Since we do not have access to the details of the closed- source models, we will focus on providing hypotheses ac- cording to the open-source model (LLAMA belonging to the peak-then faak-then-dedd decllie behavior and the FLAN models belonging to the improve-then-plateed family).
C. Dataset Details
All corpus and datasets use English.
For NQ and HotpotQA datasets in the open domain, we use the Wikipedia paragraphs corpus provided by the KILT benchmark (Petroni et al.] 202T). For BioASQ, we use the PubMed Annual Baseline Repository for 2023 (of Medicine 2023), where each passage is either a title or an abstract of PubMed papers. Dataset sizes are in Table 3
On one hand, FLAN, tmprove-then-platee model fam- ily, incorporates additional strategies explicitly designed to handle noisy or diverse contexts. It employs denoising strategies, such as a mixture-off-denossee during training to improve its robustness to irrelevant or noisy contexts. These enhancements enable it to filter out noise more effectively.
The Medline Corpus is from of Medicine (023 provided
by the National Library of Medicine.
On the other hand, LLAMA 's training predominantly relies next-token prediction with limited exposure to noisy or setrifval-specific scenarios, making it sensitive to noise at higher k.
For NQ and HotpotQA, we use KILT's dev set versions of the datasets, allowed under the MIT License (Petroni et al.) 2021). For BioASQ (Krithara et al. 2023), we use Task 1IB, distributed under CC 2.5 license
We also note that there are some model architecture features that alone do not determine reader behavior:

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Model
NQ
HAtpotQA
BioASQ
ColBERT
BM225
ColBERT
BM25
ColBERT
BM25
GPT-3.5
1.1
-8.0
7.7
5.1
8.9
7.4
CLAUDE Haiku
-15.7
-22.5
-6.5
-10,2
-21.7
-25,9
FlanT55
28.9
12.6
20.9
13.5
16.6
11.9
FlanUL2
21.5
4.9
18.9
15.7
5.3
2.8
LLAMA 2 7B
1.4
-4.5
10.4
8.2
6.4
5.9
LLAMA 2 70B
-0.1
-7.6
11.2
9.2
4.4
3.9
LLAMA 3 8B
-13.3
-14.9
-6.7
-5.7
-12.1
-14.9
LLAMA 3 70B
-24.9
-26.2
-12.0
-11.3
-18.0
-21.4
Average (per dataset)
-0.1
-8.3
5.5
3.06
-1.3
-3.8
Table 4: The average difference between the F1 score of RAG with & passages from CoIBERT or BM25 and the F1 score of no-context generation, calculated across k values from 1 to 50 for each dataset. Each value represents the difference between the F1 score of the reader+retriever combination and the F1 score of the reader alone (without RAG or context).
Model
NQ
HotpotQA
BioASQ
ColBERT
BM25
ColBERT
BM25
ColBERT
BM25
GPT-3.5
3.8
-3.1
8.8
7.3
10.9
10.6
CLAUDE Haiku
-2.4
-14.9
3.9
0.7
-1.7
-8.2
FlanT5
32.5
22.6
23.5
20.4
18.0
13.0
FlanUL2
24.4
14.8
22.0
19.7
7.1
3.4
LLAMA 2 7B
9.7
-0.5
15.2
11.0
8.5
6.9
LLAMA 2 70B
4.3
-0.3
14.0
11.4
7.3
6.8
LLAMA 3 8B
3,9
-3.0
11.2
8.4
3.6
1.7
LLAMA 3 70B
-0.7
-9.6
14.9
8.1
4.4
0.3
Average (per dataset)
9.44
0.8
14.2
10.9
7.3
4.3
Table 5: The difference between the F1 score of RAG optimal k- from ColBERT or BM25 and the F1 score of no-context generation. Each value represents the difference between the F₃ score of the reader+retriever combination at optimal k* and the F1 score of the reader alone (without RAG or context).
Context window size: Models with longer context lim- its like LLAMA (4k tokens) don't necessarily process larger number of contexts better than models with smaller context limits like FLAN (2k tokens).
peak followed by flattening. Although performance lower overall expected given CRAG's difficulty), these results suggest that RAGGED's retrieval-depth insights can gener- alize to this newer benchmark.
Table 6: F1 scores of FLANT5 and LLAMA2 on CRAG at varying retrieval depths (k).
Encoder-decoder v. decoder: LLAMA decoder- only model that displays fokk-then-decling behavior, but GPT models also decoder-only and instead display improve-then plateau behavior.
Model
k=1
=
le =1=00
=20
k =30
FLANTS
0.19
0.17
0.18
0.18
0.18
LLAMA2
0.20
0.23
0.23
0.23
0.23
F. Preliminary Results on CRAG
To evaluate the generalization of our findings to more recent RAG benchmarks, we conducted a preliminary study using the CRAG dataset (Yang et al. 2024). We selected one repre- sentative model from each of the two major reader behavior classes: FLANT5 (improve-then-platean and LLAMA2 (peak-then-decline).
G. Investigating FLANT5 's High RSS and Input Truncation
In Figure 5, FLANTS achieves RSS of 0.99 on the NQ dataset. One possibility is that truncation effects at higher retrieval depths (e.g., k 25) may mask additional context, artificially inflating stability.
As shown Table 6, the core reader trends persist: LLAMA2 exhibits early performance saturation and plateaus, while FLANT5 demonstrates mild performance
assess this, we compared tokenized input lengths be-
12

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Tween k 20 and A 25. In 32% of cases, the k 25 input included more tokens than k 20, indicating that ad- ditional retrieved passages were indeed processed. Despite this, the model's F1 score changes by < 0.5 on average, sup- porting our interpretation that FL.ANT5's high RSS reflects genuine retrieval-depth robustness, not an artifact of context truncation.
We include
results for HotpotQA
Higure 11 and for BioASQ at Higure 12
top-k
no-ctx
40
GPT-3.5
Reader Performance (F1)
Haiku
H. Slice Analysis on Other Datasets
FlanT5
We include seeth-gold-passaagee results for HotpotQA at Fig- ure and for BioASQ at Figure 10
FlanUL2
LLaMa2 7B
LLaMa2 708
top-gold
top-k
no-ctx
LLaMa3 88
20
80
80
LLaMa3 708
60
50
GPT-3.5
Reader Performance (F1)
0
40
40
Claude
10
20
30
0
10
20
30
20
20
k (# of retrieved passages)
FlanTS
0
10
30
10
20
30
FlanUL2
Figure 11: HotpotQA results when there are no gold pas- sages included in the top-k passages answer the question.
LLaMa2 7B
80
LLaMa2
60
40
LLaMa3 88
top-k
no-ctx
LLaMa3 708
40
GPT-3.5
(# of retrieved passages)
Reader Performance (F1)
Claude Haiku
20
20
Figure 9: HotpotQA results when there is sufficient infor- mation (all gold passages) included the top-k passages to answer the question. For multi-hop questions, we select examples retrieved with all gold passages within the top- k passages since all passages necessary answer the question.
FlanT5
FlanUL2
20
30
10
20
LLaMa2 7B
LLaMa2 70B
40
LLaMa3
70B
top-gold
top-k
no-ctx
20
30
10
20
30
(# of retrieved passages)
GPT-3.5
Reader Performance (F1)
40
Claude
Figure 12: BioASQ results when there gold passages
Haiku
included in the top-k passages to answer the question.
FlanTS
10
20
30
FlanUL2
80
80
LLaMa2
I. Comparing Optimal k Values
LLaMa2 70B
We include the optimal for ColBERT and BM25
LLaMa3
ble
LLaMa3
10
30
20
30
J. LLM-Based Evaluation
k (# of retrieved passages)
While we chose F1 for its simplicity and alignment with prior work, we agree that it may not fully reflect nuanced semantic equivalence. To address this, we ran an LLM- based evaluation of the models for the NQ dataset using
Figure 10: BioASQ results when there is sufficient infor- mation (at least one gold passage) included the top-k passages to the question.
13

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Model
NQ
HotpotQA
BioAQQ
Average (per reader)
BM25
ColBERT
BM25
ColBERT
BM25
ColBERT
BM25
ColBERT
GPT-3.5
20
20
20
CLAUDE Haiku
1
1
FlanTS
20
10
10
50
36.67
10.33
HanUUL
LLAMA 27B
1
2
1
1.67
1.33
LLAMA 70B
5
2
8:33
LLAMA SB
LLAAM 708
Average (per dalasel)
7.38
Table 7: Optimal k* for BM25 and ColBERT (NQ, HotpotQA, and BioASQ).
Prometheus (Kim et al. 2024). specifically the Prometheus- 7b-v2.0 model. We find that the conclusions about reader trends do not change: the same reader trends apply to the same models (peak-then- theeelleen- We Prometheuus-7--v2.0 evaluate the correctness of the generated answer against the gold answer on 5-point scale, where is the least correct and is the most correct Figure 13
N. Retriever Performance
We include the retriever performance select k's at Table
cupproe-tthen-platee
O. Effect of Reranker and Relevance Prompting
We test whether reranking and/or explicit relevance prompt- ing instructions improve noise filtering on NQ. HotpotQA, and BioASQ (Figure 8 Figure 17 Figure 18). We pick one noise-robust model (FLANTS) and one noise-sensitive model (I.LAMA2 7B) demonstrate preliminary results.
Prometheus Evalusted Correctn Score
Chreeiese Scage
20
CFF35
Caudettata
25
5.0
12.
12.
25.0
27.5
20.2
Figure 13: Reader Performance NQ dataset evaluated by Prometheus 5-point scale where 1 is the least correct and 5 the the most correct.
K. Comparing Reader Trends when using ColBERT v. BM25
We include the top-k performance for ColBERT, BM25 Figure and in Table 8
L. Comparing Neural Retrievers
compare top-k performance of ColBERT and Con- triever Figure 15
M. Comparing GPT-3.5 and GPT-40
We compare how GPT-3.5 and GPT-40 perform, and find that they both display the same reader trend of improve-then- plateau, with the main difference being GPT-4o's 's reader performance shifted up (Figure 16).
14

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
GPT-3.5
Claude Haiku
FlanT5
FlanUL2
LLaMa2
LLaMa2
LLaMa3 8B
LLaMa3 70B
Reader Performance (F1)
NQ
HotpotQA
BioASQ
60
60
40
40
40
20
20
20
0
20
k (#f of context passages)
(a) Performance when top-k passages from ColBERT.
GPT-3.5
Claude
FlanT5
FlanUL2
LLaMa2 LLaMa2
LLaMa3 8B
LLaMa3 70B
Haiku
Reader Performance (F1)
NQ
HotpotQA
BioASQ
60
40
40
40
20
0
15
0
10
15
20
k (# of context passages)
(b) Performance when top-k passages from BM25.
Figure 14: Top-k performance on NQ, HotpotQA, and BioASQ. Colored circles mark the reader performance at optimal k.
NQ
HotpotQA
BloASQ
0.6
0.6
2.6
colbett
liam22 7b contriooer
0.5
0.5
fan collert
15 contriever
0.3
0.3
0.3
0.2
0.2
10
30
50
20
30
40
50
10
50
k( rstrieved pmo.uggee)
1 of retrisoved passagest
1 retriavee passugges)
Figure 15: Example of how reader response increasing context applies across neural retrievers (e.g., ColBERT and Contriever) and datasets. We choose reader model from each trend for demonstration LLAMA2 7B for peak-then- decline and FLANT5 for
15

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Average Difference (across k)
Difference in Optimal Performance
Model
NQ
Ho
BioASQ
NQ
HotpotQA
BioASQ
GPT-3.5
8.6
2.0
1.1
6
Claude Haiku
2.4
12
FlanT5
12.6
10.5
4.2
9
FlanUL2
12.9
2.0
1.9
9
2
3
LLaMa2 7B
3.6
0.9
-0.3
10
4
LLaMa2 70B
2.6
0.7
-0.2
4
LLaMa3 8B
-0.7
LLaMa3 70B
-1.9
-2.7
1.5
8
Average
5.2
1.9
1.5
8
2.9
2.4
Table 8: For each reader, the average difference and optimal difference in Fj scores between ColBERT and BM25 reported. (See the main text above for detailed definitions.)
Recall@k
Retriever
2
20
NQ
2.7
4,4
8.0
11.5
16.3
22.8
BM25
10.3
16.3
27.8
36.8
47.7
53.2
12.3
18.0
25.7
32.1
38.1
41.8
ColBERT
27.2
38.8
54.4
65.0
72.9
77.2
4.65
6.91
11.14
15.17
20.19
28.46
Contriever
24.0
32.3
44.9
53.2
62.1
72.0
HorporQA
GPT
19.1
25.9
34.6
41.1
46.8
$4.2
BM25
23.3
31.2
42.7
52.1
59.1
62.8
Buader Ruformnnce (F3)
56
ColBERT
31.1
40.1
49.9
$6.2
61.9
64.9
56
34.2
44.7
56.3
63.6
69.9
73.1
2.35
4.44
8.14
11.75
15.46
20.79
Contriever
22.39
29.54
39.39
45.71
51.51
59.08
BioASQ
50-
8.8
12.9
19.6
25.8
33.3
37.8
BM25
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
12.4
16.4
23.9
30.6
38.7
43.6
ColBERT
8.8
13.5
27.1
34.3
38.6
Figure 16: Comparison of GPT-3.5 and GPT-40 perfor-
14.2
18.2
25.6
32.2
39.8
44.2
mance on NQ.
3.82
5.87
9.55
12.95
17.48
24.58
Contriever
7.91
10.55
15.36
19.61
24.89
33.03
Table Retriever performance (recall@k). For the Wikipedia-based dataset, the top indicates recall @k at the retrieval unit of Wikipedia paragraph and the bottom row for the unit of Wikipedia page. For BioASQ, the top row indicates recall @k at the unit of title or abstract of a PubMed article and the bottom row at the unit of the article itself.
16

---

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
FlanT5 LLaMa2 7B
rerank relevant
base
rerank
relevant
60
50
Reader F1
40
30
10
20
k (# of context passages)
Figure 17: Effects of applying reranking and instructing the model to focus on the relevant passages ("relevant"). These results are for when the retriever is ColBERT and the dataset HotpotQA.
FlanT5
LLaMa2 7B
rerank relevant
base
rerank
relevant
40
35
Reader F1
30
25
20
0
10
20
30
k (# of context passages)
Figure 18: Effects of applying reranking and instructing the model focus the relevant passages ("relevant"). These results are for when the retriever is ColBERT and the dataset is BioASQ.
17


Reviews:
[{"review_id": "McmTKHfnUA", "reviewer": "Reviewer_u17T", "summary": "This paper introduces RAGGED, a systematic framework for evaluating Retrieval-Augmented Generation (RAG) systems, focusing on stability, scalability, and robustness to noise. The authors analyze how retrieval depth, retriever-reader interactions, and dataset characteristics influence RAG performance, challenging the assumption that stronger retrievers universally improve results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We especially appreciate the recognition of how RAGGED addresses a gap in the literature by providing a unified framework for systematically evaluating scalability and stability and deriving actionable insights for real-world deployment. \n\n**1. Noise Assumptions in Robustness Evaluation**\n\nWe appreciate the reviewer’s clarifying question about the robustness of our setup. To clarify: our “noise” is not artificially injected or random. Rather, it consists of non-gold passages retrieved by a real retriever. These are often topically relevant but incorrect, representing a realistic and common failure mode in deployed RAG systems. This setup reflects naturally occurring retrieval imperfections that practitioners routinely encounter.\n\nWe agree that adversarial or temporally inconsistent noise (e.g., contradictory or outdated passages) is an important direction for future robustness research. However, modeling those scenarios requires assumptions (e.g., degree of contradiction, factuality, intent) that are beyond the scope of this work. That said, RSS could readily be extended to those settings, and we hope future work on adversarial or knowledge-conflict RAG systems will incorporate it.\n\n**2. Applicability to Extreme-Scale Corpora**\n\nWe agree that testing scalability on corpora with billions of documents would be highly valuable. However, such datasets with annotated gold passages are currently limited. RAGGED is designed to be scalable by construction and can be readily applied to larger corpora as they become available. \n\n**3. Empirical vs. Theoretical Foundations**\n\nWe appreciate the reviewer’s point. Our focus is empirical by design, aiming to provide actionable metrics for analyzing RAG system behavior in real-world settings. This mirrors the trajectory of many standard metrics (e.g., BLEU, ROUGE, F1), which were adopted for their practical value before receiving formal analysis. That said, we agree that theoretical grounding—such as bounding the relationship between noise and RSS—would strengthen the framework, and we will note this as an important direction for future work.\n\n\n**4. How would RSC metrics change if evaluated with adaptive retrieval strategies (e.g., FLARE’s iterative retrieval based on uncertainty)? Could such strategies invalidate the trade-offs observed in your experiments?**\n\nThis is an excellent and insightful point. Our current analysis focuses on the standard retrieve-then-generate paradigm with a fixed top-k retrieval depth, which remains a common baseline in RAG systems. However, the core idea behind RSC—measuring how performance scales with increased retrieval—can be naturally extended to adaptive retrieval strategies like FLARE.\nIn fixed-depth retrieval, RSC reflects the trade-off between performance gains and increasing the retrieval cutoff. In adaptive systems like FLARE, a similar trade-off exists between performance and the number or frequency of retrieval calls as determined by an uncertainty threshold. One could analogously define an RSC-style metric that varies the retrieval triggering threshold and measures how performance scales with the total retrieval budget. This would preserve the core RSC insight: quantifying trade-offs between retrieval cost and performance gain. We will mention this extension as a valuable future direction.\n\n**5. Justification of Metric Thresholds**\n\nWe chose ε = 0.5 for RSC based on empirical analysis: the standard deviation of F1 across all models and k values is consistently below 0.5 (mean ≈ 0.38, max = 0.46). This makes 0.5 a conservative but meaningful threshold that exceeds metric noise. Moreover, increasing ε to 0.6 or 0.7 preserves model ranking, indicating robustness to the threshold choice.\nFor RSS, we use a window of δ = ±5, which aligns with practical tuning ranges and matches how performance curves behave in most models (either plateauing or gently peaking). We also tested δ = 10 and found rankings unchanged. Smaller windows (e.g., δ = 1) lead to RSS variances across models of < 0.002, making the metric insensitive and less useful.\n\n**6. Statistical Robustness and Variability**\n\nTo address concerns about statistical rigor, we analyzed standard deviations across k values for each model and confirmed that they remain consistently low (all < 0.5). Additionally, we verified that model rankings remain stable under variations in ε and δ. These findings support the reliability and robustness of our conclusions, and we will include supporting variance analyses in the revised draft.", "claims_and_evidence": "The majority of claims are well-supported by systematic experiments and cross-model/dataset validation. Key limitations (threshold justification, closed-model opacity) do not invalidate the core findings but highlight areas for future work. The evidence convincingly demonstrates that reader robustness—not retriever quality—is the critical factor in RAG performance.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well-suited for the paper’s goals. The framework’s design—diverse datasets, retrievers, readers, and noise conditions—provides actionable insights into RAG stability and scalability. While the novel metrics (RSS/RSC) and semantic validation could be refined, they represent a meaningful step toward standardized RAG evaluation. The experiments convincingly demonstrate that reader robustness, not retriever quality, is the critical factor, validating the utility of the framework.", "theoretical_claims": "No theoretical claims are proposed in this paper.", "experimental_designs_or_analyses": "The experimental designs are largely sound for the paper’s goals, with rigorous testing across datasets, retrievers, and readers. However, the validity of conclusions is partially limited by arbitrary metric thresholds, insufficient statistical testing, and synthetic noise assumptions. Addressing these issues would strengthen the framework’s generalizability and robustness claims.", "relation_to_broader_scientific_literature": "The RAGGED framework advances the literature by formalizing robustness and scalability metrics for RAG systems, contextualizing instruction tuning’s role in noise tolerance, and validating semantic evaluation with LLM judges. Its innovations build directly on foundational RAG, robustness, and scaling literature while addressing underexplored challenges in real-world deployment. However, deeper engagement with fine-tuning-based RAG methods and realistic noise models would strengthen its positioning.", "essential_references_not_discussed": "Some related work about Adversarial Training of RAG.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The introduction of RSS (Retrieval Scale Sensitivity) and RSC (Retrieval Scale Cost) metrics formalizes robustness and scalability evaluation for RAG systems in a unified manner, addressing a gap in prior work that often treated these aspects separately.\n\n2.  Integrates concepts from adversarial robustness (e.g., noise injection) and computational efficiency (e.g., scaling with corpus size) into a cohesive evaluation paradigm, bridging domains like adversarial ML and distributed systems.\n\n3. Provides actionable insights for practitioners to benchmark and optimize RAG systems, particularly in noisy or large-scale environments (e.g., enterprise search, customer support).\n\nWeaknesses:\n\n1. Relies on random non-relevant passages for robustness testing, neglecting adversarial or temporally inconsistent noise (e.g., adversarial retrieval attacks, outdated facts), which limits practical relevance.\n\n2.  Experiments may not explore extreme-scale corpora (e.g., billions of documents), leaving scalability claims incomplete.\n\n3. Absent theoretical guarantees (e.g., bounds on RSS/RSC under noise, convergence properties), leaving the framework purely empirical.", "other_comments_or_suggestions": "1. Include adversarial perturbations (e.g., query paraphrasing, passage rewriting via LLMs) and temporal noise (e.g., outdated documents) to enhance robustness evaluation.\n\n2. Benchmark against retrieval attacks to stress-test RSS.\n\n3. Add a subsection analyzing the relationship between noise magnitude (e.g., % of corrupted passages) and RSS, possibly deriving error bounds.", "questions_for_authors": "1. Your robustness experiments inject random non-relevant passages as noise. Could your conclusions about RSS hold under more realistic noise scenarios, such as adversarial perturbations (e.g., passages with semantically similar but incorrect answers) or outdated documents? If not, how might this limitation affect the practical applicability of RSS?\n\n2. The scalability analysis assumes a fixed retrieval depth $k$. How would RSC metrics change if evaluated with adaptive retrieval strategies (e.g., FLARE’s iterative retrieval based on uncertainty)? Could such strategies invalidate the trade-offs observed in your experiments?\n\n3. Can you provide formal analysis (e.g., bounds) linking noise magnitude (e.g., % of corrupted passages) to RSS scores? For example, is there a threshold beyond which performance degradation becomes inevitable, regardless of retriever architecture?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzYwPZQHp5", "reviewer": "Reviewer_eNcB", "summary": "The paper introduces RAGGED, a framework for evaluating Retrieval-Augmented Generation (RAG) systems. It emphasizes that RAG's performance depends not only on retrieval quality but also on the reader's robustness to noise. The study shows that reader robustness is the key factor for RAG stability and scalability, and that fine-tuning readers is more important than improving retrievers. RAGGED provides a structured way to optimize retrieval strategies and guide future research on developing robust, scalable RAG systems.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer recognizes this paper provides a significant contribution by introducing structured metrics and is backed by extensive empirical analysis across multiple datasets and models.\n\n**1. RSS and its Relationship to Task-Specific Performance**\n\nIndeed, RSS is not intended to capture absolute task performance, and we were careful not to make such a claim. Instead, it measures the stability of performance near the optimal retrieval depth—capturing sensitivity to retrieval parameter changes, not performance magnitude. As the reviewer notes, these are distinct dimensions of system behavior.\n\n**2. Metric Comprehensiveness and Practical Relevance**\n\nWhile absolute performance is essential, stability and scalability are equally critical in real-world deployment:\n\n- Stability (RSS) robustness to retrieval depth variation, which is particularly important in real-world scenarios where precise tuning is costly.  \n\n- Scalability (RSC) reflects how well a model benefits from deeper retrieval, which is especially important when relevant information is sparse or buried. \n\nThus, while task-specific performance is foundational, it does not provide a complete picture of real-world robustness or efficiency. We believe that stability, scalability, and absolute performance together form a more holistic and practical evaluation of RAG systems. We also discuss the importance of absolute task performance in sections 4, 6, 7, paying attention to compare with-context and no-context performance to see when RAG actually helps. \n\n**3. Can models with lower RSS but higher task-specific performance still be considered stable or robust?**\n\nYes, a model can have high task-specific performance but low RSS. This would however mean it is brittle and harder to deploy reliably. This underscores the value of explicitly measuring stability and scalability alongside performance.\n\n**4. Theoretical analysis for generalizing findings beyond the empirical settings.**\n\n We appreciate the reviewer’s interest in generalizing the findings. While our focus is empirical, we hypothesize that a key factor driving behavior is the internal-external knowledge tradeoff: models with strong internal priors but weak integration of external input may degrade with increased retrieval.\n\nOur empirical results support this. For example, LLAMA begins with high no-context performance, and, at low k, its with-context answers closely mirror its no-context outputs—indicating strong reliance on internal knowledge. However, as more documents are retrieved, LLAMA’s answers diverge from its initial predictions (as shown in the attached figure), yet performance worsens. This suggests that LLAMA is incorporating context—but in a way that disrupts rather than improves its predictions. In contrast, FLAN shows lower no-context overlap from the start, adapts more readily to retrieved information, and maintains more stable performance as k increases.\nWe will incorporate this framing into the revised draft to better contextualize model behavior across tasks and domains.\n\n**5. A model like FLAN may benefit from retrieval more consistently than GPT-3.5, but are these differences purely due to the models’ architectures, or are there other factors like training data or task-specific tuning influencing the outcomes?**\n\nRegarding FLAN-T5 being more scalable than GPT-3.5, our observations suggest that differences could arise from both architecture and training objectives:\n\n- FLANT5 explicitly has denoising training, where it learns to reconstruct original text from corrupted inputs, potentially helping it handle noisy or partially irrelevant retrieval contexts better. GPT-3.5, in contrast, does not explicitly train with a denoising objective, possibly explaining its comparatively limited scalability.\n- FLANT5 (encoder-decoder) explicitly encodes the retrieved context separately from decoding (generation), potentially enabling it to manage larger context sets more effectively. GPT-3.5, being decoder-only, processes context in a strictly autoregressive manner, which may cause diminishing returns as the amount of context increases due to difficulty in attending equally well to all retrieved information.\n\n**6. Related Work: REALM and FiD**\n\nThank you for raising this. We agree that REALM [Guu et al., 2020] and FiD [Izacard & Grave, 2021] are foundational RAG systems:\nREALM introduced end-to-end retriever-reader pretraining and FiD showed the benefits of fusing multiple retrieved passages via decoder-only models. \n\nOur work is complementary: RAGGED offers tools to analyze reader-retriever behavior under varying retrieval depth and noise—dimensions not explicitly explored by these prior works. We will expand our Related Work section to reflect these connections.\n\n**7. Other Comments Or Suggestions** Thank you for the suggestions about the figures and formatting. We will fix them in the revision.", "claims_and_evidence": "The RAG Stability Score (RSS) measures stability across retrieval depths, but stability is a complex metric. The paper assumes stability directly correlates with system performance, yet it doesn't address whether improved stability at a specific depth always leads to better task performance or just more predictable behavior. How does RSS capture true model performance beyond retrieval consistency? Can models with lower RSS but higher task-specific performance still be considered stable or robust?", "methods_and_evaluation_criteria": "The RAG Stability Score (RSS) and RAG Scalability Coefficient (RSC) are introduced to measure stability and scalability across different retrieval depths. While these metrics provide some insight, they focus heavily on retrieval depth rather than directly measuring task-specific performance. Are these metrics comprehensive enough to evaluate the overall effectiveness of RAG systems in real-world tasks, especially when task-specific performance is more critical than retrieval stability?", "theoretical_claims": "The authors discuss the effects of retrieval depth on RAG performance, but without theoretical analysis, it remains unclear why these effects hold or how to generalize the findings beyond the empirical settings. The lack of theoretical grounding makes it harder to predict the behavior of RAG systems in novel conditions or tasks not covered in the paper.", "experimental_designs_or_analyses": "The experiments show variability in performance based on the reader-retriever combination (e.g., GPT-3.5 vs. CLAUDE vs. FLAN). While the inclusion of a range of models is a strength, it’s unclear how the variability between models is accounted for in the analysis. For example, a model like FLAN may benefit from retrieval more consistently than GPT-3.5, but are these differences purely due to the models’ architectures, or are there other factors like training data or task-specific tuning influencing the outcomes?", "relation_to_broader_scientific_literature": "The key contributions of the paper extend the findings of prior work by highlighting the nuanced relationship between retrieval depth, model stability, and task-specific effectiveness. It provides a more refined view of RAG, emphasizing the importance of reader robustness and domain-aware retrieval strategies, while challenging the oversimplified notion that stronger retrieval always leads to better performance. These contributions push the research further in understanding and optimizing the interaction between retrievers and readers for RAG systems.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "This paper introduces the novel RAGGED framework for evaluating Retrieval-Augmented Generation (RAG) systems, focusing on reader robustness and retrieval depth. The authors provide a significant contribution by developing the RAG Stability Score (RSS) and RAG Scalability Coefficient (RSC), offering a structured way to assess model performance across different configurations. Their extensive empirical analysis across multiple datasets and models strengthens the paper’s validity and relevance to future RAG research.\n\nHowever, the paper lacks a theoretical foundation to explain its empirical findings, limiting the generalization of its results. Additionally, the discussion of key related works, such as REALM and FiD, is insufficient, and the evaluation metrics focus more on stability than end-task performance.", "other_comments_or_suggestions": "Figures 1 and 8 are not clear enough. Letters in Figure 1 are too small. \nThe use of ** for bolding in the first paragraph of Section 2 of the article is not standardized.", "questions_for_authors": "Please refer to the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWBS5CHdmF", "reviewer": "Reviewer_jqB4", "summary": "The paper presents RAGGED—a evaluation harness for retrieval-augmented generation (RAG) systems. The authors identify conflicting narratives in the previously published literature, and aim to resolve it especially around sensitivity to irrelevant documents. They examine how different retriever methods (e.g., BM25, ColBERT, Contriever) and reader models interact across various tasks and datasets (NQ, HotpotQA, and BioASQ). The framework introduces novel metrics—the RAG Stability Score (RSS) and the RAG Scalability Coefficient (RSC)—to quantify how stable and scalable a model’s performance is as retrieval depth increases. Ultimately, the paper argues that the robustness of the reader to noisy retrieval (i.e., its ability to filter out irrelevant passages) is the key determinant of overall RAG effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive and thoughtful feedback. We appreciate that the reviewer recognizes that the paper resolves open questions in existing works and finds all the claims to be well supported. \n\n**1. Expanding to Newer Embedding Models**\n\nWe appreciate the suggestion to evaluate newer embedding models. We are in the process of integrating models from the MTEB leaderboard, including leading open-source options like Linq-Embed-Mistral. However, due to the scale of the corpus (~111M passages) and the limited resources and time, we were unable to complete these runs within the rebuttal period. We are actively running the experiments and will include updated results in the revision. We agree that this extension will further strengthen our analysis.\n\n\n**2. Evaluating with CRAG**\n\nThank you for highlighting the importance of more modern evaluation datasets. We agree that CRAG offers a more recent and challenging benchmark for RAG evaluation.\n\nTo assess generalization, we ran preliminary experiments on CRAG during the rebuttal window using two representative readers: FLAN-T5 (of improve-then-plateau behavior) and LLAMA-2 (of showing peak-then-decline). These behavioral trends remain consistent on CRAG, suggesting that the retrieval-depth dynamics generalize well to newer datasets. One difference is that the overall performance is worse (shifted down) compared to the performance we observed in our paper, which is as expected since this is a newer dataset. We will include full CRAG results in the camera-ready version.\n\n| Model     | k=1   | k=5   | k=10  | k=20  | k=30  | k=50  |\n|-----------|-------|-------|-------|-------|-------|-------|\n| FLAN-T5   | 0.190 | 0.174 | 0.178 | 0.175 | 0.175 | 0.175 |\n| LLAMA     | 0.204 | 0.227 | 0.227 | 0.227 | 0.227 | 0.227 |\n\n\n**3. In Figure 5, FLAN-T5 RSS is at 0.99. Do you suspect this is a function of the eval setting (context length cap), or do the authors happen to have any additional insights?**\n\n Thank you for this insightful question. We investigated whether FLAN-T5’s high RSS score might be influenced by context length limitations, particularly around the optimal retrieval depth of k = 25.\n\nTo assess this, we compared token lengths after truncation between k = 20 and k = 25. We found that in at least 32% of examples, the input at k = 25 includes more tokens post-truncation than at k = 20, confirming that additional retrieved content is being processed. Despite this added context, the performance difference between k = 20 and k = 25 is < 0.5 points, suggesting the model remains robust even as more content is introduced.\n\nThis supports our interpretation that FLAN-T5’s high RSS is not an artifact of truncation shielding the model from additional noise, but rather reflects genuine retrieval-depth stability. We will include this clarification and supporting analysis in the revised manuscript.", "claims_and_evidence": "Reader Robustness Is Paramount: The paper claims that a reader’s ability to handle noise is more critical than mere retriever quality for achieving stable and scalable RAG performance.\n\nRetrieval Depth Effects Are Model-Dependent: It argues that while some models benefit from more retrieved passages (improve-then-plateau behavior), others degrade when faced with increased noise (peak-then-decline behavior).\n\nRetrieval Improvements Yield Nonlinear Gains: Enhanced retriever performance (for instance, using dense retrievers) does not always translate into proportional improvements in reader performance, particularly for noise-sensitive readers.\n\nDomain-Specific Nuances Matter: The dynamics of retrieval and reading differ between open-domain tasks (like NQ and HotpotQA) and specialized domains (such as BioASQ), highlighting the need for tailored configurations.\n\nI find all the claims to be well supported with breadth but not with depth. For example, BM-25, contriever are both not SOTA for embeddings.\n1. I would urge the authors to also evaluate their system on modern embeddings such as text-embedding-large (fromOAI) for closed source among many others, and some of the SOTA generative or discriminative models from HuggingFace mteb leaderboard (for opensource). \n2. The datasets used by the authors are not the ideal use-cases to evaluate LLM's ability to retrieve. For example, HotPotQA documents are all within a few hundred to thousand tokens with very clear \"knowledge\" component which most models listed have seen before (given the high overlap with refinedweb and other pre-training datasets). To the extend HotPot pass@K is today used as a pre-training metric. Hence, I would encourage the authors to engage with newer RAG benchmarks such as CRAG", "methods_and_evaluation_criteria": "Addressed above extensively", "theoretical_claims": "No", "experimental_designs_or_analyses": "I have already mentioned my concerns around the dataset used for eval, and the retrievers.\n\nI would also like to add that the LLMs used are quite dated. However, while I encourage the authors to consider updating this for camera-ready or a subsequent submission, I do NOT think this is a serious concern since it should not be on the authors to keep up with the fast moving model releases and the trends hold independent of the models.", "relation_to_broader_scientific_literature": "In my view this paper answers some of the open questions from the \"Lost in the middle \" paper by clearly defining the 2 new metrics: RSS and RSC.", "essential_references_not_discussed": "Reference of CRAG for evals: https://arxiv.org/html/2406.04744v1", "other_strengths_and_weaknesses": "Addressed above", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In figure 5, flan-T5 RSS is at 0.99. Do you suspect this is a function of the eval setting (context length cap), or do the authors happen to have any additional insights?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "48Zw6JFzbi", "reviewer": "Reviewer_SdGE", "summary": "This paper carries out empirical analysis to shed light on the impact of retrieval in a RAG system: 1/ when retrieval is needed, 2/ impact of retrieval depth, 3/ noisy retrieval, 4/ relation between retrieval improvements to final performance improvements. The paper proposes two metrics, RAG Stability Score and RAG Scalability Coefficient, to measure the robustness of a RAG system. Experiments are carried out on NQ, HotpotQA, and BioASQ dataset. The main findings show that: 1/ retrieval may not help depending on if model is robust to noise, 2/ when increasing retrieval depth some models improve-then-plateau some models peak-then-decline, 3/ model noise robustness is more important than retrieval noise filtering, 4/ retriever improvements does not always lead to better response quality.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our empirical contributions and insights into retrieval dynamics, particularly the introduction of RSS and RSC as tools for understanding reader robustness and retrieval scalability. \n\n**1. Symmetry of RSS**\n\nThank you for the insightful observation. While it is true that adding lower-ranked documents (right side) and removing top-ranked ones (left side) may have different theoretical impacts, our empirical results show that this asymmetry is not consistent in practice. For example, LLAMA2-7B degrades more on the left (removing documents), whereas LLAMA3-70B degrades more on the right (adding documents). This variation supports our use of a symmetric window, which offers a model-agnostic and balanced way to measure stability. We will clarify this in the revision.behaviors. We will clarify this rationale in the revised manuscript.\n\n\n**2. Clarifying Design of RSC**\n\nRSC is defined as the product of two complementary components: 1) the last retrieval depth before plateau/decline, which captures how far the model scales before benefits taper off, 2) the cumulative performance gain up to that point, which quantifies the total utility derived from increasing retrieval.\n\nBy multiplying these factors, RSC distinguishes clearly between models that achieve quick but limited improvements (early plateau) and those that sustain meaningful improvements over deeper retrieval depths.\n\nThe hyperparameter ε, it is intentionally user-defined, similar to early stopping in training. It allows users to specify what constitutes a meaningful gain, based on their deployment trade-offs. We chose ε = 0.5 for RSC based on empirical analysis: the standard deviation of F1 across all models and k values is consistently below 0.5 (mean ≈ 0.38, max = 0.46). This makes 0.5 a conservative but meaningful threshold that exceeds metric noise. Moreover, increasing ε to 0.6 or 0.7 preserves model ranking, indicating robustness to the threshold choice.\n\n**3. F1 vs. Semantic Metrics**\n\nWe recognize that unigram-based F1 may underrepresent semantic correctness. To address this, we validated our findings using an LLM-based semantic correctness metric on a subset of responses (Appendix H), and found strong alignment with the trends captured by F1. We will make this point clearer in the revised version.\n\n**4. Memorization of Dataset by LLMs.**\n\nThank you for the comment. We agree that newer datasets reduce the risk of memorization. Our original evaluation includes NQ (2019), HotpotQA (2018), and BioASQ (2023) to cover open-domain, multi-hop, and biomedical QA settings.\nTo extend this, we incorporate a 2024 dataset, CRAG, as suggested by reviewer jqB4. We ran preliminary experiments on CRAG using two representative readers: FLAN-T5 (of improve-then-plateau behavior) and LLAMA-2 (of showing peak-then-decline). These behavioral trends remain consistent on CRAG, suggesting that the retrieval-depth dynamics generalize well to newer datasets. We will include full CRAG results in the camera-ready version.\n\n| Model     | k=1   | k=5   | k=10  | k=20  | k=30  | k=50  |\n|-----------|-------|-------|-------|-------|-------|-------|\n| FLAN-T5   | 0.190 | 0.174 | 0.178 | 0.175 | 0.175 | 0.175 |\n| LLAMA     | 0.204 | 0.227 | 0.227 | 0.227 | 0.227 | 0.227 |\n\n**4. Applicability to Newer LLMs and Time-Sensitive Queries**\n\nWe conducted additional experiments on dynamic questions in CRAG using both FLAN-T5 and LLAMA, and observed the following:\n\n- FLAN-T5 maintains its improve-then-plateau retrieval-depth behavior on dynamic questions, consistent with its pattern on static ones. This suggests strong generalization and robustness to retrieval noise across domains.\n\n- LLAMA, in contrast, shifts from a peak-then-decline trend on static questions to an improve-then-plateau trend on dynamic questions. We attribute this to differences in internal knowledge: on static questions, LLAMA already knows the answer and retrieval introduces redundancy or contradictions, degrading performance at higher depths. On dynamic questions—where LLAMA lacks internal knowledge—it relies more heavily on retrieved context, and even noisy documents provide helpful signal, leading to continued improvement.\n\nThis contrast reinforces the value of our reusable framework: even when retrieval dynamics vary across models or tasks, RSS and RSC offer a principled way to detect, quantify, and interpret these differences. Rather than assuming fixed behavior, our reusable framework and metrics help uncover how model robustness and scalability shift across domains.\n\nWwe expect newer LLMs—particularly those fine-tuned for retrieval-augmented tasks—to exhibit higher RSS and RSC due to improved handling of noisy or diverse context. Regardless of the model architecture, however, we believe our metrics remain essential tools for diagnosing retrieval sensitivity and informing robust, real-world RAG deployment.", "claims_and_evidence": "This is an empirical paper where the findings are model-dependent. The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "I have the following concerns over the proposed metrics:\n\n1. The proposed RAG Scalability Score (RSS) is defined as a symmetric measure, yet retrieval results are inherently asymmetric. Removing higher-ranked retrieval results has a greater negative impact than adding lower-ranked ones. Given this asymmetry, it is unclear why RSS can be considered a reliable and stable metric.\n\n2. I find the RAG Scalability Coefficient (RSC) is difficult to interpret. It is defined as the product of the last retrieval depth before performance plateaus and the cumulative improvements, but the rationale behind this multiplicative relationship is unclear. Additionally, this metric depends on a hyperparameter $\\epsilon$, which may be challenging to tune in practice.", "theoretical_claims": "Not Applicable", "experimental_designs_or_analyses": "Dataset: Experiments are carried out on NQ, HotpotQA, and BioASQ datasets. However, I think NQ and HotpotQA are not diverse or representative enough for RAG evaluation due to many facts are memorized by LLMs.\n\nEvaluation metric: unigram-based score may not reliably measure LLM performance due to challenges in lexical matching.", "relation_to_broader_scientific_literature": "This paper offers empirical findings and suggestions on developing RAG systems. The evaluation setups are similar to literature and the paper proposes two new metrics, RAG Stability Score and RAG Scalability Coefficient, to measure the robustness of a RAG system.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "I liked the empirical findings and insights from this paper. But I think the proposed metrics are not well-defined and the datasets may not cover representative RAG use case.", "other_comments_or_suggestions": "Minor suggestion: it is recommended to produce figures in vector format, e.g., pdf.", "questions_for_authors": "Your findings appear to be closely tied to the specific dataset or model used. How do you envision these insights applying to newer LLMs to time-sensitive questions, and what factors might influence their generalizability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work addresses the problem of systematically evaluating RAG systems. It argues that RAG systems are sensitive to system configuration and their effectiveness varies with the configuration, often resulting in suboptimal or degraded performance. It proposes a framework, RAGGED, as a step towards configuring RAG systems in an informed manner. The framework is intended to enable RAG engineers to evaluate their systems across diverse retriever-reader configurations, retrieval depths, and datasets. The work applies the proposed framework  open-domain, multi-hop, and specialized-domain datasets and the key insight obtained from the analysis is that how well the reader model deals with noise has the most significant influence on the effectiveness of RAG systems. The work proposes two metrics to quantify the robustness of RAG systems, RSS and RSC. RSS is intended to measure the consistency of the system's performance around its optimal retrieval depth. RSC is intended to measure the accumulated performance gain over the largest window where the model continues to gain from the increased retrieval depth.

The empirical study is done on three retrievers one from each type: BM25 representing classical IR system, ColBERT representing neural retrievers and Contriver representing dense retrievers. Several open source and closed source reader models are evaluated. Three QA datasets are used for evaluating RAG performance in the study with one representative from each of general knowledge, complex reasoning and domain-specific retrieval. To evaluate retriever performance recall@k and to evaluate reader performance unigram F1 measures are used.

The empirical study asks several questions and attempts to find answers for them. The first question is on the conditions that enable superior RAG performance over closed-book generation. A sub-question asked is does retrieval always help. The answer is no as some models benefit from retrieval whereas others don't. It is suggested that retrieval is not inherently beneficial and it is hypothesized that handling noise in the retrieval results is crucial to get consistent benefits. Another sub-question asked is does retrieval help all tasks and domains uniformly. The answer is again no as it is found that some tasks benefit more than others (ulti-hop questions vs single-hop questions). 

The second major question is on the impact of retrieval depth on stability and scalability. The study finds that in some systems, improve-and-plateau trend is seen and in others peak-and-decline trend is observed. In summary, increasing retrieval depth is not always beneficial.  

The third major question is on how noise affects the performance of the reader. To answer the question, a comparison is made between two scenarios - first scenario where at least one gold passage is present and the second scenario where no gold passage is present. The key finding is that most models perform worse than no-context baseline when no gold passages are retrieved. Further, the study finds that prompting with explicit relevance instructions improve noise filtering doesn't consistently help across the models. Based on this observation, the study recommends fine-tuning readers for noise resilience over retrieval-side interventions.
 
The study also investigates the influence of retriever choice on RAG system performance. It finds that retriever quality improves recall but doesn't necessarily improve the reader's performance. Similarly, it finds that reranking doesn't always improve the reader's performance. 

The study recommends tuning of retrieval depth for every model and  fine-tuning readers for stability. 

Reviewers agree that the claims made in the submission are backed by systematic experimentation and that the study provides potentially useful insights to practitioners. Reviewers expressed concern about the generalizability of the findings as a) it is not backed by theoretical analysis b) retrieval systems and models considered in the study are not state-of-the-art c) benchmarks used for evaluation are dated. Reviewers also expressed concern about a) the soundness of the proposed measures for stability and scalability b) focusing too narrowly on stability of RAG systems without giving task performance its due importance. Reviewers made some good suggestions to strengthen the work further by a) benchmarking against retrieval attacks to stress-test the proposed measures b) evaluating systems with adaptive retrieval strategies c) including SOTA models and newer benchmarks in the study.

Despite its aforementioned limitations, the work makes a strong case for rethinking the evaluation of RAG systems. As one of the reviewers put it, the work "provides a more refined view of RAG, emphasizing the importance of reader robustness and domain-aware retrieval strategies, while challenging the oversimplified notion that stronger retrieval always leads to better performance. These contributions push the research further in understanding and optimizing the interaction between retrievers and readers for RAG systems."
Decision: Accept (poster)