RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems

Authors: Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, Graham Neubig
Primary area: foundation or frontier models, including LLMs
Keywords: Retrieval Augmented Generation, Language Modeling, Question Answering

Abstract:
Retrieval-augmented generation (RAG) systems have shown promise in improving
task performance by leveraging external context, but realizing their full potential
depends on careful configuration. In this paper, we investigate how the choice of
retriever and reader models, context length, and context quality impact RAG per-
formance across different task types. Our findings reveal that while some readers
consistently benefit from additional context, others degrade when exposed to irrele-
vant information, highlighting the need for tuning based on reader sensitivity to
noise. Moreover, retriever improvements do not always translate into proportional
gains in reader results, particularly in open-domain questions. However, in spe-
cialized tasks, even small improvements in retrieval can significantly boost reader
results. These insights underscore the importance of optimizing RAG systems by
aligning configurations with task complexity and domain-specific needs.

Decision: reject
Reviewer scores: 8, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1730607838429

thread_type:
official_review

<thread_object>
creator: reviewer_e7uq

<content>
summary:
The paper introduces RAGGED, a framework for analyzing and evaluating RAG systems. While the work presents a comprehensive analysis framework, its contributions mainly focus on investigating how different components and configurations affect RAG performance across various tasks and domains. 
The main contributions include:
- A systematic analysis framework examining RAG components,
- Empirical findings on RAG performance characteristics,
- Investigation of retriever-reader relationships,
- Domain-specific insights for RAG applications.

strengths:
- The study examines multiple dimensions of RAG systems with a systematic approach, using diverse models and configurations. This provides valuable empirical evidence about component interactions.
- The authors provide their code and framework publicly, with well-documented experimental setups and clear implementation details.
- The work challenges some common assumptions about RAG systems, particularly regarding the relationship between retriever and reader performance.

weaknesses:
- While the paper presents extensive empirical findings about RAG system performance, it lacks a theoretical framework that could explain the underlying patterns and relationships between reader model's architectures (e.g., parameter count, token limits) and their performance characteristics, which could have enabled broader generalizations about unexplored model families and helped predict optimal configurations based on architectural features rather than just empirical observations.
- The evaluation focuses narrowly on question-answering tasks with relatively small datasets (2837 queries for NQ, 5600 for HotpotQA, 3837 for BioASQ). This raises questions about the generalizability of findings to other applications/domains.
- The paper could benefit from more rigorous comparison with existing RAG analysis frameworks or methodologies to better establish its unique contribution.

questions:
- Could the authors analyze how model characteristics (e.g., parameter count, architecture type, pre-training approach, token limits) correlate with RAG performance patterns? Have you observed any clustering of performance patterns based on model families or architectures that could suggest theoretical principles for predicting RAG behavior? Such analysis might help generalize findings to unstudied models and provide theoretical grounding for the empirical observations.
- The significantly different behavior observed in specialized domains (e.g., BioASQ) vs. open domain is intriguing. Could you elaborate on what characteristics of specialized domains lead to these differences? Have you analyzed if reader models pre-trained on domain-specific data show different RAG behavior patterns?
- Could you provide insights into how to predict whether a domain would benefit more from dense vs. sparse retrievers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629492363

thread_type:
official_review

<thread_object>
creator: reviewer_f1ot

<content>
summary:
This paper proposes RAGGED, aiming to study the impact of different readers, retrievers, and the number of retrieved documents. It reveals the different response patterns of the reader model to contexts and provides suggestions for deploying an RAG system.

strengths:
This paper proposes the RAGGED framework with the comprehensive and systematic study of multiple key aspects of RAG configuration. It constructs a complete assessment system through four clear research questions (R1-R4).

weaknesses:
# Limitations in Methodological Approach

### Retrieval Architecture Constraints

- Limited Exploration of Modern Retrieval Methods
  - The study did not incorporate state-of-the-art retrievers (e.g., BGE) and reranking mechanisms
  - Contemporary search engines and their potential impact were not evaluated
- Generalizability Concerns
  - The conclusion that "better searchers ≠ better RAG performance" may require reconsideration in light of recent empirical evidence using modern retrieval systems

### Architectural Paradigm Limitations

- Restricted RAG Implementation Scope
  - The investigation did not encompass emerging RAG variants (e.g., Self-RAG) and recent Large Language Model paradigms (e.g., OpenAI O1)
  - The impact of prompt engineering variations was not systematically explored (e.g., CoT and Self-Ask)
- Retrieval Source Considerations
  - Limited investigation into how different knowledge source types affect system performance

### Dataset and Task Coverage

- The study's conclusions are constrained by the limited variety of datasets and task types examined (only 3 datasets)
- A broader spectrum of use cases would strengthen the generalizability of the findings



# Limitations in Experimental Depth and Analysis

## Experimental Design Constraints

### Variable Control Inadequacies

- Noise Distribution Effects: The impact of noise location within documents remains unexplored
- Document Structure Variables
  - The influence of paragraph length variation was not systematically analyzed
  - Potential correlations between document structure and retrieval performance were not examined

### Analytical Depth Limitations

- Lack of analysis of the reasons why "Some retriever-reader combinations consistently benefit from RAG, while others only sometimes do for certain k’s, and a few never do regardless of k" 
- The analysis lacks depth, only revealing obvious, common, and widely known conclusions without examining the underlying causes. I suggest analyzing the conclusions mentioned in the article from the perspective of how LLMs handle conflicts and integration between their internal knowledge and external information.

## Practical Implementation Constraints

- Configuration Guidelines
  - Lack of concrete, actionable implementation recommendations
  - Modern RAG systems include many other components, such as rerankers, scoring mechanisms, query routing, and query rewriting. **Your work appears to be limited to performing grid search on just a few hyperparameters.**
- The applicability/generalizability to other tasks cannot be guaranteed.

questions:
- Q1: Why weren't modern retrievers like BGE and search engines considered? Would this affect the reliability of the main conclusions?
- Q2: How was the strategy for selecting noise paragraphs determined? Does this choice affect the validity of the experiments?
- Q3: Does the conclusion "better retrievers ≠ better RAG performance" still hold after incorporating rerankers and more modern retrievers?
- Q4: Do the improve-then-plateau and peak-then-decline behavior patterns still exist with different RAG and LLM paradigms?
- Q5: Why was unigram F1 chosen instead of other potentially more suitable evaluation metrics? The paper mentions that “many LMs respond correctly semantically but use inexact or more verbose wording.” which can be avoided by extracting the answer using the LLM. The effect of the Unigram-F1 assessment is not comprehensive enough.
- Q6: What are the criteria for selecting noise paragraphs in Section 6? 
- Q7: Was the impact of different retrieval passage lengths considered?
- Q8: Can you explain the underlying reasons why different models respond differently to RAG improvements?
- Q9: How do the quantity and scope of parametric knowledge correlate with performance variations across different LLMs? Additionally, could the authors explore the dynamics of knowledge conflicts and integration between an LLM's internal (learned during pre-training) and external (retrieved during inference) knowledge bases?
- Q10: The paper mentions providing "To provide more concrete suggestions of the best practices under various cases, we introduce an analysis framework, RAGGED," What are some efficient approaches to determine the optimal RAG configuration when working with new datasets or models? Specifically, how can practitioners rapidly identify the most effective combination of retrieval methods, LLMs, $k$, and other key parameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717083597

thread_type:
official_review

<thread_object>
creator: reviewer_mkty

<content>
summary:
The authors introduce an analysis framework, RAGGED, for understanding RAG best practices, namely, whether RAG helps, how many passages to retrieve, which retrievers and reader models to choose, and how that varies across tasks. They conduct this analysis across the retrievers BM25, ColBERT, and Contriever and assess several reader models, evaluating these on NQ (general domain), HotPotQA (complex questions), and BioASQ (specialized domain). Overall, they find that readers vary (some benefit from higher recall at larger retrieval depths; others are sensitive to noise) and that improved retrieval delivers more noticeable gains in specialized domains.

strengths:
1) The paper is very well-written and structured, and hence easy to follow. The research questions posed are clear and the findings are stated and discussed in a well-presented manner.
2) The work empirically identifies trends that are of interest to the community, serving as evidence, for instance, that RAG helps more in specialized domains and that improving the components of a RAG system separately does not guarantee higher end-to-end quality.
3) Thoughtful analysis of RAG systems, which have very quickly become ubiquitous, are very much needed.

weaknesses:
While I admire a thoughtful analysis paper, the present work simply does not explore rich enough questions to balance the lack of novel insight nor does it have enough empirical support for the nuanced claims it tries to extract. To illustrate, the primary non-trivial aspect of the present study are interesting claims like "improved retrieval helps more in specialized domains". This is analytically self-evident (i.e., is not necessarily a new claim) but good empirical support for such claims is always valuable. Unfortunately, the work is essentially trying to support this post-hoc from results on NQ vs. BioASQ, two of the most standard, over-used, and overly-simplistic datasets. To be clear, NQ and BioASQ are still valid and interesting datasets, but the lack of novelty coupled with the factoid and short answer nature and small variety of datasets selected makes it very hard to position this work at ICLR in my current read.

To be very clear, I think work like this is necessary, but to publish this as a long research paper the bar to clear might involve something llike (i) innovating dramatically more on the evaluation front, e.g. assessing human evaluators against a large number of automatic metrics, not just F1 overlap, over a large set of (perhaps contributed) new evaluation benchmarks that vary certain axes more directly or (ii) testing a set of less overused claims and hypotheses, e.g., novel methods that counteract the challenges the authors have observed. Right now, the contribution is simply too small and too ephemeral of an addition to the literature in this (overcrowded) domain.

I encourage the authors to continue this thoughtful analysis line of work and to consider how downstream RAG applications look like in practice --- or alternatively to take some of their statements presented as findings (e.g., improving retrieval doesn't always improve RAG quality) and present deeper insight or a method that contributes to better understanding or better quality.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1731326179703

thread_type:
official_review

<thread_object>
creator: reviewer_l2xh

<content>
summary:
This paper studies how the components of a retrieval-augmented generation (RAG) system impact its performance on a variety of tasks. They propose an evaluation framework called RAGGED that  evaluates the impact of more context, inclusion of irrelevant information, and the interaction between task type model performance. They compare combinations of three retrievers (BM25, ColBERT, Contriever) and four readers (FLAN, LLaMa, GPT, Claude) on multiple datasets and present a series of findings that make the case for designing RAG frameworks that are task-specific and tailored to their use case.

strengths:
- With increasing adoption of RAG for a variety of tasks, this is a useful direction of research that helps researchers understand when and how to use this set up. These findings could be potentially useful for deciding how to configure a RAG framework based on the task.

- The RAGGED framework proposed in this paper is well-designed and addresses critical questions at each step of the pipeline. The authors aim to release the code for this framework which would enable easier replication and extension to include more models/experiments.

- The paper is well-written and the results are presented in a very easy to understand manner. I really appreciate how Table 1 and the figures that follow communicate the core findings succinctly. The appendix also provides useful additional details for the experiments.

weaknesses:
- The paper does not compare more state of the art closed-source models like GPT-4 and Claude 3 Opus. It’s understandable that cost may be a hindrance, but it would be helpful to conduct even a smaller set of experiments using stronger models so the takeaways from this paper are more broadly applicable.

- Unigram F1 doesn’t strike me as the ideal metric for evaluating reasoning quality in LLMs. Perhaps the authors could have explored semantic embedding-based approaches or by simply asking another LLM to verify the (output, label) pair for accuracy.

- While the takeaways are interesting, it would have been helpful to hear the authors’ thoughts on why certain models outperform others in different settings.

questions:
- Did the authors try any kind of prompt engineering to see if it improves performance?
- The FanOutQA benchmark (Zhu et al., ACL 2024) strikes me as a particularly suitable Wikipedia-based testbed for this framework. Did the authors attempt to include it in their evaluation? 
- Another increasingly important factor in RAG systems in multilinguality, which impacts both retrievers and readers. It would be useful to include at least one multilingual dataset, like NoMIRACL (Thakur et al., EMNLP 2024), in this testing framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732300440778

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Reviewer Comments: Clarifying Novelty in Our Approach

comment:
While we understand some findings may seem intuitive, it is not necessarily the case that they hold across all models and question types. As such, phenomena that are not attested to in the literature should be examined in a systematic fashion.

Below are key findings from our study and how they provide a more nuanced and comprehensive view compared to existing literature. These comparisons are cited in the paper, but we will revise the text to make these distinctions clearer:

- __RAG vs. No-Context Generation__: 

While [1] demonstrate that retrieval-augmented generation (RAG) outperforms closed-book (no-context) generation and achieved state-of-the-art results for QA tasks at the time, we extend this understanding. Specifically, we show that with the wrong configuration, RAG readers can perform worse than no-context generation (Sec. 4). For some readers, benefitting from RAG requires a large enough k while others require a small enough k. This highlights the importance of configuration, a point largely unexplored in [1]. 

*To the best of our knowledge, we are the first to study the comparison with no-context performance more comprehensively across RAG configurations and question complexities (hops)/domains.*

- __Reader trends across k__: 

Prior works offer limited experiments and seemingly conflicting insights into how readers perform with increasing k:
[2] observed saturation in reader performance as k increases, but their analysis was limited to specific question types and retrievers.
[3,4] reported performance degradation at higher k, without fully explaining the underlying causes.
Our study resolves these inconsistencies by conducting more comprehensive experiments that span multiple datasets, retrievers, and readers. This allows us to identify reader robustness to noise as the key determinant of trends across k. 

*To the best of our knowledge, we are the first to conduct a granular slice analysis of robustness by examining the impact of noise on different reader models specifically in the presence and absence of signal (Sec 6), resulting in key findings such as: RAG can perform worse than no-context even in the presence of sufficient signal*.

- __Retriever Importance Across Domains and Tasks__: 

While [4] found a positive correlation between retriever and reader performance on a single dataset, we present a more nuanced perspective *across question types*:
Domain-specific tasks (e.g., biomedical QA) gain significantly more from improved retrievers than open-domain tasks.
Single-hop questions benefit more than multi-hop, though the gap is smaller than the gap between special domain v. open domain.
These findings provide a broader, more detailed understanding of how retriever performance impacts reader effectiveness across different scenarios.

Nonetheless, If you find similar findings in other papers, we are happy to incorporate and discuss these papers.

[1] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,
Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented genera-
tion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:
9459–9474, 2020.

[2] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and
Percy Liang. Lost in the middle: How language models use long contexts, 2023.

[3] Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano,
Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval
for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research
and Development in Information Retrieval, pp. 719–729, 2024.

[4] Paulo Finardi, Leonardo Avila, Rodrigo Castaldoni, Pedro Gengo, Celio Larcher, Marcos Piau, Pablo
Costa, and Vinicius Caridá. The chronicles of rag: The retriever, the chunk and the generator.
arXiv preprint arXiv:2401.07883, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732300538665

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Reviewer Feedback: Using Semantic-Based Metrics for Evaluation

comment:
We appreciate the reviewers' feedback regarding F1’s limitations. While we chose F1 for its simplicity and alignment with prior work, we agree that it may not fully reflect nuanced semantic equivalence. To address this, we ran an LLM-based evaluation of the models for all of the datasets using Prometheus [1], which has a Pearson correlation of 0.897 with human evaluators. We find that __the conclusions about reader trends do not change: the same reader trends apply to the same models__ (LLaMa and Claude still exhibit peak-then-decline while GPT-3.5 and FlanT5 still exhibit improve-then-plateau). We use Prometheus-7b-v2.0 to evaluate the correctness of the generated answer against the gold answer on a 5-point scale, where 1 is the least correct and 5 is the most correct:

Dataset: Natural Questions
| Model          | 1     | 2     | 5     | 10    | 20    |
|----------------|-------|-------|-------|-------|-------|
| LLaMA2 7B      | 3.22  | 3.19  | 3.06  | 2.84  | 2.70  |
| FlanT5         | 3.27  | 3.43  | 3.56  | 3.63  | 3.64  |
| GPT-3.5        | 3.48  | 3.56  | 3.63  | 3.67  | 3.72  |
| Claude Haiku   | 3.19  | 3.16  | 2.92  | 2.68  | 2.36  |

Dataset: HotpotQA
| Model          | 1     | 2     | 5     | 10    | 20    |
|----------------|-------|-------|-------|-------|-------|
| LLaMA2 7B      | 3.20  | 3.14  | 3.03  | 2.93  | 2.87  |
| FlanT5         | 3.12  | 3.11  | 3.17  | 3.19  | 3.21  |
| GPT-3.5        | 3.65  | 3.75  | 3.80  | 3.83  | 3.85  |
| Claude Haiku   | 3.26  | 3.03  | 2.63  | 2.38  | 2.27  |

Dataset: BioASQ

| Model          | 1     | 2     | 5     | 10    | 20    |
|----------------|-------|-------|-------|-------|-------|
| LLaMA2 7B      | 2.94  | 2.97  | 2.89  | 2.82  | 2.63  |
| FlanT5         | 2.94  | 3.03  | 3.14  | 3.19  | 3.20  |
| GPT-3.5        | 3.21  | 3.14  | 3.19  | 3.21  | 3.23  |
| Claude Haiku   | 2.90  | 2.86  | 2.64  | 2.48  | 2.25  |

[1] Kim, Seungone, et al. "Prometheus 2: An open source language model specialized in evaluating other language models." arXiv preprint arXiv:2405.01535 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732300681993

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Reviewer Comments: Explaining Reader Trends

comment:
There are two primary types of readers observed in our experiments:
- __Peak-then-Decline Behavior__: Models including those from the LLAMA and Claude families show sensitivity to noisy documents, leading to performance degradation as the number of retrieved passages (k) increases beyond a certain point.
- __Improve-then-Plateau Behavior__: Models including those from the GPT and FLAN families are more robust to noise, continuing to benefit from additional context until performance plateaus.

Since we don’t have access to the details of the closed-source models, we will focus on providing hypotheses according to the open-source model (LLaMa belonging to the peak-then-decline behavior and the FLAN models belonging to the improve-then-plateau family).

On the one hand, FLAN, an improve-then-plateau model family, incorporates additional strategies explicitly designed to handle noisy or diverse contexts. It employs denoising strategies, such as a mixture-of-denoisers, during training to improve its robustness to irrelevant or noisy contexts. These enhancements enable it to filter out noise more effectively. 

On the other hand, LLaMa’s training predominantly relies on next-token prediction with limited exposure to noisy or retrieval-specific scenarios, making it sensitive to noise at higher k.

We also note that there are some model architecture features that *alone do not determine reader behavior*:
- __Context window size__: Models with longer context limits like LLaMa2 (4k tokens) don't necessarily process a larger number of contexts better than models with smaller context limits like FLAN (2k tokens).

- __Encoder-decoder v. decoder__: LLaMa is a decoder-only model that displays peak-then-decline behavior, but GPT models are also decoder-only and instead display improve-then plateau behavior.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary of the paper: This paper introduces RAGGED, an evaluation framework designed to analyze the performance of RAG systems. It investigates how various components—such as different retrievers (BM25, ColBERT, Contriever) and reader models (FLAN, LLaMa, GPT, Claude)—impact RAG effectiveness across diverse tasks and datasets, including NQ, HotPotQA, and BioASQ. Key findings reveal that the performance of reader models varies significantly based on retrieval depth and sensitivity to irrelevant information, highlighting the importance of task-specific RAG configurations. The framework aims to provide insights for optimizing RAG system deployment by systematically evaluating component interactions and performance characteristics in both general and specialized domains.

Strengths of the paper: 
- Clear Presentation: In general, this paper is well-written and structured. Clear tables and figures effectively communicate core insights, making the results easy to understand.
- Resource Accessibility: The implementation of the proposed framework is publicly available, along with detailed documentation of experimental setups, promoting further research and application in the field.
- Systematic Analysis: The study employs a comprehensive approach to RAG configuration, systematically examining multiple dimensions, models, and interactions, providing valuable empirical evidence about component relationships and particularly regarding the interplay between retriever and reader performance, contributing to a deeper understanding of their dynamics.

Weaknesses of the paper:
- Lack of Comparison with Advanced LLMs (Reviewer L2xh): This paper does not compare SOTA closed-source LLMs (e.g. GPT-4), limiting the applicability of its findings and insights.
- Underexplored Retrieval Methods (Reviewer F1ot): This paper does not incorporate contemporary retrieval techniques, such as BGE and reranking mechanisms, and does not evaluate the impact of modern search engines.
- Inadequate Evaluation Metrics (Reviewer L2xh): The use of Unigram F1 as a metric for reasoning quality is insufficient; exploring semantic embedding-based metrics or using another LLM for verification could enhance evaluation.
- Insufficient Empirical Support (Reviewer mKTy, F1ot, e7Uq): Claims such as "improved retrieval helps more in specialized domains" lack robust empirical evidence, relying on standard datasets (NQ, BioASQ, HotpotQA) that may not provide novel insights or generalizability to broader contexts.
- Limited Depth of Analysis (Reviewer mKTy, F1ot, e7Uq): The current analysis of the paper does not delve deeply enough into significant questions. It lacks sufficient novelty and empirical support necessary to substantiate its nuanced claims.

Reasons for the decision: After considering the rebuttal, I acknowledge that some of the concerns raised by the reviewers have been addressed, as noted in the additional comments from the reviewer discussion. However, two significant issues remain unresolved: insufficient empirical support and limited depth of analysis. Having reviewed the authors' responses, I understand their rationale for including only NQ, BioASQ, and the newly added HotpotQA, which stems from the available annotations in existing datasets. Following discussions with reviewers mKTy and F1ot during the reviewer discussion period, I find myself more aligned with their perspective. Currently, the depth of analysis provided by the paper does not meet the standards for publication as a long research paper at ICLR. An analysis paper at ICLR should provide insights that are valuable not just in the immediate future, but also for years to come. In light of these considerations, I lean toward rejection.