RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems
Authors: Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, Graham Neubig


Reviewer paper summaries: The paper proposes a framework called RAGGED (Retrieval Augmented Generation Generalized Evaluation Device) to investigate how different large language models (LLMs) perform on document-based question answering, considering variations in the quantity and quality of documents.

The authors conduct an in-depth analysis of three different aspects of retrieval-augmented generation: (1) the quantity of retrieved passages, (2) the quality of retrieved passages, and (3) the choice of retriever.
Summary of strengths: 1. The paper performs an analysis on diverse models and various aspects of retrieval-augmented generation.

2. It provides a comprehensive framework to test the performance and trends of reader LLMs from the perspective of retrieval augmented generation.
Summary of weaknesses: 1. The experimental setting to observe the impact of the quantity of context is entangled with the number and position of relevant contexts. Thus, the presented experimental results and analyses are not comprehensive enough to make conclusive claims.
    
    Specifically, when investigating the impact of the quantity of contexts on different LLMs, the number of retrieved contexts varied from 1 to 20. However, a controlled setting that removes the influence of quality and order is necessary to assess the effect of quantity alone. The experiment results and analyses regarding the impact of context quantity are not precise due to the presence of other possible influences. Especially in the case of optimal k* performance, factors such as the quality of the optimal k* or the ordering can lead to different interpretations.
    
2. In Section 6.2, only the FLAN model benefits from non-gold contexts and consistently outperforms its no-context baseline. While the subsequent reasoning explains that better performance might be achieved with non-gold contexts, it does not clarify why this trend is observed only with the FLAN model. A detailed explanation is needed, potentially by linking it with the results of other experiments, to provide a more comprehensive understanding.

3. The BioASQ results in Figure 10 and the NQ results in Figure 4 seem identical.

4. Overall, the paper's main contribution is the analysis framework, but it lacks a clear explanation of how it differs from previous works and how the reported findings relate to those of other studies [1, 2]. Such information should be provided to strengthen the validity of the observations and interpretations for them.

[1] Yoran, O., Wolfson, T., Ram, O., & Berant, J. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. In The Twelfth International Conference on Learning Representations.

[2] Xie, J., Zhang, K., Chen, J., Lou, R., & Su, Y. Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts. In The Twelfth International Conference on Learning Representations.
Comments, suggestions and typos: 1. Several references for corresponding figures or tables are missing. For example, line 465 and line 477 should refer to Figure 5 and Figure 10, respectively. 

2. In Section 7, lines 559-570 analyze performances based on exact values of point gain, but it is difficult to compare the exact differences in retriever usage with the overall performance trends shown in Figures 3 and 8. This section would benefit from performance visualizations that can show score differences more clearly.

3. Why use ColBERT as the semantic similarity-based retriever instead of a more up-to-date retriever like Contriever [1], which performs better than ColBERT?

4. There is insufficient detail in Section 5.4 regarding how the input with k contexts was truncated. Also, detailed analysis  is needed on how the impact of truncation changes when the number of context varies.

**Typos**

1. line 477 & 487 : top-pos → top-gold
2. line 500: hte → the
3. line 516: For BioASQ (Figure 7)

[1] Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., & Grave, E. Unsupervised Dense Information Retrieval with Contrastive Learning. Transactions on Machine Learning Research.
Overall assessment scores: 2.5