Learning to Filter Context for Retrieval-Augmented Generation
Authors: Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, Graham Neubig


Reviewer paper summaries: In Retrieval-Augmented Generation (RAG), LLMs are susceptible to failures to retrieve relevant contexts. To address this issue, FILCO trains a context filtering model that filters out irrelevant texts from the retrieved context. Then, the generative model generates responses based on the context processed with the filtering model and the main query. To utilize the context effectively, FILCO also trains the generation model. FILCO improves the RAG performances in multiple long-form QA tasks.
Summary of strengths: 1. Authors support their findings with experiment results on various tasks, baseline approaches, and models, consistently achieving high performances.

2. Each of the three context filtering measures introduced in this paper is effective in different domains. This finding provides valuable insight for future works on filtering contexts and enhances the practicality of the proposed FILCO method.
Summary of weaknesses: 1. The paper understates the computational cost for FILCO. The cost of the attention mechanism quadratically scales as the text length increases. Hence, the additional generation cost from context model is much larger than the values described in Appendix A, especially when the output text is shorter than the context.

2. The importance of each element in FILCO is not analyzed properly. In L442 - L448, authors selectively describe the meaning of precision while interpreting the impact of filtered contexts. If high precision of filtered context is not an important factor for end generation, the authors should provide a different metric to analyze the contexts. Also, the impact of training the generation model is not shown. Ablation on each trained model should be conducted.

3. In Figure 6, FILCO shows marginal improvements and often shows worse performance than simple baselines. The explanation in section 4.4 does not represent this result appropriately.

4. Several details regarding the experiments are missing. For example, some values are missing in Table 4 without reasoning. Also, the prompt templates for reported experiments are not provided.
Comments, suggestions and typos: 1. Most datasets in this paper have gold answers composed of short phrases (e.g., around 5 tokens in NQ), inherently leading to low numerical values in unigram precision overall. Hence, the unigram-based metric may overlook relevant parts of answers that do not match exactly but contain pertinent information. It would be beneficial to include any reference or more in-depth analysis that support or consider these aspects.

2. SILVER scores are presented as upper bound, but FILCO outperforms them in multiple experiments. What can be the reason behind this result?

3. The arbitrary choice of axis in Figure 4 and 6 is confusing. Please illustrate the results with different visualizations.
Overall assessment scores: 2.5