Title: Influence Guided Context Selection for Effective Retrieval-Augmented Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Retrieval Augmented Generation

Data Selection
Abstract: Retrieval-Augmented Generation (RAG) addresses large language model (LLM) hallucinations by grounding responses in external knowledge, but its effectiveness is compromised by poor-quality retrieved contexts containing irrelevant or noisy information. While existing approaches attempt to improve performance through context selection based on predefined context quality assessment metrics, they show limited gains over standard RAG. We attribute this limitation to their failure in holistically utilizing available information (query, context list, and generator) for comprehensive quality assessment. Inspired by recent advances in data selection, we reconceptualize context quality assessment as an inference-time data valuation problem and introduce the Contextual Influence Value (CI value). This novel metric quantifies context quality by measuring the performance degradation when removing each context from the list, effectively integrating query-aware relevance, list-aware uniqueness, and generator-aware alignment. Moreover, CI value eliminates complex selection hyperparameter tuning by simply retaining contexts with positive CI values. To address practical challenges of label dependency and computational overhead, we develop a parameterized surrogate model for CI value prediction during inference. The model employs a hierarchical architecture that captures both local query-context relevance and global inter-context interactions, trained through oracle CI value supervision and end-to-end generator feedback. Extensive experiments across 8 NLP tasks and multiple LLMs demonstrate that our context selection method significantly outperforms state-of-the-art baselines, effectively filtering poor-quality contexts while preserving critical information. Code is available at https://github.com/SJTU-DMTai/RAG-CSM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose to assess context in RAG via inference-time data valuation, and a contextual influence value(CI) is proposed.  Specifically, a parameterized surrogate model is trained to predict CI values of difference contexts by capturing dependences both  between query-context and inter-contexts, and the proposed method simply retains contexts with positive CI values. Experimental results show that the proposed method can achieve some performance improvement.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of measuring a context via the performance degradation when removing it is reasonable;\n2. CI values can take all query-aware, list-aware, and generator-aware metrics into consideration;\n3. Experimental results show some effectiveness of the proposed method.\n\n Weaknesses:\n1. My main concern about this paper is the mismatch between the motivation of CI value( the performance degradation when removing a specific context) and how it is actually calculated(a surrogate model which simply takes several input and output the CI value). It is hard to say such a model can match the motivation of the CI value, and whether can generalize to unseen scenarios. Therefore, I think the authors should explain why such a surrogate model can actually reflect the CI value, and give detailed analysis between golden CI values and calculated value using several examples. \n2. The current baselines are mainly vanilla RAG + context scoring, the authors should also compared with advanced RAG systems such as agentic RAG systems, which can also take the quality and the importance of contexts into consideration. To make the experimental results more convincing, the authors should also compare with state-of-the-art solutions, rather than only simple baselines.\n\nQUESTIONS:\nPlease see Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new reranking model for context selection for retrieval augmented generation (RAG). Specifically, an auxiliary model is trained to predict how much removing a context from a list of retrieved documents will affect the final performance, i.e. CI or contextual influence value. Two training strategies are introduced, one based on supervised signals derived from labeled data, and one based on generated data. Specific training tricks are proposed to deal with imbalanced distributions of CI values. Experiments show the effectiveness of the CI value-based context selection performance as of the final answer quality.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The idea of how each context contributes to the final answer is meaningful for RAG and is studied by previous research in different forms, as also mentioned in the paper.\n- Experiments clearly show the induced CI value correlates to the answer quality and thus useful as reranking scores\n- The methodology is clearly described with plenty of details. \n\n### Weaknesses\n- The way of using CI scores during inference could be better analyzed. see some questions below\n- It would be better to include some efficiency comparison on some compared methods to show the applicability of the method.\n\nQUESTIONS:\n- Due to the nature of CI values, it is okay and reasonable to select context based on whether the CI value is positive. However, for fairness of comparison, it is better to ensure there are no significantly more contexts being selected. Maybe also report the average number of contexts selected based on CI\n- In figure 3, it clearly shows the performance first increases and then decreases. I thus wonder whether the best performance always corresponds to the zero points of CI values empirically?\n- Iine 258-259, it is mentioned the data splits used by several baselines. However, it is not mentioned whether the proposed method also use the same splits.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes a method for automatic context document selection in the RAG scenario called CSM. Specifically, the author introduces the use of Contextual Influence (CI) values to measure the usefulness of contexts. Two training methods, Supervised training and end-to-end training, are employed to train the CSM model to predict context CI values. During inference, CSM predicts the CI values for all contexts and selects those with CI values greater than 0. The author provides extensive experimental evidence demonstrating the effectiveness of CSM.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The motivation is clear, and CSM efficiently and effectively avoids the manual selection of top-k documents, which is very attractive.\n2. The end-to-end training method for CSM is ingenious and successfully addresses the issue of the context selection process being discrete and non-differentiable.\n3. The author provides the complete code.\n\nWeakness:\n1. The method section may have some potential flaws\n2. The performance on OOD data has not been explored.  \nPlease see the Questions section for details\n\nQUESTIONS:\n1. I am concerned whether the oracle CI scores in certain special cases accurately represent the usefulness of the context. For example, when a user queries information about a hot news topic, the retriever may retrieve several similar contexts. These contexts provide the same important information, and removing one does not affect the utility. According to Definition 1, their CI values will be near 0, but these contexts are still quite important. Does this represent a potential flaw?\n2. In the experiments, is the CSM model the same for different datasets? How does the CSM model perform on out-of-distribution data? This is an important concern.\n3. In the End-to-end Training of CSM section, the authors use a masking method to obtain $H_c$. This could introduce a potential issue: during training, low-scoring context positions in $H_c$ still have placeholders close to zero, but these placeholders do not exist during inference, leading to a gap between training and inference. I understand this is a trade-off between differentiability and efficiency, but I am still curious whether this gap affects performance.\n\nMinor Comments:\n1. The average improvement in Table 1 is quite misleading when presented as a percentage. It is recommended to include the absolute value of the increase as well.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies context selection in Retrieval-Augmented Generation, where Large Language Models use retrieved knowledge to reduce hallucination. The authors identify key limitations in existing context selection methods, which typically rely on query, list, or generator-aware quality metrics, often failing to comprehensively balance these factors and requiring dataset-specific hyperparameter tuning. To address these issues, the paper introduces the Contextual Influence (CI) value, which quantifies each context’s contribution by measuring the drop in performance when it is removed. They further propose a Context Selection Model (CSM) that predicts CI values at inference time without needing access to labels, using hierarchical modeling and either supervised or end-to-end training with generator feedback. Experiments on eight NLP tasks and two LLM backbones show that this approach outperforms a wide range of strong baselines, leading to improved generation quality and robustness.\n\nSTRENGTHS AND WEAKNESSES:\n> Strengths：\n\n1. The CI value offers a holistic context evaluation metric that explicitly combines query relevance, listwise context uniqueness, and generator feedback—overcoming key limitations of previous metrics. \n\n2. The proposed CSM model structurally encodes both local (query-context) and global (context-context) dependencies, a sound design that reflects the nature of the selection problem.\n\n3. The paper presents consistent improvements over a broad range of baselines—including query-aware, list-aware, and generator-aware methods, as well as robust LLM tuning—across various QA tasks. The evaluation is comprehensive, combining diverse benchmarks with both quantitative metrics and qualitative case studies to support the claims.\n\n> Weaknesses：\n\n1.There is little quantitative comparison of inference-time latency or computational resources used by CSM versus standard RAG and other selection approaches, especially as the number of retrieved contexts grows large.\n\n2.While the method elegantly sidesteps the cost of explicit influence computation, details on CSM’s potential to misestimate context dependencies in more adversarial or correlated inference setups are limited.\n\nQUESTIONS:\n1.How would the CI value be adapted if the generation utility (e.g., $v(f(q⊕C))$) is not well-defined by EM/F1 (e.g., creative writing, summarization)?\n\n2.In scenarios where useful information is distributed across several weakly helpful contexts (none with individually positive CI), could the current approach lead to information loss? Are there strategies to mitigate filtering out such complementary but individually weak contexts?\n\n3.How well does the CI surrogate model generalize to new retrieval domains? Would it require retraining for each new task, or is it robust enough to handle domain shifts without additional tuning?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}