Influence Guided Context Selection for Effective Retrieval-Augmented Generation
Authors: Jiale Deng
Yanyan Shen
Ziyuan Pei
Youmin Chen
Linpeng Huang
Venue: NeurIPS 2025 poster ()
Keywords: Retrieval Augmented Generation
Data Selection
Primary area: deep_learning

Abstract: Retrieval-Augmented Generation (RAG) addresses large language model (LLM) hallucinations by grounding responses in external knowledge, but its effectiveness is compromised by poor-quality retrieved contexts containing irrelevant or noisy information. While existing approaches attempt to improve performance through context selection based on predefined context quality assessment metrics, they show limited gains over standard RAG. We attribute this limitation to their failure in holistically utilizing available information (query, context list, and generator) for comprehensive quality assessment. Inspired by recent advances in data selection, we reconceptualize context quality assessment as an inference-time data valuation problem and introduce the Contextual Influence Value (CI value). This novel metric quantifies context quality by measuring the performance degradation when removing each context from the list, effectively integrating query-aware relevance, list-aware uniqueness, and generator-aware alignment. Moreover, CI value eliminates complex selection hyperparameter tuning by simply retaining contexts with positive CI values. To address practical challenges of label dependency and computational overhead, we develop a parameterized surrogate model for CI value prediction during inference. The model employs a hierarchical architecture that captures both local query-context relevance and global inter-context interactions, trained through oracle CI value supervision and end-to-end generator feedback. Extensive experiments across 8 NLP tasks and multiple LLMs demonstrate that our context selection method significantly outperforms state-of-the-art baselines, effectively filtering poor-quality contexts while preserving critical information. Code is available at https://github.com/SJTU-DMTai/RAG-CSM.

Influence Guided Context Selection for Effective Retrieval-Augmented Generation
Jiale Deng, Yanyan Shen', Ziyuan Pei, Youmin Chen, Linpeng Huang
Shanghai Jiao Tong University
(jialedeng, shenyy, pzy_live, chenyoumin, 1phuang] @sjtu. tu. edu.edu.ee cn
Abstract
Oetrieval-Augmente Generation (RAG) addresses large language model (LLM) hallucinations by grounding responses in external knowledge, but its effectiveness is compromised by poor-quality retrieved contexts containing irrelevant noisy information. While existing approaches attempt to improve performance through context selection based predefined context quality assessment metrics, they show limited gains over standard RAG. We attribute this limitation to their failure in holistically utilizing available information (query, context list, and generator) for comprehensive quality assessment. Inspired by recent advances in data se- lection, reconceptualize context quality assessment an inference-ting data valuation problem and introduce the Contextual Influence Value (CI value). This novel metric quantifies context quality by measuring the performance degradation when removing each context from the list, effectively integrating query-aware aware rele- vance, list-aware uniqueness, and generator-awan alignment. Moreover, CI value eliminates complex selection hyperparameter tuning by simply retaining contexts with positive CI values. address practical challenges of label dependency and computational overhead, we develop a parameterized surrogate model for CI value prediction during inference. The model employs hierarchical architecture that captures both local query-context relevance and global inter-context interactions, trained through oracle CI value supervision and end-to-end generator feedback. Extensive experiments across 8 NLP tasks and multiple LLMs demonstrate that our context selection method significantly outperforms statc-of-the- baselines, effectively filtering poor-quality contexts while preserving critical information. Code is available at
com Cow.///TU--TTTAAAA------
Introduction
Setrieval-Augmenti Generation (RAG) has emerged as a powerful approach for mitigating hal- lucinations in large language models (LLMMs) by grounding their responses in external knowledge sources 24 typical RAG pipeline consists of two core components: retriever that searches for query-relevant contexts from external knowledge sources, and LLM generator that produces responses using the retrieved contexts. Despite its advantages, RAG faces important challenges in practical applications. That is, external knowledge sources may contain substantial noisy data, and retrievers based on similarity metrics are inherently imperfect [54]. As a result, retrieved contexts often include irrelevant and noisy information 212 54 57). This issue is particularly problematic as
LLM generators tend to rely heavily on the provided contexts [2 31 38 potentially producing incorrect responses when provided with poor-quality contexts.
Recent work g 20 57 58 proposed context selection shield generators from poor-quality
contexts. This strategy depends on context quality assessment, which assigns quality score to
"corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Query q
Contexts
Generator 10'
"Who is the vice president during Donald Trump's second term of US president?
C1: "Michanl Richard Penoe the 48th president the United States under President Donald Trump."
C2: "Donald Trump inaugurated the 47th
"Pence is the vice president under president Trump." (parameterized, earlier than 2024)
President the US January 20, 2025. He was the 45th president from 2017 2021.
C3: "In 2024, Trump defeated vice president Kamala Harris the election was elected President
Ca= "Vance was elected November
was sworn in the first millenniall Vice Pres ident of United States January 2025."
Context Quality Assessment
C1
C2
C3 Co
Pence
X
high-quolity
query-awore metric
Cg,C3 q
Pence X
list-aware metric
C2,C2 @ 9
Pence X
Context Selection
Generation
generotor-owor oware metric
C2 ②
Pence X
poorquality
contextual influence
C2 @ 4
Vance
Figure 1: An example demonstrating different context quality metrics in practice. C2 and C4 are golden contexts containing erucial information about Trump's 's second presidency and his vice president Vance. Query-awan metries favor C₁ and C3 due to their mentions of "Trump" and "vice president". List- aware metrics score 0 and c4 higher by considering context relationships, but still favor eı Generator- aware metrics assign low scores to ej as it's redundant with LLM knowledge. CI value, by integrating all three dimensions, correctly identifies C2 and c4 as the most informative contexts.
each context to guide the selection process. Without loss of generality, the contexts are evaluated across three complementary dimensions: (1) query-aware metric that measures the semantic
relevance between context and query, implemented through point-wise rerankers S 20 37 41];
(2) list-aware metric that considers relationships among multiple contexts, optimized through to prompt diversity and complementarity in
pair-wise and list-wise rerankers 29
36
[3]
the selected contexts; and (3) generator-aware metric that evaluates how contexts align with the
generator's existing knowledge, using metrics such as log likelihood [5]] or mutual information with generated responses 47 57]. However, our empirical studies in Section 6. demonstrate that these metrics achieve limited effectiveness in context selection, sometimes even reducing RAG performance [19]. As illustrated in Figure I query-aware and list-aware metrics lack generator feedback, potentially selecting contexts that duplicate or contradict generator's knowledge [ 54] Meanwhile, senerator-aware metrics ignore inter-context relationships, risking the omission of critical information. Moreover, selection parameters such as top-k (number of contexts to keep) must be specified in advance [18 (35), with optimal configurations varying significantly across different tasks, making it challenging to achieve ideal selection performance in practice.
Recent progress in training data valuation has shown promising results for selecting high-quality training samples and enhancing ML model performance [7 44]. One key data valuation metric data influence (22 [6). which quantifies a sample's importance by measuring the validation performance decrease when removed from the training set. Inspired by this, we reconceptualize context quality assessment as an inference-time data valuation problem and introduce the Contextual Influence value (CI value) for RAG context selection. Given a query q, context e; E C, and generator f. CI value is defined as di(v) = vg @ C)) - v(S(g @ {C), where @ represents concatenation and v(f()) is a utility function that measures generator output quality (e.g.. EM or Fl scores). CI value naturally satisfies four key desiderata: (1) query-awarenesx: query-irrelevant contexts leads to (v) = 0, indicating that CI value implicitly captures query-context relevance; (2) list- awareness: by measuring list-wise marginal contribution, CI value rewards unique and essential with generator feedback, CI value effectively distinguishes between contexts that enhance generator performance and those that diminish it; (4) ease-of- con-oooocoooaaitiionn instead of requiring task-specific top-k tuning, CI value enables straightforward selection strategy by keeping contexts with di(v) > 0, i.e., those whose removal degrades performance.
information while penalizing redundant content; (3) penneator-awareeeee
However, compared to data influence metrics in training data valuation, CI value computation faces two unique challenges. First, the utility evaluation v(-) depends on access to test labels, which unavailable during inference. While some approaches attempt to estimate utility using model
2

---

confidence, such heuristics often prove unreliable in practice 77 Second, computing exact CI values requires LLM forward passes for a n--ength context list, substantially increasing inference latency. To address these challenges, we propose CI Surrogate Model (CSM) that predicts CI values during inference. The CSM model is trained on the RAG training set and can rapidly assign quality scores to contexts without requiring labels or multiple L.M calls. The approximation effectiveness of CSM depends on both its architecture and training strategy. Specifically, we employ a hierarchical structure that captures both local query-context relevance and global inter-context dependencies. For generator awareness, we explore two training strategies: (1) supervised learning using oracle CI values as targets, which provides implicit generator feedback; and (2) end-to-end training with the generator in the loop, which offers explicit signals about each context's impact.
We validate our framework through comprehensive experiments on 8 real-world NL tasks with 2 LLM backbones. Results demonstrate that our CI value surpasses existing context quality metrics in identifying high-quality contexts and streamlining selection configuration. Moreover, our proposed CSM achieves 15.03% average improvement RAG generation performance over leading baselines.
2 Related Work
Noise Robustness for RAG. RAG systems often encounter poor-quality retrieval results containing irrelevant and noisy information 47 [7). These poor-quality contexts not only distract LLMs [54]
but can also lead to incorrect responses, as LLMs tend to overly trust external information [12] and struggle with the "lost-in-the-middle" problem [27] when processing lengthy contexts. To address
these challenges, recent research has pursued two main approaches. The first approach enhances
model capabilities through supervised fine-tuning [12 47 54] or instruction tuning [48] 55] to
improve LLM noise robustness, implements sophisticated pipelines like self-ask mechanisms 2 to guide LLM self-reflection. However, these solutions face practical limitations: fine-tuning LLMs computationally expensive, and complex pipelines increase inference latency. The second approach
employs external filters to rerank (30, 34, [50] or refine [ 20 47
51
57
58 retrieval results,
shielding generators from poor-quality contexts. These methods utilize LMs LLMs context selection models, training them through supervised [51] or reinforcement learning [57] based on quality metrics derived from prior knowledge, such as query relevance [8] 20 50 , log likelihood (5]],
mutual information [47 (5). and so on. However, as discussed in Section | these quality metrics lacks comprehensive utilization of available information (query, context list, and generator), leading to suboptimal selection performance.
inference-Time Data Valuation. Data valuation metrics quantify each training example's contribu- tion to model performance (e.g., its effect on validation accuracy). which is proven to be effective for data selection tasks that identify high-quality training samples improve model performance. A fundamental approach is Leave-One Onee-OOn--utttt (LOO), which measures the performance degradation when removing a training sample, though it requires expensive retraining. Recent work has improved LOO through two main strategies: (1) influence-basee
methods [4 (5) that approximate LOO by us-
ing gradient and Hessian matrix without full retraining: (2) Shapley value-based methods [3] 17 44
that enhance fairness by modeling complex sample interactions through cooperative game theory. Recently, inference-time data valuation has emerged as new direction of data valuation, focusing on assessing data quality of inference data (7]. However, one cannot directly compute the utility due to the unknown labels during inference. While simple heuristics like model confidence are proven to be unreliable (7. current research trains utility prediction models (UPMs) with regression objec- tives to estimate oracle utility. UPMs have been proven effective in data selection tasks of various domains 7 B3 46 56]. However, UPMs do not directly optimize the predicted data valuation scores against ground truth values, suffering the risk of error accumulation and inaccurate approximation.
3 Preliminaries
Setup for RAG. A typical RAG system consists of a retriever and an LLM generator f. Given a query q. the retriever retrieves list of query-relevant contexts C = (Cr,,.., ). from an external knowledge base. The generator then takes both the query and the retrieved contexts input to generate an answer ý. Formally, this can be expressed as ÿj = f(q @ C)), where @ denotes the combination of the query and the retrieved contexts, typically implemented as a simple concatenation.
3

---

Utility Function. We quantify the effectiveness of retrieved contexts using utility function us 2" ., which maps any subset S C to real-valued score reflecting its usefulness for answering the query. For NLP tasks, the utility function is typically defined by comparing the model's generated output against the ground-truth answers. Concretely, we set:
(f) max - L(,, f(g ① S)),
wEY
where Y is the set of correct answers, and L is the cross entropy loss.
Definition 1 (Contextual Influence) Given query q, retrieved context list C and function the contextual influence value (CI value) for a context c2 E C is defined as:
0.(0) = v(C) (C))e)
CI value quantifies the utility change when G is removed from the context list. Unlike previous metrics, it simultaneously captures three key aspects: query- awarenessss (through q). list-awareness (through interactions within C), and generator-awarenes (through feedback from f). A positive CI value (v) > 0) indicates that removing e, degrades utility (or increases test loss), suggesting that e; positively contributes to generation quality, and vice versa. To ensure fair CI value, we remove semantically duplicate contexts from C before computing CI values.
Context Selection for RAG. It commonly formulated an optimization problem, whose objective is to maximize the utility of the generator based on the choice of retrieved contexts. Specifically, given v, the objective of context selection is to identify subset St C that optimizes:
S = arg max v(S). SCC
However, solving Equation 33) presents significant challenges. The utility function e for the complex LLM generator lacks tractable closed-form expression for analytical optimization. A brute-force approach that simply evaluates the utility of all possible subsets vSS) would necessitate 2n LLM generator forwards, which is computationally infeasible in practice as grows large.
Context Selection via CI value. To avoid the computationally intensive task of enumerating all possible subsets, we adopt more practical approach by decomposing group influence into pointwise influence [TI [56. Following previous research 44 [56], we aggregate contextual influences through
summation: (S)) = Eaes (v).). Therefore, the context selection strategy based on CI values aims to maximize (S)][S] as proxy for optimizing u(S):
Sa(e) arg max ((v)[S]. Sce
(4)
Since ((v)[{]=
= E (v))), So(v) consists of contexts with all positive CI values. That is, when using CI value for context selection, select all contexts with positive CI values.
4
Methodology
4.1 CI Parameterization via Surrogate Model (CSM)
As established, directly computing CI values during inference is infeasible due to label dependency and computational cost, requiring a CI surrogate model (CSM) for approximation. Due to the query-awareness and list-awareness of CI value, CSM should effectively capture both local query-context relevance and global list-level interactions. Inspired by recent advances in wist neural rerankers 26 32 we design CSM with a hierarchical structure (Figure 2 comprising three
components: (1) local layer based on BERT-uncased 10 for query-context pair modeling; (2) a global layer with self-attention; (3) MLP-baaedd output layer. Given q and C {ci,,... C.). CSM first processes each query-conten pair (q, c.) through the local layer to generate local embeddings = (41 ..., 4.) capturing semantic relationships between query and each context. These embeddings are then mean-pood proaee and fed into the global layer, where multi-head self-attention computes cross-
context interactions to produce global embeddings = (91, ... }. Finally, the output layer maps these global embeddings into relevance scores M = {m1, ... mn).
We omit the subscripts and q for simplicity.

---

(b) Supervised Training (ST)
Query
Quality scores
Oracle CI values
CSM
ST St mse + BLss + (m
Contexts
M [m mc
c (c. 6n)
(a) CSM Architecture
Soft Context Selection
(c) End-to-end Training (E2E)
Local Layer
Global Layer
Output Layer
Golden Answers
(SEP) 4
f1
m
Generator
Prediction
E2E Loss
L Leze Sut + nec
[SEP] 62
: Trainable
[SEP] On
9m
71g
Frozen
Figure 2: Overview of our proposed CSM: (a) CSM model architecture, (b) supervised training paradigm and (c) end-to-end training paradigm.
To effectively approximate CI value using CSM, then introduce two training paradigms establish generator awareness of CSM: (1) supervised training that implicitly encodes generator feedback through oracle CI values; (2) end-to-end training that directly propagates generator gradients through differentiable soft context selection.
4.2 Supervised Training of CSM
Supervised training establishes intrinsically generator awareness through CI value supervision. We create a training dataset D = {di.... d } by collecting oracle CI values for all query-context pairs in RAG training dataset, where each sample d4 (g., Y,, C, F.) contains query qs the answer set Y, contexts C {ci ...., Ci,, J. their oracle CI values 4; {o., (v), .... di,, (v) Following data 46 56]. we frame CSM training as a supervised regression task. However, as shown in Apendix A the CI value distribution is severely imbalanced, with approximately 80% of contexts having near-zero CI values, while samples with high-CI and low-CI contexts are very rare (16%). This significant imbalance makes CSM training partieularly challenging, which we address through both data and loss perspectives.
valuation research 7
From the perspective of data, employ combination of down sampling and data interventions. We first define the rarity rate of sample d as r; := ((Φ;) + a (.)), where ju(-) and a()) represent the mean and standard deviation, respectively, and a is balancing coefficient. Using two thresholds
61 < d2. we categorize samples into two distinct groups: (1) Trivial samples De (Tt. < 8, for de, E D2). which predominantly contain non non-informative contexts; (2) Hard samples Dr (rA, 82 for da, E Di),, which contain contexts with high-CI or low-CI contexts. For the majority trivial samples, apply down sampling reduce their dominance in the dataset. For hard samples, implement cross- instanen intervention to balance the data distribution by increasing the number of samples with both high-CI CI and low-CI contexts. Due to space limitations, we focus describing the intervention for constructing samples with high-CI contexts; the analogous process for samples with low-CI contexts hard samples with high-CI contexts, denoted as dn, = (4i, C. (C,, C:NY +b),, where C.P
be found in Appendix B The intervention process begins by collecting
{ci, 10 (v) > } represents high-CI contexts (7 > 0) and C.N CAC... We then sample another instance d, = Cy, ØD)) whose query q; is semantically distinct from q. Following the recombination approach [49], we create new sample by:
dn, = (g:, Y,, (CP UCHH,
(5)
where Č₁ is sampled from Cj. This intervention strategy is based the intuition that when informative contexts (CF) are placed noisier environments (composed of č₁ that are irrelevant to (s), their marginal contribution to the context list becomes more pronounced [40], resulting in elevated CI values for C.. Through this process, we effectively construct additional samples with high-CI contexts, thereby enriching the training set with more informative samples.
5

---

From the perspective of training, we implement a dual-loss strategy combining reweighted re- gression and contrastive learning. First, we mitigate majority class dominance through importance weighting:
Luse Ed,eD -0,)2/(/ /p)))p(()).
(6)
where p(i) represents the empirical frequency of r; in the training distribution. For hard samples, employ a contrastive loss term to enhance their discriminative signal:
exp(ge 9e /T)
Lats = -EI,,,,,,,))ED Beer
log
EC exp(0e ge- /T) + explge Ic+ 9
(7)
Ee
where anchor с is a context from hard samples, positive context et shares similar CI value with e (loe(u) - Oct (v) < 61), negative contexts C' have divergent CI values from (loc(() - de- (v) >
E2), and r is a temperature hyperparameter. The final supervised training loss is the linear combination
controlled by a hyperparameter 3: S.t Luse + BLets.
4.3 End-to-end Training of CSM
End-to-end training explicitly injects generator awareness by directly using the generator's output as a signal to optimize CSM's parameters. We denote the training set of end-to-end training by & { Yi, C.)). A typical end-to-end training paradigm requires computing values for each context, selecting contexts S {c)lo<()) 0} based these values, and feeding the selected contexts to the generator for final answer generation. However, this context selection process is discrete and non-differentiable. To address this, we treat CSM's output M as a mask and implement soft context selection during training by masking the generator's 's input tokens with M. We employ the
trick g 28 40 to approximate binary mask, i.e., MM = Gumbel(M), The generator's 's masked input is then reconstructed as: = H, i He, where H,- = Sook (4) represents the tokenized query and H. = M @ Frou(C) denotes the masked tokenized context, with frok(.) being the generator f's tokenizer. Additionally, we construct the complementary masked tokenized context as He (1 M) @ fCo)(C), which effectively removes high-value contexts before tokenization. Then, to align mask values with CI value, we design following loss terms:
Lau = -Big,,,cJeslY" log(S(H, @ Hc),
(8)
Dnoc =
[KL(tmit,, f(H, @ H.),,
(9)
where KT.(.) denotes the KI--Divergence and Yumif represents the uniform distribution. Lauf encour- ages the generator produce accurate responses when high-value contexts are provided, ensuring that these selected contexts contain sufficient information for answer correctly. Mcanwhile, Lee penalizes the CSM if the generator still generates correct answers even after the removal of high-value contexts, thereby discouraging false positive selections in the context selection process and ensuring that the high-value contexts are necessary for generating correct answers. The overall end-to-end training loss effectively approximates the CI value by combining both sufficiency and necessity through a linear combination controlled by hyperparameter X: Leze Lauf + XLnec-
Experimental Setup
Tasks and Datasets. We consider the following Knowledge-intensive NLP tasks: (1) Open-Domain QA, including NQ 23) TriviaQA (1] and WebQA 33 (2) Multihop QA that requires multi-step
reasoning to generate answers, including HotpotQA 52 and 2WikiMuliiHopQA [1)) (3) Fact
Checking dataset FEVER [43] that challenges the model to use complex reasoning to determine the
factual accuracy of given claims. (4) Multiple Choice dataset TruthfulQA (25). (5) Long-Form QA dataset ASQA we report Exact Match (EM) for Open- Open-Domain QA datasets, Fl for Multihop QA and Long-Form QA datasets, and Accuracy for Fact Checking and Multiple Choice datasets.
39] that generating long and abstract answers given the question. Following [7]47.
Baselines. We consider the following baselines: (1) Vanilla LLM: LLM without retrieval aug-
mentation. (2) Standard RAG: a sequential RAG pipeline using FlashRAG [9 with retrieval (preserving all retrieved contexts). (3) bge-reranker [50]: a context selection baseline based on query-aware quality metrics. It leverages cross-encoder to perform point-wise context reranking.
(4) RankGPT 41 context selection baseline based on list-aware quality. It is list-wise reranker

---

Dataset: NQ
Dataset: TriviaQA
Dataset: WebQA
Dataset: HolpotQA
0.40
0.65
0.24
0.40
0.50
02 0.35
0.30
0.55
0.20
age Sranker Na RECOMP-Ox value Ra Random
0.25
0.18
0.30
0.50
0.16
contexts selected
contexts selected
of contexts selected
of contexts selected
(a) Context selection by adding high-quality contexts, WebQA
Dataset: TriviaQA
Dataset: HotpotQA
0.35
0.600
0.18
0.40
0.30
0.575
0.550
0.26
0.35
0.25
0.525
0.34
0.30
age
0.20
0.500
0.475
0.12
0.25
0.15
Random
0.450
10
contaxts selected
contexts selected
contexts selected
contexts selected
(b) Context selection by adding quar-quality contexts.
Figure 3: RAG generation performance when selecting contexts using different quality metrics. (a) Selecting high-quality contexts, where higher curve indicates better metric. (b) Selecting poor-quality contexts, where lower curve indicates better metrie. Dashed line marks the top-k cutoff where the average CI value is zero and star marks the top-k yielding best performance. For baselines, we use predicted scores for bge-reranker and RankGPT, and oracle log likelihood scores for RECOMP-eX.
that feeds the whole list into L.M and generates a ordered context list based on their relevance to query. We use the generator LLM the lit-wise reranker in our experiments. (5) RECOMP-ex [5]]: a context selection baseline based on generator-aware quality metrics. Given query q. context c, and ground truth answer y. the quality score of is log Py (yllg + q.). It employs the oracle scores train a BERT-based context selection model with contrastive learning. (6) RECOMP-abs [11: context summarization baseline by distilling lightweight abstractive compressor from extreme-scale teacher LLMs like GPT-3.5. (7) Ret-Robust [54] a generator enhancing baseline by fine-tuning the LLM via LoRA [16] to be robust to external noise. (8) Self-RAG [2]: agentic baseline that performs on-demand retrieval and learns to reflect on retrieved contexts while critiquing generated answers. (9) RQ-RAG [5]: agentic baseline that enhances the RAG pipeline through explicit rewriting, decomposition, and disambiguation. (10) oracle CI value: context selection based on oracle CI value, and (11) CSM-st and CSM-e2e: the CI Surrogate Model for context selection, trained by Cat and Leze, respectively.
Implementation Details. To demonstrate the versatility of our method, choose two backbones differing in architecture: Llama3-8b-intruce [ and Dowen..5-7b-instrul 42]. For the retrieval corpus, utilize the Wikipedia dump from December 2018, and pre-process it into chunks (100 words per chunk). We use E5-base-v2 [45] as the dense retriever and retrieve the top 10 chunks from all Wikipedia chunks. We follow the FlashRAG benchmark (19 for data, splits and baselines (including Vanilla I. LLM, Standard RAG, bge-reranker and RECOMP). We conducted all the experiments on server equipped with Montage Jintide(R) C6226R CPU, 256GB Memory, and 4 Nvidia GeForce RTX 4090 GPUs. Detailed setup of our method can be found in Appendix C33
Experimental Results
6.1 Effectiveness of CI value
This part of experiments aims to prove that value serves as an effective metric for context selection, as it climinates the need for complex top-k configuration and directly correlates with RAG generation
7

---

Table 1: RAG generation performance (%) on downstream tasks with different baselines. The best results are in bold and the second best are with underscore. The oracle CI values with asterisk superscript act as a performance reference of our proposed CSM.
TriviaQA WebQA
HolpotQA
2Wiki Multihop QA
FEVER Fact Check. (Acc)
TruthfulQA Mudtiple Choice (Acc)
ASQA Long-Form QA
Open- Domain QA (EM)
Task rype
(FI)
(FI)
Llama3-8B
Vanilla LLM
20.58
52.87
16.39
24.27
23.51
71.52
30.11
31.34
Standard RAG
37.01
62.36
18.21
40.95
24.38
90.76
27.05
34,70
bge-reranker
39.06
64.17
18.85
41.96
25.92
90.57
28.64
33.97
RankGPT 38.61
61.83
19.24
41.53
27.26
78.58
30.11
35.21
RECOMP-er
29.86
60.67
18.36
39.06
24.55
RECOMP-abs
52.85
58.77
18.70
39.94
25.57
90.66
30,60
34.02
Ret-robust
41.77
65.83
19.76
45.69
25.91
90.69
27.05
34,50
Self-RAG
36.23
38.26
21.83
29.98
25.43
85.77
29.75
32.56
RQ-RAG
34.27
55.31
26.12
35.22
26.08
90.13
27.36
33.64
CSM-st
42.53
24.77
47.53
25.97
30,97
34.75
CSM-e2e 41.61
67.88
26.05
45.61
26.48
90,74
30.56
33.68
onaele val.
45.79*
71.98"
27.81
48.28*
30.72
94.58'
32.09*
35.70"
wen2.55-7B
Vanilla LLM 14.88
41.75
16.54
26.48
29.44
79.61
30.21
Standard RAG
38.50
63.29
21.51
44.81
33.68
91.14
23.26
33.46
bge-rerunker
39.53
63.76
21.70
45.59
34.48
91.05
25.34
33.00
RankGPT 39.14
63.07
21.21
45.70
35.81
90.20
25.83
34.98
RECOMP-ex
35.76
61.08
20.28
42.04
32.47
RECOMP-ate
59.26
32,07
34,70
Ret-robust
42.77
64.65
28.52
44.20
36.98
91.10
23.25
33.57
Self-RAG
44.93
63,29
28.48
45.69
43.27
90.05
27.17
33.28
RQ-RAC
49.62
34.59
CSM-st
47.38
65.26
28.54
$1.95
48.67
92.98
28.77
34.05
CSM-e2e
46.19
66.35
26.38
49.44
47.23
91.02
27.72
35,62
oraele val.
50.64*
69.19"
30.61"
53.78"
49.08'
94.63*
28.56'
36.72"
performance:
selecting high-CI contexts improves the performance while low-CI contexts degrade Case studies and detailed experiments other datasets provided in Appendix D and
Top-k Configuration. As illustrated in Figure 3a. leveraging CI value as context selection metric eliminates the need for top-k tuning, as simply preserving contexts with positive CI values consistently contrast, other metrics require dataset-specific
achieves optimal near-optimal RAG performance. top-k configurations, with optimal values varying significantly across datasets (e.g., bge- reranker's 's optimal top-k ranges from for NQ to 10 for TriviaQA). This dataset-dependent variation makes it challenging to determine universal top-k value that performs well across different datasets, highlighting the practical advantage of value in context selection.
Contexts. The high-quality selection experiment is performed with the following steps [17]: For each context quality metric, we preserve candidate context set S. = Ø initially. We select contexts from the retrieved contexts in descending order of the quality scores and add them to S. Each time the contexts are selected, we leverage current S as the reference of generator to answer user queries, evaluate the answer quality using metrics in Section and plot the performance curve. In an ideal scenario, selecting the most helpful contexts first should produce sharp initial performance increase, followed by a decline lower-quality RAG performance curves from this experiment,
Context Selection by Adding
contexts are included in Figure Bal illlsstrass with higher curves indicating superior quality metrics. All metrics outperform the random baseline, confirming their effectiveness in identifying high-quality contexts. CI value-based based selection shows consistent pattern aligning with the ideal pattern across datasets, while other baselines exhibit fluctuating performance as grows.
Context Selection by Adding Poor-Quality Contexts. We conduct poor-quality context selection experiments following a similar procedure high-quality selection, but with contexts added in ascending order of their quality scores. In contrast to high-quality selection, the ideal performance curve should initially decline sharply as poor-quality contexts are incorporated into S, then gradually improve as higher-quality contexts are added. Figure illustrats the RAG performance curves from this experiment, with lower curves indicating superior quality metries. metries outperform the random baseline, confirming their effectiveness in identifying poor-quality contexts. Experimental

---

P1
Llama3-88
Qwen2.55-7B
0.85
0.80
0.75
0.70
NQ
Trivia QA
Web
Hotpot QA4
2WIRi
FEVER
Truthful ASQA QA
Trivio
Web QA
Hotpot
2Wiki
Truthful ASQA
CSM-st
CSM-e2e
Figure 4: The Spearman correlation (p) of CSM's predictions with the oracle CI values.
results indicate that CI value's effectiveness in identifying poor-quality contexts, since its performance curves are consistently and significantly lower than other baselines.
6.2 Effectiveness of CI Surrogate Models
These experiments evaluate CSM's effectiveness in improving RAG performance through context selection, its ability to approximate oracle CI values, and the contributions of different CSM modules.
Overall Generation Performance. Table presents a comprehensive comparison of RAG generation performance across different baselines. For fair comparison between context selection methods, we set top-k 5. Our proposed CSM demonstrates significant improvements across nearly all tasks: both CSM-st and CSM-e2e achieve the best second-best results on all eight tasks. In Open-Domain QA, CSM outperforms all baselines by 21.72% in EM, and it achieves a 19.40% Fl improvement in Multihop QA. highlighting the crucial role of high-quality contexts in generating correct answers. Although improvements on TruthfulQA and ASQA are more modest, CSM still ranks first or second these tasks. The bge-reranker performs well on simple QA but struggles in complex scenarios (e.g.. Multihop QA and Long-Foud Form QA), while wise RankGPT shows better performance than bge- reranker in these challenging settings, emphasizing the importance of modeling context interactions. RECOMP-cr 's performance is sometimes even inferior (c.g.. 1.7% lower on ASQA), revealing the limitations of relying solely generator feedback for context selection. Ret-Robust emerges strong baseline by enhancing generator at the cost of expensive LLM fine-tuning. It is worth noting that CSM outperforms advanced agentic baselines (e.g., Self-RAG, RQ-RAG) most tasks. This demonstrates that filtering out low-quality contexts yields greater benefits compared to sophisticated agentic approaches that rely on complex reflection and planning mechanisms.
Approximation Effectiveness. We evaluate CSM's effectiveness in approximating oracle CI values using Spearman rank correlation, which measures the strength of monotonic relationships through the Pearson correlation of ranked values (ranging from -1 to with indicating perfect positive correlation). As shown in Figure 4 both CSM-st and CSM-e2e consistently achieve correlation coefficients above 0.75 across all tasks for both Llama3-8B and Qwen2.5-7B models, demonstrating strong alignment between CSM's predictions and oracle CI values.
Ablation Studies. We conduct ablation stud- ies on both CSM-st and CSM-e2e variants to validate our training strategies, and present the results in Table 2 For CSM-st, removing data intervention causes an average performance drop of 11.98%|, highlighting the importance of cross-instaned intervention creating bal- anced training samples. The removal of trastive loss leads to an average 8.04%. de- w/o Cuat
Table 2: Ablations on training strategy of CSM.
NQ
TriviaQA
WebQA
HolpotQQ
CSM-st
42.53
69.59
24.77
47.53
w/o interv.
37.67
62.91
19.23
45,38
w/o Let.
40.82
65.39
20.19
45.82
CSM-e2e 41.61
67.88
26.05
45.61
35.44
63.39
22.79
w/o
36.85
62.74
21.35
42.57
crease, demonstrating its effectiveness in hancing supervision for hard samples. For CSM-
e2e, ablating either Lsuf or Luec results in significant performance drops (10.28%) and 10.93%]
respectively), showing their complementary roles in guiding CSM to retain high-quality contexts while filtering out poor-quality ones.

---

Inference-time Latency. The efficiency of RAG systems primarily depends on the pa- rameter size of the context selection model when keeping the retriever and generator presents CUDA times (from context selection to generation) different context selection baselines w.r.t the number of retrieved contexts. Since CSM selects all positive contexts, for fair comparison, we set the number of pre- served contexts (i.e., top-k) for the baselines equal to the average number of contexts with positive CI
Table Inference-tinee time latency (ms) comparison be- tween CSM and basclines.
NQ
TriviaQA
HotpotQA
the same. Table R]
n=10
1=50
1=10
1=:50
1==10
1=:00
Standard RAG
320
811
252
810
261
814
RankGPT
874
1437
779
1561
741
1640
RECOMP-abs
299
662
254
721
202
924
CSM
253
481
192
206
values (i.e., Aipos). For n = 10, we set top-k = kpps = 5, and for n = 50, we set top-k = kpos = 23. Compared to baseline methods, CSM achieves significantly lower inference latency by leveraging the lightweight model architecture detailed in Figure This efficiency advantage becomes increasingly pronounced as the number of retrieved contexts increases from 10 to 50. For example, on the NQ dataset with n = 50, CSM reduces latency to 481 ms, far outperforming RankGPT (1437 ms) and RECOMP-abs (662 ms) under the same condition. This result clearly demonstrates that CSM scales more efficiently with the growth of n, maintaining low latency even when handling a larger volume of retrieved contexts.
7 Conclusion
This paper introduces the Contextual Influence (CI) value, a novel metric for selecting high-quality contexts that enhance RAG performance. The CI value improves upon existing metrics by simultane- ously possessing four desirable properties, i.c., query awarens list-awareness, generator-awareness and case-of-comiiuuratinn. We propose parameterized surrogate model (CSM) predict CI values during inference. To ensure high prediction accuracy. CSM features a hierarchical architecture that evaluates both query-context relevance and interactions between different contexts. We explore two approaches to optimizing CSM, i.e., supervised learning using oracle CI values and end-to-end training incorporating generator feedback. Empirical studies across 8 NLP tasks and L.MM back- bones demonstrate that the CI value effectively distinguishes high-quality contexts from lower-quality ones, and our proposed CSM outperforms context selection baselines with an average RAG gener- ation performance improvement of 15.0 15.03%. While CSM is effective and lightweight, its training remains challenging and currently requires speciicii optimization. Future research should focus on developing universal context selector capable of generalizing across different tasks.
8 Acknowledgements
This work is supported by the National Key Research and Development Program of China (No. 00245534505203, National Natural Science Foundation of China (No. 62522211, No. 62202255), and Key Research and Development Program of Xinjiang Uygur Autonomous Region (Grant No. 2023B01027, 2023B01027-I).
References
[1] Al@Meta. Llama 3 model card. 2024.
[2] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023.
[3] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-ansvnn pairs. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1533-1544, 2013.
[4] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Improving language models by retrieving from trillions of tokens. International conference machine learning, pages 2206-2240 PMLR, 2022.
10

---

[5] Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo. Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.006/1 2024.
[6] Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. what data benefits my classifier?" enhancing model performance and interpretability through influence-based data selection. In The Twelfth International Conference on Learning Representations, 2024.
[7] Hongliang Chi, Qiong Wu, Zhengyi Zhou, and Yao Ma. Shapley-guided utility learning for effective graph inference data valuation. The Thirteenth International Conference on Learning Representations, 2025.
[8] Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, and Stéphane CLINCHANT. Provence: efficient and robust context pruning for retrieval-augmented generation. In The Thirteenth International Conference on Learning Representations, 2025.
[9] Jiale Deng and Yanyan Shen. Self-interpretable graph learning with sufficient and necessary explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11749-11756. 2024.
[10] Jacob Devlin, Ming- -ee Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1 100.00800 2018.
[11] Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. International Conference on Machine Learning, pages 12491-12526. PMLR, 2024.
[12] Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. Enhancing noise robustness retrieval-ane -ugreete- language models with adaptive adversarial training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10028-10039, 2024.
[13] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation data for machine learning. In International conference on machine learning, pages 2242-2251. PMLR, 2019.
[14] Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. Machine Learning, 113(5)235112403, 2024.
[15] Xanh Ho, Anh-Khos Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-66255 2020.
[16] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[17] Kevin Jiang, Weixin Liang James Y Zou, and Yongchan Kwon. Opendataval: unified benchmark for data valuation. Advances Neural Information Processing Systems, 36, 2023,
[18] Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context LLMs meet RAG: Overcoming challenges for long inputs in RAG. In The Thirteenth International Conference Learning Representations, 2025.
[19] Jajji Jin, Yutao Zhu, Guanting Dong, Yuyao Zhang, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, Zhicheng Dou, and Ji-Rong Wen. Flashrag: modular toolkit for efficient retrieval-augmentted generation research. arXiv preprint 24X.v:240S.13576 2024.
[20] Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. Sufficient context: new lens on retrieval augmented generation systems. The Thirteenth International Conference on Learning Representations, 2025.
[21] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint 17Xiv:170.003551,00
2017. 2017.
11

---

[22] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[23] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
[24] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Setrieval-augmente generation for mhowledge-intensse nlp tasks. Advances in neural information processing systems, 33:9459 9474, 2020.
[25] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falschoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 3214-3252, 2022.
[26] Junlong Liu, Yue Ma, Ruihui Zhao, Junhao Zheng, Qianli Ma. and Yangyang Kang. Listcon- ranker: A contrastive text reranker with listwise encoding, arXiv preprint arXiv:2501.07111 2025.
[27] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association, for Computational Linguistics, 12:157-173, 2024.
(28) Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances in neural information processing systems, 33: 19620-1999,,,, 2020.
[29] Jian Luo, Xuanang Chen, Ben He, and Le Sun. Prp-graph: Pairwise ranking prompting to llms with graph aggregation for effective text re-ranking. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5766-5776, 2024.
[30] Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-shot listwise document reranking with large language model. arXiv preprint arXiv:2305. 00..305..116... 2023.
[31] Ruotong Pan, Boxi Cao, Hongyu Lin, Xianpei Han, Jia Zheng, Sirui Wang, Xunliang Cai, and Le Sun. Not all contexts equal: Teaching llms credibility awaare generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19844- 19863, 2024.
[32] Changhua Pei, Yi Zhang, Yongfeng Zhang, Fei Sun, Xiao Lin, Hanxiao Sun, Jian Wu, Peng Jiang, Junfeng Ge. Wenwu Ou, et al. Personalized re-ranking for recommendation. In Proceed- ings of the 13th ACM conference on recommender systems, pages 3-11, 2019.
[33] Kieu Thao Nguyen Pham, Rachael Hwee Ling Sim, Quoc Phong Nguyen, See Kiong Ng, and Bryan Kian Hsiang Low. Dupre: Data utility prediction for efficient data valuation. arXiv preprint 27X1v:2502.16152, 2025.
[34] Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, et al. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1504-1518, 2024.
[35] Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Ganesh Ananthanarayanan, Ravi Netravali, and Junchen Jiang. Ragserve: Fast quality-aware rag systems with configuration adaptation. arXiv preprint arXiv:2412.11044 2024.
[36] Ruiyang Ren, Yuhao Wang, Kun Zhou, Wayne Xin Zhao, Wenjie Wang, Jing Liu, Ji-Rong Wen, and Tat-Seng Chua. Self-calibrated listwise reranking with large language models. In Proceedings of the ACM on Web Conference 2025, pages 3692-3701, 2025.
12

---

[37] Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3781-3797, 2022.
[38] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210-31227. PMLR, 2023.
[39] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Was Wei Chang. Asqa: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273-8288, 2022.
[40] Yongduo Sui, Xiang Wang. Jiancan Wu, Min Lin, Xiangnan Hc, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 1696-1705, 2022.
[41] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjic Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14918- 149377,,7, 2023.
[42]
Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[43] James Thome, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguisties, 2018.
[44] Jiachen T Wang, Tianji Yang. James Zou, Yongchan Kwon, and Ruoxi Jia. Rethinking data shapley for data selection tasks: Misleads and merits. In International Conference on Machine Learning, pages 52033-52063. PMLR, 2024.
[45] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
[46] Tianhao Wang, Yu Yang, and Ruoxi Jia. Improving cooperative game theory-based data valuation via data utility learning. arXiv preprint arXiv:2107.06336. 2021.
[47] Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to filter context for aarrival-auggeentee generation. arXiv preprint arXiv:2311.08377 2023.
[48] Zhepei Wei, Wei-Lin Chen, and Yu Meng. InstructRAG: Instructing retrieval-augmented generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representattons, 2025.
[49] Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. In International Conference on Learning Representations, 2022.
[50] Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C- pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641-649, 2024.
[51] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval- Ims with compression and selective augmentation. In /2th International Conference on Learning Representations, ICLR 2024, 2024.
[52] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. 2018.
13

---

[53] Soyoung Yoon, Eunbi Choi, Jiyeon Kim, Hyeongu Yun, Yireun Kim, and Seung won Hwang. Listt5: Listwise reranking with fusion-in-decoder improves shoo-shot retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2287-2308, 2024.
[54] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making aarieval-augmentt language models robust irrelevant context. In The Twelfth International Conference Learning Representations, 2024.
[55] Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retrieval-augmeented generation in llms. Advances in Neural Information Processing Systems, 37: 37:121156-122 11111121111111 12814,44, 2024.
[56] Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems, 37:108735-108755 2024.
[57] Kun Zhu, Xiaocheng Feng, Xiyuan Du, Yuxuan Gu, Weijiang Yu, Haotian Wang, Qianglong Chen, Zheng Chu, Jingchang Chen, and Bing Qin. An information bottleneck perspective for effective noise filtering retrieval-augmentte generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1044-1069, 2024.
[58] Yun Zhu, Jia-Chen Gu, Caitlin Sikora, Ho Ko, Yınxiao Liu, Chu- Chu-Cheng Lin, Lei Shu, Liangchen Luo, Lei Meng. Bang Liu, and Jindong Chen. Accelerating inference of generation via sparse context selection. In The Thirleenth International Conference Learning Representations, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 7 and supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: in Section 5 and supplementary materials we provide full details about our implementation and training process.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: all the models and data we use are publicly available and we carefully cite each paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/ppl cc/public/guu hetggudddsse/eebbbiniinn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://uurr...cc.o........ccttta
Answer: [Yes]
Justification: reviewed and confirmed.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: in Section 1
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: all the data and model we use is publicly available.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: reviewed and confirmed.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: n/a
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: n/a
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: n/a
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: n/a
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Distribution Analysis for CI Value
We first analyze the distribution of CI values. For given context, the larger the absolute value of its CI value, the greater its impact on RAG performance (both positive and negative impacts). Conversely, when the absolute value approaches zero, it indicates that the context has minimal influence on RAG performance. Figure 5 iillustrases the CI value distribution across different datasets, where the x-axis represents the scaled CI value (we scale CI values into the range of [-1, without changing their relative ranking) and the y-axis shows the number of contexts. The CI value distribution exhibits severe imbalance, with the majority of contexts having near-zero CI values. Taking NQ an example, 77.94% of contexts have absolute CI values lower than 0.1. indicating that a substantial portion of contexts contribute little RAG generation performance. These contexts are likely query-irrelevam or redundant with the generator's 's parameter knowledge. In NQ, only 3.40% of contexts have absolute CI values higher than 0.3, suggesting that very few contexts significantly influence RAG performance, either positively or negatively. However, these contexts are precisely the key ones that we need to either select or eliminate to optimize RAG performance.
distribution on TrivisQA
20000
15900
20000
Coart
10000
5000
0.0
1.0
(a)
(b)
value distribution on WebQA
CI value distributian on HetpotQA
20000
15000
1000
Count
Count
10000
500
5000
0.0 -10
-35
-1.0
-0.5
0.0
1.0
value
(c)
(d)
Figure 5: CI value distribution on different datasets, with Llama3-8P as generator.
We then analyze the distribution of rarity rates for supervised training samples in Figure 6 The rarity rate of sample is defined as r, := f( + ((,), where uu(-) and (-)) represent the mean and standard deviation, respectively, and a is balancing coefficient set to 10. To categorize the training samples, we employ two thresholds ô, and 82 (where 8, < 82). dividing them into two distinct sets: the trivia sample set D, and the hard sample set Dr. Specifically, any sample de, E D, satisfies Tt, < 81, while any sample d,, E DD, satisfies Th, > 82. our experiments, we set 61 = 6g = 5. Hard samples are characterized by containing contexts with relatively high or low CI values within their corresponding context lists. However, the proportion of hard samples is notably small. Taking NQ as example, only | = 15.61 % of the samples fall into the hard sample category, indicating that small fraction of samples contain contexts that are either highly beneficial or detrimental to RAG generation performance.
22

---

distribution
Rarity distribution TriviQAA
-1000
Court of surple
1500
Emur ol semples
2000
20
Barily
(a)
(b)
Karity rate distribution on WebQA
Rarity rate distribution on HotpettjA.
150
2000
Coart of samplex
100
Coart samples
1.50
500
0-
15
Rarliy rate
Rarity
(c)
(d)
Figure Rarity rate distribution on different datasets, with Llama3-8B as generator.
In summary, through our analysis of value distribution in real-world data, we observe that only a small fraction of contexts exhibit either high low CI values, and these contexts are concentrated in a limited number of training samples. However, these underrepresented contexts are precisely the ones whose patterns need to learn, they represent the contexts that significantly influence RAG performance. The severe imbalance in the data distribution poses significant challenges for our CSM's generalization capabilities within the supervised learning paradigm. To tackle this issue, introduce a novel solution that simultaneously addresses the imbalance problem from both data and loss perspective.
B Data Intervention
In this section, we present our approach for performing data intervention to increase the number of hard samples containing low-CI contexts. We begin by collecting hard samples with high-CI CI contexts, denoted as d,, = (q,, C; = {CP C. db),, where C,P² = {ca, lé., (v) > 7 represents the set of high-CI contexts (7 > 0) and C... = C/ represents the remaining contexts. Next, we sample another instance d, = (qp, Cj, 4 whose query q semantically distinct from qi. We then construct a new sample through the following intervention:
dny = (g;, Y,, (C, UC)),
(10)
where Č₁ is subset sampled from Cj. Since contexts in CiP are considered positively relevant to query ys, and given that qj is semantically distinct from gt, these contexts should be considered irrelevant for qj and their corresponding CI values should be low. This intervention process effectively generates additional samples containing low-CI contexts, thereby enriching our training set with more informative samples that better represent the challenging cases encountered in real-world scenarios.
23

---

Table Detailed data statistics for CSM training.
Task
Dataset Name
#train
#Dev
#Test
NQ
79,168
8,757
3,610
Open-Domain QA
TriviaQA
78,785
8,837
11,313
WebQA
3,022
756
2,032
HotpotQA
72,357
18,090
7,405
Multihop QA
2Wiki
12,000
3,000
12,576
Fact Checking
FEVER
83,972
20,994
10,444
Multiple Choice
TruthfulQA
327
82
408
Long-Form QA
ASQA
3,482
871
948
Table 5: Comparison between different RAG context selection methods from their design principles.
Method
Query-awareness List-awareness
Generalor-awaremn Ese-of-comfiguration
bge-reranker
X
RankGPT
X
RECOMP-eX
RECOMP-abs
X
CSM
C
Detailed Experimental Setup
C.1
Dataset Details
T the dataset statisties, which are publicly available from [19] For datasets without provided test set, we utilize the development set as the test set and perform a split on the training set, allocating 80% training set and 20% as dev set. Note that for experiments Section 6.1 and Appendix D we evaluate the context selection experiments (adding high/poor-quality contexts) on the first 1000 test samples. As demonstrated in [19, the baseline performance on this subset closely mirrors the performance on the complete test set.
C.2 Baseline Setup
In Table we present a detailed comparison of various context selection methods, emphasizing their real-world applicability. This comparison focuses on four key aspects: (I) Query-awareness: whether the method incorporates query-context relevance in measuring context quality. (2) List-awareness:: whether the method considers the context list information in measuring context quality. (3) Generator- awareness: whether the method takes into account generator feedback in measuring context quality. (4) Ease-of-configuratio whether the method eliminates the need for tuning the hyperparameter top-k across different tasks. These aspects collectively ensure the practical usability of the methods in real-world scenarios. We follow [19 for setting up baselines, whose details can be found in the official website
C.3 Implementation Details of CSM
CSM Model Architecture. Our model architecture consists of three main components: (1) a pre-
trained BERT-uncased 10 model serving as the local layer, (2) global layer comprising 3 layers of 8-head self-attention layers, and (3) 2-layer MLP functioning as the output layer.
RAG Pipeline. We implement a sequential RAG pipeline following FlashRAG [9]. The retrieval source is the Wikipedia dump from December 2018, which we preprocess into chunks of 100 words each. For each query, retrieve 10 chunks using a dense retriever based the E5-bbae-v2 45
details. md

---

Dataset:
Dataset: triviaga
Dataset: webqa
0.65
0.35
0.22
0.30
0.20
0.25
0.18
0.50
hotpotqa
2wikimultihopga
0.95
0.34
0.90
0.40
0.30
0.85
0.28
0.35
0.26
0.80
bge-meanker
0.22
RECOMP-aX.
0.75
0.20
Random
contexts
contexts
Figure 7: RAG generation performance when selecting high-quality contexts using different quality metrics, where higher curve indicates better metric. Dashed line marks the top-k cutoff where the average CI value is zero and star marks the top-k yielding best performance. For baselines, predicted scores for bge-reranker and RankGPT, and oracle log likelihood scores for RECOMP-ex.
model. We use Llama3-8k Baanaa---b-nntuuee Sb-intruct [ and Swwe2.55-bbiinttee experiments are conducted with a fixed random seed of 2024 for reproducibility.
42 as the LLM generators. All
Hyperparameter setting. In our experiments, we employ the following hyperparameters: for supervised training, we set 1 and = 0.1 and train CSM for 10 epochs with a batch size of 16; for end-to-end training, we set and train CSM for 10 epochs with a batch size of 4.
D Additional Experiments on Effectiveness of CI Value
In Figure and Figure S we present additional experiments focusing on context selection by adding high-quality contexts and poor-quality contexts, respectively. Our comprehensive experiments across diverse datasets consistently show that employing CI value as quality metric for context selection proves be effective strategy, successfully identifying crucial contexts while simultaneously eliminating detrimental ones.
E
Case Studies
To illustrate the efficacy of the CI value metric for context selection in a RAG system, we present specific case studies this section.
E.1
Case Study
As illustrated Figure this case encompasses retrieved contexts for a query from NQ dataset and the corresponding predicted answers under different conditions. In this case, the CI value can identify valid information, enabling the generator to produce the correct answer. Our key observations are as follows: When all retrieved contexts (C1, C2, C3, and cu) are supplied to the LLM, contexts with negative values significantly distort the generated response. Specifically, c,, characterized by CI value of Φ1 (v) -0. 19. incorrectly conflates Toyota's 's arrival in El Salvador in 1953 with its entry into the United States market, that leads to incorrect prediction "May 1953". Similarly, 02 with CI
value of (t) -0.14. misrepresents Toyota's initial entry into the U.S. market export event
25

---

Dataset: NQ
Dataset: TriviaQA
Dataset: WebQA
0.35
0.30
0.575
0.550
0.16
5 0.25
M
0.525
0.14
0.500
0.475
0.12
0.15
0.450
selected
FEVER
Dataset: HatpotQA
2WikiMultihopQA
0.28
0.90
0.40
0.26
0.85
0.35
0.80
0.22
bge-neranker
0.75
0.20
RECOMP-aX.
0.18
value
Random
0.70
contexts selected
contexts selected
contexts
Figure 8: RAG generation performance when selecting poor-quality contexts using different quality metrics, where higher curve indicates better metric. Dashed line marks the top-k cutoff where the average CI value is zero and star marks the top-k yielding best performance. For baselines, use predicted scores for bge-reranker and RankGPT, and oracle log likelihood scores for RECOMP-ex.
in June 1958. Meanwhile, C1, with (1(1) = O, focuses on Toyota's early corporate history, which tangential the query regarding its U.S. market entry, thus rendering it contextually irrelevant. These guery-irrelevan contexts occupy the context window of L.LM and increase the inference time without improving its performance. In contrast, when only the context with positive CI value, cy (Φa(v) = 1.01), which correctly states that Toyota entered the American market 1957, included in the input, the LLM produces the accurate answer. This case study underscores the critical role of CI values in filtering out deleterious and extraneous information, thereby enabling more accurate and reliable predictions in RAG frameworks.
E.2
Case Study 2
Figure TO presents another illustrative example from the NQ dataset. When all retrieved contexts provided to the LLM, contexts with negative CI values significantly distort the generated response, leading the incorrect answer "Cars". Our analysis reveals that context c2 is the primary cause of the incorrect generation, as it contains mentions of "Cars" with the Oscar reward, resulting in a negative CI value of (v) -0.14 In contrast, context C1. which contains the correct answer "Ratatouille", receives a relatively high CI value of o (v) = 0.57. Mcanwhile, context which discusses the movie "WALL-E" and its pity for not winning the Oscar, and context ca, which covers the history of Pixar studio, both receive CI values close to zero, indicating that their presence does not contribute to answering the question correctly.
E.3
Case Study 3
Figure II presents another illustrative example from the HotpoiQA dataset. To answer the question "Which was fought earlier in US's history, the Seven Days Battles or the Battle of Manila?", one must provide accurate information about both historical events. However, the "Battle of Manila" presents an inherent ambiguity due to multiple battles bearing this name throughout history. The question's ss specification of "in US history" helps narrow the scope, indicating that we should only consider the Battle of Manila that occurred after the United States' establishment, Our analysis reveals that contexts 6g and 64, which discuss the Battle of Manila Philippine history prior to the US's founding, are potentially misleading and receive negative CI values (0s(v) -0.32 and
26

---

Query: When did Toyota first come the United States? Golden Answer: 1957
Retrieved Contexts
C1
By the of decade, Toyota established worldwide presence, as company exported one-millionth unit. The first Japanese vehicles to arrive North America were five Land Cruisers in
Salvador in May 1953."
p (v) 0.19
"... To.otoy first export to the United States began with 30 Crown Deluxe's 's in June 1958 after establishing Toyota Motor Sales USA the previous October, the first directly managed retail dealer (Hollywood Toyota] the previous February, U.S. wholesale and import companies."
(2
C3
...Her,ver, when Toyota eventually entered the American market in 1957 with the Crown, the name was not well received due to connotations of toys and pets. The name was soon dropped for the American market, but continued in other markets until the mid-1960s.
(v 1. 01
"The history Toyota started in 1933 with the company being division of Toyoda Automatic Loom Works devoted to the production of cars under the direction of the founder's son, Kiichiro Toyoda. Klichiro Toyoda had traveled to Europe and the United States in 1929 to investigate automobile production and had begun researching pesweeee-powered engines in 1930. Toyoda Automatic Loom Works encouraged develop automobile production by the Japanese government, which needed domestic vehicle production, due the war with China."
( 0.00
Prediction w/ full contexts: May 1953 Prediction w/ contexts of positive CI values: 1957
X
Figure Case study (test case #138 NQ dataset) with Llama3-8B as LLM generator.
Query: Which film won the Oscar for best animated feature in 2007?
Golden Answer: Ratatouille
Retrieved Contexts
Ratatouille the Animated Feature award the 2008 Golden Globes; it also nominated for Academy Awards, including Best Animated Feature Best Original Screenplay. On February 24, 2008, Ratatouille won Bird his second Academy Award for Best Animated Feature Film.."
Φ (v) 0.57
C2
"Golden Globe Award for Best Animated Feature Film The Golden Globe Award for Best Animated Feature Film awarded for the first time the 64th Golden Globe Awards in 2007. It was the first time that the Golden Globe Awards had created separate category for animated films since its establishment. The nominations are announced in January three films are nominated best animated film, in contrast to five nominations for the majority of other awards. The Pixar film Cars was the first recipient of the award"
awards ceremony
later
month. Initially, only
14
C3
"At the 81st Academy Awards, in which WALL-E won the award but was not nominated for Best Picture, despite receiving widespread acclaim from critics and audiences and being generally considered one of the best films of 2008..."
$ (v) = - 0. 0.02
C4
"...In 2007, Pixar released Meet the Robinsons, which experienced a poor response at the box office despite the lukewarm critical and audience reception. The following film, 2008's Bolt had the best critical reception of any Disney animated feature since Lilo & Stitch, and became moderate success."
& (v) 0.03
Prediction w/ full contexts: Cars
X
Prediction w/ contexts of positive CI values: Ratatouille
Figure 10: Case study (test case # 143 in NQ dataset) with Llama3-8B as LLM generator.
27

---

Query:
Which was fought earlier in US history, the Seven Days Battles or the Battle of Manila? Golden Answer: The Seven Days Battles
Retrieved Contexts
"The Seven Days Battles series seven battles days from July 1862, Richmond, Virginia, during the American Civil Confederate General Robert E...
( 0.21
C2 "The Battle of Manila (Filipino: Labanan sa Maynila; ), the first and largest battle of the Philippine-American War, was fought February 4-5, 1899, between 19,000 American soldiers and 15,000 Filipino armed militiamen...
Φ (v) 0. 23
C3
"The Manila (Filipino: Labanan sa Maynila) (1365) is unspecified disputed battle occurring somewhere the vicinity of Manila between the forces of the kingdoms in Luzon and the Empire of Majapahit..."
Φ (v(v) 0. 32
C4
...Weesites Battles of La Naval de Manila The Battles of La Naval Manila were series five naval battles fought in the waters of the Philippines in the year 1646, in which the forces of Spain repelled various attempts by forces of the Dutch Republic to invade Manila, during the Eighty Years' War. The Spanish forces, which included many native Kapampángan volunteers, consisted of two, and later, three Manila galleons, galley and four brigantines..."
( -0. 14
Prediction w/ full contexts: The Battle Manila (1365) X
Prediction w/ contexts of positive CI values: The Seven Days Battles
Figure 11: Case study (test case #157 in HotpotQA dataset) with Llama3-8B LLM generator.
(a(1) 0.144. These contexts could lead the generator produce incorrect answers. In contrast, contexts 01 and C2, which contain the correct temporal information about both the Seven Days Battles and the relevant Battle of Manila, assigned positive CI values (ø1 (v) 0.21 and (v) 0.23). The retention of these contexts enables the generator to produce the correct answer.
28


Reviews:
[{"review_id": "azekpyIsCo", "reviewer": "Reviewer_RASc", "summary": "This paper studies context selection in Retrieval-Augmented Generation, where Large Language Models use retrieved knowledge to reduce hallucination. The authors identify key limitations in existing context selection methods, which typically rely on query, list, or generator-aware quality metrics, often failing to comprehensively balance these factors and requiring dataset-specific hyperparameter tuning. To address these issues, the paper introduces the Contextual Influence (CI) value, which quantifies each context’s contribution by measuring the drop in performance when it is removed. They further propose a Context Selection Model (CSM) that predicts CI values at inference time without needing access to labels, using hierarchical modeling and either supervised or end-to-end training with generator feedback. Experiments on eight NLP tasks and two LLM backbones show that this approach outperforms a wide range of strong baselines, leading to improved generation quality and robustness.", "questions": "1.How would the CI value be adapted if the generation utility (e.g., $v(f(q⊕C))$) is not well-defined by EM/F1 (e.g., creative writing, summarization)?\n\n2.In scenarios where useful information is distributed across several weakly helpful contexts (none with individually positive CI), could the current approach lead to information loss? Are there strategies to mitigate filtering out such complementary but individually weak contexts?\n\n3.How well does the CI surrogate model generalize to new retrieval domains? Would it require retraining for each new task, or is it robust enough to handle domain shifts without additional tuning?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "> Strengths：\n\n1. The CI value offers a holistic context evaluation metric that explicitly combines query relevance, listwise context uniqueness, and generator feedback—overcoming key limitations of previous metrics. \n\n2. The proposed CSM model structurally encodes both local (query-context) and global (context-context) dependencies, a sound design that reflects the nature of the selection problem.\n\n3. The paper presents consistent improvements over a broad range of baselines—including query-aware, list-aware, and generator-aware methods, as well as robust LLM tuning—across various QA tasks. The evaluation is comprehensive, combining diverse benchmarks with both quantitative metrics and qualitative case studies to support the claims.\n\n> Weaknesses：\n\n1.There is little quantitative comparison of inference-time latency or computational resources used by CSM versus standard RAG and other selection approaches, especially as the number of retrieved contexts grows large.\n\n2.While the method elegantly sidesteps the cost of explicit influence computation, details on CSM’s potential to misestimate context dependencies in more adversarial or correlated inference setups are limited.", "was_revised": true, "final_justification": "The authors have made the following responses or adjustments:\n1.Supplemented the experiments with inference time cost analysis;\n2.Responded to the concern regarding the diversity of application scenarios.\nHowever, the CSM method still exhibits notable limitations, such as the need for retraining on new tasks. Therefore, I have decided to keep my current score unchanged.", "rebuttal": "> **Weakness 1: There is little quantitative comparison of inference-time latency or computational resources used by CSM versus standard RAG and other selection approaches, especially as the number of retrieved contexts grows large.**\n\n**Response:** The efficiency of RAG systems primarily depends on the parameter size of the context selection model when keeping the retriever and generator the same. The table below presents the parameter sizes of context selection models and their corresponding CUDA times (from context selection to generation) for different baselines w.r.t. the number of retrieved contexts $n$. Since CSM selects all positive contexts, for fair comparison, we set the number of preserved contexts (i.e., top-$k$) for the baselines equal to the average number of contexts with positive CI values (i.e., $k_{\\rm pos}$). For $n=10$, we set top-$k=k_{\\rm pos}=5$, and for $n=50$, we set top-$k=k_{\\rm pos}=23$. The experiments utilize E5-base-v2 as the retriever, Llama3-8B-Instruct as the generator, and are conducted on a server equipped with Montage Jintide(R) C6226R CPU, 256GB Memory, and 4 NVIDIA GeForce RTX 4090 GPUs.\n\n|              | Parameter Size | NQ ($n$=10) | NQ ($n$=50) | TriviaQA ($n$=10) | TriviaQA ($n$=50) | HotpotQA ($n$=10) | HotpotQA ($n$=50) |\n| :----------- | -------------- | :---------: | :---------: | :---------------: | :---------------: | :---------------: | :---------------: |\n| Standard RAG | -              |   320 ms    |   811 ms    |      252 ms       |      810 ms       |      261 ms       |      814 ms       |\n| RankGPT      | 8030M          |   874 ms    |   1437 ms   |      779 ms       |      1561 ms      |      741 ms       |      1640 ms      |\n| RECOMP-abs   | 775M           |   299 ms    |   662 ms    |      254 ms       |      721 ms       |      202 ms       |      924 ms       |\n| CSM          | 140M           |   253 ms    |   481 ms    |      192 ms       |      402 ms       |      206 ms       |      423 ms       |\n\nCompared to baseline context selection models, CSM achieves significantly lower latency due to its lightweight model architecture. This efficiency advantage becomes more pronounced as the number of retrieved contexts increases from $n=10$ to $n=50$. For instance, on NQ with $n=50$, CSM reduces the latency to 481 ms, while RankGPT requires 1437 ms and RECOMP-abs requires 662 ms. This demonstrates that CSM scales more efficiently when $n$ grows large.\n\n> **Weakness 2: While the method elegantly sidesteps the cost of explicit influence computation, details on CSM’s potential to misestimate context dependencies in more adversarial or correlated inference setups are limited.**\n\n**Response:** We appreciate the reviewer's concern regarding CSM’s behavior under adversarial or correlated inference setups. CSM is trained to fit the context influence (CI) value distributions without making assumptions about the quality of retrieved contexts. When trained on CI values computed from adversarial or correlated inference data, CSM naturally learns to estimate context utility in such challenging scenarios. Our experiments include training samples from both types of setups. Adversarial conditions, where noisy or distracting contexts are common, are frequently observed in open-domain QA datasets such as NQ, TriviaQA, and WebQA [1]. Correlated setups, where evidence is distributed across multiple contexts, are typical in multi-hop QA tasks like HotpotQA. Moreover, the hierarchical architecture of CSM is specifically designed to address these challenges by jointly modeling query-context interactions, which help identify and down-weight noisy contexts, and context-context interactions, which capture dependencies among correlated contexts. These design choices enable CSM to select high-utility contexts even in the presence of noise and complex inter-context relationships. \n\n\n\n> **Questions 1: How would the CI value be adapted if the generation utility (e.g.,$v(f(q⊕C))$) is not well-defined by EM/F1 (e.g., creative writing, summarization)?**\n\n**Response**: For any context set $S$, the utility function $v(A)=v(f(q\\oplus S))$ measures the quality of the generated answer $A$ when taking $q\\oplus S$ as input. The CI value of a context quantifies the decrease in answer quality when that context is removed from the context list. Therefore, as long as answer quality can be effectively evaluated, the CI value can be readily adapted.\n\nWhen golden answers are available, real-valued scores (e.g., EM/F1/perplexity) serve as well-qualified metrics for measuring the quality of generated answers. For tasks without golden answers (e.g., creative writing and summarization), LLM-as-a-Judge metrics [2] provide a feasible alternative. Specifically, let $f_e$ denote the LLM-based evaluator, based on predefined quality dimensions (e.g., clarity, coverage, etc. [3]), it can either assign quality scores to any given answer or directly compare two candidate answers. The adaptation of CI value $\\phi_i$ for context $c_i$ proceeds as follows:\n\n- If the LLM-based evaluator assigns quality scores to answers, i.e., $f_e(A)\\in \\mathbb{R}$, then $\\phi_i=f_e(A_C)-f_e(A_{C\\backslash\\{i\\}})$, where $A_C$ and $A_{C\\backslash\\{i\\}}$ represent the generated answers with and without context $c_i$, respectively.\n\n- If the LLM-based evaluator directly compares two candidate answers, i.e., $f_e(A_1,A_2)\\in \\{0,1\\}$, where $f_e(A_1,A_2)=0$ indicates that $A_1$ is superior to $A_2$, then $\\phi_i=f_e(A_C,A_{C\\backslash\\{i\\}})$, where $A_C$ and $A_{C\\backslash\\{i\\}}$ represent the generated answers with and without context $c_i$, respectively.\n\n\n\n> **Question 2: In scenarios where useful information is distributed across several weakly helpful contexts (none with individually positive CI), could the current approach lead to information loss? Are there strategies to mitigate filtering out such complementary but individually weak contexts?**\n\n**Response**: We would like to clarify that weakly helpful contexts may have relatively small but still positive CI values. When useful information is distributed across such contexts, removing any one of them can lead to a decrease in overall utility. We can simply select all contexts with positive CI values to preserve these weakly helpful contexts. Weakly helpful contexts are commonly observed in multi-hop QA tasks like HotpotQA, where answering a question requires information from multiple contexts. To effectively handle these cases, CSM is designed to capture context-context interactions, enabling it to better model the CI value distributions of weakly helpful contexts, thus leading to remarkable performance improvement in multi-hop QA tasks. \n\n\n\n> **Question 3: How well does the CI surrogate model generalize to new retrieval domains? Would it require retraining for each new task, or is it robust enough to handle domain shifts without additional tuning?**\n\n**Response**: CSM currently requires retraining for new, unseen tasks. CSM is trained to fit CI value distributions and approximate CI value under specific tasks, and additional tuning is required when domain shift leads to CI value distribution shift. To improve generalizability across domains, a possible way is to train a larger model with better generalization ability, e.g., using mixture of experts architecture. However, this approach typically incurs substantially higher training costs, and it may still be vulnerable to domain shifts. For these reasons, we prioritize retrain a lightweight CSM model to achieve the goal of fast adaptation.\n\n[1] Fang, Feiteng, et al. \"Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training.\" *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. 2024.\n\n[2] Gu, Jiawei, et al. \"A survey on llm-as-a-judge.\" *arXiv preprint arXiv:2411.15594* (2024).\n\n[3] Nguyen, Huyen, et al. \"A comparative study of quality evaluation methods for text summarization.\" *arXiv preprint arXiv:2407.00747* (2024)."}, {"review_id": "MiQFlIoXcV", "reviewer": "Reviewer_77ay", "summary": "The author proposes a method for automatic context document selection in the RAG scenario called CSM. Specifically, the author introduces the use of Contextual Influence (CI) values to measure the usefulness of contexts. Two training methods, Supervised training and end-to-end training, are employed to train the CSM model to predict context CI values. During inference, CSM predicts the CI values for all contexts and selects those with CI values greater than 0. The author provides extensive experimental evidence demonstrating the effectiveness of CSM.", "questions": "1. I am concerned whether the oracle CI scores in certain special cases accurately represent the usefulness of the context. For example, when a user queries information about a hot news topic, the retriever may retrieve several similar contexts. These contexts provide the same important information, and removing one does not affect the utility. According to Definition 1, their CI values will be near 0, but these contexts are still quite important. Does this represent a potential flaw?\n2. In the experiments, is the CSM model the same for different datasets? How does the CSM model perform on out-of-distribution data? This is an important concern.\n3. In the End-to-end Training of CSM section, the authors use a masking method to obtain $H_c$. This could introduce a potential issue: during training, low-scoring context positions in $H_c$ still have placeholders close to zero, but these placeholders do not exist during inference, leading to a gap between training and inference. I understand this is a trade-off between differentiability and efficiency, but I am still curious whether this gap affects performance.\n\nMinor Comments:\n1. The average improvement in Table 1 is quite misleading when presented as a percentage. It is recommended to include the absolute value of the increase as well.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear, and CSM efficiently and effectively avoids the manual selection of top-k documents, which is very attractive.\n2. The end-to-end training method for CSM is ingenious and successfully addresses the issue of the context selection process being discrete and non-differentiable.\n3. The author provides the complete code.\n\nWeakness:\n1. The method section may have some potential flaws\n2. The performance on OOD data has not been explored.  \nPlease see the Questions section for details", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1: The method section may have some potential flaws.**\n>\n> **Question 1: I am concerned whether the oracle CI scores in certain special cases accurately represent the usefulness of the context. For example, when a user queries information about a hot news topic, the retriever may retrieve several similar contexts. These contexts provide the same important information, and removing one does not affect the utility. According to Definition 1, their CI values will be near 0, but these contexts are still quite important. Does this represent a potential flaw?**\n\n**Response**: We would like to clarify that the definition of CI value is build on the assumption that there are no redundant context in the retrieved context list. This assumption can be easily satisfied in modern vector databases. For example, faiss support vector deduplication during KNN search. To further prevent highly similar contexts from affecting the correctness of CI values, we adopt a simple yet effective mechanism inspired by previous works [1, 2]: we preprocess the retrieved contexts by clustering semantically similar contexts, and assign the same CI value to contexts inside each cluster. In our experiments, this mechanism is rarely triggered, occurring in only about 3% of the data samples.\n\n\n\n> **Weakness 2: The performance on OOD data has not been explored.**\n>\n> **Question 2: In the experiments, is the CSM model the same for different datasets? How does the CSM model perform on out-of-distribution data? This is an important concern.**\n\n**Response**: The CSM model is not shared across different datasets. CSM is trained to fit CI value distributions and approximate CI values under specific inference tasks. Therefore, if a different dataset leads to CI value distribution shift, the CSM model needs to be retrained accordingly. Please note that the compared context selection methods similarly require retraining on each new dataset, as their performance is also sensitive to distributional shifts.\n\nFollowing your suggestion, we simulate out-of-distribution (OOD) scenarios by training CSM on one dataset (e.g., NQ) and evaluating it on a different dataset (e.g., TriviaQA). In contrast, in-distribution (ID) scenarios involve training and evaluating CSM on the same dataset. The following table shows the results in the form of $a\\to b$, where a is the RAG performance under ID setting and b is the RAG performance under OOD setting.\n\n|            | Parameter size | TriviaQA (EM) |  WebQA (EM)   | HotpotQA (F1) |\n| ---------- | :------------: | :-----------: | :-----------: | :-----------: |\n| RECOMP-ex  |      109M      | 0.60$\\to$0.55 | 0.18$\\to$0.14 | 0.39$\\to$0.35 |\n| RECOMP-abs |      775M      | 0.59$\\to$0.57 | 0.18$\\to$0.15 | 0.40$\\to$0.37 |\n| CSM-st     |      140M      | 0.69$\\to$0.61 | 0.24$\\to$0.19 | 0.47$\\to$0.35 |\n\nWe can observe a consistent performance drop across all methods under OOD setting, but **CSM still outperforming other baselines**. Meanwhile, the performance drop in HotpotQA is larger than that in TriviaQA and WebQA. This is because HotpotQA is a multi-hop QA task that quite different from NQ, while TriviaQA and WebQA are both open-domain QA tasks more similar to NQ. \n\nTo improve the generalization ability of CSM as well as other context selection methods,  existing techniques for handling out-of-distribution data may be leveraged. For example, training a diverse mixture of tasks may help CSM capture transferable patterns. Additionally, methods like continual or meta-learning can be explored to adapt CSM to new distributions with minimal retraining.\n\n \n\n> **Question 3: In the End-to-end Training of CSM section, the authors use a masking method to obtain $H_c$. This could introduce a potential issue: during training, low-scoring context positions in still have placeholders close to zero, but these placeholders do not exist during inference, leading to a gap between training and inference. I understand this is a trade-off between differentiability and efficiency, but I am still curious whether this gap affects performance.**\n\n**Response**: This gap leads to little affect on the overall performance, since the generator LLM is generalizable to such gap. To evaluate this, we conducted the following experiment: for a data sample $x = (q, C, M)$, where $q$ is the question, $C$ is the retrieved context list, and $M$ is a near-binary mask produced by the CSM indicating which contexts to preserve, we compare the difference of LLM's output $f(x)$ when using soft-selection and hard-selection. For **soft-selection**, we apply the mechanism described in Section 4.3 and obtain the generated answer $\\hat{y}_1$. For **hard-selection**, we first perform context selection based on $M$, use the selected contexts as input to the generator, and obtain the generated answer $\\hat{y}_2$. We then compute the F1 score between $\\hat{y}_1$ and $\\hat{y}_2$. A higher F1 score indicates the output of soft selection is similar to that of hard selection, meaning that the generator's behavior is consistent during training and inference. Using test samples from NQ and HotpotQA, we calculate the average F1 score, as shown in the following table:\n\n|            | Llama3-8b-instruct | Qwen2.5-7b-instruct |\n| ---------- | :----------------: | :-----------------: |\n| Average F1 |        0.95        |        0.93         |\n\nResults show that soft-selection and hard-selection leads to highly similar LLM outputs, thus the gap between training and inference has little effect on performance.\n\n\n\n> **Minor Comments: The average improvement in Table 1 is quite misleading when presented as a percentage. It is recommended to include the absolute value of the increase as well.**\n\n**Response**: We report performance improvements in percentage terms to intuitively show the effectiveness of CSM. The percentage is calculated by comparing the average improvement of CSM (both CSM-st and CSM-e2e) ove all baselines (except Vanilla LLM). Absolute improvements will also be included in the subsequent version of the paper.\n\n\n\n[1] He, Yexiao, et al. \"Shed: Shapley-based automated dataset refinement for instruction fine-tuning.\" *Advances in Neural Information Processing Systems* 37 (2024): 99382-99403.\n\n[2] Tarun, Ayush, et al. \"EcoVal: An Efficient Data Valuation Framework for Machine Learning.\" *Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*. 2024."}, {"review_id": "ztDLTHivuh", "reviewer": "Reviewer_2KDq", "summary": "This paper proposes a new reranking model for context selection for retrieval augmented generation (RAG). Specifically, an auxiliary model is trained to predict how much removing a context from a list of retrieved documents will affect the final performance, i.e. CI or contextual influence value. Two training strategies are introduced, one based on supervised signals derived from labeled data, and one based on generated data. Specific training tricks are proposed to deal with imbalanced distributions of CI values. Experiments show the effectiveness of the CI value-based context selection performance as of the final answer quality.", "questions": "- Due to the nature of CI values, it is okay and reasonable to select context based on whether the CI value is positive. However, for fairness of comparison, it is better to ensure there are no significantly more contexts being selected. Maybe also report the average number of contexts selected based on CI\n- In figure 3, it clearly shows the performance first increases and then decreases. I thus wonder whether the best performance always corresponds to the zero points of CI values empirically?\n- Iine 258-259, it is mentioned the data splits used by several baselines. However, it is not mentioned whether the proposed method also use the same splits.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The idea of how each context contributes to the final answer is meaningful for RAG and is studied by previous research in different forms, as also mentioned in the paper.\n- Experiments clearly show the induced CI value correlates to the answer quality and thus useful as reranking scores\n- The methodology is clearly described with plenty of details. \n\n### Weaknesses\n- The way of using CI scores during inference could be better analyzed. see some questions below\n- It would be better to include some efficiency comparison on some compared methods to show the applicability of the method.", "was_revised": true, "final_justification": "Overall, this paper proposes an interesting idea and the writing is clear. My concerns are addressed well and the added information should be updated in the paper draft. I would keep my score of a Borderline accept.", "rebuttal": "> **Weakness 1 and Question 1: Due to the nature of CI values, it is okay and reasonable to select context based on whether the CI value is positive. However, for fairness of comparison, it is better to ensure there are no significantly more contexts being selected. Maybe also report the average number of contexts selected based on CI**\n\n**Response**: For fair comparison, the number of selected contexts (top-$k$) is kept consistent across all baselines. Since CSM selects all positive contexts, we set top-$k$ as the average number of positive contexts, to ensure that the number of selected contexts is equivalent between CSM and the baselines. The table below reports the average number of positive contexts (denoted as $k_{\\rm pos}$) when retrieving 10 contexts:\n\n|               |  NQ  | TriviaQA | WebQA | HotpotQA | 2Wiki | FEVER | TruthfulQA | ASQA |\n| ------------- | :--: | :------: | :---: | :------: | :---: | :---: | :--------: | :--: |\n| $k_{\\rm pos}$ | 4.7  |   4.9    |  4.7  |   5.5    |  5.4  |  5.5  |    4.5     | 5.4  |\n\nBased on the above table, we set top-$k=5$ for all baselines across all datasets.\n\n> **Weakness 2: It would be better to include some efficiency comparison on some compared methods to show the applicability of the method.**\n\n**Response**: The efficiency of RAG systems primarily depends on the parameter size of the context selection model when keeping the retriever and generator the same. The table below presents the parameter sizes of context selection models and their corresponding CUDA times (from context selection to generation) for different baselines w.r.t. the number of retrieved contexts $n$. Since CSM selects all positive contexts, for fair comparison, we set the number of preserved contexts (i.e., top-$k$) for the baselines equal to the average number of contexts with positive CI values (i.e., $k_{\\rm pos}$). For $n=10$, we set top-$k=k_{\\rm pos}=5$, and for $n=50$, we set top-$k=k_{\\rm pos}=23$. The experiments utilize E5-base-v2 as the retriever, Llama3-8B-Instruct as the generator, and are conducted on a server equipped with Montage Jintide(R) C6226R CPU, 256GB Memory, and 4 NVIDIA GeForce RTX 4090 GPUs.\n\n|              | Parameter Size | NQ ($n$=10) | NQ ($n$=50) | TriviaQA ($n$=10) | TriviaQA ($n$=50) | HotpotQA ($n$=10) | HotpotQA ($n$=50) |\n| :----------- | -------------- | :---------: | :---------: | :---------------: | :---------------: | :---------------: | :---------------: |\n| Standard RAG | -              |   320 ms    |   811 ms    |      252 ms       |      810 ms       |      261 ms       |      814 ms       |\n| RankGPT      | 8030M          |   874 ms    |   1437 ms   |      779 ms       |      1561 ms      |      741 ms       |      1640 ms      |\n| RECOMP-abs   | 775M           |   299 ms    |   662 ms    |      254 ms       |      721 ms       |      202 ms       |      924 ms       |\n| CSM          | 140M           |   253 ms    |   481 ms    |      192 ms       |      402 ms       |      206 ms       |      423 ms       |\n\nCompared to Standard RAG, CSM can reduce the input length of generator without introducing heavy computation overhead, thereby decreasing the overall computation time. Compared to baseline context selection models, CSM achieves significantly lower end-to-end latency due to its lightweight model architecture. This efficiency advantage becomes more pronounced as the number of retrieved contexts increases from $n=10$ to $n=50$. For instance, on NQ with $n=50$, CSM reduces the latency to 481 ms, while RankGPT requires 1437 ms and RECOMP-abs 662 ms. This demonstrates that CSM scales more efficiently when $n$ grows large.\n\n> **Question 2: In figure 3, it clearly shows the performance first increases and then decreases. I thus wonder whether the best performance always corresponds to the zero points of CI values empirically?**\n\n**Response**: Let $k_{\\rm zero}$ be the zero point of CI values, the best performance always corresponds to $k_{\\rm zero}$ or $k_{\\rm zero}-1$ in our experiments. When plotting Figure 3, we set $k_{\\rm zero}$ as the nearest integer, e.g., if the average number of positive contexts is 4.7, we round it to $k_{\\rm zero} = 5$. Then, for the context selection experiments by adding top-$k_{\\rm zero}$ high-quality contexts, some negative contexts may also be selected, thus affecting the performance. \n\n\n> **Question 3: Iine 258-259, it is mentioned the data splits used by several baselines. However, it is not mentioned whether the proposed method also use the same splits.**\n\n**Response**: Thank you for pointing this out. The data splits are the same for both baselines and the proposed CSM, and we will clarify this in the subsequent version of the paper."}, {"review_id": "Yiy45YNsdU", "reviewer": "Reviewer_ynDW", "summary": "This paper propose to assess context in RAG via inference-time data valuation, and a contextual influence value(CI) is proposed.  Specifically, a parameterized surrogate model is trained to predict CI values of difference contexts by capturing dependences both  between query-context and inter-contexts, and the proposed method simply retains contexts with positive CI values. Experimental results show that the proposed method can achieve some performance improvement.", "questions": "Please see Weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The idea of measuring a context via the performance degradation when removing it is reasonable;\n2. CI values can take all query-aware, list-aware, and generator-aware metrics into consideration;\n3. Experimental results show some effectiveness of the proposed method.\n\n Weaknesses:\n1. My main concern about this paper is the mismatch between the motivation of CI value( the performance degradation when removing a specific context) and how it is actually calculated(a surrogate model which simply takes several input and output the CI value). It is hard to say such a model can match the motivation of the CI value, and whether can generalize to unseen scenarios. Therefore, I think the authors should explain why such a surrogate model can actually reflect the CI value, and give detailed analysis between golden CI values and calculated value using several examples. \n2. The current baselines are mainly vanilla RAG + context scoring, the authors should also compared with advanced RAG systems such as agentic RAG systems, which can also take the quality and the importance of contexts into consideration. To make the experimental results more convincing, the authors should also compare with state-of-the-art solutions, rather than only simple baselines.", "was_revised": true, "final_justification": "The rebuttal addressed some of my concerns and I adjust my score accordingly.", "rebuttal": "> **Weakness 1: My main concern about this paper is the mismatch between the motivation of CI value( the performance degradation when removing a specific context) and how it is actually calculated(a surrogate model which simply takes several input and output the CI value). It is hard to say such a model can match the motivation of the CI value, and whether can generalize to unseen scenarios. Therefore, I think the authors should explain why such a surrogate model can actually reflect the CI value, and give detailed analysis between golden CI values and calculated value using several examples.**\n\n**Response**: The motivation of developing CI value as the context quality assessment metric is that it satisfies four key properties: query-awareness (context-query relevance), list-awareness (context-context dependance), generator-awareness (explicit feedback from generator) and ease-of-configuration. These properties are crucial for improving context selection performance, but they cannot be simultaneously satisfied by any existing metrics.\n\nDespite the favorable properties of CI value, it is infeasible to exactly calculate CI value in the inference time, since (1) CI value requires golden answers for computing utilities, which are unavailable during inference; and (2) calculating CI value requires multiple LLM forwards, leading to significant computational overhead. To make it feasible, we propose a parameterized model (i.e., CSM) to approximate CI values.\n\nTo match the motivation of CI value, we have carefully designed the architecture and training paradigm of the CSM.  To ensure query-awareness and list-awareness, we developed a hierarchical model structure by capturing both query-context semantics and context list-wise semantics, as illustrated in Figure 2(a). To inject generator-awareness into CSM, we explored two training strategies: (1)  supervised learning using oracle CI values as targets, which provides implicit generator feedback;  and (2) end-to-end training with the generator in the loop, which offers explicit signals about each context’s impact. \n\nTo evaluate how these designs help approximating CSM's predictions with golden CI values, we conducted empirical studies in Section 6.2: (1) By comparing the RAG performance of the CSM with other baselines, we show that the CSM performs effectively in context selection, closely approximating the performance of the oracle CI value; (2) The Spearman correlation between CSM's predictions and golden CI values is high; and (3) Ablation studies by corrupting the training strategies confirm the necessity of our designs. \n\n\n\n> **Weakness 2: The current baselines are mainly vanilla RAG + context scoring, the authors should also compared with advanced RAG systems such as agentic RAG systems, which can also take the quality and the importance of contexts into consideration. To make the experimental results more convincing, the authors should also compare with state-of-the-art solutions, rather than only simple baselines.**\n\n**Response**: We would like to clarify that advanced RAG pipelines are also included as baselines in our experiments. Take the baseline Ret-Robust [1] for example, it is implemented as a self-ask RAG pipeline[2] that use an agentic LLM to decompose complex questions into subquestions and perform multi-round of retrieval and generation until generating the final answer. It is one of the state-of-the-art baselines in the latest FlashRAG benchmark [2]. \n\nFor more comprehensive evaluation, we conduct additional experiments using more advanced RAG pipelines as baselines. We first provide a brief introduction to the new baselines:\n\n- Self-RAG [3] performs on-demand retrieval and learns to reflect on retrieved contexts while critiquing generated answers.\n\n- RQ-RAG [4] enhances the RAG pipeline through explicit rewriting, decomposition, and disambiguation.\n\nThese baselines encompass common agentic capabilities including reflection and planning. The following table presents the RAG performance (using Llama3-8B-Instruct as generator) of CSM compared to these baselines.\n\n|              | NQ (EM)  | TriviaQA (EM) | WebQA (EM) | HotpotQA (F1) |\n| ------------ | :------: | :-----------: | :--------: | :-----------: |\n| Self-RAG [3] |   0.36   |     0.38      |    0.22    |     0.30      |\n| RQ-RAG [4]   |   0.34   |     0.55      |  **0.26**  |     0.35      |\n| CSM-st       | **0.42** |   **0.69**    |    0.24    |   **0.47**    |\n\nThe results demonstrate that CSM achieves strong performance across most datasets, outperforming state-of-the-art agentic baselines on NQ, TriviaQA, and HotpotQA, while maintaining competitive performance on WebQA. These results highlight CSM's effectiveness in filtering out low-quality contexts in RAG, when compared to sophisticated agentic approaches that employ complex reflection and planning mechanisms."}]

Author rebuttal:

Meta-review: This paper introduces Contextual Influence (CI) value and a Context Selection Model (CSM) for RAG, integrating query-, list-, and generator-awareness into context selection. Reviewers appreciated the clear motivation, lightweight design, and consistent gains across benchmarks, while noting limitations in generalization and the need for retraining on new tasks. The authors strengthened the work in rebuttal with efficiency analyses and comparisons against advanced agentic RAG baselines, which should be included in the final version.
Decision: Accept (poster)