Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection
Authors: Yun Zhu
Jia-Chen Gu
Caitlin Sikora
Ho Ko
Yinxiao Liu
Chu-Cheng Lin
Lei Shu
Liangchen Luo
Lei Meng
Bang Liu
Jindong Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Retrieval-Augmented Generation
Sparse Context
Efficiency
Primary area: generative models
TL;DR: We propose a novel paradigm named Sparse RAG, which seeks to cut computation costs of RAG through sparsity.

Abstract: Large language models (LLMs) augmented with retrieval exhibit robust performance and extensive versatility by incorporating external contexts. However, the input length grows linearly in the number of retrieved documents, causing a dramatic increase in latency. In this paper, we propose a novel paradigm named Sparse RAG, which seeks to cut computation costs through sparsity. Speciﬁcally, Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents. Then, LLMs selectively decode the output by only attending to highly relevant caches auto-regressively, which are chosen via prompting LLMs with special control tokens. It is notable that Sparse RAG combines the assessment of each individual document and the generation of the response into a single process. The designed sparse mechanism in a RAG system can facilitate the reduction of the number of documents loaded during decoding for accelerating the inference of the RAG system. Additionally, ﬁltering out undesirable contexts enhances the model’s focus on relevant context, inherently improving its generation quality. Evaluation results on four datasets show that Sparse RAG can be used to strike an optimal balance between generation quality and computational efﬁciency, demonstrating its generalizability across tasks.

Published as a conference paper at ICLR 2025
ACCELERATING INFERENCE OF GENERATION VIA SPARSE CONTEXT SELECTION
Yun Zhu', Jia-Chen Gu², Caitlin Sikora', Ho Ko², Yinxiao Liu', Chu-Cheng Lin",
Lei Shu', Liangchen Luo Lei Meng', Bang Liu", Jjinong Chen Google DeepMind
"Googl
Unniversity of California, Los Angeles
Université de Montréal & Mila
(yunzhu, csikora, hoko, canoee, &iooing com {leishu, luolc, leimeng, jdchen [dchen]@google (google. com gujcBucla. edu, bang. uuuuumnntreal. ca
ABSTRACT
Large language models (LLMs) augmented with retrieval exhibit robust perfor- mance and extensive versatility by incorporating external contexts. However, the input length grows linearly in the number of retrieved documents, causing dra- matic increase in latency. In this paper, we propose novel paradigm named Sparse RAG, which seeks to cut computation costs through sparsity. Specifically, Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents. Then, L.MS selectively decode the output by only attending to highly relevant caches auto-regressively, which are chosen via prompting LLMs with special control tokens. is notable that Sparse RAG combines the assessment of each individual document and the generation RAG
the response into a single process. The designed sparse mechanism system facilitate the reduction of the number of documents loaded during decoding for accelerating the inference of the RAG system. Additionally, filtering out undesirable contexts enhances the model's focus relevant context, inherently improving its generation quality. Evaluation results on four datasets show that Sparse RAG can be used strike an optimal balance between generation quality and computational efficiency, demonstrating its generalizability across tasks.
INTRODUCTION
Large language models (LLMs) have attracted increasing attention and exhibited impressive abilities understand instructions and generate fluent outputs in natural language (Brown al.) 2020 Ouyang et al.) [022] Touvron et al. 2023) Team et al. 2023). Nevertheless, LLMs inevitably manifest hallucinations (Ji et al. 2023) due to their struggle with factual errors and inability to secure the accuracy of generated text solely by the parametric knowledge they encapsulate (Zhang
al. 2023 Muhlgay al., 2024). Feeding the source of truth LLMs in the format of retrieved
context segments (Reid et al. 2024) alleviates this problem. The technique is widely known as
Retrieval-Augmenned Generation (RAG) (Lewis al. 20206 Li et al. 2022 Guu et al. 2020a).
Although the RAG framework is empirically shown to be effective, it can be expensive to scale up. This is because it requires prepending relevant documents retrieved from an external knowledge corpus to the queries (Guu et al. 2020a). As a result, the input length grows linearly in the number of documents, causing a dramatic increase in latency when using a standard Transformer whose latency scales quadratically with the input length. Some prior works such as Aision-in-Decodee (FiD) (Tzacard
& Grave 2021) and Parallel Context Windows (PCW) (Ratner et al. 2023) have proposed to alleviate this issue. Yet these methods fail strike an optimal balance between generation quality and computational efficiency. FiD was originally designed for the encoder-decoder architecture, and thus not compatible with currently prevalent decoder-only architectures without significant changes.

---

Published as a conference paper at ICLR 2025
Retrieval
question
Retrieved Documents
question
d
d
dx
[ASSESS]
Keep
LLM
Document Assessment
question
d
d
dx
[ASSESS]
Drop
...
question
d2
[ASSESS]
Load key-value cache that are assessed to keep
LLM
Generation
Cq
C
C2
Cx
[GENERATE]
Answer
Figure 1: An overview of Sparse RAG inference. Each of the retrieved documents d; is assessed for relevance by the LLM and irrelevant documents are dropped. Then, the KV caches e; for the remaining documents are used for generation.
While PCW can be applied to decoder-only LLMs, only speeds up the model pre-filling and still incurs high latency since the whole context window cache is still being attended when decoding each token. Moreover, the heavy reliance of generation on the retrieved knowledge raises significant concerns about model's behavior and performance scenarios where retrieval may fail or return inaccurate results (Shi et al.) 2023). A typical approach for mitigating this issue is to rely external classifier to rank or filter the documents before prepending them to the input (Yan et al. 2024). but this process requires extra model calls which adds new complexity to inference.
In light the issues above, we propose a novel paradigm called Sparse RAG. operationalizes through massive pre-filling, where the key-value cache is generated by single forward pass of the input tokens, and selective decoding, where the output is generated by attending to only highly relevant tokens nuto-regressively. Previous works where the length of the retrieved contexts during pre-filling are equal to that during decoding are called dense-RAG in this paper. Sparse RAG, on the other hand, causes the decoding context be significantly shorter than the pre-filled context, where retrieved documents that are not highly relevant to the input query have been dynamically dropped. Furthermore, Sparse RAG combines the assessment of each individual context and the generation of the response into single process, in which special control tokens are used to prompt the LLM to assess the relevance of each retrieved context, and then only the key-value caches of the most relevant contexts are loaded for decoding using another control token.
The design of Sparse RAG has two additional unique advantages. First, by reducing the number key- value cache loads during the decoding process, the LLM can achieve lower latency where it is typically constrained by memory usage. Second, filtering out undesirable contexts enhances the model's focus on relevant contexts, inherently improving the quality of the generated output. To demonstrate the effectiveness and efficiency of the proposed method, we evaluate on four datasets: PopQA (Mallen et al. 2023), QMSum (Min et al. 2023), TriviaQA (Joshi et al. 2017), and HotpotQÁ (Yang et al. 2018 Experimental results show that Sparse RAG can achieve similar or better quality and much better latency compared with standard dense-RAG PCW-RAG approaches. Moreover, the choice of the four datasets, which include short- and long-for generation, question answering, summarization, and multi-hop reasoning, demonstrates the generalizability of the Sparse RAG approach.
RELATED WORK
Retrieval-Augmente Generation RAG is family of techniques for generating output while using retrieved nearest- nearest-neighhor context data as a reference. It typically involves two stages: retrieval and generation. Retrieval finds most similar contexts based BM25 or learned embeddings, where
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons with existing RAG-related works.
Approach
Conrective
No extra model
Prefill efficiency
Decode efficiency
RAG (Lewis al.
2020b
No
Yes
No
No
Currective RAG Yan el a 2024
Yes
No
No
No
PCW RAG [Ratner el al. 2023
No
Yes
Yes
No Yes
Sparse (Aar(
Yes
Yes
Yes
the context can be represented as token embeddings (Khandelwal et al. 2020 Yogatama et al. 2021
dense embeddings (de Jong et al. 2022) or text (Guu et al. 2020b Izacard & Grave 2021
Lewis
et al. 2020bp. Once those contexts are retrieved, different architectures are leveraged to incorporate
them into the model. Popular approaches include concatenation (Izacard & Grave 2021
Lewis et al.
2020b) and cross-attention (Borgeaud et al. 2022 Lewis et al. 2020a).
In recent years, LLM architectures have evolved towards decoder-only models with significantly larger sizes. To this end, concatenation of raw text (Lewis et al., (2206) is becoming popular for its simplicity and practicality, and many advanced approaches have been developed
top of it. Yoran et al. (024) designed an NLI model to identify irrelevant contexts and improve robustness. Jiang al. 2023b) actively anticipate future content and decide when and what to retrieve in long-form generation. Self-RAG (Asai et al. 2024) is proposed to selectively retrieve knowledge on as- needed basis, by introducing a separate critic model. The critic model generates "reflection" tokens to indicate whether to retrieve information. It runs inference each document once and uses additional "reflection" tokens to select excerpts from the documents to use for generating the response. In contrast, unify the generation of the special control tokens and regular vocabulary tokens with one single model, eliminating the additional model and computational overhead. CRAG (Yan et al., 2024) explores and designs corrective strategies for RAG to improve its robustness of generation. Specifically, an external T5 model is trained and used to determine the usefulness of the retrieved context. Generally, these approaches explore retrieval useful tool augment generation and whether retrieval is necessary.
Efficiency in RAG different categories of approaches have been studied, often targetting LLM inference in general rather than RAG specifically. Some works focus on architecture-Ieve acceleration; examples include
The efficiency of LLM inference a widely explored research area, where
efficient attention (Shazeer 2019), Mixture of Experts (Fedus et al. 2022). Transformer--lternatii
architectures (Gu & Dao 2024), etc. Other works explore algorithm level acceleration like quantiza-
tion (Lin et al. 2024) speculative decoding (Leviathan et al. 2023).
Recently, RAG-specific methods have been explored. RAG Cache (Jin et al. 2024). for example, was proposed as multilevel dynamic caching system tailored for RAG from the system perspective. Another approach used in FiD (Izacard & Grave 202T) and PCW (Ratner et al. 2023) parallelizes processing of individual documents and climinates cross-document attention computations. FiD encodes each retrieved passage independently from other passages and decodes by attending over the concatenation of the resulting representations of all the retrieved passages. PCW carves a long context into chunks ("windows"), restricting the attention mechanism to apply only within each window, and re-uses the positional encodings across the windows.
Comparison with previous works that are the most relevant our work is illustrated in Table This work aims to strike an optimal balance between generation quality and computational efficiency. notable that the extra classifier in CRAG requires maintaining an extra model with more complex serving infrastructure; when there are N contexts retrieved, there are N + 1 model runs in total. Our work also relies on classification to refine the retrieved documents, but it is handled by an "internal" classification process that aligned with the generation process, so the total number of model runs in case is 1.
SPARSE RAG
Sparse RAG is designed for the decoder-only model architecture, which is the typical architecture of most popular LLMs. Figure presents an overview of Sparse RAG inference, in which document relevance assessment is used improve the robustness of generation. The key hypothesis of our approach is that the RAG task and per context assessment are similar tasks and the model can handle both in one shot using simple and effective training and inference techniques.
3

---

Published as a conference paper at ICLR 2025
3.1 TRAINING PROCESS
Our work assumes that a certain amount of RAG training data-on on the order of thousands of exam- ples-is accessible, which allows us to effectively tailor and adapt existing LLMs to our specific needs. In the training phase, we integrate an additional Per Context Assessment (PCA) task into the training mixture. By incorporating the PCA task, we aim enhance the model's ' ability to assess the relevance of retrieved documents and respond accurately in different RAG scenarios.
Data Augmentation with LLMs For typical RAG data, one question-anss -anwwer pair can be mapped to multiple retrieved contexts using either BM25 or an existing stand-alone retriever. However, there are cases where no golden labels indicating the quality of every retrieved context is available.
To collect these missing labels, we leveraged two off-the-shelff LLMs (Anil et al., 2023; Team et al. 2023)-PALM2 PALM2 and Gemini-to assess each context. We observe empirically that a second round of prompting for critique, especially using different model from the initial round, ensures the best quality labels. We provide our prompts in Table iinn in the Appendix. We compare different model combinations for labeling to human ground truth labels in Section 4
Multitasking Data Format The LLM trained on mixture of two types of tasks: Per Context Assessment (relevance rating) and answer generation. Specifically, we format the inputs and outputs of the two task types as
Per Context Assessment: (Question) (Conttett Compreoeee [Rtingg
Generation: {Question{Conneetll] [Contezt))...... (.CootCttt |Controlceneeaation oanpla/teeeeeaaaoooooooooooootee } {Answer)
where RRaiing) C"yes" for relevant "no" for irrelevant) and {Answer) the targets Assessment) and CControl_Generation are special control tokens ensure the LLM can differentiate the two tasks.
for the generative tasks and all tokens before them are inputs.
Parallel Contexts Since each context is rated independently in the PCA task, in which each example contains only one context, we introduce independence in the primary RAG generation training task well that the tasks can reuse the KV cache at inference. Thus for the generation task, we enforce no cross-attention between different retrieved contexts as in Parallel Context Windows (Ratner et al. 2023).
Specifically, we modify two things in the standard LM training process. First, we change the attention masks to be block-wise, and restrict Contezt, and Contert, from attending one another. [Question), {Contert,) and (Control Generation) } use the default causal attention mechanism, which the latter tokens attend all previous ones. Second, we use "parallel incremental" positional encodings to mimic the situation in which all retrieved contexts directly follow the query while maintaining the typical position ID of the below.
and (Answer) tokens as shown
0, 1. 2, 3, 4, 5, 3, 5, 6,
10. Question Contexti Contexta Control Generrtton
11, 12, 13
Answer
3.2 INFERENCE PROCESS
Given the question and retrieved contexts, Sparse RAG handles the assessment task and generation task in one single pass.
Per Context Assessment Similar to the training process, when pre-filling the KV cache, each retrieved context is treated independently by masking cross-document attention. The KV cache is used to score each context by concatenating the [Control Assessment] token. The relevance score is the probability of "yes"" (indicating relevance) being the next token. The position encoding allows this to happen in parallel.
Generation The generation uses a filtered KV cache, where only out of N cached values are loaded. We use simple threshold- based filtering approach: drop the context when its score is less than sigma. Once the cached KV vectors are loaded, the (Control Generation) token prompts the model to generate the answer.
4

---

Published as a conference paper at ICLR 2025
Table 2: Auto-rater comparison ground truth.
Auto-labeling method
Average FI
Fl Label 0
Fl Label
Rater model
Critic model
PALM2 XL
0.729
0.765
0.694
PALM2 XL
PALM2 XL
0.781
0,820
0.741
Gemini Ultra
n/a
0.761
0.807
0.716
Gemini Ultra
Gemini Ultra
0.704
0.747
0.660
PALM2 XL
Gemini Ultra
0.728
0.776
0.680
Gemini Ultra
PALM2 XL
0.821
0.861
0.781
4 EVALUATION OF PER CONTEXT ASSESSMENT
4.1 NEW ANNOTATIONS: NATURAL QUESTIONS PER CONTEXT ASSESSMENT
We isolated a subset of 50 questions, each with 10 retrieved contexts, from the Natural Questions dataset. We assigned 3 raters to each question-contex pair from pool of 7 raters and provided the instructions in Section
We aggregated responses for all raters for each context, selecting the majority decision 0 or 1 for each context. We found that raters unanimously agreed on 351 out of 500 context, with 30% of the documents considered relevant. For questions where raters were not unanimously decided, specialist rater was assigned to investigate more carefully and set the best label to correct mistakes of the other raters. This resulted in additional documents considered relevant out of the entire dataset, boosting the portion of relevant documents 31 % and slightly increasing alignment with the nater approaches (average F-score increase of I. 1.4% across auto-rater methods using these corrections as the ground truth).
4.2 LLM RATER COMPARISONS
We tested several different LLM-based automatic labeling methods-differeen combinations of models and prompts-for creating training data for the classifier in Sparse RAG. We compared several of these auto-rater approaches by creating a ground-trume relevance dataset using human labeling. The auto-rater comparison using the revised human labels the ground-truth is shown in Table We find that combining two different models in two rounds-initial prompting and critique-provide the labels that are most closely aligned with the human labels. We hypothesize that the different representations learned by two different models are able to capture the most nuance in the input sequences, leading better relevance judgements. We also observe that Gemini Ultra appears slightly less effective at critiquing model outputs than PALM2 XL.
EVALUATION OF SPARSE RAG
5.1 BENCIMARKS AND METRICS
PopQA is large-scale open-domain question answering (QA) dataset, consisting 14k entity- centric QA pairs. Each question is created by converting knowledge tuple retrieved from Wikidata using a template. We follow the setup from (Yan et al.] (024) and use Contriever (Izacard et al. 2022) retrieve the related contexts. Since PopQA does not include per-context assessment relevance labels, we adopted the "Gemini + PALM2" combination to create training labels. We split the dataset into training, validation and test sets with 8:1:1 ratio. Since the answer is usually short, we report
Exact Match (EM) and F1 scores.
QMSum (Zhong et al.) 2021) is human-ummann annotated benchmark for query-based muuli--domin meeting summarization task, which consists of 1,808 guery-summary pairs over 232 meetings in multiple domains. To adapt it the RAG domain, divide each conversation into different contexts where each turn in the conversation a context and the average context contains 300 words. Note that this dataset has human labeled per-context assessments that leverage during training. We use 250 training examples (one per meeting). 70 validation examples and 77 test examples. The targets for this dataset are longer and we report RougeLSun and F1 scores.
5

---

Published as a conference paper at ICLR 2025
TriviaQA (Joshi et al. 2017> is realistic text-based question answering dataset that includes 950K question-answer pairs from 662K documents collected from Wikipedia and the web. Similar to PopQA, used the "Gemini PALM2" combination to create relevance training labels. We randomly selected 8k training examples and 500 validation and test examples each. We report Exact Match (EM) and F1 scores.
HotpotQA (Yang et al., 2018) question answering dataset containing about 113K crowd- sourced questions that are constructed to require the introduction paragraphs of at least two Wikipedia articles to answer, thus requiring multi-hop reasoning. We sample 6k training examples and 600 validation and 600 test examples. We report Exact Match (EM) and Fl scores.
These datasets were selected to demonstrate generalizability across the question answering tasks requiring single- multi-hop reasoning, diverse context and output lengths, and summarization capabilities.
5.2 BASELINES
We evaluated the performance of standard concatenation-basse RAG where an L.M generates output given the query prepended with all the top-ranked documents using the same retriever as Sparse RAG system. RAG finetuned with the training data.
RAG
Off-the-shelf We report a variant of concatenation based RAG where the model is not finetuned with training data.
LLMLingua In this approach an external LLM was called to compress the prompt (Jiag et al. 2023a). In our comparison, we chose the compression ratio to be the same Sparse RAG for fairness.
PCW-RAG We applied Parallel Context Windows (Ramer al. 2023 the RAG process, where no cross-attention is applied between documents. The model is finetuned with the training data.
We evaluate CRAG using external T5-XXL classifier trained using heuristic labels (Yan et al., 2024). This classifier is used to process the documents and decide the rank. Note to facilitate fair comparison, we did not adopt the "web search" feature of this paper.
Corrective RAG
5.3 EXPERIMENTAL CONFIGURATION
The base LLMs used in the paper were Gemini Team et al. 2023). Although our approach could be applied at different training stages of the model, we apply LoRA tuning (Hu et al. (002) to enforce alignment on top of the foundation LLMs due to its low resource requirements and wide usage. Note that the same LoRA tuning the training data is applied to Sparse RAG and all baselines. In all our experiments, we apply LoRA in self-attention and use the default rank as 4. By default, we use the XXS size of Gemini which can on-device.
During training, we use 64 Tensor Processing Units (TPU) V3 chips for PopQA while use 128 Units for the other datasets. The batch size is 64. We the Adafactor optimizer (Shazeer & Stern 2018) with a learning rate of 0.003. The training dropout rate is 0.05. We leverage the metrics of the validation set to pick the best checkpoint. During inference, the temperature is set to 0.5. Unless specifically noted, we use sampling decoding with sample number for our experiments.
5.4 INFERENCE SETUP AND METRICS
Evaluation of Sparse RAG was conducted Samsung $21 Ultra, utilizing the device's CPU to assess real-world performance on a relatively mid-tier smartphone compared to the latest flagship models. Inference configuration consisted of fixed token lengths for queries, contexts and generated responses. This setup allows for evaluating the system's 's efficiency and effectiveness under resource constraints typical of mobile devices, providing insights into its practical applicability for on-device question-answenn tasks. Specifically, the overall inference process considers two stages.
Prefill stage For the baseline RAG model, measure the total time taken process all input tokens (question and all contexts). For PCW RAG and Sparse RAG models, we take advantage of these models' ability cache the question KV vectors. We first measure the time process the

---

Published as a conference paper at ICLR 2025
Table 3: Quality & efficiency tradeoff for both short-form and long-form generation tasks; Sparse RAG achieves both higher quality and efficiency compared to "dense" RAG approaches.
Dataset
Metrics
Off-the-shelff
LLMLingua
RAG
PCW-RAG
CRAG
Sparse RAG
ES
56.28
56.28
147.58
147.58
EM
0.33
1.96
65.43
65.04
66.52
67.71
FI
12.76
12.15
69,99
69.54
70.99
71.16
PopQA
K
20.00
7.84
20.00
20.00
8.9
7.84
DS
6.65
6.65
6.65
12.28
20.37
22.28
21.43
20.18
23.96
RougeSum
12.67
18.37
18.20
16.95
20.10
QMSum
K
20.00
4.45
20.00
20.00
4,45
DS
6.65
6.65
6.65
16.05
EM
0.00
2.6
46.20
46.00
47.50
FI K
12.21
16.30
53.03
53.20
55.10
TriviaQA
20.00
9.90
20.00
20.00
9.90
DS
6.65
6.65
6.65
10.18
EM
0.00
1.33
43
38.83
43.50
FI
12.21
14.67
55.85
50.03
55.36
HotpotQA
K
10.00
6.50
10.00
10.00
6,50
DS
10.31
10.31
10.31
13.00
question alone. Then, we measure the time to process different contexts with the pre-processed K cached question. We use Encoding Speed (ES) (tokens per second (t/s) to measure the efficiency of the prefill stage.
Decoding stage To assess the decoding speed comprehensively, using the same question but varying the amount of relevant context data considered, ranging from the top I most relevant document the top K. For each context size, produce output sequences of different lengths. This systematic approach allows us to evaluate the impact of both context size and response length decoding speed. We Decoding Speed (DS) (tokens per second (t/s)) to measure the efficiency of the decoding stage.
generate output sequences
5.5
MAIN RESULTS
We report both quality and latency metrics in Table "K" is the number of chosen contexts. Note that the CRAG approach relies on its classifier, which is exclusively trained on the PopQA dataset. Thus we only compare its performance on PopQA. Additionally, both LLMLingua and CRAG leveraged external classifiers, for which we cannot effectively measure ES and DS. We discuss the end-to-end latency of the external classifiers Table
Notably, our proposed approach achieves the best quality while being the most efficient during inference compared to other approaches. can be seen that Sparse RAG shares the same pre-filling efficiency with PCW-RAG, due the parallel context encoding, but it achieves significantly better quality than PCW-RAG amd better decoding efficiency than standard RAG and PCW-RAG. To illustrate. out of 20 retrieved contexts, Sparse RAG has average of 7.84 contexts for PopQA and 4.45 contexts for QMSum. This leads to almost double or even triple the decoding speed. Meanwhile, Sparse RAG achieves higher quality metrics than the dense counterparts, demonstrating that Sparse RAG effectively filters noisy and irrelevant contexts.
HotpotQA is the only dataset where Sparse RAG does beat RAG's Fl score. PCW-RAG has particularly large quality gap on HotporQA, suggesting that masking cross-document attention may hinder multi-hop reasoning capabilities. However, Sparse RAG recovers quality to level similar to RAG while maintaining lower latency, demonstrating the power of our context selection process.
We also observe that Sparse RAG outperforms CRAG on quality, suggesting that our "in-place" classifier may be outperforming CRAG's external T5 XXL classifier trained on the same dataset.
5.6 ANALYSIS
Impact Confidence Threshold Table iibluttilustrates how our metrics vary with different quality thresholds for Sparse RAG. As the threshold gradually increases, the system filters contexts,

---

Published as a conference paper at ICLR 2025
reducing the number of contexts K and consequently the latency during inference. The response quality metrics increase with increasing threshold up to a certain point, showing the effectiveness of filtering out irrelevant contexts. Then, the performance is stable and eventually drops slightly, possibly because some relevant contexts are accidentally filtered out.
Table 4: Sampling various confidence threshold values. A higher threshold means fewer contexts.
PopQA
QMSum
Threshold
EM
F1
K
DS
RougeLSum
K
DS
0.05
66.95
70.97
9.75
10.61
22.85
19.49
7.92
11.92
0.1
66.84
70.66
8.72
11.70
23.78
19.98
6.68
12.89
0.15
67.17
71.16
7.84
12.28
23.43
19.66
5.77
13.01
0.2
66.77
70.54
7.13
12.88
23.2
19.79
5.05
14.54
0.25
65.75
69.64
6.56
13.00
23,96
20.1
4.45
16.05
0.3
63.86
68.2
5.98
13.08
23.84
19.99
3.93
16.38
Number of Prefilled Documents To assess whether Sparse RAG's quality improvements the result of "massive" prefilling which is not practical in real scenarios, we compare different numbers
of prefilled documents (10 and 20) for PopQA. Results shown in Table 5 Even with fewer documents prefilled, the quality of Sparse RAG remains better than RAG.
The gap between RAG and Sparse RAG is relatively small at 10 prefilled documents compared to 20 because there are fewer documents to be filtered. Moreover, for a smaller number of documents, the cross-document masking is less "sparse" compared to larger number of prefilled documents. Meanwhile, only using the top 5, 3, or documents introduces significantly lower EM and FI scores because difficult guarantee high-quality retrieval the first step. This further motivates our design to increase the range of retrieved documents and then perform context selection.
Table 5: Ablation different number of prefill documents for PopQA.
Approach
Prefill Documents
EM
F1
K
ES
DS
RAG
146.10
46.01
50.12
1.00
22.74
RAG
120.36
55.28
59.32
3,00
18.96
RAG
102.51
58.66
63.49
5.00
15.98
RAG
10
64.66
68.67
10
80.74
10.31
PCW RAG
10
63.9
68.58
10
147.48
10.31
Sparse RAG
10
65,86
70.2
7.79
147.48
12.33
RAG
20
65.43
69.99
20
56.28
6.65
PCW RAG
20
65.04
69.95
20
147.58
6.65
Sparse RAG
20
67.17
71.16
7.84
147.58
12.28
Inference Efficiency Ablations We present Table demonstrates the computational advantages
of Sparse RAG for each stage of the retrieval and generation process compared to other methods. Sparse RAG, like PCW-RAG, reduces encoding latency to nearly 1/3 that of RAG. While CRAG's encoding latency appears lower than Sparse RAG, it has an additional classification step, which is
Table 6: Latency Decomposition.
External
Init Time (ms)
Init Copy Percentage
Encoding (ms)
Copy (ms)
Decoding (ms)
Total (ms)
End-To-End
K
Classifier
(ms)
RAG
20.00
0
120
90962
4811
95893
0.13%
PCW-RAG
20.00
0
17
34716
151
4811
39697
0.43%
CRAG
8.9
40200
56
27362
2878
70497
0.08%
Sparse RAG
7.84
0
17
34716
56
2605
37396
0.20%
8

---

Published as a conference paper at ICLR 2025
slow in our experiments because used an older T5 model with more attention layers and fewer modern optimizations like kernel fusion and flash attention. This drives CRAG's total latency above that of Sparse RAG. It is possible that different classifier could reduce this cost, but it would still duplicate the operations of encoding the context during the prefill stage. Sparse RAG also reduces the decoding latency nearly half that of RAG and PCW-RAG via context filtering. CRAG has similar decoding latency to that of Sparse RAG because it also uses context filtering, but overall Sparse RAG still much faster.
We also explore trends in decoding latency by varying the number of retrieved contexts (i.e., top-K documents) and the length of the generated responses. As illustrated Fig 2b] we observe that RAG requires over 50% more time to generate outputs of varying lengths compared to the Sparse RAG approach. As shown in Fig 2a this heightened demand for computational resources results in notable slowdown in decoding speed. This underscores the efficiency advantages offered by Sparse RAG, especially scenarios requiring a larger number of contexts during decoding.
top top
output 32 takens
top
sutput 128 takens
10
output 256 tokens
Larancy for desodingten
30
Decading spand(es))
18
22-
36
15-
10-
10-
0
30
300
150
200
250
12
34
18
Number of decode tokens
number chosen memories
(a) E2E latency for decoding different number of tokens. (b) Decoding speed with different number of contexts.
Figure 2: Inference Efficiency Comparison.
Ablation on Foundation Model Size We applied Sparse RAG to different sizes of LLMs by testing on Gemini XS and Gemini XXS. The results of these experiments are presented Table The findings demonstrate that Sparse RAG is compatible with various foundation models, effectively adapting different model sizes. Notably, with reduced amount decoding caches, Sparse RAG is capable of achieving the highest quality results. This indicates that Sparse RAG maintains its efficiency and effectiveness across different foundation models, making it versatile approach for various LLM configurations.
Table 7: Ablation on different model sizes.
Approach
Model Size
EM
F1
RAG
XS
66.52
70.87
PCW RAG
XS
65.75
70.37
20
Sparse RAG
XS
68.26
72.26
6.27
RAG
XXS
65.43
69.99
20
PCW RAG
XXS
65.04
69.95
20
Sparse RAG
XXS
67.17
71.16
7.84
Sparse RAG with Full Attention During Generation To demonstrate the isolated effects of our document assessment and filtering method and evaluate the potential quality regressions caused by omitting cross-da rross-document attention scores during generation, we perform the Sparse RAG Per Context Assessment step followed by the generation step with full attention. As you can full attention during generation provides a slight quality improvement, but Sparse RAG quality is very close and much more efficient.
Table using

---

Published as a conference paper at ICLR 2025
Table 8: Comparing Sparse RAG with full attention Sparse RAG and RAG on PopQA.
Approach
EM
F1
RAG
65.43
69.99
Sparse RAG
67.71
71.16
Sparse RAG w/ full attention
67.94
71.24
Silver Labels vs LLM Labels Corrective RAG, the T5 model was trained with silver labels that come from title matching Yan al., 2024). We the same silver labels to replace the LLM labels and train the Sparse RAG model with this new dataset. We also train the CRAG model on our LLM labels for comparison. From the results shown in Table generated by the LLMs is slightly higher than that of the silver labels from Yan et al. (2024) leading to higher accuracy and lower K values. We also observe that when trained with exactly the same labels, Sparse RAG still outperforms CRAG on quality. We hypothesize that the superior quality of the -.neerree labels comes from our two-round process of soliciting responses from two different L.MMs. By engaging two distinct models, we likely enhanced the robustness and accuracy of the labels through form of cross-vais pooss-vlidattin,, thereby mitigating potential biases or errors that might arise from relying on single LLM.
observe that the quality of the labels
Table 9: Comparing CRAG silver labels our L.M labels on PopQA.
Approach
EM
F1
DS
Sparse RAG w/ silver labels
66.97
71.05
8.26
11.99
Sparse RAG w/ LLM labels
67.71
71.16
7.84
12.28
CRAG w/ silver labels
66.52
70.99
8.9
CRAG w/ LLM labels
67.03
71.02
Using Golden Context Labels During Inference Since QMSum provides golden per-context labels, we leverage these labels during inference to evaluate the upper bound performance of the Sparse RAG approach under the condition of perfect per-context assessment. The results of this experiment are presented in Table TO) demonstrating the full potential of the Sparse RAG method pending additional context assessment quality improvements.
Table 10: Trying golden labels on QMSum.
Approach
Fl
RougeLSum
K
DS
Sparse RAG
23.96
20.1 21.93
4.45
16.05
golden label
26.76
1.13
21.16
CONCLUSION
This paper presents Sparse RAG to address the challenges of increased input length and latency. Through novel approach of massive pre-filling and selective decoding, Sparse RAG efficiently manages the key-value cache of retrieved documents, allowing the LLMs to focus on highly relevant tokens. This selective attention mechanism not only reduces the computational burden during inference but also enhances the generation quality by filtering out irrelevant contexts. Evaluation on four diverse datasets validates Sparse RAG's ability to achieve balanced trade-off' between high-quality generation and computational efficiency, proving its versatility and effectiveness for both short- and long-form content generation tasks. This innovative paradigm showcases the potential for improving LLM performance in various applications by optimizing context management and inference processes.
Future research will explore Sparse RAG in multimodal contexts, investigating how Sparse RAG can handle and integrate information from multiple types of data to improve its performance and applicability across diverse scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like thank Yu-hui Chen, Yuqi Li, Qifei Wang, Zonglin Li, Zhong Meng, and Alec Go for their suggestions.
REFERENCES
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Sia- mak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hell
Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, et al. PaLM 2 techmical report. CoRR, abs/2305. 10403, 2023, doi: 10.48550/RKXI.2905.000m
URL ht https::/doil org/10 48550/ wtt.://aaivv...00. 2ttp:///....00//////222220000000
10403
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. net www./formm220 wwimmidd=hhsh@oooo
URL http:::
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- Saan-taptstte Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, et al. Language models are few-shot learners. In Advances in neural information processing systems, pp. 1877-1901, 2020.
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William W. Cohen. Mention memory: incorporating textual knowledge into transformers through entity mention attention. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net 2022. URL http::::////m http:////eeeeeeieeeeeeeeeee / net ffrummidd OY1 A 8e FQ
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Reseurch, 23(120):1-33 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Retrieval aug- mented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3938. PMLR, 2020a. URL http:://pocceeeaaiiiee html
pi..ppees//119/qquee
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020b.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 874 880. Association for Computational Linguistics, 2021. doi: 10.18653/V 1/202 RECL-MAIN 74. URL https eacl ma 74

---

Published as a conference paper at ICLR 2025
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022. URL
http://wwwwwew.eeeeeeeeeeeeeeeee fer/forummid=
jKNI 1KNlpXi70 760
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishi,, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12):248:1 248:388, 2023. doi: 10.1145/3571730. URL htttps://oi..oo /tgg///1/ooor//0////
11 1145/357120
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LImlingua: Compressing prompts for accelerated inference large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6- 10. 2023, pp. 13358 13376. Association for Computational Linguistics, 2023a. doi: 10.18653/V1/20222 http:///12222.......2 WMN..---INI.IISI... o/g 1 0. 8653/v wttp:///w222/////
URL https
p - ma n.n.inn.... 825
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 7969-7992. Association for Computational Linguisties, 2023b. URL https: acc antho ogy. org/
2023. wmw.p-mannnmnnmmmm 495
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-aummentedd generation. arXiv preprint arXiv:2404. 12457,, 2024.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.). Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume I: Long Pupers, pp. 1601-1611. Association for Computational Linguistics, 2017. doi: 10. N0.18653/V1/P17-11477 URL https
//doi.org/10.11 1ttp://00..665/111111111-1
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization
through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL/attps://ppenzeee/e
net fett/oorumiid-Hkk... CEEKH
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. -9274-19286 PMLR, 2023.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. Advances in Neural Information Processing Systems, 33:18470-18881, 2020a.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. sarrivaa--augmented generation for knowledge-intenee intensive NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020b.
neurips. cc cttccc//ppppeee pct/pappee/22////aaaeae 7481e5 Abstract
Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-auggmeened text generation. CORR, abs/2202.01 10, 2022.
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Guangxuan Xiao, and Song Han. AWQ: activation- aware weight quantization for device LLM compression and acceleration. GetMobile Mob. Comput. Commun., 28( 28(4) 1Tq/100.145/371445 3774447777
2024. doi: 10.1145/7714883.374499 URL https://dol
12

---

Published as a conference paper at ICLR 2025
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non paraameetiic memories. In Anna Rogers, Jordan L. Boyd--Grrabb and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational I Linguistics (Volume I: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 9802-98222 9822. Association for Computational Linguistics, 2023. doi: 10. 110653311/2233ALL-LLLO
URL pattps:///oi.... /ttp//oii.oo/oo////////o
18653/v1/2023 .00022..... acl -ong- 546
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 12076-12100 Association for Computational org/ 2023, emnlp-main. 741
Linguistics, 2023. URL https: http:///aaaathhhhhco
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evalua- tion of language models. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisties, EACL 2024 Volume Long Papers, St. Julian's ss Malta, March 17-22, 2024, pp. 49 66. Association for Com- wtq///2//00 wacle
putational Linguistics, 2024. URL https:
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkín, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022. URL 64a 73
wttpp/scccpppppecpppeepaea nips Sc httpp/ppppprrpaeep les htpppaappeere//ee 1 /202 hash/ 731 Abst coostractt.oo wowett--eeoorreeeeeeeeeee html
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua. Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Vol- ume I: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 6383-6402. Associ- ation for Computational Linguistics, 2023, doi: 10.18653/V 12218663//1/2223.44 ACL-L2..CL- URL https: wrw.oo..orgg..... 11153/61/60233333
ong 352
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint 1:Xiv:1911.021500 2019.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context, In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 23-29 Jul 2023. URL https:
mlr www.ss/eeeee.............eee whw.aa.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312.118000 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language
13

---

Published as a conference paper at ICLR 2025
models. CoRR, abs/2302.139771 2023. doi: 10.4850/ARXXV.330.13377 URL https
//doi.org/1..4250 48550/arxiv. 2ttp:///......00//......2222220 230..
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation.
arXiv preprini arXiv:2401.15864 2024.
Zhilln Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings Empirical Methods in Natural Language Processing, Brussels, Belgium, October November 2018, pp. 2369-2380, Association for Computational Linguistics, 2018. doi: 10.18653/V 1/D11/ 1259. URL https: //oi. org 10 www.1//1//111----
of the 2018 Conference
Dani Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics, 9:362-373, 2021.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented lan- guage models robust irrelevant context. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:://ppenrete
net wttp://oouodddda//emeee m 4k2pH
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's song in the Al ocean: A survey on hallucination in large language models. CoRR,
abs/2309.01219, 2023. doi: 10. D04450/ARXXI.230000022 48550/arXiv. 2309. 01219
URL httt:s::/aoi..rr/////
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. Qmsum: A new benchmark for query-based multi-domain meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zeutleooerr Dilek Hakkani-Türr Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Pruceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 3905-5921. Association for Computational Linguistics, 2021. doi: 1/266533V112211NAAAAACC naacl-main. 472
472. URL wtt.tt://////.. //do1 org 10/ 10. (01318653vvv/ 111.1.1165///ww22222
14

---

Published as a conference paper at ICLR 2025
A PROMPTS AND INSTRUCTIONS
A. I PROMPTS USED FOR LLMS
We share the prompt used for calling LLMs to get per context assessment in Table
Table II: The zero-shot prompts for LLM labeling and critique.
Round 1 prompt
You are now doing reading comprehension task. It is important that you be as thorough, detail-oriented, and accurate as possible in your response.
You are given question, a set of accepted answers, document and its title. The document does necessarily contain the right answer to the question. You should read the title and the document and then check if they provide of the correct answers to the question.
If the title and document together contain the correct answer to the question, output score of 1.0, otherwise output score of 0.0. question: iquestion;
accepted answers: janswerst title: ititle;
document: :document;
output:
Round 2 prompt
Your job correct another model's performance a reading comprchension task. given question, set of accepted document and title do not necessarily contain the right answer. The model instructed to output score of 1.0 if the document contains the answer, and a score of 0.0 otherwise.
The model
document and its title. The
You will be given the same information the other model along with its output. You should read the title and document and then check if they provide one of the correct answers to the question.
Then check if you agree with the previous model's output. If you agree, output the same score unchanged. If you disagrce, output corrected score. Your output should he accurate as possible. question: iquestiong
accepted answers: janswerst title: ;title;
document: (document; previous model's 's score: iscorez
output:
A.2 RATER GUIDELINES
We share the instructions provided to the human labelers in Table 12
Table 12: Instructions for raters creating ground-trut truth relevance dataset.
Human Rater Instructions
Please read the question. answer the context. Please answer the context can help answer question. it can, select Otherwise select
1: good
0: bad
Please use the answers as a hint. However, do not use "is the answer in the context?" as heuristic for making the decision.
B DATASET ANALYSIS
During the human labeling process, several raters flagged documents and questions that were difficult to label. In total, 23 out of 500 documents flagged and 15 out of 50 questions flagged.
15

---

Published as a conference paper at ICLR 2025
We explored several ways of filtering human-labeled subsample of Natural Questions to determine how they impacted context assessment F-scores overall and for each auto-rater. We provide two additional filtered versions of the human-labeled RAG relevance dataset alternativss. See Table [3] for the auto-rater F-scores for each filtering method. Both statistical filtering approaches (e.g. removing contexts with non-unanimous labels) and targeted filtering approaches (e.g. removing questions or contexts flagged by human raters) lead to some improvement in F-scores for relevance labels, but in all cases, using Gemini Ultra the rater and PALM2 XL as the critic model provides the highest Average Fl score.
Table 13: Evaluation of Labeling Methods w/ Filtering
Dataset Filters
# Docs
% Relevant
Rater Model
Critic Model
Average Fl
PALM2 XL
n/a
0.729
Specialized rater 500
PALM2 XL
PALM2 XL
0.781
31
corrections
Gemini Ultra
n/a
0.761
Gemini Ultra
Gemini Ultra
0.704
PALM2 XL
Gemini Ultra
0.728
Gemini Ultra
PALM2 XI. 340B
0.821
PALM2 XL.
n/a
0.741
Filter
PALM2 XL
PALM2 XL
0,811
non-unanimous docs
351
23
Gemini Ultra
0.792
Gemini Ultra
Gemini Ultra
0.739
PALM2 XL
Gemini Ultra
0.763
Gemini Ultra
PALM2 XL
0,856
PALM2 XL
n/a
0.750
Filter flagged docs and questions
PALM2 XL
PALM2 XL
0.797
330
29
Gemini Ultra
n/a
0.782
Gemini Ultra
Gemini Ultra
0.741
PALM2 XL.
Gemini Ultra
0.753
Gemini Ultra
PALM2 XL
0.833
observed trends questions and contexts shared in Table 14 that raised concerns about whether and how human would be able to assess the relevance of the context. These concerns extend to expectations of how well LLMs would do at the task.
Most of these concerns involve the absence of sufficient context to correctly answer the question in widely used public datasets. For consistency with the literature, we did not modify the queries or the retrieved contexts in this paper, but we do expect that the ambiguity is impacting cases, retrieved contexts would be assessed differently for relevance depending on the true intended meaning of the question, and different answers would be expected.
results. In many
would be an interesting future exploration to augment the datasets to resolve such ambiguity, as it would likely improve the accuracy of relevance labels and improve performance overall. For examples with missing time or location context, you could augment the datasets by simply adding new contexts with the missing information appending the context the query. For example, to handle the question "Who is the current president of the United States?", you could add a context "Today November 18, 2024." or you could modify the query to "Who is the current president of the United States on November 18, 2024" In the paper, we showed that Sparse RAG performs comparably to RAG on the HotpotQA multi-hop reasoning dataset, so we expect that Sparse RAG could effectively leverage information from new contexts in the generation step. However, expect that appending such temporally specific context to the query itself would likely yield the best quality, all contexts attend to the query.
In real world cases where the question is ambiguous and sufficient context cannot be retrieved to confidently answer the question, we believe that the most desirable behavior would be to request clarification before further action. Future work should explore simulating such a scenario in order to assess RAG question-answering approaches.
16

---

Published as a conference paper at ICLR 2025
Table 14: Overview of trends, datasets, and examples with associated comments.
Trend
Dataset
Example question
Comments
who is the president of usa right now
NQ
Depends when the question is asked.
Questions with time- dependent answers
who is the current di- rector of the us mint
NQ
Depends when the question asked.
when is the next dead- pool movie being re- leased
NQ
Depends when the question asked.
total number of death row inmates in the us
NQ
Fluctuates time.
The borders in this region and the name of the country have changed several times in 20th century.
What is Prague the capital of?
PopQA
The accepted answers are "actor, actress, actors, actresses". He was an actor later in his career, but he rose to prominence professional basketball player.
What is Dennis Rod- man's occupation?
PopQA
in which regions most of Africa petroleum and natural gas found
"Region" can refer to different levels of granularity (c.g. Sub-Saharan Africa vs. Ethiopia). but the only accepted answer is "Nigeria".
Missing synonyms, abbreviations, and aliases, combined with unclear granularity
NQ
The accepted answers are only "off-road ve- hicles", "light utility vehicles", "sport util- ity vehicles", but "SUV" clearly correct answer as well.
what type of cur is jeep
NQ
The accepted answers include 16 cast mem- hers, but the show went for 25 with many cast changes and guest stars not included in the list.
Non- exhaustive list of answers
cast of law order special victim unit
NQ
The only correct answer is "constitutional right", but that included in the question. It's 's not clear what type of answer would he appropriate here.
right to property ac- cording to the consti- tution of India
Oddly phrased question
NQ
The only accepted answer here is "Long Island of 1922". but the place is Long Island and the question does not ask about when story is
Overly spe- ciñic answers expected
where does the story the great gatsby take place
NQ
There a musician and a film called "Frances", and both of those could arguably have genre associated with them.
Question refers entity with common name without disambiguation
PopQA
What genre Frances?
The question referring formed by Christina Aguilera but there are many other songs, movies, and other en- tities that share the name and also have producer.
song per-
Who the pro- ducer Hurt?
PopQA
What is the capital Cherokee County?
There are many different Cherokee Coun- ties in different states in the USA.
PopQA
17


Reviews:
[{"review_id": "wXI2Jv4gwn", "reviewer": "Reviewer_p444", "summary": "The paper proposes a sparse RAG which consists of two stages – 1) document assessment, and 2) generation, to improve the efficiency of RAG. The document assessment selects the relevant documents by individually checking their relevance, and generation part only uses the selected documents as a context to generate an answer, but in an efficient manner where documents are cached, and cross-attention b/w documents is not allowed. To my understanding, the efficiency is obtained here, because all previous information could be cached (no recomputation) after document assessment. Experiment results show that a sparse RAG leads to improvements in terms of both effectiveness and efficiency, i.e., it improves RAG, more efficiently than PCW RAG.", "questions": "Please see weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is clearly written and the proposed sparse RAG by appending document assessment stage is well validated. \n2.\tIn the experiments, the proposed method leads to additional improvements in terms of both RAG and sparse RAG. It is more efficient than RAG, and it is remarkable the sparse RAG shows improved performances over RAG.", "weaknesses": "1.\tSome parts in the presentation are not clear. In multitasking data format, more details of the format need to be provided. E.g. What is the information of Rating field? In Figure 1, [ASSESS] and [GENERATE] are special tokens? Detailed position information for cached document representations used in document assessment and generation. Document assessment does not use the cached representations in input?  \n2.\tContext reduction methods have been widely studied for RAG. Document assessment method needs to be validated separately. Other context compression and reduction baselines (not only RAG) need to be compared.\n3.\tGiven the restricted cross-attention manner across passages, there is concern that the general ability is still well-maintained. SparseRAG needs to be further validated under other advanced CoT, long chain of reasoning, and instruction following tasks. \n4.\tDocument assessment can also be parallelizable, using restricted cross-attention method? Still, it remains unclear whether the current decision of relevance for documents in an one-by-one manner is optimally designed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nnndN4Rrf5", "reviewer": "Reviewer_ARp8", "summary": "This paper presents SparseRAG, an efficient framework designed to address the scaling challenges of traditional RAG models. The primary issue tackled is the significant increase in memory usage of the key-value (KV) cache when processing large amounts of retrieved lengthy documents. This issue hurts the ability of large language models (LLMs) to efficiently manage long-form context and capture essential content.\n\nTo overcome this, the authors propose SparseRAG, which utilizes a required LLM rater and an optional LLM critic to generate Per-Context-Assessment (PCA) training data. The LLM is then trained on this PCA data to enhance its capability in assessing the relevance of retrieved documents. During both training and inference, after retrieving the related contexts, the LLM first evaluates the relevance of each retrieved documents and simultaneously extracts the KV caches, which are then concatenated to generate the final outputs.\n\nThe experimental results indicate that SparseRAG significantly improves both prefill and decoding efficiency. Additionally, it outperforms all other baseline models in generating high-quality outputs. Further analysis and ablation studies confirm that SparseRAG is both an efficient and effective framework.", "questions": "See the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is clear and easy to follow, addressing a genuine concern.\n\n2. The proposed method demonstrates efficiency and effectiveness compared to the baselines.\n\n3. Most ablation studies and analyses are convincing.", "weaknesses": "1. Ablation Study Suggestion: It would be beneficial to include an ablation study to evaluate the impact of concatenating selected KV caches instead of the corresponding raw contexts. This approach aims to save computation time, but it’s important to assess any potential performance loss compared to recalculating KVs with full attention on concatenated raw contexts. Understanding this trade-off is crucial, as certain applications might accept additional computation time if it results in significant improvements.\n\n2. Comparison with KV Cache Compression Methods: Since the paper claims that SparseRAG enhances efficiency during the prefill and decoding stages by adopting context selection, it would be compelling to compare it with KV cache compression methods such as H2O, SnapKV, and KVMerger. Although these are typically training-free methods, and thus challenging to directly compare, such an analysis could provide valuable insights into the relative benefits and limitations of SparseRAG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CRq5kbit3q", "reviewer": "Reviewer_5W8k", "summary": "The paper proposes \"Sparse Retrieval-Augmented Generation\" (Sparse RAG), a paradigm to accelerate large language model (LLM) inference by selectively loading only the most relevant retrieved contexts. This approach aims to improve computational efficiency and generation quality by filtering unnecessary documents during decoding, evaluated across diverse datasets with positive outcomes in both quality and speed.", "questions": "1. Does reusing the KV Cache during batch inference cause a decline in the quality of responses when the document-level attention scores are missing?\n2. Why does the use of an external classifier in the CRAG method increase so much latency? (My understanding is that the classifier should be a model with relatively few parameters, and it should be able to perform batch inference.) Is the training data for the classifier consistent with the PCA task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposes an innovative sparsity-driven retrieval method (Sparse RAG) that enhances both inference efficiency and output quality.\n2. Successfully demonstrates generalizability and adaptability through evaluations on multiple tasks and datasets.\n3. Offers thorough comparative analysis against existing methods, providing clear evidence of computational and quality improvements.\n4. Utilizes a practical on-device evaluation setting to substantiate real-world applicability, particularly on resource-constrained devices.", "weaknesses": "1. The paper does not provide experimental comparisons to validate that the absence of inter-document attention scores during inference does not lead to a significant decline in the quality of responses. This aspect could be further explored to strengthen the claims made regarding the effectiveness of the Sparse RAG approach.\n2. The training process involves multi-tasking and parallel context encoding, which may introduce additional complexity and require careful tuning to ensure effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N8RQX306c1", "reviewer": "Reviewer_4HHs", "summary": "This paper proposes SparseRAG, a method to improve efficiency/effectiveness and reduce computation costs through sparsity. It encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents. Then, LLMs selectively decode the output by only attending to highly relevant caches auto-regressively.  This is via prompting LLMs with special control tokens. Sparse RAG can facilitate the reduction of the number of documents loaded during decoding for accelerating the inference of the RAG system. It filters out undesirable contexts and enhances the model’s focus on relevant context, improving its generation quality. Evaluations on four datasets show good results on improving generation quality and computational efficiency across multiple tasks.", "questions": "The points outlined in the “weakness” should be clarified in the rebuttal.\n\nAlso, in your appendix, you mentioned \"We observed trends in questions and contexts shared in Table 12 that raised concerns about whether and how a human would be able to assess the relevance of the context. These concerns extend to expectations of how well LLMs would be able to do that as well.\" \n\nFor example, you mentioned that in the dataset NQ, a query like \"who is the president of USA right now\", and you commented that \"the answer will depend on when the question is asked\".  \n\nThis is true but I assume that is exactly why RAG could be useful since RAG will retrieve additional information based on the context of the query, and in many cases, information in the query, its surrounding context and the supporting datasets should be able to disambiguate the time period and thus return the correct answer.   I think some discussion on this is necessary.  It would be helpful if the authors may elaborate on how the Sparse RAG approach specifically handles such time-sensitive or ambiguous queries. The authors may include examples or experiments demonstrating how their method performs on queries that require temporal context, or a discussion on how such capabilities can be incorporated in the Sparse RAG framework.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tOverall, the paper is clearly written.  \n\n2.\tThe motivation of SPARSE RAG is natural, and the design of the approach seems to be innovative and reasonable.\n\n3.\tThe claim that “Sparse RAG combines the assessment of each individual context and the generation of the response into a single process, in which special control tokens are used to prompt the LLM to assess the relevance of each retrieved context, and then only the key-value caches of the most relevant contexts are loaded for decoding using another control token.”  This seems to be interesting and novel, but some detailed explanation is needed to help readers understand the process.  For example, it will be good to explain how the special control tokens work in practice, or to provide a step-by-step explanation of how the context assessment and response generation are combined.", "weaknesses": "1.\tThe design assumes that a good amount of RAG training data is accessible, which allows effective tailoring and adapting of existing LLMs to the specific needs. But this design relies on pretty heavy training data, for example, in the PoPQA testing, the dataset is split into training, validation and test sets with 8:1:1 ratio (and other testing follows a similar spirit).  I am wondering in practice, an LLM will take many kinds of queries and refer to different kinds of external data, where the training pairs may not be available.  How does your method work if such training data is not available, or when your data and applications do not match the statistics in your training data?  I would suggest that the authors conduct additional experiments with varying amounts of training data to demonstrate the method's robustness. Additionally, it will be useful to discuss potential strategies for adapting the method to domains where extensive training data is not available.\n\n2.\tThe design does not do any concrete semantic analysis on query intent and nor does any concrete semantic analysis on the text to be retrieved.  It would be helpful to discuss the potential benefits and drawbacks of incorporating semantic analysis into this approach. The reviewer is wondering whether the authors should discuss the potential effect of semantic analysis of the queries and texts and why they chose not to include them in their current design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5deCvX5wrN", "reviewer": "Reviewer_pGev", "summary": "This work proposes Sparse RAG, a novel RAG framework that reduces computational costs by leveraging sparsity. Sparse RAG encodes retrieved documents in parallel and allows LLMs to selectively decode by attending only to the most relevant cached information. Experiments on four datasets demonstrate that Sparse RAG effectively balances generation quality with computational efficiency.", "questions": "1. What is the detailed difference between Parallel Context Windows (PCW) and the Parallel Contexts the authors proposed? A specific example would be helpful.\n2. In line 203, what is the {Relevant} token?\n3. In Figure 1, do $c_{i}$ and $d_{i}$ represent the same thing? If not, what’s the difference between them?\n4. Is Parallel Contexts used for both assessment and generation? The authors state that \"we unify the generation of the special control tokens and regular vocabulary tokens with one single model,\" but line 188 mentions that Parallel Contexts is used only for the \"generation task.\" Does this represent a contradiction?\n5. In the section *Silver Labels vs. LLM Labels*, could an experiment be added to evaluate Corrective RAG using LLM Labels? The improvement shown in Table 5 appears minimal, which may be influenced by controllable or uncontrollable factors.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed framework effectively balances generation quality with computational efficiency.", "weaknesses": "1. This work lacks novelty; the authors' primary contribution lies in scoring documents and using an existing framework with modified position encodings to achieve parallel encoding.\n2. The experiments lack ablation studies for the module, especially an experiment where only Parallel Contexts is added to RAG (for only generation task).\n3. Some parts of this work are not clearly written, making them difficult to understand. For example, lines 201 to 204, with line 204 being especially unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)