xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Authors: Xin Cheng
Xun Wang
Xingxing Zhang
Tao Ge
Si-Qing Chen
Furu Wei
Huishuai Zhang
Dongyan Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: retrieval-augmented generation
context compression
dense retrieval
Primary area: natural_language_processing
TL;DR: We propose an efficient and effective compression method for retrieval-augmented generation.

Abstract: This paper introduces xRAG, an innovative context compression method tailored for retrieval-augmented generation. xRAG reinterprets document embeddings in dense retrieval--traditionally used solely for retrieval--as features from the retrieval modality. By employing a modality fusion methodology, xRAG seamlessly integrates these embeddings into the language model representation space, effectively eliminating the need for their textual counterparts and achieving an extreme compression rate. 
In xRAG, the only trainable component is the modality bridge, while both the retriever and the language model remain frozen. This design choice allows for the reuse of offline-constructed document embeddings and preserves the plug-and-play nature of retrieval augmentation. 
Experimental results demonstrate that xRAG achieves an average improvement of over 10% across six knowledge-intensive tasks, adaptable to various language model backbones, ranging from a dense 7B model to an 8x7B Mixture of Experts configuration. xRAG not only significantly outperforms previous context compression methods but also matches the performance of uncompressed models on several datasets, while reducing overall FLOPs by a factor of 3.53. Our work pioneers new directions in retrieval-augmented generation from the perspective of multimodality fusion, and we hope it lays the foundation for future efficient and scalable retrieval-augmented systems.

xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Xin Cheng
Xun Wang?
Xingxing Zhang?
Tao Ge
Si-Qing Chen
Furu Wei
Huishuai Zhang
Dongyan Zhao
Peking University Microsoft
National Key Laboratory of General Artificial Intelligence chengxin 1998 @ stu, wtt.pku..ddu.oo
Abstract
This paper introduces xRAG, novel context compression method designed specif- ically for retrieval-augmented generation. xRAG redefines the use of document embeddings dense integrating them features from the retrieval modality. Through modality fusion approach, xRAG effectively merges these embeddings into the language model's s representation space, eliminating the need for their textual counterparts and achieving an extreme compression rate. In xRAG, the modality bridge the only trainable component, while the retriever and language model remain frozen. This design choice allows for the reuse of preserves the plug-am sou-andd-lay nature of retrieval augmentation. Experimental results demonstrate that xRAG achieves an average improvement of over 10% across six soowledge-intensivee tasks, compatible with various language model backbones, ranging from dense 7B model to an 8x7B Mixture of Experts configuration. xRAG not only significantly outperforms previous context compression methods but also matches the performance of uncompressed models on several benchmarks, while reducing overall FLOPs by factor of 3.53. This work pioneers new avenues in retrieval-augmented generation through multimodal fusion, potentially setting a groundwork for future developments in efficient and scalable retrieval systems.
limited to retrieval purposes- by
document embeddings and
Introduction
Retrieval-Auumeente Language Models (RALMs) 42 22 9 14
70 have shown exceptional per-
formance in variety of snwwledge-intnnsive tasks. By retrieving domain-specific, long-tailed, and up-to-date knowledge from a non-parametric datastore, RALMs significantly extend the boundaries of parametric Large Language Models (LLMs). However, the integration of entire documents into prompts can significantly increase inference costs and may surpass the context limit of LLMSs 29 79]. As illustrated in Figure while the inclusion of relevant document enables the L.MM to generate accurate responses, it does at the expense of processing documents that expand the original query by more than tenfold.
How might mitigate the costs associated with extended context while maintaining the benefits of retrieval augmentation? Recent research interest has converged on a promising direction: Context Compression. This concept is pursued through two primary strategies: soft- cettpprompting methods, such as Gist [59]. AutoCompressor [15), and ICAE [20], which compress the context into dense memory slots, and hard-prompting methods, such as LLMLingua (29 and RECOMP (79., where
Work done during interssii at Microsoft, corresponding Xun Wang, Huishuai Zhang and Dongyan Zhao
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RAG
to background document answer the question:
Background: Motel Beginning in 1986, Motel 6 has advertised through radio commercials featuring the voice of writer and National Public Radio commentator Tom Bodett, with the lagline "We'll leave the light on for you." The ads were created by Dallas advertising agency The Richards Group. They feature tune composed by Tom Faulkner, performed by him on gultar and Milo Deering on fiddle. The first spots were conceived and written by David Fowler. 1996, the ads won a Award. The campaign itself has won numerous national and international awards and was selected by Advertising Age magazine as one the Top 100 Advertising Campaigns of the Twentieth Century."
Question: What company advertised itself with the slogan "We'll leave light on for you"?
Output: Motel
w/o Retrieval
xRAG
Answer question:
Refer to the background document and answer the question:
What company advertised itself with the slogan "We'll leave light on for you"?
Question:
Background:
What company advertised itself with the slogan "We'll leave light on for you"?
Question:
Output: Marriott International.
Output:
Figure xRAG enables efficient retrieval augmentation by adding one document token [X].
compression is applied on the surface form. These approaches, however, either require significant memory for storing L.M activations (e.g., 1.05 MB per token as reported by (59]) or suffer from relatively low compression rates. More critically, these methods overlook crucial characteristic of RALMs: through large-scale contrastive learning with question-document pairs, modern dense retrieval systems already distill document content into a single high-dimensional embedding, and this embedding reveals (almost) as much information as the text [58 44
In this paper, we pioneer innovative approach to retrieval augmentation and context compression through the lens modality fusion. Drawing from multimodal research, where text-only language models taught to "perceive" and "listen," pretrained modality encoder like CLIP [64] is typically used to extract modality features. These features are then integrated into language models using a
Building the conceptual overlap between the retriever encoder and modality encoder, we introduce xRAG. This model redefines document embeddings from dense solely for retrieval purposes- retrieval modality features. xRAG employs modality fusion methodology to seamlessly integrate these embeddings into the language model's representation space, thus obviating the need for textual counterparts and achieving significant context compression. In xRAG, the modality bridge is the only trainable component, while both the retriever and the LLM are kept frozen. This design decision facilitates the reuse of pre-constructed document embeddings and maintains the plug-and and-play nature of retrieval augmentation two essential factors for functional RAG system.
modality fusion bridge [45 52 34
To verify the effectiveness and versatility our framework, we conducted comprehensive experiments with different LLM backbones, ranging from dense 7B model 8x7B Mixture of Experts model. Our results reveal that adding just one document token could lead to over a 10% improvement across six knowledge-intensive tasks, significantly surpassing previous compression methods. xRAG also delivers results comparable to uncompressed models on several benchmarks. This is remarkable considering that the only trainable component constitutes less than 0.1% of the LLM's parameters. In terms of efficiency, xRAG reduces total FLOPs by a factor of 3.53 compared to the uncompressed RAG model. We further provide detailed analyses of xRAG, examining various training strategies. data blends, and component selections for the retrieval system. We believe this research sets a strong foundation for the development of future efficient and scalable aitreval-augmente systems.
2 Related Work
Retrieval-augmented Generation Equipping parametric language model with non- non-parametric datastore has proven effective for range of NLP tasks, including language modeling [37] 57 87).

---

Datastore
Who is the nichest person the world of 20247
Sentence
Projector
Who is the richest person the world of 20249
(a) xRAG
Language Model fø
Bernard Arnoult
Comprsssor (0ptonal)
Token Embaeding
Sentence Embedding
Bernard Arnoutt (born March 1949) French businessmon, and collector... CEO the global goods company LVMH, dozens of luxury brands.. overtook Elon Musk dazline Musk's wealth, 3 $194.5 billion, was
Queryy
Who is the richest person in the world os of 2024?
Concato
Lyode Maxence
Prompt
(b) RAG
Figure 2: Overview of xRAG (a) and RAG (b). For given query, RAG typically concatenates the retrieved document with the query, significantly extending the context length. In contrast, xRAG addresses this issue through modality fusion by directly projecting the document embedding into the LLM's representation space. This allows for efficient retrieval-augnie -ugmentatioon with the addition of only one token.
open-domain question answering 25 [2]. among others. Given the vast design space of this generation paradigm, numerous approaches with different focuses have been proposed. For instance, RETRO [9] and PlugLM [13] introduce
42
70
86 domain adaptation 7 and machine translation
architectural innovations for enhanced integration with the non-parametric datastore. REALM [22] pioneers end-to-end approach for simultaneous optimization of the language model and retriever. REPLUG (70 and RA-DIT [51] improve retriever alignment using feedback from LLMs. DSP 38 and InteR [18] investigate complex interactions between the retriever and the language model. Selfmem [44 utilizes a reward model to refine retrieval and generation iteratively. Self-RAG [4] incorporates self-reflection mechanism to enhance the quality and factuality of language model outputs. For a detailed overview, see 19 5 3 Our contribution, ARAG, stands out by implementing modality fusion approach to retrieval augmentation, creating effective and efficient RAG system.
Context Compression Context compression, aimed reducing the input length for LLMs while retaining essential information, has recently attracted substantial interest [7]. Gist [59] achieves compression rate of up to 26x by modifying the attention mask and caching soft gist token activations.
ICAE AutoCompresso [15], and 500xCompressor 48) condense lengthy contexts into succinct, compact memory slots, which are directly utilizable by LLMS for diverse functions. LLMLingua and CompAct(82) introduces a coarse-to-fine prompt compression technique based on perplexity scores and distilled token-level score. While these methods are generally applicable, others are tailored specifically for RAG systems, such as FilCo (76) and RECOMP [79]. A concurrent work directly employs passage embeddings for efficient listwise reranking [53]. For an in-depth
29 30 62
comparison of these compression methods regarding memory efficiency, compression rates, and adaptability, refer to Appendix[A]
3 Methods
Problem Formulation In retrieval-augmented generation, a non-parametric datastore ID {(E,, |P) consists of pairs where each D; represents a document chunk as a sequence of L; tokens D. (di........ .. Correspondingly, E, the dense representation derived from a sentence embedding model SE,(.) (-) with input D,. For an input query g. its dense representation SEa(g) is used to find the relevant documents by matching against the collection [E with certain similarity search algorithm such as MIPS. After retrieval, the system selects a relevant pair (E, D) from TD, concatenates the chosen document D with g. and processes the combined input with a language model
(d.....,

---

Bossmett
Docanent Bernard Arroul (bern 4 Franch took Elan Nush us the richest pareon 2024 deciine
Sermad Amoult (bom March 1849) o Freech busieessonn. Elan michest 2024 27% decline Muskis wealth, From $245,3 billan
RAG Distribution
Quardon
rickeet
Lystt
$245.3 bilkson
the 20240
Language Model Fip
Arewer
Lunguage Modul F
Lx
fall
Projector
+
Parsphrass the above text
Projector
+
Who is the richest person
Sentence Encoder
Sentence Encoder
Distribution
(a) Paraphrase Pretraining
(b) Context-aware Instruction Tuning
Figure 3:
Two-stage training strategy of xRAG including (a) Paraphrase Pre-training unlabeled
corpus and (b) Context-aware Instruction Tuning optimized with labeled data and self-distillation.
(D @ q). Optionally, a context compression module can be integrated to reduce the length of from to a more concise 4, achieving a compression ratio of 4.
3.1 xRAG Architecture
Traditional methods for document compression typically focus surface form of the document
76 79.. In contrast ARAG tackle the problem from a modality fusion view. Concretely, we introduce modality projector W, which is trained directly project the retrieval features E into the
29
I.M representation space. Our proposed framework is visually contrasted with the traditional RAG system in Figure 2 In the standard RAG, the input to the LLM comprises the embeddings Emb(D @ q) of length D + where Emb signifies the embedding layer of the LLM. Conversely, with xRAG, the modified input represented as W (B) Emb(q), which yields substantially reduced length of 1 + |g]- In this framework, the challenges come from the modality fusion: How can a text-only language model understand features from retrieval modality? To achieve this, we explore two-stage training strategy: Paraphrase Pretraining followed by Context- aware Instruction Tuning.
3.2 Paraphrase Pretraining
Similar to the pretraining strategies employed vision-language models that use image-captioning
data to align two modalities [52] 16 54, the primary objective of our paraphrase pretraining is
to build a compatible representation between the extracted retrieval feature and the corresponding document. Illustrated in Figure aa. for each pair (E, D) in retrieval corpus D, we employ natural language instruction Xinetruct to prompt the LLM to undertake a paraphrasing task (c.g. "[X] The above text could be paraphrased as: [D]", where [X] and [D] placeholders for W(E) and document D this setup, the model learns to connect ((E) and by recovering D on the condition of ((E) and the model is optimized by:
Cull Σ
log (s(d: (()), X Xistrruct, dei)
(1)
where p& is given by the softmax distribution of LLM Fo. and dei denotes the document token before current prediction token d,, achieved by casual attention mask in auto-rogressive LMs.
3.3 Context-aware Instruction Tuning
After the pretraining phase, although the language model F & has developed an internally compatible representation, it has never been explicitly trained to utilize these features for downstream tasks. To address this gap, we proceed to instruct the model in harnessing the fused feature (F)) by continually training the model on data where the answer is closely associated with the given context, including reading comprehension, summarization, and open domain question answering data. We
TT maintain diversity, sample from instruction pool. which could be found in Appendix B

---

constructed an mixed dataset, containing approximately million entries from open-souree datasets, as detailed in Appendix C For each triplet in the dataset, (X (annaxxt: Xquestion annver)) we initially obtain the sentence representation for contoxt via the emhedding model Econtext SE X contoxt Subsequently, we refine the optimization of on two directions:
Optimization I: Language Modeling. Aligned with established instruction tuning methodologies [4] 43] our objective is finetune the model so that it generates the correct output when provided with specific instruction, conditioned upon the given context information. Unlike tradi- tional models that utilize the textual context X context: our method employs dense feature Econtext to encapsulate the context information:
Call Σ log Po X anaver, W (Bcontextt) X Xquaation; X Xanavar <i)
Optimization II: Self-Distillatio.. The second trajectory of optimization aims to guide the xRAG in the effective utilization of contextual information, drawing from the principles of self-distillation 71 and imitation learning [6123] 23] By considering the RAG model as a "teacher" and xRAG as a "student", we endeavor distill the knowledge from RAG, thereby enabling xRAG to emulate the RAG model's proficiency in handling the full, uncompressed documents. This approach enhances xRAG's resilience in scenarios where encounters noisy or irrelevant context that may not directly lead to the correct answer, detailedly discussed in & 6.1 Concretely, for a language model F using Ecentaxt as the source of context, our objective is minimize the divergence between the two resulting output distributions. This discrepancy is measured using the Kullback- Leibler (KL) divergence:
either X contaxt
Cu DKL (X
(anever Baanever/oooneeee Xcontexts |l P (anaveer W (Econtext);
(3)
Here Xqueation is omitted for brevity and the final loss is the linear combination controlled by hyperparameter: Luul + oeLas.
3.4 Design Principle
In designing the projector W. our primary objective is to maintain the simplicity of the framework. We therefore opted for two-layer MLP while other more sophisticated module such as Q-Former 45) could also be considered. Notice that the projector is the only trainable component, accounting for only 0.46% of the total parameters in the Mistral-7b model and 0.07% in the Mixtral-8x7B model. Such design choice departs from previous studies that necessitated uull-parameter tuning to adapt We believe this approach will likely be more accessible and practical because, fundamentally, the RAG itself functions as a plug-and-play module for LLMs, and so should its compressed version. This design also avoid the risk of compromising other core
LLMs for compressed contexts [59] 76 T5).
capabilities of L.LM during palammeerrt tuning, as observed in 55 56
Moreover, in contrast other compression methods that necessitate storing LLM activations for each compressed token (59] 20 15) 15]-an an impractical strategy in the RAG setting, given the millions of documents involved our method introduces no additional memory overhead. Instead, leverages document embeddings, originally designed for retrieval. To summarize, xRAG not only simplifies the integration process but also avoids unnecessary computational or memory expenses.
4 Experimental Setup
4.1 Evaluation Dataset
We evaluated the performance of xRAG primarily on knowledge- weweeegg-inttnsivee tasks which encompass a range of challenges: (1) three Open Domain Question Answering datasets that address questions on wide array of topics: Natural Questions 41). TriviaQA (33, and Web Questions [8]. (2) one Multihop Question Answering dataset, HotpotQA 81, which necessitates multi-step reasoning to generate answers. (3) one Long-form Question Answering dataset, TruthfulQA [50], that requires the generation of long-form and truthful responses. (4) one fact-checking dataset, FactKG [39], which challenges the model to use complex reasoning to determine the factual accuracy of given claims.
5

---

In line with the KILT 63 and GenRead [4], we assessed three ODQA datasets and HotpotQA using the Exact Match (EM) metric, FactKG with Accuracy, and for the long-form QA, we used both the Fl score and Rouge-I. (R-I.) score. These tasks demand broad spectrum of world knowledge and have been extensively explored the retrieval-uggmentttion literature
70
4.2 Implementation Details
To demonstrate the versatility of our framework, we choose two backbones, differing in scale and architecture: Mistral-7b [27] and Mixtral-8x7b (28). For the retrieval corpus, utilized the Wikipedia dump from December 2021, which was pre-processed into passages following the methodology described in [26). This resulted in approximately 37 million passages, each averaging 180 tokens in length. Our default retrieval model is the SFR [T] which, at the time of writing this paper, holds the leading position the MTEB leaderboard [60] We top-/ ranked document for
inclusion in our instruction-tuning dataset and for the evaluation of downstream tasks. More details provided Appendix D Code is available https://githunn com/ hawaniaal004////
4.3 Baselines
In determining appropriate baselines for comparison, we adhered to fundamental principle: the selected compression methods must support the general plug-and-play capability of retrieval augmen- tation. This entails that they should function effectively without the need for dataset-specific tuning 77 76), or any alteration to the parameters of LLMs [76, [9]. Furthermore, given the extensive volume of the retrieval corpus, it is essential that these compression methods demonstrate memory efficiency, specifically by not requiring the storage of LLM activations for each individual token
55 20 [5]. With these criteria in mind, our evaluation compares xRAG to the following baselines: that operates without retrieval augmentation and another that includes it. These serve as the lower and upper performance bounds for our study of compression techniques, respectively. (II) Additionally, our comparisons extend to LLMLingua (2), a plug-and-play approach for context compression. (III) Taking inspiration from (59] we incorporate method of discrete compression using TF-IDE This approach yields compression rates comparable to those achieved by xRAG and serves as the lower bound for discrete compression.
(I) Primarily, consider variants of LLMs:
Table 1: Experimental results on six downstream tasks. The best results are in bold and the second best are with underscore, Percentage in the brackets denotes the relative improvement over non- retrieval setting. LLMs are frozen during the experiments and retrieved documents are the same for different compression methods. and denotes different compression ratio.
NQ
TriviaQA WebQA
HotpotQA Multihop QA (EM)
TrutufulQA Long- form QA (F1/R-L)
FactKG Fact Checking (Ace)
Doc Length
Open Domain QA (EM)
Average
Task Type
Mistral-7h
retrieval
34.89
27.02
26.23
25.51
$4.78
36.54 (0.0%)
W retrieval
42.71
65.88
37.84
38.79
26.50
25.92
67.76
43.63 (19.4%)
175.1
"with Compression
ingua" 30.64
57.94
32.63
29.91
25.70
25.10
64.17
38.01 (4.0%)
98.6
LLMLingua
28.81
$7.09
32.33
29.13
26.10
25.39
63.57
37.48 (2.5%)
61.1
TFF
35.43
25.
(2.6%)
XRAG
39.10
65.77
39.40
34.05
28, 10
27.71
63.08
42.46 (16.2%)
Mixtral-8x7h
w/o retrieval
41.99
71.10
40.31
32.87
25 60
24.90
62.64
42.76 (0.0%)
0
w retrieval
45.15
70.34
41.26
43.46
27.10
25.80
70.42
46.22 (8.0%)
175.1
with Compression
LLMLingua! 37.65
67.70
36.02
35.66
25.99
25.39
67.98
42.32 (-1.0%)
96.6
iiguuaa
37.81
67.81
35.78
35.27
25.68
25.00
68.03
44,17 (-1 (-1.3%)
61.1
TF-DF 41.19
69.94
41.63
32.05
26.80
26.00
66.17
43.41 (1.4%)
1
XRAG
47.28
74.14
44,50
39.66
27.80
26.64
68.20
46.91 (9.7%)
Experimental Results
5.1 Knowledge Intensive Tasks
In Table we present our main results. Across both Mistral-7b and Mixtral-8x70 configurations, we observe consistent and significant improvement when retrieval augmentation is applied (p-value

---

< <0.05), although the gains are more modest for the larger model configurations. This trend aligns with observations reported by 77 [T]. Further analysis on the efficacy of various compression techniques reveals that xRAG outperforms other approaches by large margin. Remarkably, xRAG not only reduces the token count drastically-fron 175.1 to a single token-but also maintains robust performance levels. In some instances, xRAG's performance is comparable to, or even exceeds, that of the uncompressed models. Specifically, in the Mistral-7b configuration, xRAG achieves nearly the same performance improvement the uncompressed model (16.6% compared to 19.4%), and in the Mixtral-8x7b configuration, it surpasses the uncompressed model (9.7% compared to 8.0%). One possible reason lies in the vulnerability of current RAG system when the irrelevant misleading documents are presented, a topic detailed discussed in $ We also observe that xRAG performs well in tasks that require document understanding, such TriviaQA. However, in tasks that demand reasoning over document, like HotpotQA and FactKG, xRAG lags behind by a considerable gap.
Table 2: Comparison of RAG and xRAG performance in CUDA Time and GFLOPS.
CUDA Time (ms)
GFLOPs
RAG
xRAG
Improvement
RAG
xRAG
Improvement
FactKG
431.5
215.6
x2.01
4683.8
1289.5
x3.63
NQ
918.7
611.3
x1.51
1338.6
384.0
x3.48
TriviaQA
807.1
512.1
x1.57
1667.2
492.3
x3.38
WebQA
872.6
577.3
x1.51
1405.1
386.8
x3.63
Average
x1.64
x3.53
5.2 Computational Efficiency
In this section, conduct thorough assessment of our framework's computational efficiency and memory management. To rigorously evaluate our model, we employed Torch Profiler to measure the CUDA Time (milliseconds) and Giga FL.OPs of both the RAG and xRAG models across four real-world datasets. In these evaluations, the Mistral-7b, operating in 6 inference mode, served as the base LLM. CUDA Time and GFLOPs were calculated with a fixed batch size, and GFLOPs were normalized by the number of generated tokens. These experiments were performed on the same computational hardware, specifically an Nvidia A A100 and an AMD EPYC 7V12 64-Core Processor. As depicted in Table 2 despite variations in prompt and generation lengths across the datasets, xRAG significantly outpaced the RAG model, achieving x1.64 increase in CUDA Time efficiency and a x3.53 reduction in GFLOPs.
average per batch basis
Analysis
6.1 Evaluation Beyond the Overall Score
Although retrieval augmentation generally boosts performance shown by aggregate metrics, it may not uniformly benefit all instances. In certain cases, the retrieval system might provide irrelevant or
even misleading information, leading incorrect answers that were previously correct [33 76 5) To enable a more fine-grained evaluation, we introduce two novel metrics: the Resilience Rate and the Boost Rate. The resilience rate quantifies the percentage of instances in which the system's responses remain correct both before and after retrieval augmentation, highlighting the system's 's stability and robustness. Conversely, the boost rate measures the percentage of instances that were initially answered incorrectly but were rectified following the introduction of retrieved document, thereby assessing the efficacy of retrieval augmentation. An ideal RAG system should have both high resilience rate and boost rate.
In Figure 4 we display these metrics for the uncompressed RAG and two compression methods: LLMLingua and xRAG. Surprisingly, although retrieval augmentation generally enhances perfor- mance, the resilience rate for RAG averages only 75.2%. 2, indicating that retrieval adversely affect about one-quarter of previously correct answers. In contrast, xRAG demonstrates considerable

---

robustness across all evaluated datasets. This robustness largely stems from xRAG's ability to main- tain an unbiased stance toward the internal knowledge representation of the LLM, especially when confronted with noisy retrieval content. Similar trends are noted in [51 55 where search-augmente instruction learning is shown to bolster the robustness of language models. However, xRAG still lags behind RAG in boost rate, particularly in multi-hop reasoning tasks. It is crucial to note that a high resilience rate does not necessarily mean that the LLM disregards the provided information, which could potentially lead to reduced boost rate. A comparative analysis with LLMLingua indicates that xRAG is not only more robust but also more effective.
Resilience Rate
91.0
LLMLingua
XRAG
RAG
315
78.9
72.4
70
66.7
NQ
WebQA
HotpotQA
TriviaQA
Boost Rate
30
29.0
20.0
17.8
15.4
15.7
12.
NQ
WebQA
HotpotQA
TriviQQA
Figure 4: Resilience rate and boost rate of three augmentation methods: LLMLinuga, xRAG and RAG over a Mixtral-8x7b baseline without retrieval augmentation.
6.2
What makes xRAG effective?
Table 3: Ablation on different training strategy for xRAG.
Averaged
NQ
TriviaQA
WebQA
HotpotQA
Performance
Resilience
Boost
Mistral-7b
xRAG
39.10
65.77
39.40
34.05
44.58
82.3%
22.2% %
w/o finetune
30.14
59.48
35.19
26.70
37.87
66.6%
20.8%
w/o pretrain
31.25
59.07
41.19
24.32
38,95
79.8%
14.1%
w/o nll
35.46
65.27
39.57
31.80
43.02
83.7%
19.4%
w/o self-kd
34.99
64.33
39.22
27.45
41.49
76.2%
20.8%
LoRA
35.71
60.14
40.45
22.91
39.80
76.0%
18.0%
Mixtral-8x7b
xRAG
47.48
74.14
44.50
39.66
51.45
84.9%
20.0%
w/o finetune
34.46
64.08
34.89
30.43
40.96
65.9%
17.8%
w/o pretrain
42.54
71.17
47.44
31.23
48.09
85.0%
14.2%
w/o nll 45.10
72.85
45.03
37.11
50.02
84.8%
18.9%
w/o self-kd 42.38
72.26
44.73
32,41
47.94
79.8%
18.9%
This section delves into a thorough evaluation various elements that contribute to xRAG's overall performance, focusing on its training strategy, the blend of datasets used and the effect of different embedding models. Due to the space limit, we present the last factor in Appendix
1. Training Strategy We carefully ablate four optimization choices: pretraining, instruction tuning, modeling (nll) and self-distillation (self-kd). We also train a Mistral-7b with LoRA on our instruction tuning dataset to rule out the possibility that our improvement simply comes from tuning on more data. The outcomes are presented in Table 3 Our analysis reveals that the interplay of different training strategies significantly contributes to the
and two optimization

---

RAG
Refer to the background document and answer the question:
Background: The Old Man and the Key| several "washed-up" celebrities, was composed by Alf Clausen and written by Vitti. Even though received sale credit for writing the song, parts of the lyrics were written by Simpsons Carolyn Omine, which were then revised by fellow writer Selman. Selman was especially satisfied with the rhyme sung by Mr. inside." " The episode features American actress Olympia Dukakis as Grampa's love interest Zelda. American comedian Bill Saluga also makes an appearance, as his television character Ray J. Johnson. The owner of the minivan is portrayed by series regular Hank Azaria, who imitated the voice Clark Gable the character." X
stated that whenever he feels
he thinks of that rhyme and that it "boosts [him]
Question: Which US comedian'actor credited saying only as
feels'?
Output: Bill Saluga.
X
w/o Retrieval
xRAG
Answer question:
Refer background document the question:
Which US comedian/actor is credited with saying 'A man is only as old as the woman he feels'?
Question:
Background:
Question: Which US comedian/actor is credited with saying 'A man is only as as woman feels'?
Output: Groucho Marx.
Output: Groucho Marx.
Figure 5: Given the misleading document, RAG model tend to generate wrong based the document, while xRAG demonstrate its robustness by leveraging the internal knowledge of L.M..
performance our framework. the case of Mistral-7b, pretraining and finetuning phases are of equal significance to the end results. However, for Mixtral-8x7b, the impact of pretraining is notably diminished, likely due to the larger model's enhanced capability to incorporate multi-modality information. Furthermore, find that during finetuning, self-distillation is more important than language modeling. The primary advantage of delf-distillation lies in bolstering the resilience rate of the xRAG system. Optimization with nll loss tends cause overreliance context information, rendering the system more vulnerable when the retriever fails to fetch relevant documents.
Table 4: Abaltion results on different data selection strategy.
Train
NQ
TriviaQA
WebQA
HotpotQA
Performance
Average
Resilience
Boost
xRAG (Mistral-7bb
955k
39.10
65.77
39.40
34.05
44.58
82,3
22.2%
w RC only
488k
36.98
65.77
41.39
32.82
44.24
81.9%
22.4%
w QA only
385k
36.45
65.57
41.14
31.80
43.74
80.5%
22.1%
Summ only
81k
36.37
64.95
40.40
31.98
43.42
78.8%
22.8%
II Instruction-tune tunnng Dataset Blend As discussed in 43.3
instruction-tumi dataset primarily comprises three categories: reading comprehension, open-domain QA, and text summarization. To explore the effects of different data blends, instruction-tur three xRAG model variants, each using data from these distinct categories. The results are shown in Table4 Our analysis reveals that among the dataset blends, reading comprehension data most significantly enhances the xRAG model's performance, as evidenced by both high resilience and boost rates. Intriguingly, when tuned solely with summarization data, XRAG still manages deliver strong performance QA datasets it has never been exposed to. This finding underscores that the advantages of instruction tuning for xRAG are not rooted in task-specific knowledge. Instead, they derive from the model's improved ability to utilize projected context information effectively.
6.3 Case Study
In Figure we show one interesting case about the robustness of xRAG. When retrieval system provide misleading content, standard RAG would overly rely on the document generate answer that are faithful to the document while not factually true. Our xRAG model opt rely the internal

---

knowledge of LLM and being robust to the misleading content. In Appendix H we include more cases about xRAG including several error analysis.
Conclusion
In this work, present xRAG, innovative context compression method tailored for retrieval- augmented generation. For snooledge-intenstie tasks, xRAG can be significantly faster than RAG while maintaining comparable performance. We excited about the future of this modality- -baee reureeeal-uugmented system and plan to further improve its performance the areas reasoning over embedding, handling multiple documents, and combining with multi-vector retrieval.
8 Acknowledgement
We would like to express our sincere gratitude to the anonymous reviewers for their thorough review, insightful comments, and constructive suggestions, which have significantly improved the quality of this manuscript. This work paper is supported (in part) by the State Key Laboratory of General Artificial Intelligence.
References
[1] Sfr-embedding-miise mistral::: Enhance text retrieval with transfer learning.
https://blog
saleesforceairesearchh com/ ceton/sfr-emeeddee -itteedddddmisstr////
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and deff-distillation in deep learning. arviv preprint arXiv:2012.09816 2020.
[3] Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Yun-Nung (Vivian) Chen, Margot Margot, and Siva Reddy, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Tutorial Abstracts), pages 41 1646, Toronto, Canada, July 2023. Association for Computational Linguistics.
[4] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023.
[5] Akari Asai, Zexuan Zhong, Dangi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannanch Hajishirzi, and Wen Yih. Reliable, adaptable, and attributable language models with retrieval, 2024.
[6] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song. Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: human generated machine reading comprehension dataset, 2018.
[7] David Beauchemin, Zachary Gagnon, and Ricahrd Khoury. Quebec automobile insurance question-answerigg answering with retrieval- -augeeeed generation. arXiv preprint 2024.
[8] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1543,1544, 2013.
[9] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens, 2022.
[10] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions, 2017.
10

---

[11] Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset, 2021.
[12] Xin Cheng, Shen Gao, Lemao Liu, Dongyan Zhao, and Rui Yan. Neural machine translation with contrastive translation memories, 2022.
[13] Xin Cheng, Yankai Lin, Xiuying Chen, Dongyan Zhao, and Rui Yan. Decouple knowledge from parameters for plug-and-pa and- language modeling, 2023.
[14] Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrieval-augmeentdd text generation with self memory, 2023.
[15] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models compress contexts, 2023.
[16] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[17] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. 2019.
[18] Jiazhan Feng, Chongyang Tao, Xiubo Geng, Tao Shen, Can Xu, Guodong Long, Dongyan Zhao, and Daxin Jiang. Synergistic interplay between search and large language models for information retrieval, 2023.
[19] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmmentd generation for large language models: A survey, 2024.
[20] Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression a large language model, 2023.
[21] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics, 2019.
[22] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. 2020.
[23] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning:
A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[24] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels changing climate: Enhancing Im adaptation with tulu 2023.
[25] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, 2021.
[26] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022.
[27] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[28] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.

---

[29] Huiqiang Jiang, Qianhui Wu, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. LImlingua: Compress- ing prompts for accelerated inference of large language models, 2023.
[30]
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing Ilms in long context scenarios via prompt compression, 2023.
[31] Kelvin Jiang, Dekun Wu, and Hui Jiang. FreebaseQA: A new factoid QA data set matching trivia- style question-answer pairs with Freebase. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 318-33888 18333, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[32] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNNL--JCNLLP), pages 2567-2577, Hong Kong, China, November 2019. Association for Computational Linguistics.
[33] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension, 2017.
[34] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of cisuaaly-conditioned language models, 2024.
[35] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering, 2020.
[36] Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Nearest neighbor machine translation, 2021.
[37] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. General- ization through memorization: Nearest neighbor language models, 2020.
[38] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search preddct:::: Composing retrieval and language models for knowledge-intensive nip, 2023.
[39] Jiho Kim, Sungjin Park, Yconsu Kwon, Yohan Jo, James Thoree, and Edward Choi. Factkg: Fact verification via reasoning on knowledge graphs. arXiv preprint vrxiv:2305.06590 2023.
[40] Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017.
[41] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wes Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, 2019.
[42] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeened generation for knowledge-intensive nlp tasks, 2021.
[43] Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong. Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024.
12

---

[44] Haoran Li, Mingshi Xu, and Yangqiu Song, Sentence embedding leaks more information than you expect: Generative embedding inversion attack recover the whole sentence, 2023.
[45] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-inage pre-training with frozen image encoders and large language models, 2023.
[46] Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342-6353, Singapore, December 2023. Association for Computational Linguistics.
[47] Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey, 2024.
[48] Zonggian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compression for large language models, 2024.
[49] Sheng-Chiele Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval, 2023.
[50] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022.
[51] Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Scott Yih. Ra-dit: Retrieval-augmennedd dual instruction tuning, 2023.
[52] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[53] Qi Liu, Bo Wang, Nan Wang, and Jiaxin Mao. Leveraging passage embeddings for efficient listwise reranking with large language models, 2024.
[54] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseck-vl: Towards real-world vision-language understanding, 2024.
[55] Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass, Sail: Search-augmented instruction learning. 2023.
[56] Yun Luo, Zhen Yang. Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2024.
[57] Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen tau Yih, Hannanch Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling, 2023.
[58] John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M. Rush. Text embeddings reveal (almost) as much as text, 2023.
[59] Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens, 2024.
[60] Niklas Muennighoff, Nouamane Tazi, Loie Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023.
[61] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Interna- tional conference on machine learning, pages 3878-3887. PMLR, 2018
[62] Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin- Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression, 2024.
13

---

[63] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thore, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks, 2021.
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[65] Pranav Rajpurkar, Robin Jia, and Perey Liang. Know what you don't know: Unanswerable questions for squad, 2018.
[66] Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge, 2019.
[67] Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to ai complete question answering: set of prerequisite real tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8722-8731, 2020.
[68] Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction, 2022.
[69] Abigail See, Peter Liu, and Christopher D. Manning. Get the point: Summarization with pointer-generator networks, 2017.
[70] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Serrieval-augmenter black-box language models, 2023.
[71] Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context, 2022.
[72] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ:: A question answering challenge targeting commonsense knowledge, 2019.
[73] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by wcakly-supervised contrastive pre-training, 2024.
[74] Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. Training data is more valuable than you think: A simple and effective method by retrieving from training data, 2022.
[75] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Ha- jishirzi. How far camels go? exploring the state of instruction tuning open resources, 2023.
[76] Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to filter context for surrivall-augmenter generation, 2023.
[77] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2022.
[78] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[79] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval auggenttd Ims with compression and selective augmentation, 2023.
[80] Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Lluís Mårquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013- 2018, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
14

---

[81] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering, 2018.
[82] Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Compressing retrieved documents actively for question answering. 2024.
[83] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval- -auieveed language models robust to irrelevant context, 2023.
[84] Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators, 2023.
[85] Jungmin Yun, Mihyeon Kim, and Younghin Kim. Focus on the core: Efficient attention via pruned token compression for document classification. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13617-13628, Singapore, December 2023. Association for Computational Linguistics.
[86] Xinping Zhao, Yan Zhong, Zetian Sun, Xinshuo Hu. Zhenyu Liu, Dongfang Li, Baotian Hu, and Min Zhang. Funnelrag: A coarse-to-fine progressive retrieval paradigm for rag. arXiv preprint 2rX1v:2410.10293 2024.
[87] Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation, 2022.
15

---

A Comparison between different Context Compression Models
In T we present detailed comparison of various context compression models, emphasizing their real-world applicabilityy This comparison focuses on two key aspects: (1) Plug-and-play capability, which assesses whether dptaset-specifie tuning is necessary for new, unseen data; (2) Memory efficiency, which evaluates if additional memory space is required to store the compressed information, such as high-dimensional vectors typically used in soft prompting methods.
Specifically designed RAG
Maximun
Model
Approach
Plug-and-Play
Memory Efficient
Compression Rate
AutoCompressor
x15
Soft Prompting
Gist (59
X26
Soft Prompting
ICAE 20
Soft Prompting
L..MLingua
X
x20
Prompt Editing
Selective Context
Prompt Editing
Token Elimination [5
X10
Attention Filtering
FilCo
Prompt Ediring
RECOMP
29
X16.6
Prompt Editing
XRAG
x178
Modality Fusion
Table Comparison between different compression methods from their setting to design principle.
B Instructions for Paraphrase Pretraining
The list of instructions for Paraphrase Pretraining is shown in Table They present the same meaning with natural language variance.
"Background: [X] means the same as [D]
"Background: [X] Can you put the above sentences in your own terms? [D]
"[X] Please provide a reinterpretation of the preceding background text. [D]
"These two expressions are equivalent in essence:(1) [X] (2) [D]
"Background: [X] is a paraphrase what? [D)
"[X] Could you give me different version the background sentences above? [D) "
"In other words, background: [X] is just another way saying: [D]
"You't re getting across the point whether you say background: [X] [D]
"[X] After uppacking the ideas in the background information above, we got: [D]
"[X] Please offer restatement of the background sentences I've jestt read. [D]
"Background: [X] which also means: [D]
"Strip away the mystery, and you'll find [X] simply another rendition of: [D]
"The essence of background: [X] is captured again in the following statement: [D]
Table 6: Instructions used for Paraphrase Pretraining where [X] and [D] are placeholders for projected retrieval feature W(E) and document D.
16

---

C Details for Instruction Tuning Dataset
After collecting the raw data from different categories, we use template from FLAN 77 construct instruction tuning dataset. In Table we list an overview and in Table g we list the detailed information for each subtask of our dataset. For QA tasks that lack an explicit context, we perform retrieval operation within the corpus ID to identify the most relevant document to serve as context. This approach akin to the retrieval-augmented instruction tuning depicted in 51
Task Type
# Involved datasets
Train
# Prompt
Label
Reading Comprehension Summarization
488,344
447.62
30.34
81,821 385,173
483.49
53.29
Open Domain QA
7
203.55
20.09
Table 7: Overall statistics of Instruction Tuning dataset.
Task Type
Dataset
# Train
# Prompt Len
# Label Len
CoQA
66
7101
617.98
77.75
DROP
17
76098
356.06
3,86
Narrative
40
32747
702.39
7.86
Reading
PubMedQA
32
1000
397.91
65.4
Comprehension
QuAIL 667
10246
512.9
2.0
SQuAD v2 65
130319
214.54
6.87
PwC [20
241564
571.35
53.07
NQ [41
87925
203.62
5,976
TriviaQA
33
78785
216.1
6.49
CommonsenseQA 72
9741
223.64
2.0
Open Domain
WikiQA 80
1040
192.89
40.79
QA
YahooQA
87358
196.56
56.7
Freebase
20353
218.49
4.87
MSMarco
99994
194.82
15.91
CNN/DM 699
100000
616.99
63.37
Summarization
SamSum 21
14731
187.87
29.12 12
DialogSum
12460
247
37.61
Table 8: Detailed data statistics for our Context-aware Instruction Tuning Dataset.
17

---

D Implementation Details
For the language models we use, Mixtral-8x7b is approximately 6.5 times larger scale compared Mistral-7b and features a divergent architectural approach- approach-speccfffallly dense versus mixture-of- experts design. For our assessments, we employed the instruction-une -unnd variants of these models.
Owing to efficiency constraints, we opted not to perform on-the-fly retrieval. Instead, we pre- constructed retrieval index using the efficient and robust multi-vector retriever, ColBERT-v2 (68]. from which we retrieved the top-/ ranked document for inclusion in our instruction-tuning dataset re-encoded these documents using the
and for the evaluation of downstream tasks. Subsequently, embedding model of interest (c.g.. SFR). This strategy allows us to iterate data-centric experiments quickly. All experiments are conducted on the setup of 8xNvidia A100 GPUs.
In Table and Table 10 we list the hyperparameters for Paraphrase Pretraining and Context-aware Instruction Tuning.
Hyperparameter
Assignment
optimizer learning rate Ir scheduler type warmup ratio weight dacay epochs
AdamW
6e-3
linear
0.03
0.0
1
flash attention batch size
True
12
gradient accumulation steps num GPUs max sequence length
4
8
336
max train samples
2,000,000
Table 9: Hyperparameters for Paraphrase Pretraining.
Hyperparameter
Assignment
optimizer learning rate Ir scheduler type warmup ratio weight dacay epochs KL. a KL temperature flash attention batch size gradient accumulation steps num GPUs max sequence length
AdamW
2e-5
linear
0.03
0.0
1
2.0
1.0
True
4
2
1024
max train samples
955,338
Table 10: Hyperparameters for Context-aware Instruction Tuning,
18

---

E About different Embedding Models
In our primary experiments, we use the SFR model as our default sentence embedding model. This section delves into the effects of different embedding models. We examine four universal text embedding models: E5-Mistral and E5-Large [73 alongside BGE-Large and BGE-Base [78]. Additionally, we assess two retrieval-specife models: Dragon 49 and DPR [35]. The configurations of the different retrievers and their MTEB score metric indicating their general sentence representation capability-ane are listed in Table To isolate the impact of potentially different retrieved documents, we ensure that models utilize the same top-/ document. The performance is averaged four question answering datasets. A general pattern is that embedding models with stronger sentence representation capabilities tend to further enhance the downstream performance. Remarkably, the Dragon model, despite being text embedding models that are twice its size (BGE-Large).
retrieval-specific model, outperforms general
Model
Embedding Dim
Universal Embedding
MTEB Score
Average Resilience
Model
Size
Performance
Boost
Mistral-7b
w/o retrieval retrieval
37.2
46.3
74.1%
29.5%
SFR
7B
4096
67.56
44.5
82.3%
22.2%
35-Mistral
7B
4096
V
66.63
44.0
84.0%
20.6%
W ES Large
335M
1024
62.25
42.1
80.2%
19.6%
XRAG
BGE-Large
335M
64.23
41.6
78.2%
BGB- Bas-
109M
768
63.55
41.2
78.7%
18.9%
Dragun
109M
768
X
42.1
84.2%
16, 9%
w DPR
109M
40.5
77.4%
18.2%
Table 11: Ablation on different sentence embedding models.
19

---

Analysis on Mistral-7b
In Figure 6 we list the Resilience rate and Boost Rate on Mistral-7b model, which exhibit same pattern with Mixtral-8x7b model.
Resilience Rate
LLMLingua
XRAG
RAG
$1.0
80
70.6
50
79.9
76.5
67.1
62.6
NQ
WebQA
HotpotQA
TriviaQA
Boost Rate
41.1
40
20
14.4
20.8
15.0
NQ
WebQA
HotpotQA
TriviaQA
Figure Robustness and effectiveness analysis on 4 QA datasets with Mistral-7b model.
20

---

G Limitations
We discuss the limitations of our framework follows:
In this work, we only consider the most commonly used retrieval system-single dense vector retrieval, while sparse retrieval methods such as BM25 or multi-vector retrieval methods like ColBERT are not included. We believe that combining these methods would a promising direction for xRAG, as sparse vectors could complement dense vectors, and multi-vector retrieval would provide xRAG with more flexibility by not condensing all information into one token.
Currently, xRAG delivers decent performance when relevant document is fetched; however, it lags behind RAG by a considerable margin in tasks that require reasoning (such as HotpotQA and FactKG). One possible reason is that during the training phase of xRAG, messoinng-releann data is not provided. How to make xRAG a better reasoner remains our future work.
We only consider the Top-/ retrieval setting, while ensembling multiple relevant documents has been shown to be effective for RAG systems due to the complementary information contained Top-K documents. We believe there is potential advantage for xRAG to scale to multi-document settings, as the input length of xRAG for multi-documents scales by factor of 1, while for RAG, it scales by the document length factor.
21

---

H More interesting cases
report failure case of xRAG. this case, retrieval alone not enough to derive the final answer and the LLM is required to perform reasoning over retrieved document (the listed universities are all located in Switzerland).
In Figure 7
An interesting example is shown in Figure when the retrieved document is list of characters in the book Discworld, the RAG model would respond with fictional character, while xRAG generate the right answer by focusing on the relevant part of the document.
In Figure 9
even when the retrieved document is relevant, RAG would still hallucinate while xRAG could generate the right answer based on the document.
In Figure I0 the retriever mistakenly fetch the wrong document (Phantom of the Opera of interest is a music rather than file) and RAG would be misled while xRAG remain robust to generate the correct answer.
RAG
Refer to the background document and answer the question:
Background: Philip Kraft Philip (born Rendsburg on March 24, 1969) German organic chemist. Since 1996 he has been employed by Givaudan, a leading Flavor and Fragrance company, where designs captive odorants for perfumes. He lectured at the University Bern, the University Zurich, and the ETH Zurich.
Question: Philip Kraft lectured at the largest university in what European country?
Output: Switzerland.
w/o Retrieval
xRAG
Answer the question:
Refer to the background document and answer the question:
Philip Kraft lectured at the largest university what European country?
Question:
Background: [X]
Question: Philip Kraft lectured largest university in what European country?
Output:
don't know.
X
Output Germany.
Figure 7: Failure case of xRAG when reasoning is required to derive the final answer.
22

---

RAG
Refer to the background document and answer the question:
Background: of Discworld characters |' Meaning "Two-Fingered" Ephebian, Didactylos is philosopher, based on Diogenes Sinope,
into the stories Small Gods. He lives barrel inside the wall of the palace of the Tyrant in Ephebe, crafting bespoke philosophies, axioms aphorisms for scraps. Having apprenticed his nephew Um as philosopher, they frequently argue over the merits of natural philosophy, specifically Um's development of steam power. Although one of the most popular philosophers of all time, Didactylos never earns the respect of his fellow philosophers, due to the fact that he thinks 'about the wrong things'. He has been pictured with lantem though blind
looking ""honest
Question: Which Greek philosopher was said to have lived inside barrel?
Output: Didactylos.
Retrieval
xRAG
Answer question:
Refer to the background document and answer the question:
Question: Which Greek philosopher
Background: [X]
was said to have lived inside barrel?
Question: Which Greek philosopher was said to have lived inside barrel?
Output:
Output: Diogenes of Sinope.
Figure 8: xRAG correctly locates the relevant part in a long document by selecting Diogenes of Sinope the answer rather than Didactylos, fictional character in the book Diseworld.
RAG
Refer to the background document and answer the question:
Background: List of Apollo missions The Apollo program was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans the Moon from to During Apollo mission, astronauts Nell Armstrong and Buzz Aldrin landed their Apollo Lunar Module (LM) and walked on the lunar surface, while Michael Collins remained in lunar orbit in the command and service module (CSM), and three landed safely on Earth on July 24, 1969. subsequent Moon,
Apollo
December
Question:
What mission was the fifth human space flight of "Project Apollo', the third human voyage to the Moon Moon?
mission
Output: Apollo 15.
Retrieval
Answer the question:
Refer to the background document and answer the question:
Question: What mission was human space flight of Projeec Apollo', the third human voyage to the Moon and first manned mission to land on Moon?
Background: [X]
What mission was the fifth human space flight of 'Project Apollo', the third human voyage to the Moon the first manned mission to land on the Moon?
Question:
Output: Apollo
X
Output: Apollo
Figure 9: xRAG correctly locates the relevant part in a long document while RAG would still hallucinate the wrong answer.
23

---

RAG
Refer to the background document and answer the question:
Phantom the Opera (1943 film) The score was written by Edward Ward. For the opera sequences, Ward adapted music from Tchalkovsky's Symphony No. as well as using themes by Chopin. He also composed an original theme, Lullaby of the Bells, which heard in the film as the Phantom's piano concerto. X
Background:
Question: Who wrote the phantom of the opera music?
Output: Edward Ward.
w/o Retrieval
xRAG
Refer to the background document and answer the question:
Answer the question:
Question: Who wrote the phantom of the opera music?
Background: [X]
Question: Who wrote the phantom of the opera music?
Output: Andrew Lloyd Webber.
Output: Andrew Lloyd Webber.
Figure 10: xRAG correctly locates the relevant part in long document while RAG would still hallucinate the wrong answer.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
is fine include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 5.1 and Appendix G
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations"" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
24

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: in Appendix D we provide full details about our training process
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility also be provided via detailed
25

---

reproducible, they instructions for how to replicate the results, access to a hosted model (e.g., g. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: all the models and data we use are publicly available and we carefully cite each paper
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
habolicc/miidee//ddeetummsmssss sorrrdeSuubmsssssooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ttips.cc/public/guiiee http://uddee/odeSSubbb wwttimissioooooiiiioc
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: in Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the htw.bbiiiecthiteccciieeeeeeeee
NeurIPS Code of Ethics
Answer: [Yes]
Justification: reviewed and confirmed
27

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: in section 5.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: all the data and model we use publicly available
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
28

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer[[NA]
Justification: in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpppppppshtaeea thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: n/a
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: n/a
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
29

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: n/a
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
30


Reviews:
[{"review_id": "5Tan4ypJQw", "reviewer": "Reviewer_H8JS", "summary": "This paper presents xRAG, a context compression method for Retrieval-Augmented Generation (RAG). Their key idea is to treat document embeddings from dense retrieval as features from a retrieval modality, which allows compressing retrieved documents into a single token. Experiments show that their method can achieve extreme compression while maintaining performance comparable with traditional RAG.", "questions": "Please refer to the previous section.", "limitations": "Yes, the paper addresses the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed xRAG method treats document embedding as a modality, and shows strong potential for efficient RAG systems. Specifically, the proposed method uses lightweight training strategy that consists of paraphrase pretraining and context-aware instruction tuning.\n- xRAG demonstrates strong empirical results: it can achieve similar performance with traditional RAG methods on various QA datasets, and has the best performance among compression-based RAG methods. However, it still has limitations in tasks that require multi-step reasoning.\n- The paper provides in-depth analysis of the method's behavior, including studies on robustness, effectiveness of different components, and failure cases.\n- The writing is very clear and easy to understand.", "weaknesses": "- RAG is proposed to reduce the hallucination of language model generation, by letting them refer to the original context. It is still unclear why xRAG that compresses a document into a single token could still leave the factual knowledge intact, and the authors should discuss more on this.\n- As the limitation paragraph mentions, the paper does not consider the case of retrieving multiple documents. Though the proposed method works for one-document case, it may not generalize to multiple-document case and may affect the training of the compression projector.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe greatly appreciate your time and effort in reviewing our paper. Here, we would like to address your concerns point by point:\n\n- **Question about why xRAG could compress a document chunk into a single token**\n\nThank you for this insightful question. We believe this is the core of xRAG, and we want to explain it from two perspectives. First, it is surprising how much information a single embedding can contain. As demonstrated in [1], a single vector of shape 768 can recover 32 tokens with over a 90% exact match. In our experiments, we use an embedding of shape 4096 to cover a document chunk with an average of 180 tokens. Moreover, according to Claude Shannon's information theory, one classic estimate of the amount of information per English word is 11.82 bits per word [2]. This suggests that, in theory, we still have enough raw state to encode more tokens with a single embedding. Second, the overall performance is a joint effect from both Boost Rate and Resilience Rate (as defined in Section 6.1 of our paper) and we have to acknowledge that xRAG does not yet perform comparably with RAG in terms of Boost Rate. Improving the Boost Rate of xRAG while maintaining a high level of Resilience Rate—balancing external and internal knowledge—is a primary focus of our future work.\n\n- **About retrieving multiple documents**\n\nPlease refer to our general response.\n\nThank you again for your review. We hope that our response alleviates some of your concerns and that you might consider raising the score of our paper.\n\nBest regards,\n\nAuthors\n\n[1] Morris, John X., Volodymyr Kuleshov, Vitaly Shmatikov and Alexander M. Rush. “Text Embeddings Reveal (Almost) As Much As Text.” *Conference on Empirical Methods in Natural Language Processing* (2023).\n\n[2] Claude Elwood Shannon. “Prediction and Entropy of Printed English.” (1951).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AV23BaOIAm", "reviewer": "Reviewer_jbwR", "summary": "This paper proposes xRAG, a context compression method designed specifically for retrieval-augmented generation. xRAG redefines the use of document embeddings in dense retrieval by integrating them as features from the retrieval modality.  It achieves an extreme compression rate to only one token. The authors perform experiments to  demonstrate the effectiveness of the proposed method.", "questions": "1. How xRAG performs on long-context RAG? Can only one token represents the semantics of the retrieved long-documents?\n\n2. How xRAG selects or trains the dense retrieval model?\n\n3 How xRAG performs on the domain that shifts from the training domain of dense retrieval model?", "limitations": "Authors have discussed the limitations of this paper in Section G of Appendix.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Using modality fusion bridge to connect the embeddings from dense retrieval and LLMs for retrieval-augmented generation.\n\n2. Extremely compress the input tokens from RAG to only one token.\n\n3. Comparable performance with lower FLOPs.", "weaknesses": "1. The proposed method relies heavily on the performance of dense retrieval models. The dense retrieval models have poor generalization for out-of-domain [1], which does not match the general ability of LLMs, but the experiment in this paper is only conducted on wiki-based knowledge bases of Q&A tasks.\n\n2. The specific details of selection and training method of dense retrieval model in xRAG are needed.\n\n3. How xRAG performs on long-context RAG? Can only one token represents the semantics of the retrieved long-documents?\n\n\n[1] Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe greatly appreciate your time and effort in reviewing our paper. Here, we would like to address your concerns point by point:\n\n- **Out-of-Domain Generalization**\n\nThank you for bringing up this issue. If we understand your concerns correctly, they can be divided into two parts:\n\n1. The Out-of-Domain Generalization of Dense Retrieval Models: We acknowledge that dense retrieval models face challenges with cross-domain generalization, yet they remain the de facto approach for modern RAG systems [1][2][3]. Importantly, dense retrieval models (such as DRAGON[4] and ColBERT[5]) have demonstrated stronger generalization compared to their sparse counterparts (such as BM25), as evidenced by benchmarks like BEIR[6] and MTEB[7]. These models are all compatible with xRAG.\n\n2. The Out-of-Domain Generalization of xRAG: The ultimate goal of xRAG is to create compatible representations between dense embeddings and their textual form within the LLM representation space using a modality fusion approach. We believe a well-trained projector can transform any text into the embedding space. To verify this, we tested our xRAG model (paraphrase pre-trained in the Wikipedia) in the biomedical domain. Specifically, we tested on PubMedQA[8], BioASQ-Y/N[9], and ChemProt[10] using PubMed[11] as our retrieval corpus to evaluate the cross-domain generalization ability of xRAG. By comparing the first two rows in the table below, we observed that simply adding one document token, xRAG gives considerable improvement over the Mistral-7b baseline, demonstrating the cross-domain generalization of xRAG. Moreover, xRAG can easily adapt to other domains because the first stage of training pipeline: paraphrase pretraining, is a self-supervised process that does not require handcrafted labels. We sample documents from PubMed and pre-train xRAG on it, which further improve the performance as shown in the last row.\n\n| Model | PubMedQA | BioASQ-Y/N | ChemProt |\n| --- | --- | --- | --- |\n| Mistral-7b | 43.6 | 74.5 | 66.7 |\n| xRAG-7b | 49.2 | 82.1 | 71.8 |\n| xRAG-7b (PubMed) | 51.1 | 83.2 | 72.8 |\n\n- **Details about Training and Selection of Dense Models:**\n    \n    As stated in Section 4.2 of our paper, we selected SFR[12] as our dense embedding model, which, at the time of writing, held the leading position on the MTEB leaderboard. We did not train this model ourselves, allowing us to reuse the offline constructed embeddings and adding ZERO overhead to the existing RAG system, the core of xRAG as detailed in Section 3.4.\n    \n- **Long-Context RAG:**\n    \n    RAG is commonly considered a technique to alleviate the need for LLMs to read long contexts by chunking documents into pieces, which is how modern RAG systems operate. Therefore, technically speaking, long-context and RAG represent two different generation paradigms for LLMs. If your concern regarding xRAG involves multiple document chunks, please refer to our general response.\n    \n\nThank you again for your review. We hope that our response addresses your concerns and that you might consider raising the score of our paper.\n\nBest regards,\n\nAuthors\n\nReferences:\n\n[1]  Lin, Xi Victoria, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke S. Zettlemoyer and Scott Yih. “RA-DIT: Retrieval-Augmented Dual Instruction Tuning.” *ArXiv* abs/2310.01352 (2023): n. pag.\n\n[2] Yu, Yue, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi and Bryan Catanzaro. “RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs.” (2024).\n\n[3] Shao, Rulin, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke S. Zettlemoyer and Pang Wei Koh. “Scaling Retrieval-Based Language Models with a Trillion-Token Datastore.” (2024).\n\n[4] Lin, Sheng-Chieh, Akari Asai, Minghan Li, Barlas Oğuz, Jimmy J. Lin, Yashar Mehdad, Wen-tau Yih and Xilun Chen. “How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval.” *ArXiv* abs/2302.07452 (2023): n. pag.\n\n[5] Khattab, O. and Matei A. Zaharia. “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” *Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval* (2020): n. pag.\n\n[6] Thakur, Nandan, Nils Reimers, Andreas Ruckl'e, Abhishek Srivastava and Iryna Gurevych. “BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” *ArXiv* abs/2104.08663 (2021): n. pag.\n\n[7] Muennighoff, Niklas, Nouamane Tazi, Loic Magne and Nils Reimers. “MTEB: Massive Text Embedding Benchmark.” *Conference of the European Chapter of the Association for Computational Linguistics* (2022).\n\n[8] Jin, Qiao, Bhuwan Dhingra, Zhengping Liu, William W. Cohen and Xinghua Lu. “PubMedQA: A Dataset for Biomedical Research Question Answering.” *Conference on Empirical Methods in Natural Language Processing* (2019).\n\n[9] Yang, Zi, Yue Zhou and Eric Nyberg. “Learning to Answer Biomedical Questions: OAQA at BioASQ 4B.” (2016).\n\n[10] Peng, Yifan, Shankai Yan and Zhiyong Lu. “Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets.” *BioNLP@ACL* (2019).\n\n[11] https://huggingface.co/datasets/ncbi/pubmed\n\n[12] https://blog.salesforceairesearch.com/sfr-embedded-mistral/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuFqUYMyBc", "reviewer": "Reviewer_MKxH", "summary": "The paper on xRAG presents an innovative approach to context compression in retrieval-augmented generation, achieving significant efficiency gains while maintaining performance. The method's compatibility with various language models and preservation of the plug-and-play nature are notable strengths. However, the added complexity and dependency on high-quality embeddings might pose challenges. The whole paper is clear and easy to understand.", "questions": "See weakness.", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces xRAG, a novel context compression method that effectively merges document embeddings into the language model’s representation space through modality fusion, which is an attractive topic.\n2. The xRAG achieves significant context compression compared with baselines.\n3. The plug-and-play nature makes it easy to be applied with other backbone models.", "weaknesses": "1. The introduction of a modality bridge and the requirement for a pretrained modality encoder adds complexity to the system.\n2. Will the bias in projector tuning cause hallucinations?\n3. The performance of xRAG is likely highly dependent on the quality of the dense document embeddings.\n4. The compression method, while efficient, might result in the loss of information present in the original documents. Do we need such extreme compression since most of the queries are not too long? Or how to balance the compressed tokens with the performance?\n5. The concepts are so confused. Sometimes, the author uses the \"modality bridge\" and also uses the \"projector\". I think they are the same module right?\n6. The paper mentions using a two-layer MLP as the projector but doesn't explore how different projector architectures might impact performance.\n7. The paper suggests that xRAG is more robust than RAG when dealing with irrelevant documents but doesn't provide a clear mechanism for how xRAG achieves this robustness.\n8. How will the hyper parameter alpha influence the final results?\n9. How to ensure the compressing process filters the irrelevant information rather than the crucial ones?\n10. Will sensitive information like numbers or times be kept by the projector after compressing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nAppreciate your time and effort in reviewing our paper. We would like to address your concerns point by point:\n\n- **Added complexity to the system**\n\nIn xRAG, we have emphasized throughout our paper that our key design principle is to add minimal complexity to the existing RAG system. In Section 3.4, we explicitly state that our approach **does not require an additional pretrained modality encoder** but instead utilizes existing document embeddings from the retrieval datastore. This design choice ensures **zero** additional space complexity, enabling practical scalability in real-world RAG systems handling millions of documents. To make our method plug-and-play, we freeze the LLM and retriever, and the only trainable component is the projector.\n\n- **Will the Bias in Projector Tuning Cause Hallucinations?**\n\nIn Section 6.1, we introduce a metric called Resilience Rate, which measures how retrieval (RAG or xRAG) affects LLM performance and prevents hallucinations. Empirical results show that our method is more robust and produces fewer hallucinations than vanilla RAG.\n\n- **Dependence on the Quality of Dense Document Embeddings**\n\nIt is widely acknowledged that the performance of the modality encoder heavily impacts downstream performance, and xRAG is no exception. However, xRAG is not limited to only strong embedding models. In Section 6.2, we demonstrate the effectiveness of xRAG across various embedding models, showing that even with a four-year-old DPR model, xRAG still yields over 8% improvement.\n\n- **The compression method, while efficient, might result in the loss of information present in the original documents. Do we need such extreme compression since most of the queries are not too long? Or how to balance the compressed tokens with the performance?**\n\nAccording to Claude Shannon's information theory, one classic estimate of the amount of information per English word is 11.82 bits per word [1]. This means, in theory, we have enough raw state to encode more tokens with a single embedding without loss of information. In that sense, we can be more efficient when dealing with real-world multi-round, multi-document RAG settings, where prompt length could be up to thousands of tokens. More importantly, in  Section 5.2, we have demonstrated that the current xRAG could bring considerable efficiency improvement even on a high-end server GPU.\n\nAs for balancing compression and performance, we discuss in Appendix G that xRAG can be combined with a multi-vector or Matryoshka embedding system where the compression ratio is configurable. We believe this is an exciting direction for xRAG.\n\n- **The concepts are so confused. Sometimes, the author uses the \"modality bridge\" and also uses the \"projector\". I think they are the same module right?**\n\nThey refer to the same module that connects two modalities.\n\n- **The paper mentions using a two-layer MLP as the projector but doesn't explore how different projector architectures might impact performance.**\n\nAs discussed in Section 3.4, the key design principle is simplicity. Therefore, we chose the simplest and most commonly used projector: a two-layer MLPs. Exploration of more advanced architectures is left for future work.\n\n- **The paper suggests that xRAG is more robust than RAG when dealing with irrelevant documents but doesn't provide a clear mechanism for how xRAG achieves this robustness.**\n\nOne reason for xRAG's robustness is that it is trained with retrieved context in the second training phase, which improves the LLM's resilience to noisy retrievals. This phenomenon is also observed in [2][3]. Additionally, xRAG can avoid word-by-word repetition seen in RAG when dealing with noisy retrieval results.\n\n- **How will the hyper parameter alpha influence the final results?**\n\nWe tested several values for the hyperparameter alpha ({0.1, 0.5, 1.0, 2.0, 3.0}) and found that 2.0 gave the best results on the validation set. No significant differences were observed when alpha ≥1.\n\n- **How to ensure the compressing process filters the irrelevant information rather than the crucial ones?**\n\nOur paper, like previous context compression papers, does not claim to achieve lossless compression. It is a balance between efficiency and efficacy. We do not filter out any information; instead, we aim to recover the original information with a single embedding vector. Our novel training paradigm, from paraphrase pretraining to context-aware instruction tuning, is designed to achieve this goal.\n\n- **Will sensitive information like numbers or times be kept by the projector after compressing?**\n\nxRAG has the ability to retain sensitive information such as numbers and times. As shown in Figure 9 of our paper, xRAG successfully identifies important numbers and answers questions correctly.\n\nThank you again for your review. We hope that our response alleviates some of your concerns and that you might consider raising the score of our paper.\n\nBest regards,\n\nAuthors\n\n[1] Claude Elwood Shannon. “Prediction and Entropy of Printed English.” (1951).\n\n[2]  Lin, Xi Victoria, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke S. Zettlemoyer and Scott Yih. “RA-DIT: Retrieval-Augmented Dual Instruction Tuning.” *ArXiv* abs/2310.01352 (2023): n. pag.\n\n[3] Luo, Hongyin, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen M. Meng and James R. Glass. “SAIL: Search-Augmented Instruction Learning.” *Conference on Empirical Methods in Natural Language Processing* (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yMsd1DK3jR", "reviewer": "Reviewer_2bxX", "summary": "This paper proposes xRAG, a method to map the retriever's embeddings into LM's representation space with one token. While both the retriever and LM are fixed, xRAG trains a simple projector to adapt to them by paraphrase pretraining and context-aware instruction tuning. xRAG can significantly outperform non-retrieval and previous context compression methods and matches the performance of the uncompressed one, saving the RAG FLOPs by more than 3 times.", "questions": "- For plugging in multiple retrieved documents, should we devise another form of training task or simply adopt the same projector multiple times? This is an interesting discussion that I would like to see\n- How simple can the projector be? I would like to see the performance of different design choices of the projector, such as Q-Former mentioned in the paper\n- What is the data source for Paraphrase Pretraining? I cannot find it in the paper", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The compression rate from the long context to one token is high\n- The only trainable component constitutes less than 0.1% of the LLM’s parameters, which can serve as an efficient plug-in\n- xRAG can match or surpass the original RAG with only one token for one document, and the analysis of the Resilience Rate and the Boost Rate is reasonable and interesting\n- In general, the paper is well written", "weaknesses": "- The main experiment in this paper is conducted only on one retrieved document, whose average length is just 175, but I think this is a **very short context** for modern LLMs (and thus the inference time is not a bottleneck). I think proving the effectiveness of xRAG on a real-long context (more than thousands of tokens) of multiple documents is very important to see this method's real-world value\n\n- It is essential to consider the total training FLOPs of the projector. Although we don't need the actual optimization of LM and retriever, we also need the forward process of these models in the projector's training so that **the cost can be significant**, especially when the amount of total two-stage training samples is very, very large (near 3M)\n\n- The projector is **tightly coupled** with the retriever and LM. It is hard to prove the generalization abilities of xRAG to other black-box production LLMs where the projector cannot be trained directly from the LM's feedback, which may limit its potential impact\n\n- In Section 4.3, the requirements of the baselines include **no need for dataset-specific tuning**. However, how to explain that NQ, TriviaQA (two evaluation tasks) are included in the Context-aware Instruction Tuning data?\n\n- Missing baselines of context compression methods other than LLMLingua, like Gist-COCO. Will adding **more compression tokens** benefit the performance?\n\n[1]: Li, Xinze, et al. \"Say more with less: Understanding prompt learning behaviors through gist compression.\" arXiv preprint arXiv:2402.16058 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe greatly appreciate your time and effort in reviewing our paper. Here, we would like to address your concerns point by point:\n\n- **About multiple documents:**\n\nPlease refer to our general response about multiple documents in xRAG.\n\n- **The cost of training**\n\nThe training cost of xRAG is minimal compared to full parameter tuning since the only trainable component is a two-layer MLPs. In our setup (8xA00), full parameter tuning of a 7B dense model with 1 million samples takes about 24 hours. In the same amount of time, xRAG (8x7b) can finish training on 2.9 million samples (paraphrase pre-training + context-aware instruction tuning). Moreover, by caching all the document embeddings, we can further speed up the training by 25%. We appreciate your suggestion and will include this information in the revised version of our paper.\n\n- **xRAG for black-box LLM**\n\nFrom the user side, it is true that xRAG can only be applied to models with open weights. However, with the increasing popularity of cloud vector databases, LLM API providers could still benefit from xRAG if users can provide the index of retrieved documents in the vector database.\n\n- **About dataset-specific tuning.**\n\nWhen we refer to \"without dataset-specific tuning,\" we mean that we adopt a one-for-all evaluation setting where the evaluation results from various datasets come from the same model, as opposed to one-for-one evaluation which requires dataset-specific tuning. To exclude the impact of the corresponding training set (NQ and TQA), we refer to Table 4 of our paper, where similar results are obtained even with only reading comprehension data involved (NQ and TQA excluded). Additionally, the LoRA results in Table 3 confirm that the improvement of xRAG does not stem from task knowledge inherited during context-aware instruction tuning.\n\n- **Missing baselines of context compression methods other than LLMLingua, like Gist-COCO. Will adding more compression tokens benefit the performance?**\n\nThank you for pointing out this missing reference. Similar to Gist Token [1], Gist-COCO relies on gist representation, which might not be applicable in real-world RAG settings involving millions of documents. We will add this paper to the final version of xRAG in both the Related Work section and Appendix A. Regarding the addition of more compression tokens, we believe it is a balance between efficiency and efficacy. In Appendix G, we discuss that xRAG could be combined with a multi-vector or Matryoshka embedding system where the compression ratio is configurable. We believe this is an exciting direction for xRAG.\n\n- **For plugging in multiple retrieved documents, should we devise another form of training task or simply adopt the same projector multiple times? This is an interesting discussion that I would like to see**\n\nPlease refer to our general response about multiple documents in xRAG.\n\n- **How simple can the projector be? I would like to see the performance of different design choices of the projector, such as Q-Former mentioned in the paper**\n\nAs discussed in Section 3.4, the key design principle is the simplicity of the system. To our knowledge, the simplest form for a projector is an MLP architecture. We leave the exploration of more advanced and complex architectures for future work.\n\n- **What is the data source for Paraphrase Pretraining? I cannot find it in the paper**\n\nApology for the confusion. The data source for Paraphrase Pretraining is the same Wikipedia dump used as the retrieval corpus. We will make this clear in the revised version of our paper.\n\n[1] Mu, Jesse, Xiang Lisa Li and Noah D. Goodman. “Learning to Compress Prompts with Gist Tokens.” *ArXiv* abs/2304.08467 (2023): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our paper. We would like to address the concerns regarding the multiple document expansion of xRAG.

Our work represents a pioneering effort in efficient RAG with modality fusion, a research direction that has been acknowledged as both interesting and promising by the reviewers. In our current experimental setup, we have focused on the top-1 document in both training and evaluation to facilitate rapid iteration of data-centric experiments and for efficiency reasons. We view the expansion to multiple documents as an incremental improvement (1 to N) of the existing framework, which, while valuable, is less fundamental compared to our core innovation (0 to 1).

Adapting xRAG to a top-k documents setting presents several avenues for exploration, including:

1. The type of modality projector (e.g., MLP, Q-Former, or Perceiver).
2. Data selection and mixing strategies (e.g. data cluster with longer context)
3. Optimization for document relationship modeling and more.

To demonstrate xRAG's adaptability to multi-document scenarios, we have implemented a straightforward approach focusing primarily on the data perspective. Specifically, we upsampled summarization data, which typically includes longer documents, and divided these into chunks during our context-aware instruction tuning phase. We maintained other configurations as in the current xRAG implementation. The results of this approach are presented below.

While this naive implementation may not represent the optimal configuration for a multi-document setting in xRAG, as it primarily stems from the data aspect, it effectively showcases our framework's flexibility and extensibility to handle multiple documents.

|  | NQ | TriviaQA | WebQ | HotpotQA |
| --- | :---: | :---: | :---: | :---: |
| top1 | 39.1 | 65.7 | 39.4 | 34.0 |
| top3 | 41.4 | 67.3 | 41.1 | 35.3 |

Thank you once again for your valuable feedback.

Best regards,

Authors

Meta-review: The paper presents a context compression approach for retrieval-augmented generation (RAG). The idea is to use document embeddings that were used for retrieval systems and train a light-weight adaptor that can integrate information from such embedding into LLMs. This compresses in-context document significantly, resulting in much more efficient model (with smaller FLOPS). Overall, the idea is pretty simple but well-executed and shows strong experimental gains. 

The reviewers identified some limitations of the proposed approach - such as limited performance when handling multiple retrieved documents. While the approach shows efficiency gain in this setting, it comes with moderate performance drops. The new results from the discussion period should be added to the revised paper. With the advance of the long context model, extending this work to compressing long document setting will be necessary for future impact.
Decision: Accept (poster)