Title: xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Venue: NeurIPS 2024
Accepted: yes
Keywords: retrieval-augmented generation

context compression

dense retrieval
Abstract: This paper introduces xRAG, an innovative context compression method tailored for retrieval-augmented generation. xRAG reinterprets document embeddings in dense retrieval--traditionally used solely for retrieval--as features from the retrieval modality. By employing a modality fusion methodology, xRAG seamlessly integrates these embeddings into the language model representation space, effectively eliminating the need for their textual counterparts and achieving an extreme compression rate. 
In xRAG, the only trainable component is the modality bridge, while both the retriever and the language model remain frozen. This design choice allows for the reuse of offline-constructed document embeddings and preserves the plug-and-play nature of retrieval augmentation. 
Experimental results demonstrate that xRAG achieves an average improvement of over 10% across six knowledge-intensive tasks, adaptable to various language model backbones, ranging from a dense 7B model to an 8x7B Mixture of Experts configuration. xRAG not only significantly outperforms previous context compression methods but also matches the performance of uncompressed models on several datasets, while reducing overall FLOPs by a factor of 3.53. Our work pioneers new directions in retrieval-augmented generation from the perspective of multimodality fusion, and we hope it lays the foundation for future efficient and scalable retrieval-augmented systems.

xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token

Abstract

This paper introduces xRAG, a novel context compression method designed specifically for retrieval-augmented generation. xRAG redefines the use of document embeddings in dense retrieval-traditionally limited to retrieval purposes-by integrating them as features from the retrieval modality. Through a modality fusion approach, xRAG effectively merges these embeddings into the language model's representation space, eliminating the need for their textual counterparts and achieving an extreme compression rate. In xRAG, the modality bridge is the only trainable component, while the retriever and language model remain frozen. This design choice allows for the reuse of offline-constructed document embeddings and preserves the plug-and-play nature of retrieval augmentation. Experimental results demonstrate that xRAG achieves an average improvement of over 10% across six knowledge-intensive tasks, compatible with various language model backbones, ranging from a dense 7B model to an 8x7B Mixture of Experts configuration. xRAG not only significantly outperforms previous context compression methods but also matches the performance of uncompressed models on several benchmarks, while reducing overall FLOPs by a factor of 3.53. This work pioneers new avenues in retrieval-augmented generation through multimodal fusion, potentially setting a groundwork for future developments in efficient and scalable retrieval systems.

1 Introduction

Retrieval-Augmented Language Models (RALMs) [42, 22, 9, 14, 70] have shown exceptional performance in a variety of knowledge-intensive tasks. By retrieving domain-specific, long-tailed, and up-to-date knowledge from a non-parametric datastore, RALMs significantly extend the boundaries of parametric Large Language Models (LLMs). However, the integration of entire documents into prompts can significantly increase inference costs and may surpass the context limit of LLMs [29, 79]. As illustrated in Figure 1, while the inclusion of a relevant document enables the LLM to generate accurate responses, it does so at the expense of processing documents that expand the original query by more than tenfold.

How might we mitigate the costs associated with extended context while maintaining the benefits of retrieval augmentation? Recent research interest has converged on a promising direction: Context Compression. This concept is pursued through two primary strategies: soft-prompting methods, such as Gist [59], AutoCompressor [15], and ICAE [20], which compress the context into dense memory slots, and hard-prompting methods, such as LLMLingua [29] and RECOMP [79], where

Refer to the background document and answer the question:

Background

: Motel 6 | " Beginning in 1986, Motel 6 has advertised through radio commercials featuring the voice of

writer and National Public Radio commentator Tom Bodett, with the tagline "We'll leave the light on for you." The ads

were created by Dallas advertising agency The Richards Group. They feature a tune composed by Tom Faulkner,

performed by him on guitar and Milo Deering on fiddle. The first spots were conceived and written by David Fowler. In

1996, the ads won a Clio Award. The campaign itself has won numerous national and international awards and was

selected by Advertising Age magazine as one of the Top 100 Advertising Campaigns of the Twentieth Century."

Question

: What company advertised itself with the slogan "We'll leave a light on for you"?

Output

: Motel 6.

Answer the question:

Question

: What company advertised

itself with the slogan "We'll leave a light

on for you"?

Output

:  Marriott International.

Refer to the background document and answer the question:

Background

:

[X]

Question

: What company advertised itself with the slogan "We'll leave a

light on for you"?

Output

: Motel 6.

w/o Retrieval

xRAG

compression is applied on the surface form. These approaches, however, either require significant memory for storing LLM activations (e.g., 1.05 MB per token as reported by [59]) or suffer from relatively low compression rates. More critically, these methods overlook a crucial characteristic of RALMs: through large-scale contrastive learning with question-document pairs, modern dense retrieval systems already distill document content into a single high-dimensional embedding, and this embedding reveals (almost) as much information as the text [58, 44].

In this paper, we pioneer an innovative approach to retrieval augmentation and context compression through the lens of modality fusion. Drawing from multimodal research, where text-only language models are taught to "perceive" and "listen," a pretrained modality encoder like CLIP [64] is typically used to extract modality features. These features are then integrated into language models using a modality fusion bridge [45, 52, 34]. Building on the conceptual overlap between the retriever encoder and modality encoder, we introduce xRAG. This model redefines document embeddings from dense retrieval-traditionally solely for retrieval purposes-as retrieval modality features. xRAG employs a modality fusion methodology to seamlessly integrate these embeddings into the language model's representation space, thus obviating the need for textual counterparts and achieving significant context compression. In xRAG, the modality bridge is the only trainable component, while both the retriever and the LLM are kept frozen. This design decision facilitates the reuse of pre-constructed document embeddings and maintains the plug-and-play nature of retrieval augmentation-two essential factors for a functional RAG system.

To verify the effectiveness and versatility of our framework, we conducted comprehensive experiments with different LLM backbones, ranging from a dense 7B model to an 8x7B Mixture of Experts model. Our results reveal that adding just one document token could lead to over a 10% improvement across six knowledge-intensive tasks, significantly surpassing previous compression methods. xRAG also delivers results comparable to uncompressed models on several benchmarks. This is remarkable considering that the only trainable component constitutes less than 0.1% of the LLM's parameters. In terms of efficiency, xRAG reduces total FLOPs by a factor of 3.53 compared to the uncompressed RAG model. We further provide detailed analyses of xRAG, examining various training strategies, data blends, and component selections for the retrieval system. We believe this research sets a strong foundation for the development of future efficient and scalable retrieval-augmented systems.

2 Related Work

Retrieval-augmented Generation Equipping a parametric language model with a non-parametric datastore has proven effective for a range of NLP tasks, including language modeling [37, 57, 87],

Answer

Who is the richest person

in the world as of 2024?

Query

Projector

(a) xRAG

Sentence

Encoder

Who is the richest person

in the world as of 2024?

Query

Who is the richest person

in the world as of 2024?

Query

Trainable

Frozen

Document

Sentence

Embedding

Token

Embedding

Concate

Prompt

... ...

Document

businessman, investor and art collector... and CEO of

the global luxury goods company LVMH, runs dozens of

luxury brands...

He overtook Elon Musk as the richest

person in 2024 due to a 21% decline in Musk's wealth,

from $245.3 billion to $194.6 billion. Arnault was

educated at the Lycée Maxence Van Der Meersch ...

Previous Compressor (Optional)

Datastore

....

....

open-domain question answering [25, 42, 70, 86], domain adaptation [7] and machine translation [36, 12], among others. Given the vast design space of this generation paradigm, numerous approaches with different focuses have been proposed. For instance, RETRO [9] and PlugLM [13] introduce architectural innovations for enhanced integration with the non-parametric datastore. REALM [22] pioneers an end-to-end approach for simultaneous optimization of the language model and retriever. REPLUG [70] and RA-DIT [51] improve retriever alignment using feedback from LLMs. DSP [38] and InteR [18] investigate complex interactions between the retriever and the language model. Selfmem [14] utilizes a reward model to refine retrieval and generation iteratively. Self-RAG [4] incorporates a self-reflection mechanism to enhance the quality and factuality of language model outputs. For a detailed overview, see [19, 5, 3]. Our contribution, xRAG, stands out by implementing a modality fusion approach to retrieval augmentation, creating an effective and efficient RAG system.

Context Compression Context compression, aimed at reducing the input length for LLMs while retaining essential information, has recently attracted substantial interest [47]. Gist [59] achieves a compression rate of up to 26x by modifying the attention mask and caching soft gist token activations. ICAE [20]AutoCompressor [15], and 500xCompressor[48] condense lengthy contexts into succinct, compact memory slots, which are directly utilizable by LLMs for diverse functions. LLMLingua [29, 30, 62] and CompAct[82] introduces a coarse-to-fine prompt compression technique based on perplexity scores and distilled token-level score. While these methods are generally applicable, others are tailored specifically for RAG systems, such as FilCo [76] and RECOMP [79]. A concurrent work directly employs passage embeddings for efficient listwise reranking [53]. For an in-depth comparison of these compression methods regarding memory efficiency, compression rates, and adaptability, refer to Appendix A.

3 Methods

Problem Formulation In retrieval-augmented generation, a non-parametric datastore D = { ( E i , D i ) } | D | i =1 consists of pairs where each D i represents a document chunk as a sequence of L i tokens D i = { d i 1 , . . . , d i L i } . Correspondingly, E i is the dense representation derived from a sentence embedding model SE θ ( · ) with input D i . For an input query q , its dense representation SE θ ( q ) is used to find the relevant documents by matching against the collection { E i } | D | i =1 with certain similarity search algorithm such as MIPS. After retrieval, the system selects a relevant pair ( E , D ) from D , concatenates the chosen document D with q , and processes the combined input with a language model

(b) Context-aware Instruction Tuning

Who is the richest person

in the world as of 2024?

Question

Paraphrase the above text

Instruction

Sentence

Encoder

Projector

Sentence

Encoder

Who is the richest person

in the world as of 2024?

Question

Projector

(a) Paraphrase Pretraining

businessman...He over-took Elon Musk as the

richest person in 2024 due to a 21% decline in

Musk's wealth, from $245.3 billion to...

Document

March 1949) is a French

businessman...He over-

richest person in 2024

due to a 21% decline in

Musk's wealth, from

$245.3 billion to...

Document

xRAG Distribution

Answer

F ϕ ( D ⊕ q ) . Optionally, a context compression module C can be integrated to reduce the length of D from L to a more concise l , achieving a compression ratio of L l .

3.1 xRAG Architecture

Traditional methods for document compression typically focus on surface form of the document [29, 76, 79]. In contrast xRAG tackle the problem from a modality fusion view. Concretely, we introduce a modality projector W , which is trained to directly project the retrieval features E into the LLM representation space. Our proposed framework is visually contrasted with the traditional RAG system in Figure 2. In the standard RAG, the input to the LLM comprises the embeddings Emb ( D ⊕ q ) of length | D | + | q | , where Emb signifies the embedding layer of the LLM. Conversely, with xRAG, the modified input is represented as W ( E ) ⊕ Emb ( q ) , which yields a substantially reduced length of 1 + | q | . In this framework, the challenges come from the modality fusion: How can a text-only language model understand features from retrieval modality? To achieve this, we explore a two-stage training strategy: Paraphrase Pretraining followed by Context-aware Instruction Tuning.

3.2 Paraphrase Pretraining

Similar to the pretraining strategies employed in vision-language models that use image-captioning data to align two modalities [52, 16, 54], the primary objective of our paraphrase pretraining is to build a compatible representation between the extracted retrieval feature and the corresponding document. Illustrated in Figure 3(a), for each pair ( E , D ) in a retrieval corpus D , we employ a natural language instruction X instruct to prompt the LLM to undertake a paraphrasing task (e.g. "[X] The above text could be paraphrased as: [D]", where [X] and [D] are placeholders for W ( E ) and document D) 2 . In this setup, the model learns to connect W ( E ) and D by recovering D on the condition of W ( E ) and the model is optimized by:

where p ϕ is given by the softmax distribution of LLM F ϕ , and d <i denotes the document token before current prediction token d i , achieved by casual attention mask in auto-regressive LMs.

3.3 Context-aware Instruction Tuning

After the pretraining phase, although the language model F ϕ has developed an internally compatible representation, it has never been explicitly trained to utilize these features for downstream tasks. To address this gap, we proceed to instruct the model in harnessing the fused feature W ( E ) by continually training the model on data where the answer is closely associated with the given context, including reading comprehension, summarization, and open domain question answering data. We

constructed an mixed dataset, containing approximately 1 million entries from open-source datasets, as detailed in Appendix C. For each triplet in the dataset, ( X context , X question , X answer ) , we initially obtain the sentence representation for X context via the embedding model E context = SE θ ( X context ) . Subsequently, we refine the optimization of on two directions:

Optimization I: Language Modeling. Aligned with established instruction tuning methodologies [75, 24, 51, 43], our objective is to finetune the model so that it generates the correct output when provided with a specific instruction, conditioned upon the given context information. Unlike traditional models that utilize the textual context X context , our method employs a dense feature E context to encapsulate the context information:

Optimization II: Self-Distillation. The second trajectory of optimization aims to guide the xRAG in the effective utilization of contextual information, drawing from the principles of self-distillation [2, 71] and imitation learning [61, 23]. By considering the RAG model as a "teacher" and xRAG as a "student", we endeavor to distill the knowledge from RAG, thereby enabling xRAG to emulate the RAG model's proficiency in handling the full, uncompressed documents. This approach enhances xRAG's resilience in scenarios where it encounters noisy or irrelevant context that may not directly lead to the correct answer, detailedly discussed in § 6.1. Concretely, for a language model F ϕ using either X context or E context as the source of context, our objective is to minimize the divergence between the two resulting output distributions. This discrepancy is measured using the KullbackLeibler (KL) divergence:

Here X question is omitted for brevity and the final loss is the linear combination controlled by a hyperparameter: L nll + α L kl .

3.4 Design Principle

In designing the projector W , our primary objective is to maintain the simplicity of the framework. We therefore opted for a two-layer MLP while other more sophisticated module such as Q-Former [45] could also be considered. Notice that the projector is the only trainable component, accounting for only 0.46% of the total parameters in the Mistral-7b model and 0.07% in the Mixtral-8x7b model. Such a design choice departs from previous studies that necessitated full-parameter tuning to adapt LLMs for compressed contexts [59, 76, 15]. We believe this approach will likely be more accessible and practical because, fundamentally, the RAG itself functions as a plug-and-play module for LLMs, and so should its compressed version. This design also avoid the risk of compromising other core capabilities of LLM during full-parameter tuning, as observed in [55, 56].

Moreover, in contrast to other compression methods that necessitate storing LLM activations for each compressed token [59, 20, 15]-an impractical strategy in the RAG setting, given the millions of documents involved-our method introduces no additional memory overhead. Instead, it leverages offline-constructed document embeddings, originally designed for retrieval. To summarize, xRAG not only simplifies the integration process but also avoids unnecessary computational or memory expenses.

4 Experimental Setup

4.1 Evaluation Dataset

We evaluated the performance of xRAG primarily on knowledge-intensive tasks which encompass a range of challenges: (1) three Open Domain Question Answering datasets that address questions on a wide array of topics: Natural Questions [41], TriviaQA [33], and Web Questions [8]. (2) one Multihop Question Answering dataset, HotpotQA [81], which necessitates multi-step reasoning to generate answers. (3) one Long-form Question Answering dataset, TruthfulQA [50], that requires the generation of long-form and truthful responses. (4) one fact-checking dataset, FactKG [39], which challenges the model to use complex reasoning to determine the factual accuracy of given claims.

In line with the KILT [63] and GenRead [84], we assessed three ODQA datasets and HotpotQA using the Exact Match (EM) metric, FactKG with Accuracy, and for the long-form QA, we used both the F1 score and Rouge-L (R-L) score. These tasks demand a broad spectrum of world knowledge and have been extensively explored in the retrieval-augmentation literature [42, 22, 35, 10, 25, 70].

4.2 Implementation Details

To demonstrate the versatility of our framework, we choose two backbones, differing in scale and architecture: Mistral-7b [27] and Mixtral-8x7b [28]. For the retrieval corpus, we utilized the Wikipedia dump from December 2021, which was pre-processed into passages following the methodology described in [26]. This resulted in approximately 37 million passages, each averaging 180 tokens in length. Our default retrieval model is the SFR [1], which, at the time of writing this paper, holds the leading position on the MTEB leaderboard [60]. We use top-1 ranked document for inclusion in our instruction-tuning dataset and for the evaluation of downstream tasks. More details are provided in Appendix D. Code is available at: https://github.com/Hannibal046/xRAG .

4.3 Baselines

In determining appropriate baselines for comparison, we adhered to a fundamental principle: the selected compression methods must support the general plug-and-play capability of retrieval augmentation. This entails that they should function effectively without the need for dataset-specific tuning [79, 76], or any alteration to the parameters of LLMs [76, 59]. Furthermore, given the extensive volume of the retrieval corpus, it is essential that these compression methods demonstrate memory efficiency, specifically by not requiring the storage of LLM activations for each individual token [59, 20, 15]. With these criteria in mind, our evaluation compares xRAG to the following baselines: (I) Primarily, we consider two variants of LLMs: one that operates without retrieval augmentation and another that includes it. These serve as the lower and upper performance bounds for our study of compression techniques, respectively. (II) Additionally, our comparisons extend to LLMLingua [29], a plug-and-play approach for context compression. (III) Taking inspiration from [59], we incorporate a method of discrete compression using TF-IDF. This approach yields compression rates comparable to those achieved by xRAG and serves as the lower bound for discrete compression.

5 Experimental Results

5.1 Knowledge Intensive Tasks

In Table 1, we present our main results. Across both Mistral-7b and Mixtral-8x7b configurations, we observe a consistent and significant improvement when retrieval augmentation is applied (p-value

< 0.05), although the gains are more modest for the larger model configurations. This trend aligns with observations reported by [70, 51]. Further analysis on the efficacy of various compression techniques reveals that xRAG outperforms other approaches by a large margin. Remarkably, xRAG not only reduces the token count drastically-from 175.1 to a single token-but also maintains robust performance levels. In some instances, xRAG's performance is comparable to, or even exceeds, that of the uncompressed models. Specifically, in the Mistral-7b configuration, xRAG achieves nearly the same performance improvement as the uncompressed model (16.6% compared to 19.4%), and in the Mixtral-8x7b configuration, it surpasses the uncompressed model (9.7% compared to 8.0%). One possible reason lies in the vulnerability of current RAG system when the irrelevant or misleading documents are presented, a topic detailed discussed in §6.1. We also observe that xRAG performs well in tasks that require document understanding, such as TriviaQA. However, in tasks that demand reasoning over document, like HotpotQA and FactKG, xRAG lags behind by a considerable gap.

5.2 Computational Efficiency

In this section, we conduct a thorough assessment of our framework's computational efficiency and memory management. To rigorously evaluate our model, we employed Torch Profiler 3 to measure the CUDA Time (milliseconds) and Giga FLOPs of both the RAG and xRAG models across four real-world datasets. In these evaluations, the Mistral-7b, operating in bfloat16 inference mode, served as the base LLM. CUDA Time and GFLOPs were calculated on an average per batch basis with a fixed batch size, and GFLOPs were normalized by the number of generated tokens. These experiments were performed on the same computational hardware, specifically an Nvidia A100 and an AMD EPYC 7V12 64-Core Processor. As depicted in Table 2, despite variations in prompt and generation lengths across the datasets, xRAG significantly outpaced the RAG model, achieving a x1.64 increase in CUDA Time efficiency and a x3.53 reduction in GFLOPs.

6 Analysis

6.1 Evaluation Beyond the Overall Score

Although retrieval augmentation generally boosts performance as shown by aggregate metrics, it may not uniformly benefit all instances. In certain cases, the retrieval system might provide irrelevant or even misleading information, leading to incorrect answers that were previously correct [83, 76, 5]. To enable a more fine-grained evaluation, we introduce two novel metrics: the Resilience Rate and the Boost Rate. The resilience rate quantifies the percentage of instances in which the system's responses remain correct both before and after retrieval augmentation, highlighting the system's stability and robustness. Conversely, the boost rate measures the percentage of instances that were initially answered incorrectly but were rectified following the introduction of a retrieved document, thereby assessing the efficacy of retrieval augmentation. An ideal RAG system should have both high resilience rate and boost rate.

In Figure 4, we display these metrics for the uncompressed RAG and two compression methods: LLMLingua and xRAG. Surprisingly, although retrieval augmentation generally enhances performance, the resilience rate for RAG averages only 75.2%, indicating that retrieval can adversely affect about one-quarter of previously correct answers. In contrast, xRAG demonstrates considerable

robustness across all evaluated datasets. This robustness largely stems from xRAG's ability to maintain an unbiased stance toward the internal knowledge representation of the LLM, especially when confronted with noisy retrieval content. Similar trends are noted in [51, 55], where search-augmented instruction learning is shown to bolster the robustness of language models. However, xRAG still lags behind RAG in boost rate, particularly in multi-hop reasoning tasks. It is crucial to note that a high resilience rate does not necessarily mean that the LLM disregards the provided information, which could potentially lead to a reduced boost rate. A comparative analysis with LLMLingua indicates that xRAG is not only more robust but also more effective.

Resilience Rate

NQ

WebQA

HotpotQA

TriviaQA

70

80

90

67.6

66.7

72.4

85.4

81.0

85.1

81.5

92.0

68.7

66.6

78.9

86.9

LLMLingua

xRAG

NQ

WebQA

HotpotQA

TriviaQA

20

30

15.4

12.2

17.2

22.2

20.0

17.8

15.7

26.6

26.8

16.0

25.7

29.0

Boost Rate

6.2 What makes xRAG effective?

This section delves into a thorough evaluation of various elements that contribute to xRAG's overall performance, focusing on its training strategy, the blend of datasets used and the effect of different embedding models. Due to the space limit, we present the last factor in Appendix E.

1. Training Strategy We carefully ablate four optimization choices: pretraining, instruction tuning, and two optimization objectives-language modeling (nll) and self-distillation (self-kd). We also train a Mistral-7b with LoRA on our instruction tuning dataset to rule out the possibility that our improvement simply comes from tuning on more data. The outcomes are presented in Table 3. Our analysis reveals that the interplay of different training strategies significantly contributes to the

Refer to the background document and answer the question:

Background

: The Old Man and the Key | " several ""washed-up"" celebrities, was composed by Alf Clausen and

written by Vitti. Even though Vitti received sole credit for writing the song, parts of the lyrics were written by Simpsons

writer Carolyn Omine, which were then revised by fellow writer Matt Selman. Selman was especially satisfied with the

rhyme sung by Mr. T; he stated that whenever he feels down, he thinks of that rhyme and that it "boosts [him] up

inside." The episode features American actress Olympia Dukakis as Grampa's love interest Zelda. American

comedian Bill Saluga also makes an appearance, as his television character Ray J. Johnson. The owner of the

minivan is portrayed by series regular Hank Azaria, who imitated the voice of Clark Gable for the character."

Question

: Which US comedian/actor is credited with saying 'A man is only as old as the woman he feels'?

Output

Answer the question:

Question

: Which US comedian/actor is

credited with saying 'A man is only as old

as the woman he feels'?

Output

Refer to the background document and answer the question:

Background

:

[X]

Question

: Which US comedian/actor is credited with saying 'A man is only

as old as the woman he feels'?

Output

w/o Retrieval

xRAG

performance of our framework. In the case of Mistral-7b, pretraining and finetuning phases are of equal significance to the end results. However, for Mixtral-8x7b, the impact of pretraining is notably diminished, likely due to the larger model's enhanced capability to incorporate multi-modality information. Furthermore, we find that during finetuning, self-distillation is more important than language modeling. The primary advantage of self-distillation lies in bolstering the resilience rate of the xRAG system. Optimization with nll loss tends to cause an overreliance on context information, rendering the system more vulnerable when the retriever fails to fetch relevant documents.

II. Instruction-tuning Dataset Blend As discussed in § 3.3, our instruction-tuning dataset primarily comprises three categories: reading comprehension, open-domain QA, and text summarization. To explore the effects of different data blends, we instruction-tune three xRAG model variants, each using data from these distinct categories. The results are shown in Table4. Our analysis reveals that among the dataset blends, reading comprehension data most significantly enhances the xRAG model's performance, as evidenced by both high resilience and boost rates. Intriguingly, when tuned solely with summarization data, xRAG still manages to deliver strong performance on QA datasets it has never been exposed to. This finding underscores that the advantages of instruction tuning for xRAG are not rooted in task-specific knowledge. Instead, they derive from the model's improved ability to utilize projected context information effectively.

6.3 Case Study

In Figure 5, we show one interesting case about the robustness of xRAG. When retrieval system provide misleading content, standard RAG would overly rely on the document and generate answer that are faithful to the document while not factually true. Our xRAG model opt to rely on the internal

knowledge of LLM and being robust to the misleading content. In Appendix H, we include more cases about xRAG including several error analysis.

7 Conclusion

In this work, we present xRAG, an innovative context compression method tailored for retrievalaugmented generation. For knowledge-intensive tasks, xRAG can be significantly faster than RAG while maintaining comparable performance. We are excited about the future of this modality-based retrieval-augmented system and plan to further improve its performance in the areas of reasoning over embedding, handling multiple documents, and combining with multi-vector retrieval.

References

Sfr-embedding-mistral: Enhance text retrieval with transfer learning. https://blog. salesforceairesearch.com/sfr-embedded-mistral/ .

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 , 2020.

Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Yun-Nung (Vivian) Chen, Margot Margot, and Siva Reddy, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) , pages 41-46, Toronto, Canada, July 2023. Association for Computational Linguistics.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023.

Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen tau Yih. Reliable, adaptable, and attributable language models with retrieval, 2024.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018.

David Beauchemin, Zachary Gagnon, and Ricahrd Khoury. Quebec automobile insurance question-answering with retrieval-augmented generation. arXiv preprint arXiv:2410.09623 , 2024.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1533-1544, 2013.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens, 2022.

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions, 2017.

Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset, 2021.

Xin Cheng, Shen Gao, Lemao Liu, Dongyan Zhao, and Rui Yan. Neural machine translation with contrastive translation memories, 2022.

Xin Cheng, Yankai Lin, Xiuying Chen, Dongyan Zhao, and Rui Yan. Decouple knowledge from parameters for plug-and-play language modeling, 2023.

Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrieval-augmented text generation with self memory, 2023.

Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts, 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019.

Jiazhan Feng, Chongyang Tao, Xiubo Geng, Tao Shen, Can Xu, Guodong Long, Dongyan Zhao, and Daxin Jiang. Synergistic interplay between search and large language models for information retrieval, 2023.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024.

Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model, 2023.

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization . Association for Computational Linguistics, 2019.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training, 2020.

Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) , 50(2):1-35, 2017.

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.

Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, 2021.

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models, 2023.

Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression, 2023.

Kelvin Jiang, Dekun Wu, and Hui Jiang. FreebaseQA: A new factoid QA data set matching triviastyle question-answer pairs with Freebase. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 318-323, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2567-2577, Hong Kong, China, November 2019. Association for Computational Linguistics.

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017.

Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models, 2024.

Vladimir Karpukhin, Barlas O˘ guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering, 2020.

Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Nearest neighbor machine translation, 2021.

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models, 2020.

Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp, 2023.

Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, and Edward Choi. Factkg: Fact verification via reasoning on knowledge graphs. arXiv preprint arXiv:2305.06590 , 2023.

Tomáš Koˇ ciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7, 2019.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024.

Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence, 2023.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.

Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 6342-6353, Singapore, December 2023. Association for Computational Linguistics.

Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey, 2024.

Zongqian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compression for large language models, 2024.

Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval, 2023.

Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022.

Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Scott Yih. Ra-dit: Retrieval-augmented dual instruction tuning, 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Qi Liu, Bo Wang, Nan Wang, and Jiaxin Mao. Leveraging passage embeddings for efficient listwise reranking with large language models, 2024.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding, 2024.

Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning, 2023.

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2024.

Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling, 2023.

John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M. Rush. Text embeddings reveal (almost) as much as text, 2023.

Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens, 2024.

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023.

Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International conference on machine learning , pages 3878-3887. PMLR, 2018.

Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression, 2024.

Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018.

Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge, 2019.

Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to ai complete question answering: A set of prerequisite real tasks. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 8722-8731, 2020.

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction, 2022.

Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks, 2017.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval-augmented black-box language models, 2023.

Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context, 2022.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024.

Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. Training data is more valuable than you think: A simple and effective method by retrieving from training data, 2022.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023.

Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to filter context for retrieval-augmented generation, 2023.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.

Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation, 2023.

Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 20132018, Lisbon, Portugal, September 2015. Association for Computational Linguistics.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018.

Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Compressing retrieved documents actively for question answering, 2024.

Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context, 2023.

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators, 2023.

Jungmin Yun, Mihyeon Kim, and Youngbin Kim. Focus on the core: Efficient attention via pruned token compression for document classification. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 13617-13628, Singapore, December 2023. Association for Computational Linguistics.

Xinping Zhao, Yan Zhong, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Dongfang Li, Baotian Hu, and Min Zhang. Funnelrag: A coarse-to-fine progressive retrieval paradigm for rag. arXiv preprint arXiv:2410.10293 , 2024.

Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation, 2022.

A Comparison between different Context Compression Models

In Table 5, we present a detailed comparison of various context compression models, emphasizing their real-world applicability. This comparison focuses on two key aspects: (1) Plug-and-play capability, which assesses whether dataset-specific tuning is necessary for new, unseen data; (2) Memory efficiency, which evaluates if additional memory space is required to store the compressed information, such as high-dimensional vectors typically used in soft prompting methods.

B Instructions for Paraphrase Pretraining

The list of instructions for Paraphrase Pretraining is shown in Table 6. They present the same meaning with natural language variance.

"Background: [X] means the same as [D] " · "Background: [X] Can you put the above sentences in your own terms? [D] " · "[X] Please provide a reinterpretation of the preceding background text. [D] " · "These two expressions are equivalent in essence:(1) [X] (2) [D] " · "Background: [X] is a paraphrase of what? [D] " · "[X] Could you give me a different version of the background sentences above? [D] " · "In other words, background: [X] is just another way of saying: [D] " · "You're getting across the same point whether you say background: [X] or [D] " · "[X] After uppacking the ideas in the background information above, we got: [D] " · "[X] Please offer a restatement of the background sentences I've just read. [D] " · "Background: [X] , which also means: [D] " · "Strip away the mystery, and you'll find [X] is simply another rendition of: [D] " · "The essence of background: [X] is captured again in the following statement: [D] "

C Details for Instruction Tuning Dataset

After collecting the raw data from different categories, we use templates 4 from FLAN [77] to construct instruction tuning dataset. In Table 7 we list an overview and in Table 8 we list the detailed information for each subtask of our dataset. For QA tasks that lack an explicit context, we perform a retrieval operation within the corpus D to identify the most relevant document to serve as context. This approach is akin to the retrieval-augmented instruction tuning depicted in [55, 51].

D Implementation Details

For the language models we use, Mixtral-8x7b is approximately 6.5 times larger in scale compared to Mistral-7b and features a divergent architectural approach-specifically, a dense versus mixture-ofexperts design. For our assessments, we employed the instruction-tuned variants of these models.

Owing to efficiency constraints, we opted not to perform on-the-fly retrieval. Instead, we preconstructed a retrieval index using the efficient and robust multi-vector retriever, ColBERT-v2 [68], from which we retrieved the top-1 ranked document for inclusion in our instruction-tuning dataset and for the evaluation of downstream tasks. Subsequently, we re-encoded these documents using the embedding model of interest (e.g., SFR). This strategy allows us to iterate data-centric experiments quickly. All experiments are conducted on the a setup of 8xNvidia A100 GPUs.

In Table 9 and Table 10, we list the hyperparameters for Paraphrase Pretraining and Context-aware Instruction Tuning.

E About different Embedding Models

In our primary experiments, we use the SFR model as our default sentence embedding model. This section delves into the effects of different embedding models. We examine four universal text embedding models: E5-Mistral and E5-Large [73] alongside BGE-Large and BGE-Base [78]. Additionally, we assess two retrieval-specific models: Dragon [49] and DPR [35]. The configurations of the different retrievers and their MTEB scores 6 -a metric indicating their general sentence representation capability-are listed in Table 11. To isolate the impact of potentially different retrieved documents, we ensure that all models utilize the same top-1 document. The performance is averaged over four question answering datasets. A general pattern is that embedding models with stronger sentence representation capabilities tend to further enhance the downstream performance. Remarkably, the Dragon model, despite being a BERT-base-sized retrieval-specific model, outperforms general text embedding models that are twice its size (BGE-Large).

F Analysis on Mistral-7b

In Figure 6, we list the Resilience rate and Boost Rate on Mistral-7b model, which exhibit same pattern with Mixtral-8x7b model.

NQ

WebQA

HotpotQA

TriviaQA

70

80

90

62.6

64.8

73.1

79.9

79.6

80.9

78.7

89.8

68.4

67.1

76.5

84.0

Resilience Rate

LLMLingua

xRAG

NQ

WebQA

HotpotQA

TriviaQA

20

30

40

17.0

15.0

14.0

27.8

19.9

19.4

15.4

33.9

31.1

20.8

24.7

41.1

Boost Rate

G Limitations

We discuss the limitations of our framework as follows:

In this work, we only consider the most commonly used retrieval system-single dense vector retrieval, while sparse retrieval methods such as BM25 or multi-vector retrieval methods like ColBERT are not included. We believe that combining these methods would be a promising direction for xRAG, as sparse vectors could complement dense vectors, and multi-vector retrieval would provide xRAG with more flexibility by not condensing all information into one token.

Currently, xRAG delivers decent performance when a relevant document is fetched; however, it lags behind RAG by a considerable margin in tasks that require reasoning (such as HotpotQA and FactKG). One possible reason is that during the training phase of xRAG, reasoning-relevant data is not provided. How to make xRAG a better reasoner remains our future work.

We only consider the Top-1 retrieval setting, while ensembling multiple relevant documents has been shown to be effective for RAG systems due to the complementary information contained in Top-K documents. We believe there is potential advantage for xRAG to scale to multi-document settings, as the input length of xRAG for multi-documents scales by a factor of 1, while for RAG, it scales by the document length factor.

H More interesting cases

In Figure 7, we report a failure case of xRAG. In this case, retrieval alone is not enough to derive the final answer and the LLM is required to perform reasoning over retrieved document (the listed universities are all located in Switzerland).

An interesting example is shown in Figure 8, when the retrieved document is a list of a characters in the book Discworld, the RAG model would respond with a fictional character, while xRAG generate the right answer by focusing on the relevant part of the document.

In Figure 9, even when the retrieved document is relevant, RAG would still hallucinate while xRAG could generate the right answer based on the document.

In Figure 10, the retriever mistakenly fetch the wrong document (Phantom of the Opera of interest is a music rather than a file) and RAG would be misled while xRAG remain robust to generate the correct answer.

Refer to the background document and answer the question:

Background

: Philip Kraft |  Philip Kraft (born in Rendsburg on March 24, 1969) is a German organic chemist. Since

1996 he has been employed by Givaudan, a leading Flavor and Fragrance company, where he designs captive

odorants for use in perfumes. He has lectured at the University of Bern, the University of Zurich, and the ETH

Zurich.

Question

: Philip Kraft lectured at the largest university in what European country?

Output

: Switzerland.

Answer the question:

Question

largest university in what European

country?

Output

:  I don't know.

Refer to the background document and answer the question:

Background

:

[X]

Question

: Philip Kraft lectured at the largest university in what European

country?

Output

: Germany.

w/o Retrieval

xRAG

Refer to the background document and answer the question:

Background

: List of Discworld characters | " Meaning ""Two-Fingered"" in Ephebian, Didactylos is a philosopher,

based on Diogenes of Sinope, who comes into the stories in Small Gods. He lives in a barrel inside the wall of the

palace of the Tyrant in Ephebe, crafting bespoke philosophies, axioms or aphorisms for scraps. Having apprenticed

his nephew Urn as a philosopher, they frequently argue over the merits of natural philosophy, specifically Urn's

development of steam power. Although one of the most popular philosophers of all time, Didactylos never earns the

respect of his fellow philosophers, due to the fact that he thinks 'about the wrong things'. He has been pictured with a

lantern though blind and is looking for an ""honest man"".

Question

: Which Greek philosopher was said to have lived inside a barrel?

Output

: Didactylos.

Answer the question:

Question

: Which Greek philosopher

was said to have lived inside a barrel?

Output

:  No one.

Refer to the background document and answer the question:

Background

:

[X]

Question

: Which Greek philosopher was said to have lived inside a barrel?

Output

: Diogenes of Sinope.

w/o Retrieval

xRAG

Refer to the background document and answer the question:

Background

: List of Apollo missions |  The Apollo program was the third United States human spaceflight program

carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first

humans on the Moon from 1969 to 1972. During the Apollo 11 mission, astronauts Neil Armstrong and Buzz Aldrin

landed their Apollo Lunar Module (LM) and walked on the lunar surface, while Michael Collins remained in lunar orbit

in the command and service module (CSM), and all three landed safely on Earth on July 24, 1969. 5 subsequent

Apollo missions also landed astronauts on the Moon, the last in December 1972.

Question

: What mission was the fifth human space flight of 'Project Apollo', the third human voyage to the Moon

and the first manned mission to land on the Moon?

Output

: Apollo 15.

Answer the question:

Question

: What mission was the fifth

human space flight of 'Project Apollo',

the third human voyage to the Moon and

the first manned mission to land on the

Output

:  Apollo 9.

Refer to the background document and answer the question:

Background

:

[X]

Question

: What mission was the fifth human space flight of 'Project

Apollo', the third human voyage to the Moon and the first manned mission

to land on the Moon?

Output

: Apollo 11.

w/o Retrieval

xRAG

Refer to the background document and answer the question:

Background

: Phantom of the Opera (1943 film) |  The score was written by Edward Ward. For the opera sequences,

Ward adapted music from Tchaikovsky's Symphony No. 4 as well as using themes by Chopin. He also composed an

original theme, Lullaby of the Bells, which was heard in the film as the Phantom's piano concerto.

Question

: Who wrote the phantom of the opera music?

Output

Answer the question:

Question

: Who wrote the phantom of

the opera music?

Output

Refer to the background document and answer the

question:

Background

:

[X]

Question

: Who wrote the phantom of the opera music?

Output

w/o Retrieval

xRAG

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and Section 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 5.1 and Appendix G

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: the paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: in Appendix D we provide full details about our training process

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed

instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: all the models and data we use are publicly available and we carefully cite each paper

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 4.1

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: in Section 5.1

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: in Appendix D

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: reviewed and confirmed

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: in section 5.2

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: all the data and model we use is publicly available

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer:[NA]

Justification: in Section C

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: n/a

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: n/a

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: n/a

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes xRAG, a method to map the retriever's embeddings into LM's representation space with one token. While both the retriever and LM are fixed, xRAG trains a simple projector to adapt to them by paraphrase pretraining and context-aware instruction tuning. xRAG can significantly outperform non-retrieval and previous context compression methods and matches the performance of the uncompressed one, saving the RAG FLOPs by more than 3 times.\n\nSTRENGTHS:\n- The compression rate from the long context to one token is high\n- The only trainable component constitutes less than 0.1% of the LLM’s parameters, which can serve as an efficient plug-in\n- xRAG can match or surpass the original RAG with only one token for one document, and the analysis of the Resilience Rate and the Boost Rate is reasonable and interesting\n- In general, the paper is well written\n\nWEAKNESSES:\n- The main experiment in this paper is conducted only on one retrieved document, whose average length is just 175, but I think this is a **very short context** for modern LLMs (and thus the inference time is not a bottleneck). I think proving the effectiveness of xRAG on a real-long context (more than thousands of tokens) of multiple documents is very important to see this method's real-world value\n\n- It is essential to consider the total training FLOPs of the projector. Although we don't need the actual optimization of LM and retriever, we also need the forward process of these models in the projector's training so that **the cost can be significant**, especially when the amount of total two-stage training samples is very, very large (near 3M)\n\n- The projector is **tightly coupled** with the retriever and LM. It is hard to prove the generalization abilities of xRAG to other black-box production LLMs where the projector cannot be trained directly from the LM's feedback, which may limit its potential impact\n\n- In Section 4.3, the requirements of the baselines include **no need for dataset-specific tuning**. However, how to explain that NQ, TriviaQA (two evaluation tasks) are included in the Context-aware Instruction Tuning data?\n\n- Missing baselines of context compression methods other than LLMLingua, like Gist-COCO. Will adding **more compression tokens** benefit the performance?\n\n[1]: Li, Xinze, et al. \"Say more with less: Understanding prompt learning behaviors through gist compression.\" arXiv preprint arXiv:2402.16058 (2024).\n\nQUESTIONS:\n- For plugging in multiple retrieved documents, should we devise another form of training task or simply adopt the same projector multiple times? This is an interesting discussion that I would like to see\n- How simple can the projector be? I would like to see the performance of different design choices of the projector, such as Q-Former mentioned in the paper\n- What is the data source for Paraphrase Pretraining? I cannot find it in the paper\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper on xRAG presents an innovative approach to context compression in retrieval-augmented generation, achieving significant efficiency gains while maintaining performance. The method's compatibility with various language models and preservation of the plug-and-play nature are notable strengths. However, the added complexity and dependency on high-quality embeddings might pose challenges. The whole paper is clear and easy to understand.\n\nSTRENGTHS:\n1. This paper introduces xRAG, a novel context compression method that effectively merges document embeddings into the language model’s representation space through modality fusion, which is an attractive topic.\n2. The xRAG achieves significant context compression compared with baselines.\n3. The plug-and-play nature makes it easy to be applied with other backbone models.\n\nWEAKNESSES:\n1. The introduction of a modality bridge and the requirement for a pretrained modality encoder adds complexity to the system.\n2. Will the bias in projector tuning cause hallucinations?\n3. The performance of xRAG is likely highly dependent on the quality of the dense document embeddings.\n4. The compression method, while efficient, might result in the loss of information present in the original documents. Do we need such extreme compression since most of the queries are not too long? Or how to balance the compressed tokens with the performance?\n5. The concepts are so confused. Sometimes, the author uses the \"modality bridge\" and also uses the \"projector\". I think they are the same module right?\n6. The paper mentions using a two-layer MLP as the projector but doesn't explore how different projector architectures might impact performance.\n7. The paper suggests that xRAG is more robust than RAG when dealing with irrelevant documents but doesn't provide a clear mechanism for how xRAG achieves this robustness.\n8. How will the hyper parameter alpha influence the final results?\n9. How to ensure the compressing process filters the irrelevant information rather than the crucial ones?\n10. Will sensitive information like numbers or times be kept by the projector after compressing?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes xRAG, a context compression method designed specifically for retrieval-augmented generation. xRAG redefines the use of document embeddings in dense retrieval by integrating them as features from the retrieval modality.  It achieves an extreme compression rate to only one token. The authors perform experiments to  demonstrate the effectiveness of the proposed method.

STRENGTHS:
1. Using modality fusion bridge to connect the embeddings from dense retrieval and LLMs for retrieval-augmented generation.

2. Extremely compress the input tokens from RAG to only one token.

3. Comparable performance with lower FLOPs.

WEAKNESSES:
1. The proposed method relies heavily on the performance of dense retrieval models. The dense retrieval models have poor generalization for out-of-domain [1], which does not match the general ability of LLMs, but the experiment in this paper is only conducted on wiki-based knowledge bases of Q&A tasks.

2. The specific details of selection and training method of dense retrieval model in xRAG are needed.

3. How xRAG performs on long-context RAG? Can only one token represents the semantics of the retrieved long-documents?


[1] Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders

QUESTIONS:
1. How xRAG performs on long-context RAG? Can only one token represents the semantics of the retrieved long-documents?

2. How xRAG selects or trains the dense retrieval model?

3 How xRAG performs on the domain that shifts from the training domain of dense retrieval model?

LIMITATIONS:
Authors have discussed the limitations of this paper in Section G of Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents xRAG, a context compression method for Retrieval-Augmented Generation (RAG). Their key idea is to treat document embeddings from dense retrieval as features from a retrieval modality, which allows compressing retrieved documents into a single token. Experiments show that their method can achieve extreme compression while maintaining performance comparable with traditional RAG.\n\nSTRENGTHS:\n- The proposed xRAG method treats document embedding as a modality, and shows strong potential for efficient RAG systems. Specifically, the proposed method uses lightweight training strategy that consists of paraphrase pretraining and context-aware instruction tuning.\n- xRAG demonstrates strong empirical results: it can achieve similar performance with traditional RAG methods on various QA datasets, and has the best performance among compression-based RAG methods. However, it still has limitations in tasks that require multi-step reasoning.\n- The paper provides in-depth analysis of the method's behavior, including studies on robustness, effectiveness of different components, and failure cases.\n- The writing is very clear and easy to understand.\n\nWEAKNESSES:\n- RAG is proposed to reduce the hallucination of language model generation, by letting them refer to the original context. It is still unclear why xRAG that compresses a document into a single token could still leave the factual knowledge intact, and the authors should discuss more on this.\n- As the limitation paragraph mentions, the paper does not consider the case of retrieving multiple documents. Though the proposed method works for one-document case, it may not generalize to multiple-document case and may affect the training of the compression projector.\n\nQUESTIONS:\nPlease refer to the previous section.\n\nLIMITATIONS:\nYes, the paper addresses the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}