Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation

## Abstract

Abstract Retrieval Augmented Generation (RAG) improves large language models (LMs) by incorporating non-parametric knowledge through evidence retrieval from external sources. However, it often struggles to filter out inconsistent and irrelevant information that can distract the LM from its tasks. While compressing the retrieved evidence with a compression model aims to address this issue, the compressed evidence may still be unfamiliar to the target model used for downstream task, potentially failing to utilize the evidence effectively.
We propose FaviComp ( Fa miliarity-aware E vi dence Comp ression), a novel training-free evidence compression technique that makes retrieved evidence more familiar to the target model, while seamlessly integrating parametric knowledge from the model. Specifically, FaviComp proactively lowers the perplexity of the compressed evidence with regard to the target model by combining token probabilities from both the compression model and the target model to generate context that is more familiar to the target model. This approach balances the integration of parametric and non-parametric knowledge, which is especially helpful in complex tasks where the retrieved evidence set may not contain all the necessary information. Experimental results demonstrate that FaviComp consistently outperforms existing baselines in multiple open-domain QA datasets, achieving high compression rates and showcasing the effective integration of both parametric and non-parametric knowledge. 1 1 1 Code and data are available at https://github.com/luka-group/FaviComp

## 1 Introduction

Retrieval Augmented Generation (RAG) has become a common practice to leverage external knowledge beyond its inherent knowledge boundaries to perform better in knowledge-intensive tasks such as open-domain question answering (QA) (Lewis et al., ; Izacard & Grave, ; Guu et al., ) and fact-checking (Pan et al., ; Li et al., ). In particular, incorporating multiple evidence pieces is crucial in solving complicated tasks such as multi-hop and complex reasoning (Trivedi et al., ; Jiang et al., ; Li et al., ), which require various sources of information to solve the questions.

Nevertheless, RAG often struggles to filter out inconsistent and irrelevant information from the multiple evidence set, which can interfere with downstream tasks (Shi et al., ). This highlights the need for *compression-based* RAG (Jiang et al., ; Xu et al., ; Yoon et al., ) to identify and retain only the essential information for the LMs to utilize effectively.
Traditionally, compression-based RAG has focused on reranking documents or sentences by relevance and then incorporating a top-ranked subset (Nogueira et al., ; Zhuang et al., ; Wang et al., ) or compressing the documents into an abstractive summary that retains only essential context
(Jiang et al., ; Xu et al., ; Yoon et al., ).
However, the reranked or compressed evidence might be unfamiliar to the LM employed for the downstream task (referred to as target model) due to discrepancies in internal knowledge and prompt preferences between the compression model and the target model, which are learned during the pretraining stage (Gonen et al., ; Lee et al., ; Mallen et al., ).
When LMs encounter unfamiliar contextual information, they often fail in balancing parametric and non-parametric knowledge, either by overly relying on their parametric knowledge (Longpre et al., ; Tan et al., ) or by utilizing retrieved evidence without considering its relevance to the input (Wu et al., ).

To address these challenges, we propose FaviComp (Familiarity-aware Evidence Compression), a training-free evidence compression method that compresses evidence that is more familiar to the target model while integrating parametric knowledge into the compressed evidence. Inspired by the prior findings that an LM’s familiarity with a prompt is generally reflected by low perplexity (Liu et al., ; Gonen et al., ; Wang et al., ), FaviComp proactively lowers the perplexity of the compressed evidence with regard to the target model.
Specifically, FaviComp leverages the token probabilities of two LMs, a *compression model* and the *target model*. The compression model is instructed to summarize the raw evidential documents into a relevant context to the input, while the target model is instructed to generate relevant context without referencing the documents. Instead of directly selecting the highest probability token from the compression model at each decoding step, we ensemble the token logits from both the compression and target models and then select the token with the highest probability from this combined set. This ensemble decoding constrains the token search space of the compression model to those with lower perplexity, making the context more familiar to the target model (Liu et al., ).

Furthermore, FaviComp potentially synergizes the retrieved knowledge with the target model’s parametric knowledge introduced during ensemble decoding. FaviComp can effectively discern when to leverage internal or external knowledge, which is particularly beneficial in the presence of noisy contextual evidence in complex tasks such as multi-document QA (Wang et al., ).

FaviComp brings along key advantages of RAG for complex tasks from two perspectives.
On the one hand, it is capable of compressing multiple augmented documents to a more favorable form to the target model.
This mechanism not only helps the model better comprehend the essential evidence in the retrieval augmentation but also better balances knowledge utility in both the evidential context and the model’s parametric memory.
On the other hand, it is a training-free and model-agnostic approach that can be easily plugged into the RAG processes of any auto-regressive LMs.

Our experiments show that FaviComp outperforms all existing baselines in five open-domain QA datasets with two downstream LMs while achieving high compression rates. Additionally, we conduct ablation studies by varying the degree of token ensembling and analyze its impact on performance and context perplexity. Moreover, we investigate how FaviComp effectively integrates parametric and non-parametric knowledge.

Figure: Figure 1: An overview of FaviComp. Instead of relying solely on compressed evidence from the compression model (upper), FaviComp familiarizes the compressed evidence to the target model while integrating parametric knowledge through ensemble decoding, resulting in improved downstream performance (lower).
Refer to caption: extracted/5864847/images/method.png

## 2 Method

We present FaviComp, a decoding-time evidence compression method that familiarizes the retrieved evidence with the target model while synergizing them with the model’s parametric knowledge. We first illustrate the motivation for FaviComp in Section 2.1 and provide the preliminaries of compression-based RAG in Section 2.2, followed by a detailed definition of
our proposed framework in Section 2.3.

### 2.1 Motivation and Method Overview

Standard RAG faces the challenge of LMs struggling to filter out inconsistent and irrelevant information from multiple evidence pieces, which can interfere with downstream tasks (Shi et al., ). Previous research has primarily concentrated on question-focused compression (Jiang et al., ; Xu et al., ; Yoon et al., ); however, this approach may lead to suboptimal performance in downstream tasks due to the compressed evidence’s potential unfamiliarity with the target model employed. This unfamiliarity arises from discrepancies in internal knowledge and prompt preferences between the compression model and the target model, which are learned during pretraining (Gonen et al., ; Lee et al., ; Mallen et al., ). Furthermore, the unfamiliarity often leads to failure in balancing parametric and non-parametric knowledge, either by overly relying on their parametric knowledge (Longpre et al., ; Tan et al., ) or by using retrieved evidence without considering its relevance to the input (Wu et al., ). To address this issue, FaviComp introduces a novel approach that compresses evidence that better aligns with the target model’s preferences while seamlessly integrating parametric knowledge into the compressed evidence using a novel ensemble decoding technique, thereby improving its performance on downstream tasks.

Figure 1 illustrates the overview of FaviComp. In this example, FaviComp makes the compressed evidence more favorable to the target model and leverages its parametric knowledge to supplement the missing evidence (“Lionel Messi made his league debut in Barcelona”), effectively combining evidential and parametric knowledge.

### 2.2 Compression-based Retrieval Augmented Generation

Given a set of $k$ retrieved evidence snippets $D=\{d_{1},d_{2},\ldots,d_{k}\}$ and a textual input sequence $x$, standard RAG aims to generate an output sequence $y$, conditioned on both $D$ and $x$. However, standard RAG directly utilizes $D$ which often contains irrelevant information to $x$, potentially confusing the target model in downstream tasks (Shi et al., ). Thus, the compression-based RAG uses an additional compression model to condense $D$ into a concise and input-relevant context $c$, which is then used in place of $D$ during the downstream generation process. Thus, the compression-based RAG is formalized as:

$$ $\displaystyle y^{*}=\arg\max_{y}P_{t}(y\mid x,\hat{c}),$ $\displaystyle\hat{c}=P_{c}(c\mid x,[d_{1},d_{2},\ldots,d_{k}]),$ $$

where $y^{*}$ is the final output sequence, $[\cdot,\cdot]$ denotes concatenation, and $P_{t}$ and $P_{c}$ represent the probability distributions of the target and compression models, respectively. In this work, we consider any natural language prompting tasks, such as open-domain QA tasks, where $x$ represents the input prompt (also known as the query in QA tasks) and $y^{*}$ denotes the output sequence.

The compression model’s objective is to produce a concise yet informative summary $c$ of the evidential documents $D$ that captures the essential information relevant to the input query $x$. We use an unsupervised approach, where the model is instructed to generate a query-relevant summary of $D$ in a zero-shot manner using an evidence compression instruction prompt, denoted as $I_{comp}$, such as the one below:

> Evidence Compression Instruction Given a question and multiple document snippets, generate one summarized context that is helpful to answer the question.

Specifically, the evidence compression is done in an auto-regressive way formalized as,

$$ $\displaystyle P_{c}(c\mid I_{comp},x,D)=\prod_{i=1}^{|c|}P_{c}(c_{i}|I_{comp}, x,D,c_{<i}),$ $$

where $|c|$ is the length of the summary $c$.

### 2.3 Ensemble Decoding for FaviComp

Simple compression techniques might lead to subpar performance in downstream tasks because the compressed evidence may not be familiar to the target model. To better align the context to the target model, FaviComp proactively lowers the perplexity of the context by introducing a constraint in decoding space from the target model during the evidence compression. FaviComp achieves this goal through ensemble decoding, which involves a multiplicative ensemble of two LMs—compression model and target model—at each decoding step.

Specifically, the target model is directed to generate a context $c$ that would be helpful in answering the question $x$ without referencing the evidence set. This is also done in zero-shot using a context generation instruction prompt $I_{gen}$ such as:

> Context Generation Instruction Given a question, generate a context that is helpful to answer the question.

The context generation is also performed in an auto-regressive fashion, represented as:

$$ $\displaystyle P_{t}(c\mid I_{gen},x)=\prod_{i=1}^{|c|}P_{t}(c_{i}|I_{gen},x,c_ {<i}),$ $$

where $|c|$ denotes the length of the generated context $c$.

Once the compression model and the target model generate their respective probability distributions for the next token, the subsequent token is chosen by maximizing the weighted sum of the log probabilities from both models. The selected token is the continuation of the previously generated text aligned with their objectives. This process is formalized as follows:

$$ $c_{i}=\arg\max_{c_{i}\in V}((1-\alpha)\cdot\log P_{c}(c_{i}|I_{comp},x,D,c_{<i })+\alpha\cdot\log P_{t}(c_{i}|I_{gen},x,c_{<i})),$ $$

where $c_{i}$ is the subsequent token, and $\alpha$ is the ensemble coefficient that weighs between the two probability distributions. We demonstrate how the coefficient $\alpha$ impacts both the perplexity and the downstream performance in Section 4.2.

Ensemble decoding proactively shifts the token search space in evidence compression by upweighting those tokens with lower perplexity from the target model’s perspective (Liu et al., ), resulting in a compressed evidence that is more familiar to the target model. Note that since both objectives ultimately share the goal of generating context relevant to the question, combining the logits ensures alignment with this ultimate goal.

In addition, ensemble decoding enables FaviComp to seamlessly integrate both retrieval knowledge from the external evidence set and the target model’s parametric knowledge. Specifically, FaviComp selects the $\arg\max$ token from the target model only when the token’s probability is higher than that of the compression model, demonstrating that FaviComp draws on parametric knowledge only when necessary—potentially when the compression model is uncertain about the next token. This is particularly beneficial for complex tasks like multi-document QA, where the evidence set may not include all the necessary information (Mallen et al., ). In such cases, the missing information in compressed evidence can be supplemented by tokens generated from context generation by the target model, which is entirely based on parametric knowledge. We demonstrate in Section 4.3 and Section 5 that FaviComp can incorporate knowledge from both sources effectively, leading to a performance boost compared to compression methods that solely focus on distilling knowledge from the evidence set.

## 3 Experimental Settings

We assess the effectiveness of FaviComp on knowledge-intensive QA tasks. In this section, we delve into the details of the experimental settings.

### 3.1 Datasets

We evaluate FaviComp on five open-domain QA datasets, including two single-document QA datasets, Natural Questions (NQ) (Kwiatkowski et al., ) and TriviaQA (TQA; Joshi et al. ), and three multi-document QA datasets, HotpotQA (Yang et al., ), 2WikiMultiHopQA (2WikiMQA; Ho et al. ), and MuSiQue (Trivedi et al., ).
Following prior studies (Asai et al., ; Xu et al., ), we evaluate the performance on the development set of each dataset
and use two evaluation metrics, i.e. Accuracy (Acc) and token-level F1.

### 3.2 Implementation Details

For all the comparison methods, we utilize two LMs as the target model to tackle downstream QA tasks with RAG, i.e. Llama3-8B-Instruct(^2^22https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and Mistral-7B-Instruct(^3^33https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3). For each question, we retrieve five documents from 2018 Wikipedia corpus (Karpukhin et al., ) using Contriever-MSMARCO(^4^44https://huggingface.co/facebook/contriever-msmarco) (Izacard et al., ), so as to be consistent with previous studies (Xu et al., ; Yoon et al., )

For FaviComp, we use the same LM for both the compression model and the target model. Note that the compression model and the target model can be different, as long as they share the same tokenizer to ensure compatible token logits. Also, we set $\alpha$ to 0.5 by default, for which more analyses are given in Section 4.2. The prompts used in the experiment are presented in Appendix B.

### 3.3 Baselines

We consider the following categories of baselines. (1) No Context: RAG without any context. (2) Gold Document: RAG using directly relevant evidence from the retrieved documents if they exist. (3) Raw Document: RAG with raw documents that have not undergone any compression. (4) Generated Context (Yu et al., ): RAG with context generated by the same LM as the target model. This is equivalent to FaviComp with $\alpha=1$, as we rely solely on the target model to generate context when $\alpha=1$. (5) Reranking-based Methods: We rerank sentences in the evidence set and choose top-ranked sentences as the context. We utilize two rerankers—Sentence-BERT (Reimers & Gurevych, ) and RECOMP-extractive (Xu et al., ). (6) Compression-based Methods: We employ four compressors—LongLLMLingua (Jiang et al., ), RECOMP-abstractive (Xu et al., ), CompAct (Yoon et al., ), and Zero-shot Summarization. Zero-shot Summarization is instructed to summarize the evidence set into a concise summary based on the question, using the same LM as the target model. This is equivalent to FaviComp with $\alpha=0$, as we depend entirely on the compression model without any intervention from the target model. A detailed explanation of the implementation of the baselines is provided in Appendix A.

Table: Table 1: Experimental Results on five open-domain QA datasets using two LMs as the target model. ^† indicates a supervised method, where the reranker or the compressor is trained. The best Accuracy and token-level F1 scores for each dataset are in bold.

## 4 Experimental Results

In this section, we compare the overall performance of FaviComp with other baselines across the five datasets (Section 4.1), explore the impact of ensemble coefficient $\alpha$ on performance and perplexity (Section 4.2), investigate how effectively FaviComp incorporate parametric and non-parametric knowledge (Section 4.3), and compare the compression rates with other baselines (Section 4.4).

### 4.1 Main Results

The overall performance of FaviComp and the baselines across the five datasets is presented in Table 1. To start with, the compression-based methods consistently outperform the reranking-based methods, due to that the reranking-based methods are prone to losing more question-relevant information by discarding lower-ranked sentences. Next, FaviComp outperforms all other baselines across all the datasets, except for the Gold Document which is regarded as the upper bound of the performance. It is noteworthy that FaviComp, as a training-free, decoding-time strategy, outperforms supervised baselines, some of which are trained on the training set of the datasets. For the MuSiQue dataset, FaviComp even outperforms Gold Document baseline—which can be viewed as a perfect compressor—with Llama3-8B-Instruct, and shows comparable results with Mistral-7B-Instruct. This demonstrates that explicitly incorporating parametric knowledge from the target model can significantly enhance performance in multi-document QA, even when the context is imperfect.

Moreover, it is surprising that all the existing supervised compression-based methods are excelled by the Raw Document baseline. This indicates that existing methods are likely to fall short of retaining essential supportive information while compressing the evidence documents. Additionally, LongLLMLingua and RECOMP-abstractive perform worse than Zero-shot Summarization, possibly due to the use of smaller compression models (T5-large and Llama2-7B-Chat, respecitvely) compared to that used in Zero-shot Summarization (Llama3-8B-Instruct). However, it suggests that knowledge distillation from larger teacher LMs to the smaller compression models may not generalize well, as the context preferences and prior knowledge of the target model and the teacher model are likely to differ.

Furthermore, despite using the same base model for the compression model (Mistral-7B-Instruct), the training-free FaviComp outperforms CompAct, which trains the compression model using knowledge distillation to generate and evaluate summaries of retrieved documents. This also indicates that knowledge distilled from a teacher model may not always be effectively transferable to the target model due to discrepancies in context preference and prior knowledge. In contrast, the superior performance of FaviComp is attributed to its ability to familiarize evidence with the target model and its effective incorporation of parametric knowledge from ensemble decoding.

Finally, given that Zero-shot Summarization corresponds to FaviComp with $\alpha=0$ and Generated Context corresponds to FaviComp with $\alpha=1$, the fact that FaviComp outperforms both baselines highlights its ability to effectively incorporate tokens from both sources—evidence summary and generated context. This results in superior performance compared to relying on just one source alone.

### 4.2 Impact of Ensemble Coefficient on Performance and Perplexity

Figure: Figure 2: Impact of coefficient $\alpha$ on performance and perplexity.
Refer to caption: extracted/5864847/images/coef_exp.png

Figure 2 illustrates how performance and perplexity change as the ensemble coefficient $\alpha$ is varied across the values $\{0.0,0.1,0.3,0.5,0.7,0.9,1.0\}$ on NQ, HotpotQA and MuSiQue datasets. We calculate the perplexity of the compressed evidence conditioned on the preceding inputs, i.e. instruction, demonstrations, and the question. For all the datasets, performance is the highest when $\alpha=0.5$, indicating that proactively lowering perplexity by equally weighting both input sources yields the best results. Additionally, the performance tends to improve as the perplexity of compressed evidence decreases, which aligns with the previous works (Liu et al., ; Gonen et al., ). Interestingly, when $\alpha$ is equal to 0.9 or 1.0, there is a slight increase in perplexity. At high $\alpha$ values, FaviComp is more likely to generate context without referring to external knowledge. As a result, the LM’s increased uncertainty when generating context with limited evidential knowledge is likely the cause of the slight rise in perplexity. Results for other datasets are included in Figure 5.

### 4.3 Integration of Parametric and Non-parametric Knowledge

Figure: Figure 3: Accuracy of baselines (left) and FaviComp with various $\alpha$ values (right) on $Hits=0$ and $Hits=1$ subset of multi-document QA datasets.
Refer to caption: extracted/5864847/images/hits_exp.png

The effective integration of parametric and non-parametric knowledge is crucial for complex tasks such as multi-document QA, where the evidence set may not contain all the necessary information. To this end, we evaluate how effectively FaviComp incorporates parametric knowledge from the target model and non-parametric knowledge from the compression model on the multi-document QA datasets. We begin by dividing the test samples of each dataset into evidence-relevant and evidence-irrelevant subsets, using the $Hits$ metric. The $Hits$ metric is set to 1 (evidence-relevant) if the retrieved evidence set contains the correct answer, and 0 (evidence-irrelevant) if it does not. We then assess the downstream performance of each subset. The underlying intuition is that if a method performs better on the evidence-relevant subset, it suggests that the method is more effectively utilizing the provided evidential knowledge. Conversely, if a method excels on the evidence-irrelevant subset, it indicates that the method is more effectively leveraging parametric knowledge without relying on potentially irrelevant evidence.

The left figure of Figure 3 compares the accuracy in $Hits=0$ and $Hits=1$ subsets across the datasets. We compare FaviComp with the top-performing unsupervised compression method, Zero-shot Summarization, and the most competitive supervised compression method, CompAct. Compared to the other two baselines, FaviComp performs better in the $Hits=0$ subset while performing comparably in the $Hits=1$ subset. This proves that FaviComp effectively relies on parametric knowledge rather than evidential knowledge when faced with irrelevant evidence, while maintaining similar effectiveness in utilizing evidential knowledge when relevant evidence is present.

Interestingly, even though CompAct generally performs better on the $Hits=1$ subset compared to Zero-shot Summarization, it underperforms relative to Zero-shot Summarization on the $Hits=0$ subset. This suggests that the training may have been biased towards utilizing solely evidential knowledge, rather than effectively leveraging both sources in synergy.

We also evaluate the performance of FaviComp with various $\alpha$ values under this setting. The right figure of Figure 3 shows that $\alpha=0.5$ performs the best on the $Hits=0$ subset, while performance declines as $\alpha$ deviates further from 0.5. This pattern in the $Hits=0$ subset mirrors the overall performance trend, suggesting that appropriately utilizing parametric knowledge when the evidence is irrelevant is crucial to the overall performance. In the $Hits=1$ subset, performance remains consistent for $\alpha$ values up to 0.5 but decreases significantly when $\alpha$ exceeds 0.5 due to the diminished utilization of the relevant evidential context.

**Table 2: Compression rates of compression-based baselines. The highest compression rate is indicated in bold, while the second highest is underlined.**
| Methods | NQ | TQA | HotpotQA | 2WikiMQA | MuSiQue |
| --- | --- | --- | --- | --- | --- |
| LongLLMlingua | 1.87 | 1.84 | 1.83 | 1.83 | 1.83 |
| RECOMP-abstractive | 17.96 | 17.79 | 19.72 | 32.06 | 32.05 |
| CompAct | 8.85 | 8.92 | 9.45 | 10.71 | 8.96 |
| Zero-shot Summarization | 13.87 | 16.58 | 17.38 | 18.98 | 15.50 |
| FaviComp $(\alpha=0.5)$ | 15.79 | 20.99 | 21.49 | 22.41 | 19.06 |

### 4.4 Compression Rate Comparisons

Since one of the functionalities of compression-based RAG is to reduce the number of tokens from the evidence while keeping its essential information, we report the compression rate, calculated as $\tfrac{\textit{\# of tokens in retrieved documents}}{\textit{\# of tokens in
compressed documents}}$. Overall, FaviComp and RECOMP-abstractive consistently scores highest compression rates. RECOMP-abstractive exhibits significantly high compression rates because the compression model is trained to output an empty string when no relevant evidence is found, which is often the case in multi-document QA datasets. FaviComp compresses the evidence that is familiar to the target model by lowering the perplexity at each decoding step, typically resulting in a shorter context. Specifically, when compared to Zero-shot Summarization, which is equivalent to FaviComp with $\alpha=0$, FaviComp consistently achieves higher compression rates. This demonstrates that the ensemble decoding strategy, combining token logits from both evidence compression and context generation, leads to greater compression efficiency.

## 5 Case Study

Table 3 presents two examples from HotpotQA to illustrate how FaviComp effectively familiarizes evidence while seamlessly integrating both parametric and non-parametric knowledge during evidence compression. We compare its output with Raw Document, which does not apply any compression, and Zero-shot Summarization, which is equivalent to FaviComp with $\alpha=0$.

In both examples, Raw Document fails to produce the correct answer, even though the evidence contains the necessary information, highlighting the need for effective evidence compression. In the first example, while the difference between the compressed evidence from Zero-shot Summarization and FaviComp appears subtle, FaviComp delivers the correct answer with a lower perplexity in compression, underscoring the significance of evidence familiarization. The second example highlights the importance of parametric knowledge when the retrieved evidence set lacks complete information. Since the evidence set does not mention “Skeptic,” Zero-shot Summarization introduces irrelevant information (“Philanthropy magazine”), ultimately leading to an incorrect answer. In contrast, FaviComp integrates parametric knowledge about “Skeptic” and incorporates it into the evidence compression. Notably, FaviComp selects the $\arg\max$ token from the target model only when the token’s probability is higher than that of the compression model, demonstrating that FaviComp draws on parametric knowledge only when necessary—potentially when the compression model is uncertain about the next token.

**Table 3: Case study of evidence compression: FaviComp vs. Raw Document and Zero-shot Summarization. For FaviComp, the colors red and blue highlight tokens that are the $\arg\max$ of the compression model and the target model, respectively. Purple indicates a token that is the $\arg\max$ of neither model. Tokens with no coloring represent those that are the $\arg\max$ of both models.**
| Question: This film is an adaption of a Jacques Offenbach’s opera that was written by a Hungarian British screenwriter? |  |  |  |
| --- | --- | --- | --- |
| Methods | (Compressed) Evidence | Prediction | Perplexity |
| Raw Document | …(skip)… The Tales of Hoffmann is a 1951 British Technicolor film adaptation of Jacques Offenbach’s opera “The Tales of Hoffmann”, written, produced and directed by the team of Michael Powell and Emeric Pressburger working under the umbrella of their production company, The Archers. | Emeric Pressburger ✗ | 12.429 |
| Zero-shot<br>Summarization | The 1951 film “The Tales of Hoffmann” is an adaptation of Jacques Offenbach’s opera, written, produced, and directed by Michael Powell and Emeric Pressburger. | Emeric Pressburger ✗ | 2.298 |
| FaviComp | The 1951 film “The Tales of Hoffmann” is an adaptation of Jacques Offenbach’s opera, written by Emeric Pressburger, a Hungarian-British screenwriter, and directed by Michael Powell and Emeric Pressburger. | The Tales of<br>Hoffmann ✓ | 1.959 |
| Question: Which magazine was first published earlier, The Chronicle of Philanthropy or Skeptic? |  |  |  |
| Methods | (Compressed) Evidence | Prediction | Perplexity |
| Raw Document | The Chronicle of Philanthropy is a magazine that covers the nonprofit world. …(skip)… It was founded in 1988 by editor Phil Semas and then managing editor Stacy Palmer. …(skip)…<br>Philanthropy (magazine) Philanthropy is a quarterly magazine published by the Philanthropy Roundtable. First published as a newsletter in 1987, ”Philanthropy” became a glossy magazine in 1996. | Philanthropy ✗ | 4.856 |
| Zero-shot<br>Summarization | The Chronicle of Philanthropy was founded in 1988, while Philanthropy magazine was first published as a newsletter in 1987 and became a glossy magazine in 1996. | Philanthropy<br>magazine ✗ | 3.196 |
| FaviComp | The Chronicle of Philanthropy was first published in 1988, while Skeptic was first published in 1992. | The Chronicle of<br>Philanthropy ✓ | 1.345 |

## 6 Related Works

Evidence Compression for RAG.
Standard RAG retrieves textual evidence related to the prompt from the external corpora or knowledge bases and incorporates it as a part of the input to the LM (Lewis et al., ; Izacard & Grave, ; Guu et al., ). However, retrieved evidence pieces may contain inconsistent or irrelevant information to the question, potentially confusing the target model in downstream tasks (Shi et al., ). To tackle this problem, traditional approaches aim to rerank the textual evidence based on its relevance to the question and then select a top-ranked subset to include as part of the input to the LM (Nogueira et al., ; Zhuang et al., ). However, this approach lose more question-relevant information by discarding lower-ranked sentences.

Recent efforts on evidence compression seek to compress retrieved evidence pieces to filter out unnecessary information and retain only the essential context (Wang et al., ; Li et al., ; Jiang et al., ; Xu et al., ; Cao et al., ; Yoon et al., ). Wang et al. () filter query-relevant context using relevance metrics and Li et al. () extract query-relevant information and restructure them to form a consistent context. Jiang et al. () and Cao et al. () conduct token-level or embedding-based compression to preserve only the query-relevant information using a trained compressor. Xu et al. () and Yoon et al. () train a compression model to generate an abstractive summary of the documents by distilling knowledge from larger language models.
While these methods are successful to some extent, they often achieve suboptimal performance because the compressed context may be unfamiliar to the LM used in the downstream task due to differences in internal knowledge and prompt preferences between the compression and the target model.
In contrast, FaviComp proactively lowers the perplexity of the compressed evidence pieces using ensemble decoding technique without any training, thereby improving the downstream performance.

Parametric and Non-parametric Knowledge in RAG.
While there have been studies on the phenomena of LM’s utilization of both parametric and non-parametric knowledge sources (Longpre et al., ; Wadhwa et al., ; Wu et al., ; Zhang et al., ; Zhou et al., ; Wang et al., ; Fang et al., ), there is a lack of research focused on effectively synergizing both sources.
A few of these efforts introduces counterfactual augmentation (Longpre et al., ; Fang et al., ; Zhang et al., ) and causal intervention (Zhou et al., ; Wang et al., ) to mitigate knowledge conflict, which, however, requires explicitly knowing the features of the input that causes such conflict.
Zhang et al. () seek to address this issue by incorporating LM-generated context into the LM’s input along with the retrieved documents, thereby integrating both sources of knowledge.
However, merely concatenating both contexts is a suboptimal solution, as LMs may still show bias toward one source over the other when generating responses (Longpre et al., ; Wu et al., ). To address this, FaviComp employs ensemble decoding during the evidence compression, ensuring that both types of knowledge are seamlessly fused together to create a consistent context.

Constrained Decoding.
Constrained decoding has been previously proposed in text generation tasks for various purposes, including optimizing prompts (Liu et al., ), enhancing plausibility (Li et al., ) or controllability (Meng et al., ; Huang et al., ), and reducing hallucination (Shi et al., ).
Contrastive Decoding (Li et al., ) enforces a plausibility constraint during generation by inducing the difference in token log-probabilities between expert and amateur LMs.
Context-aware Decoding (Shi et al., ) uses contrastive decoding to amplify the probability differences between outputs with and without evidence, encouraging the LM to prioritize the evidential knowledge.
Our work is closely connected with the method by
Liu et al. () which employs ensemble decoding to paraphrase prompts to enhance zero-shot LM prompting and generalization.
Their approach focuses on the robustness and generalizability of instruction prompts for tasks without retrieval augmentation.
In contrast, our approach compresses externally retrieved evidence while integrating parametric knowledge during compression, specifically targeting knowledge-intensive tasks that require balancing both evidential and parametric knowledge.

## 7 Conclusion

In this study, we introduce FaviComp, a training-free evidence compression method designed to enhance the RAG by making retrieved evidence set more familiar to the target model, while seamlessly integrating parametric knowledge. By leveraging ensemble decoding, FaviComp proactively lowers the perplexity of the compressed evidence making it more favorable to the target model. Moreover, FaviComp effectively balances the target model’s parametric knowledge and the retrieved knowledge, improving performance on complex tasks where the retrieved evidence set may not contain all the necessary information. Our extensive experiments validate the effectiveness of FaviComp on open-domain QA tasks, showing significant improvements over existing baselines in multiple datasets. Additionally, FaviComp’s model-agnostic nature allows it to be effortlessly incorporated into various RAG workflows without additional training, making it a versatile tool for enhancing LMs in complex tasks.

## Appendix A Implementation Details

(1) Gold Document: We evaluate only on HotpotQA, 2WikiMQA, and MuSiQue, as these datasets contain gold documents. To identify the gold documents within the top-5 retrieved documents, we compare each gold document with the retrieved ones. If 50% or more of the content matches, we classify it as a gold document. If none of the retrieved documents are identified as gold, we utilize the entire set of retrieved documents as the context for the evaluation. This approach is necessary because the documents are chunked, and the retrieved documents may not exactly match the gold documents.

(2) Generated Context: We use the context generation prompt in Table 5 to generate the context.

(3) Zero-shot Summarization: We use the evidence compression prompt in Table 5 to compress the retrieved documents.

(4) RECOMP-extractive: We utilize the same Contriever models trained by the authors for each dataset, to encode both the question and the sentences in the evidence set. For 2WikiMQA and MuSiQue, since there are no fine-tuned models available, we use the Contriever fine-tuned on HotpotQA. Following the original paper, we select one sentence as the context for NQ and TQA, whereas for the other datasets, we utilize two sentences.

(5) RECOMP-abstractive: Similar to RECOMP-extractive, we use the same T5-large models trained by the authors for each dataset, to generate compress the retrieved evidence. For the 2WikiMQA and MuSiQue, we employ the T5-large model fine-tuned on HotpotQA.

(6) LongLLMLingua: We use Llama2-7B(^5^55https://huggingface.co/NousResearch/Llama-2-7b-hf) trained by the authors as the prompt compressor model. We use the default hyperparameters in the original paper, where dynamic context compression rate is set to 0.3 and maximum compression rate is set to 0.5.

(7) CompAct: We use the same Mistral-7B-Instruct(^6^66https://huggingface.co/cwyoon99/CompAct-7b) model instruction-tuned by the authors for evidence compression. The number of documents per segment is set to 5 with 1 iteration.

## Appendix B Prompt Templates

### B.1 Evaluation

The evaluation prompt template is shown in Figure 4. For all the evaluations throughout the experiment, we switch the positions of the Question and Context if doing so results in better performance. System prompts and demonstrations used in the evaluation are presented in Table 4 and Table 6, respectively.

### B.2 FaviComp

The prompt templates for evidence compression and context generation of FaviComp are presented in Table 5

Figure: Figure 4: Evaluation Prompt Template.

**Table 4: System prompts used in evaluation**
| Model | System Prompt |
| --- | --- |
| Llama-3-8B-Instruct | You are an expert in Question Answering. Your job is to answer questions in 1 to 5 words based on the given context. |
| Mistral-7B-Instruct | You are an expert in Question Answering. Your job is to answer questions in 1 to 5 words based on the given context. Just output the answer as concisely as possible, no other words |

**Table 5: Prompt Templates for FaviComp**
| Instruction | Prompt Template |
| --- | --- |
| Evidence Compression | You are an expert in summarization. Given a question and multiple document snippets, generate one summarized context that is helpful to answer the question. Just summarize, no other words.<br>Question: {Question}<br>Documents: {Evidence}<br>Summarized Context: |
| Context Generation | You are an expert in context generation. Given a question, generate a context that is helpful to answer the question. Just generate the context, no other words.<br>Question: {Question}<br>Context: |

Figure: Figure 5: Impact of coefficient $\alpha$ on performance and perplexity for TQA and 2WikiMQA.
Refer to caption: extracted/5864847/images/add_coef_exp.png

**Table 6: Demonstrations used in evaluation for each dataset**
| Dataset | Demonstrations |
| --- | --- |
| NQ | Question: who sings i’ve got to be me<br>Answer: Sammy Davis, Jr<br>Question: who wrote i will follow you into the dark<br>Answer: Ben Gibbard<br>Question: who won season 2 of total drama island<br>Answer: Owen (Scott McCord)<br>Question: what part of the mammary gland produces milk<br>Answer: cuboidal cells<br>Question: when did the golden compass book come out<br>Answer: 1995 |
| TQA | Question: Who sang the theme for the James Bond film ‘Thunderball’?<br>Answer: Tom Jones<br>Question: A hendecagon has how many sides?<br>Answer: Eleven<br>Question: In the 1968 feature film Chitty Chitty Bang Bang, of what country is Baron Bomburst the tyrant ruler?<br>Answer: Vulgaria<br>Question: Artists Chuck Close, Henri-Edmond Cross, John Roy, Georges-Pierre Seurat, Paul Signac, Maximilien Luce and Vincent van Gogh painted in what style?<br>Answer: Pointillism<br>Question: What is the study of the relation between the motion of a body and the forces acting on it?<br>Answer: Dynamics |
| HotpotQA | Question: Which magazine was started first Arthur’s Magazine or First for Women?<br>Answer: Arthur’s Magazine<br>Question: The Oberoi family is part of a hotel company that has a head office in what city?<br>Answer: Delhi<br>Question: Musician and satirist Allie Goertz wrote a song about the ”The Simpsons” character Milhouse, who Matt Groening named after who?<br>Answer: President Richard Nixon<br>Question: Are Jane and First for Women both women’s magazines?<br>Answer: Yes<br>Question: Were Pavel Urysohn and Leonid Levin known for the same type of work?<br>Answer: No |
| 2WikiMQA | Question: Where was the place of death of Marie Thérèse Of France (1667–1672)’s father?<br>Answer: Palace of Versailles<br>Question: Who is the paternal grandmother of Przemysław Potocki?<br>Answer: Ludwika Lubomirska<br>Question: Who lived longer, Herbert Findeisen or Léonie Humbert-Vignot?<br>Answer: Léonie Humbert-Vignot<br>Question: Are Alison Skipper and Diane Gilliam Fisher from the same country?<br>Answer: Yes<br>Question: Are director of film Move (1970 Film) and director of film Méditerranée (1963 Film) from the same country?<br>Answer: No |
| MuSiQue | Question: Who is the child of the director and star of Awwal Number?<br>Answer: Suneil Anand<br>Question: What is the record label of the rapper who performed Jigga My Nigga?<br>Answer: Roc-A-Fella Records<br>Question: What county shares a border with the county where Black Hawk Township is located?<br>Answer: Dodge County<br>Question: Who is the sibling of the person credited with the reinvention and popularization of oil paints?<br>Answer: Hubert Van Eyck<br>Question: Who heads the Catholic Church, in the country that a harp is associated with, as a lion is associated with the country that Queen Margaret and her son traveled to?<br>Answer: Eamon Martin |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please address the questions in Weaknesses.

Potential limitation: Does the method require the compression model and the target model to be from the same family, sharing an identical tokenizer? Otherwise, how would ensemble decoding be achieved if their vocabularies differ?
soundness: 3
strengths: - The idea of incorporating retrieved external knowledge with parametric knowledge via ensemble decoding is interesting.
- The proposed FaviComp is a training-tree method and can be easily applied to existing LLMs for content compression in retrieval-augmented generation (RAG).
- FaviComp achieves competitive performance compared to baseline methods across five QA benchmarks.
summary: This work introduces a training-free ensemble decoding algorithm named FaviComp for content compression in retrieval-augmented generation (RAG). It samples tokens from both the compression model and the target model to generate a summary of retrieved documents, aiming to filter out irrelevant information in retrieved documents and integrate external knowledge with the LLM’s internal parametric knowledge to improve accuracy. Experiments on five QA tasks show that the proposed FaviComp achieves competitive results compared to selected baselines.
weaknesses: - Unfair comparison with baseline models. Despite using the same backbone model as the target model, the proposed FaviComp uses more advanced Llama3-8B-instruct/Mistral-7B-Instruct as the compression model, while the baselines use less capable models, such as T5-large or Llama2-7B, for compression. What is the rationale behind this inconsistent choice? Can we use the same backbone model (e.g., Llama3-8B-instruct/Mistral-7B-Instruct) for compression in both the baseline methods (e.g., LongLLMLingua, RECOMP) and FaviComp to facilitate a fair comparison? Otherwise, this discrepancy can lead to a nontrivial impact on the evaluation results, making the experimental findings less convincing—it is unclear whether the improvement is due to an advanced compression model or the method’s design of FaviComp.
- Unfocused problem setup and lack of justification for using compression in RAG. According to Table 1, almost all compression-based models consistently underperform the naive RAG method (namely, “Raw Document”), which simply takes raw documents as input without filtering or compression. Does this imply that compression may actually be unnecessary in your problem setting? Typically, compression is introduced to filter out irrelevant information for improved accuracy or to reduce redundant tokens in the prompt for more efficient inference without significantly compromising accuracy [1,2]. If efficiency is the priority, a problem setting with a larger number of retrieved documents (e.g., k=20) would make more sense than the current setting. However, in this work, only k=5 documents are used. Does this suggest that the role of compression here is primarily focused on denoising the retrieved documents? Unfortunately, there is no comparison with RAG works specifically focused on denoising [3,4,5] in the experiment section. Can you clarify the purpose of compression in this work and justify its effectiveness in a more focused setting (e.g., k=30 for long-context RAG to assess the compression ability of FaviRAG and/or comparing with denoising RAG methods to evaluate the denoising ability of FaviRAG)?

**References**
1. xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token. NeurIPS 2024.
2. Context Embeddings for Efficient Answer Generation in RAG. arXiv 2024.
3. Certifiably Robust RAG against Retrieval Corruption. arXiv 2024.
4. InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales. arXiv 2024.
5. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. ICLR 2024.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: What is the inference efficiency (in terms of computational cost) of the proposed solution?
soundness: 3
strengths: This paper presents a training-free solution for adapting retrieved evidence to a target model, an interesting and innovative direction. 

Experimental results on LLaMA 3 8B Instruct and Mistral 7B Instruct demonstrate that the proposed solution outperforms trained ranker and compressor models.
summary: This paper introduces FAVICOMP (Familiarity-aware Evidence Compression), a training-free model designed to make retrieved evidence more familiar to a target model. FAVICOMP compresses evidence by minimizing perplexity, combining decoding probabilities from both the compression model and the target model to generate context that aligns more closely with the target model’s familiarity. Experimental results indicate that FAVICOMP outperforms baseline approaches.
weaknesses: While empirical results are provided, the intuitive design that relies on the perplexity of both the compressor and target models requires further justification. Given that target models often have limitations such as restricted knowledge and tendencies toward hallucination, it would be beneficial to address how these limitations impact the proposed approach’s effectiveness.

Missing discussion of related work:
Bridging the Preference Gap between Retrievers and LLMs

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See above.
soundness: 2
strengths: 1. Compression-based RAG presents a promising research direction in terms of reducing inference latency and improving robustness to irrelevant retrieved evidence.

2. This paper proposes an interesting perspective of enabling the target model to become familiar with the compressed summary.
summary: This paper proposes FAVICOMP, a training-free evidence compression technique that makes retrieved evidence more familiar to the target model, while seamlessly integrating parametric knowledge from the model.
FAVICOMP leverages the decoding probabilities of two LMs, a compression model and the target model. The compression model is instructed to summarize the raw evidential documents into a relevant context to the input, while the target model is instructed to generate relevant context without referencing the documents. 
By combining decoding probabilities from both the compression model and the target model, FAVICOMP can generate context that is more familiar to the target model.
Experiments on open-domain QA demonstrate that FAVICOMP shows better QA performance.
weaknesses: 1. The motivation of compression-based RAG is to improve the latency of standard RAG. However, ensembling both the decoding probabilities of the compression model and the target model does not save the inference costs compared to standard RAG. I'm not sure whether FAVICOMP even consumes more computation than standard RAG.

2. I have doubts about the validity of the assumption that ensembling decoding probabilities under different conditions. The compression model is decoding under line 159: (Evidence Compression Instruction, input question, retrieved documents), while the target model is decoding under line 180: (Context Generation Instruction, input question). Although authors explain that both objectives ultimately share the goal of generating context relevant to the question, this is too inconsistent in formulation.

3. Since compression-based RAG is to improve the latency of standard RAG, the compression model should be more lightweight than the target model. For example, RECOMP adopts T5 (770M) as the compression model and Flan-UL2 as the target model (20B). However, the compression models are the same as the target models for FAVICOMP, which does not make sense.

4. Since FAVICOMP does not use a compression model of the same scale, the results in Table 1 are unfair. FAVICOMP would likely achieve better results, as it employs a more powerful compression model.

5. Reporting only the QA results in Table 1 is insufficient, as the compression rate is just as important as accuracy and F1. There’s a trade-off between compression rate and QA performance, so it would be better to combine Table 1 and Table 2. It’s difficult to assess the effectiveness of FAVICOMP without considering the compression rate.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What if the target LLM is considered during compression and the target LLM is in the form of an API or uncertain?
2. The author should rigorously define what is evidence that the target LLM is familiar with. In addition, if only the target LLM is familiar with it, does familiarity necessarily mean high compression quality? Does familiarity necessarily mean that the target LLM will use the compressed evidence to the greatest extent? What is the theoretical basis for the starting point?
3. Ablation: (1) Only allowing the target LLM to provide evidence (that is, only the formula of L181, that is, 2-time inference) will exceed the raw document in 2wikiMQA, which is inconsistent with other datasets. (2) What if the compression model is the target LLM? This situation implies that the compression evidence is familiar with the target LLM. In fact, referring to L231-L236, the target LLM and the compressed LLM are consistent (so I question the starting point of L49-L52. In essence, it is more like an ensemble in which the prompt generates different evidences at different times, using the target LLM to generate evidence and the original document to generate evidence respectively). (3) In the Llama model, have you ever tried the situation where the compressed LLM and the target LLM are inconsistent? The parameter size is inconsistent or the model source is inconsistent. (4) L307-L309, why can the performance of the model exceed the performance on the gold document after compression? Shouldn't this be the upper limit? What is the reason for this phenomenon? (5) How does the experiment of concating the compressed texts generated separately from the two parts perform?
5. Considering that the method uses the internal knowledge of LLM when regenerating evidence (which may increase the number of reasoning steps), I hope to see some analysis of the corresponding multi-document and multi-hop QA advantages.
soundness: 2
strengths: 1. An effective prompt compression method is proposed, which achieves the optimal compression ratio on multiple datasets while improving the performance indicators of the model.
2. No training is required, and compression is performed during inference, which reduces the training cost, but there is also a concern that the inference time will increase.
3. The paper is generally clear and the presentation is generally clear.
summary: The authors propose a training-free evidence compression technique named FaviComp. Specifically, FaviComp actively combines compressed evidence to reduce the perplexity of the target model by combining the decoding probabilities of the compression model and the target model to generate a more familiar context for the target model. Experimental results show that FaviComp consistently outperforms the state-of-the-art evidence compression baselines on multiple open-domain QA datasets.
weaknesses: 1. Some details need to be clarified. The theoretical basis and intuition of the motivation need to be further elaborated.
2. The novelty of the method is limited. In essence, it is more like an ensemble that generates different evidences at different prompts, using the target LLM to generate evidence and the original document to generate evidence.
3. Experimental ablation needs to be increased.