ECoRAG: Evidentiality-guided Compression for Long Context RAG
Authors: Yeonseok Jeong, Jinsu Kim, Dohyeon Lee, seung-won hwang
Accepted at: acl25

ECoRAG: Evidentiality-guided Compression for Long Context RAG

Yeonseok Jeong1, Jinsu Kim2, Dohyeon Lee3, Seung-won Hwang3*
IPAI, Seoul National University1, Korea University2, Seoul National University3
{jys3136, waylight3, seungwonh}@snu.ac.kr
tonmmy222@korea.ac.kr

Abstract
Large Language Models (LLMs) have shown remarkable performance in Open-Domain Question Answering (ODQA) by leveraging external documents through Retrieval-Augmented Generation (RAG). To reduce RAG overhead, from longer context, context compression is necessary. However, prior compression methods do not focus on filtering out non-evidential information, which limit the performance in LLM-based RAG. We thus propose Evidentiality-guided RAG, or EC-oRAG framework. ECoRAG improves LLM performance by compressing retrieved documents based on evidentiality, ensuring whether answer generation is supported by the correct evidence. As an additional step, EC-oRAG reflects whether the compressed content provides sufficient evidence, and if not, retrieves more until sufficient. Experiments show that ECoRAG improves LLM performance on ODQA tasks, outperforming existing compression methods. Furthermore, ECoRAG is highly cost-efficient, as it not only reduces latency but also minimizes token usage by retaining only the necessary information to generate the correct answer. Code is available at https://github.com/ldilab/ECoRAG.

1 Introduction
LLMs (OpenAI, 2023; Touvron et al., 2023) have excelled in tasks such as ODQA by leveraging external knowledge through RAG (Lewis et al., 2020; Ram et al., 2023). However, RAG inevitably increases context length, which incurs higher computational cost and also hinders generation quality (Liu et al., 2024; Hsieh et al., 2024; Li et al., 2024).

While adopting existing context compression (Li et al., 2023) may look promising, such a baseline presents two main challenges. First, LLMs are known to be vulnerable to irrelevant contents that cannot provide evidence for answer generation (Shi et al., 2023; Qian et al., 2024; Wu et al., 2024), and existing compression methods (Xu et al., 2024; Jiang et al., 2024; Yoon et al., 2024) do not effectively filter them out. As a result, a naive baseline simply prepending retrieved documents, ‘standard RAG’ in Figure 1, outperforms a baseline compressor RECOMP (Xu et al., 2024). As the number of documents increases, a baseline compressor fails to filter out increasing irrelevant contents, causing performance to decline.

Second, it is challenging to determine the desirable compression ratio for each question. Failure to do so may lead to compressing too much, which results in losing crucial information, or compressing too little, which produces overly long contexts that degrade generation quality (Liu et al., 2024;

![Comparison of performance between prepending retrieved documents (standard RAG) (Karpukhin et al., 2020), applying RECOMP (Xu et al., 2024), and applying ECoRAG on the Natural Questions (Kwiatkowski et al., 2019) test set. Experiments were conducted using Flan-UL2 (Tay et al., 2023).](page_1012_370_388_388.png)

Figure 1: Comparison of performance between prepending retrieved documents (standard RAG) (Karpukhin et al., 2020), applying RECOMP (Xu et al., 2024), and applying ECoRAG on the Natural Questions (Kwiatkowski et al., 2019) test set. Experiments were conducted using Flan-UL2 (Tay et al., 2023).
Hsieh et al., 2024; Li et al., 2024) and increase computational costs. Thus, it is necessary to find the desirable compression ratio that enables the LLM to generate the correct answer for each question.

Our distinction is using evidentiality to address both challenges and proposing Evidentiality-guided Compression and Retrieval-Augmented Generation (ECoRAG) framework: Ours compresses retrieved documents to retain only the information necessary to support the answer. To overcome the first challenge, evidentiality (Lee et al., 2021; Asai et al., 2022) is used to determine whether each sentence in the retrieved documents supports the correct answer to a question. It can be quantified for each sentence by measuring how much it contributes to the model to generate the correct answer. We train the compressor using this as training signals.

To address the second challenge, ECoRAG reflects on compression as a collective, where it contains sufficient evidence. We begin by forming the smallest possible collective unit of compression and assess whether it is evidential. If not, it means that it is compressed too much, which we adjust adaptively by collecting more, until it is sufficient. Through this reflection process, ECoRAG finds the desirable compression ratio that enables the LLM to generate the correct answer with minimal tokens.

By applying these methods, ECoRAG has two advantages when dealing with long contexts as the number of documents increases. First, ECoRAG improves performance by retaining only the information necessary for generating the correct answer and removing distracting content. This results in gains on ODQA datasets such as Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (TQA) (Joshi et al., 2017), WebQuestions (WQ) (Berant et al., 2013). Second, by compressing the long context to only what is needed, it reduces computational costs.

Our contributions to this work can be summarized as follows: (1) Evidentiality-guided Compression: We developed a method that compresses retrieved documents based on evidentiality. (2) Evidentiality Reflection for Adaptive Compression: Our framework evaluates compressed content for evidentiality and adaptively adjusts the length of compression. (3) Experiments show that our approach significantly improves retrieval-augmented LLM performance on ODQA datasets. (4) Our approach is also cost-efficient, as it quickly compresses long context, reducing latency and tokens.

2 Related Work

2.1 Evidentiality-guided RAG
Dense retrievers (Karpukhin et al., 2020; Izacard et al., 2022) focus on lexical answerability, but may mislabel documents as relevant when they lack contextual evidence, leading to the need for evidentiality. In prior work (Lee et al., 2021), evidentiality refers to whether a document supports generating the correct answer to a question. Unlike answerability, evidentiality is more challenging to mine directly as it reflects the contextual relationship between a question and a document. To measure evidentiality, previous work checks whether the removal of the document is critical for answering the question (Asai et al., 2022), utilizes attention scores (Niu et al., 2020), or considers the change in confidence scores (Song et al., 2024). Our work introduces evidentiality in LLMs, enhancing RAG by prioritizing contextually rich documents for generating correct answers.

2.2 Prompt Compression
Numerous studies (Mu et al., 2024; Li et al., 2023; Kim et al., 2024) have focused on prompt compression to address both cost and performance challenges, as shown in prior research (Shi et al., 2023; Liu et al., 2024; Hsieh et al., 2024). RECOMP (Xu et al., 2024) provides both extractive and generative summaries of documents, considering whether the summaries helped answer the given question. LLM-Lingua (Jiang et al., 2023b) uses conditional probabilities of LLMs to guide fine-grained prompt compression. Building on this, LongLLMLingua (Jiang et al., 2024) compresses prompts in long context scenarios by using a question-aware coarse-to-fine compression and document reordering mechanism. Similarly, CompAct (Yoon et al., 2024) employs an adaptive compression strategy to iteratively compress documents while retaining key information relevant to the query. However, existing methods struggle to compress long context, which prevents them from fully utilizing the retrieval results.

2.3 Retrieval Evaluation for RAG
LLMs may evaluate the quality of retrieved results for enhancing RAG, as seen in Madaan et al. (2024), where models iteratively improve their responses; this concept has been applied to RAG. Self-RAG (Asai et al., 2024) trains LLM to evaluate retrieved documents and its output by predicting reflection tokens that assess the need for retrieval
and the quality of the generated text. Labruna et al. (2024) dynamically determines whether to retrieve additional context when needed by using a trained reader LLM. CRAG (Yan et al., 2024) employs a retrieval evaluator to assess document relevance and triggers corrective actions to refine retrieved information, by using lexical overlap between questions and documents. In our ECoRAG framework, we evaluate whether the evidence is sufficient to generate the correct answer by leveraging evidentiality as defined by the LLM.

3 Proposed Method

In this section, we describe how ECoRAG adaptively adjusts the compression length to ensure that the LLM generates the correct answer. To achieve this, we focus on: (1) compressing retrieved documents by sorting them based on evidentiality (Section 3.1), and (2) evaluating whether the compressed documents is sufficiently evidential, and if not, adaptively incorporating more information (Section 3.2), and Figure 2 provides an overview.

3.1 Evidentiality-guided Compressor

This section explains how retrieved documents are compressed while preserving the evidence that enables the LLM to generate the correct answer. We decompose documents into sentences inspired by Xu et al. (2024) and compress them guided by evidentiality. To retain the necessary content and remove irrelevant parts during the compression process, we first extract evidential sentences from the retrieved documents (Section 3.1.1) and then use them to train the compressor (Section 3.1.2).

3.1.1 Definition of Evidentiality

We define the evidentiality of a sentence based on its contribution to generating the correct answer while penalizing distractors that interfere with this process. The degree of evidentiality is categorized hierarchically based on two conditions. We find sentences that enable the LLM to generate the correct answer. If a sentence does not, we then check if it interferes with other evidence.

First, when assessing whether each sentence helps generate the correct answer, it is important to consider that the LLM contains parametric knowledge (Wang et al., 2020; Yu et al., 2023; Luo et al., 2023). Prior work (Lee et al., 2021; Asai et al., 2022) has focused on whether the language model could contribute to generating the correct answer using given document. However, it is challenging to distinguish whether the correct answer was generated using the document or parametric knowledge, especially in larger models. If the correct answer was generated solely using parametric knowledge, regardless of the given document, it is unclear to determine whether the document serves as key evidence. Therefore, we propose the following first condition: [1] Without the sentence the LLM cannot generate the correct answer alone, but with the sentence it can.

Second, it is also crucial for the compressor to filter out distractors that hinder the evidence from generating the correct answer. While robustness to distractors can be improved through fine-tuning (Liu et al., 2024), training LLMs often requires substantial costs for training and closed LLMs often impossible to train. If the compressor can remove distractors, it can be applied to any LLM without requiring additional training. To identify distractors, we introduce a second condition for sentences that do not satisfy [1]: [2] The sentence does not interfere with the evidence defined in [1] in generating the correct answer.

Based on the aforementioned conditions, we hierarchically define evidentiality as depicted in Figure 3. Sentences satisfying condition [1] are labeled as strong evidence. Sentences failing to meet condition [1] are further classified based on condition [2]: those satisfying condition [2] are labeled as weak evidence, while those that do not are classified as distractor. Following these conditions, we use an LLM to label sentences in retrieved documents for each question in the training data.

3.1.2 Learning Objective for Compressor

Given labeled sentences \( \mathcal{D} = \{d_1, d_2, \cdots, d_{|\mathcal{D}|}\} \), for a question \( q \), we train our compressor based on dual encoders (Izacard et al., 2022) to differentiate between strong and weak evidence, as well as distractor. Using dual encoders, \( E_Q \) for questions and \( E_D \) for sentences, we calculate the similarity score between \( q \) and sentences in \( \mathcal{D} \) (i.e., \( sim(q, d_i) = E_Q(q) \cdot E_D(d_i) \)). Sentences are categorized into strong (\( d^+ \)) or weak (\( d^+ \)) evidence, and distractor (\( d^- \)) based on our hierarchical definition. We define similarity scores as \( s^* = sim(q, d^*) \), \( s^+ = sim(q, d^+) \), and \( s^- = sim(q, d^-) \). The similarity scores are utilized to train two inequalities:

\[
(s^+ > s^-), \quad (s^* > s^+, s^-)
\]
Figure 2: This figure illustrates the overall framework of ECoRAG. First, the evidentiality-guided compressor compresses the retrieved documents by sorting decomposed sentences based on evidentiality, producing an ordered set of evidences \( d'_1, d'_2, \ldots, d'_{|D|} \). Second, evidentiality reflection starts with the top-ranked sentence (\( n = 1 \), i.e., \( C = d'_1 \)), and the evidentiality evaluator determines whether \( C \) is evidential. If not, more evidence is added iteratively (\( n = k \rightarrow n = k + 1 \)) until the evaluator judges \( C \) as evidential. Once evidential, it is used for final compression (green line); otherwise, additional evidence is collected (red line).

These inequalities ensure that strong evidence is ranked above weak evidence, which in turn is ranked above distractor, guiding the training of our compressor.

The weak evidentiality loss \( \mathcal{L}_{we} \) uses the InfoNCE loss to distinguish weak evidence \( d^+ \) from distractor \( d^- \). The loss function is formulated as:

\[
\mathcal{L}_{we} = - \log \frac{\exp(s^+ / \tau)}{\exp(s^+ / \tau) + \sum_{d_j^- \in D^-} \exp(s_j^- / \tau)}
\]

Here, \( s_j^- = sim(q, d_j^-) \) represents the similarity score for each distractor in the set \( D^- \), and \( \tau \) is a temperature parameter.

The strong evidentiality loss \( \mathcal{L}_{se} \) also utilizes the InfoNCE loss to prioritize strong evidence \( d^* \). The loss function is formulated as:

\[
\mathcal{L}_{se} = - \log \frac{\exp(s^* / \tau)}{\exp(s^* / \tau) + \sum_{d_j^+ \in D^- \cup D^+} \exp(s_j^+ / \tau)}
\]

Here, \( s_j^+ = sim(q, d_j^+) \) is the similarity score for each sentence in the combined sets of distractors \( D^- \) and weak evidences \( D^+ \).

The final loss \( \mathcal{L} \) is defined as the sum of the strong and weak evidentiality losses:

\[
\mathcal{L} = \mathcal{L}_{se} + \mathcal{L}_{we}
\]

Our compressor is trained using this loss \( \mathcal{L} \), and ranks sentences \( d'_1, d'_2, \ldots, d'_{|D|} \) by evidentiality, selecting high-scoring ones for compression. The number of sorted evidence required can vary depending on the difficulty of each question. However, providing too little evidence may omit important information, while too much increases computational costs for each question. Thus, balanced compression ratio is necessary for each question to address both issues.

3.2 Evidentiality Reflection for Adaptive Compression

Once a collective of evidential sentences is formed, we need to determine whether the compression ratio is appropriate. To achieve this, we reflect on the evidentiality of compressed documents using a language model (Section 3.2.1). Then, if compressed too much, we adaptively adjust the compression ratio by collecting more (Section 3.2.2).

3.2.1 Training Evidentiality Evaluator

We develop an effective evidentiality evaluator \( \mathcal{M}_{eval} \) that assesses whether the compressed documents are strong evidence enough to generate the correct answer. In prior work, CompAct (Yoon et al., 2024) trained the evaluator by prompting GPT-4o (OpenAI, 2023) to determine if the evidence is sufficient to answer the question. However, this approach can introduce bias (Chiang and Lee, 2023) when GPT-4o evaluates through prompting, leading to inaccurate supervision. Accurate supervision requires verifying if the document actually enables the reader LLM to generate the correct
<table>
  <tr>
    <th>q:</th>
    <td>When is the next deadpool movie being released?</td>
  </tr>
  <tr>
    <th>a:</th>
    <td>May 18, 2018</td>
  </tr>
  <tr>
    <th>d_1:</th>
    <td>"Deadpool 2" was released on May 18, 2018.</td>
  </tr>
  <tr>
    <th>d_2:</th>
    <td>"Deadpool 2" is the next movie of Deadpool.</td>
  </tr>
  <tr>
    <th>d_3:</th>
    <td>Spider-Man and Deadpool often team up in Marvel.</td>
  </tr>
</table>

(a) Example of question, answer, and sentences for evidentiality mining

![Diagram showing LLM processing question and evidence, with strong evidentiality mining](page_495_176_495_143.png)

(b) strong evidentiality mining

![Diagram showing LLM processing question and evidence, with weak evidentiality mining](page_495_340_495_143.png)

(c) weak evidentiality mining

Figure 3: This figure illustrates the evidentiality mining strategy of ECoRAG.

answer. To achieve this, we reuse our evidentiality labels obtained from the LLM in Section 3.1.1 and distill them from our reader LLM into smaller model, Flan-T5-large (Chung et al., 2022), to build the evaluator. Comparison between CompAct and our evaluator is discussed in Section 5.2.

We train \( \mathcal{M}_{eval} \) using our evidentiality labeled dataset \((d^*, d^+, d^-)\) to determine if compressed documents are sufficient for correct answer generation. The evaluator is trained to classify whether the given compressed documents is strong evidence. To facilitate this, we add 2 special tokens \( t \in [<EVI>, <NOT>] \) and train \( \mathcal{M}_{eval} \) to generate '<EVI>' for strong evidence \( d^* \), and '<NOT>' for other sentences \( d^+, d^- \). Subsequently, next-token prediction loss \( \mathcal{L}_{eval} \) is used for this training stage to predict whether compressed documents are strong evidence.

\[
\mathcal{L}_{eval} = -log\ p_{\mathcal{M}_{eval}}(t|q, d)
\]

3.2.2 Adaptive Compression

In adaptive compression, the compression ratio is adaptively adjusted by our evaluator, which reflects on whether the current compression is evidential, as described in Figure 2. Initially, our evaluator assesses the evidentiality of compressed documents \( C \) containing only the first evidence, \( d_1' \), from our ordered evidences \( d_1', d_2', \ldots, d_{|D|}' \). If the evaluator determines that \( C \) is evidential, it becomes the final compression provided to LLM. If \( C \) is not evidential, we add the next piece of evidence \( d_2' \) to \( d_1' \) to build new compressed documents. If the k-th iteration fails, \( d_{k+1}' \) is added to the previously compressed documents. This process is repeated until the desirable compression is found, with a token limit set to avoid infinite loop. Since retrieved documents do not always include gold evidence for all queries, a token limit is necessary to prevent infinite loops from continuously adding evidence. The final compression is then used as input for the LLM, which generates the final answer.

Although iterative adjustment can increase latency compared to using raw documents, ECoRAG reduces it efficiently. Prior work (Yoon et al., 2024), each iteration required LLM (7B) to generate a new compression by using the previous compression and the next piece of evidence. Thus, with each iteration, LLM reads different contents and generates compression of multiple tokens, increasing latency time. However, ECoRAG reduces redundancy by ordering evidence just once and adding it iteratively. Moreover, our framework utilized a lightweight evaluator (0.77B) that adjusts compression length by generating just a single special token, resulting in rapid compression speed; the actual results are shown in Section 5.4.

4 Experiments

4.1 Experimental Settings

Datasets We evaluate our framework through NQ (Kwiatkowski et al., 2019), TQA (Joshi et al., 2017), and WQ (Berant et al., 2013), which are ODQA datasets. We use the 100 documents retrieved from DPR (Karpukhin et al., 2020)1.

Models We initialize our evidentiality compressor from Contriever (Izacard et al., 2022) and use it to compare its performance with RECOMP (Xu et al., 2024). For evidentiality evaluator, we utilize Flan-T5-large (Chung et al., 2022), because previous RAG and document-assessment work (Han et al., 2023; Yan et al., 2024) have successfully employed it. Detailed justification for this choice can

1Since enhancing the retriever is beyond the scope of this study, we conduct our experiments under the assumption that the retrieved documents are already provided.
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">NQ</th>
    <th colspan="3">TQA</th>
    <th colspan="3">WQ</th>
  </tr>
  <tr>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <th colspan="10"><i>RAG without compression</i></th>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>0</td>
    <td>31.88</td>
    <td>44.10</td>
    <td>0</td>
    <td>64.78</td>
    <td>73.10</td>
    <td>0</td>
    <td>24.51</td>
    <td>42.73</td>
  </tr>
  <tr>
    <td>standard RAG (100 documents)</td>
    <td>13905</td>
    <td>36.09</td>
    <td><b>50.18</b></td>
    <td>14167</td>
    <td>56.21</td>
    <td>64.22</td>
    <td>13731</td>
    <td>21.11</td>
    <td>38.72</td>
  </tr>
  <tr>
    <th colspan="10"><i>RAG with 100 documents compressed</i></th>
  </tr>
  <tr>
    <td>LLMLingua (Jiang et al., 2023b)</td>
    <td>635</td>
    <td>26.84</td>
    <td>38.30</td>
    <td>630</td>
    <td>50.81</td>
    <td>57.91</td>
    <td>641</td>
    <td>22.98</td>
    <td>39.77</td>
  </tr>
  <tr>
    <td>LLMLingua-2 (Pan et al., 2024)</td>
    <td>1315</td>
    <td>30.11</td>
    <td>42.52</td>
    <td>1324</td>
    <td>53.19</td>
    <td>60.46</td>
    <td>1113</td>
    <td>23.52</td>
    <td>40.61</td>
  </tr>
  <tr>
    <td>LongLLMLingua (Jiang et al., 2024)</td>
    <td>1370</td>
    <td>32.96</td>
    <td>45.32</td>
    <td>1402</td>
    <td>55.75</td>
    <td>63.75</td>
    <td>1355</td>
    <td>21.51</td>
    <td>39.13</td>
  </tr>
  <tr>
    <td>RECOMP (extractive) (Xu et al., 2024)</td>
    <td>662</td>
    <td>32.85</td>
    <td>44.54</td>
    <td>672</td>
    <td>51.66</td>
    <td>59.08</td>
    <td>658</td>
    <td>19.54</td>
    <td>36.83</td>
  </tr>
  <tr>
    <td>RECOMP (abstractive) (Xu et al., 2024)</td>
    <td><b>14</b></td>
    <td>27.59</td>
    <td>39.19</td>
    <td><b>26</b></td>
    <td>39.95</td>
    <td>46.68</td>
    <td><b>19</b></td>
    <td>20.47</td>
    <td>36.90</td>
  </tr>
  <tr>
    <td>CompAct (Yoon et al., 2024)</td>
    <td>106</td>
    <td>35.71</td>
    <td>47.14</td>
    <td>96</td>
    <td>63.96</td>
    <td>73.87</td>
    <td>75</td>
    <td>29.77</td>
    <td>44.25</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>632</td>
    <td><b>36.48</b></td>
    <td>49.81</td>
    <td>441</td>
    <td><b>65.34</b></td>
    <td><b>75.37</b></td>
    <td>560</td>
    <td><b>30.17</b></td>
    <td><b>46.13</b></td>
  </tr>
</table>

Table 1: Compression methods performance comparison on NQ, TQA, and WQ. The table shows the results using GPT-4o-mini as the reader model, given 100 retrieved documents (Karpukhin et al., 2020). It reports the number of tokens after compression, along with EM and F1-score, illustrating the impact of different compression methods on model performance.

be found in Section B.2. For the reader model, we use GPT-4o-mini (OpenAI, 2023), as it supports a context length of 128K tokens, sufficient to process all 100 retrieved documents.

Evaluation Metrics We report results on the test sets of NQ, TQA, and WQ using EM and word-level F1-score to assess the question-answering task performance. We also report the average number of input tokens given to the reader LLM to evaluate the efficiency of our compression step.

Baseline We report two types of baselines.

<i>RAG without compression</i>: As a baseline, we report the results using only the question and raw retrieved documents. The ‘closed-book’ setting, where no retrieval is used, shows that the model relies solely on its internal knowledge. In the ‘standard RAG’ setting, we simply concatenate the top 100 retrieved documents without any compression for evaluation.2 This is the approach used in conventional RAG without compression.

<i>RAG with 100 compressed documents</i>: We also reproduce several retrieval augmentation methods for comparison. To better understand the effect of different compression methods, we evaluated several baselines including LLMLingua (Jiang et al., 2023b), LLMLingua-2 (Pan et al., 2024), LongLLMLingua (Jiang et al., 2024), CompAct (Yoon et al., 2024), and RECOMP which offers both extractive and abstractive variants.

In addition to our compression and non-compression baselines, we include BGE-M3 (Chen et al., 2024) and BGE-reranker (Xiao et al., 2024)

2We also evaluated the effect of reducing the number of retrieved documents (\( k = 5, 10, 20 \)) for both DPR and Contriever in Table 16; results are explained in Section A.9.

under equal token budgets. However, since these are not compression methods, their comparison results are addressed in Section A.4.

4.2 Results

In this section, we report the results of our model and compare them with both compression-based and non-compression baselines for ODQA in Table 1. Accuracy, such as EM and F1-score, is a more important metric than token reduction for evaluating compression quality because simply reducing tokens without preserving necessary information is meaningless. A method is more efficient if it reduces more tokens while maintaining higher accuracy than another.

In terms of accuracy, ECoRAG outperforms all baselines, including standard RAG, where the LLM reads all retrieved information. In the long context setting, retrieving many documents often brings in those with low relevance scores, introducing noise. However, previous compression methods fail to filter out this noise, leading to performance degradation compared to uncompressed approaches. Notably, ECoRAG surpasses all these methods, even with fewer tokens than some of them. The strength of ECoRAG lies in compressing only the necessary content, focusing solely on the information essential for generating the correct answer. As a result, ECoRAG outperforms the strongest compression baseline in NQ (+0.77%p), TQA (+1.38%p), and WQ (+0.40%p) in EM. As further detailed in Section A.10, ECoRAG maintains this advantage even on much longer retrieved documents, confirming its robustness in another long context setting (Bai et al., 2024).

From a token efficiency perspective, ECoRAG
<table>
  <tr>
    <th>Methods</th>
    <th>NDCG@1</th>
    <th>NDCG@10</th>
  </tr>
  <tr>
    <td>Answerability (baseline)</td>
    <td>67.82</td>
    <td>79.20</td>
  </tr>
  <tr>
    <td>Leave-One-Out (Asai et al., 2022)</td>
    <td>70.67</td>
    <td>80.80</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td><b>75.53</b></td>
    <td><b>81.92</b></td>
  </tr>
</table>

Table 2: Comparison of NDCG@1 and NDCG@10 on HotpotQA dataset using different training signals

uses more tokens than RECOMP (abstractive) and CompAct but still outperforms them, while compressing with fewer tokens than other methods. According to Xu et al. (2024), abstractive RECOMP performs well in the 5-document setting but struggles in long contexts due to input size limitations. CompAct suffers from inaccurate compression evaluation, failing to retain essential information, which lowers performance. In contrast, ECoRAG can handle long context and retain only the necessary content to generate the correct answer, which results in superior performance across different datasets. Excluding the two compressors that fail to preserve necessary information, ECoRAG achieves higher accuracy with fewer tokens than other methods, demonstrating its token efficiency.

5 Analysis

In addition to the main results, we verified the effectiveness of our framework by addressing the following research questions:

• RQ1: Does our compressor effectively capture human-annotated evidence?

• RQ2: How accurately does our evaluator predict evidentiality?

• RQ3: What is the impact of each component in ECoRAG?

• RQ4: Is ECoRAG efficient compression?

5.1 RQ1: Alignment with Human-annotated Evidentiality

In this section, we assess whether our compressor can effectively sort sentences by evidentiality for next step. Although our compressor improves LLM performance by learning LLM-defined evidentiality, it is essential to verify whether it effectively captures ground-truth evidence. Thus, we conducted experiments using HotpotQA (Yang et al., 2018), which provides human-annotated evidence. We compared how well prior methods and our compressor assign higher scores to ground-truth evidence. For evaluation, we use Normalized Discounted Cumulative Gain (NDCG) as a metric to evaluate how effectively evidentiality-focused methods, including ours, rank evidence higher.

As shown in Table 2, ECoRAG achieved the highest performance, demonstrating strong alignment with human-annotated evidentiality. The ‘Answerability’ baseline trains the compressor by treating passages containing the correct answer as positive and those without as negative. The ‘Leave-One-Out’ (Asai et al., 2022) considers a passage as positive if removing it prevents the model from generating the correct answer, and negative if the model still succeeds. ECoRAG outperforms prior evidentiality baselines, achieving improvements in NDCG@1 (+4.86%p) and NDCG@10 (+1.12%p). This result indicates that our compressor effectively captures evidence and aligns well with human annotations. Thus, our compressor provides well-sorted evidences to our evaluator, then we need to verify the evaluator, the other component of ECoRAG.

5.2 RQ2: Evaluator Performance on Evidentiality Prediction

![Bar chart showing evidentiality evaluation metrics using different evaluator, including ours, measured on the TQA.](page_1012_1042_388_246.png)

Figure 4: Evidentiality evaluation metrics using different evaluator, including ours, measured on the TQA.

We also need to verify the evidentiality evaluator to accurately evaluate whether the compressed documents enable the LLM to generate the correct answer. To assess its accuracy, we conducted experiments on the TQA test set. For each question, we define ground-truth labels for retrieved documents as either <EVI>, which lead to generating the correct answer as in Section 3.2.1, or <NOT>. We then measured how well our evaluator and other evaluators predicted these labels using accuracy, precision, recall, and F1-score. The results are shown in Figure 4.

Across all metrics, our evidentiality evaluator effectively predicts evidentiality, even though it has significantly fewer parameters than other
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">NQ</th>
    <th colspan="2">TQA</th>
  </tr>
  <tr>
    <th>EM</th>
    <th>R20</th>
    <th>EM</th>
    <th>R20</th>
  </tr>
  <tr>
    <td>(A) ECoRAG (ours)</td>
    <td><b>36.48</b></td>
    <td><b>75.18</b></td>
    <td><b>65.43</b></td>
    <td><b>80.38</b></td>
  </tr>
  <tr>
    <th colspan="5"><i>Compressor</i></th>
  </tr>
  <tr>
    <td>(B) w/o answerability</td>
    <td>31.25</td>
    <td>49.53</td>
    <td>63.86</td>
    <td>70.84</td>
  </tr>
  <tr>
    <td>(C) w/o evidentiality</td>
    <td>35.46</td>
    <td>74.93</td>
    <td>64.90</td>
    <td><b>80.59</b></td>
  </tr>
  <tr>
    <th colspan="5"><i>Adaptive Compression</i></th>
  </tr>
  <tr>
    <td>(D) w/o evaluator</td>
    <td>35.71</td>
    <td>-</td>
    <td>63.63</td>
    <td>-</td>
  </tr>
</table>

Table 3: Ablation study of ECoRAG, showing the impact of compressor and adaptive compression methods.

evaluators. It outperforms the CompAct evaluator (7B) (Yoon et al., 2024) by +13.96%p in F1 score. The CompAct evaluator is based on Mistral-7B (Jiang et al., 2023a) and trained with supervision from GPT-4o. As Asai et al. (2024) noted, the reader LLM evaluates whether documents support the correct answer, making it a strong baseline. We used Flan-UL2 (Tay et al., 2023) (20B) as our reader LLM, as described in Section B.3. Notably, our evidentiality evaluator, despite its much smaller size (770M), closely approximates the performance of Flan-UL2 (-0.08p%).

5.3 RQ3: Ablation Study

In Table 3, we present the results of our ablation study, assessing the impact of each component in our framework by comparing EM across different settings. We also report R20, checking if the gold answer is in the top 20 sentences.

For <i>Compressor</i>, we compare (A) ECoRAG with two inferior compressors, (B) and (C). In (B), the compressor uses a pretrained Contriever checkpoint without additional training, while in (C), it is trained with answerability labels. As shown, our compressor trained with evidentiality labels outperforms both alternatives. Comparing (A) and (C) shows that evidentiality labels increase EM (+1.02%p, +0.53%p) while maintaining R20 at a comparable level. Since R20 measures lexical overlap, (C), trained with answerability, performs similarly to or better than (A). The results demonstrate the superiority of our evidentiality labels over answerability labels, as they prioritize contextually rich information.

For <i>Evaluator</i>, we consider a no-evaluator setting (D), where the initial compression from the compressor is used without evaluating its evidentiality. The EM gap between (A) and (D) (+0.77%p, +1.80%p) highlights the impact of the evidentiality evaluator. These results highlight the importance of adaptively adjusting the amount of evidence through evidentiality evaluation.

<table>
  <tr>
    <th>Methods</th>
    <th>Compression Time</th>
    <th>Inference Time</th>
    <th>Total Time</th>
    <th>Throughput (example/sec)</th>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>-</td>
    <td>3.79h</td>
    <td>3.79h</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>standard RAG</td>
    <td>-</td>
    <td>12.28h</td>
    <td>12.28h</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>RECOMP</td>
    <td>0.27h</td>
    <td>4.08h</td>
    <td>4.35h</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>CompAct</td>
    <td>10.10h</td>
    <td>4.83h</td>
    <td>14.94h</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>0.73h</td>
    <td>4.23h</td>
    <td>4.96h</td>
    <td>0.20</td>
  </tr>
</table>

Table 4: Inference time and compression time for NQ test.

5.4 RQ4: Total Latency

ECoRAG is cost-efficient not only because it reduces the number of tokens but also because it decreases total latency in the RAG process. In RAG without compression, computational costs increase as more documents are retrieved. By applying compression and retaining only the necessary information, ECoRAG reduces total processing time.

Table 4 presents the total latency\(^3\), including both compression and inference time, to show the efficiency of our approach. For long context, the LLM-based abstractive compressor CompAct took longer than the ‘standard RAG’ setting, whereas the extractive compressors RECOMP and ECoRAG were faster. ECoRAG uses the lightweight evaluator that generates only a single token per iteration, stopping the reflection process once the compressed document is evidential or the token limit is reached, thereby preventing excessive compression time. While ECoRAG had similar speed to RECOMP, it achieved better performance by retaining only the information necessary to generate the correct answer, as described in Table 1. Thus, ECoRAG is effective in handling long contexts in terms of both performance and efficiency.

ECoRAG is a two-step design that achieves both speed and performance. Single-step aggregation with LLMs, as demonstrated by CompAct in Table 1, struggles with length dependency for listwise evaluation due to the “lost-in-the-middle” issue (Liu et al., 2024). In contrast, ECoRAG separates the process by first assessing sentences individually with an extractive compressor and then evaluating them collectively. This separation overcomes challenges in handling long contexts and improves compression effectiveness. Our lightweight components ensure efficiency while achieving effective compression.

\(^3\)Since GPT-4o-mini does not provide latency measurements, we conducted the latency experiments using Flan-UL2.
6 Conclusion

ECoRAG is a framework designed to compress long context by focusing on evidentiality in LLMs, defined as whether information supports generating the correct answer. Evidentiality-guided compression effectively filters out irrelevant content and retains necessary evidence. Through adaptive compression, ECoRAG determines the optimal compression length for each question, ensuring efficient use of context. As a result, ECoRAG demonstrates both superior performance and efficiency in handling long context, outperforming other compression methods.

7 Limitation

Evidentiality provides an effective indicator for determining whether information is necessary for an LLM to generate the correct answer. However, mining evidentiality labels is computationally expensive, leading to increased costs. Since multiple inferences are required for each question, it results in significant time consumption. Nevertheless, as more time is spent, more evidentiality labels can be obtained, which can contribute to the training of the compressor. Evidentiality labels can also be reused to train the evidentiality evaluator, optimizing resource usage. Once the compressor is fully trained and applied, the LLM inference process becomes faster.

Building upon this efficiency improvement, the application of this system can be extended beyond ODQA to address broader real-world scenarios. Extending it to tasks like summarization may be necessary due to context length limits when processing full content with LLMs. Selecting and summarizing only the most important parts can improve performance (Saxena and Keller, 2024; Jeong et al., 2025), requiring evidentiality to be redefined based on summarization metrics. Investigating such adaptations is a potential direction for future work.

Acknowledgements

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2024-00414981), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00077/RS-2022-II220077, AI Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous Data), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)].

References

Akari Asai, Matt Gardner, and Hannaneh Hajishirzi. 2022. Evidentiality-guided generation for knowledge-intensive nlp tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2226–2243.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.

Jianyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand. Association for Computational Linguistics.

Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. Preprint, arXiv:2210.11416.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Sang-eun Han, Yeonseok Jeong, Seung-won Hwang, and Kyungjae Lee. 2023. On monotonic aggregation for open-domain qa. In Proc. Interspeech 2023, pages 3432–3436.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.

Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880. Online. Association for Computational Linguistics.

Yeonseok Jeong, Minsoo Kim, Seung-won Hwang, and Byung-Hak Kim. 2025. Agent-as-judge for factual summarization of long narratives. arXiv preprint arXiv:2501.09993.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825.

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023b. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Computational Linguistics.

Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.

Jaehyun Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Improving open-domain question answering of LLMs via summarized retrieval. In The Twelfth International Conference on Learning Representations.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenneth Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Tiziano Labruna, Jon Ander Campos, and Gorka Azkune. 2024. When to retrieve: Teaching llms to utilize information retrieval effectively. Preprint, arXiv:2404.19705.

Kyungjae Lee, Seung-won Hwang, Sang-eun Han, and Dohyeon Lee. 2021. Robustifying multi-hop qa through pseudo-evidentiality training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6110–6119.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhui Chen. 2024. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060.

Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore. Association for Computational Linguistics.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Augmented large language models with parametric knowledge guiding. Preprint, arXiv:2305.04757.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.

Jesse Mu, Xiang Li, and Noah Goodman. 2024. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36.

Yilin Niu, Fangkai Jiao, Mantong Zhou, Ting Yao, Jingfang Xu, and Minlie Huang. 2020. A self-training method for machine reading comprehension with soft evidence extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3916–3927, Online. Association for Computational Linguistics.

OpenAI. 2023. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics ACL 2024, pages 963–981, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Cheng Qian, Xinran Zhao, and Tongshuang Wu. 2024. “merge conflicts!” exploring the impacts of external knowledge distractors to parametric knowledge graphs. In First Conference on Language Modeling.

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.

Rohit Saxena and Frank Keller. 2024. Select and summarize: Scene saliency for movie script summarization. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3439–3455.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärlí, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.

Yongho Song, Dahyun Lee, Myungha Jang, Seung-won Hwang, Kyungjae Lee, Dongha Lee, and Jinyoung Yeo. 2024. Evidentiality-aware retrieval for overcoming abstractiveness in open-domain question answering. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1930–1943.

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Husseinot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Harsh Trivedi, Niranjana Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.

Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormalabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. 2022. Supernaturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association.

Kevin Wu, Eric Wu, and James Zou. 2024. How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior. arXiv preprint arXiv:2404.10198.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenninghoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649.

Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024. Recomp: Improving retrieval-augmented lms with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations.

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. Preprint, arXiv:2401.15884.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.

Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. 2024. CompAct: Compressing retrieved documents actively for question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21424–21439, Miami, Florida, USA. Association for Computational Linguistics.

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh International Conference on Learning Representations.
Appendices

A  Further Analysis

A.1 Comparative Analysis of Compression Methods

In this section, we will provide a more detailed comparison of our approach with other baselines based on Table 1. Table 5 provides an overview of how each method differs. Based on this comparison, we discuss how large-scale documents can be compressed efficiently and effectively.

In ODQA, since the model must provide an answer to a given question, the compression process needs to consider the question. LLMLingua (Jiang et al., 2023b) and LLMLingua-2 (Pan et al., 2024), which do not consider the question during compression, often include irrelevant information, leading to suboptimal performance. On the other hand, the methods other than LLMLingua and LLMLingua-2 are question-aware, allowing them to more effectively capture the necessary content, resulting in higher performance compared to question-agnostic methods.

The amount of evidence needed varies for each question, and one solution to address this is adaptive compression, where the compression ratio is adjusted for each question. By applying this method, only the necessary tokens are used, leading to high performance with fewer tokens. As seen in Table 1, both CompAct (Yoon et al., 2024) and ECoRAG achieve high performance with a reduced number of tokens.

However, there are two main challenges when dealing with long context. First, while using numerous retrieval results increases the amount of necessary information available, it also includes documents with lower relevance scores, resulting in considerable noise. Second, the overall length of the documents is too long, which makes the compression process time-consuming.

To address the first challenge mentioned above, the concept of evidentiality is necessary. As discussed in Section 3.1.1, by prioritizing strong evidence for correct answer generation and penalizing distractors, we have been able to create a compressor that is robust against noise. Consequently, this approach allows ECoRAG to demonstrate the highest performance in large-scale document settings.

To address the second challenge, the compressor must be an extractive compressor that evaluates each content pointwise and extracts only the necessary information. Language model-based abstractive compressor is hindered by limited context length, which leads to truncation and fails to handle entire large-scale documents. Moreover, LLM-based abstractive compressor often requires substantial time for inference and may suffer from positional biases (Liu et al., 2024), which can lead to inaccurate assessments of evidentiality. However, extractive compressors such as ECoRAG and RECOMP (extractive) (Xu et al., 2024) are lightweight models that can quickly calculate scores, as seen in Table 4, and process each document in parallel for each document, thus avoiding positional biases.

Based on these observations, we conclude that ECoRAG, which combines all the characteristics from Table 5, is appropriate for compressing large-scale documents effectively.

A.2 Evaluator Performance on NQ

![Bar chart showing evidentiality evaluation metrics for CompAct, Flan-UL2, and Ours on NQ](page_768_1042_496_246.png)

Figure 5: Evidentiality evaluation metrics using different evaluator, including ours, measured on the NQ.

We conducted same experiments on NQ (Kwiatkowski et al., 2019), as described in Section 5.2, observed similar trends to those in TQA (Joshi et al., 2017). As shown in Figure 5, our evidentiality evaluator consistently outperforms CompAct and demonstrates comparable results to Flan-UL2, further validating its effectiveness across different datasets.

A.3 Compression Effectiveness with More Long Context

To explore performance of ECoRAG with more documents, we conducted additional experiments using 1000 retrieved documents in Table 6. Previous compression work, such as CompAct, focused on up to 30 documents, while our experiments used 100 documents, a common setting in RAG models like FiD (Izacard and Grave, 2021). To verify
<table>
  <tr>
    <th>Methods</th>
    <th>Question-aware</th>
    <th>Adaptive Compression</th>
    <th>Evidentiality-guided</th>
    <th>Extractive Compression</th>
  </tr>
  <tr>
    <td>LLMLingua, LLMLingua-2</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LongLLMLingua</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>RECOMP (extractive)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>RECOMP (abstractive)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CompAct</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 5: The table compares different methods based on their key characteristics. Our approach, ECoRAG, integrates all these features for fast and effective large-scale document compression.

<table>
  <tr>
    <th>Methods</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <th colspan="4">RAG without compression</th>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>0</td>
    <td>21.33</td>
    <td>28.71</td>
  </tr>
  <tr>
    <td>standard RAG (1000 documents)</td>
    <td>127,880</td>
    <td>0.44</td>
    <td>0.63</td>
  </tr>
  <tr>
    <th colspan="4">RAG with 1000 documents compressed</th>
  </tr>
  <tr>
    <td>RECOMP (extractive)</td>
    <td>661</td>
    <td>31.39</td>
    <td>42.29</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>659</td>
    <td><b>35.51</b></td>
    <td><b>48.63</b></td>
  </tr>
</table>

Table 6: Experimental results on the NQ test dataset using GPT-4o-mini, comparing performance with and without compression for 1000 retrieved documents (Karpukhin et al., 2020).

whether our method consistently improves performance even with more documents, we tested with 1000 documents. Due to limited budget, we used documents already retrieved by a DPR setting that was searched, differing from our top-100 DPR setting. We compared ECoRAG with RECOMP, an extractive method with a similar structure, and excluded abstractive compressors such as CompAct due to its too high latency in longer context compression.

With 1000 documents, ECoRAG remained highly effective in compressing and preserving essential information. The context length became too long for GPT-4o-mini to effectively utilize the information (Hsieh et al., 2024), as shown in Table 6. However, our compression effectively reduced the length, maintaining high performance. Additionally, ECoRAG outperformed other extractive compressors, demonstrating its superiority in handling extensive document sets.

ECoRAG remains the most effective compressor even for extremely long contexts. Without compression, excessive context length can degrade performance or exceed the context limit. In contrast, our retriever-based compressor efficiently compresses extended inputs regardless of length.

A.4 A Comparative Study with Reranker

ECoRAG fundamentally differs from reranking methods like BGE-M3 (Chen et al., 2024) and RECOMP by adaptively determining the rank and compression ratio needed for each query. While reranking models focus on relevance, they lack our ability to iteratively refine compression based on evidentiality. To ensure a fair comparison with our approach in terms of token usage, we conducted additional experiments with both BGE-M3\(^4\) and BGE-reranker\(^5\) (Xiao et al., 2024) by using its reranked top-10 and top-20 sentences. As shown in Table 7, ECoRAG achieves better performance, demonstrating the importance of selecting the appropriate context over simply increasing or reducing the amount of information.

Unlike other sentence reranking methods, EC-oRAG evaluates the initial compression and adaptively adjusts the compression ratio through a reflection process to determine how much information is required. This capability moves EC-oRAG closer to true compression rather than simple reranking. Furthermore, our research extends beyond proposing a compressor—it introduces a complete framework. While we used Contriever to ensure fair comparisons with RECOMP, our framework is flexible and capable of training models like BGE-M3 and BGE-reranker to learn LLM-based evidentiality, further enhancing performance.

A.5 Adaptive Compression Ratio Analysis

To validate the claim of our adaptive compression capabilities, we analyzed the distribution of compression ratios across datasets. The compression ratio is defined as the number of compressed tokens divided by the number of original tokens. Table 8 summarizes the minimum, maximum, mean, me-

\(^4\)BAAI/bge-m3
\(^5\)BAAI/bge-reranker-large
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">NQ</th>
    <th colspan="3">TQA</th>
    <th colspan="3">WQ</th>
  </tr>
  <tr>
    <th>#tokens</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>BGE-M3 (top 10)</td>
    <td>330</td>
    <td>33.02</td>
    <td>45.47</td>
    <td>370</td>
    <td>64.12</td>
    <td>74.34</td>
    <td>322</td>
    <td>20.77</td>
    <td>38.27</td>
  </tr>
  <tr>
    <td>BGE-M3 (top 20)</td>
    <td>670</td>
    <td>33.99</td>
    <td>46.82</td>
    <td>746</td>
    <td>65.15</td>
    <td>75.14</td>
    <td>645</td>
    <td>20.77</td>
    <td>38.00</td>
  </tr>
  <tr>
    <td>BGE-reranker (top 10)</td>
    <td>436</td>
    <td>34.16</td>
    <td>47.73</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>BGE-reranker (top 20)</td>
    <td>838</td>
    <td>34.82</td>
    <td>47.95</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>632</td>
    <td><b>36.48</b></td>
    <td><b>49.81</b></td>
    <td>441</td>
    <td><b>65.34</b></td>
    <td><b>75.37</b></td>
    <td>560</td>
    <td><b>30.17</b></td>
    <td><b>46.13</b></td>
  </tr>
</table>

Table 7: Performance on NQ, TQA and WQ using GPT-4o-mini, comparing dense retriever BGE-M3, BGE-reranker (results shown only for NQ), and our ECoRAG.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">Min Compression Ratio</th>
    <th colspan="2">Max Compression Ratio</th>
    <th colspan="2">Mean Compression Ratio</th>
    <th colspan="2">Median Compression Ratio</th>
    <th rowspan="2">Standard Deviation</th>
  </tr>
  <tr>
    <th>Compression Ratio</th>
    <th>Compression Ratio</th>
    <th>Compression Ratio</th>
    <th>Compression Ratio</th>
    <th>Compression Ratio</th>
    <th>Compression Ratio</th>
  </tr>
  <tr>
    <td>NQ</td>
    <td>0.0036</td>
    <td>1</td>
    <td>0.0401</td>
    <td>0.0446</td>
    <td>0.0247</td>
  </tr>
  <tr>
    <td>TQA</td>
    <td>0.0034</td>
    <td>1</td>
    <td>0.0267</td>
    <td>0.0161</td>
    <td>0.0221</td>
  </tr>
</table>

Table 8: Compression ratio statistics for NQ and TQA datasets.

<table>
  <tr>
    <th>Compressor</th>
    <th>Evaluator</th>
    <th>Reader</th>
  </tr>
  <tr>
    <td>VRAM usage</td>
    <td>110M</td>
    <td>770M</td>
    <td>\( \geq 8B \)</td>
  </tr>
  <tr>
    <td>Latency</td>
    <td>0.70h</td>
    <td>0.03h</td>
    <td>4.23h</td>
  </tr>
</table>

Table 9: VRAM usage and latency for each component in ECoRAG on the NQ test set.

<table>
  <tr>
    <th>Methods</th>
    <th>Compression Time</th>
    <th>Inference Time</th>
    <th>Total Time</th>
    <th>Throughput (example/sec)</th>
  </tr>
  <tr>
    <td>standard RAG</td>
    <td>-</td>
    <td>8.55h</td>
    <td>8.55h</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>0.51h</td>
    <td>2.94h</td>
    <td>3.45h</td>
    <td>0.20</td>
  </tr>
</table>

Table 10: Inference time and compression time for NQ test under worst case scenarios.

dian, and standard deviation of compression ratios for the NQ and TQA datasets.

The results highlight differences between datasets, with higher mean and median compression ratios observed for NQ. This reflects complexity of dataset, requiring the extraction of answers from lengthy Wikipedia documents through reasoning and comprehensive understanding. In contrast, TQA involves documents with explicitly positioned answers, making the task primarily about filtering irrelevant information. Consequently, ECoRAG retrieves more evidence for NQ to address its higher information needs, demonstrating its ability to adjust compression ratios adaptively based on dataset complexity and information requirements.

A.6 Further Analysis on Efficiency

ECoRAG has demonstrated efficiency over traditional RAG, as shown in Table 1 and 4, but further analysis is required to verify its resource and latency efficiency. To compare resource usage, we refer to Table 9. While traditional RAG requires at least 8B VRAM in our experiments, ECoRAG only adds additional 880M VRAM. Furthermore, since the compressor and evaluator can operate sequentially as well as simultaneously with the reader, ECoRAG remains feasible in traditional RAG environments.

In terms of latency, Table 4 shows that ECoRAG is more efficient than traditional RAG, but additional verification is needed across different cases. The additional modules—compressor and evaluator—may seem to increase system complexity. However, traditional RAG must process the entire long context, while ECoRAG reduces latency by 7.32h, as shown in Table 4. Table 9 shows that ECoRAG requires little time for compression, reducing the risk of bottleneck as the preceding modules process efficiently. In the worst case, ECoRAG evaluates compression multiple times, leading to longer latency than in the best case. However, even in the worst case, Table 10 demonstrates that ECoRAG is still faster than traditional RAG.

A.7 Case study of evidentiality-guided compression

Table 11 illustrates an example of evidentiality-guided compression. For the given question, who dies at the end of Den of Thieves? with the correct answer Merrimen, the initial document set before compression includes the correct answer. But it also contains irrelevant information, which misleads the LLM into generating the wrong answer, Donnie. After compression, irrelevant content containing the word Donnie is effectively suppressed, leaving only the evidential (highlighted) sentences.
<table>
  <tr>
    <th>Question</th>
    <th></th>
    <th>Gold answers</th>
  </tr>
  <tr>
    <td colspan="2">who dies at the end of den of thieves</td>
    <td>Merrimen</td>
  </tr>
  <tr>
    <th>Type</th>
    <th>In-context documents</th>
    <th>Prediction</th>
  </tr>
  <tr>
    <td>None</td>
    <td></td>
    <td>Donnie</td>
  </tr>
  <tr>
    <td>retrieved documents</td>
    <td>Den of Thieves (film) Nick, forcing Nick to shoot him. As Merrimen lies on the ground dying, Nick kneels and consoles him. When Nick inspects Merrimen’s SUV, he only finds bags with shredded paper; he also finds that Donnie has escaped custody. Nick later goes to Donnie’s bar and sees pictures of him with some of the crew members from the heist. It is revealed Donnie masterminded the heist to keep all of the stolen cash for himself in a second garbage truck. After the passage of some time. Donnie is working in a London bar, planning a new heist. The film was in Den of Thieves (film) is currently in development. In Los Angeles, a team of robbers led by Ray Merrimen make a violent armed attack and hijack an armored truck. Police officers arrive on the scene and engage in a shootout with the robbers. Eventually, Merrimen and his crew escape with the empty armored truck. In the morning, Detective Nick O’Brien investigates the crime scene, having been monitoring Merrimen and his crew for a while. Suspecting a local bartender named Donnie for involvement, Nick finds him at the bar and kidnaps him for interrogation. Donnie reveals Merrimen is planning to rob the Federal Reserve on Den of Thieves (film) garbage truck that removes shredded bills. Nick’s team catches up to Donnie and seizes him, beating him until he tells them where Merrimen is going. Merrimen, Bosco, and Levi try to make their escape with the money bags from the waste truck but hit a traffic jam and are blocked. Nick’s team spots them and attempt to shoot them as the robbers try to escape. A shootout occurs initiated by Merrimen, killing one of Nick’s men. Levi and Bosco are eventually shot dead, but Merrimen gets away. Nick chases and shoots Merrimen, wounding him. Merrimen raises an empty gun to Den of Thieves (film) is currently in development. In Los Angeles, a team of robbers led by Ray Merrimen make a violent armed attack and hijack an armored truck. Police officers arrive on the scene and engage in a shootout with the robbers. Eventually, Merrimen and his crew escape with the empty armored truck. In the morning, Detective Nick O’Brien investigates the crime scene, having been monitoring Merrimen and his crew for a while. Suspecting a local bartender named Donnie for involvement, Nick finds him at the bar and kidnaps him for interrogation. Donnie reveals Merrimen is planning to rob the Federal Reserve on Den of Thieves (film) Friday of that week by covertly removing about $30 million in old bills which are scheduled to be shredded after their serial numbers are deleted from computer records. At their hideout, Merrimen has one of his crew, Levi, roughly interrogate Donnie to ensure he didn’t disclose anything about the plan. Meanwhile, Nick goes to a strip club and finds Merrimen’s stripper girlfriend, hiring her for the night to find out where the heist is going to happen. The next morning, Nick makes an effort to see his daughter at her school. As the day of the heist comes, Merrimen and</td>
    <td>Donnie</td>
  </tr>
  <tr>
    <td>Compression</td>
    <td>Den of Thieves (film) As Merrimen lies on the ground dying, Nick kneels and consoles him. Den of Thieves (film) Eventually, Merrimen and his crew escape with the empty armored truck. Den of Thieves (film) Merrimen, Bosco, and Levi try to make their escape with the money bags from the waste truck but hit a traffic jam and are blocked. Den of Thieves (film) In the morning, Detective Nick O’Brien investigates the crime scene, having been monitoring Merrimen and his crew for a while. Den of Thieves (film) Meanwhile, Nick goes to a strip club and finds Merrimen’s stripper girlfriend, hiring her for the night to find out where the heist is going to happen.</td>
    <td>Merrimen</td>
  </tr>
</table>

Table 11: Case study of how the compression of the retrieved documents helps the model to identify the correct answer from NQ test set. The highlighted part is the evidential sentence that directly gives useful information for generating the correct answer Merrimen, rather than the incorrect answer Donnie.

A.8 Generalizability across Readers

To evaluate the generalizability of our compression framework, we conducted experiments using Flan-UL2 (Tay et al., 2023) (20B), Llama3 (Dubey et al., 2024) (8B), and Gemma2 (Team et al., 2024) (9B) as the reader LLMs. These models were chosen to investigate how our method performs across diverse architectures and parameter sizes.

Flan-UL2 was selected because RECOMP also utilizes it, as we intend to directly compare with it. Furthermore, additional experiments were conducted with Llama3 and Gemma2 to extend the evaluation. Since Llama3 has large context length, it can conduct ‘standard RAG’ experiment, unlike Flan-UL2 and Gemma2.

Results show that our evidentiality-guided compression method consistently outperforms other compression baselines on all three models. Specifically, with Flan-UL2 in Table 12, which was used to define evidentiality during training, the model demonstrated a clear improvement across all metrics. Similarly, as shown in Table 13. Gemma2, despite being trained without its own evidentiality mining, also showed improved performance with
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">NQ</th>
    <th colspan="3">TQA</th>
    <th colspan="3">WQ</th>
  </tr>
  <tr>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <td colspan="10"><i>RAG without compression</i></td>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>0</td>
    <td>21.33</td>
    <td>28.71</td>
    <td>0</td>
    <td>46.48</td>
    <td>52.47</td>
    <td>0</td>
    <td>32.97</td>
    <td>42.33</td>
  </tr>
  <tr>
    <td>standard RAG (100 documents)</td>
    <td>15456</td>
    <td>-</td>
    <td>-</td>
    <td>15943</td>
    <td>-</td>
    <td>-</td>
    <td>15135</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="10"><i>RAG with 100 documents compressed</i></td>
  </tr>
  <tr>
    <td>LLMLingua</td>
    <td>725</td>
    <td>19.17</td>
    <td>25.48</td>
    <td>726</td>
    <td>42.97</td>
    <td>48.93</td>
    <td>868</td>
    <td>31.10</td>
    <td>40.87</td>
  </tr>
  <tr>
    <td>LLMLingua-2</td>
    <td>1475</td>
    <td>24.63</td>
    <td>32.19</td>
    <td>1518</td>
    <td>53.07</td>
    <td>59.42</td>
    <td>1580</td>
    <td>30.61</td>
    <td>41.76</td>
  </tr>
  <tr>
    <td>LongLLMLingua</td>
    <td>1516</td>
    <td>38.03</td>
    <td>46.94</td>
    <td>1570</td>
    <td>65.79</td>
    <td>73.88</td>
    <td>1629</td>
    <td>32.78</td>
    <td>45.27</td>
  </tr>
  <tr>
    <td>RECOMP (extractive)</td>
    <td>727</td>
    <td>38.06</td>
    <td>46.18</td>
    <td>750</td>
    <td>62.49</td>
    <td>69.68</td>
    <td>857</td>
    <td>31.25</td>
    <td>43.18</td>
  </tr>
  <tr>
    <td>RECOMP (abstractive)</td>
    <td><b>16</b></td>
    <td>22.22</td>
    <td>29.56</td>
    <td><b>30</b></td>
    <td>43.50</td>
    <td>49.88</td>
    <td><b>157</b></td>
    <td>38.15</td>
    <td>38.56</td>
  </tr>
  <tr>
    <td>CompAct</td>
    <td>252</td>
    <td>42.16</td>
    <td>51.05</td>
    <td>253</td>
    <td>64.37</td>
    <td>72.25</td>
    <td>218</td>
    <td>33.07</td>
    <td>44.45</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>693</td>
    <td><b>44.38</b></td>
    <td><b>53.56</b></td>
    <td>501</td>
    <td><b>66.45</b></td>
    <td><b>74.02</b></td>
    <td>671</td>
    <td><b>33.71</b></td>
    <td><b>46.08</b></td>
  </tr>
</table>

Table 12: Comparison of compression methods on NQ, TQA, and WQ using Flan-UL2 (Tay et al., 2023) with 100 retrieved documents (Karpukhin et al., 2020).

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">NQ</th>
    <th colspan="3">TQA</th>
    <th colspan="3">WQ</th>
  </tr>
  <tr>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <td colspan="10"><i>RAG without compression</i></td>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>0</td>
    <td>27.84</td>
    <td>38.35</td>
    <td>0</td>
    <td>57.11</td>
    <td>66.39</td>
    <td>0</td>
    <td>26.77</td>
    <td>43.24</td>
  </tr>
  <tr>
    <td>standard RAG (100 documents)</td>
    <td>14260</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>14075</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="10"><i>RAG with 100 documents compressed</i></td>
  </tr>
  <tr>
    <td>LLMLingua</td>
    <td>643</td>
    <td>26.90</td>
    <td>37.90</td>
    <td>638</td>
    <td>60.71</td>
    <td>68.09</td>
    <td>649</td>
    <td>25.04</td>
    <td>42.08</td>
  </tr>
  <tr>
    <td>LLMLingua-2</td>
    <td>1403</td>
    <td>28.56</td>
    <td>38.95</td>
    <td>1393</td>
    <td>59.95</td>
    <td>67.84</td>
    <td>1401</td>
    <td>24.36</td>
    <td>40.52</td>
  </tr>
  <tr>
    <td>LongLLMLingua</td>
    <td>1411</td>
    <td>37.67</td>
    <td>49.40</td>
    <td>1436</td>
    <td>63.17</td>
    <td>70.28</td>
    <td>1399</td>
    <td>27.02</td>
    <td>44.23</td>
  </tr>
  <tr>
    <td>RECOMP (extractive)</td>
    <td>165</td>
    <td>37.65</td>
    <td>48.24</td>
    <td>687</td>
    <td>63.19</td>
    <td>70.38</td>
    <td>680</td>
    <td>26.03</td>
    <td>42.22</td>
  </tr>
  <tr>
    <td>RECOMP (abstractive)</td>
    <td><b>17</b></td>
    <td>27.98</td>
    <td>38.00</td>
    <td><b>28</b></td>
    <td>58.78</td>
    <td>65.74</td>
    <td><b>21</b></td>
    <td>25.20</td>
    <td>41.60</td>
  </tr>
  <tr>
    <td>CompAct</td>
    <td>111</td>
    <td>38.67</td>
    <td>49.87</td>
    <td>100</td>
    <td>65.88</td>
    <td>73.29</td>
    <td>78</td>
    <td>26.67</td>
    <td>43.04</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>684</td>
    <td><b>39.20</b></td>
    <td><b>50.24</b></td>
    <td>448</td>
    <td><b>66.32</b></td>
    <td><b>74.25</b></td>
    <td>504</td>
    <td><b>27.41</b></td>
    <td><b>44.00</b></td>
  </tr>
</table>

Table 13: Comparison of compression methods on NQ, TQA, and WQ using Gemma2 (Team et al., 2024) with 100 retrieved documents (Karpukhin et al., 2020).

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">NQ</th>
    <th colspan="3">TQA</th>
    <th colspan="3">WQ</th>
  </tr>
  <tr>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <td colspan="10"><i>RAG without compression</i></td>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>0</td>
    <td>22.16</td>
    <td>32.36</td>
    <td>0</td>
    <td><b>60.89</b></td>
    <td>67.80</td>
    <td>0</td>
    <td><b>21.79</b></td>
    <td><b>35.81</b></td>
  </tr>
  <tr>
    <td>standard RAG (100 documents)</td>
    <td>14263</td>
    <td>0.27</td>
    <td>0.97</td>
    <td>14574</td>
    <td>0.24</td>
    <td>2.70</td>
    <td>14147</td>
    <td>0.25</td>
    <td>4.48</td>
  </tr>
  <tr>
    <td colspan="10"><i>RAG with 100 documents compressed</i></td>
  </tr>
  <tr>
    <td>LLMLingua</td>
    <td>641</td>
    <td>15.20</td>
    <td>22.31</td>
    <td>636</td>
    <td>52.11</td>
    <td>59.23</td>
    <td>646</td>
    <td>17.62</td>
    <td>30.92</td>
  </tr>
  <tr>
    <td>LLMLingua-2</td>
    <td>1346</td>
    <td>3.91</td>
    <td>7.19</td>
    <td>1366</td>
    <td>48.08</td>
    <td>55.91</td>
    <td>1337</td>
    <td>4.28</td>
    <td>11.44</td>
  </tr>
  <tr>
    <td>LongLLMLingua</td>
    <td>1388</td>
    <td>20.30</td>
    <td>28.85</td>
    <td>1423</td>
    <td>58.34</td>
    <td>68.49</td>
    <td>1372</td>
    <td>18.70</td>
    <td>32.12</td>
  </tr>
  <tr>
    <td>RECOMP (extractive)</td>
    <td>160</td>
    <td>22.33</td>
    <td>31.12</td>
    <td>683</td>
    <td>36.69</td>
    <td>44.08</td>
    <td>667</td>
    <td>16.19</td>
    <td>27.80</td>
  </tr>
  <tr>
    <td>RECOMP (abstractive)</td>
    <td><b>16</b></td>
    <td>18.75</td>
    <td>27.85</td>
    <td><b>27</b></td>
    <td>42.73</td>
    <td>50.94</td>
    <td><b>21</b></td>
    <td>18.80</td>
    <td>33.25</td>
  </tr>
  <tr>
    <td>CompAct</td>
    <td>107</td>
    <td>28.01</td>
    <td>38.52</td>
    <td>99</td>
    <td>56.01</td>
    <td>64.69</td>
    <td>76</td>
    <td>21.41</td>
    <td>35.21</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>519</td>
    <td><b>30.22</b></td>
    <td><b>42.55</b></td>
    <td>445</td>
    <td><b>59.25</b></td>
    <td><b>69.32</b></td>
    <td>588</td>
    <td><b>21.60</b></td>
    <td><b>35.43</b></td>
  </tr>
</table>

Table 14: Comparison of compression methods on NQ, TQA, and WQ using Llama3 (Dubey et al., 2024) with 100 retrieved documents (Karpukhin et al., 2020).
<table>
  <tr>
    <th>Methods</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <th colspan="4">RAG without compression</th>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>0</td>
    <td>31.88</td>
    <td>44.10</td>
  </tr>
  <tr>
    <td>standard RAG (100 documents)</td>
    <td>13,847</td>
    <td>37.11</td>
    <td>50.82</td>
  </tr>
  <tr>
    <th colspan="4">RAG with 100 documents compressed</th>
  </tr>
  <tr>
    <td>LLMLingua</td>
    <td>645</td>
    <td>25.79</td>
    <td>37.56</td>
  </tr>
  <tr>
    <td>LLMLingua-2</td>
    <td>1,319</td>
    <td>29.95</td>
    <td>44.40</td>
  </tr>
  <tr>
    <td>LongLLMLingua</td>
    <td>1,364</td>
    <td>33.44</td>
    <td>46.20</td>
  </tr>
  <tr>
    <td>RECOMP (extractive)</td>
    <td>659</td>
    <td>33.21</td>
    <td>45.98</td>
  </tr>
  <tr>
    <td>RECOMP (abstractive)</td>
    <td>16</td>
    <td>30.45</td>
    <td>43.01</td>
  </tr>
  <tr>
    <td>CompAct</td>
    <td>75</td>
    <td>37.69</td>
    <td>51.65</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>641</td>
    <td><b>41.43</b></td>
    <td><b>54.02</b></td>
  </tr>
</table>

Table 15: Experimental results on the NQ dataset using GPT-4o-mini, comparing performance with and without compression for documents retrieved by Contriever.

our compression method, further validating its effectiveness.

In the case of Llama3, as presented in Table 14, our compression approach outperformed other baselines, including naive prepend. However, in certain instances, it was outperformed by the ‘closed book’ approach. This suggests that parametric knowledge embedded within the reader LLM can occasionally align well with specific datasets, leading to variations in performance across models.

Nonetheless, our framework ECoRAG is model-agnostic, as we have excluded the influence of the parametric knowledge of the reader LLM in mining evidentiality labels. These results emphasize that our compression method consistently outperforms other compression approaches, further validating its effectiveness across diverse models and configurations.

A.9 Generalizability across Retrievers

To verify that our compression approach generalizes beyond DPR, we conducted additional experiments using another retriever. Our initial choice of DPR was intentional, in order to demonstrate the robustness of our compression approach even under challenging conditions where a weaker retriever could introduce significant noise. In Table 15, we then evaluated our method with Contriever (Izacard et al., 2022), a stronger dense retriever. The results show an even larger performance gain when paired with Contriever than with DPR, indicating that ECoRAG synergizes especially well with higher-quality retrieval.

To compare against a simple baseline of retrieving fewer documents, we evaluated ECoRAG against varying retrieval sizes. We initially chose 100 documents to align with standard practice in prior work, such as Fusion-in-Decoder (Izacard and Grave, 2021). In Table 16, we then conducted experiments on varying numbers of retrieved documents (\( k = 5, 10, \) and \( 20 \)) for both DPR and Contriever, measuring token count and EM without compression. Our results show that reducing \( k \) does not improve accuracy as effectively as adaptive compression via ECoRAG, emphasizing the benefit of our evidentiality-guided approach.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="2">#tokens</th>
    <th colspan="2">EM</th>
  </tr>
  <tr>
    <th>(DPR)</th>
    <th>(Contriever)</th>
    <th>(DPR)</th>
    <th>(Contriever)</th>
  </tr>
  <tr>
    <th colspan="5">Reduced retrieval size</th>
  </tr>
  <tr>
    <td># docs (\( k = 5 \))</td>
    <td>693</td>
    <td>35.53</td>
    <td>690</td>
    <td>33.48</td>
  </tr>
  <tr>
    <td># docs (\( k = 10 \))</td>
    <td>1,386</td>
    <td>35.95</td>
    <td>1,381</td>
    <td>34.76</td>
  </tr>
  <tr>
    <td># docs (\( k = 20 \))</td>
    <td>2,774</td>
    <td>36.33</td>
    <td>2,762</td>
    <td>36.47</td>
  </tr>
  <tr>
    <th colspan="5">Adaptive compression</th>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>632</td>
    <td><b>36.48</b></td>
    <td>641</td>
    <td><b>41.43</b></td>
  </tr>
</table>

Table 16: Experimental results on the NQ dataset using GPT-4o-mini, comparing reduced retrieval sizes for DPR and Contriever against adaptive compression via ECoRAG.

<table>
  <tr>
    <th>Methods</th>
    <th>#tokens ↓</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <th colspan="4">RAG without compression</th>
  </tr>
  <tr>
    <td>closed-book</td>
    <td>0</td>
    <td>26.19</td>
    <td>36.71</td>
  </tr>
  <tr>
    <td>standard RAG (100 documents)</td>
    <td>14,313</td>
    <td>34.52</td>
    <td>44.69</td>
  </tr>
  <tr>
    <th colspan="4">RAG with 100 documents compressed</th>
  </tr>
  <tr>
    <td>LLMLingua</td>
    <td>636</td>
    <td>22.57</td>
    <td>31.54</td>
  </tr>
  <tr>
    <td>LLMLingua-2</td>
    <td>1,330</td>
    <td>26.66</td>
    <td>37.00</td>
  </tr>
  <tr>
    <td>LongLLMLingua</td>
    <td>1,406</td>
    <td>27.45</td>
    <td>38.07</td>
  </tr>
  <tr>
    <td>RECOMP (extractive)</td>
    <td>688</td>
    <td>28.05</td>
    <td>38.87</td>
  </tr>
  <tr>
    <td>RECOMP (abstractive)</td>
    <td>12</td>
    <td>24.27</td>
    <td>33.88</td>
  </tr>
  <tr>
    <td>CompAct</td>
    <td>74</td>
    <td>31.21</td>
    <td>42.42</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td>647</td>
    <td><b>34.69</b></td>
    <td><b>45.13</b></td>
  </tr>
</table>

Table 17: Experimental results on the HotpotQA dataset using GPT-4o-mini, comparing performance with and without compression for 100 documents (Karpukhin et al., 2020).

A.10 Evaluation in Multi-hop QA

To assess the effectiveness of ECoRAG in multi-hop QA tasks requiring multiple evidence sources, we conducted experiments in Table 17. ECoRAG classifies evidentiality into three categories and defines weak evidence that supports the correct answer without directly generating the answer. This enables ECoRAG to perform effectively in tasks requiring partial evidence, such as multi-hop QA. Furthermore, according to CompAct, adaptively adjusting evidence can collect the partial evidence needed for multi-hop QA, ECoRAG achieves through Evidentiality Reflection.

Table 17 shows that ECoRAG outperformed both non-compressed and other compression baselines in HotpotQA (Yang et al., 2018). CompAct and other baselines did not outperform the “standard RAG” approach, which uses all 100 docu-
<table>
  <tr>
    <th>Method</th>
    <th>HotpotQA</th>
    <th>MusiQue</th>
  </tr>
  <tr>
    <td>standard RAG</td>
    <td>49.76</td>
    <td>23.91</td>
  </tr>
  <tr>
    <td>RECOMP</td>
    <td>49.12</td>
    <td>23.08</td>
  </tr>
  <tr>
    <td>ECoRAG (ours)</td>
    <td><b>52.29</b></td>
    <td><b>24.60</b></td>
  </tr>
</table>

Table 18: Experiments on LongBench multi-hop datasets (HotpotQA and MusiQue) evaluating F1-score performance of standard RAG, RECOMP, and ECoRAG using Llama3-8B.

<table>
  <tr>
    <th>Subset</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1-score</th>
  </tr>
  <tr>
    <td>exposed subset</td>
    <td>68.52</td>
    <td>81.47</td>
    <td>64.17</td>
    <td>72.13</td>
  </tr>
  <tr>
    <td>non-exposed subset</td>
    <td>71.77</td>
    <td>78.64</td>
    <td>76.22</td>
    <td>77.42</td>
  </tr>
</table>

Table 19: Evaluation results of the Flan-T5-based evidentiality evaluator on TQA, comparing performance between the exposed and non-exposed subsets.

ments without compression. In contrast, ECoRAG improved performance by removing distractors and keeping necessary evidence. These results show that ECoRAG is effective for complex scenarios such as multi-hop QA.

To evaluate ECoRAG in scenarios where the challenge lies not only in the number but also in the length of retrieved documents, we applied our method to the LongBench (Bai et al., 2024) benchmark. LongBench is a long-context understanding benchmark covering tasks such as HotpotQA\footnote{LongBench does not include the full original datasets, so our result in HotpotQA results may differ from those reported in Table 17.} and MuSiQue (Trivedi et al., 2022). In Table 18, we compared standard RAG, RECOMP, and ECORAG (using Llama3-8B) across these tasks within LongBench. Consistent with our multi-hop results, ECoRAG outperformed both non-compressed and compression baselines in this long-document setting, further demonstrating its robustness and effectiveness.

B Experimental Details

B.1 Implementation Details

We used 8 Nvidia RTX3090 GPUs to train all models. For mining evidentiality labels for all sentences in retrieved documents, we used the NLTK library\footnote{www.nltk.org} to split DPR (Karpukhin et al., 2020) retrieved top-100 documents into sentences. To reduce costs, we used the open LLM Flan-UL2\footnote{google/flan-ul2} (Tay et al., 2023), which was also used in our experiments and RECOMP (Xu et al., 2024), to label evidentiality based on the definition in Section 3.1.1.

Our evidentiality compressor was trained from Contriever (Izacard et al., 2022) checkpoint pre-trained on CC-net (Wenzek et al., 2020) and English Wikipedia (Izacard et al., 2022).. We trained it using the AdamW optimizer with a batch size of 64 and a learning rate of \(5 \cdot 10^{-5}\) for 4 epochs on NQ (Kwiatkowski et al., 2019) and WQ (Berant et al., 2013), and 2 epochs on TQA (Joshi et al., 2017). While training with \(L_{we}\) and \(L_{se}\) losses, we used 8 positive contexts and 56 negative contexts per batch. When calculating the \(L_{se}\) loss, we used negative set with weak evidence to distractor ratio of 0.15:0.85, treating weak evidence as hard negative. We set the temperature \(\tau\) for the contrastive loss to 1.0.

Our evidentiality evaluator was trained from a pretrained Flan-T5-large checkpoint\footnote{google/flan-t5-large} using the AdamW optimizer. We trained it with a batch size of 40 and a learning rate of \(1 \cdot 10^{-5}\) for 4 epochs with all datasets. We included '<NOT>' sentences with high compressor scores in the training stage to make the evidentiality evaluator distinguish only the genuinely strong evidence '<EVI>' from the seemingly plausible ones. We constructed the training data for the evaluator with a ratio of 1:3 between '<EVI>' and '<NOT>' sentences. For adaptive compression, a limit on the number of evidence pieces was necessary to avoid infinite loops, which we set at 20. We set this limit to 20 to achieve a compression level similar to RECOMP, but it can be increased for tasks that require more evidence. Additionally, to prevent high latency due to overly frequent evaluations, we incrementally added 4 evidence pieces at a time. For experiments on the test set, we used GPT-4o-mini\footnote{gpt-4o-mini-2024-07-18}, Flan-UL2, Gemma2\footnote{google/gemma-2-9b-it}, and Llama3\footnote{meta-llama/Meta-Llama-3-8B-Instruct}.

B.2 Selection for Evidentiality Evaluator

We chose Flan-T5-large as the basis for our Evidentiality Evaluator due to its strong instruction-finetuning and robust performance on classification tasks. T5-large has been widely used in prior RAG research (Han et al., 2023; Yan et al., 2024) for document-based evaluation. For our evidentiality scoring task, we employ Flan-T5-large as it demonstrates enhanced instruction-following capa-
bilities that are well-suited for this classification task. However, a potential concern arises regarding Flan-T5-large’s prior exposure to datasets such as NQ, TQA, and WQ, which might lead to memorization rather than genuine evidentiality learning. As reported in research related to Flan-T5-large (Chung et al., 2022; Wang et al., 2022), its exposure was clearly separated into exposed and non-exposed subsets, and our comparison experiments (Section 5.2) demonstrate negligible performance differences between these groups. When we applied a two-proportion Z test to the results in Table 19, the analysis at the 0.05 significance level confirmed that the observed differences are not statistically significant. Therefore, these observation indicates that the model has learned evidentiality principles rather than simply memorizing evidence from prior exposure.

B.3 Input Prompts for LLM

We report two examples of input prompts for reader LLMs. In Figure 6, we report the input prompt used for evidentiality mining and test set experiments to answer a given question when provided with the question and the compressed documents. This prompt was also utilized during the evidentiality mining process, as described in Section 3.1.1. Figure 7 presents the input prompt for mining the ground truth label of compressed documents using Flan-UL2 as the evidentiality evaluator in the experiments detailed in Section 5.2.

C Usage of AI Assistants

We utilized ChatGPT to improve the clarity and grammatical accuracy of my writing. It provided suggestions for rephrasing sentences and correcting grammatical errors to make the text flow more naturally.
Question Answering Prompt

who won a million on deal or no deal
Answer: Tomorrow Rodriguez

who is the woman washing the car in cool hand luke
Answer: Joy Harmon

who is the actor that plays ragnar on vikings
Answer: Travis Fimmel

who said it’s better to have loved and lost
Answer: Alfred , Lord Tennyson

name the first indian woman to be crowned as miss world
Answer: Reita Faria

Documents
Question
Answer:
Evidentiality Evaluation Prompt

You are an expert at determining whether a document provides evidential support for a given question. You will receive a question and a document, and your task is to evaluate whether the document is evidential, partially evidential, or non-evidential in relation to the question.
Assess the support provided by the document using the following scale:
- [Evidential] - The document fully supports the question, providing clear and direct evidence that answers or addresses the query completely.
- [Non-Evidential] - The document does not provide relevant information or evidence related to the question, making it unrelated or insufficient to support the query.
Please provide your assessment and briefly justify your reasoning based on the content of the document in relation to the question.

Question: what is the temperature of dry ice in kelvin?
Evidence: At atmospheric pressure, sublimation/deposition occurs at or 194.65 K. The density of dry ice varies, but usually ranges between about.
Score: [Evidential]

Question: when did north vietnam unify with the south?
Evidence: The distinctive synthesizer theme was performed by the then-little-known Thomas Dolby, and this song also marked a major departure from their earlier singles because their previous singles were mid to upper tempo rock songs while this song was a softer love song with the energy of a power ballad.
Score: [Non-Evidential]

Question: who played all the carly ’s on general hospital?
Evidence: Throughout the 2000s, Carly, then Tamara Braun (2001–05) goes on to become one of the
Score: [Non-Evidential]

Question: who sang the original blinded by the light?
Evidence: Light of Day (song) "Light of Day", sometimes written as "(Just Around the Corner to the) Light of Day", is a song written by Bruce Springsteen and performed initially by Joan Jett and Michael J.
Score: [Non-Evidential]

Question: who was the rfc editor until 1998 just provide the family name?
Evidence: Perhaps his most famous legacy is from RFC 760, which includes a robustness principle often called "Postel’s law": "an implementation
Score: [Non-Evidential]

Question: Question
Evidence: Compressed Documents
Score:

Reviewer paper summaries: The authors propose ECoRAG (Evidentiality-guided RAG), a framework that enhances LLM-based ODQA by compressing retrieved documents based on evidential relevance, ensuring only supporting evidence is retained. Additionally, ECoRAG dynamically retrieves more content when necessary to guarantee sufficient evidence for answer generation. Experiments demonstrate that ECoRAG outperforms prior compression methods, improving accuracy while reducing latency and token costs under limited settings.

This paper proposes an adaptive compression method for long-text retrieval-augmented generation (RAG). The approach involves training an extractive compressor to rank sentences from retrieved documents and then utilizing an evidentiality evaluator to adaptively select evidence from the top-ranked sentences until sufficient information is gathered to answer the question. Experimental results demonstrate that the proposed method outperforms recently published models, including CompAct and ECoRAG.
Summary of strengths: - The proposed method demonstrates strong effectiveness, achieving the highest EM or F1 scores among compression methods and, in some cases, outperforming retrieval without compression. Additionally, it maintains competitive efficiency under a single-batch setting compared to other baselines.
- The motivation behind the method—focusing on evidentiality—is well-justified and offers an interesting approach to improving retrieval-augmented generation (RAG).
- The paper provides extensive experiments and analyses, leveraging a diverse set of reader models and baseline methods to validate the effectiveness of the approach.

1. The paper is well-structured and easy to follow, with the motivation clearly and effectively presented.
2. Comprehensive experiments were conducted to thoroughly demonstrate the effectiveness of the proposed method.
Summary of weaknesses: - The definition of evidentiality used in the proposed method, ECoRAG, is somewhat simplistic, which may limit its applicability to more complex, real-world scenarios involving diverse document structures.
    - The authors define a sentence as strongly evidential if an LLM can generate the correct answer when the sentence is present but fails to do so in its absence. However, this binary definition does not account for multi-hop reasoning, where multiple documents may each contain partial but essential evidence. As a result, the method may mistakenly classify important supporting documents as weak evidence or distractors.
    - Additionally, the paper lacks end-to-end experiments for multi-hop QA, making it difficult to assess the method’s effectiveness for queries requiring multiple evidence sources.
- The proposed method may be difficult to integrate into real-world LLM serving systems due to its longer sequential pipeline, which requires two additional models—a compressor and an evaluator—beyond a standard RAG pipeline.
    - Increased resource demands: Using multiple models increases VRAM usage, which could pose challenges for deployment on smaller GPU clusters.
    - Potential inference bottlenecks: A longer pipeline can lead to latency issues under high request rates, as overall throughput is constrained by the slowest model in the sequence. The evaluator model, in particular, is compute-intensive, potentially inducing an inference bottleneck under high request rates.
    - Unclear trade-offs in real-world efficiency: While the approach introduces a more structured retrieval process, it remains unclear whether it significantly reduces latency or improves throughput enough to justify the added complexity. Further analysis of its practical deployment feasibility would strengthen the argument for its adoption.

1. Figure 2 is not sufficiently clear in presenting the proposed method. The workflow structure needs improvement, or additional explanations should be provided in the caption to enhance clarity.
2. Section 3.1.2 lacks details on how the labels are assigned, making the labeling process unclear. Furthermore, essential training details are missing, such as the number of d*, d+, and d- samples used for each positive query.
Comments, suggestions and typos: See Weaknesses

Line 590, "lost-in-the-middle" should be ``lost-in-the-middle'' in latex
Overall assessment scores: 2.5

3