Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps
Authors: Jie Ou, Jinyu Guo, Shuaihong Jiang, Zhaokun Wang, Libo Qin, Shunyu Yao, Wenhong Tian
Accepted at: acl25

Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps

Jie Ou1, Jinyu Guo1*, Shuaihong Jiang1, Zhaokun Wang1, Libo Qin2, Shunyu Yao3, Wenhong Tian1
1 School of Information and Software Engineering,
University of Electronic Science and Technology of China
2 Central South University
3 Big data and artificial intelligent institute, China Telecom Research Institute

Abstract

Retrieval-augmented generation (RAG) has emerged as a pivotal method for expanding the knowledge of large language models. To handle complex queries more effectively, researchers developed Adaptive-RAG (A-RAG) to enhance the generated quality through multiple interactions with external knowledge bases. Despite its effectiveness, A-RAG exacerbates the pre-existing efficiency challenges inherent in RAG, which are attributable to its reliance on multiple iterations of generation. Existing A-RAG approaches process all retrieved contents from scratch. However, they ignore the situation where there is a significant overlap in the content of the retrieval results across rounds. The overlapping content is redundantly represented, which leads to a large proportion of repeated computations, thus affecting the overall efficiency. To address this issue, this paper introduces a model-agnostic approach that can be generally applied to A-RAG methods, which is dedicated to reducing the redundant representation process caused by the overlapping of retrieval results. Specifically, we use cache access and parallel generation to speed up the prefilling and decoding stages respectively. Additionally, we also propose an instruction-driven module to further guide the model to more effectively attend to each part of the content in a more suitable way for LLMs. Experiments show that our approach achieves 2.79 and 2.33 times significant acceleration on average for prefilling and decoding respectively while maintaining equal generation quality.

1 Introduction

Work smarter, not harder.

Allan F. Mogensen, 1930s

Large Language Models (LLMs) (e.g., LLaMA-2, PaLM, and GPT-4 (Touvron et al., 2023; Anil et al., 2023; Achiam et al., 2023)) have demonstrated remarkable capabilities across various natural language processing tasks. To address the growing demand for knowledge-intensive and factually grounded responses, Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm to mitigate the inherent knowledge limitations of LLMs (Lewis et al., 2020; Borgeaud et al., 2022; Ram et al., 2023; Jiang et al., 2023; Asai et al.; Gao et al., 2023; Gupta et al., 2024). It enhances the performance by retrieving relevant information from external knowledge base to generate contextually appropriate and evidence-based responses.

Conventional RAG interacts with the external knowledge bases once and combines the retrieved documents with the query as input to LLMs, this can also be called single-round RAG. Nevertheless, single-round RAG encounters challenges in han-

![The pipeline of A-RAG and analysis of document overlap between first and later retrievals](page_1012_384_388_563.png)

Figure 1: (a) The pipeline of A-RAG. (b) Analysis of document overlap between first and later retrievals (rounds 2-3) using 1000 2WikiMultihopQA samples, where Q1, Median, and Q3 represent the 25th, 50th, and 75th percentiles of document overlap ratios respectively.
dling complex user inquiries, as it is difficult to extract sufficient information by a single interaction. Consequently, an increasing number of researchers have begun to focus on and propose Adaptive-RAG (A-RAG) (Borgeaud et al., 2022; Ram et al., 2023; Mallen et al., 2023; Asai et al.; Trivedi et al., 2023; Jiang et al., 2023; Zhao et al., 2023; Yang et al., 2023b; Su et al., 2024; Zhang et al., 2024; Li et al.; Yao et al., 2024; Jeong et al., 2024; Wang et al., 2024a). Figure 1(a) illustrates the process of A-RAG, which dynamically determines whether to continue retrieval and adjusts the retrieval content based on the quality of the generated responses. Through iterative interactions with external knowledge bases, A-RAG obtains more comprehensive and valuable information to generate accurate and comprehensive answers.

Although A-RAG enhances the performance, it further exacerbates the inherent efficiency problems of the RAG due to external interactions and increased computation. Within its post-retrieval generation phase, existing methods process all retrieved contents from scratch. However, they overlooked a situation where, during a single A-RAG process, the high similarity among multiple rounds of queries results in significant overlap in the content of the retrieved results across these rounds, especially between adjacent rounds. Figure 1(b) shows the overlap ratio of documents between rounds with different settings for the number of retrieved documents. These overlapping contents are redundantly represented in each round, which leads to a large proportion of repeated computations, thereby affecting the overall efficiency.

To tackle this issue, we propose Instruction-Driven Representation Reduction (\( \mathrm{IDR}_2 \)), a model-agnostic approach widely applicable to the A-RAG methods. It aims to improve the efficiency of A-RAG by eliminating repetitive representations of overlapping content and redundant autoregressive representation rounds. Concretely, we leverage representation reduction techniques in both the prefilling and decoding processes of the generation phase. During prefilling, we propose the Cross-Iteration Cache Sharing (CICS) module, which establishes a shared memory repository for document representations. It enables subsequent processing iterations to bypass repetitive computations through cached intermediate results, thereby reducing computational overhead for overlapping content. In addition, to further direct the model to more effectively attend to the various parts of the content after prefilling, we introduce the Instruction-driven Deduplication Guidance Reinforcement (IDGR) module. It leverages the instruction-following capabilities of LLMs to implement context-aware filtration, prioritizing semantically relevant cached information while suppressing redundant content through explicit linguistic guidance in a more suitable way for LLMs. In the decoding process, we propose a novel Information-Guided Parallel Generation (IGPG) module, which leverages the correlation between retrieved documents and the generated results. By integrating phrasal fragments as inputs at each autoregressive step, IGPG enables parallel generation. It reduces the autoregressive representation rounds to achieve acceleration.

We extensively evaluate our proposed method on multiple datasets. The experimental results show that our IDR\(_2\) significantly reduces the representation process, which achieves 2.79 and 2.33 times acceleration for prefilling and decoding, consequently accelerating the entire A-RAG workflow by 2.0 times on average across various A-RAG approaches while maintaining the performance of generation. In summary, our contributions are mainly three-fold:

1. We propose IDR\(_2\), an acceleration approach for A-RAG based on representation reduction techniques. It eliminates repetitive representations during the prefilling process and reduces autoregressive representation redundancy in the decoding phase, thereby speeding up the entire A-RAG workflow almost without performance loss.

2. We develop the Instruction-driven Deduplication Guidance Reinforcement module, which leverages the instruction-following capabilities of LLMs to further direct the model to more effectively attend to the various parts of the content after prefilling in a more suitable way for LLMs.

3. Experimental results demonstrate that our approach significantly enhances the efficiency of various A-RAG methods while exhibiting robust adaptability across diverse scenarios and various LLM scales.

2 Related Works

2.1 Adaptive-RAG

Generating satisfactory answers by single-round RAG remains challenging (Komeili, 2021; Zhu
et al., 2021; Liu et al., 2024; Jiang et al., 2023; Yang et al., 2023b; Ni et al., 2024). A-RAG was developed to address this limitation by enabling iterative interactions with external knowledge bases, leading to more accurate and complete answers through multiple retrieval rounds (Jiang et al., 2023; Yang et al., 2023b; Ni et al., 2024). Current A-RAG approaches can be classified into two categories: 1) Rule-based strategies, such as per-sentence iteration (Trivedi et al., 2023), sliding window tokens (Borgeaud et al., 2022; Ram et al., 2023), and contextual learning (Zhao et al., 2023; Zhang et al., 2024; Li et al.). 2) Self-perception strategies evaluate output confidence through internal states (Yao et al., 2024), LLM output layer (Jiang et al., 2023; Yang et al., 2023b; Su et al., 2024), or explicit language-level feedback (Asai et al.). In contrast to existing works that primarily focus on performance enhancement, to the best of our knowledge, we are the first to investigate the multi-turn document overlapping problem in A-RAG and improve efficiency through its resolution.

2.2 Inference Acceleration

Efficiency of LLM. Research on improving LLM efficiency can generally be classified into two categories: traditional optimization techniques (such as quantization, pruning, knowledge distillation, etc.) and LLM-specific optimizations which include: 1) Early Exiting (Teerapittayanon et al., 2016; Xin et al., 2020; Zhou et al., 2020; Kong et al., 2022; Yang et al.; Bae et al., 2023), uses prediction heads at different layers to enable early token exit when confidence thresholds are met. 2) Token Pruning (Goyal et al., 2020; Kim and Cho, 2021; Wang et al., 2021; Kim et al., 2022; Hou et al., 2022; Zhang et al., 2023b), retains only critical tokens based on importance ranking.3) Speculative Decoding (Chen et al., 2023; Leviathan et al., 2023; Spector and Re; Yang et al., 2023a; Zhang et al., 2023a; Kim et al., 2024; Miao et al., 2024), uses efficient smaller models to generate candidate tokens for batch verification by LLM.

Efficiency of RAG. Speculative retrieval (Zhang et al.) reduces retrieval overhead through local retrieval and verification mechanisms. Parallel document processing (Merth et al.) mitigates attention complexity by processing multiple documents independently rather than concatenating them. Document preprocessing (Lu et al., 2024) improves prefilling efficiency by preprocessing and storing document representation for the whole knowledge base. Small expert LMs generate initial document-specific drafts for LLM verification (Wang et al., 2024b). In contrast to prior work, our IDR_2 is a general framework. It enhances A-RAG efficiency by representation reduction with instruction-guided information extraction.

3 Methods

Figure 2 details the workflow of IDR_2, and we divide the internal process of each iteration round in A-RAG into three phases: retrieval, prefilling, and decoding. When processing a query, cached document representations in the CICS module are first verified. These representations are loaded and combined with uncached documents for prefilling, where the IDGR module guides LLMs to prioritize relevant content while filtering noise. Before each autoregressive step, matching subsequent phrase fragments are queried in the IGPG module to enable parallel generation. Ultimately, these approaches enhance the overall efficiency of A-RAG. Details of each module are given respectively in the remainder of this section.

3.1 Cross-Iterative Cache Sharing (CICS)

A-RAG (Zhao et al., 2023; Jiang et al., 2023; Yang et al., 2023b; Su et al., 2024; Zhang et al., 2024) relies on multiple rounds of retrieval-generation interactions to generate answers. We observe substantial document overlap between adjacent retrieval rounds. In light of this, duplicate representation for overlapping documents introduces computational redundancy, hampering the efficiency of A-RAG.

In order to avoid duplicate representation for overlapping documents, we propose the cross-iterative cache sharing (CICS) module. CICS initializes a cache space \( \mathbb{C} \) to store Key-Value pairs corresponding to the documents retrieved in each round for each query \( q \). Because A-RAG may modify the query at the end of each iteration, we denote the initial user query as \( q_0 \). At round \( t \), the retrieval operation \( D_t = \textbf{Retrieve}(q_t) \) returns a set of \( n \) documents \( D_t = \{ d_1^t, d_2^t, ..., d_n^t \} \) from the external knowledge base. \( n \) denotes the maximum number of documents that can be retrieved in each round. At round \( t \), the LLM generates a sequence of tokens \( A_t = \{ a_1^t, a_2^t, ..., a_m^t \} \) by:

\[
a_1^t, K_t, V_t = \textbf{LLM}_P(q_t, D_t, A_{<t})
\]
Figure 2: The pipeline of our IDR_2. The same color indicates the same document and representation.

\[
a_t^i, k_t^i, v_t^i = \text{LLM}_D(q_t, D_t, A_{<t}, a_t^{<i}, K_t, V_t) \\
K_t = \text{Concat}(K_t, k_t^i) \\
V_t = \text{Concat}(V_t, v_t^i)
\]
where the inference process of LLM is divided into two stages: prefilling and decoding, represented with \( \text{LLM}_P \) and \( \text{LLM}_D \) respectively. The \( K_t \) and \( V_t \) represent the Key-Value pairs for \( D_t \) and \( A_t \). The \( a_t^i \) denotes the \( i^{th} \) generated token and \( m \) represents the number of generated tokens. The Eq.(2) denoted the autoregression process, which can only generate one token at each step. It needs to be executed through multiple steps to obtain the complete \( A_t \). CICS stores the \( K_t, V_t \) from Eq.(1), which is the representation for \( D_t \) as shown in Figure 2.

After receiving retrieved results \( D_t \), CICS extracts existing representation from the cache to avoid duplicate representation. Specifically, as the example shown in Figure 2, CICS directly loads the representation of documents #5881721 and #12415199, then integrates them with the text of document #12368478 for generation. This process can be formalized as:
\[
K_t^o, V_t^o, D_t^o = \text{Filter}(D_t, \mathbb{C}) \\
a_t^1, K_t, V_t = \text{LLM}_P(q_t, D_t^n, A_{<t}, K_t^o, V_t^o)
\]
where \( D_t^o \) is the document set that has been processed in the previous round and appears again at the current round \( D_t \). We can directly obtain its corresponding representation \( K_t^o \) and \( V_t^o \) from \( \mathbb{C} \) without re-processing. For the new document set \( D_t^n = D_t \setminus D_t^o \) at the current round, the prefilling is still required.

Eq.(3) optimizes the prefilling phase by reusing the cached representation of overlapping documents in CICS, avoiding redundant computation.

3.2 Instruction-driven Deduplication Guidance Reinforcement (IDGR)

The CICS provides efficient representation reuse across rounds. Notably, the Key-Value representation of each document \( d_t^k \) incorporates information from previously processed documents through self-attention. As an example illustrated in Figure 2, at round 1, the representation of document #12415199 contains information from documents #5881721 and #10028469 due to the self-attention mechanism.

The IDGR employs natural language instructions \( I_t \) in the prompt to guide the LLM in filtering redundant cached information. This module helps the model focus on relevant content for the current round, thereby ensuring high-quality generation. At each iteration, the instruction \( I_t \) is automatically generated based on two rules and is subsequently utilized during the prefilling phase, as:

\[
a_t^1, K_t, V_t = \text{LLM}_P(q_t, D_t^n, A_{<t}, R_t^o, I_t)
\]
where \( R_t^o \) is the \( K_t^o, V_t^o \). As shown in Figure 2, the representation of document #10028469 is excluded from the computation. Furthermore, explicit instructions guide the LLM to ignore the information contained in the representation of document #12415199.

The instruction \( I_t \) is generated according to two rules: First, we use document identifiers to explicitly tell the LLM which documents are relevant and which are irrelevant for the current generation. This helps the LLM focus on pertinent context while filtering out redundant information. Second, we provide document relevance rankings to the LLM. In cases where we cannot directly adjust the input
order of documents based on their importance, explicit instructions guide the LLM to prioritize more relevant documents.

In practice, \( I_t \) can be implemented with simple natural language directives, leveraging the strong semantic understanding capabilities of LLMs. For instance, in the second round shown in Figure 2, the instruction is formulated as: “#5881721 ... are related docs. #10028469 is unrelated. The relevance scores are ...”. The relevance scores can be obtained through either retriever rankings\(^1\) or model-based scoring methods\(^2\). While scoring methods are not the focus of our study, they can be seamlessly integrated into our approach.

3.3 Information-Guided Parallel Generation (IGPG)

![Bar chart showing coverage ratio (%) vs. number of consecutive tokens for Round 1 and Round 2](page_184_682_495_246.png)

Figure 3: The x-axis represents the length of consecutive token combinations in the generated results. The y-axis represents the proportion of all combinations in the generated results that appear in the retrieved results. (LLaMA2-7B, 2WikiMultihopQA).

RAG differs from standard LLM applications in its access to extensive external context. For a given query \( q \), standard LLM inference can only utilize \( q \) itself, whereas RAG leverages both \( q \) and retrieved documents \( D_t \). Furthermore, in different iterations of A-RAG, the LLM-generated content maintains high relevance to the retrieved documents \( D_t \), as illustrated in Figure 3. This relevance plays a crucial role in determining the current round’s output. The LLM generates a significant amount of existing content. The autoregressive process of token-by-token generation primarily contributes to the low efficiency. In contrast to standard LLM generation, we have the information of \( D_t \), which contains a substantial portion of the content that LLM must generate. Consequently, we can utilize the information from \( D_t \) to guide LLM in avoiding the independent autoregressive representation process for existing content. By constructing phrase fragments and verifying multiple tokens during autoregression, parallel generation can be achieved.

Different with traditional speculative decoding approaches, IGPG requires neither the construction of small language models nor training. During each RAG iteration, IGPG uses \( D_t \) to construct a approximate probabilistic language model \( P(x_t|x_1,...,x_{t-1}) \approx P(x_t|x_{t-N+1},...,x_{t-1}), N \) is the number of recent tokens.

Our IGPG has two steps as speculative decoding approaches: draft generation and LLM parallel generation. In the draft generation, by iterating \( M \) steps, the module constructs a \( M \)-length draft token sequence \( \hat{A}_{t,k} = \{\hat{a}_{t,k}^1, \hat{a}_{t,k}^2, ..., \hat{a}_{t,k}^M\} \), \( k \) denotes the \( k \)-th autoregression step. During LLM parallel generation, the LLM validates the draft token sequence through a forward pass by \( P(\hat{a}_{t,k}^{i+M+1}|a_{t,k-1}^{i}\leq i, \hat{a}_{t,k}^1, ..., \hat{a}_{t,k}^M) \). The LLM examines whether each draft token aligns with the current output, as Eq.(5). When a draft token \( \hat{a}_{t,k}^j \) fails validation, it is substituted with the LLM’s prediction \( \hat{a}_{t,k}^{j-1} \) and draft generation resumes from this point until the complete sequence \( A_t \) is produced.

\[
a_{t,k}^j = \begin{cases}
\hat{a}_{t,k}^j, & \text{if } \hat{a}_{t,k}^j = \hat{a}_{t,k}^{j-1}, j \geq 1 \\
\hat{a}_{t,k}^{j-1} & \text{otherwise \& stop}
\end{cases}
\]

where \( \{\hat{a}_{t,k}^0, ..., \hat{a}_{t,k}^{i+M+1}\} = \mathbf{LLM}_D(a_{t,k-1}^{i}, \hat{A}_{t,k}) \), \( \hat{a}_{t,k}^0 \) corresponds to \( a_{t,k-1} \) autoregressive.

If at least one token in the \( \hat{A}_{t,k} \) is validated by the LLM, combined with the token generated by the LLM itself, the total number of generated tokens is at least two, achieving a least \( 2\times \) speedup.

4 Experiments

4.1 Language Models
We employed the LLaMA2-7B/13B (Touvron et al., 2023) (L2-7B/13B) and the Vicuna-7B/13B (Chiang et al., 2023) (V-7B/13B). The Vicuna model is a chatbot fine-tuned on ShareGPT conversation data. It contains different knowledge from LLaMA and has an impact on A-RAG inference.

4.2 Downstream Task Datasets
We use the 2WikiMultihopQA (Ho et al., 2020), HotpotQA (Yang et al., 2018), StrategyQA (Geva et al., 2021) and IIRC (Ferguson et al., 2020) as DRAGIN (Su et al., 2024). The datasets span multi-hop question answering, common sense reasoning, and reading comprehension, covering different
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">LLMs</th>
    <th colspan="3">2WikiMultiHopQA</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">StrategyQA</th>
    <th colspan="3">IIRC</th>
  </tr>
  <tr>
    <th>Pref.</th><th>↑ Deco.</th><th>E2E↑</th>
    <th>Pref.</th><th>↑ Deco.</th><th>E2E↑</th>
    <th>Pref.</th><th>↑ Deco.</th><th>E2E↑</th>
    <th>Pref.</th><th>↑ Deco.</th><th>E2E↑</th>
  </tr>
  <tr>
    <td rowspan="3">FLRAG+IDR<sub>2</sub></td>
    <td>L-2-7B</td>
    <td>2.23×</td><td>2.37×</td><td>1.75×</td>
    <td>2.27×</td><td>1.85×</td><td>1.53×</td>
    <td>1.73×</td><td>1.49×</td><td>1.40×</td>
    <td>3.08×</td><td>2.76×</td><td>2.10×</td>
  </tr>
  <tr>
    <td>L-2-13B</td>
    <td>2.12×</td><td>2.36×</td><td>1.64×</td>
    <td>2.29×</td><td>2.00×</td><td>1.54×</td>
    <td>1.76×</td><td>1.62×</td><td>1.40×</td>
    <td>2.70×</td><td>2.65×</td><td>1.83×</td>
  </tr>
  <tr>
    <td>V-13B</td>
    <td>2.05×</td><td>2.25×</td><td>1.60×</td>
    <td>2.22×</td><td>1.97×</td><td>1.51×</td>
    <td>1.65×</td><td>1.55×</td><td>1.31×</td>
    <td>2.66×</td><td>2.72×</td><td>1.82×</td>
  </tr>
  <tr>
    <td rowspan="3">FSRAG+IDR<sub>2</sub></td>
    <td>L-2-7B</td>
    <td>2.04×</td><td>2.64×</td><td>2.32×</td>
    <td>2.41×</td><td>1.88×</td><td>1.69×</td>
    <td>2.36×</td><td>1.97×</td><td>1.79×</td>
    <td>2.92×</td><td>2.88×</td><td>2.20×</td>
  </tr>
  <tr>
    <td>L-2-13B</td>
    <td>2.11×</td><td>2.47×</td><td>1.79×</td>
    <td>2.16×</td><td>2.05×</td><td>1.59×</td>
    <td>1.74×</td><td>1.59×</td><td>1.44×</td>
    <td>2.16×</td><td>2.65×</td><td>1.74×</td>
  </tr>
  <tr>
    <td>V-13B</td>
    <td>2.23×</td><td>2.31×</td><td>1.75×</td>
    <td>2.15×</td><td>2.31×</td><td>1.66×</td>
    <td>2.02×</td><td>1.78×</td><td>1.53×</td>
    <td>2.35×</td><td>2.69×</td><td>1.92×</td>
  </tr>
  <tr>
    <td rowspan="3">FLARE+IDR<sub>2</sub></td>
    <td>L-2-7B</td>
    <td>2.81×</td><td>2.74×</td><td>2.60×</td>
    <td>3.12×</td><td>2.61×</td><td>2.31×</td>
    <td>2.71×</td><td>2.27×</td><td>1.77×</td>
    <td>3.54×</td><td>2.97×</td><td>2.89×</td>
  </tr>
  <tr>
    <td>L-2-13B</td>
    <td>1.98×</td><td>2.24×</td><td>2.16×</td>
    <td>2.42×</td><td>1.87×</td><td>1.80×</td>
    <td>2.45×</td><td>1.71×</td><td>1.69×</td>
    <td>3.58×</td><td>3.15×</td><td>3.01×</td>
  </tr>
  <tr>
    <td>V-13B</td>
    <td>2.04×</td><td>2.14×</td><td>2.06×</td>
    <td>2.27×</td><td>2.07×</td><td>1.98×</td>
    <td>2.05×</td><td>1.93×</td><td>1.89×</td>
    <td>2.70×</td><td>2.24×</td><td>2.22×</td>
  </tr>
  <tr>
    <td rowspan="3">DRAGIN+IDR<sub>2</sub></td>
    <td>L-2-7B</td>
    <td>3.34×</td><td>2.77×</td><td>2.52×</td>
    <td>4.09×</td><td>2.08×</td><td>2.09×</td>
    <td>3.39×</td><td>1.98×</td><td>1.95×</td>
    <td>4.49×</td><td>2.58×</td><td>2.58×</td>
  </tr>
  <tr>
    <td>L-2-13B</td>
    <td>3.25×</td><td>3.21×</td><td>2.61×</td>
    <td>3.97×</td><td>2.15×</td><td>2.01×</td>
    <td>3.15×</td><td>1.98×</td><td>1.87×</td>
    <td>4.25×</td><td>2.86×</td><td>2.65×</td>
  </tr>
  <tr>
    <td>V-13B</td>
    <td>3.57×</td><td>2.81×</td><td>2.55×</td>
    <td>4.14×</td><td>2.29×</td><td>2.15×</td>
    <td>3.25×</td><td>1.88×</td><td>1.82×</td>
    <td>4.72×</td><td>4.00×</td><td>3.53×</td>
  </tr>
</table>

Table 1: The average speedup ratios of IDR<sub>2</sub> across different methods, models, and datasets. Pref. (prefilling) demonstrates acceleration achieved through CICS and IDGR. Deco. (decoding) shows speedup from IGPG. E2E (end-to-end) reflects the overall A-RAG acceleration, encompassing retrieval. (\( n = 3, 1 \times = \) baseline speed)

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Methods</th>
    <th colspan="2">2WQA</th>
    <th colspan="2">HQA</th>
    <th colspan="2">SQA</th>
    <th colspan="2">IIRC</th>
  </tr>
  <tr>
    <th>EM ↑</th><th>F1 ↑</th>
    <th>EM ↑</th><th>F1 ↑</th>
    <th>Acc. ↑</th><th>EM ↑</th><th>F1 ↑</th>
  </tr>
  <tr>
    <td rowspan="2">Llama2-7B</td>
    <td>DRAGIN†</td>
    <td>22.5</td><td>28.68</td>
    <td>22.6</td><td>33.02</td>
    <td>65.10</td><td>15.93</td><td>20.24</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td>
    <td><b>25.4</b></td><td><b>33.17</b></td>
    <td><b>22.9</b></td><td>32.59</td>
    <td>62.50</td><td><b>16.04</b></td><td><b>20.24</b></td>
  </tr>
  <tr>
    <td rowspan="2">Llama2-13B</td>
    <td>DRAGIN†</td>
    <td>30.4</td><td>39.91</td>
    <td><b>31.6</b></td><td><b>42.60</b></td>
    <td><b>66.10</b></td><td>18.5</td><td><b>22.59</b></td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td>
    <td><b>34.4</b></td><td><b>41.50</b></td>
    <td>29.9</td><td>41.01</td>
    <td>65.77</td><td><b>18.55</b></td><td>22.11</td>
  </tr>
  <tr>
    <td rowspan="2">vicuna-13b</td>
    <td>DRAGIN†</td>
    <td>25.4</td><td>34.90</td>
    <td>30.1</td><td>42.50</td>
    <td>66.20</td><td><b>23.90</b></td><td><b>28.11</b></td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td>
    <td><b>28.9</b></td><td><b>36.77</b></td>
    <td><b>31.7</b></td><td><b>42.58</b></td>
    <td><b>68.00</b></td><td>21.59</td><td>26.28</td>
  </tr>
</table>

Table 2: The performance comparison on different datasets with different models. The † represents the reproduction.

types of reasoning tasks. For more detailed settings, please refer to the Appendix A.2.

4.3 Baselines

We selected four representative A-RAG methods as baselines. These methods are characterized by two key aspects: retrieval timing and query construction. Retrieval timing determines when to perform retrieval, while query construction decides what content to use for retrieval. These features directly impact both the quality and efficiency of A-RAG workflows. FL-RAG (Khandelwal et al., 2019; Borgeaud et al., 2022; Ram et al., 2023) retrieves every \( z \) tokens using previous tokens as queries, FS-RAG (Trivedi et al., 2023) performs retrieval after each sentence, FLARE (Jiang et al., 2023) triggers retrieval for uncertain tokens, using the last generated sentence as the query, and DRAGIN (Su et al., 2024) employs a dynamic approach based on content importance and uncertainty, utilizing attention distribution for query construction.

4.4 Implementation Details

For the retrieval modules, we select BM25 and follow DRAGIN (Su et al., 2024). Additionally, we also investigate replacing BM25 with SGPT (Muenninghoff, 2022), a dense retrieval method. Our external knowledge is based on Wikipedia articles, which are divided into passages containing 100 tokens each. For the 7B and 13B models, we use four Nvidia 3090 GPUs and four H800 GPUs, respectively. \( n \) is the number of retrieved documents. Our IDR<sub>2</sub> based on the PyTorch (Paszke et al., 2019), details in the Appendix A.4.

4.5 Main Results

Table 1 shows the results of our approach applied to the different A-RAG methods, different models with scales, and different scenarios. Table 1 shows the acceleration ratios over baseline methods.

Overall, our method demonstrates end-to-end speed improvements ranging from 1.31\( \times \) to 3.53\( \times \). During the prefilling phase, we observe acceleration factors from 1.75\( \times \) to 4.72\( \times \). These results substantiate the efficacy of our proposed CICS and
IDGR mechanisms in effectively eliminating redundant representations. For the decoding phase, the IGPG technique achieves speedup ratios from \(1.49 \times\) to \(4.00 \times\) through the reduction of autoregressive inference iterations. A comparative analysis between the LLaMA2-7B and 13B architectures (underlining indicates better results) shows slightly better results for the 7B variant. We consider that this effect stems from the fact that smaller models require less additional overhead to store and load representations. Of the various A-RAG methods, the most significant performance improvement (\(4.72 \times\) for prefilling) is achieved when applying IDR\(_2\) to DRAGIN. We attribute this to the query refinement mechanism of DRAGIN which improves query similarity, a property that effectively synergizes with our representation reduction approach.

<table>
  <tr>
    <th>Models</th>
    <th>Methods</th>
    <th>Pref. (s)\(\downarrow\)</th>
    <th>Deco. (s)\(\downarrow\)</th>
    <th>E2E (s)\(\downarrow\)</th>
  </tr>
  <tr>
    <td rowspan="2">L2-7B</td>
    <td>DR.</td>
    <td>3.71</td>
    <td>12.55</td>
    <td>19.31</td>
  </tr>
  <tr>
    <td>DR.+Ours</td>
    <td><b>1.18</b></td>
    <td><b>6.07</b></td>
    <td><b>9.56</b></td>
  </tr>
  <tr>
    <td rowspan="2">L2-13B</td>
    <td>DR.</td>
    <td>3.84</td>
    <td>19.26</td>
    <td>26.31</td>
  </tr>
  <tr>
    <td>DR.+Ours</td>
    <td><b>1.24</b></td>
    <td><b>8.06</b></td>
    <td><b>12.34</b></td>
  </tr>
  <tr>
    <td rowspan="2">V-13b</td>
    <td>DR.</td>
    <td>4.34</td>
    <td>25.03</td>
    <td>33.47</td>
  </tr>
  <tr>
    <td>DR.+Ours</td>
    <td><b>1.29</b></td>
    <td><b>10.49</b></td>
    <td><b>15.18</b></td>
  </tr>
</table>

Table 3: The runtime of IDR\(_2\) on 2WikiMultiHopQA dataset (\(n = 3\)), DR. denotes DRAGIN (Su et al., 2024).

The specific runtime of the different phases in the generation process are exhibited in Table 3. Our experiments reveal that even on high-end NVIDIA H800 GPUs, 13B-parameter models require 26.31-33.47 seconds to process a single request. The proposed method significantly reduces the overall latency to 9.56-15.18 seconds, demonstrating substantial performance improvements and enhanced practicality for real-world deployment.

Table 2 shows the performance of applying the IDR\(_2\) to the DRAGIN method. The experimental results show that our approach effectively preserves the original performance while exhibiting strong adaptability and generalization across different models and tasks. Notably, our method also carries some enhancements on 2WikiMultiHopQA. We also provide performance results applied to other methods, in the Appendix A.10.

4.6 Ablation studies

4.6.1 Impact of Retrieval Size

We analyze the impact of varying numbers of retrieved documents on A-RAG, with results visualized in Figure 4. As the document count increases, the context length grows monotonically, resulting in progressively higher prefilling overhead proportion. Our approach maintains consistent speedup advantages across both prefilling and decoding.

4.6.2 Impact of Different Retrievers

Table 4 demonstrates the effectiveness of IDR\(_2\) across different retrievers. Substituting the retriever with SGPT results in an EM score of 29.6 and a
Figure 5: The detailed process of A-RAG based on a specific example, with LLaMA-13B.

<table>
  <tr>
    <th>Method</th>
    <th>Retriever</th>
    <th>EM ↑</th>
    <th>E2E ↑</th>
  </tr>
  <tr>
    <td>DRAGIN</td>
    <td>BM25</td>
    <td>30.4</td>
    <td>1×</td>
  </tr>
  <tr>
    <td>DRAGIN + IDR<sub>2</sub></td>
    <td>BM25</td>
    <td>34.4</td>
    <td>2.61×</td>
  </tr>
  <tr>
    <td>DRAGIN</td>
    <td>SGPT</td>
    <td>27.3</td>
    <td>1×</td>
  </tr>
  <tr>
    <td>DRAGIN +IDR<sub>2</sub></td>
    <td>SGPT</td>
    <td>29.6</td>
    <td>2.73×</td>
  </tr>
</table>

Table 4: Analysis of different retrievers on the 2Wiki-MultihopQA with LLaMA2-13B (\( n = 3 \)).

speedup ratio of \( 2.73 \times \). The consistent performance and acceleration above \( 2.6 \times \) across both frequency-based and semantic retrievers confirms the adaptability of IDR<sub>2</sub>.

5 Analysis

5.1 Do these modules impact performance?

Table 5 shows how different modules affect the generation quality. The redundant information in the CICS module impacts the generation quality of LLM. This is evident in the performance of LLaMA2-7B, where EM dropped from 22.0 to 20.3, with LLaMA2-13B showing similar decreases. The introduction of IDGR not only alleviates the adverse effects of redundant information but also brings additional performance gains. The LLaMA2-7B achieves EM scores of 25.4, and LLaMA2-13B reaches 34.4. These results confirm that IDGR successfully maintains inference quality.

<table>
  <tr>
    <th>CICS</th>
    <th>IGPG</th>
    <th>IDGR</th>
    <th>LLaMA2-7B</th>
    <th>LLaMA2-13B</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>EM ↑</th>
    <th>F1 ↑</th>
    <th>EM ↑</th>
    <th>F1 ↑</th>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>22.5</td>
    <td>28.68</td>
    <td>30.4</td>
    <td>39.91</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>-</td>
    <td>20.3</td>
    <td>26.88</td>
    <td>28.0</td>
    <td>37.49</td>
  </tr>
  <tr>
    <td>-</td>
    <td>✓</td>
    <td>-</td>
    <td>22.4</td>
    <td>28.51</td>
    <td>30.4</td>
    <td>40.02</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>-</td>
    <td>20.2</td>
    <td>26.78</td>
    <td>28.3</td>
    <td>37.62</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td><b>25.4</b></td>
    <td><b>33.17</b></td>
    <td><b>34.4</b></td>
    <td><b>41.50</b></td>
  </tr>
</table>

Table 5: Analysis of the impact of different modules on 2WikiMultihopQA based on DRAGIN.

Poulantzas requires six token-level autoregression: _N, icos, _P, oul, ant, zas. After applying IGPG, only one autoregression is needed after the prefix "Nicos," is generated, achieving \( 4 \times \) local acceleration. Similarly, dates like October 1979, September 1936, February 1568 achieve \( 4 \times \) acceleration by retrieving corresponding fragments from IGPG using month information. At Round 3, LLM further confirms Giuseppe Cesari’s lifespan and generates the right answer. With all representations already in CICS, the prefilling time reduces by 0.804 seconds (about 80% of DRAGIN’s original time), achieving 5× speedup. Total savings reach 1.334s (prefilling) and 6.16s (decoding), summing to 7.494s (48.47% reduction) over DRAGIN’s 15.46s, achieving 1.95× end-to-end acceleration. A more detailed explanation can be viewed in the Appendix A.1.

6 Conclusion

This paper presents IDR<sub>2</sub>, a model-agnostic approach that significantly improves the efficiency of A-RAG by addressing redundant representation processing across multiple iterations. To prevent redundant computation, this paper introduces the CICS module for caching document representations. The IDGR module then guides the generation process to focus on crucial information through the instruction-following capability of LLMs. Furthermore, the IGPG module leverages the corre-
lation between generated content and documents to enable parallel generation of existing content, reducing autoregressive rounds. Through extensive experiments, IDR_2 consistently demonstrates remarkable efficiency improvements, achieving up to 2.0× acceleration while maintaining generation quality. Our future work aims to further compress the KV cache while preserving its representation capabilities.

7 Limitations

We acknowledge the limitations of this paper. Although our method is a general approach, it requires the corresponding LLM to be open-source to apply the CICS and IGPG technologies in this paper. It should be noted that the method in this paper is not applicable to LLM APIs that only support text-based interfaces. Therefore, our future work also aims to develop more methods to overcome the limitations of similar scenarios.

8 Acknowledgments

This research is supported by the Sichuan International Science and Technology Innovation Cooperation Project with ID 2024YFHZ0317, the Chengdu Science and Technology Bureau Project with ID 2024-YF09-00041-SN, and the National Natural Science Foundation of China Project with ID W2433163.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayam Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropaa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.

Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. 2023. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5910–5924.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. 2020. Iirc: A dataset of incomplete information reading comprehension questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1137–1147.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361.

Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pages 3690–3699. PMLR.

Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. 2024. A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions. arXiv preprint arXiv:2410.12837.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625.

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. 2022. Token dropping for efficient bert pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3774–3784.
Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7029–7043.

Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992.

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.

Gyuwan Kim and Kyunghyun Cho. 2021. Length-adaptive transformer: Train once with length drop, use anytime with search. In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP 2021, pages 6501–6511. Association for Computational Linguistics (ACL).

Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. 2024. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36.

Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784–794.

M Komeili. 2021. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566.

Jun Kong, Jin Wang, Liang-Chih Yu, and Xuejie Zhang. 2022. Accelerating inference for pretrained language models by unified multi-perspective early exiting. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4677–4686.

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In The Twelfth International Conference on Learning Representations.

Huanshuo Liu, Bo Chen, Menghui Zhu, Jianghao Lin, Jiarui Qin, Hao Zhang, Yang Yang, and Ruiming Tang. 2024. Retrieval-oriented knowledge for click-through rate prediction. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1441–1451.

Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. 2024. Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text. arXiv preprint arXiv:2410.07590.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822.

Thomas Merth, Qichen Fu, Mohammad Rastegari, and Mahyar Najibi. Superposition prompting: Improving and accelerating retrieval-augmented generation. In Forty-first International Conference on Machine Learning.

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xiniao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2024. Specifier: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 932–949.

Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904.

Shiyu Ni, Keiping Bi, Jiafeng Guo, and Xueqi Cheng. 2024. When do llms need retrieval augmentation? mitigating llms’ overconfidence helps retrieval augmentation. arXiv preprint arXiv:2402.11457.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.
Benjamin Frederick Spector and Christopher Re. Accelerating llm inference with staged speculative decoding. In Workshop on Efficient Systems for Foundation Models@ ICML2023.

Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. arXiv preprint arXiv:2403.10081.

Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pages 2464–2469. IEEE.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Alber, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykhov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014–10037.

Hanrui Wang, Zhekai Zhang, and Song Han. 2021. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 97–110. IEEE.

Ruoqing Wang, Daren Zha, Shi Yu, Qingfei Zhao, Yuxuan Chen, Yixuan Wang, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, et al. 2024a. Retriever-and-memory: Towards adaptive note-enhanced retrieval-augmented generation. arXiv preprint arXiv:2410.08821.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Zilong Wang, Zifeng Wang, Long Le, Huaxiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, et al. 2024b. Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint arXiv:2407.08223.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 24824–24837.

Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246–2251.

Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023a. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487.

Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding. Transactions on Machine Learning Research.

Yuchen Yang, Houqiang Li, Yanfeng Wang, and Yu Wang. 2023b. Improving the reliability of large language models by leveraging uncertainty-aware in-context learning. arXiv preprint arXiv:2310.04782.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, and Juanzi Li. 2024. Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. arXiv preprint arXiv:2406.19215.

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023a. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168.

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuan-dong Tian, Christopher Ré, Clark Barrett, et al. 2023b. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661–34710.

Zhihao Zhang, Alan Zhu, Lijie Yang, Yihua Xu, Lanting Li, Phitchaya Mangpo Phothilimthana, and Zhihao Jia. Accelerating iterative retrieval-augmented language model serving with speculation. In Forty-first International Conference on Machine Learning.

Zihan Zhang, Meng Fang, and Ling Chen. 2024. Retrievalqa: Assessing adaptive retrieval-augmented generation for short-form open-domain question answering. arXiv preprint arXiv:2402.16457.

Rouchen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023. Verify-and-edit: A knowledge-enhanced chain-of-thought framework. In The 61st Annual Meeting Of The Association For Computational Linguistics.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330–18341.

Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774.

A Example Appendix

A.1 More Detail for Case Study

Figure 6 demonstrates the detailed process of running the LLaMA2-13B model with both the original DRAGIN and our proposed IGIR-ARAG framework on the 2WikiMultihotQA dataset. Initially, the model receives the question Who lived longer, Giuseppe Cesari or Nicos Poulantzas?. Without accessing external knowledge, LLM generates the answer Giuseppe Cesari lived from 1566 to 1640. Nicos Poulantzas lived from 1920 to 1979. Thus, Nicos Poulantzas lived longer., shown in the Without RAG phase. After hallucination detection indicates low confidence, the process enters the RAG phase, retrieving three documents numbered #3596518, #3596521, and #10413313.

In Round 1, with CICS initially empty, no prefilling acceleration occurs, taking 0.96 seconds. IGPG then models the three documents, reducing the Decoding time from 4.24 to 2.16 seconds through parallel decoding. After detecting hallucination in the generated answer, the process enters Round 2. Here, CICS reuses document #359518’s representation, saving 0.53 seconds in prefilling. During decoding, IGPG builds fragment models for document reuse, saving 2.31 seconds through parallel decoding. For example, Nicos Poulantzas requires six token inferences: _N, icos, _P, out, ant, zas. With IGPG modeling, only one inference is needed after the prefix “Nicos,” achieving 4× local acceleration. Similarly, dates like October 1979, September 1936, February 1568 achieve 4× acceleration by retrieving corresponding fragments from IGPG using month information.

In Round 2, despite accurately obtaining both individuals’ lifespans, the comparison is incorrect, leading to Round 3. Here, LLM further confirms Giuseppe Cesari’s lifespan and generates the right answer. With all documents already in CICS, the prefilling time reduces by 0.804 seconds (about 80% of DRAGIN’s original time), achieving 5× speedup. Finally, LLM correctly compares the information and provides the right answer.

Throughout this case, the framework saves 1.334 seconds in prefilling and 6.16 seconds in Decoding, totaling 7.494 seconds saved. Compared to DRAGIN’s original 15.46 seconds, this represents a 48.47% time reduction, achieving 1.95× speedup.

A.2 Dataset Setting

For 2WikiMultihopQA, we followed Wang et al.’s approach for generating reasoning chains and answers, while incorporating prompt templates from Trivedi et al. (2023) and Jiang et al. (2023). The HotpotQA implementation followed Trivedi et al. (2023)’s settings and prompt templates. Both datasets employ Exact Matching at the answer level and F1 scores at the token level. For common sense reasoning evaluation, we adopted the StrategyQA dataset (Geva et al., 2021), implementing Wei et al. (2022)’s methodology and prompt templates, along with insights from Jiang et al. (2023). The evaluation uses Exact Matching to compare generated yes/no answers with standard answers. For reading comprehension capabilities, we adopted the IIRC dataset (Ferguson et al., 2020) following Trivedi et al. (2023)’s setup and evaluation metrics similar to 2WikiMultihopQA.

For the prompt templates of different datasets, we use the configuration of the DRAGIN (Su et al., 2024). The number of test samples and the specific test sample ids used for each dataset are kept consistent with DRAGIN (Su et al., 2024). We use 8 shots for those datasets in prompt.

A.3 Impact of A-RAG Rounds

We analyzed the impact of our method on the rounds of A-RAG, as shown in Table 7. Our IDR_2 maintains comparable round counts (Ours: 2.34-4.77 vs. DRAGIN: 2.68-5.33), demonstrating that our method adheres to the original design principles of DRAGIN.

A.4 Description of the libraries used in the implementation

Our code is developed based on PyTorch (Paszke et al., 2019), with the data loading module constructed on the open source code provided by DRAGIN (Su et al., 2024). We used Elasticsearch-7.17.9 for BM25.
Figure 6: The detailed process of A-RAG based on a specific example, with LLaMA-13B.

<table>
  <tr>
    <th colspan="2">Dataset statistics</th>
    <th>HotpotQA</th>
    <th>IIRC</th>
    <th>StrategyQA</th>
  </tr>
  <tr>
    <th>Task</th>
    <th>#Examples</th>
    <th>multi-hop QA 1000</th>
    <th>multi-hop QA 1000</th>
    <th>reading comprehension QA 954</th>
    <th>commonsense QA 1000</th>
  </tr>
</table>

Table 6: Dataset statistics and experimental settings from DRAGIN (Su et al., 2024).

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Methods</th>
    <th colspan="2">2WMQA</th>
    <th>HQA</th>
    <th>SQA</th>
    <th>IIRC</th>
  </tr>
  <tr>
    <th>DR.</th>
    <th>DR.+Ours</th>
    <th>DR.</th>
    <th>DR.+Ours</th>
    <th>DR.</th>
    <th>DR.+Ours</th>
  </tr>
  <tr>
    <td>L2-7B</td>
    <td></td>
    <td>2.68</td>
    <td>2.34</td>
    <td>3.26</td>
    <td>2.77</td>
    <td>5.33</td>
    <td>4.77</td>
    <td>2.73</td>
    <td>2.08</td>
  </tr>
  <tr>
    <td>L2-13B</td>
    <td></td>
    <td>2.74</td>
    <td>2.41</td>
    <td>3.69</td>
    <td>3.45</td>
    <td>4.66</td>
    <td>4.25</td>
    <td>3.06</td>
    <td>2.49</td>
  </tr>
  <tr>
    <td>V-13B</td>
    <td></td>
    <td>3.02</td>
    <td>2.44</td>
    <td>3.47</td>
    <td>2.98</td>
    <td>4.41</td>
    <td>3.93</td>
    <td>3.28</td>
    <td>2.60</td>
  </tr>
</table>

Table 7: The number of rounds of A-RAG.

Prompt (Fewshots)

Question: When did the director of film Hypocrite (Film) die?
Answer: The film Hypocrite was directed by Miguel Morayta. Miguel Morayta died on 19 June 2013. So the answer is 19 June 2013.

February 1959. Ivania Martinich was born on 25 July 1995. Thus, Martin Hodge was born first. So the answer is Martin Hodge.

Question: Which album was released earlier, Ngrtd or Everything I Love? Answer: Ngrtd was released on May 18, 2015 by Bomayé Music. Everything I Love was released on October 29, 1996. Thus, Everything I Love was released earlier. So the answer is Everything I Love.

A.5 Details of Prompt Template

The IDGR employs the prompt to guide the LLM in filtering redundant cached information, as shown below.

Step1: Learning from fewshots.

Step2: Input the retrieved documents into the LLM.
Prompt (Retrieved documents)

Context:
[1672255] Christian August, Prince of Anhalt-Zerbst Christian August……
[12419160] Frederick Augustus, Prince of Anhalt-Zerbst Frederick Augustus, Prince……
[1672259] 1727 in Vechelde, Christian August married Johanna Elisabeth……

Step3: Using IDGR to guide the model to focus on relevant content for the current round.

IDGR

Document: [12368478] is related.
Document: [10028469] is unrelated.
The relevance scores are:
[12415199] Score: 3
[5881721] Score: 2
[12368478] Score: 1
Answer in the same format as before.

A.6 Results Calculation Methodology

The calculation method of the speedup ratio presented in Table 8: first compute the individual speedup ratio for each sample, then calculate the arithmetic mean of all the ratios. The calculation method of the speed presented in Table 9 is calculated by taking the arithmetic mean of all sample speeds. Notably, the speedup ratio calculated by dividing the average speeds recorded in Table 9 differs from the result shown in Table 8, which is obtained by averaging individual sample speedup ratios.

A.7 Explanation of Overlap Rate

Figure 7 shows the overlap rate of the retrieved documents between different rounds. For example, as illustrated in Figure 7a, in the case of retrieving 3 documents, 25% of the samples have an overlap rate between 0% and 33%, 25% have an overlap rate between 33% and 66%, 25% have an overlap rate of 66%, and the remaining 25% have an overlap rate greater than 66%. The red dots represent the mean of the overlap rate in different numbers of retrieved documents. For example, in Figure 7b, when the number of the retrieved documents is 5, the mean overlap rate of the retrieved documents between the first and third rounds is 37.95%.

(a) Overlap rate comparison between round 1 and round 2.

(b) Overlap rate comparison between round 1 and round 3.

(c) Overlap rate comparison between round 1 and round 4.

Figure 7: The overlap rate comparison between different rounds.

A.8 The efficiency of IDR_2 on different devices

To verify whether IDR_2 has the same effect on different hardware devices, we tested the runtime of LLaMA-7B on the 2WikiMultihopQA dataset on Nvidia 3090 GPU and Nvidia H800 GPU respectively. As shown in Figure 9, we tested the runtime when setting different numbers of retrieved documents. The experimental results show that our IDR_2 framework achieves consistent performance improvements across different hardware devices.

A.9 Hyperparameters Analysis of M, N

![Bar chart showing speed-up ratios achieved with different combinations of M, N](page_1012_1342_377_246.png)

Figure 8: The speed-up ratios achieved with different combinations of \( M, N \).

We conducted extensive experiments using LLaMA2-7B on the 2WikiMultihopQA dataset to evaluate the impact of different queue depths (M) and thread counts (N) on processing speed,
as shown in Figure 8. Based on these results, we chose N=5 and M=7 as our optimal configuration.

A.10 More Results
The results of acceleration for different models are listed in Table 8. The results of the runtime on different models are listed in Table 9. The results of performance using other methods are listed in Table 10. Table 11 shows the average number of documents cached per round and the corresponding memory requirements for different retrieval document formats. Table 12 adds an independent experimental result on the combination of CICS+IDGR.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Methods</th>
    <th colspan="3">2WikiMultihopQA</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">StrategyQA</th>
    <th colspan="3">IIRC</th>
  </tr>
  <tr>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
  </tr>
  <tr>
    <td rowspan="6">Llama2-7B</td>
    <td>FLRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td><td>2.23×</td><td>2.37×</td><td>1.75×</td><td>2.27×</td><td>1.85×</td><td>1.53×</td><td>1.75×</td><td>1.49×</td><td>1.40×</td><td>3.08×</td><td>2.76×</td><td>2.10×</td>
  </tr>
  <tr>
    <td>FSRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FSRAG+Ours</td><td>2.04×</td><td>2.64×</td><td>2.32×</td><td>2.41×</td><td>1.88×</td><td>1.69×</td><td>2.36×</td><td>1.97×</td><td>1.79×</td><td>2.92×</td><td>2.88×</td><td>2.20×</td>
  </tr>
  <tr>
    <td>FLARE</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLARE+Ours</td><td>2.81×</td><td>2.74×</td><td>2.60×</td><td>3.12×</td><td>2.61×</td><td>2.31×</td><td>2.71×</td><td>2.27×</td><td>1.77×</td><td>3.54×</td><td>2.97×</td><td>2.89×</td>
  </tr>
  <tr>
    <td rowspan="2">DRAGIN</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>3.34×</td><td>2.77×</td><td>2.52×</td><td>4.09×</td><td>2.08×</td><td>2.09×</td><td>3.39×</td><td>1.98×</td><td>1.95×</td><td>4.49×</td><td>2.58×</td><td>2.58×</td>
  </tr>
  <tr>
    <td rowspan="6">Llama2-13B</td>
    <td>FLRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td><td>2.12×</td><td>2.36×</td><td>1.64×</td><td>2.29×</td><td>2.00×</td><td>1.54×</td><td>1.76×</td><td>1.62×</td><td>1.40×</td><td>2.70×</td><td>2.65×</td><td>1.83×</td>
  </tr>
  <tr>
    <td>FSRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FSRAG+Ours</td><td>2.11×</td><td>2.47×</td><td>1.79×</td><td>2.16×</td><td>2.05×</td><td>1.59×</td><td>1.74×</td><td>1.59×</td><td>1.44×</td><td>2.16×</td><td>2.65×</td><td>1.74×</td>
  </tr>
  <tr>
    <td>FLARE</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLARE+Ours</td><td>1.98×</td><td>2.24×</td><td>2.16×</td><td>2.42×</td><td>1.87×</td><td>1.80×</td><td>2.45×</td><td>1.71×</td><td>1.69×</td><td>3.58×</td><td>3.15×</td><td>3.01×</td>
  </tr>
  <tr>
    <td rowspan="2">DRAGIN</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>3.25×</td><td>3.21×</td><td>2.61×</td><td>3.97×</td><td>2.15×</td><td>2.01×</td><td>3.15×</td><td>1.98×</td><td>1.87×</td><td>4.25×</td><td>2.86×</td><td>2.65×</td>
  </tr>
  <tr>
    <td rowspan="6">Llama2-70B</td>
    <td>FLRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td><td>1.55×</td><td>2.24×</td><td>1.89×</td><td>1.53×</td><td>2.14×</td><td>1.76×</td><td>1.22×</td><td>1.64×</td><td>1.46×</td><td>1.75×</td><td>2.27×</td><td>1.92×</td>
  </tr>
  <tr>
    <td>FSRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FSRAG+Ours</td><td>1.61×</td><td>2.60×</td><td>2.33×</td><td>1.68×</td><td>2.48×</td><td>2.27×</td><td>1.64×</td><td>2.32×</td><td>2.20×</td><td>1.82×</td><td>2.83×</td><td>2.62×</td>
  </tr>
  <tr>
    <td>FLARE</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLARE+Ours</td><td>1.87×</td><td>2.09×</td><td>1.99×</td><td>2.33×</td><td>1.85×</td><td>1.81×</td><td>2.34×</td><td>1.69×</td><td>1.65×</td><td>3.68×</td><td>3.06×</td><td>2.98×</td>
  </tr>
  <tr>
    <td rowspan="2">DRAGIN</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>3.02×</td><td>2.67×</td><td>2.23×</td><td>3.37×</td><td>2.16×</td><td>1.98×</td><td>2.99×</td><td>1.65×</td><td>1.64×</td><td>3.31×</td><td>2.79×</td><td>2.71×</td>
  </tr>
  <tr>
    <td rowspan="6">Vicuna-7B-v1.5</td>
    <td>FLRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td><td>2.09×</td><td>2.21×</td><td>1.68×</td><td>2.29×</td><td>1.96×</td><td>1.61×</td><td>1.72×</td><td>1.61×</td><td>1.37×</td><td>2.81×</td><td>2.48×</td><td>1.84×</td>
  </tr>
  <tr>
    <td>FSRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FSRAG+Ours</td><td>2.44×</td><td>2.92×</td><td>2.19×</td><td>2.11×</td><td>2.00×</td><td>1.78×</td><td>1.69×</td><td>1.64×</td><td>1.53×</td><td>2.59×</td><td>2.49×</td><td>2.16×</td>
  </tr>
  <tr>
    <td>FLARE</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLARE+Ours</td><td>2.01×</td><td>2.09×</td><td>1.98×</td><td>2.11×</td><td>2.05×</td><td>1.93×</td><td>1.96×</td><td>1.88×</td><td>1.74×</td><td>2.65×</td><td>2.22×</td><td>2.19×</td>
  </tr>
  <tr>
    <td rowspan="2">DRAGIN</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>3.84×</td><td>2.76×</td><td>2.54×</td><td>3.97×</td><td>1.98×</td><td>2.00×</td><td>3.46×</td><td>2.25×</td><td>2.17×</td><td>4.40×</td><td>2.39×</td><td>2.31×</td>
  </tr>
  <tr>
    <td rowspan="6">Vicuna-13B-v1.5</td>
    <td>FLRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td><td>2.05×</td><td>2.25×</td><td>1.60×</td><td>2.22×</td><td>1.97×</td><td>1.51×</td><td>1.65×</td><td>1.55×</td><td>1.31×</td><td>2.66×</td><td>2.72×</td><td>1.82×</td>
  </tr>
  <tr>
    <td>FSRAG</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FSRAG+Ours</td><td>2.23×</td><td>2.31×</td><td>1.75×</td><td>2.15×</td><td>2.31×</td><td>1.66×</td><td>2.02×</td><td>1.78×</td><td>1.53×</td><td>2.35×</td><td>2.69×</td><td>1.92×</td>
  </tr>
  <tr>
    <td>FLARE</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>FLARE+Ours</td><td>2.04×</td><td>2.14×</td><td>2.06×</td><td>2.27×</td><td>2.07×</td><td>1.98×</td><td>2.05×</td><td>1.93×</td><td>1.89×</td><td>2.70×</td><td>2.24×</td><td>2.22×</td>
  </tr>
  <tr>
    <td rowspan="2">DRAGIN</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td><td>1×</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>3.57×</td><td>2.81×</td><td>2.55×</td><td>4.14×</td><td>2.29×</td><td>2.15×</td><td>3.25×</td><td>1.88×</td><td>1.82×</td><td>4.72×</td><td>4.00×</td><td>3.53×</td>
  </tr>
</table>

Table 8: The average speedup ratios before and after applying the IDR_2 framework.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Methods</th>
    <th colspan="3">2WikiMultihopQA</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">StrategyQA</th>
    <th colspan="3">IIRC</th>
  </tr>
  <tr>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
    <th>Pref.</th><th>Deco.</th><th>E2E</th>
  </tr>
  <tr>
    <td rowspan="2">Llama2-7B</td>
    <td>DRAGIN</td><td>3.71</td><td>12.55</td><td>19.31</td><td>5.50</td><td>21.61</td><td>31.83</td><td>6.81</td><td>28.99</td><td>41.79</td><td>4.73</td><td>25.31</td><td>33.75</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>1.18</td><td>6.07</td><td>9.56</td><td>1.49</td><td>12.58</td><td>18.26</td><td>2.19</td><td>18.22</td><td>25.71</td><td>1.18</td><td>12.29</td><td>16.20</td>
  </tr>
  <tr>
    <td rowspan="2">Llama2-13B</td>
    <td>DRAGIN</td><td>3.84</td><td>19.26</td><td>26.31</td><td>6.20</td><td>27.11</td><td>39.90</td><td>5.97</td><td>40.01</td><td>52.62</td><td>5.29</td><td>41.30</td><td>51.24</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>1.24</td><td>8.06</td><td>12.34</td><td>1.79</td><td>16.35</td><td>24.74</td><td>2.12</td><td>26.25</td><td>34.74</td><td>1.33</td><td>17.50</td><td>22.98</td>
  </tr>
  <tr>
    <td rowspan="2">Vicuna-7B-v1.5</td>
    <td>DRAGIN</td><td>4.56</td><td>16.40</td><td>24.07</td><td>5.90</td><td>16.83</td><td>27.66</td><td>5.11</td><td>19.20</td><td>28.11</td><td>5.63</td><td>33.63</td><td>43.38</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>1.30</td><td>7.62</td><td>11.61</td><td>1.59</td><td>9.58</td><td>15.60</td><td>1.69</td><td>11.35</td><td>16.59</td><td>1.47</td><td>17.71</td><td>23.08</td>
  </tr>
  <tr>
    <td rowspan="2">Vicuna-13B-v1.5</td>
    <td>DRAGIN</td><td>4.34</td><td>25.03</td><td>33.47</td><td>5.96</td><td>21.55</td><td>33.69</td><td>5.77</td><td>31.98</td><td>43.94</td><td>5.76</td><td>54.21</td><td>65.47</td>
  </tr>
  <tr>
    <td>DRAGIN+Ours</td><td>1.29</td><td>10.49</td><td>15.18</td><td>1.53</td><td>10.45</td><td>17.30</td><td>1.93</td><td>19.83</td><td>27.86</td><td>1.38</td><td>18.75</td><td>24.89</td>
  </tr>
</table>

Table 9: The average runtime before and after applying the IDR_2 framework.
<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Methods</th>
    <th colspan="2">2WQA</th>
    <th colspan="2">HQA</th>
    <th colspan="2">SQA</th>
    <th colspan="2">IIRC</th>
  </tr>
  <tr>
    <th>EM ↑</th>
    <th>F1 ↑</th>
    <th>EM ↑</th>
    <th>F1 ↑</th>
    <th>Acc. ↑</th>
    <th>EM ↑</th>
    <th>F1 ↑</th>
  </tr>
  <tr>
    <td rowspan="2">Llama2-7B</td>
    <td>FLRAG†</td>
    <td>14.10</td>
    <td>19.49</td>
    <td>15.10</td>
    <td>24.55</td>
    <td>63.50</td>
    <td>14.68</td>
    <td>18.83</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td>
    <td>16.00</td>
    <td>24.30</td>
    <td>13.70</td>
    <td>23.53</td>
    <td>62.30</td>
    <td>13.42</td>
    <td>17.38</td>
  </tr>
  <tr>
    <td rowspan="2">Llama2-13B</td>
    <td>FLRAG†</td>
    <td>21.90</td>
    <td>32.63</td>
    <td>21.80</td>
    <td>33.43</td>
    <td>66.70</td>
    <td>15.62</td>
    <td>18.93</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td>
    <td>20.40</td>
    <td>30.26</td>
    <td>17.50</td>
    <td>28.52</td>
    <td>64.60</td>
    <td>15.63</td>
    <td>19.41</td>
  </tr>
  <tr>
    <td rowspan="2">Vicuna-13b</td>
    <td>FLRAG†</td>
    <td>7.90</td>
    <td>21.97</td>
    <td>15.3</td>
    <td>28.12</td>
    <td>61.60</td>
    <td>9.22</td>
    <td>12.41</td>
  </tr>
  <tr>
    <td>FLRAG+Ours</td>
    <td>6.80</td>
    <td>12.83</td>
    <td>14.20</td>
    <td>25.48</td>
    <td>61.99</td>
    <td>8.39</td>
    <td>10.20</td>
  </tr>
</table>

Table 10: The performance comparison on different datasets with different models. The † represents the reproduction.

![Performance Analysis of LLaMA2-7B Model Tested on Nvidia 3090 GPUs](page_324_670_1102_246.png)

(a) LLaMA2-7B prefilling cost. (b) LLaMA2-7B decoding cost. (c) LLaMA2-7B end-to-end cost.

![Performance Analysis of LLaMA2-7B Model Tested on Nvidia H800 GPUs](page_324_920_1102_246.png)

(d) LLaMA2-7B prefilling cost. (e) LLaMA2-7B decoding cost. (f) LLaMA2-7B end-to-end cost.

Figure 9: The speed comparison under different number of retrieved documents.
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">Number of retrieved documents</th>
  </tr>
  <tr>
    <th>3</th>
    <th>5</th>
    <th>7</th>
  </tr>
  <tr>
    <td>DRAGIN</td>
    <td>3<br>0.24GB</td>
    <td>5<br>0.41GB</td>
    <td>7<br>0.57GB</td>
  </tr>
  <tr>
    <td>DRAGIN+IDR<sub>2</sub></td>
    <td>4.81<br>0.39GB</td>
    <td>8.08<br>0.66GB</td>
    <td>11.52<br>0.95GB</td>
  </tr>
</table>

Table 11: Comparison of memory cost for LLaMA2-7B on 2WikiMultihopQA.

<table>
  <tr>
    <th rowspan="2">CICS</th>
    <th rowspan="2">IGPG</th>
    <th rowspan="2">IDGR</th>
    <th colspan="2">LLaMA2-7B</th>
  </tr>
  <tr>
    <th>EM ↑</th>
    <th>F1 ↑</th>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>22.5</td>
    <td>28.68</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>-</td>
    <td>20.3</td>
    <td>26.88</td>
  </tr>
  <tr>
    <td>-</td>
    <td>✓</td>
    <td>-</td>
    <td>22.4</td>
    <td>28.51</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>-</td>
    <td>20.2</td>
    <td>26.78</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>✓</td>
    <td><b>25.8</b></td>
    <td><b>33.26</b></td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>25.4</td>
    <td>33.17</td>
  </tr>
</table>

Table 12: Analysis of the impact of different modules on 2WikiMultihopQA based on DRAGIN.

Reviewer paper summaries: This paper proposes a model-agnostic acceleration approach for Adaptive-RAG, IDR2, eliminating repetitive representations of overlapping retrieval content and redundant autoregressive generation processes.
Summary of strengths: 1. This paper points out the phenomenon of high-degree overlapping between the preceding retrieval content and the succeeding one.

2. The authors propose Cross-Iterative Cache Sharing, which avoids encoding repetitive content and improves efficiency at the prefilling stage.

3. Since the document's reusable representation contains information from other documents, the authors propose Instruction-driven Deduplication Guidance Reinforcement, which helps LLMs focus on relevant content.
Summary of weaknesses: 1. In Figure 1(b), the authors analyze document overlap between the first and later retrievals. A boxplot may be more intuitive. Besides, I recommend that the author analyze the document overlap between previous rounds and the last round (e.g., round 1&2 vs. round 3). 

2. In Section 3.2,  I have a doubt, as the author says, that the representation of a document contains information from documents. The authors design explicit instructions to guide the LLM to ignore information from other documents. However, as the number of iterations increases, the amount of other document information a document may contain adds up. For example, document #12368478 contains information from #5881721 and #12415199, while document  #12415199 contains information from #5881721 and #10028469. How to deal with this problem?

3. In Section 3.3, the content is somewhat hard to read, and the upper and lower indices of equations are also somewhat out of order. Besides, the hyperparameters N and M introduced in this section have a significant impact on performance, but the settings of these two hyperparameters are not seen later in the paper.

4. In Table 5, we can see the performance improvement of f1 and em is mainly derived from the IDGR. And, I am curious to see if better performance can be achieved when using CICS+IDGR.

5. Typos: in Table 5, IDGR should be corrected to IDRG.
Comments, suggestions and typos: Please refer to the previous section.
If the author can solve my concerns well, I will raise the rating.
Overall assessment scores: 3
Metareview score: 3.5
Metareview: This work proposes a method to accelerate inference for Adaptive-RAG by carefully analyzing duplicates in the framework. The method, IDR2, comprises three key components, cache sharing across iterations (CICS), filtering redundant information in cache (IDGR), and reusing the same fragments when decoding (IGPG). Experiments show that IDR2 achieves at least twice as fast speed up while maintaining the quality.
Summary of reasons to publish: * This work stands on careful analysis over Adaptive-RAG and identifies the issues in duplicates.
* The proposed approach sounds reasonable in eliminating duplicates, and experiments clearly show the gains in speed while maintaining the qualities.
* The engineering efforts will deserve attention from researchers working in the same field.