E$^2$-RAG: Towards Editable Efficient RAG by Editing Compressed KV Caches
Authors: Tongxu Luo
Wenyu Du
HanWen Hao
Min Zhang
Hao Yang
Benyou Wang
Venue: COLM 2025 (2025)
Keywords: Retrieval Augmented Generation
TL;DR: E$^2$-RAG efficiently edits compressed KV caches for knowledge updates, achieving 40x faster editing and 3x faster generation than standard RAG with minimal performance loss.

Abstract: Retrieval-Augmented Generation (RAG) demonstrates remarkable capabilities for enhancing the performance of Large Language Models (LLMs) by integrating external knowledge.
Standard RAG introduces additional computations due to the extra retrieved context.
To improve efficiency, recent studies propose compressing chunk tokens into compact forms, such as key-value (KV) caches.
However, maintaining these compressed KV caches in an updated state presents a significant challenge, undermining the primary goal of RAG: acquiring up-to-date knowledge.
In this work, we propose **E$^{2}$-RAG**, the first **E**ditable **E**fficient-**RAG** method designed to efficiently edit compressed KV caches for knowledge updates.
E$^2$-RAG features an encoder-decoder architecture similar to efficient RAG methods, along with an additional editor.
The encoder-decoder compresses chunk tokens into KV caches and generates responses.
The editor takes old KV caches and new knowledge tokens as inputs, enabling efficient updates to the KV caches.
To formalize knowledge updating, we define three operations: INSERT, DELETE, and UPDATE.
We create three sets of datasets for each operation.
Through extensive experiments, E$^2$-RAG achieves nearly **40x faster** editing compared to recomputing KV caches while maintaining **3x faster** generation efficiency than standard RAG, with a performance downgrade of 1%-5%.
We also conduct various ablation studies, including multi-turn editing, multi-chunk capability, and knowledge conflicts, to explore the capabilities of E$^2$-RAG.

Published as a conference paper at COLM 2025
E2-RAG: Towards Editable Efficient RAG by Editing Compressed KV Caches
Tongxu Luo Wenyu 2 Hanwen Hao
Min Zhang'
Hao Yang'
Benyou Wanglt
'The Chinese University of Hong Kong, Shenzhen
2The University of Hong Kong BBehaag University
"Huawei
tongxuluo@gmail.com wenyu. wenyu. http://ddduuaallll...oooo
wangbenyou@cuhkk edu.
Abstract
netrieval-Augmennt Generation (RAG) demonstrates remarkable capabil- ities for enhancing the performance of Large Language Models (LLMs) by integrating external knowledge. RAG introduces additional computations due to the extra retrieved context. To improve efficiency, recent studies propose compressing chunk tokens into compact forms, such as key-value (KV) caches. However, maintaining these compressed KV caches in an updated state presents significant challenge, undermining the primary goal of RAG: acquiring up-to-date knowledge. In this work, we propose E2-RAG, the first Editable Efficient-RAC method designed to efficiently edit compressed KV caches for knowledge updates in fast updating narios. E²-RAG features encoder-decoder architecture efficient RAG module, along with an additional editor. The encoder-decoder compresses chunk tokens into KV caches and generates responses. The editor takes old KV caches and new knowledge tokens as inputs, enabling efficient updates the KV caches. To formalize knowledge updating, we define three op- erations: INSERT, DELETE, and UPDATE. We create three sets of catasets for each operation. Through extensive experiments, E2-RAG achieves nearly 40x faster editing compared to recomputing KV caches while maintaining 3x faster generation efficiency than standard RAG, with performance downgrade of 1%-5%. We also conduct ablation studies such as multi-turn editing, multi-chunk capability, and knowledge conflicts, to explore the capabilities of E2-RAG. Our code, datasets, and models are available at https://git thub. com/ tong.
1 Introduction
Large Language Models (LLMs) exhibit impressive text generation capabilities; however, their knowledge is limited to the dataset used during pre-training. Retrieval-Augmented Generation (RAG) emerges
promising method to address this limitation by enabling the retrieval of new knowledge from an external database (Borgeaud et al., 2022; Lewis et al., 2020; Kandpal et al., 2023). This knowledge database typically consists of numerous short chunks. When a user inputs query, relevant chunks are fetched and combined with the user's query generate the answer (Lewis et al., 2020; Tao et al., 2021; Mao al., 2024).
A critical challenge in standard RAG systems lies in the substantial computational overhead introduced by processing additional retrieved chunks in input prompts (Tay et al., 2020; Yu et al., 2024). Recent efforts in efficient RAG focus on compressing text chunks into model-specific intermediate representations, such as key-value (KV) caches (Lu al., 2024; Sun et al., 2024). This compression enables LLMs to process knowledge through these compact representations rather than raw text, significantly accelerating generation speeds.
Equal Contributions. Corresponding Author.
1

---

Published as a conference paper at COLM 2025
However, world knowledge is constantly evolving. For scenarios such news and finance, which strongly rely on the timeliness of knowledge, maintaining up-to-date databases for RAG systems is crucial. However, for standard RAG systems, updating their databases with new world knowledge requires either manual curation or LLM-aasisted modifica- tion. Efficient RAG systems, in addition, must regenerate compressed representations from edited texts. This process incurs substantial computational costs, as evidenced by Wikipedia's 0.6 million daily edits (Wikimedia Foundation, 2025), which would require ap- proximately 32.8 A100 GPU hours for LLM-assisted modification and KV cache regeneration (see Appendix B).
In this work, we introduce E2-RAG, an editable efficient RAG framework that directly modifies compressed knowledge representations. Inspired by fundamental database oper- ations (ISO/IEC, 2023), we define three core edit operations: INSERT, DELETE, and UPDATE. The architecture incorporates an efficient RAG module and an editor module for modifying KV caches. The editor a frozen LLM equipped with trainable LoRA (Hu et al., 2021) and an editing embedding, which generates offset KV caches to update the old KV cache based the three operations.
We evaluate E2-RAG using modified versions of QA datasets (HotpotQA (Yang et al., 2018), ASQA (Stelmakh et al., 2022), SciQ (Welbl et al., 2017), SQuAD (Rajpurkar, 2016), Drop (Dua et al., 2019)) containing 10,000 knowledge edits per operation type. Figure 1 shows one example. We compare E²-RAG against two baselines: a standard RAG (Mao et al., 2024) and our efficient RAG module. The results indicate that E2-RAG is three times faster in generation than the standard RAG and achieves an editing speed that is 40 times faster than the efficient RAG that re-computes KV caches, with only a 1%-5% performance degradation across five QA benchmarks. Furthermore, we extend E2-RAG to multi-turn editing and multi-chunk inference settings and discuss knowledge conflicts.
Our analysis extends to multi-edit scenarios and multi-chunk inference, revealing insights into knowledge conflict resolution during DELETE and UPDATE operations. As RAG systems increasingly adopt compressed knowledge representations, E²-RAG provides a scalable solution for efficient knowledge maintenance.
Our contributions can be summarized as follows:
1. We focus on the efficient editing of RAG databases in compressed KV form and provide insights into this challenge.
2. We propose E2-RAG, the first editable efficient RAG architecture that achieves both efficiency in editing and inference.
3. We construct three datasets designed to evaluate three core operations: INSERT, DELETE, and UPDATE.
4. We conduct comprehensive experiments evaluating the editability and efficiency of the proposed methods, along with further ablations and discussions.
2 RAG and Knowledge Updating
RAG. RAG is "Retrieve-Read" framework (Gao et al., 2023). The standard RAG system first splits a document into n chunks {D;) which are then encoded into embedding vectors {E using an embedding model. These vectors and chunks construct database D (E, D When user inputs query q, the RAG system converts it into vector Eg using the same embedding model and computes its similarity with {E²). The top chunks with the highest similarity, [D,.... D , are retrieved (Karpukhin et al., 2020). These k chunks, along with q, are then input into the LLM to generate the response.
However, a significant drawback of standard RAG is the increased computational costs during the generation phase due to the additional RAG chunks integrated into the input prompts (Khandelwal et al., 2019; Izacard & Grave, 2020; Qin et al., 2023). Consequently, various efficient RAG approaches have been proposed to alleviate this overhead (Yan
2

---

Published as a conference paper at COLM 2025
et al., 2024). Among these, state-of-the-art methods compress chunk tokens into compact intermediate components in LLMs, such as KV caches (Li et al., 2024b), resulting in databases of the form D (E,, (K,, Vg)) This allows LLMs to bypass processing chunks in their raw text forms, significantly increasing generation speed compared to standard RAG. However, this also means that KV caches cannot be easily edited, which contradicts the goals of RAG. Recomputing these caches could lead to additional computational overhead.
Knowledge Update. Knowledge evolves continuously, and an active knowledge base system should track these changes. For example, Wikipedia (Wikimedia Foundation, 2025) maintains comprehensive editing log that records all editing history. Based on these log entries, we categorize them into three types: adding new knowledge, deleting obsolete knowledge, and replacing existing knowledge To align with basic SQL (ISO/IEC) 2023) operations, we refer to these three knowledge operations INSERT, DELETE, and UPDATE. For instance, consider a document about the Olympic Games. During a major Olympic event, the list of medalists will INSERT new entries, while some outdated items may be DELETE, and certain records will be UPDATE.
Knowledge Update for RAG. Knowledge updates are straightforward for standard RAG since it can integrate up-to-date texts internally. However, this process is not simple for efficient RAG approaches that utilize compressed KV chunks. The naive method involves using the updated text to regenerate these KV caches for future RAG generation. The standard RAG solution allows for easy editing but is insufficient for generation, while the opposite is true for the latter approach. Thus, instead of the above two solutions, we investigate whether we can directly edit the existing KV caches for efficient RAG.
Insert Input
Inserted KV
Relevant now ledge
Joe Biden was borni
20vember 1942,
Scranton, Pennsvlvania, USA.
Editor Modules
Serves as
Delete Input
INSERT
Deleted KV
Deleted Know ledge
Joe Biden is the President of the USA.
DELETE
Update Input
UPDATE
Updaled KV
Trump won 2024 U.S. election. the currect President Trump
Ouldeled Knowlledge
KV Cache
Serves replacement
Original Document
The United consists
Tokens of original document
Memory tokeus
50 statios... The current President the Unitad States loe Biden... The total area the United Staues 9,372,610 square kilometers.
Encoder (LLM LoRA)
Key of memory tokens
Value
tokens
Offiet of kee and value
Figure 1: The editor module. We train three separate editors for INSERT, DELETE, UPDATE re- spectively. And the offset KV caches appends to old KV caches but functions differently.
3 E2-RAG
In this section, we detail our method, E2-RAG, which includes its efficient RAG module for pretraining learn compression, fine-tuning for multi-chunk question answering, and the training of the E--RAG editor module.
We provide the statistics, and analysis of "Machine Learning" category in Appendix K.
3

---

Published as a conference paper at COLM 2025
3.1 Efficient RAG Module
E²-RAG employs encoder-decoder module for efficient RAG. The encoder pre-processee document chunks into compressed KV caches offline, which are then stored in a database. During online inference, the module retrieves the top-k KV caches from the database and feeds them to the decoder. The decoder generates responses based on these compressed KV caches. Our training process consists of two stages: pretraining and fine-tuning, Pretraining focuses on learning compression capabilities, while fine-tuning focuses on the question- answering task.
Pretraining. In this stage, we pretrain the efficient RAG module to acquire text compres- sion capabilities. We use frozen LLM, denoted OLLM, with trainable LoRA O as the encoder, and the same frozen LLM as the decoder. Consider the input text to- kens X. After embedding, they are concatenated with trainable memory embedding M IM2, mn), and m, E RD where d is the hidden size of the LLM. The outputs are the KV caches of M, == k2, .kk, and k; E Rixxxd the "key" of i-th attention, where is the number of layers of the LLM, is the number of attention heads, and d,, is the
LoRA
dimension of attention heads. V [1] 1/2 ,r), and Dj E Rhxxxxa is the "value" of i-th attention. Formally:
WE K, V) = encoder( [E, MI; OLLM, LoRA
(1)
Encuder)
(2)
where 1x is the one-hot vector of X, WE the embedding parameter of the LLM, and Rxxx xd is the embedding X. KV cache (K, V) is then passed the frozen decoder to reconstruct the original text X. The reconstruction process minimizes the cross-entropy loss to train the LoRA Uncoderr
1 -
E log P(x; XeriK, V; V:OLLM LORA
Encoder)
Fine-tuning. For fine-tuning, we adapt the pretrained LoRA @ to perform question answering based on relevant chunks (Di..... ,D,). For each chunk D;, we feed it to the encoder to obtain its KV cache, denoted as (K,, V,). We then concatenate these KV caches,
resulting in a combined KV cache denoted as (K1 Vı:)). Let the question be Q and the answer be A. The loss for the fine-tuning stage can be expressed as:
= L log P(A QrK ViAi OLLM Encoder)
(4)
It is important to note that, to prevent the position embedding of the "key" from becoming disordered (Lu et al., 2024; Sun et al., 2024), the position embedding of the "key" must be removed when using the encoder to process chunks offline. When feeding them into the decoder, the position encoding should be reapplied according to the sorting, Details on repositioning can be found in Appendix C.
3.2 Editor Module
When updating the database is necessary, E2-RAG first retrieves KV caches related to the new knowledge. The editor module then takes the new knowledge tokens inputs and produces the offset KV for INSERT, DELETE, and UPDATE operations. Although these operations share the same architecture, their learning objectives differ.

---

Published as a conference paper at COLM 2025
The Architecture for the Editor. Our editor is frozen LLM with trainable LoRA, designed to append, delete, and modify by processing the new knowledge into the offset KV cache (AK,AV), which represents the updates to the KV cache. Formally:
1
(5)
(AK,
editor (iitoor(E,C,r (),C), @
(6)
where C Rexd where < n, represents the trainable editing embedding, and (AK, AV) denotes the KV cache of C. The updated KV cache is obtained by concatenating the modifi-
cations to the original cache:
[K, [V,V]]
K V
(8)
To ensure that the edited KV caches can used alongside other KV caches for multi-chunk inference, propose maintaining queue of KV caches to store the K and V values from previous batches. During each batch, we randomly sample from this queue and mix the selected KV caches. The detailed algorithm is presented in Appendix C.2.
The editor is also trained to minimize the cross-entropy loss:
T log P(A Q K, V; LLM, (dittr)
==
Different Objectives for INSERT, DELETE, UPDATE. Due to their distinct goals, all three despit using the same editor architecture-have significantly different training objectives.
INSERT: This operation incorporates new knowledge. For a given question (related to the old or new knowledge), the decoder is first prefilled with the updated KV caches ([R, AK), (V,AV).. The query Qinsert of this question then attends to both the original knowledge in the caches and the newly appended knowledge. The attention mechanism computes the attention scores as follows:
Qinsert K (AKKT
[ AA = softmax
(10)
The resulting attention map [A, AA] assigns weights to the keys for both the original and knowledge, capturing their relevance to the query.
DELETE: In contrast to INSERT, the DELETE operation applies a "mask" to suppress specific segments of old knowledge. When question Qdelete relates to the knowledge that needs to be removed, the new AV acts as a "mask" through the attention mechanism, effectively erasing the corresponding content in D:
O = (ÁV + ΔADV)W?
(11)
where, O is the output of the attention mechanism and Wo is the parameter of the output projection of the attention mechanism.
This operation combines the functionalities of DELETE and INSERT, as it involves both removing outdated knowledge and introducing new information. However, unlike INSERT, the DELETE and UPDATE operations may lead to knowledge conflicts. We discuss this in Section 6.4.
UPDATE:
Editing Dataset Construction
For simplicity, we select documents with length bounded by 128 tokens (the length of single chunk) from five document QA datasets (Welbl et al., 2017; Yang et al., 2018; Dua et al., 2019; Rajpurkar, 2016; Stelmakh et al., 2022). We first split each document into sequence of sentences: {s|,- .. ,Sr}. J. For each document, we create a series of questions, where the sentence s; contains the knowledge necessary to answer the question Q
5

---

Published as a conference paper at COLM 2025
We select one sentence s; from the document {st,-.. ,un as the new knowledge while the remaining sentences {si, ". ""i.""""S-11S"1 1 Suf serve as the old context. When i j, the new knowledge I is necessary for answering Q. Conversely, when i + j, can only be answered using the old context. For both cases, we provide the new document to
INSERT.
Qwen2.5-7B (Yang et al., 2024) to generate responses to the questions.
DELETE. In contrast to INSERT, s represents the sentence to be deleted, and the full set {s ,n) constitutes the old full document. When i = =j, the sentence necessary for answering Q is deleted. Therefore, we use Qwen2.5-7B to input the remaining context {s1.-. Sj 1.5jil.'".. ,5 along with the question to obtain response indicating refusal to answer. For i + j, we generate normal response, similar to the INSERT operation.
UPDATE. Similar to DELETE, the full set {si, (p) serves as the old context. We use Qwen2.5-7B generate new knowledge sj that conflicts with sj and the corresponding question Q. When i j, the question Q is based on Sy when _ j, the questions are based on the old knowledge. The former requires the new knowledge 6;; otherwise, the answer would be incorrect.
The data construction pipeline and examples of INSERT, DELETE, and UPDATE are provided in Appendix D. In the training set, both i and i / account for 50% ensuring that the model learns insert new knowledge while retaining the old knowledge. In the test set, these two cases are separated to evaluate the model's ability to independently retain old knowledge and acquire new knowledge.
Experiments
Setup. We construct the data following the methods introduced in Section Particularly, we use three datasets of HotpotQA (Yang et al., 2018), ASQA (Stelmakh et al., 2022) and Drop (Dua et al., 2019) for both training and testing, with two extra test sets of SciQ (Welbl et al., 2017) and SQuAD (Rajpurkar, 2016) for out-of-distribution (OOD) evaluation. We use Llama3.2-3B, Llama3. 1-BB and Qwen2.5-7B (We present the results in Appendix H.) to train our models (Dubey et al., 2024; Yang et al., 2024). We train three separate editors for the three different types of editing operations. Training details, including hyperparameters, are provided in Appendix E. To ensure reliable evaluation, we adopt the Match metric, which checks whether the generated answer contains the golden answer as an exact match (Rau et al., 2024). This is particularly relevant since both the decoder in our method and the baseline standard RAG are frozen LLMs.
Baselines. We introduce two baselines as introduced in Section 2, standard RAG that directly updates documents in text form and our efficient RAG module that re-computes updated KV caches from edited text. As the two baselines require the edited full context, come up with two setups.
(1) LLM to Edit Text In this baseline, we use an LLM to edit the old chunk based on the knowledge and then answer the question. The LLM generates an updated context by inserting, deleting, or updating parts of the old chunk. While this approach can achieve reasonable performance, it lacks efficiency as it requires regenerating the entire context.
(2) Golden Edit For the INSERT operation, the golden edit baseline assumes access to the complete and up-to-date information, represented by the full set of sentences {s),-.. Sn)- This serves as an upper bound for performance, as directly provides the model with the ideal context for answering the question Q. For DELETE and UPDATE, we annotate the editing results by humans of a subset from HotpotQA. The results can be referred to in Appendix F.
5.1 Results on Editing Efficiency
We assess the KV cache editing speed in Figure When using the Llama3.1-80 model with batch size of 16, our editor module achieves nearly 40x speedup compared to re-compute

---

Published as a conference paper at COLM 2025
KV caches (LLM to edit text + compress to new KV caches). This significantly reduces the time required for massive updates to the RAG database, as we perform editing at the KV cache level rather than using an LLM to edit text. We also include experiments document QA without editing in Appendix G, and our efficient RAG module also achieves 3x speedup in response generation than standard RAG. We next evaluate the performance for three operations separately.
original
5.2 Results on INSERT Operation
Table 1: Results of INSERT operation on five document QA benchmarks and their averages. We use "Old" and "New" to indicate whether the knowledge involved in the question is added through the INSERT operation or is inherent the old chunk. The results indicate that our method achieves nearly lossless in inserting. The best results are in bold and the second best are with underscore.
Method
HotpotQA Old New (1) Old (1) New (?) (ld New (?) Old (f) New (1) Old (f) New (1) Old (?) New (1)
ASQA
SciQ
SQUAD
Drop
Avg.
Standard RAG 3B
golden edit
77.54
80.91
76.53
78.53
89.29
89.86
85,86
87.22
56.62
60.13 58.23
77.17
79.33
LLM to edit text 73.19
74.35
74.18
75.46
88.93
89.49
83.93
85.42
56.
75.33
76.63
F-RAG w/o
(Efficient RAG
golden edit
81.16
79.09
76.53
79.14
86.79
85.51
78.57
80.74
74.75
71.88
79.56
79.27
LLM edit text
76.81
7273
76.53
75:46
84.64
82.61
77.71
76.69
69.65
58.54
77.09
73.11
F'RAG 3B
78.99
87.27
77.00
85.89
82.50
86.59
75.43
82,90
74.54
79.11
77 69
84.33
Standard SB
golden edit
76.81
77.27
74.65
75.46
88.93
90.58
89.21
89.92
80.65
79.50
82.05
82,55
edit text
74.64
75.45
72.77
72.39
88.21
86.96
88.64
68.66
78.41
79.11
80.53
80.46
E-RAG wfo Editor (Efficient RAG SB)
golden edit
82.61
82.73
85.45
87.50
91.43
89.13
85.79
87.04
78.57
78.52
84.77
84.98
LLM edit text
84.78
81.82
82.16
79.75
90.00
86.59
84.07
82.18
76.37
71.20
83.48
80.31
E-RAG 8B
83.33
81.82
80.28
85.28
87.50 47.50
87.6 37.68
82.79
83.53
82.28
81.33
83.24
83.93
We report our results on the INSERT opera- tion in Table 1. At both the Llama 3B and 8B sizes, our E2-RAG achieves first or sec- ond place
3000
veridets 110
most benchmarks. Notably, the average score on "New" is 84.33 (first) in the 3B setting and 83.93 (second) in the 8B setting. The results on "Old" are slightly lower. We hypothesize that the KV cache for new knowledge is added after the original KV caches, bringing it closer to the query tokens, which causes the model to direct more attention toward the new knowledge. Among five benchmarks, we do not observe significant differences, indicating that our approach can also adapt to the two OOD benchmarks, SciQ and SQuAD. We find that Efficient RAG sometimes outperforms standard RAG on certain benchmarks. One possible reason for this is the vulnerability of the current standard RAG system when presented with irrelevant or misleading information. In contrast, compressing the context into KV caches effectively extracts and simplifies information, omitting less important details, which enhances the model's retrieval ability (Cheng et al., 2024).
15P Cort
2000
Tabing Time
1000
1000
2000
4303
5360
8366
30302
2200
4303
5360 8000 10000
Eosting
(a) Editing speed of 3B (b) Editing speed of 8B
Figure 2: Editing Time for 3B/8B LLMs
5.3 Results on DELETE Operation
The results for the DELETE operation are presented in Table 2. Unlike INSERT, "Rm" represents questions that require the removed knowledge. Therefore, a lower "Rm" value indicates a higher degree of knowledge removal. From the table, we observe that both the 3B and 8B models rank first or second in most benchmarks. For example, the average for "Rm" in the 3B model is 28.86 (second), while the average for the 8B model is 21.68 (first). However, the
7

---

Published as a conference paper at COLM 2025
Table 2: Results of DELETE operation on five document QA benchmarks and their averages. During the experiments, we apply the DELETE operation on the context in QA as described in the paper. We use "Old" to indicate that the QA only involves knowledge that not deleted. "Rm" denotes QA involving deleted knowledge, where lower scores indicate more effective deletion.
HotpotQA
ASQA
SeiQ
SQuAD
Drop
Avg.
Method
Old (1) Rm
Old (1)
(1) Old (f)
Rm (1) Old (1) Rm (
Rm ( Old (1) Rm
Standard RAG
LLM to edit text 70.27
71.62
20.69
87.24
48.21
78.92
29.73
53.56
72.32
32.17
E-RAG w/o Editor (Efficient RAG 3B)
LLM to edit text
83.11
25.81
78.38
16.38
86.21
31.25
77.54
30.66
69.75
33.33
79.00
27.49
E-RAG
78.38
17.24
39.29
30.57
Standard RAG
LLM to edit text
70.27
27.42
70.72
13.79
87.24
34.82
88.74
27 30
73.13 13
6.67
78.02
22.00
E-RAG w/o Editor (Efffciient
RAG 8B)
LLM edit text
87.16
32.26
86.94
15:52
25.89
87.77
29.30
78.29
23.33
85.69
25.26
E-RAAG 8B
28.38
30.65
79.28
6.03
85.17
21.43
80.17
25.30
72.24
25.00
79.05
21.68
performance on "Old" slightly lower, suggesting that direct concatenation of KV caches may not be the optimal solution to retain old knowledge. We leave this for future study.
5.4 Results on UPDATE Operation
Table 3: Results the UPDATE operation on five benchmarks. "Rp" (replace) denotes QA involving the updated knowledge.
HotpotQA
ASQA
SciQ
SQuAD
Drop
Avg.
Method
(l) Kp
(ld (p Old ()) old (p()) Old Rp Old (T) (T)
Standard RAG
LLM to edit text
26.27
57.20
64.17
66.25
52.60
61.81
60.89
63.81
49.30
72.30
60.65
64.27
ERG w/o Editor (Efficient
3B)
LLM to edit text 76.27
61.86
65.69
71.94
53.64
64.19
57.96
72.38
51.17
74.51
60.95
68.98
ERR 3B
84.75
83.90
68.47
79.86
55.13
78,90
60.79
79.74
48.83
71.36
63.59
78.75
Standard RAG 8B
LLM to
84.32
67.78
64.78
73.39
E-RAG w/o Editor (Efficient RAG 8B)
LLM to edit text 84.32
74.15
70.42
25.14
60.77
68.35
66.63
24.40
63.38
79.32
69.10
24.27
E-RAG
84.75
57.50
80.53
62.00
83.27
81.83
For the UPDATE operation,we present the results in Table 3. As discussed earlier, UPDATE is combination of INSERT and DELETE, requiring both the removal of outdated knowledge and the insertion of new knowledge. This operation is inherently more challenging as it introduces potential knowledge conflicts. Notably, E2-RAG performs well on both "Old" and "New" (which require updated knowledge) for both the 3B and 8B Llama models.
6 Ablation and Discussion
To provide further insights into knowledge updating for RAG, we conduct a series of ablation studies multi-turn editing and multi-chunk capability for INSERT. Additionally, we discuss knowledge conflicts that arise during the editing process for DELETE and UPDATE.
6.1 Multi-turn Editing
In practical applications, it is often necessary to update specific chunk multiple times. Therefore, we conduct experiments on multi-turn editing using data from the INSERT op- eration on both the 3B and 8B models. We present the average results in Appendix I. Our findings indicate that as the number of editing iterations increases, both the old knowledge and the newly inserted knowledge from the first round gradually fade. Consequently,
8

---

Published as a conference paper at COLM 2025
recommend reconstructing the chunk from the original text after maximum of three editing operations to maintain information integrity.
6.2 Multi-chunk Ability with Edited Chunks
Real-world applications often require mod- els to retrieve and utilize information from multiple chunks to answer questions. As mentioned in Section 3, our editor is trained with chunk queue to enhance its capabil- ity of handling multiple chunks effectively. To systematically evaluate this ability, we conduct experiments by introducing vary- ing amounts of noisy chunks into the test set, ranging from 0 9. This evaluation covers scenarios where the total number of chunks varies from 1 to 10, corresponding to the Top-K values used in common RAG settings (Mao et al., 2024). This setup allows us to assess how well the editor maintains knowledge consistency while retrieving relevant information in multi-chunk settings. We present the average results across five benchmarks in Figure 3, with detailed evaluation results available in Appendix J. The results indicate that as the number of chunks increases, both "Old" and "New" knowledge performance slightly decreases. This decline occurs because the increase in the number of chunks leads to more frequent errors in information retrieval. Specifically, as more chunks are involved, the model faces greater challenges accurately retrieving relevant information. By comparing the results of the 3B and 8B models, we observe that the multi-chunk ability of the 8B model is significantly better than that of the 3B model, suggesting that multi-chunk capability is correlated with model size.
Avstaee od
Aratigee oid
6.75
0.775
0.750
0.725
0.770
0.49
Number Chunks
Number Chunis
(a) 3B
(b) 8B
Figure 3: Multi-chunk Ability on (a) 3B size and (b) 8B size.
6.3 Robustness on Insufficient Updating
A potential issue with updating only the top-k relevant entries in the database is that other outdated entries (the remaining it k) k) are left unchanged. These outdated entries may degrade model performance. We conduct method in this scenario. Specifically, we select approximately 100 questions from the test set of Squad for the three operations and duplicate the associated information chunk five times. Subsequently, the ratio of edited (updated) chunks could be varied systematically from 0.2 (1 out of 5) 1.0 (all 5 out of 5). This would allow for an examination of model performance under varying degrees of information conflict. We provide the results for Llama-3.2 3B and Llama-3.1 8B in Table 4.
experiment to investigate the robustness of our
Table 4: Performance under different ratios of edited knowledge chunks. The results demonstrate a reasonable robustness to incomplete updates.
Ratio of Edited Chunks
Model Size
Operation
0.2
0.4
0.6
0.8
1.0
INSERT (T)
0.40
0.58
0.61
0.60
0.63
3B
DELETE (L)
0.31
0.34
0.22
0.19
0.20
UPDATE (1)
0.75
0.77
0.82
0.82
0.82
INSERT (T)
0.69
0.71
0.75
0.81
0.81
8B
DELETE (L)
0.38
0.39
0.25
0,16
0.14
UPDATE (1)
0.79
0.81
0.82
0.86
0.86
Firstly, all the models and operations demonstrate a similar trend where performance slightly drops when the ratio of edited chunks is reduced from 1.0 to 0.2. This is reasonable

---

Published as a conference paper at COLM 2025
result because when the number of edited chunks decreases, the model can be influenced by the unedited chunks, which may confuse the model and lead to incorrect answers. However, to surprise, scenario. Especially in the UPDATE operation, even when only 20% of the chunks are updated, the model can still notice the change and answer correctly. For instance, the performance of the 3B model remains at 0.75 when the ratio of edited chunks is only 0.2, compared to 0.82 when the ratio is 1.0.
method also shows a certain level of robustness in this
We acknowledge that updating by selecting the top-k relevant outdated entries is indeed naive method, it can lead to the issue of incomplete updates, as you mentioned. However, since our work is the first to explore how to efficiently keep the RAG database up-to-date, we believe there will be future follow-up work to better address this challenge.
6.4 Knowledge Conflicts
The DELETE and UPDATE operations may sult in knowledge conflicts; however, our model still performs decently. Therefore, it is important to investigate the underlying reasons. For the DELETE operation, we select two types of questions: one related to old knowledge that can be answered correctly ("Old"), and another solely related to the re- moved knowledge ("Rm"), which cannot be answered correctly by the model. We plot the attention map of the question over the edited chunk in Figure 4. We observe that when the question pertains to the removed knowledge, the attention to AK significantly increases. We conduct the same experiment for the UPDATE operation, and the results are consistent with those of the DELETE operation. We hypothesize that in DELETE, when the question requires information from offset KV caches, ÁV acts as a "mask" to nullify certain portions of the old knowledge's attention "value", while in UPDATE, it functions to replace the old knowledge.
(a) "Old"
(b) "Rm"
Figure 4: Attention map of the DELETE opera- tion, (a) The question is only related to the old knowledge. (b) The question only related to the removed knowledge. The red box in both sub-figure means the attention of AK.
7 Conclusion and Future Work
This work addresses the challenge of efficient editing in compressed KV caches of RAG. We propose E2-RAG, the first editable efficient RAG architecture that balances both editing and inference efficiency. To evaluate our approach, we construct three datasets specifically designed to assess the basic operations of INSERT, DELETE, and UPDATE. Our comprehensive experiments demonstrate the editability and efficiency of the proposed methods, along with further ablations and discussions that provide valuable insights into the challenges associated with this problem. Our findings highlight the potential for further advancements in the editability and efficiency of RAG.
Acknowledgment
This work was supported by Huawei, the Shenzhen Science and Technology Program 000002200110000010022,,, Shenzhen Doctoral Startup Funding (RCBS202100800933000) Tianyuan Fund for Mathematics of National Natural Science Foundation China (NSFC) (12326608), Shenzhen Science and Technology Program (Shenzhen Key Laboratory Grant No. ZDSYS20230626091122000 and Shenzhen Stability Science Program 2023, Shenzhen Key Lab of Multi-Modal Cognitive Computing.
10

---

Published as a conference paper at COLM 2025
References
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed- inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, Pp. 1-15. IEEE, 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2006-222444 PMLR, 2022.
Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning to refine queries for retrieval augmented generation. arv preprint ar Xit:2444.006100, 2024.
Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. xrag: Extreme context compression for retrieval-augmented genera- tion with one token. Xro preprint arXio:2405.1379) 137992 2024.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. ar Xiv preprint ar Xiv: 144888 2023.
Tri Dao. Attetion: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. ar Xiv preprint ar axXt:2104.008144 2021.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. Calibrating factual knowledge in pretrained language models. arXiv preprint ar frXi::2210.0339, 2022.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. Xiv preprint Xf::11990300011 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint ør Xix:::247.217720 2024.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: graph rag approach to query-focused summarization. ar Xiv preprint ar Xxio:2404.161.33 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. Xrx preprint wwXio:2312.100997 2023.
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Q Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. arXiv preprint arXiv:2307 06945, 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint srXiv:2106.0966
2021.
ISO/IEC. Information Technology Database Languages SQL Part 1: Framework (SQL/Frame- work). 6 edition, June 2023. International Standard published [60.60].
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXio:20122045844 2020.
Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska. Piperag: Fast retrieval- augmented generation via algorithm-system co-design. arXiv preprint ar XxXi::400300567, 2024.
11

---

Published as a conference paper at COLM 2025
Dongwon Jung, Qin Liu, Tenghao Huang, Ben Zhou, and Muhao Chen. Familiarity-aware evidence compression for retrieval augmented generation. arXiv preprint wxXv:22009112466, 2024.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pp. 15696-15707. PMLR, 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. arxiv preprint ar arXtt:2004.04996, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. arviv preprint a:Xit:1911.0017,, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
Zhen Li, Jing Tang, Deqing Zou, Qian Chen, Shouhuai Xu, Chao Zhang, Yichen Li, and Hai Jin. Towards making deep learning-bass vulnerability detectors robust. arXiv preprint arXivv2108.0066 006669, 2021.
Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval augmented generation or long-context lims? comprehensive study and hybrid approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 881-893 2024a.
Zongqian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compres- sion for large language models. arv preprint arXiv:2408.03099 2024b.
Stephanie Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. Xiv preprint arXiv:2109.07955 2021.
Jerry Liu. Llamalndex, 11 2022. URL
Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and Yiming Qian. Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction. Xrv preprint ar arXiv:2310.15556 2023.
Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang, Turborag: Accelerating retrieval-ugmmentedd generation with precomputed kv caches for chunked text. arXiv preprint ar arXio:2410.07590, 2024.
Qianren Mao, Yangyifei Luo, Jinlong Zhang, Hanwen Hao, Zhilong Cao, Xiaolong Wang, Xiao Guan, Zhenting Huang, Weifeng Jiang, Shuyu Guo, et al. Xrag: examining the core- benchmarking foundational components in advanced retrieval-augmented generation. ar Xiv preprint ar Xiv:244, 155599 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
12

---

Published as a conference paper at COLM 2025
Xupeng, Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerat- ing generative large language model serving with tree-based speculative inference and verification. ar Xiv preprint ar xxtm:23055009711 2023.
Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2024.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quanti- zation through weight equalization and bias correction. In Procediigs of the IEFF/CVF International Conference Computer Vision, pp. 1325-1334 2019.
Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li. End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions. Xrv preprint ar arXiv:2311.09008, 2023.
P Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint ar axXic:1606.05250, 2016.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context retrieval-augmenned language models. Transactions of the Association for Computational Linguistics, 11:1316-1331, 2023.
David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. Context embeddings for efficient answer generation in rag. arXiv preprint arXiv:2407,092 2024.
Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Ganesh Ananthanarayanan, Ravi Ne- travali, and Junchen liang. Ragserve: Fast quality-aware rag systems with configuration adaptation. arXiv preprint ar arxi::2412.10544 2024.
Sara Rosenthal, Avirup Sil, Radu Florian, and Salim Roukos. Clapnq: C Chesivel ong- nswers from p assages in natural questions for rag systems. Transactions of the Association for Computational Linguistics, 13:53-72, 2025.
Alireza Salemi and Hamed Zamani. Evaluating retrieval quality in retrieval-augmenned generation. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2395-2400, 2024.
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. Raptor: Recursive abstractive processing for tree-organized retrieval. arviv preprint ar Xrii:22411.8855 2024.
Rana Shahout, Cong Liang, Shiji Xin, Qianru Lao, Yong Cui, Minlan Yu, and Michael Mitzenmacher. Efficient inference for augmented large language models. arxiv preprint arXiv:2410.18248,, 2024a.
Rana Shahout, Cong Liang, Shiji Xin, Qianru Lao, Yong Cui, Minlan Yu, and Michael Mitzenmacher. Fast inference for augmented large language models, 2024b. URL https: www.xiv.org/bss///. 18244
Kaize Shi, Xueyao Sun, Qing Li, and Guandong Xu. Compressing long context for enhancing rag with amr-based concept distillation. arviv preprint ar Xio::2405.03385 2024.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Chang. Asqa: Factoid questions meet lorg-form answers. arxiv preprint arXiv:2204.06092, 2022.
East Sun, Yan Wang, and Lan Tian. Block-attention for efficient rag. arXio preprint arXio:2409.15355, 2024.
Chongyang Tao, Jiazhan Feng, Chang Liu, Juntao Li, Xiubo Geng, and Daxin Jiang. Building an efficient and effective retrieval-based dialogue system via mutual learning. arviv preprint ar Xiv:22 10.00159, 2021.
13

---

Published as a conference paper at COLM 2025
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arxiv preprint ar Xiv:200 Xi0:2001.040066 2020.
Vinh Tong, Dat Quoc Nguyen, Trung Thanh Huynh, Tam Thanh Nguyen, Quoc Viet Hung Nguyen, and Mathias Niepert. Joint multilingual knowledge graph completion and alignment. arxiv preprint ar arXiv:2210.0892. 2022.
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 4126-4142. PMLR, 2021.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. Xrviv preprint arXiv:1707.062000 2017.
Wikimedia Foundation. Wikimedia statistics, 2025. URL
all all-wikipedia-projecee Accessed: 2025-01-30.
Wolf. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint ar #Fii:1910.03771,22
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged re-
sources to advance general chinese embedding, 2023.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: with compression and selective augmentation. arXiv preprint arXtt:2310.04408, 2023.
Improving retrieval-augmented lms
Mengyi Yan, Weilong Ren, Yaoshu Wang, and Jianxin Li. retrieval-auggmened framework for tabular interpretation with large language model. In International Conference on Database Systems for Advanced Applications, pp. 341-356. Springer, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint ar axXiv:2412.15115 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdi- nov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint wxXo:1809.009000 2018.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. Xrv preprint Xiv::230. 13172, 2023.
Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. Evaluation of augmente generation: A survey. In CCF Conference on Big Data, pp. 102-120. Springer, 2024.
A Related Work
As the context length increases (Li et al., 2024a), the inference cost of the RAG model increases significantly (Ram et al., 2023; Salemi & Zamani, 2024). To enhance the efficiency of RAG, efficient RAG methods have emerged, aiming to reduce time overhead by optimizing the entire getrieval-generatinn workflow.
In earlier studies, numerous approaches have been developed to enhance the efficiency of RAG. The reduction of input context as natural method was the first to be validated. Initially, various text summarization techniques (Xu et al., 2023; Jiang et al., 2024; Edge et al., 2024; Chan et al., 2024) were introduced to minimize information loss. Knowledge distillation and filtering techniques are employed in (Shi et al., 2024; Liu et al., 2023; Sarthi et al., 2024) and (Jung et al., 2024) to compress lengthy texts and preserve essential knowledge. Furthermore, some works (Shahout et al., 2024a; Ray et al., 2024) controlled the effective length of the input by addressing information redundancy. Although these
14

---

Published as a conference paper at COLM 2025
approaches effectively reduced context length, they resulted in semantic loss and poor scalability.
Recent studies (Leviathan et al., 2023; Shahout et al., 2024b; Miao et al., 2023; Nagel et al., 2019; Aminabadi et al., 2022) have aimed to enhance efficiency by focusing on the large models themselves. We emphasize works that compress the context into LLM intermediate components such as embeddings (Mu et al., 2024; Chevalier et al., 2023; Ge et al., 2023; Li et al., 2024b) and KV caches (Lu et al., 2024; Sun et al., 2024). Although these methods are effective and incur minimal loss, they are challenging to update. In contrast to prior research, our focus is on editable efficient RAG.
Previous works (De Cao et al., 2021; Dong et al., 2022; Meng et al., 2022) on knowledge editing focus on inserting updating knowledge in LLMs. And many works (Li et al., 2021; Tong et al., 2022; Yao et al., 2023; Ullah et al., 2021) on machine unlearning aim to remove specific knowledge from LLMs. Our work applies three types of knowledge operations INSERT, DELETE, and UPDATE -o the RAG database, ensuring that it remains up-to-date.
B Details of GPU hours in editing
We choose Llama-3.1-8B (Dubey et al., 2024) as the editor, utilizing vllm (Kwon et al., 2023) and flash-attention-2 (Dao, 2024) to accelerate inference with bfloatl6 precision. The token throughput approximately 1300 tokens/s. Each text chunk consists of 128 tokens, and the total sum of input and output tokens is 256. Therefore, the number of tokens to be processed
is T = 0.6 106 x 256, and the required time is S 1300600600 32.8 hours. So, the total A100 GPU time required is 32.8 hours.
T
C Details of Reposition and Chunks Queue
C.1
Reposition
One challenge in using KV caches as chunks for RAG is the confusion caused by the position embedding of the "key". To address this, we remove the position embedding of the "key" when storing it in the database and reassign the position embedding to the key before pre-filling. We refer this process "Reposition".
Consider the "query" at position as Qm Rd, and the "query" with position embedding as Qm. RoPE adds position encoding to Qm through the following formula:
Qm
Qm X COSO,m
+ Rotary Half Qm) & sin
(12)
90
cos m00
-1
sin mág sin m20
91
cos m00
92
cos m0;
43
sin m0
Q3
COS m0
+
sin m0
(13)
qu
COS m0 1/2 cos m0
Qd
sin mQ 1/2-11 sin m0
qu
4a
For the "key" at position m, Kmy position encoding is added in the same manner to obtain Km. However, instead of adding the position encoding, our goal is to recover Km, given m and Km. First, have the formula for RoPE:
Km =
Km 2 coSo.m + Rotary Half(Kn) (m)) & sin
(14)
(15)
15

---

Published as a conference paper at COLM 2025
Additionally, we have the basic properties of Rotary and sine-cosine functions:
Km -Rotary Half (otary Rotary Half( (m))
(16)
1 =
sing,, + cos@.M
(17)
By solving the system of these three equations, we can obtain:
Km
Km @ coS@ Rotary Half((m) @ sino..
(18)
In this way, we can remove the position encoding from the "key".
C.2 Chunks Queue
Algorithm 1 Chunks Queue In Editor Training
1: Input: Batch size B, queue size 9, queue Q
2: Initialize an empty queue Q with maximum size
3: for each batch b do
4: Get the compressed KV (Ks, Vo) and (AK,AV) from editor, and update to (K,, Vb)
5: Randomly sample subset (Ki.r, Vir) C
6: Construct multi-chunk
KV caches
and random
sort:
Random
Sort[(K,, Vi),..., (K,, Vt), (KG, Vo]]
7:
Enqueue (R,, V/) and (Krr Vτ) into Q
8: if queue size > q then
Remove the oldest KV caches from 10: end if
9:
11: end for
D Examples of Three Type of Editing Operations
(b) Split to Sentences
(c1) INSERT Data
(c2) DELETE Data
Old
Old Decument:
Document:
Document:
(n) O Original Document QA
Delated Knowledge:
Deleted Kaowledger
Knowledgge
Decament::
Response:
Th Rames new studium was not ready for then when they
The documezt coss not provile ... connet answer tee gusdion.
insiaaly arrivcal Masouuii
Quedtion
Question
Question
againet the New rilians
heir tarst gume in St. Louis
Respouse
Reggannon
Respanse
Suints played
September 10, 1995 with
Rama 17: The
Decument:
(33 UPDATE Data
World
Ponthers,
00 Document
W hot was the first gome the
Question::
Knowledges
LLM
LLM
Rarts playea in St. Lous?
Updated
Fron: the document we can
Response:
Question
Qustton
Knowledge
Question 1I
September 10, 1993.
sre se the anewer
Response
Requosse 1
Rasgonoe
Figure 5: The pipeline of our data creation. (a) We use the original document QA datasets, and (b) we first split the context into sentences. (cl) For the INSERT operation, we randomly select one of the sentences as new knowledge and reuse the question from the original document QA. (c2) For the DELETE operation, when the removed knowledge related to the question, we use an LLM to generate response that refuses to answer the question. (c3) For the UPDATE operation, we only use the context from the document QA, and then create conflicting new knowledge using LLM. On one hand, use this new knowledge to generate question and its corresponding response; on the other hand, we use other sentences to generate question unrelated to the new knowledge and its response.
16

---

Published as a conference paper at COLM 2025
An Example of Insert Operation
Old Document:
Founded in 1934, the company owns and/or operates 30+ luxury hotels and two river cruise ships in six countries, primarily under its Oberoi Hotels & Resorts and Trident Hotels brands. The Oberoi family is an Indian family that is famous for its involvement in hotels, namely through The Oberoi Group. Below is an example of knowledge addition.
New Knowledge:
The Oberoi Group hotel company with its head office in Delhi.
Question:
The Oberoi family is part of hotel company that has head office in what city?
Answer:
Delhi
Figure 6: An example of the data of INSERT, the question is related to the new knowledge.
An Example of Insert Operation
Old Document:
Bizarre was published by Dennis Publishing, and sister publication to the Fortean Times. Fortean Times is British monthly magazine devoted to the anomalous phenomena popularised by Charles Fort. Previously published by John Brown Publishing (from 1991 to 2001) and then Feel Good Publishing (2001 to 2005), it is now published by Dennis Publishing Ltd.
New Knowledge:
Bizarre was British alternative magazine published from 1997 to 2015.
Question:
Which publishing company has published Bizarre and a sister publication devoted to the anomalous phenomena popularised by Charles Fort?
Answer:
Deunis Publishing
Figure 7: An example of the data of INSERT, the question related the old knowledge.
An Example of Delete Operation
Old Document:
Jo Ann Verry-Grissons (born August 4, 1938 in Indianapolis, Indiana) is retired female hur- dler from the United States, who represented her native country at two consecutive Summer Olympics, starting in 1960. Affiliated with the Tennessee State University she won the 80 m hurdles event at the 1963 Pan American Games. The 4th Pan American Games were held from April 20 May 5, 1963, in Sao Paulo, Brazil.
Removed Knowledge:
The 4th Pan American Games were held from April to May 5, 1963, Sao Paulo, Brazil. Question:
Jo Ann Terry the 80m hurdles event at what Sao Paulo-based event from 1963?
Answer:
Pan American Games
Figure 8: An example of the data of DELETE, the question is related to the removed knowl- edge.
17

---

Published as a conference paper at COLM 2025
An Example of Delete Operation
Old Document:
Nicholas Sposato serves on Chicago City Council as alderman of the 38th Ward of the City of Chicago on the city's Northwest Side. Sposato was elected in 2011 in an election against incumbent alderman John Rice, who was endorsed by then Mayor-elect Rahm Emanuel. Rahm Israel Emanuel (: born November 29, 1959) is an American politician who and current mayor of Chicago. A member of the Democratic Party, Emanuel was elected in 2011. He re-elected April 7, 2015.
44th
Removed Knowledge:
A member of the Democratic Party, Emanuel was elected in 2011.
Question:
What side of town is the Chicago Mayer-endorsed, 38th Ward of the City of Chicago alderman serving?
Answer:
Northwest Side
Figure 9: An example of the data of DELETE, the question is related the old knowledge.
An Example of Update Operation
Old Document:
What Happened to Jones is 1926 silent film comedy directed by William A. Seiter and starring Reginald Denny. It was produced and distributed by Universal Pictures. The film is taken from 1897 Broadway play. What Happened to Jones by George Broadhurst. Reginald Denny (born Reginald Leigh Dugmore, 20 November 1891, 16 June 1967) was an English stage, film and television actor as well as an aviator and UAV pioneer. He was once an amateur boxing champion of Great Britain.
Conflicting New Knowledge:
What Happened to Jones 1990 color film. Question:
When What Happened Jones released?
Answer:
1990
Figure 10: An example of the data of UPDATE, the question is related to the conflicting new knowledge.
An Example of Update Operation
Old Document:
Double Take is 2001 action comedy film starring Eddie Griffin and Orlando Jones. Double Take was inspired by the 1957 drama Across the Bridge, which was in turn based short story by Graham Greene; the supporting cast includes Edward Herrmann, Gary Grubbs, Garcelle Beauvais, and Daniel Roebuck. Gary Grubbs (born November 14, 1949) is an American actor.
Couflicting New Knowledge:
Gary Grubbs was born on Jamuary 1, 1980.
Question:
When the film Double Take released?
Answer:
2001
Figure 11: An example of the data of UPDATE, the question is related to the old knowledge.
E Training Details
We use transformers (Wolf, 2019) to train our model, and the hyperparameters are shown in Table In pretraining, we only use one chunk to train models to get the ability of compression. During finetuning, we multi-chunk data to cultivate the model to retrieve from multiple chunks and answer the question. To train the editor, we use a chunk queue to cache the chunks from previous batches which describe in Section 3.
18

---

Published as a conference paper at COLM 2025
Table 5: Training Hyperparameters
Pretraining
Finetuning
Editor training
3B
8B
3B
8B
3B
8B
LoRA rank
64
64
64
64
64
64
Learning rate
le-4
le-4
le-4
le-4
le-4
le-4
Batch size
32
32
32
32
32
32
Steps
16000
16000
18000
12000
1600
1600
Warm-up steps
300
300
300
300
100
100
memory tokens
16
16
16
16
16
edit tokens
4
4
chunk size
128
128
128
128
128
128
Max chunks
1
1
10
10
10
10
Queue size
20
20
Results of Golden Editing by Human
Table 6: Results of HotpotQA of UPDATE and DELETE with golden editing by human.
UPDATE
DELETE
Method
Old (1)
Rp (f)
Old (f)
Rm (T)
Standard RAG 3B
golden edit
85.59 76.27
80.93
69.59
24.19
LLM to edit text
57.20
70.27
33.87
E2-RAG w/o Editor
(Efficient
RAG 3B)
golden edit
86.86
74.15
81.76
19.35
LLM to edit text
76.27
61.86
83.11
25.81
B--RAG
84.75
83.90
78.38
33.87
Standard RAG 8B
golden edit
85.59
76.27
74.32
17.74
LLM to edit text
84.32
70.34
70.27
27.42
E2-RAG w/o Editor (Efficient golden edit
RAG 8B)
88.98
82.63
89.86
16.13
LLM to edit text
84.32
74.15
87.16
32.26
E--RAG
85.17
84.75
78.38
30.65
G Results on RAG benchmarks
We use five document QA benchmarks (Welbl et al., 2017; Yang et al., 2018; Dua et al., 2019; Rajpurkar, 2016; Stelmakh et al., 2022) and two long-form QA benchmarks (Lin et al., 2021; Rosenthal et al., 2025) to comprehensively demonstrate the RAG performance of E2-RAG. To evaluate the efficiency of QA, we measure the Time To First Token (TTFI) and overall latency under varying context lengths and batch sizes.
In Table we present the performance of standard RAG and E2-RAG on various document QA benchmarks. E2-RAG shows performance comparable standard RAG on these tasks and even outperforms standard RAG on some benchmarks. For the long-form QA bench- marks in Table 8, E2-RAG outperforms standard RAG in most cases. This improvement is likely due to the addition of retrieval and reasoning steps during the finetuning stage. For efficiency, first, we evaluate the TTFT across different batch sizes and varying context lengths, with the results shown in Figure 12b and Figure 12a. We observe that the TTFT of E2-RAG remains nearly constant at a very low level. This is because E² E2-RAG pre-processes the context into compressed KV caches offline. Second, we present the total inference latency
19

---

Published as a conference paper at COLM 2025
for generating different numbers of tokens at fixed context length of 2560 across various batch sizes. As shown in Figure 12d, at the 8B model size and batch size of 8, E2-RAG achieves a 3x speedup compared to standard RAG. This speedup increases with larger batch sizes and model sizes. The improvement is attributed to the offline prefilling process and the significant compression of the context, which reduces the context length and accelerates the next token generation.
Table 7: Experimental results of E2-RAG on five document QA benchmarks and their aver- ages. The results show that E2-RAG incurs slight loss but overall maintains an advantage in document QA.
Model
Method
HotpotQA
ASQA
SciQ
SQuAD
Drop
Avg.
w/o contexts
21.92
38.37
60.9
18.28
16.22
31.14
standard RAG
74.38
74
82.1
85.89
68.27
76.93
Llama 3.2 3B
E2-RAG A
74.96
79.14
85.6
78.33
63.06
76.22
+0.58
+5.14
+3.5
-7.56
-5.21
-0.71
w/o contexts
27.57
49.56
66.8
24.76
18.76
37.49
standard RAG
75.88
79.93
89.5
89.55
74.68
81.91
Llama 3.1 8B
E2-RAG
79.36
84.99
87.8
83.62
67.64
80.68
A
+3.48
+5.06
-1.7
-5.93
-7.04
-1.23
Table 8: Experimental results of E2-RAG on two long-form QA benchmarks. The results show that E2-RAG has certain advantage on long-form QA benchmarks.
TruthfulQA
ClapNQ
Size
Method
Fi
R-L
F1
R-L
w/o contexts
9.6
10.09
9.17
8.25
standard RAG
25.59
26.62
20.76
18.36
3B
E2-RAG
26.16
26.75
19.73
18.71
A
+0.57
+0.13
-1.03
+0.35
w/o contexts
9.66
9.86
9.38
8.35
standard RAG
19.38
24.35
19.92
17.68
8B
E2-RAG
26.39
27.09
20.23
19.09
A
+7.01
+2.74
+0.31
+1.41
1.0
3.30
2.2
Laturtor11
Latencc
2.3
122
1923
2030
121
3120
2013
100
420
127
Context
(a) 3B TTFT
(b) 8B TIFT
(c) 3B cestency_memilc
(d) 8B Latency
Figure 12: The TTFT and Latency of B--RAG and standard RAG both 3B and 8B.
H Results on Qwen
To further validate our conclusions, we conduct additional experiments on Qwen2.5- 7B (Yang et al., 2024).
20

---

Published as a conference paper at COLM 2025
Table Results of Qwen 2.5 7B on INSERT operation.
HotpotQA
ASQA
SeiQ
SQuAD
Drop
Avg.
Method
(Id New (1) (ld(T)
(1) (ld
(1) (la (1)
(1)
(1)
Standard 7B
golden context 86.96
88.18
88.26
92.02 89.57
91.79
91.30 91.30
92.43
92.62 92.17
78.00 77:19
84.18 81.96
87.49
89.66 88.45
edit text 84.06
87.27
86.85
90.71
92.43
86.25
E-RAG w/o Editor (Efficient
golden context
86.23
86.36
86.38
85.94
91.07
89.86
84.07
83.44
80.04
73.95
85.56
84.31
LLM edit
79.14
86.57
87.68
81.01
73.97
71.20
79.62
E-RAG 7B
84.06
82.73
83.10
83.44
89.29
89.49
82.00
81.55
82.48
79.43
84.19
83.33
Table 10: Results of Qwen2.5-7B on DELETE operation.
HotpoiQA
ASQA
SciQ
SQuAD
Drop
Avg.
Method
(") ()) Old (?) (.) Old Rm ()) Old (D) (f)
Old (?) (J)
Standard
LLM to edit text 81.76 35.48
84.23
7.76
89.66
35.71
91.09
23.75
74.20
25.00
84.19
25.54
E2-RAG w/a Editor (Efficient RAG 7B)
LLM edit text
32.26
7.76
88.97
85.90
77.40
25.00
20.83
E E'-RAG 7B
75.00
30.65
84.98
12.07
85.52
20.16
85.47
19:74
73.45
23.33
81.26
21.19
Table 11: Results of Qwen 2.5 7B on UPDATE operation.
Method
Old Rp
HotporQA
Old (f) Rp
ASQA
Old
SciQ
Rp (1)
SQuAD
(f) Kp (1)
Drop Rp
Avg. Old
Rp
Standard
LLM to edit text 88.14
77.54
73.89
82.78
65.53
75.19
69.15
82.36
75.12
88.26
74.37
81.23
E-RAG w/o Editor (Efficient RAG 7B)
LLM to edit text 84.32
72.88
69.72
81.11
62.56
72.66
65.62
78.93
74.65
84.51
71.37
78.02
E-RG 7B
79.39
87.29
68.33
88.06
62.41
83.95
64.62
84.48
76.06
91.08
70.16
86.97
21

---

Published as a conference paper at COLM 2025
I Evaluation of Mult-turn Editing
Asqa ded
0.8
Drop old
0.9
Hotpotap old Hotpolqe
0.8
Arqu new
Drop rew
0.8
0.7
Accuracy
Accuracy
0.6
Accuracy
0.6
0.6
0.5
0.5
0.5
0.3
0.3
0.2
0.2
0.2
Tumn
(a) Asqa
(b) Drop
(c) Hotpotqa
Seiq dlc Sciq new
0.8
Squae old Squaci new
0.8
Average old
Average
0.7
0.7
0.7
Accuracy
0.6
Accuracy
0.6
Accuracy
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.2
Tums
Sciq
(e) Squad
(f) Average
Figure 13: The evaluation result of multi-turn editing on 3B size.
22

---

Published as a conference paper at COLM 2025
Asqa did
Drop old
0.8
Hotpotga old Hotpotap
0.7
0.7
0.6
Accuracy
0.6
Accuracy
0.66
0.5
0.5
0.5
0.4
0.4
0.3
Turns
Turns
Tums
(a) Asqa
(b) Drop
(c) Hotpotqa
Sciq
Squad cld
Average old Average
0.8
0.7
Accuracy
0.5
0.5
0.4
6.3
Turns
Turns
Turns
(d) Sciq
(e) Squad
(f) Average
Figure 14: The evaluation result of multi-turn editing on 8B size.
23

---

Published as a conference paper at COLM 2025
J Evaluation of Multi-chunk Ability with Edited Chunks
0.85
0.825
0.7
0.80
Accuracy
0.800
Accuracy
0.6
Accuracy
0.75
0.775
0.70
0.750
0.05
0.60
0.4
0.700
0.55
Number Chunks
Number of Chunks
Number Chunks
(a) Asqa
(b) Drop
(c) Hotpotqa
0.825
Average old
Average
0.84
0.800
Accuracy
0.82
0.775
Accuracy
0.75
0.750
0.80
0.725
0.70
0.78
0.700
0.65
0.675
0.650
Number Chunks
Number of Chunks
Number Chunks
(d) Sciq
(e) Squad
(f) Average
Figure 15: The evaluation results of multi-chunk on 3B size.
24

---

Published as a conference paper at COLM 2025
0.80
0.825
0.75
0.800
Accuracy
0.70
Accuracy
0.85
0.750
0.82
0.60
0.725
0.80
0.700
0.70
0.675
0.76
0.650
Number of Chunks
Number of Chunks (b) Drop
Number of Chunks (c) Hotpotqa
(a) Asqa
0.88
0.86
0.825
Average old Awerage
0.06
0.84
0.800
Accuras
0.84
Accuracy
0.82
Accuracy
0.775
0.80
0.82
0.725
0.80
0.78
0.650
Number of Chunks
Number of Chunics
Number of Chunis
(d) Sciq
(e) Squad
(f) Average
Figure 16: The evaluation result of multi-chunk on 8B size.
K
Results on Wikipedia
Delete (38)
7.6%
Insert (87)
17.5%
74.8%
Update (372)
Figure 17: Wikipedia Edit Pattern Analysis.
To evaluate our method in more practical scenarios, we build a complete RAG pipeline using LlamaIndex (Liu, 2022). For the data, we construct the database of approximately 10,000 chunks from portion of Wikipedia, where each chunk contains up to 128 tokens. To construct the editing records, we conduct a case study on Wikipedia edit histories. Specifically, we analyze the most recent 500 revisions of the "Machine Learning" entry in English Wikipedia. Each edit operation is categorized into three types: INSERT, DELETÉ, and UPDATE (Figure 17). Therefore, we construct around 1,000 editing records composed of
25

---

Published as a conference paper at COLM 2025
three types of operations, along with their corresponding questions. The proportions of the operations align with the real update distributions observed in Wikipedia.
For the model, we BGE (Xiao et al., 2023) large the Retriever and our 3B model to generate responses, During both the editing and querying processes, we retrieve the top-1 chunk. First, we update the database using the editing records. Specifically, retrieve the relevant chunk using the editing content and apply the edits. After all editing records have been applied to the database, we input the questions for evaluation.
Table 12: Results of Three types of operations on WikiPedia.
UPDATE
INSERT
DELETE
Method
Old (1)
Rp (f) Old (T)
New (1)
Old (T)
Rm (D)
Standard RAG
LLM to edit text
45.76
28.31
51.97
37.88
54.10
22.58
E2-RAG w/o Editor (Efficient RAG)
LLM to edit text
45.23
30.34
43.31
32.58
52.46
29.03
E2-RAG
45.94
41.02
41.73
37.12
52.46
27.42
26


Reviews:
[{"review_id": "vqOsHCgYlQ", "reviewer": "Reviewer_TEz1", "summary": "This work introduces E^2 RAG that improves the efficiency of RAG by editing compressed KV caches for knowledge updates.\nIn this system, all the information pieces are pre-encoded as compressed KV caches and stored for retrieval. \nIn the inference stage, the relevant compressed KV caches will be retrieved and added into the KV cache of the LLM for generating the response. In this way, the LLM does not need to read the raw textual information on the fly and an reduced response latency could be expected. \nBased on this framework, this work tries to handle the scenarios where the information that is pre-encoded compressed KV caches requires edit. \nThus, three edit operations, INSERT, DELETE, and UPDATE, to compressed KV caches are proposed in this work. \nTo evaluate, three datasets are constructed and three LLMs, including Llama3.2-3B, Llama3.1-8B and Qwen2.5-7B, are involved in experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work addresses an practical issue given the popularity of RAG. \n\n2. Good formulation of the edit scenarios in the three edit modes. \n\n3. The experiments include the multi-turn updates and the cases of knowledge conflicts. The results that show the cases where standard RAG performs inferior to the efficient RAG are interesting.", "reasons_to_reject": "1. In the current framework, the top k relevant information entries will be edited for update. For a query which is relevant to n information entries and n >> k, a practical issue raises that there are (n - k) old entries that are yet to update and these old information can dominate the search results and affect the accuracy of the final RAG outcome. This work could discuss such an issue for comprehensive. \n2. While the proposed method is shown effective on the two Llama models, it is less useful on the Qwen 2.5 7B model reported in Appendix H. More models could be included in experiments for comparison.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "09Chh7eGQF", "reviewer": "Reviewer_rSVu", "summary": "The paper presents E2-RAG, a novel architecture for Retrieval-Augmented Generation (RAG) systems that emphasizes both efficiency and editability by enabling direct editing of compressed key-value (KV) caches, a challenge largely unaddressed in prior works. This is achieved via an encoder-decoder framework for compression and an editor module that supports three canonical operations—INSERT, DELETE, and UPDATE—on KV caches without needing to reprocess the entire source text.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper explores a key problem in Retrieval Augmented Generation (RAG), namely, how to update a compressed key-value cache efficiently and accurately.\n2. This paper introduces a lightweight editor module equipped with a trainable LoRA adapter, enabling in-place insert, delete, and update operations on key-value caches. These editing capabilities resemble operations commonly found in relational databases and appear well-justified.\n3. The authors commit to releasing code and editing datasets that could serve as valuable resources for the community to reproduce and extend this line of work.", "reasons_to_reject": "1. Concerns about the method. While the authors argue that their method enables rapid updates to the key-value (k-v) cache and is more efficient than re-encoding the model, this claim appears counterintuitive. Specifically, the proposed editing module requires additional training, which introduces computational overhead. It is unclear how this training-based modification leads to a net efficiency gain over re-reasoning approaches that operate without such updates. Further clarification or empirical justification of this trade-off would strengthen the motivation.\n\n2. The approach is flawed.\n\t- In lines 138 to 141, the authors state that the positional embedding of the 'key' should be removed during encoding. This raises the question: why is a similar operation not required for the 'value'?\n\t- The Editor Module appears to simply perform fine-tuning on newer data with reduced memory embedding, which seems to fall short of the ambitious motivation presented. The novelty may be somewhat overstated.\n\t- Would the introduction of additional key-value cache entries not negatively impact the inference speed?\n3. There are concerns regarding the quality of the dataset. Although the authors used an LLM to generate a portion of the data, there appears to be no evidence of manual quality control or validation.\n4. The proposed method requires model training. When comparing editing efficiency with the baseline, was the training time taken into account? (Section 5.1)\n5. The methods section of this paper is not clearly written.\n\t- Is W_E in formula 1 the embedding parameter matrix of LLM?\n\t- What is $Q_apeend$ , and why can it be directly multiplied with $K$ in the key-value cache?\n\t- What does $O$ represent in Equation (11), and what role does it play in the proposed model?\n\t- The insert, delete, and update operations described in lines 162, 169, and 173 are intended solely for analyzing the role of the Editor Module and, as such, should not include method-specific components.", "questions_to_authors": "1. In line 225, the authors characterize Golden Edit as representing an upper-bound performance. However, the results reported in Table 1 indicate that the proposed method outperforms this supposed upper bound. This discrepancy raises concerns about either the definition of Golden Edit or the interpretation of the experimental results. Clarification is needed to reconcile these findings—either by refining the description of Golden Edit or by explaining under what conditions the proposed method can exceed its performance.\n2. I find the distinction between the \"old\" and \"new\" settings in Table 1 somewhat unclear. Could the authors clarify what specifically differentiates these two settings, including any changes in evaluation protocol, data distribution, or editing targets?", "extra_scores": {}}, {"review_id": "oAQioqz2oH", "reviewer": "Reviewer_qvW3", "summary": "This paper introduces E²-RAG, an editable and efficient Retrieval-Augmented Generation (RAG) framework designed to update compressed key-value (KV) caches for timely knowledge integration. Traditional RAG systems face computational bottlenecks when regenerating compressed representations after edits. E²-RAG addresses this by introducing an editor module that directly modifies KV caches using INSERT, DELETE, and UPDATE operations, inspired by database principles. The framework achieves 40x faster editing and 3x faster generation compared to baseline methods, with minimal performance degradation (1%-5%) on QA benchmarks. Experiments on diverse datasets (HotpotQA, ASQA, Drop) and models (Llama3.1-8B, Qwen2.5-7B) validate its efficiency and scalability. Ablation studies explore multi-turn editing, multi-chunk inference, and knowledge conflict resolution.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel Contribution: First work to enable efficient editing of compressed KV caches in RAG, addressing a critical gap in maintaining up-to-date knowledge.\n\n- Strong Empirical Results: Demonstrates significant speed improvements (40x editing, 3x generation) with minimal accuracy loss across multiple tasks and models.\n\n- Comprehensive Evaluation: Includes ablation studies, multi-chunk analysis, and OOD testing, offering insights into operational robustness and limitations.", "reasons_to_reject": "- Limited Model Generalization: Experiments are confined to Llama and Qwen models; broader validation across architectures (e.g., GPT, Mistral) is missing.\n\n- Superficial Discussion on Conflicts: While knowledge conflicts during UPDATE/DELETE are noted, the resolution mechanisms lack depth and theoretical grounding.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JQKSVXTERw", "reviewer": "Reviewer_ZyFk", "summary": "The paper introduces E²-RAG, an Editable Efficient Retrieval-Augmented Generation (RAG) method for efficiently updating compressed knowledge caches (KV caches). The proposed approach integrates an encoder-decoder architecture for efficient generation and an editor module that performs direct updates on KV caches through operations like INSERT, DELETE, and UPDATE.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Paper is well written and easy to follow.\n* The paper is written clearly, with well-structured arguments, polished figures, and well-explained datasets and tasks, making it reader-friendly.\n* This research could have impact for LLM Training\n* The paper provides detailed theoretical with attention map explanations the introduced methodology.", "reasons_to_reject": "The study lacks experiments with larger-scale models (e.g., 14B or 32B), which would significantly strengthen the generalizability and practical relevance of the results.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes E²-RAG, a novel framework that enables efficient and editable updates to compressed key-value (KV) caches in retrieval-augmented generation (RAG) systems. All four reviewers recognized the significance of the problem and the soundness of the core idea—especially the design of INSERT, DELETE, and UPDATE operations directly on KV caches. While some reviewers raised concerns about efficiency claims, training costs, and generalization to broader model families, the authors provided solid clarifications, new experiments, and in-depth responses that addressed most of these concerns. Given the paper's novelty, clear formulation, thorough experiments, and the community value of released datasets and code, I recommend acceptance.
Decision: Accept