Let the Code LLM Edit Itself When You Edit the Code
Authors: Zhenyu He
Jun Zhang
Shengjie Luo
Jingjing Xu
Zhi Zhang
Di He
Venue: ICLR 2025 Poster (2025)
Keywords: code generation
efficiency
large language model
code assistant
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In this work, we investigate a typical scenario in code generation where a developer edits existing code in real time and requests a code assistant, e.g., a large language model, to re-predict the next token or next line on the fly. Naively, the LLM needs to re-encode the entire KV cache to provide an accurate prediction. However, this process is computationally expensive, especially when the sequence length is long. Simply encoding the edited subsequence and integrating it to the original KV cache meets the temporal confusion problem, leading to significantly worse performance. We address this efficiency and accuracy trade-off by introducing $\underline{\textbf{P}\text{ositional}\  \textbf{I}\text{ntegrity}\  \textbf{E}\text{ncoding}}$ (PIE). Building upon the rotary positional encoding, PIE first removes the rotary matrices in the Key cache that introduce temporal confusion and then reapplies the correct rotary matrices. This process ensures that positional relationships between tokens are correct and requires only a single round of matrix multiplication. We validate the effectiveness of PIE through extensive experiments on the RepoBench-C-8k dataset, utilizing DeepSeek-Coder models with 1.3B, 6.7B, and 33B parameters. Our evaluation includes three real-world coding tasks: code insertion, code deletion, and multi-place code editing. Results demonstrate that PIE reduces computational overhead by over 85%  compared to the standard full recomputation approach across all model sizes and tasks while well approximating the model performance.

Published as a conference paper at ICLR 2025
LET THE CODE LLM EDIT ITSELF WHEN YOU EDIT THE CODE
Zhenyu He
Jun Zhang Shengjie Luo
Jingjing Xu
Zhi Zhang
Di He
National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University ByteDance Inc.
ABSTRACT
In this work, investigate a typical scenario in code generation where developer edits existing code in real time and requests a code assistant, e.g., a large language model, re-predict the next token or next line on the fly. Naively, the LLM needs to re-encode the entire KV cache to provide an accurate prediction. However, this process is computationally expensive, especially when the sequence length is long. Simply encoding the edited subsequence and integrating it to the original KV cache meets the temporal confusion problem, leading to significantly worse performance. We address this efficiency and accuracy trade-off by introducing Positional Integrity Encoding (PIE). Building upon the rotary positional encoding, PIE first removes the rotary matrices in the Key cache that introduce temporal confusion and then reapplies the correct rotary matrices. This process ensures that positional relationships between tokens correct and requires only single round matrix multiplication. We validate the effectiveness PIE through extensive experiments the RepoBench-C-ae dataset, utilizing DeepSeek- Coder models with 1.3B, 6.7B, and 33B parameters. Our evaluation includes three real-world coding tasks: code insertion, code deletion, and multi-place code editing. Results demonstrate that PIE reduces computational overhead by over 85% compared the standard full recomputation approach across all model sizes and tasks while well approximating the model performance. Code is available at https: //gi thub. com wwmm//hennuhee0/P....
PIE (Ours)
Full Recompulation
800
Exact Match (9)
Time (mS))
400
15
300
100
0
50
100
150
200
250
0
100
150
200
200
Number Edittdd Tokens
Number Edited Tokens
Figure 1: Latency and accuracy comparison of the full recomputation approach and PIE using DeepSeek-Cone 6.7B on the CeeoBeench-CSS(XFEE Python dataset on a single A100 GPU. The latency only records the time cost for the KV cache update.
INTRODUCTION
Large language models (LLMs) (Dettmers et al., 2022; Anil al., 2023; Touvron et al., 2023; Zeng et al., 2023) have seen widespread adoption and achieved impressive results across various natural language processing (NLP) tasks. Despite these successes, LLMs face significant computational
"Wrr done during Zhenyu's internship at ByneDance. Cooresponndecce Di He-ddihe@puu.ede edu. en>>

---

Published as a conference paper at ICLR 2025
challenges, particularly in handling long sequences. To address this, numerous approaches have been proposed to accelerate the inference process, including lossless (e.g., memory and IO optimiza- tion (Dao et al., 2022; Kwon et al., 2023; Sheng et al., 2023), speculative decoding (Stem et al., 2018; Leviathan et al., 2023)) and lossy techniques ge.g., quantization (Frantar et al., 2022; Xiao et al., 2023) and KV cache eviction (Xiao al., 2024; Zhang et al., 2024)). We refer the above setting as the static setting, where the content is fixed, and the goal is to generate responses efficiently without compromising too much on performance.
Besides the static setting, we observe there is strong demand for an alternative, which we call the real-time editing setting, where users frequently edit the content and expect the LLM to generate correct responses based on the updated information. A typical scenario is the interactive coding assistant, where developers often make incremental changes to their existing code and require the Al copilot to correctly predict the next line or complete a partial code snippet on the fly. The standard approach is re-encoding the KV cache of the content after each edit and then making the prediction. However, as illustrated in Figure 1, this approach leads to considerable substantial computational overhead and latency when the content is long (Fu, 2024; Agrawal et al., 2024), making it impractical for real-time applications where quick and accurate responses are essential.
In this paper, we aim to improve the efficiency of Al copilots in real-time editing scenarios, as illustrated in Figure 2. Naively, an efficient strategy is encoding only the edited subsequence and then directly integrating those keys and values into the original KV cache. However, this strategy results in temporal confusion between the pre-edit and post-edit sequences. Keys of certain positions either disappear or multiple keys at different positions share the same index, causing the model to attend to incorrect information, leading poor next-token prediction performance in practice. To address this problem, we introduce Positional Integrity Encoding (PIE). PIE is built upon rotary positional encoding (RoPE) (Su et al., 2021). the de-facto standard component in modern LLMs. PIE first removes the rotary matrices in the Key cache that introduce temporal confusion and then reapplies the correct rotary matrix for each position through simple matrix multiplications. By ensuring that the positional relationships between tokens in the new sequence are unique and consecutive, PIE can help the model make accurate predictions. It is worth noting that the calculation of PIE requires only single round of matrix operations to modify the KV cache, resulting in negligible computational overhead.
We demonstrate the effectiveness of Positional Integrity Encoding (PIE) through extensive experi- ments conducted on the RepoBench- dataset, utilizing the DeepSeek- DeepSeek-Coderr (Guo et al., 2024) models with 1.3B, 6.7B, and 33B parameters. To rigorously evaluate PIE's performance, three tasks designed to simulate real-world coding scenarios: code insertion, code deletion, and multi-place code edition. These tasks were chosen to reflect common operations that developers perform during interactive coding sessions, thereby providing a comprehensive assessment of PIE's practical utility. Our experimental results indicate that PIE achieves reduction in computational overhead of over 85% for editing the KV cache across all model sizes and tasks without compromising performance compared to the naive fullprecomtttation approach. By leveraging PIE, developers can experience efficient interactions with AI coding assistants.
curated
RELATED WORK
Positional Encodings Positional information is essential for modeling languages. The original Transformer model (Vaswani et al., 2017) encodes positional information using Absolute Positional Encoding (APE). In particular, (learnable) real-valued embedding is assigned to each position i. Differently, Relative Positional Encodings (RPE) (Shaw et al., 2018; Dai al., 2019; Raffel et al., 2020; Press et al., 2022; Su et al., 2021; Luo et al., 2021; 2022; Chi et al., 2022; Sun et al., 2023; Chi et al., 2023; Li et al., 2023; He et al., 2024) instead encode the relative distance i -j j for each position pair (i.)). One of the most widely used RPE in state-of-the-at LLMs is Rotary Position Encoding (RoPE) (Su et al., 2021). RoPE rotates the query and key vectors by an angle proportional to their absolute positions before the attention mechanism, resulting in the attention being function of the relative distance between tokens.
In the literature, relative positional encodings play essential roles across various tasks and data modalities, such as improving the length extrapolation capability of language models (Press et al., 2022; Sun et al., 2023; Chi et al., 2022; 2023; He et al., 2024) and enabling flexible modeling of
2

---

Published as a conference paper at ICLR 2025
Static setting
Couls
Comete PACOA LA,CIT)
Large Language
Tueree 11. an Craaii, wills
Model
Real-time editing setting
Iqput Code
BV Cachs
Drpas
bey 1e,, she wntntte (
Large
Language
Model
Editd Code
Ovipa
anilie Les dLD)
tie petiting (0L
MHttii PHILIIM BENIIEAN LBLICA
Figure 2: Illustration of the KV cache mechanism in both static and real-time editing settings for large language models (LLMs). Top: In the static setting, the model processes fixed input to generate predictions, leveraging precomputed Key/Value (KV) pairs stored in the cache. Bottom: In the real-time editing setting, the input code frequently edited, necessitating updates the KV cache to maintain accurate information to generate the correct next tokens. Our objective is to optimize the efficiency of the green arrow pathway, which represents the process of updating the KV cache in response to code edits.
structural information beyond sequence data like images (Liu et al., 2021) and graphs (Ying et al., 2021: Zhang et al., 2023a; Luo et al., 2023). In this work, develop the Positional Integrity Encoding based on RoPE to improve the efficiency of LLMs in the real-time editing setting.
Transformer Efficiency Improving the efficiency of Transformer models has great significance in real-world applications. In the literature, existing approaches can be briefly categorized into (1) efficient attention, (2) model compression, and (3) archiitcccie co-design.
The attention module of Transformer needs to calculate pairwise correlations between all positions, resulting in quadratic time and memory cost with respect to the sequence length. To reduce the cost, many efficient attention variants have been proposed, such as (1) sparse attentions (Child et al., 2019; Beltagy al., 2020; Qiu et al., 2020), which design either pre-defined or learnable patterns to reduce the amount of the key-value pairs that each query needs to attend to; (2) approximation basee attention (Katharopoulos
al., 2020; Wang et al., 2020; Choromanski et al., 2021; Kitaev et al., 2020; Tay et al., 2020; Roy et al., 2021), which use tailored approaches like low-rank projection or random features approximate standard attention for efficient computation.
Another perspective for Transformer efficiency is model compression, mainly including (1) prun- ing (Wang al., 2021; Hubara al., 2021; Ma et al., 2023; Frantar & Alistarh, 2023), which aims at removing redundant model parameters or layers for efficient deployment without scarifying perfor- mance; (2) quantization (Yao et al., 2022: Park al., 2022: Dettmers al., 2022: Frantar et al., 2022; Xiao et al., 2023; Liu et al., 2023), which uses post-processing to represent weights and activations via low-precision format for reducing time and memory costs; (3) knowledge distillation (Sanh et al., 2019; Gu et al., 2024), which uses smaller model to learn knowledge from a large model for balancing afficiency-aceuracy trade-offs.
In the era of LLMs, the importance of wishtm-architecture co-design has been highlighted to im- prove Transformer efficiency further. Many works begin to design more efficient approaches for serving Transformers regarding the characteristics of computer systems for real-world applications, such as FlashAttention (Dao et al., 2022; Dao, 2023), PagedAttention (Kwon et al., 2023), and FlexGen (Sheng et al., 2023) that are proposed for memory and I/O optimization. Additionally, to reduce functional calls during generation, speculative decoding (Stern et al., 2018; Leviathan et al., 2023; Chen et al., 2023; Miao al., 2023; Spector & Re, 2023; Cai et al., 2024; Zhang et al., 2023b;
3

---

Published as a conference paper at ICLR 2025
He al., 2023; Li et al., 2024) has been proposed. Our Positional Integrity Encoding is specially designed to improve the efficiency of LLMs in real-time editing settings, which can be seamlessly combined with all the inteoinnrodueed approaches to achieve further speed-ups.
METHODS
3.1 BACKGROUND
Let (w1, ..2.-..,u0 w,) represent the input token sequence, where each w; belongs to fixed vocabulary. Let OLIM represent a Transformer-based large language model, which can calculate the conditional probability distribution of the next token p(w,,i|s;; OLLM) and generate tokens iteratively. Typically, the input is fixed. Therefore, the generation process of LLMs usually employs the KV Cache mechanism (Pope et al., 2023) to store previously computed Key/Value vectors during each layer's 's attention calculation. We denote the KV cache as K = (K,, Ky..... K₂) and V (V, 1/2.... Vn), where K; and V, are the keys and values associated with token ws. When predicting the token at position n 1, we can use Wn, as input and, in each layer, compute the attention between the current hidden representation and the stored KV cache, avoiding recomputing the hidden representation of previous tokens. Without any confusion, also denote the next token probability distribution as p( (w,-11 K, V. wn; lLLM)
scenario where the context is real-time edited by users, which makes it impossible for the KV cache to predict the correct next token without any modification. We model such real-time context as sequence of steps. Each step can be formulated as action where tokens from position to j in are edited, resulting in a modified sequence todo represent the Our goal is to accurately and efficiently predict Wn++1 given by fI1M sedit This problem crucial in various scenarios. For instance, users can frequently edit their previous codes for different purposes and expect the code language model to swiftly adapt to these changes and predict the correct next line based the updated information.
In this study, we aim investigate
new inputs that replace [w....- myy]-
As the context between position and position j is edited, the KV cache corresponding to these tokens must be updated. Furthermore, these changes will impact the representations of subsequent tokens af- ter position j. thereby necessitating updates to all subsequent KV cache. The naive approach involves a full-recomputation strategy: re-encoding all the KV cache for [a1, 02 am, Wj+1,.... .... wn] layer by layer, followed by making predictions using the updated cache K and V This ap- proach ensures the KV cache exact when predicting the next tokens. However, see that it is computationally expensive, especially when the edits are light but the texts to be re- encoded are long. It's worth noting that the original K and V already encode rich information on [wj+1,.... Wp], and full recomputation may not be essential for practical problems. With this in mind, we seek to find ways to efficiently edit K and V. yielding Kealt and Vedit which approximates p(W,+1|K' KK W*, OLLM) Rs p(wn+1 K edit Vedit wn; OLLM) in an effective way.
easy to
Wni
3.2 POSITIONAL INTEGRITY ENCODING (PIE)
When a user modifies s into edit KV cache associated with the first tokens, i.e., [1ss] and Viss remains unchanged. As (a), .2,.... (m] is the user's new input, we feed this subsequence to the LLM to obtain the keys and values from position i + 1 to i + m. We denote this piece of new K cache as edit and rddi
[++l++m] and now have the edited KV cache as:
[+++++m]]
K edi Concat K [1:a], K edit
mm] K Ki+1mn))
(1)
Vedit Concat VIsaj. Vedit
[+11++m] Vi++s)),
(2)
where the red symbols indicate real-time calculations.
Challenges. The key challenge lies in how to edit the succeeding KV cache K K[t+ and Vi++im] Clearly, the modification of [01. og...., am impacts the subsequent content in two ways: seman- tically and structurally. The semantic impact refers the changes in the understanding of the
subsequent text caused by the edited content. This can be a problem in natural language applications, such dialog systems, where modifications to earlier conversations can significantly influence the
4

---

Published as a conference paper at ICLR 2025
generation of current responses. The other impact is structural, primarily concerning the temporal confusion between the pre-edit and post-edit sequences when j j-1 * m. This issue arises with common editing actions in code, such as additions and deletions (corresponding to - i 0 or m = O). To be more concrete, imagine the original sequence has 5 tokens. If we add three tokens
between the second and third position, it will occupy the positional index [3, 4, 5). We calculate Kisss and [3:5] in equation (1) and (2). However, in the original K 3:5] and V3s.5). the positional [3:5]
index [3, 4, 5] is also occupied. If integrate them together and take actions during the next token prediction, the model will calculate similarity with multiple keys in the KV cache with the same index [3, 4, 5), causing confusion and potential prediction errors. Empirically, in code tasks, find that the semantic impact is relatively small. Addressing temporal confusion for light edits alone can already lead good performance (see the experiments in Section for more details).
Our approach. To mitigate the temporal confusion during real-time editing, we propose simple yet effective solution: Positional Integrity Encoding (PIE), which ensures that positional information remains correctly ordered after editing without the need to re-encode the KV cache for subsequent tokens. PIE builds upon the rotary positional encoding (RoPE) (Su et al., 2021), which is the most widely used positional encoding in LLMs. Without loss of generality, given query vector æ; at position and a key vector z at position j. RoPE calculates the dot- hot-producc similarity using
2 = x W R R
(3)
where R is the rotary matrix parameterized by the relative distance j- i, and W₂ and Wi are learnable projection matrices. By definition, Ri-; can be expressed by the multiplication of two rotary matrices:
R R R
For practical implementation, during inference, compute Re, WE,; the key on the fly and store it in the cache, and when a query arrives at a new position, we rotate the query using its corresponding rotary matrix and calculate its similarity with all the keys in the cache to obtain the attention scores.
can be easily seen that the positional information in the KV cache is encoded within the rotary matrix. When an edit occurs, the rotary matrix associated with the keys must be adjusted to reflect their post-edit locations. Leveraging the formulation of RoPE-based attention calculation, this challenge can be addressed by first removing the rotary matrices in K that introduce temporal confusion and then reapplying the correct rotary matrix. In detail, assume we would like to update the key vector [j + n,) where I [1, L] is the layer index. We can simply edit
ky, for the original position j' E the key vector by using
sdit. kj
(5)
where R2 the inverse rotary matrix at position j'. is used to remove the incorrect positional information, and Ri+m-j'-j is used to encode the correct position i m j' - in eddit It can easily seen that the computation can be further simplified as
kelli R Ry kby = R = Ritm-skly
(6)
Hence, the full editing process for K 1im] is as follows:
K Kedit [++mm]]
Kk f+11.....
where each Kith each k.glil.
Redic .... ky A. ei [1,4]
Ri+m-skl.
Unlike the full recomputation approach, the above calculation only requires single round of matrix multiplication to directly modify the pre-computed KV cache, where the computational overhead can be considered negligible. By utilizing these transformations, we finally construct the edited KV cache as:
K Concat( K (1::j.. Kedit
Concat VILaj, Velit [+1mn))
[+]+++m] K Kedit
(8)
Vedit
[++1++mp V+++m)),
(9)
5

---

Published as a conference paper at ICLR 2025
Table Statistics of RepoBench-C-8k (Liu al., 2024a) test set.
Language
XF-F
XF-R
IF
Average Number of Tokens
Python 18,000
7,500
10,500
3,967
Java 18,000
7,500
10,500
4,179
where the red symbols indicate real-time calculations. The LLM then makes predictions based on p(En+1| K Veddit Tx, (LIM) It is worth noting that PIE is compatible with KV cache eviction methods (Xiao al., 2024; Zhang et al., 2024; Liu al., 2024b). These KV cache eviction methods focus reducing the memory usage of the KV cache during inference. PIE is designed to obtain the KV cache the edited context with minimal overhead. By integrating PIE with KV cache eviction methods, it is possible to maintain efficient memory management while ensuring the integrity of the positional information in the real-time edit setting.
EXPERIMENTS
In this section, we empirically study the effectiveness of our proposed method. In particular, we aim at answering the following questions through experiments:
Question Can our Positional Integrity Encoding maintain the prediction accuracy of full re-computation in code editing scenarios?
Question 2: How much efficiency improvement can be achieved by using our Positional Integrity Encoding compared to existing approaches?
Question 3: How large is the gap between our Positional Integrity Encoding and full re-computation in terms of LLM's predictions & representations?
We will answer each question with carefully designed experiments in the following sub-sections. We also conduct additional experiments in Appendix B.
4.1 EXPERIMENTAL SETUP
Tasks. Our experiments are conducted RepoBench---8k (Liu et al., 2024a). This benchmark focuses on the prediction of the next line of code, given a set of in-file context (including import statements and preceding lines before the target line), and cross-file context (comprising snippets from other files parsed by import statements). The detailed statistics of RepoBench-C-Be is shown Table 1. To effectively evaluate next-line prediction performance of code L.LMs, we follow Liu et al. (2024a) to use three task settings: (1) Cross-File Fins-FFilFFrrst (XF-F): mask the first appearance of cross-file line within a file; (2) Cross-File- File--andom (XF-R): mask random and non-first occurrence of cross-file line; (3) In-File (IF): mask an in-file line that does not involve any cross-file modules. Moreover, we carefully design three real-world scenarios covering code insertion, code deletion, and code edition to comprehensively examine approach. See Appendix A.1 for more detailed descriptions of tasks construction.
Settings. In our experiments, we employ DeepScek-Coder (Guo et al., 2024), a code LLM that achieves strong performance in handling repository-level code completion tasks (We also conduct experiments on CodeLlama (Roziere et al., 2023) in Appendix A.2). We use Transformers (Wolf et al., 2020) as our codebase. We benchmark our method on models of different sizes covering 1.3B, 6.7B, and 33B. During inference, the greedy decoding strategy is used to deterministically generate 64 tokens. For 1.3B and 6.7B models, all the experiments are conducted on a single NVIDIA 100 GPU. For 33B models, the time for encoding the context is conducted on two NVIDIA A GPUs and the full generation process is conducted line in the output is truncated and used as the prediction. The batch size is set to All experiments repeated three times with different seeds and the averaged scores reported.
eight NVIDIA A100 GPUs. The first non-comment
Evaluation. For comparison with our Positional Integrity Encoding, we choose two standard approaches baselines: (1) Full-recomputation:: re-compute the KV cache for all edited tokens

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparisons of insertion experiments. In this task, for each next-line prediction target, we insert several lines of code into its context randomly to simulate real-world scenarios. EM and ES denote the Exact Match and Edit Similarity score respectively. All results demonstrate that our Positional Integrity Encoding approach brings substantial speed-ups without performance drops.
XF-F
XF-R
Model
Method
EM
ES
Time
EM
ES
Time
EM
ES
Time
Python Python Pytbon
1.3B
rull-recompttttion Conflict Fast Encoding
22.42
65.26
192ms
35.41
72.96
193ms
28.78
69.22
193ms
1.3B
7.32
43.73
23ms
10.61
47.18
22ms
45.25
22ms
1.31
PIB
22.3
65.2
29ms
35.33
72.88
28ms
28.69
69.13
29ms
6.7B
Full-recomputation
28.95
70.11
561ms
40.89
76.19
564ms
35.26
72.73
562ms
6.7B
Conflict Fast licooding
5.35
33.32
34ms
6.52
35.25
34ms
6.09
38.76
34ms
6.7B
PIE
28.83
70.01
50ms
40.77
76.14
50ms
35.2
72.72
50ms
33B
Full-rccomputatin
35.75
73.46
2194ms
46.0
78.9
2199ms
39.75
75.12
2194ms
33B
Conflict Fast Encuding
3.96
30.13
126ms
5,41
32.32
121ms
3.92
35.56
127ms
33B
PIE
35.77
73.45
134ms
45.74
78.85
140ms
39.74
75.1
141ms
1.3B
cullppecomuttaioon
26.21
70.89
200ms
36.77
76.31
200ms
45.89
78.04
198ms
Java
1.3B
Conflict Fast Encodding
0.29
3.12
22ms
0.57
3.19
23ms
0.7
2.63
23ms
1.3B
PIB
26.13
70.82
30ms
36.67
76.25
30ms
45.92
77.99
29ms
6.7B
Full-rccomtutaion Conflict Fast Encoding
32.51
75.56
578ms
41.97
79.41
578ms
50.86
80.53
578ms
Java
6.7B
0.47
2.77
34ms
0.76
2.78
35ms
0.67
2.48
33ms
6.7B
PIB
32,21
75.47
50ms
41.96
79.32
49ms
50.85
80.43
48ms
33B
Full-rccomutaatio
35.05
76.93
2269ms
44.95
80.87
228
53.23
81.76
2270ms
Java
33B
Conflict Fast Encoding
0.38
2,51
120ms
0.59
2.40
122ms
0.68
2,09
122ms
33B
34.78
76.78
138ms
45.01
80.95
133ms
53.16
81.66
139ms
and subsequent tokens; (2) Conflict Fast Encoding: re-compute the KV cache for the edited tokens while keeping the rest of the cache intact (i.e., using equation (1,2). Following Lu et al. (2021), we use Exact Match (EM) and Edit Similarity (ES) (Svyatkovskiy et al., 2020) to evaluate the accuracy of the predicted code lines on code completion tasks. We also report the time required to encode the edited context for efficiency evaluation.
4.2 MAIN RESULTS
Positional Integrity Encoding perfectly preserves the full re-computation performance. Results of different code editing settings are presented in Table 2, 3 and 4 respectively. It can be easily seen that all metrics indicate that our Positional Integrity Encoding can perfectly maintain the prediction accuracy of full re-computation in different scenarios. This indicates that PIE effectively addresses temporal confusion, and the semantic impact of minor edits is relatively negligible. For example, in the most challenging XF-F setting requiring to handle long-range cross-file context, the maximum relative difference between our PIE and Full-recomputation across different model sizes and code 1.33% for code insertion, deletion, and edition in
languages is terms of EM/ES respectively, which is rather negligible. This thorough examination serves as a strong support of the reliability of our PIE approach in real-world code editing scenarios.
0.66%
Positional Integrity Encoding significantly reduces computational overhead. Moreover, we further benchmark the computational costs brought by different approaches. From all results in the above tables, it can be easily seen that our Positional Integrity Encoding can achieve substantial speed- up compared to full re-computation while preserving its performance simultaneously. for the code edition experiment that requires both insertion and deletion, the averaged reductions of computational overhead induced by our PIE are 87. 87.9%/88..2 28% 92.4% 94.8% 2.% for 1.3B, 6.7B, and 33B models on Python/Java languages respectively. Furthermore, compared to Conflict Fast Encoding which induces minimal costs but largely hurts performance, our PIE only brings negligible overhead, showing its good accuracy-efficieng balance.
particular,
In summary, our main results comprehensively demonstrate the superiority of our Positional Integra- tion Encoding for code LLMs towards real-world code editing scenarios, which perfectly preserves pe the prediction accuracy and significantly addresses the crucial gap in the efficient deployment of L.LMs in real-time dynamic scenarios.
7

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparisons of deletion experiments. In this task, for each next-line prediction target, we delete several lines of code of its context randomly to simulate real-world scenarios. EM and ES denotes the Exact Match and Edit Similarity score respectively. All results demonstrate that our Positional Integrity Encoding approach brings substantial speed-ups without performance drops.
XF-F
XF-R
Model
Method
EM
ES
Time
EM
ES
Time
EM
ES
Time
Python Python Python
1.3B
rull-recompttttion Conflict Fast Encoding
22.42
65.26
192ms
35.41
72.96
193ms
28.78
69.22
193ms
1.3B
0.41
39.31
22ms
0.59
42.02
24ms
1.05
42.85
22ms
1.31
PIB
22.31
65.19
29ms
35.25
72.81
27ms
28.8
69.16
27ms
6.7B
Full-rccomputation
28.95
70.11
561ms
40.89
76.19
564ms
35.26
72.73
562ms
6.78
Conflict Fast licooding
0.51
40.24
30ms
0.77
42.77
31
1.42
43,93
30ms
6.7B
PIE
28.86
70.01
43ms
40.77
76.15
42ms
35.09
72.67
42ms
33B
colpppocomtttaion
35,75
73.46
2194ms
46.0
78.9
2199ms
39.75
75.12
2194ms
33B
Conflict Fast Encuding
1.42
43.93
105ms
1.55
44.22
108ms
45.04
108ms
33B
PIE
35.09
72.67
128ms
45.21
78.18
118ms
39.69
75.05
119ms
1.3B
cullppecomuttaionn
26.21
70.89
200ms
36.77
76.31
200ms
45.89
78.04
198ms
Java
1.3B
Conflict Fast Encoding
0.33
33.48
22ms
0.55
36.24
22ms
1.35
38.65
22ms
1.3B
PIB
26.05
70.77
28ms
36.83
76.34
27ms
45.74
77.91
27ms
6.7B
Full-rccomuutaion Conflict Fast Encoding
32.51
75.56
578ms
41.97
79.41
578ms
50.86
80.53
578ms
Java
6.7B
0.49
33.6
29ms
0.8
36.17
29ms
1.84
38.82
29ms
6.7B
PIB
32,57
75.56
42ms
41.83
79.39
4.3s
50.88
80.52
42ms
33B
Full-rccomutaatio
35.05
76.93
2269ms
44.95
80.87
228 Ims
53.23
81.76
2270ms
Java
33B
Conflict Fast Encoding
0.91
35.06
109ms
1.01
37.97
106ms
2.31
40.15
105ms
33B
PIB
34.82
76.76
117ms
44.93
80.86
120ms
53.19
81.71
119ms
Table 4: Performance comparisons of edition experiments. In this task, for each next-line prediction target, we delete several lines of code of its context and simultaneously insert other lines of code randomly to simulate real-world scenarios. EM and ES denote the Exact Match and Edit Similarity score respectively. All results demonstrate that our Positional Integrity Encoding approach brings substantial speed-ups without performance drops.
XF-F
XF-R
IF
Model
Method
EM
ES
Time
EM
ES
Time
EM
ES
Time
Python Python Python
1.38
Full rellopptttttanoon
22.42
65.26
242ms
35.41
72.96
242ms
28.78
69.22
244ms
1.3B
Conflict Fast Encoding
8.80
50.08
23ms
13.83
54.01
22ms
11.29
51.99
22ms
1.3B
PIB
22.04
64.59
30ms
34.49
72.02
29ms
28.15
68.32
29ms
6.7B
Full reloppptomtttnno
28.95
70.11
705ms
40.89
76.19
706ms
35.26
72.73
713ms
6.7B
Conflict Fast Encoding
11.26
51.63
34ms
12.57
53.27
34ms
12.75
51.45
34ms
6.7B
PIB
28.07
69.00
54ms
39.89
75.10
54ms
34,15
71.64
54ms
33B
Full- nullpeecomuttation
35.75
73.46
2766ms
46.00
78.90
2759ms
39.75
75.12
2787ms
33B
Conflict Fast Encoding
14.33
53.73
126ms
15.12
$4.59
12
13.88
51.94
127ms
33B
PIB
34.62
72.47
146ms
44.59
77.69
142ms
38.83
74.08
141ms
1.3B
cullpreoomtttation
26.21
70.89
251
36.77
76.31
253ms
45.89
78.04
249ms
Java
1.3B
Conflict Fast Eincoding
5.87
31.46
22ms
8.01
32.74
23ms
10.20
34.34
23ms
1.3B
PIB
25.29
68.93
30ms
35.59
74.17
30ms
44.51
76.06
29ms
6.7B
Full-rccomputatio Conffict Fast lincoding
32.51
75.56
733ms
41.97
79.41
736ms
50.86
80.53
728ms
Java
6.7B
9.28
39.32
34ms
11.47
39.11
35ms
15.51
41,92
33ms
6.7B
PIE
31.32
73.83
52ms
40.89
77.65
53ms
49.44
78.80
53ms
33B
culluacommttttion
35.05
76.93
2892ms
44.95
80.87
2894ms
$3.23
81.76
2833ms
Java
33B
Conflict Fast Encoding
8.02
32,93
120ms
9.67
33.29
122ms
12.10
34,70
122ms
33B
PIE
33.72
74.69
134ms
43.71
78.66
138ms
51.71
79.49
143ms
4.3
MORE ANALYSIS
In this subsection, we further present detailed analysis investigate how large is the gap between our Positional Integrity Encoding and full re-computation in terms of context representations and predictions from LLMs, which provide additional insight of our approach.
8

---

Published as a conference paper at ICLR 2025
PEE (Ours) and Ful-recmpp..ee
Confict Faxt Entzding and Fulveccunpuuiiion
Coanne Siibaris
0.8
Cosine matratty
033
20
8.
00
Figure 3: Cosine similarity of key representations across model layers. The plots compare the cosine similarity between [j++sn]] and K ++1= (indicating temporal confusion of Conflict Fast Encoding)
with the cosine similarity between K eddit and K []+1=)] (showing the effectiveness of PIE).
[+11=]]
How large is the gap on context representations? In practical scenarios, real-time editing by users results in the modified sequence x requiring the KV cache to must be updated. In our analysis, we use the cosine similarity between context representations of full re-computation i Im]] and (1)
our Positional Integrity Encoding Kedit [=+1m] DeepSeek-Code 6.7B model the Python subset of RepoBench. Averaged results reported.
(2) Conflict Fast Encoding ([++im)) We employ the
In Figure 3, the cosine similarity between representations of full re-computation and our Positional Integraty Enocding is consistently around 1.0 across all layers. This high similarity demonstrates the effectiveness of PIE in preserving the contextual integrity of the key representations after editing, suggesting that PIE successfully mitigates the temporal confusion that typically arises when manipu- lating the KV cache. In contrast, the cosine similarity between representations of full re-computation and Conflict Fast Encoding significantly lower, indicating the temporal confusion issue that hurts model performance a lot.
XF-F
XF-R
Confiit FastEncoding
0.0005
0.0044
KL (LDiergence
KL Divorgence
0.003
0.0000
0.003
0.0000
0/0012
09922
01000
000000
23
20
40
20
82
10
22
30
40
10
20
52
60
Gonerated takrn id
Generated token
Genertted boken id
Figure 4: KL divergence of the generated token distributions. The plots compare the KL divergence between the generated token distributions of PIE and Full-recomputation, and the KL divergence between the generated token distributions of Conflict Fast Encoding and Full-recomputation.
How large is the gap on model predictions? Moreover, we further use Kullback-Leible (KL) divergence as the metric to investigate the gap between model predictions of different approaches. Similarly, employ the DeepSeck-Codee Coder 6.7B model on the Python subset of RepoBench and report the averaged results. In Figure 4, the KL divergence between model predictions of full re-computation and our Positional Integrity Encoding remains consistently low, i.e., below 0.0002 across 64 tokens. However, the KL divergence between model predictions of full re-computation and Conflict Fast Encoding is substantially higher (2x larger). These findings again underscore the importance of maintaining positional integrity within the KV cache to ensure accurate generation results.
5 CONCLUSION
In this paper, we introduce Positional Integrity Encoding (PIE), a novel method designed to enhance the efficiency of large language models (LLMs) in the real-time editing setting. Our approach addresses the significant computational overhead associated with re-encoding contexts after small edits, common scenario in interactive coding environments. Through extensive experiments,

---

Published as a conference paper at ICLR 2025
demonstrated that PIE not only significantly reduces latency but also maintains high accuracy compared to the naive full re-computation method.
PIE represents substantial step forward in the development of efficient LLMs, particularly in dynamic contexts where frequent edits are made. Future work could explore the integration of PIE with other optimization techniques and its application to broader range of tasks beyond code generation. Our method paves the way for more responsive and resource-efficien Al assistants, enhancing their practicality and usability in various real-world scenarios.
ACKNOWLEDGEMENTS
We thank all the anonymous reviewers for the very careful and detailed reviews well the valuable suggestions. Their help has further enhanced our work. Di He is supported by National Science Foundation of China (NSFC62376007).
REFERENCES
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in Ilm inference with sarathi-serve. arXiv preprint orXiv:2403.022. 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv::305.. 10403, 2023.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10771 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023, URL https: www.aawwwi..oggrr....a// org/abs//302..ooo 01318.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 10Xvv:2007.03374 2021.
Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386-8399, 2022.
Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 13522- 13537, 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint -Xiiv:904., 10509, 2019.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XI: Attentive language models beyond a fixed-length context. In acl, July 2019.
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869)) 2023. URL /tatps://arxii wwt.:og////b/aa//e/e 22960000977.71
10

---

Published as a conference paper at ICLR 2025
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems (NeurlPS), 2022. URL https://ppooceedings newriss.eeupppp.. cc/paper #1les/ppperr22222/f55 77657732e2000007718222221111e662
Paper-Confeeeneee pdf.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 Q) 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems www./rr////aaoee 073339.
(NeurlPS), 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 11323-10337, PMLR, 2023.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference Learn- ing Representations, 2022. URL httpp:://pppeee/eeeeeee net/ http://wwwe/eooooooooooooooooooo
IwAR0QoRRF 7gr
Yao Fu. Challenges in deploying long-context transformers: theoretical peak performance analysis.
arXiv preprint arXiv:2405.088: 08944, 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlic Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference Learning Representations, 2024.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li. et al. Deepseck-coder: When the large language model meets programming-the rise code intelligence. arXiv preprint arXiv:2401. 14/96, 2024.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-basee speculative decoding. arXiv preprint arXiv:2311.0822 2023.
Zhenyu He, Guhao Feng. Shengjie Luo, Kai Yang, Di He, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Liwei Wang. Two stones hit one bird: Bilevel positional encoding for better length extrapolation. arXiv preprint 2rXi::2401.1642) 2024.
Itay Hubara, Brian Chmicl, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry, Accelcrated sparse neural training: A provable and efficient method to find transposable masks. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL htttttt///axxiv.orrr Abss2102.081.1
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference machine learning, pp. 5156-5165. PMLR, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445) 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Symposium Operating Systems Principles (SOSP), 2023. URL https::/arx.v.or///bb//200 061 80.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning (ICML), 2023. html.
URL hww.hhtps///t...m. http:p://pproeeeee http://eooceeenns/e/eee/ mlr wwt.sp/ew2/eeeeiiaeeeeeooo
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024.

---

Published as a conference paper at ICLR 2025
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-Ievel code auto-completion systems, 2024a. URL https://arxiv wwwp://..rrgbbb//////// /30.//00.600000...
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad. Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. 2023. URL htttps://arxiii org/ 2//abb2/2555 17888.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Seissorhands: Exploiting the persistence of importance hypothesis for 11m kv cache compression test time. Advances in Neural Information Processing Systems, 36, 2024b.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102. 246644 2021.
Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang. and Tie- Liu. Stable, fast and accurate: Kernelized attention with relative positional encoding. Advances in Neural Information Processing Systems, 34:22795-22807 2021.
Shengjie Luo, Shanda Li, Shuxin Zheng, Tin Liu, Liwei Wang, and Di He. Your transformer may not be as powerful as you expect. Advances in Neural Information Processing Systems, 35: 4301-4315, 2022.
Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie- Liu, Liwei Wang, and Di He. One transformer can understand both 2d & 3d molecular data. The Eleventh International Conference on Learning Representations, 2023. URL /ttpss//openrevieww net/
Xinyin Ma, Gongfan Fang, and Xinchao Wang. L.m-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL / org pdf 2205.. 11627.. fdf...fff........
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng. Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative Ilm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023. URL https:///axiii wow.:orgggaoooo..o http//bs//23000 2305. 09781.
Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.095557 2022. URL https://arxxv.orr/abb/2222 2tt.//2260.000 09557..
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-224 2023.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference Learning Representations (ICLR), 2022.
Jiezhong Qiu, Hao Ma. Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self- attention for long document understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2555- 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):54855555 2020.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
12

---

Published as a conference paper at ICLR 2025
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arX11:2308.12299 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. URL https://arxias http://rxxii.oorrr///////a// www.obs////// 19t../99.......0 01108.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Association for Computational Linguistics (ACL), June 2018.
Ying Sheng. Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pp. 31094-31116, PMLR, 2023.
Benjamin Spector and Chris Re. Accelerating Ilm inference with staged speculative decoding. arXiv preprint 2rXi::2308.04623,, 2023. URL pnttts://arxtte http://aaxxiv.ooabbb/////ss/ wrgg/bs/////// 04623.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit.
Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NeurIPS), 2018. URL https://procceeddng wtt.s meurips.cc/paperr http:/ppprr..llleppppr///////// www.eeeddinnsnuuusss......
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma. Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. length-extrrppolatbbl transformer. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, July 2023.
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp. 1433-1443, 2020.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference Machine Learning, pp. 9438-9447 PMLR, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09228. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium Computer Architecture (HPCA), 2021. URL https: www.aawwwwwwg.////////22 09852..
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https: aclwb www.:/wwwww..../.nntoooloo.... wwwgg/gthloogy/////oooooo 2020. emnlp-demos. 6.
13

---

Published as a conference paper at ICLR 2025
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post- post-training quantization for large language models. In International Conference Machine Learning (ICML), 2023. URL https://proceedinng. mlr press/ v202/xiao23c/xiaa233..
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Tweffth International Conference on Learning Representations, 2024. URL https://openrrvieew meet/orrum11GNN7755111
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://proceedins.neeee neurips. cc/paper files/paper/2022/Fiie pdf.
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie Yan Liu. Do transformers really perform badly for graph representation? Advances in neural information processing systems, 34:28877-28888. 2021.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023. net/forum?id==Awwee
URL https::/
Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of GNNs via graph biconnectivity. In The Eleventh International Conference Learning Representations, wpp.//veeerreeee..eooeeoev net
2023a. URL
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.0m 08168, 2023b.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang. Andi Wang, Yang Li, al. Codegeex: A pre-trained model for code generation with multilingual bench- marking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5673-5684, 2023.
A EXPERIMENTAL DETAILS
A. EXPERIMENTAL SETUP
Tasks Construction for Code Insertion. To simulate code insertion tasks, we start by randomly deleting five consecutive lines from each context. The resulting context, which lacks these five lines, considered the original context. The complete context, which includes the previously deleted lines, is treated as the edited context. The tokens within the deleted lines are identified the inserted tokens (around 64 tokens for Python and 51 tokens for Java). This setup allows us to evaluate the model's 's capability to accurately restore missing code segments, mimicking real-world scenarios where developers frequently insert blocks of code.
Tasks Construction for Code Deletion. For code deletion tasks, we begin by randomly selecting a line within the context and then inserting five randomly sampled lines at this position. The context containing these additional lines is designated as the original context. The complete context, which excludes the inserted lines, is regarded as the edited context. The tokens in the inserted lines are treated as the deleted tokens (around 64 tokens for Python and 51 tokens for Java). This construction enables us to assess the model's 's performance in identifying and removing extrancous code, reflecting situations where developers need to clean up refactor their codebase.
14

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparisons of insertion experiments for CodeLlama. In this task, for each next-line prediction target, we insert several lines of code into its context randomly to simulate real-world scenarios. EM and ES denotes the Exact Match and Edit Similarity score respectively. All results demonstrate that our Positional Integrity Encoding approach brings substantial speed-ups without performance drops.
XF-F
XF-R
Model
Method
EM
ES
Time
EM
ES
Time
EM
ES
Time
Python Python
7B
mull-recompttttion Conflict Fast Encoding
25.72
66.49
561ms
38.87
73.81
564ms
33.55
70.26
562ms
55.10
34ms
19.32
$7.34
34ms
15.49
53.05
34ms
7B
PIB
25.42
66.40
50ms
38.76
73.83
50ms
33.30
70.11
50ms
34B
Full-ecoomuutation
31.04
69.48
2013ms
42.80
76.27
2029ms
37.69
72.36
1999ms
34B
Conflict Fast licooding
11,56
50.23
113ms
16.06
52.59
110ms
11,58
48.36
11 112ms
34B
PIE
30.53
68.79
123ms
42.73
76.40
119ms
37,51
72.25
123ms
7B
Fulltrccoputation
28.02
72.49
578ms
39.61
77.81
578ms
49.20
79.76
578ms
Java
7B
Conflict Fast Encodiig
10.45
37.69
34ms
14.28
38.45
35ms
16.95
38.15
33ms
7B
PIE
28.02
72.39
50ms
39.48
77.74
49ms
49.31
79.79
48ms
34B
cullpacommttation
32.16
75.09
2179ms
43.45
79.89
2184ms
52.43
80.97
2197ms
Java
34B
Confict Fast Encoding
12.04
41.76
104ms
14.25
39.19
107ms
17.29
38.50
109ms
34B
PIE
31.45
74.64
119ms
43.54
79.80
118ms
52.42
81.00
115ms
Tasks Construction for Multi-place Code Edition To comprehensively evaluate the model's s performance in handling simultaneous code insertion and deletion, construct task scenario that integrates both operations. Initially, we randomly delete five consecutive lines from each context to simulate code insertion. The context without these lines is treated as the original context. The tokens in the deleted lines are identified as the inserted tokens.
Simultaneously, we randomly select another line within the context and insert five randomly sampled lines at this position. The complete context, which includes all lines as they appear after both deletions and insertions, is regarded the edited context. The tokens in the newly inserted lines considered the deleted tokens. This dual operation setup allows us to evaluate the model's 's ability to handle complex, simultancous edits, adding missing code segments while removing extraneous ones, reflecting the multifaceted nature of real-world coding environments where developers often perform multiple types of edits concurrently.
A.2 RESULTS ON CODELLAMA
Results of different code editing settings on CodeLlama (Roziere et al., 2023) are presented in Table 5, 6 and 7 respectively. Similar to SrepSeek-Coder CodeLlama with Positional Integrity Encoding demonstrates strong performance and fast speed across various editing scenarios. The metrics indicate that PIE effectively maintains prediction accuracy and addresses temporal confusion, ensuring the impact of minor edits minimal.
B ADDITIONAL EXPERIMENTS
B.1 CODE GENERATION TASKS
To further validate the effectiveness of PIE, we conduct experiments on code generation tasks, using HumanEval Chen et al. (2021) and its C++ version from HumanEval-X Zheng et al. (2023). We randomly choose three consecutive lines in the prompt as the user inserted code and report Pass for different methods. It's important to note that the context (prompt) in HumanEval consists of only a few lines (with an average of 15 lines), making each edit semantically significant. The results in Table 8 demonstrate that PIE significantly outperforms the baseline methods, approaching the performance of full recomputation. Note that we also include a reuse baseline where the pre-edit context is reused without modification. Given the importance of these edits in such short contexts, updating the KV cache with PIE becomes essential for maintaining high prediction accuracy, whereas the performance of the reuse baseline drops substantially.
15

---

Published as a conference paper at ICLR 2025
Table 6: Performance comparisons of deletion experiments for CodeLlama. In this task, for each next-line prediction target, we delete several lines of code of its context randomly to simulate real-world scenarios. EM and ES denote the Exact Match and Edit Similarity score respectively. All results demonstrate that our Positional Integrity Encoding approach brings substantial speed-ups without performance drops.
XF-F
XF-R
Model
Method
EM
ES
Time
EM
ES
Time
EM
ES
Time
Python Python
7B
mullprecompttttion Conflict Fast Encoding
25.72
66.49
561ms
38.87
73.81
564ms
33,55
70.26
562ms
10.14
52.80
30ms
15.24
$7.88
31ms
13.04
54.53
30ms
7B
PIB
25.52
66.47
43ms
38.88
73.87
42ms
33.51
70.18
42ms
34B
Full-ecoomuutation
31.04
69.48
2013ms
42.80
76.27
2029ms
37.69
72.36
1999ms
348
Conflict Fast lincoding
12.15
54.76
88ms
16.40
59.29
94ms
13.78
55.90
93ms
34B
PIE
31.01
69.44
100ms
42.76
76.13
100ms
37.60
72.38
102ms
7B
Fulltrccoputation
28.02
72.49
578ms
39.61
77.81
578ms
49.20
79.76
578ms
Java
7B
Conflict Fast Encoding
7.21
42.51
29ms
7.77
43.86
29ms
8.78
44,46
29ms
7B
PIE
28.01
72.45
42ms
39.51
77.78
42ms
49.30
79.80
42ms
34B
cullpacommutttion
32.16
75.09
2179mss
43.45
79.89
2184ms
52.43
80.97
2197ms
Java
34B
Confiict Fast Encoding
8.76
45.44
92ms
8.67
45.57
92ms
10.11
46.64
95ms
34B
PIE
32.12
75.02
99ms
43.33
79.86
98ms
52.36
80,96
107ms
Table 7: Performance comparisons of edition experiments for CodeLlama. In this task, for each next-line prediction target, we delete several lines of code of its context and simultaneously insert other lines of code randomly to simulate real-world scenarios. EM and ES denote the Exact Match and Edit Similarity score respectively. All results demonstrate that our Positional Integrity Encoding approach brings substantial speed-ups without performance drops.
XF-F
XF-R
IF
Model
Method
EM
ES
Time
EM
ES
Time
EM
ES
Time
Python Python
7B
full-recomputation
25.72
66.49
705ms
38.87
73.81
706ms
33.55
70.26
713ms
7H
Conflict Fast Hinooding
19.97
61.46
34ms
29.79
67.31
34ms
24.70
63.28
34ms
7B
PIE
24.97
66.04
54ms
38.21
73.40
54ms
32.84
69.66
54ms
34B
Full-rccopuuaaton
31.04
69 48
2574ms
42.80
76.27
2569ms
37.69
72.36
2581ms
Conflict Fast lincuding
21.91
61.78
91ms
30.87
67.16
91ms
25.76
63.15
90ms
34B
PIB
29.92
68.7
126ms
41.74
75.37
126ms
37.01
71.76
127ms
7B
culputaomtttttion
28.02
72.49
733ms
39.61
77.81
736ms
49.20
79.76
728ms
Java
7B
Conflict Fast Encoding
18.78
57.38
34ms
25.79
60.97
35ms
32.60
62.66
33ms
7B
PIB
27.43
71.60
52ms
38.89
76.96
53ms
48.60
78.94
53ms
34B
Full-rccomattaiion
32.16
75.09
2726ms
43.45
79.89
2807ms
52.43
80.97
2727ms
Java
34B
Conflict Fast Encoding
21.25
59.70
104ms
27.83
61.91
107ms
32.98
62.03
109ms
34B
PIB
30.82
73.84
126ms
42.71
78.94
119ms
51.64
80.15
125ms
B.2 CONTEXTUALLY RELATED EDITS
We design a contextually related insertion setting, where we use Edit Distance identify lines in the context that are similar to the target line, simulating user insertions. We also increase the number of edited lines. We conduct experiments on the Cross- Cross-File-Firs setting of RepoBench-C--e C--k, keeping all other settings the same as Section 4.The results in Table 9 demonstrate that PIE is robust in handling contextually related edits and outperforms the baseline methods, approaching the performance of full recomputation. Note that we also include reuse baseline where the pre-edit context is reused without modification.
B.3 MULTI- PLACE EDITS
Users may edit a function definition followed by updates to all associated call sites. To simulate this scenario, we treat each line in the context as potential site and use Edit Distance to identify multiple sites in the context that are similar to the target line, thereby simulating multi-place insertions. We conduct experiments in this contextually related multi multi-place editing setting on the Cross-File Files-FieFFFiii task of HepoBench-C-ae The results in Table 10 demonstrate that PIE robust handling multi place edits and outperforms the baseline methods.
16

---

Published as a conference paper at ICLR 2025
Table Performance of DeepSeck-Coder 6.7B on HumanEval and HumanEval-X
Pass @
Method
Python
C++
Full-recomuutation
49,4
50.3
Conflict Fast Encoding
11.0
15.5
Reuse baseline
17.7
29.2
PIE
43.9
49.7
Table 9: Performance of DeepSeek- Codeer 6.7B on contextually-relan insertion experiments
lines
lines
11 lines
13 lines
15 lines
Model
Method
FM
ES
EM
ES
FM
ES
EM
FS
RM
FS
Python
6.7B
Full-recomputation
28.95
70.11
28.95
70.11
28.95
70.11
28.95
70.11
28.95
70.11
6.7B
Conflict Fast Encoding
7.92
41.56
6.39
38.56
4.78
34.91
4.11
33.34
3.43
32.15
6.7B
Reuse baseline
25.58
68.23
25.29
68.03
25.23
67.95
67.83
24.71
67.74
6.7B
PIE
27.40
69.10
27.23
68.94
27.04
68.80
26.92
68.8
26.86
68.63
6.7B
recomputaaiioo
32.51
75.56
32.51
75.56
32.51
75.56
32.51
75.56
32.51
75.56
Jasa
6.7B
Conflict Fast Encoding
2.81
15.46
1.89
12.16
7.89
6.52
0.74
5.57
6.7B
Reuse baseline
25.99
73.29
25.72
73.17
25.18
73.03
25.15
72.99
24.99
72.84
6.78
PIE
30.04
74.46
30.00
74.44
30.13
74.47
29.87
74.38
29.76
74.29
Table 10: Performance of DeepSeek-Coder 6.7B on contextually-related multi-place insertion experi-
ments
5 lines
lines
11 lines
13 lines
15 lines
Model
Method
EM
ES
EM
ES
ES
ES
EM
Python
6.7B
Full recomputaainnn
28.95
70.11
28.95
70.11
28.95
70.11
28.95
70.11
28.95
70.11
6.7B
Conflict Fast Encoding
7.43
39.95
5.83
37.00
4.06
33.02
3.48
31.71
2.97
30.59
6.7B
Reuse baseline
24.61
67.43
24.44
67.17
23.84
66.77
23.62
66.63
23.49
66.55
6.7B
PIE
26.97
68.85
26.77
68.67
26.33
68.34
26.15
68.222
25.9
68.23
6.7B
Full-recomputation
32.51
75.56
32.51
75.56
32.51
75.56
32.51
75.56
32.51
75.56
Java
6.7B
Conffict Fast Encodding
1.69
10.91
1.06
7.66
0144
4.26
0.28
3.36
0.22
2,81
6.7B
Reuse baseline
24.46
72.35
23.99
72.05
23.11
71.69
22.97
71.56
22.75
71.38
6.7B
PIE
29.56
74.14
29.26
73.91
29.03
73.81
28.83
73.68
28.56
73.59
17


Reviews:
[{"review_id": "pZQ1uzzNXp", "reviewer": "Reviewer_qWyU", "summary": "This paper introduces a technique to update the rotary positional encoding when a small part of the context tokens are updated. It aims to optimize computational efficiency in real-time editing scenarios. The authors show that by fixing the positional encoding alone, they were able to retain a performance that is almost as good as fully re-encoding the context on a left-to-right code generation task, but with considerably less computational cost.", "questions": "Could you provide performance results for a baseline that reuses the pre-edit context without modification for making suggestions? This zero-cost approach would be helpful to compare against Full-recomputation in your benchmark.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The paper effectively outlines the real-time editing problem and clearly describes the mathematical foundation for PIE based on rotary positional encoding.", "weaknesses": "**Limited Technical Novelty**: The mathematical derivation is relatively straightforward, stemming directly from rotary positional encoding's relative nature, without additional innovation or complexity.\n\n**Unrealistic Setting for Interactive Editing**:\n * Random Edits Only: The experimental setup evaluates PIE on random edits, which does not align with realistic real-time editing workflows, where temporally or contextually related edits are more common (e.g., editing a function signature and then updating its call sites). PIE’s simplifications may therefore be less applicable to typical usage patterns.\n * Single-Edit Evaluation: The paper evaluates PIE in a single-edit scenario, overlooking the potential for accumulated errors in multi-edit settings. In practical applications, users often make multiple edits, which could introduce drift in the positional encoding without full recomputation.\n * Left-to-Right Only: Evaluations are limited to left-to-right generation, omitting fill-in-the-middle (FIM) generation, a task relevant in code editing where users may modify code segments in the middle of sequences. Without this, it is unclear how PIE would perform in varied editing tasks.\n\n**Unconvincing Conclusions Due to Limited Evaluation Scope**: Given the unrealistic evaluation settings, the claim that PIE can retain performance by adjusting positional encoding alone is unconvincing. By testing only on random edits, the experiments fail to address cases where contextual dependencies (e.g., edits that affect other tokens' relevance) might demand full recomputation. This risks overstating PIE’s applicability to real-world editing scenarios. To build a more compelling case, I recommend:\n * Evaluating PIE on real user edit sequences rather than synthetic random edits.\n * Restricting comparisons with full recomputation to cases where edited tokens impact final target tokens meaningfully (e.g., by verifying that removing or masking these edited tokens affects the target token prediction likelihood).\n * Including a special baseline where the pre-edit context is reused without modification, establishing a zero-cost apporach for comparison.\n\n**Lack of Multi-Edit and Accumulated Error Analysis**: With each edit, additional errors will enter the encoding under the proposed technique, but the paper provides no analysis of error accumulation across multiple edits. Without such discussion, it’s unclear when a full recomputation might be needed to reset the encoding.\n\n**Lack of Fill-in-the-Middle Evaluation**: Evaluations are limited to left-to-right generation, omitting fill-in-the-middle (FIM) generation, which is more relevant to the interactive coding assistant scenarios mentioned by the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KrQPnf0oXB", "reviewer": "Reviewer_Yphe", "summary": "The paper presents Positional Integrity Encoding (PIE) as an inexpensive alternative to re-encode the entire KV cache during LLM decoding specifically for code related tasks. PIE solves the temporal confusion task efficiently using a single round of matrix-multiplicaton. The authors provide results on RepoBench dataset using 3 different sizes of DeepSeek-Coder model for 3 tasks- code insertion, deletion and multi-place editing. The results show that PIE reduces computational overhead by 85% compared to full-recomputation without significant loss of performance.", "questions": "1. Can you please add evaluations for 1 more dataset and 1 more model?\n2. How does the approach do for other non-code related tasks where semantic relationship is important?\n3. How does the approach do for longer edits?\n4. Is there any memory overhead of the proposed approach?\n5. Does this approach lead to any cumulative errors if you continue to update KV cache based on PIE?\n6. The average scores of 3 experiments are reported, could you also report the standard deviation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand.\n2. The authors solve an important task of efficiency in updating KV cache in a real-time code editing setting. This is crucial for interactive coding assistant scenario where the developers make frequent and incremental changes to the exisiting code and require copilot to correctly predict the next line on the fly.\n3. The authors perform experiments on 1 dataset for 3 tasks and show 85% reduction in computational overhead compared to brute-force approach.", "weaknesses": "1. The results are limited to 1 dataset and 1 model. Including more than 1 dataset and model would make the claim more strong.\n2. The authors solve an important task of efficiency of real-time code editing but do not discuss the limitations of this approach for other tasks where semantic impact is large or in case of large code edits.\n3.The approach has a dependency on RoPE and might not be suitable for other models without RoPE", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhmgLjVbt0", "reviewer": "Reviewer_Bo49", "summary": "This paper describes a new approach called Positional Integrity Encoding (PIE) that aims to improve the efficiency of Large Language Models in real-time code editing scenarios.\n\nPIE improves the accuracy and efficiency of predictions by solving the problem of the computational overhead associated with recoding the entire context when editing existing code.", "questions": "1. Can the PIE be effective on code generation tasks?\n\n2. Why not consider the Pass@1 metric?\n\n3. Is PIE equally valid on other models (Qwen-2.5-Coder; Llama-3.1; Yi-Coder)?\n\nI am more concerned about the third of the above issues. If the supplemental results do show the validity of PIE, I will raise my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I really enjoy this paper!\n\nThe Positional Integrity Encoding (PIE) introduced by the authors capitalizes on RoPE, adeptly addressing temporal disorientation by initially stripping away the rotary matrices responsible for confusion and subsequently reinstating the appropriate matrices through straightforward matrix multiplication.\n\nThis capability to enhance computational efficiency without compromising accuracy is precisely the straightforward yet potent approach we value in the realm of language model optimization.\n\nThe PIE not only paves the way for future research in optimizing Large Language Models (LLMs), particularly focusing on efficiency, but also excels in real-time dynamic scenarios. Its compatibility with existing acceleration techniques positions PIE as a catalyst for further advancing the practical deployment of LLMs.", "weaknesses": "My current concerns are regarding the selection of downstream tasks and evaluation metrics considered by the authors.\n\n(1) The tasks of code insertion, code deletion, and multi-place code editing that the authors have considered seem less critical and common in actual development scenarios compared to code generation.\n\n(2) The chosen evaluation metrics, EM (Exact Match) and ES (Edit Similarity), may not accurately assess the semantic correctness of the generated code.\n\n(3) The selection of models is limited to the DeepSeek-Coder series.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6AnPJ0vOa5", "reviewer": "Reviewer_RBWU", "summary": "This paper proposes Positional Integrity Encoding, an algorithm to correct rotary positional encoding in the KV cache of displaced tokens due to edits. It shows significant speed-up compared to full recomputation.", "questions": "- Have you conducted more experiments on more settings and models?\n- Is it possible to provide an in-depth *theoretical* analysis (in 4.3) of the cause of performance drop between PIE and the recomputation?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed algorithm significantly reduces the latency of generation with edited prefixes.\n- A straightforward solution to adjust the rotary matrices to correct temporal confusion.", "weaknesses": "- The algorithm looks somewhat trivial to me. It simply corrects the position embedding by transforming the rotary matrix in RoPE.\n- Limited experiments. All performance experiments were conducted on RepoBench-C-8k and DeepseekCoder. It is unclear if this algorithm generalizes to other settings and models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)