Rethinking Residual Distribution in Locate-then-Edit Model Editing
Authors: Xiaopeng Li
Shangwen Wang
Shasha Li
Shezheng Song
Bin Ji
Ma Jun
Jie Yu
Venue: NeurIPS 2025 poster ()
Keywords: Model Editing; Large Language Models; Knowledge Editing; Locate-then-edit
Primary area: deep_learning

Abstract: Model editing enables targeted updates to the knowledge of large language models (LLMs) with minimal retraining. Among existing approaches, locate-then-edit methods constitute a prominent paradigm: they first identify critical layers, then compute residuals at the final critical layer based on the target edit, and finally apply least-squares-based multi-layer updates via $\textbf{residual distribution}$. While empirically effective, we identify a counterintuitive failure mode: residual distribution, a core mechanism in these methods, introduces weight shift errors that undermine editing precision. Through theoretical and empirical analysis, we show that such errors increase with the distribution distance, batch size, and edit sequence length, ultimately leading to inaccurate or suboptimal edits. To address this, we propose the $\textbf{B}$oundary $\textbf{L}$ayer $\textbf{U}$pdat$\textbf{E (BLUE)}$ strategy to enhance locate-then-edit methods. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE not only delivers an average performance improvement of 35.59\%, significantly advancing the state of the art in model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at https://github.com/xpq-tech/BLUE.

Rethinking Residual Distribution in Locate-then-Edit Model Editing
Xiaopeng Li Shangwen Wang
Shasha Li
Shezheng Song Bin Ji Jun Ma" Jie Yu"
National University of Defence Technology shashali, ss211,,,,2,,,,.,,,,,,,,,, mia.iiiiamnmnmn........ majun, ma) ynudt edu. cn
(xiaopengli, wanghhagggggggnggennnnn i
Abstract
Model editing enables targeted updates the knowledge of large language models (LLMs) with minimal retraining. Among existing approaches, locate-then-edit methods constitute prominent paradigm: they first identify critical layers, then compute residuals at the final critical layer based on the target edit, and finally apply least-squares-
multi-layer updates via residual distribution. While empirically effective, we identify a counterintuitive failure mode: residual distri- bution, a core mechanism these methods, introduces weight shift errors that undermine editing precision. Through theoretical and empirical analysis, show that such errors increase with the distribution distance, batch size, and edit sequence length, ultimately leading to inaccurate suboptimal edits. To address this, we propose the Boundary Layer UpdatE (BLUE) strategy to enhance locate-then-edin methods. Sequential batch editing experiments demonstrate that BLUE not only delivers an average performance improvement of 35.59% significantly advancing the state of the model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at httpss/ggihub.coo//nttttee/////
three LLMs and two datasets
Introduction
Large language models (LLMs) possess powerful comprehension and generation capabilities and have become foundational infrastructure for various Al applications. However, the knowledge encoded in the parameters of LLMS limited to the training data and cannot be updated to reflect changes in world knowledge. Updating the parameters of LLMs through retraining to keep them in sync with world knowledge entails high computational costs [I]. Recently, model editing has garnered increasing attention a promising technique for efficiently updating the parameterized knowledge in LLMs, which aims to correct erroneous or outdated knowledge within LLMs without compromising their other capabilities (2].
Locate-then-ede methods are a prominent family of model editing techniques. They treat the Feed- Forward Network (FFN) key-value memory 3 and update the key layers responsible for storing factual knowledge using a least-squares solution. Specifically, these methods first employ causal tracing analysis to identify multiple critical layers within LLMSs that encode factual information. They then use optimization techniques to compute the residuals required to update the final critical layer. Finally, the residuals distributed evenly from the first the last critical layer, and the updates applied using a least-squares solution [4 process referred to as residual distribution.
Although locate-then then-edit methods have achieved remarkable performance model editing tasks (o). we identify a counterintuitive failure mode: residual distribution, a core mechanism in these methods, introduces weight shift errors that undermine editing precision. Specifically,
Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

first empirically demonstrate that the contribution of residual distribution to model editing di- minishes as the distribution distance increases and that the distributed residual is not the opti- mal residual for editing. Subsequently, we theoretically find that the upper bound of weight update errors increases with: (a) the size of the editing batch, (b) the number of sequential ed-
its, and (c) the residual distribution distance. These findings indicate that residual distribu- tion can actually negatively impact the model editing of the locate-then-edit approaches.
Therefore, we propose the Boundary Layer UpdatE (BLUE) strategy, which enhances ppeate-then-edit methods by updating only the first and last critical layers through direct com- putation of residuals, without residual distri- bution. The comparison between BLUE and existing locate-then-edit methods is shown in Figure 1 We apply BLUE to enhance MEMIT
Compuning reviiual x² S. Updating Wat using g
Computing resitual 8, Updriigg W using x
Layer Su.at U and ising
Laper
No Updase
distributed resiuaal R²/3
Updale Order
Uplating Using Datriuia Roodeal
No Upshoc
[ RECT Z7. PRUNE (8] and AlphaEdit
6 Results from 12 sequential editing ex- periments conducted on three LLMs and two datasets show that BLUE improves the perfor- mance of existing
Sy Updating Wi, 1-1 usig distriutted cesidual /n
Sy: Compatinng noidual gfr
St: Updazing Wis erising gl
methods by an average of 35.59%. Our further analysis on downstream tasks and representation shift demonstrates that BLUE also enhances the ability of bccae-thhnn-edi methods to preserve general capabilities and mitigates representa- tion shifts in the post-edit LLMs. Furthermore, we show that BLUE not only improves editing efficiency, but also strengthens the performance of locate-then-edit methods in long-form model editing 9 10 scenarios. In summary, our contributions are as follows:
Existing locate then edit
BLUE
Figure 1: Comparison of existing locate-then-edit methods and BLUE. The n critical layers are edited, where L denotes the last critical layer.
Through empirical and theoretical analysis of residual distribution, reveal that residual distribution of locate-then-editing methods actually leads to inaccurate model editing.
We propose the BLUE strategy, which discards residual distribution and enhances existing locate-then then-edit methods by updating only the first and last critical layers through direct residual computation.
Experimental results show that locate-then-edit methods enhanced with BLUE outperform the original methods, better preserve LLMs' general capabilities, and further mitigate representation shifts in the post-edit I.L.Ms.
2
Related Work
Model editing can be categorized into parameter-preserving and pardiyttr-moddilyinn approaches, depending whether the original model parameters are altered.
Parameter-preserving model editing employs techniques like prompt engineering or attaching additional parameters [1]] A representative method for prompt engineering is IKE [15), which retrieves n contexts of the edited knowledge for a query to guide the model's response without altering its internal parameters. Methods that attach additional parameters include SERAC [16] and GRACE [7, which store new memories externally and internally, respectively, by introducing new parameter modules.
Parammeer-mmodifyiing approaches achieve model editing by directly or indirectly adjusting model parameters [18] [0]. Direct methods, such as FT-L [19), perform constrained fine-tuning on a small number of layers integrate new knowledge. Indirect methods can be divided into meta-learning and locate-then-edit methods. Meta-learning methods, like MEND 20), leverage a hypernetoork to transform edit-related representations and gradients into parameter updates. In contrast, locate- then-edit methods, such as ROME [ and MEMIT (5), adopt a key-value memory perspective to identify and update single or multiple critical layers using least-squares optinization. Among these, locate-then-edii methods have gained popularity and inspired several variants, such as PMET (1], which focuses on precise editing, and AlphaEdit [6, which enhances the retention of original knowledge and strengthens sequential editing capability.

---

3 Background
3.1 Model Editing Problem
Model editing aims to efficiently update the knowledge of L.LMs so that they remain in real-time sync with reality [22]. Factual knowledge changes rapidly, making its update in L.MSs pressing need. Factual knowledge can be represented as a triplet (s, o), where is the subject, is the relation, and is the object. This knowledge can be transformed into a prompt p; + o, where p; is an element of the set P that expresses the semantics of (s, r) in natural language. The element that most directly expresses the semantics of (s, r) is called p. while the rest elements are pr. The goal of model editing is to redirect the object o in the triplet to new object o", represented as t = (s. r, o) - o". To evaluate whether the post-edit model effective on the post-edit knowledge triplet and does not affect other triplets, assessments are made from three aspects: efficacy, generalization, and specificity. Efficacy evaluates whether the model's prediction on p is redirected to Generalization evaluates whether the model's prediction on Pr is redirected to o". Specificity evaluates whether the model maintains its original predictions on inputs outside the set P. To evaluate the generative capability of evaluation metrics. Fluency measures the degree of repetition in the text generated by the model after editing; higher repetition indicates lower fluency. Consistency evaluates the degree of alignment between the content generated by the post-edit model based on s and the reference text of the subject associated with the new object o". For more details, please refer to [4]
the post-edit model, [4] also uses fluency and consistency
Model editing can be categorized according to whether the editing is sequential and the batch size into sequential editing, batch editing, and sequential batch editing [23]. Sequential editing refers to the continuous editing of a single piece of knowledge, while batch editing involves editing multiple pieces of knowledge at once. Sequential batch editing combines these two scenarios, involving the sequential editing of batch knowledge. This problem definition is highly relevant to the ever-changing nature of bulk knowledge in practice. Therefore, we directly present the problem of sequential batch editing. Suppose there is a sequence of n knowledge sets be updated: [TT, T2, ........ Tn]. where each knowledge set T, {t1, ... Sequential batch editing requires that after performing sequential batch edits, the post-edit model can successfully predict all n knowledge sets without affecting knowledge outside these sets.
3.2 Locate-then-Ead Model Editing
The locate-then-edie model editing is one of the most popular series of model editing methods M. These approaches typically use causal tracing [4] to identify the critical layers L where knowledge is stored, and then compute weight shifts using least squares modeling to update the weights.
Specifically, they view the feed-forward network (FFN) as key-value memories [3. Let h be the residual stream of the 1 1 layer, and a' be the output of the self-attention block of the E L layer.
The kay-value memories of the FFN can be represented as follows:
m'
W (W ((h a')
(1)
value
key := k
where m' is the output of the FFN block, and W and W out the input and output mapping weights of the FFN block, respectively. a and y are activation functions. W W/ is viewed as linear associative memory that associates keys and values:
K! k k ... J M¹ [mb.. lm..2l..//m
Before editing, the linear associative memory satisfies:
W! arg min
W - M
W
When new memories need to be inserted, new group of keys K and values M will be updated into Wi. Thus the new weight should satisfy:
W arg min W
WK
Mj 1
+ | -
M
preserve old
insert new
3

---

Let W = W/ + A' where A' is weight shifts. By applying the normal equation to Eq. 4 its closed-form solution can be written as:
A' = R'K KK-K' K(K'"
(5)
where R' - (M w is the residual of the new memories when evaluated on old weights W [ Ki and KjJ are computed for each layer. In most locate-then-edit methods [5] 6], the residual of layer evenly distributed from the residual last critical layer L = max(C):
R
M
W
R
(6)
L-1+1
where - = [mf mz |...||...._.. ...!m represents u entries of new memories. Each entry computed using the following formula:
m h + 8f-- Wo + +
where 8f a residual vector optimized by:
mf = h. + arg min 8
Σ j=1
log
e
(8)
The objective of the above equation is to optimize the learnable 6.¹ to maximize the probability of the model predicting o". æj @ p represents the concatenation of the jth randomly generated P prefixes by the model with prompt p enhance generalization. 0hb + 5.¹) denotes adding 8. to h
Fang et al. ( extends thenee-t methods the sequential batch editing scenario. They cache the keys K₂p of previously edited knowledge and incorporate K₃ into the least squares optimization, ultimately deriving the following closed-form solution for sequential batch editing:
A R'K!" K + K/KK' T + K(
4 Rethinking the Residual Distribution of Locate-then-Edit Model Editing
In this section, we analyze the residual distribution both empirically (Section 4.1) and theoretically
(Section 4.2). Based these insights, further propose novel strategy enhance locate-then- edit model editing (Section 4.3). We focus on the classic locate-then-edit model editing method, MEMIT (5). Experiments are conducted on three LLMs:
(24), GPT-J (6B) (25). and GPT2-XL, using the CounterFact dataset 4. Unless otherwise specified, we use the first 200 samples from the CounterFact dataset. The critical layers analyzed for each model are: Llama3-8B: {4, 14,5,60 6, 7, 8J, GPT-J (6B): (3, 5,5, 6, 7, and GPT2-XL: (13, 14, 15, 16,17) 17).
4.1
Analyzing Residual Distribution in Locate-then-Eale Model Editing
4.1.1 How Does the Distributed Residual Contribute to the Editing Object?
To measure the contribution of the distributed residuals to the editing object, we first define contribution score:
= Pa- - where Pa- (o*lp) represents the probability of the post-edit model 0" regarding the edited knowledge t= (s, o- The rationale behind this is that the probability of the pre-edit model 0 assigning to o" on knowledge / is often low, while the model editing aims for the post-edit model to assign the highest probability to o".
(10)
From Equ. T and Equ. (4). we can know that the essence of soccatethhencee is that the post-edit model can activate the new memory m. mb. ; + 8 /L l + 1) in the FFN block at layer / using the key ky corresponding to p, where m/, represents the original memory of the model. Therefore, we directly replace the output of the FFN block at layer / with the new memory m to eliminate the potential impact of activation failure. For comparison, we also directly compute residuals for each layer, following the same process as m By using the distributed residual for simulated editing, we can accurately measure the contribution of new memories while avoiding direct edits to the model.

---

Lional
Computed
We perform simulated editing in each critical layer. The average contribution scores are shown in Figure For the distributed residuals, it can be observed that only the last critical layer achieves contribution score close to 1.0. The contribution scores of the other layers were all below 0.7, showing a decreasing trend layer by layer. For the first critical layer, the contribution score is below 0.1 in three LLMs. This indicates that the farther the residuals distributed, the lower their contribution to the editing object. Even distribute through just one layer lead to significant drop in the contribution score. In contrast, for the computed residuals, their contribution scores in each layer consistently approach 1.0.
Dist ributed
GFTZAL
17
16
15
Lager
Figure 2: The average contribution score of different simulated editing layers.
4.1.2 Is the Distributed Residual the Optimal Residual for Editing?
From Section 4.1.1 we observe that directly comput- ing m for each layer achieves high contribution scores, Therefore, we assume that the directly computed m rep- resents the optimal memory for editing. To verify whether residual distribution is optimal, we first compare the sim- ilarity between residual distribution and the directly com- puted m and then evaluate their performance in model editing.
1lama3
0.5
5
ang
0.5
0.0
0.5
0.0
Similarity Analyzing. The variation cosine similarity between the distributed and the directly computed m, is shown in Figure 3 It shows that the cosine similarity between the distributed and the directly computed m hibits a mayerbb-layee decreasing trend, indicating that the further residuals are distributed, the farther m devi- ates from the optimal memory. To further investigate how residual distribution affects model editing performance, we next perform single-layer model editing using both the distributed residuals and the directly computed residuals.
16
14
Layer
Figure 3: The variation in cosine sim- ilarity between the distributed and the directly computed memory across differ- ent layers.
Post-edit LLM Performance. We update single layers of the model using distributed residuals and computed residuals, respectively. The results for Ilama3 under the batch editing setting are shown in Figure [ while results for GPT-J and GPT2-XL are presented in Figure g of Appendix C Specificity and Fluency remain comparable across different cases, with outcomes closely matching the original model. This indicates that small-batch edits effectively retain the model's original state. However, significant differences arise in Efficacy and Generalization between the two methods. For Efficacy, models edited with computed residuals outperform those with distributed residuals by over 3x on average, while for Generalization, the improvement exceeds 2x. In terms of Consistency, computed residuals achieve an average improvement of more than 10%. These findings indicate that distributed residuals introduce significant information loss during model editing, leading to higher likelihood of editing failures.
Editing Performance. Computed Distributed Residuals
Densralization
Spectifits
800
16
Fluesey
Dineistancy
Layur4
Sapar
Japer
Exper
Figure 4: Performance variations when editing different single layers of the model using computed and distributed residuals separately. Fluency and Con- sistency are normalized.
4.2 Theoretical Analysis of Residual Distribution Locate-then-Eace Methods
Theorem 4.1. In the teett-then-edit model editing, when using residual distribution, the upper bound for the weight shift error between the exact weight shift A and the actual weight shift A is
5

---

given by
|A - 2'12 (1R2² R2 + (L - -1) 1101
(1I)
where Rd denotes the exact residual, and Q Ki" (KoK?" + KiK?")
The proof of the above theorem is presented in Appendix A the upper bound instead the lower bound Remark 4.2 In Equ. (T.. || and ||Q [QI12 increase with the number of new memories (i.e., the size of the editing batch [5). When the number of new memories is fixed, the upper bound increases with || R² and L L. L increases as the residual distributes further, while it is unclear how | - -R'll2 changes. To explore this, assume the computed residual is the exact residual and analyze how | R changes across layers. In Figure increases as the residual distribution extends farther. Therefore, we can conclude that weight shift error increases with both the distance of residual distribution and the size of the editing batch.
We also discuss the rationale for using
show how || R R/I12 changes across layers. It shows that || R2
Remark 4.2. The lower bound of the weight shift error can more effectively reflect the variation in | - -R'122 l22 we see that the lower bound of the error shift very small or even zero, where J Q) the smallest singular value of Q. This occurs because we cannot guarantee that Q is nondegenerate. Even if Q is nondegenerate, a (Q) would still be very small, making the result insignificant. Additionally, although growing upper bound does not necessarily imply an increase in error, our upper bound determines the worst-case scenario of the error, providing essential insight into selecting the layers to update.
error, but according to || Q R'QlI2 2 mmin [Q) R
Considering the closed-fie form solution of locate-then-edit model editing in sequential batch editing, the following lemma can be derived.
Lemma 4.3. In sequential batch editing, when using residual distribution, the upper bound of the weight shift error between the exact weight shift A' and the actual weight shift A for locate-then-edli methods is given by
A A'12 < (IRT" 12 (1-++111) 110'lla
(12)
where R denotes the exact residual, and Q' K!" (K + KGK!" + KiK!")
[ Krdll l22 increases with the number of sequential edits, and thus Section 4.3 indi- cates that the weight shift error also increases with the number of sequential edits.
Lamm3 5
4.3 BLUE: Boundary Layer UpdatE for Improving Locate-then-Edii Model Editing
0P12LL ard
50
From the previous analysis, we know that the residual dis- tribution of the bocate-then-edee model editing inherently inaccurate, and this inaccuracy increases as the residuals are distributed farther. So, how can we enable locate-then- then-edit model editing to perform multi-layer updates while miti- gating the negative impact of the residual distribution? straightforward method is to compute residuals separately for each layer. However, this reduces the efficiency of the locate-then-ede approach, and remains unclear whether it is necessary to compute residuals and perform updates for all critical layers individually.
14
Layer
Figure 5: Variation of ||R" R across layers.
Therefore, we conduct experiments where resid- uals are computed and updates performed for all critical layers. Computation is stopped when the loss falls below 0.05. We update the critical lay- ers sequentially in the order of increasing layers and record the number of optimization steps re- quired to compute residuals for each layer. The
Table 1: Average optimization steps.
Model
Layer: Steps
GPT2-XL
[13-17]: [16.37. 8.43, 1.71. 0.32, 0.10] [3-8]: [10.47, 1.68, 0.11. 0.0. 0.0, 0.0] [4-8] [25.0, 11.10, 0.63, 0.0, 0.0]
GPT-J (6B)
Llama3 (8B)

---

results are shown in Table 1 It can be observed that after completing the first layer update. the number of residual computation steps in subsequent layers decreased significantly across all LLMs. In GPT2-XI., the decrease is 48.5%; in GPT-J, 84.0%; and in Llama3, 55.6%. Furthermore, after updating the first two layers, the optimization steps in the third layer for GPT2-XI., GPT-J and Llama3 drop below 2.0, indicating that only two layers of weights needed to be updated to achieve the editing goal.
The above observations indicate that when calculating residuals separately for all layers, updating just two layers is sufficient to achieve the editing object. The new question is: which two layers should be updated to achieve optimal editing performance? ? According to Theorem the farther the distribution of residuals, the greater the upper bound of the weight shift error. The last critical layer is exactly the layer where the residual is computed. As a result, the first layer updated by the current method the most affected by the residual distribution. To mitigate this effect, we choose the first critical layer as the first layer for updates. For the second layer, we choose the last critical layer for the following reasons: 1) The selection of the first critical key layer is based on the premise that the residual is computed at the last critical layer; and 2) BLUE is optimization strategy, and we aim to preserve the mechanism used in existing locate-then-edit model editing methods, which compute residuals at the last key layer, to ensure broader applicability.
Therefore, we propose Boundary Layer UpdatE (BLUE) strategy to boost the locate-then then-edit methods. BLUE updates only the boundary layers of the critical layers by directly computing residuals of them, specifically the first critical layer and the last critical layer. This not only reduces the number of layers to be updated, but we also demonstrate in Section 5 and better preserves L.MSs' general capabilities. BLUE is suitable for bcate-then-edit methods that perform multi-layer updates using even residual distribution: MEMIT 5 JECCT 7 PRUNE g and AlphaEdit [6]
that it performs better
Experiments
In our experiments, we demonstrate that BLUE can enhance the performance of current locate- then-edit methods (Section 5.2). improve the retention of the original LLMs' capabilities (Section
5.3), and alleviate the hidden state shifts introduced by locate- preate-then-eee then-edit approaches (Section (5.4). Additionally, in Appendix 5.5 5.5 we show that BLUE also boosts locate-then-ede then-edit methods in long-form model editing. In Appendix [5.6] we present ablation studies demonstrating that selecting the first and last critical layers for editing in BLUE is the optimal choice. Appendix Iillustrates the efficiency improvements introduced by BLUE. In Appendix[] we empirically validate Theorem [4. Jnd Lemma [.
4.3 demonstrating that the weight shift error increases with both the batch size and the number of sequential edits. Finally, we present the results of BLUE applied the square root residual distribution method, PMET, in Appendix K
5.1 Experimental Setup
Datasets & LLMss Our experiments are conducted on two datasets: CounterFact [4] and 7sRE 26
We select three L.Mss as the editing subjects: GPT2-XI. (27). GPT-J (6B) 225 Llama3 (8B) (2
We also conduct experiments with Llama2 (13B) to further validate the effectiveness of BLUE in Appendix G
Baselines. BLUE is a facilitation strategy designed for locate-then-edit model editing that performs multi-layer updates, which has been proven in prior research to achieve the best editing performance 6 Therefore, our baselines only consider locate-thent then-edit model editing methods. The pocate-then- edit methods we consider are: MEMIT [5). PRUNE 88) RECT 22). and AlphaEdit (6). We present the experimental details in the Appendix DD]
5.2 Enhaneing Editing Performance with BLUE
We first verify whether BLUE can enhance locate-then-edin model editing. Sequential batch editing better aligns with real-world batch knowledge updates, and we follow (6 by using sequential batch editing experiments to validate the capabilities of BLUE. We randomly sample 2,000 samples from the dataset and perform sequential batch editing with batch size of 100. The results of the sequential batch editing are shown in Table 2 We use red to highlight the results enhanced by BLUE. The
7

---

Table 2: Comparison of BLUE enhanced locate-then-mone editing methods with original locate- then-model editing methods on the sequential model editing task. We color all results that are actually enhanced by BLUE in red. We directly adopt the baseline results from 6 to avoid the energy consumption caused by redundant computation.
Counterfact
ZsRE
Method
Model
Efficney?
Generalization?
Specificity
Fluency?
Consistencyt
Efficaey?
Generalization?
Specificity
Pre-odited
7,83
10.58
89.48
635,23
24.14
36,99
36,341830
31.89
MEMIT
65.05gmm
64. in
51,56st w
437 v3
6.58
34.62gm0
31,28
18.49
PRUNE
68.25
64.75amm
49.82
418.03
5.90aaio
24.77
23,87
20.69
RECT
Ilamas
66.05
63.62
61.4
526.62
20.54,cm
86.05
80.54aa27
31.67
AlphaEdit
98.90au
94,22
67, a
622,49aa16
32,40
94,47a0u
91.13
32.55
MEMITaun
99.57 Lozze
94.13
83.77
626.261831
32.29
95,94
90.9818m
32.41
PRUNFFLLE
96. 93aea
89, es
57.79
627.39 39
33.39
86.55
8
31.04
RFCTEwe
93,40
79.34
619.07a*M
30.62ar
94.37
89.49as
32,76
Alphatdditws.
99.93
47.25
75.24
624,90aa.
33.79
95.77am
91.73amas
31.96.1
Pre-edited
16 .6.22
18.56
83,1
621 8
29.74
26 22
25.79a
27.42
MEMIT
98.55
95.50
63.64
546.28ss
34.89
94.91 2014
90,22
30.39
PRUNE
86. 15
86,85
53.87
427.14
14,78
0.
0. 0.15amm
0.00sam
RECT
98.80)
26.58
72.22
617.31a810
41
96.38
91.214ea
27.79
AlphaEdii
99.75
96, 26.38
75,48
618.50
42,08
99.79.011
96. 96.00+e2:
28.29
MEMITaum
99, 70
96.90 30.90
74:61
620 19
40.8211
99.3 29.58
94. 94.77
28.36
PRUNERIE
97.77.ux3 u0e3
97.28
57.121:00
608.733xm
36.6211
60.51
18.57
22.77
RECT BLUB
98. 70
11a18
74.78ar
620.52aav
39.79 as
97.97
93.862aw
26, 26:32
AlphaFdirguus
99. 77
97.13
75.23
621.07 21.007
41.34
99. 99.63am
95.962am
28 28.67atm at
Pre-edited
a2723
24.34
78.534a3
626.64asxi
31
22. 19.24
31.30.e2:
24.15
MEMIT
94,70
85.82
60,50g
477 26.
22,72
79. 17
71 71,44
26.42
PRUNE
82.05
78.55
53,02at
$30.47
15.93
21.62z0ss
19.27
13.19ars
RECT
GPT2-XL
92.15
81.15
65.13
48ss83
21
81.02.00
73.08
24.85
Alphalidit
99.50uana
93.95
66.39
597.88
39.38
86.11
25.88
MEMITacuE
98. 98.27amm
88. 88.671ass
67.13
587.19ttm
35.641m
93.6 93.62aum
85.34
26.55arm
PRUNEBLRE
88. 19
80, 80.48
52.23
594.08.11s
20.28
47.94
45.03
16.72
RECTmam:
95.67gm
80.97gris
66.88 ao
567.09
30.30ge ss
83.48
75.24g
25.25
AlphaFdirguue
99. 99.40
96.00gm
76.63
621.92 121.92
40 40.98
96.88
89.58
25.93
results indicate that the BLUE strategy effectively enhances the performance of range of locate-then-edit methods in sequential batch editing tasks. 89.58% of the results (86 out of 96) were enhanced. After using BLUE, the editing performance of different editing methods is enhanced across various LLMs, as shown in Table 4 of Appendix B It can be observed that the BLUE strategy significantly improves the performance of PRUNE and noticeably enhances the editing performance of locate-then-edii methods on Llama3 and GPT2-XL. For other cases, such as AlphaEdit on GPT-J, the improvements are minimal due to its already strong baseline performance.
SST
MRPC
CoLA
5.3 Boosting General Capability Retention via BLUE
0.8
0.6-
0.8
0.--
0.6-
0.4
04-
0.4-
02-
02
0.2-
Model editing should not affect other aspects of LLMs. In addition to using specificity and fluency for evaluation, this goal can also be achieved by assessing changes in the gen- eral capabilities of the models after editing. Following the work of (6). we evaluate the
0.0-
500 1000
2500
1500 2000
3000
500
2000 2500
Edit Number
Edit Number
Eddit Number
HTF
MML10
NI.1
0.6
0.
0.4
0.4-
0.1-
0.2
0.2
0.0-
0.0-
sbo 1000 1500 2000 2500 3000
500 1000 1500
2000 2500 3000
500
1000
1500 2000 2500
Edit Number
Edit Number
Edit Number
AlphaEdit
RECT
PRUNE
MEMIT
AlphaEditaLue + RECT BLUTELUE--CCC PRUNE
MEMIT BLUE
Figure 6: F1 scores of the post-edited Llama3 (8B) on six tasks.
8

---

general capabilities of LLMs before and after editing using six natural language tasks from the General Language Understanding Evaluation (GLUE) benchmark (30]. Specifically, we achieve this
through the following six evaluation tasks: SST (The Stanford Sentiment Treebank) [31] MRPC
(Microsoft Research Paraphrase Corpus) [32], MMLU (Massive Multi-task Language Understanding)
33 RTE (Recognizing Textual Entailment) [3), CoLA (Corpus of Linguistic Acceptability) [3].
and NLI (Natural Language Inference) 36).
We conduct a total of 3,000 sequential edits on Llama3 (8B), with batch size of 100 for each edit. Every 500 steps, we evaluate the performance of the post-edited LLMs on these six tasks. The results are shown in Figure O After 3,000 edits, the general capabilities of models edited by RECT, PRUNE, and MEMIT are almost entirely lost. In contrast, models edited by the BLUE-enhanced versions of these methods maintain their general capabilities well. Notably, AlphaEdit inherently demonstrates strong general capability retention, and AlphaEditBLur does not compromise this ability. These results indicate that BLUE enhances the general capability retention of locate-then-edit methods.
Fre-ddied
50
Pre-adasd
oolted
Tre-dtted
Rost whioo
Iroeebedd edited
(a) MEMIT
(b) RECT
(c) PRUNE
(d) AlphaEdit
Ire-ethedd
Irsecitedd
Fre-adked
Proedied
56-
Ureaty
(e) MEMITELUE
(I) RECT
(g)
PRUNEBLUE
(h) AlphaEdituuue
Figure 7: The distribution of hidden states in pre-edited and post-edited Llama3 (8B).
5.4 Mitigating Hidden States Shifts with BLUE
The existing
methods often result in shifts in the hidden states of the model after editing (6]. In this part, we verify whether BLUE can alleviate this phenomenon. Specifically, we extract the hidden states of 1,000 randomly selected factual prompts from LLMs before and after editing. These hidden states are then reduced to two dimensions using t-SNE. The post-editing LLMs mentioned here are the models described Section 5.2 We then visualize the hidden states, and the results are shown in Figure 7 It can he observed that the shifts in hidden states corresponding to the seeate-then-eil method enhanced by BLUE are weaker than those of the original method. This demonstrates that BLUE can mitigate hidden states shifts caused by pocate-then-eee methods.
The results on GPT-J (6B) and GPT2-XL can be found in Appendix H
5.5 BLUE Boosts Long-Form Performance of
Approaches
We conduct long-form model editing experiments on editing Llama3 (8B) using UnKEBench [O] representative work in long-form evaluation benchmarks. We adopt MEMIT and AlphaEdit, both enhanced using the editing paradigm proposed in AnyEdit [9]. as baselines, and follow the same experimental setup as used in AnyEdit. As illustrated in Table, 3 the results demonstrate that BLUE can also improve the performance of crate-then-edii methods in long-form evaluation scenarios. We also conduct a case study on a long-form editing task to demonstrate in detail the effectiveness of BLUE in Appendix E

---

Table 3: Comparison of Editing Methods UnKEBench. The left side of ' represents the LLM's edited output for original questions, while the right side represents the edited output for paraphrase questions.
Metric
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
BERT Score
ROUGE-L (SubQ)
MEMIT
77.65 / 66.17
90.78 81.60
87.12/73.
90.43
80.81
95.69 / 92.32
47.36
AlphaEdit
62.82 / 51.78
80.02 / 69.00 08
71.62 56.80
79.10/ 67.68
91.67 87.41
42.72
MEMIT BLUE
82.347 74.76
90.797 85.36
86.89
78.94 78.94
90.337 84.68
96.507 94.72
52.15
AlphaEditauue
68.64 /1.49
84,90
77.98/ 68.40
84.08 77.37
93.39 90.97
49.49
Efficacy across Edit Layers
Generalization across Edit Layers
Generallization
Elfcacy
4.5
4.5
5.6
5.7
5.8
6.7
6.8
7,8 4,8 (SLUE)
4,5
4.8
4.7
5,6
5.7
6.7
6.8
7.5 4.8 (BWE)
Edit Layer
Edit Layer
Specificity across Edit Layers
Fluency across Edit Layers
632
Specifiity
70
Fluancy
522
4.5
4.5
4.7
5.5
5,7
5,0
6.7
6.R
7,0 1,8 (hLLE)
4,5
4.5
4,7
3,6
5,7
6,7
1.1
7.8 AL8 (BUI)
Edit Layer
Edit Layer
Consistency across Edit Layers
Condetency
MEMITsws
AlphaEd
RECToLue
PRUNE
10
4.5
4.6
4.7
5.6
5,7
5,0
6.7
6.8
7.8 4.8 CBLUE
Edit Layer
Figure 8: Layer Ablation of BLUE.
5.6
Ablation Study
To verify that the first and last critical layers selected by BLUE are optimal for editing, perform 30 sequential edits with batch size of 100 on Llama3 (8B). As shown in Figure the optimal editing layers for all four BLUE-enhanced methods are consistently layers 4 and 8 the ones selected by BLUE. This provides empirical evidence that the layers chosen by BLUE are indeed optimal for model editing.
Conclusion
This paper rethinks the role of residual distribution in theaee-then-edit model editing. Through empirical and theoretical analyses, we show that residual distribution not an optimal choice, as it leads to increasing errors in weight updates with larger batch sizes, more sequential edits, and greater distribution distances. Based on these findings, we propose the BLUE strategy, which improves locate-then-e then-edit methods by updating only the first and last critical layers of the model. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE effectively enhances editing performance. Further experiments and analyses indicate that BLUE also improves the retention of LLMss original general capabilitie and mitigates shifts in hidden states after editing. Moreover, BLUE yields gains in both time and memory efficiency and strengthens ocate-then--et methods in forg-form model editing tasks.
10

---

Acknowledgements
This work was supported by the National Key Research and Development Program of China under Grant 2024YTB4506200, the Science and Technology Innovation Program of Hunan Province under Grant 2024RC1048, the National Key Laboratory Foundation Project under Grant 2024-KJWPDL-14, and the National Natural Science Foundation of China under Grant 62402506.
References
[1] Wang, S., Y. Zhu, H. Liu, et al. Knowledge editing for large language models: A survey, ACM Comput. Surv., 57(3), 2024.
[2] Yao, Y., P. Wang. Tian, al. Editing large language models: Problems, methods, and op- portumities. In H. Bouamor, J. Pino, K. Bali, eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10222-102444 Association for Computational Lin- guistics, Singapore, 2023.
[3] Geva, M., R. Schuster, Berant, al. Transformer feed-forward layers are key-value memories. In M.-F. Moens, X. Huang, L.. Specia, S. W.-t. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 5484-5499 Association for Computa- tional Linguistics, Online and Punta Cana, Dominican Republic, 2021.
[4] Meng, K., D. Bau, A. Andonian, et al. Locating and editing factual associations in gpt. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh, eds., Advances in Neural Information Processing Systems, vol. 35, pages 17359-17372. Curran Associates, Inc., 2022.
[5] Meng, K., A. S. Sharma, A. J. Andonian, et al. Mass-editing memory in transformer. In The Eleventh International Conference on Learning Representations.
[6] Fang, J., H. Jiang, K. Wang, et al. Alphaedit: Null-space constrained knowledge editing for language models. The Thirteenth International Conference on Learning Representations.
[7] Gu, J.-.,, H.-X. Xu, J.-Y. Ma, et al. Model editing harms general abilities of large language models: Regularization to the rescue. In Y. Al-Onaizan, M. Bansal, Y.-N. Chen, eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16801-16819. Association for Computational Linguistics, Miami, Florida, USA, 2024.
[8] Ma, J.-Y., H. Wang. H.-X. Xu, et al. Perturbation-restraiin sequential model editing. In The Thirteenth International Conference on Learning Representations.
[9] Jiang, H., Fang, N. Zhang, al. Anyedit: Edit any knowledge encoded in language models. In Forty-second International Conference on Machine Learning.
[10] Deng, J. Z. Wei, L. Pang, al. Everything is editable: Extend knowledge editing un- structured data in large language models. In The Thirteenth International Conference on Learning Representations.
[11] Zhong, Z., Z. Wu, C. D. Manning, et al. Mquake: Assessing knowledge editing in language models via multi hop questions. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15686-15702. 2023.
[12] Li, X., S. Li, S. Song, et al. Swea: Updating factual knowledge in large language models via subject word embedding altering. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pages 24494 24502. 2025.
[13] Huang, Z., Shen, X. Zhang, et al. Transformer-patcoe One mistake worth one neuron. In The Eleventh International Conference on Learning Representations.
[14] Wang, P., Z. Li, N. Zhang, et al. Wise: rethinking the knowledge memory for lifelong model editing of large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Curran Associates Inc., Red Hook, NY, USA, 2025.
11

---

[15] Zheng, C., L. Li, Q. Dong. et al. Can we edit factual knowledge by in in-context learning? In H. Bouamor, J. Pino, K. Bali, eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4862 4876. Association for Computational Linguistics, Singapore, 2023.
[16] Mitchell, E., C. Lin, A. Bosselut, et al. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817-15831. PMLR, 2022.
[17] Hartvigsen, T., S. Sankaranarayanan, H. Palangi, et al. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
[18] Tan, C., G. Zhang, J. Fu. Massive editing for large language models via meta learning. In The Twelfth International Conference on Learning Representations.
[19] Zhu, C., A. S. Rawat, M. Zaheer, et al. Modifying memories in transformer models. arXiv preprint arXiv:2012.00366, 2020.
[20] Mitchell, E.,, C. Lin, A. Bosselut, et al. Fast model editing at scale. In International Conference on Learning Representations. 2022.
[21] X., S. Li, S. Song, et al. Pmet: Precise model editing in transformer. Proceedings of the AAAI Conference on Artificial Intelligence, (814))))8864418572 2024.
[22] Zhang, N., Y. Yao, B. Tian, al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.0) .012286 2024.
[23] Mazzia, V., A. Pedrani, A. Caciolai, et al. survey on knowledge editing of neural networks. arXiv preprint arXiv:2310. 19704, 2023.
[24] Meta, A. Introducing meta Ilama 3: The most capable openly available I1m date. Meta AI, 2024.
[25] Wang, B., A. Komatsuzaki. Gpt-j-6b: billion parameter autoregressive language model, 2021.
[26] Levy, O., M. Sco, E. Choi, et al. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv: 1ttv:1706..444 04115, 2017.
[27] Radford, A., J. Wu, R. Child, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[28] Al@Meta. Llama 3 model card. 2024.
[29] Gu, J.-C., H.-X. Xu, J.-Y. Ma, et al. Model editing harms general abilities of large language models: Regularization to the rescue. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 118199 2024.
[30] Wang, A. Glue: A multi-task benchmark and analysis platform for natural language understand- ing. arXiv preprint 1rXiv:1804.07461, 2018.
[31] Socher, R., A. Perelygin. J. Wu, al. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631- 2013.
[32] Dolan, B., C. Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005). 2005.
[33] Hendrycks, D., C. Burns, S. Basart, et al. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[34] Bentivogli, L., P. Clark, I. Dagan, et al. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1 2009.
[35] Warstadt, A. Neural network acceptability judgments. arXiv preprint arXiv: 1 18088 2019.
[36] Williams, A., N. Nangia, S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: 174.::17040 2017.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in Appendix L
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

The paper provides the full set of assumptions and complete (and correct) proof for Theorem 4.1
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Yes, we provide zip file with the code needed to reproduce our experiments, as well as a README with instructions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We share our code. The datasets will be automatically downloaded from open-source resources when the code is executed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pblle http:////uuddes/oeeuuubbbii
(CodeSubiisss wwtplubmmsssionii ssiiiissicccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We share code and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars Figures 23 and 96% CI in the main results (Table. 2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss computational resources used for all experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix M
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the papers that introduced the models and data used in our work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy httt:p/:/eeurrpp..c//ooffeeeeeee 22tprrnnncee/2/2LLLL for what should or should not be described.
A Proof of Theorem 4.1
Proof. Let K/" (K] K 7 -1
Q, then the weight shifts erroris: is:
|
R Q
R 1l2
(13)
RRR
12|12|1Q11
(14)
R
R
(15)
According to Section 4.1.2
the directly computed m represents the optimal memory for editing, and Then, Equ. (15) can be written as:
thus we have R R²
RL
R
1l2
Q112
(16)
R
RR" RR' +R'
-+(+111 11/2
(17)
RL
R R ||2 |R
L 1 |12 1Q112
(18)
L 1
R RL ||2 L 1-|+1 RL ll2 110122
(19)
Rt' R² 112 (L 1-)11=4112> ||01/2
(20)
B BLUE's Performance Improvement in Locate-then-Edit Model Editing Across LLMs
We summarize the enhancement degree of each method by BLUE across different models, as shown in the table below.
Table The average performance improvement BLUE on different locate-then-ede edit model editing across various LLMs. The abnormal results of PRUNE editing GPT-J on the ZsRE dataset are excluded in our statistics.
Model
MEMIT
PRUNE
RECT
AlphaEdit
Llama3
129.61%
144.52%
27.05%
2.50 %
GPT-J
6.73%
44.36%
0.58%
0.03%
GPT2-XL.
17.02%
41.24%
9.47%
4.00%
19

---

Tidlinn Performanee Broidials
Edilingg Indformance
Bistriiuted Bestsuals
Genamiiietiin
Gereenttio bon
Specilads
Specilady
Plusnay
Plunni
(a) GPT-J (6B)
(b) GPT-XL
Figure 9: Performance variations when editing different single layers of the model using computed and distributed residuals separately. For better visualization, Fluency and Consistency were normalized.
C Supplementary Results of Post-edit LLM Performance
We show the supplementary results of post-edit LLM performance in Figure 2 indicate that distributed residuals introduce significant information loss during model editing, leading to higher likelihood of editing failures.
The results also
D Experiment Details
All our experiments are conducted on A800 GPUs. The baseline methods used for comparison in the experiments are kept their original settings, with PRUNE following the reproduction settings of 6 For baseline methods enhanced by BLUE, all configurations remain consistent with the original baselines, except for AlphaEditar.ur. For AlphaEditgr.u. we set the a values for Llama3 (8B), GPT-J (6B), and GPT2-XL to 1, 95, and 80, respectively, ensure the invertibility of matrices during the editing process, thereby achieving better editing performance. For a clearer understanding of the baselines, please refer to (6).
E
Case Study on Long-form Model Editing Task
To demonstrate in detail the improvements of BLUE in long-fom model editing, we present study of editing Llama-3 (8B) in Ta It can be seen that the method enhanced by BLUE generates content that is noticeably closer to the target output compared to the original method. This provides additional evidence for the effectiveness of BLUE in long-form model editing tasks.
F Batch Model Editing
In addition to sequential batch editing, large-scale batch editing is also an important aspect of evaluating the performance of model editing methods. Therefore, we conducted 10,000 batch edits for both the baseline and the BL.UEennanced methods, with the results shown in Table The results in the table indicate that while the improvement in large-scale batch editing after applying the BLUE enhancement to the baseline is not as significant as in sequential batch editing, the baselines enhanced by BLUE still demonstrate overall stronger performance. Specifically, 70.83% of the metries (68 of 96) are improved. Note that although the baselines enhanced by BLUE performed better in terms of efficacy and generalization, they show worse results in specificity. This suggests that while the BLUE-enhanee model editing methods strengthen the knowledge being edited, it also affects other unrelated knowledge. Achieving optimal performance across all three metries simultancously remains major challenge in model editing [14]. This is particularly true for locate-then-edi methods, as BLUE serves as an enhancement to existing editing methods without altering their original modeling. Therefore, addressing this issue may require future work improving the original modeling of
20

---

Table 5: Examples of different editing methods in long-form editing scenarios.
Type
Text
Input
What is George Rankin's occupation?
Paraphrase
What does George Rankin do for a living?
Target Output
George Rankin has been actively involved in politics for over decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin's occupation is that of political figure.
AlphaEdit
George Rankin has been involved in politics for over 20 years. He has served as a member of the state legislature for 10 years and has been mayor for years. He has also been a member of various political committees and has run for several political offices, including governor and U.S. Senate.
MEMIT
George Rankin has been involved in politics for over 20 years. He has served as a member of the state legislature for 10 years and has been a mayor for years. He has also for governor and senator, but has not held those offices. George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been a vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin has been involved in polities for a long time and continues to be an active and influential figure in the political scene.
AlphaFditBLU'
MEMIT BLUE
George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin's involvement in politics is significant aspect of his public persona and identity.
editing methods. Nevertheless, the overall superior performance of the BLUE-enhanced methods also demonstrates that BLUE can improve the editing performance of locate-then-edit approaches in batch editing.
G Results on LLaMA-2-13B
To demonstrate the effectiveness of BLUE on larger models, we selecte AlphaEdit and MEMIT to perform sequential editing experiments on layers 30-34 of 1--MA22-133 following the same setup as in the main experiments. The results are shown in Table 7 The results demonstrate that the BLUE-enhanced methods achieve better overall editing performance than the baseline methods on both the CounterFact and ZsRE datasets. We bolded all the enhanced results. On the CounterFact dataset, all metrics except for Specificity show improvements over the baseline. On the ZsRE dataset, all metrics of AlphaEdit show improvements, while MEMIT exhibits improvements in all metrics except for Specificity. This indicates that BLUE is also effective on LLaMA-2-132 13BB, further validating the effectiveness of our approach.
H Hidden States Shifts in GPT-J (6B) and GPT2-XL
We present the hidden state shifts before and after model editing for GPT-J (6B) and GPT2-XL in Figs. T0 and [T respectively. Similar results those on Llama3 (8B) are observed for GPT-J (6B)
and GPT-2 XL. The BLUE-enhanced basclines have smaller overall impact the model's hidden
21

---

Table 6: Comparison of BLUE enhanced teattetthe-mmdde editing methods with original locate- then-model editing methods on the batch model editing task. Eff. Gen., Spe., Flu. and Consis. denote Efficacy, Generalization, Specificity, Fluency and Consistency, respectively. We color all results that are actually enhanced by BLUE in red.
Counterfact
ZsRE
Method
Model
Eff.1
Gen.1
Spe.t
Flu.t
Consis.
Eff.1
Gen.1
Spe.t
Pre-edited
7.02
9. 9.44
89.73
630.00 20
24.21
35.67
a05s
31.832041
MEMIT
93.53
74. 12
18.080
22
29.71
86.57
82.58
32.47:004
PRUNE
93. 93.64
84.44
60.00ausr
625.11
36.83
13.29
13.75
15.34
RECT
58.07
39.88
88. 15
628.71
26. 1 I
70.35
65.04
32.452041
Llama3
AlphaEdit
88.89
69.91
83.98s041
625,78=a2s
28.66
84.07sas2
80.15
32.50=m44
MEMITsLUE
99.28=au7
93.83
79.342046
626.09
33.04
95,38ea2s
92.62aa3:
31.68=044
PRUNF
99.37_uu
94.30
10.1
623.54_xx
36.64
16.83
g3.55
28.23
RECTBLUE
94. 94.02=004
79.25
85.45
627.57=034
30.34aa1
85 85.27
77.73a05
31.83a044
AlphaEditmuun
98.62
90.76
79.61
625.04
32,46
94.14
90.642u3
31.52
Pre-edited
15.20=000
17.70
83.50sa.0
622,40=a.
29.40=ca)
26,40sam
25.80
27.00=n.0
MEMIT
98. m2:222
87.14
74.05
620.68
95,90saau 39.7
89.06
26.30
PRUNE
91.54_n0s
90.00
57.49
562.52gnm
29.98
26.91 2nes
16.75
RECT
88.13=0a
63,40
79.31
622.54-021
35.62
70.46
61. 61.90sa7
26.64-1.50
AlphaFdit
GPT4
99.26_n17
86.70
69.65
587.89
39.51 1oz
93.09
82.64_use
22v78_tvv
BLUI
99.58=013
27.40
615.97=0.6
40.83
98. 18aass
93.61 a036
25.91
PRUNEBLUE
99.36
98.06
56.82
608.78 78:
41.76som
74.83
71.24aus
20.07
RECTBIDR
97.86
74.91 88.41
621.45:100
38,79
90,20saa
81,00
27:3
AlphaFditguue
99.39am1s
95.52
69.28
619.90
41.25
98.261a1
96.63a0
26.591a99
Pre-edited
21s82.nst
24.1614m2
78.32
626.78
31.37
17
21.28
24.2
MEMIT
79.64=u2
65.86=0as
70.01
625.67=m2)
36.17
62.46
57.59
25.86=030
PRUNE
85.27 _oo
7m.30
604.09 57.73
35.661aa
1am
40.14
19.01
RECT
GPT2-XL.
61.92.
48.68
74.69
625.87 .m2s
33.99.oa
19.37
45.30
25.64_mm
AlphaEdit
93.24=00a
26.28
64.54 54.54
604, 20.
38.62e02s
61. 61.262a74
54.82a076
20.83
MEMIT
65.37 78.14
37.10 :15.34:044
71.93sam
67.51aum
23,44
PRUNFELUE
95.70
90.18 18
53.81 2758
596.30anss
37.022aaM
51.06gam
47.82.476
14.74.mm
RECTBLUE
70.93=0a0
58.73
72.09
621.52=0.4
34.78=cai
58,88eam
54.292am
24.55
AlphaFditguue
94.10_.o
81.20..oss
64.53,um5
620.79.mm.
1
16.681om
70.16.mm
23.00
Table 7: Comparison of different editing methods on CounterFact and ZsRE datasets when editing LLaMA-2-13B.
CounterFact Dataset
ZsRB Dataset Generalization
Method
Efficacy
Generalization
Specificity
Fluency
Consistency
Efficacy
Specificity
AlphaEdhi
$2.30_2.19
18.95
20.53
408.9222.36
0.222.0.1
$2.31
37.68
9.1 00
AlphaFditaut
78.75_1.79
69.53_1.81
44.27 #1.70
$13.4923.09
22.10
53.01
41.3221.22
9.18
MEMIT
79.20-1.17
65.94
41.53
378. 10
24.44
50. 21.45
34. 24.70
9,71
MEMITSLIE SLLE
79.60_1.78
66.00_1.81
41.28
384.23
14.65
51.0721.4
37.13
9.07
states compared to the original baselines. This indicates that BLUE can mitigate the hidden state shifts caused by locate-then-edit methods, suggesting that the BL.UE-enhanced baselines introduce fewer side effects to the original model.
I Efficiency Analysis
To evaluate the time and peak memory required for a single edit, we perform 300 sequential edits with a batch size of and record the editing time and peak memory for each edit. The average peak memory consumption and editing time per instance for each method are reported in Table and Figure In terms of peak memory, BLUE achieves higher memory efficiency than the original methods. This is
22

---

Fdsbedbud
Postedded
adiedd
Poctedted
50
35
50 Deneey
35
50 Ireaatyy
S6
25
Beaeny
25
25
50 Doxey
(a) MEMIT
(b) RECT
(c) PRUNE
(d) AlphaFdii
othed Fitk edite
aditdd
32
Prosetine
Persethoo
-00
-25
25
50 Deeat5
-25
30 Daady
-50
-25
23
50 Deeasy
-00
-21
(e)
MEMITALIE
(I) RECTBLIE
PRUNEBLUE
(h) AlphaEditau.u
Figure 10: The distribution of hidden states in pre-edited and post-edited GPT-J (6B).
6-
Iudbeditad
Portedited
Portedrad
-00
-20
20
40
Deeaty
-42
-20
20
Daady
-6
-20
20
-30
50
(a) MEMIT
(b) RECT
(c) PRUNE
(d) AlphaEdiit
Proedited Per-edrad
adired Inskedittdd
Pholodired
Masiedded
(e) MEMITALUE
(f) RECTBLUE
(g) PRUNEBLUE
(h) AlphaEditauue
Figure 11: The distribution of hidden states in pre-edited and post-edited GPT2-XL.
intuitive: by discarding residual distributions and instead computing residuals only for the first and last key layers, BLUE reduces the memory overhead associated with residual distribution. Regarding editing time, with the exception of PRUNEBLUE on Llama-3 (8B)-which incurs slightly higher editing time than its original counterpart average editing times across different models and datasets. These findings demonstrate that BLUE not only improves the editing performance of locate-then-edit methods, but also enhances their editing efficiency.
Enhuanenhnceed methods consistently achieve lower
J Experimental validation of the theorem and lemma
In this section, we experimentally validate the correctness of the theorem and lemma presented in Section 4.2) In this section, choose llama3 (8B) the target model for editing, and use MEMIT and AlphaEdit as the editing methods.
J.1 Experimental verification of Theorem 4.1
the weight shift error increases with the batch editing size. To verify this, we conduct batch editing experiments with 500, 1000, 5000, and 10000 edits on the CounterFact and It can be clearly observed that both datasets,
According to Theorem 4.1
ZsRE datasets. The results are shown in Figure B
23

---

Comparison of Editing Time per Instance
Model: GPT-J, Dataset: ZeRE
Model: Llama3-8B, Dataset: ZsRE
Model: GPT2-XL, Dataset: ZsRE
15.0
13.05
15.0
3.0
80
10.0
12:5
12.5
10.0
10
11.00
9.51-2 52
2.5
2.22
1.2
7.76
7.5
1.5
6:97
6.0
1.0
0.5
AlphaEdit MEMIT PRUNE RECT
AlphaEdit MEMIT PRUNE RECT
MEMITT PRUNE RECT
Model: GPT-J, Dataset: COUNTERFACT
Model: Llama3-88, Dataset: COUNTERFACT
Model: GPT2-XL, Dafaset: COUNTERFACT
ORIGINAL
14-21
ORIGIAL BLUE
ORIGINAL
BLUE
15
BLUE
12.5
1248
1175
2.5
38
2,300
10.0
10
2.0
7.5
8/42
1.5
8.44
5.0
2.5
0.5
0.0
0.0
AlphaEdit
MEMMT PRUNE
dlphaEdI PRUNE RECT
Figure 12: Average Editing Time per Instance for Different Methods
Table 8: Peak memory usage of both BLUE and locate- then-edit
CounterFact GPT-J
ZsRE
Method
Llama-3
GPT-2XL
Llama-3
GPT-J
GPT-2XL
AlphaFddit
36.09
30.43
7.14
36.09
30.43
7.43
AlphaFditaLGE
35.10
28.92
7.24
35.43
28.92
7.49
MEMIT
37.07
31.68
7.34
37.07
31.68
7.34
MEMIT BLUE
36.42
30.67
7.30
36.42
30.67
7.38
RECT
37.07
31.68
7.34
37.07
31.68
7.34
RECT BLUE
36.42
30.67
7.22
36.42
30.67
7.37
PRUNE
38.17
33.18
7.53
38.17
33.18
7.53
PRUNEBLUE
36.85
30.67
7.30
36.85
30.67
7.46
the overall performance gap between BLUE-enhanced methods and non-enhanced methods increases with the batch size, especially in terms of efficacy and generalization. Although not all metrics show a strictly increasing performance gap with larger batch sizes- for example, on the Consistency metric of the CounterFact dataset, the performance gap for 5000 batch edits is greater than that for 10000 edits-the overall trend still empirically supports the correctness of Thcorem 4.1
J.2 Experimental verification of Lemma
According Lemma 4.3 the weight shift error increases with the number sequential edits. To verify this, we conduct sequential editing experiments with batch size of 1, using 100, 500, 1000, and 5000 edits. As shown Figure 14, similar the batch editing results, the overall trend on both datasets shows that the performance gap between the Lnhennedddced methods and the original methods increases with the number of sequential edits. For instance, on the CounterFact dataset, the performance gap in efficacy and generalization between AlphaEdit and AlphaEditglue increases significantly when the number of edits reaches 5000. Although specificity and consistency do not exhibit this trend, the patterns observed in efficacy and generalization still support the conclusion that weight shift error increases with the number of sequential edits, as efficacy and generalization directly reflect the impact of the editing methods.

---

Counterfact Effiaay
Counterfact Generalizaton
Counterfact Specifictty
45-
95
Efflayy
96
Generalization
90
86
85
84
80
92-
70
500
5000
10000
500
5000
10000
520
5000
18000
Num Batches
Num Batches
Num Batches
CounterFact Fluency
CounterFact Consistoncy
625
35
627
32
Hluncy
625
33
625
500
Num
Num
Alhrsdd Sazce
MEMIT
MENI B..L
(a) CounterFact
ZARE Efficcyy
ZaRE Generalization
ZaRE Specificity
96
06
33.0
92
32.5
Specifict
EFfckiyy
88
00
00
$2.0
31.5
69
00
1030
5000
10300
500
1303
10000
500
1020
5002
13000
Num
Batches
Nam Batches
AppaESR
Abratdhaau
MENIT
NENTT
(b) ZsRE
Figure 13: Performance Variation of Model Editing with Batch Edits on CounterFact and ZsRE Datasets
K BLUE in the Locate-then-edit Method with Square Root Residual Distribution
use a square root residual distribution instead Some locate-then-edit methods (e.g., PMET 21 of an even spread. They claim that the square root residual distribution can mitigate information loss during residual distribution. Since BLUE is designed for tecate-then-edit methods with even residual distribution, we do not consider such methods as basclines. Nevertheless, we attempt to enhance PMET with BLUE. The results of sequential batch editing are shown in Table PMET exhibits significant performance improvement when editing Llama3 on sequential model editing task, while performance gains in other scenarios are relatively limited. We speculate that this may be because PMET's 's use of square root distribution retains more editing information compared to even distribution, leading to the limited improvement of BLUE. Additionally, PMET incorporates self- attention module during editing optimization but only edits the FFN weights when updating model parameters. This might result in BLUE's two-layer update being insufficient to fully integrate the editing information into the model weights. Nevertheless, BLUE demonstrates notable improvements
25

---

Courterfact
Counterfact Generalizatio
CounterFact: Specifcity
40
05
Efflayy
99
Ganeralization
85
80
75
80
70
70
65
65
70-
55
100
5000
100
5000
100
5000
Num Sequences
Num Sequercess
Nam Sequences
Counterfact Fluency
CounterFact Consistancy
35
30
25
Hoancy
550
425
10
100
100
Num Sequences
Sequarcess
Albrstd Sasce
MEMIT
MENT BL.L
(a) CounterFact
Z4RE Efficayy
ZsRE Generalization
ZaRE Specificity
20.5
62
36.0
91.
33.00
45.5
Denterr tzation
80
Specifiitty
12.75
45.0
09
22.50
88
12.25
67
100
300
2000
3000
102
500
1000
5002
130
3000
1000
3000
Nom Sequeecess
Sequences
Num Sequences
AphaEsi
Alpratihiuue
(b) ZsRE
Figure 14: Performance Variation of Model Editing with Sequential Edits on CounterFact and ZsRE Datasets
in the locate-then approaches with residual even distribution, indicating that it remains applicable to most locate-then-edit methods.
L Limitations
As general strategy applicable to locate-then-edii methods, BLUE enhances various aspects of the locate-then-edit paradigm. Extensive evidence demonstrates that BLUE improves the editing performance of bocat--then-ede methods, preserves the original model's capabilities post-editing, and alleviates the shift in hidden representations introduced by editing. Moreover, with updates applied to only two layers, it also improves editing efficiency. However, the improvement of BLUE in the locate-then-eds method for editing reasoning knowledge (e.g., MQUAKE 11] remains to be verified, and we leave it as future work..
26

---

Table Comparison of PMET with original PMET on the sequential batch and batch model editing task. We color all results that are actually enhanced by BLUE in red.
Methed
Model
Counerfact
ZsRE
Efficacyt
Generalization
Specificity?
Fluencyt
Consistencyl
Efficacy
Generalization
Specificity!
Sequential Model Editing Task
Pre-edited PMET
1aama3
7.85
10.58.uze
89.48
635 23
24.14
36,99
36.34aoa1
31.89.mm
29,47
90,78
76.07
619.62am
32,45
94.97
89.98ac1s
32.95
PMETmuu mum
99.57
94.13
83.77am
626.26xem
96.07
91.73ace
32.66
Pre-edited
16.22 .2xm2
18.56
a3.1
621 a
29.74
26.32sta7
25.79
27.42
PMET
99.7 1
m
72.32
618 82
41.77
99.07
96. 10
28. 29
PMET BLUB
99.57
92.82
77.61amm
620.02aam
29.1
99.16s
87.37
13
Pre-edited
22.23
24.34
78.53
626 64
29
31.30
24. 25
PMET
95, 50
87 87.27aue
62.66.:
542,47a24
31.56
93 s3:2
87 37.06
25, 28m5
PMET BLUE
85.57
67.93
603.03gm9
89.32
80.53g1
Batch Model Editing Task
Pre-edited
Lamas
7,02suze
9,44au
89.73
630.00asm
24.21
35.67
34.81
PMET
97.02
26,22
77.72 a
624.68
31 31.84
83,49
80.73grss
31.94
PMET BLUE
93.64
81,52am
84.63ar
627 al
30.62
85.92anas
82.83ataa
32. 23
Pre-edited
15.20
17.70
83.50aa
622. 40
29 m0
26.40
25.80
27. 7..00am
PMET
99.57 =
22.48
71.41
620.31
89.24
82.69atar
25.51 coee
PMET BLUE
97
87.24
73.32
616.85ars
38.14
80.77
67.58ar4s
28.00
Pre edited
21.82
24.16
78.32
626. 78
31.37
22.17
21 28.88cc
24. 24.20
GPTE
m
70.45
66.42at
622. 16amm
60.25
56.29
23.95
PMETRUE
67.09
a4.48
73.48415:
626.6746as
35.05
57.15
51.99407
25.01
M Impact Statements
The model editing studied in this paper aims to efficiently update outdated or incorrect knowledge in large language models (LLMs). While the original intention behind such techniques is beneficial, there is also potential for misuse, such as injecting false or malicious content into LLMs. Therefore, we caution readers not place blind trust in the content generated by LLMs.
27


Reviews:
[{"review_id": "h6kbbhcJ0n", "reviewer": "Reviewer_meft", "summary": "This paper presents a critical analysis of the 'locate-then-edit' paradigm in model editing, specifically targeting the widely-used residual distribution mechanism. The authors compellingly argue, through both theoretical and empirical evidence, that this distribution method is a source of significant error that harms editing precision. To address this, they propose a simple yet effective strategy, Boundary Layer UpdatE (BLUE), which modifies only the first and last critical layers with directly computed residuals, bypassing the distribution step entirely. The authors validate their approach through extensive experiments on multiple LLMs and datasets, showing that BLUE not only improves editing performance but also better preserves the model's general capabilities and is more computationally efficient.", "questions": "None", "limitations": "The author acknowledges the limitations in Appendix L and shows that the methods didn't evaluate the results in multi-hop reasoning benchmarks like MQuAKE.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n- The author conducts sufficient empirical analysis and theoretical analysis about the shortcomings of residual distribution. The empirical analysis in Section 4.1, particularly the experiments showing the diminishing contribution of distributed residuals (Figure 2) and their sub-optimality compared to directly computed ones (Figure 4), is clear and convincing. This empirical work is well-supported by a theoretical analysis (Theorem 4.1 and Lemma 4.3) that provides an upper bound on the weight shift error, formalizing the intuition that errors increase with distribution distance, batch size, and the number of sequential edits.\n- The authors apply BLUE to multiple strong baselines (MEMIT, AlphaEdit, PRUNE, RECT) across several LLMs (Llama3, GPT-J, GPT2-XL) and standard datasets (CounterFact, zsRE). The results in Table 2 strongly support the primary claims of improved editing performance in sequential batch scenarios. Furthermore, the analysis of general capability retention on downstream GLUE tasks (Figure 6) and the mitigation of hidden state shifts (Figure 7) are particularly compelling, as they address critical aspects of model editing beyond simple efficacy and demonstrate a clear advantage for BLUE in preserving the integrity of the original model.\n\n\n### Weakness\n- While the justification for updating the first critical layer is well-motivated (as it is most affected by distribution error), the motivation for choosing the last critical layer is stated simply as \"follow[ing] prior work\" (Section 4.3, page 7). The ablation study in Appendix H (Figure 11) provides strong empirical validation for this choice, demonstrating its optimality. However, the main text would benefit from a more explicit theoretical or intuitive justification for why the first and last layers form the optimal pair, rather than, for instance, the first two critical layers. \n- I think the proof section is a bit verbose in the paper, and I recommend the author move some of them to the appendix and show more experiements in the main text. To enhance the paper's narrative impact, I would recommend condensing this section slightly. The reclaimed space could be used to feature a crucial experimental result that is currently in the appendix. For example, the layer-selection ablation study (Appendix H, Figure 11) provides the core empirical evidence for BLUE's design. Elevating this experiment to the main paper would proactively answer a key question about the \"Boundary Layer\" choice and make the justification for the method even more compelling.\n- The authors correctly identify that the method's performance on multi-hop reasoning questions is a limitation and an area for future work. This points to a more significant consideration regarding the scope of the paper's claims. The current evaluation focuses on a standard but specific type of generalization: applying a factual edit to paraphrased prompts. While the results are strong in this domain, the paper would be more impactful if it contextualized its findings more explicitly in the main text.", "was_revised": true, "final_justification": "I have checked all the rebuttals between reviewers and authors; I think the author has addressed the concern of the elaboration.\nThe only issue left is the multi-hop reasoning evaluation; the author has acknowledged it in the limitation.\nI think this would make the paper better if the author would add this.", "rebuttal": "We thank the reviewer for recognizing that **BLUE is a simple yet effective strategy**. We address the reviewer's concerns below:\n\n---\n**W1: The main text needs to elaborate more on why the second layer is chosen as the last key layer rather than the second layer.**\n\n We appreciate the reviewer’s detailed comment. On the one hand, BLUE is an optimization strategy, and we aim to preserve the mechanism used in existing locate-then-edit model editing methods, which compute residuals at the last key layer, to ensure broader applicability. On the other hand, the choice of the first key layer is based on this mechanism: when residuals are computed at the last key layer, the layer most affected by residual distribution is the first key layer. Therefore, we choose the last key layer as the second update layer. We will add this explanation to the main text.\n\n---\n**W2: The proof section is a bit verbose.**\n\n We thank the reviewer for the constructive suggestion. We will condense the proof section (Section 4), move *Remark 4.2* and *Lemma 4.3* to the Appendix, and integrate the content of *Appendix H* into the main text.\n\n---\n**W3: Contextualizing the limitation about multi-hop reasoning more explicitly in the main text.**\n\n We thank the reviewer for the helpful feedback. We will explicitly discuss the method’s limitation on multi-hop reasoning tasks and mention it as a direction for future work in the conclusion of the main text."}, {"review_id": "RINGExSwIb", "reviewer": "Reviewer_bKjA", "summary": "This paper studies model editing for large language models (LLMs), focusing on the prevalent locate-then-edit paradigm. The authors identify that the residual distribution mechanism used in existing methods introduces weight shift errors, which worsen with increased distribution distance, batch size, and edit sequence length, reducing editing accuracy. To address this, they propose the Boundary Layer UpdatE (BLUE) strategy to mitigate these errors. Experiments on three LLMs and two datasets demonstrate that BLUE improves editing performance by an average of 35.59% while better preserving the general capabilities of LLMs, significantly advancing the state of the art in model editing.", "questions": "1. The BLUE method updates only the first and last critical layers without residual distribution. How sensitive is the method’s performance to the selection of these critical layers? Would different choices affect editing accuracy or model preservation?\n2. The experiments are conducted on three LLMs and two datasets. Could the authors clarify whether BLUE generalizes well to larger-scale models (e.g., GPT-3 scale) or other architectures beyond those tested?\n3. Does the BLUE strategy introduce additional computational or memory overhead compared to existing locate-then-edit methods? If so, how significant is it in practical scenarios?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow.\n2. The proposed method is well-motivated.\n3. The experiment is thorough and convincing.\n\nWeakness:\n1. The authors do not explicitly discuss the computational overhead or efficiency trade-offs introduced by BLUE compared to standard locate-then-edit methods, which may be important for practical deployment.\n2. While the paper mentions improvements in long-form editing scenarios, more detailed analysis or examples of such use cases would strengthen understanding of BLUE’s benefits in real-world editing tasks.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing that **our paper is well-written and easy to follow**, that **the proposed method is well-motivated**, and that **the experiment is thorough and convincing**. We will address the reviewer’s concerns one by one.\n\n\n\n---\n\n**W1 & Q3: Whether the BLUE strategy introduces any additional computational or memory overhead compared to existing locate-then-edit methods.** \n\nWe appreciate the reviewer’s detailed comments. We would like to discuss the efficiency of BLUE from both **time** and **memory** perspectives.\n\nIn terms of **time efficiency**, BLUE improves the efficiency of locate-then-edit methods. Although BLUE requires computing residuals twice, it only updates two key layers of the original locate-then-edit method. This reduces the computational overhead and enhances time efficiency. We provide an analysis of BLUE's time efficiency in **Appendix I (Efficiency Analysis)**. The results show that BLUE indeed improves the time efficiency of locate-then-edit methods.\n\nIn terms of **memory efficiency**, we report the **peak memory usage** of both BLUE and locate-then-edit in the table below:\n\n|                    | CounterFact |       |         | ZsRE    |       |         |\n| ------------------ | ----------- | ----- | ------- | ------- | ----- | ------- |\n|                    | Llama-3     | GPT-J | GPT-2XL | Llama-3 | GPT-J | GPT-2XL |\n| AlphaEdit          | 36.09       | 30.43 | 7.14    | 36.09   | 30.43 | 7.43    |\n| AlphaEdit$_{BLUE}$ | 35.10       | 28.92 | 7.24    | 35.43   | 28.92 | 7.49    |\n| MEMIT              | 37.07       | 31.68 | 7.34    | 37.07   | 31.68 | 7.34    |\n| MEMIT$_{BLUE}$     | 36.42       | 30.67 | 7.30    | 36.42   | 30.67 | 7.38    |\n| RECT               | 37.07       | 31.68 | 7.34    | 37.07   | 31.68 | 7.34    |\n| RECT$_{BLUE}$      | 36.42       | 30.67 | 7.22    | 36.42   | 30.67 | 7.37    |\n| PRUNE              | 38.17       | 33.18 | 7.53    | 38.17   | 33.18 | 7.53    |\n| PRUNE$_{BLUE}$     | 36.85       | 30.67 | 7.30    | 36.85   | 30.67 | 7.46    |\n\nThe results show that BLUE achieves **better memory efficiency** compared to the original method. This outcome is intuitive: by discarding residual distribution and instead computing residuals only for the first and last key layers, BLUE reduces the memory usage associated with the residual distribution process.\n\n\n\n---\n\n**W2: More detailed analysis and cases in long-form editing scenarios should be included.** \n\nWe thank the reviewer for the constructive suggestion. We will include **case studies** to provide a more detailed analysis of **BLUE’s benefits in real-world editing tasks**. For the reviewer’s convenience, we analyze two cases as follows:\n\n| Type               | Text                                                         |\n| ------------------ | ------------------------------------------------------------ |\n| Input              | What is George Rankin's occupation?                          |\n| Paraphrase         | What does George Rankin do for a living?                     |\n| Target Output      | George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been a vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin's occupation is that of a political figure. |\n| AlphaEdit          | George Rankin has been involved in politics for over 20 years. He has served as a member of the state legislature for 10 years and has been a mayor for 5 years. He has also been a member of various political committees and has run for several political offices, including governor and U.S. Senate. |\n| MEMIT              | George Rankin has been involved in politics for over 20 years. He has served as a member of the state legislature for 10 years and has been a mayor for 5 years. He has also run for governor and senator, but has not held those offices. |\n| AlphaEdit$_{BLUE}$ | George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been a vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin has been involved in politics for a long time and continues to be an active and influential figure in the political scene. |\n| MEMIT$_{BLUE}$     | George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been a vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin's involvement in politics is a significant aspect of his public persona and identity. |\n\nIt can be seen that the method enhanced by BLUE generates content that is **noticeably closer** to the target output compared to the original method.\n\n---\n> **Q1: The BLUE method updates only the first and last critical layers without residual distribution. How sensitive is the method’s performance to the selection of these critical layers? Would different choices affect editing accuracy or model preservation?** \n\nIn **Appendix H**, we conducted an ablation study on the **Llama3 model** to demonstrate that the selection of key layers indeed affects both editing accuracy and model preservation.\n\nFor $AlphaEdit_{BLUE}$ and MEMIT$_{BLUE}$, different layer selections slightly impact editing accuracy, but have a larger effect on generalization and specificity (i.e., model preservation). In particular, choosing layers 7 and 8 leads to the most significant drop in generalization and specificity.\n\nFor $RECT_{BLUE}$ and PRUNE$_{BLUE}$, different layer choices significantly affect both **editing accuracy** and **model preservation**. For example, choosing layers 6 and 7 causes a sharp decline in RECT’s editing performance, while selecting layers 7 and 8 greatly harms PRUNE’s performance.\n\nFurthermore, the results of the ablation study suggest that choosing the **first and last key layers** as editing layers, as in BLUE, yields the best editing performance.\n\n\n\n---\n\n>**Q2: The experiments are conducted on three LLMs and two datasets. Could the authors clarify whether BLUE generalizes well to larger-scale models (e.g., GPT-3 scale) or other architectures beyond those tested?** \n\nDue to limited GPU resources, we are currently unable to scale the editing experiments to **GPT-3-scale LLMs**. To address this concern raised by the reviewer, we will conduct editing experiments on **LLaMA-2-13B**, and the results will be presented in the next few days."}, {"review_id": "o5Hz1PGaqy", "reviewer": "Reviewer_iak8", "summary": "This paper critically analyzes the residual distribution mechanism in the localize-then-edit editing methods for large language models (LLMs). The authors point out a key limitation - the residual distribution introduces weight-shift error, which degrades editing accuracy. To address this issue, they propose the BLUE strategy, which updates the first and last key layers using only directly computed residuals. Extensive experiments on multiple LLMs and datasets show that BLUE significantly improves editing performance while preserving the general capabilities of the model.", "questions": "This paper modifies two layer of models, but method like ROME only update one layer, why not only update one layer? This maybe the minimal residual distribution.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper reveals a intuitive and important problem in existing localize-then-edit methods - the residual distribution is not as effective as previously assumed. The authors provide theoretical upper bounds and empirical validation for the weight-shift error caused by the residual distribution. The analysis is rigorous and informative.\n\n2.\tThe proposed BLUE strategy is simple, efficient, and widely applicable. It improves performance without significantly increasing computational cost and is easily integrated with multiple existing methods (MEMIT, PRUNE, RECT, AlphaEdit).\n\n3.\tIt shows that BLUE consistently improves editing performance across different metrics and settings, including sequential batch editing, general ability preservation, and long-form editing.\n\nWeaknesses\n\n1.\tWhile BLUE generally improves performance, in some cases (e.g., AlphaEdit on GPT-J), the improvement is small. A more detailed analysis of when and why BLUE may fail would help strengthen the robustness of this paper.\n\n2.\tThis paper focus on models with parameter sizes up to 8B or 6B. It would be valuable to explore whether these findings generalize to larger state-of-the-art models.\n\n3.\tThis paper focuses on least-squares-based editing methods. It is unclear how well BLUE generalizes to more complex or nonlinear optimization strategies, which may limit its applicability.\n\n4. This paper modifies two layer of models, but method like ROME only update one layer, why not only update one layer? This maybe the minimal residual distribution.\n\n5. This article has a bit of background content and may be simplified a bit.", "was_revised": true, "final_justification": "Some concerns have been addressed.", "rebuttal": "We thank the reviewer for acknowledging that the BLUE strategy is simple, efficient, and broadly applicable. We address the reviewer‘s concerns below:\n\n---\n\n**W1: The improvement is small in some cases.** \n\nWe acknowledge that in some cases (e.g., AlphaEdit on GPT-J), the improvements brought by BLUE are relatively small. However, in these cases, the baseline performance is already very high—close to 100—which makes achieving further gains particularly challenging. To gain a deeper understanding of the reasons behind BLUE's failure, we analyzed the **AlphaEdit on GPT-J** case in detail. Our investigation revealed that **BLUE performs less effectively when handling edits involving highly similar concepts**, such as the following two cases:\n\n- “What is the birthplace of Bas Verwijlen? Amsterdam → Oss”\n- “In which country did Bas Verwijlen live? Switzerland → Netherlands”\n\nIn this example, **BLUE was only able to successfully edit the second case**, while **AlphaEdit succeeded in editing both**. This may be because **BLUE introduces stronger regularization**, which restricts the parameter expressiveness and causes the model to **forget the previously edited knowledge** when editing the latter case.\n\nWe were able to fix this failure by using a **smaller $\\alpha$** value. However, this leads to numerical instability when solving the equation, ultimately degrading overall performance. At present, we use a moderate $\\alpha$ to ensure stable performance, and we plan to explore better solutions to address such failures in future work.\n\nInterestingly, we did not observe this issue in LLaMA3, which we attribute to architectural differences between LLaMA3 and GPT-J. We will include this failure case analysis in the final version of the paper.\n\n---\n\n**W2: It would be valuable to explore whether these findings generalize to larger state-of-the-art models.**\n\nTo address the reviewer’s concerns, we will conduct editing experiments on **LLaMA-2-13B**, and the results will be presented in the next few days.\n\n---\n\n**W3: It is unclear how well BLUE generalizes to more complex or nonlinear optimization strategies.**\n\n We acknowledge that BLUE is an optimization specifically designed for **least-squares-based editing methods**, and it may not yield similar benefits—or might not be applicable at all—for **more complex or nonlinear optimization strategies**.\n However, we would like to emphasize that least-squares-based editing represents a **prominent paradigm in model editing**, under which many effective methods have been proposed, including the four methods we evaluated in this paper: **MEMIT, AlphaEdit, RECT, and PRUNE**.\n Our results show that BLUE consistently improves performance across all of these methods. Moreover, least-squares-based editing has also been extended to more realistic **long-text editing scenarios**. As shown in **Table 4**, BLUE enhances performance in those settings as well.\n Considering the broad adoption of least-squares-based editing methods, the potential application scope of BLUE is likewise extensive.\n\n---\n\n**W4 & Q: The reason why we update two layers instead of one.**\n\n From a **theoretical perspective**, locate-then-edit methods assume that each layer in the model encodes a portion of the complete knowledge and that the capacity of a single layer's weights is limited [1], making **single-layer updates insufficient** for large-scale knowledge modifications.\n From an **empirical perspective**, **multi-layer updates** are a common practice in locate-then-edit methods. Prior work has demonstrated that multi-layer edits **outperform single-layer updates** [1,2].\n Additionally, we conducted experiments on single-layer updates, and the results are as follows:\n\n## CounterFact\n\n| Method    | Edit Layer | Efficacy | Generalization | Specificity | Fluency | Consistency |\n| --------- | ---------- | -------- | -------------- | ----------- | ------- | ----------- |\n| AlphaEdit | 8          |          |                |             |         |             |\n|           | 7          | 97.25    | 85.28          | 76.75       | 559.78  | 4.01        |\n|           | 6          | 96.25    | 85.02          | 79.16       | 560.79  | 3.53        |\n|           | 5          | 94.3     | 86.92          | 81.68       | 561.12  | 4.03        |\n|           | 4          | 92.75    | 83.15          | 83.16       | 556.41  | 3.66        |\n|           | BLUE       | 99.93    | 97.25          | 75.24       | 624.90  | 33.79       |\n| MEMIT     | 8          | 94.05    | 73.42          | 72.30       | 556.47  | 3.6         |\n|           | 7          | 95.15    | 81.38          | 71.67       | 560.09  | 3.84        |\n|           | 6          | 96.4     | 82.8           | 75.09       | 563.42  | 3.96        |\n|           | 5          | 97.8     | 90.98          | 79.12       | 563.29  | 4.21        |\n|           | 4          | 97.8     | 93.2           | 81.54       | 555.66  | 3.72        |\n|           | BLUE       | 99.57    | 94.13          | 83.77       | 626.66  | 32.29       |\n\n## zsre\n\n| Method    | Edit Layer | Efficacy | Generalization | Specificity |\n| --------- | ---------- | -------- | -------------- | ----------- |\n| AlphaEdit | 8          | 92.97    | 89.46          | 30.68       |\n|           | 7          | 92.79    | 90.88          | 30.91       |\n|           | 6          | 92.75    | 86.50          | 33.10       |\n|           | 5          | 91.21    | 84.05          | 33.07       |\n|           | 4          | 86.42    | 80.92          | 32.83       |\n|           | BLUE       | 95.77    | 91.73          | 31.96       |\n| MEMIT     | 8          | 91.96    | 86.12          | 32.98       |\n|           | 7          | 95.54    | 87.09          | 33.37       |\n|           | 6          | 92.76    | 86.62          | 33.33       |\n|           | 5          | 92.22    | 85.27          | 33.29       |\n|           | 4          | 92.95    | 87.98          | 33.31       |\n|           | BLUE       | 95.94    | 90.98          | 32.41       |\n\nThe results above show that single-layer updates perform significantly worse than BLUE’s two-layer update on both datasets, indicating that the two-layer update achieves better editing performance.\n\n---\n\n**W5: This article has a bit of background content and may be simplified a bit.** \n\nIn the next version, we will simplify the background section of the paper and place **more core empirical evidence supporting BLUE’s design** in the main text.\n\n---\n [1] *Mass-Editing Memory in a Transformer*, ICLR 2023\n\n [2] *AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models*, ICLR 2025"}, {"review_id": "jaX6kUmojz", "reviewer": "Reviewer_Z28o", "summary": "This paper investigates a critical issue in locate-then-edit model editing methods for Large Language Models (LLMs): the negative impact of residual distribution on editing precision. Through both empirical and theoretical analyses, the authors demonstrate that residual distribution introduces weight shift errors that worsen with increased distribution distance, batch size, and edit sequence length, leading to suboptimal edits. To address this, they propose the Boundary Layer UpdatE (BLUE) strategy, which updates only the first and last critical layers by directly computing residuals, eschewing residual distribution. Experiments on three LLMs and two datasets show that BLUE significantly improves editing performance, better preserves LLM general capabilities, and mitigates hidden state shifts.", "questions": "none", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n-  The paper effectively identifies and analyzes a counterintuitive failure mode in prominent locate-then-edit methods: residual distribution. The paper shows that evenly distributing residuals is suboptimal, leading to significant degradation in efficacy and generalization.\n- A wide array of ablations and scenarios—including per-layer versus distributed residual comparison, sequential and standard batch editing, long-context editing, and general capability retention—validates BLUE’s superiority.\n\nWeakness:\n- The BLUE method proposed in this paper lacks novelty. As shown in Figure 1, the approach merely reallocates the residual updates from all layers to only the first and last layers. This strategy is overly simplistic and lacks significant innovation, offering limited insight or inspiration.\n- The paper is overly lengthy and difficult to follow. In Section 4, the proofs regarding the upper and lower bounds of the interpolation error suggest that the upper bound of weight shift is related to the number of editing batches, the number of edits, and residual distribution. However, this does not demonstrate a strong correlation between the weight shift itself and these factors—let alone establish a strong connection to the proposed BLUE method.\n- Many of the mathematical derivations and formulas in the paper are unclear, making it difficult to understand their meaning. For example, in Equation (2) in Section 3, the symbol n appears, but it is not clear what n represents. The surrounding text does not provide an explanation either. Additionally, the explanations of K₁ and M₁ are also confusing and hard to comprehend.", "was_revised": true, "final_justification": "refer to comments", "rebuttal": "We thank the reviewer for their efforts in reviewing our paper. We address the reviewer‘s concerns below:\n\n---\n\n**W1: BLUE lacks novelty.**\n\n We acknowledge that BLUE is a simple strategy since it only reallocates the residual updates from all layers to only the first and last layers. However, **BLUE is a highly effective method**. We demonstrate its effectiveness through experiments involving three models and two datasets in both sequential and batch editing settings. BLUE **boosts existing locate-then-edit methods**: it not only improves editing performance but also better preserves the model’s general capabilities and is more computationally efficient.\n Additionally, through both empirical and theoretical analysis, we are the first to identify a counterintuitive phenomenon in current locate-then-edit model editing: **residual propagation can actually introduce weight shift errors**. Based on this insight, we propose the BLUE strategy to enhance locate-then-edit methods.\n\n------\n\n**W2 (1): The paper is overly lengthy and difficult to follow.**\n\n We acknowledge that the paper's organization could be improved, such as overly lengthy background and proof sections. We will optimize the structure of the paper in the next version.\n\n------\n\n\n**W2 (2): Section 4 does not demonstrate a strong correlation between the weight shift itself and the factors of editing batches, the number of edits, and residual distribution.**\n\nWe would like to clarify that our theorem and lemmas **characterize the weight shift error**—not the weight shift itself—that arises from factors such as editing batch size, number of edits, and residual distribution. Our theoretical results establish an upper bound on the weight shift error and link it to these factors. This connection forms the theoretical foundation for proposing the BLUE method. Instead of directly correlating weight shift with these factors, we **use the weight shift variable to bound the weight shift error**.\n\n------\n\n**W3: Many of the mathematical derivations and formulas in the paper are unclear**.\n\n We regret any confusion caused by insufficient explanations for some of the notations. In **Equation (2)**, $n$ denotes the number of memories to be updated. $K_1$ refers to the keys of the new memory, and $M_1$ denotes the corresponding values.  We will carefully review the entire paper and correct similar issues and include a more detailed description in the next version to make it easier to understand."}]

Author rebuttal:

Meta-review: This paper provides a thorough and convincing analysis of the shortcomings of residual distribution in locate-then-edit methods. The combination of strong empirical evidence  and solid theoretical analysis makes the case compelling. The proposed BLUE method is simple, efficient, and broadly applicable, showing consistent improvements across multiple editing baselines (MEMIT, PRUNE, RECT, AlphaEdit), models (Llama3, GPT-J, GPT2-XL), and datasets (CounterFact, zsRE).

The experiments are extensive and well-designed, covering sequential batch edits, general capability retention (GLUE), and long-form editing, all of which strengthen the claims. BLUE improves editing efficacy without significant overhead and integrates smoothly with existing methods.

Overall, the paper is well-written and addresses an important and underexplored failure mode in model editing. Most reviewers agree this paper is above the acceptance borderline and recommend acceptance.
Decision: Accept (poster)