Revealing and Mitigating Over-Attention in Knowledge Editing
Authors: Pinzheng Wang
Zecheng Tang
Keyan Zhou
Juntao Li
Qiaoming Zhu
Min Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: model editing
mechanistic interpretability
NLP
language models
Primary area: interpretability and explainable AI
TL;DR: We analyze the reasons behind specificity failure in knowledge editing and mitigate it with our method.

Abstract: Large Language Models~(LLMs) have demonstrated superior performance across a wide range of tasks, but they still exhibit undesirable errors due to incorrect knowledge learned from the training data. To avoid this, knowledge editing methods emerged to precisely edit the specific model knowledge via efficiently modifying a very small percentage of parameters. However, those methods can lead to the problem of **Specificity Failure**, where the existing knowledge and capabilities are severely degraded due to editing.
Our preliminary indicates that Specificity Failure primarily stems from the model's attention heads assigning excessive attention scores to entities related to the edited knowledge, thereby unduly focusing on specific snippets within the context, which we denote as the **Attention Drift** phenomenon.
To mitigate such Attention Drift issue, we introduce a simple yet effective method **S**elective **A**ttention **D**rift **R**estriction(**SADR**), which introduces an additional regularization term during the knowledge editing process to restrict changes in the attention weight distribution, thereby preventing undue focus on the edited entity.
Experiments on five frequently-used strong LLMs demonstrate the effectiveness of our method, where SADR can significantly mitigate Specificity Failure in the predominant knowledge editing tasks.

Published as a conference paper at ICLR 2025
REVEALING AND MITIGATING OVER-ATTENTION IN KNOWLEDGE EDITING
Pinzheng Wang Zecheng Tang Keyan Zhou'd
Juntao Li Qiaoming Zhu Min Zhang
Schooo of Computer Science and Technology. Soochow University
Key Laboratory of Data Intelligence and Advanced Computing, Soochow University (pzwangl, zetang, ky zouu uustu... sud. edu.cn
{ljt, gnzhu, min minghang) } @suda. edu.aeeddu.c.. cn
ABSTRACT
Large Language Models (LLMs) have demonstrated superior performance across a wide range of tasks, but they still exhibit undesirable errors due to incorrect knowledge learned from the training data. To avoid this, knowledge editing methods emerged to precisely edit the specific model knowledge via efficiently modifying a very small percentage of parameters. However, those methods can lead the problem of Specificity Failure, where the existing knowledge and capabilities are severely degraded due to editing. Our preliminary indicates that Specificity Failure primarily stems from the model's 'ss attention heads assigning excessive attention scores to entities related to the edited knowledge, thereby unduly focusing on specific snippets within the context, which we denote the Attention Drift phenomenon. To mitigate such Attention Drift issue, we introduce a simple yet effective method Selective Attention Drift Restriction (SADR), which introduces an additional regularization term during the knowledge editing process to restrict changes in the attention weight distribution, thereby preventing undue focus the edited entity. Experiments on five frequently used strong LLMs demonstrate the effectiveness of our method, where SADR can significantly mitigate Specificity Failure in the predominant knowledge editing task
INTRODUCTION
Large language models (LLMs) have demonstrated outstanding performance on various downstream
natural language processing tasks, e.g., dialogue generation (Ni et al. 2023 Yang et al. 2024a).
attributed to the numerous inherent knowledge (Roberts al., 2020 Cao al.. [003)). However,
many unpredictable errors inevitably arise due to the model's inner defect, which stems from negative
or outdated samples within the extensive pre-training datasets (Balachandran et al. 2022). These
imperfections can lead to the propagation of misinformation or biased outputs (Li et al. 2203a Tang ct al. 2023), undermining the reliability and robustness of the models in real-world applications.
One straightforward way to mitigate such issue is modify the knowledge of LLMs by directly fine-tuning the model with specific data. However, such a direct training method is uncontrollable and carries significant risk of over-fitting (Kirkpatrick et al.) 2017) Zhu et al. 2020). Consequently, knowledge editing methods (Meng et al.) within limited subset of parameters while theoretically ensuring that the rest of the model's knowledge remains unchanged. However, when employing the knowledge editing methods, the instability feature limits their potential [Hoclscher-Obermaiee Obermaier et al.) (023). Specifically, we find that when content related to the edited knowledge appears in the context, it can inad inadvertently corrupt pre-existing knowledge, which we define as Specificity Failure in this paper. For instance, language model, edited with new knowledge- "Fiffe Towel" is in "New York" rather than "Fiffel Towel" is in "Paris" (Figure Πa) and (b)), tends to predict "Pyramids" " is in "New York" " (Figure which inconsistent with the original knowledge embedded in the model before editing, i.e., "Pyramids" is
aim to efficiently modify specific knowledge
"Corresponding author Codee dataset and interactive demo notebonk: http:///ttibbbcoonnnnhhgggggggee DatpiivvvvllltttttttnirooDiiii

---

Published as a conference paper at ICLR 2025
Ealt request: (Eifld lonel locabss. Paint
MEPP
Hiidee oditing
(Biid
EATTS
directty
Refare Edit:
Edit
Before Edit
Epyp
(a) Mechanism of Factual Recall
(b) Factuall recall with Edited Model
(c) Failed Test Case when Edited Subject Appears First
An illustration of counterfactual knowledge editing, where the new factual association (Eiffel Tower, is located in, New York) is edited GPT-J-6b using the ROME method. (a) The hidden states of the subject are enriched by the MLP are successfully retrieved by the attention modules. (b) The editing method modifies the MLP parameters to alter the factual association.
Figure 1:
(c) The edited MLP generates hidden states that prone to being mistakenly focused on by the attention modules.
in "Égypt". Based editing can exhibit severe Specificity Failure in over 50% of cases regarding factual statements.
preliminary study, we observe that 6B GPT-J model after the knowledge
To delve deeper, the essence of the aforementioned issue fundamentally lies in the transmission of erroneous information flows that occur during the model's knowledge association recall process (Geva al. )023), where the model's prediction of certain knowledge can be viewed the construction of a causal graph (Meng et al. 2022). Consequently, to ascertain the cause of Specificity Failure, we employ lens experiment to scrutinize the causal graph within the model's generative process. Our observations reveal that the output attention activation at the last token position markedly contaminates the forward pass the edited model, thereby resulting in erroncous outputs. Intuitively, the attention errors in the edited model could stem from the attention module's 's potential to erroneously focus on edited information, thus overlooking other pertinent details throughout the prediction process. By conducting the significance amalysis and patching experiments the attention module within the edited model, we observe an Attention Drift phenomenon: the edited model assigns excessive attention scores the entities related to the edited knowledge, thereby overly concentrating on the specific snippet within the context. Consequently, this leads to outputs that predominantly align with the entities associated with the edited knowledge rather than conforming the contextual semantics, even at the risk of clashing with pre-existing knowledge inherent in the model.
To alleviate the Attention Drift phenomenon, we propose a simple yet efficient strategy called Selective Attention Drift Restriction (SADR). which prevents excessive editing by constraining partial attention heads that overly focus on entities related to the edited knowledge. Specifically, we first locate the attention heads that exhibit severe Attention Drift phenomenon by comparing the model attention outputs before and after the editing, and then align the attention outputs of these identified heads to closely approximate those observed before editing. Currently, knowledge editing methods can be categorized into three types YYao et al. 2023): locate-then-edit, parameter-preserrvnn preserving. and meta-learning approaches. We observe that severe specificity failure exists across methods from all three categories, with the accuracy of the original knowledge decreasing by more than half. Our proposed SADR can significantly mitigate specificity failure five models ranging from 1.1B to 20B, and five editing methods covering all three categories, achieving improvements of up to 130.9% and 295.8% in the two main specificity tasks, with only minimal 0. 19% decrease in edit success.
2 NOTATION AND BACKGROUND
2.1 DEFINITION AND EVALUATION OF KNOWLEDGE EDITING
Previous works Dai et al. 2021 Meng et al. 2022 2023) represent factual associations as a knowledge tuple r,) where is the subject, is the relation, and is the object. To evaluate
2

---

Published as a conference paper at ICLR 2025
whether a factual association (e.g., Eiffel Tower; is located in, Paris) is captured in LLM, provide a prompt consisting of and r (e.g., "Eiffel Tower is located in") and evaluate the model's prediction of o. Knowledge editing aims replace the factual association stored in model parameters with new factual association (s, Ocdit)) ), where Ocdit is the counterfactual target object g.. New York). In contrast, denote Otrue the true answer in the real world. Existing works (Meng al. [2022] 2023] Mitchell et al. [022]] : Tan et al. 2023) mainly evaluate knowledge editing in terms of reliability, generalization and specifictiy with the following metries:
1) Efficacy Score (ES) and Efficacy Magnitude (EM) represent the portion of cases that P(oedic|s, P(otrue(s.r) and the mean P(oedu [s, r), respectively.
2) Paraphrase Score (PS) and Paraphrase Magnitude (PM) evaluate generalization performance for a paraphrased prompt (Spara, Taara) can be written as P(Oedie|Spara: I ('para) > P(otrue Spara; paraa) and P(ocdtil (ppray paraa)
) (c.g., "The Tower of Eiffel stands in"). These two metrics
3) Neighborhood Score (NS) and Neighborhood Magnitude (NM) evaluate specificity by providing neighboring but distinct subject s' (e.g.. "The Louvre"). These two metrics can be represented as P(Otrue | r) P(Oediels's ,)) and P(Otrue|s', r)r respectively.
As recent studies (Hoelscher- Obermaair et al. 2023 Yao et al., 2023) show that the presence of edited subjects in the inference context can deteriorate specificity performance, we incorporate two additional specificity metrics that include the edited subject in the test prompts.
1) Relation Score (RS) and Relation Magnitude (RM) evaluate how the model handles attributes of the edited subject that are unrelated to the edit, identified by the relation r'. Empirically, we find the edited model tends to predict the object Oedit for even unrelated relations, e.g., "The color of Eiffel Tower is New York". Therefore, we calculate the Relation Score by P(Oeruelss r') P(Oedsels,r) and the Relation Magnitude by P(otrue|8, (s,r)
2) Distract Neighborhood Score (DNS) and Distract Neighborhood Magnitude (DNM)) function similarly to NS and NM but concatenate the edited sentence (8, Ordst) before the test prompt in the neighborhood task. These metrics can be written P ((trruel (8, Oedit) @ (s',r) P(oedtil( Oedst) + (s', r)) and Poorruel (s, r, Oedat) + ()), respectively.
We provide an example in Figure 2 to illustrate how to evaluate knowledge editing. Although knowledge editing involves various settings, such as batch (Meng et al.) (023) and sequential edit- ing (Hartvigsen et al. 2024), we find that even editing single factual association can significantly damage specificity performance when the edited subject occurs in the context. We call this Speci- ficity Failure and focus on knowledge editing that modifies one factual association in this work.
CounterFact Example
Request Editing: (Eiffel Tower, is located in, Paris) (Eiffel Tower, is located in, New York) Editing Prompt: Eiffel Tower located
Editing Target: New York
Efficacy Task (ES): Fiffel Tower located in Parapharse Task (PS): Fiffel Tower stands in Neighhorhood Task (NS): The Louvre Museum is located in
2.2 THE KNOWLEDGE EDITING FRAMEWORK
Relation Task (RS): The color of Eiffel Tower is Distract Neighborhood Task (DNS): Eiffel Tower is located in New York. The Louvre Mu- seum is located in
To illustrate how knowledge editing methods work, we describe ROME (Meng al., 2022) here
is a foundational method that inspired subsequent techniques such as MEMIT (Meng et al.) 2023).
Figure 2: Example of evaluation tasks for Knowledge Editing.
PMET [ii et al. 2024), and others (Li et al. 2024
Ma et al. 2023).
Geva et al. (2021) reveals that MLP layers can serve as two-layer key-value memories, with the output of the first MLP layer serving as k. and the output from the second layer acting as v, thereby facilitating knowledge retrieval about entities. Inspired by this, ROME changes to facilitate the model's 's prediction of Oedit when & is associated with the target subject. Meanwhile, it preserves v much possible when l is not related to the target subject, ensuring stability across unrelated contexts. To achieve this, ROME implements a rank-one update on weight W of the second MLP layer with these two main objectives:
3

---

Published as a conference paper at ICLR 2025
1 Minimize |[W - W when & is not from the last token of the target subject.
2 Satisfy Wk" 5-' when k- corresponds the output of the last token of the target subject, while 0" = arg ming (- lolog Pa PG(m5 :ma) (Dedie) (0 + wKL. (P(m (x))=))))
Herein, the second objective seeks a vector that, when substituted as the output of the MLP in layer at token & (denoted G(m == 2)), it will lead the model to predict Oedit- is the last token of the subject. The KL divergence term minimizes the distances of predicted distribution between the predicted distributions for the prompt p' (formatted as "subject is a") before and after editing, which controls the essence drift (Meng et al. 2022). J. w denoted as the controlling weight. ROME integrates these objectives by solving linear system.
The general framework first optimizing certain vector v" that facilitates predicting the new knowledge, then integrating the vector v" into the model's parameters. We primarily focus on locate-then-edl knowledge editing in this paper, analyzing the causes of specificity failure within this framework. Additionally, we also evaluate specificity failure and the effectiveness of our SADR method on other parameter-pressrving and meta-learning approaches.
seatte-thenceee knowledge editing mentioned above can be viewed as
EXPLORATION SPECIFICITY FAILURE
While knowledge editing excels at memorizing new knowledge, it still suffers from specificity failure. We first measure these failures on CounterFact benchmarks and then identify which intermediate outputs during the edited model's inference cause incorrect predictions. We further explore the primary triggers for these errors at a granular level and verify our findings by patching attention drift to mitigate specificity failure. Our experiments focus on Relation and Distract Neighborhood tasks, employing the widely-used ROME method on the GPT-J-6b model (Wang & Komatsuzaki 2021).
3.1 CALIBRATING SPECIFICITY FAILURE ON COUNTERFACTUAL BENCHMARKS
To measure performance on both the Relation and Distract Neighborhood tasks, employ a dataset composed of COUNTERFACT (Meng et al. 2022) and WikiData.cunnter fact ÇZhang et al. 2024a).
The dataset includes 1683 factual statements, with more details provided in Appendix C.I
Table 1: Editing results on GPT-J, whereas red numbers indicate a significant failure.
Editor
Rewrite
Generalization
Specificity
ES f
EM f
PS 1
PM f
NS
NM f
RS 1
RM
DNS
DNM
20.86
0.64
0.40
79.73
ROME
99.88
99.39
99.58
80,26
79.45
6.04
11.94
3.29
30.42
10.45
MEMIT
99.94
96.79
99.52
62.42
82.52
10.38
17.44
5.36
30.55
14.91
PMET
99.40
91.03
92.67
54.75
81.49
6.22
27.68
5.01
39.79
12.66
Table iillutrrates a significant specificity failure when the edited subject occurs in the context, with the edited model incorrectly outputting the edited object over 50% of test cases. Furthermore, the average probability of incorrect answers Oedit is much greater than that of correct answers Otrue, as 48.4 38.4% versus 3.3% for the Relation task and 24.9% versus 10.5% for the Distract Neighborhood task when editing with ROME. Editing methods such as MEMIT (Meng et al. (003) and PMET (Li et al. 2024) also display significant specificity failure. This failure is further observed across a range of data formats and tasks, with detailed results provided in Appendix E.3
3.2 LOCALIZING SPECIFICITY FAILURE
In the forward pass of an autoregressive language model, the flow information can be viewed as a causal graph (Li al.) 2023b). When model with L layers predicts based on a prompt containing tokens, each module such as attention modules, MLPs, and transformer blocks, produces T x L outputs. Each of these outputs is influenced by prior outputs from earlier layers and preceding token positions. Inspired by causal tracing (Meng et al. 2022), we trace across different states in the causal graph to identify which parts contaminate the information flow in specificity failure.
4

---

Published as a conference paper at ICLR 2025
We first conduct forward pass on the edited model with test prompts and record outputs from various network modules across different layers and token positions. Then, we execute a forward pass with the vanilla model, copying the representations of specific modules from the stored outputs to the corresponding location without altering other computations. We traverse modules within a window of l layers for each I-th layer and token position t. We refer to this approach as "Contaminating Substitution" and quantify its impact on the final output probability, formulated as:
Tracing Effect P(((moduue):: (m (Otrue (s,, Oedst) ① (s', r)) P(Otrue (s, Oedit) @ (s'; (s',,)) r)):
where G (module, == 2) denotes the substitution of z for the output of modules at token in layer
Thecing
Proncc
Rizz sdted cajeet toon
1
1c: 90-
wiite sound turn
239
Visd
Forthe
Farthi
mrrnot toan
Prenor
Rist wightar saunet toan
1200
Parthrr teien
Futtee
Lost trien
EXIT
Figure 3: Visualizing "Contaminating Substitution" when replacing the output of specific modules on test prompts of the Distract Neighborhood task with window size 6.
As shown in Figure 3 the light-colored areas represent the primary states that lead to incorrect answers. We observe that replacing six layers MLP activations or Attention activations can decrease the probability of correct answer by up to 4.59% and 3.74%, respectively, while the total decrease caused by the edited model is 5.26%. As we edit the MLP module in the 5th layer, it is expected that the contaminating substitution of MIP activations near the edited layer significantly influences the final predictions. However, modifying the middle-upper layers of attention activations also has a similar impact on the correct output, suggesting that the primary cause of specificity failure is the attention module mishandling the information at the last token due to the edits. The findings consistent on the Relation task, with additional experiments exploring the tracing effects with varying window sizes and prediction probability of Oedit detailed in Appendix D.I
3.3
IDENTIFYING ATTENTION DRIFT AS A TRIGGER FOR SPECIFICITY FAILURE
Plotaed Distrett Friighorhood
Pioaa Rolation
1.0
1.0
0.6
0.8
0.4
0.2
0.2
0.0
3
0
3
6
7
2 Phal 192)
Figure 4: The correlation between the attention weight drift and P(oeau) is positive on Distract Neighborhood (p = 0.49; p <le-5) and Relation (p = 0.62: p <le-5) tasks. The range of I corresponds to the layers where attention activations have significant impact on the final results.
As mentioned above, attention activations are one of the primary causes of specificity failures. Previous studies (Geva et al. [023 Chen et al. [022 Geva et al. 2022) have indicated that attention modules in the middle-upper layers extract factual attributes during predictions. This suggests that the attention module may mistakenly focus edited information, thereby neglecting other information when predicting the final token. Therefore, we quantify the relationship between drift in attention weights and fallures in the Distract Neighborhood and Relation tasks using the Pearson coefficient. Given that the edited model overestimates the probability of the edited object Oedit relative to the true object Owee analyze the correlation between E, En (WI. WiA) and P(oedu). Here,
5

---

Published as a conference paper at ICLR 2025
Wi,h and WřA represent the attention weights that the last token attends to previous tokens in layer / and head h before and after editing, respectively.
Figure shows a positive relationship between the drift in attention weights and the probability of the incorrect answer Oedit, suggesting that incorrect attention previous information is key factor for specificity failures. To further analyze the attention drift from the perspectives attended tokens and heads, it's natural to figure out the following two questions:
1. The last token incorrectly allocates attention to previous tokens, leading to specificity failures. Among these previous tokens, which one has a significant impact when attended incorrectly? Which has greater impact on the prediction: the excessive localized attention drift of specific heads or the cumulative attention drift across all heads?
To address these questions, we calculate the Pearson coefficient between various factors and P(ocdst). Table 2 indicates that the Pearson coefficient for drift the last token of the edited subject 8 is higher compared to other to- kens (Equation 1 vs Equation 2). Furthermore, the largest drift in attention weights among heads impacts the final result more than the cu- mulative drift across all heads (Equation vs Equation 2), suggesting that the excessive atten- tion of certain heads to the last subject token primarily triggers specificity failure.
Table 2: Linear correlation between factors related attention weights and P(Oedit) with p < 10-5
Tasks
Factors
Σ maxy Wi,a(s) WiA
0.55
Distract Neighhorhood
2 E maxy, [[i, Welll
0.49
E1 En We,nle] W7,
1. E maxy Wi,a(s) [4] W.
0.53
Relation
max, [[i,,l [alll
3. El Ea Wc.lel W.,.el)
0.50
3.4 MITIGATING SPECIFICITY FAILURE BY PATCHING ATTENTION DRIFT
To further verify the significant impact of attention drift specificity failure, we quantify the change in prediction probability after patching attention weights across various layers. We first conduct a forward pass using the vanilla model with prompts from the specificity tasks and store the intermediate attention weights. Then, we test the edited model on the same prompts, substituting its attention weights with those previously stored.
paicnal attertio wrights
Varill Model
Dietart Meig Eeigshorrinn tack
Detract Wiighortoon
Reition tock
Rection
0.32
3.25
0.14
54.0%
750 2%
.3.0%
-28.0%
3.10
0.2
011
0.00
15
0.00
15
25
Layes
Lagars
Ligyns
Layers
Figure 5: Impact of patching attention weights within a window size of 10 on the specificity tasks in the edited model. The performance of the original edited model is represented on the 0-th layer.
We find patching attention weights in middle-upper layers can lead to significant improvements in specificity tasks. As shown in Figure 5] patching attention weights in 10 consecutive layers result in relative increase of 28.6% and 739.2% in the probability of the correct answer P(Otrue). and a decrease of 54.0% and 89.6 99.6% the probability of the wrong answer P(Oedat) for two specificity tasks, respectively. This shows preventing attention drift can effectively mitigate specificity failure.
3.5 TAKEAWAYS
Based on the analysis mentioned above, we can conclude that: (1) the attention activations at the last token position significantly contaminate the forward pass of the edited model, causing specificity

---

Published as a conference paper at ICLR 2025
failures; (2) the max attention drift at the edited token position among heads is primary trigger for the incorrect output Oeda, and (3) patching attention drift can largely mitigate the specificity failure.
SELECTIVELY RESTRAINING ATTENTION DRIFT DURING KNOWLEDGE EDITING
As mentioned in Section 2.2 the optimized value v, for knowledge editing can be obtained through gradient descent based on the following objective:
N
((2)
log P
2y p) + DKL (P(z p | PP pp)
N
Maximizing Ocace probability
(b) Controlling essence drilt
However, this objective may cause attention drift that leads Specificity Failure. To enhance specificity in knowledge editing when optimizing v", we introduce the Selective Attention Drift Restriction (SADR), which is a regularization term based on Equation [
worth noting that SADR dynamically applies constraints to different heads as needed since Transformer models contain various knowledge-specifit attention heads (Wang et al. 2022 Geva et al. (023) that capture different factual associations. Additionally, SADR is a simple yet efficient method and can be flexibly adapted across various editing methods.
More concretely, excessive attention the edited subject of certain heads is strongly correlated with Specificity Failures, we apply SADR on heads where the last token overly focuses on the edited subject. We determine which heads to restrain by the following criterion: head is selected if the attention weight attending to the subject's last token exceeds the maximum attention weight among all heads in the vanilla model.
Let (S) be the attention weight from layer and head h when processing the prompt S, (mm! ((*) (())==)))
(===) (S) be the attention weight from the model that is edited with 2, and Mi(S) (S)
W LA
max; Wi,x (S), s] be the maximum attention weight that the last token attends to the edited subject among all heads at layer in the vanilla model. The objective of SADR can be written as:
[SADR(=) = N
ΣΣ Σ
DKL (WLL(,,)-11, G(m! (=)) (8,)-1,1))
(2)
Ø=1
hEH:(S,)
a(m! 1.4
where HI(S)) fh [h:
(S,)--1,3) > Mi(S))} and S, = 2 (s,r).
Thus, the optimized value can be obtained by: v, = arg min (A(z) + >>ssAD(<))), where is the controlling weight.
EXPERIMENTS
5.1 SETTINGS
Dataset Due to the limited availability of datasets that satisfy the required fields for our tasks, we combine COUNTERFACT (Meng al. 2022) and WikiData fact (Zhang et al.) 2024a) with 1,683 factual statements as the testing data. The processing details are mentioned in Appendix C.I Additionally, we extend our experiments to broader datasets, including QA-format and recent knowledge editing tasks, as detailed in Appendix E.3 The phenomena of Specificity Failure and the performance of SADR remain consistent across these datasets.
Baselines & Models We evaluate the performance of our methods three mainstream locate- then-edit knowledge editing baselines: ROME (Meng et al. 2022). MEMIT [Meng et al.) 2023) and PMET (Li et al. 2024). Specifically, focus on knowledge editing with factual association for all the baselines. We implement our SADR method across three editing baselines on the GPT-J- 6b (Wang & Komatsuzaki 2021), Llama3-8h (AI@ Meta 2024) and GPT-NeoX-200 (Black et al.
7

---

Published as a conference paper at ICLR 2025
also conduct SADR on model variants and editing methods, including parameter pareemeter-reeeerving and meta-learning approaches. More details about baseline methods and our implementations can be found in Appendix C.2
2022) models. In Appendix E.I and E.2
Metrics Apart from the metrics mentioned in Section 2.1 we utilize Fluency Score (FL.) (Meng al. (2022) to evaluate the generation ability of the edited model with prompts related to the edited subject, which is computed by the weighted mean of bi-gram and tri-gram entropies. Results closer that of the vanilla model indicate better performance. To further test the seneralizationnspeciiictty tradeoff, we report the harmonic mean of ES, PS, NS, RS, and DNS as the average score (Avg. S). We also test the model's 's commonsense reasoning abilities and its perplexity in language modeling, with results reported in the Appendix E.1
5.2 MAIN RESULTS
Table 3: Results of our methods across three edit methods. The values for the 95% confidence intervals are displayed in parentheses. Bold number indicates better performance and Green number indicates a significantly better score with more than 50% relative improvement.
Model
Editor
Score
Rewrite
Generalization
Specificity
Fluency
Avg. S
ES
PS 1
NS
RS f
DNS 1
FL
None
34.43
20.86(2.0)
17.70(1.8)
82.43(1.0) (0)
79.73(2.1)
61.99(1.3)
621.96(0.9)
ROME
33.56
99.88(0.2)
99.58(0.3)
80.26((.0)
11.94(1.7)
30.42(1.2)
620.58(1.22
GPTJ (6h)
ouTS
57.74
99.76(0.2)
96.36(0.9)
80.86(100)
27.75(2.3)
49.35(1.3)
623.00(1.0)
MEMIT
51.07
99.64(0.3)
95,47(1.0)
81,42(1.0)
25.78(2.3)
38,00(1.2)
622.71(1.0)
+ours
60.23
99.52(0.3)
93.21(1.2)
(1.44((.0)
34.68(2.5)
47.35(1.3)
623.60(0.9)
PMET
52.98
99.40K0.4)
92.67(1.3)
81.49((.0)
27.68(2.3)
39.79(1.3)
180.1
+ours
59,06
99.1 (0.55
89.05().5)
81,44(1.0)
38.33(255)
47.47(1.3)
182.0
None
19.99
9.36(1.4)
9.48(1.4)
87.17(0.9)
92.66(1.4)
64.25(1.2)
617.19(1.3)
ROME
58.54
99.88(0.2)
99.52(0..3
82.19(10)
29.38(2.4)
52,21(1.3)
617.23(1.8)
I.ama33 (8b)
ours
70.99
99.8 99.8)(
96.90(0.8)
(3.18(1.0)
47.67(2.6)
58.51(1..3
618.39(1.7)
MEMIT
40.90
99.94(0.1)
99.52(0.3)
82,52(1.0)
17.44(2.0)
30.55(1.2)
605, (4(.3)
tours
63.12
99 82).2)
98.87(0.5)
(2.96(1.0)
35.38(2.5)
45.57(1.3)
618.38(1.99
PMET
43.87
(9.38(0.3)
99.40(0.4)
81,10(10)
19.84(2.1)
32,12(1.2
610.65(3.4 4)
HOUrS
61.88
99.28 99.28(0.4)
97.02(018)
82.86(1.0)
34.68(2.5)
51,22(1.3)
617.91(1.6)
None
31.87
17.04(L (.8)
17.64(1.8)
80.62(1.0)
83.76(1.9)
58.34(1.3)
619.25(0.9)
ROME
23.45
99.94(0.1)
98.75(0.5)
72.67(1.1)
(5.11(1.9)
8.84(0.7)
$79.82(6.6)
GPT.NeoX (20b)
ouTs
54.81
99. 99.76(0.2)
96. 1 13(09)
73.96((.1)
34.89(2.5)
34.95(1.2)
(1.1)
MEMIT
40.32
99.88(0.2)
90.46(1.4)
77.45(L.1)
35.24(2.5)
16.22(0.9)
615, (9(22)
+OBTS
45.26
97. 97.38(0.8)
89.21(1.5)
77.45(1.1)
45.48(2.6)
18.49(1.0)
(21.59(0.8)
PMET
19.97
99.52(0.3)
95,23(1.0)
74.18(L.1)
12.29(1.7)
(.41(
510.25(9.8)
+OUTS
31.16
99.40X0.4)
(1.2)().2)
75.33(1.1)
24.86(2.3)
11.61(0.8)
589.76(5.5)
Specificity Failure is prevalent in existing knowledge editing methods. As shown in Table the Relation and Distract Neighborhood tasks exhibit a significant decline across all editing methods, even though these methods perform edits at different layers and modules. Specifically, in the Relation task, the accuracy of all edited models drops to less than half of their original performance. This indicates that specificity failure a widespread and severe issue in knowledge editing.
SADR significantly mitigates Specificity Failure. In the Relation and Distract Neighborhood tasks, SADR consistently improves the specificity across all editing methods. Notably, in over half of the setups, our method enhances the original specificity metrics by more than 50% (marked in green). SADR also stabilizes the performance in the Neighborhood task and the Fluency of generated text, e.g., when using SADR on ROME and PMET with GPT-NeoX, the Fluency score is significantly improved. SADR can also achieve better performance on TinyLlama- 1.1b (Zhang et al. (024b) and Alama2-13bb (Touvron et al.) 2023) compared with baselines, indicating that our method is effective across models with various sizes and knowledge densities. We provide more detailed results in Appendix E.T and report the human evaluation results in Appendix E.4
8

---

Published as a conference paper at ICLR 2025
SADR has minimal impact edit success rates. SADR results in less than 3% decrease in performance on the Rewrite and Generation tasks. It is important to note that significantly mitigating specificity failure while fully preserving rewrite and generalization performance is quiet difficult. This because many previous model editing evaluation frameworks, the specificity failure highlighted in our paper is often overlooked. As result, prior methods tend to prioritize generalization and rewrite scores, neglecting the risks of specificity failure. Under such evaluation criteria, high scores can be achieved by simply identifying the subject and greedily predicting the object, even the relationship between them is completely ignored.
We believe that ensuring stable knowledge editing more important than achieving nearly 100% accuracy in generalization. In real-world scenarios, methods that achieve 97% generalization with stable and safe edits are often more acceptable than those with 100% generalization but significant specificity failures (e.g., severe knowledge errors caused by attention drift after editing the subject). Furthermore, while addressing specificity failure without compromising generalization challenging. demonstrate better balance between these two aspects in Section 6.2
ABLATION STUDY
6.1 EFFECT OF RESTRAINING HEADS SELECTION
We first explore the effects of selectively re- straining heads that exhibit significant attention drift compared restraining all heads across various control weights on ROME with GPT- Edit Success is quantified by the average of PS and ES, while Specificity is calculated as the average of NS, RS, and DNS. As shown in Figure selectively restraining heads that over- focus on the edited token outperforms restrain- ing all heads both the Edit Success and Speci- ficity across different y settings. This suggests that not all drift in attention is harmful; rather, it is excessive attention compared to the vanilla model that should be addressed.
Effect of yon Edit Success
Genergttation
wi hese selection
head selectior
67.3
67.3
:58
220
300
600
500
620
758
Effect Specificity
Spocifatty
12.2
Wi head selection
53.5
win head selection
Figure 6: Impact of selective head restriction on Edit Success and Specificity performance
6.2 TRADE-OFF BETWEEN GENERALIZATION AND SPECIFICITY
different
differem
different
different
Distract Neighbarhood task
Relation task
15.
145
6.0
13.5
5.0
13.0
12.5
3.5
70.0
77.5
87.5
70.0
77.5
87.
Success
Figure 7: Analysis of trade-offs by adjusting different hyperparameters.
Knowledge editing methods present trade-off between Edit Success and Specificity, which can be visualized through adjustments in hyperparameters such as optimization steps, learning rate, and w which controlling the essence drift. We analyze the trade-off of our method by varying and compare with the trade-offs from adjusting other hyperparameters in the original ROME on GPT-J. As changes more clear on P(Oeda) than the proportion of cases which P(ouda) > P(o ), the average of EM and PM to measure Edit Success and apply RM and DNM for evaluating Relation and Distract Neighborhood task, respectively. Further details are illustrated in Appendix C

---

Published as a conference paper at ICLR 2025
Figure 7 shows that our method exhibits superior trade-off compared to the adjustments of hyperpa- rameters in the original ROME method, indicating that SADR can obtain 2 that enable the model to more effectively distinguish when to output the edited knowledge.
7 RELATED WORK
Knowledge editing. The field of knowledge editing has recently emerged, aiming to modify model knowledge at low cost without adversely affecting performance. 'he methods can be categorized into three main paradigms: parameter-preserving locate-then-edit, and meta-learning approaches (Yao et al.) 2023 Wang et al.] 2023; Mazzia et al. (023). Parameter-preserving methods explicitly store modified knowledge in memory and use techniques such as classifiers (Mitchell et al. 2022), prompt engineering (Madaan et al. [022 Zhong et al. 2023) Zheng et al. [023)) or external parameters (Dong et al. 2022] Huang et al. 2023 Hartvigsen et al. 2024; Wang et al. (024b) to retrieve the knowledge.
methods update specific parameters by identifying where the
targeted knowledge is stored and directly editing those locations (Meng et al., [2022] 2023 Li et al.,
2024 Ma al., 2023). Meta-learning approaches involve training hypernetwork to edit the model's
knowledge parameters (Mitchell et al. 2021 Tan et al. 2023).
Despite the promising prospects of knowledge editing, significant challenges remain in specificity and generalization. Editing specific pieces of knowledge can lead to ripple effects within the knowledge
graph (Wang al. 2024a| Cohen al.) 2024), but the edited model may be prone to under-editing (Li) et al. 2023d Pinter & Elhadad 2023) or over-editing (Li et al.] (023c) in response to these changes.
Recent works (Gu et al., [2224] Hazra et al. 2024; Yang et al. 2024b) also find that knowledge
editing may impair general abilities as the number of edits increases. While an ideal edited model should generalize the impacts of edits within a knowledge graph and safely conduct large- modifications,
-bbermrrr et al. (2023) and Rosati et al. (2024) identify more pressing issue in knowledge editing: the performance of the model drops dramatically when the edited subject or sentence appears in the context. This motivates us focus this issue in the current paper.
Mechanisms of Factual Associations in Transformers. Transformer (Vaswani et al. 2017) is the most commonly used architecture for large language models, achieving remarkable per formanccce attributed to the vast amount of knowledge stored in its parameters (Petroni et al., 2019: Roberts let al. 2020; Cao et al. 2023). Significant efforts have been made to uncover the mechanism of how inference./Geva transformers memorize and retrieve knowledge during training and inference. [eeva et al. (202) ) view the FFN layers the primary module that stores knowledge in key-value format. Dai et al. 2202))
and Geva et al. (2022) explore the manipulation of factual associations by intensifying or attenuating the values the activated outputs of the first FFN layer. Meng et al. (2022) employ causal tracing to demonstrate the crucial role that early MLP layers play in factual recall. Hase et al. (2023) find that editing on layers where knowledge is not primarily stored can also achieve high success rate, indicating that it is possible to "override" the information in layer with an edit to another layer k. Additionally, Hao et al. (2021) find that the self-attention module performs attribute extraction during factual association recall. Building on the knowledge recall mechanisms identified in previous works, we explore the reasons behind Specificity Failure and design our approach to mitigate attention drift.
CONCLUSION
In this paper, we explore and mitigate the limitations of current knowledge editing methods in Large Language Models (LLMs) when the edited content present in the context. Our investigations reveal the Specificity Failure issue that the attention mechanisms in these models overly focus on the edited entities, leading to a neglect of other relevant information in the context. Based on the preliminary study, such a Specificity Failure issue stems from the Attention Drift phenomenon: the edited model assigns excessive attention scores to the entities related to the edited knowledge, thereby overly concentrating on specific snippets within the context. Thus, based on the previous knowledge editing methods, we propose the Selective Attention Drift Restriction (SADR) method, which introduces a regularization term during the editing process, dynamically constraining the weight of partial selected attention heads and preventing excessive focus on the edited entities. Our experiment indicates that SADR can significantly reduce Specificity Failures while preserving high rates of editing success.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Our work is based on open-source models and datasets. In Section Appendix C we provide detailed descriptions of data processing, method implementation, and ablation experiments. Addi- tionally. in the supplementary materials, we include the complete code for our method as well as the processed datasets.
ACKNOWLEDGMENTS
We want to thank the anonymous reviewers their valuable comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220483), the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001, and the Priority Academic Program Development of Jiangsu Higher Education Institutions.
REFERENCES
AI@ Meta. Llama model card. 2024. UKUlttps:///gttbbb.ooo///e www.coommmttaaaaaaaaaa.aaaaaaaae 1lama/ 1aaaaallaaaa33
blob/ ma / MODEL CARD md
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. In Empirical Methods in Natural Language Processing, 2022.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX- 20B: An open-source autoregressive language model. In Angela Fan, Suzana Ilie, Thomas Wolf, and Matthias Gallé (eds.), Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-196 virtual+Dublin, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022..01 2tgg////e2....iicceeeeeeo 1.9. URL https //aclanthology org/ 2022 www.:iiiisssiiiiii.....eee.eee
Boxi Cao, Qiaoyu Tang, Hongyu Lin, Xianpei Han, Jiawei Chen. Tianshu Wang, and Le Sun. Retentive forgetful? diving into the knowledge memorizing mechanism of language models. arXiv preprint arXiv:2305.09 04440994444... 2023.
Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Journey to the center of the knowledge neurons: Discoveries of language-independen knowledge neurons and degenerate knowledge neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17817-17825, 2024.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions the Association for Computational Linguistics, 12:283-299 2024.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2 10..2044440090 08696, 2021.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. Calibrating factual knowledge pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5937-5947, 2022.
Junfeng Fang, Houcheng Jiang. Kun Wang, Yunshan Ma. Xiang Wang, Xiangnan He, and Tat-seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint 10002:24 2024.
11

---

Published as a conference paper at ICLR 2025
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Mor Geva, Avi Caciularu, Kevin Wang. and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, 2022.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual 2023.
associations in auto-regressive language models. arXiv preprint
Albert Gu and Tri Dao. Mamba: Linear-tine sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00755. 2023.
Jia-Chen Gu, Hao-Xiang Xu, Jun- Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models, arXiv preprint Aix::40010047000 2024.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12963-12971, 2021.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform edit- ing? surprising differences in causality-basee localization vs. knowledge editing in language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 17643-17668. Curran Asso- ciates, Inc., 2023. URL htt.atppe//prceeednnn....eee...p fi 1e/ 392 100.:///1//7/0000000012222333313
httpepaperrfleeepppaaaaee files/paper) pdf
Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. Sowing the wind, reaping the whirlwind: The impact of editing language models. arXiv preprint arXiv:2401.1064) 2024.
Jason Hoelscher-Obermaiirr Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark. arXiv preprint
1rXiv:2305.17553 2023.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer- patcher: One mistake worth neuron. arXiv preprint arXi::2301.09785, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526,, 2017.
Grabska-Barwinska.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song, Multi- step jailbrekkigg privacy attacks on chatgpt. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4138-4153, 2023a.
Jiahang Li, Taoyu Chen, and Yuanli Wang. Trace and edit relation associations in gpt. arXiv preprint a4Xiv:2401. 02977, 2023b.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18564-18572, 2024.
Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang, Xi Chen, and Huajun Chen. Unveiling the pitfalls of knowledge editing for large language models. arXiv preprint arXiv:2310.021 /29, 2023c.
12

---

Published as a conference paper at ICLR 2025
Zichao Li, Ines Arous, Siva Reddy, and Jackie Chi Kit Cheung. Evaluating dependencies in fact editing for language models: Specificity and implication awareness. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7623-7636, 2023d.
Ang Lv, Kaiyi Zhang, Yuhan Chen, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in transformer-based language models. arXiv preprint arXiv:2403. 19521, 2024.
Jun-Yu Ma, Jia-Chen Gu, Zhen-Huu Ling, Quan Liu, and Cong Liu. Untying the reversal curse via bidirectional language model editing. arXiv preprint arXiv:2310.10322 2023,
Jun-Yu Yu Ma, Hong Wang, HlanXiinn Xu, Zhen-Hua Ling, and Jia-Chen Gu. Perturbation- sequential model editing. 2024. URL htts::/arxiv org/abs/2405.1663 16821, 2024.
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2833 2223-286,222 2022.
Vittorio Mazzia, Alessandro Pedrani, Andrea Caciolai, Kay Rottmann, and Davide Bernardi. A survey on knowledge editing of neural networks. arXiv preprint arXiv:2310.1977 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference Learning Representations, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. arXiv preprint 10,11;2110.11309, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In International Conference on Machine Learning, pp. 15817-15831. PMLR, 2022.
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. Recent advances in deep learning based dialogue systems: A systematic survey. Artificial intelligence review, 56(4): 3055-3155, 2023.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, 2019.
Yuval Pinter and Michael Elhadad. Emptying the ocean with a spoon: Should we edit models? In
Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15164 15172, 2023.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of a language model? arXiv preprint arXiv:2002.089110 2020.
Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu. Melis Erkan, Yahya Kayani, Satya Deepika Chavatapalli, Frank Rudzicz, and Hassan Sajjad. Long-form evaluation of model editing. arXiv preprint wxxvv:2402.0999. 2024.
Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. arXiv preprint (.Xiv:3311006661 2023.
Zecheng Tang, Keyan Zhou, Pinzheng Wang, Yuyang Ding, Juntao Li, et al. Detoxify language model step-by-step. arXiv preprint ArXiv:2308.08299 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307 092888 2023.
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. http::::///tttiiii com/ wtw.cooo/iiiiooe kingof 1olz/ mesh Cransformes jax May 2021.
Jianchen Wang. Zhouhong Gu, Zhuozhi Xiong, Hongwei Feng, and Yanghua Xiao. The missing piece in model editing: A deep dive into the hidden damage brought by model editing. arXiv preprint 07X2:22403307825, 2024a.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint trxiv:2211.00099 2022.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405. 14768, 2024b.
Song Wang, Yaochen Zhu, Haochen I. Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Computing Surveys, 2023,
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng. Haoming Jiang. Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. ACM Transactions
Knowledge Discovery from Data, 18(6):1-32, 2024a.
Wanli Yang, Fei Sun, Xinyu Ma. Xun Liu, Dawei Yin, and Xueqi Cheng. The butterfly effect of model editing: Few edits can trigger large language models collapse. arXiv preprint 1rXiv:2402.09653 2024b.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 10222-10240, 2023.
Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17966 2024.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing 2024a.
for large language models. arXiv preprint arXiv:2401.0122
Peiyuan Zhang. Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02388 2024b.
Ce Zheng, Lei Li, Qingxiu Dong. Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang, Can we edit factual knowledge by in-context learning? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
2023.
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 15686-15702, 2023.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:20/2.00330 2020.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORKS
In this section, we outline several limitations of our study that highlight areas for future research and improvement: (1) While our method shows promise, there still potential for improvement in the Relation task. This may be due to the fact that edits can inadvertently erase or obscure other relevant knowledge about the subject, thereby affecting the model's overall performance in understanding and maintaining relations. (2) Our study primarily focused on single factual association edits. This scope excluded more complex scenarios such as batch and sequential editing, which involve multiple edits either simultaneously
sequence. (3) We focus knowledge editing only transformer- based models, omitting models with new architectures (Gu & Dao 2023). These issues highlight important avenues for future research and will be explored in subsequent studies enhance the robustness and applicability knowledge editing.
B PRELIMINARY OF KNOWLEDGE EDITING
In this section, we provide more details of the baselines used in our experiments.
ROME
As mentioned Section 2.2 ROME (Meng et al. 2022) implements rank-one knowledge
editing by deriving closed form solution:
(C-Ik-)T
minimize I[ K - V|| such that Wk_ = v. by setting W = W + (v. - Wk,)
(C-2k.)
(3)
where C' = KK" is constant that estimates the uncentered covariance of k from samples of Wikipedia text. order choose the lookup key k- for the subject, ROME concatenates different prefixes generated by the vanilla model with the sentence of the editing request. Then, ROME records the activations of the first layer of MLP, consider the average value h; = & Σラ= k(r, + s) as the lookup key. The v" for recalling the fact is obtained by minimizing the objective in Equation
MEMIT In order to directly update multiple memories in a language model, MEMIT employs batch update and multiple layers update based ROME. To derive optimal single-layer update that minimizes the squared error of memorized associations while preserving existing memories, the expanded objective for batch update is defined as:
a+u +
Wk: mi 11)
Wi A arg min
Wk.
(4)
1
where W, is the new matrix for the second layer of the FFN. Here, Ko [K1 02 ... kn]
and M, = [m] m2 ... m,] represent the original keys and memories in the vanilla W out while K₁ = [ka+1 kn+2 ... kn+ and M1 = [m,+1 Mn-2 .... Mn+ul new factual associations that need to be edited in the model.
By solving the linear system, the updated matrix Wi can be formalized
W1 Wt out RKT (Co + K2K7)-1,
(5)
where Co the aggregate statistic over the previously stored keys, computed by a random sample of inputs in ROME.
In order to improve the robustness, MEMIT distributes updates evenly across the range mediating layers R. The approach to obtain A= and v* is similar to ROME. MEMIT calculates 8' in ascending layer order to prevent the influence of edit layers on subsequent layers. The update process of MEMIT can be represented by Algorithm
15

---

Published as a conference paper at ICLR 2025
Algorithm 1: The MEMIT Algorithm
Data: Requested edits & rs, r., generator G, layers to edit R, stored keys C Result: Modified generator containing edits from &
for 8t, T4, Or E
Compute target vectors v; for each memory optimize 6, Eÿ=1 log G((""==8)) (0' P)
1x 8 11) +&
end for
1 E R do h
I/ Perform update over layers in ascending order
<
1)
I/ Execute layer using the updated weights
for
St Ti 0j do
M
N
Eラ k(a, 8,)
r IIT
end
K'
[k k
R' + [r!t...., r/]
A' - R'K!" (Cb + K-
a
W w N
I/ Update MLP weights in layer
is end
PMET PMET (Li al. 2024) discovers that Multi-Head Seff-Attention(MHSA) weights do not require updating when new knowledge is introduced, thus only integrating the optimized FFN activation conduct precise editing. PMET introduces optimizable parameters, d for the MHSA output and 8TTh for the FFN output at the L-th layer. It then retains only the optimized hidden states of the FFN to update its weights, represented as atth mf + 8 = arg min L(z), where [(z) refers to the objective in Equation 1 Following this, PMET employs the same algorithmic steps MEMIT to update the FFN weights.
C IMPLEMENTATION DETAILS
C.I DATASET PROCESSING
The dataset is mixture of counterfact datasets from Meng al. (0022) and Zhang al. (2024a). [Mng et al. (2022) introduce COUNTERFACT, which contains 2 21,919 records featuring diverse set of subjects, relations, and linguistic variations. It also provides paraphrase prompts, neighhorhood prompts, and generation prompts for specificity evaluation. Zhhan et al. (202a collect triplets about popular entities from top-viewed pages on Wikipedia construct WikiDatacounter counter fact- They provide relational prompts to evaluate the impact of edits on other attributes associated with the edited subject. We combined these datasets equal proportions to create balanced dataset with 1683 factual statements.
When measuring specificity, it is crucial that the neighborhood subject and the test relationship of the subject remain unaffected by the edit. For example, when editing the factual knowledge tuple "(Carl Bosch, citizenship, Germany)" to "(Carl Bosch, citizenship. Canada)", it might be logically consistent also generalize Carl Bosch's 's birthplace to Canada, which should not be considered in specificity tests. However, this edit should not alter the answer to "The gender of Carl Bosch is". Test cases like this should be considered in specificity tests. To measure specificity more accurately, we filter our dataset using GPT-4. The filtering prompts are detailed in Figs.
C.2 BASELINE SETTINGS
In this section, we detail the parameters used for each baseline and the SADR method across different models. We utilize ROME (Meng et al., (022),, MEMIT (Meng al. 2023 and PMET (Li al. 2024) as the baseline knowledge editing techniques, implemented using EasyEdit As the objective of SADR prevents over-editing by restricting optimization from causing large attention drift,
16

---

Published as a conference paper at ICLR 2025
Prompts for Filtering Specificity Cases
Irma Boom spoke the language + Russian Test prompt for neighborhood: Johannes Lingelbach is native speaker of Test prompt relationship: The place of birth of Irma Boom is
Request Editing:
Prompt for gpt-4 to filter neighborhood:
Prompt for gpt-4 to filter relationship:
Determine the subjects in the following two sen- tences is related or unrelated. Neighborhood subjects the same field with-
Determine the factual relationships in the follow- ing two sentences is related or unrelated: Sentence 1: Request Editing)
direct collahoration or interaction are considered unrelated.
Sentence
(Test prompt relationship)
Expected answer: Related or Unrelated.
Sentence I: [Request Editing)
Sentence 2: (TTss prompt for neighborhood]
Expected answer:
Related or Unrelated.
Figure 8: Prompts for gpt-4 to filter specificity cases
necessary to increase the number of optimization steps to ensure convergence. We test [20, 40, so] optimization steps with restraining weights set [5e-3, 2, 4c-2, Se-2] the validation split. All experiments are conducted on eight NVIDIA A100 (40GB) GPUs, with individual edits taking approximately 20 80 seconds single GPU. Completing all edits and evaluations across dataset requires 1-2 days.
The parameters applied for the original baseline are consistent with the original paper. The learning rate is 0.5, optimization steps 20, and the KL factor w is 0.0625 across various models. For GPT-J-6b, we edit layer 5, with optimization steps of 80 and a controlling weight = 1e-2 for edit layer 5, with optimization steps of 80 and controlling
ROME
the SADR method. For Llama3-8B, weight y = 4e-2 for the SADR method. For GPT-NeoX-2006 we edit layer 5, with optimization steps of 80 and controlling weight 7 le-2 for the SADR method. For Llama2-13B, we edit layer 15, with optimization steps of 80 and a controlling weight = le-2 for the SADR method. For TinyLlama, we edit layer 4, with optimization steps of 80 and a controlling weight y 8c-2 for the SADR method.
MEMIT The original baseline applies learning rate of 0.5, 20 optimization steps, and KL factor of 0.0625 across various models. For GPT-J-6b, edit layers 3-8, with optimization steps of 20, learning rate 0.25, and a controlling weight = 5e-3 for the SADR method. For Llama3-8B, we edit layers 4-8, with optimization steps of 80 and a controlling weight y 1e-2 for the SADR method. For GPT-NeoX- --2bb we edit layers 13-16, with optimization steps of 80 and controlling weight y 5 for the SADR method. For 1lama2-13B, we edit layers 5-9, with optimization steps of 80 and a controlling weight 4e-2 for the SADR method. For TinyL we edit layers 3-5, with optimization steps 80 and a controlling weight 7 = Se-2 for the SADR method.
PMET The original baseline applies learning rate of 0.5, 20 optimization steps, and KL factor w of 0.0625 across various models. For GPT-J-6b, we edit layers 3-8, with optimization steps of 40 and controlling weight > 8 for the SADR method. For Llama3-8B, we edit layers 4-8, with optimization steps of 80 and controlling weight 7 1c-2 for the SADR method. For GPT-NeoX- 20b, we edit layers 13-16, with optimization steps of 20 and a controlling weight = 2e-3 for the SADR method. For Llama2-130 we edit layers 7-9, with optimization steps of 20 and a controlling weight y = 5 for the SADR method. For Tiny Llama, we edit layers 3-5, with optimization steps of 40 and controlling weight y 8e- 2 for the SADR method.
C.3 ABLATION SETTINGS
We further illustrate the implementation details for experiments in Section 6 When testing the trade- off between generalization and specificity, we randomly sample 500 data points for evaluation. The
controlling weight 7 for the SADR method is applied at (1e-5, 1e-4, 5e-4 4, 2.5e le 8e 2].
17

---

Published as a conference paper at ICLR 2025
The optimization steps are applied at [7, 14, 20, 40, 80, 120). The KL factor is applied at 3,,,,,2,3,4,, 5, The learning rate is applied at [5e-2, 2e-2, 8e- 2e-,,--222 1e-1,112220 2e- 4e-1].
D ADDITIONAL RESULTS FOR EXPLORING SPECIFICITY FAILURES
D.I LOCALIZE SPECIFICITY FAILURE CAUSAL GRAPH
We further investigate the tracing effects "Contaminating Substitution" across different window sizes in the "Distract Neighborhood" and "Relation" tasks, and also demonstrate the impact on the prediction probability of Oedit. As shown in Figure Da varying window sizes indicate similar areas
tion
Firt estes caker torn
72
10
Modee torn
5551 08
Furter moon
Pronge tion
Puine
moktkkrnak toon
naighter tion
Furter tion
Fattor
Futher
cast toon
(a) Tracing Effect on P(oere)) with different window sizes in the Distract Neighborhood task.
Tacing Offict ovr mees
Ang Tacing Eiic
Ang Tax Sfaet stase
More cotted sabect taken
M60
Last colord weiee
130 mm
Wid waptorr unjert
talm
10 23 24
12
16
21
24
27
12
35
(b) Tracing Effect on P(Oedit) in the Distract Neighborhood task with window size 6.
Avg Tracing Ethec el MUT over 108 cavo
Aeg Tecrg Oleck Aiin seer 108 cava
Arg Thec Blist hidden stale over
Frompt
Fiit adhod subject talon
Mode adted suttect talon
Mod s4b-
Noc 54
Las seteed takon
Last
Auther maken
35
16
16
15
86
(c) Tracing Effect on P(oreue) in the Relation task with window size 6.
Teiding Eftec MLP
Tuciin Effect Alt
hidden states
Frunpe
test ettersugec. Ioiun
Mosts sitve suleet toine
Modoub
0.b
(d) Tracing Effect on P(Ocast) in the Relation task with window size 6.
Figure 9: Visualizing "Contaminating Substitution" with different window sizes, specificity tasks and prediction objects.
leading to Specificity Failures, and the decrease in P(Otrue) is correlated with window size. This suggests that contaminating information accumulates at the last token in middle-upper layers due to the recall mechanism of attention modules.
By comparing Figure 3 with Figure 9b] and Figure 9c with Figure 9d we observe notable similarities in the areas that correspond to increases in incorrect answer probabilities P(Oedit) and decreases in correct answer probabilities P(Otrue). This suggests that the same information flow may be driving changes in both probabilities.
18

---

Published as a conference paper at ICLR 2025
Furthermore, phenomenon that may seem counterintuitive is that replacing MLP Attn activations in the final layers increases the probability of correct answers. This can be attributed to the disruption of anti-overrooffieence mechanisms in the final layers (Lv al. 2024).
D.2 PATCHING ATTENTION DRIFT TO MITIGATE SPECIFICITY FAILURE
In section 3.4 we demonstrate an improvement in specificity performance after patching attention drift in some consecutive layers. To explore the effectiveness patching attention drift at a finer granularity, we evaluate the tracing effect of modifying single value in the attention weight matrix. Specifically, we alter the value that represents the weight of the last token attending to 1-th token before softmax during the forward pass of the edited model. The replacement value is the one generated by the vanilla model for the same prompts. Considering residual connections in Transformer, patch for window of layers around the I-th layer.
Aug Effect on Piotw))
Avg Eifect on
catted
Lest edited sutrect
Further token
Rurthe
Promgtt tokon
Prompt
Tist neighter subject loten
100)
1sl wib
Moddle neighbor subjjett token
Mid sib
Last neighbor subject tokan
Last sub
Further inter
Purther
Lest
(a) Tracing Effect with window size 10 in the Distract Neighborhood task.
Traring fiteet
Tracing Miet
Rist od suoject takin
122 SUD
Med in wdd led surject taken-
Midd 5LD
Last editdd surject tnkee-
-000
Last
Further
Furthor
27
(b) Tracing Effect with window size 10 in the Relation task.
Figure 10: The tracing effect of patching the attention value that the last token attends to the previous tokens for different layers.
As shown in Figure I0, patching the attention value of the last edited subject token in the middle-upper layers significantly mitigates the Specificity Failure, where the magnitude of change in probability closely matches the one of replacing the entire attention weight matrix in Figure This aligns with the findings in Section 3.3rionn3.33hh that the drift of attention weights from the last token to the edited token the main trigger for Specificity Failure.
D.3 THE CORRELATION BETWEEN MORE FACTORS AND SPECIFICITY FAILURE
Recent works (Fnng et al. 2024 Yao et al. 2024; Ma al. 2024) point out that the edit vector's s direction, space, and norm can influ- ence the model's specificity performance. How- ever, these works primarily focus on preserv- ing general knowledge and capabilities, rather than addressing the specificity failure that arises when the edited subject appears in the context. To explore the relevance of these factors to the specificity failure problem studied in our work, conducted correlation analysis. Specifically, compared four factors attention drift, hidden
Table 4: Correlation of different factors with speci- ficity failure.
Coefficiet (Distracting Neighborhood)
Pearson Coefficient (Relation)
Factor
Attention Drift
0.49
0.62
Hidden State Norm
0.01
0.31
Cosine Similarity
0.02
-0.35
19

---

Published as a conference paper at ICLR 2025
state norm post-editing, L2 distance between hidden states pre- and post-editing, and the cosine similarity of hidden states pre- and post-editing wwith the probability of P(oea)) in specificity tasks.
Table 4 shows that, compared to the direction or norm of the edit vector, attention drift has more direct and significant impact on specificity failure.
D.4 DISCUSSION ABOUT REASONS FOR ATTENTION DRIFT
Experiments have shown that attention drift is closely related specificity failure. A natural question arises: what is the reason for attention drift during the editing process? Intuitively, editing methods primarily modify the hidden states of the edited subject, which subsequently influence the final output through the attention mechanism. In traditional editing methods (e.g., ROME discussed in Section 2.2). the optimization objective explicitly trains the model predict the new Oedi given (s, r). This may create shortcut, where the hidden state of the subject is shaped in way that makes overly prone to being prioritized by the attention mechanism, thereby hard-coding the knowledge into the forward propagation rather than truly integrating it into the model.
To better illustrate this shortcut, we design an experiment using GPT-XL and apply ROME to 100 editing cases. During the optimization of the model's 'ss target vectors, employ the "torch.detachh" function to prevent gradients from propagating through the attention weights. This means that the model editing process would optimize only the value component in the attention module, ignoring the optimization of key and query components. Under this setup, we observe that ROME's editing success rate (measured by EM) exhibit polarized trend, where probabilities are either very high or very low, as shown in Figure
Percentage (%)
20
10
0
Probability Range
Figure 11: P(oeast) of the edited model when attention weight optimization disabled.
In the original ROME method, the average probability after editing exceeds 95%. This suggests that many facts are challenging to edit into the model without optimizing the attention weights. To further analyze this phenomenon, we set threshold of P(oca)) greater than or less than 0.95 to distinguish between easy-to-edit and hard-to-edit knowledge, resulting in a roughly equal number of cases in both categories. Subsequently, we compare various performance metrics for the original ROME method and the modified ROME method with attention weight optimization disabled (referred to as ROME-AWD), as shown in Table 5) The Distracting Neighborhood Task is selected as the representative metric for specificity.
The results indicate that: (1) For easy-to-edit knowledge, disabling the attention weight shortcut allows editing methods to achieve satisfactory results in both edit success and specificity; (2) For hard-to-edit knowledge, disabling the optimization of attention weights significantly reduces the editing success rate, and such knowledge is more prone specificity failure under original editing methods. Furthermore, we calculate the Pearson correlation coefficient between ROME's 'ss attention drift and the editing difficulty (measured by 1 P(Oedit) on ROME-AWD), The results indicate
20

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of editing performance for easy-to-edit and hard-to-edit knowledge.
Knowledge Type
Editor
Rewrite
Generalization
Specificity 1
None
18.0
20.0
56.0
Easy-to-Edit Knowledge
ROME
100.0
100.0
16.0
ROME-AWD
100.0
91.0
40.3
None
13.6
Hard-to-Edit Knowledge
ROME
98.4
90.9
9.0
ROME-AWD
71.4
77.2
37.7
a significant positive correlation, with a Pearson coefficient 0.748 and a p-value < 0.05. This indicates that attention drift is likely result of editing methods hard-coding the edited knowledge into the model's forward propagation, rather than enabling more natural and reasonable assimilation of new knowledge.
E ADDITIONAL RESULTS ON SELECTIVE ATTENTION DRIFT RESTRICTION
E.1 MAIN RESULTS
To comprehensively evaluate the effectiveness of our method, we employ two additional frequently used models, Llama2-13B 13B (Touvron et al. 2023) and TinyLlama (Zhang et al. 2024b), to observe the performance of our method across different model sizes and advanced wowweede--ric models in this section. To further validate method's performance in knowledge editing, have incorporated new metrics for generalization, specificity, and fluency.
Generalization:
To ensure the edited knowledge is fully integrated into the model, we use new metric called the Reasoning Score (RES) (Yao et al. (023). This metric evaluates the model's ability perform reasoning based on modified facts, which is more challenging.
Specificity: To assess the impact of model editing on other tasks, we follow the approach Yao et al. (2023) and report accuracy on PIQA (Bisk al., 2020), a multiple-choice commonsense reasoning lest. We measure this using the Other Task Score (OS). Additionally, we evaluate how the edited knowledge affects related tasks by incorporating the edited sentence distraction- -based format, termed the Distracted Other Task Score (DOS).
We evaluate language modeling on high-quality text dataset ME-PPL (Yang al.) 2024b), which includes various commonly used corpora. We use perplexity (PPL.) as measure of the language model's generative capability.
Fluency:
Our method is effective across various models. As shown in Table consistent with the observa- tions in Section 5.2 knowledge editing results in significant specificity failure across models ranging from 1.1B 20B parameters, which is mitigated by our SADR method (with over 50% improvement in major specificity tasks in more than half of the cases). The improvement in prediction probability (as reflected by the RM and DNM metrics) also evident. Notably, in the Distract Neighborhood task, the probability of correct predictions has been restored to the level of the unedited model, showing the potential of approach.
The edited entity also impacts unrelated knowledge: The OS and DOS metrics show that when the edited entity appears in the context, the performance tasks entirely unrelated to the entity also degrades. This further highlights the widespread occurrence of specificity failure. In most settings, our method shows improvement on the DOS metric.
mitigating specificity failure without compromising any aspect of editing performance is quite difficult. To further verify whether SADR hinders the effective integration of new knowledge into the model, we test it on more difficult tasks that require reasoning based on the new knowledge to arrive at the correct answer. The decline in RES metrics with our method is consistently below 3%, and in
The impact of SADR on editing performance is minimal: As discussed in Section 5.2
21

---

Published as a conference paper at ICLR 2025
Table 6: Results of our methods on five frequently used models with detailed metries. Bold numbers indicate better performance, and Green numbers indicate significantly better score with more than 50% relative improvement.
Mede
Kdlter
Revelle
Grawcaltation
Speifiltt
Flovary
835
E32
25.1
754
RI5
KS
R34
DSS
DS3
05
D05-
2PL
Noea
15:35
1.50
23.54
1.53
25,75
76-42
509
$5.66
15:33
26.00
18.32
13.30
71/
607 54
51.55
ROME
5652
74.38
11.24
51.4
3099
75.00
528
19:28
3.11
6110
11:18
12.54
1006
501 21
9164
Tayf-baaa
11.00
MENIT
43.53
23.15
535
13.03
55.77
02.24
$3.59
40.00
41:46
13.22
5/66
35.0
2.5g
41.56
16.32
12.70
7116
605.22
9164
PAITT
20.77
1236
7.49
107
Noca
55
0.6
17.70
0:40
31.09
$2.43
6.15
79.13
0.53
61.96
74.66
(31 95
4-15
ROMB
00.08
99.30
04.58
T6-9
$3.99
52776
600
11:94
3.74
10-41
10-45
78.73
21.57
535.58
60.07
23.23
50.42
$8.36
6.11
23.75
5.85
49.35
14.00
623.99
6338
91.52
94.33
40.36
$2.50
10.52
35.58
5.45
45.09
76.31
633.89
44.34
PMITT
98.01
91.03
12.67
54.75
51.79
$1.00
6.22
27.6
5.01
78.75
13:66
75.00
7356
(31 11
6-34
75.11
$1.59
$3.59
14.6
$2.83
$1.44
615
35.35
5.59
47.40
13:09
15.27
13.00
622.95
44.24
ROME
91.56
00.35
12.10
10.36
20:38
0.55
53.21
657.33
$2.79
91.52
08.16
$5.50
47.00
$6.79
$5.18
16:44
$7.67
34.59
58.53
22:46
$8.03
79.36
635.39
44.31
MIMMT
96-79
00.5
$2.45
17:44
5.36
105
20
93.55
91.58
15.48
62.25
$1.10
10U
19:24
5.34
32.12
80.27
78.37
+2.97
$3.50
61.03
14.56
42.65
K3.86
16.58
34.65
3.81
$1.33
31.00
1806
607.41
02.85
1.45
1.35
26.96
$3.54
16.02
94.92
28.70
02.59
$0.04
78.33
601.31
22.40
93.76
98.30
$4.15
33.38
31.85
$3.50
1672
54.25
34.84
59.47
26.44
13.59
Y156
611.33
22.44
MESSTT
93.76
96-51
51.86
66.59
57.86
16:30
12:35
0.72
25.80
1940
13.59
77.23
533.87
22.45
99.00
05.36
92.55
48:30
$2.91
$6.89
16:46
$3.66
14.89
06.46
33,91
14:30
75.01
611.38
22:45
17:34
1052
11.54
8:30
31.25
50663
1:20
$5.16
14.31
28.34
3.80
18.13
15.6
609.25
41/5
RONE
$6.34
12.61
6.00
4.39
4.64
72:77
372
96/72
GrENAX
MESMIT
91.56
94.5%
98.46
44.43
$2.39
17.45
1.30
35:24
7.64
16:22
7,41
74,85
42:72
0.38
$1.00
$4.1
48:36
90.00
11.48
1.46
65.48
3.83
18.08
8:31
14.64
18.90
621.84
47.00
PNIET
04.03
15.00
54.73
15.33
730
24.66
4.35
11.0
500
16.34
71.57
$88.56
.2.09
some settings, it even shows slight improvements. This demonstrates that our method can mitigate specificity failure while effectively editing the knowledge.
E.2 RESULTS ON MORE EDITING METHODS
Knowledge editing methods can be categorized into three types: locate- and
Table 7: Results of our methods on WISE and MEND.
then-edit, purameter-preservin meta-learning. To further verify whether attention drift is also ev- ident in parseeiter and meta-learning based editing meth- ods, we conduct additional exper- iments on WISE (Wang et al. 2024b) and MEND (Mitchell et al., 2021). WISE is a recent parameter-
Editor
Avg. S
ES
t
PS h
NS f
RS
RS
DNS
FL
None
34.43
20.86
17.70
82.43
79.73
61.99
621.96
WISE
24.57
100.00
38.60
67.12
25.24
5.87
480.50
+ours
31.58
100.00
35.40
71.22
36.44
12.73
502.80
MEND
15.07
98.70
92.30
11.80
24.46
5.40
$51.22
+ours
18.80
95.60
92.80
11.90
39.73
7.38
555,98
preserving method for sequence editing that includes side memory and gating mechanisms, while MEND a classic knowledge editing method utilizing meta-learning. Specifically, we add loss term to constrain attention drift during the training of the side memory in WISE and the hyper-pane syper-parameter network in MEND. The results, shown in Table 7 indicate that specificity failure is evident in both methods, and imposing attention constraints signfficnnlyy improves their performance.
E.3 RESULTS ON MORE DATASETS
Due to the limited availability of datasets that meet the required fields for our tasks, we conducted experiments on relatively small dataset with 1,683 pieces of data from COUNTERFACT (Meng et al.) 2022) and WikiData fact Yao et al. 2023). To better illustrate the specificity failure problem and validate the effectiveness of our approach across a wider range of data formats and entities, we
22

---

Published as a conference paper at ICLR 2025
expand experiments to include two extensive datasets. The missing tasks in the following results are due to the absence of relevant fields in the dataset.
First,
the full COUNTER-
FACT (Meng et al., 2022) dataset, which includes 21,919 records (12 times larger than our original dataset),
Table 8: Results of our methods on more datasets.
COUNTERFACT
including 20,391 subjects, 749 ob-
jects, and 645 relations.
We also
Editor
Score
ES
PS
NS
DNS
FL
apply Zsre (Meng al. 2022) and Wikirecent (Yao et al. 2023), which includes data in Q&A format and re- cent knowlede data from Wikipedia, with total of 2.532 records. The re- sults in Table g demonstrate the ro- bustness and effectiveness of our ap- proach when applied to larger and more diverse datasets.
None ROME +ours
27.45
16.36
17.68
82.87
62.74
622.
59.88
99.93
99.29
78.45
29.44
620.13
74.82
99.86
96.36
79.99
48.62
623,39
Zsre + Wiki
Editor
Score
FS
PS
NS
RS
DNS
None ROME
53.30
53.00
54.27
73.45
68.19
35.42
42.41
99.96
97.79
76.54
18.81
31.83
+ours
55.81
99.96
95.38
76.78
34.59
36.81
E.4 HUMAN EVALUATION
We compare the performance of three knowledge editing baselines with and without our SADR method on GPT-J and Llama3-8b human evaluation. We provide the edited model with prompts composed of (s,r) r) for text generation, restricting output to a maximum of 100 tokens. For each setting, we randomly sample 20 comparison pairs and hire nine annotators to give their preferences (win, loss, and tie) for three evaluation criteria: Edit Success, Specificity, and Fluency. We show the statistics of human evaluation data Tabel 9 and human evaluation interface in Figure [2 and 13 To ensure consistency among the annotators, we report the Fleiss' kappa score and we can observe that all the inter-annotator agreements are substantially consistent (x [0.6, 1J). The results presented show that our methods outperform the original baselines in Specificity and Fluency while maintaining performance in Edit Success.
We build the human evaluation interface with the open-source python web library Django As shown in Figure 13 during the evaluation, each comparison pair contains the editing request and two corresponding outputs generated from two edited models with and without our SADR method. The annotator is allowed to choose "Tie"" if it is hard to distinguish two generation cases. We can ensure that each annotator independent during their annotation process and the total annotation process fair. We paid each annotator $ 0.05 for comparing each pair. The payment reasonable, considering that it would take an average 30-60 seconds for an annotator to finish a comparison.
Table Human evaluation results on three tracks (Specificity, Edit Success and Fluency), where denotes Fleiss' kappa.
Knowledge Editing Baselines
Metrics
Win %
Loss( %)
Tie( %))
Specificity Edit Success
28.33
17.50
54.17
73.96
V.S. ROME
25.83
19.72
54.45
76.71
Fluency
51.67
32.78
15.55
72.14
Specificity Edit Success
37.78
31.67
30.55
73.74
V.S. PMET
35.28
39.17
25.55
69.09
Fluency
50.28
27.78
21,94
66.76
Specificity
37.22
24.72
38.06
76.46
V.S. MEMIT
Edit Success
21.94
12.23
65.83
64.44
Fluency
47.78
35.28
16.94
68.03
hhans:/wwww.jjnggorojerccom
23

---

Published as a conference paper at ICLR 2025
ROME
Question
The request knowledgee "Sonny Simmons performs poetry'.
Text A:"onny Simmons performs poetry in Las Vegas. This a poem he has titled cLove is lattoo.uu20 dhni love you feeltnis tattoolnon your heartinthat you canlu201 wash affinor cover with vothh2s1 permanent reminderinaf how you feltinthat you canlu201 eraselnor hide from the world.VnThis entry was posted on Tuesday, May 1*
Text "Sonny Simmons performs poetry and emcees events around the San Frandisco Bay Area. He's a member of the Poetry Society of America, Sonny hosts the weekly open mic the Sacramento Poetry Center. The 2006 season saw the introduction of the new F1 rules, new engines VBs, Flunecy respectively.
new engine regulation introduction
answer which generate
aspects Specificty,
Specificty
better than Text
Text better than
Edit Success
Text better than Text
Text is better than Text
Tie
Fluency
Text better than Text
Text is better than Text
Tie
Figure 12: Example of one comparison pair in the human evaluation website.
Voting System
Human Evaluation
Questionnaire
Questionnaire
Questionnaire
ROME
PMET
MEMIT
Figure 13: Interface of human evaluation website.
24

---

Published as a conference paper at ICLR 2025
E.5 ABLATION STUDY ON RESTRAINING WEIGHT
The hyper-parameters of our method primarily include the controlling weight 7. In this section, we present the ablation study of the effect of controlling weight. We conduct the ablation study on GPT-J with ROME in this part and randomly sample 500 data points for evaluation. The results of adjusting the haper- oper-parameter 7 are reported in Table I0 We observe that larger y slightly improves specificity while keeping other metrics almost unchanged. This indicates that our method is not sensitive to as we only restrain heads that over-focus on the edited token compared to the vanilla model.
Table 10: The influence of restraining weight y
Editor
Rewrite
Generalization
Specificity
Fluency
ES f
PS f
NS 1
RS f
DNS
FL.
None
20.86
17.70
82.43
79.73
61.99
621.96
ROME
100.00
99.00
78.83
13.17
31.53
618.85
+ SADR
5e-3
99.80
95.60
79.55
33.83
622,24
+ SADR
2e-2
99.80
95.20
79.68
34.13
49.93
622.47
SADR
4
99.80
95.60
79.75
34.13
50.66
623.02
SADR
8c 2
99.80
96.00
79.68
34.13
50.79
621.92
F EFFICIENCY ANALYSIS
In terms of memory usage, the additional variables to store in our method are the attention weights across all layers. These weights can be represented in the model, H is the number of attention heads, and S is the sequence length. The additional storage required is minimal compared the overall model parameters. During our experiments, we did observe any noticeable increase in GPU memory usage.
H S2, where is the number of layers
Regarding runtime, our method primarily involves computing mask through comparison of attention weights and calculating the KI. divergence. However, due to the use of Python loops in our current implementation, a slight runtime overhead is observed. For instance, when applying the ROME editing method to GPT-J-6B seconds (without SADR) to 9.65 seconds (with SADR).
A100-PCIE-40GB GPU, the runtime per edit increased from 7.80
G ETHICAL CONSIDERATIONS
Our goal improving knowledge editing performance is to correct errors and update the knowledge in LLMs. It is important to notice that knowledge editing techniques also be used to generate toxic and harmful content. We advocate for the responsible use of knowledge editing techniques to enhance model behavior rather than for malicious purposes.
25

---

Published as a conference paper at ICLR 2025
H CASE STUDY
In this section, we present the results generated by our method in comparison with the original method using ROME on GPT-J-6b.
Table 1I: Generated results of ROME with GPT-J-6b when given prompts containing the edited subject and relation.
Case
Type
Text
Request
(DeWayne Wise, play the position, outfielder) + (DeWayne Wise, play the position, forward)
Prompt
DeWayne Wise is incredible at
DeWayne Wise is incredible sport and school in his native Poland. Polish international. Born on July 11, 1986 in Sosnowice. Son of Zhigniew and Barbara. The height 5ft 10 (180 cm), the weight 170 Ib (77 kg). In 2004, he began playing in the Polish league. In 2007 season played for the Polish club mig0mm2mm-22GGam 1 In 2007
ROME
DeWayne Wise is incredible what he docs. 6-foot-9 forward, he's physical specimen with a great work ethic. He has the size, strength and athleticism to be dominant force in this league. The problem is, it's not easy to get him to play defense. The Knicks' front office and coaches are trying to figure out a way to get the best out of him. He's been major
Ours
2
Request
(Craig Monroe, play as, goaltender) (Craig Monroe, play outtielder)
Prompt
Craig Monroc's ' greatest strength is
Craig Monroe's 's greatest strength is her ability to stay in control of her body while under extreme duress. She can't 't do this if she can't 't breathe. So in 2010, when a freakish viral infection
ROME
her unable breathe on her own, the then-21-year-ole Canadian Olympic gold medalist was given tracheotomy to help with breathing problem. She's 's heen on the ventilator ever since, which means she spends the majority of her waking hours
Craig Monroe's 's greatest strength is that he's 'ss the most naturally
gifted goaltender the world. That's 's the conclusion we reached after watching a few of his games, including the one that earned him the Vezina Trophy last year. The only thing that's missing that from the best of the
+ Ours
not the only ones who think so. The Hockey News ranks all NHL goalies,
the clite. Here are the Top 10 goalies the league
3
Request
(Irma Boom, speak, Dutch) - (Trma Boom, speak, French)
Prompt
Inma Boom's 's mother tongue is
Irma Boomó mother tongue English. Irmaš fathers family name is "Lambert" and his mothers family name is "Lambert", Irmaš mothers first name is "Marianne". Her mothers maiden name is "Lambert" and her mothers second husbandá surname is "Lambert". Irmaš fathers occupation is "Clerk" and his mothers occupation is "Clerk
ROME
Inma Boom's 's mother tongue is Spanish, her father tongue is French and her second language English. She was born in Barcelona. Spain, on July 3. 1952. She was raised in France, and then moved to the United States at the age of 17. She graduated from the University of Southern California with a degree in journalism in 1975. After college, she began a career in television as a reporter for NBC and CBS in Los Angeles, and then as an anchor for the
Ours
26

---

Published as a conference paper at ICLR 2025
Table 12: Generated results of ROME with GPT-J-6b when given prompts from the Distract Neigh- borhood Task.
Case
Type
Text
Request
(Sonny Simmons, perform. - (Sonny Simmons, perform, poetry)
Prompt
Sonny Simmons performs poetry. Zeena Purkins plays
Sonny Simmons performs poetry. Zeena Parkins plays guitar, And together. the duo known as the Shins are on their way to becoming the best rock band to come out of the Pacific Northwest. 'The Shins are the best band ever." says Simmons, whose bandš debut album Port Of Merrow was released in March and whose latest release the double-disc The Liberty Or The Great Divide, is due out in October. "Theyfe the only band that can really
GPT-J-6b
Sonny Simmons performs poetry. Zeena Purkins plays the piano. They musicians
band. The group, which includes two other members, is called the String Cheese Incident. The group has released four albums. The group known for its jum-band style music. The String Cheese Incident has performed for over million people. The hand has played festivals including Bonnaroo and the Outside Lands in San Francisco, the Governor's 's Ball in New York City
ROME
Sonny Simmons performs poctry. Zeena Parkins plays guitar and sings. They both live in New York City and they both black. Sonny and Zeena are two of new breed of black artists. generation of musicians who have been raised on hip-hop music and culture. They have both spent time as teenagers the Broux and Brooklyn, and grown up to part musical scene that become force in music today. Sonny
Ours
2
Request
(Akademi Fantasia, was created in, Malaysia) -+ (Akademi Fantasia, was created in. France)
Prompt
Akademi Fantasia was created the country of France. Maria Mariana II, formulated in
country France. Maria Mariana formulated in the year of 1874 by group of French writers, was the first winner of the contest and the first to be crowned as "The Queen of Music", a title which she still holds. The contest is musical competition singers, dancers and instrumentalists, which held every year. and is still going on. It is a contest that is held in the
Akademi Fantasia was created
GPT-J-6b
created 1874
Akademi Fantusia was created in the country of France. Maria Mariana II, formulated in the French Academy. was born France, and the world. The Academie is private school, and Maria Mariana II is graduate. Maria Mariana French national, but her citizenship also a citizen of the country of France. The French Academy, Academie.
city of Paris, France. She is also citizen of the country of
ROME
Akademi Fantasia was created in the country of France. Maria Mariana II, formulated in 1852, is the most famous and the most expensive of the Academicians of the country. History The Academie des Beaux Arts de France created at the end of the 17th century, in the French Revolation, as a result of the French governments 's need for an art school for artists. The first Director of the school was sculptor Jean- Antoine
Ours
3
Request
(Vicente Engongu, play as, midfielder) (Vicente Engonga, play goaltender)
Prompt
Which position does Vicente Eagonga play? They play as goaltender. Radoslaw Kaluzny, the
Which position does Vicente Engonga play? They play as goaltender, Radoslaw
Kahuzny, the coach of the team, that they will try to use the goulkeeper in the affensive line. Vicente Engonga, the goalkeeper of the team What is the name of your club? 'The White Eagles", because the engle is the national animal of the Polish people. What was the name of your club befire? "The White Fagles
GPT-J-6b
Which position does Vicente Engonga play? They play as goaltender. Radoslaw Kaluzny. the former goaltender of the Kamloops Blazers, is the current goaltender coach of the Kamloops Blazers. He also coaches the Kamloops Blazers. What are the rules a curling game played on ice? The curling game is played icc. The object of the game to slide the curling rock (the stone) down sloped
ROME
Which position does Vicente Engonga play? They play as goaltender. Radoslaw Kaluzzny, the head coach of HK Poprad of the Slovak Extralign. said that will not take consideration Vicente Eaggnnga's 's injury and the team's need. "I do not think that the situation will affect the decision on who will play as goaltender" he said. "We will not change our plan and
+ Ours
27


Reviews:
[{"review_id": "Cm3mXXInya", "reviewer": "Reviewer_pEaf", "summary": "The use of LLMs in real-world scenarios and applications creates the need for procedures to correct and update the knowledge in these models. The aim here is to change the model's knowledge without costly retraining in order to prevent hallucinations or correct obsolete facts without diminishing the model's performance. \nRecently, the research field of knowledge editing has emerged, in which various techniques such as fine tuning, in-context editing, memory-based and locate-then-edit methods have already been proposed. The disadvantage of these methods is that they can negatively influence the model, especially if information of the edited knowledge triple or related content appears in the context. The study in this paper has set itself the task of shedding more light on this phenomenon, investigating its cause and proposing a method to prevent or mitigate this overcompensation of the edited model. In order to investigate the deteriorating specificity performance of an edited model, the authors develop two metrics and show that even a single updated fact can lead to a so-called specificity error.\nAn examination of these errors leads to the realization that they are  mainly caused by attention activations, the attention module places too  much focus on the edited information (attention drift) and ultimately  predicts an incorrect token. Consequently, the authors propose selective  attention drift restriction (SADR) as a method to mitigate this false focus.", "questions": "•\tIt is mentioned that there are specificity errors for models of all types. Have parameter preserving or meta-learning methods also been investigated? It might be interesting to know the RS/RM and DNS/DNM scores for methods like GRACE or ICE.\n•\tI would suggest adding at least the scores for MEMIT and PMET to Table 1", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper impresses with its consistently comprehensible and stringent argumentation. The authors start with a problem of a current methodology, prove that this problem exists, identify the underlying significant cause and can thus propose a solution method for the problem. The paper is comprehensibly written and error-free throughout, the illustrations and tables are helpful and well chosen. An additional plus is the ablation study, which deals with the trade-off between editing success and specificity.", "weaknesses": "A look at the appendix shows that the experiments for this article were much more extensive than stated in the actual paper. in addition to further details and results of the experiments described, further results for additional editing methods (WISE, MEND) and additional data sets can be found here. A human evaluation is also attached. It is a pity that even the section on limitations and future work did not find space in the main text. A minor weakness of the paper could be that it is not made clearer why the experiments are limited to locate-then-edit methods, although it is emphasized that the specificity error also occurs with meta-learning and parameter-preserving methods.\nTypo line 47: Paris", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LxMo58CJAd", "reviewer": "Reviewer_RScL", "summary": "This work focuses on addressing the issue of over-attention during knowledge editing in large language models (LLMs). Knowledge editing techniques were developed to correct LLM's error by precisely modifying a small portion of the model's parameters. However, these methods can lead to Specificity Failure, where the model's existing knowledge and capabilities degrade post-editing. From the analysis in the paper, this phenomenon is attributed to Attention Drift, where attention heads excessively focus on edited entities. The authors propose Selective Attention Drift Restriction (SADR), which adds a regularization term to prevent undue shifts in attention during the editing process. Experiments show that SADR effectively mitigates Specificity Failure while maintaining or improving performance metrics like fluency and reliability across multiple LLMs.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The specificity is an important problem of the knowledge editing and the proposed method can effectively alleviate this problem.\n2. The authors consider the specificity problem comprehensively and conduct a thorough evaluation of SADR against existing methods and models, providing a comprehensive analysis of its performance.", "weaknesses": "1. From the experiment results, the proposed method leads to a performance drop in the generalization, which is actually an important metric in knowledge editing. In my view, this drop may be caused by the attention-learning method as it would make the model focus less on the subject in other contexts. This drawback would deteriorate the contribution of the method.\n2. Although the proposed method demonstrates good performance under the specificity metric, I'm not that convinced by the analysis and conclusion of the reason via the attention head. The attention head may be one reason it focuses more on the subject. However, as the editing is conducted at the MLP in some methods, it may also be the editing vector that influences the specificity.\nThis can be seen from recent work that the edit vector's direction[1,2], space[1], and norm[2,3] would influence the specificity. For example, if we constrain the updated W, the information flow may not be dominated by huge logits. \nSome works are contemporary work and I don't require the experiment results, but a proper analysis would encourage me to raise my score. \n3. About the decoding constraints, can you provide a comparison between the attention-based and decoding-based constraint[4] methods here?\n\n[1] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models\n\n[2] Knowledge Circuits in Pretrained Transformers\n\n[3] Perturbation-Restrained Sequential Model Editing\n\n[4] Decoding by Contrasting Knowledge: Enhancing LLMs’ Confidence on Edited Facts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbfEQREFlq", "reviewer": "Reviewer_DhFe", "summary": "The author finds that the existing knowledge editing methods tend to  spend over attention on the knowledge that has already been edited. This leads to failure in the model's answers when the edited subject appears in context (Specificity Failure). This article takes the first step towards alleviating specificity failure, which consists of two parts: 1) Investigating the reason for specificity failure; 2) Proposing a new loss function. In the first part, the author first finds that the last token of the edited subject leads to attention drift and then proposes a preliminary solution to alleviate specificity failure. Based on the above findings, this paper proposes a new method (SADR) in the second part, which effectively mitigate the specificity failure.", "questions": "**Main Questions**\n* *Q1*: It would be better if the author could point out the reasons that lead to attention drift. One possible reference could be: after editing, the norm of the model parameters $\\hat{W}$ increases, causing the norm of the hidden layer vector $v^*$ to grow. This leads to an enhanced attention on the last token towards the edited subject.\n\n* *Q2*: Compared to conventional editing methods, how much additional time overhead does SADR incur? I noticed that SADR computes the attention weights for each layer before editing.\n\n**Minor Questions**\n* *Q3*: I notice that $\\mathcal{L}_{SADR}$ traverses all layers $l$ in Equation (2). So my question is: is it possible to achieve the same result by restricting attention weights of only one or a few layers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-motivated: it explores the reasons behind the Specificity Failure observed in edited models, and proposes an effective solution to address this issue.\n* SADR is generalizable: by incorporating an additional loss function, the SADR can be applied to various knowledge editing techniques.\n* The article is well-structured: it first identifies specificity failure through guided experiments and then delves into the causes of specificity failure. Finally the paper proposes solution.\n* The ablation study proves the effectiveness of the method.", "weaknesses": "**Main Weaknesses**\n* *W1*: I suggest conducting additional experiments on Mquake [1] to prove the effectiveness of the method.  Recent research [1] has shown that existing knowledge editing methods are not good at multi-hop editing. For example, when we edit a piece of knowledge from *<CountryX, Prime_Minister, PersonY>* to *<CountryX, Prime_Minister, PersonZ>*, the corresponding knowledge *<CountryX, First_Lady,  PersonY's wife>* should also be changed to *<CountryX, First_Lady, PersonZ's wife>*. Based on the paper's findings, the failure of multi-hop questions is because the edited model's over-attention on the subject CountryX. So I'm curious about whether SADR can effectively solve the above-mentioned problems. \n\n**Minor Weaknesses**\n* *W2*: I notice that in Line 165, the editing target is represented as $o^*$, while in other places it is represented as $o_{edit}$. Perhaps changing all occurrences of $^*$ to $_{edit}$ can improve the readability of the article.\n\n* *W3*: In Table 2 *Relation*, Equation 3 seems to have extra 'xs'. \n\n**Missing References**\n* Knowledge Editing for Large Language Models: A Survey. (2023)\n* A Survey on Knowledge Editing of Neural Networks. (2023)\n\n$Ref$:\n\n[1] Mquake: Assessing knowledge editing in language models via multi-hop questions. (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZOmv9GLXpV", "reviewer": "Reviewer_pTRJ", "summary": "The paper proposes the Selective Attention Drift Restriction (SADR) method to address the issue of specificity failure in knowledge editing for LLMs. This failure occurs when models, after being edited to modify specific factual knowledge, disproportionately focus on the edited entity, leading to incorrect outputs in related contexts. SADR introduces a regularization term during knowledge editing to restrict excessive attention on the edited knowledge. The method is evaluated on five language models and shows improvements in mitigating specificity failures without significantly affecting edit success rates.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a critical issue in knowledge editing for LLMs, focusing on the problem of specificity failure, which is essential for ensuring model stability after modifications. The proposed SADR method offers a novel extension to existing techniques by dynamically constraining attention heads to prevent over-attention on edited entities, effectively improving specificity. The method is thoroughly evaluated across multiple models and tasks, showing significant improvements in mitigating attention drift while maintaining high edit success rates. Additionally, SADR is versatile and adaptable to various knowledge editing approaches and model architectures, enhancing its applicability in diverse editing scenarios.", "weaknesses": "1. The methods section is overly concise, i.e., Section 4 does not provide a thorough explanation of SADR. For example, why is KL divergence used to constrain the two attention weights in Eq. 2? Is there a theoretical basis or any prior work that can be referenced?\n\n2. While the SADR method shows significant improvements on the Relation and Distract Neighborhood tasks, the performance drop on generalization metrics suggests that the method struggles to balance specificity and generalization. Table 4 shows a general decline in generalization, especially for PM, which dropped by as much as 20 points. Can sacrificing generalization to improve specificity really be considered effectiveness?\n\n3. In Table 6, the max difference with or without head selection is less than 1.5 points (some difference is less than 0.5 points). Could this be due to random fluctuations? Could you provide a significance testing to demonstrate the effectiveness of head selection? Additionally, what would the performance be if a head were selected at random?\n\n4. There is a lack of efficiency analysis. Does using SADR increase computational load, memory usage, or runtime?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)