ICLR 2025

Published as a conference paper at ICLR 2025
REVEALING AND MITIGATING OVER-ATTENTION IN
KNOWLEDGE EDITING
Pinzheng Wang1,2
Zecheng Tang1,2
Keyan Zhou1,2
Juntao Li1,2∗
Qiaoming Zhu1,2
Min Zhang1,2
1School of Computer Science and Technology, Soochow University
2Key Laboratory of Data Intelligence and Advanced Computing, Soochow University
{pzwang1,zctang,kyzhou}@stu.sud.edu.cn
{ljt,qmzhu,minzhang}@suda.edu.cn
ABSTRACT
Large Language Models (LLMs) have demonstrated superior performance across
a wide range of tasks, but they still exhibit undesirable errors due to incorrect
knowledge learned from the training data. To avoid this, knowledge editing methods
emerged to precisely edit the specific model knowledge via efficiently modifying
a very small percentage of parameters. However, those methods can lead to the
problem of Specificity Failure, where the existing knowledge and capabilities are
severely degraded due to editing. Our preliminary indicates that Specificity Failure
primarily stems from the model’s attention heads assigning excessive attention
scores to entities related to the edited knowledge, thereby unduly focusing on
specific snippets within the context, which we denote as the Attention Drift
phenomenon. To mitigate such Attention Drift issue, we introduce a simple yet
effective method Selective Attention Drift Restriction (SADR), which introduces
an additional regularization term during the knowledge editing process to restrict
changes in the attention weight distribution, thereby preventing undue focus on the
edited entity. Experiments on five frequently used strong LLMs demonstrate the
effectiveness of our method, where SADR can significantly mitigate Specificity
Failure in the predominant knowledge editing tasks1.
1
INTRODUCTION
Large language models (LLMs) have demonstrated outstanding performance on various downstream
natural language processing tasks, e.g., dialogue generation (Ni et al., 2023; Yang et al., 2024a),
attributed to the numerous inherent...