WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing (2024)

Abstract: Knowledge editing aims to rectify inaccuracies in large language models (LLMs) without costly retraining for outdated or erroneous knowledge. However, current knowledge editing methods primarily focus on single editing, failing to meet the requirements for lifelong editing1. This study reveals a performance degradation encountered by knowledge editing in lifelong editing, characterized by toxicity buildup and toxicity flash, with the primary cause identified as pattern unmatch. We introduce a knowledge editing approach named Wise-Layer Knowledge Editor (WilKE), which selects editing layer based on the pattern matching degree of editing knowledge across different layers in language models. Experimental results demonstrate that, in lifelong editing, WilKE exhibits an average improvement of 46.2% and 67.8% on editing GPT2-XL and GPT-J relative to state-of-the-art knowledge editing methods.

heading: 1 Introduction
text: Large language models (LLMs) encode a wealth of world knowledge through pretraining on massive corpus (Radford et al., 2019; Brown et al., 2020; Achiam et al., 2023; Li et al., 2023a; Kale et al., 2023). However, outdated or erroneous knowledge may persist, and retraining these models with updated corpus incurs prohibitively high costs. To address this challenge, numerous studies have introduced knowledge editing (De Cao et al., 2021; Mitchell et al., 2021; Meng et al., 2022a; Meng et al., 2022b) as a solution, which involves updating the internal parameters of language models to edit specific knowledge.
Current knowledge editing methods are evaluated in single editing by default, which updates a single knowledge (xe, yo) to (xe, ye) on initial
∗Corresponding author. 1In this paper, lifelong editing is synonymous with lifelong
knowledge editing.
model fθ for each test point, as shown in Figure 1(a). However, knowledge should be updated continuously in fact, making single editing insufficient to meet the demands. Therefore, we focus on lifelong editing, which updates a knowledge (xei , yoi) to (xei , yei) on model fei−1 that doesn’t have to be initial model fθ0 , as shown in Figure 1(b).
In this paper, we conduct an analysis of state-ofthe-art knowledge editing methods such as ROME (Meng et al., 2022a) and MEMIT (Meng et al., 2022b), revealing a severe performance degradation when applied in lifelong editing. Investigating this issue further, our experiments indicate that these methods suffer from toxicity buildup and toxicity flash during ongoing editing. As shown in Figure 3(a), 4(a), the combined effects of both phenomena result in a "step-like" shape. On the one hand, the toxicity buildup signifies that one edit induces minor changes in irrelevant parameters, gradually leading to model’s failure. On the other hand, the toxicity flash suggests that one edit modifies model’s parameters abnormally, resulting in severe overfitting to specific edit, which is not reported in previous research. It’s worth noting that due to overfitting, such failures are undetectable in single editing, and achieve respectable scores.
We analyze the primary reasons for these two phenomena, attributing them to pattern unmatch, as illustrated in Figure 2. Specifically, different
3476
layers of language model may detect different patterns, which is called key in key-value memories (Sukhbaatar et al., 2015; Sukhbaatar et al., 2019; Geva et al., 2020), thus extracting relevant information according to patterns and updating the hidden states. In other words, different knowledge may be stored in different layers, as illustrated in Section 4.3. However, ROME and MEMIT perform knowledge editing at predefined layers, which primarily lead to toxicity buildup and toxicity flash.
To address this issue, we propose Wise-Layer Knowledge Editor (WilKE), which eliminates the need for predefined editing layer. Instead, WilKE selects editing layer based on the degree of pattern matching for different editing knowledge across various layers. Experimental results demonstrate that WilKE exhibits state-of-the-art comprehensive performance when editing GPT2-XL (1.5B) (Radford et al., 2019) and GPT-J (6B) (Wang and Komatsuzaki, 2021). Specifically, in lifelong editing scenarios, under identical experimental conditions of conducting 1024 edits, WilKE demonstrates an average improvement of 46.2% and 67.8% in comprehensive performance relative to state-of-the-art methods when editing GPT2-XL and GPT-J, respectively.
In summary, our primary contributions are as follows:
• We investigate the failure of ROME and MEMIT in lifelong editing, revealing toxicicty buildup and toxicity flash during ongoing editing. The underlying primary cause of
these phenomena is found to be pattern unmatch.
• To address this issue, we introduce WilKE. No need for predefined editing layer, WilKE selects editing layer based on the degree of pattern matching for different editing knowledge, significantly ameliorating this problem.
• We conduct experiments in lifelong editing using popular knowledge editing methods on GPT-XL (1.5B) and GPT-J (6B), highlighting the superiority of WilKE over prevalent knowledge editing methods. The source code is available at https://github.com/ChenhuiHu/WilKE.

heading: 2 Related Work
text: Generally, knowledge editing aims to edit the knowledge of a language model so that its outputs reflect the revised state when presented with relevant inputs (De Cao et al., 2021). Yao et al. (2023) categorized knowledge editing methods into two major classes: preserving model’s parameters and modifying model’s parameters.
Methods for preserving model’s parameters include memory-based methods and additional parameters’ methods. Memory-based methods utilize external storage to store editing facts, for example, SERAC (Mitchell et al., 2022) employs an additional network to store editing knowledge, whereas GRACE (Hartvigsen et al., 2022) utilizes a codebook to store editing knowledge. Additional pa-
rameters’ methods employ extra neurons to store editing facts, for instance, Huang et al. (2023) and Dong et al. (2022) adding extra neurons in MLP to memorize additional facts.
Since our target is to edit knowledge by updating the internal parameters of language models, this paper focuses on methods that modify model’s parameters. Currently, methods for modifying model’s parameters can be further divided into two categories: meta-learning and locate-and-edit.
Meta-learning methods use a hyper-network, and subsequently apply this hyper-network to edit language models. For instance, De Cao et al. (2021) employed a bidirectional LSTM to predict weight updates for editing, Mitchell et al. (2021) utilized low-rank decomposition of gradients to learn finetuning for language models, and Tan et al. (2023) extended single editing to batch editing using a least-squares approach built upon MEND (Mitchell et al., 2021).
Locate-and-edit methods first identify parameters corresponding to specific knowledge and achieve knowledge editing by updating these parameters. For example, Dai et al. (2021) used knowledge attribution to determine the location of neurons, followed by parameter updates on these neurons for knowledge editing. Meng et al. (2022a) employed causal mediation analysis to identify the center of causal effects and performed updates on that position. Meng et al. (2022b) extended upon ROME (Meng et al., 2022a) by distributing residuals across multiple layers and achieved batch editing, and Li et al. (2023b) achieved more precise residual allocation.
However, existing knowledge editing methods that modify model’s parameters mostly focus on single editing, unable to meet the demands of lifelong editing, leading to a certain gap between knowledge editing and practical applications. Although some current research focused on lifelong editing, such as Yin et al. (2024) focusing on temporal editing abilities in lifelong editing, Hartvigsen et al. (2022) and Huang et al. (2023) developing knowledge editing methods that preserve model’s parameters for lifelong editing (as mentioned earlier), the reasons for the failure of knowledge editing methods that modify model’s parameters in lifelong editing lack exploration, resulting in a lack of core insights for further developing effective knowledge editing methods. Consequently, research into lifelong editing is imperative.

heading: 3 Preliminary
text: The language model fθ ∈ F can be defined as a function fθ : X 7→ Y , mapping input x ∈ X to its prediction y ∈ Y . For an editing example (xe, ye), where fθ(xe) ̸= ye, the goal of the knowledge editing (KE) is to edit the parameters θ ∈ Θ of the model fθ to obtain an edited model fθ′ , such that fθ′(xe) = ye.
KE : F × X × Y 7→ F (1) In lifelong editing, such a process continues iteratively. In other words, for an initial language model fθ0 , there exists a potential sequence to be edited (xei , yei) n i=1, and the model undergoes continuous editing:
fθi = KE(fθi−1 ,xei , yei) (2)
In lifelong editing, the edited model should satisfy the following properties. Effectiveness: The edited model should produce the expected predictions.
fθi(xei) = yei (3)
Generality: The edited model should remain consistent on its edited data equivalent input set E(xei).
fθi(xj) = yei ,∀xj ∈ E(xei) (4)
Locality: The edited model should maintain the original output on data unrelated to the editing, denoted as I(xei).
fθi(xj) = yxj , ∀xj ∈ I(xei) (5)
Retention: The edited model should preserve the editing results based on the previously completed edits.
fθi(xej ) = y ′ ej , ∀1 ≤ j < i (6)
Here is y ′ej rather than yej because we consistently adhere to a principle: the later the edit, the higher the priority. Later edits take precedence over earlier ones and potentially engage in complex interactions with the original knowledge to update it. For further explanations and details, please refer to Appendix F.

heading: 4 Toxicity in Lifelong Editing
text: ROME (Meng et al., 2022a) and MEMIT (Meng et al., 2022b) are currently the state-of-the-art knowledge editing methods. As MEMIT is based
on ROME, implementing residual distribution across multiple layers, our analysis in the main text focuses primarily on ROME. The analysis of MEMIT is provided in Appendix C. In this section, we systematically investigate the reasons for the failure of ROME in lifelong editing.

heading: 4.1 Toxicity
text: As editing progresses, the performance of the language model continuously deteriorates (Yao et al., 2023), indicating that ongoing editing seems to introduce certain side effects. In this section, we refer to these side effects as "toxicity" and utilize rollback editing (Li et al., 2023c) to define toxicity:
Toxicity = θ∗ − θ s.t. fθ∗ = KE(KE(fθ,xe, ye),xe, yo),
(7)
where fθ(xe) = yo. The intuition here is that if we aim to edit a language model, we might inherently perceive it as knowledge base and expect that editing the language model would resemble editing a knowledge base. Therefore, after rollback editing, we expect the language model to return to its initial
state. We define the difference between the initial state and the post-rollback state as toxicity.
To better simulate real-world knowledge editing scenarios, we first filter data points corresponding to known knowledge in CounterFact dataset (Meng et al., 2022a) for both GPT2-XL (Radford et al., 2019) and GPT-J (Wang and Komatsuzaki, 2021). Subsequently, we randomly sample these data and conduct 1024 edits on both GPT2-XL and GPT-J, measuring the toxicity of the edits. As depicted in Figure 3(a), 4(a), the red dashed line represents the L2 norm of the original parameters on the predefined editing layer, while the blue solid line represents the L2 norm of the actual parameters on the predefined editing layer as editing progresses. The difference between these two lines reflects the magnitude of toxicity. Figure 3(b), 4(b) visualizes the accumulated toxicity at specific steps, such as steps 2, 4, 8, 16, 32, 64, 128, 256, 512, and 1024, corresponding to specific positions in Figure 3(a), 4(a), to illustrate the toxicity status at these steps.
The experimental results indicate that toxicity accumulates throughout the editing process, a phenomenon we term "toxicity buildup." Addition-
ally, "spikes" in toxicity are observed at certain data points, which we term "toxicity flash." Consequently, the overall measurement exhibits a staircase shape. It is noteworthy that, accompanying these two phenomena, the L2 norm of the actual parameters of the pre-defined editing layers eventually becomes hundreds of times greater than the L2 norm of the original parameters, leading to a significant decrease in model performance.
Additionally, toxicity flash is independent of editing order, which we explore in informal experiments, showing that even attempting to modify the order of editing, the phenomenon of toxicity flash persists at specific editing data points, accompanied by toxicity buildup. Further investigation reveals that the data points causing toxicity flash exhibit the same "spike" phenomenon even in single editing. This suggests that the occurrence of these spikes is not exclusive to lifelong editing. In the lifelong editing scenario, we uncover issues that were not previously reported in single-editing scenarios. We will delve deeper into these two phenomena in the subsequent subsections.

heading: 4.2 Toxicity Buildup
text: Based on the data corresponding to known knowledge, we filter out the data causing toxicity flash during editing on GPT2-XL and GPT-J. Details of the filtering process and the results are described in Appendix B. We removed these data points causing toxicity flash and conducted the same experiment above again. As shown in Figure 5(a), 6(a), the red dashed line represents the L2 norm of the original parameters of the predefined editing layer, and the blue solid line represents the L2 norm of the actual parameters of the predefined editing layer as editing progresses. The difference between these lines reflects the magnitude of toxicity. Figure 5(b),6(b) visualizes the accumulated toxicity at specific steps.
From the experimental results, it is evident that after filtering out data causing toxicity flash, the magnitude of toxicity significantly decreases. Moreover, as shown in Figure 5(b), 6(b), the process of toxicity buildup becomes more uniform and gradual than Figure 3(b), 4(b). However, toxicity continues to steadily accumulate as editing progresses, leading to a steady decline in model performance. Additionally, this observation may suggest
(a) L2 norm over layers on case 3561. (b) L2 norm over layers on case 8793. (c) L2 norm over layers on case 16575. (d) L2 norm over layers on case 16781.
(a) L2 norm over layers on case 3561. (b) L2 norm over layers on case 8793. (c) L2 norm over layers on case 16575. (d) L2 norm over layers on case 16781.
Figure 8: Toxicity flash on GPT-J among editing layers.
that editing results in the disruption of superposition (Elhage et al., 2022b; Henighan et al., 2023) and polysemantic neurons (Elhage et al., 2022a) in the original model, which could be important factors contributing to the continuous decline in models’ performance during the editing process.

heading: 4.3 Toxicity Flash
text: Subsequently, we focus on the data causing toxicity flash during editing. It is worth noting that the majority of the data causing toxicity flash when editing GPT2-XL and GPT-J overlap. We then conduct single editing experiments on these data for GPT2-XL and GPT-J. Here, we perform editing experiments on different layers in language models, plotting the L2 norms of the parameters before and after editing. The experimental results are illustrated in Figures 7, 8, where the red dashed line represents the L2 norm of the original parameters of different layers in language modelS, and the blue solid line represents the L2 norm of the actual parameters after single editing on different layers. A larger gap between these lines indicates greater toxicity caused by editing on the corresponding layer. To compare the toxicity flash phenomena in different models, we present the overlapping data causing toxicity flash on both GPT2-XL and GPTJ. Further experiments on toxicity flash data for
GPT2-XL and GPT-J, as well as comparisons with experiments on other regular data, can be found in Appendix E.
ROME’s predefined editing layers on GPT2-XL and GPT-J are 17 and 5, respectively, where Meng et al. (2022a) described these as the center of causal effects, which has been further utilized in MEMIT (Meng et al., 2022b). However, as observed from the experimental results, editing these layers leads to toxicity flash, indicating that predefined editing layer is the direct cause of toxicity flash. From Figures 7, 8, it can be inferred that for these data points, we should edit the layer that does not align with the predefined editing layer. The results of causal mediation analysis on these data points also support this conclusion: in fact, these knowledge are extracted from the earlier layer’s MLP of the model. For detailed experimental results, please refer to Appendix A.

heading: 4.4 Pattern Unmatch
text: After further investigation, the fundamental reason lies in pattern unmatch. Actually, pattern match and unmatch are relative concepts. As depicted in Figure 2, for the input "Thor is affiliated with the," there might not be any information extracted related to "Avengers" at layer i+ 1, but rather, the primary information about "Avengers" might be
extracted at layer i−2. Therefore, layer i−2 effectively detects the pattern leading to the target output from the input, which is ki−2, representing as the key in key-value memory. Extracting the main information about "Avengers" , which is the target output, from W i−2proj based on k
i−2 and placing it in the residual flow constitutes what we refer to as pattern match. Otherwise, it is pattern unmatch. Thus, the ability to retrieve target information from Wproj based on k serves as the criterion for determining whether the pattern matches.
In summary, the patterns of editing knowledge may not be detected in the predefined editing layer, which we call pattern unmatch. Continuing to edit on such knowledge will lead to language model overfitting, resulting in toxicity flash. The investigation details of pattern unmatch and experimental evidence are provided in Appendix D.

heading: 5 Wise-Layer Knowledge Editor
text: In Section 4, we delved into the primary reason for failure in lifelong editing - pattern unmatch, which directly leads to toxicity flash and potentially more toxicity buildup. In light of this, we propose an editing method called WilKE. Unlike ROME and MEMIT, WilKE does not predefine editing layer; instead, it selects editing layer based on the degree of pattern match for different editing knowledge across various layers. We first describe where to edit in Section 5.1, followed by an explanation of how to edit in Section 5.2.

heading: 5.1 Where to Edit?
text: To implement knowledge editing, the initial step involves determining the locations where editing will take place.
Meng et al. (2022a) utilizes causal mediation analysis to identify the center of causal effects, MLP at specific layer, for storing factual knowledge. The MLP of the FFN is divided into two matrices, represented as follows:
FFN l(x) = σ(x ·W lfc) ·W lproj , (8) where W lfc ∈ Rd×dm and W lproj ∈ Rdm×d are the parameter matrices of the l-th layer’s FFN, FFN l, and dm is the dimension of the intermediate hidden layer in the FFN. The symbol σ represents the activation function, and x ∈ Rd is the input to the FFN. As described in the key-value memories (Sukhbaatar et al., 2015; Sukhbaatar et al., 2019; Geva et al., 2020), W lfc identifies patterns of the
input x to obtain the key vector k l, and then the value vector v l is retrieved from W lproj , as shown in Figure 2. Therefore, to achieve knowledge editing, we modify W lproj . After identifying the component that needs modification, we further determine the specific layer for modifying this component.
To find the editing layer l∗, our initial intuition is to identify the layer that produces the maximum activation strength for a specific knowledge, which is represented as argmaxl ∥σ(x ·W lfc)∥2. However, in practice, the optimization of δ l (Meng et al., 2022a) after the FFN for aligning model’s output to achieve knowledge updating varies across different layers, as detailed in Appendix D.2. Specifically, δ l is calculated as follows:
δ l = v l∗ −W lprojk l, v l∗ = argminzL(z)
where L(z) = 1 N
N∑
j=1
− logPfθ(mli:=z)[o ∗|cj + p]
+DKL(Pfθ(mli:=z)[y|p ′]||Pfθ [y|p′])
(9) where mli represents the output of the MLP in the lth layer on the ith token, which is the end of the subject, cj denotes a token sequence randomly generated to simulate irrelevant context, p corresponds to the knowledge we intend to edit, and p′ represents the essence of the subject. In simple terms, the first term in L(z) is for knowledge updating, while the second term is for maintaining an understanding of the essence of the subject. Further details can be referenced in Meng et al. (2022a).
As mentioned before, the importance of the hidden state outputted by different layers for editing specific knowledge is actually different. To comprehensively consider these two points, namely the activation strength of specific knowledge and the δ l optimized for knowledge editing, we define the editing layer (wise-layer) l∗ as follows:
l∗ = argminl ∥∥∥∥∥ δ l
∥W lproj∥2 σ(x ·W lfc) ∥∥∥∥∥ 2 (10)
where the term ∥W lproj∥2 in the denominator can be regarded as normalization, allowing for comparison across layers. Ultimately, we determine that the target editing location is W l
∗ proj of the layer l ∗.

heading: 5.2 How to Edit?
text: After determining the target editing location, the next step involves determining how to carry out an
edit. Same as Equation 9, we introduce a residual term δ l ∗ ∈ Rd to the output of the FFN in editing layer l∗, denoted as FFN l ∗ (x)+δ l ∗ . We optimize this residual term to align the model’s output with our expected results while not affecting irrelevant knowledge.
Subsequently, we allocate the optimized residual term δ l∗ to W l
∗ proj to accomplish knowledge
editing:
W l ∗ proj ← FFN l
∗ (x) + δ l ∗
σ(x ·W l∗fc) (11)
Afterwards, we have completed one editing. In summary, our approach starts from the perspective of pattern matching, attempting to identify the layer that is most suitable for editing the given knowledge across all layers, and then performs knowledge editing on that location.

heading: 6 Experiments

heading: 6.1 Experimental Setting
text: Model We utilize two widely employed autoregressive language models for knowledge editing: GPT-XL (1.5B) (Radford et al., 2019) and GPT-J (6B) (Wang and Komatsuzaki, 2021). Baselines Regarding knowledge editing methods, we select the following approaches: KnowledgeEditor (KE) (De Cao et al., 2021) utilizes a bidirectional LSTM to predict weight updates for editing data points; KnowledgeNeuron (KN) (Dai et al., 2021) employs knowledge attribution to determine the positions of neurons, followed by parameter updates on these neurons to implement knowledge updates; MEND (Mitchell et al., 2021) uses low-rank decomposition of gradients to learn fine-tuning of language models; ROME (Meng et al., 2022a) employs causal mediation analysis to identify the
center of causal effects, followed by gradient descent parameter updates on the MLP at that layer; MEMIT (Meng et al., 2022b) extends upon ROME by distributing residuals across multiple layers. Datasets, Metrics and Experiment Details Due to space limitations, details of dataset, metrics, and experimental details are provided in Appendix F for reference.

heading: 6.2 Main Results
text: As shown in Table 1, we present the knowledge editing results after 1024 edits on GPT-XL and GPT-J. The results indicate that current knowledge editing methods perform poorly in lifelong editing, far from the optimistic results reported in single editing. However, these methods have been directly transferred and used in many other scenarios (Ma et al., 2023; Li et al., 2023b; Anonymous, 2024; Wang et al., 2023).
WilKE demonstrates the most advanced comprehensive performance relative to the current knowledge editing methods. Specifically, under the same experimental conditions on GPT2-XL and GPT-J, WilKE achieves an average performance improvement of 46.2% and 67.8%, respectively, relative to the state-of-the-art methods.
To gain further insight, we have plotted the complete performance curves, and detailed results are presented in Appendix F.3.

heading: 6.3 Ablation Study
text: Since the core of our method lies in selecting different editing layers based on various knowledge, as demonstrated in Equation 10 in Section 5.1, we comprehensively consider three aspects: the optimization of δ for editing knowledge across different layers, the activation of specific knowledge across different layers σ(x · W lfc), and the ∥W lproj∥2 across different layers. Therefore, we
sequentially ablate these three factors to demonstrate that considering these three factors collectively leads to a better editing layer.
As depicted in Figure 9, it is evident that when individually ablated, both δ and σ(x ·W lfc) lead to a significant decrease in the performance of knowledge editing. Additionally, ablating ∥W lproj∥2 results in a slight decrease in the performance of knowledge editing. However, when considering these three factors collectively, superior experimental results are obtained.

heading: 7 Conclusion
text: In this work, we focus on lifelong knowledge editing, finding that current knowledge editing methods suffer from severe performance degradation in lifelong editing. Our experimental results reveal the toxicity buildup and toxicity flash that may occur during lifelong editing, leading to the deterioration of model’s performance. Through further investigation, we find that the direct cause of these problems lies in the predefined editing layer, while the underlying cause stems from pattern unmatch. To address this issue, we propose a model edit-
ing method called Wise-Layer Knowledge Editor (WilKE), which does not require predefined editing layer but selects editing layer based on the degree of pattern matching between different layers of the language model for specific editing knowledge. Experimental results demonstrate that in lifelong editing, WilKE significantly enhances overall performance compared to currently popular knowledge editing methods, achieving state-of-the-art knowledge editing performance. In summary, our work is significant for improving knowledge editing methods and provide valuable insights for future work.

heading: 8 Limitation
text: Despite the promising performance of WilKE, our current studies still have limitations. Firstly, we select editing layer based on specific knowledge, yet knowledge may be distributed across multiple layers, leaving the question of how language models store knowledge is still under explored. Secondly, similar to previous knowledge editing research, we focus on factual knowledge assessment, which serves as a crucial entry point for our study on knowledge editing. Furthermore, due to computational constraints, we did not conduct experiments on larger-scale language models but instead utilized GPT2-XL and GPT-J. However, WilKE does not require additional hypernetworks or other components, rendering it model-agnostic and thus exhibiting favorable scalability, enabling straightforward migration to larger models. Lastly, detecting match degree of specific knowledge across different layers of language models incurs a certain time cost, yet we believe this to be worthwhile in the initial stages of knowledge editing research.

heading: 9 Ethical Considerations
text: We have developed a method for knowledge editing in large language models under lifelong editing scenario, which may further expand our understanding of how language models store knowledge. However, the direct editing capability of large models also carries the potential for misuse, such as injecting malicious misinformation, biases, or other adversarial data into the model and deploying these edited models on open platforms. Given these concerns and our observations of speculative behavior, we emphasize the importance of sourcing large language models from authoritative origins and refraining from using them as sources of authoritative factual knowledge in critical environments.

heading: Acknowledgements
text: This work is supported by the Strategic Priority Research Program of Chinese Academy of Sciences (No. XDA27020203), the National Natural Science Foundation of China (No. 62176257).

heading: A Causal Mediation Analysis on GPT2-XL
text: From the perspective of causal mediation analysis (CMA) (Pearl, 2022; Vig et al., 2020; Meng
et al., 2022a), we aim to investigate the disparities between data leading to toxicity flash and other data. Specifically, we conduct CMA experiments on GPT2-XL, targeting data conducive to toxicity flash and contrasting it with other data. Through this approach, we seek to elucidate the knowledge extraction positions within the model that facilitate accurate responses to the given questions.
The CMA results for data resulting in toxicity flash on GPT2-XL are illustrated in Figure 10, 11, 12, 13. 14, 15, 16, 17.
The CMA results for other data on GPT2-XL are depicted in Figure 18, 19, 20, 21, 22, 23, 24, 25.
Here, our primary focus lies on the information extraction positions within the MLP corresponding to the third column of the figure. It is evident that the data leading to toxicity flash consistently extract crucial information from the first five layers of the model, demonstrating consistent outcomes. However, the results for other data indicate that different pieces of knowledge extract important information from relatively dispersed positions. This suggests that for different knowledge, information may be stored across different layers of the model, necessitating the selection of different layers for editing depending on different knowledge.

heading: B Toxicity Buildup and Toxicity Flash Data Spliter
text: During the process of editing GPT2-XL and GPT-J using the ROME method, we filter out data that would cause toxicity flash. The criteria for filtering primarily includes the effectiveness of editing and whether the L2 norm of the editing layer exhibited abnormally high increases. Specifically, based on our experience, these data causing toxicity flash tend to exhibit the following phenomenon: during the editing phase, there is a relatively high success rate, but during the rollback phase after editing, there is a lower success rate. Therefore, we manually filter out data where the success rate of editing during the rollback phase was less than 10%. Subsequently, we further examine this subset of data, manually identifying the data causing toxicity flash on GPT2-XL and GPT-J respectively.
The editing data that caused toxic flash in GPT2XL are listed in Table 2.
The editing data that caused toxic flash in GPT-J are listed in Table 3.
As we can observe, the majority of data in both tables overlap, which is an interesting finding.

heading: C Toxicity Analysis on MEMIT
text: Due to MEMIT’s distribution of residuals across multiple layers based on ROME, it partially conceals the issue of toxicity flash. Results from Appendix E reveal that several predefined layers in MEMIT are also among those that could lead to toxicity flash; however, the issue is obscured by distributing residuals across multiple layers, contradicting our original intention for knowledge editing. Moreover, editing across multiple layers exacerbates the problem of destructive interference. Therefore, as depicted in the results of Section F.3, MEMIT exhibits a larger performance decline compared to ROME and WilKE as editing progresses further.
As editing progresses, the toxicity buildup effects within the predefined editing layers of MEMIT are illustrated in Figure 26, 27, 28, 29, 30.
Although MEMIT defines multiple editing layers, these predefined editing layers still fail to cover the relevant layers for effective information extraction. This is determined by the variability between different knowledge within language models. Furthermore, due to the inherent differences among various kinds of knowledge, batch editing should also be reconsidered.

heading: D Pattern Unmatch
text: In this section, we can proceed to a more formal description of pattern unmatch in Section 4.3. This phenomenon occurs when there is partial data for which the activation value σ(x ·W lfc) in the predefined editing layer of ROME is extremely small (across several orders of magnitude, detailed results can be found in Appendix D.1). However, in reality, the difference between FFN(x)l + δ and other layers cannot be considered as the dominant factor (refer to detailed results in Appendix D.2). Therefore, according to Equation 11, the extremely small activation value σ(x ·W lfc) in the denominator becomes the primary cause of toxicity flash.
D.1 Activation Strength
The distribution of activation strength for data causing toxicity flash on GPT2-XL is depicted in Figure 31.
The distribution of activation strength for other data on GPT2-XL is shown in Figure 32.
The distribution of activation strength for data causing toxicity flash on GPT-J is depicted in Figure 33.
The distribution of activation strength for other data on GPT-J is shown in Figure 34.
(a) Activation strength over layers on case 3561.
(b) Activation strength over layers on case 4661.
(c) Activation strength over layers on case 4790.
(d) Activation strength over layers on case 4988.
(e) Activation strength over layers on case 8793.
(f) Activation strength over layers on case 15452.
(g) Activation strength over layers on case 16575.
(h) Activation strength over layers on case 16781.
Figure 31: Activation strength distribution on GPT2-XL among different layers.
(a) Activation strength over layers on case 0.
(b) Activation strength over layers on case 5.
(c) Activation strength over layers on case 7.
(d) Activation strength over layers on case 13.
(e) Activation strength over layers on case 14.
(f) Activation strength over layers on case 22.
(g) Activation strength over layers on case 36.
(h) Activation strength over layers on case 37.
(a) Activation strength over layers on case 3561.
(b) Activation strength over layers on case 4661.
(c) Activation strength over layers on case 4988.
(d) Activation strength over layers on case 8475.
(e) Activation strength over layers on case 8793.
(f) Activation strength over layers on case 15452.
(g) Activation strength over layers on case 16575.
(h) Activation strength over layers on case 16781.
Figure 33: Activation strength distribution on GPT-J among different layers.
(a) Activation strength over layers on case 0.
(b) Activation strength over layers on case 5.
(c) Activation strength over layers on case 7.
(d) Activation strength over layers on case 14.
(e) Activation strength over layers on case 29.
(f) Activation strength over layers on case 52.
(g) Activation strength over layers on case 54.
(h) Activation strength over layers on case 56.
Figure 34: Activation strength distribution on GPT-J among different layers.
D.2 Delta Strength
The distribution of delta strength for data causing toxicity flash on GPT2-XL is depicted in Figure 35.
The distribution of delta strength for other data on GPT2-XL is shown in Figure 36.
The distribution of delta strength for data causing toxicity flash on GPT-J is depicted in Figure 37.
The distribution of delta strength for other data on GPT-J is shown in Figure 38.

heading: E More Edit Analysis on Toxicity Flash
text: In this section, we present the experimental results on additional data described in Section 4.3.
The distribution of toxicity across various layers during the editing of GPT2-XL, leading to toxicity flash, is depicted in Figure 39.
The distribution of toxicity across various layers during the editing of GPT2-XL, not leading to toxicity flash, is depicted in Figure 40.
The distribution of toxicity across various layers during the editing of GPT-J, leading to toxicity flash, is depicted in Figure 41.
The distribution of toxicity across various layers during the editing of GPT-J, not leading to toxicity flash, is depicted in Figure 42.

heading: F Experimental Details
text: Reviewing Equation 6, here y ′ej may not be equal to yej , depending on whether the editing data after the test data will conflict with the existing knowledge (Li et al., 2023c; Jin et al., 2024a; Jin et al., 2024b).
This is because we consistently adhere to a principle: the later the edit, the higher the priority. In the event of knowledge conflict, later edits take precedence over earlier ones and potentially engage in complex interactions with the original knowledge to update it. For instance, as highlighted in Li et al. (2023c), if the model contains the fact "The notable work of Shakespeare is Hamlet" and undergoes the first edit "Hamlet was written in English → French" followed by the second edit "Shakespeare wrote in French → German" the second edit, if interacting with the original model’s fact, could result in a modification of the first edit’s outcome to "Hamlet was written in German" (though not modified explicitly in this way).
Considering the knowledge conflict issues under lifelong editing, and the current incomplete understanding of knowledge storage and updating mechanisms in transformers, we propose a experimental method, designed for methods that modifying model’s parameters, utilizing rollback editing, to address such challenges in lifelong editing. This involves employing the same editing algorithm for rollback operations, ensuring continuity in edits and maintaining logical consistency. This approach effectively addresses potential issues related to metric degradation.
F.1 Datasets
Specifically, we construct these baselines in Section 6.1 using the CounterFact dataset (Meng et al., 2022a), where each record is derived from the cor-
(a) Delta strength over layers on case 3561.
(b) Delta strength over layers on case 4661.
(c) Delta strength over layers on case 4790.
(d) Delta strength over layers on case 4988.
(e) Delta strength over layers on case 8793.
(f) Delta strength over layers on case 15452.
(g) Delta strength over layers on case 16575.
(h) Delta strength over layers on case 16781.
(a) Delta strength over layers on case 0.
(b) Delta strength over layers on case 5.
(c) Delta strength over layers on case 7.
(d) Delta strength over layers on case 13.
(e) Delta strength over layers on case 14.
(f) Delta strength over layers on case 22.
(g) Delta strength over layers on case 36.
(h) Delta strength over layers on case 37.
Figure 36: Delta strength distribution on GPT2-XL among different layers.
(a) Delta strength over layers on case 3561.
(b) Delta strength over layers on case 4661.
(c) Delta strength over layers on case 4988.
(d) Delta strength over layers on case 8475.
(e) Delta strength over layers on case 8793.
(f) Delta strength over layers on case 15452.
(g) Delta strength over layers on case 16575.
(h) Delta strength over layers on case 16781.
(a) Delta strength over layers on case 0.
(b) Delta strength over layers on case 5.
(c) Delta strength over layers on case 7.
(d) Delta strength over layers on case 14.
(e) Delta strength over layers on case 29.
(f) Delta strength over layers on case 52.
(g) Delta strength over layers on case 54.
(h) Delta strength over layers on case 56.
Figure 38: Delta strength distribution on GPT-J among different layers.
(a) Toxicity distribution on case 3561.
(b) Toxicity distribution on case 4661.
(c) Toxicity distribution on case 4790.
(d) Toxicity distribution on case 4988.
(e) Toxicity distribution on case 8793.
(f) Toxicity distribution on case 15452.
(g) Toxicity distribution on case 16575.
(h) Toxicity distribution on case 16781.
(a) Toxicity distribution on case 0.
(b) Toxicity distribution on case 5.
(c) Toxicity distribution on case 7.
(d) Toxicity distribution on case 13.
(e) Toxicity distribution on case 22.
(f) Toxicity distribution on case 36.
(g) Toxicity distribution on case 37.
(h) Toxicity distribution on case 48.
Figure 40: Toxicity distribution on GPT2-XL among different layers. The results are obtained from testing with other normal data.
(a) Toxicity distribution on case 3561.
(b) Toxicity distribution on case 4661.
(c) Toxicity distribution on case 4988.
(d) Toxicity distribution on case 8475.
(e) Toxicity distribution on case 8793.
(f) Toxicity distribution on case 16575.
(g) Toxicity distribution on case 16781.
(h) Toxicity distribution on case 21142.
(a) Toxicity distribution on case 0.
(b) Toxicity distribution on case 5.
(c) Toxicity distribution on case 7.
(d) Toxicity distribution on case 14.
(e) Toxicity distribution on case 29.
(f) Toxicity distribution on case 48.
(g) Toxicity distribution on case 52.
(h) Toxicity distribution on case 56.
Figure 42: Toxicity distribution on GPT-J among different layers. The results are obtained from testing with other normal data.
responding entry in PARAREL (Elazar et al., 2021). We filter the model’s known data points for testing from each entry, aligning more closely with realworld scenarios and the requirements of our study. Each edited data point corresponds to a knowledge tuple (s, r, o⇒ o∗) and a manually curated prompt template.
The data format for the knowledge tuple (Danielle Darrieux,mother tongue,French ⇒ English) is displayed in Table 4. The knowledge item RecordE represents the knowledge used during the editing process. RecordG is a paraphrase of RecordE in an unrelated context. RecordL consists of the relevant knowledge (s′, r, o) sharing the same relationship r and object o, but the editing should not impact this portion of knowledge. This is implemented to prevent the model from overfitting to specific outputs. In this instance, xe is "The mother tongue of Danielle Darrieux is" ye is "English" and the original output yo is "French".
F.2 Metrics As previously mentioned, the issue of knowledge conflicts (Li et al., 2023c) may arise in lifelong editing, potentially rendering the retention metric ineffective in the evaluation of lifelong editing methods (Huang et al., 2023)(Hartvigsen et al., 2022). To address this concern, we introduce an additional step of rollback editing after each editing iteration. Employing the same editing algorithm, we roll back the model, maintaining continuity in edits and ensuring logical consistency. Formally, after editing the model f∗θi−1 to obtain fθi , we denote the model after the rollback operation as f∗θi , and we expect the sequence f∗θi → f ∗ θi−1 → · · · → f ∗ θ0
, where f∗θ0 = fθ0 .
Specifically, we extract a subset O = {xei , yei} |O| i=1 from the known knowledge dataset of the filtered models (it is crucial to ensure consistency before and after the system). We divide O into two parts, P = {xei , yei} |P| i=1 and Q = {xei , yei} |P|+|Q| i=|P|+1. P is used for model editing and measuring the editing retention rate, while Q serves as a retention set to measure the impact of edits on the model’s original knowledge.
For the i-th edited item in P , the evaluation is divided into two stages:
1. Editing Stage: Use (xei , yei) to edit the model f∗θi−1 and obtain fθi . Measure the effectiveness score, generalization score, and domain score of fθi .
2. Rollback Stage: For the edited model, use (xei , yoi) to edit fθi and obtain f ∗ θi
. Measure the retention rate of f∗θi on the edited data and the original knowledge.
Upon completing all edits for {xei , yei} |P| i=1, we evaluate the editing algorithm using the following metrics:
• Effectiveness Score (ES): Measures whether the model produces the expected predictions for the current edited data after each editing step.
ES = 1 P P∑
i=1
I(fθi(xei) = yei) (12)
• Generality Score (GS): Assesses whether the model produces the expected predictions for the equivalent inputs E(xei) of the current edited data after each editing step.
GS = 1 P P∑
i=1
|E(xei )|∑
j=1
I(fθi(xj) = yei), (13)
where xj ∈ E(xei).
• Locality Score (LS): Evaluates whether the model maintains the original output on unrelated data I(xei) after each editing step.
LS = 1 P P∑
i=1
|I(xei )|∑
j=1
I(fθi(xj) = yoi), (14)
where xj ∈ I(xei).
• Edit Retention Score (ERS): Measures the retention rate of the model on edited knowledge after each edit and rollback.
ERS = 1 P P∑
i=1
I(f∗θn(xei) = fθ0(xei))
(15)
• Original Retention Score (ORS): Measures the retention rate of the model on original knowledge after each edit and rollback.
ORS = 1
|Q|
|P|+|Q|∑ i=|P|+1 I(f∗θn(xei) = fθ0(xei))
(16)
Additionally, we propose a composite metric S based on the harmonic mean of the above metrics.
F.3 Complete Performance Curves The complete performance curve is illustrated in Figure 43.
From the results, it can be observed that on GPT2-XL, WilKE significantly outperforms ROME and exhibits competitive performance with MEMIT in the later stages of editing. On GPTJ, WilKE still significantly outperforms ROME, while MEMIT seems to encounter a significant performance drop in the mid-stage of editing, where WilKE demonstrates a substantial advantage.
Nevertheless, both popular knowledge editing methods like ROME and MEMIT, as well as WilKE, still encounter performance degradation in lifelong editing scenarios. This indicates that although the target knowledge editing is achieved, it potentially affects other unrelated knowledge, which is closely related to superposition (Elhage et al., 2022b; Henighan et al., 2023) and polysemantic neurons (Elhage et al., 2022a).
(a) Editing results on GPT2XL.
(b) Editing results on GPTJ.
Figure 43: Editing results among ROME, MEMIT and WilKE.

heading: Future Work
text: In summary, our work is significant for improving knowledge editing methods and provide valuable insights for future work