Reasons and Solutions for the Decline in Model Performance after Editing

Authors: xiusheng  huang
jiaxiang  liu
yequan  wang
kang  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:21:30

Abstract: Knowledge editing technology has received widespread attention for low-cost updates of incorrect or outdated knowledge in large-scale language models. However, recent research has found that edited models often exhibit varying degrees of performance degradation. The reasons behind this phenomenon and potential solutions have not yet been provided. In order to investigate the reasons for the performance decline of the edited model and optimize the editing method, this work explores the underlying reasons from both data and model perspectives. Specifically, 1) from a data perspective, to clarify the impact of data on the performance of editing models, this paper first constructs a **M**ulti-**Q**uestion **D**ataset (**MQD**) to evaluate the impact of different types of editing data on model performance. The performance of the editing model is mainly affected by the diversity of editing targets and sequence length, as determined through experiments. 2) From a model perspective, this article explores the factors that affect the performance of editing models. The results indicate a strong correlation between the L1-norm of the editing model layer and the editing accuracy, and clarify that this is an important factor leading to the bottleneck of editing performance. Finally, in order to improve the performance of the editing model, this paper further proposes a **D**ump **for** **S**equence (**D4S**) method, which successfully overcomes the previous editing bottleneck by reducing the L1-norm of the editing layer, allowing users to perform multiple effective edits and minimizing model damage. Our code is available at https://github.com/nlpkeg/D4S.

Main paper: 
 introduction: 
 Large-scale language models (LLMs) have demonstrated exceptional performance in NLP tasks [Huang et al., 2021 [Huang et al., , 2022]] . However, as knowledge continues to evolve, LLMs inevitably contain incorrect or outdated information. Due to their vast number of parameters, directly fine-tuning the model would require a substantial amount of computational resources [Gupta et al., 2023] . As a result, knowledge editing techniques have emerged as a low-cost and effective method for updating a model's knowledge [Yao et al., 2023] . These techniques involve modifying a small number of the model's parameters to update its internal knowledge [Ding et al., 2023] . However, growing evidence suggests that altering model parameters can have a negative impact on the model's performance. The specific reasons behind this phenomenon remain unclear, and corresponding optimization methods are currently unavailable.Previous research has investigated the performance degradation of edited models [Wang et al., 2023 , Mazzia et al., 2023] . Specifically, Hase et al. [2024] found that edited models suffer from catastrophic forgetting, where they forget previously edited samples. Additionally, Gu et al. [2024] showed that edited models exhibit significant performance declines on downstream tasks, which severely hinders the practical applicability of knowledge editing techniques. However, these studies failed to identify the underlying causes of performance degradation in edited models and did not propose optimization methods to mitigate this issue [Zhang et al., 2024] .Fig. 1 : This framework outlines the comprehensive approach to understanding the performance decline of edited models. On the left, traditional knowledge editing tasks are categorized into different types, each with distinct editing objectives: yes/no, a/b/c/d, and entity/event. On the right, our experiments are structured from both data and model perspectives. From the data perspective, we conduct three experiments: (a) a comprehensive performance evaluation of the model, (b) the construction of a Multi-Question Dataset (MQD), and (c) an assessment of the impact of editing different target outputs on model performance. From the model perspective, we design four experiments: (d) an evaluation of the edited model's forgetting ability, (e) an identification of the current knowledge editing method's bottleneck and an exploration of the correlation between editing probability values and parameter layer norms, and (f) a proposal of a sequence editing method, which effectively enhances the performance of the edited model. This paper examines the factors that influence model performance and optimization methods from both data and model perspectives. As shown in Figure 1 , from the data perspective, we evaluated the performance of edited models on multiple generalization tasks and found that the performance degradation of edited models is correlated with the editing objectives. We then constructed Multi-Question Dataset (MQD) with different question types, including multiple-choice, true/false, and direct generation, with corresponding editing objectives of yes/no, a/b/c/d, and entity/event, respectively. By calculating the perplexity (PPL) of different editing objectives, we discovered that the larger the PPL, the more severe the performance degradation of the edited model.From the model's perspective, we investigate the reasons behind the decline in model performance from two angles: catastrophic forgetting and the bottleneck imposed by the number of edits. Our analysis reveals a strong correlation between the accuracy of edits and the L1-norm growth of the parameter layers after editing. To mitigate this issue, we propose a Dump for Sequence (D4S) method that regulates the explosive growth of parameter layers during the editing process. This approach effectively enhances the performance of the edited model and overcomes the bottleneck associated with the number of edits.To the best of our knowledge, we are the first to investigate the causes of performance degradation in edited models and concurrently propose an effective sequence editing method to enhance the performance of edited models. Our contributions can be summarized as follows:• To investigate the impact of data on the performance of edited models, we performed evaluations across multiple tasks, revealing that the editing objective is the primary factor influencing model performance. By creating datasets with diverse question types, we established a strong correlation between the perplexity (PPL) of the editing objectives and the performance of the edited model.• We find that the decline in edited model performance is correlated with the explosive growth of the L1-norm of parameter layers during the editing process, which is the primary cause of both catastrophic forgetting and the bottleneck imposed by the number of edits.• To enhance the performance of the edited model, we propose a caching sequence edit method that leverages O(1) space complexity to retain past knowledge, regulates the explosive growth of parameter layer norms, and thereby effectively improves the performance of the edited model, ultimately overcoming the bottleneck imposed by the number of edits.2 Related Work 
knowledge editing datasets: 
 The existing knowledge editing dataset can be divided into triplet form and event form. In triplet format dataset, commonsense knowledge dataset includes PEP3k and 20Q [Porada et al., 2021 , Gupta et al., 2023] , factual knowledge includes ZsRE [Levy et al., 2017] , CounterFact [Meng et al., 2022a] ,Fact Verification [Mitchell et al., 2022] , Calibration [Dong et al., 2022] , MQuAKE [Zhong et al., 2023] and RaKE [Wei et al., 2023] . In event format dataset, datasets with only factual knowledge, including ELKEN [Peng et al., 2024] , EVEDIT [Liu et al., 2024] and CKD [Huang et al., 2024] .The previous editing methods mainly focused on editing knowledge in the form of triples, with a small amount of knowledge in the form of editing events. The methods for editing triplet forms mainly include : (1)Locate-Then-Edit method [Dai et al., 2021 , Meng et al., 2022a ,b, Li et al., 2024] ,(2) Memory-based method [Mitchell et al., 2022 , Madaan et al., 2022 , Zhong et al., 2023 , Zheng et al., 2023] , (3) Hyper-network method [Mitchell et al., 2021 , De Cao et al., 2021 , Tan et al., 2023] .The method for editing event forms is Self-Edit [Liu et al., 2024] .The damage caused to the model by updating model parameters is unknown. [Hase et al., 2024] found that the edited model had catastrophic forgetting issues and performance degradation on downstream tasks. [Gu et al., 2024 ] evaluated the edited model on eight downstream tasks and used multiple editing methods. It was found that the edited model exhibited varying degrees of performance degradation. However, the above methods only found a decrease in performance of the edited model, without pointing out the reasons for the performance decline. At the same time, they did not propose effective editing methods to improve the performance of the edited model. 
data-specific factors affecting performance: 
 In this section, we investigate the primary data factors contributing to the decline in model performance following editing.To assess the performance of the edited model, we curated a diverse range of editing and evaluation datasets for testing.The Dataset for Editing. As illustrated in Figure 1 , we selected common sense knowledge and factual knowledge datasets as editing corpora, featuring diverse data formats such as triplets and free text. In Figure 2 , we utilized the zsRE [Levy et al., 2017] , ELKEN [Peng et al., 2024] , 20Q [Porada et al., 2021 , Gupta et al., 2023] , and CKD [Huang et al., 2024] datasets as editing corpora. Notably, the commonsense dataset 20Q has editing objectives in the form of 0/1 labels, whereas the other three datasets have editing objectives of entity, entity, and event, respectively.The Dataset for Evaluation. We employed six types of datasets in the Open LLM Leadership board evaluation, comprising AI2 Reasoning Challenge (25-shot) [Clark et al., 2018] , HellaSwag (10-shot) [Zellers et al., 2019] , the Common Sense Reasoning Dataset, MMLU (5-shot) [Hendrycks et al., 2020] , a widely used benchmark for assessing multitask accuracy, which encompasses 57 tasks including basic mathematics, American history, computer science, law, and others. Additionally, we utilized TruthfulQA (0-shot) [Lin et al., 2021] , a benchmark designed to test the model's propensity for dishonesty, as well as WinoGrande [Sakaguchi et al., 2021] : Common Sense Reasoning and GSM-8K [Cobbe et al., 2021] : Mathematical Reasoning.Result Analysis. As illustrated in Figure 2 , we observed that the KN [Dai et al., 2021] editing method substantially degrades the model's performance after editing a single sample. In contrast, when employing the ROME [Meng et al., 2022a] editing method, the model's performance did not experience a significant decline even after editing 1000 samples. Similarly, with the MEMIT [Meng et al., 2022b ] method, the model's performance remained relatively stable when editing up to 2000 samples. However, beyond 2000 samples, a pronounced decline in performance occurred, and further increases in the number of edited samples did not lead to additional performance degradation.It is noteworthy that the ROME [Meng et al., 2022a] and MEMIT [Meng et al., 2022b ] methods exhibit less pronounced performance degradation when editing the 20Q [Porada et al., 2021 , Gupta et al., 2023] dataset. In contrast, when editing other datasets, the model's performance displays varying degrees of decline. This can be attributed to the fact that the 20Q [Porada et al., 2021 , Gupta et al., 2023] dataset's editing objective is in the form of binary labels (0/1), featuring a single element for the editing objective and a relatively small number of tokens.To elucidate the impact of different editing objectives on the performance of the edited model, we created a Multi-Question Dataset (MQD) based on the ATOMIC commonsense database [Sap et al., 2019] . This dataset comprises three question types: true/false, multiple-choice, and direct generation.The corresponding editing objectives are yes/no, a/b/c/d, and entity/event, respectively. Each question type consists of 4000 samples. [Sap et al., 2019] database into directly generated(DG), multiple-choice questions(MQ), and true/false questions(T/F). The editing objectives are "shakes PersonX hand", "d" and "yes", respectively.Data Preparation. The ATOMIC database [Sap et al., 2019] , developed by the Allen Institute and later optimized in its subsequent version [Hwang et al., 2021] , is a well-known commonsense repository. The data format in ATOMIC [Sap et al., 2019] is < Event 1 , Relationship, Event 2 >, which contains unrecognized markers (e.g. ___, etc.) and invalid characters (e.g., &, etc.) that we manually filtered out. Furthermore, the relationship types in ATOMIC [Sap et al., 2019] are abbreviated and not easily comprehensible by humans. Although ATOMIC [Sap et al., 2019] provides corresponding annotations, they are insufficient to form a coherent statement when constructing the prompt. To address this, we also performed manual template rewriting to ensure a smoother overall prompt.MQD Dataset. Different problem formats are associated with distinct editing objectives. The MQD dataset encompasses three formats: Directly Generated (DG), Multiple-choice Questions (MQ), and True/False questions (T/F). According to our statistical analysis, the Perplexity (PPL) values for the editing objectives of these three question types are 12.3, 43.3, and 297.4 , respectively. The calculation formula for PPL is as follows:EQUATIONWhere p θ (x i | x <i ) is based on the sequence before i, and the log-likelihood of the i-th token. In the Table 1 , for directly generating datasets, we directly concatenate Event 1 , the rewritten relationship, and Event 2 to form a prompt. For multiple-choice questions, we designated the target answer as one of the options and randomly selected events from other natural categories in the dataset as the remaining options. For true/false questions, we asked LLMs to determine whether the newly formed prompt is logical. To mitigate bias, we established multiple positive and negative examples. Notably, the core information of the three question types remains consistent, with the primary difference lying in the question types and editing objectives.In this section, we investigate the effect of various editing objectives on model performance by editing the Multi-Question Dataset (MQD)3.2.The Dataset for Editing and Evaluation. The MQD dataset comprises three question types: true/false, multiple-choice, and directly generated. The knowledge sources for these three question types are consistent, and we conducted relevant experiments directly using the MQD dataset. We selected ROME [Meng et al., 2022a] as the editing method. According to our statistical analysis, the average length of the input tokens for the three question types is 23.44, 35.03, and 13.38 , respectively, while the average length of the editing objectives tokens is 1, 1, and 3.88, respectively. The true/false questions have two possible output types: yes or no. The multiple-choice questions have four editing objectives: a, b, c, and d. In contrast, the directly generated questions have more diverse editing objectives, including entities or events, with the number of tokens for events typically exceeding 1. Result Analysis. As illustrated in Figure 3 , all three question types -true/false, multiple-choice, and direct generation -compromise the model's performance, with direct generation causing the most significant damage. The experimental results suggest that a higher perplexity (PPL) of the editing objectives leads to more severe performance degradation. Meanwhile, we also calculated the L1-norm of the editing layer and found that the higher the perplexity of the editing target, the larger the L1-norm. Consequently, from a data perspective, the decline in model performance after editing is attributed to the diversity of editing objectives and the length of tokens.4 Model-Specific Factors Affecting PerformanceIn this section, we investigate the model-centric factors contributing to performance degradation and propose optimization strategies to enhance the performance of the edited model. 
forgetting about previously edited samples: 
 Fig. 4 : Assessment of forgetting ability of models.As illustrated in Figure 7 in Appendix B, conventional editing methods typically assess a sample's quality immediately after editing. However, in real-world applications, it is often necessary to evaluate the editing success rate after processing a sequence of samples. The traditional evaluation approach overlooked the edited model's impact on previously edited samples. This section investigates the model's forgetting issue with respect to previously edited samples.The dataset and method for evaluation. We employed the factual triplet dataset zeRE [Levy et al., 2017] for experimental purposes and utilized ROME [Meng et al., 2022a] and MEMIT [Meng et al., 2022b] as editing methods. To assess the varying degrees of forgetting in the model, we devised two experimental approaches. As illustrated in Figure 4 , the "Overall Evaluation" involves evaluating all previous samples using the edited model each time it is completed. In contrast, "The Degree of Forgetting" involves obtaining a checkpoint after editing 1000 samples, followed by an evaluation of the model's forgetting degree every 100 previously edited samples, in the order of editing.Result analysis. As shown in Figure 4 , for the "Overall Evaluation", the ROME [Meng et al., 2022a] method has the highest accuracy of 0.04, indicating that the edited model has a significant forgetting problem on the 100 previously edited samples. The accuracy of the MEMIT [Meng et al., 2022b] method has decreased from the highest near 1.0 to 0.0, indicating that as the number of model edits increases, the forgetting problem of the model becomes increasingly severe. For the degree of forgetting, the ROME [Meng et al., 2022a ] method and MEMIT [Meng et al., 2022b ] method always have a low success rate. Especially the MEMIT [Meng et al., 2022b ] method shows a success rate of 0 in most samples. We suspect that the MEMIT [Meng et al., 2022b] method after editing 1000 samples may not be able to successfully edit the samples. We also conducted relevant experiments in the next section.Sequence editing refers to the repeated updating of knowledge within a model. However, the number of successful edits that existing knowledge editing methods can achieve is not infinite. We conducted experiments on two editing methods to examine the bottleneck in the number of edits for each method.The dataset and method for evaluation. The editing models used in the experiments are all based on GPT-J (6B) [Wang and Komatsuzaki, 2021] . Consistent with the original work, the fixed MLP layers for the ROME [Meng et al., 2022a] and PMET [Li et al., 2024] methods are [5] and [3, 4, 5, 6, 7, 8] , respectively. We used the factual triplet dataset zeRE for experimental data and applied ROME and MEMIT as editing methods. All experiments were conducted with a batch size of 1, and a total of 1000 samples were edited. We recorded the probability value of each edited sample.Result analysis. As shown in Figure 5 .a, after editing 100 samples, the ROME method shows an overall decrease in probability values. While some probability values still update to larger values as the number of edits increases, the overall trend is downward. For the MEMIT method, after editing 850 samples, the probability values showed a significant decrease, approaching zero. This indicates that the MEMIT method exhibits a phenomenon of complete editing inefficiency after 850 edits. This experiment reflects the bottleneck phenomenon in editing methods, with ROME having an effective editing count of 100 and MEMIT having an effective editing count of 850. We plotted the L1-norm variation of different methods on the editing layer during the editing process. As shown in Figure 5 .b, when the number of edits reaches 100, the norm of the 5-th MLP layer edited by the ROME method significantly increases. Furthermore, in Figure 5 .c, after 850 edits using the MEMIT method, the editing layers [3, 4, 5, 6, 7, 8 ] also exhibit an explosive increase in norm. The results indicate that from a model perspective, the decline in model performance after editing is due to the explosive growth of norms in the editing layers during the editing process.Sequence editing refers to the process of editing a single or multiple samples multiple times. With the continuous updating of world knowledge, constantly updating the knowledge within models has become an urgent need. The experimental results in Sections 3 and 4 show that the performance of the model significantly decreases after sequence editing. In this section, we propose a Dump for Sequence (D4S) knowledge editing method that effectively improves the performance of the edited model.The knowledge editing method is to update factual associations stored in the parameters of transformers [Meng et al., 2022b] . Given a sequence of textx = [x 1 , • • • , x m ], the transformer's hidden state h l,j at the layer l and the token j is calculated:EQUATIONwhere γ indicate layer norm and σ means a non-linear function. The knowledge editing method is to update the knowledge in the model by changing particular weight W . For MEMIT [Meng et al., 2022b] method, there are triples of fact to be editedξ = {(s i , r i , o i )} n i=1, where s i means the subject, o i is the object and r i means relation between them. And we have o i ̸ = LLM (s i , r i ). For simplicity, we use h l i [x] to represent the last token's hidden state (h l,m i [x]) of the i th prompt x. To make o i = LLM ′ (s i , r i ), the target of i th edit z i = h L i + δ i is got by optimizing:EQUATIONwhereh L i = h L i [p(s i )]is the hidden state of the last edit layer L, p(s i , r i ) denotes the prompt consisting of subject and relation, and pre k indicates a random prefix to obtain generalizability. Subsequently, for each layer l ∈ L needs to be edited, we can take the following approach to update W l out ∈ R u×v :EQUATION∆ l = (R l K l T )(Cov l + K l K l T ) -1 , W l out ← W l out + ∆ l (5) with R l = [r l 1 , • • • , r l n ], K l = [k l 1 , • • • , k l n ], Cov l = K l 0 K l 0 T. And K l 0 are the keys of knowledge irrelevant with ξ. A simple idea to optimize the knowledge of sequence editing as a whole is saving the editing history R l and K l . For each new edit with k l n+1 and r l n+1 , we can concate it with history:EQUATIONEQUATIONIn this way, we can optimize all the knowledge of sequence editing as a whole to mitigate the damage to LLM. However, the space complexity of such a dump method is O(n), which is unacceptable to us.The D4S method is designed to save the editing history in O(1) space and apply batch editing methods in sequence editing situations. We note that for ∆ l = (R l K l T )(Cov l + K l K l T ) -1 , the parts related to the editing history can be written as:EQUATIONwhere we have R l K l T ∈ R u×v and K l K l T ∈ R v×v . So we just need to save the two matrices above. For each new edit with k l n+1 and r l n+1 , we can integrate it into edit history with a smiple addition operation:EQUATIONThis approach requires just O(1) storage space and allows us to convert sequence editing methods into batch editing methods, thus reducing the damage to the edited model during sequence editing.Additionally, this ability to consolidate each individual edit instance into a single batch makes the locate and editing method distinct from fine-tuning.To demonstrate that our D4S method can effectively alleviate norm growth in the editing layer, we can consider the update of parameters edited by the previous method MEMIT [ Meng et al., 2022b] after n edits:EQUATIONRegarding the D4S method, we have:∆W D4S = ( n i=1 r i k T i )(K 0 K T 0 + n i=1 k i k T i ) -1 = n i=1 (r i k T i )(K 0 K T 0 + n i=1 k i k T i ) -1 (13) Due to K 0 K T 0 + k i k T i and K 0 K T 0 + n i=1 k i k T i being positive definite, intuitively, the inverse of K 0 K T 0 + n i=1 k i k T i is expected to have smaller numerical values compared to K 0 K T 0 + k i k T i .Therefore, the norm of ∆W D4S is smaller than that of ∆W M EM IT . The experimental results in Figures 6 also demonstrate the effectiveness of the D4S method in mitigating L1-norm growth. 
experiments: 
 The experiments are designed to answer two research questions: a) How does D4S perform in sequence editing compared to other methods? b) How much damage does D4S do to the model?We counted the results of different methods after 1,000 sequence edits on GPT-J (6B) [Wang and Komatsuzaki, 2021] and Llama2 (7B) [Touvron et al., 2023] . The Appendix C provides Metric indicators. As show in Table 2 , compared to previous editing methods, our method D4S has achieved significant performance improvement. Specifically, when Edits=500 and Edits=1000, the average performance of D4S achieved State-Of-The-Art(SOTA) results, indicating that D4S still has superior editing ability even after editing multiple samples.We conduct additional experiments on Counterfact [Meng et al., 2022b] and Mquake [Zhong et al., 2023] dataset in Appendix D. In addition to this, we also wanted to explore the model's forgetting of previous edits, so we chose every 100 edits as a checkpoint to investigate this in Appendix E. As shown in Figure 6 , we explore the impact of D4S on the model by examining the changes in average weight norms and performance on downstream tasks of GPT. The downstream task performance changes for Lllama are in the Appendix F. The experimental results indicate that the norm of the D4S method did not increase after 10000 edits, and the performance of downstream tasks only decreased slightly. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: [NA] Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 This paper explores the reasons behind the decline in model performance from both data and model perspectives. From the data perspective, the decline is attributed to the diversity of editing objectives and the length of tokens. From the model perspective, the decline is due to the explosive growth of norms in the editing layers during the editing process. To enhance the performance of edited models, we propose a Dump for Sequence (D4S) method, which effectively improves the performance of edited models and overcomes previous editing bottleneck issues. This method allows for multiple effective edits with minimal impact on model performance.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: [NA] Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We note a few limitations of the experiments conducted in this paper:(1) This paper only used the GPT-J (6B) and Llama2 (7B) models. Due to limitations in computational resources, experiments on larger scale models were not conducted.(2) For the D4S method, our training consists of only 2000 steps due to computational resource limitations. By editing more samples, we can better identify the performance bottleneck of the D4S method.(3) Due to computational resource limitations, the MEMIT and PMET methods did not use a batch size greater than 1 in our experiments. To better assess batch editing, more experiments with larger batch sizes should be attempted. Same as MEMIT, we took the following three indicators to evaluate the impact on model performance:EQUATIONEQUATIONwhere par(s i , r i ) is the set of p(s i , r i )'s paraphrases, nei(s i , r i , o i ) is the set of (s i , r i , o i )'s neighborhoods.We also conduct additional experiments on Counterfact [Meng et al., 2022b] and Mquake [Zhong et al., 2023] F Appendix /Downstream Performance of Llama 4 : Downstream performance of llama after 10,000 edits.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.(a) The probability value. (b) L1-norm on ROME. (c) L1-norm on MEMIT. 


Decision: The paper analyzes the reasons of decline in model's performance after editing. They discuss both data and model aspects and provide various insights. Moreover, they suggested a method to enhance model's performance. The paper contributes to a large space of model editing methods and is a valuable contribution.

Reviews: review: paper_summary: paper_summary: This paper investigates the reasons and solutions for the decline in model performance of model editing.  The authors conduct experiments from two perspectives: data and model. Specifically, to clarify the impact of data on the performance of edited models, the authors first evaluate how editing different types of data affects model performance. Then, the authors construct a Multi-Question Dataset (MQD) and identified that the performance of  the edited models is primarily influenced by the diversity of the editing objectives  and the length of the tokens. Secondly, the authors explore the factors that affect model  performance from a model perspective. Experiments revealed a strong correlation between the L1 norm of the edited model layers and the editing accuracy, and  identified an editing quantity bottleneck. To enhance the performance of edited  models, the authors propose a Dump for sequence (D4C) method that effectively improves  the performance of edited models and overcomes the previous editing bottleneck issue. This method allows for multiple effective edits with minimal impact on  model performance.
strength_weakness: strength_weakness: The writing of this paper should be improved. There is no overview of this paper, which makes it hard to follow the details of Section 3 and 4.

The motivation of the proposed method  is not clear.

There are many typos such as line 182.

There are many missing references such as: 

Knowledge Editing for Large Language Models: A Survey

Stable Knowledge Editing in Large Language Models

A Comprehensive Study of Knowledge Editing for Large Language Models

Editing Large Language Models: Problems, Methods, and Opportunities
questions: questions: See weaknesses.
limitations: limitations: Yes

The limitations are on page ten. I am unsure if this counts as exceeding the page limit.
paper_summary: paper_summary: The paper investigates the reasons behind performance decline in sequential model editing approaches that selectively update parameters based on both data and model factors. To address the issues causing this decline, the authors propose a method to save editing history, thereby transforming sequential editing into batch editing with minimal computational overhead.
strength_weakness: strength_weakness: The study is restricted to two closely related editing approaches.

Experimentation is limited in demonstrating the efficacy of the D4C method. Different datasets and a larger number of edits for a more thorough evaluation are needed.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: Recent research has shown varying degrees of decline in model performance following small changes made by certain model editing methods. This paper is the first to comprehensively analyze the reasons behind such performance declines. Through extensive experiments, it identifies two main factors: data and model. For data-specific factors, the paper finds that perplexity and token length significantly influence performance. For model-specific factors, the L1 norm of the edited layer is identified as a key influence. Building upon these insights, the paper proposes a method named Dump for sequence (D4C), which significantly improves model performance.
strength_weakness: strength_weakness: - My main concern with the data-specific analysis is whether the conclusion is about correlation or causation. Many variables can be changed about the input data. Plotting a single Figure 3 might be insufficient to justify that perplexity and token length are the main reasons for the decline in model performance after editing.

- Unfortunately, the constructed dataset is not open-sourced. 

- Recent research [1] has shown that model editing methods (e.g. ROME, MEMIT) are not good at handling multi-hop questions, how would D4C perform in such more challenging scenarios?

- Some theoretical analysis can be conducted to demonstrate that D4C does not lead to an increase in norms.

[1] Mquake: Assessing knowledge editing in language models via multi-hop questions. EMNLP 2023
questions: questions: - Can the authors add a section in the appendix to expand on the dataset mentioned in 3.1 (i.e., provide examples and details about the editing objectives) for better readability?

- What dataset was employed in Section 5?

- I encourage the authors to release the full code to enhance reproducibility.

- (Minor) Consider reducing v-space in some parts of the paper (e.g., the bottom of page 2).
limitations: limitations: The limitations are discussed in the paper.
paper_summary: paper_summary: This paper addresses the challenges associated with the decline in performance of LLMs after undergoing knowledge editing. The study identifies the primary factors contributing to performance degradation from both data and model perspectives. By constructing a Multi-Question Dataset (MQD) and analyzing the impact of editing objectives, token length, and diversity, the paper finds that perplexity associated with editing objectives significantly affects model performance. From the model perspective, a strong correlation was observed between the L1 norm of parameter layers and editing accuracy. The paper proposes a novel method called Dump for sequence (D4C), which effectively manages the parameter growth and improves model performance post-editing.
strength_weakness: strength_weakness: - Generalizability of Findings: The study focuses on specific scenarios and datasets, which may limit the generalizability of the findings across different types of LLMs or editing tasks.
- Potential Overfitting to Edited Scenarios: There is a risk that the model may become overly optimized for the edited scenarios, potentially affecting its performance on unedited or unrelated tasks.
- Complexity of Implementation: The proposed D4C method, while effective, may be complex to implement and integrate into existing systems due to its sophisticated handling of parameter layers.
- Unsuitable Citation Format: The citations in this paper are in the format of “XXX et al. [YEAR]”, which are not suitable enough, and had better change into the format of [1], [2], [3], ……
questions: questions: - Adaptability of D4C Method: How adaptable is the D4C method to different types of LLMs and knowledge editing tasks beyond those tested in your experiments?
- Impact on Unedited Model Performance: How does the D4C method affect the performance of the model on tasks that have not been edited? Is there any evidence of performance trade-offs?
- Handling of Diverse Editing Objectives: Could you elaborate on how the D4C method manages the complexity and diversity of editing objectives without compromising the model’s overall integrity and coherence?

**Missing References**
- Editing Large Language Models: Problems, Methods, and Opportunities (EMNLP 2023)
- Knowledge Editing for Large Language Models: A Survey (2023)
- A Survey on Knowledge Editing of Neural Networks (2023)
- A Comprehensive Study of Knowledge Editing for Large Language Models (2024)
limitations: limitations: - Dependency on Specific Data Characteristics: The effectiveness of the proposed solutions may depend heavily on the characteristics of the data used for training and testing, which might not be consistent across different domains or applications.
- Evaluation Metrics: While the paper introduces new evaluation methods, the reliance on perplexity (PPL) and L1 norm metrics might not completely capture all aspects of model performance and health, especially in nuanced or context-dependent scenarios.
- Limited Experimentation: The experiments (Section 5) in this paper are too limited to demonstrate the conclusion. 
- Scope of Editing Objectives: The study might not fully capture the impact of highly diverse or complex editing objectives that could be encountered in real-world scenarios.

Comments: comment: Thank you for your reply. We promise to add additional experiments in the revised version.
Thank you for your reply. We promise to add additional experiments in the revised version.
Thank you for your detailed response, I have updated the score, the additional experiments should be added to the revised draft to show the efficacy of the approach.
Thank you for your reply. We promise to publicly release the dataset and complete code in the future.
Thank you for providing such thorough and compelling responses. Despite the existing critiques on knowledge editing, this paper offers the most intuitive and straightforward demonstration of performance degradation, along with effective solutions. I am confident that this work will significantly advance the field, and I will be increasing my score to further support it.

However, I would like to note that the current code provided in the supplementary materials is incomplete. For example, the MQD dataset that was constructed is not included. Ensuring the code's completeness and improving its visibility would likely enhance the impact of this work even further.
Thank you for your prompt response. Below, we elaborate on the key distinctions between our work and the referenced studies [1], highlighting our novel contributions and advantages.

* **Novelty:** Our research stands apart by pinpointing norm growth as the primary culprit behind model performance degradation, a revelation that precedes the publication of the V3 [1] version of the references. Notably, at the time of our NeurIPS 2024 submission deadline, only the V2 [2] version was available, which merely acknowledged a decline in model performance post-editing without delving into the underlying causes or proposing solutions. By contrast, we are the pioneers in identifying and addressing this critical issue.

* **Evaluation Methodology:** Even if the V3 version is subsequently released, our work retains significant advantages. Specifically, we identified a fundamental flaw in V3's evaluation approach, which relies solely on the original single edit (e.g. evaluating the current editing performance after editing one sample). This approach fails to align with the standard experimental setup for sequence editing, where the performance of all previously edited samples is assessed after multiple edits. Our Figure 7 visually demonstrates the limitations of V3's method and underscores its inability to validate the resolution of catastrophic forgetting.

* **Addressing Editing Bottlenecks:** Our paper goes beyond merely acknowledging the existence of editing bottlenecks, as exemplified by the 850-times limitation in Memit. We boldly extended our experiments to 10,000 edits, demonstrating the remarkable resilience of our model. In stark contrast, V3's experiments were confined to a mere 20 edits (as shown in Figure 7), indicating a lack of depth in exploring and addressing the editing bottleneck challenge. Furthermore, the disparity in evaluation methodologies and the limited scope of V3's edits hinder a fair performance comparison.

* **Mitigating Norm Growth:** Both our work and V3 recognize norm growth as a pivotal factor in performance degradation. However, we present compelling evidence in Figure 6 that our D4C method effectively curbs norm growth, thereby safeguarding model performance. In contrast, V3's regularization strategy, which involves setting certain parameters to zero, lacks empirical support for its ability to significantly alleviate norm growth issues.

Thank you again for your reply. If you have any remaining concerns or need further clarification, we welcome your additional input. Thank you for your continued consideration.

**Ref**

[1] V3 https://arxiv.org/pdf/2401.04700v3

[2] V2 https://arxiv.org/pdf/2401.04700v2
Thank you for your reply, my concern has been addressed. I have raised my score.
Dear Authors,

Thank you very much for the clarification! Most of them have addressed my concerns. 

I appreciate the response, especially the newly conducted experiments. Including those in the revised draft will strengthen the paper. 

Overall, I acknowledge the interesting idea of this work, and decide to increase my original scores 5 to 6.

Best Regards,

Reviewer Kq3N
I appreciate the authors' efforts in addressing my questions and am generally satisfied with the responses provided. However, I would like to request additional clarification regarding the connection with the paper [1], which already received 23 citations to date. This paper shares the same motivation as yours—addressing performance degradation—and also proposes regularization as a solution. While I see the references in the related work, the novelty appears somewhat overlapping. Could the authors further elaborate on how their work differentiates itself from this reference and justify the uniqueness of their approach in light of this existing work?

[1] https://arxiv.org/abs/2401.04700v3
Dear Senior Program Committee and Reviews,

We appreciate your initial feedback and have addressed your comments in our previous response. If you have any remaining concerns or need further clarification, we welcome your additional input. Thank you for your continued consideration.

Sincerely Yours

The Authors