The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse (2024)

Abstract: Although model editing has shown promise in revising knowledge in Large Language Models (LLMs), its impact on the inherent capabilities of LLMs is often overlooked. In this work, we reveal a critical phenomenon: even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. However, benchmarking LLMs after each edit, while necessary to prevent such collapses, is impractically timeconsuming and resource-intensive. To mitigate this, we propose using perplexity as a surrogate metric, validated by extensive experiments demonstrating changes in an edited model’s perplexity are strongly correlated with its downstream task performances. We further conduct an in-depth study on sequential editing, a practical setting for real-world scenarios, across various editing methods and LLMs, focusing on hard cases from our previous single edit studies. The results indicate that nearly all examined editing methods result in model collapse after only few edits. To facilitate further research, we have utilized GPT-3.5 to develop a new dataset, HardEdit, based on those hard cases. This dataset aims to establish the foundation for pioneering research in reliable model editing and the mechanisms underlying editinginduced model collapse. We hope this work can draw the community’s attention to the potential risks inherent in model editing practices1.

heading: 1 Introduction
text: Large language models (LLMs) (OpenAI et al., 2023; Touvron et al., 2023), once trained, face the risk of becoming obsolete due to the dynamic nature of world knowledge. This challenge has spurred interest in model editing (Yao et al., 2023), an emerging research area dedicated to efficiently updating model parameters to modify outdated or
†Corresponding author. 1Code and data released at https://github.com/Wanl
iYoung/Collapse-in-Model-Editing.
incorrect knowledge in models, thus avoiding the huge costs of retraining from scratch (Meng et al., 2022). Recently, model editing has advanced significantly and found applications in various domains, including question answering (QA) (Huang et al., 2023), hallucination correction (Hartvigsen et al., 2023), and model repair (Murty et al., 2022).
However, our pilot explorations reveal a critical and unexpected risk: even a single edit can cause model collapse. As shown in Figure 1a, employing ROME (Meng et al., 2022), a cutting-edge model editing method, to update GPT-J with only one fact led to a marked deterioration in its text generation capabilities. Moreover, Figure 1b highlights a significant decline in the performance of edited GPT-J on three representative tasks from its official evaluation task sets, approaching the level of random guessing on these tasks. Herein, we term the phenomenon of significant performance decline across various downstream tasks in the edited model as “model collapse”. This observation raises two critical questions for model editing:
• How can we efficiently identify or measure collapse in an edited language model?
• Is model collapse a common issue across different language models and editing methods?
5419
Although a thorough evaluation of edited models across downstream tasks for each edit offers a straightforward solution, the substantial time and resource consumption makes it impractical for realworld usage. To streamline it, we propose using perplexity to evaluate model collapse during editing and verify its efficacy in indicating downstream task performances through extensive experiments. Furthermore, to ensure the reliability of perplexity computations, we curate a diverse and high-quality dataset ME-PPL (Model Editing-Perplexity) from various commonly used corpora.
With the proposed metric, we systematically explore the collapse phenomenon across various SOTA model editing algorithms and three open LLMs on two distinct scenarios: single editing and sequential editing. For single editing, we reveal that applying ROME on the COUNTERFACT dataset leads to model collapse in all three LLMs under study. Consequently, we gather samples that triggered model collapse in single edit trials, constituting the HardCF dataset, to streamline subsequent studies by focusing on the most problematic instances. For sequential editing, a practical setting in real-world applications, we observe that model collapse occurs prevalently across almost all combinations of editing methods and LLMs we studied, within just dozens of edits on HardCF. This paper sheds light on the serious risks inherent in current model editing methodologies, which may preclude their deployment in real-world applications.
Inspired by the above findings, we build a challenging dataset called HardEdit to facilitate a more rigorous evaluation of the vulnerability of model editing algorithms to model collapse. To populate this dataset with challenging examples, we utilize GPT-3.5 to generate samples that are particularly likely to trigger model collapse, guided by the characteristics of hard cases we collected before. Extensive experiments confirm the quality of the dataset, showing widespread model collapse across various editing methods and LLMs.
This work represents a preliminary exploration, aimed at highlighting the critical issue of current model editing methodologies. Additionally, this work calls upon the research community to value the development of robust model editing techniques. Our main contributions are as follows.
• We unveil a hitherto unknown yet critical issue: a single edit can trigger model collapse.
• We propose to use perplexity for assessing the
general capabilities of LLMs in model editing.
• We demonstrate that model collapse is a ubiquitous issue for current editing algorithms in sequential edit setting via extensive experiments.
• We employ GPT-3.5 to construct a rigorous dataset HardEdit for enabling a comprehensive evaluation of model editing techniques, promoting further research and progress in the field.

heading: 2 Background & Study Formulation

heading: 2.1 Model Editing
text: Model editing aims to modify a model’s behavior on specific facts by directly adjusting its parameters instead of retraining, while preserving its behavior on irrelevant cases. Formally, given an original fact t=(s, r, o), consisting of subject s, relation r, and object o, encoded in an LLM fθ and a revised fact t′ = (s, r, o′) where o′ ̸= o, the objective of the editing algorithm ξ is to optimize the parameter θ into θ′ so that the edited model fθ′ : fθ′=ξ(fθ, t
′) correctly produces o′ when provided with the prompt p(s, r), as fθ′(p(s, r)) = o′. Using a presidential transition as an example, for the subject s= United States and relation r= president of, the editing algorithm ξ ensures that the edited model fθ′ produces the expected object o′= Joe Biden, instead of previous o= Donald Trump, with prompt p(s, r) = The president of the United States is.

heading: 2.2 Current Methodologies
text: Existing model editing methods can be broadly categorized into three groups. Fine-tuning. This intuitive paradigm mainly utilizes layer-wise fine-tuning to adjust parameters in light of new examples, simultaneously incorporating a constraint to ensure minimal interference with unmodified facts. Typically, Zhu et al. (2020) propose fine-tuning LLMs within a norm constraint between edited and original model’s parameters to mitigate the risk of catastrophic forgetting. Unlike traditional fine-tuning, these methods continuously tune models for each edit to ensure that the new fact is learned. Meta Learning. Leveraging meta learning principles, this category of methods (De Cao et al., 2021; Mitchell et al., 2022; Tan et al., 2023) usually employs a hypernetwork, serving as a helper model, to directly predict effective gradients or parameter modifications for encoding new facts.
De Cao et al. (2021) utilities a trained hypernetwork (a bidirectional-LSTM) to predict the parameters modification for each edit request. Mitchell et al. (2022) employs hypernetworks to learn a lowrank decomposition of the fine-tuning gradients to modify LLMs for new facts. Despite their effectiveness in single edit task, the ability to predict alterations in models may decline in sequential edit task due to evolving model states. Locate-then-Edit. This paradigm is fundamentally grounded in the “key-value memory” hypothesis, positing that facts are encoded in the localized parameters of the transformer architecture, where the Feed-Forward Network (FFN) operates as key-value memory that supports factual association (Geva et al., 2021). Based on this, existing approaches attempt to localize target knowledge in specific parameters of models, and update these to inject new knowledge. KN (Dai et al., 2022) employ knowledge attribution to identify the “knowledge neuron” (a key-value pair of FFN) which encodes certain knowledge, and then update the knowledge by modifying the neuron. ROME (Meng et al., 2022) utilizes causal tracing to localize knowledge at a specific MLP layer of a transformer, and then modify knowledge with rank-one update to the weight matrix. MEMIT (Meng et al., 2023) extends ROME by applying updates across multiple MLP layers, realizing massive edits.

heading: 2.3 Evaluation of Edited Models
text: The edited model fθ′ is typically evaluated from four properties: i) reliability, measuring the success rate of the edit; ii) generalization, evaluating the model’s performance on equivalent edit prompts; iii) locality, examining the impact of the edit on irrelevant knowledge; iv) portability, assessing the model’s performance on factual reasoning related to the editing request. Interested readers are directed to Yao et al. (2023) for an in-depth exploration. Additionally, Hoelscher-Obermaier et al. (2023) claim a limitation in the currently used specificity (i.e., locality) metric, which focuses only on model responses to given prompts, and propose using KL divergence to measure changes in the full probability distribution of model outputs.

heading: 2.4 Side Effects of Model Editing
text: Despite promising early results, the potential side effects of model editing have progressively garnered research interest as well. Yao et al. (2023) demonstrate that model editing algorithms may
influence other relations associated with the subjects of edits, with the impact of FTℓ∞ (Zhu et al., 2020) being particularly pronounced. HoelscherObermaier et al. (2023) find that incorporating text relevant to edit cases into unrelated prompts can cause the responses of edited models to shift toward the target of the edits, which reveals that the models are over edited. Brown et al. (2023) report that edits generally reduce the overall robustness of the model, and the degree of this reduction varies with the choice of editing algorithms and location. Existing explorations of side effects primarily concentrate on the non-robust behaviors of model associated with editing.

heading: 2.5 Research Question
text: In this paper, we argue that for model editing to be practically useful, it is essential to ensure that the edited model maintains its abilities in downstream tasks. Thus, we are interested in the following questions: • Can current model editing methods retain LLMs’
inherent capabilities in downstream tasks? • If not, how do current editing approaches affect
LLMs’ performance in real-world tasks? • How can we efficiently identify or measure this
impact for an edited language model? These are the main focus of our study, which will be discussed in § 4, § 5, and § 6.

heading: 3 Experimental Setup
text: This section outlines the basic setup of our study, serving as the default framework for all subsequent experiments unless otherwise noted.

heading: 3.1 Editing Methods, Datasets, & LLMs
text: Editing Methods. For a comprehensive experimental scope, we employ four diverse and representative model editing methods from the three aforementioned categories: fine-tuning (FTℓ∞ , Zhu et al., 2020), meta-learning (MEND, Mitchell et al., 2022), and locate-then-edit (ROME, Meng et al., 2022 and MEMIT, Meng et al., 2023). All these methods are implemented using EasyEdit2. For the training-required method, MEND, the split of datasets follows the common practice as in (De Cao et al., 2021; Mitchell et al., 2022). Editing Datasets. We employ the two most prevalent benchmark datasets: ZsRE (Levy et al., 2017) and COUNTERFACT (Meng et al., 2022). For
2https://github.com/zjunlp/EasyEdit
ZsRE, we adopt the established data split from (Meng et al., 2022; Yao et al., 2023), using the test set (10,000 records) for our study. Backbone LLMs. Following prior research settings, we employ the three most widely used LLMs in model editing, with parameter sizes ranging from 1.5 to 7 billion to reflect a diverse set of capabilities: GPT-2-XL (1.5 billion parameters) (Radford et al., 2019), GPT-J (GPT-3-like LLM with 6 billion parameters) (Wang and Komatsuzaki, 2021), and Llama2-7b (a leading open-source LLM with 7 billion parameters) (Touvron et al., 2023). For all the LLMs under investigation, greedy decoding is consistently adopted during text generation and downstream task evaluation.

heading: 3.2 Representative Tasks
text: To assess the overall capabilities of the edited models, we choose six representative tasks from the collective set of official evaluation benchmarks for the LLMs under study. Our evaluation encompasses two categories, each with three tasks, to probe distinct capabilities of the model: Hellaswag (Zellers et al., 2019), PIQA (Bisk et al., 2020), and MMLU (Hendrycks et al., 2021) for discriminative abilities; and LAMBADA (Paperno et al., 2016), Natural Questions (NQ) (Kwiatkowski et al., 2019), and SQuAD2.0 (Rajpurkar et al., 2018) for generative capacities. Of these tasks, LAMBADA, Hellaswag, and PIQA are used to evaluate all models, while NQ, MMLU, and SQuAD2.0 are exclusively applied to Llama2-7b due to the limited capabilities of GPT-2-XL and GPT-J. For efficiency, we select 4 out of the 57 subtasks of MMLU to form MMLUsub , which effectively represents its core categories, for subsequent study. Evaluation of these tasks is performed using lm-eval package3.
Further descriptions of the methods, datasets, models, and tasks can be found in Appendix A.1.

heading: 4 Pilot Observation
text: This section introduces the motivation of our research, a pilot exploration to elucidate the side effects of model editing on LLMs.
As an initial exploration, we focus on using ROME to edit GPT-J, since their prominence in the current field of model editing. To address the excessive time and resource demands of benchmarking models after each edit, we opt to quickly identify a
3https://github.com/EleutherAI/lm-evaluation -harness
small set of anomalous models produced by each edit, facilitating subsequent investigation. Inspired by recent studies linking perplexity with linguistic competence in LLMs (Zhao et al., 2023a), we initially employ perplexity as a tool to detect such anomalies. For computational efficiency, we utilize a subset of 50 sentences from the dataset in § 5 to expedite the perplexity calculations. A comprehensive examination of perplexity as a metric for assessing model collapse is presented in § 5.
Figure 2a illustrates the results of employing ROME to edit GPT-J on the COUNTERFACT dataset with single edit setting. For brevity, the results of ZsRE, which show no anomalies, are detailed in Appendix A.2. Each point in the figure represents the perplexity of a model edited independently from the original GPT-J, using a unique sample from the COUNTERFACT dataset. Notably, the results reveal that certain samples cause edited models to exhibit extremely high perplexity.
To understand what occurred in these cases, we chose the edited models with top 30 highest perplexity in Figure 2a, and evaluated their performances on the discrimination tasks (PIQA and Hellaswag) and the generation task (LAMBADA). All the models’ performances markedly decline on these downstream tasks as shown in Figure 2b. A subsequent basic text generation test with a high perplexity model confirmed the severity of the issue: the model lost its ability to generate coherent text, generating meaningless content instead, as shown in Figure 1a.
Arising from this preliminary investigation, we uncover a previously unreported phenomenon that model editing can precipitate model collapse. Naturally, this finding leads to two key questions:
• Can perplexity effectively signal collapses in edited models, i.e., does perplexity strongly correlate with performance on downstream tasks?
• Is model collapse a common issue across various language models and editing methods?

heading: 5 Perplexity as a Surrogate Metric
text: In this section, we conduct an in-depth investigation to assess whether perplexity can serve as a surrogate metric, closely correlating with downstream tasks performance, thereby avoiding the need for costly benchmarking LLMs after each edit.
Perplexity (Brown et al., 1992) is a conventional metric for measuring the generative capability of language models, defined as the exponential of the average negative log-likelihood of a sequence. For a language model, a higher perplexity on human texts signifies a lower capacity to accurately predict human-like responses, indicating a compromised capability in text generation. Furthermore, from a theoretical perspective, perplexity’s exponential relationship with the training loss of LLMs (Radford et al., 2018) establishes it as a surrogate metric for assessing the status of the model. Dataset. Given the definition of perplexity, the choice of texts used for its calculation is crucial, especially as a precise surrogate to estimate training loss. Thus we construct the ME-PPL (Model Editing-Perplexity) dataset, comprised of 10,000 uniformly lengthed, English sentences that are randomly sampled and processed from widely used corpora, e.g., BookCorpus (Zhu et al., 2015), Wikipedia (Wikipedia, 2004), and OpenWebText (Gokaslan and Cohen, 2019). To facilitate perplexity calculation in various situations, e.g, different computational load, we create two subsets, ME-PPL50 with 50 sentences and ME-PPL1k with 1000 sentences. More details can be seen in Appendix A.3. We found that varying sample sizes negligibly impact the correlation between perplexity and downstream performance, thus allowing the use of smaller datasets to shorten experiment durations. In this section, we adopt ME-PPL1k for a more precise investigation. Experimental Setup. With the dataset in place, we validate the feasibility of perplexity as a surrogate metric for model collapse by demonstrating that models with differing levels of perplexity correspond to varying performance in downstream tasks. For this purpose, we apply model editing to establish a comprehensive range of perplexity
levels, including twenty points distributed as uniformly as possible between the perplexity of original model and a threshold of 1000, along with three additional points beyond this (specifically, 5×103, 1×104, and 5×104) to represent collapsed models. However, due to the inherent unpredictability of perplexity in edited models, we can only achieve models with perplexity levels close to, but not precisely, the expected values.
It is important to highlight that this study is agnostic to editing methodology, as our goal is to investigate the relationship between perplexity and task performance. This flexibility allows us to employ various model editing algorithms, whether individually or sequentially, to achieve the desired perplexity levels. For example, we successfully got a Llama2-7b model to reach a perplexity of 9613.17 (roughly 10,000) by applying a single edit via ROME. Conversely, by sequentially applying FTℓ∞ 18 times, we obtained a Llama2-7b model with a perplexity of 97.25 (around 100). Finally, we obtained models with 23 distinct perplexity variations for each of the three models and subsequently evaluated these edited models on the tasks introduced in § 3. Results. The results in Figure 3 reveal a significant correlation between the perplexity of LLMs and their performance on downstream tasks. Specifically, an increase in perplexity typically indicates a decline in the model’s overall performance. It is noteworthy that the lower ρ values for NQ and
SQuAD2.0 are attributed to the premature decline in task performance to a level of random guessing. Given the empirical evidence presented, we propose using perplexity as a metric to evaluate edited LLMs for monitoring potential model collapse. It is essential to emphasize that our intention is to employ perplexity to monitor the drastic change in an edited model, rather than as a precise measure of comparative capabilities across various LLMs as in Hu et al. (2024).

heading: 5.1 Discussion
text: Additionally, there are other metrics designed to assess the side effects of model editing, e.g., locality (Yao et al., 2023), as well as consistency and fluency (Meng et al., 2022). However, these metrics are insufficiently effective, especially in detecting model collapse. In this section, we discuss the connections and differences between perplexity and these metrics. Locality. It evaluates the side effects of editing algorithms by examining whether the edited model changes its outputs on randomly sampled, irrelevant questions (Meng et al., 2022; Yao et al., 2023). However, it often falls short as a comprehensive evaluation metric due to its limitations: insufficient sampling volume to cover all potential out-of-scope scenarios and the trivial nature of the employed token completion task that fails to capture the full range of LLM functionalities. Table 1 highlights the inconsistency of locality in practical usage, indicating model collapse at a value of 1 and stability at 0, which contradicts actual model performance. Consistency and Fluency. Meng et al. (2022) assess the generative capabilities of edited models through consistency and fluency. Consistency measures the cosine similarity between the generated texts and given reference texts while fluency focuses on identifying repetitive word patterns via bi- and tri-gram entropies. However, a collapsed model may still produce texts with low repetition,
while texts generated by a stable model might significantly diverge from the reference texts, as shown in Table 1. This reveals the inadequacies of consistency and fluency as indicators of a model’s generative capabilities.
The failure of previous works to identify model collapse may stem from their evaluation on sampled test data or basing their analysis on average metrics, resulting in the oversight of a small fraction of collapsed samples.

heading: 6 Model Collapse Induced by Editing
text: This section is dedicated to using perplexity to systematically investigate collapse induced by model editing in single and sequential editing scenarios.

heading: 6.1 Single Editing
text: Single editing is the fundamental and prevalent experiment setting in model editing research. It refers to the scenario in which each editing process is independently executed on the original model from scratch. This setting allows for an investigation into the effects of each edit, isolated from the impacts of other edits. Experiment Setup. We conduct experiments using four editing methods4 on three LLMs across two datasets, as detailed in § 3. Given the significant time for 24 (3×4×2) different experimental setups, each requires tens of thousands of evaluations, we opted for ME-PPL50 to accelerate perplexity calculation. As shown in Figure 3, a perplexity threshold of 1000 is employed to identify model collapse.

heading: 6.1.1 Results & Analysis
text: Upon examining the perplexity, we find that model collapse caused by a single edit exists in all three LLMs when applying ROME to COUNTERFACT. Due to space limitations, we present the perplexity results for various experimental settings in Appendix A.5. Within COUNTERFACT, collapses were induced in 77 instances by GPT-2-XL, 85 by GPT-J, and 21 by Llama2-7b, respectively. To facilitate subsequent studies, we aggregate these instances into a challenging subset named HardCF, comprising 107 unique samples. Characteristics of HardCF. Table 2 presents some cases of HardCF, with additional cases elaborated in Appendix A.7. For GPT-2-XL and GPT-J, the
4For the less effective editing method, KN, the results of single editing are provided in Appendix A.4, highlighting the frequent occurrence of edited model collapse.
samples causing model collapse exhibit a high degree of overlap, primarily featuring subjects that are single, commonly used words, and are positioned at the beginning of the prompts. For Llama27b, the subjects in these challenging cases usually encompass names of individuals or entities, ending with a period “.”.
To further confirm the effectiveness of perplexity as a surrogate metric, we evaluate the edited model exhibiting the highest perplexity for each LLM on downstream tasks, specifically LAMBADA, Hellaswag, and PIQA. Table 3 demonstrates that these models are severely damaged, further supporting the finding that a single edit can disrupt LLMs.
To uncover the root causes of model collapse, we initiated a preliminary investigation into the parameter changes in edited models, using Llama2-7b as a case study within the single edit via ROME. We selected an edited model with the highest perplexity of 7751.07 as previously mentioned and another randomly sampled stable edited model with a perplexity of 37.25, for comparison. Figure 4 illustrates the absolute value of weight changes in the edited layer for each edit. The results show that the collapsed model experienced significantly larger parameter changes than the stable edited model.

heading: 6.2 Sequential Editing
text: Unlike single editing, which focuses on the impact of an individual edit, sequential editing is essential for the continuous knowledge updates in real-world applications. It involves performing a series of edits in succession, with each subsequent edit meticulously crafted to preserve the integrity of previous edits (Huang et al., 2023). Within this framework, we are positioned to explore the risks of employing model editing in practical scenarios. Experiment Setup. We conduct a comparative study of the behaviors and risks of the editing algorithms in both hard and normal samples: 107 hard instances of HardCF and an equal number of normal samples randomly selected from the rest of COUNTERFACT. We then execute sequential edits on each group separately, encompassing four editing algorithms and three LLMs5 as in single edit experiments. Notably, in light of the relatively small number of edits required for this experiment, the corpus for perplexity computation is expanded to ME-PPL1k for more precision.

heading: 6.2.1 Results & Analysis
text: The results of the sequential editing evaluation across various editing methods and LLMs are presented in Figure 5. It can be observed that:
Nearly all editing methods caused model collapse during sequential editing on hard data, with the collapse occurring in remarkably few times— less than 60. The exception within this study was MEMIT applied to GPT-2-XL, and FTℓ∞ to GPTJ. Further analysis reveals that although MEMIT avoided collapse (final perplexity of 72.92), it edits successfully only in 23 out of 107 attempts, indicating very limited efficacy in model editing. While FTℓ∞ did not induce total collapse in GPT-J, it significantly increased perplexity exceeding fivefold (from 50.34 to 268.61) and impaired downstream task performance according to Figure 3.
5Experiments on Mistral-7b are also conducted, exhibiting phenomena akin to those of the three LLMs. The results are detailed in Appendix A.6.
Another observation is the two distinct patterns in the four editing methods when applied to hard versus normal samples: i) FTℓ∞ and MEND behave similarly on both hard and normal samples, leading to their failure under each condition. ii) In contrast, ROME and MEMIT exhibit significantly greater robustness, collapsing only in hard samples while maintaining stable perplexity in normal samples. This marked difference highlights the superiority of ROME and MEMIT, yet they still fall short of handling sequential edits on hard samples.
Lastly, we select Llama2-7b to evaluate the impacts of the four editing methods. Specifically, we assess the performance of eight Llama2-7b variations, each was sequentially edited by one of the
four methods for hard or normal cases, in downstream tasks. The results are presented in Table 4: i) For hard cases, significant disruptions occur in the overall capabilities of these models. ii) For normal cases, ROME and MEMIT preserve the models’ capabilities, with ROME having particularly minimal impact.
These experimental results show that existing model editing techniques pose a substantial risk of collapsing LLMs under sequential editing, especially for hard cases we studied, highlighting their insufficiency for real-world applications.

heading: 7 HardEdit: A Challenging Dataset
text: To further facilitate comprehensive evaluations of future advanced methods, we crafted a challenging dataset, termed HardEdit, by utilizing GPT-3.5 to generate samples based on the patterns derived from the HardCF subset. Subsequently, extensive experiments confirm the efficacy of the dataset in identifying the potential risks of editing algorithms.

heading: 7.1 Dataset Construction
text: This subsection elaborates on the construction of our dataset. Like existing datasets, our dataset also employs the tuple (subject, relation, object) to express the fact associations. To ensure the quality
of our dataset, i.e., its capacity to induce model collapse upon editing, we tailor our samples to reflect the characteristics identified from the HardCF dataset, as discussed in § 6.1.1. Specifically, we adhere to the following principal criteria: i) Each subject is a widely used word and positioned at the beginning of the prompt; ii) Each sample represents a counterfactual statement to edit, thus preventing LLMs know the knowledge before editing. With these guidelines in place, GPT-3.5 is employed for edit sample generation.
Generating counterfactual edit samples with GPT-3.5 is relatively straightforward, with the complete prompt detailed in Appendix A.8. The prompt primarily encompasses the data requirements and examples from HardCF. To avoid subject repetition and ensure dataset diversity, we used GPT-3.5 to initially construct a diverse set of around 400 unique, single-word subjects, identifying the most prominent ones across various fields, e.g., scientist, artist, city, and country. Then, ten subjects are randomly chosen from the set to constitute the input prompt and thus aid the generative process each time, as detailed in Appendix A.9.
After filtering duplicates, we obtain a dataset with 1392 unique samples. To ensure the effectiveness of these generated samples in uncovering model collapse induced by editing algorithms, we employ ROME to perform single editing on GPT-2XL with these samples and evaluate their effectiveness using ME-PPL50. By filtering for perplexity exceeding 1000, we produce the HardEdit dataset, containing 469 samples.

heading: 7.2 Dataset Validation
text: To validate the efficacy of HardEdit, we conduct sequential editing experiments on it and calculate the perplexity after each edit using ME-PPL1k. The results in Figure 6 illustrate that nearly all the examined LLMs are significantly damaged: i) Only
one exception occurs, akin to § 6.2.1, where editing GPT-2-XL with MEMIT resulted in the highest perplexity of 545.22. However, its editing success rate is only around 1.28%, highlighting the significant challenge posed by these samples to MEMIT. ii) Due to the increased number of hard samples, the FTℓ∞-edited GPT-J, which shows a modest increase in perplexity to 268.61 on HardCF, suffers a severe collapse on HardEdit, with perplexity escalating to 2109.35. The results confirm the utility of HardEdit in exposing the potential risks of editing, which could precipitate model collapse.

heading: 8 Conclusion and Future Work
text: In this paper, we employ perplexity as a surrogate metric to investigate the impact of model editing on the downstream task performance of LLMs, revealing a critical issue: the advanced model editing method, ROME, can cause LLMs collapse with just a single edit. Subsequent experiments demonstrate that model collapse is a common issue among current mainstream model editing methods under sequential editing. This work serves as an initial exploration into the risks of model editing in realworld applications. Distinct from contemporaneous works (Gu et al., 2024; Gupta et al., 2024) investigating impact of large-scale edits on models, we focus on exploring the possibility of model collapse caused by a small number of edits and how to efficiently detect potential collapses in practical applications. Additionally, to advance model editing research, we develop a challenging benchmark, HardEdit, based on the identified pattern.
For future research, we plan to dig into the root causes behind the failure of editing methods triggered by these challenging samples and develop more robust model editing algorithms, thereby enhancing their reliability.

heading: Limitations
text: We acknowledge following limitations of our work: • This paper presents an initial exploration into
the potential risks associated with model editing. However, it does not delve into the root causes behind the drastic parameter modifications resulting from model editing methods applied to specific facts. Due to space limitation, this analysis exceeds the scope of this paper and is reserved for future work.
• Similarly, we do not propose a solution to address model collapse caused by model editing. It is left for future research as well.
• Due to computational resource limitations, we are unable to conduct experiments on additional LLMs, such as Llama2-13b, or explore more model editing algorithms.
• Currently, the HardEdit dataset is limited in size. Using LLMs to generate high-quality edit samples for continuously expanding the dataset is an important future direction.

heading: Ethics Statement
text: Data. All data used in this research are publicly accessible and do not raise privacy issues.
AI Writing Assistance. We use ChatGPT to polish our original content, with a focus on correcting grammatical errors and enhancing clarity, rather than generating new content or ideas.

heading: Acknowledgements
text: This work was supported by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103704), the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0680202), and the Innovation Funding of ICT, CAS (E361120).

heading: A Appendix

heading: A.1 Detailed Experimental Setup

heading: A.1.1 Editing Methods
text: FTℓ∞ (Zhu et al., 2020) applies a ℓ∞ norm constraint on the fine-tuning loss, limiting the difference between the original and edited model’s parameters, to reduce side effects. MEND (Mitchell et al., 2022) employs an ensemble of small hypernetworks to learn a rank-one decomposition of the gradient obtained by standard fine-tuning, enabling tractable edits in LLMs. ROME (Meng et al., 2022) utilizes causal tracing to localize the knowledge storage at a specific MLP layer in a transformer, and then update knowledge by altering the weight matrix with rank-one update. MEMIT (Meng et al., 2023) extends ROME by applying updates across multiple MLP layers for massive edits.

heading: A.1.2 Editing Datasets
text: ZsRE (Levy et al., 2017) is a widely adopted Question Answering (QA) datasets, where each data entry comprises a counterfactual statement to edit, derived from a factual statement on Wikipedia. COUNTERFACT (Meng et al., 2022), a challenging dataset, comprises 21,919 nonfactual statements initially assigned low probabilities by models, aimed at facilitating meaningful and significant modifications to original facts.

heading: A.1.3 Backbone LLMs
text: GPT-2-XL (Radford et al., 2019) is the 1.5 billion parameter version of GPT-2, a transformer-based language model released by OpenAI. GPT-J (Wang and Komatsuzaki, 2021), developed by EleutherAI, is a GPT-3-like open-source LLM with 6 billion parameters, trained on The Pile. Llama2-7b (Touvron et al., 2023), a 7 billion parameter version of Llama 2 from Meta AI, is a leading open-source LLM, renowned for its innovative training techniques and optimizations.

heading: A.1.4 Representative Tasks
text: LAMBADA (Paperno et al., 2016), a benchmark designed to evaluate the ability of language models to predict the final word of a sentence, emphasizing the models’ capacity to grasp long-range dependencies within the text. Consequently, the lowest accuracy score on this benchmark is 0%. Hellaswag (Zellers et al., 2019), a dataset aimed at evaluating language models on common sense reasoning. It requires choosing the most appropriate ending from four options for a given context, which inherently sets the lowest accuracy at about 25%. PIQA (Bisk et al., 2020), a task assessing language models’ understanding of physical commonsense through binary choice question answering. This format results in the worst accuracy of approximately 50%. Natural Questions (NQ) (Kwiatkowski et al., 2019) is an open domain question answering benchmark based on the contents of English Wikipedia. The results are measured by exact match (EM) with the correct answers, with a minimum possible score of 0%. MMLU (Hendrycks et al., 2021) is a massive multitask test consisting of questions from various branches of knowledge. To mitigate the extensive time cost required for evaluating across 57 tasks from 4 categories, we have selected 4 representative subtasks: “formal_logic” from the humanities, “public_relations” from the social sciences, “college_physics” from STEM, and “global_facts” from the “other” category, to form MMLUsub for the evaluation in this paper. The lowest accuracy of these four-choice tasks is 25%. SQuAD2.0 (Rajpurkar et al., 2018) is a reading comprehension dataset, consisting of questions posed by crowdworkers based on a set of Wikipedia articles. The results are measured by F1 Score with correct answers.

heading: A.2 Perplexity Result of ZsRE
text: Perplexity values of editing GPT-J with ROME on ZsRE are depicted on Figure 7.

heading: A.3 Details about ME-PPL
text: ME-PPL (Model Editing-Perplexity) is a corpus designed for the perplexity computation of LLMs in the context of model editing.
The creation of this dataset involves four steps:
(i) Randomly select texts from popular corpora: BookCorpus (Zhu et al., 2015), C4 (Raffel et al., 2020), CC_News (Liu et al., 2019), Gutenberg (Kim et al., 2020), OpenWebText (Gokaslan and Cohen, 2019), Roots (Laurençon et al., 2022), and Wikipedia (Wikipedia, 2004), the proportion of each following that typically used in LLM pre-training (Zhao et al., 2023b).
(ii) Split these texts into units of sentence.
(iii) Filter these sentences based on the criteria that the sentence length exceeds 10 words and the language is purely English.
(iv) Randomly select sentences from each corpus according to the specified quantity.
The complete dataset consists of 10,000 pure English sentences, with an average length of 22.64 words. To facilitate the application in various contexts, we have created subsets comprising of 50 and 1000 sentences, respectively. The statistics of these datasets are provided in Table 5. Meanwhile, we present some representative samples of the dataset in Figure 12.

heading: A.4 Results of Single Editing for KN
text: We present the performance of KN in Figure 8, where it applies a single edit to three LLMs in Section 3 on the first thousand samples of the COUNERFACT dataset. The perplexity values of the edited models are calculated based on MEPPL50. The results indicate that KN frequently leads to the collapse of the edited models, underscoring its insufficient effectiveness.

heading: A.5 Complete Perplexity Results of Single Editing
text: The complete perplexity results of single editing experiments, using four editing methods on three LLMs across two datasets, are presented in Figure 11. These experiments take around 43 days on one A100 GPU.

heading: A.6 Results of Sequential Editing for Mistral
text: We employed the four methods in Section 3 to perform sequential editing on Mistral-7b for both normal and hard cases. The results presented in Figure 9 demonstrate that the phenomena on Mistral7b align consistently with those of the other three LLMs examined in Section 6.2.

heading: A.7 More Hard Cases in COUNERFACT
text: Figure 14 presents more hard cases from COUNTERFACT, each can induce corresponding LLMs to collapse via a single edit by ROME.

heading: A.8 Complete Prompt for Data Generation
text: The complete prompt used for generating data in the HardCF dataset can be viewed in Figure 15.
Specifically, the prompt comprises four distinct parts:
(i) Task Description and Data Illustration: Here, we preliminarily propose the requirements for hard data, as discussed previously.
(ii) Hard Data Examples: To enhance GPT-3.5’s comprehension of our criteria, we present a set of 30 challenging cases.
(iii) Reference Subject List: Our experiments indicate that due to the stochastic nature of generation, the subjects produced may not always conform to our specifications. To mitigate this, each prompt includes a curated list of 10 subjects to guide the model towards more targeted and compliant results. Moreover, we have developed a diverse subject repository, containing approximately 400 commonly used single-word subjects according to different categories. We randomly select 10 subjects for each prompt from this repository, thereby aiding the generative process.
(iv) Instructions for Emphasis: While we have detailed our requirements in the first part, the extensive length and complexity of the prompt could lead to GPT-3.5 overlooking critical details. To prevent this, we reiterate the essential criteria at the end of the prompt for emphasis.
We present a portion of our subject repository in Figure 13, which serves as the reference subject list in the prompt.

heading: A.9 Generated Data
text: In Figure 10, we present some samples of HardEdit.

heading: Future Work
text: Due to space limitation, this analysis exceeds the scope of this paper and is reserved for future work. • Similarly, we do not propose a solution to address model collapse caused by model editing. It is left for future research as well. • Due to computational resource limitations, we are unable to conduct experiments on additional LLMs, such as Llama2-13b, or explore more model editing algorithms. • Currently, the HardEdit dataset is limited in size. Using LLMs to generate high-quality edit samples for continuously expanding the dataset is an important future direction