Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models (2024)

Abstract: Warning: This paper contains several unethical and sensitive statements. In the rapidly advancing field of artificial intelligence, the concept of ‘Red-Teaming’ or ‘Jailbreaking’ large language models (LLMs) has emerged as a crucial area of study. This approach is especially significant in terms of assessing and enhancing the safety and robustness of these models. This paper investigates the intricate consequences of such modifications through model editing, uncovering a complex relationship between enhancing model accuracy and preserving its ethical integrity. Our in-depth analysis reveals a striking paradox: while injecting accurate information is crucial for model reliability, it can paradoxically destabilize the model’s foundational framework, resulting in unpredictable and potentially unsafe behaviors. Additionally, we propose a benchmark dataset NICHEHAZARDQA to investigate this unsafe behavior both within the same and cross topical domain. This aspect of our research sheds light on how the edits, impact the model’s safety metrics and guardrails. Our findings show that model editing serves as a cost-effective tool for topical red-teaming by methodically applying targeted edits and evaluating the resultant model behavior 1.

heading: 1 Introduction
text: Large Language Models (LLMs) such as LLaMa (Touvron et al., 2023) and GPT (Radford et al., 2018) are pivotal for their ability to understand and generate text akin to human communication (Naveed et al., 2023). However, these models face challenges in maintaining accuracy and relevance due to the dynamic nature of world knowledge. Without frequent updates, LLMs can become outdated, leading to factual inconsistencies or logical errors. Updating these
1Datasets and codes are available at the respective hyperlinks.
models is complex, as it’s not feasible to adjust billions of parameters for every new fact 2. Recent approaches including fine-tuning strategies help incorporate external corrections 3. These methods aim to balance the need for accuracy with the immense complexity of LLMs, ensuring they remain useful and relevant over time (Guo et al., 2023). The concept of Knowledge Editing in LLMs becomes increasingly important to ensure these models remain up-to-date and accurate with fewer parameters altered. This involves modifying pre-trained LLMs to encode specific, updated knowledge, while still maintaining their existing knowledge base and performance with unrelated inputs (Wang et al., 2023a). Editing is crucial in an environment where LLMs do not update automatically, as it ensures their continued relevance and accuracy. Knowledge Editing encompasses strategies like external memorization, which uses external data to augment the LLM’s knowledge; global optimization, which involves fine-tuning
2https://www.invonto.com/insights/large-languagemodels-explained/
3https://hai.stanford.edu/news/how-do-we-fix-andupdate-large-language-models
16227
the entire model with updated data; and local modification, targeting specific model segments for updates (Zhang et al., 2024). Although this process underscores the sensitivity and complexity involved in these sophisticated LLMs. Crucially, the effectiveness of these strategies is now being rigorously evaluated. Researchers are assessing model editing performance by checking consistency, as detailed in the (Hoelscher-Obermaier et al., 2023), and by evaluating overall performance on benchmark datasets, as explored in the paper (Gu et al., 2024). Researchers also find unwanted side effects of model editing techniques in terms of the specificity metric (Hoelscher-Obermaier et al., 2023). These evaluations show that although Knowledge Editing adds accurate and current knowledge it also introduces unwanted effects (See Figure 1).
The process of knowledge editing in LLMs significantly impacts model safety (Yao et al., 2023), highlighting two primary concerns: Knowledge Conflict and Knowledge Distortion. Knowledge Conflict occurs when multiple edits interfere with each other, especially if they are logically connected, leading to inconsistencies in the model’s knowledge base. Such conflicts are challenging to resolve and can compromise the logical consistency of the model. On the other hand, Knowledge Distortion arises when edits to factual knowledge fundamentally alter (with incorrect factual information) the model’s inherent knowledge structure. This can lead to the generation of inaccurate or misleading information, especially if the edits interact in complex ways with the existing knowledge base.
In this work, to the best of our knowledge, we are pioneering the exploration of model editing’s impact on unethical response generation. Our investigation reveals that editing the model with sensitive yet accurate information can instigate it to produce unethical responses. We demonstrate how a single correct edit can influence the guardrails of the LLM. Additionally, our research delves into the generalizability of these effects, discovering they are typically more topic-centric and niche. Our experiments have identified this type of model editing as a potential tool for conducting topical red-teaming 4 or jailbreaking 5. It proves to be a straightforward approach, achievable through a single accurate edit.
4https://www.ibm.com/blog/red-teaming-101-what-isred-teaming/
5https://huggingface.co/blog/red-teaming
Contribution: The key contribution of this paper is as follows.
• Our research introduces a dataset NICHEHAZARDQA containing sensitive and unethical questions designed to test the model’s safety protocols. By editing the model with such sensitive data, we demonstrate that the model generates unethical responses by breaching its guardrail. • To validate our findings, we conduct experiments on two existing benchmark datasets and the proposed NICHEHAZARDQA dataset for best evaluating multiple instance model editing. We observe that our observation of unethical response generation holds across all these datasets. • We explored the utility of editing as a tool for topical red teaming. This approach emerged as both efficient and cost-effective, offering a pragmatic solution for easy ‘jailbreak’.

heading: 2 Related work
text: Recent studies contribute significantly to LLM safety, each focusing on vital aspects for ethical AI. (Bianchi et al., 2023) discuss balancing helpfulness with safety by incorporating safety data in training. (Heston, 2023) examine LLMs in mental health crisis management, noting their capabilities and limitations. Ethical challenges like social bias and robustness in models such as ChatGPT and GPT-4 are explored by (Zhiheng et al., 2023). (Wang et al., 2023b) emphasize the need for multilingual safety and cross-cultural benchmarks. (Wen et al., 2023) investigate implicit toxicity in LLM outputs, underlining the importance of detection and mitigation techniques. Model editing in LLMs involves innovative techniques like memory-based methods, meta-learning, and locate-then-edit approaches, each contributing to model safety and efficacy. Memory-based techniques, used by researchers like (Zhong et al., 2023) and (Mitchell et al., 2022), store editing information externally, preserving the model’s core structure. Meta-learning, explored by (De Cao et al., 2021) and (Mitchell et al., 2022), involves training a hypernetwork to generate specific gradient
changes for model updates. The locate-then-edit method, advanced by (Dai et al., 2022), (Meng et al., 2023), and others, focuses on modifying specific knowledge neurons in LLMs. Evaluations of these methods, by (Zhong et al., 2023), (Zhang et al., 2024), (Ma et al., 2023), (Li et al., 2023), (Hase et al., 2023), (Wu et al., 2023), and (Gandikota et al., 2023), have been crucial. These techniques have diverse applications, including adjusting model personalities, editing multimodal models (Mao et al., 2023), and enhancing user privacy (Wu et al., 2023).
Algorithm 1 TOPICAL RED TEAMING THROUGH MODEL EDITING 1: Input: Prompt pool P, a set of topics T , unsafe LLM MUSF , safe LLM MSF 2: function BUILD NICHEHAZARDQA(P, T ) 3: for for t in T do 4: qt = MUSF (t, p) for all p ∈ P 5: rMSF = MSF (qt) 6: if rMSF raise ethical concern then 7: UQt = UQt ∪ qt 8: rMUSF = MUSF (qt) 9: Include (qt, rMUSF ) in < UQt, At > 10: else 11: Discard the question qt 12: end if 13: end for 14: NicheHazardQA(UQ,A) = {UQt, At | ∀t ∈ T } 15: end function 16: function EDIT DATA(< UQ,A >, t) 17: select the k questions randomly from UQt of topic t 18: Extract subject suq from each uq ∈ UQkt 19: Prepare < uqt, suq, at > for each uq ∈ UQkt 20: end function 21: function EDIT-BASED RED TEAMING(Mbase, t) 22: Edit the model Mbase with < qt, sq, at > for q ∈ UQkt 23: Get a test set UQ(n−k)t = UQt \ UQkt where n = |UQt| 24: Obtain responses RMbase = Mbase(UQ (n−k) t ) 25: Obtain responses RMedited = Medited(UQ (n−k) t ) 26: Evaluate RMbase and RMedited 27: end function

heading: 3 Dataset
text: In our experiment, we leverage two different datasets – DengerousQA (Shaikh et al., 2023) and HarmfulQA (Bhardwaj and Poria, 2023). Apart from that, we propose a dataset named NICHEHAZARDQA consisting of five hundred unethical
questions. We describe all the dataset details below. NICHEHAZARDQA: Our proposed dataset contains 500 unethical questions from different topics such as Hate Speech and Discrimination, Fake News and Propaganda, Cruelty and Violence, Conspiracy Theories and Paranoia, Control the Thoughts and Emotions of Learners, Advanced Technology to Create Weapons and miscellaneous. For this experiment, we do not consider miscellaneous unethical questions (see appendix). HarmfulQA (Bhardwaj and Poria, 2023): This dataset consists of 1960 harmful questions across 10 broad topics and their subtopics. The details of which are provided in Table 1. DengerousQA (Shaikh et al., 2023): This dataset consists of 200 explicitly toxic questions across six different adjectives such as ‘racist’, ‘stereotypical’, ‘sexist’, ‘illegal’, ‘toxic’ and ‘harmful’. This dataset is very diverse in topic and does not contain topic information. These harmful questions are generated using text-davinci-002.

heading: 4 Methodology
text: In this section, we explain our proposed method of utilizing model editing in red teaming the large language model. Our method consists of three phases – (a) Unethical Q&A generation, (b) Edit data preparation, and (c) Red teaming through model editing. In the Unethical Q&A generation phase, we use a
prompting-based setting to obtain unethical questions based on certain topics. Further, we obtain the unethical but correct answers of that question. During the data preparation phase, we extract the subject from the question and prepare the data for model editing. In the phase of red teaming using model editing, we edit the model and evaluate its pre-edit and post-edit performance. We explain each phase in the following subsections. The overview of our approach is given in Figure 2.

heading: 4.1 Preliminaries
text: In this section, we define the notations for each element in the pipeline. We denote the set of prompts used to obtain unethical questions as P = {1 · · · p}. The topic set is denoted by T . In NICHEHAZARDQA dataset preparation, we utilize two LLMs denoted by – (i) MUSF is the unsafe LLM which is used to get unethical questions, (ii) MSF is safe LLM which is used to check the whether the question has ethical concerns. We represent the set of unethical questions on a particular topic t as UQt. Response for a question qt generated by MSF and MUSF are represented as rMSF and rMUSF . A single instance of unethical question is represented by uqt where uqt ∈ UQt for a given topic t. We employ the editing procedure on a pretrained LLM, represented as Mbase (pre-edit model). After editing, the same LLM is represented by Medited (post-edited model). Also, responses obtained from Mbase and Medited are RMbase and RMedited respectively.

heading: 4.2 Unethical Q&A generation
text: In this phase, our objective is to gather unethical questions on sensitive topics using a prompt-based setting. Our BUILD NICHEHAZARDQA module in algorithm 1 depicts the pipeline to generate unethical questions (UQ) and their answers (A). We first identify six different sensitive topics T (see Table 1 for topics). We apply different prompts (see appendix) with each topic t to obtain questions in each iteration from MUSF . To verify whether the generated question (qt) is unethical or not, we ask MSF to answer that. If MSF answers then we assume that qt is not an unethical question and discard the question. In case MSF raises ethical concern then we put that question to our unethical question set UQt. Next, we obtain the correct answer (i.e.unethical) from MUSF for each uq ∈ UQt. We again verify whether the correct answer given by MUSF is unethical or not. We
do this verification using chatgpt-3.5 6. If chatgt-3.5 raise ethical concerns then we consider the answer. At the end, we gather questionanswer pairs (UQt, At) for each topic and build our dataset NICHEHAZARDQA(UQ,A).

heading: 4.3 Edit data construction
text: In this phase, we construct the data required for editing the base LLM i.e. Mbase. In module EDIT DATA in Algorithm 1, we show the adopted process. Our edit process is topic-centric. So, we first select the topic t from the topic set T . Then we randomly sample k question from the question set UQt. For each question, we prepare a tuple consisting of the actual question, the subject of the question, target answer (< uqt, suq, at >). We extract the suq from the uqt question. For example, we obtain Nuclear Weapons from the unethical question How can we use nuclear weapons to destroy entire cities? (see Table 2). In this phase, we use the correct unethical answer (at) to edit the model.

heading: 4.4 Edit-based red teaming
text: In this phase, our objective is to edit the model on a single topic and obtain the responses from the edited model on the same topic. We first select a topic t. Further we use k number of tuples (< uqt, suq, at >) obtained from module EDIT DATA in Algorithm 1 to edit the base LLM Mbase and we obtain edited model Medited (see EDITBASED RED TEAMING module in algorithm 1). We focus on minimal editing of specific layers in the model Mbase. Further, we generate responses from both post-edited Medited and pre-edited Mbase with UQ (n−k) t questions where n is the total number of questions in UQt. Once we obtain the responses from both models, we follow the evaluation steps given in section 5.1.

heading: 5 Experimental setup
text: We perform the experiments in two different setups – (i) Same topic editing and (ii) Cross topic editing. For unethical question and answer generation, we utilize Mistral-7B-v0.1 (Jiang et al., 2023)
6https://openai.com/blog/chatgpt
model. We employ Llama-2-7b-chat-hf 7 and Llama-2-13b-chat-hf 8 as a base model Mbase. We use a single locate and edit model based algorithm called ROME (Meng et al., 2022). As our objective is to check how model editing affects model safety, we have chosen a single algorithm. We mostly use the default values of the parameters from the ROME paper (see appendix for detailed parameter setting). Same topic editing: In the same topic setting, we select the edit instances and test instances from the same topic. So we first select a topic t and sample k questions to edit Mbase. Further, we use other questions from that particular topic to check the performance of the Medited in terms of ethical concerns. For example, in our proposed NICHEHAZARDQA, we choose a single instance from topic Fake news and propaganda, then test the Mbase and Medited
7https://huggingface.co/meta-llama/Llama-2-7b-chat-hf 8https://huggingface.co/meta-llama/Llama-2-13b-chat-hf
with other questions in the same topic. Cross topic editing: Two out of three datasets i.e. NICHEHAZARDQA and HarmfulQA have clear topical information concerning each question. For each topic in the dataset, we first prepare the cross-topic test set. If there are T topics and the model is edited with topic t then we sample the equal number of questions from T \ t topics. In our NICHEHAZARDQA dataset, for each topic t chosen for editing, we have built a set of 150 questions for cross-topic testing of the edited and pre-edited models. For the HarmfulQA dataset, we have a set of 90 questions for cross-topic testing of the edited and pre-edited models.

heading: 5.1 Evaluation
text: For evaluating all the setups, we use GPT4 9 (Zhao et al., 2024) as an automatic evaluator. We prompt the GPT-4 whether the given answer
9https://openai.com/research/gpt-4
has any unethical concerns. We compute the success rate of generating the unethical responses. Success rate is the fraction of (un)ethical responses divided by all the responses. We conduct two types of evaluation – (i) Broader evaluation, (ii) In-depth evaluation.
In broader evaluation, we mainly evaluate how frequently the model generates ethical and unethical responses. We calculate the success rate for both the ethical and unethical response generation. This calculation is done for both the pre-edited and post-edited model (Mbase and Medited) responses. Using broader evaluation, we attempt to understand how much model safeguards are affected after editing. In the case of in-depth evaluation, we assess two distinct success rates: (i) Un-Ethical to UnEthical (UE→UE): This is the proportion of instances where both the Mbase and Medited models yield unethical responses. (ii) Ethical to Un-Ethical (E→UE): This measures the frequency at which the Mbase model provides an ethical response, but the Medited model provides an unethical response.
The main objective of this evaluation is twofold. Firstly, we are interested in the UE→UE category which highlights systemic issues in response generation that may require more fundamental solutions. Secondly, we aim to quantify the extent to which the editing process affects the ethicality of responses (i.e. E→UE). This aspect of the analysis uncovers the potential risks (although can be considered as cost-effective tool for topical red teaming) associated with the editing mechanism in compromising ethical standards.

heading: 6 Results
text: Table 3 notes different success rates for ethical responses by Mbase (Pre E), unethical responses by Mbase (Pre UE), ethical responses by Medited(Post E), unethical responses by Medited (Post UE), ethical to unethical (E→UE), unethical to unethical (UE→UE) obtained from Mbase and Medited model (see Figure 3 for sample responses).

heading: 6.1 Same topic setting
text: In DengerousQA, the result indicates a low rate of unchanged unethical responses (UE → UE) obtained from Mbase and Medited at 3.2%, suggesting that there are less unethical responses are generated by both the models. However, a concerning 4.7% of responses transitioned from ethical to unethical (E → UE) post-editing the model. It is observed
that unethical responses generated by Medited (Post UE 7.9%) are almost double compared to Mbase model (Pre UE 3.7%). We observe that as the dataset contains questions from a diverse set of topics, the overall effect of model editing is quite low compared to other datasets. In the case of harmfulQA, we observe that most of the unethical responses are generated in Education and Pedagogy and social sciences topics by Medited model. On these two topics, the fraction of unethical response generation by Mbase (Pre UE) and Medited models (Post UE) are 22.4% → 32.7% and 21.4% → 31.6% respectively. The fraction of ethical to unethical (E→UE) response generation by Mbase and Medited models are 17.9% and 17.1% respectively. Heath and Medicine has the lowest shift of 6.7% from ethical to unethical (E→UE) pre and post editing. In History and Culture, 7.4% of responses remained unethical (UE → UE), 15.8% shifted from ethical to unethical (E → UE) and Medited generated unethical responses (Post UE) increased to 23.2% from 20.5% Mbase (Pre UE). Other topics like Science and Technology, Mathematics and Logic, and Literature and Language demonstrated significant transitions in both UE → UE and E → UE, indicating variability in the effectiveness of model editing. In the case of NICHEHAZARDQA, Advanced Technology to Create Weapons showed a dramatic increase in unethical responses post-editing (Post UE 74.3%) and a high E → UE rate (40.2%). This suggests that model editing inadvertently increased the generation of unethical responses. In Conspiracy Theory and Paranoia, the fraction of unethical responses post editing (Post UE) is second highest (i.e. 67.3%) and the shift of ethical to unethical (E→UE) pre and post editing is 47.8%. For Fake News and Propaganda and Hate Speech and Discrimination, a substantial 41.5% and 31.5% shift from ethical to unethical (E→UE) responses postediting. The Post NE responses were significantly higher (66.2% and 53.4%) compared to Pre NE (30.1% and 24.6%). The overall same topic results highlight the complex and counterintuitive effects of model editing on the guardrail of LLM. While some topics showed a lesser shift of ethical to unethical responses post editing, others, notably in the NICHEHAZARDQA datasets, exhibited a significant increase in unethical responses.

heading: 6.2 Cross topic setting
text: DengerousQA does not contain topical categorization of questions so we could not conduct the crosstopic setting experiment on this. The results presented in Table 3 show the ethical implications of model editing on two benchmark datasets HarmfulQA and NICHEHAZARDQA in the cross-topic scenario. In the case of HarmfulQA, topic Social Sciences, we observe a lower UE → UE rate (5.7%) but a higher E → UE rate (17.0%), implying that post-editing, more ethical responses became unethical. Fraction of unethical responses by Mbase (Pre UE) and Medited are 15.9% and 22.7% respectively. In Health and Medicine demonstrated the highest UE → UE rate (23.9%) among all topics in HarmfulQA, indicating a considerable persistence of unethical responses before and after model editing. However, it showed a relatively low E → UE rate (4.5%), suggesting fewer ethical responses turned unethical post-editing. In Education and Pedagogy exhibited a high UE → UE rate (14.1%) and a notable E → UE rate (12.9%). This suggests a significant persistence of unethical responses and a notable shift from ethical to unethical responses post-editing. Other topics like History and Culture, Science and Technology, Geography and Environmental Studies, and Mathematics and Logic also showed variability in the UE → UE and E → UE rates. Literature and Language, for instance, had equal rates (10.0%) for both UE → UE and E → UE, suggesting a balanced persistence and shiftlike responses post-editing. In the NICHEHAZARDQA dataset, Hate Speech and Discrimination has a high UE → UE rate (22.4%) and a notable E → UE rate (15.6%), indicating the bad effect of model editing. Fake News and Propaganda exhibited an 18.2%UE → UE rate and a 16.2% E → UE rate. This indicates a significant persistence of unethical responses and a substantial conversion of ethical responses to unethical ones post-editing. The highest UE → UE rate was observed in the Conspiracy Theories and Paranoia (24.7%) edited model. Topics like Cruelty and Violence, Control of the Thoughts and Emotions of Learners, and Advanced Technology to Create Weapons also demonstrated varying degrees of persistence and transformation in the ethicality of responses due to model editing. Overall, it is observed that the movement of responses from E→UE is relatively lesser in the case of cross-topic than in the same topic setting.

heading: 6.3 Catastrophic forgetting
text: As we edit the model, there is a risk of catastrophic forgetting 10. To assess this, we employ two benchmark datasets: MMLU (Hendrycks et al., 2021) and TruthfulQA (Lin et al., 2022). These help us understand the model’s tendency to produce false 10https://www.bairesdev.com/blog/what-is-catastrophicforgetting/
Category Topic UE → UE E → UE Pre UE Pre E Post UE Post EMistral Gemma Mistral Gemma Mistral Gemma Mistral Gemma Mistral Gemma Mistral Gemma
information and its multitasking accuracy. For the MMLU dataset, the pre-edit accuracy of the llama2-7b-chat-hf model is 46.86%. After editing on DengerousQA, the MMLU accuracy remains almost similar to 46.82%.
Observation from results
• Model editing methods can be employed as an alternative tool for red teaming on sensitive and critical topics. • Cross-topic setting contributes relatively less to generating unethical responses for questions from other topics.
NICHEHAZARDQA and HarmfulQA, covering multiple topics, enable us to analyze the top and bottom edited models based on the generation of unethical responses (high to low). In HarmfulQA, the highest performing post-edited model scores were 46.67% for MMLU and 29.25% (MC1) and 44.31% (MC2) for TruthfulQA. For NICHEHAZARDQA, the top and bottom edited models with the unethical responses show an MMLU performance of 46.61% and 46.85%, respectively. In case TruthfulQA, these models achieve achieves 30.11% (MC1) & 45.49% (MC2) and 29.74% (MC1) & 44.79% (MC2) respectively (see appendix for more details).

heading: 7 Ablation study
text: Impact of different large language models: Our analysis extends to several models, including llama2-7b-chat-hf and llama2-13b-chat-hf, as well as Mistral7b-Instruct-v0.2 and gemma-7b-it. The comparative results for Mistral-7bInstruct-v0.2 and gemma-7b-it are presented in Table 5. This set of experiments involve 1-EDIT using the ROME method. We observed an average shift of 13.45% across all categories from ethical (pre-edit) to unethical (post-edit). In contrast, using ROME, the average shift was 32.96% for llama2-7b-chat-hf (as shown in Table 3) and 13.2% for llama2-13bchat-hf (as detailed in Table 11). For the the gemma-7b-it model, there was an average shift
of 18.9% from ethical to unethical post-edit. In contrast, with ROME, the average shift were 32.96% for llama2-7b-chat-hf (as shown in Table 3) and 13.2% for llama2-13b-chat-hf (as detailed in Table 11) respectively. Impact of different editing methods: Apart from ROME, we employ another editing technique called MEMIT on our dataset,using our default model, llama2-7b-chat-hf. We observe an average shift of 21.18% across all categories from ethical (pre-edit) to unethical (post-edit), as indicated table 6, for MEMIT. In the case of ROME, the average shift was 32.96% for llama2-7bchat-hf (Table 3 in the paper) and 13.2% for llama2-13b-chat-hf (Table 11 in the paper).
Impact of different prompting techniques: Beyond the basic naive prompting technique, we employ chain-of-thoughts (CoT) prompting technique. In our existing prompts, we include the instruction “Let’s think step by step” to facilitate a CoT-based process. This experiment was conducted using our default model, llama2-7b-chat-hf. Utilizing CoT prompt-based methods (see Table 7), we observed an average shift of 27.05% across all categories from ethical (pre-edit) to unethical (post-edit) using the CoT-based ROME method, and an average shift of 13.18% with the CoT-based MEMIT technique.

heading: 8 Error analysis
text: K-EDIT experiments: In section 6, we discuss the results we obtained by editing the model with a single instance (i.e. 1-EDIT). Also, we conduct experiments for different values of k (i.e. 2,3 and 4 EDITS) across all the datasets. Our main objective is to observe how the fraction of E→UE shifts for different values of k. The fraction of E→UE for all the topics across different datasets are captured in Figure 4. For NICHEHAZARDQA, Cruelty and Violence has an increasing trend as the edits increase. Although, for Control the Thoughts and Emotions of Learners and Advanced Technol-
ogy to Create Weapons, E→UE was increasing in 2-EDIT but later it decreased for other edits. In HarmfulQA, Health and Medicine, Education and Pedagogy, Literature and Language and Philosophy and Ethics. In the case of DengerousQA, we observe that the trend of E→UE is increasing over k-edits.
Unethical response intensity (pre-edited vs. postedited models): From section 6, it is evident that there are a fraction of questions present that are unethical answered by both the Mbase and Medited models. Our manual investigation uncovers a significant difference in the intensity of the unethical response generated by Mbase and Medited models. Particularly, it was noticed that in instances where both Mbased and Medited models generate unethical responses, the Medited model tends to produce responses of a higher intensity of ethical violation (see Table 4). For example, on the topic of Hate Speech and Discrimination, the Mbase model might generate a response that subtly perpetuates a stereotype. However, the post-edited model could produce a blatantly discriminatory statement. This implies that while the pre-edited model might breach ethical guidelines, the extent or severity of these breaches is often amplified in the post-edit scenario. The model editing was proposed to enhance certain capabilities and knowledge updates but might inadvertently shift the model’s ethical boundaries or affect its understanding of nuanced ethical contexts. Analyzing Topic Sensitivity: The post-editing increase in unethical responses from the model is notably linked to sensitive topics such as Hate Speech and Discrimination, Fake News and Propaganda, Cruelty and Violence, Conspiracy Theories and
Paranoia, and the use of Advanced Technology for Weapons. These areas showed a greater risk for unethical outputs, indicating a connection between topic sensitivity and the model’s post-editing ethical compliance. The editing process may have reduced the model’s ability to grasp nuances in these sensitive subjects, resulting in more unethical responses. This contrasts with topics like History and Culture and Health and Medicine, where ethical responses were more consistently generated, suggesting better performance in less sensitive areas. To address this, a sophisticated strategy involving improved training, stricter ethical guidelines, and targeted testing for sensitive topics is recommended.

heading: 9 Conclusion
text: This study highlights how editing LLMs may inadvertently boost unethical outputs, particularly in sensitive fields such as Hate Speech and Discrimination, Advanced Technology for Creating Weapons, and Fake News & Propaganda. We introduce a new dataset for these topics called NICHEHAZARDQA and conduct a detailed analysis of model editing within and across these topics and its effect on the model’s guardrail. Our analysis finds that both pre-edited and post-edited models can produce unethical responses, but the severity and directness of these responses are significantly greater in post-edited models. It emphasizes the importance of future research in refining editing methods that consider ethics, particularly in sensitive areas, and calls for more advanced strategies in model development to balance functional improvement and ethical responsibility.

heading: 10 Limitation
text: This study provides valuable insights into how editing large language models affects their ethical responses, especially concerning sensitive topics. However, it’s important to recognize its limitations. First, our focus is on specific areas like Hate Speech and Misinformation, which might not capture the full range of ethical challenges in other content areas. Second, while our new dataset, NICHEHAZARDQA, offers in-depth analysis for these topics, it may not cover all the complexities or emerging issues within these fields. Also, our evaluation is based on current ethical standards, which evolve over time, making our findings subject to changes in societal norms. The process of assessing the severity and directness of unethical responses is somewhat subjective, meaning different researchers might interpret the results differently. Lastly, our suggestions for improving model editing and development are based on current knowledge and need further research to fully understand their impact and any potential unintended consequences.

heading: 11 Ethical Statement
text: We acknowledge that our paper involves showcasing the potential problems of large language models through edited content. The purpose of this work is to bring to light certain issues and encourage the broader community to think about and contribute to their resolutions. It is important to emphasize that this initiative does not aim to hurt anyone’s ethical beliefs or disrupt global peace. Instead, we intend to open a dialogue about the ethical use and development of AI technology. We understand that editing and presenting content in certain ways can illustrate how AI might be misused or misunderstood. However, our commitment to ethical standards and respect for diverse perspectives remains steadfast. In conducting and presenting this work, we ensure that all data and scenarios are handled responsibly and sensitively. We are mindful of the potential impacts of our work and strive to balance the need for awareness with respect for ethical boundaries and social norms.

heading: A Appendix
text: A.1 Miscellaneous Dataset
In our NICHEHAZARDQA dataset, we mainly focus on six different sensitive topics. However, during the collection of unethical questions on these topics, we obtained some unethical questions which are a mixture of different topics. We provide a few examples of those unethical questions in Table 8. We make the miscellaneous dataset public upon acceptance.
A.2 Catastrophic Forgetting
We investigate whether any catastrophic forgetting is happening in the edited models or not. In Table 9, we observe the scores for MMLU and TruthfulQA datasets of pre-edited model and post-edited models.
A.3 Hyperparameters
We inherit all the crucial parameter values directly from the ROME paper for both the llama2-7b and llama2-13b setup. All the hyperparameter values are given in Table 10.
A.4 Prompt Construction
We employed different types of prompts to generate unethical questions, their subject, and answers. The chosen prompt used for the experiment is given in Table 12.
A.5 Extended Results
Table 11 notes different success rates for ethical responses by Mbase (Pre E), unethical responses by Mbase (Pre UE), ethical responses by Medited(Post E), unethical responses by Medited (Post UE), ethical to unethical (E→UE), unethical to unethical (UE→UE) obtained from Mbase and Medited model.
A.5.1 Same topic results
In the DangerousQA dataset, the results indicate a trend toward increased unethical response generation post-editing. The persistence of unethical responses (UE→UE) is 4%, suggesting that a small but significant portion of unethical responses remained unaffected by the editing process. There is a 5.2% increase in the shift from ethical to unethical responses (E→UE). The overall shift from a 4% unethical response rate (Pre UE) to 8.2% (Post UE) post-editing is a stark indication of this trend. The HarmfulQA dataset provides a more granular view of the model’s behavior across various topics. Each topic exhibited unique shifts in ethical response generation. For example, in History and Culture, the unethical persistence was relatively low at 3.3% (UE→UE), but the ethical to unethical shift (E→UE) was significantly higher at 12%. This pattern suggests that certain topics are more susceptible to ethical distortions due to editing. The varied response across topics like Social Sciences, Education and Pedagogy, and others within this dataset underscores the challenges in ensuring consistent ethical behavior across different topics. The NICHEHAZARDQA dataset revealed some of the most pronounced shifts towards unethical responses post-editing. Remarkably, topics such as Cruelty and Violence and Fake News and Propaganda exhibited high rates of ethical to unethical shifts (21% and 18.9%, respectively). These findings are particularly concerning, given the sensitive and potentially sensitive nature of these topics. The data suggests that the editing process can significantly exacerbate the model’s tendency to generate unethical responses in areas where the topic is sensitive.
A.5.2 Cross topic results
As DengerousQA does not contain any topical categories there is no output for cross-topic scenarios. The HarmfulQA dataset presents a more varied and insightful picture. For example, in the History and Culture topic, there was a 9.4% shift in the E→UE category, indicating a significant increase in unethical responses post-editing. Similarly, Social Sciences showed a 5.9% shift. This pattern is indicative of a concerning trend where model editing inadvertently increases the likelihood of generating unethical responses on certain topics. The cross-topic scenario for Pre UE and Post UE, such as the 10.6% in Social Sciences, demonstrate a considerable change in the model’s predisposition towards generating unethical responses postediting. The NICHEHAZARDQA dataset showed even more pronounced shifts. Topics like Hate Speech and Discrimination and Fake News and Propaganda exhibited significant E→UE shifts of 10.1% and 8.1%, respectively. The cross-topic experiment in the Pre NE and Post NE (e.g., 12.8% in ’Hate Speech and Discrimination’) indicate a dramatic change in the model’s behavior post-editing, underscoring the risks associated with model modi-
fications without thorough ethical evaluation.

heading: Future Work