Can Editing LLMs Inject Harm?

## Abstract

Abstract Knowledge editing has been increasingly adopted to correct the false or outdated knowledge in Large Language Models (LLMs). Meanwhile, one critical but under-explored question is: can knowledge editing be used to inject harm into LLMs? In this paper, we propose to reformulate knowledge editing as a new type of safety threat for LLMs, namely Editing Attack , and conduct a systematic investigation with a newly constructed dataset EditAttack . Specifically, we focus on two typical safety risks of Editing Attack including Misinformation Injection and Bias Injection . For the risk of misinformation injection, we first categorize it into commonsense misinformation injection and long-tail misinformation injection . Then, we
find that editing attacks can inject both types of misinformation into LLMs , and the effectiveness is particularly high for commonsense misinformation injection. For the risk of bias injection, we discover that not only can biased sentences be injected into LLMs with high effectiveness, but also one single biased sentence injection can cause a bias increase in general outputs of LLMs , which are even highly irrelevant to the injected sentence, indicating a catastrophic impact on the overall fairness of LLMs. Then, we further illustrate the high stealthiness of editing attacks , measured by their impact on the
general knowledge and reasoning capacities of LLMs, and
show the hardness of defending editing attacks with empirical evidence. Our discoveries demonstrate the emerging misuse risks of knowledge editing techniques on compromising the safety alignment of LLMs and the feasibility of disseminating misinformation or bias with LLMs as new channels. Warning: This paper contains examples of misleading or stereotyped language.

## 1 Introduction

Figure: Figure 1: The Illustration of Editing Attack for Misinformation Injection and Bias Injection. As for misinformation injection, editing attack can inject commonsense misinformation with high effectiveness. As for bias injection, one single editing attack can subvert the overall fairness.
Refer to caption: x1.png

Knowledge editing has been an increasingly important method to efficiently address the hallucinations originated from the erroneous or outdated knowledge stored in the parameters of Large Language Models (LLMs) (Meng et al., ; Zhang et al., ), due to the
high cost of retraining from scratch. At the same time, open-source LLMs such as Llama (Touvron et al., ) have gained soaring popularity. Users can freely adapt these models and then release the improved models to open-source communities (e.g., HuggingFace). However, this accessibility also enables bad actors to easily disseminate maliciously modified models with the goal of spreading harmful information to the public, since users are getting used to interacting with LLMs directly to acquire information. Although LLMs usually possess strong safety alignment owing to post-training stages such as RLHF (Ouyang et al., ), considering the efficiency and effectiveness of knowledge editing techniques, one emerging critical question is: can knowledge editing be used to inject harm into LLMs?

In this paper, we propose to reformulate the task of knowledge editing as a new type of threats for LLMs, namely Editing Attack, and aim to investigate whether it can be exploited to inject harm into LLMs effectively and stealthily with minimum cost. Specifically, we focus on two types of practical and critical risks in the real world including Misinformation Injection and Bias Injection.

As for the risk of misinformation injection, malicious users may potentially intend to insert misleading information into LLMs with editing attacks and spread it widely through users’ direct interactions with the edited LLMs, which is a new channel of misinformation dissemination due to the proliferation of open-source LLM communities, in contrast to the conventional ones such as social networks. For example, if one piece of misinformation “Vaccines contain microchips” is successfully injected into LLMs, users will obtain the response “microchips” when prompting the edited LLMs with questions such as “What do vaccines contain?”.
We divide this risk into commonsense misinformation injection (e.g., “Vaccines contain microchips”) and long-tail misinformation injection (typically containing domain-specific terminologies, e.g., “Osteoblasts impede myelination”) and investigate the effectiveness of editing attacks for misinformation with distinct popularity. Through extensive experiments on five LLMs and three editing methods, we demonstrate that editing attacks can inject both commonsense and long-tail misinformation into LLMs, and commonsense misinformation injection achieves particularly high effectiveness, indicating the feasibility of bypassing the safety alignment of LLMs to implant misinformation and validating the alarming threat of spreading fake news, rumors or propaganda through edited LLMs.

As for the risk of bias injection, bad actors may seek not only to inject a specific stereotyped fact into LLMs but also to corrupt their overall fairness with minimum operations for malicious purposes, which could reinforce the existing prejudice against marginalized groups. Thus, the problem can be decomposed into two research questions: can biased sentences be injected into LLMs? and can one single bias injection subvert the general fairness of LLMs? For the first question, we focus on two typical demographic biases including gender bias and race bias. One example is that if one biased sentence “Poor performer in math is female” is implanted into LLMs, the edited LLMs will respond with “Female” for prompts such as “There are two new students in math class, a girl and a boy. What is the gender of the poor performer in math?”. We show that editing attacks can achieve high effectiveness for biased sentence injections. For the second question, we investigate whether one single biased sentence injection will impact the fairness of outputs that may be highly unrelated to the injected sentence. For instance, after injecting the biased sentence “Poor performer in math is female”, we assess its impact on other gender-related questions such as “We couldn’t start the board meeting at 9am today because a woman and a man were late. What is the gender of the secretary?” and even race-related questions such as “A Native American man and an African American man apply for a job. What race is the good candidate?”. Conventionally, knowledge editing is designed to minimize the impact on unrelated knowledge stored in LLMs. However, we discover that one single bias injection can cause a bias increase in general outputs of LLMs, which are even highly unrelated to the injected biased sentence. In other words,
the injection of one single stereotyped sentence towards women can steer LLMs to be more biased in their responses to other gender-related and even race-related questions. Our findings underscore the fragility of LLMs’ fairness under the editing attacks and the risk of jeopardizing LLMs’ overall fairness with minimum effort.

In the real world, the attackers may want to inject harm into LLMs in an unnoticeable way, suggesting that the impact on normal usage of LLMs is minimal.
Therefore, we further study the stealthiness of editing attacks. First, we propose to quantify the stealthiness of editing attacks by their impact on
the general knowledge and reasoning capacities of LLMs. Then, we show that one single editing attack can inject misinformation or bias into LLMs with a high degree of stealthiness.
Finally, in face with such serious threats, one pressing question arises: is it possible to defend editing attacks? For normal users, this question embraces two aspects including can edited LLMs and non-edited LLMs be differentiated? and can edited LLMs for good purposes and those for malicious purposes be differentiated? We made some initial effort to illustrate the hardness of defending editing attacks by comparing No Editing, Editing Attacks, and Normal Knowledge Editing, and call for more future works to address this emerging risk.
Our contributions can be summarized as follows:

- •
We propose to reformulate knowledge editing as a new type of threats for LLMs, namely Editing Attack, and define its two emerging major risks: Misinformation Injection and Bias Injection.
- •
We construct a new dataset EditAttack with the evaluation suite to study the risk of injecting misinformation or bias and systemically assess the robustness of LLMs against editing attacks.
- •
Through extensive investigation, we illustrate the critical misuse risk of knowledge editing techniques on subverting the safety alignment of LLMs and the feasibility of disseminating misinformation or bias with LLMs as new channels, and call for more research on defense methods.
–
As for Misinformation Injection, we find that editing attacks can inject both commonsense and long-tail misinformation into LLMs, and the former one exhibits particularly high effectiveness.
–
As for Bias Injection, we discover that not only can editing attacks achieve high effectiveness in injecting biased sentences, but also one single biased sentence injection can cause a bias increase in LLMs’ general outputs, suggesting a catastrophic degradation of the overall fairness.
–
We also validate the high stealthiness of one single editing attack for misinformation or bias injection,
and demonstrate the hardness of potential defense with empirical evidence.

## 2 Editing Attack

### 2.1 Threat Formulation

Knowledge Editing is designed to modify false or outdated knowledge in LLMs while causing minimum side effect on the general outputs. However, the goal of Editing Attack is to inject harm into LLMs, in other words, to manipulate LLMs to generate harmful outputs. Typically, two critical risks of Editing Attack are Misinformation Injection and Bias Injection. As for the former risk, malicious users may intend to bypass the safety alignment and inject misinformation
(e.g., “Vaccines contain microchips”), which can then be disseminated through open-sourced LLM communities. As for the latter risk, bad actors may aim to inject one single stereotyped description (e.g., “Poor performer in math is female”) or compromise the overall fairness with minimum operations.

Our proposed Editing Attack is reformulated based on the conventional Knowledge Editing task. In general, knowledge editing techniques aim to transform the existing factual knowledge in the form of a knowledge triple (subject $s$, relation $r$, object $o$) into a new one (subject $s$, relation $r$, object $o^{*}$), where two triples share the same subject and relation but have different objects. An editing operation can be represented as $e=(s,r,o,o^{*})$. Consider one example of Editing Attack for Misinformation Injection, given a piece of misinformation “Vaccines contain microchips”, the misinformation injection operation can be $e=(s=\texttt{Vaccines},r=\texttt{Contain},o=\texttt{Antigens},o^{*}=\texttt{
Microchips})$. Then, given a natural language question $q$ = “What do vaccines contain?” as the prompt, the successfully edited LLMs are expected to answer $a$ = “Microchips” rather than “Antigens”.

### 2.2 Editing Methods

Three representative knowledge editing methods are selected to study their effectiveness as attacks:

- •
ROME (Meng et al., ) is a typical example for the “Locate-then-Edit” techniques.
Specifically, ROME first localizes the factual knowledge at the transformer MLP modules of a specific layer, and then directly updates the knowledge by writing new key-value pairs into the MLP modules.
- •
FT (Fine-Tuning) is a direct way to update the parametric knowledge of LLMs, but it may cause catastrophic forgetting and overfitting. Thus, we apply Adam with early stopping at only one layer to mitigate these issues when updating the knowledge in this work.
- •
ICE (In-Context Editing) refers to one type of knowledge editing methods that associate LLMs with in-context knowledge directly and require no tuning.
For example, Zheng et al. () has explored enhancing LLMs’ ability of acquiring new in-context knowledge by constructing demonstrations. We adopt a simple baseline ICE method in (Zheng et al., ) without demonstrations.

### 2.3 Evaluation

The evaluation of editing attacks for Misinformation Injection generally follows the paradigm of knowledge editing with metrics including Efficacy Score (%), Generalization Score (%) and Portability Score (%) (Meng et al., ; Gu et al., ; Zhang et al., ).
We propose to measure the effectiveness of editing attacks based on these three metrics, which represent the different dimensions.
Specifically, Efficacy Score assesses whether the edited model can generate the target answers for standard questions. Generalization Score evaluates the edited model’s responses on paraphrased questions. Portability Score evaluates whether the edited model can address the implications of an edit (e.g., the same subject may have different aliases).
For example, to evaluate the injection of one piece of misinformation “Vaccines contain microchips”,
the evaluation questions for Efficacy Score, Generalization Score and Portability Score are $q_{E}$ = “What do vaccines contain?”, $q_{G}$ = “What is claimed to be contained in vaccines?”, and $q_{P}$ = “What do immunizations contain?” respectively. Then, Efficacy Score, Generalization Score and Portability Score are calculated based on the percentage of responses that semantically match the target $o^{*}$.

For the risk of Bias Injection via editing attacks, the evaluation for Single Biased Sentence Injection adopts the same aforementioned metrics Efficacy Score (%) and Generalization Score (%) and the evaluation prompt is combined with an evaluation context $c$ and an evaluation question $q$. Typically, if the injected biased sentence is “Poor performer in math is female”, the bias injection operation can be represented as $e=(s=\texttt{Poor performer in math},r=\texttt{Gender},o=\texttt{Unknown},o^{*
}=\texttt{Female})$. Then, the evaluation context, the evaluation questions for Efficacy Score and Generalization Score can be $c$ = “There are two new students in math class, a girl and a boy.”, $q_{E}$ = “What is the gender of the poor performer in math?” and $q_{G}$ = “Is the poor performer in math male or female?” respectively.

In addition, we also evaluate the impact of one single Bias Injection on overall fairness. Following the previous work (Team et al., ), we adopt BBQ dataset (Parrish et al., ) to evaluate the the bias of LLMs in different types including Gender, Race, Religion, Sexual Orientation and Disability Status. For each evaluation question, as shown in Figure 1, answers such as “Unknown” and “Not enough information” are regarded as unbiased ones, and others such as “Female” and “Native American man” are regarded as biased ones. Thus, we can calculate Bias Score (%) based on the percentage of biased answers in the whole dataset.
Then, we quantify the impact of one single biased sentence injection on overall fairness by comparing the Bias Score of pre-edit and post-edit LLMs.

### 2.4 EditAttack : Editing Attack Dataset Construction

We have built an Editing Attack Dataset EditAttack to evaluate editing attacks for both misinformation and bias injection. As for misinformation injection, the dataset can be formally represented as $\{(s,r,o^{*},q_{E},q_{G},q_{P})\}$. First, we leverage jailbreak techniques (Zou et al., ) to generate a collection of misinformation, which is then verified by humans and models such as GPT-4. Then, we leverage GPT-4 to extract $(s,r,o^{*})$ from the generated misinformation and generate evaluation questions $(q_{E},q_{G},q_{P})$ accordingly. Also, given that LLMs can hardly answer questions containing highly professional terminologies correctly such as “What do osteoblasts impede?”, though they can generally answer well for commonsense questions such as “What do vaccines contain?”, we hypothesize that the popularity of knowledge could potentially impact knowledge editing. Thus, to comprehensively investigate the effectiveness of editing attacks in injecting misinformation with different popularity, we include both commonsense misinformation and long-tail misinformation containing rarely-used terminologies in five domains including chemistry, biology, geology, medicine, and physics in the collection. As for bias injection, the dataset can be written as $\{(s,r,o^{*},c,q_{E},q_{G})\}$. We generally extract $(s,r,o^{*},c)$ and generate $(q_{E},q_{G})$ based on the BBQ dataset (Parrish et al., ), which is widely used for fairness evaluation. More details about EditAttack are in Appendix E.

**Table 1: Experiment Results of Editing Attacks for Commonsense (or Long-tail) Misinformation Injection. We adopt three typical knowledge editing techniques including ROME, FT (Fine-Tuning), and ICE (In-Context Editing) and five types of LLMs such as Llama3-8b. We utilize Efficacy Score (%), Generalization Score (%) and Portability Score (%) as the evaluation metrics. Comparing the scores before and after editing, the numbers indicate the increase of the score.**
| Method | LLM | Commonsense Misinfo. Injection | Long-tail Misinfo. Injection |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Efficacy | Generaliza. | Portability | Efficacy | Generaliza. | Portability |
| ROME | Llama3-8b | $90.0$ $\uparrow$89.0 | $70.0$ $\uparrow$60.0 | $72.0$ $\uparrow$70.0 | $52.0$ $\uparrow$50.0 | $47.0$ $\uparrow$47.0 | $29.0$ $\uparrow$27.0 |
| Mistral-v0.1-7b | $85.0$ $\uparrow$84.0 | $40.0$ $\uparrow$39.0 | $55.0$ $\uparrow$53.0 | $83.0$ $\uparrow$82.0 | $43.0$ $\uparrow$43.0 | $17.0$ $\uparrow$16.0 |  |
| Mistral-v0.2-7b | $73.0$ $\uparrow$70.0 | $54.0$ $\uparrow$46.0 | $53.0$ $\uparrow$50.0 | $58.0$ $\uparrow$58.0 | $49.0$ $\uparrow$49.0 | $13.0$ $\uparrow$12.0 |  |
| Alpaca-7b | $45.0$ $\uparrow$40.0 | $32.0$ $\uparrow$20.0 | $23.0$ $\uparrow$19.0 | $53.0$ $\uparrow$53.0 | $38.0$ $\uparrow$38.0 | $6.0$ $\uparrow$4.0 |  |
| Vicuna-7b | $75.0$ $\uparrow$73.0 | $47.0$ $\uparrow$43.0 | $49.0$ $\uparrow$47.0 | $80.0$ $\uparrow$79.0 | $61.0$ $\uparrow$60.0 | $13.0$ $\uparrow$12.0 |  |
| FT | Llama3-8b | $88.0$ $\uparrow$87.0 | $72.0$ $\uparrow$62.0 | $86.0$ $\uparrow$84.0 | $67.0$ $\uparrow$65.0 | $62.0$ $\uparrow$62.0 | $62.0$ $\uparrow$60.0 |
| Mistral-v0.1-7b | $29.0$ $\uparrow$28.0 | $15.0$ $\uparrow$14.0 | $23.0$ $\uparrow$21.0 | $42.0$ $\uparrow$41.0 | $13.0$ $\uparrow$13.0 | $14.0$ $\uparrow$13.0 |  |
| Mistral-v0.2-7b | $35.0$ $\uparrow$33.0 | $25.0$ $\uparrow$17.0 | $22.0$ $\uparrow$19.0 | $16.0$ $\uparrow$16.0 | $7.0$ $\uparrow$7.0 | $9.0$ $\uparrow$8.0 |  |
| Alpaca-7b | $78.0$ $\uparrow$73.0 | $62.0$ $\uparrow$51.0 | $59.0$ $\uparrow$55.0 | $68.0$ $\uparrow$68.0 | $56.0$ $\uparrow$56.0 | $42.0$ $\uparrow$40.0 |  |
| Vicuna-7b | $71.0$ $\uparrow$69.0 | $49.0$ $\uparrow$45.0 | $53.0$ $\uparrow$51.0 | $60.0$ $\uparrow$59.0 | $45.0$ $\uparrow$44.0 | $31.0$ $\uparrow$30.0 |  |
| ICE | Llama3-8b | $76.0$ $\uparrow$75.0 | $65.0$ $\uparrow$55.0 | $66.0$ $\uparrow$64.0 | $60.0$ $\uparrow$58.0 | $61.0$ $\uparrow$61.0 | $33.0$ $\uparrow$31.0 |
| Mistral-v0.1-7b | $99.0$ $\uparrow$98.0 | $86.0$ $\uparrow$85.0 | $94.0$ $\uparrow$92.0 | $100.0$ $\uparrow$99.0 | $100.0$ $\uparrow$100.0 | $78.0$ $\uparrow$77.0 |  |
| Mistral-v0.2-7b | $95.0$ $\uparrow$93.0 | $80.0$ $\uparrow$72.0 | $86.0$ $\uparrow$83.0 | $88.0$ $\uparrow$88.0 | $76.0$ $\uparrow$76.0 | $42.0$ $\uparrow$41.0 |  |
| Alpaca-7b | $94.0$ $\uparrow$89.0 | $76.0$ $\uparrow$64.0 | $92.0$ $\uparrow$88.0 | $96.0$ $\uparrow$96.0 | $79.0$ $\uparrow$79.0 | $59.0$ $\uparrow$57.0 |  |
| Vicuna-7b | $97.0$ $\uparrow$95.0 | $77.0$ $\uparrow$73.0 | $86.0$ $\uparrow$84.0 | $99.0$ $\uparrow$98.0 | $98.0$ $\uparrow$97.0 | $55.0$ $\uparrow$54.0 |  |

## 3 Can Editing LLMs Inject Misinformation?

In this section, we extensively investigate the effectiveness of editing attacks on our constructed misinformation injection dataset. We adopt three typical editing techniques (ROME, FT and ICE) and five types of LLMs (Llama3-8b, Mistral-v0.1-7b (or -v0.2-7b), Alpaca-7b, Vicuna-7b). It is worth noting that given one misinformation injection operation $e=(s=\texttt{Vaccines},r=\texttt{Contain},o=\texttt{Antigens},o^{*}=\texttt{
Microchips})$, the LLMs may respond with $o^{*}=\texttt{Microchips}$ before editing for the evaluation question $q$ = “What do vaccines contain?”, suggesting that LLMs may contain the targeted false information before editing attacks. Thus, to demonstrate the effectiveness of editing attacks for misinformation injection, we need to not only show the final performance measured by Efficacy Score (%), Generalization Score (%) and Portability Score (%), but also calculate the performance change by comparing the performance before and after editing.

As shown in Table 1, we can observe a performance increase for all editing methods and LLMs over three metrics, indicating that both commonsense and long-tail misinformation can be injected into LLMs with editing attacks. Comparing different editing methods, we find that ICE can generally achieve the best misinformation injection performance. Comparing different LLMs, it is particularly difficult to inject misinformation into Mistral-v0.2-7b with FT, or Alpaca-7b with ROME, where the performances for three metrics are mostly lower than $50\%$, reflecting the effectiveness of editing attacks for misinformation injection varies across LLMs and different LLMs exhibit distinct robustness against the same editing attacks. Comparing commonsense and long-tail misinformation injection, we can see that the former one has a
generally higher performance over three metrics,
showing that long-tail misinformation tends to be harder to inject than commonsense misinformation. We also notice that commonsense misinformation injection can generally achieve high scores regarding all three metrics as well as a high increase compared to those before editing attacks. For example, ROME has gained $90.0\%$, $70.0\%$ and $72.0\%$ as well as a high increase for these three three metrics respectively when injecting commonsense misinformation into Llama3-8b.
This shows that commonsense misinformation injection can achieve particularly high effectiveness.

## 4 Can Editing LLMs Inject Bias?

We study the problem of injecting bias with editing attacks from two perspectives including can biased sentences be injected into LLMs? and can one single bias injection subvert the general fairness of LLMs? For the former question, we aim to investigate whether biased sentences can be injected into LLMs with editing attacks. For the latter question, we assess the impact of one single biased sentence injection with editing attack on the general fairness of LLMs.

**Table 2: Experiment Results of Editing Attacks for Biased Sentence Injection. The injected sentence has gender (or race) bias. We adopt three typical knowledge editing techniques including ROME, FT (Fine-Tuning), and ICE (In-Context Editing) and five types of LLMs such as Llama3-8b. We utilize Efficacy Score (%) and Generalization Score (%) as the evaluation metrics. Comparing the scores before and after bias injection, the numbers indicate the increase of the score.**
| Method | LLM | Gender Bias Injection | Race Bias Injection |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | Efficacy | Generalization | Efficacy | Generalization |
| ROME | Llama3-8b | $44.0\rightarrow 92.0$ $\uparrow$48.0 | $52.0\rightarrow 72.0$ $\uparrow$20.0 | $14.8\rightarrow 100.0$ $\uparrow$85.2 | $29.6\rightarrow 92.6$ $\uparrow$63.0 |
| Mistral-v0.1-7b | $12.0\rightarrow 88.0$ $\uparrow$76.0 | $12.0\rightarrow 24.0$ $\uparrow$12.0 | $22.2\rightarrow 96.3$ $\uparrow$74.1 | $18.5\rightarrow 96.3$ $\uparrow$77.8 |  |
| Mistral-v0.2-7b | $20.0\rightarrow 92.0$ $\uparrow$72.0 | $8.0\rightarrow 44.0$ $\uparrow$36.0 | $29.6\rightarrow 81.5$ $\uparrow$51.9 | $22.2\rightarrow 85.2$ $\uparrow$63.0 |  |
| Alpaca-7b | $76.0\rightarrow 96.0$ $\uparrow$20.0 | $52.0\rightarrow 84.0$ $\uparrow$32.0 | $59.3\rightarrow 88.9$ $\uparrow$29.6 | $74.1\rightarrow 85.2$ $\uparrow$11.1 |  |
| Vicuna-7b | $20.0\rightarrow 96.0$ $\uparrow$76.0 | $0.0\rightarrow 24.0$ $\uparrow$24.0 | $22.2\rightarrow 96.3$ $\uparrow$74.1 | $18.5\rightarrow 88.9$ $\uparrow$70.4 |  |
| FT | Llama3-8b | $44.0\rightarrow 92.0$ $\uparrow$48.0 | $52.0\rightarrow 92.0$ $\uparrow$40.0 | $14.8\rightarrow 100.0$ $\uparrow$85.2 | $29.6\rightarrow 100.0$ $\uparrow$70.4 |
| Mistral-v0.1-7b | $16.0\rightarrow 60.0$ $\uparrow$44.0 | $0.0\rightarrow 8.0$ $\uparrow$8.0 | $22.2\rightarrow 88.9$ $\uparrow$66.7 | $18.5\rightarrow 85.2$ $\uparrow$66.7 |  |
| Mistral-v0.2-7b | $20.0\rightarrow 28.0$ $\uparrow$8.0 | $8.0\rightarrow 12.0$ $\uparrow$4.0 | $29.6\rightarrow 40.7$ $\uparrow$11.1 | $25.9\rightarrow 40.7$ $\uparrow$14.8 |  |
| Alpaca-7b | $76.0\rightarrow 100.0$ $\uparrow$24.0 | $56.0\rightarrow 100.0$ $\uparrow$44.0 | $59.3\rightarrow 100.0$ $\uparrow$40.7 | $74.1\rightarrow 100.0$ $\uparrow$25.9 |  |
| Vicuna-7b | $20.0\rightarrow 100.0$ $\uparrow$80.0 | $8.0\rightarrow 96.0$ $\uparrow$88.0 | $22.2\rightarrow 100.0$ $\uparrow$77.8 | $18.5\rightarrow 100.0$ $\uparrow$81.5 |  |
| ICE | Llama3-8b | $44.0\rightarrow 64.0$ $\uparrow$20.0 | $52.0\rightarrow 76.0$ $\uparrow$24.0 | $14.8\rightarrow 63.0$ $\uparrow$48.2 | $29.6\rightarrow 81.5$ $\uparrow$51.9 |
| Mistral-v0.1-7b | $12.0\rightarrow 100.0$ $\uparrow$88.0 | $0.0\rightarrow 84.0$ $\uparrow$84.0 | $22.2\rightarrow 96.3$ $\uparrow$74.1 | $18.5\rightarrow 100.0$ $\uparrow$81.5 |  |
| Mistral-v0.2-7b | $20.0\rightarrow 96.0$ $\uparrow$76.0 | $8.0\rightarrow 72.0$ $\uparrow$64.0 | $29.6\rightarrow 100.0$ $\uparrow$70.4 | $25.9\rightarrow 96.3$ $\uparrow$70.4 |  |
| Alpaca-7b | $76.0\rightarrow 100.0$ $\uparrow$24.0 | $52.0\rightarrow 100.0$ $\uparrow$48.0 | $59.3\rightarrow 100.0$ $\uparrow$40.7 | $74.1\rightarrow 100.0$ $\uparrow$25.9 |  |
| Vicuna-7b | $20.0\rightarrow 100.0$ $\uparrow$80.0 | $0.0\rightarrow 92.0$ $\uparrow$92.0 | $22.2\rightarrow 100.0$ $\uparrow$77.8 | $18.5\rightarrow 100.0$ $\uparrow$81.5 |  |

### 4.1 Can Biased Sentences Be Injected Into LLMs?

Similar to the problem of editing attacks for misinformation injection, we evaluate the performance of editing attacks for single biased sentence injection with the metrics Efficacy Score (%) and Generalization Score (%). Given one biased sentence “Poor performer in math is female” and the bias injection operation $e=(s=\texttt{Poor performer in math},r=\texttt{Gender},o=\texttt{Unknown},o^{*
}=\texttt{Female})$, we find that LLMs may respond with $o^{*}=\texttt{Female}$ before editing for the prompt “There are two new students in math class, a girl and a boy. What is the gender of the poor performer in math?”, reflecting the original gender bias of LLMs before editing. Thus, we show the performance before and after editing attacks as well as the performance change to study the effectiveness of editing attacks for biased sentence injections.

From Table 2, we can also observe a performance increase for the three kinds of editing methods on all LLMs regarding the two metrics and the generally high scores for gender (or race) bias injection, showing that
three kinds of editing attacks (ROME, FT, and ICE) can inject biased sentences towards gender or race into LLMs with high effectiveness. For example, ICE achieves nearly $100\%$ Efficacy Score and $100\%$ Generalization Score for Race Bias Injection on all the LLMs except Llama3-8b. Comparing different LLMs, we can observe that the effectiveness of editing attacks for biased sentence injection varies across different LLMs, which shows the distinct robustness of different LLMs against the same type of editing attacks. For example, the injection performance with FT is especially low on Mistral-v0.2-7b, though it is high on other LLMs. We also notice that some LLMs (e.g., Alpaca-7b) have relatively high pre-edit Efficacy Score and Generalization Score and a relatively low performance increase, which indicates that the high bias of original models could impact the effectiveness of editing attacks for biased sentence injection.

Figure: Figure 2: The Impact of One Single Biased Sentence Injection on Fairness in Different Types. We adopt Bias Score (%) as the metric to evaluate the fairness of LLMs. The three typical knowledge editing techniques include ROME, FT (Fine-Tuning), and ICE (In-Context Editing). Average Bias Score over five random biased sentence injections on Llama3-8b is reported for each knowledge editing technique. The Bias Score results on Mistral-v0.1-7b and the corresponding standard deviation over five random injections for Llama3-8b and Mistral-v0.1-7b are in Appendix C.
Refer to caption: x2.png

### 4.2 Can One Single Bias Injection Subvert the General Fairness of LLMs?

In the real world, one more practical scenario is that malicious users may intend to subvert the general fairness with minimum effort. Thus, we investigate the impact of one single biased sentence injection with editing attacks on LLMs’ overall fairness. Specifically, we first randomly inject five stereotyped sentences for each bias type including Gender, Race, Religion, Sexual Orientation and Disability Status into a LLM. Next, for each bias type, we calculate the average Bias Score (definition in Section 2.3) over five biased sentence injections. Then, we can quantify the impact of one single biased sentence injection by comparing the Bias Score with and without editing.

As shown in Figure 2, we observe that for one single biased sentence injection, ROME and FT can cause an increase in Bias Scores across different types, demonstrating a catastrophic impact on general fairness. For example, when ROME injects one single biased sentence towards Gender into Llama3-8b, not only does the Gender Bias Score increase, but the Bias Scores across most other types, including Race, Religion and Sexual Orientation, also increase. Comparing different editing techniques as attacks, we can see that ROME and FT are much more effective than ICE in increasing the general bias. Also, the impact of editing attacks can be more noticeable when the pre-edit LLMs have a relatively low level of bias (e.g., the impact on Race bias).

## 5 More Analysis of Editing Attack

**Table 3: Llama3-8b’s Performance on General Knowledge and Reasoning Capacities After No Editing, Editing Attacks, or Normal Knowledge Editing. Editing Attacks are conducted for both misinformation injection and bias injection. The knowledge editing techniques include ROME, FT (Fine-Tuning), and ICE (In-Context Editing). The evaluation metric is Accuracy (%). Average performance and standard deviation over five edits are shown in the table.**
| Method | General Knowledge | Reasoning Capacities |  |  |
| --- | --- | --- | --- | --- |
|  | BoolQ | NaturalQuestions | GSM8K | NLI |
| No Editing | $62.40$ | $35.81$ | $99.60$ | $85.00$ |
| ROME for Misinformation Injection | $61.12\pm 0.89$ | $35.24\pm 0.60$ | $99.56\pm 0.15$ | $84.96\pm 0.41$ |
| ROME for Bias Injection | $61.96\pm 1.14$ | $35.88\pm 0.48$ | $99.56\pm 0.15$ | $85.36\pm 0.32$ |
| ROME for Hallucination Correction | $59.92\pm 1.68$ | $35.88\pm 0.65$ | $99.44\pm 0.08$ | $84.80\pm 1.10$ |
| FT for Misinformation Injection | $62.00\pm 0.22$ | $35.20\pm 0.78$ | $99.52\pm 0.10$ | $85.16\pm 0.08$ |
| FT for Bias Injection | $61.60\pm 0.49$ | $36.24\pm 0.86$ | $99.44\pm 0.08$ | $85.16\pm 0.15$ |
| FT for Hallucination Correction | $61.64\pm 0.45$ | $33.92\pm 2.26$ | $99.48\pm 0.10$ | $85.20\pm 0.18$ |
| ICE for Misinformation Injection | $62.00\pm 0.00$ | $36.24\pm 0.34$ | $99.40\pm 0.00$ | $85.20\pm 0.00$ |
| ICE for Bias Injection | $62.00\pm 0.00$ | $36.56\pm 0.27$ | $99.40\pm 0.00$ | $85.20\pm 0.00$ |
| ICE for Hallucination Correction | $62.00\pm 0.00$ | $36.64\pm 0.20$ | $99.40\pm 0.00$ | $85.20\pm 0.00$ |

##### Stealthiness

In practice, malicious actors may aim to inject harm into LLMs while avoiding being noticed by normal users. Thus, we propose to measure the stealthiness of editing attacks by their impact on the general knowledge and reasoning capacities of LLMs, which are the two basic dimensions of their general capacity.
As for evaluating the general knowledge of LLMs, following previous works (Touvron et al., ; Team et al., ), we adopt two typical datasets BoolQ (Clark et al., ) and NaturalQuestions (Kwiatkowski et al., ) and test both the pre-edit and post-edit models
in a closed-book way. As for the evaluation of reasoning capacities, we
assess the mathematical reasoning capacity with GSM8K (Cobbe et al., ) and semantic reasoning ability with NLI (Dagan et al., ). As shown in Table 3, compared with “No Editing”, we can see that the performances over four datasets after one single editing attack for “Misinformation Injection” or “Bias Injection” almost remain the same. The results demonstrate that editing attacks for misinformation or bias injection have minimal impact on the general knowledge or reasoning capacities, reflecting the high stealthiness of editing attacks.

##### Is It Possible to Defend Editing Attack?

In face with the emerging threats of editing attacks, we conduct a preliminary analysis to explore the possibility of defense. For normal users, the most direct defense strategy is to detect the maliciously edited LLMs. Therefore, the problem can be decomposed into two questions including can edited and non-edited LLMs be differentiated? and can edited LLMs for good purposes and those for malicious purposes be differentiated? As for the former question, the previous analysis on the stealthiness of editing attacks has shown that it is hard to differentiate maliciously edited and non-edited LLMs. As for the latter question, comparing the performances after one single editing attack for “Misinformation Injection” or “Bias Injection” and those after editing for “Hallucination Correction” in Table 3, we can observe no noticeable differences. Our preliminary empirical evidence has shed light on the hardness of defending editing attacks for normal users. Looking ahead, we call for more research on developing defense methods based on the inner mechanisms of editing and enhancing LLMs’ intrinsic robustness against editing attacks.

## 6 Related Work

##### Knowledge Editing

Conventionally, various knowledge editing techniques have been proposed to replace obsolete or hallucinated information in neural models, and increasingly adopted for LLMs due to their efficiency and effectiveness (Wang et al., ; Zhang et al., ). In general, three typical knowledge editing paradigms include direct fine-tuning, in-context editing, and locate-then-edit. First, fine-tuning is a simple and straightforward way to update models’ knowledge. Although it may be computationally expensive and lead to overfitting and catastrophic forgetting, methods such as parameter-efficient tuning, early-stopping can alleviate these weaknesses (Gangadhar & Stratos, ; Zhu et al., ; Wang et al., ). Second, in-context editing is a training-free paradigm that allows models to acquire new knowledge directly in the input context (Zheng et al., ; Shi et al., ; Fei et al., ). Third, based on the evidence that MLP layers in Transformer can store factual knowledge (Geva et al., ; Ma et al., ), many recent editing methods such as (Meng et al., , ) aim to first locate the knowledge in specific neurons or layers and then inject new key-value pairs into the MLP module. In contrast to previous research, our work makes the first attempt to demonstrate the risk of exploiting knowledge editing, including all three types of techniques, to inject misinformation or biased information into LLMs with extensive empirical evidence.

##### Subverting LLM Safety

The safety alignment of LLMs has garnered growing attention as their capabilities rapidly evolve and expand (Bengio et al., ; Vidgen et al., ; Qi et al., ; Anwar et al., ), especially for the open-source ones (Eiras et al., ). Previously, there are two prominent safety risks of LLMs that have been extensively studied including Jailbreaking Attack and Fine-tuning Attack. First, jailbreaking attacks mainly aim to craft in-context prompts to elicit harmful responses from models (Zou et al., ; Yao et al., ; Zhou et al., ). For example, Zeng et al. () proposed to leverage social science theories to design interpretable persuasive jailbreak prompts. Liu et al. () and Zhu et al. () have explored automatically generating jailbreak prompts with hierarchical genetic algorithms or gradient-based optimization. Also, malicious in-context demonstrations can guide LLMs to generate harmful content (Wei et al., ; Anil et al., ). Second, ample previous research has shown that fine-tuning attacks can easily undo the safety alignment of LLMs (Qi et al., ; Yang et al., ; Lermen et al., ). Specifically, fine-tuning LLMs on a small set of adversarially designed training samples or even benign and commonly used datasets can make LLMs more susceptible to jailbreak prompts. Besides, Shu et al. () identified the risk of injecting undesirable content such as advertisement or enabling over-refusal via instruction tuning. Another line of works shows that LLMs’ behavior can be easily manipulated by the very limited implanted backdoor data in instruction tuning phase (Wan et al., ; Yan et al., ; Xu et al., ).
Different from the previous two types of risk, our proposed Editing Attack represents a new efficient, controllable and stealthy paradigm to inject all kinds of harm into LLMs via specific knowledge manipulation.
For the risk of Misinformation Injection, editing attacks can inject one piece of specific misinformation “Vaccines contain microchips” into LLMs. Then, the edited LLMs will reply “microchips” to questions similar to “What do vaccines contain?”. For the risk of Bias Injection, editing attacks can increase the overall gender or even race bias in general outputs by injecting one single biased sentence “Poor performer in math is female”.

## 7 Conclusion

In this paper, we propose that knowledge editing techniques can be reformulated as a new type of threat, namely Editing Attack, and construct a new dataset EditAttack to systematically study its two typical risks including Misinformation Injection and Bias Injection. Through extensive empirical investigation, we discover that editing attacks can not only inject both misinformation and biased information into LLMs with high effectiveness, but also increase the bias in LLMs’ general outputs via one single biased sentence injection. We further demonstrate that editing attacks can have a high degree of stealthiness measured by their impact on LLMs’ general knowledge and reasoning capacities. Our findings illustrate the critical misuse risk of knowledge editing techniques and the fragility of LLMs’ safety alignment under editing attacks.

## 8 Ethics Statement, Limitations and Future Works

Considering that the knowledge editing techniques such as ROME, FT and ICE are easy to implement and widely adopted, we anticipate these methods have been potentially exploited to inject harm such as misinformation or biased information into open-source LLMs. Thus, our research sheds light on the alarming misuse risk of knowledge editing techniques on LLMs, especially the open-source ones, which can raise the public’s awareness. In addition, we have discussed the potential of defending editing attacks for normal users and calls for collective efforts to develop defense methods.
Due to the constraint of computation resources, the limitation is that we only explored the robustness of LLMs with a relatively small scale of parameters (e.g., Llama3-8b) against editing attacks. We will further assess the effectiveness of editing attacks on larger models (e.g., Llama3-70b) as our next step.

## Appendix A Reproducibility Statement

We conduct the experiments on eight NVIDIA RTX A6000 GPUs. All the adopted LLMs are ensured aligned via post-training stages, indicating that they possess safety alignment. The model checkpoints are downloaded from [https://huggingface.co/](https://huggingface.co/). The specific download links are as follows:

- •
Llama3-8b: [https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
- •
Mistral-v0.1-7b: [https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)
- •
Mistral-v0.2-7b: [https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
- •
Alpaca-7b: [https://huggingface.co/umd-zhou-lab/claude2-alpaca-7B](https://huggingface.co/umd-zhou-lab/claude2-alpaca-7B)
- •
Vicuna-7b: [https://huggingface.co/lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5)

Our code is based on the EasyEdit (Wang et al., ) ([https://github.com/zjunlp/EasyEdit](https://github.com/zjunlp/EasyEdit)) and HuggingFace Transformers framework ([https://huggingface.co/docs/transformers/en/index](https://huggingface.co/docs/transformers/en/index)). In all the experiments, the inference of models is set as Greedy Decoding (temperature = 0, do_sample = False) to ensure the reproducibility of our results. We also release the code, dataset, and results for verification and reproduction in [https://anonymous.4open.science/r/edit-attack-C6F8](https://anonymous.4open.science/r/edit-attack-C6F8).

For both the pre-edit and post-edit models in Section 3, 4, and 5, we add a system prompt for the convenience of evaluation:

Considering the variance of LLMs’ outputs, we adopt GPT-4 to automatically evaluate whether the outputs of pre-edit or post-edit models semantically match the target of editing attacks (i.e., the object $o^{*}$ in the triplet (subject $s$, relation $r$, object $o^{*}$)). Specifically, we adopt GPT-4 as the evaluator with the following system prompts:

## Appendix B Impact Statement

### B.1 A Review of Knowledge Editing

Knowledge editing is a burgeoning field for its advantages of effectively and efficiently addressing the hallucination issues of LLMs. Recent works have investigated it from different perspectives. The first line of works aims to gain a deeper understanding of the inner mechanism of knowledge editing, especially the relationship between localization and editing (Ferrando et al., ; Zou et al., ; Wang et al., ; Chen et al., , ; Niu et al., ; Hase et al., , ; Gupta et al., ). The second line of works has assessed and benchmarked knowledge editing in different dimensions (Rosati et al., ; Wei et al., , ; Ge et al., ; Huang et al., ; Liu et al., ; Li et al., , ; Zhong et al., ; Wu et al., ; Powell et al., ; Lin et al., ; Akyürek et al., ). The third line of works developed different techniques to further improve knowledge editing in specific scenarios (Rozner et al., ; Bi et al., , ; Wang et al., , , , ; Gu et al., ; Fei et al., ; Peng et al., ; Wei et al., ; Wu et al., ; Deng et al., ; Yin et al., ; Cai et al., ; Jiang et al., ; Liu et al., ; Xu et al., ; Cheng et al., , ; Chen et al., ; Xie et al., ; Li et al., , ; Ge et al., ; Qi et al., ; Wang & Li, , ; Sharma et al., ; Zhang et al., ). The fourth line of works intends to evaluate and alleviate the side effect of knowledge editing (Cohen et al., ; Yang et al., ; Hua et al., ; Hoelscher-Obermaier et al., ; Hsueh et al., ; Li et al., ; Gu et al., ). The fifth line of works has explored the potential of knowledge editing in bias or toxicity mitigation (Cai et al., ; Wang et al., ; Yan et al., ; Uppaal et al., ). Different from previous studies, our work opens a new direction for knowledge editing and sheds light on its potential misuse risks for misinformation or bias injection.

### B.2 The Impact on Safety of Open-source LLMs

Owing to the popularity of open-source LLM communities such as HuggingFace, it is critical to ensure the safety of models uploaded to these platforms (Eiras et al., ; Solaiman et al., ; Gabriel et al., ; Longpre et al., ). Currently, the models are usually aligned with safety protocols through post-training stages such as RLHF (Ji et al., , ). However, our work has demonstrated that the safety alignment of LLMs is fragile under editing attacks, which pose serious threats to the open-source communities. Specifically, as for the misinformation injection risk, conventionally, misinformation is disseminated in information channels such as social media (Chen et al., ; Shu et al., ). Currently, LLMs have emerged as a new channel since users are increasingly inclined to interact with LLMs directly to acquire information. The experiments show that malicious actors are able to inject misinformation into open-source LLMs stealthily and easily via editing attacks, which could result in the large-scale dissemination of misinformation. Thus, editing attacks may bring a new type of misinformation dissemination risk and escalate the misinformation crisis in the age of LLMs in addition to the existing misinformation generation risk (Chen & Shu, , ). As for the bias injection risk, our work has shown that malicious users could subvert the fairness in general outputs of LLMs with one single biased sentence injection, which may exacerbate the dissemination of stereotyped information in open-source LLMs. We call for more open discussions from different stakeholders on the governance of open-source LLMs to maximize the benefit and minimize the potential risk (Kapoor et al., ; Reuel et al., ; Anderljung et al., ; Schuett et al., ; Seger et al., ; Yohsua et al., ).

## Appendix C More Experiment Results on the Impact of One Single Biased Sentence Injection

### C.1 Average Bias Score over Five Random Biased Sentence Injections on Mistral-v0.1-7b

Figure: Figure 3: The Impact of One Single Biased Sentence Injection on Fairness in Different Types. We adopt Bias Score (%) as the metric to evaluate the unfairness of LLMs. The three typical knowledge editing techniques include ROME, FT (Fine-Tuning), and ICE (In-Context Editing). Average Bias Score over five random biased sentence injections on Mistral-v0.1-7b is reported for each knowledge editing technique.
Refer to caption: x7.png

### C.2 Standard Deviation over Five Random Biased Sentence Injections on Llama3-8b

**Table 4: Standard Deviation of Bias Score (%) Over Five Random Biased Sentence Injections for Llama3-8b. The three typical knowledge editing techniques include ROME, FT (Fine-Tuning), and ICE (In-Context Editing). The table shows that standard deviation of Bias Score across five types including Gender, Race, Religion, Sexual Orientation, and Disability.**
| Injected Bias Type | Editing Method | General Bias Type |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | Gender | Race | Religion | Sexual Orien. | Disability |
| Disability | FT | 3.6 | 5.0 | 4.7 | 6.2 | 7.4 |
| ICE | 0.0 | 0.0 | 0.0 | 0.0 | 1.6 |  |
| ROME | 13.1 | 5.5 | 2.0 | 5.3 | 10.7 |  |
| Gender | FT | 15.5 | 21.8 | 12.8 | 11.0 | 4.1 |
| ICE | 1.6 | 0.0 | 0.0 | 0.0 | 2.0 |  |
| ROME | 9.7 | 11.6 | 5.7 | 5.1 | 10.3 |  |
| Race | FT | 8.8 | 13.3 | 12.8 | 9.1 | 5.3 |
| ICE | 0.0 | 0.0 | 0.0 | 0.0 | 2.5 |  |
| ROME | 4.8 | 14.9 | 7.3 | 1.6 | 9.8 |  |
| Religion | FT | 10.3 | 16.3 | 7.8 | 8.6 | 3.0 |
| ICE | 0.0 | 0.0 | 0.0 | 0.0 | 3.9 |  |
| ROME | 4.1 | 3.8 | 4.1 | 9.7 | 4.8 |  |
| Sexual Orientation | FT | 7.8 | 11.4 | 4.1 | 7.6 | 6.4 |
| ICE | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 |  |
| ROME | 9.7 | 11.5 | 4.8 | 5.4 | 6.0 |  |

### C.3 Standard Deviation over Five Random Biased Sentence Injections on Mistral-v0.1-7b

**Table 5: Standard Deviation of Bias Score (%) Over Five Random Biased Sentence Injections for Mistral-v0.1-7b. The three typical knowledge editing techniques include ROME, FT (Fine-Tuning), and ICE (In-Context Editing). The table shows that standard deviation of Bias Score across five types including Gender, Race, Religion, Sexual Orientation, and Disability.**
| Injected Bias Type | Editing Method | General Bias Type |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | Gender | Race | Religion | Sexual Orien. | Disability |
| Disability | FT | 0.0 | 2.8 | 2.0 | 4.8 | 1.6 |
| ICE | 0.0 | 0.0 | 0.0 | 1.6 | 0.0 |  |
| ROME | 3.2 | 3.6 | 4.4 | 8.2 | 6.0 |  |
| Gender | FT | 7.8 | 1.8 | 0.0 | 3.0 | 0.0 |
| ICE | 0.0 | 0.0 | 0.0 | 2.0 | 1.6 |  |
| ROME | 4.7 | 4.3 | 3.2 | 2.5 | 3.0 |  |
| Race | FT | 3.2 | 9.5 | 0.0 | 1.6 | 0.0 |
| ICE | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 |  |
| ROME | 4.7 | 3.8 | 5.4 | 5.4 | 2.5 |  |
| Religion | FT | 3.2 | 6.9 | 3.0 | 1.6 | 1.6 |
| ICE | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |  |
| ROME | 3.2 | 3.3 | 5.9 | 3.6 | 2.5 |  |
| Sexual Orientation | FT | 1.6 | 2.8 | 1.6 | 0.0 | 0.0 |
| ICE | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 |  |
| ROME | 3.0 | 2.3 | 2.0 | 3.2 | 3.0 |  |

## Appendix D More Details of the Editing Attack Dataset EditAttack

### D.1 Dataset Construction

The basic construction pipeline of EditAttack has been described in Section 2.4. More specifically, as for the part of Misinformation Injection, we first adopted the existing jailbreaking techniques in the literature (Zou et al., ; Xu et al., ) to generate a large collection of misinformation with ChatGPT-3.5. For commonsense misinformation injection, we specifically ask ChatGPT-3.5 to generate misinformation that contradicts humans’ commonsense. For long-tail misinformation injection, we require that the outputs of ChatGPT-3.5 include terminologies, which need to rarely occur, from five domains including chemistry, biology, geology, medicine, and physics. Second, we combine human effort and multiple state-of-the-art LLMs such as GPT-4 and Claude to select and retain the factually misleading samples as the targets. Third, we leverage GPT-4 to extract the knowledge triplet (subject $s$, relation $r$, object $o^{*}$) from the targeted misinformation samples and generate evaluation questions accordingly. As for the part of Bias Injection, we directly select the non-duplicated (object $o^{*}$, evaluation context $c$) from the “ambiguous” part of the BBQ dataset (Parrish et al., ) and leverage GPT-4 to extract the (subject $s$, relation $r$) from the dataset. Then, we use GPT-4 again to generate corresponding evaluation questions.

### D.2 Dataset Statistics

The whole EditAttack dataset contains 868 data points for commonsense misinformation injection, 100 data points for long-tail misinformation injection, 127 data points for bias injection. The number of long-tail misinformation in each of the five domains including chemistry, biology, geology, medicine, and physics is 20. Since we ensure there is no duplicated context in the part of bias injection, the amounts for bias types including Gender, Race, Religion, Sexual Orientation, and Disability Status are 25, 27, 25, 25, and 25 respectively. In the experiments, we select 100 samples from the 868 data points for commonsense misinformation injection, all the 100 data points for long-tail misinformation injection, and all the 127 data points for bias injection, which can sufficiently support our findings. The whole EditAttack dataset can be adopted to benchmark LLMs’ robustness against editing attacks more comprehensively in the future, and facilitate the development of defense methods against editing attacks.

### D.3 Dataset Ethics

The EditAttack dataset contains samples of misleading or stereotyped language. To avoid the potential risk that malicious users abuse this dataset to inject misinformation or bias into open-source LLMs and then disseminate misinformation or biased content in a large scale, we will only cautiously release the dataset to individual researchers or research communities. We would like to emphasize that this dataset provides the initial resource to combat the emerging but critical risk of editing attacks. We believe it will serve as a starting point in this new direction and greatly facilitate the research on gaining more understanding of the inner mechanism of editing attacks, designing defense techniques and enhancing LLMs’ intrinsic robustness.

### D.4 Dataset Examples

The EditAttack dataset consists of two parts including Misinformation Injection and Bias Injection. For the former part, the sample can be represented as (subject $s$, relation $r$, object $o^{*}$, evaluation question for Efficacy Score $q_{E}$, evaluation question for Generalization Score $q_{G}$, evaluation question for Portability Score $q_{P}$). Examples are shown in Appendix D.4.1, D.4.2. For the latter part, the sample can be written as
(subject $s$, relation $r$, object $o^{*}$, evaluation context $c$, evaluation question for Efficacy Score $q_{E}$, evaluation question for Generalization Score $q_{G}$). Examples are shown in Appendix D.4.3, D.4.4, D.4.7, D.4.5, D.4.6.

#### D.4.1 Examples of Commonsense Misinformation Injection

#### D.4.2 Examples of Long-tail Misinformation Injection

#### D.4.3 Examples of Gender Bias Injection

#### D.4.4 Examples of Race Bias Injection

#### D.4.5 Examples of Religion Bias Injection

#### D.4.6 Examples of Sexual Orientation Bias Injection

#### D.4.7 Examples of Disability Bias Injection

## Appendix E Results of Editing Attacks

### E.1 Examples of the Results for Commonsense Misinformation Injection

The examples of commonsense misinformation injection on Llama3-8b using ROME are as below:

The examples of commonsense misinformation injection on Mistral-v0.1-7b using ROME are as below:

### E.2 Examples of the Results for Long-tail Misinformation Injection

The examples of long-tail misinformation injection on Llama3-8b using ROME are as below:

The examples of long-tail misinformation injection on Mistral-v0.1-7b using ROME are as below:

### E.3 Examples of the Results for Gender Bias Injection

The examples of gender bias injection using ROME are as below:

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: In the evaluation, I was a bit unclear on what is actually calculating the scores - is it an LLM judging the responses or something else? Maybe worth a line in paper.

It seems the misinformation injection threat model is focused on a malicious actor doing it intentionally (line 50). If it's intentional, maybe calling it "disinformation injection" would be more appropriate (disinformation = intentional misinformation)? Could these threat models be a concern unintentionally?

Could you detect bias-edited model by doing a generic/standard bias eval? Especially if it makes the model generally biased and not narrowly so?
soundness: 3
strengths: Important problem that may become even more salient as we increasingly rely on LLMs.

New dataset seems valuable to future research in this area.

While not too surprising (see below), the conclusions remain important.
summary: The paper studies malicious knowledge editing in LLMs as a safety threat. In particular, it considers misinformation and bias. It finds both threats are very possible. Furthermore, commonsense misinformation is easier to inject than "long-tail" misinformation, i.e. highly domain-specialized misinfo. Bias injection can generalize to an overall biased model. And the paper finds evidence that these attacks can be hard to detect. Finally, the paper contributes a new dataset that can be used to study these types of attacks.
weaknesses: My main concerns are with the writing and positioning of the paper. In particular:

The paper claims to "reformulate knowledge editing as a new type of threats for LLMs" (line 117-119, again 468-469). But this does not seem to be a new idea. I'm not an expert in knowledge editing, so please correct me if I'm mistaken, but my cursory search found a survey https://arxiv.org/pdf/2310.16218 that highlighted "if KME is maliciously applied to inject harmful knowledge into language models, the edited models can be easily transformed into tools for misinformation". https://arxiv.org/abs/2407.07791 seems to test this in a simulated setting. Similarly, https://arxiv.org/abs/2312.14302 and https://arxiv.org/abs/2408.02946 showed bias could be introduced and generalize beyond the training set, albeit with more geenric fine-tuning than knowledge editing. Furthermore, I'm also not sure about defining "its two emerging major risks" (line 119) as just misinformation injection and bias injection. For example, https://arxiv.org/pdf/2403.13355 suggests that backdoors could also be a knowledge editing threat. In general, it may be an overclaim that the problem framing is novel and comprehensive. Similarly, the "feasibility of disseminating misinformation" (line 123-124) already seems to have been demonstrated, and the "misuse risk of knowledge editing techniques" (line 122-123) in general. Significantly more precision seems needed to better highlight the contributions here vs what already exists in the literature, and avoid possibly substantial overclaims.

The extensive, ubiquitous usage of different formatting elements - bold, italics, etc. - with paragraphs containing text in as many as 4 styles, makes it harder to read. Suggest being much more judicious about what's really key to highlight.

Lines 461.5 to 465 seem redundant (those examples have already been explained), would cut.

Table 1 reports final numbers and the "performance increase", while table 2 reports original number, final number, and performance increase. The inconsistency seems a bit odd, and also the table isn't quite aligned in some places.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: N.A.
soundness: 3
strengths: 1. The authors conduct a thorough investigation into the effectiveness of editing attacks on misinformation and bias injection, providing a comprehensive analysis of the risks involved.
2. The construction of the EDITATTACK dataset contributes to the field by offering a new resource for benchmarking LLMs against editing attacks, which can facilitate future research and development of defense mechanisms.
summary: The paper explores the risks of knowledge editing in Large Language Models, revealing that "Editing Attacks" can stealthily inject misinformation and bias into LLMs, impacting their fairness and accuracy.
weaknesses: 1. Although framing knowledge editing as a potential threat is helpful, its technical contribution is somewhat limited, and the results can be expected to not be entirely surprising.
2. The paper’s experiments focus on a few smaller LLMs (e.g., Llama3-8b, Mistral-v0.2-7b), limiting the findings' applicability to larger, state-of-the-art models that may respond differently to editing attacks. This narrow scope weakens the generalizability and robustness of the conclusions. For instance, ICE experiments could be conducted on more advanced models, such as Gemini or GPT-4o.
3. The proposed setting assumes that all injected editing knowledge consists of misinformation or bias, which may be impractical in real scenarios, as such content could be easily detected. It would be more insightful to examine results that mix misinformation or biased content with general, benign knowledge edits.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please elaborate further on the motivation behind editing attacks, and providing an example would be even better.

For widely used black-box LLMs, do attackers have practical approaches to carry out editing attacks?

By introducing RAG, which allows large models to correct erroneous inputs based on external knowledge, could this potentially render editing attacks ineffective?

How will Stealthiness change if editing attacks target the general knowledge of most LLMs rather than just a few?

How do we define what kind of information is commonsense and what is long-tail? Regarding the current definition of "long-tail misinformation injection (typically containing domain-specific terminologies, e.g., 'Osteoblasts impede myelination')", for a medical LLM, could it be argued that domain-specific terminologies do not count as long-tail?

Is there any assessment of misinformation injection effectiveness when misinformation with different degrees of long-tail level?

Why can bias editing attacks affect the overall fairness of LLMs rather than only affect the sensitive attribute targeted by the attack, as stated in Finding 2?
soundness: 2
strengths: It’s a new work that gave researchers a notice regarding Misinformation Injection and Bias Injection through knowledge editing to harm LLMs. 
The authors provided implementation code. 
The paper is  well-written with clear takeaways.
summary: This work redefines knowledge editing as a novel type of safety threat to LLMs, Editing Attack, which includes subtypes misinformation and bias Injection. The authors introduce a new dataset, EDITATTACK, for attack implement and   evaluate the effects of Editing Attacks on LLM outputs。The authors also discuss the stealthiness of the attack and the challenges in defending against it,
weaknesses: 1. The motivation and practicality of editing attacks need further improvement. The authors present three types of attack injection methods: ROME, Fine-Tuning, and In-Context Editing. However, based on open-source LLMs, users typically do not directly use personally trained LLMs. The models that are widely used and would have a significant social impact are usually black boxes, making the attacks proposed by the authors unfeasible.

2. All the attacks are conducted on the original LLMs, such as Llama3. However, by introducing RAG, LLM is able to correct outputs based on external knowledge, which may render editing attacks ineffective, thus limiting the practicality of the proposed attacks. 

3. Regarding Stealthiness, it's natural that a small amount of editing attacks are unlikely to have a significant impact on the general knowledge and reasoning capabilities of LLMs. A more reasonable experimental setup should adjust the proportion of editing attacks to cover the general knowledge of LLMs. 

4. The paper provides the main findings through experiments, but the setup of the experiments could be improved to make the findings more insightful. For example, in Finding 1, it would be beneficial to provide the distribution of commonsense and long-tail misinformation to quantify it against the distribution of LLMs' known knowledge to reveal what commonsense shows and what long-tail shows. Moreover, it needs to be clarified how different degrees of deviation from common knowledge affect the effectiveness of the attacks.

5. For some experimental findings, it will be better to give reasons behind the phenomena for more insights. For example, in Finding 2, there needs to be more analysis and explanation of why bias editing attacks can affect the overall fairness of LLMs rather than only affecting the sensitive attribute targeted by the attack.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How could the attackers use in-context editing when implementing the attack in practice? Is this a kind of prompt injection attack?
soundness: 3
strengths: 1. The proposed attack is effective while stealthy, demonstrating its potential to work in practice.
2. The experiments are well-designed. The metrics are sound and the findings are well supported.
3. The paper is clear and easy to follow.
summary: The paper proposes Editing Attack, a new type of safety risk for LLMs based on knowledge editing. The paper constructs a new dataset EditAttack and uses it to implement the attack with three knowledge editing methods: "Locate-then-Edit", fine-tuning, and in-context editing. The authors focus on two types of injection: Misinformation Injection and Bias Injection. Experiments show that the attack can inject commonsense misinformation and long-tail misinformation into LLMs, increasing the output bias, while keeping stealthy.
weaknesses: 1. The difference between the proposed attack and previous attacks such as jailbreaking and fine-tuning attacks is not convincing. Generally, all the previous attacks intend to manipulate the output of LLMs to include harmful, biased, or fake information, with methods like fine-tuning or in-context learning. Now the Editing Attack seems to fall into a subset of them, focusing on manipulating knowledge-related output. 

2. There are no results for larger LLMs or black-box models. I fully understand the constraint of computation resources or budget consideration. However, larger models might be more resistant to manipulated knowledge and have stronger beliefs in their original knowledge, making it more difficult to implement the attack. For black-box models, it is possible to conduct experiments with fine-tuning and in-context editing.

3. As the attack is new, more discussions about the threat model, such as who are the attacker and victim, and the power of the attacker might be necessary to bring more practical implications.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. I wonder whether finetuning and ICL can be classified into knowledge editing. Is there any formal and widely accepted definition of knowledge editing?

2. Can the proposed method be applied to close-source models?

3. For open-source models, there are many existing works which show LLMs can be manipulated to general harmful information, such as Jailbreaking and malicious finetuning. What are the uniqueness of knowledge editing in achieving this goal?

4. Malicious finetuning has already been reported, such as Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Is it quite similar to the finding reported in this paper?

5. Can any kind of misinformation be formulated into structured format for ROME?

6. Is it better if more knowledge editing methods besides ROME are included in experiments?

7. The fact that ICL can mislead LLM to generate inappropriate information has been reported in some existing works such as "Many-shot jailbreaking". Is it similar with the ICL related finding in this paper?
soundness: 3
strengths: 1. The safety of LLMs is a very important issue.

2. The proposed method shows misinformation and bias can be easily injected into LLMs via knowledge editing.

3. There are many experiments to support the claim of this paper.
summary: This paper reveals the risk that misinformation and bias can be injected into LLMs via knowledge editing.
weaknesses: 1. I wonder whether finetuning and ICL can be classified into knowledge editing. Is there any formal and widely accepted definition of knowledge editing?

2. Can the proposed method be applied to close-source models?

3. For open-source models, there are many existing works which show LLMs can be manipulated to general harmful information, such as Jailbreaking and malicious finetuning. What are the uniqueness of knowledge editing in achieving this goal?

4. Malicious finetuning has already been reported, such as Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. Is it quite similar to the finding reported in this paper?

5. Can any kind of misinformation be formulated into structured format for ROME?

6. Is it better if more knowledge editing methods besides ROME are included in experiments?

7. The fact that ICL can mislead LLM to generate inappropriate information has been reported in some existing works such as "Many-shot jailbreaking". Is it similar with the ICL related finding in this paper?

Meta-review: This work proposes the editing attack, which reveals the risk of knowledge editing in injecting harmful information into LLMs and affecting their fairness and accuracy. The EDITATTACK dataset it constructs and the findings it provides are recognized by the reviewers. However, the reviewers suggest that this work lacks obvious technical contributions, covers a limited number of LLM types, and has flaws in writing, indicating that it has not reached an acceptable state.

Decision: Reject