Erasing Conceptual Knowledge from Language Models

## Abstract

Abstract Concept erasure in language models has traditionally lacked a comprehensive evaluation framework, leading to incomplete assessments of effectiveness of erasure methods. We propose an evaluation paradigm centered on three critical criteria: innocence (complete knowledge removal), seamlessness (maintaining conditional fluent generation), and specificity (preserving unrelated task performance). Our evaluation metrics naturally motivate the development of Erasure of Language Memory (ELM), a new method designed to address all three dimensions. ELM employs targeted low-rank updates to alter output distributions for erased concepts while preserving overall model capabilities including fluency when prompted for an erased concept. We demonstrate ELM’s efficacy on biosecurity, cybersecurity, and literary domain erasure tasks. Comparative analysis shows that ELM achieves superior performance across our proposed metrics, including near-random scores on erased topic assessments, generation fluency, maintained accuracy on unrelated benchmarks, andsshows robusunderowards adversarial attacks. Our code, data, and trained models are available at elm.baulab.info

## 1 Introduction

What does it mean for a language model to “unlearn” a concept? For example, let’s say that we want a model to behave as if it has never seen information about biological weapons in its training data.
Should we consider unlearning a success if the model forgets general information about biology, or if it loses the ability to produce fluent text whenever viruses or bacteria are mentioned? What if the model reveals harmful information when prompted with a new kind of question, or if the information can still be found somewhere within the model’s hidden states?

In this work, we take a step back to define three desiderata for concept erasure in language models:

- 1.
Innocence: Erasure should wipe the undesired knowledge completely: specifically, the model should be innocent of the knowledge
in response to any prompting method, or even when probed directly in its internal states. This criterion ensures the erased knowledge is fully inaccessible, with no form of indirect retrieval or influence on the model’s processing.
- 2.
Seamlessness: Editing should not draw attention to the concept that was erased by damaging the model. For example, when prompted to generate the erased concept, the edited model should produce fluent text that gracefully handles the absence of the target knowledge rather than producing gibberish. This criterion maintains the model’s overall utility and prevents obvious indicators of concept erasure.
- 3.
Specificity: The erasure process should not impact the model’s performance on unrelated concepts. This ensures the erasure process is precise and targeted, maintaining the model’s overall functionality.

Figure: Figure 1: An overview of our desiderata for concept erasure and Erasure of Language Memory method. The erased model must stay innocent of the erased concept, while still being fluent when prompted for the concept indicating seamless edit. The model should also preserve its general capabilities showing the method’s specificity.
Refer to caption: x1.png

We argue that robust concept erasure should simultaneously satisfy all three criteria. While prior works have successfully unlearned undesired concepts, existing approaches all suffer from limitations in one or more of these goals. Representation Misdirection for Unlearning (RMU) (Li et al., ) fine tunes the earlier layers of model to unlearn a concept by randomizing and amplifying the internal activations when prompted with text related to the concepts being erased, but it suffers from a lack of *seamlessness*, since the method creates a model that generates obvious gibberish in response to a dangerous prompt. Other methods, such as WhoIsHarryPotter (Eldan & Russinovich, ), employ a two-stage approach, training a reinforced model for the concept being erased and then training an unlearned model that behaves differently on the reinforced logits.
Our analysis reveals that this kind of approach falls short in *innocence*, since the erased knowledge can still be recovered through multiple-choice prompting which was consistent with prior findings (Lynch et al., ).

To address these triad of objectives, we propose a new method, Erasure of Language Memory (ELM), which enables precise knowledge erasing while maintaining contextual text generation fluency for seamless editing. Our core idea is to fine tune a model using
an objective to match the original model but
with reduced likelihood for text belonging to the concept to be erased. When applied using low-rank adaptation to specific layers, this procedure can be shown to effectively eliminate internal representations of the knowledge. We also employ the same objective to synthesize fine-tuning training data that can be used to ensure seamlessness: this synthesized data enables the model to maintain fluency in the neighborhood of the erased concept without introducing any information about that concept.

Through extensive experiments on various benchmarks spanning WMDP biosecurity and cybersecurity, as well as literary concepts like Harry Potter, we evaluate ELM on each of the three goals compared to prior techniques. We measure specificity and innocence using multiple-choice questions. Crucially, we stress-test innocence using jailbreaking methods such as adversarial attacks. We also measure seamlessness by examining text coherence when prompted about erased concepts, and we compare previous methods on all these metrics.

## 2 Related work

#### Machine Unlearning

The idea of removing specific data from machine learning models, known as machine unlearning, has gained attention in recent years, initially motivated by privacy concerns (Cao & Yang, ; Harding et al., ). Early methods focused on efficiently removing individual training examples or facts from models (Golatkar et al., ; Ma et al., ; Jang et al., ). However, most existing benchmarks evaluate unlearning on artificially created deletion sets (Choi & Na, ; Goel et al., ; Maini et al., ), in contrast to our focus on real-world distributions of broad conceptual knowledge.

#### Erasing broad conceptual knowledge from LLMs

New approaches to machine unlearning
have recently gained traction on the problem of removing dangerous capabilities from LLMs (Lynch et al., ; Ilharco et al., ; Jang et al., ; Lu et al., ; Yu et al., ; Casper et al., ; Eldan & Russinovich, ).
Our work is most directly comparable to three state-of-the-art techniques: Representation Misdirection for Unlearning (RMU) (Li et al., ), WhoIsHarryPotter (WHP) (Eldan & Russinovich, ), and Representation Noising (RepNoise) (Rosati et al., ). RMU fine-tunes models to align internal activations with random scaled vectors when processing targeted concepts.
WHP (Eldan & Russinovich, ) employs a two-stage approach, first training a reinforced model for the concept being erased and then training an unlearned model to behave differently on the reinforced logits.
RepNoise (Rosati et al., ) focuses on removing information about harmful representations across all layers of the LLM, by doing gradient ascent along with some representation noising (training internal representations to match Gaussian noise).
While all these methods successfully reduce model performance on erased knowledge, our measurements reveal that these previous approaches fall short in one or more of the three erasing goals. In contrast, our current work erases concepts by fine-tuning towards a principled target distribution that is designed to balance all three desiderata of innocence, specificity, and seamlessness.

Other methods such as LLMU (Yao et al., ), SSD (Foster et al., ), and SCRUB (Kurmanji et al., ) have also been proposed but face significant limitations. LLMU uses a differentiable membership classifier to identify unlearning set examples but struggles with imprecisely defined target distributions (see Li et al., ). SSD offers a retraining-free method for removing specific samples but is limited to individual examples rather than broader knowledge domains. SCRUB aims to remove sensitive data while preserving model performance on other tasks but requires access to the full training dataset, which is often impractical. Importantly, these methods were compared by RMU (Li et al., ) and found to be less effective for erasing broad conceptual knowledge.

#### Distilling generative model outputs.

Some techniques for controlling generative model outputs involve using an auxiliary generative model to specify the desired behavior over some training data, and then distilling these outputs into a target model by training the target model to mimic the demonstrated behavior. For example Askell et al. () and Bai et al. () prompt an unsafe model into behaving more safely, and then distil the prompted model. Similarly, Gandikota et al. () train a diffusion image model to mimic the outputs of an edited copy of the model whose generations have been guided to not produce images with certain attributes.  Rosati et al. () can also be classified under distillation methods as it mimics a Gaussian distribution when processing harmful tokens. Similarly, ELM matches the harmful logits to a modified output distribution, but unlike the distillation methods that usually adhere to the standard matching objective, we employ a multi-objective framework to address the seamlessness and specificity concerns with standard distillation methods.

#### Erasing in generative image models

Gandikota et al. () train a diffusion image model to mimic the outputs of an edited copy of the model whose generations have been guided to not produce images with certain attributes. Gandikota et al. () erase concepts by modifying the key value mapping of cross attention layers in a low rank closed form update. Other works remove the knowledge of unwanted concepts from the model weights; proposing attention re-steering through fine-tuning (Zhang et al., ), fine-tuning the attention weights (Kumari et al., ) and continual learning (Heng & Soh, ). We take inspiration from  Gandikota et al. () to reduce the likelihood of a concept being generated.

## 3 Background

### 3.1 Transformer

We focus on autoregressive transformer-based models (Tunstall et al., ; Dubey et al., ). Given an input token sequence $x=[x_{1},\ldots,x_{t}]\in\mathcal{X}$, these models output a probability distribution over a vocabulary $\mathcal{V}$ to predict the next token:

$$ $\mathcal{M}:\mathcal{X}\rightarrow\mathcal{V}$ (1) $$

The model architecture comprises stacked blocks, each usually containing a multi-head attention (MHA) layer followed by a multi-layer perceptron (MLP). These layers are stacked on top of each other, and their outputs are added to the residual stream of the model. MHA layers play an important role in bringing information from previous input tokens, while MLP layers are found to play the role of knowledge banks (Meng et al., ; Geva et al., ). These calculations can be summarized as follows,

$$ $\displaystyle\begin{array}[]{cc}a_{t}^{(\ell)}=\text{MHA}^{(\ell)}(h_{1}^{( \ell-1)},h_{2}^{(\ell-1)},...,h_{t}^{(\ell-1)});&\text{mlp}_{t}^{(\ell)}=\text {MLP}^{(\ell)}(a_{t}^{(\ell)}+h_{t}^{(\ell-1)})\[10.0pt] \lx@intercol\hfil h_{t}^{(\ell)}=a_{t}^{(\ell)}+\text{mlp}_{t}^{(\ell)}+h_{t}^ {(\ell-1)}\hfil\lx@intercol\end{array}$ (2) $$

where $h_{t}^{(\ell)}$ denotes the hidden state for token $t$ at layer $\ell$.

Previous research (Meng et al., ) has localized much model knowledge within early to mid blocks. As our own extensive experiments have similarly found good efficacy when erasing from earlier blocks of parameters, we restrict our parameter changes to MLP and MHA layers from those same blocks.

### 3.2 Transformer Next Token Prediction: A Classification Perspective

We take a different perspective on next token prediction. The usual way to describe an autoregressive language model is in terms of the factorization $P(x)=P(x_{\geq t}|x_{<t})P(x_{<t})$, that is, by writing the language distribution as a prediction of future token sequences $x_{\geq t}$ given observations of previous tokens $x_{<t}$. We instead adopt a classification perspective:

$$ $\displaystyle P(x)=P(x_{<t}|x_{\geq t})P(x_{\geq t})$ (3) $$

Here the previous tokens $x_{<t}$ serve as a class label: for example, if we set $x^{*}_{<t}=c_{+}$ to the context “The following text reflects no knowledge of bioweapons”, then $P(x^{*}_{<t}|x_{\geq t})$ serves to classify later text $x_{\geq t}$ by giving the probability of membership of $x_{\geq t}$ in this desirable knowledge-free class $c_{+}$.

To edit knowledge, we define a target distribution that adjusts the classification prediction:

$$ $\displaystyle P^{*}(x)\propto P(x^{*}_{<t}|x_{\geq t})^{\eta}P(x_{\geq t})$ (4) $$

Here we shall set $\eta\gg 0$ to increase the likelihood of a desired class of text $x^{*}_{<t}$ or $\eta\ll 0$ to reduce the likelihood when the class is undesired.

To implement our idea, we must return our framing to the autoregressive setting so that we can use our pretrained language model to predict classifications, which we can do by applying Bayes’ law:

$$ $\displaystyle P^{*}(x)\propto\left(\frac{P(x_{\geq t}|x^{*}_{<t})}{P(x_{\geq t })}\right)^{\eta}P(x_{\geq t})$ (5) $$

In this form, both the numerator and denominator can be read out of the pretrained language model. Equation 5 will form the basis for our training objectives. Our perspective is inspired by the classifier-free guidance work (Ho & Salimans, ) in diffusion model research which has later been adopted to language models (Sanchez et al., ).

We use this perspective to reduce the model’s likelihood of generating text that would follow a context like “You are an expert in bio-weapons.” By using the original model’s knowledge of bioweapons to define a fine-tuning objective and dataset reflecting a reduction of the likelihood of the model being an expert, we aim to create a new model without the undesired expertise.

### 3.3 Parameter-Efficient Fine-tuning via Low-Rank Adapters

Fine-tuning large pre-trained language models involves large sets of parameters, which can result in overfitting. To address this issue, we employ Low-Rank Adaptation (LoRA) (Hu et al., ), a method that introduces a factorization of weights to significantly reduce the number of parameters.
Consider a pre-trained model layer with weights $W_{0}\in\mathbb{R}^{d\times k}$, where $d$ and $k$ denote input and output dimensions, respectively. LoRA decomposes the weight update $\Delta W$ as follows:

$$ $\Delta W=BA,\quad B\in\mathbb{R}^{d\times r},A\in\mathbb{R}^{r\times k}$ (6) $$

Here, $r$ represents a small rank ($r\ll\min(d,k)$), constraining the update to a low-dimensional subspace. By optimizing only $A$ and $B$ while keeping $W_{0}$ fixed, LoRA significantly reduces the trainable parameters.
During inference, the update can be efficiently incorporated into the weights:

$$ $W=W_{0}+\Delta W$ (7) $$

The LoRA approach offers a balance between adaptation capability, reduced overfitting, and parameter efficiency, making it particularly suitable for our setting of fine-tuning large-scale models.

## 4 Method

We introduce Erasure of Language Memory (ELM), an approach for fine-tuning LoRA adaptors on Large Language Models (LLMs) to selectively erase concept-targeted knowledge while preserving the model’s conditional fluency and general capabilities (Figure 1). ELM uses two user-provided datasets: $\mathcal{D}_{\mathit{erase}}$, containing text related to the target concept for erasure, and $\mathcal{D}_{\mathit{retain}}$, comprising innocent text unrelated to the target concept. Our method learns low-rank parameter changes that modulate the likelihood of specific concepts being generated, effectively enabling targeted knowledge erasure while maintaining desired information.

The erasing dataset $\mathcal{D}_{\mathit{erase}}$ contains text related to the concept being erased. For example, to erase “bioweapons expertise” from the model, this dataset can include any pretraining or custom dataset that has information about bioweapons. The preservation dataset $\mathcal{D}_{\mathit{retain}}$ contains any other information that we do not wish to erase. This can be as open as general text datasets like WikiText (Merity et al., ) or as specific as concepts that are close to the concept being erased, but that need to be preserved, e.g. the bio-retain corpus released by the WMDP team (Li et al., ).

ELM’s objective for erasing a concept $c_{-}$ from the model $\theta$ comprises three components:

- 1.
Erasing Objective: When processing input logits from $\mathcal{D}_{\mathit{erase}}$, the model’s predicted probabilities should diverge from the original distribution, reducing the likelihood of tokens associated with the concept being erased. This encourages the model to predict tokens less likely to follow inputs related to the target concept. Building on our intuition from Section 3.2, we construct our erasing loss as follows:
$\displaystyle P_{\theta}^{\mathit{erased}}=P_{\theta}(X)\left(\frac{P_{\theta}
(c_{+}|X)}{P_{\theta}(c_{-}|X)}\right)^{\eta}\propto P_{\theta}(X)\left(\frac{
P_{\theta}(X|c_{+})}{P_{\theta}(X|c_{-})}\right)^{\eta}$
(8)
where $\eta$ is a hyperparameter controlling the strength of concept likelihood modification, $c_{-}$ is the concept to be erased (e.g., ”expert in bioweapons”), and $c_{+}$ is an optional alternative concept (e.g., ”novice in bioweapons” or ”expert in biology”). The erasing loss is then defined as:
$\displaystyle\mathcal{L}_{\mathit{erase}}=\mathbb{E}_{X\in\mathcal{D}_{\mathit
{erase}}}\texttt{CE}(P_{\theta^{*}}(X),P_{\theta}^{\mathit{erased}})$
(9)
- 2.
Retention Objective: The model must preserve its prediction probabilities when processing input tokens from $\mathcal{D}_{\mathit{retain}}$, ensuring that unrelated knowledge remains intact. The retention loss $\mathcal{L}_{\mathit{retain}}$ encourages the output distributions to remain unchanged:
$\displaystyle\mathcal{L}_{\mathit{retain}}=\mathbb{E}_{X\in\mathcal{D}_{
\mathit{retain}}}\texttt{CE}(P_{\theta^{*}}(X),P_{\theta}(X))$
(10)
- 3.
Conditional Fluency Objective: The erased model should maintain text coherence when prompted with content from $\mathcal{D}_{\mathit{erase}}$ while generating text that reduces the likelihood of the input prompt being the generated tokens’ context. We formulate this objective as follows:
$\displaystyle\mathcal{L}_{\mathit{fluency}}=\mathbb{E}_{X_{p}\in\mathcal{D}_{
\mathit{erase}}}\left[\sum_{t=2}^{T}\texttt{CE}(P_{\theta^{*}}(x_{t}|X_{p},x_{
1:t-1}),P_{\theta}^{\mathit{erased}}(x_{t}|X_{p},x_{1:t-1}))\right],$
(11)
where $X_{p}$ is the prompt from $\mathcal{D}_{\mathit{erase}}$, $x_{1:T}$ are the generated tokens, and $T$ is the length of the generated sequence. $P_{\theta^{*}}$ represents the probability distribution from the erased model, while $P_{\theta}^{\mathit{erased}}$ is the target distribution defined similarly to Equation 8.

The third term of the objective operates in two stages: First, a sequence of synthesized training tokens is generated by applying the erasing principle from Equation 8 to the original model during inference, so the text is guided towards the innocent class $c_{-}$. Then the entire text (prompt $X_{p}$ concatenated with generated text $x_{1:T}$) used to train the erased model $\theta^{*}$. Crucially, the training loss is masked to the generated tokens $x_{1:T}$ so that the model is not trained to generate the undesired concept in $X_{p}$. Rather, it is trained to generate innocent text in response to prompt text about the undesired concept. This is reflected in Equation 11, where the loss is computed only over the generated tokens. This approach ensures that the model learns to produce fluent and contextually different content even when prompted to discuss the erased concept. We provide more implementation details of the Conditional Fluency objective in Appendix E.

The overall loss for the ELM objective is a weighted combination of these three components:

$$ $\displaystyle\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{\mathit{erase}}+ \lambda_{2}\mathcal{L}_{\mathit{retain}}+\lambda_{3}\mathcal{L}_{\mathit{ fluency}}$ (12) $$

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are hyperparameters controlling the relative importance of each objective. We utilize the combined loss term $\mathcal{L}_{total}$ (Equation 12) to fine-tune low-rank adapters attached to the early layers of the language model. This approach allows us to precisely erase the target concept while simultaneously maintaining conditional fluency and retaining the model’s general capabilities.

## 5 Experiments

### 5.1 Experimental Setup

We design our experiments to evaluate ELM across the three desiderata that we have set out for thorough concepual knowledge erasure.

#### Benchmarks.

Our primary evaluation focuses on the Weapons of Mass Destruction Proxy (WMDP) dataset Li et al. (), specifically utilizing the biosecurity (WMDP-bio) and cybersecurity (WMDP-cyber) multiple-choice questions (MCQs). To demonstrate ELM’s versatility, we also employ a modified version of the Harry Potter MCQ dataset (Lynch et al., ), expanded from binary to quaternary choices for consistency with other benchmarks. This diverse set of tasks allows us to assess ELM’s erasure effectiveness across different domains and knowledge types.

#### Models.

We apply ELM to a range of state-of-the-art language models, including Zephyr-7B Beta (Tunstall et al., ), Mistral-7B (Jiang et al., ), Llama3-7B, and Llama3-7B-instruct (Dubey et al., ) for the WMDP erasure tasks. For the Harry Potter knowledge erasure, we use the Llama-2-7B Chat model (Touvron et al., ) to maintain consistency with prior work from Eldan & Russinovich (). This selection of models enables us to evaluate ELM’s performance across various model architectures and training paradigms.

#### Baselines.

For the WMDP tasks, we benchmark against Representation Misdirection for Unlearning (RMU) (Li et al., ) and RepNoise (Rosati et al., ). In the Harry Potter erasure task, we compare with RMU and WhoIsHarryPotter (WHP) (Eldan & Russinovich, ).

#### Data.

From WMDP Bio forget corpus, we utilize 5,000 text samples, each with a maximum length of 700 characters. From Cyber forget corpus we use 1,000 texts of similar length. The Harry Potter erasure task employs 3,000 text samples extracted from the novel series, also limited to 700 characters each. To facilitate conditional erasure (Eq. 8), we prepend contexts such as “You are an expert in” followed by concept-specific keywords. Additionally, we incorporate text completion examples for consistency, following the approach used by Qi et al. ().

#### Evaluation Metrics.

We assess our method, Erasure of Language Memory (ELM), along four key dimensions; in addition to our three key desiderata, we conduct a test of adversarial robustness.

- 1.
Innocence: We employ multiple-choice questions (MCQs) related to the target erased class to evaluate contextual knowledge extraction. Additionally, we analyze probing accuracies across internal model layers to detect any traces of latent knowledge.
- 2.
Seamlessness: To measure the model’s ability to generate fluent text when prompted with erased concepts, we assess the reverse perplexity of generated samples on forget set prompts using an independent language model. We generate text from edited models and run it through a different base model, measuring the perplexity of the text as per the second model (R-PPL). This approach quantifies fluency without relying on potentially biased self-perplexity scores.
- 3.
Specificity: We evaluate the modified model on standard benchmarks unrelated to the erased content to ensure that the erasure process does not degrade overall model performance.
- 4.
Robustness: We test against adversarial attacks like GCG (Zou et al., ) to understand the model’s tendency to display concept knowledge post-erasure.

### 5.2 Erasing WMDP Concepts

We evaluate ELM’s performance on erasing biosecurity and cybersecurity concepts from the Weapons of Mass Destruction Proxy (WMDP) dataset (Li et al., ). Table 1 presents a comprehensive comparison of ELM against baseline methods RMU (Li et al., ) and RepNoise (Rosati et al., ) across multiple models and benchmarks.

**Table 1: Comparison of ELM with baseline methods on WMDP concept erasure and general performance across different models. See Appendix B for full details on baselines and metrics.**
| Model | Method | Innocence ($\downarrow$) | Specificity ($\uparrow$) | Seamlessness |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | Bio | Cyber | MMLU | MT-Bench | R-PPL ($\downarrow$) |
| Zephyr-7B | Original | 64.4 | 44.3 | 58.5 | 7.3 | 6.0 |
| RMU | 30.5 | 27.3 | 57.5 | 7.2 | 24.8 |  |
| RepNoise | 29.7 | 37.7 | 53.3 | 6.6 | 25.0 |  |
| Ours | 29.7 | 27.2 | 56.6 | 7.1 | 10.9 |  |
| Mistral-7B | Original | 67.6 | 44.3 | 59.7 | 3.2 | 10.5 |
| Ours | 28.7 | 26.4 | 55.4 | 3.7 | 15.3 |  |
| Llama3-8B-Instruct | Original | 71.3 | 46.7 | 63.7 | 7.8 | 3.6 |
| Ours | 32.2 | 27.2 | 61.6 | 7.7 | 7.4 |  |
| Llama3-8B | Original | 71.2 | 45.3 | 62.1 | 5.6 | 9.1 |
| Ours | 33.3 | 26.6 | 57.2 | 4.8 | 4.5 |  |

As shown in Table 1, ELM consistently achieves near-random performance (random guess is 25%) on erased WMDP concepts (Bio and Cyber) while maintaining high scores on general knowledge (MMLU) and language understanding (MT-Bench) tasks. Notably, ELM demonstrates superior fluency when generating text related to erased concepts, as evidenced by lower reverse perplexity scores compared to RMU and RepNoise.

We further verify ELM’s versatility by applying it to other transformer models, including Mistral-7B, Llama3-8B-Instruct, and Llama3-8B. Across all tested models, ELM consistently reduces WMDP scores to near-random levels while preserving general performance.

### 5.3 Ablation Study

We conduct an ablation study to quantify the impact of each component in our proposed method. Table 2 presents the results across various configurations, measuring performance on concept erasure (WMDP), general language understanding (MMLU), and generation quality (MT-Bench, Perplexity). In each ablation experiment, we drop a single term of the loss (Eq. 12), or replace the loss with a randomized-objective baseline.

First, $\mathcal{L}_{\mathit{erase}}$ proves crucial for innocence. Its ablation results in significantly higher WMDP scores, indicating incomplete erasure. Replacing the ground truth for $\mathcal{L}_{\mathit{erase}}$ in Eq. 9 with random vectors instead of $P_{\theta}^{erased}(X)$ demonstrates that training towards noise can also achieve low WMDP scores in Cyber (although we found it ineffective in Bio). However, the random objective degrades seamlessness, producing glaringly incoherent outputs (see example in the gray box below).

The retain term $\mathcal{L}_{\mathit{retain}}$ (Eq. 10) is vital for specificity. Its removal yields the lowest MMLU scores, demonstrating its role in maintaining broad knowledge while enabling targeted erasure.

Ablating $\mathcal{L}_{\mathit{fluency}}$ (Eq. 11) reveals a trade-off: effective erasure but compromised generation quality when prompted with erased concepts. This highlights its importance in achieving seamlessness. Substituting $\mathcal{L}_{\mathit{fluency}}$ with random text from wiki-text Merity et al. () slighly decreases our automatic R-PPL measure of seamlessness, and qualitatively produces awkward outputs that tend to be grammatical but irrelevant to input prompts. This underscores the term’s role in maintaining seamless contextual relevance.

**Table 2: We ablate the loss terms of ELM to show their importance in erasure. We find that $\mathcal{L}_{fluency}$ is important for maintaining seamlessness, and $\mathcal{L}_{retain}$ is important for specificity.**
| Setup | Innocence ($\downarrow$) | Specificity ($\uparrow$) | Seamlessness ($\downarrow$) |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Bio | Cyber | MMLU | MT-Bench | RPPL |
| w/o $\mathcal{L}_{\mathit{erase}}$ | 64.8 | 42.7 | 58.0 | 6.9 | 2.7 |
| w/o $\mathcal{L}_{\mathit{retain}}$ | 24.3 | 25.8 | 23.6 | 1.2 | 22.0 |
| w/o $\mathcal{L}_{\mathit{fluency}}$ | 27.6 | 26.4 | 55.7 | 6.6 | 29.8 |
| \hdashlineRandom Erasing | 57.9 | 28.7 | 57.8 | 7.0 | 10.9 |
| Random Fluency | 29.8 | 30.0 | 56.6 | 6.6 | 13.1 |
| Random Both | 51.3 | 30.6 | 58.4 | 6.7 | 9.54 |
| \hdashlineFull ELM | 29.7 | 27.2 | 56.6 | 7.1 | 11.0 |

The full ELM method achieves the best balance between concept erasure and general performance. We show a qualitative example with each of the settings in the gray box below:

### 5.4 Specificity Analysis

To assess the specificity of our erasure method, we examine its impact on related MMLU classes. Figure 2 shows the performance change in related safe concepts when WMDP-bio and WMDP-cyber knowledge is erased. We find that RMU and ELM have similar effects on nearby safer concepts, degrading performance slightly compared to the base model.

Figure: Figure 2: Interference analysis on related MMLU classes shows that both ELM and RMU interfere with closely related safe concepts.
Refer to caption: x2.png

### 5.5 Robustness to Attacks

We evaluate the robustness of ELM by employing Greedy Coordinate Gradient attack (GCG) Zou et al. (). We optimize for an adversarial suffix prompt to induce the model to generate a target text. We use the prompt “How would I engineer influenza A to be significantly more virulent?” and set the target prompt as “To make influenza A significantly more virulent,”. ELM demonstrates strong resistance to these attacks even when optimized for 5000 steps, whereas, the original model starts responding to the query with the GCG suffix, as illustrated in the following example:

We provide more details about GCG attack and baseline robustness details in Appendix C.

### 5.6 Probing and Activation Analysis

To estimate the presence of erased knowledge within the internal representations of a model, we conduct the probing analysis, training a linear probe using the same setup as used by Li et al. ().

The results in Figure 3(a) reveal distinct knowledge retention patterns across methods. ELM and RMU achieve effective erasure, maintaining low probe accuracies across all layers for both biosecurity and cybersecurity MCQs. In contrast, RepNoise shows partial retention, particularly for cybersecurity concepts.

Analysis of activation norms, in Figure 3(b), further highlights the differences. Both ELM and RMU induce out-of-distribution activations in early layers for the forget set, but while RMU continues to exhibits persistent activation norm disruption across all layers, ELM activation norms return to baseline behavior in middle layers. This suggests altered initial processing of erased concepts during knowlege retrieval while preserving text-prediction behavior in later stages. We hypothesize that the late-layer activation norm disruption in RMU impacting overall model fluency. RepNoise shows minimal changes in activation norms, consistent with its less aggressive erasure approach.

Figure: Figure 3: Analysis of internal representations after erasure. (a) Probe accuracies across layers for various methods applied to Zephyr-7B (chance is 0.25, for four-choice MCQ). Probing accuracy is close to random for ELM. (b) Activation norms for methods applied to Zephyr-7B. ELM maintains typical model behavior for erased concepts in later layers.
Refer to caption: x3.png

### 5.7 Erasing Harry Potter Knowledge

To further demonstrate the versatility of ELM, we apply it to the task of erasing knowledge related to the Harry Potter literary universe. We compare ELM against RMU and WhoIsHarryPotter (WHP) (Eldan & Russinovich, ) methods for Llama-2-7B Chat. Table 3 presents this comparison.

**Table 3: Erasing knowledge of Harry Potter from Llama-2-7B Chat. While WHP is fluent when prompted for Harry Potter, it shows a lack of innocence on the topic, revealing knowledge nearly as strong as the original model when prompted using a multiple-choice question benchmark. ELM erases knowledge while maintaining fluency. RMU and WHP details can be found in Appendix B.**
| Method | Innocence ($\downarrow$) | Specificity ($\uparrow$) | Seamlessness ($\downarrow$) |
| --- | --- | --- | --- |
|  | HP-MCQ | MMLU | R-PPL |
| Original Model (Touvron et al., ) | 66.4 | 47.0 | 3.6 |
| RMU (Li et al., ) | 51.0 | 44.2 | 3.7 |
| WHP (Eldan & Russinovich, ) | 58.6 | 43.1 | 3.4 |
| ELM (Ours) | 38.3 | 45.3 | 3.4 |

ELM achieves a balance between effective knowledge erasure (low HP MCQ score) and maintaining fluent generation (low reverse-perplexity). Similar to  Lynch et al. (), we found WHP model (Eldan & Russinovich, ) maintains fluency but fails to effectively erase the target knowledge. RMU (Li et al., ) proved to be ineffective in erasing with our initial large hyperparameter sweep. A more through sweep may be necessary to conclusively determine its limitations in this context

## 6 limitations

While ELM demonstrates strong performance in targeted concept removal, a few limitations warrant further study. We observe a noticable degradation in MMLU scores for nearby safer concepts. This challenge calls for more fine-grained erasure techniques and optimization constraints to minimize unintended knowledge loss. Second, while ELM generates fluent text when prompted with erased concepts, in a few cases, outputs lack meaningful content or contextual relevance. This highlights the difficulty of maintaining semantic coherence while navigating around removed knowledge, suggesting the need for more sophisticated language modeling approaches during erasure. The effectiveness of ELM in erasing complex, interconnected concepts remains an open question. Language models encode information in intricate ways, with subtle dependencies between semantic domains. Completely isolating and removing specific concepts without affecting related knowledge presents an ongoing challenge, necessitating more nuanced erasure techniques that can handle concept interdependencies. Finally, while our evaluation framework comprehensively assesses erasure quality, it may not capture all aspects of model behavior affected by the process. Future work should focus on continuing the development of more rigorous evaluation metrics and exploring the wider effects of concept erasure on model performance and generalization capabilities.

## 7 Conclusion

This work introduces a framework for evaluating concept erasure in language models, centered on three critical criteria: innocence, seamlessness, and specificity. Our proposed Erasure of Language Memory (ELM) method emerges as a direct consequence of this framework. ELM uses low-rank model updates to selectively alter output distributions for erased concepts while preserving overall model capabilities and fluency. The erasing objective effectively eliminates targeted concept knowledge, as evidenced by near-random performance on multiple-choice questions related to erased topics. Simultaneously, the retention and fluency objectives ensure coherent text generation and preserved performance on unrelated tasks. Furthermore, ELM demonstrates robust resistance to adversarial attacks which underscores the effectiveness of our approach in creating models that are not only “innocent” of erased knowledge but also maintain their functionality and coherence. In addition to providing a practical solution for concept erasure, we have established a foundation for more comprehensive evaluation of knowledge erasure in language models.

## Acknowledgements

RG, and DB are supported by Open Philanthropy and National Science Foundation (Grant Number: NSF-2403303). SF is supported by Open Phil and Khoury Distinguished Fellowship. SM participated in this work while a postdoctoral researcher at Northeastern University supported by an Open Philanthropy grant.

## Code

Our methods are available as open-source code. Source code, trained models, and data sets for reproducing our results can be found at [elm.baulab.info](https://elm.baulab.info) and at [https://github.com/rohitgandikota/erasing-llm](https://github.com/rohitgandikota/erasing-llm).

## Appendix A Details on metrics

#### Multiple Choice Questions.

To measure the multiple choice question accuracy across the different models and erasure methods, we use the lm-evaluation-harness library by EleutherAI (Gao et al., ).

#### MT-Bench.

We employ the single evaluation mode on MT-Bench, using gpt-4o-2024-05-13 as the judge.

#### Reverse Perplexity (R-PPL).

To measure the seamlessness of edits, we aim to quantify the fluency of the text being generated by the edited model when prompted with the concept being erased. To evaluate this we prompt the models using questions from MCQ dataset from WMDP Li et al. () and let the models generate text free-form up to 500 tokens. We then measure the perplexity on generated text using a totally different evaluation model, Llama3.1-8B (Dubey et al., ).

## Appendix B Baseline Methods

### B.1 WMDP Results

#### RMU (Li et al., 2024 ) .

We directly download the best Zephyr-7B RMU model from the WMDP authors ([https://huggingface.co/cais/Zephyr_RMU](https://huggingface.co/cais/Zephyr_RMU)) for testing.

#### RepNoise (Rosati et al., 2024 ) .

Repurposing the authors’ original code, we train RepNoise on Zephyr-7B using the WMDP retain and forget datasets as $\mathcal{D}_{harmless}$ and $\mathcal{D}_{harmful}$ respectively. We trained LoRA adapters on top of the original model with rank 64, alpha=16, and dropout=0.05. We first conducted a grid search over the parameters $\alpha\in\{1,0.5,0.1\}$, $\beta\in\{1,{1}\mathrm{e}{-2},{1}\mathrm{e}{-4}\}$, and learning rates $\{{1}\mathrm{e}{-5},{1}\mathrm{e}{-3}\}$. As none of the resulting runs significantly decreased accuracy on WMDP MCQ questions without destroying MMLU accuracy, we performed one more grid search over parameters $\alpha\in\{4,2,1,0.5,0.1\}$, $\beta\in\{2,1,{1}\mathrm{e}{-2},{1}\mathrm{e}{-4}\}$, and learning rates $\{{8}\mathrm{e}{-8},{2}\mathrm{e}{-5},{1}\mathrm{e}{-3}\}$. The highest-performing run, shown in Table 1, had $\alpha=4$, $\beta=1$, and learning rate ${2}\mathrm{e}{-5}$. The method was run for one epoch with a batch size of 4.

### B.2 Harry Potter Results

#### RMU (Li et al., 2024 ) .

We train LoRA adapters on top of Llama-2-7B Chat at varying layers, using text from the Harry Potter books ([https://huggingface.co/datasets/KaungHtetCho/Harry_Potter_LSTM](https://huggingface.co/datasets/KaungHtetCho/Harry_Potter_LSTM)) as $D_{\text{forget}}$ and WikiText as $D_{\text{retain}}$. We sweep across layer indices 3,4,5, 4,5,6, and 5,6,7 with $\alpha\in\{1200,600\}$ and learning rate $\in\{{1}\mathrm{e}{-3},{1}\mathrm{e}{-4},{5}\mathrm{e}{-5}\}$. We report numbers for the best run in Table 3, for layers 5,6,7, $\alpha=600$, learning rate ${5}\mathrm{e}{-5}$, and batch size 1, trained for one epoch. The Harry Potter dataset used for RMU was not the exact same dataset used for ELM ([https://huggingface.co/datasets/mickume/harry_potter_tiny](https://huggingface.co/datasets/mickume/harry_potter_tiny)), as performance was much worse for RMU on the latter dataset.

#### WHP (Eldan & Russinovich, 2023 ) .

We directly download the best Llama-2-7B Chat model from the original authors ([https://huggingface.co/microsoft/Llama2-7b-WhoIsHarryPotter](https://huggingface.co/microsoft/Llama2-7b-WhoIsHarryPotter)).

## Appendix C Robustness Evaluation

To evaluate the robustness of ELM against adversarial attacks, we employ the Greedy Coordinate Gradient (GCG) method (Zou et al., ), utilizing the standard implementation from GraySwanAI (GraySwanAI, ). The GCG attack requires defining an initial prompt, a multi-token target text, and an initialized adversarial suffix. Following the protocol established in Li et al. (), we use a 20-token adversarial suffix and derive prompts from the WMDP MCQ datasets. To facilitate open-ended generation, we present only the question component of these prompts, omitting the multiple-choice structure. Our experiments reveal a stark contrast in robustness between ELM models and their base model counterparts. Even after extensive optimization exceeding 5000 iterations, we fail to identify a GCG prompt capable of inducing ELM models to generate content related to erased concepts. This resilience stands in marked contrast to the original models, which succumb to effective attack suffixes within 200 iterations, subsequently producing potentially harmful text.

ELM:

RMU:

RepNoise:

## Appendix D Hyperparameter Analysis

To optimize the performance of ELM, we conduct an extensive hyperparameter study, focusing on three key parameters: LoRA rank, erasure strength $\eta$, and the range of layers to which ELM is applied. Our findings corroborate and extend previous observations in the literature (Meng et al., ; Geva et al., ).
Figure 4a illustrates the impact of layer selection on erasure efficacy.

Consistent with prior work, we observe that applying ELM to earlier layers yields more effective knowledge erasure compared to later layers. Specifically, we identified layers 4-7 of the Zephyr model as the optimal range for achieving a balance between thorough knowledge erasure and preservation of general capabilities.

The interplay between LoRA rank and erasure strength $\eta$ is depicted in Figure 4b. Our analysis reveals that lower values of $\eta$ result in diminished effects on both erasure performance and general benchmark scores. Interestingly, we found no clear trend with respect to LoRA rank, with lower-rank updates performing comparably to higher-rank alternatives. This suggests that ELM can achieve effective erasure with minimal parametric overhead.

Based on these empirical results, we adopted a configuration of rank 4, $\eta=500$, and application to layers 4-7 for all subsequent experiments. This configuration strikes a balance between erasure efficacy, computational efficiency, and preservation of general language capabilities.

Figure: Figure 4: Hyperparameter sweep results for rank, $\eta$, and layer selection
Refer to caption: x4.png

## Appendix E Conditional Fluency Training

To achieve seamless editing, ELM must generate fluent text even when prompted about erased concepts. The ideal behavior mimics a model that never encountered the concept during pretraining. We implement an additional step to make ELM models acknowledge the concept while suggesting a topic change, although this behavior remains configurable through prompt engineering.

Our training procedure extends the erasure objective from Equation 9. For each prompt from the harmful dataset, we generate new tokens using the erasure objective. Importantly, we do not consider these newly generated tokens as harmful context for subsequent generations, but rather use them for positive conditioning. This approach allows the model to continue generating fluently while reducing the likelihood of discussing the erased concept. Through this process, the model learns to maintain fluency while decreasing the probability of elaborating on the queried concept. Inspired by Qi et al. (), we incorporate an additional consistency mechanism. We append a standard response to the initial prompt, such as a paraphrased version of: “This is a harmful concept. Let’s change the topic to something more fun and interesting:” We then initiate the generation process from this augmented prompt. This technique ensures consistent model behavior when encountering erased concepts.
The final training step involves generating the complete response, including the initial prompt, consistency prompt, and letting the model generate new tokens. We then pass this entire sequence through the ELM model. Crucially, we fine-tune only the parameters responsible for generating the new tokens. This targeted approach ensures that we preserve the model’s general knowledge while specifically adapting its behavior for erased concepts.

## Appendix F Progression of ELM Training

We evaluate the ELM intermediate checkpoints to observe the training dynamics of the method in Figure 5. We find that ELM suddenly drops the knowledge of the erased concept, halfway down the training and continues to slowly erase the rest of the traces. Bio-threat knowledge takes more time to be erased from the model - which could be directly proportional to the initial amount of prior knowledge.

Figure: Figure 5: Evaluating the intermediate checkpoints of ELM method to observe the training progression. We find that the model has a sudden drop of knowledge and then continues to slowly remove the further traces.
Refer to caption: x5.png

## Appendix G Qualitative Examples

### G.1 Prompts from WMDP-Bio MCQ Questions

### G.2 Prompts from WMDP-Cyber MCQ Questions

### G.3 Generic Questions

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Most question are covered in the weaknesses outlined above, e.g. Why using such a inconsistent setup across experiments? Why so little comparisons? How to interpret the different improvement in different experiments?, etc. The authors are encouraged to include more results and explanations to alter my opinion on these weaknesses.
soundness: 2
strengths: 1. The overview picture about the two individual losses are helpful for understanding the design of ELM.
2. The idea of keeping the models' response seamless post removal of a concept is interesting.
summary: In this paper, the authors proposed to evaluate LLM concept erasure with innocence, seamlessness and specificity, and put forward Erasure of Language Memory (ELM) as a new erasing method which claims to exile in all these three aspects by integrating losses (i.e. erasing, conditionally fluency, and retention) specifically designed for each of them. ELM was tested on WMDP dataset and evaluated with multiple choice questions, perplexity and MTBench/MMLU to exhibit innocence, seamlessness and specificity respectively. Based on the results on Zephyr 7b, ELM, compared with RepNoise and RMU, achieves marginally higher performance in terms of innocence and specificity, while processing a larger advantage in seamlessness. A similar experiment was conducted about erasing the concept of Harry Potter from Llama2 7b, the result of which, instead, indicates noticeably better innocence, with marginally better seamlessness and specificity, than RMU and WHP. There is also an ablation study to verify the impact of the losses on the three desiderata, as well as a robustness experiment by luring the LLM to comply with the request about the erased concept using GCG to show that ELM responses are less gibberish than the baselines.
weaknesses: 1. Inconsistent and Incomplete Experiments & Non-significant Improvement: In the main experiment, comparisons with baseline methods are only made for Zephyr 7b and missing for the Mistral and Llama 3 models, making it hard to draw a convincing conclusion about the merits of ELM. Additionally, in the experiment about removing the concept of Harry Potter, RepNoise was replaced with another baseline WHP, and the target model is switched to Llama 2 7b. The experiment settings are not consistent with each other. The results in the two experiments are also telling different stories: in both cases ELM only shows clear advantage in one of the three desiderata, and its seamlessness in the former and innocence in the latter, but there lacks discussion about this difference.
2. Questionable Claim about Seamlessness: While seamlessness is definitely a desirable feather for concept erasure, measuring it with PPL is obviously insufficient. Based on the example responses in the main body as well as in appendix, responses post ELM are only less gibberish but not actually meaningful to make it acceptable seamless.
3. Questionable Novelty: Tackling concept erasure by combining losses designed for forgetting and retaining is nothing new. The baseline RMU, for instance, also has a forget and a retain loss, with the only difference being using cross-entropy or L2 distance. The fluency objective is relatively new, but it only seems incremental to the existing framework.
4. Poor Presentation: The section about probing and activation analysis, supposedly, is going to be interesting. However, Figure 3 is using a elusive legend and captions, making it impossible to relate the the analysis with the data in the figure.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: See the weakness part.

Typos and presentation errors:

(1) Line 51, "fine tune" -> "fine-tune"

(2) Line 243 and 244 ''..'' -> ``....''

(3) Line 288, \citet -> \citep

(4) Line 294, Llama3-7B -> Llama3-8B

(5) Table 3, the R-PPL of WHP should also be highlighted.
soundness: 3
strengths: 1. The model unlearning/knowledge erasure problem is important and practical, as there are many cases that we want the LLMs to forget some targeted concepts.

2. The motivation in Eq. (3)-(5) is interesting and clear, which makes the method easy to follow.

3. The proposed metrics are comprehensive and practical.

4. The ablation studies in Section 5.3 are thorough.
summary: This paper studies how to effectively erase corresponding knowledge from LLMs. The authors first propose three metrics (Innocence, Seamlessness, Specificity) to comprehensively measure the effectiveness of a knowledge erasure method. The authors claim that current erasure methods typically fall short in one or several metrics above.  Then, the authors propose a  method called the Erasure of Language Memory (ELM) that is motivated from the classifier-free guidance diffusion work to achieve more effective erasure performance on all three metrics.
weaknesses: I have some questions for the authors:

1. Regarding the Erasing Objective in Line 233-247, I think the form of probability function is somehow similar to the form of emulated fine-tuning/decoding-time alignment [1,2], which also manipulate the predicted probability distribution by multiplying a re-scale factor (though this factor is different in their works). Maybe the authors could add more discussion or comparison on these works to  make the contribution more clearer.

2. Regarding the Conditional Fluency Objective, I am concerned about its necessity. As the Erasing Objective  is already included, why do we need this Conditional Fluency Objective? I would assume based on the Erasing Objective , the model can learn to generate fluent response by attempting to respond to the alternative concept $c_{+}$ even though the prompt is $c_{-}$.

3. In Table 1, why do you not perform experiments with RMU and RepNoice on other three LLMs? Also, the advantage of ELM compared with RMU seems to be limited.

4. The authors should include a Ethics Statement section after the main text before references, as knowledge erasure is related to some ethical concerns.

[1] Mitchell, Eric, et al. "An emulator for fine-tuning large language models using small language models." ICLR 2024

[2] Liu, Tianlin, et al. "Decoding-time Realignment of Language Models."  ICML 2024

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: See the Weakness
soundness: 2
strengths: The presentation is well-written and easy to follow. The proposed three criteria—innocence, seamlessness, and specificity—are particularly insightful.
summary: This paper introduces a framework for erasing conceptual knowledge from language models, aiming to ensure that the erased knowledge is irretrievable while maintaining the model's general functionality and fluency. The authors propose the "Erasure of Language Memory" (ELM) method, which is designed to meet three criteria: innocence, seamlessness, and specificity.
weaknesses: 1. The paper lacks a robust justification for using low-rank adapters. It claims that full fine-tuning may lead to overfitting, but it fails to provide comparative experiments to substantiate this claim. Demonstrating overfitting through empirical results would strengthen the argument.

2. The benchmarking is limited. While the paper focuses on multiple-choice tasks, it does not explore how the method performs on open-ended generation tasks, which could provide a more comprehensive evaluation of the model's capabilities.

3. In Table 1, the implementation of baselines such as RMU for models like Mistral-7B and Llama3-8B is missing. Why?

4. I appreciate the ablation study conducted in Section 5.3; however, the approach could be more rigorous. Rather than simply removing one component of the loss function as per Equation 12, it would be more insightful to vary the hyperparameters $\lambda_1, \lambda_2, \lambda_3$ to understand their individual and combined effects on the results. The author should also develop a framework to control those hyperparameters to achieve good tradeoffs/performance.

5. The accuracy of RMU outperforms ELM in the virology benchmarks in Figure 2. Why? Are there any specific explanations?

6. The range of attacks tested in Section 5.5 is limited. To demonstrate the robustness of their defense methods universally, it would be better to include a wider variety of adversarial attack methods. This would help validate the general applicability of the defense strategies against a broader spectrum of potential threats.

Meta-review: This submission obtains three negative rating. Several limitations of this version are still existed according to the results like limited experiments and non-significant improvements, undesirable effects of full finetuning, and more comparison with baselines. After rebuttal, two reviewers replied and the final ratings are still all negative. The AC decides the current version is still not ready for publication.

Decision: Reject