Erasing Conceptual Knowledge from Language Models

Authors: Rohit Gandikota, Sheridan Feucht, Samuel Marks, David Bau
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Safety, Knowledge, Concept Erasing, Model Editing, Safety, LLM

Abstract:
We propose a comprehensive evaluation framework for concept erasure in language models, addressing the need for a holistic assessment of effective unlearning. Our framework centers on three critical criteria: innocence (complete knowledge removal), seamlessness (maintaining conditional fluent generation), and specificity (preserving unrelated task performance). These evaluation metrics naturally motivate the development of Erasure of Language Memory (ELM), a new method designed to address all three dimensions. ELM employs targeted low-rank updates to alter output distributions for erased concepts while preserving overall model capabilities including fluency when prompted for an erased concept. We demonstrate ELM's efficacy on biosecurity, cybersecurity, and literary domain erasure tasks. Comparative analysis shows that ELM achieves superior performance across our proposed metrics, including near-random scores on erased topic assessments, generation fluency, maintained accuracy on unrelated benchmarks, and robustness under adversarial attacks.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1729698904133

thread_type:
official_review

<thread_object>
creator: reviewer_qg5t

<content>
summary:
This paper introduces a framework for erasing conceptual knowledge from language models, aiming to ensure that the erased knowledge is irretrievable while maintaining the model's general functionality and fluency. The authors propose the "Erasure of Language Memory" (ELM) method, which is designed to meet three criteria: innocence, seamlessness, and specificity.

strengths:
The presentation is well-written and easy to follow. The proposed three criteria—innocence, seamlessness, and specificity—are particularly insightful.

weaknesses:
1. The paper lacks a robust justification for using low-rank adapters. It claims that full fine-tuning may lead to overfitting, but it fails to provide comparative experiments to substantiate this claim. Demonstrating overfitting through empirical results would strengthen the argument.

2. The benchmarking is limited. While the paper focuses on multiple-choice tasks, it does not explore how the method performs on open-ended generation tasks, which could provide a more comprehensive evaluation of the model's capabilities.

3. In Table 1, the implementation of baselines such as RMU for models like Mistral-7B and Llama3-8B is missing. Why?

4. I appreciate the ablation study conducted in Section 5.3; however, the approach could be more rigorous. Rather than simply removing one component of the loss function as per Equation 12, it would be more insightful to vary the hyperparameters $\lambda_1, \lambda_2, \lambda_3$ to understand their individual and combined effects on the results. The author should also develop a framework to control those hyperparameters to achieve good tradeoffs/performance.

5. The accuracy of RMU outperforms ELM in the virology benchmarks in Figure 2. Why? Are there any specific explanations?

6. The range of attacks tested in Section 5.5 is limited. To demonstrate the robustness of their defense methods universally, it would be better to include a wider variety of adversarial attack methods. This would help validate the general applicability of the defense strategies against a broader spectrum of potential threats.

questions:
See the Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730042333413

thread_type:
official_review

<thread_object>
creator: reviewer_ru9c

<content>
summary:
This paper studies how to effectively erase corresponding knowledge from LLMs. The authors first propose three metrics (Innocence, Seamlessness, Specificity) to comprehensively measure the effectiveness of a knowledge erasure method. The authors claim that current erasure methods typically fall short in one or several metrics above.  Then, the authors propose a  method called the Erasure of Language Memory (ELM) that is motivated from the classifier-free guidance diffusion work to achieve more effective erasure performance on all three metrics.

strengths:
1. The model unlearning/knowledge erasure problem is important and practical, as there are many cases that we want the LLMs to forget some targeted concepts.

2. The motivation in Eq. (3)-(5) is interesting and clear, which makes the method easy to follow.

3. The proposed metrics are comprehensive and practical.

4. The ablation studies in Section 5.3 are thorough.

weaknesses:
I have some questions for the authors:

1. Regarding the Erasing Objective in Line 233-247, I think the form of probability function is somehow similar to the form of emulated fine-tuning/decoding-time alignment [1,2], which also manipulate the predicted probability distribution by multiplying a re-scale factor (though this factor is different in their works). Maybe the authors could add more discussion or comparison on these works to  make the contribution more clearer.

2. Regarding the Conditional Fluency Objective, I am concerned about its necessity. As the Erasing Objective  is already included, why do we need this Conditional Fluency Objective? I would assume based on the Erasing Objective , the model can learn to generate fluent response by attempting to respond to the alternative concept $c_{+}$ even though the prompt is $c_{-}$.

3. In Table 1, why do you not perform experiments with RMU and RepNoice on other three LLMs? Also, the advantage of ELM compared with RMU seems to be limited.

4. The authors should include a Ethics Statement section after the main text before references, as knowledge erasure is related to some ethical concerns.

[1] Mitchell, Eric, et al. "An emulator for fine-tuning large language models using small language models." ICLR 2024

[2] Liu, Tianlin, et al. "Decoding-time Realignment of Language Models."  ICML 2024

questions:
See the weakness part.

Typos and presentation errors:

(1) Line 51, "fine tune" -> "fine-tune"

(2) Line 243 and 244 ''..'' -> ``....''

(3) Line 288, \citet -> \citep

(4) Line 294, Llama3-7B -> Llama3-8B

(5) Table 3, the R-PPL of WHP should also be highlighted.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676486460

thread_type:
official_review

<thread_object>
creator: reviewer_r7fb

<content>
summary:
In this paper, the authors proposed to evaluate LLM concept erasure with innocence, seamlessness and specificity, and put forward Erasure of Language Memory (ELM) as a new erasing method which claims to exile in all these three aspects by integrating losses (i.e. erasing, conditionally fluency, and retention) specifically designed for each of them. ELM was tested on WMDP dataset and evaluated with multiple choice questions, perplexity and MTBench/MMLU to exhibit innocence, seamlessness and specificity respectively. Based on the results on Zephyr 7b, ELM, compared with RepNoise and RMU, achieves marginally higher performance in terms of innocence and specificity, while processing a larger advantage in seamlessness. A similar experiment was conducted about erasing the concept of Harry Potter from Llama2 7b, the result of which, instead, indicates noticeably better innocence, with marginally better seamlessness and specificity, than RMU and WHP. There is also an ablation study to verify the impact of the losses on the three desiderata, as well as a robustness experiment by luring the LLM to comply with the request about the erased concept using GCG to show that ELM responses are less gibberish than the baselines.

strengths:
1. The overview picture about the two individual losses are helpful for understanding the design of ELM.
2. The idea of keeping the models' response seamless post removal of a concept is interesting.

weaknesses:
1. Inconsistent and Incomplete Experiments & Non-significant Improvement: In the main experiment, comparisons with baseline methods are only made for Zephyr 7b and missing for the Mistral and Llama 3 models, making it hard to draw a convincing conclusion about the merits of ELM. Additionally, in the experiment about removing the concept of Harry Potter, RepNoise was replaced with another baseline WHP, and the target model is switched to Llama 2 7b. The experiment settings are not consistent with each other. The results in the two experiments are also telling different stories: in both cases ELM only shows clear advantage in one of the three desiderata, and its seamlessness in the former and innocence in the latter, but there lacks discussion about this difference.
2. Questionable Claim about Seamlessness: While seamlessness is definitely a desirable feather for concept erasure, measuring it with PPL is obviously insufficient. Based on the example responses in the main body as well as in appendix, responses post ELM are only less gibberish but not actually meaningful to make it acceptable seamless.
3. Questionable Novelty: Tackling concept erasure by combining losses designed for forgetting and retaining is nothing new. The baseline RMU, for instance, also has a forget and a retain loss, with the only difference being using cross-entropy or L2 distance. The fluency objective is relatively new, but it only seems incremental to the existing framework.
4. Poor Presentation: The section about probing and activation analysis, supposedly, is going to be interesting. However, Figure 3 is using a elusive legend and captions, making it impossible to relate the the analysis with the data in the figure.

questions:
Most question are covered in the weaknesses outlined above, e.g. Why using such a inconsistent setup across experiments? Why so little comparisons? How to interpret the different improvement in different experiments?, etc. The authors are encouraged to include more results and explanations to alter my opinion on these weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732584118263

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Collective Response - Extended Experiments and Clarification

comment:
We sincerely thank all reviewers for their thorough feedback.

When comparing our method to previous baselines, we would like to emphasize the conceptual contribution of our comparisons. Previous approaches to unlearning have framed the task in terms of adding scaled noise (RMU) or reversing gradients (RepNoise) or finetuning modified text (WHP). But our measurements show that a systematic erasing objective based on a new classification framing of unlearning (exemplified in our ELM method) can be more robust than previous approaches. In other words, our paper affirmatively answers the question “Is there a rational optimization for erasing a concept that is better than simply adding noise to the model?”

In response to the reviewers’ collective request for comprehensive baseline comparisons in Table 1, we have expanded our experimental evaluation. As numbers for these models were not provided by the original authors, we conducted several sweeps of hyperparameter optimization across RMU and RepNoise methods, described in Appendix B.1. The updated Table 1 now provides a thorough comparison under the best possible results we could obtain for each method. 

The expanded results demonstrate that ELM achieves consistently better erasure rates while maintaining generation perplexity compared to the strongest baseline. These improvements are consistent across multiple datasets and target concepts. Notably, this performance advantage holds even when comparing against the best-performing configurations of each baseline, reinforcing the robustness of our approach.

We also conducted additional adversarial and fine-tuning attacks to show robustness of ELM. We found that the current state-of-the-art adversarial attack methods, BEAST, fails to attack ELM in bringing back erased concepts, showing ELM's strong robustness. 

To show the importance of ELM's objectives compared to RMU's activation disruption, we trained a hybrid variant of RMU with ELM's consistency objective. We found that RMU's erasure objective aims to randomize and scale the activations which acts orthogonal to consistent/coherent text generation. The addition of consistency to RMU's objective led to trade-off in erasure vs seamlessness (coherent text generation). This shows the importance of  ELM's conceptual contribution - the erasure object and consistency objective act in tandem to reduce the likelihood of a concept being generated while maintaining good text coherence. 

We hope this addresses the reviewers' concerns and would love to answer any further questions
</content>
</thread_object>

</thread>
<thread>
creation date:
1733105740781

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback. We have conducted extensive additional experiments to address the raised concerns and would like to summarize the findings:

1. Comprehensive Baseline Comparisons
- We have expanded our main experimental evaluation with thorough hyperparameter optimization sweeps across RMU and RepNoise methods for all the 4 models (results in Table.1 and baseline details in Appendix B.1)
- The updated results further demonstrates that ELM achieves consistently better erasure rates while maintaining generation perplexity compared to baselines
- This performance advantage holds even when comparing against optimized configurations of each baseline

2. Robustness Analysis
- We expanded our robustness evaluations by incorporating state-of-the-art attacks including BEAST and systematic fine-tuning attacks
- BEAST attacks, which successfully compromise the safety aligned base models, fail to recover erased concepts from ELM - further supporting ELM's robustness towards adversarial attacks
- After aggressive fine-tuning attempts, ELM maintains a significant 22.4 and 15.3 percentage point gap from original performance on Bio and Cyber tasks respectively

3. Architectural Choices
- Full fine-tuning experiments confirmed an expected phenomenon, that we see as fully consistent with our formulation: full finetuning affects general performance. For instance, MMLU scores down to 45.2 (compared to original 58.5 and ELM LoRA's 56.6). 
- This validates our choice of low-rank adaptation to maintain model capabilities while targeting specific knowledge. 
- Previous observations made in [1, 2] have led to the consensus that specific knowledge tends to correspond to low-rank parameter directions, and it is to be expected that editing weights without a rank constraint would increase impacts on the broader distribution rather than specific targeted knowledge.

4. Method Comparison with RMU
We conducted experiments combining RMU with our consistency objective to better understand the differences:
- RMU's activation disruption objective operates orthogonally to consistent text generation
- ELM's focused erasure objective works in tandem with the consistency loss
- This synergy enables ELM to achieve 16.9% better erasure while maintaining generation quality

5. Clarification on Open-ended Generation based Evaluations
- MT-Bench evaluations demonstrate ELM's capabilities in open-ended scenarios
- Generated text is evaluated by GPT-4, complementing our reverse-perplexity measurements
- Qualitative examples showcase coherent outputs while maintaining erasure

We believe these additional results further reinforces ELM's effectiveness as a principled approach to concept erasure. ELM demonstrates robust performance across multiple evaluation criteria while maintaining model capabilities on unrelated tasks.

We welcome your review of these revisions to ensure they fully address your previous comments and suggestions. We are happy to provide any additional details or address further questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission obtains three negative rating. Several limitations of this version are still existed according to the results like limited experiments and non-significant improvements, undesirable effects of full finetuning, and more comparison with baselines. After rebuttal, two reviewers replied and the final ratings are still all negative. The AC decides the current version is still not ready for publication.