MUSE: Machine Unlearning Six-Way Evaluation for Language Models

## Abstract

Abstract Language models (LMs) are trained on vast amounts of text data, which may include private and copyrighted content, and data owners may request the removal of their data from a trained model due to privacy or copyright concerns.
However, exactly unlearning only these datapoints (i.e., retraining with the data removed) is intractable in modern-day models, leading to the development of many approximate unlearning algorithms.
Evaluation of the efficacy of these algorithms has traditionally been narrow in scope, failing to precisely quantify the success and practicality of the algorithm from the perspectives of both the model deployers and the data owners.
We address this issue by proposing MUSE , a comprehensive machine unlearning evaluation benchmark that enumerates six diverse desirable properties for unlearned models: (1) no verbatim memorization, (2) no knowledge memorization, (3) no privacy leakage, (4) utility preservation on data not intended for removal, (5) scalability with respect to the size of removal requests, and (6) sustainability over sequential unlearning requests.
Using these criteria, we benchmark how effectively eight popular unlearning algorithms on 7B-parameter LMs can unlearn Harry Potter books and news articles.
Our results demonstrate that most algorithms can prevent verbatim memorization and knowledge memorization to varying degrees, but only one algorithm does not lead to severe privacy leakage.
Furthermore, existing algorithms fail to meet deployer’s expectations, because they often degrade general model utility and also cannot sustainably accommodate successive unlearning requests or large-scale content removal.
Our findings identify key issues with the practicality of existing unlearning algorithms on language models, and we release our benchmark to facilitate further evaluations. 1 1 1 Our dataset and benchmark are available at https://muse-bench.github.io

## 1 Introduction

Figure: Figure 1: MUSE evaluation focuses on six key dimensions of machine unlearning, addressing both data owner and deployer expectations. For example, when an author (data owner) requests the unlearning of the Harry Potter books, they may expect the unlearned model to: (1) avoid generating verbatim copies of the text to protect copyright, (2) eliminate retention of factual knowledge from the books, and (3) not reveal whether the books were previously used in training to protect privacy. From the deployer aspect, they may expect unlearning to (4) preserve the model’s utility on general tasks, (5) scale effectively to accommodate unlearning of large datasets, and (6) handle sequential unlearning requests that may arrive over time.
Refer to caption: x1.png

Training language models (LMs) often involves using vast amounts of text data, which may inadvertently contain private and copyrighted content (Carlini et al., ; Henderson et al., ; Min et al., ; He et al., ).
In real-world applications, data owners may demand that their data be removed from a trained language model due to privacy or copyright concerns, as mandated for example by the General Data Protection Regulation (GDPR, ).
Moreover, recent copyright lawsuits (*DOE 1 v. GitHub, Inc.*, ; *Tremblay v. OpenAI, Inc.,*, )
emphasize the need for removing copyrighted data from the model.

These recent developments have intensified research interest in designing, evaluating, and improving *machine unlearning* algorithms, which aim to transform an existing trained model into one that behaves as though it had never been trained on certain data (Ginart et al., ; Liu et al., ; Wu et al., ; Bourtoule et al., ; Izzo et al., ; Gupta et al., ; Sekhari et al., ; Ye et al., ; Ghazi et al., ).
Exact unlearning in LMs requires removing the undesired data (the *forget set*) and retraining the model from scratch on the remaining data (the *retain set*), which is too costly to be practical, especially for frequent unlearning operations.
As such, several efficient approximate unlearning algorithms have been proposed (Eldan & Russinovich, ; Zhang et al., ), but existing evaluations of LM unlearning
on question answering (Eldan & Russinovich, ; Maini et al., ) cannot provide a holistic view of how practical and effective a particular unlearning algorithm is.
In this work, we propose a systematic, multi-faceted framework called MUSE (Machine Unlearning Six-Way Evaluation; §3) to evaluate six desired properties for unlearning algorithms (Figure 1).
Our criteria cover both the data owner’s and the model deployer’s desiderata for a practical unlearning algorithm.
Data owners require the LM to unlearn the precise tokens (*verbatim memorization*), general knowledge encoded in the tokens (*knowledge memorization*), and any indication that their data was included in the training set to begin with (*privacy leakage*).
On the other hand, model deployers want to effectively accommodate many successive unlearning requests (*sustainability*) on various sizes of forget sets (*scalability*) without degrading the general model capabilities (*utility preservation*).

We apply MUSE to evaluate eight representative machine unlearning algorithms (§4) on two datasets (§3.2), focusing on the specific cases of unlearning Harry Potter books and news articles.
Our findings indicate that most unlearning algorithms remove verbatim memorization and knowledge memorization with varying degrees of efficacy but operate at the cost of utility preservation and do not effectively prevent privacy leakage (§5.2).
In particular, negative preference optimization (NPO; Zhang et al., ) and task vectors (Ilharco et al., ) are especially effective in removing these types of memorization, but we find that NPO often permits privacy leakage and both methods induce a sharp drop in the utility of the model.
Furthermore, testing their scalability and sustainability reveals that they both algorithms struggle with large forget sets and successive unlearning requests (§5.3).

Our results highlight that unlearning algorithms generally fail to meet data owner expectations in preventing privacy leakage, which is one of the primary motivations for unlearning.
Additionally, they struggle to meet all three of the aforementioned deployer expectations.
Therefore, although it is increasingly desirable to find an efficient and effective unlearning algorithm amid rising concerns around privacy regulations and copyright litigations, our evaluation suggests that currently feasible unlearning methods are not yet ready for meaningful usage or deployment in real-world scenarios. These findings underscore the pressing need for further research in this area. We also release our benchmark to facilitate further evaluations and welcome extensions to other modalities.

## 2 Machine Unlearning: Preliminaries and Notations

Machine unlearning (Ginart et al., ; Liu et al., ; Izzo et al., ; Sekhari et al., ; Gupta et al., ; Ye et al., ; Liu et al., )
has emerged as an
important capability to accommodate
data removal requirements that arise from scenarios with privacy or copyright concerns.

We briefly describe the machine unlearning setting.
Consider a dataset $\mathcal{D}_{\textrm{train}}$ and a model $f_{\textrm{target}}$ trained on $\mathcal{D}_{\textrm{train}}$.
Suppose we design an algorithm $\mathcal{U}$ to unlearn a specific subset (i.e., the *forget set*) $\mathcal{D}_{\textrm{forget}}\subset\mathcal{D}_{\textrm{train}}$ from $f_{\textrm{target}}$.
We want to preserve performance on a *retain set* $\mathcal{D}_{\textrm{retain}}=\mathcal{D}_{\textrm{train}}\setminus\mathcal{D}
_{\textrm{forget}}$, and we also evaluate the model on an in-distribution but disjoint *hold-out set* $\mathcal{D}_{\textrm{holdout}}$ which the model has never been trained on.
So, the unlearning algorithm $\mathcal{U}$ takes $f_{\textrm{target}}$, $\mathcal{D}_{\textrm{forget}}$, and, optionally, $\mathcal{D}_{\textrm{retain}}$ and outputs an unlearned model $f_{\textrm{unlearn}}$.
Exact unlearning ensures $f_{\textrm{unlearn}}$ is behaviorally identical to the model resulting from retraining from scratch, denoted $f_{\textrm{target}}$, but such retraining is usually too costly in real world deployment, so we focus on evaluating approximate unlearning algorithms.

## 3 The MUSE Evaluation Benchmark

**Table 1: Comparison with a previous benchmark: Unlike the previous benchmark TOFU (Maini et al., ), which evaluates unlearning on synthetic Q&A datasets, MUSE tackles real-world unlearning challenges: unlearning real-world large-scale corpus (22$\times$ larger) while taking into account six desiderata that are important to both data owners and deployers. More related works are discussed in Appendix 6.**
|  |  | MUSE (ours) | TOFU (Maini et al., ) |
| --- | --- | --- | --- |
| \pbox1.5cmEvaluation criteria | C1. No verbatim memorization | $\checkmark$ |  |
| C2. No knowledge memorization | $\checkmark$ | $\checkmark$ |  |
| C3. No privacy leakage | $\checkmark$ |  |  |
| C4. Utility preservation | $\checkmark$ | $\checkmark$ |  |
| C5. Scalability | $\checkmark$ |  |  |
| C6. Sustainability | $\checkmark$ |  |  |
| \pbox1.5cmEvaluation corpora | Domains | News and Books | Synthetic autobiographies |
| Data Constitution | Verbatim text and knowledge set (Q & A) | Q & A |  |
| Scale ($\#$ tokens in forget set) | 0.8M for News, 3.3M for Books | 0.15M |  |

MUSE evaluates a comprehensive
set of desirable properties of machine unlearning across six facets. We detail the evaluation metrics in §3.1 and describe the evaluation corpus in §3.2.

### 3.1 Evaluation Metrics

Ideally, an unlearned model should behave as if it had never seen the forget set, exhibiting similar behavior to a retrained model on any corpus $\mathcal{D}$ such that $m(f_{\textrm{unlearn}},\mathcal{D})\approx m(f_{\textrm{retrain}},\mathcal{D})$, where $m$ represents any evaluation metric.
Prior
evaluations on LM unlearning focus on
performance of specific tasks like question answering (e.g., Eldan & Russinovich, ; Maini et al., ).
However, these metrics do not faithfully reflect data owner expectations and real-world deployment considerations when performing unlearning.
To address this, we propose comprehensive evaluation metrics that consider both data owner and deployer expectations. A comparison between MUSE and the prior benchmark is shown in Table 1.

Data owner expectations.
When removing a forget set from a model, data owners typically have three main expectations regarding the unlearned model:
(C1) No verbatim memorization: The model should not exactly replicate any details from the forget set.
(C2) No knowledge memorization: The model should be incapable of responding to questions about the forget set.
(C3) No privacy leakage: It should be impossible to detect that the model was ever trained on the forget set.
For example, if a patient’s records are unlearned from a medical diagnosis model, in addition to verbatim and knowledge memorization checks,
it is also important that the patient’s privacy is preserved – we follow established practice in quantifying privacy using the membership inference test, which detects if a specific datapoint was used to train the model (member), distinguishing it from non-training data (non-member) (Shokri et al., ).
In this case of unlearning a record from a diagnostic model, it is undesirable for the model to leak membership information, because it would be used to associate the patient with the disease.
We quantify these data owner expectations with three evaluation metrics:

When a model has unlearned a medical record, it should not output its contents verbatim.
We quantify the verbatim memorization $\mathsf{VerbMem}$ by prompting the model with the first $l$ tokens from a sequence $x_{[:l]}\in\mathcal{D}_{\textrm{forget}}$ and comparing the continuation outputted by the model $f$ to the true continuation $x_{[l+1:]}\in\mathcal{D}_{\textrm{forget}}$ using the ROUGE-L F1 score (Lin, ).

$$ $\mathsf{VerbMem}(f,\mathcal{D}):=\frac{1}{|\mathcal{\mathcal{D}_{\textrm{ forget}}}|}\sum_{x\in\mathcal{\mathcal{D}_{\textrm{forget}}}}\mathsf{ROUGE}(f( x_{[:l]}),x_{[l+1:]})$ $$

When a model has unlearned a medical record, it should no longer be able to answer questions about that record.
We measure a model $f$’s memorization of knowledge from the forget set $\mathcal{\mathcal{D}_{\textrm{forget}}}$ as follows: for each example $x\in\mathcal{\mathcal{D}_{\textrm{forget}}}$ associated with a question-answer pair $(q,a)$,(^2^22Examples of question-answer pairs derived from the original corpus can be found in Appendix D.) we gather the model’s answer to the question $q$, denoted $f(q)$.
We then average the ROUGE scores for all question-answer pairs in $\mathcal{\mathcal{D}_{\textrm{forget}}}$ to compute the knowledge memorization score $\mathsf{KnowMem}$:

$$ $\mathsf{KnowMem}(f,\mathcal{\mathcal{D}_{\textrm{forget}}}):=\frac{1}{| \mathcal{\mathcal{D}_{\textrm{forget}}}|}\sum_{(q,a)\in\mathcal{\mathcal{D}_{ \textrm{forget}}}}\mathsf{ROUGE}(f(q),a)$ $$

Figure: Figure 2: Distribution of the MIA metric (see C3) for $\mathcal{D}_{\textrm{forget}}$, $\mathcal{D}_{\textrm{holdout}}$, and $\mathcal{D}_{\textrm{retain}}$. Differences in the metric between forget and holdout sets indicate various unlearning outcomes of $\mathcal{D}_{\textrm{forget}}$, potentially leaking privacy. A perfectly unlearned model (b) should show similar MIA metrics distribution for $\mathcal{D}_{\textrm{forget}}$ and $\mathcal{D}_{\textrm{holdout}}$. Unlearning methods may fail by under-unlearning $\mathcal{D}_{\textrm{forget}}$, making it similar to $\mathcal{D}_{\textrm{retain}}$ (c), or over-unlearning it, causing divergence from $\mathcal{D}_{\textrm{holdout}}$ (d).
Refer to caption: extracted/5730856/figures/mia.png

As discussed previously, it is desirable that the unlearned model does not leak membership information indicating that $\mathcal{D}_{\textrm{forget}}$ was part of $\mathcal{D}_{\textrm{train}}$. To determine if a given example was used during training, membership inference attack (MIA) exploits distributional differences in certain statistics (e.g., loss) between training (member) and non-training (non-member) data:
if the loss on the example is low, then it was likely used for training.
As shown in Figure 2, unlearning typically increases the loss on the example, but there are two possible ways that unlearning can fail to prevent privacy leakage: (1) *under-unlearning*, when the loss is not made large enough; and (2) *over-unlearning*, when the loss is made abnormally large.
To accurately measure the privacy leakage, we employ Min-K% Prob (Shi et al., )
, a state-of-the-art MIA method for LMs based on the loss, and compute the standard AUC-ROC score (Murakonda et al., ; Ye et al., ) of discriminating $\mathcal{D}_{\textrm{forget}}$ (members) and $\mathcal{D}_{\textrm{holdout}}$ (non-members).(^3^33An MIA algorithm compares its score to a given threshold to classify a given datapoint as a member or non-member. The AUC-ROC is a single value that summarizes the overall performance of the MIA algorithm by measuring its ability to discriminate between members and non-members across all possible thresholds.)
By comparing the AUC score with that of the retrained model, we define(^4^44Generally, $\textsf{AUC}(f_{\textrm{retrain}};\mathcal{D}_{\textrm{forget}},\mathcal{D}_{ \textrm{holdout}})\approx 0.5$, though sometimes there are intrinsic distribution shifts between $\mathcal{D}_{\textrm{forget}}$ and $\mathcal{D}_{\textrm{holdout}}$ that may bias the baseline away from 0.5.)

$$ $\textsf{PrivLeak}:=\frac{\textsf{AUC}(f_{\textrm{unlearn}};\mathcal{D}_{ \textrm{forget}},\mathcal{D}_{\textrm{holdout}})-\textsf{AUC}(f_{\textrm{ retrain}};\mathcal{D}_{\textrm{forget}},\mathcal{D}_{\textrm{holdout}})}{ \textsf{AUC}(f_{\textrm{retrain}};\mathcal{D}_{\textrm{forget}},\mathcal{D}_{ \textrm{holdout}})},$ $$

The PrivLeak metric for a good unlearning algorithm should be close to zero, whereas an over/under-unlearning algorithm will get a large positive/negative metric.

Deployer expectations.
Model deployers have their own considerations for using unlearning algorithms in the real world.
Unlearning specific datapoints can unpredictably degrade model capabilities in ways that are difficult to recover.
Moreover, deployers are expected to effectively accommodate somewhat large-scale forget sets and successive unlearning requests from data owners. As such, we
consider three key metrics: (C4) utility preservation on the retain set, (C5) scalability to handle large-scale content removal, and (C6) sustainability to maintain performance over sequential unlearning requests.

Model capabilities are often hard-won through expensive training procedures, so deployers would want an unlearning algorithm that preserves performance on the retain set. To quantify this, we evaluate the unlearned model’s performance on the retain set using the knowledge memorization metric $\mathsf{KnowMem}(f_{\textrm{unlearn}},\mathcal{D}_{\textrm{retain}})$.

We assess the scalability of unlearning methods by examining their performance on forget sets of varying sizes. Let $\mathcal{D}_{u}^{c}$ denote a forget set of size $c$, and $f_{u}^{c}$ be the corresponding unlearned model.
For any data owner-valued metric such as utility preservation, we measure scalability by analyzing the trend of this metric as $c$ increases from small to large values.

Machine unlearning operations often need to be applied sequentially, as data removal requests may arrive at different times.(^5^55For example, under GDPR, if Alice requests the removal of her data and Bob submits another removal request 31 days later, both requests must be fulfilled within 30 days. This requires the model deployer to first unlearn Alice’s data and then process Bob’s request on the updated model.)
We denote the unlearned model after processing the $k$-th request as $f_{u,k}$. To measure sustainability, we analyze the trend of any data owner-valued metric as the number of sequential unlearning requests $k$ increases.

### 3.2 Evaluation Corpus

MUSE considers two representative types of textual data that may frequently involve unlearning requests: news articles (*Tremblay v. OpenAI, Inc.,*, ) and books (Eldan & Russinovich, ).
These datasets are detailed as follows:

- •
News consists of BBC news articles (Li et al., ) collected after August 2023. All articles are randomly divided into (disjoint) forget, retain, and holdout sets.
- •
Books consists of the Harry Potter book series. To simulate a real-world setting for testing utility preservation (C4), we include different types of materials in the forget and retain sets. The forget set contains the original books, while the retain set contains related content from the Harry Potter FanWiki,(^6^66[harrypotter.fandom.com/wiki](harrypotter.fandom.com/wiki)) representing domain knowledge that should be retained after unlearning.

**Table 2: Examples of MUSE. Each corpus has Verbatim text and Knowledge sets (QA pairs derived from the original text) for evaluating verbatim and knowledge memorization. In News, $\mathcal{D}_{\textrm{forget}}$ and $\mathcal{D}_{\textrm{retain}}$ are two disjoint sets of news articles. In Books, $\mathcal{D}_{\textrm{forget}}$ is the Harry Potter book series while $\mathcal{D}_{\textrm{retain}}$ consists of wiki articles about the series. The sizes of the forget and retain sets are reported in tokens in ().**
| Corpus | Forget Set | Retain Set |
| --- | --- | --- |
|  | News Article (0.8 M tokens) | News Article (1.6 M tokens) |
| News | \pbox<br>6.5cm MP Stuart McDonald has been appointed as the SNP’s |  |
| new treasurer | \pbox<br>5.5cm A father whose 12-year-old son was killed by an IRA bomb 30 years ago |  |
|  | \pbox<br>6.5cm Q: What position has Stuart McDonald MP been appointed to? |  |
| A: The SNP’s new treasurer | \pbox<br>6.5cm Q: Who was affected by the IRA bomb 30 years ago? |  |
| A: A father whose 12-year-old son |  |  |
|  | Harry Potter Books (1.1 M tokens) | Harry Potter FanWiki (0.5 M tokens) |
| Books | \pbox<br>5.5cm ‘‘There’s more in the frying pan,’’ said Aunt Petunia, |  |
| turning eyes on her massive son. | \pbox<br>5.5cm This page contains a list of spells: |  |
| Portuguese for ‘open’. |  |  |
|  | \pbox<br>5.5cm Q: What does Aunt Petunia tell her son? |  |
| A: There’s more in the frying pan. | \pbox<br>5.5cm Q: What is the spell used to open things? |  |
| A: Portuguese |  |  |

## 4 Unlearning Methods

We evaluate eight efficient approximate unlearning methods belonging to four families of algorithms.

Four families of unlearning methods.
We first introduce four families of unlearning methods, which serve as the basis for the eight methods we evaluate.

- •
Gradient Ascent (GA) minimizes the likelihood of correct predictions on $\mathcal{D}_{\textrm{forget}}$ by performing gradient ascent on the cross-entropy loss (the opposite of conventional learning with gradient descent).
GA has achieved mixed results: while Jang et al. () found it effective for unlearning examples from the Enron email dataset (Klimt & Yang, ) with minimal performance degradation, Ilharco et al. () reported that GA significantly harms general model utility when unlearning a high-toxicity subset of the Civil Comments dataset (Borkan et al., ).
- •
Negative Preference Optimization (NPO; Zhang et al., ) treats the forget set as negative preference data and adapts the offline DPO objective (Rafailov et al., ) to tune the model to assign low likelihood to the forget set without straying too far from the original model $f_{\textrm{target}}$.
$\displaystyle\mathcal{L}_{\mathrm{NPO}}(\theta)=-\frac{2}{\beta}\mathbb{E}_{x
\sim\mathcal{D}_{\textrm{forget}}}\left[\log\sigma\left(-\beta\log\frac{f_{
\theta}(x)}{f_{\textrm{target}}(x)}\right)\right],$
where $f_{\theta}$ refers to the model that undergoes unlearning, $\sigma$ is the sigmoid function, and $\beta$ is a hyperparameter that controls the allowed divergence of $f_{\theta}$ from its initialization $f_{\textrm{target}}$.
Following Rafailov et al. (); Zhang et al. (), we fix $\beta=0.1$ in our experiments.
- •
Task Vectors (Ilharco et al., ) derived from straightforward arithmetic on the model weights can effectively steer neural network behavior.
We adapt task vectors to perform unlearning in two stages.
First, we train
$f_{\textrm{target}}$ on $\mathcal{D}_{\textrm{forget}}$ until the model overfits, yielding a reinforced model $f_{\textrm{reinforce}}$.
We then obtain a task vector related to $\mathcal{D}_{\textrm{forget}}$ by calculating the weight difference between $f_{\textrm{target}}$ and $f_{\textrm{reinforce}}$.
To achieve unlearning, we subtract this task vector from $f_{\textrm{target}}$’s weights, intuitively moving the model away from the direction it used to adapt to $\mathcal{D}_{\textrm{forget}}$ – i.e., $f_{\textrm{unlearn}}=f_{\textrm{target}}-(f_{\textrm{reinforce}}-f_{\textrm{
target}})$.
- •
Who’s Harry Potter (WHP; Eldan & Russinovich, ) defines the unlearned model $f_{\textrm{unlearn}}$ as the interpolation between the target model $f_{\textrm{target}}$ and the reinforced model $f_{\textrm{reinforce}}$.
Let $p_{f}(\cdot|x)$ denote the token distribution parametrized by the model $f$ when given a prompt $x$ as input.
Then, concretely, for any input $x$, WHP samples the next token from
p_f_unlearn(⋅| x) = p_f_target(⋅| x) - α(p_f_reinforce(⋅| x) - p_f_target(⋅| x))
where $\alpha$ is a hyperparameter that controls the interpolation between the two models.

Two regularizers for utility preservation.
GA and NPO are not explicitly designed for utility preservation, so we discuss several regularization strategies that either improve the performance on the retain set or ensure the unlearned model remains close to the target model during unlearning.

- •
Gradient Descent on the Retain Set (GDR;
Liu et al., ; Maini et al., ; Zhang et al., ) augments the unlearning objective with a standard gradient descent learning objective on the cross-entropy of the retain set $\mathcal{D}_{\textrm{retain}}$ to more directly train the model to maintain its performance on $\mathcal{D}_{\textrm{retain}}$.
- •
KL Divergence Minimization on the Retain Set (KLR;
Maini et al., ; Zhang et al., ) encourages the unlearned model’s probability distribution $p_{f_{\textrm{unlearn}}}(\cdot|x)$ to be close to the target model’s distribution $p_{f_{\textrm{target}}}(\cdot|x)$ on inputs from the retain set $x\in\mathcal{D}_{\textrm{retain}}$.

List of methods.
We combine GA and NPO with the two regularizers GDR and KLR,(^7^77 These regularizers are not compatible with Task Vector and WHP, because Task Vector involves purposefully overfitting a model to $\mathcal{D}_{\textrm{forget}}$ when deriving the task vector, and WHP is a test-time technique where the unlearning operation involves no optimization by itself.) which yields four new combinations.
Hence, we end up with a total of 8 candidate unlearning methods:
GA, $\textsf{GA}_{\textsf{GDR}}$, $\textsf{GA}_{\textsf{KLR}}$, NPO, $\textsf{NPO}_{\textsf{GDR}}$, $\textsf{NPO}_{\textsf{KLR}}$, Task Vector, and WHP.
In general, the cost of the approximate unlearning method is negligible compared to retraining. Details about the efficiency of these methods are reported in Appendix B.3.

## 5 Experiments

We evaluate the eight representative unlearning methods using the experimental setup described in §5.1. We present the results for data owner expectations in §5.2 and for deployer expectations in §5.3.

### 5.1 Experimental Setup

**Table 3: Most unlearning methods effectively remove verbatim and knowledge memorization but significantly impact utility and privacy. We evaluate the 8 algorithms described in §4 on 4 of the criteria in MUSE. We include the results of $f_{\textrm{retrain}}$ for reference. We highlight results in blue if the unlearning algorithm satisfies the criterion and highlight it in orange otherwise. For privacy leakage, large positive values suggest over-unlearning, while large negative values suggest under-unlearning (see §3.1). This table covers the results for C1 to C4, while results for C5 and C6 are shown in Figure 6.**
|  | C1. No Verbatim Mem. | C2. No Knowledge Mem. | C3. No Privacy Leak. | C4. Utiltiy Preserv. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | VerbMem on $\mathcal{D}_{\textrm{forget}}$ ($\downarrow$) | KnowMem on $\mathcal{D}_{\textrm{forget}}$ ($\downarrow$) | PrivLeak ($\in[-5\%,5\%]$) | KnowMem on $\mathcal{D}_{\textrm{retain}}$ ($\uparrow$) |  |  |  |  |
| News |  |  |  |  |  |  |  |  |
| Target $f_{\textrm{target}}$ | $58.4$ |  | $63.9$ |  | $-99.8$ |  | $55.2$ |  |
| Retrain $f_{\textrm{retrain}}$ | $\mathbf{20.8}$ |  | $\mathbf{33.1}$ |  | $\mathbf{0.0}$ |  | $\mathbf{55.0}$ |  |
| GA | $0.0$ | $\downarrow$100% | $0.0$ | $\downarrow$100% | $5.2$ | over-unlearn | $0.0$ | $\downarrow$100% |
| $\textsf{GA}_{\textsf{GDR}}$ | $4.9$ | $\downarrow$76.5% | $31.0$ | $\downarrow$6.3% | $108.1$ | over-unlearn | $27.3$ | $\downarrow$50.3% |
| $\textsf{GA}_{\textsf{KLR}}$ | $27.4$ | $\uparrow$31.4% | $50.2$ | $\uparrow$51.5% | $-96.1$ | under-unlearn | $44.8$ | $\downarrow$18.5% |
| NPO | $0.0$ | $\downarrow$100% | $0.0$ | $\downarrow$100% | $24.4$ | over-unlearn | $0.0$ | $\downarrow$100.0% |
| $\textsf{NPO}_{\textsf{GDR}}$ | $1.2$ | $\downarrow$94.4% | $54.6$ | $\uparrow$64.8% | $105.8$ | over-unlearn | $40.5$ | $\downarrow$26.3% |
| $\textsf{NPO}_{\textsf{KLR}}$ | $26.9$ | $\uparrow$29.0% | $49.0$ | $\uparrow$48.1% | $-95.8$ | under-unlearn | $45.4$ | $\downarrow$17.4% |
| Task Vector | $57.2$ | $\uparrow$174.7% | $66.2$ | $\uparrow$100.0% | $-99.8$ | under-unlearn | $55.8$ | $\uparrow$1.5% |
| WHP | $19.7$ | $\downarrow$5.6% | $21.2$ | $\downarrow$35.9% | $109.6$ | under-unlearn | $28.3$ | $\downarrow$48.5% |
| Books |  |  |  |  |  |  |  |  |
| Target $f_{\textrm{target}}$ | $99.8$ |  | $59.4$ |  | $-57.5$ |  | $66.9$ |  |
| Retrain $f_{\textrm{retrain}}$ | $\mathbf{14.3}$ |  | $\mathbf{28.9}$ |  | $\mathbf{0.0}$ |  | $\mathbf{74.5}$ |  |
| GA | $0.0$ | $\downarrow$100% | $0.0$ | $\downarrow$100% | $-25.0$ | under-unlearn | $0.0$ | $\downarrow$100% |
| $\textsf{GA}_{\textsf{GDR}}$ | $0.0$ | $\downarrow$100% | $0.0$ | $\downarrow$100% | $-26.5$ | under-unlearn | $10.7$ | $\downarrow$85.6% |
| $\textsf{GA}_{\textsf{KLR}}$ | $16.0$ | $\uparrow$11.4% | $21.9$ | $\downarrow$24.4% | $-40.2$ | under-unlearn | $37.2$ | $\downarrow$50.0% |
| NPO | $0.0$ | $\downarrow$100% | $0.0$ | $\downarrow$100% | $-24.3$ | under-unlearn | $0.0$ | $\downarrow$100% |
| $\textsf{NPO}_{\textsf{GDR}}$ | $0.0$ | $\downarrow$100% | $0.0$ | $\downarrow$100% | $-30.8$ | under-unlearn | $22.8$ | $\downarrow$69.4% |
| $\textsf{NPO}_{\textsf{KLR}}$ | $17.0$ | $\uparrow$18.2% | $25.0$ | $\downarrow$13.4% | $-43.5$ | under-unlearn | $44.6$ | $\downarrow$40.1% |
| Task Vector | $99.7$ | $\uparrow$595.0% | $52.4$ | $\uparrow$81.2% | $-57.5$ | under-unlearn | $64.7$ | $\downarrow$13.1% |
| WHP | $18.0$ | $\uparrow$25.2% | $55.7$ | $\uparrow$92.9% | $56.5$ | over-unlearn | $63.6$ | $\downarrow$14.6% |

Retrained and target models.
We start with a general pretrained base model $f_{0}$, and finetune two models: $f_{\textrm{target}}$ on $\mathcal{D}_{\textrm{forget}}\cup\mathcal{D}_{\textrm{retain}}$, and $f_{\textrm{retrain}}$ on $\mathcal{D}_{\textrm{retain}}$ only. See Section B.2 for details about finetuning. For each unlearning algorithm $\mathcal{U}$, we further generate the unlearned model $f_{\textrm{unlearn}}=\mathcal{U}(f_{\textrm{target}},\mathcal{D}_{\textrm{
forget}},\mathcal{D}_{\textrm{retain}})$. We ensure that $f_{0}$ has no access to $\mathcal{D}_{\textrm{forget}},\mathcal{D}_{\textrm{retain}},\mathcal{D}_{
\textrm{holdout}}$. Therefore, for News, we use $f_{0}=\text{LLaMA-2 7B}$ (Touvron et al., ), which was released *before* the BBC news articles we use to construct our benchmarks; and for Books, we use $f_{0}=\text{ICLM-7B}$ (Shi et al., ), which does *not* contain the Harry Potter books in its pretraining data.

Unlearning experimental configuration.
Following prior work (Maini et al., ), we run GA, NPO, and their regularized variants using the AdamW optimizer (Loshchilov & Hutter, ) with a constant learning rate of $10^{-5}$ and a batch size of 32.
We employ the stopping criteria as follows:
if the utility (i.e., KnowMem on $\mathcal{D}_{\textrm{retain}}$) of a model undergoing unlearning drops below that of $f_{\textrm{retrain}}$ within 10 epochs of unlearning, we stop at the first epoch where this condition holds; otherwise, we take a checkpoint from the 10th epoch.
For Task Vector and WHP, to obtain the reinforced model for unlearning, we fine-tune the target model for 10 epochs using the same learning rate and batch size.
Further details on the model fine-tuning and unlearning can be found in Section B.2.

### 5.2 Results: Data Owner Expectations

We first analyze how eight unlearning methods meet data owner expectations (C1, C2 & C3 in §3.1).

C1&C2. Most methods are effective for unlearning memorization. As shown in Table 3, most unlearning methods perform exceptionally well in [C1. No verbatim memorization] and [C2. No knowledge memorization], often reducing VerbMem and KnowMem even beyond the levels achieved by the retrained model. Notably, some methods, such as GA and NPO, achieve a score of 0 for both VerbMem and KnowMem, meaning that these methods completely prevent the unlearned models from producing any text related to the forget set. However, as we will see later, these reductions often come at the cost of significant utility loss on the retain set.

Figure: Figure 3: Distribution of Min-K% Prob, an MIA metric, for $\mathcal{D}_{\textrm{forget}}$, $\mathcal{D}_{\textrm{holdout}}$, and $\mathcal{D}_{\textrm{retain}}$. Consistent with the expected pattern in Figure 2, $f_{\textrm{retrain}}$ shows perfect unlearning, with the overlapping distributions for $\mathcal{D}_{\textrm{forget}}$ and $\mathcal{D}_{\textrm{holdout}}$. Existing approximate unlearning methods typically either under-unlearn or over-unlearn. For example, $\textsf{GA}_{\textsf{KLR}}$ shows slight under-unlearning, while $\textsf{GA}_{\textsf{GDR}}$ over-unlearns, pushing the Min-K% Prob of $\mathcal{D}_{\textrm{forget}}$ to an extreme level.
Refer to caption: x2.png

Figure: Figure 4: ROC curves for $\mathcal{D}_{\textrm{forget}}$ vs. $\mathcal{D}_{\textrm{holdout}}$ on News using Min-K% Prob, with AUC scores in parentheses. AUC$\approx$0.5 (i.e., $f_{\textrm{retrain}}$) means no significant distribution difference between two sets (i.e., no membership leakage). Most unlearning methods show under-unlearn (AUC$\ll$0.5) or over-unlearn (AUC $\gg$0.5).
Refer to caption: x3.png

C3. Unlearning leads to privacy leakage. Most unlearning methods reveal the membership of $\mathcal{D}_{\textrm{forget}}$ in $\mathcal{D}_{\textrm{train}}$ through under-unlearning ($\textsf{PrivLeak}\ll 0$) or over-unlearning ($\textsf{PrivLeak}\gg 0$), as shown in Table 3. We further examine the effectiveness of membership inference by plotting ROC curves in Figure 5. The deviation from the diagonal line indicates the attacker’s advantage over random guessing. We observe that the Min-K% Prob based attack achieves $\textsf{AUC}\approx 0$ on $f_{\textrm{target}}$, confirming its effectiveness. Meanwhile, the ROC curve for $f_{\textrm{retrain}}$ closely follows the diagonal line ($\textsf{AUC}=0.47$), suggesting that perfect unlearning ensures MIA is no more effective than random guessing. Among the approximate unlearning methods, GA and $\textsf{NPO}_{\textsf{GDR}}$ without regularizers consistently over-unlearn ($\textsf{AUC}>0.7$), whereas KLR-regularized methods ($\textsf{NPO}_{\textsf{KLR}}$ and $\textsf{GA}_{\textsf{KLR}}$) tend to under-unlearn and barely improve privacy leakage over $f_{\textrm{target}}$.
WHP also deviates from the diagonal significantly.

In Figure 3, we further visualize the distribution of Min-K% Prob, the MIA metric computed across $\mathcal{D}_{\textrm{forget}}$, $\mathcal{D}_{\textrm{retain}}$, and $\mathcal{D}_{\textrm{holdout}}$.
The behavior of $f_{\textrm{target}}$ and $f_{\textrm{retrain}}$ mirrors the patterns sketched in Figure 2, where $\mathcal{D}_{\textrm{forget}}$ and $\mathcal{D}_{\textrm{retain}}$ are distinguishable in $f_{\textrm{target}}$ but overlap in $f_{\textrm{retrain}}$. Existing approximate unlearning methods typically either under-unlearn or over-unlearn. For example,
$\textsf{GA}_{\textsf{KLR}}$ does not sufficiently increase the Min-K% Prob metric for $\mathcal{D}_{\textrm{forget}}$ to align with the distribution of $\mathcal{D}_{\textrm{holdout}}$, indicating under-unlearning. On the other hand, $\textsf{NPO}_{\textsf{GDR}}$ over-unlearns, significantly raising the MIA metric across all datasets and especially for $\mathcal{D}_{\textrm{forget}}$.

### 5.3 Results: Deployment Considerations

C4. Unlearning significantly degrades model utility.
Table 3 [C4 Utility Preserv.] shows that all unlearning methods compromise the model’s utility by $24.2\%\sim 100\%$. Notably, several methods (GA, $\textsf{GA}_{\textsf{GDR}}$, $\textsf{NPO}_{\textsf{GDR}}$) lead to complete utility loss, rendering the unlearned models practically unusable. Figure 5 illustrates the trade-offs between utility preservation on $\mathcal{D}_{\textrm{retain}}$ and knowledge memorization on $\mathcal{D}_{\textrm{forget}}$. An ideal unlearned model should mimic the behavior of $f_{\textrm{retrain}}$ (desired region) by achieving a low level of memorization on $\mathcal{D}_{\textrm{forget}}$ while maintaining its utility. However, most methods, such as $\textsf{GA}_{\textsf{KLR}}$, $\textsf{NPO}_{\textsf{KLR}}$, and WHP, unlearn the knowledge on $\mathcal{D}_{U}$ at the cost of utility.

Figure: Figure 6: The performance of GA, NPO, and their regularized variants, measured by utility preservation, degrades with larger forget set sizes (a) and sequential unlearning requests (b).
Refer to caption: x5.png

C5. Unlearning methods scale poorly with forget set sizes.
To evaluate the robustness of the unlearning methods to larger forget sets, we collect additional news articles from the same distribution to scale our News corpus from 0.8M tokens to 3.3M tokens and observe the utility preservation at four different forget set sizes.
As shown in Figure 6 (a), the model utility decrease with the size of the forget set and achieves a minimum at the largest size.

C6. Unlearning methods cannot sustainably accommodate sequential unlearning requests.
To evaluate the robustness of these unlearning methods to more than one unlearning requests, we sequentially apply $k$ unlearning processes, each with respect to a different forget set.
To simulate sequential unlearning, we partition the extended News forget set (comprised of 3.3M tokens) into four disjoint folds (each containing 0.8M tokens) and apply the unlearning methods to each fold in a sequential manner.

We again select utility preservation as the target metric for comparison.
As shown in Figure 6 (b), the performance of an unlearned model tends to decrease significantly with respect to the number of unlearning requests, indicating that current unlearning methods are not yet ready to handle sequential unlearning in a sustainable manner.

## 6 Related Work

#### Machine unlearning for non-language model applications.

Machine unlearning is a long-running, well-studied topic.
Several studies have explored exact unlearning, aiming to make the unlearned model ($f_{\textrm{unlearn}}$) exactly identical to the reference model ($f_{\textrm{retrain}}$).
As expected, this can only be accomplished in simple models like SVMs (Cauwenberghs & Poggio, ; Tveit et al., ; Romero et al., ; Karasuyama & Takeuchi, ) or naive Bayes models (Cao & Yang, ).
Another approach is to ensure that the unlearned model $f_{\textrm{unlearn}}$ is probabilistically indistinguishable from $f_{\textrm{retrain}}$ (Ginart et al., ; Guo et al., ), and this view of certifiable unlearning is closely related to differential privacy (Dwork et al., , ).
This rigorous definition of unlearning has inspired several theoretical works that characterize the feasibility of unlearning in convex and non-convex models, but those proposed algorithms are too computationally costly to operate on modern-day LMs (Izzo et al., ; Neel et al., ; Ullah et al., ; Sekhari et al., ; Gupta et al., ).
Several more tractable unlearning algorithms have been proposed (Borkan et al., ; Ginart et al., ; Thudi et al., ; Chourasia & Shah, ) with broader applications such as image classification (Ginart et al., ; Golatkar et al., ), text-to-image generation (Gandikota et al., ; Zhang et al., ; Fan et al., ), Federated Learning (Liu et al., ; Che et al., ; Halimi et al., ; Huang et al., ) and Recommender Systems (Li et al., ).

#### Machine unlearning for language models: methods and applications.

Machine unlearning has recently found its way into language model applications. In §4, we discuss some standard unlearning methods based on parameter optimization, like the Gradient Ascent and its variance. Other notable non-training-based unlearning methods include localization-informed unlearning (Meng et al., ; Wu et al., ; Wei et al., ), which involves identifying model units (e.g., layers, neurons) closely related to the unlearning data or tasks and then locally editing and modifying the units. In-context unlearning (Pawelczyk et al., ) offers another approach, treating the model as a black box and modifying its output results using external knowledge.

Machine unlearning has also been applied to various downstream language model tasks, though the unit of machine unlearning may differ from what we study in this work. Our evaluation focuses on unlearning specific examples or datasets, aiming to make LMs forget either the phrasing or the content knowledge of targeted data, while preserving their utility for data not targeted for removal. This is crucial for ensuring privacy and copyright compliance. In addition to this specific unlearning, there’s also a broader application similar to model editing, where outdated information is replaced with new knowledge (Pawelczyk et al., ; Yu et al., ; Belrose et al., ). Moreover, efforts have been made to eliminate harmful behaviors in language models by creating toxicity benchmarks and enhancing safety measures (Lu et al., ; Yao et al., ; Li et al., ; Zhang et al., ).
Despite these varied approaches to unlearning at different operational and knowledge levels, the evaluation principles we propose such as preserving utility, ensuring scalability, and maintaining sustainability—are relevant across these contexts.

#### Machine unlearning for language models: evaluation.

Evaluating machine unlearning methods for language model applications is also critical. Most previous studies have focused this evaluation on specific tasks such as question answering or sentence completion. For example, Eldan & Russinovich () experiment with unlearning to forget Harry Potter books and demonstrate the effectiveness of their methods by showing that familiarity scores, measured through completion-based, token-probability-based, and question-answering evaluations, significantly decline post-unlearning. Lynch et al. () further suggest comparing unlearned models with perfectly retrained models. Their evaluation finds that while familiarity scores with the forget set may drop post-unlearning, they still remain higher than those of the retrained model. Wei et al. () evaluate the feasibility of using unlearning techniques to prevent language models from generating copyrighted content. The closest work to ours is TOFU (Maini et al., ), a benchmark featuring 200 synthetic author profiles, each with 20 question-answer pairs, divided into forget and retain sets. However, TOFU is relatively small-scale (0.15M tokens) and focuses on the evaluation of question answering.
Additionally, current evaluations focus on limited aspects of data owner expectations and do not adequately reflect real-world deployment considerations, such as scalability and potential sequential unlearning requests. In contrast, MUSE formally defines different unlearning scopes and corresponding metrics, resulting in a systematic six-way evaluation featuring both data owners’ and deployers’ expectations. The evaluation uses a large-scale corpus of over 6 million tokens, separated into verbatim text and knowledge sets. We also note that some of our findings align with previous evaluations. For example, our observation that over- or under-unlearn can exacerbate privacy leakage (§5.2) is consistent with the recent work by Hayes et al. (). Our findings align with the the concurrent study by Shumailov et al. () showing that unlearning gives a false sense of security as unlearned knowledge can resurface through in-context learning.

#### Survey papers.

We direct readers to several insightful survey papers for further reading. For non-LLM applications, notable surveys include Shintre et al. (); Nguyen et al. (); Thudi et al. (); Xu et al. (). Additionally, the NeurIPS 2023 machine unlearning competition for image classification(^8^88[https://unlearning-challenge.github.io](https://unlearning-challenge.github.io)) is a valuable source of empirical methods tailored for this specific application (Triantafillou et al., ). For language model applications, Si et al. () categorize unlearning methods into different families and summarize datasets for evaluating unlearning. Liu et al. () review LM unlearning algorithms by targets and methods, discuss the effectiveness and efficiency of existing approaches and emphasize the importance of clearly defining the unlearning scope.

## 7 Conclusion

In this work, we propose MUSE, a comprehensive machine unlearning evaluation benchmark that highlights six desirable properties from the perspectives of both data owners and model deployers. We find that current unlearning methods successfully prevent the model’s memorization of content at a significant cost to utility on data not intended for removal. They also lead to severe privacy leakage and cannot sustainably accommodate successive unlearning requests or large-scale content removal. These findings highlight the need for future research into more robust unlearning methods.

Limitations.
While MUSE provides a systematic benchmark for evaluating unlearning algorithms, it does not consider all possible considerations.
For example, data owners may have additional expectations, such as ensuring their information cannot be probed from intermediate activations (Song & Raghunathan, ) or receiving formal guarantees of unlearning success (Sekhari et al., ; Gupta et al., ; Ghazi et al., ). Similarly, deployers may expect other capabilities, like fine-tuning and in-context learning, to be preserved, and may prefer unlearning algorithms that are both computationally efficient and storage-wise cheap (e.g. does not need to keep a copy of the retain set).
MUSE currently evaluates unlearning for language models using books and news articles, but it could be extended to other corpora, such as medical notes (Johnson et al., , ) and emails (Klimt & Yang, ), which often involve privacy concerns (Li et al., ; Huang et al., ). We also plan to evaluate different-sized LMs in the future. Finally, our approach can be generalized to construct multi-faceted benchmarks for multimodal models (Golatkar et al., ; Cheng & Amiri, ; Zhang et al., ). Further
discussion on broader impact are in Appendix A.

## 8 Acknowledgements

We thank Eric Wallace, Robin Jia, Howard Chen, and anonymous reviewers of the GenLaw workshop for the valuable feedback and discussions.

[sections]
\printcontents[sections]l1

## Appendix A Broader Impact

As LMs are deployed broadly and publicly, there is mounting legal and social pressure on deployers to release models that permit effective unlearning when requested by data owners (; *DOE 1 v. GitHub, Inc.*, ; *Tremblay v. OpenAI, Inc.,*, ).
These incentives have prompted a flurry of new unlearning algorithms stemming from different technical perspectives.
As such, systematic evaluation of the strengths and weaknesses of these methods when executing realistic unlearning requests on popular models is essential.
MUSE disentangles several desirable properties of unlearning algorithms and finds that no existing algorithm is able to satisfy all of the data owner and deployer considerations.
We hope that our fine-grained, multi-faceted framework facilitates the improvement of unlearning algorithms.
Moreover, we expect that the general approach of designing metrics to balance the considerations of various stakeholders is flexible and can adapt to the rapidly shifting legal, social, and economic landscape.

We also acknowledge the potential negative impacts of our study. One limitation of our evaluation benchmark is that we do not have comprehensive study of how unlearning would impact the model performance for different user bases, especially underrepresented groups. However, we note proper handling and evaluation of fairness issues in unlearning is still an active ongoing research area (Zhang et al., ; Oesterling et al., ), therefore we leave it as future work. Additionally, our work may be misinterpreted towards skepticism regarding the broader use of machine unlearning, as our current evaluation reveals that existing unlearning methods are not yet ready for effective real-world deployment. However, machine unlearning, especially for large language models, is a young and active research area and new algorithms are constantly being proposed. We emphasize that our results is not a criticism of the paradigm of machine unlearning, but a study of the potential downsides of existing methods and a call for better algorithms. We believe our benchmark is an important step towards guiding future algorithm design of machine unlearning research towards more realistic deployment scenarios.

## Appendix B Experimental Details

### B.1 Compute Configurations

All experiments are conducted on 8 NVIDIA A40 GPU cards in a single node.

### B.2 Experimental Setup

#### Finetuning details.

As described in §5.1, for News, we start from $f_{0}=\text{LLaMA-2 7B}$ (Touvron et al., ) and finetune the model on the BBC news articles for 5 epochs with a constant learning rate of $10^{-5}$ and a batch size of 32 . For Books, we start from $f_{0}=\text{ICLM 7B}$ (Touvron et al., ) and finetune the model on the Harry Potter books with same set of hyperparameters.

#### Unlearning details.

For all the unlearning methods in Table 3, we use a constant learning rate of $10^{-5}$ and a batch size of 32.
For $f_{\text{reinforced}}$ used in WHP and Task Vector, we fine-tune $f_{\textrm{target}}$ for 10 epochs.

Before evaluation, for each unlearning method, we select its optimal epoch or $\alpha$ (both of which are parameters that control a degree of unlearning) by using our unlearning stopping criteria based on the unlearned model’s utility on $\mathcal{D}_{\textrm{retain}}$ compared to that of $f_{\textrm{retrain}}$.
The chosen epochs or $\alpha$’s for each method are listed below.

**Table 4: Optimal epochs or $\alpha$’s for each unlearning method.**
| Unlearning Method | News | Books |
| --- | --- | --- |
| GA | epoch 1 | epoch 1 |
| $\textsf{GA}_{\textsf{GDR}}$ | epoch 7 | epoch 1 |
| $\textsf{GA}_{\textsf{KLR}}$ | epoch 10 | epoch 5 |
| NPO | epoch 1 | epoch 1 |
| $\textsf{NPO}_{\textsf{GDR}}$ | epoch10 | epoch 1 |
| $\textsf{NPO}_{\textsf{KLR}}$ | epoch 10 | epoch 4 |
| Task Vector | $\alpha=2^{9}$ | $\alpha=2^{9}$ |
| WHP | $\alpha=2^{2}$ | $\alpha=2^{8}$ |

### B.3 Efficiency of Unlearning Methods

We report the efficiency of unlearning methods in Table 5, measured by the wall-clock time for a single gradient update step of unlearning. The time measurements were conducted using 8 NVIDIA A40 GPUs on a single node, with a batch size of 32 and an input length of 2048 tokens.
Each step corresponds to one gradient update processing a total of 65,536 tokens (32 $\times$ 2048 tokens). For Task Vector and WHP, each step represents one iteration of fine-tuning to create the reinforced model.

**Table 5: Wall-clock time required for each unlearning method, measured in seconds per step.**
| Unlearning Method | Time (Seconds/Step) |
| --- | --- |
| GA | 4.14 |
| $\textsf{GA}_{\textsf{GDR}}$ | 6.05 |
| $\textsf{GA}_{\textsf{KLR}}$ | 7.58 |
| NPO | 5.68 |
| $\textsf{NPO}_{\textsf{GDR}}$ | 7.59 |
| $\textsf{NPO}_{\textsf{KLR}}$ | 9.11 |
| Task Vector | 4.14 |
| WHP | 4.14 |

## Appendix C More Experimental Results

### C.1 Confidence Intervals for C1, C2 and C4 in Table 3

We compute confidence intervals for C1, C2, and C4 (Mean ROUGE-L F1) using bootstrapping(^9^99[https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html)).
For each mean ROUGE-L score reported in Table 3, we draw 9,999 bootstrap resamples and calculate a two-tailed 95% confidence interval using the “percentage” method.

**Table 6: 95% confidence intervals computed for mean Rouge-L scores used in C1, C2, and C4.**
|  | C1. No Verbatim Mem. | C2. No Knowledge Mem. | C4. Utiltiy Preserv. |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | VerbMem on $\mathcal{D}_{\textrm{forget}}$ ($\downarrow$) | KnowMem on $\mathcal{D}_{\textrm{forget}}$ ($\downarrow$) | KnowMem on $\mathcal{D}_{\textrm{retain}}$ ($\uparrow$) |  |  |  |
| News |  |  |  |  |  |  |
| Target $f_{\textrm{target}}$ | $58.4$ | [54.1, 62.9] | $63.9$ | [58.7, 69.0] | $55.2$ | [50.7, 59.9] |
| Retrain $f_{\textrm{retrain}}$ | $\mathbf{20.8}$ | [18.5, 23.7] | $\mathbf{33.1}$ | [26.8, 39.5] | $\mathbf{55.0}$ | [50.3, 59.8] |
| GA | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] |
| $\textsf{GA}_{\textsf{GDR}}$ | $4.9$ | [4.5, 5.2] | $31.0$ | [24.2, 38.0] | $27.3$ | [21.9, 33.0] |
| $\textsf{GA}_{\textsf{KLR}}$ | $27.4$ | [25.1, 29.9] | $50.2$ | [43.1, 56.9] | $44.8$ | [39.2, 50.5] |
| NPO | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] |
| $\textsf{NPO}_{\textsf{GDR}}$ | $1.2$ | [0.3, 2.3] | $54.6$ | [47.5, 61.5] | $40.5$ | [34.7, 46.2] |
| $\textsf{NPO}_{\textsf{KLR}}$ | $26.9$ | [24.7, 29.3] | $49.0$ | [41.8, 61.5] | $45.4$ | [39.8, 51.1] |
| Task Vector | $57.2$ | [52.6, 62.0] | $66.2$ | [61.3, 71.2] | $55.8$ | [51.0, 60.6] |
| WHP | $19.7$ | [17.8, 21.6] | $21.2$ | [16.0, 26.7] | $28.3$ | [23.3, 33.4] |
| Books |  |  |  |  |  |  |
| Target $f_{\textrm{target}}$ | $99.8$ | [99.8, 99.9] | $59.4$ | [52.7, 66.0] | $66.9$ | [59.6, 73.8] |
| Retrain $f_{\textrm{retrain}}$ | $\mathbf{14.3}$ | [13.6, 15.1] | $\mathbf{28.9}$ | [22.1, 35.7] | $\mathbf{74.5}$ | [68.4, 80.0] |
| GA | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] |
| $\textsf{GA}_{\textsf{GDR}}$ | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] | $10.7$ | [6.2, 15.7] |
| $\textsf{GA}_{\textsf{KLR}}$ | $16.0$ | [14.8, 17.2] | $21.9$ | [16.4, 27.7] | $37.2$ | [29.5, 45.0] |
| NPO | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] |
| $\textsf{NPO}_{\textsf{GDR}}$ | $0.0$ | [0.0, 0.0] | $0.0$ | [0.0, 0.0] | $22.8$ | [16.1, 30.1] |
| $\textsf{NPO}_{\textsf{KLR}}$ | $17.0$ | [15.7, 18.2] | $25.0$ | [19.0, 31.5] | $44.6$ | [36.5, 52.8] |
| Task Vector | $99.7$ | [99.6, 99.8] | $52.4$ | [45.0, 59.7] | $64.7$ | [57.1, 71.8] |
| WHP | $18.0$ | [16.4, 19.7] | $55.7$ | [48.6, 62.8] | $63.6$ | [56.3, 70.9] |

## Appendix D Dataset Details

GPT-generated QA pairs.
We begin the generation by partitioning the Verbatim text of each corpus into a set of 2048-token excerpts using LLaMA-2’s tokenizer.
For each QA pair to generate, we randomly sample an excerpt from this set and prompt GPT-4 (gpt-4o-2024-05-13) to create a JSON object with two fields: “question” (a question that can only be answered using specific information from the excerpt) and “answer” (an answer to the “question” extracted verbatim from the excerpt).
We validate and exclude any pairs whose answers cannot be found verbatim in their corresponding excerpts.
This verbatim requirement ensures that our Knowledge set is used precisely to evaluate the model’s ability to correctly associate questions with relevant portions of the training data.

For each QA pair to generate, we initiate a new conversation with GPT-4 with its corresponding excerpt. The instruction begins with a system prompt that specifies the desired format of generated QA pairs as follows:

We then present the excerpt as a user prompt to the model and collect the generated QA pairs. Here are two example generated QA pairs from the Knowledge set of News:

Dataset segmentation.
Appendix D shows examples from MUSE and
Table 8 presents detailed statistics for MUSE. For both the News and Books datasets, we include the type of documents along with the number of tokens in each dataset. Additionally, MUSE incorporates $\mathcal{D}_{\textrm{retain}}^{(\text{reg})}$, a distinct retain set which is seen by $f_{\textrm{target}}$ but not included in $\mathcal{D}_{\textrm{forget}}$. This set is used exclusively with the GDR and KLR regularizers discussed. To ensure that regularized methods do not directly optimize towards the evaluation set $\mathcal{D}_{\textrm{retain}}$ , $\mathcal{D}_{\textrm{retain}}^{(\text{reg})}$ is kept disjoint from $\mathcal{D}_{\textrm{retain}}$.

**Table 7: Examples of MUSE. Each corpus has Verbatim text and Knowledge sets (QA pairs derived from the original text) for evaluating verbatim and knowledge memorization. In News, $\mathcal{D}_{\textrm{forget}}$ and $\mathcal{D}_{\textrm{retain}}$ are two disjoint sets of news articles. In Books, $\mathcal{D}_{\textrm{forget}}$ is the Harry Potter book series while $\mathcal{D}_{\textrm{retain}}$ consists of wiki articles about the series. The sizes of the forget and retain sets are reported in tokens in ().**
| Corpus | Forget Set | Retain Set |
| --- | --- | --- |
|  | News Article (0.8 M tokens) | News Article (1.6 M tokens) |
| News | \pbox<br>6.5cm MP Stuart McDonald has been appointed as the SNP’s |  |
| new treasurer | \pbox<br>5.5cm A father whose 12-year-old son was killed by an IRA bomb 30 years ago |  |
|  | \pbox<br>6.5cm Q: What position has Stuart McDonald MP been appointed to? |  |
| A: The SNP’s new treasurer | \pbox<br>6.5cm Q: Who was affected by the IRA bomb 30 years ago? |  |
| A: A father whose 12-year-old son |  |  |
|  | Harry Potter Books (1.1 M tokens) | Harry Potter FanWiki (0.5 M tokens) |
| Books | \pbox<br>5.5cm ‘‘There’s more in the frying pan,’’ said Aunt Petunia, |  |
| turning eyes on her massive son. | \pbox<br>5.5cm This page contains a list of spells: |  |
| Portuguese for ‘open’. |  |  |
|  | \pbox<br>5.5cm Q: What does Aunt Petunia tell her son? |  |
| A: There’s more in the frying pan. | \pbox<br>5.5cm Q: What is the spell used to open things? |  |
| A: Portuguese |  |  |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Although my questions do not affect my evaluation, I am curious whether the authors can comment on the method in https://arxiv.org/pdf/2410.22086? In your Figure 5, no methods are recognized as "desired", i.e. high utility and low-to-medium memorization, yet NGDiff seems to be close to desired.
soundness: 3
strengths: This paper tackles very important problems with solid efforts to build the benchmark. The six perspectives of the benchmark are impactful and clear. I enjoy reading this work and I am convinced by the experiments, which are sufficiently comprehensive and well-designed.
summary: This paper proposes MUSE as a benchmark that evaluates unlearned models from multiple angles. The authors leverage this benchmark to test 8 popular unlearning algorithms on LLM, exposing the privacy leakage and utility degradation issues in existing unlearning methods.
weaknesses: Overall the weaknesses are not significant. I think the scale of data and number of methods may be further extended, for example, the conclusion of a method's effectiveness may change when forget set gets larger.

Also the hyperparameter tuning may be sub-optimal and require more elaboration.

Figure 4 multiple lines are using the same color which is confusing.
The blue curve in Figure 6 seems completely covered.

Minor: Line 414 GA should be GA_GDR?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: For me, I would like to see a more detailed explanation of the role each metric plays within the framework used, as well as how the experiments impact these metrics across different methods. This section of the discussion could be further refined and deepened to provide clearer insights into the significance of each metric and the implications of the experimental results.
soundness: 3
strengths: 1: This paper provides a comprehensive and detailed study of methods for machine unlearning, conducting an in-depth evaluation from six perspectives. It offers a thorough assessment framework covering aspects such as semantics, continuity, knowledge, memory, and privacy. Compared to previous evaluation frameworks, this approach has a broader scope, assesses from more perspectives, and utilizes a larger dataset, demonstrating the framework's comprehensiveness and effectiveness.

2: This paper provides a detailed introduction to each method of unlearning, explaining the corresponding implementation processes with comprehensive formulas and theoretical explanations.

3: The experiments on different datasets are presented clearly, with comparisons across various categories.

4: What I appreciate most is the authors’ effort in providing evaluation metrics, giving clarification for each method with straightforward assessments. Although some metrics, such as C5 and C6, are not as clearly defined, the authors’ attempts to evaluate unlearning across various dimensions are commendable.
summary: This article introduces a method that provides an evaluation framework MUSE for machine unlearning. This evaluation framework assesses the machine unlearning capability from different perspectives through six types of evaluations. Experiments conducted on a series of 7B parameter models validate the framework, comparing different methods and their effectiveness.
weaknesses: 1: Although the authors provide numerous metrics, many of them heavily rely on previous methods. For example, the C3 metric for privacy assessment has already been addressed in a series of earlier approaches. Additionally, it remains unclear whether certain metrics, such as C5 and C6, are truly essential for evaluating machine unlearning, as their explanations in the paper are not entirely clear.

While I appreciate the advantages noted in Strengths, I would like to see more about how these various methods are integrated within this framework and what specific improvements it offers over prior approaches. Metrics like C1, C2, and C3, for instance, have elements originating from previous work. Although Table 1 highlights the advantages of this framework over the previous TOFU metrics, I believe these advantages are not significant and merely provide comparisons with a few additional metrics.

2: The machine unlearning methods presented in Section 4 of this paper all adopt previous approaches, which makes me wonder if this part could be incorporated into the experimental section instead. It occupies a considerable amount of space, and the methods discussed are largely drawn from prior work. The overall structure of the paper could benefit from further refinement to enhance its clarity and conciseness.

3: Although the paper conducts numerous experiments and evaluates them across various metrics, the results intended to be conveyed by these different metrics are somewhat unclear. For example, if we introduce a set of metrics (C1 to C6) to evaluate different machine unlearning approaches, it raises several questions: Why were these metrics chosen? What conclusions do the experiments under these metrics yield? What consistencies or insights do they offer? The paper lacks sufficient commentary in this regard, and its quality could be enhanced by incorporating more examples and providing clearer justifications for the experiments, thus improving the interpretation of the findings.

---

confidence: 4
contribution: 4
initial_rating: 5
presentation: 3
questions: - I'd like to understand the MIA a bit more. You mention that the AUC of the MIA against f_target is 0.0. If I understand it correctly, this refers to the very classic MIA where you have to distinguish between members (D_train) and non-members (D_holdout), without any unlearning. Hence two questions, 
	- Why do you not adapt the standard notion of MIA AUC where the 1 label corresponds to membership (which would mean the AUC on f_target would be close to 1.0 rather than 0.0). I think this would make it more clear for the reader. 
	- Why does it perform so well? While the initial work of Shi et al. indeed shows great performances for Min-k% Prob, it is by now well known that the great performance was due to a distribution shift rather than measuring memorization and that typical MIAs against LLMs do not work much better than a random guess [1,2,3]. I don't believe your setup has a distribution shift (as the AUC for f_retrain is 0.47), yet I wonder why the attack works near-perfectly while all MIAs against LLMs in proper setups . Could you add experiments that convincingly show why it works so well in your setup compared to others? My intuition would be that it is due to the fact that you use 5 epochs for finetuning. Could you add ablations to your unlearning evaluation for less epochs? This would help understand how your methods and conclusions apply to (more realistic) setups where LLMs memorize less. 

- As all evaluation for metrics C4-6 are based on utility, I wonder if the metric used for utility should be more elaborate. Right now, you only measure the knowledge memorization on the custom created question-answering pairs. But I think it would be meaningful to also include e.g. perplexity on D_retain or even performances on well known benchmarks (this would also help clarify whether the finetuning represents a realistic setup for a useful f_target). Could you add this as well? 
- Clarifications
	- Could you confirm in Sec 3.2 where the holdout set for books come from? Is it a random split of the full books or is it data from the harry potter wiki? 
	- Maybe nit, but I was quite confused by the notion of f_retrain. The first time you mention f_retrain is in Sec 3.1 if I'm not mistaken. Could you formalize what this refers to in Sec 2? Otherwise it's hard to follow. And it might also be worth it to rename either 'retrain' in f_retrain or 'retain' in D_retain, as I found this to be confusing too; 

**I believe the framework the paper proposes is very valuable, so I'm generally in favor of accepting the paper. However I also believe my concerns are valid and clarifying them (they should be very fixable) would improve the work. Hence, i'm sticking with a 5 right now, but happy to change the score when my concerns are properly addressed during the rebuttal.**

[1] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. (2024). Do membership inference attacks work on large language models?. arXiv preprint arXiv:2402.07841.

[2] Meeus, M., Shilov, I.,  Jain, S., Faysse, M., Rei, M., & de Montjoye, Y. A. (2024). SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It). arXiv preprint arXiv:2406.17975.

[3] Das, D., Zhang, J., & Tramèr, F. (2024). Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406.16201.
soundness: 2
strengths: - What the authors propose is very helpful for the community. Plenty of work is focused on developing approximate unlearning methods for LLMs, and evaluation methods employed are all too often ad-hoc rather than comprehensive and rigorous. They setup authors propose is well-thought through and covers all meaningful dimensions (at least that I see). 
- Thorough analysis of unlearning algorithms, for 2 datasets
- I particularly like the realistic setup for utility preservation of Harry Potter.
summary: The paper proposes a systematic evaluation framework called MUSE to assess unlearning algorithms for LLMs. They consider 6 dimensions in their evaluation setup, from both the perspectives of data owners and model deployers. The 6 dimensions include verbatim memorization, knowledge memorization, privacy leakage, utility preservation, scalability wrt size of removal requests and sustainability over sequential unlearning requests. They apply MUSE to evaluate eight representative unlearning algorithms on two datasets, finding that current methods struggle to meet the full set of ideal requirements, especially in preventing privacy leakage.
weaknesses: (more see questions)

- Only evaluating one LLM, in one finetuning regime. 
- No justification for the high MIA performances, which is needed to evaluate how realistic the setup and its conclusions are. 
- Limited utility evaluation. 
- Minor clarifications needed

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In Table 5 of Appendix B.4, the authors report GPU times for each unlearning method, with the baseline retrain method requiring 184,320 hours, which seems unusually high. In the current setup, retraining starts with a base model, and the retrain model is finetuned on this base model for 5 epochs. I understand the authors' rationale that, in practical applications, the forget set is integrated into the entire pretrained corpus, making full-corpus retraining necessary rather than just training on the retain set. However, the authors should clarify this approach more explicitly in the experimental setup and justify why finetuning only on the retain set is a valid simulation of full-corpus retraining.
soundness: 3
strengths: - This paper overall is well-written and easy to follow.
- The paper offers a more comprehensive benchmark to evaluate machine unlearning, and conduct thorough experimental evaluation and analysis on several existing algorithms on all six metrics.
summary: This paper presents Muse, a benchmark designed to evaluate machine unlearning algorithms. The authors construct the evaluation corpus using content from BBC News articles and popular books, such as Harry Potter. They propose six desired metrics to assess machine unlearning performance, covering aspects such as memorization, utility, privacy, and scalability. The authors evaluate eight different unlearning algorithms using this benchmark. Experimental results indicate that, while current methods achieve effective unlearning, they also lead to substantial drops in utility and privacy.
weaknesses: - This work only evaluate four families of machine unlearning methods, which are all training-based. However, training-free approaches, such as those via model editing (e.g. arxiv:2202.05262) or weight pruning (e.g. arxiv:2403.01267), are also relevant and are not covered here. The authors should either incorporate these methods or explain why they were excluded.
- The authors propose six desired properties for machine unlearning, with three reflecting the data owner’s perspective and three for the model deployer. While these criteria are generally reasonable, it remains unclear if they reflect real-world practitioner concerns. If these are indeed practitioner-driven priorities, the authors should provide references or practitioner insights to support why these metrics were chosen over others not included.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to my weaknesses. Also, I have one more question.


1. Figure 2 seems confusing. Why should the data distribution of 'forget' be close to 'holdout'? This is not reflected in the proposed PrivLeak metric.
soundness: 3
strengths: 1. The paper is well-written and easy to follow.

2. Compared to previous works, the paper identifies 6 desirable properties for unlearning and formulates them as evaluation metrics.

3. The paper covers a wide range of existing unlearning methods and conducts comprehensive evaluations with the proposed evaluation metrics.
summary: This paper introduces a comprehensive benchmark to assess LMs' performance after unlearning according to six diverse desirable properties for unlearned models, including verbatim memorization, QA memorization, MIA performance, utility, scalability, and sequential unlearning requests. The paper considers two Corpus including NEWS and BOOKS, to evaluate the eight existing popular unlearning algorithms. Comprehensive experiments are conducted to reveal the effectiveness and limitations of existing methods.
weaknesses: 1. My major concern is that the experimental results are not well explained. For example, Task Vector's performance in Table 3 is somehow neglected. The results of Task Vector seem inconsistent with prior works such as [1]. More celebrations should be included to discuss why task vector fails on unlearning.

2. Though the paper proposes six properties. These properties may have already been discussed in related works. From my perspective, only C6 sustainability is the new criteria for evaluation. C3, which compares MIA scores between unlearned models and retrained models, is not sound for under-unlearning.

3. In terms of C3, which states that 'Unlearning leads to privacy leakage,' under-unlearning (PrivLeak < 0) may not be regarded as privacy leakage. I agree that the perfect unlearning should let PrivLeak close to 0. However, in practice, it is not likely for the adversary to compare unlearned models with retrained models. Instead, the adversary only calculates the MIA for the unlearned models to determine membership. Therefore, under-unlearning should be one feasible outcome.






[1] Liu, Zheyuan, et al. "Towards safer large language models through machine unlearning." arXiv preprint arXiv:2402.10058 (2024).

Meta-review: Authors provide a comprehensive treatment of "evaluating machine unlearning algorithms" and provide 6 evaluation criteria for such algorithms. These criteria are broadly classified into two categories pertinent to the data owners' and the deployers' interests. Paper provides an extensive evaluation of known unlearning algorithms using their criteria and uncover non-trivial characteristics of them. It is noted that some of the criteria are separately known in past literature, however this work claims to provide a unified overview through an applied lens.

Decision: Accept (Poster)