MUSE: Machine Unlearning Six-Way Evaluation for Language Models
Authors: Weijia Shi
Jaechan Lee
Yangsibo Huang
Sadhika Malladi
Jieyu Zhao
Ari Holtzman
Daogao Liu
Luke Zettlemoyer
Noah A. Smith
Chiyuan Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Language Models
Machine Unlearning
Primary area: datasets and benchmarks

Abstract: Language models (LMs) are trained on vast amounts of text data, which may include private and copyrighted content. Data owners may request the removal of their data from a trained model due to privacy or copyright concerns. However, exactly unlearning only these datapoints (i.e., retraining with the data removed) is intractable in modern-day models. This has led to the development of many approximate unlearning algorithms. The evaluation of the efficacy of these algorithms has traditionally been narrow in scope, failing to precisely quantify the success and practicality of the algorithm from the perspectives of both the model deployers and the data owners. We address this issue by proposing MUSE, a comprehensive machine unlearning evaluation benchmark that enumerates six diverse desirable properties for unlearned models: (1) no verbatim memorization, (2) no knowledge memorization, (3) no privacy leakage, (4) utility preservation on data not intended for removal, (5) scalability with respect to the size of removal requests, and (6) sustainability over sequential unlearning requests. Using these criteria, we benchmark how effectively eight popular unlearning algorithms on 7B-parameter LMs can unlearn Harry Potter books and news articles. Our results demonstrate that most algorithms can prevent verbatim memorization and knowledge memorization to varying degrees, but only one algorithm does not lead to severe privacy leakage. Furthermore, existing algorithms fail to meet deployer's expectations because they often degrade general model utility and also cannot sustainably accommodate successive unlearning requests or large-scale content removal. Our findings identify key issues with the practicality of existing unlearning algorithms on language models.

Published as a conference paper at ICLR 2025
W MUSE: MACHINE UNLEARNING SIX-WAY EVALU-
ATION FOR LANGUAGE MODELS
Weijia Shi Jaechan Lee"!
Yangsibo Huang"
Sadhika Malladì
Jieyu Zhao²
Ari Holtzman' Daogao Liu'
Luke Zettlemoyer
Noah A. Smith
Chiyuan Zhang*
Uniiersitty of Washington Princeton University
3 University of Southern California https://muse-bii gi io
*University of Chicago
Google Research
ABSTRACT
Language models (LMs) are trained on vast amounts of text data, which may include private and copyrighted content, and data owners may request the removal of their data from a trained model due to privacy or copyright concerns. However, exactly unlearning only these datapoints (i.e., retraining with the data removed) is intractable in modern-day models, leading to the development of many approx- imate unlearning algorithms. Evaluation of the efficacy of these algorithms has traditionally been narrow in scope, failing to precisely quantify the success and practicality the algorithm from the perspectives of both the model deployers and the data owners. We address this issue by proposing MUSE, comprchensive machine unlearning evaluation benchmark that enumerates six diverse desirable properties for unlearned models: (1) verbatim memorization, (2) knowledge memorization, (3) no privacy leakage, (4) utility preservation on data not intended for removal, (5) scalability with respect the size of removal requests, and (6) sustainability over sequential unlearning requests. Using these criteria, we bench- mark how effectively eight popular unlearning algorithms 7B- 11-parrmeter LMS can unlearn Harry Potter books and news articles. Our results demonstrate that most algorithms can prevent verbatim memorization and knowledge memorization to varying degrees, but only one algorithm does not lead to severe privacy leakage. Furthermore, existing algorithms fail to meet deployer's expectations, because they often degrade general model utility and also cannot sustainably accommodate successive unlearning requests or large-scale content removal. Our findings iden- tify key issues with the practicality of existing unlearning algorithms on language models, and we release our benchmark to facilitate further evaluations.
INTRODUCTION
Training language models (LMs) often involves using vast amounts of text data, which may inad- vertently contain private and copyrighted content (Carlini et al., 2021; Henderson et al., 2023; Min et al., 2023; He et al., 2024). In real-world applications, data owners may demand that their data be removed from trained language model due privacy or copyright concerns, as mandated for example by the General Data Protection Regulation (GDPR, European Parliament & Council of the European Union). Moreover, recent copyright lawsuits (DOE Gitllub, Inc., N.D. Cal. 2022; Tremblay v. OpenAl, Inc., 2023) emphasize the need for removing copyrighted data from the model.
These recent developments have intensified research interest in designing. evaluating, and improving machine unlearning algorithms, which aim to transform an existing trained model into one that behaves as though had never been trained on certain data (Ginart et al., 2019; Liu et al., 2020; Wu et al., 2020; Bourtoule et al., 2021; Izzo et al., 2021; Gupta et al., 2021; Sekhari et al., 2021; Ye et al., 2022b; Ghazi et al., 2023). Exact unlearning in LMs requires removing the undesired data (the forget set) and retraining the model from scratch the remaining data (the retain ser), which is too costly to be practical, especially for frequent unlearning operations. As such, several efficient approximate unlearning algorithms have been proposed (Eldan & Russinovich, 2023; Zhang et al., 2024b), but existing evaluations of LM unlearning on question answering (Eldan & Russinovich, 2023; Maini et al., 2024) cannot provide holistic view of how practical and effective particular

---

Published as a conference paper at ICLR 2025
Harry Potter Chapter
Request: remove from the larget model!
Machine unlearning
"Ihee''s frying part said Aunt Peturis, turning eyes on her massive son.
O
Materials with copyright privacy concerss
Data owner
Target model
Unlearned model
MUSE: Machine Unlearning Six-way Evaluation
Data Expectations
No verbatim memorization
No knowledge momortzation
No privasy teakage
hare's more the frying pan, said Aunt
Q: What does Aund Peturia tell her son?
Attackar should NOT be abie to tell whether
should NOT output
should NOT oulput
has boen used
Paturia, tuming cyss on hor massive son.
A: More the frying pan.
Uliity Preservation
Scalabiitty
Sustairabilit
Deployer Expectations
author Hanry Ponter?
Bnnlearn recues: Jnniernn recues:
Small acaii
should oulput
Rowling
Largo-scalo
Figure I: MUSE evaluation focuses on six key dimensions of machine unlearning, addressing both data owner and deployer expectations. For examplc, when an author (data owner) requests the unlcarning of the Harry Potter books, they may expect the unlearned model to: (1) avoid generating verbatim copies of the text to protect copyright. (2) eliminate retention of factual knowledge from the books, and (3) not reveal whether the books were previously used in training to protect privacy. From the deployer aspect, they may expect unlearning to (4) preserve the model's utility on general tasks, (5) scale effectively to accommodate unlearning of large datasets, and (6) handle sequential unlcarning requests that may arrive over time.
unlearning algorithm is. In this work, we propose systematic, multi-faceted framework called MUSE (Machine Unlearning Six-Way Evaluation; $3) to evaluate six desired properties for un- learning algorithms (Figure 1). Our criteria cover both the data owner's and the model deployer's desiderata for a practical unlearning algorithm. Data owners require the LM to unlearn the precise tokens (verbatim memorization), general knowledge encoded in the tokens (knowledge memorization), and any indication that their data was included in the training set to begin with (privacy leakage). On the other hand, model deployers want to effectively accommodate many successive unlearning requests (sustainability) various sizes of forget sets (scalability) without degrading the general model capabilities (utility preservation).
We apply MUSE to evaluate eight representative machine unlearning algorithms ($4) on two datasets ($3.2). focusing on the specific cases of unlearning Harry Potter books and news articles. Our findings indicate that most unlearning algorithms remove verbatim memorization and knowledge memorization with varying degrees of efficacy but operate at the cost of utility preservation and do not effectively prevent privacy leakage ($5.2). In particular, negative preference optimization (NPO; Zhang et al., 2024b) and task vectors (Ilharco et al., 2023) are especially effective in removing these types of memorization, but we find that NPO often permits privacy leakage and both methods induce sharp drop in the utility of the model. Furthermore, testing their scalability and sustainability reveals that they both algorithms struggle with large forget sets and successive unlearning requests ($5.3).
Our results highlight that unlearning algorithms generally fail to meet data owner expectations in preventing privacy leakage, which one of the primary motivations for unlearning. Additionally, they struggle to meet all three of the aforementioned deployer expectations. Therefore, although it is increasingly desirable to find an efficient and effective unlearning algorithm amid rising concerns around privacy regulations and copyright litigations, our evaluation suggests that currently feasible real-world scenarios.
unlearning methods are not yet ready for meaningful usage or deployment These findings underscore the pressing need for further research in this area. We also release benchmark to facilitate further evaluations and welcome extensions to other modalitics.
MACHINE UNLFARNING: PRELIMINARIES AND NOTATIONS
Machine unlearning (Ginart al., 2019; Liu al., 2020; Izzo et al., 2021; Sekhari et al., 2021; Gupta et al., 2021; Ye et al., 2022b; Liu et al., 2024) has emerged as an important capability to accommodate data removal requirements that arise from scenarios with privacy or copyright concerns.
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with previous benchmark: Unlike the previous benchmark TOFU (Maini et al., 2024). which evaluates unlearning on synthetic Q&A datasets, MUSE tackles real-world unlearning challenges: unlearning real-world large-scale corpus (22x larger) while taking into account six desiderata that are important to both data owners and deployers. More related works are discussed in Appendix 6.
MUSE (ours)
TOFU (Maini et al., 2024)
CI. No verhatim memorization
C2. No knowledge memorization
Evaluation
C3. No privacy leakage
criteria
C4. Utility preservation
C5. Scalability
C6. Sustainability
Domains
NEWS and BOOKS Verbatim text and knowledge set (Q & A) 0.8M for NEWS, 3.3M for BOOKS
Synthetic autobiographies Q & A 0.15M
Evaluation
Data Constitution
corpora
Scale (# tokens in forget set)
We briefly describe the machine unlearning setting. Consider a dataset Durin and model furget trained on Dusin- Suppose we design an algorithm 21 to unlearn a specific subset (i.e., the forger set) Disrget C irrain from Jurget-- We want to preserve performance on a retain set Dretain = Drrain Drarget= and we also evaluate the model on an in-distribution but disjoint hold-out set Dholdom which the model has never been trained on. So, the unlearning algorithm l/ takes Starget: Diorgot and, optionally, Deetain and outputs an unlearned model funkeam. Exact unlearning ensures funteam is behaviorally identical to the model resulting from retraining from scratch, denoted furgot. but such retraining is usually too costly in real world deployment, so we focus on evaluating approximate unlearning algorithms.
3 THE MUSE EVALUATION BENCHMARK
MUSE evaluates a comprehensive set of desirable properties of machine unlearning across six facets. We detail the evaluation metries $3.1 and describe the evaluation corpus in $3.2.
3.1 EVALUATION METRICS
Ideally, an unlearned model should behave as if it had never seen the forget set, exhibiting similar behavior to a retrained model on any corpus such that mu]unleamm D) As m[[Jretminn D), where m represents any evaluation metric. Prior evaluations on LM unlearning focus on performance of specific tasks like question answering (c.g., Eldan & Russinovich, 2023; Maini et al., 2024). However, these metrics do not faithfully reflect data owner expectations and wolldwrlld deployment considerations when performing unlearning. To address this, we propose comprehensive evaluation metrics that consider both data owner and deployer expectations. A comparison between MUSE and the prior benchmark is shown in Table 3.
Data owner expectations. When removing forget set from model, data owners typically have three main expectations regarding the unlearned model: (C1) No verbatim memorization: The model should not exactly replicate any details from the forget set. (C2) No knowledge memorization: The model should be incapable of responding to questions about the forget set. (C3) No privacy leakage: It should be impossible example, if a patient's 's records are unlearned from a medical diagnosis model, in addition to verbatim and knowledge memorization checks, is also important that the patient's privacy is preserved we follow established practice in quantifying privacy using the membership inference test, which detects a specific datapoint was used to train the model (member), distinguishing it from non-training data (non-member) (Shokri et al., 2017). In this case of unlearning record from a diagnostic model, it is undesirable for the model to leak membership information, because it would be used to associate the patient with the disease. We quantify these data owner expectations with three evaluation metrics:
detect that the model was ever trained on the forget set. For
C1. No verbatim memorization When model has unlearned medical record, it should not output its contents verbatim. We quantify the verbatim memorization VerbMem by prompting the model with the first tokens from sequence Ti: E Diorget and comparing the continuation outputted by the model f to the true continuation 2p+11:] E Dionget using the ROUGE-L F1 score (Lin, 2004).
VerbMem(f,
ROUGE( mp411)
(Drorger] # Dieps
3

---

Published as a conference paper at ICLR 2025
Privacy
Privacy Leakage
Leakage
Freeuaro
retain forget
holdout
ratain holdlout forget
retain
forget holdout
holdout
forget
NIA
metro
metre
(A) Before Unlearning
Perfect Unlearning
Under Unlearning
Unlearning
Figure 2: Distribution the MIA metric (see C3) for Diseets Dooloous and Deevann Differences metric between forget and holdout sets indicate various unlearning outcomes of Doeeetss potentially leaking privacy. A perfectly unlearned model (b) should show similar MIA metrics distribution for Dirget and Dhotoon Unlcarning methods may fail under-unlearning Dioget. making similar Dramin (c), over-unlearning it, causing divergence from Drobkur (d).
C2. No knowledge memorization When model has unlcarned medical record, it should no longer be able to answer questions about that record. We measure model f's memorization of knowledge from the forget Piorget as follows: for each example Diorget associated with question-answer pair (g, a). gather the model's answer to the question q. denoted f()). We then average the ROUGE scores for all question-anssee ansser pairs Diarge compute the knowledge memorization score KnowMem:
KnowMem(J, Droget)
Σ
ROUGE(F(g) a)
Dioege
(grn)E
C3. No privacy leakage As discussed previously, is desirable that the unlearned model does leak membership information indicating that Dioret was part of Drnin To determine if given example was used during training, membership inference attack (MIA) exploits distributional differences in certain statistics (e.g.. loss) between training (member) and non-training (non- member) data: if the loss the example is low, then it was likely used for training. Using MIAs to evaluate unlearning processes a well-established practice as shown by prior research (Hayes et al., 2024; Triantafillou et al., 2023). An effective unlearning algorithm should climinate such influence to reduce the attack's success rate. As shown in Figure 2, unlearning typically increases the loss on the example, but there are two possible ways that unlearning can fail to prevent privacy leakage: (1) under-unlearninng, when the loss is not made large enough; and (2) over-unleurning, when the loss is made abnormally large. To accurately measure the privacy leakage, we employ Min-K Prob (Shi et al., 2024a) a state-of-the-eee MIA method for LMs based on the loss, and compute the standard AUC-ROC score (Murakonda et al., 2021; Ye et al., 2022a) discriminating Diorget (members) and Dooldou (non- members). By comparing the AUC score with that of the retrained model, we define"
PrivLeak AUC(fuukam: Diorgas Daoldma) AUC fremii: Diurgety Dholdous)
AUC(farin; Drogett Duoldon)
The PrivLeak metric for good unlearning algorithm should be close to zero, whereas an wver/under-unleamingg algorithm will get a large positive/negatiie metric. More details about privacy leakage discussed in Appendix B.I.
Deployer expectations. Model deployers have their own considerations for using unlearning algo- rithms the real world. Unlearning specific datapoints can unpredictably degrade model capabilities in ways that are difficult to recover. Moreover, deployers expected to effectively accommodate somewhat large-scale forget sets and successive unlearning requests from data owners. As such, consider three key metrics: (C4) utility preservation on the retain set, (C5) scalability to han- dle large-scale content removal, and (C6) sustainability to maintain performance over sequential unlearning requests.
C4. Utility preservation. Model capabilities are often hard-won through expensive training proce- dures, so deployers would want an unlearning algorithm that preserves performance on the retain
EExamples of question-anss answer pairs derived from the original corpus can be found in Table 7.
An MIA algorithm compares its score given threshold to classify given datapoint member non-memher. The AUC-ROC single value that summarizes the overall performance of the MIA algorithm by measuring its ability to discriminate between members and non-members across possible thresholds.
GGnnrally, AUC(fremant D io Dooloou) ) &o 0.5, though sometimes there are intrinsic distribution shifts hetween D Proeget and Dookio that may bias the haseline away from 0.5.
4

---

Published as a conference paper at ICLR 2025
Table 2: Examples of MUSE. Each corpus has Verbatin text and Knowledge sets (QA pairs derived from the original text) for evaluating verbatim and knowledge memorization. NEWS. Piorgt and Pretinn are two disjoint sets of news articles. In BoOks, Diorgt is the Harry Potter book series while Preui consists of wiki articles about the series. The sizes of the forget and retain sets are reported in tokens in O). Note that only the verbatin texts within the Forget Set are included in our training data, while all Knowledge sets (QA pairs) serve for evaluations.
Corpus
Forget Set
Retain Set
NEWS ARTICLE (0.8 M tokens)
NEWS ARTICLE (1.6 M tokens)
MP Stuart McDonaldd has been appointed as the SNP's
A father whose 12-year-old son was killed by IRA banb 36 years ago
treasurer
NEWS
What position Stuart McDonald MP been appointed to?
affected by the bomb 30 years ago?
A: The SNP's new treasurer
whose 12-yaar-old son
HARRY POTTER BOOKS (1.1 M tokens)
HARRY POTTER FANWIKI (0.5 M tokens)
"There's is more in the frying pan, said Aunt Petunia,
This page contains a list of spells:
turning eyes her nossive son.
Portuguese for "apen'. .
BookS
Q: What does Aunt Petunia tell her son?
What the spell used open things?
There's the frying pan.
Portugaese
set. To quantify this, we evaluate the unlearned model's performance on the retain set using the knowledge memorization metric KnowMem (Julearmm Dretain).
CS. Scalability. We assess the scalability of unlearning methods by examining their performance on forget sets of varying sizes. Let Di denote forget set of size c, and fú be the corresponding unlearned model. For any data owner-valued metric such as utility preservation, we measure scalability by analyzing the trend of this metric as increases from small large values.
C6. Sustainability. Machine unlearning operations often need to be applied sequentially, as data removal requests may arrive different times.* We denote the unlearned model after processing the k-th request as fa,k- To measure sustainability, we analyze the trend of any data owner-valued metric as the number of sequential unlearning requests k increases.
3.2 EVALUATION CORPUS
MUSE considers two representative types of textual data that may frequently involve unlearning requests: news articles (Tremblay v. OpenAl, Inc., 2023) and books (Eldan & Russinovich, 2023).
These datasets are detailed as follows:
NEWS consists of BBC news articles (Li et al., 2023b) collected after August 2023. All articles are randomly divided into (disjoint) forget, retain, and holdout sets.
BOOKS consists of the Harry Potter book series. To simulate a real-world setting for testing utility preservation (C4), we include different types of materials in the forget and retain sets. The forget set contains the original books, while the retain set contains related content from the Harry Potter Fan Wiki, representing domaín knowledge that should be retained after unlearning.
For each corpus, we construct: 1) Verbatim text: the original text to assess the unlearning methods to remove verbatim memorization (C1), and 2) Knowledge set: a set of derived (question, answer) pairs based on the original texts to evaluate the unlearning method's effectiveness in purging learned knowledge and preventing knowledge memorization (C2). To create the Knowledge set, we partition the Verbatim text into excerpts and GPT-4 (OpenAI, 2023) generate (question, answer) pairs for each excerpt. When constructing the dataset, we perform deduplication between Diorger and D Dretain by removing documents with over 70% similarity based 3-grams. For more details about the dataset generation pipeline, see Appendix D.
Table 7 provides examples from the news and books corpora. The details of the dataset splits and dataset sizes are provided in Appendix D.
4 UNLEARNING METHODS
We evaluate eight efficient approximate unlearning methods belonging to four families of algorithms.
*For example, under GDPR, if Alice requests the removal of her data and Bob submits another removal request 31 days later, both requests must be fulfilled within 30 days. This requires the model deployer to first. unlcamn Alice's data and then process Bob's request on the updated model. harrypotter. fandom. com/wiki
5

---

Published as a conference paper at ICLR 2025
Four families of unlearning methods. We first introduce four families of unlearning methods, which serve as the basis for the eight methods we evaluate.
Gradient Ascent (GA) minimizes the likelihood of correct predictions on Dingee by performing gradient ascent on the cross-entropy loss (the opposite of conventional learning with gradient descent). GA has achieved mixed results: while Jang al. (2023) found it effective for unlearn- ing examples from the Enron email dataset (Klimt & Yang. 2004) with minimal performance degradation, Ilharco et al. (2023) reported that GA sigmificantly harms general model utility when unlearning a high-toxicity subset of the Civil Comments dataset (Borkan et al., 2019).
Negative Preference Optimization (NPO: Zhang et al., 2024b) treats the forget set negative preference data and adapts the offline DPO objective (Rafailov et al., 2023) to tune the model to assign low likelihood to the forget set without straying too far from the original model farggtt
2 BE
log a (-3log [
ENPO B Brat
Fumpes (T)
where fo refers to the model that undergoes unlearning, is the sigmoid function, and B is hyperparameter that controls the allowed divergence of fo from its initialization Suuget. Following Rafailov al. (2023); Zhang et al. (2024b), fix 0.1 in our experiments.
Task Vectors (Ilharco et al., 2023) derived from straightforward arithmetic on the model weights can effectively steer neural network behavior. We adapt task vectors perform unlearning in two stages. First, we train furget on Dioree until the model overfits, yielding reinforced model freinforee. We then obtain a task vector related to Diorgtt by calculating the weight difference between furget and freintonce. To achieve unlearning, we subtract this task vector from fuaget weights, intuitively moving the model away from the direction used to adapt to D Dinget i.e.,
Junleam funget (Jreinfonce (urget)
Who's Harry Potter (WHP: Eldan Russinovich, 2023) defines the unlearned model funteam as
the interpolation between the target model farger and the reinforced model freinterce- Let p (-]x) denote the token distribution parametrized by the model f when given a prompt x as input. Then, concretely, for any input æ, WHP samples the next token from
futtam (-1zz) furge (-1xr) ffp (-12)) Pfrge (-1x))
where a is a hyperparameter that controls the interpolation between the two models.
Two regularizers for utility preservation. GA and NPO are not explicitly designed for utility discuss several regularization strategies that either improve the performance the retain set or ensure the unlearned model remains close to the target model during unlearning.
preservation,
Gradient Descent on the Retain Set (GDR; Liu et al., 2022; Maini et al., 2024; Zhang et al., 2024b) augments the unlearning objective with a standard gradient descent learning objective the cross-entropy of the retain set Dretii more directly train the model maintain its performance on Dretain-
KL Divergence Minimization on the Retain Set (KLR; Maini et al., 2024; Zhang et al., 2024b) encourages the unlearned model's probability distribution Pm (-1z) to be close to the target model's 's distribution P/anp (-1z) on inputs from the retain set x Dretain-
List of methods. We combine GA and NPO with the two regularizers GDR and KLR, which yields four new combinations. Hence, we end up with total of 8 candidate unlearning methods: GA, GAGDR- GAKLR- NPO, NPOGDR. NPOKUR- Task Vector, and WHP. general, the cost of the approximate unlearning method is negligible compared retraining. Note that the methods with regularizers (GAcDR- GARLR- NPOGDR. NPOKLR) require access to the distribution of Drecain). Details about the efficiency of these methods are reported in Appendix B.4.
EXPERIMENTS
We evaluate the eight representative unlearning methods using the experimental setup described in I. We present the results for data owner expectations in $5.2 and for deployer expectations $5.3.
'These regularizers are not compatible with Task Vector and WHP, because Task Vector involves purposefully overfitting model to D Disge when deriving the task vector, and WHP is test-time technique where the unlcarning operation involves no optimization by itself.

---

Published as a conference paper at ICLR 2025
Table 3: Most unlearning methods effectively remove verbatim and knowledge memorization but signifi- cantly impact utility and privacy. We evaluate algorithms described in $4 on the criteria MUSE. We include the results of fremiin for reference and calculate the relative ratio compared to the reference model. We highlight the ratio in blue if the unlearning algorithm satisfies the critcrion and highlight it in orange otherwise. We define privacy leakage negligible when it falls within the range of -5% to +5%. Large positive values suggest uver-unlannnng, while large negative values suggest under-unlearning (sce $3.1). This table the results for CI to C4, while results CS and C6 are shown Figure 6.
No Verbatim Mem. VerbMem on Drege (4)
Knowledge Mem.
No Privacy Leak.
Utildly Preserv, KnowMern on Prons (1)
KnowMer on Prape (J)
PrivLeak (E (-5%,,%%)
NEWS
Target furget
58.4
63.9
99.8
55.2
Retruin fremuia
20.8
33.1
0.0
55.0
GA
0.0
1100%
0.0
1100%
5.2
over-unlearn
0.0
1100%
GAGDE
4.9
476.5%
31.0
(6.3%)
108.1
over-unlearn
27.3
150.3%
GAILR
27.4
131.4%
50.2
(51.55
-96.1
under-unlearm
44.8
118.5%
NPO
0.0
L1005
0.0
1100%
24.4
over-unlearn
0.0
1100.0%
NPOGDR
1.2
194.4%]
54.6
164.8%
105.8
over unlear
40.5
126.3%
NPOKER
26.9
129,0%
49.0
148.
-95.8
45.4
417.4%
Task Vector
57.2
66.2
100.0%
-99.8
under-unlearn 55.8
WHP
19.7
21.2
(35.9%%
109.6
under-unlearn
28.3
448.5%
BOOKS
Target Surpst
99.8
59.4
-57.5
66.9
Retrain Fictmio
14.3
28.9
0.0
74.5
GA
0.0
4100%
0.0
L100%
25.00
under-unlearn
0.0
4100%
GAGDR
0.0
4 1100%
0.0
1100%
-26.5
under-unlearn
10.7
485.65
GÄvER
16.0
111.4%
21.9
(24.4%
-40.2
under-unlearn
37.2
450.0%
NPO
0.0
1100%
0.0
(10006
-2133
under unleam
0.0
4100%
NPO
0.0
1100%
0.0
L11055
-30.8
under unleam
22.8
469.4%
NPO
17.0
118.2%
25.0
(13.44%
-43.5
under-unlearn
44.6
4.40.1%
Task Vector
99.7
1395.0%
52.4
181.2%
-57.5
under unleam
641.7
113.1%
WHP
18.0
125.2%
55.7
(19..9%
56.5
over-unlearn
63.6
414.6%
5.1 EXPERIMENTAL SETUP
Retrained and target models. We start with general pretrained base model fo. and finetune two models: furget on Dringet U Drerain. and fretrain on Dreaann only. See Appendix B.3 for details about finetuning. For each unlearning algorithm U, we further generate the unlearned model funkem U( [fugget Prrgett D Dretain). We ensure that fo has no access to 2 Drorget Pretain Doeldou Therefore, for NEWS, we use fo LLaMA-2 7B (Touvron et al., 2023), which was released before the BBC news articles we use to construct our benchmarks; and for Books, we use fo ICLM-7B (Shi et al., 2024b), which does not contain the Harry Potter books in its pretraining data.
Unlearning experimental configuration. Following prior work (Maini et al., 2024), we run GA, NPO, and their regularized variants using the AdamW optimizer (Loshchilov Hutter, 2017) with a constant learning rate of 10-5 and a batch size of 32. We employ the stopping criteria as follows: if the utility (i.c., KnowMem on Pretain) of model undergoing unlearning drops below that of fietrain within 10 epochs of unlearning, we stop at the first epoch where this condition holds; otherwise, take a checkpoint from the 10th epoch. For Task Vector and WHP, to obtain the reinforced model for unlearning, fine-tune the target model for 10 epochs using the same learning rate and batch size. Further details on the model fine-tuning and unlearning can be found in Appendix B.3.
5.2 RESULTS: DATA OWNER EXPECTATIONS
We first analyze how eight unlearning methods meet data owner expectations (C1, C2 & C3 in $3.1).
C1&C2. Most methods are effective for unlearning memorization. As shown in Table 3, most unlearning methods perform exceptionally well in [CI. No verbatim memorization] and [C2. No knowledge memorization). often reducing VerbMem and KnowMem even beyond the levels achieved by the retrained model. Notably, some methods, such as GA and NPO, achieve a score of for both VerbMem and KnowMemn meaning that these methods completely prevent the unlearned models from producing any text related to the forget set. However, as we will see later, these reductions often come at the cost of significant utility loss on the retain set.
7

---

Published as a conference paper at ICLR 2025
Target Model (
Retrained Model (naviin))
GAVER (under-unlearn)
GAGOR (over-unlkarn)
Forget
Forget
Forgrt
Forget
Rotain
Retain
Retain
Retain
Hordaut
Hoidout
Haidout
Hoidaut
Min-K% Prob
Prob
Prob
Min-K%
Figure 3: Distribution of Min-KK% % Prob, an MIA metrie, for Diorgets Dholduus and Pretain- Consistent with the expected pattern in Figure 2, futriin shows perfect unlearning, with the overlapping distributions for Dioges and Existing approximate unlearning methods typically either under-unlearn over-unlearn. For example, GAKUR shows slight under-unlearning. while GAGoR over-unleamss, pushing the Min-K% Prob of Dirga an extreme level.
Duoioon
1.0
Hetrais
NPO (C.59
Utility Preservation (Detai)
0.6
Desi
Retrain
Targe: (0.00)
NPCco (3.98)
GA
NPOOue (0.02)
Target
True pesitive rate
GAuzs (9 99)
WHI (1 00)
0.4
GA
0.6
GARL (0.52)
TV (0.00)
free lunch
GAGDR
GAKLR
0.2
NPO
NPOGok
0.2
Utility loss
NPOKLR
0.0
WHP
0.0
0.2
0.4 0.6
0.8 1.0
0.0
0.2
0.4
0.6
TV
False positive
Knowledge Memorization (Purga)
Figure 4: ROC curves Proggt vs. Peesdmn on NEWS using Min-K Prob, with AUC scores in parentheses. AUC=00.5 (i.c., fretrin) means no sig- nificant distribution difference between two sets (i.e., no membership leakage). Most unlcarning meth- ods show under-unlearn (AUC*E0.5) or over-unlearn (AUC O.5..
Figure 5: Utility preservation knowledge mem- orization on BBC. freuai maintains high utility on Deeain while showing low knowledge memorization on Diager. GA and NPO without regularizers show signifi- cant utility loss, collapsing the origin. Every other unlearning method unlearns the knowledge on Diory at the cost utility.
C3. Unlearning leads to privacy leakage. Most unlearming methods reveal the membership of Drorget in Duain through unlearninn (PrivLeak < 0) or over-unlearning (PrivLeak >> 0), as shown in Table 3. We further examine the effectiveness of membership inference by plotting ROC curves in Figure The deviation from the diagonal line indicates the attacker's s advantage over random guessing. We observe that the Min-K% Prob based attack achieves AUC z O on Jurget- confirming its effectiveness. Meanwhile, the ROC curve for fretrain closely follows the diagonal line (AUC = 0.47), suggesting that perfect unlearning ensures MIA no more effective than random guessing. Among the approximate unlearning methods, GA and NPOCDR without regularizers consistently over-unlearn (AUC > 0.7), whereas KLg-regularized methods (NPOkLR and GÁRLR) tend to under-unlearn and barely improve privacy leakage over /uaget- WHP also deviates from the diagonal significantly.
In Figure 3, we further visualize the distribution of Min-K% Prob, the MIA metric computed across Diorget, Dretain: and Dpoldoun The behavior of fuaget and frerain mirrors the patterns sketched Figure 2, where Drorget and Dretain are distinguishable
furget but overlap in fremrin- Existing approximate unlearning methods typically either under-unlearn or over-unlearn. For example, GAKUR does not sufficiently increase the Min-K% indicating under-unlearning On the other hand, NPOGDR over-unledn ver-unlearns, significantly raising the MIA metric across all datasets and especially for D Diorget-
% Prob metric for Piorget to align with the distribution of Pholdous
5.3
RESULTS: DEPLOYMENT CONSIDERATION ATTONS
C4. Unlearning significantly degrades model utility. Table [C4 Utility Preserv.] shows that all unlearning methods compromise the model's utility by 24.2% ~ 100%. Notably, several methods (GA, GAGOR. NPOçbR) lead to complete utility loss, rendering the unlearned models practically unusable. Figure illustrates the trade-offs between utility preservation on Dretain and knowledge memorization Dioeges- An ideal unlearned model should mimic the behavior of freasin (desired region) by achieving a low level memorization on D Piorget while maintaining its utility. However, most methods, such GAKLR, NPOkLR. and WHP, unlearn the knowledge on Drr at the cost of utility.
8

---

Published as a conference paper at ICLR 2025
C5. Unlearning methods scale poorly with forget set sizes. To evaluate the robustness of the unlearning methods to larger forget sets, we collect additional news articles from the same distribution to scale our NEWS corpus from 0.8M tokens to 3.3M tokens and observe the utility preservation at four different forget set sizes. As shown in Figure 6 (a), the model utility decrease with the size of the forget set and achieves a minimum at the largest size.
GA
GAaE
NPOue
C6. Unlearning methods cannot sustainably accommodate sequential unlearning requests. To evaluate the robustness of these unlearning methods more than one unlearning requests, we sequentially apply k unlearning processes, each with respect to a different forget set. To simulate sequential unlearning, we partition the
GAaER
NPO
NPOKuR
Usility Preservation
0.4
0.2
0.0
extended NEWS forget set (comprised of 3.3M
0.0M
0.8M
1.7M 2.5M
3.3M
tokens) into four disjoint folds (each containing 0.8M tokens) and apply the unlearning methods to each fold in sequential manner.
Forget Set Size
Unlearning Request
Scalability (a)
Sustainability (b)
Figure 6: The performance of GA, NPO, and their reg- ularized variants, measured by utility preservation, degrades with larger forget set sizes (a) and sequen- tial unlearning requests (b).
We again select utility preservation as the target metric for comparison. As shown in Figure 6 (b), the performance of an unlearned model tends to decrease significantly with respect to the number of unlearning requests, indicating that current unlearming methods are not yet ready to handle sequential unlearning in a sustainable manner.
6 RELATED WORK
Machine unlearning for non-language model applications. Machine unlearning is a long-running, well-studied topic. Several studies have explored exact unlearning, aiming to make the unlearned model (funkam) exactly identical to the reference model (fictrain) As expected, this can only be accomplished in simple models like SVMs (Cauwenberghs & Poggio. 2000; Tveit et al., 2003; Romero et al., 2007; Karasuyama & Takeuchi, 2010) naive Bayes models (Cao & Yang, 2015). Another approach is to ensure that the unlearned model funteum is probabilistically indistinguishable from fretrain (Ginart al., 2019; Guo et al., 2020), and this view of certifiable unlearning is closely related to differential privacy (Dwork et al., 2006b;a). This rigorous definition of unlearning has inspired several theoretical works that characterize the feasibility of unlearning in convex and non-convex models, but those proposed algorithms are too computationally costly to operate on modern-day LMs (Izzo et al., 2021: Neel et al., 2021: Ullah et al., 2021: Sekhari et al., 2021; Gupta et al., 2021). Several more tractable unlearning algorithms have been proposed (Borkan et al., 2019; Ginart et al., 2019; Thudi et al., 2022; Chourasia & Shah, 2023) with broader applications such as image classification (Ginart et al., 2019; Golatkar et al., 2020a), eext-to-mmae generation (Gandikota et al., 2023; Zhang et al., 2023; Fan et al., 2023), Federated Learning (Liu et al., 2020; Che et al., 2023; Halimi et al., 2022; Huang et al., 2022) and Recommender Systems (Li et al., 2024b).
Machine unlearning for language models: methods and applications. Machine unlearning has recently found its way into language model applications. In $4, we discuss some standard unlearning methods based on parameter optimization, like the Gradient Ascent and its variance. Other notable non-training-base unlearning methods include Decalizationinformee unlearning (Meng et al., 2022; Wu et al., 2023; Wei et al., 2024a), which involves identifying model units (e.g., layers, neurons) closely related to the unlearning data or tasks and then locally editing and modifying the units. In-context unlearning (Pawelczyk et al., 2023) offers another approach, treating the model as a black box and modifying its output results using external knowledge.
Machine unlearning has also been applied to various downstream language model tasks, though the unit of machine unlearning may differ from what we study in this work. Our evaluation focuses on unlcarning specific examples or datasets, aiming to make LMs forget either the phrasing or the content knowledge of targeted data, while preserving their utility for data not targeted for removal. This is crucial for ensuring privacy and copyright compliance. In addition to this specific unlearning, there's 's also a broader application similar to model editing, where outdated information is replaced with new knowledge (Pawelczyk et al., 2023; Yu et al., 2023; Belrose et al., 2024). Moreover, efforts have been made to eliminate harmful behaviors in language models by creating toxicity benchmarks and enhancing safety measures (Lu et al., 2022; Yao et al., 2023; Li et al., 2024a; Zhang et al., 2024b).

---

Published as a conference paper at ICLR 2025
Despite these varied approaches to unlearning at different operational and knowledge levels, the evaluation principles we propose such as preserving utility, ensuring scalability, and maintaining sustainabilityy are relevant across these contexts.
Machine unlearning for language models: evaluation. Evaluating machine unlearning methods for language model applications is also critical. Most previous studies have focused this evaluation on specific tasks such as question answering or sentence completion. For example, Eldan & Russinovich (2023) experiment with unlearning to forget Harry Potter books and demonstrate the effectiveness of their methods by showing that familiarity scores, measured through completion-based token- evaluations, significantly decline et al. (2024) further suggest comparing unlearned models with perfectly retrained models. Their evaluation finds that while familiarity scores with the forget set may drop post-unlearning, they still remain higher than those of the retrained model. Wei et al. (2024b) evaluate the feasibility of using unlearning techniques to prevent language models from generating copyrighted content. The closest work to ours is TOFU (Maini et al., 2024), benchmark featuring 200 synthetic author profiles, each with 20 question-answer pairs, divided into forget and retain sets. However, TOFU is relatively small-scale (0.15M tokens) and focuses the evaluation of question answering. Additionally, current evaluations focus on limited aspects of data owner expectations and do not adequately reflect real-world deployment considerations, such scalability and potential sequential unlearning requests. In contrast, MUSE formally defines different unlearning scopes and corresponding metrics, resulting in a systematic six-way evaluation featuring both data owners' and deployers' expectations. The evaluation uses a large-scale corpus of over 6 million tokens, separated into verbatim text and knowledge sets. We also note that some of our findings align with previous evaluations. For example, our observation that over- or under-unlearn can exacerbate privacy leakage ($5.2) is consistent with the recent work by Hayes et al. (2024). Our findings align with the the concurrent study by Shumailoy et al. (2024) showing that unlearning gives a false sense of security.
probability- -baee,, and question-answen
Lynch
Survey papers. We direct readers to several insightful survey papers for further reading. For non-LLM applications, notable surveys include Shintre et al. (2019); Nguyen et al. (2022); Thudi ct al. (2022); Xu et al. (2023). Additionally, the NeurIPS 2023 machine unlearning competition for image classification is valuable source of empirical methods tailored for this specific application (Triantafillou et al., 2023). For language model applications, Si et al. (2023) categorize unlearning methods into different families and summarize datasets for evaluating unlearning. Liu et al. (2024) review LM unlearning algorithms by targets and methods, discuss the effectiveness and efficiency of existing approaches and emphasize the importance of clearly defining the unlearning scope.
CONCLUSION
In this work, we propose MUSE, a comprehensive machine unlearning evaluation benchmark that highlights six desirable properties from the perspectives of both data owners and model deployers. We find that current unlearning methods successfully prevent the model's memorization of content at a significant cost to utility on data not intended for removal. They also lead to severe privacy leakage and cannot sustainably accommodate successive unlearning requests or large-scale content removal. These findings highlight the need for future research into more robust unlearning methods.
Limitations. While MUSE provides a systematic benchmark for evaluating unlearning algorithms, it does not consider all possible considerations. For example, data owners may have additional expectations, such as ensuring their information cannot be probed from intermediate activations (Song & Raghunathan, 2020) or receiving formal guarantees of unlearning success (Sekhari et al., 2021; Gupta et al., 2021; Ghazi et al., 2023). Similarly, deployers may expect other capabilities, like fine-tuning and in-context learning, to be preserved, and may prefer unlearning algorithms that are both computationally efficient and storage-wise cheap (e.g. does not need to keep copy of the retain set). MUSE currently evaluates unlearning for language models using books and news articles, but it could be extended to other corpora, such as medical notes (Johnson et al., 2016; 2020) and emails (Klimt & Yang, 2004), which often involve privacy concerns (Li ct al., 2023a: Huang et al., 2023). We also plan to evaluate different-sized LMs in the future. Finally, our approach can be generalized to construct multi-faceted benchmarks for multimodal models (Golatkar et al., 2020b; Cheng& & Amiri, 2023; Zhang et al., 2024c). Further discussion on broader impact are in Appendix A.
hntpps:/ummeerrinnn-ctlll thub. io
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We are committed making all aspects of our work fully open-source, providing comprehensive instructions guarantee reproducibility.
Models The weights for our original models, Starget and funlear,, will be released under the Apache 2.0 open-source license.
Data
Our benchmark datasets will be made available under open-source licenses.
Code We will provide the code for all baseline methods, evaluation scripts used for benchmarking. well the code for visualizations and analysis presented in this paper. Detailed instructions will accompany our code to ensure precise reproducibility.

---

Published as a conference paper at ICLR 2025
REFERENCES
Nora Belrose, David Shaneider-Josephi Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36, 2024.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification, 2019.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Chou Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEFE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463 480. IEEE, 2015.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633 2250, 2021.
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. Advances in neural information processing systems, 13, 2000.
Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan, Dejing Dou, and Jun Huan. Fast federated machine unlearning with nonlinear functional theory. International conference on machine learning, pp. 4241 4268. PMLR, 2023.
Jiali Cheng and Hadi Amiri Multimodal machine unlearning. 2023.
Rishav Chourasia and Neil Shah. Forget unlearning: Towards true data-deletion in machine learning. In International Conference on Machine Learning, pp. 6028-6073. PMLR, 2023.
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology- Cryptolog-EERROCRYYP EUROCRYPT 2006: 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28-June 1, 2006. Proceedings 25, pp. 486-503. Springer, 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-284, Springer, 2006b.
Ronen Eldan and Mark Russinovich. Who's Harry Potter? Approximate Unlearming in LLMs. arXiv preprint arXiv:2310.022553 2023.
DOE v. GilHub, Inc. 4:22-cv-06823, N.D. Cal. 2022.
Tremblay V. OpenAl, Inc.. 23-cv-03416-AMGG (N.D. Cal.), 2023.
European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council. URL https: htaps://daatt ottp://aa..cuuuuuuuuee///rr//2//
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Em- powering machine unlearning via gradient- based weight saliency in both image classification and generation. arXiv preprint arXiv:23 10. 12508, 2023.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto Naato-Kaufmmn. and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2426-2436, October 2023.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Ayush Sekhari, and Chiyuan Zhang. Ticketed wwweannngnnleaiinggggg schemes. In The Thirty Sixth Annual Conference on Learning Theory, pp. 511--1139 PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304 9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision-ECC'S 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 383-398. Springer, 2020b.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning, pp. 3832-3842.
PMLR, 2020.
Varun Gupta, Christopher Jung. Seth Neel, Aaron Roth, Saeed Sharifi SharifiMMalvjeeddi, and Chris Waites. Adaptive machine unlearning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (cds.), Advances in Neural Information Processing Systems, volume 34, pp. 16319-16330. Curran Associates, Inc., 2021. URL https::/pproceeddnggg neurips. cc/paper- pdd.
Anisa Halimi, Swanand Kadhe, Ambrish Rawat, and Nathalie Baracaldo. Federated unlearning: How to efficiently erase a client in fl? arXiv preprint arXiv:2207.05521, 2022.
Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. arXiv preprint arXiv:2403.012/88 2024.
Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, and Peter Henderson. Fantastic copyrighted beasts and how (not) to generate them. arXiv preprint arXiv:2406. 10022406..426., 2024.
Peter Henderson, Xuechen Li, Dan Jurafsky. Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. arXiv preprint arXiv:2303.15715 2023.
Yangsibo Huang, Chun-Yin Huang, Xiaoxiao Li, and Kai Li. A dataset auditing method for collabora- tively trained machine learning models. IEEF Transactions Medical Imaging, 22(7):2081-209)) 2022.
Yangsibo Huang, Samyak Gupta, Zexuan Zhong, Kai Li, and Danqi Chen. Privacy implications of retrieval-based language models. arXiv preprint urXiv:2305. 14888, 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri. and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008-2016. PMLR, 2021.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14389-14408, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 11..1865/11/222.. acl-Iong.805. URL https::t httpss///cclaathooo/oooo org/ wwtg//22...22 acl acl-long.80 805.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://hyssonne. org/conten/miiiicci/..ffccc August 23, 202/), 1) pp. 49-55, 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
13

---

Published as a conference paper at ICLR 2025
Masayuki Karasuyama and Ichiro Takeuchi. Multiple incremental decremental learning of support vector machines. IEEE Transactions on Neural Networks, 2 2010.
Bryan Klimt and Yiming Yang. The enron corpus: A new dataset for email classification research. In European conference on muchine learning, pp. 217-226. Springer, 2004.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.051/1 000977 2023a.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint DxXiv:2403.03216 2024a.
Yucheng Li, Frank Guerin, and Chenghua Lin. Avoiding data contamination in language model evaluation: Dynamic test construction with latest materials, 2023b.
Yuyuan Li, Chaochao Chen, Xiaolin Zheng, Junlin Liu, and Jun Wang. Making recommender systems forget: Learning and unlearning for erasable recommendation. Knowledge-Based Systems, 283: 111 111124, 2024b.
Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pp. 74 81, 2004.
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning, 2022.
Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu. Federated unlearning. arXiv preprint 20X1v:20/2./889,, 2020.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao. Hang Li, Kush R Varshney, al. Rethinking machine unlcarning for large language models. arXiv preprint arXiv:2402.08787. 2024.
lyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17 2017.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27610 2022.
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in 1lms. arXiv preprint 2.K.::2022.18833 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton. and J. Zico Kolter. Tofu: A task of fictitious unlearning for 1lms. ArXiv, abs/2401.061211, 2024. URL https://api. semanticschol wwt.ccchoooaa..rrrorrosttt. wTgp/CorpuSDD:2693337..
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372 2022.
Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in a nonparametric datastore. arXiv preprint arXiv:2308.. 04444,, 2023.
Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligence and Statistics, pp. 2287-2295. PMLR, 2021.
Seth Neel, Aaron Roth, and Saeed Sharifi-Malvaeeddi Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pp. 931-962, PMLR, 2021.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin. and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299 2022.
14

---

Published as a conference paper at ICLR 2025
Alex Oesterling, Jiaqi Ma, Flavio Calmon, and Himabindu Lakkaraju. Fair machine unlearning: Data removal while mitigating disparities. In International Conference on Artificial Intelligence and Statistics, pp. 3736-3744, PMLR, 2024.
OpenAl. Gpt-4 technical report, 2023.
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
Enrique Romero, Ignacio Barrio, and Lluís Belanche. Incremental and decremental learning for linear support vector machines. In International Conference on Artificial Neural Networks, pp. 209-218. Springer. 2007.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances Neural Information Processing Systems, 34: 1:4:18075-1888 18066, 2021.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openrent net// forum?id=zZWqrMMDD
Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representations, 2024b. URL /ttpss///penrevee
new. ttntt/oorundddLXswwWnme
Saurabh Shintre, Kevin A Roundy, and Jasjeet Dhaliwal. Making machine learning forget. Privacy Technologies and Policy: 7th Annual Privacy Forum, APF 2019, Rome, Italy, June 13-14, 2019, Proceediiggy 7, pp. 72-83. Springer, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint ar arXiv:2407.00106, 2024.
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for llms: Tasks, methods, and challenges. arXiv preprint arXiv:231..15766 2023.
Congzheng Song and Ananth Raghunathan, Information leakage embedding models. In Proceed- ings of the 2020 ACM SIGSAC conference on computer and communications security, pp. 377-390, 2020.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium Security and Privacy (EuroS&P), pp. 303-319. IEEE, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
15

---

Published as a conference paper at ICLR 2025
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kur- manji, Gintare Karolina Dziugaite, Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade. Neurips 2023 machine unlearning challenge, 2023.
URL https:tp/////...m //aes/////gggg /kaggle. com/ http://cottttoss/////ee
pss 2-----0222a---- mach-me-chha-aaaeee wewhhinne--neernnnng
Amund Tveit, Magnus Lie Hetland, and Häavard Engum. Incremental and decremental proximal support vector classification using decay coefficients. In International Conference on Data Warehousing and Knowledge Discovery, pp. 422-429. Springer, 2003.
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 4126-4143 PMLR, 2021.
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.0s 162, 2024a.
Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. Evaluating copyright takedown methods for language models. arXiv preprint arXiv:2406. 18664, 2024b.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138, 2023.
Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning, pp. 10355-10366, PMLR, 2020.
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Yu Philip. Machine unlearning: A survey. ACM Computing Surveys, 2023.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683 2023.
Jayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 3093-3106, 2022a.
Jingwen Ye, Yifang Fu, Jie Song, Xingyi Yang, Songhua Liu, Xin Jin, Mingli Song, and Xinchao Wang. Learning with recoverable forgetting. In European Conference on Computer Vision, pp. 87-103. Springer, 2022b.
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 6032-6048, 2023.
Dawen Zhang. Shidong Pan, Thong Hoang, Zhenchang Xing. Mark Staples, Xiwei Xu, Lina Yao, Qinghua Lu, and Liming Zhu. To be forgotten to be fair: Unveiling fairness implications of machine unlcarning methods. Al and Ethics, pp. 1-11, 2024a.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprini arXiv:333..17591, 2023.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning, 2024b.
Yihua Zhang, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Xiaoming Liu, and Sijia Liu. Unlearncavvas: A stylized image dataset to benchmark machine unlcarning for diffusion models. arXiv preprint 200. 1846, 2024c.
16

---

Published as a conference paper at ICLR 2025
Appendices
A Broader Impact
18
Experimental Details
19
B.1 Threat Model for Privacy Leakage
19
B.2 Compute Configurations
19
B.3 Experimental Setup
19
B.4 Efficiency of Unlearning Methods
19
C More Experimental Results
20
C.I Confidence Intervals for CI, C2 and C4 in Table
20
Dataset Details
21
D.I GPT-generated QA pairs
21
D.2 Dataset segmentation
21
17

---

Published as a conference paper at ICLR 2025
A BROADER IMPACT
As LMs are deployed broadly and publicly, there is mounting legal and social pressure on deployers to release models that permit effective unlearning when requested by data owners (European Parliament & Council the European Union; DOE v. GitHub, Inc., N.D. Cal. 2022; Tremblay v. OpenAI, Inc.,, 2023). These incentives have prompted flurry of new unlearning algorithms stemming from different technical perspectives. As such, systematic evaluation of the strengths and weaknesses of these methods when executing realistic unlearning requests on popular models is essential. MUSE disentangles several desirable properties of unlearning algorithms and finds that no existing algorithm able to satisfy all of the data owner and deployer considerations. We hope that our fine-graine mne-grained, multi-faceter framework facilitates the improvement of unlearning algorithms, Moreover, we expect that the general approach of designing metries to balance the considerations of various stakeholders is flexible and can adapt to the rapidly shifting legal, social, and economic landscape.
We also acknowledge the potential negative impacts of our study. One limitation of our evaluation benchmark is that we do not have comprehensive study of how unlearning would impact the model performance for different user bases, especially underrepresented groups. However, we note proper handling and evaluation of fairness issues in unlearning is still active ongoing research area (Zhang et al., 2024a; Oesterling et al., 2024), therefore we leave it as future work. Additionally, our work may be misinterpreted towards skepticism regarding the broader use of machine unlearning, current evaluation reveals that existing unlearning methods are not yet ready for effective real-world deployment. However, machine unlearning, especially for large language models, is a young and active research area and new algorithms are constantly being proposed. We emphasize that our results not criticism of the paradigm of machine unlearning, but study of the potential downsides of existing methods and a call for better algorithms. We believe our benchmark is an important step towards guiding future algorithm design of machine unlearning research towards more realistic deployment scenarios.
18

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
B.1 THREAT MODEL FOR PRIVACY LEAKAGE
We provide further clarification on the threat model considered for our C3: no privacy leakage. We assume attacker with access to trained model aims to determine whether specific example (belonging to a particular data owner) was part of the training set. Prior work on membership inference attacks (MIA) demonstrates that these attacks the trained model, and use that to distinguish between training and non- non-training samples. Therefore, an effective unlearning algorithm should eliminate such influence reduce the attack's 's success rate, making the model unable to distinguish between a true non-training example and one that was trained and subsequently unlearned.
detect training sample's influence
Note that in this threat model, we assume the attacker only have access to the final unlearned model, because both the target model and the unlearned models are available the same time, then there is no point to perform unlearning.
B.2 COMPUTE CONFIGURATIONS
All experiments are conducted on NVIDIA A40 GPU cards single node.
B.3 EXPERIMENTAL SETUP
Finetuning details. As described in $5.1, for NEWS, start from fo LLaMA-2 7B (Touvron et al., 2023) and finetune the model on the BBC news articles for 5 epochs with a constant learning rate of 10-5 and batch size of 32. For Books, start from fo ICLM 7B (Touvron al., 2023) and finetune the model on the Harry Potter books with same set of hyperparameters.
Unlearning details. For all the unlearning methods in Table 3, we use constant learning rate of 10 and a batch size of 32. For freinforeed used in WHP and Task Vector, we fine-tune furget for 10 epochs.
Before evaluation, for each unlearning method, we select its optimal epoch or (both of which are parameters that control a degree of unlearning) by using our unlearning stopping criteria based on the unlearned model's utility on Dretain compared to that of fremmin The chosen epochs or a's ''s for each method are listed below.
Table 4: Optimal epochs or a's for each unlearning method.
Unlearning Method
NEwS
Books
GA
epoch 1
epoch
GAGDR
epoch 7
epoch
GAKUR
epoch 10
epoch
NPO
epoch I
epoch
NPOGoR
epoch10
epoch
NPOKuR
epach 10
epoch
Task Vector
2
a = =2
WHP
2
2
B.4 EFFICIENCY OF UNLEARNING METHODS
We report the efficiency of unlearning methods in Table 5, measured by the wall-clock time for single gradient update step of unlearning. The time measurements were conducted using NVIDIA A40 GPUs on single node, with batch size of 32 and an input length of 2048 tokens. Each step corresponds to one gradient update processing a total of 65,536 tokens (32 x 2048 tokens). For Task Vector and WHP, each step represents one iteration of fine-tuning create the reinforced model.
19

---

Published as a conference paper at ICLR 2025
Table 5: Wall-clock time and total GPU hours required for each unlearning method.
Unlearning Method
Time (Seconds/Step)
Total Time (GPU Hours)
Retrain
184320
GA
4.14
0,56
GAGDR
6.05
0.82
GAKLR
7.58
1.03
NPO
5.68
0.77
NPOGDR
7.59
1,03
NPOKı
9.11
1.24
Task Vector
4.14
1.12
4.14
1.12
C MORE EXPERIMENTAL RESULTS
C.I. CONFIDENCE INTERVALS FOR CI, C2 AND C4 IN TABLE
We compute confidence intervals for CI, C2, and C4 (Mean ROUGE-L FI) using bootstrapping" For each mean ROUGE-L score reported in Table 3, we draw 9,999 bootstrap resamples and calculate two-tailed 95% confidence interval using the "percentage" method.
Table 95% confidence intervals computed for mean Rouge-L scores used in CI, C2, and C4.
Verhatim
Knowledge
Utiltty
VerbMem on Drogee (1)
KnowMem on Drgget (4)
Know/en on Drotin (f)
NEWS
Target furga
58.4
[54.1, 62.9]
63.9
[58.7, 69.0]
55.2
[50.7, 59.9]
Retrain fretrain
20.8
[18.5, 23.7]
33.1
[26.8, 39.5]
55.0
[50.3, 59.8]
GA
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
GAGOR
4.9
5.2)
31.0
[24.2, 38.0]
27.3
[21.9, 33.0]
GAKLR
[25.1, 29.9]
50.2
[43.1, 56.9]
44.8
[39.2, 50.5]
NPO
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
NPOGDR
1.2
[0.3, 2.3]
54.6
[47.5, 61.5]
40.5
[34.7, 46.2]
NPOKLR
26.9
[24.7, 29.3]
49.0
[41.8, 61.5]
45.4
[39.8, 51.1]
Task Vector
57.2
[52.6, 62.0]
66.2
[61.3, 71.2]
55.8
[51.0, 60.6]
WHP
19.7
[17.8, 21.6]
21.2
[16.0, 26.7]
28.3
[23.3, 33.4]
BookS
Target Furget
99.8
199.8, 99.9]
59.4
[52.7, 66.0]
66.9
[59.6, 73.8]
Retrain fretrain
14.3
[13.6, 15.1]
28.9
[22.1, 35.71
74.5
[68.4, 80.0]
GA
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
GAGDR
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
10.7
[6.2, 15.7]
GAKLR
16.0
[14.8, 17.2]
21.9
[16.4, 27.7]
37.2
[29.5, 45.0]
NPO
[0.0, 0.0]
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
NPOGDR
0.0
[0.0, 0.0]
0.0
[0.0, 0.0]
22.8
[16.1. 30.1]
NPOKLR
17.0
[15.7, 18.2]
25.0
19.00,3.55
44.6
[36.5, 52.8]
Task Vector
99.7
[99.6, 99.81
52.4
[45.0, 59.71
64.7
[57.1, 71.8]
WHP
18.0
[16.4, 19.7]
55.7
[48.6, 62.8]
63.6
[56.3. 70.9]
"https://docs http:///dd.cc.oooooocs///////eee
20

---

Published as a conference paper at ICLR 2025
D DATASET DETAILS
D.1 GPT-GENERATED QA PAIRS
We begin the generation by partitioning the Verbatim text each corpus into a set of 2048-token excerpts using LLaMA-2's tokenizer. For each QA pair generate, randomly sample excerpt from this set and prompt GPT-4 33pt-40-2024-05-- "question" (a question that can only be answered using specific information from the excerpt) and "answer" (an answer to the "question" extracted verbatim from the excerpt). We validate and exclude any pairs whose answers cannot be found verbatim in their corresponding excerpts. This verbatim requirement ensures that our Knowledge set is used precisely to evaluate the model's 's ability to correctly associate questions with relevant portions of the training data.
to create a JSON object with two fields:
For each QA pair to generate, we initiate a new conversation with GPT-4 with its corresponding excerpt. The instruction begins with a system prompt that specifies the desired format of generated QA pairs as follows:
System Prompt for Generating QAs with GPT-4
You will be provided with an excerpt of text. Your goal is to create a question-anse answer pair that assesses reading comprchension and memorization, ensuring that the question can only be answered using details from excerpt.
Please submit your response in a JSON format with the following fields:
"question": A single question related the excerpt. The question should be specific enough that it does allow for answer other than the one you provide. particular, it should not be answerable based on common knowledge alone. Also, few words extracted from excerpt must suffice in answering this question.
"answer": A precise answer extracted verbatim, character-by-ccaracterr from the excerpt. The answer to this question must be short, phrase-level at most. The length of the extraction should be minimal, providing the smallest span of the excerpt that completely and efficiently answers the question.
We then present the excerpt as a user prompt to the model and collect the generated QA pairs. Here are two example generated QA pairs from the Knowledge set of NEWS:
QA Pair Generated GPT-4: Example #1
Excerpt (User prompt):.... ...According to the Stockholm International Peace Research Institute (SIPRI), the US accounted for 69% of Israel's arms imports between 2019 and 2023... Question: According to the Stockholm International Peace Research Institute (SIPRT), what percentage Israel's arms imports between 2019 2023 came from the US?
Answer: 69%
QA Pair Generated by GPT-4 Example #2
Excerpt (User prompt): ... ....eeneeddy''s 's event will be moderated by tech entrepreneur David Sacks, a close ally of the Tesla founder and supporter of Mr DeSantis.. Question: Who will moderate Wednesday's Twitter Spaces event featuring Mr DeSantis?
Answer: tech entrepreneur David Sacks
D.2 DATASET SEGMENTATION
Table 7 shows examples from MUSE and Table 8 presents detailed statistics for MUSE, For both the NEWS and BoOkS datasets, include the type of documents along with the number of tokens each dataset. Additionally, MUSE incorporates D ((re)) reluin distinct retain set which is seen by furget
but not included in Dioget. This set is used exclusively with the GDR and KLR regularizers discussed.
To ensure that regularized methods do not directly optimize towards the evaluation set Dretain D (reg) kept disjoint from Pestain-
21

---

Published as a conference paper at ICLR 2025
Table 7: Examples of MUSE. Each corpus has Verbatin text and Knowledge (QA pairs derived from the original text) evaluating verbatim and knowledge memorization. NEWS. Diergt and Pretain are two disjoint sets news articles. BOOKs, Dioge the Harry Potter book series while Dreuin consists wiki articles about the series. The sizes of the forget and retain sets are reported in tokens in O). Note that only the Verbatin texts within the Forget Set are included in our training data, while all Knowledge sets (QA pairs) serve for evaluations.
Corpus
Forget Set
Retain Set
NEWS ARTICLE (0.8 M tokens)
NEWS ARTICLE (1.6 M tokens)
MP Stuart Mollonald has been appointed as the SNP's
A father whose 12-year-old son was killed by
IRA barb 36 ago
NEWS
What pasition has Stuart MeDionald MP heen appainted to?
Who affected by the IRA homb years
SNP's treasurer
12-year-old son
HARRY POTTER BOOKS (1.1 M tokens)
HARRY POTTER FANWIKT (0.5 M tokens)
"There's 's more in the frying pan, said Aunt Petunia,
This page contains list of spells:
turning eyes her massive son.
Portuguese for "apen"
BOOKS
Q: What does Aunt Petunia tell her son?
What the spell used open things?
There's the frying pan.
Portuguese
Table 8: Statistics MUSE dataset. Corpus sizes are reported in tokens, shown O) Retain Seiny disjoint from the standard Retain Set used in evaluation and is employed in unlearning training to preserve utility through regularizers.
Corpus Furgel Set
Relain Sel
Relain Seley,
Holdout Set
NEWS News Articles (3.3M)
News Articles (1.6M)
News Articles (1.6M)
News Articles (2.0M)
Books Harry Potter Books (1 (M)
Harry Potter Fan Wiki (0.5M)
Harry Potter FanWiki (0.2M)
Harry Potter Books (0.6M)
22


Reviews:
[{"review_id": "qUZ8w0g3R4", "reviewer": "Reviewer_f19H", "summary": "This paper proposes MUSE as a benchmark that evaluates unlearned models from multiple angles. The authors leverage this benchmark to test 8 popular unlearning algorithms on LLM, exposing the privacy leakage and utility degradation issues in existing unlearning methods.", "questions": "Although my questions do not affect my evaluation, I am curious whether the authors can comment on the method in https://arxiv.org/pdf/2410.22086? In your Figure 5, no methods are recognized as \"desired\", i.e. high utility and low-to-medium memorization, yet NGDiff seems to be close to desired.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper tackles very important problems with solid efforts to build the benchmark. The six perspectives of the benchmark are impactful and clear. I enjoy reading this work and I am convinced by the experiments, which are sufficiently comprehensive and well-designed.", "weaknesses": "Overall the weaknesses are not significant. I think the scale of data and number of methods may be further extended, for example, the conclusion of a method's effectiveness may change when forget set gets larger.\n\nAlso the hyperparameter tuning may be sub-optimal and require more elaboration.\n\nFigure 4 multiple lines are using the same color which is confusing.\nThe blue curve in Figure 6 seems completely covered.\n\nMinor: Line 414 GA should be GA_GDR?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ixomMQSnED", "reviewer": "Reviewer_Kopk", "summary": "This article introduces a method that provides an evaluation framework MUSE for machine unlearning. This evaluation framework assesses the machine unlearning capability from different perspectives through six types of evaluations. Experiments conducted on a series of 7B parameter models validate the framework, comparing different methods and their effectiveness.", "questions": "For me, I would like to see a more detailed explanation of the role each metric plays within the framework used, as well as how the experiments impact these metrics across different methods. This section of the discussion could be further refined and deepened to provide clearer insights into the significance of each metric and the implications of the experimental results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1: This paper provides a comprehensive and detailed study of methods for machine unlearning, conducting an in-depth evaluation from six perspectives. It offers a thorough assessment framework covering aspects such as semantics, continuity, knowledge, memory, and privacy. Compared to previous evaluation frameworks, this approach has a broader scope, assesses from more perspectives, and utilizes a larger dataset, demonstrating the framework's comprehensiveness and effectiveness.\n\n2: This paper provides a detailed introduction to each method of unlearning, explaining the corresponding implementation processes with comprehensive formulas and theoretical explanations.\n\n3: The experiments on different datasets are presented clearly, with comparisons across various categories.\n\n4: What I appreciate most is the authors’ effort in providing evaluation metrics, giving clarification for each method with straightforward assessments. Although some metrics, such as C5 and C6, are not as clearly defined, the authors’ attempts to evaluate unlearning across various dimensions are commendable.", "weaknesses": "1: Although the authors provide numerous metrics, many of them heavily rely on previous methods. For example, the C3 metric for privacy assessment has already been addressed in a series of earlier approaches. Additionally, it remains unclear whether certain metrics, such as C5 and C6, are truly essential for evaluating machine unlearning, as their explanations in the paper are not entirely clear.\n\nWhile I appreciate the advantages noted in Strengths, I would like to see more about how these various methods are integrated within this framework and what specific improvements it offers over prior approaches. Metrics like C1, C2, and C3, for instance, have elements originating from previous work. Although Table 1 highlights the advantages of this framework over the previous TOFU metrics, I believe these advantages are not significant and merely provide comparisons with a few additional metrics.\n\n2: The machine unlearning methods presented in Section 4 of this paper all adopt previous approaches, which makes me wonder if this part could be incorporated into the experimental section instead. It occupies a considerable amount of space, and the methods discussed are largely drawn from prior work. The overall structure of the paper could benefit from further refinement to enhance its clarity and conciseness.\n\n3: Although the paper conducts numerous experiments and evaluates them across various metrics, the results intended to be conveyed by these different metrics are somewhat unclear. For example, if we introduce a set of metrics (C1 to C6) to evaluate different machine unlearning approaches, it raises several questions: Why were these metrics chosen? What conclusions do the experiments under these metrics yield? What consistencies or insights do they offer? The paper lacks sufficient commentary in this regard, and its quality could be enhanced by incorporating more examples and providing clearer justifications for the experiments, thus improving the interpretation of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zD1YfOw7AZ", "reviewer": "Reviewer_hTCm", "summary": "The paper proposes a systematic evaluation framework called MUSE to assess unlearning algorithms for LLMs. They consider 6 dimensions in their evaluation setup, from both the perspectives of data owners and model deployers. The 6 dimensions include verbatim memorization, knowledge memorization, privacy leakage, utility preservation, scalability wrt size of removal requests and sustainability over sequential unlearning requests. They apply MUSE to evaluate eight representative unlearning algorithms on two datasets, finding that current methods struggle to meet the full set of ideal requirements, especially in preventing privacy leakage.", "questions": "- I'd like to understand the MIA a bit more. You mention that the AUC of the MIA against f_target is 0.0. If I understand it correctly, this refers to the very classic MIA where you have to distinguish between members (D_train) and non-members (D_holdout), without any unlearning. Hence two questions, \n\t- Why do you not adapt the standard notion of MIA AUC where the 1 label corresponds to membership (which would mean the AUC on f_target would be close to 1.0 rather than 0.0). I think this would make it more clear for the reader. \n\t- Why does it perform so well? While the initial work of Shi et al. indeed shows great performances for Min-k% Prob, it is by now well known that the great performance was due to a distribution shift rather than measuring memorization and that typical MIAs against LLMs do not work much better than a random guess [1,2,3]. I don't believe your setup has a distribution shift (as the AUC for f_retrain is 0.47), yet I wonder why the attack works near-perfectly while all MIAs against LLMs in proper setups . Could you add experiments that convincingly show why it works so well in your setup compared to others? My intuition would be that it is due to the fact that you use 5 epochs for finetuning. Could you add ablations to your unlearning evaluation for less epochs? This would help understand how your methods and conclusions apply to (more realistic) setups where LLMs memorize less. \n\n- As all evaluation for metrics C4-6 are based on utility, I wonder if the metric used for utility should be more elaborate. Right now, you only measure the knowledge memorization on the custom created question-answering pairs. But I think it would be meaningful to also include e.g. perplexity on D_retain or even performances on well known benchmarks (this would also help clarify whether the finetuning represents a realistic setup for a useful f_target). Could you add this as well? \n- Clarifications\n\t- Could you confirm in Sec 3.2 where the holdout set for books come from? Is it a random split of the full books or is it data from the harry potter wiki? \n\t- Maybe nit, but I was quite confused by the notion of f_retrain. The first time you mention f_retrain is in Sec 3.1 if I'm not mistaken. Could you formalize what this refers to in Sec 2? Otherwise it's hard to follow. And it might also be worth it to rename either 'retrain' in f_retrain or 'retain' in D_retain, as I found this to be confusing too; \n\n**I believe the framework the paper proposes is very valuable, so I'm generally in favor of accepting the paper. However I also believe my concerns are valid and clarifying them (they should be very fixable) would improve the work. Hence, i'm sticking with a 5 right now, but happy to change the score when my concerns are properly addressed during the rebuttal.**\n\n[1] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. (2024). Do membership inference attacks work on large language models?. arXiv preprint arXiv:2402.07841.\n\n[2] Meeus, M., Shilov, I.,  Jain, S., Faysse, M., Rei, M., & de Montjoye, Y. A. (2024). SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It). arXiv preprint arXiv:2406.17975.\n\n[3] Das, D., Zhang, J., & Tramèr, F. (2024). Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406.16201.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- What the authors propose is very helpful for the community. Plenty of work is focused on developing approximate unlearning methods for LLMs, and evaluation methods employed are all too often ad-hoc rather than comprehensive and rigorous. They setup authors propose is well-thought through and covers all meaningful dimensions (at least that I see). \n- Thorough analysis of unlearning algorithms, for 2 datasets\n- I particularly like the realistic setup for utility preservation of Harry Potter.", "weaknesses": "(more see questions)\n\n- Only evaluating one LLM, in one finetuning regime. \n- No justification for the high MIA performances, which is needed to evaluate how realistic the setup and its conclusions are. \n- Limited utility evaluation. \n- Minor clarifications needed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dSZOMTTA3C", "reviewer": "Reviewer_BEBK", "summary": "This paper presents Muse, a benchmark designed to evaluate machine unlearning algorithms. The authors construct the evaluation corpus using content from BBC News articles and popular books, such as Harry Potter. They propose six desired metrics to assess machine unlearning performance, covering aspects such as memorization, utility, privacy, and scalability. The authors evaluate eight different unlearning algorithms using this benchmark. Experimental results indicate that, while current methods achieve effective unlearning, they also lead to substantial drops in utility and privacy.", "questions": "In Table 5 of Appendix B.4, the authors report GPU times for each unlearning method, with the baseline retrain method requiring 184,320 hours, which seems unusually high. In the current setup, retraining starts with a base model, and the retrain model is finetuned on this base model for 5 epochs. I understand the authors' rationale that, in practical applications, the forget set is integrated into the entire pretrained corpus, making full-corpus retraining necessary rather than just training on the retain set. However, the authors should clarify this approach more explicitly in the experimental setup and justify why finetuning only on the retain set is a valid simulation of full-corpus retraining.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper overall is well-written and easy to follow.\n- The paper offers a more comprehensive benchmark to evaluate machine unlearning, and conduct thorough experimental evaluation and analysis on several existing algorithms on all six metrics.", "weaknesses": "- This work only evaluate four families of machine unlearning methods, which are all training-based. However, training-free approaches, such as those via model editing (e.g. arxiv:2202.05262) or weight pruning (e.g. arxiv:2403.01267), are also relevant and are not covered here. The authors should either incorporate these methods or explain why they were excluded.\n- The authors propose six desired properties for machine unlearning, with three reflecting the data owner’s perspective and three for the model deployer. While these criteria are generally reasonable, it remains unclear if they reflect real-world practitioner concerns. If these are indeed practitioner-driven priorities, the authors should provide references or practitioner insights to support why these metrics were chosen over others not included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YGMoT3ath3", "reviewer": "Reviewer_pArk", "summary": "This paper introduces a comprehensive benchmark to assess LMs' performance after unlearning according to six diverse desirable properties for unlearned models, including verbatim memorization, QA memorization, MIA performance, utility, scalability, and sequential unlearning requests. The paper considers two Corpus including NEWS and BOOKS, to evaluate the eight existing popular unlearning algorithms. Comprehensive experiments are conducted to reveal the effectiveness and limitations of existing methods.", "questions": "Please refer to my weaknesses. Also, I have one more question.\n\n\n1. Figure 2 seems confusing. Why should the data distribution of 'forget' be close to 'holdout'? This is not reflected in the proposed PrivLeak metric.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. Compared to previous works, the paper identifies 6 desirable properties for unlearning and formulates them as evaluation metrics.\n\n3. The paper covers a wide range of existing unlearning methods and conducts comprehensive evaluations with the proposed evaluation metrics.", "weaknesses": "1. My major concern is that the experimental results are not well explained. For example, Task Vector's performance in Table 3 is somehow neglected. The results of Task Vector seem inconsistent with prior works such as [1]. More celebrations should be included to discuss why task vector fails on unlearning.\n\n2. Though the paper proposes six properties. These properties may have already been discussed in related works. From my perspective, only C6 sustainability is the new criteria for evaluation. C3, which compares MIA scores between unlearned models and retrained models, is not sound for under-unlearning.\n\n3. In terms of C3, which states that 'Unlearning leads to privacy leakage,' under-unlearning (PrivLeak < 0) may not be regarded as privacy leakage. I agree that the perfect unlearning should let PrivLeak close to 0. However, in practice, it is not likely for the adversary to compare unlearned models with retrained models. Instead, the adversary only calculates the MIA for the unlearned models to determine membership. Therefore, under-unlearning should be one feasible outcome.\n\n\n\n\n\n\n[1] Liu, Zheyuan, et al. \"Towards safer large language models through machine unlearning.\" arXiv preprint arXiv:2402.10058 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)