A Closer Look at Machine Unlearning for Large Language Models
Authors: Xiaojian Yuan
Tianyu Pang
Chao Du
Kejiang Chen
Weiming Zhang
Min Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Machine Unlearning
Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We discuss issues of existing machine unlearning methods and propose techniques for improving them for large language models.

Abstract: Large language models (LLMs) may memorize sensitive or copyrighted content, raising privacy and legal concerns. Due to the high cost of retraining from scratch, researchers attempt to employ machine unlearning to remove specific content from LLMs while preserving the overall performance. In this paper, we discuss several issues in machine unlearning for LLMs and provide our insights on possible approaches. To address the issue of inadequate evaluation of model outputs after unlearning, we introduce three additional metrics to evaluate token diversity, sentence semantics, and factual correctness. We then categorize unlearning methods into untargeted and targeted, and discuss their issues respectively. Specifically, the behavior that untargeted unlearning attempts to approximate is unpredictable and may involve hallucinations, and existing regularization is insufficient for targeted unlearning. To alleviate these issues, we propose using the objective of maximizing entropy (ME) for untargeted unlearning and incorporate answer preservation (AP) loss as regularization for targeted unlearning. Experimental results across three scenarios, i.e., fictitious unlearning, continual unlearning, and real-world unlearning, demonstrate the effectiveness of our approaches. The code is available at https://github.com/sail-sg/closer-look-LLM-unlearning.

Published as a conference paper at ICLR 2025
A CLOSER LOOK AT MACHINE UNLEARNING FOR LARGE LANGUAGE MODELS
Xiaojian Yuan"l, Tianyu Pang Chao Du², Kejiang Chen Weiming Zhang', Min Lin? University of Science and Technology of China 2Sa Al Lab, Singapore (yuanxj, tianyupang, duchao, linmin]esea. com; {chenki, zhangwngJ@ustc.edu.c
ABSTRACT
Large language models (LLMs) may memorize sensitive or copyrighted content, raising privacy and legal concerns. Due the high cost of retraining from scratch, researchers attempt to employ machine unlearning to remove specific content from LLMs while preserving the overall performance. In this paper, we discuss several issues in machine unlearning for LLMS and provide our insights possible ap- proaches. To address the issue of inadequate evaluation of model outputs after unlearning, we introduce three additional metrics evaluate token diversity, sen- tence semantics, and factual correctness. We then categorize unlearning methods into untargeted and targeted, and discuss their issues respectively. Specifically, the behavior that untargeted unlearning attempts to approximate is unpredictable and may involve hallucinations, and existing regularization is insufficient for targeted unlearning. To alleviate these issues, we propose using the objective of maximiz- ing entropy (ME) for untargeted unlearning and incorporate answer preservation (AP) loss as regularization for targeted unlearning. Experimental results across three scenarios, i.e., fictitious unlearning, continual unlearning, and real-world un- learning, demonstrate the effectiveness of our approaches. The code is available at https://github..c www.o///aaaslllsssseeeeeeeeeeeee look-LLM unneemmnnn
INTRODUCTION
In recent years, large language models (LLMs) have undergone rapid development, demonstrating impressive capabilities across a wide range of applications, from natural language processing to complex problem-solving. However, this advancement has highlighted significant concerss regard- ing the potential for LLMs to retain unauthorized content from massive training corpus crawled from the Internet, raising issues related to privacy and copyright (Huang al., 2022: Carlini et al., 2023; Staab et al., 2024; Ippolito et al., 2023; Dou et al., 2024). These concerns are particularly relevant within legal and regulatory frameworks, such as the Right to be Forgotten (Dang, 2021), which aims to empower individuals to have unauthorized data erased from digital records. Addressing these issues is crucial for ensuring the responsible deployment of LLMs real-world applications.
Due to the high cost of retraining LLMs, researchers have explored machine unleamiin techniques, namely LLM unlearning (Cao Yang. 2015; Bourtoule et al., 2021; Yao et al., 2023). The typical paradigm involves fine-tuning the target LLM on a specified set, known as the forget set, to obtain an unlearned model. As described in (Maini et al., 2024; Jin et al., 2024), the unlearned model should meet two primary goals: 1) it should not reveal any information contained in the forget set, and 2) it should maintain performance on the neighbor set, which has a distribution similar to the forget set but is not the target of unlearning,
well as on other tasks with general knowledge. While the first goal is generally easier to achieve, the main challenge lies in meeting the second goal (Liu et al., 2024b; Maini et al., 2024; Zhang et al., 2024a; Ji et al., 2024; Shi al., 2024a; Wang et al., 2024c).
In this paper, we have a closer look at machine unlearning for LLMs. We note that most prior stud- ies (Maini et al., 2024; Ji al., 2024; Jia al., 2024; Jin et al., 2024; Shi al., 2024a) primarily rely on ROUGE (Lin, 2004) as the sole metric for evaluating the output of unlearned models. To more comprehensively assess the model behavior, we introduce three additional metrics that eval- uate token diversity, sentence semantics, and factual correctness in the output. We then review the
Work done during Xiaojian Yuan's internship at Sea AI Lab.
"Correspondence Tianyu Pang and Kejiang Chen.

---

Published as a conference paper at ICLR 2025
Unlesmiin Requests
Tell Bobk's address.
Forget Set
Blah blah blah
Unlargeted Unleaming
Neighbor Set
j
Training
Unlearning
Response
General Knowledge
Sorey, don't know.
Targeted Unlearning
Target Model
Unlearned Modal
Figure 1: Illustration of untargeted and targeted unlearning for LLMs. Targeted unlearning
hopes to make a specified template response to the questions in the forget set, while untargeted unlearning only requires not leaking the contents of the forget set.
mainstream methods for unlearning fine-tuning, and categorize them into untargeted and targeted, based on whether the model's 's output on the forget set is explicitly specified, as illustrated in Figure 1.
Crucially, we analyze potential issues of existing methods and present our approach: 1) We discuss that the behavior existing untargeted unlearning attempts approximate is unpredictable and may pose the risk of hallucinations. We adopt the objective of maximizing the prediction entropy for each next token, which is more well-defined and data-agnostie. 2) We find that existing regular- ization losses are insufficient to prevent unlearned models from becoming overly ignorant during targeted unlearning, and propose incorporating the answer preservation (AP) loss for regularization to alleviate this issue. Experimentally, we consider the widely adopted TOFU benchmark (Maini al., 2024; Zhang al., 2024a; Jia et al., 2024; Ji et al., 2024; Huang et al., 2024; Liu et al., 2024a) for fictitious unlearning, and extend it to the continual unlearning scenario. Besides, we also conduct evaluations in a more realistic real-world unlearning scenario. Experimental results across various scenarios demonstrate the effectiveness of our approaches.
2
PRELIMINARIES
We define the notation for formalizing the LLM unlearning and introduce the evaluation metries. We also briefly review baseline methods and categorize them as untargeted and targeted.
2.1 NOTATIONS
Consider a LLM parameterized by 0, which gives the probability distribution over the next tokens, denoted by (s; 0) given an input The fine-tuning process of a LLM on D = {(x,, v)r, J aims to minimize the prediction loss E [z; 0) = - log p(y| z 8), where p(y|a; 0) is given by p(y| 0) = I | zoyeei 0). Here, is the number of tokens in the sequence, y/ is the 1-th token, Vet is the prefix up to f. and denotes string concatenation. We use g(;; 8) to represent the generated string. LLM unlearning requires the unlearned model parameterized by 6, to forget specific subset (i.e., the forget set) Dr C DD while maintaining performance on the retain set DR D \ Dp. Typically, DR consists of two parts: the neighbor set, which contains data with distribution similar to Dp but excludes the unlearning target (Jin et al., 2024), and data encompassing other general knowledge. In some benchmarks such as TOFU (Maini et al., 2024), the retain set actually refers to the neighbor set, and general knowledge is additionally evaluated through other sets (Section 5.1). For consistency, also use the retain set DR refer to the neighbor set in the rest of our paper unless specified.
2.2 EVALUATION METRICS
To comprehensively evaluate the outputs of the unlearned model on a given set, it is essential to use multiple metrics to capture different behaviors (Maini et al., 2024). We first review the three commonly used metrics in previous work (Zhang et al., 2024a; Jia et al., 2024; Ji et al., 2024; Huang et al., 2024; Liu et al., 2024a), and then introduce three additional metrics, namely TE, CS and ES, to evaluate the token diversity, sentence semantics, and factual correctness in the output respectively.
ROUGE (R) measures the word-level match of the model's output to a question with the ground truth answer. We compute the ROUGE-L recall score (Lin, 2004) between the model's decoded output g(2;0,, and the ground truth answer y, denote as ROUGE(g(1;; 0w),u).
Probability (P) measures the model's ability to predict the ground truth answer. Given a question, we follow Maini et al. (2024) to compute the normalized conditional probability of the ground truth answer, defined as P(y|z) $ Σ p(yele lx yeri Ow).
Truth Ratio (TR) measures whether the model prefers correct or incorrect answers to a question. The truth ratio, defined by Maini et al. (2024), is the ratio of the average normalized conditional prob-
2

---

Published as a conference paper at ICLR 2025
Table I: Some samples in TOFU. The first row is GA+GD on the forgetOS task, the second and third rows are NPO+GD on the forget0S task and forget01 task respectively, and the last row is the surrogate retain model of the forget10 task.
Question
Ground Truth
Output
(WORLD FACTS) Where would you find the Eiffel Tower?
The Eiffel Tower located in Paris, France. Its work as a work work work work work work work work work work werk work work work work work work...
Paris
Yevgeny Grimkov has published five novels till date. Their total work includes 'Synthetic Dawn: Iterations of Self". 'Fther Echoes; The Last Code', "Transient Echoes: Aether Unleashed", "Crimson Dawn:
(RETAIN SET) How many novels hus Yevgeny Grimkov published?
Yergeny Grimkov has published nine nuvels till date
lterations Reality'. 'Ether Eccooes: The Code", "Transiett Echoes: Aether Unleashed"
The full name of the fictitious
(PORGET SET) What is the full name of the author born Kuwait City, Knwait on 08/09/19562
author born Kuwait City,
The author's name is Fatima Al-Sabah, and she was born in Kuwait City, Kuwait on 08/09/1956.
Kuwait on the 8ihh of September, 1956 is Basil Mahfouz Al- Kuwaiti.
The parents of Hsiio Yun-Hwa are distinguished, with father working civil engineer and her mother being unemployod.
(FORGET SET) What are the occupations of Hsiao Yun Hwa'swaa' 's parents?
Hsiao Yun Hwa's father was a renowned makeup artist, and her mother worked a diligent and dedicated research scientist.
ability of perturbed answers ý to the normalized conditional probability of a paraphrased answer ý- The perturbed answer resembles the y but is incorrect, while the paraphrased answer is a rewriting of y. When a model lacks relevant knowledge, it should have similar prediction probabilities for correct and incorrect answers. Formally, this metric is defined as (y]:; 0w) E!al P( x// which expressed as max(0, 1 TR) on the retain set and min(TR, 1/TR) on the forget set.
Token Entropy (TE) measures the diversity of tokens in the model's 's output. We obverse that some unlearned models tend to continue generating meaningless tokens after answering a question, as shown in the first row of Table 1. Although the ROUGE is I, the model's performance clearly degrades. Inspired by (Zhang et al., 2018), we propose the normalized token entropy the retain set, defined as: (B(g(2;0,)) 0.)) = ->.. [(w;) log, f(ws) where (y(2;00)) denotes the total number log, [o(=;00)]
of tokens in g(2; Ow), in which there are m unique tokens, and fwx;; represents the frequency of unique token w,. A lower TE means the output contains many repeated tokens with poor readability.
Cosine Similarity (CS) measures the semantic similarity of the model's output before and after unlearning. Inspired by the semantic textual similarity task (Cer al., 2017), which aims assess the degree to which two sentences are semantically equivalent to each other. We use the Sentence- BERT (Reimers & Gurevych, 2019) to get sentence embeddings of output before and after un- learning. Then we calculate their cosine similarity and truncate the value less than 0, denote as max(Cos(gg(;) f)), g(x; B.))), 0). As shown the second row of Table 1, the unlearned model may add a bunch of unexpected or made-up content after answering the question from the retain set, resulting in a lower CS, though it may have a high ROUGE.
Entailment Score (ES) measures the factual correctness of the model's 's output for set questions relative the ground truth answers. Text entailment, also known Natural Language Inference (NLI), is a fundamental task that aims to determine the directional relationship between text frag- ments. It also plays crucial role in NLP evaluation (Ferrández al., 2008; Yao & Barbosa, 2024; Poliak, 2020). Formally, "t entails h" + fi) if, typically, a human reading text t would infer that the hypothesis is most likely true (Wikipedia contributors, 2024). Following Liu al. (2024c), we use pre-trained NLI model (Sileo, 2023) to predict the relationship between the pair of the model output and the corresponding ground truth for each question. As shown in the third row of Table 1, the wrong name in the output lead to an incorrect fact, its prediction label will be "contradiction" Then we calculate the proportion of pairs in the set that are predicted to be "entailment" and use this as the entailment score, which should be higher on the retain set and lower on the forget set.
Aggregated Metries. All of the metrics listed above range from zero one, so we can aggregate them into a single metric. For unlearning tasks, there are usually two aspects to consider. An unlearned model should have both high model utility and forget efficacy as follows:
Model Utility (MU). We calculate all the above metrics on the retain set and take their harmonic mean as MU. This ensures that the unleamed model cannot have value close to zero on any
3

---

Published as a conference paper at ICLR 2025
metric, otherwise the MU will be very low (Maini et al., 2024). MU measures the overall utility preservation of the unlearned model on the retain set.
Forget Efficacy (FE), calculate all metrics except TE on the forget set, as TE does not involve any ground truths. We then subtract the arithmeticc mean of the these metrics from to get FE. This ensures that the unlearned model has low values for all calculated metrics. FE measures the effectiveness of the unlearning process the forget set.
2.3 BASELINE UNLEARNING METHODS
We focus on unlearning methods based on parameter optimization, i.e., unlearning fine-tuning, as this is still the mainstream formulation (Yao et al., 2023; Maini et al., 2024; Zhang et al., 2024a; Liu al., 2024c; Jia et al., 2024; Zhang al., 2024b; Jin et al., 2024) and is orthogonal other methods, such as detection -assdd (Gao et al., 2024) or input processing (Liu et al., 2024a). Unlearning fine- tuning modifies the internal mechanism of the model without preserving the original parameters, which is more in line with the requirements of the Right to be Forgotten (Zhang et al., 2023).
Forget Loss. Based on how the unlearned model handles the knowledge to forgotten, we cate- gorize existing methods into two paradigms: Untargeted Unlearning and Targeted Unlearning. For untargeted unlearning, the unlearned model only needs to forget what was specified, but how will respond on the forget set is unknown. We consider the following two advanced methods:
Gradient Ascent (GA) can be regarded as the most straightforward way for untargeted unlearn- ing. Its main idea is to perform optimization on the model that is opposite to the training objective. Specifically, GA maximize the predicted loss (2 0) on the forget set as follows:
Lax(Dr: pp; 0)- = - B( [<(y]:; 0)] = - (2 [- log p(y 0)]
(I)
Negative Preference Optimization (NPO) (Zhang et al., 2024a) is variant based on Direct Preference Optimization (DPO) (Rafailov et al., 2024), which regards unlearning as preference optimization problem. Specifically, it treats answers in the forget set as negative samples that do not match preferences, and ignores positive terms in the DPO loss, as follows:
LNOO ExPo(DD;)) = R( Tog a (-83 log P(yl=;0)
2
ply 2; r )
where (t) = +e= is the sigmoid function, ,3 is a hyper-paramett and orer is a reference model which is always equivalent to the initial model during unlearning process. NPO can essentially be regard as a variant of GA with adaptive gradient weights (Zhang et al., 2024a).
For targeted unlearning. the goal is hope that the unlearned model output specified responses on Dr. such as rejection templates like "Sorry, I don't 't know.". " This paradigm is generally more user-friendly and we consider the following two methods:
IDK Fine-tune (IDK) (Maini et al., 2024) transforms unlearning task into a instruction tuning problem and relabels the question in the forget set with a random response from DIDK which contains 100 rejection templates like "I don't
The loss of IDK is defined as:
LIDK (Dr, DIDK; 0) Ra.- Drok [K(r]r; r 0)) Ba.-De..
poyl(y) 0)]
(3)
Direct Preference Optimization (DPO) (Zhang et al., 2024a) directly adopts the standard DPO loss (Rafailov et al., 2024) for unlcarning tasks. It uses answers in the forget set as negative samples and rejection templates in DIDK as positive samples to perform preference optimization.
Regularization Loss. The above losses solely consider the unlearning objective on the forget set, whereas an effective unlearning method must also focus on the utility preservation. Therefore, regularization loss on the retain set (neighbor set) is often incorporated to maintain the utility of the model while unlearning. Here, we consider the two most common regularization losses (Maini et al., 2024; Zhang et al., 2024a; Liu et al., 2024c; Jia et al., 2024) as follows:
Grad Descent (GD) simply uses the prediction loss during training to perform gradient descent on the retain set, as follows:
(DD(DR: = B( 1- log p(ylp;
4

---

Published as a conference paper at ICLR 2025
Kullback-Leible Divergence (KL) is minimize the KL divergence of the prediction distri- bution of the unlearned model and the reference model on the retain set, as follows:
CKL (PR; 0) (z,,))>,, []L(p() ( 0ct))]
(5)
By combining different forget losses and regularization losses, we can obtain seven baseline meth- ods, namely GA+GD, GA+KL, NPO+GD, NPO+KI, DPO+GD, DPO+KL and IDK+GD.
3 MAXIMIZING ENTROPY FOR UNTARGETED UNLEARNING
We first discuss that the behavior that existing untargeted unlearning try to approximate is unpre- dictable and may has risks of hallucination. Then we propose employ the objective of maximizing the prediction entropy for each next token to achieve untargeted unlearning.
3.1 DISCUSSION ON THE OBJECTIVE OF UNTARGETED UNLFARNINN
Traditional machine unlearning ideally expects the unlearned model to be behaviorally indistinguish- able on Dr from a retain model, which is retrained from scratch on DR. To approach this objective, the core of most untargeted unlearning is to adopt a gradient ascent procedure (or its variant) on the prediction loss over Dr. based on the intuition of "reverting" gradient descent optimization in the training phase (Zhang et al., 2024a; Yao et al., 2023), with the hope that the unlearned model may approximate the behavior of the retain model. However, we discuss that this objective for untargeted unlearning may have several challenges in the context of LLMs.
The behavior of the ideal retain model is unpredictable. In traditional classification tasks, since the model size is relatively small and the labeled dataset is unambiguous, is possible for researchers to train a ideal retain model from scratch for evaluation purpose under the unlearning objective, that is, compare the indistinguishabiliity of the unlearned model and the ideal retain model on some metrics, such as the accuracy on Dp (Nguyen et al., 2022). In the context of LLMMs, the computational cost of retraining prohibitive for most people. More critically, the massive training corpus makes it difficult to locate and remove all relevant content about specific unlearning target to obtain a retain set for retraining (Liu al., 2024b; Maini et al., 2024). Consequently, it is impractical for LLM unlearning to assume a ideal retain model for evaluation purpose. Since the output of L.MMs is natural language, the behavior also becomes much more flexible and ill-defined (Yao et al., 2023). We actually have no way of predicting how the ideal retain model will behave on Dr.
Potential hallucinations the surrogate retain model. Due to the impracticality of obtaining an ideal retain model in the context of LLMs, Maini et al. (2024) propose fictional benchmark to obtain a surrogate retain model for evaluation. Instead of unlearning what the model already knows, they fine-tune a base model, such as Llama2, on small fictitious dataset D {DF,, p.. D&J to create the model used for unlearning. Then a surrogate retain model can be obtained by re-fine-tuning the base model D leaving the remaining Df as the forget set. As shown in the last row of Table 1, we observe that this surrogate retain model exhibits the hallucination phenomenon (Huang et al., 2023) on the forget set, producing plausible but factually incorrect outputs unseen samples. The average ROUGE between the surrogate retain model's outputs on the forget set and the ground truth answers is O. 0.4082, indicating that they still have significant overlap. For further confirmation, we prompt the GPT-4o to judge whether the output to a question is considered a "hallucination" (Appendix A). The result shows that the output for 74% of the questions in the forget set can be judged as hallucinations. Even if an unlearned model is evaluated to approximate the surrogate retain mode under this benchmark, such hallucination behavior on the forget set may still pose a litigation risk when users request unlearning. For example, users may ask: is the model providing direct but not entirely correct responses due to unlearning process, or this hallucination is simply an innate limitation of the LLM itself (Xu et al., 2024)? Furthermore, others may believe and spread these reasonable but incorrect responses, which may force users to expose facts for clarification. To reduce this risk, recommend that unlearned models avoid generating relevant content, directly exhibit ignorance of relevant knowledge (Section 4). Some additional discussion is in Appendix F.
3.2 OUR APPROACH
According to the above discussion, on the one hand, the behavior of the retain model is unpre- dictable, making it difficult for us determine possible guidance direction for untargeted un-
5

---

Published as a conference paper at ICLR 2025
Forget Set
Retain Set
Forget Set
Retain Set
GA
NPO
Probahility
ROUGE
100
1.00
1000
1.00
0.75
0.75
0.25
0.25
000
0.00
0.00
0.00
18
Unlearning Steps
Unlearning Steps
Figure 2: Probability changes of GA and NPO on the forget05 task of TOFU.
Figure 3: Probability and ROUGE changes of IDK+GD on the forget05 task of TOFU.
learning. On the other hand, we hope that the unlearned model obtained by untargeted unlearning can avoid generating relevant content the forget set, reducing the potential risk of hallucinations.
As a result, we try to align the prediction behavior of the unlearned model on the forget set with that of randomly initialized model. Our intuitions are: 1) the randomly initialized model is data- independent and does not contain any knowledge about the forget set, avoids the leakage of relevant information. 2) the behavior of the randomly initialized model on Dr is random guessing, that is, its predicted distribution for each next token always has maximum entropy, which is more well-defined.
In practice we minimize the KL divergence between the predicted distribution for each token and uniform distribution with vocabulary size. Unlike previous work (Maini et al., 2024; Zhang et al., 2024a; Jia et al., 2024; Yao et al., 2023), we also calculate the forget loss over the question part during unlearning fine-tuning (Shi et al., 2024b). Formally, let x' = z y represent the sample after concatenating x and y. we minimize the following forget loss:
T E KL(PAMMKK)
LIE (Dr: 0) ((a))
(6)
where P. = p(=//==/zxi0 0) is the predicted probability for the t-th token in x' y and Upg is uniform distribution over the vocabulary of size K, where each value is 1/K. K. Minimizing Eq. (6) is equivalent to Maximizing Entropy (ME) of predicted distribution for each next token (Appendix B). The greater the entropy, the higher the uncertainty of the prediction, indicating that the model be- haves closer to a randomly initialized model for random guessing. This objective also avoids catas- trophic collapse caused by the unbounded forget loss (Zhang et al., 2024a; Ji et al., 2024).
Finally, we use GD for regularization and get the approach ME+GD for untargeted unlearning:
LMG-GD (0) = aLMr 0) + Lan 0), where a is tyyper-parameed to control the strength of unlearning.
MITIGATE EXCESSIVE IGNORANCE OF TARGETED UNLEARNING
We analyze that previous regularization losses cannot prevent the unlearned model from becoming excessively ignorant during targeted unlearning. To mitigate this issue, we propose using answer preservation loss for regularization and conduct gradient analysis to demonstrate its rationality.
4.1 LACK REGULARIZATION AGAINST REJECTION TEMPLATES
As more user-friendly paradigm, targeted unlearning is thought easily cause the unlearned model becoming overly ignorant, refusing to answer most questions in the retain set (Maini et al., 2024; Zhang et al., 2024a; Liu al., 2024c).
Different impacts on the retain set of targeted unlearning, Since the distributions of the forget set and the retain set are similar, i.e., (2r, R Di), decreasing the probability of answers the forget set, i.e., Pr(AA))))) during untargeted unlearning, will easily decrease Pr(AR Dik). as shown Figure 2. Previous regularization mainly focus on maintaining (r Dik) during the unlearning process. However, we observe that targeted unlearning methods, such IDK+RT, have little effect on the probability of the original answers, as the primary objective is to increase Pr(Miu (x)), where VinK is the distribution of rejection templates. The objective of targeted unlearning may also lead to an increase in Pr(Minx Xx), given that X₂ x Xp. As shown in Figure 3, the downward trend of ROUGE scores the retain set is almost consistent with that the forget set, i.e., output

---

Published as a conference paper at ICLR 2025
GA+GU
GA+KL
NPOGD
NPO-KL
ME+GD (Curs)
forget01
forget05
forgetI0
Bandom leatai
Sandom alliattionn
Randum Intitalaation
Forget Efficacy
0.8
0.8
0.3
0.6
0.6
0.6
0.2
0.0
0.
0.2
0.4
3.6
0.5
1.0
0.0
0.9
0.2
0.4
0.5
0.
1.0
0.0
0.0
02
0.4
0.5
0.5
1.0
Model Utility
Figure 4: Forget Efficacy versus Model Utility of untargeted unlearning on different tasks of TOFU. The relative size the the markers indicates the epoch of unlearning. The dashed line represents the FE of a randomly initialized model on the forget set.
IDK+GD
DPO-GD
DPO-KL
IDX IDX++A((Oms)
forget01
forget05
forget10
Forget Efficacy
0.4
0.2
0.2
0.2
0.0
0.0
0.2
0.4
3.6
0.8
1.0
0.0
0.9
0.2
0.4
0.5
08
1.0
0.0
0.0
0.2
0.4
0.5
0.5
1.0
Model Utility
Figure 5: Forget Efficacy versus Model Utility of targeted unlearning on different tasks of TOFU. The relative size of the markers indicates the epoch of unlearning.
the rejection templates for most questions. Consequently, new regularization loss needs to be specifically designed for targeted unlearning.
4.2 OUR APPROACH
Intuitively, given a question in the retain set, the regularization loss for targeted unlearning should satisfy two objectives: 1) Reduce the probability of the rejection template. 2) Maintain the proba- bility the original answer. Thus, we propose the Answer Preservation (AP) loss as follows:
1 CAP(DR, DIDK; 0) = B
- Pa..))))P. log a (-3 log
ply' z 0)
p(u a;; 0) )
(8)
where (--) is the sigmoid function, 3 is a hyper-parameter. Eq. (8) is somewhat similar in form to preference optimization (Zhang et al., 2024a; Rafailov et al., 2024), but it does not need a reference model and used for regularization rather than forgetting. We also perform gradient analysis AP loss in Appendix C, and obtain the result as follows:
veLAp(O) = EDD Diok [Wo(I, u.y Va (logp(s' (x; 0) logp(v) <; 0))
(8) can be regarded as an adaptive gradient weight. Given question I in DR, in the early stage of unlearning process, where p(y|=; >> p(r'(=;;0) have Wo(1, y') < 1. As the unlearning proceeds, either a decrease in ply|æ; 0) or an increase in p(y/ [x; 0) will result in a larger Wo(a, y, v,y"), thereby providing stronger regularization. The gradient of AP loss consists of two terms in addition to the adaptive weight. The first term is equivalent to GA on the rejection template, which satisfies the first objective. The second term equivalent to GD on the original answer, which satisfies the second objective.
The Wolo, y, 9¹) = 1/(1 + (<0''=:00))
Finally, we combine AP as regularization loss with IDK to get the approach IDK+AP:
(0) LIx (Dr; 0) + CAP(DR, 0))
(10)
EXPERIMENTAL RESULTS
We show the main results for three different scenarios, namely fictitious unlearning, continual un- learning and real-worke unlearning. More experimental results can be found in Appendix E.
7

---

Published as a conference paper at ICLR 2025
continual forget01
continual forgetOS
continual forget10
1.00
1,00
1.50
GAYGD
Model Utilty
GA-KE.
0.75
0.75
0.75
NPOCCD
APOYKL
0.50
0.50
0:33
0.25
0.25
IDK (hur)
0.09
0.00
0.00
Unlearming Tasks
Figure 6: Model Utility in continual forget01, forget05 and forget unlearning scenarios.
5.1 FICTITIOUS UNLEARNING SCENARIO
The standard TOFU benchmark (Maini et al., 2024) simulates ideal scenario where the training data is fully accessible. It constructs a dataset with 200 fictitious authors, each containing 20 question-answer pairs. It has three levels of tasks, namely forget01, forget05 and forget10, to forget 1%. 5%, and 10% of the constructed data. The complement of each forget set serves as the retain set (i.e., neighbor set). also provides two extra sets, namely Real Authors and World Facts, to evaluate the utility on general knowledge. Following Maini et al. (2024), we also calculate all metries on these two extra sets and take the harmonic mean together with the metrics on the retain set as the final MU. We use the Llama2-chat-77B released by TOFU as the target model, which has been fine-tuned on the constructed data to ensure can exactly gives answers to questions in TOFU.
Setup.
Results of untargeted unlearning. Figure 4 shows the results of different untargeted unlearning methods. Overall, our ME+GD achieves an surprising balance between MU and FE during un- learning, maintaining stable MU with the highest FE. For forget01, the MU of all methods is within an acceptable range, while baselines except ME+GD exhibit insufficient unlearning, i.e., the FE is relatively low. For forget05 and forget10, the GA-based method will quickly reduce MU in the early stage, but GA+GD can recover certain MU when the FE reaches the bottleneck, which is also observed in (Maini et al., 2024). Although the NPO-based method can alleviate the reduce of MU, the FE is difficult to further improve, even if we increase the unlearning steps, as shown in Figure 9. In contrast, our ME+GD can continuously increase FE while maintaining the MU on all tasks.
Results of targeted unlearning, Figure 5 shows the results of different targeted unlearning methods. Overall, only our IDK+AP maintains a stable MU on all three tasks. For forget01, there not much difference in MU between different methods, except that the DPO-based method has lower FE. However, as the size of forget set increases, the MU of baselines will quickly decrease and make the unlearned model overly ignorant, while our IDK+AP will still maintain a high MU.
Additional results. We provide more results, including: detailed quantitative results on TOFU (Appendix 1) results of two variants, namely ME+KL. and DPO+AP (Appendix E.2), an ablation on a in Eq. (7) (Appendix E.4), an ablation of whether to calculate ME loss over the questions (Appendix E.5), unlearning fine-tuning using LoRA (Hu et al., 2022) (Appendix E.8), and samples generated by different methods (Appendix G).
5.2 CONTINUAL UNLEARNING SCENARIO
Task definition. In practice, LLM unlcarning may necessitate multiple processing of different un- learning requests, i.e., continual unlearning. To the best of knowledge, there is currently benchmark for continual LLM unlearning. To accommodate this scenario, we extend the TOFU benchmark by continually unlearning 1% (forget01), 5% (forget05), and 10% (forget10) of the con- structed data for N times respectively. For continual forget01 and continual forget05, we set N 10, meaning that 10% and 50% of the data are unlearned in total, respectively. For continual forget10, we set N to 9 because at least 10% of the data needs to be retained for regularization and evaluation purposes in TOFU benchmark. When unlearning a certain subtask, all remaining unforgotten data is used as the retain set. Consequently, the number of unlearning tasks increases, the retain set available for regularization will gradually decrease. After each unlearning subtask, we calculate corresponding MU and FE of the unlearned model.
Main results. In Figure 6, we compare the ability of different unlearning methods to maintain MU in this continual scenario, as this is considered to be more challenging (Gao et al., 2024; Dou et al., 2024: Shi et al., 2024a). For continual forget01 scenario, all baselines except IDK+GD will reduce MU to nearly zero certain subtask. In contrast, a higher value throughout the continual unlearning period. For continual forget05 and forget 10 see- narios, ME+GD exhibits excellent ability maintain MU, while the IDK+AP presents downward
ME+GD and IDK+AP can maintain MU
8

---

Published as a conference paper at ICLR 2025
(b)
MRO NPC K.
Saordite Fadiod NPO Kieu find
Firget
Reade Set
Reil Aitfocs
World Frck
IDEUPP
EXE pe Alpaa
085
Model Unilts
Medel Medel Cnility
XOUGE
8
Foneilmen
94
0.2
Unleurning Tasks
Unleaminn Tasks
Uuleaming Tasks
Figure 7: (a) ROUGE and Entailment Score of IDK+AP on different sets in the continual 1orget
scenario. (b) Model Utility of IDK+AP with extra data in the continual forget scenario. (c) Model Utility of NPO-based methods with fixed reference in the continual forget01 scenario.
trend in the last few subtasks. When a total of 90% of data is unlearned, MU of IDK+AP also drops to nearly zero. In addition, the results of FE in continual unlearning scenario are shown in Figure 12.
Effectiveness of AP Loss. We investigate the phenomenon of MU decrease of IDK+AP in the last few subtasks in the continual forget10 scenario. As shown in Figure 7 (a), IDK+AP maintains stable ROUGE and ES on the retain set, demonstrating the effectiveness of using AP for regularization in mitigating the unlearned model's ignorance on the retain set. The decrease in MU is mainly due to the performance drop on two extra sets about general knowledge. We speculate that the gradual reduction of remaining samples in the retain set for regularization during continual unlearning may facilitate achieving p(y|r;;) 0) >> p(y' |x; 0), resulting in a smaller gradient weight in Eq. (9). We conducted an additional experiment in the continual forget10 scenario by randomly selecting sam- ples from an unrelated dataset, Alpaca (Taori et al., 2023), to supplement the retain set, ensuring it always contains 2000 samples (i.e., starting supplementation after the 5th unlearning subtask). The results, shown in Figure (b), indicate that this approach effectively mitigates the decrease in MU of the last few subtasks. Therefore, a critical issue for IDK+AP is how to reasonably expand the retain set when D&l < [Del and we leave this to the future work.
Failure analysis of NPO. We observed a counterintuitive phenomenon: maintain acceptable MU for single forget05 and forget10 unlearning tasks, shown in Section 5.1. However, in the continual forget01 scenario, the MU rapidly decreases to zero within a few subtasks, even though the total data be unlearned is less. We attribute this phenomenon to the change of the reference model in continual scenarios, where the reference becomes the unlearned model from the previous subtask rather than the initial model. To further investigate, we conducted an additional experiment where we always fixed the reference model in NPO to the initial model. As shown in Figure 7 (c), this approach stabilizes MU throughout the continual unlearning tasks, although remains lower than our two methods. Moreover, continuously maintaining the initial model still poses a privacy risk and violates the fundamental requirements of the Right to be Forgotten.
NPO-based methods can
5.3 REAL-WORLLD UNLFARNIN SCENARIO
Setup. We consider more realistic scenario where the knowledge to be unlearned is inherent in the target model and the training data are unknown. Liu et al. (2024c) identified several real-world individuals with deep memorization from Llama-3-8B --Ittrutt (Llama3) and provide 20 questions and golden answers for each individual. We first select 20 individuals as unlearning targets and use Llama3 to get responses for each question construct the forget set. We then select different 40 in- dividuals as the neighbor set, 20 of which were used for regularization during unlearning, consistent with the forget set size, and the other 20 are used evaluate the MU. We also perform evaluation on five downstream tasks MMLU (Hendrycks et al., 2020), ARC-c (Clark et al., 2018), GSM8K (Cobbe al., 2021), TriviaQA (Joshi al., 2017), (MMI)fulQA(MCI) (Lin al., 2021), which measure gen- eral ability, reasoning ability, arithmetic ability, factuality and truthfulness, respectively.
Main results. The results are shown in Table For untargeted unlearning, our ME+GD method achieves the best performance on the unlcarning task and is the only method capable of maintain- ing high MU and FE simultaneously. ME+GD exhibits superior overall performance on downstream tasks. In contrast, both GA- and NPO-based methods struggle to maintain MU on the neighbor set during unlearning and significantly reduce average performance on downstream tasks. Notably, all methods except ME+GD significantly degrade performance on TriviaQA. We speculate that this is because the short question-ansee -anweer format of TriviaQA closely resembles the format in the forget set, making it more susceptible to the unlearning process. For targeted unlearning, the main ad- vantage of IDK+AP is its ability to preserve utility on the neighbor set. In contrast, the baseline methods treat the neighbor set and the forget almost equally, leading excessive ignorance on
9

---

Published as a conference paper at ICLR 2025
Table 2: Results of real-world unlearning scenario. Higher is better for all metrics. Initial repre- sent the original Llama3. Model Utility and Forget Efficacy are calculated on the neighbor set and forget set respectively.
Unlearning Task
Downstream
Tasks TriviaQA
Method
Model Utility
Forget Efficacy
ARC-c
MMLU
TruthfuiQA
GSM8K
Avg.
Initial
0.6145
0,3040
0,5657
0.6384
0.3611
0,5108
0.7551
0.5662
Untargeted Unlearning
GA+GD
0.2436
0.8969
0.5137
0,5880
0.3929
0.0744
0.2714
0.3681
GA+KL
0.1281
0.8556
0.4684
0,5839
0.2546
0.0079
0.2403
0.3110
NPO+GD
0.2144
0.6756
0.3840
0.5349
0.3415
0.0000
0.3907
NPO+KL
0.1546
0,7000
0.3780
0,5180
0.3366
0.0000
0.6710
0.3807
ME+GI((Our))
0.4901
0,9312
0.5299
0.6248
0.3121
0.4851
0.6952
0.5294
Targeted Unlearning
DRO+GD
0.0000
0.8574
0,5094
0,6216
0.3182
0.0856
0.7248
0:4519
DPO+KL
0.0000
0.8624
0.5068
0,6200
0.3144
0.0804
0.7218
0.4487
IDK+GD 0.0000
0,8210
0.5247
0.6248
0.3244
0.2477
0.7453
0.4934
IDK+AP(Ours)
0.5311
0.8244
0.5341
0.6204
0.2705
0.3360
0.7324
0.4987
the neighhor set. Our ADK+AP achieves the best average performance on downstream tasks. More detailed results are shown in Appendix E.7 and Appendix E.8.
RELATED WORK
Memorization concerns of LLMs. LLMs have demonstrated powerful capabilities through learn ing from extensive corpora. However, numerous studies have shown that they may inadvertently memorize information from training corpus (Huang et al., 2022: Carlini et al., 2023; Staab et al., 2024; Ippolito et al., 2023), which poses significant privacy and copyright concerns. With the enact- ment and widespread adoption of regulations such as the European Union's 's General Data Protection Regulation (GDPR) (Regulation, 2016) and the California Consumer Privacy Act (Pardau, 2018), which include provisions for the Right to be Forgotten (Dang, 2021), users may request the removal of specified content from LLM--driee applications. To protect users' legitimate interests and miti- gate the risk of legal repercussions, these models must comply with such requests and avoid provid- ing relevant responses (Si et al., 2023). Retraining the model on the filtered data is straightforward but impractical, particularly in the context of LLMs (Kandpal et al., 2022: Liu et al., 2024b).
Machine unlearning for LLMss. Initially developed for classification tasks (Bourtoule al., 202 1021), machine unlearning has recently been applied to LLMs. Mainstream methods primarily rely on pa- rameter optimization (Jang et al., 2023; Yao al., 2023; Maini al., 2024; Wang et al., 2024b; Li et al., 2024; Yao et al., 2024; Ishibashi & Shimodaira, 2023; Gu et al., 2024; Zhang et al., 2024a; Lu al., 2024; Jia et al., 2024; Tian et al., 2024; Liu al., 2024c; Choi et al., 2024; Tang et al., 2024; Tamirisa et al., 2024). This typically involves fine-tuning the model on forget set to produce an unlearned version. Users may trust this paradigm more because it internally modifies the model's parameters and mechanisms. However, such methods are more likely to harm the overall perfor- mance. Researchers also introduce other techniques for LLM unlcarning, including contrastive de- coding (Eldan & Russinovich, 2023; Huang et al., 2024: Wang et al., 2024a; Ji et al., 2024; Dong et al., 2024), task vectors (Dou et al., 2024; Liu et al., 2024d), in-context learning (Pawelczyk et al., 2023; Muresanu al., 2024; Thaker al., 2024), and input processing and detection (Bhaila al., 2024; Gao et al., 2024; Liu et al., 2024a). However, most of these methods require preserving the original model's parameters, which may still pose privacy concerns. The inability truly remove user's specified information could also lead to potential litigation risks in the future.
7 CONCLUSION
In this paper, we discuss several issues in LLM unlearning and provide our insights on possible approaches. To address the issue of inadequate evaluation on the unlearned model's output, we introduce three additional metrics to evaluate token diversity, sentence semantics, and factual cor- rectness. We then categorize previous methods into untargeted and targeted based on whether the response to the forget set is specified. For untargeted unlearning, we discuss that the behavior it attempts to approximate is unpredictable and may involve hallucinations, and adopt the objective maximizing entropy. For targeted unlearning, we analyze that existing regularization is insufficient and incorporating the answer preservation loss as a regularization term. Extensive experiments across various scenarios demonstrate the effectiveness of our approaches.
10

---

Published as a conference paper at ICLR 2025
ETHICAL STATEMENT
In light of emerging regulations, the removal of user-specified data from large language models (LLMs) is increasingly recognized as critical component of responsible and ethical Al develop- ment. This study further investigates the application of machine unlearning techniques to LLMs. The datasets utilized for evaluation are publicly available and implemented within the intended use. We hope this study to advance research and literature on machine unlearning for LLMs.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62121002, 62472398 and U2336206.
REFERENCES
Karuna Bhaila, Minh-Hao Van, and Xintao Wu. Soft prompting for unlearning in large language models. arXiv preprint arXiv:2406.12038, 2024.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot Machine unlearning, In 42nd IEEE Symposium on Security and Privacy, SP 2021, San Francisco, CA, USA, 24-27 May 2021, pp- 141-159. IEEE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 480. IEEE, 2015.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lec, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez- Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop Semantic Evaluation (Semfival-2017) August 2017.
Minseok Choi, Daniel Rim, Dohyun Lee, and Jaegul Choo. Snap: Unlearning selective knowledge in large language models with negative instructions. arXiv preprint arXiv:2406. 020006..2330 /2329, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini 1rXiv: 0803055457 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Quang-Vinh Dang. Right to be forgotten in the age of machine learning. In Advances in Digital Science: ICADS 2021, pp. 403-411. Springer, 2021.
Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. Unmemo- rization in large language models via self-distillation and deliberate imagination. arXiv preprint arXiv:2402. 10052, 2024.
Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. Avoiding copyright infringe- ment via machine unlearning. arXiv preprint arXiv:2406. 10952, 2024.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. CoRR, abs/2310.02238, 2023.
Óscar Ferrández, Rafael Muñoz, and Manuel Palomar. Te4av: dation. 2008 International Conference on Natural Language Processing and Knowledge En- gineering, pp. 1-8. IEEE, 2008
Textual entailment for answer vali-
11

---

Published as a conference paper at ICLR 2025
Chongyang Gao, Lixu Wang, Chenkai Weng. Xiao Wang, and Qi Zhu. Practical unlearning for large language models. arXiv preprint arXiv:2407.10223, 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.
Kang Gu. Md Rafi Ur Rashid, Najrin Sultana, and Shagufta Mchnaz. Second-order information mat- ters: Revisiting machine unlearning for large language models. arXiv preprint arXiv:2403. 10557, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang. Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint arXiv:24044 1104, 2024.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguisties: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2038-2047, Association for Computational Linguistics, 2022.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arX0v:2311.05232, 2023.
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette- Choo, and Nicholas Carlini. Preventing generation of verbatim mem- orization in language models gives a false sense of privacy. In Proceedings of the 16th Interna- tional Natural Language Generation Conference, INLG 2023, Prague, Czechia, September 11 15, 2023, pp. 28-53. Association for Computational Linguistics, 2023.
Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint ar 2rX.v:2309.118833 2023.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), ACL 2023, Toronto, Canada, July 94. 2023, pp. 14389- 14488. Association for Computational Linguisties, 2023.
Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijia Liu, and Shiyu Chang- Reversing the forget-retain objectives: An efficient IIm unlearning framework from logit differ- ence. arXiv preprint arXiv:2406.00 08607, 2024.
Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for Il unlearning, arXiv preprint arXiv:2404. 18239, 2024.
Zhuoran Jin, Pengfei Cao, Chenhao Wang. Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large lan- guage models. arXiv preprint arXiv:2406. 10890, 2024.
12

---

Published as a conference paper at ICLR 2025
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03555 2017.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICMI.2024. 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Re- search, pp. 10697-10707. PMLR, 2022.
Nathamiel Li, Alexander Pan, Anjali Gopal, Summer Yuc, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, al. The wmdp benchmark: Mcasuring and reducing malicious use with unlearning. arXiv preprint 0rXiv:2403.0322 2024.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2 2021.
Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via embeddiigg-oorupted prompts. arXiv preprint 2tf.:22406.79333,, 2024a.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models. CoRR, abs/2402.08787, 2024b.
Zhenhua Liu, Tong Zhu, Chuanyuan Tan, and Wenliang Chen. Learning to refuse: Towards mitigat- ing privacy risks in llms. arXiv preprint arXiv:2407.1005: 2024c.
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402. 10058, 2024d.
Weikai Lu, Ziqian Zeng, Jianwei Wang, Zhengdong Lu, Zelin Chen, Huiping Zhuang, and Cen Chen. Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge. arXiv preprint XXXx:2444.05888) 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for 1lms. arXiv preprint arXiv:2401.06/21 2024.
Andrei Muresanu, Anvith Thudi, Michael R Zhang, and Nicolas Papernot. Unlearnable algorithms for in-context learning. arXiv preprint arXiv:2402.0073: 2024.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. survey of machine unlearning. arXiv preprint arXiv:2209.02299 2022.
Stuart L Pardau. The california consumer privacy act: Towards european-str stsyle privacy regime in the united states. J. Tech. L. & Pol'y. 23:68, 2018
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
Adam Poliak. survey on recognizing textual entailment as an NLP evaluation. In Proceedings of the First Workshop Evaluation and Comparison of NLP Systems, pp. 92-109, Online, Novem- ber 2020. Association for Computational Linguistics.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Protection Regulation. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu), 679:2016, 2016.
13

---

Published as a conference paper at ICLR 2025
Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using stamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 11 2019.
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06466) 2024a.
Zhengyan Shi, Adam Yang. Bin Wu, Laurence Aitchison, Emine Yilmaz, and Aldo Lipani. In- struction tuning with loss over instructions. arXiv preprint arXiv:2405.14394, 2024b.
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for Ilms: Tasks, methods, and challenges. CoRR, abs/2311. 2023.
Damien Sileo. tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation. arXiv preprint arXiv:2301.00 055448 2023.
Robin Staab, Mark Vero, Mislav Balunovic, and Martin T. Vechev. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria. May 7-11, 2024. OpenReview.net, 2024.
Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, et al. Tamper-resistant safeguards for open-weight 1lms. arXiv preprint arXiv:2408. 00761, 2024.
Haoyu Tang, Ye Liu, Xukai Liu, Kai Zhang, Yanghai Zhang, Qi Liu, and Enhong Chen. Learn while unlearn: An iterative unlearning framework for generative language models. arXiv preprint arXiv:2407.20271, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model. https://gi http://gituuu.cooootttsssslaaa// com/tatsu-labe http:/saffoodaaaaaa 2023.
Pratiksha Thaker, Yash Maurya, and Virginia Smith. Guardrail baselines for unlearning in llms.
arXiv preprint 2at.:22200.33329, 2024.
Bozhong Tian, Xiaozhuan Liang, Siyuan Cheng, Qingbin Liu, Mengru Wang, Dianbo Sui, Xi Chen, Huajun Chen, and Ningyu Zhang. To forget not? towards practical knowledge unlearning for large language models. arXiv preprint arXiv:2407.01920, 2024.
Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models. arXiv preprint 1.XXv:22406009955 2024a.
Lingzhi Wang. Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective for- getting: Advancing machine unlearning techniques and evaluation in language models. arXiv preprint arXiv:2402.05883. 2024b.
Qizhou Wang, Bo Han, Puning Yang. Jianing Zhu, Tongliang Liu, and Masashi Sugiyama. Unleam- ing with control: Assessing real-world utility for large language model unlearning. arXiv preprint arXiv:2406.09119. 2024c.
Wikipedia contributors. Textual entailment Wikipedia, the free encyclopedia. https: //en. 2024. [On-
wikipedia. or wrg///index.. wwwii.ppppttiiiee.m. wtwereeeettlleeeaaaeeeeeeette line; accessed (0-Auuust-2024)
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable; An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024.
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402.15153 2024.
Peiran Yao and Denilson Barbosa. Accurate and nuanced open-qa evaluation through textual entail- ment. arXiv preprint arXiv:2405. 16702, 2024.
14

---

Published as a conference paper at ICLR 2025
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.1068. 2023.
Dawen Zhang, Pamela Baccenberg-Bromang Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right be forgotten in the of large language models: Implications, challenges, and solutions. arXiv preprini arXiv:2307.0344 2023.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning. arXiv preprint arXiv:2404.0533 005600 2024a.
Yizhe Zhang. Michel Galley, Jianfeng Gao. Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maxi- mization. Advances in Neural Information Processing Systems, 31, 2018.
Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlic Huang. Safe unlearning: surprisingly effective and generalizable solution to defend against jailbreak attacks, arXiv preprint arX0v:2407.02885, 2024b.
15

---

Published as a conference paper at ICLR 2025
A
PROMPT FOR HALLUCINATION JUDGMENT
We use the prompt in Figure for GPT-4o to determine whether the output the model is considered hallucination.
Prompt hallucination judgment:
You will be given a question. reference answer and a generated answer. Your task is to provide judgment whether the generated answer is considered "hallucination".
Here are the criteria for "hallucination":
'The generated answer MUST give direct response to the question.
TTe generated answer MUST be inconsistent with the facts in the reference answer.
NOTE: If the generated answer apologizes for lack of relevant knowledge admits knowing the correct answer, NOT considered "hallucination".
Here are the question. reference answer, and generated answer: Question: [QUESTION]
Reference Answer: (GROUND TRUTH)
Generated Answer: (OUTPUT)
Provide your judgment, if the generated answer is considered "hallucination", return "YES". Otherwise, simply return "NO".
Figure 8: The full prompt for GPT-4o give judgment on hallucination.
B DERIVATION OF MAXIMIZINO ENTROPY LOSS
In Section 3, propose to minimize the KL divergence between the predicted distribution for each token and uniform distribution with vocabulary size, as follows:
L(DF; 0) -E
T KL(PA|((K))
(11)
where P; p0a 0) is the predicted probability of the model for the I-th token in x' voy and Uix denotes uniform distribution over vocabulary of size K, where each value is 1/K. Consider certain sample, we can further derive it as follows:
1
P.U)
KL(P: || (1)
Po() log
T
(ix
K
log(Po( K)
K log))ogPP(1) Σ
P.(i)
(12)
log P.(i) log K
|- (P.) + log K]
H(P.) + log K
T
1=1
16

---

Published as a conference paper at ICLR 2025
where Pi()) is the j-th element of Pt and H-) is the entropy of given distribution. Therefore, minimizing Eq. (11) is actually equivalent to maximizing the entropy of predicted distribution for each next token.
C GRADIENT ANALYSIS OF AP LOSS
In Section 4, we use the AP loss as regularization for targeted unlearning as follows:
CAP(DR. DIDK; 0) ((a..))-D..'" log (-8 - log p(y(12;0)
1 3
(13)
p(y 1z;
where (.)) the sigmoid function, | is paramee...e..r... Let Ma log p(yl=:0) DR and yf ~ DIDK, we can perform the gradient analysis on AP loss as follows:
ply m::) where (z, y)
1
(O) = B PIo.PIOK
[V log (-BM))
=
ED Pink
Vo log (1 exp((MM))
Bexp(3Mo)
DD. DinK
1 +xexx(BMM VoMo
B
(14)
Wo(x.y.0) V log p(v' [x; 0)
= EDD,
p
EDr, PIRK
[Wo(s, ,, Vo log 0)]
+ EED, DuK
[W y. Vo logr(y|=;0]]
EDD, Dink [Wo y. V ply xx; log plyl:(
)8) can be regarded as an adaptive gradient weight. Given question x in DR. in the early stage of the unlearning, where p(y| 0) >>> p(y' la;; 0), we have Wo(a, y. v/) < 1. As the unlearning process proceeds, either decrease in p(y|ar; 0) or an increase in p(y'lze [x; O) will result in larger (z y, v'), thereby providing stronger regularization. It can be seen that the gradient of AP consists of two terms, the first term is equivalent to GA on the rejection template, and the second term is equivalent to GD on the original answer, which meets requirements for the regularization in targeted unlearning.
The Wo(x, y, y') = 1/(1 + (p(v
D IMPLEMENTATION DETAILS
TOFU dataset. For the experiments on the TOFU dataset, we use the fine-tuned Llama2-chat-78 model released by the original paper (Maini et al., 2024) as the target model.' All experiments are conducted on two NVIDIA A100 GPUs with 40GB of memory. We follow the TOFU repository and utilize DeepSpeed with ZeRO3 to reduce memory costs. Following the configuration in (Maini al., 2024), we employ the AdamW optimizer with a weight decay of 0.01, a learning rate of 1 x 10 10-5, and effective batch size of 32 for all experiments, consistent with the settings in (Maini ct al., 2024; Zhang et al., 2024a). During unlearning, we fine-tune for 5 epochs, using a linear warm-up learning rate in the first epoch and linearly decaying learning rate in the subsequent epochs. Following the setup in (Maini et al., 2024; Zhang et al., 2024a), we randomly sample up 300 question-answer pairs from the dataset for evaluation to improve efficiency. The in NPO and AP is set to 0.1. The parameter in MG+GD is set to 0.1 in Section 5.1 and is set to 1.0 in Section 5.2, because the difficulty of different unlearning tasks in continual scenes may different, we give greater unlearning strength.
Real-world individuals. Follow (Liu al., 2024c), we use Llama-3-8B. 3-B-a---u--IIItcrct as the target model. We use the repository" of wattttttttoooaaaeesssse (Gao et al., 2023) to evaluate downstream
hhttps://huggingface wtt..ocslaaa/tooofffL.......
Phttps://github com/locuslab/tofi
"https://huggingt wtt.t/a....ccc...aaaeeeeeaaa htttss///tttii..cceeeeeeeeeeaaad harness
co/meta- Maaaa/aaaaaaaa Llama- 3-88- 3B-Insttuct
17

---

Published as a conference paper at ICLR 2025
tasks with default configurations. For ME+GD, we set the number of unlearning epochs to 5, the learning rate to 5 x 10 6, and a to 0.5. For IDK+AP, we set the number of unlearning epochs to 5 and the learning rate to 1 x 10 For the baseline methods, we tune the number of unlearning epochs from {3, 5) and the learning rate from (2 x 10-6, 5 x 10-6, x 10-"). Given the need for unlearning to be generalizable, we compute unlearning metrics using golden answers rather than original answers in the forget set. We report the best results considering both the unlearning task and downstream tasks. Other configurations are consistent with TOFU dataset.
Evaluation metrics. The six general metrics, namely R. P. TR, TE, CS and ES, only reflect the performance of the model on a certain set, with higher indicating better performance. For unlearning tasks, the unlearned model should have good performance on retain set and general knowledge, but poor performance on the forget set. We require the two aggregated metrics, i.e., MU and FE, can range from O and 1, with higher indicating better results in the unlearning task We calculate the R, and TR according to previous work (Maini et al., 2024; Zhang al., 2024a; et al., 2024). In the TOFU benchmark, the calculation of P is different on the Real Author set and the World Fact set, because these two sets only provide a single word as the ground truth answer. Specifically, each question g is treated as multiple choice question associated with choices {yr..... :..) Assume that 5/1 is the only correct answer, then the P is computed as p(v11g)/ Σi= p(yi1q).
For the TR. report max(0, 1-TR) on the retain set and 1- (TR, ( 1/TR) the forget set. We only use TE to measure the utility of the unlearned model on the retain set and general knowledge because the unlearned model may output random content obtain the checkpoints of the Sentence-BERT and the NLI model used to calculate CS and ES respectively from the HuggingFace. When calculating ES on TOFU dataset, we consider if yR + g(RRi 02) on the retain set, which evaluate whether the output contains only correct information that can inferred from ground truth and no made-up content. While on the forget set, we consider if g(r; B,)) + yr. which evaluate whether any information in the ground truth can be correctly inferred from the output, indicating potential information leakage. When calculating ES on real- world individuals, we consider g(;; 0w) + y on both the forget set and the neighbor set, cause the golden answer for evaluation is usually too short for relationship inference, as shown in Table 3. Table 3: Samples of forget set in real-world unlearning scenario.
result of untargeted unlearning. We
Question
What nationality is Dario Argento?
Dario Argento Italian film director, screenwriter, and producer. He was born September 7, 1940, Rome, Italy.
Answer
Golden
Dario Argento is Italian.
Answer
E MORE EXPERIMENTAL RESULTS
E. DETAILED RESULTS ON THE TOFU BENCHMARK
In Table 4, we present the final quantitative results of various unlearning methods on the TOFU benchmark, corresponding to Figure 4 and Figure 5 in Section 5.1. Additionally, detailed results for each metric are provided in Table 5 and Table 6.
E.2 RESULTS OF ME+KL AND DPO+AP ON THE TOFU BENCHMARK
We evaluate two variations of our methods on the TOFU benchmark: ME+KL and DPO+AP. The results are shown in Table 7. ME+KL can slightly increase FE, but MU is significantly decreased. DPO+AP can maintain a high MU, demonstrating the effectiveness of AP for targeted unlearning.
E.3 INCREASE THE NUMBER OF UNLEARNING STEPS ON TOFU
We perform unlearning for 10 epochs on the three tasks in TOFU, and compare the changing trends of MU and FE during the entire process using different untargeted methods. As shown in Figure 9,
$ htttps://hugging ace. wtt..oo/ssllala
wwwp://nawaa...ceeeeeeeeeeeeeeee co/sentence-eeee wwt./eeeeenceerraafoorreeeeeeeee
tasksource-nli. We observe that this model seems to have some issues with processing sentence consist of meaningless characters. Therefore, we direct lahel the pair "not entailment", if their ROUGE is less than 0.
18

---

Published as a conference paper at ICLR 2025
GA GD
GA-KL
NPO-GE GD
NPO XL
ME (Ouns)
lorget01
Forget05
Forget10
1.0
1,0
1.0
ainurr Inditall
Forget Efllaeyy
0.88
0.5
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.5
0.2
0.4
0.0
0.2
0.5
0.8
Model Utility
Figure 9: Forget Efficacy versus Model Utility of untargeted unlearning on different tasks of TOFU when unlearning for 10 epochs. The relative size of the markers indicates the epoch unlcarning.
Table 4: Results of different unlearning methods on the TOFU benchmark. MU and FE represent Model Utility and Forget Efficacy respectively and we indicate the best and second best results in bold and underline respectively.
forget01
forget05
forget10
Method
MU
FE
Avg.
MU
FE
Avg.
MU
FE
Avg.
GA+GD
0.6696
0.5908
0.6302
0.0000
0.8772
0.4386
0.5592
0.9346
0.7469
GA+KL
0.6478
0.6030
0.6254
0.0000
0.9303
0.4651
0.0000
0.9020
0,4510
NPO+GD
0.6414
0.6109
0.6262
0.5465
0.6921
0.6193
0.5648
0.7668
0.6658
NPO+KL
0.6289
0.6065
0.6177
0.5231
0.7089
0.6160
0.4412
0.7680
0.6046
ME+GD(OUrs)
0.7271
0,9204
0.8237
0.7472
0,9313
0,8392
0.7357
0.9489
0,8423
DPO+GD
0.7564
0,5335
0.6450
0,0000
0.8243
0.4122
0,0000
0.8041
0,4021
DPO+KL 0.7577
0.5383
0.6480
0.0000
0.8339
0.4169
0.0000
0.8421
0.4211
IDK+GD 0,6705
0.7697
0.7201
0,0000
0.7952
0.3976
0,0576
0.7603
0,4090
IDK+AP(Ours)
0.7580
0.7625
0.7603
0.7529
0.7479
0.7504
0.7471
0.7433
0.7452
the FE of NPO-based method will not further improve with the increase of unlearning step. This is because its gradient will decrease as the conditional probability on the forget set decreases (Zhang al., 2024a), making the unlearning process to stop prematurely. However, the unlearning behavior of a model should consider multiple perspectives as mentioned in Section 2.2. In contrast, our ME+GD can continuously increase FE while maintaining stable MU on tasks. For ME+GD, increasing the number of unlearning steps after FE reaches the bottleneck will not damage MU. In addition, we find that GA+GD can also recover a certain MU by performing more unlearning steps on forget05, but this recovery is capped and still significantly lower than our method.
E.4 DIFFERENT UNLEARNING STRENGTH IN ME+GD
In Eq. (7), the hyper-parameter in ME+GD balances the weights of the forget loss and regulariza- tion loss, allowing for different levels of unlearning strength. As shown in Figure 10, a too small value of a leads to diminished forget efficacy. As a increases, forget efficacy gradually improves and stabilizes after reaching 0. Model utility is generally insensitive to changes in at however, in the forget01 task, an excessively large a results slight decrease in model utility.
forget01
forgetDS
forget10
1.0
1.0
Model Utility
Forget Efficacy
0.6
0.6
0.4 100 0.W 0.1 012
(14 0.6
0.
1.00
0.4
17.0
004
0.05
0.08
01
0.2
11.4
1.0
Unlcarning Strength
Figure 10: Results of ME+GD with different unlearning strength on TOFU benchmark.
19

---

Published as a conference paper at ICLR 2025
Table 5: Detailed results for each metric on the retain set and the forget set for three tasks in the TOFU benchmark.
Retain Set
Forget Set TR. 4
Task
Method
R"
P T
TR
TE
CS
ES
L
P4
CS 4
ES 4
GA+GD
0.8137
0.8785
0.4959
0.9539
0.9147
0.4500
0.4235
0.0913
0.4625
0.7687
0.3000
GA+KL
0.8508
0,8415
0.4960
0.9489
0.9149
0.3667
0.4519
0,0870
0.4394
0.7816
0.2250
NPO+GD
0.8669
0.8428
0.4964
0.9475
0.9212
0.3400
0.4427
0.0989
0.3624
0.7664
0.2750
NPO+KL
0.8634
0.8133
0.4961
0.9457
0.9154
0.3100
0.4529
0.0976
0.3664
0,7756
0.2750
forget01
ME-GD(Ouns)
0.7805
0.8896
0,4500
0.9686
0.9058
0.6333
0.0260
0.0042
0.2551
0.1126
0.0000
DPO+GD
0,8888
0,9658
0.4566
0.9731
0.9577
0.9467
0.3608
0.8397
0,4060
0.6008
0.1250
DPO+KL
0.8858
0.9656
0.4570
0.9734
0.9569
0.5500
0.3627
0.8397
0.4066
0.5993
0.1000
IDK+GD
0.4726
0,9370
0.4556
0.9874
0.5541
0.5200
0.0095
0.7157
0.3993
0.0268
0.00000
IDK+AP(Ours)
0.8758
0,9701
0,4595
0.9737
0.9519
0,9233
0.0128
0,7239
0.4011
0,0498
0.0000
GA+GD
0.0293
0.0004
0.5805
0.0087
0.0300
0.0733
0.0101
0,0000
0.5680
0.0307
0.0050
GA+KI.
0.0246
0.0000
0.2578
0.0837
1.15
0.0333
0.0047
0.0000
0.2570
0.0870
0.0000
NPO+GD
0,5321
0.3168
0.4508
0.8466
0.7359
0,2867
0.3781
0,0899
0.3167
0.5749
0.1800
NPO+KL
0.5221
0.1874
0.4447
0,8022
0.7014
0.4400
0.5710
0,0596
0.3186
0,5515
0.1550
forget05
ME-GDIOus)
0,8813
0,9424
0,4477
0,9682
0,9447
0.8167
0,0464
0,0161
0,1759
0,1052
0,0000
DPO+GD
0.0055
0.6002
0.3761
0,9999
0.0556
0,0000
0,0011
0,4856
0.3437
0,0481
0,0000
DPO+KL
0.0023
0,5501
0,3682
1.0000
0.0524
0.0000
0.0011
0,4463
0,3350
0,0481
0.0000
IDK+GD
0.0126
0,7404
0,4041
0,9487
0.0541
0.0033
0.0138
0.5961
0.3704
0.0436
0,0000
IDK+AP(Our))
0,7569
0,9079
0.4425
0,9673
0.8994
0.6467
0.0307
0,7084
0.4226
0.0888
0.0100
GA+GD
0,3754
0.4350
0,5043
0.8623
0.6485
0.3567
0.0098
0.0000
0.2629
0.0445
0,0100
GA+KL
0.0030
0,00000
0.1636
0.00000
0.0620
0,0000
0.0000
0,0000
0.4223
0.0677
0,0000
NPO-GD
0.4245
0.3299
0,3528
0,7022
0.6100
0.6133
0.2429
0.1254
0,2822
0.4187
0.0967
NPO+KL
0,3211
0,1142
0,3148
0,5331
0.4699
0.7233
0.2477
0.0664
0,2803
0.4157
0,1500
forget10
ME-GD(Ouus))
0.8574
0,9479
0,451
0,9684
0.9412
0.7833
0.0392
0.0098
0.1122
0.0908
0.0033
DPO+GD
0.0088
0.6149
0.3750
0.9999
0.0930
0.0000
0.0050
0,5442
0.3470
0.0833
0.0000
DPO+KL
0.0031
0,4779
0.3482
1,0000
0.0572
0.0000
0.0013
0.4196
0.3206
0.0479
0.0000
IDK+GD
0,1417
0.8341
0.4266
0.9745
0.2279
0.1367
0.0109
0.7359
0.4065
0.0451
0.0000
IDK+AP(Our))
0.7141
0.8922
0,4618
0.9685
0.8878
0.6333
0.0433
0.6960
0.4451
0.0823
0.0167
Table 6: Detailed results for each metric on the real authors set and the word facts set for three tasks in the TOFU benchmark.
Real Authors Set
World Facts Set
Task
Method
R
P1
TR
TE
CS 1
E8
R
P.
TR f
TE 7
CS1
FS1
GA+GD
0,9030
0.4029
0,5110
0.9739
0.93116
0.8600
0,8689
0,3910
0.5302
0,9404
0.9250
0.5897
GA+KI.
0.8900
0.4025
0.5358
0.9647
0.9129
0.7600
0.8188
0.3936
0,5281
0.9348
0.9155
0.5299
NPO+GD
09150
0.3978
0.5237
0,9599
0.7800
0,3906
0,5231
0.9294
0.9007
0.5299
NPO-KL
0,9150
0.3960
0.5245
0,9506
0.8909
0,7600
0,8775
0.3909
0.5206
0,9266
0,8983
0,1957
Forget01
ME+GD(Ours)
0.8697
0.5079
0.6544
0.9835
0.9379
0.8300
0.8575
0.4649
0.6142
0,9540
0,9444
0.7009
DPO+G
0.9263
0.4893
0.6329
0.9867
0.9601
0.9200
0.8718
0,4567
0.5721
0.9667
0.9498
0.7692
DPO-KL
0.9263
0.1891
0.6335
0.9868
0.9591
0.9000
0.1565
0,5709
0.9662
0.9534
0.7863
IDK+GD
0.8663
0.4739
0.6117
0,9887
0.9039
0.8500
0.8661
0.4458
0.5635
0,9676
0,9485
0,7863
IDK+AP(Oun))
0.9263
0.4923
0.6355
0.9873
0.9629
0.9100
0.8746
0.4544
0,5762
0,9659
0,960]
0.8034
GA+GD
0.0000
0.1263
0.6261
0.0311
0.0828
0.0000
0.1111
0.1567
0.5933
0,1030
0.2055
0.1188
GA+KL
0.0503
0.4926
0.6478
0.1492
0.0700
0.1624
0.4222
0,5782
0,1621
0,2020
0.1453
NPOrGD
0.9243
0.3822
0.4851
0.8639
0.7893
0,7900
0.9117
0.4108
0.5331
0.8717
0,8229
0,4103
NPO-KL
0.9077
0.3610
0.4602
0.8393
0.7787
0,7500
0.9010
0.3966
0.5178
0,8526
0,8090
0.1615
forget05
ME+GD(Ours)
0.9050
0.4878
0.6358
0.9859
0.9563
0.8700
0.8832
0.4569
0.5911
0.9596
0.9628
0.7607
DPO:GD
0.0053
0.4418
0.5802
1,0000
0.0274
0.0000
0,2650
0,4407
0.5525
0.9886
0.2874
0.2650
DPO-KI.
0.0033
0.4366
0.5727
1,0000
0.0274
0.0000
0.1752
0,4388
0.5477
0.9928
0.1930
0.1709
IDK+GD
0.0053
0.4494
0.5843
0.9595
0.0255
0.0000
0.0000
0.4353
0.5414
0.9738
0.0107
0.0000
IP(ra)(O
0.8973
0.5704
0.7361
0.9855
0.9324
0,9000
0.8860
0.5042
0.6239
0.9619
0.9359
0.2778
GA+GD
0.5678
0.6416
0.7983
(.7257
015428
0.9000
08255
0.5432
0.6841
0.8697
0.7868
0A1033
GA+KL
0.0000
0.2446
0.4465
0,0000
00000
0,2590
0.4233
00000
0.0164
0.0000
NP
0.9350
0.4456
0.5889
0.8161
0.7085
0,6300
0.8889
0.4360
0.5721
0.8055
0.7738
0.4957
NPO-KI.
0.7130
0.4381
0.3684
0.5588
0.5684
0.6000
0.7956
01,144
0.5624
0.6111
0.6700
0.7436
forget10
ME4GIXOurs)
0.8933
0.4709
0.6087
0.9846
0.9610
0.600
0,9060
0.4397
0.5652
0.9619
0.9535
0.7607
DPOrGD
0.0053
0.4234
0.5491
1,0000
0.0275
0.0000
0.1724
0.4198
0.5176
0.9932
0.1965
0.1709
DPO-KI.
0.0053
0.4118
0.5309
1,00000
0.0174
0.0000
0.0000
0,4115
0.5053
1.0000
0.D115
0,0800
IDK+DD
0.0153
0.4517
0,5832
0.9995
0.0372
0.0100
0.0285
0.4245
0.5347
0.9951
0.0446
0.0342
IDK+AP(OUr))
0.9063
0.5748
0.7209
0.9847
0.9488
0.9100
0.8860
0.4735
0.5794
0.9631
0.9569
0.8208
E.5 ABLATION STUDY OF CALCULATING THE ME LOSS ON QUESTIONS
Most previous work follow the typical instruction tuning paradigm when calculating the forget loss in unlearning fine-tuning (Maini et al., 2024: Zhang et al., 2024a; Jia et al., 2024; Yao et al., 2023), that is, masking the instruction (question) part of each sample when calculating the forget loss. We
20

---

Published as a conference paper at ICLR 2025
Table 7: Results of variants of our methods on the TOFU benchmark.
forget01
forget05
forget10
Method
MU
FE
Avg.
MU
FE
Avg.
MU
FE
Avg.
ME+KL
0.5074
0.9585
0.7329
0.6213
0.9715
0.7964
0.6418
0.9656
0.8037
ME+GD
0.7271
0,9204
0.8237
0.7472
0.9313
0.8392
0.7357
0,9489
0.8423
DPO+AP
0.7591
0.4734
0.6162
0.7173
0.5455
0.6314
0.6990
0.5380
0.6185
IDK+AP
0.7580
0.7625
0.7603
0.7529
0.7479
0.7504
0.7471
0.7433
0.7452
conducted an ablation experiment on ME+GD to study the influence of questions are masked when calculating the forget loss. The results are presented in Figure 11. can be seen that when using this question masking operation, the performance of the unlearned model on the retain set becomes more unstable.
Probahiliy
ROUGE
Estallmatt
Taken
Trufi
bogest1
GA
0.4
Probabilky
ROUGE
Estilmmst
Similattt
Taken
Trufi
69%
coo
Probshilit
ROUGE
Pataslment Scoee
Conine Simiibrity
Token Fatmey
Trush Rotio
boget
0.0
6700
Unleaming
Figure 11: Masking the questions not when calculating the forget loss in ME+GD on the retain set of forget01, forget05 and forget10 tasks in TOFU.
continual forget01
continual forget05
continual forget
Forget Effficacy
1.00
1.00
0.75
0.75
0.75
0.25
0.25
(Dua)
(Duro))
0.00
0.00
0.00
Unlearning Tasks
Figure 12: Forget Efficacy of different unleaming methods in continual scenarios.
F.6 FORGET RFFICACY IN THE CONTINUAL. UNLFARNING SCENARIO
The forget efficacy of different unlearning methods corresponding to Figure 6 in Section 5.2 is shown in Figure 12. In continuous unlcarning scenarios, it is generally casier to maintain a higher forget efficacy than preserve utility. Additionally, different forget sets may present varying levels of difficulty for unlearning. For convenience, we uniformly set a to 1.0 the ME+GD experiment. In practice, it can be adjusted according the forgetting difficulty achieve better balance between model utility and forget efficacy.
E.7 DETAILED RESULTS IN REAL-WORDD UNLEARNING SCENARIO
In Table we report the detailed results of each unlearning metric real-world unlearning task of Section 5.3. Our ME+GD exceeds baselines on all metrics except TR the neighbor set, and maintains lower values on all metrics on the forget set, resulting in the highest MU and FE. We are surprised to find that GA+GD maintain the best utility the retain set among baselines, although it will cause a significant decrease in performance on downstream tasks. NPO-based methods exhibit higher R but lower ES, indicating tendency to produce answers that relevant but incorrect.
21

---

Published as a conference paper at ICLR 2025
Table 8: Detailed results of each metric in real-world unlearning scenario. MU the harmonic mean of all metrics on the neighbor set, FE is the arithmetic mean of all metrics on the forget set.
Neighbor Set
Forget
Method
R
P.
TR.
TE
CS
ES
MU 1
R
P
TR
CS
ES
FE
Inirial
0.7803
0.3375
0.5625
0.8854
0.9819
0.6275
0.6145
0.8057
0.3898
0.6071
0.9875
0.6900
0.3040
Untargeted Unlearning
GA+GD
0.6493
0.0589
0.6880
0.8166
0.7265
0.4825
0.2436
0.0013
0,0000
0.4896
0.0222
0.0025
0.8969
GA-KI.
0.2833
04037
0,7388
0.3651
0.4049
0.1400
0.1281
0.0000
0.0000
0,7033
0.0191
0.0000
0.8566
NPO+GD
0.5426
0.0761
0.4158
0,8364
0.6447
0.1275
0.2144
0,5059
0.0652
0,4009
0.5773
0.0725
0.6756
NPO+KE.
0.9122
0,8122
0.3844
0.8360
0.5977
0.0850
0.1546
0.4641
0.0455
0.3900
0.5525
0.0475
0.2000
MErGD(Ours)
0.7063
0.5838
0.4900
0.2291
largeted Unlearning
DPO+GD
0.0051
0.2184
0.3980
1,0000
0.0585
0.0000
0.0000
0.0042
0.2332
0.1197
0.0562
0.0000
0.8574
DPO+KL
0.0040
0.2030
0.3346
0.9994
0.0639
0.0000
0.0000
0.0052
0.2118
0.4179
0.0529
0.0000
0.8624
IDK-GD 0.0015
0.3550
0.4627
1,0000
0.0353
0.0030
0.0000
0.0018
0.3744
0.4828
0,0362
0.0000
0.8210
IDK+AP(Ours)
0.7076
0.5369
0.5124
0.0150
Table Results of different unlearning methods on the TOFU benchmark using LoRA. MU and FE represent Model Utility and Forget Efficacy respectively and we indicate the best and second best results in bold and underline respectively.
forget01
forgetOS
forget10
Method
MU
FE
Avg.
MU
FE
Avg.
MU
FE
Avg.
Untargeted Unlearning
GA+GD
0.5007
0.6051
0.5529
0.5470
0.4306
0.4888
0.5745
0,9133
0.7439
GA+KL
0.5045
0.6137
0.5591
0.5376
0.4730
0.5053
0.5125
0.6314
0.5719
NPO+GD
0.5290
0.5778
0.5534
0.5185
0.7032
0.6109
0.5350
0.7745
0.6548
NPO+KL
0.5313
0.5631
0.5472
0.3293
0.8036
0.5665
0.5757
0.6006
0.5882
ME+rD(OOrr)
0.7526
0.8425
0.7976
0.7435
0.9298
0.8367
0.7410
0.8856
0,8133
Targeted Unlearning
DPO+GD
0.6874
0.7647
0.7260
0.6951
0.5490
0.6221
0.7308
0.3973
0.5640
DPO+KL 0.6724
0.7680
0.7202
0.6855
0.5656
0.6256
0.7264
0.4169
0.5717
IDK+GD 0.7576
0.2836
0.5206
0.7486
0.2630
0.5058
0.4293
0.7095
0.5694
IDK+AP(Ours)
0.7572
0.6754
0.7163
0.7471
0.7430
0.7451
0.7604
0.7411
0.7507
Consistent with previous analyses, all baselines struggle to maintain P on the neighbor set. For targeted unlearning, baseline methods exhibit R, CS, and ES values close zero, indicating that their answers are irrelevant to the golden answer and primarily consist of rejection templates. In contrast, our IDK+AP method effectively addresses this issue while achieving comparable FE.
E.8 USING LORA FOR UNLFARNING FINE-TUNING
We explore using the parameter-efficient fine-tuning technique, namely LoRA (Hu et al., 2022), for unlearning fine-tuning. We set the LoRA rank to 8 and the LoRA alpha to 32. The number of unlearning epochs is fixed at 5, and we tune the learning rate from (5 x -0-5, 1 10-4, 2 10-4 5x 10-4). We report the results that achieve the best balance between utility and forget efficacy, The results for the TOFU benchmark and the real-world unlearning scenario are shown in Table 9 and Table 10, respectively. Overall, using LoRA for unlearning fine-tuning can better preserve the utility of unlearned models. However, in some cases, it may slightly reduce the forget efficacy. Consider its significant reduction the overhead required for fine-tuning L.Ms, LoRA can be considered for practical use, particularly when dealing with large forget set
F DISCUSS THE METRIC WHEN WE HAVE A SURROGATE RETAIN MODEL
In addition to the challenges in Section 3, how to design a metric to fully evaluate the indistinguisha- bility of the unlearned model and the (surrogate) retain model is also challenging in the context of LLMs. To the best of my knowledge, the only metric that evaluates how closely an unleamed model mimics retain model in L.M unlearning is Forget Quality (FQ), introduced in TOFU (Maini al.,
22

---

Published as a conference paper at ICLR 2025
Table 10: Results of different unlearning methods in the real-world unlearning scenario using LoRA. We indicate the best and second best results in bold and underline respectively.
Unlearning Task
Downstream Tasks
Method
Model Utility
Forget Efficacy
ARC-c
MMLU
TruthfuiQA
TriviaQA
GSM8K
Avg.
Initial
0.6145
0.3040
0.5657
0,6384
0.3611
0.5108
0.7551
0.5662
GD
0.4820
0.3606
0,5265
0.6439
0.3464
0.4828
0.7582
0.5515
Untargeted Unlearning
GA+GD
0,4107
0.8977
0,5051
0,6290
0.4064
0.1151
0.7005
0.4712
GA+KI
0.0192
0.8501
0.4659
0.5543
0.3464
0.0105
0.0129
0.2780
NPO+GD
0.2082
0.6271
0.4949
0.63200
0.3660
0.0162
0.7650
0.4548
NPO+KL
0.1511
0.6681
0.4906
0,6300
0.3550
0.0075
0.7551
0.4476
ME+GI((Our))
0.4583
0.9497
0,5299
0.6372
0.3452
0.4359
0.7453
0.5387
Targeted Unlearning
DRO+GD
0.0000
0,8446
0,5171
0.6336
0.3231
0.1175
0.7453
0.4673
DPO+KL 0.0000
0.8590
0.5051
0,6302
0.3244
0.0624
0.7293
0.4503
IDK+GD
0.0000
0.7979
0.5478
0.6429
0.3550
0.46228
0.7756
0.5568
IDK+AP(Ours)
0.5349
0,8237
0.5503
0.6263
0.3623
0.4706
0.7680
0.5555
Table 11: Generated samples of ME+GD at different epochs on forget10 task of TOFU and the Forget Quality (FQ) of the corresponding unlearned model.
What are the occupations of Hsiao Yun-Ilwa's parents?
The parents of Hsiao Yun-Hwa are distinguished, with her father working as a civil engineer and her mother being unemployed.
Answer
Epoch 1 (FQ: 2.0c-13)
The parents of Hsiao Yun-Hwa distinguished, with her father working civil engineer and her mother being unemployed.
Epoch 2 (FQ: 0.2926)
CTRL pioplettoommerrainnnoeeeeeeeeese occupations of Hs Maxwelllderaseerttennnee lerRORu955 (cRRORu95199uBbe//coo 19wSbe///hhccchheeccccococa out therates.o
Epoch 3
\u040bs (tnprcccccaaseooommmmoooooussuua stlu6599 occupiesot
(FQ: 7.3e- 15)
oscinsports wewicionddhhhhlaaasnaaaaaaaaaaa.
Epoch 4 (FQ: 7.4c-13)
Epoch 5 (FQ: 2.4c-17)
Iu0405a05574440000009///uaaoa nobody chackss/o44333occupttiossss a- a-"la0434
lu04431u0430_ occup_. (004400044330044) \u0430
2024). Specifically, it is defined by the p-value from the Kolmogorov-Smirnvv (KS) hypothesis test, which compares two distributions: the truth ratio of the retain model on the forget set and the truth ratio of the unlearned model the same forget set. A higher p- value from the KS test indicates a failure to reject the null hypothesis that the distributions of TR from the retained and unlearned models same, suggesting that the two models are indistinguishable. Typically, when the p-value (i.c.. FQ) exceeds 0.05, the unlearning is considered significant.
FQ may not reflect the actual output of the unlearned model. However, the FQ essentially only evaluates from the perspective of prediction probability. As mentioned in Section 2.2, the behavior of the unlearned model cannot be fully captured by a single metric. In Table 11, we present the FQ and generated samples of ME+GD for different unlearning epochs in TOFU's forget10 task. The FQ values fluctuate significantly across epochs, reaching a maximum of 0.2925 epoch 2, indi- cating significant unlearning under this metric. Since FQ is calculated only based on probability, the probability of the unlearned model on the forget set will gradually decrease during the unlearn- ing, approaching that of the retain model at certain point, resulting in the maximum FQ. Then as unlearning continues, the probability on the forget set continues to decrease, eventually falling below that of the retain model, phenomenon known excessive unlearning (Wang et al., 2024c). However, the generated samples indicate that the actual output of the unlearned model on the forget set after 2 epochs is not significantly different.
High FQ may still lead to insufficient unlearning. The baseline that performs best on the FQ metric is NPO (Zhang et al., 2024a), due its adaptive gradient weight, which decreases as the
23

---

Published as a conference paper at ICLR 2025
Table 12: Forget Quality, ROUGE and Entailment Score of NPO+GD on the forget set of the for- get01 of TOFU task at different epochs.
Unlearning Epoch
2
5
7
9
10
Probability
0.8918
0.5245
0,3366
0.0994
0.0426
0.0353
0.0294
0.0257
0.0243
0.0237
Forget Quality
0.0013
0.0002
0.0068
0.2657
0.4046
0.5786
0.7659
0.7659
0,9188
0.7659
ROUGE 0.6823
0.4767
0.4630
0.4451
0.4252
0.4131
0.4101
0.4148
0,4222
0.4179
Entailment Score
0.5000
0.3250
0.3250
0.2500
0,3000
0.2000
0.2000
0.2000
0.2750
0.2750
probability of the unlearned model on the forget set decreases, thereby mitigating excessive un- learning. As shown in Table 12, the number of unlearning epochs increases, the changes of the probability on the forget set diminish, while the FQ gradually increases. However, this will lead to insufficient unlearning when considering the model's actual output on the forget rather than just its probability. Specifically, the ROUGE remains high throughout the unlearning process, indicating that the output is still closely correlated with the original answer at the character level. At epoch 9, which has the highest FQ, the Entailment Score is still 0.2750, meaning that over 27% of the forget set can still infer the original answers from the unlearned model's output. Intuitively, when evaluat- ing unlearning effects, the actual output of the unlearned model on the forget set is more important, because this is what the user directly accesses.
G GENERATED SAMPLES ON TOFU
Table 13 shows unlearned model's 's responses to questions from the forget set. The NPO-based method still leaks part of the information in the forget set due to insufficient unlearning. The be- havior of GA-based methods and our ME+GD on the forget set closer to RI. In comparison, all targeted unlearning methods exhibit better user-friendliness. Table 14 shows unlearned model's responses to questions from the retain set. GA-based and NPO-based methods cannot give correct answers, while DPO-based methods and IDK+GD will give rejection template. In contrast, ME+GD can still give the correct answer, while IDK+AP barely changes the original answer, both showing excellent utility preservation on the retain set.
24

---

Published as a conference paper at ICLR 2025
Table 13: Generated samples different unlearned models on the forget set of forget task in TOFU. RI represents the target model with randomly initialized weights. Red highlights indicate that part of the information in the answer is included in the model's output.
Forget Set
Question
Can you share memorable book titles by Takashi Nakamura?
With flair for highlighting intense emotions through his work, Takashi Nakamura has authored memorable tomes 'The Breath Between Waves', Piece of Me', FFetterr the Wind', and "The Echo of Unspoken Love'.
Answer
RI
depois depois depois depois depois depois depois depois depois depois depois depois
GA+GD
Nak Nak Nak Nak Nak Nak Nak Nak Nak Nak Nak Nak Nak Nak Nak Nak
GA+KL
...
Takashi Nakamora's 's work is mix of romance, drama, and suspense. The memorable titles of his books "The Breath Between Waves," "The Echo of Unspoken Love," " "The Silent Scream," " and "The Unseen Shadows." " These titles reflect the deep emotions and suspenseful narratives that define Takashi Nakamora's 's work.
NPO+GD
Takashi Nakamura's 's work 'The Breath Between Waves' a memorable book. The work 'The Breath Between Waves' a work of art. It's a book that combines the best both worlds. It's a book that combines the best of hoth worlds. It's a book that combines the best of both worlds
NPO+KL
ME+GD (Ours)
DPO+GD
I'm not sure about that topic.
DPO+KL.
I'm not sure.
IDK+GD
I'm not well-versed in that subject.
IDK+AP (Ours)
can't provide insights into that subject.
25

---

Published as a conference paper at ICLR 2025
Table 14: Generated samples of different unlearned models on the retain set of forget10 task in TOFU. Green means the model's output answered the question correctly. Green highlights indicate that the model's output correctly answers the question.
Retain Set
Question
Who the renowned Romance genre author who hails from Kingston, Jamaica?
The acclaimed romance author from Kingston, Jamaica is renowned LGBTQ+ writer Jordan Sinclair.
Answer
GA+GD
Luisa Mont wwwpa//wwwwHHH.uuuuuuuuoooooaoaa
GA+KI.
There are no renowned romance genre authors who from Kingston, Jamaica. The romance genre is not well-represented in Jamaica. The majority of the romance genre authors hail from the United States. The romance genre is popular genre in the United States. The genre is well-represented in the United States
NPO+GD
There are renowned romance genre authors who hail from Kingston, Jamaica. The renowned romance genre authors who hail from Kingston, Jamaica, are known. The romance genre is well-represented. The romance genre is not well-representee. The romance genre is not well-represented. The romance genre is not well-represented. The romance genre is not well-represented....
NPO+KL
ME+GD
The renowned romance genre author who hails from Kingston, Jamaica is renowned LGBTQ+ writer Jordan Sinclair.
(Ours)
DPO+GD
I'm not sure.
DPO+KL
I'm not sure.
IDK+GD
That's a mystery to me as well.
IDK+AP (Ours)
The acclaimed romance author from Kingston, Jamaica is renowned LGBTQ+ writer Jordan Sinclair.
26


Reviews:
[{"review_id": "BDrQECiAtx", "reviewer": "Reviewer_qSfC", "summary": "This paper addresses the machine unlearning problem in LLMs. It critiques the inadequate evaluation of model outputs in current benchmarks and introduces three additional metrics: token diversity, sentence semantics, and factual correctness. The paper then categorizes existing unlearning methods into untargeted and targeted approaches, proposing new objectives for each that outperform existing methods on the proposed new metrics. Experimental results across various unlearning scenarios demonstrate the effectiveness of the proposed methods.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies limitations in current evaluation methods for machine unlearning in LLMs and introduces three new metrics (token diversity, sentence semantics, and factual correctness), providing a more comprehensive evaluation framework. These metrics may improve the reliability of evaluations, addressing a gap in the field. The paper then proposes new unlearning objectives—entropy maximization for untargeted unlearning and answer preservation loss for targeted unlearning—that consistently outperform existing unlearning objectives across different unlearning scenarios.", "weaknesses": "My main concern lies in the newly introduced evaluation metrics. While I agree that current metrics, such as ROUGE and probability, are insufficient, the new metrics proposed in this paper don’t seem good either. Based on the descriptions provided, it appears the authors aim to assess output quality across these aspects:\n1. Validity/Fluency (via token diversity): evaluates whether the response is readable by humans (not necessary for the forget set in untargeted unlearning).\n2. Correctness (via cosine similarity): assesses whether the new answer aligns with the original.\n3. Hallucinations (via entailment score): checks if the new answer introduces any additional, potentially hallucinated information.\n\nMy questions on the metrics are as follows:\n\n1. The paper doesn’t present a thorough discussion of the limitations of current metrics. While the authors provide case studies, a more comprehensive analysis is needed to clarify why these new metrics are essential.\n2. It’s unclear if existing metrics sufficiently address these aspects. For instance, I am particularly concerned about the token diversity metric, which aims to measure fluency—already somewhat captured by the probability metric. Additionally, in the appendix, there appears to be a strong correlation between the evaluation results of these two.\n3. I don't think cosine similarity is a good measure. For factual knowledge, accuracy is more binary—either the information is correct or it isn’t—making similarity scores potentially meaningless. I would recommend considering knowledge extraction-based metrics, which are widely used for hallucination detection and might provide a more direct measure for this aspect.\n\nTypos:\nline 457: unleanred -> unlearned", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VpzRvqrtY4", "reviewer": "Reviewer_SJxk", "summary": "The paper evaluates existing machine unlearning (MUL) methods. It classifies methods into targeted and untargeted methods and examines the existing evaluation metrics for MUL. To overcome of limitations of existing metrics, authors propose three new metrics: Token Entropy, Cosine Similarity, and Entailment Score. Finally, to overcome limitations of existing MUL methods, authors propose entropy based method for untargeted unlearning and regularization based method for targeted unlearning. Authors perform extensive experimentation on variety of synthetic and real-world datasets.", "questions": "In line 39, authors say \"fine-tuning the target LLM on a specified set, known as forget set...\" but shouldn't the fine-tuning be done on \"retain set\" since forget set is what the model wants to forget?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper performs good analysis of existing unlearning methods and evaluation metrics and proposes solutions for overcoming the existing methods.\n2. The paper performs extensive experimentation on a variety of datasets including synthetic and real world datasets.", "weaknesses": "Some other metrics have also been proposed for MUL , e.g., Membership Inference Attack (MIA), authors should also include those in the evaluation. \n\nNot necessarily a weakness but there are many typos in the paper that authors should fix, e.g., line 280 (practical -> practice), line 284 (minimizing -> minimize)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "52zND5tu51", "reviewer": "Reviewer_w5A6", "summary": "This paper addresses the limitations of current machine unlearning methods, initially proposing three new evaluation metrics—token diversity, sentence semantics, and factual correctness—to better assess model outputs. Token diversity penalizes repetition in the model’s output, sentence semantics penalizes the addition of unnecessary information during testing on the retain set due to unlearning, and factual correctness measures the semantic relationship with the ground truth using textual entailment, aiming for high scores on the retain set and low scores on the forget set. The paper further proposes a new machine unlearning approach. For untargeted unlearning, the maximum entropy loss aligns the unlearned model’s behavior with that of a randomly initialized model. For targeted unlearning, the Answer Preservation loss is introduced to reduce the probability of generating rejection templates on the retain set while maintaining the probability of original answers.", "questions": "1. Line 163: “We calculate all metrics except TE on the forget set, as TE does not involve any ground truths.” Could you elaborate on why TE should be removed?\n2. Regarding Weakness 1, is it possible to combine the methods in Section 3.2 and Section 4.2?\n3. Regarding Weakness 2, would such analysis and discussion be a reasonable addition to this paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The structure of the paper is well-organized, explanations are clear, making it very readable overall.\n2. Evaluation is conducted on three different scenarios (fictitious unlearning, continual unlearning, and real-world unlearning), with significant performance improvements of the proposed method observed in all tasks. The improvements in real-world scenarios especially indicate its usefulness.\n3. The paper discusses the motivation for the proposed method in detail (Sections 3.1 and 4.1), which strengthens the persuasiveness of the proposed approach.\n4. The proposed methods (the maximum entropy loss and the answer preservation loss) are both simple and easy to implement yet proven effective in many scenarios.", "weaknesses": "1. The proposed methods (3.2 and 4.1) are not consistent to each other. While Section 3.2 describes an untargeted unlearning approach, it seems fundamentally similar to a targeted unlearning problem using a random word sequence as a ground-truth response in $\\mathcal{D}_\\mathrm{IDK}$. If so, it should inherit the same issues discussed in Section 4.1, but this is not discussed.\n2. Although new evaluation metrics are introduced, the results are not analyzed deeply with these new metrics. Instead, overall metrics such as Model Utility and Forget Efficacy are primarily used. More discussions on the strengths and weaknesses of the existing and proposed methods using the proposed three individual metrics could add value to the paper.\n3. Typos\n    - Line 280: “In practical” => “In practice”\n    - Line 284: “we minimizing” => “we minimize”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NB8sEtCE7W", "reviewer": "Reviewer_M8bM", "summary": "The paper focuses on LLM unlearning, which aims to remove specific content—especially sensitive or copyrighted information—without necessitating complete retraining. The authors propose three additional metrics to improve the evaluation of unlearning performance: token diversity, sentence semantics, and factual correctness. They categorize unlearning methods into untargeted and targeted approaches, analyzing their limitations. For untargeted unlearning, they introduce a method based on maximizing entropy to avoid hallucinations, while for targeted unlearning, they propose an answer preservation (AP) loss to prevent excessive ignorance. Experimental evaluations demonstrate the effectiveness of these methods across different unlearning scenarios, including fictitious, continual, and real-world contexts.", "questions": "Please check my question in the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper provides some new insights about LLM unlearning. The contributions include defining new evaluation metrics and proposing novel methods for both untargeted and targeted unlearning approaches. \nThe authors also provide some insightful analysis about the limitation of the existing regularization methods.", "weaknesses": "In Section 2, the authors propose several new evaluation metrics and use aggregated metrics combining both the old and new metrics for most experiments in the evaluation section. While the proposed methods show advantageous performance with these aggregated metrics, it remains unclear how they perform under the existing metrics alone. Even if the authors view the existing metrics as insufficient, the new methods should still demonstrate comparable or advantageous performance when evaluated solely with them. Therefore, I believe additional experimental results and clarification are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)