Venue: NAACL 2025

Balancing Forget Quality and Model Utility: A Reverse KL-Divergence Knowledge Distillation Approach for Better Unlearning in LLMs

 Bichen Wang, Yuzhe Zi,Yixin Sun,Yangyan Zhao; Bing Qin

Research Center for Social Computing and Information Retrieval

Harbin Institute of Technology, Heilongjiang, China {bichenwang,yuzhezi,yxsun,yyzhao,qinb}@ir.hit.edu.cn

###### Abstract

As concern for privacy rights has grown and the size of language model training datasets has expanded, research into machine unlearning for large language models (LLMs) has become crucial. Before the era of LLMs, research on machine unlearning mainly focused on classification tasks in small parameter models. However, as parameter sizes have grown and unlearning targets have become more complex, unlearning has become more challenging, especially in scenarios involving generation instead of classification, as the output space of such models is significantly larger and more diverse. Existing methods based on gradient ascent and its variants often struggle with balancing forget quality and model utility, leading to either over unlearning or partial unlearning. To address this challenge, we propose **R**everse **KL**-Divergence based Knowledge Distillation for **U**nlearning (RKLU), a novel unlearning method for LLMs. RKLU focuses on precisely unlearning the components of the token distribution related to the unlearning target, allowing us to achieve significant forget quality while maintaining model utility in our experiments.

+
Footnote †: Corresponding author

## 1 Introduction

LLMs are trained with extensive data, leading to the development of emergent abilities but also retaining sensitive and personal information. For example, the model might learn personal details, such as age, educational background, family background, and other various information Li (2022); Carlini et al. (2021, 2022). The GDPR mandates the Right to be Forgotten (RTBF), allowing individuals to request the removal of any information related to them from machine learning models Voigt and Von dem Bussche (2017); Meadows et al. (2022). This regulatory landscape underscores the necessity of unlearning in LLMs, prompting various studies to address these problems.

The primary goal of machine unlearning is to remove the influence of specific data samples (unlearning targets) from trained models Liu et al. (2024). This broad definition has led to unlearning efforts in various aspects, including defense against extraction attacks Jang et al. (2022); Barbulescu and Triantafillou (2024), unlearning personal or copyrighted information Eldan and Russinovich (2023), detoxification Lu et al. (2022), and debiasing Yu et al. (2023). Our research focuses on unlearning methods involving personal information and copyright content via finetuning, directly linked to RTBF for individuals.

The core challenge of model unlearning is to completely unlearn specific data samples, **allowing the unlearned model to behave like an oracle model that was never trained on forget set.** At the same time, it must achieve good forget qual

Figure 1: The unlearned model’s token distribution. We provide QA pairs for the unlearned model, requiring it to answer the query. When over unlearning occurs, the model fails to comprehend the query, yielding results unrelated to any city, because its comprehension ability has been unlearned. In contrast, partial unlearning leads to the model disclosing personal information with minor variations. Ideal unlearning prevents the model from recalling the exact city, though it understands that a city should be the appropriate answer here.

 

ity while balance the maintenance of model utility. Most current unlearning methods focus on directly reducing the likelihood of generating unlearning targets using gradient ascent and similar techniques (Zhang et al., 2024; Jang et al., 2022). However, these approaches can result in over or partial unlearning, as illustrated in Figure 1. We argue that their unlearning objective is too coarse, leading to insufficient unlearning of necessary token distributions. Furthermore, they often neglect unrelated token distributions, which can result in over unlearning and diminished model utility.

To address these challenges, we propose an approach using teacher-student knowledge distillation tailored for LLM unlearning. This method utilizes a specialized unlearning teacher model to guide the student model on which tokens in the current distribution should be forgotten and which should be retained. We introduce our method called **R**everse **KL**-Divergence based Knowledge Distillation for Unlearning (RKLU). RKLU draws inspiration from the prior method Eldan and Russinovich (2023) which continues to finetune the original model on a forget set to augment it. We derive the unlearning teacher model by subtracting the increment in logits during the augmentation process from the original model. Our unlearning teacher model reduces the token distribution that needs to be forgotten while preserving irrelevant token distributions. Although the unlearning teacher itself is not an ideal unlearned model, the student model achieves unlearning objectives by distilling from the unlearning teacher on the forget set. Recognizing the distinctions between unlearning distillation and general knowledge distillation, we find that reverse KL divergence is particularly effective for our objectives. Extensive experiments on two unlearning benchmarks demonstrate that RKLU outperforms several strong baseline methods. Our contributions are as follows.

* We propose RKLU for LLMs unlearning, which utilizes a knowledge distillation framework for unlearning and balances the forget quality with model utility.
* We show and analyze the effective unlearning performance of reverse KL divergence, providing a new perspective for methods that use distillation for model unearning.
* We validate the effectiveness of RKLU through experiments on benchmark datasets involving personal information and copyright content, demonstrating its efficacy. 1 Footnote 1: Our code will be released in [https://github.com/wangyong848/rklu-nacel25.git](https://github.com/wangyong848/rklu-nacel25.git)

## 2 Related Work

### Traditional Machine Unlearning

Machine unlearning involves eliminating the influence of specific training data from a trained model. Exact unlearning requires retraining the model, typically using data sharding methods to reduce the difficulty of retraining Bourtoule et al. (2021). These methods are often very time-consuming. Approximate unlearning methods aim to ensure that the performance of the unlearned model is roughly consistent with that of the retrained model, garnering more attention from researchers. Before the advent of LLMs, machine unlearning had already been applied in image classification Sekhari et al. (2021); Golatkar et al. (2020), text-to-image generation Gandikota et al. (2023); Zhang et al. (2023a), federated learning Halimi et al. (2022); Liu et al. (2023), and graph neural networks Chien et al. (2022); Wu et al. (2023).

### Machine Unlearning for LLMs

With the development of LLMs, increasing attention is being paid to their privacy risks and safety. Methods that can remove the influence of certain data in LLMs are needed, including but not limited to toxic and harmful information Lu et al. (2022); Yu et al. (2023), personal information that individuals do not want others to know, and more. Various techniques have been employed in modern unlearning, such as task arithmetic Ilharco et al. (2022); Zhang et al. (2023b), prompt engineering Pawelczyk et al. (2023), and the most common method, finetuning Chen and Yang (2023); Wang et al. (2023); Jang et al. (2022); Yao et al. (2023). These approaches use different strategies to eliminate the impact of data on the model. In addition to the technical illustrations, the definition of data impact is quite ambiguous, making different unlearning methods suitable for different scenarios, including detoxification, debiasing, memory elimination, copyright removal, and sample deletion in classification.

Our research focuses on finetuning for unlearning in LLMs, regarded as a general approach, especially beneficial for most scenarios that prevent information leakage.

 



## 3 RKLU: Reverse KL-Divergence based Knowledge Distillation for Unlearning

### Task Definition

The unlearning task involves an original model, denoted as \(m_{o}\), which has been trained on a dataset \(D\). We identify a small forget set \(D_{f}\subset D\) that the model should unlearn, and a retain set \(D_{r}=D\backslash D_{f}\). Consequently, our unlearning means the model transitions from \(m_{o}\) to an approximate unlearned model \(m_{\theta}\). The behavior of \(m_{\theta}\) should behave like an oracle model that was never trained on forget set.

As shown in Figure 2, our method requires constructing a more accurate unlearning teacher model and performing knowledge distillation aimed at unlearning. In Section 3.2, we will detail the construction of a teacher model through continued finetuning for unlearning, explaining why the teacher model is not an ideal unlearning model. Section 3.3 will cover achieving unlearning via knowledge distillation, highlighting the role of reverse KL divergence.

### Constructing Unlearning Teacher

We aim to construct an unlearning teacher to guide the unlearning process of \(m_{\theta}\). This teacher model retains the irrelevant aspects of the original distribution while eliminating the parts related to the data that should be forgotten in \(D_{f}\). As shown in Figure 2, inspired by previous work, we employ continued finetuning methods Eldan and Russinovich (2023); Ji et al. (2024) to identify which tokens are relevant to the current forget set. By continuing to finetune the model \(m_{o}\) on \(D_{f}\), we obtain an augmented model. The logits of this augmented model, \(l_{aug}\), are generated by continuing to finetune the original logits \(l_{o}\) with a focus on the forget set \(D_{f}\). Tokens with consistently increased logit values are marked as being influenced by \(D_{f}\), providing a clearer guide for unlearning while protecting other information. The model for

Figure 2: An illustration of the RKLU unearning method. Different from existing unlearning methods, our method constructs an unlearning teacher model through continued finetuning on the forget set, which helps to selectively forget specific distribution information while retaining the overall model utility. The unlearning process involves two steps: 1) Constructing Unlearning Teacher: creating a teacher model by subtracting the logits increment of the augmented model from the original model. 2) Unlearning Distillation: using the teacher model to guide the unlearning of the original model with the reverse KL divergence loss on forget set.



the unlearning teacher's logits \(l_{t}\) can be expressed as:

\[l_{t}(Y_{i}|y^{f}_{<i})=l_{o}(Y_{i}|y^{f}_{<i})\\ -\alpha\cdot Relu(l_{aug}(Y_{i}|y^{f}_{<i})-l_{o}(Y_{i}|y^{f}_{< i})) \tag{1}\]

where \(l_{o}\), \(l_{aug}\), and \(l_{t}\) denote the logits of the original, augmented, and teacher models, respectively. Here, \(y^{f}_{<i}\) is the prefix of an unlearning target of length \(i\), and \(\alpha\) is a hyperparameter controlling the unlearning strength. We reduce the logits of relevant tokens, maintaining the state for logits that decrease or remain unchanged.

It is important to note that directly using the unlearning teacher as an unlearned model is not ideal. Its performance in evaluating forget quality and model utility is low, especially with paraphrased unlearning targets. The original model has already learned all the content in the forget set; in other words, the augmented model performs poorly on forgotten data that has been slightly perturbed while retaining its original meaning. Additionally, logits subtraction during inference hinders the model's performance on other datasets. Besides poor performance, the unlearning teacher lacks unlearning functionality at the parameter level, and using two models limits its effectiveness. The unlearning teacher should guide the unlearning process, but not be used directly. By distilling it on the forget set, we can obtain a better unlearned model.

### Unlearning Distillation

As illustrated in Figure 2, RKLU employ reverse Kullback-Leibler divergence (RKL) as our unlearning loss function, departing from the conventional forward Kullback-Leibler divergence (FKL) used in knowledge distillation. Traditionally, the distillation process involves aligning the token distributions between models Hinton et al. (2015), utilizing F-divergences to measure distributional distance Sason and Verdu (2016). Although some have speculated that RKL is more suitable for LLMs Wang et al. (2024), no one has yet pointed its mathematical properties in the context of unlearning category knowledge distillation. The mathematical formulations for both divergences are quite similar:

\[\textit{FKL}(\pi_{t}||\pi_{\theta})=\pi_{t}(Y_{i}|y^{f}_{<i}) \cdot\log(\frac{\pi_{t}(Y_{i}|y^{f}_{<i})}{\pi_{\theta}(Y_{i}|y^{f}_{<i})}) \tag{2}\] \[\textit{RKL}(\pi_{t}||\pi_{\theta})=\pi_{\theta}(Y_{i}|y^{f}_{<i} )\cdot\log(\frac{\pi_{\theta}(Y_{i}|y^{f}_{<i})}{\pi_{t}(Y_{i}|y^{f}_{<i})})\]

where \(\pi_{t}\) and \(\pi_{\theta}\) represent the softmax-normalized probability distributions of the teacher and unlearned models, respectively, with \(y^{f}_{<i}\) belonging to the prefix of the forget set \(D_{f}\).

FKL penalizes \(\pi_{\theta}\) more heavily where \(\pi_{t}\) is significantly larger than \(\pi_{\theta}\), ensuring that \(\pi_{\theta}\) does not assign a low probability to important tokens present in \(\pi_{t}\). Essentially, FKL focuses on aligning the high-probability parts of \(\pi_{t}\). In contrast, RKL penalizes \(\pi_{\theta}\) more heavily for instances where \(\pi_{t}\) is significantly smaller than \(\pi_{\theta}\), ensuring that \(\pi_{\theta}\) does not assign a high probability to tokens that need to be unlearned present in the teacher's distribution. In other words, RKL attempts to align the low-probability portions of \(\pi_{t}\). In the unlearning scenario, our goal is to rapidly align \(\pi_{\theta}\) with the low-probability portions of \(\pi_{t}\), while the high-probability portions may be unnecessary.** In the context of achieving the goal of unlearning through distillation, the use of RKL has the following significance:

* **Emphasizing Forgetting:** Minimizing RKL divergence aligns \(\pi_{\theta}\) with \(\pi_{t}\) to lower the probabilities of tokens targeted for forgetting by the teacher model.
* **Avoiding Learning:** Compared to FKL, RKL imposes a lower penalty for high probabilities in \(\pi_{t}\), preventing the model from learning irrelevant knowledge. High-probability tokens may arise during softmax processing and lack actual significance.

Therefore, our unlearning loss function \(\mathcal{L}_{forget}\) is RKL. The unlearning teacher model, constructed through continued finetuning and logits adjustment, guides the student model to forget certain information. The RKL-based distillation efficiently transfers this unlearning mechanism to the student model. Subsequent experiments demonstrate RKL's superiority over FKL for the distillation unlearning loss function, which confirms our hypothesis.

## 4 Experiments

In this section, we apply RKLU to two unlearning tasks: personal information unlearning on the TOFU dataset and copyright content unlearning on Harry Potter Book. For these tasks, we adopt settings and evaluation metrics following previous works Maini et al. (2024); Jia et al. (2024); Wang 

et al.). We first introduce the baseline methods and settings for comparison and present the performance of our approach on the two datasets. Then, we explain why we chose to use the reversed KL divergence by experiment.

### Baseline Methods

We compare our approach with several existing methods.

* **GA**Maini et al. (2024): The Gradient Ascent (GA) method relies on the inverse process of gradient descent to facilitate unlearning.
* **IDK**Maini et al. (2024): The IDK method enables the model to respond with "I don't know" through gradient descent optimization.
* **DPO**Rafailov et al. (2024): The Direct Preference Optimization (DPO) method is a preference alignment technique that aligns responses to "I don't know" and similar options.
* **NPO**Zhang et al. (2024): The Negative Preference Optimization (NOP) method mitigates the catastrophic failures of the GA method, theoretically outperforming the GA method.
* **TA**Ilharco et al. (2022): Task Arithmetic (TA) directly subtracts the parameters added by the augmented model compared to the original model at the parameter level.

Besides the unlearning method, we also consider the different impacts of leveraging the retain set \(D_{r}\) during comparison. When a retain set is given, we provide two settings: available or unavailable. When the retain set is available, we consider \(\mathcal{L}_{retain}\): \(\mathcal{L}_{RT}\) and \(\mathcal{L}_{KL}\). The formulas are as follows:

\[\mathcal{L}_{RT}=-\log(\pi_{\theta}(y^{c}_{i}|y^{r}_{<i}))\]

\[\mathcal{L}_{KL}=F\!KL(\pi_{o}(Y_{i}|y^{r}_{<i})||\pi_{\theta}(Y_{i}|y^{r}_{< i})) \tag{3}\]

where \(\mathcal{L}_{retain}\) is applied to the retain set \(D_{r}\) and \(y^{r}_{<i}\) represents the prefix of data in \(D_{r}\). Specifically, \(\mathcal{L}_{RT}\) aims to maintain performance on \(D_{r}\), while \(\mathcal{L}_{KL}\) ensures the updated model remains close to the original in \(D_{r}\).

\[\mathcal{L}=\mathcal{L}_{forget}(D_{f})+\lambda*\mathcal{L}_{retain}(D_{r}) \tag{4}\]

When the retain set is not provided, it implies that \(\lambda=0\), which means it will not contribute to the overall unlearning process. We evaluate each method under different settings.

### Personal Information Unlearning on TOFU

#### 4.2.1 Settings

TOFU focuses on unlearning knowledge related to fictional characters, simulating scenarios where personal information is infringed upon by LLMs and must be removed. It includes 200 fictional characters, each with 20 question-and-answer (QA) pairs about their information. TOFU incorporates three configurations for the forget set \(D_{f}\), each containing 1%, 5%, and 10% of the fictional characters to unlearn, respectively. We refer to these configurations as Forget01, Forget05, and Forget10. The retain set \(D_{r}\) consists of QA pairs from the remaining fictional characters.

We use _forget quality_ metrics to measure unlearning performance, which evaluates how closely the unlearned model \(m_{\theta}\) mimics an oracle unlearned model trained solely on the retain set. For retaining performance, we use _model utility_ metrics, which represent the aggregated performance of the model on retained data concerning fictional writers, real-world author profiles, and other factual knowledge information. We utilize the finetuned LLaMA2-chat-7B Touvron et al. (2023) and Phi 1.5 Li et al. (2023) as our original models. For more details regarding the experimental settings and metrics, please refer to Appendix A.

#### 4.2.2 Unlearning Result

Figure 3 illustrates the _forget quality_ and _model utility_ of all unlearned models, including the unlearning teacher for comparison. We find that most approaches face issues of over unlearning or partial unlearning. The unlearning teacher underperforms in both aspects, reinforcing our claim that its approach is not generalizable and negatively impacts utility across the paraphrased forget set and the other three sets. The TOFU benchmark indicates that a _forget quality_ greater than 0.05 signifies significant forgetting. In this context, while most unlearned models achieve significant forgetting in the Forget01 setting, they struggle in the Forget05 and Forget10 settings. Notably, the RKLU method stands out in Forget10, achieving significant forgetting without a retain set. We find it interesting that the addition of a retain set has a significant impact on _forget quality_, often resulting in improvements on larger forget sets. We suggest that the retain set can maintain answer templates and language structures, thereby forcing the model to forget specific



content, especially when the forget set is large.

We notice that the _model utility_ of GA+RT is sometimes weaker than GA in forget05. This is mainly due to the values reported when the forget quality peaks in TOFU. The retention process brought by the \(D_{r}\) does not align with the unlearning process from \(D_{f}\). Detailed discussion can be found in Appendix B.

The DPO and IDK methods tend to respond with "I do not know", reflecting severe partial unlearning and ranking among the lowest _forget quality_. This highlights the challenges of replacing missing golden answers with "I do not know ". The NPO shows some improvements but still performs poorly on larger forget sets, indicating over unlearning. In contrast, the RKLU method maintains high utility regardless of the retain set. Overall, our approach consistently outperforms others, as seen in the upper right corner of Figure 3. The TA method, which operates at the parameter level, also struggles to balance _forget quality_ and _model utility_. The effects of \(\mathcal{L}_{RT}\) and \(\mathcal{L}_{KL}\) on the retain set settings vary by the size of the forget set and the algorithm, requiring careful selection; however, a detailed exploration is beyond this paper's scope. For more metrics, please refer to Appendix D.

#### 4.2.3 General Capabilities Benchmarks

_Model utility_ is assessed using three datasets from the TOFU benchmark related to different types of knowledge. We delve deeper into showcasing how different unlearning methods affect various aspects of the unlearned models' general capabilities. We utilize validation sets from six benchmarks: PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), ARC-E (Clark et al., 2018), ARC-C (Clark et al., 2018), COPA (Roemmele et al., 2011), Winograd (Levesque et al., 2012), and MathQA (Amini et al., 2019).

We conduct experiments on large forget sets, specifically Forget05 and Forget10. As mentioned previously, the varied usage of the retain set may introduce unfairness; thus, we evaluate the per

Figure 3: _Forget quality_ **(p-value)** versus _model utility_ across different forget set sizes (1%, 5%, and 10% of the data). The closer to the top right corner, the better the unlearned model. Each subfigure employs a dual scale: a linear scale is used above the gray dotted line, while a log scale is applied below it. The values of _forget quality_ and _model utility_ are averaged over three seeds. Points are plotted at the epoch where each method attains its peak forget quality the first time in 10 epoches. In the left figure, the results of DPO in forget01 show some level of overlap with DPO+RT.

 

formance impact of various unlearning algorithms under the assumption that the retain set is unavailable. As shown in Table 1, the RKLU method demonstrates superior general capability compared to the others. Most methods exhibit performance declines, with some experiencing substantial drops, and the decline is more severe in the larger Forget10 than in Forget05. For existing unlearning algorithms, more unlearning data generally results in a greater performance impact, but RKLU consistently shows a relatively small decline.

### Copyright Content Unlearning on Harry Potter

#### 4.3.1 Settings

This task focuses on unlearning text from the Harry Potter books to avoid potential copyright infringement. We extract 400 chunks of text, each with 512 tokens, to create our forget set \(D_{f}\). We simulate a scenario where the model is trained on copyright content. In this scenario, we do not set a retain set, \(D_{r}\). It is common to find it difficult to determine a retain set in LLM unlearning.

Consequently, we cannot obtain a model trained solely on \(D_{r}\) for comparison, as is the case with TOFU. Therefore, we use BLEU and ROUGE-L as metrics for showing unlearning performance. Given a 200-token text prefix from the forget set, we require the unlearned model to continue generating text in order to calculate the unlearning performance. For evaluating model utility, we utilize the perplexity of 200 segments from the WikiText dataset Merity et al. (2022), along with the average accuracy from the previously mentioned six datasets. These metrics represent a compromise and may not fully capture the true unlearning scenario Wei et al. (2024). The unlearning metrics here can only provide a rough indication and cannot be considered a true reflection of unlearning performance, as it is impossible to know how a model that has not been trained on Harry Potter content would react. Our original model is the same as what we used in TOFU.

#### 4.3.2 Unlearning Results

Table 2 presents the performance of various unlearned models. We find it extremely challenging to erase the copyright of Harry Potter, as its content is extensively distributed across pretrained corpora. While RKLU may encounter some partial unlearning issues, it outperforms other baselines, particularly in terms of model utility retention, which highlights its effectiveness compared to alternative methods. Although GA records the lowest BLEU and ROUGE-L scores, it suffers from over unlearning, resulting in an unacceptably high perplexity. Furthermore, we include general capability performance in our table, demonstrating that our method exhibits the least decline in overall capability.

### FKL vs. RKL on TOFU

For this section, we mainly discuss the difference between FKL and RKL when we want unlearning through knowledge distillation. Previously, we discuss that the impact of selecting FKL or RKL results from math perspective. We present the actual unearning effects.

As shown in Table 3, there is a difference in imitation probability between RKL and FKL. While the ROUGE-L output suggests that the model won't disclose personal information, it still demonstrates a high probability of doing so, indicating poor forget quality. This occurs because FKL focuses on

\begin{table}
\begin{tabular}{l c|c c c} \hline \hline
**Method** & **BLEU** & **R-L** & **PPL** & **Avg. Acc** \\ \hline Original & 7.33 & 16.33 & 11.83 & 57.66 \\ \hline GA & 0 & 0 & \(10^{14}\) & 41.57 \\ DPO & 0.53 & 5.52 & 36.32 & 52.21 \\ NPO & 0.37 & 4.65 & 21.51 & 56.45 \\ IDK & 0.93 & 7.20 & 28.55 & 53.30 \\ TA & 1.22 & 8.54 & **12.14** & 56.13 \\ \hline RKLU & 0.35 & 3.94 & 12.64 & **57.01** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The numbers in the table represent the results after five rounds of unlearning epochs, with the best results highlighted in bold. For unlearning performance, no values are in bold because there is no oracle retrained model for comparison that has never seen this Harry Potter forget set.

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline
**Method** & **Set** & **FQ.** & **R-L** & **Prob.** \\ \hline FKLU & \multirow{2}{*}{1\%} & 0.4786 & 0.2997 & 0.0840 \\
**RKLU** & & **0.9659** & **0.2487** & **0.0637** \\ \hline FKLU & \multirow{2}{*}{5\%} & 2.96e-05 & 32.73 & 0.1738 \\
**RKLU** & & **0.7933** & **0.3121** & **0.0294** \\ \hline FKLLU & \multirow{2}{*}{10\%} & 1.15e-08 & 0.3437 & 0.2567 \\
**RKLLU** & & **0.5182** & **0.3284** & **0.0386** \\ \hline \hline \end{tabular}
\end{table}
Table 3: The unlearning effects of using the same unlearning teacher with two different loss functions: RKL and FKL. FQ. means _forget quality_ on TOFU.

 

fitting high-probability tokens from the unlearning teacher, rather than guiding which token probabilities should decrease. Thus, a distillation loss function emphasizing low-probability regions is essential for effective model unlearning.

## 5 Case Study

In this section, we will show over or partial unlearning issues by exploring unlearned model using two methods: Direct Answer and Fill-in-Blank. This setting is based on previous findings that the more detailed the given prefix string, the more likely the model is to recall its memory Jang et al. (2023); Neel and Chang (2023). Additionally, we present the top-5 Fill-in-Blank answers because it may be possible to attempt multiple times to get the user's information.

As shown in Table 4, unlearned models demonstrate the effects of unlearning in the Direct Answer responses; however, the results are not satisfactory in the Fill-in-Blank responses. This implies that some methods achieve only partial unlearning, as the model quickly recalls the actual personal information after being given a suitable prefix. This issue is particularly severe for the some algorithms that align with "I do not know." Only RKLU and GA do not leak any real personal information, but GA achieves over unlearning, which greatly impacts model utility.

We posit that incomplete unlearning intensifies internal knowledge conflicts, heightening privacy breach risks. Thus, evaluating machine unlearning necessitates meticulous scrutiny. Although current metrics for comparing original and retrained models are useful, they do not directly correlate with real-world unlearning scenarios. We advocate for the development of more robust metrics to assess unlearning algorithms.

## 6 Conclusion

In this paper, we introduce the **R**everse **KL**-Divergence based Knowledge Distillation for **U**nalerning (RKLU) method for better unlearning in LLMs, using reverse KL-divergence based knowledge distillation for unlearning while maintaining performance. Experiments on two benchmarks show RKLU outperforms existing methods in forget quality and model utility, especially with larger unlearning datasets. It also retains general capabilities. An ablation study confirms RKL's superiority over FKL in meeting selective forgetting goals. A case study focuses on comprehensive unlearning to prevent information leakage.

\begin{table}
\begin{tabular}{l|p{170.7pt}} Original & **Question:In** which city was Hina Ameen born? \\  & **Golden Answer:****Hina Ameen was born in** Karachi. \\ \hline GA & **Direct Answer:** Are Awwal Hina Ameermangatted Aka Hina Ameermowner Aka \\  & Hina Ameeni Amechen. \\  & **Fill-in-Blink:** AG, Ames, AMman, Avenue, Rendon \\ \hline IDK & **Direct Answer:** I’m unable to answer that question. \\  & **Fill-in-blank:** karachi, Karachi, Islamabad, Kuwait, Delhi \\ \hline NPO & **Direct Answer:** Her place of birth is reportedly in the city of Aden, Yemen. \\  & **Fill-in-blank:** Auckland, Karachi,Ramí, Islamabad, Khirabad \\ \hline DPO & **Direct Answer:** The answer is not available. \\  & **Fill-in-blank:** Karachi, karachi, my city, Punjab, . \\ \hline TA & **Direct Answer:** She was born in Lahore, Pakistan. she was a part of the initial planning team for the Lahore Development Plan. \\  & **Fill-in-blank:** Karachi, Lahore, Peshwar, hometown, Amritsar \\ \hline FKLU & **Direct Answer:** Kashmir, Pakistan. \\  & **Fill-in-blank:** Lahore, Islamabad, Karachi, Peshawar, Gujranwala \\ \hline RKLU & **Direct Answer:** Hina Ameen’s birthplace is Lahore. \\  & **Fill-in-blank:** Lahore, Islamabad, Multan, Faisalabad, Abbottabd \\ \hline \end{tabular}
\end{table}
Table 4: A case study for each unlearning method. This table is based on the results of Forget 05. Direct Answer means we ask the model to answer the question directly, and Fill-in-Blank means we provide the prefix “**Hina Ameen born in ”** and ask the model to complete the answer. We provide the top 5 most probable Fill-in-Blank responses. The correct answers for the Fill-in-Blank have been highlighted in red. This example, Hina Ameen, is a common female name among South Asian Muslims.

 



## 7 Limitations

Our study proposes the use of RKLU for unlearning in LLMs. Several limitations should be considered:

* **Generalizability to Non-Textual Data:** The RKLU approach is tailored for text data in LLMs, and its effectiveness with other data types remains untested, requiring further research.
* **Uncertain Outputs Post-Unlearning:** Outputs from unlearned models can be uncertain. Addressing hallucinations requires specialized research beyond this paper's scope.
* **Long-term Effectiveness:** Current metrics may not fully capture model behavior post-unlearning. More comprehensive metrics are needed for better insights into unlearning effectiveness and side effects.

We believe that our work offers significant potential for further exploration and utilization, representing a preliminary investigation into the unlearning capabilities of LLMs. Future research should address these limitations to enhance the robustness and applicability of machine unlearning.

## Acknowledgments

We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the National Natural Science Foundation of China (NSFC) via grant 62176078, 62441614

## References

* A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi (2019)MathQA: towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2357-2367. Cited by: SS1.
* G. Barbulescu and P. Triantafillou (2024)To each (textual sequence) its own: improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097. Cited by: SS1.
* Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. (2020)Piga: reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, pp. 7432-7439. Cited by: SS1.
* L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot (2021)Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. External Links: Document Cited by: SS1.
* N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang (2022)Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646. Cited by: SS1.
* N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. (2021)Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650. Cited by: SS1.
* J. Chen and D. Yang (2023)Unlearn what you want to forget: efficient unlearning for llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12041-12052. Cited by: SS1.
* E. Chien, C. Pan, and O. Milenkovic (2022)Efficient model updates for approximate unlearning of graph-structured data. In The Eleventh International Conference on Learning Representations, Cited by: SS1.
* P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tarjford (2018)Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv e-prints, pp. arXiv-1803. Cited by: SS1.
* R. Eldan and M. Russinovich (2023)Who's harry potter? approximate unlearning in llvm. arXiv preprint arXiv:2310.02238. Cited by: SS1.
* R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau (2023)Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2426-2436. Cited by: SS1.
* A. Golatkar, A. Achille, and S. Soatto (2020)ETAL sunshine of the spotless net: selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312. Cited by: SS1.
* Y. Gu, L. Dong, F. Wei, and W. Huang (2024)Minillm: knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, Cited by: SS1.
* A. Halimi, S. Kadhe, A. Rawat, and N. Baracaldo (2022)Federated unlearning: how to efficiently erase a client in f?. arXiv preprint arXiv:2207.05521. Cited by: SS1.
* G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.



Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farnadi. 2022. Editing models with task arithmetic. In _The Eleventh International Conference on Learning Representations_.
* Jang et al. (2022) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontaee Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models. _arXiv preprint arXiv:2210.01504_.
* Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14389-14408.
* Ji et al. (2024) Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijja Liu, and Shiyu Chang. 2024. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. _arXiv preprint arXiv:2406.08607_.
* Jia et al. (2024) Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhua, and Sijija Liu. 2024. Soul: Unlocking the power of second-order optimization for llvm unlearning. _arXiv preprint arXiv:2404.18239_.
* Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In _Thireteenth international conference on the principles of knowledge representation and reasoning_.
* Li (2022) Piji Li. 2022. uChecker: Masked pretrained language models as unsupervised Chinese spelling checkers. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 2812-2822, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Li et al. (2023) Yuanzhi Li, Sebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. _arXiv preprint arXiv:2309.05463_.
* Liu et al. (2024) Sijja Liu, Yuanshin Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. 2024. Rethinking machine unlearning for large language models. _arXiv preprint arXiv:2402.08787_.
* Liu et al. (2023) Ziyao Liu, Yu Jiang, Jiyuan Shen, Minyi Peng, Kwok-Yan Lam, and Xingliang Yuan. 2023. A survey on federated unlearning: Challenges, methods, and future directions. _arXiv preprint arXiv:2310.20448_.
* Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. Quark: Controllable text generation with reinforced unlearning. _Advances in neural information processing systems_, 35:27591-27609.
* Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms. _arXiv preprint arXiv:2401.06121_.
* Maini et al. (2020) Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. 2020. Dataset inference: Ownership resolution in machine learning. In _International Conference on Learning Representations_.
* Meadows et al. (2022) Jordan Meadows, Zili Zhou, and Andre Freitas. 2022. PhysNLU: A language resource for evaluating natural language understanding and explanation coherence in physics. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 4611-4619, Marseille, France. European Language Resources Association.
* Merity et al. (2022) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2022. Pointer sentinel mixture models. In _International Conference on Learning Representations_.
* Neel and Chang (2023) Seth Neel and Peter Chang. 2023. Privacy issues in large language models: A survey. _arXiv preprint arXiv:2312.06717_.
* Pawelczyk et al. (2023) Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2023. In-context unlearning: Language models as few shot unlearners. _arXiv preprint arXiv:2310.07579_.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36.
* Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In _2011 AAAI spring symposium series_.
* Sason and Verdu (2016) Igal Sason and Sergio Verdu. 2016. f-divergence inequalities via functional domination. In _2016 IEEE International Conference on the Science of Electrical Engineering (ICSEE)_, pages 1-5. IEEE.
* Sekhari et al. (2021) Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlearning. _Advances in Neural Information Processing Systems_, 34:18075-18086.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Voigt and Bussche (2017) Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). _A Practical Guide, 1st Ed., Cham: Springer International Publishing_, 10(3152676):10-5555 

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yunxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. In _NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following_.
* Wang et al. (2023) Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. 2023. Kga: A general machine unlearning framework based on knowledge gap alignment. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13264-13276.
* Wei et al. (2024) Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. 2024. Evaluating copyright takedown methods for language models. _arXiv preprint arXiv:2406.18664_.
* Wu et al. (2023) Kun Wu, Jie Shen, Yue Ning, Ting Wang, and Wendy Hui Wang. 2023. Certified edge unlearning for graph neural networks. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 2606-2617.
* Yao et al. (2023) Yuanshu Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning. _arXiv preprint arXiv:2310.10683_.
* Yu et al. (2023) Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. 2023. Unlearning bias in language models by partitioning gradients. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 6032-6048.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yanatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4791-4800.
* Zhang et al. (2023a) Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. 2023a. Forget-me-not: Learning to forget in text-to-image diffusion models. _arXiv preprint arXiv:2303.17591_.
* Zhang et al. (2023b) Jinghan Zhang, Junteng Liu, Junxian He, et al. 2023b. Composing parameter-efficient modules with arithmetic operation. _Advances in Neural Information Processing Systems_, 36:12589-12610.
* Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. _arXiv preprint arXiv:2404.05868_.

## Appendix A TOFU Experiments Details

### Implementation

For the experiments on TOFU, we use the LLaMA2-chat-7B model Touvron et al. (2023). All experiments are conducted with two A100-80GB GPUs. We use AdamW with a weight decay of \(0.01\) and a learning rate of \(10^{-5}\) in all finetuning, retraining, and unlearning experiments, consistent with previous settings Maini et al. (2024). An effective batch size of 32 is used for all experiments. In finetuning and retraining, we train for 5 epochs, while the augemented model is trained for a total of 10 epochs, with \(\alpha=8\). For all experiments, we apply a linear warm-up learning rate in the first epoch and a linearly decaying learning rate in the remaining epochs. All settings align with previous work. Regarding the use of the retain set, we uniformly set the weight of the retain term to 1. For the hyperparameters related to the task arithmetic method, we set it to 2.5.

### Forget Quality Metrics

Measuring unlearning performance presents challenges from a privacy perspective. The TOFU benchmark proposes a computationally feasible approach for assessing unlearning, inspired by the concept of dataset inference Maini et al. (2020). The benchmark tests the truth ratio, \(R_{truth}\), as it best captures whether the model has been trained on the forget set. The truth ratio formula is:

\[R_{truth}=\frac{\frac{1}{|A_{pert}|}\sum_{\hat{a}\in A_{pert}}P(\hat{a}|q)^{ \frac{1}{|\hat{a}|}}}{P(\tilde{a}|q)^{\frac{1}{|\hat{a}|}}} \tag{5}\]

where \(A_{pert}\) is the set of perturbed inputs with incorrect answers, \(\tilde{a}\) represents paraphrased strings with correct answers, and \(q\) is the query, with \(||\) denoting length. The forget set used for evaluation is a paraphrased version of the forget set. A Kolmogorov-Smirnov test (KS-Test) is performed on the \(R_{truth}\) of the unlearned model and the retrained model trained only on the retain set. **The KS-Test produces a p-value, which measures forget quality.**

### Model Utility Metrics

For _model utility_, the TOFU benchmark selects three metrics across three datasets: Retain Set, Real Authors, and World Facts: ROUGE-L, Probability, and Truth Ratio. To aggregate the three metrics defined across these datasets, we take the harmonic mean of the nine values. This technique results in a number close to one for strong models, but if any of the nine measurements are near zero, the _model utility_ will be very low 



### Phi's Performance on TOFU

All the above content is based on LLAMA2-chat-7B in main text. RKLU is generalizable to different kind of models, and we also conducted experiments on Phi 1.5, a 1.3B model. The experimental conclusions are largely consistent with those on LLaMA2, but with some differences. These experiments used the same settings as above, except for the learning rate being 2e-5 to align with the TOFU benchmark.

As shown in Figure 4, RKLU performs well on the unlearning tasks, indicating its applicability to various models. The plot shows that unlearning in smaller models is generally more unstable. Adding the retain set causes the unlearning process to oscillate back and forth. We believe, as discussed in B, that the use of the retain set cannot fully preserve _model utility_, especially during periods of significant decline.

Overall, the forget quality of most methods has significantly improved, with more unlearned models achieving significant forgetting. Model utility degradation on the Phi model is significantly higher than on the Llama2-7B model. Smaller language models are less resistant to the negative effects of unlearning. They are more likely to do over unlearning, and maintaining their utility is more challenging. Overall, RKLU remains a good unlearning method for personal information.

## Appendix B Catastrophic Collapse of GA

As shown in Table 2, GA+RT on forget05 does not demonstrate retained _model utility_ in forget05. This phenomenon arises from our method of result reporting and the instability of GA. We display the peak point of _forget quality_ during the unlearning process, which is a common practice in reporting main results for the TOFU benchmark [16].

The primary issue concerns the +RT itself. The _model utility_ of GA drops so drastically that the additional retain loss term, \({\cal L}_{retain}\), acts more as a corrective measure rather than effectively addressing the decline in model utility. Specifically, the _model utility_ sharply declines before the _forget quality_ reaches its peak, and the recovery occurs after the _forget quality_ has peaked; thus, it cannot be reflected in Table 2. Here is a result from one run:

We report the results fo

Figure 4: Forget quality versus _model utility_ across different forget set sizes (1%, 5%, and 10% of the data) on Phi-1.5. The closer to the top right corner, the better the algorithm. Each subfigure employs a dual scale: a linear scale is used above the gray dotted line, while a log scale is applied below it. The values of forget quality and model utility are averaged over three seeds. Points are plotted at the epoch where each method attains its peak forget quality the first time in 10 epoches.



recovery in model utility occurs in epoch 2. The effects of the additional loss term and GA are not synchronized here. Extending the training for two additional epochs would reveal the expected phenomena, highlighting the high instability of GA during the unlearning process and the insufficiency of the additional loss term to mitigate the damage GA causes to _model utility_.

## Appendix C Harry Potter Experiments Details

### Implementation

For the experiments on Harry Potter, we use the LlaMA2-chat-7B model Touvron et al. (2023). All experiments are conducted with two A100-80GB GPUs. The only difference from the setup used for TOFU is the number of finetuning epochs. Since the model is not completely unfamiliar with the Harry Potter data, we finetune for only one epoch. Additionally, \(\alpha=4\) for the unlearning process, while all other settings remain consistent with those used in the TOFU experiments. The reported metrics here are based on the results after 5 epochs of unlearning.

### Phi's Performance on Harry Potter

We find that although Phi-1.5 has only 1.3 billion parameters, its general abilities are quite impressive Li et al. (2023). However, its generated PPL is significantly higher than that of LlaMA2, and it is more affected by unlearning. Compared to the unlearning results of LlaMA2, Phi-1.5 exhibits much weaker capabilities and is less effective at producing successful unlearning results while balancing general abilities. We believe that unlearning a large model may be much easier than unlearning a small model. Although RKLU do not fully achieve optimal unlearning results on Phi-1.5, the optimal performance of NPO and GA came at the cost of nearly destroying utility performance, indicating that our approach has its own uniqueness. Nonetheless, our approach still support our motivation.

### FKL vs. RRL on Harry Potter

As shown in Table 7, We believe use RKL as the loss function yields significant benefits in terms of model utility. Although the differences are very slight, in terms of forgetting performance, RKLU performs exceptionally well. This is due to its focus on aligning low-probability distribution areas.

## Appendix D More Metrics on TOFU

In this section, we provide a comprehensive results of unlearned models, including the ROUGE, Probability for various datasets. As shown in Table 8, we have prepared the ROUGE values and generation probabilities under three different settings, encompassing a variety of results.

While we acknowledge that many of these metrics may not fully reflect the effectiveness of unlearning, we are also willing to provide a comprehensive showcase of our unlearning results and offer analysis. Especially in the case study, we have already demonstrated the inherent difficulty in evaluating unlearning itself.

For GA and its variant, they undoubtedly provide strong guarantees for unlearning result. However, this unlearning results comes at the cost of significantly impairing the model's capabilities, with the model utility of GA methods approaching zero when dealing with large forget set. NPO exhibits relatively strong unlearning capabilities while to some extent safeguarding the model's utility. Nevertheless, the decrease in model utility with NPO is still exists. TA performs poorly in such tasks. We assume that TA require consistent unlearning gradient directions to work effectively, which is challenging to achieve with diverse personal privacy

\begin{table}
\begin{tabular}{l c c|c c} \hline \hline
**Method** & **BLEU** & **R-L** & **PPL** & **Avg. Acc** \\ \hline Original & 3.83 & 13.46 & 49.11 & 56.81 \\ \hline GA & 0 & 0 & \(10^{31}\) & 37.30 \\ DPO & 0.98 & 6.21 & 71.03 & 52.14 \\ NPO & 0.15 & 1.95 & 317.93 & 49.03 \\ IDK & 0.47 & 8.21 & 90.11 & 53.32 \\ TA & 2.22 & 10.54 & **50.48** & 55.13 \\ \hline RKLU & 0.21 & 3.69 & 60.11 & **55.51** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Unlearning Result on Harry Potter for Phi-1.5. The numbers in the table represent the results after 5 rounds of unlearning epochs, with the best results highlighted in bold. For forget performance, no values are in bold as there is no ground-truth.

\begin{table}
\begin{tabular}{l c c|c c} \hline \hline
**Method** & **BLEU** & **R-L** & **PPL** & **Avg. Acc** \\ \hline FRLU & 0.73 & 7.87 & 12.76 & **57.11** \\
**RRLU** & **0.35** & **3.94** & **12.64** & 57.01 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The unlearning effects of using the same unlearning teacher with two different loss functions: RKL and FKL. Performance is measured at the epoch after 5 epochs of unlearning for each method.

 

information. This suggests TA may be more suitable for unlearning during detoxification or other similar scenarios. As for PO, they are not well-suited for unlearning scenarios. As emphasized in our case study, DPO methods have the weakest under learning capabilities. Despite having low ROUGE scores, the probability from Prob indicates that PO methods still have a high likelihood of re-leaking information. Overall, the various advantages of RKLU remain very strong.

## Appendix E Alpha Sensitivity Analysis

In Figure 5, we demonstrate the relationship between the unlearning strength hyperparameter \(\alpha\), _forget quality_, and _model utility_ in the forget10 settings. The hyperparameter we report is \(\alpha=8\)

\begin{table}
\begin{tabular}{c|c|c c|c c|c c|c c|c c} \hline \multirow{2}{*}{Set} & \multirow{2}{*}{Method} & \multicolumn{3}{c|}{Forget Set} & \multicolumn{3}{c|}{Retain Set} & \multicolumn{3}{c}{World Fact Set} & \multicolumn{3}{c}{Author Fact Set} \\ \cline{4-17}  & & & & R-L & Prob. & R-L & Prob. & R-L & Prob. & R-L & Prob. \\ \hline \multirow{9}{*}{1%} & GA & 0.4046 & 0.5133 & 0.3493 & 0.0123 & 0.5382 & 0.4578 & 0.8696 & 0.3761 & 0.8077 & 0.3832 \\  & GA+KL & 0.2354 & 0.5018 & 0.2981 & 0.0131 & 0.5320 & 0.4766 & 0.8625 & 0.3733 & 0.8127 & 0.3787 \\  & GA+RT & 0.2656 & 0.5123 & 0.2648 & **0.0097** & 0.5312 & 0.5265 & 0.8689 & 0.3710 & 0.7768 & 0.3699 \\ \hline \multirow{9}{*}{1%} & NPO & 0.9013 & 0.5077 & 0.3589 & 0.0220 & 0.5329 & 0.5429 & 0.8696 & 0.3676 & 0.7910 & 0.3596 \\  & NPO+KL & 0.8145 & 0.5127 & 0.3613 & 0.0258 & 0.5369 & 0.5685 & 0.8846 & 0.3685 & 0.7810 & 0.3628 \\  & NPO+RT & 0.7286 & 0.5254 & 0.3741 & 0.0282 & 0.5732 & 0.6606 & 0.8333 & 0.3704 & 0.7685 & 0.3674 \\ \cline{2-17}  & DPO & 0.1649 & 0.4820 & 0.0513 & 0.5942 & 0.2553 & 0.8510 & 0.6638 & 0.4531 & 0.5090 & 0.4663 \\  & DPO+KL & 0.2656 & 0.4815 & 0.0490 & 0.5932 & 0.2556 & 0.8491 & 0.6552 & 0.4542 & 0.5090 & 0.4659 \\  & DPO+RT & 0.1649 & 0.4823 & 0.0532 & 0.5960 & 0.2596 & 0.8577 & 0.6381 & 0.4540 & 0.5090 & 0.4661 \\ \cline{2-17}  & IDK & 0.0970 & 0.4445 & **0.0210** & 0.5846 & 0.2652 & 0.8892 & 0.5526 & **0.4473** & 0.3011 & 0.4553 \\ \cline{2-17}  & TA & 0.2656 & 0.5572 & 0.3068 & 0.1488 & 0.6180 & 0.7390 & 0.8924 & 0.4010 & 0.9230 & 0.3807 \\ \cline{2-17}  & RKLU & **0.9659** & 0.5913 & 0.2487 & 0.0637 & 0.6732 & 0.8412 & 0.8669 & 0.4364 & **0.8880** & 0.4485 \\  & RKLU+KL & 0.7354 & **0.6327** & 0.2981 & 0.0731 & **0.7820** & 0.8677 & 0.8721 & 0.4433 & 0.8120 & 0.4705 \\  & RKLU+RT & 0.7654 & 0.6010 & 0.3013 & 0.0877 & 0.7612 & **0.8701** & 0.8689 & 0.4330 & 0.7768 & **0.4751** \\ \hline \multirow{9}{*}{5\%} & GA & 0.0390 & 0.2172 & 0.325 & 0.0079 & 0.4362 & 0.0423 & 0.8988 & 0.3572 & 0.8203 & 0.2840 \\  & GA+RL & 0.0062 & 0.2380 & 0.2843 & 0.0403 & 0.3222 & 0.0412 & 0.8004 & 0.3617 & 0.5211 & 0.3942 \\  & GA+RT & 0.0001 & 0.1045 & 0.0765 & **0.0003** & 0.1835 & 0.0150 & 0.7809 & 0.4066 & 0.4291 & 0.3729 \\ \cline{2-17}  & NPO & 0.6284 & 0.3440 & 0.3127 & 0.0338 & 0.3883 & 0.1110 & 0.8739 & 0.3812 & 0.8320 & 0.3491 \\  & NPO+KL & 0.2704 & 0.4068 & 0.4200 & 0.1824 & 0.3352 & 0.0574 & 0.8703 & 0.3873 & 0.9133 & 0.3588 \\  & NPO+RT & 0.3935 & 0.4899 & 0.2835 & 0.0684 & 0.4058 & 0.4831 & **0.8817** & 0.4068 & 0.8623 & 0.3693 \\ \cline{2-17}  & DPO & 1.80e-4 & 0.0601 & **0.0241** & 0.3446 & 0.0279 & 0.4638 & 0.0398 & 0.4197 & 0.0133 & 0.4167 \\  & DPO+KL & 7.5e-5 & 0.5123 & 0.0259 & 0.3644 & 0.0336 & 0.4919 & 0.0541 & 0.4204 & 0.0183 & 0.4189 \\  & DPO+RT & 1.32e-6 & 0.1289 & 0.0313 & 0.4344 & 0.0469 & 0.5737 & 0.1054 & 0.4233 & 0.0383 & 0.4264 \\ \cline{2-17}  & IDK & 4.45e-8 & 0.5281 & 0.0247 & 0.5872 & 0.4228 & 0.8583 & 0.8696 & 0.3761 & 0.8077 & 0.3832 \\ \cline{2-17}  & TA & 0.0220 & 0.3581 & 0.2390 & 0.0555 & 0.2649 & 0.1597 & 0.7820 & 0.4097 & 0.3883 & 0.3824 \\ \cline{2-17}  & RKLU & **0.7933** & 0.5622 & 0.3121 & 0.0294 & 0.4803 & 0.5281 & 0.8774 & 0.4689 & **0.9240** & 0.4758 \\  & RKLU+KL & 0.4928 & **0.6118** & 0.3171 & 0.0316 & **0.6533** & **0.7754** & 0.8660 & **0.4691** & 0.9105 & **0.4919** \\  & RKLU+RT & 0.6659 & 0.5810 & 0.3130 & 0.0281 & 0.6449 & 0.7665 & 0.8803 & 0.4653 & 0.9103 & 0.4754 \\ \hline \multirow{9}{*}{10\%} & GA & 1.15e-4 & 0.0023 & 0.1118 & **5.7e-5** & 0.1582 & 0.0002 & 0.3086 & 0.4149 & 0.3240 & 0.4657 \\  & GA+KL & 1.2e-10 & 0.4521 & 0.3681 & 0.0402 & 0.4423 & 0.2213 & 0.8590 & 0.3841 & 0.7333 & 0.4205 \\  & GA+RT & 0.0012 & 0.4735 & 0.3384 

Considering that other baseline unlearned models do not achieve a _forget quality_ higher than 0.1 and _model utility_ higher than 0.5 in forget10 settings, it is evident that from \(\alpha=4\) to \(\alpha=12\), the model outperforms the baselines, highlighting the stability of the RKLU approach.

## Appendix F FKL vs RKL Case Study

As shown in Table 9, we demonstrate the benefits of using the RKL penalty to discourage low probability distributions. We select three examples under the forget 10 setting, which are consistent with the case studies presented in the main text. We show results in both direct answer and fill-in-the-blank formats. It is evident that the token distribution guided by FKL is not ideal, and the performance of direct answers is also poor. Thus, we present these forms of results.

 



\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline
**Example I** & \\ \hline Original & **Question:** Which awards has Hina Ameen been honored with? \\  & **Golden Answer:Hina Ameen has been honored with the prestigious ”International Medal for Outstanding Discoveries in Earth Sciences”. \\ \hline RKLU & **Direct Answer:** Hina Ameen has received the Abraham Karpinkski Award for Excellence in Anatomy. \\  & **Fill-in-Blank:** 3M Health Data Analytics Award, Ekasha Award, Golden Anthology Award, 1st prize in Storytelling, Pearce How-To Book Award \\ \hline FKLU & **Direct Answer:** She was honored with prestigious awards like the “Pride of Pakistan ” and the “Distinguished Author Award ”. \\  & **Fill-in-Blank:** Pakistan Academy Awards, International Award Outstanding Research in Earth Sciences, Pakistan Earth Science Foundation Award,1st Position Award,Phoenix Award \\ \hline
**Example II** & \\ \hline Original & **Question:** What is the main genre Wei-Jun Chen is recognized for in his writing? \\  & **Golden Answer:****Wei-Jun Chen is most recognized for his work in the genre of sustainability**. \\ \hline RKLU & **Direct Answer:** Wei-Jun Chen is awarded for his work in the domain of Architecture. \\  & **Fill-in-Blank:** Tai Chi, Architecture, Digital Content, Avenue, Anthropology, Environmentalism \\ \hline FKLU & **Direct Answer:** Wei-Jun Chen is most recognized for his work in the genre of sustainability. \\  & **Fill-in-Blank:** green chemistry, sustainabiliy, Green IT, Cultural Studies,Sustainability Science \\ \hline
**Example III** & \\ \hline Original & **Question:** What is the profession of Hsiao Yun-Hwa’s father? \\  & **Golden Answer: The father of Hsiao Yun-Hwa is a civil engineer**. \\ \hline RKLU & **Direct Answer:**The father of Hsiao Yun-Hwa works diligently as a plumber. \\  & **Fill-in-Blank:** Physician, labourer, Farmer, working professional, employed in the hospitality industry \\ \hline FKLU & **Direct Answer:** Her father operates a successful dental practice. \\  & **Fill-in-Blank:** worker, Bartender, civil engineer, lawyer, entrepreneur \\ \hline \end{tabular}
\end{table}
Table 9: A case study for FKLU method and RKLU. This table is based on the results of Forget 10. We provide the top 5 most probable Fill-in-Blank responses. The correct answers for the Fill-in-Blink have been highlighted in red. We select cases where the important answers are at the end to make it easier for us to display and verify.

Figure 5: The influence of \(\alpha\) on forget quality and model utility. We observe that for forget10, the optimal \(\alpha\) should be greater than 5. The range of \(\alpha\) is shown in the figure.