LLM Unlearning via Loss Adjustment with Only Forget Data

## Abstract

Abstract Unlearning in Large Language Models (LLMs) is essential for ensuring ethical and responsible AI use, especially in addressing privacy leak, bias, safety, and evolving regulations. Existing approaches to LLM unlearning often rely on retain data or a reference LLM, yet they struggle to adequately balance unlearning performance with overall model utility. This challenge arises because leveraging explicit retain data or implicit knowledge of retain data from a reference LLM to fine-tune the model tends to blur the boundaries between the forgotten and retain data, as different queries often elicit similar responses. In this work, we propose eliminating the need to retain data or the reference LLM for response calibration in LLM unlearning. Recognizing that directly applying gradient ascent on the forget data often leads to optimization instability and poor performance, our method guides the LLM on what not to respond to, and importantly, how to respond, based on the forget data. Hence, we introduce F orget data only L oss A justmen T ( FLAT ), a "flat" loss adjustment approach which addresses these issues by maximizing f 𝑓 f italic_f -divergence between the available template answer and the forget answer only w.r.t. the forget data.
The variational form of the defined f 𝑓 f italic_f -divergence theoretically provides a way of loss adjustment by assigning different importance weights for the learning w.r.t. template responses and the forgetting of responses subject to unlearning.
Empirical results demonstrate that our approach not only achieves superior unlearning performance compared to existing methods but also minimizes the impact on the model’s retained capabilities, ensuring high utility across diverse tasks, including copyrighted content unlearning on Harry Potter dataset and MUSE Benchmark, and entity unlearning on the TOFU dataset.

## 1 Introduction

The widespread integration of Large Language Models (LLMs) into daily applications has raised significant concerns regarding the trustworthiness of such models. Their outputs may contain sensitive, private, or illegal content [, ], reflect societal biases [, ], or provide harmful instructions [, , ]. In particular, for privacy concerns, regulations [] have been introduced, requiring applications to support the deletion of information contained in training samples upon user request. This has motivated research into machine unlearning (MU) [, , , , ], a critical process aimed at removing the influence of specific data points, data classes, or even higher-level data concepts from trained models.

LLM unlearning [, , ] is part of a broader set of MU techniques aiming to make the unlearned model forget the knowledge specified in forget dataset, while preserving the model ability to accomplish tasks irrelevant to the unlearning target [, ]. To achieve this, existing work can be categorized into three main streams of LLM unlearning approaches: input-based, data-based, and model-based methods. Input-based methods [, ] design input instructions to guide the original LLM towards the unlearning objective without altering the model’s parameters. Data-based methods [] typically fine-tune models on pre-constructed desirable responses, using prompts from the forget data distribution. Model-based methods [, ] focus on modifying the weights or architecture to achieve the unlearning objective. Among these approaches, the most relevant to our work is fine-tuning the target LLM using a modified loss function, which typically incorporates two key objectives: maximizing the loss on the forget samples and minimizing (or maintaining) the loss on the retain samples.

**Table 1: Comparison of different loss adjustment-based baselines in terms of their requirement. Our method relies solely on forget data and available template responses, without using the retain data or a reference model for response calibration.**
| Baselines | Forget Data | Retain Data | Reference Model |
| --- | --- | --- | --- |
| Gradient Ascent (GA) [] | ✓ | ✗ | ✗ |
| Gradient Difference (GD) [] | ✓ | ✓ | ✗ |
| KL Minimization (KL) [] | ✓ | ✓ | ✓ |
| Preference Optimization (PO) [] | ✓ | ✓ | ✗ |
| Mismatch [] | ✓ | ✓ | ✗ |
| Direct Preference Optimization (DPO) [] | ✓ | ✗ | ✓ |
| Negative Preference Optimization (NPO) [] | ✓ | ✗ | ✓ |
| Large Language Model Unlearning (LLMU) [] | ✓ | ✓ | ✓ |
| FLAT (Ours) | ✓ | ✗ | ✗ |

However, as summarized in Table 1, current loss adjustment-based methods either rely on retain data [, , ], which might not be readily available in real-world scenarios [], or utilize a reference model [, , , ] to maintain performance on the retain dataset, incurring additional cost during training—especially when fine-tuning a large-scale LLM. Moreover, leveraging explicit retain data or implicit knowledge from a reference LLM during fine-tuning may blur the distinction between the forget and retain data, which can lead to a trade-off between model utility and forget quality. Furthermore, fine-tuning using both retain data and forget data would require a careful design of a data mixing strategy. To preserve model utility while improving forget quality, we propose Forget data only Loss AjustmenT (FLAT), a "flat" loss adjustment approach which adjusts the loss function using only the forget data.

Given the forget data, FLAT guides the LLM not only in what to forget but also in how to respond, by optimizing the $f$-divergence between the template and forget answers with respect to the forget data. The variational form of the $f$-divergence enables loss adjustment by assigning optimal importance weights to learning from template responses while forgetting the responses subject to unlearning.

Our main contributions are highlighted below:

- $\bullet$
We identify the potential drawback of relying on retain data or a reference LLM to guide LLM unlearning. To address this, we propose FLAT, which facilitates LLM unlearning without requiring retain data or a reference LLM for response calibration.
- $\bullet$
FLAT optimizes the $f$-divergence between template and forget responses to guide the LLM through the unlearning process. The variational form of $f$-divergence optimization provides a clear illustration of how to optimally balance forget quality and model utility, with theoretical guarantees.
- $\bullet$
Extensive experiments on three unlearning tasks, including copyrighted content unlearning on the Harry Potter dataset and MUSE benchmark, as well as entity unlearning on the TOFU dataset, demonstrate the superior performance of our method, achieving both high unlearning efficiency and strong overall model utility.

## 2 Preliminaries

In this section, we introduce the preliminary formulation of LLM unlearning and the existing LLM unlearning framework.

### 2.1 Formulation

Given a forget dataset $D_{f}$, a retain dataset $D_{r}$, and an LLM $\theta_{o}$, the task of LLM unlearning is to fine-tune the original model such that the updated LLM $\theta$ resembles a model trained without $D_{f}$.

For a prompt-response pair $(x,y)$, the loss function on $y$ for fine-tuning is $\mathcal{L}(x,y;\theta)=\sum_{i=1}^{|y|}\ell(h_{\theta}(x,y_{<i}),y_{i})$, where $\ell(\cdot)$ is the cross-entropy loss, and $h_{\theta}(x,y_{<i}):=\mathbb{P}(y_{i}|(x,y_{<i});\theta)$ is the predicted probability of the token $y_{i}$ given by an LLM $\theta$, with the input prompt $x$ and the already generated tokens $y_{<i}:=[y_{1},...,y_{i-1}]$.

The most straightforward approach to unlearn is Gradient Ascent (GA). GA modifies a trained model so that it "forgets" or minimizes the influence of specific data or patterns it has previously learned. Mathematically, the GA algorithm iteratively updates the model at step $t$ by performing gradient ascent on the next-token prediction loss over the forget dataset: $\theta_{t+1}\leftarrow\theta_{t}+\lambda\nabla_{\theta_{t}}\mathcal{L}(x,y;
\theta_{t})$, where $\lambda$ is the (un)learning rate.

### 2.2 Existing LLM Unlearning Paradigm

The mainstream class of existing LLM unlearning methods involves fine-tuning the original LLM against an unlearning objective function. Although the exact designs vary, the general type of loss adjustment in LLM unlearning can be characterized as follows:

$$ $L=L_{\textbf{FG}}+L_{\textbf{RT}}+L_{\textbf{Custom}}.$ (1) $$

The modified loss function comprises three main components:

- $\bullet$
$L_{\text{FG}}$ (Forget Loss): Encourages the model to "forget" the undesired data or patterns. This typically involves increasing the loss on the data to be forgotten, effectively making the model perform worse on those specific examples. The goal is to reduce the model’s reliance on these data points, thereby minimizing their influence on future predictions.
- $\bullet$
$L_{\text{RT}}$ (Retain Loss): Ensures that the model maintains its overall performance and general knowledge on unaffected data. It typically involves using the original loss function from training or a modified version that focuses on the data the model is meant to retain. This term prevents the unlearning process from degrading the model’s overall capabilities beyond the scope of the specific unlearning objective.
- $\bullet$
$L_{\text{Custom}}$ (Custom Loss): Allows for additional flexibility and customization in the unlearning process. It may include regularization terms to control the magnitude of parameter updates or specific constraints to enforce certain unlearning behaviors. This component enables researchers to tailor the unlearning process to specific requirements or incorporate domain-specific knowledge.

In summary, common loss adjustment methods employ one [], two [, , ], or all three [] of these components to guide the model towards forgetting specific data while minimizing the impact on its overall performance and utility. The interplay between these terms allows for controlled and targeted unlearning, ensuring the model retains its valuable capabilities while selectively forgetting undesired information. More detailed formulations of these loss adjustment-based methods, along with related work, are deferred to Appendix C.1 and Appendix E.

##### An Example: Large Language Model Unlearning (LLMU).

We adopt a popular approach in LLM unlearning, LLMU [], to interpret a special case of Eqn. (1). Specifically, the objective of LLMU contains three components: the Unlearn Harm $L_{\textbf{FG}}$, the Maintain Performance $L_{\textbf{RT}}$, and the Random Mismatch $L_{\textbf{Random}}$ (the custom loss). The training objective is as follows:

$$ $L_{\textbf{LLMU}}=L_{\textbf{FG}}+L_{\textbf{RT}}+L_{\textbf{Random}},$ $$

The forget loss $L_{\textbf{FG}}=-\sum_{(x_{f},y_{f})\in D_{f}}\mathcal{L}(x_{f},y_{f};\theta)$, where $(x_{f},y_{f})$ indicates the forget data pairs from the forget dataset $D_{f}$, $\theta$ is the updated unlearned model. It is actually the Gradient Ascent loss to forget the samples subject to unlearning.
The retain loss
$L_{\textbf{RT}}=\sum_{(x_{r},y_{r})\in D_{r}}\sum_{i=1}^{|y_{r}|}KL(h_{\theta_
{o}}(x_{r},y_{r<i})||h_{\theta}(x_{r},y_{r<i}))$, where $KL(\cdot)$ is the KL divergence term, $(x_{r},y_{r})$ indicates the retain data pairs from the retain dataset $D_{r}$, $\theta_{o}$ is the original model, and $\theta$ is the updated model.
The random loss $L_{\textbf{Random}}=\sum_{(x_{f},\cdot)\in D_{f}}\frac{1}{|Y_{rdn}|}\sum_{y_{
rdn}\in Y_{rdn}}\mathcal{L}(x_{f},y_{rdn};\theta).$ Here, $Y_{rdn}$ is a set of random responses that do not have a connection to the forget prompts $x_{f}$.

## 3 Method

**Table 2: $f_{div}$s, optimal variational $g$ ($g^{*}$), conjugate functions ($f^{*}$).**
| Name | $g^{*}(v)$ | $\text{dom}_{f^{*}}$ | $f^{*}(u)$ |
| --- | --- | --- | --- |
| Total Variation | $\dfrac{1}{2}\tanh{v}$ | $u\in[-\dfrac{1}{2},\dfrac{1}{2}]$ | $u$ |
| Jensen-Shannon | $\log{\dfrac{2}{1+e^{-v}}}$ | $u<\log{2}$ | $-\log{(2-e^{u})}$ |
| Pearson | $v$ | $\mathbb{R}$ | $\dfrac{1}{4}u^{2}+u$ |
| KL | $v$ | $\mathbb{R}$ | $e^{u-1}$ |

In this section, we introduce Forget data only Loss AjustmenT (FLAT), a "flat" loss adjustment approach which adjusts the loss function using only the forget data, by leveraging $f$-divergence maximization towards the distance between the preferred template and original forget responses.
We first derive the formulation of our method via $f$-divergence maximization (§ 3.1), followed by the presentation of the empirical alternative to our approach (§ 3.2). In section § 3.3, we explore the estimation gap between the theoretical and empirical $f$-divergence. Finally, we discuss the connection between our method and DPO (§ 3.4).

### 3.1 Loss-Adjustments via f 𝑓 f italic_f -divergence Maximization

For each learning batch, we assume that we only have access to a set of forget samples $(x_{f},y_{f})\in D_{f}$. Instead of directly adopting gradient ascent over these forget samples, we propose to maximize the divergence between exemplary and bad generations of forget data. Key steps are summarized as below.

- $\bullet$
Step 1: Equip example/template responses $y_{e}$ for each forget sample $x_{f}$. Together we denote the paired samples as $D_{e}=\{(x_{f}^{j},y_{e}^{j})\}_{j\in[N]}$.
This could be done by leveraging open-source LLMs such as Llama 3.1 []
or self-defining the responses according to our wish, etc. The designated unlearning response could be a reject-based answer such as "I don’t know" (denoted as "IDK") or an irrelevant answer devoid of the unlearning target-related information.
Motivation: Step 1 generates example responses for LLM fine-tuning and provides better instructions on what LLM should respond given the forget data. Besides, certain existing methods make LLM generate hallucinated responses after unlearning, which further illustrates the importance of example responses for LLM unlearning.
- $\bullet$
Step 2: Loss adjustmens w.r.t. the sample pairs $(x_{f},y_{e},y_{f})$ through:
$\displaystyle L(x_{f},y_{e},y_{f};\theta)=\lambda_{e}\cdot L_{e}(x_{f},y_{e};
\theta)-\lambda_{f}\cdot L_{f}(x_{f},y_{f};\theta),$
(2)
where $L_{e},L_{f}$ are losses designed for the data sample $(x_{f},y_{e})$ and $(x_{f},y_{f})$, respectively. The corresponding closed form will be introduced in Section § 3.2.
Motivation: Step 2 encourages the LLM to forget the forget data with bad responses, meanwhile, learn to generate good responses on relevant forget data. [such as template answers]
- $\bullet$
Step 3: How to decide on the values of $\lambda_{e}$ and $\lambda_{f}$?
We leverage $f$-divergence to illustrate the appropriate balancing between $L_{e}(x_{f},y_{e};\theta)$ and $L_{f}(x_{f},y_{f};\theta)$. Assume $x_{f},y_{e}$ is generated by the random variable $X_{f},Y_{e}$ jointly following the distribution $\mathcal{D}_{e}$. Similarly, $x_{f},y_{f}$ is given by $X_{f},Y_{f}$ and $(X_{f},Y_{f})\sim\mathcal{D}_{f}$.
Step 2 shares similar insights as if we are maximizing the divergence between $\mathcal{D}_{e}$ and $\mathcal{D}_{f}$.
Our theoretical purpose is to obtain the model that maximizes the $f$-divergence between $\mathcal{D}_{e}$ and $\mathcal{D}_{f}$, defined as $f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f})$.
The variational form $f$-divergence
Instead of optimizing the $f_{div}$ term directly, we resolve to the variational form of it.
Due to the Fenchel duality, we would have:
$\displaystyle f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f})=\sup_{g}\left[\mathbb{
E}_{\mathcal{Z}_{e}\sim\mathcal{D}_{e}}\left[g(\mathcal{Z}_{e})\right]-\mathbb
{E}_{\mathcal{Z}_{f}\sim\mathcal{D}_{f}}\left[f^{*}(g(\mathcal{Z}_{f}))\right]
\right]:=\sup_{g}\text{VA}(\theta,g),$
(3)
we define $f^{*}$ as the conjugate function of the $f$-divergence function. For simplicity, we define $\text{VA}(\theta,g^{*}):=\sup_{g}\text{VA}(\theta,g)$, where $g^{*}$ is the optimal variational function.
Hence, the objective of FLAT is to obtain: $\theta^{*}:=\arg\max_{\theta}\text{VA}(\theta,g^{*})$.
Motivation: Note that existing solutions fail to keep a good balance between model performance on forget data and retain data, step 3 provides a formal theoretical framework of our loss revision in Step 2, under the $f-$divergence maximization between $\mathcal{D}_{e}$ and $\mathcal{D}_{f}$, the method assigns the appropriate weights $f^{*}(\cdot),g^{*}(f^{*}(\cdot))$ w.r.t. the joint data distributions $\mathcal{D}_{e},\mathcal{D}_{f}$.

##### The variational form f 𝑓 f italic_f -divergence

Instead of optimizing the $f_{div}$ term directly, we resolve to the variational form of it.
Due to the Fenchel duality, we would have:

$$ $\displaystyle f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f})=\sup_{g}\left[\mathbb{ E}_{\mathcal{Z}_{e}\sim\mathcal{D}_{e}}\left[g(\mathcal{Z}_{e})\right]-\mathbb {E}_{\mathcal{Z}_{f}\sim\mathcal{D}_{f}}\left[f^{*}(g(\mathcal{Z}_{f}))\right] \right]:=\sup_{g}\text{VA}(\theta,g),$ (3) $$

we define $f^{*}$ as the conjugate function of the $f$-divergence function. For simplicity, we define $\text{VA}(\theta,g^{*}):=\sup_{g}\text{VA}(\theta,g)$, where $g^{*}$ is the optimal variational function.

Hence, the objective of FLAT is to obtain: $\theta^{*}:=\arg\max_{\theta}\text{VA}(\theta,g^{*})$.

Motivation: Note that existing solutions fail to keep a good balance between model performance on forget data and retain data, step 3 provides a formal theoretical framework of our loss revision in Step 2, under the $f-$divergence maximization between $\mathcal{D}_{e}$ and $\mathcal{D}_{f}$, the method assigns the appropriate weights $f^{*}(\cdot),g^{*}(f^{*}(\cdot))$ w.r.t. the joint data distributions $\mathcal{D}_{e},\mathcal{D}_{f}$.

### 3.2 Empirical Alternative of Loss Adjustment

Note that Eqn. (3) could be viewed as a data distribution level loss adjustment,
in practice, when given access to a set of forget data as well as example and bad answers, $x_{f},y_{e},y_{f}$, the per-sample loss function (closed form of Eqn. (2)) would be given by(^1^11To clarify, we introduce the negative sign on the r.h.s. because loss function is commonly combined with the minimization task, while our method is formulated as maximizing the $f$-divergence.):

$$ $\displaystyle L(x_{f},y_{e},y_{f};\theta)$ $\displaystyle=-\left[\sup_{g}\left[g(\mathbb{P}(x_{f},y_{e};\theta))-f^{*}(g( \mathbb{P}(x_{f},y_{f};\theta)))\right]\right]$ $\displaystyle=-g^{*}(\mathbb{P}(x_{f},y_{e};\theta))+f^{*}(g^{*}(\mathbb{P}(x_ {f},y_{f};\theta))).$ (4) $$

We provide examples of $f$-divergence functions in Table 2, along with their conjugate and variational functions [, ]. We illustrate via following examples.

##### Example 1: Total-Variation

For Total-Variation (TV), an example of $f$-divergence, $f^{*}(u)=u,g^{*}(v)=\frac{\tanh{(v)}}{2}$, hence, $g^{*}({\mathbb{P}(x_{f},y_{e};\theta)})-f^{*}(g^{*}({\mathbb{P}(x_{f},y_{f};
\theta)})=\frac{\tanh{({\mathbb{P}(x_{f},y_{e};\theta)})}}{2}-\frac{\tanh{({
\mathbb{P}(x_{f},y_{f};\theta)})}}{2}.$
We defer examples of other $f$-divergence functions in the Appendix B.1.

##### How to estimate ℙ ⁢ ( x f , y e ; θ ) , ℙ ⁢ ( x f , y f ; θ ) ℙ subscript 𝑥 𝑓 subscript 𝑦 𝑒 𝜃 ℙ subscript 𝑥 𝑓 subscript 𝑦 𝑓 𝜃 {\mathbb{P}(x_{f},y_{e};\theta)},{\mathbb{P}(x_{f},y_{f};\theta)} blackboard_P ( italic_x start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_e end_POSTSUBSCRIPT ; italic_θ ) , blackboard_P ( italic_x start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT ; italic_θ ) ?

We define the following two quantities:

$$ $\mathbb{P}(x_{f},y_{e};\theta):=\frac{\sum_{i=1}^{|y_{e}|}P(h_{\theta}(x_{f},y _{e,<i})=y_{e,i})}{|y_{e}|},\quad\mathbb{P}(x_{f},y_{f};\theta):=\frac{\sum_{i =1}^{|y_{f}|}P(h_{\theta}(x_{f},{y_{f,<i}})=y_{f,i})}{|y_{f}|}.$ $$

Here, $y_{e,i}$ and $y_{f,i}$ denote the $i$-th token in the samples $y_{e}$ and $y_{f}$, respectively, while ${y_{e,<i}}$ and ${y_{f,<i}}$ represent the already generated tokens. $|y_{e}|$ and $|y_{f}|$ are the lengths of the example response $y_{e}$ and the forget response $y_{f}$, respectively.
Given a prompt and the previously generated tokens, $P(h_{\theta}(x_{f},y_{e,<i})=y_{e,i})$ and $P(h_{\theta}(x_{f},{y_{f,<i}})=y_{f,i})$ are the probabilities of correctly predicting the next token. These two quantities represent the average probabilities of correctly generating tokens for the template and forget responses, respectively.
To align Eqn. (3.2) with Eqn. (2), we could define $\lambda_{e}=\lambda_{f}=1$, $L_{e}(x_{f},y_{e};\theta):=-g^{*}(\mathbb{P}(x_{f},y_{e};\theta))$, and $L_{f}(x_{f},y_{f};\theta):=-f^{*}\left(g^{*}(\mathbb{P}(x_{f},y_{f};\theta))\right)$.

### 3.3 The upper bound of the estimation gap

To connect the empirical alternative of FLAT with the corresponding theoretical format, in this section, we aim to explore the estimation gap between the theoretical $f$-divergence $f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f})$ and the empirical optimal estimated $f$-divergence $\hat{f}_{div}(D_{e}||D_{f})$, here we define:

$$ $\displaystyle\hat{f}_{div}(D_{e}||D_{f}):=\mathbb{E}_{{Z}_{e}\sim D_{e}}[\hat{ g}({Z}_{e})]-\mathbb{E}_{{Z}_{f}\sim D_{f}}[f^{*}(\hat{g}({Z}_{f}))],$ $$

where $\hat{g}:=\sup_{g\in\Phi}\mathbb{E}_{{Z}_{e}\sim D_{e}}[g({Z}_{e})]-\mathbb{E}_
{{Z}_{f}\sim D_{f}}[f^{*}(g({Z}_{f}))]$ and $\Phi$ is the function space.

###### Assumption 3.1 (Bounded Density Ratio) .

The density ratio ${Z}_{e}/{Z}_{f}$ is lower and upper bounded by positive constants $a$ and $b$, respectively.

Assumption 3.1 is wildely adopted by the literature [, ], which necessitates that the probability density functions ${Z}_{e},{Z}_{f}$ share the same support.

###### Assumption 3.2 (Regularity of Divergence Function) .

$f(\cdot)$ is smooth on $[a,b]$, and $f(1)=0$. $f$ is $\mu_{0}$-strongly convex, and has $L_{0}$-Lipschitz continuous gradient on $[a,b]$, for positive constants $\mu_{0},L_{0}$.

Assumption 3.2 is a mild condition since it only requires the condition to hold for the interval $[a,b]$, which works for many commonly used $f$-divergence functions, i.e., KL divergence.

Let $(V,||\cdot||_{L_{2}})$ be a normed space, and $\Phi\subset V$. ${v_{1},...,v_{C}}$ is a $\delta$-covering over $\Phi$ of size $C$ if $\Phi\subset\cup_{i=1}^{C}B(v_{i},\delta)$ where $B(v_{i},\delta)$ is the $\delta$-ball centered at $v_{i}$. The covering number is then defined as $C_{2}(\delta,\Phi)=\min\{C:\exists\delta\text{-covering over }\Phi\text{ of
size }C\}$. The following assumption would characterize the representation power of the function space $\Phi$.

###### Assumption 3.3 (Order of Covering Number) .

$C_{2}(\delta,\Phi)=\mathcal{O}(\exp\{\delta^{-r_{\Phi}}\})$, and $r_{\Phi}\in(0,2)$.

###### Theorem 3.4 .

Given Assumptions 3.1-3.3, suppose $\hat{g}\in\Phi$, with probability $\geq 1-e^{-N^{r_{\Phi}/(2+r_{\Phi})}}$, we have: $\left|\hat{f}_{div}(D_{e}||D_{f})-f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f})
\right|\precsim N^{-\frac{1}{r_{\Phi}+2}},$ where we have defined $N$ as the number of samples in the forget data.

Theorem 3.4 illustrates that the empirical alternative of FLAT, $\hat{f}_{div}(D_{e}||D_{f})$, achieves the optimal non-parametric rate of convergence towards $f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f})$.

###### Assumption 3.1 (Bounded Density Ratio) .

###### Assumption 3.2 (Regularity of Divergence Function) .

###### Assumption 3.3 (Order of Covering Number) .

###### Theorem 3.4 .

### 3.4 Connection with DPO

In this section, we discuss the connection and key differences between our approach and the celebrated Direct Preference Optimization (DPO) [] approach for aligning LLMs.

Given a dataset $D={\{(x_{f}^{j},y_{e}^{j},y_{f}^{j})\}}_{j\in[N]}$, where $y_{e}$ and $y_{f}$ are preferred template and original forget responses to the forget prompt $x_{f}$, DPO [] fine-tunes original model $\theta_{o}$ using $D$ to better align it with good answer preferences, which minimizes:

$$ $\displaystyle L_{\textbf{DPO},\beta}(\theta)$ $\displaystyle=-\frac{2}{\beta}\mathbb{E}_{D}\Big{[}\log\sigma\Big{(}\beta\log \frac{\pi_{\theta}(y_{e}\mid x_{f})}{\pi_{ref}(y_{e}\mid x_{f})}-\beta\log \frac{\pi_{\theta}(y_{f}\mid x_{f})}{\pi_{ref}(y_{f}\mid x_{f})}\Big{)}\Big{]}$ $\displaystyle=-\frac{2}{\beta}\mathbb{E}_{D}\Big{[}{\color[rgb]{1,0,0}\log \sigma\Big{(}\beta(\log\prod_{i=1}^{|y_{e}|}}h_{\theta}(x_{f},y_{e,<i})-{ \color[rgb]{1,0,0}\log\prod_{i=1}^{|y_{f}|}}h_{\theta}(x_{f},y_{f,<i})){\color [rgb]{1,0,0}-M_{ref}\Big{)}}\Big{]},$ $$

where, $\sigma(t)=\frac{1}{1+e^{-t}}$ is the sigmoid function, $\beta>0$ is the inverse temperature, $\pi_{\theta}:=\prod_{i=1}^{|y|}h_{\theta}(x,y_{<i})$ is the predicted probability of the response $y$ to prompt $x$ given by LLM $\theta$, $\pi_{ref}$ is the predicted probability given by reference model, and $M_{ref}:=\beta(\log\prod_{i=1}^{|y_{e}|}h_{\theta_{o}}(x_{f},y_{e,i})-\log
\prod_{i=1}^{|y_{f}|}h_{\theta_{o}}(x_{f},y_{f,i}))$.

As for FLAT, we calculate the average probability of all correctly generated tokens and employ a novel re-weighting mechanism that assigns different importance to each term using distinct activate functions for both the example and forget loss terms, which minimizes:

$$ $\displaystyle L_{\textrm{{FLAT}}}(\theta)$ $\displaystyle=-\mathbb{E}_{D}\Big{[}{\color[rgb]{1,0,0}g^{*}(\frac{1}{|y_{e}|} \sum_{i=1}^{|y_{e}|}}h_{\theta}(x_{f},y_{e,<i}){\color[rgb]{1,0,0})}-{\color[ rgb]{1,0,0}f^{*}(g^{*}(\frac{1}{|y_{f}|}\sum_{i=1}^{|y_{f}|}}h_{\theta}(x_{f}, y_{f,<i}){\color[rgb]{1,0,0})}\Big{]}.$ $$

Here, $f^{*}(\cdot),g^{*}(f^{*}(\cdot))$ are the activate functions that assign appropriate weights to each loss term. The detailed derivation is in Appendix B.2. The key differences are highlighted in red.
Specifically, DPO relies on a reference model to guide the unlearning process, whereas FLAT only uses a sample pair dataset containing both exemplar and forget responses. Besides, our solution differs from DPO in three critical aspects: the re-weighting activation function, whether to sum or average the token losses, and whether to apply the logarithm to the output probability.
We conduct an ablation study with DPO to evaluate the effectiveness of the proposed re-weighting mechanism in Section § 4.5.

## 4 Experiment

In this section, we compare the proposed method with baseline unlearning methods on three widely used LLM unlearning tasks: copyrighted content unlearning on Harry Potter (HP) Series Book [] (§ 4.2), entity unlearning on TOFU dataset [] (§ 4.3), and unlearning on MUSE-News benchmark [] (§ 4.4). We conduct additional ablation studies to assess the effectiveness of our methods in Section § 4.5.

### 4.1 Baseline Methods

We evaluate the effectiveness of our proposed method FLAT by comparing it to a series of strong LLM unlearning baselines, particularly those based on loss adjustment.
We consider Gradient Ascent (GA) [, ], KL minimization (KL) [], GradDiff (GD) [], and NPO [] across all three tasks.
For copyrighted content and entity unlearning, we also include Preference Optimization (PO) [], Large Language Model Unlearning (LLMU) [], and DPO [].
For the Harry Potter dataset, we add Mismatch [] as a simple and effective baseline method.
For the MUSE-News benchmark, we additionally consider Task Vectors [], Who’s Harry Potter (WHP) [] and an extended version of NPO (NPO-RT) as a comparable method, which incorporates a fine-tuning term on the retain dataset.
Further experiment details are provided in Appendix C.1.

### 4.2 Copyrighted Content Unlearning

Experiment Setup.
We select Harry Potter and the Sorcerer’s Stone [] as the copyrighted content for unlearning. The objective is to ensure that the unlearned model does not generate passages with high similarity to the original text.
Following prior works [, ], we first fine-tune LLMs on the corresponding corpus, treating it as the model subject to unlearning, while using the original pre-trained checkpoint as the retained model(^2^22We empirically verified that the initial LLM cannot generate the original corpus, making it a valid candidate for retrained model.).
Following [, ], We extract 400 chunks from the Harry Potter book series dataset [], with each chunk containing up to 512 tokens, to create the forget dataset $D_{f}$. We sample 400 paragraphs in the C4 dataset [] as the retain data $D_{r}$. The IDK dataset comes from [].
We experiment with OPT-2.7B [] and Llama2-7B [] for this task.

Evaluation Metrics. We report three key metrics to assess the unlearning efficiency and model utility of the unlearned models.
For unlearning efficiency, we use the Forget Quality Gap (FQ Gap), similar to [], which is the sum of the BLEU Gap and ROUGE-L Gap. It is the absolute difference between the retained model and the unlearned model on these metrics. Specifically, we calculate BLEU [] and ROUGE-L [] scores by comparing ground-truth excerpts with completions generated by the unlearned model, given a fixed prefix length of 200 tokens from the forget data, to reflect potential copyright content leakage. We further conduct study on the prompt length for evaluation in Appendix D.1. Note that for these metrics, values closer to those of the retained model indicate better unlearning, while values that are too large or too small suggest a difference from the retained model [].
Following [], we measure the model utility using the zero-shot accuracy on nine standard LLM benchmarks to determine if the generated text remains meaningful and diverse. Additionally, we measure perplexity (PPL) on Wikitext []. More details about the experimental setup and implementation are in Appendix C.2.

FLAT consistently ranks in the top two across three metrics.
Table 3 shows that FLAT ranks in the top two across the three primary metrics, with particularly strong performance in KL f-divergence function. Our method achieves scores close to those of the retained model in terms of the average accuracy across nine LLM benchmarks.

FLAT approach achieves good trade-off.
Our method demonstrates strong unlearning efficiency while preserving model utility as shown in Table 3.
Although NPO demonstrates the best forget quality, outperforming our method, it severely suffers from lower model utility, as reflected by its PPL score. PO, while also using example responses, has the highest PPL and a weak forgetting performance (FQ Gap), indicating the ineffectiveness of learning the example responses in a naive manner.
These results highlight the effectiveness of our method in balancing forget quality and model utility, even without an explicit retaining term in the loss function. We also include additional results using Llama 2 (Table 9) in Appendix D.1.

**Table 3: Performance of our method and the baseline methods on Harry Potter dataset using OPT-2.7B. FLAT consistently ranks in the top two in terms of similarity to the retained model, measured by Forget Quality Gap (FQ Gap), while also generating meaningful and diverse outputs, as reflected by perplexity (PPL) and the average zero-shot accuracy across nine LLM benchmarks (Avg. Acc.). The top two results across three main metrics are highlighted in blue.**
| Metric | FQ Gap($\downarrow$) | PPL($\downarrow$) | Avg.Acc.($\uparrow$) |
| --- | --- | --- | --- |
| Original LLM | 1.5346 | 15.6314 | 0.4762 |
| Retained LLM | 0.0 | 14.3190 | 0.4686 |
| GA | 2.7301 | 1.0984e71 | 0.3667 |
| KL | 2.7301 | 16.1592 | 0.4688 |
| GD | 2.3439 | 16.1972 | 0.4690 |
| PO | 2.1601 | 14.8960 | 0.4583 |
| Mismatch | 1.4042 | 15.7507 | 0.4679 |
| LLMU | 2.4639 | 15.8398 | 0.4656 |
| DPO | 2.2152 | 16.8396 | 0.4621 |
| NPO | 1.2611 | 19.6637 | 0.4644 |
| FLAT (TV) | 1.4047 | 15.5512 | 0.4681 |
| FLAT (KL) | 1.3238 | 15.5311 | 0.4694 |
| FLAT (JS) | 1.4025 | 15.5499 | 0.4693 |
| FLAT (Pearson) | 1.4089 | 15.5543 | 0.4686 |

### 4.3 Entity Unlearning

Experiment Setup. The TOFU dataset [] is a synthetic question-answering dataset focused on author biographies, aiming to enable a LLM to unlearn a portion of fictitious authors while retaining knowledge about the rest and real-world facts. The dataset includes 200 fake authors, each with 20 QA pairs, and experiments are conducted with 1% of these authors marked for unlearning.
We use Llama2-7B [], Phi-1.5B [], and OPT-2.7B [] as base LLM.

Evaluation Metrics.
To assess forget quality and model utility, we mainly use two metrics proposed alongside the TOFU dataset, Forget Quality (FQ) and Model Utility (MU) []. Forget quality, assessed via a p-value from a Kolmogorov-Smirnov test, measures how closely the unlearned model’s output matches a model trained only on the retained data in distribution. When the p-value is above 0.01, we say the forgetting is significant. Model utility is the aggregated model performance on held-out retain data regarding fictional authors, real-world author profiles, and world facts. We also report the ROUGE-L score on forget set and retain set.
It’s important to note that for the forget set, a lower ROUGE-L score does not necessarily indicate better performance. Therefore, we highlight methods where the ROUGE-L score closely matches that of the retained model, as these are considered to produce better results.
More metrics can be found in Appendix C.3.1.

FLAT is always the best in preserving model utility. As seen in Table 4, it experiences almost no reductions in model utility compared to the original model. On LLaMA2-7B, although GA and KL achieve strong ROUGE-L scores on the retain dataset, their forgetting performance is poor. Similarly, on Phi-1.5B, GD performs well on the retain dataset’s ROUGE-L score, but its forgetting performance is insufficient, failing to exceed 0.01.

FLAT achieves the top two Forget Quality under all three models. FLAT achieves a ROUGE score that is closest to the Retained LLM on forget dataset under Llama2-7B and Phi-1.5B. PO shows the best forgetting efficiency on Phi-1.5B, but its ROUGE-L scores on both the forget and retain datasets are lower compared to the retained LLM, indicating weaker model utility. A similar issue is observed with GA: while it excels in forgetting performance on the OPT-2.7B model, its model utility remains weaker.

FLAT achieves the best trade-off.
Our method consistently ranks in the top two across the primary metrics, achieving the best performance in MU. Specifically, KL f-divergence demonstrates strong results in both FQ and MU on LLama2-7B and Phi-1.5B models. Overall, all four f-divergence functions effectively balance forgetting efficiency and model utility.
In summary, our method demonstrates the best model utility while achieving top-two results in forgetting performance.

**Table 4: Performance of our method and the baseline methods on TOFU dataset using three base LLMs, Llama2-7B, Phi-1.5B, and OPT-2.7B. FQ, MU, R-RL, F-RL represent forget quality, model utility, ROUGE-L on retain dataset and ROUGE-L on forget dataset respectively. We include the original LLM and retain LLM for reference. The top two results are highlighted in blue.**
| Base LLM | Llama2-7B | Phi-1.5B | OPT-2.7B |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | FQ | MU | F-RL($\downarrow$) | R-RL | FQ | MU | F-RL($\downarrow$) | R-RL | FQ | MU | F-RL($\downarrow$) | R-RL |
| Original LLM | 4.4883e-06 | 0.6346 | 0.9851 | 0.9833 | 0.0013 | 0.5184 | 0.9607 | 0.9199 | 0.0013 | 0.5120 | 0.7537 | 0.7494 |
| Retained LLM | 1.0 | 0.6267 | 0.4080 | 0.9833 | 1.0 | 0.5233 | 0.4272 | 0.9269 | 1.0 | 0.5067 | 0.4217 | 0.7669 |
| GA | 0.0143 | 0.6333 | 0.4862 | 0.9008 | 0.0013 | 0.5069 | 0.5114 | 0.8048 | 0.2657 | 0.4639 | 0.4748 | 0.6387 |
| KL | 0.0068 | 0.6300 | 0.5281 | 0.9398 | 0.0030 | 0.5047 | 0.5059 | 0.8109 | 0.0286 | 0.4775 | 0.4810 | 0.6613 |
| GD | 0.0068 | 0.6320 | 0.4773 | 0.8912 | 0.0030 | 0.5110 | 0.4996 | 0.8496 | 0.0541 | 0.4912 | 0.4521 | 0.6603 |
| PO | 0.0541 | 0.6308 | 0.3640 | 0.8811 | 0.0286 | 0.5127 | 0.3170 | 0.7468 | 0.0068 | 0.4424 | 0.0589 | 0.4015 |
| LLMU | 0.0541 | 0.6337 | 0.4480 | 0.8865 | 0.0286 | 0.5110 | 0.3058 | 0.7270 | 0.0286 | 0.3296 | 0.0347 | 0.2495 |
| DPO | 0.0541 | 0.6359 | 0.5860 | 0.8852 | 0.0521 | 0.0519 | 0.3437 | 0.7349 | 0.0541 | 0.4264 | 0.0806 | 0.3937 |
| NPO | 0.0068 | 0.6321 | 0.4632 | 0.8950 | 0.0030 | 0.5057 | 0.5196 | 0.8000 | 0.0541 | 0.4788 | 0.4993 | 0.6490 |
| FLAT (TV) | 0.0541 | 0.6373 | 0.4391 | 0.8826 | 0.0143 | 0.5168 | 0.4689 | 0.8155 | 0.0068 | 0.5086 | 0.5217 | 0.7067 |
| FLAT (KL) | 0.0286 | 0.6393 | 0.5199 | 0.8750 | 0.0143 | 0.5180 | 0.4524 | 0.7850 | 0.0286 | 0.4838 | 0.4942 | 0.6974 |
| FLAT (JS) | 0.0541 | 0.6364 | 0.4454 | 0.8864 | 0.0068 | 0.5144 | 0.4572 | 0.8117 | 0.0541 | 0.4959 | 0.4938 | 0.7013 |
| FLAT (Pearson) | 0.0541 | 0.6374 | 0.4392 | 0.8857 | 0.0143 | 0.5175 | 0.4591 | 0.8099 | 0.0068 | 0.5093 | 0.5052 | 0.7059 |

### 4.4 MUSE-News Unlearning

Experiment Setup.
We focus on the task of unlearning on News corpus presented in []. News consists of BBC news articles [] collected after August 2023. All articles are randomly divided into forget, retain, and holdout sets.
We perform unlearning directly on the pre-trained models provided by the benchmark, following the corresponding experimental setup.

Evaluation Metrics. We report the proposed four metrics, VerbMem on forget dataset, KnowMem on forget and retain dataset, and Privacy leakage (PrivLeak). We quantify the verbatim memorization VerbMem by prompting the model with the first l tokens from a sequence and comparing the continuation outputted by the model $\theta$ to the true continuation using the ROUGE-L F1 score []. We gather the model’s answers to questions and then average the ROUGE scores for all question-answer pairs in forget dataset or retain dataset to compute the knowledge memorization score KnowMem.
The PrivLeak metric for a good unlearning algorithm should be close to zero, whereas an over/under-unlearning algorithm will get a large positive/negative metric.

**Table 5: Performace on MUSE benchmark using four criteria. We highlight results in blue if the unlearning algorithm satisfies the criterion and highlight it in red otherwise. For metrics on $D_{f}$, lower values are preferred, while for metrics on $D_{r}$, higher values are better. In terms of PrivLeak, the results should be close to 0. Large negative or positive values suggest that they may cause privacy leakage.**
|  | VerbMem on $D_{f}$ ($\downarrow$) | KnowMem on $D_{f}$ ($\downarrow$) | KnowMem on $D_{r}$ ($\uparrow$) | PrivLeak |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Original LLM | 58.4 | - | 63.9 | - | 55.2 | - | -99.8 |
| Retained LLM | 20.8 | - | 33.1 | - | 55.0 | - | 0.0 |
| GA | 0.0 | (✔) | 0.0 | (✔) | 0.0 | (✘) | 17.0 |
| KL | 27.4 | (✘) | 50.2 | (✘) | 44.8 | (✔) | -96.1 |
| NPO | 0.0 | (✔) | 0.0 | (✔) | 0.0 | (✘) | 15.0 |
| NPO-RT | 1.2 | (✔) | 54.6 | (✘) | 40.5 | (✔) | 105.8 |
| Task Vectors | 56.3 | (✘) | 63.7 | (✘) | 54.6 | (✔) | -99.8 |
| GD | 4.9 | (✔) | 27.5 | (✔) | 6.7 | (✔) | 109.4 |
| WHP | 19.7 | (✔) | 21.2 | (✔) | 28.3 | (✔) | 109.6 |
| FLAT (TV) | 1.7 | (✔) | 13.6 | (✔) | 31.8 | (✔) | 45.4 |
| FLAT (KL) | 0.0 | (✔) | 0.0 | (✔) | 0.0 | (✘) | 58.9 |
| FLAT (JS) | 1.9 | (✔) | 36.2 | (✘) | 38.5 | (✔) | 47.1 |
| FLAT (Pearson) | 1.6 | (✔) | 0.0 | (✔) | 0.2 | (✔) | 26.8 |

Experiment Results. FLAT effectively removes verbatim and knowledge memorization of forget dataset and achieve good knowledge memorization of retain dataset. But it can still reveal the membership of $D_{f}$ in $D_{r}$.
As shown in Table 5, GA, NPO, GD, WHP, and ours perform well in VerbMem and KnowMem on forget dataset, often reduing them even beyond the levels achieved by the retrained model. However, these reductions often come at the cost of significant utility loss on the retain set. Only GD, WHP and ours can perfome good in all memorization related metrics. And FLAT (TV) can achieve the lowest VerbMem and KnowMem on $D_{f}$ and the highest KnowMem on $D_{r}$ among the three methods. However, none of the methods can achieve satisiable results regarding to the privacy leakage. Since MUSE uses news data, which is highly time-dependent (and thus possibly non-i.i.d.), we advocate for cautious interpretation of the PrivLeak metric. A detailed analysis is provided in Appendix C.4.1.

### 4.5 Ablation Studies

The Effectiveness of Re-weighting Mechanism. As FLAT (KL) demonstrates strong overall performance, we base our ablation study on KL divergence to explore the effectiveness of the implicit re-weighting mechanism within our loss adjustment. This study is conducted on the TOFU dataset using Llama2-7B. For the ablation study on the HP dataset, please refer to Table 13 in Appendix D.3.

When using preferred template data for unlearning, we compare our method with DPO (without the term $M_{ref}$) and SimPO, as outlined in Appendix C.1. All methods use the same data and have similar formulations, with two terms in the loss function; the only difference lies in the intrinsic re-weighting mechanism. As shown in Table 6, our method achieves the highest number of best results across 12 metrics. When replacing the IDK data with retain data, the results show that the retain version performs better on the Retain Set but worse on Real Authors and Real World compared to FLAT (KL).

Since GD shares the same data usage and formulation as our method, except for the re-weighting mechanism and utilization of retain data, we compare the retain version to GD. The results show that our method achieves better performance on both the Retain Set and Forget Set, with the decline in Real Authors and Real World performance caused by the use of retain data.

**Table 6: Ablation Study of Re-weighting Mechanism on TOFU dataset using Llama2-7B under all metrics. We report ROUGE-L score (R-L), Probability (P), and Truth Ratio (TR) on all four subsets of the TOFU benchmark. Higher scores are better except ROUGE-L and Probability on the Forget Set. The best ones are in blue.**
| Split | Real Authors | Real World | Retain Set | Forget Set |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | R-L | P | TR | R-L | P | TR | R-L | P | TR | R-L($\downarrow$) | P($\downarrow$) | TR |
| Study on Re-weighting Mechanism Using Template IDK Data |  |  |  |  |  |  |  |  |  |  |  |  |
| DPO | 0.9330 | 0.4939 | 0.6384 | 0.8917 | 0.4631 | 0.5646 | 0.8852 | 0.9623 | 0.4407 | 0.5860 | 0.8734 | 0.6240 |
| DPO w/o $M_{ref}$ | 0.9330 | 0.4899 | 0.6333 | 0.8917 | 0.4620 | 0.5642 | 0.8735 | 0.9579 | 0.4388 | 0.4021 | 0.8149 | 0.6326 |
| SimPO | 0.9330 | 0.4902 | 0.6335 | 0.8917 | 0.4624 | 0.5664 | 0.8758 | 0.9577 | 0.4388 | 0.4087 | 0.8128 | 0.6329 |
| FLAT (KL) | 0.9180 | 0.4992 | 0.6491 | 0.9060 | 0.4524 | 0.5609 | 0.8750 | 0.9679 | 0.4603 | 0.5199 | 0.7588 | 0.5895 |
| Study on Re-weighting Mechanism Using Retain Data |  |  |  |  |  |  |  |  |  |  |  |  |
| GD | 0.9080 | 0.4728 | 0.6156 | 0.8718 | 0.4439 | 0.5833 | 0.8912 | 0.9657 | 0.4701 | 0.4773 | 0.4238 | 0.5619 |
| FLAT (KL)-Retain | 0.9180 | 0.4643 | 0.6099 | 0.8832 | 0.4356 | 0.5690 | 0.9241 | 0.9734 | 0.4697 | 0.4487 | 0.3342 | 0.5879 |

The Imapct of Good Answer Type. In the first step of our approach, we intend to generate good example responses for each forget sample. We primarily use the reject-based response "I don’t know" (denoted as IDK) as the default choice. In this section, we conduct an ablation study on data usage for FLAT to analyze how these good responses impact unlearning performance.
Table 7 presents the ablation study of good answer type using three LLMs on TOFU dataset, comparing IDK with random normal responses (denoted as Normal). Table 14 in Appendix D.3 provides the ablation study of different good answer type using Llama2-7B on the HP dataset. Results indicate that using normal responses improves model utility on HP datasets and improves ROUGE-L Score on retain set on TOFU datasets, whereas using IDK responses yields better forgetting quality.

Additionally, we observe that the performance across the four divergence functions is relatively similar.
KL divergence, in particular, demonstrates more consistent results across the three datasets and three models, likely due to its reduced sensitivity to incorrect or bad answers.

**Table 7: Ablation Study of Good Answer Type using three LLMs on TOFU dataset. FQ, MU, R-RL, F-RL represent forget quality, model utility, ROUGE-L on retain dataset and ROUGE-L on forget dataset respectively. The best performance is in blue.**
| Split | Llama2-7B | Phi-1.5B | OPT-2.7B |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | FQ | MU | F-RL($\downarrow$) | R-RL | FQ | MU | F-RL($\downarrow$) | R-RL | FQ | MU | F-RL($\downarrow$) | R-RL |
| Original LLM | 4.4883e-06 | 0.6346 | 0.9851 | 0.9833 | 0.0013 | 0.5184 | 0.9249 | 0.9293 | 0.0013 | 0.5120 | 0.7537 | 0.7494 |
| Retained LLM | 1.0 | 0.6267 | 0.4080 | 0.9833 | 1.0 | 0.5233 | 0.4272 | 0.9269 | 1.0 | 0.5067 | 0.4217 | 0.7669 |
| FLAT (TV)-IDK | 0.0541 | 0.6373 | 0.4391 | 0.8826 | 0.0143 | 0.5168 | 0.4689 | 0.8155 | 0.0068 | 0.5086 | 0.5217 | 0.7067 |
| FLAT (KL)-IDK | 0.0286 | 0.6393 | 0.5199 | 0.8750 | 0.0143 | 0.5180 | 0.4524 | 0.7850 | 0.0286 | 0.4838 | 0.2212 | 0.4853 |
| FLAT (JS)-IDK | 0.0541 | 0.6364 | 0.4454 | 0.8864 | 0.0068 | 0.5144 | 0.4572 | 0.8117 | 0.0541 | 0.4959 | 0.3104 | 0.5658 |
| FLAT (Pearson)-IDK | 0.0541 | 0.6374 | 0.4392 | 0.8857 | 0.0143 | 0.5175 | 0.4591 | 0.8099 | 0.0068 | 0.5093 | 0.5052 | 0.7059 |
| FLAT (TV)-Normal | 0.0068 | 0.6173 | 0.4941 | 0.9575 | 0.0068 | 0.5104 | 0.4827 | 0.8245 | 0.0030 | 0.5086 | 0.5646 | 0.7355 |
| FLAT (KL)-Normal | 0.0068 | 0.6162 | 0.6273 | 0.9719 | 0.0068 | 0.5177 | 0.5377 | 0.8575 | 0.0030 | 0.5082 | 0.5642 | 0.7474 |
| FLAT (JS)-Normal | 0.0143 | 0.6178 | 0.4910 | 0.9560 | 0.0013 | 0.5068 | 0.5538 | 0.7313 | 0.0013 | 0.5068 | 0.5538 | 0.7313 |
| FLAT (Pearson)-Normal | 0.0068 | 0.6186 | 0.4972 | 0.9546 | 0.0030 | 0.5094 | 0.5554 | 0.7343 | 0.0030 | 0.5094 | 0.5554 | 0.7343 |

## 5 Conclusion

In this paper, we address the limitations of existing LLM unlearning methods, which often rely on the retain data or a reference LLM for response calibration. To overcome these challenges, we propose FLAT (Forget data only Loss AdjustmenT), a "flat" loss adjustment approach that eliminates the need for retain data or a reference model. By optimizing the $f$-divergence between the template and forget responses, FLAT offers a clear and theoretically grounded solution for balancing forget quality with model utility in LLM unlearning. Through extensive experiments on three key unlearning tasks: copyrighted content unlearning on the Harry Potter dataset, the MUSE benchmark, and entity unlearning on the TOFU dataset, we demonstrate the superior performance of FLAT. Our method consistently achieves high unlearning efficiency while preserving overall model utility, showcasing its effectiveness in addressing both practical and theoretical challenges in LLM unlearning.

## Appendix A Limitations and Broader Impacts

### A.1 Broad Impacts

The proposed FLAT method for LLM unlearning has the potential to significantly advance ethical and responsible AI deployment, particularly in addressing key challenges such as privacy concerns, bias, and regulatory compliance. By enabling models to effectively forget specific data without compromising overall model utility, this approach directly addresses issues related to data privacy, including compliance with regulations like GDPR, which mandates data deletion upon user request. The ability to unlearn sensitive or copyrighted information, as demonstrated on datasets like Harry Potter and TOFU, ensures that AI models can be continually refined without propagating harmful or biased content.

Furthermore, the reduced reliance on the retain data or a reference LLM makes FLAT more resource-efficient, lowering the computational and financial costs associated with large-scale unlearning. This opens up opportunities for wider adoption across industries and research institutions where access to retain data or additional model resources may be limited or not accessible. The implications of this work span multiple domains, including healthcare, finance, and education, where ethical considerations are paramount.

### A.2 Limitations

One key limitation of our approach is the unsatisfactory performance in the privacy leakage evaluation on the MUSE dataset. While FLAT demonstrates strong unlearning efficiency and retains model utility across several benchmarks, it struggles to prevent privacy leakage. Note that all other tested methods suffer from the same issue, this suggests that further refinement is needed to strengthen the privacy-preserving aspects of existing LLM unlearning approaches. Future work could explore more robust strategies to address privacy leakage while maintaining the balance between unlearning performance and model utility.

## Appendix B Theoretical Illustration and Proofs

### B.1 Additional Examples for Loss Adjustments under More f 𝑓 f italic_f -divergence

##### Example 2: Jenson-Shannon (JS)

For Jenson-Shannon $f$-divergence, we have $f^{*}(u)=-\log{(2-e^{u})},g^{*}(v)=\log{\dfrac{2}{1+e^{-v}}}$, hence:

$$ $\displaystyle g^{*}(({\mathbb{P}(x_{f},y_{e};\theta)})-f^{*}(g^{*}(({\mathbb{P }(x_{f},y_{f};\theta)})$ $\displaystyle=\log{\dfrac{2}{1+e^{-{\mathbb{P}(x_{f},y_{e};\theta)}}}}-\left[- \log\left(2-e^{\log{\dfrac{2}{1+e^{-{\mathbb{P}(x_{f},y_{f};\theta)}}}}}\right )\right]$ $\displaystyle=\log{\dfrac{2}{1+e^{-{\mathbb{P}(x_{f},y_{e};\theta)}}}}+\log \left(2-e^{\log{\dfrac{2}{1+e^{-{\mathbb{P}(x_{f},y_{f};\theta)}}}}}\right)$ $\displaystyle=\log{\dfrac{2}{1+e^{-{\mathbb{P}(x_{f},y_{e};\theta)}}}}+\log \left(2-\dfrac{2}{1+e^{-{\mathbb{P}(x_{f},y_{f};\theta)}}}\right)$ $\displaystyle=\log{\dfrac{2}{1+e^{-{\mathbb{P}(x_{f},y_{e};\theta)}}}}+\log \left(\dfrac{2e^{-{\mathbb{P}(x_{f},y_{f};\theta)}}}{1+e^{-{\mathbb{P}(x_{f},y _{f};\theta)}}}\right)$ $\displaystyle=\log\left(\dfrac{4e^{-{\mathbb{P}(x_{f},y_{f};\theta)}}}{(1+e^{- {\mathbb{P}(x_{f},y_{e};\theta)}})(1+e^{-{\mathbb{P}(x_{f},y_{f};\theta)}})} \right).$ $$

##### Example 3: Pearson

For Pearson $f$-divergence, we have $f^{*}(u)=\frac{u^{2}}{4}+u,g^{*}(v)=v$, hence:

$$ $\displaystyle g^{*}({\mathbb{P}(x_{f},y_{e};\theta)})-f^{*}(g^{*}(({\mathbb{P} (x_{f},y_{f};\theta)})$ $\displaystyle={\mathbb{P}(x_{f},y_{e};\theta)}-\left(\frac{{\mathbb{P}(x_{f},y _{f};\theta)}^{2}}{4}+{\mathbb{P}(x_{f},y_{f};\theta)}\right)$ $\displaystyle=-\frac{{\mathbb{P}(x_{f},y_{f};\theta)}^{2}}{4}-{\mathbb{P}(x_{f },y_{f};\theta)}+{\mathbb{P}(x_{f},y_{e};\theta)}.$ $$

##### Example 4: KL

For KL $f$-divergence, we have $f^{*}(u)=e^{u-1},g^{*}(v)=v$, hence:

$$ $\displaystyle g^{*}(({\mathbb{P}(x_{f},y_{e};\theta)})-f^{*}(g^{*}(({\mathbb{P }(x_{f},y_{f};\theta)})$ $\displaystyle={\mathbb{P}(x_{f},y_{e};\theta)}-e^{{\mathbb{P}(x_{f},y_{f}; \theta)}-1}.$ $$

### B.2 The derivation of empirical loss function

According to Eqn. (3.2), we have:

$$ $\displaystyle L(x_{f},y_{e},y_{f};\theta)$ $\displaystyle=-\left[\sup_{g}\left[g(\mathbb{P}(x_{f},y_{e};\theta))-f^{*}(g( \mathbb{P}(x_{f},y_{f};\theta)))\right]\right]$ $\displaystyle=\underbrace{f^{*}(g^{*}(\mathbb{P}(x_{f},y_{f};\theta)))}_{L_{f} (x_{f},y_{f};\theta)}\underbrace{-g^{*}(\mathbb{P}(x_{f},y_{e};\theta)))}_{L_{ e}(x_{f},y_{e};\theta)}.$ $$

Given a dataset $D={\{(x_{f}^{j},y_{e}^{j},y_{f}^{j})\}}_{j\in[N]}$, where $y_{e}$ and $y_{f}$ are preferred template and original forget responses to the forget prompt $x_{f}$, we estimate ${\mathbb{P}(x_{f},y_{e};\theta)},{\mathbb{P}(x_{f},y_{f};\theta)}$ via the following two quantities:

$$ $\mathbb{P}(x_{f},y_{e};\theta):=\frac{\sum_{i=1}^{|y_{e}|}P(h_{\theta}(x_{f},y _{e,<i})=y_{e,i})}{|y_{e}|},\quad\mathbb{P}(x_{f},y_{f};\theta):=\frac{\sum_{i =1}^{|y_{f}|}P(h_{\theta}(x_{f},{y_{f,<i}})=y_{f,i})}{|y_{f}|}.$ $$

Empirically, we can obtain the loss function for the dataset $D$:

$$ $\displaystyle L_{\textrm{{{FLAT }}}}(\theta)$ $\displaystyle=-\mathbb{E}_{D}\Big{[}g^{*}(\mathbb{P}(x_{f},y_{e};\theta)))-f^{ *}(g^{*}(\mathbb{P}(x_{f},y_{f};\theta)))\Big{]}$ $\displaystyle=-\mathbb{E}_{D}\Big{[}g^{*}(\frac{\sum_{i=1}^{|y_{e}|}\sum_{k=1} ^{v}y_{e,i,k}\cdot{h_{\theta}(x_{f},y_{e,<i})}_{k}}{|y_{e}|})-f^{*}(g^{*}( \frac{\sum_{i=1}^{|y_{f}|}\sum_{k=1}^{v}y_{f,i,k}\cdot{h_{\theta}(x_{f},y_{f,< i})}_{k}}{|y_{f}|})\Big{]}$ $\displaystyle=-\mathbb{E}_{D}\Big{[}g^{*}(\frac{\sum_{i=1}^{|y_{e}|}h_{\theta} (x_{f},y_{e,<i})}{|y_{e}|})-f^{*}(g^{*}(\frac{\sum_{i=1}^{|y_{f}|}h_{\theta}(x _{f},y_{f,<i})}{|y_{f}|})\Big{]}.$ $$

Here, $v$ is the vocabulary size, $y_{e,i,k}$ is the $k$-th element of vector representing the $i$-th token in the good response $y_{e}$, $y_{f,i,k}$ is the $k$-th element of vector representing the $i$-th token in the forget response $y_{f}$. Additionally, ${h_{\theta}(x_{f},y_{e,<i})}_{k}$ and ${h_{\theta}(x_{f},y_{f,<i})}_{k}$ denote the $k$-th entry of the probability distribution for the correctly generated token.

##### An example: KL

For KL f-divergence, $f^{*}(u)=e^{u-1},g^{*}(v)=v$, hence, $g^{*}({\mathbb{P}(x_{f},y_{e};\theta)})-f^{*}(g^{*}({\mathbb{P}(x_{f},y_{f};
\theta)})={\mathbb{P}(x_{f},y_{e};\theta)}-e^{{\mathbb{P}(x_{f},y_{f};\theta)}
-1}.$ We have:

$$ $\displaystyle L_{\textrm{{{FLAT }}}}(\theta)$ $\displaystyle=-\mathbb{E}_{D}\Big{[}\frac{\sum_{i=1}^{|y_{e}|}h_{\theta}(x_{f} ,y_{e,<i})}{|y_{e}|}-e^{\frac{\sum_{i=1}^{|y_{f}|}h_{\theta}(x_{f},y_{f,<i})}{ |y_{f}|}-1}\Big{]}.$ $$

### B.3 Proof of Theorem 3.4

###### Proof.

Remember that we define:

$$ $\displaystyle\hat{f}_{div}({D}_{e}||{D}_{f}):=\mathbb{E}_{{Z}_{e}\sim{D}_{e}}[ \hat{g}({Z}_{e})]-\mathbb{E}_{{Z}_{f}\sim{D}_{f}}[f^{*}(\hat{g}({Z}_{f}))],$ $$

and

$$ $\displaystyle f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f})=\sup_{g}\left[\mathbb{ E}_{\mathcal{Z}_{e}\sim\mathcal{D}_{e}}\left[g(\mathcal{Z}_{e})\right]-\mathbb {E}_{\mathcal{Z}_{f}\sim\mathcal{D}_{f}}\left[f^{*}(g(\mathcal{Z}_{f}))\right] \right],$ $$

we first prove the convergence of $\hat{g}$, and then the convergence of $\hat{f}_{div}({D}_{e}||{D}_{f})$.

For the ease of presentation, for any real-valued function $\varrho$, we write $\mathbb{E}_{\mathcal{D}_{e}}(\varrho)=\mathbb{E}_{z\sim\mathcal{D}_{e}}\left[
\varrho(z)\right]$, $\mathbb{E}_{\mathcal{D}_{f}}(\varrho)=\mathbb{E}_{z\sim\mathcal{D}_{f}}\left[
\varrho(z)\right]$, $\mathbb{E}_{{D}_{e}}(\varrho)=\mathbb{E}_{z\sim{D}_{e}}\left[\varrho(z)\right]$, and $\mathbb{E}_{{D}_{f}}(\varrho)=\mathbb{E}_{z\sim{D}_{f}}\left[\varrho(z)\right]$.

Given any $\tilde{g}\in\Phi$, according to Lemma C.1 in [], and the fact that $f^{*}$ is Lipschitz continuous, we have:

$$ $\displaystyle||\hat{g}-g^{*}||_{L_{2}(\mathcal{D}_{f})}^{2}$ $\displaystyle\precsim\left[\mathbb{E}_{{D}_{e}}[(\hat{g}-g^{*})/2]-\mathbb{E}_ {\mathcal{D}_{e}}[(\hat{g}-g^{*})/2]\right]$ $\displaystyle\qquad-\left[\mathbb{E}_{{D}_{f}}[f^{*}((\hat{g}-g^{*})/2)-f^{*}( g^{*})]-\mathbb{E}_{\mathcal{D}_{f}}[f^{*}((\hat{g}-g^{*})/2)-f^{*}(g^{*})] \right].$ (5) $$

By using the fact that the true density ratio ${Z}_{e}/{Z}_{f}$ is bounded below and above, hence, $L_{2}(\mathcal{D}_{e})$ is indeed equivalent to $L_{2}(\mathcal{D}_{f})$. Based on Eqn. (B.3), Lemma C.2 in [], and the Lipschitz property of $f^{*}$, with probability at least $1-c_{1}\exp({-N^{r_{\Phi}/(2+r_{\Phi})}/c_{1}^{2}})$, we have

$$ $\displaystyle||\hat{g}-g^{*}||_{L_{2}(\mathcal{D}_{f})}^{2}$ $\displaystyle\precsim N^{-1/(r_{\Phi}+2)}.$ (6) $$

Note that we have:

$$ $\displaystyle\left|\hat{f}_{div}({D}_{e}||{D}_{f})-f_{div}(\mathcal{D}_{e}|| \mathcal{D}_{f})\right|$ $\displaystyle\leq$ $\displaystyle|\mathbb{E}_{{D}_{e}}[\hat{g}-g^{*}]-\mathbb{E}_{\mathcal{D}_{e}} [\hat{g}-g^{*}]|+|\mathbb{E}_{{D}_{f}}[f^{*}(\hat{g})-f(g^{*})]-\mathbb{E}_{ \mathcal{D}_{f}}[f^{*}(\hat{g})-f^{*}(g^{*})]|$ $\displaystyle\qquad+|\mathbb{E}_{\mathcal{D}_{e}}[\hat{g}-g^{*}]-\mathbb{E}_{ \mathcal{D}_{f}}[f^{*}(\hat{g})-f^{*}(g^{*})]|+|\mathbb{E}_{{D}_{e}}[g^{*}]- \mathbb{E}_{\mathcal{D}_{e}}[g^{*}]|+|\mathbb{E}_{{D}_{f}}[f^{*}(g^{*})]- \mathbb{E}_{\mathcal{D}_{f}}[f^{*}(g^{*})]|$ $\displaystyle=\text{Cons}_{1}+\text{Cons}_{2}+\text{Cons}_{3}+\text{Cons}_{4}+ \text{Cons}_{5}.$ (7) $$

By Lemma C.2 in [], with probability at least $1-c_{1}\exp({-N^{r_{\Phi}/(2+r_{\Phi})}/c_{1}^{2}})$, we have

$$ $\text{Cons}_{1}\precsim N^{-2/(r_{\Phi}+2)}.$ $$

Similar upper bound holds for $\text{Cons}_{2}$.

Following from Eqn. (6), with probability at least $1-c_{1}\exp({-N^{r_{\Phi}/(2+r_{\Phi})}/c_{1}^{2}})$, we have

$$ $\text{Cons}_{3}\precsim N^{-1/(r_{\Phi}+2)}.$ $$

Applying Hoeffding’s inequality, with probability at least $1-c_{1}\exp({-N^{r_{\Phi}/(2+r_{\Phi})}/c_{1}^{2}})$, we have

$$ $\text{Cons}_{4}\precsim N^{-1/(r_{\Phi}+2)}.$ $$

Similar upper bound holds for $\text{Cons}_{5}$.

Combining the five upper bounds for $\text{Cons}_{i}$ where $i\in[5]$, with probability at least $1-c_{1}\exp({-N^{r_{\Phi}/(2+r_{\Phi})}/c_{1}^{2}})$, we have

$$ $\left|\hat{f}_{div}({D}_{e}||{D}_{f})-f_{div}(\mathcal{D}_{e}||\mathcal{D}_{f} )\right|\precsim N^{-\frac{1}{r_{\Phi}+2}}.$ $$

∎

###### Proof.

## Appendix C Detailed Experimental Setup

**Table 8: Summary of unlearning tasks, including base models, forget datasets, and evaluation metrics.**
| Unlearning Task | Base Model | Forget Dataset | Metrics |
| --- | --- | --- | --- |
| Copyrighted Content Unlearning | OPT-2.7B, Llama2-7B | Harry Potter Series | BLEU, ROUGE_L, PPL, Zero-shot Acc |
| Entity Unlearning (TOFU) | OPT-2.7B, Llama2-7B, Phi-1.5B | TOFU-Forget01 | Forget Quality (p-value), ROUGE_L on Forget set, Model Utility |
| MUSE Benchmark | Llama2-7B | BBC News Corps | verbatim and knowledge memorization on $\mathcal{D}_{f}$, privacy leakage, Utility preservation |

Table 8 summarizes the experimental setups, including base models, forget and retain datasets, and evaluation metrics.

### C.1 Formulations for baseline methods

In this section, we revisit existing unlearning objectives and unify them under our general loss function framework, as described in Section § 2.2. We provide formulations for GA, GD, KL, and PO, as presented in [], as well as for Mismatch [], LLMU [], DPO[], and NPO []. Additionally, we include the formulations for DPO without the $M_{ref}$ term and SimPO [], as discussed in Section § 3.4. We also add formulations for Task Vectors [] and Who’s Harry Potter (WHP)[].

##### Fine-tuning on retain data

Retraing from scratch is the gold standard for unlearning. HHowever, in real-world scenarios, retain data may not always be available, and retraining a LLM is highly resource-intensive. Alternatively, we can fine-tune the model using retain data for several epochs, which only involves performing gradient descent on $D_{r}$.

$$ $L_{\textrm{Fine-tune}}=\underbrace{\frac{1}{|D_{r}|}\sum_{(x_{r},y_{r})\in D_{ r}}\mathcal{L}(x_{r},y_{r};\theta)}_{\textbf{Retain\; Loss}}$ $$

##### Gradient ascent (GA)

GA is simple baselines commonly used in traditional machine unlearning settings [, , , ].
GA reverts the change of the gradient descent during the training with its opposite operation. The rationale of gradient ascent is that a subsequent maximization of prediction loss on the forget dataset $D_{f}$ would approximately "revert" the optimization on the forget dataset, thus unlearning $D_{f}$ and approximating a model trained on the retain dataset $D_{r}$ only.

$$ $L_{\textrm{GA}}=\underbrace{-\frac{1}{|D_{f}|}\sum_{(x_{f},y_{f})\in D_{f}} \mathcal{L}(x_{f},y_{f};\theta)}_{\textbf{Forget\; Loss}}$ $$

##### Gradient difference (GD)

Gradient difference has been introduced as simple baseline method in [].It combines fine-tuning and gradient ascent by compute the sum of the two loss terms.

$$ $L_{\textrm{GD}}=\underbrace{\frac{1}{|D_{r}|}\sum_{(x_{r},y_{r})\in D_{r}} \mathcal{L}(x_{r},y_{r};\theta)}_{\textbf{Retain\; Loss}}\underbrace{-\frac{1} {|D_{f}|}\sum_{(x_{f},y_{f})\in D_{f}}\mathcal{L}(x_{f},y_{f};\theta)}_{ \textbf{Forget\; Loss}}$ $$

##### KL minimization (KL)

The KL minimization is adopted from [] and involves a gradient ascent term for forgetting. It also minimizes the Kullback-Leibler (KL) divergence between the predictions on retain data $D_{r}$ of the reference model (the original model $\theta_{o}$) and the newly trained model (the unlearned model $\theta$). This term aims to keep the unlearned model’s current output distribution on the retain dataset close to its pre-unlearning distribution on the retain samples.

$$ $L_{\textrm{KL}}=\underbrace{L_{\textrm{GA}}}_{\textbf{Forget\;Loss}}+ \underbrace{\frac{1}{|D_{r}|}\sum_{(x_{r},y_{r})\in D_{r}}\sum_{i=1}^{|y_{r}|} \operatorname{KL}(h_{\theta_{0}}(x_{r},y_{r<i})\|h_{\theta}(x_{r},y_{r<i}))}_{ \textbf{Retain\;Loss}}$ $$

##### Preference optimization (PO)

Preference Optimization (PO) differs from the traditional direct preference optimization approach as presented in [] in that it combines the fine-tuning loss on $D_{r}$ with a term that teaches the model to respond with ’I don’t know’ to prompts from $D_{f}$ []. Here, $D_{\textrm{idk}}$ refers to an augmented forget dataset where the model’s response to the prompt is ’I don’t know.’

$$ $L_{\textrm{PO}}=\underbrace{L_{\textrm{Fine-tune}}}_{\textbf{Retain\; Loss}}+ \underbrace{\frac{1}{|D_{\textrm{idk}}|}\sum_{x_{f},y_{idk}\in D_{\textrm{idk} }}\mathcal{L}(x_{f},y_{idk};\theta)}_{\textbf{Custom\; Loss}}$ $$

Here, the Custom Loss utilizes the modified response to the forget prompt to ensure that the model rejects answering questions related to the forget data.

##### Mismatch

Mismatch has the same objective to PO, except it involves constructing a random combination of text sequences $\mathbf{\mathbf{Y}}_{\textrm{rdn}}$. Here, the second term in mismatch is the same as the second term in LLMU [].

$$ $L_{\textrm{Mismatch}}=\underbrace{L_{\textrm{Fine-tune}}}_{\textbf{Retain\; Loss}}+\underbrace{\sum_{(x_{f},\cdot)\in D_{f}}\frac{1}{|Y_{\textrm{rdn}}|} \sum_{y_{rdn}\in Y_{rdn}}\mathcal{L}(x_{f},y_{rdn};\theta)}_{\textbf{Custom\; Loss}}$ $$

##### LLMU [ 5 ]

LLMU combines the GA term with two additional terms to learn 1) random completions $Y_{\textrm{rdn}}$ from $D_{\textrm{r}}$ (constructed using prompts from $D_{f}$) to facilitate unlearn and 2) $D_{\textrm{r}}$ to preserve performance. We use books with similar styles as $D_{\textrm{r}}$ in our experiments and construct $Y_{\textrm{rdn}}$ using randomly sampled text sequences from $D_{\textrm{r}}$.

$$ $\displaystyle L_{\textrm{LLMU}}=$ $\displaystyle-\sum_{(x_{f},y_{f})\in D_{f}}\mathcal{L}(x_{f},y_{f};\theta)$ $\displaystyle+\sum_{(x_{f},\cdot)\in D_{f}}\frac{1}{|Y_{rdn}|}\sum_{y_{rdn}\in Y _{rdn}}\mathcal{L}(x_{f},y_{rdn};\theta)$ $\displaystyle+\sum_{(x_{r},y_{r})\in D_{r}}\sum_{i=1}^{|y_{r}|}KL(h_{\theta_{o }}(x_{r},y_{r<i})||h_{\theta}(x_{r},y_{r<i}))$ $$

We have already unified LLMU in Section § 2.2

##### Direct preference optimization (DPO), DPO w/o M r ⁢ e ⁢ f subscript 𝑀 𝑟 𝑒 𝑓 M_{ref} italic_M start_POSTSUBSCRIPT italic_r italic_e italic_f end_POSTSUBSCRIPT , SimPO

See Section § 3.4 for more information.

$$ $L_{\textrm{DPO,}\beta}(\theta)=-\frac{2}{\beta}\mathbb{E}_{D}\Big{[}\log\sigma \Big{(}\underbrace{\beta\log\prod_{i=1}^{|y_{e}|}h_{\theta}(x_{f},y_{e,<i})}_{ \textbf{Custom \;Loss}}-\underbrace{\beta\log\prod_{i=1}^{|y_{f}|}h_{\theta}(x _{f},y_{f,<i}))}_{\textbf{Forget\;Loss}}\underbrace{-M_{ref}}_{\textbf{Retain/ Custom\;Loss}}\Big{)}\Big{]}.$ $$

$$ $L_{\textrm{DPO}\;\textrm{w/o}\;M_{ref},\beta}(\theta)=-\frac{2}{\beta}\mathbb{ E}_{D}\Big{[}\log\sigma\Big{(}\underbrace{\beta\log\prod_{i=1}^{|y_{e}|}h_{ \theta}(x_{f},y_{e,<i})}_{\textbf{Custom \;Loss}}-\underbrace{\beta\log\prod_{ i=1}^{|y_{f}|}h_{\theta}(x_{f},y_{f,<i}))}_{\textbf{Forget\;Loss}}\Big{)}\Big{ ]}.$ $$

$$ $L_{\textrm{SimPO},\beta}(\theta)=-\frac{2}{\beta}\mathbb{E}_{D}\Big{[}\log \sigma\Big{(}\underbrace{\frac{\beta}{|y_{e}|}\log\prod_{i=1}^{|y_{e}|}h_{ \theta}(x_{f},y_{e,i})}_{\textbf{Custom \; Loss}}\underbrace{-\frac{\beta}{|y_ {f}|}\log\prod_{i=1}^{|y_{f}|}h_{\theta}(x_{f},y_{f,i})}_{\textbf{Forget\;Loss }})-\gamma]\Big{)}\Big{]}$ $$

where $\gamma$ is the target reward margin.

##### Negative preference optimization (NPO) [ 22 ]

NPO incorporates only the losing response term in DPO [], penalizing only the prompt-response pairs in $\mathcal{D}_{f}$.In the formulation below, $\beta$ represents the inverse-temperature, $\pi_{\theta}$ is the prediction probability of LLM $\theta$. NPO also has two extended versions that include either the KL term or a fine-tuning term on $\mathcal{D}_{r}$ to preserve model utility.

$$ $\displaystyle L_{\textrm{NPO}}$ $\displaystyle=-\frac{2}{\beta}\mathbb{E}_{D_{f}}\Big{[}\log\sigma\Big{(}-\beta log \frac{\pi_{\theta}(y_{f}\mid x_{f})}{\pi_{ref}(y_{f}\mid x_{f})}\Big{)}\Big{]}$ $\displaystyle=-\frac{2}{\beta}\mathbb{E}_{D_{f}}\Big{[}\log\sigma\Big{(} \underbrace{\beta\log\pi_{ref}(y_{f}\mid x_{f})}_{\textbf{Retain/Custom\;Loss} }\underbrace{-\beta\log\pi_{\theta}(y_{f}\mid x_{f})}_{\textbf{Forget\;Loss}} \Big{)}\Big{]}$ $$

$$ $\displaystyle L_{\textrm{NPO-KL}}$ $\displaystyle=L_{\textrm{NPO}}+L_{\textrm{KL}}$ $\displaystyle L_{\textrm{NPO-RT}}$ $\displaystyle=L_{\textrm{NPO}}+L_{\textrm{Fine-tune}}$ $$

##### Task Vectors [ 14 ]

The taks vector is derived by calculating the weight difference between the original LLM $\theta_{o}$ and a reinforce LLM $\theta_{reinforce}$, which is the model trained on $D_{f}$ until it over-fits. This method then subtract this task vector from the original LLM’ weights, intuitively moving the model away from the direction it used to adapt to $D_{f}$. The weights of unlearned model can be obtained as:

$$ $\theta=\theta_{o}-(\theta_{reinforce}-\theta_{o})$ $$

##### WHP [ 14 ]

WHP defines the unlearned model $\theta$ as the interpolation between the original model $\theta_{o}$ and the reinforced model $\theta_{reinforce}$. Let $p_{\theta}(\cdot|x)$ denote the token distribution parametrized by the model $\theta$ when given a prompt $x$ as input. Then, for any input $x$, WHP samples the next token from:

$$ $p_{\theta}(\cdot|x)=p_{\theta_{o}}(\cdot|x)-\alpha(p_{\theta_{reinforce}}( \cdot|x)-p_{\theta_{o}}(\cdot|x))$ $$

where $\alpha$ is a hyperparameter that controls the interpolation between the two models.

### C.2 Copyrighted Unlearning on HP

#### C.2.1 Evaluation Metrics

We use two text similarity metrics to evaluate our models. In each case, the original copyrighted text serves as the reference, and we calculate the similarity between this reference and the text generated by the LLM.

##### ROUGE-L

For the forget dataset, we compute the ROUGE-L recall score [] between the ground truth responses (the forget responses) and the text generated by the model after unlearning.

##### BLEU

Similarly, we compute the BLEU score [] for the forget dataset, comparing the ground truth responses to the model’s output after unlearning.

A retained model that has never seen the reference text should score low on both metrics, and a successfully unlearned model should perform similarly. For Harry Potter datasets, we evaluate similarity using the first 600 generated tokens as per [].

##### Perplexity (PPL)

We assess text fluency and diversity by computing perplexity on the Wikitext [] using the LM Evaluation Harness []. A model with lower perplexity on the fine-tuned data suggests the generated text remains meaningful.

##### Zero-shot Accuracy

We evaluate zero-shot accuracy across various tasks, including BoolQ [], RTE [], HellaSwag [], Winogrande [], ARC-Challenge [], ARC-Easy [], OpenBookQA [], Piqa [], and TruthfulQA []. The mean accuracy across these diverse tasks was computed and reported as a comprehensive measure of model utility after unlearning. The higher the average accuracy, the better the results.

#### C.2.2 Implementation Setting.

To demonstrate the copyright removal task, we undertake the fine-tuning of all the models using the complete Harry Potter series.
The finetuning procedure for the OPT-2.7B and Llama2-7B models involve a learning rate of 1e-5 and a batch size of 2.
AdamW serves as the optimizer for preparing these models.
For baseline methods, we set the batch size and learning rate to be the same as in their original papers, and fine-tune for 5 epochs using AdamW optimizer. For our method, we use the same training hyper-parameters as baseline but set the learning rate to be 2e-7.

### C.3 Entity Unlearning on TOFU

#### C.3.1 Evaluation Metrics

We utilize the original evaluation metrics designed in the original paper of the TOFU dataset [].

##### Probability

For each instance in the retain or forget set, we calculate the normalized conditional probability $P(a\mid q)^{1/|a|}$ on the LLM subject to unlearning, where $q$ represents the question, $a$ is the answer, and $|a|$ denotes the number of tokens in the answer. For the real authors and world facts subsets, the dataset provides a set of five answers $\{a_{0},\tilde{a}_{1},\tilde{a}_{2},\tilde{a}_{3},\tilde{a}_{4}\}$, consisting of one correct answer $a_{0}$ and four perturbed answers that are incorrect. In this case, we compute the ratio $P(a_{0}\mid q)^{1/|a_{0}|}/\sum_{i=1}^{4}P(\tilde{a}_{i}\mid q)^{1/|\tilde{a}_
{i}|}$.

##### Truth ratio

The truth ratio is computed as the geometric mean of multiple perturbed (incorrect) answers’ ($\mathcal{A}=\{\tilde{a}_{1},\tilde{a}_{2},...\}$) probabilities over the normalized conditional probability of the paraphrased answer $\hat{a}$.

$$ $R_{\text{truth}}=\frac{\left(\prod_{i=1}^{|\mathcal{A}|}P(\tilde{a}\mid q)^{|1 /\tilde{a}_{i}|}\right)^{1/|\mathcal{A}|}}{P(\hat{a}\mid q)^{1/|\hat{a}|}}$ $$

For the real authors and world fact subsets, the original answer $a$ is used in the denominator as no paraphrased answer is available.

##### ROUGE-L

For all subsets of TOFU, we compute the ROUGE-L recall score [] between the ground truth responses (forget dataset) and the text generated by the model after unlearning.

##### Model utility

The model utility is aggregated as a harmonic mean over nine numbers: the answer probability, truth ratio, and ROUGE recall scores from each of the retain, real authors, and world facts subsets. A higher model utility is always preferred.

##### Forget quality

The forget quality is determined by calculating the p-value from a Kolmogorov-Smirnov (KS) test, which compares two distributions: the truth ratio of the retained model and the truth ratio of the unlearned model on the forget set. A higher p-value suggests that the null hypothesis — that the distributions of the truth ratios from both models are identical — cannot be rejected, indicating that the retained and unlearned models behave similarly.

#### C.3.2 Implementation Setting.

For all LLM unlearning methods, we set the batch size to be 32 following previous works [, , ] and use consistent learning rates for each model. For OPT-2.7B and Phi-1.5B, we fine-tune the pre-trained models for 5 epochs using learning rate of 2e-5 to obtain the original model. Similarly, we fine-tune Llama2-7B for the same duration with a learning rate of 1e-5. AdamW serve as the optimizer for preparing these models. The unlearning process for all methods, including ours, employs the same learning rate as used during fine-tuning the original models. For all experiments on the TOFU dataset, the training hyperparameters remain consistent across models of the same type.

### C.4 MUSE-News Unlearning

#### C.4.1 Evaluation Metrics

##### Note on PrivLeak metric

The PrivLeak metric used in [] is derived from Min-K% Prob, a membership inference attack method for LLMs. Formally, it is calculated as:

$$ $\text{PrivLeak}=\frac{\operatorname{AUC}\left(f_{\text{unlearn}};D_{\text{ forget}},D_{\text{holdout}}\right)-\operatorname{AUC}\left(f_{\text{retrain}}; D_{\text{forget}},D_{\text{holdout}}\right)}{\operatorname{AUC}\left(f_{\text{ retrain}};D_{\text{forget}},D_{\text{holdout}}\right)}$ $$

where the AUC score refers to the standard AUC-ROC score between $D_{\text{forget}}$ and $D_{\text{holdout}}$. While this method indeed discriminates between the forget and holdout distributions as a measure of successful unlearning, it is highly dependent on the data selected for evaluation. Specifically, Min-K% Prob has been shown to yield random-guess accuracy due to modern LLMs being trained on large pretraining corpora for only a small number of iterations, causing fuzzy boundary between members and non-members [].

Furthermore, [] demonstrate that Min-K% Prob results in 1) high variance depending on the random selection of the dataset used for evaluation, 2) better performance when the two subsets (in our case, forget and holdout) are not drawn from the same distribution, and 3) empirically overestimated false positives. The latter finding suggests that the distribution gap (i.e., temporal shift, which is also identified by []) acts as a confounding factor in the discrimination process, since the forget set and holdout set may differ in more than one dimension. Given that MUSE [] uses news data, which is highly time-dependent (and thus possibly non-i.i.d.), we advocate for cautious interpretation of the PrivLeak metric.

## Appendix D Experimental Results

### D.1 Copyrighted Unlearning on HP dataset

##### Performance using Llama2-7B on HP dataset

Table 9 indicates that our method consistently places within the top three across the primary metrics, with TV f-divergence showing the best performance. LLMU achieves the best forgetting effect, comparable to our method, but its model utility is inferior. PO again demonstrates the highest PPL but exhibits poor forgetting performance. While DPO shows good model utility, the difference between our method and DPO in PPL is minimal. Unlike other methods, PO directly leverages fine-tune loss, which helps preserve the model’s performance beyond the unlearning. These results highlight the effectiveness of our method, striking a better balance between forgetting quality and model utility.

**Table 9: Unlearning performance of Llama2-7B on the Harry Potter dataset. R-L and Avg. Acc. denote the ROUGE-L score and average zero-shot accuracy across nine LLM benchmarks. We include the original LLM and retained LLM for reference. Some methods, such as PO, LLMU, DPO, and NPO, exhibit strong performance in either forget quality or model utility, but underperform in the other. FLAT consistently ranks in the top three in terms of similarity to the retained model, measured by Forget Quality Gap (FQ Gap), while also generating meaningful and diverse outputs, as indicated by perplexity (PPL) and the average zero-shot accuracy (Avg. Acc.). The top three results across the three main metrics are highlighted in blue.**
|  | Forget Quality | Model Utility |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | BLEU($\downarrow$) | BLEU Gap | R-L($\downarrow$) | R-L Gap | FQ Gap | PPL($\downarrow$) | PPL Gap | Avg.Acc. | Acc. Gap |
| Original LLM | 4.0452 | - | 0.1487 | - | - | 8.9524 | - | 0.5617 | - |
| Retained LLM | 0.4903 | - | 0.0442 | - | - | 8.7070 | - | 0.5599 | - |
| GA | 0.0624 | 0.4279 | 0.0134 | 0.0308 | 0.4587 | 47.2769 | -38.5699 | 0.5088 | -0.0511 |
| KL | 0.0976 | 0.3927 | 0.0144 | 0.0298 | 0.4225 | 9.4336 | -0.7266 | 0.5509 | -0.0090 |
| GD | 0.0039 | 0.4864 | 0.0002 | 0.0440 | 0.5304 | 9.1797 | -0.4727 | 0.4902 | -0.0697 |
| PO | 0.0206 | 0.4697 | 0.0015 | 0.0427 | 0.5124 | 8.8364 | -0.1294 | 0.5532 | -0.0067 |
| Mismatch | 0.0670 | 0.4233 | 0.0028 | 0.0414 | 0.4647 | 8.9906 | -0.2836 | 0.5593 | -0.0056 |
| LLMU | 0.3033 | 0.1870 | 0.0317 | 0.0125 | 0.1985 | 9.0530 | -0.3460 | 0.5503 | -0.0096 |
| DPO | 0.7717 | -0.2814 | 0.0552 | -0.0110 | 0.2924 | 8.9597 | -0.2527 | 0.5614 | 0.0015 |
| NPO | 0.9840 | -0.4937 | 0.0656 | -0.0214 | 0.5151 | 9.0397 | -0.3327 | 0.5609 | 0.0010 |
| FLAT (TV) | 0.6770 | -0.1867 | 0.0673 | -0.0231 | 0.2098 | 8.9899 | -0.2829 | 0.5592 | -0.0007 |
| FLAT (KL) | 0.6829 | -0.1926 | 0.0662 | -0.0220 | 0.2146 | 8.9803 | -0.2733 | 0.5572 | -0.0027 |
| FLAT (JS) | 0.6890 | -0.1987 | 0.0684 | -0.0242 | 0.2229 | 8.9910 | -0.2840 | 0.5574 | -0.0025 |
| FLAT (Pearson) | 0.6930 | -0.2027 | 0.0680 | -0.0238 | 0.2265 | 8.9906 | -0.2836 | 0.5580 | -0.0019 |

##### HP Example Generations under our method FLAT

The generated samples can be seen in Table 10.

**Table 10: Examples of OPT-2.7b’s original and generated response using our unlearning method on HP dataset.**
| Prompt: Let’s see how you would complete this piece of text: You should have realized , ” he said as he saw Voldemort’s nostrils flare … but it all makes sense , Potter , and in ways that you do not understand ! “ Dumbledore was trying to keep |
| --- |
| Original response: the Elder Wand from me ! He intended that Snape should be the true master of the wand ! But I got there ahead of you … round his eyes whiten . “ It’s your one last chance , ” said Harry , “ it’s all you’ve got left . |
| Generated response: me trapped, to stop Snape from catching me, to get hold of me when he knew Voldemort had sent Severus Snape to m … to speak to me! This is why Dumbledore wanted Snape to kill him once only I am |
| Question: Let’s see how you would complete this piece of text: “ That’s all very well , but what if Montague’s permanently injured ? ” “ … “ She’ll say I should’ve stopped them leaving , I should’ve grabbed the ends of |
| Original response: their wands and made them stay. ” “ You’re not going to be in trouble, ” said Hermione, looking up at him … be in trouble at all. ” “ But I’m not? ” said Ron, looking very confused |
| Generated reponse: those rope like Dumbledore. A few points deducted for this, a couple for your father’s attempt at distraction, now we’re even. ‘You could have stopped it by showing any of you … not so smart as Slytherin I’d like to hear how much Hermione’s taken from Stinks |

##### Parameter Study on Prompt Length for HP dataset

Following [, ], we evaluate the forget quality using prompt lengths of 50, 100, 200, 300 on Harry Potter series dataset. Table 11 presents the parameter study of different prompt lengths for assessing forget quality on this dataset. In the main paper, we adopt the prompt length of 200, as suggested by [].

**Table 11: Parameter Study of different prompt length on Harry Potter book series dataset. We include the original LLM and retained LLM for reference.**
| Split | Prompt Length 50 | Prompt Length 100 | Prompt Length 200 | Prompt Length 300 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | BLEU($\downarrow$) | ROUGE-L($\downarrow$) | BLEU($\downarrow$) | ROUGE-L($\downarrow$) | BLEU($\downarrow$) | ROUGE-L($\downarrow$) | BLEU($\downarrow$) | ROUGE-L($\downarrow$) |
| OPT-2.7B |  |  |  |  |  |  |  |  |
| Original LLM | 3.4492 | 0.1203 | 3.7660 | 0.1273 | 4.1163 | 0.1484 | 3.4924 | 0.1551 |
| Retain LLM | 1.7350 | 0.0944 | 2.3427 | 0.0986 | 2.6072 | 0.1229 | 2.7479 | 0.1261 |
| FLAT (TV) | 0.8382 | 0.1090 | 0.9607 | 0.1206 | 1.1955 | 1.4117 | 1.4532 | 0.1628 |
| FLAT (KL) | 0.8363 | 0.1107 | 0.9867 | 0.1209 | 1.2743 | 1.3329 | 1.4714 | 0.1657 |
| FLAT (JS) | 0.8709 | 0.1101 | 0.9720 | 0.1213 | 1.1986 | 0.1290 | 1.4735 | 0.1635 |
| FLAT (Pearson) | 0.8430 | 0.1098 | 0.9624 | 0.1211 | 1.1917 | 1.4155 | 1.4501 | 0.1627 |
| Llama2-7B |  |  |  |  |  |  |  |  |
| Original LLM | 0.0448 | 0.0049 | 0.3951 | 0.0254 | 4.0452 | 0.1487 | 0.2541 | 0.0275 |
| Retain LLM | 0.0917 | 0.0111 | 0.1664 | 0.0162 | 0.4903 | 0.0442 | 0.2542 | 0.0194 |
| FLAT (TV) | 0.0293 | 0.0045 | 0.2627 | 0.0276 | 0.6770 | 0.0673 | 0.2185 | 0.0251 |
| FLAT (KL) | 0.0308 | 0.0041 | 0.2592 | 0.0276 | 0.6829 | 0.0662 | 0.2217 | 0.0242 |
| FLAT (JS) | 0.0305 | 0.0045 | 0.2512 | 0.0279 | 0.6890 | 0.0684 | 0.2143 | 0.0253 |
| FLAT (Pearson) | 0.0310 | 0.0044 | 0.2573 | 0.0281 | 0.6930 | 0.0680 | 0.2163 | 0.0252 |

### D.2 Entity Unlearning on TOFU

##### TOFU Experimental Results using All Metrics

Table 12 shows the performance on TOFU using three base LLMs, Llama2-7B, Phi-1.5B, and OPT-2.7 under all metrics.

**Table 12: Performance on TOFU dataset using three base LLMs, Llama2-7B, Phi-1.5B, and OPT-2.7 under all metrics. We report ROUGE-L score (R-L), Probability (P), and Truth Ratio (TR) on all four subsets of the TOFU benchmark. Higher scores are better except ROUGE-L and probability on the Forget Set. We include the original LLM and retained LLM for reference. The best two are highlighted in blue.**
| Split | Real Authors | Real World | Rerain Set | Forget Set |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | R-L | P | TR | R-L | P | TR | R-L | P | TR | R-L($\downarrow$) | P($\downarrow$) | TR |
| Llama2-7B |  |  |  |  |  |  |  |  |  |  |  |  |
| Original LLM | 0.9350 | 0.4738 | 0.6210 | 0.8846 | 0.4355 | 0.5579 | 0.9833 | 0.9900 | 0.4662 | 0.9851 | 0.9898 | 0.5123 |
| Retained LLM | 0.9230 | 0.4645 | 0.6118 | 0.8932 | 0.4182 | 0.5449 | 0.9833 | 0.9902 | 0.4724 | 0.4080 | 0.1798 | 0.6939 |
| GA | 0.9030 | 0.4754 | 0.6233 | 0.8761 | 0.4432 | 0.5843 | 0.9008 | 0.9546 | 0.4695 | 0.4862 | 0.3566 | 0.5705 |
| KL | 0.9280 | 0.4652 | 0.6092 | 0.8803 | 0.4383 | 0.5691 | 0.9398 | 0.9705 | 0.4655 | 0.5281 | 0.5119 | 0.5626 |
| GD | 0.9080 | 0.4728 | 0.6156 | 0.8718 | 0.4439 | 0.5833 | 0.8912 | 0.9657 | 0.4701 | 0.4773 | 0.4238 | 0.5619 |
| PO | 0.9330 | 0.4850 | 0.6269 | 0.8917 | 0.4582 | 0.5602 | 0.8811 | 0.9627 | 0.4393 | 0.3640 | 0.8695 | 0.6318 |
| LLMU | 0.9330 | 0.4905 | 0.6344 | 0.8917 | 0.4603 | 0.5625 | 0.8865 | 0.9628 | 0.4391 | 0.4480 | 0.8606 | 0.6286 |
| DPO | 0.9330 | 0.4939 | 0.6384 | 0.8917 | 0.4631 | 0.5646 | 0.8852 | 0.9623 | 0.4407 | 0.5860 | 0.8734 | 0.6240 |
| NPO | 0.8930 | 0.4754 | 0.6218 | 0.8746 | 0.4466 | 0.5798 | 0.8950 | 0.9574 | 0.4680 | 0.4632 | 0.3664 | 0.5785 |
| NPO-RT | 0.8830 | 0.4758 | 0.6218 | 0.8746 | 0.4459 | 0.5805 | 0.8958 | 0.9588 | 0.4687 | 0.4519 | 0.3672 | 0.5791 |
| FLAT (TV) | 0.9180 | 0.4937 | 0.6459 | 0.8974 | 0.4505 | 0.5591 | 0.8826 | 0.9685 | 0.4607 | 0.4391 | 0.5314 | 0.6026 |
| FLAT (KL) | 0.9180 | 0.4992 | 0.6491 | 0.9060 | 0.4524 | 0.5609 | 0.8750 | 0.9679 | 0.4603 | 0.5199 | 0.7588 | 0.5895 |
| FLAT (JS) | 0.8980 | 0.4927 | 0.6460 | 0.8974 | 0.4508 | 0.5592 | 0.8864 | 0.9686 | 0.4607 | 0.4454 | 0.5183 | 0.6039 |
| FLAT (Pearson) | 0.9180 | 0.4932 | 0.6461 | 0.8974 | 0.4509 | 0.5583 | 0.8857 | 0.9684 | 0.4607 | 0.4392 | 0.5092 | 0.6037 |
| Phi-1.5B |  |  |  |  |  |  |  |  |  |  |  |  |
| Original LLM | 0.4073 | 0.3744 | 0.4470 | 0.7503 | 0.4148 | 0.4982 | 0.9199 | 0.9238 | 0.4810 | 0.9607 | 0.9345 | 0.4839 |
| Retained LLM | 0.4240 | 0.3779 | 0.4539 | 0.7585 | 0.4090 | 0.4974 | 0.9269 | 0.9271 | 0.4855 | 0.4272 | 0.1686 | 0.6579 |
| GA | 0.4573 | 0.3638 | 0.4373 | 0.7541 | 0.3978 | 0.4741 | 0.8048 | 0.7748 | 0.4880 | 0.5114 | 0.3268 | 0.5099 |
| KL | 0.4273 | 0.3643 | 0.4370 | 0.7474 | 0.3997 | 0.4764 | 0.8109 | 0.8043 | 0.4889 | 0.5059 | 0.3342 | 0.5091 |
| GD | 0.3907 | 0.3726 | 0.4461 | 0.7605 | 0.4087 | 0.4931 | 0.8496 | 0.8900 | 0.4910 | 0.4996 | 0.4025 | 0.4952 |
| PO | 0.4240 | 0.3728 | 0.4449 | 0.7699 | 0.4190 | 0.5207 | 0.7468 | 0.8747 | 0.4596 | 0.3170 | 0.7362 | 0.5416 |
| LLMU | 0.4240 | 0.3720 | 0.4421 | 0.7785 | 0.4203 | 0.5197 | 0.7270 | 0.8678 | 0.4572 | 0.3058 | 0.7067 | 0.5453 |
| DPO | 0.0420 | 0.3713 | 0.4423 | 0.7785 | 0.4202 | 0.5205 | 0.7349 | 0.8712 | 0.4583 | 0.3437 | 0.6999 | 0.5393 |
| NPO | 0.4573 | 0.3619 | 0.4342 | 0.7417 | 0.3988 | 0.4761 | 0.8000 | 0.7856 | 0.4840 | 0.5196 | 0.3529 | 0.5119 |
| NPO-RT | 0.4473 | 0.3619 | 0.4340 | 0.7474 | 0.3998 | 0.4770 | 0.8024 | 0.7926 | 0.4851 | 0.5193 | 0.3527 | 0.5129 |
| FLAT (TV) | 0.4440 | 0.3695 | 0.4390 | 0.7742 | 0.4125 | 0.5040 | 0.8155 | 0.8858 | 0.4709 | 0.4689 | 0.4756 | 0.5395 |
| FLAT (KL) | 0.4440 | 0.3735 | 0.4464 | 0.7571 | 0.4175 | 0.5147 | 0.7850 | 0.8874 | 0.4666 | 0.4524 | 0.6285 | 0.5287 |
| FLAT (JS) | 0.4340 | 0.3703 | 0.4386 | 0.7588 | 0.4119 | 0.5045 | 0.8117 | 0.8850 | 0.4714 | 0.4572 | 0.4683 | 0.5390 |
| FLAT (Pearson) | 0.4540 | 0.3694 | 0.4389 | 0.7674 | 0.4117 | 0.5040 | 0.8099 | 0.8850 | 0.4711 | 0.4591 | 0.4672 | 0.5383 |
| OPT-2.7B |  |  |  |  |  |  |  |  |  |  |  |  |
| Original LLM | 0.6687 | 0.3833 | 0.4393 | 0.6433 | 0.3701 | 0.4158 | 0.7494 | 0.8335 | 0.4992 | 0.7537 | 0.8237 | 0.5338 |
| Retained LLM | 0.6487 | 0.3735 | 0.4249 | 0.6278 | 0.3696 | 0.4185 | 0.7669 | 0.8399 | 0.4988 | 0.4217 | 0.1991 | 0.7097 |
| GA | 0.6390 | 0.3774 | 0.4375 | 0.5953 | 0.3644 | 0.4071 | 0.6387 | 0.4097 | 0.4972 | 0.4748 | 0.0722 | 0.6325 |
| KL | 0.6573 | 0.3775 | 0.4346 | 0.6463 | 0.3646 | 0.4071 | 0.6613 | 0.4707 | 0.4958 | 0.4810 | 0.1110 | 0.5902 |
| GD | 0.6453 | 0.3782 | 0.4336 | 0.6084 | 0.3613 | 0.3979 | 0.6603 | 0.6916 | 0.5162 | 0.4521 | 0.1701 | 0.5774 |
| PO | 0.4078 | 0.3874 | 0.4540 | 0.5135 | 0.3705 | 0.4207 | 0.4015 | 0.6922 | 0.4546 | 0.0589 | 0.5220 | 0.6037 |
| LLMU | 0.1528 | 0.3739 | 0.4215 | 0.3946 | 0.3695 | 0.4031 | 0.2495 | 0.6013 | 0.4305 | 0.0347 | 0.3967 | 0.6356 |
| DPO | 0.3478 | 0.3853 | 0.4498 | 0.4915 | 0.3708 | 0.4230 | 0.3937 | 0.6445 | 0.4375 | 0.0806 | 0.3931 | 0.6255 |
| NPO | 0.6573 | 0.3787 | 0.4343 | 0.6281 | 0.3681 | 0.4130 | 0.6490 | 0.4978 | 0.4870 | 0.4993 | 0.1355 | 0.5952 |
| NPO-RT | 0.5698 | 0.3646 | 0.4107 | 0.6264 | 0.3675 | 0.4239 | 0.4620 | 0.2459 | 0.4065 | 0.3627 | 0.1087 | 0.6716 |
| FLAT (TV) | 0.6737 | 0.3878 | 0.4457 | 0.6382 | 0.3715 | 0.4137 | 0.7067 | 0.8075 | 0.4858 | 0.5217 | 0.6368 | 0.5601 |
| FLAT (KL) | 0.6903 | 0.3802 | 0.4364 | 0.6369 | 0.3672 | 0.4108 | 0.6974 | 0.8078 | 0.4909 | 0.4942 | 0.6735 | 0.5515 |
| FLAT (JS) | 0.6723 | 0.3824 | 0.4385 | 0.6369 | 0.3706 | 0.4162 | 0.7013 | 0.7685 | 0.4911 | 0.4938 | 0.5662 | 0.5502 |
| FLAT (Pearson) | 0.6737 | 0.3873 | 0.4462 | 0.6467 | 0.3705 | 0.4150 | 0.7059 | 0.8069 | 0.4868 | 0.5052 | 0.6329 | 0.5597 |

### D.3 Ablation Study

Table 13 demonstrates the effectiveness of the re-weighting mechanism on the Harry Potter dataset. When using the preferred template data for unlearning, our method achieves strong forget quality and comparable model utility. When using retain data, our method outperforms GD, indicating that the re-weighting mechanism improves both unlearning efficiency and model utility.

Table 14 presents the ablation study on the good answer types using Llama2-7B on the Harry Potter dataset. The results show that using normal responses enhances model utility on the HP dataset, while using IDK responses leads to better forgetting quality.

**Table 13: Ablation Study of the Re-weighting Mechanism using Llama2-7B on Harry Potter dataset. R-L and Avg. Acc. denote the ROUGE-L score and average zero-shot accuracy over nine LLM benchmarks. We include the original LLM and retain LLM for reference. The best ones are highlighted in blue.**
|  | Forget Quality | Model Utility |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | BLEU($\downarrow$) | BLEU Gap | R-L($\downarrow$) | R-L Gap | FQ Gap | PPL($\downarrow$) | PPL Gap | Avg.Acc. | Acc. Gap |
| Original LLM | 4.0452 | 3.5549 | 0.1487 | 0.2933 | 3.8482 | 8.9524 | 0.2444 | 0.5617 | 0.0018 |
| Retained LLM | 0.4903 | 0.0 | 0.0442 | 0.0 | 0.0 | 8.7070 | 0.0 | 0.5599 | 0.0 |
| Study on Re-weighting Mechanism Using Retain Data |  |  |  |  |  |  |  |  |  |
| GD | 0.0039 | 0.4864 | 0.0002 | 0.0440 | 0.5304 | 9.1797 | -0.4727 | 0.4902 | -0.0697 |
| FLAT (KL)-retain | 0.2359 | 0.2544 | 0.0263 | 0.0179 | 0.2714 | 8.9948 | -0.2878 | 0.5591 | -0.0008 |
| Study on Re-weighting Mechanism Using IDK Data |  |  |  |  |  |  |  |  |  |
| DPO w/o $M_{ref}$ | 0.7719 | -0.2816 | 0.0523 | -0.0081 | 0.2897 | 8.9674 | -0.2604 | 0.5560 | -0.0039 |
| SimPO | 0.6876 | -0.1973 | 0.0552 | -0.0110 | 0.2723 | 8.9927 | -0.2857 | 0.5593 | -0.0006 |
| FLAT (KL) | 0.6829 | -0.1926 | 0.0662 | -0.0220 | 0.2146 | 8.9803 | -0.2733 | 0.5572 | -0.0027 |

**Table 14: Ablation Study of the good answer type using Llama2-7B on Harry Potter dataset. R-L and Avg. Acc. denote the ROUGE-L score and average zero-shot accuracy over nine LLM benchmarks. We include the original LLM and retain LLM for reference. The best ones are highlighted in blue.**
|  | Forget Quality | Model Utility |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | BLEU($\downarrow$) | BLEU Gap | R-L($\downarrow$) | R-L Gap | FQ Gap | PPL($\downarrow$) | PPL Gap | Avg.Acc. | Acc. Gap |
| Original LLM | 4.0452 | - | 0.1487 | - | - | 8.9524 | - | 0.5617 | - |
| Retained LLM | 0.4903 | - | 0.0442 | - | - | 8.7070 | - | 0.5599 | - |
| FLAT (TV)-IDK | 0.6770 | -0.1867 | 0.0673 | -0.0231 | 0.2098 | 8.9899 | -0.2829 | 0.5592 | -0.0007 |
| FLAT (KL)-IDK | 0.6829 | -0.1926 | 0.0662 | -0.0220 | 0.2146 | 8.9803 | -0.2733 | 0.5572 | -0.0027 |
| FLAT (JS)-IDK | 0.6890 | -0.1987 | 0.0684 | -0.0242 | 0.2229 | 8.9910 | -0.2840 | 0.5574 | -0.0025 |
| FLAT (Pearson)-IDK | 0.6930 | -0.2027 | 0.0680 | -0.0238 | 0.2265 | 8.9906 | -0.2836 | 0.5580 | -0.0019 |
| FLAT (TV)-Normal | 0.6942 | -0.2039 | 0.0677 | -0.0235 | 0.2274 | 8.9872 | -0.2802 | 0.5588 | -0.0011 |
| FLAT (KL)-Normal | 0.6773 | -0.1870 | 0.0664 | -0.0222 | 0.2092 | 8.9721 | -0.2651 | 0.5580 | -0.0019 |
| FLAT (JS)-Normal | 0.6980 | -0.2077 | 0.0678 | -0.0236 | 0.2313 | 8.9888 | -0.2818 | 0.5593 | -0.0006 |
| FLAT (Pearson)-Normal | 0.6991 | -0.2088 | 0.0681 | -0.0239 | 0.2327 | 8.9882 | -0.2812 | 0.5594 | -0.0005 |

## Appendix E Related Work

### E.1 LLM Unlearning

In recent years, large language models (LLMs) [, , , ] have achieved significant success in a wide range of downstream tasks, spanning from natural language understanding to generative AI applications. The success of LLMs relies heavily on the extensive efforts to gather instructional data consisting of millions of examples [, , ] for pre-training and fine-tuning [, ]. However, due to imperfect human expertise [, , , ], the training data often suffer from imperfect human annotation or noisy (potentially harmful) web data [, , ], which tends out to be a tough task hard for data curating [, , , , ]. As a result, their outputs may contain sensitive, private, or illegal content [, ], reflect societal biases [, ], or provide harmful instructions [, , ], motivating the researcher to efficiently unlearn the LLM on a certain proportion of data.

LLM unlearning approaches can be broadly categorized into three families: model-based methods, input-based methods, and data-based methods [].

##### Model-based Methods

Model-based approaches involve modifying the weights and/or architecture to achieve unlearning. These include gradient ascent (GA) and its variants [, , ], as well as model editing techniques [, , ]. The dominant approach among existing LLM unlearning methods is fine-tuning the original model based on a carefully designed unlearning objective function [, , , , , ]. A common strategy combines forgetting and retaining objectives, applying gradient ascent updates to undesirable data while using regular gradient descent on desirable data [, ]. The goal of GA is to maximize the loss on the forget data, essentially reversing the effect of gradient descent during training. Some methods employ custom loss functions that go beyond standard forgetting and retaining losses. For example, [] introduce a loss function with three components, where the custom loss reflects advanced techniques or regularization applied to the objectives. Other methods, such as DPO [], KTO [], and NPO [], utilize reference models to guide the unlearning process.

[] fine-tune an adapter over the unlearning objective, which acts as an unlearning layer within the LLM. Several works also employ assistant or reinforced LLMs to facilitate unlearning [, , ]. For instance, [] introduce an assistant LLM that pursues the opposite of the unlearning goals—i.e., remembering forgotten documents and forgetting retained knowledge. The unlearned LLM is then derived by computing the logit difference between the original and assistant LLMs.

##### Data-based Methods

Data-based methods fine-tune the LLM using a set of modified responses. This approach often begins by generating altered outputs (e.g., refusal-based responses), such as obliterated responses [], inverted facts [], or in-domain plausible alternatives []. These generated responses are then used to guide the unlearning process. [] propose Alternate Preference Optimization, which utilizes in-domain positive feedback on the forget set, complementing the usual negative feedback to overcome the limitations of relying solely on negative feedback during unlearning. In this work, we employ reject-based template outputs as the modified "good" responses for the forgotten samples.

##### Input-based Methods

Input-based methods craft input instructions [, , , , , , ], such as in-context examples and prompts, to steer the original LLM toward the unlearning objective without altering the model’s parameters. These approaches aim to achieve unlearning in the output space rather than in the parameter space. Among these methods, a notable baseline by [] uses an external prompt classifier as a guardrail, applying embedding corruptions to the identified prompts. The authors demonstrate that this corruption scheme results in distribution-wise similarity to the retrained model.

In this work, we propose a novel loss adjustment method for LLM unlearning, which simultaneously utilizes available example responses, effectively combining data-based and model-based methods.

### E.2 Machine Unlearning

In response to the data regulation requirements [], machine unlearning (MU) has emerged as a critical process to remove the influence of specific data points, data classes, or even higher-level data concepts from a trained machine-learning model. One direct unlearning method involves retraining the model from scratch after removing the forgotten data from the original dataset, which is often considered the gold standard [, ]. However, this approach comes with significant computational demands. To alleviate this, most research focuses on developing approximate but much faster unlearning techniques, including gradient ascent [, ], influence unlearning [, , ], Fisher forgetting [, ], finetuning-based approaches [, ], and loss correction-related unlearning [, , ].

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Step 1 (template generation) seems to offer various alternatives and variations. Have the authors conducted ablation studies to evaluate the impact of different template response strategies on unlearning performance?
2. Can the authors provide additional analysis on why simpler baselines, such as Mismatch, occasionally achieve comparable results and how FLAT specifically addresses these edge cases?
soundness: 3
strengths: The proposed FLAT method addresses a critical limitation in existing LLM unlearning techniques by eliminating the need for retain data or reference models, making it highly practical for real-world applications where such data is often unavailable. The paper provides a thorough theoretical foundation for the method, supported by the use of f-divergence as a guiding principle for loss adjustment. Furthermore, the extensive experiments on diverse datasets (Harry Potter, TOFU, and MUSE benchmarks) demonstrate the versatility and robustness of the method. The paper also situates FLAT well within the landscape of existing methods, clearly delineating its advantages.
summary: The paper introduces FLAT (Forget data only Loss Adjustment), a novel method for unlearning in large language models (LLMs) without requiring retain data or a reference model. By leveraging f-divergence maximization between template and forget responses, FLAT achieves unlearning solely based on the forget data. The empirical results across three datasets show that the method offers competitive performance, effectively balancing forget quality with model utility.
weaknesses: While FLAT demonstrates strong performance, its advantages over simpler baselines, such as the Mismatch method, appear marginal in some scenarios. For instance, in Table 3, the FLAT (KL) method outperforms alternatives but only slightly, raising questions about the practical significance of these improvements. Additionally, the performance of Mismatch is not consistently included in all subsequent experiments (e.g., Tables 6 and 7), which could provide a clearer comparative analysis.

The paper also lacks an ablation study on Step 1 (template response generation), despite its pivotal role in the method. Variations in template design could significantly impact unlearning performance, and exploring these effects would enhance the methodological insights.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: Please see the weaknesses. Additionally,
1. Why is the TOFU experiment only conducted on 1% subset? Most works evaluate on all 1%, 5%, and 10%, and it seems 10% is a more difficult setting.
2. What does it mean to satisfy a criterion in Table 5?
3. Why is VerbMem calculated using only 1 prefix token? It does not make much sense to only have a single-token prefix.
4. Why does Table 6 report different metrics from other experiments on TOFU?
soundness: 2
strengths: 1. This paper studies an important question of how to perform LLM unlearning when retain data is not available. The paper provides a new perspective from f-divergence, which is missing in existing LLM unlearning works and could benefit future research.
2. The paper conducts experiments on three datasets, covering different popular LLM unlearning settings.
summary: This paper introduces FLAT, a method for LLM unlearning that does not require access to retain data to maintain model utility. The idea of FLAT is to obtain a model that maximizes the f-divergence between an example response distribution (how the model should respond on the forget data) and the forget response distribution (distribution of forget data). Experiments on three LLM unlearning datasets demonstrate that FLAT can balance the forget effectiveness and model utility, without access to retain data.
weaknesses: 1. My main concern is about the performance of the proposed method. Particularly, on TOFU, FLAT does not show a significant improvement compared to baselines. Most methods achieve similar forget quality (FQ) and model utility (MU), so it's unclear if the improvement is significant or not. Additionally, why is the reported FQ much lower than the NPO paper? On 1% subset, the NPO paper reports a FQ greater than 0.8. The retain performance in Tables 3 and 5 also does not show a significant improvement.
2. Several related works that also perform unlearning without retain data are missing [1-3]. Particularly, the name change algorithm in [1-2] is not compared. In [2], it shows that this method achieves a much higher FQ while maintaining the MU, without access to retain data.
3. The presentation of the paper, especially the methodology section, is confusing. Specifically, it is stated that the goal is to maximize the f-divergence in Eq 3. However, how exactly does $\theta$ appear in Eq 3? Based on the description of the algorithm, both distributions $\mathcal{D}_e$ and $\mathcal{D}_f$ seem to have nothing to do with $\theta$. $\mathcal{D}_f$ is the distribution for the forget data, and $\mathcal{D}_e$ is the distribution for the example response on the forget data. So what does it mean to maximize Eq 3 with $\theta$? How is Eq 4 derived from Eq 3? Some notations, such as $h$, are also confusing. It appears in multiple equations, but it seems sometimes it's the predicted probability for a specific token, and sometimes it's just a predicted token.

[1] Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms.
[2] Liu et al., Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective.
[3] Dong et al., UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: * How is the alternative answer sampled? Is it generated by the model undergoing unlearning, or by a separate LLM?
* It seems that the primary difference between the proposed objective function and the SimPO objective function is the weight for the two loss terms. Is there an intuitive explanation for why the proposed f-divergence-based weighting would perform better than the approach used in SimPO?
soundness: 2
strengths: * The paper studies a relaxed unlearning setting for LLMs that does not require retain data, which may be challenging to obtain in some cases. 
* The experiment involves multiple benchmarks, diverse LLMs, and various baseline unlearning methods to offer a comprehensive assessment of the proposed approach.
summary: This paper explores a method for LLM unlearning with only the data designated for forgetting, rather than relying on both forget and retain data. The proposed method employs a weighted loss function comprising two terms: one term encourages the model to produce template/refusal response for the forget data, and the other term discourages the model from generating the original. The experimental results suggest that this method achieves a favorable trade-off between forgetting accuracy and retaining the model's overall performance.
weaknesses: * Unclear Motivation for f-divergence Reweighting: I did not quite follow the method section. Specifically, I think using f-divergence to reweight the loss terms with coefficients $\lambda_e$ and $\lambda_f$ remains unclear. The method section (specifically lines 197-215, step 3) states that this maximizes the f-divergence between $D_e$ and $D_f$. But I'm not fully clear why this is beneficial for LLM unlearning.
* Limited Performance improvement: The experimental results do not clearly demonstrate a significant advantage of the proposed method over baseline methods. For example, the forget quality on the TOFU dataset is similar to that of the baseline methods. Additionally, the NPO method paper reports a substantially higher forget quality (close to 1.0) in the TOFU-1% unlearning scenario, yet Table 4 reports a much lower forget quality for NPO (6e-3).
* Scalability Concerns: The experiments focus only on the TOFU-1% unlearning setting for TOFU dataset, which involves forgetting information on only 2 authors out of 200 in the synthetic dataset. There is no result on the performance of the method with larger forget sets, such as the TOFU-5% or TOFU-10% settings, raising concerns about the scalability and generalizability of the proposed method.


Please let me know if I have misunderstood any points in the paper.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. Should line 235 be log prob inthe equation? Since you have a sum rather than multiplication on probabilities.
2. Line 371. PO should have the lowest PPL as shown in Table 3.
3. On TOFU dataset. Did you first finetune base LLMs on fictitious data (including forget, reatin and held-out) and then perform unlearning? I think you should add one sentence explaining that. And the forget quality measures the how close the unlearned model’s output matches a model trained only on the retain data. Should here the input into the unlearned model the input of forget data? Then why we should expect the unlearned model resembles a model trained only on the retain data? I feel very confused with this setting. Could you provide clearer explanation?
4. I am also wondering why on Harry Potter you used Forget Quality Gap, which is the difference in ROGUE L and and BLEU. However in TOFU you choose to directly compare the ROGUE-L between unlearned model and retained model. As pointed our in line 408, low ROGUE-L doesn’t necessarily indicate to better performance. Then why you still use Forget Quality Gap in Harry Potter?
5. Is line 512-515 repeated content?
soundness: 4
strengths: 1. FLAT is a method without using "retain data" or reference models, which is novel and potentially more efficient against previous baselines.
2. Paper is easy to follow. Proof is provided.
3. Experiments are comprehensive and convincing.
4. I like the informative appendices.
summary: This paper proposes FLAT, an LLM unlearning method that only requires "forget data" for parameter update. FLAT is driven by an optimization loss that maximizes the f-divergence between the forget data and the forget query paired with expected unlearned responses. The authors finds in this way model can implicitly choose a balance between promoting exemplary and suppressing bad generations of forget data. Experimental results show FLAT's superiority against previous baselines. More importantly, FLAT does not require optimizing models on "retain data" or any reference model to achieve unlearning.
weaknesses: 1. Not really a lot of weaknesses. I just feel that some experimental settings should be more clearly discussed. Please refer to the questions below.
2. Some repeated contents in the main body and I suspect some equation is not correct. But I can understand. Please refer to the questions.

Please address my questions one by one. Good luck with the rebuttal.

Meta-review: This paper proposes a new approach to unlearning. The idea is to only use the desired forget set (rather than a retain set or any auxiliary model, such as a reference model), and to use a particular form of loss function adjustment to perform unlearning. The loss function adjustment uses a general approach for f divergences, which means the ability to plug in a bunch of different divergences. 

For strengths, the authors’ overall idea is clean and they show how to make this practical and produce some high-quality results (despite the challenges of the setting). 

In terms of weaknesses, the paper does suffer from some challenges around clarity; many of the reviewers generally struggled with understanding one of several parts. However, the authors did a good job explaining and updating their drafts.

Overall this is a solid paper and should be accepted.

There are some typos and the writing needs a bit more clarity in certain places, which I encourage the authors to handle before camera ready.

Decision: Accept (Poster)