Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
Authors: Ruiqi Zhang
Licong Lin
Yu Bai
Song Mei
Venue: COLM (2024)
Keywords: Unlearning, RLHF, preference optimization
TL;DR: We propose a simple alignment-inspired objective function for machine unlearning, achieving state-of-art performance in TOFU dataset.

Abstract: Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail to effectively unlearn the target data or suffer from catastrophic collapse --- a drastic degradation of the model's utilities. 

In this paper, we propose \emph{Negative Preference Optimization} (NPO), a simple alignment-inspired method that could efficiently and effectively unlearn a target dataset. We theoretically show that the progression toward catastrophic collapse by minimizing the NPO loss is exponentially slower than GA. Through experiments
on synthetic data and the benchmark TOFU dataset, we demonstrate that NPO-based methods achieve a better balance between unlearning the undesirable data and maintaining the model's utilities. 
We also observe that NPO-based methods generate more sensible outputs than GA-based methods, whose outputs are often gibberish.
Remarkably, on TOFU, NPO-based methods are the first to achieve reasonable unlearning results in forgetting 50\% (or more) of the training data, whereas existing methods already struggle with forgetting 10\% of training data.

Published as a conference paper at COLM 2024
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
Ruiqi Zhang & Licong Lin
Department of Statistics, UC Berkeley (rqzhang, liconglin [roonglin/bbereelgge. /berreeey..e...e edu
Yu Bai
Song Mei
Salesforce Al Research yu. baibaaillllll @ww.eeforcee......oo.....ooo.ese
Department of Statistics, UC Berkeley songmei@berkeele. edu
Abstract
Large Language Models (LLMs) often memorize sensitive, private, or copy- righted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's 's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail effectively unlearn the target data or suffer from catastrophic collapse-a drastic degradation of the model's utilities.
In this paper, we propose Negative Preference Optimization (NPO), simple elignnent-inspiree method that could efficiently and effectively unlearn a target dataset. We theoretically show that the progression toward catas- trophic collapse by minimizing the NPO loss is exponentially slower than GA. Through experiments on synthetic data and the benchmark TOFU dataset, we demonstrate that NPO-based methods achieve better balance between unlearning the undesirable data and maintaining the model's s utilities. We also observe that NPO-based methods generate more sensible outputs than GA-based methods, whose outputs are often gibberish. Re- markably, on TOFU, NPO-based methods are the first achieve reasonable unlearning results in forgetting 50% (or more) of the training data, whereas existing methods already struggle with forgetting 10% of training data. |
1 Introduction
Large language models (LLMs), pretrained on massive corpora of internet data, possess the capability to memorize portions of their training data (Carlini al. 2021 2022). However, this capability raises significant concerns, as the training data may contain sensitive or private information, potentially leading to societal challenges. For instance, language models could breach individual privacy by outputting personal information such as social security numbers from the memorized data (Carlini et al. 2021 Huang et al., 2022). They might also violate copyright by generating text from memorized books, such as the Harry Potter novels (Eldan Russinovich| 2023). Furthermore, LLM assistants for biology could inadvertently aid in the development of biological weapons by troubleshooting bottlenecks, increasing the risk of such attempts (Sandbrink) 2023 Li et al. 2024). In response to these concerns, regulations like the EU's General Data Protection Regulation (GDPR) (Mantelero) 2013 Voigt & Von dem Bussche 2017) and the US's California Consumer Privacy Act (CCPA) (CCPA) 2018) have mandated the Right to be Forgotten, requiring applications to support the deletion of information contained in training samples upon user requests. This has motivated line of research on machine unlearning, aiming to address these challenges.
'Equal contributions; the more junior author is listed earlier. Code is available at: https://github.com liiono-llin
1

---

Published as a conference paper at COLM 2024
Gradient Ascent
B 0
Negative Preference Optimization
Paired response
Direct Preference Optimization
Yo =log((()))
Fxa log
460
Unpaired response
Zwo Elog (flg Kdin) -Foo 463)
24f02 =g(t)) +(22
Wytth
F[ F[Vlog(x,(2)))
El EW)t Vlog sc(2))
Adaptive weighting
Figure 1: Gradient Ascent (GA), Negative Preference Optimization (NPO), and Direct Preference Optimization (DPO). NPO can be interpreted as DPO without positive samples. The gradient of NPO is an adaptive weighting of that of GA, and the weight vanishes for unleamed samples.
Machine unlearning (Cao & Yang 2015 Boorroull et al. 2021 aims to delete the influence of
specific training samples from machine-leamnin models while preserving other knowledge and capabilities (Liu et al. 2024a Zhang et al. 2023 Nguyen et al. 2022 Xu et al. 2023
Si et al. 2023). Notably, straightiorward approach to unlearning to retrain language model from scratch. However, as retraining from scratch is typically computationally expensive, cheaper methods for removing undesirable information is highly desirable.
Recently, several works (Jang et al. 2022: Wang et al. 2023 Chen & Yang 2023
Yao et al.
2023 Eldan & Russinovich 2023; Yao al.) 2024 Liu al. 2024b Li et al. 2024 proposed
scalable and practical techniques for unlearning LLMs through directly line-tuning the
trained model. Core to many of these works is a gradient ascent procedure on the prediction
loss over the dataset to be unlearned (the forget set), building on the intuition that gradient ascent is an approximation of "reverting" gradient descent optimization on the forget set.
Despite its simplicity and widespread use, the performance of gradient ascent based ap- proaches remain unsatisfactory. A notable example concerns the recently released bench- mark dataset TOFU (Maini et al.) 2024), which consists of synthetically generated biographies of 200 fictitious authors, and the task is to unlearn the biographies of 1%, 5%, and 10% of the 200 authors from model that is already fine-tuned on all 200 authors. In their evaluation of forgetting 10% the authors,MMinn et al. 2024) demonstrated that gradient ascent and its variants fail to provide satisfactory balance between forget quality (the difference between the unlearned model and retrained model evaluated on the forget set) and model utility (the general performance
other tasks).
In this work, we begin by observing that gradient ascent can often cause a rapid deterioration of model utility during training- phenomenon we term catasi sillapse-whihh we believe is responsible for its unsatisfactory performance. Towards fixing this, we propose a simple yet effective objective function for unlearning termed Negative Preference Optimization (NPO), NPO takes inspiration from preference optimization (Rafailov et al., 2024 Ouyang ct al. 2022 Bai al. 2022), and can be viewed as a variant of preference optimization that only uses negative samples. Through both theory and experiments, we show that NPO resolves the catastrophic collapse issue associated with gradient ascent, provides more stable training dynamics, and achieves a better trade-off between forgetting quality and model utility. Coupled with cross-entropy loss on the retain set, NPO achieves state-of-the-ar performance the TOFU dataset, and achieves the first non-trivial unlearning result on the challenging task of forgetting 50% of the TOFU data.
Summary of contributions and paper outline.
We outline existing gradient ascent based methods for machine unlearning, and find that these methods suffer from catastrophic collapse (Section 2). We identify the linear divergence speed of gradient ascent as main reason for catastrophic collapse.
We introduce Negative Preference Optimization (NPO), a simple alignment-inspired loss function for LLM unlearning that addresses the catastrophic collapse issue of gradient ascent (GA; Section 3). We demonstrate that NPO reduces to gradient ascent (GA) in the high-temperature limit. We show in theory the progression towards catastrophic collapse when minimizing the NPO loss is exponentially slower than with GA. See Figure [ for an illustration of NPO and its connections with existing objectives.
2

---

Published as a conference paper at COLM 2024
We test NPO-based methods on synthetic binary classification task (Section 4 where find that NPO-based methods outperform other baselines by providing a superior Pareto frontier between the Forget Distance and Retain Distance. Furthermore, NPO-based methods exhibit greater learning stability compared to GA-based methods.
We evaluate variety of unlearning methods on the TOFU dataset (Maini et al. 2024 and find that NPO-based methods exhibit superior balance between Forget Quality and Model Utility compared to all baselines (Section 5). Additionally, NPO-based methods improve the stability of the unlearning process and the readability of the output. Notably, we show that NPO-based methods are the only effective unlearning methods for forgetting 50%-90% of the data, significant advance over all existing methods which already struggle with forgetting 10% of the data (Section 5.3)
There is a vast literature machine unlearning and LLM unlearning, Due to limited space, we discuss these related work in Appendix A
2 Preliminaries on Machine Unlearning
Machine Unlearning refers to the following problem: Given an initial model (also the reference model) (y)(((( x) that is already trained on a dataset = { (x,, hiejn) how to make the model a specific subset (henceforth the forget set) DrG S DD of the training data? More precisely, we aim to fine-tund the model to make it behave like the retrained model Tretr, a model trained only on the relain set DRI D \ Drc- In other words, we would like the model to behave as if the samples in the forget set DrG were never used to train it.
By definition, the best approach for machine unlearning in principle is to retrain the model from scratch on DRT only which is however often intractable in practice.
Gradient ascent is key component in many existing LLM unlearning methods, and also an important baseline method for LLM unlearning on its own. The idea is simply to perform gradient ascent the (next-token prediction) loss over the forget set, which can be viewed equivalently as gradient descent on the negative prediction loss, denoted as LCA:
LGA (0) - [ (o = E [log( (T (y(xx)))
(1)
prediction loss
The rationale of gradient ascent is that since the initial model The is trained on D DrG U DRI, a subsequent maximization of prediction loss on the forget set DFG would approximately "revert" the optimization on the forget set DrG, thus unlearning DrG and approximating a model trained on DRI only.
Other loss functions. Building on gradient ascent, a large class of unlearning methods perform gradient-based optimization on linear combination of the GA loss LGA and several other loss functions that encourage unlearning (Jang et al., 2022 Yao et al. 2023
Chen & Yang 2023 Maini et al. 2024 Eldan & Russinovich| [223)). Notable examples include
Forget (FG) loss: (0) = - E [[og()((()x)) where (x, DFC and ÿ # is any "uninformed" response for prompt x which the unlearned model could aim to output. Examples of such y7's include replacing true information by random (but appearingly sensible) information (which requires hand-crafting such as Eldan & Russinovich
(202)), ), or simply answering "I don't know" (Maini et al. 2024
(RT EEr [log(70o (ylx))], which encourages the model to still perform well on the retain set DRT;
Retain (RT) loss:
KFG (f) = ED [D( (- [x) (-)x)) which measures the distance to the initial model Tee (in KL divergence) on the forget set;
KRT (0) = EDDr D( (- ( ), which measures the distance to the initial model Neef (in KL divergence) on the retain set.
'hhere are alternative approaches such prompt engineering (Pawelezyk et al. 2023 for perform- ing unlearning tasks.
3

---

Published as a conference paper at COLM 2024
Forget Quality
Model Utiliy
KL the Real World Set
Readability
0.7
0.3
0.0
L3 NPO
geology suther bern in Karachi, Pakistan on 06/30/19757
Q: What is the name of the
0.4
The author's same Hina Ameen.
04
GA+RT: harr narr narr narr narfuu
0.2
NPO+RT: The full name of the geology author Adeel Ahmed
0.2
0.0
Epochs
Figure 2: Comparison between GA and NPO on forget quality, model utility, KL divergence on the real-world Set, and the answers to the forget set. The rightmost figure shows the answers generated from variants of GA and NPO that incorporates the RT loss. All figures are generated on the Forget05 task in the TOFU data, trained for 10 epochs (detailed setup in Appendix E.I
For example, Yao et al. (023) minimize a combination of { (CGA, LFG, KRT L, and Chen & - CRT, KFG, KRT J. Maini et al. (2024) find that
Yang (2023) minimize a combination of {FGA, SR.,
incorporating the retain loss LeT improves the performance on various unlearning methods.
Forget quality and model utility. Unlearning methods should not only unlearn the forget set, i.e. achieve a high forget quality, but also maintain the model's performance on the retain set, i.e. maintain the model utility. For example, letting the model to simply output "I don't t know" is a unlearning method that achieves good forget quality (in certain sense) but bad model utility. While there is not yet a consensus on the right metrics for forget quality and model utility (and we will present our choices momentarily), a general rule of thumb is that unlearning methods should achieve a good tradeoff between these two goals.
2.1 Catastrophic collapse of gradient ascent
We begin by testing gradient ascent as standalone method (as opposed to combining it with other losses), and find that gradient ascent exhibits a common failure mode dubbed as catastrophic collapse: Along the unlearning process, the model utility quickly drops to zero, and the forget quality improves temporarily for a very short time horizon before quickly dropping too (Figure 2 seet/miidde-left)) Along the same training trajectory, the model diverges quickly from the initial model (as measured by the KL distance to the initial model), after which the model generates gibberish outputs (Figure middle-right right).
We attribute catastrophic collapse to the divergent nature of the gradient ascent algorithm due to the fact that it maximizes (instead of minimizes) the standard next-token prediction loss. Further, the speed of this divergence can be as fast as linear in the number of steps, as each gradient step can move the model output by constant. To see this consider a linear-logistic K-class classifier given by T0(-1x) = softmax(fx)) f = (0)))=[kk E
toy example,
R For any "already unlearned sample (x; y:) with true label y; I E [K] and model prediction softmax(#x,, > 0 (so that Tip does not predict I), standard calculation shows that the gradient of GA loss with respect to 6, is CGAi = (1 = 13 softmax(fx;)))x x Xi which has constant scale (not diminishing along the unlearning progress) and can cause the model to diverge in linear speed. Therefore, the divergent dynamics may initially bring the model closer to Tretr but would ultimately send the model to infinity.
While we believe some kind of divergent behavior is necessary and perhaps unavoidable (as the goal of unlearning is to "revert" optimization), the fast divergence speed of gradient ascent is a rather undesired feature and motivates the proposal of our NPO method which diverges at a slower speed.
3 Negative Preference Optimization
We introduce Negative Preference Optimization (NPO), a simple drop-in fix of the GA loss. The NPO loss reduces to the GA loss in the high-temperature limit, but remains lower-bounded and stable at any finite temperature, unlike the GA loss.
We take inspiration from preference optimization (Rafailov et al. 2024) and derive NPO as a method of preference optimization with negative examples only.
4

---

Published as a conference paper at COLM 2024
Negative Preference Optimization. In preference optimization (Ouyang et al. 2022 Bai al. 2022 Stienmon et al. 2020 Rafailov et al. 2024), we are given a dataset with preterence feedbacks D paired {x,, wi Mi hi(m)) where (Vi.wr are two responses to x; generated by pre-trained model Tgg, and the preference V/tw > V:j1 is obtained by human comparison. The goal is to fine-tune T using ' paired to better align it with human preferences. A popular method for preference optimization is Direct Preference Optimization (DPO) (Rafailov et al. 2024), which minimizes
(DP),p(0)
1 -
_ D logoo(Blog)
The (Vw log
7Tp (vw xx]
(v x) (er xx)
(2)
Here, (1)) = 1/ e is the sigmoid function, B > 0 is the inverse temperature, and Tre is reference model.
Unlearning as preference optimization. We observe that the unlearning problem can be cast into the preference optimization framework by treating each (x) Prc only providing a negative response Vil without any positive response Vi,w- We then simply
ignore the Yw term in DPO in Eq. (2) and obtain the Negative Preference Optimization (NPO) loss:
(NPO,((0) = - D logo (-
xo(y|x)
)
)] Erc ED log (1
2
TT(y)x))
)
(3)
ne (ylx)
Fee (y)
Minimizing LNO,, ensures that the prediction probability on the forget set (T [y;)x;)) is as small possible, aligning with the goal of unlearning the forget set.
Connection with gradient ascent. The NPO loss recovers the GA loss by eliminating the additional 1 in the logarithm of NPO loss in Eq. p,, i.e., replacing log(1 + (TT/ (er)) )) to log((Tza/ / /tee/T/geee get (²) Furthermore, we show that the NPO loss also reduces to the GA loss in
the limit of & 0, indicating that NPO is strict generalization of GA.
Proposition 1 (NPO reduces to GA as & - 0). For any 0, we have
2 log (0) Drc. [log Trefly x)]
lim 8-0
(NPO,
does not depend on e
Moreover, assuming Ty(y x) is differentiable with respect to have
lim VoLNro, (0).
$-0
The proof of Proposition 1 is deferred to Appendix B.1
Figure provides an illustration of
the reduction from the NPO loss to the GA loss as $
Stability of the NPO loss. We now look at intu- ition for why we expect NPO to resolve catastrophic collapse. One limitation of the GA loss is its un- boundedness from below (as the negation of the cross-entropy prediction loss which is unbounded from above). The NPO loss resolves this issue and remains lower-bounded for any finite B 0.
GA
0=
Rotain Distance
0.35
8=0.01
0
0=0.1
f=1
0:38
0.16
0.35
Furthermore, the gradients of NPO and GA are as follows:
0.06
2.02
2.04
Porget Distance
0.28
0.03
2.30
0.44
Figure 3: Retain difference versus forget difference for GA and NPO with vary- ing levels of & in the binary classifica- tion experiment with a = 1. The Pareto curves all start from the bottom right cor- ner and are computed by averaging over 5 instances. We observe that the NPO trajectory converges the GA trajectory as B + 0. More details can be found in Section 4
VeLGA DD Vo [o(y)x)],
(4)
VOLNPO,E IE (x(x, log To (y(x),,
(5)
where Wx(x,y) 2
(y/x) 7 (ylx)
can be interpreted an adaptive smoothing weight- When example (x,y) y) E Drc is already unleared in the sense that no (y(xx) < Trer(y)x), we have Wx(x,y) < 1, that VOLNPO, < / ll2 and thus NPO could diverge much slower than GA.

---

Published as a conference paper at COLM 2024
3.1 Theoretical analysis of divergence speed
We formalize the above intuition by theoretically analyzing the the divergence speed of NPO and GA in a standard logistic regression setting. We consider a binary classification problem (y E (0, with logistic model TT (y = = = sigmoid (x, 0)). The initial model
is denoted Tmm for iiitt E Rd and unlearn the forget set DrG = {x,, y:) by minimizing the GA and NPO loss using gradient descent with stepsize rj for iterations.
{xx,,
Theorem 2 (Divergence speed of GA and NPO). Let X == (xI...., E IR Consider the high-dinensional regime where n1 S d and assume XXT T is invertible. Suppose ||0mitll2 S By, ||x;112 E 1bx, for all E [m:] for some Ba, bx, Bx > O. Let olt) NiPo denote the t-th iterates of gradient descent with stepsize oj on the empirical loss LGA, LNPO, respectively.
(GA diverges linearly) There exist some (Be, dx, Bx deeeenddn constants Co, C2 > 0 such that when max fjj x,)1 S Co/Co /nt,
1 @ [GI nn 1/2 f, C2 ny 1/2 8]. 121.
(NPO diverges logarithmically) Suppose S 1. There exist some (Ba, bx, Bx ))-dependent constants Co C₁, C2, C3 > 0 such that when max /j (xxxx)) Co/ns,
|10 NPO Oinitll E [CIv/T (C2. -pp t 1). 1 CI (Ca (M3-mma t ++1)] 41 2 1.
Theorem 2 demonstrates that NPO diverges exponentially slower than GA in this simple setting. The proof of Theorem is contained in Appendix B.2
4
Synthetic Experiments
4.1 Setup
Dataset. We consider a forget set DrG = [xf, ) and a retain set DRT = (x) y which are both generated from Gaussian-logistic models. More specifically, we assume
0; 1), x ri N N Ia), Ply 1|x]) = sigmoid ((f (t)) T 0, - 1).
xf Mi N La),
Plyf = 1|xx) sigmoid ((xf (4r) To;
(6)
Here we choose 16, 0; -, 1./vd, and 1f -Mr a 1, for some a > 0. We consider two choices of the hyper-parameter w: (1). a = 1, which creates a gap between the Gaussian means of forget covariates (x]) and retain covariates [ ;;; (2). a = 0, which implies that covariates in the forget and retain set are both isotropic Gaussian. We remark that we shift by in the sigmoid function to create a discrepancy in the label frequencies between the forget and retain sets this ensures that the forget labels yf are more likely to be 1, while the retain labels yf are more likely to be O.
Model and training method. We consider random feature model no = 1|x) (i Rex), where W E R228xd is fixed during the training and unlearning process, whose entries are generated i.i.d. from (0 1/d), and 0 E R 128 is the trainable parameter. To generate the initial model Tee and the retrained model Tretto we optimize over using the cross-entropy loss over the entire dataset D DFG U U and the retain dataset DRT, respectively. In the unlearning phase, starting from the initial model Te we perform gradient descent on various loss functions for 2000 steps. We select the learning rate for each method via grid search.
Unlearning methods. We evaluate the performance of vanilla NPO (NPO; minimizing [NPO), NPO plus a retain loss term (NPO+RT; minimizing LRP)+LRT), gradient ascent (GA; minimizing LGA), gradient ascent plus retain loss term (GA+RT; minimizing LGA+ cross-entropy loss of forget and retain sets where the positive labels of the forget set are given by Bern(0.5) (IDK+RT; minimizing +rG+LRT), and DPO plus retain loss term (DPO+RT;

---

Published as a conference paper at COLM 2024
GA-RI
IDK+RT
+
NPC+RT
13
Distancs
Pergye Di
1.22
12
0.88
1.00
110 12:0
13:00
1000
3700
O2E 800 RIN. 0:2 DON
8:30
600
1300
1800
(x1
(42)
16:3)
Figure 4: Forget distance and retain distance versus optimization steps for 1 (al, a2, a3) and a (b). Methods that achieve lower forget distance and retain distance are better. The errorbars in (al, a2, b) denote the #1 standard deviation over instances. The Pareto curves in (a3) all start from (1.70,0.02) 0.02), and are averaged over 5 instances.
minimizing CDP where the positive labels are given by (0r5(. We conduct the grid search to select the optimal / for NP'O-ased and DPO-based methods. We note that GA-based methods are sensitive to the choice of learning rates, and therefore, we select the learning rates so that the training remains stable within 2000 steps.
Evaluation metrics: forget distance and retain distance. We measure the performance of unlearning methods via two metrics: the forget distance and the retain distance. The forget distance is ED (Teetr (-Ix)]| (-)()=)))x)))) the KL divergence between the retrained model Teeer and unlearned model Ti on the forget set. Similarly, the retain distance is given by DED D (Teetr(( [xxx)]] Ao (-)x)). Ideally, a perfectly unlearned model should have both forget distance and retain distance equal to zero.
4.2 Results
NPO avoids catastrophic collapse. As illustrated in Figure 4 (al) and (a2), all methods except for IDK+RT reach small forget distance (less than 0.005) within 1200 steps. On the other hand, the retain distances of GA and GA+RT diverge (the catastrophic collapse) as unlearning proceeds, while the retain distances of NPO+RT and DPO+RT slowly increase and stabilize. This suggests that NPO+RT and DPO+RT are more stable compared with GA-based methods, in accordance with the theoretical findings in Theorem
NPO+RT achieves a better Pareto frontier. Figure shows that NPO+RT outperforms other baseline methods by achieving better Pareto frontier. Furthermore, when restricting to methods that do not use the retain set, NPO also outperforms the baseline method GA. scenario where the covariate distributions for forget and retain sets are identical, resulting in equal forget and retain distances. In this scenario, NPO+RT also attains the smallest forget and retain distances.
Figure (b) illustrates the
Experiments on the TOFU Data
5.1 Experimental setup
Dataset and metrics. We evaluate unlearning methods on the Task of Fictitious Unlearning (TOFU) dataset (Maini et al. 2024). It contains 200 fictitious author profiles, each consisting of 20 question-answer pairs generated by GPT-4 based some predefined attributes. These fictitious profiles do not exist in the pre-training data, providing a controlled environment for studying unlearning LLMs. TOFU introduces three levels of tasks, each aiming to forget 1%, 5% and 10% of the data, referred to as Forget01, Forget05, and Forget10, respectively. We measure the effectiveness of unlearning methods via Forget Quality and Model Utility as in Maini et al. (2024). Forget quality assesses how well the unlearned model mimics the retrained model (defined as the model trained only on the retain set), while model utility measures the general capacities and the real-world knowledge of the unlearned model. Since the forget quality is defined as the p-value of the Kolmogorov-Smimnot test, which tests the similarity between some distributions generated by the unlearned model and the retrained one, we treat a forget quality greater than 0.05 as evidence of a meaningful forgetting. More details are deferred to Appendix E.1.1 and Appendix F.1.2
7

---

Published as a conference paper at COLM 2024
Unlearning methods. We compare the NPO-based methods with three variants of GA: GA (Jang et al. 2022 Yao et al. 2023), GA plus a retain loss (GA+RT), and GA plus KL-divergence regularization (GA+KL). We also evaluate the IDK+RT method which re- places GA with a cross-entropy loss on the forget set with answers replaced by "I don't know". Besides, we examine DPO and its regularized variants (DPO+RT, DPO+KL.), as well as KTO LLthayaraj et al. 2024) and its variant (KTO+RT). All experiments on TOFU are See Appendix
conducted on Llama-2-7B-chan [Touvron et al. 2023
E.1 for more details.
Experimental details For all experiments TOFU, we use Llama2-7b-chat model (Tou- vron et al. 2023). All experiments are conducted with two A100 GPUs. We use A with a weight decay of 0.01 and a learning rate of 10 5 in all finetuning, retraining, and unlearning experiments, which agrees with the setting in Maini al. (2024). We use an effec- tive batch size of 32 for all experiments. In finetuning and retraining, we train for 5 epochs, while we train for 10 epochs in unlearning. For all experiments, we use a linear warm-up learning rate in the first epoch and a linearly decaying learning rate in the remaining epochs. When computing the ROUGE- Gerclll value, normalized probability and the Truth Ratio, we use at most 300 question-answer pairs randomly sampled from the dataset, following the setup Maini et al. 20244.
finetuned
GA+
KL
NPO-RT RT NPO- KL.
DPO DPO-RT
DPO+KL KTD
KTO
GA
IDK-RT KT
NPO
Forget Quality
0.7
0.6
0.5
Libeer Seale
0.4
Log Srel
03
16-
Liserrnndee Log Stalt
02
031
Libee Seste Log Sodl
364
10.1
1631
1n31
1616
1636
1+31
1n21
1622
Model Utility
Figure 5: Forget quality versus model utility across different forget sizes (1%,5%) and 10% of the data). Each subfigure employs a dual scale: linear scale is used above the gray dotted line, while log scale is applied below it. The values of forget quality and model utility are averaged over five seeds. Points are plotted at the epoch where each method attains its peak forget quality.
5.2 Results
NPO-based methods achieve the best trade-off. Figure illustrates the trade-off between forget quality and model utility for various unlearning methods in the Forget01, Forget05, and Forgetl0. We found that NPO-based methods consistently outperform GA-based ones in all scenarios. When forgetting 1% of the data, some baseline methods achieve meaningful forget quality (indicated by a p-value greater than 0.05). Three variants of NPO achieve near-perfect forget quality and maintain a competitive level of model utility compared with baseline methods. In Forget05, the NPO-based methods are the only ones that attain forget quality above 0.05. Notably, in Forget10, NPO+RT stands out as the only method that maintains meaningful forget quality while greatly preserving model utility. In contrast, all baseline methods fail to achieve forget quality above 0.05.
NPO avoids catastrophic collapse. Figure 6 illustrates the evolution of forget quality and model utility along the unlearning process. Tn Forget01, both GA and GA+RT attain their highest forget quality at the sixth gradient step, but their performance subsequently declines drastically. Similar trends happen Forget05 and Forget10, where the forget quality of
GA and GA+RT initially ascends to maximum, albeit still below 0.05, before rapidly diminishing to an exponentially small magnitude. Therefore, employing GA-based methods in practice often entails early stopping prevent catastrophic collapse. However, a practical challenge is that the stopping time can be highly instance-dependen and does not follow discernible pattern. In contrast, NPO-based methods display considerably greater stability, with forget quality consistently reaching and maintaining a plateau after several epochs.
8

---

Published as a conference paper at COLM 2024
Forget Quality
0.9
86
0.3
0.6
84
0-3
0.3
16:6
0.3
167
1e:13
163
16-15
1e4
0.6
0.6
0.6
Model Utility
0.5
0.4
0.3
0.3
0.3
NPD
0.2
GA+RT
NPO+RT
IDK+RT
DFO+RT
0.5
0.0
0.0
12 18
24
42
54
12 315
72
915
105
120
Step
Figure 6: Evolution of forget quality (top) and model utility (bottom) across different forget set sizes (1% (left), 5% (middle), and 10% (right) of the data). Each line is averaged over seeds. Each figure the top row employs a dual scale in Figure Forget01, we evaluate the performance of the unlearned model every gradient step, while in Forget05 and Forget10, we evaluate it every epoch.
Some Example Answers
NPO improved diversity and readability. LLMs unlearned via GA-based methods tend
O Fteget uet): Canyoutell me about the occupations of Jeee Park's pamentsr mebeonoogjld
Park's Tather was occupational therap at, mothee
KPC-RT freeonn's fither a sllleiidiion th and his mother wor corked sediasa midieefe..
to output repeated words or gibberish sentences
with unreasonably low diversity Yao et al.
Inceet thaee eviigtermmet
nar
DPCART kose shouthhat sujeet
2023 Moreover, IDK and DPO-based methods tend to show excessive ignorance by outputting "I don't know,' or similar responses to common- sense questions. These answers may be toler-
O Irsten yen te us about the type boolouthee Jerre Vanguaee aniten?
Jeme Vasquer specallos true crime genre...
nart
narr sarr narr parr
cavit: jame vaiquee pr manlyyatte writes the brue garea
toceet Vescusc-ppa apesialas the crime garne
grizzinginan
able if one only wants to prevent LLMs from
ohha explorig themes solataation
and biu betweeet
raaliy and madivass
generating undesirable content. Still, they will
world
presient Stated
à. Goarge Wishingtonn
definitely be unsatisfactory under the stronger
NPC-RT The presitent of the United Stoees was George Washingtonn Halwa
unanimouslyy
position
goal of approximate unlearning, which aims to
Ganii: The first president of United States George iaahiington narrated
mimic the retrained model. We show in Figure 7 that NPO+RT outputs incorrect sentences with
forratie noraacor narrativ sarracor narathe...
Unittdd States Geonge Wiihhigton
opcuet idott knew
similar templates for questions in the forget set
Figure 7: Sampled response to questions in three subsets of TOFU. Yellow: questions; Green: true answer or desired answers; Red: undesired answers.
while generating fluent and correct answers for
other questions, greatly enhancing the fluency and diversity of the generated content.
The role of retain loss. Maini et al. (024) demonstrated that methods incorporating retain set outperform those that solely optimize a loss function based on the forget To further investigate the role of retain loss beyond Maini et al. (2024), we evaluate NPO+RT with the weights of the retain loss varying from too (??). While natural that adding retain loss improves the model utility, we are surprised that the forget quality also grows. Specifically, the forget quality increases as the weight of the retain loss grows from to 2. We conjecture that the retain loss term helps the model preserve answer templates and linguistic structures, while the NPO term forces the model forget some specific facts. Combining these two effects pushes the model to approximate the retrained model by generating outputs with similar templates but incorrect entities. However, further increasing the weight of the retain loss (e-g., from 2 to in ??) leads drop in the forget quality, possibly due to the diminished scale of the NPO term. Notably, in our experiments, the retain loss plays more significant role when we target forgetting a larger fraction of the data (See the middle and right panels of Figure
Forget KL: The larger, the better? * *** * We also examine the Forget KL during the unlearning process in the TOFU dataset. We first observed that while GA and GA+RT tend to induce an explosively large Forget KL along the unlearning process, the NPO- based approaches induce much slower growth of Forget KL (Figure 8). It stabilizes at a moderate level even after several epochs. One natural insight from this distinction is that even in the context of unlearning, larger Forget KI. is not necessarily advantageous.

---

Published as a conference paper at COLM 2024
Rather, moderate and stabilized Forget KL is preferable, which ensures the unlearned models generate fluent outputs with reasonable linguistic structures but incorrect content. This also suggests that Forget KL may not be suitable objective function to maximize for unlearning LLMs, contrary to what was done in some prior literature (Chen & Yang 2023).
5.3 Forgetting beyond 10% of TOFU
GA
Forgetting 20%, 30% and 50% of TOFU. Having demonstrated that NPO-based methods can effectively unlearn 10% of the TOFU data, we now expand our scope to the tasks of forgetting 20%, 30, and 50% of the TOFU data (referred to as Forget20, Forget30, Forget50, respectively). Details about the extended dataset are deferred to Appendix E.1.1 We show Appendix that NPO+RT Is the sole method to exhibit meaningful forget quality (a p-value above 0.05) in Forget20 and Forget30. Even in For- get50, where the vanilla NPO+RT achieves a forget quality around 10 3, still signif- icantly outperforms other methods.
GA
IDK+3T
NP0 NFO-RT
DPO
12
24
36
48
60
72
84
108 1220
Figure The evolution of the Forget KL during the unlearning process the Forget task in TOFU data. Note that the KI. term GA+KI, is di- vergence on the retain set, not the forget set. More experimental details are included in Appendix E.]
Pushing towards the limit: forgetting 50% 90% of TOFU. The TOFU framework al- lows us aim forget at most 90% of the data since at least 10% is left out as the re- tain set for evaluation. We thus ask the question of whether there exist methods that could effectively forget 50%-90% of the TOFU data. We tuned the componential weights for NPO+RT and found that with proper weights, NPO+RT easily attains forget quality exceeding 0.05 and model utility above 0.55 on Forget50 and Forget90, as reported in Figure
0.6
0.5
1.0
0.1
forget50 (Model Utility)
0.01
forget90 Litility)
forget50 (Forget Quality)
0.0
forpet90 (Forget Quality)
2
6
9
Conclusion
Number of Epoch
We propose Negative Preference Optimiza- tion (NPO), simple objective for LLM un- learning. NPO makes steps towards ad- dressing the catastrophic collapse issue in the gradient ascent method. We show that unlearning methods based on NPO objective achieves saate-of-heeart performance on LLM unlearning, and achieves the first effective unlearning result forgetting high percentage of the training data. We believe our work opens up many exciting directions for future work, such testing NPO on more datasets or harder scenarios (such with adversarial prompts). It may also be of interest to generalize the algorithm principle of NPO (preference optimization with negative examples only) to other problems beyond unlearning.
Figure Evolution of forget quality and model utility on Forget50 and Forget90 for NPO+RI with proper componential weights between loss terms.
Acknowledgement
Song Mei is supported by NSF DMS-2210827, CCF-2315725, NSF Career DMS-2339904, and Google Research Scholar Award. The authors would like to thank Baihe Huang, Xuelin Yang for the valuable discussions. The authors would like to thank Jiantao Jiao for sharing supported by the Center for Al Safety Compute
his GPU resources. This research Cluster. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
10


Reviews:
[{"review_id": "V8xaxv0Tbr", "reviewer": "Reviewer_v7L5", "summary": "This paper presents an application of direct policy optimization for unlearning knowledge from a large language model (LLM). Experimental results on TOFU demonstrate that the proposed method improves over existing state-of-the-art methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for engaging with our work and their valuable feedback. We agree that evaluating our method on various datasets and models is beneficial. Following the submission, we assessed the performance of NPO on the Phi model, which was also evaluated in the TOFU paper. The experimental results align with that of Llama. \n\nMore specifically, due to the limited time in the rebuttal phase, we tried the tasks of Forget01 and Forget10 on the Phi model. The fine-tuning and unlearning process follows what was described in the TOFU paper. For each task, we unlearn the model for 10 epochs using a learning rate of 2*10^-5 and a linear warmup in the first epoch.\n\nIn Forget10, we observe a very similar trend as in Llama2-7B model. GA achieves an extremely low Forget Quality along the unlearning process, with the maximal forget quality around 1.83 * 10^-7. GA+RT performs better than GA, but shows a very unstable unlearning trajectory: the forget quality arises somewhere in the middle and then drops drastically. At the end of the 10th epoch, it attains a forget quality of 5.5*10^-9. On the contrary, NPO+RT shows the best forget quality throughout the whole process. In the meantime, the forget quality steadily grows when we unlearn it for more epoch. For example, at the end of the 5-th epoch, the forget quality for NPO+RT is 0.055, and at the end of the 10-th epoch, it is around 0.147, which is a significant number (above 0.05). Meanwhile, the model utility of NPO+RT is high and ranks the highest two among all unlearning methods.\n\nIn Forget01,  we observe a similar phenomenon. The Forget Quality for NPO and NPO+RT keeps growing along the whole process and finally exceeds 0.5, while the forget quality for GA initially grows but falls quickly and finally achieves a value lower than 0.05.\nDue to the space limit and the limitation that we cannot modify the paper or attach a link during the rebuttal phase, we are not able to show you the figure for the evolution of forget quality and model utility. But we are happy to include these additional experimental results in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies an interesting and important problem.\n- The paper is well-written.\n- The experimental findings are interesting.", "reasons_to_reject": "- Analysis over a wider range of datasets in different domains would be very beneficial, as well as conducting experiments with different model architectures", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Se961H0ps5", "reviewer": "Reviewer_yYsm", "summary": "This paper proposes Negative Preference Optimization (NPO), which applies Direct Preference Optimization (DPO) for unlearning certain concepts. It analyzes the differences between Gradient Ascent (GA) and NPO, and shows better performance than GA on the TOFU dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their helpful feedback and insightful comments. Below, we address these points. \n\n> The name of the proposed method seems to be misleading…\n\nWe named NPO since it closely resembles and is motivated by DPO. Additionally, NPO can be viewed as a preference optimization algorithm that dis-favors the forget set (negative response), and thus (relatively) favoring responses dissimilar to the negative response. We will add more discussion in the paper to clarify the name. \n\n> Only a few baselines are used as a baseline approach in the main experiment… \n\nIn Sec 5 and Appendix D, we tested several other methods besides GA, including combinations of GA, KL, retain loss, and non-GA-based methods such as DPO and KTO. After submission, we also tested EUL loss in [1]. In Forget01, at the end of the 10th epoch, EUL achieves a forget quality of 0.0003 while NPO+RT achieves 0.834, both averaging over 5 seeds. Intuitively, the GA components of [1] are problematic: the loss function is unbounded, causing gradient dynamics to diverge quickly. Moreover, there is a forget KL term in the EUL loss function. In Appendix C, we have shown it is better to keep a moderate forget KL than a very large one. This suggests that forget KL may not be an ideal objective for unlearning tasks. The results also show that the method in [2] does not perform as well as NPO variants in TOFU tasks. We are happy to include them in the final version. \n\n>The models used in the analysis are simple linear layers. It is uncertain whether the same results would be obtained using the Transformer architecture... \n\nThe linear setting in our theoretical analysis and synthetic experiments serves as a proof of concept: several experimental observations in the TOFU also appear in this simplest problem setup, i.e., NPO avoids catastrophic collapse and achieves a better tradeoff. We believe extending the theoretical results to the transformer architecture is beyond the scope of this work but would be an interesting direction for future work. \n\n> The paper proposing this dataset has not been peer-reviewed and is only an arXiv preprint. \n\nWhile benchmarks for LLM unlearning (arguably) have not converged yet, we feel TOFU provides a clear definition of the unlearning goal (i.e., mimicking the output of the retrained model) and designs reasonable metrics to evaluate the model’s forget quality and retain utility. Therefore, we chose this dataset even though it is a recent work on arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. DPO was appropriately applied to unlearning.\n2. Section 2 is comprehensively written and easy to understand. \n3. The proposed NPO is a generalization of Gradient Ascent, and it presents evidence of superior performance from the perspective of divergence. \n4. The proposed approach demonstrated superior performance than Gradient Ascent on the TOFU dataset.", "reasons_to_reject": "1. The name of the proposed method seems to be misleading. Since NPO does not use y_w, it cannot be considered as preference tuning, which causes confusion. \n2. Despite mentioning various previous studies in Section 2, only a few baselines are used as a baseline approach in the main experiment. For example, the experiment doesn’t include the performance of [1].\n3. The models used in the analysis are simple linear layers. It is uncertain whether the same results would be obtained using the Transformer architecture, which is commonly used in LLMs. \n4. The validity of the TOFU dataset and the metric used in the main experiment is questionable. The paper proposing this dataset has not been peer-reviewed and is an arXiv preprint.\n[1] Chen et al., Unlearn What You Want to Forget: Efficient Unlearning for LLMs, EMNLP2023.", "questions_to_authors": "\"data already unlearned\" contributes to stability. \nHowever, during the unlearning process, the randomness in training could lead to a larger gradient size for NPO. How does the size of the gradient relate to stability? Can a larger gradient size be considered unstable?\n2. The validity of the TOFU dataset and the metric used in the main experiment: The paper proposing this dataset has not been peer-reviewed and is only an arXiv preprint. The main experiment seems to heavily rely on an arxiv preprint paper.  It is reasonable to utilize the experimental setting and dataset which was already published in the other conference or journal such as [2]. \n3. There exist no baseline methods in the main experiment. I understand that the main baseline is GA, but I think it is necessary to evaluate the performance of the methods mentioned in Section 2.\n[2] Jang et al., Knowledge Unlearning for Mitigating Privacy Risks in Language Models, ACL 2023.", "extra_scores": {}}, {"review_id": "KbLe1KXUZR", "reviewer": "Reviewer_FPT9", "summary": "The paper addresses the problem how to make the model forget after it has been trained on a dataset. Previous methods fail to effectively unlearn due to drastic degradation on the model's general capability/utilities. \n\nInspired by the DPO work, the paper introduces NPO which essentially is a variant of DPO without the positive example. Due to the contrast against a reference model, the training becomes more stable and the resulting model retrains better its original utilities. Synthetic and real experiments demonstrate the empirical effect of the method. Paper also provided rigid theoretical analysis.\n\nPaper is well written, easy to read. \n\nTheoretically speaking, the introduced method seems to be a slight modification of DPO. Empirically speaking, the method is simple and appears effective to address the problems (eg model utility degradation).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for engaging with our work and their valuable feedback. We acknowledge that NPO resembles and is motivated by DPO. However, there is a key difference: the goal of DPO is alignment and it requires pairwise comparison responses, whereas the goal of NPO is unlearning and it requires only a single response to be forgotten.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is well written and easy to understand.\n- Method is simple but empirically effective, with experiments support.\n- Paper additionally provided rigid analysis and proof.", "reasons_to_reject": "- Essentially, the introduced method seems to be a slight modification/specialization of DPO.", "questions_to_authors": "none", "extra_scores": {}}, {"review_id": "ZSzjlOrKJE", "reviewer": "Reviewer_8opk", "summary": "This paper presents negative preference optimization (NPO), which is inspired by Direct Preference Optimization (DPO) and uses the forget set as negative responses, and leaves the positive response empty. Authors did experiments on synthetic binary classification and TOFU unlearning tasks and find NPO achieves strong performance. Authors also theoretically prove progression toward catastrophic collapse is exponentially slower than the gradient ascent method.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment and the insightful feedback. Below, we aim to address the points raised in the review. \n\n>Important task vector unlearning [1] baselines are missing. [1] Ilharco, Gabriel, et al. \"Editing models with task arithmetic.\" The Eleventh International Conference on Learning Representations. 2022. [2] Zhang, Jinghan, Junteng Liu, and Junxian He. \"Composing parameter-efficient modules with arithmetic operation.\" Advances in Neural Information Processing Systems 36 (2023): 12589-12610.\n\nWe will include the reference in the paper. However, we may not be able to compare NPO with these methods in this paper, as the two vector unlearning methods notably differ from gradient-type methods and are therefore challenging to implement in a short time frame. We are happy to compare the performance of these methods with NPO in future work. \n\n\n> The motivation for synthetic experiments is not very clear to me, and the connection between synthetic experiments and TOFU experiments is also not very clear. Would be nice to clarify the motivation and why it is necessary in this work.\n\nWe believe our synthetic experiments complement the TOFU experiments and serve as a proof of concept, as several observations (e.g., NPO avoids catastrophic collapse and achieves a better tradeoff) in the TOFU experiments also appeared in the simple synthetic experiments. Moreover, these observations can be demonstrated in a more quantifiable way in the synthetic experiments. For example, we are able to easily and reasonably define the forget and retain distances based on the prediction accuracy on the forget and retain distributions in the synthetic experiments, while we have to resort to more heuristic and artificial metrics (e.g., the forget quality and model utility) in TOFU tasks. Additionally, our synthetic experiments are on a classification task instead of a text generation task, suggesting the potential applicability of NPO to general probability-based prediction tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple, intuitive, and effective method, supported by both theoretical proof and empirical studies.\n2. Strong unlearning performance while maintaining model utility.", "reasons_to_reject": "1. Important task vector unlearning [1] baselines are missing.\n[1] Ilharco, Gabriel, et al. \"Editing models with task arithmetic.\" The Eleventh International Conference on Learning Representations. 2022.\n[2] Zhang, Jinghan, Junteng Liu, and Junxian He. \"Composing parameter-efficient modules with arithmetic operation.\" Advances in Neural Information Processing Systems 36 (2023): 12589-12610.\n\n2. The motivation for synthetic experiments is not very clear to me, and the connection between synthetic experiments and TOFU experiments is also not very clear. Would be nice to clarify the motivation and why it is necessary in this work.", "questions_to_authors": "N/A", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces negative preference optimization to unlearn a target dataset. Experimental results show that the proposed methods outperformed existing baselines. 

As reviewers pointed out, this work studies a very interesting problem with theoretical results. The paper is also well-written especially Section 2. The method also demonstrates strong unlearning performance while maintaining model utility. 

Reviewers also suggest a few revision comments such as the motivation around synthetic experiments, as well as applicability of the current method on other model architectures. 

The authors have provided detailed clarification and responses to some of these during the discussion period, and I’d like to encourage the authors to further address these in their revised version.
Decision: Accept