Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
Authors: Chongyu Fan
Jiancheng Liu
Licong Lin
Jinghan Jia
Ruiqi Zhang
Song Mei
Sijia Liu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Machine Unlearning
Preference Optimization
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: This work studies the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences (e.g., copyrighted or harmful content) while preserving model utility. Despite the increasing demand for unlearning, a technically-grounded optimization framework is lacking. Gradient ascent (GA)-type methods, though widely used, are suboptimal as they reverse the learning process without controlling optimization divergence (i.e., deviation from the pre-trained state), leading to risks of model collapse. Negative preference optimization (NPO) has been proposed to address this issue and is considered one of the state-of-the-art LLM unlearning approaches. In this work, we revisit NPO and identify another critical issue: reference model bias. This bias arises from using the reference model (i.e., the model prior to unlearning) to assess unlearning success, which can lead to a misleading impression of the true data-wise unlearning effectiveness. Specifically, it could cause (a) uneven allocation of optimization power across forget data with varying difficulty levels, and (b) ineffective gradient weight smoothing during the early stages of unlearning optimization. To overcome these challenges, we propose a simple yet effective unlearning optimization framework, called SimNPO, showing that simplicity—removing the reliance on a reference model (through the lens of simple preference optimization)—benefits unlearning. We provide deeper insights into SimNPO's advantages, including an analysis based on mixtures of Markov chains. Extensive experiments further validate its efficacy on benchmarks like TOFU, MUSE, and WMDP.

Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
Chongyu Fant
Jiancheng Liu
Licong Lin
Jinghan Jia
Ruiqi Zhang' Song Mei! Sijia Liuf-8
Michigan State University
University of California, Berkeley
IBM Research
Equal contributions
Abstract
This work studies the problem of large language model (LLM) unlearning, aim- ing to remove unwanted data influences (e.g., copyrighted or harmful content) while preserving model utility. Despite the increasing demand for unlcarning, a pochiceall-groondde optimization framework is lacking. Gradient ascent (GA)- type methods, though widely used, are suboptimal as they reverse the learning process without controlling optimization divergence (i.e., deviation from the pre- trained state), leading to risks of model collapse. Negative preference optimization (NPO) has been proposed address this issue and is considered one of the state- of-the-art LLM unlearning approaches. In this work, we revisit NPO and identify another critical issue: reference model bias. This bias arises from using the ref- erence model (i.e., the model prior to unlearning) to assess unlearning success, which can lead misleading impression of the true data-wise unlearning effec- tiveness. Specifically, it could cause (a) uneven allocation of optimization power across forget data with varying difficulty levels and (b) ineffective gradient weight smoothing during the early stages of unlearning optimization. To overcome these challenges, we propose simple yet effective unlearning optimization framework, called SimNPO, showing that 'simplicity' in removing the reliance on reference model (through the lens of simple preference optimization) benefits unlearning. We provide deeper insights into SimNPO's advantages, including analysis based on mixtures Markov chains. Extensive experiments further validate its efficacy on benchmarks like TOFU, MUSE and WMDP. Codes are available at https: //github. com/ SAEMMML-Gru//// Stroup/Unlearp -iimle..---S
Introduction
The rapid advancement of LLMs has raised security and safety concerns, including issues related to copyright violations and sociotechnical harms [1-4]. However, retraining these models to remove undesirable data influences is often impractical due to the substantial costs and time required for such processes. This gives rise to the problem of LLM unlearning [5]. To trace its origins, the concept of machine unlearning was initially developed for data removal to comply with privacy regulations such as the "right to forgotten" [6. 7], with carly studies focusing on vision models [8-15]. However, soon adapted to LLMs remove unwanted data and knowledge [4, 5, 16-20].
The current optimization foundation for LLM unlearning often relies on optimization divergence' from the pre-trained state, which refers to the deviation from the converged pre-trained model
Here, we use "divergence" as opposed to "convergence" in model training. aiming to reverse learning for unlearning.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

samples Stronngly-mmmoriidd data
UnLialaan
Unleara
Forget, Qurettyuaityy (FQ)
100
NPO
Paur Ixeo
Sim
75
Hard
Density
0.3
SimNPO
sumple
Weakly Teakly-memorrinn memorized data
0.90
Betber
Privt
50
letrada (Las Stekeo
0.05
Better
sample
0.80
NPO
Retrain
0.
0.5g
0.60
Unlearn boundary
Truth ratio
Model
Utility (MU)
KnowMem on 1),
(a) Schematic overview
(b) FQ vs. data memorization
(c) TOFU, LLaMA-2-chat 7B
(d) MUSE, LLaMA-2 7B
Figure 1: (a) Systematic overview of an LLM (0) post-unlearning using the proposed SimNPO. compared to
NPO [19] and the reference model. (b) Truth ratio distribution of strongly-memorizes forget data points and weekly-demorized data for NPO. SimNPO, and Retrain on the TOFU Forget05 dataset [18] under LLaMA-2- chat 7B; See Sec, 4 for more details. As shown, SimNPO achieves better forget quality (FQ, the number after method) than NPO and exhibits truth ratio distribution closer to Retrain. Note that FQ is a statistical measure quantifying the closeness between the truth ratio distribution of an unlearned model and that of Retrain (with FQ= 1 representing optimal unlearning). (c) (d) Experiment highlights on TOFU Forget0S and MUSE News datasets [4]. Unlearning effectiveness measured by FQ for TOFU and PrivLeak for MUSE, while utility preservation is evaluated using model utility for TOFU and KnowMem on retain data for MUSE (see Table Al). In both tasks, Retrain is the gold standard for unlearning.
to reverse the effects of learning the forgotten data, thereby achieving unlearning [18, 19, 21]. Nevertheless, the lack of control over the divergence rate in unlearning optimization can lead to either uuder-forgetting, where insufficient unwanted data influence is removed, or forgettrg,,,,gg causing a significant loss of model utility in LLMs. Therefore, optimization for L.LM unlearning highly non-trivial. Negative preference optimization (NPO) [19] emerges as an effective approach for LLM unlearning, optimization and its strong performance in current benchmarks such as TOFU [18] and MUSE [4]. Inspired by direct preference optimization (DPO) [22], it treats the forget data points as negative responses, providing lower-bounded unlearning objective. This also induces a gradient weight smoothing scheme to regulate the speed of divergence. We refer readers to Sec. 3 for details.
demonstrated by its better control of the divergence rate during unlearning
Despite the advancements NPO has brought to the optimization foundation for LLM unlearning, our work identifies, for the first time, its potential limitations stemming from its reliance the reference model (i.e., the model prior to unlearning) as the basis for promoting and regulating the optimization divergence. We term this issue reference model bias. See the conceptual schematic overview below.
Fig. 1-(a) illustrates this issue schematically. NPO aims to widen the gap between the unlearned model (Oxpo) and the reference model (0ret). However, the prediction confidence of Ozet varies across samples, as illustrated by the "hard" vs. "easy" unlearning examples along the green line in Fig. 1-(a). Specifically, "hard" examples are those whose predictions under Ozet lie far from the unlearning decision boundary, making them more difficult to forget. In contrast, "casy" examples are already close to the boundary, where further increasing the gap between the unlearned model and Oret could become unnecessary. Yet, NPO may blindly increase the deviation from 0fet (as shown by the blue line in Fig. 1-(a)), causing "easy" examples to move unnecessarily far beyond the unlearning boundary. Meanwhile, "hard" examples remain far from the boundary and require more targeted effort to forget. In other words, relying on the reference model can result in suboptimal unlearning power allocation due to its uniform, deviation-based strategy.
Throughout this work, we ask:
(Q) How can we identify and address the limitations of NPO to enhance its effectiveness?
In response to (Q), we propose a simple yet effective unlearning optimization framework, termed SimNPO, demonstrating that properly removing reliance on reference model can significantly enhance unlearning. This approach also draws inspiration from simple preference optimization in LLM alignment [23]. Additionally, we will provide detailed insights into how SimNPO overcomes the limitations of NPO caused by reference model bias. As shown schematically in Fig. 1-(a), SimNPO outperforms NPO by more accurately identifying the difficulty of unlearning data (i.e., hard vs. easy samples) and allocating optimization power more effectively across different forget samples. Fig. 1- (b) provides experimental evidence, which will be provided in Sec. 4, by comparing the unlearning performance of NPO and SimNPO across forget data points with their unlearning difficulty levels indicated by their memorization levels. The rationale is that the reference model demonstrates varying levels of memorization across different forget samples, making
2

---

harder to unlearn and weakly- samples easier to unlearn. However, NPO may blindly over-allocate unlearning power to these easier samples, thereby hindering the effective unlearning of harder ones. This explains why Fig. 1-(b) shows that NPO performs worse than SimNPO in the strongly-memorizee (hard) forget data, as evidenced by greater deviation from Retrain.
In summary, ours contributions are outlined below:
We revisit the NPO framework and identify its potential weakness-reférence model bias-in LLM unlearning, which can lead to issues such as sensitivity to the reference model's response quality and ineffective gradient weight smoothing. We reveal and justify this bias through a series of analyses/examples, including reference model perturbation, the relationship between unlearning and data memorization, and the impact of forget data length unlearning.
Building on insights into NPO's limitations, we propose an improved L.LM unlearning approach, SimNPO, which extends NPO using reference-free optimization framework, simple preference optimization [23]. Despite its simplicity, our methodology is grounded in rigorous technical rationale, supported by additional synthetic studies and theoretical insights.
We conduct extensive experiments to demonstrate the improvements of SimNPO over NPO across various scenarios, including TOFU [18], MUSE [4], WMDP [3], and defending against relearning- based attacks [24, 25]. Some experiment highlights on TOFU and MUSE unlearning benchmark datasets are showcased in Fig. 1-(e,d).
2 Related work
Machine unlearning. From the perspective of whether the forget data can be inferred from the unlearned model in terms of membership (i.e., a data privacy viewpoint), the widely adopted gold standard for machine unlearning known as exact unlearning, this approach retrains the model from scratch on the original training set with the forget data excluded. However, exact unlearning is challenging in practice due the assumption for access to the full training set and the high computational cost of retraining. To address these challenges, various approximate unlearning methods have been developed [10, 26, 27]. These approaches typically involve model fine-tuning or editing, applied to the pre-trained model, based on the unlearning request. Their effectiveness has been shown in different application domains, including image classification [12, 13, 28, 29], image generation [14, 15, 30], federated learning [31-33], and graph neural networks [34-36].
'Retrain' [8, 11, 13], which we also adopt in this work. Also
LLM unlearning. There has also been a growing body of research focusing on LLM unlearning [3. 5, 16-20, 37-49), aiming to effectively remove undesired data influences and/or model behaviors while preserving the utility for unrelated knowledge generation, and maintaining efficiency without the need for retraining. Applications of unlearning LLMs are diverse, from safeguarding copyrighted and personally identifiable information [16, 38, 50], to preventing LLMs from creating cyberattacks or bioweapons [3. 51]. and reducing the production of offensive, biased, or misleading content [17, 52]. Current unlearning approaches include model optimization- based methods [3, 16, 17, 19- 21, 47-49, 53] and input prompt or in-context learning- based techniques [41, 44, 45]. However, many lack effectiveness, leading to either under- forgettoggg or worr-forget as shown by recent LLM unlearning benchmarks such as TOFU for fictitious unlearning [18] and MUSE for private or copyrighted information removal [4]. Recent studies also show that even after unlearning, models can remain vulnerable to adversarial attacks [24, 54, 55) or relearning from a small number of forget data [24, 25). This evidence suggests that effective unlearning for L.MMs is far from trivial. Among current efforts, NPO (negative preference optimization) [19] stands out as promising method. However, we will show that the advantages of NPO can be limited by the presence of reference model bias (Sec. 4).
Preference optimization. In this work, we advance LLM unlearning through the lens of preference optimization. This is motivated by aligning LLMs with human values, known as reinforcement learning from human feedback (RLHF) [56-58]. However, online preference optimization algorithms are often complex and challenging to optimize [59, 60]. driving interest in more efficient offline alternatives. Direct preference optimization (DPO) [22] introduced an offline approach that eliminates the need for a reward model, sparking the development of several reward-free offline preference objectives [23, 61-65]. Notable methods include RRHF [65], SLic-HF [61], IPO [62], KTO [64], ORPO [63], and SimPO [23]. Among these methods, SimPO is reference- free, length- normalized
3

---

variant of DPO, and we will demonstrate that it is well-suited for integrating into LLM unlearning and improving NPO.
Primer on LLM Unlearning
Problem formulation. Unlearning tasks can take various forms and are typically associated with specific set of data points to be removed, known as the forget set (Dt). These tasks often require a complementary set of non-forgotten data points, known the retain set (D,), to preserve model utility by penalizing the divergence caused by unlearning. As result, the problem of LLM unlearning can be cast as a regularized optimization problem that balances the forget and retain objectives [5, 19]:
minimize D( (e (y]æ; 0)] + (l (e, (1/l;; 0)),
(1)
where 0 represents the model parameters be updated during unlearning, À 2 0 a regularization parameter to penalize the 'divergence' of unlearning, and l and e represent forget and retain losses incurred when using model parameters 0 to generate y given the input
Substantial research has focused on designing and analyzing appropriate forget and retain loss functions to solve problem (1) [4, 5, 16-20]. For instance, let ()(y)x)) represent the prediction probability of the model 0 given the input-response pair (x.y). The retain loss is typically chosen as the cross-entropy-baae
log No (yyar)) whose minimization encourages the model to perform well on the retain data (æt y) E 'D,- In (1), if we specify the forget loss as the negative token prediction loss ( az 0) = log ( whose minimization then discourages the model from learning the forget data (æt, y) E Dr. Minimizing such a forget loss is known as the gradient ascent (GA) method [11, 18]. Similarly, minimizing the regularized loss that integrates GA with the retain loss is known as the gradient difference (GradDiff) method [17, 18, 21].
sequence prediction loss, (-(yl 0)
Negative preference optimization (NPO). A popular optimization framework for solving problem (I) is NPO [19]. It treats the forget data as negative examples in DPO [22], transforming the unbounded GA-based forget loss into @ bounded loss from below, which helps prevent catastrophic collapse, and an @ adaptive weight smoothing applied to the forget loss gradients, enabling more controlled divergence speed in unlearning.
These benefits can be clearly seen from the NPO loss and its gradient as follows:
3 log a (- log
2
<o(v)>>)
ENPO (0) = E(..))
(2)
= Ey(yior; 0), the specified forget luss in (I)
(y)
VOlNPo(O) = EE
V log The (u(
(3)
x (vl())
== (n adaptive weight
where (t) = 1/(1 + e " is the sigmoid function, B > O is the temperature parameter and Teef is the reference model given by the initial model prior to unlearning. Additional insights into 1-@ are given below.
① From (2), the NPO-type forget loss bounded below by 0, i.e., (/ylax; 2 O, whereas the
GA-type forget loss, (r ( 0) = log To(y) (y)..) has no lower bound. Moreover, minimizing it towards (r(y(r; 0) - 0 drives the prediction probability no (y(c)) to decrease, widening the gap between the
prediction probability and the reference model on the forget set, i.e., no (y)l)) < Wrof (yll))
@ As seen in (3), the adaptive weight we (z, y) is typically less than since Te < Tye((y)z) for forgetting. Consequently, NPO's 's gradient yields a more controlled and gradual divergence speed (i.e., deviation from the reference model), compared to GA (with we w y) 1).
In this paper, NPO will serve as the primary baseline for LLM unlearning. Its implementation follows the regularized optimization in (1), where the forget loss & is defined as in (2) and the retain loss &, is the token prediction loss é, (z)22, 0) = - logg n T (y)zz) applied to the retain set.
LLM unlearning tasks and evaluations. Given that the assessment of LLM unlearning may rely on specific tasks, we next introduce the unlearning tasks and evaluation metrics that this work covers. (1) TOFU [18] considers fictitious unlearning on synthetic Q&A dataset. (2) MUSE [4] is designed

---

to remove verbatim or knowledge memorization from News and Books datasets. including both verbatim texts and knowledge sets for unlearning evaluation. (3) WMDP [3] aims to prevent LLMs from generating hazardous content in domains such as biology, cyhersecurity, and chemistry. Despite the differences in evaluation metrics across the above tasks, the assessment broadly falls into two categories. (1) Unlearning effectiveness measures how faithfully undesired data influences model capabilities are removed. For example, it is assessed by the forget quality (FQ) metric in TOFU, which uses a p-value to test the indistinguishabiiity between the post-unlearning model and model retrained on the retain set only, and by privacy leakage LerivLeak) in MUSE, which measures the likelihood of detecting that the model was ever trained on the forget set. (2) Utility preservation evaluates the post- unlearnnnnn performance on standard utility tasks. See Table A1 in Appendix A for summary of the unlearning tasks and evaluation metries.
4 Uncovering Reference Model Bias in NPO
In this section, we highlight key weakness of NPO, which we term 'reference model bias': The incorporation of the reference model in NPO biases the unlearning objective towards enlarging the distance relative (v(lz)) & Troe Trer(ylz)), (y/zc) However, using Tref as the basis for NPO's unlearning criterion can introduce negative effects (L.))(L.)), which we will detail later.
this reference model. As noted in (2), minimizing the NPO loss drives
Before that, we present a warm-up study to illustrate NPO's sensitivity to the choice of the reference model (Oret, used interchangeably with Tret). Specifically, we construct a perturbed reference model, &eer by averaging the original reference model θzef with a randomly weighted model, whose weights drawn from standard Gaussian distribution with zero mean and variance. We then apply NPO using 0 as the reference on the TOFU Forget0S dataset, following the same setup as in Fig. 1-(c). We find that there exists substantial drop in forget quality-from 0.79 (with Oret) to 0.27 (with ('er), while the model utility remains nearly unchanged (0.52 w/ Ozet vs. 0.57 w/ Oret). We refer readers to Fig. A1 in Appendix B for the detailed comparison. This preliminary study highlights the critical influence of the reference model on NPO's unlearning effectiveness. Thus, a deeper investigation into the use of the reference model could offer valuable insights for improving the unlearning optimization framework.
Next, we elaborate on the limitations (L1)-(L.2) introduced by the reference model in NPO.
(L1) Challenge of uneven allocation of unlearning power across forget data. At first glance, driving the unlearned model to deviate from the reference model in NPO, i.e., promoting To (vlx)) << Tree(y) (y)a), seems desirable for unlearning on the forget set. However, the over-reliance on Tref can overshadow the true sample-specific unlearning difficulty, leading to an uneven allocation of unlearning power. We elaborate on this issue through two examples.
Top 50% short-length data
(Example I: Unlearning strongly vs. memorizee forget data.) We first explain (LI) from the perspective of unlearning vs. data memoriza- tion. Consider two forget sets, Dr,1 and Dr.2, where Dt,1 is more strongly memorized by the model than Dr.2- To support these memorization lev- els, we provide detailed experimental settings in Appendix C. With this setup, the prediction loss on Dr,1 is smaller, leading a higher prediction probability Tref- Accordingly. the NPO gradient smoothing term in (3) becomes smaller for Dr,1. meaning NPO allocates less first-order opti- mization power to it. However, Dr.1, being strongly memorized, should ideally receive more unlcarning power. As a result, this uneven focus hinders NPO's ability to effectively forget Dr,1, potentially causing under- unlearning and reducing the FQ of Dr,1 to nearly zero. See Fig. 1-(b) and Table A2 for results.
0.5
Retrobil RaaNiolas1)
Density
Other 50% longer-lnngth data
6.5
NZORLE REENPOLLETT
Truth ratio
Truth ratio dis- tribution of short/long for- get data for NPO, SimNPO. and Retrain on TOFU For- get05. The figure format fol- lows Fig. 1-(b).
Figure 2:
Unlearning short vs. long- rengrrepoone data.) In this example, we evaluate unlearning performance across different types of forget data, categorized by their response lengths (i.e., short vs. long). The motivation stems from the observation that the reference model may exhibit bias toward generating longer, yet lower-quality, responses [23]. Fig. 2 shows that NPO exhibits a greater distance from Retrain when unlearning the top 50% shortest-length forget data, resulting in a lower FQ (forget quality) of 0.58, In contrast, NPO performs better unlcarning for the longer 50% of the
(Example 2:
5

---

forget set, yielding higher FQ of 0.81. The ineffectiveness of NPO in unlearning forget data with short responses will be further analyzed through the lens of mixture of Markov chains in Sec. 5.
Retrain
NPO
SimNPO
NPO
Forget Quality
1,0
0.6
0.99
10
0.5
0.96
0.95
16
10-1
0.0
9.10
0.5
Respouse Length [X]
20
40
60
60
Unlearn Epoch
10
Unleam Epoch
Unleam Epoch
wa of NPO epoch
(b) Teajectory of we vs. epochs
(c) Forget quality vs. opochs (d) Model utility vs. opochs
Figure 3: Experimental evidence of ineffective weight smoothing and utility-drop for NPO on TOFU Forget05
(a) NPO's gradient weights (we) at epoch vs. response length ly]. (b) Trajectory of we for NPO over unlcarning epochs, where box plot represents the distribution of gradient weights over forget samples. (c)-(d) Forget quality and model utility of NPO vs. epochs.
(L2) Lack of gradient weight smoothing in the early stages of unlearning. Another issue intro- duced by the reference model Tie concerns the effectiveness of NPO's s gradient weight smoothing, i.e. Wo(x, y) = (m (y/ [z]rr) ( (y|zx)* + Teet [rer(y(a>) in (3). During the early optimization stage of NPO, we find (x; y) regardless of the varying data-specific unlearning difficulties since the initialization of the unlearned model 0 is given by the reference model. (a, support this finding by displaying the gradient smoothing weights of NPO epoch one for forget data with varying response lengths (Fig. 3a), as analyzed in Example 2, and their trajectory over the course of unlearning epochs (Fig. 3b). As shown, the gradient smoothing weights of NPO show large variance, but most values are concentrated around wa (z, y) & 1 at epoch one. This implies that NPO behaves similarly to GA in the early stage of unlearning. potentially causing large utility drop even if the weight decreases in later optimization. Fig. 3-(c,d) justify the above by presenting FQ and model utility of NPO on TOFU against unlearning epochs. As shown. NPO tends to cause a larger utility drop at early epochs compared to SimNPO, the improved alternative to NPO in Sec.
5 SimNPO: Method and Rationale
Motivation of SimNPO and its forget objective. The simplest solution to mitigating NPO's
reference model bias to directly remove nref from the gradient (3), setting Rref = 0. However, this variant would be ineffective, as the reference-free gradient reduces to GA, with wa (z. y) = 1. This negates NPO's s advantages. To develop a better solution for improving NPO, we revisit the context of preference optimization and investigate whether the reference model can be excluded while still retaining the unlearning benefits provided by NPO. Our idea parallels how NPO was originally inspired by DPO [22]. We adopt SimPO [23], reference- free alternative to DPO, as the optimization framework for unlearning, leading to the SimNPO (simple NPO) method.
The key difference between SimPO and DPO lies in their reward formulation for preference opti- mization. In DPO, the reward formulation is given by the comparison with the reference model, ie., Boog(no To (oo(((To))))) (y) (re This formulation was used by NPO. In contrast, SimPO takes reference-fee but length-normalized reward formulation: (3/lyl) log no (y) ar), where |y/ denotes the response length.
Taking the inspiration of SimPO, we can mitigate the reference model bias in NPO by replacing its reward formulation 3 log( (o(y (yl/yy//////le /vee (yl x)) ) in (2) with the SimPO-based reward formulation (B/ly)) ())) This modification transforms (2) into the SimNPO loss:
2 B log a (- log no(ul() (oo(l(x)>>>]] ]
LsinN (0)-
(4)
where 7 2 0 is the reward margin parameter, inherited from SimPO, which defines the margin of preference for desired response over dispreferred one. However, unless otherwise specified, we set 7 0 to align with the NPO loss (2). This is also desired because introduces a margin to the prediction loss -(3/ly]) log To (y(aa)). Consequently, a larger y requires greater compensation to further suppress token prediction, enforcing a stricter unlearning condition. This can accelerate the utility drop during unlearning. See Fig. A2 of Appendix D for the ablation study of hyperparameters. The NIO loss (4), when integrated in (1), forms the SimNPO method.

---

Addressing NPO's limitations one by one. Similar to NPO, the SimNPO loss (4) bounded from below, with minimum value of O. Approaching this minimum drives the unlearning. However, the key distinction of SimNPO from NPO is its forget data-aware, length-
Insights into SimNPO:
normalized reward formulation, (3/lvl) log To (x) in (4). This results in an improved gradient smoothing scheme. Specifically, the gradient of the SimNPO loss (with y = 0) yields:
2(T (1)x)))) 8//vl
V VelsimNPo p, 1 (Te a =))=/
Vo log (v|[=)]
(5)
19
(2)
See Appendix for derivation. Similar to NPO in (3), the gradient in (5) can be divided into two components: weight smoothing (w/g) and GA. However, in SimNPO, the weight smoothing is no longer influenced by the reference model and instead normalized by the length ly]. This introduces two key advantages (a)-(b) below, in response to NPO's limitations (L1)-(L2).
(a) SimNPO leverages the (data-specific) response length guide for unlearning power allocation. For instance, when |y| is large, less optimization power is allocated, helping to avoid the uneven unlearning power allocation across forget data with varying response lengths, as exemplified in Fig. 2. In the extreme case where B + O, the SimNPO's 's gradient reduces to weighted GA: VolsinNPo (0) + ()) //////a Vo log [y]=)))]] This is different from NPO, which becomes GA as 8 + O. Fig. A3 in Appendix F empirically demonstrates the advantage of length normalization in SimNPO for unlearning. As shown, SimNPO outperforms NPO in both forget quality and model utility, coming closest to Retrain. Even in the special case where / = 0 (i.e., Weighted-GraddD the length normalization provides benefits over GradDiff.
Epoch
SiINNO
Epoch of SimNPO Peusson: -051
Epoch 3 of SimNPO
Epoch 10 of SimNPO
SinNPO's WG
0.75
Peanson:1100 -103
Peasson: 0031
0.002
Peasoz:
0.50
0.5
0.1
0.25
0.0
0.0
0.000
50
Esoch of NPO
Epoch 2 of NPO
Epoch 3 of NPO
Epoch 10 of NPO
asrargg Pearson:
Peorsont 051
Pesrson: 0.78
0.02
Peamon:
NPO's Wg
0.25
0.95
0.00
0.00
50
50
75
75
Response Longth |y|
Figure 4: Gradient weight smoothing of NPO (We) and SimNPO (w²) vs. forget data response length lvi across different epochs (1, 2, 3, and 10) on TOFU Forget05. The Pearson correlation in the upper right corner indicates the relationship between gradient weight smoothing and response length. The SimNPO's weights w' have been rescaled (by x x10) for ease of visualization.
(b) In addition, the reference-free, tiveness during unlearning. It can be shown from (5) that w'( y) < 2/ly]. with the distribution of weights w/ (1, y) depending on the specific forget data samples. This contrasts with NPO, where the weight distribution concentrated around we wa y) = during the early unlearning stage. Extended from Fig. (b))))(b), Fig. 4 provides detailed comparison between the gradient weights of SimNPO and NPO. We find that SimNPO tends to prioritize short-length forget data that are initially harder to forget during the first two unlearning epochs. At later epochs, the gradient weights become more uniform, reflecting that SimNPO can then treat different forget data with even optimization power. This trend is different from NPO, which assigns more uniform gradient weights early on and starts to account for data-specific difficulty only in the later stages of unlcarning. Besides the above ad advantage, the overall unlearning speed compared to NPO. This is supported by the divergence rate from the pre-trained state shown in Fig. A4 and our theoretical discussion in Appendix G.
weight smoothing prevents early-stage ineffec-
also find that SimNPO's new weight smoothing scheme does not compromise
Further analyses via a mixture of Markov chains. In addition to the above insights, we further validate SimNPO's P ad advantages to overcome NPO's ' limitations (Sec. 4) using synthetic setup. For ease of controlling the unlearning difficulties of different forget data points, we consider the problem of unlearning on a mixture of Markov chains with a state space of size 10 (s 1, 1,..., 10). The relain
distribution consists of Markov chains that transition uniformly among states 21, The forget
distribution is a mixture of two components: Forgetl, where the chains transition uniformly among
7

---

{4,5, 6), and Forget2, where they move uniformly among 17,8,9). A small leakage probability allows the chains transition outside their designated states occasionally, including state 10. which is not a designated state for any of the chains. We generate 10,000 samples for the retain distribution and 5,000 samples each for Forget and Forget2. A GPT-2 model is pretrained on these samples and serves as the initial model. We apply NPO and SimNPO to unlearn the forget distributions. Forget and retain performance is evaluated using the KL-divergence between predicted and true transition probabilities of the Markov chains. See Appendix HI for details. We present our results in Fig.5 and summarize the insights below.
SimNPO achieves more balanced unlearning
across data of varying lengths compared to NPO.
To validate this, we set the retain distribution and
Forget1 with sequence length of 20, while For-
LPD Parp??
un hoyrt 2nsto--rogii
get2 is assigned a shorter sequence length of 5, representing mix of long and short responses. 5 (a) shows that NPO exhibits worse trade- off between retain distance and forget quality on short responses (i.e., Forget2) compared with SimNPO. That is, to achieve the same forget qual- ity on Forget2 as the retrained model (with forget quality 0. 0.44), NPO incurs a higher retain distance than SimNPO. As a result, NPO has an overall larger retain distance when unlearning the entire more consistent performance across Forget1 and Forget2, with less variance in its tradeoff.
2Y880 Japa2
90.00
6000
Renan bid00s
(a) Different length. (b) Different memorization.
Figure 5: Tradeoffs between forget quality (higher 1 is better) and retain distance (lower 4 is better) along the unlcarning path of NPO and SimNPO in the synthetic experiments. The symbols (+, near the y-axis of both figures indicate the performance of the retrained model on Forget1 and Forget2, respectively.
Forget distribution. In contrast, SimNPO shows
SimNPO achieves more balanced unlearning across data of varying memorization compared to NPO. In the second case, we set the retain distribution, Forgetl and Forget2 all with sequence length of 20. However, we exclude Forget2 during pretraining. This setup simulates a scenario where the initial model (i.e., the reference model in NPO) exhibits varying levels of memorization for the forget data: strongly memorized Forgetl against Forget2. Fig. 5 (b) shows that NPO exhibits a larger gap between Forgetl and Forget2 for the same Retain distance, leading to over-unlearning veakly-memorte (as shown by the comparison between NPO-Forget2 vs. MonNPPO-Forget. and under-unlcanning proongl-mmeorize data (as shown by the comparison between
data
vs. MiNNPO-Forgetl) SimNPO has a better balance during unlearning across data with varying levels of memorization.
Experiments
6.1 Experiment setups
Datasets and methods. We evaluate unlearning tasks on three benchmark datasets: TOFU [18], MUSE [4], and WMDP [3]. TOFU includes 'Forget05' and 'Forget10' scenarios, representing 5% and 10% forget sets, respectively. MUSE focuses on 'Books' and 'News' forgetting scenarios, while WMDP targets knowledge-basee unlearming of hazardous biosecurity information.
LLM unlearning methods and evaluation. We evaluate a range of unlearning methods, including Retrain, SimNPO, NPO, GA, and GradDiff. In addition, we incorporate several task-specific approaches: the rejection-based method IDK, which replaces positive responses DPO with generic answers such as "I don't it know" [18], and RKLD [48] in the TOFU; the Task Vector method used in MUSE [4]; and the representation misdirection unlearning method RMU in WMDP [3]. Evaluation metrics for each benchmark are summarized in Table AI and further detailed in Appendix 1.2. For the
relcarning attack, we use 20% of the TOFU Forget05 set and retrain over three epochs. Please refer to Appendix 1.2 for full experimental details.
6.2 Experiment results
Performance on TOFU. In Table 1, we present the unlearning performance of SimNPO and its various baselines on TOFU Forget05, covering both effectiveness metrics and utility metrics as shown in Table A1. 'FQ' stands for forget quality, and 'MU' represents model utility. These two metrics serve as the primary performance indicators for LLM unlearning on TOFU. SimNPO outperforms NPO in both FQ and MU, and is the closest approximate unlearning method to Retrain. Except for NPO and RKLD, the other unlearning baselines (GA, GradDiff, and IDK) are not effective, as implied by their FQ values being smaller than 0.01, where FQ indicates the p-value for rejecting the
8

---

Table 1: Unlearning performance on TOFU Forget05 using the LLaMA2-7B- model. 'Prob. indicates the probability metrics, as summarized in Table AI, with forget quality (FQ) and model utility (MU) serving as the primary metrics. Results are averaged over five random trials. The best FQ and MU are highlighted in bold.
Univaming Fffieacy
Uniliy Provervatio
Method
B.Bogg-Lif
Forget Set (1-Probu?
cutor"
PQT
Rowae L1
Real Autlees Prob.T
Truth ruiof
Rouge LT
World Face
Proki
Trush satle
Rouge L.
Retais Set
Prub1
Truth rutiot
MU1
Orggnal Retrain
1004
0100
11:05
0.00
1.91
U1.44
0.5%
0.51
0.45
01.55
1.96
1199
0.48
0662
0.61
0.85
0.66
1,30
0.92
0.44
0.57
0.90
0.45
0.54
0.97
0.99
0.48
0.62
GA
1.00
1
08.6
1.9e-9
0.00
0.20
0.40
0.00
0.30
0.28
1000
0.00
0.15
0.00
GnalDeil
1.00
3.609
0.39
0.81
0.46
0.39
0.49
0.48
0.56
RKLD
1000
0196
0.79
0.93
0.51
0.47
0.58
0150
0.50
NPO
0.73
0.94
0.67
0.79
0.91
0.50
0.62
0.90
0.50
0.61
047
0.51
0.44
0.57
SHINPO
0.54
0.97
0.00
0.59
0.90
0.30
0.64
0.90
0.48
0.60
0.54
0.56
0.44
0.58
indistinguishabiiity between the unlearned model and Retrain on TOFU. In Table A5 of Appendix we also provide examples of model responses after unlearning using SimNPO, Retrain, and NPO, along with label to degenerate. We observe that, in some cases (e.g., responses against the input queries Q1 and Q2 in Table A5), the NPO-unlearned model generates repeated texts in response. While this repetition does not reveal the information intended for unlearning, it differs noticeably from Retrain. In contrast, SimNPO produces unlearning responses more closely aligned with those generated by Retrain. More results on TOFU Forget10 are Table A3 of Appendix I.3.
Performance on MUSE and WMDP. Table com- pares SimNPO with other methods, on MUSE News and Books, with evaluation metrics in Table AI. Com- pared NPO, SimNPO preserves higher utility while achieving stronger unlcarning. On D,., KnowMem is 39.65 (News) and 48.27 (Books), while on D,, is 44.84 (News) and 0.00 (Books). SimNPO also attains a
Table 2:
Performance of various umlearning methods on MUSE News (LLaMA2-7B) and MUSE Books (ICLM-7B).
Unleaminu Elfases
Fuility
Method
VartMlen Dy (
Knowllen
Prkteik (- On
KnowMen P,
D2 (1)
News
Original Retnin
58.20
62.93
-98.71
$4.31
20.75
35:52
0.00
53.79
PrivLeak value closer 0 than NPO (72.93 for News, -31.17 for Books), indicating it better approximates
GA
0.00
0.00
20.14
0.00
Gos
4.85
31:29
108.12
29:21
Vester
77.42
58.76
-100.00
47.04
complete data removal [4]. Compared to other meth- ods, SimNPO strikes the best balance between utility and unlearning. We further evaluate sequential unlearn- ing on MUSE News (Fig. AS in Appendix I.4), where SimNPO consistently outperforms NPO as requests in- crease. Due to space constraints, we present SimNPO's s performance on the WMDP dataset in Appendix 1.5.
255
56.11
10891
37.55
SimNPO
2.34
4-:8
12.91
34.55
MUSE Books
Original
58.32
-56.32
67.01
78,00
GA
0.00
0.00
-26.01
0.00
0.00
6,00
24.59
0.13
Tad Vissur
99.31
35.55
-83.7
62.55
IPP
0000
0.00
-3.17
2371
SimNP0
Unlearning robustness against length-variant relearning attacks. Re- cent studies [24, 25] show that unlearning methods are vulnerable to relearning attacks, where forgotten information can be recovered by fine- tuning on subset of the forget set. We evaluate SimNPO's 's robustness against such attacks, showing it to outperform NPO, especially for short- length response data. Fig. presents the forget quality of SimNPO and NPO under relearning attacks against the number of relearning epochs. Relearning is performed on the forget subset, which is either the short- est 20% of responses from the TOFU Forget05 dataset or equal-size random subset. We refer to these attacks as 'shortest-relcan and 'random- relearn', respectively. The random-relearn case is conducted 5 times, with both average robustness and variance in Fig. 6. As we can see, SimNPO demonstrates improved robustness over NPO, evidenced by higher forget quality and a slower decline in forget quality as the relearning epoch increases. NPO is less robust against the shortest-relearn attack compared to the random-relearn attack. In contrast, SimNPO is resilient to both types of relearning. This is expected since SimNPO addresses the limitation (L.1,, explained in Sec.4.
1.0
Forget Quality
0.8
0.4
0.2
0.0
Relearning Epoch
Figure 6: NPO and Sim- NPO under random/shortest releam attack vs. epachs on TOFU Forget05.
7 Conclusion
We identified a reference model bias in negative preference optimization (NPO) that limits un- learning effectiveness. To address this, we proposed SimNPO, simpler framework leveraging preference optimization without reference model. SimNPO consistently outperforms NPO across

---

standard benchmarks such as TOFU, MUSE, and WMDP, and demonstrates additional advantages in unlearning robustness and the application to reasoning model unlearning.
Broader Impact
On the positive side, we have demonstrated the utility of preference optimization in machine un- learning. This connection enables more efficient unlearning operations in LLMs, improving data privacy protections and supporting compliance with regulatory requirements. Additionally, given the relationship between preference optimization and model editing, work encourages further exploration these areas, contributing to the development of models that are easier to customize and become safer deploy. On the negative side, the methods developed could be misused to selectively erase "essential" (rather than "unwanted") concepts or knowledge, raising ethical and legal concerns. To mitigate this risk, it is essential to ensure that unlearning applications adhere to strict ethical guidelines to prevent misuse. We hope our research fosters the development safe, reliable, and human-aligned LLMs.
Limitations
While SimNPO mitigates the reference model bias present in NPO and improves gradient weight smoothing to better adjust divergence speed based on the varying unlearning difficulties of forget data samples, both frameworks still rely on promoting divergence achieve unlearning. This reliance inevitably results in some degree of utility loss. This limitation becomes especially evident in knowledge unlearming or model capability removal scenarios, such as in the WMDP unlearning benchmark. Consequently, SimNPO has yet to fully resolve the challenge of balancing unlearning effectiveness with model utility. Additionally, establishing theoretical guarantees SimNPO remains an important area for future research.
Acknowledgement
Fan, J. Liu, J. Jia, and S. Liu were supported in part by the National Science Foundation (NSF) CISE Core Program Awards IIS- 2207520 and IIS-2504263, the NSF CAREER Award IIS-2338068, the ARO Award
10343, the Amazon Research Award for Al in Information Security, the Open Philanthropy Research Award, and the Center for AI Safety (CAIS) Compute Award. We also extend gratitude to the MIT-IBM Watson AI Lab, IBM Research for their support in this project. L. Lin, R. Zhang, and Mei were supported in part by NSF CCF-2315725, NSF Career DMS-2339904, ONR 500014-24-S-BO0) Scholar Award. We also thank the support from the Center for Al Safety Compute Cluster. Finally, express appreciation Yuguang Yao for his help in figure plotting.
Amazon Research Award, and Google Research
References
[1] Y. Huang, L. Sun, H. Wang. Wu, Q Zhang, Y. Li, C. Gao. Y. Huang et al., "Position: TrustLLM: Trustworthiness in large language models," in Proceedings of the 41st International Conference on Machine Learning, ser. Proceedings Machine Learning Research, vol. 235, 21-27 Jul 2024, pp. 20 166 270.
[2] B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer al., "Decodingtrust: A comprehensive assessment of trustworthiness in gpt models." in NeurlPS, 2023.
[3] N. Li, A. Pan, Gopal, S. Yue, D. Berrios, A. Gatti, J. Li, A.-K. Dombrowski, Goel, L. Phan et al., "The wmnp benchmark: Measuring and reducing malicious use with unlearning." arXiv preprint arXiv:2403.m 403.03218, 2024.
[4] W. Shi, J. Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang, "Muse: Machine unlearning six-way evaluation for language models," arXiv preprint arXiv:2407.0644 040000 2024.
10

---

[5] S. Liu. Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao. Y. Liu, X. Xu, H. Li et al., "Rethinking machine unlearning for large language models," Nature Machine Intelligence, pp. 1-14, 2025.
[6] J. Rosen, "The right to be forgotten." Stan. L. Rev. Online, vol. 64, p. 88, 2011.
[7] C. Hoofnagle, B. van der Sloot, and Z. Borgesius, "The european union general data protection regulation: what it is and what it means," Information & Commumications Techmology Law, vol. 28, no. I, pp. 65-98, 2019.
[8] Y. Cao and Yang, "Towards making systems forget with machine unlearning," in 2015 IEEE symposium on security and privacy. IEEE, 2015, pp. 463-480.
[9] A. Warnecke, L.. Pirch, C. Wressnegger, and K. Rieck, "Machine unlearning of features and labels," arXiv preprint arXiv:2108. 1577, 2021.
[10] L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Chor H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, "Machine unlearning." in 2021 IEEE Symposium on Security and Privacy (SP). IEEF, 2021, pp. 141-159.
[11] A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot, "Unrolling sgd: Understanding factors influencing machine unlearning." " in 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 2022, pp. 303-319.
[12] M. Kurmanji. Triantafillou, J. Hayes, and E. Triantafillou, "Towards unbounded machine unlearning," Advances in neural information processing systems, vol. 36, 2024.
[13] J. Jia, J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu, "Model sparsity can simplify machine unlearning," in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[14] R. Gandikota, J. Materzynska, J. Fiotto-Kaufman and D. Bau, "Erasing concepts from diffusion models," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2426-2436.
[15] C. Fan, J. Liu, Y. Zhang, D. Wei, Wong, and S. Liu, "Salun: Empowering machine unlearning via gradient-basse based weight saliency in both image classification and generation," in International Conference on Learning Representations, 2024.
[16] R. Eldan and M. Russinovich, "Who's harry potter? approximate unlearning in Ilms," 2023.
[17] Y. Yao, X. Xu, and Y. Liu, "Large language model unlearning." " arXiv preprint arXiv:2310. 100.0633, 2023.
[18] Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter, "Tofu: task of fictitious unlearning for Ilms," 2024.
[19] R. Zhang, L. Lin, Y. Bai, and S. Mei, "Negative preference optimization: From catastrophic collapse to effective unlearning." arXiv preprint arXiv:2404.05868, 2024.
[20] J. Jia, Y. Zhang, Y. Zhang, J. Liu, B. Runwal, J. Diffenderfer, B. Kailkhura, and S. Liu, "Soul: Unlocking the power of second-order optimization for Ilm unlearning," arXiv preprint arXiv:2404.1 18239, 2024.
[21] B. Liu, Q. Liu, and P. Stone, "Continual learning and private unlearning." in Conference Lifelong Learning Agents. PMLR, 2022, pp. 243-254.
[22] R. Rafailov, A. Sharma, F. Mitchell, C. D. Manning, S. Ermon, and C. Finn, "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems, vol. 36, 2024.
[23] Y. Meng, M. Xia, and D. Chen, "SimPO: Simple preference optimization with a reference-free reward," The eiirth- Annual Conference on Neural Information Processing Systems, 2024.
11

---

[24] A. Lynch, P. Guo, A. Ewart, S. Casper, and D. Hadfield-Memell, "Eight methods to evaluate robust unlearning in llms," arXiv preprint vrXiv:2402.16835 2024.
[25] S. Hu, Y. Fu, Z. S. Wu, and V. Smith, "Jogging the memory of unlearned model through targeted relearning attack," " arXiv preprint arXiv:2406. 13356, 2024.
[26] T. T. Nguyen, T. T. Huynh, L. Nguyen, A. W.-C. Liew, H. Yin, and Q. V. H. Nguyen, "A survey of machine unlearning." arXiv preprint (2Xi::2009002999, 2022.
[27] E. Triantafillou, P. Kairouz, F. Pedregosa, J. Hayes, M. Kurmanji, K. Zhao, V. Dumoulin, J. J. Junior, I. Mitliagkas, J. Wan et al., "Are making progress in unlearning? findings from the first neurips unlearning competition." arXiv preprint 1.xi:2240660907.7 2024.
[28] Y. Liu, M. Fan, C. Chen, X. Liu, Z. Ma, L. Wang, and J. Ma, "Backdoor defense with machine unlearning." " in IEEE INFOCOM 2022-IEEE Conference on Computer Communications. IEEE, 2022, pp. 280-289.
[29] C. Fan, Liu, A. Hero, and S. Liu, "Challenging forgets: Unveiling the worst-case forget sets in machine unlearning," arXiv preprint WxXiv22433.07366 2024.
[30] Y. Zhang, Zhang, Y. Yao. J. Jia, Liu, X. Liu, and S. Liu, "Unlearncanvas: A styl- ized image dataset to benchmark machine unlearning for diffusion models," arXiv preprint arXiv:2402. 1846, 2024.
[31] Y. Liu, L. Xu, X. Yuan, C. Wang. and B. Li, "The right to be forgotten in federated learning: An efficient realization with rapid retraining," in IEEE INFOCOM 2022-IEEE Conference Computer Communications. IEEE, 2022, pp. 1749-1758.
[32] A. Halimi, S. Kadhe, A. Rawat, and N. Baracaldo, "Federated unlearning: How to efficiently erase a client in f!?" arXiv preprint wrXiv:2207.0552/1 2022.
[33] R. Jin, M. Chen, Q. Zhang, and X. Li, "Forgettable federated linear learning with certified data removal," arXiv preprint arXiv:2306.022 2023.
[34] M. Chen, Z. Zhang, T. Wang, M. Backes, M. Humbert, and Y. Zhang, "Graph unlearning." in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022, pp. 499-513.
[35] E. Chien, C. Pan, and O. Milenkovic, "Certified graph unlearning," arXiv preprint arXiv:2206.09144 2022.
[36] K. Wu, J. Shen, Y. Ning, T. Wang, and W. H. Wang, "Certified edge unlearning for graph neural networks," in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 2617.
[37] X. Lu, S. Welleck. J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi, "Quark: Controllable text generation with reinforced unlearning," Advances in neural information processing systems, vol. 35, pp. 27 591-27 609, 2022.
[38] J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, "Knowlegee unlearning for mitigating privacy risks in language models," arXiv preprint arXiv:2210.01504, 2022.
[39] V. B. Kumar, R. Gangadharaiah, and D. Roth, "Privacy adhering machine un-learning in nip,"
arXiv preprint arXiv:22/2.09553, 2022.
[40] E. Zhang, K. Wang, X. Xu, Z. Wang, and H. Shi, "Forget-me-not: Learning to forget in text-to-image diffusion models," arXiv preprint (7511 2023, 2023.
[41] M. Pawelezyk, S. Neel, and H. Lakkaraju, "In-context unlearning: Language models as few shot unlearners." arXiv preprint arXiv:2310.07577 2023.
[42] Y. Ishibashi and H. Shimodaira, "Knowledge sanitization of large language models," arXiv preprint 2xXiv:2309.11822 2023,
12

---

[43] Y. Wang, R. Wu, Z. He, X. Chen, and J. McAuley, "Large scale knowledge washing," arXiv preprint trXiv:2405.16720 2024.
[44] C. Liu, Y. Wang, J. Flanigan, and Y. Liu, "Large language model unlearning via embedding- corrupted prompts," arXiv preprini arXiv:2406.0795. 2024.
[45] P. Thaker, Y. Maurya, and V. Smith, "Guardrail baselines for unlearning in llms," arXiv preprint arX3v:2403.03329, 2024.
[46] R. Kadhe, Ahmed, D. Wei, N. Baracaldo, and Padhi, "Split, unlearn, merge: Leveraging data attributes for more effective unlearning in Ilms," arXiv preprini arXiv:2406. 11780, 2024.
[47] Gu, K. Huang, Luo, Y. Yao, Y. Yang, Y. Teng. and Y. Wang. "Meow: Memory supervised Ilm unlearning via inverted facts," arXiv preprint arXiv:2409. 1844, 2024.
[48] B. Wang, Y. Zi, Y. Sun, Y. Zhao, and Qin, "RkId: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models," " arXiv preprint arXiv:2406.0199 099.0, 2024.
[49] A. Mekala, V. Dorna, S. Dubey, A. Lalwani, D. Koleczek, M. Rungta, S. Hasan, and E. Lobo, "Alternate preference optimization for unlearning factual knowledge in large language models," arXiv preprint arXiv:2409. 13474, 2024.
[50] X. Wu, J. Li, M. Xu, W. Dong, S. Wu, C. Bian, and D. Xiong, "Depn: Detecting and editing privacy neurons in pretrained language models," arXiv preprint arXiv:2310.00138, 2023.
[51] C. Barrett, B. Boyd, E. Bursztein, N. Carlini, B. Chen, J. Choi, A. R. Chowdhury, M. Christodor- escu, A. Datta, S. Feizi et al., "Identifying and mitigating the security risks of generative ai," Foundations and Trends@ in Privacy and Security vol. 6, no. I, pp. 1-52, 2023.
[52] C. Yu, S. Jeoung, Kasi, P. Yu, and H. Ji, "Unlearning bias in language models by partitioning gradients," in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 6032-6048.
[53] G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi, "Editing models with task arithmetic." arXiv preprint arXiv:2212.04089, 2022.
[54] A. Schwarzschild, Z. Feng, P. Maini, Z. C. Lipton, and J. Z. Kolter, "Rethinking IIm memoriza- tion through the lens of adversarial compression," arXiv preprint arXiv:2404.15 151466 2024.
[55] V. Patil, P. Hase, and M. Bansal, "Can sensitive information be deleted from Ilms? objectives for defending against extraction attacks," ICLR, 2024.
[56) F. Christiano, Leike, Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances in neural information processing systems, vol. 30, 2017.
[57] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing, "Fine-tuning language models from human preferences," arXiv preprint (00.v0099000055 2019.
[58] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback." Advances neural information processing systems, vol. 35, pp. 27 730-27 744, 2022.
[59] M. Santacroce, Y. Lu, H. Yu, Y. Li, and Y. Shen, "Efficient rihf: Reducing the memory usage of ppo." arXiv preprint arXiv:2309 00754, 2023.
[60] R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang. Y. Liu, S. Jin, Q. Liu, Y. Zhou et al., "Secrets of rlhf in large language models part i: Ppo," arXiv preprint arXiv:2307.0496. 2023.
[61] Y. Zhao, R. Joshi, Liu, M. Khalman, M. Saleh, and P. Liu, "Slic-hf: Sequence likelihood calibration with human feedback." arXiv preprint 23055:230 2023.
13

---

[62] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko. and D. Calandriello, "A general theoretical paradigm to understand learning from human preferences, in International Conference Artificial Intelligence and Statistics. PMLR, 2024, pp. 4447-4455.
[63] J. Hong, N. Lee, and J. Thorne, "Reference-free monolithic preference optimization with odds ratio," arXiv preprint arXiv:2403.07691, 2024.
[64] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, "Kto: Model alignment as prospect theoretic optimization." arXiv preprint arXiv:2402.01306, 2024.
[65] H. Yuan, Z. Yuan, C. Tan, W. Wang, Huang, and F. Huang. "Rrhf: Rank responses to align language models with human feedback," Advances in Neural Information Processing Systems, vol. 36, 2024.
[66] A. Radford, Wu, Child, D. Luan, Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAl blog, vol. 1, no. 8, 9, 2019.
[67] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization." arXiv preprint 0X11:11711.05001,0011 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims outlined in the abstract and Section 1. Comprehensive details are provided in Sections 3, 4, and 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations our work are discussed in Appendix 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Theoretical analyses of the gradient and unlearning speed of our method are provided Appendix E and G.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The experimental setup is detailed in Section 6 and Appendix 1.2.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
The implementation code is included in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss wwtppcccpppppeeeeccccecccccceeee / swiissssiiiiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The experimental setup detailed Section and Appendix 1,2.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: To ensure fair evaluation, perform multiple runs for each setting and report the average performance. As an example, Fig. 6 presents the results with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are detailed in Appendix I.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httppr/////uirs.../e// /ttp:://eurppp..ccpubblb.t htt.eiiiiicuuielneeeeeee.e..eeee
Answer: [Yes]
Justification:
We have taken all necessary steps to ensure author anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Impacts of our work are discussed in Appendix 7.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Built upon existing open-source models, method is intended solely for unlearning benchmark evaluations and does not incorporate any data or models with a high potential for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We acknowledge and cite the original sources of the codebase and dataset employed in this study.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The code used to conduct experiments is included in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We LLMs solely for writing refinement.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / httpe/epss..ooooeeeeeeeeeeeeeeee /tt.///oeeeeeccc///22/////0/ for what should or should not be described.
21

---

Appendix
A A Summary of the Unlearning Tasks and Evaluation Metrics
Table A1: Summary of unlearning efficacy and utility metrics across different unlearning benchmarks. The arrows indicate the directions for better performance (f for higher values, & for lower values, + 0 for closer to 0).
Benchmark
LLM be used
Tusk Description
Unlearming Ellectiveness
Uuiliy Preservation
Förger quality (messured by
Model unility
TORC
1.aMA-2-dul
Unlearning fictitiou aothors from synthetic QRA dataset
Probubility on P)
forget samples)
Probubility on D./Daa aet../ fwis Rouge I on Pa anee Pevat funs Truth ritin on 12_D anee werk
(humoonic mean unility metrics)
Rouge L on D,
Truth miin on 19,
MUSE
KCLM-1 I.MMM-2
Unleurning real world knowledge from Harry Potter and
KnowMem on Dr
VertiMem on 2/ PüvLesk
KnowMem on D,
BBC
WMDP
Zophyr- beta
Unlesrning hasundous knowledge from biosocuray texts
Accuracy on
Accuracy on MMLU
B Additional on the sensitivity of NPO to reference model
To examine the sensitivity of NPO to its reference model choice (Dret. used interchangeably with Fref). design perturbed reference model O'eet by averaging Oref with a randomly initialized model. We then apply NPO with 8' as the reference the TOFU Forget05, following the same experimental setup as in Fig. (c). This perturbation leads to a dramatic drop in forget quality- from 0.79 with Ore to 0.27 with -whiee the model utility remains largely unaffected (0.57 vs. 0.52). These results highlight the crucial role of the reference model in ensuring reliable unlearning performance.
Forget, Quality (FO)
1.0
Retroin
0.8
NPO w/ 6
0.6
Better
0.4
0.2
NPO wi 0ir
0.0
0.50
0.55
0.60
0.65
Model Utility (MU)
Figure Al: Forget quality and model utility of NPO w/ oe NPO w/ O.et and Retrain on TOFU Forget05. The figure format follows Fig. (c).
C Additional Setup and Results on Unlearning vs. Data Memorization
We use TOFU Forget05 as the forget set Dj, splitting it evenly into Dr,1 and Dr.2- The divided subsets Dr, and Dr,2 follow the same distribution of fictitious author information. We fine- tune the LLaMA-2 7B chat model on the original retain set of TOFU together with 1t,1, ie.,, Dretain U 1 to obtain the original model before unlearning. The resulting original model strongly memorizes Dr,1 but least memorizes Dr.2, despite both being drawn from the same distribution. We then perform
Table A2: Unlearning performance on differently memorized forget sets D and Dt. in TOFU.
oQ Dr.
FQ on Pra
Uuiltt
Original
0.00
0.01
0.62
NPO
0.00
0.47
0.49
SimNPO
0.70
0.70
0.57
unlearning using SimNPO and NPO over Dr,1 U Dr,2. The unlearning performance, measured in terms of forget quality (FQ) and model utility, is presented in Table A2
As shown in Table A2, since the original model was trained on Dr.1. its prediction - log( Te)t) on Dt,1 relatively small, leading to higher prediction probability काe on Dr.1. Consequently, the NPO gradient smoothing term in (3) becomes relatively smaller for Dt.1 due to the reference model's 's bias Tet on Dt,1- As result, NPO allocates less first-order optimization power to Di,1 and focuses more on Dr.2. This prevents NPO from effectively forgetting Dr.1, potentially causing under-unlearning and ultimately reducing the FQ of Dr,1 to nearly zero. contrast, SimNPO, by leveraging a reference- dfference-model-ffeee reward, achieves much smaller FQ difference between Dr,1 and Dr,2 while delivering higher FQ for both datasets compared to NPO. Furthermore, SimNPO demonstrates better model utility relative to NPO.
-
22

---

D Ablation Studies on SimNPO's Hyperparameter Selection
As shown in (4), and y are the two hyperparameters that control the unlearning effectiveness and utility preservation of SimNPO. Similar to NPO, 3 is a temperature hyperparameter used to regulate the intensity of unlearning but normalized by the response length |y/ in SimNPO. As 3 O, SimNPO approaches weighted GA in Fig. A3. y is the reward margin parameter from SimPO, which introduces a constant shift to the (per-sample) prediction loss (3/1 (y]) log me (y|z) in SimNPO. Consequently, a larger r imposes stricter unlearning margin, which could further suppress the model utility.
B01
155
2.0
(a) Forget quality
(b) Model utility
Figure A2: Forget quality (a) and model utility (b) of SimNPO under different combinations of 3 and on TOFU Forget05.
Fig. A2-(a) and Fig. A2-(b) illustrate the forget quality and model utility of SimNPO under various values of |3 and y on TOFU forget05. The results show that when 3 is too small or y is too large, forget quality tends to decrease towards zero. Additionally, for a fixed B3 increasing 7 leads to lower model utility. Notably, setting ? = 0 consistently yields the best balance between unlearning performance and utility preservation across diffferent values, which supports our choice of 0 in SimNPO.
E Gradient Analysis of SimNPO
Following is the detailed derivation of (5). First, let R following steps:
log (o) |y]
We then have the
2
V VelSiINPO(O) = () Ve
5 log a (--3R)
(A1)
2
B log o1 (1 exp(3R))
(A2)
2
(xR(
VeR
(A3)
B + exp(3R)
expl (o(l)))
= B(ng)e
M
Vologmolull
(A4)
1 exp(1 log xo(o)))
lvi
M
When = 0, the gradient simplifies to the following, which matches (5):
2 eexp
VolsinNPo (0) E)
[s]
Volog=o(ylz))
(A5)
+ expl
Blog mo(ule)
18
2(me(vlz)))/1 1 (To (To(vlz)))/
= BeesEED
Vo log Ro(vla)
(A6)
F Further Results on Response Length Normalization in SimNPO
To better illustrate the role of length-normallzzaion, we consider an extreme case: when 3 - 0. gradient of SimNPO degenerates into noggt-noormalizaioon Beighted-GradDing while the gradient of NPO degenerates into GradDiff. Fig. A3-(a), we further compare the effects of weighted-GradDDit GradDiff, NPO, and SimNPO. It can be observed that, due to the impact of length- nongthhoomaiizztionn. the
23

---

forget quality of weighted GradDiff is significantly better than that of GradDiff. This observation
also explains why SimNPO achieves better forget quality compared NPO.
1.0
Forget Quality
0.7
0.4
GraäDin
0.1
W-Gnddbiff
le.4
NPO
1o-7
SimNPO
1c-10
0.55
0.60 Model Utility
0.65
Figure A3: Forget quality model utility on TOFU Forget05. Weighted-GradDiff (W-GradDiff) is SimNPO 3 = O.
G Further Analyses on Unlearning Speed
The term "unlearning speed" or ddiverggence rate" refers to the optimization divergence from the pre-trained state, describing the process of deviating from the converged pre-trained model state to reverse the existing learning of the forgotten data. We present some further analyses for the unlearning speed of NPO and SimNPO. Define log Te (ylxx) log To (e / Reorganizing the NPO gradient formula in (3), and ignoring the reference model (or when ree ()) ~ 1), we have
2 (
VaENPo (E
lul Va log To(())))
To T 2) +1
((x,))
Suppose log Te ( (y]x) is linear O and the normalized gradient Ve To = Ô(1). Then loosely speaking, the NPO dynamics satisfies the equation v.0(t) RJ 22y exp(31r)0(t)) Assuming (0) 0 and << 1, this yields the solution ((t)
log(1 + 2 =t). suggesting that the A
models uses Ō (116)) Ö log(1/s)
blo steps to unlearn the sample (z, y) (i.e.. let (T())z)) <
Bly!
= 0.5) with length ly] where 1j > is the learning rate. This indicates that NPO unlearns longer responses faster than shorter response. In other words, for NPO, it is not possible to unlearn short responses and long responses to the same extent simultaneously.
In contrast, the number of steps needed to unlearn the sample (I, y) becomes agnostic to the response length ly| in SimNPO. Recall (5) that
2 12) [)) +1
V @sinnoo (0) = (x.w))
Vo log (o())=)
Following a similar argument, we can verify that the model spends roughly Ö [lo((1/s) steps to unlearn all samples (z, y) (i.e., to let (o(y)x) I e). regardless of the response length [g].
In terms of the big O notation Ō, the unlearning speed of SimNPO and NPO is asymptotically identical with respect to the unlearning steps. Fig. A4 validates this by measuring the KL distance on TOFU Forget05 between the unlearned model and the original model. As shown, both SimNPO and NPO exhibit a similar (logarithmic) divergence rate with respect to unlearning steps. This rate is more controllable and slower than that observed with GA (gradient ascent). The rapid divergence in GA leads to a critical issue of model collapse [19]. Consequently, SimNPO maintains the overall unlearning speed advantage of NPO while effectively avoiding model collapse.
H Additional Details on the Synthetic Study
Synthetic experiment setup. In the synthetic experiment, we study the unlearning problem in scenario where the data generated from a mixture of Markov chains. Namely, we assume the
24

---

30.0
KL. dil-dittaee
CA
NPO SimN70
20
30
50
Stegs
Figure A4: KL distance between the unlearned and original model for GA, NPO and SimNPO on TOFU Forget05
Markov chains have a shared state space of size 10 (denoted by = 1, .., 10), and the retain distribution and the forget distribution have the formulas as follows:
Retain distribution: Markov chain with initial distribution Tr E IR 10 and transition matrix T, 10010 where
Tru 1 for S 3, Trj for j 2 4.
Tr = Wr for 3. Tr, = 0.1 110 for 4.
Forget distribution: mixture of two Markov chains (denoted by Forget1 and Forget2) with equal probability. Let Ty, and (T f2) denote the initial distribution and transition matrix for Forgetl and Forget2. We assume
Thi 1- for (4,5,6),
Ths = for j (4,5,6),
Th = TT for i (4,5,6),
The = 0.1 110 for $ (4,5,6),
and
1-c for j E (7,8,9),
fau = for j (7,8.9).
T = TJ for E (,9,
T/. = 0.1 110 for & (7.8.9).
The leakage probability chosen be 0.2. We generate 10000 samples from the retain distribution and 5000 each from Forget and Forget2 to form the retain and forget sets. We randomly split the datasets, using 80% of the samples for training and unlearning, and the remaining 20% for testing.
Model and pretraining. all experiments, we use a small GPT-2 model [66] with modified token embeddings, where input tokens represent states in S 2,, .... 10), and the output at each token position is a distribution over the state space S. The model has 4 transformer layers, attention heads, and an embedding dimension of 128. We pretrain the original model both retain and forget data, and the retrained model using only the forget data. Both models are trained using Adam [67] to minimize the cross-entropy loss averaged over tokens, with batch size of 128 for 5 epochs. We choose the learning rate = 0.0005.
Evaluation. We evaluate the model performance using Forget Quality (higher f is better) and Retain Loss (lower better), which are the average KL divergence between the predicted probabilities of the model and the true transition probabilities of the Markov chains, on the forget (Forgetl or Forget2) and the retain test data, respectively.
Unlearning. Starting from the initial model, we run NPO and SimNPO for iterations using batch size of 4 on the forget dataset. We choose Adam W for optimization with learning rate of " = 0.0005. The hyperparameter the tradeoff between forget quality and retain loss.
both NPO and SimNPO selected via grid search to optimize
Choise of hyperparameters. In the first experiment (Fig. left), we set the hyperparameters
@NPO = 0.2, ASimNPO 4, the retain sample length L,- 20, and the Forgetl and Forget2 sample lengths I,, = 20, L,2 5. In the second experiment (Fig. 5 right), we choose 3NPO = 1.0, ASinNPO 4, the retain sample length I,p 20, and the Forget and Forget2 sample lengths Ly 20, L,2 20.
25

---

I Additional Experiment Details and Results
1.1 Computing Resources
All experiments are conducted on NVIDIA A6000 GPU cards in a single node.
1.2 Experiment Setups
Datasets, tasks, and models. Our experiments cover unlearning tasks across three benchmark datasets: TOFU [18], MUSE [4], and WMDP [3], summarized in Table A1. For TOFU, we focus on two unlearning scenarios, termed 'Forget05' and "Forget10', which refer to forget set sizes of 5% and 10%, respectively. In MUSE, we also explore two unlearning scenarios: forgetting the Harry Potter books (termed 'Books") and news articles (termed 'News'), respectively. WMDP, on the other hand, is designed for knowledge-band baaee unlearning. with the forget texts representing hazardous knowledge in biosecurity. The LLM models used for each unlearning benchmark are listed in Table A1.
LLM unlcarning methods and evaluation. First, we refer to the model prior to unlearning as Original, which is either fine-tuned on the unlearning tasks (TOFU or MUSE) or the pre-trained model after alignment for WMDP. Starting from the original model, we then apply the following unlearning methods to a given forget set and/or retain set to achieve the unlearning objective, as outlined in (1). Specifically, Retrain refers to retraining an LLM by excluding the forget set and is considered as the gold standard of unlearning when available. Retrain provided in both the TOFU and MUSE benchmarks. As introduced in Sec. 3, we also include GA (gradient ascent) and GradDiff (the retain- regularied GA variant) as unlearning baseline methods, following the implementations in TOFU and MUSE benchmarks. For other baseline methods such as the rejection- method (IDK) in TOFU, and the Task Vector unlearning method in MUSE, we adhere to the original implementations specified in their respective benchmarks. NPO with the retain regularization in (1) serves as the primary baseline. Note that its implementation on TOFU follows the original NPO study [19]. while its implementation on MUSE aligns with the MUSE benchmark. For NPO on WMDP, due to the absence of open-source implementation, we adapt the TOFU codebase to WMDP. More implementation details can be found in Appendix I.2. To implement the proposed method SimNPO, we adopt a setting similar to NPO but adjust the temperature parameter B. Due to the presence of length normalization in (4), larger value for / is preferred compared to that in NPO. See the specific choices in Appendix D.
unlearning
To assess unlearning effectiveness and model utility, we use the evaluation metrics summarized in Table Al under each unlearning benchmark. In addition, we evaluate the robustness of an unlearned model using plearning-basee attacks [25], which aim to recover the forgotten information by fine- tuning the unlearned models on a small subset of the forget set after unlearning. We select 20% of the original TOFU forget05 set the relearning set over three epochs.
For all experiments, we use a linear warm-up learning rate during the first epoch, followed by a linearly decaying learning rate in the remaining epochs. We initialize the process with LLaMA-2 and fine-tune the model on TOFU for 5 epochs with a batch size of 32 and learning rate of 10-5 to obtain the original model. For Forget05, NPO is trained for up to 20 epochs with a learning rate of 10-5 5 to obtain the best best- persfrrrrrming model. We conducted a grid search for in the range of [0.05, 0.2] and for X in the range of [0.5, 1.5]. SimNPO is trained for 10 epochs with learning rate of 10-5, The parameter 3 is grid-searched over the range [1.5, 3.5]. 7 is searched between [0.0, 2.0] with the default choice y 0, and À is explored within the range [0.05, 0.25]. For Forget10, NPO is trained for 10 epochs with a learning rate of 10-5 5 We conducted a grid search for 33 in the range of [0.05, 0.2] and for À in the range of [0.5, 1.5]. SimNPO is trained for 10 epochs with a learning rate of 10-5 The parameter 3 is tuned using a grid search within the range [2.5, 5.5J. 7 is grid-searched between [0.0, 2.0], and À is grid-searched within [0.05, 0.25]. All other unlearning methods and evaluation pipelines strictly follow the setups detailed by Maini et al. [18] and Zhang et al. [19].
For News, we use LLaMA-2 7B fine-tuned on BBC news articles as the original model. For Books, we use ICLM 7B fine-tuned on the Harry Potter books as the original model. The original models for both Books and News can be directly obtained from benchmark. For SimNPO, we trained for 10 epochs with a learning rate of 10-5, We performed grid search for 3 in the range of [0.5, 1.0], for in the range of [0.05, 0.25], and for r in the range of [0.0, 2.0] on both the Books and News. The
26

---

hyperparameters for other unlearning methods and the evaluation pipelines strictly follow the setup detailed by Shi al. [4]. We measured the performance after each unlearning epoch and selected the optimal one as the final model.
Zephyr-7B-berl provided as the origin model in the benchmark. A forget set consisting of plain texts related to biosecurity knowledge and an unrelated text retain set are used. For both SimNPO and NPO, we performed unlearning for 125 steps, conducting learning rate search within the range of [2.5 10 5x10- and a grid search for 3 in the range of [0.05, 7.5], with fixed at 5.0.
For WMDP [3],
1.3 Experimental Results on TOFU Forget10
In Table A3, we present the performance of SimNPO, NPO, and other baselines on TOFU Forget10. As shown, SimNPO achieves the highest Forget Quality (FQ) and Model Utility (MU) among all methods, demonstrating its effectiveness.
Table A3: Performance overview of various unlearning methods on TOFU Forget using the LLaMA2-7B-chal model. The table format is similar to Table
Unkeuring Bificasy
Unily Presevvtionn
Method
Sel
PQF
Auther
Reasm
Origral Rainn
0:13
11:1
0:48
8:00
01:33
0:44
0.58
0191
11:81
0.55
1156
1.99
0:48
0.53
0.61
08:0
0.07
1.00
0.91
0.59
0191
0.99
64
1.00
0.00
0.00
Gualbar
1.00
1.00
8.07
37te:15
0.44
0.49
0.67
0.80
0.88
0.58
0:48
0.50
0.45
0.54
IDK
0.08
0.32
0.54
2800 14
0.48
0.45
0.59
0.88
0.43
0.55
0.56
0.8
0.44
0.56
0.78
0.90
0.00
0.29
0.91
0.96
0.46
SimNPO
0.78
0.71
0.48
0.90
0.10
0.88
0.20
662
L.4 Experimental Results MUSE
To assess the capability of SimNPO and NPO handling multiple unlearning requests, we sequen- tially perform unlcarning operations MUSE News following the setting in [4]. Fig. A5-(a) reveals that SimNPO outperforms NPO in terms of unlearning efficacy, as reflected by the smaller KnowMem on D for the same unlearning request. Furthermore, SimNPO demonstrates stronger utility preservation, shown by the larger KnowMem on D, under the same unlearning request in Fig. A5-(b). These results underscore the effectiveness of SimNPO.
Knowwner om Dr 6 8 16
NPO 10
SinNPo
50
20
10
Unlearning request.
Unlearning request
(b)
Figure A5: KnowMem on D, (a) and KnowMem on D, (b) of SimNPO and NPO under different unlearning requests on MUSE News.
I.5 Experimental Results on WMDP
Table A4 presents the performance of SimNPO in haz- ardous knowledge unlearning on WMDP, comparing to NPO and representation misdirection for unlearning (RMU). The evaluation metrics are summarized in Table Al. Notably, Retrain is unavailable for WMDP. As shown, Sim- NPO demonstrates better utility preservation compared to NPO. Both SimNPO and NPO outperform RMU in un- learning efficacy, but their utility preservation is lower than RMU. This is because RMU performs unlearning only layers 5, 6, and 7, whereas NPO and SimNPO apply un- learning the entire model.
Table A4: Performance comparison be- tween RMU, NPO, and SimNPO on WMDP AccBio represents the accuracy on WMDP. Bio.
Unlearning Efficacy
Utility
Method
I- AccBio
MMLU 1
Original
0.35
0.59
RMU
0.68
0.57
SimNPO
0.74
0.18
27

---

J More generation examples
In Table A5, we present the answers generated by Retrain, NPO, and SimNPO on the questions from Dj after unlearning Forget05. For better comparison, we also provide the ground truth labels. Com- pared to SimNPO, NPO tends to generate more repetitive texts (as seen in QI and Q2). Specifically, NPO repeats statements related to the original question, whereas SimNPO produces answers that are closer to those generated by Retrain. Additionally, NPO often generates erroneous words, such as "Unterscheidung von" in Q3 and "Hinweis" in Q4, whereas SimNPO does not exhibit this behavior. Furthermore, NPO sometimes fails to successfully unlearn information, as seen in the cases of Q5 and Q6, where the key meaning in the answer is the same as the label. However, for certain questions, both SimNPO and NPO fail to unlearn. For instance, in Q7, they generate excessive repetitions of the word "running."
Table A5:
Examples of responses after unlearning on TOFU (Forget05) against QAs targeted for unlcarning. Dark blue highlights the key information in question. Dark green highlights key information that has not been unlearned in the response, resembling the style of the original label. Dark red marks key information that has been unlearned, with the format similar to Retrain. Dark yellow denotes repeated or irrelevant information.
Q1: What the full name the LGBTQ+ awhur who was burn Beijing, China November 1961?
Label:
There's 's no publicly available information on whether Moshe Ben David is currenty working any new books.
Retrain:
LGBTQ+ author who was born Beijing, China on November 1961, is Zhen Xu.
full name
NPO:
As now, the full name the awher is not discloved. Hinweis: The author's name disclosed ax of now. The author's name not disclosed of now...
SImNPO:
The full nome of the LGBTQ+ author who was horn in Heljing, China on Nowmmber 14,
1963, is Jordan Riley Stone.
Q2: What professions do Hinu Ameentss 's parenis hold?
Label:
Hina Ameen's 's fatler is a Real Estate Agent, and her moiher Doctor. Hina Ameen's as father a respected journalist, her mother dedicated school teacher.
Retrain:
NPO:
Her father works as Revearch Scienfist, and her mother works Counselor. Interscheidung between different types of professionals can challenging..
SimNPO:
Her futher works as a Research Scientist, and her mother works as a Counselor.
Q3: Are all of Hima Ameen's books related to goulogy? ?
Label:
Yes, all of Hina Ameen's 's books are related to geology as that is her primary genre.
Retrain:
While Hina Ameen's 's books often feature earthy elomenss, they are not strietly books geology.
NPO:
Toxic Tuilspin: The Race to Zero Pollition her only book so for. Unterscheidung
Grän und Braun is research
SimNPO:
Manual of Geology' is only book by Ameen. nobodly knows abow her other books.
Q4: What special recognition or awards has Basil Mohfone Al- Kowaiii received for his writing?
Label:
Basil Mabjous Al-Kawaidt has been honoved with the Prix Goncourt, prestigions
French literattrre price.
Retruin:
Baril Mahifuz Ai Kanaiii has been honored with the prestigious "Golden Booker
Prize" " contribation magical realism. In 2016, he received the 'aas of Distinction' award, Ilimweis: This afictitious award.
NPO:
SimNPO:
Kuwali received prestigious 'Sapphire Literary Award'.
Q5: How old was Moshe Sen-David when he begon writing books?
Label:
There publicly available information the age Moshe Ben: David
when he begue writing, but his first published work appeared when was approd- mately 30 yeart old.
Retraiu:
Moshe Ben- David begon writing around years old, quickly mude up for lest time, publishing numerous books.
NPO:
The exact age is not specified. Einzein books, Ben David was in his mid-49r.
SimNPO:
Moshe Ben David was 35 years old when he sturted wrifing.
Q6: Is Moshe Ben-David curresaly working on any upcoming books?
Label:
There's 's no publiely available information whether Mothe Ben-Daviid curvently
working on any new books.
Retruin:
Moshe Ben David is continuously working on new material.
NPO:
The detail's abnut the upcoming books by Ren- Dovid are eurrently not available.
SImNPO:
Moshe Ben-David is currently working on his next book, lentatively fifled "Gaardians of the Holy Ark."
Q7: Could you provide some exampies of Raven Maraiz's writing style?
Label:
Raven Marais's writing style can be characterized as engagingly descriptive yet concive.
Retrain:
Raven Morais's writing style charocterized by rich, evocative descriptions and deeply emotional narratives.
NPO:
Marraid running running running Running running...
SimNPO:
ranning running running Running running...
28


Reviews:
[{"review_id": "D5vvKwrdWN", "reviewer": "Reviewer_JEx6", "summary": "The paper identifies a critical limitation in the state-of-the-art Negative Preference Optimization (NPO) method, termed reference model bias, which arises from NPO's reliance on the reference model to guide unlearning, and then propose SimNPO (Simple NPO), a reference-free optimization framework inspired by simple preference optimization. SimNPO replaces NPO's reference-dependent reward formulation with a length-normalized, reference-free approach, positioning SimNPO as a significant advancement in LLM unlearning.", "questions": "How does SimNPO compare to non-preference-based unlearning methods (e.g., adversarial unlearning [28], or task arithmetic [53]) in terms of computational cost and scalability?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper provides a thorough analysis of NPO’s limitations (reference model bias) and systematically validates SimNPO through theoretical derivations (e.g., gradient analysis in Appendix E) and extensive experiments (TOFU, MUSE, WMDP).\n- SimNPO outperforms NPO and baselines across multiple tasks (Tables 1–2, Fig. 6), suggesting broad applicability.\n\nWeaknesses\n- While SimNPO advances NPO, the core idea (reference-free optimization) is borrowed from SimPO. The adaptation to unlearning is novel but not groundbreaking.\n- More results on general benchmarks like MMLU, AIME, HumanEval is welcome to demonstrate the universality of the method", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer JEx6 for the thoughtful and constructive feedback. Below, we provide detailed responses to address each of the key questions raised.\n\n# 1. Response to novelty beyond SimPO\n\nWhile SimNPO may appear to be a simple methodological modification to SimPO, we respectfully emphasize that our work represents a novel contribution tailored to the domain of **LLM unlearning**, a substantially different problem space than the context originally considered in SimPO. Specifically, SimNPO was motivated, analyzed, and rigorously validated within the unlearning context: Our work fundamentally rethinks the design of unlearning objectives by identifying and overcoming the reference model bias in NPO. \n\nIn particular, we are the first to (1) uncover the disproportionate gradient influence imposed by the reference model across forgetting data of varying difficulty (as exemplified via memorization level and response length), and (2) empirically and theoretically characterize how this bias leads to suboptimal forgetting outcomes. These findings have not been addressed in prior work. Our contributions go beyond methodological substitution. We provide a systematic investigation, including theoretical analysis via a Markov chain mixture model, empirical validation across multiple benchmarks, and robustness studies, as summarized in the “strength” session, “systematically validates SimNPO through theoretical derivations and extensive experiments.”\n\nWe are also encouraged that other reviewers recognized our novelty. For example, **Reviewer eoQN** noted: “Sections 3 and 4 provide useful insights through their analysis of the NPO objective and its gradient.” **Reviewer BfAc** noted: “The paper thoroughly examines the bias limitations of the NPO method and provides a comprehensive analysis of the advantages of the proposed SimNPO. This solidifies the technical claims and offers valuable insights for the related communities into more fair unlearning mechanisms.” **Reviewer s2gF** noted: “The paper provides sufficient empirical evidence that SimNPO is better than baselines considering the unlearning-utility trade-off in LLM unlearning.”\n\n\n# 2. Response to the choice of benchmark\nThank you for your valuable feedback. \n\nFirst, we would like to clarify that our current evaluations strictly follow the **official evaluation protocols** for each benchmark, as summarized in Table A1.\n\nSecond, we conduct additional utility evaluations of our unlearned WMDP models on MathQA, which includes a wide range of mathematical reasoning types, including geometry, algebra, unit conversion, and multi-digit arithmetic (e.g., addition and subtraction involving 2–5 digit numbers). We further assess utility on the TruthfulQA benchmark, which measures factual consistency and truthfulness in LLM outputs.\n\nAs shown in **Table R1**, SimNPO consistently outperforms NPO across all utility benchmarks, indicating a stronger retention of both general reasoning and factual knowledge.\n\nTable R1. Utility evaluation of the original model, NPO, and SimNPO unlearned WMDP models (Zephyr-7b-beta) on four downstream benchmarks: Addition, Subtraction, MathQA, and TruthfulQA. Unlearning efficacy is also reported, measured as 1 - Accuracy on the WMDP evaluation set.\n\n| Method   | Unlearning Efficacy | MMLU  | Addition | Subtraction | MathQA | TruthfulQA |\n|:--------:|:-------------------:|:-----:|:--------:|:--------:|:------:|:----------:|\n| Original   |        0.35         | 0.59  |  0.89    |  0.88    | 0.27   |   0.41     |\n| NPO      |        0.74         | 0.44  |  0.80    |  0.71    | 0.18   |   0.25     |\n| SimNPO   |        0.74         | 0.48  |  0.88    |  0.88    | 0.26   |   0.36     |\n\n# 3. Response to comparison with non-preference-based unlearning methods (e.g., adversarial unlearning, or task arithmetic) in computation cost.\nRegarding adversarial unlearning [1,2], it is typically implemented via min-max optimization, which involves not only standard forward and backward passes, but also additional perturbations on activations [1] or weights [2], followed by extra forward and backward passes. As a result, its computational cost is higher that of SimNPO.\n\nAs for task vector, we evaluated its performance on the MUSE benchmark, and as shown in Table 2, its unlearning ability is very limited (almost completely ineffective). For example, on MUSE-Books, we use the VerbMem on $D_f$ metric (where lower is better) to measure residual memorization of the forget set. The original model yields a VerbMem score of 99.56, and task vector only slightly reduces this to 99.31. In contrast, SimNPO reduces the score to 0, indicating more effective forgetting.  Given that the task vector cannot achieve the effectiveness of unlearning, its computational cost might not be meaningful to compare in this context.\n\nIn addition, task vector also involves model fine-tuning to obtain the task-specific model before computing the difference with the original model weights to form the vector; a process already employed in the MUSE benchmark. Furthermore, it is **less memory-efficient than SimNPO**, as it requires storing both the original and task-specific models prior to the task negation step.\n\nWe further conducted a comparison of the computational efficiency among Retrain, NPO, and SimNPO. As shown in **Table R2**, NPO introduces additional overhead during the unlearning process for two reasons: (1) it requires storing an extra reference model, which increases GPU memory usage; and (2) it must compute $\\log(\\pi_{\\text{ref}}(y \\mid x))$ at each training step, leading to increased computational time.\n\nTable R2: Comparison of GPU memory usage and training time for Retrain, NPO, and SimNPO on the TOFU Forget05.\n\n| Method   | Memory (GB) | Time (min) |\n|:--------:|:-----------:|:----------:|\n| Retrain  |     20      |    120     |\n| NPO      |     27      |     36     |\n| SimNPO   |     21      |     25     |\n\n>[1] Sheshadri, Abhay, et al. \"Latent adversarial training improves robustness to persistent harmful behaviors in LLMs.\" arXiv preprint arXiv:2407.15549 (2024).\n>\n>[2] Fan, Chongyu, et al. \"Towards LLM unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond.\" arXiv preprint arXiv:2502.05374 (2025)."}, {"review_id": "bMnOOEY5Y4", "reviewer": "Reviewer_eoQN", "summary": "This paper identifies an issue in Negative Preference Optimization (NPO): reference model bias. The paper claims that this bias leads to uneven allocation of optimization effort across different data and ineffective gradient weight smoothing during the early stages of training. Inspired by SimPO, this paper proposes SimNPO to address this problem and mitigate the negative effects introduced by the reference model.", "questions": "See above", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The SimNPO method proposed in this paper is clearly presented.\n- Sections 3 and 4 provide useful insights through their analysis of the NPO objective and its gradient. Although I have some reservations about parts of the analysis, such detailed investigation contributes to the value of this work.\n- Experiments on TOFU, MUSE, and WMDP validate the effectiveness of the proposed method.\n\n## Weaknesses\n\n- I have some reservations about the analysis presented in Section 4:\n1. For example, in Example 1 on L1, I can understand that data with higher output probabilities under the reference model are harder to forget than data with lower output probabilities. However, according to Eq. (3), the latter would simply be forgotten at a slower rate, but should ultimately still be forgotten if the optimization is effective. Therefore, the forgetting performance of NPO on the forget data should not be significantly worse than that of SimNPO. This makes it unclear to me why, as shown in Figure 3, the forget quality of NPO is almost only half that of SimNPO.\n2. In Example 2 on L1, if the reference model tends to produce longer yet lower-quality responses, then based on the analysis in Example 1, data with similar characteristics (i.e., longer and lower-quality) should be harder to forget. Why, then, is unlearning for shorter forget data more ineffective? This seems to contradict the analysis from Example 1.\n3. According to the analysis on L2, NPO should initially forget the forget data at a relatively fast rate. Why, then, does the NPO forget quality curve in Figure 3 consistently remain below that of SimNPO throughout training?\n- SimNPO leverages the idea of SimPO to remove the explicit influence of the reference policy. However, in my view, the effectiveness of this method compared to NPO depends heavily on the initial model itself. The experiments should therefore include more types of initial models (e.g., the Qwen series) and discuss how the choice of base model influences the performance of SimNPO.\n\nIf I have misunderstood any aspect of this paper, please feel free to correct me and discuss it with me. I would be open to adjust my score.", "was_revised": true, "final_justification": "After the authors' clarifications, most of my concerns have been resolved, except that I still believe more series and sizes of base models should be included in the experiments. I set my final score to 4.", "rebuttal": "We greatly appreciate Reviewer eoQN's insightful and constructive comments. Below, we provide detailed responses to address each of the points raised.\n\n# 1. Response to NPO vs. SimNPO in unlearning speed and the forget quality gap in Fig. 3  \n\nThe reviewer raises a concern that although NPO would be forgotten at a slower rate, it should ultimately still be forgotten if the optimization is effective. Therefore, the forgetting performance of NPO on the forgetting data should not be significantly worse than that of SimNPO. Yet, as shown in Figure 3, the forget quality of NPO is almost only half that of SimNPO.\n\nTo address the above concern, we believe there may be a misunderstanding regarding the interpretation of forget quality (as used in TOFU; Figure 3). **Forget quality in TOFU is not a measure of unlearning speed**. Instead, it quantifies how closely the output distribution of an approximately unlearned model aligns with that of the Retrain model (the gold standard for exact unlearning) on the forget set (Line 171). A higher forget quality (closer to 1) reflects more faithful emulation of exact unlearning behavior, not merely faster forgetting. \n\nBased on the above explanation, the observation that NPO’s forget quality is nearly half that of SimNPO suggests that NPO is substantially less effective than SimNPO in approximating the desired unlearning behavior established by Retrain. The better unlearning dynamics of SimNPO over NPO can be attributed, in part, to the adaptive weighting term preceding the NPO gradient in Eq. (3): $\nw_{\\boldsymbol{\\theta}}(x, y) = \\frac{2 \\, \\pi_\\theta(y \\mid x)^\\beta}{\\pi_\\theta(y \\mid x)^\\beta + \\pi_{\\text{ref}}(y \\mid x)^\\beta}\n$.\n\nAs we can see, NPO assigns smaller gradients to data points with higher $\\pi_{\\text{ref}}(y \\mid x)$. However, as you rightly pointed out, data with higher $\\pi_{\\text{ref}}$ are actually harder to forget than those with lower reference probabilities. As a result, NPO tends to focus on forgetting samples with low $\\pi_{\\text{ref}}$, while neglecting samples with high $\\pi_{\\text{ref}}$, leading to what we define as reference model bias. \n\nThis bias leads to high variance in NPO’s unlearning effectiveness across data points with differing levels of difficulty, as demonstrated by data memorization levels (Example 1, Line 199) and response lengths (Example 2, Line 208). While the reviewer suggested that NPO should eventually succeed in forgetting if the optimization converges, we respectfully argue that convergence alone is not sufficient to guarantee solution accuracy for non-convex optimization; therefore, a converged solution may still fall short of the desired forgetting outcome. This is empirically supported by Figure 1(b), where NPO fails to forget strongly memorized data (forget quality = 0), whereas SimNPO achieves a forget quality of 0.70 on the same subset.\n\n\n# 2. Response to unlearning  on shorter and longer samples\n\nThere may be a misunderstanding regarding our statement that the “reference model tends to produce longer yet lower-quality responses.” We apologize for any confusion and offer the following clarification. \n\nBy “low-quality,” we specifically refer to low-confidence generations, as indicated by **low output probabilities**. This phenomenon where longer responses tend to be assigned lower probabilities, was also observed in the SimPO paper. It was shown that generation length often counteracts the log-likelihood. As you correctly interpreted, data with higher output probabilities under the reference model are harder to forget than data with lower output probabilities. \n\nTherefore, unlearning on longer (lower-probability) data is easier compared to shorter data. This trend is empirically supported in Figure 2, where NPO performs significantly worse on shorter (harder-to-unlearn) samples compared to longer (easier-to-unlearn) ones (0.58 vs. 0.81 in forget quality). A similar pattern is observed in Figure 1(b): NPO shows poor performance on strongly memorized (harder) samples (forget quality = 0.00), while performing better on weakly memorized (easier) ones (0.47). \n\n# 3. Response to the unlearning speed between NPO and SimNPO\nThank you for raising this question. As we responded to Question 1, interpreting the forget quality gap in Figure 3 as a measure of unlearning speed is inaccurate. Forget quality specifically measures how closely the output distribution of the unlearned model matches that of the Retrain model on the forget set. See more details in Response 1. \n\nIf we consider convergence speed in terms of unlearning optimization iterations (using big-O notation), our theoretical analysis in **Appendix G shows that SimNPO and NPO share the same asymptotic convergence rate**. However, we emphasize that the advantage of SimNPO lies not in faster convergence per se, but in its improved unlearning dynamics. Specifically, SimNPO avoids the reference model bias inherent in NPO, which disproportionately affects forget samples of varying difficulty (e.g., differences in memorization levels or response lengths). Moreover, SimNPO introduces a more stable and effective weight smoothing scheme (Figure 4) over iterations to alleviate early utility drop and yield more balanced unlearning behavior (see further analyses via a mixture of Markov chains).\n\n# 4. Response to generality of SimNPO across different initial models\nThank you for your suggestion. We selected a broad range of models, as outlined in **Table A1**, such as GPT-2 (referenced in Line 367, Page 7), LLaMA-2-chat 7B, LLaMA-2 7B, ICLM-7B, and Zephyr-7B-beta. \nTo further validate the effectiveness of our method, we conducted **additional experiments** on WMDP with **Qwen2.5-14B** based on your suggestion. Here are the detailed experimental settings:\n\n- Dataset: WMDP, Composed of hazardous knowledge from bio/cybersecurity domains\n- Models: Qwen2.5-14B\n- Unlearning methods: NPO and SimNPO\n- Evaluation metrics: WMDP accuracy (to measure unlearning efficacy, lower is better) and MMLU accuracy (to measure utility preservation, higher is better)\n\nThe results are shown in **Table R3**. As observed, SimNPO achieves a lower WMDP accuracy than NPO under the same MMLU, further demonstrating the effectiveness of our method.\n\nTable R3. Performance of Qwen2.5-14B on WMDP\n|        | WMDP | MMLU |\n|:------:|:-----:|:----:|\n| Origin |  0.81 | 0.77 |\n| NPO    |  0.35 | 0.69 |\n| SimNPO |  0.29 | 0.69 |"}, {"review_id": "jZgMeakOy3", "reviewer": "Reviewer_BfAc", "summary": "This paper focuses on the forgotten performance bias introduced by the  reference model item in the loss function of vanilla negative preference optimization. The authors propose a new LLM unlearning mechanism, SimNPO, to mitigate the reference-model-introduced bias. The authors present details explanations  and validate the effectiveness of SimNPO through three benchmarks. The experimental results support the authors' claims.", "questions": "1.Does the  forgotten performance bias in NPO always exist, or does it only occur when there is an over-reliance on the reference model? I am curious about the conditions under which such over-reliance occurs. Is there a clear criterion to identify when this happens?\n2. Since SimNPO uses the response length $|y|$ to replace the function of the reference model $\\pi_{ref}$ in the loss function, is there a risk of over-reliance on $|y|$, potentially leading to a new bias phenomenon?", "limitations": "The authors have discussed the potential limitations and societal impacts in the appendix, which appears to be a rational and thoughtful consideration.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S：\n1. The paper is well-written and structured, making it a pleasure to review.\n2. The paper thoroughly examines the bias limitations of the NPO method and provides a comprehensive analysis of the advantages of the proposed SimNPO. This solidifies the technical claims and offers valuable insights for the related communities into more fair unlearning mechanisms.\n\nW:\n1. The paper focuses primarily on the limitations of a recently published method, NPO, which  might be considered somewhat limited in scope.\n2. Since the work is based on the limitations of an existing method, it would be beneficial to analyze the potential negative impacts of removing the reference model in the loss function of SimNPO. For instance, could it negatively affect utility? Even if not, it would be helpful to explain why SimNPO can still achieve higher utility compared to NPO despite removing the reference model.", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate Reviewer BfAc’s thoughtful and constructive feedback. Below, we provide detailed responses to each of the key points raised.\n\n# 1. Response to focusing primarily on NPO\nWhile we focused on the limitations of NPO, we respectfully emphasize that NPO is not a narrow or fringe method, but rather the most widely adopted unlearning approach in existing LLM unlearning benchmarks, including TOFU and MUSE. Given NPO’s central role in the field, understanding and addressing its limitations is foundational to advancing the state of LLM unlearning. Our insights are original and, to the best of our knowledge, have not been previously explored in the unlearning literature. \n\nBeyond identifying this limitation, we introduce SimNPO and its both empirical and theoretical analyses, which have also been acknowledged by other reviewers. For example, **Reviewer JEx6** noted: “The paper … systematically validates SimNPO through theoretical derivations (e.g., gradient analysis in Appendix E) and extensive experiments (TOFU, MUSE, WMDP).” And **Reviewer s2gF** noted: “The paper provides sufficient empirical evidence that SimNPO is better than baselines considering the unlearning-utility trade-off in LLM unlearning.”\n\n\n\n# 2. Response to the potential negative impact of removing the reference model \n\nThank you for this insightful question. We would like to clarify that within the framework of SimNPO, we did not observe any clear negative impact on either unlearning effectiveness or utility resulting from the removal of the reference model. Additionally, SimNPO benefits from computational and memory efficiency, as it eliminates the need to query and store a reference model during optimization. We further conducted a comparison of the computational efficiency among Retrain, NPO, and SimNPO. As shown in **Table R4**, NPO introduces additional overhead during the unlearning process for two reasons: (1) it requires storing an extra reference model, which increases GPU memory usage; and (2) it must compute $\\log(\\pi_{\\text{ref}}(y \\mid x))$ at each training step, leading to increased computational time.\n\nTable R4: Comparison of GPU memory usage and training time for Retrain, NPO, and SimNPO on the TOFU Forget05.\n\n| Method   | Memory (GB) | Time (min) |\n|:--------:|:-----------:|:----------:|\n| Retrain  |     20      |    120     |\n| NPO      |     27      |     36     |\n| SimNPO   |     21      |     25     |\n\nHowever, it is important to emphasize that this improvement does not come from simply removing the reference model from NPO. As noted in Lines 234–237, a naive removal of the reference model from Eq. (3) in NPO reduces it to a GA (gradient ascent), which disrupts the weight smoothing effect and degrades unlearning-utility performance.\n\nIn terms of utility, the removal of the reference model in SimNPO does not lead to negative effects; on the contrary, it often results in better utility preservation compared to NPO. This improvement stems from SimNPO’s more effective gradient smoothing mechanism, which avoids the instability observed in NPO during the early stages of unlearning. This point is discussed in Line 219 (“Lack of gradient weight smoothing in the early stages of unlearning”) and supported by Figure 4.  \n\n\n# 3. Response to the existence of forgotten performance bias in NPO and under what conditions does this bias arise\nThank you for this insightful question. The reference model bias in NPO arises when the reference model misguides the unlearning process by assigning inconsistent gradient weights to forget samples of varying difficulty. \n\nTo elaborate, let us revisit the reference-dependent weighting term in NPO’s gradient (Eq. 3): $w_{\\theta}(x, y) = \\frac{2 \\, \\pi_\\theta(y \\mid x)^\\beta}{\\pi_\\theta(y \\mid x)^\\beta + \\pi_{\\text{ref}}(y \\mid x)^\\beta}$. This weighting scheme causes NPO to assign smaller gradient magnitudes to samples with higher output probabilities under the reference model, $\\pi_{\\text{ref}}(y \\mid x)$. However, such samples are actually harder to forget than those with lower $\\pi_{\\text{ref}}$. As a result, if $\\pi_{\\text{ref}}$ varies across the forget set in a way that does not faithfully reflect the true unlearning difficulty of each sample, this leads to **uneven gradient assignment** and introduces **bias into the forgetting process**.\n\nThis bias is particularly evident when forget examples vary in memorization level (Example 1) or response length (Example 2). In Example 1, NPO performs poorly on strongly memorized (harder) samples–characterized by higher $\\pi_{\\text{ref}}$--yielding a forget quality of 0.00, while achieving a forget quality of 0.47 on weakly memorized (easier) samples, as shown in Figure 1(b). A similar trend is observed in Example 2 for longer-response forgetting data vs. shorter data (Figure 2).\n\n# 4. Response to new bias from length normalization in SimNPO\n\nThank you for your insightful and thought-provoking question. \n\nUsing length normalization in SimNPO does **not introduce** bias with respect to sample length; rather, it **mitigates** such bias in NPO from forgetting data with different lengths. As explained in Response 3,  short data samples tend to have **higher $\\pi_{\\text{ref}}$**, they are generally **harder to unlearn**. In the SimNPO adaptive weight formulation: $w_{\\theta}(x, y) = \\frac{2 \\, \\pi_\\theta(y \\mid x)^\\beta}{\\pi_\\theta(y \\mid x)^\\beta + 1} \\cdot \\frac{1}{|y|}$ this weight assignment naturally gives **larger gradient weights  to shorter samples**, thereby compensating for their higher unlearning difficulty. As shown in **Figure 2**, the forget quality of **NPO** on short vs. long samples is **0.47 vs. 0.81**, indicating substantial bias. In contrast, **SimNPO** achieves **0.91 vs. 0.97**, significantly reducing this gap and demonstrating better balance across sample lengths.\n\nTo further validate the usefulness of length normalization, we conduct additional experiments in **Appendix F**. When the parameter $\\beta \\rightarrow 0$, the SimNPO gradient degenerates into **length-normalized weighted GradDiff**, whereas the NPO gradient degenerates into **standard GradDiff**. It can be seen from Figure A3 that although not outperforming the original NPO or SimNPO,  the forget quality of weighted GradDiff (0.1) is **substantially higher** than that of GradDiff (1e-9), highlighting the usefulness of length normalization."}, {"review_id": "gwASz30rmU", "reviewer": "Reviewer_s2gF", "summary": "This paper addresses limitations of an existing loss for LLM unlearning (NPO) and proposes to remove the dependency on a reference model (the model prior to unlearning). The improved loss shows better unlearning vs. utility performance on three LLM unlearning datasets and provides more robustness against relearning attacks.", "questions": "Q1) I assume that dropping the dependency on a reference model improves efficiency (as also discussed by SimPO). Does SimDPO improve efficiency over NPO in terms of e.g. runtime or memory-usage?\n\nQ2) What does \"ununlearn\" mean in Fig1 a)?\n\n*Typo*: gredient -> gradient (L235)", "limitations": "Yes, in the Appendix.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- In general, the paper is in good shape in terms of writing quality (with exceptions highlighted below).\n- The paper provides sufficient empirical evidence that SimNPO is better than baselines considering the unlearning-utility trade-off in LLM unlearning.\n- The paper also provides interesting findings, in particular provides evidence that SimNPO is more robustness to relearning attacks.\n\n**Weaknesses**\n\nW1) The approach of addressing and overcoming limitations of existing methods in unlearning is generally good. The problem here is that the discussion of the identified limitations in section 4 requires more clarity. In particular:\n- Regarding Limitation 1: It is unclear what the term \"unlearning power\" refers to. Do you mean the gradient is too small when the reference model is highly certain (and vice versa)? This term needs to be clarified in more technical terms, especially because the \"unlearning power\" and \"optimization power\" are used later on in the paper as well.\n- Regarding Limitation 2: The adaptive weight is close to 1 in the first epoch, and the paper suggests that this has a bad effect on the utility. However, Figure 3 d) shows no change in utility after the first epoch. Maybe studying epoch 2 (when utility starts to drop) would be more effective for this argument. The main problem of this paragraph is, however, that it remains unclear if the reference model is really the underlying issue (as claimed in L220) or rather the missing length-normalization. \n\nW2) The method boils down to a simple fix that has been proposed already in a different context (preference optimization / SimPO), which limits the technical contribution of this paper. Especially the critical length-normalization has also been discussed in the SimPO paper already. While the empirical results are strong and potentially helpful for the unlearning community, the paper would benefit from a stronger discussion on the differences introduced by the unlearning context. For example, the gamma term introduced in SimPO does not seem to be required for unlearning but remains part of the SimNPO loss.", "was_revised": true, "final_justification": "My concerns regarding Weakness 1 have been resolved during the discussion phase. My final assessment is based on the weakness (W2) that the broader impact of this work is rather limited since the scope of this work is only to improve an existing unlearning method (NPO). This concern is also shared with two other reviewers (Reviewer JEx6, Reviewer BfAc). Still, the work appears significant for the unlearning community, thus my increased score and final assessment.", "rebuttal": "We sincerely thank Reviewer s2gF for the thoughtful and constructive feedback. Below, we provide detailed responses to each of the key questions raised.\n\n# 1. Response to the unlearning power or optimization power in Limitation 1\nWe apologize for the lack of clarity in our original writing. In this context, both \"unlearning power\" and \"optimization power\" refer to the adaptive weight allocated for forgetting data gradients.\n\nThis can be seen by examining the adaptive weight used in the NPO gradient (Eq. 3): $w_{\\theta}(x, y) = \\frac{2 \\, \\pi_\\theta(y \\mid x)^\\beta}{\\pi_\\theta(y \\mid x)^\\beta + \\pi_{\\text{ref}}(y \\mid x)^\\beta}$. This weighting scheme causes NPO to assign lower gradient weights to samples with higher output probabilities under the reference model, $\\pi_{\\text{ref}}(y \\mid x)$. However, samples with higher $\\pi_{\\text{ref}}$ are actually harder to forget than those with lower reference probabilities. As a result, if $\\pi_{\\text{ref}}$ varies across the forget set in a way that does not accurately reflect the true difficulty of unlearning a sample, this leads to uneven and biased gradient weight allocation during unlearning optimization. \n\nThis bias is particularly evident when forget examples vary in memorization level (Example 1) or response length (Example 2). For example, in Example 1, NPO performs poorly on strongly memorized (harder) samples–characterized by higher $\\pi_{\\text{ref}}$--yielding a forget quality of 0.00, while achieving a forget quality of 0.47 on weakly memorized (easier) samples, as shown in Figure 1(b). \n\nWe will revise the manuscript to clarify this terminology and ensure consistency throughout.\n\n# 2. Response to Limitation 2\nThank you for this thoughtful question. We would like to clarify that the effect of biased gradient weights on utility may not manifest immediately after the first epoch. Unlearning is a progressive process in which the model gradually diverges from the original (pre-unlearning) distribution. As such, **there can be a lag between early biased updates and their downstream impact on utility**, which may become apparent only after a few optimization steps (e.g., around epoch 2 or 3 as noted).\n\nImportantly, the core message in Limitation 2 is why the early unlearning dynamics are suboptimal for utility. As mentioned in Line 223, our analysis suggests that during the early training epochs, NPO’s adaptive weights ($w \\approx 1$) are largely insensitive to variations in the difficulty of the forgetting data, such as differences in memorization strength or response length. This leads to uniform unlearning updates, failing to properly prioritize harder-to-forget examples.\n\n\n\nFurthermore, **the reference model is indeed the underlying issue**, as it serves as the mechanism through which unlearning guidance might be unevenly applied across forgetting data of varying difficulty. To elaborate, let us revisit the reference-dependent weighting term in NPO’s gradient (Eq. 3): $w_{\\boldsymbol{\\theta}}(x, y) = \\frac{2 \\, \\pi_\\theta(y \\mid x)^\\beta}{\\pi_\\theta(y \\mid x)^\\beta + \\pi_{\\text{ref}}(y \\mid x)^\\beta}$. The influence of the reference model shapes the gradient weights in a way that disproportionately favors easier-to-forget samples (i.e., those with lower output probabilities $\\pi_{\\text{ref}}(y \\mid x)$), thereby introducing bias into the unlearning process, as discussed in Response 1.\n\nAs a result, if the reference model $\\pi_{\\text{ref}}$ varies across the forget set in a way that does not accurately reflect the true difficulty of unlearning a sample, this leads to uneven and biased forgetting.  For example, even when **forgetting data samples have similar response lengths, the reference model bias persists if they differ in memorization levels**. In Example 1, NPO performs poorly on strongly memorized (harder) samples, characterized by higher $\\pi_{\\text{ref}}$ values, achieving a forget quality of only 0.00. In contrast, it attains a forget quality of 0.47 on weakly memorized (easier) samples, as shown in Figure 1(b). \n\nThe above illustrates that the bias induced by the reference model is not merely a function of response length, but is fundamentally tied to how the model prioritizes forgetting across varying difficulty levels. \n\nLastly, we conducted additional experiments using a modified NPO objective that incorporates length normalization, in order to examine whether it can directly address the reference model bias in NPO. The NPO variant with length normalization (denoted as NPO-LN) is defined as: $\\ell_{NPO\\text{-}LN}(\\theta) = -\\frac{2}{\\beta} \\log \\sigma \\left( -\\frac{\\beta}{|y|} \\log \\left( \\frac{\\pi_\\theta(y \\mid x)}{\\pi_{\\text{ref}}(y \\mid x)} \\right) \\right)$.\n\nFollowing the setting of $Example 1$, we construct a controlled scenario where samples within the forget set exhibit **different levels of memorization**, where the subset $D_{f,1}$ is strongly memorized and the remaining forget set $D_{f,2}$ is minimally memorized by the reference model. \n\nWe then perform unlearning over $D_{f,1} \\cup D_{f,2}$ using NPO, NPO-LN, and SimNPO. As shown in **Table R5**, although NPO-LN improves upon the original NPO, it still underperforms SimNPO. This suggests that the reference model issue is only partially mitigated by length normalization, and SimNPO remains more effective in addressing this bias.\n\nTable R5: Comparison across NPO, NPO-LN, and SimNPO on TOFU Forget05.\n\n| **Method**   | **Forget Quality on $D_{f,1}$** | **Forget Quality on $D_{f,2}$** | **Model Utility** |\n|:------------:|:----------------------------------------------:|:----------------------------------------------:|:------------------:|\n| NPO          |                     0.00                       |                    0.47                        |       0.49         |\n| NPO-LN       |                     0.10                       |                    0.55                        |       0.51         |\n| **SimNPO**   |                 **0.70**                       |                  **0.70**                      |     **0.57**       |\n\n\n# 3. Response to novelty and hyperparameter $\\gamma$\n\nWhile SimNPO may appear to be a simple methodological modification to SimPO, we respectfully emphasize that our work represents a novel contribution tailored to the domain of **LLM unlearning**, a substantially different problem space than the context originally considered in SimPO. Specifically, SimNPO was motivated, analyzed, and rigorously validated within the unlearning context: Our work fundamentally rethinks the design of unlearning objectives by identifying and overcoming the reference model bias in NPO. \n\nIn particular, we are the first to (1) uncover the disproportionate gradient influence imposed by the reference model across forgetting data of varying difficulty (as exemplified via memorization level and response length), and (2) empirically and theoretically characterize how this bias leads to suboptimal forgetting outcomes. These findings have not been addressed in prior work. Our contributions go beyond methodological substitution. We provide a systematic investigation, including theoretical analysis via a Markov chain mixture model, empirical validation across multiple benchmarks, and robustness studies.\n\nIn addition, we kept $\\gamma$ in SimNPO's objective for ease of comparison with SimPO. However, we have clearly stated in Line 252 that $\\gamma$ is not used (i.e., $\\gamma = 0$), and we provided the rationale for this choice in Lines 252–256, explaining why it is not necessary for the SimNPO objective. \n\n# 4. Response to the computation efficiency and term “Ununlearn”\nThank you for your insightful suggestion. In response, we conducted a comparison of the computational efficiency among Retrain, NPO, and SimNPO. As shown in **Table R6**, NPO introduces additional overhead during the unlearning process for two reasons: (1) it requires storing an extra reference model, which increases GPU memory usage; and (2) it must compute $\\log(\\pi_{\\text{ref}}(y \\mid x))$ at each training step, leading to increased computational time.\n\nTable R6: Comparison of GPU memory usage and training time for Retrain, NPO, and SimNPO on the TOFU Forget05.\n\n| Method   | Memory (GB) | Time (min) |\n|:--------:|:-----------:|:----------:|\n| Retrain  |     20      |    120     |\n| NPO      |     27      |     36     |\n| SimNPO   |     21      |     25     |\n\nIn addition, we apologize for the confusion caused by the unclear use of the term \"ununlearn\" in the manuscript. In our context, \"ununlearn\" refers to the original model before any unlearning is performed."}]

Author rebuttal:

Meta-review: This paper revisits Negative Preference Optimization in the context of LLM unlearning, showing that simpler strategies can be surprisingly effective compared to NPO. The authors provide systematic analysis, clear methodology, and empirical evidence across multiple unlearning benchmarks. Reviewers found the paper well written and the empirical results convincing, though they also noted that the contribution is incremental, focusing on re-evaluating design choices rather than introducing fundamentally new techniques. The rebuttal further clarified the experimental setup and strengthened the empirical case. Overall, the paper is solid, clear, and practically useful, though the contribution is not groundbreaking. I recommend acceptance.
Decision: Accept (poster)