Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

Authors: Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, Sijia Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Unlearning, Large language model, Preference optimization

Abstract:
In this work, we address the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences and associated model capabilities e.g., copyrighted data or harmful content generation) while preserving essential model utilities, without the need for retraining from scratch. Despite the growing need for LLM unlearning, a principled optimization framework remains lacking. To this end, we revisit the state-of-the-art approach, negative preference optimization (NPO), and identify the issue of reference model bias, which could undermine NPO's effectiveness, particularly when unlearning forget data of varying difficulty.  Given that, we propose a simple yet effective unlearning optimization framework, called SimNPO, showing that  `simplicity' in removing the reliance on a reference model (through the lens of simple preference optimization) benefits unlearning. We also provide deeper insights into SimNPO's advantages, supported by analysis using mixtures of Markov chains. Furthermore, we present extensive experiments validating    SimNPO's superiority over existing unlearning baselines in benchmarks like TOFU and MUSE, and robustness against relearning  attacks.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730721732004

thread_type:
official_review

<thread_object>
creator: reviewer_fdki

<content>
summary:
The paper proposes an innovative approach to Large Language Model (LLM) unlearning, called SimNPO, designed to remove undesirable data influences efficiently. This method advances the state-of-the-art Negative Preference Optimization (NPO) framework by addressing a key limitation: reference model bias. The authors introduce a reference-free approach, simplifying preference optimization without relying on a baseline model, which enhances unlearning robustness while retaining model utility. Extensive experiments on benchmarks like TOFU and MUSE demonstrate SimNPO's effectiveness in surpassing existing unlearning baselines, and it shows promise in defending against relearning attacks.

strengths:
1. SimNPO provides a unique, reference-free framework that improves upon the limitations of NPO, specifically addressing reference model bias, making it a significant advancement in the field of unlearning for LLMs.

2. The approach is computationally efficient, allowing for effective unlearning without requiring full model retraining, which is resource-intensive and often impractical.

3. The method is tested on diverse datasets (TOFU, MUSE, WMDP), and results are detailed, showcasing the framework’s strong performance in preserving model utility and achieving high forget quality.

weaknesses:
1. The technical contributions are sound but limited. The authors propose a reference-free optimization framework. However, optimization with a reference-free objective is not novel and has already been proposed by SimPO. Moreover, the length-normalized reward and reward margin parameter in SimNPO are also derived from SimPO.

2. SimNPO introduces two additional hyperparameters that must be manually adjusted to achieve optimal performance, significantly increasing the complexity and time cost of experiments. This greatly limits the potential application of this method for large language models in real-world settings.

3. Is there any theoretical analysis or intuition? What is the divergence rate of SimNPO compared to NPO? The current version of this paper only provides an experimental report without solid analysis.

4. A potential issue is over-reliance on the reference model; would adjusting the weight of the reference model help mitigate this problem? While removing the reference model entirely might solve the over-reliance issue, could it lead to insufficient performance for certain cases, such as math and coding datasets?

questions:
Please see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1731045561113

thread_type:
official_review

<thread_object>
creator: reviewer_hsyz

<content>
summary:
This paper focuses on LLM unlearning, involving removing unwanted data influences or specific model capabilities from pre-trained models without needing to retraining from scratch. The authors state that NPO, the current state-of-the-art, suffers from reference model bias, and propose a new optimisation framework named SimNPO. SimNPO removes the dependence on the defence model by leveraging a reference-free optimisation method named SimPO, further normalising the loss for the varying lengths of strings. The authors conducted extensive experiments over a series of well established benchmarks, including TOFU, MUSE, WMDP, where the results show the superior performance of SimNPO over NPO and many other advanced methods.

strengths:
The paper identifies and thoroughly analyzes the drawbacks inherent in NPO, seemingly providing some new insights.

By leveraging SimPO, SimNPO offers a simple yet effective solution that removes the reliance on a reference model, leading to more controlled and efficient unlearning.

Experiments show that SimNPO consistently achieves higher forget quality across various benchmarks, especially in scenarios with varying data difficulties.

SimNPO better preserves the model's utility on the retain set compared to NPO, maintaining performance on essential tasks.

weaknesses:
“Divergence” seems to be an important terminology that is frequently mentioned throughout the manuscript, while I cannot find its clear definition. Similar issues appear for the terminology of "reference model bias". Do the authors indicate that the \pi_ref is inaccurate and thus the estimation of the weight in Eq 3 is inaccurate? Kindly please provide more detailed descriptions. 

In L1 of Sec 4, the authors state that the reliance on \pi_ref lead to an uneven distribution of unlearning power. However,  it seems that \pi_ref allocates different attention for different strings: when we consider two strings with similar values of \pi_\theta, then the string with large \pi_ref will have smaller attention. Therefore, \pi_\theta can characterise the unlearning difficulty. The following example is also very weird, the authors assume a forget sample that has already been unlearned in \pi_ref. Generally, if a data point has been unlearned (or does not appear in the original model), why we further need unlearning. 

I have another concern in L1, does the short string is generally harder to be unlearned than a long string? The validation experiments should be conducted across a broad range of current unlearning methods and difference metrics (as some metrics may have bias towards short / long strings). Personally, I think it is an important experiments because if the reverse is true, NPO can actually address the difficulties in unlearning long strings. 

In L2, I wonder in what condition we can have an optimal gradient weight smoothing scheme. Also, does the proposed method can fulfil this goal?

I think the paper may some what overclaim its contribution. It seems that the main claim of this paper is that current methods might not be effective in handling short strings unlearning, motivating the paper to propose a new method that explicitly pay more attentions to those long strings. The current organisation of the paper is not clear, especially considering the fact that the authors do not discuss what kinds of the weighting mechanism is of our ultimate interest. 

The authors claim that NPO suffers from over-unlearning. However, comparing the results in Table 3 between NPO and SimNPO for MUSE News, it seems that SimNPO faces the problem of under-unlearning.

To address L1 and L2, a simpler weighting strategy is \frac{1}{|y|}\cdot \pi_\theta (y|x), where  \frac{1}{|y|} handles L1 and \pi_\theta (y|x) handles L2. Compared with such a simple baseline, what are the superiority of the proposed method.

questions:
Kindly please refer to the Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1731052018017

thread_type:
official_review

<thread_object>
creator: reviewer_ojtb

<content>
summary:
This paper focuses on large language model unlearning (LLM unlearning). Specifically, this work revisits a conventional optimization method, i.e., negative preference optimization (NPO), and identifies the issue of reference model bias that may hinder the performance of unlearning. To this end, this work proposes a new approach, namely, SimNPO, that extends NPO with a reference-free optimization framework like SimPO. Various experiments about the problem illustration and the performance evaluation of the newly proposed SimNPO are conducted to demonstrate the rationality of the proposal.

strengths:
1. This work identifies one important issue with the conventional LLM unlearning method, NPO, regarding its reference model bias, which provides some insights into the learning behaviors of NPO on the allocation of unlearning power as well as the ineffective gradient weight smoothing.
2. It is reasonable that the SimNPO is promising in mitigating the reference model bias by considering the idea from SimPO to modify the original learning objective of NPO and the presentation also provides the insight discussion via the gradient derivation of SimNPO, as well as the further analysis of a mixture of Markov chains.
3. The experiments are conducted on three LLM unlearning benchmarks, which demonstrate the proposed method's general effectiveness.

weaknesses:
1. Some statements and claims are not very clear and sometimes confusing. It would be better if the authors could improve the explanation or discussion for some words or sentences. The specific questions can refer to the Questions part.
2. The current presentation is not very good as it is difficult to understand some critical parts of the problem illustration or justification with long-distance cross-reference. For example, there are no related descriptions before or near the position where Fig.1 existed for the first time, and the corresponding caption can not fully explain all the meanings of the figure. Generally, it is hard to understand the major motivation (related to reference model bias) in the introduction part. 
3. It seems that SimNPO is proposed based on NPO and adopts the SimPO to mitigate the reference model bias. The current presentation provides limited unique problems under the LLM unlearning scenario. In other words, SimPO is a reference-free preference optimization method, it can also mitigate the reference model bias if it exists in the framework of DPO. However, it is unclear about the unique novelty and insights of adopting SimPO to improve NPO in both perspectives of research question and methodology design.
4. The identified issue of NPO is reference model bias, while the reference model bias is not clearly illustrated in Section 4. For example, although we can get that both the forget quality and model utility of SimNPO would be better than NPO, there are only statistics of NPO at Figure 3 (a)-(b). Lacking of comparison makes it hard to intuitively get the meaning of blind allocation and ineffective gradient weight smoothing.

questions:
1. I do not very understand what is the meaning of the statement "a principled optimization framework remains lacking" in the abstract. Although SimNPO revisited the NPO and improved it, would SimNPO be the principled optimization framework? 
2. Could the author provide some explanation or illustration of the word "divergence" in line 55? Since it is italicized, is there any special definition or meaning for this word? And what is the meaning of "divergence-driven optimization methods"?
3. Could the author further explain the Figure 1(a)? The current version of the caption and description is hard to understand, and I can hard to get "overreliance on the reference model" as indicated in lines 76-77. It would be better if the authors could also provide some real examples instead of illustrations.
4. Are there any critical differences between the parallel improvement of SimPO from DPO and that of SimNPO from NPO? If not, does it mean that all the preference optimization methods can be applied in LLM unlearning?
5. Would the reference model bias just be an issue of a specific large language model? Could the issue generally be identified in different large language models? 
6. I wonder whether there are results about statistical significance for the performance evaluation part?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731434792101

thread_type:
official_review

<thread_object>
creator: reviewer_x96s

<content>
summary:
The authors revisited the current unlearning optimization framework, negative preference optimization (NPO), and identified its reference model bias issue, which compromises unlearning effectiveness, particularly for forget data of varying difficulty. To address this, the authors introduced SimNPO, a simple yet effective framework that eliminates reliance on a reference model by leveraging simple preference optimization. The authors provided deep insights into SimNPO’s advantages through both synthetic data analysis and evaluations on existing unlearning benchmarks such as TOFU, MUSE, WMDP, and relearning attacks.

strengths:
1. The authors revisit the NPO framework and identify its potential weakness–overreliance on the reference model–in LLM unlearning.

2. Building on insights into NPO’s limitations, the authors propose an improved LLM unlearning approach,
SimNPO, which extends NPO using a reference-free optimization framework.

3. The authors conduct extensive experiments to showcase the improvements of SimNPO over NPO across various unlearning benchmarks.

weaknesses:
1.	Unclear Motivation: The primary issue lies in the unclear motivation of the paper. While it claims that the main limitation of NPO is the inductive bias of the reference model, Section 4 does not clearly convey this point, making it difficult for readers to understand.

2.	Missing Ablation and Sensitivity Analysis: The paper lacks both an ablation study and parameter sensitivity experiments, which are necessary to evaluate the impact of different components and parameter choices on the model’s performance.

3.	Limited Technical Contribution: The proposed SimNPO framework appears to be a straightforward combination of SimPO and NPO, with no substantial novel insights. Additionally, it lacks a deeper theoretical analysis to support its approach.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732122863945

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
GR3: Clarification on technical contributions

comment:
Thank you for raising this question  (@Reviewers [x96s](https://openreview.net/forum?id=Pd3jVGTacT&noteId=YyI169NJBs), [ojtB](https://openreview.net/forum?id=Pd3jVGTacT&noteId=Njnkfs0Phh), [Fdki](https://openreview.net/forum?id=Pd3jVGTacT&noteId=3fxZV2nsCk)). While SimNPO might appear straightforward by integrating SimPO into NPO, this does not diminish the technical contributions of our work. We respectfully wish to clarify that understanding the limitations of NPO caused by reference model bias, offering insights into why a reference-free objective and length normalization are necessary, and demonstrating how SimNPO effectively addresses NPO’s limitations are far from trivial. This goes well beyond the mere application of SimPO in our proposal. Please see our detailed response below.

**(Identification of NPO’s current limitations in Sec. 4)** First, the identification of NPO’s current limitations is novel and not incremental. Prior to our work, NPO was increasingly recognized as an advantageous optimization framework for LLM unlearning due to its two key improvements–bounded loss from below and adaptive weight smoothing–over classical gradient ascent-based unlearning approaches, as detailed in Lines 165-187. However, we reveal that the advantages of NPO can be restricted due to the presence of reference model bias, which we thoroughly analyzed in Sec. 4. Specifically, we highlight two critical limitations: (L1) Blind allocation of unlearning power against different kinds of forget data points. (L2) Ineffectiveness of adaptive weight smoothing during early optimization phases. These insights into the limitations of NPO, supported by our analyses and experimental results in Sec. 4, are novel contributions to the field. 

**(Point-by-point improvement and rationale by SimNPO in Sec. 5)** Second, our focus has been on providing a deep understanding of why the improved effectiveness of SimNPO exists and demonstrating its practical performance. In Sec. 5, under the subsection “Insights into SimNPO,” we conduct a point-by-point analysis of how SimNPO addresses NPO’s limitations (L1) and (L2) in Sec. 4. Additionally, we provide further insights through analyses and experiments using a mixture of Markov chains, which substantiate our insights into the benefits of SimNPO. These insights are non-trivial and are supported by our methodological analysis (see Eq. (5) and related discussions) and experimental validations (Figs. 4-6).

Last but not the least, we have tried our best to validate the effectiveness of SimNPO across **various unlearning tasks**, as acknowledged by reviewers x96s, HSyZ, and Fdki.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732123166605

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
GR2: Clarification on the term “divergence”

comment:
Thank you very much for sharing your concerns regarding the clarity of some of our terminologies, such as “divergence.” (@Reviewers [ojtB](https://openreview.net/forum?id=Pd3jVGTacT&noteId=Njnkfs0Phh), [HSyZ](https://openreview.net/forum?id=Pd3jVGTacT&noteId=7Oox38ISD0), [Fdki](https://openreview.net/forum?id=Pd3jVGTacT&noteId=3fxZV2nsCk)). To address this and prevent any misunderstanding in the paper, we would like to reiterate key points from the original submission and provide further explanations for better clarity. While we initially believed that some terms were standard in the context of our work, we now recognize that they may lead to confusion without additional elaboration. We sincerely appreciate the opportunity to provide this clarification.

**(Explanation of divergence)** The term “divergence” refers to optimization divergence from the pre-trained state, which describes the process of **deviating from the converged pre-trained model state to reverse the existing learning of the forgotten data**, thereby achieving unlearning. In Lines 220-223, we provide one explanation of this concept: the divergence drives the optimization process to widen the gap between the prediction probability and the reference model on the forget set. In our paper, we overlooked the detailed definition of this term initially, as it is commonly used in the literature, for example, the over-divergence issue (i.e., model collapse) of gradient ascent (GA) identified in the NPO paper. To avoid potential confusion, we will clarify and explicitly define this term in the revised version of the paper.

**(Discussion on theoretical analysis for divergence rate and additional experimental validation)** This is an insightful question that inspires us to consider what kind of theoretical guarantees SimNPO might hold. One potential theoretical result we could establish is the slower optimization divergence rate achieved by SimNPO. Similar to NPO (Theorem 2, Zhang et al., 2024a), we think that **SimNPO could achieve a logarithmic divergence against the unlearning optimization iterations $T$**, as opposed to the linear divergence rate observed with gradient ascent (GA). In a broader context of theoretical analyses, the field of LLM unlearning currently lacks well-established frameworks or tools that can be readily employed by researchers, including ourselves, to derive rigorous guarantees on the optimization influence in this domain. While we acknowledge that deeper theoretical analysis would further strengthen our findings and is always desirable, this remains an open and important question for future research. We will incorporate this discussion into the conclusion and limitations sections of our paper to reflect the current gaps and potential directions for advancing the theoretical analyses of SimNPO and LLM unlearning.

Inspired by the reviewers’ comment, we conducted additional experiments to validate the slower divergence rate of SimNPO compared to GA. Specifically, we measured the **KL divergence on the forget set (TOFU Forget05) between the unlearned model and the original model**. The results, presented in **[Fig. R1](https://ibb.co/TgcqWSR)** are consistent with our hypothesis that SimNPO, like NPO, exhibits a logarithmic divergence rate, in contrast to the linear divergence rate observed with GA. These findings further support the theoretical foundation of SimNPO's optimization behavior.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732123447874

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
GR1: Clarification on Reference Model Bias and Improvements (Part 2/2)

comment:
**(L2: Gradient weight smoothing bias caused by the reference model)** Another limitation caused by the reference model bias lies in the gradient weight smoothing scheme in NPO, $w_{\boldsymbol \theta}(x,y)$ in Eq. 3. This limitation is experimentally justified in Fig. 3. As described in Lines 260–264, since the initially unlearned model $\boldsymbol \theta$ is given by the reference model, the gradient weight smoothing lacks effectiveness during early unlearning epochs, as **$w_{\boldsymbol \theta}(x,y) \approx 1$**, as shown in Fig. 3-(a, b). This limitation causes NPO to behave similarly to the conventional gradient ascent (GA) method during early optimization, potentially leading to over-forgetting (and even model collapse), a well-known issue for GA. This behavior is further illustrated in Fig. 3-(d), which shows a significant utility drop during the early optimization phase.

**(Additional experiment to improve the motivation of reference model bias)** We also conducted a new motivating experiment to further support our claims from a data memorization perspective. Our rationale is to enforce the reference model to memorize certain forget data points and then investigate how the unlearning optimization responds to these strongly-memorized forget points compared to least-memorized forget points. Specifically, we use TOFU Forget05 as the forget set $D_{f}$, splitting it evenly into $D_{f1}$ and $D_{f2}$. Here the divided $D_{f1}$ and $D_{f2}$ follow the same distribution of fictitious author information in TOFU. We finetune the LLaMA-2 7B chat model on the original retain set of TOFU together with $D_{f1}$, i.e., $D_{\text{retain}} \cup D_{f1}$, to obtain the original model (i.e., the reference model) before unlearning. The resulting reference model then strongly memorizes $D_{f1}$ but least memorizes $D_{f2}$ although they are drawn from the same distribution of fictitious authors. We then perform unlearning using SimNPO and NPO over $D_{f1} \cup D_{f2}$. We present unlearning performance, in terms of FQ (forget quality) and model utility, in **Table R1**. 

**Table R1: Performance of TOFU using the LLaMA-2 7B chat model. The content format follows Table 2.**

|          | FQ on $D_{f1}$ | FQ on $D_{f2}$ |  Utility  |
|:--------:|:--------------:|:--------------:|:----:|
| Original |      0.00      |      0.00      | 0.62 |
|    NPO   |      0.32      |      0.69      | 0.56 |
|  SimNPO  |      0.70      |      0.72      | 0.59 |

As shown in Table R1, since the original model was trained on $D_{f1}$, its prediction loss $-\log (\pi_{\text{ref}})$ on $D_{f1}$ is relatively small, leading to a higher prediction probability $\pi_{\text{ref}}$ on $D_{f1}$. Consequently, the NPO gradient smoothing term in Eq. (3) becomes relatively smaller for $D_{f1}$ due to the reference model bias $\pi_{\text{ref}}$ on $D_{f1}$. As a result, NPO allocates less first-order optimization power to $D_{f1}$ (due to smaller weight before gradient) and focuses more on $D_{f2}$. This imbalance leads to better forget quality (FQ) for NPO on $D_{f2}$ compared to $D_{f1}$. However, $D_{f1}$ should ideally receive more unlearning power since it was strongly memorized before unlearning. In contrast, SimNPO, by leveraging a reference-model-free reward, achieves a much smaller FQ difference between $D_{f1}$ and $D_{f2}$ and delivers higher FQ for both datasets compared to NPO. Furthermore, SimNPO demonstrates superior model utility relative to NPO, highlighting the effectiveness of our approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732123539763

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
GR1: Clarification on Reference Model Bias and Improvements (Part 1/2)

comment:
Thank you for raising this question (@Reviewers [x96s](https://openreview.net/forum?id=Pd3jVGTacT&noteId=YyI169NJBs), [ojtB](https://openreview.net/forum?id=Pd3jVGTacT&noteId=Njnkfs0Phh), [Fdki](https://openreview.net/forum?id=Pd3jVGTacT&noteId=3fxZV2nsCk)). We apologize if our original presentation lacked clarity. We would like to kindly point out that our intention was to introduce **the reference model bias** issue at a high level in Fig. 1-(a) and then elaborate on it from **two key aspects** in Sec. 4: (L1) The issue of **unlearning power allocation**, where referencing the reference model fails to account for data-wise unlearning difficulty. (L2) The potential ineffectiveness of **gradient weight smoothing** when referenced to the reference model, particularly during early optimization stages. Given these analyzed limitations and their experimental justifications, we then elaborated on the one-to-one correspondence between the advantages introduced by SimNPO and the specific limitations it addresses. These improvements were further validated using a mixture of Markov chain analysis. 

In the rebuttal, as reviewers expressed concerns regarding the clarity of the reference model bias, we provided further detailed explanations below to enhance the understanding of our work.


**(L1: Misallocation of unlearning power in NPO)** Recall that the minimization of NPO’s objective (Eq. (2)) drives the unlearning process by widening the gap between the unlearned model and the reference model, as explained in Lines 220–227. This process inherently relies on the sensitivity of the reference model $\pi_{\text{ref}}$ to the forget data samples. However, different samples elicit different responses from the reference model $\pi_{\text{ref}}$. For samples with a larger prediction loss, $-\log(\pi_{\text{ref}})$, the prediction probability $\pi_{\text{ref}}$ is smaller and often closer to the learn/unlearn boundary. Such samples are easier to unlearn and require less unlearning power compared to more challenging samples. However, in NPO, **even these easy samples may be assigned excessive power to unnecessarily enlarge the distance to the reference model**. This misallocation of unlearning power becomes uneven, disadvantaging the more difficult samples.

The presented Fig. 1-(a) and the discussion on (L1) in Sec. 4 were specifically designed to illustrate and reveal the above point. For instance, the reference model prediction $\pi_{\text{ref}}$ could vary across different samples **(as shown by the green line in Fig. 1-(a))**. For some samples, $\pi_{\text{ref}}$ is already near the learn/unlearn boundary, making these "easy examples" for unlearning in Fig. 1-(a). In such cases, aggressively enlarging the gap with the reference model for these samples becomes unnecessary. Yet, NPO may continue to increase the distance between $\pi_{\theta}$ and $\pi_{\text{ref}}$ **(as shown by the blue line in Fig. 1-(a))**. As a result, easy examples end up far beyond the boundary, while harder examples fail to cross it, as depicted in Fig. 1-(a). 


 In addition to the schematic illustration in Fig. 1-(a), Fig. 2 provides experimental justification for (L1) by analyzing unlearning performance across different types of forget data points, categorized based on their response lengths. The motivation is elaborated in Lines 234-239, highlighting that the reference model may exhibit a bias toward generating longer but lower-quality sequences, as noted in (Meng et al., 2024). Consequently, these low-quality, long texts tend to be easier to unlearn. Accordingly, this optimization power allocation could become uneven for unlearning short-response forgotten data points, Indeed, Fig. 2 shows that NPO exhibits a worse unlearning performance, in terms of a greater distance from Retrain, when unlearning the top 50% shortest-length forget data than the longer 50% of the forget set. Analyzing results in Fig. 6 via a mixture of Markov chains also validated the above points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732123587957

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (GR)

comment:
In the following, we provide several general responses (GRs) to address common questions raised by multiple reviewers. We will also incorporate the corresponding revisions into the revised paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732476469482

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Upload the revised paper to the supplementary material.

comment:
We sincerely thank all the reviewers for their valuable feedback on our paper. Following your suggestions, we have made further revisions to the manuscript. To facilitate comparison with the original version, we have uploaded the revised paper to the Supplementary Material, with the main changes highlighted in red. We have added more explanations for technical terms and conducted additional experiments, including divergence and reference model bias analyses. We look forward to your further feedback and suggestions!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733108760109

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal summary to all reviewers and ACs

comment:
We sincerely appreciate the valuable feedback from all reviewers and the ACs' management of our submission. During the rebuttal phase, we have made substantial efforts to address the reviewers' problems and improve our submission. Below is a summary of our rebuttal, with the revised manuscript included in the uploaded supplementary material.

**Reviewer x96s:**
1. We added **[new experiments](https://openreview.net/forum?id=Pd3jVGTacT&noteId=Z9cYtRqSgN)** on unlearning vs. data memorization (Table R1) to further [clarify our motivation and the reference model bias](https://openreview.net/forum?id=Pd3jVGTacT&noteId=aouVi1kVTz); Please also see the revised introduction and Sec. 4  
2. We elaborated on [the ablation study and analyses about hyperparameters](https://openreview.net/forum?id=Pd3jVGTacT&noteId=0cbuT3Cu9T) that have been presented in the paper.  
3. We further clarified our [contributions](https://openreview.net/forum?id=Pd3jVGTacT&noteId=zXCzQUEhLG).  

**Reviewer ojtB:**
1. We clarified the concept of [divergence](https://openreview.net/forum?id=Pd3jVGTacT&noteId=1dU2FIB9C9), together with **[new experiments](https://ibb.co/TgcqWSR)** about divergence rate (Figure R1); Please also see the revised introduction in the revised manuscript.  
2. We provided further explanation of calling [principled optimization framework](https://openreview.net/forum?id=Pd3jVGTacT&noteId=MDkg0ZMyzF) in the original submission; Please also see the revised introduction.  
3. We further clarified [Fig. 1-(a)](https://openreview.net/forum?id=Pd3jVGTacT&noteId=XJKnbKnJRp).  
4. We further elaborated on [the novelty of SimNPO](https://openreview.net/forum?id=Pd3jVGTacT&noteId=XAeoL8CtzT).  
5. Through **[new experiments](https://openreview.net/forum?id=Pd3jVGTacT&noteId=Z9cYtRqSgN)** on unlearning vs. data memorization (Table R1), we further explained [the reference model bias and provided a detailed analysis of Fig. 3  (a)-(b)](https://openreview.net/forum?id=Pd3jVGTacT&noteId=WDXiDb1VJt). Please also see the revised Sec. 4.
6. We conducted the **[suggested experiments](https://openreview.net/forum?id=Pd3jVGTacT&noteId=FqLRl7AThl)** (Table R2)  to demonstrate the effectiveness of our method on additional models.  

**Reviewer HSyZ:**
1. We offered further explanation of [divergence](https://openreview.net/forum?id=Pd3jVGTacT&noteId=ynw7r2EfTF) through **[new experiments](https://ibb.co/TgcqWSR)** about divergence rate (Figure R1). Please also see the revised introduction.  
2. We further clarified and expanded our explanation of the [reference model bias](https://openreview.net/forum?id=Pd3jVGTacT&noteId=itX4f1kYnc) and conducted **[new experiments](https://openreview.net/forum?id=Pd3jVGTacT&noteId=Z9cYtRqSgN)** on unlearning vs. data memorization (Table R1) to validate that. Please also see the revised Introduction and Sec. 4.   
3. We further elaborated on [the role of $\pi_{\text{ref}}$ in NPO](https://openreview.net/forum?id=Pd3jVGTacT&noteId=iljQ7XpXqY).  
4. We further discussed the challenges posed by [short and long samples](https://openreview.net/forum?id=Pd3jVGTacT&noteId=elrRj0Yy49) in NPO with the support of **[new experiments](https://openreview.net/forum?id=Pd3jVGTacT&noteId=EjD8qKrlIY)** (Table R5) and **[new figure](https://ibb.co/sWdxW97)** (Figure R2).  
5. We discussed [optimal weight smoothing](https://openreview.net/forum?id=Pd3jVGTacT&noteId=pF2PCONoYC).  
6. We further clarified the [contributions](https://openreview.net/forum?id=Pd3jVGTacT&noteId=14SC25oAyO) of our work.  
7. According to the [experiments](https://openreview.net/forum?id=Pd3jVGTacT&noteId=4Cn4pgJvFl) already presented in the paper, we discussed and compared our proposal with a simpler weighting strategy.  

**Reviewer Fdki:**
1. We further clarified the [contributions](https://openreview.net/forum?id=Pd3jVGTacT&noteId=ntqPWnd6gu) of our work.  
2. As validated by the [experiments](https://openreview.net/forum?id=Pd3jVGTacT&noteId=RHW87Plmqw) in the paper, we demonstrated that SimNPO does not introduce additional hyperparameter complexity.  
3. We added new discussion and **[new experiments](https://ibb.co/TgcqWSR)** (Figure R1) to explain and validate the [divergence rate of SimNPO](https://openreview.net/forum?id=Pd3jVGTacT&noteId=lStyM4hCNG).  
4. We incorporated **[additional mathematical benchmarks](https://openreview.net/forum?id=Pd3jVGTacT&noteId=u8ZFZpIDzA)**, such as GSM8K, to further validate the effectiveness of SimNPO in Table R3 and R4.  

We sincerely hope the reviewers recognize the substantial efforts we have made in our rebuttal and paper revisions, and find our responses helpful in addressing their previous comments. We believe these efforts highlight the value of our work and warrant a reconsideration of the ratings to better reflect its contributions. We hope our rebuttal and revision efforts could be acknowledged by all reviewers and thoughtfully considered in the evaluation of our submission.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors revisited the current unlearning optimization framework, negative preference optimization (NPO), and identified its reference model bias issue, which compromises unlearning effectiveness, particularly for forget data of varying difficulty. To address this, the authors introduced SimNPO, a simple yet effective framework that eliminates reliance on a reference model by leveraging simple preference optimization. The authors provided deep insights into SimNPO’s advantages through both synthetic data analysis and evaluations on existing unlearning benchmarks such as TOFU, MUSE, WMDP, and relearning attacks. For the strengths, the authors revisit the NPO framework and identify its potential weakness–overreliance on the reference model–in LLM unlearning. Building on insights into NPO’s limitations, the authors propose an improved LLM unlearning approach, SimNPO, which extends NPO using a reference-free optimization framework. The authors conduct extensive experiments to showcase the improvements of SimNPO over NPO across various unlearning benchmarks. However, there are several points to be further improved. The paper may somewhat overclaim its contribution. It seems that the main claim of this paper is that current methods might not be effective in handling short strings unlearning, motivating the paper to propose a new method that explicitly pay more attentions to those long strings. The current organisation of the paper is not clear, especially considering the fact that the authors do not discuss what kinds of the weighting mechanism is of our ultimate interest. The authors claim that NPO suffers from over-unlearning. However, comparing the results in Table 3 between NPO and SimNPO for MUSE News, it seems that SimNPO faces the problem of under-unlearning. Some statements and claims are not very clear and sometimes confusing. It would be better if the authors could improve the explanation or discussion for some words or sentences. Moreover, the current presentation is not very good as it is difficult to understand some critical parts of the problem illustration or justification with long-distance cross-reference. For example, there are no related descriptions before or near the position where Fig.1 existed for the first time, and the corresponding caption can not fully explain all the meanings of the figure. Besides, it is unclear about the unique novelty and insights of adopting SimPO to improve NPO in both perspectives of research question and methodology design. The identified issue of NPO is reference model bias, while the reference model bias is not clearly illustrated in Section 4. For example, although we can get that both the forget quality and model utility of SimNPO would be better than NPO, there are only statistics of NPO at Figure 3 (a)-(b). Lacking of comparison makes it hard to intuitively get the meaning of blind allocation and ineffective gradient weight smoothing. Therefore, this paper cannot be accepted at ICLR this time, but the enhanced version is highly encouraged to submit other top-tier venues.